I really should replace the batteries in this guy (freaking sweet UPS by the way, highly recommended) so he’ll last more than a few minutes.

A friend of mine did a bit more controlled test of his UPS and here’s his blog post with graphs also.

I have zoomed the graph a bit more to get more detail on the last 20 minutes of my servers uptime.

I find it interesting that the UPS temperature actually dropped when the thing was on batteries. I remember reading somewhere that the fan only runs when the inverter is actually active, so I guess that would make sense. I also find it interesting that the input frequency never changed, perhaps the UPS only records that value once every now and then, and doesn’t update it. I would think that with 0 input voltage I wouldn’t have any measurable frequency either.

I wonder if I could/should get those AFCI breakers replaced. They tend to not keep my computers on when there’s a lightning storm, and the only way to turn them back on is to go into the garage and flip it by hand.

Boy, was I wrong. Not that upstart is “bad.” Just that there’s something much better.

This actually makes things better instead of just providing the ability to make things better. It involves less shell, and more configuration. Dependencies are automatic, instead of manually specified. The author of the previously linked post goes into great detail regarding the reasoning behind the decisions made. It’s quite a good, albeit long, post.

I think I will have to get this working on my linux box that I intend to build. Possibly even on my laptop, as the features for enabling and disabling bluetooth would be quite advantageous. Also for the wifi and such. It’d be great if when I fired up the bluetooth, it were to start various things, like trying to connect to my mouse, or enabling the obex ftp thing.

As for a server, it’d be really nice to have if I can ever get my fibre channel setup to work. The Xen VMs won’t start until the Fibre Channel drives are available, and then they’ll all start up. Events could be fired depending on when the UPS was online or on battery.

I’m very impressed with it in concept, I’ll have to see how it goes in practice. Oh, and it’s going to be in Fedora 14. Woot.

You need to install openssl-devel, zlib-devel, bison, gcc, make, patch, tar, and maybe gcc-c++ (although I don’t think this one is needed).

Go get the latest ruby 1.9 source, as of this writing 1.9-p378, and extract it somewhere. Then go get the patches on this bug, at the specific comment. You will need to apply at least the openssl-build-fix patch, since fedora uses openssl 1.0 and it’s not yet into ruby 1.9. Then follow your typical ./cofnigure, make, and make install stuff. I installed mine into a prefix of /opt/ruby so that it wouldn’t affect any fedora ruby stuff that it might want. I then added ruby’s path to the end of my user’s PATH variable.

That’ll get you a working ruby 1.9 in Fedora.

I also like that it claims it can distribute across multiple hosts (‘claims’ because I haven’t verified it yet.) I wonder how difficult it will be to set up a switch that extends to my router, so I can just emulate my entire network’s switch in OpenVswitch. It’s certainly got some interesting potential.

There’s potential to get better information on the traffic happening in my network thanks to the netflows provided by this. Maybe it will be enough information to figure out why my traffic shaping doesn’t work well, and how I can improve it.

Woot.

EDIT:

Man, this guy has all sorts of good stuff: http://www.vim.org/account/profile.php?user_id=9012

cucumber, rails, ruby, git-vim integration. VIM FTW!

So I’m back to the original setup, with an old (1998) left-over 2GB hard-drive, and a modified Source Mage install. It’s in good shape, with some tmpfs stuff for things that write to the disk in order to hopefully prolong the life of the disk. I took a clonezilla snapshot of it as well, so when the drive dies, I can throw in a new (also old, I have several 2 to 4 GB drives that are in the closet that still work) one, and the router will be back up in a few moments.

Whilst it was unfortunate that the router wasn’t a good fit for my particular use, perhaps I’ll find another use for it. Since there are two Serial ports trivially accessible on the device, perhaps I’ll get a GPS module and set it up in the car as a mobile automatic war-driving toy. I’ve found instructions to hook up an SD card to it as well, that would give it plenty of easy to replace storage.

It was an interesting experiment, and I’m somewhat sad it didn’t work out.

Turns out these things are really just WRT54GL‘s, the Linux one that they sell for like $70 that they have not screwed with so you can hack them and run any of the (not all)Linux wrt distros out there. Except it’s even better, because it’s got 32MB of ram and 8mb of flash, instead of half that like the standard WRT54GL. It requires a bit more voodoo to unlock (and after flashing dd-wrt, you can flash openwrt, or anything else,) but not at all difficult.

With the recent release of BackFire from OpenWRT, which coincided with my old SMGL router’s hard drive failure, I have set up that router on my network. It worked well for a few weeks, but recently has started flaking out and completely disappearing from my network. A gateway that refuses to respond to anything on the network is not particularly useful. So, to the internets I went. I discovered how trivial it would be to hook up one of the serial ports on the thing to my computer. A bit of soldering later, and I have a log of everything that comes out of the console of that box on my server. Hopefully, the next time it flakes out, I’ll have a logged kernel panic that I can submit it in a bug report. I might even be able to figure out the solution and post it along with the bug.

Additionally, as a solution to keep the network alive right now, I have set up a Nagios event handler to tell the UPS the router is attached to, to restart. This successfully power cycles the two gateway devices: ATT’s 2-wire thingy, and my OpenWRT router. So, my network has a limited self-healing ability now, which is, of course, a band-aid. However, until I can figure out what’s wrong with the router, it’s the best option that I’ve got to keep things functional. Especially when I’m not at home, and it dies (which seems to be the only time it dies…)

All of this is using xen 4.0 hypervisor, the kernel from here, and a pvops domU kernel built by hand.

• Using blktap2 backed disks in the vhd format (vhd-util create -n name -s 1024) xm pause fails. The user-space disk devices are never released, and only a hard reboot will get it to function again.
• Using a different driver for the blktap2, tap:tapdisk:aio, also fails to pause.
• Using phy:/ backed disks, say through LVM works fine. The domU will pause, and resume nicely.

It would seem that there is something not quite ready to be used regarding the blktap2 backend, so I wouldn’t reccomend using it.

I also tried to use the original tap:aio drivers, but I cannot as they don’t seem to exist. I guess I’ll be using the plain old phy:/ drivers for now. That’s not a big deal as I don’t have any second xen machine to do anything fancy that the tap:aio drivers would provide. I guess I should poke on a mailing list or something.

Finding this information should’ve been obvious, but for some reason it wasn’t. Maybe I had some silly mental block or something. Anyways, get the kernel source from here. Then build it for your architecture. If you want to build a 32-bit kernel for some 32-bit domUs, and you’re running on 64-bit, it’s as easy as make ARCH=i386 <target>. This does work and I’ve verified it. Had to fool a Fedora domU into beleiving that it really was a 32-bit system.

I’ve found make tarbz2-pkg to be quite handy in packaging up the kernel to be deployed.

I have verified that this works in Xen 3.4.2 on a patched 2.6.29.6 kernel.

On my Xen 4.0 test box, it fails to suspend. I’m using the blktap2 stuff, and perhaps that doesn’t pause well. As far as I understand it, that blktap2 should not affect suspending, but I’m going to test it with some logical volume based vms. I usually use the lvm backed disks anyway, but the blktap2 stuff offers some interesting features, if it were to work. Not being able to pause stuff probably isn’t good, but again, I’m not sure what’s causing it.

Apparently, that is what’s causing it. The domU will pause and unpause just fine when using an lvm backed disk, rather than the blktap2 ones. I should test a couple of other combinations to see if things like tap:aio will behave, as that’s needed for nifty things like remus.

I hacked up some backup scripts in ruby, I probably could have done it in pure bash, but I felt that this was cleaner.

I’ve got two scripts, one for mysql, and one for the postgresql database. They’re almost exactly the same, except for the differences in how to talk to the database, and the interfacing objects. They create one .sql file for each database, and that will automatically adapt when you add more databases. It will keep around 7 copies of each file, so if you run it daily, you’ll have a week’s worth of database backups. They’re bzip2 –best’d so that it doesn’t waste too much space.

MySQL:

#!/usr/bin/ruby

require "mysql"
require "fileutils"

# designed to be run daily, will keep #{files} number of files around

backupDir = "/srv/database/backup/mysql"
username = "SOMEUSER"
password = "PASSNERD"
host = "localhost"
files = 7

FileUtils.mkdir_p backupDir

currentDir = FileUtils.pwd()
FileUtils.cd(backupDir)
begin
        dbh = Mysql.real_connect(host, username, password)
        result = dbh.query("show databases;")
        while row = result.fetch_row do
                begin
                        dbname = row[0]
                        # do file rotation stuffs
                        FileUtils.rm("#{dbname}.#{files}.sql.bz2", :force => true) # never will complain
                        (1..(files-1)).each do |a|
                                n = files -a
                                FileUtils.mv("#{dbname}.#{n}.sql.bz2", "#{dbname}.#{n+1}.sql.bz2", :force => true) #no errors
                        end
                        # move the one without a number to 1.sql.bz2
                        FileUtils.mv("#{dbname}.sql.bz2", "#{dbname}.1.sql.bz2", :force => true)

                        #dump the sql!
                        command = "mysqldump -u #{username} -p'#{password}' -h #{host} #{dbname} | bzip2 --best > #{dbname}.sql.bz2"
                        `#{command}`
                rescue Mysql::Error => err
                        # note it and continue with the next database
                        p "Database: #{dbname}\n\terr"
                end
        end
        result.free
rescue Mysql::Error => e
        puts "Error code: #{e.errno}"
        puts "Error message: #{e.error}"
        puts "Error SQLSTATE: #{e.sqlstate}" if e.respond_to?("sqlstate")
ensure
        dbh.close if dbh
        FileUtils.cd(currentDir)
end

Postgresql:

#!/usr/bin/ruby

require "pg"
require "fileutils"

# designed to be run daily, will keep #{files} number of files around

backupDir = "/srv/database/backup/postgres"
username = "SOMEUSER"
password = "PASSNERD"
host = "localhost"
files = 7

FileUtils.mkdir_p backupDir

currentDir = FileUtils.pwd()
FileUtils.cd(backupDir)
begin
        conn = PGconn.new(:user => username, :password=> password, :host=>host)
        result = conn.exec("select datname from pg_database")
        result.each do |row|
                begin
                        dbname = row['datname']
                        if !( /template.*/ =~ dbname)
                                # do file rotation stuffs
                                FileUtils.rm("#{dbname}.#{files}.sql.bz2", :force => true) # never will complain
                                (1..(files-1)).each do |a|
                                        n = files - a
                                        FileUtils.mv("#{dbname}.#{n}.sql.bz2", "#{dbname}.#{n+1}.sql.bz2", :force => true) #no errors
                                end
                                # move the one without a number to 1.sql.bz2
                                FileUtils.mv("#{dbname}.sql.bz2", "#{dbname}.1.sql.bz2", :force => true)

                                #dump the sql!
                                command = "PGPASSWORD='#{password}' pg_dump -U #{username} -h #{host} #{dbname} | bzip2 --best > #{dbname}.sql.bz2"
                                `#{command}`
                        end
                rescue PGError => err
                        # note it and continue with the next database
                        p "Database: #{dbname}\n\terr"
                end
        end
        result.clear
rescue PGError => e
        p e
ensure
        conn.close if conn
        FileUtils.cd(currentDir)
end