Xen stats FTW!

(The UPS graphs I need to fix too, somethings broken in the PHP datasource it uses. So if you’ve got a reliable Network UPS Tools graphing source for cacti, other than this one, lemme know.)

Daily Spam Statistics for my email

I recently repaired the DSPAM statistics on my mail server so I can look at the ridiculous amounts of spam that I receive. I knew the number was rather high, because I kept having to train a bunch of misses. Especially when the spam format dramatically changes. It seems to take more corrective action than it used to. I wonder if perhaps I need to clean out the tokens in the database. It’s possible that I’ve got a large amount of stale tokens that are leaning higher for other emails than it should be.

These statistics are from my older email server. I’ve got a new one that has been built, but not put into practice yet. Still trying to figure out how to get the Dovecot Sieve to actually work. Then I can finally ditch procmail and use a smarter server-side mail filtration language. One that doesn’t require a shell account. It’s also possible that I could offer email accounts @shlrm.org. Not that anyone would actually want those. Heh.

Hopefully soon it’ll go stable, and I can do something with it. I’m also hoping that fedora will have RPMs for it, even unofficially, so that it’s easier to install than building by hand. Or maybe better building documentation would help: detailed dependency information or something. Maybe the stuff exists, but it’s hard to find (or I’m looking in all the wrong places.)

Or I could install CentOS as my domain0 and use that old xen kernel. I don’t need any of the new features, I think. And I don’t think they’ve gotten in online LVM resizing, which is something I’d really want. I can’t remember if they got the USB to PV guests either. I don’t need the overhead of fully virtualized guests, but I would really like to be able to forward a USB device to a specific guest.

So yeah, unimpressed.

It would appear that Xen 4.0 is due out Q1 2010. That’s not very far off. I will perhaps wait on redoing my Xen network setup until that time. I need to get it working better. I’ve been having problems suspending VMs. Supposedly this is due to the guest kernel not behaving nicely. Patches have been submitted upstream from what I’ve been told and the pull request has been made to Linus to pull it into the kernel.

There’s lots of good stuff coming in Xen 4.0 too. Remus is particularly interesting. I will have to experiment with that. Live failover of boxes. I don’t know what I’d do with that to be honest. I guess you could set up two physical boxes, with the same virtual machines on them, and have 100% uptime, well like 9 9’s of uptime. (Assuming no stupid admin errors.)

Groovy Code:

import java.util.concurrent.atomic.AtomicInteger
import java.util.concurrent.Executors
import java.util.concurrent.Callable

def moveSingleDisk(source, dest) {
	//nothing!
}

def moveTower(number, source, dest, temp) {
	if (number > 0) {
		moveTower(number -1, source, temp, dest)
		moveSingleDisk(source, dest)
		moveTower(number - 1, temp, dest, source)
	}
}

AtomicInteger counter = new AtomicInteger()
counter.set(1)

synchronized out(message) {
	println message
}

def logic = {
	start = System.nanoTime()
	moveTower(it, 'A','B','C')
	stop = System.nanoTime()
	out("${it} took ${(stop-start)/ 10**6} milliseconds")
}

def THREADS=16

pool = Executors.newFixedThreadPool(THREADS)
println "Creating a pool for $THREADS threads"

1.upto(64) {
	def value = it
	//out "Submitting job $value"
	task = {c -> pool.submit( c as Callable)}
	task{logic(value)}
}

pool.shutdown()

Java code:

/*
 * To change this template, choose Tools | Templates
 * and open the template in the editor.
 */
package hanoi2;

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

/**
 *
 * @author brad
 */
public class Main {

	static private ExecutorService exec = Executors.newFixedThreadPool(8);

	static void moveSingleDisk(char src, char dst) {
		// System.out.println(src + " => " + dst);
	}

	static void moveTower(int n, char src, char dst, char tmp) {
		if (n > 0) {
			moveTower(n - 1, src, tmp, dst);
			moveSingleDisk(src, dst);
			moveTower(n - 1, tmp, dst, src);
		}
	}

	public void run() {
	}

	public static void main(String[] args) {

		System.out.println("Starting with a thread pool of 8");

		for (int i = 0; i < = 64; i++) {
//      long start, stop;
			final int blah = i;
			exec.submit(new Runnable() {
				public void run() {
					long start = System.nanoTime();
					moveTower(blah, 'A', 'B', 'C');
					long stop = System.nanoTime();
					System.out.println(blah + " took " + ((stop - start) / 1000000) + "ms");

				}
			});
		}
	}
}

• xen-3.4.2-1.fc12.x86_64.rpm
• xen-debuginfo-3.4.2-1.fc12.x86_64.rpm
• xen-devel-3.4.2-1.fc12.x86_64.rpm
• xen-doc-3.4.2-1.fc12.x86_64.rpm
• xen-hypervisor-3.4.2-1.fc12.x86_64.rpm
• xen-libs-3.4.2-1.fc12.x86_64.rpm
• xen-runtime-3.4.2-1.fc12.x86_64.rpm

After many hours searching the intertubes for a srpm or a spec file for building xen 3.4.2 for fedora, I decided that I would hack an existing one to 3.4.2 and hope for the best.

It built after I removed a few patches. The patches mostly complained about being already applied, so I assume the bugs the patches were addressing were fixed in 3.4.2. I have generated rpms for x86_64 and put the spec file up on my site. Enjoy.

• xen-3.4.2-5.fc11.x86_64.rpm
• xen-debuginfo-3.4.2-5.fc11.x86_64.rpm
• xen-devel-3.4.2-5.fc11.x86_64.rpm
• xen-doc-3.4.2-5.fc11.x86_64.rpm
• xen-hypervisor-3.4.2-5.fc11.x86_64.rpm
• xen-libs-3.4.2-5.fc11.x86_64.rpm
• xen-runtime-3.4.2-5.fc11.x86_64.rpm

Anyways. I had this plan for a long time to write my resume in Docbook. Why Docbook? Because I could collect all my data in one big file, or multiple files, and then apply XSLT filters to prune it down into what I wanted to generate. It sounded great in my head, and it sounds awesome to implement. I mean every programmer’s dream. A professional looking résumé generated from code. I could push commits to a git repo, and hack up a hook to have it automatically update my résumé in both PDF and HTML formats all at once. Unfortunately, the learning curve for docbook to do such typesetting is rather difficult. I even found a free, open source docbook editor (only does docbook 4.5, not the new 5.0).

I spent about a day trying to figure it out. I read Docbook: The Definitive Guide and the XSL Guide. I know I could do it, but I don’t believe it would be a simple task. I’m not even sure the end result would be worth it. Doing documentation in docbook, that would be worth it. I think that would be a better place to start, writing a short book, or article, in docbook. Something that doesn’t require lots of special formatting, like a résumé does. Something a bit simpler to get my feet wet.

As an alternative, I have turned to LyX, a LaTeX editor thingy that seems to be more suited to doing it. Certainly their editor is a bit better. As I type this paragraph, however, I wonder if I ever searched the internet for a docbook stylesheet for résumés that someone already created.

Wouldn’t you know it, it exists. So yeah, I’m going to go do that now.

First I’ll rave about pygrub. This allows me to give the owner of a paravirtualized box the ability to install their own kernel. There’s only one constraint: The kernel must support being a domU. So it doesn’t quite get you unmodified guest kernel support, but you can still install your own kernel if you want a specific kernel for some reason. Also, vanilla kernels starting from about 2.6.27 onward have support for running in a domU, so long as you turn it on. Fedora 11’s kernel has this ability, so you can do a bit of configuration, and install a Fedora 11 domU right off the CD. That’s pretty awesome.

I use LVM backed disks for domains rather than disk image files. I’ve found that the LVM backed disks perform much better than the file backed ones. The installs that I’ve done with Fedora 11 are actually using the disk I have given the domU as a physical disk, partitioning it, installing LVM and everything. I was concerned about the performance of two levels of LVM, so I ran bonnie++ on both the domU and the dom0. The dom0 is one level of LVM, and the domU will have two levels of LVM.

Domain 0 IO test result:

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
fedoraxen.shl 1512M 50362  83 41511  11 22227   1 46086  92 72866   0 207.4   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  9622  98 +++++ +++ +++++ +++  9834  99 +++++ +++ +++++ +++

Fedora 11 DomU (on the same box):

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
TestVM           2G 52894  96 65639  31 29314  14 36553  69 79138  11 189.6   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  7932  82 +++++ +++ 30738  99  9571  99 +++++ +++ 31084  99

So it seems that performance isn’t hit that badly, things are a bit slower, as to be expected with two levels of LVM, but not horribly so.