Shlrm.org Blog

Linux, Java, Ruby, and Politics

Scala for Very Fast Servers: Revisited

| Comments

In a previous entry I had a demo of using scala for very fast servers. It resulted in a bug being filed on the spray framework for a rather inefficient copying of data. That has since been resolved, and I have re-ran the tests, both on a “cold” jvm, and after “warming” it up with about 20 runs of the apache benchmark command, the same one used to collect the benchmarks.

The performance is roughly on par with node.js and go, fitting about halfway between the two, slightly closer to go. Doing nearly 2000 requests per second on a single machine. Which is somewhat less than Go’s 2300 requests/sec. The average time/request is about 10ms longer on a warm JVM than go. Yay numbers! Results follow.

Scala for Very Fast Servers?

| Comments

EDIT: Updated version of spray produces better results!

I have been following Go somewhat in recent history, mostly blog posts about it and friends talking about how awesome it is. It intrigues me. It also irritates me. I feel like I’ve spent all this time learning Scala, and it’s impressive functional/ObjectOriented hybrid style, and now this upstart Go comes in and everyone is all excited about it.

I get all huffy reading this article because I figured that Scala wasn’t getting the love it deserves. People see JVM and go, “EWW JAVA IS SLOW.” Which is a load of crap. The JVM is a very mature Virtual Machine compared to many of the newer language VMs. It still suffers from some monolithic problems, which the so-called Jigsaw project would help with, making it much easier to load only the parts you care about. Thus, the JRE would load up way faster, and could be targeted better for specific environments.

Regardless, I decided that, for giggles of course, I would implement the same benchmarks there on my workstation and see how scala measured up.

My Love/Hate Relationship With DSpam

| Comments

I really hate dspam, but it is the absolute best anti-spam solution I’ve ever seen. It works stupidly well. I can turn it on, and it catches all the spam. I don’t care if my email address is on a whole pile of lists. Dspam doesn’t care. It does a fantastic job of marking that crap, and then I happily filter it into a folder, and only keep like 30 days worth, just in case of a false positive. Mostly because I’m paranoid.

But Dspam’s documentation is absolutely horrid. The C code is full of loops and GOTOs. It is horrid. There’s shared code but the log messages don’t tell you in which process the thing actually had a problem in, additionally, it doesn’t actually log anything useful. Often times you just get Segmentation fault, or if you’re lucky you get the usage of the application. Bleeding worthless. Debug mode doesn’t help much more either.

But all that aside, I met someone awesome in the #postfix channel on irc.freenode.net that helped me solve all these stupid problems. I’ve used the Dovecot Antispam Plugin to solve my retraining problems. It truly is the easiest way to retrain. Basically, any move into or out of a configureable folder (unless you’re moving into the trash, like deleting a message) it will call the appropriate spam retraining method. I tell dspam to retrain on a signature. Peace of cake. Works in all cases. Except where DSpam sucks, which I’ll detail.

Multicast Snooping Is Bad

| Comments

I spent most of the day debugging ganglia when the problem wasn’t even ganglia, but I’m mentioning ganglia, and gmond, here so that if someone else searches google for the same problems that I was having using the same words, they’ll be able to find the appropriate solution.

tl;dr: If you can’t get your gmetric in xen using multicast to cluster across a linux bridged interface, disable multicast snooping in dom0.

echo 0 > /sys/devices/virtual/net/<bridge>/bridge/multicast_snooping

First Experiences With Puppet (Where I Built a Server)

| Comments

I’ve played with puppet in the past, but I never actually deployed anything significant with it. I have built manifests for work, to help deploy packages, and I’ve been tinkering with one that does some fancy stuff to install elasticsearch.

This weekend I actually spent the time to get a puppet master set up on ArchLinux, and it went relatively smoothly. I had to apply two patches, however, and one I got to write myself..

Chrootception

| Comments

I thought this would be worth sharing.

I’ve got an old box under my desk that can run xen, but doesn’t have the virtualization extensions. This makes it somewhat difficult to do things like install certain linux distributions coughcentoscough that don’t provide a simple network installation mechanism, or their net install stuff requires being fully virtualized.

I found instructions on building your own chroot within a centos box to run chrooted services, which isn’t what I wanted, but I can extrapolate from a chroot into a xen vm. Just need to make a simple grub menu.lst and make sure that it has a real init system to actually boot from.

Following those instructions, I was able to construct a chroot on my fedora desktop. Unfortunately, after chrooting into the box itself, I wasn’t actually able to yum install anything, since the database formats were from the yum on my fedora 17 desktop, which is significantly newer than old CentOS 6.

Solution was to simply build a chroot within the chroot. Creating database files that are the appropriate format for centos 6. Then I pulled that chroot out and did away with the intermediary chroot. A bit of yum groupinstall Base and I had a working system that I could boot in a paravirtualized way. Fantastic.

Scala Play and the SBT IDEA Gen Plugin

| Comments

Just a short one, but something that I thought I should document, because it bit me.

When I took the excellent Coursera course on functional programming in Scala, I wanted to use IntelliJ IDEA, because I wasn’t particularly enthralled with trying to learn eclipse as well as learn Scala. I found the SBT plugin to generate IDEA project files, and configured my sbt globally to take advantage of it.

After the completion of the course, I started working on learning Play, just to start a new project. Unfortunately, Play comes with it’s own, and yet exactly the same, version of the IDEA generation plugin. It is in the same namespace, and so collides with the existing one. The only solution is to disable the globally configured SBT one for the duration of working with Play.

It’s not a very good solution, but it does get it working again.

OSX, Java 7, and UTF-8. Oh My!

| Comments

tl;dr this.

It’s about a quarter to three in the afternoon on a Thursday and the twitter goes off! An open source project that I haven’t worked on in a while is having some trouble with UTF-8. I was stuck on some code that was proving to be kind of boring, so I decided to give it a whirl to try and shake my brain up a bit.

My initial solution was way off base. This solution didn’t deal with filenames, it dealt with data in the file. Cucumber-JVM is already setting UTF-8 in all places like it’s supposed to. That wasn’t the solution, it wasn’t even close. The adrenaline fires up as I realize this is going to be a challenge. All thoughts of doing work previously are gone, now this challenge is all that matters. I try my search again, looking to see what else I could find about UTF-8 and filenames.

Keyboard Command of Pithos and Clementine Without Media Keys

| Comments

I love my DasKeyboard, but it doesn’t have any media keys. That makes it somewhat tricky to do things like controlling Clementine and Pithos, my two favorite ways of listening to music.

Originally, I’d gone down the path of simulating multimedia keys. That was the most frustrating 30ish minutes of my day. Do not want.

Since the recommended method of talking to pithos was over dbus, I wondered if clementine also had a dbus method of control. I opted to use the MPRIS 2 API as that provides the PlayPause call, which behaved the way I wanted.

And so, the solution was to write a script to send messages over dbus to the system if it’s running, so I can pause my music with a keyboard command.

Here’s my ruby script that uses the native Ruby DBUS interface (yeah, I know it’s got an “invalid” self-signed ssl certificate, annoying, but cheaper than paying for one :) ) to send the commands. It should be resonably easy to now extend to other media apps if I desire to. No big deal. Even if they’re command line apps, I’ve got one centralized, simple script that can talk to anything I want it to. It’s supposed to be magicked away by media keys and stuff within your window manager, but it’s made more difficult if your keyboard doesn’t have media keys. No longer will I be discriminated against! I will rise up with my awesome DasKeyboard and type to victory! Er, well, listen to music.

Ruby Dbus media command script
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#!/usr/bin/env ruby
#
# ARGV.first should be play, stop, next, prev
require 'dbus'

command = ARGV.first
exit 1 unless %w{PlayPause Next Previous Stop}.include? command

#play with dbus stuff
bus = DBus::SessionBus.instance
begin
  clementine = bus.service("org.mpris.clementine").object('/org/mpris/MediaPlayer2')
  clementine.introspect
  clementine.default_iface = 'org.mpris.MediaPlayer2.Player'

  #clementine.PlayPause
  #clementine.Next
  #clementine.Previous
  #clementine.Stop
  clementine.send(command.to_sym)
rescue DBus::Error => e
  # don't really care, means it's probably not there
end

begin
  pithos = bus.service("net.kevinmehall.Pithos").object('/net/kevinmehall/Pithos')
  pithos.introspect
  pithos.default_iface = 'net.kevinmehall.Pithos'

  if command == "Next"
    pithos.send(:SkipSong)
  elsif command == "PlayPause"
    pithos.send(:PlayPause)
  end

rescue DBus::Error => e
  # don't really care again
end

There’s one more bit necessary, you’d have to hook it up to some kind of keymapper. I used xbindkeys, so it’s not particularly bound to any particular window manager, even though I use KDE4.

xbindkeys example
1
2
3
4
5
6
7
8
9
#media magic controls!
"/home/dkowis/Dropbox/scripts/mediamagic.rb PlayPause"
  control+alt+End
"/home/dkowis/Dropbox/scripts/mediamagic.rb Stop"
  control+alt+Home
"/home/dkowis/Dropbox/scripts/mediamagic.rb Next"
  control+alt+Next
"/home/dkowis/Dropbox/scripts/mediamagic.rb Prev"
  control+alt+Prior

Hopefully this will save you time if you’re trying to accomplish the same thing I was trying to accomplish.

Building Ruby 1.8.7 on Fedora 17 Using RVM

| Comments

UPDATE: I’m a moron. Latest RVM (1.14.5) handles all this for you. I’ll leave the post here since it’s already on the internet, and if anyone wants to see how to do things anyway, heh.

Unfortunately, the default RHEL system ruby is 1.8.7, which is quite old. This also means that I will have to develop against 1.8.7, and so it needs to be installed on my local development box. I use RVM, and so it makes sense to have RVM install 1.8.7 for me. Unfortunately, the first thing it does is explode horribly :(

AUGH, Explodie!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
gcc -I/opt/local/include -I. -I/opt/local/include -I../.. -I../../. -I../.././ext/dl -DHAVE_DLFCN_H -DHAVE_DLOPEN -DHAVE_DLCLOSE -DHAVE_DLSYM -DHAVE_DLERROR    -I. -fPIC -g -O2   -fno-defer-pop -fno-omit-frame-pointer  -c ptr.c
gcc -I/opt/local/include -I. -I/opt/local/include -I../.. -I../../. -I../.././ext/dl -DHAVE_DLFCN_H -DHAVE_DLOPEN -DHAVE_DLCLOSE -DHAVE_DLSYM -DHAVE_DLERROR    -I. -fPIC -g -O2   -fno-defer-pop -fno-omit-frame-pointer  -c handle.c
Generating callback.func
Generating cbtable.func
gcc -I/opt/local/include -I. -I/opt/local/include -I../.. -I../../. -I../.././ext/dl -DHAVE_DLFCN_H -DHAVE_DLOPEN -DHAVE_DLCLOSE -DHAVE_DLSYM -DHAVE_DLERROR    -I. -fPIC -g -O2   -fno-defer-pop -fno-omit-frame-pointer  -c dl.c
In file included from dl.c:104:0:
callback.func:1:1: warning: data definition has no type or storage class [enabled by default]
callback.func:1:7: error: expected identifier or ‘(’ before ‘long’
In file included from dl.c:104:0:
callback.func:78:33: error: expected ‘)’ before ‘(’ token
callback.func:79:3: warning: data definition has no type or storage class [enabled by default]
callback.func:79:24: error: ‘proc’ undeclared here (not in a function)
callback.func:79:39: error: ‘argc’ undeclared here (not in a function)
callback.func:79:45: error: ‘argv’ undeclared here (not in a function)
callback.func:82:1: error: expected identifier or ‘(’ before ‘}’ token
dl.c:106:1: error: expected ‘;’, ‘,’ or ‘)’ before ‘static’
make[1]: *** [dl.o] Error 1