org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:hibernate3-maven-plugin:2.2:hbm2ddl (create-schema) on project incident-service:
   Execution create-schema of goal org.codehaus.mojo:hibernate3-maven-plugin:2.2:hbm2ddl failed:
    Could not parse configuration: file:/var/lib/jenkins/jobs/IncidentService/workspace/target/classes/hibernate.cfg.xml

Except, there were no changes to the code; nothing changed in this file, and it’s been working fine for about a month or two. The environment the build server is on changed however. It cannot get to certain parts of the internet:

jenkins@hudson1:~$ wget http://www.hibernate.org/dtd/hibernate-configuration-3.0.dtd
--2011-10-10 10:48:06--  http://www.hibernate.org/dtd/hibernate-configuration-3.0.dtd
Resolving www.hibernate.org... 209.132.182.21
Connecting to www.hibernate.org|209.132.182.21|:80... failed: Connection timed out.
Retrying.

It’s not actually able to get to the internet to validate the DTD within that hibernate.cfg.xml file. This should not break our build. It was a source of much frustration for about 4 hours of the day. Various hibernate forums were of limited help.

Apparently it should get it from the jar, rather than the internet, if the jar is available on the class path and the DTD matches. First our DTD was incorrect. Rectified that, and yet it still tried to get it off the internet. WTF?

Turns out the hibernate3-maven-plugin doesn’t include any hibernate jars when it executes.

Solution is to duplicate the necessary hibernate dependencies for the plugin to have the necessary DTD data available:

<dependencies>
  <!--
    Need to include the following hibernate dependencies so that it can find the DTD
    without having to get it off the internet.
  -->
  <dependency>
    <groupId>org.hibernate</groupId>
    <artifactId>hibernate-core</artifactId>
    <version>${hibernate.version}</version>
  </dependency>
  <dependency>
    <groupId>org.hibernate</groupId>
    <artifactId>hibernate-validator</artifactId>
    <version>4.0.0.GA</version>
    <exclusions>
      <exclusion>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
      </exclusion>
    </exclusions>
  </dependency>
  <dependency>
     <groupId>org.hibernate.javax.persistence</groupId>
     <artifactId>hibernate-jpa-2.0-api</artifactId>
     <version>1.0.0.Final</version>
  </dependency>
</dependencies>

So some context: This is to support preventing the hibernate3-maven-plugin from going to the internet to get the DTD every time it parses that hibernate.cfg.xml file. There’s no data in this file, only a shell of a hibernate.cfg.xml; just enough of a placeholder to keep the hbm2ddl Goal from failing, as we’ll eventually have it build DDL for us.

Content:

<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE hibernate-configuration PUBLIC
    "-//Hibernate/Hibernate Configuration DTD 3.0//EN"
    "http://www.hibernate.org/dtd/hibernate-configuration-3.0.dtd">

<hibernate-configuration>

    <session-factory>
        <!-- specify annotated classes here, because the hbm2ddl plugin isn't capable of searching for them :( -->
    </session-factory>

</hibernate-configuration>

Quite an irritating error for something that seems to actually do nothing. Apparently it does. Stupid XML

I updated my postgresql database, as it was version 8 and version 9 has been out for quite a while. That actually went painlessly. Backup the SQL, terminate the db server, build/install the new one, and restore the SQL. Then tewak configuration files to get everything back the way it was before. No problem.

Then realized I needed to rebuild a pg gem on the deployed host so that it’d talk to postgresql 9 now. Did that. Suddenly the app server no longer responds. it just sits there forever waiting on … something. No amount of debugging output from Passenger will help. It must be something else. So now I’m updating ruby and rebuilding a new Passsenger against it in the hopes that it’s a bug in ruby that I’ve encountered that evidences itself with postgresql 9 or something. Otherwise, I might be spending the day getting the newer revision of my application ready to go a bit earlier than I had planned.

Had I had an automated deployment script built in, I could’ve just clobbered the environment and redeployed within the new fresh environment. Instead I’m having to monkey around getting everything to behave well, and generally failing at it.

Instead, it’s simply “not due for delivery,” and so you keep it. I wish I’d checked before I left work, then I could’ve gone and picked it up, because you’re too lazy to get it to my house today.

If you weren’t better than UPS at smashing boxes for me, I’d have shipped it UPS, and they’d have delivered it a day early.

Unfortunately, that means for those that are still using cuke4duke, you’ve got problems to deal with. The voodoo magic that is cuke4duke does not install gems very well. I don’t know precisely why this is, but it ends up with many problems. Things that will install just fine using a jruby command line, or in native ruby barf a horrible death. Often regarding some YAML::Syck thing.

ERROR:  While executing gem ... (ArgumentError)
      undefined class/module YAML::Syck::DefaultKey

Really sucks and makes it difficult to actually use cuke4duke unless you’ve already got all the gems you need installed.

After much investigation, I discovered that cuke4duke uses a GEM_HOME of ~/.m2/repository/.jruby . So a simple command can get the gems you want installed into that folder:

java -jar jruby-complete-1.6.4.jar -S gem install json --version=1.5.4 --no-rdoc --no-ri -i /home/dkowis/.m2/repository/.jruby

Unfortunately, if there’s another “gem” command in your path, it will use that instead of the one in your jar file, so you have to munge your path as well to prevent it from doing so.

Finally, I needed a way to get this into our project’s Maven POM so that we could easily set up other peoples boxes and the configuration isn’t lost in some terribly manual process of running that command over and over.

This is the solution I came up with:

        <profile>
            <id>install-gems</id>
            <build>
                <plugins>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-antrun-plugin</artifactId>
                        <version>1.6</version>
                        <executions>
                            <execution>
                                <phase>generate-test-resources</phase>
                                <configuration>
                                    <target>
                                        <property file="gems.properties"/>
                                        <macrodef name="install-gem">
                                            <attribute name="gem"/>
                                            <attribute name="version"/>
                                            <sequential>
                                                <java jar="${maven.dependency.org.jruby.jruby-complete.jar.path}"
                                                      fork="true"
                                                      failonerror="true"
                                                      maxmemory="64m"
                                                      newenvironment="true">
                                                    <arg value="-S"/>
                                                    <arg value="gem"/>
                                                    <arg value="install"/>
                                                    <arg value="@{gem}"/>
                                                    <arg value="--version=@{version}"/>
                                                    <arg value="-i"/>
                                                    <arg value="${user.home}/.m2/repository/.jruby"/>
                                                    <env key="PATH" path="${java.home}/bin"/>
                                                </java>
                                            </sequential>
                                        </macrodef>
                                        <install-gem gem="json" version="1.5.4"/> <!-- needed to get json to behave with jruby -->
                                        <install-gem gem="cucumber" version="1.0.3"/>
                                        <install-gem gem="cuke4duke" version="${gem.cuke4duke.version}"/>
                                        <install-gem gem="nokogiri" version="${gem.nokogiri.version}"/>
                                        <install-gem gem="rspec" version="${gem.rspec.version}"/>
                                        <install-gem gem="rest-client" version="${gem.restclient.version}"/>
                                        <install-gem gem="jruby-openssl" version="${gem.jrubyopenssl.version}"/>
                                    </target>
                                </configuration>
                                <goals>
                                    <goal>run</goal>
                                </goals>
                            </execution>
                        </executions>
                    </plugin>
                </plugins>
            </build>
        </profile>

One can then run the command ‘mvn test -Pinstall-gems’ and the gems will be installed as part of the test-resource-generation phase. This only has to be done once whenever new gems are added, or on initial setup. Once it’s done, you’re good to go with the normal cuke4duke workflow.

Hopefully this will save other people some time :)

I kept having a horrible error building collectd 4.10.x on my Source Mage servers:

libiptc.c:85: error: redefinition of ‘struct xt_error_target’

The internets couldn’t really help me with the error, I think I was searching for the wrong words. There was also issues regarding warnings that gcc now treats as errors.

I ended up spending probably about an hour cloning their git repo, cherry-picking a few commits into the 4.10.3 tag, and then submitting the patches to the bug I created.

End result is that collectd-4.10.3 now builds on gcc 4.4.3 and 4.6.1. The fix is in the Grimoire on Source Mage, and should trickle out to stable eventually. I think I’m the only one who uses collectd, so it’s not all that important, heh.

Well, I was using an internal lighttpd to render the same thing internally, and I didn’t disable its directory listing at all. Whilst I was working on setting up collectd to monitor my Apache scoreboards, I noticed that it could also monitor the lighttpd scoreboard. So I was going about the config to set that up, and I noticed that the Apache wasn’t proxying it’s requests through to lighttpd, and was taking a lot of time and resources to do something simple.

Welp, having ADD like I do, I got sidetracked on doing that instead and started to rewrite my configs to not serve files directly, but proxy it to the lighttpd internally.

Originally, I had two aliases set up “/sourcemage” and “/sourcemage/fallback” which both pointed to the same spot on disk, This way you could reference things using “/sourcemage/file.tar.bz2″ or “/sourcemage/fallback/file.tar.bz2″. The reasoning behind this is simple: I set up my fallback mirror before Source Mage standardized on a fallback URL, and I wanted to keep both paths functional. Easy to do with aliases. Oh, and I also have an alias to “/sourcemage/codex” for all my local codex needs.

Switching to a proxied setup was a bit more complicated than I anticipated. The alias stuff doesn’t work the same way everything else does, in that what it finds first it goes with. I needed to have mod_rewrite rules wired in to properly redirect things and manipulate the URL sufficiently. But I have two special cases. I can’t simply redirect everything “/sourcemage/*” to “/sourcemage/fallback.” Also, I wanted to be able to continue to use “/sourcemage/file.tar.bz2″

My solution is relatively simple and follows:

#aliases
Alias /sourcemage/codex    "/srv/webMirrors/sourcemage.org/codex"
#rewrite /sourcemage/ to /sourcemage/fallback
#some very fancy rewrite rules to keep the old alias functionality
RewriteEngine On
RewriteRule ^/sourcemage/?$ /sourcemage/fallback/ [R]
RewriteRule ^/sourcemage/codex/?$ - [L]
RewriteRule ^/sourcemage/([a-zA-Z0-9.i_\-+]+)$ /sourcemage/fallback/$1 [R]
# do the actual passing through to the lighttpd (which handles huge
# directory listing much much better
ProxyPass /sourcemage/fallback http://fallback.shlrm.org/
ProxyPassReverse /sourcemage/fallback http://fallback.shlrm.org/

# old notes kept here for reference as to what the above rewrite
# rules are doing -- dkowis 2011-09-07
# going to proxy these to the lighttpd
#Alias /sourcemage/fallback "/var/spool/sorcery/"
#Alias /sourcemage          "/var/spool/sorcery/"
# directories!
#<Directory "/var/spool/sorcery/" >
#       Options none
#    AllowOverride None
#    Order allow,deny
#    Allow from all
#</Directory>

<Directory "/srv/webMirrors/sourcemage.org/codex/" >
        Options Indexes
        Order allow,deny
        Allow from all
</directory>

# vi: set ft=apache:

The rewriting rules are the most complicated part of this. The trick was getting it not to try to rewrite anything when the url matched “/sourcemage/codex” After that, the rest was really easy. Some matching logic to pick up on “/sourcemage/file.tar.bz2″ and redirect that to “/sourcemage/fallback/$1″ and everything worked as it did before, except way faster.

Far more customizable also. I’m satisfied.

I wanted to figure out how to require/load/import additional files into the deploy.rb file so that I would be able to drop recipes in to a config/recipes directory.

Turns out this is the solution.

I ended up using this to simply load all the recipes in the recipes directory:

#load in all the other recipes
$LOAD_PATH.unshift File.join(File.dirname(__FILE__), 'recipes')
Dir['config/recipes/**/*.rb'].each { |recipe| require  File.basename(recipe, '.rb') }

I looked into upgrading cucumber-rails, and apparently they recommend using Capybara instead of webrat for Rails 3. So I make the necessary changes. Unfortunately Capybara only has a get method. You can only visit pages. Any posts should be exercised by the web forms. This is far less than useful. It has proven to be a huge pain the butt. Hours of trying to find how people do this to no avail. Until I find an obscure answer, at the bottom of a StackOverflow posting.

That code on github is the key to testing APIs. It appears to use some of the more internal guts of the Capybara page drivers to accomplish it’s goal. Works for me.

I am surprised to see that there are few integration test frameworks that support this kind of web service test. Especially since where I work, this is what we build. Having Cucumber features describing those API calls makes everyone’s job easier. I find it unfortunate that it seems to be somewhat ignored by the testing community. Hopefully this will contain the necessary words for other people that are searching for the same thing I spent hours searching for, and will find it in far less time.