Particularly deployment automation, in my case something like capistrano. It goes further than just the application, however. It goes into being able to ensure that the deployment environment, including the database and the application container all behave nicely. I have failed in this aspect, as my rails app is no longer behaving, and I’ve been reduced to updating things in the hopes that one of the things will fix it :(
I updated my postgresql database, as it was version 8 and version 9 has been out for quite a while. That actually went painlessly. Backup the SQL, terminate the db server, build/install the new one, and restore the SQL. Then tewak configuration files to get everything back the way it was before. No problem.
Then realized I needed to rebuild a pg gem on the deployed host so that it’d talk to postgresql 9 now. Did that. Suddenly the app server no longer responds. it just sits there forever waiting on … something. No amount of debugging output from Passenger will help. It must be something else. So now I’m updating ruby and rebuilding a new Passsenger against it in the hopes that it’s a bug in ruby that I’ve encountered that evidences itself with postgresql 9 or something. Otherwise, I might be spending the day getting the newer revision of my application ready to go a bit earlier than I had planned.
Had I had an automated deployment script built in, I could’ve just clobbered the environment and redeployed within the new fresh environment. Instead I’m having to monkey around getting everything to behave well, and generally failing at it.
Sorry for your troubles, however, slightly relieved to see that not even the elite are immune to stuff just being broke.