I have a plugin called Broken Links Checker that I have used in the past to find broken links within the wordpress postings. I’ve used it in the past without consequence. It would dutifully go through my posts and report back to me the links that no longer work. Pretty handy, I can go and disable those links, make them strike-through, etc.
I updated it along with updating wordpress to 3.0.3 the other day. I fired off the Broken Links Checker, and as it usually takes a while, I ignored it and went on playing Eve and writing some ruby code (a simple little project to organize a whole lot of files into 4.7GB disks and include manifests and md5s of all the files for an archive. Still doesn’t work yet.) The next morning, I woke up to a large amount of email. I figured it was just spam or something that was significantly different than the normal spam. Unfortunately it was notifications from my Nagios telling me that the box doesn’t respond to ssh or http anymore. Uh oh.
Luckily, I’m running Xen, so it is trivial to get to the console over ssh. I connect to the console and try to log in. The box is totally wedged. OOM Killer had gone crazy with httpd, and anything else that was on that box. So xm destroy. Fire it back up again, wait for it to replay all the transactions, and it had decided to check the disks, since they haven’t been checked in 214 days. All came back up fine. Nagios was again satisfied. However, none of the sites worked. “Error: Cannot access database.” Oh noes. Started the normal debugging process, looking through logs and what not. Found a bunch of local requests from Broken Links Checker checking links. It appears that it caused a DOS to myself. Wonderful. Still no luck with MySQL, so I tried to connect to the mysql server via the command line, I had already determined that it was up, and it’s on a different box, so it didn’t get oom killed when apache killed the box. I get an error telling me that MySQL was denying connections from this host due to too many errors. So a flush-hosts later, and it all works again. Fun.
Guess I should set up some kind of resource limits for Apache on that box. Also, I’m removing that Broken Links Checker plugin, I don’t need to find my broken links that badly…
