BloaterPaste commented recently about how StackOverflow answers were amazingly accurate. I believe it is a huge boon to productivity, and a testament to gameificiation in its purest form.

Recently, as some of you may have noticed, the Smashball website has been going down. Frequently. According to NAGIOS in the last week I’ve logged more downtime than uptime. The problem turned out to be a full disk (well, vmdk in any case). It’s running on a tiny VM in my house running on KUSH, my old ass XP gaming rig. Amusingly, the Ubuntu VM on KUSH is a bit better at serving pages than my AWS infrastructure was. But I digress.

I traced the problem to a 16GB ibdata1 file in /var/lib/mysqld (Ubuntu). I went hunting for a solution and found this awesome post on StackOverflow about purging ibdata1 files.

It is not the answer you want to hear, but it will work. The above SO answer walked me through the procedure of backing up my databases, dropping them, stopping MySQL, deleting the ibdata1 and logfiles, then restarting. Not fun, but I find that events like this make for good practice. I don’t get to lay my hands on a keyboard in the name of sysadmin stuff much anymore in my day job so it’s nice to brush off the rust every so often. To even get to a point where I could perform the above procedure I had to:

  1. create a new smallish vmdk attached to the existing Ubuntu VM
  2. Use LVM to create a couple of logical volumes, laid down ext3 on both
  3. rsync’d a couple of directories over to the logical volumes
  4. did the rm/mount shuffle on those directories
  5. modified fstab so the new volumes mount on boot

You will notice that the site is now back up. Let’s hope for more uptime now that that 16GB ibdata file is gone!