Woke up this morning to find out MongoDB grew to all available disk space for some reason. I cannot repair the db to compact it, as I have no space left. I can't even start postgres to remove an old database which would free up a few gigs more. I literally have zero free disk space.
I did delete a 9GB file but the space is not showing up at all when I "df -h".
I have to clear up space on the drive asap as its a production server and we're down. What to do?
the space will not show up until all processes release the file. Try using lsof to see what's using the file.
Although the answer provided is valid in many cases, in my scenario I ended up falling into the ext3 5% reserved root space. I could have recovered this space had I known about it at the time. I ended up using a USB drive as a repair path to mongodb reclaiming back 850GB.
This is how to recover that ext3 5% should anyone be interested:
http://www.andremiller.net/content/recovering-reserved-space-ext2-and-ext3-filesystems