Out of disk space, what's the source?

I think you have something writing to a file that has been deleted from the drive but not yet closed by the application/server, so the space remains allocated on disk but can't be seen by du since the file was removed from the filesystem. The lsof program lists processes that have files open. If you had more filesystems mounted and the number didn't fluctuate so much, then I would have suggested that you had a filesystem mounted on top of a directory that wasn't empty (though you could try umount /var/lib/ureadahead/debugfs and make sure that directory is empty and there isn't a bunch of junk written to the directory hiding under that mountpoint).

If this is the case, then you should easily find these with sudo lsof | grep deleted. lsof includes (deleted) in the last column if a file has been deleted while a process still has it open. The first column is the name of the command, the second column is the PID. You can get a more detailed look at the command using ps for instance ps auxww | grep PID, or ps auxwwf | less -S to see the process list in "forest" mode so you can see what process that PID came from. Once you've tracked down the process(es) that are holding open giant files, you can stop it to free the drive space, then figure out how to fix it to close the file properly. The usual cause of this is a logrotate script that renames/deletes log files but doesn't notify the application that it has done so (either through an appropriate signal with kill or by restarting the application), so the application continues to hold the old log file open.

Run

du -h --max-depth=1 /

And it should give a clearer picture. If its coming and going it sounds like temp files being created and then not deleted once done with, until whichever process is causing it crashes. What OS is this server runnng and is it running anything in particular?

It looks like the problem is /var/lib/ureadahead/debugfs. It appears this is a known problem, here is a link to ubuntuforums with more information http://ubuntuguide.net/howto-fix-ureadahead-problem-after-upgrading-to-ubuntu-10-04. The tl;dr seems to be update and upgrade, sudo mv /etc/init.d/ureadahead.conf /etc/init.d/ureadahead.conf.disabled, then reboot. Of course, I am assuming you are running 10.04.

My guess is the log files; I had so many PHP 5.3 "deprecated" warnings in my Apache logs on a dev server that I wasn't really paying attention to that it chewed up all 8GB of space on my var partition (as a sidebar to the problem: you should always put /var on a separate partition that your root partition as running out of space can cause system instability issues).

If the space was consumed very quickly (not in ages), Its probably just file allocation.

The cause could be huge swap or temporary files for some application, Which are emptied after its process.

Do a du --max-length=1 when space is consumed a lot.

If you think your root folder is taking too much (3.3 GB) try ll -a / and post the results.

It seems like /var/lib/ureadahead/debugfs may be a red-herring. Here's why...

While /var/lib/ureadahead/debugfs does exist in /etc/mtab, it is not found in /proc/mounts:

$ mount | grep debug
none on /sys/kernel/debug type debugfs (rw)
none on /var/lib/ureadahead/debugfs type debugfs (rw,relatime)

$ cat /proc/mounts | grep debug
none /sys/kernel/debug debugfs rw,relatime 0 0

The df command seems to be reporting exactly the same thing for /var/lib/ureadahead/debugfs and /

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             10321208   1681128   8115792  18% /
none                    830388       120    830268   1% /dev
none                    880752         0    880752   0% /dev/shm
none                    880752        60    880692   1% /var/run
none                    880752         0    880752   0% /var/lock
none                    880752         0    880752   0% /lib/init/rw
none                  10321208   1681128   8115792  18% /var/lib/ureadahead/debugfs
/dev/sdb             153899044    192068 145889352   1% /mnt

Creating a 1GB file in /tmp:

$ dd if=/dev/zero of=/tmp/carypjunk.out bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 52.7234 s, 20.4 MB/s

Shows the size reported in both places:

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             10321208   2730216   7066704  28% /
none                    830388       120    830268   1% /dev
none                    880752         0    880752   0% /dev/shm
none                    880752        60    880692   1% /var/run
none                    880752         0    880752   0% /var/lock
none                    880752         0    880752   0% /lib/init/rw
none                  10321208   2730216   7066704  28% /var/lib/ureadahead/debugfs
/dev/sdb             153899044    192068 145889352   1% /mnt

So, it seems like /var/lib/ureadahead/debugfs device is a red-herring as it is just mirroring the stats from /. If you are running out of space, it is due to something filling up your root filesystem. I would check your /var/log first.

The problem was being initiated by a cron task executing a php CLI command every minute. The PHP code seemed to be stuck in some kind of insanity loop of caught errors and massive amount of debug data growing at the speed of the processor.

As the php code being executed took longer than a minute it did not consider the job done, it kept executing again and again increasing the speed of the growth of the (temporary?) data.

The same task has been running for close to a month with no problems, so it was not in my mind as a cause.

The strange thing is that the php script sets the max execution time manually

I checked the php.ini for clues

; Maximum execution time of each script, in seconds
; http://php.net/max-execution-time
; Note: This directive is hardcoded to 0 for the CLI SAPI
max_execution_time = 30

; Maximum amount of time each script may spend parsing request data. It's a good
; idea to limit this time on productions servers in order to eliminate unexpect$
; long running scripts.
; Note: This directive is hardcoded to -1 for the CLI SAPI
; Default Value: -1 (Unlimited)
; Development Value: 60 (60 seconds)
; Production Value: 60 (60 seconds)
; http://php.net/max-input-time
max_input_time = 60

It says that values are hardcoded to unlimited for the CLI! O_o