ubuntu server slowly filling up

Use du to track down the directory that contains the file(s) that are filling the disk.

cd /
du -h --max-depth 1

will show you which directory in / uses the most space. Traverse the filesystem running the du command to find the culprit.

e.g.

cd /
du -h --max-depth 1

shows /usr is usong 2.3G of the 3.5G used on the system.

cd /usr
du -h --max-depth 1

shows /usr/lib uses 1.1G of the 2.3 in /usr ...

This may also be caused by an open file that has been deleted.

You can use lsof to find files that are open but unlinked (deleted)

lsof +L1

should do the trick. As the man page states:

A specification of the form +L1 will select open files that have been unlinked. A specification of the form +L1 <file_system> will select unlinked open files on the specified file system.

(This is a Linux focused answer. Other UNIX variants may be different.)

There are two pieces of information relevant to your problem: (1) which files are filling up your file system, and (2) which processes are writing to those files.

Notes

Below, when I put the $ character in commands, that's probably a place holder where you need to substitute a real value. Hopefully, it's obvious where to do that and where not to.

Which Files?

Be aware that there are really two resources in most file system types that can be used up by individual files: meta-data (e.g. inodes), and real data. You can see the number of inodes (search Google for a definition, but they're "pointers" to the structures that make up your files) with a command like:

df -i

... and as you already know, something like this will show the space being used by real data:

df -h

Also, be aware that file system space can be taken up by files that don't exist on disk. These files are still in the open state by some process, but have been removed (we'll cover that below).

Once you've identified the full file system(s), then you need to start looking for lots of little files, a few big files, or both. Running out of meta-data resources is usually caused by having lots of little files, whereas running out of real data resources is usually caused by a few big files. I like to use this command to help find the big files:

sudo find $file_system -mount -ls | awk '{print $7, $11}' | sort -rn > $output

... and this command to help find directories with lots of little files (Update:: added null termination to improve file name handling):

sudo find . -mount -print0 | xargs -0n 1 dirname | sort | uniq -c | sort -rn > $output

... be aware that these commands may take a while to run, and do a lot of I/O, depending. Once run, you can read through $output to find the offending files or directories. The name and location of each may give you a hint about where the data's coming from, but requires some Linux experience.

Once you've identified the offenders, you can rm $file to get rid of the problem.

Which Processes?

The most straight forward way to find the processes potentially filling up your file system is to run a command like:

fuser -c $file_system 2>/dev/null

... which will tell you the PID of processes that have open file descriptors (files and network sockets) for the given file system (the 2>/dev/null part gets rid of some information you don't need). You might be able to deduce just from these PIDs which process is filling up your file system. Search for the processes with:

ps -ef | grep $pid

You can also try running this command which will give you even more detail (and help identify open files with no corresponding file name on the disk -- I mentioned this above):

sudo lsof $file_system | grep $directory_filling_up

... and if you've identified a suspect PID from the fuser command, you can do this:

sudo lsof -p $pid

The problem with fuser and lsof is that they only give you a snapshot of the system at the time your run the command. If the offending process doesn't happen to be writing when you run them, you're out of luck. You can counter this by repeatedly running them over time and saving the output. This will require reading through the output to find patterns, or writing a program to do it for you. An alternative is to use a tool like SystemTap. SystemTap allows you to trap all kinds of useful information, and is "programmable." It even comes with some sample source files that would allow you see what processes are writing to what files over some span of time. It would be perfect, but it's an advanced tool and requires a lot of Linux knowledge.

Once you've identified the offending process(es), you can kill (and maybe restart them). If the process is associated with the operating system or some well packaged software, there will probably be a mechanism for restarting them, but it will depend on your Linux distro (I think Ubuntu will allow you to run something like /etc/init.d/$init_script restart, but you'll have to check your distro's documentation). Otherwise, you can kill it with kill $pid or kill -9 $pid if it's not behaving. Be careful to note how the process was running (e.g. what were it's arguments shown in ps -ef) in case you need to restart it (you may need to refer to that software's documentation).

Something is filling up the / partition. It is probably something in /var/log , or in /home . This depends on your setup. Also look in places where your users have access.

Run the following command in each of the directories in question. This will show you the sub-directories which are the largest consumers of space.

cd /directory
du -cks -x * .* |sort -n

This idea is borrowed from the ducks script (du -cks) from Linux Server Hacks from O'Reilly. I run this command often.

In my experience, this is almost always due to large, growing logfiles. In this case, use Logrotate, and be sure to use compression. Using gzip compression with the default compression ratio, your logfiles will be made smaller by 80-95% (A 1GB /var/log/messages can be easily compressed down to 200MB or less). This puts a moderate amount of load on the CPU, but I have rarely seen this impact the real performance of a server. Some people prefer to use Bzip2 compression, or use gzip --best but in my experience this causes alot of CPU overhead with little added benefit. gzip with the default ratio is usually good enough.

And obviously, this problem is sometimes due to a user doing bad things. Use the du command above to find the culprit.

I would use the du command to see which directories are taking up more space, which should suggest which programs are using that space. If you can run graphical apps, there are some nice ones which will help to summarize du's output, such as KDirStat.

The likely culprit is the logs, but here is a command that will sort recently modified (or created) files by size:

D=$(date --rfc-3339 date);
sudo sh -c 'find / -xdev -mtime -1 -type f -print0 |xargs -0 du -0sbc' \
  |tee ~/recent-files.$D |sort -zn |tee ~/recent-by-size.$D |xargs -0n1

You could run this command daily; there's probably a way to do something SQL-ish to sort these files by daily growth.

(edit) To monitor growth, use gt5

sudo aptitude install gt5
cd /
gt5

One day later; look for ± signs

gt5

Log files may be filling up your hard drive. Use logrotate to stop that.

Thanks everyone for your help

It turns out that the culprit was a hidden .recycler folder in each of the shared director that was hidden.

If you do a ls -a you can see them.