Home / server / Questions / 445445
Accepted
Jacek Konieczny
Asked: 2012-11-05 23:43:29 +0800 CST

tmpfs fills up, although hardly used. How can I debug this

  • 772

I have a system with / on tmpfs. Most / subdirectories have aufs mounted overlaying the read-write root filesystem with read-only base filesystem (the system boots from a read-only medium). Earlier, I used to use unionfs instead of aufs. It has been working properly until recently the tmpfs started to fill up. I am not sure what triggered the change. It could be the unionfs to aufs change, a kernel upgrade or some changes in the system and how it accesses the file systems.

Anyway, it seems it is the tmpfs which behaves somehow wrong.

Although the system should not write a lot to tmpfs, quite a bit of it is used up:

# df -m /
Filesystem     1M-blocks  Used Available Use% Mounted on
tmpfs                200    50       151  25% /

while:

# du -smx /
2       /

This is my test system, doing basically nothing. Things are wore on production system when usage quickly reach over 90% and the system crashes.

I would suspect these are deleted files still open, but:

# lsof | grep deleted

shows nothing.

The other idea was, that some files on / are masked by a file system mounted over it, so I tried this:

# mount --bind / /mnt
# du -sm /mnt
2       /mnt

Still, no trace of the 48MB lost.

How can I find out what is using up my tmpfs file system?

System information:

# uname -rm
3.4.6 i686

Update: I have tried kernels 3.4.17 and 3.6.6 – no change.

linux

2 Answers

  1. Best Answer

    I have solved the mystery myself, with help of the aufs maintainer, Junjiro Okajima.

    The first step, to debug the problem, was to reproduce it in a controlled way. It took me some time (now I wonder why so much) to find out, that the problem occurs when files are written and deleted via aufs.

    Reproducing the problem

    create mount points:

    # cd /tmp
# mkdir rw
# mkdir mnt

    mount the tmpfs:

    # mount -t tmpfs none /tmp/rw

    mount the aufs, overlaying /usr with /tmp/rw:

    # mount -t aufs  -n -o "br:/tmp/rw:/usr" none "/tmp/mnt"

    now I can see /usr contents under /tmp/mnt:

    # ls /tmp/mnt
bin  games  include  lib  lib64  local  sbin  share  src

    what I am interested in is the used/available space on the tmpfs below:

    # du -sk /tmp/rw   
0   /tmp/rw
# df /tmp/rw  
Filesystem     1K-blocks  Used Available Use% Mounted on
none             1031128    24   1031104   1% /tmp/rw

    No files in /tmp/rw, but 24 blocks allocated. Still not a big problem.

    I can write a file to the aufs, it will be stored on tmpfs in /tmp/rw:

    # dd if=/dev/zero of=/tmp/mnt/test bs=1024 count=100
100+0 records in
100+0 records out
102400 bytes (102 kB) copied, 0.000343903 s, 298 MB/s
# du -sk /tmp/rw
100 /tmp/rw
# df /tmp/rw
Filesystem     1K-blocks  Used Available Use% Mounted on
none             1031128   128   1031000   1% /tmp/rw

    Note how the usage stats changed. du show 100kB added, as expected, but the 'Used' value in the df output increased by 104 blocks.

    When I remove the file:

    # du -sk /tmp/rw   
0   /tmp/rw
# df /tmp/rw
Filesystem     1K-blocks  Used Available Use% Mounted on
none             1031128    28   1031100   1% /tmp/rw

    Four blocks are lost.

    When I repeat the dd and rm commands a few times I get:

    # df /tmp/rw                                         
Filesystem     1K-blocks  Used Available Use% Mounted on
none             1031128    36   1031092   1% /tmp/rw

    More and more tmpfs blocks were gone and I didn't know where…

    Where I did the same – dd and rm directly on /tmp/rw nothing was lost this way. And after un-mounting the aufs, the lost space on tmpfs was recovered. So, at least, I knew it was aufs, not tmpfs to blame.

    What has been happening

    Knowing what to blame, I described my problem on the aufs-users mailing list. I have quickly received first answers. The one from J. R. Okajima helped me to explain what is happening to the missing tmpfs blocks.

    It was a deleted file, indeed. It wasn't shown by lsof or anywhere in /proc/<pid>/* as the file was not opened or mmaped by any user-space process. The file, the 'xino file', is aufs' external inode number translation table and is used internally by the kernel aufs module.

    Path to the file can be read from sysfs:

    # cat /sys/fs/aufs/si_*/xi_path         
/tmp/rw/.aufs.xino

    But, as the file is deleted, it cannot be seen directly:

    # ls -l /tmp/rw/.aufs.xino
ls: cannot access /tmp/rw/.aufs.xino: No such file or directory

    Though, information about its size and sizes of the other special aufs files can be read from the debugfs:

    # for f in /sys/kernel/debug/aufs/si_8c8d888a/* ; do echo -n "$f: " ; cat $f ; done 
/sys/kernel/debug/aufs/si_8c8d888a/xi0: 1, 32x4096 132416
/sys/kernel/debug/aufs/si_8c8d888a/xi1: 1, 24x4096 626868
/sys/kernel/debug/aufs/si_8c8d888a/xib: 8x4096 4096
/sys/kernel/debug/aufs/si_8c8d888a/xigen: 8x4096 88

    The details are described in the aufs manual page.

    The solution

    The 'xino file' can be manually truncated by:

    # mount -o remount,itrunc_xino=0 /tmp/mnt

    Automatic xino file truncation can be requested by using trunc_xino option while mounting the aufs:

    # mount -t aufs -n -o "br:/tmp/rw:/usr,trunc_xino" none "/tmp/mnt"

    I still don't know how does it affect file system performance or if this will really solve my out-of-tmpfs-space problems on production… but I have learned a lot.

    • 10

  2. I have seen this happen where files were deleted but processes were still holding on to the file which meant that the space was not freed until the process was restarted. I have seen this with Apache Log files. It seemed to continue to write to the now deleted log file and the space was not cleared until it was restarted.

    To find out which process might be holding on to deleted files, you might try restarting each process and see if that clears the space. If it does, you have found your culprit.

    HTH

    • 2