Home / server / Questions / 73051
Accepted
mark
Asked: 2009-10-10 10:46:04 +0800 CST

"No space left on device", df shows discrepancy

  • 772

A few hours ago my root partition filled up, I moved files away from it and df reports:

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1             183G  174G     0 100% /

So there should be 9GB free, but avail reports 0 and Use is still at 100%.

I tested as root, e.g.

# echo test >a ; cat a
test

it works as expected; however as a normal user, I still get the error:

$ echo test >a ; cat a
bash: echo: write error: No space left on device

The root home directory where I conducted the positive test and my home directory are on the same partition.The fstab entry is:

/dev/hda1 / ext3 noatime,defaults,errors=remount-ro 0 1
linux

6 Answers

  1. Best Answer

    Most filing systems reserve a certain percentage for root, so you can still log in as root and solve out of diskspace issues. Usually this is 5%. 9GB is roughly 5% of 183GB, so this would make sense. You can see how much is reserved using tune2fs:

    # tune2fs -l /dev/sda1 | grep -i reserved
Reserved block count:     936488
Reserved GDT blocks:      1019
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)

    You can modify it using 

    # tune2fs -m 3 /dev/sda1
tune2fs 1.41.9 (22-Aug-2009)
Setting reserved blocks percentage to 3% (561893 blocks)

    On modern large drives 5% is probably a little excessive, and you probably want to set it lower. You don't want to set it to zero.

    • 36

  2. im with bob, try df -i, if you have a bunch of ill output crontabs for instance, your /var/spool/clientmqueue/ directory can get filled up

    • 5

  3. Take a look at INODES too. In a "vanilla" installation, if you have to many small files they can consume inodes, but not space. You'll see that you have available space, but since your inodes are full, you'll not be able to use this space.

    • 3

  4. "By default, every filesystem in Unix has some space reserved for the superuser (root). This means that no regular Unix user can fill your filesystem up to 100%, and so it's always going to have enough free space to continue normal function."

    From: http://www.unixtutorial.org/commands/tune2fs/

    • 2

  5. Another thing to check is to see if there are any open files on that filesystem (especially logs). Deleting files won't clear the space on the disk until the file is actually closed.

    • 1

  6. I'd bet $1 that Clyde has the answer. A process has a file open on that device. In linux, the file isn't actually removed until the process holding it open lets go of it.

    I'd start with: lsof | grep hda1

    • 0