The monitoring system we use shows several servers, related to the same application servers, are out of disk space. The application owner stated they cleared Gigs of data but this change was not reflected.
$ df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/sdc1 59G 56G 554M 100% /storage
$ sudo du -hs /storage
56G /storage
The difference between Size and Used indicates there should be some free space. A few posts recommended that a reboot would correct this issue but it's been over a month with no maintenance window.
Is there another way to to update the percentage to confirm the disk space is free?
Another site/post suggested looking for deleted files still held by a process. None of those found are in the mount in question.
sudo lsof +L1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
tuned 1392 root 7u REG 253,0 4096 0 8133 /tmp/*** (deleted)
gnome-she 5256 gdm 19u REG 253,0 12288 0 701887 /tmp/*** (deleted)
pulseaudi 5693 gdm 6u REG 0,4 67108864 0 32735 /memfd:*** (deleted)
I dont know which FS type you are using, but in ext3/ext4 there is a 5% of space that will get reserved for special processes so that they can continue to run when diskspace is running low.
Thats why you only have 56GB of usable space on your FS