Home / server / Questions / 814430
In Process
Shailesh Sutar
Asked: 2016-11-12 07:57:21 +0800 CST

Inode usage gone up to 100% hence can't uninstall any package

  • 772

My server's inode usage has gone up to 100% due to which I am not able to uninstall extra linux-kernel-headers and linux-kernel-images 

$ df -i
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
/dev/xvda1     524288 523970    318  100% /
none           481856      2 481854    1% /sys/fs/cgroup
udev           480561    396 480165    1% /dev
tmpfs          481856    317 481539    1% /run
none           481856      1 481855    1% /run/lock
none           481856      1 481855    1% /run/shm
none           481856      3 481853    1% /run/user
/dev/xvdb      262144     11 262133    1% /mnt

Also HDD usage are up to only 78%

$ sudo df -kh
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  5.7G  1.7G  78% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            1.9G   12K  1.9G   1% /dev
tmpfs           377M  368K  377M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            1.9G     0  1.9G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/xvdb       3.9G  8.1M  3.7G   1% /mnt

When I try to remove headers and images using sudo apt-get autoremove command, It gives me below error.

$ sudo apt-get autoremove
Reading package lists... Done
Building dependency tree       
Reading state information... Done
You might want to run 'apt-get -f install' to correct these.
The following packages have unmet dependencies:
 linux-headers-3.13.0-83-generic : Depends: linux-headers-3.13.0-83 but it is not installed
E: Unmet dependencies. Try using -f.

When I do sudo apt-get -f install, It gives me below error. 

dpkg: error processing archive /var/cache/apt/archives/linux-headers-3.13.0-83_3.13.0-83.127_all.deb (--unpack):
 unable to create `/usr/src/linux-headers-3.13.0-83/arch/sh/include/asm/sparsemem.h.dpkg-new' (while processing `./usr/src/linux-headers-3.13.0-83/arch/sh/include/asm/sparsemem.h'): No space left on device
No apport report written because the error message indicates a disk full error
                                                                              dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/linux-headers-3.13.0-83_3.13.0-83.127_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

When I do $ sudo dpkg --configure -a, It gives me below error.

    libpostfix-dns.so.1 -> libpostfix-dns.so.1.0.1
/sbin/ldconfig.real: Can't create temporary cache file /etc/ld.so.cache~: No space left on device
dpkg: error processing package libc-bin (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: error: unable to create new file '/var/lib/dpkg/status-new': No space left on device

Now I am not sure how to tackle this issue. Any experts advice would be a great input. I am not sure from where should I start deleting files. Here is description of filesystem.

/$ sudo du -sh *
9.6M    bin
418M    boot
12K dev
6.9M    etc
216K    home
0   initrd.img
0   initrd.img.old
789M    lib
4.0K    lib64
16K lost+found
4.0K    media
28K mnt
684M    opt
du: cannot access ‘proc/21817/task/21817/fd/4’: No such file or directory
du: cannot access ‘proc/21817/task/21817/fdinfo/4’: No such file or directory
du: cannot access ‘proc/21817/fd/4’: No such file or directory
du: cannot access ‘proc/21817/fdinfo/4’: No such file or directory
0   proc
23M root
372K    run
9.4M    sbin
4.0K    srv
0   sys
19M tmp
2.9G    usr
669M    var
0   vmlinuz
0   vmlinuz.old

Note: I have already deleted log files older than 20 days. It didn't help to minimise inode usage and I am stuck now.

linux apt inode linux-kernel ubuntu-14.04

2 Answers

  1. Given your df -i output and seeing that you are not using any LVM - the only solution you have is to start deleting files.
    Every file present on the system / partition will consume one inode.
    Directories also consume inode.
    So you need to figure out what's driving this number of files up.
    Perhaps it's some process creating many temp files.
    Perhaps it's log files.

    You can use below command to see which directory is the biggest offender

    find / -printf "%h\n"|grep -v "^/proc"|grep -v "^/sys"|cut -d\/ -f1-3|sort|uniq -c|sort -rn
    • 0

  2. In my case the / partition was showing as 100% inode usage due to which I was not able to uninstall any extra linux-headers and extra kernel linux-images. 1. First I deleted all the logs which were older than 20 days 2. I moved some files from /var/cache/ folder to /mnt/test/ folder which caused 3% free inodes on my system. 

        $ sudo apt-get clean 
    $ sudo apt-get autoremove
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    The following packages will be REMOVED:
      linux-headers-3.13.0-67 linux-headers-3.13.0-67-generic
      linux-headers-3.13.0-68 linux-headers-3.13.0-68-generic
      linux-headers-3.13.0-70 linux-headers-3.13.0-70-generic
      linux-headers-3.13.0-71 linux-headers-3.13.0-71-generic
      linux-headers-3.13.0-73 linux-headers-3.13.0-73-generic
      linux-headers-3.13.0-74 linux-headers-3.13.0-74-generic
      linux-headers-3.13.0-76 linux-headers-3.13.0-76-generic
      linux-headers-3.13.0-77 linux-headers-3.13.0-77-generic
      linux-image-3.13.0-67-generic linux-image-3.13.0-68-generic
      linux-image-3.13.0-70-generic linux-image-3.13.0-71-generic
      linux-image-3.13.0-73-generic linux-image-3.13.0-74-generic
      linux-image-3.13.0-76-generic linux-image-3.13.0-77-generic
    0 upgraded, 0 newly installed, 24 to remove and 144 not upgraded.
    After this operation, 955 MB disk space will be freed.
    Do you want to continue? [Y/n] Y

    After uninstalling those headers and images, inode usages are dropped to 64% now

    $ df -i
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
/dev/xvda1     524288 331671 192617   64% /
none           481856      2 481854    1% /sys/fs/cgroup
udev           480561    396 480165    1% /dev
tmpfs          481856    317 481539    1% /run
none           481856      1 481855    1% /run/lock
none           481856      1 481855    1% /run/shm
none           481856      3 481853    1% /run/user
/dev/xvdb      262144   1933 260211    1% /mnt

    of course this is not the only way to deal with 100% inode usage problem. But In my case it worked since my /mnt/ partition had free inodes.

    • 0