We are facing a problem with a ceph osd node which uses an XFS file system.
The mount has become non-writeable. It gives the following error:
root@fab-prod-ceph-h3:/var/lib/ceph/osd/ceph-6# touch hrushi
touch: cannot touch `hrushi': No space left on device
Both, df -i and df -h, indicate a consumption of about ~40%.
Here is the output:
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 178921472 60895 178860577 1% /
udev 8240393 597 8239796 1% /dev
tmpfs 8242709 582 8242127 1% /run
none 8242709 3 8242706 1% /run/lock
none 8242709 1 8242708 1% /run/shm
/dev/sdd1 73111872 25981652 47130220 36% /var/lib/ceph/osd/ceph-8
/dev/sdc1 73111872 30445183 42666689 42% /var/lib/ceph/osd/ceph-7
/dev/sde1 97530624 1913758 95616866 2% /var/lib/ceph/osd/ceph-15
/dev/sdf1 97530624 2680246 94850378 3% /var/lib/ceph/osd/ceph-16
/dev/sdg1 195206272 2206318 192999954 2% /var/lib/ceph/osd/ceph-17
/dev/sdb1 146223870 33812416 112411454 24% /var/lib/ceph/osd/ceph-6
10.100.112.53:/srv/data/Alcatraz 98304000 15954028 82349972 17% /exporter
root@fab-prod-ceph-h3:/var/lib/ceph/osd/ceph-6# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 2.7T 313G 2.2T 13% /
udev 32G 12K 32G 1% /dev
tmpfs 13G 364K 13G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 32G 0 32G 0% /run/shm
/dev/sdd1 2.8T 1.8T 1010G 64% /var/lib/ceph/osd/ceph-8
/dev/sdc1 2.8T 1.7T 1.2T 60% /var/lib/ceph/osd/ceph-7
/dev/sde1 3.7T 158G 3.5T 5% /var/lib/ceph/osd/ceph-15
/dev/sdf1 3.7T 185G 3.5T 5% /var/lib/ceph/osd/ceph-16
/dev/sdg1 7.3T 155G 7.2T 3% /var/lib/ceph/osd/ceph-17
/dev/sdb1 2.8T 1.9T 937G 67% /var/lib/ceph/osd/ceph-6
10.100.112.53:/srv/data/Alcatraz 1.5T 901G 502G 65% /exporter
A few community forum postings suggested increasing the space allocated to save inodes.
Initial:
root@fab-prod-ceph-h3:/var/lib/ceph/osd/ceph-6# xfs_info /dev/sdb1
meta-data=/dev/sdb1 isize=2048 agcount=32, agsize=22847480 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=731119355, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=356991, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Extended
root@fab-prod-ceph-h3:/var/lib/ceph/osd/ceph-6# xfs_growfs -m 10 /var/lib/ceph/osd/ceph-6
meta-data=/dev/sdb1 isize=2048 agcount=32, agsize=22847480 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=731119355, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=356991, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
inode max percent changed from 5 to 10
root@fab-prod-ceph-h3:/var/lib/ceph/osd/ceph-6# xfs_db -f -c "sb 0" -c "p" /dev/sdb1 | grep imax_pct
imax_pct = 5
Note:
This value did not change.
How do I resolve this problem?
I have tried upgrading the kernel version from 3.5 to 3.15 as well, but without success.