We are facing a problem with a ceph osd node which uses an XFS file system.
The mount
has become non-writeable. It gives the following error:
root@fab-prod-ceph-h3:/var/lib/ceph/osd/ceph-6# touch hrushi
touch: cannot touch `hrushi': No space left on device
Both, df -i
and df -h
, indicate a consumption of about ~40%.
Here is the output:
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 178921472 60895 178860577 1% /
udev 8240393 597 8239796 1% /dev
tmpfs 8242709 582 8242127 1% /run
none 8242709 3 8242706 1% /run/lock
none 8242709 1 8242708 1% /run/shm
/dev/sdd1 73111872 25981652 47130220 36% /var/lib/ceph/osd/ceph-8
/dev/sdc1 73111872 30445183 42666689 42% /var/lib/ceph/osd/ceph-7
/dev/sde1 97530624 1913758 95616866 2% /var/lib/ceph/osd/ceph-15
/dev/sdf1 97530624 2680246 94850378 3% /var/lib/ceph/osd/ceph-16
/dev/sdg1 195206272 2206318 192999954 2% /var/lib/ceph/osd/ceph-17
/dev/sdb1 146223870 33812416 112411454 24% /var/lib/ceph/osd/ceph-6
10.100.112.53:/srv/data/Alcatraz 98304000 15954028 82349972 17% /exporter
root@fab-prod-ceph-h3:/var/lib/ceph/osd/ceph-6# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 2.7T 313G 2.2T 13% /
udev 32G 12K 32G 1% /dev
tmpfs 13G 364K 13G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 32G 0 32G 0% /run/shm
/dev/sdd1 2.8T 1.8T 1010G 64% /var/lib/ceph/osd/ceph-8
/dev/sdc1 2.8T 1.7T 1.2T 60% /var/lib/ceph/osd/ceph-7
/dev/sde1 3.7T 158G 3.5T 5% /var/lib/ceph/osd/ceph-15
/dev/sdf1 3.7T 185G 3.5T 5% /var/lib/ceph/osd/ceph-16
/dev/sdg1 7.3T 155G 7.2T 3% /var/lib/ceph/osd/ceph-17
/dev/sdb1 2.8T 1.9T 937G 67% /var/lib/ceph/osd/ceph-6
10.100.112.53:/srv/data/Alcatraz 1.5T 901G 502G 65% /exporter
A few community forum postings suggested increasing the space allocated to save inodes.
Initial:
root@fab-prod-ceph-h3:/var/lib/ceph/osd/ceph-6# xfs_info /dev/sdb1
meta-data=/dev/sdb1 isize=2048 agcount=32, agsize=22847480 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=731119355, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=356991, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Extended
root@fab-prod-ceph-h3:/var/lib/ceph/osd/ceph-6# xfs_growfs -m 10 /var/lib/ceph/osd/ceph-6
meta-data=/dev/sdb1 isize=2048 agcount=32, agsize=22847480 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=731119355, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=356991, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
inode max percent changed from 5 to 10
root@fab-prod-ceph-h3:/var/lib/ceph/osd/ceph-6# xfs_db -f -c "sb 0" -c "p" /dev/sdb1 | grep imax_pct
imax_pct = 5
Note:
This value did not change.
How do I resolve this problem?
I have tried upgrading the kernel version from 3.5 to 3.15 as well, but without success.
We ended up cloning the drive to larger sized disk and then expanding it to some extent. Drives became mountable and we could continue using an osd device.