Home / ubuntu / Questions / 1377316
In Process
Ahmad Ahmadi
Asked: 2021-11-25 01:24:01 +0800 CST

ceph stuck in active+remapped+backfill_toofull after lvextend an OSD's volume

  • 772

I have a 2 OSDs ceph cluster. The initial size of backing volumes was 16GB. Then I shutdown OSDs, did a lvextend on both, and turn OSDs on again. Now ceph osd df shows:

ceph osd df

But ceph -s show it's stucked at active+remapped+backfill_toofull for 50 pgs: ceph -s

I tried to understand the mechanism by reading CRUSH algorithm but seems a lot of effort and knowledge is required. It would be very appreciated if anyone could describe the behaviour (why stuck in toofull despite the free space is increased significantly.) and help me to resolve this state.

ceph

1 Answers

  1. Your RAW USE is times larger than DATA. Note: solution not tried, still this is what I've found.

    Re: Raw use 10 times higher than data use

    Probably the first thing to check is if you have objects that are under the min_alloc size. Those objects will result in wasted space as they will use the full min_alloc size.

    Similar advice: https://stackoverflow.com/questions/68185503/ceph-df-octopus-shows-used-is-7-times-higher-than-stored-in-erasure-coded-pool/68186461#68186461

    This is related to bluestore_min_alloc_size_hdd=64K (default on Octopus).

    If using Erasure Coding, data is broken up into smaller chunks, which each take 64K on disk.

    Another way to fix the problem one may try to follow instructions on your second screen:

    add storage if this doesn't resolve itself

    • 0