Home / server / Questions / 1151862
Accepted
Mikko Rantalainen
Asked: 2024-01-19 21:58:16 +0800 CST

How to find the underlying device for a file on top of ext4 on top of LVM on top of multiple underlying storage devices?

  • 772

I have a system with 3 SSD devices (/dev/sda, /dev/sdb, /dev/sdc) which contain a single LVM logical volume that span over all the devices. I have a single ext4 partition on the logical volume.

I think that one of the SSD devices (/dev/sdb) might be somewhat faulty and have reduced performance compared to other devices.

Is there a command to get the list of files that are backed by that device?

I know that I can get list of logical segments with sudo pvdisplay -m and the output looks like following:

  --- Physical volume ---
  PV Name               /dev/sda
  VG Name               storage
  PV Size               <1,82 TiB / not usable <1,09 MiB
  Allocatable           yes (but full)
  PE Size               4,00 MiB
  Total PE              476932
  Free PE               0
  Allocated PE          476932
  PV UUID               h3x3O1-1KWj-3pY6-kZ24-MVV4-54UE-ltEdfA
   
  --- Physical Segments ---
  Physical extent 0 to 476931:
    Logical volume  /dev/storage/vm
    Logical extents 0 to 476931
   
  --- Physical volume ---
  PV Name               /dev/sdb
  VG Name               storage
  PV Size               <3,64 TiB / not usable <3,84 MiB
  Allocatable           yes (but full)
  PE Size               4,00 MiB
  Total PE              953861
  Free PE               0
  Allocated PE          953861
  PV UUID               MsNlhh-W2It-CbX4-IxJn-lXJN-hlcd-EpBh9Q
   
  --- Physical Segments ---
  Physical extent 0 to 953860:
    Logical volume  /dev/storage/vm
    Logical extents 476932 to 1430792
   
  --- Physical volume ---
  PV Name               /dev/sdc
  VG Name               storage
  PV Size               <3,64 TiB / not usable <3,84 MiB
  Allocatable           yes (but full)
  PE Size               4,00 MiB
  Total PE              953861
  Free PE               0
  Allocated PE          953861
  PV UUID               sklK6w-XZd6-DqIp-ZT1g-O9rj-1ufw-UaC0z4
   
  --- Physical Segments ---
  Physical extent 0 to 953860:
    Logical volume  /dev/storage/vm
    Logical extents 1430793 to 2384653

So I know that logical extends 476932 to 1430792 are the potentially problematic area. How to map this logical segment range to actual files on the filesystem (ext4) on top of the LVM?

Basically I'm trying to figure out if the device is actually faulty or if the usage pattern for these files might be unlucky enough that I'm hitting usage pattern that's problematic for the hardware and the performance is worse than expected. No device is showing any errors and all the data looks good, but the performance of this single device appears to be worse than expected.

The system is in use so I'd prefer to diagnose this online without overwriting any data. I know that if I could simply take the potentially problematic storage device offline and overwrite its contents, I could use fio to benchmark it to see if it's working below spec or not.

$ lsblk -s
...
storage-vm 253:0    0   9,1T  0 lvm  /mnt/storage
├─sda        8:0    0   1,8T  0 disk 
├─sdb        8:16   0   3,7T  0 disk 
└─sdc        8:32   0   3,7T  0 disk

I'm basically asking how to get list of files backed by a single storage device when the filesystem spans over multiple storage devices.

Or if you can provide instructions how to figure where a given file is actually stored, that would be fine, too. I would then run that routine for every file to figure out which files are backed by the device I'm interested in. I'm aware that it could be that a single big file is backed by all devices if the file is fragmented over a big range of locigal segments so the answer could be that a single file is backed by all the devices but I have currently no idea how to do this either.

$ sudo vgdisplay 
  --- Volume group ---
  VG Name               storage
  System ID             
  Format                lvm2
  Metadata Areas        3
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                3
  Act PV                3
  VG Size               <9,10 TiB
  PE Size               4,00 MiB
  Total PE              2384654
  Alloc PE / Size       2384654 / <9,10 TiB
  Free  PE / Size       0 / 0   
  VG UUID               MOrTMY-5Dly-48uQ-9Fa8-JNvf-tont-9in7ol

$ sudo lvdisplay 
  --- Logical volume ---
  LV Path                /dev/storage/vm
  LV Name                vm
  VG Name                storage
  LV UUID                RDkaLH-mh6C-cXxT-6ojc-DxkB-o4jD-3CMHdl
  LV Write Access        read/write
  LV Creation host, time staging, 2021-01-21 09:57:06 +0200
  LV Status              available
  # open                 1
  LV Size                <9,10 TiB
  Current LE             2384654
  Segments               3
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0
performance

2 Answers

  1. You might be able to get a general idea of where the file is located using debugfs, but it will greatly depend on how your LVs were created. If they are using the striped type, the data isn't written sequentially to the drives (first drive1, then drive 2, etc), so the extents would be divided up.

    Remember that the file system layer (ext4) doesn't know or care about the underlying block device. It just sees it as a large chunk of space it can use to create files. Likewise, the LV doesn't know or care about the overlaying file system, as its job is to just manage its own underlying physical devices.

    Because of this, what the LV calls extent 476932 to 1430792 isn't necessarily what the filesystem calls those extents. However, since it's such a large range of extents, you might be able to at least get in the ballpark.

    An example of using debugfs on /dev/xvda2, which is my root (/) partition:

    # debugfs /dev/xvda2
debugfs 1.46.5 (30-Dec-2021)
debugfs:  ls /root
 262146  (12) .    2  (12) ..    268932  (12) .ssh
 268409  (16) .bashrc    268410  (16) .profile    276186  (16) .cache
 268256  (16) .ansible    268318  (24) .vim
 268408  (24) .bash_completion
 267590  (16) .vimrc    262167  (24) .bash_history
debugfs:  stat /root/.bashrc
Inode: 268409   Type: regular    Mode:  0644   Flags: 0x80000
Generation: 4101506038    Version: 0x00000000:00000001
User:     0   Group:     0   Project:     0   Size: 10137
File ACL: 0
Links: 1   Blockcount: 24
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x64c7f06b:37648614 -- Mon Jul 31 12:33:31 2023
 atime: 0x660f7b2c:32146cc8 -- Thu Apr  4 23:16:44 2024
 mtime: 0x64c7ef9d:00000000 -- Mon Jul 31 12:30:05 2023
crtime: 0x641493b2:49f692b4 -- Fri Mar 17 11:22:10 2023
Size of extra inode fields: 32
Inode checksum: 0x8cc75e77
EXTENTS:
(0-2):4145465-4145467

    You can see that this file is located at extents 4145465-4145467. If the underlying LV was created as a linear LV, it's possible these extents are the same or very similar.

    • 1
  2. Best Answer

    First of all, use filefrag utility to find the list of all file extents:

    merlin@uc-s4m75657:~$ sudo filefrag /mnt/spool/merlin/VMs/slax-64bit-11.4.0-merlin.iso -e
Filesystem type is: ef53
File size of /mnt/spool/merlin/VMs/slax-64bit-11.4.0-merlin.iso is 322979840 (78853 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..    8191:    2703360..   2711551:   8192:            
   1:     8192..   14335:    2719744..   2725887:   6144:    2711552:
   2:    14336..   57343:    2732032..   2775039:  43008:    2725888:
   3:    57344..   65535:    2783232..   2791423:   8192:    2775040:
   4:    65536..   71679:    2797568..   2803711:   6144:    2791424:
   5:    71680..   78852:    2811904..   2819076:   7173:    2803712: last,eof
/mnt/spool/merlin/VMs/slax-64bit-11.4.0-merlin.iso: 6 extents found

    physical_offset will give you an overview where in the filesystem the extents of the file are located. Notice these numbers are in terms of filesystem blocks, which are 4k in this case.

    For example, the second extents of this file starts at byte 11140071424 from the filesystem start.

    Next, explore your LVM layout. Run sudo lvm vgcfgbackup -v, it will dump a layout of each VG into /etc/lvm/backup/<vgname> in a textual form (and -v switch will make it tell you these names, if you lazy). It's important to use a freshly created backup dump, for the "hint" device names in it to be actually used devices at the moment (an old dump will refer to the VG state when you last time made changes in it, which may have happened a few reboots ago, so the actual device names may have been changed since then).

    Read the dump for the corresponding VG. It is quite verbose, but easy to understand. First of all, it lists various common information about the VG, of which PE size is important. Then it lists PVs and make note of which PV you are interested in.

    And below, all LVs are listed, as the set of segments. For each segment, it says where it is mapped, PV and location, in terms of LVM extents.

    Look at the example (I stripped unrelated parts):

    system {
#...
        extent_size = 8192              # 4 Megabytes
#...
        physical_volumes {
                pv0 {
#...
                        device = "/dev/sda4"    # Hint only
#...                        
                        pe_start = 2048
                        pe_count = 57105        # 223,066 Gigabytes
                }
        }

        logical_volumes {
#...
                spool {
#...
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 41361    # 161,566 Gigabytes
#...
                                stripes = [
                                        "pv0", 15744
                                ]
                        }
                }
        }

}

    This is the LV where filesystem from above is located. We see that it is fully located on the pv0, and the above extent of the file starts at byte 11140071424 + 15744 * 4MiB = 77175193600 of the device sda4. If the filesystem was spanning several segments, I'd have to subtract their size (in bytes) from the file extent byte location (11140071424), until I ended up in the middle of a certain segment.

    • 1