Home / server / Questions / 429985
In Process
Matt Mencel
Asked: 2012-09-20 10:48:29 +0800 CST

Creating Persistent Drive Labels With UDEV Using /dev/disk/by-path

  • 772

I have a new BackBlaze Pod (BackBlaze Pod 2.0).

It has 45 3TB drives and they when I first set it up they were labeled /dev/sda through /dev/sdz and /dev/sdaa through /dev/sdas.

I used mdadm to setup three really big 15 drive RAID6 arrays.

However, since first setup a few weeks ago I had a couple of the hard drives fail on me. I've replaced them but now the arrays are complaining because they can't find the missing drives. When I list the the disks...

ls -l /dev/sd*

I see that

/dev/sda
/dev/sdf
/dev/sdk
/dev/sdp

no longer appear and now there are 4 new ones...

/dev/sdau
/dev/sdav
/dev/sdaw
/dev/sdax

I also just found that I can do this...

ls -l /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root 10 Sep 19 18:08 pci-0000:02:04.0-scsi-0:0:0:0 -> ../../sdau
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:02:04.0-scsi-0:1:0:0 -> ../../sdb
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:02:04.0-scsi-0:2:0:0 -> ../../sdc
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:02:04.0-scsi-0:3:0:0 -> ../../sdd
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:02:04.0-scsi-0:4:0:0 -> ../../sde
lrwxrwxrwx 1 root root 10 Sep 19 18:08 pci-0000:02:04.0-scsi-2:0:0:0 -> ../../sdae
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:02:04.0-scsi-2:1:0:0 -> ../../sdg
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:02:04.0-scsi-2:2:0:0 -> ../../sdh
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:02:04.0-scsi-2:3:0:0 -> ../../sdi
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:02:04.0-scsi-2:4:0:0 -> ../../sdj
lrwxrwxrwx 1 root root 10 Sep 19 18:08 pci-0000:02:04.0-scsi-3:0:0:0 -> ../../sdav
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:02:04.0-scsi-3:1:0:0 -> ../../sdl
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:02:04.0-scsi-3:2:0:0 -> ../../sdm
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:02:04.0-scsi-3:3:0:0 -> ../../sdn
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:02:04.0-scsi-3:4:0:0 -> ../../sdo
lrwxrwxrwx 1 root root 10 Sep 19 18:08 pci-0000:04:04.0-scsi-0:0:0:0 -> ../../sdax
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:04:04.0-scsi-0:1:0:0 -> ../../sdq
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:04:04.0-scsi-0:2:0:0 -> ../../sdr
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:04:04.0-scsi-0:3:0:0 -> ../../sds
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:04:04.0-scsi-0:4:0:0 -> ../../sdt
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:04:04.0-scsi-2:0:0:0 -> ../../sdu
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:04:04.0-scsi-2:1:0:0 -> ../../sdv
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:04:04.0-scsi-2:2:0:0 -> ../../sdw
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:04:04.0-scsi-2:3:0:0 -> ../../sdx
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:04:04.0-scsi-2:4:0:0 -> ../../sdy
lrwxrwxrwx 1 root root  9 Sep 19 18:08 pci-0000:04:04.0-scsi-3:0:0:0 -> ../../sdz

I didn't list them all....you can see the problem above. They're sorted by scsi id here but sda is missing...replaced by sdau...etc...

So obviously the arrays are complaining. Is it possible to get Linux to reread the drive labels in the correct order or am I screwed?

My initial design with 15 drive arrays is not ideal. With 3TB drives the rebuild times were taking 3 or 4 days....maybe more. I'm scrapping the whole design and I think I am going to go with 6 x 7 RAID5 disk arrays and 3 hot spares to make the arrays a bit easier to manage and shorten the rebuild times. But I'd like to clean up the drive labels so they aren't out of order. I haven't figured out how to do this yet.

Does anyone know how to get this straightened out?

Thanks,

Matt

linux

1 Answers

  1. My Solution

    I'm not sure it's the best solution, and I'm still interested in other answers, but here's what I did.

    I could not figure out how to force it to map based on the SCSI path (e.g. pci-0000:02:04.0-scsi-0:0:0:0). So instead I am mapping based on the UUID, which is the drive models and serial numbers. I just have to remember that when I swap drives I have to update the UDEV entries.

    Finding the UUIDs

    Used entries from the following commands to identify the current drive names, what SCSI PATH they were mapped to, and what their UUIDs were.

    # ls -l /dev/disk/by-path
# /lib/udev/scsi-id -gu /dev/sdX

    Created UDEV entries

    I put entries like this into a new file I called /etc/udev/rules.d/59-persistent-disk.rules. There are three rows of Three backplanes each so I did something like this. 

    # ROW 1 BACKPLANE 1
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0076306", SYMLINK+="pci24scsi00"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0073437", SYMLINK+="pci24scsi01"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0073085", SYMLINK+="pci24scsi02"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0072374", SYMLINK+="pci24scsi03"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0071564", SYMLINK+="pci24scsi04" 

# ROW 1 BACKPLANE 2
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0077571", SYMLINK+="pci24scsi20"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0077346", SYMLINK+="pci24scsi21"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0074773", SYMLINK+="pci24scsi22"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0071238", SYMLINK+="pci24scsi23"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0076264", SYMLINK+="pci24scsi24"
.......etc

    Creating the Arrays

    Now that I have persistent device names I created five 9 drive arrays (RAID5 with one spare) as follows. I used 1 drive per backplane in each array so I could lose an entire backplane (or even two) and the arrays would stay running. Performance may suffer for doing it this way, but I bought this for archiving and offsite backups so performance isn't the primary concern.

    mdadm --create --verbose /dev/md0 --level=5 --raid-devices=8 --spare-devices=1 /dev/pci24scsi00 /dev/pci24scsi20 /dev/pci24scsi30 /dev/pci44scsi00 /dev/pci44scsi20 /dev/pci44scsi30 /dev/pci64scsi00 /dev/pci64scsi20 /dev/pci64scsi30
mdadm --create --verbose /dev/md1 --level=5 --raid-devices=8 --spare-devices=1 /dev/pci24scsi01 /dev/pci24scsi21 /dev/pci24scsi31 /dev/pci44scsi01 /dev/pci44scsi21 /dev/pci44scsi31 /dev/pci64scsi01 /dev/pci64scsi21 /dev/pci64scsi31
mdadm --create --verbose /dev/md2 --level=5 --raid-devices=8 --spare-devices=1 /dev/pci24scsi02 /dev/pci24scsi22 /dev/pci24scsi32 /dev/pci44scsi02 /dev/pci44scsi22 /dev/pci44scsi32 /dev/pci64scsi02 /dev/pci64scsi22 /dev/pci64scsi32
mdadm --create --verbose /dev/md3 --level=5 --raid-devices=8 --spare-devices=1 /dev/pci24scsi03 /dev/pci24scsi23 /dev/pci24scsi33 /dev/pci44scsi03 /dev/pci44scsi23 /dev/pci44scsi33 /dev/pci64scsi03 /dev/pci64scsi23 /dev/pci64scsi33
mdadm --create --verbose /dev/md4 --level=5 --raid-devices=8 --spare-devices=1 /dev/pci24scsi04 /dev/pci24scsi24 /dev/pci24scsi34 /dev/pci44scsi04 /dev/pci44scsi24 /dev/pci44scsi34 /dev/pci64scsi04 /dev/pci64scsi24 /dev/pci64scsi34

    Using LVM to Create VGs and LVs

    These were my LVM steps to create the volumes so they could be iSCSI mounted remotely.

    PVCREATE

    pvcreate /dev/md0
pvcreate /dev/md1
pvcreate /dev/md2
pvcreate /dev/md3
pvcreate /dev/md4

    VGCREATE

    vgcreate VgArr0 /dev/md0
vgcreate VgArr1 /dev/md1
vgcreate VgArr2 /dev/md2
vgcreate VgArr3 /dev/md3
vgcreate VgArr4 /dev/md4
vgcreate VgArr5 /dev/md5

    LVCREATE for iSCSI

    Create the LVs but do not format or mount the volumes. They get formatted/mounted remotely

    lvcreate -L100%FREE VgArr0 -n LvISCSI0

    iSCSI

    Edit the /etc/iet/ietd.conf and /etc/iet/initiators.conf

    Map the iSCSI LUNs remotely on the other host

    • 4