I have a server which runs Proxmox VE 8.2 (based on Debian 12), fully updated, which uses multipath SAN. There we have a space allocated, ten devices of 2T each, which are seen 16 times each (the number of paths) and which are mapped into 10 multipathed virtual devices and collected into a single LVM group, where a logical volume is carved. No problems on this part.
It was initialized "on the fly" and running all right, until the host was rebooted. After that I noticed that whenever one runs any LVM-related command (lvs
, etc.) in the system shell, it emits the warning in the form:
WARNING: Device mismatch detected for <LV> which is accessing <list of devices like
/dev/sdak, /dev/sdaz, /dev/sdbn, ...> instead of <list of the devices
like /dev/mapper/oamdwhdg-01, /dev/mapper/oamdwhdg-02, ...>
These /dev/mapper/oamdwhdg-XX
are in reality the multipath devices, they were named that way for operational reasons in /etc/multipath.conf
:
multipaths {
...
multipath {
wwid 36...
alias oamdwhdg-04
}
...
}
I.e. for whatever reason, the LV is mapped using backend devices, instead of using multipath virtual devices.
So I updated the filter in /etc/lvm/lvm.conf
, which now looks like this:
global_filter=[ "a|/dev/sda3|", "a|/dev/mapper/oamdwhdg|", "r|.*|" ]
(I updated it, not added, it was global_filter=["r|/dev/zd.*|","r|/dev/rbd.*|"]
before with the comment added by pve-manager to avoid scanning ZFS zvols and Ceph rbds
. I believe my filter is much more strict than this one.)
/dev/sda3
is the local disk where PVE is installed and it contains the VG pve
which doesn't depend on SAN, so it is the only /dev/sd...
which is not using multipath and it was whitelisted.
I tested this filter using vgscan
and it shows both groups to be found.
Then I executed update-initramfs
and rebooted. During boot, server failed into emergency shell. When I hit Ctrl+D
there, it, however, booted almost normally: the multipathed VG is seen but not activated (as if vgchange -an oamdwhdg
was not yet run). But then I activated it by hand perfectly normally with the said command and it worked flawlessly.
I suspect this is because multipath isn't correctly initialized before LVM in the initramfs, so it tried to set up mappings using /dev/sdXX devices. But why it failed to emergency shell when I added the filter I don't understand.
Two (very related) questions here: why emergency shell a.k.a. what is wrong and how to make it work as expected?
It turned out to be very silly problem. I need no custom filters. I needed to have the proper multipath inclusion into initramfs, which
multipath-tools
alone doesn't do.For that, there's a separate package
multipath-tools-boot
in Debian-based systems. It made dm-multipath modules and configuration from /etc to appear inside initramfs.I removed all the custom changes to
lvm.conf
and run:Reboot, and voila, everything came up in the right state.
Important: after any changes to the multipath configuration or adding devices or whatever that changes
/etc/multipath/wwids
or/etc/multipath.conf
, one needs to rebuild the initramfs to include updated versions: