We are having a weird problem when attaching an EBS volume to a running Linux (NixOS in this case) instance (for the purpose of growing the file system on that attached volume; on it is the NixOS root file system for another machine that we shut down).
Before the attach ing, all is normal:
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 100G 0 disk
└─xvda1 202:1 0 100G 0 part
After the attaching, lsblk
oddly claims that the attached volume's partition contains the mounted /
partition of the current machine:
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 100G 0 disk
└─xvda1 202:1 0 100G 0 part /nix/store
xvdf 202:80 0 400G 0 disk
└─xvdf1 202:81 0 200G 0 part /
This makes no sense at all:
Just "plugging in" that disk makes Linux think that the root file system mount just "flipped over" to the new disk. The /nix/store
(which is a NixOS read-only bind mount) remains on the proper disk somehow.
There are no messages in dmesg
/journalctl
beyond Linux noticing that a disk was attached:
Apr 28 11:57:21 mymachine kernel: blkfront: xvdf: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled;
Apr 28 11:57:21 mymachine kernel: xvdf: xvdf1
In fdisk -l
, the two disks look normal, and have different Disk identifier
s.
It is impossible to umount /dev/xvdf1
; it says the mount is busy.
For the goal of growing the partition, growpart /dev/xvdf 1
works anyway, but resize2fs /dev/xvdf1
fails with:
Filesystem at /dev/xvdg1 is mounted on /; on-line resizing required
old_desc_blocks = 25, new_desc_blocks = 50
resize2fs: No space left on device While checking for on-line resizing support
What's going on here, why does Linux confuse these disks?
The reason is
by-label
mounts.For reasons of declarative automation (we have many machines), each of our machine's root file systems has the same ext4 file system label
nixos
:Attaching two of those to the same machine confuses Linux.
So the solutions are:
by-label
with the same label.