Home / server / Questions / 1061854
Accepted
nh2
Asked: 2021-04-29 04:33:56 +0800 CST

Linux confused by identical mounts on different disks after attaching disk

  • 772

We are having a weird problem when attaching an EBS volume to a running Linux (NixOS in this case) instance (for the purpose of growing the file system on that attached volume; on it is the NixOS root file system for another machine that we shut down).

Before the attach ing, all is normal:

# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  100G  0 disk
└─xvda1 202:1    0  100G  0 part

After the attaching, lsblk oddly claims that the attached volume's partition contains the mounted / partition of the current machine:

# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  100G  0 disk
└─xvda1 202:1    0  100G  0 part /nix/store
xvdf    202:80   0  400G  0 disk
└─xvdf1 202:81   0  200G  0 part /

This makes no sense at all:

Just "plugging in" that disk makes Linux think that the root file system mount just "flipped over" to the new disk. The /nix/store (which is a NixOS read-only bind mount) remains on the proper disk somehow.

There are no messages in dmesg/journalctl beyond Linux noticing that a disk was attached:

Apr 28 11:57:21 mymachine kernel: blkfront: xvdf: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled;
Apr 28 11:57:21 mymachine kernel:  xvdf: xvdf1

In fdisk -l, the two disks look normal, and have different Disk identifiers.

It is impossible to umount /dev/xvdf1; it says the mount is busy.

For the goal of growing the partition, growpart /dev/xvdf 1 works anyway, but resize2fs /dev/xvdf1 fails with:

Filesystem at /dev/xvdg1 is mounted on /; on-line resizing required
old_desc_blocks = 25, new_desc_blocks = 50
resize2fs: No space left on device While checking for on-line resizing support

What's going on here, why does Linux confuse these disks?

linux mount nixos

1 Answers

  1. Best Answer

    The reason is by-label mounts.

    For reasons of declarative automation (we have many machines), each of our machine's root file systems has the same ext4 file system label nixos:

    # cat /etc/fstab
/dev/disk/by-label/nixos / ext4 x-nixos.autoresize 0 1

    # cat /etc/mtab
/dev/disk/by-label/nixos /nix/store ext4 rw,relatime 0 0

    Attaching two of those to the same machine confuses Linux.

    So the solutions are:

    • Do the file system growing by attaching the EBS volume to an AWS machine whose root file system is not mounted by-label with the same label.
    • Change the file system label first, then do the change, then change it back to the desired one.
    • 0