After an apt updgrade, I ran into some boot issues involving systemd dependency cycles (like in zfs-linux#1875577). So, since I’m running 20.10 on ZFS (with encryption), I decided to revert the system state using the GRUB ZFS history entries. However, this apparently caused me to run into this bug with the rpool/ROOT/ubuntu_ dataset (see this question), and so I still can’t boot my system. Moreover, now mounting /var/lib/NetworkManager and /var/lib/apt was failing during the boot process.
I tried to see if I can investigate by mounting the ZFS file system from a live USB session:
zpool export -a
zpool import -N -R /mnt rpool
zpool import -N -R /mnt bpool
cryptsetup open --type luks /dev/zvol/rpool/keystore keystore
mkdir -p /run/keystore/rpool
mount /dev/mapper/keystore /run/keystore/rpool
zfs load-key -a
zfs mount rpool/ROOT/ubuntu_<UUID>
zfs mount bpool/BOOT/ubuntu_<UUID>
zfs mount -a
The result of zfs mount -a showed similar errors as while booting:
cannot mount '/mnt/var/lib/NetworkManager': failed to create mountpoint
cannot mount '/mnt/var/lib/apt': failed to create mountpoint
Trying to see what’s going on, I get these strange question marks in ls for /mnt/var/lib/NetworkManager and /mnt/var/lib/apt, but also access errors for these paths.
$ ls -ahl /mnt/var/lib/
ls: cannot access '/mnt/var/lib/NetworkManager': No such file or directory
ls: cannot access '/mnt/var/lib/apt': No such file or directory
total 718K
drwxr-xr-x 80 root root 80 Nov 18 00:53 .
drwxr-xr-x 16 root root 18 Oct 22 14:17 ..
drwxr-xr-x 4 root root 4 Oct 22 14:01 AccountsService
drwxr-xr-x 2 root root 2 Jan 31 2019 acpi-support
drwxr-xr-x 2 root root 3 Dec 1 13:38 alsa
drwxr-xr-x 5 root root 5 Nov 18 00:54 apache2
drwxr-xr-x 4 root root 4 Dec 1 11:25 app-info
d????????? ? ? ? ? ? apt
drwxr-xr-x 2 root root 4 Nov 19 21:58 aptitude
drwxr-xr-x 2 root root 27 Oct 22 14:12 aspell
drwxr-xr-x 2 avahi-autoipd avahi-autoipd 2 Oct 22 14:09 avahi-autoipd
drwxr-xr-x 2 root root 5 Nov 15 23:06 binfmts
drwxr-xr-x 3 root root 3 Nov 15 13:40 bluetooth
drwxr-xr-x 2 root root 2 Jun 16 14:01 boltd
drwxrwxrwt 2 root root 2 Jul 27 15:38 BrlAPI
drwxr-xr-x 4 colord colord 6 Nov 15 15:01 colord
drwxr-xr-x 2 root root 4 Dec 1 11:25 command-not-found
drwxr-xr-x 2 root root 3 Nov 15 15:01 dbus
drwxr-xr-x 2 root root 2 Aug 12 09:33 dhcp
drwxr-xr-x 5 root root 5 Oct 22 14:08 dictionaries-common
drwxr-xr-x 3 root root 4 Dec 1 12:05 dkms
drwxr-xr-x 7 root root 18 Dec 1 12:06 dpkg
drwxr-xr-x 3 root root 3 Oct 22 14:02 emacsen-common
drwxr-xr-x 2 root root 2 Apr 16 2020 fprint
drwxr-xr-x 6 root root 7 Nov 15 23:05 fwupd
drwxr-xr-x 5 gdm gdm 6 Dec 1 13:30 gdm3
drwxr-xr-x 3 root root 3 Nov 15 22:59 gems
drwxr-xr-x 2 geoclue geoclue 2 Feb 28 2020 geoclue
drwxr-xr-x 4 root root 4 Oct 22 14:03 ghostscript
drwxr-xr-x 2 root root 2 Jun 9 20:50 git
drwxr-xr-x 4 root root 4 Oct 22 14:04 grub
drwxr-xr-x 2 root root 2 Jun 22 10:09 hp
drwxr-xr-x 2 root root 6 Nov 25 11:32 initramfs-tools
drwxr-xr-x 2 root root 3 Oct 22 14:02 ispell
drwxr-xr-x 3 root root 3 Oct 22 14:05 libreoffice
drwxr-xr-x 3 root root 3 Nov 15 22:59 libxml-sax-perl
drwxr-xr-x 3 root root 3 Oct 22 14:12 locales
drwxr-xr-x 2 root root 3 Nov 30 23:00 logrotate
drwxr-xr-x 2 root root 3 Oct 22 14:10 man-db
drwxr-xr-x 2 root root 2 Oct 16 15:40 misc
d????????? ? ? ? ? ? NetworkManager
drwxr-xr-x 3 root root 3 Oct 22 14:10 openvpn
drwxr-xr-x 2 root root 2 Dec 1 12:55 os-prober
drwxr-xr-x 2 root root 3 Dec 1 13:31 PackageKit
drwxr-xr-x 2 root root 8 Nov 19 21:57 pam
drwxr-xr-x 4 root root 4 Nov 15 23:35 php
drwxr-xr-x 2 root root 3 Nov 15 15:01 plymouth
drwx------ 3 root root 3 Oct 22 13:59 polkit-1
drwx------ 2 root root 2 Oct 22 13:59 private
drwxr-xr-x 2 root root 3 Oct 22 13:59 python
drwxr-xr-x 2 root root 2 Sep 8 18:15 realmd
drwxr-xr-x 2 root root 4 Nov 19 21:57 sgml-base
drwxr-xr-x 3 root root 5 Nov 15 15:18 shim-signed
drwxr-xr-x 24 root root 26 Dec 1 13:30 snapd
drwxr-xr-x 2 root root 2 Aug 6 15:42 snmp
drwxr-xr-x 10 root root 10 Oct 22 14:09 sss
drwxr-xr-x 3 root root 3 Oct 22 13:59 sudo
drwxr-xr-x 2 root root 2 Feb 28 2020 synaptic
drwxr-xr-x 12 root root 13 Nov 15 15:01 systemd
drwxr-xr-x 5 root root 5 Nov 15 22:59 tex-common
drwxr-xr-x 9 root root 16 Nov 18 00:54 texmf
drwx--S--- 3 127 135 9 Dec 1 13:38 tor
drwxr-xr-x 2 tss tss 2 Dec 3 2019 tpm
drwxr-xr-x 2 root root 3 Nov 15 13:46 ubiquity
drwxr-xr-x 2 root root 2 Sep 1 17:14 ubuntu-advantage
drwxr-xr-x 2 root root 3 Nov 15 15:01 ubuntu-drivers-common
drwxr-xr-x 2 root root 2 Nov 28 13:17 ubuntu-release-upgrader
drwxr-xr-x 3 root root 21 Nov 18 01:28 ucf
drwx------ 2 root root 3 Dec 1 13:30 udisks2
drwxr-xr-x 2 root root 2 Aug 5 12:03 unattended-upgrades
drwxr-xr-x 2 root root 2 Jul 23 19:50 update-manager
drwxr-xr-x 4 root root 6 Dec 1 12:06 update-notifier
drwxr-xr-x 2 root root 6 Dec 1 13:05 upower
drwxr-xr-x 2 root root 2 Jul 27 16:30 usb_modeswitch
drwxr-xr-x 2 root root 3 Oct 22 14:02 usbutils
drwxr-xr-x 3 root root 3 Oct 22 13:59 vim
drwxr-xr-x 2 root root 3 Dec 1 13:30 whoopsie
drwxr-xr-x 2 root root 3 Oct 22 14:10 xfonts
drwxr-xr-x 2 root root 3 Oct 22 14:04 xkb
drwxr-xr-x 2 root root 6 Oct 22 14:11 xml-core
Trying to access or delete these paths claims that no such files exist, while trying to mkdir claims that the files do already exist! (This is why the mount point cannot be created.)
$ ls /mnt/var/lib/NetworkManager
ls: cannot access '/mnt/var/lib/NetworkManager': No such file or directory
$ rm /mnt/var/lib/NetworkManager
rm: cannot remove '/mnt/var/lib/NetworkManager': No such file or directory
$ rmdir /mnt/var/lib/NetworkManager
rmdir: failed to remove '/mnt/var/lib/NetworkManager': No such file or directory
$ mkdir /mnt/var/lib/NetworkManager
mkdir: cannot create directory ‘/mnt/var/lib/NetworkManager’: File exists
What happened here? How can I get rid of these path entries?
0 Answers