I found a question in the Ubuntu_20.04_LTS_STIG 1.5.2 that reads,
The Ubuntu operating system must automatically remove or disable emergency accounts after 72 hours...
What is an emergency account? Is it different from booting in rescue mode?
After laptop (Lenovo T520) fell on the ground, my HDD (Crucial_CT240M500SSD3) started making clicking sounds during boot and at startup started receiving following message:
2100: Detection error on HDD0 (Main HDD)
So I presume my disk has been damaged and cannot reach login screen in Ubuntu as it always boots into Emergency mode.
You are in emergency mode.
After logging in, type "journalctl -xb" to view system logs,
"systemctl reboot" to reboot, "systemctl default"
or ^D to try again to boot into default mode".
My main goal is to retreive personal data and save it to an external disk.
The problem is that my /home directory is empty.
I have found many methods which point toward resolving the issue, but nothing really worked and I don't know what else to try.
In my /etc/fstab I have separate entries for / and /home, but I haven't created a separate partition for home during Ubuntu installation (at least not manually):
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sdb1 during installation
UUID=99d3a843-b520-41c0-bc80-e64fb67439c8 / ext4 errors=remount-ro 0 1
# /home was on /dev/sda1 during installation
UUID=f0b370b3-90d8-46ca-956f-f620b5c40536 /home ext4 defaults 0 2
# swap was on /dev/sdb5 during installation
UUID=0e4c9122-3317-4478-bc6b-852415505118 none swap sw 0 0
journalctl -xb snippet where the mountin of /home UUID fails
...
dec 10 12:11:18 jp-pc kernel: input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input17
dec 10 12:11:18 jp-pc systemd[1]: Received SIGRTMIN+20 from PID 384 (plymouthd).
dec 10 12:11:27 jp-pc kernel: thinkpad_acpi: EC reports that Thermal Table has changed
dec 10 12:12:45 jp-pc systemd[1]: dev-disk-by\x2duuid-f0b370b3\x2d90d8\x2d46ca\x2d956f\x2df620b5c40536.device: Job dev-disk-by\x2duuid-f0b370b3\x2d90d8\x2d46ca\x2d956f\x2df620b5c40536.device/start timed out.
dec 10 12:12:45 jp-pc systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-f0b370b3\x2d90d8\x2d46ca\x2d956f\x2df620b5c40536.device.
-- Subject: Unit dev-disk-by\x2duuid-f0b370b3\x2d90d8\x2d46ca\x2d956f\x2df620b5c40536.device has failed
-- Defined-By: systemd
-- ...
--
-- Unit dev-disk-by\x2duuid-f0b370b3\x2d90d8\x2d46ca\x2d956f\x2df620b5c40536.device has failed.
--
-- The result is RESULT.
dec 10 12:12:45 jp-pc systemd[1]: Dependency failed for /home.
-- Subject: Unit home.mount has failed
-- Defined-By: systemd
-- ...
--
-- Unit home.mount has failed.
--
-- The result is RESULT.
dec 10 12:12:45 jp-pc systemd[1]: Dependency failed for Local File Systems.
-- Subject: Unit local-fs.target has failed
-- Defined-By: systemd
-- ..
--
-- Unit local-fs.target has failed.
--
-- The result is RESULT.
dec 10 12:12:45 jp-pc systemd[1]: Dependency failed for Clean up any mess left by 0dns-up.
-- Subject: Unit dns-clean.service has failed
-- Defined-By: systemd
-- ...
--
-- Unit dns-clean.service has failed.
--
-- The result is RESULT.
dec 10 12:12:45 jp-pc systemd[1]: dns-clean.service: Job dns-clean.service/start failed with result 'dependency'.
dec 10 12:12:45 jp-pc systemd[1]: local-fs.target: Job local-fs.target/start failed with result 'dependency'.
dec 10 12:12:45 jp-pc systemd[1]: local-fs.target: Triggering OnFailure= dependencies.
dec 10 12:12:45 jp-pc systemd[1]: home.mount: Job home.mount/start failed with result 'dependency'.
dec 10 12:12:45 jp-pc systemd[1]: Dependency failed for File System Check on /dev/disk/by-uuid/f0b370b3-90d8-46ca-956f-f620b5c40536.
-- Subject: Unit systemd-fsck@dev-disk-by\x2duuid-f0b370b3\x2d90d8\x2d46ca\x2d956f\x2df620b5c40536.service has failed
-- Defined-By: systemd
-- ...
--
-- Unit systemd-fsck@dev-disk-by\x2duuid-f0b370b3\x2d90d8\x2d46ca\x2d956f\x2df620b5c40536.service has failed.
--
-- The result is RESULT.
dec 10 12:12:45 jp-pc systemd[1]: systemd-fsck@dev-disk-by\x2duuid-f0b370b3\x2d90d8\x2d46ca\x2d956f\x2df620b5c40536.service: Job systemd-fsck@dev-disk-by\x2duuid-f0b370b3\x2d90d8\x2d46ca\x2d956f\x2df620b5c40536.service/start failed with result 'dependency'.
dec 10 12:12:45 jp-pc systemd[1]: dev-disk-by\x2duuid-f0b370b3\x2d90d8\x2d46ca\x2d956f\x2df620b5c40536.device: Job dev-disk-by\x2duuid-f0b370b3\x2d90d8\x2d46ca\x2d956f\x2df620b5c40536.device/start failed with result 'timeout'.
...
fdisk says that partition does not start on physical sector boundary (I noticed that sda2 and sda5 reside on the same interval):
$ fdisk -l
Disk /dev/sda: 223.57 GiB, 240057409536 bytes, 468862128 sectors
Disk model: Crucial_CT240M50
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x021adb68
Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 429799423 429797376 204.9G 83 Linux
/dev/sda2 429801470 468860927 39059458 18.6G 5 Extended
/dev/sda5 429801472 468860927 39059456 18.6G 82 Linux swap / Solaris
Partition 2 does not start on physical sector boundary.
blkid (removed loop entries for readability):
$ blkid
/dev/sda1: UUID="99d3a843-b520-41c0-bc80-e64fb67439c8" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="021adb68-01"
/dev/sda5: UUID="0e4c9122-3317-4478-bc6b-852415505118" TYPE="swap" PARTUUID="021adb68-05"
/dev/sda2 is missing, but when I explicitly call:
$ blkid /dev/sda2
/dev/sda2: PTUUID:"d5620a3d" PTTYPE="dos" PARTUUID="021adb68-02"
lsblk in LiveOS (removed loop entries for readability)
$ lsblk -o NAME,RM,SIZE,TYPE,MOUNTPOINT,UUID
NAME RM SIZE RO TYPE MOUNTPOINT UUID
sda 0 223.6G 0 disk
├─sda1 0 204.9G 0 part /home/xx/mnt 99d3a843-b520-41c0-bc80-e64fb67439c8
├─sda2 0 1K 0 part
└─sda5 0 18.6G 0 part 0e4c9122-3317-4478-bc6b-852415505118
sr0 1 1024M 0 rom
There is no entry for the UUID of /home (taken from /etc/fstab) in /dev/disks/by-uuid/. Only for sda1 and sda5.
Please let me know if any additional info is required.
What I've tried already:
- In emergency mode:
$ mount -o rw,remount /
$ mount -a
mount: /home: can't find UUID=f0b370b3-90d8-46ca-956f-f620b5c40536.
In LiveOS I ran
fsckon/dev/sda1which found a lot of bad sectors, but now it returns clean.
$ fsck /dev/sda2
fsck.ext2: Attempt to read block from filesystem resulted in short read while trying to open /dev/sda2
Could this be a zero-length partition?
Same error with fsck.ext3 and fsck.ext4
- Commenting out
/homepart in/etc/fstabbrings me to Ubuntu login screen, but after entering the password it freezes. I presume this is due/homenot mounted properly by the system
I thought locked SSH access to my IP would be a good thing but now I'm completely locked out and nothing I do in rescue mode seems to work.
I've tried changing/removing/uninstalling iptables but nothing I do in rescue mode seems to save and as soon as I boot the machine back up from the HDD all the settings have returned :/
I'm planning a system upgrade from Eoan to Focal on a remote machine where I only have SSH access. I've previously dealt with broken packages causing the system to boot into rescue mode, so I want to ensure whatever happens I can keep my access to the server. To further secure the process, I'm doing a rehearsal using a local virtual machine running the target system.
With my knowledge in Ubuntu, I did the following tasks beforehand:
systemctl edit ssh.serviceand insertDefaultDependencies=noto [Unit] section andWantedBy=rescue.targetto [Install] section, thensystemctl enable ssh.service. Systemd reports "created symbolic link".systemctl edit rescue-ssh.targetand insertWantedBy=rescue.targetto [Install] section, thensystemctl enable rescue-ssh.target.
Then I edited grub to enable boot menu, update-grub then systemctl reboot, and use e to edit the default entry. I scrolled down to the first linux line and appended rescue and booted this entry with Ctrl-X, which led to this screen:
Everything seems fine here as I can SSH into this VM right at this point.
My question begins:
I tried again in a different way to test SSH availability in rescue mode. First I booted the system normally into graphical mode (with GNOME and whatever), started a terminal and ran systemctl isolate rescue.target. I see the same prompt text again, but when trying to SSH into the VM, I got the following result:
ubuntu@WSL:~ $ ssh root@vmwares
Warning: Permanently added '192.168.2.5' (ECDSA) to the list of known hosts.
Timeout, server 192.168.2.5 not responding.
255|ubuntu@WSL:~ $
Running systemctl status ssh.service highlighted the following text:
pam_systemd(sshd:session): Failed to create session: Failed to activate service 'org.freedesktop.login1': timed out (service_start_timeout=25000ms)
Restarting systemd-logind.service makes no difference. Going back to graphical interface with systemctl isolate default.target brings everything back to normal again.
Question: Why can't I login to SSH when the rescue target is isolated from graphical.target? How to I resolve this?
I think there's something to do with "isolating targets", but apparently things have gone beyond my knowledge.
How do I edit the GRUB boot sequence from:
linux /boot/vmlinuz... quiet splash
so that it boots to the console? I've tried removing "quiet" and "splash", as well as adding "text". I can also boot to rescue mode, or edit the boot sequence.
Else booting hangs at the splash screen.