How to figure out how to fix hibernation?

Backstory

I followed this answer about how to enable hibernation two months ago and it worked. As documented in my question on How to resume Ubuntu 18.04 after hibernation it worked the first time, but not after a complete reinstall.

xenoid kindly pointed out a generally helpful article on the ubuntu wiki.

Dead, blank, or black screen on resume In some cases, a machine can hibernate just fine, and resume without issue, with the exception of waking up to a blacked-out screen. In other words, the computer is running just fine, but the display appears dead.

They recommend adding nomodeset to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub.

Hibernation worked. Sometimes the laptop woke itself up, but lenovo has addressed an issue like that in the latest bios and fimware updates now.

Then I realized that I had sound problems. I first found one fix that broke my brightness keys in turn and then undid it when I found out that sound and brightness keys worked flawlessly on a clean install. nomodeset was the culprit.

So now I am back to square one regarding hibernation. Or, you would think so. But actually, I'm even worse off than before: In How to resume Ubuntu after hibernation it seemed like the hibernate part worked and the resume part did not.

Problem Statement

When I run sudo systemctl hibernate or sudo pm-hibernate the laptop shuts down. Closing and Reopening the lid does start it again, but like a reboot it brings me to the grub menu instead of restoring my session as it was.

Testing

kernel.org states that one can debug hibernation issues using this snippet

# echo devices > /sys/power/pm_test
# echo platform > /sys/power/disk
# echo disk > /sys/power/state

with one of those options

freezer
test the freezing of processes
devices
test the freezing of processes and suspending of devices
platform
test the freezing of processes, suspending of devices and platform global control methods [1]
processors
test the freezing of processes, suspending of devices, platform global control methods [1] and the disabling of nonboot CPUs
core
test the freezing of processes, suspending of devices, platform global control methods[1], the disabling of nonboot CPUs and suspending of platform/system devices

[1] (1, 2, 3) the platform global control methods are only available on ACPI systems and are only tested if the hibernation mode is set to “platform”

They state that this is only available on kernels compiled with CONFIG_PM_DEBUG but I assume this is the case for me since the file /sys/power/pm_test exists.

root@motorbrot:~# echo freezer > /sys/power/pm_test
root@motorbrot:~# echo disk > /sys/power/state
## results in a short freeze of the laptop

root@motorbrot:~# echo devices > /sys/power/pm_test
root@motorbrot:~# echo disk > /sys/power/state
## results in a black screen for a few seconds. Power Button LED stays on. Then screen turns back on.

root@motorbrot:~# echo platform > /sys/power/pm_test
root@motorbrot:~# echo disk > /sys/power/state
## takes longer than before until the screen turns dark, otherwise the same behaviour as with the "devices" test.

root@motorbrot:~# echo processors > /sys/power/pm_test
root@motorbrot:~# echo disk > /sys/power/state
## nothing noticeably different to the previous two tests

root@motorbrot:~# echo core > /sys/power/pm_test
root@motorbrot:~# echo disk > /sys/power/state
## Again nothing new. The screen turns dark, takes a moment, turns on again.

I interpret those results as "all tests succeed".

However, there is a log file in /var/crash/susres.2020-06-08_17:46:27.952596.crash. Either from when I last actually ran hibernate, or from the "devices" test. It starts with

ProblemType: KernelOops
Annotation: This occurred during a previous hibernation, and prevented the system from resuming properly.
Architecture: amd64
Date: Mon Jun  8 17:46:27 2020
DistroRelease: Ubuntu 18.04
ExecutablePath: /usr/share/apport/apportcheckresume
ExecutableTimestamp: 1589407934
Failure: hibernate/resume
InterpreterPath: /usr/bin/python3.6
Package: linux-image-5.3.0-53-generic 5.3.0-53.47~18.04.1

LED

Looks fine. One last question, now that it works, what does the Power button LED indicates when you hibernate? Completely off? dimmed? "Breathing"? – xenoid Mar 19 at 20:07

@xenoid it blinks twice short, once long, three times short and then it remains off like everything else (apart from the "I'm charging!-led" – lucidbrot Mar 19 at 21:07

This quote form the last time I asked about the hibernation problem no longer applies. Currently, the power button LED stays on for a while after systemctl hibernate, blinks twice, then is on for a while, and then off like after a shutdown.

Does systemctl suspend work?

When I do systemctl suspend the power button LED goes into breathing mode and the screen turns dark. Touching the trackpad or clicking, or typing does not cause any reaction.
Closing and reopening the lid works: The display turns back on and presents the login screen.

System Info

When I started writing this question, I was on

root@motorbrot:~# uname -a
Linux motorbrot 5.3.0-53-generic #47~18.04.1-Ubuntu SMP Thu May 7 13:10:50 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

root@motorbrot:~# swapon --show
NAME           TYPE       SIZE USED PRIO
/dev/nvme0n1p6 partition 18.6G   0B   -2

I have since upgraded to the 5.3.0-59 kernel, but the behaviour remains the same.

/etc/fstab and friends

Perhaps that is helpful?

generic@motorbrot:~$ blkid
/dev/nvme0n1p1: LABEL="Recovery" UUID="885E1DE35E1DCAB8" TYPE="ntfs" PARTLABEL="Basic data partition" PARTUUID="2aac3226-cbc2-46d4-9bd4-5ce4695533fc"
/dev/nvme0n1p2: UUID="C21E-C17B" TYPE="vfat" PARTLABEL="EFI system partition" PARTUUID="beb37963-eae9-4a51-a29e-8850fd392a7f"
/dev/nvme0n1p5: UUID="81e29369-ff69-4424-858c-3489283588d7" TYPE="ext2" PARTUUID="f68aad7f-bfc7-404d-b64b-072ee19d56dd"
/dev/nvme0n1p6: UUID="1ba104f0-35be-42f7-bf71-65e43f6fbcc3" TYPE="swsuspend" PARTUUID="1a651275-5ad0-4ead-9bdf-66e5e9a28b85"
/dev/nvme0n1p7: UUID="d2a1f26d-4442-4eae-9a62-2c176268208d" TYPE="ext2" PARTUUID="d296b2e6-2f0e-4393-9b54-6ed0d11f22ac"
/dev/nvme0n1p8: UUID="003fdbc2-f3fe-4e5d-a945-c2568048589a" TYPE="ext4" PARTUUID="503fddfe-0898-4ed1-a43e-1866a29abd8f"
/dev/nvme0n1p9: LABEL="aquarium" UUID="13317063534776643493" UUID_SUB="8783866549508142042" TYPE="zfs_member" PARTUUID="9bdfdcf4-f586-4603-90fb-a51ed366de3e"
/dev/nvme0n1p10: LABEL="tank" UUID="2491194531467516534" UUID_SUB="922708726245396109" TYPE="zfs_member" PARTUUID="77970109-18ef-4ebd-b49d-6ae04c88aeaa"

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/nvme0n1p8 during installation
# but that was before I moved it over to zfs
#UUID=003fdbc2-f3fe-4e5d-a945-c2568048589a /               ext4    errors=remount-ro 0       1
# I haven't specified the zfs volume yet because I'm not sure I'd need it... in fact I don't need it, as proven by usage.
#
# /boot is its own partition distinct from /boot/efi, but for some reason it was not set up correctly, it seems. Trying to fix that now with the following line:
UUID=81e29369-ff69-4424-858c-3489283588d7  /boot           ext2    defaults        0       2
# /boot/efi was on /dev/nvme0n1p2 during installation
UUID=C21E-C17B  /boot/efi       vfat    umask=0077      0       1
# Use swap partition instead of swap file
#/swapfile                                 none            swap    sw              0       0
UUID=1ba104f0-35be-42f7-bf71-65e43f6fbcc3  none            swap    sw              0       0

I have a custom grub menuentry because generating it with update-grub fails for the encrypted zfs partition (that's a different problem which seems to be a bug with the update-grub script, not my current question).

#/etc/grub.d/40_custom
#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.


menuentry "Encryptioned Magic 18.04" --id encryptioned-magic-id {
setparams 'Ubuntu'

    recordfail
    load_video
    gfxmode $linux_gfx_mode
    insmod gzio
    if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
    insmod part_gpt
    insmod ext2
     # not needed. because grub has already set its root correctly to that partition(hd0,gpt5).
     # but also not harmful.
     # Find the /boot partition
     if [ x$feature_platform_search_hint = xy ]; then
       search --no-floppy --fs-uuid --set=root 81e29369-ff69-4424-858c-3489283588d7 
     else
       search --no-floppy --fs-uuid --set=root 81e29369-ff69-4424-858c-3489283588d7 
     fi
    # The relevant part
    # nomodeset was for (hopefully) fixing hibernation startup problems where screen stayed black, but I removed it again since it caused sound and brightness problems.
    # resume=SWAP-PARTITION-UUID-AS-PER-lsblk is for restoring from swap space after hibernation
    # acpi_backlight=video fixed my screen brightness keys, but no longer needed since I removed nomodeset
    linux /vmlinuz-5.3.0-59-generic root=ZFS=tank/ds1/u18 ro resume=1ba104f0-35be-42f7-bf71-65e43f6fbcc3
    initrd /initrd.img-5.3.0-59-generic
}

policy kit in local authority

As recommended in the answer about how to enable hibernation

#/etc/polkit-1/localauthority/50-local.d/com.ubuntu.enable-hibernate.pkla
[Re-enable hibernate by default in upower]
Identity=unix-user:*
Action=org.freedesktop.upower.hibernate
ResultActive=yes

[Re-enable hibernate by default in logind]
Identity=unix-user:*
Action=org.freedesktop.login1.hibernate;org.freedesktop.login1.handle-hibernate-key;org.freedesktop.login1;org.freedesktop.login1.hibernate-multiple-sessions;org.freedesktop.login1.hibernate-ignore-inhibit
ResultActive=yes

[Enable hibernate to be run via cron]
Identity=unix-user:*
Action=org.freedesktop.login1.hibernate;org.freedesktop.login1.hibernate-multiple-sessions
ResultAny=yes

Output from update-initramfs -u -k all

generic@motorbrot:~$ sudo update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-5.3.0-59-generic
I: The initramfs will attempt to resume from /dev/nvme0n1p6
I: (UUID=1ba104f0-35be-42f7-bf71-65e43f6fbcc3)
I: Set the RESUME variable to override this.
update-initramfs: Generating /boot/initrd.img-5.3.0-53-generic
I: The initramfs will attempt to resume from /dev/nvme0n1p6
I: (UUID=1ba104f0-35be-42f7-bf71-65e43f6fbcc3)
I: Set the RESUME variable to override this.
update-initramfs: Generating /boot/initrd.img-5.3.0-51-generic
I: The initramfs will attempt to resume from /dev/nvme0n1p6
I: (UUID=1ba104f0-35be-42f7-bf71-65e43f6fbcc3)
I: Set the RESUME variable to override this.

Output from update-grub

The device-mapper issue is (as mentioned earlier) likely due to me using an encrypted zfs root and I circumvent that issue by using a custom grub entry.

generic@motorbrot:~$ sudo update-grub
Sourcing file `/etc/default/grub'
Generating grub configuration file ...
Found theme: /boot/grub/themes/poly-dark/theme.txt
Found linux image: /boot/vmlinuz-5.3.0-59-generic
Found initrd image: /boot/initrd.img-5.3.0-59-generic
Found linux image: /boot/vmlinuz-5.3.0-53-generic
Found initrd image: /boot/initrd.img-5.3.0-53-generic
Found linux image: /boot/vmlinuz-5.3.0-51-generic
Found initrd image: /boot/initrd.img-5.3.0-51-generic
Found linux image: /boot/vmlinuz-4.15.0-101-generic
Found initrd image: /boot/initrd.img-4.15.0-101-generic
device-mapper: reload ioctl on osprober-linux-nvme0n1p10  failed: Device or resource busy
Command failed
Found Windows Boot Manager on /dev/nvme0n1p2@/EFI/Microsoft/Boot/bootmgfw.efi
Found Ubuntu 18.04.4 LTS (18.04) on /dev/nvme0n1p8
Adding boot menu entry for EFI firmware configuration
done

Question

How do I fix this or how do I go about finding out where the problem lies? The end goal is to be able to write the ram to the swap partition and resume right where I was after hibernation.

Edit (14. Jul - One month after posting this question)

It's working now:

• systemctl hibernate works correctly from a running system without nomodeset to a rebooted system without nomodeset.
• systemctl hibernate works correctly from a running system without nomodeset to a rebooted system with nomodeset.
• The sound and brightness keys work in both cases.

I hope this was fixed in a kernel update. My current system information is

$ uname -a
Linux motorbrot 5.3.0-62-generic #56~18.04.1-Ubuntu SMP Wed Jun 24 16:17:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

The Question remains How did it become fixed and how would one go about that if it were to happen again?