Questions[dracut](server)

Okay, so I have a problem with Fedora 34. More specifically, I have a problem with a custom built image that is part of the services that we deliver.

The Problem

The problem has to do with initramfs. See, we've been using initrd up until now, but the amount of packages and binaries have increased since our last upgrade from Fedora 32 to Fedora 34. Without manually adding new packages, the base, release and dependency packages have now increased to a point where initrd has become over 500MB in size.

The size of this initrd makes this image very unstable: legacy PXE doesn't work for instance, because this image is too big for TFTP and will freeze on most networks. However, the biggest, most crucial issue we have, is that GRUB2 is not allowing this initrd to boot anymore, mainly on HP laptops. Whenever we go back to Fedora 32, the images seem to be more stable. We think it's because of initrd being smaller here. The default Fedora Workstation (34) images also work on systems where our custom image is getting kernel panics.

Our image uses a /vmlinuz and /initrd file to boot into a RAMDisk using either USB (GRUB2) or PXE (iPXE). This is by design: Our application is a Chromium-based, integrated kiosk web browser, running on Fedora (34). It is supposed to be volatile and stateless.

Initramfs

I've been looking into initramfs as a replacement to initrd, since it's much newer, much lighter, and the size of this file may not trigger any bugs in GRUB2. Perhaps this will also enable us the use of legacy PXE implementations like PXELINUX and Windows Deployment Services. I figured I could easily replace the use of CPIO + XZ with Dracut. But I've been stuck on an issue I can't really wrap my head around...

The Paradox

Replacing initrd with initramfs in GRUB2 isn't that hard. If I use the same name for both files (not at the same time), it would look for the file name in the config and find the file. Easy, right? But then, the kernel messages pop up, and it fails to boot reading something like: Failed to switch root: Specified switch root path /sysroot does not seem to be an OS tree. os-release file is missing.

Needless to say, I checked out the contents of /sysroot: empty. There should be an os-release file in /etc. Reading up on "Sysroot", it's supposed to mount the OS directory structure, which is probably why it's telling me that it's missing some files.

Sysroot

Here's where I got lost: The directory structure is inside initramfs, right? But in order to run initramfs through GRUB2, I need to mount the directory structure to Sysroot. That means I can only reach /sysroot from inside initramfs. But I can't mount initramfs, because I need the OS-release file in /sysroot. I also can't mount the directories inside initramfs, because it's an image file. Can you see my paradox? How am I supposed to mount to /sysroot, when this requires something inside initramfs itself? I've already tried the root= kernel parameter with a PARTUUID or LABEL, but I can't use UUIDs since it's vFAT over UEFI.

Did I miss something here? Am I even supposed to use initramfs, when using bootable USB RAMDisks? If not, should I look into splitting up initrd into multiple files? How do I even do this?

NOTE: We are already using the highest possible compression (XZ level 9). I've tried splitting initrd before, but I have not been able to do it properly.

I personally don't like asking for help, but when all the Google search results are in purple, I may need a point in the right direction. A lot of systems are not working after our upgrade to F:34 and I would really like to get this fixed. Does anybody know how to use Dracut properly? Anyone with experience in generating initramfs images? Why is my /sysroot not already mounted?

Help is much appreciated, thank you.

I use the boot options biosdevname=1 net.ifnames=1 in order to get consistent, predictable device names. I'm starting to notice a problem where in some cases, the network device names are not consistent. For example, if I drop to a dracut debug shell and look at the output of rdsosreport.txt, I see this:

+ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: p3p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b4:56:50:97:08 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b4:56:50:97:09 brd ff:ff:ff:ff:ff:ff

Notice that there is a mix of consistent (p3p1) and legacy style (eth1) naming. However, if I look at the the interfaces from the dracut debug shell, I see this:

initqueue:/run/initramfs# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: p3p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b4:56:50:97:08 brd ff:ff:ff:ff:ff:ff
3: p3p2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b4:56:50:97:09 brd ff:ff:ff:ff:ff:ff

p3p1/p3p2 are the correct expected names. For some reason, early in the initrd sequence, they are coming up in the mixed format. My assumption is that there is some sort of race going on here and given a bit more time, it (udev?) settles into the correct state, but I'm not sure exactly where it is. Unfortunately, this is causing problems for some of our automated server builds, because servers are coming up after (postinstall) first-boot and trying to bring up eth1 when the real interface name is p3p2.

I've been digging through the dracut modules to try and figure out where the problem may lie, but haven't been able to determine it conclusively yet, so looking for suggestions.

Also, this behavior doesn't happen all the time. The same server, booting the same image sometimes works fine, and other times gets this mixed naming behavior. Which also sort of tells me this is some kind of race - sometimes the race is won, and sometimes it is lost.

I never used dracut so far. I understood dracut filesystems (Wiki) as temporary systems that help mounting system specific kernel modules during the boot process. But after successful boot of my SLES 12 Server VM I still have a dracut file system under /var/tmp/dracut.2jqPCx.

Do I still need it? Where do I find information about how to get rid of this extra file system? Or, if it is essential, were do I find information about the reasoning for keeping it after the boot process?

Sequence of events:

1. Start install CentOs 7.2 using Everything ISO in Vmware vSphere 5.5
2. Choose Server with Gui, manual partitioning, encrypted sdd contains /home, /data
3. Initial boot is great, fast, works; requests partition password, login with password
4. Start terminal, run yum update and then yum upgrade -y (I know wrong order)
5. Do other stuff, eventually it HANGS in VMRC console, and I reboot gracefully with guest tools shutdown
6. On restart, GRUB2 has two kernels to choose, either one has same result
7. In either case, hangs before requesting partition password, eventually times out during plymouth and I get dracut shell

It warns me that there is no swap partition detected at /dev/mapper/swap. Note that I put swap to cover the entire sdc on its own, so it takes most of the space.

I am so lost, so any help is appreciated.

after doing a kernel-update (only via yum update) my centOS 7 (64bit) does no longer boot.

I now booted into rescue-mode and tried to mount /dev/md2 - this worked fine:

Any ideas what I could do here?