Zero0ne's questions

My apologies for the incoming wall of text but I'm hoping to find a solution to a strange problem that "might" be a bug. I'm in the process of creating an offline (air-gapped) automated install of Ubuntu 24.04.1 LTS using a combination of PXE booting the ISO and pointing it to a user-data file that contains my autoinstall and user-data configuration for a fully automated bare-metal installation.

Anyone familiar with this process knows that its a multi-phase install process, with phase 1 being the base install processing anything in the "autoinstall:", formatting disks and copying files from the ISO to the disk, etc... and phase 2 basically processing anything defined in the "user-data:" section of your user-data file. Additionally, it is possible to configure packages to be installed during the phase 1 install phase (for example openssh-server). That said, as noted on the AutoInstall reference page (https://canonical-subiquity.readthedocs-hosted.com/en/latest/reference/autoinstall-reference.html#ssh):

Note that Desktop installation ISOs do not include openssh-server, so installations of Desktop require Ubuntu archive access for install-server to be successful.

So if I understand correctly, all this means is that in order to install openssh-server, or any package not contained on the ISO the installer needs to be able to reach an Ubuntu noble archive to be able to download and install the package.

Now as mentioned, I'm trying to perform the automated install in an air-gapped network. I have mirrored the Ubuntu noble repositories from the internet and made them available in the air-gap network. I have also validated that it works by performing a manual install of Ubuntu 24.04.1 LTS and updating the /etc/apt/sources.list.d/ubuntu.sources file to point to the air-gapped mirror and successfully performing an apt update/upgrade and installing openssh-server (and ansible).

So the issue I need to overcome is to somehow configure the Phase 1 (autoinstall:) portion to use my local mirror as a source for package installation.

This process on how to do this is documented here:

https://canonical-subiquity.readthedocs-hosted.com/en/latest/reference/autoinstall-reference.html#apt https://curtin.readthedocs.io/en/latest/topics/apt_source.html

However, after several hours of trying various combinations of trying to configure the "apt:" section in my user-data file what I'm observing is that anything put into the "apt:" section seems to only be applied to the /target/etc/apt/ configurations, not the live session. The apt cache seems to only hold the packages that are available on the ISO. Because of this, specifying additional "packages:" to be installed causes the autoinstall to fail.

Here are the various configurations I've tried:

Using the deb format:

#cloud-config
autoinstall:
  apt:
    preserve_sources_list: false
    mirror-selection:
      primary:
        - uri: "http://mirror.local.net/ubuntu"
          arches: [default]
    fallback: offline-install
    geoip: false

Using the deb822 format:

#cloud-config
autoinstall:
  apt:
    sources:
      local.sources:
        source: |
          Types: deb
          URIs: http://mirror.local.net/ubuntu
          Suites: $RELEASE $RELEASE-updates $RELEASE-backports $RELEASE-security $RELEASE-proposed
          Components: main restricted universe multiverse
          Signed-by: /usr/share/keyrings/ubuntu-archive-keyring.gpg
    preserve_sources_list: false
    fallback: offline-install
    geoip: false

In my testing, it "appears" that when I specify the deb formatted "apt:", there's no indication in the installer console that it even tries to contact the mirror at any point to either update the cache (apt update) or to install the package. When I specify the deb822 formatted "apt:", one of the first things the installer appears to do is validate it can contact mirror.local.net, as well as archive.ubuntu.com and security.ubuntu.com.

So my thoughts were that using the deb822 formatted "apt:" would seem to be the right thing to do however, later in the install when it goes to install any defined packages I can clearly see in the logs:

Sep 05 10:30:00 ubuntu subiquity_log.4233[15323]: Get1 file:/cdrom nobel InRelease
Sep 05 10:30:00 ubuntu subiquity_log.4233[15323]: Ign1 file:/cdrom nobel InRelease
Sep 05 10:30:00 ubuntu subiquity_log.4233[15323]: Get2 file:/cdrom nobel Release [1072 B]
Sep 05 10:30:00 ubuntu subiquity_log.4233[15323]: Get2 file:/cdrom nobel Release [1072 B]
Sep 05 10:30:00 ubuntu subiquity_log.4233[15323]: Reading package lists...
...
Sep 05 10:30:00 ubuntu subiquity_log.4233[15312]: Running command ['unshare', '--fork', '--pid', '--mount-proc=/target/proc', '--', 'chroot', '/target', 'apt-get', '--quiet', '--assume-yes', '--option=Dpkg::options::=--force-unsafe-io', '--option=Dpkg::Options::=-force-confold', 'install', '--download-only', 'openssh-server'] with allowed return codes [0] (capture=False)
Sep 05 10:30:00 ubuntu subiquity_log.4233[15432]: Reading package lists...
Sep 05 10:30:00 ubuntu subiquity_log.4233[15432]: Building dependency tree...
Sep 05 10:30:00 ubuntu subiquity_log.4233[15432]: Reading state information...
Sep 05 10:30:00 ubuntu subiquity_log.4233[15432]: E: Unable to locate package openssh-server

When the installer crash's while still in the Live session, if I pop into the terminal and check the following, there are no references to "mirror.local.net" in any of the following files:

/etc/apt/sources.list
/etc/apt/sources.list.d/ubuntu.sources
/target/etc/apt/sources.list
/target/etc/apt/sources.list~
/target/etc/apt/sources.list.d/ubuntu.sources.curtin.orig

The only reference I can find to what I specified in the "apt:" section can only be found in /etc/cloud/cloud.cfg.d/91_kernel_cmdline_url.cfg file which appears to be a copy of the user-data file.

The other observation I've made is that if I move all of the extra software installation into Phase 2 of the install (1st boot after base install) where the user-data section "stuff" gets processed, the local.sources file does exist in the /etc/apt/sources.list.d directory and if I perform a runcmd: apt install openssh-server -y then the packages install fine.

My question is has anyone else observed this issue or does anyone else have a working airgapped/offline Ubuntu 24.04.1 LTS installation and can verify that they can install openssh-server from an internal Ubuntu repository mirror?

I fully admit that I might be running into a bug with the new installer but I wanted to rule out me being an idiot first. If anyone has an example of a working "apt:" section for specifying a local mirror I'd truly appreciate some guidance.

Thanks for reading and any feedback!

TLDR; "apt:" configurations in autoinstall for Ubuntu 24.04.1 LTS seems to be ignored during the initial install phase.

This may be more of a networking question but very long story short, I'm in the process of setting up a PXEBoot server (TFTP/HTTP) to automate network installs of Linux and the last thing I need to do/test is how to make it work within an existing DHCP environment. I've gone ahead and tested/documented the steps on how to configure DHCP Options 60 & 67 on an existing DHCP server. It works fine as long as your DHCP server can provide the logic necessary to determine the platform difference between a BIOS based or a UEFI based system.

The other method to configure this to install isc-dhcp-server on your TFTP server and either place it in the same subnet or modify the IP Helper options on your routers so its receives the DHCP REQ's. This is what I need is some help understanding how this is supposed to work.

Normally, when designing a subnet with multiple DHCP servers, unless the DHCP servers have some way to replicate or notify each other of which IP's it's handed out, to avoid IP address conflicts you configure them to hand out ranges that don't overlap.

So imagine (for simplicity sake) a flat 192.168.1.0/24 network with an existing DHCP server configured to issue out IP's for the entire range. Is there a way to configure isc-dhcp-server's dhcpd.conf file to only issue the DHCP Options 60/67 and let the existing server hand out remaining network info?

OR... am I expected to redesign my network to only hand out half of the IP's from the existing DHCP server, and the other half from the new isc-dhcp-server? If this is the case, and I only have 1 of the 2 DHCP servers configured to hand out DHCP Options 60/67 won't I run into an issue where sometimes when booting from the network it will get it's info from the DHCP server that doesn't have the necessary information?

I guess what I'm really looking for is for someone who has this all working to explain at a high level how this is all supposed to work together avoiding IP address conflicts.

Thx for any help or guidance!

In summary, my issue specifically is when I PXE boot a UEFI system, I get stuck at a grub> prompt. It appears that the UEFI systems aren't loading the grub.cfg file and I'll be damned if I can figure out why.

Since 22.04 is no longer providing us a mini.iso or netboot.iso option, I've been building (in our lab) a POC of the 'new' way to automate deployment of systems. For the most part I've been following the instructions from these two URLs:

• https://www.molnar-peter.hu/en/ubuntu-jammy-netinstall-pxe.html
• https://www.golinuxcloud.com/uefi-pxe-boot-server-ubuntu-20-04-cloud-init/

For now my PoC is all sitting in a small Hyper-V lab. The network config is pretty simple in that I have a pfSense FW giving my lab network access to the internet and everything behind my FW is on a closed (private) network. This allows me full control over things like DHCP and such.

On what I'm calling my PXE sever, I have installed:

• apache2
• isc-dhcp-server
• tftpd-hpa

TFTP Directory structure:

├── boot
│   ├── bootx64.efi
│   ├── grub
│   │   ├── font.pf2
│   │   ├── grub.cfg
│   │   ├── unicode.pf2
│   │   └── x86_64-efi
│   │       ├── command.lst
│   │       ├── crypto.lst
│   │       ├── fs.lst
│   │       └── terminal.lst
│   ├── grubx64.efi
│   ├── jammy
│   │   ├── initrd
│   │   └── vmlinuz
│   ├── ldlinux.c32 -> ./syslinux/bios/ldlinux.c32
│   ├── libutil.c32 -> ./syslinux/bios/libutil.c32
│   ├── menu.c32 -> ./syslinux/bios/menu.c32
│   ├── pxelinux.0
│   ├── pxelinux.cfg
│   │   └── default
│   └── syslinux
│       └── bios
│           ├── ldlinux.c32
│           ├── libutil.c32
│           └── menu.c32
└── grub -> ./boot/grub/

/etc/dhcp/dhcpd.conf

allow booting;
allow bootp;
option arch code 93 = unsigned integer 16;

subnet 172.16.1.0 netmask 255.255.255.0 {
    range 172.16.1.100 172.16.1.110;
}

host test1 {
    hardware ethernet 00:15:5d:01:21:07;
    if option arch = 00:07 {
        filename "boot/bootx64.efi";
    } else {
        filename "boot/pxelinux.0";
    }
    next-server 172.16.1.50;
    fixed-address 172.16.1.60;
}

host test2 {
        hardware ethernet 00:15:5d:01:21:08;
        if option arch = 00:07 {
               filename "boot/bootx64.efi";
        } else {
                filename "boot/pxelinux.0";
        }
        next-server 172.16.1.50;
        fixed-address 172.16.1.61;
}

NOTE: I want DHCP to be issued by my pfSense server but I discovered the only way to get isc-dhcp-server to start was to specify some sort or range. I need to resolve this but for now it does seem that since everything is on the same subnet things are working well enough with this config.

/etc/default/tftpd-hpa

# /etc/default/tftpd-hpa

TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/tftp"
TFTP_ADDRESS=":69"
TFTP_OPTIONS="--secure

/etc/default/isc-dhcp-server

# Defaults for isc-dhcp-server (sourced by /etc/init.d/isc-dhcp-server)

# Path to dhcpd's config file (default: /etc/dhcp/dhcpd.conf).
#DHCPDv4_CONF=/etc/dhcp/dhcpd.conf
#DHCPDv6_CONF=/etc/dhcp/dhcpd6.conf

# Path to dhcpd's PID file (default: /var/run/dhcpd.pid).
#DHCPDv4_PID=/var/run/dhcpd.pid
#DHCPDv6_PID=/var/run/dhcpd6.pid

# Additional options to start dhcpd with.
#       Don't use options -cf or -pf here; use DHCPD_CONF/ DHCPD_PID instead
#OPTIONS=""

# On what interfaces should the DHCP server (dhcpd) serve DHCP requests?
#       Separate multiple interfaces with spaces, e.g. "eth0 eth1".
INTERFACESv4="eth0"
INTERFACESv6="eth0"

/tftp/boot/grub/grub.cfg

set timeout=30

loadfont unicode

set menu_color_normal=white/black
set menu_color_highlight=black/light-gray

menuentry "Install Ubuntu Jammy (22.04)" {
        set gfxpayload=keep
        linux   /boot/jammy/vmlinuz ip=dhcp cloud-config-url=/dev/null url=http://172.16.1.50/isos/jammy/ubuntu-22.04-live-server-amd64.iso autoinstall ds="nocloud-net;s=http://172.16.1.50/cloud-init/jammy/" --- # Don't forget the slash at the end.
        initrd  /boot/jammy/initrd
}

tcpdump -n -i eth0 port 69

tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
01:26:56.188414 IP 172.16.1.60.1332 > 172.16.1.50.69: TFTP, length 59, RRQ "boot/bootx64.efi" octet tsize 0 blksize 1482 windowsize 4
01:26:56.189458 IP 172.16.1.60.1333 > 172.16.1.50.69: TFTP, length 51, RRQ "boot/bootx64.efi" octet blksize 1482 windowsize 4
01:26:56.225401 IP 172.16.1.60.1334 > 172.16.1.50.69: TFTP, length 50, RRQ "boot/grubx64.efi" octet blksize 512 windowsize 4

grub> echo $prefix

(tftp,172.16.1.50)/boot/grub

Screenshot of Grub> Info

For what it's worth, I have validated that the Apache2 directories are being served correctly and in fact, with the above config I am able to PXE boot BIOS based systems using the pxelinux.0.

For whatever reason UEFI systems (or in my case Hyper-V Gen2) get 'stuck' at the grub> menu.

Any help or advice would be appreciated. I know I'm close.

Cheers!

I'm looking for some advice on how I can upgrade 100's of Ubuntu 20.04.X installations to get all that 20.04.4 goodness.

First some background:

I manage an environment where we have 100's of Ubuntu 20.04.X installations. Up until now, we've just lived with the fact that they are using lvmpipe instead of the built-in/onboard Intel graphics. Anyways for poops and giggles I decided to download the latest 20.04.4 ISO and throw it on a system and to my shock and surprise it was listed and working!

I immediately tried some upgrading of a few of our existing systems and while it did upgrade to 20.04.4, we did not enjoy the same results.

Of note, I noticed that the upgrade kept us on the 5.04 Kernel. Of course I did (and have in the past) try upgrading to the 5.13 HWE kernel and that didn't seem to work.

So I'm wondering... short of having to re-install the OS on all these systems (which really isn't an option we want to explore), does anyone know of some dark magic that I can use to upgrade an existing installation to get all the bits and parts that are obviously included on the ISO?