Debian 10 cloud-init waiting for DHCP on boot with static network configuration

I encountered the sample problem -- using a static network configuration (NoCloud provider meta-data ENI, or network-config v1/v2) does not disable the DHCP client.

Seems a network config is applied from a template (/etc/network/cloud-interfaces-template) before the cloud-init configuration is written.

auto $INTERFACE
allow-hotplug $INTERFACE

iface $INTERFACE inet dhcp

You can test that this template is the culprit by changing the cloud-image before first start:
(patching the image as changing network config in e.g. bootcmd is too late.)

qemu-nbd --connect=/dev/nbd0 /tmp/debian-10-genericcloud-amd64-20210208-542.qcow2
fdisk /dev/nbd0 -l
mkdir /tmp/nbd
mount /dev/nbd0p1 /tmp/nbd
sed -i 's/dhcp/manual/' /tmp/nbd/etc/network/cloud-interfaces-template
umount /tmp/nbd
rmdir /tmp/nbd
qemu-nbd --disconnect /dev/nbd0

I still need to find a way to apply this change or prevent the use of this template with cloud-init though.

That template seems to be processed by /etc/network/cloud-ifupdown-helper, so that script could be changed or influenced perhaps.

I met the same problem.

Here is a better way to resolve it, just set DHCP timeout to a shorter time.

# virt-edit debian-10-generic-amd64.qcow2 /etc/dhcp/dhclient.conf

timeout 15;

Then this image can function correctly in NoCloud environment or DHCP network.

Great pointers by @zany

In my case I was trying to configure a Debian 11 generic cloud image with cloud-init and a static IP on my KVM host (using dmacvicar libvirt Terraform provider)

My network-config file was:

version: 2
ethernets:
  ens3:
    dhcp4: false
    addresses: [10.1.0.100]
    gateway4: 10.1.0.1
    nameservers:
      addresses: [10.1.0.1 1.1.1.1]
      search: [home.lab]

Then I was surprised that during VM creation, the interface was requesting a DHCP lease (journalctl is your friend) before cloud-init config would actually kick in and configure the interface as per my static settings (exacltly like the OP described)

After a minute or so, the "mysterious" dhclient gave up waiting for an offer (that was expected as DHCP is disabled on my libvirt network) and was then left running in the background. Then the boot sequence continues and cloud-init kicks in, rendering the correct static config in /etc/network/interfaces.d/50-cloud-init.cfg. At that point, the interface gets the expected static IP (ip address show proves that, and you can also ping things by IP), however is leaving DNS resolution broken. I guess it's a side effect of the dhclient fiasco.

Well after some digging it turns out the /etc/network/interfaces file, in addition to sourcing source-directory /etc/network/interfaces.d it also sources the extra directory /run/network/interfaces.d/. To my surprise, that /run directory contains an interface definition for ens3 where it is being configured in dhcp mode!

So now that I knew where the unexpected dhcp request was coming from, it was a matter of disabling it, since it was conflicting with the correct settings in /etc/network/interfaces.d/50-cloud-init.cfg.

Unfortunatelly disabling the intial dhcp request happens before cloud-init kicks in, so really no easy way to prevent dhclient wasting a precious minute or so trying to get an offer that will never come.

What I was able to accomplish though, was fixing DNS resolution by using the following bootcmd: block in my user-data

bootcmd:
  - cloud-init-per once down ifdown ens3
  - cloud-init-per once bugfix rm /run/network/interfaces.d/ens3
  - cloud-init-per once up ifup ens3

In the above commands, I'm bringing the interface down which stops the dormant dhclient process, then I'm removing the interface definition file that initially sets ens3 in dhcp mode, and finally I'm bringing the ens3 interface back up, which applies what's set in /etc/network/interfaces.d/50-cloud-init.cfg like a champ.

With that, the subsequent cloud-init stages in the initial boot process were now able to fully reach the internet by name. That was critical for the later stages such the packages: block to succeed, since it needed DNS working to resolve the apt repo server name.

Here's the more detailed user-data excerpt:

bootcmd:
  - cloud-init-per once ifdown ifdown ens3
  - cloud-init-per once bugfix rm /run/network/interfaces.d/ens3
  - cloud-init-per once ifup ifup ens3

packages:
  - qemu-guest-agent
  - locales-all

package_update: true
package_upgrade: true
package_reboot_if_required: true

runcmd:
  - [ systemctl, start, qemu-guest-agent ]

final_message: "The system is finally up, after $UPTIME seconds"

Despite not being on Debian10, the issue sounded so familiar that thought I'd share my experience in case you face this issue in newer releases.

References:

• https://cloudinit.readthedocs.io/en/latest/topics/network-config.html#network-configuration
• https://cloudinit.readthedocs.io/en/latest/topics/network-config-format-v2.html
• https://cloudinit.readthedocs.io/en/latest/topics/modules.html#bootcmd
• https://cloudinit.readthedocs.io/en/latest/topics/modules.html#package-update-upgrade-install
• https://cloud.debian.org/images/cloud/bullseye/latest/debian-11-genericcloud-amd64.qcow2