I have four Azure VM machines running Ubuntu 20.04.3 that I regularly shut down. Yesterday I shut them down, and early today I restarted them. Usually this presents no problems.
Today, one of the machines started giving me the dreaded WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED:
» ssh machine-two
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: POSSIBLE DNS SPOOFING DETECTED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The ECDSA host key for machine-two.westeurope.cloudapp.azure.com has changed,
and the key for the corresponding IP address 23.231.121.245
is unknown. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:lU2tqbkQU+e3l+hymVr+lU2tqbkQUWo3/wXxdT/MGA4.
Please contact your system administrator.
Add correct host key in /home/theuser/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/theuser/.ssh/known_hosts:92
remove with:
ssh-keygen -f "/home/theuser/.ssh/known_hosts" -R "machine-two.westeurope.cloudapp.azure.com"
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.11.0-1021-azure x86_64)
I have not reinstalled the machine - maybe I did a system update yesterday, but for sure no reinstallaion.
What I see is that the host keys are indeed updated:
» ls -l /etc/ssh/
total 576
-rw-r--r-- 1 root root 535195 Jul 23 14:55 moduli
-rw-r--r-- 1 root root 1603 Jul 23 14:55 ssh_config
drwxr-xr-x 2 root root 4096 Jul 23 14:55 ssh_config.d
-rw------- 1 root root 1393 Nov 24 04:05 ssh_host_dsa_key
-rw-r--r-- 1 root root 610 Nov 24 04:05 ssh_host_dsa_key.pub
-rw------- 1 root root 513 Nov 24 04:05 ssh_host_ecdsa_key
-rw-r--r-- 1 root root 182 Nov 24 04:05 ssh_host_ecdsa_key.pub
-rw------- 1 root root 411 Nov 24 04:05 ssh_host_ed25519_key
-rw-r--r-- 1 root root 102 Nov 24 04:05 ssh_host_ed25519_key.pub
-rw------- 1 root root 2610 Nov 24 04:05 ssh_host_rsa_key
-rw-r--r-- 1 root root 574 Nov 24 04:05 ssh_host_rsa_key.pub
-rw-r--r-- 1 root root 342 Oct 21 23:45 ssh_import_id
-rw-r--r-- 1 root root 3287 Oct 21 23:46 sshd_config
drwxr-xr-x 2 root root 4096 Oct 22 00:09 sshd_config.d
And the update time matches the boot time:
» last -5
azureuse pts/1 216.129.44.18 Wed Nov 24 04:10 still logged in
azureuse pts/0 216.129.44.18 Wed Nov 24 04:10 still logged in
reboot system boot x.yy.0-zzzz-azur Wed Nov 24 04:04 still running
azureuse pts/0 216.129.44.18 Tue Nov 23 08:22 - 23:02 (14:39)
reboot system boot x.yy.0-zzzz-azur Tue Nov 23 03:58 - 23:02 (19:04)
wtmp begins Tue Oct 26 20:19:31 2021
The other machines are not affected by this, and have never been. In fact, this is the first time I see that a host key changes "spontaneously".
Why would a reboot update the host keys? What else should I verify? For the time being, I am connecting with StrictHostKeyChecking no
Run
grep ssh /var/log/cloud-init.log. Apparently cloud-init will regenerate keys/fingerprints if the VM's instance ID changes. This file seems to have a long history so it should be easy to tell if it happened around your timeframe.