I could not ssh into one of my servers this morning after successfully doing so most recently yesterday.
ssh hostname
Connection closed by hostname port 22
I rebooted system and it did not fix the issue.
I dug out a keyboard and a display and could not find any issues or messages in /var/log/messages /var/log/warn or journalctl.
I stopped sshd (systemctl stop sshd), ran sshd from the command line (sshd -ddd) and there are no error messages, and no connection. sshd does unexpectedly terminate with an exit value 255. Note that when running as a system daemon it is not exiting, just not reporting anything.
Another symptom is that ssh does not prompt for a password for either a non-existing user or an existing user. sshd will however log that the non-existing user is invalid:
Invalid user bob from [host] port 40718
While investigating this issue, I found that usermod fails.
usermod -a -G nogroup nobody
usermod: PAM: Authentication failure
Thinking this is a symptom of my sshd failure I started to investigate usermod failing and found the deep pit of despair that is PAM. This question Switch on PAM debugging to Syslog documents the zero ability to debug anything within PAM without recompling from source.