As recently as yesterday, I was able to connect to our development instance via my local ssh client, using one of several ssh keys listed under sshKeys in the instance's custom metadata (the old-fashioned way). ("Block project-wide ssh keys" is checked). Also that afternoon, while connected, I was doing something where a second ssh session would be convenient. I tried opening a second connection through the local ssh client, and got "Permission denied (publickey)." I tried opening a second connection from the instance console page, and it wouldn't connect, either. Meanwhile, the ssh session I was using continued to work just fine.
This morning I can't get an ssh session on that instance AT ALL.
The only thing I can remember doing yesterday, that was the least bit unusual, was a chmod on my home directory, setting it to "777."
I tried yanking out the sshKeys metadata, and trying the console ssh again. No joy. Then I looked at the serial port 1 log, and noticed this, with timestamps showing GMT for approximately when I tried it:
Nov 28 21:26:49 bitnami-trac-dm-87ea google-accounts: INFO Removing user <redacted>.
Nov 28 21:26:49 bitnami-trac-dm-87ea google-accounts: INFO Removing user <redacted>.
Nov 28 21:26:49 bitnami-trac-dm-87ea google-accounts: INFO Removing user <redacted>.
Nov 28 21:26:49 bitnami-trac-dm-87ea google-accounts: INFO Removing user <redacted>.
Nov 28 21:26:49 bitnami-trac-dm-87ea google-accounts: INFO Removing user <redacted>.
Nov 28 21:26:49 bitnami-trac-dm-87ea google-accounts: INFO Removing user <redacted>.
Nov 28 21:26:49 bitnami-trac-dm-87ea google-accounts: INFO Removing user <redacted>.
Nov 28 21:26:49 bitnami-trac-dm-87ea google-accounts: INFO Removing user <redacted>.
I can get to the password prompt on the serial port, but I've never set myself up with password access, to the best of my recollection.
This is probably what locked you out.
The SSH daemon is pretty paranoid (for good reason), and it'll check various permissions before it allows a connection. If your
.ssh/authorized_keys
file is writable by other users on the server, SSH won't allow incoming connections for the public keys listed in there.(It'll also do the same for outbound connections if it detects your private keys are too readable/writeable.)