Home / server / Questions / 1048065
In Process
Kristi Jorgji
Asked: 2020-12-31 09:24:05 +0800 CST

KOPS Kubernetes cannot login to bastion host, ssh public key permission error

  • 772

I am learning about Kubernetes and wanted to provision one such cluster in AWS via the KOPS tool. Followed official tutorial then for short this one too https://medium.com/andcloudio/kubernetes-kops-cluster-on-aws-f55d197d8304

I also made sure to add the ssh key before trying to connect to the bastion host as explained here too https://kops.sigs.k8s.io/bastion/#using-the-bastion

All goes fine, nodes, workes, load balancers etc get created and bastion host too.

The only problem is that I cannot ssh into the bastion host with the key. I runed the ssh with -vvv to see verbose output and log is below. I don't understand what is the problem

ssh -A admin@${bastion_elb_url} -vvv

Warning: Permanently added 'bastion-single-k8s-local-noarfe-151938406.eu-central-1.elb.amazonaws.com,3.121.65.83' (ECDSA) to the list of known hosts.
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /root/.ssh/id_rsa (0x55d6af4ea570), agent
debug2: key: /root/.ssh/id_dsa ((nil))
debug2: key: /root/.ssh/id_ecdsa ((nil))
debug2: key: /root/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,[email protected],ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected]>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ecdsa
debug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ed25519
debug3: no such identity: /root/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).

I am also posting key results to help troubleshooting:

root@vagrant:/srv# ssh-add -L
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCq9cN3EAEy0WiASY/IBkF9SPIpLv/bZt1tpLc95cb5fG++ac5VX36rA4XukJFtCAk6I4P82ysuqfZGUQNsB57yibz9rbKZ1bFfxRPyGZS22/1Omqb/8B2NlNpJx42sK4odyUj3G+KLCGCmID/AEDhbjeY7d99ZuE6g8aqrtSo0fwsmNHnpvDS8Dt0IjbLxg41Sms9tmYDLlc/tncAs9BmRvuhPbg+BDw+z7ecLneI7+TexDfhXbnZkYfjFLsfI8vWivOu8ptuGVvPkQz/MJo+MokZEzoGbVCAZP5mYSIz+LIFnnCoh5WOMsB3OZuwvelR5bBgWjQhvOaWOX8BuSU5v /root/.ssh/id_rsa
ssh amazon-web-services kubernetes

1 Answers

  1. As you can see in the verbose output access was denied based on the publickey that was used when trying to authenticate:

    debug1: Authentications that can continue: publickey
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ecdsa
debug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ed25519
debug3: no such identity: /root/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).

    You can see above that all files that are checked by default (if the particular file wasn't specified with -i flag) were not found in your /root/.ssh/ directory.

    As already discussed in comments it turned out that you used the admin user that wasn't defined on the remote host. You confirmed that with ubuntu user you managed to login successfully: "now tried with ubuntu user and logged in successfully to the bastion host" As it was clafiried sufficiently I will focus only on answering your additional question, posted in comments:

    Then from it had to repeat process copy keys from host -> bastion, then ssh from bastion to kubernetes master. This worked now but I expected the -A flag as doumented officially to forward somehow to the master but did not happen. Had to double ssh manually and copy keys to bastion – Kristi Jorgji Dec 31 '20 at 19:35

    The ssh login process you described is known as ssh-ing via so called a jump host. Keep in mind that it doesn't work this way out of the box and requires additional configuration. Take a look at this article, as it clearly explains everything, you need to know about setting up SSH agent forwarding which needs to be done if you want to use your local keys to ssh not only to the bastion host (which happens to be a jump host in this scenario) but also to automatically ssh from there to another remote host.

    In short, you need to create ~/.ssh/config file on your local machine, if it doesn't exist, and set the host you want to allow your local ssh keys to be forwarded to and set ForwardAgent to yes:

    Host example.com # it can be either domain name or IP address
  ForwardAgent yes

    Additionally make sure your jump host allows SSH agent forwarding on inbound connections:

    Agent forwarding may also be blocked on your server. You can check that agent forwarding is permitted by SSHing into the server and running sshd_config. The output from this command should indicate that AllowAgentForwarding is set.

    Now you should be able to ssh directly to your destination remote host via a jumphost directly from your local machine with one ssh command. It's well described here:

    Dynamic jumphost list

    You can use the -J option to jump through a host:

    user $ ssh -J host1 host2

    If usernames or ports on machines differ, specify them:

    user $ ssh -J user1@host1:port1 user2@host2:port2
    Multiple jumps

    The same syntax can be used to make jumps over multiple machines:

    user $ ssh -J user1@host1:port1,user2@host2:port2 user3@host3
    • 0