Home / server / Questions / 1098184
In Process
Mtl Dev
Asked: 2022-04-09 11:18:51 +0800 CST

Tailscale: SSH works with manually compiled Openssh, but not system installed Openssh

  • 772

(This question should be tagged with "Tailscale", but that doesn't exist yet - requires knowledge of Tailscale)

I have a standard Ubuntu 20.04 laptop, connecting via ssh to a remote Raspberry Pi (port 22). At the remote location, there is port forwarding on the router (59995->22).

Note: I have removed ~/.ssh/config during testing.

SSH always connects successfully, using:

ssh user@public_ip -i ~/.ssh/id_rsa -p 59995

However, when I use the Tailscale IP, it never connects, just hangs for 120s then times out:

ssh user@tailscale-ip -i ~/.ssh/id_rsa (does not work)

However, if I manually compile OpenSSH v8.2, and use that version, instead of the /usr/bin/ssh binary, then it works perfectly! Even though /usr/bin/ssh is also v8.2

./ssh user@tailscale-ip -i ~/.ssh/id_rsa (this works!)

Recap: the system binary works fine with "normal" ssh, but not with ssh-via-tailscale. However a manually compiled (same) version of Openssh works fine with both regular ssh, and ssh-over-tailsale. Does anyone know what might be causing this?

Additional Notes

I don't think it's any obvious problem with the tailscale network: i.e if I probe via tailscale IP then nmap scan shows correct ports are open, and I can netcat directly to tailscale_ip:22, it is receving and responding.

diff ~/openssh-8.2p1/ssh /usr/bin/ssh shows that the binary's differ. (huge size difference too, about 3MB vs 0.7MB)

/usr/bin/ssh -V : OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f 31 Mar 2020

~/openssh-8.2p1/ssh -V : OpenSSH_8.2p1, OpenSSL 1.1.1f 31 Mar 2020

Log of /usr/bin/ssh -vvv user@tailscale_ip -i ~/.ssh/id_rsa (ip redacted)

OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 100.1.1.1 is address
debug2: ssh_connect_direct
debug1: Connecting to 100.1.1.1 [100.1.1.1] port 22.
< 2 minute hang here >
ssh: connect to host 100.1.1.1 port 22: Connection timed out
ssh

1 Answers

  1. I had similar issue because my ssh client didn't show Tailscale's 2FA message, with default ACL policy for ssh/action set to "check".

    Try checking ACL settings for your user/tag? What is action value for ssh? If it set to "check" - try changing it to "access" and try to access your host via ssh. If this work - I'll suggest to look into how to add 2FA into ssh client so it would show Tailscale's authentication link which is required in "check" mode.

    Some links that helped me to investigate and fix similar issue:

    • 0