ssh through openVPN on openWRT hangs on SSH2_MSG_KEXINIT sent

I'm encountering a problem where ssh to a remote server hangs with the message SSH2_MSG_KEXINIT sent. I have already explored the MTU options with no luck.

Case 1 (fails)

This happens when I ssh from my Linux PC through an openWRT router running the openVPN client. I have the router set to forward from the LAN 44.20.11.0/24 to the VPN tun0. I can successfully ping the remote ssh server from my PC, so I don't think forwarding is the problem.

This openVPN PC client version is 2.5.2 mips-openwrt-linux-gnu. The server-side openWRT openVPN version is openvpn-2.3.8-16.26.1.x86_64.

Here is the ssh client's output for the failed case:

ssh -vvvv [email protected]
OpenSSH_8.8p1, OpenSSL 1.1.1m  14 Dec 2021
debug1: Reading configuration data /home/myuser/.ssh/config
debug1: Reading configuration data /usr/etc/ssh/ssh_config
debug1: /usr/etc/ssh/ssh_config line 24: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /usr/etc/ssh/ssh_config line 26: Applying options for *
debug2: resolve_canonicalize: hostname 10.84.96.200 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/myuser/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/myuser/.ssh/known_hosts2'
debug3: ssh_connect_direct: entering
debug1: Connecting to 10.84.96.200 [10.84.96.200] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: Connection established.
debug1: identity file /home/myuser/.ssh/id_rsa type 0
debug1: identity file /home/myuser/.ssh/id_rsa-cert type -1
debug1: identity file /home/myuser/.ssh/id_dsa type -1
debug1: identity file /home/myuser/.ssh/id_dsa-cert type -1
debug1: identity file /home/myuser/.ssh/id_ecdsa type 2
debug1: identity file /home/myuser/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/myuser/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/myuser/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/myuser/.ssh/id_ed25519 type -1
debug1: identity file /home/myuser/.ssh/id_ed25519-cert type -1
debug1: identity file /home/myuser/.ssh/id_ed25519_sk type -1
debug1: identity file /home/myuser/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/myuser/.ssh/id_xmss type -1
debug1: identity file /home/myuser/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6
debug1: compat_banner: match: OpenSSH_7.6 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 10.84.96.200:22 as 'myuser'
debug3: record_hostkey: found key type ECDSA in file /home/myuser/.ssh/known_hosts:828
debug3: record_hostkey: found key type RSA in file /home/myuser/.ssh/known_hosts:2050
debug3: record_hostkey: found key type DSA in file /home/myuser/.ssh/known_hosts:2051
debug3: record_hostkey: found key type ED25519 in file /home/myuser/.ssh/known_hosts:2052
debug3: load_hostkeys_file: loaded 4 keys from 10.84.96.200
debug1: load_hostkeys: fopen /home/myuser/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type [email protected], using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent```

Here is the PC's route for this scenario (Case 1, failure):

default via 44.20.6.1 dev br0
10.0.0.0/8 via 44.20.11.11 dev enp1s0
44.20.6.0/24 dev br0 proto kernel scope link src 44.20.6.2
44.20.11.0/24 dev enp1s0 proto kernel scope link src 44.20.11.187

Case 2 (works)

When I run the openVPN client directly on the Linux PC, I can successfully ssh to the remote ssh server. This makes me think the problem lies in the openWRT/openVPN router.

This openvpn client version is 2.5.5 x86_64-suse-linux-gnu

Here is the Linux PC's route when bypassing the openWRT router and running the openVPN client directly on the PC:

default via 44.20.6.1 dev br0
44.20.6.0/24 dev br0 proto kernel scope link src 44.20.6.2
44.20.11.0/24 dev enp1s0 proto kernel scope link src 44.20.11.187

I only have one instance of the openVPN client running at any given time.

I've read all the threads suggesting changing the MTU. I've tried setting it to 1492, 1000, and 576 on the client side's network devices, with no luck. The server-side openVPN server's MTU is set to 1492.

Both the Linux PC and openWRT/openVPN VPN clients use the same client config (relevant portions included):

client
dev tun
keepalive 10 300
reneg-sec 0
auth-user-pass
script-security 3
pull
persist-key
persist-tun
ns-cert-type server
comp-lzo
verb 6
mute 20
auth SHA512
cipher AES-256-CBC

My observations:

Because this works when running the VPN client directly on the Linux PC, I do not believe this is an openVPN client or server misconfiguration. I suspect the problem lies in between, in the openWRT/openVPN router. When ssh'ing directly from the openWRT router, the connection also hangs.

mtr 10.84.96.200 shows no packet loss.

I suspect the problem could be one of these things:

1. I didn't change MTU on the correct network devices
2. MTU should be a value other than 1492, 1000 or 576
3. There is a firewall setting on the openWRT router that's breaking this
4. Something in openWRT makes this fail

The openWRT device is a gl.inet Slate travel router. I can access the logs from both the client and server openVPNs, but I can't change the server side openVPN. I can access and change the server-side ssh server.

Here is a diagram to hopefully explain more clearly: