Home / server / Questions / 1005084
Accepted
Chau Chee Yang
Asked: 2020-03-01 00:50:54 +0800 CST

SSH Client freeze connect to SSH server with multiple network interface

  • 772

Case 1: SSH host has ONE network interface

Routing table for host SSH:

[SSH] $ ip route
default via 192.168.211.1 dev eth0
192.168.211.0/24 dev eth0 proto kernel scope link src 192.168.211.119

Initiate ssh connection from SSH_Client to host SSH on LAN_0 works fluently:

[SSH_CLIENT] $ ssh [email protected] -p 22

Case 2: SSH host has TWO network interface

Routing table for host SSH:

[SSH] $ ip route
default via 192.168.0.1 dev eth1
192.168.0.0/24 dev eth1 proto kernel scope link src 192.168.0.113
192.168.211.0/24 dev eth0 proto kernel scope link src 192.168.211.119

Initiate ssh connection from SSH_Client to host SSH on LAN_0 works fine:

[SSH_Client] $ ssh [email protected]

Initiate ssh connection from SSH_Client to host SSH on LAN_211 works for few seconds and freeze thereafter:

[SSH_Client] $ ssh [email protected]
ssh pfsense

1 Answers

  1. Best Answer

    It's a case of asymmetric routing, caused by the weak host model used in Linux: any interface can be used for any of the IPs belonging to the host.

    • host client sends packets to an address outside of its own LAN so routes them through the pfSense router.
    • host SSH receives a packet on eth0 from 192.168.0.136 to 192.168.211.119
    • host SSH when replying follows its routing table and thus uses eth1 to reply to client using source IP 192.168.211.119: client sees now the connection as established.
    • conjecture: router pfSense, which sees only half of the traffic (from client to SSH but not from SSH to client), detects out of window non-ACKed TCP traffic and drops further packets after a short time.

    If host SSH had enabled Strict Reverse Path Forwarding, this wouldn't have worked at all, which would probably have been better considering the current outcome, because SSH would then have dropped all packets having IP from LAN 0 received on LAN 211's interface.

    The solution is to use policy routing on host SSH to correct the behaviour of this multi-homed Linux system: have it answer using the same side where the IP is assigned. This is done by creating additional routing tables where, on purpose, only a partial (and perhaps simplified) copy of the main table is present, so that each table ignores the interface which shouldn't be used. While usually only one additional table is needed to have this working, using a symmetric design with two tables is cleaner.

    Also, a default route (through pfSense) is added back in the LAN 211 specific table.

    Then ip rules will call those routing tables before defaulting to the main table.

    Let's use tables with arbitrarily chosen values 10000 and 10211 for respectively LAN 0 and LAN 211 handling.

    # ip route add table 10000 192.168.0.0/24 dev eth1
# ip route add table 10000 default via 192.168.0.1 dev eth1
# ip route add table 10211 192.168.211.0/24 dev eth0
# ip route add table 10211 default via 192.168.211.1 dev eth0

# ip rule add from 192.168.0.113 lookup 10000
# ip rule add from 192.168.211.119 lookup 10211

    Now one can verify that each source IP address will use the correct path, by asking the kernel which path it will choose:

    # ip rule
0:      from all lookup local
32764:  from 192.168.211.119 lookup 10211
32765:  from 192.168.0.113 lookup 10000
32766:  from all lookup main
32767:  from all lookup default

# ip route get 192.168.0.136 from 192.168.0.113
192.168.0.136 from 192.168.0.113 dev eth1 table 10000 uid 0 
    cache 
# ip route get 192.168.0.136 from 192.168.211.119
192.168.0.136 from 192.168.211.119 via 192.168.211.1 dev eth0 table 10211 uid 0 
    cache

    The SSH host can now accept connections from the two sides simultaneously: to each of its two IP addresses and still route correctly each traffic on the correct interface.

    As a side note, those rules won't match when an IP address has not yet been bound on the socket (example: outgoing connection initiated from host SSH as a standard client). The main table will then still be used:

    # ip route get 192.168.0.136
192.168.0.136 dev eth1 src 192.168.0.113 uid 0 
    cache

    You'll have to figure yourself how to integrate those settings to the network manager in use on your specific system. It's often distribution or tool dependent. Note that when an IP address is removed (then added back) or an interface is brought down (then up), tables 10000 and 10211 will be flushed: they have to be updated again each time after such an event happens, contrary to entries in the main table where the kernel handles the LAN routes automatically (but yet, not the default route).

    Since the setup has been corrected, maybe it's time, by principle, to enable Strict Reverse Path Forwarding (the two last commands are needed in case the value was set to 2 rather than 0 before: as the documentation for rp_filter tells higher value overcomes lower value):

    # sysctl -w net.ipv4.conf.all.rp_filter=1
# sysctl -w net.ipv4.conf.eth0.rp_filter=1
# sysctl -w net.ipv4.conf.eth1.rp_filter=1
    • 4