I have a recipe I already use on many cases, but this time doesn't works on Debian 11 (kernel 5.10.0-10-amd64)
my setup is basically an internal interface eth0 for a RFC1918 LAN, and two external interfaces connected to some ISP's Box:
eth1 for ISP1 as default router at 10.0.0.254 with public IP 1.2.3.4 (figuratively)
eth2 for ISP2 has a router at 10.0.3.254 with public 2.3.4.5
I have different possible route. I want to control which route my packet takes, so I create some rule and fwmark. First I append 2<tab>secondrouter in /etc/iproute2/rt_tables
ip rule add fwmark 0x3 lookup secondrouter
ip route add default via 10.0.3.254 table secondrouter
everythings is fine regarding ip route list table secondrouter and ip rule list
at this time I am able to do:
curl -4 ifconfig.me
1.2.3.4 #<- public ip address of my default route
Then I do
iptables-legacy -t mangle -A OUTPUT -d 34.117.59.81 -j MARK --set-mark 0x3
now if I do
curl -4 ifconfig.me
<timeout>
Where I expected 2.3.4.5 as public IP. So clearly the marked packet do not take the route from the ip route table, worse, it timeouts.
If I do this exactly the same way on older Debian, its works perfectly.
NB:if I do a
ip route add 34.117.59.81 via <second router IP>
my curl test works perfectly as expected
curl -4 ifconfig.me
2.3.4.5 #<- Pub ip address of my second router
My problem occurs when using iptables or iptables-legacy to mark packets to route.
BTW I have plenty of iptables rules that works fine, so it do not looks like an iptables issue.
iptables-legacy-save
# Generated by iptables-save v1.8.7 on Sun Jan 23 22:35:06 2022
*mangle
:PREROUTING ACCEPT [41:5019]
:INPUT ACCEPT [41:5019]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [44:4752]
:POSTROUTING ACCEPT [44:4752]
-A OUTPUT -d 192.168.0.0/16 -j RETURN
-A OUTPUT -d 172.16.0.0/12 -j RETURN
-A OUTPUT -d 10.0.0.0/8 -j RETURN
-A OUTPUT -d 34.117.59.81/32 -j MARK --set-xmark 0x3/0xffffffff
COMMIT
# Completed on Sun Jan 23 22:35:06 2022
# Generated by iptables-save v1.8.7 on Sun Jan 23 22:35:06 2022
*filter
:INPUT DROP [19:7746]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [846:62420]
-A INPUT -m state --state INVALID -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -p udp -m udp --dport 500 -j ACCEPT
-A INPUT -p esp -j ACCEPT
-A INPUT -p ah -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -s 192.168.0.0/16 -p tcp -m tcp --dport 3128 -j ACCEPT
-A INPUT -s 172.16.0.0/12 -p tcp -m tcp --dport 3128 -j ACCEPT
-A INPUT -s 10.0.0.0/8 -p tcp -m tcp --dport 3128 -j ACCEPT
-A FORWARD -m state --state INVALID -j DROP
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.0.0/16 -d 10.0.6.0/24 -j ACCEPT
-A FORWARD -s 10.0.6.0/24 -d 192.168.0.0/16 -j ACCEPT
COMMIT
# Completed on Sun Jan 23 22:35:06 2022
When the packet get out of the machine it is given a source IP address corresponding to the default output interface.
So it is required to add a MASQUERADE rule on the output interface because the mark rules gives our packet another destination than the planned one.
for the sake of understanding netfilter order of processing : (via Plouf)