I have a Debian server acting as a router for a small network. It has 3 network interfaces, one LAN and 2 WANs (two different ISPs, let's call them A and B, each of them with a public static IP address).
The LAN has 5 subnets, some access the internet using ISP A and some use ISP B (plus there's a very simple script-based failover mechanism, which redirects all traffic through one ISP if the other one fails and an OpenVPN server).
This has been working well for quite some time. But up until now we had only used ISP A's public IP address to access the server from the outside. Now we need to be able to access through both public IP addresses, and I'm seeing ISP B's does not work.
Using tcpdump I've noticed that ping requests arrive to ISP B's network interface, but the responses don't make it back. Instead they seem to be going back through ISP A.
I suppose there's something wrong in my routing table, but I'm not sure what. Could you help me figure it out?
Here's how the router looks like (I've replaced the actual public IP addresses for 11.11.11.11 for ISP A and 22.22.22.22 for ISP B):
# ifconfig
eth1 Link encap:Ethernet HWaddr 94:0c:6d:82:0d:98
inet addr:10.1.1.1 Bcast:10.1.1.255 Mask:255.255.255.0
inet6 addr: fe80::960c:6dff:fe82:d98/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:70084654 errors:0 dropped:713 overruns:0 frame:0
TX packets:87266365 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2955150829 (2.7 GiB) TX bytes:3255030277 (3.0 GiB)
Interrupt:20 Base address:0x2000
eth1:0 Link encap:Ethernet HWaddr 94:0c:6d:82:0d:98
inet addr:10.1.2.1 Bcast:10.1.2.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:20 Base address:0x2000
eth1:1 Link encap:Ethernet HWaddr 94:0c:6d:82:0d:98
inet addr:10.1.3.1 Bcast:10.1.3.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:20 Base address:0x2000
eth1:2 Link encap:Ethernet HWaddr 94:0c:6d:82:0d:98
inet addr:10.1.4.1 Bcast:10.1.4.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:20 Base address:0x2000
eth1:3 Link encap:Ethernet HWaddr 94:0c:6d:82:0d:98
inet addr:10.1.5.1 Bcast:10.1.5.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:20 Base address:0x2000
eth3 Link encap:Ethernet HWaddr 94:0c:6d:82:c8:72
inet addr:22.22.22.22 Bcast:22.22.22.255 Mask:255.255.255.0
inet6 addr: fe80::960c:6dff:fe82:c872/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2013773 errors:0 dropped:0 overruns:0 frame:0
TX packets:52720 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:128125141 (122.1 MiB) TX bytes:4658309 (4.4 MiB)
Interrupt:19 Base address:0x6000
eth4 Link encap:Ethernet HWaddr 6c:f0:49:84:79:ca
inet addr:11.11.11.11 Bcast:255.255.255.255 Mask:255.255.255.0
inet6 addr: fe80::6ef0:49ff:fe84:79ca/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:57255186 errors:0 dropped:0 overruns:0 frame:0
TX packets:39862172 errors:0 dropped:0 overruns:0 carrier:1
collisions:0 txqueuelen:1000
RX bytes:1933578821 (1.8 GiB) TX bytes:328150009 (312.9 MiB)
Interrupt:27
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:952741 errors:0 dropped:0 overruns:0 frame:0
TX packets:952741 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:116644740 (111.2 MiB) TX bytes:116644740 (111.2 MiB)
tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:10.8.0.1 P-t-P:10.8.0.2 Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:7907032 errors:0 dropped:0 overruns:0 frame:0
TX packets:6371185 errors:0 dropped:5 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:497588957 (474.5 MiB) TX bytes:3980021182 (3.7 GiB)
# ip rule list
0: from all lookup local
32763: from 10.1.5.0/24 lookup adsl
32764: from 10.1.3.0/24 lookup adsl
32765: from 10.1.2.0/24 lookup adsl
32766: from all lookup main
32767: from all lookup default
# ip route show table main
10.8.0.2 dev tun0 proto kernel scope link src 10.8.0.1
10.8.0.0/24 via 10.8.0.2 dev tun0
11.11.11.0/24 dev eth4 proto kernel scope link src 11.11.11.11
22.22.22.0/24 dev eth3 proto kernel scope link src 22.22.22.22
10.1.4.0/24 dev eth1 proto kernel scope link src 10.1.4.1
10.1.5.0/24 dev eth1 proto kernel scope link src 10.1.5.1
10.1.1.0/24 dev eth1 proto kernel scope link src 10.1.1.1
10.1.2.0/24 dev eth1 proto kernel scope link src 10.1.2.1
10.1.3.0/24 dev eth1 proto kernel scope link src 10.1.3.1
default via 11.11.11.1 dev eth4
default via 22.22.22.1 dev eth3 metric 100
# ip route show table adsl
10.8.0.2 dev tun0 proto kernel scope link src 10.8.0.1
22.22.22.0/24 dev eth3 proto kernel scope link src 22.22.22.22
11.11.11.0/24 dev eth4 proto kernel scope link src 11.11.11.11
10.8.0.0/24 via 10.8.0.2 dev tun0
10.1.4.0/24 dev eth1 proto kernel scope link src 10.1.4.1
10.1.5.0/24 dev eth1 proto kernel scope link src 10.1.5.1
10.1.1.0/24 dev eth1 proto kernel scope link src 10.1.1.1
10.1.2.0/24 dev eth1 proto kernel scope link src 10.1.2.1
10.1.3.0/24 dev eth1 proto kernel scope link src 10.1.3.1
default via 22.22.22.1 dev eth3
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.8.0.2 0.0.0.0 255.255.255.255 UH 0 0 0 tun0
10.8.0.0 10.8.0.2 255.255.255.0 UG 0 0 0 tun0
11.11.11.0 0.0.0.0 255.255.255.0 U 0 0 0 eth4
22.22.22.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3
10.1.4.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
10.1.5.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
10.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
10.1.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
10.1.3.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
0.0.0.0 11.11.11.1 0.0.0.0 UG 0 0 0 eth4
0.0.0.0 22.22.22.1 0.0.0.0 UG 100 0 0 eth3
And here's the tcpdump test (here 22.22.22.22 is ISP B's IP and 99.99.99.99 is the IP of the remote box from which I'm pinging):
# Here's ISP B's interface, eth3
# tcpdump -i eth3 -qtln icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth3, link-type EN10MB (Ethernet), capture size 65535 bytes
IP 99.99.99.99 > 22.22.22.22: ICMP echo request, id 8099, seq 1, length 64
IP 99.99.99.99 > 22.22.22.22: ICMP echo request, id 8099, seq 2, length 64
IP 99.99.99.99 > 22.22.22.22: ICMP echo request, id 8099, seq 3, length 64
IP 99.99.99.99 > 22.22.22.22: ICMP echo request, id 8099, seq 4, length 64
# Here's ISP A's interface, eth4
# tcpdump -i eth4 -qtln icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth4, link-type EN10MB (Ethernet), capture size 65535 bytes
IP 22.22.22.22 > 99.99.99.99: ICMP echo reply, id 8099, seq 9, length 64
IP 22.22.22.22 > 99.99.99.99: ICMP echo reply, id 8099, seq 10, length 64
IP 22.22.22.22 > 99.99.99.99: ICMP echo reply, id 8099, seq 11, length 64
IP 22.22.22.22 > 99.99.99.99: ICMP echo reply, id 8099, seq 12, length 64
You forgot the rule for each interface. And better create 2 tables instead of main and adsl just for the better readability, eg. ispa and ispb
For eth4 add (you can do that with post-up in /etc/network/interface):
(exchange the 11.11.11.12/32 with the IP of your gateway).
Same goes for eth3 with it's addresses.
I don't know if the static routing for the subnets is wanted (it perhaps is), you should consider balancing the traffic between both wan interfaces.
If you need any help on this, this is a great help to setup routing for multiple uplinks:
http://www.debian-administration.org/article/377/Routing_for_multiple_uplinks
The behaviour you are experiencing is expected. This blogpost is a bit old but explains pretty well how to make it work as you want to: configuring-multiple-default-routes-in-linux
EDITED: Using default routes, your outgoing packets will be routed through the default route with biggest preferece, no matter which way they came in.
You need to use policy based routing, so I think that in your case you could just add this:
Check your rules are placed properly with
ip rule list. Anyway in the end this is the same as @Broco's answer.