I have a multihomed Ubuntu server with a set of interfaces that includes:
eth2: 10.10.0.131/24
eth3: 10.20.0.2/24
The default interface is eth2, with a gateway of 10.10.0.1. Here's what the routing table looks like:
root@c220-1:~# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.10.0.1 0.0.0.0 UG 0 0 0 eth2
10.10.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
10.20.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3
10.30.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.40.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
From a separate network (192.168.3.5/24) I can reach this machine on the eth2 interface (the one with the default gateway), but not the eth3 interface. I can ping the eth3 interface from a router on the same network (10.20.0.1) without a problem.
If I ping 10.10.0.131 from 192.168.3.5, the packets reach the machine, but it doesn't send any replies:
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 0, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 1, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 2, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 3, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 4, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 5, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 6, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 7, length 64
If I ping from the router (10.20.0.1) on the same network, the server replies properly:
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80
Note that, as per the answer in this similar question, I have rp_filter off on all of the interfaces, but it doesn't resolve the issue:
$ for i in eth0 eth1 eth2 eth3 all default
> do
> cat /proc/sys/net/ipv4/conf/$i/rp_filter
> done
0
0
0
0
0
0
The problem is that since the default route is through eth2, the ping responses get sent through eth2 even though the requests were received on eth3. (If you tcpdump eth2 you should see the responses being sent.) There's then probably some device which is dropping the packets because they have an invalid source IP for the network they're on. You need some source policy routing in order to get the responses to be sent out the interface they were received on.
Create a new routing table (only needs to be done once):
Add a default route to this new table going out eth3:
Add a policy rule to use this new table for packets with source address of eth3's IP:
It sounds like you are missing a route for 192.168.3.5/24 from the 10.30.0/24 subnet. You should add a network diagram and traceroutes for each network from each device.