I have a Linux based router with four interfaces (each with its own private subnet).
When I directly connect a device directly (i.e. no switch, just a patch cable) to one interface and another device directly to another, as below, then the router works perfectly.
DEVICE1
192.168.8.11 ------- 192.168.8.254
ROUTER
10.58.129.254 ------- DEVICE2
10.58.129.1
When I connect the router with our switches in between, as below, then router doesn't work.
DEVICE1
192.168.8.11 ----------- switch1
|
switch2
|
switch3
|
192.168.8.254
ROUTER
10.58.129.254 -------- switch3
|
DEVICE2
10.58.129.1
All the switches are Layer 3, Switch1 (Dell PowerConnect 3548P) has a fibre connection to Switch2 (Dell PowerConnect 6224F) which is our core switch that handles routing between most VLANs. This is connected via fibre to Switch3 (Dell PowerConnect 6224).
Routing on the core switch isn't enabled either of the two VLANs (192.168.8.11 or 10.58.129.254). The reason for this is because our core switch doesn't support policy based routing, hence the reason behind this Linux box to perform routing on these VLANs.
With the router connected via the switches, from Device1, I can ping the interface 192.168.8.254 on the Linux router, but can't ping the other interface (10.58.129.254).
Switch2 configuration/diagnostics
switch2#show ip route
Route Codes: R - RIP Derived, O - OSPF Derived, C - Connected, S - Static
B - BGP Derived, IA - OSPF Inter Area
E1 - OSPF External Type 1, E2 - OSPF External Type 2
N1 - OSPF NSSA External Type 1, N2 - OSPF NSSA External Type 2
S 0.0.0.0/0 [50/0] via 10.58.3.16, vlan 3
C 10.58.3.0/24 [0/0] directly connected, vlan 3
C 10.58.4.0/24 [0/0] directly connected, vlan 4
C 10.58.5.0/24 [0/0] directly connected, vlan 5
C 10.58.9.0/24 [0/0] directly connected, vlan 9
C 10.58.10.0/24 [0/0] directly connected, vlan 10
C 10.58.11.0/24 [0/0] directly connected, vlan 11
C 10.58.12.0/24 [0/0] directly connected, vlan 12
S 10.58.64.0/24 [40/0] via 10.58.3.17, vlan 3
S 10.58.128.0/24 [40/0] via 10.58.3.254, vlan 3
S 10.58.129.0/24 [1/0] via 10.58.3.254, vlan 3
S 192.168.8.0/24 [1/0] via 10.58.3.254, vlan 3
switch2#ping 10.58.129.254
Pinging 10.58.129.254 with 64 bytes of data:
----10.58.129.254 PING Statistics----
4 packets transmitted,0 packets received,100% packet loss
round-trip (ms) min/avg/max = 0/NaN/0
switch2#ping 192.168.8.254
Pinging 192.168.8.254 with 64 bytes of data:
----192.168.8.254 PING Statistics----
4 packets transmitted,0 packets received,100% packet loss
round-trip (ms) min/avg/max = 0/NaN/0
Router diagnostics
router# traceroute -d 192.168.8.11
traceroute to 192.168.8.11 (192.168.8.11), 30 hops max, 60 byte packets
1 192.168.8.11 (192.168.8.11) 0.237 ms 0.222 ms 0.211 ms
router# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.58.3.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.58.128.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3
10.58.129.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
192.168.8.0 0.0.0.0 255.255.255.0 U 0 0 0 eth4
router# ping 192.168.8.11
PING 192.168.8.11 (192.168.8.11) 56(84) bytes of data.
64 bytes from 192.168.8.11: icmp_seq=1 ttl=128 time=2.23 ms
64 bytes from 192.168.8.11: icmp_seq=2 ttl=128 time=0.237 ms
Device1 diagnostics
(device1)c:\>route print
===========================================================================
Interface List
0x1 ........................... MS TCP Loopback interface
0x2 ...bc 30 5b d8 41 c3 ...... Broadcom NetXtreme 57xx Gigabit Controller - Pac
ket Scheduler Miniport
===========================================================================
===========================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 192.168.8.254 192.168.8.11 20
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1
192.168.8.0 255.255.255.0 192.168.8.11 192.168.8.11 20
192.168.8.11 255.255.255.255 127.0.0.1 127.0.0.1 20
192.168.8.255 255.255.255.255 192.168.8.11 192.168.8.11 20
224.0.0.0 240.0.0.0 192.168.8.11 192.168.8.11 20
255.255.255.255 255.255.255.255 192.168.8.11 192.168.8.11 1
Default Gateway: 192.168.8.254
===========================================================================
Persistent Routes:
None
(device1)c:\>tracert -d 10.58.129.254
Tracing route to 10.58.129.254 over a maximum of 30 hops
1 * * * Request timed out.
2 * * * Request timed out.
3 * * * Request timed out.
4 * * * Request timed out.
(etc. until 30 hops).
So from device1, running ping 10.58.129.254, and with tcpdump running on the 192.168.8.254 interface of the Linux router, I can see ICMP echo requests and replies
router# tcpdump -i eth4
17:08:08.326221 IP 192.168.8.11 > 10.58.129.254: ICMP echo request, id 512, seq 63746, length 40
17:08:08.326240 IP 10.58.129.254 > 192.168.8.11: ICMP echo reply, id 512, seq 63746, length 40
But the reply never returns to device1.
Does anybody know what the problem might be? tcpdump on eth2,3 & 4 also reveals the following output (I've not seen it on eth0, which is the one VLAN of the above that is routed by the core switch):
19:49:16.246286 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id
8000.a4:ba:db:69:74:91.8014, length 43
19:49:18.257007 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id
8000.a4:ba:db:69:74:91.8014, length 43
I understand this is spanning tree, but I don't know if this is a bad thing or not. Does this offer any clues? For info, the hardware address in the above STP message, is that of switch3.
In your second topology, you appear to have a split subnet. 192.168.8.0/24 spans multiple switches which you state are layer 3. In your output from switch2, you have a static route for this /24 pointing at a single interface:
This means that traffic hitting switch2 destined for either 192.168.8.254 or 192.168.8.11 will be forwarded to the same next hop. At least one of those destinations
In order for this to work as you intend, you have a couple of options:
Due to not being able to resolve this problem, a different approach is now being utilised which will vastly simplify the configuration. Instead of routing some VLANs by the core switch, and others by a Linux box, I'm now allowing the core switch to perform all routing between VLANs, with ACLs performing the segregation that two routers would have provided.
I'm going to move the Linux box to being the default gateway to the outside world for the entire network, with the help of iproute2, to perform source IP based routing, so the correct systems use the correct gateways.