Home / server / Questions / 997968
In Process
iBug
Asked: 2020-01-08 08:28:09 +0800 CST

Linux packets going to strange interfaces when NATing between bridges

  • 772

I have a desktop computer (Ubuntu 19.10) running containers and VMs. Its ethernet enp2s0 is bridged with systemd-networkd:

[Match]
Name=enp2s0

[Network]
Bridge=bridge
LinkLocalAddressing=no

The bridge interface is named exactly bridge and has IP addresses when shown with ip a, for example, 192.168.1.2/24. It also has a wireless interface wlp3s0 with IP address 192.168.5.2/24 and an OpenVPN interface ovpn at 10.0.1.2/24.

My containers and VMs are running under an internal bridge virbr0 where the host has the address 172.17.0.1/16. The strange thing is, no container or VM can reach the network outside host interface bridge, but all of them can reach anything from host interfaces wlp3s0 (e.g. 192.168.5.1) and ovpn (e.g. 10.0.1.1).

I tried pinging different locations from a VM and watching packets on the host.

root@iBug-Server:~# tcpdump -leni bridge icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bridge, link-type EN10MB (Ethernet), capture size 262144 bytes
00:05:36.068658 86:80:93:9a:83:87 > d8:67:d9:70:e9:41, ethertype IPv4 (0x0800), length 98: 192.168.1.2 > 192.168.1.1: ICMP echo request, id 6227, seq 1, length 64
00:05:36.068902 d8:67:d9:70:e9:41 > 86:80:93:9a:83:87, ethertype IPv4 (0x0800), length 98: 192.168.1.1 > 192.168.1.2: ICMP echo reply, id 6227, seq 1, length 64
00:05:36.068930 86:80:93:9a:83:87 > d8:67:d9:70:e9:41, ethertype IPv4 (0x0800), length 98: 192.168.1.1 > 172.17.2.120: ICMP echo reply, id 6227, seq 1, length 64
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel
root@iBug-Server:~# tcpdump -leni wlp3s0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wlp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
00:06:29.306316 12:04:21:2d:39:18 > 74:f8:db:6a:3e:d1, ethertype IPv4 (0x0800), length 98: 192.168.5.2 > 192.168.5.1: ICMP echo request, id 6231, seq 1, length 64
00:06:29.317283 74:f8:db:6a:3e:d1 > 12:04:21:2d:39:18, ethertype IPv4 (0x0800), length 98: 192.168.5.1 > 192.168.5.2: ICMP echo reply, id 6231, seq 1, length 64
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel
root@iBug-Server:~# tcpdump -leni enp2s0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
00:09:12.098863 86:80:93:9a:83:87 > d8:67:d9:70:e9:41, ethertype IPv4 (0x0800), length 98: 192.168.1.2 > 192.168.1.1: ICMP echo request, id 6232, seq 1, length 64
00:09:12.099100 d8:67:d9:70:e9:41 > 86:80:93:9a:83:87, ethertype IPv4 (0x0800), length 98: 192.168.1.1 > 192.168.1.2: ICMP echo reply, id 6232, seq 1, length 64
00:09:12.099145 86:80:93:9a:83:87 > d8:67:d9:70:e9:41, ethertype IPv4 (0x0800), length 98: 192.168.1.1 > 172.17.2.120: ICMP echo reply, id 6232, seq 1, length 64
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel

As can be seen above, the 3rd packet from bridge or enp2s0 appears to be going to the wrong place (it should have been to virbr0) and also having wrong MAC addresses. Here are the results from virbr0. Note how the ICMP reply didn't appear when pinging 192.168.1.1 (gateway of host's bridge interface).

root@iBug-Server:~# tcpdump -leni virbr0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on virbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
00:21:49.815124 00:16:3e:03:1a:f8 > 52:b4:4b:3f:45:e2, ethertype IPv4 (0x0800), length 98: 172.17.2.120 > 192.168.1.1: ICMP echo request, id 6238, seq 1, length 64
^C
1 packet captured
1 packet received by filter
0 packets dropped by kernel
root@iBug-Server:~# tcpdump -leni virbr0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on virbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
00:21:59.268210 00:16:3e:03:1a:f8 > 52:b4:4b:3f:45:e2, ethertype IPv4 (0x0800), length 98: 172.17.2.120 > 192.168.5.1: ICMP echo request, id 6239, seq 1, length 64
00:21:59.277884 52:b4:4b:3f:45:e2 > 00:16:3e:03:1a:f8, ethertype IPv4 (0x0800), length 98: 192.168.5.1 > 172.17.2.120: ICMP echo reply, id 6239, seq 1, length 64
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel

My iptables has an empty FORWARD chain with policy ACCEPT, and I didn't touch the nat table after setting up KVM (with libvirt). What's going wrong here and how do I fix it?

linux

1 Answers

  1. So stupid of me!

    I wanted to make sure packets coming from the ethernet interface go the same way back, so I had these lines among my startup scripts:

    ip rule add dev bridge table 1
ip route add table 1 default via 192.168.1.1 dev bridge

    This is causing the NAT'd ICMP reply packet going back to the bridge interface.

    To solve this, copy everything necessary from the main table:

    ip route add table 1 192.168.5.0/24 dev wlp3s0
ip route add table 1 172.17.0.0/16 dev virbr0
... and so on
    • 0