Home / server / Questions / 1008179
In Process
Matt Clark
Asked: 2020-03-25 07:03:49 +0800 CST

CentOS7 packets disappear after PREROUTING

  • 772

Packets are disappearing while trying to get them forwarded to libvirt machines.

The incoming requests hit my interface / public IP

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  165  8580 DNAT       tcp  --  em4    *       0.0.0.0/0            50.x.y.z            tcp dpt:80 to:192.168.122.100
...

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all  --  *      *       192.168.122.0/24    !192.168.122.0/24
    0     0 MASQUERADE  all  --  *      em4     0.0.0.0/0            0.0.0.0/0
...

And I have the rules on both FORWARD and OUTPUT to match, both with no hits.

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            192.168.122.100      tcp dpt:80
...

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            192.168.122.100      tcp dpt:80
...

Forwarding is enabled for all interfaces:

net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.em4.forwarding = 1
net.ipv4.conf.virbr0.forwarding = 1
net.ipv4.conf.vnet5.forwarding = 1

I believe routing is correct, I can hit the endpoint properly from my host

# ip route get 192.168.122.100
192.168.122.100 dev virbr0 src 192.168.122.1 uid 0
    cache

# curl --head http://192.168.122.100/
HTTP/1.1 302 Found
...

I have complete rule chains, including all default libvirt rules. The chains all end with LOG / DROP at the end, but nothing related ever hits those. I have been using tcpdump across all interfaces, I see the syn packets from the request which are the packets seen in the initial PREROUTING rule, but then nothing.

# uname -r
5.5.10-1.el7.elrepo.x86_64

Where else could these packets be going after the PREROUTING rule? Is there something I've missed?

edit

Gerrit led me to look at the rp_filter which appears to be dropping my packets. I do have MASQUERADE rules, but it looks like they might be wrong.

kernel: IPv4: martian source 192.168.122.100 from 24.xx.yy.zz, on dev em4
kernel: ll header: 00000000: 78 2b cb 34 d5 70 98 52 4a 60 53 5d 08 00        x+.4.p.RJ`S]..
routing iptables

1 Answers

  1. The incoming packets are dropped by the rp_filter on em4 because of the way the routing is split over multiple interface aliases on this interface.

    Usually with a DNAT rule, the reply packet NAT'ing should automatically be reversed on its way out so that the answer would seem to come from the interface where the DNAT was activated. This does not seem to function here, and the rp_filter indicates it sees the unmangled source address in the reply packet (This turns out to be caused by the DNAT, it confuses the kernel message).

    Or the actual incoming packet IS a martian somehow, and your system has a different route to it then the interface where it comes in, and it is the kernel log message that is confusing because of the DNAT (There is indication that this message indeed shows the wrong address being called a martian when DNAT is involved).

    Basically if ip route get address/32 is not equal to the incoming interface then the packet is a Martian! You can also use loose rp filtering (setting of 2) to only filter if the address is not anywhere on the machine. Or you can set static routes yourself to unmartian those martians.

    In iptables you cannot use interface aliases like em0:2 or something. It will only accept the main device.

    • 1