I have a server configured with multiple interfaces and multiple VLANs. It works perfectly fine for all the local networks, it drops packets forwarded through my router for some reason though. And it's not even consistent. Sometimes I can get it working for a couple days before it starts dropping packets again. I would love to keep digging but the only things but the only results I can get out of Google are people who need help setting up iptables.
Configurations
$ cat /etc/issue
Ubuntu 18.04.4 LTS \n \l
$ cat /etc/netplan/50-cloud-init.yaml
network:
version: 2
ethernets:
enp10s0:
dhcp4: true
dhcp6: true
enp6s0:
dhcp4: false
dhcp6: false
vlans:
vlan18:
id: 18
link: enp6s0
dhcp4: true
optional: true
vlan150:
id: 150
link: enp6s0
dhcp4: true
optional: true
vlan155:
id: 155
link: enp6s0
dhcp4: true
optional: true
The interface in question is enp10s0. I had it on enp6s0 in the VLANs for a while but moved it to a seperate NIC to isolate variables. That didn't change anything.
$ netstat -s enp10s0
Ip:
Forwarding: 2
4207683 total packets received
11 with invalid addresses
0 forwarded
0 incoming packets discarded
4197424 incoming packets delivered
2183348 requests sent out
21 outgoing packets dropped
Tcp:
1634 active connection openings
1615 passive connection openings
150 failed connection attempts
1100 connection resets received
43 connections established
4207863 segments received
2190261 segments sent out
596 segments retransmitted
0 bad segments received
222 resets sent
Testing
Setup
I add the following first line to my iptables INPUT chain:
-p tcp -m tcp --dport 22 -j LOG --log-prefix "IPTABLES SEEN: "
I watch traffic using tcpdump:
tcpdump -n -e -vv -i enp10s0 port 22
Step 1: Prove it works locally
From my router 10.8.10.1 telnet to the server in question 10.8.10.11 port 22.
iptables log:
Jul 15 23:58:04 meji kernel: IPTABLES SEEN: IN=enp10s0 OUT= MAC=60:a4:4c:60:ce:ce:e0:63:da:21:c1:a5:08:00 SRC=10.8.10.1 DST=10.8.10.11 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=44677 DF PROTO=TCP SPT=48770 DPT=22 WINDOW=14600 RES=0x00 SYN URGP=0
tcpdump log:
23:58:04.335447 e0:63:da:21:c1:a5 > 60:a4:4c:60:ce:ce, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 44677, offset 0, flags [DF], proto TCP (6), length 60)
10.8.10.1.48770 > 10.8.10.11.22: Flags [S], cksum 0xbb2d (correct), seq 978415077, win 14600, options [mss 1460,sackOK,TS val 25150304 ecr 0,nop,wscale 7], length 0
SYN ACK follows like you would expect and everything is fine.
Step 2: compare to the router forwarded connection
I use nc -vz from my remote server (34.73.148.195) to connect to the same ip/port.
tcpdump log:
00:54:44.427670 e0:63:da:21:c1:a5 > 60:a4:4c:60:ce:ce, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 56, id 18829, offset 0, flags [DF], proto TCP (6), length 60)
34.73.148.195.50176 > 10.8.10.11.22: Flags [S], cksum 0x8c20 (correct), seq 1566819019, win 65320, options [mss 1420,sackOK,TS val 1249821436 ecr 0,nop,wscale 6], length 0
iptables log:
nothing. nothing is logged.
No SYN ACK and a retransmit attempt comes through a moment later. The NIC is not reporting errors, iptables sees nothing, and I am left scratching my head. Where can I even look from here? Start digging in the kernel? The network drivers?
Additional Requested Information
$ ip route show
default via 10.8.8.1 dev vlan18 proto dhcp src 10.8.8.11 metric 100
default via 10.8.50.1 dev vlan150 proto dhcp src 10.8.50.5 metric 100
10.8.8.0/24 dev vlan18 proto kernel scope link src 10.8.8.11
10.8.8.1 dev vlan18 proto dhcp scope link src 10.8.8.11 metric 100
10.8.10.0/24 dev enp10s0 proto kernel scope link src 10.8.10.11
10.8.50.0/24 dev vlan150 proto kernel scope link src 10.8.50.5
10.8.50.1 dev vlan150 proto dhcp scope link src 10.8.50.5 metric 100
10.8.55.0/24 dev vlan155 proto kernel scope link src 10.8.55.5
More iptables stuff. But when I clear out all rules and change all policies to ACCEPT I still have the same problem. I'm confident I've eliminated iptables rules as the culprit.
# iptables -vL
Chain INPUT (policy DROP 442K packets, 81M bytes)
pkts bytes target prot opt in out source destination
348M 494G ACCEPT all -- any any anywhere anywhere state RELATED,ESTABLISHED
1692K 301M ACCEPT all -- lo any anywhere anywhere /* Loopback Interface */
327K 24M ACCEPT all -- vlan18 any anywhere anywhere
174K 14M ACCEPT all -- vlan155 any anywhere anywhere
0 0 ACCEPT tcp -- enp10s0 any anywhere anywhere tcp dpt:ssh state NEW,ESTABLISHED /* Ssh Passthrough */
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 190M packets, 17G bytes)
pkts bytes target prot opt in out source destination
# iptables --list --table raw
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
# iptables --list --table mangle
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
# iptables --list --table nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source
