Home / server / Questions / 830447
Accepted
Volodymyr Melnyk
Asked: 2017-02-04 07:49:52 +0800 CST

An odd-looking TCP-session script

  • 772

I've faced a quite odd issue, which I'd like to share with you. Maybe you'd help me with some ideas on what's going on.

There are 3 virtual machines on a KVM-powered host. Actually there are about 50 VMs, but they all are working fine, although these 3 VMs behavior is kind a bit unusual.

Here's a diagram illustrating these 3 VMs and 2 links between them: enter image description here

When everything is OK, the TCP-session ("GET / HTTP/1.0" - "HTTP 200 OK") between these ones is looking like that:

00:58:43.885118 IP 192.168.111.2.55480 > 192.168.113.2.http: Flags [S], seq 926382744, win 14600, options [mss 1460,sackOK,TS val 277997 ecr 0,nop,wscale 7], length 0
00:58:43.885380 IP 192.168.113.2.http > 192.168.111.2.55480: Flags [S.], seq 1849545379, ack 926382745, win 14480, options [mss 1460,sackOK,TS val 3702103 ecr 277997,nop,wscale 7], length 0
00:58:43.885957 IP 192.168.111.2.55480 > 192.168.113.2.http: Flags [.], ack 1, win 115, options [nop,nop,TS val 277998 ecr 3702103], length 0
00:58:43.886000 IP 192.168.111.2.55480 > 192.168.113.2.http: Flags [P.], seq 1:213, ack 1, win 115, options [nop,nop,TS val 277998 ecr 3702103], length 212
00:58:43.886061 IP 192.168.113.2.http > 192.168.111.2.55480: Flags [.], ack 213, win 122, options [nop,nop,TS val 3702104 ecr 277998], length 0
00:58:43.922286 IP 192.168.113.2.http > 192.168.111.2.55480: Flags [P.], seq 1:372, ack 213, win 122, options [nop,nop,TS val 3702140 ecr 277998], length 371
00:58:43.922335 IP 192.168.113.2.http > 192.168.111.2.55480: Flags [F.], seq 372, ack 213, win 122, options [nop,nop,TS val 3702140 ecr 277998], length 0
00:58:43.923150 IP 192.168.111.2.55480 > 192.168.113.2.http: Flags [.], ack 372, win 123, options [nop,nop,TS val 278035 ecr 3702140], length 0
00:58:43.923622 IP 192.168.111.2.55480 > 192.168.113.2.http: Flags [F.], seq 213, ack 373, win 123, options [nop,nop,TS val 278036 ecr 3702140], length 0
00:58:43.923671 IP 192.168.113.2.http > 192.168.111.2.55480: Flags [.], ack 214, win 122, options [nop,nop,TS val 3702142 ecr 278036], length 0

OK, everything is fine so far.

Then we save the pfSense configuration, destroy this VM, create a new one, install pfSense from scratch and restore its configuration from the backup file.

And here's what we see after that:

00:46:39.218193 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [S], seq 3622924060, win 14600, options [mss 1460,sackOK,TS val 674608862 ecr 0,nop,wscale 7], length 0
00:46:39.218316 IP 192.168.113.2.http > 192.168.111.2.51674: Flags [S.], seq 152904245, ack 3622924061, win 14480, options [mss 1460,sackOK,TS va l 2977436 ecr 674608862,nop,wscale 7], length 0
00:46:39.218570 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [.], ack 1, win 115, options [nop,nop,TS val 674608862 ecr 2977436], length 0
00:46:40.417623 IP 192.168.113.2.http > 192.168.111.2.51674: Flags [S.], seq 152904245, ack 3622924061, win 14480, options [mss 1460,sackOK,TS val 2978636 ecr 674608862,nop,wscale 7], length 0
00:46:40.417947 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [.], ack 1, win 115, options [nop,nop,TS val 674610062 ecr 2977436], length 0
00:46:43.158907 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674612803 ecr 2977436], length 16
00:46:43.360103 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674613004 ecr 2977436], length 16
00:46:43.761787 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674613406 ecr 2977436], length 16
00:46:44.565890 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674614210 ecr 2977436], length 16
00:46:46.174039 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674615818 ecr 2977436], length 16
00:46:49.389921 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674619034 ecr 2977436], length 16
00:46:51.753723 IP 192.168.113.2.http > 192.168.111.2.51672: Flags [F.], seq 1, ack 1, win 114, options [nop,nop,TS val 2989972 ecr 674560137], length 0
00:46:55.821824 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674625466 ecr 2977436], length 16
00:46:57.221625 IP 192.168.113.2.http > 192.168.111.2.51672: Flags [F.], seq 1, ack 1, win 114, options [nop,nop,TS val 2995440 ecr 674560137], length 0
00:47:08.157575 IP 192.168.113.2.http > 192.168.111.2.51672: Flags [F.], seq 1, ack 1, win 114, options [nop,nop,TS val 3006376 ecr 674560137], length 0
00:47:08.685886 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674638330 ecr 2977436], length 16
00:47:30.029609 IP 192.168.113.2.http > 192.168.111.2.51672: Flags [F.], seq 1, ack 1, win 114, options [nop,nop,TS val 3028248 ecr 674560137], length 0
00:47:34.413785 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674664058 ecr 2977436], length 16
00:47:40.478757 IP 192.168.113.2.http > 192.168.111.2.51674: Flags [F.], seq 1, ack 1, win 114, options [nop,nop,TS val 3038697 ecr 674610062], length 0
00:47:34.413785 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674664058 ecr 2977436], length 16
00:47:40.478757 IP 192.168.113.2.http > 192.168.111.2.51674: Flags [F.], seq 1, ack 1, win 114, options [nop,nop,TS val 3038697 ecr 674610062], length 0
00:47:40.479216 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [FP.], seq 17:19, ack 2, win 115, options [nop,nop,TS val 674670123 ecr 3038697], length 2
00:47:45.946604 IP 192.168.113.2.http > 192.168.111.2.51674: Flags [F.], seq 1, ack 1, win 114, options [nop,nop,TS val 3044165 ecr 674610062], length 0
00:47:45.946979 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [.], ack 2, win 115, options [nop,nop,TS val 674675591 ecr 3044165,nop,nop,sack 1 {1:2}], length 0

It looks like... I don't know, like they don't hear each other. They can ping each other, they even interact with each other, but it look like they just ignore some of the packets.

Both VMs are showing the same, so pfSense doesn't drop any of their packets. Although it seems that something wrong happens to there packets. Like they've been mangled by something.

This is the thing I can't understand, really. If you share any ideas with me, it will be fantastic.

Thanks in advance to everyone!

networking virtual-machines tcpdump tcp pfsense

1 Answers

  1. Best Answer

    Does it work after you reboot the pfsense VM?

    According to their VirtIO docs here

    With the current state of VirtIO network drivers in FreeBSD, it is necessary to check the Disable hardware checksum offload box under System > Advanced on the Networking tab and to manually reboot pfSense after saving the setting, even though there is no prompt instructing to do so

    I assume if you're just loading the configuration, that does the same as checking the box, but it still needs the manual reboot.

    Your behavior also matches the cited bug. It's caused by the checksum for TCP packets not being calculated correctly causing the packet to appear corrupt to the receiver and dropped, which is why ICMP packets like ping work fine.

    • 1