Questions[tcp-window-scaling](server)

During the 3-way handshake, sender (10.16.0.128) and recipient (10.16.0.5) advertised their window size of 42340 and 65535 respectively.

From the 3rd packet (syn+ack+ack) onwards sender again advertised widow size as 166, while receiver advertised 512.

Can you please let me know why the window size was reduced without sending any segment? Does it keep changing on every ack? Why they didn't commit window size during negotiations.

    10.16.0.128.59570 > 10.16.0.5.22: Flags [S], cksum 0x7da0 (incorrect -> 0xf0a5), seq 2355031382, win 42340, options [mss 1460,sackOK,TS val 4284732064 ecr 0,nop,wscale 8], length 0

09:50:58.661172 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.16.0.5.22 > 10.16.0.128.59570: Flags [S.], cksum 0x2d67 (correct), seq 1148159519, ack 2355031383, win 65535, options [mss 1460,sackOK,TS val 216174882 ecr 4284732064,nop,wscale 7], length 0

09:50:58.661229 IP (tos 0x0, ttl 64, id 49621, offset 0, flags [DF], proto TCP (6), length 52)
    10.16.0.128.59570 > 10.16.0.5.22: Flags [.], cksum 0x7d98 (incorrect -> 0x5b8c), seq 1, ack 1, win 166, options [nop,nop,TS val 4284732065 ecr 216174882], length 0

09:50:58.661455 IP (tos 0x0, ttl 64, id 49622, offset 0, flags [DF], proto TCP (6), length 98)
    10.16.0.128.59570 > 10.16.0.5.22: Flags [P.], cksum 0x7dc6 (incorrect -> 0x5b5f), seq 1:47, ack 1, win 166, options [nop,nop,TS val 4284732065 ecr 216174882], length 46

09:50:58.661543 IP (tos 0x0, ttl 64, id 8746, offset 0, flags [DF], proto TCP (6), length 52)
    10.16.0.5.22 > 10.16.0.128.59570: Flags [.], cksum 0x5a03 (correct), seq 1, ack 47, win 512, options [nop,nop,TS val 216174883 ecr 4284732065], length 0

09:50:58.661543 IP (tos 0x0, ttl 64, id 8746, offset 0, flags [DF], proto TCP (6), length 52)
    10.16.0.5.22 > 10.16.0.128.59570: Flags [.], cksum 0x5a03 (correct), seq 1, ack 47, win 512, options [nop,nop,TS val 216174883 ecr 4284732065], length 0```

The TCP window size is the amount of data that the sender will send before waiting for a TCP ack. Does the receiver have a way to control this (e.g., as part of the TCP handshake), or is it only the sender who can control?

We are having some issues with several clients running Windows 10 downloading files from our HTTP file delivery server.

We can't reproduce the error on our end but notice that TCP window scaling is not "ramping up" and throughput between the Linux server and Windows 10 client remains low.

We have run a TCPDUMP on the server capturing packets from the client and can see the small window size:

Example:

00:00:00.321292 IP 85.144.19.44.58799 > 195.224.144.66.80: Flags [.], ack 554881, win 1025, options [nop,nop,sack 1 {556241:564401}], length 0
 00:00:00.321348 IP 85.144.19.44.58799 > 195.224.144.66.80: Flags [.], ack 554881, win 1025, options [nop,nop,sack 1 {556241:565761}], length 0
 00:00:00.321359 IP 195.224.144.66.80 > 85.144.19.44.58799: Flags [.], seq 590241:591601, ack 702, win 30, length 1360: HTTP
 00:00:00.322417 IP 85.144.19.44.58799 > 195.224.144.66.80: Flags [.], ack 554881, win 1025, options [nop,nop,sack 1 {556241:572561}], length 0
 00:00:00.322482 IP 85.144.19.44.58799 > 195.224.144.66.80: Flags [.], ack 554881, win 1025, options [nop,nop,sack 1 {556241:573921}], length 0
 00:00:00.322497 IP 195.224.144.66.80 > 85.144.19.44.58799: Flags [.], seq 595681:597041, ack 702, win 30, length 1360: HTTP
 00:00:00.322528 IP 85.144.19.44.58799 > 195.224.144.66.80: Flags [.], ack 554881, win 1025, options [nop,nop,sack 1 {556241:575281}], length 0
 00:00:00.322559 IP 195.224.144.66.80 > 85.144.19.44.58799: Flags [.], seq 597041:598401, ack 702, win 30, length 1360: HTTP
 00:00:00.322570 IP 85.144.19.44.58799 > 195.224.144.66.80: Flags [.], ack 554881, win 1025, options [nop,nop,sack 1 {556241:576641}], length 0
 00:00:00.322983 IP 85.144.19.44.58799 > 195.224.144.66.80: Flags [.], ack 554881, win 1025, options [nop,nop,sack 1 {556241:578001}], length 0
 00:00:00.323007 IP 195.224.144.66.80 > 85.144.19.44.58799: Flags [.], seq 598401:599761, ack 702, win 30, length 1360: HTTP
 00:00:00.323015 IP 85.144.19.44.58799 > 195.224.144.66.80: Flags [.], ack 554881, win 1025, options [nop,nop,sack 1 {556241:579361}], length 0
 00:00:00.323026 IP 195.224.144.66.80 > 85.144.19.44.58799: Flags [.], seq 599761:601121, ack 702, win 30, length 1360: HTTP
 00:00:00.323268 IP 85.144.19.44.58799 > 195.224.144.66.80: Flags [.], ack 587521, win 1025, length 0
 00:00:00.332204 IP 85.144.19.44.58799 > 195.224.144.66.80: Flags [.], ack 591601, win 1025, length 0
 00:00:00.332242 IP 195.224.144.66.80 > 85.144.19.44.58799: Flags [.], seq 606561:609281, ack 702, win 30, length 2720: HTTP

Notice the small window size (1025).

We have tested the Windows 10 client throughput using another server on our network (macOS) and can get full throughput along with increased scaling window sizes so the problem seems to be this specific to this Linux server.

We have made some changes to the network stack on this particular Linux server which we include in sysctl.conf:

kernel.panic = 3
net.core.somaxconn = 1024
net.ipv4.tcp_ecn = 0
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_mtu_probing = 1
vm.swappiness = 1
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.route.flush = 1
net.ipv6.route.flush = 1

Does anyone have any ideas as to why Windows 10 won't negotiate better TCP window sizes with our Linux machine? It's not an issue of middleboxes as stated above, we can achieve full throughput by testing with a different server on our same delivery network. It's just this particular machine.

Any thoughts, feedback, tips, and advice are greatly appreciated in advance.

Many thanks.

I am currently dealing with a networking problem on high latency (100-400ms) Internet links. I run a Minecraft network, and I recently moved it to a separate datacenter to get a server with a better CPU and more ram. The users of this server are spread all over the world. Before the switch, the server was in Montreal and the users in Europe got ~100-200ms latency, and those in Australia got ~200-300ms latency. After the switch, the server is in Germany, the users in North America get ~100-200ms latency, and the users in South America and Australia get 200-400ms latency. Overall, the latency is quite similar, but who gets great latency and who gets tolerable latency varies (note that Minecraft isn't very latency sensitive in general, especially compared to most video games). There also isn't any significant packet loss, as measured by the MTR and ping tools. Furthermore, the software on both servers is almost identical. Both servers run Debian 10, and I tar'd all software + configuration that's not in APT repositories to send that over, while reinstalling the exact same packages via apt. As such, the software configuration should be essentially identical.

Yet, many users are having connection trouble. It seems to only occur around 6:00 PM (± a few hours) in the eastern United States. The connection trouble specifically takes the nature of the throughput of all TCP connections being atrociously low. With an SSH+SOCKS proxy, it took minutes to load a normal web page (Gmail), and in-game, it often takes minutes for even a simple chat message to get through if a few MBs of world data are currently being transferred. The effective latency of a TCP connection (e.g. the time it takes a chat message to get through it) increases unreasonably massively when any data is put over that TCP connection. A normal SSH session with just a terminal is basically fine, and the game is mostly fine if not much is going on, but as soon as anything of significant size gets sent over TCP and it is during the aforementioned time, then the throughput breaks down and latency over TCP (but not via ping) becomes unreasonable, even to multiple minutes in the worst cases. When this issue first took place, there was significant packet loss (~25%) which I thought was to blame, but, that packet loss is no longer occurring (according to ping, etc.) yet the issue remains. The packet loss, but not the problematic symptoms, disappeared after I made a general report to the new host about packet loss, but before I could give them more detailed data with MTR as they requested in their response to that report. My impression from the host is that they did not change anything, but, who knows really.

As such, at this point, I suspect that the relevant difference between the servers is that the old host (OVH) does some sort of tuning of their OS images (something that I know to be the case), and that the new host (Hetnzer) does not.

I suspect that this tuning has something to do with TCP window size, but when I attempted to manipulate those settings to make changes, the settings didn't seem to do what they're supposed to. Specifically, when I set the various net.ipv4.mem or net.core.mem settings that I find listed on the Internet via sysctl, the window size that iperf selects (or the maximum that it is allowed to select, when using the -w option) seems to take a random value with no discernible-to-me relation to the values that I set via sysctl, as opposed to behaving in the way that I expect, where it's maximum value is simply whatever I set via sysctl. Note that iperf -s misbehaves even before a client connects to it, so failing to make the same changes on the client is not a plausible explanation.

As such, I'm wondering 2 things:

1) How can I fix my server, and allow the latency in TCP connections to be similar to the actual latency on the link, even at peak times and under moderate load (a few mbps)?

2) How can I reliably and predictably change the TCP window size of all applications? (or, equivalently, what's going on with the sysctl settings applying in seemingly random ways? / what is the pattern that I'm missing?)

I have a very odd issue with our cisco firewall,

If on our osx machines we run the following to disable window scaling support:

sudo sysctl -w net.inet.tcp.rfc1323=0

We see near 40% improvement in download speeds (tested using varying linux iso image mirrors).

If I sit my machine directly onto the outside modem (Manually setting up an external ip adress within the range etc), the bandwidth remains constant with rfc1323 enabled and disabled, so it has to be the cisco.

Something else to note:

• None of our linux devices suffer this problem

Has anyone come across this issue before? I seem to find some vague references in old mailing lists detailing PIX workarounds which are no longer applicable to cisco ios.

Thanks in advance.