Windows TCP Window Scaling Hitting plateau too early

Have you tried enabling Compound TCP (CTCP) in your Windows 7/8 clients.

Please read:

Increasing Sender-Side Performance for High-BDP Transmission

http://technet.microsoft.com/en-us/magazine/2007.01.cableguy.aspx

...

These algorithms work well for small BDPs and smaller receive window sizes. However, when you have a TCP connection with a large receive window size and a large BDP, such as replicating data between two servers located across a high-speed WAN link with a 100ms round-trip time, these algorithms do not increase the send window fast enough to fully utilize the bandwidth of the connection.

To better utilize the bandwidth of TCP connections in these situations, the Next Generation TCP/IP stack includes Compound TCP (CTCP). CTCP more aggressively increases the send window for connections with large receive window sizes and BDPs. CTCP attempts to maximize throughput on these types of connections by monitoring delay variations and losses. In addition, CTCP ensures that its behavior does not negatively impact other TCP connections.

...

CTCP is enabled by default in computers running Windows Server 2008 and disabled by default in computers running Windows Vista. You can enable CTCP with the netsh interface tcp set global congestionprovider=ctcp command. You can disable CTCP with the netsh interface tcp set global congestionprovider=none command.

Edit 6/30/2014

to see if CTCP is really "on"

> netsh int tcp show global

i.e.

PO said:

If I understand this correctly, this setting increases the rate at which the congestion window is enlarged rather than the maximum size it can reach

CTCP aggressively increases the send window

http://technet.microsoft.com/en-us/library/bb878127.aspx

Compound TCP

The existing algorithms that prevent a sending TCP peer from overwhelming the network are known as slow start and congestion avoidance. These algorithms increase the amount of segments that the sender can send, known as the send window, when initially sending data on the connection and when recovering from a lost segment. Slow start increases the send window by one full TCP segment for either each acknowledgement segment received (for TCP in Windows XP and Windows Server 2003) or for each segment acknowledged (for TCP in Windows Vista and Windows Server 2008). Congestion avoidance increases the send window by one full TCP segment for each full window of data that is acknowledged.

These algorithms work well for LAN media speeds and smaller TCP window sizes. However, when you have a TCP connection with a large receive window size and a large bandwidth-delay product (high bandwidth and high delay), such as replicating data between two servers located across a high-speed WAN link with a 100 ms round trip time, these algorithms do not increase the send window fast enough to fully utilize the bandwidth of the connection. For example, on a 1 Gigabit per second (Gbps) WAN link with a 100 ms round trip time (RTT), it can take up to an hour for the send window to initially increase to the large window size being advertised by the receiver and to recover when there are lost segments.

To better utilize the bandwidth of TCP connections in these situations, the Next Generation TCP/IP stack includes Compound TCP (CTCP). CTCP more aggressively increases the send window for connections with large receive window sizes and large bandwidth-delay products. CTCP attempts to maximize throughput on these types of connections by monitoring delay variations and losses. CTCP also ensures that its behavior does not negatively impact other TCP connections.

In testing performed internally at Microsoft, large file backup times were reduced by almost half for a 1 Gbps connection with a 50ms RTT. Connections with a larger bandwidth delay product can have even better performance. CTCP and Receive Window Auto-Tuning work together for increased link utilization and can result in substantial performance gains for large bandwidth-delay product connections.

Clarifying the Problem:

TCP has two windows:

• The receive window: How many bytes are left in the buffer. This is flow control imposed by the receiver. You can see the size of the receive window in the wireshark since it is made up of the window size and windowing scaling factor inside the TCP header. Both sides of the TCP connection will advertise their receive window, but generally the one you care about is the one receiving the bulk of the data. In your case, it is the "server" since the client is uploading to the server
• The congestion window. This is flow control imposed by the Sender. This is maintained by the operating system and does not show up in the TCP header. It controls the rate how fast data will be sent.

In the capture file you provided. We can see that the receive buffer is never overflowing:

My analysis is that the sender isn't sending fast enough because the send window (aka the congestion control window) isn't opening up enough to satisfy the RWIN of the receiver. So in short the receiver says "Give me More", and when Windows is the sender it isn't sending fast enough.

This is evidenced by the fact that in the above graph the RWIN stays open, and with the round trip time of .09 seconds and a RWIN of of ~ 500,000 bytes we can expect a max throughput according to the bandwidth delay product to be (500000/0.09) * 8 =~ 42 Mbit/s (and you are only getting about ~5 in your win to Linux capture).

How to Fix it?

I don't know. interface tcp set global congestionprovider=ctcp sounds like the right thing to do to me because it would increase the send window (which is another term for the congestion window). You said that is isn't working. So just to make sure:

1. Did you reboot after enabling this?
2. Is Chimney offload on? If it is maybe try turning it off as an experiment. I don't know what exactly gets offloaded when this enabled, but if controlling the send window is one of them, maybe congestionprovider has no effect when this is enabled... I'm just guessing...
3. Also, I think this may be pre windows 7, but you might try adding and playing with the two registry keys called DefaultSendWindow and DefaultReceiveWindow in HKEY_LOCAL_MACHINE-System-CurrentControlSet-Services-AFD-Parameters. If these even work, you probably beed it have ctcp off.
4. Yet another guess, try checking out netsh interface tcp show heuristics. I think that might be RWIN, but it doesn't say, so maybe play with disabling/enabling that in case it impacts the send window.
5. Also, make sure your drivers are up to date on your test client. Maybe something is just broken.

I would try all these experiments with all you offloading features off to start with to eliminate the possibility that the network drivers are doing some rewriting/modifying of things (keep an eye CPU while offloading is disabled). The TCP_OFFLOAD_STATE_DELEGATED struct seems to at least imply that CWnd offloading is at least possible.

There's been some great info here by @Pat and @Kyle. Definitely pay attention to @Kyle's explanation of the TCP receive and send windows, I think there has been some confusion around that. To confuse matters further, iperf uses the term "TCP window" with the -w setting which is kind of an ambiguous term with regards to the receive, send, or overall sliding window. What it actually does is set the socket send buffer for the -c (client) instance and the socket receive buffer on the -s (server) instance. In src/tcp_window_size.c:

if ( !inSend ) {
    /* receive buffer -- set
     * note: results are verified after connect() or listen(),
     * since some OS's don't show the corrected value until then. */
    newTCPWin = inTCPWin;
    rc = setsockopt( inSock, SOL_SOCKET, SO_RCVBUF,
                     (char*) &newTCPWin, sizeof( newTCPWin ));
} else {
    /* send buffer -- set
     * note: results are verified after connect() or listen(),
     * since some OS's don't show the corrected value until then. */
    newTCPWin = inTCPWin;
    rc = setsockopt( inSock, SOL_SOCKET, SO_SNDBUF,
                     (char*) &newTCPWin, sizeof( newTCPWin ));
}

As Kyle mentions, the issue isn't with the receive window on the Linux box, but the sender isn't opening up the send window enough. It's not that it doesn't open up fast enough, it just caps at 64k.

The default socket buffer size on Windows 7 is 64k. Here's what the documentation says about the socket buffer size in relation to throughput at MSDN

When sending data over a TCP connection using Windows sockets, it is important to keep a sufficient amount of data outstanding (sent but not acknowledged yet) in TCP in order to achieve the highest throughput. The ideal value for the amount of data outstanding to achieve the best throughput for the TCP connection is called the ideal send backlog (ISB) size. The ISB value is a function of the bandwidth-delay product of the TCP connection and the receiver's advertised receive window (and partly the amount of congestion in the network).

Ok, blah blah blah, Now here we go:

Applications that perform one blocking or non-blocking send request at a time typically rely on internal send buffering by Winsock to achieve decent throughput. The send buffer limit for a given connection is controlled by the SO_SNDBUF socket option. For the blocking and non-blocking send method, the send buffer limit determines how much data is kept outstanding in TCP. If the ISB value for the connection is larger than the send buffer limit, then the throughput achieved on the connection will not be optimal.

The average throughput of your most recent iperf test using the 64k window is 5.8Mbps. That's from Statistics > Summary in Wireshark, which counts all the bits. Likely, iperf is counting TCP data throughput which is 5.7Mbps. We see the same performance with the FTP test as well, ~5.6Mbps.

The theoretical throughput with a 64k send buffer and 91ms RTT is....5.5Mbps. Close enough for me.

If we look at your 1MB window iperf test, the tput is 88.2Mbps (86.2Mbps for just TCP data). The theoretical tput with a 1MB window is 87.9Mbps. Again, close enough for government work.

What this demonstrates is that the send socket buffer directly controls the send window and that, coupled with the receive window from the other side, controls throughput. The advertised receive window has room, so we're not limited by the receiver.

Hold up, what about this autotuning business? Doesn't Windows 7 handle that stuff automatically? As has been mentioned, Windows does handle auto-scaling of the receive window, but it can also dynamically handle the send buffer too. Let's go back to the MSDN page:

Dynamic send buffering for TCP was added on Windows 7 and Windows Server 2008 R2. By default, dynamic send buffering for TCP is enabled unless an application sets the SO_SNDBUF socket option on the stream socket.

iperf uses SO_SNDBUF when using the -w option, so dynamic send buffering would be disabled. However, if you don't use -w then it doesn't use SO_SNDBUF. Dynamic send buffering should be on by default, but you can check:

netsh winsock show autotuning

The documentation says you can disable it with:

netsh winsock set autotuning off

But that didn't work for me. I had to make a registry change and set this to 0:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AFD\Parameters\DynamicSendBufferDisable

I don't think disabling this will help; it's just an FYI.

Why isn't your send buffer scaling above the default 64k when sending data to a Linux box with plenty of room in the receive window? Great question. Linux kernels also have an autotuning TCP stack. Like T-Pain and Kanye doing an autotune duet together, it just might not sound good. Perhaps there's some issue with those two autotuning TCP stacks talking to each other.

Another person had a problem just like yours and was able to fix it with a registry edit to increase the default send buffer size. Unfortunately, that doesn't seem to work anymore, at least it didn't for me when I tried it.

At this point, I think it's clear the limiting factor is the send buffer size on the Windows host. Given that it doesn't seem to be dynamically growing properly, what's a girl to do?

You can:

• Use applications that allow you to set the send buffer i.e. window option
• Use a local Linux proxy
• Use a remote Windows proxy?
• Open a case with Microsofhahahahahahaha
• Beer

Disclaimer: I have spent many many hours researching this and it is correct to the best of my knowledge and google-fu. But I wouldn't swear on my mother's grave (she's still alive).

Once you have the TCP stack tuned, you might still have a bottleneck in the Winsock layer. I have found that configuring Winsock (Ancillary Function Driver in the registry) makes a huge difference for upload speeds (pushing data to the server) in Windows 7. Microsoft has acknowledged a bug in the TCP autotuning for non-blocking sockets -- just the kind of socket that browsers use ;-)

Add DWORD key for DefaultSendWindow and set it to the BDP or higher. I am using 256000.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\AFD\Parameters\DefaultSendWindow

Changing the Winsock setting for downloads might help - add a key for DefaultReceiveWindow.

You can experiment with various socket level settings by using the Fiddler Proxy and commands to adjust the client and server socket buffer sizes:

prefs set fiddler.network.sockets.Server_SO_SNDBUF 65536 

fiddler.network.sockets.Client_SO_SNDBUF
fiddler.network.sockets.Client_SO_RCVBUF
fiddler.network.sockets.Server_SO_SNDBUF
fiddler.network.sockets.Server_SO_RCVBUF

Having read all the analysis in the answers, this problem very much sounds like you might be running Windows7/2008R2 aka Windows 6.1

The networking stack (TCP/IP & Winsock) in Windows 6.1 was horribly flawed and had a whole host of bugs and performance problems which Microsoft eventually addressed over many years of hotfixing since the initial release of 6.1.

The best way to apply these hotfixes is to manually sift through all the relevant pages on support.microsoft.com and manually request and download the LDR versions of the network stack hotfixes (there are many dozens of these).

To find the relevant hotfixes, you must use www.bing.com with the following search query site:support.microsoft.com 6.1.7601 tcpip.sys

You also need to understand how LDR/GDR hotfix trains work in Windows 6.1

I generally used to maintain my own list of LDR fixes (not just network stack fixes) for Windows 6.1 and then proactively applied these fixes to any Windows 6.1 server/client that I came across. It was a very time-consuming task to regularly check for new LDR hotfixes.

Luckily, Microsoft has stopped the practice of LDR hotfixes with newer OS versions and bugfixes are now available through automatic update services from Microsoft.

UPDATE: Just one example of many network bugs in Windows7SP1 - https://support.microsoft.com/en-us/kb/2675785

UPDATE 2: Here's another hotfix that adds a netsh switch to force Window scaling after the second retransmission of a SYN packet (by default, window scaling is disabled after 2 SYN packets are retransmitted) https://support.microsoft.com/en-us/kb/2780879

I see this is a little bit older post but it could help others.

In short you have to enable "Receive Window Auto-Tuning":

netsh int tcp set global autotuninglevel=normal

CTCP means nothing without above enabled.

If you disable "Receive Window Auto-Tuning" you will be stuck at 64KB packet size which has negative impact over long RTT's in high broadband connections. You can also experiment with "restricted" and "highlyrestricted" option.

Very good reference: https://www.duckware.com/blog/how-windows-is-killing-internet-download-speeds/index.html

I was experiencing a similar issue with Windows Clients (Windows 7). I went through most the debugging you have gone through, disabling Nagle algorithm, TCP Chimney Offloading, and tons of other TCP related setting changes. Non of them had any effect.

What finally fixed it for me was modifying the default send window in the registry of the AFD service. The issue seems to be related to the afd.sys file. I tested several clients, some exhibited the slow upload, and some didn't, but all were Windows 7 machines. Machines that exhibited the slow behavior had the same version AFD.sys. The registry workaround is needed for computers with certain versions of AFD.sys (sorry, do no recall the version #'s).

HKLM\CurrentControlSet\Services\AFD\Parameters

Add - DWORD - DefaultSendWindow

Value - Decimal - 1640960

That value is was something I found here: https://helpdesk.egnyte.com/hc/en-us/articles/201638254-Upload-Speed-Slow-over-WebDAV-Windows-

I think to use the proper value, you should calculate it yourself using:

eg. Advertised Upload: 15 Mbps = 15,000 Kbps

(15000 / 8 ) * 1024 = 1920000

From what I understand, client software should generally override this setting in the registry, but if they don't, the default value is used, and apparently the default value is very low in some versions of the AFD.sys file.

I noticed that mostly MS products had the slow upload problem (IE, Mini-redirector(WebDAV), FTP through Windows Explorer, etc...) When using 3rd party software (ex. Filezilla) I did not have the same slow-downs.

The AFD.sys effects all Winsock connections, so this fix should apply to FTP, HTTP, HTTPS, etc...

Also, this fix was listed above somewhere as well, so I don't want to take credit for it if it works for anybody, however there was so much information in this thread that I was afraid it might have been glossed over.

Well, I've run into a similar situation myself (my question here), and in the end I had to disable TCP scaling heuristics, manually set the autotuning profile and enable CTCP:

# disable heuristics
C:\Windows\system32>netsh interface tcp set heuristics wsh=disabled
Ok.

# enable receive-side scaling
C:\Windows\system32>netsh int tcp set global rss=enabled
Ok.

# manually set autotuning profile
C:\Windows\system32>netsh interface tcp set global autotuning=experimental
Ok. 

# set congestion provider
C:\Windows\system32>netsh interface tcp set global congestionprovider=ctcp
Ok. 

I don't have enough points to comment, so I'll post an "answer" instead. I'm having what appears to be a similar/identical problem (see serverfault question here). My (and probably your) problem is the send buffer of the iperf client on windows. It does not grow beyond 64 KB. Windows is supposed to dynamically grow the buffer when it is not explicitly sized by the process. But that dynamic growth is not happening.

I'm not sure about your window scaling graph that shows the window opening up to 500,000 bytes for your "slow" Windows case. I expected to see that graph open to only ~64,000 bytes given that you are limited to 5 Mbps.

This is a fascinating thread and exactly matches issues I've had using Win7/iperf to test throughput on long fat pipes.

The solution for Windows 7 is to execute the following command on both the iperf server AND the client.

netsh interface tcp set global autotuninglevel=experimental

NB: Before you do this, be sure to record the current status of autotuning:

netsh interface tcp show global

Receive Window Auto-Tuning Level : disabled

Then run the iperf server/client at each end of the pipe.

Reset the autotuning value following your tests:

netsh interface tcp set global autotuninglevel=

   autotuninglevel - One of the following values:
                     disabled: Fix the receive window at its default
                         value.
                     highlyrestricted: Allow the receive window to
                         grow beyond its default value, but do so
                         very conservatively.
                     restricted: Allow the receive window to grow
                         beyond its default value, but limit such
                         growth in some scenarios.
                     normal: Allow the receive window to grow to
                         accomodate almost all scenarios.
                     experimental: Allow the receive window to grow
                         to accomodate extreme scenarios.