Home / server / Questions / 320254
In Process
incognito2
Asked: 2011-10-11 10:42:07 +0800 CST

Fixing window scaling problems with Linux TCP

  • 772

I'm trying to improve the throughput of one of my servers overseas and after monitoring a transfer between the server and my home computer with wireshark I'm pretty sure I have a problem with the window size.

For an ftp transfer I get a receive window size of 14720.

Window size value: 115
Calculated window size: 14720
Window size scaling factor: 128

My send window looks like what I set it to:

Window size value: 65335
Calculated window size: 261340
Window size scaling factor: 4

So how do I fix the rwindow? I've gone through the linux tcp settings on my server and everything seems normal. Timestamps are on, syncookies are off, scaling is on, sacks are on, cubic is the congestion control method, max receive and send window size is 3mb. I've tried changing the default tcp_wmem and tcp_rmem values but it does nothing.

EDIT:

When I turn off autotuning and/or window scaling on the server the window shrinks to 14600, which is basically 10x the MSS. 

5337    4.268584    2.2.2.2 1.1.1.1 FTP 106 Response: 227 Entering Passive Mode (2,2,2,2,240,15).
5338    4.268640    1.1.1.1 2.2.2.2 TCP 74  59855 > 61455 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=431721460 TSecr=0 WS=128
5364    4.300368    1.1.1.1 2.2.2.2 TCP 54  57609 > ftp [ACK] Seq=217 Ack=648 Win=15744 Len=0
5480    4.346856    2.2.2.2 1.1.1.1 TCP 66  61455 > 59855 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1 WS=128
5481    4.346867    1.1.1.1 2.2.2.2 TCP 54  59855 > 61455 [ACK] Seq=1 Ack=1 Win=14720 Len=0
5482    4.346893    1.1.1.1 2.2.2.2 FTP 70  Request: STOR 100mb.bin
5570    4.428061    2.2.2.2 1.1.1.1 FTP 109 Response: 150 Opening BINARY mode data connection for 100mb.bin
5571    4.428078    1.1.1.1 2.2.2.2 TCP 54  57609 > ftp [ACK] Seq=233 Ack=703 Win=15744 Len=0
5572    4.428155    1.1.1.1 2.2.2.2 FTP-DATA    2974    FTP Data: 2920 bytes
5573    4.428166    1.1.1.1 2.2.2.2 FTP-DATA    1514    FTP Data: 1460 bytes
5662    4.505384    2.2.2.2 1.1.1.1 TCP 60  61455 > 59855 [ACK] Seq=1 Ack=1461 Win=8832 Len=0
5663    4.505392    1.1.1.1 2.2.2.2 FTP-DATA    2974    FTP Data: 2920 bytes
5664    4.505421    2.2.2.2 1.1.1.1 TCP 60  61455 > 59855 [ACK] Seq=1 Ack=2921 Win=11776 Len=0
5665    4.505429    1.1.1.1 2.2.2.2 FTP-DATA    2974    FTP Data: 2920 bytes
5666    4.505535    2.2.2.2 1.1.1.1 TCP 60  61455 > 59855 [ACK] Seq=1 Ack=4381 Win=14720 Len=0
5667    4.505543    1.1.1.1 2.2.2.2 FTP-DATA    2974    FTP Data: 2920 bytes
5734    4.583769    2.2.2.2 1.1.1.1 TCP 60  61455 > 59855 [ACK] Seq=1 Ack=5841 Win=17536 Len=0
5735    4.583778    1.1.1.1 2.2.2.2 FTP-DATA    2974    FTP Data: 2920 bytes
5736    4.583781    2.2.2.2 1.1.1.1 TCP 60  61455 > 59855 [ACK] Seq=1 Ack=7301 Win=20480 Len=0
5737    4.583787    1.1.1.1 2.2.2.2 FTP-DATA    2974    FTP Data: 2920 bytes
linux optimization tcp

2 Answers

  1. Starting with the 2.6.17 (or there abouts) linux kernel, the default scale factor was increased. The downside of this is that there appear to be routers/firewalls/etc. that do not correctly deal with TCP Window Scaling (the rfc is only ~16 years old). If you have to traverse one of these devices then you need to turn off the window scaling or else things get real slow.

    On a Redhat/Redhat like system the "fix" is to:

      /bin/cat <<'EOT'>>/etc/sysctl.conf

  # Turn off the tcp_window_scaling
  net.ipv4.tcp_window_scaling = 0
  EOT

  /sbin/sysctl -p
    • 4

  2. hmmm. Could you give us some more info ? Such as ...

    TCP releases {Reno, Vegas, etc} Trasfer direction { desktop -> server, server->desktop, something else} what are you using for the measurement ? iperf ? harpoon ?

    • 0