Home / server / Questions / 365975
Accepted
Quintin Par
Asked: 2012-03-04 04:25:56 +0800 CST

Linux Slow Start: Changing ip route does not have any effect on initial window

  • 772

I changed the tcp initial window in my machine to 10 as shown below 

[user@site etc]$ sudo ip route change default via 17.255.209.1 dev eth0  proto static initcwnd 10

And changed tcp_slow_start_after_idle as shown below 

[user@site etc]$ sudo sysctl -a | grep tcp_slow_start_after_idle
net.ipv4.tcp_slow_start_after_idle = 0

a ip route show confirmation is given below 

[user@site etc]$ ip route show
default via 17.255.209.1 dev eth0  proto static  initcwnd 10
169.254.0.0/16 dev eth0  scope link  metric 1002
17.255.209.0/24 dev eth0  proto kernel  scope link  src 17.255.209.19

Now when I do a tcpdump on the website I don’t seem to see a change in the initial window with the WIN/MSS remaining 4 as default. 5840/1460=4

[user@site etc]$ sudo tcpdump -n -i any 'tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn and port 80'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
11:17:45.048174 IP 21.101.151.198.45873 > 17.255.209.19.http: Flags [S], seq 2008673341, win 5840, options [mss 1460,sackOK,TS val 1724223146 ecr 0,nop,wscale 6], length 0

The curl hit I did to the webpage requested around 30 KB of data. 

[user@machine ~]$ curl http://www.site.com/js/main.js > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 88212  100 88212    0     0   179k      0 --:--:-- --:--:-- --:--:--  272k

What could be wrong in my approach?

Kernel 

[user~]$ uname -r
3.0.4x86_64-linode21

As an update, here' are the results when I try google.com

[user@site ~]$ sudo tcpdump -n -i any 'tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn and host www.google.com'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
17:20:28.033236 IP 17.255.209.19.42799 > 74.125.127.106.http: Flags [S], seq 3148947324, win 14600, options [mss 1460,sackOK,TS val 193695310 ecr 0,nop,wscale 4], length 0

As you can see WIN/MSS is 14600/1460=10 in this case

I tried hitting my site from the server machine itself through curl and here's the result:

[user@site ~]$ sudo tcpdump -n -i any 'tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn and host www.site.com'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
17:25:14.584338 IP 17.255.209.19.35008 > 17.255.209.19.http: Flags [S], seq 3894567470, win 32792, options [mss 16396,sackOK,TS val 193981861 ecr 0,nop,wscale 4], length 0

WIN/MSS is 32792/16396=2 in this case

linux networking optimization tcp tcp-slow-start

2 Answers

  1. Best Answer

    I think you are misunderstanding how TCP works.

    Each packet sent will always advertise a receiver window (aka. RWIN) and an optional scaling factor, see RFC 1323

    The sender is not allowed to send more than the amount of data specified in the RWIN without it getting acknowledged. Depending on the congestion window, the sender may decide to fill up the RWIN or not.

    So, there are two bits of information that are public in the TCP packets. The RWIN on the Server and the RWIN on the client. Both of these figures dictate what the maximal size of the congestion window can be on both ends.

    The RWIN on the server is interesting when we are trying to optimise performance for say file uploads.

    The RWIN on the client is interesting when we are trying to determine download speed.

    Neither of these numbers make the congestion window on the other end public.

    SO if I have an RWIN of 64k, the congestion window on the server can be ANY number lower than 64k.

    The only way to determine what the actual congestion window is to count packets.

    If I know:

    1. My round trip time (RTT) is ~200ms.
    2. I just requested a resource that is 100k.
    3. I have an RWIN of 64k.

    If I get 2 packets back from the server that are 1452 bytes long within ~200ms, it is likely the congestion window on the server is smaller than 4356, cause if it were bigger 3 packets would be sent. If IW was set to 10, I would see a burst of 10 packets around the 200ms mark.

    If you change your IW and want to confirm the change worked, you need to count packets to get an estimate on the congestion window size on the server.

    Keep in mind, you probably want to look at the conversation directly after the SYN, SYN-ACK, ACK to ensure you are not looking at the middle of a conversation (where the congestion window could have already grown).

    • 9

  2. The window size will be whichever is smaller: server init window size or client RWIN. Since 5840 is the default RWIN for Linux 2.6, it seems that your client is the limiting factor here.

    Try from a windows box. Windows XP has an RWIN of 64k, newer version 8k.

    Source: http://www.cdnplanet.com/blog/tune-tcp-initcwnd-for-optimum-performance/ (The interesting part is below the video)

    Edit: Expanding the answer to make it clearer:

    • In the TCP handshake, the client sends a SYN packet to the server sending its maximum allowed window size. (As your tcpdump output shows, these are 5840 bytes)
    • The server now responds with SYN ACK and the window size it would like to agree on. This window size can only be smaller than what the client proposed, not bigger. No matter how the server is configured, it can never have bigger window sizes than 5840 bytes with that client.
    • Client returns ACK and they happily exchange data ever after.

    Edit2: The tcpdumps added to the question show the server opening connections to google and itself ACTING AS THE CLIENT.

    • 7