Home / server / Questions / 509121
Accepted
Crash Override
Asked: 2013-05-20 14:48:42 +0800 CST

Stuck TCP Connection in Linux

  • 772

Strange problem. I have server "ALPHA" in Dallas and server "BETA" in Hong Kong, about 195ms ping times apart from each other. ALPHA is connected at 1gbps full, and BETA at 10mbps full. Occasionally when they communicate to each other over TCP, the connection hangs. Eventually ALPHA gives up and closes the connection, but BETA still sees the connection as established and waits for a long time until it eventually times out.

Example.

On BETA, wget -O /dev/null ALPHA:50001/1mb.test

--2013-05-19 02:45:54--  http://ALPHA:50001/1mb.test Resolving dfw... ip.address.redacted Connecting to ALPHA|ip.address.redacted|:50001... connected. HTTP request sent, awaiting response... 200 OK Length: 1000000 (977K)
[application/octet-stream] Saving to: `/dev/null'

and it freezes at 0 bytes received. Note that it did get a 200 OK from the server, so they did seem to handshake successfully. To eliminate wget as part of the problem, I did a telnet:

telnet ALPHA 50001
Trying ip.address.redacted...
Connected to ALPHA.
Escape character is '^]'.
GET /1mb.test HTTP/1.0

HTTP/1.1 200 OK
Server: nginx
Date: Sun, 19 May 2013 22:43:48 GMT
Content-Type: application/octet-stream
Content-Length: 1000000
Last-Modified: Sat, 21 Jan 2012 21:47:29 GMT
Connection: close
ETag: "4f1b3271-f4240"
Accept-Ranges: bytes

It also freezes at this point.

Both servers do have iptables rules of -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

Here is a tcpdump that was running on ALPHA during the above wget freeze. http://pastebin.com/axX7ap6T

This problem happens only occasionally. Sometimes it works just fine. Actually, if I run about 3-5 wget's, usually everything starts working on about the fourth or fifth try.

BETA can communicate with GAMMA, DELTA, and my other servers without problem. Only seems to happen when talking to ALPHA. ICMP pings show no packet losses, but even if there was packet loss, it would slow it down, not disconnect. Both machines have the same CentOS 6.4 64-bit OS, kernel 2.6.32-358.6.2.el6.x86_64. Nothing useful logged in /var/log files. ALPHA is listening on 50001 using nginx-1.4.1-1.el6.ngx.x86_64. The nginx access log file shows a 200 (request OK) 43440 bytes delivered during the frozen connection.

The problem does not seem to be nginx related, since I have had an ssh connection also freeze in the middle of communicating on the same day. Other servers connect to that nginx and never had any problems.

In summary, this appears to be what is happening in this particular communication:

BETA -> ALPHA "SYN!"
ALPHA -> BETA "SYN Acknowledgement"
BETA -> ALPHA "ACK"
ALPHA -> BETA "Established"
BETA -> ALPHA "GET /file HTTP/1.0"
ALPHA -> BETA "200 OK, here's a bunch of data"
BETA freezes, gets no data.

Anyone seen a situation like this?

linux

3 Answers

  1. Best Answer

    Maybe an MTU/MSS problem? The small initial packets get through, the big ones (data transfer) get killed somewhere?

    A work-around is to reduce the MTU (on both sides) a bit, from 1500 to 1450 e.g. Or to use the Netfilter target TCPMSS just for the affected connections (or differently for separate connections).

    But the real solution is to get path MTU discovery working again. So check whether ICMP packets (fragmentation needed: Type 3, Code 4) are blocked. Use tcpdump -i $ifname -n icmp to see whether such packets arrive.

    • 6

  2. There is indeed defragmentation occurring. That's what IPTABLES does to inspect packets. It must reassemble them in order to validate content.

    In my case, I would see periodic ping dropouts and web browser stalls while I was web browsing on a CentOS 7 system, or my PuttY sessions from windows would mysteriously hang -- permanently.

    I used a variant of method Mick so kindly provided to determine the MTU size. The problem with that method is that if you've set the MTU size small, you get a false reading on the fragmentation. So set the size large (jumbo frame/9000) and then begin the ping probes.

    After setting MTU to 1472 (what I determined was the correct value) the ping problem when browsing went away. Hopefully when I next ssh from windows, that will work too.

    • 2

  3. My first guess is that it's a TCP Window scaling issue:

    https://en.wikipedia.org/wiki/TCP_window_scale_option#Possible_side_effects

    Disable it on both ends and see if it the problem goes away. Another option (as Hauke suggested) is there's defragmention occurring.

    You can use ping to check for packet de-fragmentation, a simple HOWTO is here:

    http://muzso.hu/2009/05/17/how-to-determine-the-proper-mtu-size-with-icmp-pings

    Edit: missed a word.

    • 1