Home / server / Questions / 483038
In Process
Omid Kosari
Asked: 2013-02-28 06:42:58 +0800 CST

Squid TPROXY connection fails on specific sites?

  • 772

When squid setup as TPROXY, returns ERR_CONNECT_FAIL on some sites . The server which squid runs on it is able to open those sites by lynx, wget, curl etc . Even if we manually set proxy in browser or use simple REDIRECT or DST-NAT, those sites could open.

From this page http://wiki.squid-cache.org/SquidFaq/InterceptionProxy

It causes path-MTU (PMTUD) to fail, possibly making some remote sites inaccessible. This is not usually a problem if your client machines are connected via Ethernet or DSL PPPoATM where the MTU of all links between the cache and client is 1500 or more. If your clients are connecting via DSL PPPoE then this is likely to be a problem as PPPoE links often have a reduced MTU (1472 is very common).

But i have same problem with ethernet

Here is tcpdump from one client:

Click here to see tcpdump when i use tproxy and the problem appears

and

Click here to see tcpdump when i manually set proxy in my browser and everyhting works fine

GET / HTTP/1.0
Host: 80.75.1.4
Accept: text/html, text/plain, text/css, text/sgml, */*;q=0.01
Accept-Encoding: gzip, compress, bzip2
Accept-Language: en
User-Agent: Lynx/2.8.8dev.2 libwww-FM/2.14 SSL-MM/1.4.1

HTTP/1.0 504 Gateway Time-out
Server: squid
Mime-Version: 1.0
Date: Wed, 27 Feb 2013 15:39:03 GMT
Content-Type: text/html
Content-Length: 376353
X-Squid-Error: ERR_CONNECT_FAIL 110
X-Cache: MISS from cache.mysquid.com
X-Cache-Lookup: MISS from cache.mysquid.com:3128
Connection: close

ping -s 1500 80.75.1.4

PING 80.75.1.4 (80.75.1.4) 1500(1528) bytes of data.
1508 bytes from 80.75.1.4: icmp_req=1 ttl=58 time=5.28 ms
1508 bytes from 80.75.1.4: icmp_req=2 ttl=58 time=3.96 ms
1508 bytes from 80.75.1.4: icmp_req=3 ttl=58 time=4.28 ms

ping -s 1473 80.75.1.4 -M do

PING 80.75.1.4 (80.75.1.4) 1473(1501) bytes of data.
From 109.110.160.171 icmp_seq=1 Frag needed and DF set (mtu = 1500)
From 109.110.160.171 icmp_seq=1 Frag needed and DF set (mtu = 1500)

--- 80.75.1.4 ping statistics ---
0 packets transmitted, 0 received, +2 errors

ping -s 1472 80.75.1.4 -M do

PING 80.75.1.4 (80.75.1.4) 1472(1500) bytes of data.
1480 bytes from 80.75.1.4: icmp_req=1 ttl=58 time=4.33 ms
1480 bytes from 80.75.1.4: icmp_req=2 ttl=58 time=4.32 ms

--- 80.75.1.4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 4.320/4.329/4.338/0.009 ms

traceroute --mtu 80.75.1.4

traceroute to 80.75.1.4 (80.75.1.4), 30 hops max, 65000 byte packets
 1  x.x.x.10 (x.x.x.10)  0.820 ms F=1500  0.681 ms  0.243 ms
 2  y.y.y.1 (y.y.y.1)  2.969 ms  3.185 ms  2.994 ms
 3  217.218.181.193 (217.218.181.193)  2.836 ms  2.381 ms  2.487 ms
 4  217.218.185.22 (217.218.185.22)  3.617 ms  2.957 ms  3.176 ms
 5  78.38.119.237 (78.38.119.237)  2.050 ms  1.808 ms  2.264 ms
 6  217.11.30.250 (217.11.30.250)  3.522 ms  3.962 ms  2.674 ms
 7  * 80.75.1.4 (80.75.1.4)  3.507 ms *

tracepath 80.75.1.4

 1:  cache.mysquid.com                                     0.092ms pmtu 1500
 1:  x.x.x.10                                              0.380ms
 1:  x.x.x.10                                              0.309ms
 2:  y.y.y.1                                               3.390ms asymm  7
 3:  217.218.181.193                                       2.671ms asymm  5
 4:  217.218.185.22                                        2.944ms asymm  5
 5:  78.38.119.237                                         1.684ms
 6:  217.11.30.250                                         4.020ms
 7:  80.75.1.4                                             3.915ms reached
     Resume: pmtu 1500 hops 7 back 58

Also tried these steps http://wiki.squid-cache.org/SquidFaq/SystemWeirdnesses#Can.27t_connect_to_some_sites_through_Squid

I don't know related or not but i also have following configs

echo 1025 65000 > /proc/sys/net/ipv4/ip_local_port_range
echo 0 > /proc/sys/net/ipv4/tcp_syncookies
echo 131072 > /proc/sys/net/ipv4/tcp_max_syn_backlog
echo 1 > /proc/sys/net/ipv4/ip_forward
echo 0 > /proc/sys/net/ipv4/conf/lo/rp_filter
echo 1 > /proc/sys/net/ipv4/ip_forward
echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/p2p1/rp_filter
echo 524288 > /proc/sys/net/netfilter/nf_conntrack_max

and enabled/disabled following 

#echo 0 > /proc/sys/net/ipv4/tcp_window_scaling
#echo 0 > /proc/sys/net/ipv4/tcp_ecn

Here is TPROXY route rules

/sbin/iptables -t mangle -N DIVERT
/sbin/iptables -t mangle -A DIVERT -j MARK --set-mark 1
/sbin/iptables -t mangle -A DIVERT -j ACCEPT
/sbin/iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
/sbin/iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3129
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

Note : Connection to internet providers is over GRE Tunnel with MTU 1476.

linux

3 Answers

  1. From the tcpdump output you provided, following lines indicate a problem, i.e. your host sending a SYN to destination, and then sending an RST to the destination as well, but ofcourse TTLs are different, so seems like some TPROXY magic going on:

    12:55:17.255717 IP (tos 0x0, ttl 64, id 11025, offset 0, flags [none], proto TCP (6), length 56)
    x.x.x.150.62568 > 80.75.1.4.80: Flags [S], cksum 0x5f7e (incorrect -> 0x3c92), seq 572035649, win 14600, options [mss 1460,sackOK,TS val 66378265 ecr 0], length 0
12:55:17.258877 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    x.x.x.150.62568 > 80.75.1.4.80: Flags [R], cksum 0xa779 (correct), seq 572035650, win 0, length 0
12:55:18.253670 IP (tos 0x0, ttl 64, id 11026, offset 0, flags [none], proto TCP (6), length 56)
    x.x.x.150.62568 > 80.75.1.4.80: Flags [S], cksum 0x5f7e (incorrect -> 0x3b98), seq 572035649, win 14600, options [mss 1460,sackOK,TS val 66378515 ecr 0], length 0
12:55:18.257916 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    x.x.x.150.62568 > 80.75.1.4.80: Flags [R], cksum 0xa779 (correct), seq 572035650, win 0, length 0

    It doesn't seem like an MTU problem. To start with troubleshooting, I suggest removing TPROXY (assuming you're using that) and switch to REDIRECT based rule and see if problem goes away. It would be helpful if you could paste your REDIRECT or related iptables rules.

    • 2

  2. The Unix error 110 means "Connection timed out". The remote site never responded to the request.

    When I tried to reach 80.75.1.4, I also was not able to reach it. It appears from ping and traceroute that there is significant packet loss on the network after it enters the state of Iran. This is probably why the site is failing to respond.

    $ traceroute 80.75.1.4
traceroute to 80.75.1.4 (80.75.1.4), 30 hops max, 60 byte packets
 1  hosted.by.leaseweb.com (198.7.57.254)  1.275 ms  1.227 ms  1.888 ms
 2  108.59.15.23 (108.59.15.23)  0.249 ms  0.243 ms  0.216 ms
 3  be4.cr2.wdc1.leaseweb.net (108.59.15.108)  0.922 ms be3.cr2.wdc1.leaseweb.net (108.59.15.102)  0.922 ms ae1.cr1.wdc1.leaseweb.net (108.59.15.116)  0.275 ms
 4  xe-7-2-0-690.edge1.Washington1.Level3.net (4.28.127.57)  1.258 ms xe-5-3-0-673.edge3.Washington1.Level3.net (4.59.144.161)  1.181 ms  1.247 ms
 5  ae-4-90.edge1.Washington4.Level3.net (4.69.149.207)  1.779 ms  1.749 ms ae-3-80.edge1.Washington4.Level3.net (4.69.149.143)  1.670 ms
 6  ash-bb1-link.telia.net (213.248.81.117)  1.499 ms ash-bb1-link.telia.net (213.248.89.177)  15.220 ms ash-bb1-link.telia.net (213.248.103.233)  1.505 ms
 7  ash-bb4-link.telia.net (213.155.130.58)  1.560 ms  1.637 ms *
 8  ldn-bb2-link.telia.net (80.91.246.67)  76.651 ms  76.782 ms  76.761 ms
 9  ldn-b1-link.telia.net (80.91.248.93)  77.581 ms  77.552 ms  77.515 ms
10  telecominfra-ic-132493-ldn-b1.c.telia.net (213.248.76.42)  234.204 ms  233.666 ms  233.731 ms
11  * * *
12  * 10.10.53.222 (10.10.53.222)  251.938 ms *
13  217.11.30.246 (217.11.30.246)  250.353 ms  250.725 ms  250.322 ms
14  80.75.1.4 (80.75.1.4)  240.361 ms  240.269 ms  240.327 ms

    What's worse, is it looks like someone has used RFC 1918 addresses on the public Internet in this path. Among other things, this causes Path MTU discovery to break, which means many TCP connections will also break.

    To quote from one source, Cisco's IP Addressing Best Practices (PDF):

    Any router-to-router links connecting to areas of the network with public addressing should be addressed with public IP addresses. Routers serving specific areas of the network using and continuing to use only private addresses may use private addresses on the router-to-router links.

    This requirement enables and helps ensure that path MTU discovery (RFC 1191) works properly; routers must be able to send "packet-too-big" errors and must be assured that the packets are likely to arrive at the original source host. If router-to-router links are addressed with RFC 1918 addresses, the Internet Control Message Protocol (ICMP) messages generated by the router will come from an RFC 1918 address. Networks filtering out incoming packets with RFC 1918 source IP addresses, or using unicast reverse path forwarding (uRPF), will likely drop these packets, breaking TCP for those applications. This will cause large packet transfer across a TCP connection to fail completely or perform suboptimally.

    In short, there are multiple network issues here which will cause the site to be unreachable for many or most Internet users.

    • 1

  3. The prossible problems could be MTU and also routing problems, as you could open addresses via links or curl the issue may related to MTU.

    To check MTU you need to use ping to check correct Path MTU and manually set correct MTU to your network adapter. also you can use Change-MSS in your iptables to fix this issue I try yo explain both solutions:

    for MTU checking try to ping destination with the following command:

    ping -M do -s 1472 <ip address>

    the -s argeument sets ping package size you may alter this value (normally decrease) until you find the value at that ping your packages start to be fragmented. then add 28 bytes (ping header size) to this value and this is your PMTU size. you may also use tracepath to discover the PMTU.

    then you must set a iptables rule to set change packets MSS size accourding to your Path MTU size. 

     iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss <PMTU size>

    you can also use --clamp-mss-to-pmtu to ask iptables to check for PMTU automatically:

     iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

    Thats all I can say right now, hope it helps

    P.S. Do you use TProxy for transparent proxy?

    • 0