sajal's questions

I am trying to figure out a way to detect round trip times for each IP that connects to port 80 on my server. What is a good way to do this?

There are far to many connections, and pinging each ip is not practical. I was thinking, somehow measuring the time difference between when server send the syn/ack packet and when server receives the ack for it.

Is there a way to log this cheaply? Maybe using some magic tcpdump filter?

The goal of this exercise is to aggregate the data and figure if there are any routing inefficiencies. Say one ISP in one region maybe routing incorrectly, etc.

FWIW the webserver is nginx.

My new server has 2 x X5570 CPUs.

Now here is the output of grep -i hz /proc/cpuinfo

model name : Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
cpu MHz  : 1600.231
model name : Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
cpu MHz  : 1600.231
model name : Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
cpu MHz  : 1600.231
model name : Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
cpu MHz  : 1600.231
model name : Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
cpu MHz  : 1600.231
model name : Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
cpu MHz  : 1600.231
model name : Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
cpu MHz  : 1600.231
model name : Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
cpu MHz  : 1600.231
model name : Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
cpu MHz  : 1600.231
model name : Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
cpu MHz  : 1600.231
model name : Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
cpu MHz  : 1600.231
model name : Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
cpu MHz  : 1600.231
model name : Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
cpu MHz  : 1600.231
model name : Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
cpu MHz  : 1600.231
model name : Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
cpu MHz  : 1600.231
model name : Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
cpu MHz  : 1600.231

It always remains the same.. no matter how much load is mysql or any other app hogging. Even when mysql eats 2 or 3 CPUs at 100% each, the output of cpuinfo is the same.

If fact mysql performance for some heavy inserts is poorer than my old E5430 server.

Any clues? I contacted the server provider, they tried turning off SpeedStep and still we see the same results.

Any insights would be helpful cause I am paying heavily for this box and would love to milk all juice i can.

I have a strange issue, hosting at Softlayer Washington data center.

Test from an EC2 instance to my server at Softlayer :-

[root@EC2-box ~]# ping -f -c 100 MY-SL-BOX
PING MY-SL-BOX (MY-SL-BOX) 56(84) bytes of data.
..................................................
--- MY-SL-BOX ping statistics ---
100 packets transmitted, 50 received, 50% packet loss, time 743ms
rtt min/avg/max/mdev = 2.464/2.906/3.606/0.220 ms, ipg/ewma 7.505/2.927 ms
[root@EC2-box ~]# 

Test from an EC2 instance to www.softlayer.com (or any other website):-

[root@EC2-box ~]# ping -f -c 100 www.softlayer.com
PING www.softlayer.com (66.228.118.51) 56(84) bytes of data.

--- www.softlayer.com ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 1168ms
rtt min/avg/max/mdev = 35.143/35.485/36.216/0.211 ms, pipe 4, ipg/ewma 11.804/35.518 ms
[root@EC2-box ~]#

Test from my server at softlayer to EC2 instance :-

[root@MY-SL-BOX ~]# ping -f -c 100 7EC2-box
PING 7EC2-box (7EC2-box) 56(84) bytes of data.
.................................................
--- 7EC2-box ping statistics ---
100 packets transmitted, 51 received, 49% packet loss, time 866ms
rtt min/avg/max/mdev = 2.067/2.378/2.679/0.131 ms, ipg/ewma 8.756/2.414 ms

Test from my server at softlayer to www.softlayer.com instance :-

[root@MY-SL-BOX ~]# ping -f -c 100 www.softlayer.com
PING www.softlayer.com (66.228.118.51) 56(84) bytes of data.
....................................................
--- www.softlayer.com ping statistics ---
100 packets transmitted, 48 received, 52% packet loss, time 1424ms
rtt min/avg/max/mdev = 33.677/34.041/34.693/0.240 ms, pipe 4, ipg/ewma 14.386/34.083 ms

Note.. in all the above cases, when I don't use the "-f" option, there is no packet loss at all. 7EC2-box and MY-SL-BOX are real IPs. I masked them to post here.

If I do a traceroute to any other IP from my server, and even ping -f the first node shown there, it shows a 50% packet loss, and no loss without "-f".

I spoke to the support, and they said that since there is no problem without the "-f" there is no issue here.

Are they right?

It's odd that the "ping -f" from any external network to any site shows no loss but only to mine.

I've been searching for an hour now, would be very grateful if someone can point me to some detailed instructions for testing packet loss.