Home / server / Questions / 170262
In Process
Brent
Asked: 2010-08-14 08:12:12 +0800 CST

Mysterious WAN Connection Throttling

  • 772

Setup

2x Cisco ASA 5520's

ASA 8.3(1), ASDM 6.3(1)

IPSec Crypto Settings:

ESP-3DES-SHA, bidirectional, perfect forwarding secrecy group 1, 8 HR lifetime or 4,608,000 Kilobytes, NAT-T enabled

Network A, Chicago 10.110.1.0/24 100 MBPS Cogent connection

Network B, Denver 10.110.2.0/24 Into datacenter fiber network at 60 MBPS

Issue

Any traffic between Chicago and Devner (IPSec, HTTP, FTP, etc) hits a wall at 7 MBPS Denver -> Chicago and 1.5 MBPS Chicago -> Denver.

Tests

  1. Microsoft updates download at 60+MBPS from either network so I know that the WAN connection at either end is fast.
  2. Rebooted both firewalls just for good measure.
  3. Transfers from the Chicago ASA to another Chicago ASA ran at the expected 45-50 MBPS. So it doesn't look like my ASA are going to be rate limiting or IPSec is having trouble with the through output. CPU sits at 3-4% during the day and 10% during the 7/1.5 MBPS transfers.
  4. Traceroute between Chicago and Denver comes to 16 hops, no hop higher than 60ms
  5. Have transferred files over the IPSec tunnel in Windows, outside the IPSec tunnel as HTTP or FTP traffic and still see the 7/1.5 MBPS limits.
  6. Have connected a desktop to our Cogent switch in Chicago, assigned a WAN IP, transferred HTTP and FTP traffic and still see a 7/1.5 MBPS limit.
  7. Looked at TCP Window size on transfers while I was seeing a limit between Chicago and Denver, and while just downloading a file. Both were at 65,535.
  8. Asked Datacenter to look at the line to see if there were any lost packets, CRC errors, etc. They saw nothing.
  9. 24 hour ping shows no packet loss and average response time of 85ms.
  10. Double-checked Cisco ASA setting to make sure there was no "police output" commands issued. Our ASA is pretty vanilla as we have some NATs, ACL for access, and a IPSec VPN back home to Chicago.

Questions

  1. I can't figure out what OSI layer to deal with when troubleshooting.
  2. Do I engage the ISPs and tell them my issues? Who should I point the finger at?
vpn cisco bandwidth cisco-asa

2 Answers

  1. It looks like it is time to run wireshark on both ends. My guess is it will help you find the traffic that is causing the problem. We recently had a similar (it was causing our iscsi san not to sync) it ended up being a bad data center switch. I used the wireshark logs to show them the problem and they replaced the switch.

    • 3

  2. my thoughts on this would be first, to try running multiple concurrent connections to see if you achieve your allotted speeds through multiple simultaneous connections, then you have pretty much ruled out any layer 1, 2 or 3 difficulties

    possibly try a UDP throughput test, if you can send 70mbps in UDP traffic but you are limited to 7mbps on your TCP throughput, then I would strongly lean towards it being some sort of TCP sliding window issue

    we ran into something very similar to this scenario a few months ago, and it was resolved by a switch that did not have the proper MTU setting on it, which I guess was limiting the TCP sliding window from hitting it's peak performance. I would verify you can send 1500 byte packets (the standard size for internet packets) from end to end with the do not fragment bit set on the ping.

    hope this helps

    • 3