Background
I have a new hosted, dedicated database server behind a dedicated firewall (Cisco ASA 5505 Sec+). The plan is to have a virtual (aka "cloud") web server or two on the other side of the firewall connecting back to the backend DB server.
While setting up the server, I was unimpressed by it's network performance. Turns out, though the 2 servers have GigE - the firewall only supports 100Mb - so most of the performance issues I had can be adequately explained by that.
Problem
However, as part of troubleshooting, I ran a series of pings to the firewall from the dedicated server. These pings came back with some interesting results - specifically, the distribution of 100 pings was:
57% < 1ms
14% between 1ms and 2ms
12% between 2ms and 3ms
11% between 3ms and 6ms
6% >= 6ms
Min/Avg/Max: 0/1/8 ms
I would've expected first hop to be < 1ms consistently (and can't honestly recall any hard wired environment where it wasn't). Subsequent tests were pretty similar, and have been for so me number of days - so this doesn't appear to be an isolated incident. No retransmits or dropped packets have been observed. Pinging across the firewall shows similar performance:
58% < 1ms
14% between 1ms and 2ms
8% between 2ms and 2ms
14% between 3ms and 6ms
6% >= 6ms
Min/Avg/Max: 0/2/56 ms
Troubleshooting
The hoster has checked the server, firewall and intervening switch(es) and sees no issues. They also point out that they "deprioritize" ICMP traffic on the network. They noticed some recent port flapping (likely caused, I believe, by configuration of the server) and will "continue to monitor" the situation. The port flapping isn't numerous enough or time correlated enough to explain the ping times, though it may be possible it's a(nother) symptom of an underlying issue.
I don't have direct access to the ASA - but the hoster ran some stats on it as part of troubleshooting:
# ping ***** (series of 5-packet pings from firewall to server, edited for brevity)
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/10 ms
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/8/10 ms
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/6/10 ms
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
# show cpu usage
CPU utilization for 5 seconds = 13%; 1 minute: 11%; 5 minutes: 10%
# show mem
Free memory: 341383104 bytes (64%)
Used memory: 195487808 bytes (36%)
------------- ----------------
Total memory: 536870912 bytes (100%)
# show int eth0/1
Interface Ethernet0/1 "", is up, line protocol is up
Hardware is 88E6095, BW 100 Mbps, DLY 100 usec
Full-Duplex(Full-duplex), 100 Mbps(100 Mbps)
Available but not configured via nameif
MAC address *****, MTU not set
IP address unassigned
5068644 packets input, 5077178693 bytes, 0 no buffer
Received 4390 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 L2 decode drops
387883 switch ingress policy drops
3220647 packets output, 1648213382 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
0 input reset drops, 0 output reset drops
0 rate limit drops
0 switch egress policy drops
Except for some seemingly high CPU usage for a firewall with a few ACLs and only possibly an RDP session going through it, I don't see anything alarming about the ASA stats. It certainly doesn't appear over-taxed IMHO.
Question
Considering we're approaching disk seek times, and there's no production traffic on the firewall or server yet - I'm still a little concerned. What do you guys think? Is this an issue? Is this normal in a larger data-center environment?