Home / server / Questions / 271225
Accepted
Henno
Asked: 2011-05-19 13:37:53 +0800 CST

10 gigabit NICs showing only half the throughput

  • 772

I have 2 physical servers:

  1. Two way Intel E5504 @ 2GHz, 24GB RAM, 12x32GB Intel X25-E SSDs in RAID10.
  2. Intel Core2 6400 @ 2.12GHz, 3GB RAM, simple 80GB SATA drive.

Both machines run Windows Server 2008 R2 now and have 10Gbit Supermicro AOC-STGN-i2S (actually they are Intel 82599 bearing Supermicro logo) in PCIe x4 slots- with a SFP+ direct attached twin axial cable between them.

The second server is for testing only.

First I installed ESXi on the 2nd and used the 1st as a datastore.

I noticed that according to CrystalDiskMark, a VM on ESX only got 325 MB/s seq transfer rate (tried with both NFS and ISCSI).

I ran the same test on the first server locally and got ~1000 MB/s. I wondered if the network link really kills 2/3 of speed, so I replaced 2nd's hard drive and installed Windows Server 2008 R2 and tried Jperf and NTTtcp. Jperf showed 400 MB/s and NTttcp showed 4300-4600Mbit/s. Windows Task Manager showed some 600 000 000 bytes per interval which translates to 4.47 Gigabits.

I verified that both ends had full duplex and tried toggling jumbo frames on and off both ends but the difference was only 580 000 000 vs 600 000 000 bytes per interval.

Why the throughput I'm seeing is only about half the theoretical maximum of 10 gigabits?

ADDENDUM

NTTtcp command lines:

ntttcpr -m 6,0,192.168.137.1 -a 6 (receiver)
ntttcps -m 6,0,192.168.137.1 -a 6 (sender)
windows networking 10gige

1 Answers

  1. Best Answer

    I'd suspect your PCI-e x4 slots are the bottleneck. The theoretical throughput on those slots should be in the range of 16 Gbps (saturating the NIC with room to spare), but that's not always well-implemented from the controller standpoint.

    Got an x8 or higher slot you can steal from something else to test?

    • 3