Home / server / Questions / 378265
In Process
NPE
Asked: 2012-04-11 12:03:20 +0800 CST

Very low throughput on 10GbE network

  • 772

I have two Linux machines, each equipped with a Solarflare SFN5122F 10GbE NIC. The two NICs are connected together with an SFP+ Direct Attach cable.

I am using netperf to measure TCP throughput between the two machines. On one box, I run:

netserver

and on the other:

netperf -t TCP_STREAM -H 192.168.x.x -- -m 32768

I get:

MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.x.x (192.168.x.x) port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  32768    10.02    1321.34

The measured throughput is 1.3Gb/s. This is 7.5x below the theoretical maximum, and only 30% faster than 1GbE.

What steps can I take to troubleshoot this?

performance networking ethernet 10gbethernet

1 Answers

  1. few things:

    • did you try adjusting mtu to make use of jumbo-frames?
    • are you absolutely sure that the link between the two servers does not have any packet losses?
    • does ethtool show you any errors on the interfaces on both ends?
    • what does top/atop say during the prolonged test - do you see any of the cores fully occupied by iowait?

    you'll [most probably] not achieve full 10gbit on a single tcp session but you can do some additional tuning to get closer to it by tweaking tcp handling and communication with the network card itself - take a look here or here .

    • 6