Home / server / Questions / 309513
Accepted
grs
Asked: 2011-09-09 11:31:07 +0800 CST

How to test 10 gigabit Intel X520 on Ubuntu

  • 772

I got Intel X520 and connected it to PCI-E x16. The card is recognized as 10 Gbit by ethtool: 

 # ethtool eth5
Settings for eth5:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full 
                                10000baseT/Full 
        Supports auto-negotiation: Yes
        Advertised link modes:  10000baseT/Full 
        Advertised auto-negotiation: Yes
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
        Link detected: yes

The switch also recognizes the link as 10 Gbit. I ran iperf tests from 12 different machines (2 times x 6). Every single connection is 1 Gbit, except the server one. So, I expect to see around 6 Gbit total (6 machines x 1 Gbit each). I got only total of 1 Gbit on the server side. My questions are:

  1. am I doing something wrong?
  2. how to properly test the new card?

The OS is Ubuntu Karmic, the MB is Supermicro X8DAi. Thanks!

EDIT: The switch is single Netgear GSM7328S-200NAS. Here is a diagram: 6 1 Gbps Clients to 10 Gbps server

I test by running:

Server: iperf -s
All clients, simultaneously: iperf -c server_IP

I see values around 150-160 Mbps/client, which match to 1 Gbps link to the server. The IP I assigned to the new card was never used, so I suppose no ARP issues. However, all traffic is on one subnet, as the same clients use the old 1 Gbps card and IP of the server.

No virtualization here, just plain Ubuntu install, serving NFS mostly.

Update: after further tests, I found that the clients' ARP tables contain wrong entry for server's 10 Gbps NIC. The HW address listed there is the same as the server's 1 Gbps NIC. I deleted it and manually inserted the correct entry: arp -d server_IP_10g, arp -s server_IP_10g HW:address. After that I was unable to ping server_IP_10g.

How can I fix it without disabling the 1 Gbps NIC in the server?

UPDATE2: sysctl -w net.ipv4.conf.all.arp_ignore=1 sysctl -w net.ipv4.conf.all.arp_announce=2 does not work.

linux ubuntu networking arp

1 Answers

  1. Best Answer

    After carefully revising pros and cons and what actually is possible to test, I went and disabled eth3 on the server (the 1 Gbps NIC) and set up eth5 (the 10 Gbps) with its address. Then the clients found the new interface and iperf showed 10 Gbps from 10 x 1 Gbps clients. So, all is good. I think @sciurus' question, posted above, would be great, if I needed both interfaces permanently. +1 from me on the good reference!

    • 2