Home / server / Questions / 395264
Accepted
Rayne
Asked: 2012-06-04 23:37:59 +0800 CST

Transmit rate decreases when using both Intel NIC 10G ports

  • 772

I'm using the Intel 82599EB 10G NIC on a server running RHEL6. I ran a program where packets are generated at high rates. The NIC was previously installed on another PC where I was able to generate traffic at 8-9Gbps on each port concurrently. On this server, I was able to generate at 8-9Gbps only if one port is in use. If I were to generate on both ports concurrently, then my total transmit rate drops to about 12Gbps (~6Gbps on each port).

My relevant "lspci -vvv" output shows the following:

0b:00.0 Ethernet controller: ...
      Capabilities: [a0] Express (v2) Endpoint, MSI 00
            LnkCap: Port #1, Speed 5GT/s, Width x8
            LnkSta: Speed 5GT/s, Width x4
            LnkCtl2: Target Link Speed: 5GT/s

0b:00.1 Ethernet controller: ...
      Capabilities: [a0] Express (v2) Endpoint, MSI 00
            LnkCap: Port #1, Speed 5GT/s, Width x8
            LnkSta: Speed 5GT/s, Width x4
            LnkCtl2: Target Link Speed: 2.5GT/s

Is my transmit rate limited by the "x4" in LnkSta for both ports? And what does it mean to have "Target Link Speed: 5GT/s" on one port and "Target Link Speed: 2.5GT/s" on the other port? Does this contribute to the rate decrease too?

Also, I see a couple of PCIE slots that are longer than my NIC's PCIE connectors. Can I install my NIC on these longer slots instead?

traffic nic intel rhel6

1 Answers

  1. Best Answer

    Also, I see a couple of PCIE slots that are longer than my NIC's PCIE connectors. Can I install my NIC on these longer slots instead?

    MOSTLY yes. As in: Yes in theory, not sure in practice - a bad BIOS may blow that. FIor example in the past the Graphics port (a ncice pCIEx16) with a RAI card may result in boot failures due to the BIOS Not handling that well... THAT though is years in the past, these days such problems should not happen. I do that regularly, running RAID controllers from Graphics slots and using on board graphics for server scenarios.

    Try and see waht hapens is the only solution here - PCIe is compatible in that CAN put a smaller connector into a larger slot from the standard. The larger connector must not even connect all pins - you could have a PCIex1 link on a PCIex16 connector, advantage clearly that all cards fit but only work with PCIEx1... I have some convertrs here to run graphics cards on PCIEx1 slots for number crunching.

    Is my transmit rate limited by the "x4" in LnkSta for both ports?

    Well, the physical limit of the port obviously limits the speed of the line, or? And yes, that is a physical limit - nothing you can program around.

    And what does it mean to have "Target Link Speed: 5GT/s" on one port and "Target Link Speed: 2.5GT/s" on the other port? Does this contribute to the rate decrease too?

    Someone else will have to answer what, but I found this in google:

    http://www.xilinx.com/support/answers/37497.htm

    If that is as I read it, that is the configured uppe limit - so yes, that owuld be a THE Reason, but the question is why, so it may be a symptom of something else.

    • 3