Gigabit really gigabit?

Two cards with a crossover cable:

How are you testing it? How are you measuring it? When you say 20-40% tell us exactly what you mean in each instance, and how you're getting that figure.

The ethernet overhead is about 38 bytes per packet, and TCP/IP has about 52 bytes overhead. With 1500 byte packets, running TCP/IP over ethernet, you're looking at 94% possible data throughput.

The sending application may be sending smaller chunks of data, though. Take UDP (used for video streaming, voice, etc). If you're sending 256 bytes per packet, plus 90 bytes overhead, you're going to get 74% utilization.

Then you have the overhead of the OS and TCP/IP stack - both of which add latency, which will affect the bandwidth in complex ways due to the way TCP/IP manages loading of the network (algorithms attempt to use the maximum bandwidth without losing packets).

Then you have other loading on the computer. If you're copying fragmented data from a slow hard drive, you may not be seeing the network bottleneck at all, but another bottleneck elsewhere in the PC.

Are the 100base-T NICs also server grade? Are you certain they don't offload any processing to the OS that should be done on the NIC?

What bus are the NICs on? PCI, PCIe, etc? What else is on that bus? Is the NIC sharing an IRQ with any other devices?

Wre you using the latest BIOS for the motherboard and NIC (if applicable)? Latest drivers?

Is the OS a server OS, or desktop (the scheduler, stacks, etc are configured differently for different performance scenarios)?

There are many, many reasons why you might not be getting the throughput you desire. The simple question you're asking above doesn't provide enough information to know where the real problem is.

In a normal network, could be caused by collisions and other network traffic

Ethernet is awesome for what it does, but it is not synchronized. This means that two cards can start transmitting at once, which causes a collision.

A collision is generally detected by both cards, and they both back off for a random period of time before attempting a resend.

This, coupled with the overhead of the ethernet packet and encoding, means that even under the best circumstances you can't really expect more than 80-90% utilization.

You can alleviate this by using routers and switches rather than hubs between computers, but even routers and switches will drop packets if there's too much data, and they can only improve total utilization a few percent.

The ethernet packet itself includes overhead for clocking, addressing, etc. If you enable big packets on your network, then the overhead of the packet becomes small relative to the data size, so for large transfers you can get better utilization. This doesn't help if your application requires a lot of smaller packets, though, as in video or audio streaming applications. Even if you enable big packets, you may still only be sending small packets because that's what the application is sending.

-Adam

My guess would be that it's a limitation in your testing methodology. For example, if you're testing by transferring a file between two systems, than the performance of your filesystem may be a limiting factor. You'll also need to account for overhead of whatever protocol you're using, etc. Of course, there's some overhead inherent in ethernet and IP, so you're unlikely to ever see 100% utilization.

I'd suggest using a tool like iperf (http://en.wikipedia.org/wiki/Iperf) that's designed for testing throughput. There's also a GUI for iperf called Jperf, but the basic iperf command line isn't hard to figure out and hey... how often are ya gonna need to use it anyway right?

BTW. It is not, as some commenter have suggested, related to collisions. Collisions don't happen on full duplex ethernet and there's no such thing as half duplex gigabit ethernet. :-) It's also unlikely to have anything to do with needing jumbo frames, unless the two PC's you're using for testing have really bad ethernet cards or really low end CPU's. Jumbo frames is NOT necessary for full utilization of gigabit ethernet, although it can reduce CPU utilization during high bandwidth transfers. Unfortunately it can cause more problems than it solves and I would definitely NOT recommend enabling it unless you're quite sure what you're doing.

What are you using to test your link?

It's very easy to saturate a 100 Mbps link and most of a 1 Gbps link if you're just pushing stuff out from memory.

If you're trying to read stuff from disk and transfer to the other system, though, you'll be constrained by disk performance.

Are the nic's configured as full duplex rather than half duplex?

When I test network throughput, it's best to remove as many variables as possible such as disk i/o or CPU limitations. Here's the procedure I use, which can illustrate the typical max you will ever see in your network.

Tools:

Both these utilities should be available on most modern *nix systems, either preinstalled, via packages, or able to be compiled from source.

• Netcat
• PV

Procedure:

1. Run the following command on one computer to set up the UDP listener. (Using UDP will give higher performance numbers because of the lack of TCP overhead to transfer.)

   nc -ulp 5000 > /dev/null

2. On the other machine, run the following command, substituting the listener's ip address, to send a stream of zeros across the network to the nc listener. The pv utility will show the current performance as tracked by the receiver.

   pv < /dev/zero | nc -u listener-ip-address 5000

Notes:

• The theoretical max on a Gig Ethernet connection is 119MB/s (notice, MB, not Mb)
• TCP can also be tested by removing the -u in both nc commands
• In real world applications, you'll never reach the performance numbers illustrated in the pv output.

This method gives predictable results in a well designed network with properly functioning equipment. For that reason, it is a good tool to help discover if the network itself or the networking equipment is contributing to the perceived "slowness".

If you want to use the full throughput of Gigabit Ethernet, you'll need to ensure that Jumbo frames are configured throughout your full network -- or at least on all machines and switches on which you're measuring the performance.

However, keep in mind that a single hard drive's maximum transfer rate is usually 30-40 MByte/sec, which is not close to a Gigabit/sec.

I managed to get 80%-90% utilization just by copying files between network shares, on 100m and gigabit

Adam & the full-duplex guys are right. Also, don't forget that your underlying hardware might not be up to the task of saturating 1Gbps/100Mbps as well. Unless you have a decent RAID config, decent SSDs, or some kind of RAM-based drive, you're probably not going to see those kind of transfer rates for extended periods.

I've never seen more than about 50% utilisation on a copper GigE link but I have seen high-90%s on saturated trunk ports between Cisco switches over fibre.