I have a Windows Server 2008 R2 x64 server that is AD controller and file server. I have a problem that Windows XP clients experience terribly slow (less than 10Mbps, really less than ten-megabits-per-second) downloading of files from a share.
Server is connected to 1Gbps switch using 1Gbps Nvidia NForce card and client is connected using 100Mbps built-in card.
This slow downloading can also be seen when I've booted client computer from CentOS Linux 5.5 Live-USB and used smbclient for downloading. But downloading from a Samba share on Linux server, which is also connected using 1Gbps link is fast.
What is very strange I've created a pair of programs (attached below) which test plain TCP throughput in C#, and they're performing as expected — at about 89Mbps.
I've disabled firewall on client and I'm using dot_nc_l 21000 > NIL on client and dot_nc [client_ip] < 100m.dat on Windows server. And I get about 9 seconds, when copying the same 100MB file from share takes over 2 minutes.
How to eliminate this?
Some pictures generated with wireshark on Linux client:
Downloading 100MB file from Windows 2008 CIFS file server connected with 1Gbps NIC to Centos 5 Linux client connected with 100Mbps NIC with smbclient:
Downloading 100MB file from Fedora Linux CIFS file server on Samba connected with 1Gbps NIC to Centos 5 Linux client connected with 100Mbps NIC with smbclient (same scale as above):
Here are these programs (linked are compiled using mono's gmcs, require .NET2):
using System;
using System.IO;
using System.Diagnostics;
using System.Net.Sockets;
public class dot_nc
{
public static void Main(string[] args) {
string hostname = args[0];
int port = int.Parse(args[1]);
Stopwatch stopwatch = new Stopwatch();
stopwatch.Start();
TcpClient client = new TcpClient(hostname, port);
stopwatch.Stop();
Console.WriteLine("Connection: {0}ms", stopwatch.ElapsedMilliseconds);
stopwatch.Reset();
stopwatch.Start();
byte[] buffer = new byte[4096];
{
Stream stdin = Console.OpenStandardInput();
NetworkStream netout = client.GetStream();
while ( true ) {
int bytesread = stdin.Read(buffer, 0, buffer.Length);
if ( bytesread <= 0 ) {
break;
}
netout.Write(buffer, 0, bytesread);
}
}
stopwatch.Stop();
Console.WriteLine("Sending: {0}ms", stopwatch.ElapsedMilliseconds);
client.Close();
}
}
using System;
using System.IO;
using System.Diagnostics;
using System.Net;
using System.Net.Sockets;
public class dot_nc
{
public static void Main(string[] args) {
int port = int.Parse(args[0]);
TcpListener server = new TcpListener(IPAddress.Any, port);
server.Start();
TcpClient client = server.AcceptTcpClient();
NetworkStream netin = client.GetStream();
byte[] buffer = new byte[4096];
Stream stdout = Console.OpenStandardOutput();
int processed_bytes = 0;
int processed_chunks = 0;
while ( true ) {
int bytesread = netin.Read(buffer, 0, buffer.Length);
if ( bytesread <= 0 ) {
break;
}
stdout.Write(buffer, 0, bytesread);
processed_bytes += bytesread;
processed_chunks++;
}
netin.Close();
client.Close();
server.Stop();
Console.Error.WriteLine(
"Received: {0} chunks of data of {1} average size",
processed_chunks, processed_bytes/processed_chunks
);
}
}
The problem was caused by:
Because flow control was disabled, Windows was sending packets up to window size in one batch using 1Gbps connection. Since 100Mbps client receive packets much more slowly, almost all data up to window size needed to be buffered by a switch. As this cheap switch has very small buffers (buffer sizes aren't even stated in specifications but it has to be less than 64kB per port, as even disabling window scaling did not help) it had to drop excess packets. Packet loss caused a delay of about 0.25s seen on a graph. But congestion avoidance algorithm, used in File Services, or lack thereof, did not reduce TCP window size, so the next batch of packets wasn't smaller — it congested connection again and again causing congestion collapse.
Standard TCP connections (not File Services) must use different congestion control algorithm and do not get congested repeatably. I suppose treating File Services specially by Windows TCP stack helps in benchmarks against for example Samba.
So the solutions are:
Enable flow control in network adapter properties. It isn't an ideal solution, as any File Services transfer to 100Mbps client will also slow down concurrent transfers to 1Gbps clients to less than 100Mbps speeds.
Or connect 100Mbps clients to an enterprise class switch with much bigger buffers. This is a solution I've used. I have a 10 year old "3Com SuperStack 3 3300 SM" switch with one 1000Base-SX fiber optic gigabit Ethernet MT-RJ port. I bought a Cisco 1000BASE-SX mini-Gbic module (MGBSX1) with LC port for my Linksys gigabit switch and LC/MT-RJ multi-mode fiber patchcord (about $150 for both) and connected all 100Mbps clients to this 3com switch. I've also enabled flow control but it should not cause slowdowns with no 100Mbps client connected.
Thanks to SpacemanSpiff, whose comments helped to resolve this.
Does the Windows server have SMB Signing enabled? SMB signing adds slowness, and is enabled by default on Domain Controllers.
Might it be the 100Mbps card/switch? You mention that the same client works properly when it is on 1Gbps.
Feels like a lower-level network issue. My guesses:
You could try to copy via esetutil (if you have an exchange server there)
Check this: http://blogs.technet.com/b/askperf/archive/2007/05/08/slow-large-file-copy-issues.aspx
This could be an test if you copy large files from client to server or vice versa to test if esetutil get better performance.
also i had similar Problems with a Windows 2008 and Linux Server with an option named NetDMA (last section). This solved my problems (was an Broadcom Network Adapter with teaming)
How to enable and disable NetDMA in Windows Server 2008 To enable or disable NetDMA, follow these steps: Click Start, click Run, type regedit, and then click OK. Locate the following registry subkey, and then click it: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters Double-click the EnableTCPA registry entry. Note If this registry entry does not exist, right-click Parameters, point to New, click DWORD Value, type EnableTCPA, and then press ENTER. To enable NetDMA, type 1 in the Value data box, and then click OK. To disable NetDMA, type 0 in the Value data box, and then click OK. If the EnableTCPA registry entry does not exist, enable the NetDMA functionality. The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these product
from Support Microsoft KB 951037
Check if the BIOS of your servers/clients has a "CPU C State" parameter (probably inside the Power Saving category). If present, try setting that parameter to DISABLE.