Porch's questions

I have a really strange one.

I have packet loss with Excessive 'TCP Dup ACK' & 'TCP Fast Retransmission' when I download files (and only download) from two different Windows 2008 servers. Upload speed is fine.

This ONLY occurs if the client computers(Win7) is connected at 100mb/s. At 1GB, no errors and I get full speed. If I set the client nic to 100Mb/s, I get a lot of 'TCP Dup' errors and the download speed drops to around 2-5MB/s. Upload speed is 10MB/s or above.

This only happens to the Windows 2008 Server boxes (Dell, but different hardware). This problem does not occur if I transmit between the Win7 clients and the Linux servers.

It's like Server 2008 is unable to scale the TCP window properly, overloads the switch or something, then pauses traffic for a bit.

Parts of the network run at 100Mb/s due to older equipment, so this is really causing a problem in some buildings.

I have uploaded a pcap file from the client here. https://dl.dropboxusercontent.com/u/24907255/slow.pcap.gz

It shows a 50MB file being written to the server, then read back from the server with the errors.

Thanks for any help. I am stumped.

11/28/13 More Information.

I shutdown the entire network so that only one client and one server are on the network. No change in the problem.

If I set every interface, server, client and Cisco 2960 switch to 100Mbs full, then the problem goes away. If I set the server and switch interface auto or 1Gbs, the problem is back.

If I bypass the switch with a Netgear 10/100 switch and set both client and server to auto, I have no problems.

I did discover this. In the normal setup, with server to switch at 1Gbs, I plug in the Netgear 10/100 switch between the client and Cisco switch, my speed problem is even worse. Speeds go from 5-7MB/s to 2-3MB/s, and yes I have tried fixed and auto network speeds. This would explain why some of the buildings that have a 2 switch hop between them and the main Cisco switch have more of a speed problem.

On to pinging. With everything at 1GB/s, I can ping a full TCP payload, ping -l 65500 and it works. With the client at 100Mbs, the max size I can ping is 17752. Anymore and it fails, to the Windows servers only, no problem on the Linux boxes. With the Netgear 10/100 between the server and client, no problems pinging at 65500.

Update 3

I swapped in a PowerConnect 2748 switch. Same problem with the server at 1Gbs and the client at 100Mbs. I can ping over 17752 now tho. Strange. So I don't think it's the Cisco switch.

Update 4. I am trying to get some hard numbers by using ipref. All systems connected to the same switch, with the client set to 100Mbs and running the command ipref.exe -c -u -b 10m. So sending to the server. One server is 2008 with no load on it right now, other is a Ubuntu with a load avg of .20.

At 10m

• Linux jitter 0.022ms, packet loss is 0/8505
• Server 2008 jitter 1.859, packet loss 68/8505

Pushing it to 100m

• Linux jitter 0.445, packet loss 0/26634
• Server 2008 jitter 0.542, packet loss 94/26596

Now for stats sending TO the client at 10m

• Linux jitter 0.271 ms, 0/ 8500 (0%) 1 datagrams received out-of-order
• Server 2008 jitter .063, 20/8505 (0.24%)

Pushing it to 100m

• Linux jitter 0.230 ms 4083/85443 (4.8%), 1 datagrams received out-of-order, 95.7Mbs
• Server 2008 jitter 0.237, 28174/81718 (47%), 51.1mbs

So Server 2008 is poor in general, but you can see the huge packet loss 47% when the connection is pushed to the clients 100mbs limit.

Update 5.

When I tested with the PowerConnect 2748 switch, I used different cat5 cable between the server and switch and client and switch. This should rule out cabling or switch issues.

I have two Windows 2008 Servers in this environment, installed at different times, and on different hardware. The only thing they share is a Broadcom branded nic, but the chipset is different. Both experience the same problem, but I am doing my main testing on one so in case something goes wrong, the other will still work.

The one server has a built on BCM5709C with two ports, and an add-on card, pci express I think, card also with the same BCM5709C chipset and two ports. I have tried all of them and the problem still exist. So this should rule out any hardware problems.

Update 6 12/3/13 I installed the Intel nic. No change. I played around with the ctcp settings and no change there. I even turned off SMB2 and no difference.

I did some more testing at 100Mbs Copying a 3GB ISO image TO the server, drag and drop, averages out at 10MB/s. Copying the same 3GB ISO image FROM the server, averages out at 6.3MB/s.

With all network interfaces set to Auto and at 1Gbs. Copying the ISO TO the server, averages 101MB/s Copying the ISO FROM the server, averages 57MB/s

So read speeds from the server are almost half the write speeds.

I have been tasked with setting up and supporting some computers for some people who have sociological problems. Troubled teens/adults and the like. They need computers to do very basic tasks like web browse, but they know little about computers and will delete and destroy anything they can without care.

So I am building some cheap white boxes and plan to put a locked down version of Ubuntu on it with some built in recovery options. Any suggestions on a ChromeOS like Distro would be appreciated.

My big problem is remote administration. They will not have a fixed IP and I won't control anything about the network. So port forwarding is out.

An option is to buy an EC2 and have each client connect with OpenVPN when they come online. I could then SSH in and fix problems. But EC2 is not free and I will not be making any money from this.

My preferred way is BotNet style. I have a friend that runs a private IRC server. I could set up a channel and have all the clients log in and I could issue commands. A remote bash shell to IRC gateway would be nice.

I do something like this only over XMPP with Nagios with my servers. It's real handy to quickly look down a list and see who is online, who is having problems, and issue commands.

Anyone know of such an IRC bot like a BotNet? Or any better way to do this?

As part of a larger complex, is a small office building with 4 computers. It has a single cat5 run to a non-managed switch that has 4 computers connected to it.

Since I can't ping the switch, the only way I can tell if that network is up is by pinging the computers. But the ladies in that office tend to turn the computers off on holidays, weekend, when they fell like it, etc. 99% of the time, at least 1 computer is on.

What I want to do is have Nagios send a notification only if all 4 computers go missing. Aka, the switch died or a FSB (fiber seeking backhoe) had some fun in the area. I don't think Nagios host check dependencies will do this from what I have been reading.

Any other way to do this?

I setup a small box with Server 2003 64bit to be used as a webserver and email server for a small school. Real simple stuff for a few users. A simple website and a handful of emails. rDNS and spf records setup and pass every test I found including test at dnsstuff.com.

Email sending to almost every email address (google, hotmail, aol, whatever) works. However, with one domain, I get an bounce back with the error.

550 Requested action not taken: mailbox unavailable

It's another school running Exchange judging from some packet sniffing with WireShark. Every email on this domain I have tried sending to gives this error. The email address is valid as I can send to it from my personal, and gmail account without a problem.

Does anyone know of some anti-spam software that gives an 550 error like the above? What else could this be?

Thanks for any suggestions.

Packet capture of the two servers communicating look like this.

220 <server snip> Microsoft ESMTP MAIL Service, Version: 6.0.3790.3959 ready at  Sat, 2 Oct 2010 12:48:17 -0700 
EHLO <email snip>
250-<server snip> Hello [<ip snip>]
250-TURN
250-SIZE
250-ETRN
250-XXXXXXXXXX
250-DSN
250-ENHANCEDSTATUSCODES
250-8bitmime
250-BINARYMIME
250-XXXXXXXX
250-VRFY
250-X-EXPS GSSAPI NTLM LOGIN
250-X-EXPS=LOGIN
250-AUTH GSSAPI NTLM LOGIN
250-AUTH=LOGIN
250-X-LINK2STATE
250-XXXXXXX
250 OK
MAIL FROM: <email snip>
250 2.1.0 <email snip>....Sender OK
RCPT TO:<email snip>
250 2.1.5 <email snip> 
DATA
354 Start mail input; end with <CRLF>.<CRLF>
<email body here>
.

550 Requested action not taken: mailbox unavailable
QUIT
221 Goodbye