Geoff Dalgas's questions

We run a Netgear GS724Tv3 switch and part of the configuration per port is an option for "Lower Power Mode". The documentation for the switch lists this feature as a "Green Ethernet Configuration" feature:

A cable test is performed when this is enabled and the port link is up at 1 Gbps: if the cable is less than 10 m, PHYs are placed in low power mode (nominal power)

We have purchased a set of short patch cables of lengths less than 1 m, and we might possibly be having issues with a few of them (testing is still under way). We have this feature disabled at the moment.

Should this feature be enabled if the cables between a NIC / SWITCH are less than 10 m where power consumption is not a concern? As a follow-up question, is there a minimum cable length requirement for Cat 6 gigabit?

TL;DR version: Turns out this was a deep Broadcom networking bug in Windows Server 2008 R2. Replacing with Intel hardware fixed it. We don't use Broadcom hardware any more. Ever.

We have been using HAProxy along with heartbeat from the Linux-HA project. We are using two linux instances to provide a failover. Each server has with their own public IP and a single IP which is shared between the two using a virtual interface (eth1:1) at IP: 69.59.196.211

The virtual interface (eth1:1) IP 69.59.196.211 is configured as the gateway for the windows servers behind them and we use ip_forwarding to route traffic.

We are experiencing an occasional network outage on one of our windows servers behind our linux gateways. HAProxy will detect the server is offline which we can verify by remoting to the failed server and attempting to ping the gateway:

Pinging 69.59.196.211 with 32 bytes of data:
Reply from 69.59.196.220: Destination host unreachable.

Running arp -a on this failed server shows that there is no entry for the gateway address (69.59.196.211):

Interface: 69.59.196.220 --- 0xa
Internet Address      Physical Address      Type
69.59.196.161         00-26-88-63-c7-80     dynamic
69.59.196.210         00-15-5d-0a-3e-0e     dynamic
69.59.196.212         00-21-5e-4d-45-c9     dynamic
69.59.196.213         00-15-5d-00-b2-0d     dynamic
69.59.196.215         00-21-5e-4d-61-1a     dynamic
69.59.196.217         00-21-5e-4d-2c-e8     dynamic
69.59.196.219         00-21-5e-4d-38-e5     dynamic
69.59.196.221         00-15-5d-00-b2-0d     dynamic
69.59.196.222         00-15-5d-0a-3e-09     dynamic
69.59.196.223         ff-ff-ff-ff-ff-ff     static
224.0.0.22            01-00-5e-00-00-16     static
224.0.0.252           01-00-5e-00-00-fc     static
225.0.0.1             01-00-5e-00-00-01     static

On our linux gateway instances arp -a shows:

peak-colo-196-220.peak.org (69.59.196.220) at <incomplete> on eth1
stackoverflow.com (69.59.196.212) at 00:21:5e:4d:45:c9 [ether] on eth1
peak-colo-196-215.peak.org (69.59.196.215) at 00:21:5e:4d:61:1a [ether] on eth1
peak-colo-196-219.peak.org (69.59.196.219) at 00:21:5e:4d:38:e5 [ether] on eth1
peak-colo-196-222.peak.org (69.59.196.222) at 00:15:5d:0a:3e:09 [ether] on eth1
peak-colo-196-209.peak.org (69.59.196.209) at 00:26:88:63:c7:80 [ether] on eth1
peak-colo-196-217.peak.org (69.59.196.217) at 00:21:5e:4d:2c:e8 [ether] on eth1

Why would arp occasionally set the entry for this failed server as <incomplete>? Should we be defining our arp entries statically? I've always left arp alone since it works 99% of the time, but in this one instance it appears to be failing. Are there any additional troubleshooting steps we can take help resolve this issue?

THINGS WE HAVE TRIED

I added a static arp entry for testing on one of the linux gateways which still didn't help.

root@haproxy2:~# arp -a
peak-colo-196-215.peak.org (69.59.196.215) at 00:21:5e:4d:61:1a [ether] on eth1
peak-colo-196-221.peak.org (69.59.196.221) at 00:15:5d:00:b2:0d [ether] on eth1
stackoverflow.com (69.59.196.212) at 00:21:5e:4d:45:c9 [ether] on eth1
peak-colo-196-219.peak.org (69.59.196.219) at 00:21:5e:4d:38:e5 [ether] on eth1
peak-colo-196-209.peak.org (69.59.196.209) at 00:26:88:63:c7:80 [ether] on eth1
peak-colo-196-217.peak.org (69.59.196.217) at 00:21:5e:4d:2c:e8 [ether] on eth1
peak-colo-196-220.peak.org (69.59.196.220) at 00:21:5e:4d:30:8d [ether] PERM on eth1

root@haproxy2:~# arp -i eth1 -s 69.59.196.220 00:21:5e:4d:30:8d
root@haproxy2:~# ping 69.59.196.220
PING 69.59.196.220 (69.59.196.220) 56(84) bytes of data.
--- 69.59.196.220 ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6006ms

Rebooting the windows web server solves this issue temporarily with no other changes to the network but our experience shows this issue will come back.

Swapping network cards and switches

I noticed the link light on the port of the switch for the failed windows server was running at 100Mb instead of 1Gb on the failed interface. I moved the cable to several other open ports and the link indicated 100Mb for each port that I tried. I also swapped the cable with the same result. I tried changing the properties of the network card in windows and the server locked up and required a hard reset after clicking apply. This windows server has two physical network interfaces so I have swapped the cables and network settings on the two interfaces to see if the problem follows the interface. If the public interface goes down again we will know that it is not an issue with the network card.

(We also tried another switch we have on hand, no change)

Changing network hardware driver versions

We've had the same problem with the latest Broadcom driver, as well as the built-in driver that ships in Windows Server 2008 R2.

Replacing network cables

As a last ditch effort we remembered another change that occurred was the replacement of all of the patch cords between our servers / switch. We had purchased two sets, one green of lengths 1ft - 3ft for the private interfaces and another set of red cables for the public interfaces. We swapped out all of the public interface patch cables with a different brand and ran our servers without issue for a full week ... aaaaaand then the problem recurred.

Disable checksum offload, remove TProxy

We also tried disabling TCP/IP checksum offload in the driver, no change. We're now pulling out TProxy and moving to a more traditional x-forwarded-for network arrangement without any fancy IP address rewriting. We'll see if that helps.

Switch Virtualization providers

On the off chance this was related to Hyper-V in some way (we do host Linux VMs on it), we switched to VMWare Server. No change.

Switch host model

We've reached the end of our troubleshooting rope and are now formally involving Microsoft support. They recommended changing the host model:

• http://en.wikipedia.org/wiki/Host_model
• http://technet.microsoft.com/en-us/magazine/2007.09.cableguy.aspx

We did that, and we also got some unpublished kernel hotfixes which were presumably rolled into 2008 R2 SP1. No fix.

Replacing network card hardware

Ultimately, replacing the Broadcom network hardware with Intel network hardware fixed this issue for us. So I am inclined to think that the Broadcom Windows Server 2008 R2 drivers are at fault!

http://blog.serverfault.com/post/broadcom-die-mutha/

We currently have our DNS SOA record set to the following for stackoverflow.com:

    primary name server = ns1.p19.dynect.net
    serial  = 2009090909
    refresh = 3600 (1 hour)
    retry   = 600 (10 mins)
    expire  = 604800 (7 days)
    default TTL = 60 (1 min)

Are there better choices for our refresh / retry / expire / default TTL for a site like stackoverflow.com which receives close to 1M pageviews per day?

We have recently implemented HAProxy for stackoverflow.com. We decided on using TProxy to maintain the source address for clients connecting so our logs and other IIS modules which depend on the client IP address would not require modification. So the packets arrive spoofed as if they have come from an external internet IP address, when in reality they came from a local 192.168.x.x HAProxy IP on our local network.

Both of our web servers have two NICs - one routable class B address on the public internet with a static IP, DNS, and default gateway and one private unroutable class C address configured with a default gateway pointed at the private IP for HAProxy. HAProxy has two interfaces - one public and one private and performs the job of routing packets transparently between interfaces and directing traffic to the appropriate web server.

Ethernet adapter Internet:

   Description . . . . . . . . . . . : network card #1
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 69.59.196.217 (Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.240
   Default Gateway . . . . . . . . . : 69.59.196.209
   DNS Servers . . . . . . . . . . . : 208.67.222.222
                                       208.67.220.220
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Private Local:

   Description . . . . . . . . . . . : network card #2
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 192.168.0.2 (Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.0.50
   NetBIOS over Tcpip. . . . . . . . : Enabled

We have disabled automatic metrics on each of the web servers and assigned the routable public class B a metric of 10 and our private interface a metric of 20.

We have also set both of these registry keys:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"DeadGWDetectDefault"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"EnableDeadGWDetect"=dword:00000000

About twice per day we see issues where one of the web servers cannot contact DNS or make connections out to any other servers on the public internet.

We suspect dead gateway detection is falsely detecting an outage on the public gateway and is switching all traffic to the private gateway which has no DNS access at this point but have no way of verifying this.

1. Is there a way to know if dead gateway detection is running or even an option in Windows 2008 server?

2. If so, is there a way to disable dead gateway detection in Windows 2008 server?

3. If not could there be other reasons that we lose the ability to resolve DNS or connect out for a short time?

When I configure a new remote desktop connection to connect to servers I often forget to uncheck the option inside of local devices and resources for remote printers. This creates exceptions in the event log for the server I connect of source: TerminalServices-Printers for the printer drivers I use for PDF document conversion.

I'd love to know if it's possible to disable remote printers by default either on the client remote desktop connection options, or on the server so these exceptions no longer occur.

We have an SMTP only mail server behind a firewall which will have a public A record of mail.. The only way to access this mail server is from another server behind the same firewall. We do not run our own private DNS server.

Is it a good idea to use the private IP address as an A record in a public DNS server - or is it best to keep these server records in each servers local hosts file?

Microsoft has a lot of documentation about their somewhat new load balancing and request routing module for IIS7 located here: http://blogs.iis.net/bills/archive/2009/02/16/iis7-request-routing-and-load-balancing-module-released.aspx, but I'd like to know if anyone has experience using it in production. What are a few pros / cons to using this module instead of another solution such as HAProxy?