I am looking at a performance problem of an asp.net web site.
It appears that when we hit 1000 concurrent connections per server things slow down.
We have several web servers behind a hardware load balancer.
Is there any default setting that is 1000 connections?
What hardware load balancer are you using? Does it perhaps not have a license limit of 1000 concurrent connections? I recall in the past network appliances like Cisco PiX firewalls having a "maximum number of concurrent connections" limit imposed by the license key.
Check your ASP.NET
<processModel \>
section for the framework. c:\windows\microsoft.net\framework(64)(version)\config\web.config and machine.config:MSDN: <processModel> Element
That has some limits on the request queues and threads.
However, the bigger issue is why you have 1000 concurrent connections. If you can get users of the server quicker then it won't build that high. If you can find a couple problem pages and speed up performance then the server will be able to keep up and the 1000 limit will be a non-issue. Generally speaking, 1000 concurrent connection is a LOT of traffic and each web server doesn't normally intend to have to handle that much.
It looks like this was a problem with the TCP Chimney offload in windows 2008 R2.
http://weblogs.asp.net/steveschofield/archive/2010/08/31/troubleshooting-iis-7-network-performance-issues-and-tcp-chimney-offload-receive-side-scaling-and-network-direct-memory-access.aspx
Basically it offloads work to the network card, but if the network card gets maxed out things slow down, but all the normal performance values show little load.