What would be the best approach for load-balancing at least 2-3 Windows 2008 R2 IIS webservers running a multitude of .NET applications? My choices appear to be:
1) Hardware-based network device load balancer, like a Cisco CSS
2) Windows NLB
3) Some sort of linux based proxy, either haproxy or other
The three servers sit as VMs on a vSphere farm, so I have the ability to clone to up the instance count in times of high load. I control the switch that the vSphere hosts are plugged into (Cisco 3750), but don't control the switching/routing infrastructure beyond that to the clients.
(1) Is too expensive, and probably overkill for my needs. I've included this in case someone figures out a cunning way to do it on my existing network kit, which I doubt.
(2) would seem to be the obvious "built-in" option, but seems to be quite fiddly messing around with network interfaces, multicast, and generally other things that seem to be needlessly complex. It's also fairly stupid, in that it can't remove hosts from the pool if they start throwing 500 errors or otherwise go wrong
(3) is the most interesting option, as it would appear to offer the most flexibility and customizability, but without having to mess around with the network. However, while I'm familiar with the reverse-proxy capabilities of lighttpd etc, I'm not that well read on other options like HAProxy, which might be able to offer a lot more.
Which would you go for, and is there anything I've not thought of?
At Stackoverflow we use HAProxy to balance against our Windows Server 2008 R2 IIS 7 web servers with great sucess. We love HAProxy and find it to be very flexible.
How much traffic are you talking about? Citrix offers a free version of their Netscaler load balancer as a virtual machine for ESX, Hyper-V, and XenServer. It is called Netscaler VPX Express and offers up to 5Mb of throughput for load balancing. It also includes a 10 user SSL VPN license as a remote access solution. I've been using it internally with great success.
Here's a recent blog post by Citrix that discusses the recent bump of the free VM from 1Mb to 5Mb: http://community.citrix.com/display/ocb/2011/02/28/NetScaler+VPX+Express+Sets+Your+Networks+(Even)+Free(-er+than+Before!)
I have used NLB for IIS webservices for years with good experience. Its not really a load balancer, more of a redundancy tool. The load is not shared intelligently between the servers but it is really really nice to be able to take down one server for upgrades and have everything running with full uptime. I do recommend you start with NLB and see if that is good enough for you. Its free and very simple after all.
Not really an answer, but this (essay?) might give you some good background information to help in your choice: Making Applications Scalable With Load Balancing