I am setting up a web application which uses SSL for all requests. It needs to have room for scaling and also be highly available. It seems that the "recommended" way to handle this is to setup a pair of Layer 4 load balancers for HA, which pass through to a farm of servers for SSL decryption, which then go to a farm of Layer 7 load balancers, and then finally to the web servers. (See: http://1wt.eu/articles/2006_lb/index_09.html)
I've used HAProxy in the past, so I gave that a try with a setup like this. Basically I had HAProxy in TCP mode => STunnel => HAProxy in HTTP mode => Apache. When I did this, the client IP that was available to apache was that of the 2nd HAProxy server. I could improve that slightly by having STunnel add an X-Forwarded-For header, but that only gave me the IP address of the 1st HAProxy server. I was not able to find a way to let Apache know the original client's IP address.
Based on the documentation I have read, I don't think that HAProxy is capable of passing through the original client's IP address in tcp mode. Is that correct? If so, are there any opensource, software load balancers that can do this? If not, are there any commercial/hardware load balancers that can?
I guess I should also note that this load balancer doesn't, strictly, need to be a layer 4 load balancer. I just needs to be able to pass through the SSL traffic to STunnel (or whatever I end up using for SSL decryption).
As a side note, I tried setting it up without the layer 4 load balancer (i.e. STunnel is the first part of the chain). This solved the client IP problem. However, as predicted in the above article, STunnel was the bottleneck. It started losing requests somewhere between 500 and 600 requests per second. And, of course, this can't scale (while maintaining HA) by just adding more boxes. Based on the requirements I have been given, this app should be able to handle peak traffic of 1000-5000 requests per second.
I highly recommend looking into LVS. http://www.linuxvirtualserver.org/
It has the ability to do normal NAT routing, but also IP tunneling and direct routing which send directly back to the client.
Have you looked at: http://httpd.apache.org/docs/2.3/mod/mod_remoteip.html that will rewrite the IP based on the X-Forwarded-for for header.
The BigIP from F5 is a commercial solution that works very well. X-Forwarded-For is implemented as an http profile option. It isn't cheap though.
Apache TrafficServer can be set up to insert an X-Forwarded-For header and it doesn't cost anything. I've never used it though but it does have a configuration variable: proxy.config.http.append_xforwards_header