I'm testing running HAProxy as a dedicated load balancer behind Apache 2.2, replacing our current configuration where we use Apache's load balancer. In our current, Apache only, set-up if all the backend (origin) servers are down Apache will serve a 503 service unavailable message. With HAProxy I get a 502 bad gateway response.
I'm using a simple reverse proxy rewrite rule in Apache
RewriteRule ^/(.*) http://127.0.0.1:8000/$1 [last,proxy]
In HAProxy I have the following (running in default tcp mode)
defaults
log global
option tcp-smart-accept
timeout connect 7s
timeout client 60s
timeout queue 120s
timeout server 60s
listen my_server 127.0.0.1:8000
balance leastconn
server backend1 127.0.0.1:8001 check observe layer4 maxconn 2
server backend1 127.0.0.1:8001 check observe layer4 maxconn 2
Testing connecting directly to the load balancer when the backend servers are down:
[root@dev ~]# wget http://127.0.0.1:8000/ test.html
--2012-05-28 11:45:28-- http://127.0.0.1:8000/
Connecting to 127.0.0.1:8000... connected.
HTTP request sent, awaiting response... No data received.
So presumably this is down to the fact that HAProxy accepts the connection and then closes it.
In TCP mode, haproxy does not emit any status code so the only remaining point clearly is apache. I think that it's simply because haproxy accepts then closes the connection that makes apache return a 502, which is expected.
So the behaviour you're observing is correct. Anyway it's generally better to work in HTTP mode. I also suggest that you enable "option httplog" which will provide you with very detailed logs, and "option http-server-close" to take advantage of apache's ability to maintain keep-alive with haproxy, it will significantly reduce local source port consumption on the machine.
I couldn't get this working in tcp mode but if you switch to http mode then you do get the 503