Home / server / Questions / 991016
In Process
olivierg
Asked: 2019-11-08 12:19:49 +0800 CST

Layer 4 Load balancer in front of many apache frontends serving tomcat backends that can crash

  • 772

i have a setup with :

HAProxy acting as a layer 4 LB (just forwarding requests, passthrough, TCP mode for all frontend/backend) to multiple Apache frontends. Each of these apache frontends is sending the traffic to a tomcat backend where an application is running.

Now, if one of these tomcats/applications is crashed, the corresponding apache frontend reports an error 503 (which is normal), others are OK.

problem is : as HAProxy is just load balancing to each Apache randomly (based on source IP in my case), some users get the 503 errors (pointing to a tomcat not working), while others are working (pointing to a tomcat that is OK)

is there a way to detect that in any way ? (and ensure users are only redirected to a working frontend/backend)

i suppose i can't do anything on the HAproxy side as HAproxy is forwarding to the "Apache" backends (and reports them as healthy as port 443 is OK) and does not see the real backends (the tomcat servers). i suppose i have to do something on the apache frontends (such as "if there is a 503, stop apache" or something).

any idea on how to handle this ?

thanks !

ps : let me know if you want to see the config

apache-2.4

1 Answers

  1. You need to define a health check for your backends.

    Find some simple request that tomcat can serve without incurring too much processing overhead; generally something like /img/empty.gif will be used. Point is that this can be checked to give a HTTP status 200 when tomcat is up and running, and some other error when tomcat is not working.

    Modify your haproxy.conf:

    listen app1 0.0.0.0:8080
  mode http
  option httpchk GET /img/empty.gif HTTP/1.0
  http-check expect status 200
  server backend1 10.0.0.1:443 check fall 3 rise 2 ssl verify none
  server backend2 10.0.0.2:443 check fall 3 rise 2 ssl verify none

    The thing you need to add is the option httpchk and http-check lines, I assume you already have the check in the server line.

    If a server does not respond with status 200 when the httpchk is performed, then it will be taken out of the pool. It will be added back when it responds normally again.

    • 1