Home / server / Questions / 973858
In Process
Christopher Perrin
Asked: 2019-07-04 07:01:18 +0800 CST

Apache Webserver block frontend connection on reverse proxy error

  • 772

We have a problem with a relatively dumb loadbalancer that sits in front of an Apache Webserver. The Apache Webserver again is a reverse proxy for an application server.

The problem starts when the application goes down. This makes the Apache throw an error code.

The problem is that the loadblancer only considers a hard tcp error for removing a server from the pool. This means that error pages will go through the loadbalancer to the user instead of the server just being removed from the pool.

Is it possible to configure Apache to reject a tcp request on backend a backend error?

reverse-proxy

3 Answers

  1. Haproxy can reject tcp connections based on dynamic conditions, including the number of servers available on the backend. The config keyword is tcp-request connection. (Link to docs below.)

    So, maybe put haproxy in front of Apache (I know, I know! Three layers?!?!) or maybe use haproxy instead of Apache.

    Might look like this (not tested):

    frontend http-in
  bind *:80
  mode http
  tcp-request connection reject if nbsrv(appfarm) lt 1
  default_backend appfarm

backend appfarm
  server appsrv01 127.0.0.1:8888 check

    I'd be happy to flesh this out a bit if you think you could use it. It's an interesting problem.

    • 2

  2. This will probably not work with apache httpd itself. The problem is that your failed application makes apache httpd generate an error. This error only happens when you actually request an URL from that application. Apache httpd needs the URL to decide wether this is an error or not and to get the URL from the client it needs the TCP connection.

    What you can do as a workaround: You can write a rather simple script that reads the ErrorLog stream on stdin and then blocks requests from your loadbalancers (maybe iptables) once the errors start to happen. That script can then be configured as a sink of apache httpd's error log using the ErrorLog directive.

    This would disable an apache httpd backend server, but it will never reenable it automatically.

    • 0

  3. I will move the logic to the load balancer, if it is not possible to replace the existent load balancer, install a small nginx in each Apache server to proxy the connection.

    [User] - [Load Balancer] - [Nginx] - [Apache]

    By using nginx as load balancer or reverse proxy you have access to the upstream responses (In your case, Apache), and you can make rules that evaluate the response code before replying to the end client:

    $upstream_status keeps status code of the response obtained from the upstream server. Status codes of several responses are separated by commas and colons like addresses in the $upstream_addr variable. If a server cannot be selected, the variable keeps the 502 (Bad Gateway) status code.

    Nginx example rules

    Call load balancer API to remove node, if available:

    if ($upstream_status != 200) {
    return 301 return 301 http://loadbalance/api/remove/node;
}

    Call local script (please review security) to kill Apache and tell user to reload page (it will use another backend next time). It uses ngx_http_lua_module.

    if ($upstream_status != 200) {
   content_by_lua_block {
     os.execute("/bin/killapache.sh")
   } 
   return 301 https://$host$request_uri;
}
    • -1