Home / server / Questions / 697530
Accepted
slm
Asked: 2015-06-09 18:21:41 +0800 CST

Nginx upstream fails over from serverA to serverB, and then back to serverA, why?

  • 772

Background

I have the following Nginx configuration which is load balancing 2 Tomcat servers running on 2 different servers:

   ### http://nginx.org/en/docs/http/ngx_http_upstream_module.html#hash
   upstream backend_w_affinity { 
     hash $request_uri consistent;
     server 192.168.110.11:8080;
     server 192.168.110.12:8080; 
   }

   server {
            listen 80 default_server;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            location ~ ^/path/to/rest/endpoint$ { proxy_pass http://backend_w_affinity; }
  }

When I use the following curl command to drive traffic at Nginx it seems fine.

$ watch "curl -iI http://10.128.13.153/path/to/rest/endpoint"
HTTP/1.1 401 Unauthorized
Server: nginx/1.8.0
Date: Tue, 09 Jun 2015 01:31:13 GMT
Content-Type: application/xml;charset=ISO-8859-1
Content-Length: 231
Connection: keep-alive

NOTE: I realize I'm getting 401's above. There's definitely Tomcat's up and if I change the pathing around so that it points to a static page within Tomcat so that I get 200's it still behaves the same way, so that doesn't seem to be an issue at the moment.

Initially seems fine

With the above curl command running, all my traffic is directed to the first Tomcat server. If I stop Tomcat, then all the traffic fails over to the second Tomcat server.

My Problem

When I bring the first Tomcat server back up, I'm expecting all the traffic to remain on the second server now. However as soon as I bring it back up, the traffic is directed back to the first Tomcat server.

How can I achieve getting Nginx, to leave the traffic on the second server, and not redirect it back?

nginx

1 Answers

  1. Best Answer

    You need to use "sticky" sessions if you want traffic to stick to a server even in the face of changes to the upstream group's membership. Consistent hashing chooses the same server so long as it is available, which is not what you seem to want.

    In your case, you're hashing the URI, which is good for "cache locality" if you are using proxy_cache in nginx. However, consistent hashing means that one server will always be the "first choice" for a given URI, so as soon as it comes back online, nginx will switch back to it.

    There are sticky session modules available for open-source nginx from 3rd parties, or there is a native sticky session option in commercial "NGINX Plus".

    It is possible to do sticky sessions without addtional modules or commercial nginx subscription, but it is a bit ugly. You can use a combination of map, proxy_pass with a variable hostname in the destation, and a fallback location. You also need ot set a unique header in each of your backend servers identifying them (so they sent an "X-Loadbalace-ID" header. This could be part of a session cookie such as a Tomcat JVM Route. We actually still run this configuration in production, as it predates the availability of sticky sessions on nginx.

    Something like this might work for you (untested simplification of what we do):

    upstream backend_w_affinity { 
  #use round-robin
  server 192.168.110.11:8080;
  server 192.168.110.12:8080; 
  }

server {
  listen 80 default_server;
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

  map $http_cookie_lb $mybackend {
    server1 192.168.110.11:8080;
    server2 192.168.110.12:8080;
    #default is something that will fail quickly
    #when no correct load balancing cookie is set
    #putting us into the named location that uses
    #the upstream block
    default 127.0.0.1:65432;
    }

  location ~ ^/path/to/rest/endpoint$ { 
      proxy_pass http://$mybackend;
      #if the backend selected is down, fallback to upstream logic
      error_page 503 = @backend_down;
      }
  location @backend_down { 
      proxy_pass http://backend_w_affinity;
      add_header "Set-Cookie" "lb=$upstream_http_X_Loadbalance_ID";
      }
}
    • 1