nginx reverse proxy - try upstream A, then B, then A again

Key points:

• Don't bother with upstream blocks for failover, if pinging one server will bring another one up - there's no way to tell nginx (at least, not the FOSS version) that the first server is up again. nginx will try the servers in order on the first request, but not follow-up requests, despite any backup, weight or fail_timeout settings.
• You must enable recursive_error_pages when implementing failover using error_page and named locations.
• Enable proxy_intercept_errors to handle error codes sent from the upstream server.
• The = syntax (e.g. error_page 502 = @handle_502;) is required to correctly handle error codes in the named location. If = is not used, nginx will use the error code from the previous block.

Here is a summary:

server {
    listen ...;
    server_name $DOMAINS;

    recursive_error_pages on;

    # First, try "Upstream A"
    location / {
        error_page 418 = @backend;
        return 418;
    }

    # Define "Upstream A"
    location @backend {
        proxy_pass http://$IP:81;
        proxy_set_header  X-Real-IP     $remote_addr;
        # Add your proxy_* options here
    }

    # On error, go to "Upstream B"
    error_page 502 @handle_502;

    # Fallback static error page, in case "Upstream B" fails
    root /home/nginx/www;
    location = /_static_error.html {
        internal;
    }

    # Define "Upstream B"
    location @handle_502 { # What to do when the backend server is not up
        proxy_pass ...;
        # Add your proxy_* options here
        proxy_intercept_errors on;          # Look at the error codes returned from "Upstream B"
        error_page 502 /_static_error.html; # Fallback to error page if "Upstream B" is down
        error_page 451 = @backend;          # Try "Upstream A" again
    }
}

Original answer / research log follow:

Here's a better workaround I found, which is an improvement since it doesn't require a client redirect:

upstream aba {
    server $BACKEND-IP;
    server 127.0.0.1:82 backup;
    server $BACKEND-IP  backup;
}

...

location / {
    proxy_pass http://aba;
    proxy_next_upstream error http_502;
}

Then, just get the control server to return 502 on "success" and hope that code is never returned by backends.

Update: nginx keeps marking the first entry in the upstream block as down, so it does not try the servers in order on successive requests. I've tried adding weight=1000000000 fail_timeout=1 to the first entry with no effect. So far I have not found any solution which does not involve a client redirect.

Edit: One more thing I wish I knew - to get the error status from the error_page handler, use this syntax: error_page 502 = @handle_502; - that equals sign will cause nginx to get the error status from the handler.

Edit: And I got it working! In addition to the error_page fix above, all that was needed was enabling recursive_error_pages!

You could try something like the following

upstream backend {
    server a.example.net;
    server b.example.net backup;
}

server {
    listen   80;
    server_name www.example.net;

    proxy_next_upstream error timeout http_502;

    location / {
        proxy_pass http://backend;
        proxy_redirect      off;
        proxy_set_header    Host              $host;
        proxy_set_header    X-Real-IP         $remote_addr;
        proxy_set_header    X-Forwarded-for   $remote_addr;
    }

}