Questions[503-error](server)

I have an unusual problem (as everyone does on this site). I have HAProxy listening on port 80 and 443. I am offloading SSL encryption to HAproxy and passing all traffic to my web servers on port 80. There is a firewall in front of my HAProxy server with NAT rules pointing to my internal listening IP addresses of HAProxy. We are getting an unusual 503 message for random IP source addresses over SSL on port 443. Other random IP addresses are working fine over SSL. Our service is an API and most traffic goes through but a very small percentage gets a 503.

Here is an HAProxy log entry that works and one that doesn't working:

localhost haproxy[5404]: XXX.XXX.XXX.XXX:54787 [15/Jun/2016:22:46:57.592] https_in_ssl~ http_www2/web1 32/0/0/232/264 200 747 - - ---- 5/4/0/1/0 0/0 "POST /webservices/ourService.asmx HTTP/1.1"

not working:

localhost haproxy[5404]: XXX.XXX.XXX.XXX:55494 [15/Jun/2016:22:46:39.514] https_in_ssl~ https_in_ssl/<NOSRV> -1/-1/-1/-1/227 503 212 - - SC-- 3/2/0/0/0 0/0 "POST /webservices/ourService.asmx HTTP/1.0"

One thing I notice is the front-end and back-end are the same in the not working log entry.

Here is my config file:

peers prodHAproxypeers
        peer haproxylb1 10.0.0.145:1024
        peer haproxylb2 10.0.0.146:1024

global
        log     127.0.0.1 local0
#       log /dev/log local0
#       log /dev/log local1 notice
        chroot /var/lib/haproxy
        stats socket /var/lib/haproxy/stats
        stats timeout 30s
        tune.ssl.default-dh-param 2048
        user haproxy
        group haproxy
        daemon

defaults
        log global
        mode http
        option httplog
        option dontlognull
        option redispatch
        option forwardfor
        option http-server-close
        maxconn 5000
        timeout connect 5s
        timeout client 5h
        timeout server 5h
        timeout queue 30s
        timeout http-request 5s
        timeout http-keep-alive 15s

listen stats *:1936
        mode http
        stats enable
        stats hide-version
        stats realm Haproxy\ Statistics
        stats uri /haproxy_stats
        stats auth admin:hardPassword
        stats admin if TRUE

frontend http_in
        bind *:80
        ###Add new acl and use_backend entry for each new site
        ###new backend sections will be needed as well
        acl is_www1 hdr(host) -i www1.domainname.com
        acl is_www2 hdr(host) -i www2.domainname.com
        acl is_www3 hdr(host) -i www3.domainname.com
        acl is_www4 hdr(host) -i www4.domainname.com
        acl is_wildcardwww hdr_end(host) -i domainname.com
        use_backend http_www1 if is_www1
        use_backend http_www2 if is_www2
        use_backend http_www3 if is_www3
        use_backend http_www4 if is_www4
        use_backend http_www5 if is_www5
        option forwardfor
        option http-server-close

frontend https_in_ssl
        mode http
        bind *:443 ssl crt /etc/ssl/private/ no-sslv3
        reqadd X-Forwarded-Proto:\ https
        use_backend http_www1 if { ssl_fc_sni www1.domainname.com }
        use_backend http_www2 if { ssl_fc_sni www2.domainname.com }

        acl is_ssl_www5 hdr_end(host) -i domainname.com
        use_backend http_www5 if is_ssl_www5


backend http_www1
        balance source
        cookie SRV_ID prefix
        stick-table type ip size 1m expire 6h peers prodHAproxypeers
        stick on src
        ###This site does not use host header - only the page name is needed###
#       option httpchk HEAD /Default.aspx
        ###Added host header so haproxy can route around NLB - use below for checking###
        option httpchk HEAD /Default.aspx HTTP/1.1\r\nHost:\ www1.domainname.com
        server p-websvr01 10.0.0.10:80 cookie pweb1 weight 45 check
        server p-websvr02 10.0.0.11:80 cookie pweb2 weight 45 check
        server p-websvr03 10.0.0.115:80 cookie pweb3 weight 5 check
        server p-websvr04 10.0.0.118:80 cookie pweb4 weight 5 check

backend http_www2
        balance roundrobin
        stick-table type ip size 1m expire 6h peers prodHAproxypeers
        stick on src
        ###This site uses host headers so this type of check is required###
        option httpchk HEAD /default.htm HTTP/1.1\r\nHost:\ www2.domainname.com
        server p-websvr01 10.0.0.10:80 cookie pweb1 weight 45 check
        server p-websvr02 10.0.0.11:80 cookie pweb2 weight 45 check
        server p-websvr03 10.0.0.113:80 cookie pweb3 weight 5 check
        server p-websvr04 10.0.0.116:80 cookie pweb4 weight 5 check

backend http_www3
        balance roundrobin
        cookie SRV_ID prefix
        stick-table type ip size 1m expire 6h peers prodHAproxypeers
        stick on src
        ###This site does not use host header - only the page name is needed###
        option httpchk HEAD /login.aspx HTTP/1.1\r\nHost:\ www3.domainname.com
        server p-websvr01 10.0.0.10:80 cookie pweb1 weight 45 check
        server p-websvr02 10.0.0.11:80 cookie pweb2 weight 45 check
        server p-websvr03 10.0.0.113:80 cookie pweb3 weight 5 check
        server p-websvr04 10.0.0.116:80 cookie pweb4 weight 5 check

backend http_www4
        balance roundrobin
        cookie SRV_ID prefix
        stick-table type ip size 1m expire 6h peers prodHAproxypeers
        stick on src
        ###This site uses host header so this type of check is required###
        option httpchk HEAD /default.aspx HTTP/1.1\r\nHost:\ www4.domainname.com
        server p-websvr01 10.0.0.10:80 cookie pweb1 weight 45 check
        server p-websvr02 10.0.0.11:80 cookie pweb2 weight 45 check
        server p-websvr03 10.0.0.113:80 cookie pweb3 weight 5 check
        server p-websvr04 10.0.0.116:80 cookie pweb4 weight 5 check

backend http_www5
        balance roundrobin
        cookie SRV_ID prefix
        stick-table type ip size 1m expire 6h peers prodHAproxypeers
        stick on src
        ###This site uses host header so this type of check is required###
        option httpchk HEAD /default.aspx HTTP/1.1\r\nHost:\ www5.domainname.com
        server p-websvr01 10.0.0.10:80 cookie pweb1 weight 45 check
        server p-websvr02 10.0.0.11:80 cookie pweb2 weight 45 check
        server p-websvr03 10.0.0.115:80 cookie pweb3 weight 5 check
        server p-websvr04 10.0.0.117:80 cookie pweb4 weight 5 check

I am using HAproxy in front of an app that is on multiple other servers. I would like to set up a vhost in a local nginx instance that only serves a 503 for every request (to use as the 'backup' server when all backends are down). I know how to set the custom error page (as in How can I make Nginx return HTTP 503 when my proxied app server is down?) but how I can make it so that every request to nginx returns a 503?

We're getting "HTTP Error 503. The service is unavailable.". Googling has provided lots of near hits, but none have worked so far.

The flow is: - confirm all app pools started - open browser to http://localhost, get 503 error - app pool is now stopped

The event log has 5 warnings of

"A listener channel for protocol 'http' in worker process 'nnnn' serving application pool 'DefaultAppPool' reported a listener channel failure. The data field contains the error number."

with the source WAS, details: ProtocolID: http FC2A0780 followed by 1 error

"Application pool 'DefaultAppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool."

Event ID 5002 Windows 2008 Enterprise, IIS7

All patches current as of 12/7/2011

Default Web Site, using DefaultAppPool

DefaultAppPool - tried Classic, Integrated modes, .NET 2 & 4, 32bit on and off, Identity of ApplicationPoolIdentity and an Administrator account, Ping enabled and not, Rapid-Fail enabled and not. Nothing in wwwroot except iisstart.htm, welcome.png and aspnet_client folder. Reinstalled IIS, .NET Framework.

Turning off Rapid-Fail simply resulted in never ending warnings and no logged error, but still got the 503 on the client.

The machine does not have RPC over HTTP or Sharepoint or Exchange installed, but does have SQL Server 2008 R2.

We've tried dozens of configurations, but all result in the same errors.

Any ideas?

I have a setup with Apache2 as a front-end server for multiple python apps served by gunicorn. My Apache2 setup using mod_proxy looks like this:

<VirtualHost *:80>
    ServerName example.com
    UseCanonicalName On
    ServerAdmin webmaster@localhost

    LogLevel warn
    CustomLog /var/log/apache2/example.com/access.log combined
    ErrorLog /var/log/apache2/example.com/error.log
    ServerSignature On

    Alias /media/ /home/example/example.com/pysrc/project/media/

    ProxyPass /media/ !
    ProxyPass / http://127.0.0.1:4711/
    ProxyPassReverse / http://127.0.0.1:4711/
    ProxyPreserveHost On
    ProxyErrorOverride Off
</VirtualHost>

Generally, this setup works pretty well. I have one problem though: When I restart the gunicorn process (takes 2-5 seconds) and there is a request from Apache, that request will fail with a 503 error. So far so good. But Apache keeps returning 503 errors, even after the gunicorn process is back up. It resumes serving content from the proxied server only after a full restart of Apache.

Is there a way around this behaviour?

I have an nginx web server sitting in front of a Jetty Java app server running a web application.

I'd like to configure nginx to return a HTTP 503 if the proxied Jetty server is down.

This way when I bring the app server down for maintenance, of if it happens to crash, a 503 is returned to any clients letting them know the site is temporarily unavailable.

However, with the basic configuration, Nginx returns a HTTP 502 Bad Gateway when it can't connect to the proxy_pass location.

Is it possible to configure nginx to report a 503 instead?