Home / server / Questions / 185894
In Process
Martin
Asked: 2010-09-30 09:35:12 +0800 CST

Proxy Error 502 "Reason: Error reading from remote server" with Apache 2.2.3 (Debian) mod_proxy and Jetty 6.1.18

  • 772

Apache is receiving requests at port :80 and proxying them to Jetty at port :8080

The proxy server received an invalid response from an upstream server
The proxy server could not handle the request GET /.

My dilemma: Everything works fine normally (fast requests, few seconds or few tens of seconds long requests are processed ok). Problems occur when request processing takes long (few minutes?).

If I issue request instead directly to Jetty at port :8080 the request is processed OK. So problem is likely to sit somewhere between Apache and Jetty where I am using mod_proxy. How to solve this?

I have already tried some "tricks" related to KeepAlive settings, without luck. Here is my current configuration, any suggestions?

#keepalive Off                     ## I have tried this, does not help
#SetEnv force-proxy-request-1.0 1  ## I have tried this, does not help
#SetEnv proxy-nokeepalive 1        ## I have tried this, does not help
#SetEnv proxy-initial-not-pooled 1 ## I have tried this, does not help
KeepAlive 20                       ## I have tried this, does not help
KeepAliveTimeout 600               ## I have tried this, does not help
ProxyTimeout 600                   ## I have tried this, does not help

NameVirtualHost *:80
<VirtualHost _default_:80>
    ServerAdmin [email protected]

    ServerName www.mydomain.fi

    ServerAlias mydomain.fi mydomain.com mydomain www.mydomain.com

    ProxyRequests On
    ProxyVia On
    <Proxy *>
            Order deny,allow
            Allow from all
    </Proxy>

    ProxyRequests Off
    ProxyPass / http://www.mydomain.fi:8080/ retry=1 acquire=3000 timeout=600
    ProxyPassReverse / http://www.mydomain.fi:8080/

    RewriteEngine On
    RewriteCond %{SERVER_NAME} !^www\.mydomain\.fi
    RewriteRule /(.*) http://www.mydomain.fi/$1 [redirect=301L]

    ErrorLog /var/log/apache2/error.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn

    CustomLog /var/log/apache2/access.log combined
    ServerSignature On

</VirtualHost>

Here is also the debug log from a failing request:

74.125.43.99 - - [29/Sep/2010:20:15:40 +0300] "GET /?wicket:bookmarkablePage=newWindow:com.mydomain.view.application.reports.SaveReportPage HTTP/1.1" 502 355 "https://www.mydomain.fi/?wicket:interface=:0:2:::" "Mozilla/5.0 (Windows; U; Windows NT 6.1; fi; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10"
[Wed Sep 29 20:20:40 2010] [error] [client 74.125.43.99] proxy: error reading status line from remote server www.mydomain.fi, referer: https://www.mydomain.fi/?wicket:interface=:0:2:::
[Wed Sep 29 20:20:40 2010] [error] [client 74.125.43.99] proxy: Error reading from remote server returned by /, referer: https://www.mydomain.fi/?wicket:interface=:0:2:::
apache-2.2 jetty

6 Answers

  1. I have solved the problem. The Keepalive=On should be inserted into ProxyPass config line:

    ProxyPass / http://www.dom.fi:8080/ retry=1 acquire=3000 timeout=600 Keepalive=On

    See that 

    Keepalive=On

    there? It is critical ;)

    • 108

  2. This error can also occur if you don't end your proxy url with a /. Either both paths should end with a / or neither.

    • 9

  3. Have you tried setting setenv proxy-initial-not-pooled 1?

    Reference here

    • 5

  4. Looking at the log, there's something that times out at 5 minutes (=300 seconds). That's a pretty long time to wait for a response. When you access the Jetty server directly, does this resource really take that long to produce a response?

    If the five minutes really is within possible response times, you might try tweaking the ProxyTimeout configuration directive.

    Depending on your network set-up, it might well be that there's no reason to even try to use any keepalive system (is there a firewall between the app server and proxy which might be configured to drop sessions that are idle for too long?), but the ProxyTimeout would affect the behaviour of the proxy itself.

    If the same proxy also serves other backends, it would be better to keep the current ProxyTimeout, and configure the timeout in the ProxyPass directive (see mod_proxy documentation).

    If, however, the responses without the proxy are consistently something much less than the five minutes see here as the cut-off limit, then there might really be some odd interference between the proxy and app server, but you're not providing anything of value for identifying what it might be.

    • 1

  5. For me removing a header value called Transfer-Encoding" (binary) in my server-app (PHP) solved the problem for:

    [proxy_http:error] [pid 17623] (22)Invalid argument: [client 127.0.0.1:44929] AH01102: error reading status line from remote server 0.0.0.0:80

    All the other suggestions like SetEnv proxy-initial-not-pooled or Keep-Alive did not.

    • 0

  6. If the above solutions don't work, one thing you can try is to enable all your apache modules to make sure that there isn't some module you need that somehow got accidentally disabled.

    For example, how I found the cause of my problem was to replace all instances of #LoadModule with LoadModule in all my Apache config files. Since that solved the problem for me, therefore I knew that my problem was not a missing "KeepAlive" directive argument, but rather, my problem was a missing dependency.

    Because, remember, .so files are basically static libraries. Having a module enabled does not mean it will get used, but having one disabled does mean that it can't get used, and therefore, anything that depends on it will necessarily fail.

    Note: this answer has received some down-votes due to the fact that my initial answer seemed to suggest leaving all the modules enabled, forever. While you could theoretically do that without necessarily breaking anything, it's obviously not a best practice solution.

    So, please understand, I am merely suggesting this as a troubleshooting step, not a final solution.

    Also, please note: I use a special git project to track all of my local machine's apache config files. That way I can do these sorts of global search-and-replace operations in my apache config working directory, as a troubleshooting step. If enabling all modules succeeds, then try disabling them again one-by-one and restarting apache in between, until you find which module it is that needs to remain enabled. Once you've figured that out, then reset the repo back to its original state, and only enable that one module that needs to remain enabled.

    You will also find that using git to track your apache config files cleans up those directories, since you won't be needing those old-fashioned .bak and .default files anymore.

    • 0