Questions[502-error](server)

I frequently get this error on random pages when the are like ~+10K connections (netstat -an |grep 80 |wc -l). This happens while there is still +10GB of free memory and server load is less than 3.

My relevant configs:

/etc/php5/fpm/php.ini

memory_limit = 1024M
default_socket_timeout = 120

/etc/php5/fpm/pool.d/www.conf

pm = dynamic 
pm.max_children = 30 
pm.start_servers = 5
pm.min_spare_servers = 2 
pm.max_spare_servers = 5 
;pm.max_requests =20000 #uncommented with any value didn't fixed
request_terminate_timeout = 120s

I also doubled the above vlues (i.e 60 children, 10 start, 10 max servers) with no avail.

/etc/nginx/nginx.conf

worker_connections 4024;
keepalive_timeout 10;

Nginx itself behind Varnish and memcached is enable. page rendering is pretty fast in general.

This seems to be a quite common problem you I could not find any helpful solution or even analysis of the possible causes. So your hints/solutions are greatly appreciated.

I'm experiencing 502 Gateway errors when accessing a PHP file in a directory (http://example.com/dev/index.php). The logs simply says this:

2011/09/30 23:47:54 [error] 31160#0: *35 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xx.xx.xx, server: domain.com, request: "GET /dev/ HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "domain.com"

I've never experienced this before. What is the solution for this type of 502 Gateway error?

This is the nginx.conf:

user www-data;
worker_processes 4;
pid /var/run/nginx.pid;

events {
    worker_connections 768;
    # multi_accept on;
}

http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    # server_tokens off;

    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##

    gzip on;
    gzip_disable "msie6";

    # gzip_vary on;
    # gzip_proxied any;
    # gzip_comp_level 6;
    # gzip_buffers 16 8k;
    # gzip_http_version 1.1;
    # gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

    ##
    # Virtual Host Configs
    ##

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

I am running Nginx and proxying php requests via FastCGI to PHP-FPM for processing. I will randomly receive 502 Bad Gateway error pages - I can reproduce this issue by clicking around my PHP websites very rapidly/refreshing a page for a minute or two. When I get the 502 error page all I have to do is refresh the browser and the page refreshes properly.

Here is my setup:

nginx/0.7.64 PHP 5.3.2 (fpm-fcgi) (built: Apr 1 2010 06:42:04) Ubuntu 9.10 (Latest 2.6 Paravirt)

I compiled PHP-FPM using this ./configure directive

./configure --enable-fpm --sysconfdir=/etc/php5/conf.d --with-config-file-path=/etc/php5/conf.d/php.ini --with-zlib --with-openssl --enable-zip --enable-exif --enable-ftp --enable-mbstring --enable-mbregex --enable-soap --enable-sockets --disable-cgi --with-curl --with-curlwrappers --with-gd --with-mcrypt --enable-memcache --with-mhash --with-jpeg-dir=/usr/local/lib --with-mysql=/usr/bin/mysql --with-mysqli=/usr/bin/mysql_config --enable-pdo --with-pdo-mysql=/usr/bin/mysql --with-pdo-sqlite --with-pspell --with-snmp --with-sqlite --with-tidy --with-xmlrpc --with-xsl

My php-fpm.conf looks like this (the relevant parts):

 ...   
<value name="pm">
     <value name="max_children">3</value>
     ...
     <value name="request_terminate_timeout">60s</value>
       <value name="request_slowlog_timeout">30s</value>
       <value name="slowlog">/var/log/php-fpm.log.slow</value>
       <value name="rlimit_files">1024</value>
       <value name="rlimit_core">0</value>
       <value name="chroot"></value>
       <value name="chdir"></value>
       <value name="catch_workers_output">yes</value>
       <value name="max_requests">500</value>
...

I've tried increasing the max_children to 10 and it makes no difference. I've also tried setting it to 'dynamic' and setting max_children to 50, and start_server to '5' without any difference.

I have tried using both 1 and 5 nginx worker processes.

My fastcgi_params conf looks like:

fastcgi_connect_timeout 60;
fastcgi_send_timeout 180;
fastcgi_read_timeout 180;
fastcgi_buffer_size 128k;
fastcgi_buffers 4 256k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
fastcgi_intercept_errors on;

fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;

fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;
fastcgi_param  REDIRECT_STATUS    200;

Nginx logs the error as:

[error] 3947#0: *10530 connect() failed (111: Connection refused) while connecting to upstream, client: 68.40.xxx.xxx, server: www.domain.com, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com"

PHP-FPM logs the follow at the time of the error:

[NOTICE] pid 17161, fpm_unix_init_main(), line 255: getrlimit(nofile): max:1024, cur:1024 
[NOTICE] pid 17161, fpm_event_init_main(), line 93: libevent: using epoll 
[NOTICE] pid 17161, fpm_init(), line 50: fpm is    running, pid 17161 
[DEBUG] pid 17161, fpm_children_make(), line 403: [pool default] child 17162 started 
[DEBUG] pid 17161, fpm_children_make(), line   403: [pool default] child 17163    started 
[DEBUG] pid 17161,  fpm_children_make(), line 403: [pool default] child 17164 started 
[NOTICE] pid 17161, fpm_event_loop(), line 111: ready to handle connections

My CPU usage maxes out around 10-15% when I recreate the issue. My Free mem (free -m) is > 130MB

I had this intermittent 502 Bad Gateway issue when in was using php5-cgi to service my php requests as well. Does anyone know how to fix this?

EDIT/UPDATE: I am using supervisor to launch php-fpm (not deamonized because of this).

I have experience with nginx but it's always been pre-installed for me (via VPS.net pre-configured image). I really like what it does for me, and now I'm trying to install it on my own server with apt-get. This is a fairly fresh Debian 5 install. I have few extra packages installed but they're all .deb's, no manual compiling or anything crazy going on.

Apache is already installed but I disabled it. I did apt-get install nginx and that worked fine. Changed the config around a bit for my needs, although the same problem I'm about to describe happens even with the default config.

It took me a while to figure out that the default debian package for nginx doesn't spawn fastcgi processes automatically. That's pretty lame, but I figured out how to do that with this script, which I found posted on many different web sites:

#!/bin/bash

## ABSOLUTE path to the PHP binary
PHPFCGI="/usr/bin/php5-cgi"

## tcp-port to bind on
FCGIPORT="9000"

## IP to bind on
FCGIADDR="127.0.0.1"

## number of PHP children to spawn
PHP_FCGI_CHILDREN=10

## number of request before php-process will be restarted
PHP_FCGI_MAX_REQUESTS=1000

# allowed environment variables sperated by spaces
ALLOWED_ENV="ORACLE_HOME PATH USER"

## if this script is run as root switch to the following user
USERID=www-data

################## no config below this line

if test x$PHP_FCGI_CHILDREN = x; then
  PHP_FCGI_CHILDREN=5
fi

ALLOWED_ENV="$ALLOWED_ENV PHP_FCGI_CHILDREN"
ALLOWED_ENV="$ALLOWED_ENV PHP_FCGI_MAX_REQUESTS"
ALLOWED_ENV="$ALLOWED_ENV FCGI_WEB_SERVER_ADDRS"

if test x$UID = x0; then
  EX="/bin/su -m -c \"$PHPFCGI -q -b $FCGIADDR:$FCGIPORT\" $USERID"
else
  EX="$PHPFCGI -b $FCGIADDR:$FCGIPORT"
fi

echo $EX

# copy the allowed environment variables
E=

for i in $ALLOWED_ENV; do
  E="$E $i=${!i}"
done

# clean environment and set up a new one
nohup env - $E sh -c "$EX" &> /dev/null &

When I do a "ps -A | grep php5-cgi", I see the 10 processes running, that should be ready to listen.

But when I try to view a web page via nginx, I just get a 502 bad gateway error.

After futzing around a bit, I tried telneting to 127.0.0.1 9000 (fastcgi is listening on port 9000, and nginx is configured to talk to that port), but it just immediately closes the connection.

This makes me think the problem is with fastcgi, but I'm not sure what I can do to test it. It may just be closing the connection because it's not getting fed any data to process, but it closes immediately so that makes me think otherwise.

So... any advice? I can't figure it out. It doesn't help that it's 1AM, but I'm going crazy here!

I'm a developer and I'm having trouble setting up some hosting, and I hope you guys can help me out.

I've got a domain I'm setting up for a friend. URLs work fine when I use the short form, like http://example.com. That is, the pages render properly and I get HTTP/200 responses. When I try http://www.example.com, however, I get a HTTP/502 error. Pinging it gives me,

Ping request could not find host www.example.com. Please check the name and try again.

I'm stumped, because I don't know enough terminology to formulate a good keyword search. Can anyone give me some info on what I might need to do to set up some kind of alias so that the standard www. form will work? I don't know whether to talk to my registrar or my hosting company, what to ask them for, or where to look in the various control panels.

Any help at all — even just naming the different pieces — would be helpful. I don't know my TLDs from my domain aliases from my CNAMEs.