Questions[tuning](server)

I'm using Asp.Net together with MySQL. In the .Net connection string, I have set Max Pool Size to 150.

If I run the following I get these values:

SHOW GLOBAL STATUS LIKE 'max_used_connections'; gives 66
SHOW GLOBAL STATUS LIKE 'Threads_created'; gives 66
SHOW GLOBAL STATUS LIKE 'connections'; gives 474

Which gives Threads_created / Connections = 0,1392.

So from that it seems like I need to increase thread_cache_size.

But if I run SHOW PROCESSLIST I always see that I have a lot of connections open (most of them sleeping) because of the pool created by .Net. Do I still need to set the thread_cache_size as I still will reuse the connections from the connection pool? If the Pool Size is 150 do you think a good value would be to set thread_cache_size to 150+? Would this affect CPU and memory a lot?

I am running an nginx server that acts as a proxy to an upstream unix socket, like this:

upstream app_server {
        server unix:/tmp/app.sock fail_timeout=0;
}

server {
        listen ###.###.###.###;
        server_name whatever.server;
        root /web/root;

        try_files $uri @app;
        location @app {
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header Host $http_host;
                proxy_redirect off;
                proxy_pass http://app_server;
        }
}

Some app server processes, in turn, pull requests off /tmp/app.sock as they become available. The particular app server in use here is Unicorn, but I don't think that's relevant to this question.

The issue is, it just seems that past a certain amount of load, nginx can't get requests through the socket at a fast enough rate. It doesn't matter how many app server processes I set up.

I'm getting a flood of these messages in the nginx error log:

connect() to unix:/tmp/app.sock failed (11: Resource temporarily unavailable) while connecting to upstream

Many requests result in status code 502, and those that don't take a long time to complete. The nginx write queue stat hovers around 1000.

Anyway, I feel like I'm missing something obvious here, because this particular configuration of nginx and app server is pretty common, especially with Unicorn (it's the recommended method in fact). Are there any linux kernel options that needs to be set, or something in nginx? Any ideas about how to increase the throughput to the upstream socket? Something that I'm clearly doing wrong?

Additional information on the environment:

$ uname -a
Linux servername 2.6.35-32-server #67-Ubuntu SMP Mon Mar 5 21:13:25 UTC 2012 x86_64 GNU/Linux

$ ruby -v
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]

$ unicorn -v
unicorn v4.3.1

$ nginx -V
nginx version: nginx/1.2.1
built by gcc 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)
TLS SNI support enabled

Current kernel tweaks:

net.core.rmem_default = 65536
net.core.wmem_default = 65536
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_mem = 16777216 16777216 16777216
net.ipv4.tcp_window_scaling = 1
net.ipv4.route.flush = 1
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
net.core.somaxconn = 8192
net.netfilter.nf_conntrack_max = 524288

Ulimit settings for the nginx user:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

I run a rather busy site, and during peek hours I see over 10.000 open connections to my database server on my webserver when a run a netstat command. 99% of the connections are in the TIME_WAIT state.

I learned about this mysql variable: wait_timeout http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html#sysvar_wait_timeout today. Mine is still set at the default 28.800 seconds.

Is lowering this value safe?

Non of my queries usually takes over a second. So it seems silly to keep a connection open for 480 minutes.

I also heard about using mysql_pconnect instead of mysql_connect, but i've been reading nothing but horror stories about it, so I think i'll stay away from that.

What kernel parameter or other settings control the maximum number of TCP sockets that can be open on a Linux server? What are the tradeoffs of allowing more connections?

I noticed while load testing an Apache server with ab that it's pretty easy to max out the open connections on the server. If you leave off ab's -k option, which allows connection reuse, and have it send more than about 10,000 requests then Apache serves the first 11,000 or so requests and then halts for 60 seconds. A look at netstat output shows 11,000 connections in the TIME_WAIT state. Apparently, this is normal. Connections are kept open a default of 60 seconds even after the client is done with them for TCP reliability reasons.

It seems like this would be an easy way to DoS a server and I'm wondering what the usual tunings and precautions for it are.

Here's my test output:

# ab -c 5 -n 50000 http://localhost/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 5000 requests
Completed 10000 requests
apr_poll: The timeout specified has expired (70007)
Total of 11655 requests completed

Here's the netstat command I run during the test:

 # netstat --inet -p | grep "localhost:www" | sed -e 's/ \+/ /g' | cut -d' ' -f 1-4,6-7 | sort | uniq -c 
  11651 tcp 0 0 localhost:www TIME_WAIT -
      1 tcp 0 1 localhost:44423 SYN_SENT 7831/ab
      1 tcp 0 1 localhost:44424 SYN_SENT 7831/ab
      1 tcp 0 1 localhost:44425 SYN_SENT 7831/ab
      1 tcp 0 1 localhost:44426 SYN_SENT 7831/ab
      1 tcp 0 1 localhost:44428 SYN_SENT 7831/ab

How do you determine that the performance of your Linux server is I/O bound and, perhaps more importantly, what process or processes are casuing the problem?