When I run this:
nslookup some.host.com
I get results like this:
Server: 4.2.2.1
Address: 4.2.2.1#53
Non-authoritative answer:
Name: some.host.com
Address: #.#.#.#
"4.2.2.1" is at the top of my /etc/resolv.conf, so that makes sense, but what does the #53 mean? Googling around, this seems to be commonly appended to the address here in nslookup results, but I can't find any information about what it actually means.
I am running an nginx server that acts as a proxy to an upstream unix socket, like this:
upstream app_server {
server unix:/tmp/app.sock fail_timeout=0;
}
server {
listen ###.###.###.###;
server_name whatever.server;
root /web/root;
try_files $uri @app;
location @app {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://app_server;
}
}
Some app server processes, in turn, pull requests off /tmp/app.sock as they become available. The particular app server in use here is Unicorn, but I don't think that's relevant to this question.
The issue is, it just seems that past a certain amount of load, nginx can't get requests through the socket at a fast enough rate. It doesn't matter how many app server processes I set up.
I'm getting a flood of these messages in the nginx error log:
connect() to unix:/tmp/app.sock failed (11: Resource temporarily unavailable) while connecting to upstream
Many requests result in status code 502, and those that don't take a long time to complete. The nginx write queue stat hovers around 1000.
Anyway, I feel like I'm missing something obvious here, because this particular configuration of nginx and app server is pretty common, especially with Unicorn (it's the recommended method in fact). Are there any linux kernel options that needs to be set, or something in nginx? Any ideas about how to increase the throughput to the upstream socket? Something that I'm clearly doing wrong?
Additional information on the environment:
$ uname -a
Linux servername 2.6.35-32-server #67-Ubuntu SMP Mon Mar 5 21:13:25 UTC 2012 x86_64 GNU/Linux
$ ruby -v
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]
$ unicorn -v
unicorn v4.3.1
$ nginx -V
nginx version: nginx/1.2.1
built by gcc 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)
TLS SNI support enabled
Current kernel tweaks:
net.core.rmem_default = 65536
net.core.wmem_default = 65536
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_mem = 16777216 16777216 16777216
net.ipv4.tcp_window_scaling = 1
net.ipv4.route.flush = 1
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
net.core.somaxconn = 8192
net.netfilter.nf_conntrack_max = 524288
Ulimit settings for the nginx user:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
According to the Unicorn docs, there are different binaries for Rails apps and other Rack apps:
non-Rails Rack applications
In APP_ROOT, run:
unicorn
for Rails applications (should work for all 1.2 or later versions)
In RAILS_ROOT, run:
unicorn_rails
They seem to also take the same command-line parameters. But Rails is built on top of rack, so I don't understand why this dichotomy is required. Is there any reason you can't just use unicorn for Rails apps?
I am running Ubuntu 11.10.
This is the only line in my /etc/apt/sources.list with the text "google" or "chrome" in it:
deb http://dl.google.com/linux/chrome/deb/ stable main
Sometimes (not always) when I run apt-get update, I get these warnings at the end of the output:
Reading package lists... Done
W: Duplicate sources.list entry http://dl.google.com/linux/chrome/deb/ stable/main amd64 Packages (/var/lib/apt/lists/dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages)
W: Duplicate sources.list entry http://dl.google.com/linux/chrome/deb/ stable/main i386 Packages (/var/lib/apt/lists/dl.google.com_linux_chrome_deb_dists_stable_main_binary-i386_Packages)
W: You may want to run apt-get update to correct these problems
If I run apt-get update again immediately after receiving these warnings, they don't show up the second time (or third time, etc...). But then the warnings always come back eventually.
In any case, installing/updating google-chrome works fine, but those warnings are annoying (and since I skim over them now, I may inadvertently miss some more important warning if one ever comes up).
Any way to get rid of these warnings permanently?
I want to upgrade nginx, and I want to use the same ./configure parameters I used the last time (--prefix=..., --conf=..., etc...). If I don't have access to the original installation directory, is there any way to query nginx or any config/cache file that I could use to obtain these parameters? (If not, is there any way to determine what these parameters were if I do have access to the original installation directory?)
I'm attempting to compile nginx (version 1.0.11) with gridfs support (nginx-gridfs version 0.8) on Ubuntu 11.10 (gcc version 4.6) using the nginx module here:
https://github.com/mdirolf/nginx-gridfs
The installation instructions described there are simple:
* Clone the nginx-gridfs repository (`git clone https://github.com/mdirolf/nginx-gridfs`)
* Check out the mongo-c-driver submodule (`git submodule init` then `git submodule update`)
* Download and unpack the nginx source (from `http://nginx.org/download/nginx-1.0.11.tar.gz`)
* Run `./configure` with `--add-module=/path/to/nginx-gridfs/repository`
* `make` and `sudo make install`
These steps worked fine for me last time I set up nginx-gridfs on a machine (about four months ago on Ubuntu 10.04, with gcc 4.5 and nginx version 1.0.5), but now (on a fresh Ubuntu install) I'm getting an error. The ./configure is working fine, but make fails with:
/path/to/nginx-gridfs/mongo-c-driver/src/mongo.c: In function 'mongo_count':
/path/to/nginx-gridfs/mongo-c-driver/src/mongo.c:939:5: error: missing initializer [-Werror=missing-field-initializers]
/path/to/nginx-gridfs/mongo-c-driver/src/mongo.c:939:5: error: (near initialization for 'out.dataSize') [-Werror=missing-field-initializers]
/path/to/nginx-gridfs/mongo-c-driver/src/mongo.c: In function 'mongo_simple_int_command':
/path/to/nginx-gridfs/mongo-c-driver/src/mongo.c:981:5: error: missing initializer [-Werror=missing-field-initializers]
/path/to/nginx-gridfs/mongo-c-driver/src/mongo.c:981:5: error: (near initialization for 'out.dataSize') [-Werror=missing-field-initializers]
/path/to/nginx-gridfs/mongo-c-driver/src/mongo.c: In function 'mongo_simple_str_command':
/path/to/nginx-gridfs/mongo-c-driver/src/mongo.c:1013:5: error: missing initializer [-Werror=missing-field-initializers]
/path/to/nginx-gridfs/mongo-c-driver/src/mongo.c:1013:5: error: (near initialization for 'out.dataSize') [-Werror=missing-field-initializers]
/path/to/nginx-gridfs/mongo-c-driver/src/mongo.c: In function 'mongo_cmd_get_error_helper':
/path/to/nginx-gridfs/mongo-c-driver/src/mongo.c:1055:5: error: missing initializer [-Werror=missing-field-initializers]
/path/to/nginx-gridfs/mongo-c-driver/src/mongo.c:1055:5: error: (near initialization for 'out.dataSize') [-Werror=missing-field-initializers]
/path/to/nginx-gridfs/mongo-c-driver/src/mongo.c: In function 'mongo_cmd_ismaster':
/path/to/nginx-gridfs/mongo-c-driver/src/mongo.c:1098:5: error: missing initializer [-Werror=missing-field-initializers]
/path/to/nginx-gridfs/mongo-c-driver/src/mongo.c:1098:5: error: (near initialization for 'out.dataSize') [-Werror=missing-field-initializers]
cc1: all warnings being treated as errors
make[1]: *** [objs/addon/src/mongo.o] Error 1
Any idea why this would fail?
The cc1: all warnings being treated as errors leads me to believe that I may be able to add -Wno-error to CFLAGS to tell it not treat warnings as errors to avoid this issue. But I'm worried that treat-warnings-as-errors is enabled here for a reason, and that turning it off would only make the compilation successful on the surface, but not actually work.
Anyone know what's going on here?