Stefano's questions

I upgraded our gentoo server, got lost in the messages and, to make the story short, I ended up removing my postgresql 9.1 BEFORE I run a pg_upgrade to 9.2.. Then I discovered that pg_upgrade requires the binary of the previous version to still be there!

How can I re-install postgresql 9.1 without removing 9.2?

I know I can specify a version with /etc/portage/package.mask

eg.

>dev-db/postgresql-server-9.1.9
<dev-db/postgresql-server-9.1.9

but I'm afraid doing so I would replace the postgresql installed and end up more screwed up than before.

I'm using nginx mostly as a reverse cachining proxy in front of several gunicon/mod_wsgi applications and of course to server static files.

I find that quickly my nginx confs become impossible to maintain; the issue is that I have a few patterns that are similar (or even identical) but I can't manage to make it clean.

One of the biggest issues I have is that I would love to use named locations as a way to group a set of confs, eg.

location @django_modwsgi {
    include proxy.conf;
    proxy_pass  http://127.0.0.1:8080;        
}

location @django_gunicorn {
    include proxy.conf; # this could also be included directly in the server {} block?
    proxy_pass  http://gunicorn_builder;
}

NB. The issue is not having both gunicorn and wsgi. That's just an example. Another one is:

location @namedlocation_1 {
     some cache settings;
     some cache_key settings;
     ignore some headers;
     expires;
     proxy_pass
}

location @namedlocation_2 {
     other cache settings;
     other cache_key settings;
     ignore some headers;
     expires;
     proxy_pass
}

but to call a named location the only way I found is :

location /somelocation {
    try_files $uri @named_location;
}

This already does not feel right, I do not want nginx to go look for static files, I want it to go directly to the named location! Is there a way to "call" a named location directly?!

Another way I thought I could go for dry is a lot of include...

location /somelocation {
    include django_unicorn.conf;
}

But is this a good way to do it? It sounds ok for very generic settings (eg. proxy ones), but it is not very readable to have to go open different files to get the full conf.

Also, in some cases I can group a few locations with a regexp, but I like doing so ONLY when they are logically related not just to be able to put common settings in the same block.

The Question

Is there an "official" best practice to write good, DRY nginx configurations?

I would love to find a pattern like:

location / {
    common confs
    try_files $uri @name_location
}

** but how do I write specific cases for different locations? **

Could I simply add several locations with the uncommon part of the conf and the common one in the @named_location?

location /1/ {
    some cache expire settings;
    NOTHING ELSE;
}

location /2/ {
    some other cache expire settings;
    NOTHING ELSE;
}

location / {
    common settings
    try_files
}

location @named_location {
    other common settings for this named location only
    proxy_pass
}

When I have different url pointing to the same resource can I simply do a rewrite?

location /1/ {
    rewrite  ^  /3/  last;
}

location /2/ {
    rewrite ^   /4/  last; 
}

location / {
    common settings
    try_files
}

location @named_location {
    other common settings for this named location only
    proxy_pass
}

or should those be all grouped into the one location?

location / {
    rewrite ^/1/$  /3/  last;
    rewrite ^/2/$   /4/  last; 

    common settings
    try_files
}

location @named_location {
    other common settings for this named location only
    proxy_pass
}

Related

I could not find much in the mailing list, even less so in the wiki.

Please note this is /not/ the same as question NGinx Best Practices - that's a very generic question.

This other one is more relevant: How do I DRY up this Nginx configuration?

I'm configuring gunicorn (monitored by supervisord and behind an nginx frontend) and got a bit confused about the optimal number of processes to setup.

In the documentation it is clearly explained that:

workers = multiprocessing.cpu_count() * 2 + 1

My machine is a quad core so that should count for 9 workers.

But I want to run several applications, each one listening to a different port.

Should the count then be (truncated):

workers_per_application = int(workers / NUM_APPLICATIONS)

Or should each have the above number of workers?

I think this question actually applies not only to gunicorn but to every similar kind of listening server...

I've been experiment quite some time with a "typical" django setting upon nginx+apache2+mod_wsgi+memcached(+postgresql) (reading the doc and some questions on SO and SF, see comments)

Since I'm still unsatisfied with the behavior (definitely because of some bad misconfiguration on my part) I would like to know what a good configuration would look like with these hypotesis:

• Quad-Core Xeon 2.8GHz
• 8 gigs memory
• several django projects (anything special related to this?)

These are excerpts form my current confs:

EDIT: I've added more stuff to make this complete but following Graham's suggestion I will follow up on the wsgi mailing list

apache 2 (>apache2 -v)

Server version: Apache/2.2.12 (Ubuntu)
Server built:   Nov 18 2010 21:16:51
Server's Module Magic Number: 20051115:23
Server loaded:  APR 1.3.8, APR-Util 1.3.9
Compiled using: APR 1.3.8, APR-Util 1.3.9
Architecture:   64-bit
Server MPM:     Worker
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/worker"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT=""
 -D SUEXEC_BIN="/usr/lib/apache2/suexec"
 -D DEFAULT_PIDLOG="/var/run/apache2.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types"
 -D SERVER_CONFIG_FILE="/etc/apache2/apache2.conf"

apache2 conf

PidFile ${APACHE_PID_FILE}
Timeout 60
KeepAlive Off

ServerSignature Off
ServerTokens Prod
#MaxKeepAliveRequests 100
#KeepAliveTimeout 15
# worker MPM
<IfModule mpm_worker_module>
    StartServers          2
    ServerLimit           4
    MinSpareThreads       2
    MaxSpareThreads       4
    ThreadLimit          32
    ThreadsPerChild      16
    MaxClients          64#128
    MaxRequestsPerChild   10000
</IfModule>

...

SetEnv VHOST null 
#WSGIPythonOptimize 2

<VirtualHost *:8082>
    ServerName subdomain.domain.com
    ServerAlias www.domain.com
    SetEnv VHOST subdomain.domain
    AddDefaultCharset UTF-8
    ServerSignature Off

    LogFormat "%{X-Real-IP}i %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" custom
    ErrorLog  /home/project1/var/logs/apache_error.log
    CustomLog /home/project1/var/logs/apache_access.log custom

    AllowEncodedSlashes On

    WSGIDaemonProcess subdomain.domain user=www-data group=www-data threads=25
    WSGIScriptAlias / /home/project1/project/wsgi.py
    WSGIProcessGroup %{ENV:VHOST}
</VirtualHost>

wsgi.py

Currently using version 3.3 built from source

import os
import sys

# setting all the right paths....


_realpath = os.path.realpath(os.path.dirname(__file__))
_public_html = os.path.normpath(os.path.join(_realpath, '../'))    

sys.path.append(_realpath)
sys.path.append(os.path.normpath(os.path.join(_realpath, 'apps')))
sys.path.append(os.path.normpath(_public_html))
sys.path.append(os.path.normpath(os.path.join(_public_html, 'libs')))
sys.path.append(os.path.normpath(os.path.join(_public_html, 'django')))


os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'

import django.core.handlers.wsgi

_application = django.core.handlers.wsgi.WSGIHandler()

def application(environ, start_response):
    """
    Launches django passing over some environment (domain name) settings
    """

    application_group = environ['mod_wsgi.application_group']
    """
    wsgi application group is required. It's also used to generate the
    HOST.DOMAIN.TLD:PORT parameters to pass over
    """
    assert application_group
    fields = application_group.replace('|', '').split(':')
    server_name = fields[0]
    os.environ['WSGI_APPLICATION_GROUP'] = application_group
    os.environ['WSGI_SERVER_NAME'] = server_name
    if len(fields) > 1 :
        os.environ['WSGI_PORT'] = fields[1]
    splitted = server_name.rsplit('.', 2)    
    assert splitted >= 2
    splited.reverse()
    if len(splitted) > 0 :
        os.environ['WSGI_TLD'] = splitted[0]
    if len(splitted) > 1 :
        os.environ['WSGI_DOMAIN'] = splitted[1]
    if len(splitted) > 2 :
        os.environ['WSGI_HOST'] = splitted[2]
    return _application(environ, start_response)`

folder structure

in case it matters (slightly shortened actually)

/home/www-data/projectN/var/logs
                       /project (contains manage.py, wsgi.py, settings.py)
                       /project/apps (all the project ups are here)
                       /django
                       /libs

Please forgive me in advance if I overlooked something obvious.

My main question is about the apache2 wsgi settings. Are those fine? Is 25 threads an /ok/ number with a quad core for one only django project? Is it still ok with several django projects on different virtual hosts? Should I specify 'process'? Any other directive which I should add? Is there anything really bad in the wsgi.py file?

I've been reading about potential issues with the standard wsgi.py file, should I switch to that?

Or.. should this conf just be running fine, and I should look for issues somewhere else?

So, what do I mean by "unsatisfied": well, I often get quite high CPU WAIT; but what is worse, is that relatively often apache2 gets stuck. It just does not answer anymore, and has to be restarted. I have setup a monit to take care of that, but it ain't a real solution. I have been wondering if it's an issue with the database access (postgresql) under heavy load, but even if it was, why would the apache2 processes get stuck?

Beside these two issues, performance is overall great. I even tried New Relic and got very good average results.

edit I will not be able to provide an answer myself as I have, temporarily, moved to an nginx+gunicorn environment.

Also follow up on google groups for my personal situation and issues! Sounds like Graham is, of course, really busy (mod_wsgi is a free side project!) but moving to Read The Docs sounds great, and solving that one backlog issue would be totally awesome. That and the new Apache 2.4 might make me reconsider the best combo (currently nginx+gunicorn, then I might drop nginx for a varnish+apache+mod_wsgi setting)

From apache2 doc:

The LockFile directive sets the path to the lockfile used when Apache is used with an AcceptMutex value of either fcntl or flock.

And about AcceptMutex

The AcceptMutex directives sets the method that Apache uses to serialize multiple children accepting requests on network sockets.

I am running apache 2.2 with mod_wsgi for a django website on an Ubuntu 64bit box.

what is it for? is this even ever used?

I can't find a file in the specified location.

My /usr/sbin/apache2 -V tells me:

-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE

which should mean that sysvsem is used. Does that mean that LockFile is not used?

We are running a production server based on Ubuntu 9.10 Karmic Koala, kernel is almost up-to-date (2.6.38.2-grsec-xxxx-grs-ipv6-64) but karmic package repository is now ridiculously outdated, eg. Nginx is 0.7.62 - really buggy - while latest stable is 1.0.x!

In addition, Karmic just reached its end of life.

This question: Best practices for keeping UNIX packages up to date? looks similar but actually only includes some suggestions about package managers; not at all what I need!

So the options that I see are:

1. Get a new machine, install it from scratch, migrate
2. Distribution upgrade
3. Use a different repository (launchpad/ppa / backport / pinning)
4. Build your own

The disadvantages of #1 are quite obvious.

I do not dare do a dist-upgrade path though, as downtime and possible catastrophic consequences are just impossible to predict for a production server, and currently am mostly re-building my own required packages. But I'm sure I might be missing some.

It is not really clear to me what the risks are (stability/compatibility) of using Ubuntu backports, in addition, nothing is officially provided for 9.10 anymore. Launchpad are individual-builds, similar question - how much better is this than compiling my own?

Building packages seems fine, but:

1. Sometimes I have trouble reproducing the correct ./configure options in order to re-use my existing configuration files
2. I am sure there are tons of packages and dependencies that are now pretty outdated and possible sources of bugs

Finally… what about "old" packages in a recent distribution? I guess there's no other way than re-building them myself? Is a combination of 2 and 4 finally the best path?

Is there any objective consensus on what is the best way to do this, or reasons why some of my options are fine/not fine?

If really there isn't, I will accept that the question gets closed before creating an endless thread!

I just inherited an intranet where a Mandriva Directory Server 5.0 acts as internal DNS and DHCP server, based on an LDAP server for configuration storage.

The idea is that all intranet hosts get a DNS entry name-ipdaddres where the ip address is in a NATted private space 10.0.*.*

Some of those hosts are also declared on a real public DNS (internet :-) ) pointing to our router public IP address; it takes care of some port-forwarding and http-redirections to allow some intranet services to be available from outside the office without a VPN-like or SSH-tunnel access.

The MDS DNS server thus only covers private ip addresses and relies on public DNS servers for the rest.

Now I have a couple of strictly related questions about this setup; I am not used to mandriva configuration files, and am a bit confused about the correct practice for this 'double' dns setup

1. WHERE can I modify the 'public' nameservers for MDS?

   Is it in /etc/sysconfig/network-scripts/ifcfg-eth0 , DNS1 and DNS2 ? They have changed since.

2. What's the best practice to have an host first check the MDS DNS, then use the public DNS? Should hosts only use the MDS DNS, and MDS goes to public DNSs to get the missing entries, or...?

edit

Advancing but still the main issue seems the same.

Please not that the server is currently working and most domain names are resolved, but some are not, and likely it was using and old, outdated public DNS.

Following @TiZon advise I have edited /etc/resolvconf/resolv.conf.d/tail to add the public nameservers.

Now this is what happens. When the main nameserver fails, it does not fallback to the public ones:

dig rueducommerce.com @10.0.0.10

; <<>> DiG 9.5.0-P2 <<>> rueducommerce.com @10.0.0.10
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 55845
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;rueducommerce.com.             IN      A

;; Query time: 0 msec
;; SERVER: 10.0.0.10#53(10.0.0.10)
;; WHEN: Thu May 19 15:50:55 2011
;; MSG SIZE  rcvd: 35

Dig (or nslookup, same behaviour) without specifying a server will get the fail code and try the next server in /etc/resolv.conf and correctly return the result.

dig rueducommerce.com

;; Got SERVFAIL reply from 127.0.0.1, trying next server

; <<>> DiG 9.5.0-P2 <<>> rueducommerce.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38942
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;rueducommerce.com.             IN      A

;; ANSWER SECTION:
rueducommerce.com.      440     IN      A       178.251.201.141
rueducommerce.com.      440     IN      A       178.251.200.141

;; Query time: 279 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu May 19 15:51:51 2011
;; MSG SIZE  rcvd: 67

but stackoverflow works

dig stackoverflow.com @10.0.0.10

; <<>> DiG 9.5.0-P2 <<>> stackoverflow.com @10.0.0.10
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40964
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 0

;; QUESTION SECTION:
;stackoverflow.com.             IN      A

;; ANSWER SECTION:
stackoverflow.com.      1800    IN      A       64.34.119.12

;; AUTHORITY SECTION:
stackoverflow.com.      38      IN      NS      ns3.serverfault.com.
stackoverflow.com.      38      IN      NS      ns2.serverfault.com.
stackoverflow.com.      38      IN      NS      ns1.serverfault.com.

;; Query time: 230 msec
;; SERVER: 10.0.0.10#53(10.0.0.10)
;; WHEN: Thu May 19 16:02:24 2011
;; MSG SIZE  rcvd: 117

Qhat can I debug to understand why stackoverflow is working but rueducommerce is not?

I could add a public DNS as a secondary DNS in the DHCP (how?) and the clients are probably going to behave better (same as this command line), but is that the correct way to solve this problem? To me it sounds like clients could start to get contraddictory results from the private and public server.

Is there a way for bind/named to go fetch those results or tell the client where to go fetch them?

Hope the question is clear enough, don't hesitate adding comments should I need to state it more clearly or add details.. thanks!