Benjamin Wohlwend's questions

I found some weird entries in my ELB logs that I can't really explain. They all look like this (parsed into JSON):

{
    "backend": "-",
    "backend_processing_time": "-1",
    "backend_status_code": 0,
    "client_ip": "XXX.XXX.XXX.XXX",
    "client_port": "52084",
    "elb": "ELB1-PROD",
    "elb_status_code": 504,
    "http_method": "POST",
    "http_version": "HTTP/1.1",
    "received_bytes": 16051,
    "request_processing_time": "-1",
    "request_uri": "https://example.com:443/example/",
    "response_processing_time": "-1",
    "sent_bytes": 0,
    "ssl_cipher": "ECDHE-RSA-AES128-GCM-SHA256",
    "ssl_protocol": "TLSv1.2",
    "timestamp": "2015-10-29T04:41:47.251500Z",
    "user_agent": "client/4.0"
}

e.g. "backend_processing_time": "-1" or "backend_status_code": 0 seem really weird.

Any idea what this could mean? I suspected that the backend servers were overloaded, but we didn't experience a higher load than usual when these errors happened, and the errors have all been triggered by the same client IP.

We're building a load balanced setup with two load balancers (that also terminate SSL) and several upstream servers. Both the load balancers and the upstream servers run nginx. The network on which requests are forwarded to the upstream servers cannot be trusted, hence we have to re-encrypt it after SSL termination on the load balancer. We're wondering which route we should take for re-encrypting the connection:

• SSL: the load balancer connects to an upstream server using SSL.
  • positive: fairly resilient, since every connection is established from the ground up
  • negative: added latency from the SSL handshake
• persistent SSH tunnel
  • positive: no (or less?) added latency
  • negative: tunnel may break down and has to be monitored with e.g. autossh

Additionally, I'm not sure how both solutions compare in throughput. Does anybody have some experience with one or both of these solutions? Or are there better approaches?

We use nagios to monitor our server farm, and generally it works great. From time to time, though, the host where nagios runs loses connectivity for a couple of minutes, which makes nagios believe that all servers and services it monitors are down. The result is hundreds of alert mails, shortly followed by hundreds of recovery mails.

Is there any way to configure nagios in such a way that it tests its own connectivity before releasing an avalanche of alert mails?

I have a relatively straight forward config:

upstream appserver-1 {
    server unix:/var/www/example.com/app/tmp/gunicorn.sock fail_timeout=0;
}
server {
    listen  80;
    server_name  example.com;

    location / {
        proxy_pass http://appserver-1;
        proxy_redirect              off;
        proxy_set_header            Host $host;
        proxy_set_header            X-Real-IP $remote_addr;
        proxy_set_header            X-Forwarded-For $proxy_add_x_forwarded_for;

        auth_basic                  "Restricted";
        auth_basic_user_file        /path/to/htpasswd;

    }

    location /api/ {
        auth_basic          off;
    }
}

The goal is to use basic auth on the whole website, except on the /api/ subtree. While it does work with respect to basic auth, other directives like proxy_pass are not in effect on /api/ as well.

Is it possible to just disable basic auth while retaining the other directives without copy&pasting everything?

I have a setup with Apache2 as a front-end server for multiple python apps served by gunicorn. My Apache2 setup using mod_proxy looks like this:

<VirtualHost *:80>
    ServerName example.com
    UseCanonicalName On
    ServerAdmin webmaster@localhost

    LogLevel warn
    CustomLog /var/log/apache2/example.com/access.log combined
    ErrorLog /var/log/apache2/example.com/error.log
    ServerSignature On

    Alias /media/ /home/example/example.com/pysrc/project/media/

    ProxyPass /media/ !
    ProxyPass / http://127.0.0.1:4711/
    ProxyPassReverse / http://127.0.0.1:4711/
    ProxyPreserveHost On
    ProxyErrorOverride Off
</VirtualHost>

Generally, this setup works pretty well. I have one problem though: When I restart the gunicorn process (takes 2-5 seconds) and there is a request from Apache, that request will fail with a 503 error. So far so good. But Apache keeps returning 503 errors, even after the gunicorn process is back up. It resumes serving content from the proxied server only after a full restart of Apache.

Is there a way around this behaviour?

I implemented a SSO authentication using mod_auth_kerb on Apache. My config looks like this:

<Location /login/ >
    AuthType Kerberos
    AuthName "Kerberos Login"
    KrbMethodNegotiate on
    KrbAuthoritative on
    KrbVerifyKDC on
    KrbAuthRealm D.ETHZ.CH
    Krb5Keytab /etc/HTTP.keytab
    KrbSaveCredentials on
    RequestHeader set KERBEROS_USER %{REMOTE_USER}s
</Location>

My problem is that, without require valid-user, mod_auth_kerb doesn't even try to authenticate the user and KERBEROS_USER ends up being (null). If I add require valid-user, the user is authenticated automatically if the browser supports it, but gets shown an ugly modal login form (ala HTTP Basic Auth) if the browser doesn't support Kerberos Negotiate.

What I want to achieve is that if a user visits /login/, mod_auth_kerb tries to authenticate the user through Kerberos Negotiate. If that fails, a normal HTML login form will be presented to the user.

Is it possible to configure Apache/mod_auth_kerb in such a way?

I'm on RHEL 5.4, and I've set up a Django project using PostgreSQL as the database (I've used MySQL before, but want to try something new :) ). When I run the Django test server, everything works, I can also connect trough the command line with password authentification:

psql -U myuser -h 127.0.0.1

But when I try to access my project over Apache, I get a 500 error code and this python traceback in apache's error log:

mod_wsgi (pid=7026): Exception occurred processing WSGI script '/var/www/myproject/myproject/hook.wsgi'.
Traceback (most recent call last):
  File "/var/www/myproject/myproject/src/django/django/core/handlers/wsgi.py", line 245, in __call__
    response = middleware_method(request, response)
  File "/var/www/myproject/myproject/src/django/django/contrib/sessions/middleware.py", line 36, in process_response
    request.session.save()
  File "/var/www/myproject/myproject/src/django/django/contrib/sessions/backends/db.py", line 55, in save
    obj = Session(
  File "/var/www/myproject/myproject/src/django/django/contrib/sessions/backends/base.py", line 152, in _get_session_key
    self._session_key = self._get_new_session_key()
  File "/var/www/myproject/myproject/src/django/django/contrib/sessions/backends/base.py", line 144, in _get_new_session_key
    if not self.exists(session_key):
  File "/var/www/myproject/myproject/src/django/django/contrib/sessions/backends/db.py", line 29, in exists
    Session.objects.get(session_key=session_key)
  File "/var/www/myproject/myproject/src/django/django/db/models/manager.py", line 132, in get
    return self.get_query_set().get(*args, **kwargs)
  File "/var/www/myproject/myproject/src/django/django/db/models/query.py", line 336, in get
    num = len(clone)
  File "/var/www/myproject/myproject/src/django/django/db/models/query.py", line 81, in __len__
    self._result_cache = list(self.iterator())
  File "/var/www/myproject/myproject/src/django/django/db/models/query.py", line 269, in iterator
    for row in compiler.results_iter():
  File "/var/www/myproject/myproject/src/django/django/db/models/sql/compiler.py", line 672, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/var/www/myproject/myproject/src/django/django/db/models/sql/compiler.py", line 726, in execute_sql
    cursor = self.connection.cursor()
  File "/var/www/myproject/myproject/src/django/django/db/backends/__init__.py", line 75, in cursor
    cursor = self._cursor()
  File "/var/www/myproject/myproject/src/django/django/db/backends/postgresql_psycopg2/base.py", line 136, in _cursor
    self.connection = Database.connect(**conn_params)
OperationalError: could not connect to server: Permission denied
\tIs the server running on host "127.0.0.1" and accepting
\tTCP/IP connections on port 5432?

I checked the conn_params dictionary, it contains the correct username and password. My pg_hba.conf looks like this (I have the same exact configuration on an Ubuntu 9.04 server, where everything works peachy):

local   all         postgres                          ident
local   all         all                               md5
host    all         all         127.0.0.1/32          md5
host    all         all         ::1/128               md5

What am I doing wrong?