Vladimir Panteleev's questions

When signing a message with DKIM, the choice of which domain's signature to use seems to lie on the sender - as far as I can tell, there is nothing prescriptive in the DKIM specification which says which certain specific domains must be signed.

So, which domain(s) should the DKIM d field match?

Some of the options:

• The domain name of the address from the From header
• The domain name of the envelope From
• The host name of the mail server, as used in HELO/EHLO / Received headers (when forwarding mail?)
• The reverse hostname of the sender IP?
• The domain name of the Sender header (as mentioned in the DKIM RFC)?

Since DKIM allows multiple signatures in a single message, one approach is to go ahead and sign the message for all involved domains for which we have signing keys of.

Are there any definitive rules, though?

I would like to perform incremental backups (for the entire filesystem) from a machine. rsync does this quite will, however I would like to also preserve file ownership - meaning, make it possible to restore it.

Is this possible to do without running rsync as root on the target machine (storing the backups)?

Some ideas...

• Is there a way to mount a filesystem (FUSE?) in such a way as to allow chown for a non-root user? (I guess it would probably need to be noexec to forbid elevation.)
• Some way to store and restore the ownership in metadata files instead of the filesystem itself?
• tar can store file ownership, though getting it to work with rsync or incremental backups would be a bit more involved. It would also be nice to be able to browse the backups like a regular filesystem.
• Perhaps some kind of fake root environment? A virtual machine would work, but would be nice to avoid the associated maintenance and performance overhead.

I'm looking for a way to implement something close to the following backup scheme:

1. Initially, a full image is copied to the backup target.
2. Periodically (e.g. nightly), only blocks that have changed since the last backup are copied to the backup target.
3. Ideally, it should be possible to mount snapshots from any point in time, or delete (flatten) some snapshots selectively.

Can this be implemented using LVM (or some other way)? It needs tracking which blocks have become dirty since the last backup, which I'm not sure LVM can do... I'd rather avoid the permanent performance cost of running on an LVM snapshot at all times.

I have some software which provides dynamic information via DNS, similar to a DNSBL.

How do I run it on the same server as a BIND server that's also serving DNS requests for other domains? I just need BIND to redirect queries to a certain subdomain (*.myservice.myname.net) to a different port on the same machine, and I can't just delegate the subdomain because I don't have an IPv4 address to spare.

In qmail, you could put this in a .forward file:

|/usr/local/bin/myfilter | forward [email protected]

This would pipe the whole message, incl. headers, to the myfilter program, then forward the message printed to its standard output to [email protected].

However, Exim doesn't have a forward program, and generally ignores the output of pipe transports.

How can one do this with Exim .forward files?

I'm trying to set up nginx as a reverse proxy, with a large number of backend servers. I'd like to start up the backends on-demand (on the first request that comes in), so I have a control process (controlled by HTTP requests) which starts up the backend depending on the request it receives.

My problem is configuring nginx to do it. Here's what I have so far:

server {
    listen 80;
    server_name $DOMAINS;

    location / {
        # redirect to named location
        #error_page 418 = @backend;
        #return 418; # doesn't work - error_page doesn't work after redirect

        try_files /nonexisting-file @backend;
    }

    location @backend {
        proxy_pass http://$BACKEND-IP;
        error_page 502 @handle_502; # Backend server down? Try to start it
    }

    location @handle_502 { # What to do when the backend server is not up
        # Ping our control server to start the backend
        proxy_pass http://127.0.0.1:82;
        # Look at the status codes returned from control server
        proxy_intercept_errors on;
        # Fallback to error page if control server is down
        error_page 502 /fatal_error.html;
        # Fallback to error page if control server ran into an error
        error_page 503 /fatal_error.html;
        # Control server started backend successfully, retry the backend
        # Let's use HTTP 451 to communicate a successful backend startup
        error_page 451 @backend;
    }

    location = /fatal_error.html {
        # Error page shown when control server is down too
        root /home/nginx/www;
        internal;
    }
}

This doesn't work - nginx seems to ignore any status codes returned from the control server. None of the error_page directives in the @handle_502 location work, and the 451 code gets sent as-is to the client.

I gave up trying to use internal nginx redirection for this, and tried modifying the control server to emit a 307 redirect to the same location (so that the client would retry the same request, but now with the backend server started up). However, now nginx is stupidly overwriting the status code with the one it got from the backend request attempt (502), despite that the control server is sending a "Location" header. I finally got it "working" by changing the error_page line to error_page 502 =307 @handle_502;, thus forcing all control server replies to be sent back to the client with a 307 code. This is very hacky and undesirable, because 1) there is no control over what nginx should do next depending on the control server's response (ideally we only want to retry the backend only if the control server reports success), and 2) not all HTTP clients support HTTP redirects (e.g. curl users and libcurl-using applications need to enable following redirects explicitly).

What's the proper way to get nginx to try to proxy to upstream server A, then B, then A again (ideally, only when B returns a specific status code)?

I'm using cgroups with the memory controller to set a memory limit for each user (using the memory.limit_in_bytes setting).

The problem is that this setting also accounts cache usage. Therefore, if the limit is 1GB, and the user merely downloads or copies a 1GB file, their processes get killed. What's worse, the cached pages remain in memory, so the user's "memory usage" remains close to 1GB even when they have zero processes running.

Naturally, this makes no sense. I only want to limit total private (non-anonymous) memory usage per user. How can I achieve that?

Alternatively, get the OOM killer to try dropping the user's cached pages before going off killing processes, which doesn't even free the cached pages.

(Sorry if this is off-topic - technically this is a home workstation setting, but is regarding mostly-server technology.)

I have a GA-X79S-UP5-WIFI motherboard and four ST3000DM001 drives, and the IRST manager isn't letting me put the drives in a RAID (it lists them, but the option to select the C600-series controller is simply grayed out).

Have I done a mistake? Do I need SAS drives to RAID them on the SAS controller?

Cron sends emails with the "From" header always set to "From: [email protected] (Cron Daemon)" for all users, even though the envelope address (seen in the "Delivered-To" header in received email) is the user's correct email address. Is it possible to create a header rewrite rule which fixes the "From" header for outgoing emails?

nginx has the $scheme variable usable in its log_format lines.

%H is the request protocol (e.g. "HTTP/1.1").

How can I do the same with Apache?

Let's say I have two Apache servers (one per vhost) behind an nginx server.

The problem is that I'd like Apache to do the authentication of client certificates.

Would it be possible for nginx to do the SNI part of SSL (so it knows to which Apache instance to forward to), and forward the rest to Apache?

I'm currently administrating a Gentoo server hosting a few dozen websites for friends and people with common interests. I may be moving to a new server soon, and with that comes the possibility to switch to another OS. I've enjoyed Gentoo's ability to let me tinker with USE flags and automatically patch upstream source code right before it's compiled and installed, but it does have some quirks (tail and cron having subtly broken behavior, deadlocks in kernel cgroups code, files being installed with randomly broken permissions), and the upkeep can be considerable (I can't seem to update perl without breaking half the system).

I conjecture that for a shared hosting system, two things are of most importance: security and stability. While I generally trust my users not to do something intentionally malicious, I can't trust them to never allow their accounts to be compromised (e.g. by installing and not updating a web app for which a vulnerability has been found). A user should not be able to access other users' data, interfere with the accessibility of other websites, or trash the server.

CentOS seems to be very popular on servers, although my personal limited experience with rpm/yum was rather discouraging (took me an hour to install one package). I heard nice things about Ubuntu Server as well. Or should I just stick to what I'm familiar with, and hope that my Gentoo problems are simply a result of misconfiguration or accumulated cruft that won't manifest in a clean install? (CentOS and Ubuntu Server come with SELinux/AppArmor, Gentoo has the Hardened Gentoo project.)

There's also the question of Apache configuration... running mod_php with a standard MPM is out of the question, as it'll allow scripts access to all users' data. CGI is a lot slower. I'm using the itk MPM right now, and it seems to work okay, but I wonder if there's a better solution - how do the big shared hosting providers (DreamHost etc.) do it?

Is it possible to set a limit of simultaneous requests for each vhost? Basically to prevent connections to a single vhost to fill all connection slots.

My guess would be "no", because Apache has to accept a connection in order to know the vhost, so the "real" solution would be to use different IPs. Maybe it's possible at a lower level (e.g. before it runs CGI/mod_php)?

I maintain a Gentoo server with a few services, including Apache. It's fairly low-end (2GB of RAM and a low-end CPU with 2 cores). My problem is that, despite my best efforts, an over-loaded Apache crashes the entire server. In fact, at this point I'm close to being convinced that Linux is a horrible operating system that isn't worth anyone's time looking for stability under load.

Things I tried:

1. Adjusting oom_adj for the root Apache process (and thus all its children). That had close to no effect. When Apache was overloaded it would bring the system to a grind, as the system paged out everything else before it got to kill anything.
2. Turning off swap. Didn't help, it would unload memory paged to binaries of processes and other files on /, thus causing the same effect.
3. Putting it in a memory-limited cgroup (limited to 512 MB of RAM, 1/4th of the total). This "worked", at least in my own stress tests - except the server keeps crashing under load (basically stalling all other processes, inaccessible via SSH, etc.)
4. Running it with idle I/O priority. This wasn't a very good idea in the end, because it just caused the system load to climb indefinitely (into the thousands) with almost no visible effect - until you tried to access an unbuffered part of the disk. This caused the task to freeze. (So much for good I/O scheduling, eh?)
5. Limiting the number of concurrent connections to Apache. Setting the number too low caused web sites to become unresponsive due to most slots being occupied with long requests (file downloads).
6. I tried various Apache MPMs without much success (prefork, event, itk).
7. Switching from prefork/event+php-cgi+suphp to itk+mod_php. This improved performance, but didn't solve the actual problem.
8. Switching I/O schedulers (cfq to deadline).

Just to stress this out: I don't care if Apache itself goes down under load, I just want the rest of my system to remain stable. Of course, having Apache recover quickly after a brief period of intensive load would be great to have, but one step at a time.

Right now I am mostly dumbfounded by how can humanity, in this day and age, design an operating system where such a seemingly simple task (don't allow one system component to crash the entire system) seems practically impossible - or at least, very hard to do.

Please don't suggest things like VMs or "BUY MORE RAM".

Some more information gathered with a friend's help: The processes hang when the cgroup oom killer is invoked. Here's the call trace:

[<ffffffff8104b94b>] ? prepare_to_wait+0x70/0x7b
[<ffffffff810a9c73>] mem_cgroup_handle_oom+0xdf/0x180
[<ffffffff810a9559>] ? memcg_oom_wake_function+0x0/0x6d
[<ffffffff810aa041>] __mem_cgroup_try_charge+0x32d/0x478
[<ffffffff810aac67>] mem_cgroup_charge_common+0x48/0x73
[<ffffffff81081c98>] ? __lru_cache_add+0x60/0x62
[<ffffffff810aadc3>] mem_cgroup_newpage_charge+0x3b/0x4a
[<ffffffff8108ec38>] handle_mm_fault+0x305/0x8cf
[<ffffffff813c6276>] ? schedule+0x6ae/0x6fb
[<ffffffff8101f568>] do_page_fault+0x214/0x22b
[<ffffffff813c7e1f>] page_fault+0x1f/0x30

At this point, the apache memory cgroup is practically deadlocked, and burning CPU in syscalls (all with the above call trace). This seems like a problem in the cgroup implementation...

I need to set a property on the root Apache process which in inherited by its children (e.g. oom_adj, cgroup membership). However, Apache will fork itself as soon as it starts.

• Using $$ in the init.d script doesn't seem to work - nothing is affected after the init script exits.
• Using the value from the PID file is too late - only the root process is affected, after it has created workers.

Any solutions? (aside from hacky ideas like enumerating "apache2" processes)

I have a Windows web service application running on HTTP, no HTTPS support. Can someone recommend some lightweight HTTPS reverse proxy in front of it, so connections to it from the outside can go through HTTPS?

I rent a Gentoo server with the usual LAMP stack (prefork Apache MPM) + suPHP.

From time to time, my server runs out of memory and slows down to a crawl (responds to pings, but it's practically impossible to log in, and keystrokes sent through SSH can take minutes to be echoed back, much less processed). Lots of oom_killer stuff in the system logs, too.

This is what I see in top during one of these moments:

top - 16:45:05 up 22 days,  8:08,  3 users,  load average: 104.26, 103.87, 93.3
Tasks: 393 total,   1 running, 388 sleeping,   0 stopped,   4 zombie
Cpu(s):  4.6%us,  9.3%sy,  0.8%ni,  0.0%id, 84.8%wa,  0.0%hi,  0.5%si,  0.0%st
Mem:   2042128k total,  1634392k used,   407736k free,     1792k buffers
Swap:        0k total,        0k used,        0k free,    27724k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3125 apache    20   0  288m 105m 1368 S    0  5.3   0:01.00 apache2
 2886 apache    20   0  285m 102m 1368 S    0  5.1   0:02.44 apache2
 3048 apache    20   0  279m  96m 1192 D    0  4.8   0:01.58 apache2
 3037 apache    20   0  278m  95m 1076 S    0  4.8   0:02.23 apache2
 3014 apache    20   0  278m  94m 1204 D    0  4.8   0:01.81 apache2
 2859 apache    20   0  274m  91m 1368 S    0  4.6   0:00.63 apache2
 3016 apache    20   0  269m  86m 1368 S    0  4.3   0:01.49 apache2
 2887 apache    20   0  269m  86m 1192 D    0  4.3   0:01.06 apache2
 2753 apache    20   0  269m  86m 1368 S    0  4.3   0:01.09 apache2
 3036 apache    20   0  266m  83m 1372 S    1  4.2   0:01.10 apache2
 3006 apache    20   0  266m  83m 1368 S    0  4.2   0:01.98 apache2
 3007 apache    20   0  265m  82m 1372 S    0  4.1   0:02.00 apache2
 3064 apache    20   0  264m  81m 1368 S    0  4.1   0:00.57 apache2
 3045 apache    20   0  263m  80m 1368 S    0  4.1   0:00.60 apache2
 2888 apache    20   0  263m  79m  416 S    0  4.0   0:01.09 apache2
 2862 apache    20   0  260m  77m 1368 S    1  3.9   0:01.95 apache2
 2891 apache    20   0  259m  76m 1332 D    0  3.9   0:01.98 apache2
 3046 apache    20   0  258m  75m 1080 S    0  3.8   0:01.20 apache2
 2873 apache    20   0  255m  72m 1380 S    0  3.6   0:01.51 apache2
 2987 apache    20   0  252m  69m 1368 S    0  3.5   0:01.04 apache2
 2666 apache    20   0  250m  67m 1368 S    0  3.4   0:00.72 apache2
 2903 apache    20   0  248m  66m 1368 S    0  3.3   0:01.02 apache2
 3013 apache    20   0  247m  63m  416 S    0  3.2   0:01.02 apache2

Note that PHP is running in CGI mode, so this is just Apache without any PHP modules.

Frankly I don't understand why else would it be slow other than running oun of RAM, yet it claims it has 400 MB of free RAM. "84.8%wa" also indicates that the system is waiting for I/O operations (paging?).

Things I tried:

• Disabling swap, with the hope that things that start eating memory rampantly will just crash instead of bringing the server down to a grind - this didn't work, it probably just began paging out memory-mapped files (executables and SOs)
• Setting oom_adj of the root Apache process to 15

• Tweaking the MPM settings:

  StartServers         5
  MinSpareServers      5
  MaxSpareServers      10
  MaxClients           50
  MaxRequestsPerChild  10
  MaxMemFree           1024
  

For now I've reduced MaxClients to 25, but now page requests take several seconds to process, and some kid with FlashGet can theoretically clog all Apache processes and effectively make all websites inaccessible :/

Questions:

1. Can anyone suggest some Apache configuration tweaks which could radically improve my situation?

2. Is it possible to tell Linux to not swap/page out sshd, bash, and everything else required for me to ssh in and kill runaway processes?

3. If the answer to the above question is 'no', someone please explain to me how is it that in this day and age modern operating systems are such horrendously flawed. Sounds like epic fail in OS design to me :(

I need to set up a CentOS 5.3 server to forward e-mails to another e-mail address. The server doesn't currently have any mail server software installed. What's the simplest way to do this? (aside setting up MX records for the domain)

(I'm aware I could just pick up any mailserver software and configure it to do this, I'd just like to know the simplest solution.)

There are some programs which can display used disk space using a treemap, such as WinDirStat for Windows and KDirStat for KDE/Linux:

I'm looking for something similar, but for a headless Linux box. (E.g. run console data collection program on the server, then load the file in a graphical program in a GUI environment.)

Alternatively, what are other good ways to get a structured used disk space representation, with just SSH access?

We have a server (4GB RAM, two 4-core Intel Xeon E5420) performing the following tasks:

• website with static content (currently just some CMS, we plan to add a caching reverse-proxy)
• customer control panel (basically web interface to a database)
• backend scripts for our desktop software - a few PHP scripts which perform a few DB queries/updates, but will be hit up every 5 minutes per customer
• database storing data for the above two

We are expecting a big (up to hundreds of thousands) influx of users in the upcoming weeks. My superiors are worried that our current setup may not be able to handle the load, and are considering moving to cloud hosting - however, I'm not convinced that this would help. From what I understand of cloud computing, virtualization won't be of much help unless we can split the workload across several machines (unless they have individual servers significantly more powerful than ours). They offer MySQL clusters, however the backend scripts simply read/write some values in a few rows and don't perform any computationally-intensive SQL queries, so again I'm not sure MySQL clusters would make much of a difference compared to Apache/PHP overhead + MySQL network latency. As for the website, we could also move most of the static content to a CDN as we have a few video clips which could stress our bandwidth.

So, would cloud hosting benefit us in any way?