sa289's questions

We moved a site from Apache 2.2 to 2.4 (same PHP version) and now code that used to work is getting a 500 error due to [core:error] [pid ######] [client ##.##.##.##:#####] malformed header from script 'index.php': Bad header:

We identified the issue in the code which sent a bad header and can fix it, but if we wanted legacy code not to break if there are other occurrences buried on various sites that we plan to move, is there some way to tell Apache not to fail on this? Like can we lower the strictness of errors it cares about or something?

Ran into a weird issue where a wildcard CNAME record (i.e. *.example.com) was overriding specific A records (i.e. host1.example.com, host2.example.com). It only affected Verizon Wireless's nameservers. The authoritative nameservers are controlled by Network Solutions (ns1.dnsbycomodo.net and ns2.dnsbycomodo.net).

Other providers' nameservers returned the correct results (OpenDNS and mxtoolbox.com), and it can't be a caching issue because the incorrect IPs (via the CNAME lookup) being returned were never previously used, and on top of that, the change was made 12 hours prior and the TTL on the records was only 7200.

Deleting the wildcard CNAME record appears to have solved the issue. Any thoughts on what happened? Has anyone else run into this? Is this just some bug with Verizon's DNS servers talking to Network Solutions'? Supposedly wildcard CNAME records have been valid for a while (Is a wildcard CNAME DNS record valid?).

EDIT:

Here's the order that things happened

Original config:
A *.example.com -> 1.1.1.1
A host1.example.com -> 2.2.2.2
A host2.example.com -> 3.3.3.3

Changed to:
Removed "A" *.example.com
Added CNAME *.example.com -> hostalias.example.net which resolves to 4.4.4.4

Outcome:
On Verizon queries to host1.example.com and host2.example.com started returning 4.4.4.4 whereas on OpenDNS and mxtoolbox.com, they still correctly returned 2.2.2.2 and 3.3.3.3, respectively.

In moving a website with a dedicated IP from one server to another, to minimize downtime due to DNS propagation delays, there's the approach of using IP forwarding so that all traffic to the original IP gets forwarded to the new IP.

Is there anything important to know when doing this? Here are the steps I plan to use. Is there anything from a security perspective or otherwise that I'm missing?

1. echo "1" > /proc/sys/net/ipv4/ip_forward (or set it permanently)
2. iptables -t nat -A PREROUTING -d original.ip.goes.here -p tcp --dport 80 -j DNAT --to-destination new.ip.goes.here
3. iptables -t nat -A POSTROUTING -p tcp -d new.ip.goes.here --dport 80 -j MASQUERADE
4. Repeat #2 and #3 but for port 443 instead of 80 if the site has SSL

I understand downtime can be reduced without resorting to this by lowering the TTL of the DNS records far enough in advance of the change, but that's still not quite as good at this at minimizing downtime since supposedly some DNS servers (and perhaps clients) will cache records for longer than the TTL says if it's short.

EDIT:

Part of what got me wondering if there's something I'm missing is the question of why ip_forward isn't always set to 1 and instead defaults to 0 - like is there some security risk or undesired behavior if having it set to 1 in certain situations.

I spun up an Amazon virtual instance for Windows 2008 R2 Datacenter Edition and when I went into Windows Updates in control panel, automatic updates were turned off. I turned them on and checked for updates and it found security updates, so I'm assuming I took the correct action, but is there something I'm missing and they should be disabled? Is there a reason why they start out disabled?

Sometimes if text that is on the clipboard is pasted into shell and it contains a return character it will execute even though that wasn't the intention. Sometimes even multiple lines may be pasted in by mistake and then unintended commands could be run.

I understand in a perfect world, these mistakes would never happen, but is there a way to prevent this to avoid the risk of human error?

We get an issue where messages intermittently fail with a "550 Unauthenticated senders not allowed". We're working on getting to the bottom of that, but in the meantime, I want to configure exim such that when messages fail with that error, they go into a queue to be automatically retried (since the error is intermittent, they may work fine a few minutes later so that'd be a fine workaround in the meantime).

Currently what happens is they bounce and get frozen. If I thaw one of the messages, it's not the original message, but the bounce message.

Is this possible?

If a server is configured to check the RAID weekly using /usr/sbin/raid-check and the output of smartctl -a is being monitored, is it worthwhile to also have regular SMART short and long self tests be configured to run, or would that be overkill?

In case it matters, this is for Intel DC S3500 SSD drives in a RAID configuration that can tolerate 2 drive failures.

When yum installs updates sometimes, it will give some message like:

warning: /etc/ssh/sshd_config created as /etc/ssh/sshd_config.rpmnew

My thought is that it would be wise to act upon these because perhaps occasionally there's some tweak to a config file that is important to be performed for security reasons; however, I'm wondering if I'm just being too cautious and that's just a theoretical concern that isn't really an issue in practice.

I guess what I'm asking is anyone aware of any case in the past few years where not merging in a .rpmnew file would have had some noteworthy undesirable implication - especially security-wise, but other angles like stability or desirability of configuration may be worth mentioning.

Is there any issue with having a high number of spare httpd processes always running? I ran a test where I increased StartServers and MinSpareServers by 1,000 and measured the increase in memory usage and it was only 500MB.

In light of this I was thinking that since we have a lot of RAM, to get optimal performance during bursty traffic periods, we might as well have StartServers and MinSpareServers set to around 1,000 and of course set ServerLimit and MaxRequestWorkers (formerly MaxClients) to something higher.

Are there any drawbacks of doing this that I'm not aware of assuming our server is capable of handling that many requests at once and we use MaxConnectionsPerChild as a precaution against memory leaks?

As a side note for anyone thinking 0.5MB per httpd process is incorrect, from what I've read the reason why Apache memory usage is way less than what's reported for an individual process by top is that it uses shared libraries.

If using entire drives for an mdadm RAID which will be the boot device as well, is it more correct / standard to:

1) Configure the RAID out of partitions that encompass the whole drive (like /dev/sda1 + /dev/sdb1) and then partition the resulting single md device into the various partitions.

OR

2) Create all the partitions on each drive in their desired sizes and then create RAIDs of those (e.g. sda1 + sdb1, sda2 + sdb2, sda3 + sdb3, etc.)

I'm thinking the benefit to #1 would be ease of drive replacement, and also I was told that #1 allows mdadm to parallelize reads across the various member drives more effectively.

Is there some authoritative link which talks about one as being the preferred way to go?

We have a three-way RAID 1 mirror which is powered by mdadm. I think I read that mdadm is supposed to take multiple simultaneous read requests and distribute them across different drives in the mirror (parallelize the reads) to improve read performance, but in our testing and watching the output of iostat -xm 1, it appears only /dev/sda is being used even though I/O to that device is being saturated from 5 different md devices.

Am I misunderstanding something? Does mdadm need to be configured differently? Does our version (CentOS 6.7) not support this? I'm not sure why it's behaving this way.

Benchmark setup - run the following commands simultaneously:

dd if=/dev/md2 bs=1048576 of=/dev/null count=25000
dd if=/dev/md3 bs=1048576 of=/dev/null count=25000
dd if=/dev/md4 bs=1048576 of=/dev/null count=25000
dd if=/dev/md5 bs=1048576 of=/dev/null count=25000
dd if=/dev/md6 bs=1048576 of=/dev/null count=25000

While those are going watch the output of iostat -xm 1 (sample output included below - the mirror is made up of sda, sdb, and sdc).

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sda           100669.00     0.00 10710.00    0.00   435.01     0.00    83.18    33.28    3.11   0.09 100.00
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdd               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md1               0.00     0.00 19872.00    0.00    77.62     0.00     8.00     0.00    0.00   0.00   0.00
md2               0.00     0.00 18272.00    0.00    71.38     0.00     8.00     0.00    0.00   0.00   0.00
md5               0.00     0.00 18272.00    0.00    71.38     0.00     8.00     0.00    0.00   0.00   0.00
md7               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md6               0.00     0.00 18240.00    0.00    71.25     0.00     8.00     0.00    0.00   0.00   0.00
md4               0.00     0.00 18208.00    0.00    71.12     0.00     8.00     0.00    0.00   0.00   0.00
md3               0.00     0.00 18528.00    0.00    72.38     0.00     8.00     0.00    0.00   0.00   0.00
md0               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

When doing some performance tests writing to a SSD-based 3-way RAID 1 mirror powered by mdadm, it appears we suffer a significant write penalty of about 2.2x slower than doing the same test on just one individual drive. We're reading from and writing to the same underlying physical drives in this test because that simulates our real world test case we're interested in.

Is this slowdown due to SATA III speed limitations or something else? I'm surprised RAID 1 would have a write penalty because I'd think it could write to all three drives simultaneously at the same speed it could write to one of them.

All three drives present:
dd if=/dev/md3 of=test.file bs=1048576 count=37193
...207.748 s, 188 MB/s

Just two drives present (i.e. normal two-drive RAID 1)
dd if=/dev/zero of=test.file bs=1048576 count=37193
...119.016 s, 328 MB/s

Just one drive present (no redundancy)
dd if=/dev/zero of=test.file bs=1048576 count=37193
...93.794 s, 416 MB/s

To ensure there aren't security weaknesses that can be exploited via .htaccess, how can we see all possible handlers that Apache can use via directives like SetHanlder and AddHandler.

I know I can find some by looking at the output of mod_info or grepping through conf files (which was an approach I found when trying to find an answer to this question), but that doesn't mean there aren't handlers available that aren't listed that way. I imagine I could download and search through all the source code of Apache and every module that's loaded and see what they are registering, but is there a better way?

A quick fix would be to not allow AddHandler and SetHandler in .htaccess, but that has its drawbacks. For example, Drupal has a valid use-case for the SetHandler directive as it uses it as part of defense in depth for preventing PHP scripts from running if someone somehow manages to upload them to the site through a file uploader.

Here are some examples of what I'm referring to:

AddHandler server-parsed .shtml

SetHandler server-info

AddHandler application/x-httpd-php5 .php

Here's Drupal's use case from sites/default/files/.htaccess if anyone's interested:

# Set the catch-all handler to prevent scripts from being executed.
SetHandler Drupal_Security_Do_Not_Remove_See_SA_2006_006
<Files *>
  # Override the handler again if we're run later in the evaluation list.
  SetHandler Drupal_Security_Do_Not_Remove_See_SA_2013_003
</Files>

EDIT July 30, 2015: This is not an exhaustive answer, but in case it's helpful to anyone, I found some handlers listed at the top of http://httpd.apache.org/docs/2.4/handler.html and also read that custom handlers can be created by the Action directive (see http://httpd.apache.org/docs/2.4/mod/mod_actions.html#action).

EDIT Sept 1, 2015: I wonder if there's some way to dump the memory of an httpd process and find the handlers names in there. I tried this using gcore, but didn't have success (though I've never used that before and got a warning when doing it, so I'm not sure if I'm doing something wrong).

I noticed between two servers I get about 500Mbps of throughput and between two others I get about 25Mbps. While it may not be related to my issue (which I'm not interested in solving for the purposes of this question), this got me thinking about how you could have a 1Gbps port speed at a data center but if it's 20 low quality hops from any given end user or other servers (such as because of being on a carrier with very poor peering), then you're more apt to get subpar performance.

I was wondering if anyone is aware of a good way to test the quality of a server's connection with regards to this? I'm thinking maybe a website that you visit and the server uploads and downloads data and measures latency and jitter to 100 different machines that represent a variety of major ISPs and data center carries - almost like a speedtest.net x100.

We're planning to move from a 4-drive RAID6 to a SSD RAID 1, and I was analyzing smartctl data to see how many GB we've writen to the drives over the past year so I can estimate the lifetime of the SSDs based on their endurance rating.

I'm thinking that if 1GB is written to the RAID 6, then to put it simply each of the 4 drives incurs 500MB of writing (250MB for data and 250MB for parity), so I should take whatever number of GB was written to each drive for the year and multiply it by 2 to get the GB per year estimate for the SSDs in the RAID1.

Does this seem correct or am I missing something? I didn't know if RAID6 has some additional write amplification due to its algorithm or something like that.

EDIT #2 July 23, 2015: Looking for a new answer that identifies an important security item missed in the below setup or can give reason to believe everything's covered.

EDIT #3 July 29, 2015: I'm especially looking for a possible misconfiguration like inadvertently permitting something that could be exploited to circumvent security restrictions or worse yet leaving something wide open.

This is multi-site / shared hosting setup and we want to use a shared Apache instance (i.e. runs under one user account) but with PHP / CGI running as each website's user to ensure no site can access another site's files, and we want to make sure nothing's being missed (e.g. if we didn't know about symlink attack prevention).

Here's what I have so far:

• Make sure PHP scripts run as the website's Linux user account and group, and are either jailed (such as using CageFS) or at least properly restricted using Linux filesystem permissions.
• Use suexec to ensure that CGI scripts can't be run as the Apache user.
• If needing server-side include support (such as in shtml files), use Options IncludesNOEXEC to prevent CGI from being able to be run when you don't expect it to (though this shouldn't be as much of a concern if using suexec).
• Have symlink attack protection in place so a hacker can't trick Apache into serving up another website's files as plaintext and disclosing exploitable information like DB passwords.
• Configure AllowOverride / AllowOverrideList to only allow any directives that a hacker couldn't exploit. I think this is less of a concern if the above items are done properly.

I'd go with MPM ITK if it wasn't so slow and didn't run as root, but we're specifically wanting to use a shared Apache yet make sure it's done securely.

I found http://httpd.apache.org/docs/2.4/misc/security_tips.html, but it wasn't comprehensive on this topic.

If it's helpful to know, we're planning to use CloudLinux with CageFS and mod_lsapi.

Is there anything else to make sure to do or know about?

EDIT July 20, 2015: People have submitted some good alternate solutions which are valuable in general, but please note that this question is targeted only regarding the security of a shared Apache setup. Specifically is there something not covered above which could let one site access another site's files or compromise other sites somehow?

Thanks!

Does anyone know when the end of life date will be for Apache 2.2.x? Judging from historical patterns, I'm guessing in 2016 or 2017 (both 1.3 and 2.0 were between 11 and 12 years, and 2.2 came out in 2005).

I was hoping to find something authoritative such as a planned end of life date or a minimum committed end of life date (i.e. the Apache Software Foundation guarantees it will be supported until at least 20##), but I couldn't find anything online other than what's happened historically.

See table on https://en.wikipedia.org/?title=Apache_HTTP_Server#Development for historical release and EOL dates.

In setting up a hosting environment with multiple sites that should be isolated from each other, I did the obvious step of having PHP be configured so it runs as the user that's associated with each website rather than as the Apache user, but is there some way to be sure there's no way that one site can run a script written in some other language (e.g. Python) as the Apache user? I've seen clever attacks like the symlink attack which uses .htaccess rules and a symlink to trick Apache into serving PHP files from other websites as plaintext (this isn't related to the question, just an example), and since I'm not very familiar with the setup of server-side languages other than PHP, I didn't know what to check and/or prohibit such as via the Apache conf file in order to make sure scripts written in other languages can't be run as the Apache user.

For example, even though PHP is set to run as the website user, if a website got hacked, could the hacker create an .htaccess file which sets .abc files to run as Perl scripts and if things aren't configured properly on the server, then those run as the Apache user?

What's the best way to approach this?

Is a 3-way RAID1 using mdadm a good solution to be able to sustain any two drives failing without the RAID failing? I know this costs extra in the sense of only being able to use 1/3 of the disk space (1 out of the 3 drives), but what about aside from that?

Wikipedia says "RAID 2 is the only standard RAID level, other than some implementations of RAID 6, which can automatically recover accurate data from single-bit corruption in data."

Does anyone know if the RAID 6 mdadm implementation in Linux is one such implementation that can automatically detect and recover from single-bit data corruption. This pertains to CentOS / Red Hat 6 if those are different from other versions. I tried searching online but didn't have much luck.

With SATA error rates being 1 in 1E14 bits, and a 2TB SATA disk containing 1.6E13 bits, this is especially relevant to preventing data corruption.

EDIT 17-Jun-2015

I believe this is less of a concern that I originally thought - see Hard disk / SSDs - detection and handling of errors - is silent data corruption reliably prevented? for more details