Tom Marthenal's questions

I run a web server with multiple users. The machine provides Apache-based virtual hosts to a bunch of users, who can serve static content or use PHP. No other forms of server-side scripting are available.

Here's how a typical request is handled:

HTTPS traffic has its SSL stripped by Pound, which acts as a reverse proxy and forwards the request on to Varnish, which also acts as a reverse proxy, finally forwarding the request on to Apache.

By the time the HTTPS traffic reaches Apache, the protocol, port, and full URL have changed from, say, https://example.com:443/ to http://example.com:8443/. This is a result of all the proxies before Apache.

Is there a way I can trick the PHP scripts into thinking the request came from the original URL, port, and protocol, without modifying the PHP code?

This is important because the users will wish to run Joomla, WordPress, and other PHP-based CMSes which detect the URL and cause trouble with redirects, links, etc.

I could probably patch mod_php and build it from source with the necessary changes for my specific scenario, but is there a better way to give the PHP scripts fake environmental variables, perhaps via some setting in php.ini?

I have a postfix server running amavis and SpamAssassin to help filter spam. Messages that are detected as spam look like this when delivered into the user's Maildir:

From tom@tom-mint Fri Mar 15 01:46:20 2013
Return-Path: <tom@tom-mint>
X-Original-To: [email protected]
Delivered-To: [email protected]
X-Virus-Scanned: Debian amavisd-new at my-server.com
X-Spam-Flag: YES
X-Spam-Score: 6.463
X-Spam-Level: ******
X-Spam-Status: Yes, score=6.463 tagged_above=2 required=6.31
    tests=[DRUGS_ERECTILE=2.221, FH_FROMEML_NOTLD=0.18,
    FSL_HELO_NON_FQDN_1=0.001, HELO_NO_DOMAIN=0.001, RCVD_IN_PBL=3.558,
    RDNS_DYNAMIC=0.363, TO_NO_BRKTS_DYNIP=0.139] autolearn=no
Date: Fri, 15 Mar 2013 01:46:19 -0400
To: [email protected]
Subject: ***SPAM*** hello
User-Agent: Heirloom mailx 12.5 6/20/10
Content-Type: text/plain; charset=us-ascii
From: tom@tom-mint (Tom)
Status: RO

buy some viagra!

Note specifically the subject, which is prepended with ***SPAM***, and the X-Spam-Flag: YES header.

What's the best way to configure postfix to store these messages in a special junk folder?

From my research, it seems like I will need to use a different mail delivery tool than postfix to accomplish this, but I am not sure what approach is the best. There seems to be a lot of dated information, and I am wondering whether postfix is capable of placing messages into a folder or not.

The mail directories are all Maildir.

I had two of my CPUs lock up on one of my servers. From dmesg:

BUG: soft lockup - CPU#1 stuck for 23s! [vmx-vcpu-0:6148]

and later:

BUG: soft lockup - CPU#2 stuck for 23s! [vmx-vcpu-0:6148]

I'm trying to figure out why this would happen; the processor has 4 cores with hyperthreading, so the OS sees it as 8 cores. But my main question is related to this:

When looking at htop post-freeze from SSH, I see that CPUs #2 and #3 (guessing these correspond to #1 and #2 from dmesg) are both stuck at 100% with apparently no processes using them:

None of the processes were using more than 5% CPU. Why would these display 100% utilization? Are they still considered locked by the kernel?

I have a slow web app that I've placed Varnish in front of. All of the pages are static (they don't vary for a different user), but they need to be updated every 5 minutes so they contain recent data.

I have a simple script (wget --mirror) that crawls the entire website every 15 minutes. Each crawl takes about 5 minutes. The point of the crawl is to update every page in the Varnish cache so that a user never has to wait for the page to generate (since all pages have been generated recently thanks to the spider).

The timeline looks like this:

• 00:00:00: Cache flushed
• 00:00:00: Spider starts crawling to update cache with new pages
• 00:05:00: Spider finishes crawling, all pages are updated until 00:15:00

A request that comes in between 0:00:00 and 0:05:00 might hit a page that hasn't been updated yet, and will be forced to wait a few seconds for a response. This isn't acceptable.

What I'd like to do is, perhaps using some VCL magic, always foward requests from the spider to the backend, but still store the response in the cache. This way, a user will never have to wait for a page to generate since there is no 5-minute window in which parts of the cache are empty (except perhaps at server startup).

How can I do this?

My understanding is that running rm on a file simply unlinks it, marking the space as free in the filesystem. It should then follow that deleting one file always takes roughly the same amount of time (i.e. delete speed is proportional to number of files, not size of files).

So why does deleting a 15 GB file take over a minute with a simple rm file.tar.gz?

I'm in the process of migrating an aging shared-hosting system to more modern technologies. Right now, plain old insecure FTP is the only way for customers to access their files.

I plan on replacing this with SFTP, but I need a way to create multiple SFTP users that correspond to one UNIX account. A customer has one account on the machine (e.g. customer) with a home directory like /home/customer/.

Our clients are used to being able to create an arbitrary number of FTP accounts for their domains (to give out to different people). We need the same capability with SFTP.

My first thought is to use SSH keys and just add each new "user" to authorized_keys, but this is confusing for our customers, many of whom are not technically-inclined and would prefer to stick with passwords.

SSH is not an issue, only SFTP is available. How can we create multiple SFTP accounts (customer, customer_developer1, customer_developer2, etc.) that all function as equivalents and don't interfere with file permissions (ideally, all files should retain customer as their owner)?

My initial thought was some kind of PAM module, but I don't have a clear idea of how to accomplish this within our constraints. We are open to using an alternative SSH daemon if OpenSSH isn't suitable for our situation; again, it needs to support only SFTP and not SSH.

Currently our SSH configuration has this appended to it in order to jail the users in their own directories:

# all customers have group 'customer'
Match group customer
    ChrootDirectory /home/%u    # jail in home directories
    AllowTcpForwarding no
    X11Forwarding no
    ForceCommand internal-sftp  # force SFTP
    PasswordAuthentication yes  # for non-customer accounts we use keys instead

Our servers are running Ubuntu 12.04 LTS.

We are in the process of setting up a hosting machine that clients will have SSH access to. We don't want them to be able to see a list of user accounts, but the problem we have run into are commands like w and who.

We could disable those, but then they could just bring their own binary. I just tried on a shared webhost, and I was the only user who showed up in w, even though it's a large shared box and I doubt I'm the only one on.

How can I prevent users from seeing who else has logged in via SSH?

We have about 20 Macs in our shop, all of which have been setup and configured as they were purchased. We have a mix of computers and years (some Mac Minis, some iMacs, some MacBook Pros). They're not even all running the same version of OS X.

We'd like to setup a fresh install of the latest OS X (Mountain Lion as of today) with all applications installed and all network settings configured, and then clone that to all of the Macs.

My first thought is to simply clone the hard drive with dd, but I'm not sure this will actually work due to hardware differences. In addition, computer names and the like will need to be changed (similar to Sysprep on Windows).

What software or process should I use to image computers running OS X?

I am in charge of hosting a PHP application that is large and slow, but easy to scale. The application is entirely static, with writable disk storage needed. We've profiled the application, and the main bottleneck appears to come from loading the application and not the work the application does. The application is not CPU-intensive, although it does use a fair amount of memory (think Magento).

Currently we distribute it by having a series of servers with the same PHP files on their hard drive and a load balancer in front of them. Easy but expensive.

I've been reading about RAM disks and the IO benefits they offer, and was wondering if they would be well-suited to PHP applications.

Since PHP applications are loaded from disk for every request and often involve lots of different files (as opposed to being kept in memory like with a Java application), I would figure that disk performance can be a severe bottleneck.

Would placing the PHP files on a RAM disk and using the mount point as Apache's document root offer performance benefits? A startup script could create the RAM drive and then copy the files (which are plain-text and small) from a permanent location to the temporary RAM drive.

Does this make sense, or should I just trust the linux kernel to cache the appropriate files in memory by itself?

We all know what 127.0.0.1 is used for (loopback).

What are uses cases for the rest of the reserved 127.0.0.0/8 loopback space?

I manage a server (running Ubuntu) which hosts our client's sites with a few dozen different PHP-based websites, mostly small sites but also some installations of CMSes and forums.

I used the get_loaded_extensions() method to see what extensions I have loaded. To help streamline the server (remove unnecessary extensions to make upgrading easier and marginally improve speed), I'd like to remove extensions that aren't being used by any of the sites.

I currently have 54 different extensions loaded.

I can easily eliminate some of these from the list which I know are used, but others I am less sure about. Is there some way that I can see extensions which have not been used recently?

I am using s3cmd to upload to S3:

# s3cmd put 1gb.bin s3://my-bucket/1gb.bin
1gb.bin -> s3://my-bucket/1gb.bin  [1 of 1]
  366706688 of 1073741824    34% in  371s   963.22 kB/s

I am uploading from Linode, which has an outgoing bandwidth cap of 50 Mb/s according to support (roughly 6 MB/s).

Why am I getting such slow upload speeds to S3, and how can I improve them?

Update:

Uploading the same file via SCP to an m1.medium EC2 instance (SCP from my Linode to the instance's EBS drive) gives about 44 Mb/s according to iftop (any compression done by the cipher is not a factor).

Traceroute: Here's a traceroute to the server it's uploading to (according to tcpdump).

# traceroute s3-1-w.amazonaws.com.
traceroute to s3-1-w.amazonaws.com. (72.21.194.32), 30 hops max, 60 byte packets
 1  207.99.1.13 (207.99.1.13)  0.635 ms  0.743 ms  0.723 ms
 2  207.99.53.41 (207.99.53.41)  0.683 ms  0.865 ms  0.915 ms
 3  vlan801.tbr1.mmu.nac.net (209.123.10.9)  0.397 ms  0.541 ms  0.527 ms
 4  0.e1-1.tbr1.tl9.nac.net (209.123.10.102)  1.400 ms  1.481 ms  1.508 ms
 5  0.gi-0-0-0.pr1.tl9.nac.net (209.123.11.62)  1.602 ms  1.677 ms  1.699 ms
 6  equinix02-iad2.amazon.com (206.223.115.35)  9.393 ms  8.925 ms  8.900 ms
 7  72.21.220.41 (72.21.220.41)  32.610 ms  9.812 ms  9.789 ms
 8  72.21.222.141 (72.21.222.141)  9.519 ms  9.439 ms  9.443 ms
 9  72.21.218.3 (72.21.218.3)  10.245 ms  10.202 ms  10.154 ms
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

The latency looks reasonable, at least until the server stopped responding to ping requests.

I've been using Linode for over a year now, and, unlike some lesser-known VPS hosts I've used, I've never been required to shut down my VPS by Linode. The only restarts have been ones I've initiated.

How do they go for years on end without requiring restarts and with no downtime? Isn't downtime inevitable when upgrading some parts of the host system? Do they simply perform as few updates as possible?

This isn't meant to be a Linode-specific question; I am only using them as an example because I have experience with them.

I have a VPS with Linode right now. I was alerted by my monitoring service that a site I was hosting had gone down. I used Lish, Linode's method of getting direct out-of-band access to the console over an SSH connection but without using SSH, to see any error messages. This is what I saw:

I checked my Munin logs to see if there was a spike in memory usage, and indeed there is a spike at the appropriate time for the swap graph:

However, there was no spike on the memory graph (although swap does seem to be rising slightly):

I restarted the server and it has been working fine since. I checked Apache access and error logs and saw nothing suspicious. The last entry in syslog prior to the server restart was an error with the IMAP daemon and does not appear to be related:

Oct 28 18:30:35 hostname imapd: TIMEOUT, [email protected], ip=[::ffff:XX.XX.XX.XX], headers=0, body=0, rcvd=195, sent=680, time=1803
# all of the startup logs below here
Oct 28 18:40:33 hostname kernel: imklog 5.8.1, log source = /proc/kmsg started.

I tried checking dmesg but didn't see anything suspicious either. The last few lines:

VFS: Mounted root (ext3 filesystem) readonly on device 202:0.
devtmpfs: mounted
Freeing unused kernel memory: 412k freed
Write protecting the kernel text: 5704k
Write protecting the kernel read-only data: 1384k
NX-protecting the kernel data: 3512k
init: Failed to spawn console-setup main process: unable to execute: No such file or directory
udevd[1040]: starting version 173
Adding 524284k swap on /dev/xvdb.  Priority:-1 extents:1 across:524284k SS
init: udev-fallback-graphics main process (1979) terminated with status 1
init: plymouth main process (1002) killed by SEGV signal
init: plymouth-splash main process (1983) terminated with status 2
EXT3-fs (xvda): using internal journal
init: plymouth-log main process (2017) terminated with status 1
init: plymouth-upstart-bridge main process (2143) terminated with status 1
init: ssh main process (2042) terminated with status 255
init: failsafe main process (2018) killed by TERM signal
init: apport pre-start process (2363) terminated with status 1
init: apport post-stop process (2371) terminated with status 1

I tried Googling the error message (kernel BUG at mm/swapfile.c:2527!) and found a few Xen related topics (Linode uses Xen):

• Xen-devel Re: kernel BUG at mm/swapfile.c:2527! was 3.0.0 Xen - Xen Source
• Mailing List Archive: Re: Re: kernel BUG at mm/swapfile.c:2527! was 3.0.0 Xen pv guest - BUG: Unable to handle

However, none of the information I found seemed to point to any solution. I am going to upgrade to the latest kernel Linode offers (from 2.6.39.1-linode34 to 3.0.4-linode38).

Is there anything else I can do to diagnose this problem now, or in the future if it should happen again? Is there anything I missed? Does anybody have ideas for what may have triggered this?

Please let me know if there's any other information I can provide. Thanks a ton.

I have several virtual hosts, and I want to run each one of them on a separate user to isolate the websites from each other. I have searched for a solution, but I can't find an up-to-date solution. What's the best way to run each virtual host asa separate user?

I am running Ubuntu 11.10.

I've setup a working postfix server except that all incoming mail is rejected.

When I try to send mail via telnet:

MAIL FROM: <[email protected]>
250 2.1.0 Ok
RCPT TO: <[email protected]>
554 5.7.1 <my.host.name[1.2.3.4]>: Client host rejected: Access denied

My postconf -n

alias_database = hash:/etc/postfix/aliases
alias_maps = hash:/etc/postfix/aliases
append_dot_mydomain = no
biff = no
broken_sasl_auth_clients = no
config_directory = /etc/postfix
delay_warning_time = 4h
inet_interfaces = all
mailbox_size_limit = 0
masquerade_domains = mail.mydomain.com www.mydomain.com
maximal_backoff_time = 8000s
maximal_queue_lifetime = 7d
minimal_backoff_time = 1000s
mydestination = 
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
myorigin = techxonline.net
readme_directory = no
recipient_delimiter = +
relayhost = 
smtp_helo_timeout = 60s
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_banner = $myhostname ESMTP $mail_name
smtpd_client_restrictions = reject_rbl_client sbl.spamhaus.org,         reject_rbl_client blackholes.easynet.nl,        reject_rbl_client dnsbl.njabl.org
smtpd_data_restrictions = reject_unauth_pipelining
smtpd_hard_error_limit = 12
smtpd_helo_restrictions = permit_mynetworks, warn_if_reject reject_non_fqdn_hostname,       reject_invalid_hostname, permit
smtpd_recipient_limit = 16
smtpd_recipient_restrictions = reject_unauth_pipelining, permit_mynetworks,         permit_sasl_authenticated, reject_non_fqdn_recipient, reject_unknown_recipient_domain,      reject_unauth_destination, check_policy_service inet:127.0.0.1:10023, permit
smtpd_sasl_auth_enable = yes
smtpd_sasl_local_domain = 
smtpd_sasl_security_options = noanonymous
smtpd_sender_restrictions = permit_sasl_authenticated, permit_mynetworks,       warn_if_reject reject_non_fqdn_sender, reject_unknown_sender_domain,        reject_unauth_pipelining, permit
smtpd_soft_error_limit = 3
smtpd_tls_cert_file = /etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file = /etc/ssl/private/ssl-cert-snakeoil.key
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtpd_use_tls = yes
unknown_local_recipient_reject_code = 450
virtual_alias_maps = mysql:/etc/postfix/mysql_alias.cf
virtual_gid_maps = static:5000
virtual_mailbox_base = /var/spool/mail/virtual
virtual_mailbox_domains = mysql:/etc/postfix/mysql_domains.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql_mailbox.cf
virtual_uid_maps = static:5000

In /var/log/syslog after sending from Gmail:

Oct 18 21:30:01 appman postfix/smtpd[25307]: connect from mail-gx0-f181.google.com[209.85.161.181]
Oct 18 21:30:01 appman postfix/smtpd[25307]: NOQUEUE: reject: RCPT from mail-gx0-f181.google.com[209.85.161.181]: 554 5.7.1 <mail-gx0-f181.google.com[209.85.161.181]>: Client host rejected: Access denied; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<mail-gx0-f181.google.com>
Oct 18 21:30:01 appman postfix/smtpd[25307]: disconnect from mail-gx0-f181.google.com[209.85.161.181]

How can I get my postfix server to accept mail? If there is any other information I can provide please let me know.

EDIT: It seems like the server is requiring authentication to receive mail here. It doesn't seem to be host-restricted—using telnet from the server itself still causes the mail to be rejected. Authenticating with SASL and then sending the email works fine.

So, it seems that the problem is the server expects authentication for mail to be delivered at the final destination, which it shouldn't. Ideas?

I've followed this guide to setup an email server with IMAP and SMTP. IMAP is working great, but SMTP doesn't work.

If I try to send a message using telnet:

MAIL FROM: <[email protected]>
250 2.1.0 Ok
RCPT TO: <[email protected]>
554 5.7.1 <[email protected]>: Relay access denied

Attempting to login with my mail client (Mail.app on OS X Lion) gives this error in the system log:

postfix/smtpd[13322]: warning: my.computer.hostname[12.34.56.78]: SASL PLAIN authentication failed: no mechanism available

The client says "The server rejected the password given."

What can I do to get email relaying working?

To clarify:

I'm using my public hostname to connect to a MySQL database. The hostname resolves to my server's external IP (e.g. 1.2.3.4). Is the data I'm sending/receiving via the MySQL connection going over the internet at all? Would it be faster to use localhost? Will it take up my server's bandwidth?

Right now I have a small server handling a few things—DNS, HTTP serving, email, etc—and am storing all my data in /data/, a directory I create on all servers I administrate.

The reason I like to store data in /data/ is so that I can easily back up the important data by just tar-ing the entire directory. If I were to backup the entire system, I'd have a lot of extra data that I don't need (at the moment, I backup the system once a week and the /data/ directory nightly).

Since I'm planning a relatively large setup right now (10+ machines), I'm wondering if there's any reason I should store data in /var/ or another directory instead of /data/.

Thanks for the advice.

I just ran (not on purpose!) rm -rf /bin.

I've booted down the computer and am using Finnix to try to recover from it. I have succeeded in mounting the drive, and confirmed that, yes, the entire /bin folder is deleted.

Is it possible to recover from this without reinstalling the OS?

I'm thinking that I could setup a VM with the same OS and architecture (Ubuntu Server 11.10 alpha release, x86) and install all the packages I had installed on the server, then just copy the /bin folder.

Will this work? Am I better off just starting over?