Chris Allen Lane's questions

My company uses several EC2 servers that will scale up and down according to the load we're receiving on our sites at any given moment. For the sake of our discussion here, we're running four instances:

• master.ourdomain.com - the file syncing "hub" of the webservers
• www1/www2/www3.ourdomain.com - three webservers which turn on or off as dictated by load

I'm using lsyncd to keep all of the webservers in sync, and for the most part, it's working quite well. We're using a two-way syncing scheme, such that each webserver syncs against master, and master syncs against each webserver. Thus, the webservers are kept in sync, even though they aren't syncing against each other directly.

I'm having one problem that I'm having a hard time solving,though. It occurs under these circumstances:

• When changes are made on master (perhaps after we've pushed new code), while some of the redundant webservers are sleeping
• And then a sleeping webserver wakes-up to absorb load

Under that circumstance, I would like the following to happen:

• First, the newly-awoken webserver should sync its file structure - one way - against master, to bring its web application code up-to-date.
• Then, and only then, should it begin pushing changes in its file structure back to master.

Unfortunately, currently, when a sleeping server is started, when lsyncd starts up, it pushes changes back to master before updating its own codebase, thus overwriting new code with old.

Thus, before lsyncd starts, I'd like to be able to synchronize the webservers code against master's, perhaps by running a simple one-way rsync against the two machines.

We're running lsyncd v.2, and I've tried to make this happen by using the "bash" configuration options documented in the lsyncd manual. My configuration file looks like this:

settings = {
   logfile      = "/home/user/log/lsyncd/log.txt",
   statusFile   = "/home/user/log/lsyncd/status.txt",
   maxProcesses = 2,
   nodaemon     = false,
}

bash = {
    onStartup = "rsync [email protected]:/home/user/www /home/user/www"
}

sync{
    default.rsyncssh,
    source="/home/user/www/",
    host="[email protected]",
    targetdir="/home/user/www/",
    rsyncOpts="-ltus",
    excludeFrom="/home/user/conf/lsyncd/exclude"
}

(I've obviously redacted that file somewhat to protect the identities of the guilty.)

Simply put, though, this just isn't working.

How else might I approach this problem? I was looking at the --delete-after option in man rsync, but I don't think that does what I'm looking for.

Are there any suggestions about how I should approach this problem?

Thanks for lending your time and expertise.

Chris

I'm running something of a bare-bones server (based on Ubuntu 11.04) on an Amazon EC2 micro instance, whose purpose is simply to coordinate the activities of a few webservers. The machine ran well for a few weeks, but now is hanging frequently with its CPU redlined at 100%.

I logged into the machine over SSH and ran a top, which revealed that the landscape-sysinfo process was the perpetrator consuming all of the system resources. A pstree revealed where it was situated:

init─┬─atd
     ├─cron
     ├─dhclient3
     ├─dovecot─┬─2*[dovecot-auth]
     │         ├─3*[imap-login]
     │         └─3*[pop3-login]
     ├─6*[getty]
     ├─master─┬─pickup
     │        └─qmgr
     ├─mountall
     ├─mysqld───11*[{mysqld}]
     ├─rsyslogd───3*[{rsyslogd}]
     ├─sshd─┬─sshd───sshd───bash
     │      ├─sshd───sshd───bash───top
     │      ├─sshd───sshd───bash───pstree
     │      └─sshd───sh───run-parts───50-landscape-sy───landscape-sys+
     ├─udevd───2*[udevd]
     ├─upstart-socket-
     ├─upstart-udev-br
     └─vsftpd

The offending process is listed here as the last child of sshd. If I manually kill landscape-sysinfo, the machine returns to normal - until the process spontaneously respawns, usually a few moments later. (I can "vouch for" the other sshd processes in the above tree. They were legitimate.)

I have no idea why landscape-sysinfo is spawning itself randomly. I doubly have no idea why it's the child of sshd.

I'm obviously none too thrilled about having an SSH processes running on my machine that I can't account for. Initially I feared a breach/trojan/backdoor, so I ran chkrootkit and rkhunter, but they both came up clean.

Does anybody have any idea what could be causing this process to run wild? Any thoughts on how to stop it from respawning?

I've got an Apache question here.

For my entire career as a developer, I've been restarting Apache like this:

sudo /etc/init.d/apache2 restart

I just today deployed my first Rails application, but I was having trouble getting Apache to restart on the host machine. When I tried it that way, Apache would try to restart, but would ultimately hang.

This, however, worked fine:

sudo apachectl restart

What are the differences between these two techniques? I had assumed that the latter was, at best, "syntactic sugar" for the first, but that must not be the case, given that the latter worked while the former did not.

If it's relevant (and I suspect it may be), this Rails application uses Phusion Passenger. My prior development experience (whereby the former method works fine) is mostly with PHP. I'm pretty new to Rails.

The server in question is runs Ubuntu 11.04 (Natty).