Chris Adams's questions

I have a few S3 buckets that I want to hand access over to another organisation, as am handing over an existing hosting client to them.

After looking through the docs, if I no longer want to be responsible for the bucket, it seems the simplest option is to copy the contents of the new bucket across to a new bucket controlled by the new organisation, and make any existing apps write files to the new bucket from now on.

The bucket isn't very big:

aws s3 ls --human-readable --recursive --summarize s3://some-client-bucket

 # (snip… lots of files listed, all less than 10mb)

 # Total Objects: 22764
 # Total Size: 2.4 GiB

But some the bucket is versioned, and I have daily snapshots of some files going back the last year which I also want to be able to transfer across.

Is there a straightforward way to do this?

I understand how I might copy the current contents of a bucket to to new one controlled by another organisation after they grant me access using something like:

aws s3 sync s3://some-client-bucket s3://new-client-bucket --recursive

However, I don't think this will move across the versions too, and I'm I've been relying on S3's per file versioning rather than timestamping the files myself.

Do I have to jerry-rig some script to:

• download each versioned file
• rename it with a timestamp
• upload it to the new bucket

Or is there some nifty extra feature in S3 to do this automagically for me?

I'm working on a site that I'd like to serve from a naked domain like foo.com, and serve it from AWS S3'.

However, the documentation I can find with AWS gave me the impression that if I want to do this, I'll need to move move all my DNS over to Amazon's Route53.

I'd rather not do that, but as I can't use CNAMES for naked domains, nor point A-records point to a fixed IP (because they might change), I'm not sure what my options are.

Is it possible to only delegate the serving of a naked domain to another provider, without needing to move everything across?

I see snippets like this in the AWS docs, which I could add to my DNS with Gandi, but I understand this would delegate all domain handling to the Amazon Name servers, which I don't think I want.

foo.com. 172800 IN NS ns-9999.awsdns-99.com.

If my provider, Gandi doesn't support ALIAS or ANAME records, what are my options?

I understand I could spin an instance up at an IP I point to with an A record, then use it to bounce requests along from the naked domain to a static sited hosted at www.foo.com, but are there other options, but this feels like it would defeat the purpose of having static serving, so other than using a provider like Cloudflare, or wwwizer, who handle the naked domain japery for you, what else can you do?

I'm currently trying to spec up a horizontally scalable cluster for a drupal based web app, that looks something like the colorful diagram below:

The load balancer implements sticky sessions, so a user keeps state once they've been allocated a server to work with.

Each app server has the following:

• varnish at the front
• drupal 6 in the middle running on the lamp stack
• memcached at the back

The two mysql database servers are on a shared IP, and they're in a HA cluster with DRBD, and hearbeat, so that losing one won't bring down the whole platform.

There are a few things I'm not certain about that I'd appreciate your opinions on:

How should file storage scale horizontally?

I'm thinking of using NFS to mount a shared files directory on each app server, so a file uploaded in once place is available on all of them. I'm thinking of NFS because it's been around for ages, and I've no experience with MogileFS or GlusterFS, and it's something we've used before, so we're more familiar with it.

Are there any guidelines around to follow for working out how many servers it's wise to share a directory over NFS this way?

How should HA be provided on the shared file storage here?

One problem here is that the NFS server is a single point of failure.

We're already using Heartbeat and DRBD on the Mysql servers, and I'd prefer to keep the number of technologies involved in a stack as low as possible - what pitfalls would there be if I was to use the same HA strategy for the file servers too?

An alternative approach

This is for an internal facing site, with a finite number of users that occasionally use the site very intensively for short periods, when an internal initiative is on. So this doesn't need to scale infinitely like some startup.

Given that

• there is an upper limit to traffic we can expect
• adding adding HA to the file servers, and designing a setup to scale horizontally like this introduces considerable complexity

I'm also considering just making the two web servers beefier so they that would handle the peak load between them, and setting up unison, or rsync across the two on a cron job, so that:

• they files are in still in sync (sticky sessions keep a user on the same server they uploaded a file to)
• losing one means the site is still operational.

Does this sound like a possible way to get around any possible NFS/DRBD HA complexity headaches?

Thanks,

C

I'd like some feedback about this failover strategy for a couple of MySQL servers I'm speccing up for a cluster, and I want to check if there's something obvious I'm not missing here.

One app server that connects to a mysql master server in day to day operations, and that has a mysql-server set up as a slave to it for master-slave replication purposes.

If the mysql server fails, I want to the web app to try connecting to the master, and then after n failed attempts, perform the following:

• assume the master will no longer be available
• send a signal to the slave server to stop replication
• send a signal to the slave server to tell it to act as the new mysql master
• begin connecting to the server again, and treat it like the master from now on

Once the app is up again, and serving users, I'd like to be able to spin up a new slave server in the background, once it's ready to serve requests, set up master slave replication once again to provide the same failover support as before.

I'm pretty sure this has been done before, but I can't see any guides on this, so I'm assuming there must be some obvious reason you wouldn't try this, that I haven't thought of yet.

What are the pitfalls of taking this approach for providing automated failover like this with MySQL?

As an aside, I'm aware of master-master replication, but a) I've seen it go horribly wrong, and it b) seems worryingly over-complicated.

Thanks

I'm in the position of needing to make a few RPMs at work, which is something fairly new to me.

I'd really love to hear which docs, tutorials. or presos have been useful for you for understanding:

a) how to make them well without making noobish mistakes

b) any other handy tricks that you would otherwise miss if you didn't spend a day poring over the man pages.

What's been the most helpful for you?

I'm coming from a developer background, so I'm comfortable with making and compiling software manually, and I've fiddled around with making formula for homebrew but rpm is a new, powerful and shiny tool to me.

We currently serve most of our Rails and LAMP apps over Apache and Apache passenger, but we're considering adding Nginx or Varnish as a reverse proxy to reduce the load on our servers somewhat.

I'm aware that you can use Varnish and Nginx together, but given that there's investment of time in learning how both work, and where possible we want to keep the number of 'moving parts' in our infrastructure as low as possible, I'm trying to work out what the advantages and disadvantages are between using:

• nginx by itself as a the reverse reverse proxy/cache
• varnish alone as the reverse proxy/cache
• nginx and varnish together

I understand nginx is known to be extremely fast, and it's gaining prominence as a full blown http server as it gets more popular, so I can see the argument for investing some time in learning how this server works, but Varnish is still an unknown to me.

Why would I use Varnish if nCache is now in Nginx?

Thanks

I'm planning to migrate a Zimbra server with about 200 GB of data from a server hosted in an office into a datacentre, to increase uptime (we've had a couple of outages when our network here started flaking out, and we have people in other countries relying on this server too).

However, I'm not sure how best to migrate the data into the data centre without rendering the connection unusable during office hours, because there's far too much to send in over night over the 2 Mbit/s upstream connection we have here.

I'm familiar with using tools like nice to stop a long running process degrading machine performance - is there a simple way to throttle a connection between office hours, so the long running transfer doesn't block the pipe, but then opens up outside of office hours to make the most of the bandwidth?

I'm aware the alternative here is to simply mail a hard drive to the data centre, but I'd like to avoid doing that if I could.

We're using CentOS Linux for our servers, in the office and the datacentre, so extra points for an open source Linux answer.

I'm trying to consolidate domain names for the servers I look after to just use one panel instead of 3 or 4, and one thing stopping me is that the provider I originally wanted to move them to only lets me the following kinds of records:

• A
• MX
• NS
• CNAME
• TXT

The first four I understand, but I'm not sure about the relationship (if any) between SRV records and TXT records.

Can I use TXT records in the place of SRV records? They both seem to be general text records to just point at a particular server without needing to specify a particular protocol, so it doesn't sound like a totally unreasonable assumption, but I'd rather check here before I break something.

If I can only set the above records, does that mean I'm essentially unable to so any SRC record redirecting?

Thanks!

I'm having a horrible time getting a piece of software to compile on a CentOS box, and the problem I've listed below is specific, but I'm hoping the correct approach that comes out of this is one that can be applied to similar problems, as this doesn't seem to be one that's specific to just this piece of software. Getting to the point though...

I've been stuck with a compilation problem when building a standardised virtual machine on CentOS 5.4, and I'm in the dark here as to a) why this error is occurring, and b) how to fix it, and in the hope that someone else stumbles across this problem too, I'm hoping someone can help me find the solution here.

I'm getting a configure: error: newly created file is older than distributed files! error when trying to compile Ruby Enterprise like below when I try to run the installer, and the solutions offered to on the forums (of checking the tine, and touching the files to update the time associated with them) don't seem to be helping here.

What steps can I take to work out what the cause of this problem?

[vagrant@vagrant-centos-5 ruby-enterprise-1.8.7-2009.10]$ sudo ./installer
Welcome to the Ruby Enterprise Edition installer
This installer will help you install Ruby Enterprise Edition 1.8.7-2009.10.
Don't worry, none of your system files will be touched if you don't want them
to, so there is no risk that things will screw up.

You can expect this from the installation process:

  1. Ruby Enterprise Edition will be compiled and optimized for speed for this
     system.
  2. Ruby on Rails will be installed for Ruby Enterprise Edition.
  3. You will learn how to tell Phusion Passenger to use Ruby Enterprise
     Edition instead of regular Ruby.

Press Enter to continue, or Ctrl-C to abort.

Checking for required software...

 * C compiler... found at /usr/bin/gcc
 * C++ compiler... found at /usr/bin/g++
 * The 'make' tool... found at /usr/bin/make
 * Zlib development headers... found
 * OpenSSL development headers... found
 * GNU Readline development headers... found
--------------------------------------------
Target directory

Where would you like to install Ruby Enterprise Edition to?
(All Ruby Enterprise Edition files will be put inside that directory.)

[/opt/ruby-enterprise] : 
--------------------------------------------
Compiling and optimizing the memory allocator for Ruby Enterprise Edition
In the mean time, feel free to grab a cup of coffee.

./configure --prefix=/opt/ruby-enterprise --disable-dependency-tracking
checking build system type... i686-pc-linux-gnu
checking host system type... i686-pc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... configure: error: newly created file is older than distributed files!
Check your system clock

This is a virtual machine running on virtualbox, and the time of the host and the virtual machine are identical, and up to date. I've also tried running this after updating time with an ntp-client, so no avail. I tried this after reading this post here of someone having a similar problem

[vagrant@vagrant-centos-5 ruby-enterprise-1.8.7-2009.10]$ date
Tue Apr 27 08:09:05 BST 2010 

The other approach I've tried is to touch the top level the files in the build folder like suggested here, but this hasn't worked either (an to be honest, I'm not sure why it would have worked either)

[vagrant@vagrant-centos-5 ruby-enterprise-1.8.7-2009.10]$ sudo touch ruby-enterprise-1.8.7-2009.10/*

I'm not sure what I can do next here - the problem seems to be the bash configure script that returns this error error: newly created file is older than distributed files!, at line :2214

{ echo "$as_me:$LINENO: checking whether build environment is sane" >&5
echo $ECHO_N "checking whether build environment is sane... $ECHO_C" >&6; }
# Just in case
sleep 1
echo timestamp > conftest.file
# Do `set' in a subshell so we don't clobber the current shell's
# arguments.  Must try -L first in case configure is actually a
# symlink; some systems play weird games with the mod time of symlinks
# (eg FreeBSD returns the mod time of the symlink's containing
# directory).
if (
   set X `ls -Lt $srcdir/configure conftest.file 2> /dev/null`
   if test "$*" = "X"; then
      # -L didn't work.
      set X `ls -t $srcdir/configure conftest.file`
   fi
   rm -f conftest.file
   if test "$*" != "X $srcdir/configure conftest.file" \
      && test "$*" != "X conftest.file $srcdir/configure"; then

      # If neither matched, then we have a broken ls.  This can happen
      # if, for instance, CONFIG_SHELL is bash and it inherits a
      # broken ls alias from the environment.  This has actually
      # happened.  Such a system could not be considered "sane".
      { { echo "$as_me:$LINENO: error: ls -t appears to fail.  Make sure there is not a broken
alias in your environment" >&5
echo "$as_me: error: ls -t appears to fail.  Make sure there is not a broken
alias in your environment" >&2;}
   { (exit 1); exit 1; }; }
   fi
   ### PROBLEM LINE ####
   # this line is the problem line - this is returned true, sometimes it isn't and I can't 
   # see a pattern that that determines when this will test will pass or not.
   test "$2" = conftest.file
   )
then
   # Ok.
   :
else
   { { echo "$as_me:$LINENO: error: newly created file is older than distributed files!
Check your system clock" >&5
echo "$as_me: error: newly created file is older than distributed files!
Check your system clock" >&2;}
   { (exit 1); exit 1; }; }
fi

the thing that makes this really frustrating is that this script works sometimes, when the VM has been running for an hour or so it works, but not at boot. There's nothing I see in the crontab that suggests any hourly tasks are run that might change the state of the system enough make a difference to this script working.

I'm totally at a loss when it comes to debugging beyond here. What's the best approach to take here?

Thanks

I currently have an email address, [email protected], that I currently use to forward to a gmail account that I've used for years. I currently have it set, so I can send email from my [email protected] instead of [email protected].

This is done through the web ui on my current domain name, provider, gandi.net.

What are the main differences between this arrangement, and setting up a Google Apps for your domain setup, where I'm explicitly listing in the dns records to point records to point to Google's app mail server?

I have a few domains with Dreamhost where I have set a load of records using their web based domain name system, and I've like to move them to another provider that lets me enter info directly as a text file for their name server, bind 9 to use. (If you're interested, I'm moving them to Gandi.net).

Previously when I used a cpanel based system to do something similar, there was a tool that let me simply enter a domain name, and any available domains were automagically entered into a system, saving me typing it myself (and bringing down sites with silly typos in the process).

What open source tool can I use to query a domain for all the relevant subdomains and records and list them in a format like a zone file, that I can use with other name servers?

I've started administering a clutch of CentOS based webservers in a new job, and after a couple of years of using debian based linux servers, I've got fairly comfortable with the apache file structure along the lines of below:

    /etc/apache2
        apache2.conf
        conf.d/
        envvars
        mods-available
        mods-enabled
        ports.conf
        sites-available
        sites-enabled

This makes sense to me modules that are available to me but not necessarily in use are in mod-available, and ones in user are in mods-enabled, likewise with the sites directly. Granted, these are mainly symlinks, but I can understand why it's done like this; coming to the system fresh I have a rough idea of what's up on the system, by looking in the relevant *enabled directories.

By comparison CentOS's setup is like so:

    /etc/httpd
        conf
        conf.d
        logs -> ../../var/log/httpd
        modules -> ../../usr/lib64/httpd/modules
        run -> ../../var/run

I can understand that I can see which hosts are enabled by calling apachectl -S, and apachectl -M to see the modules, and renaming /conf.d/example.conf to conf.d/example.conf.disabled would allow me to disable sites fairly simply too, but I don't understand why these are different in the first place, and right now, I'm not sure of the advantages of doing it one way or another.

Are there any pages, blog posts or sites around to help explain the differences between the two, and possibly some of the thinking around why they're set up like this?

Right now I'm blindly following the convention I'm seeing in the standard CentOS file system without understanding why, which I can't help feeling will bite me when I least it expect it, like when I blunder across selinux's permissions layer, or something similar.

I'm looking after a rails app running Passenger, and MySQL, on a VPS running Ubuntu Intrepid and earlier this morning, the webserver stopped being able to connect to the server. My exception notifier sent me a message with this error:

ActionView::TemplateError: Mysql::Error: MySQL server has gone away: SELECT * FROM `table_name` WHERE (`table_name`.`id` = 70)

Shelling into the vps trying to open up mysql gave me this:

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

I cleared this error by stopping apache, rebooting the database server and restarting passenger, which seems to have everything humming along fine again, but this doesn't solve the problem of working out why this happened in the first place.

Which places should I be looking to find the cause of this error? (other than production.log, or a /var/log ), so I can avoid this in future, and seeing that this was so simple to fix, would it be worth automating a reboot of the two servers if these conditions occurred again in future?

I'm trying to work out why email I send from one domain I own is rejected by another that I own, and while I think it may be related to how I've setup spf records, I'm not sure what steps I need to take to fix it.

Here's the error message I receive:

Technical details of permanent failure:
Google tried to deliver your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 550 550-Verification failed for <[email protected]>
550-No Such User Here
550 Sender verify failed (state 14).

Here's the response from [email protected]

Delivered-To: [email protected]
Received: by 10.86.92.9 with SMTP id p9cs85371fgb;
        Wed, 2 Sep 2009 22:33:32 -0700 (PDT)
Received: by 10.90.205.4 with SMTP id c4mr2406190agg.29.1251956007562;
        Wed, 02 Sep 2009 22:33:27 -0700 (PDT)
Return-Path: <[email protected]>
Received: from verifier.port25.com (207-36-201-235.ptr.primarydns.com [207.36.201.235])
        by mx.google.com with ESMTP id 26si831174aga.24.2009.09.02.22.33.25;
        Wed, 02 Sep 2009 22:33:26 -0700 (PDT)
Received-SPF: pass (google.com: domain of [email protected] designates 207.36.201.235 as permitted sender) client-ip=207.36.201.235;
Authentication-Results: mx.google.com; spf=pass (google.com: domain of [email protected] designates 207.36.201.235 as permitted sender) [email protected]; dkim=pass [email protected]
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=auth; d=port25.com;
 h=Date:From:To:Subject:Message-Id:In-Reply-To; [email protected];
 bh=GRMrcnoucTl4upzqJYTG5sOZMLU=;
 b=uk6TjADEyZVRkceQGjH94ZzfVeRTsiZPzbXuhlqDt1m+kh1zmdUEoiTOzd89ryCHMbVcnG1JajBj
   5vOMKYtA3g==
DomainKey-Signature: a=rsa-sha1; c=nofws; q=dns; s=auth; d=port25.com;
 b=NqKCPK00Xt49lbeO009xy4ZRgMGpghvcgfhjNy7+qI89XKTzi6IUW0hYqCQyHkd2p5a1Zjez2ZMC
   l0u9CpZD3Q==;
Received: from verifier.port25.com (127.0.0.1) by verifier.port25.com (PowerMTA(TM) v3.6a1) id hjt9pq0hse8u for <[email protected]>; Thu, 3 Sep 2009 01:26:52 -0400 (envelope-from <[email protected]>)
Date: Thu, 3 Sep 2009 01:26:52 -0400
From: [email protected]
To: [email protected]
Subject: Authentication Report
Message-Id: <[email protected]>
Precedence: junk (auto_reply)
In-Reply-To: <[email protected]>

This message is an automatic response from Port25's authentication verifier
service at verifier.port25.com.  The service allows email senders to perform
a simple check of various sender authentication mechanisms.  It is provided
free of charge, in the hope that it is useful to the email community.  While
it is not officially supported, we welcome any feedback you may have at
<[email protected]>.

Thank you for using the verifier,

The Port25 Solutions, Inc. team

==========================================================
Summary of Results
==========================================================
SPF check:          pass
DomainKeys check:   neutral
DKIM check:         neutral
Sender-ID check:    pass
SpamAssassin check: ham

==========================================================
Details:
==========================================================

HELO hostname:  fg-out-1718.google.com
Source IP:      72.14.220.158
mail-from:      [email protected]

----------------------------------------------------------
SPF check details:
----------------------------------------------------------
Result:         pass 
ID(s) verified: [email protected]
DNS record(s):
    stemcel.co.uk. 14400 IN TXT "v=spf1 include:aspmx.googlemail.com ~all"
    aspmx.googlemail.com. 7200 IN TXT "v=spf1 redirect=_spf.google.com"
    _spf.google.com. 300 IN TXT "v=spf1 ip4:216.239.32.0/19 ip4:64.233.160.0/19 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:209.85.128.0/17 ip4:66.102.0.0/20 ip4:74.125.0.0/16 ip4:64.18.0.0/20 ip4:207.126.144.0/20 ?all"

----------------------------------------------------------
DomainKeys check details:
----------------------------------------------------------
Result:         neutral (message not signed)
ID(s) verified: [email protected]
DNS record(s):

----------------------------------------------------------
DKIM check details:
----------------------------------------------------------
Result:         neutral (message not signed)
ID(s) verified: 

NOTE: DKIM checking has been performed based on the latest DKIM specs
(RFC 4871 or draft-ietf-dkim-base-10) and verification may fail for
older versions.  If you are using Port25's PowerMTA, you need to use
version 3.2r11 or later to get a compatible version of DKIM.

----------------------------------------------------------
Sender-ID check details:
----------------------------------------------------------
Result:         pass 
ID(s) verified: [email protected]
DNS record(s):
    stemcel.co.uk. 14400 IN TXT "v=spf1 include:aspmx.googlemail.com ~all"
    aspmx.googlemail.com. 7200 IN TXT "v=spf1 redirect=_spf.google.com"
    _spf.google.com. 300 IN TXT "v=spf1 ip4:216.239.32.0/19 ip4:64.233.160.0/19 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:209.85.128.0/17 ip4:66.102.0.0/20 ip4:74.125.0.0/16 ip4:64.18.0.0/20 ip4:207.126.144.0/20 ?all"

----------------------------------------------------------
SpamAssassin check details:
----------------------------------------------------------
SpamAssassin v3.2.5 (2008-06-10)

Result:         ham  (-2.6 points, 5.0 required)

 pts rule name              description
---- ---------------------- --------------------------------------------------
-0.0 SPF_PASS               SPF: sender matches SPF record
-2.6 BAYES_00               BODY: Bayesian spam probability is 0 to 1%
                            [score: 0.0000]
 0.0 HTML_MESSAGE           BODY: HTML included in message

I've registered the spf records for my domain, as advised here

Both domains pass validate according to Kitterman's spf record testing tools, so I'm somewhat confused about this.

I also have the catchall address set up on the stemcel.co.uk domain here, but I don't have one setup for chrisadams.me.uk.

Instead, we have the following forwarders setup

[email protected]  to  [email protected]   
[email protected]  to  [email protected]   
[email protected]     to    [email protected]   
[email protected]     to    [email protected]

Any ideas how to get this working? I'm not sure what I should be looking for here.

Let's say I have a domain that I run a web application on, for example cranketywidgets.com, and I'm using Google Apps for handling email for people working on that domain, for example, support@ cranketywidgets.com, [email protected], [email protected] and so on.

Google's own mail services aren't always the best for sending automated reminder emails, comment notifications and so on, so the current solution I plan to pursue is to create a separate subdomain called mailer.cranketywidgets.com, run a mail server off it, and create a few accounts specifically for sending these kinds of emails.

What should the MX records and A records look like here for this?

I'm somewhat confused by the fact that MX records can be names, but that they must eventually resolve to an A record. What should the records look like here?

cranketywidgets.com - A record to actual server like 10.24.233.214

cranketywidgets.com - MX records for Google's email applications

mailer.cranketywidgets.com - MX name pointing to server's IP address

I would greatly appeciate some help on this - the answer seems like it'll be obvious, but email spam is a difficult problem to solve.

I'm used to being able to pass variables inside strings in ruby, like so

"message in double quotes #{expression_or_variable_to_run}"

What's the equivalent in bash, for really quick scripting?