jberryman's questions

I'm experiencing a strange issue with a fabric script I'm using to bootstrap a server on EC2.

I launch a stock Ubuntu 12.04 AMI (ami-3d4ff254), wait for it to start, then proceed with:

with settings(host_string="ubuntu@%s" % i.dns_name, 
              connection_attempts=30):
    sudo('apt-get -qy update')
    sudo('apt-get -qy install --no-install-recommends mdadm') # don't install postfix
    #etc...

The apt-get update appears to run fine and gives no errors, however (2/3 of the time or so) installing mdadm throws a "no installation candidate" error.

When I ssh into the server and run apt-get install mdadm I get the same error. Running apt-get update by hand, then the package installs fine.

Any ideas on what might be happening, or ideas for debugging?

EDIT: It turns out by adding a sleep(10) after the SSH server comes up and before trying apt-get update it works fine. This seems to be an issue with the configuration of apt on this AMI combined with perhaps some strangeness on Amazon's end, or DNS issues.

Here are my entries in /etc/apt/sources.list:

deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise main
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise main
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise-updates main
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise-updates main
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise universe
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise universe
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise-updates universe
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise-updates universe
deb http://security.ubuntu.com/ubuntu precise-security main
deb-src http://security.ubuntu.com/ubuntu precise-security main
deb http://security.ubuntu.com/ubuntu precise-security universe
deb-src http://security.ubuntu.com/ubuntu precise-security universe

Here is the output from the apt-get update that screws up my mdadm install:

Ign http://security.ubuntu.com precise-security InRelease
Get:1 http://security.ubuntu.com precise-security Release.gpg [198 B]
Ign http://archive.ubuntu.com precise InRelease
Get:2 http://security.ubuntu.com precise-security Release [49.6 kB]
Ign http://archive.ubuntu.com precise-updates InRelease
Hit http://archive.ubuntu.com precise Release.gpg
Get:3 http://archive.ubuntu.com precise-updates Release.gpg [198 B]
Hit http://archive.ubuntu.com precise Release
Get:4 http://security.ubuntu.com precise-security/main amd64 Packages [177 kB]
Get:5 http://archive.ubuntu.com precise-updates Release [49.6 kB]
Hit http://archive.ubuntu.com precise/main amd64 Packages
Hit http://archive.ubuntu.com precise/restricted amd64 Packages
Hit http://archive.ubuntu.com precise/universe amd64 Packages
Hit http://archive.ubuntu.com precise/multiverse amd64 Packages
Hit http://archive.ubuntu.com precise/main i386 Packages
Hit http://archive.ubuntu.com precise/restricted i386 Packages
Hit http://archive.ubuntu.com precise/universe i386 Packages
Hit http://archive.ubuntu.com precise/multiverse i386 Packages
Get:6 http://security.ubuntu.com precise-security/restricted amd64 Packages [3,969 B]
Hit http://archive.ubuntu.com precise/main TranslationIndex
Get:7 http://security.ubuntu.com precise-security/universe amd64 Packages [48.6 kB]
Hit http://archive.ubuntu.com precise/multiverse TranslationIndex
Get:8 http://security.ubuntu.com precise-security/multiverse amd64 Packages [2,188 B]
Get:9 http://security.ubuntu.com precise-security/main i386 Packages [183 kB]
Hit http://archive.ubuntu.com precise/restricted TranslationIndex
Get:10 http://archive.ubuntu.com precise/universe TranslationIndex [2,922 B]
Get:11 http://archive.ubuntu.com precise-updates/main amd64 Packages [405 kB]
Get:12 http://security.ubuntu.com precise-security/restricted i386 Packages [3,968 B]
Get:13 http://security.ubuntu.com precise-security/universe i386 Packages [48.7 kB]
Get:14 http://security.ubuntu.com precise-security/multiverse i386 Packages [2,357 B]
Get:15 http://security.ubuntu.com precise-security/main TranslationIndex [73 B]
Get:16 http://security.ubuntu.com precise-security/multiverse TranslationIndex [71 B]
Get:17 http://security.ubuntu.com precise-security/restricted TranslationIndex [71 B]
Get:18 http://security.ubuntu.com precise-security/universe TranslationIndex [73 B]
Get:19 http://archive.ubuntu.com precise-updates/restricted amd64 Packages [8,215 B]
Get:20 http://archive.ubuntu.com precise-updates/universe amd64 Packages [149 kB]
Get:21 http://security.ubuntu.com precise-security/main Translation-en [86.5 kB]
Get:22 http://archive.ubuntu.com precise-updates/multiverse amd64 Packages [8,948 B]
Hit http://security.ubuntu.com precise-security/multiverse Translation-en
Get:23 http://archive.ubuntu.com precise-updates/main i386 Packages [410 kB]
Hit http://security.ubuntu.com precise-security/restricted Translation-en
Get:24 http://security.ubuntu.com precise-security/universe Translation-en [30.3 kB]
Get:25 http://archive.ubuntu.com precise-updates/restricted i386 Packages [8,218 B]
Get:26 http://archive.ubuntu.com precise-updates/universe i386 Packages [149 kB]
Get:27 http://archive.ubuntu.com precise-updates/multiverse i386 Packages [9,930 B]
Get:28 http://archive.ubuntu.com precise-updates/main TranslationIndex [3,564 B]
Get:29 http://archive.ubuntu.com precise-updates/multiverse TranslationIndex [2,605 B]
Get:30 http://archive.ubuntu.com precise-updates/restricted TranslationIndex [2,461 B]
Get:31 http://archive.ubuntu.com precise-updates/universe TranslationIndex [2,850 B]
Get:32 http://archive.ubuntu.com precise/main Translation-en [726 kB]
Get:33 http://archive.ubuntu.com precise/multiverse Translation-en [93.4 kB]
Get:34 http://archive.ubuntu.com precise/restricted Translation-en [2,395 B]
Get:35 http://archive.ubuntu.com precise/universe Translation-en [3,341 kB]
Get:36 http://archive.ubuntu.com precise-updates/main Translation-en [202 kB]
Get:37 http://archive.ubuntu.com precise-updates/multiverse Translation-en [5,606 B]
Get:38 http://archive.ubuntu.com precise-updates/restricted Translation-en [2,048 B]
Get:39 http://archive.ubuntu.com precise-updates/universe Translation-en [87.4 kB]
Ign http://archive.ubuntu.com precise/main Translation-en_US
Ign http://archive.ubuntu.com precise/multiverse Translation-en_US
Ign http://archive.ubuntu.com precise/restricted Translation-en_US
Fetched 6,308 kB in 8s (786 kB/s)
Reading package lists...

And here's the output for the one that works (i.e. after waiting a few seconds before doing the apt-get update):

Ign http://us-east-1.ec2.archive.ubuntu.com precise InRelease
Ign http://us-east-1.ec2.archive.ubuntu.com precise-updates InRelease
Get:1 http://us-east-1.ec2.archive.ubuntu.com precise Release.gpg [198 B]
Get:2 http://us-east-1.ec2.archive.ubuntu.com precise-updates Release.gpg [198 B]
Get:3 http://us-east-1.ec2.archive.ubuntu.com precise Release [49.6 kB]
Ign http://security.ubuntu.com precise-security InRelease
Get:4 http://us-east-1.ec2.archive.ubuntu.com precise-updates Release [49.6 kB]
Get:5 http://security.ubuntu.com precise-security Release.gpg [198 B]
Get:6 http://us-east-1.ec2.archive.ubuntu.com precise/main Sources [934 kB]
Get:7 http://security.ubuntu.com precise-security Release [49.6 kB]
Get:8 http://us-east-1.ec2.archive.ubuntu.com precise/universe Sources [5,019 kB]
Get:9 http://us-east-1.ec2.archive.ubuntu.com precise/main amd64 Packages [1,273 kB]
Get:10 http://security.ubuntu.com precise-security/main Sources [50.1 kB]
Get:11 http://us-east-1.ec2.archive.ubuntu.com precise/universe amd64 Packages [4,786 kB]
Get:12 http://security.ubuntu.com precise-security/universe Sources [14.5 kB]
Get:13 http://security.ubuntu.com precise-security/main amd64 Packages [177 kB]
Get:14 http://us-east-1.ec2.archive.ubuntu.com precise/main i386 Packages [1,274 kB]
Get:15 http://security.ubuntu.com precise-security/universe amd64 Packages [48.6 kB]
Get:16 http://us-east-1.ec2.archive.ubuntu.com precise/universe i386 Packages [4,796 kB]
Get:17 http://security.ubuntu.com precise-security/main i386 Packages [183 kB]
Get:18 http://us-east-1.ec2.archive.ubuntu.com precise/main TranslationIndex [3,706 B]
Get:19 http://us-east-1.ec2.archive.ubuntu.com precise/universe TranslationIndex [2,922 B]
Get:20 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main Sources [178 kB]
Get:21 http://security.ubuntu.com precise-security/universe i386 Packages [48.7 kB]
Get:22 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe Sources [60.2 kB]
Get:23 http://security.ubuntu.com precise-security/main TranslationIndex [73 B]
Get:24 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main amd64 Packages [405 kB]
Get:25 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe amd64 Packages [149 kB]
Get:26 http://security.ubuntu.com precise-security/universe TranslationIndex [73 B]
Get:27 http://security.ubuntu.com precise-security/main Translation-en [86.5 kB]
Get:28 http://security.ubuntu.com precise-security/universe Translation-en [30.3 kB]
Get:29 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main i386 Packages [410 kB]
Get:30 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe i386 Packages [149 kB]
Get:31 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main TranslationIndex [3,564 B]
Get:32 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe TranslationIndex [2,850 B]
Get:33 http://us-east-1.ec2.archive.ubuntu.com precise/main Translation-en [726 kB]
Get:34 http://us-east-1.ec2.archive.ubuntu.com precise/universe Translation-en [3,341 kB]
Get:35 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main Translation-en [202 kB]
Get:36 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe Translation-en [87.4 kB]
Fetched 24.6 MB in 12s (1,976 kB/s)
Reading package lists...

I don't know why the former is full of more Hits and Igns.

I'm new to RAID. Creating a new RAID1 array from blank, newly-partitioned 100GB disks with

sudo mdadm --create --verbose /dev/md0 --level=mirror --raid-devices=2 /dev/xvdf1 /dev/xvdg1

...looks to be taking about an hour to rebuild. My questions are:

1. are there any shortcuts that can be taken on this initial "resync", since there's not (AFAIK) any data I need copied between the empty disks

2. can I blithely format /dev/md0 and treat it like a normal disk while it's in the middle of its initial sync, as long as I'm okay with it being in a "degraded" state for a bit?

I know that I can do (edit: fixed this; I'm interested in gpg NOT openssh)

gpg --edit-key

...to change my passphrase for my key, but I'm not sure what this means.

If I'm encrypting data on box A and decrypting on box B (say with duplicity) do I have to change the passphrase on both ends? Will previous backups still work?

Is the passphrase just the key to a sort of encrypted wrapper around the key file?

Dumb question, but I don't want to screw this up. Thanks!

We have a web service running on Amazon EC2. Currently we have some live user data stored on a single disk (EBS). We are considering moving to a RAID0 setup (we don't have to be concerned about the increased failure rate).

If we do this migration, what is the quickest (to minimize site unavailability) way to reliably transfer the user data to the RAID array?

One idea I had was to take a recent snapshot of the data, copy it over to the new RAID array, then when the site goes down for maintenance use rsync to copy only the changed data over. I'm not sure if this would actually save time or ensure data integrity though.

I understand it displays the command with arguments, or when unavailable the command in square brackets.

But where do the names come from for processes such as passenger worker ruby instances, which show up as:

root@XXX:~# ps aux | grep Rails
webuser    2273  0.0  3.2  86968 57500 ?        S    17:38   0:00 Rails: /var/websites/app

...and how can I launch processes and set such a string?

I am interested in trying to package a software product consisting of a large linux software stack as a so-called "virtual appliance" to avoid a painful, painful installation procedure. My needs are probably similar to the author of this question and the answers here were helpful, but I have some specific questions.

The app I am trying to distribute consists of several rails apps and some other servers, daemons, and jobs that need to run via cron, etc. Currently it is running in a Ubuntu server.

1. What is the best way currently available to create and package a virtual machine like the above? I have been looking at JeOS which appears to be outdated, but has some tutorials written about the process. Also apparently the Open Virtualization Format is a good one for distributing virtual appliances, but I can't find any tutorials on creating such an image.

2. How would we go about distributing the image and updates in such a way that user data is preserved?

I would like to configure sudo such that users can run some specific commands without entering a password (for convenience) and can run all other commands by entering a password.

This is what I have, but this does not work; a password is always required:

Defaults    env_reset Defaults    timestamp_timeout = 1 

root    ALL=(ALL:ALL) ALL 

# Allow members of group sudo to execute any command 
%sudo   ALL=(ALL:ALL) NOPASSWD: /usr/sbin/pm-suspend, /usr/bin/apt-get, PASSWD: ALL 

#includedir /etc/sudoers.d

Note that this is a debian system which uses this adding users to the "sudo" group method.

Thanks.

I'm using an Alestic image which disables root SSH logins, but provides a user "ubuntu" with NOPASSWD sudo privileges. See here.

In the course of trying to add a new user to the sudoers file I inadvertantly created another line for the "ubuntu" user, this time without NOPASSWD. I have now apparently lost root access to this machine.

Is there some way to mount the EBS root volume on a different instance (fixing the sudoers file) and then re-launch the server?

Or am I totally screwed?

What specificaly is involved in an Elastic Load Balancer health check on an instance?

I know that it performas an HTTP(S) "ping"; does it just deem an instance "Unhealthy" if that HTTP request returns an error status $THRESHOLD number of times in a row?

Or does it take other factors into account, such as CPU usage?

I have a command that runs a disk snapshot (on EC2, freezing an XFS disk and running an EBS snapshot command), which is set to run on a regular schedule as a cron job. Ideally I would like to be able to have the command delayed for a period of time if the disk is being used heavily at the moment the task is scheduled to run.

I'm afraid that using nice/ionice might not have the proper effect, as I would like the script to run with high priority while it is running (i.e. wait for a good time, then finish fast).

Thanks.

UPDATE:

This is what I ended up going with. It checks /proc/diskstats and runs my job when the current IO activity hits 0, or we timeout. I'll probably have to tweak this when I look at what kind of IO activity our servers actually get in production:

#!/bin/bash

DEVICE=sdf

# we want to make a snapshot when IO in progresses reaches this:
LOW_THRESHOLD=0

TIMER=0
MAX_SEC_DELAY=120

# Get the number of IO operations in progress:
ioInProgress(){
    grep $DEVICE /proc/diskstats | awk '{print $12}'
}

# Wait for a good time to run snapshot, else timeout:

while [[ $TIMER -lt $MAX_SEC_DELAY && $(ioInProgress) -gt $LOW_THRESHOLD  ]]; do
    TIMER=`expr $TIMER + 1`
    sleep 0.5
done

# Recording delay required:
echo $TIMER
echo "Executing snapshot"
run-the-snapshot

I am using tiobench to test performance of an NFS mounted volume, and notice that Sequential Reads are much slower than Random Reads. This isn't the behavior when I run the same test on the disk mounted locally.

Here is a little graph, with NFS in red and sequential reads on the left:

I am exporting the volume on the server like this

/export *.internal(no_subtree_check,rw,no_root_squash)

and mounting with this:

mount -o hard,intr,async,noatime,nodiratime,noacl $NFS_SERVER:/export /nfs 

Additionally I am doing all this in amazon EC2, exporting an EBS volume with the XFS file system (redundant, I know).

I have tried using jumbo frames and various other mount options, but none seem to have much effect.

Thanks for any clues.

This may be a dumb question. I would like to use duplicity to make backups to Amazon S3 of directories, each of which contains a different user's data. Each directory could be written to at any time. So I have two questions:

Should I be concerned that a scheduled backup of a directory might occur in the middle of data being written to files in the directory, resulting in a corrupted backup?

And if that is a valid concern, how would I go about temporarily delaying an operation while IO was happening, to try to minimize that effect.

Thanks for the advice

I have two identical linux machines (identical images launched in amazon EC2) and I am trying to mount an exported directory over NFSv4. Here is what the mounted directory looks like on the client machine:

root@server:~# ls -l /websites/
drwxr-xr-x  6 4294967294 4294967294   92 2010-01-01 20:21 logs
drwxr-xr-x  2 4294967294 4294967294   20 2009-12-23 01:14 monit.d
...

I double checked to make sure that the UIDs were matching

Here is the mount command I run from the client

/sbin/mount.nfs4 $MASTER_DN:/ /websites -o rw,_netdev,async

And here is the /etc/exports entry on the server machine:

/websites 10.0.0.0/8(fsid=0,no_subtree_check,rw,no_root_squash)

I am working with EC2, and trying to set up a good system so that I can launch an instance, make changes to a few files (e.g. init scripts), and copy those changes over to the production server, with the ability to roll back changes.

I found this old thread very helpful and am thinking of putting the server's root under a git repository and adding the few files I'll be modifying under version control per this tutorial, but that seems a bit overkill, and I also want to make sure I'm not overlooking some better method for testing and copying over changes.

Thanks

After searching previous questions here, the general consensus seems to be if an instance that I own is assigned a private IP of 10.208.34.55, that only OTHER INSTANCES I OWN can reach it at that address. See:

How to encrypt traffic between two Amazon EC2 instances?

Is that correct? So I can treat all my instances as if they are on a LAN and authenticate and trust any machine coming from 10.XXX.XXX.XXX because I am sure I own it?

I just want to be sure. I'm finding that amazon seems to be rather more interested in waxing poetic about The Cloud and their 3-character abbreviations than actual providing clear technical documentation.

I will be running a server which will work with data on a local disk. I would also like to share the volume over the LAN with Samba or NFS to servers which will also be writing to data on the disc.

Should my main server act as both a Samba/NFS server and client (i.e. everything interacting with the disc through samba)? Or is it safe to mount and work with the mounted drive normally on the main server, while at the same time other servers are writing to data through Samba/NFS?

Also any opinions on choice of network file server are welcomed. Thanks!

I am helping in setting up a web service in which user data will be stored in sqlite databases on a server running perhaps Samba, and one or more web servers will read and write to those databases on the backend. I would like advice as to which network file system to use in an all-linux environment, as well as any other thoughts