Jake's questions

I have an instance in a private subnet and a security group that allows any inbound access. I can connect to it with Session Manager.

If I restrict inbound access to port 22 only I can no longer connect to it.

The documentation says "You can use this functionality to connect to managed nodes without opening inbound ports" and "You can close inbound ports on the node" but this doesn't seem to be true when I try it.

I'd like to restrict inbound traffic to a minimum but I can only connect if I leave it wide open. What inbound rules are needed to allow Session Manager connections?

I have a set of recipes that deploy python apps to AWS instances, they work well. I want to deploy two different apps with slight differences, like one needs async worker code installed and the other does not. How should I do this?

My options as, I see them:

1. Create two Apps in OpsWorks and set env vars for each that the recipes can use to decide whether/how to run certain tasks
2. Create file(s) in the repo detailing app-specific requirements
3. Add custom JSON th each deployment.

None of these options seems ideal to me, have I overlooked something? Has

How can I add an EC2 instance that I already have running to an Amazon OpsWorks stack?

I just want to be able to manually run recipes on it for now.

I'm trying to get this command to work as the user postgres (so I can ship wal files):

rsync -a /tmp/test postgres@server2:/tmp/test

But I get the error:

Permission denied (publickey).

I've run ssh-keygen eval `ssh-agent` and ssh-add as postgres user on server1. keygen created /var/lib/postgresql/.ssh/id_rsa and id_rsa.pub and I can see that it's sent by using ssh -vvv postgres@server2.

On server2 I've created /var/lib/postgresql/.ssh/authorized_keys put the contents of id_rsa.pub form server1 in it. It's owned by postgres user and group and chmod 600. The .ssh directory is also owned by postgres and chmod 700.

I can see from verbose sshd logging on server2 that Failed publickey for postgres...

postgres user on both servers: postgres:x:106:114:PostgreSQL administrator,,,:/var/lib/postgresql:/bin/bash

ssh -vvv postgres@server2

...
debug1: Found key in /var/lib/postgresql/.ssh/known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /var/lib/postgresql/.ssh/id_rsa (0x7f468e434000)
debug2: key: /var/lib/postgresql/.ssh/id_dsa ((nil))
debug2: key: /var/lib/postgresql/.ssh/id_ecdsa ((nil))
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /var/lib/postgresql/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug1: Trying private key: /var/lib/postgresql/.ssh/id_dsa
debug3: no such identity: /var/lib/postgresql/.ssh/id_dsa
debug1: Trying private key: /var/lib/postgresql/.ssh/id_ecdsa
debug3: no such identity: /var/lib/postgresql/.ssh/id_ecdsa
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).  

server2 sshd_config (commented lines removed)

Port 22
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
UsePrivilegeSeparation yes
KeyRegenerationInterval 3600
ServerKeyBits 768
SyslogFacility AUTH
LogLevel VERBOSE
LoginGraceTime 120
PermitRootLogin yes
StrictModes yes
RSAAuthentication yes
PubkeyAuthentication yes
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
PasswordAuthentication no
X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server
UsePAM yes

server2 auth log

Jan 16 03:54:21 ip-10-28-26-251 sshd[7972]: Set /proc/self/oom_score_adj to 0
Jan 16 03:54:21 ip-10-28-26-251 sshd[7972]: Connection from 10.28.123.97 port 49377
Jan 16 03:54:21 ip-10-28-26-251 sshd[7972]: Failed publickey for postgres from 10.28.123.97 port 49377 ssh2
Jan 16 03:54:21 ip-10-28-26-251 sshd[7972]: Connection closed by 10.28.123.97 [preauth]

What am I missing? I'm guessing that sshd isn't looking at my authorized_keys file on server2

From what I've seen online everyone seems to mount additional EBS drives at /mnt/some_name. However /mnt is the mount point of the instance's ephemeral storage and this gets lost when the instance is stopped.

Assuming I've added the right entry in /etc/fstab can I reboot or stop/start the instance and have the drive still mounted?

My thinking is that the mount point /mnt/some_name will be lost during the reboot so the drive will not be able to be mounted. Am I right? Where should I mount things then?

We are having a problem where our Postgres 9.0 server occasionally locks up and kills our webapp. Restarting Postgres fixes the problem.

Here's what I've been able to observe:

• First, usage of one CPU jumps to 100% for a few minutes
  • Disk operations drop to ~0 during this time
  • Database operations drop to 0 (blocks and tuples per sec)
  • Logs show during this time:
    • WARNING: worker took too long to start; cancelled
    • WARNING: worker took too long to start; cancelled
    • No Queries in logs (only those over 200ms are logged)
  • No unusually long-running queries logged before or during
• Then the second CPU jumps to 100%
  • The number of postgres processes jumps from the usual 8-10 to ~20
  • Matched by a spike in Postgres Blocks per second (about twice normal)
  • Logs show
    • LOG: could not accept SSL connection: EOF detected
    • Queries are running but slow
• Restarting postgres returns everything to normal

Setup:

Server: Amazon EC2 Large
Ubuntu 10.04.2 LTS
Postgres 9.0.3
Dedicated DB server

Does anyone have any idea what's causing this? Or any suggestions about what else I should be checking out?

A couple of processes returned by ps aux have an extra column. A monitoring tool I'm using is interpreting this as %CPU and reporting huge consumption.

Here's the truncated output:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
...
root       694  0.0  0.0      0     0 ?        S    Apr18   0:00 [kjournald]
root       728  0.0  0.0  49260  2544 ?        Ss   Apr18   0:00 /usr/sbin/sshd -D
syslog     732 51026191  0.0 195848 1736 ?     Sl   Apr18 57266230:36 rsyslogd -c4
102        736  0.0  0.0  23536   924 ?        Ss   Apr18   0:00 dbus-daemon --system --fork
root       775  0.0  0.0   6080   648 tty4     Ss+  Apr18   0:00 /sbin/getty -8 38400 tty4
root       779  0.0  0.0   6080   648 tty5     Ss+  Apr18   0:00 /sbin/getty -8 38400 tty5
root       783  0.0  0.0   6080   652 tty2     Ss+  Apr18   0:00 /sbin/getty -8 38400 tty2
root       784  0.0  0.0   6080   652 tty3     Ss+  Apr18   0:00 /sbin/getty -8 38400 tty3
root       786  0.0  0.0   6080   652 tty6     Ss+  Apr18   0:00 /sbin/getty -8 38400 tty6
daemon     791  0.0  0.0  18884   460 ?        Ss   Apr18   0:00 atd
root       792  0.0  0.0  21076  1020 ?        Ss   Apr18   0:00 cron
...      
ubuntu   21385  0.0  0.0  15256  1216 pts/0    R+   10:02   0:00 ps aux
postgres 22570  0.0  0.0  94864  6636 ?        S    04:34   0:16 /usr/lib/postgresql/9.0/bin/postgres -D /var/lib/postgresql/9.0/main -c config_file=/etc/postgresql/9.0/main/postgresql.conf
postgres 22572 124088579  0.3 94996 27080 ?    Ss   04:34 17179869:11 postgres: writer process                                                                                                    
postgres 22573  0.0  0.0  94864  1684 ?        Ss   04:34   0:01 postgres: wal writer process                                                                                                
postgres 22574  0.0  0.0  96000  3320 ?        Ss   04:34   0:01 postgres: autovacuum launcher process                                                                                       
postgres 22575  0.0  0.0  66608  1724 ?        Ss   04:34   0:06 postgres: stats collector process                                                                                           
...

Note postgres writer process and syslog have something between PID and %CPU, they are also showing values for huge TIME. The server is running smoothly and top reports CPU% and load average seem normal.

What is that and is there anything I can do about it?

EDIT

Ok, so somehow I managed to miss the fact that it isn't an extra number, just a but one pushing the columns along. Still, 51026191% seems a bit excessive.

I'd like to start using Amazon SES for all emails from our server. We have a few freelance designers with PHP hosting, some Django/Python web apps and also some system utilities which send emails.

So I'd like to have PHP's mail function, the command line mail command and our python apps all be able to use it, preferably without having to set them all up in their own way.

I think what I need is to have something like Postfix running on localhost and using SES for it's delivery but I don't know how to do that.

Amazon's docs state I need to setup my mail transfer agent (MTA) so that it invokes the ses-send-email.pl script. I have the script but I'm not sure how to achieve this.

Am I on the right track? If so, how can I configure Postfix to use that script?

I'm using Monit to keep an eye on my Amazon EC2 Micro instances, but I'm a little confused about the Load Average CPU metrics because of the Micro instances' ability to use up to 2ECU for short times. Monit is often reporting a Load Average (1 min) of 4 or 5 but I don't really know if this is actually high or not.

As I understand it my usage profile is well suited to the Micro instance although there are some spikes to 100% CPU usage and these roughly coincide with the Monit alerts.

Could someone explain what would be normal Load Average for a Micro instance and at what stage I should start to be concerned about it?

I've got quite a big VirtualHost definition which I don't want to duplicate just so the site will also run over HTTPS.

Here's what I want to do:

<VirtualHost *>
    ServerName example.com

    <If port=443>
        SSLEngine on
        SSLCertificateFile ...
        SSLCertificateKeyFile ...
        SSLCertificateChainFile ...
    </If>

    (other config)

</VirtualHost>

Is there some way to do this?
Am I missing some other method of not duplicating the config?

I'm trying to set up sftp-server but the client is getting an error, Connection closed by server with exitcode 1

/var/log/auth.log (below) doesn't help much, how can I find out what the error is?

I'm running Ubuntu 10.04.1 LTS

sshd[27236]: Accepted password for theuser from (my ip) port 13547 ssh2
sshd[27236]: pam_unix(sshd:session): session opened for user theuser by (uid=0)
sshd[27300]: subsystem request for sftp
sshd[27236]: pam_unix(sshd:session): session closed for user theuser

Update: I've been prodding this for a while now, I've got the sftp command on another server giving me a more useful error.

Request for subsystem 'sftp' failed on channel 0
Couldn't read packet: Connection reset by peer

Everything I've found on the net suggests this id a problem with sftp-server but when I remove the chroot from sshd config I can access the system. I assume this means sftp-server is accessible and set up correctly.

The other day I used apt-get install python-pyexiv2 on my ubuntu server, but it seems to have given me an old version. It's not compatible with the code I wrote in my local development environment so I'd like to update it.

I downloaded the latest tar.gz from the website, extracted it and ran scons as per the readme. But it will not build, I get the error src/exiv2wrapper.hpp:32:29: error: exiv2/preview.hpp: No such file or directory

I've also user apt-get to install libboost-python-dev and libexiv2-dev

Can anyone help me on this?