Arkaaito's questions

I'm setting up a CentOS server with the sendmail MTA to send DKIM-signed mail (sending domain is @brighter.do). I have succeeded in routing mails through the dkim milter; however, they arrive at my GMail account with dkim=fail authentication results and no further information. How do I begin to debug this (besides Googling 'debug failed dkim' and other such related terms)?

I used the config described in these instructions. The only additional config changes I made were adding my app user to /etc/mail/trusted-users, adding EXTRA_FLAGS=-R to /etc/sysconfig/dkim-milter, and switching Canonicalization to relaxed/relaxed (which didn't appear to change anything). I've verified that the key is 2048 bits using ssh-keygen -l -f app1. (It seems that a too-small key can cause GMail to reject a valid DKIM signature but that doesn't appear to be my problem.)

The contents of my relevant DNS records are:

APP1._DOMAINKEY.BRIGHTER.DO. TXT k=rsa; p=AAAAB3NzaC1yc2EAAAADAQABAAABAQC3pJ4UJW/KBQ2D6N/6kl37yqJ0F4NcKPGApyHw4wl2zohdOPp8rELvQnRgvmQUMu3hrgicD9W9LbnGx/CzakZAA4RcJk9kI51v+Y8L5j3lZURFC1ZIXoRFgfafyo31XN3rc+V0hNMXUGcxVI09oYtyS+2AuC9cULP4Nu030I3yYFd2NOwmKPY57PU3ybwGKEvuWsB/9PyWC6KVlULlkg7TB
APP1._DOMAINKEY.BRIGHTER.DO. TXT CwbMnGyavwIeoJpNlb1fINdDGWDAJvfTTpMGvIkQAehknbgBqL4IgciWQ/2xw6bMhma7MRJHzZsd7JfbNramQIpsxX6hZUkZja6HpoFJzBi1vbnLcM2n8Xhat/A1Q/F
_DOMAINKEY.BRIGHTER.DO. TXT o=~ [email protected]

The headers I'm getting are:

Delivered-To: [email protected]
Received: by 10.140.42.166 with SMTP id c35csp248278qga;
        Wed, 23 Mar 2016 14:10:12 -0700 (PDT)
X-Received: by 10.98.72.213 with SMTP id q82mr7347661pfi.164.1458767412258;
        Wed, 23 Mar 2016 14:10:12 -0700 (PDT)
Return-Path: <[email protected]>
Received: from DUALSTACK.PROD-API-478862527.US-WEST-2.ELB.AMAZONAWS.COM (outbound.brighter.do. [54.201.111.245])
        by mx.google.com with ESMTPS id m22si6801929pfi.43.2016.03.23.14.10.12
        for <[email protected]>
        (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
        Wed, 23 Mar 2016 14:10:12 -0700 (PDT)
Received-SPF: pass (google.com: domain of [email protected] designates 54.201.111.245 as permitted sender) client-ip=54.201.111.245;
Authentication-Results: mx.google.com;
       dkim=fail [email protected];
       spf=pass (google.com: domain of [email protected] designates 54.201.111.245 as permitted sender) [email protected]
Received: from DUALSTACK.PROD-API-478862527.US-WEST-2.ELB.AMAZONAWS.COM (localhost [127.0.0.1])
    by DUALSTACK.PROD-API-478862527.US-WEST-2.ELB.AMAZONAWS.COM (8.14.4/8.14.4) with ESMTP id u2NLAB8k007870
    for <[email protected]>; Wed, 23 Mar 2016 21:10:11 GMT
X-DKIM: Sendmail DKIM Filter v2.8.3 DUALSTACK.PROD-API-478862527.US-WEST-2.ELB.AMAZONAWS.COM u2NLAB8k007870
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=brighter.do; s=app1;
    t=1458767411; bh=HiluaVoCYKZyFY1h3gE73EqhCFuKBJzE8SqwhrLX5/c=;
    h=Date:Message-Id:To:Subject:MIME-Version:From;
    b=RfNNbBaAUNX+y3cdSqb+NkgC8GHa0wd/vV4LC72DQ8jbSWIHfqxZD6Qi3xGtKVnyU
     2j9FDAtI7X1B7dsPuFIw9F5m+1YoFuV+/3vCQ/zsXxCoExwml7DrxnYuWI0e5MeKma
     3K4T+R/tpNgKYVSU00RNCorLsvyia/fD8+wFTY4ZyoYOTZ4tK6gwcO4loPERiPAAOL
     HI11YagXgreCk3efJXanF8Df9ALLmTZMjMLXHHIHnSsypzhtEXYmua+EWQEZzIiVis
     paAmh9w8sRfeFww4PraRN7Caxznm51ZUIecdST29xRL276LsEgb5Nsy6TIEJyOov/N
     7AilYKXwuotHg==
Received: (from ec2-user@localhost)
    by DUALSTACK.PROD-API-478862527.US-WEST-2.ELB.AMAZONAWS.COM (8.14.4/8.14.4/Submit) id u2NLABrt007869;
    Wed, 23 Mar 2016 21:10:11 GMT
Date: Wed, 23 Mar 2016 21:10:11 GMT
Message-Id: <201603232110.u2NLABrt007869@DUALSTACK.PROD-API-478862527.US-WEST-2.ELB.AMAZONAWS.COM>
To: [email protected]
Subject: Test message
X-PHP-Originating-Script: 500:PushComponent.php
MIME-Version: 1.0
From: Brighter <[email protected]>

Note: since I know almost nothing about e-mail admin, it's possible that I'm omitting some key details here. In that event please help me improve the question by mentioning what additional info would be useful - e.g., other config files to include.

I have a number of AWS EC2 instances which are serving as the backend of a production app. Many of them predate the availability of general SSD storage; however, some of the more recent ones do use SSD for their root volume.

I'd like to convert all the ones that still use magnetic storage to take advantage of SSD storage as well, but I'd prefer not to destroy and recreate any instances which are already on SSDs. How can I determine the type of the storage associated with a particular instance's root volume?

I need to stop crond at the command line. (I'm writing a script which needs to stop it, do some work, and then start it again. I'd rather not remove and replace the crontab file because it's more prone to error.)

However, when I run service crond stop or sudo /etc/init.d/crond stop, it stops but is started again less than a minute later.

[root@prod-sphinx1 ~]# service crond stop;date
Stopping crond:                                            [  OK  ]
Wed Oct 15 05:48:50 UTC 2014
[root@prod-sphinx1 ~]# ps uefxxx | grep crond
root     11891  0.0  0.0  61192   788 pts/0    S+   05:49   0:00          \_ grep crond HOSTNAME=sphinx01.us-east-1.zoomingo.com TERM=xterm-256color SHELL=/bin/bash HISTSIZE=1000 SSH_CLIENT=50.46.219.220 51260 22 SSH_TTY=/dev/pts/0 USER=root LS_COLORS= EC2_HOME=/home/ec2 MAIL=/var/spool/mail/root PATH=/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/home/ec2/bin:/home/ec2/bin:/root/bin INPUTRC=/etc/inputrc PWD=/root JAVA_HOME=/usr/java/default LANG=en_US.UTF-8 PS1=[\u@prod-sphinx1 \W]\$  SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass SHLVL=1 HOME=/root LOGNAME=root CVS_RSH=ssh SSH_CONNECTION=50.46.219.220 51260 10.146.200.111 22 PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/local/lib/pkgconfig LESSOPEN=|/usr/bin/lesspipe.sh %s G_BROKEN_FILENAMES=1 _=/bin/grep
root     11875  0.0  0.0  17832  1044 ?        Ss   05:49   0:00 crond MONIT_DATE=Wed, 15 Oct 2014 05:49:21 +0000 MONIT_HOST=sphinx01.us-east-1.zoomingo.com PATH=/sbin:/usr/sbin:/bin:/usr/bin PWD=/ MONIT_PROCESS_PID=0 MONIT_EVENT=Started MONIT_PROCESS_MEMORY=0 SHLVL=2 MONIT_PROCESS_CPU_PERCENT=0 MONIT_SERVICE=crond MONIT_PROCESS_CHILDREN=0 MONIT_DESCRIPTION=Started _=/usr/sbin/crond

ps -l says the parent process ID is init (not surprising):

[root@prod-sphinx1 ~]# ps -l -p11875
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
1 S     0 11875     1  0  78   0 -  4458 -      ?        00:00:00 crond

And pstree says:

[root@prod-sphinx1 ~]# pstree -p 11875
crond(11875)───crond(11905)───sh(11907)───index-minutely.(11908)───indexer(11910)

Which gives me the processes it's running, but not the process that started it.

Unfortunately, pstree was the best my Google-fu could dig up. What can I do to figure out how this thing is getting started?

Edit: the box is running CentOS 5.6:

[root@prod-sphinx1 ~]# cat /etc/issue
CentOS release 5.6 (Final)
Kernel \r on an \m

We have a Sphinx install (2.0.3) running on a cluster of 3 EC2 instances (currently m3.large).

Currently we have workers = threads and max_children = 30 in our Sphinx config (same on each box). We are periodically getting the dreaded "temporary searchd error: server maxed out, retry in a second". Our instances are hovering around 5% CPU utilization. Some example top output:

top - 19:51:56 up 22:15,  1 user,  load average: 0.08, 0.04, 0.01
Tasks:  82 total,   2 running,  80 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.0%us,  0.0%sy,  0.0%ni, 98.5%id,  0.3%wa,  0.0%hi,  0.0%si,  0.2%st
Mem:   7872040k total,  2911920k used,  4960120k free,   245168k buffers
Swap:        0k total,        0k used,        0k free,  2190992k cached

All the Sphinx doc seems to say about setting max_children is that it is "useful to control server load". While searching I found a forum post indicating that setting it either too high or too low can cause "server maxed out" - I presume the former is because the individual queries are starved - but had no further tips on choosing the right level. (I can't find the link to this post again to save my life. Sorry.)

Two related questions:

• Am I right in thinking the low CPU suggests max_children could/should be higher than 30?
• How can I find the optimal number (i.e., the max number of children which [usually] does not lead to query slowdown)? I'm not entirely sure what kind of info Sphinx logs beyond query.log. Is there a tool I can use to determine whether query slowdown is occurring (due to too many parallel queries), and if not, are queries CPU-bound or memory-bound (or should I be looking at some other value entirely)?

I have a virtual development server running Ubuntu Server set up with Apache and running under VirtualBox on a Mac (host OS is OSX 10.8.1). I have a bridged adapter configured in an attempt to make it visible from my test devices (iPhone and Android) inside the network.

When I connect to the server from the "real" machine that's running it, everything's fine - webpages can be loaded, etc. However, connection attempts to this server from any of my test devices time out.

I tried setting promiscuous mode = allow all and restarting the guest OS, but it doesn't seem to have had any effect. I've also verified that the host box's firewalls are completely off (at the command line using 'sudo ipfw list' and in System Preferences > Security > Firewall).

Is there any additional setup I ought to have done in order to make this work? What additional troubleshooting steps am I missing?

(I suspect that this may already be answered somewhere, since it seems like it would be a common operation. But I can't find it, so...)

I am a relative AWS newbie. I have inherited a running Amazon EC2 instance, with various items (Apache, MySQL, Sphinx, ...) installed on it and a bunch of configuration. I'd like to turn it into an AMI that I can spin up other instances from.

I can't find any information on creating a custom AMI on Amazon's site - only the fact that you can, repeatedly referenced, as if to taunt me...

I believe this is not an EBS-backed instance, just an "ordinary" one. I do not know what AMI it was originally created from.

How would I create an AMI that I could use for spinning up other instances which will be identical except for the hostname?

Our memcache install recently started removing keys, and we're not sure why. Large groups of keys vanish at the same time.

Memcache reports that evictions are low to non-existent, and our app has no way to clear memcache (it can only delete specific keys). Even keys of which the app has no knowledge get deleted, so we're pretty convinced they're getting expired. However, our memcache configuration hasn't been touched in some time.

Has anyone debugged an issue like this before, and if so, are there any steps you'd recommend we take? How flexible is memcache's expiration policy - is it possible that we're suddenly running into a criterion based on (say) write frequency to a key?

I recently upgraded my desktop to Windows 7 and opened it up for ssh from other boxes (using OpenSSH as installed with Cygwin).

Everything works - however, there's a very, very long delay (sometimes up to 30 seconds) when I first attempt to make the connection (before it asks for the password), even from a wired connection. There's no such delay when I ssh remotely to my boxes at work.

I'll be very grateful for your insights on what I can do to debug this. I recognize there are a zillion small things that affect the speed of establishing an ssh connection, but this seems so drastically bad that there must be SOMETHING majorly wrong.

ssh -vvv output follows (warning: long) - one long pause is after it lists the identity files (identity file /Users/Arkaaito/...), and then there's another long pause after it lists the keys (key: /Users/Arkaaito...). I suppose it might be just because I don't have key-based authentication set up yet. (Ironically, I wanted to wait to add that until I had it working with password-based authentication, on the internal network.)

Arkas-MacBook-Pro:~ Arkaaito$ ssh -vvv [email protected]
OpenSSH_5.2p1, OpenSSL 0.9.8l 5 Nov 2009
debug1: Reading configuration data /etc/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.168.0.196 [192.168.0.196] port 22.
debug1: Connection established.
debug1: identity file /Users/Arkaaito/.ssh/identity type -1
debug1: identity file /Users/Arkaaito/.ssh/id_rsa type -1
debug1: identity file /Users/Arkaaito/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.1
debug1: match: OpenSSH_5.1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.2
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[email protected],hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[email protected],hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,[email protected],aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,[email protected],aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[email protected],hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[email protected],hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 127/256
debug2: bits set: 521/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: check_host_in_hostfile: filename /Users/Arkaaito/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 16
debug1: Host '192.168.0.196' is known and matches the RSA host key.
debug1: Found key in /Users/Arkaaito/.ssh/known_hosts:16
debug2: bits set: 522/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /Users/Arkaaito/.ssh/identity (0x0)
debug2: key: /Users/Arkaaito/.ssh/id_rsa (0x0)
debug2: key: /Users/Arkaaito/.ssh/id_dsa (0x0)
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug3: start over, passed a different list publickey,password,keyboard-interactive
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /Users/Arkaaito/.ssh/identity
debug3: no such identity: /Users/Arkaaito/.ssh/identity
debug1: Trying private key: /Users/Arkaaito/.ssh/id_rsa
debug3: no such identity: /Users/Arkaaito/.ssh/id_rsa
debug1: Trying private key: /Users/Arkaaito/.ssh/id_dsa
debug3: no such identity: /Users/Arkaaito/.ssh/id_dsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug3: userauth_kbdint: disable: no info_req_seen
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: 
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
[email protected]'s password: 

I need to reinstall Apache on my personal desktop. The machine now runs Windows 7 x64, so I'm thinking of installing the x64 version of Apache. However, it looks like this would lead to me needing to recompile several Apache modules (at a minimum) myself, which would probably be an annoyance. I'm also worried about what bugs might magically appear in the 64-bit version of things like APC.

I use my machine as (1) a personal SVN server, (2) a toy webserver [for when I'm debugging personal projects in PHP], (3) a toy webserver again, with mod_rewrite, mod_ssl, mod_proxy, mod_proxy_http [for when I occasionally use the machine to work on my day job], and (4) a standard development box [so I'd like Apache to not eat up too many extra cycles if it can be avoided].

Does anyone have a good feel for the performance pros and bugginess cons of going x64? Somewhat subjective, I know, but I'm hopeful that there will be a consensus among those with more experience than I.

Problem: we've grown to the point where we need a real DBA. Real DBAs are hard to find.

Possible solution: I know someone with extensive experience (currently working on MSSQL) who is looking for a new DBA job. He might be willing to consider working for a MySQL shop (I haven't asked).

Snag: I don't know to what extent MSSQL DBA skills map to MySQL DBA skills. I'm a developer, so I know enough about MySQL to develop apps which use it (including the basic performance-tuning, index selection, etc.), do schema design, and perform simple utility tasks (backup with mysqldump, scripting, etc.). I don't know anything about MSSQL. Nor do I actually know much about the boundaries of the DBA role.

Can anyone with more experience - perhaps as a DBA - weigh in on this? Are there enough similarities between MSSQL and MySQL that it's worth asking a MSSQL DBA if he'd be interested in applying for a MySQL position?

I have a desktop machine that is currently set up as an SVN server. Currently I have it set up to be accessible via http (using Apache), but only for the local network, as I wasn't confident in my handling of the security issues involved in opening it up.

My two options for handling that, as I see it, are SVN over SSH and setting up a VPN while maintaining the "local network only" configuration for SVN. Does anyone have a strong idea of the pros and cons of these two setups? In particular, I've never tried to set up a personal VPN before, so I don't know how much pain and expense is usually involved with OpenVPN and such.

[Aside: I don't know if this is actually more appropriate to SuperUser because it's a server for personal use - if it is, can someone please move it? Thanks!]