Josh Nankin's questions

I'm running a dedicated m3.medium EC2 instance. How is it possible that my top says:

Tasks:  84 total,   1 running,  83 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.3%us,  1.5%sy,  0.0%ni, 90.8%id,  0.1%wa,  0.0%hi,  0.0%si,  3.2%st
Mem:   3840444k total,  3780152k used,    60292k free,   208224k buffers
Swap:        0k total,        0k used,        0k free,  3040292k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                                                              
22440 newrelic  20   0  109m 3892 2704 S  4.0  0.1   1:35.23 nrsysmond                                                                                                                                                                                                            
 4715 www-data  20   0 83568 4928 2044 S  2.0  0.1  10:10.62 nginx                                                                                                                                                                                                                
    1 root      20   0 24308 2304 1348 S  0.0  0.1   0:01.39 init                                                                                                                                                                                                                 
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd                                                                                                                                                                                                             
    3 root      20   0     0    0    0 S  0.0  0.0   2:26.52 ksoftirqd/0                                                                                                                                                                                                          
    4 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kworker/0:0                                                                                                                                                                                                          
    5 root      20   0     0    0    0 S  0.0  0.0   0:00.03 kworker/u:0                                                                                                                                                                                                          
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0                                                                                                                                                                                                          
    7 root      RT   0     0    0    0 S  0.0  0.0   0:05.75 watchdog/0                                                                                                                                                                                                           
    8 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 cpuset                                                                                                                                                                                                               
    9 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 khelper                                                                                                                                                                                                              
   10 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kdevtmpfs                                                                                                                                                                                                            
   11 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 netns                                                                                                                                                                                                                
   12 root      20   0     0    0    0 S  0.0  0.0   0:00.00 xenwatch                                                                                                                                                                                                             
   13 root      20   0     0    0    0 S  0.0  0.0   0:00.07 xenbus                                                                                                                                                                                                               
   14 root      20   0     0    0    0 S  0.0  0.0   0:01.80 sync_supers                                                                                                                                                                                                          
   15 root      20   0     0    0    0 S  0.0  0.0   0:00.73 bdi-default                                                                                                                                                                                                          
   16 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 kintegrityd                                                                                                                                                                                                          
   17 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 kblockd                                                                                                                                                                                                              
   18 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 ata_sff                                                                                                                                                                                                              
   19 root      20   0     0    0    0 S  0.0  0.0   0:00.00 khubd                                                                                                                                                                                                                
   20 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 md                                                                                                                                                                                                                   
   21 root      20   0     0    0    0 S  0.0  0.0   0:36.58 kworker/0:1                                                                                                                                                                                                          
   23 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kworker/u:1                                                                                                                                                                                                          
   24 root      20   0     0    0    0 S  0.0  0.0   0:00.16 khungtaskd                                                                                                                                                                                                           
   25 root      20   0     0    0    0 S  0.0  0.0   0:04.74 kswapd0                                                                                                                                                                                                              
   26 root      25   5     0    0    0 S  0.0  0.0   0:00.00 ksmd                                                                                                                                                                                                                 
   27 root      20   0     0    0    0 S  0.0  0.0   0:00.00 fsnotify_mark                                                                                                                                                                                                        
   28 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ecryptfs-kthrea                                                                                                                                                                                                      
   29 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 crypto                                                                                                                                                                                                               
   37 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 kthrotld                                                                                                                                                                                                             
   38 root      20   0     0    0    0 S  0.0  0.0   0:00.00 khvcd                                                                                                                                                                                                                
   57 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 devfreq_wq                                                                                                                                                                                                           
  180 root      20   0     0    0    0 S  0.0  0.0   0:30.38 jbd2/xvda1-8                                                                                                                                                                                                         
  181 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 ext4-dio-unwrit                                                                                                                                                                                                      
  280 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kjournald                                                                                                                                                                                                            
  295 root      20   0 17232  636  448 S  0.0  0.0   0:00.07 upstart-udev-br                                                                                                                                                                                                      
  297 root      20   0 21468 1268  792 S  0.0  0.0   0:00.03 udevd                                                                                                                                                                                                                
  346 root      20   0 21464  768  360 S  0.0  0.0   0:00.00 udevd                                                                                                                                                                                                                
  347 root      20   0 21464  792  380 S  0.0  0.0   0:00.00 udevd                                                                                                                                                                                                                
  444 root      20   0 15188  396  204 S  0.0  0.0   0:00.02 upstart-socket-                                                                                                                                                                                                      
  458 root      20   0 19200 1064  788 S  0.0  0.0   0:01.73 rpcbind                                                                                                                                                                                                              
  501 root      20   0  7264 1032  532 S  0.0  0.0   0:01.00 dhclient3                                                                                                                                                                                                            
  581 root      20   0     0    0    0 S  0.0  0.0   0:17.81 flush-202:1                                                                                                                                                                                                          
  787 root      20   0 50036 2932 2320 S  0.0  0.1   0:00.02 sshd                                                                                                                                                                                                                 
  789 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 rpciod                                                                                                                                                                                                               
  802 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 nfsiod                                                                                                                                                                                                               
  811 statd     20   0 21504 1348  892 S  0.0  0.0   0:00.00 rpc.statd                                                                                                                                                                                                            
  816 syslog    20   0  247m 1524 1108 S  0.0  0.0   2:37.38 rsyslogd                                                                                                                                                                                                             
  825 messageb  20   0 23816  924  644 S  0.0  0.0   0:00.03 dbus-daemon                                                                                                                                                                                                          
  830 root      20   0 29780  716  448 S  0.0  0.0   0:00.00 rpc.idmapd                                                                                                                                                                                                           
  889 root      20   0 14504  980  812 S  0.0  0.0   0:00.01 getty                                                                                                                                                                                                                
  895 root      20   0 14504  968  812 S  0.0  0.0   0:00.00 getty                                                                                                                                                                                                                
  900 root      20   0 14504  964  812 S  0.0  0.0   0:00.00 getty                                                                                                                                                                                                                
  901 root      20   0 14504  976  812 S  0.0  0.0   0:00.00 getty                                                                                                                                                                                                                
  903 root      20   0 14504  972  812 S  0.0  0.0   0:00.00 getty                                                                                                                                                                                                                
  909 root      20   0 19112 1052  804 S  0.0  0.0   0:02.46 cron                                                                                                                                                                                                                 
  910 daemon    20   0 16908  376  216 S  0.0  0.0   0:00.03 atd

If I'm the only one on the box, who's stealing my CPU?

I have an alias for SSHing in to a specific machine. The alias uses a hostname like so:

ssh [email protected]

Example.com is a CNAME. For some reason, I can dig the hostname after I change the CNAME to point somewhere else, and it resolves correctly, but on the same machine, ssh thinks the CNAME is still the old location. How do I get ssh to resolve the new CNAME correctly?

I'm on Ubuntu 12.04.

I have the following in a chef recipe:

remote_file '/tmp/zint-2.4.2.zip' do
  source 'https://s3.amazonaws.com/mybucket/zint-2.4.2.zip'
  checksum '4dc28641ef79957e9d3f0904a5d56472ef1d66f8529e3deda25537d890a6f8f3'
  mode '0664'
  action :create_if_missing
end

On Ubuntu 14.04 with ruby1.9.1 this is throwing "Encoding::UndefinedConversionError: "\xA2" from ASCII-8BIT to UTF-8" when I run chef-solo on boot. If I run chef-solo manually after start up, the cookbook completes successfully.

This seems related to https://github.com/yevgenko/cookbook-php-fpm/issues/36 as I added the following code to my php5-fpm recipe:

if RUBY_VERSION =~ /1.9/
  Encoding.default_external = Encoding::UTF_8
  Encoding.default_internal = Encoding::UTF_8
end

I have this in my chef recipe:

service 'php5-fpm' do
  supports :restart => true
  action [ :enable, :start ]
end

However my recipe now barfs in Ubuntu 14.04, as "/etc/init.d/php5-fpm restart" does not work on Ubuntu 14.04 (it returns with 1 and there is no message).

How do I get my chef recipe to restart and manage the php5-fpm service?

On apt-get on a 11.10 machine i'm getting:

W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/oneiric-security/main/source/Sources  404  Not Found [IP: 91.189.91.14 80] 
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/oneiric-security/universe/source/Sources  404  Not Found [IP: 91.189.91.14 80]
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/oneiric-security/multiverse/source/Sources  404  Not Found [IP: 91.189.91.14 80]
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/oneiric-security/main/binary-amd64/Packages  404  Not Found [IP: 91.189.91.14 80]
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/oneiric-security/universe/binary-amd64/Packages  404  Not Found [IP: 91.189.91.14 80]
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/oneiric-security/multiverse/binary-amd64/Packages  404  Not Found [IP: 91.189.91.14 80]
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/oneiric-security/main/binary-i386/Packages  404  Not Found [IP: 91.189.91.14 80]
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/oneiric-security/universe/binary-i386/Packages  404  Not Found [IP: 91.189.91.14 80]
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/oneiric-security/multiverse/binary-i386/Packages  404  Not Found [IP: 91.189.91.14 80]

Indeed, security.ubuntu.com has packages for other releases, but not 11.10. What happened to them?

In ubuntu, I'd like to get the status of a print job using the command line. When I send a job to the printer using lp, i can see the status of the print job in the taskbar applet. For instance, I see a job that says "Pending" when I send a job to a printer that is off.

However, when I run lpstat -t, all I see is the following:

scheduler is running
system default destination: Brother_MFC-7340
device for BRFAX: usb:/dev/usb/lp0
device for Brother_MFC-7340: usb://Brother/MFC-7340?serial=000G9N299107
device for MFC-7340: usb://Brother/MFC-7340?serial=000G9N299107
device for MFC7340: usb://Brother/MFC-7340?serial=000G9N299107
BRFAX accepting requests since Tue 03 Dec 2013 10:42:29 AM CST
Brother_MFC-7340 accepting requests since Fri 06 Dec 2013 04:11:28 PM CST
MFC-7340 accepting requests since Fri 06 Dec 2013 04:11:28 PM CST
MFC7340 accepting requests since Fri 06 Dec 2013 04:11:28 PM CST
printer BRFAX is idle.  enabled since Tue 03 Dec 2013 10:42:29 AM CST
    File "/usr/lib/cups/filter/brfaxfilter" has insecure permissions (0100777/uid=0/gid=0).
printer Brother_MFC-7340 disabled since Fri 06 Dec 2013 04:11:28 PM CST -
    Unplugged or turned off
printer MFC-7340 disabled since Fri 06 Dec 2013 04:11:28 PM CST -
    Unplugged or turned off
printer MFC7340 disabled since Fri 06 Dec 2013 04:11:28 PM CST -
    Unplugged or turned off
Brother_MFC-7340-14     jnankin          77824   Tue 10 Dec 2013 06:46:52 PM CST
Brother_MFC-7340-15     jnankin         305152   Tue 10 Dec 2013 07:30:13 PM CST

How do I get lpstat to print "pending" or equivalent for this job? Likewise, how do I see if a job completed or failed if I have the job string (in this case Brother_MFC-7340-15)

I have a vagrant box running ubuntu 11.10 and a PHP script that uses swiftmailer to send email. My mail provider is SES, although I've replicated the issue with Mailgun and sendgrid as well.

My issue is that when I attempt to send mail on the vagrantbox I get "Connection to tcp://email-smtp.us-east-1.amazonaws.com:465 Timed Out". However when I run the exact same script from the host machine, it works fine.

Here's ifconfig on my vagrant box:

eth0      Link encap:Ethernet  HWaddr 08:00:27:6e:cd:85  
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe6e:cd85/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:272963 errors:0 dropped:0 overruns:0 frame:0
          TX packets:138823 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:375217285 (375.2 MB)  TX bytes:8245622 (8.2 MB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:50:d1:0c  
          inet addr:192.168.33.10  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe50:d10c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:468 (468.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:181 errors:0 dropped:0 overruns:0 frame:0
          TX packets:181 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:18416 (18.4 KB)  TX bytes:18416 (18.4 KB)

eth1 is a host only network interface with a static ip.

UPDATE: This appears to be a problem with swiftmailer. I had different copies of it between my host and guest boxes. I updated my host box, and now I'm getting the same issue where the connection times out.

Seems impossible to find any info on configuration files for Gluster's most recent version (3.2.x). I'd like to use configuration files as opposed to the CLI because I'd like to automate configuration with chef.

Is this possible? When I googled this, all I found was this, which was pretty useless: http://community.gluster.org/q/does-glusterfs-support-configuration-files/

How does one do this?

Trying to get a Symfony2 project up and running with a vagrant VM. I've seen a zillion articles on this, but getting the cache directory permissioned correctly is proving to be excruciatingly painful.

In my Vagrantfile:

config.vm.share_folder("src", "/var/www", "../www")

My VM's fstab file:

proc            /proc           proc    nodev,noexec,nosuid 0       0
/dev/mapper/ubuntu--1110-root /               ext4    acl,errors=remount-ro 0       1
# /boot was on /dev/sda1 during installation
UUID=4ed7eaaf-6f42-48ba-b5ed-f0c1df9add38 /boot           ext2    defaults        0       2
/dev/mapper/ubuntu--1110-swap_1 none            swap    sw              0       0

The acl package has been installed on the VM.

When I run the following in /var/www as per the Symfony2 install directions:

 sudo setfacl -R -m u:apache:rwx -m u:username:rwx app/cache app/logs

I get:

Operation not supported

I can run the command on any other directory or file on the FS (outside of the shared directory) and the command works. What gives?

I want to run:

knife role create blah

I use chef-solo and don't need knife to connect to a server. I just would like to run these commands for helping me prepopulate the files quickly. Everytime I run of these commands it attempts to connect to a server.

This does not happen for

knife cookbook create blah

Double-ewe tee eff.

Update:

I've noticed this is the case for databags and environments as well. I've read elsewhere that knife is primarily intended for use with the chef-server API, but it seems ridiculous that something like this can't be done. It just needs to save the darn file.

I have several processes running on servers with different roles that are monitored by monit. For example, say I have an Apache server and a DB server with MySQL. I have several monitrc files that can be dropped in to monit's configuration directory to monitor these processes. What's the most elegant way in chef to make sure the appropriate files get into the monit configuration directory, based on the role?

Options I see:

1. use a template resource in a monit cookbook that checks the node's roles to see if it should be executing
2. use a template resource in the apache and mysql cookbooks (but check to see if monit is installed somehow)
3. Create separate recipies in the monit cookbook for each role, and add them to the run list for the role (monit::mysql, monit::apache, etc.)

What I'd really like to do is put the code for ensuring the files are moved in the role's file, but I don't think that's possible. Ideas?

If i have a a software system that has a lot of dependencies on other libraries, like libssl, or libxml etc, should I be creating separate cookbooks for each of these libraries (with a recipe that really only has like 2 lines) or do people do several 'package' blocks in a cookbook to ensure the dependencies are there (and not split out these package blocks to their external cookbooks)?

I'm experimenting with chef and moving my server configuration over to it. One hurdle I'm encountering is how to deal with packages that must be compiled and installed manually (configure, make, make install) but take forever to compile. I don't want to hinder a server start up time just because a dependency takes forever to build.

Specific usecase: wkhtmltopdf. Wkhtmltopdf with full features requires a custom/patched qt that takes FOREVER to compile.

Here are what I see as my options: 1) Compile the package using chef on each startup (Con: takes forever to start a server) 2) Compile the packages once, tar and them up. Then just download the package and install it. (Con: won't work for machines with different OSes/hardware) 3) Compile the package once and just keep it in a base image and out of chef (Chef: package is not in chef)

Any ideas?

Recently I've been experimenting with various cloud management tools like RightScale, Scalr, custom scripts for managing a variety of servers, each hosting several roles (app, db, load balancer, job queues, etc).

The one thing I find lacking in most solutions is a way to do rolling deployments, i.e. running deployments sequentially across a number of servers with the same role. For instance, I dont want to build all of my webservers at the same time, as that will almost definitely result in some down time or 500s for my customers. I'd rather have one or two servers build at a time, while other servers are still available to handle requests.

The other alternative is obviously to launch new servers that automatically update themselves on boot, but this isn't as cost effective, and most likely requires more time for the build to complete (it's faster to build on an existing server than to launch a new server and kill old ones).

We've all heard of the big companies having the famous "push to build" button (companies like Twilio, Etsy, etc.) but it seems that they all have custom implementations of this. I'm not talking about a simple ssh-loop, clusterssh, or even an mcollective - I preferably want something with a nice simple interface that allows me to specify something like a RightScript or a Scalr script to run on a set of servers with a specific role, and it builds them sequentially.

Does any one know of easy ways to get this done, or is this a candidate for a new open source project?

Title says it all.

Does anyone know of a way to replicate beanstalkd such that if a beanstalk server went down, others slaves could take over?

Here's one approach I thought of: I could make beanstalk write its binlog (with the -b) to a shared location, and then somehow have a secondary/backup server start beanstalkd if the primary fails.

There must be a better way though.

1) Do you have to use a sliding rail when mounting a server in a rack cabinet, or Can you just screw the machine directly into the cabinet?

2) What size are the screws that go in the front and back of the server?

Possible Duplicate:
Finding the Public IP address in a shell script

I'm currently doing this in a script:

EXTERNAL_IP=`curl -s http://whatismyip.org`

There's got to be a built in linux command or something for this, no?

I'm getting the famous "SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed" problem when attempting to curl a domain that has a rapidssl certificate installed.

I'm running Ubuntu 10.10 and I have openssl installed with a huge /etc/ssl/certs directory. I've tried using the --cacert and --capath arguments.

Nothing seems to work with this specific domain, even though when I visit the site with my web browser it authenticates the certificate.

However, when I curl https://google.com, it has no problem.

What am I doing wrong?

I'd like to monitor NFS mounts and the NFS server process using Monit.

On the server, I'd need a PID file, but I can't seem to find a way of getting that created with existing configuration files. Is there a way to do this, or has anyone monitored the server in a different way (checking if port 53 is active, etc).

On clients, I was thinking of making Monit simply look for a specific file in an NFS mount, and if it's accessible, all is well. Problem is, if the NFS server does go down, file requests usually hang (perhaps even indefinitely, not sure). How would one get around this issue with monit?

Any configuration examples would be greatly appreciated!

Here are my MPM constraints:

<IfModule mpm_prefork_module>
    StartServers         10
    MinSpareServers      10
    MaxSpareServers      10
    MaxClients           10
    MaxRequestsPerChild  2000
</IfModule>

However despite this, I have over 20 apache processes running currently, and in the past hour or two there have been as many as 40-50. Shouldn't the MaxClient and MaxSpareServers keep the number of processes under control (i.e. about 10)?

Is there something I'm missing?