pjmorse's questions

I have a Spree site running the following stack:

• Nginx 1.0.8
• Passenger 3.0.9
• Ruby 1.9.2-p290
• Rack 1.3.6
• Rails 3.1.4
• Spree 0.70.5

I recently upgraded from Spree 0.70.3, which also brought a Deface upgrade from 0.7.x to 0.8.0. Since then things have been very unstable.

Recently we've seen some CPU-hogging processes which drive load up on the server and grind the whole thing to a stop. They're Rack processes and it looks like Passenger is starting them; they're owned by the site-runner user, an unprivileged user who owns the application code. (Passenger automatically runs the site code as the user who owns it.) If I restart Nginx and kill the runaway processes, it helps for a while, but eventually similar processes return and bog things down again.

ETA: I'm looking now at passenger-status and passenger-memory-stats which suggest these are Passenger's application processes. If it's running away or hanging, there must be an issue with my app.

What's my best option for figuring out where this is hanging?

I'm trying to chase a bug on a Rails site and I've noticed that there are recurring Ruby processes which are hogging CPU and won't go away. I'd like to suss out what's starting them and how to either stop them or give them the resources they need to complete and get their work done.

Background: The server is Ubuntu 10.04 LTS. Ruby is 1.9.2-p290. The site is Rails 3.1.4, and Passenger 3.0.9 is on Nginx 1.0.8.

The site code is deployed and owned by a non-wheel user named site-runner, so Passenger processes should be running as that user. That user has no crontab file.

This is what makes me suspicious:

top - 13:41:05 up 73 days, 20:26,  2 users,  load average: 2.11, 2.06, 2.28
Mem:    508272k total,   295660k used,   212612k free,    12608k buffers
Swap:  1048572k total,    32020k used,  1016552k free,    42580k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND 
26911 site-run  20   0  227m 116m 2656 R  101 23.4  18:08.55 ruby          
26919 site-run  20   0  227m 116m 2656 R  100 23.4  18:08.50 ruby  
[...]

I wanted to confirm this wasn't Passenger, so I used pstree:

$ pstree -apu
init,1
  |-PassengerWatchd,17903
  |   |-PassengerHelper,17908
  |   |   |-ruby,17912                                                ...
  |   |   |   |-{ruby},17938
  |   |   |   `-{ruby},26888
  |   |   |-{PassengerHelpe},17913
[Collapsed a bunch of Passenger processes]
  |   |   `-{PassengerHelpe},17926
  |   |-PassengerLoggin,17915,nobody
  |   |   `-{PassengerLoggi},17928
  |   |-{PassengerWatch},17907
  |   |-{PassengerWatch},17929
  |   `-{PassengerWatch},17930
  |-cron,2331
  |-getty,7533 -8 38400 tty1
  |-master,2536
  |   |-pickup,26864,postfix -l -t fifo -u -c
  |   |-qmgr,2543,postfix -l -t fifo -u
  |   `-tlsmgr,19889,postfix -l -t unix -u -c
  |-mysqld,23916,mysql
  |   |-{mysqld},23922
[collapsed a bunch of mysql processes]
  |   `-{mysqld},15541
  |-nginx,17931
  |   `-nginx,17932,site-runner      
  |-rsyslogd,2297,syslog -c4
  |   |-{rsyslogd},2303
  |   `-{rsyslogd},2304
  |-ruby,26911,site-runner                                      ...
  |   `-{ruby},26913
  |-ruby,26919,site-runner                                      ...
  |   `-{ruby},26921
  |-sshd,2329
  |   `-sshd,27099
  |       `-sshd,27110,parker
  |           `-bash,27111
  |               `-pstree,27218 -apu
  |-udevd,2108 --daemon
  |   |-udevd,2139 --daemon
  |   `-udevd,2142 --daemon
  `-upstart-udev-br,2066 --daemon

...and there are those ruby processes, 26911 and 26919, not children of Nginx or Passenger.

I've killed the processes, but they restart.

How can I figure out what these processes are doing and either urge them to completion or make them go away and stay away?

We've been running git over SSH on our Mac OS X server. (It happens to be a Mini.) Problem is, it's frightfully slow. We often see speeds under 100KiB/s for cloning repositories from this server, and similar speeds cloning repositories from elsewhere to this server. The issue is not the server or its network connection; we've run speed tests and seen 1.5 to 2MB/s download speeds and up to 7MB/s upload. scp transfers of large files are similarly quick.

Our theory is that git is moving a ton of tiny files, and this is causing filesystem overhead which is slowing the transfer.

• Is this a reasonable theory?
• Is there anything we can do to improve git performance on this box, or should we be looking for alternative hosting for our git repositories?

I'm pulling a bunch of company git repositories from an older host to our company git server. I started by just git clone-ing the old repos into the repositories directory on the new server. Soon we're going to point DNS for the old host to the new server.

My question is this: should I make the new repositories bare repos before this switch happens, or can they function as they are (normal, non-bare repositories with .git/ directories)? I'm guessing the former, but I'd like someone who actually knows to confirm this.

For a set of VPSes operated by my company, I'd like to get a list of domain names hosted by each. The list of names has grown in an ad-hoc way for a few years. Is there a simple way to query the Apache and/or Nginx configurations to see what domains the server answers to on port 80?

It is safe to assume (a) that I have sudo on these systems, and (b) they aren't using wildcards, although a solution which acknowledged that a wildcard was in use would be useful too if it also provided a list of explicit domains.

I have a specific URL pattern on a site which needs to be redirected to the HTTPS version. This is a Django site; Nginx checks each URL in memcached, and if it doesn't find a cached version it proxies the request to Apache/mod_python for Django to render the page.

The relevant configuration block is

    rewrite ^/certificate              https://mysite.com/certificate ;
    rewrite ^/([a-zA-Z]{2})/certificate   https://mysite.com/certificate ;

...and it doesn't appear to be working at all.

Nginx is:

$ nginx -V
nginx version: nginx/0.7.65
built by gcc 4.2.4 (Ubuntu 4.2.4-1ubuntu4)
TLS SNI support disabled
configure arguments: --prefix=/usr/local/nginx --pid-path=/var/run/nginx.pid --with-http_gzip_static_module --with-http_ssl_module

How can I figure out if the problem is my patterns not matching, or a more obscure configuration problem?

(The site is localized to three languages, and the localization is in the URL string, e.g. /US/news/, /DE/about, etc. It tracks localization in the session as well, defaulting to US, so if you just requested /news Django will rewrite to /US/news unless the user has a cookie indicating they're using a different localization. Django handles this, though, not Nginx.)

We have an Ubuntu 10.4 VPS serving a Rails site which often shows pretty high load, but doesn't have high CPU or memory numbers. Reading a lot of other questions here on Server Fault suggests to me that this is an I/O issue (i.e. there are processes which are stuck in I/O wait state and therefore driving up load). I'm trying to track down those processes, but not having much luck. I'd appreciate help with (a) ways to identify the guilty processes, and/or (b) confirmation that I'm asking the right question.

Here's a snapshot of top:

top - 18:28:49 up 5 days,  3:07,  2 users,  load average: 1.79, 1.83, 1.73
Tasks:  82 total,   1 running,  81 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.3%sy,  0.0%ni, 99.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.1%st
Mem:   1794980k total,  1780384k used,    14596k free,    13356k buffers
Swap:   524284k total,     3116k used,   521168k free,  1012272k cached

Notice low swap, CPUs mostly idle; that's why I think we're I/O bound instead of memory or CPU bound.

Here's iostat (I've obfuscated the server name):

$ iostat -x 1 3
Linux 2.6.35.2-xenU (our.server.com)     03/25/11        _x86_64_        (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.75    0.19    0.50    0.31    0.01   97.24

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
xvdap1            0.01    11.52    2.19    3.18   145.12   117.55    48.97     0.08   15.60   1.67   0.90
xvdap9            0.01     0.01    0.00    0.00     0.10     0.14    62.62     0.00   13.20   6.09   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    0.00    0.00    0.00  100.00

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
xvdap9            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    0.00    0.00    0.00  100.00

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
xvdap9            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

iotop won't run on this box:

$ iotop
Could not run iotop as some of the requirements are not met:
- Linux >= 2.6.20 with I/O accounting support (CONFIG_TASKSTATS, CONFIG_TASK_DELAY_ACCT, CONFIG_TASK_IO_ACCOUNTING): Not found
- Python >= 2.5 or Python 2.4 with the ctypes module: Found

ps seldom finds any processes in the D state:

$ sudo ps -eo pid,user,state,cmd | awk '$3 ~ /D/ { print $0 }'
  976 root     D [kjournald]
$ sudo ps -eo pid,user,state,cmd | awk '$3 ~ /D/ { print $0 }'
$ sudo ps -eo pid,user,state,cmd | awk '$3 ~ /D/ { print $0 }'
$ 

What's my next troubleshooting step?

ETA: I ran vmstat:

$ vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0   3116 509372  22880 773232    0    0    18    15   24   14  2  0 97  0

That wa value of 0 makes me wonder if I/O is really the problem.

Also, yes, I know load in the 1.x range isn't really a problem - but this app has a history of ramping up load until it chokes, and if I can track the source while it still has a low fever I might spare a fatality (to torture a metaphor).

I'm adding an SSL domain to a host. When I try to restart Nginx, it protests:

Restarting nginx: [emerg]: the "ssl" parameter requires ngx_http_ssl_module in /etc/nginx/sites-enabled/my_site_conf:16

This is frustrating because I actually rebuilt Nginx and Passenger specifically in order to make ngx_http_ssl_module available; it wasn't in the running instance, so I took the opportunity to build with Nginx 0.8.53 and Passenger 3.0.1.

The relevant configuration block is

server {
        listen  0.0.0.0:443 default ssl;
        server_name  our-site.com;

        ssl_certificate  /etc/nginx/ssl/our-site.crt;
        ssl_certificate_key  /etc/nginx/ssl/our-site.key;

        ssl_session_timeout  5m;

        ssl_protocols  SSLv2 SSLv3 TLSv1;
        ssl_ciphers  ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
        ssl_prefer_server_ciphers   on;

...etc. etc. (obviously I've obscured the host name, don't need this question coming up in searches for that domain.)

• How can I be sure the running Nginx instance has the SSL module active?
• Am I missing something obvious here?

We have a colocated Mac OS X 10.6 server, and I need to create a new user on the box. (The existing users were created by me when the box was on my desk.) I'm able to connect to the machine using SSH and also using the Server Admin utility. However, user management appears to require the Server Preferences utility. When I try to connect with Server Preferences, I get this error:

Unable to set node credentials for /LDAPv3/127.0.0.1 with the record name [FDQN]$.

There may be a problem with the Open Directory service.

(N.B. I've obscured the actual server name; where it says [FDQN] above, the actual error has the complete name of the server.)

In the server logs, which I can see using Server Admin, I see

Dec 10 11:31:05 eiffel servermgrd[63]: servermgr_accounts: got error 5203 trying to auth to local LDAP node

(N.B. eiffel is the hostname.) I think this is the relevant line; Jabber is flapping for some reason (different problem) and smothering the logs with its attempts to start up.

I'm not familiar with Open Directory, and I don't really want to have to be; I just want to administer my server, in particular to create a new unix user on the box so they can connect via SSH. Thanks for any pointers.

(ETA: My research suggests that VNC would solve this problem. Unfortunately, I need access to Server Preferences to allow VNC access, so that's a Chicken and egg problem. (See what I did there?) Setting up an SSH tunnel sounds like a great idea, but I need a bit more fine-grained advice (which ports on each end?) to put that into practice.)

ETA, 12/16: I'm able to add a new user with the Workgroup Manager utility, which is part of the server tools package. (I suspect I have it on my laptop and not my desktop, which is why I didn't find it sooner.) Even though the urgency is gone, I'm leaving the question open because I still can't connect with Server Preferences.

We have an OS X server that is co-located; that is, we have access by SSH and via the Server Admin utilities, but not via Remote Desktop.

Several months later, I'd like to install git on the server. However, it appears that Developer Tools were never installed; make and gcc aren't there. So I can't build from source. Also, without remote desktop, I can mount a .dmg file, but I can't actually run the installer.

Can anyone suggest a strategy to either run the installer from the command line, or get the relevant developer tools installed (again, these are normally a .dmg) so I can build from source?