cmckendry's questions

Attempting to locate/create an EC2 AMI that meets my current needs. HVM is preferred over PV, because the newer and larger EC2 instance types only support HVM (we're targeting r3.4xlarge) and a partitionless "whole drive" image is preferred over one with a real partition table because it's nice to be able to dynamically enlarge the file system without having to mess with the partition table on the boot drive.

Is this even possible, or does the HVM boot system fundamentally require an actual partition table? So far all of my attempts to create a working HVM AMI using a whole drive image have failed to boot with no visible error.

Running a LAMP site with a moderate amount of traffic, but which is prone to occasional spikes. Trying to diagnose a bottleneck that causes slowness during periods of higher traffic, I've been looking at the output of mod_status and I'm seeing an alarming disparity between the CPU usage reported there and the CPU usage showing up in all our other monitoring tools, as well as in top and mpstat.

Here's the output of mpstat, which is typical of the server under normal load (it's supposed to be overpowered for what we're doing, which makes the slowdowns all the more frustrating):

Linux 2.6.18-274.7.1.el5 (xxxxxxxxxx)   10/25/2012

03:58:14 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
03:58:14 PM  all    2.02    0.01    0.16    0.41    0.01    0.07    0.00   97.32   3398.80
03:58:14 PM    0    0.14    0.01    0.08    0.01    0.00    0.00    0.00   99.76    999.98
03:58:14 PM    1    0.93    0.01    0.13    0.12    0.00    0.03    0.00   98.78     58.25
03:58:14 PM    2    0.79    0.01    0.09    0.07    0.00    0.03    0.00   99.02     57.08
03:58:14 PM    3    1.44    0.01    0.17    0.57    0.00    0.03    0.00   97.78     33.10
03:58:14 PM    4    0.75    0.01    0.07    0.06    0.00    0.03    0.00   99.08     53.83
03:58:14 PM    5    0.36    0.01    0.04    0.09    0.00    0.01    0.00   99.49      8.92
03:58:14 PM    6    0.85    0.01    0.09    0.19    0.00    0.03    0.00   98.82     48.18
03:58:14 PM    7    1.34    0.00    0.13    0.54    0.00    0.03    0.00   97.96     36.87
03:58:14 PM    8    0.15    0.01    0.03    0.01    0.00    0.00    0.00   99.80      0.14
03:58:14 PM    9    0.94    0.00    0.07    0.08    0.00    0.03    0.00   98.87     51.87
03:58:14 PM   10    1.18    0.01    0.16    0.06    0.00    0.03    0.00   98.56     53.50
03:58:14 PM   11    8.35    0.01    0.58    1.55    0.02    0.30    0.00   89.20    375.46
03:58:14 PM   12    1.08    0.01    0.09    0.06    0.00    0.03    0.00   98.72     56.58
03:58:14 PM   13    1.14    0.00    0.10    0.28    0.03    0.13    0.00   98.32    907.31
03:58:14 PM   14    4.73    0.01    0.28    1.08    0.01    0.14    0.00   93.75    198.19
03:58:14 PM   15    8.15    0.01    0.48    1.76    0.02    0.29    0.00   89.30    459.54

And here, at nearly the same time, is apache's mod_status output:

Current Time: Thursday, 25-Oct-2012 15:57:55 EDT
Restart Time: Thursday, 25-Oct-2012 15:40:09 EDT
Parent Server Generation: 3
Server uptime: 17 minutes 46 seconds
Total accesses: 8606 - Total Traffic: 283.4 MB
CPU Usage: u322.08 s32.5 cu0 cs0 - 33.3% CPU load
8.07 requests/sec - 272.3 kB/second - 33.7 kB/request
20 requests currently being processed, 492 idle workers 

I don't expect these two to line up exactly, but this seems like a pretty significant difference. I am at a loss as to how apache could be arriving at this number, and documentation appears to be pretty sparse.

Is there some way that apache could be getting capped at some arbitrary level of CPU usage, and mod_status is reflecting this limit? I inherited this server and was not present for the initial setup, so it's entirely possible that some arcane setting has been set to a pathological value for reasons unknown. It would explain a lot, but I don't see anything relevant in the configuration. If that's not the case, is this being calculated in some non-obvious way wherein that number actually makes sense, or is one source or the other just displaying incorrect information?

Any insight would be greatly appreciated.

I'm running a relatively low-traffic site that experiences a large spike in visitors once a week following a site update. During this spike, site performance is extremely poor compared to the rest of the week. The actually load on the servers remains very low, reliably under 10% CPU and under 30% RAM (the hardware should be complete overkill for what we're actually doing), but for some reason Apache seems to be unable to cope with the quantity of requests. We are running apache 2.2.3 on RHEL 5.7, kernel 2.6.18-274.7.1.el5, x86_64.

Attempting to reproduce this behavior during off hours with ab, I'm finding a major drop off in performance when exceeding approximately 256 users. Running the test with the smallest possible use case I could come up with (static text file being retrieved, 223 bytes total) performance is consistently normal with 245 simultaneous requests:

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       15   25   5.8     24      37
Processing:    15   65  22.9     76      96
Waiting:       15   64  23.0     76      96
Total:         30   90  27.4    100     125

Percentage of the requests served within a certain time (ms)
  50%    100
  66%    108
  75%    111
  80%    113
  90%    118
  95%    120
  98%    122
  99%    123
 100%    125 (longest request)

But as soon as I ratchet up to 265 simultaneous requests, a subset of them start taking an absurd amount of time to complete:

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       13  195 692.6     26    3028
Processing:    15   65  21.3     72     100
Waiting:       15   65  21.3     71      99
Total:         32  260 681.7    101    3058

Percentage of the requests served within a certain time (ms)
  50%    101
  66%    108
  75%    112
  80%    116
  90%    121
  95%   3028
  98%   3040
  99%   3044
 100%   3058 (longest request)

These results are very consistent across multiple runs. Since there is other traffic going to that box, I'm not sure precisely where the hard cutoff would be, if there is one, but it seems to be suspiciously close to 256.

Naturally, I assumed that this was being caused by the thread limit in prefork, so I went ahead and adjusted the configuration to double the number of available threads and to prevent the pool of threads from growing and shrinking unneccessarily:

<IfModule prefork.c>
StartServers     512
MinSpareServers  512
MaxSpareServers  512
ServerLimit      512
MaxClients       512
MaxRequestsPerChild  5000
</IfModule>

mod_status confirms that I am now running with 512 available threads

8 requests currently being processed, 504 idle workers

However, attempting 265 simultaneous requests still yields almost identical results to before

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       25  211 714.7     31    3034
Processing:    17   94  28.6    103     138
Waiting:       17   93  28.5    103     138
Total:         57  306 700.8    138    3071

Percentage of the requests served within a certain time (ms)
  50%    138
  66%    145
  75%    150
  80%    161
  90%    167
  95%   3066
  98%   3068
  99%   3068
 100%   3071 (longest request)

After scouring the documentation (and Stack Exchange) I'm at a loss for further configuration settings to attempt to address this bottleneck. Is there something that I'm missing? Should I start looking for answers outside of apache? Has anyone else seen this behavior? Any help would be greatly appreciated.

EDIT:

As per Ladadadada's advice, I ran strace against apache. I tried with -tt and -T a few times and couldn't find anything out of the ordinary. I then tried running strace -c against all currently running apache processes, and got this:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 22.09    0.317836           5     62128      4833 open
 19.91    0.286388           4     65374      1896 lstat
 13.06    0.187854           0    407433           pread
 10.70    0.153862           6     27076           semop
  7.88    0.113343           3     38598           poll
  6.86    0.098694           1    100954     14380 read

(...abdridged)

If I'm reading this right (and bear with me, as I don't use strace very often) none of the system calls can account for the amount of time that these requests are taking. It almost looks like the bottleneck is occurring before the requests even get to the worker threads.

EDIT 2:

As several people suggested, I ran the test again on the web server itself (previously the test was run from a neutral internet location). The results were surprising:

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   11   6.6     12      21
Processing:     5  247 971.0     10    4204
Waiting:        3  245 971.3      7    4204
Total:         16  259 973.3     21    4225

Percentage of the requests served within a certain time (ms)
  50%     21
  66%     23
  75%     24
  80%     24
  90%     26
  95%   4225
  98%   4225
  99%   4225
 100%   4225 (longest request)

The bottom-line time is similar to the internet-based test, but appears to consistently be a bit worse when run locally. More interestingly, the profile has changed dramatically. Whereas before the bulk of the long-running requests' time was spent in "connect" now the bottleneck appears to be in either processing or waiting. I'm left to suspect that this may actually be a separate issue which was previously being masked by network limitations.

Running the test again from another machine on the same local network as the Apache host, I'm seeing much more reasonable results:

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    2   0.8      2       4
Processing:    13  118  99.8    205     222
Waiting:       13  118  99.7    204     222
Total:         15  121  99.7    207     225

Percentage of the requests served within a certain time (ms)
  50%    207
  66%    219
  75%    220
  80%    221
  90%    222
  95%    224
  98%    224
  99%    225
 100%    225 (longest request)

These two tests together raise a number of questions, but separately from that, there is now a compelling case to be made for some kind of severe network bottleneck happening under a certain amount of load. I think that next steps will be investigating the network layer separately.

I'm looking into deploying a small (one or two host servers) non-production virtualization initiative for our company. We need it to be high performance and host shared resources for a number of users. I'm moderately familiar with both VMWare ESXi and Citrix XenServer, and know that we would be happy with either. Where I'm hitting a snag is the administration side.

From everything I'm reading, neither product currently supports any OS other than MS Windows for using their remote admin tools. Our company has been running quite happily for over a decade without deploying any Windows machines, and this does not seem like a. Good enough reason to start now. Even with some janky workaround in place to allow some degree of administration from a Linux or OS X machine, every solution I have found requires Windows for at least one component or one step in the process. I don't want to set up a Windows VM or partition just for this purpose. This would be a major expense to introduce an unwanted element into our environment, and I would very much like to avoid it.

Do I have any options for this? I'm not going to hold my breath waiting for Citrix or VMWare to get around to supporting other operating systems, so I'm pretty open to any alternatives. Do I need to roll my own solution here, and just forego the advantages of having a hypervisor OS with remote administration tools? Or is there some way of managing these systems with third-party tools that I'm not yet aware of? Any insight would be a big help.

One of the machines on my network (Mint 12 workstation, up to date) periodically gets into a bad state wherein all outgoing SSH connections fall back to password authentication with the error "agent admitted failure to sign using the key" instead of using the key-based authentication which has been configured.

Once it's in this state it will fail 100% of the time for all outgoing connections. Incoming key-based authentication appears to connect normally. I have tried deleting and regenerating the keypair and redistributing the public key, but the error persists.

A reboot will temporarily resolve the error, but it comes back again in a few days. Doesn't seem to coincide with any particular event/workflow, but I might be missing something.

Has anyone else seen this?