Questions[sar](server)

I'm trying to narrow down the list of suspects of web servers that perform moderately well most of the time with occasional bouts of poor performance. I'm analyzing the data collected and summarized by sar. I've noticed a few things, one of which is high number of tasks in the run queue.

10:15:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
10:25:01 AM         2       150      0.05      0.05      0.06         0
10:35:01 AM         4       149      0.08      0.12      0.09         0
10:45:01 AM         6       150      0.13      0.19      0.15         0
10:55:01 AM         1       150      0.08      0.10      0.13         0
11:05:01 AM         4       150      0.20      0.35      0.23         0
11:15:01 AM         3       149      0.02      0.09      0.15         0
11:25:01 AM         7       149      0.04      0.05      0.11         0
11:35:01 AM         4       150      0.14      0.15      0.13         0
11:45:01 AM         6       150      0.27      0.18      0.16         0
11:55:01 AM         5       150      0.08      0.10      0.13         0
12:05:01 PM         3       149      0.35      0.40      0.26         0
12:15:01 PM        19       155      0.02      0.10      0.16         1
12:25:01 PM         2       150      0.00      0.07      0.12         0
12:35:02 PM         3       151      0.58      0.24      0.17         0
12:45:01 PM         8       150      0.02      0.13      0.15         0
12:55:01 PM         6       149      0.81      0.29      0.18         0
01:05:01 PM         3       148      0.00      0.09      0.13         0
01:15:01 PM         7       149      0.00      0.04      0.11         0

I believe these are 10 minute averages.

Is this an indicator that the web server is not performing as fast as it could if the average run queue length was lower?

I have a CentOS-5 server that has very little load. A typical output of "sar" command shows like this:

02:20:01 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
02:30:01 PM       all      0.63      0.00      1.23      0.31      0.00     97.84
02:40:01 PM       all      0.92      0.32      1.34      0.45      0.00     96.97
02:50:01 PM       all      0.76      0.00      0.93      0.77      0.00     97.54
03:00:01 PM       all      0.41      0.00      0.72      0.15      0.00     98.72
03:10:01 PM       all      0.78      0.32      1.49      1.83      0.00     95.58
03:20:01 PM       all      0.27      0.00      0.50      0.62      0.00     98.61

However, I am observing that every night at 4:10AM, the CPU usage is very high. This happens almost every night except few days in a month. The "sar" command shows something like this:

03:00:01 AM       all      0.10      0.00      0.10      0.11      0.00     99.68
03:10:01 AM       all      0.08      0.32      0.18      0.21      0.00     99.21
03:20:01 AM       all      0.03      0.00      0.03      0.19      0.00     99.76
03:30:01 AM       all      0.01      0.00      0.02      0.06      0.00     99.91
03:40:02 AM       all      0.75      0.32      3.21     15.43      0.00     80.28
03:50:01 AM       all      0.95      0.00      3.43      9.25      0.00     86.36
04:00:01 AM       all      0.69      0.00      3.79      4.86      0.00     90.66
04:10:01 AM       all     22.38      0.54      4.47     30.92      0.00     41.69
04:20:01 AM       all      2.99      0.02      4.58      4.95      0.00     87.46
04:30:01 AM       all      0.34      0.00      0.61     13.22      0.00     85.83
04:40:01 AM       all      0.05      0.32      0.17      0.52      0.00     98.95
04:50:01 AM       all      0.22      0.00      0.36      0.11      0.00     99.32
05:00:01 AM       all      0.23      0.00      0.36      0.12      0.00     99.28

How can I determine what is causing this peak in CPU usage and iowait etc. I examined /var/log/messages but could not determine. There is no other cron job or other things set to run at that time. Since this is showing under %user, I can only make out that it is some user process. How can I exactly determine the cause?

The sar manpage says :

pgpgin/s - Total number of kilobytes the system paged in from disk per second.

pgpgout/s - Total number of kilobytes the system paged out to disk per second.

fault/s - Number of page faults (major + minor) made by the system per second. This is not a count of page faults that generate I/O, because some page faults can be resolved without I/O.

majflt/s - Number of major faults the system has made per second, those which have required loading a memory page from disk

Does this measure swap activity? What paging does it refer to? Getting data from disk is considered paging?

Also, I've read that large and constant values for majflt/s is not good. why?

Example :

Sar -B

12:00:08 AM  pgpgin/s pgpgout/s   fault/s  majflt/s
12:10:05 AM    207.55   2522.76   5109.80      0.01
12:20:07 AM    303.83    274.64   4446.52      0.00
12:30:05 AM     53.85    251.81   4183.98      0.00
12:40:05 AM     43.19    234.05   4181.53      0.00
12:50:06 AM     88.89    265.46   4311.81      0.00
01:00:09 AM     64.60    232.72   4239.05      0.00
01:10:07 AM     69.71    216.89   4523.03      0.00
01:20:06 AM     81.37    250.02   4359.93      0.00
01:30:06 AM     79.77    246.28   4291.49      0.00
01:40:02 AM     42.89    227.22   4319.88      0.02
01:50:06 AM    214.46    441.33   4760.78      0.00
[...]

The load on my server is very high, even though there doesn't seem to be much disk activity and the CPU is idle. Using sar, I can see the run queue is getting full, but is there any way to see what's in the run queue?

When I run sar -d I receive the error message:

Requested activities not available in file /var/log/sysstat/sa24

The file does exist, and I can run sar with other flags, but can't see these particular metrics. Is there something I need to enable in sysstat to collect what I need to see this report?

Server is Ubuntu 9.04.