Home / server / Questions

Questions[vmstat](server)

Martin Hope
APZ
Asked: 2011-11-05 16:08:27 +0800 CST
  • 4

I am writing a shell script to check the vmstat si and so data at various time intervals

vmstat 1 sample output:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
0  0  45820 899252  86700 468520    0    0     0    60  127 5821 20  7 34  0
0  0  45820 899252  86704 468504    0    0     0    32   44  104  0  0 100  0

I want to use awk, sed to extract si and so into different variables for further usage. I Am new to awk, sed and am still struggling to find my way through. Can you show me how I can do this ?

vmstat awk
Martin Hope
futureal
Asked: 2011-02-04 16:34:38 +0800 CST
  • 13

I am looking to understand some server performance problems I am seeing with a (for us) heavily loaded web server. The environment is as follows:

  • Debian Lenny (all stable packages + patched to security updates)
  • Apache 2.2.9
  • PHP 5.2.6
  • Amazon EC2 large instance

The behavior we're seeing is that the web typically feels responsive, but with a slight delay to begin handling a request -- sometimes a fraction of a second, sometimes 2-3 seconds in our peak usage times. The actual load on the server is being reported as very high -- often 10.xx or 20.xx as reported by top. Further, running other things on the server during these times (even vi) is very slow, so the load is definitely up there. Oddly enough Apache remains very responsive, other than that initial delay.

We have Apache configured as follows, using prefork:

StartServers          5
MinSpareServers       5
MaxSpareServers      10
MaxClients          150
MaxRequestsPerChild   0

And KeepAlive as:

KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5

Looking at the server-status page, even at these times of heavy load we are rarely hitting the client cap, usually serving between 80-100 requests and many of those in the keepalive state. That tells me to rule out the initial request slowness as "waiting for a handler" but I may be wrong.

Amazon's CloudWatch monitoring tells me that even when our OS is reporting a load of > 15, our instance CPU utilization is between 75-80%.

Example output from top:

top - 15:47:06 up 31 days,  1:38,  8 users,  load average: 11.46, 7.10, 6.56
Tasks: 221 total,  28 running, 193 sleeping,   0 stopped,   0 zombie
Cpu(s): 66.9%us, 22.1%sy,  0.0%ni,  2.6%id,  3.1%wa,  0.0%hi,  0.7%si,  4.5%st
Mem:   7871900k total,  7850624k used,    21276k free,    68728k buffers
Swap:        0k total,        0k used,        0k free,  3750664k cached

The majority of the processes look like:

24720 www-data  15   0  202m  26m 4412 S    9  0.3   0:02.97 apache2                                                                       
24530 www-data  15   0  212m  35m 4544 S    7  0.5   0:03.05 apache2                                                                       
24846 www-data  15   0  209m  33m 4420 S    7  0.4   0:01.03 apache2                                                                       
24083 www-data  15   0  211m  35m 4484 S    7  0.5   0:07.14 apache2                                                                       
24615 www-data  15   0  212m  35m 4404 S    7  0.5   0:02.89 apache2

Example output from vmstat at the same time as the above:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 8  0      0 215084  68908 3774864    0    0   154   228    5    7 32 12 42  9
 6 21      0 198948  68936 3775740    0    0   676  2363 4022 1047 56 16  9 15
23  0      0 169460  68936 3776356    0    0   432  1372 3762  835 76 21  0  0
23  1      0 140412  68936 3776648    0    0   280     0 3157  827 70 25  0  0
20  1      0 115892  68936 3776792    0    0   188     8 2802  532 68 24  0  0
 6  1      0 133368  68936 3777780    0    0   752    71 3501  878 67 29  0  1
 0  1      0 146656  68944 3778064    0    0   308  2052 3312  850 38 17 19 24
 2  0      0 202104  68952 3778140    0    0    28    90 2617  700 44 13 33  5
 9  0      0 188960  68956 3778200    0    0     8     0 2226  475 59 17  6  2
 3  0      0 166364  68956 3778252    0    0     0    21 2288  386 65 19  1  0

And finally, output from Apache's server-status:

Server uptime: 31 days 2 hours 18 minutes 31 seconds
Total accesses: 60102946 - Total Traffic: 974.5 GB
CPU Usage: u209.62 s75.19 cu0 cs0 - .0106% CPU load
22.4 requests/sec - 380.3 kB/second - 17.0 kB/request
107 requests currently being processed, 6 idle workers

C.KKKW..KWWKKWKW.KKKCKK..KKK.KKKK.KK._WK.K.K.KKKKK.K.R.KK..C.C.K
K.C.K..WK_K..KKW_CK.WK..W.KKKWKCKCKW.W_KKKKK.KKWKKKW._KKK.CKK...
KK_KWKKKWKCKCWKK.KKKCK..........................................
................................................................

From my limited experience I draw the following conclusions/questions:

  • We may be allowing far too many KeepAlive requests

  • I do see some time spent waiting for IO in the vmstat although not consistently and not a lot (I think?) so I am not sure this is a big concern or not, I am less experienced with vmstat

  • Also in vmstat, I see in some iterations a number of processes waiting to be served, which is what I am attributing the initial page load delay on our web server to, possibly erroneously

  • We serve a mixture of static content (75% or higher) and script content, and the script content is often fairly processor intensive, so finding the right balance between the two is important; long term we want to move statics elsewhere to optimize both servers but our software is not ready for that today

I am happy to provide additional information if anybody has any ideas, the other note is that this is a high-availability production installation so I am wary of making tweak after tweak, and is why I haven't played with things like the KeepAlive value myself yet.

debian apache-2.2 amazon-ec2 vmstat performance-tuning
Martin Hope
grzaks
Asked: 2010-10-13 02:43:23 +0800 CST
  • 15

According to vmstat, my Linux server (2xCore2 Duo 2.5 GHz) is constantly doing around 20k context switches per second.

# vmstat 3
procs -----------memory----------  ---swap-- -----io----  -system-- ----cpu----
 r  b   swpd   free   buff  cache    si   so    bi    bo   in    cs us sy id wa
 2  0   7292 249472  82340 2291972    0    0     0     0    0     0  7 13 79  0
 0  0   7292 251808  82344 2291968    0    0     0   184   24 20090  1  1 99  0
 0  0   7292 251876  82344 2291968    0    0     0    83   17 20157  1  0 99  0
 0  0   7292 251876  82344 2291968    0    0     0    73   12 20116  1  0 99  0

... but uptime shows small load: load average: 0.01, 0.02, 0.01 and top doesn't show any process with high %CPU usage.

How do I find out what exactly is generating those context switches? Which process/thread?

I tried to analyze pidstat output:

# pidstat -w 10 1

12:39:13          PID   cswch/s nvcswch/s  Command
12:39:23            1      0.20      0.00  init
12:39:23            4      0.20      0.00  ksoftirqd/0
12:39:23            7      1.60      0.00  events/0
12:39:23            8      1.50      0.00  events/1
12:39:23           89      0.50      0.00  kblockd/0
12:39:23           90      0.30      0.00  kblockd/1
12:39:23          995      0.40      0.00  kirqd
12:39:23          997      0.60      0.00  kjournald
12:39:23         1146      0.20      0.00  svscan
12:39:23         2162      5.00      0.00  kjournald
12:39:23         2526      0.20      2.00  postgres
12:39:23         2530      1.00      0.30  postgres
12:39:23         2534      5.00      3.20  postgres
12:39:23         2536      1.40      1.70  postgres
12:39:23        12061     10.59      0.90  postgres
12:39:23        14442      1.50      2.20  postgres
12:39:23        15416      0.20      0.00  monitor
12:39:23        17289      0.10      0.00  syslogd
12:39:23        21776      0.40      0.30  postgres
12:39:23        23638      0.10      0.00  screen
12:39:23        25153      1.00      0.00  sshd
12:39:23        25185     86.61      0.00  daemon1
12:39:23        25190     12.19     35.86  postgres
12:39:23        25295      2.00      0.00  screen
12:39:23        25743      9.99      0.00  daemon2
12:39:23        25747      1.10      3.00  postgres
12:39:23        26968      5.09      0.80  postgres
12:39:23        26969      5.00      0.00  postgres
12:39:23        26970      1.10      0.20  postgres
12:39:23        26971     17.98      1.80  postgres
12:39:23        27607      0.90      0.40  postgres
12:39:23        29338      4.30      0.00  screen
12:39:23        31247      4.10     23.58  postgres
12:39:23        31249     82.92     34.77  postgres
12:39:23        31484      0.20      0.00  pdflush
12:39:23        32097      0.10      0.00  pidstat

Looks like some postgresql tasks are doing >10 context swiches per second, but it doesn't all sum up to 20k anyway.

Any idea how to dig a little deeper for an answer?

vmstat performance-monitoring kernel performance-tuning context-switch
Martin Hope
Mech
Asked: 2010-03-09 17:35:04 +0800 CST
  • 8

I am running into a scenario where I'm seeing a high server load (sometimes upwards of 20 or 30) and a very low CPU usage (98% idle). I'm wondering if these wait states are coming as part of an NFS filesystem connection. Here is what I see in VMStat

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  1      0 1298784      0      0    0    0    16     5    0    9  1  1 97  2  0
 0  1      0 1308016      0      0    0    0     0     0    0 3882  4  3 80 13  0
 0  1      0 1307960      0      0    0    0   120     0    0 2960  0  0 88 12  0
 0  1      0 1295868      0      0    0    0     4     0    0 4235  1  2 84 13  0
 6  0      0 1292740      0      0    0    0     0     0    0 5003  1  1 98  0  0
 4  0      0 1300860      0      0    0    0     0   120    0 11194  4  3 93  0  0
 4  1      0 1304576      0      0    0    0   240     0    0 11259  4  3 88  6  0
 3  1      0 1298952      0      0    0    0     0     0    0 9268  7  5 70 19  0
 3  1      0 1303740      0      0    0    0    88     8    0 8088  4  3 81 13  0
 5  0      0 1304052      0      0    0    0     0     0    0 6348  4  4 93  0  0
 0  0      0 1307952      0      0    0    0     0     0    0 7366  5  4 91  0  0
 0  0      0 1307744      0      0    0    0     0     0    0 3201  0  0 100  0  0
 4  0      0 1294644      0      0    0    0     0     0    0 5514  1  2 97  0  0
 3  0      0 1301272      0      0    0    0     0     0    0 11508  4  3 93  0  0
 3  0      0 1307788      0      0    0    0     0     0    0 11822  5  3 92  0  0

From what I can tell when the IO goes up the waits go up. Could NFS be the cause here or should I be worried about something else? This is a VPS box on a fiber channel SAN. I'd think the bottleneck wouldn't be the SAN. Comments?

linux centos nfs vmstat vps
Martin Hope
His
Asked: 2009-07-09 18:07:28 +0800 CST
  • 10

I need to get a memory usage in % from vmstat, but I don't quite understand some of the numbers relating to memory displayed in vmstat.

In particular, given an output like this

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0 139728  45396 199128 1236360    1    2     4   622    3    1 51  4 43  2  0

What is the amount of total memory of the system? Is it 'swpd' + 'free' + 'buff' + 'cache', which total to 1620612?

What is the free amount of memory of the system? Is it 45396, as displayed in the 'free' column?

The problem is they don't add up or even look close to what 'top' reported. The output of 'top' which I took around the same time as 'vmstat's above is:

[...]
Mem:   4059616k total,  3965504k used,    94112k free,   200020k buffers
[...]

So...I am assuming there must be hidden numbers/options in 'vmstat' that I didn't know of?

I need to use 'vmstat' for this purpose and not other commands as this is what the requirement stated.

So ideally, what I want to achieve from 'vmstat' is the % memory usage, as in:

<used_memory>/<total_memory> * 100%.

Thanks!

linux memory vmstat