Home / server / Questions / 148576
Accepted
Milan Babuškov
Asked: 2010-06-06 15:06:47 +0800 CST

Why does my VMWare / Linux host slow down every 6 hours?

  • 772

I have a VPS with standard LAMP stack and a busy website. Operating system is CentOS 5.5. Virtualization is done with VMWare. My server gets real slow about every 6 hours. Logging into it I see that 1.6GB of RAM is consumed. However, summing up the memory usage of active processes adds up only to about 700MB. Can anyone make any sense of this?

"free" shows this:

             total       used       free     shared    buffers     cached
Mem:       2059456    2049280      10176          0      14780     380968
-/+ buffers/cache:    1653532     405924
Swap:      2096472         96    2096376

While this is output of "ps":

[root@vmi29 /]# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  10348   688 ?        Rs   Jun05   0:01 init [3]
root         2  0.0  0.0      0     0 ?        S<   Jun05   0:00 [migration/0]
root         3  0.0  0.0      0     0 ?        SN   Jun05   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S<   Jun05   0:00 [migration/1]
root         5  0.0  0.0      0     0 ?        SN   Jun05   0:00 [ksoftirqd/1]
root         6  0.0  0.0      0     0 ?        S<   Jun05   0:00 [migration/2]
root         7  0.0  0.0      0     0 ?        SN   Jun05   0:00 [ksoftirqd/2]
root         8  0.0  0.0      0     0 ?        S<   Jun05   0:00 [migration/3]
root         9  0.0  0.0      0     0 ?        SN   Jun05   0:00 [ksoftirqd/3]
root        10  0.0  0.0      0     0 ?        S<   Jun05   0:06 [events/0]
root        11  0.0  0.0      0     0 ?        S<   Jun05   0:00 [events/1]
root        12  0.0  0.0      0     0 ?        S<   Jun05   0:00 [events/2]
root        13  0.0  0.0      0     0 ?        S<   Jun05   0:00 [events/3]
root        14  0.0  0.0      0     0 ?        S<   Jun05   0:00 [khelper]
root        31  0.0  0.0      0     0 ?        S<   Jun05   0:00 [kthread]
root        38  0.0  0.0      0     0 ?        S<   Jun05   0:00 [kblockd/0]
root        39  0.0  0.0      0     0 ?        S<   Jun05   0:00 [kblockd/1]
root        40  0.0  0.0      0     0 ?        S<   Jun05   0:00 [kblockd/2]
root        41  0.0  0.0      0     0 ?        S<   Jun05   0:00 [kblockd/3]
root        42  0.0  0.0      0     0 ?        S<   Jun05   0:00 [kacpid]
root       204  0.0  0.0      0     0 ?        S<   Jun05   0:00 [cqueue/0]
root       205  0.0  0.0      0     0 ?        S<   Jun05   0:00 [cqueue/1]
root       206  0.0  0.0      0     0 ?        S<   Jun05   0:00 [cqueue/2]
root       207  0.0  0.0      0     0 ?        S<   Jun05   0:00 [cqueue/3]
root       210  0.0  0.0      0     0 ?        S<   Jun05   0:00 [khubd]
root       212  0.0  0.0      0     0 ?        S<   Jun05   0:00 [kseriod]
root       302  0.0  0.0      0     0 ?        S    Jun05   0:00 [khungtaskd]
root       303  0.0  0.0      0     0 ?        S    Jun05   0:00 [pdflush]
root       304  0.0  0.0      0     0 ?        S    Jun05   0:01 [pdflush]
root       305  0.0  0.0      0     0 ?        S<   Jun05   0:05 [kswapd0]
root       306  0.0  0.0      0     0 ?        S<   Jun05   0:00 [aio/0]
root       307  0.0  0.0      0     0 ?        S<   Jun05   0:00 [aio/1]
root       308  0.0  0.0      0     0 ?        S<   Jun05   0:00 [aio/2]
root       309  0.0  0.0      0     0 ?        S<   Jun05   0:00 [aio/3]
root       515  0.0  0.0      0     0 ?        S<   Jun05   0:00 [kpsmoused]
root       582  0.0  0.0      0     0 ?        S<   Jun05   0:00 [mpt_poll_0]
root       583  0.0  0.0      0     0 ?        S<   Jun05   0:00 [mpt/0]
root       584  0.0  0.0      0     0 ?        S<   Jun05   0:00 [scsi_eh_0]
root       590  0.0  0.0      0     0 ?        S<   Jun05   0:00 [ata/0]
root       591  0.0  0.0      0     0 ?        S<   Jun05   0:00 [ata/1]
root       592  0.0  0.0      0     0 ?        S<   Jun05   0:00 [ata/2]
root       593  0.0  0.0      0     0 ?        S<   Jun05   0:00 [ata/3]
root       594  0.0  0.0      0     0 ?        S<   Jun05   0:00 [ata_aux]
root       610  0.0  0.0      0     0 ?        S<   Jun05   0:00 [kstriped]
root       631  0.0  0.0      0     0 ?        S<   Jun05   0:05 [kjournald]
root       656  0.0  0.0      0     0 ?        S<   Jun05   0:00 [kauditd]
root       689  0.0  0.0  13364   928 ?        S<s  Jun05   0:00 /sbin/udevd -d
root      2123  0.0  0.0      0     0 ?        S<   Jun05   0:00 [kmpathd/0]
root      2124  0.0  0.0      0     0 ?        S<   Jun05   0:00 [kmpathd/1]
root      2126  0.0  0.0      0     0 ?        S<   Jun05   0:00 [kmpathd/2]
root      2127  0.0  0.0      0     0 ?        S<   Jun05   0:00 [kmpathd/3]
root      2128  0.0  0.0      0     0 ?        S<   Jun05   0:00 [kmpath_handlerd]
root      2203  0.0  0.0      0     0 ?        S<   Jun05   0:00 [kjournald]
root      2613  0.0  0.0   5908   648 ?        Ss   Jun05   0:00 syslogd -m 0
root      2617  0.0  0.0   3804   424 ?        Ss   Jun05   0:00 klogd -x
root      2707  0.0  0.0  10760   372 ?        Ss   Jun05   0:02 irqbalance
apache    2910  0.5  0.6 213964 12912 ?        S    00:22   0:07 /usr/sbin/httpd
dbus      3011  0.0  0.0  21256   904 ?        Ss   Jun05   0:00 dbus-daemon --system
root      3025  0.0  0.0   3800   576 ?        Ss   Jun05   0:00 /usr/sbin/acpid
68        3038  0.0  0.2  31152  4336 ?        Ss   Jun05   0:01 hald
root      3039  0.0  0.0  21692  1176 ?        S    Jun05   0:00 hald-runner
68        3046  0.0  0.0  12324   856 ?        S    Jun05   0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.s
68        3052  0.0  0.0  12324   856 ?        S    Jun05   0:00 hald-addon-keyboard: listening on /dev/input/event0
root      3105  0.0  0.0  62624  1212 ?        Ss   Jun05   0:00 /usr/sbin/sshd
root      3264  0.0  0.0  74820  1156 ?        Ss   Jun05   0:00 crond
root      3292  0.0  0.0  18416   472 ?        S    Jun05   0:00 /usr/sbin/smartd -q never
root      3300  0.0  0.0   3792   480 tty2     Ss+  Jun05   0:00 /sbin/mingetty tty2
root      3301  0.0  0.0   3792   480 tty3     Ss+  Jun05   0:00 /sbin/mingetty tty3
root      3302  0.0  0.0   3792   484 tty4     Ss+  Jun05   0:00 /sbin/mingetty tty4
root      3304  0.0  0.0   3792   480 tty5     Ss+  Jun05   0:00 /sbin/mingetty tty5
root      3306  0.0  0.0   3792   480 tty6     Ss+  Jun05   0:00 /sbin/mingetty tty6
apache    5158  0.4  0.5 211896 11848 ?        S    00:28   0:04 /usr/sbin/httpd
apache    5519  0.4  0.5 211896 11992 ?        S    00:29   0:03 /usr/sbin/httpd
root      5649  0.0  0.0  63848  1184 pts/0    S    Jun05   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --soc
mysql     5696  2.1  1.9 411060 40392 pts/0    Rl   Jun05   2:01 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql
apache    5943  0.4  0.5 211896 12000 ?        S    00:30   0:03 /usr/sbin/httpd
apache    5976  0.6  0.5 211896 11792 ?        S    00:30   0:04 /usr/sbin/httpd
apache    6073  0.4  0.5 211896 11208 ?        S    00:31   0:03 /usr/sbin/httpd
apache    6122  0.4  0.5 211896 11848 ?        S    00:31   0:03 /usr/sbin/httpd
apache    6128  0.3  0.5 211896 11940 ?        S    00:31   0:02 /usr/sbin/httpd
apache    6159  0.5  0.5 211896 11872 ?        S    00:31   0:04 /usr/sbin/httpd
apache    6636  0.4  0.6 213960 13444 ?        S    00:32   0:02 /usr/sbin/httpd
apache    6787  0.3  0.5 211884 11308 ?        S    00:33   0:02 /usr/sbin/httpd
apache    6796  0.4  0.5 211884 12024 ?        S    00:33   0:02 /usr/sbin/httpd
apache    6801  0.3  0.5 211896 11920 ?        S    00:33   0:01 /usr/sbin/httpd
apache    6804  0.4  0.5 211884 11848 ?        S    00:33   0:02 /usr/sbin/httpd
apache    6825  0.4  0.5 211896 11972 ?        S    00:33   0:02 /usr/sbin/httpd
apache    6866  0.3  0.5 210860 11044 ?        S    00:33   0:01 /usr/sbin/httpd
apache    6870  0.2  0.5 211896 11108 ?        S    00:33   0:01 /usr/sbin/httpd
apache    6872  0.3  0.5 211896 11900 ?        S    00:33   0:01 /usr/sbin/httpd
apache    6993  0.3  0.5 211896 11836 ?        S    00:33   0:02 /usr/sbin/httpd
apache    6994  0.3  0.5 211896 11792 ?        S    00:33   0:01 /usr/sbin/httpd
apache    7136  0.2  0.5 211896 11432 ?        S    00:34   0:01 /usr/sbin/httpd
apache    7143  0.2  0.5 210860 11052 ?        S    00:34   0:01 /usr/sbin/httpd
apache    7145  0.2  0.5 211896 11136 ?        S    00:34   0:01 /usr/sbin/httpd
apache    7266  0.2  0.6 213952 12748 ?        S    00:34   0:01 /usr/sbin/httpd
apache    7299  0.2  0.5 211884 11276 ?        S    00:34   0:01 /usr/sbin/httpd
apache    7311  0.2  0.5 211884 11300 ?        S    00:34   0:01 /usr/sbin/httpd
apache    7313  0.3  0.5 211884 11876 ?        S    00:34   0:01 /usr/sbin/httpd
apache    7345  0.2  0.5 210872 11100 ?        S    00:34   0:01 /usr/sbin/httpd
apache    7349  0.2  0.5 210860 11008 ?        S    00:34   0:01 /usr/sbin/httpd
apache    7350  0.2  0.5 211896 11832 ?        S    00:34   0:01 /usr/sbin/httpd
apache    7351  0.1  0.5 211884 11072 ?        S    00:34   0:00 /usr/sbin/httpd
apache    7352  0.2  0.5 210872 11096 ?        S    00:34   0:01 /usr/sbin/httpd
apache    7449  0.1  0.5 210860 11020 ?        S    00:35   0:00 /usr/sbin/httpd
root      7490  0.3  0.0      0     0 ?        S    Jun05   3:11 [vmmemctl]
root      7597  0.0  0.0  72656  1260 ?        Ss   Jun05   0:06 /usr/lib/vmware-tools/sbin64/vmware-guestd --background /va
apache    7720  0.1  0.5 210860 10748 ?        S    00:36   0:00 /usr/sbin/httpd
apache    7726  0.1  0.4 209836  9304 ?        R    00:36   0:00 /usr/sbin/httpd
apache    7727  0.1  0.5 210860 10916 ?        S    00:36   0:00 /usr/sbin/httpd
apache    7731  0.1  0.5 210860 10780 ?        S    00:36   0:00 /usr/sbin/httpd
apache    7732  0.3  0.5 210860 10916 ?        S    00:36   0:01 /usr/sbin/httpd
apache    7733  0.1  0.5 210872 11000 ?        S    00:36   0:00 /usr/sbin/httpd
apache    7735  0.1  0.5 211884 11048 ?        S    00:36   0:00 /usr/sbin/httpd
apache    7761  0.1  0.5 210860 10552 ?        S    00:36   0:00 /usr/sbin/httpd
apache    7776  0.1  0.4 209836  8648 ?        R    00:37   0:00 /usr/sbin/httpd
apache    7790  0.2  0.3 208812  7724 ?        R    00:40   0:00 /usr/sbin/httpd
apache    7800  0.2  0.3 208812  8088 ?        R    00:40   0:00 /usr/sbin/httpd
root      7801  0.0  0.0   3792   484 tty1     Ss+  00:41   0:00 /sbin/mingetty tty1
apache    7820  0.2  0.3 208812  7552 ?        R    00:41   0:00 /usr/sbin/httpd
apache    7834  0.2  0.3 207788  6756 ?        R    00:42   0:00 /usr/sbin/httpd
apache    7864  0.2  0.2 207788  6148 ?        R    00:42   0:00 /usr/sbin/httpd
apache    7872  0.3  0.2 207788  5856 ?        R    00:43   0:00 /usr/sbin/httpd
apache    7874  2.5  0.3 207788  6336 ?        R    00:43   0:00 /usr/sbin/httpd
root      7875  0.3  0.0  63844  1056 ?        S    00:43   0:00 sh -c lsb_release -sd 2>/dev/null
root      7879  1.6  0.0  65604   964 pts/0    R+   00:43   0:00 ps aux
root     16316  0.0  0.1  90128  3272 ?        Ss   Jun05   0:00 sshd: milanb [priv]
milanb   16358  0.0  0.0  90128  1752 ?        S    Jun05   0:00 sshd: milanb@pts/0
milanb   16360  0.0  0.0  66076  1480 pts/0    Ss   Jun05   0:00 -bash
root     16875  0.0  0.0 101068  1324 pts/0    S    Jun05   0:00 su -
root     16877  0.0  0.0  66184  1692 pts/0    S    Jun05   0:00 -bash
root     24373  0.0  0.3 206764  7348 ?        Rs   Jun05   0:01 /usr/sbin/httpd

UPDATE:

Here, it happens again (about a hour later this time). 

[root@vmi29 ~]# cat /proc/meminfo
MemTotal:      2059456 kB
MemFree:         17340 kB
Buffers:         17788 kB
Cached:         407804 kB
SwapCached:          0 kB
Active:         517704 kB
Inactive:       122188 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      2059456 kB
LowFree:         17340 kB
SwapTotal:     2096472 kB
SwapFree:      2096376 kB
Dirty:             160 kB
Writeback:           0 kB
AnonPages:      214436 kB
Mapped:          13296 kB
Slab:            27392 kB
PageTables:      18780 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   3126200 kB
Committed_AS:   514788 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    264012 kB
VmallocChunk: 34359473911 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

iostat output:

[root@vmi29 ~]# iostat
Linux 2.6.18-194.3.1.el5 (vmi29)        06/06/2010

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.33    0.00    1.07    1.83    0.00   90.78

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              12.02        44.47       123.40    2526367    7011298
sda1              0.00         0.04         0.10       2036       5466
sda2              0.00         0.03         0.00       1681        200
sda3             12.01        44.39       123.30    2522226    7005632

Free:

[root@vmi29 ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          2011       2002          8          0         15        386
-/+ buffers/cache:       1600        410
Swap:         2047          0       2047

Command that Nathan Powell suggested:

[root@vmi29 ~]# ps aux | awk '{SUM += $3} END { print SUM }'
20.2

Update2:

[root@vmi29 ~]# ps aux | awk '{SUM += $4} END { print SUM }'
12.5

Free:

[root@vmi29 ~]# free
             total       used       free     shared    buffers     cached
Mem:       2059456    2038316      21140          0       7360     492800
-/+ buffers/cache:    1538156     521300
Swap:      2096472        100    2096372
linux apache-2.2 memory

4 Answers

  1. Run slabtop or parse /proc/slabinfo and look at your kernel slabs; it’s very common for the kernel to cache a lot of directory entries and inodes (dentry_cache, ext3_inode_cache) on a system, especially one with lots of file access like an Apache server that’s sending lots of static content (images, etc.). This is where your “missing” memory is usually.

    If this tweaks you out you can adjust vm.cache_pressure in /etc/sysctl.conf to reduce that usage but I highly recommend understanding why first.

    • 2
  2. Best Answer

    Check vmmemctl memory usage. I had a similar problem, see RedHat Linux: server paging, sum of RES/RSS + buffers + cached < TOTAL. Who is using my memory?

    In my case, we had an 8GB RAM server and we couldn't find which process was using. This is our vmmemctl:

    cat /proc/vmmemctl

target:              1000894 pages
current:             1000894 pages
rateNoSleepAlloc:      16384 pages/sec
rateSleepAlloc:         2048 pages/sec
rateFree:              16384 pages/sec

timer:                325664
start:                     3 (   0 failed)
guestType:                 3 (   0 failed)
lock:                3623088 (  29 failed)
unlock:               623698 (   0 failed)
target:               325664 (   2 failed)
primNoSleepAlloc:    3620199 (  11 failed)
primCanSleepAlloc:      2900 (   0 failed)
primFree:            2622165
errAlloc:                 28
errFree:                  28

getconf PAGESIZE
4096

    So vmmemctl is using 4GBs

    It's a pity the vmmemctl doesn't use a standard method to report how much memory it's using, but I think it's because how it's implemented.

    The main reference from vmware offers a lot of detail about ballooning. I quote since it's relevant to our original problem ( 'why is this server paging if it has non used memory'? ):

    "Typically, the hypervisor inflates the virtual machine balloon when it is under memory pressure. By inflating the balloon, a virtual machine consumes less physical memory on the host, but more physical memory inside the guest. As a result, the hypervisor offloads some of its memory overload to the guest operating system while slightly loading the virtual machine. That is, the hypervisor transfers the memory pressure from the host to the virtual machine. Ballooning induces guest memory pressure. In response, the balloon driver allocates and pins guest physical memory. The guest operating system determines if it needs to page out guest physical memory to satisfy the balloon driver’s allocation requests. If the virtual machine has plenty of free guest physical memory, inflating the balloon will induce no paging and will not impact guest performance. In this case, as illustrated in Figure 6, the balloon driver allocates the free guest physical memory from the guest free list. Hence, guest-level paging is not necessary.

    However, if the guest is already under memory pressure, the guest operating system decides which guest physical pages to be paged out to the virtual swap device in order to satisfy the balloon driver’s allocation requests. The genius of ballooning is that it allows the guest operating system to intelligently make the hard decision about which pages to be paged out without the hypervisor’s involvement."

    "genius of ballooning" :)

    • 2

  3. VSZ = Virtual Memory Size.

    Have a look at all your httpd (apache) processes. 200MB each times lots.

    • 0

  4. First 'free -m' which shows the usage in Megs, makes more sense visually...since it's not 1996 anymore :)

    Second, you have a lot of ram cached, so you aren't out of ram.

    3rd, you are only swapping 96K. That is trivial.

    To see what is swapping, open 'top', Then use 'O' (cap o) then 'p' then enter.

    Ram is not the only thing that will make things "slow". Look at disk i/o and Proc usage as well.

    • 0