I've noticed high memory usage on the server. Details:
OS: CentOS 6.3 - x86_64
Web server: Nginx with PHP-FPM
The server is generating PDF documents so the traffic is minimum.
top:
# top -b -n 1 -a
top - 10:04:51 up 21 days, 18:57, 1 user, load average: 0.00, 0.00, 0.00
Tasks: 92 total, 1 running, 91 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 0.2%sy, 0.0%ni, 99.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3923092k total, 3720380k used, 202712k free, 133904k buffers
Swap: 4194296k total, 12k used, 4194284k free, 147404k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15855 www-data 20 0 199m 4952 2128 S 0.0 0.1 0:00.06 php-fpm
15853 www-data 20 0 199m 4940 2028 S 0.0 0.1 0:00.06 php-fpm
15850 www-data 20 0 199m 4928 2020 S 0.0 0.1 0:00.05 php-fpm
15851 www-data 20 0 199m 4888 2020 S 0.0 0.1 0:00.06 php-fpm
15852 www-data 20 0 199m 4852 2020 S 0.0 0.1 0:00.06 php-fpm
15857 www-data 20 0 198m 4716 2020 S 0.0 0.1 0:00.06 php-fpm
17553 root 20 0 97816 3860 2924 S 0.0 0.1 0:00.03 sshd
15849 root 20 0 198m 3460 1072 S 0.0 0.1 0:00.12 php-fpm
13441 nginx 20 0 65608 2968 1604 S 0.0 0.1 0:02.06 nginx
13440 nginx 20 0 65608 2964 1600 S 0.0 0.1 0:01.87 nginx
17561 root 20 0 105m 1944 1488 S 0.0 0.0 0:00.01 bash
1150 xfs 20 0 20980 1784 704 S 0.0 0.0 0:00.13 xfs
15863 root 20 0 179m 1424 1028 S 0.0 0.0 0:00.00 rsyslogd
1 root 20 0 19224 1360 1088 S 0.0 0.0 0:17.96 init
1201 nrpe 20 0 40928 1288 704 S 0.0 0.0 3:57.64 nrpe
13226 root 20 0 114m 1216 612 S 0.0 0.0 0:00.01 crond
6691 root 20 0 64068 1156 488 S 0.0 0.0 0:09.59 sshd
13439 root 20 0 65104 1128 292 S 0.0 0.0 0:00.00 nginx
19026 root 20 0 15040 1116 844 R 0.0 0.0 0:00.00 top
451 root 16 -4 11052 1096 316 S 0.0 0.0 0:00.02 udevd
1174 root 18 -2 11048 1064 288 S 0.0 0.0 0:00.00 udevd
1175 root 18 -2 11048 1064 288 S 0.0 0.0 0:00.00 udevd
1065 root 16 -4 93168 824 560 S 0.0 0.0 0:16.00 auditd
1165 root 20 0 4056 564 480 S 0.0 0.0 0:00.00 mingetty
1167 root 20 0 4056 564 480 S 0.0 0.0 0:00.00 mingetty
1169 root 20 0 4056 564 480 S 0.0 0.0 0:00.00 mingetty
1171 root 20 0 4056 564 480 S 0.0 0.0 0:00.00 mingetty
1163 root 20 0 4056 560 480 S 0.0 0.0 0:00.00 mingetty
1176 root 20 0 4056 560 480 S 0.0 0.0 0:00.00 mingetty
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:11.75 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 44:30.28 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:03.51 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 0:11.63 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
9 root 20 0 0 0 0 S 0.0 0.0 11:35.50 ksoftirqd/1
10 root RT 0 0 0 0 S 0.0 0.0 0:03.34 watchdog/1
11 root 20 0 0 0 0 S 0.0 0.0 1:36.68 events/0
12 root 20 0 0 0 0 S 0.0 0.0 1:50.57 events/1
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cgroup
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khelper
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 netns
16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 async/mgr
17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pm
18 root 20 0 0 0 0 S 0.0 0.0 0:07.86 sync_supers
19 root 20 0 0 0 0 S 0.0 0.0 0:10.38 bdi-default
20 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/0
21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/1
22 root 20 0 0 0 0 S 0.0 0.0 0:04.35 kblockd/0
23 root 20 0 0 0 0 S 0.0 0.0 0:04.18 kblockd/1
24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpid
25 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpi_notify
26 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpi_hotplug
27 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/0
28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/1
29 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata_aux
30 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksuspend_usbd
31 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khubd
32 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kseriod
33 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/0
34 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/1
35 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/0
36 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/1
37 root 20 0 0 0 0 S 0.0 0.0 0:00.48 khungtaskd
38 root 20 0 0 0 0 S 0.0 0.0 1:07.52 kswapd0
39 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd
40 root 39 19 0 0 0 S 0.0 0.0 0:22.00 khugepaged
41 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/0
42 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/1
43 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/0
44 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/1
49 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthrotld/0
50 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthrotld/1
52 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kpsmoused
53 root 20 0 0 0 0 S 0.0 0.0 0:00.00 usbhid_resumer
83 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kstriped
233 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0
234 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1
321 root 20 0 0 0 0 S 0.0 0.0 0:00.00 virtio-blk
359 root 20 0 0 0 0 S 0.0 0.0 0:03.24 kdmflush
360 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdmflush
380 root 20 0 0 0 0 S 0.0 0.0 0:20.64 jbd2/dm-0-8
381 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ext4-dio-unwrit
382 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ext4-dio-unwrit
694 root 20 0 0 0 0 S 0.0 0.0 0:00.00 vballoon
697 root 20 0 0 0 0 S 0.0 0.0 0:00.00 virtio-net
818 root 20 0 0 0 0 S 0.0 0.0 0:00.00 jbd2/vda1-8
819 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ext4-dio-unwrit
820 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ext4-dio-unwrit
851 root 20 0 0 0 0 S 0.0 0.0 0:06.96 kauditd
1013 root 20 0 0 0 0 S 0.0 0.0 0:15.45 flush-253:0
ps:
# ps aux --sort -vsz | head
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
www-data 13213 0.0 0.1 204416 4772 ? S 08:28 0:00 php-fpm: pool default
www-data 13214 0.0 0.1 204416 4776 ? S 08:28 0:00 php-fpm: pool default
www-data 13215 0.0 0.1 204416 4832 ? S 08:28 0:00 php-fpm: pool default
www-data 13216 0.0 0.1 204416 4776 ? S 08:28 0:00 php-fpm: pool default
www-data 13218 0.0 0.1 204416 4956 ? S 08:28 0:00 php-fpm: pool default
free:
#free -m
total used free shared buffers cached
Mem: 3831 3530 300 0 130 143
-/+ buffers/cache: 3256 574
Swap: 4095 0 4095
When I stooped Nginx, PHP-FPM the memory usage was still the same.
Could you help me to investigate what is consuming the memory on the system?
Regards
Your KVM-based virtual machine is reporting much more memory used than the running processes account for.
I would guess the most likely cause of this is that the host is using memory ballooning to overcommit memory usage on the host.
You won't have much control over this unless you also control the host. If you've leased this virtual server, check with the host regarding their memory overcommit policies.
You might also want to check for ramdisk mounts. A mount|grep tmpfs should show any. If there are large files in those directories, that might cause large RAM usage without it showing up in top, though I don't think it's possible for a ramdisk to take up all the memory.
You misread the output of
free.This command show your sever has 300 MiB free and 3256 MiB used for buffers and cache. So you only have 3831-3256=575 MiB used. Which is clearly not high usage.
Edit : I took the wrong numbers. This is 3530-3256=274MiB used by programs and kernel, without counting buffers and cache.
The memory used by buffers and cache will be released if any program needs it.
As your sever don't use lots of memory, the kernel takes advantage of it to store some datas which can be later discarded. When a program needs more memory, the kernel discard some cache to give this memory to the program.
The amount of really available memory is tuned by the kernel. If the I/O load is heavy, the kernel tend to keep more memory for cache.