Questions[memory-usage](server)

This is a canonical question about how Unix operating systems report memory usage.
Similar Questions:

• Server refuses to use swap partition
• Memory Usage in LINUX

I have production server that is running Debian 6.0.6 Squeeze

#uname -a

Linux debsrv 2.6.32-5-xen-amd64 #1 SMP Sun Sep 23 13:49:30 UTC 2012 x86_64 GNU/Linux

Every day cron executes backup script as root:

#crontab -e

0 5 * * * /root/sites_backup.sh > /dev/null 2>&1

#nano /root/sites_backup.sh

#!/bin/bash
str=`date +%Y-%m-%d-%H-%M-%S`
tar pzcf /home/backups/sites/mysite-$str.tar.gz /var/sites/mysite/public_html/www
mysqldump -u mysite -pmypass mysite | gzip -9 > /home/backups/sites/mysite-$str.sql.gz
cd /home/backups/sites/
sha512sum mysite-$str* > /home/backups/sites/mysite-$str.tar.gz.DIGESTS
cd ~

Everything works perfectly, but I notice that Munin's memory graph shows increase of cache and buffers after backup.

Then I just download backup files and delete them. After deletion Munin's memory graph returns cache and buffers to the state that was before backup.

Here's Munin graph:

Externally hosted image was a dead link.

My VPS web server running on CentOS 5.4 (Linux kernel 2.6.16.33-xenU) irregularly (like once a month give or take a few weeks) becomes unresponsive due to oom-killer kicking in. Monitoring of the server shows that it doesn't normally run out of memory, just every so often.

I've read a couple of blogs that point to this page which discusses configuring the kernel to better manage overcommit using the following sysctl settings:

vm.overcommit_memory = 2
vm.overcommit_ratio = 80

My understanding of this (which may be wrong, but I can't find a canonical definition to clarify) is that this prevents the kernel over-allocating memory beyond swap + 80% of physical memory.

However, I have also read some other sources suggesting that these settings are not a good idea - although the critics of this approach seem to be saying "don't do things to break your system, rather than attempting this kludge" in the assumption that causation is always known.

So my question is, what are the pros and cons of this approach, in the context of an Apache2 web server hosting about 10 low traffic sites? In my specific case, the web server has 512Mb RAM, with 1024Mb swap space. This seems to be adequate for the vast majority of the time.

Running Ubuntu on a 2.6.31-302 x86-64 kernel. The overall problem is that I have memory in the 'cached' category that keeps on going up and will not be freed or used even when our application needs it.

So here's what I get out of the 'free' command. None of this looks out of the ordinary at first glance.

# free
             total       used       free     shared    buffers     cached
Mem:       7358492    5750320    1608172          0       7848    1443820
-/+ buffers/cache:    4298652    3059840
Swap:            0          0          0

The first thing someone's going to say is "Don't worry, linux manages that memory automatically." Yes, I know how the memory manager is supposed to work; the problem is that it's not doing the right thing. The "cached" 1.4 GB here appears to be reserved and unusable.

My knowledge of Linux tells me that 3 GB is "free"; but the behavior of the system says otherwise. When the 1.6 GB of real free memory is used up during peak usage, as soon as more memory is demanded (and the 'free' in the first column approaches 0) the OOM killer is invoked, processes are killed, and problems start to arise even though the 'free' in the -/+ buffers/cache row still has about 1.4 GB 'free'.

I've tuned the oom_adj values on key processes so it doesn't bring the system to its knees, but even then important processes will be killed, and we never want to reach that point. Especially when, theoretically, 1.4GB is still "free" if it would only evict the disk cache.

Does anyone have any idea what's going on here? The internet is flooded with the dumb questions about the Linux 'free' command and "why don't I have any free memory" and I can't find anything about this issue because of that.

The first thing that pops into my head is that swap is off. We have a sysadmin that is adamant about it; I am open to explanations if they're backed up. Could this cause problems?

Here's free after running echo 3 > /proc/sys/vm/drop_caches :

# free
             total       used       free     shared    buffers     cached
Mem:       7358492    5731688    1626804          0        524    1406000
-/+ buffers/cache:    4325164    3033328
Swap:            0          0          0

As you can see, some minuscule amount of cache is actually freed up, but around 1.4 GB appears to be "stuck." The other problem is that this value seems to rise over time. On another server 2.0 GB is stuck.

I'd really like this memory back... any help would be most appreciated.

Here's cat /proc/meminfo if it's worth anything:

# cat /proc/meminfo 
MemTotal:        7358492 kB
MemFree:         1472180 kB
Buffers:            5328 kB
Cached:          1435456 kB
SwapCached:            0 kB
Active:          5524644 kB
Inactive:          41380 kB
Active(anon):    5492108 kB
Inactive(anon):        0 kB
Active(file):      32536 kB
Inactive(file):    41380 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               320 kB
Writeback:             0 kB
AnonPages:       4125252 kB
Mapped:            42536 kB
Slab:              29432 kB
SReclaimable:      13872 kB
SUnreclaim:        15560 kB
PageTables:            0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     3679244 kB
Committed_AS:    7223012 kB
VmallocTotal:   34359738367 kB
VmallocUsed:        7696 kB
VmallocChunk:   34359729675 kB
DirectMap4k:     7340032 kB
DirectMap2M:           0 kB

Here the output of free -m:

             total       used       free     shared    buffers     cached
Mem:          7188       6894        294          0        249       5945
-/+ buffers/cache:        698       6489
Swap:            0          0          0

I can see almost 6GB(5945MB) memory out of 7GB is used in caching the files. I know how to flush the caches. My question is: Is possible see which files(or inodes) are being cached?

Why does my server show

            total      used      free   shared buffers    cached
    Mem: 12286456  11715372    571084        0   81912   6545228
-/+ buffers/cache:  5088232   7198224
   Swap: 24571408     54528  24516880

I have no idea on calculating the memory in linux. I think it says that 5088232 is used where as 7198224 is free, meaning it is actually consuming 5GB of RAM?