I am not sure how to interpret the memory usage of our servers on which WebSphere MQ (WMQ) is running. The main question is: Is WMQ using more and more memory over time (is it leaking memory) or is everything just fine and Linux is using our RAM for disk caching?
We have the following Cacti graph.
The data for this is polled from /proc/meminfo. Which currently shows the following output.
[user@server ~]$ cat /proc/meminfo
MemTotal: 32956188 kB
MemFree: 3963664 kB
Buffers: 1225024 kB
Cached: 15611124 kB
SwapCached: 34016 kB
Active: 23880484 kB
Inactive: 3279676 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 32956188 kB
LowFree: 3963664 kB
SwapTotal: 8388600 kB
SwapFree: 8354584 kB
Dirty: 1648 kB
Writeback: 0 kB
AnonPages: 10290180 kB
Mapped: 457704 kB
Slab: 1375028 kB
PageTables: 136452 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 24866692 kB
Committed_AS: 19962412 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 382196 kB
VmallocChunk: 34359356007 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
The command free -m shows the following currently.
[user@server ~]$ free -m
total used free shared buffers cached
Mem: 32183 28312 3871 0 1196 15245
-/+ buffers/cache: 11870 20313
Swap: 8191 33 8158
According to the web site "linuxatemyram dot com" only the line "-/+ buffers/cache: 11870 20313" of "free -m" is relevant.
In the Cacti graph you can clearly see that "Used Memory" is increasing since the beginning of "Week 03". On the other hand "Cache" and "Buffers" seem to be pretty constant. How is "Used Memory" relevant in this case? Is WMQ leaking memory?
Yes, something is consuming more and more memory. No, it's not the VFS (buffers and cache). It's not possible to draw any further conclusions from the data you've provided.
You really need to look at which processes / programs are using what memory - this provides a better indicator than just looking at the RSS / VSZ, but it can still be misled by COW pages. Try to get a series of snapshots and see if the change matches the change in your cacti graph.
You have ~20GB of RAM available to processes on that system. The OS will use free RAM to cache filesystem reads speed up subsequent reads of the same files. It's quite normal to see 'Used' RAM increasing slowly over time as more and more of the filesystem is accessed; you won't see any performance problems as a result of this and if a process needs to allocate this RAM later, then the OS will gladly relinquish it.
Edit
As @symcbean points out, something is slowly consuming more memory. If the server is given over to just this one application, then that's the first place to start looking, but I second that recommendation of the python script for getting a clearer picture of what's happening.