I'm running top and I see that out of 502968 KB, 48064 KB is used leaving 16884 KB free. But then when I look at the individual processes I see that mysql is consuming 9.4% of my RAM on occasion - but nothing else seems to be consuming anything. What is consuming all my RAM?
Here is a screen capture from top:
top - 20:46:07 up 1 min, 1 user, load average: 0.18, 0.05, 0.02
Tasks: 81 total, 1 running, 80 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.4 sy, 0.0 ni, 99.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 502968 total, 241236 used, 261732 free, 10488 buffers
KiB Swap: 524284 total, 0 used, 524284 free, 106756 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1584 dmackey 20 0 20508 1372 1000 R 0.4 0.3 0:00.01 top
1 root 20 0 26664 2456 1340 S 0.0 0.5 0:00.69 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
6 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u:0
7 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/u:0H
8 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
10 root 20 0 0 0 0 S 0.0 0.0 0:00.24 rcu_sched
11 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
12 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 cpuset
13 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 bdi-default
And here is one from free:
total used free shared buffers cached
Mem: 502968 355252 147716 0 10816 111548
-/+ buffers/cache: 232888 270080
Swap: 524284 0 524284
Nothing.
It's Linux, and that's how it works.
It'll use "free" memory for disk caching, because what's the point in having an under-utilised resource?
When a process wants more memory, it'll ask for it from the kernel, and (generally), get it.
Oh.. and MySQL will use as much as it needs to (as configured in
/etc/my.cnf) so that indexes, queries and tables are cached from disk, into memory to make access times faster.It's just whatever junk happens to be left in memory from however it was used last. Most likely, it's data that was read from disk or written to disk. The OS doesn't make the memory free because that's a waste for three reasons:
If the memory isn't used soon, then making it free accomplishes nothing.
If the memory is used soon, then making it free is just work that has to be undone as it has to be made used again.
If the information that was in the memory is needed again, making the memory free will mean having to get that information from disk instead of memory, which is much slower.
Modern operating systems only make memory free if they have absolutely no other choice. If you're thinking, "I want that memory free now so I can use it later", stop. You don't need the memory to be free now in order to use it later. There's no tradeoff here, it's a pure win to keep the memory in use until it's needed.
Logging top's output for future analysis is the best way to answer your question. There's no quick fix, some observed investigation over time will be necessary. A one-off I use to skim the surface of what's using my memory is putting this into root's crontab.
Then I use some text manipulation and pager-fu to track trends. This stack overflow post https://stackoverflow.com/questions/7908953/how-to-measure-cpu-usage has some more comprehensive examples of using top like this, and customizing top via
.toprc, for CPU stats, but it can just as easily be adapted to fit your needs.