Home / ubuntu / Questions / 1444396
Accepted
Artur Meinild
Asked: 2022-12-08 01:27:12 +0800 CST

How can I correctly calculate the sum of memory used by all processes?

  • 772

There recently was a couple of questions asking why Ubuntu is seemingly using more memory than accounted for by processes/services - here and here.

However, I have the exact opposite situation. On my server, the memory used by all processes seems to be higher than the total memory used. Let me provide the evidence.

Output of free -m:

               total        used        free      shared  buff/cache   available
Mem:           64277       15014       34424          31       14837       48530
Swap:           8191           0        8191

Output of arcstat:

    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  size     c  avail
09:38:57     0     0      0     0    0     0    0     0    0  9.8G  9.8G    41G

Output of ps aux | awk '{sum +=$6}END{print sum}':

6351512

So what this means for me is:

free -m reports a total of 15.0 GB memory used (shared memory is neglible). Subtract from this 9.8 GB reported by arcstat for ZFS ARC Cache. The remaining result is around 5.2 GB of "real" used memory. But the sum of column 6 in ps aux (RSS - resident set size) totals to 6.35 GB. This would mean that reported memory used by all processes is 1.15 GB higher than it actually is shown by free -m.

I wonder if there is a way I can correctly calculate the memory used by all processes, so it matches the used memory (after ZFS cache and shared ramdisks are subtracted)?

process

1 Answers

  1. Best Answer

    So as many are probably aware, the memory statistics reported by ps (and the metric RSS itself) aren't entirely accurate because of shared memory pages.

    There is another tool called smem, which correctly reports the proportionate set size for each process - that is each process' memory consumption where the shared memory is divided in a sane way between processes.

    From man smem:

    smem reports physical memory usage, taking shared memory pages into account.
Unshared memory is reported as the USS (Unique Set Size). Shared memory is 
divided evenly among the processes sharing that memory. The unshared memory
(USS) plus a process's proportion of shared memory is reported as the PSS 
(Proportional Set Size). The USS and PSS only include physical memory usage. 
They do not include memory that has been swapped out to disk.

    However, there are a few handy tricks for installing and configuring smem to easily report actual used memory by all processes.

    Installing and configuring smem

    First, if you want to install smem, you need to have the Universe repository enabled. Make sure this line is uncommented in /etc/apt/sources.list (for Ubuntu 22.04):

    deb http://archive.ubuntu.com/ubuntu jammy universe

    When you're normally trying to install smem, it wants to install a ton of dependencies (a little over 100 packages), because it has a recommended dependency on python3-matplotlib for generating graphical charts.

    Since I don't need this (and anybody with a server probably don't need it either), it's possible to only install the base package with this command:

    sudo apt install smem python3-matplotlib-

    Notice the dash after python3-matplotlib - this indicates that this packages should not be installed. Another option is to use:

    sudo apt install --no-install-recommends smem

    This has the same result, which will only install the smem package.

    The final trick is to fix the columns, so you can correctly use awk, or export the metrics in a way where you can trust the columns. To do this, we need to move the command column to the rightmost place, since the command field can include additional spaces we can't control.

    To fix the column layout and size, smem should be run with these options:

    smem -ac "user pid swap maps vss uss pss rss command"

    In addition, to include all processes, it has to be run as sudo. I've defined the following alias for smem:

    alias smem='sudo smem -ac "user pid swap maps vss uss pss rss command"'

    So in the following, when I type the command smem, it does in fact run the command with the above options.

    Comparing smem and ps metrics

    Now we can correctly compare the metrics of ps aux and smem. I was expecting a "real" used memory of 5.2 GB, but ps aux reported 6.35 GB used. Below are the results from smem.

    Count of RSS from smem:

    $ smem | awk '{sum +=$8}END{print sum}'
6464644

    Count of PSS from smem:

    $ smem | awk '{sum +=$7}END{print sum}'
5281005

    Now it's clear that the RSS reported is roughly similar (a little higher from smem). But it's also clear that the proportionate set size (PSS) more correctly matches the expected memory usage among processes - 5.2 GB.

    So to correctly measure total process memory usage, the PSS metric from smem seems to give the most meaningful result.

    • 2