Home / server / Questions / 1109552
Accepted
Leon
Asked: 2022-09-01 01:04:07 +0800 CST

How to monitor memory usage continuously with Linux?

  • 772

I have a daemon service application runing a Ubuntu server(20.04.1). The total phisical memory is 2G, and the swap space is 4G.

Sometime my service was killed by OS quietly, eventhough I add OOMScoreAdjust=-1000 into my systemd service descripting file at /etc/systemd/system/, but it is a critical app should NOT be killed. I guess that the reason why it be killed maybe is it used too more memory.

I need to get detail logs about the memory usage of my service, for example, a log entry every minute, so that I can fingure out the max value of that, and adjust the requirements of others services such as postgresql.

It is a daemon app and running a cloud VPS, so I can NOT start a top command and be watching it continuously.

Is there a way, in which I can generate such a log?

Thanks!

linux

3 Answers

  1. You can do it the professional or the hack way. You can use various monitoring and/or process accounting solutions, or you can simply run systemctl status | grep something every minute and write the memory use as reported by Systemd (which uses kernel control groups).

    • 2
  2. Best Answer

    Based on the answer by Halfgaar, doing the simple hacky way but without systemd status | grep something

    First I would strongly advise using a monitoring solution of some kind - installing Prometheus systemd-exporter should be quite easy and running Prometheus on any other machine to retrieve the stats should be to - could even be on a device at home/corp for just as long as you debug this issue. In this setup, the metrics systemd_process_resident_memory_bytes, systemd_process_virtual_memory_bytes and systemd_process_virtual_memory_max_bytes would probably be the interesting ones.

    cgroups (all of this depending on v2) have a file system under /sys/fs/cgroup in which you can query a lot of information. The most interesting one for you is memory.stat, from which the first line anon $someNumber is probably the most interesting value (giving you bytes of memory in use that are not backed by files and thus cannot be removed from memory anywhere else than swap).

    If you want that per systemd service, you can use e.g. /sys/fs/cgroup/system.slice/systemd-journald.service/memory.stat, giving you the memory for everything in systemd-journald.service (the process for this service and everything started by that). You can also differentiate between system and user (my current graphical session on my laptop, where I write this, would be in user) by using /sys/fs/cgroup/system.slice/memory.stat/.../user.slice/memory.stat instead.

    Your older Ubuntu version might have cgroups v1, in which case the paths differ - have a look at the docs for that: https://manpages.ubuntu.com/manpages/focal/man7/cgroups.7.html for cgroups in general, https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt#cat-sysfscgroupmemory0memoryusage_in_bytes for cgroup v1 memory controller.

    • 2

  3. After some experiment, I feel it's a good and simple enough solution:

    Add a crontab job to execute following command every minutes:

    */1 * * * * top -bn1|grep my_app_name >> /some/where/memory-usage.log

    With this log file, I can easily fingure out the max usage.

    • 0