Home / server / Questions / 11754
Accepted
davitenio
Asked: 2009-05-25 02:44:44 +0800 CST

Running batch jobs and getting a peak memory usage summary for each job

  • 772

I want to execute a batch of jobs and afterwards have a summary telling me what the peak memory requirement for each job was. Executing the jobs under a profiler such as valgrind is not acceptable because it would slow down the jobs. For a running job I could read the value of VmPeak under /proc/JOBPID/status, where JOBPID is the PID of a job's process. But I need to get the job's all-time maximum memory requirement, therefore I would need to get the value of VmPeak when the job's process is just about to be finished, otherwise I would just get the peak memory usage up to the moment I read VmPeak, which could increase after I read it. So unless there is a way to read the value of VmPeak of a process that has finished, this approach doesn't seem useful. Any other ideas on how to get the maximum amount of memory that had been allocated to a process from the moment it started up to the moment it finished?

linux memory

3 Answers

  1. The VmPeak answer looks like a good one to me... You could either append vmpeak to a file every x seconds and then find the highest value. Or, every x seconds, run something like: VmPeak = curVmPeak if (curVmPeak > VmPeak).

    • 1

  2. Have you tried overriding the child reaper for your master program? You probably have access to the process statistics you're interested in before you reap the dead child.

    My last bet would be overriding libc's exit() call with your own shared library and have your procedure report those statistics for you.

    • 1
  3. Best Answer

    I was hoping for a solution along the lines of using a hook script that could be triggered when a job's process finished and which would then briefly suspend the process to inspect the value of VmPeak. Modifying the source code of each job I could probably easily do something similar by writing an exit handler (using something like the atexit function). But I didn't want to touch the source.

    The solution I finally used was basically what Kyle Brandt proposed, i.e. sample the peak memory usage every few units of time and determine the maximum this way. Although I didn't have to write it myself. I found a small tool that does basically exactly that, it's called memtime.

    • 0