I am trying to find a metric to use that will tell me if starting another process estimated to use N bytes of memory will have an adverse impact on system performance.
I know that the /proc/meminfo fields MemFree, Buffers, and Cached can be added together to give the total memory available to programs. However, I would prefer to not squeeze out cache that's actually being used to good effect (the workload results in disk hotspots).
The Active entry in /proc/meminfo seems like it could be compared with MemTotal to determine how much memory is available (either by swapping something idle out, or by being free in the first place), but I have not been able to find much information about Active and so I'm not sure.
Should I aim to have Active memory as close to MemTotal without going over as I can manage?
Is there some other way to estimate the memory behavior of running another process?
After experimentation, it seems like Active memory is a good indicator.
What we observed is that as free memory dropped to zero, very low frequency page outs would occur as inactive memory started being swapped out (for our system, on the order of 10s to 100s per second). Actual swapping would result in 10-to-100 times more page outs per second. We ended up picking 1000 page outs per second as our cutoff point to differentiate between getting inactive memory out, and actual performance-impacting swapping.
We didn't have much luck pushing Active memory to 100% of total, but that's largely due to the shape of the work we're doing (we become CPU capped first). We decided that for our workload, anything more than 75% of memory Active meant that we should no longer start a new process, though our load limits and rate of spawning new processes are typically the thing actually throttling our process creation, so this comes with heavy caveats.