Home / server / Questions / 43565
In Process
Brian Spolarich
Asked: 2009-07-22 06:58:32 +0800 CST

Ubuntu 7.04 VM Runs Out of Memory

  • 772

I keep a Ubuntu 7.04 VM running at Linode.com to host a few web sites, e-mail lists, etc. It was working flawlessly for a long time. I don't patch it or really pay attention to it.

Recently (past few months) and with increasing frequency, the box runs out of memory and I have to reboot it. Its configured with 350MB real memory, plus 64MB swap. It runs Apache, Mysql, Postfix, and mailman.

The out-of-memory symptoms are obvious and I see these in the kernel.log:

Jul 21 10:16:42 grendel kernel: Out of memory: kill process 30364 (apache2) score 11205 or a child
Jul 21 10:16:42 grendel kernel: Killed process 30364 (apache2)
Jul 21 10:16:42 grendel kernel: apache2 invoked oom-killer: gfp_mask=0xa01d2, order=0, oomkilladj=0

Is there a silver-bullet method to determine what's going on here? I'm watching 'top' and while I see the available real memory decline over time I don't see an obvious culprit.

ubuntu outofmemoryerror

4 Answers

  1. Perhaps configure a cron job to mail the output of something like ps auxh | sort -r --key=4 | head -5 to you every 30 minutes or so?

    That will let you know the top 5 consumers of memory on the thing, which may help track down the culprit process. The 4th column is the percentage of memory consumed by the process.

    You might be able to spot a trend over time.

    • 2

  2. In top, try commands F o to sort by virtual (total) image size. Then you'll see the largest processes up on top.

    • 1

  3. If the Apache processes are getting large, for instance if you are running PHP scripts via mod_php that work on a fair chunk of data at times, you might not notice any one process growing massively (but every Apache process growing a little as time goes on). If this is the case then you could try the following:

    • limit the number of child processes Apache uses by tweaking the options:
      • StartServers
      • MinSpareServers
      • MaxSpareServers
      • MaxClients
    • consider using mpm_worker instead of mpm_perfork if everything you do with Apache is thread-safe
    • ensure that KeepAliveTimeout is not too high (this stops an errant client holding a processes for too long, especially important if you have drastically limited the number of children
    • decreasing the maximum time-to-live of each process with the MaxRequestsPerChild option will help if the problem is due to a memory leak, though may make your service noticeably less efficient if you pair it back too far
    • 0

  4. You can also remove the requirement for reboots by installing swapd which will automagically create swap files in case of low memory.

    • 0