Home / server / Questions / 132088
In Process
Boaz
Asked: 2010-04-14 11:43:37 +0800 CST

What can cause an increase in inactive memory and how to reclaim it?

  • 772

I have heavy application running on a CentOS server and I'm seeing a strange memory behavior. Here is a snapshot of a munin graph: alt text

As you can see the amount of committed memory increases gradually causing the swap file to be use. What strikes me odd is that the amount of inactive memory keeps growing as well. It is my understanding that the inactive memory is actually memory freed up but not yet clean by the OS and put back in the free memory pool. It seems that running out of memory is acutally caused by this lack of clean up, but I may be wrong.

Can you give some tips to find the cause of the problem and/or cause CentOS to reclaim the inactive memory?

Thanks.

Some extra info:

1) I have a tmpfs mounted on /tmp and the number of files stored there grows (but it is double the amount of the inactive memory).

2) cat /proc/meminfo (at a later stage than the image) gives:

MemTotal:     14371428 kB
MemFree:       1207108 kB
Buffers:         35440 kB
Cached:        4276628 kB
SwapCached:     785316 kB
Active:        9038924 kB
Inactive:      3902876 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     14371428 kB
LowFree:       1207108 kB
SwapTotal:    10223608 kB
SwapFree:      6438320 kB
Dirty:          627792 kB
Writeback:           0 kB
AnonPages:     7844560 kB
Mapped:          49304 kB
Slab:           146676 kB
PageTables:      27480 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  17409320 kB
Committed_AS: 16471488 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    275852 kB
VmallocChunk: 34359462007 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

3) The application is a combination of MySQL, Heritrix (http://crawler.archive.org/ ) and a Tomcat based Java servlet to manage things.

linux centos memory tmpfs

2 Answers

  1. It is my understanding that the inactive memory is actually memory freed up but not yet clean by the OS and put back in the free memory pool.

    This is false. 'inactive' memory is actively mapped memory which has not been utilized by any application for some time. When its time to swap, memory is taken from pages marked like this and swapped out. It can also be used to swap out in favour of page cache.

    As you can see the amount of committed memory increases gradually causing the swap file to be use. What strikes me odd is that the amount of inactive memory keeps growing as well.

    The two dont necessarily correlate, but to me this strongly looks as if something is leaking memory. The fact that you have pages not being accessed by any applications growing, and swap growing too suggests something is allocating memory, forgetting about it then not freeing it afterwards.

    Memory could be 'inactive', for example if malloc() is called. This is a libc call that may allocate a chunk of memory, but only a portion of it is actually utilized to do any work (less than the number of pages allocated anyway). Even if you free in malloc it doesnt actually mean you free the memory by asking the operating system to do so, its just mallocs tables might mark is as 'reusable', it might free it after.

    • 5

  2. what is this application, and does it have a memory leak? i.e if your application terminates, does the memory get returned to the operating system?

    If you application is leaking memory, you need to use something like electric fence to find those problems and correct them.

    What is you application written in? you need to use the correct debugger for the language it is written in.

    I very much doubt that the operating system is leaking memory, unless you application is using up a lot of resources and not freeing them correctly.

    Also, your memory graph is very very busy and very hard to comprehend, there's about 20 colours going on there, it's about as clear as mud.

    Do you think it's a problem with you ram disk? it's probably expanding to meet demand, however it's not scaling back when needed.

    • 0