I administer a Windows 2008 server (well, on Amazon EC2) running IIS and a .NET4 Web app. I got a memory alert the other day and went and looked, and sure enough the process memory had grown over time via some kind of slow leak. It didn't grow by much, just like 60M to 200M, but enough else was going on with the box that it went over our pretty low threshold (75%) to set off the monitor.
I recycled the app's pool and the memory freed up, and I noticed upon reviewing stats that swap space was being used significantly and that more than 1 GB of it freed up with that recycle.
Maybe this is a basic question, but I'm a UNIX guy and I'm used to swap not getting used until you're out of memory. This box has never gone above 75% memory usage. Is this a Windows thing or a .NET thing or an Amazon thing? I suspect that there's a much larger memory leak in this app than suspected - it's not leaking from 60M to 200M, it's leaking from 60M to 1.2GB, but much of that is somehow going "cold" and being pushed out to swap?
I have memory recycling set on the application pool, but it triggers off box full memory, so this app could get really, really big before it recycles automatically.
I could set up regular "timed" recycling, but that's a workaround, I'll get the dev to fix the app but need to understand what's going on here with the swap usage to make sure I am understanding this right.
Edit with more info: instance memory: 1.7 GB swap: 4.5 GB
I see the w3wp.exe process in taskmgr showing that Memory: 211,000k. But when I restarted it (it's in its own app pool, and it's the only app on the box), its memory usage went down to its normal starting point of 60M and like 1 GB+ of swap also freed up. In taskmgr I just had the usual Memory (Private Working Set) stat up, but saw the swap change via my other monitoring (Cloudkick). Going back and looking at it today, memory is back up to 195M on the process (1.2 GB total) and swap has crept from 1.0 GB to 1.1 GB, buyt not all the way back up where it was (graphing over time, it's a slow creep).
I'm less concerned about this specific app and more concerned about just understanding when Windows swaps and how it uses that, and what to be concerned about given Windows memory and swap usage in general.
Windows and linux have two different page/swap strategies.
Linux
Linux wants to avoid using swap space at all, and waits until the last possible moment. If you see a large amount of swap in linux, your system likely is or was in trouble. This strategy is good for minimizing overall disk i/o, which is the slowest part of your system, but weaker for systems with alternating periods of light and heavy load (and honestly, that's most of us). The times when your load is already heavy will now be burdened by the "extra" disk i/o, or, put another way, you need to design your server builds with an eye to having enough ram you don't swap even during the highest expected load times.
Windows
Windows wants to treat memory as a mere cache of the page file. Your real memory is always on disk, but it will read/write from the "cache" first if it can. This strategy is good for evening out your load over time; when the system gets busy and needs to swap pages, the current page is already on disk and half the work is already done. This approach made huge sense back when Windows was young, 32MB (forget GB) was still a lot of RAM, and the frequent need to use swap space was a given. Even today this is good for work-loads that alternate between light and busy loads, as it helps spread the disk i/o out more evenly over time.
Modern Windows versions have additional optimizations — such as SuperFetch — to pre-load and prepare memory pages on disk and in RAM when the load is otherwise light, to help avoid the need for extra disk writes when loading a program for the first time. All of this means you can design your system to only need enough RAM for something less than the highest expected load, so you can still have at least acceptable performance all the time, with reduced costs.
Convergence
This concept of measuring or predicting load in a test environment first and then allocating production resources when the load is known is a relatively recent development in system building, made possible, or at least practical, in part with the advent of virtual and then cloud servers. Depending on your load, you may even design the system such that it never needs to swap at all. In these cases, Windows does allow you to turn off paging and behave more like a linux system. However, you have to be careful; if your system design requires more memory than expected you can get yourself into trouble this way.
On the other hand, modern linux kernels are more willing to swap to disk opportunistically than they once were. So the difference in memory management strategies between the two systems is still present, but now less distinct than it used to be. Both systems have their merits, and each watches the other to see which advances they can copy.
Windows (and Linux & other Unix-a-like OSs) will move pages that have not been used for some time to disk to make room for buffers and cache to speed up active I/O activity. Also applications will often allocate more memory than they are immediately going to use - this may encourage the Kernel to page some things that have not recently been touched in the background, so that said applications don't see the paging delay when they do suddenly start using that allocation.
Under Linux you can tweak (or block) this behaviour by altering the relevant "swappiness" values in the
/procfilesystem - no doubt there are registry values you can tweak to alter how Windows behaves in this regard too.Another thing to be aware of is that when something has been paged out and later read back in, the kernel will not remove it from the page file until either the file is otherwise full or the page in RAM is altered. This way if it needs to page that chunk out again it can do so without having to actually write the pages to disk: the content is already there. This can greatly improve performance in situations where memory over-commit has got so bad as to cause thrashing of the page files (significant numbers of pages constantly being mapped in and out). You are likely to find that some data was pushed out by that memory leak, and has since been read back in but not wiped from disk in case those pages need to be mapped out or RAM to make room again later. Under Linux, the "SwapCached" value in
/proc/meminfoshows how much data is present in pages that have identical copies in RAM and on disk. Windows no doubt uses the same optimisation (or something similar) but I'm not exactly where to look to see how much this is happening (no doubt there are relevant performance monitor counters you can query).tl;dr: This is normal. A modern OS Kernel will try to be clever and maximise how much RAM it can use as cache to save I/O operations, and will sometimes have data copied in disk and in RAM to save I/O if it needs to page those bits out of RAM later. Chances are, counter intuitive as it may be, that this use of page files in these two ways, even if you are not currently low on RAM, is improving your overall performance rather than decreasing it.