The IIS worker processes are taking lot of memory on our servers. I want to limit the memory each application can use. I am confused whether I should set a limit on Virtual Memory Limit, or Private Memory Limit. Each application in our IIS is on its own application pool.
If I set private memory limit to 500MB and virtual memory limit to 3GB. When does the application pool recycle? Does it recycles after reaching 500MB or after reaching 3GB.
IIS will respect both of those limits.
If you set a 500MB private byte limit, as soon as a worker process attempts to commit 501MB, IIS will spin up a new worker process and kill the old one.
If you set a 3GB virtual memory limit, as soon as a worker process attempts to reserve 3.001GB, IIS will spin up a new worker process and kill the old one.
If you are on a 64bit platform, you should be aware that ASP.NET applications aggressively reserve virtual memory. As an example, I have an app on a farm that uses only 88MB of private bytes, but its sitting at 5.4GB Virtual Size right now. I believe the virtual memory reservation is a function of physical RAM on the server. It's also important to understand that on a 64bit platform, reserving large portions of virtual memory has zero performance impact.
Basically, if you are having memory consumption issues on an IIS server, the setting you want to limit is Private Memory/Bytes, this is what corresponds to actual memory usage.
I guess just to add to Dominic's older answer, on a 64bit pool you'll really want to be careful with the virtual memory allocation. I've seen the IIS worker process try to grab 22gb right away, where that was the value we were trying to set it to so the app pools crashed every minute. MS IIS best practices indicate these settings shouldn't be set, but DISA STIGs indicate you have to set them. Fun times.
https://techcommunity.microsoft.com/t5/core-infrastructure-and-security/iis-best-practices/ba-p/1241577
VS DISA V-76869 The amount of virtual memory an application pool uses must be set.
V-76871 The amount of private memory an application pool uses must be set.