I have a WCF service that might grow in memory. I put a memory limit on the private bytes to recycle the app pool after 500m of memory usage. I noticed that my w3wp.exe process can get to 600megs when it decides to recycle and any currently executing client requests get a communication error. Is there a way for iis to wait for requests to complete before recycling the process?
Recycling is killing an process when it reaches certain conditions, replacing it with another one.
In terrible-analogy-land, it's a bit like ensuring that the size of a balloon can only ever be one foot square by having pins positioned in a one-foot sphere - when the balloon hits the limit, it pops and you replace it with a new one.
Processes being recycled get up to {Shutdown Time Limit} to shut themselves down, from the point at which they're told to recycle.
If they can't finish their work in that time {90 second default}, that's it.
If you're using a WCF host with a persistent connection, then I think the answer's no; but the answer is also "pick larger limits". What you've described as your policy is: "I want my app to die when it hits 500MB." But the app itself doesn't know about that limit; it's just trying to grow as it needs to.
Load test your app without a limit; see what it gets to and what might indicate a leak; set limits at the point a leak is indicated.