We've been having Out of Memory issues with our mixed classic ASP/.NET site for quite some time. Basically, we see a couple of different cases:
- Some request (SQL query that returns a LARGE recordset) happens that causes memory to drastically spike (Private and Virtual Bytes), and the IIS worker process crashes, and the application pool recycles.
- Private Bytes remain 'normal', but Virtual Bytes is at its max (~2GB), and the IIS worker process crashes, app pool recycles.
Crashes have the EventID's: 1009, 1011, & 1013.
In both cases, ASP reports Out of Memory messages in our application log.
If we truly are 'out of memory', why is that users ALREADY in the application can still work and new users are blocked from even logging in? Wouldn't that mean that NO requests could be served?
Obviously, IIS can still serve some requests -- so the answer is probably that it depends on the specific request. Maybe one is more 'expensive' than the other?
In any case, my ultimate question is -- at what point does the application pool say enough is enough, I MUST recycle?
We ran DebugDiag and saw some heaps that were over 90% or even 100% fragmented, so does the recycle happen only when fragmentation is high AND there is no more heap space?
This is a slightly subjective question but I hope I can get some input on sort of what that threshold is!
Thanks.
Unless the worker process breaks one of your configured recycling criteria (virtual bytes limit seems like a good one?) or is failing to respond to pings, it's not recycled.
Apps (ISAPIs) can report themselves as unhealthy, which triggers a recycle, but that happens under fairly narrow conditions.
Your app is basically fragmenting memory while it runs, and the OOM reflects this - no free contiguous, unallocated memory is available for new allocations; they fail.
Your options based on the description are:
fix the allocation pattern
separate the parts of the app into separate App Pools (run .Net separately from ASP, for example) - not all apps can be broken up in this way, but it's an easy win for those that can. Two App Pools = 2 (or more) worker processes = 2GB per worker process address space
implement a virtual bytes limit to recycle the app pool when it grows too big
implement a daily recycle to prevent the app getting to the point where its entire memory space is fragmented
if your app is stateless, try web gardening; increase max # worker processes in order to stave off failure for longer
Run as a 32-bit app on a 64-bit system; 4GB is provided to worker processes.
Essentially, if (whatever) heap's become super fragmented, you're in a death spiral at that point and you need a new worker process.
IIS doesn't have inbuilt safety belts for this, so implementing a recycling limit on your process before it gets to that point - or just saying "forget that, I'm going 64-bit!" might be possible solutions.