IIS 6 + ASP.NET web service - DW20 and stackoverflow exception

The resolution to this (likely unique) problem:

• remove and recreate all app pools (likely this was overkill and unneeded)
• delete the applications files on disk
• redeploy and new build & version of the app
• ensure all references are included in the bin directory

Stackoverflow exceptions are a special case, because the affected application can't do anything anymore (e. g. logging a stack trace) - in this case, the application pool process (w3p.exe) is terminated by the OS. That's why Dr. Watson/DW20 gets involved. You could try to debug the dump that DW20 saved using WinDbg with SOS extension (expect a steep learning curve if you're not familiar with that tool set - I hope this becomes easier with VS2010 as promised).

The high CPU usage (and often high memory usage) is caused by DW20, which is especially annoying if the "crash-and-restart-loop" is faster than DW20 and so several DW20 processes accumulate.

The default IIS application pool setting is to restart crashed applications no more than 3 times in a short period of time, otherwise they will be stopped to protected the server from DoS.

Regarding the root cause, the stackoverflow: Could be everything... but how about this wild guess: Database access is failing due to misconfiguration, exception is generated, and your application is logging exceptions to the database, without catching exceptions on exception handling ;)

I had this issue and found out that I had a LINQ statement that was trying to delete some rows. It was failing every day so the amount of rows just kept going up and up. The exceptions were actually handled and logged to a table that I had made so I found it there. Found the problem table and it was 700k+ of super heavy rows. My LINQ looked like this:

var db = new DatabaseDataContext();
        var updateQueueLogs = db.UpdateQueueLogs;

        List<UpdateQueueLog> listToDelete;

        using (new TransactionScope(
            TransactionScopeOption.Required, new TransactionOptions { IsolationLevel = IsolationLevel.ReadUncommitted }))
        {
            listToDelete = (from updateQueueLog
                              in db.UpdateQueueLogs
                            where updateQueueLog.CreatedAt < DateTime.Now.AddDays(-7)
                            select updateQueueLog).ToList();
        }

        updateQueueLogs.DeleteAllOnSubmit(listToDelete);

        db.SubmitChanges();

So it was pulling the full objects and getting OutofMemory exceptions. I changed this code to a stored proc:

delete from UpdateQueueLogs
 where CreatedAt < DATEADD(day,-7,getdate())

I just wanted to post this because if you are not seeing the exceptions you might want to check some of your SQL table row counts and see if there could be some LINQ statements that are timing out.

It sounds like you may be getting an unhandled exception which in turn kills the asp.net worker process. See if this helps:

http://support.microsoft.com/?id=911816

Check in the Event Viewer for Errors (typically Application log). It is under Administrative Tools.

Also, @markus's good point of IIS having pretty hard default "no more than X thread crashes per Y time"-setup, so that if you hit the page just a couple of times with such an error, the entire Application Pool is taken down. Again check Event Viewer.