With a 4Gb RAM server and Windows 2003, I've determined by trial and error that setting a virtual RAM limit of 1.6 seems to kill the runaway processes without causing the normal processes to restart unnecessarily.
Support suggested a formula of 1.9 - all running processes = virtual memory limit, is there any theory behind this?
in theory the value should depend on the website/application running on the app pool
I think that the reason to use lower limits is the overlapping recycling that iis does. During the recycling there is a period of time that the old process will run side by side with the new process. You have to consider how this will affect the total memory usage of the server.