I have heard the old advice for Windows NT 4 which, if memory serves, was a rule of thumb "memory size plus 12 MB" (so that a complete dump could be written to disk).
Does anybody have any recommendations for Windows Server 2003 and 2008?
(Also, while we are at it, what is a good swap file setting for a VM? Like physical machine or should I give more memory to the VM and let the host do the swapping?)
About your virtualization question - never let the host do the swapping. The problem is that when the host does the swapping instead of the guest, then the guests can't tell why performance is suffering. There's no indication on the guest's performance metrics that their memory is being swapped by the virtualization host.
That's not to say you should avoid overcommitting on memory - that's okay as long as you manage VMware's memory sharing correctly. VMware basically does deduping on memory to reduce the amount actually needed. Hyper-V doesn't do that yet.
How to determine the appropriate page file size for 64-bit versions of Windows Server 2003 or Windows XP
Another good discussion:
Pushing the Limits of Windows: Virtual Memory
Start of with 1.5x your physical memory, then measure to adapt the figure to your workload
Use 1.5x physical memory for machines with less than 4-6GB RAM. After that, you can reduce the size or at least not increase it. Watch paging to determine if increases are needed. Also, use a fixed page size. This will pre-write the page file an eliminate performance hits when the page file needs to be increased.
As for working with VM's. Monitor your paging in the VM and adjust as appropriately to maximize performance. This will require using the performance monitor.