We have an ASP.NET MVC site hosted on two VMware ESX hosts. Each host has two sockets with quad core CPUs (8 logical cores per host). We have two VMs on each host. Initially, only one vCPU was allocated to each host.
We increased the vCPUs per host to 2, and then 4, and at each stage have measured an average of 30% throughput increase in our load testing. The application is CPU-bound - there is not much caching (RAM) and very little disk activity.
I am wondering if we should expect different results if we have more VMs with fewer CPUs. I've been reading a bit about how ESX schedules vCPUs and it appears that with more vCPUs and fewer VMs, the scheduling overhead may be holding us back.
Should we go with, for example, 4 VMs with 2 vCPUs? Exactly what resource bottlenecks are we trading by adjusting VM count versus vCPU count?
I think you'd be better with fewer VMs with more vCPUs - not all applications grow like yours, it's quite a luxury in fact, I wish mine did that :)
Basically don't worry about vCPU scheduling until you start seeing an actual problem, then it gets complex.