Home / server / Questions / 241935
Accepted
Josh
Asked: 2011-03-02 14:13:54 +0800 CST

Multiple vCPUs or multiple identical VMs, which is better?

  • 772

I am looking to upgrade one of our servers, an 8 core server running VMware Server 2. The VMs it runs are mostly web, file, and email servers; specifically, it runs three webservers, 2 email/file servers, and a few Jabber/XMPP Servers.

When we originally set this machine up, we had the two web servers configured with two vCPUs. We had some very serious performance issues. At the time we were only running 3 VMs with a total of 5 vCPUs. We were also running VMware Server 1 and a RAID1. We have now upgraded to a RAID10 and I would never consider anything less! :-)

The new server will have VMware ESXi on it instead of VMware Server. We are looking at either another 8-core machine with faster processors, or a 16- or 32-core server. I want to make sure that I avoid the issues we had before.

Because I had such issues with multiple vCPUs before, my plan was to replace the primary web server VM with multiple identical webserver VMs (2 - 4), each with a single vCPU, serving content from an NFS share, and place these in a load-balanced configuration. SO I would basically be replacing one VM with at least 4: 2 small webserver VMs, a file server VM, and a load balancing VM. My plan here being twofold: First, I can avoid any multiple vCPU isuses I saw before, and secondly, I can spawn new web server VMs to handle increased load.

Note that I understand I am not getting all the benefits of the load balancing because I still have a single point of failure.

Is this a good plan, or is it unnecessary? Can VMware ESXi handle multiple vCPUs better than VMware Server 1 could over three years ago? Which will perform better, a single large webserver VM with 2 - 4 vCPUs, or 4 VMs with a single vCPU, in a load-balanced configuration?

load-balancing vmware-esxi vcpu smp

2 Answers

  1. Best Answer

    Can VMware ESXi handle multiple vCPUs better than VMware Server 1 could over three years ago?

    Yes, they're totally seperate product lines, and ESXi can handle multiple vCPUs very well. I'm running a 5 node vmware farm right now with a mix of machines on each box, each with 1, 2 and in some cases 4 processors as needed and it works very well indeed.

    Which will perform better, a single large webserver VM with 2 - 4 vCPUs, or 4 VMs with a single vCPU, in a load-balanced configuration?

    That's less of a VMWare question and more of a application/website testing question. VMWare (or any other modern bare metal virtual machine) can do either way - there might be a cost to having lots of vCPUs attached to one guest but this might be offset by the resource costs of having multiple guests running and by performance issues (or improvements, for that matter) your web application might realise from a more distributed approach.

    On that last issue, its really a matter of testing and architectural preferences rather than the one true way, but my general feeling is that if you can distribute the application over more than one virtual machine easily, its better to build a few smaller virtual machines than one big one. Gives you more flexibility and helps performance in the long run as you can move things about if you have to. In other words, scaling outwards rather than upwards

    With physical machines I think we think of scaling upwards, putting as much into one OS "instance" as possible to get the most out of the cost of the licence and of the hardware, but with virtual machines those equations change a bit - if you can get more than one virtual OS instance for the price of a licence then this might change the way your server infrastructure is designed.

    edit

    http://blog.peacon.co.uk/understanding-the-vcpu/ is an article Iain mentioned while discussing this question in chat that explains in some detail why its important to think about vCPU allocation when creating your VMs.

    • 7

  2. I can tackle this question in a number of ways, but I'll start of with horizontal vs. vertical scaling.

    Horizonal scaling

    Pros

    • Can scale very large
    • Can scale across hosts (ie a vmware cluster)
    • Fault tolerant
    • Low/No downtime during maintenance

    Cons

    • More overhead from VM OSes and Load Balancers
    • More maintenance/management (eg patching, SSL certs)
    • More website version management across web servers
    • Additional testing and development may be required for web site behind load balancer
    • Licensing?

    Vertical Scaling

    Pros

    • Scales easier (just add another vCPU or two, or more memory)
    • Only one web server to manage, license, etc.

    Cons

    • Downtime during maintenance/upgrades
    • Scaling too big can cause performance issues in a VM.
    • Doesn't truly scale.

    So, both have their goods and bads, and I am a big fan of horizontal scaling, but here's what I'd recommend to you:

    1. Scale vertically to a certain point (I'll get to that certain point in a minute).
    2. Do your homework on what it is going to cost time and money wise to get going horizontally.
    3. Give yourself plenty of time to test your horizontal scaling solution.
    4. When you've reached the point where it's time to scale horizontally, you can ramp back down your vertically scaled server, make some clones, throw them behind your already configured load balancer, and watch it rock and roll.

    Okay, so what is that certain vertical scaling point?

    It depends on your hardware.

    VMware has done a lot to vCPU scheduling since ESX 2.x, and there is more room to grow vertically. How much really depends on the CPUs you get.

    Sizing VMs for performance nowadays requires understanding of NUMA (Non-Uniform Memory Access. Basically, you don't want a VM to be bigger than your CPUs NUMA node. Here are some links to help guide you in that area:

    http://frankdenneman.nl/2010/02/sizing-vms-and-numa-nodes/

    http://frankdenneman.nl/2010/09/esx-4-1-numa-scheduling/

    http://frankdenneman.nl/2011/01/amd-magny-cours-and-esx/

    I guess I'll wrap it up by saying that CPU has never been too much of an issue for me, unless you are oversubscribing CPU on your host. Instead, memory is normally the culprit. Ballooning/swapping memory os a very noticeable performance issue.

    Good luck!

    • 4