Home / server / Questions / 423828
Accepted
Martin
Asked: 2012-09-04 02:41:51 +0800 CST

How to find virtualization performance bottlenecks?

  • 772

We have recently started moving our C++ build server(s) from real machines into VMs. (MS Hyper-V)

We have some performance issues that I've currently no idea how to address. We have:

  • Test-Box - this is a piece of desktop workstation hardware my co-worker used to set up the VM before we moved it to the actual server hardware
  • Srv-Box - this is the server hardware
  • Test-Box-Real - This is Windows running directly on the Test-Box HW
  • Test-Box-VM - This is Windows in a Hyper-V VM on the Test-Box HW
  • Srv-Box-Real- This is Server2008R2 running on the Srv-Box HW.
  • Srv-Box-VM- This is Windows running in a Hyper-V VM on the Srv-Box HW, i.e. on Srv-Box-Real.

Now, the problem is that we compared Build times between Test-Box-Real and Test-Box-VM and they were basically equal (within about 2%).

Then we moved the VM to the Srv-Box machine and what we saw there is that we have a significant performance degradation between Srv-Box-Real and Srv-Box-VM, that is, where we saw no differences on the Test HW we now do see major differences in performance on the actual Server HW. (Builds about ~~ 50% slower inside the VM.)

I should add that both the Test-Box and the Srv-Box are only running this one single VM and doing nothing else.

I should also note that the "Real" OS is Win2008R2(64bit) and the VM hosted OS is Wind2003R2(32bit).

Hardware specs:

  • Srv-Box:

    • Intel XEON E5640 @ 2.67Ghz (This means 8 cores with hyperthreading on the Real system and "only" 4 cores on the VM, since Hyper-V doesn't allow for hyperthreading, but number of cores doesn't seem to explain the problem here.)
    • 16GB RAM (we have 4GB assigned to the VM)
    • Virtual DELL RAID 1 (2x 450GB HUS156045VLS600 Hitachi 15k SAS drives)

  • Test-Box:

    • Intel XEON E31245 @ 3.3GHz
    • 16GB RAM
    • WD VelociRaptor 600GB 10k RPM SATA

Note again that I'm only concerned with the differences between Srv-Box-Real and Srv-Box-VM (high) vs. the differences seen btw. Test-Box-Real and Test-Box-VM (low).

Why would one machine have parity when comparing VM vs Real performance and the other (server grade HW no less) would have a large disparity? (Both being XEON CPUs ...)

virtualization

3 Answers

  1. Start with the obvious - monitor performance counters, if possible using Ressource MOnitor. How is CPU, IO handling? The standard" bottleneck is pretty much IO, and your 2 drives sound like quite bad IO for anything that has more then 2-3 VM's, especially when something brutal as a build server for C++ hits it (that is HEAVY IO based, compilers in generally are ,but C++ is quite extensive on this area - I use a SSD just for the build).

    This is standard Admin - nothing special about a VM at this point. Basically to find the bottleneck you first find it.

    IO may by also - dynamic discs. Helps to put in a non dynamic disc ONLY for the build step (work files). There is a TON of IO happening there.

    Now, yes, a Raptor has less IOPS than a 15k SAS, but then it is likely NON SHARED.

    Does SRV-HW use up all CPU cores? Another item, then...

    Regarding your CPU limit, btw., the CURRENT version of Hyper-V supports a LOT more cores than 4 per VM, so an upgrade may be in order... Server 2012 is available for pretty much everyone not buying a box in a shop right now, I got my copy 16th of August from the MS repositories.

    But first you do normal operational step by step - find out where the bottleneck is by using ressource monitor and performance

    • 2

  2. I had a similar problem in the past, we tracked this down to write caching Disable it on your physical server and give it another go.

    The other thing to keep in mind is your simply battling a lot of differences,

    1. VM
    2. Different memory
    3. Different disks (VHD vs Physical)

    Eliminate as many as you can and try again.

    • 1
  3. Best Answer

    Check the BIOS settings wrt. performance. (We changed it from "Custom" to "Full Performance".)

    We found out that on this server, the system did not detect full load when the full load came from within the Hyper-V VM, meaning that the processor bumbled along at 1,6GHz instead of running at it's full potential. Speed Stepping did work when running on the "real" OS -- that's where the difference came from as far as we can tell.

    • 0