Home / server / Questions / 26932
Accepted
maff
Asked: 2009-06-17 15:42:23 +0800 CST

Best way to benchmark linux virtualization solutions

  • 772

I'd like to run some benchmarks on linux virtualization solutions (namely Xen, KVM and OpenVZ) and I'm currently thinking about how to set up the test environment. My current approach would be the following:

  • Set up a bare server system
  • Install all 3 solutions (kernels) on the same system and set them up (+ VMs)
  • Boot a specific kernel (disable eventual services from other solutions) and run benchmarks

Would this approach be OK or should I completely separate the 3 systems from each other (have 3 completely independent installations each one with only one solution installed)?

Additional question: any hints what kind of benchmarks I should run? I thought about some compiling (e.g. linux kernel), ab (Apache Benchmark) and OSDB (Open Source Database Benchmark). The comparison should mainly focus on overall performance, I/O performance and response time of individual VMs.

Thanks in advance, Mathias

virtualization xen kvm-virtualization benchmark openvz

3 Answers

  1. I'd read up on the benchmarking Anandtech.com has been doing on virtualization:

    And read up on why VMware doesn't allow benchmarks in their EULA. They give some really good examples of bad benchmarking:

    http://blogs.vmware.com/virtualreality/2009/03/a-big-step-backwards-for-virtualization-benchmarking.html

    • 3
  2. Best Answer

    Testing should be done as close to how you'd run it in production. So install each system separately and do the testing. This may require flattening your testing box several times, but it will give the cleanest results. Then focus on creating your test cases. Some off the top of my head:

    • Multiple VM test Run several VM's on the same host.
    • Busy VM resource contention Have one VM run something highly CPU demanding, run other tests in other VMs to see how they're affected by it.
      • Do the same, but with a high disk I/O VM, and see how other VMs are affected.
    • Multi-processor efficiency If you have the hardware for it, run VM's with more than one virtual CPU and judge efficient use of it (say, kernel compile with multiple jobs).

    In essence, brain-storm the edge cases and build tests around those cases. What happens when a VM gets piggy with resources? How well does the hypervisor handle over-subscription of resources? How does one VM locking up hard affect other VMs on the system?

    • 2

  3. Strongly suggest performing the tests with only one hypervisor set up in the test environment for each iteration of your tests. Isolating the hypervisors in this fashion will reduce the likelyhood of unwanted interactions; especially for para-virtualised VMs.

    Think of desiging an additional suite of tests that will find the breaking point of the hypervisor for each of your limited resouces:

    • CPU
    • Memory
    • I/O Bandwidth

    What you want to find is what behaviourial idiosyncrasies are experienced before & after the pathological point; and if there is some form of recovery for some of the host VMs.

    For example, when the hypervisor is overloaded; are you still able to shut down VMs.

    • 0