What are the performance and stability implications of running resource intensive services in a hypervisor vs guest OS?
I want to set up a workstation with both Linux and Windows, hosted by (probably) KVM. Both will use some of the same services: For example, the VMs will be stored on zvols and share a zpool devoted to file storage.
I am hung up on deciding which services to run in the hypervisor OS, and which to run in a VM. If I put services like ZFS in a VM, I am worried that they might be slower (as they will have to work via the hypervisor) or more error prone, as they will depend on controller pass-through (one more thing to have problems with).
On the other hand, I am worried that burdening the hypervisor with too many extra tasks will cause problems, and I am also worried about how much memory ZFS will consume if I don’t put it in a virtual machine with defined limits (I’m thinking to devote 32 Gb to whichever OS is running ZFS. And I’m worried that having too much stuff installed in the hypervisor will make conflicts which affect the whole system more likely (as those conflicts would occur inside the hypervisor).
Hypervisors, regardless of the flavor, should be 100% dedicated to serving its VMs. This is for a few reasons.
First, one of the key benefits of virtualization is hardware independence - if a host dies, it's very simple to restore those VMs to a different host and be up and running in a very short period. As soon as you start deploying services directly on the host OS, that process becomes much more complicated.
Second, hypervisor kernels are tuned to deal with virtualized load, not userland processes running directly on the host OS.
Third, process configuration and deployment becomes overly complicated if you have to consider processes running on VMs and processes running on bare metal. This isn't insurmountable, but it isn't ideal and should be avoided if possible.
So to answer your question directly: don't run any unnecessary processes directly on the host OS. Let the hypervisor do its thing, and use VMs for your services.