I've inherited a bunch of hardware from a failed project. They're SuperMicro 2U servers with 24x SATA HDD or SSDs. I've been tasked with turning this rack of generic hardware into a IaaS cluster. I'm an old hand with Xen and the show-stopper bug for us with KVM has been fixed, so I'm good to go from most aspects.
Looking over the documentation, I'm not sure right now if I can run GlusterFS (or Nimbus/Cumulus or another cloud storage layer) on the same hardware as the VM Host. I'm trying to find out if anyone has any experience building out with this kind of configuration -- since I don't have separate storage nodes, will it be possible for me to make use of this hardware fully?
Any other suggested solutions, such as doing hardware passthrough to a dedicated storage virtual machine on each node?
I'm quite sure those file systems need the boot loader if running on the physical machines which would need to be used by the hypervisor.
I use xen and gluster but never using VMs to run the distributed filesystem to run VMs on the same hypervisors. I think the problem there would be mounting ISCSI while the VMs running gluster might not be able to initiate providing the hardware pass through. It might work if the only VMs to load from a physical cold boot is the VMs running gluster, manually mount ISCSI and then manually boot VMs attached to gluster LUNs. Should be scriptable.
What are you trying to achieve with Gluster on the VM Host?
Gluster is a FUSE Filesystem. That means it runs on top of whatever other filesystem and only proives Network functionality.
If HA Disk replication with VMs on top is what you want, then Gluster would be the wrong choice. There are things like drbd for that. If however you need a shared filesystem for VMs to mount it may be a good option depending on what exactly you're trying to do.