What is recommended storage solution as VM count grows? We currently have 5 VM servers (8 core, 32GB) and one storage server. Currently we have ~80 virtual machines running and the storage server's disk IO is getting overwhelmed and VM performance is getting degraded.
We're thinking about moving to sheepdog for both redundancy and ability to scale. We don't need super fast virtual machines as most are used for development instances of production machines or intranet sites. Some info on sheepdog scaling and performance in this slide deck: http://www.osrg.net/sheepdog/_static/kvmforum2010.pdf
Other option is increase the number of disks in our storage server, add a physical raid device, and add a read/write cache. (although then i'd have to power down our entire virtual infrastructure to do a hardware upgrade this server...).
What is best practice to allow scaling up to say ~150 virtual machines?
Edit
After Chris's comment, I started pouring over the sheepdog mailing lists, and they also don't recommend for production. Shows how I need to go over fine print more...
Looks like I'll need to do more reading on storage solutions and make some purchases to upgrade ours.
Currently we have Dual Quad-core AMD Opterons 2 disk mirrored set for OS 6x2TB disk software raid for data (I think Raid 10)
Sheepdog looks like an interesting idea, though they seem to really be gunning for using "cheap" storage instead of doing a reasonable comparison of requirements and potential solutions (makes my skin crawl when people talk about price before requirements).
At any rate the website specifically says "This software is experimental and developing software." As in do not use in production unless you're ready to accept that it might hose itself at any moment and we'll not be responsible for your decisions.
Maybe someday, but not today, tomorrow, or anytime real soon.
Please describe your current storage solution. I'm assuming it's a Linux server running software RAID 5 with less than 10 SATA disks... Have you measured your IOPS requirement or count at full load?
NexentaStor is my choice for VM storage. But in general, you should engineer a new solution that's intended to scale and accounts for your anticipated workload. It sounds like you're reacting to a crisis rather than simply looking to add a bit of performance headroom. On the NexentaStor side important design factors include: intelligent write caching, working set data size (how much HOT data is there?), storage tiers (VM data in RAM -> SSD -> fast disks -> slow disks) and niceties such as snapshotting and replication.
There are also reputable storage partners out there who can help tailor a solution to your environment.