I am running VMWare ESX 4.1.0, 260247 on an intel DX58SO motherboard with a HighPoint 4310 raid controller.
The server has 2 velociraptor drives in RAID 1 and 2x WD black 2TB drives in RAID 1 as well.
The rig has been working properly for over a year, but yesterday we started noticing extremely bad disk IO on the vm's at random. Regardless of which raid the vm was stored in, we recorded 1MB/s reads.
If we restart the vm, performance goes back up to 80MB/s on reads, but in a matter of hours it goes back down to 1-3.
We've tried just leaving 1 vm on, and still experience the same issue.
I've ran into this problem before with ESXi as well, and have never found a solution. It seems to happen over time.
Thanks in advance for any feedback.
I've encountered a similar issue where the root cause was insufficient available RAM, forcing the host machine to make heavy use of swap space, and as a result causing the disk IO speed to drop heavily. Keep in mind that you should have around 1Gb available RAM for the hypervisor, in addition to the RAM required for the individual virtual machines.
Based on this VMWare Communities thread I've added this to my vmx files:
After starting the guests, performance seems to be back to normal.