I'm doing some benchmarking with IOMeter, and I'm seeing a consistent and susbtantial drop-off in performance after running the first test in my VMWare environments. These drop-offs are about the same on the three VMs I've tested on, which makes me think it's a configuration setting, or something about my VMs.
For example, one system (local RAID 10) went from 388 I/Os per second the first run, to about 211 I/Os per second on every run after that. Everything else about the test was identical, and I also bounced the machine in between runs. Testing on my local machine in a non-virtualized environment is consistent, whereas if I test it with a local VM, I also get a performance drop-off after the first run.
Any explanations or suggestions to explore?
Are you using the entire VM drive for the test? On a fresh VM when you run your test you are starting with no allocated data. There is nothing the test needs to delete to write. On your subsequent test, you are likely having to delete, then write, which is going to cause the second test to be slower.
I do performance testing on Hard Drives at EMC. Before every HDD or SSD performance test we run, we use iometer to do whats called "dirty the bits" in the drive. Of course brand new just-initialized drives are going to run faster; none of their bits have ever been accessed, and thus everything is zeroed out and better performance results are obtained. IN order to achieve results unbiased to newly initialized drives, the bits are dirtied and then the performance test is run immediately with no idle time in between. (During idle time, the VD will usually do something along the lines of "garbage collecting" clearing out all of the dirty bits [This is why you run your tests, and the first run is always better, and then after the rest of your tests have run, idle time occurs and the drives clear and everything is back to normal and the loop continues]).
Run a 512 Read on 1 worker for an hour in one test, and then run Worker 1 - 1024 read, Worker 2, 2048 read, WOrker3 - 4096 read, for 5 hours. These two tests consecutively=how to dirty your bits prior.
[Sometimes for RAID 5's over 1 TB you need to adjust the tests to run over 24 hours - i.e. test 1 4hours, and test 2 for 20 hours)
Can anyone point me to something to help visualize/parse my Iometer results? please email