TL;DR: I'm having some performance issues with my hypervisor storage. here are a bunch of test results from fio. Skip to the Results section to read about them and see my questions.
Summary
I recently purchased an R730xd so before I migrate over to it I wanted to be sure that storage was performing opmtimally. I've been running some benchmark tests with fio and have found some alarming results. Using a combination of those results and fio-plot, I've amassed a pretty large collection of graphs and charts that demonstrate the issues across my various storage backends.
However, I'm having a hard time turning it into usable information because I don't have anything to compare it to. And I think I'm having some very strange performance issues.
Disk Configuration
Here are the four types of storage exposed to my hypervisor (Proxmox):
╔═══════════╦════════════════════════════════╦═════════════╦════════════════════════════╗
║ Storage ║ Hardware ║ Filesystem ║ Description ║
╠═══════════╬════════════════════════════════╬═════════════╬════════════════════════════╣
║ SATADOM ║ 1x Dell K9R5M SATADOM ║ LVM/XFS ║ Hypervisor filesystem ║
║ FlashPool ║ 2x Samsung 970 EVO M.2 SSD ║ ZFS RAID 1 ║ Hypervisor Compute Storage ║
║ DataPool ║ 6x HGST 7200RPM HDD ║ ZFS RAID 10 ║ Redundant Data Storage ║
║ RAIDPool ║ 6x Seagate/Hitachi 7200RPM HDD ║ HW RAID 10 ║ General Purpose Storage ║
╚═══════════╩════════════════════════════════╩═════════════╩════════════════════════════╝
Storage Details
Here is a more detailed breakdown for each storage backend:
SATADOM: The
SATADOMis managed directly by Proxmox via LVM. Here is the output fromlvdisplay pve. The SATADOM is connected to the server via the internal DVD-ROM SATA port as it is unused in theR730xdmodel.FlashPool: The
FlashPoolis a simple ZFS RAID 1 comprised of dual NVMe SSDs. The goal is to use this as backing storage for my VMs. Here are the outputs for:zpool list zpool status zfs get allEach of the SSDs in the
FlashPoolare connected to the server via PCI-E -> M.2 adapters installed in x16 PCIe slots. I recognize that these are x4 PCIe adapters. However, I'm pretty sure NVMe only operates at that speed so faster adapters are not manufactured.DataPool: The
DataPoolis the only pre-existing dataset. It is a couple years old and was previously used for both Data and VM storage to the detriment of performance. It is also managed by Proxmox as a ZFS RAID 10.It was originally comprised of
6x 4TB HGST Ultrastar 7K4000 7200RPMdisks. However, as they started to fail I decided to replace them with higher density disks. As a result, the array now consists of:2x 6TB HGST Ultrastar He6 7200RPM 4x 4TB HGST Ultrastar 7K4000 7200RPMI obviously intend to eventually move entirely to 6TB disks as the older ones continue to fail. Here are the outputs for the same commands posted above for the
FlashPool.These 6 disks are connected to the server via the first 6 bays on the backplane. This backplane is connected to a Dell H730 Mini PERC RAID Controller.
RAIDPool: The
RAIDPoolis an experimental storage backend. I've never worked with hardware RAID before so I was excited for the opportunity now that I have a proper RAID Controller. Similar to theDataPool, these disks are installed in the last 6 bays on the backplane. However, instead of being passed through to Proxmox, they are managed by the PERC. They are presented to Proxmox as a single disk which is then managed by LVM and presented to the OS via logical volumes as XFS filesystems. Here is the output fromlvdisplay RAIDPool.
RAID Controller Configuration
So, you may have just noticed that both the DataPool and RAIDPool are installed and managed by the H730 RAID Controller. However, the DataPool is managed by Proxmox via ZFS and the RAIDPool is managed by the actual controller.
Here is a screenshot of the topology of the physical disks. The H730 is capable of passing disks directly through to the OS and simultaneously managing other disks. As you can see, The first 6 disks are configured in Non-RAID mode and the last 6 disks are configured in Online mode.
- Here are the configured properties for the controller from within the iDRAC UI.
- Disk Cache is enabled for both Write Back and Read Ahead on the Virtual Disk (
RAIDPool). Since this is configured specifically for the VD, it should not impact the ZFS drives. - Dick Cache for Non-RAID disks (ZFS
DataPool) is set toDisable. - The Link Speed for all of the drives is set to
auto.
Also, after stepping through all of the settings once more, I enabled Write Cache for the Embedded SATA Controller. So this might improve performance on the SATADOM from what is seen in the benchmarks below.
Benchmarking:
I have benchmarked all of these storage backends in two ways. For both tests, I ran a series of fio-plot commands in a small shell script that dumped the results in a few folders.
If you're crazy and want to parse through the raw results on your own, here they are. You will need to massage my scripts a bit to rerun then since I moved around the directory structure to organize it prior to uploading it.
In a nutshell, they ran a series of tests against each storage backend that evaluated its RANDOM bandwidth, IOPS, and latency. It then plotted these results on graphs. Some graphs compare multiple backends. Other graphs simply show results from individual backends. I did not perform any SEQUENTIAL tests. In all cases, the default block size was used for the test.
Test 1) From within Proxmox, I mounted all of the storage backends into the /mnt directory. The ZFS Pool were simply imported into the OS and both the RAIDPool and the SATADOM were presented to the OS via LVM. Each had a Logical Volume formatted as an XFS partition that was used for benchmarking. NOTE: I ran these benchmarks from the live OS so the performance of the SATADOM will be affected accordingly.
The log files were generated using these commands:
./bench_fio --target /mnt/SATADOM_Data/bm --type directory --size 450M --mode randread randwrite --output SATADOM
./bench_fio --target /mnt/RAIDPool_Data/bm --type directory --size 1G --mode randread randwrite --output RAIDPOOL
./bench_fio --target /mnt/DataPool/bm/ --type directory --size 1G --mode randread randwrite --output DATAPOOL
./bench_fio --target /mnt/FlashPool/bm/ --type directory --size 1G --mode randread randwrite --output FLASHPOOL
Test 2) I created three VMs in Proxmox. Each of which used a different backing storage from the FlashPool, DataPool, and RAIDPool. The FlashPool and DataPool VMs ran in their own ZFS dataset. The RAIDPool VM ran on its own thick-provisioned Logical Volume. All three VMs were given 4 vCPUs and 40GB of memory.
The log files were generated using these commands:
./bench_fio --target /fio --type file --size 1G --mode randread randwrite --duration 600 --output DATAPOOL_VM
./bench_fio --target /fio --type file --size 1G --mode randread randwrite --duration 600 --output RAIDPOOL_VM
./bench_fio --target /fio --type file --size 1G --mode randread randwrite --duration 600 --output FLASHPOOL_VM
Results:
The graphs in the above Imgur links should all be in the same order. The results from the two benchmarks are quite a bit different. But that is to be expected when you factor in the overhead from virtualization. What is NOT expected to me, is that they all seem to behave about the same.
For example, this chart shows that when
fiowas run from within a VM, the average write bandwidth was somewhere around 125 MB/s. Shouldn't the two NVMe SSDs in RAID 1 (FlashPool) MASSIVELY outperform theSATADOM? Instead, you can see that theFlashPoolVM took the LONGEST amount of time to complete the test and had the slowest average write bandwidth. The same situation can be seen for the Write IOPS comparison -- The average IOPS were around 3,000 and theFlashPoolVM took the longest to execute the test!Stepping away from the benchmarks taken from WITHIN a VM, and instead looking at those taken by directly interacting with the storage from the hypervisor, we can see some different behavior. For example, in this test the write bandwidth for the
FlashPoolandDataPoolwas as high as 400MB/s. However, the performance for theRAIDPoolaveraged around 10MB/s. Which coincidentally, was about the same as theSATADOM? Surely, theRAIDPoolshould have performed compatible to, if not better than, theDataPool? Given that they are comprised of similar disks present in the same RAID Controller? Similar to above, the Write IOPS show the same bizarre story.The Write Latency from the Hypervisor tests also appears to be unusual. The
RAIDPoolappears to be experiencing up to ten times worse latency than the ZFS Pools? However, if you flip over to the VM tests, the latency for the three storage backends seems to congregate around 300us. Which is pretty similar to what we were seeing in the WORST cast for theRAIDPool. Why does this smooth effect happen to write latency when the tests are run from VMs instead of hypervisor? Why does the latency for the ZFS Pools suddenly become so much worse, and comparable to, theRAIDPool?Looking at the read Bandwidth, IOPS, and latency shows a similar story. All metrics are equally slow, despite having massively different hardware configurations, when benchmarked from within a VM. However, once benchmarked from the hypervisor, the ZFS Pools suddenly greatly outperform everything else?
Questions:
These results are abnormal... right? This benchmark from this website shows a 970 EVO achieving upwards of 900MB/s random write speeds. Why are mine only coming in at 150MB/s on the hypervisor and 10MB/s in a VM? Why are these speeds so different when benchmarked from the hypervisor and from a VM?
Why does the
RAIDPoolsuddenly become abnormally slow when benchmarked from the Hypervisor? Here we see that the Read Bandwidth in a VM averages 20MB/s. However, from the hypervisor, it instead reports 4MB/s. Just like the benchmark tests I showed in question 1, shouldn't these read speeds be closer to 900MB/s?Why do the ZFS Pools suddenly perform significantly worse when benchmarked from within a VM instead of the hypervisor? For example, here we can see that the read IOPS averaged around 200,000 and latency under 650us. However, when benchmarked from within a VM, we can suddenly see that the read IOPS average around 2,500 and the latency more than quadrupled? Shouldn't the performance in both situations be about the same?
