Differences between HW RAID and ZFS

Hardware RAID vs ZFS doesn't make a lot of difference from a raw throughput perspective -- either system needs to distribute data across multiple disks, and that requires running a few bit shifting operations on cached data, and scheduling writes to underlying disks. Which processor you use for that hardly matters, and synthetic workloads like running dd can't tell you much here.

The differences are in features:

Hardware RAID is usually just a block layer, perhaps with some volume management on top, while ZFS also includes a file system layer (i.e. there is no separation of concerns in ZFS). This allows ZFS to offer compression and deduplication, while that would be hard to get right on a block layer, but for use cases where you just want a set of simple 1:1 mappings, that additional complexity will still be there.

On the other hand, hardware RAID can offer battery backed write caches that are (almost) transparent to the operating system, so it can easily compensate for the overhead of a journaling file system, and data needs to be transferred out of the CPU only once, before adding redundancy information.

Both have their use cases, and in some places, it even makes sense to combine them, e.g. with a hardware RAID controller that offers a battery backed cache, but the controller is set to JBOD mode and only re-exports the constituent disks to the operating system, which then puts ZFS on top.

In general, ZFS alone is good for "prosumer" setups, where you don't want to spend money on hardware, but still want to achieve sensible fault tolerance and some compression, and where random-access performance isn't your primary concern.

ZFS on top of JBOD is great for container and VPS hosting -- the deduplication keeps the footprint of each container small, even if they upgrade installed programs, as two containers that have installed the same upgrade get merged back into one copy of the data (which is then again kept in a redundant way).

Hardware RAID alone is good for setups where you want to add fault tolerance and a bit of caching on the outside of an existing stack -- one of the advantages of battery backed write caches is that they are maintained outside of OS control, so the controller can acknowledge a transfer as completed as soon as the data has reached the caches, and if a write is superseded later, it can be skipped, and head movements can be scheduled system-wide ignoring dependencies.

The way journaling file systems work, they will first submit a journal entry, then as soon as that is acknowledged, submit the data and after that is acknowledged, another journal entry marking the first as complete. That is a lot of head movement, especially when the disks are shared between multiple VMs that each have their own independent journaling file system, and in a busy system, the caches allow you to skip about half of the writes, but from the point of view of the inner system, the journal still behaves normally and dependent writes are performed in order.

The aspect of safely reordering dependent writes for more optimal head movements is why you want a hardware RAID at the bottom. ZFS generates dependent writes itself, so it can profit from hardware RAID too, but these are the performance bottleneck only in a limited set of use cases, mostly multi-tenant setups with little coordination between applications.

With SSDs, reordering is a lot less important, obviously, so the motivation to use hardware RAID there is mostly bulk performance -- if you've hit the point where memory and I/O interface speed on the mainboard are relevant factors, then offloading the checksum generation and transferring only a single copy one way vs multiple transfers from and to RAM (that need to be synchronized with all the other controllers in the same coherency domain) is definitely worth it. Hitting that point is a big "if" -- I haven't managed so far.

Short answer... You can use hardware RAID where it makes sense.

It really depends on where you want your RAID protection to come from and where you want your volume management to come from.

For example, I use HPE ProLiant servers...

• I'm building a 100TB storage array today.
• This is going into an environment where there won't be regular IT staff or knowledgeable support.
• I'm using HPE SmartArray RAID to build this as a RAID 60 setup across 24 disks.
• I'll set the Smart array to carve out a 100GB RAID 60 volume for the OS and the remainder to the data volume.
• ZFS will be installed on top of the RAID block device presented to the OS (e.g. a single VDEV ZFS zpool)

The reasoning for this design is that HPE SmartArray hardware RAID is reliable and consistent in operation. It's easy to direct someone to replace a disk or to build automatic spares into that setup. Considering the location is unstaffed with IT resources, this make sense for manageability reasons.

I still get the benefit of ZFS volume management and caching, compression, performance, etc.

In a more controlled environment, I may instead set the controller in HBA mode and use raw disks for ZFS.

If you use hardware raid instead of ZFS raid you will lose a couple features. Lets imagine a simple two disk mirror for the rest of this post.

1. ZFS will not be aware that there are two disks and therefore two copies of every block. So if and when it detects a checksum error all it can do is notify you that there is a file that has a corrupt block. You would need to restore this file from backup to fix. ** ALSO**: You're hindering it's ability to even detect corruption. In the scenario above when ZFS requests to read a block there are actually two disks that each have identical copies and the HW controller will give ZFS data from one of the two disks. ZFS has no control over which disk, or even awareness that there are more than one. So ZFS has no way of checking each disk individually so it's quite possible it could take multiple successive reads to finally read the bad block and even detect the corruption. At that point you don't even know which disk it is.

2. Because ZFS is also the filesystem it is aware of the FS. So if I have a mirror that is 95% free space and I replace a drive ZFS knows to only copy the 5% of actual data. HW Raid controllers are blind to the FS and have no way of distinguishing free space (or, previously used but since freed space) from data. So HW raid will blindly block copy the entire contents of diskA to diskB.

ZFS over HW RAID is fine, but only if you understand what this means. Specifically:

• not having redundancy at the vdev level, it can detect but not fix data corruption
• a faulty controller can totally trash your pool (a point only partially invalidated by the fact that a dying CPU/RAM/MB can have a similar effect)
• you depend on the controller powerloss-protected writeback cache to be healthy (PERC H730 should use a flash-based powerloss-protected cache, or FBWB, so you should be safe here)

Replying to your questions:

• What are the advantages and disadvantages of HW RAID versus ZFS? Read here for the answer

• What are the differences between HW RAID and ZFS? HW RAID is a block-level affair only, where the RAID controller follow some "simple" replication/distribution schema to remap the LBAs to the physical addresses. ZFS is a complex CoW filesystem which, by its very nature, really like as much cache you can throw at it (to avoid read/modify/write cycles). In ZFS, the "RAID layer" is handled by the SPA (storage pool allocator).

• Are HW RAID and ZFS complementary technologies or incompatible with each other? You can use ZFS on top of HW RAID, if your setup requires it. ZFS will work just fine, with the notes I already wrote above.

• Since Proxmox VE is a Debian-based Linux distribution, does it make more sense to use the H730 for RAID10 with LVM versus setting the H730 in HBA mode and using ZFS? Having a performant H730 card, I would not use MDRAID. I have a similar server (DELL R720 with PERC H710p) with HW RAID + ZFS and it works very well.

1. What are the advantages and disadvantages of HW RAID versus ZFS?

Hardware RAID can sometimes yield better performance from a base config, but ZFS is far more powerful, scales better, and when properly tuned, it can yield better performance.

2. What are the differences between HW RAID and ZFS?

ZFS offers many features that are completely unavailable in other kinds of RAID, such as snapshots, copy-on-write, send-and-receive, compression, deduplication, caching, bit rot protection, nested volumes and filesystems, and independence from a particular manufacturer's hardware RAID implementation. ZFS is lso far more flexible, and can meet multiple use cases at the same time.

3. Are HW RAID and ZFS complementary technologies or incompatible with each other?

They are completely different technologies. You can run ZFS on top of other RAIDs, but then you would lose bit rot protection.

4. Since Proxmox VE is a Debian-based Linux distribution, does it make more sense to use the H730 for RAID10 with LVM versus setting the H730 in HBA mode and using ZFS?

LVM offers some of the features ZFS does, minus bit rot and filesystem support, and doesn't perform quite as well. ZFS is built into Proxmox for a reason.

I would highly encourage you to read Ars Technica's introduction to ZFS, as it will explain this in far better detail.