I have an IBM x346 with dual 3.2Ghz Xeons and 2x36GB 15k SCSI disks. I'm running an entirely ZFS-based install of FreeBSD 8.1.
I can configure the bios to mirror the two disks, and then put ZFS on that "single" drive, or I can mirror the drives using ZFS, and install to that.
If I go the all-ZFS route, I suppose I have a little more freedom with how I use the devices (I could set up a fast non-redundant striped area, in addition to my mirrored system install, for example). Do I gain anything as far as error correction goes? Rebuild performance/ease/time?
I know that both configurations are probably the same to within a gnat's eyelash, but I'd like a second opinion.
Don't know about the speed, but here is what I believe running ZFS on RAID would means:
You lose the benefits of atomic writes because now the RAID controller has the last say on when a write happens to the disk. Which means you rely on the RAID controllers NVRAM.
ZFS also may get lied to if the data was written improperly. ZFS would have to take the RAID controllers word for it.
You would also lose repairing files because, from ZFS's point of view, you have a single disk, if the data's integrity is bad, ZFS would have no way to repair it because there is no second copy. (assuming you don't set copies=2)
If the RAID drives fall out of sync the RAID drive may take some time to sync depending on the journaling ability. ZFS will resilver the data it finds bad and at least some OSs may run a resilver periodically to ensure the integrity. Again because the RAID will only display one drive to ZFS, ZFS can't help with the repair/rebuild.
You would be able to expand the RAID (if the RAID has the capability) and maybe rebuild the ZFS data across more drives. (For me not a big plus considering the negatives so far.)
Of course all the snapshot functionality of ZFS would be unaffected (assuming the data doesn't silently get corrupted).
Hardware RAID would almost negate any advantages that ZFS would have. Personally, I wouldn't recommend using anything under ZFS, I would run ZFS on bare metal.
However, if there is an advantage I hadn't considered, I'm open to hearing it.
ZFS mirror will be much better.
Mirror is a very simple config and need almost zero resources from the CPU, but in addition you get a much better pool control in comparison with a hw or a pseudo hw raid.
In my own practice, in the modern systems even having raidz is better than a hw raid5, the performance is only 5-10% slower but ZFS benefits fully cover this (and of course you save a money cuz you don't need a card with a hw raid5 support).
One point which I would consider:
What if the RAID-Controller dies? Is the RAID compatible with other hardware so that you can easily exchange the controller?
With ZFS this is not a problem. Just plug a new controller and that's it.
I use gmirror for UFS and ZFS-RAID for ZFS-Systems ever since. I had a RAID-Controller once (cheap one, sure) which had to resync the drives before booting up the system (BIOS stage) which took endless (1.5TB drive). During that time you can't use the system.
With ZFS/gmirror you reboot the system and it syncs while you are working with it (even not recommended and slowing down the system).
But I would confirm Scott's points (even I would prefer you mark what is a pro and what is a con ;) )
One advantage to using the hardware RAID would have been potentially higher performance, since you only have to write to the raid controller once instead of twice. Probably only a little, though, and only if you're already maxed out. I'd use ZFS mirroring myself, though, for all the reasons already mentioned.
re: mirrored raid having hardware compatibility problems, i've run into it. The issue was that the raid controllers put their own identifying data somewhere on the drive. A different brand of raid controller didn't know how to interpret this.