I recently purchased a Dell R730xd that came with an H730 RAID card. I would like to bring an existing 6 disk ZFS pool to this server as well as create an additional 6 disk hardware RAID.
Is it possible to configure an H730 RAID card to operate in both HBA and RAID mode simultaneously? And then only manage 6 of the disks in a hardware RAID and allow the OS to manage the remaining 6 disks in a software RAID?
If not, can a Dell R730xd use both an H730 RAID card and an LSI 9207-8i HBA card simultaneously? Would you simply plug one SAS cable from each card into each part of the Backplane? Will I need to purchase any new cables to connect the LSI 9207-8i to the backplane in my R730xd?
Is there a better card to use for ZFS in an R730xd using spinning disks than the LSI 9207-8i?
Regarding your question about splitting the backplane across controllers, yes, it's possible. I've done this with 8-bay enclosures where two 4-lane SAS SFF-8087 connectors were present. 4 disks to one controller, 4 disks to the other. I don't know the composition or expander situation on the Dell server you're describing, though.
The only controllers that would allow this are those that have Mixed/Hybrid mode ports. HPE Smart Array controllers on Gen10 systems are a good example. They allow hardware RAID or HBA mode on a per-port/per drive basis.
I also have this running on an Avago MR9363-4i. /dev/sdy at the bottom is a RAID1 Hardware RAID1 for the OS. The remaining disks are split across two normal ZFS pools.
Replying to each questions:
while you can install another PCI-E RAID card, the backplane can be connected to a single card. So you can't concurrently use the native RAID card and the additional one to access the disks on the same internal backplane [edit: @ewwhite suggested splitting disks between controller should be possible, so maybe I am wrong here. However the only officially supported dual-controller setup on the R730xd requires the use of flex-zoning backplane option, so I am not sure if/how two controllers can be put into operation in that specific case];
probably not, but see #1
PERC H730 supports two operating modes: RAID mode and HBA mode
the standard RAID mode, which supports RAID and non-RAID disks. In RAID mode, the controller expect the disks to be part of a RAID array and will not expose the raw disk to the operating system. In non-RAID (or passthrough) mode, the disks declared as non-RAID are exposed to the operating system (see note #1). In this mode, SMART reporting is firmware dependent and, if not working, you can try using the dedicated
megaraiddriver - ie: issuing something assmartctl -a -d megaraid,0 /dev/sda;in HBA mode, the card works as a normal LSI-based SAS controller. This means any RAID capabiilty is disabled and disks are directly managed by the operating system. In such a mode SMART monitoring by the card is disabled and SMART data should be directly accessible (and monitored) by the operationg system. Additionally, you lose the card LED management and you may lose the added benefit offered by the controller writeback cache (I don't have direct confirmation), meaning you get much lower performance for random writes. However, please note that in ZFS you should use a SLOG to get high randon write performance rather than relying on the controller cache anyway.
In the end, as PERC 9/10 series cards are quite flexible you have no reason to use an additional RAID card.
Note #1, from the card manual:
I got same thoughs about mixing Perc hardware RAID and ZFS, I was thinking about splitting backplane or doing other nasty procedures. From one side I needed hardware raid volumes, from other side I needed to import existing ZFS pool.
I have readed that H730 can be set up for either HBA or RAID mode. However that's not true, at least for latest firmware. I have tested on R730, H730p mini mono and ESXI 7- everything works perfect. No need to create virtual disk. Passthrough works pretty cool!
Also, on ESXi 7 no more RDM passthrough needed!
Step 1. Upgrade PERC firmare to latest 25.5.7.005. Set it to RAID mode.
Step 2. Create Virtual Disk for needed hardware raid layout.
Step 3. Mark needed ZFS disks as Non-Raid ones.
Step 4. Im not sure if You can disable write cache for Non-Raid disks via BIOS, but I have suceeed disabling non-raid cache through OpenManage (of course cache still enabled for HW RAID):
H730p openmanage settings
Step 5. Non-raid disks and RAID disks will be normally visible on ESXi. Of course no problems with hotplugging.
Detected devices by vSphere
Step 6. Create/import FreeNAS VM. Edit VM settings, click Add Hard Disk -> New Raw Disk. Non-VMFS raw devices will be shown, and You can add them one-by one as passthrough:
New one-click RDM passthrough
RDM passthrough
Remember to leave Disk Compatibility: Physical
RDM passthrough
Step 7. Boot FreeNas, all passthrough disks will be visible as directly attached, including disks serial numbers, APM managment, etc:
PERC h730p nonraid disks in FreeNAS
Step 8. Use shell and test smartctl:
Smart results
Disks are directly visible by FreeNas. Also, I recommend upgrading to TrueNas 12.0 - iSCSI speed and stability improvments are really big step forward. No more ctl_datamove aborted errors.
You can also hotplug any other disk, set it to nonraid via iDrac and then just on-live Edit VM options and add new RAW device, it will be detected and attached to FreeNAS immediatly without rebooting of course.
I have readed too much on forums that's not possible... not sure if new firmware added such capability, but actually I am more than happy. System is very flexible that way.
From one side iDrac is fully helath-monitoring created RAID Virtual disk (incl. patrol read and all), and from other side non-perc RAID disks are monitored via FreeNAS. Also, because thats 1:1 passthrough, of course disks temperatures are monitored too.
Small tip: Don't forget to disable "Delayed ACK" when using ESXi software iscsi client. Feed FreeNas with decent amount of RAM. After disabling Delayed Ack, and upgrading NAS to v12.0 (From 11.3 Freenas to TrueNAS) actual iSCSI backup times dropped from 11+ hours to 3.5 hrs.
It will look like this from iDrac perspective:
However, remember that Write Cache can't be disabled for Non-Raid disks via iDrac, there is no such setting in controller settings. So OpenManage (confirmed), or bios maybe.
iDrac perspective