I'm configuring a brand new HPC server and am interested in using SmartOS because of it's virtualization control and zfs features. Does this configuration make sense for a SmartOS HPC, or would you recommend an alternative?
System Specs: 2x 8-core xeon 384 GB RAM 30 TB HDs with 2x512GB SSDs
Uses: - zfs for serving data to different vms, and over the network; 1 SSD for L2ARC and 1 for ZIL - typically 1-2 ubuntu instances running R and custom C/C++ code
My biggest concerns as a newbie to SmartOS and ZFS are:
(1) will I get near-metal performance from ubuntu running on SmartOS if it is the only active vm? (2) how do I serve data from the global zfs pool to the containers and other network devices?
Your server specs seem fine, but I don't know where you'll find 30TB drives (at least not in this decade).
The ZIL does not need to be very big, certainly a 512GB SSD is a waste. You'd have to calculate the "optimum" size, but it's going to be in the 2-20 GB range typically. Take the fastest 'storage' drive in your zpool, figure out the transfer speed, multiple by about 10 seconds. (eg if your fastest drive is 170MBps, then ~2GB for ZIL).
L2Arc can use as much high speed SSD as you can throw at it however. Partitioning the SSDs so that the first few gigs are used for ZIL and the rest is used for L2Arc makes a lot of sense, and ZFS will make good use of multiple devices that way.
While I'm at it.. It's been shown that leaving 20-25% of your SSDs unused aids significantly in their garbage collection algorithms, making them noticeable faster in typical use. And as always, SSDs get killed relatively quickly in heavy use (2-4 years is typical from my experience, YMMV).
You'll have to define "near-metal performance". KVM, which SmartOS uses imparts 2-10% performance penalties typically. Using a Guest OS that can take advantage of paravirtualization (virt-io, etc) will keep you near the low end of that range. Ubuntu should work well.
How you serve date from a ZFS Pool to VMs and other network devices is accomplished in the same way as with any other storage... ZFS can provide block storage or file storage. VMs can use either type. NFS and Samba/SMB/CIFS use file storage. iSCSI or FC serve Block storage over the network. SmartOS can be configured to do any of these with enough patience.