We currently have 4 Linux nodes with local storage, arranged in two active/passive pairs with storage mirrored using DRBD, running virtual machines (actually using Xen Hypervisor) for typical hosting workloads (mail, web, a couple VPS, etc.). We're approaching the (presumed) maximum IOPS of those servers, and we're planning to migrate to an external storage solution with two active nodes, with capacity for up to four active nodes.
Since we're an all-Dell shop I've done some research and found the MD3200 / MD3200i products should be the ones we're looking for. We are pretty sure we won't be attaching more than 4 hosts on a single storage and I'm wondering if there is any clear advantage for one or the other.
In theory I should be able to attach 4 SAS hosts to a single MD3200 (single links on a single controller MD3200, or dual redundant SAS links from each host to a dual-controller MD3200), or 4 iSCSI hosts to a single MD3200i (directly on its 4 GigE ports without any switch, again with dual links for the dual controller option).
Both setups should let us implement live VM migration since all hosts can access all the LUNs at the same time, and also some shared filesystem like GFS2 or OCFS2. Also, both setups should allow full redundancy of the whole system (assuming dual controllers in the storage).
One difference I can see is that the DAS solution is actually limited to 4 hosts while the iSCSI one should be able to grow to more hosts (adding two GigE switches to the mix).
One point for the iSCSI solution is that it would allow us to start out with our current nodes and upgrade them at a later time (we can't add other SAS controllers, but they already have 4 GigE ports each).
With the right (iSCSI|SAS) controllers I should be able to connect diskless nodes and boot them off the external storage which I think is a good thing (get rid of any local storage).
On the other hand, I would have thought the SAS one to be cheaper but it seems like an MD3200 actually costs a little less than an MD3200i (?)
(please note: I've used Dell gear in my examples since that's what we're looking for but I assume the same goes with other vendors)
I would like to know if my assumptions above are correct, and if I'm missing any important difference between the two setups.
DAS generally provides better performance than older (4Gbps and lower) FC and iSCSI (1Gbps * amount of paths). Setup is pretty simple and cabling is very straightforward. You also save on the extra equipment (no need for an FC or iSCSI switches). You need to make sure you get a DAS that supports clustering (usually it's a DAS that has it's own RAID controllers, and the connecting servers will only have SAS/SCSI HBAs, unlike the simpler DAS boxes that simply hook on to a controller in the server).
With FC or iSCSI you get more flexibility and scalability. Multipathing also helps in case a cable dies, and with enough paths, you can beat the speed of direct SAS. The cable lengths can be much greater as well (IIRC the max length for a SAS cable is 10m, while an ethernet or FC cable can be much longer)
You've nailed it. SAS is used for directly connecting storage, and iSCSI is used for having a bunch of servers share the storage. One point of interest is that SAS has considerably more bandwidth available than 1Gb iSCSI. If your workload is small block random IO, this won't help you, but when you do large sequential work (backups, for example), it may.