I'm building a server with 24 SATA III disks connected to a Chenbro 12G SAS expander on a 12G SAS HBA (LSI 9300) with a single SFF-8643 cable. Aggregate write speed is slower than I hoped for, and I'm wondering if this is due to the lack of Store & Forward technology. I can write to each disk individually at 170MBps, but writing to all 24 simultaneously drops to 75MBps each or 1800 aggregate.
According to this answer, SAS does not support S&F. Since the disks are 6Gbps, I have a total of 24Gbps between the HBA and the expander, or 2400MBps, and maybe there is some other SATA translation overhead limiting it further to 1800Mbps. With S&F support, I should have 4800Mbps to the expander.
But here is a draft proposal for SAS2 regarding support for S&F published in 2006. I can't find any more recent information on this, so was S&F ever implemented? If so, then what is the benefit of buying 12G SAS disks instead of 6G SAS, considering no spinning disk comes close to 6Gbps?
http://www.t10.org/ftp/t10/document.06/06-386r0.pdf
Quote from the proposal:
Such a solution would involve data transferred back and forth between expanders and target devices at 3Gb per second, and data transferred back and forth between expanders and initiator devices at 6Gb per second, without sacrificing link utilization on either end.
Whether an expander at hand is able to buffer (S&F) or not is totally up to the expander. In your case, you'd need to look for another expander or faster HDDs.
This is exactly the point for using 12G drives even though magnetic drives can't even come close to 6G. Most expanders negotiate all links at the same speed, so a single 6G drive can seriously impede your host connection.