When doing some performance tests writing to a SSD-based 3-way RAID 1 mirror powered by mdadm, it appears we suffer a significant write penalty of about 2.2x slower than doing the same test on just one individual drive. We're reading from and writing to the same underlying physical drives in this test because that simulates our real world test case we're interested in.
Is this slowdown due to SATA III speed limitations or something else? I'm surprised RAID 1 would have a write penalty because I'd think it could write to all three drives simultaneously at the same speed it could write to one of them.
All three drives present:
dd if=/dev/md3 of=test.file bs=1048576 count=37193
...207.748 s, 188 MB/s
Just two drives present (i.e. normal two-drive RAID 1)
dd if=/dev/zero of=test.file bs=1048576 count=37193
...119.016 s, 328 MB/s
Just one drive present (no redundancy)
dd if=/dev/zero of=test.file bs=1048576 count=37193
...93.794 s, 416 MB/s
It appears all three SSDs are on the same controller and that you're reaching the maximum SATA III speed rating:
Your test for writing to all three drives at the same time, so that 188MB/s is actually 564MB/sec, just short of the maximum speed.
If you can move one or more of the SSDs to separate controllers it may help.
This limitation you're hitting is one of the reasons why PCIe flash based storage is showing up in more and more high end systems.