I love the idea of using SSD EBS instance stores as L2ARC and ZIL for a zpool backed by EBS.
Going further (and into more dangerous territory), could I instead create a zpool mirror with the 2 instance stores:
zpool create vol1 mirror xvdb xvdc
and then use ZFS snapshotting/replication to keep a "warm"/eventually consistent spare on EBS?
- I would be OK with losing a few seconds of data
- I don't want to add the EBS as a hot spare, because that would limit the speed of the whole pool
Seems to me a perfectly acceptable setup if your spare writes to EBS (and you snapshot that), and you have some failover scenario, as obviously restoring your instance store cost time.
What you describe was actually the only setup we had before we had EBS. People survived for years doing exactly that.
Finally, Netflix moved away from EBS backed disks due to extra risk of failure. They just replicate using instance storage (using Casandra).
I would not mix such different disks in a mirrored volumes. I would rather use frequent send/receive iterations to have consistent, point-in-time backups of the main volume.
I would suggest not to mirror the drives. Instead, create 2 zpools with one drive each, one with the ephemeral drive then another with the EBS drive. Create a dataset then zfs send at frequent intervals to the EBS zpool from the ephemeral zpool snapshots. You can easily grow the EBS drive and zpool while your zpool is online using
awscli to grow the EBS snapshot,fdisk,partedandzpool online -e(expand). With snapshots rotation you could save space. For example, keep only the last 24hrs snapshot - if you perform a snapshot and send/recv in 10 minutes interval you would keep at a minimum 144 snapshots in a day.