I'm trying to figure out the lowest hassle way to provision 24x locally attached SSDs as a large logical volume with low-value data. I'm using them as a hot-set cache for data who's master state (about a petabyte) resides in S3, so I care more about performance, complexity of maintenance, and downtime more than lost data. Nothing will linger in the hot data set for more than a couple days, and its all easy to recreate from S3 anyway.
- Medium large instance: 32x vCPUs, 120GB RAM, Skylake
- 24x locally attached SSDs @ 375GB each = 9TB total
- Hosted on Google Cloud (GCP)
- Debian 10 (Buster)
- Access is ~4x heavier on read than write
- High number of concurrent users (human and machine) with pretty random access patterns, and very hungry for I/O.
- 90% of files are larger than 10MB
I'm thinking RAID 5 is out of the question, no chance I'm going to wait for manual rebuilds. I'm inclined toward either RAID 0, RAID 10, or.... maybe this is actually a case for a simple LVM pool with no RAID at all? Do I really lose anything by going that relatively simpler route in this case?
My ideal solution would have each subdir (I have one self contained dataset per subdir) of /
completely contained on a single disk (I can fit maybe 10 subdirs on each drive). If a drive failed, I'd have a temporary outage of the subdirs/datasets on that drive, but an easy to reason about set of "these data sets are redownloading and not available". Then I'd just rebuild the missing data sets from S3 on a new drive. I suspect LVM jbods (not sure of exactly the right word for this?) might come closest to replicating this behavior.