I have a pair of physical servers with 32TB of raw disk space each (4 * 8TB x 2) in a ZFS pool on Ubuntu. After formatting, I am currently at 86% used and my data volume is growing at a reasonably constant 50G per week. I've recently setup the ZFS volume on the 4 disks from just 4 separate volumes (synced to the other server for some redundancy) so I don't have to manage 4 volumes (x 2 servers) and do less balancing of each volume as it's now just 1 large volume.
I'm soon going to need to add another disk to the volume (technically 2, one per server), but I already know that you just can't dynamically add a disk to an existing ZFS volume set to increase it's overall size, you just end up with a "safer" old volume (with parity) and a completely "unsafe" new volume.
My question is: How do I regularly make the volume larger as the volume fills up? I'm also not able to just buy a pile of disks and make the volume huge and move this problem far into the future.
I'm wondering about the "steady state" of an ever growing volume, and will I just eventually end up where I was, with multiple volumes of different sizes and having to manually balance the usage across the volumes??
Update: Unfortunately the data is already de-duplicated, and will continue to grow for the foreseeable future. I suspect it will eventually stop growing, or old data could be purged, but I don't see that happening for a couple of years. (Thanks for the ideas though)
your problem is also well describe as the scale-up vs scale-out problem.
it is an infinite source of troubles if you want to stick with ZFS because of the replacement of some vdev, or extension of zpool, however there is a (new) solution with some recent release of "TrueNas Scale" which bring HCI (hyperconvergence infrastructure) to ZFS, so one can add "node" like structure to existing storage so you are not limited to the number of disk your server will accept, you can just add servers.
From my point of view I would prefer a Ceph infrastructure that you can make grow, but if your already familiar with ZFS "TrueNAs Scale" will be easier to manipulate.