I am looking for a multi-volume, indexed, seekable and appendable archive format.
- multi-volume = split the TB-size archive into GB-chunks
- indexed = add complete index at the archive end with offset information per file in the archive stream
- seekable = allow fast extraction of file ABC from chunk 15 and 16 with only those chunks plus the last chunk being present
- appendable = add data and update the index after initial archive creation
There is a similar question here on Askubuntu but limited to rar which apparently ticks all four boxes but isn't appendable and multi-volume at the same time. tar is appendable but needs to read in the entire archive for extraction of any part. multi-volume zip files are apparently uneditable and so is 7-zip. zpaq is pretty close but apparently the stream needs to be present in its entirety for extraction. An interesting approach is gzipi which cats gzipped chunks together and adds an index to it. The idea might be generalized for a new tool but the implementation seems to be specialized towards logs and text files only.