Home / ubuntu / Questions / 217398
In Process
Andre
Asked: 2012-11-15 17:47:46 +0800 CST

How can I set up on-the-fly/transparent compression?

  • 772

I have huge data sets to deal with, and so my 1TB drive is filling up quickly. However, the data is highly compressible when using LZMA (7zip, bzip2). The only file system with transparent compression I've discovered so far is Btrfs, which is not assumed stable yet. Is there a reliable solution for piping the data through a compression and do a bind-mount to a compressed image file?

Any idea is welcome!

I do not want to patch any file system driver though.

Thanks!

filesystem

2 Answers

  1. Fusecompress would have been a ideal, since that lives in virtual filesystem and thus has no requirement on changing the actual filesystem run. However, sadly, the project is dead upstream and I would recommend to stay away from it for this reason.

    BtrFS supports only ZLIB and LZO compression at the time of writing.

    ZFS-on-Linux (non-FUSE) offers more options, including LZ4 compression. It's actively maintained and the userbase for ZFS in general is quite large in other than-Linux Unices (Sun, BSD, etc.).

    Migrating to a "3rd generation" filesystem could be a good idea for other reasons too, but that's totally up to how flexible you are in implementing just this.

    • 2

  2. You can also try ZFS with Gzip-9 with would give some good results.

    Squashfs may also be considered (I didn't try it though).

    • 1