Home / server / Questions / 279633
Accepted
Adam Matan
Asked: 2011-06-13 06:50:40 +0800 CST

Ultra-efficient compression

  • 772

I store a very large (TBytes) amount of logs. These logs will rarely be extracted, and when they will - only a single file will be required.

Could you recommend an ultra-efficient and extremely stable compression algorithm that's considerably better than bzip2?

log-files compression

1 Answers

  1. Best Answer

    lzma (aka xz) should do notably better than bzip2, but will take a bit longer.
    paq (aka zp) will do quite a bit better yet, but will take ages to compress and just as long to decompress.

    Both are available for Windows and *nix environments (most *nix systems have packages available)

    A quick test on a smartd log:

    Original       3900K
GZip            208K    0.11s
BZip2            71K    3.07s
XZ               13K    1.76s*
ZP                6K   25.68s*

    *I've got -O3 compiled ports for xz and zp. The gzip and bzip2 binaries were precompiled with no optimization.

    • 15