Home / server / Questions / 2600
Accepted
Guy C
Asked: 2009-05-02 08:46:11 +0800 CST

How do you set bzip2 block size when using tar?

  • 772

I am using tar to backup a linux server to tape. I am using the -j option to compress the file with bzip2, however I can't see a way to adjust the block size options for bzip2 from tar. The default block size is 900,000 bytes which gives the best compression but is the slowest. I am not that bothered about the compression ratio, so am looking to make bzip2 run faster with a smaller block size.

backup linux compression

5 Answers

  1. Best Answer
    export BZIP=--fast
tar cjf foo.tar.bz2 foo

    Or pipe the output of tar to bzip2.

    Though you should note from the bzip2 man page:

        -1 (or --fast) to -9 (or --best)
              Set  the  block size to 100 k, 200 k ..  900 k when compressing.
              Has no effect when decompressing.  See MEMORY MANAGEMENT  below.
              The --fast and --best aliases are primarily for GNU gzip compat-
              ibility.  In particular, --fast  doesn't  make  things  signifi-
              cantly faster.  And --best merely selects the default behaviour.
    • 11 
  2. tar -cjf dir.tar.bz2 --options bzip2:compression-level=9 path/to/dir/
    • 4

  3. bzip2 block sizes

    bzip2 has some block size options. From the manual page bzip2(1):

    -1 (or --fast) to -9 (or --best)
       Set the block size to 100 k, 200 k ..  900 k when compressing.
       Has no effect when decompressing. See MEMORY MANAGEMENT below.
       The --fast and --best aliases are primarily for GNU gzip
       compatibility. In particular, --fast doesn't make things
       significantly faster. And --best merely selects the default
       behaviour.

    As you want faster compression with less regards to compression ratio, using bzip2, you seem to want the -1 (or --fast) option.

    Setting bzip2 block size when using tar

    You can set bzip2 block size when using tar in a couple of ways.

    The UNlX way

    My favorite way, the UNlX way, is one where you use every tool independently, and combine them through pipes.

    $ tar --create [FILE...] | bzip2 -1 > [ARCHIVE].tar.bz2

    You can read that as "create .tar with tar -> bzip it with bzip2 -> write it to [ARCHIVE].tar.bz2".

    Environment variable

    It is also possible to set bzip2 options through the environment variable BZIP2. From the manual page bzip2(1):

    bzip2 will read arguments from the environment variables BZIP2 and BZIP,
in that order, and will process them before any arguments read from the
command line. This gives a convenient way to supply default arguments.

    So to use that with tar, you could for example do:

    $ BZIP2=-1 tar --create --bzip2 --file [ARCHIVE].tar.bz2 [FILE...]

    Faster alternatives

    bzip2 uses a slow compression algorithm. If you are concerned about speed, you could investigate alternative algorithms, such as those used by gzip or lzop. Here is a nice article comparing compression tools: https://aliver.wordpress.com/2010/06/22/huge-unix-file-compresser-shootout-with-tons-of-datagraphs/

    • 2

  4. Send the tar output to stdout and then pipe it through bzip2 separately:

    % tar cvf - _file_ | bzip2 _opts_ > output.tar.bz2
    • 1

  5. Its even easier:

    % tar -cvf dir.tar path/to/dir/ && bzip2 -9 dir.tar
    • 1