Home / ubuntu / Questions / 587081
In Process
kos
Asked: 2015-02-19 12:25:45 +0800 CST

How can I archive dd's output directly possibly without using any compression?

  • 772

It has been determined already in this question that tar cannot read input from stdin.

How else can a dd output be archived directly possibly without using any compression? The purpose of doing everything in a single task is to avoid to write the dd output to the target disk twice (once as a raw file and once as an archive), and to avoid to perform two different tasks, which is a waste of time (since the input file must be read and written and the output read, processed and written again), and can be impossible if the target drive is almost full.

I'm planning to do multiple backups of drives, partitions and folders, and I'd like to benefit both from the ease of having everything stored into a single file and from the speed of each backup / potential restore task.

dd

1 Answers

  1. If you want to dump a whole block device to a file, tar won't be of any use, because it doesn't work with block devices. Instead you'll need to use dd or similar:

    dd if=/dev/sdX of=/path/to/backup bs=16m

    Even like this it would be better to use at least a little compression, as long as it doesn't slow down the transfer too much. In short, you need a compression algorithm with a throughput not much lower than that of your slowest storage medium. There are several such compression algorithms. The most notorious are Lempel–Ziv–Oberhumer, its derivate L4Z, and Snappy. There's a comparison of various compression algorithms including those three on the L4Z project page:

    Name            Ratio  C.speed D.speed
                        MB/s    MB/s
LZ4 (r101)      2.084    422    1820
LZO 2.06        2.106    414     600
QuickLZ 1.5.1b6 2.237    373     420
Snappy 1.1.0    2.091    323    1070
LZF             2.077    270     570
zlib 1.2.8 -1   2.730     65     280
LZ4 HC (r101)   2.720     25    2080
zlib 1.2.8 -6   3.099     21     300

    For the sake of this answer, I'll choose an example with LZO, because it's readily available in Canonical's repositories in the form of lzop, but ultimately all those stream compressors have front-ends that read from standard input and write to standard output.

    dd if=/dev/sdX bs=16m | lzop > /path/to/backup.lzo

    If you want to work on the same machine during backup, you may want to use ionice and/or nice/schedtool:

    ionice -c 3 dd if=/dev/sdX bs=16m | ionice -c 3 schedtool -B -n 10 -e lzop > /path/to/backup.lzo
    • 4