Break up a dd image into multiple files

It is my command line:

dd if=/dev/sda bs=4M | gzip -c | split -b 2G - /mnt/backup_sda.img.gz

It will create 2GB files in this fashion:

backup_sda.img.gz.aa
backup_sda.img.gz.ab
backup_sda.img.gz.ac

Restore:

cat /mnt/UDISK1T/backup_sda.img.gz.* | gzip -dc | dd of=/dev/sda bs=4M

Hope it helps.

You probably want to consider using tar, as KPWINC says, but to answer your question directly, you want to use dd's "skip" option.

If your first command, as stated, is:

sudo dd if=/dev/sdf1 bs=4096 count=150GB | gzip > img1.gz

Then your second would be:

sudo dd if=/dev/sdf1 bs=4096 skip=150GB count=40GB | gzip > img2.gz

and third:

sudo dd if=/dev/sdf1 bs=4096 skip=190GB count=120GB | gzip > img3.gz

That said, I'm not sure that the "GB" suffix does what you're intending. I think it just does raw math on the root number it follows, not figure out how to get that many gigabytes from the block size you've given. I would do something like this:

dd if=/dev/sdf1 bs=`expr 10 * 1024 * 1024` count=`expr 15 * 1024 * 1024 * 1024`

just to be sure of the math.

Oh, and make sure that your device isn't changing underneath you as you copy it. That would be bad.

A simple solution might be to just use "/usr/bin/split". It just breaks files up into pieces. You can use "-" as the input file name to read from standard input. The nice thing about split is that it is simple, it doesn't affect the toolchain in any real way and you can "join" the files by just using "cat" to glob them back together (or pipe them to another app).

tar

I tar might solve your issue. It has the ability to break up files into multiple volumes.

Check out this link:

http://paulbradley.tv/44/

From the page:

The two extra command line options you need to use over and above the standard syntax are -M (--multi-volume) which tells Tar you want to split the file over multiple media disks. You then need to tell Tar how big that media is, so that it can create files of the correct size. To do this you use the --tape-length option, where the value you pass is number x 1024 bytes.

The example below shows the syntax used. Lets say the largefile.tgz is 150 Meg and we need to fit the file on two 100 Meg Zip drives.

tar -c -M --tape-length=102400 --file=disk1.tar largefile.tgz

The value 102400 is 1024 x 100, which will create a 100 Meg file called disk1.tar and then Tar will prompt for volume 2 like below :-

Prepare volume #2 for disk1.tar and hit return:

In the time of tape drives you would have taken the first tape out of the machine and inserted a new tape, and pressed return to continue. As we want Tar to create the remaining 50 Meg in a separate file, we issue the following command :-

n disk2.tar

This instructs Tar to continue writing the remaining 50 Meg of largefile.tgz to a file named disk2.tar. You will then be prompted with the line below, and you can now hit return to continue.

Prepare volume #2 for disk2.tar and hit return:

You would repeat this process until your large file has been completely processed, increasing the disk number in the filename each time you are prompted.

A couple of things here. First, the command you listed probably won't work as you're expecting. It looks like you're trying to hit 150GB, but you need to factor in both the block size and the count (count is the number of blocks of block size). So if you want 150GB, you might do bs=1GB count=150. You could then pick up where you left off by adding a skip=150 to skip 150 blocks (each of 1GB) on your second run. Also, to have gzip send its output to standard out, you need to pass it the -c option.

However, before you do that, a couple of other questions. Are you using dd here because the filesystem is corrupted/damaged/etc and you need a bit-for-bit exact disk image copy of it? Or are you just trying to get the data off? A filesystem-aware tool might be more effective. Particularly if the source filesystem isn't full. Options include tar, or you might want to look into something like Clonezilla, Partclone, Partimage, or for a more Windows-specific option to directly access a Windows filesystem, Linux-NTFS (note the previously mentioned tools can handle Windows filesystems to various degrees, too).

If you are set on operating on the partition with a non-filesystem aware program, then using your dd line (as modified above to be correct) will likely work. It's hard to say how well it will compress, but it should be smaller than the original filesystem. If you have read-write access to the original filesystem, it would be worth filling up the free space with a file written from /dev/zero to zero out the unused space before saving it with dd. This will enhance gzip's ability to compress the free space in the disk image.

To operate on the second chunk, just add a skip=XXX bit to your second dd invocation, where 'XXX' is equal to the count= value you gave it the first time. If you wanted to do 150GB on your first one and 40 on your second, you might do:

sudo dd if=/dev/sdf1 bs=1GB count=150 | gzip -c > img1.gz

followed by:

sudo dd if=/dev/sdf1 bs=1GB skip=150 count=40 | gzip -c > img2.gz

Comming late for the party, but I had to do such a script to backup an unknown file system on a FAT32 stick.

This simple script works for me:

# Sets pipefail so failed dd makes the whole pipe command fail
set -o pipefail
dd_result=0
split=0
filenameSplit=
while [ $dd_result == 0 ]
do
    cmd="dd if=/dev/sdX bs=2G count=1 skip=$split | pigz --fast > \"2G.$filenameSplit.myharddisk.gz\""
    echo $cmd
    eval $cmd
    dd_result=$?
    split=$((split + 1))
    filenameSplit=$split
done

EDIT: Have uploaded a basic backup restore script that uses dd, pigz or gzip compression and splits files into chunks that can be set at commandline. Script found here: github.com/deajan/linuxscripts