How to set up disk cloning with dd, netcat and ssh tunnel?

Copying from source to target where target has sshd running:

• dd if=/dev/sda | gzip | ssh root@target 'gzip -d | dd of=/dev/sda'

Copying from source to target via sshd_host when target is not running sshd.

• Target: nc -l -p 62222 | dd of=/dev/sda bs=$((16 * 1024 * 1024))
• Source: ssh -L 62222:target:62222 sshd_host &

• Source: dd if=/dev/sda | nc -w 3 localhost 62222

  dd - if= is the source, of= is the destination, bs= is the block size. Different block sizes may improve performance. 16 is usuually a fairly reasonable starting point. You can also use count= to indicate how many blocks to copy.

  nc - -p indicates the port to use for services. -l is used to start a service. -w sets up the time to wait for data in the pipline before quiting.

  ssh - -L sets up the tunnel on the remote host. The format of the argument is, local_port:target_host:target_port. Your local program (nc) connects to the local_port, this connection is tunneled and connected to target_port on the target_host.

The options defined are just the ones used for this. Look at the man pages for more details.

A few notes:

1. If you are doing this over anything but a LAN, I'd suggest compressing the datastream with gzip or compress. Bzip2 would work too but it takes a bit more CPU time. The first one has an example of that usage.
2. Its better if the source partition is not mounted or is mounted read-only. If not you will need to fsck the destination image.
3. Unless one of the machines has netcat but not ssh, netcat isn't really needed here. That case would look like:

source machine dd -> nc -> ssh -> ssh tunnel -> sshd server -> nc on target -> dd

4. dd works best if the source and targets are the same size. If not the target must be the bigger of the 2.
5. If you are using ext2/3 or xfs, dump (or xfsdump) and restore may be a better option. It wont handle the boot sector but it works when the target and source are different sizes.

netcat is not needed.

on src machine run:

dd if=/dev/sdX bs=1M | ssh root@dstMachine " dd of=/dev/sdY bs=1M"

i assume none of partitions on sdX and sdY are mounted. you can boot both boxes with knoppix or other similar live distro.

dd - takes data from if [ if not provided - takes it from stdin ], sends data to of [ if not provided - data is sent to stdout ]. bs - block size... will speed things out.

ssh - executes command provided in quotes on remote box, all data pumped to stdin of ssh will be tunneled to remote machine and porovided as stdin to command executed there.

Host A is the one to image, host B is the one the image will be stored on:

root@A# dd if=/dev/sda | ssh root@B "dd of=/some/file"

Restoring to disk would just swap those two.

If you want use netcat without ssh. I presume that is the fastest way and not the secure one, you can copy and restore the whole disk like this:
On computer A with IP 192.168.0.1

cat /dev/hdb | nc -p 9000

nc -l 192.168.0.1 9000 > /dev/hdb

Remember that according to man nc the -l option is:

  -l  Used to specify that nc should listen for an incoming connection rather than initiate a connection to a remote host.  It is an error to use this option in conjunction with the -p, -s, or -z options.

The basic copy with netcat is described here.

If you need to get SSH involved in to this, you could use port forwarding over that,

-R [bind_address:]port:host:hostport

But, on the whole, you could just do the SSH transfer in the first place (without netcat).

So long as the filesystems are both unmounted, dd works well.

(from server1) dd if=/dev/sda bs=32k | ssh <server2> dd of=/dev/sda bs=32k

You'll need hostkey authentication setup ahead of time or else the password prompt will cause the copy to fail.

Doing this on a mounted volume will produce poor results.

Or, you could use clonezilla and "mount" your remote storage through sshfs.

I tried a combination of the options provided above, and am sharing the results with you. fastest to slowest using combinations of dd block size, gzip and gzip compression algorithm.

As you can see gzip only gave me an improvement when using the fast algorithm in conjunction with a 1M block size.

time dd bs=1M if=/dev/HypGroup00/stage-snapshot  | gzip --fast | ssh hyp5 'gzip -d | dd bs=1M of=/dev/HypGroup00/stage'
12884901888 bytes (13 GB) copied, 326.045 s, 39.5 MB/s

time dd if=/dev/HypGroup00/stage-snapshot  | gzip --fast | ssh hyp5 'gzip -d | dd of=/dev/HypGroup00/stage'
12884901888 bytes (13 GB) copied, 370.158 s, 34.8 MB/s

time dd if=/dev/HypGroup00/stage-snapshot  | ssh hyp5 dd of=/dev/HypGroup00/stage
12884901888 bytes (13 GB) copied, 370.274 s, 34.8 MB/s

time dd bs=1M if=/dev/HypGroup00/stage-snapshot  | ssh hyp5 dd bs=1M of=/dev/HypGroup00/stage
12884901888 bytes (13 GB) copied, 372.906 s, 34.6 MB/s

time dd bs=1M if=/dev/HypGroup00/stage-snapshot  | gzip | ssh hyp5 'gzip -d | dd bs=1M of=/dev/HypGroup00/stage'
12884901888 bytes (13 GB) copied, 520.116 s, 24.8 MB/s

Two fast servers were used connected with GigE via a Enterprise GigE switch using local disks via LVM.

Looks like you're using a sledgehammer to crack a nut here - or perhaps a better analogy is trying to cut your lawn with scissors :)

I would strongly suggest you look at some of the tools out there for doing a job like this unless you've got great reasons to do it in-house.

Trinity Rescue Kit is a free liveCD which supports imaging drives over multicast, and might do what you want (or indeed anyone else thinking on the same lines), without going to full-bore imaging systems.