Home / server / Questions / 254893
In Process
edotan
Asked: 2011-04-03 02:43:55 +0800 CST

1TB of folder I would like to copy to another server

  • 772

I have a 1TB folder with many files on Centos server I need to make sure another Centos server has the same folder and files for backup. right now the backup server has no files. once I copy all the folders I guess I can use "rsync" but for the first copy I think I'll need something else I have 2 NIC on both servers in cross cable which I would like to use for the backup. any good idea? thanks

centos networking storage

6 Answers

  1. you can also tar and untar over ssh in a single command, this is good for only one trancefer you can use rsync after you do a full copy 

    cd /data ; tar cf - . | ssh -C -c blowfish root@server2 "cd /data ; tar xf - "

    I am not sure, but it could be faster then rsync when there are a lot of small files

    • 6

  2. There is nothing wrong with using rsync for that - if it takes that long, just start it on a Friday ;) .

    In some cases, if there are a lot of small files and the latency is high, it might be advisable to tar the whole tree, copy the tarball and untar it at the target system. For good measure, you can also calculate checksums of the tarball on both systems.

    • 2

  3. rsync is perfectly fine for initial backup too. If you have two NICs and slow ethernet switches (or those servers are on different networks), use crossover cables and rsync. Additionally, if you have something performance critical going on, using crossover cable on secondary NIC is good idea, as it offloads traffic to separate card.

    If you have 1Gbit/s network cards, then it's actually much faster to use rsync than buying external disk.

    Already mentioned solution with tar - if there is any glitch, it'll fail, and you have to start over from the beginning.

    I really, really recommend using rsync,

    rsync -avz --progress /srv/source -e ssh [email protected]:/srv/destination

    Even if you have 100Mbit/s network, using USB disks is bad solution - it's something like 30MB/s for both writing and reading - 15MB/s in average, plus you have to manually move it and run same commands again. 100Mbit/s is about 10MB/s (some network overhead included).

    • 2

  4. Do rsync from different sessions simultaneously:

    rsync -avz --progress --stats /home/[a-l]* root@remotebkpserver:/backup/home

rsync -avz --progress --stats /home/[m-z]* root@remotebkpserver:/backup/home

rsync -avz --progress --stats /var root@remotebkpserver:/backup
    • 1

  5. 1000M ethernet nics doesn't require cross cable for peer-to-peer connection. you can use standard cat5 cable to connect between those servers. then you can use 1000M instead of 100M on the crossed cable.

    • 0

  6. Buy a 1Tb disk. Plug it into server 1, preferably using SATA or eSATA, although USB is fine too. Copy all the files to it. Plug it into server2. Your files have now arrived. Much faster than rsync.

    Here's a picture I took today while doing this very thing: 1Tb drive in colo box

    • -5