Home / server / Questions / 685800
In Process
Dr I
Asked: 2015-04-28 01:27:25 +0800 CST

Transfer large amount of small files

  • 772

I've to do a migration of two servers with large SAN attachments to our new VMWare environment.

EDIT: I have to supply some additional intelligence as I have good answers regarding VMWare solution.

Ok so, I can't attach a previous EMC LUN on the New system due to some technologies limitations on the server.

I can't use VMWare Converter to clone the missing volumes on my new VM as VMWare Converter can't see EMC PowerPath Pseudo-devices and that the previous admin used those Pseudo-device to built LVM2 and/or ASM volumes on top of.

Those two physicals servers are attached to an old EMC² CX-340 SAN and handle 5TB of data.

Those 5TB of data are small PDF and I need to transfer them to the new machine through our 1Gbit/s LAN.

I've tried using rsync, but it's really to slow and have a strong impact on the RAM and CPU performance.

I've try using NC with TAR but the transfer rate is quite slow as I've an average throughput of about 50MB/s on a 1Gbit/s link with barely zero traffic.

Could you give me some advice or return of experience with this kind of migration and how you manage to have it finished correctly within reasonable amount of time?

linux-networking

4 Answers

  1. If you really need a quick way to transfer files, and both systems are Linux-based, you can try UDR.

    This is really a form of rsync-over-UDP (using the open-source UDT framework) and is particularly handy for moving large numbers of files or transferring over high-bandwidth or high-latency links. In addition, encryption is disabled by default, so the RAM/CPU hit is lower than traditional rsync. SSH is not involved either.

    I can easily get wire-speed transfers over 1Gbps with 10-million small TIFF files in a directory tree.

    Your syntax will be slightly modified from rsync. All rsync flags need to appear before the source/destination specification:

    udr rsync -avP --stats --delete /data/ server2:/data/

    Easy to build... You'll need g++ and openssl-devel:

    git clone https://github.com/LabAdvComp/UDR.git
cd UDR/
make
cp src/udr /usr/local/bin/

    Do that on the source and destination.

    See: Possibility of WAN Optimization for SSH traffic

    • 10

  2. If used in daemon mode without encryption, rsync can efficiently transfer large amount of small files. Give it another try using it in daemon mode.

    • 3

  3. Have you not thought of exposing the SAN LUNs directly to the new VMs - this generally works just fine and can be faster than copying the files into a VMDK - though it can 'lock' the VMs onto their initial host. But you could use this to get things going then migrate the files into a VMDK at your own pace - with rsync - and later cut the link to the original LUNs.

    • 1

  4. If the destination VMs aren't yet built, you might try using the free VMware Converter to copy the data over.

    In fact, even if they are built, you could clone the disks to a dummy VM then attach them to existing VM once the clone is done.

    In any event, the converter uses two methods to clone files from source to destination, the full details of which can be found here.

    If the destination disks are configured to be smaller than the source, it will clone individual files into the new VM.

    However, if the destination disks are setup to be equal or larger, it clones blocks. This would make the amount of files on disk pretty much irrelevant and it should run relatively quickly.

    I doubt you'll fill a 1Gbps pipe, but you should get more than 50Mbps.

    Just remember that you're still looking to move 5TB, so it's going to take some time.

    • 1