Home / server / Questions / 80850
Accepted
Roy
Asked: 2009-11-04 00:20:43 +0800 CST

Replacement for FTP?

  • 772

I have some fairly large file transfers going across the atlantic every morning by ftp. From time to time the transfers will be distrupted, likely due connectivity issues, and the files will be left in an incomplete state.

I'm wondering if there is a better way to handle these file transfers. Would say, rsync or scp be more resilient to periodic loss of connectivity? The best would be if the transfer would reconnect and continue where it left off, should it be disrupted.

Large in this case means files up to several GiB and transfers spanning several hours.

linux

6 Answers

  1. Best Answer

    I belevei this is a case where rsync should work well. Rsync does a very good job at resuming where it left off when a transfer is incomplete. If an older version of the file already exists on the remote side rsync is really good at only transmitting the differences.

    I don't really think scp is what you want. If you need to transmit over ssh, then you can pass the option -e ssh to rsync and ssh will be used as the transport.

    • 10

  2. I second the rsync option, however if you want it to resume a transfer of a big file, then don't forget to use the --partial command-line option which will mean it doesn't delete the temporary file it was writing to when the link goes down. You'll probably have to call it in a loop which retries when the link goes down.

    Something like

    until rsync --partial -e ssh sourcefile remote:destination; do 
 echo Retrying; 
done
    • 7

  3. Why not BitTorrent? You get multiple bonuses if you use it. At least reliable transfers and availability to seed from multiple locations.

    • 2

  4. Definitely use rsync with -P - but for better performance over the Atlantic you probably want to use one of the UDP based file transfer tools - Aspera, FileCatalyst - or the open source Tsunami UDP.

    • 1

  5. Maybe Direct Connect (or other P2P protocol) is the best choice for you?

    • 0

  6. Have you checked out accellion? They are a ftp/webmail hybred. We use it for our graphics designers to securely and easily send large files to the printers.

    They offer a free virtual appliance. It also has encryption, delivery notification and forwarding prevention, 2GB max file size, all built-in for free.

    If you pay you can setup automation and allow up to 10GB files.

    • 0