Home / server / Questions / 26927
In Process
jimconstable
Asked: 2009-06-17 15:26:48 +0800 CST

Large File Copy Tool accross Flaky Connection

  • 772

I am looking to copy some large files accross a flaky WAN connection. Is there a tool I can use which will auto resume when the network fall over briefly and let me manually resume if I have to fix it manually?

windows synchronization wide-area-network copying

8 Answers

  1. The native windows tools to do this are xcopy /z and using BITS. Xcopy /z is pretty self explanatory. Bitsadmin is how you manage a bits tranfer in windows 2003, howver if you can install powershell v2 it contains cmdlet modules to do bits transfers. See Transferring (Large) Files Using BITs. My favorite exe tool is XXcopy.

    • 6

  2. Use something like rar to create an archive that can be broken up into chunks, say 100MB. Use xcopy or robocopy to copy the files across.

    robocopy source dest *.rar /w:30 /r:30 /z

    will wait 30 seconds when the connection drops and retry 30 times.

    I've used that when I need to copy a multi-gig file across a link that won't stay up long enough to do the copy.

    • 3

  3. curl has several retry options (along with being able to specify infinite retries).

    rsync doesn't have any built in retry schemes, you could wrap it in a script to do infinite retries until it completed successfully.

    • 1

  4. FTP client with queue function (with auto retries). Try SmartFTP or FileZilla.

    • 1

  5. My friend suggested I use KillCopy for this purpose. It works well - its user interface is a bit clunky, but it's a drag-and-drop solution (assuming you're using a destination you can access in Explorer).

    • 1

  6. How short are the network outages?

    If they are not long, then OpenVPN might be useful. I use it in various places for other reasons, but when using it over a slow unreliably GPRS connection on a moving train I found it did a good job of surviving the regular connection drops and, if the outage was not long enough to cause timeouts, the applications (web browser, mainly) transferring stuff over the link survived too.

    This in combination with a client that supports partial sends and retrys, as suggested by others, might cover all the bases for you.

    If using rsync, make sure you have the options set such that it can reuse partial transfers otherwise you may waste much bandwidth if the connection drops just before the end of synchronising a large new (or much changed) file.

    • 0

  7. It might seem like a strange suggestion, but use uTorrent. Instructions on how to do it are here:

    http://www.poromenos.org/tutorials/bittorrent

    It is great with lousy connections. I am assuming you can start the download from the other side. If not, then this idea won't work.

    • 0

  8. If you're on Windows you may want to take a look at RichCopy a 'successor' to RoboCopy mentioned previously.

    Except copying files is not always that simple. What if you're copying thousands of files across a slow connection? What happens if your network hiccups and interrupts the copy? What if you want to make sure that you preserve particular file attributes, such as a Last Modified date, but not other attributes, like security descriptors? What if you want to filter the files you're copying from source to destination based on filename or extension? If any of these apply to you, the simple Ctrl+C/Ctrl+V combination probably won't cut it. And though Robocopy GUI certainly improves on that basic functionality, it still has some limitations in terms of granular control and usability. What you need is RichCopy.

    • 0