Home / server / Questions / 248105
Accepted
nonot1
Asked: 2011-03-17 05:53:23 +0800 CST

How to upload (and measure) SFTP in parallel?

  • 772

I've got a remote server that I can access via SFTP. (Not FTP over SSL. SFTP.)

Periodically, I need to push a few thousand new files to that machine.

How can I:

  1. Upload upload these in parallel?
  2. Gather statistics, so I can judge the ideal level of parallelization given my particular server/network configuration?

A command line client would be ideal, but a GUI could work as well.

Thank you

ssh sftp copy copssh

2 Answers

  1. Best Answer
    1. Use a client like Filezilla or WinSCP that will allow multiple streams. Or script it in a parallel fashion, using either something multi-threaded, or divide the files into chunks that you can run multiple scripts on.
    2. WinSCP will keep detailed logs, so you can go back through those logs and generate your own stats. Filezilla may, I'm not sure. Your own script would generate the logs that you tell it to.
    • 2

  2. mfinni has given you the best answers I can think of using sftp.

    My suggestion to you is to investigate rsync and see if it meets your needs.
    It can be used to push, pull, synchronize, replace, etc. with appropriate command line options, implements good compression and sends deltas rather than whole files when possible. It also has decent statistics which may meet your needs.

    • 0