Home / server / Questions / 163696
Accepted
krissi
Asked: 2010-07-25 04:06:57 +0800 CST

Stop slowing down fast and verbose operations when using ssh over slow connections

  • 772

I often copy many files between 2 servers connected via 1GBit ethernet while I am connected via VPN and SSH over DSL. The problem is, that the verbose output (1 line per copied file) which needs to be sent over the slow connection to my ssh client and slows down the copy operation (it feels for me this way)

I tested this with a testfile created via

dd if=/dev/urandom | base64 | dd of=testfile count=10M bs=1

Testing run A (no output)

# time sh -c 'cat testfile > /dev/null'
sh -c 'cat testfile > /dev/null'  0.00s user 0.02s system 97% cpu 0.025 total

Testing run B (all output via ssh/vpn)

# time sh -c 'cat testfile'
sh -c 'cat testfile'  0.00s user 0.45s system 0% cpu 4:31.10 total

(I know, its not a good test, but it demonstrates the problem)

Is there a way not to slow down the operation and get the output asyncronously. I imagined something like dropping all lines except a specified count per second or something like this.

At the moment I start screen and detach while the operation is running.

I am using putty for windows and the openssh-client for linux access

Any ideas?

linux ssh slow-connection blocking

3 Answers

  1. Best Answer

    You don't say what you're using to do the copy, but I suppose it doesn't really matter.

    You might try something like this using pv:

    your-copy-command 2>&1 | tee progress.out | pv -trlb >/dev/null

    That gives you a low-overhead progress indicator:

    502 0:00:20 [25.1/s ]

    which shows the number of lines of filenames (not file contents) that you would have seen otherwise. The tee command saves this output on the remote system in case you do need to see it. Then the output is discarded.

    Here's a way to do something similar without pv

    your-copy-command 2>&1 | tee progress.out | progress >/dev/null

    where progress is a script something like:

    #!/bin/bash
SECONDS=0    # auto update variable
while read -r line
do
    if ((count++ % 10 == 9))
    then
        printf "%s sec, %s lines\r" $SECONDS $count >&2
    fi
    echo "$line"
done
    • 2

  2. what happens if you pipe the output into less ?, as in cp a b | less

    • 0

  3. If I'm understanding this correctly, it sounds like you're slowing down due to terminal buffering!

    Do you need to see the output of the transfer? If not, use the -q or appropriate options to silence the per-file transfer output. If you need that data, redirect the output to a file for later review. You could then tail -f that file.

    • 0