Home / ubuntu / Questions / 1078468
In Process
3kstc
Asked: 2018-09-26 16:40:08 +0800 CST

How to put files and delete them straight after by means of SFTP in a Bash script?

  • 772

Brief:

I have a directory where files are dropped arbitrarily in server A. I want to dump these files into server B and delete them from server A.

Background:

I would like write an Bash script where the code iterates through each file, puts the file across to server B, and deletes it straight after. This will be ammended to the cron and called hourly to scrape the files across to server B

Code:

The code I have attempted is:

#!/bin/bash

MY_SCRIPT_NAME=`basename "$0"`
PATH_TO_METRICS='/home/some/directory'

if pidof -o %PPID -x $MY_SCRIPT_NAME > /dev/null; then
    echo "$MY_SCRIPT_NAME already running; exiting"
    exit 1
fi

sftp -i my_priv_key -oPort=12345 [email protected] <<EndOfTransfer
cd $PATH_TO_METRICS
for filename in $PATH_TO_METRICS; do
   PUT "$filename"
   rm "$filename"
done

EndOfTransfer

Relevant Research:

Question:

How does one iterate through files in a given a directory and use sftp to put each of the files across to another server, and delete the file if a successful transfer has happened?

bash sftp

2 Answers

  1. You need to understand <<EOF and EOF

    In the below example <<EOF after the sftp command has the meaning, pass everything to the sftp program as standard input untill you get to EOF (End Of File).

    So you could easily modify your script with this and get it working.

    sftp YourSftpServer <<EOF
put YourFile
exit
EOF
    • 1

  2. There are two ways to do this that are much easier than using sftp itself, since you're using SSH in either case. You only need to use SFTP if you're actually running an "old school" SFTP server. If Server A/B are actually Linux machines, just use rsync or scp

    rsync --remove-source-files -r /home/some/directory user@ServerB:/somewhere/else

    This will copy all the files and has much better performance than sftp commands or scp since they typically open/close many connections. rsync will open a single connection, check to see what needs to be transfered, then transfer it all without fragmented connections.

    If for whatever reason you need to pass your private key or use custom ports to SSH you can pass the -e "ssh -i foo.key -p 1234" to rsync and it will use those keys.

    • 0