how to pipe data to sftp connection?

As this is the first result you find by googling for this question and it hasn't been mentioned already I'll add the solution I found here as well:

you can use curls sftp implementation for this. As curl is probably already installed on a lot of systems this might be preferred to the solution using custom clients.

example usage:

pg_dump -d database | pigz -1 | curl -u username -T - sftp://sftpserver/folder/dbbackup.sql.gz

curl uses your .ssh/known_hosts file for key verification. This might fail in case your ssh client does use newer encryption standards not supported by the library used in curl

to fix this you can add the other key types to the known hosts file using the following command:

ssh-keyscan sftpserver >> ~/.ssh/known_hosts

or you can disable key verification using the -k flag (I wouldn't recommend that though)

I had a lot of fun figuring out a solution to this problem. It requires the tool nc (netcat) on both machines, and SSH (SFTP isn't needed).

In this example, I shall call the machine that has the data that needs to backed up linux-a, and the machine that needs to receive the backup linux-b.

On linux-a, have netcat listen on a port (I took 2000) and redirect it to a file. This will just sit there and wait until something comes through on that port.

[kenny@linux-b /var/backups]$ nc -l 2000 > backup.tgz

On linux-b, open up an ssh tunnel to linux-a, I used port 2000 again. This will redirect anything you throw at TCP port 2000 on localhost to TCP port 2000 on linux-a, where netcat is listening.

[kenny@linux-a /var/data]$ ssh -L 2000:localhost:2000 -CfN linux-b

Now create the tar archive, but send the output to stdout (using -) and pipe it to gzip for some compression. Now pipe that to another netcat that sends it to localhost on TCP on port 2000.

[kenny@linux-a /var/data]$ tar cf - important-data | gzip -fc | nc localhost 2000

We're done! On linux-b the netcat is no longer listening, and a new file is created. The best part is that the tar archive was never placed on the hard disk of linux-a.

[kenny@linux-b /var/backups]$ file backup.tgz 
backup.tgz: gzip compressed data, from Unix, last modified: Thu Jul  5 13:48:03 2012

I know it's not exactly what you asked for in the question, but if you have netcat available, its a viable solution to your type of problem.

Edit: I forgot about one thing: if you follow these instructions, you'll still have an SSH tunnel floating around on linux-a. Find out what the process ID is and kill it.

[kenny@linux-a /var/data]$ ps -ef | grep "ssh -L"
kenny     5741     1  0 13:40 ?        00:00:00 ssh -L 2000:localhost:2000 -CfN linux-b
kenny     5940  3360  0 14:13 pts/1    00:00:00 grep --color=auto ssh -L
[kenny@linux-a /var/data]$ kill 5741

output-stream-generating-command | ssh user@remotehost 'input-stream-accepting-command' is an option, if your remote user has a valid shell.

voretaq7 pointed out, that the sftp client doesn't support piped data transfer for users, that are allowed to use sftp only to connect to the server.

fortunately, there is libssh2, that supports sftp. so we just need 2 other clients using libssh2, that i've called:

• sftp_stdin_upload (to upload to a sftp server)
• sftp_stdout_download (to download from a sftp server)

the source code can be found under the following URL: http://www.qxs.ch/2012/07/05/sftp-upload-tool/

since i'm not that experienced in libssh2 programming, i'm happy for any feedback to the source code.