We have two main environments in question:
Development and QA
Each environment has two servers:
- Jump Box
- Application server
In order to connect to the application server, you must connect to the jump box first, and then SSH to the Application server.
There are a few rules in place courtesy of the firewall:
- You MUST connect to the application server via the jump box
- The application server cannot connect to either jump boxes
- The jump boxes are on the same subnet, and CAN talk to each other.
Our Problem
We have a lot of content (670 GB) on the DEVELOPMENT APPLICATION SERVER
, and we need to get this to the QA APPLICATION SERVER
.
Copying this data to the jump boxes is not an option because they lack the required amount of space.
I did some research, and learned that we could potentially do a series of tunnels through these servers so that we can stream the data straight from one app server to the other via the tunnels. However, the problem that we cannot connect to the jump box from the application server.
Do we have any options? This is getting to be a desperate situation, and time is of the essence. We do not have time to download the data and re-upload it. Copying across the network on the servers will go quickly, as it is a gigabit connection.
By far, the easiest way is to just copy it via scp. Plus, this syntax actually works unlike some of the other suggestions.
You can't beat this syntax for ease. It allows you to recursively copy, rsync or what ever you'd like without the hassle of considering potentially complex pipes. This syntax is intuitively clear, will be more readily supportable by Sys Admins that follow you and does not make useless use of cat.
From the scp man page:
-3
Copies between two remote hosts are transferred through the local host. Without this option the data is copied directly between the two remote hosts. Note that this option disables the progress meter.In the example below
Here is an ~/.ssh/config file that sets up the direct access from your workstation to the application servers via the appropriate jump (aka bastion server).
Testing for presence of file on target server, it won't be there.
Now let's copy a file from Dev Application server to QA Application via your workstation.
Now let's check for the presence of the copied file on the QA Application Server. It will be there this time.
Note
When closing a ProxyCommand connection, you will see the warning message "Killed by signal 1". This is SSH tearing down the ProxyCommand connection and is nothing to be alarmed about. You can get rid of it by adding
LogLevel Quiet
to your bastion host config stanza.PIPES!
If the internet is a series of tubes, Unix is a series of pipes -- something like:
should work.
If you need to traverse more hosts, add more pipes (and more nested layers of
\
-escaped quotation) as needed. (Note however that if the pipeline/escaping gets so complex that you have to draw a diagram or resort to counting on your fingers to determine how many times you have to double up on escapes it's probably time to admit defeat and set up a proper VPN!)If I understand correctly, you have two jump servers (jump-qa and jump-dev) protecting two app servers (app-qa and app-dev); the jump servers can ssh to each other; no box other than the relevant jump server can ssh to the corresponding app server. The app servers can ssh to noone. A file is to be transferred from app-dev to app-qa. Both jump servers lack the space for an interim copy of the data.
You can solve this with ssh tunneling. We set up a connection to one remote app server, carrying a remote tunnel that connects back to an unused port on its jump server. We set up a second connection from one jump server to the other jump server, carrying a tunnel that picks up the dangling end of the remotely-forwarded port from tunnel one and sends it on to the other app server's ssh port.
Set up the tunnels (each of these commmands will need to be run in a separate window on
jump-qa
):You should now find that on app-qa, you can do
telnet localhost 2345
and get app-dev's ssh banner. You may then copy the datafile:OpenSSH v7.3 onward supports
-J
.For A → B → C, on A, just:
-J
option.-C
to leave the files on home folder.-C
(or tar's-z
). Particularly useful if the data is plain text (uncompressed).pv
monitor the progress of data through a pipe. An alternative could beprogress
.Inspired on Florian Fida and Dan Garthwaite's answers.