I built a web application that runs on a virtual server for one of my customers. It includes the facility for them to upload files, but they now want to store those files on a server in their office.
The easiest way to do this would be to mount their file system over the internet directly onto the virtual server. I have experience of mounting NFS over an internal network, but I am not clear if it would work in this scenario.
Any thoughts? I can always write new software to transfer the files, but this would be an easy fix!
Note: The server in the office is Ubuntu 12 server running on a virtual server in a windows hyper-v environment. The VPS with the web app is Ubuntu 11.04
No one has mentioned sshfs yet. If you're on a modern linux distro and have ssh access to the remote host, it's as simple as:
Performance is quite acceptable (but not nearly as fast as a streamed sync like rsync if you require the whole directory).
NFS is inherently insecure. It would be a very poor choice for connecting over the internet.
I like the post that mentions rsync. Instead of using cron to fire the transfer, I would hope that you could simply run the rsync job from your code that handles the file upload.
When the upload completes, rsync the file to their server, done.
You would need to set up a secure connection to their server for the transfer, I would expect.
If you wanted to, you could put the incoming files into a list for transfer, removing the names after successful copy, and give yourself some fail over capability, in case something happens to the connection.
As someone else already pointed out, rsync is designed to handle groups of files, or a hierarchy (thanks, spell check) , so this wouldn't be that hard to accomplish.
NFS may be inherently insecure but that is not the fault of the service. Telnet and FTP are also inherently insecure but both have been used for decades on the open Internet. If an encrypted VPN tunnel is established, then the lack of NFS encryption is irrelevant.
Also if a firewall is configured to only allow a specific remote host address to connect to the NFS mountpoint, then a bare NFS connection is mostly secure from hacking, except from Three Letter Agencies that sniff Internet traffic, and which have access to powerful decryption to crack your encrypted remote connections anyway.
I see that some versions of NFS use UDP by default, which is likely where the Internet reliability problems come from. UDP packets can get lost if the connection is congested, and no retransmission is automatically attempted. If you want a reliable connection, make sure your NFS is using (or is forced to use only) TCP packets over the Internet.
NFS can be done through a firewall. If you search, you will find others who have attempted and been successful with this process, however, it is not as straightforward as opening ports and appears to be dependent on the version of NFS being run.
rsync is an excellent suggestion and has a lot of flexibility. However, this would be a process run outside of your app. rsync can handle syncing of local file updates and maintain an entire directory in sync with other folders.
ssh would be another secure option and would only require a single port to be opened. I have used this in the past between Windows and Linux systems and has worked out well both from within an app as well as scheduling the process.
I'm sure there are other ways to accomplish what you are looking for, but we'd need more information about the types of files these are and if you are linking them into a page or streaming them in with something more complex.
NFS mounts can be done using automount, which will continue to try the mount on failure, but the only way I would consider using it over the pipes is with a vpn or some other secure connection (actually, I wouldn't consider it, but you sound fixed on the idea).
Even using automount, if there is a network issue, and you are putting their uploads on the NFS mount, the upload will likely fail or be corrupted, unless you are storing the file locally and then copying on upload success.