Home / server / Questions / 300086
Accepted
ErnieTheGeek
Asked: 2011-08-12 05:18:21 +0800 CST

File Transfer Alternatives

  • 772

Right now we're using the enterprise product from Aspera. We've got a license that covers only 10megabits of throughput per connection. We're looking at migrating away from Aspera as their licensing costs are a killer to get us up to 100megabits. I've been looking around and cant really find any alternatives. Right now we transfer something along the lines of 500gb a day in and out. I'd prefer something from the FOSS camp but I'm not against purchasing something Windows based.

To provide you some background on our hardware config.

(2) BL495c Blade Servers - 2x 6 core Opterons, 128gb RAM, Storage is all on HP MSA2300 FC arrays, about 10TB of data in total. Everything is running on Hyper-V Server 2008r2.

So with all of that out there the question really is:

What software/methods do you use to provide highspeed file uploads/downloads to your clients?

Edit: I'm dealing with mostly non technical end users who all live in the Windows world. So I'd like something browser based with a fairly simplistic user front end. The back end can be as technical as necessary. I think the only real requirements I've got are performance capability of approx 100megabit file transfers, user quotas, and possibly connection throttling.

file-transfer upload

5 Answers

  1. We use SFTP/SCP or rsync-over-SSH, but that depends on the client having the relevant client/server software at there end.

    It goes as fast as out links will allow, with rsync interrupted transfers are restartable, and it is all nicely scriptable (assuming you know your bash/equivalent scripting and related tools) so things can be completely automated. You'll want to rent a server somewhere to host the files rather than transferring them direct from you to the client unless you have a 100Mbit pipe locally, but that won't be expensive unless you need significant SLAs (remember to use encrypted filesystems and other such precautions if the information is sensitive, or course).

    This is bit of a DIY solution though. If you are looking for something that presents itself with a friendlier face to your clients (something akin to dropbox for example) then you'll need to add some details about what sort of thing you are looking for to your question so you get relevant answers. Remember that most of use won't have used Aspera so won't be familiar with it, and even those that do use it won't know which features you find particularly useful (and which are no concern to you) if you don't list them.

    Edit:

    This tool was recently on HN (here), which looks like it might be worth you investigating if you do have a fat outgoing connection. Obviously you'll have to look into the security side of things, such as keeping your clients separate from each other by some means (as it seems geared around synchronising personal files between locations, rather than sharing them with clients). Caveat: I've not tried it at all myself as yet.

    • 5
  2. Best Answer

    Have you considered other commercial accelerated file transfer solutions?

    Take a look at FileCatalyst www.filecatalyst.com it will give you the same speed advantage at a more affordable cost.

    You could also look at Signiant on the commercial front.

    On the open source side there are also a few options. UDT, Tsunami, GridFTP are all possible options. Also take a look at UFTP as well. I think the problem with these solutions will be the fact that you are dealing with non-technical end users. Perhaps with some scripting and one of those products running in the background you could create a working solutions.

    On my blog, I compiled a comparison of all the open source file transfer products and I tried to identify which one will work best in certain situations (ex: high bandwidth, high latency scenario etc)

    Here is the link to the blog article: http://www.filecatalyst.com/open-source-fast-file-transfers

    John

    note: This user is an employee of FileCatalyst.

    • 2

  3. Something from HPC that is used if your data sets are approaching PB size and your pipes are fat but have high RTT: GridFTP - parallel TCP streams, can be striped over multiple servers. Maybe a bit of an overkill for your application but it won't get faster than GridFTP. Depending on your security policies it may require a full blown X509 PKI.

    • 1

  4. 9 years later, the OP will have found a solution, but anyway:

    Filesender is an open source PHP/javascript solution which seems to work well to let people transfer files through their browser.

    We use it to send and receive files from people who are not used to FTP, particularly when the files are too large for the services they are used to like Dropbox, Wetransfer, etc.

    Logged-in users can send download links to anyone, and can also generate "guest vouchers" to send to people to do uploads.

    The default maximum file size is 100 GB., but it can be configured to whatever suits your needs (we increased it to 250 GB).

    The source is on Github: https://github.com/filesender/filesender#readme

    • 1

  5. like john mention, filecatalyst is an option. Please correct me if i am wrong, but from my reading, i believe filecatalyst lacks the control of bandwidth consumption. it will consume all available bandwidth. however, file catalyst provides the ability to schedule it "off-peak" as an alternative to bandwidth management.

    other solutions are open source UDT or Tsunami which filecatalyst is built upon.

    martin

    • 0