Home / server / Questions / 276139
Accepted
Edward
Asked: 2009-02-04 07:42:50 +0800 CST

Is BitTorrent good for copy files between servers in the workplace?

  • 772

I have 1 source server that contains about 30GB of files that I want to copy to 7 other servers. I currently SCP the files over to the first four servers and when that transfer is complete SCP to the last 3 servers.

Would using BitTorrent be faster to distribute the files to all 7 at once? The connection internally is already fast and consistent between each of the servers. I know it would probably take some load off the source server, but would it just take longer to distribute the files?

What's a good scriptable client I could use from the shell? I'd want everyone to stop seeding after 100%.

shell scp copy bittorrent

4 Answers

  1. Best Answer

    It depends on how your network is laid out and where the bottlenecks are, but yes, it can be faster to use BitTorrent. In theory, instead of sending out 7 copies of the same files, your source server should only need to send out 1 copy, split among the receivers. By the time that happens, the remaining peers will have shared the parts they didn't receive directly among themselves.

    If your network supports multicast, though, it's probably easier to set up udpcast, mdp, multisend, or something like that: your source will send out a single copy of the data, and the network will route it to all the recipients.

    • 7

  2. The advantage of BitTorrent over other file distribution tools, say rsync, is that clients start to offer their data to the other clients, so if the server has a slow connection to the clients, throughput will be much better.

    BT is a protocol which is designed for global sharing, though. So you will have to find a means to lock out all other clients but the 7 you want to feed (or your torrent will soon end up on The Pirate Bay).

    As for stopping at 100%, every BT client will do that: Since no client requests the data anymore, there will be no more seeding. The application will still run but they will only cause a little trickle of data between the clients and the tracker to tell it that they are still available.

    Try the original BitTorrent client which is written in Python. This will allow you to make any modifications you desire. Unfortunately, I can't find a link ATM. Anyone?

    [EDIT] Based on a comment saua: If you use a private tracker, your torrents should not spread unexpectedly. That said, you should only use a torrent client from a reliable source. Otherwise, you might risk that some cracker has put a little extra feature in the code which publishes private torrents somewhere where you don't want them to appear.

    • 3

  3. BitTorrent is a great idea if you want to replicate files through a tree of servers without for example having to issue a remote command to all servers to copy from the same place; potentially causing a bottleneck.

    Twitter engineers done this with great success with Murder, which is now on github.

    http://engineering.twitter.com/2010/07/murder-fast-datacenter-code-deploys.html

    • 1

  4. If all 8 of your servers are plugged into the same section of the network using Bit Torrent is unlikely to speed things up. In fact you would get so many collisions that it would probably be very slow. BitTorrent would only speed things up if you are using a fully switched network and the eight servers can talk to each other independently without affecting the communication between the other servers.

    • 0