We have a central repository of files in say server-1 at /srv/www. Then we have N number of servers as well. We want server-1 to be able to deploy its files in /srv/www to all servers as quickly and efficiently as possible.
Is there something like rsync, but instead of specifying a single target, specifying a cluster (N servers) of targets?
I was thinking git may work, but can we specify multiple remotes to push too?
What is the best solution, assuming N servers may eventually reach hundreds.
Well, both Twitter and Facebook have started using bittorrent in their clusters to distribute new code revs. Doing this, they're able to push code to tens of thousands of servers in a very short amount of time compared to old-school centralized deployment methods.
It doesn't sound like you're at that scale yet, but there's no harm in designing your deployment system such that it will not prove to be a bottleneck anytime soon.
I don't recommend git for the scales you're talking to. It can work, but I personally see some deficits with using that model for fetching.
There are a couple of things that determine how best to go about this:
For perfect convergence, and maximum speed you'll have to go with a network file system, such as NFSv4. The clustered filesystems I know about don't scale to 'multiple hundreds' of nodes, so it has to be a network filesystem. This presents its own challenges, but it means that you'll reach convergence the moment the files are updated on the NFS head.
For rapid convergence, you can use some rsync trickery. If the rsync daemon ends up being CPU bound, you can certainly put a couple-three rsync servers behind a loadbalancer like haproxy. Couple that with cron jobs to pull data (or some other method of triggering code updates) and you can hit convergence pretty quickly.
For both of the above, it'll probably be a good idea to put the central repository on 10GbE links for maximum throughput.
An alternate is a push-rsync, where it's run from the central repo to push updates to your servers. It won't converge as fast as either of the above, but will be more friendly to your internal bandwidth. Use multiple hosts pushing to divided ranges for better speed.
rdist may work for you.
[affiliated] Using Kwatee (http://www.kwatee.net) you can deploy on as many servers as you want. Deployments are incremental (only modified files are transmitted) and can be parallelized so its very quick. You can also configure Kwatee to be load-balancer aware so that servers are removed from the LB during updates and then re-inserted. There's a graphical interface to configure the deployment parameters and then deployments can be either manual triggered via GUI or automated using python commands.