Questions[synchronization](server)

At a company I work for we have such a thing called "playlists" which are small files ~100-300 bytes each. There's about a million of them. About 100,000 of them get changed every hour. These playlists need to be uploaded to 10 other remote servers on different continents every hour and it needs to happen quick in under 2 mins ideally. It's very important that files that are deleted on the master are also deleted on all the replicas. We currently use Linux for our infrastructure.

I was thinking about trying rsync with the -W option to copy whole files without comparing contents. I haven't tried it yet but maybe people who have more experience with rsync could tell me if it's a viable option?

What other options are worth considering?

Update: I have chosen the lsyncd option as the answer but only because it was the most popular. Other suggested alternatives are also valid in their own way.

Currently I have two directories A/ and B/ which are identical in every respect, with the exception of the timestamps. Therefore if I run the command :

rsync --dry-run -crvv A/ B/

then all files are marked "uptodate", whereas the command :

rsync --dry-run -rvv A/ B/

shows that all files are to be copied over from A/ to B/.

My question is this : given that I know the files are identical (in respect to contents), then is there any way (via rsync or otherwise) to set the timestamps for files in B/ to be identical to the timestamps of the files in A/, without copying over all the files from A/ to B/ ?

Thanks

I am writing a Linux shell script to copy a local directory to a remote server (removing any existing files).

Local server: ftp and lftp commands are available, no ncftp or any graphical tools.

Remote server: only accessible via FTP. No rsync nor SSH nor FXP.

I am thinking about listing local and remote files to generate a lftp script and then run it. Is there a better way?

Note: Uploading only modified files would be a plus, but not required.

I want to interrupt a running resync operation on a debian squeeze software raid. (This is the regular scheduled compare resync. The raid array is still clean in such a case. Do not confuse this with a rebuild after a disk failed and was replaced.)

How to stop this scheduled resync operation while it is running? Another raid array is "resync pending", because they all get checked on the same day (sunday night) one after another. I want a complete stop of this sunday night resyncing.

[Edit: sudo kill -9 1010 doesn't stop it, 1010 is the PID of the md2_resync process]

I would also like to know how I can control the intervals between resyncs and the remainig time till the next one.

[Edit2: What I did now was to make the resync go very slow, so it does not disturb anymore:

sudo sysctl -w dev.raid.speed_limit_max=1000

taken from http://www.cyberciti.biz/tips/linux-raid-increase-resync-rebuild-speed.html

During the night I will set it back to a high value, so the resync can terminate.

This workaround is fine for most situations, nonetheless it would be interesting to know if what I asked is possible. For example it does not seem to be possible to grow an array, while it is resyncing or resyncing "pending"]

I would like to sync a directory of an internal hard drive to an external hard drive.

When I write or modify a file on my internal hard drive, automatically the same file would be copied to the external hard drive... and bye bye backup ;)

This would be something like a RAID with an external hard drive.

Is this possible?