Home / server / Questions / 157901
In Process
Tom
Asked: 2010-07-07 09:09:21 +0800 CST

lsync, unison or some other inotify auto-syncing tool..?

  • 772

I have an app farm which generates thumbnails and makes them available to a web server farm using nfs. However the performance is so poor so I am going to make a local copy of the files on each web node.

I looked at unison and lsync, but there doesn't seem to be a rpm available from the centOS5.5 repo for either.

which one should I use, or some other strategy. I critically need the sync to be immediate as the user needs to display the thumbnail straight away on a web page, so inotify based tools are good!

Edit: when I say immediate, I mean pushed out rather than pulled on some schedule.

In the end, lsync have released some more versions so I used that, and lsync works well once it is running (ie syncs in a few seconds) however it needs to enumerate each directory beforehand and for a very large file-system takes many hours..

linux centos rsync inotify unison

3 Answers

  1. I critically need the sync to be immediate as the user needs to display the thumbnail straight away on a web page

    So you've no control over server affinity in your load balancing? Time to get a new load balancer.

    I looked at unison and lsync, but there doesn't seem to be a rpm available from the centOS5.5 repo for either.

    Previously I did this by building replication into the application - after all, it's the first thing to know when new content needs to be replicated - it was just a few lines of PHP to publish a notice to the other nodes that content needed fetching - when a node has been offline for a while, I ran rsync before bringing it completely back online (this approach makes handling offline nodes much simpler and non-intrusive). However if I were to implement such a system today then I do it using overlay filesystems - perhaps an NFS at the bottom of the stack and copy up on access.

    See also this post

    HTH

    C.

    • 1

  2. I don't see anything for it in the repository, but you might find that incron may be helpful.

    • 0

  3. I recently installed unison version 2.32.52 on a one of my CentOS 5.x boxes.

    With unison its important to note that all servers that will be synchronizing together will need to be running the exact same version. Unison won't work when linked to other unison versions older and newer.

    For myself I found that this rpm http://rpmfind.net/linux/RPM/dag/redhat/el5/x86_64/unison-2.32.52-1.el5.rf.x86_64.html worked for me running on centos 5.8

    As for lsync I wouldn't be able to tell you much about it as I have never personally used it. I can however say that Unison when used properly is a fantastic tool.

    You can set Unison on a crontab to check for changes every 1-10 minutes.

    • 0