Questions[unison](server)

I am syncing a local directory (say "/local") with a cloud storage server using Unison. The cloud storage is mounted to, say, "/mnt" via sshfs (with no-cache option), and I sync the two directories /local and /mnt with Unison locally. [I don't have root access to the server, and Unison is not installed on the server, so a genuine remote sync with Unison is not possible.]

In short, I am looking for a safe way to sync the local directories of multiple computers with the server (including the correct handling of removed files).

I assume that the Unison sync between local directories is safe in principle, as long as nothing happens to the connection. But what if the sshfs connection breaks while Unison is working? Unfortunately I don't understand well enough how sshfs connections work in order to judge whether there is a data loss risk.

Assume Unison is in the process of writing a modified file to /mnt when the connection breaks. Will it quit with an error (which I assume would be safe)? My fear is that it keeps writing to /mnt although the connection is broken, and then marks a file as successfully transferred in its archive although the file has not actually been written to the server. Could this happen?

How about read processes? It seems that when I disconnect while Unison is reading /mnt it always fails with the error message that /mnt has been emptied completely (good!). But is there any situation in which it might get the information that only part of the data in /mnt is missing and then wrongly assumes that these files have been deleted?

EDIT: Reading the Unison manual again, I got aware of the "mountpoint" parameter. However, the way the manual makes it sound, Unison checks for the existence of the mountpoint only once, after detecting updates. So I still see the risk of something going wrong if the connection breaks during file transfer.

EDIT2: Essentially, my question is the same as this - seemingly unanswered - question in the fuse sourceforge mailing list: https://sourceforge.net/p/fuse/mailman/message/1605382/

I'm a little confused on how to run unison to sync files with group and owner attributes on an ubuntu system, as you need to be root. But I need to do this remotely and automated. I know I can set up ssh keys and the such for my user but that doesn't matter because I still wont be able to ssh as root to do the changes on the remote system.

I guess my question is; How would I go about using root on the remote system for unison? or is there a way of setting a command for a user to 'auto-run' as root without password?

Or are there any other ways that I could do this?

Any guidance with this is appreciated.

I need to implement some sort of HA setup where 2 servers need to be able to always be in sync, no matter on which one you write on. The DB part can be covered by a master-master replication setup. However, when it comes to files and content I haven't been able to find something to suit these needs very well. I need to be able to replicate /var/www for example from one machine to the other and be able to write in any of them and always have the same content available no matter where the http request goes to.

• unison: easy to use, simple concept, however it's more like a 2-way rsync, there is no automatic propagation of file changes, unless you run it with its repeat option. I'm not sure how reliable that is. I was hoping for a daemon-like feature that would 'watch' changes to a folder's content.
• glusterfs: easy to configure, a nice project, seems to be perfect for what I need, however, it doesn't seem to handle this 2-way.
• xtreemfs: difficult to configure if you want replication (the docs are a bit hard to follow) and seems to be intended for the "distributed filesystem" part more than for the replication aspect.
• ceph: similar to gluster, but again, don't think it handles 2-way replication.
• mogilefs: not transparent, the application that you build needs to be aware of it and use it's services to access the file system. Not something I can use.

So I'm not sure if either 2-way replication is something that is usually not done and I need to reconsider this or I haven't researched this enough, but I'm at a loss. I don't seem to find other solutions.

Is there anything else out-there that can handle automatic transparent 2-way file replication?

I have an app farm which generates thumbnails and makes them available to a web server farm using nfs. However the performance is so poor so I am going to make a local copy of the files on each web node.

I looked at unison and lsync, but there doesn't seem to be a rpm available from the centOS5.5 repo for either.

which one should I use, or some other strategy. I critically need the sync to be immediate as the user needs to display the thumbnail straight away on a web page, so inotify based tools are good!

Edit: when I say immediate, I mean pushed out rather than pulled on some schedule.

In the end, lsync have released some more versions so I used that, and lsync works well once it is running (ie syncs in a few seconds) however it needs to enumerate each directory beforehand and for a very large file-system takes many hours..

We have a folder structure on our intranet which contains around 800,000 files divvied up into around 4,000 folders. We need to synchronize this to a small cluster of machines in our DMZs. The depth of the structure is very shallow (it never exceeds two levels deep).

Most of the files never change, each day there are a few thousand updated files and 1-2 thousand new files. The data is historical reporting data being maintained where the source data has been purged (i.e. these are finalized reports for which the source data is sufficiently old that we archive and delete it). Synchronizing once per day is sufficient given that it can happen in a reasonable time frame. Reports are generated overnight, and we sync first thing in the morning as a scheduled task.

Obviously since so few of the files change on a regular basis, we can benefit greatly from incremental copy. We have tried Rsync, but that can take as long as eight to twelve hours just to complete the "building file list" operation. It's clear that we are rapidly outgrowing what rsync is capable of (12 hour time frame is much too long).

We had been using another tool called RepliWeb to synchronize the structures, and it can do an incremental transfer in around 45 minutes. However it seems we've exceeded its limit, it has started seeing files show up as deletes when they are not (maybe some internal memory structure has been exhausted, we're not sure).

Has anyone else run into a large scale synchronization project of this sort? Is there something designed to handle massive file structures like this for synchronization?