I have an image in my head where a link is too slow to realize the real-time transfer of files, but fast enough to catch up every day. What I'd like to see is a master <-> master setup where when I write a file to Server A, the metadata will transfer to Server B immediately and the file will transfer at idle or immediately when Server B's client tries to read the file before Server A has sent it.
It seems that there are many filesystems which can perform well over fast links, but I don't know of any that do well with a big bottle neck and a few hours of latency.
glusterfs has a healing mode that works as you describe; it would share metadata, but only "heal" the file copies on access. The "nightly sync" process was documented to be:
Due warning: I expect that specific feature will still be available, but I have not used it in a year.
Soooo... the problem you've got there is what do you do when someone on side A wants a file that was written on side B, but hasn't been synced yet?
There's a reason why shared filesystems that work well over fast links are common -- it's possible to define standard semantics for their operation that map reasonably well to the standard POSIX filesystem semantics that we all know and love. Anything where not all of the data is actually available falls pretty much immediately into "domain-specific" territory -- your use-case for a filesystem might be such that you don't need to worry about data being missing (an effectively append-only datastore, for example, or one where you can deal with missing data effectively by recalculating it or fudging the results), but those aren't common enough situations (and are very hard to model, specify, etc) such that nobody has been bothered to write something general purpose that everybody can use.