I have a 3-layer web solution like this:
- Frontend with load balancing + proxies + static content
- Backend with 2 Apache web servers each one serving different sites
- Publishing System that pushes content to the apache web servers
So I am working in a solution with High Availability for the web servers in the backend. My idea is to replicate the content between the backend servers and if one fails the other will serve all the sites (this could be manual or using Heartbeat).
Problem is that the sites are big in terms of total size and number of files. I try to replicate the contents between servers with rsync but it takes a long time. Also I thought of using NFS to share the contents but this is not an option for High Availability. Another way is for the publishing system to push content to both web servers, but what will happen if I put another web server in the backend?
Is there a better way to do this? I don't need both servers serving the same content at the same time, but having the same content synchronized is a must.
You really should consider DRBD (RAID-1 over TCP/IP) with a multi-node filesystem such as OCFS or GFS.
You can also consider getting a SAN on which you will be able to put any one of these filesystems as well.
Use a SAN instead a NFS server, RAID will handle the high availability.
You could use HAProxy + Keepalived for the load balancers. For the replication think about optical link, if ethernet is not effcient for your needs. RSync is very efficient IMAO (with the "-z" options which compress datas, it find it very efficient). At least, if you want HIGH performance, you could host the two Apache as VM's on the same server, and add some nice disks (15K rpm) with a nice raid card. That should provide you the availability you are looking for
I use Heartbeat2 on Debian Lenny for failover and it works very well. I have a web application that is being served by one web server which will failover to another if a problem (eg. a 2-node active-passive cluster). The web application data is on the filesystem and also in a MySQL database. We use MySQL in Master-Master replication mode to handle the mirroring of the database application data. The rest is handled by rsync when we push an update live. This set-up has been working in production for the last 6 months and has worked well in real-life incidents. I think we have added another 9 to our overall uptime due to this.
I'm suprised your rsync is taking a long time given that your web servers are presumably in the same datacentre or in the same country unless they are large files like ISOs. It might be worth checking what rsync options you're using to see if this can be optimized.