We use capistrano for website deployments and our Apache document root is a symlink to a particular code release. The deployment procedure switches the symlink from the old release to the new release as the final step of the deployment.
We are migrating our webservers from real servers running RHEL 5.6 to Amazon EC2 virtual machines running Ubuntu 11.10 and the new servers are suffering from a problem where Apache doesn't immediately notice the change to it's document root when the symlink is switched. It can take a second or so (and I think I've even seen it take a couple of minutes). It's kind of like Apache has cached the physical path of the symlink for some time.
Does anyone know some Apache settings I could look at to get it to "scan" for changes to it's served files quicker.
Thoughts:
- I read that the disks on virtual machines are much slower (since they are network attached storage). Perhaps the filesystem cache somehow works differently too? If so, is there anything that can be done?
- The website runs PHP code. Perhaps there is some PHP config differences between RHEL and Ubuntu? I checked
realpath_cache_ttlbut both servers have it commented out:
e.g.
; Duration of time, in seconds for which to cache realpath information for a given
; file or directory. For systems with rarely changing files, consider increasing this
; value.
; http://www.php.net/manual/en/ini.core.php#ini.realpath-cache-ttl
;realpath_cache_ttl = 120
- We do use the APC opcode cache but don't think it's the issue due to experimentation. The PHP code is in different file paths for each deployment and we ensure
stat=1. - Here is a similar question that is very interesting: 294107 - but doesn't provide an answer for me.
- One solution would be to reload Apache everytime we modify the document root symlink. I'll do this if we can't find another solution.
If it is an EBS-backed image, note that Amazon uses some pretty aggressive caching behind-the-scenes (i.e. your settings mean nothing) to get EBS performance vaguely in the ballpark for common read tasks. Writes are slower to be committed to disk.
I have enjoyed some success with
<command to modify disk> && sync. So it's possible Amazon is trapping the sync system call and forcing EBS to sync as well. Or maybe it's the natural behavior of sync speeding up the process (i.e. pushing in-memory changes to the filesystem immediately also kicks EBS). Unfortunately Amazon is not particularly forthcoming about their EBS tech.If the problem truly lies with Apache, you don't specify if you are doing anything to try to alert Apache to pay attention... a "apachectl -k graceful" will tell each worker thread to finish what it is doing and exit, then launch a new thread to replace it, in theory one that does not have the content cached. (I doubt the destination of the symlink is cached by Apache, it might be cached in the file descriptor table for the process;
syncshould solve that.)