Home / server / Questions / 331111
Accepted
j0nes
Asked: 2011-11-16 06:10:59 +0800 CST

Serving static files with nginx over NFS?

  • 772

I have a website that is getting about 7k requests per seconds on a nginx server. This server both handles rewrites to an Apache server as well as serving static files, images, etc. directly. Static files is the biggest part in there with about 5k requests.

With an architecture upgrade, I think about using a central file server that exports a directory containing these static files via NFS. There will be no write-access to these files, so the directory could be mounted read-only on the nginx machine. My main concern is:

Is NFS fast enough for this? Is there a limit on how many requests NFS can handle? Are there some "must-have" options when going this way?

Bonus: are there other alternatives for this setup besides NFS?

Thanks!

nfs nginx

4 Answers

  1. I use cachefilesd (and a recent linux kernel, with cachefs) to cache NFS files to a local HD. This way, every read in the nfs will copy the file to a /var/cache/fs dir and next reads will be delivered from there, with the kernel checking in the nfs if the content is still valid.

    This way you can have a central NFS, but without losing the performance of local files

    Cachefilesd will take care of the cleaning of old files when the free size/inodes reach a configured level, so you can serve uncommon data from the NFS and common requests from the HD

    Of course, also use a varnish to cache the more common requests and save the nginx/NFS from serving then.

    Here is a small cachefilesd howto

    • 3
  2. Best Answer

    By setting up a central NFS server you introduce a single point of failure into your design. That alone should be a deal breaker. If not, NFS can be plenty fast enough for a load like this. The critical factors will be having enough RAM to cache files, low latency interconnections (Gig-E or better), and tuning (less so than the previous).

    You should also strongly consider using rsync or a similar tool to keep local copies of the static files updates on each individual webserver. Another option might be a SAN or redundant NFS server solution (both of which are going to be much more complicated and costly than the rsync idea).

    • 2

  3. The speed depends on many factors:

    • How are your servers going to be connected against the NFS target? A single dual-port SAS disk can utilize 6gbit/s of transfer speed. Keep this in mind if you're planning to use 1gig Ethernet (which you can subtract 20% TCP overhead from).
    • What kind of cache is the NFS server going to get? Are you using a enterprise grade array controller with lots of cache? Read cache is key in this setup
    • How many servers are going to access the same file simultaneously? NFS locking can hurt - badly

    The limit of open files via NFS is a limitation of the host operating system. FreeBSD has for example many different tuning options to support a large number of open files, but it depends on the amount of RAM in your server.

    An alternative to a central file server is to use synchronization/replication between your web servers (like Chris S suggests). rsync or DRBD might be a great and cost-effective choice.

    • 1

  4. I would advise against NFS unless you put some Caching in there. The nginx cache is better than nothing, but Varnish is better.

    With that said, if your load would change to be more dynamic content than static, it will become more important to serve apps files from local disk.

    If you put NFS in, make sure that you have redundancy.

    • 1