Home / server / Questions / 33743
Accepted
letronje
Asked: 2009-07-01 04:14:38 +0800 CST

Optimizing image serving using nfs for a system with a high read to write ratio

  • 772

Our system use NFS primarily for replicating and serving images from multiple nodes.

New images are not added very frequently. Hence writes are rare. Almost all requests will be read requests.

How can i optimize my system to get the maximum read performance(with nfs alone). Any quick reliable tests to verify if system is performing better ?

Are there better ways to serve images quickly ? Can squid/caching mechanism be used on top of nfs mount to speed things up ?

We are already using memory caching modules of apache.

server & client use - nfs v3, all running RHEL 4.

h/w config of server : RAM : 3GB, Quad core Intel(R) Xeon(TM) @ 3 Ghz.

cat /proc/net/rpc/nfsd" on server gives :

rc 0 3584710 221921733 fh 1506 0 0 0 0 io 4274869225 1474485090 th 8 581669 39835.922 7449.989 1396.547 0.000 879.767 604.030 436.573 320.797 0.000 1362.017 ra 16 3454951 13954 4091 8186 3920 6990 6421 3003 5574 2727 2467399 net 225510008 0 225488411 41217 rpc 225506272 0 0 0 0 proc2 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 proc3 22 57 144259928 359775 11723001 58910204 0 5977165 2145920 546253 3829 0 0 509776 3718 15256 165 21497 79725 10067 59 0 867450 proc4 2 0 0

linux nfs apache-2.2 squid images

6 Answers

  1. Best Answer

    Instead of using NFS, consider using rsync to push out the content to all image-serving nodes. Writes can occur at one central point and then be rsynced out either when necessary (i.e, when images are added) or with a cron job.

    This will give better performance overall, and allow for the usual filesystem cache to be used instead of having to implement something on top of NFS.

    • 3

  2. You didn't mention what is your NFS server, nor the nfs clients. I'll suppose that both are some sort of recent Linux systems.

    1. Check raw disk performance on your file server. Generally NFS performance can't be better than about 50% of raw disk performance.
    2. your NFS server network performance should be more than the sum of the clients. For instance, with 10 clients with 100Bt ethernet, have at least 1Gb interface on your server.
    3. optimize your read-ahead and caching settings on the server. I can't help you without knowing the hardware configuration.
    4. Optimize your network : compare NFS over TCP and NFS over UDP. Try jumbo frames if applicable.
    5. Run enough server processes on the NFS server. Ideally you should have as many server as active connections, but YMMV.
    • 3

  3. You say you're using apache's memory caching. Apache also has disk-based caching which could be put to use here. See the disk-based caching section in Apache's Caching Guide. This would require devoting some local disk space to an image cache, and running a utility program every so often to clean the cache.

    Solaris has a caching filesystem that can be used as a layer on top of NFS, but you tagged linux so I don't suppose that's an option for you.

    • 0

  4. You didn't mention whether or not you are considering throwing hardware at the problem, but one possible caching solution would be the NetApp FlexCache products. They're certainly not cheap, but they're made to solve exactly this problem.

    • 0

  5. Disable access time (atime) on the filesystem serving NFS reader.

    Basically, by default, your hard drive is performing a write to update the access time of each file that is being read. Disabling it can lead to improved read performance.

    More info on atime can be found here and here.

    • 0

  6. Sounds like a perfect application for SSD. It would reduce your read latency.

    • -2