We're currently working on a virtualization setup, and because of features like being able to grow volumes on our filers rather than having to go through the process resizing block devices etc. we'd like to use NFS volumes for everything, essentially, rather than exposing "block devices" through iSCSI or similar technologies.
This does however pose a little problem, as file system caching suffers quite a lot, as we cannot find a way to instruct the Linux NFS driver to treat an NFS volume as a block device in terms of caching. I know this is sort of counter intuitive as to what NFS was designed for, but I'm wondering if there's any known way, without using for example CacheFS, to make the NFS client believe that it has exclusive accesss to the volume and thus can freely cache as much in virtual memory as it wants? We can technically guarantee this exclusivity, so it's just a question of making the client believe us, essentially ;)
For the record all hosts are running Linux using kernel version 3.2 on a Xen hypervisor likewise running on a version 3.2 kernel.
Virtualization on NFS is a well established technology. There are benefits and disadvantages to NFS and block, but you can't pick and choose them. Deduplication, volume resizing, and thin provisioning on block devices are all mature technologies that work fine. Same for NFS.