Networked filesystem with user level security for linux

I am not sure I fully understand:

"I don't want to rely on machine trust like in NFSv4 because client users will have root privileges."

If you mean the client users will have root privileges on the client and you don't want them to have root on the host then you just don't use the "no_root_squash" option. You can also do things like disabling setuid to help reduce the risk to the client from the server as well.

You also have the option to use kerberos with NFSv4, see this link.

So in other words, NFSv4 might give you the security you need while being a little bit more ... (scalable?) .. the using sshfs everywhere. It still might not be what you want, but I wouldn't give up on it too soon.

With Kerberos, only the kdc server grants authentication tokens. A client machine by itself can only authenticate as a host (and that is if you give it the keytab to a matching nfs/client-hostname@REALM principal), and that only gives it the right to talk to the nfs server. It's the users who have the ability to authenticate, and the nfs server only authorizes them to access their own files. With sec=krb5p the server prevents snooping and alteration as well.

Being root won't give your users inappropriate privileges. The only way they'd get access to more files is by hacking each other's machines, hacking the nfs server or the kdc. NFSv4 with Kerberos fits your security requirements well.

Here is more on the security model:

• Designing an authentication system: a dialogue in four scenes

If you're looking at a deployment, here are some debian/ubuntu centric tutorials. I picked simple setups without LDAP. These distributions have debconf-based configuration that gets you some of the way there.

• Setting up Kerberos (note the fqdn requirements).
• Setting up NFSv4

My additions: you don't need to specify a des-cbc-crc enctype, but you need to allow_weak_crypto in krb5.conf so that the communication protocol can use des-cbc-crc for stream encryption. This will become unnecessary in 2.6.35 kernels.

If you're looking at something appliance-like, there is FreeIPA.

sshfs is the way to go. On the client:

sshfs -o idmap=user,workaround=rename user@server:/home/user/share /home/user/share

Permissions same as ssh... because you are using ssh! Nice thing is you don't have to touch anything on the server, assuming sshd is installed and running properly. Might not have the performance as the other suggestions, but it is very simple.

samba does actually sound like your best bet. samba does have unix extensions so when mounted using as cifs on linux it should show proper unix permissions and whatnot. I think that is going to be the best option for your constraints. if that doesn't work out sshfs could be used in a pinch but it won't have as good performance or integration into the OS as samba.

Does OpenAFS support user level authentication & access?

Yes, it sounds like OpenAFS would satisfy your requirements here, but so would Kerberized NFSv4. In both of those environments, you do not need to "trust" clients; access control is enforced by servers. Previous versions of NFS did require you to effectively "trust" clients, but Kerberized NFSv4 does not. OpenAFS has never required "trusted" clients, and also relies on Kerberos for authentication and security.