Samuel Lampa's questions

Is it possible to import a file system owned by a single user, (e.g. mounted via NFS) and export it again as multiple virtual volumes, owned by multiple users, where each volume is a folder on the single-user owned file system imported via NFS?

The file system does not need to have great performance, so even something that does a custom workflow like tar:ing each file that is saved, might be acceptable.

Background

At the HPC Center I'm working at, we are managing data access to an dCache based storage, which is mountable via NFS.

The problem is that we have one big storage account, without possibility to add more user accounts or groups/projects (we have one UNIX group per research project), but still we need to manage access there for hundreds of different users and groups, without access between the groups (one folder per group is enough).

So far we've done this by only letting users interact with the storage through a dropbox-like solution, so that the actual file-moving to the storage is done by our single "robot" user, which is the only one with access to the remote storage. Then we have developed custom scripts that places the files in a file structure that reflects the users' projects, and the date of the "check-in" of a fileset.

I thought that there might be a better solution with some clever import/exporting of file systems though.

Any hard numbers, or rules-of-thumgs, for what workload (e.g. in terms of simultaneously active users running average HPC center jobs) on a computer cluster, that would make a parallel network shared file system such as Lustre, GPFS, Ibrix or Panasas, a requirement?

...that is, when is an NFS shared SAN storage not enough anymore, and you need some kind of parallel NAS?

EDIT: Added "NFS shared" before "SAN".

I have been needing to automate the moving of quite large amounts of data (possibly around 1TB/day+ in average. 60TB so far, and counting) to a separate, remote storage system.

I have so far been doing this by writing custom python code, because of some special challenges related to packing data in suitable chunks for network transfer, and for the protocols available (GridFTP in our case).

I was wondering if there are any material (articles, blog posts, or books - if they are really good) providing best practices and/or hints on how to write data management scripts in a robust and fault-tolerant way?

UPDATE/CLARIFICATION: I should add that we have more or less solved the file uploading problem (with comparing remote and local checksums). We need to handle a lot more than this though, since we are providing a "dropbox" like solution for all the projects at the HPC Center, so we basically manage everything from this dropbox interface, until a user wants a special file back from storage.

So, we need to handle projects, archiving events, meta data for all files, including creating checksums and the like, and all this in a fully automated manner, where one archiving event can consist of 5TB spread out on thousands of medium to small-size files.

This calls for writing our automation software in a robust and fault-tolerant manner, since we cannot really manually keep track of every single file that is stored.

Thus I'm looking more for general hints/best practices for writing software (and in particular for data handling/doing system calls), rather than just how to verify uploads.

Things for which I'd like to see best practices and hints on, are e.g: Exception handling, Logging, Verification, Automated tests etc ... and how to combine all this to get a solution that you can rely on without continually watching it (you should get a mail when you need to, etc).

For apache, there is the htpasswd utility, which can be used to generate encrypted passwords for .htaccess access restriction etc. In Ubuntu I can install it via the apache2-utils package, but in Scientific Linux (Red Hat) I find only the following package, when I do yum search htpasswd:

perl-Apache-Htpasswd.noarch : Manage Unix crypt-style password file

but this does not seem to be the package I'm looking for, since it does not include the htpasswd command, and also when I do apt-cache search htpasswd in Ubuntu, I get:

libapache-htpasswd-perl - Manage Unix crypt-style password file
lighttpd - A fast webserver with minimal memory footprint
nanoweb - HTTP server written in PHP
apache2-utils - utility programs for webservers

... where the first one is quite obviously the one corresponding to the one I found for Red Hat above(?).

So, is there any equivalent to the apache2-utils package, or any other package including the htpassd utility, for Red Hat/Scientific Linux? At least I can't find it ...