Questions[archive](server)

I have a big zip file and I want to know what it's contain. I know I can run:

zipinfo file.zip

but the output is too verbose and there are a lot of files in the sub-directories.

I want to see a list of the files in the top level.

Example

If the normal output is:

-sh-3.2$ zipinfo file.zip
Archive:  file.zip   999999999 bytes   99999 files
-rw-r--r--  2.3 unx     3894 tx defN  3-Jul-11 13:11 file1
drwxr-xr-x  2.3 unx        0 bx stor 23-Feb-12 21:00 dir1/
-rw-r--r--  2.3 unx      269 tx defN 23-Oct-11 14:34 dir1/file2
drwxr-xr-x  2.3 unx        0 bx stor 25-Sep-11 03:53 dir1/subdir1/
...
drwxr-xr-x  2.3 unx        0 bx stor 23-Feb-12 21:00 dir2/
...

I want a command that will output:

-sh-3.2$  <answer>
Archive:  file.zip   999999999 bytes   99999 files
-rw-r--r--  2.3 unx     3894 tx defN  3-Jul-11 13:11 file1
drwxr-xr-x  2.3 unx        0 bx stor 23-Feb-12 21:00 dir1/
drwxr-xr-x  2.3 unx        0 bx stor 23-Feb-12 21:00 dir2/
drwxr-xr-x  2.3 unx        0 bx stor 23-Feb-12 21:00 dir3/
...

I want to archive files (with tar) which are below 3 MB in size. But I also want to retain the directories in which those files exist. (so I cannot use find command). I just want to avoid the files which are above 3 MB in size. How can this be done?

My lab is in the process of setting up a small server that holds data (mostly video and image data, plus a few documents) for the project our group is working on at a moment in time. Historically, after a research project ends, the data haphazardly ends up being archived in one hard drive, or a big pile of DVDs (or CDs in the olden days), and/or some of the video ended up in Sony DV cassettes or even VHS tapes (this lab has been active since the early '90s), OR a mixture of all the above...

Question: What is the best way for (1) consolidating them ALL into the same format AND storage medium, and (2) what's the best medium for long term archiving of such data for very occasional access (say, 30+ years?)? Unfortunately we don't have enterprise level budget (we are just a ~10 people lab), so can't do things that costs hundreds of thousands of dollars.

Thanks!

P.S. Considering our old video and images are of smaller resolution, but recent ones are huge, I think we are talking about 30~40 TB for the really old data, another 10~20 TB for recent data, then yearly additions of about 5 TB.

Here's my problem : I need to archive to tar files a lot ( up to 60 TB) of big files (usually 30 to 40 GB each). I would like to make checksums ( md5, sha1, whatever) of these files before archiving; however not reading every file twice (once for checksumming, twice for tar'ing) is more or less a necessity to achieve a very high archiving performance (LTO-4 wants 120 MB/s sustained, and the backup window is limited).

So I'd need some way to read a file, feeding a checksumming tool on one side, and building a tar to tape on the other side, something along :

tar cf - files | tee tarfile.tar | md5sum -

Except that I don't want the checksum of the whole archive (this sample shell code does just this) but a checksum for each individual file in the archive.

I've studied GNU tar, Pax, Star options. I've looked at the source from Archive::Tar. I see no obvious way to achieve this. It looks like I'll have to hand-build something in C or similar to achieve what I need. Perl/Python/etc simply won't cut it performance-wise, and the various tar programs miss the necessary "plugin architecture". Does anyone know of any existing solution to this before I start code-churning ?

Related to this question about using an SSD for system and HDD for data, except I would like my system to do this automatically...

Is it possible to have several layers of storage and push items automatically between them, using preferably free, open-source software?

I know of hugely expensive enterprise-class solutions like the EMC SAN->EMC Centera automatic archiving, but I was wondering if this sort of staged storage is possible to do automatically.

It would be nice to be able to have several layers in this : Memory->SSD->HDD->slower HDD or tape or some other archive solution.

Are there any filesystems which can do this automatically? (ZFS, Btrfs, HAMMER?)

Any Unix-variants are fine, as I'm interested in how this works and whether it's likely to be portable to Linux or other flavours (BSD etc).