We are a small company that does video editing, among other things, and need a place to keep backup copies of large media files and make it easy to share them.
I've got a box set up with Ubuntu Server and 4 x 500 GB drives. They're currently set up with Samba as four shared folders that Mac/Windows workstations can see fine, but I want a better solution. There are two major reasons for this:
- 500 GB is not really big enough (some projects are larger)
- It is cumbersome to manage the current setup, because individual hard drives have different amounts of free space and duplicated data (for backup). It is confusing now and that will only get worse once there are multiple servers. ("the project is on sever2 in share4" etc)
So, I need a way to combine hard drives in such a way as to avoid complete data loss with the failure of a single drive, and so users see only a single share on each server. I've done linux software RAID5 and had a bad experience with it, but would try it again. LVM looks ok but it seems like no one uses it. ZFS seems interesting but it is relatively "new".
What is the most efficient and least risky way to to combine the hdd's that is convenient for my users?
Edit: The Goal here is basically to create servers that contain an arbitrary number of hard drives but limit complexity from an end-user perspective. (i.e. they see one "folder" per server) Backing up data is not an issue here, but how each solution responds to hardware failure is a serious concern. That is why I lump RAID, LVM, ZFS, and who-knows-what together.
My prior experience with RAID5 was also on an Ubuntu Server box and there was a tricky and unlikely set of circumstances that led to complete data loss. I could avoid that again but was left with a feeling that I was adding an unnecessary additional point of failure to the system.
I haven't used RAID10 but we are on commodity hardware and the most data drives per box is pretty much fixed at 6. We've got a lot of 500 GB drives and 1.5 TB is pretty small. (Still an option for at least one server, however)
I have no experience with LVM and have read conflicting reports on how it handles drive failure. If a (non-striped) LVM setup could handle a single drive failing and only loose whichever files had a portion stored on that drive (and stored most files on a single drive only) we could even live with that.
But as long as I have to learn something totally new, I may as well go all the way to ZFS. Unlike LVM, though, I would also have to change my operating system (?) so that increases the distance between where I am and where I want to be. I used a version of solaris at uni and wouldn't mind it terribly, though.
On the other end on the IT spectrum, I think I may also explore FreeNAS and/or Openfiler, but that doesn't really solve the how-to-combine-drives issue.
LVM is actually quite heavily used. Basically, LVM sits above the hardware (driver) layer. It doesn't add any redundancy or increased reliability (it relies on the underlying storage system to handle reliability). Instead, it provides a lot of added flexibility and additional features. LVM should never see a disk disappear or fail, because the disk failure should be handled by RAID (be it software or hardware). If you lose a disk and can't continue operating (rebuild the RAID, etc), then you should be going to backups. Trying to recover data from an incomplete array should never be needed (if it is, you need to reevaluate your entire design).
Among the things you get with LVM are the ability to easily grow and shrink partitions/filesystems, the ability to dynamically allocate new partitions, the ability to snapshot existing partitions, and mount the snapshots as read only or writable partitions. Snapshots can be incredibly useful, particularly for things like backups.
Personally, I use LVM for every partition (except /boot) on every box I build, and I've been doing so for the past 4 years. Dealing with non-LVM'ed boxes is a huge pain when you want to add or modify your disk layout. If you're using Linux, you definitely want use LVM. [Note: This above stuff on LVM has been updated to better explain what it is and how it fits into the storage equation.]
As for RAID, I don't do servers without raid. With disk prices as cheap as they are, I'd go with RAID1 or RAID10. Faster, simpler, and much more robust.
Honestly though, unless you're wedded to Ubuntu (which I would normally recommend), or if the box is performing other tasks, you might want to look into OpenFiler. It turns your box into a storage appliance with a web interface and will handle all of the RAID/LVM/etc for you, and allow you to export the storage as SMB, NFS, iSCSI, etc. Slick little setup.
ZFS is really reliable and it sure does make your storage management hell of a lot easier. As a bonus: smb is integrated with ZFS in OpenSolaris and it handles Raid very well. Wait a few days, download the by then released 2009.6 version and give it a go on a test machine. I am sure you will love ZFS.
And about your comment ZFS being new: not very new anymore!
The central question is: "How important is this data?"
If the answer is "I can recreate it easily" you want RAID5, possibly with LVM on top of it for simplicity of management.
If the answer is "I can recreate it but it would take a while and people would complain" you want RAID 6 or more likely RAID 1/10.
If the answer is "Nobody does any work while I recreate it and make sure it's bit-perfect" you want ZFS/Raid-Z
Note that you're always able to recreate it. RAID isn't a backup.
To connect a lot of drives in the same chassis, a hardware RAID controller is the best tool. It will provide lots of SATA connectors for your drives, redondancy via RAID-5 or preferably RAID-6, and may provide better performance too.
Software RAID performance is often better than hardware RAID in benchmarks, however file serving and software RAID are both CPU intensive and compete for your processors while working. My experience shows that unless you use dual quad-core systems, properly configured hardware RAID will beat software RAID hands down.
Good hardware controllers with good linux support :
RAID is NOT like LVM. You can use RAID to make fault-tolerance partitions, but LVM is used to easy disk partitioning and filesystem edition. You can use RAID over LVM or ZFS (ZFS is can work both RAID and LVM). In my opinion, ZFS works better than LVM, but:
On Ubuntu i prefer to use RAID5 MD with LVM.
Take a look at what Nexenta and OpenSolaris are offering and I think you'll be very pleased with that you can get for nothing. It's rumoured that the next releases of OpenFiler will use the FreeBSD ZFS port as well (though, they're quite behind from a feature perspective).
That being said, I try to avoid doing RAID5, RAID6 or RAID50 in software and prefer to use hardware controllers to offload all the XOR work. RAID1 and RAID10 in Linux software work pretty well, and from there I put LVM on top of them to allow more flexibility in what's done with the blocks I have after the redundancy in place. RAID+LVM+XFS is my favorite Linux config., but I'd take ZFS over it anyday.
RAID vs LVM isn't really a good comparison they perform separate roles, and are frequently used together. RAID is used for drive redundancy, LVM can be used to break up your RAID device into logical volumes, it is used for easy resizing, and for taking snapshots.
I ran the file server for a very similar company/situation. Basically a 3 person graphics department with 30TB of storage and the shoestring budget of a small company. Our typical projects ran from a 0.5TB to 6TB. And the file server was serving a sizable rendering farm which could really pound on it.
In my setup I ran a 3U server running Linux with external hardware RAID6 arrays attached to it. I managed the physical and logical volumes with LVM and ran the XFS file system. What I would do is create a logical volume for each project and then expand it as the project grew. When the project was done I could archive the job to tape and blow away the logical volume. This would return that space back to the volume group where it would get reallocated to the next project(s).
This was a very clean way to utilize our storage but there are two drawbacks to this approach. You end up having to micromanage the sizes of the logical volumes trying to balance the amount of space allocated to a logical volume so that you had enough space to do your job but not over allocate it and end up wasting space. Our rendering farm was capable of generating many TB's of data a day and if you didn't pay close attention to this you would run out of space in a hurry. I eventually setup some scripts that monitored the trends in the available space on the logical volumes and would auto grow them. Even with that in place with 80 or so Logical Volumes there was a lot of unused space tied up in all the logical volumes. I've already hinted at the 2nd problem....LVM doesn't really do thin provisioning and XFS only allows you to grow a file system. So over allocation of space to a logical volume can add up to a lot of unusable space.
This was all setup about 5 years ago and if I were setting it up today I would use OpenSolaris and ZFS. The main reason is ZFS pooled storage approach means less volume management. You can still separate each project into its own file system but without having to micromanage the sizes of the individual volumes. ZFS has some other very nice features that make it a better choice but there are other questions on serverfault that go into that.
In my opinion ZFS simply is the best free solution available today.
Some things to consider if you stay with Linux:
Use the "mdadm" utility to create a RAID-5 array out of your drives.
This provides the redundancy you need, so if a drive goes dead you can replace it with no data loss, and also gives you usage of 3 out of the 4 drives.
I'd also recommend you create an LVM volume on top of the RAID, so you can partition the space out as needed.