My filer currently has two basic md-type software raid 5 sets. One 8 x 500 GB and one 4 x 1TB. To make the configuration a little more flexible I'm considering adding LVM. Before I do, I figured I should also look into ZFS.
Is ZFS available on Ubuntu, and can it replace md-raid and lvm?
ZFS was released with the CDDL license, so it cannot ever be built in to the Linux kernel.
It is possible to install ZFS on Ubuntu, and several people have released howto documents. If you Google search for "ZFS Ubuntu" you find several. Here's one: http://systembash.com/content/howto-installing-zfs-and-setting-up-a-raid-z-array-on-ubuntu/
ZFS on Ubuntu works with the "file system in userspace" (FUSE) system, rather than being a kernel module. I believe this means you will not be able to boot from ZFS, but you should be able to keep user data in it. Here is the hope page for the ZFS on FUSE project: http://www.wizy.org/wiki/ZFS_on_FUSE
ZFS has its own ways of being redundant, so yes it replaces md-raid and lvm.
Note that Btrfs is coming soon, and it will be the native Linux answer to ZFS. It is available now, but I would not advise you trusting important data to it. I expect that Btrfs will be solid in perhaps another year.
I will stick with ext4 on RAID until Btrfs is out of beta status. I would also be willing to install OpenSolaris on a spare computer and use it as a ZFS file server with NFS. I am reluctant to trust important data to beta software, and ZFS on FUSE is still considered beta.
EDIT: I'm updating this to recommend ext4 instead of ext3. Sadly, I think that BTRFS is still at least a year away from being ready, so I'm not editing that part. (BTRFS still doesn't have a working fsck that can fix errors yet!)
I would not use ZFS on Linux through FUSE. Development mostly stopped on it as far as I know and it's not close to out of beta, nowhere near ZFS on Solaris. Btrfs should be a great replacement once it gets out of development, but it's very much immature software that's hardly ready for...well...anything. I'll be sticking to ext4/Reiser4 for my Linux storage needs.
No, ZFS is not available for Linux (as a kernel-integrated filesystem). The main reason is the incompatible license. I would not recommend the user space solution with FUSE for any serious use.
The thing that comes very close to your requirements is Nexenta. It combines the OpenSolaris kernel with the Ubuntu userland. You should give it a try because ZFS is (unfortunately) miles ahead of Linux's md-raid and lvm.
Other ZFS options are:
For now it is very easy to use ZFS (Pool Version 28) for Ubuntu as kernel module.
Here is complete guide ZFS Stable Releases for Ubuntu. Also you can check out ZFS on Linux project site.
Bam! http://zfsonlinux.org/
Pre build packages for most debian based systems. They also give a bit "hand wavy" work around to the license question.
Native kernel-level ZFS is in Debian now: http://tucobsd.blogspot.com/2010/08/apt-get-install-zfsutils.html
There is a kernel implementation of ZFS for Linux, it can be found at http://wiki.github.com/behlendorf/zfs/.
I haven't used it tough nor even properly read it's documentation, so I haven't got the slightest idea about it's state or usability.
But maybe it's helpful for you.