I am setting up a 16TB Storage server (which will eventually double in capacity) to store media. This server needs to be read from by two windows server 2008 clients which host our companies applications. I've heard of iSCSI but it seems from what I've read that iScsi treats the target server as if it was directly connected hard drive thus making it very hard to share with two clients as they would randomly write and read data which would mess things up.
So, what software / hardware combination can I use to get high read speeds on the windows side? Right now I am considering 2 dual gigabit ethernet nics with TOE and NFS but I don't want to purchase the nics until I'm sure that it will be fast enough.
Thanks
Starting out: Determine a price for the data. The cost of losing the data, or loss of access per day.
Usage: Determine how many individual read and write accesses you will receive in peak times. Also determine the data rates for reading and writing.
Network: With this amount of data, you should look into 10Gb Ethernet. Do the math how long it will take to copy the data there.
Raid: On Raids of this size, automatic rebuild is not a sane idea. The rebuilds take so long, and the chances of finding a defect on an additional disc are big enough, that it is better to avoid the automatics. Go for Raid 10, or Raid 6 at least.
Sharing: If you do need backup for the thing, it is probably best to mirror the storage. Look into something with realtime replication. A NAS or SAN is probably the best way to go. Or you look into clustering your MS Servers.
Experience: Build a mockup. You are planning a complicated setup. Try to break it. You will be astounded how easy it is to put this offline if not properly planned.
Ok, so let me get this straight. You have direct-attached storage on a server but you want to share files on it with two other Windows 2008 servers? Why not use Samba and be done with it? Samba basically turns your Solaris server into a domain-member file server. Just install Samba, set up your file shares, and let the Windows 2008 servers access the files from there.
If you are wanting to do some kind of shared-cluster solution, you'll need to use a clustering filesystem. Yes, iSCSI is meant to provide block-level sharing. However, with a clustering filesystem, you would use something like iSCSI as a layer beneath it to share physical disk space.
I guess the issue is that you need to clarify what the 2008 servers will use the Solaris install for. This would at least give an idea about what kind of sharing is involved.
That's not a small amount of storage. You're talking about at least 10 disks (2 TB drives, RAID 5 + 1 hot spare) and probably 20, maybe more. And you say that it will double over the lifetime of the box. I wont say that you can't find a server to hold all those disks but you will have very few choices and will probably end up with a couple of external drive trays in any case. Rolling your own single server solution for this is certainly possible but you have to ask yourself if you are going to be comfortable with all of that data hanging off something with limited redundancy and minimal support.
You should take a look at pricing for entry level iSCSI SAN\NAS storage systems to get an idea what they will cost and see if they can deliver what you need. At the very least you will get a ballpark number for what a shrink wrapped commercial solution costs.
You are correct that iSCSI wont allow you to share between two Windows servers in an Active\Active sense but you could use iSCSI to present the storage to one of the servers and have that server share out the volume(s) so the second server could access them with SMB. You could also opt to just share out directly using SMB - pretty much all NAS boxes will do this too - and just connect both servers that way.
In terms of your nic choices my answer would be that if the data availability is important you should be considering at least two on the storage server in any case, maybe more with that amount of storage. And at least two per target server, with redundant switches in the mix as well to ensure that you have no single points of failure regardless of the performance requirements.
If you find that you need more performance than single GigE links can provide (ie bandwidth over about 80-100Megabyte/sec or random IOPS (4K blocks) >~8000) be sure to check that whatever solution you do select does actually support some mechanism to make use of multiple nics per connection (multi-pathing/channel bonding/link aggregation etc) that Windows 2008 and your choice of network hardware also support.
Just one small thing: if you go for speed and availability, forget RAID 5. Go with mirroring, 16TB is not so expensive to reach.
You'll get spread reading (up to double throughput) and when a disk fails, only part has to be rebuild with a least inexpensive operation (no special checksums have to be calculated).
Plus, up to half of the drives can go dead in best worst case™.
Otherwise I've been experimenting at home with about 360GB of mirrored SATA, Solaris on Intel and OS X client and Gbit ethernet. All normal hardware, no server components; set up jumbo frames and got to 58MB/s with SMB, very close to integrated iMac disk.
NFS should be even faster, but not on every platform, the one on OSX is not very good, don't know for windows.
Also, for tuning you have to know what kind of load you'll do to pick right drives (small number of bigger or larger number of small drives), buffers, packet sizes, ...