Home / server / Questions / 121971
Accepted
Paul Hinett
Asked: 2010-03-13 10:03:58 +0800 CST

Windows Server 2003 - Handling hundreds of simultaneous downloads

  • 772

At the moment I have a single server with 4 1TB hard disks, daily I haver over 150 MP3 music files uploaded (around 80mb each).

At busy periods there is over 300 people streaming / downloading these mixes all at once, 75% of the activity is on the most recently uploaded stuff which is all on a single hard disk.

My read speads on the hard disk are very low due to such high activity of 200+ reads all happening at the same time on a single hard disk (ran some tests with HDTach).

What would be a logical solution to solve this, a couple of ideas I had are:

  • Load balance with another server
  • Install faster hard disks (what are best these days? SCSI / SATA)
  • Spread the most accessed files over the 4 drives so it is sharing the load between all 4 disks, instead of all the most accessed (most recent) all on the most recently installed drive.

Obviouslly load balance is the most expensive option, but would it dramatically help?

Some help on this situation would be great!

hard-drive load-balancing

4 Answers

  1. Your best bet will be to get some sort of RAID setup so that the load is spread across more than one hard drive. Also this will allow you to survive a failure of on of the disks in the RAID array without losing any data.

    Based on what you've said a 4 disk RAID 5 array should work just fine for you as most of the data is reads.

    • 1

  2. So what you want is a load balancer to ensure that both machines can serve requests, if needs be, and RAID to improve reliability and balance file read/writes over their disks. Oh and a rsync type tool to ensure the files are synchronised.

    As for whether the various parts of this are "worth it", instead of guessing you need to do performance monitoring to see where the bottlenecks are. If disk storage is definately the issue then RAID is worth it. If serving network requests is the issue then server load balancing is your friend.

    • 0

  3. How much memory do you have in the server? Perhaps using a ramdisk is an option if the system is devoted to serving files. Otherwise monitor the file cache counters in perfmon such as Cache - Cache Read Hit%, and the disk counters (in particular avg disk queue length, this should generally be around 3 or less.

    Here's a document that has a few optimizations that may help.

    • 0
  4. Best Answer

    You can scale in a few different direction here depending on your budget and hardware availability. Consider these steps, probably in this order but whatever suits your situation:

    • Re-jig the disks into a striped RAID set to increase raw disk performance. This'll mean backing up the files from the disks, creating the array, then moving the data back onto the stripe. If you can afford to lose some capacity for the sake of greater fault tolerance, go for a striped & mirrored array which will survive a disk failure

    • Add a dedicated RAID disk set for the server's page/temp file storage. Make this a fast RAID0 setup with a focus on performance. Ensure your app throws the files into memory when serving them, and that the page file is large enough to accomodate the files. This is a fairly cheap, straightforwards improvement.

    • Add more RAM to the server (it's relatively cheap stuff nowadays) with a focus on the bulk total RAM, rather than pure performance. The server can probably handle several different RAM speeds - and the slower stuff may be cheaper per Gb - You don't really need uber high RAM performance, rather you need bulk.

    • Re-tier the application so that the files are served from a dedicated host, and the rest of the web/database traffic and processing is handled by a seperate machine. This'll leave one server dedicated to file serving and you'll gain a performance benefit from this.

    • Tier your storage and add a fast stripe of disks to serve the newest MP3s from. Move files over to the slower storage once demand drops off. Feasibility of this approach depends on the effort involved in moving the files regularly, and how much of it you can automate in your app.

    • Add an array of SSD drives to serve the newest files.

    Ultimately you can target a tiered storage solution that stacks up like this, in order fastest to slowest: - RAM - Pagefile - SSD - Fast SATA/SAS stripe - Bulk Disk array

    • 0