Home / server / Questions / 535049
Accepted
jl6
Asked: 2013-08-30 22:49:24 +0800 CST

How do services with large ingest rates install enough hard disks fast enough?

  • 772

Amazon, Facebook and Google must take in TBs if not PBs per day, so does that mean they have people continuously installing new hard disks and cables in new racks in the same way you might build a regular server, or is there a different technology used for connecting drives en masse?

hard-drive

5 Answers

  1. Best Answer

    I don't know if any of them are actually installing hardware one server at a time. Back in 2008 MS started building its data centers by getting servers delivered in sealed and pre-wired shipping containers of servers that they just needed to unload from a truck and plug power/network connections into. While the 08 build was a mix of containers and traditional for their most recent datacenter they've since gone to a custom prefab design that's weatherproof and doesn't need to be housed inside separate buildings.

    Both HP and IBM sell similar packages with prebuilt containers full of servers that just need power/data connections to deploy.

    Microsoft's Boyden Virginia Data Center with prefab modules exposed to the weather.

    • 16

  2. Google has several technologies which they developed internally to store these huge masses of data. Using these technologies they can actually add truck loads of hard disks into their cluster without any downtime, but yes, they do still need people doing that.

    As far as I know from the Google Blog the two main parts are the Google File System, which is a distributed file system that can scale up to really big scale: Google File System

    And on top of the Google File System they have Big Table which is some kind of Key Value database and also scales up into huge scales: Big Table

    For guaranteeing high availability everything is redundant many times, more than 3 times in most cases.

    • 7

  3. That's precisely correct. I remember that at one time, Facebook datacenters were adding three tractor-trailers full of hard drives and rack-mount servers in the average day. Of course, they have complicated schemes to make storage scalable and redundant. Google, for example, has GFS. Facebook has three data centers just for their equipment, each larger than two Wal-Marts and a new one planned four times larger than their existing centers.

    • 5

  4. With new-generation Open Compute storage solutions able to fit 180 TB of disk in 4 rack units, a PB is not really a huge amount of space: adding 1PB a day would mean racking 5 such servers a day, not a big problem. Better yet, these can be had for about $10K, which means you are paying less than $60 per TB.

    So yeah, the technology is available to all of us to do the same, at a price that doesn't break the bank.

    But you have to keep in mind that larger companies always get a better deal, and sign large contracts to procure vast amounts of storage. They may get small shipments spread out over the year, but it's not like they order just 1PB at a time.

    • 2

  5. Here is an interesting blog post from BackBlaze on how they do this (they proably don't need as much disks as Google or Facebook, but still a whole lt), and what they needed to do when there were floods in Thailand that made harddisk expensive and harder to get by:

    http://blog.backblaze.com/2012/10/09/backblaze_drive_farming/

    • 0