Issue

I have read many discussions about storage, and whether SSDs or classic HDDs are better. I am quite confused. HDDs are still quite preferred, but why?

Which is better for active storage? For example for databases, where the disk is active all the time?

About SSD.

Pros.

• They are quiet.
• Not mechanical.
• Fastest.

Cons.

• More expensive.

Question.

• When the life cycle for one cell of a SSD is used, what happens then? Is the disk reduced by only this cell and works normally?
• What is the best filesystem to write? Is ext4 good because it saves to cells consecutively?

About HDD.

Pros.

• Cheaper.

Cons.

• In case of mechanical fault, I believe there is usually no way to repair it. (Please confirm.)
• Slowest, although I think HDD speed is usually sufficient for servers.

Is it just about price? Why are HDDs preferred? And are SSDs really useful for servers?

All of my servers are currently flooded by salt water. Is it possible for each platter in a multi-platter drive to be separated, cleaned, imaged, and merged into a new virtual drive for data recovery?

This is a Canonical Question about RAID levels.

What are:

• the RAID levels typically used (including the RAID-Z family)?
• deployments are they commonly found in?
• benefits and pitfalls of each?

Prelude:

I'm a code-monkey that's increasingly taken on SysAdmin duties for my small company. My code is our product, and increasingly we provide the same app as SaaS.

About 18 months ago I moved our servers from a premium hosting centric vendor to a barebones rack pusher in a tier IV data center. (Literally across the street.) This ment doing much more ourselves--things like networking, storage and monitoring.

As part the big move, to replace our leased direct attached storage from the hosting company, I built a 9TB two-node NAS based on SuperMicro chassises, 3ware RAID cards, Ubuntu 10.04, two dozen SATA disks, DRBD and . It's all lovingly documented in three blog posts: Building up & testing a new 9TB SATA RAID10 NFSv4 NAS: Part I, Part II and Part III.

We also setup a Cacti monitoring system. Recently we've been adding more and more data points, like SMART values.

I could not have done all this without the awesome boffins at ServerFault. It's been a fun and educational experience. My boss is happy (we saved bucket loads of $$$), our customers are happy (storage costs are down), I'm happy (fun, fun, fun).

Until yesterday.

Outage & Recovery:

Some time after lunch we started getting reports of sluggish performance from our application, an on-demand streaming media CMS. About the same time our Cacti monitoring system sent a blizzard of emails. One of the more telling alerts was a graph of iostat await.

Performance became so degraded that Pingdom began sending "server down" notifications. The overall load was moderate, there was not traffic spike.

After logging onto the application servers, NFS clients of the NAS, I confirmed that just about everything was experiencing highly intermittent and insanely long IO wait times. And once I hopped onto the primary NAS node itself, the same delays were evident when trying to navigate the problem array's file system.

Time to fail over, that went well. Within 20 minuts everything was confirmed to be back up and running perfectly.

Post-Mortem:

After any and all system failures I perform a post-mortem to determine the cause of the failure. First thing I did was ssh back into the box and start reviewing logs. It was offline, completely. Time for a trip to the data center. Hardware reset, backup an and running.

In /var/syslog I found this scary looking entry:

Nov 15 06:49:44 umbilo smartd[2827]: Device: /dev/twa0 [3ware_disk_00], 6 Currently unreadable (pending) sectors
Nov 15 06:49:44 umbilo smartd[2827]: Device: /dev/twa0 [3ware_disk_07], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 171 to 170
Nov 15 06:49:45 umbilo smartd[2827]: Device: /dev/twa0 [3ware_disk_10], 16 Currently unreadable (pending) sectors
Nov 15 06:49:45 umbilo smartd[2827]: Device: /dev/twa0 [3ware_disk_10], 4 Offline uncorrectable sectors
Nov 15 06:49:45 umbilo smartd[2827]: Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
Nov 15 06:49:45 umbilo smartd[2827]: # 1  Short offline       Completed: read failure       90%      6576         3421766910
Nov 15 06:49:45 umbilo smartd[2827]: # 2  Short offline       Completed: read failure       90%      6087         3421766910
Nov 15 06:49:45 umbilo smartd[2827]: # 3  Short offline       Completed: read failure       10%      5901         656821791
Nov 15 06:49:45 umbilo smartd[2827]: # 4  Short offline       Completed: read failure       90%      5818         651637856
Nov 15 06:49:45 umbilo smartd[2827]:

So I went to check the Cacti graphs for the disks in the array. Here we see that, yes, disk 7 is slipping away just like syslog says it is. But we also see that disk 8's SMART Read Erros are fluctuating.

There are no messages about disk 8 in syslog. More interesting is that the fluctuating values for disk 8 directly correlate to the high IO wait times! My interpretation is that:

• Disk 8 is experiencing an odd hardware fault that results in intermittent long operation times.
• Somehow this fault condition on the disk is locking up the entire array

Maybe there is a more accurate or correct description, but the net result has been that the one disk is impacting the performance of the whole array.

The Question(s)

• How can a single disk in a hardware SATA RAID-10 array bring the entire array to a screeching halt?
• Am I being naïve to think that the RAID card should have dealt with this?
• How can I prevent a single misbehaving disk from impacting the entire array?
• Am I missing something?

Is there a good command line utility to monitor hard disk load on linux? Something like top but then monitoring disk activity i.s.o. cpu usage.

More specifically, I suspect that for some (heavy load) servers after several optimizations on various parts of the program(s) that run on it, right now the bottleneck is simply the logging to files on the disk. But I find it very difficult to assess how much traffic the servers can handle.

My ideal tool would be something that prints "You're using 35% of your disk bandwidth right now". Any ideas?