Prelude:
I'm a code-monkey that's increasingly taken on SysAdmin duties for my small company. My code is our product, and increasingly we provide the same app as SaaS.
About 18 months ago I moved our servers from a premium hosting centric vendor to a barebones rack pusher in a tier IV data center. (Literally across the street.) This ment doing much more ourselves--things like networking, storage and monitoring.
As part the big move, to replace our leased direct attached storage from the hosting company, I built a 9TB two-node NAS based on SuperMicro chassises, 3ware RAID cards, Ubuntu 10.04, two dozen SATA disks, DRBD and . It's all lovingly documented in three blog posts: Building up & testing a new 9TB SATA RAID10 NFSv4 NAS: Part I, Part II and Part III.
We also setup a Cacti monitoring system. Recently we've been adding more and more data points, like SMART values.
I could not have done all this without the awesome boffins at ServerFault. It's been a fun and educational experience. My boss is happy (we saved bucket loads of $$$), our customers are happy (storage costs are down), I'm happy (fun, fun, fun).
Until yesterday.
Outage & Recovery:
Some time after lunch we started getting reports of sluggish performance from our application, an on-demand streaming media CMS. About the same time our Cacti monitoring system sent a blizzard of emails. One of the more telling alerts was a graph of iostat await.
Performance became so degraded that Pingdom began sending "server down" notifications. The overall load was moderate, there was not traffic spike.
After logging onto the application servers, NFS clients of the NAS, I confirmed that just about everything was experiencing highly intermittent and insanely long IO wait times. And once I hopped onto the primary NAS node itself, the same delays were evident when trying to navigate the problem array's file system.
Time to fail over, that went well. Within 20 minuts everything was confirmed to be back up and running perfectly.
Post-Mortem:
After any and all system failures I perform a post-mortem to determine the cause of the failure. First thing I did was ssh back into the box and start reviewing logs. It was offline, completely. Time for a trip to the data center. Hardware reset, backup an and running.
In /var/syslog I found this scary looking entry:
Nov 15 06:49:44 umbilo smartd[2827]: Device: /dev/twa0 [3ware_disk_00], 6 Currently unreadable (pending) sectors
Nov 15 06:49:44 umbilo smartd[2827]: Device: /dev/twa0 [3ware_disk_07], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 171 to 170
Nov 15 06:49:45 umbilo smartd[2827]: Device: /dev/twa0 [3ware_disk_10], 16 Currently unreadable (pending) sectors
Nov 15 06:49:45 umbilo smartd[2827]: Device: /dev/twa0 [3ware_disk_10], 4 Offline uncorrectable sectors
Nov 15 06:49:45 umbilo smartd[2827]: Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
Nov 15 06:49:45 umbilo smartd[2827]: # 1 Short offline Completed: read failure 90% 6576 3421766910
Nov 15 06:49:45 umbilo smartd[2827]: # 2 Short offline Completed: read failure 90% 6087 3421766910
Nov 15 06:49:45 umbilo smartd[2827]: # 3 Short offline Completed: read failure 10% 5901 656821791
Nov 15 06:49:45 umbilo smartd[2827]: # 4 Short offline Completed: read failure 90% 5818 651637856
Nov 15 06:49:45 umbilo smartd[2827]:
So I went to check the Cacti graphs for the disks in the array. Here we see that, yes, disk 7 is slipping away just like syslog says it is. But we also see that disk 8's SMART Read Erros are fluctuating.
There are no messages about disk 8 in syslog. More interesting is that the fluctuating values for disk 8 directly correlate to the high IO wait times! My interpretation is that:
- Disk 8 is experiencing an odd hardware fault that results in intermittent long operation times.
- Somehow this fault condition on the disk is locking up the entire array
Maybe there is a more accurate or correct description, but the net result has been that the one disk is impacting the performance of the whole array.
The Question(s)
- How can a single disk in a hardware SATA RAID-10 array bring the entire array to a screeching halt?
- Am I being naïve to think that the RAID card should have dealt with this?
- How can I prevent a single misbehaving disk from impacting the entire array?
- Am I missing something?
I hate to say "don't use SATA" in critical production environments, but I've seen this situation quite often. SATA drives are not generally meant for the duty cycle you describe, although you did spec drives specifically rated for 24x7 operation in your setup. My experience has been that SATA drives can fail in unpredictable ways, often times affecting the entire storage array, even when using RAID 1+0, as you've done. Sometimes the drives fail in a manner that can stall the entire bus. One thing to note is whether you're using SAS expanders in your setup. That can make a difference in how the remaining disks are impacted by a drive failure.
But it may have made more sense to go with midline/nearline (7200 RPM) SAS drives versus SATA. There's a small price premium over SATA, but the drives will operate/fail more predictably. The error-correction and reporting in the SAS interface/protocol is more robust than the SATA set. So even with drives whose mechanics are the same, the SAS protocol difference may have prevented the pain you experienced during your drive failure.
How can a single disk bring down the array? The answer is that it shouldn't, but it kind of depends on what is causing the outage. If the disk were to die in a way that behaved, it shouldn't take it down. But it's possible that it's failing in an "edge case" way that the controller can't handle.
Are you naive to think this shouldn't happen? No, I don't think so. A hardware RAID card like that should have handled most issues.
How to prevent it? You can't anticipate weird edge cases like this. This is part of being a sysadmin...but you can work on recovery procedures to keep it from impacting your business. The only way to try to fix this right now is to either try another hardware card (not probably what you'd want to do) or change your drives to SAS drives instead of SATA to see if SAS is more robust. You can also contact your vendor of the RAID card and tell them what has happened and see what they say; they are, after all, a company that is supposed to specialize in knowing the ins and outs of wonky drive electronics. They may have more technical advice on how the drives work as well as reliability...if you can get to the right people to talk to.
Have you missed something? If you want to verify that the drive is having an edge-case failure, pull it from the array. The array will be degraded but you shouldn't have more of the weird slowdowns and errors (aside from the degraded array status). You're saying that right now it seems to be working fine, but if it's having disk read errors, you should replace the drive while you can. Drives with high capacity can sometimes have URE errors (best reason not to run RAID 5, side note) that don't show up until another drive has failed. And if you're experiencing edge-case behavior from that one drive, you don't want corrupted data migrated to the other drives in the array.
I'm not an expert, but I'm going to take a wild shot in the dark on the basis of my experience with RAID controllers and storage arrays.
Disks fail in many different ways. Unfortunately, disks can fail, or be faulty, in ways where their performance is seriously affected but the RAID controller doesn't see as being a failure.
If a disk fails in an obvious way, any RAID controller software should be pretty good at detecting lack of response from the disk, removing it from the pool and firing any notifications. However, my guess as to what's happening here is that the disk is suffering an unusual failure which, for some reason are not triggering a failure on the controller side. Therefore when the controller is conducting a write flush or a read from the affected disk, it's taking a long time to come back and in turn is hanging the whole IO operating and therefore the array. For whatever reason, this isn't enough for the RAID controller to go "ah, failed disk", probably because the data ends up coming back eventually.
My advice would be to immediately replace the failed disk. After that, I'd take a look at the configuration for your RAID card (It's 3ware, I thought they were pretty good) and find out what it considers a failed disk to be.
P.S. nice idea importing SMART into cacti.
Just a guess: the harddisks are configured to retry on read errors rather than report an error. While this is desirable behaviour in a desktop setting, it is counterproductive in a RAID (where the controller should rewrite any sector that fails reading from the other disks, so the drive can remap it).
my shot in the dark:
drive 7 is failing. it has some failure windows where it's not available.
drive 8 has some 'lighter' errors too; corrected by retrying.
RAID10 is usually "a RAID0 of several RAID1 pairs", are drive 7 and 8 members of the same pair?
if so, then it seems you hit the "shouldn't happen" case of two-disk failure on the same pair. almost the only thing that can kill a RAID10. unfortunately, it can happen if all your drives are from the same shipping lot, so they're slightly more likely to die simultaneously.
I guess that during a drive 7 failure, the controller redirected all reads to drive 8, so any error-retry caused big delays that caused an avalanche of frozen tasks, killing performance for a while.
you're lucky that drive 8 doesn't seem to be dead yet, so you should be able to fix without dataloss.
I'd start by changing both drives, and don't forget to check cabling. a loose connection could cause this, and if not routed firmly, it's more likely to happen in adjacent drives. also, some multiport cards have several two-port connectors, if drive 7 and drive 8 are on the same one, it might be the source of your trouble.
You need the features of enterprise class storage devices. Specifically, the WD RE 4 enterprise drives have two features needed to prevent this behavior in RAID Arrays. The first technology listed below prevents rotational harmonic vibration from causing needless wear on the hard drive mechanical components. The second technology is what caused your problem, SATA protocol does not have this feature. To get these features you need SAS, and if you insist on SATA drives you can purchase SAS to SATA Interposer cards such as the LSISS9252.
Enhanced RAFF technology Sophisticated electronics monitor the drive and correct both linear and rotational vibration in real time. The result is a significant performance improvement in high vibration environments over the previous generation of drives.
RAID-specific, time-limited error recovery (TLER) Prevents drive fallout caused by the extended hard drive error-recovery processes common to desktop drives.
http://en.wikipedia.org/wiki/Error_recovery_control#Overview
Also please see link below:
http://en.wikipedia.org/wiki/Error_recovery_control#Raid_Controllers
Also see: Western Digital TLER Document explaining the error recovery process in depth. Error Recovery Fallout Prevention in WD Caviar RAID Edition Serial ATA Hard Drives:
http://www.3dfxzone.it/public/files/2579-001098.pdf
SATA Interposer Cards are another solution.
I recently experienced exacly the same fate and found this thread. The overall tenor is that SAS protocol is better suited for RAID than SATA, because SATA is lacking features. This is why the same physical drives are equipped with SAS controllers, then sold as Nearline SAS.
Searching further, I found:
http://www.lsi.com/products/storagecomponents/Pages/LSISS9252.aspx
I'm investigating upgrading one of my storages with a batch of these. Right now, the price difference between 3 TB SATA vs SAS is 400% (vanilla price, same brand, specs and shop, Germany). I obviously can't tell if this strategy works out well, but it's worth a try.
Comments very welcome :-)
I have seen a SATA disk with broken electronics lock up the firmware init of an Areca 12something solidly, there was no way to access the BIOS let alone boot the machine from any medium until the offending hard drive was found by pulling disks out in a binary search fashion.