On two Supermicro 847E16-RJBOD1 enclosures, I have trouble accessing SATA disks on the rear backplane with 6gb/s. The only significant difference between rear and front backplanes is total cable length from HBA to backplane, which is about 1m + 30cm vs. 1m + 70cm.
SAS signalling shouldn't be affected at these lengths, and I always assumed HBA and backplane will connect with SAS signalling whatever disk type attached. Is this assumption wrong? Do SATA disks limit HBA to backplane cable lengths?
My understanding of SCSI timeouts is that any read, write, flush and other commands have a limited time to complete. If exceeded, the command is aborted and an error is reported to the upper layer. While waiting for the command to complete, any application depending on the I/O will stall.
My next layer would be mdraid, the Linux software RAID. From what I read, mdraid has no timeouts on its own but relies on the lower layer to timeout commands.
The default SCSI timeout value is 90 seconds for Kernel 3.2 (Debian).
A hard disk encountering a read error will try hard to correct the error within a time frame defined by firmware. That timeout is set high for desktop drives (typically stand-alone, so correction has high priority) and low for server drives (typically RAID, so report bad sector soon, let other drive answer). Sometimes it can be adjusted via smartctl (SCTERC, TLER, etc.).
So I guess if an HDD is set to a high ERC timeout, kernel will wait for 90 seconds by default before aborting the request. Only then will mdraid redirect the application's request to another disk.
90 seconds is a loooong time for a webpage to load.
Is it correct to assume the default SCSI timeout is meant for desktop purposes or non-hdd SCSI equipment (tape drive, tape library come to mind), and safe to tune down to, say, 7 seconds for RAID usage?
I have a 45-disk array of Seagate Barracuda 3 TB ST3000DM001 (yes these are desktop drives I'm aware of that) in a Supermicro sc847 JBOD, connected via LSI 9285. I have found a solution for the problem description below by reducing speed via
MegaCli -PhySetLinkSpeed -phy0 2 -a0;
for i in $(seq 48); do MegaCli -PhySetLinkSpeed -phy${i} 2 -a0; done
and rebooting.
The question remains: Is this typical for current 6 gb/s equipment? Is this the sad state of SATA storage? Or is some of my equipment (the sff-8088 cables come to mind) bad?
The Problem was:
Synchronizing HW RAID-6, disks kept offlining. Fetching SMART values reveiled that those which offlined did not increase powered-on hours anymore. That is, their firmware (CC4C) seems to crash.
Digging into the matter by switching to Software RAID-6, with the disks passed-through, I got tons of kernel messages scattered across all disks, with 6 gb/s:
sd 0:0:9:0: [sdb] Sense Key : No Sense [current]
Info fld=0x0
sd 0:0:9:0: [sdb] Add. Sense: No additional sense information
And finally, when a disk offlines:
megasas: [ 5]waiting for 160 commands to complete
...
megasas: [35]waiting for 159 commands to complete
...
megasas: [155]waiting for 156 commands to complete
...
megaraid_sas: pending commands remain after waiting, will reset adapter.
Ugly controller reset here, then minutes later:
megaraid_sas: Reset successful.
sd 0:0:28:0: Device offlined - not ready after error recovery
...
sd 0:0:28:0: [sdu] Unhandled error code
sd 0:0:28:0: [sdu] Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
sd 0:0:28:0: [sdu] CDB: Read(10): 28 00 23 21 2f 40 00 00 70 00
sd 0:0:28:0: [sdu] killing request
Reduced speed to 3 gb/s like written above, all problems vanished.
Driven by the current price difference between SATA and SAS disks on one side and the potentially bad behaviour of SATA disks in bigger storage arrays on the other side, I have found so-called SATA-to-SAS interposer cards.
Advertised as "seamlessly adding SAS capabilities to existing SATA disk drives", I wonder if anyone here has had some experience with these or similar products. The major benefits I can identify are the increased cable voltage (if all drives are SAS connected), the ability to power-cycle the drive and multipath (if desired). Obviously the SATA drive will still have to be RAID edition.
The question is: Do these cards indeed increase the overall reliability of a storage system, or will failing SATA disks cause trouble nevertheless?
Edit: I'm not asking for hypothetical answers, only actual experience please.
I'm well aware that the typical 10k SAS drive is more reliable (and better performing) than 7200 SATA drives. But how does a nearline SAS, which is phyiscally the same disk as its SATA counterpart, compare to the SATA version with interposer?
I'd like to use "ipmitool ... sol activate" to record any kernel panics that may happen. I managed to keep GRUB2 in text-mode with "GRUB_TERMINAL=console" in /etc/default/grub. But right after the message "Loading initial ramdisk" (which is still GRUB), when Linux starts printing text on VGA console, messages on the serial console stop.
I guess either GRUB or Linux is doing something at that point, like loading a font, which kills off the vga-text-to-serial line. I'd like to keep it from doing that, whatever it is.
I have tried so far:
One obvious alternative is to setup serial console redirection to the virtual COM port, which should make panics appear on the serial-over-lan as well, but I'd prefer to stay in the initial text-mode, so VGA and console are in sync and both accept input.
After some search I assume lvm2 restriping, as in take all extents of an active LV and redistribute them evenly across all current PVs in the VG, is not possible?
For example:
(having a mix of SSD and HDD in the same VG is basically a fun thing to do, because pvmove can move LVs online from SSD to HDD and back, but unfortunately it can't stripe while moving)
The offline case is straight forward: create a new striped LV across the HDDs, dd all content from old to new, rename, online again. But is there a way to do this online?
Closest thing I could come up with is write a bash script running pvmove to scatter the extents, but that wouldn't be the same (since extent size is a whopping 4 MiB opposed to typical stripe size of 64 KiB).
Background flushing on Linux happens when either too much written data is pending (adjustable via /proc/sys/vm/dirty_background_ratio) or a timeout for pending writes is reached (/proc/sys/vm/dirty_expire_centisecs). Unless another limit is being hit (/proc/sys/vm/dirty_ratio), more written data may be cached. Further writes will block.
In theory, this should create a background process writing out dirty pages without disturbing other processes. In practice, it does disturb any process doing uncached reading or synchronous writing. Badly. This is because the background flush actually writes at 100% device speed and any other device requests at this time will be delayed (because all queues and write-caches on the road are filled).
Is there a way to limit the amount of requests per second the flushing process performs, or otherwise effectively prioritize other device I/O?