Home / server / Questions / 287034
In Process
DimitrisD
Asked: 2011-07-05 23:37:55 +0800 CST

Debian squeeze, mdadm. Degraded RAID5 array after each reboot

  • 772

/etc/mdadm/mdadm.conf

 ARRAY /dev/md0 level=raid5 num-devices=4 metadata=00.90 UUID=54e55d36:c3971e21:d484c166:6e86c3d7
   devices=/dev/sdb,/dev/sdc,/dev/sdd,/dev/sde
MAILADDR root

mdadm -D /dev/md0

/dev/md0:
        Version : 0.90
  Creation Time : Thu Apr 30 23:48:58 2009
     Raid Level : raid5
     Array Size : 4395415488 (4191.79 GiB 4500.91 GB)
  Used Dev Size : 1465138496 (1397.26 GiB 1500.30 GB)
   Raid Devices : 4
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue Jul  5 10:21:41 2011
          State : clean, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 54e55d36:c3971e21:d484c166:6e86c3d7
         Events : 0.28886

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       2       8       48        2      active sync   /dev/sdd
       3       0        0        3      removed

after each reboot array is degraded and /dev/sde is removed. I re-add /dev/sde using

mdadm /dev/md0 -a /dev/sde

array getting rebuilt everything seems OK.

Until the next reboot...

EDIT

dmesg

1.275758] ata3.00: ATA-8: ST31500341AS, CC1H, max UDMA/133
[    1.275764] ata3.00: 2930277168 sectors, multi 16: LBA48 NCQ (depth 0/32)
[    1.275793] ata4.00: ATA-8: ST31500341AS, CC1H, max UDMA/133
[    1.275799] ata4.00: 2930277168 sectors, multi 16: LBA48 NCQ (depth 0/32)
[    1.296314] ata1.00: ATAPI: HL-DT-STDVD-ROM GDR8164B, 0L06, max UDMA/33
[    1.296714] ata1.01: HPA detected: current 156299375, native 156301488
[    1.296720] ata1.01: ATA-6: ST380011A, 3.06, max UDMA/100
[    1.296725] ata1.01: 156299375 sectors, multi 16: LBA48 
[    1.303263] ata3.01: ATA-8: ST31500341AS, CC1H, max UDMA/133
[    1.303267] ata3.01: 2930277168 sectors, multi 16: LBA48 NCQ (depth 0/32)
[    1.303375] ata4.01: ATA-8: ST31500341AS, CC1H, max UDMA/133
[    1.303378] ata4.01: 2930277168 sectors, multi 16: LBA48 NCQ (depth 0/32)
[    1.308845] ata1.00: configured for UDMA/33
[    1.320424] ata1.01: configured for UDMA/100
[    1.324633] scsi 0:0:0:0: CD-ROM            HL-DT-ST DVD-ROM GDR8164B 0L06 PQ: 0 ANSI: 5
[    1.324914] scsi 0:0:1:0: Direct-Access     ATA      ST380011A        3.06 PQ: 0 ANSI: 5
[    1.359697] ata4.00: configured for UDMA/133
[    1.367662] ata3.00: configured for UDMA/133
[    1.415713] ata4.01: configured for UDMA/133
[    1.424171] ata3.01: configured for UDMA/133
[    1.424284] scsi 2:0:0:0: Direct-Access     ATA      ST31500341AS     CC1H PQ: 0 ANSI: 5
[    1.424515] scsi 2:0:1:0: Direct-Access     ATA      ST31500341AS     CC1H PQ: 0 ANSI: 5
[    1.424753] scsi 3:0:0:0: Direct-Access     ATA      ST31500341AS     CC1H PQ: 0 ANSI: 5
[    1.424971] scsi 3:0:1:0: Direct-Access     ATA      ST31500341AS     CC1H PQ: 0 ANSI: 5
[    1.447254] sr0: scsi3-mmc drive: 52x/52x cd/rw xa/form2 cdda tray
[    1.447260] Uniform CD-ROM driver Revision: 3.20
[    1.447418] sr 0:0:0:0: Attached scsi CD-ROM sr0
[    1.448228] sd 0:0:1:0: [sda] 156299375 512-byte logical blocks: (80.0 GB/74.5 GiB)
[    1.448622] sd 0:0:1:0: [sda] Write Protect is off
[    1.448626] sd 0:0:1:0: [sda] Mode Sense: 00 3a 00 00
[    1.448656] sd 0:0:1:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.448848]  sda:
[    1.449175] sd 2:0:0:0: [sdb] 2930277168 512-byte logical blocks: (1.50 TB/1.36 TiB)
[    1.449246] sd 2:0:0:0: [sdb] Write Protect is off
[    1.449250] sd 2:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[    1.449281] sd 2:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.449478]  sdb:
[    1.449759] sd 3:0:0:0: [sdd] 2930277168 512-byte logical blocks: (1.50 TB/1.36 TiB)
[    1.449818] sd 3:0:0:0: [sdd] Write Protect is off
[    1.449821] sd 3:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[    1.449847] sd 3:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.450030]  sdd:
[    1.453309] sr 0:0:0:0: Attached scsi generic sg0 type 5
[    1.453369] sd 0:0:1:0: Attached scsi generic sg1 type 0
[    1.453420] sd 2:0:0:0: Attached scsi generic sg2 type 0
[    1.453476] sd 2:0:1:0: Attached scsi generic sg3 type 0
[    1.453532] sd 3:0:0:0: Attached scsi generic sg4 type 0
[    1.454032] sd 3:0:1:0: Attached scsi generic sg5 type 0
[    1.465468]  sda1 sda2 <
[    1.478018] sd 3:0:1:0: [sde] 2930277168 512-byte logical blocks: (1.50 TB/1.36 TiB)
[    1.478399] sd 2:0:1:0: [sdc] 2930277168 512-byte logical blocks: (1.50 TB/1.36 TiB)
[    1.478609]  unknown partition table
[    1.478667] sd 2:0:1:0: [sdc] Write Protect is off
[    1.478670] sd 2:0:1:0: [sdc] Mode Sense: 00 3a 00 00
[    1.478700]  unknown partition table
[    1.478819] sd 2:0:1:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.479134] sd 3:0:0:0: [sdd] Attached SCSI disk
[    1.479149]  sdc:
[    1.479284] sd 3:0:1:0: [sde] Write Protect is off
[    1.479287] sd 3:0:1:0: [sde] Mode Sense: 00 3a 00 00
[    1.479317] sd 3:0:1:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.479498]  sde: sda5 >
[    1.495522] sd 0:0:1:0: [sda] Attached SCSI disk
[    1.500601] sd 2:0:0:0: [sdb] Attached SCSI disk
[    1.500604]  unknown partition table
[    1.500852] sd 2:0:1:0: [sdc] Attached SCSI disk
[    1.508325]  unknown partition table
[    1.509078] sd 3:0:1:0: [sde] Attached SCSI disk
[    1.877343] async_tx: api initialized (async)
[    1.878206] xor: automatically using best checksumming function: pIII_sse
[    1.896005]    pIII_sse  :  6515.000 MB/sec
[    1.896009] xor: using function: pIII_sse (6515.000 MB/sec)
[    1.964021] raid6: int32x1    735 MB/s
[    2.032073] raid6: int32x2    753 MB/s
[    2.100050] raid6: int32x4    550 MB/s
[    2.168011] raid6: int32x8    490 MB/s
[    2.236012] raid6: mmxx1     2367 MB/s
[    2.304019] raid6: mmxx2     2598 MB/s
[    2.372010] raid6: sse1x1    1589 MB/s
[    2.440024] raid6: sse1x2    1978 MB/s
[    2.508009] raid6: sse2x1    2976 MB/s
[    2.576004] raid6: sse2x2    3575 MB/s
[    2.576006] raid6: using algorithm sse2x2 (3575 MB/s)
[    2.587036] md: raid6 personality registered for level 6
[    2.587040] md: raid5 personality registered for level 5
[    2.587042] md: raid4 personality registered for level 4
[    2.619394] md: md0 stopped.
[    2.622452] md: bind<sdc>
[    2.624170] md: bind<sdd>
[    2.625842] md: bind<sdb>
[    2.632060] raid5: device sdb operational as raid disk 0
[    2.632065] raid5: device sdd operational as raid disk 2
[    2.632068] raid5: device sdc operational as raid disk 1
[    2.632566] raid5: allocated 4222kB for md0
[    2.632751] 0: w=1 pa=0 pr=4 m=1 a=2 r=4 op1=0 op2=0
[    2.632755] 2: w=2 pa=0 pr=4 m=1 a=2 r=4 op1=0 op2=0
[    2.632758] 1: w=3 pa=0 pr=4 m=1 a=2 r=4 op1=0 op2=0
[    2.632761] raid5: raid level 5 set md0 active with 3 out of 4 devices, algorithm 2
[    2.632806] RAID5 conf printout:
[    2.632808]  --- rd:4 wd:3
[    2.632811]  disk 0, o:1, dev:sdb
[    2.632813]  disk 1, o:1, dev:sdc
[    2.632815]  disk 2, o:1, dev:sdd
[    2.632852] md0: detected capacity change from 0 to 4500905459712
[    2.635265]  md0: unknown partition table
[    2.692382] PM: Starting manual resume from disk
[    2.692387] PM: Resume from partition 8:5
[    2.692389] PM: Checking hibernation image.
[    2.700369] PM: Error -22 checking image file
[    2.700373] PM: Resume from disk failed.
[    2.752886] kjournald starting.  Commit interval 5 seconds
[    2.752902] EXT3-fs: mounted filesystem with ordered data mode.
[    4.475170] udev[370]: starting version 164
[    5.095175] input: PC Speaker as /devices/platform/pcspkr/input/input1
[    5.096407] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input2
[    5.096415] ACPI: Power Button [PWRB]
[    5.097714] ACPI: SSDT 1f5e74c0 0026C (v01  PmRef  Cpu0Ist 00003000 INTL 20040311)
[    5.100680] processor LNXCPU:00: registered as cooling_device0
[    5.101730] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input3
[    5.101742] ACPI: Power Button [PWRF]
[    5.105041] ACPI: SSDT 1f5e7980 00152 (v01  PmRef  Cpu1Ist 00003000 INTL 20040311)
[    5.107531] processor LNXCPU:01: registered as cooling_device1
[    5.264144] parport_pc 00:08: reported by Plug and Play ACPI
[    5.264369] parport0: PC-style at 0x378, irq 7 [PCSPP,TRISTATE]
[    6.020497] i801_smbus 0000:00:1f.3: PCI INT B -> GSI 19 (level, low) -> IRQ 19
[    6.021314] intel_rng: FWH not detected
[    6.113334] [drm] Initialized drm 1.1.0 20060810
[    6.228124] pci 0000:00:02.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[    6.228132] pci 0000:00:02.0: setting latency timer to 64
[    6.232413] pci 0000:00:02.0: irq 27 for MSI/MSI-X
[    6.232469] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
[    6.241459] Error: Driver 'pcspkr' is already registered, aborting...
[    6.255289] HDA Intel 0000:00:1b.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[    6.255345] HDA Intel 0000:00:1b.0: setting latency timer to 64
[    6.338649] input: HDA Digital PCBeep as /devices/pci0000:00/0000:00:1b.0/input/input4
[    6.989723] Adding 2650684k swap on /dev/sda5.  Priority:-1 extents:1 across:2650684k 
[    7.259580] EXT3 FS on sda1, internal journal
[    7.500343] loop: module loaded
[    7.537631] it87: Found IT8718F chip at 0x290, revision 5
[    7.537642] it87: in3 is VCC (+5V)
[    7.558886] coretemp coretemp.0: Using relative temperature scale!
[    7.558938] coretemp coretemp.1: Using relative temperature scale!
[    8.122878] kjournald starting.  Commit interval 5 seconds
[    8.128716] EXT3 FS on md0, internal journal
[    8.128724] EXT3-fs: mounted filesystem with ordered data mode.
[    8.898492] r8169: eth0: link up
[    8.898497] r8169: eth0: link up
[   16.029883] ip_tables: (C) 2000-2006 Netfilter Core Team
[   16.116193] nf_conntrack version 0.5.0 (7879 buckets, 31516 max)
[   16.116698] CONFIG_NF_CT_ACCT is deprecated and will be removed soon. Please use
[   16.116702] nf_conntrack.acct=1 kernel parameter, acct=1 nf_conntrack module option or
[   16.116704] sysctl net.netfilter.nf_conntrack_acct=1 to enable it.
[   19.080019] eth0: no IPv6 routers present
[  169.429012] md: bind<sde>
[  169.483527] RAID5 conf printout:
[  169.483532]  --- rd:4 wd:3
[  169.483537]  disk 0, o:1, dev:sdb
[  169.483540]  disk 1, o:1, dev:sdc
[  169.483544]  disk 2, o:1, dev:sdd
[  169.483547]  disk 3, o:1, dev:sde
[  169.483622] md: recovery of RAID array md0
[  169.483625] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[  169.483634] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[  169.483639] md: using 128k window, over a total of 1465138496 blocks.

EDIT-2

mdadm --assemble -v --force /dev/md0 /dev/sdb /dev/sdc /dev/sdd /dev/sde

mdadm: looking for devices for /dev/md0
mdadm: /dev/sdb is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 4.
mdadm: added /dev/sdc to /dev/md0 as 1
mdadm: added /dev/sdd to /dev/md0 as 2
mdadm: no uptodate device for slot 3 of /dev/md0
mdadm: added /dev/sde to /dev/md0 as 4
mdadm: added /dev/sdb to /dev/md0 as 0
mdadm: /dev/md0 has been started with 3 drives (out of 4) and 1 spare.
debian mdadm raid5

2 Answers

  1. I had a similar problem with a RAID-1. In my case, because of device renaming, the solution was to remove the /dev devices from the ARRAY specification and only use the UUID.

    • 2

  2. Although there is nothing specific in the log, and providing that the array finished rebuilding before the reboot, then I suspect that the disk /dev/sde has something up with it, a good guide to look into this using SMART (if supported nby the disk) can be found at cyberciti.

    If you happen to have another drive floating around that you can drop that in to test the configuration.

    • 0