I had some problems with my RAID 6 drive on mdadm recently. No LVM in between. Two drives died, and were pulled out while the machine was powered down. Now, I boot up the system, and cannot mount the raid.
mdstat says that the raid is active
root@NFS:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : active raid6 sdc[5] sdb[3] sde[1] sdd[0]
7813531648 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/4] [UU_U_U]
But when I try to mount it...
root@NFS:~# sudo mount -t ext4 /dev/md127 /media/NAS
mount: wrong fs type, bad option, bad superblock on /dev/md127,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
fsck is the same
root@NFS:~# fsck.ext4 /dev/md127
e2fsck 1.42 (29-Nov-2011)
fsck.ext4: Superblock invalid, trying backup blocks...
fsck.ext4: Bad magic number in super-block while trying to open /dev/md127
The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
So, I listed backup superblocks
root@NFS:~$ fdisk -l
Disk /dev/sda: 320.1 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders, total 625142448 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000589e8
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 608382975 304190464 83 Linux
/dev/sda2 608385022 625141759 8378369 5 Extended
/dev/sda5 608385024 625141759 8378368 82 Linux swap / Solaris
Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sdb doesn't contain a valid partition table
Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sdc doesn't contain a valid partition table
Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sdd doesn't contain a valid partition table
Disk /dev/sde: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sde doesn't contain a valid partition table
Disk /dev/md127: 8001.1 GB, 8001056407552 bytes
2 heads, 4 sectors/track, 1953382912 cylinders, total 15627063296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 524288 bytes / 2097152 bytes
Disk identifier: 0x6f6d8d7f
Disk /dev/md127 doesn't contain a valid partition table
jippen@NFS:~$ sudo mke2fs -n /dev/md127
mke2fs 1.42 (29-Nov-2011)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=128 blocks, Stripe width=512 blocks
244174848 inodes, 1953382912 blocks
97669145 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
59613 block groups
32768 blocks per group, 32768 fragments per group
4096 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544, 1934917632
And try a few...
root@NFS:~# e2fsck -b 7962624 /dev/md127
e2fsck 1.42 (29-Nov-2011)
e2fsck: Bad magic number in super-block while trying to open /dev/md127
The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
Is there still hope, or is this total data loss?
EDIT 1/12/2014
Output of fdisk -l
root@NFS:~# fdisk -l /dev/md127
Disk /dev/md127: 8001.1 GB, 8001056407552 bytes
2 heads, 4 sectors/track, 1953382912 cylinders, total 15627063296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 524288 bytes / 2097152 bytes
Disk identifier: 0x6f6d8d7f
Disk /dev/md127 doesn't contain a valid partition table
EDIT adding hexdump
root@NFS:~# hexdump -C -n 512 /dev/md127
00000000 45 eb 13 54 f6 24 03 0a 17 ad 30 a9 50 4f bd 48 |E..T.$....0.PO.H|
00000010 8c 7f 20 7f f6 5a 63 0e be bd 7f 4a c2 db ff 00 |.. ..Zc....J....|
00000020 5d c7 dc 8c f2 3f 11 fd 6a 32 00 27 03 3d 41 a7 |]....?..j2.'.=A.|
00000030 b0 c1 cf bf 3f 98 a6 b0 ce 73 df ff 00 ad 5a a5 |....?....s....Z.|
00000040 a0 92 b5 bd 0a e7 04 91 f5 c7 eb 50 ba 16 07 d0 |...........P....|
00000050 f1 fc 85 4c dc 92 07 4e 29 84 85 00 0e c7 9a b5 |...L...N).......|
00000060 b2 b7 64 53 db ee 29 95 c0 20 72 57 ad 64 5f c5 |..dS..).. rW.d_.|
00000070 87 0d ea 09 35 b7 26 77 67 eb fd 2a 9d cc 21 a3 |....5.&wg..*..!.|
00000080 3f 46 cd 6f 4e 5e f1 cd 3d 62 61 51 4e 71 b5 98 |?F.oN^..=baQNq..|
00000090 7a 1a 6d 76 2d 8e 20 a2 8a 28 00 a2 8a 28 00 a2 |z.mv-. ..(...(..|
000000a0 8a 28 00 a2 8a 28 00 a2 8a 28 02 48 f8 39 a7 cd |.(...(...(.H.9..|
000000b0 39 70 06 6a 0a 7c 6b b9 87 a6 69 03 2d 5a db 82 |9p.j.|k...i.-Z..|
000000c0 0b 9e c3 8a ba 67 54 8c 67 8c 1c 0f 5a a4 f7 3e |.....gT.g...Z..>|
000000d0 50 0a 3a 81 8a a8 ee 5c 92 7b d4 38 dd fc c6 4f |P.:....\.{.8...O|
000000e0 71 74 66 24 0e 9c ff 00 3a ad 4a 14 92 2a ec 36 |qtf$....:.J..*.6|
000000f0 d8 5d d8 eb ff 00 d6 aa b5 a2 fd 49 5d 08 22 b7 |.].........I].".|
00000100 32 30 07 8a b4 2d 82 67 d4 54 bb 42 64 83 ce 09 |20...-.g.T.Bd...|
00000110 fe 55 5a 4b 9d a7 03 f1 fa e0 d2 bf bd f2 2d f4 |.UZK..........-.|
00000120 3d 0c 60 b7 d7 a7 e9 4a 57 04 f7 a4 c6 40 f6 c7 |=.`....JW....@..|
00000130 f4 a7 85 d8 07 35 e4 4b 49 47 cd 33 d1 19 8c 90 |.....5.KIG.3....|
00000140 69 a5 70 4e 2a 43 fa d3 86 39 07 be 7f ad 5a 7a |i.pN*C...9....Zz|
00000150 3f eb b1 16 d1 15 66 b5 8a f2 36 46 fb de bf 4a |?.....f...6F...J|
00000160 e5 af f4 b3 04 8d 8e 47 51 f8 e6 ba d2 9f 78 0f |.......GQ.....x.|
00000170 4a ce b9 8f e6 04 8f af d3 9a d2 9b b5 fd 56 84 |J.............V.|
00000180 b8 9c 84 d1 15 04 1f f3 d6 a9 95 c1 c5 76 97 9a |.............v..|
00000190 72 5d c1 b9 38 65 52 7f 9d 72 13 c6 51 d8 1e a0 |r]..8eR..r..Q...|
000001a0 9f d2 bb e9 ce e9 fa 9c f5 23 aa 22 03 6d 38 70 |.........#.".m8p|
000001b0 0e 69 a7 b6 69 73 91 f8 7f 8d 6d 6f 79 7a 18 8b |.i..is....moyz..|
000001c0 d2 99 9e b4 e3 ce 73 4d 3d 6a 7a 07 51 41 e2 9d |......sM=jz.QA..|
000001d0 91 9e 7d 29 99 e2 94 f4 14 2d 9f a8 85 03 3f cb |..}).....-....?.|
000001e0 f9 52 13 82 3f 0f e9 4a 0f 14 df f0 a4 f7 19 20 |.R..?..J....... |
000001f0 e8 7e 94 11 c7 e2 0d 31 4e 0d 3f 39 07 9f f3 cd |.~.....1N.?9....|
00000200
EDIT More requests for commands
mdadm --detail
root@NFS:~# mdadm --detail /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Sat Jan 11 20:56:15 2014
Raid Level : raid6
Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
Used Dev Size : 1953382912 (1862.89 GiB 2000.26 GB)
Raid Devices : 6
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Sat Jan 11 22:01:47 2014
State : clean, degraded
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : NFS:127 (local to host NFS)
UUID : 2cde46cb:aa89730c:bb1f214a:cad5bc0a
Events : 2
Number Major Minor RaidDevice State
0 8 48 0 active sync /dev/sdd
1 8 64 1 active sync /dev/sde
2 0 0 2 removed
3 8 16 3 active sync /dev/sdb
4 0 0 4 removed
5 8 32 5 active sync /dev/sdc
And mdadm --examine
root@NFS:~# mdadm --examine /dev/sd{b,c,d,e,f}
/dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 2cde46cb:aa89730c:bb1f214a:cad5bc0a
Name : NFS:127 (local to host NFS)
Creation Time : Sat Jan 11 20:56:15 2014
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 565612d2:cad5edc9:0848ba81:e06f76a2
Update Time : Wed Jan 15 22:11:32 2014
Checksum : 47b3b275 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AA.A.A ('A' == active, '.' == missing)
/dev/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 2cde46cb:aa89730c:bb1f214a:cad5bc0a
Name : NFS:127 (local to host NFS)
Creation Time : Sat Jan 11 20:56:15 2014
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : a96bd195:c410fc2f:f0422486:6418d34c
Update Time : Wed Jan 15 22:11:32 2014
Checksum : 2047a62f - correct
Events : 4
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AA.A.A ('A' == active, '.' == missing)
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 2cde46cb:aa89730c:bb1f214a:cad5bc0a
Name : NFS:127 (local to host NFS)
Creation Time : Sat Jan 11 20:56:15 2014
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 27e6f0f2:96c41f41:e75d30cc:462cd9bf
Update Time : Wed Jan 15 22:11:32 2014
Checksum : 479d0354 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AA.A.A ('A' == active, '.' == missing)
/dev/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 2cde46cb:aa89730c:bb1f214a:cad5bc0a
Name : NFS:127 (local to host NFS)
Creation Time : Sat Jan 11 20:56:15 2014
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : fc780ffd:9daf5c50:9a1e4af7:ff5061c7
Update Time : Wed Jan 15 22:11:32 2014
Checksum : 939a669d - correct
Events : 4
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AA.A.A ('A' == active, '.' == missing)
Regarding the hexdump, you showed the first 512 bytes, but for an ext4 filesystem it's a too small value, if you see the ondisk format of ext4 in https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Layout you'll see that the first 1024 bytes are left alone as padding to allow the installation of a boot sector. The first contents start at 1024 with the contents of the (first) superblock.
In my case my ext4 filesystems contains 1024 zeros at the beginning, I don't know why do you have binary data there. My opinion is that it could be there from a previous use of the disk (perhaps some initial benchmark?).
If this dump you can see that the first 1024 (0x400) bytes are 0, and that there is a superblock with the signature of an ext filesystem at offset 0x38 of the superblock (0x438 of the full output). The magic value is defined as 0xEF53, but these are the bytes 0x53 0xEF the 16-bit value 0xEF53 in little-endian form. In my superblock you can see the label of the filesystem "portage" and that it was last mounted on "/usr/portage/distfiles" among other things. You should search for things like that in the superblock copies.
It seems that you don't have any valid copy of the superblock, so the filesystem is possibly thrashed, but why? why did the two disks failed? do you have something interesting in dmesg? in smartctl -a /dev/sd_? in the kernel log when the two previous disks failed? did the machine worked correctly before? did you shut it down correctly or it died by itself?
RAID6 should be able to survive the hardware failure of two disks, but depending on what failed you could have missed your data. At last resort you can try testdisk, photorec, sleuthkit, etc to try to recover files, but a backup restore should be a better option.
You should really investigate the reason your RAID6 array did not survive and learn from that, avoid getting in the same situation (whatever it was).
And if you can, please post it here so we can all learn.
After looking at the posted hexdump my idea is that the order sequence of the drives of the RAID6 has been scrambled somehow. I would try to investigate the first blocks of each of the other three drives and try to find the ext4 super block on one of these. Before giving up it might be even possible to guess every permutation of the drives and by chance you could get your data back.
Disclaimer: I personally stopped using RAID5 and RAID6 long ago in favor of RAID10 and so my personal knowledge about RAID modes 5 and 6 is a bit rusty. Why? read this