Brad's questions

I have an MDADM array with four 1TB disks in a RAID5 configuration. Here's the relevant part of mdadm --detail /dev/md1:

        Version : 00.90
     Raid Level : raid5
     Array Size : 2929537920 (2793.83 GiB 2999.85 GB)
  Used Dev Size : 976512640 (931.28 GiB 999.95 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 1
    Persistence : Superblock is persistent

It shows that the array is 2.8TB. Here is fdisk -l /dev/md1:

Disk /dev/md1: 2999.8 GB, 2999846830080 bytes
2 heads, 4 sectors/track, 732384480 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 196608 bytes
Disk identifier: 0xffffffff
Disk /dev/md1 doesn't contain a valid partition table

fdisk also shows it to be 3TB. However, df -Th does not agree:

Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/md1      ext3    1.8T  1.8T     0 100% /

Why does everything show /dev/md1 to be 3TB but the space is limited to only 2TB?

Other details: Ubuntu 10.10

UPDATE:

$ tune2fs -l /dev/md1:

tune2fs 1.41.12 (17-May-2010)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          1b4e9420-61ee-4ffd-817a-28831f2aeaf2
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags:         signed_directory_hash
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              122068992
Block count:              488255968
Reserved block count:     24412798
Free blocks:              69746292
Free inodes:              121882953
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      907
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
RAID stride:              16
RAID stripe width:        32
Filesystem created:       Sun Oct 17 01:14:23 2010
Last mount time:          Mon Dec  6 01:24:31 2010
Last write time:          Mon Dec  6 00:45:41 2010
Mount count:              12
Maximum mount count:      23
Last checked:             Sun Dec  5 20:37:35 2010
Check interval:           15552000 (6 months)
Next check after:         Fri Jun  3 21:37:35 2011
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
First orphan inode:       55550208
Default directory hash:   half_md4
Directory Hash Seed:      2a386fcb-b24f-4f40-bf4c-7c03489b086c
Journal backup:           inode blocks

The simple question: how does initramfs know how to assemble mdadm RAID arrays at startup?

My problem: I boot my server and get:

Gave up waiting for root device.
ALERT! /dev/disk/by-uuid/[UUID] does not exist. Dropping to a shell!

This happens because /dev/md0 (which is /boot, RAID 1) and /dev/md1 (which is /, RAID 5) are not being assembled correctly. What I get is /dev/md0 isn't assembled at all. /dev/md1 is assembled, but instead of using /dev/sda2, /dev/sdb2, /dev/sdc2, and /dev/sdd2, it uses /dev/sda, /dev/sdb, /dev/sdc, /dev/sdd.

To fix this and boot my server I do:

$(initramfs) mdadm --stop /dev/md1
$(initramfs) mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
$(initramfs) mdadm --assemble /dev/md1 /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2
$(initramfs) exit

And it boots properly and everything works. Now I just need the RAID arrays to assemble properly at boot so I don't have to manually assemble them. I've checked /etc/mdadm/mdadm.conf and the UUIDs of the two arrays listed in that file match the UUIDs from $ mdadm --detail /dev/md[0,1].

Other details: Ubuntu 10.10, GRUB2, mdadm 2.6.7.1

UPDATE: I have a feeling it has to do with superblocks. $ mdadm --examine /dev/sda outputs the same thing as $ mdadm --examine /dev/sda2. $ mdadm --examine /dev/sda1 seems to be fine because it outputs information about /dev/md0. I don't know if this is the problem or not, but it seems to fit with /dev/md1 getting assembled with /dev/sd[abcd] instead of /dev/sd[abcd]2.

I tried zeroing the superblock on /dev/sd[abcd]. This removed the superblock from /dev/sd[abcd]2 as well and prevented me from being able to assemble /dev/md1 at all. I had to $ mdadm --create to get it back. This also put the super blocks back to the way they were.

I have a 4 drive RAID array. A small part of each disk is partitioned and used in RAID 1 for /boot. Everything else is RAID 5. When I start the computer GRUB2 shows up and I can use its command line and view each of my RAID arrays (ls (md0)/ or ls (md1)/etc, for instance). However, when I actually try to have GRUB2 boot the system I get these errors:

modprobe: FATAL: Could not load /lib/modules/2.6.35-22-generic-pae/modules.dep: No such file or directory

mount: mounting /dev/disk/by-uuid/[uuid] on /root failed: Invalid argument
mount: mounting /dev on /root/dev failed: No such file or directory
mount: mounting /sys on /root/sys failed: No such file or directory
mount: mounting /proc on /root/proc failed: No such file or directory
Target filesystem doesn't have requested /sbin/init.
No init found. Try passing init= bootarg.

Here are the GRUB2 commands that are being run:

recordfail
insmod raid
insmod mdraid
insmod part_msdos
insmod ext2
set root='(md0)'
search --no-floppy --fs-uuid --set [uuid]
linux /vmlinuz-2.6.35-22-generic-pae root=UUID=[different uuid] ro quiet
initrd /initrd.img-2.6.35-22-generic-pae

I have tried

fsck /dev/md0
fsck /dev/md1

and both arrays come out clean. I've booted to the CD and looked everywhere I can think of for a discrepancy that would cause GRUB2 to fail to boot. Using the GRUB2 command line and the CD command line everything appears to be in working order. Both RAID arrays show up active and not degraded. What is keeping GRUB2 from being able to mount and boot the system?