Home / server / Questions / 427683
In Process
Adam Ryczkowski
Asked: 2012-09-16 00:48:00 +0800 CST

What parameters to mdadm, to re-create md device with payload starting at 0x22000 position on backing storage?

  • 772

I try to recover from mdadm raid disaster, which happened when moving from ubuntu server 10.04 to 12.04. I know the correct order of devices from dmesg log, but given this information, I still cannot access the data.

The superblocks look messy; the mdadm --examine for each disk is on this question on askubuntu

By inspecting the raw contents of backing storage, I found the beginning of my data (the LUKS container in my case) at position 0x22000 relative to the beginning of the first partition in the raid.

Question: What is the combination of options issued to "mdadm --create" to re-create mdadm that starts with the given offset? Bitmap size?

PS. The relevant information from syslog when the system was healthy are pasted here.

linux

1 Answers

  1. It appears, that there is no need for any tool for setting start of the RAID payload on any given offset. The problem arose from change in offset between mdadm versions above 3.00 and below. It appears, that at the moment, due to the different offsets, the mdadm from version above 3.00 (I tried 3.24) cannot cannot re-create layout made by older mdadm. One needs to use the older one.

    The problem appeared in the first place from a known bug in Ubuntu 12.04 kernel, which under certain circumstances (e.g. found during distribution upgrade...) overwrites the superblocks. The solution is to use avoid kernels from 3.2.0-22.35 to 3.2.0-24.37, which incidentaly include among other, kernel used on Ubuntu 12.04 Mini-cd installer.

    This is the conversation in [email protected] mailing list, in which Mr. Robin Hill gave me the solution:

    Looks like you've been hit by the bug mentioned here: http://neil.brown.name/blog/20120615073245

    You've got most of the required data to recreate the array. It's just the chunk size missing I think. This is probably 512K as the arrays are using 1.2 superblocks, but it may be 64K if you were using an old mdadm version.

    Your data offset is 272 sectors, which means you'll need to use an older mdadm version to re-create the array. Newer versions use 2048 sectors, but it looks like version 3.0 uses 272, so grab a copy of that if possible.

    Assuming your drives are numbered in the same order as previously, the create commands will be:

    mdadm -C /dev/md5 -e 1.2 -l 5 -n 5 /dev/sd{b,f,c,e,d}6 \
    --assume-clean --uuid=a3945c40:73237cd6:3d61998e:8f773d03

    and

    mdadm -C /dev/md6 -e 1.2 -l 6 -n 5 /dev/sd{b,f,c,e,d}5 \
    --assume-clean --uuid=2a8fd2df:3ad53552:e8b80ecf:a46d93a2

    Make sure you use the curly brace format to list the disks rather than square brackets - that'll ensure the order is maintained. Also make sure that --assume-clean is used to prevent rebuilding from kicking off, and losing data.

    After recreating the array, run a fsck -n on the array to check whether the details are correct. If not, stop the array and try recreating it again, adding "-c 64" to set the chunk size to 64K.

    Once you've got them up and fsck shows clean, you can re-add the bitmaps using mdadm -G /dev/mdX --bitmap=internal.

    Do read through Neil's blog post first as there may be something I've missed.

    So after booting from XUbuntu 10.04 live CD I was able to successfully re-create the array with the given commands, and get access to all my data.

    It is important to note, that the old mdadm was needed only for re-creation the array. The array can be normally accessed with 12.04 live CD (using usual mdadm --assemble --scan).

    If one wants to change the layout of the raid to the newer one (with the offset 2048 sectors) the path is long:

    The only way I know of to update the offset location in an existing array would be to fail the disks one at a time, zeroing the superblocks, then adding them back into the array using a new mdadm. When it's added back in, it'll use the new offset location. If you have a spare disk and a new enough kernel version, you can use the pre-emptive replacement option, which will prevent the need to run without full parity available.

    ...In hope that this post would keep at least one system administrator upgrading from 10.04 to 12.04 from heart-attack ;-)

    See also related problem on askubuntu: mdadm fails after 10.04 -> 12.04 upgrade.

    • 1