Adrian Heine's questions

I have a failing hard disk (let's call it sda) which contains, among other things, a 1.5 TB partition (let's call it sda3). There is another disk (sdb), which has a 1.5 TB partition (sdb1) as well. Both used to be part of an mdadm level 1 RAID using metadata version 1.2. Inside this RAID partition (let's call id md5), there is a LUKS encryption container (let's call it md5_uncrypted). This container should contain an ext4 partition.

At some place in time around August 10, 2012, I somehow restarted the RAID array with sdb1 missing and didn't even notice that. When I wanted to replace the RAID yesterday (three months later), I started copying data from sdb1 until I realized that it was out of date. So I took a look at the old sda3. By mistake, I ran mdadm --create instead of mdadm --assemble to restart md5 with only sda3 available. Accordingly, I ignored all warnings and let mdadm --create continue. cryptsetup didn't like the content of the new RAID. I didn't actually think mdadm --create would corrupt data if the same metadata version is used? Well, apparently it did.

I compared the first 20 MB of sda3 and sdb1 and noticed that they are equal starting at about 8 MB. So I copied the first 8 MB of sdb1 to sda3 (I have a backup of the old first 20 MB of sda3) and tried to assemble md5 (with only a single drive, sda3). Unfortunately, this gave me an error:

failed to add /dev/sdb1: Invalid argument

I also tried using the LUKS header from sdb1 on a freshly mdadm --created sda3, which cryptsetup happily did (of course), but it contained garbage.

My question is: Is there any chance to restore at least some of the data from sda3? Since I have the state from three months ago, everything helps, even just a list of files, or a list of files with modification dates.

Edit:

# mdadm --examine /dev/sdb1
         Magic : a92b4efc
       Version : 1.2
   Feature Map : 0x0
    Array UUID : 76a25475:70aa881c:dd30cc93:bbae03b7
          Name : ubuntu:0
 Creation Time : Fri Mar 16 20:52:16 2012
    Raid Level : raid1
  Raid Devices : 2

Avail Dev Size : 2930272256 (1397.26 GiB 1500.30 GB)
    Array Size : 1465129848 (1397.26 GiB 1500.29 GB)
 Used Dev Size : 2930259696 (1397.26 GiB 1500.29 GB)
   Data Offset : 2048 sectors
  Super Offset : 8 sectors
         State : clean
   Device UUID : b9012482:afa502cf:7794f4fb:2a0da196

   Update Time : Wed Nov 21 20:51:51 2012
      Checksum : 4e54a07 - correct
        Events : 15003


  Device Role : Active device 1
  Array State : .A ('A' == active, '.' == missing)

# mdadm --examine /dev/sda3
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 54ea031e:42367512:b6a8675b:91f2cb6f
           Name : willow:5  (local to host willow)
  Creation Time : Wed Nov 21 18:03:35 2012
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 2929999872 (1397.13 GiB 1500.16 GB)
     Array Size : 1464999744 (1397.13 GiB 1500.16 GB)
  Used Dev Size : 2929999488 (1397.13 GiB 1500.16 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 93c70c36:3cc540a5:13817695:bd4f327c

    Update Time : Wed Nov 21 18:03:35 2012
       Checksum : 321ddb3e - correct
         Events : 0


   Device Role : Active device 1
   Array State : .A ('A' == active, '.' == missing)

I’m running a Debian Squeeze AMD64 server. Target runlevel after boot is runlevel 2, which includes rsyslogd, cron, sshd and some other stuff, but not dovecot, postfix, apache2, etc. The system fails to reach runlevel 2 with several symptoms:

• The system hangs at trying to start rsyslogd
• Booting into runlevel 1 works, then login from the console works
• Starting rsyslogd from runlevel 1 via /etc/init.d/rsyslog hangs
• Starting runlevel 2 with rsyslogd disabled works
• But then, logging in via console fails: I get the motd, and then nothing
• Starting sshd from runlevel 1 succeeds
• But then, I cannot login via ssh. Sometimes password ssh login gives me the motd and then nothing, sometimes not even this. Trying to offer a public key seems to annoy the sshd enough to not talk to me any further.
• When rebooting from runlevel 1, the server hangs at trying to stop apache2 (which is not running, so this really should be trivial). Trying to stop apache2 when logged in in runleve 1 does hang as well.

And that’s just the stuff which fails all the time. RAM has been tested, dmesg shows no problems. I have no clue.

Update: (shortened) output from rsyslogd -c4 -d called in runlevel 1

rsyslogd 4.6.4 startup, compatibility mode 4, module path '' caller requested object 'net', not found (iRet -3003) Requested to load module 'lmnet' loading module '/user/lib/rsyslog/lmnet.so' module of type 2 being loaded conf.c requested ref for 'lmnet', refcount 1 rsylog runtime initialized, version 4.6.4, current users 1 syslogd.c requested ref for 'lmnet', refcount now 2

I can kill rsyslogd with Strg+C, then. /var/log shows none of the configured log files, though.

Update2: Thanks to @DerfK I still have no clue, but at least I narrowed down the problem. I’m now testing with /etc/init.d/apache2 stop (without an apache2 running, of course) which hangs as well and looks like an even more obvious failure.

After some testing I found out that a file with one single line:

/usr/sbin/apache2ctl configtest > /dev/null 2>&1

hangs, while the same line executed in an interactive shell works. I was not able to further reduce this line while, i. e. every single part, the stream redirections and the commando itself is necessary to reproduce the hang. @DerfK also pointed me to strace which gave a shallow hint about what kind of hang we have here:

• wait4(-1for the init scripts
• futex(0xsomepointer, FUTEX_WAIT_PRIVATE, 2, NULL for rsyslogd / apache2 binaries called by the init scripts

The system was installed as a Debian Lenny by my hoster in autumn 2011, I upgraded it to Squeeze immediately and kept it up to date with Squeeze, which then used to be testing. There were no big changes, though. I guess I never tried to reboot the system before.

Update3: I found the problem. My /etc/nsswitch.conf specified ldap as hosts lookup backup, which is not available at that time of the boot. Relying on dns solely fixes my boot problems.