Ben Hymers's questions

I have had a hard drive failure which has caused a lot of processes to go into an uninterruptable state ("D" state in ps- See this question). I cannot kill these processes, so I wish to shut down, but unfortunately halt is also now showing as "D" and the machine will not restart!

ben@jack:~$ ps aux | grep "halt"
root      7875  0.0  0.0   4148   700 ?        S    Mar08   0:00 /bin/sh /etc/rc0.d/S01halt stop
root      7880  0.0  0.0  10436   680 ?        D    Mar08   0:00 halt -d -f -i -p

How can I recover from this?

The short version: I have a failed RAID 5 array which has a bunch of processes hung waiting on I/O operations on it; how can I recover from this?

The long version: Yesterday I noticed Samba access was being very sporadic; accessing the server's shares from Windows would randomly lock up explorer completely after clicking on one or two directories. I assumed it was Windows being a pain and left it. Today the problem is the same, so I did a little digging; the first thing I noticed was that running ps aux | grep smbd gives a lot of lines like this:

ben        969  0.0  0.2  96088  4128 ?        D    18:21   0:00 smbd -F
root      1708  0.0  0.2  93468  4748 ?        Ss   18:44   0:00 smbd -F
root      1711  0.0  0.0  93468  1364 ?        S    18:44   0:00 smbd -F
ben       3148  0.0  0.2  96052  4160 ?        D    Mar07   0:00 smbd -F
...

There are a lot of processes stuck in the "D" state. Running ps aux | grep " D" shows up some other processes including my nightly backup script, all of which need to access the volume mounted on my RAID array at some point. After some googling, I found that it might be down to the RAID array failing, so I checked /proc/mdstat, which shows this:

ben@jack:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sdb1[3](F) sdc1[1] sdd1[2]
      2930271872 blocks level 5, 64k chunk, algorithm 2 [3/2] [_UU]

unused devices: <none>

And running mdadm --detail /dev/md0 gives this:

ben@jack:~$ sudo mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90
  Creation Time : Sat Oct 31 20:53:10 2009
     Raid Level : raid5
     Array Size : 2930271872 (2794.53 GiB 3000.60 GB)
  Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
   Raid Devices : 3
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Mon Mar  7 03:06:35 2011
          State : active, degraded
 Active Devices : 2
Working Devices : 2
 Failed Devices : 1
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : f114711a:c770de54:c8276759:b34deaa0
         Events : 0.208245

    Number   Major   Minor   RaidDevice State
       3       8       17        0      faulty spare rebuilding   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       2       8       49        2      active sync   /dev/sdd1

I believe this says that sdb1 has failed, and so the array is running with two drives out of three 'up'. Some advice I found said to check /var/log/messages for notices of failures, and sure enough there are plenty:

ben@jack:~$ grep sdb /var/log/messages

...

Mar  7 03:06:35 jack kernel: [4525155.384937] md/raid:md0: read error NOT corrected!! (sector 400644912 on sdb1).
Mar  7 03:06:35 jack kernel: [4525155.389686] md/raid:md0: read error not correctable (sector 400644920 on sdb1).
Mar  7 03:06:35 jack kernel: [4525155.389686] md/raid:md0: read error not correctable (sector 400644928 on sdb1).
Mar  7 03:06:35 jack kernel: [4525155.389688] md/raid:md0: read error not correctable (sector 400644936 on sdb1).
Mar  7 03:06:56 jack kernel: [4525176.231603] sd 0:0:1:0: [sdb] Unhandled sense code
Mar  7 03:06:56 jack kernel: [4525176.231605] sd 0:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Mar  7 03:06:56 jack kernel: [4525176.231608] sd 0:0:1:0: [sdb] Sense Key : Medium Error [current] [descriptor]
Mar  7 03:06:56 jack kernel: [4525176.231623] sd 0:0:1:0: [sdb] Add. Sense: Unrecovered read error - auto reallocate failed
Mar  7 03:06:56 jack kernel: [4525176.231627] sd 0:0:1:0: [sdb] CDB: Read(10): 28 00 17 e1 5f bf 00 01 00 00

To me it is clear that device sdb has failed, and I need to stop the array, shutdown, replace it, reboot, then repair the array, bring it back up and mount the filesystem. I cannot hot-swap a replacement drive in, and don't want to leave the array running in a degraded state. I believe I am supposed to unmount the filesystem before stopping the array, but that is failing, and that is where I'm stuck now:

ben@jack:~$ sudo umount /storage
umount: /storage: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

It is indeed busy; there are some 30 or 40 processes waiting on I/O.

What should I do? Should I kill all these processes and try again? Is that a wise move when they are 'uninterruptable'? What would happen if I tried to reboot?

Please let me know what you think I should do. And please ask if you need any extra information to diagnose the problem or to help!

I'm just assessing the security of my home server in preparation for opening it up to the internet. I've used nmap from a machine on my local network but I have firewall rules allowing access from the local network so this reveals more than I think will be visible from the internet.

Is there any way to 'spoof' the address nmap is scanning from, so I can see what a potential attacker would see? Any google search for that kind of thing yields a lot of very shady looking websites...

a while ago I gave root a password so I could log in as root and get some stuff done. Now I want to disable root login to tighten security, since I'm going to be exposing my serve to the internet. I've seen several ways of doing this (sudo passwd -l root, fiddling with /etc/shadow, and so on), but nowhere that says what the best/most sensible way of doing it is. I've done sudo passwd -l root but I've seen advice that says this can affect init scripts, and that it's not as secure as it looks since it still asks for a password if you try to log in, rather than flat out denying access. So what would be the way to achieve that?

EDIT: to clarify, this is for local login as root; I've already disabled remote login via SSH. Though trying to log in as root over SSH still prompts for root's password (which always fails). Is that bad?

I've got a backup script written in Python which creates the destination directory before copying the source directory to it. I've configured it to use /external-backup as the destination, which is where I mount an external hard drive. I just ran the script without the hard drive being turned on (or being mounted) and found that it was working as normal, albeit making a backup on the internal hard drive, which has nowhere near enough space to back itself up.

My question is: how can I check whether the volume is mounted in the right place before writing to it? If I can detect that /external-backup isn't mounted, I can prevent writing to it.

The bonus question is why was this allowed, when the OS knows that directory is supposed to live on another device, and what would happen to the data (on the internal hard drive) should I later mount that device (the external hard drive)? Clearly there can't be two copies on different devices at the same path!

Thanks in advance!

I'm just trying to copy a directory from one home server to another, both running Ubuntu (9.04 and 9.10), using rsync. The trouble I'm having is that some of the files in the directory (and subdirectories) are owned by root. The main administrator account on both machines is 'ben', so running this on the old server:

rsync -v -u -a -p -t -rsh=ssh --stats --progress source/ ben@newserver:/dest

Prompts me for ben@newserver's password, copies source/ but all the permissions are 'ben' at the new destination. Running this using sudo means I have access to root's files, but on the other end ben can't write them as belonging to root. Sending to root@newserver asks for root's password, which Ubuntu doesn't give you.

Can anyone explain simply how to do this? Thanks :)