Home / server / Questions / 821966
In Process
captain lizard
Asked: 2016-12-21 22:55:57 +0800 CST

How to re-add a drive when RAID is in degraded mode?

  • 772

I am running an unmanaged Ubuntu server and due to recent performance issue and sent a request to the provider only to receive the following response:

"Dear Client,

as requested we have checked your drives. Both drives pass a test without any error and do not show any critical value here. We noticed the RAID is degraded state so you will need to re-add the drive to array here:

-----------------%<-----------------
Personalities : [raid1]
md3 : active raid1 sda4[0]
      1839089920 blocks super 1.2 [2/1] [U_]

md2 : active raid1 sda3[0]
      1073610560 blocks super 1.2 [2/1] [U_]

md1 : active raid1 sda2[0]
      523968 blocks super 1.2 [2/1] [U_]

md0 : active raid1 sda1[0]
      16768896 blocks super 1.2 [2/1] [U_]

unused devices: <none>
-----------------%<-----------------"

Google says that U_ indicates a failed drive, which appears to be in contradiction to what the support said. How do I proceed from here?

Something like 

mdadm /dev/mdX -r /dev/sdaY  
mdadm /dev/mdX -a /dev/sdaY

And if so, what is X and Y for my case? If more info is needed about my configuration, please let me know.

EDIT, more info below:

root@Ubuntu-1204-precise-64-minimal /home # df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/md2       1008G   26G  931G   3% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev             16G  4.0K   16G   1% /dev
tmpfs           3.2G  504K  3.2G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none             16G     0   16G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/md1        488M   66M  397M  15% /boot
/dev/md3        1.7T   31G  1.6T   2% /home

root@Ubuntu-1204-precise-64-minimal /home # mdadm -D /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Wed Nov  6 08:02:41 2013
     Raid Level : raid1
     Array Size : 16768896 (15.99 GiB 17.17 GB)
  Used Dev Size : 16768896 (15.99 GiB 17.17 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Wed Dec 21 03:50:13 2016
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : rescue:0
           UUID : 872ad258:c42ccb36:e9e19c96:98b55ee9
         Events : 816844

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       0        0        1      removed

root@Ubuntu-1204-precise-64-minimal /home # mdadm -D /dev/md1
/dev/md1:
        Version : 1.2
  Creation Time : Wed Nov  6 08:02:41 2013
     Raid Level : raid1
     Array Size : 523968 (511.77 MiB 536.54 MB)
  Used Dev Size : 523968 (511.77 MiB 536.54 MB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Wed Dec 21 07:30:55 2016
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : rescue:1
           UUID : 18cb39fc:9eaea61c:0074a6c2:661b5862
         Events : 387

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       0        0        1      removed

root@Ubuntu-1204-precise-64-minimal /home # mdadm -D /dev/md2
/dev/md2:
        Version : 1.2
  Creation Time : Wed Nov  6 08:02:42 2013
     Raid Level : raid1
     Array Size : 1073610560 (1023.87 GiB 1099.38 GB)
  Used Dev Size : 1073610560 (1023.87 GiB 1099.38 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Wed Dec 21 07:58:52 2016
          State : active, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : rescue:2
           UUID : eb9be750:7ff778b4:31fd7ce9:9d86d191
         Events : 38799724

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       0        0        1      removed

root@Ubuntu-1204-precise-64-minimal /home # mdadm -D /dev/md3
/dev/md3:
        Version : 1.2
  Creation Time : Wed Nov  6 08:02:42 2013
     Raid Level : raid1
     Array Size : 1839089920 (1753.89 GiB 1883.23 GB)
  Used Dev Size : 1839089920 (1753.89 GiB 1883.23 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Wed Dec 21 07:57:41 2016
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : rescue:3
           UUID : c9b748ef:332d3bf9:5fa8fef1:5b433b0a
         Events : 7429895

    Number   Major   Minor   RaidDevice State
       0       8        4        0      active sync   /dev/sda4
       1       0        0        1      removed
ubuntu raid mdadm raid1 md

3 Answers

  1. Adding Back a RAID 1 Member

    To add a RAID member back into the array, use:

    mdadm <RAID Device> -a <Device to add into the array>

    For example:

    $ mdadm /dev/md0 -a /dev/sdc1
mdadm: re-added /dev/sdc1

    If you check again using mdadm --detail , you'll see that /dev/sdc1 is shown as "rebuilding":

    Number   Major   Minor   RaidDevice State
   0       8       17        0      active sync   /dev/sdb1
   2       8       33        1      spare rebuilding   /dev/sdc1

    This means that the newly added member is being synchronised with the data on the other member; if you now check proc/mdstat, you can see the synchronisation progress - as a percentage and a bar chart:

    $ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sdc1[2] sdb1[0]
      976759936 blocks [2/1] [U_]
      [>....................]  recovery =  1.2% (12409024/976759936) finish=195.2min speed=82312K/sec

unused devices: <none>

    Verify that you can still see your files on the RAID drive following the addition of the new RAID member:

    $ ls /mnt/raid1
Music  lost+found  Images Films Misc
    • 19

  2. I solved my own issue. I did not understand that mdX are partitions and sda and sdb are actual physical discs. So sdb was missing, so I needed to add it back. I executed the following commands:

    sgdisk -R /dev/sdb /dev/sda
sgdisk -G /dev/sdb
mdadm /dev/md0 -a /dev/sdb1
mdadm /dev/md1 -a /dev/sdb2
mdadm /dev/md2 -a /dev/sdb3
mdadm /dev/md3 -a /dev/sdb4

    After a day of resync (can be observed with)

    cat /proc/mdstat

    What I see today looks good again:

    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md3 : active raid1 sdb4[2] sda4[0]
      1839089920 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sdb2[2] sda2[0]
      523968 blocks super 1.2 [2/2] [UU]

md2 : active raid1 sdb3[2] sda3[0]
      1073610560 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sdb1[2] sda1[0]
      16768896 blocks super 1.2 [2/2] [UU]
    • 2

  3. Before adding the drive as suggested by @Tolsadus its a good idea to check your drive health using smartmontools by executing short and long tests:

    Initially you have to ensure that the hard disk supports SMART and is enabled:

    smartctl -i /dev/sdb

    short test:

    smartctl -t short /dev/sdb

    long test:

    smartctl -l selftest /dev/sdb

    • 1