Question 1 - Before you answer with "it just takes the smaller disk", hear me out quick. My 3TB WD Reds come out to be 3001 GB in size. Let's say I set up a mirror via mdadm for sdb1 and sdc1 which span 100% of the drive. But suddenly, one of the drives fail. The replacement is a 3TB, weighing in at 3000 GB. What happens when I put in a drive that is smaller than the one currently existent on the array? I know with a new array using 3000 vs 3001, it would build the array to be 3000. But like I said, what about a current array @ 3001 and I add a smaller drive? Does it re-structure itself during the rebuild to be 3000 GB in size?
Question 2 - In the event that I can't add a 3000 GB to the array with an existing 3001 GB and it simply downsize to 3000... can I resize the 3001 down a bit?
Question 3 - Or, a better idea. What if I downsize my 3TB drive to be 2999 GB. That way whether the drive is short by 1 MB, 1 byte, 10 KB, doesn't matter, it'll always pick up the "smaller" drive @ 2999 GB.
I came across this answer by mistake, but in case anyone is curious, here's an answer supported by experiments.
The Short Version
Bonus Question: can I create an
md(4)
RAID array out of block devices of unequal size? Yes, but the RAID array will have the size of the smallest block device (plus some overheads for its own housekeeping). If device sizes aren't within 1% of each other, you get a warning.Question 1: can I add to an existing
md(4)
RAID array a device smaller than the smallest current member? Nope, sorry.mdadm
will flat out refuse to do that to protect your data.Question 2: can you resize an existing md array? Yes (read the
mdadm
manpge!), but it may not be worth the effort. You'll have to back everything up, then resize the contents of the RAID device, then resize the device itself — all of this is quite prone to errors, miscalculations, and other things that'll cost you your data (painful experience talking).It's not worth the risk and effort. If you have a new, blank disk, here's how to resize it and also keep between one and two copies of all your data intact at all times (assuming you have 2-disk RAID1):
md(4)
array on it (with one disk missing).md(4)
superblock). If necessary, create the required partitions to match the scheme on he new disk.Note: yes, this is the same technique 0xC0000022L described in his answer.
Question 3. What if the drive is 1G short? :) Don't worry about it. Chances are your replacement drive will be bigger. In fact, with a strategy like above it pays to get cheaper larger drives whenever one fails (or for a cheaper upgrade). You can get a progressive upgrade.
Experimental Proof
Experimental Setup
First, let's fake some block devices. We'll use
/tmp/sdx
and/tmp/sdy
(each 100M), and/tmp/sdz
(99M).This sets up three files as three loopback block devices:
/dev/loop0
,/dev/loop1
and/dev/loop2
, mapping tosdx
,sdy
andsdz
respectively. Let's check the sizes:As expected, we have two loop devices of exactly 100M (102400 KiB = 100 MiB) and one of 99M (exactly 99×1024 1K blocks).
Making a RAID Array out of Identically-Sized Devices
Here goes:
Check the size:
This is precicely what we expect: one look at the mdadm manual reminds us that version 1.2 metadata take up 128K: 128 + 102272 = 102400. Now let's destroy it in preparation for the second experiment.
Making a RAID Array out of Unequally Sized Devices
This time we'll use the small block device.
Well, we got warned, but the array was made. Let's check the size:
What we get here is 101,248 blocks. 101248 + 128 = 101376 = 99 × 1024. The usable space is that of the smallest device (plus the 128K RAID metadata). Let's bring it all down again for our last experiment:
And Finally: Adding a smaller Device to a Running Array
First, let's make a RAID1 array with just one of the 100M disks. The array will be degraded, but we don't really care. We just want a started array. The
missing
keywords is a placeholder that says ‘I don't have a device for you yet, start he array now and I'll add one later’.Again, let's check the size:
Sure enough, it's 128K short of 102400 blocks. Adding the smaller disk:
Boom! It won't let us, and the error is very clear.
There are several ways to set up
mdX
devices. The method would be to usegdisk
(orsgdisk
if you prefer the command-line only version) to partition this as GPT. If you want to boot from the array create a "BIOS Boot Partition", type codeef02
. This is only necessary if you want to boot off this array, otherwise no need to care. Then, create a partition the same size or smaller than the smallest disk to be added to the array. Last but not least, copy the GPT data over to the other disk (expert menu ingdisk
, usingx
, and thenu
and specify the target device). This is a destructive process.It should be possible - if the file system allows for it - to resize an existing partition to something smaller and then use the same method to copy the GPT data. However, this gets you into a bit of a kerfuffle. Because now you have two disks, but still no
mdX
device. One of them has to be prepared asmdX
, either partition-wise (which I implied above) or disk-wise) and then the data must be moved from the existing disk to that.So:
/dev/sda
) contains data, data is smaller than 3001 GB, partitions are not/dev/sdb
gets added to the system/dev/sdb
withgdisk
mdadm -C /dev/md2 -l 1 -n 1 /dev/sdb2
)/dev/sdb
to/dev/sda
/dev/sda
into the existing arrays/proc/mdstat
to show you that the synching is doneIf you followed all steps you should now be able to boot into the new system off the mdX arrays. However, keep a rescue CD or a PXE boot option handy, just in case.
GRUB2 will not be able to recognize the setup off hand. So you need some "magic". Here's a one-liner:
Or let's be more verbose:
This creates (or overwrites) the default
/boot/grub/devicemap
with one that tells GRUB2 where to find each respective disk. The result would be something like this list:If you use legacy GRUB, you also need to create the "BIOS Boot Partition" with meta-data version 0.9, using
mdadm -e 0 ...
and the process will differ. I haven't done that, though.