Home / server / Questions / 259547
Accepted
m.sr
Asked: 2011-04-15 00:29:02 +0800 CST

Xen: Convert image to lvm with minimal downtime

  • 772

My Goals:

  • Convert all my DomUs from image-based to lvm-based
  • Minimal downtime
  • Minimal performance impact on other DomUs/Dom0

My plan / setup:

All my images are in a big lvm-volume, all lvs for the new setup are created with the filesystem (ext3/ext4). So my approche is, to do this for every single machine:

  1. make a snapshot of the big image-lv
  2. mount this snapshot (e.g. on /tmp/img_snap/)
  3. loopback-mount the image itself (e.g. on /tmp/convert_src/)
  4. mount the new lv (e.g. on /tmp/convert_dest)
  5. rsync from /tmp/convert_src to /tmp/convert_dest
  6. unmount everything
  7. remove the lv-snapshot
  8. shutdown the DomU
  9. do steps 1-7 again
  10. change the disk-settings in the DomU's config
  11. fire up the domU again

Steps 1-7 are already scripted so in step 9 I just can fire up this script again. This means a bit of overhead because the lv-snapshot is not necessary, but I'm ok with and aware of that.

Because of the fact that the original machine can run until step 8 (and step 9 should only copy the delta) the downtime should be minimal. I also renice and ionice the rsync process(es) so the impact on all other systems should be minimal. Or so i thought ...

  • Debian lenny
  • Xen 3.2 from debian
  • HP ML350 G5 with their SmartArray Controller (iirc: e220?)
  • drives connected via SATA

Questions / Problems:

  • This approach is ok and tested with small images, but I have some big images 400GB+. This takes ages. Are there any better approaches?
  • Sometimes - I don't exactly know why - the load of the Dom0 goes up way to much, the machine gets 'stuck' and i get messages like INFO: task loop27:23352 blocked for more than 120 seconds, INFO: task pdflush:7329 blocked for more than 120 seconds., ... and their call traces and stuff. Why/When? I can't figure out a pattern? Is there too much IO, too CPU? And most of all: Why doesn't get my rsync-processes blocked - i reniced (renice 20 -p $(pidof rsync)) and ioniced (ionice -c2 -n7 -p$pid) them - shouldn't these processes be the one to be first blocked?
  • Generaly any hints and improvement ideas?
xen lvm rsync image convert

1 Answers

  1. Best Answer

    rsync is a very good tool. If you use it, take the options "-aSH --delete" so you will get an amlost identical target (ctime-stamps of soft-links will have the current time).

    From your writing I understand that the time needed for the rsync seems to be your main problem, while shutdown/startup of the DomUs is pretty fast (including their applications, I assume).

    I see another problem in your snapshot of your big LV holding all your running DomUs. If you have many write-requests your snapshot will fill up pretty fast - and the snapshot will slow down the DomUs during writes.

    In this situation I would chosse a different approach: Use a software raid 1 to mirror from your images to your new LVs.

    If you want you could combine that with snapshot/rsync to reduce the amount of data that needs to be moved. But IMHO the time neede to get that one sorted out and scripted is not worth the effort.

    So here is the plan - in both cases you need block-devices (loopback-mount your images):

    A) Using md (set up via mdadm) Write a script to:

    1. Shut down DomU
    2. Loopback-Mount the DomUs image
    3. Create a degraded md-raid-1-device with that image
    4. Boot up your DomU with "phy:mdN" as new disk device
    5. Raise number of raid1-members from 1 to 2
    6. add your target LV
    7. Change the setup of your DomU so it will use the phy:vgX/lvY in the future
    8. If the mirror is complete, restart your DomU with the new setup

    Whith plan A you have two downtimes lasting one restart of the DomU. The mirroring will take place in the background as fast as possible.

    Drawback: All data will by synced (RAW mirror, not based on file-system)

    You could get around that by creating the md-raid1 with (--assume clean) without degradation. But this will be tricky (you will need rsync to get your data to the LV then mirror the delta using md-bitmap).

    B) Using drbd (set up via /etc/drbd.conf) If you plan to switch to drbd in the future you should go this way. I did not test this, but can't see a reason why drbd should not mirror between 127.0.0.1 and 127.0.0.2 ;-)

    I did not test this, but it should work - and is more complicated than plan A.

    Drawback: Complicated, not tested.

    But: If you want to switch to drbd anyway your can prepare your DomU this way. Run in disconnected mode until your cluster is ready, then connect, sync via IP and start live-migrating...

    • 0