Xen Live Migration Across Network Storage

Currently, all the visualization hyper-visors can live migrate (xen, kvm, vmware, openvz, hyper-v). But for this you need a shared storage. You have 3 options :

• SAN ( FC or iSCSI ), but you need a cluster file system on top of it (ocfs2, gfs, vmware, ...). The number of nodes is limited to the parameter of the initialization of the file system (with ocfs2 at least).
• NAS ( NFS ). I did test this solution successfully with Netapp FAS2020, and performances are comparable with iSCSI. But I did not tested enough to check the behaviour in case of failure of the host, and the data integrity. The article : http://thesantechnologist.com/?p=52 has a link with really shows excellent results on nfs. I tested and can confirm that NFS is as fast as iSCSI for few concurrent VMs.
• DRBD (active-active): This solution is very similar to SAN, but the active-active DRBD cluster is very recent, and I don't know about the reliability. I used DRBD in the past but with nbd on top of it because the active-active was not yet available. I also used NFS to share between nodes. Both solutions are working (active-passive + NFS) and (active-passive + NBD + OCFS2).

In short :

• SAN is the most proven solution, and definitely the most expensive especially if you go for FC.
• NFS is the easiest solution.
• DRBD is the cheapest solution, limited to 2 active nodes.

I'm far from being the most experiencing but here is what I think : - Unless you have a lot of €€€€, FC is useless. As virtualization slowdown really comes when you have a lot of random read, the bandwidth at this time is low, and iSCI can perfectly handle it. - iSCSI is the most scalable. This will work for an entry system, and you can get more performances by turning to a dedicated iSCSI controller, and if still not enough change your storage to a faster one. If you really need something impressively fast, Dell can sell you equalogic storages with SSD (SLC technology) only. Expensive, but the performances are that high that the performance/price ratio is still excellent. - NFS might be a very good solution too, but I miss experience, and my only fear is data loss in case of power failure. (While you can also have data loss with other technologies.) - DRBD will bring satisfaction in a SMI/very low cost solution.

I'm very satisfied with Netapp, and I would recommend 2 Netapp FAS-2020 with 2 controllers in each boxes for both virtualization and CIFS file server. NFS is also very good. I'm also very satisfied with NetApp deduplication. So, if you have some money for virtualisation, start with 2 NetApp 2020 boxes (with 15krpm drives). ( You have to count 20k - 30k€ for this )

Regarding virtualization technologies I mainly used xen, kvm and openvz. And as virtualization solutions, I'm using OracleVM in production for 5 years now, but I'm interested in changing because the storage is not efficient (only raw format with OracleVM). I think the best Open Source pretender today is Proxmox ve.

If I had to restart a virtualization project from the begining, I would also have a close look to the Intel Modular Servers (You can have prices and presentation on www.thomas-krenn.com).

Hope it helps.

Not with the open source Xen. I don't know about XenServer, though I suspect not.

Live migration only moves the memory and state, not the disk. You should shut it down to move it.

If this question has any relation to your zfs question about l2arc then I just wanted to point out one thing that you can do if you're using zfs on both hosts to minimize downtime in the cases where you have a planned move from one storage server to another.

If each VM has it's own zfs filesystem you can leverage zfs to help speed up planned migrations using snapshots and zfs send and zfs receive. zfs send can efficiently move the file systems or snapshots from one storage server to the other. So as long as you're dealing with a planned migration you can take a snapshot then use zfs send/receive to send this snapshot of the VM's file system to the other storage server. Once you've done that shutdown the VM then run zfs send/receive again this time it will only have to send the changes from the previous snapshot which should be pretty quick. Update your VM's configuration file to point to the new storage location and start it back up. I bet most of that can be automated using a few scripts.