Handling XenServer snapshotting and cloning with ZFS SAN

As other answers alluded to, the ideal approach is LUN-per-VDI. At first it didn't look like it was possible to do this, but there is an undocumented "iscsi" SR driver that will create a LUN-per-VDI SR (I found this when looking through the /opt/xensource/sm directory - see the ISCSISR.py file). You essentially setup one SR for each iSCSI target, and XenServer creates the VDIs for each LUN on that target. You can only set this up through the command-line, including creating the VBDs and attaching it to VMs. The VBDs and VDIs don't even show up in XenCenter.

Here's a sample command to set it up:

xe sr-create name-label=san type=iscsi device-config:target=<IP address of target> device-config:targetIQN=<IQN of target> shared=true

This will automatically create an VDI for each iSCSI LUN. If you end up adding a new iSCSI LUN on the SAN, XenServer will add a new VDI for it after executing the following command:

xe sr-scan uuid=<UUID of SR>

This also works when adding VDIs for cloned LUNs - the new VDI gets assigned a new UUID.

Also, if you end up resizing a LUN, XenServer does not automatically pick up on that, so you'd have to execute the following:

xe vdi-forget uuid=<UUID of VDI that you resized>
iscsiadm -m node -R
xe sr-scan uuid=<UUID of SR>

And to create a VBD and attach it to a VM:

xe vbd-create vm-uuid=<UUID of VM> device=<VBD device #, e.g. 0> vdi-uuid=<VDI of UUID>

I've done several configuration with xenserver similar to this setup

And I've used one of two methods

1. If I dont have many vm's I create the system disk with VDI per VM and data disk as direct attach Iscsi lun

2. If I have lots of vm (20+) I create a big SR and when I need to roll back I can

   A. rename the VG before connecting to xenserver (vgrename on a different machine - even virtual )

   B. You can even attach the big SR snapshot to a virtual machine and export it again with iscsi ,

I hope its not too complicated :)

Interesting configuration.. specially for ZFS.

According to my experience with XenServer, the best you can do is let the storage system manage the disk (and it snapshots and other administrative tasks).

I think you should let ZFS handle te snapshots and cloning, but I'm not sure if the best you can do is give to XenServer one big SR (I agree with what you said about roll back). This is a very complex setup, because if you have a lot of VM you'll have a lot of VDI, the administration could be a mess but you will earn the capability to rollback. I'd use one VDI per VM.

Answering your question, if you double check that you detach the SR when rolling back you wont have any problems when attaching a cloned SR. I've done that before, not in XenServer but in XCP (http://goo.gl/4wfE)

Our customers (Nexenta customers) who do not have very many VMs, fewer than 1000 mostly, choose to do a LUN per VM at times. This allows you to snapshot easily from NexentaStor of each individual VM. What's cool is that you can create one golden image, aka clome-master. Then use that clone-master to spin up new VMs, by simply cloning it. And the added benefit is that you are not using any additional disk space.

This seems relevant. The bash script below has functions for attaching and detaching a SR as well as renaming all the UUIDs in an existing (for example SAN cloned) SR. It could be used to rename the UUIDs of an old clone and attach it without conflicting with the latest version of the volume being attached.

    # returns scsi_id, scsi_lun_id and as a bonus host_uuid
    probe_sr(){
            xen_host_ip=$1
            iscsi_device_ip=$2
            iscsi_device_target_iqn=$3
            iscsi_vol_name=$4

            cmd="ssh root@$xen_host_ip xe sr-probe type=lvmoiscsi device-config:target=$iscsi_device_ip device-config:targetIQN=$iscsi_device_target_iqn:$iscsi_vol_name"
            echo $cmd
            probe_out=`$cmd 2>&1`
            #echo probe_out=$probe_out

            scsi_id=`echo $probe_out | awk '{for(i=1;i<=NF;i++){if($i=="<SCSIid>"){i+=1;o=o","$i}}}END{print substr(o,2)}'`
            echo scsi_id=$scsi_id
            if [ -z $scsi_id ];then
                    exit
            fi
            # Only allow one LUN per target. To allow for more, enhance input to specify volume serial number and search probe output for it.
            if [ `echo $scsi_id | awk -F, '{print NF}'` != 1 ];then
                    echo Only one LUN per iscsi target is supported
                    exit
            fi

            scsi_lun_id=`echo $probe_out | awk '{for(i=1;i<=NF;i++){if($i=="<LUNid>"){i+=1;o=o" "$i}}}END{print substr(o,2)}'`
            echo scsi_lun_id=$scsi_lun_id
            if [ -z $scsi_lun_id ];then
                    exit
            fi

            # This only works if it is a standalone XenServer, not in a Xen pool - so this becomes a requirement
            host_uuid=`ssh root@$xen_host_ip xe host-list --minimal`
            if [ -z $host_uuid ];then
                    echo host_uuid=$host_uuid
                    exit
            fi
    }


    # Create SR wipes out contents of previous SR
    create_sr(){
            xen_host_ip=$1
            iscsi_device_ip=$2
            iscsi_device_target_iqn=$3
            iscsi_vol_name=$4
            sr_name=$5

            probe_sr $xen_host_ip $iscsi_device_ip $iscsi_device_target_iqn $iscsi_vol_name

            cmd="ssh root@$xen_host_ip xe sr-create host-uuid=$host_uuid content-type=user name-label=$sr_name shared=true device-config:target=$iscsi_device_ip device-config:targetIQN=$iscsi_device_target_iqn:$iscsi_vol_name device-config:SCSIid=$scsi_id type=lvmoiscsi"
            echo $cmd
            $cmd
    }

    # mainly returns sr_uuid, but also pv_device used in uuid_regen_sr
    get_sr_uuid_from_file_system(){
            xen_host_ip=$1
            iscsi_device_ip=$2
            iscsi_device_target_iqn=$3
            iscsi_vol_name=$4
            scsi_lun_id=$5

            # Ensure the the VG for the new volume gets created
            ##iscsiadm -m discovery -t sendtargets -p $iscsi_device_ip
            echo ssh root@$xen_host_ip iscsiadm -m node -T $iscsi_device_target_iqn:$iscsi_vol_name -p $iscsi_device_ip:3260 --login
            ssh root@$xen_host_ip iscsiadm -m node -T $iscsi_device_target_iqn:$iscsi_vol_name -p $iscsi_device_ip:3260 --login

            # get the pv_device name
            # Some iSCSI SAN might need customization here
            #scsi_id_short=`echo $scsi_id | awk '{print substr($1,1,length("36589cfc000000"))}'`
            scsi_id_short=$scsi_id
            echo scsi_id_short=$scsi_id_short
            unset pv_device_short
            tries=1
            while [ $tries -le 3 ]&&[ -z $pv_device_short ];do
                    echo "ssh root@$xen_host_ip ls -l /dev/disk/by-scsibus/ | egrep \"$scsi_id_short.*-[0-9]+:[0-9]+:[0-9]+:$scsi_lun_id -> \" | awk '{print \$NF}' | awk -F/ '{print \$NF}'"
                    pv_device_short=`ssh root@$xen_host_ip ls -l /dev/disk/by-scsibus/ | egrep "$scsi_id_short.*-[0-9]+:[0-9]+:[0-9]+:$scsi_lun_id -> " | awk '{print $NF}' | awk -F/ '{print $NF}'`
                    pv_device=/dev/$pv_device_short
                    echo pv_device=$pv_device
                    if [ -z "$pv_device_short" ];then
                            sleep 1
                    fi
                    tries=$(( tries + 1 ))
            done
            if [ -z "$pv_device_short" ];then
                    exit
            fi
            if [ `echo $pv_device | wc -l` -gt 1 ];then
                    exit
            fi

            # pvscan ensure PVs, VGs and LVMs are visible
            echo ssh root@$xen_host_ip pvscan
            ssh root@$xen_host_ip pvscan 2>/dev/null

            # get the VG UUID which is also what Xen uses for the SR UUID
            sr_uuid=`ssh root@$xen_host_ip "pvs 2>/dev/null" | egrep "$pv_device.*VG_XenStorage-" | awk '{print substr($2,length("VG_XenStorage-")+1)}'`
            echo sr_uuid=$sr_uuid
            if [ -z "$sr_uuid" ];then
                    exit
            fi

            # Thought this might be good, but think it caused problems.  Better and seemingly required place is in detach_sr
            #echo ssh root@$xen_host_ip iscsiadm -m node -T $iscsi_device_target_iqn:$iscsi_vol_name -p $iscsi_device_ip:3260 --logout
            #ssh root@$xen_host_ip iscsiadm -m node -T $iscsi_device_target_iqn:$iscsi_vol_name -p $iscsi_device_ip:3260 --logout
    }

    attach_sr_xen(){
            xen_host_ip=$1
            iscsi_device_ip=$2
            iscsi_device_target_iqn=$3
            iscsi_vol_name=$4
            sr_name=$5

            # Attach volume as a Xen SR
            echo ssh root@$xen_host_ip "xe sr-introduce uuid=$sr_uuid name-label=\"$sr_name\" type=lvmoiscsi shared=true"
            ssh root@$xen_host_ip "xe sr-introduce uuid=$sr_uuid name-label=\"$sr_name\" type=lvmoiscsi shared=true"

            echo ssh root@$xen_host_ip "xe pbd-create sr-uuid=$sr_uuid host-uuid=$host_uuid type=lvmoiscsi device-config-target=$iscsi_device_ip device-config-targetIQN=$iscsi_device_target_iqn:$iscsi_vol_name device-config-SCSIid=$scsi_id"
            pbd_uuid=`ssh root@$xen_host_ip "xe pbd-create sr-uuid=$sr_uuid host-uuid=$host_uuid type=lvmoiscsi device-config-target=$iscsi_device_ip device-config-targetIQN=$iscsi_device_target_iqn:$iscsi_vol_name device-config-SCSIid=$scsi_id"`
            echo pbd_uuid=$pbd_uuid
            if [ -z $pbd_uuid ];then
                    exit
            fi

            echo ssh root@$xen_host_ip "xe pbd-plug uuid=$pbd_uuid"
            ssh root@$xen_host_ip "xe pbd-plug uuid=$pbd_uuid"
    }

    # Attach SR retains prior contents as opposed to create_sr
    attach_sr(){
            xen_host_ip=$1
            iscsi_device_ip=$2
            iscsi_device_target_iqn=$3
            iscsi_vol_name=$4
            sr_name=$5

            probe_sr $xen_host_ip $iscsi_device_ip $iscsi_device_target_iqn $iscsi_vol_name
            get_sr_uuid_from_file_system $xen_host_ip $iscsi_device_ip $iscsi_device_target_iqn $iscsi_vol_name $scsi_lun_id
            attach_sr_xen $xen_host_ip $iscsi_device_ip $iscsi_device_target_iqn $iscsi_vol_name $sr_name
    }

    # Detach/forget SR
    detach_sr(){
            xen_host_ip=$1
            iscsi_device_ip=$2
            iscsi_device_target_iqn=$3
            iscsi_vol_name=$4
            sr_name=$5

            echo ssh root@$xen_host_ip "xe sr-list name-label=\"$sr_name\" --minimal"
            sr_uuid=`ssh root@$xen_host_ip "xe sr-list name-label=\"$sr_name\" --minimal"`
            echo sr_uuid=$sr_uuid
            if [ -z $sr_uuid  ];then
                    exit
            fi
            echo ssh root@$xen_host_ip "xe pbd-list sr-uuid=$sr_uuid --minimal"
            pbd_uuid=`ssh root@$xen_host_ip "xe pbd-list sr-uuid=$sr_uuid --minimal"`
            echo pbd_uuid=$pbd_uuid
            if [ -z $pbd_uuid ];then
                    echo No PBD found, proceeding anyway
            fi
            echo ssh root@$xen_host_ip iscsiadm -m node -T $iscsi_device_target_iqn:$iscsi_vol_name -p $iscsi_device_ip:3260 --logout
            ssh root@$xen_host_ip iscsiadm -m node -T $iscsi_device_target_iqn:$iscsi_vol_name -p $iscsi_device_ip:3260 --logout
            if [ ! -z $pbd_uuid ];then
                    echo ssh root@$xen_host_ip "xe pbd-unplug uuid=$pbd_uuid"
                    ssh root@$xen_host_ip "xe pbd-unplug uuid=$pbd_uuid"
            fi
            echo ssh root@$xen_host_ip "xe sr-forget uuid=$sr_uuid"
            ssh root@$xen_host_ip "xe sr-forget uuid=$sr_uuid"

            echo "It is now ok to remove the volume from the storage array"

    }

    # generates new PV, SR (VG) and VDI (LVM) uuids for a cloned and unattached iSCSI SR, leaving it unattached
    uuid_regen_sr(){
            xen_host_ip=$1
            iscsi_device_ip=$2
            iscsi_device_target_iqn=$3
            iscsi_vol_name=$4
            sr_name=$5

            probe_sr $xen_host_ip $iscsi_device_ip $iscsi_device_target_iqn $iscsi_vol_name

            get_sr_uuid_from_file_system $xen_host_ip $iscsi_device_ip $iscsi_device_target_iqn $iscsi_vol_name $scsi_lun_id

            # Import the clone giving it a new PV ID and VG UUID
            new_sr_uuid=`uuidgen`
            echo ssh root@$xen_host_ip vgimportclone --basevgname VG_XenStorage-$new_sr_uuid --import $pv_device
            ssh root@$xen_host_ip vgimportclone --basevgname VG_XenStorage-$new_sr_uuid --import $pv_device

            # get the new VG UUID which is also what Xen uses for the SR UUID
            sr_uuid=`ssh root@$xen_host_ip "pvs 2>/dev/null" | egrep "$pv_device.*VG_XenStorage-" | awk '{print substr($2,length("VG_XenStorage-")+1)}'`
            echo sr_uuid=$sr_uuid
            if [ -z $sr_uuid ]||[ $sr_uuid != $new_sr_uuid ];then
                    echo bad sr_uuid
                    exit
            fi

            attach_sr_xen $xen_host_ip $iscsi_device_ip $iscsi_device_target_iqn $iscsi_vol_name $sr_name

            # pvscan to get rid of some Linux duplicate dev errors
            echo ssh root@$xen_host_ip pvscan
            ssh root@$xen_host_ip pvscan 2>/dev/null

            echo -n waiting for VDI count to settle :
            prev_num_vdis=`ssh root@$xen_host_ip "xe vdi-list sr-uuid=$sr_uuid" | grep name-label | wc -l`
            echo -n $prev_num_vdis" "
            sleep 5
            num_vdis=`ssh root@$xen_host_ip "xe vdi-list sr-uuid=$sr_uuid" | grep name-label | wc -l`
            echo -n $num_vdis" "
            while [ $num_vdis != $prev_num_vdis ];do
                    sleep 5
                    num_vdis=`ssh root@$xen_host_ip "xe vdi-list sr-uuid=$sr_uuid" | grep name-label | wc -l`
                    echo -n $num_vdis" "
                    prev_num_vdis=$num_vdis
            done
            echo ""

            metadata_sr_uuid=`echo $sr_uuid | awk '{for(i=1;i<=length($1);i++){c=substr($1,i,1);if(c=="-"){o=o"--"}else{o=o""c}};print o}'`
            echo ssh root@$xen_host_ip "cp VG_XenStorage--$metadata_sr_uuid""-MGT ."
            ssh root@$xen_host_ip "cp /dev/mapper/VG_XenStorage--$metadata_sr_uuid""-MGT ."

            for vdi_uuid in `ssh root@$xen_host_ip lvs VG_XenStorage-$sr_uuid | grep VHD\- | awk '{print substr($1,length("VHD-")+1)}'`;do
                    new_uuid=`uuidgen`
                    echo ssh root@$xen_host_ip "sed -i \"s/$vdi_uuid/$new_uuid/g\" VG_XenStorage--$metadata_sr_uuid""-MGT"
                    ssh root@$xen_host_ip "sed -i \"s/$vdi_uuid/$new_uuid/g\" VG_XenStorage--$metadata_sr_uuid""-MGT"

                    echo ssh root@$xen_host_ip "lvrename /dev/VG_XenStorage-$sr_uuid/VHD-$vdi_uuid /dev/VG_XenStorage-$sr_uuid/VHD-$new_uuid"
                    ssh root@$xen_host_ip "lvrename /dev/VG_XenStorage-$sr_uuid/VHD-$vdi_uuid /dev/VG_XenStorage-$sr_uuid/VHD-$new_uuid"
            done

            echo ssh root@$xen_host_ip "cp VG_XenStorage--$metadata_sr_uuid""-MGT /dev/mapper/VG_XenStorage--$metadata_sr_uuid""-MGT"
            ssh root@$xen_host_ip "cp VG_XenStorage--$metadata_sr_uuid""-MGT /dev/mapper/VG_XenStorage--$metadata_sr_uuid""-MGT"

            echo ssh root@$xen_host_ip "rm -f VG_XenStorage--$metadata_sr_uuid""-MGT"
            ssh root@$xen_host_ip "rm -f VG_XenStorage--$metadata_sr_uuid""-MGT"

            echo ssh root@$xen_host_ip "xe sr-scan uuid=$sr_uuid"
            ssh root@$xen_host_ip "xe sr-scan uuid=$sr_uuid"

            detach_sr $xen_host_ip $iscsi_device_ip $iscsi_device_target_iqn $iscsi_vol_name $sr_name
}