How to port a Xen VM to KVM
I have a number of existing virtual machines running under Xen. I use them to separate different bits of functionality and allow easy migration to new hardware. I used to spend days moving lots of different services to a new machine when the time arrived, but now I simply migrate the VM, which takes no longer than is required to copy the disk devices.
One point has niggled for a while though. A couple of my services require access to USB devices, and support for forwarding real USB devices through to VMs in Xen is not good. It theoretically exists, but the old management tool which supports it - xm - is now deprecated and doesn't seem to work with Xen 4.3. The replacement tool - xl - does not support it. I've tried forwarding the whole PCI controller, but the guest OS has difficulty resetting it and so can't use it.
I therefore set out to see if any of the alternative virtualisation offerings provided better support, and it seems that KVM does. I've tried it and it works well, at least for my applications. Creating new VMs under KVM is easy - much easier than under Xen - but that left me with my existing few that I really wanted to move across. You can't run both Xen and KVM on the same box, so if one guest was to use KVM, they would all have to. Fortunately I'm wanting to move them all to a new physical system, so the conversion would fit in neatly at the same time. Surely it can't be that difficult?
All the guest systems are running Debian Wheezy, and the new host is running Debian Jessie. There are lots of postings about this on the 'net, but none of them provided quite enough information to complete the process. Hence this record.
Each of my existing VMs uses two LVM logical volumes, created on the host - one for its system disk and one for swap. The boot process under Xen uses a kernel outside the VM, so although each of the VMs does have a kernel installed, they don't have grub or anything like that inside them.
For the purposes of this description, I'll focus on moving a single VM called "xronos". Its original disks are:
- xronos-disk
- xronos-swap
Each of them appears as single un-partitioned device in the guest OS, but I could really do with something to partition and on which I can install grub in the MBR. Hence I introduce a new logical volume to provide /boot, and that one I will partition with one large partition. xronos-disk gets renamed as xronos-root, so we end up with:
- xronos-boot
- xronos-root
- xronos-swap
Note that KVM expects to boot from the first device, so it's important to list xronos-boot first in its VM definition.
In overview, all we need to do is move /boot onto the new partition, install grub, redirect /etc/fstab, fix up the console (which is slightly different under Xen) and you're there. There are quite a few steps though. All of the following should be done as root, and we assume that the LVM Volume Group which you're using is called vg0.
Firstly we set up the disk structure and mount all the necessary bits. Note that the kpartx step seems to complete asynchronously. If you try to script this then the immediately following mke2fs command will fail unless you put in a small delay between the two.
At this point you should have your two target partitions mounted and ready to receive your installation of grub. To achieve this you need to create some device files in /dev of the target system, which exactly mirror how the disks are already mounted. You need them for the root disk, the boot disk, and the partition within the boot disk. If you examine /dev/mapper on the host system, you will find that each of the relevant devices there is a symlink to a file called dm-NN in the parent directory. E.g.
Firstly we set up the disk structure and mount all the necessary bits. Note that the kpartx step seems to complete asynchronously. If you try to script this then the immediately following mke2fs command will fail unless you put in a small delay between the two.
lvrename vg0 xronos-disk xronos-root
lvcreate --size 128m --name xronos-boot
fdisk /dev/mapper/vg0-xronos--boot (to create one large bootable partition)
kpartx -a -v /dev/mapper/vg0-xronos--boot
mke2fs -t ext4 /dev/mapper/vg0-xronos--boot1
mkdir /mnt/target-xronos
mount /dev/mapper/vg0-xronos--root /mnt/target-xronos
mount /dev/mapper/vg0-xronos--boot1 /mnt/target/xronos/mnt
cd /mnt/target-xronos
mv boot/* mnt
umount /mnt/target-xronos/mnt
mount /dev/mapper/vg0-xronos-boot1 /mnt/target/xronos/boot
At this point you should have your two target partitions mounted and ready to receive your installation of grub. To achieve this you need to create some device files in /dev of the target system, which exactly mirror how the disks are already mounted. You need them for the root disk, the boot disk, and the partition within the boot disk. If you examine /dev/mapper on the host system, you will find that each of the relevant devices there is a symlink to a file called dm-NN in the parent directory. E.g.
ls -l /dev/mapper/vg0-xronos* lrwxrwxrwx 1 root root 8 Apr 14 08:23 vg0-xronos--boot -> ../dm-19
lrwxrwxrwx 1 root root 8 Apr 14 08:24 vg0-xronos--boot1 -> ../dm-20 lrwxrwxrwx 1 root root 8 Apr 14 08:23 vg0-xronos--root -> ../dm-12 lrwxrwxrwx 1 root root 8 Apr 14 08:23 vg0-xronos--swap -> ../dm-11
We need files with exactly the same name in the guest systems /dev directory, but they can't be symbolic links to the existing device files because those won't be available once we execute chroot. Instead, we copy the existing device files.
cp -av /dev/dm-19 /mnt/target-xronos/dev/vg0-xronos--boot cp -av /dev/dm-20 /mnt/target-xronos/dev/vg0-xronos--boot1 cp -av /dev/dm-12 /mnt/target-xronos/dev/vg0-xonos--root
And we should have the necessary files.
