Testing system updates using libvirts checkpoint feature
If you want to test upgrades on virtual machines (running on libvit/qemu/kvm) these are usually the most common steps:
- Clone the virtual machine and test the upgrade.
- Create a snapshot beforehand, do an in-place upgrade and hope everything works. If not, revert the snapshot..
- Combine both methods
- Use LVM or filesystem snapshots (snapper, etc)
As with recent versions, both libvirt and qemu have full support for dirty bitmaps (so called checkpoints). These checkpoints, once existent, will track changes to the block level layer and can be exported via NBD protocol.
Usually one can create these checkpoints using
with a proper xml description.
Using the pull based model, the following is possible:
- Issue an backup-begin statement and freeze the virtual machines file systems during this process (using qemu-agent) so you get an consistent system state.
- Create an qcow2 image with backing image option that points to the created (readonly) NBD export via unix socket as overlay.
- Use the created overlay image to boot up the instance and test the system upgrade.
- Do so while the virtual machine is operating without any downtime.
The overlay image will only use the disk space for the blocks changed during upgrade: no need to create a full clone which may waste a lot of disk space.
In order to simplify the first step, its possible to use virtnbdbackup for creating the required consistent checkpoint and export its data using a unix domain socket.
Update: As alternative, ive just created a small utility called vircpt to create and export checkpoints.
In my example im using a debian11 virtual machine with qemu guest agent configured:
Now let virtnbdbackup create an checkpoint, freeze the filesystems during creation and tell libvirt to provide us with a usable NBD server listening on an unix socket:
We can now use nbdinfo to display some information about the NBD export:
And create a backing image that we can use to test an in-place upgrade:
Now we have various ways for booting the image:
- Create another domain config in libvirt which points to this disk image, boot.
- Extract the current qemu command line for the the running domain using
virsh domxml-to-native qemu-argv --domain debian11_defaultand execute it manually after adjusting required parameters.
- Trust for everything to work out in the simplest way and simply boot up the image via:
After performing the required tests within the virtual machine we can simply kill the active NBD “backup job”:
And remove the created qcow image: