The libvirt and qemu versions in Debian Bullseye support a new feature that allows for easier backup and recovery of virtual machines. Instead of using snapshots for backup operation, its now possible to enable dirty bitmaps. Other hypervisors tend to call this “changed block tracking”.

Using the new backup begin approach, its not only possible to create live full backups (without having to create an snapshot) but also track the changes between so called checkpoints, which is very useful for incremental backups.

Over the course of the last few months, i have been working on a simple backup and recovery utility called virtnbdbackup

It uses the pull based approach in the libvirt api set and currently supports:

• Thin provisioned full and incremental backups.
• Compression (lz4).
• Freezing the virtual machine guest fs via qemu-agent (if installed)
• Multithreaded backup, if multiple disks are attached.
• Point in time recovery to a given incremental backup.
• Includes nbdkit plugin and utility that allows to map the thin provisioned backup images to a block device, for single file or instant recovery (boot the virtual machine directly from the backup image)

Preparing the virtual machines

The dirty bitmap feature is not enabled by default, users can enable it by adding a new capability to a virtual machine configuration:

 <domain type='kvm' id='1' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
 [..]
 <qemu:capabilities>
   <qemu:add capability='incremental-backup'/>
 </qemu:capabilities>
 [..]
 </domain>

To finally enable the feature, power cycle virtual machine once.

Creating backups

By default, virtnbdbackup saves the virtual machine config, disks and its logfiles to a given target directory. Its also possible to stream the output into a uncompressed zip archive

Taking a backup is as simple as:

# virtnbdbackup -d vm2 -l full -o /tmp/WEEKLY_BACKUP/
 [..]
[2021-11-08 15:43:46] INFO virtnbdbackup - main [MainThread]: Backup jobs finished, stopping backup task.
[2021-11-08 15:43:46] INFO virtnbdbackup - main [MainThread]: Finished

# tree /tmp/WEEKLY_BACKUP/
 /tmp/WEEKLY_BACKUP/
├── backup.full.11082021154343.log
├── checkpoints
│   └── virtnbdbackup.0.xml
├── sda.full.data
├── vm2.cpt
└── vmconfig.virtnbdbackup.0.xml

From that point on, its now possible to create incremental backups:

 # virtnbdbackup -d vm2 -l inc -o /tmp/WEEKLY_BACKUP/

 # tree /tmp/WEEKLY_BACKUP/
 /tmp/WEEKLY_BACKUP/
├── backup.full.11082021154343.log
├── backup.inc.11082021154538.log
├── checkpoints
│   ├── virtnbdbackup.0.xml
│   └── virtnbdbackup.1.xml
├── sda.full.data
├── sda.inc.virtnbdbackup.1.data
├── vm2.cpt
├── vmconfig.virtnbdbackup.0.xml
└── vmconfig.virtnbdbackup.1.xml

Restoring backups

The virtnbdrestore utility can be used to reconstruct the backup sets into usable qcow images, like so:

 # virtnbdrestore -a restore -i /tmp/WEEKLY_BACKUP/ -o /tmp/VM_RESTORE

Using the --until option its also possible to only reconstruct the images to a certain checkpoint, allowing for point in time recovery.

Restoring single files

Via virtnbdmap you can map full backups back into an usable block device, without having to reconstruct the complete backup image:

# virtnbdmap -f /tmp/WEEKLY_BACKUP/sda.full.data
 [..] INFO virtnbdmap - <module> [MainThread]: Done mapping backup image to [/dev/nbd0]
 [..] INFO virtnbdmap - <module> [MainThread]: Press CTRL+C to disconnect
# fdisk -l /dev/nbd0
Disk /dev/nbd0: 2 GiB, 2147483648 bytes, 4194304 sectors

From here, you can either mount the disc and recover single files, or boot from it via:

qemu-img create -b /dev/nbd0 -f qcow2 bootme.qcow2
qemu-system-x86_64 -enable-kvm -m 2000 -hda bootme.qcow2

Check out the README for the full feature set.