Mastodon Follow

VM-Operator Logo

View GitHub Project


The Runner

The Manager

Web interface



The Runner

For most use cases, Qemu needs to be started and controlled by another program that manages the Qemu process. This program is called the runner in this context.

The most prominent reason for this second program is that it allows a VM to be shutdown cleanly in response to a TERM signal. Qemu handles the TERM signal by flushing all buffers and stopping, leaving the disks in a crash consistent state. For a graceful shutdown, a parent process must handle the TERM signal, send the system_powerdown command to the qemu process and wait for its completion.

Another reason for having the runner is that another process needs to be started before qemu if the VM is supposed to include a TPM (software TPM).

Finally, we want some kind of higher level interface for applying runtime changes to the VM such as changing the CD or configuring the number of CPUs and the memory.

The runner takes care of all these issues. Although it is intended to run in a container (which runs in a Kubernetes pod) it does not require a container. You can start and use it as an ordinary program on any system, provided that you have the required commands (qemu, swtpm) installed.

Stand-alone Configuration

Upon startup, the runner reads its main configuration file which defaults to /etc/opt/vmrunner/config.yaml and may be changed using the -c (or --config) command line option.

A sample configuration file with annotated options can be found here. As the runner implementation uses the JGrapes framework, the file follows the framework’s conventions. The top level “/Runner” selects the component to be configured. Nested within is the information to be applied to the component.

The main entries in the configuration file are the “template” and the “vm” information. The runner processes the freemarker template, using the “vm” information to derive the qemu command. The idea is that the “vm” section provides high level information such as the boot mode, the number of CPUs, the RAM size and the disks. The template defines a particular VM type, i.e. it contains the “nasty details” that do not need to be modified for some given set of VM instances.

The templates provided with the runner can be found here. When details of the VM configuration need modification, a new VM type (i.e. a new template) has to be defined. Authoring a new template requires some knowledge about the qemu invocation. Despite many “warnings” that you find in the web, configuring the invocation arguments of qemu is only a bit (but not much) more challenging than editing libvirt’s XML.

Running in a Pod

The real purpose of the runner is to run a VM on Kubernetes in a pod. When running in a Kubernetes pod, /etc/opt/vmrunner/config.yaml should be provided by a ConfigMap.

If additional templates are required, some ReadOnlyMany PV should be mounted in /opt/vmrunner/templates. The PV should contain copies of the standard templates as well as the additional templates. Of course, a ConfigMap can be used for this purpose again.

Networking options are rather limited. The assumption is that in general the VM wants full network connectivity. To achieve this, the pod must run with host networking and the host’s networking must provide a bridge that the VM can attach to. The only currently supported alternative is the less performant “user networking”, which may be used in a stand-alone development configuration.

Runtime changes

The runner supports adaption to changes of the RAM size (using the balloon device) and to changes of the number of CPUs. Note that in order to get new CPUs online on Linux guests, you need a udev rule which is not installed by default1.

The runner also changes the images loaded in CDROM drives. If the drive is locked, i.e. if it doesn’t respond to the “open tray” command the change will be suspended until the VM opens the tray.

Finally, powerdownTimeout can be changed while the qemu process runs.

Testing with Helm

There is a Helm Chart for testing the runner.

  1. The simplest form of the rule is probably:

    ACTION=="add", SUBSYSTEM=="cpu", ATTR{online}="1"