Events and handlers

Every lab daemon emits structured events as things happen: a machine starting, becoming ready, stopping or crashing; a lab coming up or down; a snapshot taken; a playbook converging; the disk running low. Any unrecoverable error is an event before it is a failure. Each event goes three places at once: the daemon's live broadcast stream, which CLI subscribers and the supervisor's host-wide aggregate read; the lab's append-only event history; and the tracing log. An on {} block in the lab file binds an event to a wscript handler. This chapter explains the stream, the handlers, the watchdogs and the logs. The full event list with payloads is in Events.

The event stream

An event is a name, the lab it came from, a timestamp and a JSON payload. Machine events carry the machine's name under vm, or under container for a container. The names a handler may bind are a closed list, and `vmlab validate rejects an on {}` that names anything else:

Other events reach the stream and the history but cannot be bound: the workspace syncer's workspace.halted, workspace.synced, workspace.rescan, workspace.deferred, workspace.skipped, workspace.refused, workspace.volume and their siblings, the per-step playbook.op.* progress, and smb.started and smb.failed from the shared-folder service. They exist so that a refusal or a skip is visible somewhere other than one developer's terminal, and vmlab logs shows them.

Handlers

An on "<event>" { run = "<script.ws>" } block at lab level binds the event to a handler script. The script's entry point is fn handle(event: Event, lab: Lab): event.name is the event, event.vm is the machine's name for machine events and empty otherwise, and event.data is the JSON payload as text. lab is the same handle a provision gets, so a handler can screenshot the machine that crashed, copy files off it, or start it again; see Guest automation with wscript.

lab "ad-lab" {
  // …
  on "vm.crashed"    { run = "scripts/collect-dumps.ws" }
  on "host.disk_low" { run = "scripts/alert.ws" }
}

use vmlab

fn handle(event: Event, lab: Lab) {
    lab.log("crash handler fired for " + event.vm + " (" + event.name + ")")
    let Ok(vm) = lab.vm(event.vm) else { return }
    match vm.screenshot("") {
        Ok(path) => lab.log("saved crash screenshot: " + path),
        Err(e) => lab.log("could not screenshot: " + e),
    }
}

targets = ["dc01", "web"] narrows a handler to named machines; with no targets it handles every occurrence. Only machine-scoped events accept targets, the vm., container. and snapshot. families, and validate rejects targets on a lab-wide event such as host.disk_low. The handler's path is relative to the lab root, must exist, and is compiled by validate along with every other script.

The lab daemon subscribes to its own stream and, for each event, runs every matching handler as its own task. Handlers run concurrently with each other and with whatever caused the event, and shutdown waits a bounded time for in-flight handlers rather than killing them. **Handler failures are logged, never fatal**: a compile error, a runtime error or an unreadable script is a warning in the daemon log and nothing else stops. There is no restart policy in the daemon; a handler that wants one calls machine.start() itself, which is what the crash handler in examples/mixed-lab demonstrates for a container.

Watchdogs

Two free-space watchdogs run, both governed by disk_low_percent in the host configuration, default 10 percent, and both checking once a minute. The supervisor watches the filesystem holding the template store, since pulls and builds land there. Each lab daemon watches the filesystem holding its own .vmlab/ directory, since linked clones grow with use. Each emits host.disk_low with the path and the free percentage when free space drops below the threshold, once, and re-arms only when space recovers above it, so a full disk does not flood the stream. The lab daemon's copy is the one an on "host.disk_low" handler sees.

The supervisor is also the watchdog over lab daemons. It reaps a daemon on a full down, vmlab lab stop or destroy, and if one dies unexpectedly it emits lab.daemon_crashed, marks the lab failed, and does not restart it; see How vmlab runs a lab.

Logs

Everything is logged as JSON lines under ~/.local/state/vmlab/labs/<lab>/: the event history in events.jsonl, the lab log in lab.log with provision output, and per machine a serial log, QEMU's stdout and stderr, and swtpm's log for a VM, or the console log for a container. The files vmlab appends to itself roll over at a fixed size, keeping one previous generation as <name>.1. Provision output is also streamed live to the CLI that ran up.

vmlab logs [lab/][machine] dumps or follows them. With no target it takes the current directory's lab, with a machine it narrows to that machine's files, -f keeps following and picks up machines that start later, -n sets the history shown, and --output jsonl emits the raw lines instead of the pretty rendering. Event lines are rendered as a timestamp plus a flattened event key=value … summary; every other stream passes through verbatim. See vmlab logs. Two related verbs read inside the guest rather than about it: vmlab tail follows a file in the guest over the agent, and vmlab eventlog follows a Windows guest's event log.