QEMU Refuses to Start — Failed to Get Write Lock

Symptom: pivirtctl vm start <name> returns an error, or the VM state flips from starting → stopped/crashed within seconds. The pivirtd log shows:

ERR failed to start VM  error="qemu exited with status 1"

The QEMU stderr (captured in pivirtd logs or the run dir) contains:

qemu-system-x86_64: Failed to get write lock on /var/lib/pivirt/vms/talos-worker-4/disks/system.raw
Is another process using the image?

Root Cause — Stale QEMU process holding the disk lock

QEMU takes an exclusive write lock on the disk image file when it starts. If the QEMU process is killed unexpectedly (e.g., SIGKILL, OOM, pivirtd crash), the lock is held until the OS fully cleans up the file descriptor. A new QEMU process for the same VM then cannot acquire the lock.

This happened repeatedly during the Talos re-provisioning when VMs were force-stopped to debug networking issues.

Diagnosis

# Find stale QEMU processes for the affected VM
sudo ps aux | grep 'qemu-system' | grep 'talos-worker-4'

# Check if the disk file is locked
sudo fuser /var/lib/pivirt/vms/talos-worker-4/disks/system.raw

# Look at pivirtd logs for the exact error
sudo journalctl -u pivirtd --since '5 minutes ago' | grep 'talos-worker-4'

Fix

Step 1 — Kill the stale QEMU process

# Graceful stop first (may fail if QEMU is wedged)
sudo pivirtctl vm stop talos-worker-4

# If still running after 5s, force-stop via pivirtctl
sudo pivirtctl vm stop -f talos-worker-4

# If pivirtctl cannot reach the process, kill by name directly
sudo pkill -9 -f 'name talos-worker-4'

Step 2 — Verify the lock is released

# Should return nothing if released
sudo fuser /var/lib/pivirt/vms/talos-worker-4/disks/system.raw

Step 3 — Clean up stale run files (optional but recommended)

Stale PID files, QMP sockets, and monitor sockets from the previous run can confuse pivirtd:

UUID="F994C870-722F-44D4-8A37-E3A27FEDFDBF"
RUN_DIR="/var/lib/pivirt/vms/talos-worker-4/run"

# Remove stale PID and socket files
sudo rm -f "${RUN_DIR}/${UUID}-qemu.pid"
sudo rm -f "${RUN_DIR}/${UUID}-qmp.sock"
sudo rm -f "${RUN_DIR}/${UUID}-monitor.sock"
sudo rm -f "${RUN_DIR}/${UUID}-serial.sock"

Step 4 — Start the VM

sudo pivirtctl vm start talos-worker-4

Prevention

Always use pivirtctl vm stop <name> before killing pivirtd or rebooting the host. This allows QEMU to flush the disk and release the lock cleanly.
Never use sudo systemctl restart pivirtd while VMs are running without first stopping all VMs gracefully. pivirtd does not (yet) send shutdown signals to child QEMU processes on exit.

Special Case — Image is corrupted

If the disk image was written to mid-commit when QEMU was killed, Talos installation may be partial. Symptoms: Talos boots but finds no STATE volume, or enters a perpetual install loop.

# Recreate the disk image from scratch (DESTRUCTIVE — wipes all data)
sudo pivirtctl vm stop -f talos-worker-4
sudo rm /var/lib/pivirt/vms/talos-worker-4/disks/system.raw
sudo qemu-img create -f raw /var/lib/pivirt/vms/talos-worker-4/disks/system.raw 10G

# Also reset the NVRAM so UEFI boot order is clean
sudo cp /usr/share/OVMF/OVMF_VARS.fd /var/lib/pivirt/vms/talos-worker-4/nvram.fd

sudo pivirtctl vm start talos-worker-4