Talos Linux Kubernetes Cluster Provisioning & Lifecycle Guide

KubernetesTalos LinuxUEFI / OVMFRoot Required

This guide explains how pivirtd provisions and manages Talos Linux Kubernetes clusters, how the hypervisor stack is deployed and configured on host nodes, and how to perform cluster lifecycle operations (scaling workers, expanding control-plane HA, and safely removing nodes) using pivirtctl and talosctl.

---

1. Stack Architecture Overview

The pivirtd hypervisor stack operates as a lightweight, zero-dependency QEMU/KVM daemon with embedded network and cloud metadata services:

flowchart TB
    subgraph Host["Host (vm3)"]
        subgraph Daemon["pivirtd"]
            API["Unix socket API / gRPC"]
            DHCP["Embedded DHCP Server (AF_PACKET)"]
            META_SRV["Cloud Metadata Server (:8080)"]
        end

        OVS["Open vSwitch (br-provision)<br/>GW: 10.0.0.1/24 | Meta: 169.254.169.254"]
        Daemon --> OVS
    end

    subgraph Cluster["Talos Cluster"]
        CP["talos-cp-0 (Control Plane)<br/>- UEFI / OVMF Boot<br/>- IP: 10.0.0.10<br/>- Fetches config from 169.254.169.254"]
        WORKERS["talos-worker-2, 3, 4 (Workers)<br/>- UEFI / OVMF Boot<br/>- IPs: 10.0.0.11, .12, .13<br/>- Fetches config from 169.254.169.254"]
    end

    OVS --> CP
    OVS --> WORKERS

Core Stack Components

pivirtd Daemon

Manages QEMU processes, VM lifecycle, networking interfaces, and storage disks under /var/lib/pivirt/.

pivirtctl CLI

Host management CLI that communicates with pivirtd over Unix socket (/var/lib/pivirt/daemon/pivirtd.sock).

Open vSwitch (br-provision)

Provides VM bridge networking with gateway IP 10.0.0.1/24 and metadata routing.

Cloud Metadata Service

Listens on 10.0.0.1:8080. IPTables DNAT redirects http://169.254.169.254:80 requests to 10.0.0.1:8080.

Talos Linux OS

Boots via UEFI (OVMF_CODE.fd), receives kernel parameters via SMBIOS OEM strings (fw_cfg), and auto-configures via metadata.

Zero-Dependency Provisioning

Talos Linux nodes require zero manual intervention. Upon booting UEFI firmware, kernel parameters passed through SMBIOS OEM strings instruct Talos to request its configuration directly from http://169.254.169.254/talos/config.

---

2. Installation of the pivirtd Stack

Choose one of the following deployment options to install or update pivirtd and pivirtctl on your host:

Automated Script (Recommended)

Use the provided automated deployment script to build and deploy to a target host:

1. Run Deployment Script:

   ./scripts/deploy-pivirtd.sh --host vm3.pilab.hu

2. Verify Daemon Service:

   ssh vm3.pilab.hu "sudo systemctl status pivirtd"

Manual Binary Installation

To compile and install the binaries manually:

1. Cross-Compile Binaries for Linux:

   GOOS=linux GOARCH=amd64 go build -o bin/pivirtd ./apps/pivirtd/cmd/pivirtd
   GOOS=linux GOARCH=amd64 go build -o bin/pivirtctl ./apps/pivirtctl/main.go

2. Copy Binaries to Target Host:

   scp bin/pivirtd bin/pivirtctl vm3.pilab.hu:/tmp/
   ssh vm3.pilab.hu "sudo mv /tmp/pivirtd /tmp/pivirtctl /usr/local/bin/ && sudo chmod +x /usr/local/bin/pivirtd /usr/local/bin/pivirtctl"

3. Restart the pivirtd Service:

   ssh vm3.pilab.hu "sudo systemctl restart pivirtd"

Host Permission Notice

Managing hypervisor interfaces, Open vSwitch bridges, and QEMU processes requires elevated sudo privileges on the host node.

---

3. Provisioning a Talos Cluster

Follow these step-by-step instructions to initialize and launch a new Talos Linux VM instance on pivirtd:

1. Generate Talos Machine Configurations

   Generate cluster configurations using talosctl:

   talosctl gen config talos-cluster https://10.0.0.10:6443 --output-dir ./cluster-config

   Configuration Artifacts

   This command produces controlplane.yaml, worker.yaml, and talosconfig.

2. Create VM Directories and System Files

   For each target VM (e.g., talos-worker-2):

   • Create Folder Structure: Create /var/lib/pivirt/vms/talos-worker-2/
   • Store Machine Configuration: Copy worker.yaml to /var/lib/pivirt/vms/talos-worker-2/talos-config
   • Initialize Disk & NVRAM State:

     qemu-img create -f raw /var/lib/pivirt/vms/talos-worker-2/disks/system.raw 10G
     cp /usr/share/OVMF/OVMF_VARS.fd /var/lib/pivirt/vms/talos-worker-2/nvram.fd

3. Define VM Specification (config.yaml)

   Create /var/lib/pivirt/vms/talos-worker-2/config.yaml with SMBIOS OEM strings:

   name: talos-worker-2
   uuid: 04a7851e-04bd-4544-970a-fa2b5198ef60
   sysInfo:
     type: smbios
     oemStrings:
       io.systemd.stub.kernel-cmdline-extra: "console=ttyS0,115200"
   os:
     type: hvm
     machine: q35
     firmware: uefi
     efi:
       loader:
         readonly: true
         type: pflash
         path: /usr/share/OVMF/OVMF_CODE.fd
       nvram:
         path: /var/lib/pivirt/vms/talos-worker-2/nvram.fd
   devices:
     networkinterfaces:
       - type: openvswitch
         source: br-provision
         mac: 52:54:00:ab:cd:0f
         model: virtio
     disks:
       - deviceType: disk
         driver:
           name: qemu
           type: raw
         path: /var/lib/pivirt/vms/talos-worker-2/disks/system.raw
         target:
           bus: virtio
           dev: vda

   Unique Address Constraints

   Always ensure each VM specification has a unique uuid and mac address. Reusing MAC addresses on br-provision leads to ARP collisions and DHCP allocation failures.

4. Start the Virtual Machine via pivirtctl

   Execute the start command:

   sudo pivirtctl vm start talos-worker-2

---

4. Cluster Lifecycle Operations

Manage post-deployment cluster changes such as scaling worker nodes, expanding control plane HA, or performing safe node decommissioning:

Scaling Worker Nodes

1. Create VM Folder: /var/lib/pivirt/vms/talos-worker-5/
2. Stage Configuration: Copy worker.yaml into /var/lib/pivirt/vms/talos-worker-5/talos-config
3. Allocate Storage & NVRAM:

   sudo qemu-img create -f raw /var/lib/pivirt/vms/talos-worker-5/disks/system.raw 10G
   sudo cp /usr/share/OVMF/OVMF_VARS.fd /var/lib/pivirt/vms/talos-worker-5/nvram.fd

4. Generate Spec: Write config.yaml with a unique UUID & MAC address.
5. Boot VM:

   sudo pivirtctl vm start talos-worker-5

6. Verify Automatic Join: The VM boots, acquires an IP via DHCP, retrieves worker.yaml from metadata, and joins Kubernetes automatically.

Adding Control Plane Nodes (HA Expansion)

1. Create Control Plane Folder: /var/lib/pivirt/vms/talos-cp-1/
2. Stage Configuration: Copy controlplane.yaml into /var/lib/pivirt/vms/talos-cp-1/talos-config
3. Prepare Storage & NVRAM:

   sudo qemu-img create -f raw /var/lib/pivirt/vms/talos-cp-1/disks/system.raw 10G
   sudo cp /usr/share/OVMF/OVMF_VARS.fd /var/lib/pivirt/vms/talos-cp-1/nvram.fd

4. Configure Specification: Write config.yaml with SMBIOS OEM strings.
5. Boot Control Plane VM:

   sudo pivirtctl vm start talos-cp-1

6. Verify Node & etcd Integration:

   kubectl get nodes
   talosctl -n 10.0.0.10 etcd members

Quorum Guidance

Maintain an odd number of control plane nodes (e.g. 3 or 5) to guarantee etcd quorum stability.

Removing Nodes

Follow these steps to safely decommission a worker or control plane node:

1. Cordon and Drain Node in Kubernetes:

   kubectl cordon <node-name>
   kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

2. Delete Node Object:

   kubectl delete node <node-name>

3. Reset Talos Node State (Optional):

   talosctl -n <node-ip> reset

4. Stop Hypervisor VM:

   sudo pivirtctl vm stop <vm-name>

5. Delete VM Registration:

   sudo pivirtctl vm delete <vm-name>

Workload Protection

Always drain workloads prior to stopping a node VM to prevent service disruption or ungraceful termination of stateful pods.

---

5. pivirtctl Command Reference

Command	Action / Description
sudo pivirtctl vm list	List all VMs registered on the local hypervisor
sudo pivirtctl vm info <vm>	Display detailed VM runtime status and configuration
sudo pivirtctl vm start <vm>	Start a virtual machine instance
sudo pivirtctl vm stop <vm>	Cleanly shut down a virtual machine via ACPI
sudo pivirtctl vm stop -f <vm>	Force stop (SIGKILL) a virtual machine
sudo pivirtctl vm reboot <vm>	Reboot a virtual machine instance
sudo pivirtctl vm delete <vm>	Remove VM registration from pivirtd

CLI Usage Tip

Use sudo pivirtctl vm info <vm> to inspect assigned MAC addresses, TAP interface bindings, and active QEMU PID details.

---

Related Documentation

pivirtd CLI ReferenceExplore all pivirtctl commands and hypervisor management features.

Troubleshooting Talos Boot LoopResolve kexec and configuration fetch issues on Talos Linux VMs.

Bare Metal InstallationDeploy pivirtd stack directly on bare metal hypervisors.

High Availability GuideArchitect fault-tolerant control plane and storage clusters.