System Architecture

System DesignConnectRPC / mTLSMicroservices

PiVirt follows a decoupled three-tier architecture designed as an open, cloud-native successor to legacy enterprise virtualization (such as VMware vSphere). Communication across tiers relies on ConnectRPC / gRPC with mTLS authentication, Unix domain sockets, and declarative GitOps manifests.

The Three-Tier Architecture Model

PiVirt establishes a clear separation of concerns across the virtualization lifecycle:

Tier	Component	Open / Free	VMware Analogue	Kubernetes Analogue	Responsibility
Host	`pivirtd`	Open Source	`vmkernel`	CRI / `containerd`	Low-level executor daemon. Manages QEMU/KVM processes, QMP sockets, veth pairs, and local disk volumes. Zero cluster awareness.
Node	`agent`	Free (≤ 4 nodes)	`hostd` / Host Client	`kubelet`	Node supervisor & mesh endpoint. Standalone mode (local auth, DCUI, max 4 nodes) or managed cluster worker streaming telemetry & executing tasks.
Cluster	`director`	Enterprise / Paid	`vCenter`	`kube-apiserver`	Cluster control plane. Multi-node DRS scheduling, cluster HA, Distributed Virtual Switches (DVS), cross-host IPAM, and multi-tenancy.

Why Are Host (`pivirtd`) and Node (`agent`) Separate Processes?

Blast Radius & Zero Workload Disruption:
- pivirtd talks directly to the Linux kernel, KVM, QEMU, and local disk images via /var/run/pivirt/pivirtd.sock.
- If the agent crashes, experiences memory pressure, rotates mTLS certificates, or undergoes an in-place software upgrade, running guest VMs never experience downtime.
Cluster Agnosticism:
- pivirtd contains zero knowledge of tenants, user organizations, cluster placement policies, or WAN networking. It remains small, fast, and secure.
Standalone vs. Managed Modes:
- The agent can run completely standalone without a Director (for homelabs and edge setups) with local authentication (identity.yaml) and the Direct Console UI (virt-appliance).
- When adopted by a Director (AdoptToCloud), the agent switches from standalone authority to an enrolled cluster worker.

High-Level Architecture

graph TB
    subgraph "User & Management Interfaces"
        REACT_UI["Director React Web UI<br/>(Admin / Self-Service Console)"]
        PVCLI["pvcli / pivirtctl<br/>(Operator CLIs)"]
        DCUI["appliance / DCUI<br/>(Host Console TUI / Webterm)"]
    end

    subgraph "Cluster Control Plane — Paid / Enterprise"
        DIRECTOR["director (vCenter Analogue)<br/>DRS Scheduler • DVS • IPAM • Multi-Tenancy"]
        DEPLOYER["deployer<br/>(Bare-Metal Provisioning)"]
        STATE_DB[("Cluster State DB")]
        DIRECTOR <--> STATE_DB
    end

    subgraph "Node Tier — Free (≤ 4 Nodes) / Standalone or Managed"
        AGENT["agent (hostd Analogue)<br/>Node Supervisor • Datastores • Local Identity • TaskExecutor"]
        ISCSIMAN["iscsiman<br/>iSCSI & OCFS2 Cluster Storage"]
    end

    subgraph "Host Tier — Free / Bare-Metal Executor"
        PIVIRTD["pivirtd (vmkernel Analogue)<br/>QEMU/KVM • AF_PACKET DHCP • Metadata Service"]
        HYPERVISOR["Linux KVM / QEMU Hypervisor"]
        PIVIRTD --> HYPERVISOR
    end

    REACT_UI -->|ConnectRPC / WS| DIRECTOR
    PVCLI -->|ConnectRPC| DIRECTOR
    PVCLI -->|ConnectRPC| AGENT
    PVCLI -->|Unix Socket gRPC| PIVIRTD
    DCUI -->|PAM / Local APIs| AGENT

    DIRECTOR -->|mTLS NodeSession Stream| AGENT
    DEPLOYER -->|SSH / ConnectRPC| AGENT

    AGENT -->|Unix Socket gRPC| PIVIRTD
    AGENT -->|gRPC| ISCSIMAN
    ISCSIMAN -->|iSCSI / OCFS2| STORAGE[(Shared Storage Pool)]

Component Responsibilities

director — Cluster Control Plane

Centralized cluster controller, scheduler, and React Web Console. Coordinates multi-node placement, live migrations, and distributed tasks.

appliance & DCUI — Host Console

Direct Console User Interface (Bubble Tea TUI) & Webterm gateway. Host-level networking, PAM authentication, hardware diagnostics, and status.

agent — Node Agent & Mesh

Node service mesh endpoint exposing ConnectRPC APIs. Handles mTLS via PiTrust PKI, service discovery, power schedules, and health checks.

pivirtd — Hypervisor Daemon

Direct QEMU/KVM management via go-qemu (QMP). Manages VM lifecycle, snapshots, live migration, console, and metrics exposition.

pvcli — Command Line Interface

User-facing CLI for all operations. Connects to director or agent via ConnectRPC with shell tab-completion and context switching.

iscsiman — Storage Manager

iSCSI target/initiator & OCFS2 cluster storage manager. Handles volume provisioning, multipath failover, and CSI driver compatibility.

deployer — Deployment Automation

Infrastructure provisioning tool. Deploys clusters on Bare Metal (IPMI/Redfish), VMware, Proxmox, and cloud providers via cloud-init.

osinfodb — OS Image Database

libosinfo parser & image database manager. Provides automated image discovery, signature verification (GPG/SHA256), and local caching.

Communication Protocols

Transport: HTTP/2 with gRPC-Web fallback for web browser clients.
Encoding: Binary Protobuf + JSON.
Auth: mTLS with PiTrust X.509 certificates.
Streaming: Supports Unary, Server Streaming, Client Streaming, and Bidi Streaming.

Transport: Raw HTTP/2 over private Unix Sockets (/var/run/pivirt/pivirtd.sock) or internal node IPs.
Usage: Inter-component communication (agent ↔ pivirtd, agent ↔ iscsiman).
Isolation: Prevents host hypervisor endpoints from being exposed to the network.

Agent-to-Director Session Protocol

The link between the agent and the director uses a resilient, bidirectional streaming protocol (NodeService.Session) over mTLS:

sequenceDiagram
    autonumber
    participant D as Director (Control Plane)
    participant A as Agent (Node Supervisor)
    participant P as pivirtd (Host Hypervisor)

    Note over A,D: mTLS Enrollment & Handshake
    A->>D: RegisterNode(node_uuid, hostname, agent_version)
    D-->>A: RegisterNodeResponse(lease_epoch)

    Note over A,D: Bidirectional Session Stream (NodeService.Session)
    A->>D: Open Stream (Session)
    A->>D: NodeEvent_Heartbeat (every 10s)
    A->>D: NodeEvent_HostReport (CPU, memory, disk, hypervisor metrics)

    opt Task Execution
        D->>A: NodeDirective_Task(Task)
        A->>P: Execute via QMP / Unix Socket
        P-->>A: Hardware confirmation
        A->>D: UpdateTaskStatus(status=RUNNING, progress=X%)
        A->>D: UpdateTaskStatus(status=COMPLETED)
    end

    opt Drift Reconciliation
        D->>A: NodeDirective_Resync
        A->>P: List physical VMs & disks
        P-->>A: Physical inventory
        A->>D: FullInventoryEvent(VMs with stamped pivirt.hu/* labels)
        Note over D: Reconciles cluster DB against bare-metal ground truth
    end

Protocol Mechanics

mTLS Bootstrap & Lease Renewal:
- The agent automatically requests its node identity certificate from the Director’s internal CA.
- It maintains an active lease epoch with 10-second heartbeats. If heartbeats cease, the Director marks the node degraded and triggers cluster HA / evacuation.
Task Queue Execution:
- The Director pushes placement tasks (NodeDirective_Task) directly down the stream. The agent’s TaskExecutor processes them locally and streams progress and logs back via TaskServiceClient.UpdateTaskStatus.
Disaster Recovery & Drift Detection (NodeDirective_Resync):
- If the Director restarts or experiences database corruption, it issues a resync.
- The agent queries pivirtd for actual running VMs, reads their stamped labels (pivirt.hu/uid, pivirt.hu/generation), and transmits a FullInventoryEvent so the Director can completely rebuild its inventory from the physical hosts.

Declarative Resource Model & Two-Generation Integrity

PiVirt supports full declarative GitOps management using Kubernetes-style manifests (VirtualMachine, StoragePool, Network, VirtualDisk, IPPool) applied directly with pivirtctl apply -f or via the Director.

The Dual-Generation Model

To ensure cluster controllers and host-level maintenance operations never conflict, PiVirt decouples host generation from cluster generation:

┌────────────────────────────────────────────────────────┐
│ Cluster Control Plane (Director)                      │
│ - Stamps desired cluster generation:                   │
│   labels: { "pivirt.hu/generation": "7" }             │
└───────────────────────────┬────────────────────────────┘
                            │ Carried opaquely
                            ▼
┌────────────────────────────────────────────────────────┐
│ Host ObjectStore (pivirtd)                             │
│ - Manages host-local generation on disk:              │
│   metadata.generation: 3                              │
│   status.observedGeneration: 3                        │
└────────────────────────────────────────────────────────┘

Host Generation (metadata.generation): Owned and incremented by pivirtd.ObjectStore whenever a resource’s spec changes on disk. Host reconcilers mark status.observedGeneration once the state is realized.
Cluster Generation (pivirt.hu/generation label): Owned by the Director and carried opaquely by pivirtd. pivirtd never writes or increments this label. Local host edits with pivirtctl never trigger false cluster drift.

Reserved Label Namespace (`pivirt.hu/*`)

Labels under the pivirt.hu/ prefix are reserved for system components (ValidateUserLabels):

pivirt.hu/uid: Cluster-wide unique identifier.
pivirt.hu/owner-org: Organization ID for tenancy and billing.
pivirt.hu/owner-project: Project ID for RBAC.
pivirt.hu/owner-principal: Identity of creator / owner.
pivirt.hu/managed-by: Automation tool (director, terraform, etc.).
Regular users cannot supply pivirt.hu/* labels in admission manifests, preventing cross-tenant privilege escalation.

Data Flow: Declarative VM Provisioning

sequenceDiagram
    participant User
    participant PVCLI as pvcli / pivirtctl
    participant Agent as agent
    participant PIVIRTD as pivirtd
    participant QEMU as QEMU/KVM

    User->>PVCLI: pivirtctl apply -f vm.yaml
    PVCLI->>PIVIRTD: ApplyResource(VirtualMachine)
    Note over PIVIRTD: Admission Chain (Decode -> Admit -> Validate -> Store)
    PIVIRTD->>QEMU: qemu-img create + launch QEMU process
    QEMU-->>PIVIRTD: Hardware created & QMP socket ready
    PIVIRTD-->>PVCLI: ApplyResult (created, generation=1)
    PVCLI-->>User: Success (VirtualMachine/web-01 created)

Security Architecture & Policies

Identity & Trust Hierarchy (PiTrust PKI)

PiTrust PKI: 2-tier X.509 authority with Root CA (PiRootR1.pem) and signing Intermediate CA (PiCA.pem).
Automated Lifecycle: Managed via PiTrust Operator (ManagedCertificate CRD) in Kubernetes or pitrust CLI on bare-metal hosts with automated 30-day pre-expiry renewal.
Mutual Authentication: Full mTLS across Director, Agent, Hypervisor (pivirtd), and Storage (iscsiman).

flowchart LR
    CP["Control Plane (Director / Agent)"] ---|mTLS (PiTrust PKI)| DP["Data Plane (pivirtd / iscsiman)"]

Storage Architecture

flowchart TB
    subgraph vm_disk_layer[VM Disk Layer]
        subgraph storage_layers[Storage Layers]
            local["Local (qcow2/raw)"]
            iscsi["iSCSI (block)"]
            nfs["NFS (file)"]
        end
        subgraph ocs2["iscsiman (OCFS2)"]
            target[Target]
            initiator[Initiator]
            cluster["Cluster (Locking)"]
        end
    end

Virtual Networking Model

Select a networking model below to inspect the packet flow topology:

Standard Linux bridge interface connecting virtual Ethernet pairs to physical uplinks:

flowchart LR
    VM --> veth
    veth --> linux-bridge
    linux-bridge --> phys
    phys --> uplink

Isolates VM traffic onto specific 802.1Q VLAN IDs:

flowchart LR
    VM --> veth
    veth --> linux-bridge["linux-bridge (vlan=100)"]
    linux-bridge --> phys
    phys --> trunk["trunk port"]

Encapsulates Layer 2 VM frames inside UDP packet overlays for multi-host clusters:

flowchart LR
    VM_local --> vxlan0
    vxlan0 --> UDP["UDP:4789"]
    UDP --> vxlan0_remote["vxlan0"]
    vxlan0_remote --> VM_remote

Scalability Limits

Dimension	Limit	Notes
VMs per Node	~500	Limited by KVM host CPU/memory overhead
Nodes per Cluster	~50	Limited by etcd Raft consensus & service mesh overhead
Volumes per Node	~1000	iSCSI session limits & OCFS2 cluster lock limits
API Throughput	~10k QPS	Multiplexed ConnectRPC over HTTP/2

Availability & Failure Domains

flowchart TB
    subgraph AZ["Availability Zone"]
        subgraph N1["Node 1"]
            P1[pivirtd]
            A1[agent]
        end
        subgraph N2["Node 2"]
            P2[pivirtd]
            A2[agent]
        end
        subgraph N3["Node 3"]
            P3["pivirtd / agent"]
        end
    end
    AZ --> STORAGE[("Shared Storage<br/>(iSCSI / NFS)")]

pivirtd ArchitectureDeep dive into the hypervisor daemon and QEMU management layer.

agent Service MeshService discovery, ConnectRPC gateway, and PiTrust PKI mesh details.

Storage Deep DiveExplore iscsiman, OCFS2 cluster locking, and volume provisioning.

Deployment GuideAutomated cluster bootstrapping across bare metal and cloud.