Clustering & High Availability

PiVirt clusters unite multiple physical hypervisor nodes into a cohesive, highly available virtualization platform with distributed state, shared storage pools, and live VM migration.

Cluster Architecture

flowchart TB
    subgraph AZ["Availability Zone"]
        subgraph N1["Node 1"]
            P1[pivirtd]
            A1[agent]
        end
        subgraph N2["Node 2"]
            P2[pivirtd]
            A2[agent]
        end
        subgraph N3["Node 3"]
            P3["pivirtd / agent"]
        end
    end

    STORAGE[("Shared Storage<br/>(iSCSI / NFS)")]

    AZ --> STORAGE

Cluster Components

Control Plane

• director / etcd: Cluster orchestration and state store
• PiTrust CA: PKI identity management, automated X.509 certificate issuance
• agent: Service mesh, API gateway

Data Plane

• pivirtd: Hypervisor daemon (per node)
• iscsiman: Storage manager (per node)
• Shared storage: iSCSI, NFS, OCFS2

Cluster Topology

Single Node

cluster:
  name: "dev-cluster"
  nodes:
    - name: "node-01"
      role: "control-plane"
      ip: "10.0.1.10"

Multi-Node

cluster:
  name: "prod-cluster"
  control_plane:
    - name: "cp-01"
      ip: "10.0.1.10"
    - name: "cp-02"
      ip: "10.0.1.11"
    - name: "cp-03"
      ip: "10.0.1.12"
  workers:
    - name: "worker-01"
      ip: "10.0.1.20"
    - name: "worker-02"
      ip: "10.0.1.21"

High Availability

Control Plane HA

• 3 or 5 etcd nodes for quorum
• PiTrust CA HA & Operator replication
• Load balancer for director / agent endpoints

Storage HA

• iSCSI multipath
• OCFS2 cluster filesystem
• SBD fencing for split-brain prevention

VM HA

• Automatic restart on failure
• Live migration for maintenance
• Anti-affinity rules

Cluster Operations

Bootstrap

# Initialize first node
pvcli cluster init --name prod-cluster

# Join additional nodes
pvcli cluster join --token <join-token> --node-ip 10.0.1.11

Scale

# Add worker nodes
pvcli cluster node add --role worker --count 3

# Remove node
pvcli cluster node remove worker-03

Maintenance

# Drain node for maintenance
pvcli cluster node drain worker-01

# Cordon node (no new VMs)
pvcli cluster node cordon worker-01

# Uncordon
pvcli cluster node uncordon worker-01

Quorum & Fencing

Quorum

• Majority of nodes must be available
• Prevents split-brain scenarios
• Configurable quorum policies

Fencing

cluster:
  fencing:
    enabled: true
    agent: "ipmi"
    devices:
      - name: "node-01-fence"
        ip: "10.0.0.11"
        user: "admin"
        password_ref: "vault:secret/data/bmc/node-01#password"

Next Steps

• Cluster Setup Guide
• High Availability
• Live Migration
• Storage Replication