Clustering Guide

PiVirt clustering enables multi-host orchestration with unified CLI control, distributed state management, shared storage pools, and live VM workload migration.

Cluster Setup

Initialize Cluster

# Single node
pvcli cluster init --name dev-cluster

# Multi-node
pvcli cluster init --name prod-cluster --control-plane 3

Join Nodes

# Generate join token
pvcli cluster token create --ttl 24h

# Join node
pvcli cluster join --token <token> --node-ip 10.0.1.11

Node Management

Add Nodes

# Add worker nodes
pvcli cluster node add --role worker --count 3

# Add specific node
pvcli cluster node add --name worker-04 --ip 10.0.1.24

Remove Nodes

# Drain node
pvcli cluster node drain worker-03

# Remove node
pvcli cluster node remove worker-03

Maintenance

# Cordon node
pvcli cluster node cordon worker-01

# Drain node
pvcli cluster node drain worker-01

# Uncordon after maintenance
pvcli cluster node uncordon worker-01

High Availability

Control Plane HA

# Initialize with 3 control plane nodes
pvcli cluster init --name prod-cluster \
  --control-plane 3 \
  --etcd-members 3

Storage HA

cluster:
  storage:
    type: "iscsi"
    multipath: true
    fencing:
      enabled: true
      agent: "ipmi"

Cluster Operations

Upgrade Cluster

# Check current version
pvcli cluster get prod-cluster -o json | jq '.version'

# Upgrade
pvcli cluster upgrade --version v0.0.90 --strategy rolling

Backup Cluster

# Backup etcd
pvcli cluster backup create --name backup-2024-01-15

# Restore
pvcli cluster backup restore --name backup-2024-01-15

Best Practices

1. Use Odd Number of Control Plane Nodes

• 3 nodes for small clusters
• 5 nodes for large clusters
• Ensures quorum

2. Distribute Across Failure Domains

spec:
  controlPlane:
    antiAffinity: true
    topologyKey: "kubernetes.io/hostname"

3. Enable Fencing

cluster:
  fencing:
    enabled: true
    agent: "ipmi"
    devices:
      - name: "node-01-fence"
        ip: "10.0.0.11"

Troubleshooting

Node Not Joining

# Check network connectivity
ping 10.0.1.10

# Check token validity
pvcli cluster token verify <token>

# Check logs
journalctl -u pivirt-agent -f

Quorum Lost

# Check etcd status
etcdctl endpoint health

# Restore from backup
pvcli cluster backup restore --name backup-2024-01-15

Related

• Clustering Concepts
• High Availability
• Cluster Bootstrap