Cluster Bootstrap

This guide provides a comprehensive walkthrough for bootstrapping a production-ready PiVirt cluster from bare metal or virtual machines, covering node initialization, control plane formation, and worker joins.

Prerequisites

Hardware Requirements

Control Plane Nodes: Minimum 3 nodes (for HA)
- 4 CPU cores
- 8 GB RAM
- 50 GB disk
Worker Nodes: As needed
- 8+ CPU cores
- 16+ GB RAM
- 200+ GB disk
Network: 1 Gbps+ connectivity between nodes
Storage: Shared storage (iSCSI/NFS) or local storage per node

Software Requirements

Linux kernel 5.10+
KVM support enabled
NTP synchronized
SSH access between nodes

Step 1: Prepare Nodes

Install Base System

# On all nodes
apt-get update && apt-get upgrade -y

# Install required packages
apt-get install -y \
  qemu-kvm \
  bridge-utils \
  open-iscsi \
  multipath-tools \
  ntp \
  curl \
  wget

Enable KVM

# Load KVM modules
modprobe kvm_intel  # or kvm_amd
modprobe vhost_net

# Make persistent
cat >> /etc/modules-load.d/kvm.conf <<EOF
kvm_intel
vhost_net
EOF

# Verify KVM
kvm-ok

Configure Networking

# Create bridge
cat > /etc/netplan/01-bridge.yaml <<EOF
network:
  version: 2
  bridges:
    virbr0:
      interfaces: [eth0]
      dhcp4: yes
EOF

netplan apply

Step 2: Initialize First Control Plane Node

Install PiVirt Components

# Download and install pivirtd
curl -L https://github.com/pilab-dev/pivirt/releases/latest/download/pivirtd-linux-amd64 -o /usr/local/bin/pivirtd
chmod +x /usr/local/bin/pivirtd

# Download and install agent
curl -L https://github.com/pilab-dev/pivirt/releases/latest/download/pivirt-agent-linux-amd64 -o /usr/local/bin/pivirt-agent
chmod +x /usr/local/bin/pivirt-agent

# Download and install pvcli
curl -L https://github.com/pilab-dev/pivirt/releases/latest/download/pvcli-linux-amd64 -o /usr/local/bin/pvcli
chmod +x /usr/local/bin/pvcli

Initialize Cluster

# Initialize cluster
pvcli cluster init --name prod-cluster \
  --control-plane 3 \
  --etcd-members 3

# Verify cluster
pvcli cluster get prod-cluster

Configure Storage

# Create storage pool
pvcli storage pool create \
  --name default \
  --type dir \
  --path /var/lib/pivirtd/images

# Or use LVM
pvcli storage pool create \
  --name default \
  --type lvm \
  --volume-group pivirtd-vg

Step 3: Join Additional Control Plane Nodes

Generate Join Token

# On first control plane node
pvcli cluster token create --ttl 24h
# Output: Join token: abc123...

Join Nodes

# On each additional control plane node
pvcli cluster join \
  --token abc123... \
  --node-ip 10.0.1.11 \
  --role control-plane

# Verify
pvcli cluster node list

Step 4: Join Worker Nodes

Join Workers

# On each worker node
pvcli cluster join \
  --token abc123... \
  --node-ip 10.0.1.20 \
  --role worker

# Verify
pvcli cluster node list

Step 5: Configure Shared Storage

iSCSI Setup

# On storage node
pvcli storage target create \
  --iqn iqn.2024-01.hu.pilab:vm-storage \
  --alias "VM Storage"

# On all nodes
pvcli storage initiator discover --portal 10.0.1.10:3260
pvcli storage initiator login \
  --target iqn.2024-01.hu.pilab:vm-storage \
  --portal 10.0.1.10:3260

OCFS2 Setup

# Format OCFS2 volume
pvcli storage ocfs2 create \
  --name vm-shared \
  --device /dev/disk/by-id/scsi-360050764008181c60000000000000001 \
  --cluster pivirt-cluster

# Mount on all nodes
pvcli storage ocfs2 mount \
  --name vm-shared \
  --all-nodes

Step 6: Configure Networking

Create VXLAN Network

pvcli network create \
  --name vxlan-net \
  --type vxlan \
  --vni 1000 \
  --local-ip 10.0.1.10 \
  --remote-ips 10.0.1.11,10.0.1.12

Step 7: Verify Cluster

Check Cluster Health

# Check cluster status
pvcli cluster get prod-cluster -o yaml

# Check node status
pvcli cluster node list

# Check component health
pvcli cluster health

Create Test VM

# Create test VM
pvcli vm create --name test-vm \
  --image ubuntu-24.04 \
  --cpu 1 \
  --memory 1Gi \
  --disk 10Gi

# Start VM
pvcli vm start test-vm

# Verify VM
pvcli vm get test-vm

Step 8: Configure Monitoring

Install Prometheus

# Deploy Prometheus
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/bundle.yaml

Configure Alerting

groups:
  - name: pivirt.rules
    rules:
      - alert: PivirtNodeDown
        expr: pivirt_cluster_node_count{status="ready"} < 3
        for: 5m
        labels:
          severity: critical

Post-Bootstrap Tasks

1. Configure Backup

# Setup etcd backup
pvcli cluster backup create --name initial-backup

2. Configure RBAC

# Create admin user
pvcli auth user create \
  --email admin@pilab.hu \
  --role admin

# Create operator user
pvcli auth user create \
  --email operator@pilab.hu \
  --role operator

3. Configure TLS

# Generate certificates
pvcli cert generate --cn agent.pilab.hu

# Distribute certificates
scp ca.crt node-02:/etc/pivirt-agent/tls/
scp ca.crt node-03:/etc/pivirt-agent/tls/

Troubleshooting

Node Won’t Join

# Check network connectivity
ping 10.0.1.10

# Check token validity
pvcli cluster token verify <token>

# Check logs
journalctl -u pivirt-agent -f

etcd Quorum Lost

# Check etcd status
etcdctl endpoint health

# Restore from backup
pvcli cluster backup restore --name initial-backup

Storage Not Accessible

# Check iSCSI sessions
iscsiadm -m session -P3

# Check multipath
multipath -ll