Live Migration

PiVirt supports live migration of running virtual machines between cluster nodes with sub-second cutover, enabling non-disruptive host maintenance, dynamic resource rebalancing, and high availability.

Migration Types

Live Migration (Shared Storage)

flowchart TB
    subgraph Source["Source Node"]
        VM_SRC["VM (running)<br/>- memory state"]
    end
    subgraph Target["Target Node"]
        VM_TGT["VM (running)<br/>- memory state"]
    end
    STORAGE[("Shared Storage<br/>(iSCSI / OCFS2)")]

    VM_SRC -->|"migrate memory"| VM_TGT
    VM_SRC --- STORAGE
    VM_TGT --- STORAGE

Live Migration (Shared-Nothing)

flowchart LR
    subgraph Source["Source Node"]
        VM_SRC["VM (running)<br/>- memory state<br/>- disk images"]
    end
    subgraph Target["Target Node"]
        VM_TGT["VM (running)<br/>- memory state<br/>- disk images"]
    end
    VM_SRC -->|"migrate memory + disk copy"| VM_TGT

Prerequisites

Network Requirements

# Verify network connectivity
ping target-node

# Check migration port range
nc -zv target-node 49152-49215

# Check bandwidth
iperf3 -c target-node

CPU Compatibility

# Check CPU model on source
pvcli node get source-node -o json | jq '.status.cpu'

# Check CPU model on target
pvcli node get target-node -o json | jq '.status.cpu'

# Verify compatibility
# Both nodes must have compatible CPU models
# Or use "host-model" instead of "host-passthrough"

Storage Accessibility

# For shared storage - verify mount
ssh target-node "mount | grep shared-storage"

# For NBD - check NBD server
ssh target-node "nbd-client -l source-node"

Performing Migration

Live Migration

# Basic live migration
pvcli vm migrate web-01 --target node-02 --live

# With bandwidth limit
pvcli vm migrate web-01 \
  --target node-02 \
  --live \
  --bandwidth-limit 1000  # Mbps

# With downtime limit
pvcli vm migrate web-01 \
  --target node-02 \
  --live \
  --downtime-limit 500  # ms

# With compression
pvcli vm migrate web-01 \
  --target node-02 \
  --live \
  --compress

Cold Migration

# Stop VM, migrate, start
pvcli vm stop web-01
pvcli vm migrate web-01 --target node-02
pvcli vm start web-01

Migration with Disk Copy

# Migrate VM with disk copy (shared-nothing)
pvcli vm migrate web-01 \
  --target node-02 \
  --copy-disks \
  --bandwidth-limit 500  # Mbps

Migration Configuration

pivirtd Configuration

/etc/pivirtd/config.yaml

migration:
  enabled: true
  transport: "tcp"  # tcp, unix, rdma
  bind_address: "0.0.0.0"
  port_range: "49152-49215"
  bandwidth_limit_mbps: 0  # 0 = unlimited
  downtime_limit_ms: 500
  compression: true
  compression_threads: 4

Network Configuration

# Firewall rules for migration
# Allow migration ports
iptables -A INPUT -p tcp --dport 49152:49215 -j ACCEPT

# Or with nftables
nft add rule inet filter input tcp dport 49152-49215 accept

Monitoring Migration

Migration Status

# Check migration status
pvcli vm get web-01 -o json | jq '.status.migration'

# Output:
# {
#   "status": "migrating",
#   "target_node": "node-02",
#   "progress_percent": 45,
#   "remaining_time_seconds": 120,
#   "downtime_ms": 0
# }

Migration Metrics

# View migration metrics
curl http://localhost:9002/metrics | grep migration

# Key metrics:
# pivirtd_migration_count{status="completed"} 42
# pivirtd_migration_duration_seconds{vm="web-01"} 123.45
# pivirtd_migration_downtime_ms{vm="web-01"} 250

Migration Logs

# View migration logs
journalctl -u pivirtd -f | grep -i migrat

# Example log:
# {"level":"info","msg":"Starting live migration","vm":"web-01","target":"node-02"}
# {"level":"info","msg":"Migration progress","vm":"web-01","progress":45}
# {"level":"info","msg":"Migration completed","vm":"web-01","downtime_ms":250}

Troubleshooting

Migration Fails

# Check network connectivity
nc -zv target-node 49152-49215

# Check disk accessibility
# For shared storage - verify mount
# For NBD - check NBD server

# View migration logs
journalctl -u pivirtd -f | grep -i migrat

# Check CPU compatibility
pvcli node get source-node -o json | jq '.status.cpu'
pvcli node get target-node -o json | jq '.status.cpu'

High Downtime

# Increase bandwidth limit
pvcli vm migrate web-01 \
  --target node-02 \
  --live \
  --bandwidth-limit 10000  # 10 Gbps

# Enable compression
pvcli vm migrate web-01 \
  --target node-02 \
  --live \
  --compress

# Reduce dirty page rate
# Pause non-critical workloads during migration

Migration Stuck

# Cancel migration
pvcli vm migrate cancel web-01

# Check migration status
pvcli vm get web-01 -o json | jq '.status.migration'

# View detailed logs
journalctl -u pivirtd -f | grep -i migrat

Best Practices

1. Pre-Migration Checks

# Verify network connectivity
ping target-node
nc -zv target-node 49152-49215

# Check CPU compatibility
pvcli node get source-node -o json | jq '.status.cpu'
pvcli node get target-node -o json | jq '.status.cpu'

# Check storage accessibility
ssh target-node "mount | grep shared-storage"

2. Bandwidth Management

# Set appropriate bandwidth limit
pvcli vm migrate web-01 \
  --target node-02 \
  --live \
  --bandwidth-limit 1000  # Don't saturate network

3. Downtime Optimization

# Set downtime limit
pvcli vm migrate web-01 \
  --target node-02 \
  --live \
  --downtime-limit 500  # 500ms max downtime

4. Compression

# Enable compression for slow networks
pvcli vm migrate web-01 \
  --target node-02 \
  --live \
  --compress

Related

• High Availability
• Clustering Concepts
• VM Management Guide