Disaster Recovery

PiVirt disaster recovery workflows ensure business continuity through multi-site replication, automated failover triggers, and cross-datacenter cluster state restoration.

DR Architecture

flowchart TB
    subgraph Primary["Primary Site"]
        P_PVC["PiVirt Cluster"]
        P_ETCD["etcd Cluster"]
        P_STOR[("Shared Storage<br/>(iSCSI / OCFS2)")]
    end

    subgraph DR["DR Site"]
        D_PVC["PiVirt Cluster<br/>(Standby)"]
        D_ETCD["etcd Cluster<br/>(Standby)"]
        D_STOR[("Shared Storage<br/>(Replica)")]
    end

    Primary -->|"Replication<br/>(async / semi-sync)"| DR

Recovery Objectives

RPO (Recovery Point Objective)

Strategy	RPO	Use Case
Synchronous replication	0	Zero data loss required
Semi-synchronous	< 1 min	Minimal data loss acceptable
Asynchronous	< 1 hour	Some data loss acceptable
Daily backup	< 24 hours	Non-critical data

RTO (Recovery Time Objective)

Strategy	RTO	Use Case
Hot standby	< 1 min	Critical services
Warm standby	< 15 min	Important services
Cold standby	< 1 hour	Non-critical services
Backup restore	< 4 hours	Archival data

Backup Strategies

etcd Backup

# Manual backup
pvcli cluster backup create --name backup-2024-01-15

# Automated backup (cron)
0 2 * * * pvcli cluster backup create --name "backup-$(date +\%Y-\%m-\%d)" --retention 7d

# Backup to S3
pvcli cluster backup create --name backup-2024-01-15 \
  --destination s3://backups.pilab.hu/etcd/

VM Backup

# Snapshot backup
pvcli vm snapshot create web-01 --name "backup-before-update"

# Full VM backup
pvcli vm backup create web-01 \
  --destination s3://backups.pilab.hu/vms/ \
  --name "web-01-backup-2024-01-15"

Storage Backup

# Volume snapshot
pvcli storage snapshot create --volume data-01 --name "data-01-backup"

# Volume backup
pvcli storage volume backup create data-01 \
  --destination s3://backups.pilab.hu/volumes/ \
  --name "data-01-backup-2024-01-15"

Replication Strategies

Synchronous Replication

/etc/iscsiman/config.yaml

target:
  replication:
    enabled: true
    mode: "sync"
    target_iqn: "iqn.2024-01.hu.pilab:replica"
    target_portal: "10.0.2.10:3260"
    bandwidth_limit_mbps: 0  # No limit

Asynchronous Replication

/etc/iscsiman/config.yaml

target:
  replication:
    enabled: true
    mode: "async"
    target_iqn: "iqn.2024-01.hu.pilab:replica"
    target_portal: "10.0.2.10:3260"
    bandwidth_limit_mbps: 1000
    compression: true

Semi-Synchronous Replication

/etc/iscsiman/config.yaml

target:
  replication:
    enabled: true
    mode: "semi-sync"
    target_iqn: "iqn.2024-01.hu.pilab:replica"
    target_portal: "10.0.2.10:3260"
    bandwidth_limit_mbps: 1000
    ack_timeout: "5s"

Failover Procedures

Planned Failover

# 1. Stop replication
pvcli storage replication pause --volume data-01

# 2. Verify replication status
pvcli storage replication status

# 3. Promote DR site
pvcli storage replication promote --volume data-01 --site dr-site

# 4. Update DNS/load balancer
# Point to DR site

# 5. Verify services
pvcli cluster get dr-cluster

Unplanned Failover

# 1. Check primary site status
# (If unreachable, proceed)

# 2. Force promote DR site
pvcli storage replication promote --volume data-01 --site dr-site --force

# 3. Update DNS/load balancer
# Point to DR site

# 4. Verify services
pvcli cluster get dr-cluster

# 5. Notify stakeholders

Recovery Procedures

Restore etcd

# List available backups
pvcli cluster backup list

# Restore from backup
pvcli cluster backup restore --name backup-2024-01-15

# Restore to specific point
pvcli cluster backup restore --name backup-2024-01-15 --force

Restore VM

# Restore from snapshot
pvcli vm snapshot restore web-01 --name "backup-before-update"

# Restore from backup
pvcli vm backup restore \
  --source s3://backups.pilab.hu/vms/web-01-backup-2024-01-15 \
  --name web-01-restored

Restore Storage

# Restore from snapshot
pvcli storage snapshot restore --volume data-01 --name "data-01-backup"

# Restore from backup
pvcli storage volume backup restore \
  --source s3://backups.pilab.hu/volumes/data-01-backup-2024-01-15 \
  --name data-01-restored

Testing DR

Regular Testing

# Test failover (non-disruptive)
pvcli dr test-failover --cluster prod-cluster --dry-run

# Test restore (non-disruptive)
pvcli dr test-restore --backup backup-2024-01-15 --dry-run

Testing Checklist

• etcd backup/restore
• VM backup/restore
• Storage backup/restore
• Failover to DR site
• Failback to primary site
• DNS/load balancer update
• Service verification
• Data integrity check

Monitoring

Replication Metrics

# View replication metrics
curl http://localhost:9003/metrics | grep replication

# Key metrics:
# iscsiman_replication_lag_seconds{volume="data-01"} 5
# iscsiman_replication_bytes_total{volume="data-01"} 1073741824
# iscsiman_replication_status{volume="data-01",status="healthy"} 1

Prometheus Alerts

prometheus-rules.yml

groups:
  - name: pivirt.dr.rules
    rules:
      - alert: PivirtReplicationLagHigh
        expr: iscsiman_replication_lag_seconds > 60
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Replication lag > 60s for {{ $labels.volume }}"

      - alert: PivirtReplicationFailed
        expr: iscsiman_replication_status{status="failed"} == 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Replication failed for {{ $labels.volume }}"

      - alert: PivirtBackupFailed
        expr: pivirt_backup_status{status="failed"} == 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Backup failed for {{ $labels.resource }}"

Best Practices

1. Regular Backups

# Daily etcd backup
0 2 * * * pvcli cluster backup create --name "backup-$(date +\%Y-\%m-\%d)"

# Weekly full VM backup
0 3 * * 0 pvcli vm backup create --all --destination s3://backups.pilab.hu/vms/

# Hourly storage snapshots
0 * * * * pvcli storage snapshot create --all-volumes --name "hourly-$(date +\%H)"

2. Test Regularly

# Monthly DR test
0 0 1 * * pvcli dr test-failover --cluster prod-cluster

3. Document Procedures

• Document failover procedures
• Document recovery procedures
• Keep runbooks up to date
• Train team members

4. Monitor Proactively

• Set up alerts for replication lag
• Monitor backup success rate
• Track RPO/RTO metrics

Troubleshooting

Replication Issues

# Check replication status
pvcli storage replication status

# Check network connectivity
nc -zv dr-site 3260

# Check logs
journalctl -u iscsiman -f | grep -i replicat

Backup Issues

# Check backup status
pvcli cluster backup get backup-2024-01-15 -o yaml

# Check logs
pvcli cluster backup logs backup-2024-01-15

Recovery Issues

# Check backup integrity
pvcli cluster backup verify --name backup-2024-01-15

# Check restore logs
pvcli cluster backup restore --name backup-2024-01-15 --verbose

Related

• High Availability
• Backup & Restore
• Storage Replication