Storage Replication

PiVirt supports flexible block and filesystem storage replication strategies to safeguard data against hardware failures, network partitioning, and site-wide outages.

Replication Types

Synchronous Replication

sequenceDiagram
    participant Primary as Primary Site
    participant Secondary as Secondary Site

    Primary->>Secondary: Write I/O (replicate)
    Secondary-->>Primary: ACK (committed)
    Note over Primary,Secondary: Write committed on both sites (RPO = 0)

Pros:

• Zero data loss (RPO = 0)
• Consistent state

Cons:

• Higher latency
• Requires low-latency network

Asynchronous Replication

sequenceDiagram
    participant Primary as Primary Site
    participant Secondary as Secondary Site

    Primary->>Primary: Write I/O (committed immediately)
    Primary-->>Secondary: Async copy (background)
    Note over Secondary: Write applied eventually

Pros:

• Lower latency
• Works over long distances

Cons:

• Potential data loss (RPO > 0)
• Inconsistent state during failure

Semi-Synchronous Replication

sequenceDiagram
    participant Primary as Primary Site
    participant Secondary as Secondary Site

    Primary->>Primary: Write I/O (committed)
    Note over Primary: Buffer writes in batch
    Primary-->>Secondary: Batch copy (periodic flush)
    Secondary-->>Primary: ACK batch

Pros:

• Balance between latency and data loss
• Configurable RPO

iSCSI Replication

LIO Target with Replication

/etc/iscsiman/config.yaml

target:
  backend: "lio"
  replication:
    enabled: true
    mode: "async"  # sync, async, semi-sync
    target_iqn: "iqn.2024-01.hu.pilab:replica"
    target_portal: "10.0.2.10:3260"
    bandwidth_limit_mbps: 1000
    compression: true

DRBD Replication

# Install DRBD
apt-get install -y drbd-utils

# Configure DRBD resource
cat > /etc/drbd.d/pivirt-data.res <<EOF
resource pivirt-data {
  protocol C;  # Synchronous

  on node-01 {
    device    /dev/drbd0;
    disk      /dev/sdb1;
    address   10.0.1.10:7789;
    meta-disk internal;
  }

  on node-02 {
    device    /dev/drbd0;
    disk      /dev/sdb1;
    address   10.0.1.11:7789;
    meta-disk internal;
  }
}
EOF

# Initialize and start
drbdadm create-md pivirt-data
drbdadm up pivirt-data
drbdadm primary pivirt-data --force

OCFS2 Replication

Cluster Filesystem with Replication

cluster:
  name: "pivirt-cluster"
  replication:
    enabled: true
    mode: "async"
    target_cluster: "pivirt-dr"
    target_nodes:
      - "10.0.2.10"
      - "10.0.2.11"
    bandwidth_limit_mbps: 1000
    compression: true
    encryption: true

Snapshot-Based Replication

# Create snapshot
pvcli storage snapshot create --volume data-01 --name "replica-snapshot"

# Replicate snapshot to secondary site
pvcli storage snapshot replicate \
  --volume data-01 \
  --snapshot "replica-snapshot" \
  --target-site dr-site \
  --target-volume data-01-dr

Ceph RBD Replication

Ceph RBD Mirror

# Enable RBD mirroring
rbd mirror pool enable pivirt-data pool

# Add peer
rbd mirror pool peer add pivirt-data client.admin@remote-cluster

# Enable image mirroring
rbd mirror image enable pivirt-data/vm-disk-01 snapshot

# Check mirror status
rbd mirror image status pivirt-data/vm-disk-01

Ceph Configuration

/etc/ceph/ceph.conf

[global]
fsid = 12345678-1234-1234-1234-123456789abc
mon_host = 10.0.1.10, 10.0.1.11, 10.0.1.12

[client.rbd.mirror]
rbd mirror pool = pivirt-data
rbd mirror mode = snapshot
rbd mirror snapshot interval = 60

NFS Replication

NFS with rsync

# Create replication script
cat > /usr/local/bin/nfs-replicate.sh <<'EOF'
#!/bin/bash
SOURCE="/mnt/nfs-primary"
TARGET="backup@10.0.2.10:/mnt/nfs-secondary"
LOG="/var/log/nfs-replication.log"

rsync -avz --delete \
  --exclude=".snapshot*" \
  "$SOURCE/" "$TARGET/" \
  >> "$LOG" 2>&1
EOF

chmod +x /usr/local/bin/nfs-replicate.sh

# Schedule with cron
echo "0 * * * * /usr/local/bin/nfs-replicate.sh" | crontab -

NFS with Snapshots

# Create NFS snapshot
ssh nfs-server "zfs snapshot tank/pivirt@$(date +%Y%m%d-%H%M%S)"

# Replicate snapshot
ssh nfs-server "zfs send tank/pivirt@latest | ssh backup-server zfs receive -F tank/pivirt-dr"

Monitoring Replication

Replication Status

# Check iSCSI replication status
pvcli storage replication status

# Check DRBD status
drbd-overview

# Check Ceph RBD mirror status
rbd mirror image status pivirt-data/vm-disk-01

Replication Metrics

# View replication metrics
curl http://localhost:9003/metrics | grep replication

# Key metrics:
# iscsiman_replication_lag_seconds{volume="data-01"} 5
# iscsiman_replication_bytes_total{volume="data-01"} 1073741824
# iscsiman_replication_status{volume="data-01",status="healthy"} 1

Prometheus Alerts

prometheus-rules.yml

groups:
  - name: pivirt.replication.rules
    rules:
      - alert: PivirtReplicationLagHigh
        expr: iscsiman_replication_lag_seconds > 60
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Replication lag > 60s for {{ $labels.volume }}"

      - alert: PivirtReplicationFailed
        expr: iscsiman_replication_status{status="failed"} == 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Replication failed for {{ $labels.volume }}"

Troubleshooting

Replication Lag

# Check network bandwidth
iperf3 -c target-site

# Check replication status
pvcli storage replication status

# Increase bandwidth limit
pvcli storage replication update \
  --volume data-01 \
  --bandwidth-limit 10000

Replication Failed

# Check logs
journalctl -u iscsiman -f | grep -i replicat

# Check network connectivity
nc -zv target-site 3260

# Restart replication
pvcli storage replication restart --volume data-01

Data Inconsistency

# Verify data integrity
pvcli storage volume verify data-01

# Resync from primary
pvcli storage replication resync --volume data-01 --force

Best Practices

1. Choose Right Replication Mode

Use Case	Recommended Mode
Zero data loss required	Synchronous
Low latency required	Asynchronous
Balance between RPO and latency	Semi-synchronous

2. Monitor Proactively

• Set up alerts for replication lag
• Monitor bandwidth utilization
• Track replication success rate

3. Test Failover Regularly

• Test failover procedures
• Verify data consistency
• Measure RTO and RPO

4. Document Procedures

• Document failover procedures
• Document recovery procedures
• Keep runbooks up to date

Related

• High Availability
• Storage Configuration
• Backup & Restore