Monitoring & Observability

PiVirt provides built-in observability through standardized Prometheus metrics, structured zerolog streams, and OpenTelemetry-compatible tracing across all daemon components.

Metrics

Prometheus Endpoints

Component	Endpoint	Port
pivirtd	`/metrics`	9002
agent	`/metrics`	8444
iscsiman	`/metrics`	9003
deployer	`/metrics`	9005

Example Prometheus Configuration

scrape_configs:
  - job_name: 'pivirtd'
    static_configs:
      - targets: ['node-01:9002', 'node-02:9002', 'node-03:9002']

  - job_name: 'pivirt-agent'
    static_configs:
      - targets: ['node-01:8444', 'node-02:8444', 'node-03:8444']

  - job_name: 'iscsiman'
    static_configs:
      - targets: ['node-01:9003', 'node-02:9003', 'node-03:9003']

Key Metrics

VM Metrics

pivirtd_vm_count{status="running"} 42
pivirtd_vm_cpu_usage_percent{vm="web-01"} 23.5
pivirtd_vm_memory_usage_bytes{vm="web-01"} 2147483648
pivirtd_vm_disk_read_bytes_total{vm="web-01"} 1073741824
pivirtd_vm_disk_write_bytes_total{vm="web-01"} 536870912
pivirtd_vm_network_rx_bytes_total{vm="web-01"} 2147483648
pivirtd_vm_network_tx_bytes_total{vm="web-01"} 1073741824

Storage Metrics

iscsiman_volume_size_bytes{volume="data-01"} 107374182400
iscsiman_volume_used_bytes{volume="data-01"} 53687091200
iscsiman_pool_capacity_bytes{pool="default"} 1099511627776
iscsiman_pool_used_bytes{pool="default"} 549755813888

Cluster Metrics

pivirt_cluster_node_count{status="ready"} 3
pivirt_cluster_vm_count{status="running"} 42
pivirt_cluster_cpu_usage_percent 45.2
pivirt_cluster_memory_usage_percent 67.8

Logging

Structured Logging

All components output structured JSON logs:

{
  "level": "info",
  "ts": "2024-01-15T10:30:00Z",
  "logger": "pivirtd",
  "msg": "VM started",
  "vm_id": "vm-abc123",
  "vm_name": "web-01",
  "duration_ms": 1234
}

Log Levels

debug - Detailed debugging information
info - General operational information
warn - Warning conditions
error - Error conditions

Log Collection

# View pivirtd logs
journalctl -u pivirtd -f

# View agent logs
journalctl -u pivirt-agent -f

# View all PiVirt logs
journalctl -u "pivirt-*" -f

Distributed Tracing

OpenTelemetry Configuration

# agent config
tracing:
  enabled: true
  endpoint: "http://jaeger.pilab.hu:14268/api/traces"
  sampling_rate: 0.1

Trace Examples

VM creation flow: pvcli → agent → pivirtd
Storage operations: agent → iscsiman → LVM/iSCSI
Migration: source agent → target agent → pivirtd

Health Checks

gRPC Health Protocol

# Check pivirtd health
grpcurl -plaintext localhost:9001 grpc.health.v1.Health/Check

# Check agent health
grpcurl -plaintext localhost:8443 grpc.health.v1.Health/Check

HTTP Health Endpoints

pivirtd exposes /healthz (liveness) and /readyz (readiness) endpoints on its Unix socket and optionally on TCP:

# pivirtd — via Unix socket
curl --unix-socket /var/lib/pivirt/daemon/pivirtd.sock http://localhost/healthz
curl --unix-socket /var/lib/pivirt/daemon/pivirtd.sock http://localhost/readyz

# pivirtd — over TCP (if listen_tcp configured)
curl http://localhost:8443/healthz
curl http://localhost:8443/readyz

# agent
curl http://localhost:8444/health

Alerting

Example Prometheus Rules

groups:
  - name: pivirt.rules
    rules:
      - alert: PivirtVMHighCPU
        expr: pivirtd_vm_cpu_usage_percent > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "VM {{ $labels.vm }} CPU usage > 90%"

      - alert: PivirtStorageLowSpace
        expr: (iscsiman_pool_used_bytes / iscsiman_pool_capacity_bytes) > 0.9
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Storage pool {{ $labels.pool }} > 90% full"

Grafana Dashboards

Pre-built dashboards available:

PiVirt Cluster Overview
VM Performance
Storage Utilization
Network Traffic