Skip to content
PiVirt

Prometheus Metrics & Alerts

PiVirt exposes standardized Prometheus metrics across all daemon services (pivirtd, agent, iscsiman, deployer), providing granular telemetry on CPU, memory, storage IOPS, and network bandwidth.

Metrics Endpoints

Section titled “Metrics Endpoints”
Component Port Path
pivirtd 9002 /metrics
agent 8444 /metrics
iscsiman 9003 /metrics
deployer 9005 /metrics

Prometheus Configuration

Section titled “Prometheus Configuration”

Basic Configuration

Section titled “Basic Configuration”
prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s


scrape_configs:
  # PiVirt components
  - job_name: 'pivirtd'
    static_configs:
      - targets:
        - 'node-01:9002'
        - 'node-02:9002'
        - 'node-03:9002'
    metrics_path: '/metrics'


  - job_name: 'pivirt-agent'
    static_configs:
      - targets:
        - 'node-01:8444'
        - 'node-02:8444'
        - 'node-03:8444'
    metrics_path: '/metrics'


  - job_name: 'iscsiman'
    static_configs:
      - targets:
        - 'node-01:9003'
        - 'node-02:9003'
        - 'node-03:9003'
    metrics_path: '/metrics'


  - job_name: 'deployer'
    static_configs:
      - targets:
        - 'deployer-01:9005'
    metrics_path: '/metrics'

Service Discovery

Section titled “Service Discovery”

File-Based Discovery

Section titled “File-Based Discovery”
prometheus.yml
scrape_configs:
  - job_name: 'pivirt'
    file_sd_configs:
      - files:
        - '/etc/prometheus/targets/pivirt-*.yml'
        refresh_interval: 30s
/etc/prometheus/targets/pivirt-nodes.yml
- targets:
  - 'node-01:9002'
  - 'node-02:9002'
  - 'node-03:9002'
  labels:
    component: 'pivirtd'
    cluster: 'production'

Consul Discovery

Section titled “Consul Discovery”
scrape_configs:
  - job_name: 'pivirt'
    consul_sd_configs:
      - server: 'consul.pilab.hu:8500'
        services:
          - 'pivirtd'
          - 'pivirt-agent'
          - 'iscsiman'

Key Metrics

Section titled “Key Metrics”

VM Metrics

Section titled “VM Metrics”
# VM count by status
pivirtd_vm_count{status="running"} 42
pivirtd_vm_count{status="stopped"} 5


# VM CPU usage
pivirtd_vm_cpu_usage_percent{vm="web-01",namespace="production"} 23.5


# VM memory usage
pivirtd_vm_memory_usage_bytes{vm="web-01",namespace="production"} 2147483648
pivirtd_vm_memory_total_bytes{vm="web-01",namespace="production"} 4294967296


# VM disk I/O
pivirtd_vm_disk_read_bytes_total{vm="web-01",disk="root"} 1073741824
pivirtd_vm_disk_write_bytes_total{vm="web-01",disk="root"} 536870912


# VM network I/O
pivirtd_vm_network_rx_bytes_total{vm="web-01",nic="eth0"} 2147483648
pivirtd_vm_network_tx_bytes_total{vm="web-01",nic="eth0"} 1073741824

Storage Metrics

Section titled “Storage Metrics”
# Storage pool capacity
iscsiman_pool_capacity_bytes{pool="default"} 1099511627776
iscsiman_pool_used_bytes{pool="default"} 549755813888
iscsiman_pool_available_bytes{pool="default"} 549755813888


# Volume metrics
iscsiman_volume_size_bytes{volume="data-01"} 107374182400
iscsiman_volume_used_bytes{volume="data-01"} 53687091200


# IOPS
iscsiman_volume_iops{volume="data-01",op="read"} 200
iscsiman_volume_iops{volume="data-01",op="write"} 100

Cluster Metrics

Section titled “Cluster Metrics”
# Node count
pivirt_cluster_node_count{status="ready"} 3
pivirt_cluster_node_count{status="not_ready"} 0


# Node resources
pivirt_cluster_node_cpu_usage_percent{node="node-01"} 45.2
pivirt_cluster_node_memory_usage_percent{node="node-01"} 67.8


# API metrics
pivirt_api_requests_total{method="CreateVM",status="200"} 1234
pivirt_api_request_duration_seconds_bucket{method="CreateVM",le="0.1"} 1000

Alert Rules

Section titled “Alert Rules”

VM Alerts

Section titled “VM Alerts”
prometheus-rules.yml
groups:
  - name: pivirt.vm.rules
    rules:
      - alert: PivirtVMHighCPU
        expr: pivirtd_vm_cpu_usage_percent > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "VM {{ $labels.vm }} CPU usage > 90%"
          description: "VM {{ $labels.vm }} in namespace {{ $labels.namespace }} has been using > 90% CPU for 5 minutes."


      - alert: PivirtVMHighMemory
        expr: (pivirtd_vm_memory_usage_bytes / pivirtd_vm_memory_total_bytes) > 0.9
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "VM {{ $labels.vm }} memory usage > 90%"
          description: "VM {{ $labels.vm }} in namespace {{ $labels.namespace }} has been using > 90% memory for 5 minutes."


      - alert: PivirtVMDown
        expr: pivirtd_vm_count{status="running"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "No VMs running"
          description: "No VMs are currently running on the node."

Storage Alerts

Section titled “Storage Alerts”
groups:
  - name: pivirt.storage.rules
    rules:
      - alert: PivirtStorageLowSpace
        expr: (iscsiman_pool_used_bytes / iscsiman_pool_capacity_bytes) > 0.9
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Storage pool {{ $labels.pool }} > 90% full"
          description: "Storage pool {{ $labels.pool }} is {{ $value | humanizePercentage }} full."


      - alert: PivirtStorageCriticalSpace
        expr: (iscsiman_pool_used_bytes / iscsiman_pool_capacity_bytes) > 0.95
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Storage pool {{ $labels.pool }} > 95% full"
          description: "Storage pool {{ $labels.pool }} is {{ $value | humanizePercentage }} full. Immediate action required."


      - alert: PivirtStorageUnavailable
        expr: iscsiman_pool_available_bytes == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Storage pool unavailable"
          description: "Storage pool {{ $labels.pool }} has no available space."

Cluster Alerts

Section titled “Cluster Alerts”
groups:
  - name: pivirt.cluster.rules
    rules:
      - alert: PivirtClusterQuorumLost
        expr: count(pivirt_cluster_node_count{status="ready"}) < 3
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Cluster quorum lost"
          description: "Cluster has fewer than 3 ready nodes."


      - alert: PivirtNodeNotReady
        expr: pivirt_cluster_node_count{status="not_ready"} > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Node not ready"
          description: "{{ $value }} node(s) are not ready."


      - alert: PivirtEtcdUnhealthy
        expr: etcd_cluster_health == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "etcd cluster unhealthy"
          description: "etcd cluster is unhealthy."

API Alerts

Section titled “API Alerts”
groups:
  - name: pivirt.api.rules
    rules:
      - alert: PivirtAPIHighErrorRate
        expr: rate(pivirt_api_requests_total{status=~"5.."}[5m]) / rate(pivirt_api_requests_total[5m]) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "API error rate > 5%"
          description: "API error rate is {{ $value | humanizePercentage }}."


      - alert: PivirtAPIHighLatency
        expr: histogram_quantile(0.95, rate(pivirt_api_request_duration_seconds_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "API p95 latency > 1s"
          description: "API p95 latency is {{ $value }}s."

Recording Rules

Section titled “Recording Rules”

Pre-computed Metrics

Section titled “Pre-computed Metrics”
groups:
  - name: pivirt.recording.rules
    rules:
      # VM resource usage
      - record: pivirt:vm:cpu_usage:avg
        expr: avg by (namespace) (pivirtd_vm_cpu_usage_percent)


      - record: pivirt:vm:memory_usage:avg
        expr: avg by (namespace) (pivirtd_vm_memory_usage_bytes / pivirtd_vm_memory_total_bytes)


      # Storage utilization
      - record: pivirt:storage:utilization:ratio
        expr: iscsiman_pool_used_bytes / iscsiman_pool_capacity_bytes


      # Cluster health
      - record: pivirt:cluster:node_health:ratio
        expr: pivirt_cluster_node_count{status="ready"} / pivirt_cluster_node_count

Grafana Dashboards

Section titled “Grafana Dashboards”

Import Dashboards

Section titled “Import Dashboards”

Pre-built dashboards are available:

  1. PiVirt Cluster Overview (Dashboard ID: 12345)
  2. VM Performance (Dashboard ID: 12346)
  3. Storage Utilization (Dashboard ID: 12347)
  4. API Performance (Dashboard ID: 12348)
Terminal window
# Import via Grafana UI
# Go to Dashboards > Import > Upload JSON file


# Or via API
curl -X POST http://localhost:3000/api/dashboards/db \
  -H "Content-Type: application/json" \
  -d @pivirt-cluster-overview.json

ServiceMonitor (Kubernetes)

Section titled “ServiceMonitor (Kubernetes)”
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: pivirt
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: pivirt
  endpoints:
    - port: metrics
      path: /metrics
      interval: 15s

Troubleshooting

Section titled “Troubleshooting”

Metrics Not Scraping

Section titled “Metrics Not Scraping”
Terminal window
# Check endpoint connectivity
curl http://node-01:9002/metrics


# Check Prometheus targets
curl http://localhost:9090/api/v1/targets


# Check Prometheus logs
journalctl -u prometheus -f

Missing Metrics

Section titled “Missing Metrics”
Terminal window
# Check component logs
journalctl -u pivirtd -f | grep -i metric


# Verify metrics endpoint
curl -v http://localhost:9002/metrics
Section titled “Related”
Open: Community & Dev•Professional: Production Clustering•Enterprise: Mission-Critical & HA
© 2026 Progressive Innovation LAB. All rights reserved.Distributed virtualization platform licensed across Open, Professional, and Enterprise tiers.