Service Mesh Architecture

PiVirt implements a lightweight, secure service mesh using ConnectRPC, embedded discovery, and PiTrust PKI for authenticated, zero-trust inter-service communication.

Architecture

flowchart TD
    subgraph Mesh["Service Mesh Architecture"]
        subgraph CP["Control Plane"]
            pitrust["PiTrust CA (Identity & PKI)"]
            etcd["etcd (Discovery)"]
            config["Config Manager (Distribution)"]
        end
        subgraph DP["Data Plane"]
            subgraph pivirtd_node["pivirtd"]
                pproxy["Hypervisor Server (:9001)"]
            end
            subgraph agent_node["agent"]
                aproxy["Agent ConnectRPC (:8443)"]
            end
            subgraph iscsiman_node["iscsiman"]
                iproxy["Storage Manager (:9002)"]
            end
        end
        CP -->|mTLS| DP
    end

Components

PiTrust CA Server

Manages service identities and issues X.509 certificates (Root CA -> X1 Intermediate CA).

# PiTrust CA Server configuration
server:
  addr: "0.0.0.0:8080"
  certs_dir: "/etc/pitrust/certs"
  db_path: "/var/lib/pitrust/certs.db"

PiTrust Kubernetes Operator

Runs in cluster and automatically reconciles ManagedCertificate CRDs into TLS Secrets.

apiVersion: certz.pilab.hu/v1alpha1
kind: ManagedCertificate
metadata:
  name: agent-tls
  namespace: pivirt-system
spec:
  commonName: agent.pivirt.svc.cluster.local
  type: mtls-server
  secretName: agent-tls-secret
  duration: "8760h"

Direct mTLS Termination

Each service terminates ConnectRPC / gRPC TLS directly:

mTLS termination: Encrypts and authenticates all traffic
Service discovery: Routes to correct endpoints via etcd
Load balancing: Distributes traffic across nodes
Observability: Prometheus metrics, OpenTelemetry tracing, structured logging

Identity Model

Service Identifiers (SANs)

PiTrust X.509 certificates encode daemon DNS and IP SANs:

pivirtd.pivirt.internal / node-01.pivirt.internal
agent.pivirt.internal / director.pivirt.internal
iscsiman.pivirt.internal

Certificate Issuance

# Issue mTLS server certificate for pivirtd
pitrust issue-mtls-server \
  --name "pivirtd.pivirt.internal" \
  --dns "pivirtd.pivirt.internal" \
  --dns "node-01" \
  --days 365

mTLS Configuration

Server Configuration

# pivirtd config
server:
  grpc_addr: "0.0.0.0:9001"
  tls:
    enabled: true
    cert_file: "/etc/pivirtd/tls/server.crt"
    key_file: "/etc/pivirtd/tls/server.key"
    ca_file: "/etc/pivirtd/tls/ca.crt"
    client_ca_file: "/etc/pivirtd/tls/ca.crt"  # Require client certs

Client Configuration

# agent config
backends:
  pivirtd:
    endpoint: "pivirtd:9001"
    tls:
      enabled: true
      cert_file: "/etc/pivirt-agent/tls/client.crt"
      key_file: "/etc/pivirt-agent/tls/client.key"
      ca_file: "/etc/pivirt-agent/tls/ca.crt"
      server_name: "pivirtd.pilab.hu"

Traffic Management

Routing

# agent config
routing:
  rules:
    - match:
        prefix: "/pivirt.vm.v1.VMService"
      route:
        cluster: "pivirtd"
    - match:
        prefix: "/pivirt.storage.v1.StorageService"
      route:
        cluster: "iscsiman"

Load Balancing

# agent config
load_balancing:
  policy: "round_robin"  # round_robin, least_connections, random
  health_check:
    enabled: true
    interval: "10s"
    timeout: "5s"

Circuit Breaking

# agent config
circuit_breaking:
  enabled: true
  thresholds:
    consecutive_5xx: 5
    interval: "10s"
    base_ejection_time: "30s"
    max_ejection_percent: 50

Observability

Metrics

# View mesh metrics
curl http://localhost:8444/metrics | grep mesh

# Key metrics:
# pivirt_mesh_requests_total{service="pivirtd",status="200"} 1000
# pivirt_mesh_request_duration_seconds{service="pivirtd",quantile="0.99"} 0.05
# pivirt_mesh_active_connections{service="pivirtd"} 42

Tracing

# agent config
tracing:
  enabled: true
  exporter: "jaeger"
  endpoint: "http://jaeger.pilab.hu:14268/api/traces"
  sampling_rate: 0.1

Logging

# agent config
logging:
  level: "info"
  format: "json"
  output: "stdout"
  access_log:
    enabled: true
    format: "json"

Security Policies

Authorization

# RBAC policy
apiVersion: rbac.pivirt.pilab.hu/v1
kind: Policy
rules:
  - roles: ["admin"]
    resources: ["*"]
    verbs: ["*"]

  - roles: ["operator"]
    resources: ["virtualmachines", "volumes"]
    verbs: ["get", "list", "create", "update"]

  - roles: ["viewer"]
    resources: ["*"]
    verbs: ["get", "list"]

Network Policies

# Network policy
apiVersion: pivirt.pilab.hu/v1
kind: NetworkPolicy
metadata:
  name: allow-agent-to-pivirtd
spec:
  ingress:
    - from:
        - service: "agent.pivirt.internal"
      to:
        - service: "pivirtd.pivirt.internal"
      ports:
        - protocol: TCP
          port: 9001

Certificate Rotation

Automated Rotation via Operator

PiTrust Operator watches ManagedCertificate resources and automatically regenerates and rotates Kubernetes TLS Secrets 30 days before expiration without downtime.

Manual Rotation

# Force certificate reissue via CLI
pitrust issue-mtls-server --name "node-01.pivirt.internal" --days 365

# Verify certificate details
openssl x509 -in /etc/pivirt/certs/node-01.crt -text -noout | grep -A1 "Subject Alternative Name"

Troubleshooting

mTLS Handshake Failed

# Check certificate validity
openssl verify -CAfile /etc/pivirt/certs/ca-chain.pem /etc/pivirt/certs/agent-server.crt

# Verify OCSP status
openssl ocsp \
  -issuer /etc/pivirt/certs/PiCA.pem \
  -cert /etc/pivirt/certs/agent-server.crt \
  -url http://pitrust.pivirt.internal:8080/ocsp \
  -CAfile /etc/pivirt/certs/PiRootR1.pem

Service Discovery Failed

# Check etcd
etcdctl endpoint health

# Check registered services
etcdctl get /pivirt/services --prefix

# Check agent logs
journalctl -u pivirt-agent -f | grep -i discovery

Connection Refused

# Check service status
systemctl status pivirtd

# Check port
ss -tlnp | grep 9001

# Check firewall
nft list ruleset | grep 9001