agent — Node Agent & Service Mesh

The agent is the node-level service mesh component that provides ConnectRPC/gRPC connectivity, service discovery, and secure communication between all PiVirt components.

Technical Specifications

Property	Value
Language	Go 1.22+
Protocol	ConnectRPC (HTTP/2), gRPC
Transport	mTLS (PiTrust PKI)
Port	8443 (ConnectRPC), 9000 (gRPC internal)
Config	/etc/pivirt-agent/config.yaml
Service Mesh	Custom, PiTrust-based

Architecture

flowchart TB
    subgraph Agent["agent Architecture"]
        subgraph Gateway["ConnectRPC Gateway (Port 8443)"]
            VM_API["VM API (proxy)"]
            ST_API["Storage API (proxy)"]
            NW_API["Network API (proxy)"]
        end

        subgraph Core["Service Mesh Core"]
            PITRUST["PiTrust PKI Identity"]
            DISC["Service Discovery"]
            SYNC["Config Sync (etcd/raft)"]
        end

        subgraph Backends["Backend Connectors"]
            PV["pivirtd (gRPC)"]
            ISC["iscsiman (gRPC)"]
            OSI["osinfodb (gRPC/HTTP)"]
        end

        Gateway --> Core
        Core --> Backends
    end

Key Responsibilities

1. API Gateway

• ConnectRPC/HTTP2 endpoint for external clients (pvcli, UIs)
• Protocol translation: ConnectRPC ↔ gRPC for backends
• Request routing based on service/method
• Rate limiting and authz enforcement

2. Service Mesh & Zero-Trust Security

• mTLS everywhere: All inter-service communication encrypted
• PiTrust PKI: X.509 certificates issued by Root & X1 Intermediate CAs
• Certificate rotation: Automatic via PiTrust Operator (30-day pre-expiry)
• Service discovery: Dynamic endpoint resolution via etcd

3. Configuration Management

• Centralized config: Distributed via etcd/raft
• Hot reload: Config changes propagate without restart
• Validation: Schema validation before apply

4. Health & Observability

• Health checks: gRPC health protocol for all backends
• Metrics: Prometheus exposition (:8443/metrics)
• Tracing: OpenTelemetry (Jaeger/Zipkin)
• Logging: Structured JSON with correlation IDs

5. Power Schedule Management

• Schedule CRUD: Create, update, delete, and list VM power schedules through the agent API
• Cron-based execution: Schedules execute on a cron expression (minute/hour/day-of-month/weekday), with power-on, power-off, and reboot actions
• Backend-agnostic: Executes against the configured hypervisor through the agent’s client pool (PowerOn, ShutdownDefault, Reboot)

Configuration

Main Config (/etc/pivirt-agent/config.yaml)

# agent configuration
server:
  connectrpc:
    addr: "0.0.0.0:8443"
    tls:
      cert_file: "/etc/pivirt/certs/agent-server.crt"
      key_file: "/etc/pivirt/certs/agent-server.key"
      ca_file: "/etc/pivirt/certs/ca-chain.pem"
      client_ca_file: "/etc/pivirt/certs/ca-chain.pem"

  grpc_internal:
    addr: "0.0.0.0:9000"
    # Inter-node communication protected via PiTrust mTLS

  metrics:
    addr: "0.0.0.0:8444"
    path: "/metrics"

tls:
  enabled: true
  ca_file: "/etc/pivirt/certs/ca-chain.pem"
  cert_file: "/etc/pivirt/certs/agent-client.crt"
  key_file: "/etc/pivirt/certs/agent-client.key"
  verify_peer: true
  require_client_cert: true

service_discovery:
  type: "etcd"
  etcd:
    endpoints:
      - "https://etcd-0.pilab.hu:2379"
      - "https://etcd-1.pilab.hu:2379"
      - "https://etcd-2.pilab.hu:2379"
    ca_file: "/etc/pivirt-agent/etcd/ca.crt"
    cert_file: "/etc/pivirt-agent/etcd/client.crt"
    key_file: "/etc/pivirt-agent/etcd/client.key"
    prefix: "/pivirt/services"

backends:
  pivirtd:
    endpoint: "unix:///run/pivirtd/pivirtd.sock"
    # Or: "dns:///pivirtd.pilab.hu:9001"
    timeout: "30s"
    retry:
      max_attempts: 3
      backoff: "100ms"

  iscsiman:
    endpoint: "unix:///run/iscsiman/iscsiman.sock"
    timeout: "10s"

  osinfodb:
    endpoint: "https://osinfodb.pilab.hu:8443"
    timeout: "30s"

auth:
  # OIDC for external clients
  oidc:
    issuer_url: "https://auth.pilab.hu/realms/pivirt"
    client_id: "pivirt-agent"
    # JWKS auto-discovered from issuer_url/.well-known/jwks.json

  # RBAC policy file
  rbac_policy: "/etc/pivirt-agent/rbac.yaml"

logging:
  level: "info"
  format: "json"
  output: "stdout"
  sampling:
    initial: 100
    thereafter: 100

tracing:
  enabled: true
  endpoint: "http://jaeger.pilab.hu:14268/api/traces"
  sampling_rate: 0.1

PiTrust PKI Identity Model

Trust Hierarchy

PiTrust Root CA (PiRootR1.pem)
└── PiTrust X1 Intermediate CA (PiCA.pem)
    ├── agent (agent-client.crt)
    ├── pivirtd (pivirtd-server.crt)
    ├── director (director-server.crt)
    └── iscsiman (iscsiman-server.crt)

Certificate Issuance & Validation Flow

sequenceDiagram
    participant Agent as pivirt-agent
    participant Operator as PiTrust Operator / CA
    participant Backend as pivirtd / iscsiman
    
    Operator->>Agent: Provision TLS Secret / Keypair
    Agent->>Agent: Load TLS Config (ca-chain.pem + client.crt)
    Agent->>Backend: ConnectRPC / gRPC Handshake (mTLS)
    Backend->>Backend: Verify Certificate with PiTrust CA Chain
    Backend-->>Agent: Connection Established (Mutual Authentication)

API Endpoints

ConnectRPC (External)

Service	Package	Path
VM	pivirt.vm.v1	/pivirt.vm.v1.VMService/*
Storage	pivirt.storage.v1	/pivirt.storage.v1.StorageService/*
Network	pivirt.network.v1	/pivirt.network.v1.NetworkService/*
Cluster	pivirt.cluster.v1	/pivirt.cluster.v1.ClusterService/*

gRPC Internal (Backend ↔ Agent)

Service	Port	Auth
pivirtd	9001	PiTrust mTLS
iscsiman	9002	PiTrust mTLS
osinfodb	8443	PiTrust mTLS

RBAC Policy Example

/etc/pivirt-agent/rbac.yaml

apiVersion: rbac.pivirt.pilab.hu/v1
kind: Policy
rules:
  # Admin full access
  - roles: ["admin"]
    resources: ["*"]
    verbs: ["*"]

  # Operator: VM lifecycle, no delete
  - roles: ["operator"]
    resources: ["virtualmachines", "volumes", "networks"]
    verbs: ["get", "list", "create", "update", "patch"]

  # Viewer: Read-only
  - roles: ["viewer"]
    resources: ["*"]
    verbs: ["get", "list"]

  # VM Owner: Own VMs only
  - roles: ["vm-owner"]
    resources: ["virtualmachines"]
    verbs: ["get", "list", "update", "patch", "start", "stop", "console"]
    conditions:
      - field: "metadata.owner"
        operator: "Equals"
        value: "{{.user.identity}}"

Service Management

The agent is managed via systemd:

# Start the agent
sudo systemctl start pivirt-agent

# Stop the agent
sudo systemctl stop pivirt-agent

# Restart the agent
sudo systemctl restart pivirt-agent

# Check status
sudo systemctl status pivirt-agent

# Enable on boot
sudo systemctl enable pivirt-agent

# View logs
sudo journalctl -u pivirt-agent -f

Health Endpoints

gRPC Health Check

# Check agent health
grpcurl -plaintext localhost:8443 grpc.health.v1.Health/Check

# Check specific service
grpcurl -plaintext -d '{"service": "pivirt.vm.v1.VMService"}' \
  localhost:8443 grpc.health.v1.Health/Check

HTTP Health Check

# Check agent health
curl http://localhost:8444/health

# Check readiness
curl http://localhost:8444/ready

# Check liveness
curl http://localhost:8444/live

Management Endpoints

Reload Configuration

curl -X POST http://localhost:8444/reload

Debug Endpoints

# Get debug info
curl http://localhost:8444/debug/vars

# Get goroutine stack traces
curl http://localhost:8444/debug/pprof/goroutine?debug=1

# Get heap profile
curl http://localhost:8444/debug/pprof/heap > heap.prof

Metrics

Prometheus Metrics

# Get all metrics
curl http://localhost:8444/metrics

# Get specific metrics
curl http://localhost:8444/metrics | grep pivirt_api_requests_total

CLI Reference

For complete CLI command reference including service management, health checks, and debug endpoints, see the agent CLI Reference.

Running the Agent

Systemd Service

/etc/systemd/system/pivirt-agent.service

[Unit]
Description=PiVirt Node Agent
After=network.target pivirtd.service

[Service]
Type=notify
ExecStart=/usr/local/bin/pivirt-agent --config /etc/pivirt-agent/config.yaml
Restart=on-failure
RestartSec=5

# Security
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/run/pivirt-agent /var/log/pivirt-agent

[Install]
WantedBy=multi-user.target

PiTrust PKI Provisioning

# Issue mTLS Server Certificate for agent
pitrust issue-mtls-server \
  --name "node-01.pivirt.internal" \
  --dns "node-01.pivirt.internal" \
  --days 365

# Issue mTLS Client Certificate for agent inter-node communication
pitrust issue-mtls-client \
  --name "pivirt-agent-node-01" \
  --days 365

Troubleshooting

Certificate Issues

# Verify certificate chain
openssl verify -CAfile /etc/pivirt/certs/ca-chain.pem /etc/pivirt/certs/agent-server.crt

# Check certificate expiration
openssl x509 -in /etc/pivirt/certs/agent-server.crt -enddate -noout

# Check agent logs
journalctl -u pivirt-agent -f | grep -i tls

Service Discovery

# Check etcd connectivity
etcdctl --endpoints=https://etcd-0.pilab.hu:2379 \
  --cacert=/etc/pivirt-agent/etcd/ca.crt \
  --cert=/etc/pivirt-agent/etcd/client.crt \
  --key=/etc/pivirt-agent/etcd/client.key \
  get /pivirt/services --prefix

# Check registered services
curl -k --cert ~/.config/pvcli/client.crt \
  --key ~/.config/pvcli/client.key \
  https://localhost:8443/metrics | grep service_discovery

Connectivity

# Test ConnectRPC endpoint
grpcurl -insecure \
  -H "authorization: Bearer $TOKEN" \
  localhost:8443 list

# Test specific service
grpcurl -insecure \
  -H "authorization: Bearer $TOKEN" \
  -d '{}' \
  localhost:8443 pivirt.vm.v1.VMService/ListVMs

Related Documentation

ConnectRPC Protocol

API protocol specifications and Protobuf schemas.
View ConnectRPC Guide

PiTrust PKI & mTLS

Automated X.509 issuance, Operator CRDs, and zero-trust mTLS.
View PiTrust Guide

RBAC Configuration

Role-based access control policies and tokens.
View RBAC Guide

Service Discovery

Dynamic service registration and etcd discovery.
View Service Discovery

CLI Reference

Agent daemon flags, systemd unit, and metrics endpoints.
View Agent CLI