Diagnostic Overview & Troubleshooting

Diagnostic guides and step-by-step runbooks for PiVirt hypervisor nodes, networking layers, and Talos Linux Kubernetes clusters. Each guide addresses a specific operational symptom with root cause analysis, diagnostic commands, and verified resolutions.

Quick Reference

Symptom	Guide
VM running but no DHCP / IP address	VM Gets No DHCP / IP Address
QEMU fails with “Failed to get write lock”	QEMU Disk Write Lock
Talos reboots in kexec loop, never installs	Talos Kexec / Reboot Loop
Kubernetes node NotReady or missing	Node NotReady or Missing
Metadata server returns 404 / VM not found	Metadata Server 404
OVS bridge missing TAP ports	OVS Bridge / TAP Port Issues

General Diagnostic Commands

Cluster Health

# Check cluster status
pvcli cluster get prod-cluster -o yaml

# Check node status
pvcli cluster node list

# Check component health
pvcli cluster health

# View cluster events
pvcli cluster events --tail 50

Component Logs

# View pivirtd logs
journalctl -u pivirtd -f

# View agent logs
journalctl -u pivirt-agent -f

# View iscsiman logs
journalctl -u iscsiman -f

# View deployer logs
journalctl -u deployer -f

Debug Logging

# Enable debug logging for pivirtd
pivirtd --config /etc/pivirtd/config.yaml --log-level debug

# Enable debug logging for agent
pivirt-agent --config /etc/pivirt-agent/config.yaml --log-level debug

Talos Stack Troubleshooting Guides

These guides were created during the July 2026 Talos re-provisioning and cover the specific failure modes encountered:

Networking Issues

• VM Gets No DHCP / IP Address — Covers TAP not attached to OVS bridge, missing MAC in ipam.json, DHCP bound to wrong bridge
• OVS Bridge / TAP Port Issues — Covers missing TAP ports, duplicate ports, missing br-provision, OpenFlow misconfiguration

VM Lifecycle Issues

• QEMU Disk Write Lock — Stale QEMU processes holding disk locks, corrupted disk images
• Talos Kexec / Reboot Loop — Missing KSPP kernel params, no network during install, stale NVRAM, wrong disk bus type
• Metadata Server 404 — Missing DNAT rule, VM IP not in metadata store, missing talos-config, metadata server not listening

Kubernetes Issues

• Node NotReady or Missing — Stale kubeconfig tokens, old node objects, CNI not initialized, kubelet bootstrap timeout

Common Issues (General PiVirt)

VM Won’t Start

Symptoms:

• VM status remains Pending or Creating
• Error message in pvcli vm get <vm> -o yaml

Resolution:

# Check pivirtd logs
journalctl -u pivirtd -f

# Verify KVM is available
kvm-ok
ls -la /dev/kvm

# Check storage pool
pvcli storage pool list

# Check network bridge
brctl show

Network Connectivity Issues

Symptoms:

• VM cannot reach network
• DHCP not assigning IP

Resolution:

# Check bridge configuration
brctl show virbr0
ip link show virbr0

# Check VM network interface
pvcli vm get web-01 -o json | jq '.spec.nics'

# Check iptables/nftables
nft list ruleset | grep virbr0

# Restart networking
systemctl restart networking

Storage Issues

Symptoms:

• Volume creation fails
• VM disk errors

Resolution:

# Check storage pool
pvcli storage pool list
pvcli storage pool get default -o yaml

# Check disk space
df -h /var/lib/pivirtd

# Check LVM (if using LVM)
lvs
vgs
pvs

# Check iSCSI sessions
iscsiadm -m session -P3

# Check multipath
multipath -ll

PiTrust PKI / mTLS Certificate Issues

Symptoms:

• mTLS handshake failures
• “certificate expired” or “unknown authority” errors

Resolution:

# Verify certificate chain
openssl verify -CAfile /etc/pivirt/certs/ca-chain.pem /etc/pivirt/certs/agent-server.crt

# Check OCSP status
openssl ocsp \
  -issuer /etc/pivirt/certs/PiCA.pem \
  -cert /etc/pivirt/certs/agent-server.crt \
  -url http://pitrust.pivirt.internal:8080/ocsp \
  -CAfile /etc/pivirt/certs/PiRootR1.pem

# In Kubernetes, check ManagedCertificate CRD status
kubectl get managedcertificate -n pivirt-system
kubectl describe managedcertificate pivirt-agent-tls -n pivirt-system

Cluster Quorum Lost

Symptoms:

• etcd cluster unhealthy
• Cannot create/update resources

Resolution:

# Check etcd status
etcdctl endpoint health --endpoints=https://etcd-0:2379,https://etcd-1:2379,https://etcd-2:2379

# Check etcd members
etcdctl member list --endpoints=https://etcd-0:2379

# Restore from backup
pvcli cluster backup restore --name backup-2024-01-15

Performance Issues

High CPU Usage

# Check VM CPU usage
pvcli vm get web-01 -o json | jq '.status.cpu'

# Check host CPU usage
top
htop

# Check CPU pinning
pgrep -af qemu-system
taskset -pc <pid>

# Enable CPU pinning in VM spec
# spec.cpu.pinning: true

High Memory Usage

# Check VM memory usage
pvcli vm get web-01 -o json | jq '.status.memory'

# Check host memory usage
free -h

# Enable hugepages
echo 1024 > /proc/sys/vm/nr_hugepages

# Enable memory ballooning
# spec.memory.balloon: true

Slow Disk I/O

# Check disk I/O
iostat -x 1

# Check storage pool type
pvcli storage pool get default -o yaml

# Enable caching
# spec.disks[].cache: "writeback"

# Use virtio-blk instead of virtio-scsi
# spec.disks[].bus: "virtio"

API Issues

Connection Refused

# Check endpoint
pvcli config view | grep endpoint

# Test connectivity
nc -zv pivirtd-prod.pilab.hu 9001

# Check TLS
openssl s_client -connect pivirtd-prod.pilab.hu:9001 -CAfile ~/.config/pvcli/ca.crt

Permission Denied

# Verify cert/key match
openssl x509 -noout -modulus -in ~/.config/pvcli/client.crt | openssl md5
openssl rsa -noout -modulus -in ~/.config/pvcli/client.key | openssl md5

# Check certificate expiry
openssl x509 -in ~/.config/pvcli/client.crt -text -noout | grep "Not After"

# Check RBAC policy
cat /etc/pivirt-agent/rbac.yaml

Getting Help

Support Channels

• GitHub Issues: pilab-dev/pivirt/issues
• Discord: PiLab Community
• Documentation: pivirt.pilab.hu

Collecting Diagnostics

# Collect cluster diagnostics
pvcli cluster diagnostics --output /tmp/pivirt-diagnostics.tar.gz

# Collect component logs
pvcli cluster logs --component pivirtd --tail 1000 > /tmp/pivirtd.log
pvcli cluster logs --component agent --tail 1000 > /tmp/agent.log

# Collect system information
pvcli cluster info --output /tmp/pivirt-info.json

Related

• Monitoring
• Deployment
• Backup & Restore