Metadata Server Returns 404 / "VM Not Found"
Symptom: Talos console shows:
[talos] retrying error: failed to fetch talos config from http://169.254.169.254/talos/config: 404Or pivirtd logs show:
WRN metadata: VM not found ip=10.0.0.19The VM boots but never receives its Talos machine config and sits in maintenance mode.
How the Metadata Service Works
Section titled “How the Metadata Service Works”VM (10.0.0.19) → HTTP GET http://169.254.169.254/talos/config → iptables DNAT: 169.254.169.254:80 → 10.0.0.1:8080 → pivirtd metadata server (10.0.0.1:8080) → looks up VM by source IP → reads /var/lib/pivirt/vms/<name>/talos-config → serves the YAML contentThere are three places this can break.
Root Cause A — DNAT iptables rule is missing
Section titled “Root Cause A — DNAT iptables rule is missing”The redirect from the link-local metadata IP (169.254.169.254) to the gateway is done by an iptables DNAT rule. If the host was rebooted or iptables was flushed, this rule may be gone.
How to check
Section titled “How to check”sudo iptables -t nat -L PREROUTING -n -v | grep 169.254.169.254Expected:
DNAT tcp -- * * 0.0.0.0/0 169.254.169.254 tcp dpt:80 to:10.0.0.1:8080sudo iptables -t nat -A PREROUTING \ -d 169.254.169.254 -p tcp --dport 80 \ -j DNAT --to-destination 10.0.0.1:8080To make this persistent across reboots, add it to the host’s iptables-persistent config or a systemd unit that runs at boot.
Root Cause B — VM IP is not registered in the metadata store
Section titled “Root Cause B — VM IP is not registered in the metadata store”pivirtd’s metadata server resolves the requesting VM by matching the source IP from the HTTP request against its in-memory store. This store is populated at startup from:
- The IPAM lease file (
/var/lib/pivirt/network/ipam.json) — MAC → IP mapping. - The VM config (
config.yaml) — MAC address of the NI.
If pivirtd starts before a DHCP lease is granted (i.e., the VM got its IP after pivirtd loaded), the IP-to-VM mapping may be missing.
How to check
Section titled “How to check”# Query the metadata server directly from the host (bypasses DNAT)curl -s http://10.0.0.1:8080/talos/config \ -H "X-Forwarded-For: 10.0.0.19" 2>&1 | head -20If you get VM not found, the store has no entry for 10.0.0.19.
The easiest fix is to restart pivirtd so it reloads the IPAM store and re-registers all VMs:
sudo systemctl restart pivirtdThis is safe as long as the VMs are already running — pivirtd re-attaches to the existing QEMU processes via PID file tracking.
Root Cause C — talos-config file is missing or empty
Section titled “Root Cause C — talos-config file is missing or empty”pivirtd reads the machine config from /var/lib/pivirt/vms/<name>/talos-config. If this file is missing, the metadata server has nothing to serve.
How to check
Section titled “How to check”ls -la /var/lib/pivirt/vms/talos-worker-4/talos-confighead -5 /var/lib/pivirt/vms/talos-worker-4/talos-configExpected: a valid Talos machine config YAML starting with:
version: v1alpha1machine: type: worker ...# Copy the worker config generated by talosctl gen configsudo cp ./cluster-config/worker.yaml /var/lib/pivirt/vms/talos-worker-4/talos-configIf the file was already served (VM already booted once), a restart is needed for the new config to take effect:
# If the node is reachable via talosctl, apply without reboot:talosctl -n 10.0.0.19 apply-config --file ./cluster-config/worker.yaml
# If it's in maintenance mode, reboot so it fetches fresh config:sudo pivirtctl vm reboot talos-worker-4Root Cause D — Metadata server not listening
Section titled “Root Cause D — Metadata server not listening”If pivirtd failed to bind the metadata server (e.g., port 8080 already in use), requests reach the host but nothing answers them.
How to check
Section titled “How to check”# Is pivirtd listening on 10.0.0.1:8080?sudo ss -tlnp | grep ':8080'
# Check pivirtd startup logsudo journalctl -u pivirtd | grep -E 'metadata|8080|listen' | head -10# Find what is occupying port 8080sudo ss -tlnp | grep 8080sudo fuser 8080/tcp
# Kill the conflicting process if safe, then restart pivirtdsudo systemctl restart pivirtdEnd-to-End Metadata Test
Section titled “End-to-End Metadata Test”Run this from inside the guest (if you have shell access via talosctl exec) or from the host using the X-Forwarded-For header trick:
# From host — simulate VM at 10.0.0.19 requesting its configcurl -v http://10.0.0.1:8080/talos/config \ -H "X-Forwarded-For: 10.0.0.19" 2>&1
# Expected: HTTP 200 with the Talos machine YAML