Kubernetes Node `NotReady` or Missing from Cluster

Symptom: The VM is running and reachable by ping, but:

• kubectl get nodes does not show the new node, or
• The node appears but stays in NotReady state for more than 5 minutes.

---

Step 0 — Confirm the VM actually got an IP and Talos installed

Before diving into Kubernetes, verify the fundamentals:

# Host — ping the VM
ping -c 3 10.0.0.19

# Check Talos is running (not still in installer loop)
sudo tail -20 /var/lib/pivirt/vms/talos-worker-4/run/console.log | grep -E 'kubelet|etcd|STATE|assigned'

If ping fails → see VM Gets No DHCP / IP Address.
If console still shows install loop → see Talos Node in Kexec / Reboot Loop.

---

Root Cause A — Node has the wrong/old kubeconfig endpoint

When Talos workers are re-provisioned, they use the same worker.yaml generated by talosctl gen config. The machine.token and CA bundle inside must match the running control plane. If the control plane was re-bootstrapped without regenerating worker configs, the tokens are stale.

How to check

# On the control plane node, check if the token the new worker presents is accepted
talosctl -n 10.0.0.10 logs kubelet 2>&1 | grep -i 'token\|bootstrap\|unauthorized' | tail -20

Fix

Regenerate machine configs and re-provision workers:

talosctl gen config talos-cluster https://10.0.0.10:6443 \
  --output-dir ./cluster-config \
  --force

# Push new worker config to each node (if reachable via talosctl)
talosctl -n 10.0.0.17 apply-config --file ./cluster-config/worker.yaml
talosctl -n 10.0.0.18 apply-config --file ./cluster-config/worker.yaml
talosctl -n 10.0.0.19 apply-config --file ./cluster-config/worker.yaml

If nodes are not reachable via talosctl yet, replace the talos-config file on disk and restart the VM:

sudo cp ./cluster-config/worker.yaml /var/lib/pivirt/vms/talos-worker-4/talos-config
sudo pivirtctl vm stop -f talos-worker-4
sudo rm /var/lib/pivirt/vms/talos-worker-4/disks/system.raw
sudo qemu-img create -f raw /var/lib/pivirt/vms/talos-worker-4/disks/system.raw 10G
sudo cp /usr/share/OVMF/OVMF_VARS.fd /var/lib/pivirt/vms/talos-worker-4/nvram.fd
sudo pivirtctl vm start talos-worker-4

---

Root Cause B — Old node object still in the cluster from previous provisioning

kubectl get nodes shows the new node’s IP with the old hostname (e.g., talos-10-0-0-19) but NotReady. This is a stale node object from a previous worker that used the same IP.

How to check

kubectl get nodes -o wide
# Look for duplicate IPs or nodes with old names

Fix

Delete the old node object and let the new worker re-register:

# Drain first if it holds any running pods
kubectl drain talos-10-0-0-19 --ignore-daemonsets --delete-emptydir-data --force

# Delete the stale node object
kubectl delete node talos-10-0-0-19

The new node re-registers within ~60 seconds.

---

Root Cause C — CNI not initialized on the new node

Even after a node joins, it stays NotReady until the CNI (e.g., Flannel, Cilium) deploys its DaemonSet pod on the node.

How to check

# Describe the node — look at Conditions and Events
kubectl describe node talos-10-0-0-19

# Check CNI pod status on the new node
kubectl get pods -n kube-system -o wide | grep talos-10-0-0-19

If the CNI pod is Pending or ContainerCreating, wait 2–3 minutes. If it is CrashLoopBackOff:

kubectl logs -n kube-system <cni-pod-name> --previous

Typical causes: missing network config, MTU mismatch, or the node cannot reach the CNI registry (ghcr.io or quay.io). Check internet connectivity from the VM:

talosctl -n 10.0.0.19 exec -- curl -I https://ghcr.io 2>&1 | head -5

---

Root Cause D — kubelet certificate bootstrap timeout

Talos kubelet uses a bootstrap token to get a signed client certificate from the API server. If the control plane is unreachable when kubelet first starts (e.g., the node booted before the control plane was ready), the bootstrap may have timed out.

How to check

talosctl -n 10.0.0.19 service kubelet status
talosctl -n 10.0.0.19 logs kubelet 2>&1 | grep -i 'error\|certificate\|bootstrap' | tail -20

Fix

Simply reboot the worker — Talos will retry kubelet bootstrap on next boot:

talosctl -n 10.0.0.19 reboot
# Or via pivirtctl:
sudo pivirtctl vm reboot talos-worker-4

---

Quick Health Check Script

#!/bin/bash
# Run from the host (vm3.pilab.hu) or any machine with kubectl + talosctl access

WORKERS=(10.0.0.17 10.0.0.18 10.0.0.19)
CP=10.0.0.10

echo "=== Kubernetes nodes ==="
kubectl get nodes -o wide

echo ""
echo "=== Talos service health ==="
for ip in "${WORKERS[@]}"; do
    echo "--- $ip ---"
    talosctl -n "$ip" service kubelet status 2>/dev/null | grep -E 'STATE|HEALTH'
done

echo ""
echo "=== Network reachability ==="
for ip in "${WORKERS[@]}"; do
    ping -c 1 -W 1 "$ip" > /dev/null && echo "$ip OK" || echo "$ip UNREACHABLE"
done

---

Related Guides

• VM Gets No DHCP / IP Address
• Talos Node in Kexec / Reboot Loop
• Metadata Server Returns 404