VXLAN Overlay Networks

VXLAN (Virtual Extensible LAN) overlay networks encapsulate layer-2 Ethernet frames within layer-3 UDP packets (port 4789), allowing VMs on different physical hypervisor hosts to communicate across flat virtual network segments.

Prerequisites

# Load VXLAN kernel module
modprobe vxlan

# Verify support
ip link add vxlan0 type vxlan id 100 dev eth0 dstport 4789
ip link delete vxlan0

Create VXLAN Network

Using pvcli

pvcli network create \
  --name vxlan-net \
  --type vxlan \
  --vni 1000 \
  --local-ip 10.0.1.10 \
  --remote-ips 10.0.1.11,10.0.1.12

Using ip command

# Create VXLAN interface
ip link add vxlan1000 type vxlan \
  id 1000 \
  dev eth0 \
  local 10.0.1.10 \
  remote 10.0.1.11 \
  dstport 4789 \
  nolearning

# Bring up interface
ip link set vxlan1000 up

Configuration

pivirtd Configuration

network:
  vxlans:
    - name: "vxlan1000"
      vni: 1000
      local_ip: "10.0.1.10"
      remote_ips:
        - "10.0.1.11"
        - "10.0.1.12"
      port: 4789
      ttl: 64
      learning: false

VM Network Interface

spec:
  nics:
    - name: eth0
      network: vxlan1000
      model: virtio
      ipam:
        mode: static
        address: "10.10.1.10/24"

Multi-Host Setup

Node 1 (10.0.1.10)

network:
  vxlans:
    - name: "vxlan1000"
      vni: 1000
      local_ip: "10.0.1.10"
      remote_ips:
        - "10.0.1.11"
        - "10.0.1.12"

Node 2 (10.0.1.11)

network:
  vxlans:
    - name: "vxlan1000"
      vni: 1000
      local_ip: "10.0.1.11"
      remote_ips:
        - "10.0.1.10"
        - "10.0.1.12"

Node 3 (10.0.1.12)

network:
  vxlans:
    - name: "vxlan1000"
      vni: 1000
      local_ip: "10.0.1.12"
      remote_ips:
        - "10.0.1.10"
        - "10.0.1.11"

Firewall Configuration

# Allow VXLAN traffic (UDP 4789)
nft add rule inet filter input udp dport 4789 accept

# Or with iptables
iptables -A INPUT -p udp --dport 4789 -j ACCEPT

Monitoring

Check VXLAN Interface

# List VXLAN interfaces
ip -d link show type vxlan

# Check VXLAN FDB
bridge fdb show dev vxlan1000

Check Connectivity

# From VM on node 1
ping 10.10.1.11  # VM on node 2
ping 10.10.1.12  # VM on node 3

Troubleshooting

VXLAN Not Working

# Check VXLAN interface
ip -d link show vxlan1000

# Check UDP connectivity
nc -zv 10.0.1.11 4789

# Check firewall
nft list ruleset | grep 4789

VMs Can’t Communicate

# Check VXLAN FDB
bridge fdb show dev vxlan1000

# Check VM network config
pvcli vm get web-01 -o json | jq '.spec.nics'

# Check IP configuration
# Inside VM
ip addr show eth0
ip route show