Multi-Tenancy, VPCs & Host Isolation

Epic PIVIRT-180Design: vps-tenancy-isolation-and-vpc.md

PiVirt’s tenancy model is built for one situation: strangers who pay share a host. A tenant sees only what their organization owns, their VMs live on a private network no other tenant can reach, and the rules that keep it that way are enforced on the host by pivirtd — not by the API that a tenant talks to.

This page describes the model as designed. Each section carries its implementation status; the status badges track the stories under epic PIVIRT-180.

Status

Shipped: V0 (authenticated console routes), V1 (tenancy in the director), V2 (PortSecurity and the hardened metadata service), V3 (OverlayNetwork), V4 (VPCRouter, ExternalIP), V5 (the director’s VPC/Subnet/FloatingIP/SecurityGroup/IPAMPool objects and the compiler that ships host kinds through node sessions) and V6 (the GraphQL API, now including the networking fields). Still planned: placement (V7), metering (V8), HA (V9).

---

The three layers

Layer	Owns	Kinds
director — cluster policy	who the tenant is, what they may do, what they are allowed to use	Organization, Project, Quota, RoleBinding, VPC, Subnet, FloatingIP, SecurityGroup, IPAMPool
agent — node reconciler	the logical network across the nodes it manages	Network
pivirtd — host mechanism	the bridges, tunnels, flows and NAT rules that make isolation true on this machine	OverlayNetwork, NetworkSegment, PortSecurity, VPCRouter, ExternalIP, DHCPLease

The director never sends a tenant object to a host. It compiles them into host kinds and delivers those over the node session as a ResourceDirective. A host has no idea what a tenant is — it only knows which MAC and IP a TAP is allowed to use, which VNI a bridge belongs to, and which external address maps to which private one.

---

Tenancy — organizations, projects, quota V1 · shipped

Every tenant-owned resource carries metadata.namespace of the form org/project. The namespace is taken from the caller’s token — never from a request argument — and every list, count and permission check is scoped by it.

apiVersion: pivirt.hu/v1
kind: Organization
metadata: { name: acme }
spec:
  displayName: ACME Ltd
  owner: user:paal
status: { phase: Active }         # Suspended → VMs powered off, mutations refused, nothing deleted
---
kind: Project
metadata: { name: prod, namespace: acme }
---
kind: Quota
metadata: { name: default, namespace: acme/prod }
spec:
  limits: { vcpu: 32, memoryMiB: 65536, diskGiB: 2000, vms: 20, vpcs: 3, floatingIps: 5, snapshots: 50 }
status:
  used: { vcpu: 12, memoryMiB: 24576, vms: 6 }   # computed by selector, never stored

• Quota usage is derived, not counted. The director sums the namespace’s machines (vCPUs, MiB of memory, GiB of disk, count) when it needs the figure. A crash halfway through a create cannot leave a counter wrong, and deleting a VM out of band lowers usage on the next check. A namespace with no Quota object is unlimited; a meter set to 0 is unlimited.
• RoleBinding { subject, role, scope } scopes the existing IAM role model. The scope is org/project, org (every project in the org, present and future) or empty — the cluster scope, which makes the subject an operator. Subjects are user:<id>.
• Suspend is the abuse control. Organization.status.phase: Suspended emits a stop task for every running VM in the org and refuses every mutation in its namespaces with FailedPrecondition naming the org. Nothing is deleted; resuming allows mutations again but does not start VMs. (Port-level drop-all arrives with V2.)
• Denial reads as not found. A caller asking for a VM, project or organisation outside their bindings gets NotFound, never PermissionDenied, so the namespace set cannot be enumerated by probing.

How a request is scoped

1. The bearer token is validated and its user id becomes the subject user:<id>.
2. The subject’s RoleBindings are loaded and folded into a caller with a set of scopes.
3. Every read filters by those scopes; every write checks the target namespace against them, then the organisation’s phase, then the quota — in that order, so the error a tenant sees is the most useful one.
4. Internal paths — node events, the reconciler — run as the system caller explicitly. A request with no caller at all is refused: a new entry point cannot forget authorisation and pass.

Migration from a pre-tenancy director

On first start with tenancy, the director stamps every machine that has no namespace with default/default, creates the default organisation and project, and binds every IAM super_admin user on the cluster scope. Nothing is orphaned; the first real organisation is a create away. The migration is idempotent.

Managing organisations with pvcli

The director serves the organisation half of the cli/v1 AdminService, so pvcli org works against a director exactly as it does against a standalone agent:

# Operator: create an org with a default project, quota and owner
pvcli org create acme --description "ACME Ltd" \
  --owner-user 6650a1… --quota-cpu 16 --quota-ram 32768 --quota-disk 500

# Anyone with a binding in the org: inspect it, with derived usage
pvcli org get acme --show-quotas

# Operator: membership and quota
pvcli org update acme --add-member 66f2c3… --quota-cpu 32
pvcli org update acme --remove-member 66f2c3…

# Refused: an org cannot be renamed — its name is every namespace's prefix
pvcli org update acme --rename acme2

# Delete: --force removes empty projects; machines must be gone first
pvcli org delete acme --force

Members added this way receive the vm_admin role at org scope. Finer bindings (org/project, other roles) are RoleBinding objects.

Creating a VM in a namespace

DropletCreateParams.namespace names the org/project. When it is empty the director fills it in only if the caller holds exactly one namespace; an operator with no namespace lands in default/default; a caller with several must say which. A namespace the caller has no binding for, a suspended organisation, or a plan that does not fit the quota refuse the create before any record is written.

---

VPC — one VNI, a CIDR, a router V5 · shipped

A VPC is a private layer-2 domain spanning every host the tenant’s VMs land on. Subnets are address ranges inside it, not separate broadcast domains — a customer sees “my private network”, and the number of tunnel ports per host stays flat.

kind: VPC
metadata: { name: vpc-1, namespace: acme/prod }
spec:
  cidr: 10.42.0.0/16
  dns: [ 10.42.0.2 ]
  mtu: 1450                                          # default; fits VXLAN on a 1500 underlay
  router: { zone: zone-a, node: "", ha: false }      # node pins the gateway host; zone is recorded for V7
status: { vni: 10042, bridge: br-vpc10042, routerNode: gw-node-a, phase: Ready }
---
kind: Subnet
metadata: { name: web, namespace: acme/prod }       # ownerRef → VPC vpc-1
spec: { vpc: vpc-1, cidr: 10.42.1.0/24, gateway: 10.42.1.1, dhcp: true }

Creating a VPC allocates its VNI (lowest free in the director’s vpc.vni_start–vpc.vni_end range), names its bridge br-vpc<vni> and creates a private IPAMPool covering the CIDR. Subnets must lie inside the VPC and may not overlap each other; the gateway defaults to the first host address. A VPC cannot be deleted while it has subnets, a subnet cannot be deleted while addresses are held in it — refusals, not cascades, because a tenant’s addresses are billed state.

Addresses are claimed atomically. Every address a VM interface, a host or a floating IP holds is an IPAllocation object named <pool>-<a-b-c-d> in the pool’s namespace. The object store’s unique (kind, namespace, name) index decides a race: two claimants insert, exactly one wins, the other moves to the next candidate. No counters, no locks, and a crash halfway leaves nothing to repair.

Routing is centralized. Each VPC gets one router — a network namespace on a gateway node — rather than distributed routing on every compute host. That keeps public-IP announcement off the compute tier, and it means a VM migration never moves a public address: only the DNAT target changes, and it lives on the gateway. Gateway HA (router.ha: true, a VRRP pair) is a later step that adds no kinds.

What the director compiles it into

The compiler (pivirt-director/internal/vpc) is a pure function of the object store: it reads the VPC, its subnets, the project’s security groups and floating IPs, and every VM whose interfaces attach to the VPC, and produces one ordered list of host objects per host. Hosts are the ones carrying a VM in the VPC plus the gateway host — the pinned router.node, else a host labelled vpc.gateway_label, else the first host. The list goes out as one ResourceDirective per host over the node session; the agent proxies each object into pivirtd’s admission chain and answers with one outcome per object. The VPC’s status.phase records the result: Ready, or Degraded naming the host and object that refused or was unreachable. Re-running is free — an unchanged object is UNCHANGED on the host.

Director object	On every compute host	On the gateway host
VPC	OVSBridge br-vpc<vni> + OverlayNetwork vpc-<vni> (VNI, this host’s underlay address, every other host as a peer)	the same, plus VPCRouter vpc-<vni>-router: inside on the bridge at the first subnet’s gateway, uplink on vpc.uplink_bridge with an address from vpc.public_pool, masquerade
Subnet	NetworkSegment vpc-<vni>-<subnet> on the overlay with a per-host address from the subnet	the same, plus the DHCP delegatedRange for the whole subnet and one static DHCPLease per VM in it — DHCP is answered from exactly one host so a broadcast gets one offer
VirtualMachine.networkInterfaces[].vpc	PortSecurity vm-<uuid8>-nic<i>: the NIC’s MAC, its allocated address, dhcpClientOnly	—
FloatingIP (attached)	—	ExternalIP vpc-<vni>-fip-<name> with namespace set to the router’s netns
SecurityGroup	PortSecurity.acl on every member port, fromGroup/toGroup resolved to the members’ /32s, defaultIngress: deny	—

Two things the host does not do yet

pivirtd’s DHCP server does not yet read a NetworkSegment’s delegatedRange or the static DHCPLease objects the compiler places (they are stored and validated, not served) — guests get their address through the metadata service’s network config until that lands. And a VM’s TAP joins br-vpc<vni> because the director writes sourceType: openvswitch and sourceBridgeName onto the interface; the VM-create pipeline that carries those to pivirtd is the subject of V10.

flowchart LR
  subgraph director
    VPC --> NET
    FIP[FloatingIP] --> EXT
    SG[SecurityGroup] --> PS
  end
  subgraph agent["agent (per node)"]
    NET[Network]
  end
  subgraph compute["pivirtd, compute host"]
    NET --> OVL[OverlayNetwork vni]
    NET --> SEG[NetworkSegment]
    SEG --> OVL
    PS[PortSecurity] --> OVL
  end
  subgraph gw["pivirtd, gateway node"]
    RTR[VPCRouter netns] --> EXT[ExternalIP]
  end
  OVL -. VXLAN .-> RTR

---

Floating IPs V4 · shippedV5 · shipped

A public address is claimed from a pool and bound to a port, not to a VM.

kind: IPAMPool                                    # operator, cluster-scoped
metadata: { name: public-zone-a }
spec: { cidr: 203.0.113.0/24, kind: public, gateway: 203.0.113.1, exclude: [203.0.113.2] }
---
kind: FloatingIP
metadata: { name: web-01-public, namespace: acme/prod }
spec:
  pool: public-zone-a
  attachTo: { vm: <vm uuid>, interface: 0 }  # optional — claimed-but-unattached is a valid state
status: { address: 203.0.113.9, phase: Attached, router: vpc-10042-router, internalIP: 10.42.1.7 }

Operators register public pools (createPublicPool, or pvcli apply of an IPAMPool with kind: public); the pool’s gateway is also the VPC routers’ default next hop. A claim takes the first free address and counts against the project’s floatingIps quota. Deleting the VM returns the address to claimed-but-unattached; the tenant keeps it (and keeps paying for it) until they release it — the behaviour people expect from a reserved IP. Release refuses while attached. On the gateway an attached address becomes an ExternalIP object inside the VPC’s VPCRouter namespace: address on the router’s uplink, DNAT in prerouting, SNAT in postrouting.

---

Security groups V2 · shippedV5 · shipped

kind: SecurityGroup
metadata: { name: web, namespace: acme/prod }
spec:
  ingress:
    - { proto: tcp, ports: [22, 80, 443], from: [ 0.0.0.0/0 ] }
    - { proto: tcp, ports: [5432], fromGroup: app }
  egress:
    - { proto: any, to: [ 0.0.0.0/0 ] }

Ingress rules take from/fromGroup, egress rules to/toGroup; a referenced group must exist in the same project and a group that others reference cannot be deleted. A VM interface names its groups in networkInterfaces[].vpc.securityGroups. The director resolves fromGroup to the current member addresses and ships one PortSecurity object per member port with defaultIngress: deny; an interface with no groups gets no ACL (the VPC’s own traffic is allowed, the router never forwards unsolicited inbound). Applying a changed group re-compiles every VPC in the project. Nothing on the host evaluates group names.

---

Host isolation — what pivirtd enforces V2 · shippedV3 · shipped

The PortSecurity kind is the enforcement point. One object per VM interface, realized as OpenFlow rules under a single cookie so that delete and prune are exact.

kind: PortSecurity
metadata: { name: web-01-nic0 }                 # ownerRef → VirtualMachine web-01
spec:
  tap: tap-web01-0
  allowedMac: 52:54:00:ab:cd:ef
  allowedIps: [ 10.42.1.10/32 ]
  dhcpClientOnly: true                          # drop DHCP-server and RA frames from the guest
  acl:
    ingress: [ { proto: tcp, ports: [22, 80, 443], from: 0.0.0.0/0 } ]
    egress:  [ { proto: any, to: 0.0.0.0/0 } ]
  defaultIngress: drop
status: { observedGeneration: 3, cookie: "0x5ec0a1", flows: 14 }

What that buys:

• No spoofing. A guest that changes its MAC or takes a neighbour’s IP sends frames that are dropped at its own port.
• No rogue DHCP or router advertisements. A guest cannot answer another guest’s DHCP request.
• A trustworthy metadata service. The IMDS (169.254.169.254) identifies the VM by its pinned address and issues IMDSv2-style per-VM session tokens (EC2 header names, so cloud-init needs no change); X-Forwarded-For is ignored.
• Layer-2 separation by VNI. Two VPCs on the same host share a bridge but never a tun_id; traffic cannot cross.

Why not enforce in the agent?

pivirtd is the only writer of Open vSwitch state on a host. The agent’s older OvsFlowSync path, which applied director-compiled flows with ovs-ofctl directly (and began with del-flows), is gone — two writers on one bridge is how a reconcile loop silently deletes someone else’s rules.

---

Console access V0 · shippedV6 · shipped

The director’s VNC, serial-console, screenshot and realtime WebSocket routes require the caller’s access token. Browsers cannot set an Authorization header on a WebSocket upgrade, so those routes accept it as a token query parameter — see API Authentication.

With V1 the console handlers resolve the caller from that token and read the VM through the same namespace check as the API: a VM outside the caller’s namespaces is 404. With V6 the preferred credential is a console ticket: the consoleTicket(vmId, kind) GraphQL mutation mints a signed, five-minute token bound to one VM, and the WebSocket routes accept it as ?ticket= — no access token leaves the console. The ?token= form remains for the older console screens until they move onto GraphQL.

---

The tenant API V6 · shipped (VM slice)

Tenants talk to the director’s GraphQL only. It has two roots:

• project(id) — every field beneath it is filtered by that namespace; the ID is checked against the caller’s memberships before anything resolves.
• admin — null unless the caller holds an operator binding; hosts, zones, pools and plans live here.

VM mutations return a Task, never the resource; the console subscribes to vmChanged(projectId) and taskProgress(id) and shows the VM going Pending → Ready. Every mutation takes a clientToken (a replay returns the original result) and updates take ifMatchGeneration (a stale write is a typed CONFLICT, not a lost update). Networking objects (createVpc, createSubnet, claimFloatingIp, attachFloatingIp, applySecurityGroup, …) are stored synchronously and returned as themselves; the host work runs behind them and shows in Vpc.phase. Moving the console’s screens off their ConnectRPC agent clients is the remaining half of V6.

---

Story map

Story	Scope	Status
V0 PIVIRT-181	Console, screenshot and realtime routes behind authentication	shipped
V1 PIVIRT-182	Organization, Project, Quota, RoleBinding; namespace on every director kind; suspend	shipped
V2 PIVIRT-183	PortSecurity; IMDS identity from the port; agent OvsFlowSync removed	shipped
V3 PIVIRT-184	OverlayNetwork host kind — one keyed tunnel port per peer; segments ride overlays	shipped
V4 PIVIRT-185	VPCRouter and ExternalIP host kinds	shipped
V5 PIVIRT-186	Director VPC, Subnet, FloatingIP, SecurityGroup, IPAMPool; the compiler	shipped
V6 PIVIRT-187	GraphQL tenant and admin roots, console tickets, subscriptions	API shipped
V7 PIVIRT-188	Placement engine and allocation ledger	planned
V8 PIVIRT-189	Usage metering ledger	planned
V9 PIVIRT-190	HA restart with storage-lease fencing	planned

Kinds reference

Host-tier kinds the compiler targets.

Declarative Envelope →

Networking architecture

Bridges, VLANs, VXLAN and SR-IOV below the tenancy layer.

Networking →