Staging fleet on Proxmox
Use this runbook to test a change on the staging fleet before you ship it. Run it for changes to the agent, the units, deploy/postgres or the node bootstrap (operations.md).
scripts/staging/staging.sh keeps a long-lived staging fleet on the Proxmox hosts pve1 and pve2. It uses the planned Robot layout on Ubuntu 24.04 amd64 VMs: one host each for the control plane, builds and databases, and two application hosts. Each environment’s mesh coordinator runs on one of its own app or db hosts (mesh coordinators). It installs them with the production procedure in deploy/README.md and robot-hosts.md. Nothing is deleted when a command ends.
Since 2026-10-23, staging also replaces the Lima harness on the Mac (test:fleet-vms, test:tenants-vms, test:control-plane-ha-vms, removed that day). What that leaves unproven is in Not proven since the Lima harness was removed. Staging is a test lab. Production does not run as VMs on these hosts.
Before you start
You need:
- bash 4+ (Homebrew),
jq,curlandpython3on the Mac. - SSH to
[email protected]. Over Tailscale, useroot@pve1and setPUSHLEFT_STAGING_PVE_HOST=pve1. - The staging API token at
~/Development/proxmox-iso/tokens/pve1-staging.json, with the host CA next to it. - For the public names, once: the Tailscale auth key at
~/.config/pushleft-staging/tailscale-authkeyand the Cloudflare token at~/.config/pushleft-staging/cloudflare-token.
State (SSH key, secrets, the owner token, logs) is in ~/.config/pushleft-staging.
Test a change on staging
-
Create the fleet and run the first smoke test. On an existing fleet, skip to step 2.
scripts/staging/staging.sh up # VMs, build, install, smoke -
Build the working tree and roll it out.
scripts/staging/staging.sh update -
Run the
pljourney.scripts/staging/staging.sh smoke -
For tenant changes, run the cross-tenant plan. It reboots the app and db VMs.
scripts/staging/staging.sh isolation -
To work on a VM, open a shell on it. You log in as
ubuntu, andsudoworks.scripts/staging/staging.sh ssh cp -
Renew the certificates at least once a month.
scripts/staging/staging.sh certs
Check it worked
-
smokeends without an error. Theweb-onlyanddb-appapps stay deployed. -
From the Mac,
https://api.stg.pushleft.dev/readyzanswers 200. -
From the Mac, the fleet admin answers 401:
curl -s -o /dev/null -w '%{http_code}\n' https://admin.stg.pushleft.dev:8447/v1/me -
In the Fleet app connected to
https://admin.stg.pushleft.dev:8447, Nodes shows CPU, memory and disk for every VM.
Commands
| Command | Effect |
|---|---|
vms | Creates the template and the missing VMs through the Proxmox API. Running VMs keep their disks. CPU and memory changes apply at the next restart of the VM |
build | Copies the working tree (committed or not) to forge and runs cargo build --release. Writes bin.tar and a version 0.1.0-stg<hash> |
install | Runs the phases network tailnet controlplane mesh nodes public expose. public sets the public names, builds and installs the console, and runs certs. expose joins the tailnet nodes and writes the DNS records. A finished phase is skipped. PUSHLEFT_STAGING_PHASES="nodes" runs the named phases again |
update | Runs build, stops the cp services, installs the binaries, runs pushleft-api migrate and the grants, and starts the services again. It installs the mesh controller on cp (put_mesh). The nodes get a signed agent release through pushleft-agent-update.service |
smoke | Runs pl login, then pl up of scripts/fixtures/web-only (Kata build), then scripts/fixtures/db-app with a managed PostgreSQL on the db VM. The apps stay deployed |
tenant | Prepares a large sample tenant app from its checkout at HEAD (scripts/tenants-prepare.sh) and runs pl up in the staging org, on managed PostgreSQL and ClickHouse |
isolation | Deploys a second sample app in a second organization next to the first. Runs the cross-tenant plan of examples/tenants/README.md in both directions, with Kata builds. K3 tears down the second app and keeps the first. K4 reboots the app and db VMs over SSH |
telemetry | Installs or converges node telemetry. install (end of the nodes phase) and update also run it |
admin | Installs or converges the fleet admin API and its path from the Mac. The public phase and update also run it |
certs | Issues or renews the two wildcard certificates when they have fewer than 30 days left. Installs them on cp, app and app2, and the fleet admin’s copy |
app2 | Converges the second application VM (stg-app2) on its own: the per-node install steps, then platform DNS. Safe to run again |
names | Deploys scripts/fixtures/names-app and checks private hostnames across the environment’s nodes |
ports | Deploys scripts/fixtures/ports-app with published ports on an extra address (10.20.1.50 on app), then tears it down |
reachability | Runs scripts/reachability-check.sh against cp, build, app and db, seen from forge (outside) and over the management tailnet |
import-drill | Runs pl db import end to end into a throwaway preview, then tears it down |
rehearse | Installs a fresh control plane (rbox1, 10.20.1.20) and database node (rdb, 10.20.1.21) from zero beside staging, then reboots rbox1. rehearse recover runs the control-plane recovery drill; rehearse destroy deletes both VMs |
destroy | Destroys data: asks for confirmation, then deletes the cp, build, app, db and forge VMs and their disks on their hosts. It does not delete stg-app2 (115): delete that in Proxmox by hand. --template also deletes both templates |
Placement
The control plane, applications and databases run on pve1. The build agent and the compiler VM run on pve2, with the GitLab runners.
| VM (VMID) | Host | Address | vCPU / RAM | Role |
|---|---|---|---|---|
stg-cp (110) | pve1 | 10.20.1.10 | 6 / 16 GiB | API behind Caddy TLS, scheduler, agent gateway, controller, telemetry gateway and the central OpenTelemetry collector, PostgreSQL 17 primary, ClickHouse (pl-clickhouse), OpenBao, zot, management Headscale, the mesh controller (pushleft-mesh) with the coordinators’ Caddy server on 192.88.99.10:9443, git over OpenSSH, dnsmasq for *.stg.pl.test, a Cloudflare API stub, the operator device (a Tailscale container) |
stg-build (112) | pve2 | 10.20.1.12 | 8 / 24 GiB | Agent with Kata Containers 3.32 (nested KVM) |
stg-app (113) | pve1 | 10.20.1.13 | 16 / 33 GiB | Agent and the ingress Caddy; coordinators of environments placed here |
stg-app2 (115) | pve1 | 10.20.1.15 | 8 / 16 GiB | A second application node, as in production: agent and the ingress Caddy |
stg-db (114) | pve1 | 10.20.1.14 | 16 / 32 GiB | Agent for managed databases; coordinators of environments placed here |
stg-forge (119) | pve2 | 10.20.1.19 | 8 / 16 GiB | Compiles the binaries and is outside the fleet |
Templates and disks
Every VM is a full clone of a local template:
stg-base(100) onpve1. It uses the reserved address10.20.1.9during setup.stg-base-build(101) onpve2, onbuild-thinstorage. It uses the reserved address10.20.1.8during setup.
Both templates use the Ubuntu cloud image with base-setup.sh. A template is built again when that script or deploy/images.lock.env changes. Existing VMs keep their disks.
A fleet VM normally has two extra disks, as in robot-hosts.md section 2:
- A 64 GiB data disk: XFS with
prjquotaat/var/lib/pushleft/data. - A 32 GiB backup disk at
/var/backups/pushleft.
stg-db is larger, for the full performance dataset of the sample tenant app: a 250 GiB root disk, a 500 GiB data disk and a 250 GiB spool disk.
- The local WAL and base archive at
/var/backups/pushleft/tenantand the offsite spool at/var/backups/pushleft/offsite/tenantshare the spool disk. The spool hard-links the archive, so a backup is stored once. (Until 2026-11-01 the archive was on the root disk with a full copy in the spool, and the copies filled the root disk.) Onstg-dbthe spool is off (PUSHLEFT_DATABASE_SPOOL_MODE=disabled,PUSHLEFT_DATABASE_BACKUP_RETENTION_DAYS=3in/etc/pushleft/agent.env) until staging has an R2 bucket to ship to. Without a shipper, the spool would keep every file the store prunes. - New managed PostgreSQL and ClickHouse instances have project quotas of 300 GiB and 75 GiB.
Thin disks reserve virtual space only. Check the physical pool headroom before a full restore.
The API and the agent gateway share a private import handoff store at /var/lib/pushleft/import-blobs. Service drop-ins set it from deploy/systemd/backup-local.conf.example. PUSHLEFT_BACKUP_LOCAL_ROOT changes backups only, and source uploads keep their own store. The directory serves pl db import and volume transfers. Nothing in it is copied off the host.
Network
The bridge vmbr20 (10.20.1.0/24) spans both hosts over a dedicated point-to-point VXLAN with MTU 1450. pve1 is the gateway and does NAT. There is no DHCP.
- The VMs on
pve2use MTU 1450. New templates advertise that MTU through virtio. - The build VM also sets the Docker option
default-network-opts.bridge.com.docker.network.driver.mtu=1450. Each build network and its Kata guest then fit the tunnel. With an MTU change on the host only, registry TLS handshakes time out inside builds. - Addresses and SSH through
pve1do not change.
In staging, the bridge is the public internet. Every private path runs over the management tailnet, as in production. Nothing is reachable from the internet.
dnsmasq on cp answers *.stg.pl.test:
api,mgmt,releasesandregistrypoint at cp.appspoints at the app VM.meshpoints at the coordinators’ public alias on cp, 192.88.99.10.cp.internal,registry.internalandvault.internalpoint at the tailnet address of cp.
TLS on the internal names comes from a staging operator CA that every VM trusts.
Reach a VM or the API by hand
-
SSH to a VM through
pve1:ssh -J [email protected] -i ~/.config/pushleft-staging/id_ed25519 [email protected] -
SSH on every VM admits only the operator device and the Proxmox host (10.20.1.1, the stand-in for the Robot console).
-
The API answers on cp at
https://api.stg.pl.test, inside the bridge only. -
To use
plon cp, runsudo -i, then:XDG_CONFIG_HOME=/root/pl-owner PUSHLEFT_API_URL=http://127.0.0.1:8081 pl list
After a reboot of cp, OpenBao unseals itself from /root/staging/bao-init.json. Production keeps the unseal shares offline (vault-recovery.md).
Public names over the tailnet
Staging also has real names in the Cloudflare zone pushleft.dev. Only Waseem’s personal tailnet reaches them.
| Name | Served by | Certificate |
|---|---|---|
https://console.stg.pushleft.dev | cp’s Caddy: the built console (npm run build), with /v1 sent to the API on the same origin | *.stg.pushleft.dev |
https://api.stg.pushleft.dev | cp’s Caddy, the API (for pl --api-url) | *.stg.pushleft.dev |
https://<branch>--<project>--<handle>.apps.stg.pushleft.dev | the ingress Caddy on app | *.apps.stg.pushleft.dev |
https://admin.stg.pushleft.dev:8447 | pushleft-admin on cp, through pve1 (Fleet admin) | *.stg.pushleft.dev |
- The apps base domain is the product setting
PUSHLEFT_PREVIEW_BASE_DOMAIN=apps.stg.pushleft.dev(API and scheduler). The API hasPUSHLEFT_CONSOLE_URL=https://console.stg.pushleft.dev. - The A records point at two Tailscale nodes tagged
tag:staging:pl-stg-cp(api, console) andpl-stg-app(*.apps). - Each node is a Tailscale container on its VM, with userspace networking in its own network namespace. It forwards only TCP 80 and 443 to the VM’s Caddy (
tailscale serve --tcp). - The VM’s own tailscaled (the management tailnet) does not change. SSH is not reachable from the tailnet.
- Warning: do not run the container in the host’s network namespace. There, userspace tailscaled forwards every inbound tailnet connection to the host’s own ports. sshd and PostgreSQL answered.
- Inside staging, dnsmasq answers the same names with the VM addresses.
smokechecks the URLs from cp. - Certificates come from Let’s Encrypt, through DNS-01 with lego in Docker on the Mac. The Cloudflare token stays on the Mac.
- The ingress uses the agent’s wildcard files (
PUSHLEFT_INGRESS_WILDCARD_CERT_FILE, mode (a) in architecture.md). The agent loads a renewed file within 30 s. certsrenews a certificate with fewer than 30 days left and installs both.PUSHLEFT_STAGING_ACME=letsencrypt-staginguses the Let’s Encrypt test CA.- After the first join, the state of the tailnet nodes is in
/var/lib/pl-public-tson the VM.
To sign in to the console:
-
On cp, get a login code:
sudo -i env XDG_CONFIG_HOME=/root/pl-owner PUSHLEFT_API_URL=http://127.0.0.1:8081 pl console-code -
Sign in as
[email protected]with the code.
Differences from production
- Names and certificates. The public names point at tailnet addresses, so nothing is reachable from the internet. Custom domains (
pl domain add) cannot work, because ACME HTTP-01 needs the internet to reach the ingress. Internal names use dnsmasq and a staging CA.plon cp talks to the API through the loopback listener of cp’s Caddy. A Cloudflare API stub takes the DNS writes of the mesh worker. - Client addresses. Tailscale
serveopens a new TCP connection to Caddy. The API sees the address of the Tailscale container for every user, so all users share the per-address rate limits. - Registry storage. zot runs as
pushleft-registry.servicefrom the shipped template, without its R2 storage driver. It stores on the disk of cp. - Control-plane PostgreSQL. Staging uses the manual mode: primary on cp, no replica (production has none either). The services on cp, the mesh controller among them, connect over loopback without TLS, without HAProxy and pg_auto_failover.
- The coordinators’ public address. Every vmbr20 address is private, and gateways reach public addresses only. cp gets the alias 192.88.99.10, where its Caddy serves the coordinators on 9443 and nothing else answers. The other VMs route it to cp.
- Build DNS. The VMs resolve through the dnsmasq of cp, which
pl_buildcannot reach. The build node setsPUSHLEFT_BUILD_DNS=1.1.1.1,9.9.9.9. - Self-update trust. The update client of the agent trusts only the built-in web roots.
updateaddsPUSHLEFT_RELEASE_CA_FILE(the staging CA) to theagent.envof each node. - Telemetry. See Node telemetry.
- Fleet admin. The operator device is a container on cp, so the Mac reaches the admin API through pve1. See Fleet admin.
- Build cache. Since 2026-10-28, build112 has a dedicated 64 GiB XFS cache disk: 10 GiB per project, 32 GiB total budget, enforced project quotas. cp110 has a separate 64 GiB disk for cache snapshots and transfer temporary files. The gateway uses
PUSHLEFT_CACHE_LOCAL_ROOT, and source and backup storage do not change. See staging-build-cache.md. - Not set up: the AI proxy, mail, error ingest, backups off the host, custom domains and ACME.
Node telemetry
Every VM sends host metrics to ClickHouse on cp, as in production (architecture.md).
- Agent hosts (build, app, db).
pushleft-otelcol.service(deploy/systemd) runsotelcol-contrib0.162.0 as thepushleft-oteluser. The binary is the upstream release tarball, with its version and checksum pinned indeploy/images.lock.env(PUSHLEFT_OTELCOL_VERSION,PUSHLEFT_OTELCOL_SHA256). agent.envhasPUSHLEFT_OTEL_GATEWAY=https://cp.internal.stg.pl.test:8445. On start, the agent renders/etc/pushleft-otel/otelcol.yamlfrom its node identity and starts the unit.- The collector sends OTLP/HTTP with a telemetry client certificate to the telemetry gateway on cp. The gateway sets
host.idto the node’sfleet_nodes.id, and setscloud.region. - cp. cp is not a fleet node. Its collector uses the same unit, with a configuration that the script writes. It has
host.id=control-planeand sends straight to the central collector on loopback. - Central collector.
pushleft-otelcol-gateway.servicewithotelcol-gateway.yamllistens on 127.0.0.1:14317/14318. It writes ClickHousepushleft_otel(90-day TTL) and creates its own tables:otel_metrics_gauge,otel_metrics_sum,otel_metrics_histogram,otel_metrics_summary,otel_metrics_exponential_histogram,otel_logs,otel_traces. - ClickHouse users.
pushleft_otel_ingestis the central collector, with create and insert onpushleft_otel.pushleft_admin_readhasSELECTonpushleft_otel.*, forpushleft-admin. The passwords are${PW}ch_otel_ingestand${PW}ch_admin_readfromenv.sh. - Every 30 s:
system.cpu.*,system.memory.usage,system.cpu.load_average.{1m,5m,15m},system.filesystem.*(with the data volume),system.disk.*,system.network.*,system.paging.*andsystem.processes.*. - Every 60 s: one row per process (
process.cpu.time,process.memory.usage), withprocess.pid,process.executable.nameandprocess.commandas resource attributes. Kernel threads are left out. - The collector has no ptrace rights. It reads the CPU, memory and command line of every process. The collector cannot read the executable path, I/O or file descriptors of processes of other users.
Staging-only parts, until the agent renders them:
- The script issues the telemetry certificates: fleet CA,
purpose='telemetry', one year, issued again with fewer than 30 days left. Production issues them by hand (fleet.md). - A drop-in (
pushleft-otelcol.service.d/process.conf) adds/etc/pushleft-otel/process.yaml. It holds the per-process scraper andcompression: none. The gateway refuses compressed requests with 415, and the collector does not retry them. The configuration of the agent does not turn compression off. Both parts are inotelcol-agent.yaml. - The drop-in also makes
otelcol.yamlreadable. The agent writes it underUMask=0077, so it is 0600 root and the user of the collector cannot open it. /var/lib/pushleftis 0700 root, so the filesystem scraper cannot read the usage of the data volume. In the namespace of the collector, the drop-in puts an empty tmpfs there and binds in only the data volume, read-only. This also hides the keys of the agent from the collector.
Fleet admin
pushleft-admin (crates/pushleft-admin/README.md) runs on cp as in production. pushleft-admin.service binds to the management-tailnet address of cp (100.64.0.1:8447) and admits only the tailnet address of the operator device ($OP_TS/32).
The operator device is the pl-operator container on cp, and the Mac is on Waseem’s personal tailnet. So the path from the Mac goes through pve1 (decided 2026-10-24, staging only):
- The Mac opens
https://admin.stg.pushleft.dev:8447. The A record is the personal-tailnet address of pve1. - On pve1,
pl-stg-admin-fwd.service(socat) forwards that port to cp10.20.1.10:18447over vmbr20. - On cp,
pl-admin-fwd.service(socat) accepts only from pve1 (10.20.1.1), through the nftables tablepl_admin_fwdand the socatrange. - For each connection, it enters the network namespace of the
pl-operatorcontainer and connects to100.64.0.1:8447over the management tailnet. It looks up the namespace per connection, so a restarted container needs nothing. - The admin API sees the source
$OP_TS. TLS is end to end: the API serves the*.stg.pushleft.devcertificate from a copy in/etc/pushleft/admin, whichcertsrenews.
The Fleet app’s Connect screen defaults to production’s admin API. For staging, enter https://admin.stg.pushleft.dev:8447.
Other parts:
- Environment.
scripts/staging/admin.shwrites/etc/pushleft/admin.env. It sets the PostgreSQL rolespushleft_admin_readandpushleft_adminover loopback (password${PW}<role>), ClickHousepushleft_admin_read, and health on127.0.0.1:9110.updatecreates roles thatdeploy/README.mdadded after the install. - Provisioning (decided 2026-10-24, staging only).
provider: proxmoxmakes VMs on pve1 (fleet.md): VMIDs 150-199 and addresses 10.20.1.40-10.20.1.79 only, as full clones ofstg-basein thestagingpool. Hetzner provisioning is not configured (409). admin.shcopies the staging token and host CA from the Mac. On cp, it creates a provisioner SSH key, a copy of the fleet CA and a management Headscale API key (90 days, replaced after 80). All are in/etc/pushleft/admin, for the service user.- The sandbox of the service needs no change. ssh runs under it and writes its known-hosts file to the private
/tmp. - What a provisioned VM does not get, compared with build, app and db:
- No node telemetry.
telemetrycovers the four roles only, so there is no otelcol, noPUSHLEFT_OTEL_GATEWAYand no telemetry certificate. The app shows no host metrics. - No
PUSHLEFT_RELEASE_CA_FILEinagent.env.updatedoes not reach it, and its self-update cannot verify the staging release mirror. - No route to the coordinators’ alias, 192.88.99.10.
- No operator-only SSH (
install-operator-ssh.sh).
- No node telemetry.
- When admitted, a provisioned VM is schedulable for customer workloads. Drain it if it is only a test.
- Inside staging, dnsmasq answers
admin.stg.pushleft.devwith the tailnet address of cp. Only the operator device gets past the address check. The ACL of the management tailnet refuses other nodes. cp itself gets 403.
To get an enrollment link for Waseem’s operator (ADMIN_OPERATOR_EMAIL in admin.sh):
-
The first run of
admincreates the operator and prints the link. The link is valid for 30 minutes. -
For a new link (expired link, lost passkey), run:
scripts/staging/staging.sh ssh cp 'sudo bash -c "set -a; . /etc/pushleft/admin.env; set +a; pushleft-admin operator add --email <email> --name Waseem"' -
Open the link in a browser on the Mac.
First Proxmox provisioning run (2026-10-24)
Through the API, as the operator device, this request made VMID 150 at 10.20.1.40:
POST /v1/nodes/provision {"provider":"proxmox","pool":"application","region":"fsn1","server_type":"pve-2c-4g"}
- Clone, configuration and start took 2 min. Boot, bootstrap (disks, tailnet, agent) and enrollment took 1.5 min. The total was 3 min 40 s.
- XFS
prjquotawas enforced on the data disk, and Docker ran with userns-remap. - The node joined the management tailnet as
node.admittook its capacity from the host facts of the agent. - Drain and decommission then deleted the VM with its disks and removed its tailnet node, in 6 s.
Two defects were found and fixed:
tailscaledran from boot, before the bootstrap installed the staging CA. It never trusted the login server, andtailscale upretried forever. The bootstrap now restarts it first, and the join times out after 120 s.- Decommission left the tailnet node of the VM registered.
Open proofs
The Lima harness ran these until 2026-10-23. Staging does not run them yet, and no other suite does:
-
Automatic control-plane PostgreSQL failover: pg_auto_failover on three VMs, with failover under writes, fencing, rejoin, switchover and off-host restore. Staging runs the manual mode.
npm run test:control-plane-hacovers the manual mode on Docker only. Last results: control-plane-postgres-failover.md. -
Management tailnet checks on the hosts:
- No node reaches the tailnet address of another node.
- The agent ports, registry, OpenBao and PostgreSQL of cp are closed on its public address.
- PostgreSQL answers no other host.
- Admin SSH works only from operator devices.
isolationchecks the tenant containers only (N4). -
A scripted cp reboot: PostgreSQL on the tailnet address, with the services reconnecting and nothing restarted. The cp reboot on 2026-10-21 was a one-off that no command runs.
-
arm64 hosts. Staging is amd64, like the Robot hosts.
Results (2026-10-21)
First build of the fleet:
- About 21 minutes for the template and six VMs.
- 7 minutes for the first release build on forge.
- 5 minutes for the five install phases.
smoke then passed. web-only built with Kata and was live in 77 s. db-app was live in 205 s, with its PostgreSQL on the db VM. A second run of vms and of every install phase broke nothing.
After a reboot of cp, OpenBao unsealed itself. The API, gateway and controller were ready again within 2 minutes, with no manual step. The apps continued to serve, and the replica returned to quorum.
The binaries, the pinned images, Caddy, OpenBao, zot and Kata all run on the Sandy Bridge CPUs of the host (x86-64-v2, no AVX2).
Public names, the same day, from the Mac over the tailnet:
- The console signed in with a login code in Chrome (session cookie,
/v1on the console origin). api.stg.pushleft.dev/readyzanswered 200.- Both smoke apps served under
*.apps.stg.pushleft.devwith Let’s Encrypt certificates. - Ports 22, 5432, 8443 and 2019 on both tailnet nodes are closed.
Fixed: Kata build network
A build container asks for 12 GiB. In the nested guest, onlining that hotplugged memory takes more than 3 s. The Kata agent waits only 3 s for the NIC that QEMU hotplugs next. The agent gives up, and the guest runs with eth0 down. BuildKit then fails with “network is unreachable”.
- Staging sets
agent.hotplug_timeout=60in the Kata kernel parameters (/etc/kata-containers/configuration.toml). - With the timeout, the NIC still comes up about 3 s after the build process starts. So the Kata entrypoint of the agent now waits up to 30 s for a default route before it starts BuildKit.
- Without the timeout, 0 of 4 test runs got a network. With both changes, 4 of 4 did.
- It is not known whether bare-metal Robot build hosts need the timeout. They boot the guest faster.
See build-executor.md.