pushleft
Deploy your first app
Manual contents
Handbook
HandbookPushleftArchitecture
Operate
Install the control planeYour own serversFleetOperations and alertsAuthenticationDomains and DNSUsage and budgetsScheduled jobsAbuse controlsAI provider proxyCustomer mailErrors and performanceLogs and metricsRetentionObject storageContainer registryGit server
Develop
The .pl.yaml manifestOnboard with an AI agentThe pl up demoBuild executorSource ingestionAgent gatewayManaged databasesSecurity findingsTunnels and shellsDev workspacesPrivate hostnamesMesh coordinatorsDesign baseline
Runbooks
Control-plane host failoverControl-plane PostgreSQL failoverControl-plane PostgreSQL restoreCustom domains and certificatesGateway key compromiseIngress host failureNode busyNode disk fullNode drain and replacementNode lostRegistry outageHetzner Robot hostsSecret rotationSecurity incidentStaging fleet on ProxmoxStaging build cacheStuck deploymentVault/OpenBao backup and recovery
Manual · docs/runbooks/staging.md

Staging fleet on Proxmox

Use this runbook to test a change on the staging fleet before you ship it. Run it for changes to the agent, the units, deploy/postgres or the node bootstrap (operations.md).

scripts/staging/staging.sh keeps a long-lived staging fleet on the Proxmox hosts pve1 and pve2. It uses the planned Robot layout on Ubuntu 24.04 amd64 VMs: one host each for the control plane, builds and databases, and two application hosts. Each environment’s mesh coordinator runs on one of its own app or db hosts (mesh coordinators). It installs them with the production procedure in deploy/README.md and robot-hosts.md. Nothing is deleted when a command ends.

Since 2026-10-23, staging also replaces the Lima harness on the Mac (test:fleet-vms, test:tenants-vms, test:control-plane-ha-vms, removed that day). What that leaves unproven is in Not proven since the Lima harness was removed. Staging is a test lab. Production does not run as VMs on these hosts.

Before you start

You need:

  • bash 4+ (Homebrew), jq, curl and python3 on the Mac.
  • SSH to [email protected]. Over Tailscale, use root@pve1 and set PUSHLEFT_STAGING_PVE_HOST=pve1.
  • The staging API token at ~/Development/proxmox-iso/tokens/pve1-staging.json, with the host CA next to it.
  • For the public names, once: the Tailscale auth key at ~/.config/pushleft-staging/tailscale-authkey and the Cloudflare token at ~/.config/pushleft-staging/cloudflare-token.

State (SSH key, secrets, the owner token, logs) is in ~/.config/pushleft-staging.

Test a change on staging

  1. Create the fleet and run the first smoke test. On an existing fleet, skip to step 2.

    scripts/staging/staging.sh up        # VMs, build, install, smoke
  2. Build the working tree and roll it out.

    scripts/staging/staging.sh update
  3. Run the pl journey.

    scripts/staging/staging.sh smoke
  4. For tenant changes, run the cross-tenant plan. It reboots the app and db VMs.

    scripts/staging/staging.sh isolation
  5. To work on a VM, open a shell on it. You log in as ubuntu, and sudo works.

    scripts/staging/staging.sh ssh cp
  6. Renew the certificates at least once a month.

    scripts/staging/staging.sh certs

Check it worked

  • smoke ends without an error. The web-only and db-app apps stay deployed.

  • From the Mac, https://api.stg.pushleft.dev/readyz answers 200.

  • From the Mac, the fleet admin answers 401:

    curl -s -o /dev/null -w '%{http_code}\n' https://admin.stg.pushleft.dev:8447/v1/me
  • In the Fleet app connected to https://admin.stg.pushleft.dev:8447, Nodes shows CPU, memory and disk for every VM.

Commands

CommandEffect
vmsCreates the template and the missing VMs through the Proxmox API. Running VMs keep their disks. CPU and memory changes apply at the next restart of the VM
buildCopies the working tree (committed or not) to forge and runs cargo build --release. Writes bin.tar and a version 0.1.0-stg<hash>
installRuns the phases network tailnet controlplane mesh nodes public expose. public sets the public names, builds and installs the console, and runs certs. expose joins the tailnet nodes and writes the DNS records. A finished phase is skipped. PUSHLEFT_STAGING_PHASES="nodes" runs the named phases again
updateRuns build, stops the cp services, installs the binaries, runs pushleft-api migrate and the grants, and starts the services again. It installs the mesh controller on cp (put_mesh). The nodes get a signed agent release through pushleft-agent-update.service
smokeRuns pl login, then pl up of scripts/fixtures/web-only (Kata build), then scripts/fixtures/db-app with a managed PostgreSQL on the db VM. The apps stay deployed
tenantPrepares a large sample tenant app from its checkout at HEAD (scripts/tenants-prepare.sh) and runs pl up in the staging org, on managed PostgreSQL and ClickHouse
isolationDeploys a second sample app in a second organization next to the first. Runs the cross-tenant plan of examples/tenants/README.md in both directions, with Kata builds. K3 tears down the second app and keeps the first. K4 reboots the app and db VMs over SSH
telemetryInstalls or converges node telemetry. install (end of the nodes phase) and update also run it
adminInstalls or converges the fleet admin API and its path from the Mac. The public phase and update also run it
certsIssues or renews the two wildcard certificates when they have fewer than 30 days left. Installs them on cp, app and app2, and the fleet admin’s copy
app2Converges the second application VM (stg-app2) on its own: the per-node install steps, then platform DNS. Safe to run again
namesDeploys scripts/fixtures/names-app and checks private hostnames across the environment’s nodes
portsDeploys scripts/fixtures/ports-app with published ports on an extra address (10.20.1.50 on app), then tears it down
reachabilityRuns scripts/reachability-check.sh against cp, build, app and db, seen from forge (outside) and over the management tailnet
import-drillRuns pl db import end to end into a throwaway preview, then tears it down
rehearseInstalls a fresh control plane (rbox1, 10.20.1.20) and database node (rdb, 10.20.1.21) from zero beside staging, then reboots rbox1. rehearse recover runs the control-plane recovery drill; rehearse destroy deletes both VMs
destroyDestroys data: asks for confirmation, then deletes the cp, build, app, db and forge VMs and their disks on their hosts. It does not delete stg-app2 (115): delete that in Proxmox by hand. --template also deletes both templates

Placement

The control plane, applications and databases run on pve1. The build agent and the compiler VM run on pve2, with the GitLab runners.

VM (VMID)HostAddressvCPU / RAMRole
stg-cp (110)pve110.20.1.106 / 16 GiBAPI behind Caddy TLS, scheduler, agent gateway, controller, telemetry gateway and the central OpenTelemetry collector, PostgreSQL 17 primary, ClickHouse (pl-clickhouse), OpenBao, zot, management Headscale, the mesh controller (pushleft-mesh) with the coordinators’ Caddy server on 192.88.99.10:9443, git over OpenSSH, dnsmasq for *.stg.pl.test, a Cloudflare API stub, the operator device (a Tailscale container)
stg-build (112)pve210.20.1.128 / 24 GiBAgent with Kata Containers 3.32 (nested KVM)
stg-app (113)pve110.20.1.1316 / 33 GiBAgent and the ingress Caddy; coordinators of environments placed here
stg-app2 (115)pve110.20.1.158 / 16 GiBA second application node, as in production: agent and the ingress Caddy
stg-db (114)pve110.20.1.1416 / 32 GiBAgent for managed databases; coordinators of environments placed here
stg-forge (119)pve210.20.1.198 / 16 GiBCompiles the binaries and is outside the fleet

Templates and disks

Every VM is a full clone of a local template:

  • stg-base (100) on pve1. It uses the reserved address 10.20.1.9 during setup.
  • stg-base-build (101) on pve2, on build-thin storage. It uses the reserved address 10.20.1.8 during setup.

Both templates use the Ubuntu cloud image with base-setup.sh. A template is built again when that script or deploy/images.lock.env changes. Existing VMs keep their disks.

A fleet VM normally has two extra disks, as in robot-hosts.md section 2:

  • A 64 GiB data disk: XFS with prjquota at /var/lib/pushleft/data.
  • A 32 GiB backup disk at /var/backups/pushleft.

stg-db is larger, for the full performance dataset of the sample tenant app: a 250 GiB root disk, a 500 GiB data disk and a 250 GiB spool disk.

  • The local WAL and base archive at /var/backups/pushleft/tenant and the offsite spool at /var/backups/pushleft/offsite/tenant share the spool disk. The spool hard-links the archive, so a backup is stored once. (Until 2026-11-01 the archive was on the root disk with a full copy in the spool, and the copies filled the root disk.) On stg-db the spool is off (PUSHLEFT_DATABASE_SPOOL_MODE=disabled, PUSHLEFT_DATABASE_BACKUP_RETENTION_DAYS=3 in /etc/pushleft/agent.env) until staging has an R2 bucket to ship to. Without a shipper, the spool would keep every file the store prunes.
  • New managed PostgreSQL and ClickHouse instances have project quotas of 300 GiB and 75 GiB.

Thin disks reserve virtual space only. Check the physical pool headroom before a full restore.

The API and the agent gateway share a private import handoff store at /var/lib/pushleft/import-blobs. Service drop-ins set it from deploy/systemd/backup-local.conf.example. PUSHLEFT_BACKUP_LOCAL_ROOT changes backups only, and source uploads keep their own store. The directory serves pl db import and volume transfers. Nothing in it is copied off the host.

Network

The bridge vmbr20 (10.20.1.0/24) spans both hosts over a dedicated point-to-point VXLAN with MTU 1450. pve1 is the gateway and does NAT. There is no DHCP.

  • The VMs on pve2 use MTU 1450. New templates advertise that MTU through virtio.
  • The build VM also sets the Docker option default-network-opts.bridge.com.docker.network.driver.mtu=1450. Each build network and its Kata guest then fit the tunnel. With an MTU change on the host only, registry TLS handshakes time out inside builds.
  • Addresses and SSH through pve1 do not change.

In staging, the bridge is the public internet. Every private path runs over the management tailnet, as in production. Nothing is reachable from the internet.

dnsmasq on cp answers *.stg.pl.test:

  • api, mgmt, releases and registry point at cp.
  • apps points at the app VM.
  • mesh points at the coordinators’ public alias on cp, 192.88.99.10.
  • cp.internal, registry.internal and vault.internal point at the tailnet address of cp.

TLS on the internal names comes from a staging operator CA that every VM trusts.

Reach a VM or the API by hand

  • SSH to a VM through pve1:

    ssh -J [email protected] -i ~/.config/pushleft-staging/id_ed25519 [email protected]
  • SSH on every VM admits only the operator device and the Proxmox host (10.20.1.1, the stand-in for the Robot console).

  • The API answers on cp at https://api.stg.pl.test, inside the bridge only.

  • To use pl on cp, run sudo -i, then:

    XDG_CONFIG_HOME=/root/pl-owner PUSHLEFT_API_URL=http://127.0.0.1:8081 pl list

After a reboot of cp, OpenBao unseals itself from /root/staging/bao-init.json. Production keeps the unseal shares offline (vault-recovery.md).

Public names over the tailnet

Staging also has real names in the Cloudflare zone pushleft.dev. Only Waseem’s personal tailnet reaches them.

NameServed byCertificate
https://console.stg.pushleft.devcp’s Caddy: the built console (npm run build), with /v1 sent to the API on the same origin*.stg.pushleft.dev
https://api.stg.pushleft.devcp’s Caddy, the API (for pl --api-url)*.stg.pushleft.dev
https://<branch>--<project>--<handle>.apps.stg.pushleft.devthe ingress Caddy on app*.apps.stg.pushleft.dev
https://admin.stg.pushleft.dev:8447pushleft-admin on cp, through pve1 (Fleet admin)*.stg.pushleft.dev
  • The apps base domain is the product setting PUSHLEFT_PREVIEW_BASE_DOMAIN=apps.stg.pushleft.dev (API and scheduler). The API has PUSHLEFT_CONSOLE_URL=https://console.stg.pushleft.dev.
  • The A records point at two Tailscale nodes tagged tag:staging: pl-stg-cp (api, console) and pl-stg-app (*.apps).
  • Each node is a Tailscale container on its VM, with userspace networking in its own network namespace. It forwards only TCP 80 and 443 to the VM’s Caddy (tailscale serve --tcp).
  • The VM’s own tailscaled (the management tailnet) does not change. SSH is not reachable from the tailnet.
  • Warning: do not run the container in the host’s network namespace. There, userspace tailscaled forwards every inbound tailnet connection to the host’s own ports. sshd and PostgreSQL answered.
  • Inside staging, dnsmasq answers the same names with the VM addresses. smoke checks the URLs from cp.
  • Certificates come from Let’s Encrypt, through DNS-01 with lego in Docker on the Mac. The Cloudflare token stays on the Mac.
  • The ingress uses the agent’s wildcard files (PUSHLEFT_INGRESS_WILDCARD_CERT_FILE, mode (a) in architecture.md). The agent loads a renewed file within 30 s.
  • certs renews a certificate with fewer than 30 days left and installs both. PUSHLEFT_STAGING_ACME=letsencrypt-staging uses the Let’s Encrypt test CA.
  • After the first join, the state of the tailnet nodes is in /var/lib/pl-public-ts on the VM.

To sign in to the console:

  1. On cp, get a login code:

    sudo -i env XDG_CONFIG_HOME=/root/pl-owner PUSHLEFT_API_URL=http://127.0.0.1:8081 pl console-code
  2. Sign in as [email protected] with the code.

Differences from production

  • Names and certificates. The public names point at tailnet addresses, so nothing is reachable from the internet. Custom domains (pl domain add) cannot work, because ACME HTTP-01 needs the internet to reach the ingress. Internal names use dnsmasq and a staging CA. pl on cp talks to the API through the loopback listener of cp’s Caddy. A Cloudflare API stub takes the DNS writes of the mesh worker.
  • Client addresses. Tailscale serve opens a new TCP connection to Caddy. The API sees the address of the Tailscale container for every user, so all users share the per-address rate limits.
  • Registry storage. zot runs as pushleft-registry.service from the shipped template, without its R2 storage driver. It stores on the disk of cp.
  • Control-plane PostgreSQL. Staging uses the manual mode: primary on cp, no replica (production has none either). The services on cp, the mesh controller among them, connect over loopback without TLS, without HAProxy and pg_auto_failover.
  • The coordinators’ public address. Every vmbr20 address is private, and gateways reach public addresses only. cp gets the alias 192.88.99.10, where its Caddy serves the coordinators on 9443 and nothing else answers. The other VMs route it to cp.
  • Build DNS. The VMs resolve through the dnsmasq of cp, which pl_build cannot reach. The build node sets PUSHLEFT_BUILD_DNS=1.1.1.1,9.9.9.9.
  • Self-update trust. The update client of the agent trusts only the built-in web roots. update adds PUSHLEFT_RELEASE_CA_FILE (the staging CA) to the agent.env of each node.
  • Telemetry. See Node telemetry.
  • Fleet admin. The operator device is a container on cp, so the Mac reaches the admin API through pve1. See Fleet admin.
  • Build cache. Since 2026-10-28, build112 has a dedicated 64 GiB XFS cache disk: 10 GiB per project, 32 GiB total budget, enforced project quotas. cp110 has a separate 64 GiB disk for cache snapshots and transfer temporary files. The gateway uses PUSHLEFT_CACHE_LOCAL_ROOT, and source and backup storage do not change. See staging-build-cache.md.
  • Not set up: the AI proxy, mail, error ingest, backups off the host, custom domains and ACME.

Node telemetry

Every VM sends host metrics to ClickHouse on cp, as in production (architecture.md).

  • Agent hosts (build, app, db). pushleft-otelcol.service (deploy/systemd) runs otelcol-contrib 0.162.0 as the pushleft-otel user. The binary is the upstream release tarball, with its version and checksum pinned in deploy/images.lock.env (PUSHLEFT_OTELCOL_VERSION, PUSHLEFT_OTELCOL_SHA256).
  • agent.env has PUSHLEFT_OTEL_GATEWAY=https://cp.internal.stg.pl.test:8445. On start, the agent renders /etc/pushleft-otel/otelcol.yaml from its node identity and starts the unit.
  • The collector sends OTLP/HTTP with a telemetry client certificate to the telemetry gateway on cp. The gateway sets host.id to the node’s fleet_nodes.id, and sets cloud.region.
  • cp. cp is not a fleet node. Its collector uses the same unit, with a configuration that the script writes. It has host.id=control-plane and sends straight to the central collector on loopback.
  • Central collector. pushleft-otelcol-gateway.service with otelcol-gateway.yaml listens on 127.0.0.1:14317/14318. It writes ClickHouse pushleft_otel (90-day TTL) and creates its own tables: otel_metrics_gauge, otel_metrics_sum, otel_metrics_histogram, otel_metrics_summary, otel_metrics_exponential_histogram, otel_logs, otel_traces.
  • ClickHouse users. pushleft_otel_ingest is the central collector, with create and insert on pushleft_otel. pushleft_admin_read has SELECT on pushleft_otel.*, for pushleft-admin. The passwords are ${PW}ch_otel_ingest and ${PW}ch_admin_read from env.sh.
  • Every 30 s: system.cpu.*, system.memory.usage, system.cpu.load_average.{1m,5m,15m}, system.filesystem.* (with the data volume), system.disk.*, system.network.*, system.paging.* and system.processes.*.
  • Every 60 s: one row per process (process.cpu.time, process.memory.usage), with process.pid, process.executable.name and process.command as resource attributes. Kernel threads are left out.
  • The collector has no ptrace rights. It reads the CPU, memory and command line of every process. The collector cannot read the executable path, I/O or file descriptors of processes of other users.

Staging-only parts, until the agent renders them:

  • The script issues the telemetry certificates: fleet CA, purpose='telemetry', one year, issued again with fewer than 30 days left. Production issues them by hand (fleet.md).
  • A drop-in (pushleft-otelcol.service.d/process.conf) adds /etc/pushleft-otel/process.yaml. It holds the per-process scraper and compression: none. The gateway refuses compressed requests with 415, and the collector does not retry them. The configuration of the agent does not turn compression off. Both parts are in otelcol-agent.yaml.
  • The drop-in also makes otelcol.yaml readable. The agent writes it under UMask=0077, so it is 0600 root and the user of the collector cannot open it.
  • /var/lib/pushleft is 0700 root, so the filesystem scraper cannot read the usage of the data volume. In the namespace of the collector, the drop-in puts an empty tmpfs there and binds in only the data volume, read-only. This also hides the keys of the agent from the collector.

Fleet admin

pushleft-admin (crates/pushleft-admin/README.md) runs on cp as in production. pushleft-admin.service binds to the management-tailnet address of cp (100.64.0.1:8447) and admits only the tailnet address of the operator device ($OP_TS/32).

The operator device is the pl-operator container on cp, and the Mac is on Waseem’s personal tailnet. So the path from the Mac goes through pve1 (decided 2026-10-24, staging only):

  1. The Mac opens https://admin.stg.pushleft.dev:8447. The A record is the personal-tailnet address of pve1.
  2. On pve1, pl-stg-admin-fwd.service (socat) forwards that port to cp 10.20.1.10:18447 over vmbr20.
  3. On cp, pl-admin-fwd.service (socat) accepts only from pve1 (10.20.1.1), through the nftables table pl_admin_fwd and the socat range.
  4. For each connection, it enters the network namespace of the pl-operator container and connects to 100.64.0.1:8447 over the management tailnet. It looks up the namespace per connection, so a restarted container needs nothing.
  5. The admin API sees the source $OP_TS. TLS is end to end: the API serves the *.stg.pushleft.dev certificate from a copy in /etc/pushleft/admin, which certs renews.

The Fleet app’s Connect screen defaults to production’s admin API. For staging, enter https://admin.stg.pushleft.dev:8447.

Other parts:

  • Environment. scripts/staging/admin.sh writes /etc/pushleft/admin.env. It sets the PostgreSQL roles pushleft_admin_read and pushleft_admin over loopback (password ${PW}<role>), ClickHouse pushleft_admin_read, and health on 127.0.0.1:9110. update creates roles that deploy/README.md added after the install.
  • Provisioning (decided 2026-10-24, staging only). provider: proxmox makes VMs on pve1 (fleet.md): VMIDs 150-199 and addresses 10.20.1.40-10.20.1.79 only, as full clones of stg-base in the staging pool. Hetzner provisioning is not configured (409).
  • admin.sh copies the staging token and host CA from the Mac. On cp, it creates a provisioner SSH key, a copy of the fleet CA and a management Headscale API key (90 days, replaced after 80). All are in /etc/pushleft/admin, for the service user.
  • The sandbox of the service needs no change. ssh runs under it and writes its known-hosts file to the private /tmp.
  • What a provisioned VM does not get, compared with build, app and db:
    • No node telemetry. telemetry covers the four roles only, so there is no otelcol, no PUSHLEFT_OTEL_GATEWAY and no telemetry certificate. The app shows no host metrics.
    • No PUSHLEFT_RELEASE_CA_FILE in agent.env. update does not reach it, and its self-update cannot verify the staging release mirror.
    • No route to the coordinators’ alias, 192.88.99.10.
    • No operator-only SSH (install-operator-ssh.sh).
  • When admitted, a provisioned VM is schedulable for customer workloads. Drain it if it is only a test.
  • Inside staging, dnsmasq answers admin.stg.pushleft.dev with the tailnet address of cp. Only the operator device gets past the address check. The ACL of the management tailnet refuses other nodes. cp itself gets 403.

To get an enrollment link for Waseem’s operator (ADMIN_OPERATOR_EMAIL in admin.sh):

  1. The first run of admin creates the operator and prints the link. The link is valid for 30 minutes.

  2. For a new link (expired link, lost passkey), run:

    scripts/staging/staging.sh ssh cp 'sudo bash -c "set -a; . /etc/pushleft/admin.env; set +a; pushleft-admin operator add --email <email> --name Waseem"'
  3. Open the link in a browser on the Mac.

First Proxmox provisioning run (2026-10-24)

Through the API, as the operator device, this request made VMID 150 at 10.20.1.40:

POST /v1/nodes/provision {"provider":"proxmox","pool":"application","region":"fsn1","server_type":"pve-2c-4g"}
  • Clone, configuration and start took 2 min. Boot, bootstrap (disks, tailnet, agent) and enrollment took 1.5 min. The total was 3 min 40 s.
  • XFS prjquota was enforced on the data disk, and Docker ran with userns-remap.
  • The node joined the management tailnet as node. admit took its capacity from the host facts of the agent.
  • Drain and decommission then deleted the VM with its disks and removed its tailnet node, in 6 s.

Two defects were found and fixed:

  • tailscaled ran from boot, before the bootstrap installed the staging CA. It never trusted the login server, and tailscale up retried forever. The bootstrap now restarts it first, and the join times out after 120 s.
  • Decommission left the tailnet node of the VM registered.

Open proofs

The Lima harness ran these until 2026-10-23. Staging does not run them yet, and no other suite does:

  • Automatic control-plane PostgreSQL failover: pg_auto_failover on three VMs, with failover under writes, fencing, rejoin, switchover and off-host restore. Staging runs the manual mode. npm run test:control-plane-ha covers the manual mode on Docker only. Last results: control-plane-postgres-failover.md.

  • Management tailnet checks on the hosts:

    • No node reaches the tailnet address of another node.
    • The agent ports, registry, OpenBao and PostgreSQL of cp are closed on its public address.
    • PostgreSQL answers no other host.
    • Admin SSH works only from operator devices.

    isolation checks the tenant containers only (N4).

  • A scripted cp reboot: PostgreSQL on the tailnet address, with the services reconnecting and nothing restarted. The cp reboot on 2026-10-21 was a one-off that no command runs.

  • arm64 hosts. Staging is amd64, like the Robot hosts.

Results (2026-10-21)

First build of the fleet:

  • About 21 minutes for the template and six VMs.
  • 7 minutes for the first release build on forge.
  • 5 minutes for the five install phases.

smoke then passed. web-only built with Kata and was live in 77 s. db-app was live in 205 s, with its PostgreSQL on the db VM. A second run of vms and of every install phase broke nothing.

After a reboot of cp, OpenBao unsealed itself. The API, gateway and controller were ready again within 2 minutes, with no manual step. The apps continued to serve, and the replica returned to quorum.

The binaries, the pinned images, Caddy, OpenBao, zot and Kata all run on the Sandy Bridge CPUs of the host (x86-64-v2, no AVX2).

Public names, the same day, from the Mac over the tailnet:

  • The console signed in with a login code in Chrome (session cookie, /v1 on the console origin).
  • api.stg.pushleft.dev/readyz answered 200.
  • Both smoke apps served under *.apps.stg.pushleft.dev with Let’s Encrypt certificates.
  • Ports 22, 5432, 8443 and 2019 on both tailnet nodes are closed.

Fixed: Kata build network

A build container asks for 12 GiB. In the nested guest, onlining that hotplugged memory takes more than 3 s. The Kata agent waits only 3 s for the NIC that QEMU hotplugs next. The agent gives up, and the guest runs with eth0 down. BuildKit then fails with “network is unreachable”.

  • Staging sets agent.hotplug_timeout=60 in the Kata kernel parameters (/etc/kata-containers/configuration.toml).
  • With the timeout, the NIC still comes up about 3 s after the build process starts. So the Kata entrypoint of the agent now waits up to 30 s for a default route before it starts BuildKit.
  • Without the timeout, 0 of 4 test runs got a network. With both changes, 4 of 4 did.
  • It is not known whether bare-metal Robot build hosts need the timeout. They boot the guest faster.

See build-executor.md.

Rendered from the repository at build time. Edit docs/runbooks/staging.md to change it.