pushleft
Deploy your first app
Manual contents
Handbook
HandbookPushleftArchitecture
Operate
Install the control planeYour own serversFleetOperations and alertsAuthenticationDomains and DNSUsage and budgetsScheduled jobsAbuse controlsAI provider proxyCustomer mailErrors and performanceLogs and metricsRetentionObject storageContainer registryGit server
Develop
The .pl.yaml manifestOnboard with an AI agentThe pl up demoBuild executorSource ingestionAgent gatewayManaged databasesSecurity findingsTunnels and shellsDev workspacesPrivate hostnamesMesh coordinatorsDesign baseline
Runbooks
Control-plane host failoverControl-plane PostgreSQL failoverControl-plane PostgreSQL restoreCustom domains and certificatesGateway key compromiseIngress host failureNode busyNode disk fullNode drain and replacementNode lostRegistry outageHetzner Robot hostsSecret rotationSecurity incidentStaging fleet on ProxmoxStaging build cacheStuck deploymentVault/OpenBao backup and recovery
Manual · docs/runbooks/control-plane-host-failover.md

Control-plane host failover

Use this runbook when the host that runs the control-plane services is lost and service_not_ready fires for every service.

All control-plane services run on one host, box 1 (pl-cp). They are the API, scheduler, agent gateway, controller, mesh controller, telemetry gateway, error ingest, the central collector, the host collector and the fleet admin API. They also include the registry (zot), OpenBao, the DERP relay, the management Headscale, ops-alerts and the control-plane Caddy in front of them. pl-cp is also the operators’ SSH jump host. This runbook moves them to a second, prepared host. The failover is active/passive and manual: you stop or power off host A, start host B and move the addresses. The database is not on A: it runs on box 2 (control-plane PostgreSQL restore).

Warning: nothing in this runbook has run on Robot hardware. The steps that move addresses are untested. There is no host B in production today.

Design

PieceWhere it livesWhy the services can move
Binaries, units, /etc/pushleft/*.envInstalled on A and B; enabled on A onlyInstall both from the same signed release. Copy the env files from A each time they change.
Control-plane databaseBox 2, at PUSHLEFT_DB_HOST (db1.internal.pushleft.dev) over the management tailnet. With the HA design: two PostgreSQL nodes + a witness, behind HAProxy on each control-plane host (db.internal.pushleft.dev → 127.0.0.2:5432)DATABASE_URL is the same on A and B. Box 2’s pg_hba and firewall admit only the cluster clients, so B’s tailnet address must be one of them.
Source objects (uploads waiting for a build)PUSHLEFT_SOURCE_S3_BUCKET (object store)With the S3 store, the API, gateway and scheduler use the same bucket from either host. With the local store (PUSHLEFT_SOURCE_ROOT), the sources on A are lost with A: queued builds fail and the user runs pl up again.
Gateway CA and certificates (/etc/pushleft/pki)Copied to B (0600, same owners)Agents trust the CA and dial agent-gateway.internal.pushleft.dev. B serves the same certificate.
Proxy config and certificatespushleft-cp-caddy with /etc/pushleft/caddy.json and /etc/pushleft/caddy/; copied to B, or B gets its own ACME certificatesThe public names do not change.
Internal names (cp.internal.<base>, admin.internal.<base>)DNS A records for A’s management tailnet address (100.64.0.1)Nodes dial the gateway, enrollment, registry, OpenBao and telemetry through cp.internal. Point the records at B’s tailnet address.
Management tailnet coordinatorpushleft-mgmt-headscale on A, state in /var/lib/pushleft/mgmt-headscaleJoined hosts keep their connections while it is down, but no host can join or change. Copy its state to B.
Fleet admin APIpushleft-admin on A’s tailnet address, port 8447 (PUSHLEFT_ADMIN_BIND in admin.env)Bound to A’s address: run install-control-plane.sh access on B to bind B’s.
Public API nameA Hetzner failover IP, or a DNS record with a short TTLYou switch the failover IP to B through the Robot API, or change the record.
Vault/OpenBao, registryTheir own recovery (Vault recovery, registry outage)Single instances on A. OpenBao keeps its data in /var/lib/openbao: restore it on B from a snapshot. zot keeps images in its R2 bucket, so the registry survives the loss of A.

Fencing. Never run two schedulers, gateways or controllers at the same time. Job leases carry fencing generations, so a second scheduler probably only wastes work, but two of each have never been tested. Stop the services on A, or power A off, before you start them on B.

Before you start

  • Root access to A (if it answers), B and the database host. With A down, pl-cp is no jump host. Reach B and box 2 through the Robot console or rescue system. Or use an operator address that their sshd and Robot firewall admit.
  • Robot console or API access, for the power-off and the failover IP.
  • B’s tailnet address as a client of box 2: install-db-node.sh cluster <db bundle> <A>/32 <B>/32 (or, with the HA design, in PUSHLEFT_PG_CLIENT_HOSTS).

Prepare B

Do this once, and again after every release, so B never runs an older release than A.

  1. On B, install the same release and units as A, and disable the services:

    systemctl disable --now pushleft-api pushleft-scheduler pushleft-agent-gateway pushleft-controller pushleft-mesh \
      pushleft-telemetry-gateway pushleft-errors pushleft-otelcol-gateway pushleft-otelcol pushleft-admin pushleft-registry \
      pushleft-openbao pushleft-derper pushleft-cp-caddy pushleft-mgmt-headscale pushleft-ops-alerts.timer
  2. From A, copy the env files, the PKI, the proxy config and the Headscale state over the management tailnet. Keep owners and modes:

    rsync -a /etc/pushleft/ root@B:/etc/pushleft/                                          # env files, pki, caddy.json, caddy/, admin/
    rsync -a /var/lib/pushleft/mgmt-headscale/ root@B:/var/lib/pushleft/mgmt-headscale/   # while Headscale is stopped

    B must already be on the management tailnet as cp (install-control-plane.sh mgmt-key cp on A), so the ACL lets it reach box 2 and the nodes.

  3. Only with the HA design: on B, add the database endpoint (haproxy-pushleft-pg.cfg) and the hosts line, then start HAProxy:

    grep -q db.internal.pushleft.dev /etc/hosts || echo '127.0.0.2 db.internal.pushleft.dev' >>/etc/hosts
    systemctl enable --now haproxy
  4. On B, check that the endpoint reaches the writable primary:

    set -a; . /etc/pushleft/controller.env; set +a; pushleft-controller fleet db-check

    Expect primary: DATABASE_URL reaches the writable primary. On a replica the command prints replica: ... and exits non-zero. Do not continue: fix the endpoint first. With the HA design, HAProxy sends the endpoint only to the writable primary.

Fail over

  1. Fence A. If A answers, stop and disable the units from Prepare B step 1.

    If A does not answer, power it off (Robot: reset, “power off”). Do not continue while A can still run the services.

  2. Point cp.internal.<base> and admin.internal.<base> at B’s management tailnet address (tailscale ip -4 on B). Do not proxy them.

  3. Start the services and the proxy on B. start waits until each service answers ready; access loads B’s firewall and SSH rules and binds the admin API to B’s address:

    systemctl enable --now pushleft-mgmt-headscale pushleft-openbao pushleft-cp-caddy pushleft-derper
    install-control-plane.sh start
    install-control-plane.sh access
    systemctl enable --now pushleft-ops-alerts.timer
  4. Check readiness. Expect 200 and "in_recovery": false, then check the health listeners on 9102–9106 and the admin API on 9110:

    curl -fsS http://127.0.0.1:8080/readyz | jq .database
  5. Move the public name. Switch the failover IP to B (Robot API POST /failover/<ip> with active_server_ip=<B>), or change the API record.

Fail back the same way when A is repaired, at a quiet time.

Check it worked

  • pl whoami works from outside.
  • Agents reconnect on their own: their next lease request reaches the gateway on B through cp.internal.<base>, once their resolvers see the new record. node_heartbeat_stale clears.

Limits

  • Active/active. One instance of each service only. The API on both hosts behind a shared edge needs the S3 source store (done) and a load balancer (not built). The enrollment rate limiter is in memory per gateway process. Two schedulers, controllers or mesh workers have not been tested.
  • Automatic host failover. Nothing watches A and moves the services. Production has no database failover either, because the HA design is not deployed.
  • Robot specifics. Failover IP switching through the Robot API comes from Hetzner’s documentation. Nobody has tried it yet.
  • Headscale and OpenBao on B. Moving the management Headscale’s state and starting OpenBao (KMS auto-unseal, same config) on B have not been tried. If OpenBao’s data did not come across, restore it on B (Vault recovery).
Rendered from the repository at build time. Edit docs/runbooks/control-plane-host-failover.md to change it.