Control-plane host failover
Use this runbook when the host that runs the control-plane services is lost and service_not_ready fires for every service.
All control-plane services run on one host, box 1 (pl-cp). They are the API, scheduler, agent gateway, controller, mesh controller, telemetry gateway, error ingest, the central collector, the host collector and the fleet admin API. They also include the registry (zot), OpenBao, the DERP relay, the management Headscale, ops-alerts and the control-plane Caddy in front of them. pl-cp is also the operators’ SSH jump host. This runbook moves them to a second, prepared host. The failover is active/passive and manual: you stop or power off host A, start host B and move the addresses. The database is not on A: it runs on box 2 (control-plane PostgreSQL restore).
Warning: nothing in this runbook has run on Robot hardware. The steps that move addresses are untested. There is no host B in production today.
Design
| Piece | Where it lives | Why the services can move |
|---|---|---|
Binaries, units, /etc/pushleft/*.env | Installed on A and B; enabled on A only | Install both from the same signed release. Copy the env files from A each time they change. |
| Control-plane database | Box 2, at PUSHLEFT_DB_HOST (db1.internal.pushleft.dev) over the management tailnet. With the HA design: two PostgreSQL nodes + a witness, behind HAProxy on each control-plane host (db.internal.pushleft.dev → 127.0.0.2:5432) | DATABASE_URL is the same on A and B. Box 2’s pg_hba and firewall admit only the cluster clients, so B’s tailnet address must be one of them. |
| Source objects (uploads waiting for a build) | PUSHLEFT_SOURCE_S3_BUCKET (object store) | With the S3 store, the API, gateway and scheduler use the same bucket from either host. With the local store (PUSHLEFT_SOURCE_ROOT), the sources on A are lost with A: queued builds fail and the user runs pl up again. |
Gateway CA and certificates (/etc/pushleft/pki) | Copied to B (0600, same owners) | Agents trust the CA and dial agent-gateway.internal.pushleft.dev. B serves the same certificate. |
| Proxy config and certificates | pushleft-cp-caddy with /etc/pushleft/caddy.json and /etc/pushleft/caddy/; copied to B, or B gets its own ACME certificates | The public names do not change. |
Internal names (cp.internal.<base>, admin.internal.<base>) | DNS A records for A’s management tailnet address (100.64.0.1) | Nodes dial the gateway, enrollment, registry, OpenBao and telemetry through cp.internal. Point the records at B’s tailnet address. |
| Management tailnet coordinator | pushleft-mgmt-headscale on A, state in /var/lib/pushleft/mgmt-headscale | Joined hosts keep their connections while it is down, but no host can join or change. Copy its state to B. |
| Fleet admin API | pushleft-admin on A’s tailnet address, port 8447 (PUSHLEFT_ADMIN_BIND in admin.env) | Bound to A’s address: run install-control-plane.sh access on B to bind B’s. |
| Public API name | A Hetzner failover IP, or a DNS record with a short TTL | You switch the failover IP to B through the Robot API, or change the record. |
| Vault/OpenBao, registry | Their own recovery (Vault recovery, registry outage) | Single instances on A. OpenBao keeps its data in /var/lib/openbao: restore it on B from a snapshot. zot keeps images in its R2 bucket, so the registry survives the loss of A. |
Fencing. Never run two schedulers, gateways or controllers at the same time. Job leases carry fencing generations, so a second scheduler probably only wastes work, but two of each have never been tested. Stop the services on A, or power A off, before you start them on B.
Before you start
- Root access to A (if it answers), B and the database host. With A down,
pl-cpis no jump host. Reach B and box 2 through the Robot console or rescue system. Or use an operator address that their sshd and Robot firewall admit. - Robot console or API access, for the power-off and the failover IP.
- B’s tailnet address as a client of box 2:
install-db-node.sh cluster <db bundle> <A>/32 <B>/32(or, with the HA design, inPUSHLEFT_PG_CLIENT_HOSTS).
Prepare B
Do this once, and again after every release, so B never runs an older release than A.
-
On B, install the same release and units as A, and disable the services:
systemctl disable --now pushleft-api pushleft-scheduler pushleft-agent-gateway pushleft-controller pushleft-mesh \ pushleft-telemetry-gateway pushleft-errors pushleft-otelcol-gateway pushleft-otelcol pushleft-admin pushleft-registry \ pushleft-openbao pushleft-derper pushleft-cp-caddy pushleft-mgmt-headscale pushleft-ops-alerts.timer -
From A, copy the env files, the PKI, the proxy config and the Headscale state over the management tailnet. Keep owners and modes:
rsync -a /etc/pushleft/ root@B:/etc/pushleft/ # env files, pki, caddy.json, caddy/, admin/ rsync -a /var/lib/pushleft/mgmt-headscale/ root@B:/var/lib/pushleft/mgmt-headscale/ # while Headscale is stoppedB must already be on the management tailnet as
cp(install-control-plane.sh mgmt-key cpon A), so the ACL lets it reach box 2 and the nodes. -
Only with the HA design: on B, add the database endpoint (
haproxy-pushleft-pg.cfg) and the hosts line, then start HAProxy:grep -q db.internal.pushleft.dev /etc/hosts || echo '127.0.0.2 db.internal.pushleft.dev' >>/etc/hosts systemctl enable --now haproxy -
On B, check that the endpoint reaches the writable primary:
set -a; . /etc/pushleft/controller.env; set +a; pushleft-controller fleet db-checkExpect
primary: DATABASE_URL reaches the writable primary. On a replica the command printsreplica: ...and exits non-zero. Do not continue: fix the endpoint first. With the HA design, HAProxy sends the endpoint only to the writable primary.
Fail over
-
Fence A. If A answers, stop and disable the units from Prepare B step 1.
If A does not answer, power it off (Robot: reset, “power off”). Do not continue while A can still run the services.
-
Point
cp.internal.<base>andadmin.internal.<base>at B’s management tailnet address (tailscale ip -4on B). Do not proxy them. -
Start the services and the proxy on B.
startwaits until each service answers ready;accessloads B’s firewall and SSH rules and binds the admin API to B’s address:systemctl enable --now pushleft-mgmt-headscale pushleft-openbao pushleft-cp-caddy pushleft-derper install-control-plane.sh start install-control-plane.sh access systemctl enable --now pushleft-ops-alerts.timer -
Check readiness. Expect 200 and
"in_recovery": false, then check the health listeners on 9102–9106 and the admin API on 9110:curl -fsS http://127.0.0.1:8080/readyz | jq .database -
Move the public name. Switch the failover IP to B (Robot API
POST /failover/<ip>withactive_server_ip=<B>), or change the API record.
Fail back the same way when A is repaired, at a quiet time.
Check it worked
pl whoamiworks from outside.- Agents reconnect on their own: their next lease request reaches the gateway on B through
cp.internal.<base>, once their resolvers see the new record.node_heartbeat_staleclears.
Limits
- Active/active. One instance of each service only. The API on both hosts behind a shared edge needs the S3 source store (done) and a load balancer (not built). The enrollment rate limiter is in memory per gateway process. Two schedulers, controllers or mesh workers have not been tested.
- Automatic host failover. Nothing watches A and moves the services. Production has no database failover either, because the HA design is not deployed.
- Robot specifics. Failover IP switching through the Robot API comes from Hetzner’s documentation. Nobody has tried it yet.
- Headscale and OpenBao on B. Moving the management Headscale’s state and starting OpenBao (KMS auto-unseal, same config) on B have not been tried. If OpenBao’s data did not come across, restore it on B (Vault recovery).