Install the control plane
This page shows you how to install the Pushleft control plane on servers. It covers which hosts run which units, the ports, service users, database roles, migrations and the control-plane PostgreSQL. It describes the layout the code supports today. Each statement was checked against the code, and open items are marked gap.
Operating Pushleft covers SLOs, alerts and recovery, and the runbooks cover incidents. Nodes are provisioned as described in the fleet substrate.
Production runs from these files on six Hetzner Robot servers since 2026-11-03: the control-plane host pl-cp, pl-build-1, pl-app-1 to pl-app-3 and pl-database-1 (Robot hosts runbook). No Hetzner Cloud fleet has run yet. The staging fleet is five Ubuntu 24.04 amd64 VMs on the Proxmox lab host. Staging installs these units, env examples, role grants, postgres/ (Debian layout, manual mode) and vault/ policies under systemd. The staging runbook lists where it differs from production. A Lima harness on the Mac ran the same files from 2026-10-20 to 2026-10-23.
Topology
| Host | Units | Notes |
|---|---|---|
| Control-plane host (one) | pushleft-api, pushleft-scheduler, pushleft-agent-gateway, pushleft-controller, pushleft-mesh (the mesh controller), pushleft-telemetry-gateway, pushleft-otelcol-gateway, pushleft-ops-alerts.timer, pushleft-admin (the fleet admin API, below), pushleft-derper (the environment meshes’ DERP relay, mesh coordinators), pushleft-host-health.timer (drive and RAID health for the Fleet app), plus a TLS-terminating proxy (for example Caddy) in front of the API, pushleft-mgmt-headscale (the management tailnet’s coordinator, behind the same proxy as mgmt.<base>) and pushleft-private-firewall (the private ports answer only on the tailnet; runbooks/robot-hosts.md) | API, scheduler and gateway must share one host and one service user. The source store is a local 0700 directory: the API writes it, the gateway reads it and the scheduler’s retention deletes from it (crates/pushleft-source, private_directory). With PUSHLEFT_SOURCE_S3_* or the shared PUSHLEFT_BLOB_S3_* bucket, the source store is an S3 bucket instead (S3-compatible source store). Running the three on separate hosts is not tested (gap). Run one instance of each service. Multiple schedulers, controllers or gateways are not tested. |
| Control-plane PostgreSQL (two, PostgreSQL 17+) | the distribution’s PostgreSQL unit, plus pushleft-postgres-basebackup.timer on the primary | Primary + synchronous streaming replica, WAL archive and PITR from deploy/postgres (below). Manual mode has no automatic promotion (see disaster recovery). Gateway transactions require PostgreSQL 17 or newer (agent-gateway.md). |
Build pool (pool=build) | pushleft-agent (with --buildkit-image and --build-runtime kata), pushleft-otelcol, Docker, Kata Containers | One Kata VM per build with rootful BuildKit inside, with egress to the public internet only (build executor). It runs on Robot hosts only (KVM). It has no tenant runtime, databases or Vault address. |
Application pool (pool=application) | pushleft-agent, pushleft-otelcol, Docker, host Caddy on 80/443 | The node is also the ingress host. The agent writes routes into the Caddy on the node that runs the web instances. Run Caddy with systemd/pushleft-ingress-caddy.service (--resume), because routes exist only in its running config. With one application node, the operator’s wildcard *.<base> record reaches it. With more (built 2026-11-03), the controller gives each platform hostname its own A record to the node holding its route (PUSHLEFT_PLATFORM_DNS_ZONE_ID, controller.env.example). A linked domain’s tunnel CNAME follows its route. On a move between live nodes, the old node serves until the hostname’s DNS has pointed at the new one for two minutes (domain registration). Roles that share a hostname are placed on one node, and a web role prefers the node already routing its environment. |
Database pool (pool=database) | pushleft-agent, pushleft-otelcol, Docker, XFS data volume with project quotas; on one database host also pushleft-garage and pushleft-garage-mirror.timer | Managed databases: PostgreSQL, ClickHouse and Valkey containers. The platform’s object storage is one Garage node on pl-database-1 (since 2026-11-04). Every night the mirror timer copies every bucket, encrypted, to its own R2 bucket and keeps 5 nights. |
| AI proxy (the control-plane host, or a host of its own) | pushleft-ai-proxy, behind a TLS-terminating proxy as https://ai.<base> | A public endpoint, because application containers can reach only public addresses (AI proxy networking). It keeps no state: limits and usage live in PostgreSQL, so replicas share them (one replica tested). It holds customers’ provider keys in memory. So it runs as its own service user, with a read-only Vault policy (vault/pushleft-ai-proxy.hcl) and a database role that can only call its three functions. |
| Mail host (a Robot server of its own) | pushleft-mail, behind a TLS-terminating proxy as https://mail.<base>; port 25 in (bounce receiver) and out (direct delivery) | Customer mail: the send API, the delivery workers on the host’s own IPv4 with matching PTR, ESP fallback and the bounce and feedback-loop receiver. It gets a host of its own, because a provider can revoke its IP reputation and its port 25 after complaints. Nothing else is affected then (Robot mail host). It holds DKIM private keys and the ESP relay credentials in memory. So it has its own service user, a read-only Vault policy (vault/pushleft-mail.hcl) and a database role that can only call the mail_* functions. It has not run on a Robot host yet (gap). |
| Error ingest (the control-plane host, or a host of its own) | pushleft-errors, behind a TLS-terminating proxy as https://errors.<base> | Sentry-compatible error ingest. It is public, because browsers and application containers send to it. It keeps no state. Limits, issues and metering are in PostgreSQL. Events and spans are in ClickHouse (pushleft_otel, schema observability/clickhouse/002_errors.sql). Because it parses untrusted payloads, it runs as its own service user. Its database role can only call its two functions, and its ClickHouse user can only insert. |
| Registry (the control-plane host) | pushleft-registry (zot in Docker, port 5000), behind the TLS proxy as the public registry host | zot on its own R2 bucket (registry). It runs next to the gateway. The gateway issues every registry token, serves the realm and renders zot’s config (/var/lib/pushleft/zot-rendered: per-org pull-through upstreams, per-repository retention, upstream passwords from Vault). The TLS proxy sends /v1/registry/token to the gateway’s realm (8446) and everything else to zot. |
| Git server (the control-plane host) | the host’s OpenSSH on public port 22 with git/sshd_config.d/pushleft-git.conf, pushleft-git-shell, pushleft-git-firewall, pushleft-git-backup.timer; set up by git/install-git-host.sh | Private repositories and push-to-deploy (git server), in the GitLab pattern. For the git user, sshd’s AuthorizedKeysCommand asks the API’s internal hook listener (127.0.0.1:8091) for the presented key. pushleft-git-shell (the key’s forced command and git’s login shell) asks it for access before it runs git on the bare repository. The hooks call the same listener. No git daemon runs. Operators log in on the same port as their own users, from operator addresses only. nftables rate-limits and bans abusive sources. Operator addresses are exempt. Hourly sealed bundles go to the backups bucket. |
| External | Vault/OpenBao, ClickHouse for telemetry, Cloudflare, Openprovider, Hetzner, R2 | Not deployed by these units. |
The fleet substrate describes the node units (pushleft-agent*.service, pushleft-otelcol.service) and their provisioning.
Ports and flows
| Listener | Default in the examples | Who connects | Auth |
|---|---|---|---|
API PUSHLEFT_BIND_ADDR | 127.0.0.1:8080 | Only the local TLS proxy (public 443 → here). CLI and console reach it through the proxy. | Production tokens/sessions; the API refuses requests that do not come from PUSHLEFT_TRUSTED_PROXIES with X-Forwarded-Proto: https (authentication.md) |
AI proxy PUSHLEFT_AI_PROXY_BIND | 127.0.0.1:8090 | Only the local TLS proxy (public 443 for ai.<base> → here). Applications and SDKs reach it through the proxy | Per-environment plai_ tokens (ai-proxy.md) |
Mail API PUSHLEFT_MAIL_BIND | 127.0.0.1:8092 | Only the local TLS proxy (public 443 for mail.<base> → here). Applications send through the proxy; ESPs post events to /v1/esp/events | Per-environment plm_ tokens; the ESP webhook a shared secret (mail.md) |
Error ingest PUSHLEFT_ERRORS_BIND | 127.0.0.1:8093 | Only the local TLS proxy (public 443 for errors.<base> → here). Sentry SDKs in browsers and applications post through the proxy | DSN public key (X-Sentry-Auth or sentry_key), bound to one environment’s project id (errors.md) |
Mail bounce receiver PUSHLEFT_MAIL_INBOUND_BIND | 0.0.0.0:25 on the mail host | Any MTA on the internet (the MX of the bounce domain) | none (receive-only: VERP and fbl@ recipients, everything else 550) |
Agent gateway PUSHLEFT_AGENT_GATEWAY_BIND | 0.0.0.0:8443, private network only | Agents (outbound from nodes) | mTLS, client certificate must be an unrevoked fleet_node_certificates row |
Enrollment PUSHLEFT_AGENT_ENROLL_BIND | 0.0.0.0:8444, provider network only | New nodes, once | Server TLS + one-time secret; token bucket per source address (10, then 1 per 6 s) and for the listener (60, then 1 per s), 429 beyond. In memory per gateway process |
Registry token realm PUSHLEFT_REGISTRY_TOKEN_BIND | 0.0.0.0:8446; public only through the TLS proxy at https://<registry host>/v1/registry/token | The scheduler (retention deletes), people and CI (docker login, username token); zot names it as auth.bearer.realm | Server TLS (gateway certificate) + HTTP Basic: the scheduler secret (delete on released builds) or a plr_ registry token (its patterns and actions, own organization only). Nodes get their tokens over 8443 instead |
Session relay PUSHLEFT_SESSION_RELAY_BIND | 127.0.0.1:8449 (loopback only; the gateway refuses another address); public only through the TLS proxy at https://api.<base>/v1/sessions/{id}/relay | pl db connect, pl tunnel, pl exec (developer access) | The session’s one-time token from POST /v1/sessions (Admin, user credential): stored as SHA-256, once, within two minutes. The agent side of the same relay is on 8443 (mTLS + lease) |
| Registry (zot) | 0.0.0.0:5000 (TLS, operator CA); public through the TLS proxy | Nodes (pull/push with gateway tokens), people and CI | Bearer tokens from the gateway only; anonymous requests get 401 |
Telemetry gateway PUSHLEFT_TELEMETRY_GATEWAY_BIND | 0.0.0.0:8445, private network only | Node collectors | mTLS, telemetry-purpose certificate |
| Central collector OTLP/HTTP | 127.0.0.1:14318 | Telemetry gateway only | loopback |
| SSH (the host’s OpenSSH) | 0.0.0.0:22 on the control-plane host | Anyone as git (git push/clone); operators as their own users | git: SSH keys that Pushleft users registered (AuthorizedKeysCommand), forced pushleft-git-shell, access checked by the API per request and again in pre-receive (git.md). Others (root included) log in with keys only, and only from the operator addresses (the management tailnet, Match block in sshd). sshd checks the address, because the Hetzner firewall must leave 22 open for git. Passwords are off. Other hosts keep 22 closed publicly. nftables: 30 new connections per minute per source, then a 15-minute ban; 10 concurrent; operator addresses exempt |
Git hook listener PUSHLEFT_GIT_HOOK_BIND | 127.0.0.1:8091 | pushleft-git-shell (key lookup, access) and the pre-receive and post-receive hooks only; never behind the public proxy | Loopback or private peers only, plus the shared secret in PUSHLEFT_GIT_HOOK_SECRET_FILE |
Fleet admin API PUSHLEFT_ADMIN_BIND | On first install, install-control-plane.sh access writes the host’s management tailnet address, e.g. 100.64.0.1:8447, and admits PUSHLEFT_OPERATORS (refused unless loopback or private). A rerun keeps the bind and operator CIDRs already in admin.env. On production’s pl-cp both are 127.0.0.1:8447 and 127.0.0.1/32,::1/128. Operators reach it as admin.internal.<base>:8447 through pl-cp’s address in the operator’s tailnet (Robot hosts) | The Fleet app on operator devices, from PUSHLEFT_ADMIN_OPERATOR_CIDRS only | Operator passkey sessions (crates/pushleft-admin/README.md) |
Health PUSHLEFT_HEALTH_BIND | 127.0.0.1:9102 gateway, 9103 scheduler, 9104 controller, 9105 mesh, 9106 telemetry gateway, 9110 fleet admin API | Local monitoring, pushleft-ops-alerts | none; bind to loopback |
| Management Headscale | 127.0.0.1:8085, public as https://mgmt.<base> through the control-plane proxy | Every Pushleft host and operator device (Tailscale) | One-use pre-auth keys per Headscale user; headscale/mgmt-acl.hujson. “Private network only” in this table means this tailnet (no vSwitch). |
| Node Caddy | 80/443 public on application nodes; admin on loopback | Internet | TLS per architecture.md “Ingress publication” |
Outbound connections:
- From the control-plane host: PostgreSQL (all services) and Vault (the gateway reads wrapped secrets and creates backup keys). The registry: scheduler retention deletes manifests with a delete token from the gateway’s realm. Cloudflare and Openprovider: the controller, and the API for domain quotes. ClickHouse: the central collector and the analytics sink.
- From the control-plane host to nodes: over the management tailnet, the mesh controller dials each node’s coordinator host API (8448). The control-plane Caddy dials the coordinators it serves (18080-19079) (mesh coordinators).
- From nodes, which otherwise only dial out: the gateway, the telemetry gateway, the registry, Vault (unwrap only, no token) and Headscale. The registry takes per-operation tokens from the gateway, and no node has a registry login.
Service users and files
Application nodes also need br_netfilter loaded, with net.bridge.bridge-nf-call-iptables=1 persisted (/etc/modules-load.d/pushleft.conf, /etc/sysctl.d/90-pushleft.conf). pushleft-agent manage-firewall refuses to run without it. Cloud-init does this on Hetzner nodes (fleet firewall).
| Unit | User | Writable paths | Secret files it reads |
|---|---|---|---|
pushleft-api, pushleft-scheduler, pushleft-agent-gateway | pushleft (shared, see Topology) | /var/lib/pushleft/sources (API creates it via StateDirectory; scheduler deletes; gateway read-only); gateway: /var/lib/pushleft/zot-rendered | gateway: server key, agent CA key, registry token signing key and scheduler realm secret, Vault token (policy vault/pushleft-agent-gateway.hcl); scheduler: registry realm secret |
pushleft-controller | pushleft-controller | none | Vault token (policy vault/pushleft-controller.hcl: read on organizations/data/+/cloudflare/connections/*, create on organizations/data/+/custom-domain-certificates/*), for custom domains linked to a Cloudflare connection; the platform’s own Cloudflare token is an environment value |
pushleft-ai-proxy | pushleft-ai | none | Vault token (policy vault/pushleft-ai-proxy.hcl: read on organizations/data/+/ai/* only). The API’s optional Vault token for storing those keys carries vault/pushleft-api.hcl (create only, same paths) |
pushleft-mail | pushleft-mail (CAP_NET_BIND_SERVICE only, for port 25) | none | Vault token (policy vault/pushleft-mail.hcl: read on organizations/data/+/mail/dkim/* and platform/data/mail/*), ESP webhook secret. The API’s Vault token creates the DKIM keys (vault/pushleft-api.hcl, create only) |
pushleft-errors | pushleft-errors | none | none (database and ClickHouse passwords are environment values) |
pushleft-telemetry-gateway | pushleft-telemetry | none | server key, analytics hash key |
pushleft-mesh (control-plane host) | pushleft-mesh, no capabilities | none | Cloudflare token for the mesh zone (no Vault token since 2026-11-03) |
pushleft-ops-alerts | pushleft-ops | none | none |
pushleft-registry | zot container run as pushleft (unit starts Docker as root) | /var/lib/pushleft/zot (metadata, pull-through downloads) | rendered config with the R2 keys and sync-credentials.json (upstream passwords), both written 0600 by the gateway; registry TLS key |
pushleft-git-shell (run by sshd), pushleft-git-backup | git (system user, login shell pushleft-git-shell, no capabilities); pushleft is in the git group to read the hook secret | /var/lib/pushleft-git (bare repositories, 0700; read-only for the backup) | hook secret (/etc/pushleft/git-hook.secret, 0640 root:git; /etc/pushleft must be traversable); backup: the API’s database login, backup bucket key, Vault token (policy vault/pushleft-git-backup.hcl: read and create organization backup keys), under /etc/pushleft/git-backup/ owned by git |
Nodes (pushleft-agent) hold no Vault token. Secrets reach them through the gateway (brokered secrets):
- The gateway reads the secrets that each leased job names, as single-use response-wrapping tokens with a 5-minute TTL. The secrets are database admin and principal passwords, the node’s own mesh pre-auth key, and for an application role its environment secrets and database login passwords. It reads only under the lease’s own organization and environment.
- It puts them in the lease response only. The agent unwraps them with its
PUSHLEFT_VAULT_ADDRand no token. - Backup keys come from
POST /v1/agent/secrets/backup-key. It answers only for an organization with a database, a volume or a leased database or volume job on that node. - The gateway stores the enrollment keys a coordinator host’s Headscale mints (
createandreadonorganizations/data/+/environments/+/mesh/*), only for segments that host coordinates (mesh coordinators). - The gateway creates missing backup keys, database admin passwords and the passwords of database logins declared in
.pl.yaml(CAS 0, neverupdate). - The API’s token writes customer environment secrets (
create,update, metadatadelete) and can never read one.
So root on a node reaches the secrets of the work placed on it, and no other tenant’s. Rotate the gateway’s token as the runbook says (secret rotation).
Vault service tokens
Every service token (agent gateway, API, controller, AI proxy, mail, the scheduler’s leftovers token) is a periodic orphan token. Its policy comes from vault/ plus default (which grants auth/token/lookup-self and renew-self):
bao token create -orphan -period=72h -policy=pushleft-agent-gateway # or vault token create ...
-
A service reads the token file once at start. It then renews the token itself (
renew-selfevery period/3, so every 24 h) for as long as it runs. A periodic token has no max TTL, and an orphan does not expire with its creator. -
pushleft-git-backup(an hourly oneshot,vault/pushleft-git-backup.hcl) renews its token once per run. -
If renewal fails, the service retries every minute and logs
vault_token_renewal_failed. When less than 24 h is left, its/readyzfails.ops-alertspages this asservice_not_ready, a day before the token expires. -
A token can have a TTL and a max TTL in place of a period. The check reports such a token the same way as it nears its max TTL.
-
npm run test:credential-renewalchecks every shipped policy on such a token, and the renewal against OpenBao. -
install-openbao.shwrites everyvault/pushleft-*.hclpolicy and issues each token into its file (SERVICES). On a control plane that is already running, add a new one by hand. For the scheduler’s leftovers token (added 2026-11-05):bao policy write pushleft-scheduler-leftovers deploy/vault/pushleft-scheduler-leftovers.hcl install -d -m 0700 -o pushleft -g pushleft /etc/pushleft/vault-scheduler (umask 077; bao token create -orphan -period=72h -policy=pushleft-scheduler-leftovers -field=token >/etc/pushleft/vault-scheduler/token) chown pushleft:pushleft /etc/pushleft/vault-scheduler/tokenThe scheduler, the API and the agent gateway all run as
pushleft. The API and gateway units hide/etc/pushleft/vault-scheduler(InaccessiblePaths), so only the scheduler reads this token.
Vault/OpenBao backup. Run pushleft-vault-snapshot.timer on the control-plane host. Every hour it:
- streams an integrated-storage (raft) snapshot through
ageto the public recipient of the offline operator key, - runs a pluggable off-host copy command (Storage Box or S3),
- keeps 35 days locally,
- writes the
last-successfile thatops-alertschecks (vault_snapshot_stale).
Its token carries only vault/pushleft-snapshot.hcl. Unseal shares and the operator key stay offline with named holders. Vault/OpenBao recovery covers setup, key custody, restore and the optional backup-key escrow.
Registry tokens. Nodes and the scheduler hold no registry login either. The gateway signs 15-minute tokens for one repository:
pull,pushfor the repository of a node’s live build lease,pullfor the repositories of instances placed on the node,- through its realm,
deletefor the scheduler on repositories with a released artifact.
Configure the registry with auth.token pointing at the gateway (registry tokens, including the migration from htpasswd). Application nodes must not keep a Docker login for the registry.
Env and secret files. /etc/pushleft/<unit>.env files are root:root 0600. Systemd reads them before it drops privileges. Each deploy/systemd/<unit>.env.example lists every variable that the binary reads, marks secret values and secret file paths, and gives the default. Secret files go under /etc/pushleft/<unit>/, owned by the service user, mode 0600. The binaries refuse key files with group or other permission bits. The three source-store services share a UID, so each can read the secret files of the others. This follows from the shared store (gap).
useradd --system --home /nonexistent --shell /usr/sbin/nologin pushleft
useradd --system --home /nonexistent --shell /usr/sbin/nologin pushleft-controller
useradd --system --home /nonexistent --shell /usr/sbin/nologin pushleft-telemetry
useradd --system --home /nonexistent --shell /usr/sbin/nologin pushleft-ops
useradd --system --home /nonexistent --shell /usr/sbin/nologin pushleft-mail # mail host only
# git: deploy/git/install-git-host.sh creates it (docs/git.md#install)
install -m 0755 target/release/pushleft-{api,scheduler,agent-gateway,controller,mesh,telemetry-gateway,errors,admin,offsite} /usr/local/bin/
install -d -m 0755 /usr/local/lib/pushleft/scripts /usr/local/lib/pushleft/deploy/observability/alerts
install -m 0755 scripts/ops-alerts.sh /usr/local/lib/pushleft/scripts/ # needs psql and curl on the host
# ops-alerts.env on the control-plane host needs PUSHLEFT_OPS_CLICKHOUSE_* (node use alerts; see the example)
install -m 0644 deploy/observability/alerts/control-plane.sql /usr/local/lib/pushleft/deploy/observability/alerts/
install -m 0644 deploy/systemd/pushleft-*.service deploy/systemd/pushleft-ops-alerts.timer /etc/systemd/system/
install -m 0600 deploy/systemd/api.env.example /etc/pushleft/api.env # then edit; repeat per unit
systemctl daemon-reload
Database roles
All services connect to one database. Run the SQL below as the database owner after the first migration. The GRANT ... ON ALL TABLES statements cover only tables that exist, so run them again after every migration that adds tables.
These roles are tested:
scripts/capacity-e2e.shcreates these roles with this exact grant set. It runs the API, scheduler, gateway,auditand the alert rules under them, and exercises deployment creation, placement and leasing (capacity).- Every staging install runs every service under these roles end to end, the API included (staging runbook).
npm run test:source-ingestionruns the API as aNOBYPASSRLSnon-owner role. Migration 0058 fixed source upload underpushleft_api.
| Role | Attributes | Used by |
|---|---|---|
pushleft_owner | owns the schema; the only role with DDL | pushleft-api migrate, pushleft-api bootstrap-owner |
pushleft_api | LOGIN NOBYPASSRLS, a non-owner role | pushleft-api service. RLS applies: tenant policies are the second line of defence behind the application predicates. |
pushleft_scheduler | LOGIN BYPASSRLS | pushleft-scheduler (placement spans organizations) |
pushleft_gateway | LOGIN BYPASSRLS, member of pushleft_agent_gateway | pushleft-agent-gateway (startup refuses a role without BYPASSRLS; artifact publication needs the group’s EXECUTE) |
pushleft_controller | LOGIN BYPASSRLS | pushleft-controller daemon; operators run pushleft-controller fleet with it too |
pushleft_telemetry | LOGIN BYPASSRLS | pushleft-telemetry-gateway (startup checks rolbypassrls) |
pushleft_mesh | LOGIN BYPASSRLS | pushleft-mesh run (publish_mesh_credential needs an explicit EXECUTE) |
pushleft_ops | LOGIN BYPASSRLS, read-only by default | pushleft-ops-alerts, pushleft-api audit |
pushleft_ai_proxy | LOGIN NOBYPASSRLS, no table grants | pushleft-ai-proxy: executes ai_proxy_admit, ai_proxy_models and ai_proxy_record only (SECURITY DEFINER, migration 0053) |
pushleft_errors | LOGIN NOBYPASSRLS, no table grants | pushleft-errors: executes errors_admit and errors_record_event only (SECURITY DEFINER, migration 0066) |
pushleft_mail | LOGIN NOBYPASSRLS, no table grants | pushleft-mail: executes the ten mail_* functions of migration 0054 only (SECURITY DEFINER) |
pushleft_admin_read | LOGIN BYPASSRLS, read-only, SELECT on the fleet, deployment, database, job and customer tables only | pushleft-admin reads (DATABASE_URL) |
pushleft_admin | LOGIN BYPASSRLS, per-table grants: operator auth tables, and exactly what the fleet and customer actions write | pushleft-admin operator sign-in, fleet and customer actions, pushleft-admin operator ... (PUSHLEFT_ADMIN_ACTION_DATABASE_URL) |
CREATE ROLE pushleft_api LOGIN PASSWORD '...' NOSUPERUSER NOBYPASSRLS;
CREATE ROLE pushleft_scheduler LOGIN PASSWORD '...' NOSUPERUSER BYPASSRLS;
CREATE ROLE pushleft_gateway LOGIN PASSWORD '...' NOSUPERUSER BYPASSRLS;
CREATE ROLE pushleft_controller LOGIN PASSWORD '...' NOSUPERUSER BYPASSRLS;
CREATE ROLE pushleft_telemetry LOGIN PASSWORD '...' NOSUPERUSER BYPASSRLS;
CREATE ROLE pushleft_mesh LOGIN PASSWORD '...' NOSUPERUSER BYPASSRLS;
CREATE ROLE pushleft_ops LOGIN PASSWORD '...' NOSUPERUSER BYPASSRLS;
CREATE ROLE pushleft_ai_proxy LOGIN PASSWORD '...' NOSUPERUSER NOBYPASSRLS;
CREATE ROLE pushleft_mail LOGIN PASSWORD '...' NOSUPERUSER NOBYPASSRLS;
CREATE ROLE pushleft_errors LOGIN PASSWORD '...' NOSUPERUSER NOBYPASSRLS;
CREATE ROLE pushleft_admin_read LOGIN PASSWORD '...' NOSUPERUSER BYPASSRLS;
CREATE ROLE pushleft_admin LOGIN PASSWORD '...' NOSUPERUSER BYPASSRLS;
GRANT USAGE ON SCHEMA public TO pushleft_api, pushleft_scheduler, pushleft_gateway,
pushleft_controller, pushleft_telemetry, pushleft_mesh, pushleft_ops;
GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO pushleft_api, pushleft_scheduler,
pushleft_gateway, pushleft_controller, pushleft_telemetry, pushleft_mesh;
REVOKE ALL ON _sqlx_migrations FROM pushleft_api, pushleft_scheduler, pushleft_gateway,
pushleft_controller, pushleft_telemetry, pushleft_mesh;
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO pushleft_api, pushleft_scheduler,
pushleft_gateway, pushleft_controller, pushleft_telemetry, pushleft_mesh;
GRANT DELETE ON organization_memberships TO pushleft_api; -- member removal
GRANT DELETE ON guest_project_grants TO pushleft_api; -- guest grant changes and member removal
GRANT DELETE ON network_attachment_requests TO pushleft_scheduler; -- pre-seal reservation reclaim
GRANT DELETE ON public_port_claims TO pushleft_scheduler; -- released port claims (0120)
GRANT DELETE ON fleet_node_addresses TO pushleft_controller; -- `fleet address-remove` (0120)
GRANT DELETE ON hostname_dns TO pushleft_controller; -- records no longer routed (0125)
GRANT DELETE ON org_budgets TO pushleft_api; -- budget removal (0051)
GRANT DELETE ON ai_provider_keys, ai_policies TO pushleft_api; -- AI key and policy replacement (0053)
GRANT DELETE ON mail_suppressions TO pushleft_api; -- suppression removal (0054)
GRANT DELETE ON environment_secrets TO pushleft_api; -- `pl secrets rm` (0057)
GRANT DELETE ON registry_upstreams, registry_retention_rules TO pushleft_api; -- registry upstream and rule removal (0063)
GRANT DELETE ON user_ssh_keys, git_webhooks TO pushleft_api; -- `pl ssh-key rm`, `pl git webhook rm` (0065)
-- 0162 grants git_credentials itself when the roles exist (SELECT, INSERT, DELETE to the API for
-- `pl git credential`; SELECT to the scheduler and the gateway for the incremental source scan).
GRANT DELETE ON projects TO pushleft_api; -- deleting a never-deployed project (0081)
GRANT DELETE ON database_datasets TO pushleft_api; -- `pl db dataset delete` (0101)
GRANT DELETE ON image_scan_findings TO pushleft_gateway; -- a newer scan replaces a deployment's findings; old ones pruned (0138)
GRANT DELETE ON source_scan_findings TO pushleft_gateway; -- a newer source scan replaces a deployment's findings (0159)
GRANT DELETE ON image_sboms TO pushleft_gateway; -- each daily scan prunes SBOMs of releases no longer live (0208, 0210)
GRANT DELETE ON cloudflare_edge_requests, cloudflare_edge_events TO pushleft_controller; -- edge traffic kept 7 days (0144)
-- 0150 adds platform_certificates (per-hostname platform certificates, M11) and
-- grants it itself when the roles exist: the gateway records CSRs and hands a node
-- its certificate, the controller issues, renews and revokes, ops alerts count
-- issuances. Roles created after migrating need, by hand:
-- GRANT SELECT, INSERT, UPDATE ON platform_certificates TO pushleft_gateway;
-- GRANT SELECT, UPDATE ON platform_certificates TO pushleft_controller;
-- 0153 adds platform_certificate_issuances (one row per issuance, for the weekly count):
-- GRANT SELECT, INSERT ON platform_certificate_issuances TO pushleft_controller;
-- GRANT SELECT ON platform_certificate_issuances TO pushleft_ops;
-- GRANT SELECT ON platform_certificates TO pushleft_ops;
-- 0051/0052 revoke PUBLIC execute: the API takes abuse reports and mails budget
-- alerts, the scheduler rolls up the usage ledger (docs/billing.md, docs/abuse.md).
GRANT EXECUTE ON FUNCTION submit_abuse_report(citext, text, text, text, text, inet),
claim_budget_alert_mail() TO pushleft_api;
-- 0063/0064: upstream host exclusivity and deploy-by-digest publication (docs/registry.md).
GRANT EXECUTE ON FUNCTION registry_upstream_host_available(uuid, text, boolean),
publish_image_artifact(uuid, uuid, text) TO pushleft_api;
-- 0137: build reuse, publishing an identical finished build (docs/build-executor.md#build-reuse).
GRANT EXECUTE ON FUNCTION publish_reused_artifact(uuid, uuid) TO pushleft_api;
-- 0065/0068: git server lookups before a tenant scope exists (repository,
-- webhook, SSH key by fingerprint and its user) and the backup index
-- (docs/git.md); the backup unit uses this login.
GRANT EXECUTE ON FUNCTION git_repository_for_push(text, text), git_webhook_for_delivery(uuid),
git_ssh_key(text), git_ssh_key_user(uuid), git_backup_index(), git_backup_recorded(uuid, text, text) TO pushleft_api;
-- 0151 prices the ledger with plans (docs/billing.md#plans). usage_rollup_next runs with the
-- scheduler's rights and calls the rollup, the budgets and the month close. The jsonb form
-- is kept for a scheduler from before 0151. 0151 grants these itself when the roles exist.
GRANT EXECUTE ON FUNCTION usage_rollup_next(timestamptz), usage_rollup_next(jsonb, timestamptz),
usage_rollup_hour(timestamptz), evaluate_budgets(timestamptz), billing_close_month(date) TO pushleft_scheduler;
GRANT EXECUTE ON FUNCTION record_storage_samples(text, uuid[], bigint[]) TO pushleft_api; -- storage sampler (0151)
GRANT SELECT ON billing_plans, org_plan_periods, billing_statements, billing_statement_lines,
storage_sample_runs, storage_usage_samples, fleet_allocation_charges TO pushleft_ops;
GRANT EXECUTE ON FUNCTION purge_expired_deployment_events(integer) TO pushleft_scheduler; -- event retention (0055)
GRANT EXECUTE ON FUNCTION purge_finished_history(integer, integer, integer) TO pushleft_scheduler; -- finished history retention (0073)
GRANT EXECUTE ON FUNCTION purge_deleted_project(integer, integer, uuid[]) TO pushleft_scheduler; -- deleted project purge (0131)
GRANT EXECUTE ON FUNCTION deleted_project_leftovers_due(text[], integer), deleted_project_leftover_done(bigint, text) TO pushleft_scheduler; -- Vault leftovers (0133)
-- 0138 adds image_scan_findings (docs/build-executor.md#findings): the gateway
-- writes it, the API reads it and copies a reused build's rows. On an existing
-- database, after migrating:
-- GRANT SELECT, INSERT, UPDATE ON image_scan_findings TO pushleft_api, pushleft_gateway;
-- GRANT SELECT ON image_scan_findings TO pushleft_ops;
-- 0159 adds source_scan_findings and source_scan_alerts (the daily source scan,
-- docs/build-executor.md#source-scan) and grants them itself when the roles exist:
-- the gateway writes both, the API reads the findings and mails the alerts.
-- Roles created after migrating need, by hand:
-- GRANT SELECT, INSERT, UPDATE, DELETE ON source_scan_findings TO pushleft_gateway;
-- GRANT SELECT, INSERT ON source_scan_alerts TO pushleft_gateway;
-- GRANT SELECT ON source_scan_findings TO pushleft_api, pushleft_scheduler;
-- GRANT SELECT ON source_scan_findings, source_scan_alerts TO pushleft_ops;
-- 0073 adds registry_keep_tag_failures (the gateway's keep-tag backoff; SELECT,
-- INSERT, UPDATE through the ON ALL TABLES grant above, RLS bypassed by
-- BYPASSRLS): on an existing database, rerun that grant after migrating.
-- 0074's deployment_is_active, append_deployment_event and billing_adjust run
-- with the caller's rights and keep PUBLIC execute: no grant needed.
GRANT EXECUTE ON FUNCTION activate_network_segments(), network_attachment_evidence_fresh(network_attachments)
TO pushleft_scheduler; -- segment activation (0059)
GRANT EXECUTE ON FUNCTION network_gateway_cpu_resize_valid(network_attachments,network_attachments)
TO pushleft_api, pushleft_scheduler, pushleft_gateway, pushleft_controller,
pushleft_telemetry, pushleft_mesh, pushleft_ops; -- CPU-only prepared intent fence (0094)
-- The mail service: no table grants, only its functions (0054); they call
-- mail_send_allowed / record_mail_event / record_usage as their owner.
GRANT USAGE ON SCHEMA public TO pushleft_mail;
GRANT EXECUTE ON FUNCTION mail_admit(bytea, text, text[]),
mail_enqueue(uuid, uuid, uuid, uuid, text, text, text[], text, bytea, text[]),
mail_lease(integer, integer, integer), mail_delivery_result(uuid, text, text, integer, text, timestamptz),
mail_record_bounce(uuid, text, text), mail_record_complaint(uuid, text), mail_find_delivery(text, text),
mail_domains_due(integer), mail_domain_checked(uuid, jsonb, boolean, text), mail_retention_sweep()
TO pushleft_mail;
-- 0066: the API mails issue alerts (docs/errors.md).
GRANT EXECUTE ON FUNCTION claim_error_alert_mail() TO pushleft_api;
GRANT EXECUTE ON FUNCTION claim_image_scan_alert_mail() TO pushleft_api;
GRANT EXECUTE ON FUNCTION image_scan_alert_mailed(uuid, text, boolean) TO pushleft_api;
GRANT EXECUTE ON FUNCTION claim_source_scan_alert_mail(), source_scan_alert_mailed(uuid, text, boolean) TO pushleft_api; -- source scan alerts (0159)
GRANT EXECUTE ON FUNCTION finished_import_archives() TO pushleft_api;
-- 0142 revokes PUBLIC execute: the API checks a node's session lane (docs/developer-access.md).
GRANT EXECUTE ON FUNCTION node_runs_sessions(uuid) TO pushleft_api;
GRANT EXECUTE ON FUNCTION import_archive_deleted(uuid) TO pushleft_api;
-- 0149: the API deletes finished `pl volume import` archives too.
GRANT EXECUTE ON FUNCTION finished_volume_import_archives(), volume_import_archive_deleted(uuid) TO pushleft_api;
-- 0133: the API deletes a purged project's blob leftovers (docs/retention.md "Deleted projects").
GRANT EXECUTE ON FUNCTION deleted_project_leftovers_due(text[], integer), deleted_project_leftover_done(bigint, text) TO pushleft_api;
GRANT EXECUTE ON FUNCTION request_custom_domain_dns(uuid, uuid, text) TO pushleft_api;
-- 0216: sign-up, the trial and Stripe payment (docs/authentication.md "Sign-up", docs/billing.md "Payment").
-- The migration grants these itself when the role exists.
GRANT EXECUTE ON FUNCTION pushleft_signup_request(citext, text, bytea, integer),
pushleft_signup_confirm(bytea, uuid, integer), pushleft_set_hosting(uuid, text),
claim_org_notice(), org_trial_blocked(uuid), pushleft_stripe_customer(uuid, text),
pushleft_payment_method_added(text), billing_statement_paid(text) TO pushleft_api;
-- 0217: notice mail retry, the paid-plan gate and a payment method removed at Stripe.
GRANT EXECUTE ON FUNCTION org_notice_mailed(uuid, text, text, boolean), billing_end_trial(uuid),
pushleft_customer_org(text), pushleft_payment_method_removed(text) TO pushleft_api;
-- The error ingest: no table grants, only its two functions (0066).
GRANT USAGE ON SCHEMA public TO pushleft_errors;
GRANT EXECUTE ON FUNCTION errors_admit(bytea, bigint, integer, integer, integer),
errors_record_event(uuid, uuid, bytea, text, text, text, text, text) TO pushleft_errors;
-- The AI proxy: no table grants (not in the GRANT ... ON ALL TABLES lists above), only its functions (0053).
GRANT USAGE ON SCHEMA public TO pushleft_ai_proxy;
GRANT EXECUTE ON FUNCTION ai_proxy_admit(bytea, text, text, integer, integer), ai_proxy_models(bytea),
ai_proxy_record(uuid, uuid, text, text, text, bigint, bigint) TO pushleft_ai_proxy;
-- The pre-tenant auth functions; migrations 0027, 0038, 0042, 0048-0050 and 0086 revoke PUBLIC execute.
GRANT EXECUTE ON FUNCTION pushleft_authenticate_token(bytea), pushleft_authenticate_session(bytea, uuid),
pushleft_redeem_login_code(citext, bytea), pushleft_rate_limit_hit(text, text, text, integer, integer),
pushleft_totp_challenge(bytea), pushleft_accept_invitation(bytea, uuid),
pushleft_login_targets(citext), -- 0048 emailed login codes
pushleft_webauthn_challenge(bytea, text, uuid, uuid), pushleft_webauthn_take(bytea, text),
pushleft_passkey(bytea), pushleft_challenge_passkeys(bytea), -- 0049 passkeys
pushleft_sso_sign_in(uuid, text, text, citext, uuid), pushleft_sso_config(uuid), -- 0050 SSO
pushleft_session_organizations(bytea), pushleft_switch_session(bytea, uuid, bytea) -- 0086 org switch
TO pushleft_api;
-- Since 0042 the API no longer creates users or lists a user's other organizations;
-- on databases set up before it: REVOKE EXECUTE ON FUNCTION pushleft_ensure_user(citext),
-- pushleft_user_memberships(uuid) FROM pushleft_api;
-- Optional: replication details (lag, sync state) in /readyz and the replay_lag alert.
GRANT pg_read_all_stats TO pushleft_api, pushleft_ops;
-- Migration 0024 creates the NOLOGIN group with EXECUTE on publish_release_artifact.
GRANT pushleft_agent_gateway TO pushleft_gateway;
-- 0082: instance samples and runtime output, admitted per node placement and org cap (docs/logs.md).
GRANT EXECUTE ON FUNCTION instance_telemetry_admit(uuid, uuid, uuid[], bigint[]) TO pushleft_gateway;
-- 0088: database insights, admitted per node placement (docs/managed-databases.md).
GRANT EXECUTE ON FUNCTION database_telemetry_admit(uuid, text[]) TO pushleft_gateway;
GRANT EXECUTE ON FUNCTION publish_mesh_credential(uuid,uuid,uuid,uuid,bigint,text,text,text,text,timestamptz,text,uuid)
TO pushleft_mesh;
-- Topology sealing checks each member's reservation (migration 0022 revokes PUBLIC).
GRANT EXECUTE ON FUNCTION reserved_environment_workload(uuid,uuid,uuid) TO pushleft_mesh;
-- A revised topology admits a workload only when no reserved member lags (migration 0136).
GRANT EXECUTE ON FUNCTION reserved_environment_workload(uuid,uuid,uuid) TO pushleft_scheduler;
-- An idle member leaves a sealed network (migrations 0147 and 0148, deployed apart:
-- docs/mesh-coordinators.md#member-removal): the gateway leases
-- and completes its teardown; the scheduler releases one that never had an attachment.
GRANT EXECUTE ON FUNCTION network_member_removal_authorized(uuid), finish_network_member_removal(uuid,uuid,uuid)
TO pushleft_gateway;
GRANT EXECUTE ON FUNCTION finish_network_member_removal(uuid,uuid,uuid) TO pushleft_scheduler;
GRANT EXECUTE ON FUNCTION network_topology_revision_valid(network_segments,network_segments)
TO pushleft_api, pushleft_scheduler, pushleft_gateway, pushleft_controller,
pushleft_telemetry, pushleft_mesh, pushleft_ops; -- topology revision fence (0136, revised in 0147)
GRANT SELECT ON ALL TABLES IN SCHEMA public TO pushleft_ops;
ALTER ROLE pushleft_ops SET default_transaction_read_only = on;
-- The fleet admin API (crates/pushleft-admin, migration 0084): BYPASSRLS because
-- the fleet and operator tables have no tenant policy; table grants listed one by one.
GRANT USAGE ON SCHEMA public TO pushleft_admin_read, pushleft_admin;
GRANT SELECT ON fleet_nodes, fleet_allocations, fleet_events, deployments, application_instances,
database_instances, jobs, organizations, projects, environments, platform_operators TO pushleft_admin_read;
-- The pipeline view: build failures' code and message, and capacity waits (never the build log).
GRANT SELECT ON deployment_events TO pushleft_admin_read;
-- The deployment page: its live routes and whether a hostname is a custom domain.
GRANT SELECT ON ingress_analytics_routes, custom_domains TO pushleft_admin_read;
-- The Backups view: volumes' backup evidence and the agent's message of a failed backup job.
GRANT SELECT ON application_volumes, agent_job_results TO pushleft_admin_read;
-- Fleet admin: a network gateway placement names its environment.
GRANT SELECT ON network_attachment_requests, network_segments TO pushleft_admin_read;
-- Node reads: reserved CPU as charged (production 2.5x, migrations 0151/0152).
GRANT SELECT ON fleet_allocation_charges TO pushleft_admin_read;
-- The arena's bots: requests Cloudflare blocked on linked domains (0144; the controller fills it).
GRANT SELECT ON cloudflare_edge_events TO pushleft_admin_read;
-- The Organizations view: tier limits, the open plan period and role-size override.
GRANT SELECT ON org_tiers, org_plan_periods, org_role_resource_overrides TO pushleft_admin_read;
-- The Customers page (migration 0220): users, their memberships and newest console session
-- (last sign-in; never the session hash), and the plans.
GRANT SELECT (id, email, display_name, created_at, disabled_at) ON users TO pushleft_admin_read;
GRANT SELECT (user_id, created_at) ON console_sessions TO pushleft_admin_read;
GRANT SELECT ON organization_memberships, billing_plans TO pushleft_admin_read;
ALTER ROLE pushleft_admin_read SET default_transaction_read_only = on;
-- Operator sign-in: operators, passkeys, enrollment links, sessions.
GRANT SELECT, INSERT, UPDATE ON platform_operators, platform_operator_passkeys,
platform_operator_enrollments, platform_operator_sessions TO pushleft_admin;
-- The `fleet` library functions behind the actions: provision/enroll-host/admit/
-- drain/decommission (fleet_nodes, certificate revocation), migrate-volumes
-- (application_volumes + events), relocate (request_database_operation, whose
-- FOR UPDATE lock needs a column UPDATE grant), and the audit row.
GRANT SELECT, INSERT, UPDATE ON fleet_nodes TO pushleft_admin;
GRANT SELECT, UPDATE ON fleet_node_certificates, application_volumes TO pushleft_admin;
GRANT SELECT ON fleet_allocations, jobs, database_instances, database_operations TO pushleft_admin;
-- Fleet's Backup files page queues `manage_backups` jobs (migration 0227).
GRANT INSERT ON jobs TO pushleft_admin;
GRANT UPDATE (updated_at) ON database_instances TO pushleft_admin;
GRANT INSERT ON application_volume_events, database_operations, database_operation_events, fleet_events
TO pushleft_admin;
-- Organization policies an operator sets in the app (PUT /v1/orgs/{id}/settings/...): these columns only.
GRANT SELECT (id, allow_external_images, image_scan_policy, root_role_policy),
UPDATE (allow_external_images, image_scan_policy, root_role_policy) ON organizations TO pushleft_admin;
-- The Customers page's actions (POST /v1/customers/.../actions/..., migration 0220). Suspend,
-- unsuspend and approve cloud access: these columns and the audit row.
GRANT SELECT (suspended_at, cloud_access),
UPDATE (suspended_at, suspended_reason, cloud_access, hosting) ON organizations TO pushleft_admin;
GRANT INSERT ON auth_events, org_notices TO pushleft_admin;
-- Set plan, and the plan change of an approval.
GRANT SELECT ON org_plan_periods, billing_plans TO pushleft_admin;
GRANT EXECUTE ON FUNCTION billing_assign_plan(uuid, text, timestamptz, text) TO pushleft_admin;
-- Disable a user: the account, its sessions and its API tokens.
GRANT SELECT (id, email, disabled_at), UPDATE (disabled_at) ON users TO pushleft_admin;
GRANT SELECT (user_id, revoked_at), UPDATE (revoked_at) ON console_sessions TO pushleft_admin;
GRANT SELECT (id, org_id, user_id, revoked_at), UPDATE (revoked_at) ON api_tokens TO pushleft_admin;
-- Drain and decommission count the workspaces on a node (migration 0220).
GRANT SELECT (node_id, deleted_at) ON dev_workspaces TO pushleft_admin;
Every table except _sqlx_migrations has RLS enabled (checked on a migrated database). So tenant policies bound the broad table grant to pushleft_api.
- In production authentication mode, the API refuses to start if its role is a superuser or has
BYPASSRLS. It also refuses if the role owns any table inpublic(directly or through membership). The exception isPUSHLEFT_MIGRATE_ON_START=1, which names the owner path explicitly. - No table uses
FORCE ROW LEVEL SECURITY.pushleft-api migrate, migration backfills andbootstrap-ownerrun as the non-superuser owner and rely on the owner bypassing tenant policies. Forced RLS would break them. The startup refusal keeps the owner out of the API. - Several tables have a read-only tenant policy or none. They refuse API writes whatever the grant.
- A per-table least-privilege grant set for the service roles is not worked out (gap).
- The
fleetCLI deletes nothing. The DELETE needs of the controller daemon are audited only by its test suite, which runs as the owner.
After migration 0094, a new service database role that updates network_attachments also needs EXECUTE ON FUNCTION network_gateway_cpu_resize_valid(network_attachments,network_attachments). The migration grants this to the existing API, scheduler, gateway, controller, telemetry, mesh and ops roles. The helper allows only CPU changes that capacity accounts for, and keeps the existing credential and network-policy fences (tenant gateway CPU).
Migrations
Migrations are the embedded sqlx set in crates/pushleft-api/migrations/. You do not need an external sqlx-cli.
pushleft-api migrateapplies pending migrations withDATABASE_URLand exits. Use the owner role.pushleft-api(the service) also applies them at startup whenPUSHLEFT_MIGRATE_ON_START=1. The default is1in development authentication mode and0in production mode. In production,1is the explicit owner path, and it also skips the RLS-bypass refusal above. Any other value refuses to start. Even with nothing pending, sqlx runsCREATE TABLE IF NOT EXISTS _sqlx_migrations, which needsCREATEon schemapublic. So a non-ownerpushleft_apirole fails at startup unless this is0(verified on PostgreSQL 17 with a role that had onlyUSAGE).pushleft-api bootstrap-owneralso migrates before it creates the first owner.
To upgrade:
pg_dump --format=custom --file=pushleft-pre-$(date -u +%Y%m%dT%H%M%SZ).dump "$OWNER_URL" # see the restore runbook
systemctl stop pushleft-api pushleft-scheduler pushleft-agent-gateway pushleft-controller pushleft-mesh pushleft-telemetry-gateway
DATABASE_URL="$OWNER_URL" /usr/local/bin/pushleft-api migrate
# rerun the GRANT ... ON ALL TABLES / SEQUENCES statements above if the release added tables
# and run any GRANT DELETE / GRANT EXECUTE line above that the database lacks (e.g. database_datasets, added 2026-11-04)
systemctl start pushleft-api pushleft-scheduler pushleft-agent-gateway pushleft-controller pushleft-mesh pushleft-telemetry-gateway
Migrations are forward-only, and there are no down migrations. To roll back, restore the pre-upgrade dump (control-plane PostgreSQL restore). Agents keep running during the window. Leases and heartbeats fail, and agents retry until the gateway is back. In-flight leases can expire, and the jobs retry under the normal fencing. Version skew between binaries and schema is not tested, so stop all services across a migration.
Control-plane PostgreSQL HA
Files: postgres/primary.conf, postgres/replica.conf, postgres/pg_hba.replication.conf, postgres/pushleft-pg-ha.sh, systemd/pushleft-postgres-basebackup.{service,timer}, systemd/postgres-ha.env.example. Install the postgres/ directory as /etc/pushleft/postgres/ (root-owned, 0755, files 0644, the script 0755) on both nodes. archive_command and restore_command call the script there.
| Choice | Setting | Why |
|---|---|---|
| Streaming replication through a slot | wal_level=replica, slot pushleft_replica, max_slot_wal_keep_size=20GB | The replica never misses WAL; a dead replica cannot fill the primary’s disk (past 20 GB it needs a re-base) |
| Synchronous commit | synchronous_commit=on, synchronous_standby_names='ANY 1 (pushleft_replica)' via enable-sync | Losing one node loses no acknowledged write (job leases, fencing generations, revocations). Cost: writes wait while the replica is down until disable-sync. Promotion switches the new primary to async until a replica rejoins |
| WAL archive | archive_mode=on, archive_timeout=60, local PUSHLEFT_PG_ARCHIVE_DIR (/var/backups/pushleft/wal-archive, on the backup filesystem), each segment gzipped under its own name (a quiet minute’s 16 MB segment is mostly zeros); pushleft-postgres-basebackup.timer prunes it to the oldest kept base backup | PITR to any second covered by archive + base backups; the copy is atomic and refuses to overwrite a different file. Off-host copies are the operator’s (gap) |
| Fencing | pushleft-pg-ha.sh fence (standby.signal) or provider power-off/firewall, before promote (which refuses without PUSHLEFT_PG_OLD_PRIMARY_FENCED=yes) | No split-brain writer: a fenced node restarts read-only, and every service’s readiness refuses a standby |
| Rejoin | Re-base with pg_basebackup (bootstrap-replica, PUSHLEFT_PG_REBASE=1) | Always works after divergence; wal_log_hints=on leaves pg_rewind possible (untested) |
Setup, once (as postgres, with /etc/pushleft/postgres-ha.env loaded: set -a; . /etc/pushleft/postgres-ha.env; set +a):
-
Primary: run
pushleft-pg-ha.sh configure-primary, then restart PostgreSQL (wal_level, archive_mode, listen_addresses). The command:- includes
primary.conf, which listens onlocalhostonly; - sets
listen_addresses = 'localhost,<PUSHLEFT_PG_LISTEN_ADDRESSES>'inpostgresql.auto.conf.PUSHLEFT_PG_LISTEN_ADDRESSESis the node’s management tailnet address. Install99-pushleft-nonlocal-bind.confin/etc/sysctl.d/so that PostgreSQL binds before tailscaled assigns that address at boot; - deletes the package or image catch-all
host all all all(or0.0.0.0/0,::/0) hba line; - appends the replication line for
PUSHLEFT_PG_REPLICA_HOSTS. It appends onehostssl all allline per entry ofPUSHLEFT_PG_CLIENT_HOSTS(the control-plane hosts). UsePUSHLEFT_PG_CLIENT_HBA_TYPE=hoston test servers without a certificate; - creates the
pushleft_replicatorrole and slot.
On the Debian layout,
pg_hba.confis outside the data directory. Copy the same lines to the replica’s file by hand. - includes
-
Replica, with PostgreSQL stopped and the data directory empty: run
pushleft-pg-ha.sh bootstrap-replicawith the replica’s ownPUSHLEFT_PG_LISTEN_ADDRESSES, then start it. The command runspg_basebackupthrough the slot and includesreplica.conf. It writesprimary_conninfo(with the password) and the replica’slisten_addressestopostgresql.auto.conf(0600), and writesstandby.signal.pitr-restoredrops the copiedlisten_addresses(loopback only) unlessPUSHLEFT_PG_LISTEN_ADDRESSESis set. -
Primary: run
pushleft-pg-ha.sh enable-sync. Check thatSELECT sync_state FROM pg_stat_replicationisquorum. -
Primary: run
systemctl enable --now pushleft-postgres-basebackup.timer. Copy/var/backups/pushleft/postgresand the archive off the host. -
In
ops-alerts.env, setPUSHLEFT_OPS_REPLICA_DATABASE_URLfor the standby. The read-onlypushleft_opsrole exists there through replication.
The migration-owner role also needs CREATEROLE, because migration 0024 creates the pushleft_agent_gateway group role.
npm run test:control-plane-haruns all of this on Docker against the pinnedPUSHLEFT_POSTGRES_17_IMAGE(image layout: config in the data directory). Production’s control plane runs PostgreSQL 18 from PGDG (Robot hosts). Managed databases default to 18.- The go-live layout (from 2026-11-02, until revenue) may put this database on the control-plane host itself. Then it has no replica: the WAL archive, base backups and off-host copies are its protection.
- The staging fleet uses the Debian/Ubuntu layout in
postgres-ha.env.example(config in/etc/postgresql/17/main, binaries in/usr/lib/postgresql/17/bin). Staging ranconfigure-primary,bootstrap-replicaandenable-synctosync_state = quorum(staging runbook). Failover and PITR have not run on it. - Failover and rejoin are in the failover runbook. PITR is in the restore runbook.
Automatic failover and off-host copies
The same script also sets up pg_auto_failover 2.2 from PGDG:
- a monitor on a third host as the witness (
auto-monitor), - the two data nodes (
auto-node,postgres/auto-failover.conf), run bysystemd/pushleft-pg-autoctl.service.
The moving endpoint is HAProxy on every control-plane host (postgres/haproxy-pushleft-pg.cfg, 127.0.0.2 db.internal.pushleft.dev). It checks systemd/pushleft-pg-health.socket (port 8008, 200 only on the writable primary). So DATABASE_URL stays the same, and the services follow a promotion without a restart. Service connections are TLS-only (hostssl, sslmode=verify-full), and the server certificate must name db.internal.pushleft.dev.
systemd/pushleft-postgres-offsite.{service,timer} (pushleft-pg-ha.sh offsite) encrypts every base backup and archived WAL file with age to the operator key, and runs a pluggable copy command. ops-alerts.sh fires pg_archive_copy_stale from its last-success (PUSHLEFT_OPS_PG_ARCHIVE_COPY_STATUS_FILE).
The manual commands above stay as the fallback. They refuse to run on a node that pg_auto_failover manages.
Gap: a three-VM Lima suite proved failover, fencing, rejoin, switchover and an off-host restore on 2026-10-20. The suite was removed with the Lima harness on 2026-10-23. Staging runs the manual mode, so no suite proves automatic failover today.
The failover runbook has the setup, behaviour and those results. Control-plane host failover covers moving the services themselves.
Health and shutdown
| Service | Liveness | Readiness |
|---|---|---|
| API | GET /healthz on its listener (204) | GET /readyz on its listener: 200 when the database answers within 2 s and is not in recovery, else 503. The JSON body reports in_recovery, standby replay lag and the primary’s replicas |
| AI proxy | GET /healthz on its listener (204) | GET /readyz on its listener: the same database check as the API |
| Agent gateway | PUSHLEFT_HEALTH_BIND /healthz (200). The mTLS /healthz needs an agent certificate. | PUSHLEFT_HEALTH_BIND /readyz: SELECT NOT pg_is_in_recovery() true within 2 s |
| Telemetry gateway | PUSHLEFT_HEALTH_BIND /healthz | /readyz: the same database check |
| Scheduler, controller, mesh | PUSHLEFT_HEALTH_BIND /healthz | /readyz: the same database check, and the control loop completed an iteration in the last 30 s |
| Agent | none on the node | the control plane sees fleet_nodes.last_seen_at (alert node_heartbeat_stale); pushleft-agent doctor on the host |
The health listener (crates/pushleft-ops) is plain HTTP with no authentication and no data in the response. Bind it to loopback. It answers only the two exact request lines, and 404 for anything else.
Every binary handles SIGTERM (which systemd sends on stop) and SIGINT the same way:
- API and gateways: stop accepting, finish in-flight requests, then exit. Open SSE event streams end cleanly, and
pl up/pl logsreconnect from their cursor after 0.5 s. If no API answers at that moment, the CLI exits with an error, and the deployment continues on the server. A drain deadline (30 s for the API and agent gateway, 20 s for the telemetry gateway) closes what is still open.TimeoutStopSec=45ssends SIGKILL after that. - Scheduler and controller: the current reconciliation tick finishes (each runs in its own transaction), then the process exits. The scheduler’s retention task stops mid-pass. Its deletes are safe to run again (verified-delete, then record).
- Mesh: the tick loop stops, and in-flight coordinator workers are aborted. Their job leases expire and retry under the existing fencing.
- Agent: all five lanes are cancelled and in-flight child processes are killed, the same as Ctrl-C (agent gateway).
Related
- Operating Pushleft: SLOs, alerts, capacity and disaster recovery.
- Fleet substrate: node provisioning, enrollment and the node units.
- Staging fleet: these units installed on five VMs.
- Handbook: the infrastructure in one screen.
- Control-plane PostgreSQL failover: failover and rejoin.