Hetzner Robot hosts
Use this runbook to add a Hetzner Robot (dedicated) server to the fleet.
The Hetzner Cloud API cannot create Robot servers, so fleet provision does not apply. You install the host yourself. Then pushleft-controller fleet enroll-host writes the same bootstrap that cloud-init runs on Cloud nodes, and you run it over SSH. After enrollment, both kinds of node work the same way: admit, drain, decommission, lost-node reclamation, CA rotation and self-update.
Section 2 and the control-plane host packages of section 3 ran on the first Robot server, pl-cp, on 2026-11-03. We wrote the rest from the installimage and Robot firewall documentation and from the preconditions of the agent. The host preparation after installation runs on every staging install on Ubuntu 24.04 amd64 VMs. That covers the XFS data partition with prjquota, enroll-host, the bootstrap over a shell, Kata and admit.
Before you start
- A Robot account with your SSH key added.
- The fleet environment on your operator machine (listed in section 5).
- Shell access to the control-plane host, for the management tailnet keys.
1. Order
- Order an AX-line server with two NVMe drives. The AX42, EX63 and AX102 have ECC memory. The AX41 does not.
- Order it in the region its deployments will run in. Auction listings can be in any location. Before you install, check the data centre in Robot (Server → the server, or
GET https://robot-ws.your-server.de/serverand itsdcfield, e.g.FSN1-DC1). The round trip between FSN1 and HEL1 is about 36 ms, too slow between an application and its database. Nodes are matched to deployments byregion(fsn1,nbg1,hel1). Keep the workload nodes in one region unless you want to spread them. The control plane may be in another region. It reaches nodes over the management tailnet. - Add your SSH key in Robot before you order, so the rescue system accepts it.
2. Install Ubuntu 24.04
Every agent host needs /var/lib/pushleft/data as its own mount point (ConditionPathIsMountPoint in pushleft-agent.service). Database hosts also need XFS project quotas on it. The agent checks that accounting and enforcement are on, and refuses database leases otherwise. Use the same layout on every pool, so that any host can take any pool. A customer-owned node does not use this runbook: its bootstrap prepares the data disk and writes the role settings (customer-owned nodes).
-
Boot the rescue system (Robot → Server → Rescue → Linux 64-bit).
-
SSH in, save this config as
/root/setup.confand runinstallimage -a -c /root/setup.conf. Check the image name withls /root/images: the rescue system ships.tar.zstimages. SetHOSTNAMEto the host’s Hetzner label (pl-cp,pl-build-1,pl-app-1orpl-database-1):DRIVE1 /dev/nvme0n1 DRIVE2 /dev/nvme1n1 SWRAID 1 SWRAIDLEVEL 1 HOSTNAME pl-cp BOOTLOADER grub IMAGE /root/images/Ubuntu-2404-noble-amd64-base.tar.zst PART /boot ext4 1G PART / ext4 100G PART /var/lib/pushleft/data xfs allOn a UEFI server, add
PART /boot/efi esp 256Mbefore the/bootline, or the installed system does not boot.[ -d /sys/firmware/efi ]in the rescue system shows UEFI, and the newer Ryzen and Intel Core boxes have it.Reboot when it reports “INSTALLATION COMPLETE”. The RAID resync runs in the background after the first boot. If it is still running when the agent starts,
node_disk_warningtickets the rebuilding array until it ends. -
After the first boot, turn on project quotas. XFS reads quota options only at mount time, and only from fstab for a non-root filesystem.
sed -i 's#\(/var/lib/pushleft/data\s\+xfs\s\+\)defaults#\1defaults,prjquota#' /etc/fstab umount /var/lib/pushleft/data && mount /var/lib/pushleft/data xfs_quota -x -c 'state -p' /var/lib/pushleft/data # "Accounting: ON", "Enforcement: ON"
The backup directory
There is no backup partition (decided 2026-11-01). Backups keep 1 day on the data filesystem; the off-host bucket (R2) keeps the rest. /var/backups/pushleft is a bind mount of a directory there, so every check that wants it as a mount point still holds:
mkdir -p /var/lib/pushleft/data/backups /var/backups/pushleft
echo '/var/lib/pushleft/data/backups /var/backups/pushleft none bind,x-systemd.requires-mounts-for=/var/lib/pushleft/data 0 0' >>/etc/fstab
systemctl daemon-reload && mount /var/backups/pushleft
mountpoint /var/backups/pushleft
The backup directory is outside the data root’s instance directories, so nothing of it is mounted into a container. It has no project quota: backups count against the data filesystem as a whole, watched by node_disk_high (80%) and node_disk_full (95%). Until R2 ships, the spool keeps every file the 1-day local retention drops, so install the R2 token before the disk fills.
/var/backups/pushleft holds the local backups (tenant: tenant/, one day) and the off-host spool (offsite/<kind>/). Tenant backups are stored once: the spool hard-links the store, and a shipped tenant file leaves the spool when the store’s retention drops it. Other kinds keep 40 days of what they shipped (pushleft-offsite --keep-local-days). The control-plane host also keeps 7 days of local base backups and WAL.
- The agent keeps tenant backups in
/var/backups/pushleft/tenantand hard-links them into/var/backups/pushleft/offsite/tenant. It refuses a spool on/; mount the bind mount above before the agent starts. pushleft-agent doctormust printok backup spool ... is on its own filesystem(here: not on/).- The spool layout is
<org>/<source>/{base/<id>/,wal/,last-success}. - On database hosts, run
scripts/ops-alerts.shas root withPUSHLEFT_OPS_TENANT_SPOOL_ROOT=/var/backups/pushleft/offsite/tenant. The alerttenant_backup_spool_stalethen watches the spool. - Every host with a spool runs
pushleft-offsite.timer, which ships the spool to R2 (offsite.env.example, onePUSHLEFT_OFFSITE_S3_PREFIXper host). It skips dotfiles andlast-success. It alerts throughoffsite_copy_stale(PUSHLEFT_OPS_OFFSITE_STATUS_FILE=/var/lib/pushleft-offsite/last-success).
To restore a tenant backup from R2:
- Sync
<prefix>/tenant/<org>/<source>/into a directory with the same layout. - Run
pushleft-agent restore-from-spool --organization <org> --source <src> <dir>against the spool or its off-host copy. - Run the normal restore task (managed databases).
Garage off-host copy
On the Garage host (pl-database-1), scripts/garage-mirror.sh copies every Garage bucket to R2 every night, from pushleft-garage-mirror.timer (02:00 UTC plus up to 1 h random delay). It writes to the R2 bucket pushleft-garage-mirror (EU), separate from pushleft-offsite. Settings: /etc/pushleft/garage-mirror/r2.env (example). The API token pushleft-garage-mirror-pl-database-1 has Object Read & Write on that bucket only. Its secret is in /etc/pushleft/garage-mirror/r2-secret-access-key (0600), and an operator copy is in ~/.config/pushleft/r2-garage-mirror-pl-database-1.env.
-
Each run reads every bucket with the read-only Garage key
pushleft-mirror. The script creates the key on its first run and stores it in/etc/pushleft/garage-mirror/garage-key.env. It grants the key read access to every bucket on every run, so new buckets are included. -
Each run writes a full snapshot to
garage/<UTC stamp>/<bucket>/. Object names and contents are encrypted with rclone crypt. The run checks each bucket withrclone cryptcheck. Then it writes<stamp>/COMPLETE(one line per bucket:<bucket> <objects> <bytes>, not encrypted) and/var/lib/pushleft-garage-mirror/last-success(<epoch> <objects>). -
The run never overwrites or deletes in R2. The bucket’s lock rule
lock-5drefuses that for 5 days. Its lifecycle ruleexpire-5ddeletes objects after 5 days, and unfinished multipart uploads after 1 day (Waseem, 2026-11-05: keep 5 days). Thus the last 5 nights are kept, and you can recover an object that was deleted or overwritten in Garage up to 5 days ago. -
Older copies are in
pushleft-offsite. They are the first two snapshots of 2026-11-05 (pushleft-offsite/pl-database-1/garage/20261105T155831Z/and the one of 17:11 CET) and the test objectspl-database-1/garage/test-501,-501band-501c. That bucket’s 45-day lock refuses their deletion, and its lifecycle rule deletes them around 2026-12-20. Leave them. Do not shorten that bucket’s rules, because they also protect the database and WAL backups. -
The agent reports
last-successwith its host facts.garage_mirror_stale(ticket) fires after 26 h without a complete run. It also fires for a database host older than 26 h that never recorded a success, for example when the first run failed. -
The crypt key is in
/etc/pushleft/garage-mirror/crypt.conf(0600 root,RCLONE_CONFIG_MIRROR_PASSWORD{,2}, obscured by rclone). An age copy is on the operator Mac:~/.config/pushleft/garage-mirror-crypt-2026-11-05.age, encrypted topushleft-offline.age.pub. Without this key, the snapshots cannot be read. The script does not create the key. To create it on a new host (then make the age copy):install -d -m 0700 /etc/pushleft/garage-mirror && umask 077 for n in PASSWORD PASSWORD2; do printf 'RCLONE_CONFIG_MIRROR_%s=%s\n' $n "$(openssl rand -base64 32 | tr -d '\n' | rclone obscure -)"; done > /etc/pushleft/garage-mirror/crypt.conf -
rclone is Ubuntu 24.04’s package, pinned and held:
apt-get install rclone=1.60.1+dfsg-3ubuntu0.24.04.6 && apt-mark hold rclone. -
Install: copy the script to
/usr/local/bin/pushleft-garage-mirrorand the two units to/etc/systemd/system/, thensystemctl enable --now pushleft-garage-mirror.timer. To run it now:systemctl start pushleft-garage-mirror.service, log injournalctl -u pushleft-garage-mirror.
To restore, on the Garage host as root, set the same RCLONE_CONFIG_* variables as the script (r2.env, crypt.conf, garage-key.env), then:
- Find the latest complete snapshot:
rclone lsf r2:pushleft-garage-mirror/garage/, and check that<stamp>/COMPLETEexists. - Set
RCLONE_CONFIG_MIRROR_REMOTE=r2:pushleft-garage-mirror/garage/<stamp>/<bucket>, thenrclone copy mirror:<key or directory> <local dir>(decrypts). - To put objects back into Garage, use a key with write access to that bucket (
pushleft-mirroris read-only):rclone copy <local dir> <remote>:<bucket>.
Disk use
/ holds Docker images. BuildKit state is not written to disk. Each build keeps it in a size-limited tmpfs (isolation decision), which needs 12 GiB of memory per build. The agent workspace under /var/lib/pushleft/builds peaks at about 8.3 GiB per build.
3. Private network: the management tailnet
There is no vSwitch. Every Pushleft host (Robot, Hetzner Cloud or on premises) reaches the control plane over one self-hosted tailnet. The management Headscale on the control-plane host coordinates it. Its ACL (deploy/headscale/mgmt-acl.hujson) decides which host reaches which port. Agent hosts reach only the agent ports of the control plane, and never each other.
The private listeners answer only on the tailnet and loopback. They are the registry token realm 8446, registry 5000, OpenBao 8200, PostgreSQL 5432 and ClickHouse. The agent gateway 8443, enrollment 8444 and telemetry gateway 8445 are public since 2026-11-07, for customer-owned nodes, as gateway.<base>; each authenticates its callers (agent certificates, one-use enrollment secrets, telemetry certificates).
Below, hs means docker exec pushleft-mgmt-headscale /ko-app/headscale on the control-plane host.
Control-plane host, once
deploy/install-control-plane.sh installs box 1 from the key ceremony’s box1/ bundle, and deploy/postgres/install-db-node.sh installs the database host (box 2) from its db/ bundle. The control-plane PostgreSQL and ClickHouse run on box 2. staging.sh rehearse rehearses this order on fresh VMs, including a reboot.
Box 1 serves its public names itself: api, console, errors (error ingest), mgmt, registry and releases, each under <base>. Caddy gets their certificates with ACME (HTTP-01 on port 80, TLS-ALPN on 443). So each name needs an unproxied DNS A record for the public IPv4 of box 1.
Do these steps in this order. Run each command as root.
- On box 1, install the host packages, then write
/etc/pushleft/install.env(the variables are at the top ofinstall-control-plane.sh). The installers expect these, and stop when one is missing:- Tailscale (every other host, step 1).
- From apt:
docker.io nftables xfsprogs jq zstd openssl age git, and PostgreSQL 18 from the PGDG repository (postgresql-common, then/usr/share/postgresql-common/pgdg/apt.postgresql.org.sh -y, thenpostgresql-18 postgresql-client-18). Ubuntu 24.04 ships PostgreSQL 16. The package creates the18/maincluster thatinstall-db-node.shuses. - Caddy 2.11.7 and OpenBao 2.7.1 from their GitHub release tarballs into
/usr/local/bin, each checked against its SHA-256 (the values are inscripts/staging/base-setup.sh). - The release binaries in
/usr/local/bin:pushleft-api pushleft-git-shell pushleft-scheduler pushleft-agent-gateway pushleft-controller pushleft-mesh pushleft-telemetry-gateway pushleft-errors pushleft-admin pushleft-offsite, built withcargo build --release --lockedfrom a committed tree (git archive). For drive and RAID health in the Fleet app, also installpushleft-agent(the binary only; the host runs no agent) andsmartmontools. Without themstartskipspushleft-host-health.timerand says so. - The console build as a tar file (
npm run build, thentar -C dist -cf console.tar .). For mail (login codes, invitations, security alerts), do this once first. In Amazon SES (regioneu-west-1, where the account sends, unlessPUSHLEFT_SES_REGIONsays otherwise), verify the sending domain with Easy DKIM and add its three DKIM CNAME records. Then create the platform sender as indeploy/aws/README.md. The sender is an IAM user under/pushleft/mail/that may send only as that domain, with an access key and its SES SMTP password. SetPUSHLEFT_SES_SMTP_USERto the access key id andPUSHLEFT_SES_SMTP_PASSWORDto the SMTP password. SetPUSHLEFT_MAIL_FROMto an address on the domain, in quotes because the file is a shell file:PUSHLEFT_MAIL_FROM='Pushleft <[email protected]>'. The API sends throughemail-smtp.<region>.amazonaws.com:465with implicit TLS. Without all three settings, the API sends no mail. While SES is in its sandbox it delivers only to verified addresses. Then runinstall-control-plane.sh prepare <box1 bundle> <console.tar>. It installs the service users, OpenBao, zot, the env files and the units. It also installs the control-plane Caddy and the management Headscale withmgmt-acl.hujson, and joins box 1 to the tailnet ascp.
- Run
tailscale ip -4on box 1 to get its tailnet address. Allocation is sequential, so the first host is100.64.0.1. Add a DNS A recordcp.internal.<base>for that address. Do not proxy it. - Use that name everywhere a node connects to the control plane:
PUSHLEFT_FLEET_CONTROL_PLANE=https://cp.internal.<base>:8443,PUSHLEFT_FLEET_ENROLL_URL=https://cp.internal.<base>:8444, and the registry, OpenBao and telemetry URLs. The server certificate of the gateway must carry the name. Box 1 and box 2 may be one host (the go-live layout from 2026-11-02 until revenue). Then every “box 2” step runs on box 1:
- Skip the
dbkey and box 2’s tailnet join in step 4. - Use box 1’s tailnet IPv4 wherever these steps say box 2’s (
PUSHLEFT_DB_LISTEN,PUSHLEFT_DB_ADDRESS). - List box 1’s own tailnet IPv4 as a
clusterclient. - Put
PUSHLEFT_CLICKHOUSE_PORT=8543in box 1’sinstall.envbefore the firstinstall-db-node.shrun.
Box 1’s agent gateway holds 8443. install-db-node.sh reads the port from install.env for both its firewall and ClickHouse, and refuses 8443 on that host. Skip the OpenBao copy to box 2.
-
On box 1, run
install-control-plane.sh mgmt-key db. On box 2, join the tailnet with that key, as under every other host. -
On box 2, run
PUSHLEFT_DB_LISTEN=<box 2 tailnet IPv4> install-db-node.sh cluster <db bundle> <box 1 tailnet IPv4>/32. Box 1 is the only client: the mesh controller runs there too. It sets up PostgreSQL with TLS for that client, listening on loopback and the tailnet address only. It also loads the firewall of box 2. Then runPUSHLEFT_DB_LISTEN=<box 2 tailnet IPv4> install-db-node.sh clickhouse <db bundle>(see ClickHouse on the database host). -
On box 1, add
PUSHLEFT_DB_ADDRESS=<box 2 tailnet IPv4>toinstall.envand runinstall-control-plane.sh migrate. -
On box 2, run
install-db-node.sh roles <db bundle>. -
On box 1, run
install-control-plane.sh openbao-initonce and record the recovery share. Then runinstall-control-plane.sh start. It waits until every service answers ready, the error ingest and the central OpenTelemetry Collector included.prepareinstalls the collector (otelcol-contrib, version and SHA-256 pinned indeploy/images.lock.env). It writes to ClickHouse on box 2. -
Join your own devices with
operatorkeys. AddPUSHLEFT_OPERATORStoinstall.env, then runinstall-control-plane.sh access. It allows admin SSH from operator addresses only, installs the git host and loads the private-port firewall of box 1. It also starts the fleet admin API on the tailnet address of box 1, port 8447, for the operator addresses only. Add a DNS A recordadmin.internal.<base>for that tailnet address. Do not proxy it. Its certificate comes from Pushleft’s internal CA. Each operator device must trust that CA before the browser can open the passkey link or the fleet app can connect. Copyinternal-ca.crtfrom the box 1 bundle to the device, then on macOS runsudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain internal-ca.crt, or on Linuxsudo install -m 0644 internal-ca.crt /usr/local/share/ca-certificates/pushleft-internal.crt && sudo update-ca-certificates. Check it withcurl -sS https://admin.internal.<base>:8447/(no certificate error). Then create the first operator. The command prints a passkey link that is valid for 30 minutes, so run it when you are ready to open that link:set -a; . /etc/pushleft/admin.env; pushleft-admin operator add --email <you> --name <name> -
From a machine outside the tailnet, check that the private ports are closed. Each command must fail. (8443–8445 answer publicly since 2026-11-07.)
for port in 5000 8200 8446 5432; do nc -z -w 3 <public IP> $port; done -
From a node, check that the gateway answers. This command must connect.
curl -sk https://cp.internal.<base>:8443/
ClickHouse on the database host
The control-plane ClickHouse runs on the database host (box 2), next to the control-plane PostgreSQL. Do this after box 2 joined the tailnet and ran install-db-node.sh cluster.
-
On box 2, run the installer with the
db/bundle of the key ceremony. Use the tailnet address of box 2:PUSHLEFT_DB_LISTEN=<box 2 tailnet IPv4> deploy/postgres/install-db-node.sh clickhouse <db bundle>It runs the locked ClickHouse image in Docker, with its data in
/var/lib/pushleft/clickhouse. It serves HTTPS only, on portPUSHLEFT_CLICKHOUSE_PORTof that address, with the PostgreSQL certificate of the bundle. The port is 8443 on its own box, and 8543 when box 1 runs on the same host. It applies the schema and creates one user for each control-plane service, with the passwords of the bundle. The admin user can log in only from inside the container (docker exec).PUSHLEFT_CLICKHOUSE_MEMORYsets the memory limit (default8g). You can run it again. -
On box 1,
install-control-plane.sh preparesets up the connection. Caddy listens on127.0.0.1:8123and forwards to box 2 over HTTPS, and the API and agent gateway env files use that address. -
The management ACL lets only
cp@reachdb@:8443. On a shared host the connection stays on the box and no ACL rule is needed. -
Box 2 has its own firewall.
install-db-node.sh cluster(withPUSHLEFT_DB_LISTEN) writes/etc/pushleft/network/db-nftables.nft(tableinet pushleft_db) and enablespushleft-db-firewall.service, beside box 1’spushleft-private-firewallwhen both run on one host. PostgreSQL (5432) answers on loopback and on the tailnet address; ClickHouse (its configured port) answers only on the tailnet address. Ontailscale0both accept only the client addresses of that run. When you runclusteragain with a different list, the firewall changes too.
The mesh controller
There is no mesh host (since 2026-11-03). install-control-plane.sh prepare installs the mesh controller on box 1 from the box 1 bundle, which carries the coordinators’ wildcard pair and the pushleft_mesh password. start starts it and waits until it answers ready. Before prepare, add PUSHLEFT_MESH_ZONE_ID to install.env, and PUSHLEFT_MESH_CLOUDFLARE_TOKEN if the controller’s Cloudflare token cannot edit DNS in the mesh zone. The controller writes preview hostnames into PUSHLEFT_PLATFORM_DNS_ZONE (default <base>), which must hold the preview base; when that is not the mesh zone, also set PUSHLEFT_PLATFORM_DNS_ZONE_ID. prepare stops otherwise.
Each environment’s coordinator runs on one of its own application or database hosts. Those hosts need nothing extra: their agent serves the coordinator host API on its tailnet address once the host has joined the tailnet (mesh coordinators). Box 1’s Caddy serves the coordinators on port 9443, so the box 1 firewall template opens it (rule 2 below).
Box 1 also runs the environment meshes’ DERP relay (relay), so relayed environment traffic stays inside Hetzner. Build derper at the version pinned in deploy/images.lock.env and copy it to /usr/local/bin/derper before prepare. prepare checks its SHA-256. Build on the operator machine, with GOBIN= because go install refuses to cross-compile into a set GOBIN:
GOBIN= CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go install -trimpath tailscale.com/cmd/[email protected]
shasum -a 256 ~/go/bin/linux_amd64/derper # = PUSHLEFT_DERPER_SHA256
DERP rides on 9443 (Caddy, derp.<mesh domain>) and STUN on udp 61478, which rule 7 admits: no extra Robot rule. Its plain-HTTP port 3340 is dropped off loopback by pushleft-private-firewall.
Every other host, before enrollment
-
Install Tailscale from its apt repository (
pkgs.tailscale.com). -
On the control-plane host, create a one-use key with the
nodeuser.hs preauthkeys create --user <id> --expiration 15m -
On the new host, join the tailnet and check the path.
install -m 0600 /dev/stdin /root/mgmt.key # paste the key, then Ctrl-D tailscale up --login-server https://mgmt.<base> --auth-key file:/root/mgmt.key \ --hostname pl-<pool>-<n> --accept-dns=false --accept-routes=false --ssh=false rm /root/mgmt.key tailscale ping cp.internal.<base> # "via <ip>:<port>" is direct, "via DERP" is relayed -
Connect again over the tailnet (
ssh root@<tailnet address>). -
Copy
deploy/sshto the host, for example withscp -r. -
Limit SSH to the tailnet addresses of the operator devices. Never allow the whole tailnet: it holds the nodes that run customer code. The control-plane host does this through
PUSHLEFT_GIT_OPERATORSinstead (operator access).PUSHLEFT_OPERATORS=100.64.0.7,fd7a:115c:a1e0::7 deploy/ssh/install-operator-ssh.sh sshd -t && systemctl reload ssh -
Allow keys only. Ubuntu’s sshd allows passwords by default, and
PermitRootLogin without-passwordcovers only root:printf 'PasswordAuthentication no\nKbdInteractiveAuthentication no\nPermitRootLogin prohibit-password\n' >/etc/ssh/sshd_config.d/10-pushleft-keys-only.conf sshd -t && systemctl reload ssh sshd -T | grep passwordauthentication # "passwordauthentication no"This was set on all six production boxes on 2026-11-05. Before that every one allowed passwords, though no account had one. The same day
install-operator-ssh.shran on all six, pl-cp included, withPUSHLEFT_OPERATORS=92.110.103.161. The office address is used because no operator device is on the management tailnet yet. pl-cp serves no git over SSH yet. Add each operator device’s tailnet address once it joins.
The host now reaches the control plane privately, so the bootstrap in section 5 enrolls over the tailnet. Hosts behind NAT, such as the on-premises build VMs, connect through DERP relays. Image pushes (up to 4 GiB) need a direct path, so allow UDP 41641 to these hosts, or at least outbound UDP.
If the state of the management Headscale is lost, existing tunnels keep working. Rebuild it, then join every host again with a new key.
Operator tailnet access
Decided 2026-11-05 (Waseem): operators keep their own Tailscale tailnet and reach the fleet through pl-cp as the jump host, not by joining the management tailnet.
-
pl-cp runs a second Tailscale,
pushleft-operator-tailscale.service, in userspace networking (no tun, routes or firewall rules, so it cannot interfere with the management tailscaled). It joins the operator’s tailnet astag:pushleft-opswith a one-time, pre-approved auth key from that tailnet’s admin console. The operator’s tailnet policy lets only the operator’s own user reach the tag. -
Inbound connections to its address reach pl-cp’s local ports, so sshd sees
127.0.0.1; pl-cp’sPUSHLEFT_OPERATORStherefore lists127.0.0.1,::1next to the office address. -
The management tailnet policy lets
cp@reachnode@:22(deploy/headscale/mgmt-acl.hujson), and every other host lists pl-cp’s management address (100.64.0.1) inPUSHLEFT_OPERATORS. The login to a box is still authenticated end to end with the operator’s key; pl-cp only forwards the connection. -
To set it up on pl-cp: install
deploy/systemd/pushleft-operator-tailscale.service,systemctl enable --now pushleft-operator-tailscale, then join with the key (in a file,0600) and delete the key file:tailscale --socket=/run/pushleft-operator-tailscale/tailscaled.sock up \ --auth-key=file:<key file> --hostname=pushleft-ops --advertise-tags=tag:pushleft-ops --accept-dns=false -
In the operator’s tailnet policy, allow the operator’s devices
tcp:22andtcp:8447totag:pushleft-ops, and nothing from the tag back. -
From the operator’s machine:
ssh -J root@pushleft-ops [email protected](pl-app-1;.2pl-build-1,.5pl-app-2,.6pl-database-1,.7pl-app-3). -
The Fleet app’s Terminal (on each server’s page, macOS and Linux) runs the Mac’s own
ssh -t -J pl-cp root@<management address>, orssh -t pl-cpfor the control plane. It needs a host entry namedpl-cpin~/.ssh/config:Host pl-cp pushleft-ops HostName <pl-cp's address in the operator's tailnet> User root -
The fleet admin API is reached the same way. On pl-cp,
/etc/pushleft/admin.envsetsPUSHLEFT_ADMIN_BIND=127.0.0.1:8447andPUSHLEFT_ADMIN_OPERATOR_CIDRS=127.0.0.1/32,::1/128, and the DNS A recordadmin.internal.<base>points at pl-cp’s address in the operator’s tailnet. The Fleet app’s Connect screen defaults tohttps://admin.internal.pushleft.dev:8447.install-control-plane.sh accesswrites box 1’s management tailnet address andPUSHLEFT_OPERATORSinto those two settings only on first install. A rerun keeps the values already inadmin.env.
4. Robot firewall
The Robot firewall filters traffic before it reaches the host, like the Cloud firewall that fleet provision attaches. The firewall is stateless, so a TCP ACK rule must allow return traffic for outbound connections. Rules apply in order, and the firewall drops all traffic that no rule matches.
-
scripts/robot-firewall.pycreates the templates and applies them through Robot’s web service (built 2026-11-03). The templates arepl-cp,pl-publicfor the application host andpl-privatefor build and database hosts, with IPv6 filtered and serving nothing. Or create one template per role by hand (Robot → Firewall templates) with these rules:# Name Source Protocol / port Hosts 1 operator ssh your operator CIDRs tcp 22 all, until the host has joined the tailnet; afterwards operators use SSH over the tailnet. To keep a fixed public address as a fallback, keep this rule and also list the address in PUSHLEFT_OPERATORS: sshd refuses addresses that are not listed there2 public ingress any tcp 80, 443; on the control plane also tcp 8443–9443: the agent gateway, enrollment and telemetry (8443–8445, for customer-owned nodes, decided 2026-11-07) and the environment coordinators (9443). The host’s nftables drops every other port of that range control plane (API proxy, mgmt.<base>, coordinators); the application host too, through the migration: its directcurl --resolvechecks against its public IP and the DNS switch need 443. Closing it there afterwards, once every hostname runs through the Cloudflare Tunnel, is a separate decision3 git ssh any tcp 22 control plane only (git.md); sshd admits admins only from the operator devices’ tailnet addresses ( PUSHLEFT_GIT_OPERATORS), never from the whole tailnet, which holds customer nodes4 tailscale any udp 41641 all (direct tailnet paths) 5 icmp any icmp all 6 tcp established any tcp, flags ack, dst port 32768–65535all 7 udp replies any udp, dst port 32768–65535 all (DNS, NTP, tailscale); environment gateways’ fixed direct-path ports (direct paths); on the control plane also the DERP relay’s STUN, udp 61478 Turn on IPv6 filtering in each server’s firewall settings, or the rules cover IPv4 only. After two Robot hosts have joined the tailnet, run
tailscale ping <other host>between them: it should end with a direct path (via <public IP>:41641). A relay can show while the direct path is still being set up, so judge by the end of the ping. Onetailscale statuscan show that early relay too. If it stays on a relay, check rule 4 and that tailscaled listens on 41641 (ss -lunp | grep 41641) on both hosts.Published ports (manifest) need a rule each on the application host, for example
any → tcp 25, 587(a mail server) andany → udp 3478(TURN). A TURN relay range above 32768 is already open through rule 7. The Robot firewall holds at most 10 rules per server, so combine ports where you can. -
Apply the template to each server.
-
Turn on “Hetzner services” in the firewall settings, so the rescue system and monitoring keep working.
-
Leave IPv6, which has its own rule set, at the drop default unless a role needs it.
Private traffic needs no rule, because it all runs inside the tailnet.
An extra address (failover IP)
A role that sends mail directly gets its own IPv4, so its reputation and PTR are its own and survive replacing the host:
- Order a failover IP in Robot and route it to the application host.
- Add it to the host’s public interface (netplan
addresses:, thennetplan apply). - Set its reverse DNS (Robot → IPs → the address) to the mail hostname. For mail on port 25, ask Hetzner to lift the port 25 block on the account.
- Assign it:
pushleft-controller fleet address-add <node id> <address> --org <organization id>.fleet addresseslists them with their claimed ports;fleet address-remove <address>refuses while a port on it is claimed. - The organization names it in its manifest (
address:on each port). To move the address with a new host, route the failover IP there, remove and re-add it on the new node, and redeploy.
5. Enroll
-
On the operator machine, load the fleet environment:
DATABASE_URL,PUSHLEFT_FLEET_ENROLL_URL,PUSHLEFT_FLEET_CONTROL_PLANE,PUSHLEFT_FLEET_GATEWAY_CA_FILE,PUSHLEFT_AGENT_RELEASE_URL,PUSHLEFT_AGENT_VERSIONandPUSHLEFT_RELEASE_PUBKEY. You do not needPUSHLEFT_FLEET_OPERATOR_CIDRSorHETZNER_API_TOKEN. -
Write the bootstrap script.
--private-ipv4is the tailnet address of the host (tailscale ip -4).pushleft-controller fleet enroll-host --pool database --region fsn1 \ --public-ipv4 203.0.113.21 --private-ipv4 100.64.0.21 --model EX63 --out node-21.sh -
Run it on the host within 15 minutes, before the enrollment secret expires. The command waits until the agent has enrolled.
ssh [email protected] 'bash -s' < node-21.sh -
Delete the script. It holds the one-time enrollment secret, which is worthless after use. The file is created
0600and never overwritten.rm node-21.sh -
On a build host, set up Kata Containers now (section 6).
-
Nothing else is done by hand. With the
PUSHLEFT_FLEET_PLATFORM_*settings andPUSHLEFT_FLEET_INTERNAL_CA_FILE(written byinstall-control-plane.shintoapi.env; load it with the fleet environment), the bootstrap of a platform node also:- points the agent at the management tailnet (
PUSHLEFT_FLEET_PLATFORM_CONTROL_PLANE,_ENROLL_URL) instead of the public gateway that customer nodes use; - writes the internal CA as
/etc/pushleft/pki/internal-ca.crtand sets Vault, the registry and the mesh to trust it; - finds the uplink (
PUSHLEFT_PUBLIC_INTERFACE) on application and database hosts; - installs the pinned node collector. The agent asks the gateway for its telemetry client certificate and renews it (
PUSHLEFT_TELEMETRY_CERT_VIA_GATEWAY=1), as on customer nodes; - on an application host, installs the pinned ingress Caddy and creates the empty wildcard files the agent fills from the gateway;
- on a database host, allows the pgvector image (
PUSHLEFT_POSTGRES_REPOSITORIES).
Without those settings the bootstrap writes only the enrollment lines, and the steps above are yours (2026-11-19: pl-app-3, enrolled again, needed eleven agent.env lines copied from another node and a hand-issued telemetry certificate).
- points the agent at the management tailnet (
-
Run
pushleft-agent doctoron the host, with the agent environment loaded:set -a; . /etc/pushleft/agent.env; set +a; pushleft-agent doctor. Without it, doctor stops at “PUSHLEFT_CONTROL_PLANE is not set”. -
Admit the node.
pushleft-controller fleet admit <node-id> -
Check the capacity with
fleet list.
The host joined the tailnet in section 3, so the bootstrap reaches PUSHLEFT_FLEET_ENROLL_URL over it. There is no Vault token file to place: nodes hold none. Application and database hosts set only PUSHLEFT_VAULT_ADDR.
admit sets the capacity of the node from the facts its agent reported (CPUs, memory, free space on the data partition). It subtracts the host reserve of 0.5 vCPU, 1 GiB and 20 GB. A build host must keep at least one build after the reserve (2 CPUs, 4096 MiB, 20 GB, so at least 3 vCPUs). admit refuses a smaller host and leaves it unschedulable.
The control-plane host has no agent, so install-control-plane.sh gives it its own host collector for the Fleet app. prepare writes deploy/observability/otelcol-cp-host.yaml as /etc/pushleft-otel/otelcol.yaml, with the short hostname as host.id, and installs pushleft-otelcol.service and pushleft-host-health.timer. start starts the collector, which reports to the central collector on loopback. It also starts the timer, which writes pushleft-agent disk-health to /var/lib/pushleft-host-health/<host>.json every 10 minutes for the admin API (PUSHLEFT_ADMIN_HOST_HEALTH_DIR).
6. Build hosts: Kata Containers
Every build runs in its own Kata Containers VM (Kata host setup). While PUSHLEFT_REQUIRE_VM_ISOLATED_BUILD is on (the default), only such hosts get builds. Robot servers expose KVM. Hetzner Cloud servers do not.
-
Check for KVM. If it is missing, enable SVM (AMD) or VT-x (Intel) in the BIOS through a Robot KVM console session.
apt-get install -y cpu-checker && kvm-ok -
Install Kata Containers 3.x from the release tarball (
kata-static-<version>-amd64.tar.zst) under/opt/kata. Verify the checksum first. -
Link
/opt/kata/bin/containerd-shim-kata-v2and/opt/kata/bin/kata-runtimeinto/usr/local/bin. -
Raise the wait of the Kata agent for hotplugged devices to 60 s. A build asks for 12 GiB, and to online that memory can take longer than the default 3 s wait for the network card. The guest then has no network (found on the staging fleet).
install -d /etc/kata-containers sed -E 's/^kernel_params = "(.*)"/kernel_params = "\1 agent.hotplug_timeout=60"/' \ /opt/kata/share/defaults/kata-containers/configuration-qemu.toml >/etc/kata-containers/configuration.toml -
Run
kata-runtime check. It must report that the system can run Kata Containers. -
Add the runtime to
/etc/docker/daemon.jsonand restart Docker. Keep theuserns-remapline that the bootstrap wrote (workload hardening). The file then reads:{ "userns-remap": "default", "runtimes": { "kata": { "runtimeType": "io.containerd.kata.v2" } } }The build container runs with
--userns=host, because the VM is the boundary of Kata. The remap does not affect builds. Under the remap, a Kata container without--userns=hostfails withfailed to create shim task: EPERM, so the check below passes it too. -
Check the runtime. The first command lists
kata. The second prints the kernel version of the guest.docker info --format '{{json .Runtimes}}' docker run --rm --userns=host --runtime kata alpine uname -r -
Set these values in
/etc/pushleft/agent.env:PUSHLEFT_BUILDKIT_IMAGE=moby/buildkit@sha256:79cc6476ab1a3371c9afd8b44e7c55610057c43e18d9b39b68e2b0c2475cc1b6(rootful v0.25.1)PUSHLEFT_BUILD_RUNTIME=kataPUSHLEFT_BUILD_SCAN_IMAGE=aquasec/trivy@sha256:2416d1f74045b57274df1a340eee6b510cacd73c66a6ccb8da092b117eb7d222(Trivy 0.76.0). Required withPUSHLEFT_BUILD_RUNTIME: without it the agent exits at start.PUSHLEFT_BUILD_DNS, only if the host resolves through a private address. A Kata build cannot use the embedded DNS of Docker. So the agent writes theresolv.confof the build from the public upstream resolvers of the host. Hetzner’s 185.12.64.1/2 qualify. With a private resolver and noPUSHLEFT_BUILD_DNS, the agent refuses to start.
-
Do not set
PUSHLEFT_VAULT_ADDRorPUSHLEFT_VAULT_TOKEN_FILE, and do not create/etc/pushleft/credentials/vault-token. Build hosts run tenant code and hold no tenant secret.pushleft-agent doctorwarns about a token file on any host. -
Install the
nftablespackage. The agent installs its own nftables table (inet pl_build) for the build bridges, and needsnftat/usr/sbin/nft. -
Give the build cache its own XFS partition with project quotas, so a full cache cannot fill the root disk. The simplest way is a fourth
installimageline in section 2,PART /var/lib/pushleft/build-cache xfs all, afterPART /var/lib/pushleft/data xfs 120G. Then addprjquotato its fstab line too. Remount it as section 2 step 3 does for the data partition (XFS reads quota options only at mount time):sed -i 's#\(/var/lib/pushleft/build-cache\s\+xfs\s\+\)defaults#\1defaults,prjquota#' /etc/fstab umount /var/lib/pushleft/build-cache && mount /var/lib/pushleft/build-cache xfs_quota -x -c 'state -p' /var/lib/pushleft/build-cache # "Accounting: ON", "Enforcement: ON"On a host installed without it, for example with a free partition
/dev/md3:mkfs.xfs -f /dev/md3 echo '/dev/md3 /var/lib/pushleft/build-cache xfs defaults,prjquota 0 2' >> /etc/fstab mkdir -p /var/lib/pushleft/build-cache && mount /var/lib/pushleft/build-cache && chmod 700 /var/lib/pushleft/build-cache xfs_quota -x -c 'state -p' /var/lib/pushleft/build-cache # "Accounting: ON", "Enforcement: ON" -
Once that state shows enforcement on, set the cache values in
/etc/pushleft/agent.env:PUSHLEFT_BUILD_CACHE_ROOT=/var/lib/pushleft/build-cache,PUSHLEFT_BUILD_CACHE_QUOTA=xfs, and optionallyPUSHLEFT_BUILD_CACHE_PROJECT_MAX_MB(default 10240). The agent refuses to start if the partition does not enforce project quotas.
New build hosts warm their cache from the cache blob store of the gateway (PUSHLEFT_CACHE_S3_* or the shared bucket). Build hosts get no bucket credential.
Keep build networking NAT only. Each VM sits behind the network namespace of its build container, and leaves through the Docker bridge and masquerade. Never bridge VMs onto the public NIC (macvlan, or a host bridge with enp9s0). Hetzner flags unknown MAC addresses on the switch port of a dedicated server and can lock the server. Builds cannot reach the management tailnet either: the pl_build table drops 100.64.0.0/10 and every host address.
Nothing in this section has run on a Robot host yet. Steps 2 to 10 run on every staging install with Kata 3.32 in a nested amd64 VM (staging).
Check it worked
xfs_quota -x -c 'state -p' /var/lib/pushleft/datashows “Accounting: ON” and “Enforcement: ON”.pushleft-agent doctorpasses, including the backup spool line.pushleft-controller fleet listshows the node admitted, with capacity.- On a build host,
docker run --rm --userns=host --runtime kata alpine uname -rprints the guest kernel.
Bringing a lost host back
When the agent of a lost host polls again, the host returns to service (return of a lost node). Enroll a decommissioned or wiped host again as a new node:
fleet decommission <old node>. It refuses while the node holds an allocation. A deleted or expired volume on a lost node is released by the scheduler without a final backup (its last backup is kept), so only a volume that a role still mounts blocks it: remove the role from the manifest and deploy, thenpl volume delete.- On the host, remove the old node’s containers:
docker rm -f $(docker ps -aq). Its empty environment bridges can stay: the agent replaces an empty bridge of its own whose spec is out of date. - Enroll and admit as in section 5.
pushleft-agent enrollreplaces a staged identity of another node, so the old, revoked certificate can stay where it is. - Point placement preferences at the new node:
fleet place-environment <project> <environment> --node <new node>.
Retiring a Robot host
- Drain the node with
fleet drain. - Wait until
fleet listshows it empty. - Run
fleet decommission <node>. This retires the row and revokes the certificates. It does not change the server. - Destroys data: wipe the server with
installimageagain, or cancel it in Robot.
To replace a host, enroll the new host, admit it, then drain the old one. fleet replace refuses BYO nodes.