pushleft
Deploy your first app
Manual contents
Handbook
HandbookPushleftArchitecture
Operate
Install the control planeYour own serversFleetOperations and alertsAuthenticationDomains and DNSUsage and budgetsScheduled jobsAbuse controlsAI provider proxyCustomer mailErrors and performanceLogs and metricsRetentionObject storageContainer registryGit server
Develop
The .pl.yaml manifestOnboard with an AI agentThe pl up demoBuild executorSource ingestionAgent gatewayManaged databasesSecurity findingsTunnels and shellsDev workspacesPrivate hostnamesMesh coordinatorsDesign baseline
Runbooks
Control-plane host failoverControl-plane PostgreSQL failoverControl-plane PostgreSQL restoreCustom domains and certificatesGateway key compromiseIngress host failureNode busyNode disk fullNode drain and replacementNode lostRegistry outageHetzner Robot hostsSecret rotationSecurity incidentStaging fleet on ProxmoxStaging build cacheStuck deploymentVault/OpenBao backup and recovery
Manual · docs/runbooks/vault-recovery.md

Vault/OpenBao backup and recovery

Use this runbook to set up Vault snapshots, to answer the vault_snapshot_stale alert, or to restore Vault after its host or data is lost.

Vault/OpenBao holds:

  • The backup key of every organization (organizations/<org>/backup-key).
  • Every admin and principal password of the managed databases.
  • The mesh pre-auth keys (mesh coordinators).

Warning: if Vault is lost without a backup, no sealed tenant backup can be opened again.

The commands use bao. With HashiCorp Vault, use vault with the same subcommands.

Key hierarchy

LayerProtects againstHeld by
Integrated-storage (raft) snapshot, hourlyLost or corrupted Vault host or dataThe off-host target (Storage Box or object store) and a 35-day local copy
age encryption of each snapshot to the offline operator keyA leaked snapshot or a compromised off-host targetThe offline operator key: never on any server
Vault’s own barrier encryption inside the snapshotSomeone who has both the snapshot and the operator key, but not the unseal sharesThe unseal-key holders (table below)
Optional backup-key escrowTotal Vault loss: no usable snapshot, or the unseal shares are lostThe offline operator key

A restore needs three things, each in a different place:

  • A snapshot from off-host.
  • The offline operator key.
  • A quorum of unseal shares.

None of them is on the Vault host or the control-plane host.

Key custody

Record the holders here when the production Vault is initialized (bao operator init -key-shares=5 -key-threshold=3, or the recovery-key equivalent with an auto-unseal).

SecretHolderMedium and locationLast verified
Unseal (or recovery) share 1<name><for example: sealed envelope, home safe><date>
Unseal share 2<name><medium, location><date>
Unseal share 3<name><medium, location><date>
Unseal share 4<name><medium, location><date>
Unseal share 5<name><medium, location><date>
Offline operator key (age identity), copy A<name><for example: YubiKey with age-plugin-yubikey, or paper in a safe><date>
Offline operator key, copy B<name><medium, location><date>

Rules:

  • Never store a share, the root token or the operator private key on a server or in this repository. Never store them in a shared password vault that one person can open.
  • Revoke the initial root token after setup (bao token revoke -self). When you need one, generate a new one from the shares (bao operator generate-root).
  • No person holds a quorum of shares.
  • Each holder checks once a quarter that their share and medium still exist, and updates “Last verified”.
  • When a holder leaves, rekey (bao operator rekey) and update the table the same day.

A service token stopped working

Each service token is periodic, with a period of 72 hours. A running service renews its own token. A service that is stopped for longer than 72 hours loses its token, and OpenBao answers permission denied.

  1. On the OpenBao host, set BAO_ADDR and BAO_CACERT as for install-openbao.sh restore.

  2. Pipe the recovery share in:

    age -d -i <offline key> <recovery share file> | jq -r '.recovery_keys_b64[0]' | ssh <host> 'BAO_ADDR=... BAO_CACERT=... install-openbao.sh reissue'
  3. The command lists each token as ok or expired. It issues a new token for each expired one, and keeps the working ones. The root token it uses is revoked when it ends.

  4. Restart each service whose token was reissued.

Install on box 1

Box 1’s OpenBao unseals itself through an AWS KMS key at every start, so a reboot needs no operator. The key stays in AWS. Whoever can call Decrypt on it can unseal box 1’s storage, so its IAM user can do nothing else. Since OpenBao 2.7 the AWS seal is an external plugin (openbao-plugins kms-aws). install-openbao.sh downloads it and checks it against pinned checksums.

Before this, run the key ceremony on an offline machine: deploy/install-keys.sh writes box1/, db/, mesh/ and safe/. Only box1/ goes to box 1. The internal CA key and the release signing key stay in safe/, encrypted to the offline operator key, in two copies. test:install-keys checks the chains, that offline keys stay out of the server bundles, the file modes and the release signature. Use box1/cp.crt and cp.key for OpenBao’s TLS below.

  1. In AWS (eu-central-1), create a symmetric encryption key, pushleft-box1-openbao. Create an IAM user with only this policy, and an access key for it:

    {"Version": "2012-10-17", "Statement": [{"Effect": "Allow",
      "Action": ["kms:Encrypt", "kms:Decrypt", "kms:DescribeKey"],
      "Resource": "arn:aws:kms:eu-central-1:<account>:key/<key id>"}]}
  2. On box 1, copy openbao.env.example to /etc/pushleft/openbao.env and set the key id, the access key and the API address. Then run:

    deploy/vault/install-openbao.sh config <tls.crt> <tls.key>   # the internal CA's vault certificate
    BAO_ADDR=https://vault.internal.<zone>:8200 BAO_CACERT=<internal CA> deploy/vault/install-openbao.sh init
  3. init prints one recovery share and the root token, once, to the terminal. Write the share down twice, for Waseem and for the safe (custody 1-of-2; OpenBao refuses two shares with a threshold of 1). Do not store the root token: init revokes it when setup ends. Setup enables organizations/, writes every policy in deploy/vault/, and writes each service token to its file.

The recovery share cannot unseal. It authorizes bao operator generate-root and rekey. If KMS is unreachable or the key is disabled, OpenBao stays sealed and the services fail their readiness checks, which pages. OpenBao unseals at its next start once KMS answers again.

test:openbao (scripts/openbao-autounseal-e2e.sh) proves this against a KMS stand-in, with the same rendered config, plugin and init. It checks the service tokens and their policies, that the root token is revoked, and an unseal after a restart. It also checks that OpenBao stays sealed without KMS and unseals when KMS is back, and that a second init is refused.

Set up snapshots (once per Vault)

  1. On an offline machine, create the offline operator key. Keep two copies (see Key custody).

    age-keygen -o operator.key
  2. Copy only the public line (age1...) to the control-plane host as /etc/pushleft/vault-snapshot/recipients.txt. age-keygen -y operator.key also prints it.

    • To encrypt to several keys, put one recipient on each line.
    • scripts/vault-snapshot.sh refuses a recipients file that contains a private key.
  3. Create the snapshot policy and token. The token can read sys/storage/raft/snapshot and nothing else (deploy/vault/pushleft-snapshot.hcl).

    bao policy write pushleft-snapshot deploy/vault/pushleft-snapshot.hcl
    bao token create -orphan -period=72h -policy=pushleft-snapshot -field=token \
      | sudo install -m 0600 -o pushleft-vault-snapshot -g pushleft-vault-snapshot /dev/stdin /etc/pushleft/vault-snapshot/token

    The token is periodic, and each run renews it. If the timer stops for 72 h, the token expires. Create a new one with the same command.

  4. Set the off-host target.

    • The default copy command puts the encrypted snapshot into the spool, /var/backups/pushleft/offsite/vault/.
    • pushleft-offsite.timer sends it to the R2 bucket under <host prefix>/vault/ (decided 2026-10-29, offsite.env.example).
    • The lifecycle rule of the bucket must keep objects for at least 35 days, the same as the local copy.
    • Box 2 also keeps a copy (decided 2026-10-30), so a snapshot survives the loss of box 1 even before it reaches R2. On box 1, run install-control-plane.sh vault-copy-key. It makes an SSH key for the snapshot user, reads the host key of box 2 over the tailnet, and prints a command. Run that command on box 2: install-db-node.sh vault-copy "<key>" <box 1 tailnet address>. The key can only store one snapshot under its own name in /var/backups/pushleft/vault-copies, and only from box 1. Box 2 deletes copies older than 35 days. The copy command in the example then pushes each snapshot to box 2 after the spool. Replace the host name in it with PUSHLEFT_DB_HOST.
    • To use a different target, set PUSHLEFT_VAULT_SNAPSHOT_COPY_COMMAND (vault-snapshot.env.example).
  5. Install the units. The script needs curl and age. Edit /etc/pushleft/vault-snapshot.env after you copy it.

    useradd --system --home /nonexistent --shell /usr/sbin/nologin pushleft-vault-snapshot
    install -d -m 0700 -o pushleft-vault-snapshot -g pushleft-vault-snapshot /etc/pushleft/vault-snapshot
    install -m 0755 scripts/vault-snapshot.sh /usr/local/lib/pushleft/scripts/   # needs curl and age
    install -m 0644 deploy/systemd/vault-snapshot.env.example /etc/pushleft/vault-snapshot.env   # then edit
    install -m 0644 deploy/systemd/pushleft-vault-snapshot.{service,timer} /etc/systemd/system/
    systemctl daemon-reload && systemctl enable --now pushleft-vault-snapshot.timer
    systemctl start pushleft-vault-snapshot.service && journalctl -u pushleft-vault-snapshot -n 5
  6. Turn on the alert. In /etc/pushleft/ops-alerts.env, set:

    PUSHLEFT_OPS_VAULT_SNAPSHOT_STATUS_FILE=/var/lib/pushleft-vault-snapshot/last-success

    Run the snapshot timer on the same host as pushleft-ops-alerts.timer.

  7. Run the restore once as a drill on a scratch instance before you rely on it. Record the date and duration here.

Retention

The local copy keeps 35 days (PUSHLEFT_VAULT_SNAPSHOT_RETENTION_DAYS). Tenant database backups keep 14 days and the retention window of released volumes is 30 days (retention.md).

A backup key never changes after it is created. Any snapshot taken after a key was created can open every backup sealed with that key. The long retention lets you go back to a snapshot from before a corrupted or tampered Vault.

Alert: vault_snapshot_stale (ticket)

scripts/ops-alerts.sh fires this alert when last-success is missing or older than 2 hours (PUSHLEFT_OPS_VAULT_SNAPSHOT_MAX_AGE_SECONDS). The script writes the file only after the snapshot is encrypted and the off-host copy succeeds.

  1. Read the status and the journal of the service.

    systemctl status pushleft-vault-snapshot.service
    journalctl -u pushleft-vault-snapshot -n 50
  2. Find the cause in the journal and fix it:

    Journal showsCauseFix
    403 or permission denied from renew-selfThe token expired or was revokedCreate a new token (setup step 3)
    Connection errorsVault is down or sealed. The gateway also cannot hand out backup keys, so new databases get no backupsUnseal Vault first
    Copy command failedThe spool partition is full, or the shipper failedCheck free space on the spool partition. If offsite_copy_stale also fires, read journalctl -u pushleft-offsite
    snapshot is implausibly smallVault returned an empty bodyCheck bao operator raft list-peers and the Vault logs
  3. Start the service by hand.

    systemctl start pushleft-vault-snapshot.service
  4. Check that last-success has a new time.

Restore

Use this when the Vault host or its data is lost, or the data is bad. Practise it as a drill against a scratch instance first. Warning: never point a drill at the production address.

Box 1 (AWS KMS auto-unseal). Rebuild box 1 with install-control-plane.sh prepare, then run install-control-plane.sh openbao-restore <snapshot> with the recovery share on stdin instead of openbao-init. That runs steps 4 to 7 and 10 below in one go:

  • a throwaway init, then the snapshot restored over it;
  • KMS unseals the restored data, since the KMS key is unchanged;
  • a root token generated with the original recovery share;
  • new service tokens, then the root token revoked.

OpenBao 2.5.3 and later refuse the unauthenticated root-generation endpoints on its listeners. Box 1 serves them only on a local socket that only root reaches (/var/lib/openbao/admin.sock, mode 600). test:openbao and staging.sh rehearse recover prove this: a secret written before the loss is read back after box 1 is destroyed and rebuilt. Then do steps 8, 9, 11 and 12.

  1. Fetch the newest good snapshot from the off-host target onto the operator’s workstation. Do not fetch it onto a server.

  2. Decrypt it there with the offline key.

    age -d -i operator.key -o vault.snap vault-<timestamp>.snap.age

    The decrypted file is still barrier-encrypted. Delete it when you are done.

  3. Start a fresh instance with integrated storage and the same config as production.

  4. Initialize it as a throwaway, then unseal it and log in with the throwaway key and root token.

    bao operator init -key-shares=1 -key-threshold=1
  5. Destroys data: restore the snapshot. This replaces all data on the instance.

    bao operator raft snapshot restore -force vault.snap

    -force is required because the snapshot comes from the keyring of another cluster. The instance then seals itself, and the throwaway key and token stop working.

  6. Unseal with the original shares. The holders in Key custody each run bao operator unseal until the threshold is reached.

  7. If the original root token was revoked at setup and you need one, the same holders run bao operator generate-root.

  8. Check that a known organization’s backup key opens a backup.

    bao kv get -field=key organizations/<org>/backup-key           # for a known org
    pushleft-agent open-backup --organization <org> --source volume-<env>-<name> \
      --backup-id <id> --output /tmp/check.tar --vault-address https://<restored> --vault-token-file <operator token>

    open-backup verifies the manifest hash and every AEAD tag again. A wrong key cannot produce silent garbage.

  9. Point PUSHLEFT_VAULT_ADDR of every service that uses Vault at the restored instance, or move the DNS name (vault.internal.<base>).

  10. Issue new service tokens: the agent gateway, API, controller, scheduler, AI proxy, mail, git backup and snapshot tokens (the list in deploy/vault/install-openbao.sh). Tokens created after the snapshot are not in the restored data (secret rotation).

  11. Restart those services: pushleft-agent-gateway pushleft-api pushleft-controller pushleft-scheduler, and the AI proxy, mail and git backup where they run.

  12. Deal with the gap after the snapshot. Backup keys and database passwords created after it are missing.

    • The next request creates a missing backup key again. Backups sealed with the lost key cannot be opened. Escrow covers this case.
    • For missing database passwords, rotate the credentials of those instances.

npm run test:vault-backup runs steps 1 to 6 and step 8 end to end. It uses real OpenBao with raft storage, a real agent-sealed backup, the shipped snapshot script, destroy, restore, the original share and open-backup.

Check it worked

  • last-success is less than 2 hours old, and vault_snapshot_stale does not fire.
  • After a restore, pushleft-agent open-backup writes /tmp/check.tar without an error.
  • The /readyz of the gateway, the API and the scheduler answer 2xx (Service not ready).

Optional: backup-key escrow

scripts/vault-escrow-backup-keys.sh <output.age> reads the backup key of every organization. It writes one age file, encrypted to the offline operator key. The keys go from Vault through jq into age, so no plaintext is written.

Run it on the operator’s workstation, with a short-lived token that has deploy/vault/pushleft-backup-key-escrow.hcl. That policy lists organizations and reads backup-key, and nothing else.

  1. Create the policy and a 15-minute token. Keep the token file at mode 0600, and delete it after use.

    bao policy write pushleft-backup-key-escrow deploy/vault/pushleft-backup-key-escrow.hcl
    bao token create -policy=pushleft-backup-key-escrow -ttl=15m -field=token > ~/.escrow-token   # 0600, delete after
  2. Write the escrow file.

    VAULT_ADDR=https://vault... VAULT_TOKEN_FILE=~/.escrow-token PUSHLEFT_VAULT_SNAPSHOT_RECIPIENTS_FILE=recipients.txt \
      scripts/vault-escrow-backup-keys.sh escrow-$(date -u +%Y%m%d).age
  3. Store the output off-host, with the snapshots.

Keys never change, so run it after new organizations get databases. Weekly is enough. The script stops on any read error other than “no key yet”, so it never skips a key without telling you.

Import after a total loss

Import into a new, empty Vault with the organizations KV v2 mount. Each key is written create-only, so an existing key is never replaced.

age -d -i operator.key escrow-<date>.age | while read -r org key; do
  [ "$org" = "#" ] && continue
  bao kv put -cas=0 organizations/$org/backup-key key="$key"
done

Trade-off

  • Without escrow, an attacker needs a snapshot, the operator key and a quorum of unseal shares to read backup keys.
  • With escrow, an attacker with the escrow file and the sealed backups opens the backup key of every organization with the operator key alone. The operator key becomes the most valuable secret.
  • Use escrow only if that key is held as strongly as the unseal shares: a hardware token, two named holders, never online.
  • The gain: tenant backups survive the loss of Vault and of the unseal quorum. Snapshots alone cannot give that.
  • Escrow covers backup keys only. After such a loss, rotate the database passwords.

Proof status

  • Proven by npm run test:vault-backup (Docker, OpenBao 2.4.1, single share):
    • The snapshot policy cannot read tenant secrets.
    • The snapshot is encrypted before it touches disk, and it is a valid raft archive.
    • A destroyed Vault restored from it seals itself and opens only with the original share.
    • The restored key opens an agent-sealed backup. An escrow import into an empty Vault opens it too.
    • vault_snapshot_stale fires when the status file is stale or missing.
  • Not proven: the systemd units on a real host, a real Storage Box or S3 copy (the test copies into a local directory) and off-host retention. The tests also skip a real unseal ceremony with named holders, restore time at production size, and HashiCorp Vault (the API calls are the same).
  • Not built: a 3-node raft cluster, and write-once off-host retention that a compromised control-plane host cannot delete. With one instance, a host loss is an outage until the restore is done.
Rendered from the repository at build time. Edit docs/runbooks/vault-recovery.md to change it.