Vault/OpenBao backup and recovery
Use this runbook to set up Vault snapshots, to answer the vault_snapshot_stale alert, or to restore Vault after its host or data is lost.
Vault/OpenBao holds:
- The backup key of every organization (
organizations/<org>/backup-key). - Every admin and principal password of the managed databases.
- The mesh pre-auth keys (mesh coordinators).
Warning: if Vault is lost without a backup, no sealed tenant backup can be opened again.
The commands use bao. With HashiCorp Vault, use vault with the same subcommands.
Key hierarchy
| Layer | Protects against | Held by |
|---|---|---|
| Integrated-storage (raft) snapshot, hourly | Lost or corrupted Vault host or data | The off-host target (Storage Box or object store) and a 35-day local copy |
age encryption of each snapshot to the offline operator key | A leaked snapshot or a compromised off-host target | The offline operator key: never on any server |
| Vault’s own barrier encryption inside the snapshot | Someone who has both the snapshot and the operator key, but not the unseal shares | The unseal-key holders (table below) |
| Optional backup-key escrow | Total Vault loss: no usable snapshot, or the unseal shares are lost | The offline operator key |
A restore needs three things, each in a different place:
- A snapshot from off-host.
- The offline operator key.
- A quorum of unseal shares.
None of them is on the Vault host or the control-plane host.
Key custody
Record the holders here when the production Vault is initialized (bao operator init -key-shares=5 -key-threshold=3, or the recovery-key equivalent with an auto-unseal).
| Secret | Holder | Medium and location | Last verified |
|---|---|---|---|
| Unseal (or recovery) share 1 | <name> | <for example: sealed envelope, home safe> | <date> |
| Unseal share 2 | <name> | <medium, location> | <date> |
| Unseal share 3 | <name> | <medium, location> | <date> |
| Unseal share 4 | <name> | <medium, location> | <date> |
| Unseal share 5 | <name> | <medium, location> | <date> |
Offline operator key (age identity), copy A | <name> | <for example: YubiKey with age-plugin-yubikey, or paper in a safe> | <date> |
| Offline operator key, copy B | <name> | <medium, location> | <date> |
Rules:
- Never store a share, the root token or the operator private key on a server or in this repository. Never store them in a shared password vault that one person can open.
- Revoke the initial root token after setup (
bao token revoke -self). When you need one, generate a new one from the shares (bao operator generate-root). - No person holds a quorum of shares.
- Each holder checks once a quarter that their share and medium still exist, and updates “Last verified”.
- When a holder leaves, rekey (
bao operator rekey) and update the table the same day.
A service token stopped working
Each service token is periodic, with a period of 72 hours. A running service renews its own token. A service that is stopped for longer than 72 hours loses its token, and OpenBao answers permission denied.
-
On the OpenBao host, set
BAO_ADDRandBAO_CACERTas forinstall-openbao.sh restore. -
Pipe the recovery share in:
age -d -i <offline key> <recovery share file> | jq -r '.recovery_keys_b64[0]' | ssh <host> 'BAO_ADDR=... BAO_CACERT=... install-openbao.sh reissue' -
The command lists each token as
okorexpired. It issues a new token for each expired one, and keeps the working ones. The root token it uses is revoked when it ends. -
Restart each service whose token was reissued.
Install on box 1
Box 1’s OpenBao unseals itself through an AWS KMS key at every start, so a reboot needs no operator. The key stays in AWS. Whoever can call Decrypt on it can unseal box 1’s storage, so its IAM user can do nothing else. Since OpenBao 2.7 the AWS seal is an external plugin (openbao-plugins kms-aws). install-openbao.sh downloads it and checks it against pinned checksums.
Before this, run the key ceremony on an offline machine: deploy/install-keys.sh writes box1/, db/, mesh/ and safe/. Only box1/ goes to box 1. The internal CA key and the release signing key stay in safe/, encrypted to the offline operator key, in two copies. test:install-keys checks the chains, that offline keys stay out of the server bundles, the file modes and the release signature. Use box1/cp.crt and cp.key for OpenBao’s TLS below.
-
In AWS (eu-central-1), create a symmetric encryption key,
pushleft-box1-openbao. Create an IAM user with only this policy, and an access key for it:{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": ["kms:Encrypt", "kms:Decrypt", "kms:DescribeKey"], "Resource": "arn:aws:kms:eu-central-1:<account>:key/<key id>"}]} -
On box 1, copy
openbao.env.exampleto/etc/pushleft/openbao.envand set the key id, the access key and the API address. Then run:deploy/vault/install-openbao.sh config <tls.crt> <tls.key> # the internal CA's vault certificate BAO_ADDR=https://vault.internal.<zone>:8200 BAO_CACERT=<internal CA> deploy/vault/install-openbao.sh init -
initprints one recovery share and the root token, once, to the terminal. Write the share down twice, for Waseem and for the safe (custody 1-of-2; OpenBao refuses two shares with a threshold of 1). Do not store the root token:initrevokes it when setup ends. Setup enablesorganizations/, writes every policy indeploy/vault/, and writes each service token to its file.
The recovery share cannot unseal. It authorizes bao operator generate-root and rekey. If KMS is unreachable or the key is disabled, OpenBao stays sealed and the services fail their readiness checks, which pages. OpenBao unseals at its next start once KMS answers again.
test:openbao (scripts/openbao-autounseal-e2e.sh) proves this against a KMS stand-in, with the same rendered config, plugin and init. It checks the service tokens and their policies, that the root token is revoked, and an unseal after a restart. It also checks that OpenBao stays sealed without KMS and unseals when KMS is back, and that a second init is refused.
Set up snapshots (once per Vault)
-
On an offline machine, create the offline operator key. Keep two copies (see Key custody).
age-keygen -o operator.key -
Copy only the public line (
age1...) to the control-plane host as/etc/pushleft/vault-snapshot/recipients.txt.age-keygen -y operator.keyalso prints it.- To encrypt to several keys, put one recipient on each line.
scripts/vault-snapshot.shrefuses a recipients file that contains a private key.
-
Create the snapshot policy and token. The token can read
sys/storage/raft/snapshotand nothing else (deploy/vault/pushleft-snapshot.hcl).bao policy write pushleft-snapshot deploy/vault/pushleft-snapshot.hcl bao token create -orphan -period=72h -policy=pushleft-snapshot -field=token \ | sudo install -m 0600 -o pushleft-vault-snapshot -g pushleft-vault-snapshot /dev/stdin /etc/pushleft/vault-snapshot/tokenThe token is periodic, and each run renews it. If the timer stops for 72 h, the token expires. Create a new one with the same command.
-
Set the off-host target.
- The default copy command puts the encrypted snapshot into the spool,
/var/backups/pushleft/offsite/vault/. pushleft-offsite.timersends it to the R2 bucket under<host prefix>/vault/(decided 2026-10-29,offsite.env.example).- The lifecycle rule of the bucket must keep objects for at least 35 days, the same as the local copy.
- Box 2 also keeps a copy (decided 2026-10-30), so a snapshot survives the loss of box 1 even before it reaches R2. On box 1, run
install-control-plane.sh vault-copy-key. It makes an SSH key for the snapshot user, reads the host key of box 2 over the tailnet, and prints a command. Run that command on box 2:install-db-node.sh vault-copy "<key>" <box 1 tailnet address>. The key can only store one snapshot under its own name in/var/backups/pushleft/vault-copies, and only from box 1. Box 2 deletes copies older than 35 days. The copy command in the example then pushes each snapshot to box 2 after the spool. Replace the host name in it withPUSHLEFT_DB_HOST. - To use a different target, set
PUSHLEFT_VAULT_SNAPSHOT_COPY_COMMAND(vault-snapshot.env.example).
- The default copy command puts the encrypted snapshot into the spool,
-
Install the units. The script needs
curlandage. Edit/etc/pushleft/vault-snapshot.envafter you copy it.useradd --system --home /nonexistent --shell /usr/sbin/nologin pushleft-vault-snapshot install -d -m 0700 -o pushleft-vault-snapshot -g pushleft-vault-snapshot /etc/pushleft/vault-snapshot install -m 0755 scripts/vault-snapshot.sh /usr/local/lib/pushleft/scripts/ # needs curl and age install -m 0644 deploy/systemd/vault-snapshot.env.example /etc/pushleft/vault-snapshot.env # then edit install -m 0644 deploy/systemd/pushleft-vault-snapshot.{service,timer} /etc/systemd/system/ systemctl daemon-reload && systemctl enable --now pushleft-vault-snapshot.timer systemctl start pushleft-vault-snapshot.service && journalctl -u pushleft-vault-snapshot -n 5 -
Turn on the alert. In
/etc/pushleft/ops-alerts.env, set:PUSHLEFT_OPS_VAULT_SNAPSHOT_STATUS_FILE=/var/lib/pushleft-vault-snapshot/last-successRun the snapshot timer on the same host as
pushleft-ops-alerts.timer. -
Run the restore once as a drill on a scratch instance before you rely on it. Record the date and duration here.
Retention
The local copy keeps 35 days (PUSHLEFT_VAULT_SNAPSHOT_RETENTION_DAYS). Tenant database backups keep 14 days and the retention window of released volumes is 30 days (retention.md).
A backup key never changes after it is created. Any snapshot taken after a key was created can open every backup sealed with that key. The long retention lets you go back to a snapshot from before a corrupted or tampered Vault.
Alert: vault_snapshot_stale (ticket)
scripts/ops-alerts.sh fires this alert when last-success is missing or older than 2 hours (PUSHLEFT_OPS_VAULT_SNAPSHOT_MAX_AGE_SECONDS). The script writes the file only after the snapshot is encrypted and the off-host copy succeeds.
-
Read the status and the journal of the service.
systemctl status pushleft-vault-snapshot.service journalctl -u pushleft-vault-snapshot -n 50 -
Find the cause in the journal and fix it:
Journal shows Cause Fix 403orpermission deniedfromrenew-selfThe token expired or was revoked Create a new token (setup step 3) Connection errors Vault is down or sealed. The gateway also cannot hand out backup keys, so new databases get no backups Unseal Vault first Copy command failed The spool partition is full, or the shipper failed Check free space on the spool partition. If offsite_copy_stalealso fires, readjournalctl -u pushleft-offsitesnapshot is implausibly smallVault returned an empty body Check bao operator raft list-peersand the Vault logs -
Start the service by hand.
systemctl start pushleft-vault-snapshot.service -
Check that
last-successhas a new time.
Restore
Use this when the Vault host or its data is lost, or the data is bad. Practise it as a drill against a scratch instance first. Warning: never point a drill at the production address.
Box 1 (AWS KMS auto-unseal). Rebuild box 1 with install-control-plane.sh prepare, then run install-control-plane.sh openbao-restore <snapshot> with the recovery share on stdin instead of openbao-init. That runs steps 4 to 7 and 10 below in one go:
- a throwaway init, then the snapshot restored over it;
- KMS unseals the restored data, since the KMS key is unchanged;
- a root token generated with the original recovery share;
- new service tokens, then the root token revoked.
OpenBao 2.5.3 and later refuse the unauthenticated root-generation endpoints on its listeners. Box 1 serves them only on a local socket that only root reaches (/var/lib/openbao/admin.sock, mode 600). test:openbao and staging.sh rehearse recover prove this: a secret written before the loss is read back after box 1 is destroyed and rebuilt. Then do steps 8, 9, 11 and 12.
-
Fetch the newest good snapshot from the off-host target onto the operator’s workstation. Do not fetch it onto a server.
-
Decrypt it there with the offline key.
age -d -i operator.key -o vault.snap vault-<timestamp>.snap.ageThe decrypted file is still barrier-encrypted. Delete it when you are done.
-
Start a fresh instance with integrated storage and the same config as production.
-
Initialize it as a throwaway, then unseal it and log in with the throwaway key and root token.
bao operator init -key-shares=1 -key-threshold=1 -
Destroys data: restore the snapshot. This replaces all data on the instance.
bao operator raft snapshot restore -force vault.snap-forceis required because the snapshot comes from the keyring of another cluster. The instance then seals itself, and the throwaway key and token stop working. -
Unseal with the original shares. The holders in Key custody each run
bao operator unsealuntil the threshold is reached. -
If the original root token was revoked at setup and you need one, the same holders run
bao operator generate-root. -
Check that a known organization’s backup key opens a backup.
bao kv get -field=key organizations/<org>/backup-key # for a known org pushleft-agent open-backup --organization <org> --source volume-<env>-<name> \ --backup-id <id> --output /tmp/check.tar --vault-address https://<restored> --vault-token-file <operator token>open-backupverifies the manifest hash and every AEAD tag again. A wrong key cannot produce silent garbage. -
Point
PUSHLEFT_VAULT_ADDRof every service that uses Vault at the restored instance, or move the DNS name (vault.internal.<base>). -
Issue new service tokens: the agent gateway, API, controller, scheduler, AI proxy, mail, git backup and snapshot tokens (the list in
deploy/vault/install-openbao.sh). Tokens created after the snapshot are not in the restored data (secret rotation). -
Restart those services:
pushleft-agent-gateway pushleft-api pushleft-controller pushleft-scheduler, and the AI proxy, mail and git backup where they run. -
Deal with the gap after the snapshot. Backup keys and database passwords created after it are missing.
- The next request creates a missing backup key again. Backups sealed with the lost key cannot be opened. Escrow covers this case.
- For missing database passwords, rotate the credentials of those instances.
npm run test:vault-backup runs steps 1 to 6 and step 8 end to end. It uses real OpenBao with raft storage, a real agent-sealed backup, the shipped snapshot script, destroy, restore, the original share and open-backup.
Check it worked
last-successis less than 2 hours old, andvault_snapshot_staledoes not fire.- After a restore,
pushleft-agent open-backupwrites/tmp/check.tarwithout an error. - The
/readyzof the gateway, the API and the scheduler answer 2xx (Service not ready).
Optional: backup-key escrow
scripts/vault-escrow-backup-keys.sh <output.age> reads the backup key of every organization. It writes one age file, encrypted to the offline operator key. The keys go from Vault through jq into age, so no plaintext is written.
Run it on the operator’s workstation, with a short-lived token that has deploy/vault/pushleft-backup-key-escrow.hcl. That policy lists organizations and reads backup-key, and nothing else.
-
Create the policy and a 15-minute token. Keep the token file at mode 0600, and delete it after use.
bao policy write pushleft-backup-key-escrow deploy/vault/pushleft-backup-key-escrow.hcl bao token create -policy=pushleft-backup-key-escrow -ttl=15m -field=token > ~/.escrow-token # 0600, delete after -
Write the escrow file.
VAULT_ADDR=https://vault... VAULT_TOKEN_FILE=~/.escrow-token PUSHLEFT_VAULT_SNAPSHOT_RECIPIENTS_FILE=recipients.txt \ scripts/vault-escrow-backup-keys.sh escrow-$(date -u +%Y%m%d).age -
Store the output off-host, with the snapshots.
Keys never change, so run it after new organizations get databases. Weekly is enough. The script stops on any read error other than “no key yet”, so it never skips a key without telling you.
Import after a total loss
Import into a new, empty Vault with the organizations KV v2 mount. Each key is written create-only, so an existing key is never replaced.
age -d -i operator.key escrow-<date>.age | while read -r org key; do
[ "$org" = "#" ] && continue
bao kv put -cas=0 organizations/$org/backup-key key="$key"
done
Trade-off
- Without escrow, an attacker needs a snapshot, the operator key and a quorum of unseal shares to read backup keys.
- With escrow, an attacker with the escrow file and the sealed backups opens the backup key of every organization with the operator key alone. The operator key becomes the most valuable secret.
- Use escrow only if that key is held as strongly as the unseal shares: a hardware token, two named holders, never online.
- The gain: tenant backups survive the loss of Vault and of the unseal quorum. Snapshots alone cannot give that.
- Escrow covers backup keys only. After such a loss, rotate the database passwords.
Proof status
- Proven by
npm run test:vault-backup(Docker, OpenBao 2.4.1, single share):- The snapshot policy cannot read tenant secrets.
- The snapshot is encrypted before it touches disk, and it is a valid raft archive.
- A destroyed Vault restored from it seals itself and opens only with the original share.
- The restored key opens an agent-sealed backup. An escrow import into an empty Vault opens it too.
vault_snapshot_stalefires when the status file is stale or missing.
- Not proven: the systemd units on a real host, a real Storage Box or S3 copy (the test copies into a local directory) and off-host retention. The tests also skip a real unseal ceremony with named holders, restore time at production size, and HashiCorp Vault (the API calls are the same).
- Not built: a 3-node raft cluster, and write-once off-host retention that a compromised control-plane host cannot delete. With one instance, a host loss is an outage until the restore is done.