pushleft
Deploy your first app
Manual contents
Handbook
HandbookPushleftArchitecture
Operate
Install the control planeYour own serversFleetOperations and alertsAuthenticationDomains and DNSUsage and budgetsScheduled jobsAbuse controlsAI provider proxyCustomer mailErrors and performanceLogs and metricsRetentionObject storageContainer registryGit server
Develop
The .pl.yaml manifestOnboard with an AI agentThe pl up demoBuild executorSource ingestionAgent gatewayManaged databasesSecurity findingsTunnels and shellsDev workspacesPrivate hostnamesMesh coordinatorsDesign baseline
Runbooks
Control-plane host failoverControl-plane PostgreSQL failoverControl-plane PostgreSQL restoreCustom domains and certificatesGateway key compromiseIngress host failureNode busyNode disk fullNode drain and replacementNode lostRegistry outageHetzner Robot hostsSecret rotationSecurity incidentStaging fleet on ProxmoxStaging build cacheStuck deploymentVault/OpenBao backup and recovery
Manual · docs/retention.md

Retention

Choose how long Pushleft keeps your releases, sources and deployment events, and when it tears down idle previews. Retention never releases a deployment that runs or carries traffic. It never tears down production.

Console Retention settings per organization, within the limits of the tier.

Quick start

  1. In the console, open Settings and go to Retention.
  2. Fill in the settings that you want to change. Leave a field empty to use the platform default:
    • Releases kept per environment: successful releases kept for rollback (production only).
    • Days a retired release is kept: source and image of a superseded, failed or cancelled release (production only).
    • Tear down idle previews after (days): days without a new deployment. Empty turns it off.
    • Days deployment events are kept: events and build logs of finished deployments.
  3. Select Save retention. Only owners and admins can change retention, and the Allowed column shows your tier’s limits.

You can also set the policy with the API:

curl -X PUT https://api.pushleft.dev/v1/orgs/<org id>/retention \
  -H "Authorization: Bearer $PUSHLEFT_TOKEN" -H 'Content-Type: application/json' \
  -d '{"retained_releases": 5, "source_retention_days": 14, "preview_idle_teardown_days": 7, "event_retention_days": 30}'

For container images that you push, set rules per repository with pl registry retention set (registry retention).

Features

  • Rollback to any retained release of a production environment (3 per environment by default).
  • Sources and images of retired production releases kept for 7 days by default. A preview keeps no release history (decided 2026-11-05).
  • Optional teardown of previews with no new deployment for N days. Production is never torn down.
  • A window for deployment events and build logs. The newest event of each deployment always stays.

Internals

This page covers:

  • The per-organization policy, with idle preview teardown and event retention.
  • Finished history, deleted projects and the stale-deployment reaper.
  • Registry blob GC and build cache snapshots.
  • The S3-compatible store for sources, cache and backups.

Source ingestion describes source and image GC itself.

Organization retention policy

The policy sets the retention windows for one organization (migration 0055_retention_policy.sql). Owners and admins set it in the console (Settings, Retention) or with PUT /v1/orgs/{id}/retention. Every member can read it with GET. The audit trail records a change as retention_policy_changed.

A field that is empty or null uses the platform default, which is the behaviour before the policy existed.

FieldPlatform boundsDefault (null)new / standard / trusted tier cap
retained_releases: successful releases per environment kept for rollback1–100PUSHLEFT_RETAINED_RELEASES (3)5 / 10 / none
source_retention_days: how long a superseded, failed or cancelled release keeps its image. The source goes when the release finishes (source ingestion)1–90PUSHLEFT_RELEASE_RETENTION (7 days)14 / 30 / none
preview_idle_teardown_days: tear down previews with no new deployment for N days1–90offnone
event_retention_days: how long events of finished deployments are kept7–365kept30 / 90 / none

The API refuses a value above the platform bound or the org’s tier cap. If an operator lowers an org’s tier later, the scheduler applies the lower cap to the stored value. The caps live in org_tiers (trust tiers).

  • Releases and sources. The scheduler’s retention pass runs once a minute. It uses the org’s window and release count in place of the platform settings. Torn-down deployments are still released at once. Nothing that runs or carries traffic is released.

  • Only production keeps history. In a preview environment, a superseded, failed or cancelled release gives up its source and image at once. The environment keeps no rollback targets. The window and the release count apply to production environments only.

  • Idle previews. The org must opt in. A preview environment whose newest deployment is older than N days has its serving deployments (live, degraded, ready, running) set to teardown_queued.

    • This is the same path as pl down: the scheduler tears down every teardown_queued deployment.
    • Each one gets a teardown_requested event with {"reason":"preview_idle","idle_days":N}.
    • Production environments are never torn down.
    • Inactivity means “no new deployment”. Request traffic does not count.
  • Events. deployment_events stays append-only. The table trigger allows a delete only when all of these are true:

    • The org has an event window, and the event is older than it (after the tier cap).
    • The deployment is superseded, cancelled, failed or destroyed.
    • A newer event of the same deployment exists.

    The newest event is never deleted, so a sequence number (an SSE cursor) is never reused. The scheduler calls purge_expired_deployment_events(5000) (SECURITY DEFINER) once a minute. Build logs travel in events, so this window is also the log window. The auth_events audit trail is not affected.

Finished history

Finished history is the job, instance and route rows of work that is done. The hygiene pass removes the rows that no live state, billing or audit needs (migrations 0073_hot_path_indexes_and_history_retention.sql and 0142_job_retention_and_lookup_indexes.sql, scheduler src/reaper.rs).

The pass runs once a minute, together with the reaper. It calls purge_finished_history (SECURITY DEFINER) in batches of 5 000 rows per table, up to 20 batches while a batch is full. It deletes these rows:

RowsDeleted whenSettingDefault
Periodic network, mesh and name jobs and developer sessions (manage_network, reconcile_mesh_routes, provision_tailnet, sync_environment_names, open_session, no deployment) and their agent resultsfinished (completed or failed) longer ago than the windowPUSHLEFT_CHECK_JOB_RETENTION_HOURS3 h (minimum 1 h; 24 h before 2026-11-06, when these jobs, about 10,000 an hour, made the jobs table 1 GB)
Published-port jobs (sync_published_ports) and their agent resultsas above, and a later-completed job of the same node and environment with at least the same generation exists (a failed one needs none)samesame
Jobs of a finished deployment (superseded, cancelled, failed, destroyed) and their agent resultsfinished longer ago than the windowPUSHLEFT_HISTORY_RETENTION_DAYS30 days (minimum 7)
removed application instances of a finished deploymentremoved longer ago than the window, and none of their jobs is leftsamesame
failed ingress routes of a finished deploymentfailed longer ago than the window, and a newer route generation of the same hostname existssamesame

These are kept:

  • Jobs of a deployment that is not finished, whatever their age. A running cron role reads its placement job and its run history. A live release reads its release run.
  • Any job that a release artifact, a source upload claim or a mesh credential publication references (build jobs, enrollments).
  • The newest completed published-port set of each node and environment. The scheduler holds an environment’s port claims until its nodes have confirmed their current generation.
  • Other jobs without a deployment (DNS, databases, backups, volumes, enrollments). Their idempotency keys are not time-bucketed.
  • lost instances, and an instance that a replacement references.
  • The newest route generation of every hostname, so a generation is never reused.
  • Deployments themselves, their events (see event retention above), sources, releases, fleet_allocations (the usage ledger’s input) and usage_ledger.

agent_job_results and ingress_analytics_routes stay append-only otherwise. Their triggers allow a delete only for these rows:

  • A result whose job finished more than an hour ago.
  • A failed, superseded route of a finished deployment older than 7 days.

The scheduler role needs GRANT EXECUTE ON FUNCTION purge_finished_history(integer, integer, integer) TO pushleft_scheduler (database roles). Without it, the purge logs finished history purge deferred and nothing else changes.

Deleted projects

A project deleted with pl project delete (projects) keeps its history for 7 days. Then the hygiene pass removes it (decided 2026-11-04; migration 0131_deleted_project_purge.sql, scheduler src/reaper.rs).

SettingDefaultMinimum
PUSHLEFT_DELETED_PROJECT_RETENTION_DAYS: days after deleted_at71

The pass calls purge_deleted_project (SECURITY DEFINER) up to 20 times a minute. Each call purges one project in one transaction. It locks the project row and its environments (FOR UPDATE, skipping a locked project) and deletes at most 1 000 rows per table. A call that fills a batch ends with more, and the next call continues. An interrupted purge loses nothing: each call commits completely or not at all.

Checks before anything is deleted. The purge checks again what pl project delete refused, because something can come back to an environment after the deletion:

  • a deployment that is not finished, a job that is not finished, an instance that is not removed or lost, a route that can carry traffic;
  • a database that is not destroyed, a database operation or import that is still running;
  • a volume that is not released, a bucket, a custom domain that is not removed, a secret, an AI provider key, a token or DSN that still works;
  • a network, attachment or attachment request that is not released;
  • a public port claim, or a node whose port set is not confirmed empty;
  • an allocation (reserved or active) of any of its instances, databases, volumes or network attachments.

If it finds one, it deletes nothing and logs deleted project not purged with the item. It checks again on every pass.

Waits for the verified GC. A deleted project’s releases are not rollback targets. The source and image GC releases them at once, like a torn-down deployment, without the retention window or the last N kept. The purge waits until the GC has deleted every source object, released every snapshot and image, and its registry keep tags are removed. A build image must be deleted (or unsupported by the registry). When an artifact of another project uses the same digest, that artifact deletes it. The purge also waits until database import archives are deleted. Until then it logs deleted project purge waits for retention. With the GC off (PUSHLEFT_SOURCE_ROOT unset) or no registry configured, the project waits.

Deleted: deployments and their events, jobs, agent results, image scan alerts and waivers; source snapshots, upload claims, release artifacts and keep tags; application instances and ingress routes; destroyed databases with their catalogs, logins, operations, operation events and imports; released volumes with their events and imports; removed custom domains with their events and certificates; deleted buckets; the network (segment, members, attachments, attachment requests, enrollments, credential publications); mail messages and deliveries, mail, AI and error tokens; AI policies; error projects, issues, alerts, releases and source map records; name sets and port sets.

Kept:

  • The project row (its icon is removed and history_purged_at is set), its environment rows and their ingress_hostnames. They keep the public id, the hostname label and the preview hostnames taken, so an old URL never resolves to a new project.
  • The project_deleted audit event and every other auth_events row.
  • Billing: fleet_allocations (the compute meters’ input), usage_events, usage_ledger, ai_usage_daily and runtime_log_days. None of them references the project.
  • Backups. A deleted database’s backups and the final backup stay for their normal retention. The agent sweeps the archive by its removed-at stamp, and the off-host bucket has its own lifecycle (managed databases). Neither reads the database row. The archive is <organization>/db_<id> in the backup store.
  • A released volume’s last backup on its node (<backup_root>/<organization>/volume-<environment>-<name>). Volume release does not stamp it removed, so the agent’s sweep of removed archives does not delete it. The gap is open and applies to every released volume in every project.
  • Volume migration and relocation archives in the backup store (volumes/, relocation/). They are left to the bucket’s lifecycle rule.
  • AI provider keys and Cloudflare tunnel tokens in Vault. They belong to the organization, not the project, and a removed key stays in Vault.

Leftovers outside PostgreSQL (decided 2026-11-05; migration 0133_deleted_project_leftovers.sql). The purge queues, in its own transaction, what the project left in Vault and the blob stores. Each target belongs to one side, by its store, and the other side never asks for it:

  • the scheduler (src/reaper.rs, in the hygiene pass every minute) deletes the Vault targets, with its own token that may only list and delete;
  • the API (src/leftovers.rs, every 5 minutes) deletes the blob targets, because it holds the stores.
StoreTargetWhatDeleted by
Vaultorganizations/<organization>/environments/<environment>/every secret under it: database admin and login passwords, mesh pre-auth keys, any secret leftscheduler
Vaultorganizations/<organization>/custom-domain-certificates/<certificate>an uploaded or issued origin certificate keyscheduler
artifactssourcemaps/<organization>/<environment>/stored source map filesAPI
backupsvolume-imports/<organization>/<volume>/pl volume import archivesAPI
cache<organization>/<project>/build cache snapshots, and the previews’ under <organization>/preview/<project>/API
  • The purge queues the environment and project targets when it sets history_purged_at. It queues a volume or certificate target when it deletes that row, because the id is gone after the purge. Projects purged before 0133 get their environment and project targets from the migration; their volume and certificate ids are already gone.
  • Every target is made of ids, and a prefix ends in /. A CHECK on deleted_project_leftovers and the code on each side refuse any other shape, so a target never matches the objects of another project.
  • Each side takes up to 50 due targets of a purged project (deleted_project_leftovers_due, with its own stores). In Vault, the scheduler lists the tree and deletes every version and the metadata of each secret. In a store, the API lists the prefix and deletes each object. An absent object counts as deleted, so a second run changes nothing.
  • A failure is logged (purged project Vault leftover not deleted, purged project leftover not deleted), recorded with the error and the attempt count (deleted_project_leftover_done) and tried again after 10 minutes. It never holds up the SQL purge, which is already done.
  • A store that a side does not have is not tried; its targets wait. Without Vault settings on the scheduler, the Vault targets wait. Without cache (PUSHLEFT_CACHE_S3_* or the shared bucket) on the API, build cache snapshots stay.
  • Both roles need GRANT EXECUTE ON FUNCTION deleted_project_leftovers_due(text[], integer), deleted_project_leftover_done(bigint, text) (TO pushleft_api and TO pushleft_scheduler).
  • The scheduler’s token carries only pushleft-scheduler-leftovers.hcl: list and delete on organizations/metadata/+/environments/+/*, delete on organizations/metadata/+/custom-domain-certificates/*, no read or write. It is a periodic orphan token at /etc/pushleft/vault-scheduler/token, renewed by the scheduler like every service token (Vault service tokens). The API keeps its create-only policy.

The append-only triggers of these tables let a delete through only in the purge’s own transaction, which marks itself in project_purge_marks. Only the function can write that table. Everywhere else they refuse as before. A deleted project stays terminal: the only change allowed is the purge setting history_purged_at once and removing the icon.

The scheduler role needs GRANT EXECUTE ON FUNCTION purge_deleted_project(integer, integer, uuid[]) TO pushleft_scheduler. Without it, the pass logs deleted project purge deferred.

Stale deployment reaper

The reaper fails deployments that stop making progress, so they do not hold capacity forever. It lives in the scheduler (src/reaper.rs) and runs once a minute. It fails a deployment in queued, building or deploying whose last progress is older than its threshold:

StatusSettingDefault
queuedPUSHLEFT_REAP_QUEUED_AFTER3600 s
buildingPUSHLEFT_REAP_BUILDING_AFTER2700 s (the agent’s 30-minute build timeout plus 15 minutes)
deployingPUSHLEFT_REAP_DEPLOYING_AFTER3600 s
queued or deploying with a role still unplacedPUSHLEFT_REAP_PLACEMENT_AFTER600 s, counted from the role’s first waiting_for_capacity event, whatever else progresses

The placement limit makes a release that cannot be placed fail within ten minutes (decided 2026-11-05). pl up in CI then fails with the waiting_for_capacity reason and does not block the pipeline for an hour. The reap reason names the wait. The minimum for each setting is 600 s. Progress is the latest of these:

  • The deployment’s updated_at.
  • Its newest event.
  • Its jobs’ lease expiry, which a heartbeat extends.

The scheduler targets a build at a build node even when that node is busy. The build waits there for the node’s single build slot. While it waits, the node’s last build activity (a live build lease or a finished build) also counts as progress. A deployment queued behind a long build is therefore not reaped. If the build node does nothing while the build waits, the reaper can still fail it. Other placement deferrals (for example, waiting for application capacity) are not progress.

The reaper never touches a deployment with a live lease, or with a lease that expired less than 60 s ago. It also skips suspended organizations, whose work waits on purpose.

In one transaction, the reaper locks the deployment and its open jobs (the gateway leases with SKIP LOCKED) and checks the leases again. Then it:

  1. Fails every queued or expired leased job and clears its lease. A late heartbeat, completion or artifact publication from the node is then a stale-lease conflict (the same fence as lost-node reclamation).
  2. Sets the deployment to failed.
  3. Appends deployment_reaped with reason (for example, building for 2712 s without progress (limit 2700 s)), previous_status, idle_seconds, threshold_seconds and cancelled_jobs.

Capacity comes back through the existing paths for any failed deployment:

  • Reservations of instances that never started are released.
  • Started instances are stopped with stop_application once no route of the deployment can carry traffic. The agent removes their containers, secret directories and, when no other container uses it, the image. Nothing starts a stopped release again; a rollback deploys it anew.
  • A previous live release keeps its capacity.

Retention abandons the source claim after PUSHLEFT_SOURCE_CLAIM_TTL, because failed is not a protected state.

Registry blob GC

The registry is zot. Zot collects garbage online, inside the serving process. There is no read-only window and no restart, and pushes keep working.

  1. Release retention deletes a built image’s manifest by digest. The scheduler does this with a delete token from the gateway’s realm.
  2. Zot’s GC frees the blobs after gcDelay (1 h), on its next periodic run, which starts in the nightly window (gcTimeWindow, 02:00–03:00 UTC). A registry restart also starts a pass, at any hour.

Release retention never deletes images that an organization deploys by digest. The gateway only removes their pl-keep-<deployment> tag. The repository’s retention rule decides the rest.

The offline GC with a read-only window (scripts/registry-gc.sh, pushleft-registry-gc.timer) was for distribution (registry:2) and is removed. npm run test:registry proves on Garage that zot deletes old untagged images and their blobs while keeping every digest a retained deployment uses.

Build cache snapshots

Each project has two build caches in the blob store’s cache use: production’s (<org>/<project>/) and the previews’ (<org>/preview/<project>/, decided 2026-11-05). The gateway keeps the newest two snapshots of each and deletes older ones whenever it stores a new one. Build nodes evict local caches least recently used first. When a deleted project is purged, the API deletes both (leftovers).

S3-compatible source store

One adapter stores sources, build cache snapshots, registry blobs and encrypted backups: the pushleft-blob crate (re-exported by pushleft-source). It has a BlobStore trait and two stores with the same contract:

  • S3Store: a path-style S3 client with AWS Signature V4 (no SDK). Cloudflare R2 in production; Garage in tests.
  • LocalStore: a private directory tree (0700), for tests and single-host installs.
MethodContract
put_from(key, reader) / put_new(key, bytes)Write-once. A replay of identical bytes is accepted, different bytes are refused, nothing is overwritten. Returns the length and SHA-256, computed while streaming.
get_into(key, writer, max_len) / get(key, max_len)Streams the object, bounded by max_len; returns the length and SHA-256 for the caller to compare with the recorded digest. Absent is None.
head(key)The length, or None.
list(prefix)Every key under the prefix with its length, sorted; S3 pages through ListObjectsV2. Retention uses it.
delete(key)Absent is not an error.

Keys use [A-Za-z0-9/._-], at most 512 bytes, without empty, . or .. segments.

Large objects. An object up to 16 MiB is a single PUT. A larger one is a multipart upload in parts of exactly 16 MiB (PART_SIZE), with only the last part smaller.

  • R2 refuses a multipart upload whose non-trailing parts differ in length (“All non-trailing parts must have the same length”). The client checks this rule itself before each part.
  • Every part’s SHA-256 is signed, so the store verifies each body.
  • An upload that fails is aborted.
  • The size limit is 10,000 parts (about 156 GiB).

Write-once on S3. Every put first sends HEAD. If the key exists, the client hashes its input and the stored object, and accepts only identical bytes.

The PUT (and CompleteMultipartUpload) also carries If-None-Match: *. On a backend that honours it (R2, AWS), two writers that race with different bytes have one winner. Garage ignores that header (checked on v2.4.1). On Garage, a race between different bytes is last-writer-wins. The object is still one writer’s complete bytes, and readers detect the swap through the digest. Pushleft’s keys are content-addressed or per generation, so such a race needs a writer outside Pushleft.

The read-verify-delete in GC is not atomic, because there is no conditional delete on the ETag.

Configuration

pushleft_blob::from_env(Use::Sources | Cache | Registry | Backups) picks the store for each use. The first match wins:

  1. A bucket for that use alone: PUSHLEFT_SOURCE_S3_*, PUSHLEFT_CACHE_S3_*, PUSHLEFT_REGISTRY_S3_* or PUSHLEFT_BACKUP_S3_*.
  2. The shared bucket PUSHLEFT_BLOB_S3_*, under <PREFIX>sources/, cache/, registry/ or backups/.
  3. For backups only, a private local tree at PUSHLEFT_BACKUP_LOCAL_ROOT. The API and gateway must share it.
  4. A local tree, PUSHLEFT_BLOB_LOCAL_ROOT/<use>.

With none of these, sources use the local PUSHLEFT_SOURCE_ROOT tree. Set the same source setting on the API, the scheduler and the agent gateway at the same time. Those three services can then run on separate hosts.

Setting (<NAME> is PUSHLEFT_BLOB_S3 or a per-use name)Meaning
<NAME>_BUCKETSwitches that use to S3
<NAME>_ENDPOINTOrigin, https:// (plain HTTP only for loopback)
<NAME>_REGIONDefault us-east-1; auto for R2
<NAME>_PREFIXOptional, ends with /
<NAME>_ACCESS_KEY_ID, <NAME>_SECRET_ACCESS_KEY_FILECredentials; the file must be mode 0600

For Cloudflare R2, use:

  • Endpoint https://<account id>.r2.cloudflarestorage.com.
  • Region auto.
  • Path-style addressing (the only style the client speaks).
  • An R2 API token with Object Read & Write on the bucket.

For example:

PUSHLEFT_BLOB_S3_BUCKET=pushleft-prod
PUSHLEFT_BLOB_S3_ENDPOINT=https://<account id>.r2.cloudflarestorage.com
PUSHLEFT_BLOB_S3_REGION=auto
PUSHLEFT_BLOB_S3_ACCESS_KEY_ID=<token access key id>
PUSHLEFT_BLOB_S3_SECRET_ACCESS_KEY_FILE=/etc/pushleft/r2-secret   # 0600

PUSHLEFT_SOURCE_ROOT must still be set, because it turns source handling on. Pushleft does not use it when a blob store is configured. Reads of sources are bounded and checked against the recorded length and digest. GC reads the object, checks the digest and only then deletes.

Tests

npm run test:blob (in test:all) runs the adapter conformance suite (crates/pushleft-blob/src/tests.rs). It runs on LocalStore and on S3 against Garage. The Garage fixture (scripts/fixtures/garage.sh) uses dxflrs/garage v2.4.1 pinned by digest, a one-node layout applied through its admin CLI, an imported key and region auto. The suite covers:

  • put/get/head/delete, write-once with replay, bounded reads and list by prefix.
  • Multipart objects. One is 32 MiB plus 5 bytes, read back part by part with HEAD ?partNumber= as 16 MiB, 16 MiB, 5 bytes. One is about 33 MiB, replayed and tampered.
  • SHA-256 integrity end to end.
  • Concurrent writers. With the same bytes, all succeed. With different bytes, LocalStore has exactly one winner and Garage keeps one writer’s whole bytes.

npm run test:source-ingestion runs the source store contract against Garage. Garage is AGPL-3.0 and runs only as a separate test service.

Planned:

  • The adapter has not run against real R2.
  • Encryption is whatever the bucket enforces. Backups will be encrypted client-side with the organization keys before upload.
  • Objects are not replicated beyond what the provider does.

Off-host backup copies

Database and volume backups do not go through this interface. The agent spools them, sealed with the organization’s key, and pushleft-offsite.timer ships the spool to R2 (off-host spool). The backups use carries database imports, named datasets, volume handoffs, relocation archives and git repository backups. A copy through the adapter would look like this:

use anyhow::Context;
use pushleft_blob::{BlobStore, Use};
let store = pushleft_blob::from_env(Use::Backups)?   // inside a Tokio runtime
    .context("no blob store configured")?;             // None when nothing is set
// from spawn_blocking, per backup generation:
store.put_from("organizations/<org>/databases/<instance>/<generation>/<file>", &mut file)?; // every file first, streamed
store.put_new(".../<generation>/manifest.json", &manifest)?;                                // the manifest last
store.get_into(key, &mut out, max_len)?;  // restore/verify: Option<Stored { len, sha256 }>
store.list("organizations/<org>/databases/<instance>/")?;  // retention: generations present
store.delete(key)?;  // only after retention re-verified the oldest kept base; the manifest first

A copy without a manifest is incomplete and must be ignored.

Rendered from the repository at build time. Edit docs/retention.md to change it.