pushleft
Deploy your first app
Manual contents
Handbook
HandbookPushleftArchitecture
Operate
Install the control planeYour own serversFleetOperations and alertsAuthenticationDomains and DNSUsage and budgetsScheduled jobsAbuse controlsAI provider proxyCustomer mailErrors and performanceLogs and metricsRetentionObject storageContainer registryGit server
Develop
The .pl.yaml manifestOnboard with an AI agentThe pl up demoBuild executorSource ingestionAgent gatewayManaged databasesSecurity findingsTunnels and shellsDev workspacesPrivate hostnamesMesh coordinatorsDesign baseline
Runbooks
Control-plane host failoverControl-plane PostgreSQL failoverControl-plane PostgreSQL restoreCustom domains and certificatesGateway key compromiseIngress host failureNode busyNode disk fullNode drain and replacementNode lostRegistry outageHetzner Robot hostsSecret rotationSecurity incidentStaging fleet on ProxmoxStaging build cacheStuck deploymentVault/OpenBao backup and recovery
Manual · docs/storage.md

Object storage

This page shows how you give an environment its own private S3 bucket. Each bucket has an access key that works on that bucket only. Any S3 client works with it: the AWS SDKs, boto3, aws s3 and rclone.

Quick start

  1. Create a bucket for this checkout’s preview environment, or for production:

    pl storage bucket create uploads --set-secret          # this checkout's preview environment
    pl storage bucket create uploads --production --set-secret

    create prints the key once:

    VariableValue
    S3_ENDPOINTThe S3 endpoint (path-style addressing)
    S3_BUCKETThe bucket’s name, pl-<organization handle>-<name>-<suffix>
    AWS_REGIONThe endpoint’s region
    AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEYThe bucket’s key: read and write on this bucket only
  2. --set-secret also stores the five values as environment secrets. List them under secrets: with as: env in .pl.yaml:

    secrets:
      - { name: S3_ENDPOINT, as: env }
      - { name: S3_BUCKET, as: env }
      - { name: AWS_REGION, as: env }
      - { name: AWS_ACCESS_KEY_ID, as: env }
      - { name: AWS_SECRET_ACCESS_KEY, as: env }
  3. Deploy with pl up. The next deployment gets the variables.

Commands

CommandAPI
pl storage bucket create <name> [--production | --project P --environment E] [--set-secret] [--public <domain>]POST /v1/storage/buckets {"project", "environment", "name", "public_domain"} (members)
pl storage bucket verify <id>POST /v1/storage/buckets/{id}/verify (members). Checks the TXT record, then serves the bucket on its domain.
pl storage bucket listGET /v1/storage/buckets: buckets with their usage (bytes, objects) and quota
pl storage bucket rotate <id> [--set-secret], then --finishPOST /v1/storage/buckets/{id}/rotate[?finish=true] (members). This prints a new key once. The previous key works until --finish, so deploy with the new secrets between the two steps.
pl storage bucket delete <id>DELETE /v1/storage/buckets/{id} (admins). Refused while the bucket holds objects. The bucket’s keys are deleted with it.

The audit trail records create, rotate and delete (storage_bucket_created, storage_key_rotated, storage_bucket_deleted), and each key pl storage cp/ls/rm uses (storage_access_issued).

Files in a bucket

You can copy, list and delete objects with pl, without the AWS CLI and without the bucket’s key on disk. pl uses your login. For a production bucket you need the admin role, and for a preview bucket the member role.

pl storage cp ./dist/app-1.2.0.dmg downloads:desktop/1.2.0/      # upload one file under a prefix
pl storage cp -r ./dist downloads:desktop/1.2.0                  # upload a directory
pl storage cp latest.json downloads:desktop/latest.json --content-type application/json
pl storage ls downloads:desktop/                                  # objects and sub-prefixes
pl storage cp downloads:desktop/latest.json .                     # download
pl storage rm -r downloads:desktop/1.1.0/                         # delete a prefix
  • Name the bucket by its name in the checkout’s environment (add --production, or --project and --environment), by its S3 name, or by its id.
  • An upload replaces the object, so you can run the same command again after a failure. A file larger than 16 MiB goes up in 16 MiB parts, and Pushleft tries each part up to five times.
  • Each command first gets a key that works on this bucket only and expires after an hour (POST /v1/storage/buckets/{id}/access). It deletes the key when it is done (DELETE /v1/storage/buckets/{id}/access/{key}). The audit trail records each key as storage_access_issued.
  • Keys are letters, digits, /, -, _ and ..
  • rm -r <bucket>:<prefix> deletes the objects under <prefix>/, so desktop/1.1 does not take desktop/1.10. cp -r skips symbolic links.

Limits

A bucket is private unless you create it with --public. A private bucket has no anonymous access. To give its files to a browser, your application serves them or gives a presigned URL.

The organization’s trust tier sets the limits:

TierBuckets per organizationQuota per bucket
new31 GiB
standard2020 GiB
trusted100200 GiB

On a paid plan (Cloud, or Cloud with your own servers), you choose the quota, from 1 to 1024 GiB. Stored GB are billed. Set it at creation, or change it later:

pl storage bucket create files --production --set-secret --quota 50
pl storage bucket quota <id> 100

Without --quota, a bucket gets the tier’s quota. On the self-hosted plan, or with no plan, the tier’s quota is the most. The number of buckets always follows the tier.

A write past the quota gets 403 (Bucket size quota is reached).

Public buckets

A public bucket is for files that every visitor may load, such as videos and images on your pages. It is served on your own domain only, never on a Pushleft name. Anyone can read its objects at:

https://<your domain>/<key>
  1. Create the bucket with the domain. pl prints the key once and two DNS records:
    pl storage bucket create media --production --public media.example.com
  2. Add the two records at your DNS provider:
    • TXT _pushleft-challenge.media.example.com with the printed value. It proves that you own the domain.
    • CNAME media.example.com to s3.pushleft.dev.
  3. Run pl storage bucket verify <id>. Nothing is served until it passes.
  4. Upload with any S3 client and the printed key. Set Cache-Control on each object, for example public, max-age=31536000, immutable. Give a changed file a new name.
  5. Link to https://media.example.com/<key> from your pages.
  • Without --set-secret, pl stores no secrets. Your environment’s existing AWS_* secrets stay as they are.
  • Pushleft gets the domain’s certificate when the first visitor arrives.
  • Range requests work, so video players can seek.
  • To cache the files at Cloudflare’s edge, proxy the CNAME in your own Cloudflare zone after verify passes. Set the zone’s SSL mode to Full or Full (strict).
  • Only GET and HEAD are served. Directories are not listed.
  • HTML in a public bucket gets a sandbox Content-Security-Policy, so it runs no script and submits no form.
  • A domain serves one bucket. Public is set at creation: to make a bucket private, create a new bucket and move the objects.
  • The new tier cannot create public buckets.

Operating it

The included tier is one platform Garage node (v2.4.1, the digest is in scripts/fixtures/garage.sh). It uses replication 1 on a disk of its own. The API manages it through the Garage admin API v2:

SettingMeaning
PUSHLEFT_STORAGE_GARAGE_ADMIN_URLThe admin API, for example https://garage-admin.internal.pushleft.dev:3903/. Private: the tailnet only.
PUSHLEFT_STORAGE_GARAGE_ADMIN_TOKEN_FILEIts admin_token, mode 0600
PUSHLEFT_STORAGE_GARAGE_S3_ENDPOINTThe S3 endpoint that applications get. Workloads reach only public addresses, so this must be a public HTTPS name, for example https://s3.pushleft.dev.
PUSHLEFT_STORAGE_GARAGE_REGIONGarage’s s3_region (default garage)

Without these settings, pl storage answers 503.

Production (since 2026-11-04): the node runs on pl-database-1 as pushleft-garage.service (host network, --userns host, 1 GiB memory). Its data is under /var/lib/pushleft/data/garage, and its config and secrets are in /etc/pushleft/garage/ (garage.toml, rpc-secret, admin-token, 0600). Its S3 API (3900) and admin API (3903) listen on the node’s tailnet address only. RPC listens on loopback. The management tailnet ACL lets only cp reach them (mgmt-acl.hujson). cp’s Caddy serves https://s3.pushleft.dev (an A record to cp, not proxied by Cloudflare) through PUSHLEFT_STORAGE_GARAGE_S3_UPSTREAM in install-control-plane.sh. The API reads the four settings from pushleft-api.service.d/storage.conf, the token from /etc/pushleft/storage/garage-admin-token. The layout is zone fsn1, 500 GB. The region is garage. Since 2026-11-05 every bucket is copied off the host each night, encrypted, to the R2 bucket pushleft-garage-mirror (pushleft-garage-mirror.timer). Each night is a full copy, and R2 keeps the last 5 nights (a 5-day lock and expiry). garage_mirror_stale (ticket) fires after 26 h without a complete copy (Garage off-host copy).

Public buckets (since 2026-11-10). Garage’s web endpoint listens on 100.64.0.6:3902. Like the rest of garage.toml, its [s3_web] section (bind_addr = "100.64.0.6:3902", root_domain unused) was added by hand on pl-database-1 on 2026-11-10. A new Garage host needs it before PUSHLEFT_STORAGE_GARAGE_WEB_UPSTREAM is set, or verified buckets get 502. Caddy keeps a normal certificate policy for the platform’s names, and the on-demand policy covers only other names. It serves only buckets with website access, by the request’s Host: a bucket whose global alias is that name. The API adds the customer’s domain as the alias and turns on website access only after verify finds the TXT record. cp’s Caddy sends every name that no other route handles to the web endpoint (pushleft-public-buckets, the last route), when PUSHLEFT_STORAGE_GARAGE_WEB_UPSTREAM is set in install.env. Caddy gets certificates on demand, and only for names that GET /v1/storage/tls-permission?domain= confirms. The API needs PUSHLEFT_STORAGE_PUBLIC_ORIGIN=s3.pushleft.dev in storage.conf: the name a customer’s CNAME points to. The management ACL lets cp reach port 3902. No Pushleft name serves a public bucket (Waseem, 2026-11-10: a bad upload must not hurt pushleft.dev).

scripts/storage-e2e.sh runs the API against a real Garage and a migrated PostgreSQL. It checks:

  • a bucket with its scoped key and quota,
  • writes through Pushleft’s S3 client,
  • the two-step rotation (the old key stops working after --finish),
  • the refused delete of a bucket that holds objects,
  • the bucket limit of the tier.

Plan: the paid R2 tier (R2 buckets with tokens minted through the Cloudflare API, metered), a console Storage page and usage metering. None of these are built yet. (The nightly copy of the Garage buckets to R2 is built, 2026-11-05.)

Rendered from the repository at build time. Edit docs/storage.md to change it.