Container registry
Pushleft includes a private container registry for each organization. Push your own images from a laptop or CI, pull public images through a cache, and deploy an image by digest without a build.
Quick start
-
Log in to the registry. This creates a 30-day push token and runs
docker loginwith it:pl registry login -
Show the registry host and your organization’s namespaces:
pl registry info -
Tag and push an image into your namespace:
docker tag server:latest <registry host>/organizations/<org>/api/server:v1 docker push <registry host>/organizations/<org>/api/server:v1 -
To deploy that image without a build, name it by digest in
.pl.yaml, then runpl up. Your organization must allow image deployments (deploy an image by digest):project: api image: organizations/<org>/api/server@sha256:<64 hex> roles: web: { port: 8080 }
For CI, print a pull-only token and skip the login: pl registry login --print --pull-only.
Features
- Tokens for people and CI, per repository pattern, with
pullorpush(and pull) access. - A pull-through proxy for Docker Hub, GHCR, GitLab and other registries, per organization. Upstream passwords live in Vault.
- Deploy an image by digest without a build. The setting is per organization, because such images skip the isolated build.
- Retention rules per repository, with safe defaults. The defaults keep every tag, the last 10 untagged images, and untagged images pushed in the last 30 days. Pushleft never deletes a digest that a live or retained deployment uses.
- Storage on R2, in a bucket of its own.
For users
Namespaces
Every organization has two namespaces. Both carry its organization ID:
| Namespace | What it holds | Who writes |
|---|---|---|
organizations/<org>/... | Your images. organizations/<org>/projects/<project> holds Pushleft’s builds. | You (push tokens), except projects/, which only Pushleft builds write |
proxy/<org>/<upstream>/... | Images pulled through from an upstream you configured | Nobody pushes; zot fetches on the first pull |
pl registry info (or GET /v1/registry) shows the registry host, your namespaces, tokens, upstreams and retention rules. No token ever reaches the namespaces of another organization.
Tokens for people and CI
Organization admins create registry tokens.
- A token names 1 to 20 repository patterns inside your namespaces.
*matches within one path segment, and**matches any number of segments. - It grants
pull, orpushandpull. - It expires after 1 to 365 days (default 90).
- Pushleft shows the secret (
plr_...) once.
pl registry login # creates a 30-day push token and runs docker login (username `token`)
pl registry login --print --pull-only # for CI: print the token instead
pl registry token create ci --repo 'organizations/<org>/app/**' --push --days 90
pl registry token rotate <id> --overlap-minutes 10
pl registry token revoke <id>
docker login <registry host> --username token # paste the plr_ token
Rules the registry enforces:
- Push only under
organizations/<org>/. Pushes underorganizations/<org>/projects/(Pushleft builds) andproxy/(pull-through caches) are refused. - No token can delete. Images go away through retention (below), so a delete can never remove an image that a deployment uses.
- The registry refuses a revoked or expired token at once. Registry access tokens that were already issued live at most 15 minutes.
- Rotation keeps the old secret working for the overlap. An overlap of 0 revokes it at once.
- The registry refuses the tokens of a suspended organization.
- The audit log records each token creation, rotation and revocation (
registry_token_*).
API: POST /v1/registry/tokens {"name", "repositories": [...], "access": "pull"|"push", "expires_in_days"}, POST /v1/registry/tokens/{id}/rotate {"overlap_minutes"}, DELETE /v1/registry/tokens/{id}.
Pull-through upstreams
pl registry upstream set hub --url https://registry-1.docker.io # anonymous
pl registry upstream set gitlab --url https://gitlab.example.com:5050 --username deploy-bot # password on stdin
docker pull <registry host>/proxy/<org>/hub/library/nginx:1.27
pl registry upstream rm gitlab
The first pull of proxy/<org>/<name>/<path> fetches <path> from the upstream and stores it. The registry serves later pulls itself.
The password goes to Vault (organizations/<org>/registry/upstreams/<name>). The API can write and replace it, but never read it. GET /v1/registry shows only has_password. Only the registry reconciler reads it.
Limits:
- Upstream URLs are
https://host[:port]with no path, on a public host. When you save the upstream, Pushleft refuses (400) a private, loopback, link-local or reserved IP literal, or a name that resolves to one. The check uses the shared SSRF filter,pushleft_core::ssrf(AI proxy: SSRF guard). zot resolves the host again when it pulls, and Pushleft does not check that later answer.PUSHLEFT_REGISTRY_ALLOW_PRIVATE_UPSTREAMS=1(tests only) skips the check. - One organization per credentialed host. zot keys upstream credentials by host, for every upstream at that host. When one organization stores credentials for a host, no other organization can add an upstream at that host (409). A host that another organization already uses cannot get credentials.
- Shared public hosts are anonymous only. Every organization pulls
docker.io,registry-1.docker.io,index.docker.io,ghcr.io,registry.gitlab.comandquay.ioanonymously, so credentials for them are refused (400). You cannot pull private images on those hosts through yet. Push them toorganizations/<org>/instead. - Docker Hub’s anonymous pull limit applies to the registry’s address, which every organization shares.
Deploy an image by digest
In place of build:, the manifest can name an image in your namespaces, pinned by digest:
project: api
image: organizations/<org>/api/server@sha256:<64 hex> # or proxy/<org>/hub/library/nginx@sha256:...
roles:
web: { port: 8080 }
You can leave the registry host out, or give the Pushleft registry’s own host.
pl upstill uploads the checkout, because it carries the manifest. No build runs.- The API publishes the release artifact directly, with
source = imageand no build node. - The deployment event log shows
image_resolvedin place ofbuild_completed. - Placement, the node’s per-placement pull token, health checks and routing are the same as for a built release.
- Pushleft pulls a
proxy/image through when it first protects it (retention). - A single role can also run its own image beside a built release, with the same rules (a second image).
- Pushleft does not scan the image at deploy, and no scan gates it. A pre-built image is the deployer’s responsibility (decided 2026-10-30). In production, a daily scan reports its findings and alerts on fixable CRITICAL ones (report only, decided 2026-11-07; see pre-built images).
An organization can deploy images only if its setting allows it, because such images never ran through the isolated build. The default comes from the trust tier: off for new, on for standard and trusted. An operator changes it with pushleft-api org external-images <org> on|off|default --reason <text> (audited).
- A refused deployment answers 403
external_images_disabled. - An image outside your namespaces answers 403
image_outside_organization.
Retention
With no rule, every repository keeps:
- every tag;
- the 10 most recently pushed untagged images;
- every untagged image pushed in the last 30 days.
Pushleft deletes older untagged images, and zot’s online GC then frees their blobs. An image becomes untagged when its tag moves to another image.
pl registry retention set --repo 'organizations/<org>/api/**' --keep-tags '^v' --keep-tag-count 20 --keep-untagged 5 --untagged-days 14
pl registry retention set --repo 'proxy/<org>/hub/**' --keep-untagged 3 --untagged-days 0 # 0: no age rule
pl registry retention rm --repo 'organizations/<org>/api/**'
-
--keep-tagskeeps only the tags that match the regular expressions.--keep-tag-countkeeps only the most recently pushed of them. Without--keep-tags, every tag stays. -
--keep-untaggedand--untagged-daysbound untagged images. The tier caps them:Tier --keep-untagged--untagged-daysnew1 to 10 up to 30 standard1 to 50 up to 90 trusted1 to 1000 up to 3650 -
The most specific pattern wins (longest first).
-
The gateway renders a changed rule within 30 seconds. zot applies rules on its next garbage collection, in the nightly window (02:00–03:00 UTC).
-
zot never deletes an untagged image younger than 24 hours.
Pushleft never deletes a digest that a live or retained deployment uses.
- The gateway tags the digest of every unreleased release artifact
pl-keep-<deployment>. Every policy keepspl-keep-*tags, whatever its tag rules say. - When release retention releases the artifact, the gateway removes the tag. Pushleft then deletes built images by digest as before, and a deployed image falls back under its repository’s rule.
Sometimes a tag cannot be set or removed (registry down, digest missing). The gateway then records it in registry_keep_tag_failures (0073), with its attempt count and last error. It retries that artifact after 30 s × 2^attempts, at most 6 h apart, so the failure does not hold back the artifacts behind it. It never gives up. After you fix the cause, delete its row to retry at once.
API: PUT /v1/registry/retention {"repository", "keep_tag_patterns", "keep_tag_count", "keep_untagged_count", "keep_untagged_days"} (admins), DELETE /v1/registry/retention?repository=....
Internals
The registry runs on zot v2.1.22. The engine spike at the end of this page chose it. It stores builds, images that people and CI push, and images pulled through from other registries.
The agent gateway is its only token issuer. Nodes, the scheduler, people and CI all present the same short-lived ES256 tokens.
Decided 2026-10-20: the registry replaces the GitLab registry.
For operators
Deployment
The registry runs on the control-plane host, next to the gateway. The gateway renders zot’s configuration.
| File | Purpose |
|---|---|
deploy/registry/zot-config.json → /etc/pushleft/registry/zot-config.json (root:pushleft 0640) | Template: R2 storage, TLS, bearer auth with the realm, docker2s2 compatibility, online GC, the default retention policy, sync with no upstreams |
deploy/systemd/pushleft-registry.service | Runs PUSHLEFT_ZOT_IMAGE (from deploy/images.lock.env, the digest the spike ran) as the pushleft user, port 5000 |
/var/lib/pushleft/zot-rendered/ (pushleft 0700) | The gateway’s rendered config.json and sync-credentials.json (both 0600). zot reads them read-only |
/var/lib/pushleft/zot/ | zot’s metadata database and pull-through download directory |
/etc/pushleft/registry/tls.{crt,key}, pushleft-token.crt | Registry TLS (operator CA) and the gateway’s token certificate (PUSHLEFT_REGISTRY_TOKEN_CERT_FILE) |
Fill in the template’s REPLACE_* values: the registry’s own R2 bucket, endpoint and keys.
- Use a bucket of its own, not a prefix of the shared blob bucket. zot then runs without a
rootdirectory, which avoids the listing problem in finding 2. - Give that bucket no lifecycle rule and no lock rule, unlike
pushleft-offsite. zot’s retention and GC decide what goes. A lifecycle rule would delete blobs that live images still use. A lock would refuse GC’s deletes, so R2 would only grow. - zot’s S3 sizes must be JSON strings (finding 4).
Gateway settings (deploy/systemd/agent-gateway.env.example):
- rendering:
PUSHLEFT_ZOT_CONFIG_TEMPLATE,PUSHLEFT_ZOT_CONFIG_OUT,PUSHLEFT_ZOT_CREDENTIALS_OUT; - keep tags:
PUSHLEFT_REGISTRY_URLandPUSHLEFT_REGISTRY_CA_FILE; PUSHLEFT_REGISTRY_SYNC_INTERVAL_SECS(default 30).
The gateway unit can write only /var/lib/pushleft/zot-rendered. Its Vault policy reads organizations/data/+/registry/upstreams/*. The API’s policy writes there (create, update, metadata delete). Database grants are in deploy/README.md: publish_image_artifact, registry_upstream_host_available, and DELETE on registry_upstreams and registry_retention_rules.
Reaching the realm. zot’s auth.bearer.realm is the URL where every client gets tokens.
- Nodes never use it. They get tokens over mTLS.
- The scheduler calls the gateway directly (
PUSHLEFT_REGISTRY_TOKEN_URL). - People and CI use it. The public TLS proxy therefore serves the registry host. It forwards
/v1/registry/tokento the gateway’s realm listener (PUSHLEFT_REGISTRY_TOKEN_BIND, 8446), and everything else to zot on 5000.
The realm answers 401 to anything but the scheduler secret or a live plr_ token. A plr_ secret has 256 random bits, so the realm is safe to expose.
Startup order. The gateway renders the config on its first pass. The unit refuses to start zot before config.json exists. zot polls the file every second and reloads sync, retention and GC settings without a restart. A changed password rewrites the credentials file and then the config, and the config change makes zot reload.
Token decisions
GET /v1/registry/token?service=pushleft-registry&scope=repository:<name>:<actions> (scope can repeat) with HTTP Basic:
- The scheduler’s secret gets
deleteon one repository that holds a released built artifact (unchanged). - A
plr_token (any username):- The token must be live (unrevoked and unexpired), and its organization must not be suspended.
- For each scope, the access is the requested actions that the token holds. The name must be inside the organization’s namespaces and match one of the token’s patterns. Push is allowed only under
organizations/<org>/, outsideprojects/. - The registry leaves out scopes that grant nothing. A request without scope (
docker login) therefore gets a token with no access. - Tokens are ES256, valid 15 minutes, with
sub=registry-token:<id>. The gateway recordslast_used_at.
- Anything else gets 401.
Node tokens (POST /v1/agent/registry/token) are unchanged. A node gets pull on the repository of any release artifact placed on it. This now includes deployed images and proxy/ images.
Migration from distribution
Nothing in production runs on distribution. For an existing registry:2 (a test fleet), copy every repository by digest and keep the digests. Then switch the units:
# with an admin token for both registries (or a plr_ push token per organization on the zot side)
for repo in $(crane catalog old.example:5000); do
for tag in $(crane ls old.example:5000/$repo); do crane copy old.example:5000/$repo:$tag registry.example/$repo:$tag; done
done
# digest-only images (every Pushleft build): copy the digests the database still references
psql -Atc "SELECT image FROM release_artifacts WHERE image_released_at IS NULL" | while read -r image; do
crane copy "$image" "registry.example/${image#*/}"
done
crane copy and regctl image copy keep manifests byte for byte, so digests do not change.
If the registry host changes, release_artifacts.image keeps the old host. Keep the old name resolving to the new registry (the simplest), or redeploy. You can also configure the old registry as a temporary zot sync upstream with onDemand, and let the first pull copy each image.
The offline GC for distribution is removed: scripts/registry-gc.sh, pushleft-registry-gc.timer and deploy/registry/config.yml. zot collects garbage online and has no read-only window.
zot metadata
zot keeps push and pull times and tags in a local database (/var/lib/pushleft/zot). Retention decisions depend on it.
- If the database is lost, zot rebuilds what it can from storage on start (
--force-reparse, untested). - Push times are then unknown, and zot keeps untagged images without statistics.
- Back up the directory with the control-plane host, or accept that retention restarts its clocks after a rebuild.
A finding while building retention: a tag delete in zot leaves the image untagged without statistics, so retention keeps it forever. The gateway therefore never deletes a pl-keep-* tag directly:
- It moves the tag onto a two-byte placeholder manifest. The released image keeps its statistics, as after any tag move.
- It then deletes the tag.
One placeholder per repository remains.
Multipart part sizes on R2
R2 rejects a multipart upload whose non-trailing parts differ in size (finding 1). zot turns each PATCH of a chunked upload into its own S3 part. Only clients that send several uneven PATCHes are affected. Measured on Garage through the recording proxy (npm run test:registry):
| Client | Upload | S3 parts |
|---|---|---|
| crane 0.21.7 | one streamed PATCH | 16, 16, 8 MiB (equal) |
docker push (Docker Desktop, Docker 29.6, containerd image store) | one streamed PATCH | 16, 16, 16, 12 MiB (equal) |
| the agent’s build push | one PUT | equal (spike) |
a client that sends 7, 13 and 21 MiB PATCHes | three PATCHes | 7, 13, 16, 5 MiB (uneven) |
Decision: no front proxy for now. The common clients send one stream.
- A client that chunks unevenly fails at the end of its push on R2 (
CompleteMultipartUploadrefused). No partial image is stored. - If that happens in practice, add a small normalizing proxy in front of zot.
- Not yet checked: BuildKit
--push,skopeowith a chunk size, GitLab CI tooling, and anything on real R2.
Tests
npm run test:registry is in test:all and takes about 3 minutes. It runs zot on Garage behind the recording proxy, the real API for two organizations, the gateway (realm and reconciler). It also runs OpenBao with the shipped policies, a credentialed local upstream, crane and docker push. It checks:
- token scope, cross-organization refusal, and revoked, expired and rotated tokens;
docker loginandpl registry login;- pull-through with credentials: the API and the other organization cannot read them, and the host cannot be shared;
- deploy by digest: refused until allowed, with built and
proxy/images; - retention: the live digest is kept, older untagged images are deleted, and the released digest is deleted;
- the part sizes above.
PUSHLEFT_REGISTRY_E2E_DOCKERHUB=1 adds a live Docker Hub pull-through. npm run test:pl-up-demo deploys a plr_-pushed image by digest through the real runtime path (pl up demo). test:build-executor, test:retention and test:pl-up-demo run against zot.
Gaps
- Per-role images (
image:is for the whole deployment, likebuild:). - Private images on shared public hosts (GHCR, gitlab.com, Docker Hub), because of zot’s per-host credentials.
- The console has no registry page yet (CLI and API only).
- Production stores images on R2. The uneven-part check (finding 1) has not run against R2 yet.
- Not yet in the multi-VM fleet test (it still runs
registry:2.8.3). - zot’s metadata database has no tested restore.
Engine spike
scripts/registry-spike.sh compares two engines on the same setup. It is not in test:all, takes a few minutes and uses pinned images.
- distribution 3.1.2 (
registry@sha256:ddf75434…), the successor of the engine Pushleft ran before. - zot v2.1.22 (
ghcr.io/project-zot/zot@sha256:96cda114…).
The setup:
- Both engines run on Garage (the R2 stand-in in tests,
scripts/fixtures/garage.sh), through a recording proxy (scripts/fixtures/s3-recording-proxy.py). The proxy logs every S3 request with its body length and copy range. That log shows the multipart part sizes each engine sends. - Garage itself accepts uneven parts, so the proxy is the check for R2’s rule.
- Tokens come from
scripts/fixtures/registry-realm.py. It signs exactly like the gateway (ES256,x5c, issuerpushleft-agent-gateway, audiencepushleft-registry, 15 minutes), with the keys fromscripts/fixtures/registry-token.sh. - The pull-through upstream is a local
registrywith htpasswd credentials. Docker Hub is left out to avoid rate limits. - The S3 part size is 16 MiB in both engines (
chunksize,multipartcopychunksize,multipartcopythresholdsize), the same as the blob store.
Results (2026-10-20)
| Check | distribution 3.1.2 | zot 2.1.22 |
|---|---|---|
| Push and pull, 3 layers including 110 MiB, each layer read back by digest (crane, token realm) | pass | pass |
| Anonymous read refused (401) | pass | pass |
| Token scope: wrong repository, missing action, outside the user’s prefix | pass | pass |
Bearer token sent directly (the agent’s Docker registrytoken path) | pass | pass |
| Gateway-style token accepted | yes, through rootcertbundle and x5c | yes, verified with the key of the certificate in auth.bearer.cert (the gateway’s) |
| Monolithic 41 MiB PUT (the agent’s push path) | pass; S3 parts 16, 16, 9 MiB | pass; S3 parts 16, 16, 9 MiB |
| One streamed PATCH, 110 MiB layer (crane) | parts 6 × 16 MiB + 14 MiB | parts 6 × 16 MiB + 14 MiB |
| Three uneven PATCHes (7, 13, 21 MiB) | uneven parts: 7, 13, 16, 5 MiB | uneven parts: 7, 13, 16, 5 MiB |
| Commit (server-side copy to the final path) | equal parts | equal parts |
| Delete manifest by digest (202, then 404) | pass | pass |
| Garbage collection | offline only (registry garbage-collect); removed the deleted image’s blobs and kept the live one | online, inside the serving process; removed the deleted image’s blobs from S3 and kept the live one |
| Native retention (keep last N, delete untagged) | none | yes: keepTags with mostRecentlyPushedCount, deleteUntagged after a delay; kept v2 and v3 of three v* tags and removed v1 and an untagged manifest |
| Pull-through from a credentialed upstream | pass, but one upstream per instance, and that instance cannot take pushes | pass (sync with onDemand), several upstreams in one instance, each with its own credentials and destination prefix, next to normal repositories |
Findings
- Uneven multipart parts. Both engines turn each
PATCHof a chunked upload into its own S3 part, once it is at least 5 MiB. A client that sends uneven chunks therefore produces uneven non-trailing parts. R2 should then refuseCompleteMultipartUpload, and the push fails. The failure is expected and not yet seen on R2.- The two engines produced identical part shapes in every case. zot uses distribution’s S3 storage driver, so the result fits.
- Clients that send one stream are fine: the agent (one
PUT) and crane (onePATCH). - Before user and CI pushes open, check other clients on real R2:
docker push, BuildKit/containerd,docker buildx --push, GitLab CI tooling, andskopeowith a chunk size.
- distribution with a
rootdirectoryon Garage deletes everything in GC. Withrootdirectoryset, the driver lists repositories withstart-after=<root>. Garage answers that with an empty list whenstart-aftersorts before the prefix. The catalog is then empty, andgarbage-collectmarks 0 blobs and deletes every blob, live images included.- The spike showed this, and
curlagainst Garage directly reproduced it. - The spike runs without
rootdirectory(one bucket per instance). - R2’s behaviour here is unverified.
- Never run GC on a new storage backend without a
--dry-runthat marks the live images first.
- The spike showed this, and
- Digest-only pushes and
deleteUntagged. Pushleft pushes every built image by digest, without a tag. zot’sdeleteUntagged(like distribution’s--delete-untagged) would delete all of them. Retention policies must exclude platform repositories, or the platform must tag what it deploys (for example, by release ID). zot has no rule to keep a digest that a live deployment uses. Pushleft enforces it by tagging live digests or by excluding repositories. - zot details.
- Docker media types (crane’s default, and many Docker Hub images) need
http.compat: ["docker2s2"]. - Sync together with S3 storage needs a local
sync.downloadDir. - The S3 sizes in
storageDrivermust be JSON strings. zot read the number16777216as1, and startup failed. - zot keeps its metadata database on local disk unless it gets Redis or DynamoDB, so a zot instance is not stateless. That local state needs a backup or a rebuild path. A reparse of storage on startup exists (
--force-reparse). This last point is from the zot documentation and was not tested.
- Docker media types (crane’s default, and many Docker Hub images) need
- Garage ignores
If-None-Match: *on PUT. The registries never rely on it, but it limits what the blob store tests can prove (retention: S3-compatible source store).
Recommendation
zot. It covers three of the four scope items natively. zot has online GC, retention rules per repository, and a pull-through cache for several upstreams in one instance next to Pushleft’s own repositories. distribution would need three extra proxy instances (one per upstream), a retention engine of Pushleft’s own, and the offline GC window. Both engines accept the gateway’s tokens unchanged and behave the same on S3. The token issuer and the blob store need no change for either engine.
What the choice does not settle:
- The uneven-part limit (finding 1) affects both engines. Before user and CI pushes open, verify the common chunked clients against real R2. Then decide whether to refuse or merge uneven chunks in front of the registry.
- zot’s local metadata (finding 4) needs a backup and restore story, or Redis.
- The rule to never delete a live digest stays with Pushleft (finding 3).
Next steps, in order:
- Run the spike’s S3 checks against a real R2 bucket.
- Switch the build and runtime paths to zot behind the same token realm (
docker2s2on). - Map per-repository user and CI tokens onto the gateway realm.
- Set up upstream credentials from Vault for
sync. - Add retention defaults with live digests protected.
Related
- The manifest:
image:andbuild:. - Isolated build executor: how Pushleft builds and pushes images.
- Retention: release retention and registry blob GC.
- Agent gateway: the registry’s token issuer.
- Registry outage runbook