pushleft
Deploy your first app
Manual contents
Handbook
HandbookPushleftArchitecture
Operate
Install the control planeYour own serversFleetOperations and alertsAuthenticationDomains and DNSUsage and budgetsScheduled jobsAbuse controlsAI provider proxyCustomer mailErrors and performanceLogs and metricsRetentionObject storageContainer registryGit server
Develop
The .pl.yaml manifestOnboard with an AI agentThe pl up demoBuild executorSource ingestionAgent gatewayManaged databasesSecurity findingsTunnels and shellsDev workspacesPrivate hostnamesMesh coordinatorsDesign baseline
Runbooks
Control-plane host failoverControl-plane PostgreSQL failoverControl-plane PostgreSQL restoreCustom domains and certificatesGateway key compromiseIngress host failureNode busyNode disk fullNode drain and replacementNode lostRegistry outageHetzner Robot hostsSecret rotationSecurity incidentStaging fleet on ProxmoxStaging build cacheStuck deploymentVault/OpenBao backup and recovery
Manual · docs/runbooks/staging-build-cache.md

Staging build cache

Use this runbook to turn on, check or roll back the build cache on the staging fleet.

The cache is optional. Without PUSHLEFT_BUILD_CACHE_ROOT, every build starts with empty BuildKit state and does not ask the gateway for a snapshot. A successful image push does not fill the cache for the next build.

Before you start

  • You need SSH to build112 (stg-build, on pve2) and cp110 (stg-cp, on pve1). See staging.md.
  • The gateway binary on cp110 must support the cache-specific local root (PUSHLEFT_CACHE_LOCAL_ROOT). Save the current binary for rollback.
  • Check the physical space of the thin pools (build-thin on pve2, ssd-thin on pve1). The virtual size of a thin disk does not reserve space.

Add the cache disks

The 2026-10-28 rollout added one 64 GiB virtual disk, virtio3, to each of build112 and cp110. The existing data disk and backup disk keep their roles.

  1. Add a 64 GiB disk as virtio3 to build112 (build-thin) and to cp110 (ssd-thin).
  2. On each VM, check the serial and size of the new device.
  3. Destroys data: format only the new, empty device as XFS. Do not copy a mkfs -f command onto an existing host.
  4. Add the filesystem to /etc/fstab by UUID, with the options defaults,prjquota,nodev,nosuid.

Configure the build host (build112)

  1. Mount the cache at /var/lib/pushleft/build-cache, owned by root, mode 0700.

  2. Write /etc/pushleft/build-cache-agent.env:

    PUSHLEFT_BUILD_CACHE_ROOT=/var/lib/pushleft/build-cache
    PUSHLEFT_BUILD_CACHE_QUOTA=xfs
    PUSHLEFT_BUILD_CACHE_PROJECT_MAX_MB=10240
    PUSHLEFT_BUILD_CACHE_MAX_MB=32768
  3. Write /etc/systemd/system/pushleft-agent.service.d/build-cache.conf:

    [Unit]
    RequiresMountsFor=/var/lib/pushleft/build-cache
    ConditionPathIsMountPoint=/var/lib/pushleft/build-cache
    
    [Service]
    EnvironmentFile=/etc/pushleft/build-cache-agent.env

    The existing write permission of the service on /var/lib/pushleft covers this mount.

  4. Check that project accounting and enforcement are on:

    xfs_quota -x -c 'state -p' /var/lib/pushleft/build-cache

    The agent also checks this at startup. It applies a 20 GiB project quota to each cache directory, for the old and new generation together. Since 2026-11-05 a project has two: <project id> for production and <project id>-preview for its previews.

Configure the gateway host (cp110)

  1. Mount /var/lib/pushleft/build-cache-snapshots, owned by pushleft:pushleft, mode 0700.

  2. Create cache and tmp below it, with the same owner and mode.

  3. Write /etc/pushleft/build-cache-gateway.env:

    PUSHLEFT_CACHE_LOCAL_ROOT=/var/lib/pushleft/build-cache-snapshots/cache
    PUSHLEFT_GATEWAY_CACHE_MAX_MB=11264
    TMPDIR=/var/lib/pushleft/build-cache-snapshots/tmp

    Warning: do not use PUSHLEFT_BLOB_LOCAL_ROOT here. It also moves source delivery. PUSHLEFT_BACKUP_LOCAL_ROOT keeps the import and backup handoff store separate, and source delivery keeps PUSHLEFT_SOURCE_ROOT.

  4. Write /etc/systemd/system/pushleft-agent-gateway.service.d/build-cache.conf:

    [Unit]
    RequiresMountsFor=/var/lib/pushleft/build-cache-snapshots
    ConditionPathIsMountPoint=/var/lib/pushleft/build-cache-snapshots
    
    [Service]
    EnvironmentFile=/etc/pushleft/build-cache-gateway.env
    ReadWritePaths=/var/lib/pushleft/build-cache-snapshots

Restart the services

  1. Make both environment files root-owned, mode 0600.

  2. Reload systemd on both hosts.

    systemctl daemon-reload
  3. Check that no builds are active. Restarting the agent stops its current builder.

  4. Restart only the affected service: pushleft-agent on build112, pushleft-agent-gateway on cp110.

The mount requirements stop the services when a disk is absent. Cache writes then cannot go to the root filesystem by accident.

Check it worked

Use a small separate project first. Never remove the cache of another project for this check.

  1. Run a production build. Its cache result reports imported: none, exported: true, uploaded: true.
  2. Change a file that only a late COPY reads, and run a second production build. It reports imported: local, with the earlier RUN steps cached. Do not resend an identical tree. Since 2026-11-05 an identical finished build of the organization is reused without a build, with no cache result.
  3. Run a preview. It imports and writes the project’s preview cache, not production’s, and reports an export and upload. On an agent that does not advertise build_cache_scopes, a preview still imports the production cache read-only and reports no export or upload.
  4. Park the local generation of the canary while its builder is idle. The next production build reports imported: snapshot.
  5. Remove the canary deployments.

A rolling application deployment also needs room for the replacement instance while the current one serves traffic. The sample tenant app needs 2 more CPUs and 4 GiB on the application pool. During the rollout, the app node had only 1 CPU free. The canary build checks completed, but runtime placement waited for capacity. Do not tear down unrelated applications to make room without approval.

Check with the sample tenant app

  1. Build the sample tenant app from an isolated source checkout. Keep its secrets, data stores and volume.

  2. Leave out the canceled login optimization from the shared prepared checkout.

  3. Check the startup script and the nginx configuration together. The rebuilt source declares user root and master_process off in nginx.conf, so its runtime command is [/bin/sh, /app/start.sh].

    Warning: the old pinned image needed a wrapper that injected these directives. With that wrapper in the rebuilt image, nginx fails with a duplicate user directive.

  4. Build again with an identical application context. Check that:

    • The Yarn step is cached.
    • No dependency download repeats.
    • The cache report has no error.
    • The application health check passes.
  5. To check the production cache read-only, use a one-shot preview role with the command [/bin/sh, -c, 'true']. This avoids another handoff of the volume of the serving application. This worked on 2026-10-28, when every preview read the production cache. A preview on an agent that advertises build_cache_scopes now uses its own preview cache, so this check no longer reads production’s.

    • Declare no databases, secrets or volumes for that preview.
    • Put its deployment manifest outside the frozen build context. A changed role definition then does not change the COPY . ./ input of the API.
  6. Tear the preview down after the check.

Sample app cache invalidation

The sample tenant app Dockerfile keeps Yarn installation apart from the UI source:

  • node_modules is in a dependency stage, exported by mode=max. That stage is bind-mounted only for compilation.
  • This is a BuildKit stage bind mount. Pushleft does not export type=cache mounts.
  • API publishing runs after UI compilation, in sequence.
  • A broad COPY . ./ invalidates API publish when any included source changes. This includes frame/, the deployment scripts and the SOURCE_REVISION of the prepared artifact. NuGet restore is part of that step.
  • A change to the package manifests, the lockfile or the Yarn configuration invalidates the UI dependency stage.
  • A change to the UI source invalidates compilation and keeps the dependency layer.
  • Unchanged inputs can reuse both compiled stages.

A changed base-image digest invalidates the layers that depend on it, so mutable FROM tags can cause misses. A different date or file modification time alone does not expire a layer. Cache reuse follows the Dockerfile instructions and the content and metadata of copied files. It does not depend on whether another build succeeded that day.

Capacity and limits

The _MB values of the agent are MiB.

  • The limit is 10 GiB per project. The agent tries to keep at least 20 GiB free on the cache partition before and after a build. It evicts other projects by last use.
  • A 64 GiB filesystem with a 32 GiB steady cache budget leaves room for a new 10 GiB generation and filesystem overhead.
  • The cache of the running project is never evicted.
  • The total budget and the free-space target are best effort. If no other project can be evicted, the build continues. The XFS project quota is the hard limit during export.

Cache snapshots are uncompressed tar archives of the OCI layout.

  • The gateway keeps two per project and scope (production, and previews under <org>/preview/<project>/). A third is present for a short time during replacement.
  • The gateway uses temporary files for transfers.
  • The dedicated disk is the only limit on total disk use. There is no snapshot byte budget for the whole gateway, and no eviction across projects. Monitor the disk as more projects start to cache.
  • When the disk is full, snapshot transfer stops. A valid cache on the node stays usable.
  • Local snapshots on cp are not off-host backups.

The builder keeps its 8 GiB tmpfs and 12 GiB memory limit with disk caching on.

  • Its default workspace, /var/lib/pushleft/builds, still uses the root disk for source, image output and cache transfer tar files.
  • A production build fed from a snapshot can hold an incoming and an outgoing tar there at the same time. At the configured project limit, budget up to 20 GiB, plus source, the image tar (up to 4 GiB) and unpacked image blobs.
  • Gateway temporary files use the snapshot disk through TMPDIR.

The node checks an OCI layout against the project limit, but the limit on a downloaded snapshot includes tar overhead. An export near the limit can stay usable on the node and still fail snapshot restoration. Measure the real cache sizes and leave a margin below 10 GiB.

Results, 2026-10-28

All builds used the staging Kata runtime with enforced XFS project quotas. The times are cache.build_ms. That covers cache attachment and BuildKit. It does not cover the final layout check on the host, snapshot upload or image publication, so these are not end-to-end deployment times.

BuildSecondsCache result
Small fixture, cold production23.928no import; exported and uploaded
Same fixture, warm production14.986local import; both RUN steps cached
Same fixture, parked local generation15.120snapshot import; both RUN steps cached
Same fixture, preview14.344local import; no export or upload
Sample tenant app, cold production1165.581valid 1,559,715,840-byte cache exported and uploaded
Same sample tenant app context, one-shot preview86.653local import; Yarn install, UI build and API publish all CACHED; no export or upload
  • Both builds of the sample tenant app produced the image digest sha256:f4b2a2489a42b02445321b4de1c6f0fd956906ced6cdc61fcc25428e0a7e55b7.
  • The serving deployment is 01a1846a-5de0-7669-8e70-4b65989c2165. It starts directly, and /health reports PostgreSQL and ClickHouse connected.
  • The web-only and dbapp smoke deployments were stopped, with explicit approval, to free application capacity.
  • The retained dbapp PostgreSQL instance is still ready. Its existing retention policy expires on 2026-11-04 at 21:17:53 UTC.
  • All cache-validation deployments were torn down.

The first runtime rollout carried the old nginx wrapper into the rebuilt configuration, and startup failed. The single-writer volume handoff had already stopped the previous instance at 21:24:11 UTC. The corrected command restored the route at 21:29:59 UTC, an interruption of about six minutes.

Warning: for a role with a volume, do not expect rolling to keep the previous writer running until the replacement passes its health checks.

The audit, build events, step-level cache checks, health results and rollback manifests are kept on the operator machine in ~/.config/pushleft-staging/build-cache-rollout/. The shared prepared checkout and its canceled login optimization were not changed or deployed.

Roll back

  1. Remove only the two cache drop-ins and their environment files.
  2. Reload systemd.
  3. Restart the affected services while they are idle.
  4. If needed, restore the saved gateway binary.
  5. To roll back the application, deploy its recorded previous image and manifest again. The cache rollout records keep that manifest.

Keep the disks and their data mounted, for inspection or a later re-enable. Do not remove the backup and import drop-in, and do not change the source-store settings.

Rendered from the repository at build time. Edit docs/runbooks/staging-build-cache.md to change it.