Staging build cache
Use this runbook to turn on, check or roll back the build cache on the staging fleet.
The cache is optional. Without PUSHLEFT_BUILD_CACHE_ROOT, every build starts with empty BuildKit state and does not ask the gateway for a snapshot. A successful image push does not fill the cache for the next build.
Before you start
- You need SSH to build112 (
stg-build, onpve2) and cp110 (stg-cp, onpve1). See staging.md. - The gateway binary on cp110 must support the cache-specific local root (
PUSHLEFT_CACHE_LOCAL_ROOT). Save the current binary for rollback. - Check the physical space of the thin pools (
build-thinonpve2,ssd-thinonpve1). The virtual size of a thin disk does not reserve space.
Add the cache disks
The 2026-10-28 rollout added one 64 GiB virtual disk, virtio3, to each of build112 and cp110. The existing data disk and backup disk keep their roles.
- Add a 64 GiB disk as
virtio3to build112 (build-thin) and to cp110 (ssd-thin). - On each VM, check the serial and size of the new device.
- Destroys data: format only the new, empty device as XFS. Do not copy a
mkfs -fcommand onto an existing host. - Add the filesystem to
/etc/fstabby UUID, with the optionsdefaults,prjquota,nodev,nosuid.
Configure the build host (build112)
-
Mount the cache at
/var/lib/pushleft/build-cache, owned by root, mode 0700. -
Write
/etc/pushleft/build-cache-agent.env:PUSHLEFT_BUILD_CACHE_ROOT=/var/lib/pushleft/build-cache PUSHLEFT_BUILD_CACHE_QUOTA=xfs PUSHLEFT_BUILD_CACHE_PROJECT_MAX_MB=10240 PUSHLEFT_BUILD_CACHE_MAX_MB=32768 -
Write
/etc/systemd/system/pushleft-agent.service.d/build-cache.conf:[Unit] RequiresMountsFor=/var/lib/pushleft/build-cache ConditionPathIsMountPoint=/var/lib/pushleft/build-cache [Service] EnvironmentFile=/etc/pushleft/build-cache-agent.envThe existing write permission of the service on
/var/lib/pushleftcovers this mount. -
Check that project accounting and enforcement are on:
xfs_quota -x -c 'state -p' /var/lib/pushleft/build-cacheThe agent also checks this at startup. It applies a 20 GiB project quota to each cache directory, for the old and new generation together. Since 2026-11-05 a project has two:
<project id>for production and<project id>-previewfor its previews.
Configure the gateway host (cp110)
-
Mount
/var/lib/pushleft/build-cache-snapshots, owned bypushleft:pushleft, mode 0700. -
Create
cacheandtmpbelow it, with the same owner and mode. -
Write
/etc/pushleft/build-cache-gateway.env:PUSHLEFT_CACHE_LOCAL_ROOT=/var/lib/pushleft/build-cache-snapshots/cache PUSHLEFT_GATEWAY_CACHE_MAX_MB=11264 TMPDIR=/var/lib/pushleft/build-cache-snapshots/tmpWarning: do not use
PUSHLEFT_BLOB_LOCAL_ROOThere. It also moves source delivery.PUSHLEFT_BACKUP_LOCAL_ROOTkeeps the import and backup handoff store separate, and source delivery keepsPUSHLEFT_SOURCE_ROOT. -
Write
/etc/systemd/system/pushleft-agent-gateway.service.d/build-cache.conf:[Unit] RequiresMountsFor=/var/lib/pushleft/build-cache-snapshots ConditionPathIsMountPoint=/var/lib/pushleft/build-cache-snapshots [Service] EnvironmentFile=/etc/pushleft/build-cache-gateway.env ReadWritePaths=/var/lib/pushleft/build-cache-snapshots
Restart the services
-
Make both environment files root-owned, mode 0600.
-
Reload systemd on both hosts.
systemctl daemon-reload -
Check that no builds are active. Restarting the agent stops its current builder.
-
Restart only the affected service:
pushleft-agenton build112,pushleft-agent-gatewayon cp110.
The mount requirements stop the services when a disk is absent. Cache writes then cannot go to the root filesystem by accident.
Check it worked
Use a small separate project first. Never remove the cache of another project for this check.
- Run a production build. Its
cacheresult reportsimported: none,exported: true,uploaded: true. - Change a file that only a late
COPYreads, and run a second production build. It reportsimported: local, with the earlierRUNsteps cached. Do not resend an identical tree. Since 2026-11-05 an identical finished build of the organization is reused without a build, with no cache result. - Run a preview. It imports and writes the project’s preview cache, not production’s, and reports an export and upload. On an agent that does not advertise
build_cache_scopes, a preview still imports the production cache read-only and reports no export or upload. - Park the local generation of the canary while its builder is idle. The next production build reports
imported: snapshot. - Remove the canary deployments.
A rolling application deployment also needs room for the replacement instance while the current one serves traffic. The sample tenant app needs 2 more CPUs and 4 GiB on the application pool. During the rollout, the app node had only 1 CPU free. The canary build checks completed, but runtime placement waited for capacity. Do not tear down unrelated applications to make room without approval.
Check with the sample tenant app
-
Build the sample tenant app from an isolated source checkout. Keep its secrets, data stores and volume.
-
Leave out the canceled login optimization from the shared prepared checkout.
-
Check the startup script and the nginx configuration together. The rebuilt source declares
user rootandmaster_process offinnginx.conf, so its runtime command is[/bin/sh, /app/start.sh].Warning: the old pinned image needed a wrapper that injected these directives. With that wrapper in the rebuilt image, nginx fails with a duplicate
userdirective. -
Build again with an identical application context. Check that:
- The Yarn step is cached.
- No dependency download repeats.
- The cache report has no error.
- The application health check passes.
-
To check the production cache read-only, use a one-shot preview role with the command
[/bin/sh, -c, 'true']. This avoids another handoff of the volume of the serving application. This worked on 2026-10-28, when every preview read the production cache. A preview on an agent that advertisesbuild_cache_scopesnow uses its own preview cache, so this check no longer reads production’s.- Declare no databases, secrets or volumes for that preview.
- Put its deployment manifest outside the frozen build context. A changed role definition then does not change the
COPY . ./input of the API.
-
Tear the preview down after the check.
Sample app cache invalidation
The sample tenant app Dockerfile keeps Yarn installation apart from the UI source:
node_modulesis in a dependency stage, exported bymode=max. That stage is bind-mounted only for compilation.- This is a BuildKit stage bind mount. Pushleft does not export
type=cachemounts. - API publishing runs after UI compilation, in sequence.
- A broad
COPY . ./invalidates API publish when any included source changes. This includesframe/, the deployment scripts and theSOURCE_REVISIONof the prepared artifact. NuGet restore is part of that step. - A change to the package manifests, the lockfile or the Yarn configuration invalidates the UI dependency stage.
- A change to the UI source invalidates compilation and keeps the dependency layer.
- Unchanged inputs can reuse both compiled stages.
A changed base-image digest invalidates the layers that depend on it, so mutable FROM tags can cause misses. A different date or file modification time alone does not expire a layer. Cache reuse follows the Dockerfile instructions and the content and metadata of copied files. It does not depend on whether another build succeeded that day.
Capacity and limits
The _MB values of the agent are MiB.
- The limit is 10 GiB per project. The agent tries to keep at least 20 GiB free on the cache partition before and after a build. It evicts other projects by last use.
- A 64 GiB filesystem with a 32 GiB steady cache budget leaves room for a new 10 GiB generation and filesystem overhead.
- The cache of the running project is never evicted.
- The total budget and the free-space target are best effort. If no other project can be evicted, the build continues. The XFS project quota is the hard limit during export.
Cache snapshots are uncompressed tar archives of the OCI layout.
- The gateway keeps two per project and scope (production, and previews under
<org>/preview/<project>/). A third is present for a short time during replacement. - The gateway uses temporary files for transfers.
- The dedicated disk is the only limit on total disk use. There is no snapshot byte budget for the whole gateway, and no eviction across projects. Monitor the disk as more projects start to cache.
- When the disk is full, snapshot transfer stops. A valid cache on the node stays usable.
- Local snapshots on cp are not off-host backups.
The builder keeps its 8 GiB tmpfs and 12 GiB memory limit with disk caching on.
- Its default workspace,
/var/lib/pushleft/builds, still uses the root disk for source, image output and cache transfer tar files. - A production build fed from a snapshot can hold an incoming and an outgoing tar there at the same time. At the configured project limit, budget up to 20 GiB, plus source, the image tar (up to 4 GiB) and unpacked image blobs.
- Gateway temporary files use the snapshot disk through
TMPDIR.
The node checks an OCI layout against the project limit, but the limit on a downloaded snapshot includes tar overhead. An export near the limit can stay usable on the node and still fail snapshot restoration. Measure the real cache sizes and leave a margin below 10 GiB.
Results, 2026-10-28
All builds used the staging Kata runtime with enforced XFS project quotas. The times are cache.build_ms. That covers cache attachment and BuildKit. It does not cover the final layout check on the host, snapshot upload or image publication, so these are not end-to-end deployment times.
| Build | Seconds | Cache result |
|---|---|---|
| Small fixture, cold production | 23.928 | no import; exported and uploaded |
| Same fixture, warm production | 14.986 | local import; both RUN steps cached |
| Same fixture, parked local generation | 15.120 | snapshot import; both RUN steps cached |
| Same fixture, preview | 14.344 | local import; no export or upload |
| Sample tenant app, cold production | 1165.581 | valid 1,559,715,840-byte cache exported and uploaded |
| Same sample tenant app context, one-shot preview | 86.653 | local import; Yarn install, UI build and API publish all CACHED; no export or upload |
- Both builds of the sample tenant app produced the image digest
sha256:f4b2a2489a42b02445321b4de1c6f0fd956906ced6cdc61fcc25428e0a7e55b7. - The serving deployment is
01a1846a-5de0-7669-8e70-4b65989c2165. It starts directly, and/healthreports PostgreSQL and ClickHouse connected. - The
web-onlyanddbappsmoke deployments were stopped, with explicit approval, to free application capacity. - The retained
dbappPostgreSQL instance is still ready. Its existing retention policy expires on 2026-11-04 at 21:17:53 UTC. - All cache-validation deployments were torn down.
The first runtime rollout carried the old nginx wrapper into the rebuilt configuration, and startup failed. The single-writer volume handoff had already stopped the previous instance at 21:24:11 UTC. The corrected command restored the route at 21:29:59 UTC, an interruption of about six minutes.
Warning: for a role with a volume, do not expect rolling to keep the previous writer running until the replacement passes its health checks.
The audit, build events, step-level cache checks, health results and rollback manifests are kept on the operator machine in ~/.config/pushleft-staging/build-cache-rollout/. The shared prepared checkout and its canceled login optimization were not changed or deployed.
Roll back
- Remove only the two cache drop-ins and their environment files.
- Reload systemd.
- Restart the affected services while they are idle.
- If needed, restore the saved gateway binary.
- To roll back the application, deploy its recorded previous image and manifest again. The cache rollout records keep that manifest.
Keep the disks and their data mounted, for inspection or a later re-enable. Do not remove the backup and import drop-in, and do not change the source-store settings.