pushleft
Deploy your first app
Manual contents
Handbook
HandbookPushleftArchitecture
Operate
Install the control planeYour own serversFleetOperations and alertsAuthenticationDomains and DNSUsage and budgetsScheduled jobsAbuse controlsAI provider proxyCustomer mailErrors and performanceLogs and metricsRetentionObject storageContainer registryGit server
Develop
The .pl.yaml manifestOnboard with an AI agentThe pl up demoBuild executorSource ingestionAgent gatewayManaged databasesSecurity findingsTunnels and shellsDev workspacesPrivate hostnamesMesh coordinatorsDesign baseline
Runbooks
Control-plane host failoverControl-plane PostgreSQL failoverControl-plane PostgreSQL restoreCustom domains and certificatesGateway key compromiseIngress host failureNode busyNode disk fullNode drain and replacementNode lostRegistry outageHetzner Robot hostsSecret rotationSecurity incidentStaging fleet on ProxmoxStaging build cacheStuck deploymentVault/OpenBao backup and recovery
Manual · docs/scheduled-jobs.md

Scheduled jobs

A scheduled job runs a script from your repository on a schedule, in a throwaway VM that can run Docker. Pushleft sends mail when the script fails. A run that succeeds sends no mail.

Scheduled jobs and cron roles

A cron role and a scheduled job both run on a five-field UTC schedule. Neither starts a second run while one is active. After an outage, both start one run for the latest tick.

Needcron roleScheduled job
CodeThe image of the live releaseA checkout of the production branch at one commit
DockerNoYes
Time limit1 hourYour timeout, up to 6 hours
Files between runsNoThe paths in cache:
Failure reportThe last lines, on the deployment’s eventsMail with the failed lanes and tests
Runs onThe application nodesThe workspace pool

Use a cron role for a small task inside a release, such as a cleanup query. Use a scheduled job for a test suite, a nightly gate or other work that needs Docker or a long run.

pl schedules list shows only scheduled jobs. To see your cron roles, use pl cron list:

pl cron list --production

It prints one line per cron role: its schedule, its next run, and the start and result of its last run. Use --environment <slug> for another environment. pl logs <deployment-id> shows the output of each run.

The manifest

Add a schedules: block to .pl.yaml. A project can have 20 schedules. The key is the schedule’s name, a DNS-safe slug.

schedules:
  nightly-gate:
    cron: "0 1 * * *"
    branch: main
    run: GLUON_RUN_GATE=1 scripts/push
    runner: { cpu: 14, memory: 32GiB, disk: 200GiB }
    timeout: 3h
    cache: [~/.cargo, target, ~/.local/share/pnpm, ~/.cache/ms-playwright]
    environment: ci
    secrets: [ANTHROPIC_API_KEY]
    report:
      to: [owners]
      summary: reports/gate-summary.json
FieldRequiredMeaning
cronYesFive fields, UTC.
branchYesThe production branch of the project.
runYesA bash command of up to 4096 bytes. It starts in the checkout.
runner.cpuYes1 to 64 vCPUs.
runner.memoryYes1GiB to 512GiB, in whole GiB.
runner.diskYes10GiB to 2048GiB, in whole GiB.
timeoutYes1m to 6h, written as 90s, 45m or 3h.
cacheNoUp to 16 paths that stay between runs. A path starts with ~/ or is relative to the checkout. See the cache.
environmentWith secretsThe environment that holds the secrets.
secretsNoNames of the secrets the job receives.
report.toNo[owners], the default and the only value.
report.summaryNoThe path of the summary file, relative to the checkout.

pl plan checks this block. When you are signed in, it also checks the branch, the environment and the secrets on the server.

The production branch

A schedule runs only on the production branch. The job runs code from its branch, so anyone who can push to that branch can read the job’s secrets.

A hosted repository or a repository with a webhook names its production branch. For any other repository, the production branch is the branch of the production deployment. An admin who runs pl up --production from another branch moves it.

At each tick, Pushleft reads the head commit of the branch and records that commit on the run. This works when deploy_on_push is off.

Changes to a schedule

A production deploy checks its schedules. An invalid schedule refuses the deploy. The schedules take effect when the production release goes live. A release that fails leaves the old schedules in place.

Repository access

Pushleft reads the repository when each run starts: the head of the branch, as an archive. A production deploy with schedules needs a read token for the repository’s host:

pl git credential add <host> --provider <github|gitlab> --token-stdin

A push webhook is not enough. Deploy from a checkout with an origin remote, so that Pushleft knows the repository.

The checkout is the git host’s archive of that commit: every tracked file, symlinks included, and nothing that git does not track. It is at most 2 GiB compressed.

The runner

Each run gets one Kata VM with the runner size. Pushleft destroys the VM when the run ends.

  • The script runs as root in the VM.
  • Docker is available. The script starts its own databases and other services, and they end with the VM.
  • Rust, C and C++, Node.js and pnpm are installed.
  • The VM reaches the public internet only.

Secrets and the environment

A job receives only the secrets it lists, as environment variables. They come from the one environment in its environment field. A job gets production secrets only when the field says production.

pl plan and the API refuse these:

  • secrets with no environment;
  • an environment that does not exist;
  • a secret that the environment does not hold.

The cache

cache: is opt-in. Without it, a run gets one work disk, and nothing stays after the run.

With cache:, the listed paths stay between runs:

  • The volume takes half of runner.disk. The work disk gets the rest.
  • One volume exists per schedule and branch on a node. Stale files are the job’s concern.
  • The volume stays on the node that made it. The scheduler picks that node first when the node qualifies.
  • When that node doesn’t qualify, the run goes to another node with an empty cache. The run is slower, and the console marks it Cold cache. A later run uses the newest volume.
  • A volume unused for 30 days is deleted.
  • pl schedules cache-clear <name> deletes the volumes of a schedule. The next run starts empty.

Cost

A run uses the workspace prices of your plan. CPU and memory count only during a run.

A kept cache volume is billed for the whole month as reserved disk. On your own server, runs and cache cost nothing. A budget with hard_stop stops new runs.

The summary file

The script can write this JSON file at the path in report.summary:

{
  "lanes": [
    { "name": "rust", "status": "passed", "failed_tests": [] },
    { "name": "browser", "status": "failed", "failed_tests": ["checkout pays with a saved card"] }
  ]
}

Pushleft keeps 100 lanes, 200 failed tests per lane and 200 characters per text value. It ignores other fields. It doesn’t read the file after a timeout.

Mail

Mail goes to the owners of the organization. It links to the run in the console.

ResultMail
failedYes
timed_outYes
no_capacityYes
succeededNo
skipped, the first in a rowNo
skipped, the second in a rowOne mail, because the job runs longer than its schedule

With a summary file, the mail lists the failed lanes and tests. Without it, the mail shows the exit code and the last 50 lines. Each mail names the commit and the duration of the run.

Run results

StatusMeaning
pendingThe run waits for a node. Pushleft gets the commit when the run starts.
waitingNo node has room yet. The scheduler checks again every minute.
runningThe script runs.
succeededThe script exited with 0.
failedThe script exited with another code, or the run could not start.
timed_outThe run reached timeout, and Pushleft stopped the VM.
no_capacityNo node had room for 1 hour.
skippedThe previous run was still active. Runs never queue.
cancelledSomeone cancelled the run. No mail is sent.

To stop a run that is stuck or no longer needed, select Cancel run on its page in the console, or run pl schedules cancel <name>. Members and above can cancel. The run stops within a minute: Pushleft removes its VM and keeps the cache. A run whose VM stops by itself ends as failed within 30 seconds.

Output and limits

  • A run can store 200 MiB of output. Past that limit, the log keeps the first and last 100 MiB and marks the cut.
  • Output is stored with your tier’s log retention of 7, 14 or 30 days.
  • Output doesn’t count against the daily output limit of services.
  • The Deployments page and pl logs don’t list scheduled runs.

The Schedules page of the console lists each schedule and its runs. The page of one run shows the lanes, the failed tests and the log. You can search the whole log and download it.

Commands

Run these in the directory that holds .pl.yaml.

CommandResult
pl schedules listEach schedule, its next run and its last result
pl cron list [--production | --environment <slug>]Each cron role of the environment, its next run and its last result
pl schedules run <name>Starts a run now. Viewers can’t. A schedule with an active run refuses it
pl schedules logs <name> [--run <id>] [-f]The output of the newest run, or of the run you name. -f keeps reading while the run is active
pl schedules cache-clear <name>Deletes the cache volumes of the schedule
pl schedules cancel <name> [--run <id>]Stops the active run, or the run you name. Viewers can’t. No mail is sent

Hosts

The scheduler picks the host when each run starts. No manifest or setting names a host. Only nodes of the workspace pool in your organization qualify. Application and database nodes never run a job.

A node qualifies when these are true:

  • its free capacity fits the runner size;
  • its heartbeat is fresh;
  • its CPU and memory use over the last 15 minutes leaves room for the runner.

Among those nodes, the scheduler takes the one with the most free CPU.

To use your own server:

  1. Prepare a workspace host. The host needs KVM. Enrollment for the workspace pool stops without /dev/kvm.
  2. Set PUSHLEFT_RUNNER_IMAGE in /etc/pushleft/agent.env to the runner image, and restart the agent.
Rendered from the repository at build time. Edit docs/scheduled-jobs.md to change it.