pushleft
Deploy your first app
Manual contents
Handbook
HandbookPushleftArchitecture
Operate
Install the control planeYour own serversFleetOperations and alertsAuthenticationDomains and DNSUsage and budgetsScheduled jobsAbuse controlsAI provider proxyCustomer mailErrors and performanceLogs and metricsRetentionObject storageContainer registryGit server
Develop
The .pl.yaml manifestOnboard with an AI agentThe pl up demoBuild executorSource ingestionAgent gatewayManaged databasesSecurity findingsTunnels and shellsDev workspacesPrivate hostnamesMesh coordinatorsDesign baseline
Runbooks
Control-plane host failoverControl-plane PostgreSQL failoverControl-plane PostgreSQL restoreCustom domains and certificatesGateway key compromiseIngress host failureNode busyNode disk fullNode drain and replacementNode lostRegistry outageHetzner Robot hostsSecret rotationSecurity incidentStaging fleet on ProxmoxStaging build cacheStuck deploymentVault/OpenBao backup and recovery
Manual · docs/git.md

Git server

Pushleft hosts private git repositories for your code and deploys every push. Push the production branch to update production, and push any other branch to get a preview. Code that stays on GitHub or GitLab can deploy the same way through a push webhook.

Console SSH keys for git push, on the Security page.

Quick start

  1. Sign in with a fresh login code, then add your SSH public key. Adding a key needs a sign-in from the last 10 minutes (SSH keys):

    pl login --email <you> --code <mailed code>
    pl ssh-key add                     # ~/.ssh/id_ed25519.pub
  2. Create a repository (organization admins). The command prints the clone URL:

    pl git create web                  # ssh://[email protected]/<handle>/web.git
  3. Add the remote and push. A checkout with a .pl.yaml deploys on push:

    git remote add pushleft ssh://[email protected]/<handle>/web.git
    git push pushleft main             # deploys to production
    git push pushleft feature/login    # deploys to preview/feature-login
  4. Follow the deployment with the command that the push prints:

    pl logs <deployment id>

Features

  • Repositories are org-namespaced: <org handle>/<name>, private, created explicitly.
  • Push to deploy. The production branch (default main) deploys to production, every other branch to preview/<branch>. Only admins and owners push the production branch. Deleting a branch tears down its preview.
  • GitHub and self-hosted GitLab push webhooks deploy the same way.
  • Backups of every repository to R2, sealed with the organization’s backup key.
  • SSH only. No web UI, no code review, no HTTP or git-daemon access.
  • Per-tier repository quotas (abuse and limits).

For users

SSH keys

A key belongs to your user, not to one organization: it opens the repositories of every organization you belong to. Adding or removing a key therefore needs a sign-in from the last 10 minutes with a factor you own, like TOTP and passkeys: a login code mailed to you (console “email me a code”, or pl login --email <you> --code <mailed code>), your TOTP or your passkey. An API token cannot add a key, and neither can a session from your organization’s SSO, an invitation, or a login code issued through the API (pl console-code, or by an admin): the organization holds those, and the key would open your other organizations too (docs/authentication.md, “Fresh login”).

pl ssh-key add                     # ~/.ssh/id_ed25519.pub
pl ssh-key add ~/.ssh/work.pub --name work-laptop
pl ssh-key list                    # id, SHA256 fingerprint (as ssh-keygen -l prints it), name
pl ssh-key rm <id>

The console has the same list under Settings → SSH keys. Accepted: Ed25519, ECDSA and their security-key forms, and RSA of 2048 bits or more. One key belongs to one user; a key that another user registered is refused (409). A user has at most 20 keys.

Repositories

Organization admins create repositories. There is no create-on-push: a mistyped remote cannot create a repository, and the tier’s repository cap holds.

pl git create web                       # prints ssh://[email protected]/<handle>/web.git
pl git create site --production-branch trunk
pl git create docs --no-deploy
pl git set web --deploy false           # deploy on push off
pl git set web --production-branch release
pl git list                             # clone URLs, settings, webhooks

<handle> is the organization’s handle (the same 8 or more hex characters as in preview hostnames). GET /v1/git returns the clone URLs, settings, webhooks and the tier limits.

Access follows the organization role, re-read at every push:

RoleClone and fetchPush
vieweryesno
memberyesyes, except the production branch
admin, owneryesyes
suspended organizationyesno
anyone else, or a key no Pushleft user registerednono

Push to deploy

When the pushed commit of a branch has a .pl.yaml, Pushleft deploys it:

RefEnvironment
the production branch (default main)production
any other branch, e.g. feature/Login-Formpreview/feature-login-form (the same name pl up uses)
tags, deleted branchesnot deployed

The source is the pushed commit, packed exactly as pl up packs a checkout: .plignore and .gitignore apply, the same paths are excluded and the same limits hold (source-ingestion.md). The snapshot digest equals the one pl plan --source prints for a checkout of that commit. The push itself never waits for the build; the pusher sees the deployment id:

remote: pushleft: deployment 01a1598d-… queued for production (main @ 5218ac04d2ef)
remote: pushleft: follow it: pl logs 01a1598d-…

A push that cannot deploy (an invalid .pl.yaml, a budget hard stop, the deployment quota) is still accepted; the reason is printed instead. Pushing the same commit to the same branch again replays the existing deployment, it does not create another one.

GitHub and GitLab webhooks

For code that stays on GitHub or a self-hosted GitLab, an admin adds a push webhook. Pushleft fetches the pushed commit’s archive with a read-only token you supply, then deploys it as above.

# GitHub: a fine-grained token with Contents: read on the repository
pl git webhook add --provider github --repository acme/web <<<"$GITHUB_TOKEN"
# GitLab (self-hosted): a project or deploy token with read_repository
pl git webhook add --provider gitlab --repository group/web --api-url https://gitlab.example.com <<<"$GITLAB_TOKEN"
pl git webhook rm <id>

The command prints the webhook URL and a signing secret, once. In GitHub, add a webhook with that URL, content type application/json, the secret, and the push event. In GitLab, add a webhook with that URL, the secret as the secret token, and push events. GitHub Enterprise would take --api-url https://<host>/api/v3 (not tested; only redirects to the same host are followed).

Every delivery is checked before anything else happens:

  • GitHub: X-Hub-Signature-256 must be the HMAC-SHA256 of the exact body under the secret. GitLab: X-Gitlab-Token must equal the secret. Both compare in constant time. A failure answers 401.
  • The delivery id (X-GitHub-Delivery, X-Gitlab-Event-UUID) is accepted once per webhook. A replay answers 409 delivery_replayed.
  • Events other than a push are acknowledged and ignored.

The secret and the token are stored in Vault. No endpoint returns them; pl git list shows the webhook without them. The provider host must be public: api_url may not be a private or reserved IP literal, and every host the API dials (the redirect target included) goes through the shared SSRF filter in the HTTP client’s resolver (pushleft_core::ssrf: private, loopback, link-local, shared, reserved, benchmarking, documentation, NAT64, 6to4-of-private and Teredo addresses are refused), so the address that is checked is the one used. GitHub’s redirect to codeload.github.com is followed; no other redirect is.

The webhook endpoint is public, so deliveries are rate limited per webhook and sender address (git_webhook bucket, PUSHLEFT_AUTH_RATE_TOKEN_IP_LIMIT per PUSHLEFT_AUTH_RATE_WINDOW_SECONDS, 429 with Retry-After beyond it) before the webhook is looked up or its secret is read from Vault.

Internals

The server is the control-plane host’s own OpenSSH on port 22, in the pattern GitLab uses: the git user’s keys come from Pushleft, and a small shell, pushleft-git-shell, checks every request with the API before it runs git on a bare repository.

Decided 2026-10-20. Scope:

  • Users, SSH keys and repositories come from Pushleft. Nothing is mirrored: sshd asks the API for each presented key, and the API decides each clone and push.
  • Abuse limits on the public SSH port, and per-tier repository quotas.
  • One port 22. Operators log in on the same sshd as their own users, from operator addresses only (decided 2026-10-20; replaces an earlier plan in which Soft Serve took port 22).
git push ──ssh:22──▶ OpenSSH ──AuthorizedKeysCommand──▶ pushleft-git-shell authorized-keys %f ──▶ API hook listener: key by fingerprint
                        │
                        └──forced command──▶ pushleft-git-shell ──▶ API: may this key read/write <handle>/<repo>?
                                                   │
                                                   └──exec git receive-pack ──pre-receive──▶ API: role, suspension, size
                                                                               └──post-receive: git archive ──▶ API: pack_tar → deploy → source ingest
GitHub/GitLab ──https──▶ API /v1/git/webhooks/{id}: signature, delivery id ──▶ provider archive ──▶ same path
  • Keys. sshd’s drop-in deploy/git/sshd_config.d/pushleft-git.conf has, for Match User git, AuthorizedKeysFile none and AuthorizedKeysCommand /usr/local/bin/pushleft-git-shell authorized-keys %f, run as the git user. The shell sends the presented key’s SHA-256 fingerprint to the API, which answers with the one key registered under it (git_ssh_key(), migration 0068), or nothing. The shell prints it as command="pushleft-git-shell key-<key id>",restrict <key>. Nothing lists more than that one key, and keys of disabled users are not returned. restrict turns off the pty, port, agent and X11 forwarding and ~/.ssh/rc; the Match block turns them off as well and accepts public keys only.
  • Every request. sshd runs the forced command through the git user’s login shell, which is pushleft-git-shell too. It reads SSH_ORIGINAL_COMMAND and accepts exactly git-upload-pack, git-receive-pack or git-upload-archive followed by one single-quoted <org handle>/<name> path, with or without a leading / and a trailing .git. Handles and names are lowercase letters, digits, - and _, so no path can contain . segments or leave the repository root. It then asks the API whether the key’s user may read (upload-pack, upload-archive) or write (receive-pack) that repository now: viewers read, members and above write, a suspended organization reads only, anyone else gets the same “not found” as for a missing repository. Any other answer, or none, refuses. Only then does it exec git -c core.hooksPath=/usr/local/lib/pushleft/git-hooks <command> <data>/repos/<handle>/<name>.git. An interactive login, ssh git@host ls, a chained command and an unknown path are refused with a one-line message.
  • No ForceCommand. sshd’s ForceCommand overrides the per-key command=, so the key id would be lost. The fallback is the login shell instead: a session for git that somehow lacks the forced command still starts pushleft-git-shell, which refuses everything that is not -c 'pushleft-git-shell key-<id>'.
  • Repositories. Bare repositories under /var/lib/pushleft-git/repos/<handle>/<name>.git, owned by git (mode 0700). A repository exists in Pushleft first (pl git create); its directory is created on the first permitted clone or push. The hooks are forced with core.hooksPath and are root-owned, so a repository’s own hooks/ directory never runs and git cannot change them.
  • Pushleft decides every push twice. The shell checks write access before receive-pack starts, and the pre-receive hook asks again with the received size, so a role change or a suspension during a long push still refuses it. No answer refuses the push.
  • The production branch is protected. The pre-receive hook sends the pushed ref updates. Pushleft refuses the whole push if a member creates, updates or deletes the production branch: “only admins and owners push main; push another branch for a preview”. The push endpoint checks the role again before it deploys production. A webhook deploys what the provider already accepted, so the provider’s own branch protection applies there.

Deleting a branch tears down its preview. post-receive posts each deleted branch to /internal/git/delete. Pushleft queues the teardown of every running deployment of that preview/<branch> environment whose source.git_url is this repository, and prints “preview/: N deployment(s) tearing down”. A provider webhook for a deleted branch does the same. Deleting the production branch never tears production down; use pl down. Matching needs PUSHLEFT_GIT_PUBLIC_URL; without it, Pushleft says so and leaves the preview.

Hook listener. The shell and the hooks call a separate API listener (PUSHLEFT_GIT_HOOK_BIND, /internal/git/{authorized-key,access,pre-receive,push,delete}) that accepts loopback and private peers only and a shared secret (PUSHLEFT_GIT_HOOK_SECRET_FILE, compared in constant time). The public proxy never routes to it. The secret file is 0640 root:git; the API’s user is in the git group. The hooks write the secret into a temporary header file with a shell builtin, so it never appears in a process list; the shell sends it over its own socket.

  • The shell. crates/pushleft-api/src/bin/pushleft-git-shell.rs, standard library only, so it starts quickly once per connection and links nothing it does not use. Defaults: hook listener http://127.0.0.1:8091, secret /etc/pushleft/git-hook.secret, data /var/lib/pushleft-git; /etc/pushleft/git-shell.env may override PUSHLEFT_GIT_HOOK_URL, PUSHLEFT_GIT_HOOK_SECRET_FILE and PUSHLEFT_GIT_DATA. It passes PUSHLEFT_GIT_REPOSITORY and PUSHLEFT_GIT_USER to the hooks.
  • One pipeline. Both the hook and the webhook hand a gzip tarball to pushleft_source::pack_tar. It writes regular files and directories into a private temporary directory (links and special files are refused as in pl up, excluded paths skipped, limits enforced while unpacking) and calls the same pack as the CLI. The API then creates the deployment (POST /v1/deployments logic) and ingests the snapshot (PUT /v1/deployments/{id}/source logic), so admission, quotas, budgets, suspension and the source claim are unchanged. The idempotency key is git: + SHA-256 of (repository or webhook id, ref, commit).
  • Who deployed. A push records the pusher as the actor. A webhook deployment has no user and acts with the member role. source.git_url is the Pushleft clone URL or the provider repository.

Soft Serve comparison

The first version (2026-10-20) ran Soft Serve v0.12.3 on port 22, with a reconciler in the API that mirrored users, keys, repositories and collaborators into it over its admin SSH. With the host’s OpenSSH in front, Soft Serve would only have stored repositories and run hooks, which bare repositories and core.hooksPath do. It was dropped with its reconciler, its admin key, its SQLite state and its service. What it gave up: nothing that was used. Access is now decided by the API per request instead of by a mirror that could lag by a reconciler pass. The hooks, the push pipeline, the webhooks and the backups are unchanged.

Operator access

Port 22 is the host’s sshd for everyone. Users other than git log in with keys only and only from the operator addresses: a Match User *,!git Address *,!<operator CIDRs> block turns off every authentication method for anyone else, which works on OpenSSH 9.x (Ubuntu 24.04) as well as 10.x. Passwords are off for everyone; root logs in by key only, and like every operator only from an operator address. The Hetzner firewall cannot express this, because port 22 must stay open to everyone for git; sshd enforces it. The operator addresses are required (PUSHLEFT_GIT_OPERATORS, no default): each operator device’s tailnet address (tailscale ip). Never the whole tailnet: it also holds every node, and nodes run customer code. The install script refuses an IPv4 range that overlaps 100.64.0.0/10 and any IPv6 range (decided 2026-10-20). Add a device by running the install script again with the new list. Operators keep their own ~/.ssh/authorized_keys; AuthorizedKeysFile none applies to git only.

Install

On the control-plane host, as root, from a checkout:

cargo build --release -p pushleft-api --bin pushleft-git-shell
PUSHLEFT_GIT_OPERATORS=100.64.0.7,fd7a:115c:a1e0::7 deploy/git/install-git-host.sh target/release/pushleft-git-shell   # your devices' tailnet addresses
sshd -t && systemctl reload ssh
install -m 0644 deploy/systemd/pushleft-git-{firewall,backup}.service deploy/systemd/pushleft-git-backup.timer /etc/systemd/system/
systemctl daemon-reload && systemctl enable --now pushleft-git-firewall pushleft-git-backup.timer

install-git-host.sh is idempotent. It creates the system user git (home /var/lib/pushleft-git, login shell pushleft-git-shell, password * so sshd does not treat it as locked), installs the shell and the hooks root-owned, creates the hook secret if absent (0640 root:git) and adds the API’s user pushleft to the git group, and writes the sshd drop-in and /etc/pushleft/git/git-nftables.nft with the operator addresses. Keep a second operator session open while reloading sshd.

Abuse and limits

LimitnewstandardtrustedEnforced by
git_max_repositories350noneAPI, 429 git_repository_quota
git_max_repository_mb (repository plus the push being received)2001 0245 120pre-receive

pushleft-git-firewall.service loads after local filesystems and nftables.service, before both ssh.service and ssh.socket. It uses DefaultDependencies=no: the usual service dependency on basic.target creates a cycle with early socket startup, which can cause systemd to drop the SSH socket job at boot. Explicit shutdown ordering keeps the firewall stop operation ahead of shutdown.

The SSH port is public port 22 on the control-plane host, the host’s own sshd. deploy/git/git-nftables.nft (loaded by pushleft-git-firewall.service) limits each source address to 30 new connections per minute, with a burst of 30. A source over the limit is dropped for 15 minutes. No source holds more than 10 connections at once. Loopback and the operator addresses are exempt, so an attack on the git port cannot lock operators out. The rate limit and ban were checked in a container with the Soft Serve version of this file (the 31st connection in a minute was dropped and the address was in the ban set); the operator exemption is not yet proven on a host. sshd itself drops unauthenticated connections from 50 on (MaxStartups 50:30:200) and gives a login 30 seconds; OpenSSH 9.8 and later also penalise sources that fail authentication repeatedly (PerSourcePenalties, on by default); where sshd supports it, the install script exempts the operator addresses (PerSourcePenaltyExemptList). test:git hit this: without the exemption, its refusal checks got the test client’s address penalised and later logins closed. Each unknown key a client offers costs one API lookup.

Backups

pushleft-git-backup.timer runs pushleft-api git backup every hour as the git user. For each repository whose refs changed since the last backup, it writes git bundle create --all and seals it with the organization’s backup key (vault://organizations/<org>/backup-key#key, created if absent). The seal is the agents’ PLSEAL1 format, now in the pushleft-seal crate. The sealed bundle goes to the blob store’s backups use as git/<org>/<repository id>/<UTC time>.bundle.sealed. The newest PUSHLEFT_GIT_BACKUP_KEEP (default 14) are kept. An empty repository, or one never cloned or pushed (no directory yet), is skipped; a backed-up repository whose directory disappeared fails the run. A failed repository makes the run fail, after the others were tried.

Restore

# On the git host, as the git user with the backup unit's environment:
sudo systemd-run --wait --pipe -p User=git -p EnvironmentFile=/etc/pushleft/git-backup.env \
  /usr/local/bin/pushleft-api git restore <handle>/<name> --into /var/lib/pushleft-git/repos/<handle>/<name>.git
# A specific object instead of the newest: add --object git/<org>/<repository id>/<time>.bundle.sealed

The restore downloads the object, opens it with the organization key (a changed or truncated object fails authentication), runs git bundle verify, and fetches every ref into the target (+refs/*:refs/*, created bare if absent). It prints the restored refs.

  • One repository. Restore into its directory (created bare if absent).
  • The whole host. Run the install above, then restore each repository listed by git_backup_index(). Keys and access need no restore: they live in Pushleft.

Pushes made after the restored bundle are lost; the pushers still have them and can push again.

Configuration

SettingWhereMeaning
PUSHLEFT_GIT_HOOK_BIND, PUSHLEFT_GIT_HOOK_SECRET_FILEAPIHook listener (loopback or private address only) and its secret, at least 32 characters
PUSHLEFT_GIT_PUBLIC_URLAPIClone URL base with the git user, e.g. ssh://[email protected]
PUSHLEFT_GIT_PROVIDER_ALLOW_HTTP, PUSHLEFT_GIT_PROVIDER_ALLOW_PRIVATEAPITests only: plain HTTP and private provider hosts
PUSHLEFT_GIT_HOOK_URL, PUSHLEFT_GIT_HOOK_SECRET_FILE, PUSHLEFT_GIT_DATA/etc/pushleft/git-shell.env (optional)Overrides of pushleft-git-shell’s defaults http://127.0.0.1:8091, /etc/pushleft/git-hook.secret, /var/lib/pushleft-git
PUSHLEFT_GIT_OPERATORSinstall-git-host.shOperator CIDRs for sshd and nftables (default the management tailnet)
PUSHLEFT_GIT_DATA, PUSHLEFT_GIT_BACKUP_KEEP, PUSHLEFT_BACKUP_S3_*, Vaultgit-backup.envBackups

Units, env examples and Vault policies: deploy/systemd/pushleft-git-{firewall,backup}.service, pushleft-git-backup.timer, git-backup.env.example, deploy/git/ (sshd drop-in, hooks, nftables, install script), deploy/vault/pushleft-git-backup.hcl. The API’s Vault policy gains create, read and delete on organizations/*/git/webhooks/*. See deploy/README.md.

Tests

npm run test:git (in test:all) runs the production-auth API as the NOBYPASSRLS role with the hook listener; an OpenSSH server container (Debian trixie by digest) set up by install-git-host.sh with the shipped drop-in, hooks and pushleft-git-shell built from source; OpenBao with the shipped policies, Garage and a fake GitHub/GitLab archive endpoint. It checks:

  • Keys need a fresh login with a mailed code (local SMTP sink): a bootstrap token, and the fresh pl login of a token-issued or admin-issued code, are refused. Keys are listed with ssh-keygen’s fingerprint, belong to one user, and private keys are refused.
  • Viewers cannot create repositories, and the new tier stops at 3.
  • sshd’s AuthorizedKeysCommand prints exactly one command="pushleft-git-shell key-<id>",restrict line for a registered key, nothing for an unregistered one, refuses a non-fingerprint argument, and the key lookup refuses a caller without the secret.
  • A member’s push of main is refused; the owner’s creates a production deployment and a member’s branch push a preview, and deleting that branch queues the preview’s teardown and leaves production. Both snapshot digests equal pl plan --source of git archive of the pushed commit, and .plignore applies.
  • A replayed hook call returns the same deployment. The hook listener refuses a missing or wrong secret.
  • These are refused: an unregistered key, a viewer’s push, another organization’s clone (both directions), a member demoted to viewer (at once), a shell with and without a pty, ssh git@host ls /, a chained command, four path-traversal forms (.. inside and before the handle, an absolute path into the data directory) both raw and through git clone, and any clone while the shell cannot reach the API.
  • An operator logs in with a shell from an operator address, and is refused from any other address; no git user key logs in as root.
  • Signed GitHub and GitLab deliveries deploy through the fake provider with the right environment and digest. A bad signature and a wrong token give 401, a replayed delivery 409, a deleted webhook 404. The API never returns the deploy token.
  • A backup writes a PLSEAL1 object to Garage (no plaintext commit ids), skips unchanged repositories and empty ones, and a restore reproduces every ref.

Unit tests (cargo test -p pushleft-api git::, --bin pushleft-git-shell, -p pushleft-source foreign) cover signature checks (GitHub’s documented example), ref-to-environment mapping, idempotency keys, key parsing and fingerprints, the hook peer check, the shell’s command parser (traversal, quoting, chained commands), its authorized_keys line and foreign tarball packing.

Gaps

  • No repository deletion or rename (pl git has neither); repositories are kept.
  • The nftables limits were checked in a container, not on the control-plane host; the operator exemption and the sshd operator Match block are not yet run on a VM with real addresses (test:git proves the Match block in a container).
  • The pre-receive size check uses du of the repository, including the quarantined push; a push larger than the cap is received before it is refused.
  • Webhook delivery ids are kept forever (no purge yet).
  • Each SSH connection costs the API two lookups (key, access) and a push two more (hooks). Not load-tested.
  • Only one control-plane host serves git; on failover, restore from the backups (above).
  • No ssh git@host greeting or pl-style diagnostics over SSH: every non-git command is refused with one line.
Rendered from the repository at build time. Edit docs/git.md to change it.