pushleft
Deploy your first app
Manual contents
Handbook
HandbookPushleftArchitecture
Operate
Install the control planeYour own serversFleetOperations and alertsAuthenticationDomains and DNSUsage and budgetsScheduled jobsAbuse controlsAI provider proxyCustomer mailErrors and performanceLogs and metricsRetentionObject storageContainer registryGit server
Develop
The .pl.yaml manifestOnboard with an AI agentThe pl up demoBuild executorSource ingestionAgent gatewayManaged databasesSecurity findingsTunnels and shellsDev workspacesPrivate hostnamesMesh coordinatorsDesign baseline
Runbooks
Control-plane host failoverControl-plane PostgreSQL failoverControl-plane PostgreSQL restoreCustom domains and certificatesGateway key compromiseIngress host failureNode busyNode disk fullNode drain and replacementNode lostRegistry outageHetzner Robot hostsSecret rotationSecurity incidentStaging fleet on ProxmoxStaging build cacheStuck deploymentVault/OpenBao backup and recovery
Manual · docs/developer-access.md

Tunnels and shells

Three commands and a console terminal give your machine access to one environment’s private services. Your machine does not join the environment network, and no container runs an SSH server.

Console A shell in a running web instance, from the Services page. pl exec opens the same session.
  • pl db connect forwards a managed database to localhost.
  • pl tunnel forwards one port of a running service to localhost.
  • pl exec runs a command, or opens a shell, in a running service.
  • pl cp copies a file or directory to or from a running service.
  • The Shell button on the console’s Services page opens a terminal in a running service.

You need the Admin or Owner role in the organization. A Member runs pl exec, pl cp, the console’s shell, pl tunnel and pl db connect in preview environments (Who). Built 2026-11-04; the console’s shell and pl exec without a command added 2026-11-06; pl cp, the shell’s instance choice, the sessions list and the database Connect panel added 2026-11-06; exec for Members in preview environments decided and built 2026-11-13; tunnels and database connections for Members in preview environments on 2026-11-14.

Connect to a database

  1. Find the database ID:

    pl db list
  2. Forward it. pl prints the connection string on standard output, and the rest on standard error:

    $ pl db connect 0193f2c4-8a1e-7c3b-9d40-5e6f7a8b9c0d
    postgres://[email protected]:54012/shop
    The password is not shown: the API never reads database credentials back. Use the one in the role's DATABASE_URL (or PG*/CLICKHOUSE_URL/REDIS_URL) variables.
    Forwarding until you press Ctrl-C. Session 0193f2c5-…; a new one takes over before its four-hour limit.
  3. In a second terminal, connect with the printed address and the password from the role’s DATABASE_URL:

    psql "postgres://[email protected]:54012/shop"
  4. Press Ctrl-C in the first terminal when you are done. pl prints Session ended: closed.

Use --port <local port> for a fixed local port, for example to keep a saved connection in a database GUI. A port that is in use fails before a session opens.

EnginePort forwardedPrinted address
PostgreSQL5432postgres://<login>@127.0.0.1:<port>/<database>
ClickHouse8123 (HTTP)http://<login>@127.0.0.1:<port>/?database=<database>
Valkey6379redis://<login>@127.0.0.1:<port>/0

The address has the manifest login and its database, but no password: the API never reads database credentials back from Vault. Take the password from the role’s connection variables (DATABASE_URL, PG*, CLICKHOUSE_URL or REDIS_URL). When an instance has several logical databases, the address names the first. The tunnel reaches all of them: change the name in the address. Without a ready login with migrate, the address has no <login>@. The database must be ready or degraded in pl db list; otherwise the API answers 409.

In the console. A browser cannot forward a port. So a database’s page on the Data page has a Connect panel instead. It shows the pl db connect <id> command with a copy button, and the database and login the address will name. It never shows a password. It names the secret that holds it: the role’s DATABASE_URL (or PG*), CLICKHOUSE_URL or REDIS_URL.

Open a shell

From a checkout, open a shell in the web role of the checkout’s preview environment:

$ pl exec web
/app #

Without a command, pl exec runs sh. Add --production for production, or --project and --environment for another environment. Exit the shell to end the session.

Run one command and use its output and exit code in a script. Put the command after --:

$ pl exec web --production -- ls /app
package.json
server.js
$ echo $?
0

pl exec passes standard input, output and error, and exits with the command’s exit code. When standard input and output are a terminal, the command gets a terminal of the same size, and a resize follows. Without a terminal, you can pipe into it: pl exec web -- sh -c 'cat > /tmp/seed.sql' < seed.sql. When the session ends before the command exits, pl prints Session ended: <reason>. and exits with 255.

In the console. On the Services page, an Admin or Owner sees a Shell button on each web, private or worker service. A Member sees it in a preview environment. The service needs at least one ready instance. When it has several ready instances, a list next to the button picks one, by ordinal and short ID (web.1 · 0193bbbb). With one, there is no list. It opens a terminal below the services: bash -l where the image has bash, else sh -l, with TERM=xterm-256color. The terminal is ghostty-web, Ghostty’s terminal emulator compiled to WebAssembly, loaded only when a shell opens. The console and pl exec open the same exec session, with the same checks and record. When the shell exits, the panel shows the exit code. Close ends the session.

Changes you make in a shell stay in that one instance until its next deployment. Put lasting changes in the image or the manifest.

Choose an instance

A service with replicas has several instances. pl exec, pl cp and pl tunnel take --instance <id>. The IDs are under Technical details on the service’s card, and in the console’s shell list. Without --instance, the command picks the first ready instance of the newest live release.

Copy files

Copy a local directory into the web role, and a log file back:

$ pl cp ./fixtures web:/app/tmp/fixtures
$ pl cp web:/var/log/app.log ./

Name one side role:/absolute/path. The other side is a local path. Write ./a:b for a local name with a colon. When the destination is a directory that exists, the copy goes into it under its own name. Otherwise it gets the destination’s name, as with cp -r.

pl cp sends a tar archive over an exec session. It runs sh and tar in the instance, so the image needs both. Without tar, it stops with pl cp: the image has no tar. It takes the same options as pl exec: --instance, --project, --environment, --production and --path. It exits with the remote command’s exit code. The archive streams through the session, so a copy of any size holds only a small buffer in memory. A failed download may leave part of the copy, and pl cp says so.

Forward a port

Forward port 9090 of the web role, for example an admin page that is not public:

$ pl tunnel web 9090
127.0.0.1:54013
Forwarding until you press Ctrl-C. Session 0193f2c6-…; a new one takes over before its four-hour limit.

Then open http://127.0.0.1:54013 in a browser. A private role works the same way: pl tunnel search 7700. Use --local-port <port> for a fixed local port.

Options and limits

  • pl tunnel and pl exec take the environment like pl logs: the checkout’s project and branch by default, or --project, --environment, --production and --path. pl db connect takes only the database ID.
  • Without --instance <id>, pl tunnel, pl exec and pl cp pick a ready instance of the newest live release. The console’s shell does the same unless you pick an instance (Choose an instance).
  • The role must be long-running (web, private or worker). exec into a database is refused.
  • A tunnel carries up to 32 connections at once.
  • A session lasts up to four hours. It also ends after 30 minutes without traffic. When a session ends, pl prints why (When a session ends).
  • pl tunnel and pl db connect keep the local port until you press Ctrl-C, across sessions. Ten minutes before a session’s four-hour limit, pl opens a new session for the new connections. The connections that are open finish on the old session until its limit. When a session ends early, for example when the node’s agent updates or after 30 idle minutes, the next connection opens a new session. Only the connections that were open on the ended session are lost. A revoked session, or a role that no longer allows it, ends the tunnel.
  • pl exec and pl cp end with their session. Run the command again.

Session access

The Admin or Owner role in the organization, with a user credential: a console or pl login session, or a user’s API token. Viewers, read-only tokens and service tokens are refused.

A Member opens sessions in preview environments only: pl exec, pl cp, the console’s shell, pl tunnel and pl db connect. A Member gets 403 for each of them in production. A shell in an instance can read that environment’s variables and secrets, and a tunnel or database connection reaches what the app reaches. A Member can already deploy code to a preview environment, so these sessions give no new access there.

SessionAdmin, OwnerMember
exec, tunnel, database in a preview environmentYesYes
exec, tunnel, database in productionYesNo

An open session ends when its creator’s role no longer allows it: an Admin loses the role, a Member leaves the organization, or the environment is no longer a preview.

Recorded

Every session is a developer_sessions row (migration 0130). The row records who opened it, the environment and the target. A target is a database, a role, instance and port, or a role, instance and command. The row also records the node, when it was created, when pl connected, and when and why it ended. Opening and revoking are audit events (developer_session_opened, developer_session_revoked).

See and revoke sessions

Settings in the console has a Developer sessions panel for Admins and Owners. It lists the open sessions first, then the last 50 that ended. Each row shows the kind, the target, the project and environment, who opened it, and when it opened, connected and ended. An ended row shows why. An open row has a Revoke button. The relay then closes both sides within five seconds.

API

POST /v1/sessions with {"kind":"database","database":<id>}, {"kind":"tunnel","project","environment","role","port","instance"?} or {"kind":"exec","project","environment","role","instance"?,"command":[…],"tty"} answers 201 with the session, its one-time token, the relay URL and the connection details. GET /v1/sessions lists the organization’s open sessions, then the last 50 ended, with the project and environment slugs and the creator’s email. GET /v1/sessions/{id} shows a session and why it ended. The list needs the Admin role, because a target can name a command line. A Member reads only the sessions they opened. DELETE /v1/sessions/{id} revokes it: the relay closes both sides within five seconds, and a job that no node has leased yet fails.

Internals

Agents get work only by polling (POST /v1/agent/lease, agent gateway), so there is no channel the control plane can push a byte stream over. All three commands use one relay:

  1. pl asks the API for a session. The API checks the role, finds the target and the node that runs it, records the session and queues one open_session job for that node. The job carries the one target the session may reach. It runs once (max_attempts 1), so a session is never retried or renewed. The API answers 409 when that node’s agent has no session support yet (it does not report session_lane).
  2. pl opens the relay with the session’s one-time token.
  3. The node’s agent leases the job in a lane of its own (session). An open tunnel or shell never holds up deployments, database work or name updates in the other execution lanes. The lane runs up to eight sessions at once per node. The agent checks the target and opens its own side of the relay with its node certificate and the lease. It renews the lease while the session is open. The relay pairs the two sides and copies bytes both ways.
  4. On the node, the agent connects to the target. For each connection pl opens, it makes a TCP connection to the container’s address on the environment bridge at the session’s port. For exec, it runs docker exec in the container.

pl and the agent exchange small frames over the relay: a kind, a channel and a length. They carry each tunnel connection on its own channel, and for exec the standard input, output and error, the terminal size and the exit code. The relay never reads them.

Relay placement

Decided 2026-11-04 while building it: in the agent gateway process, not in the API and not as a new service.

  • The two sides must meet in one process to be paired in memory. The agent side already has its trust model in the gateway. That model is mutual TLS with the node certificate, checked against the database on each request, and the job lease. The API authenticates people and tokens, never nodes.
  • pl cannot reach the gateway’s agent listener: it is on the private network only and requires a client certificate. The gateway gets a third listener for the pl side, plain HTTP on loopback (PUSHLEFT_SESSION_RELAY_BIND, 127.0.0.1:8449 in the examples). Only the control-plane TLS proxy reaches it, as https://api.<base>/v1/sessions/{id}/relay. The registry token realm uses the same pattern: the proxy also sends it to the gateway. The gateway refuses a non-loopback address for it.
  • The pl side’s only credential is the session’s token: 256 random bits, issued by the API in the POST answer. The API stores only its SHA-256. The token works once, within two minutes of issue, and only while the session has not ended. Any other token, a second use, an expired or a revoked session is 401.
  • The agent side is GET /v1/agent/jobs/{id}/relay on the mTLS listener with the header x-pushleft-lease-token. It needs the current lease of that node’s open_session job for a session that has not ended; otherwise 409.
  • Both sides are HTTP/1.1 upgrades (Upgrade: pushleft-session). The relay is then a byte copy, and no new library is needed: axum, hyper and reqwest already do upgrades. The TLS proxy passes an upgrade through as it does a WebSocket.
  • A browser cannot send that upgrade, so the pl side also takes a WebSocket (decided 2026-11-06, for the console’s shell). A browser cannot set Authorization on a WebSocket, and a token in the URL would reach access logs. The token is therefore the second subprotocol: Sec-WebSocket-Protocol: pushleft-session, pls_…. The relay answers with pushleft-session. The frames travel in binary messages, in any chunking, and the relay turns them back into the byte stream the agent reads. The console’s host routes /v1/sessions/*/relay to the relay, as the API’s host does.
  • The first side waits up to 60 seconds for the other. A session whose agent does not join in that time ends as not_connected.

Reachable addresses

The target is fixed when the API creates the session, and checked three times. Nothing in the byte stream names an address, a port or a container.

  1. API. The database, role or instance must belong to the caller’s organization; otherwise 404. A role must be long-running (web, private or worker) with a ready instance in a live release. A database must be ready or degraded. The task names one container (pushleft-db-<runtime name>, or pl-app-<instance>-g<generation>), the environment’s bridge (pl_<segment>) and, for a tunnel, one port. exec is refused for databases.
  2. Gateway, at lease. The session must be live and issued less than two minutes ago. The bridge must be the environment’s. The target must still be on this node: the database’s active runtime, or the instance at that deployment and generation, ready. Otherwise the job fails and the session ends as not_connected. A customer-owned node leases only its own organization’s sessions, as for every job.
  3. Agent. The container must be running and carry the Pushleft labels of the target. For applications, the labels are managed, the organization, the environment, deployment, instance and generation. For databases, they are managed, the organization, the database runtime and bridge. Its address is the one Docker gives it on the task’s bridge, and it must be inside the environment networks (10.128.0.0/9). A tunnel dials only that address at the task’s port. The host reaches it the way the node’s ingress reaches its upstreams. exec runs through Docker’s API in that container only.

Session lifetime

ReasonWhen
client_closedpl closed its side (Ctrl-C, or the command ended).
agent_closedThe agent closed its side, for example after the command exited.
idle30 minutes without traffic in either direction.
lease_lostThe agent’s lease of the job is no longer current.
revokedAn admin revoked the session.
max_durationFour hours after the session was created: the lease protocol’s cap for one attempt. A new session is a new job, checked and recorded again.
not_connectedThe agent did not join within 60 seconds, or the lease was refused (see above).
access_removedThe creator’s role no longer allows the session (Who).

The relay checks the conditions every five seconds and records the reason before it closes either side.

A relay that stops before it records an end leaves the session open, for example when the agent gateway restarts. The scheduler closes such a session within a minute or two:

ReasonWhen the scheduler records it
not_connectedNobody joined within five minutes.
lease_lostThe agent’s lease of the job is gone.
max_durationFour hours and five minutes after the session was created.

The tailnet alternative

The Tailscale client would have to trust the coordinator’s internal certificate authority. The environment network admits every member to every service. A coordinator that moves starts with fresh state, so every joined machine would have to enroll again. pl is Rust, and there is no embedded Tailscale client for it. A long-lived device that joins with per-user access rules stays a later option.

Limits

  • Traffic goes through the control plane, so the tunnels suit administration and debugging. Use pl db backups and imports for bulk transfers.
  • One gateway process. Both sides must reach the same gateway process. With more than one gateway behind the proxy, the relay needs sticky routing by session or pairing through the database (gap).
  • A gateway restart ends every open session (its 30-second drain).
  • A tunnel’s address is taken when the session starts. If the instance is replaced meanwhile, new connections fail; open a new session.
  • The data path through a tunnel is tested on Linux only; Docker Desktop does not route the host to container addresses.

Rollout (operators)

Deploy in this order:

  1. Migration 0130 (the API runs it at start). Then run the GRANT ... ON ALL TABLES statements again (database roles), so the API and the gateway can use developer_sessions. From migration 0142, the API also needs GRANT EXECUTE ON FUNCTION node_runs_sessions(uuid) TO pushleft_api.
  2. Gateway. Set PUSHLEFT_SESSION_RELAY_BIND=127.0.0.1:8449 and add the proxy route for /v1/sessions/*/relay on the API host (deploy/install-control-plane.sh does both). The gateway must know the session lane before an agent asks for it. An older gateway answers 400 to that lane, and the agent logs the error every five seconds.
  3. API. Until agents report session_lane, it answers 409 for every session.
  4. Agents. The general lane then reports session_lane. An agent from before the lane asks for general and never sees a session job. Only agents from before the durability lane (all or workload, before migration 0095) would lease one and fail it as an unsupported kind. The API never targets them, because they do not report session_lane.
  5. CLI. A pl with db connect, tunnel and exec. pl cp needs only a new pl.
  6. Console shell. The gateway with WebSocket support, the console’s Caddy route for /v1/sessions/*/relay (deploy/install-control-plane.sh), then the console build.
  7. Sessions list. The API with GET /v1/sessions, then the console build. No migration.

Tests

  • Unit: the task’s structure and refusals (pushleft-core session::tests: no exec into a database, no other bridge, unknown fields refused), the frames, and a tunnel carrying several connections to its one target.
  • Unit: the agent’s target check (pushleft-agent session::tests: another organization’s, environment’s or generation’s container, a stopped one, and addresses outside the environment networks are refused).
  • Against migrated PostgreSQL, the API (sessions_tests: viewers, read-only and service tokens refused; a Member opens exec, a tunnel and a database in a preview environment and gets 403 for each in production, and reads only the sessions they opened; another organization’s database, role or instance 404; a node without the lane 409; the session, job, lane and audit; revoke; the list, open first, and only for admins).
  • Against migrated PostgreSQL, the gateway (sessions_tests: the job leases only in the session lane; the relay refuses another node, a wrong lease and a revoked session; a stale session or a moved target fails at lease; a customer-owned node of another organization never sees it; the token works once, only within two minutes, never for another session or a revoked one; a session ends with access_removed when its creator’s role no longer allows it).
  • End to end: scripts/developer-access-e2e.sh runs the API, the gateway with its relay listener, the real agent and pl against a labelled container on an environment bridge. It checks pl exec output, errors, exit code, standard input and a terminal; pl cp of a directory and a file both ways, and a missing path failing; the console’s shell code (src/shell.js, through scripts/console-shell-check.mjs) joining as a WebSocket, with standard input, output and the exit code; a spent token refused; pl tunnel, with bytes through it on Linux; a revoke ending a running tunnel, reported by pl. Passed on macOS (Docker Desktop) on 2026-11-04, without the tunnel’s data path. With pl cp, passed on macOS on 2026-11-06.
Rendered from the repository at build time. Edit docs/developer-access.md to change it.