pushleft
Deploy your first app
Manual contents
Handbook
HandbookPushleftArchitecture
Operate
Install the control planeYour own serversFleetOperations and alertsAuthenticationDomains and DNSUsage and budgetsScheduled jobsAbuse controlsAI provider proxyCustomer mailErrors and performanceLogs and metricsRetentionObject storageContainer registryGit server
Develop
The .pl.yaml manifestOnboard with an AI agentThe pl up demoBuild executorSource ingestionAgent gatewayManaged databasesSecurity findingsTunnels and shellsDev workspacesPrivate hostnamesMesh coordinatorsDesign baseline
Runbooks
Control-plane host failoverControl-plane PostgreSQL failoverControl-plane PostgreSQL restoreCustom domains and certificatesGateway key compromiseIngress host failureNode busyNode disk fullNode drain and replacementNode lostRegistry outageHetzner Robot hostsSecret rotationSecurity incidentStaging fleet on ProxmoxStaging build cacheStuck deploymentVault/OpenBao backup and recovery
Manual · docs/logs.md

Logs and metrics

Pushleft collects what your web, private and worker instances write to stdout and stderr. It also records their CPU, memory and network use. You see the output on the console Logs page and three charts per instance on the Metrics page.

Console Service output from every instance, filtered by service or by text.
Console CPU, memory and network per instance, and latency per endpoint.

Quick start

  1. Write your application’s logs to stdout or stderr. You do not install an agent or a library. A logger that writes one JSON object per line also works.

  2. Deploy the application:

    pl up --production
  3. In the console, open Logs. New lines arrive about every 15 seconds. Open Metrics for the CPU, memory and network charts of each instance.

  4. Or read the output in a terminal. In the checkout, pl logs prints the newest lines of the preview environment of your branch. Add --production for production, --role web for one role, and --follow (-f) to keep printing new lines until you stop it:

    pl logs --production --role web -f

    Outside a checkout, give --project (slug or id) and --environment.

    To read an earlier time window, give --since and --until in UTC (RFC 3339). pl logs then prints every line written in that window, oldest first:

    pl logs --production --role api --since 2026-11-16T06:30:00Z --until 2026-11-16T06:41:00Z

    The window can go back as far as the environment keeps logs.

pl logs <deployment> shows the events of one deployment instead: build, release, rollout, and after it is live, route moves, cron runs and recoveries. It keeps reading until the deployment is torn down or fails, or until you stop it. --after <sequence> skips the events up to that number.

Features

  • The output of every web, private and worker instance, with the time, the stream (stdout or stderr) and the instance.
  • CPU, memory and network charts per instance. The API returns up to the last 168 hours.
  • A mail to the organization’s owners when an instance restarts or runs out of memory (Crashes), or when a service returns many server errors (Server errors).
  • A daily output limit and a retention period of 7, 14 or 30 days, by trust tier (Limits, retention and billing).
  • Lines up to 64 KiB. A longer JSON line stays valid JSON.

Release, job and cron runs are not included. Their last lines stay with the deployment’s events. For application errors with stack traces, use error tracking.

Requests that Cloudflare answers itself (blocks, rate limits, an unreachable tunnel) never reach the app, so they are not in the service output. For domains linked to a Cloudflare connection, Logs → Edge shows them (edge traffic).

Collected data

About every 15 seconds, the agent on each node reads two things for each application container it runs.

Output lines. The lines the container wrote since the previous report, with their time and stream (stdout or stderr):

  • The agent removes control characters other than tab.
  • It keeps a line up to 64 KiB.
  • A longer JSON object stays valid JSON. An example is a structured log line with a stack trace, from any logger that writes one JSON object per line. The agent shortens its longest string fields until it fits, and adds "@pl_truncated": true.
  • The agent cuts other text, and any line past 1 MiB, at 64 KiB.

Metrics. CPU in millicores, memory in use and the memory limit, and the bytes received and sent since the container started.

The agent sends them to the agent gateway with its node certificate. The gateway keeps only instances that the control plane placed on that node. It takes the organization, environment, deployment and service from that placement, never from the report. A node therefore cannot write into another organization’s logs.

Report limits. A report carries at most 2 MB of lines, counted as encoded JSON, and 5,000 lines per instance. If a service writes faster, the agent reads the rest on the next report.

Retries.

  • The agent sends a report again, byte for byte, when the gateway cannot take it yet. The causes are a network error, a 5xx, or 401, 403, 408 or 429. Up to 16 MB of reports can wait.
  • The agent drops a report that the gateway rejects as malformed.
  • The agent skips a container whose logs it cannot read, and reads it on a later round.
  • The gateway keys admission, the charge and its ClickHouse inserts on the SHA-256 of the body. A resent report is stored and charged once. A different body under the same id counts as a new report.
  • The agent tracks lines that share one timestamp by count, so it never skips or sends one twice.

Output written while the agent is down is not reported.

Crashes

The agent reports two kinds of crash for each application instance:

  • Restart. The process exited and Docker started the container again. A stop by Pushleft, for a deploy or a scale-down, is not a crash.
  • Out of memory. The kernel stopped a process at the role’s memory limit (resources.memory). The container can survive this when the process was not its main one.

Each crash adds an instance crashed event to the deployment, with the role, the exit code and the restart count. Pushleft also mails the owners of the organization, at most once an hour per environment and role. The mail links to the role’s metrics, and it counts the crashes in the hour before it.

Pushleft does not report a crash while the agent is down, or a restart before the agent first sees the container. A failed start during a deploy shows in the deployment’s events, as before.

Server errors

Every 5 minutes, Pushleft counts each service’s requests and 5xx answers over the last 10 minutes. It counts from the ingress access log, the same data as the Traffic page. A service alerts when both of these are true:

  • It returned at least 20 server errors.
  • Server errors were at least 10% of its requests.

Each alert adds an error rate high event to the deployment, with the counts and the most common status. Pushleft also mails the owners of the organization, at most once an hour per environment and role. A 502, 503 or 504 usually means that the ingress found no ready instance. Another 5xx comes from the service itself, so check its logs and errors.

Requests that Cloudflare answers itself never reach the ingress, so they do not count here. Logs → Edge shows them (edge traffic). With PUSHLEFT_WEB_ANALYTICS=0 on a node, its services send no access log and get no alert.

Limits, retention and billing

TierOutput per dayKept for
new100 MB7 days
standard2 GB14 days
trusted20 GB30 days

The daily limit counts bytes of output per organization per UTC day. When a report would pass it, Pushleft drops that report’s lines and all later ones until the day ends. Pushleft still accepts metrics. Pushleft also drops output while the organization is suspended or its budget is spent.

Stored output is metered as runtime_log_gb on the usage page, in GB, at the price the operator sets (billing). Metrics are not metered.

Pushleft keeps samples and lines for the tier’s retention. Each row gets its retention when it arrives, so a tier change applies to new rows only.

API

Both endpoints need the viewer role. Both take the project (slug or id) and the environment slug.

GET /v1/logs?project=&environment=[&role=][&since=][&until=][&after=] returns the newest 500 lines stored in the last day, in the order Pushleft stored them.

  • since and until (RFC 3339) keep the lines written in that window. until needs since. With since, the first page holds the oldest 500 lines from that time instead of the newest, and the window can reach back past the last day.

  • Each line has time, role, instance_id, stream, line and cursor. The page has the cursor of its last line.

  • Set after to a page’s cursor to get the next 500 lines stored since then. The response has more: true when there may be more lines.

  • Paging follows the storage order, so a report that arrives late still shows.

  • The console polls every 3 seconds, and at once while more is true. pl logs -f polls every 2 seconds.

GET /v1/instances/metrics?project=&environment=[&hours=] returns about 90 points per instance over the window. The default window is 1 hour, and the maximum is 168.

  • Each point has instance_id, role, t (Unix seconds), cpu_millis (average), memory_bytes (peak), memory_limit_bytes, and the rx_bytes and tx_bytes counters.
  • A counter drops when the container restarts.

Operating it

The gateway writes to ClickHouse: pushleft_otel.instance_samples and instance_logs, with the schema in deploy/observability/clickhouse/003_instances.sql.

  • Set PUSHLEFT_INSTANCES_CLICKHOUSE_ENDPOINT, _USERNAME and _PASSWORD on the gateway. Use a ClickHouse user that can only insert into those two tables.
  • Without these settings, the gateway refuses reports with 503, the agents keep retrying, and the console says that storage is not configured.
  • The API reads with the error-tracking ClickHouse user (PUSHLEFT_ERRORS_CLICKHOUSE_*, error tracking). It needs SELECT on both tables.
  • The gateway’s database role needs EXECUTE on instance_telemetry_admit (deploy/README.md).
Rendered from the repository at build time. Edit docs/logs.md to change it.