Logs and metrics
Pushleft collects what your web, private and worker instances write to stdout and stderr. It also records their CPU, memory and network use. You see the output on the console Logs page and three charts per instance on the Metrics page.
Quick start
-
Write your application’s logs to stdout or stderr. You do not install an agent or a library. A logger that writes one JSON object per line also works.
-
Deploy the application:
pl up --production -
In the console, open Logs. New lines arrive about every 15 seconds. Open Metrics for the CPU, memory and network charts of each instance.
-
Or read the output in a terminal. In the checkout,
pl logsprints the newest lines of the preview environment of your branch. Add--productionfor production,--role webfor one role, and--follow(-f) to keep printing new lines until you stop it:pl logs --production --role web -fOutside a checkout, give
--project(slug or id) and--environment.To read an earlier time window, give
--sinceand--untilin UTC (RFC 3339).pl logsthen prints every line written in that window, oldest first:pl logs --production --role api --since 2026-11-16T06:30:00Z --until 2026-11-16T06:41:00ZThe window can go back as far as the environment keeps logs.
pl logs <deployment> shows the events of one deployment instead: build, release, rollout, and after it is live, route moves, cron runs and recoveries. It keeps reading until the deployment is torn down or fails, or until you stop it. --after <sequence> skips the events up to that number.
Features
- The output of every web, private and worker instance, with the time, the stream (stdout or stderr) and the instance.
- CPU, memory and network charts per instance. The API returns up to the last 168 hours.
- A mail to the organization’s owners when an instance restarts or runs out of memory (Crashes), or when a service returns many server errors (Server errors).
- A daily output limit and a retention period of 7, 14 or 30 days, by trust tier (Limits, retention and billing).
- Lines up to 64 KiB. A longer JSON line stays valid JSON.
Release, job and cron runs are not included. Their last lines stay with the deployment’s events. For application errors with stack traces, use error tracking.
Requests that Cloudflare answers itself (blocks, rate limits, an unreachable tunnel) never reach the app, so they are not in the service output. For domains linked to a Cloudflare connection, Logs → Edge shows them (edge traffic).
Collected data
About every 15 seconds, the agent on each node reads two things for each application container it runs.
Output lines. The lines the container wrote since the previous report, with their time and stream (stdout or stderr):
- The agent removes control characters other than tab.
- It keeps a line up to 64 KiB.
- A longer JSON object stays valid JSON. An example is a structured log line with a stack trace, from any logger that writes one JSON object per line. The agent shortens its longest string fields until it fits, and adds
"@pl_truncated": true. - The agent cuts other text, and any line past 1 MiB, at 64 KiB.
Metrics. CPU in millicores, memory in use and the memory limit, and the bytes received and sent since the container started.
The agent sends them to the agent gateway with its node certificate. The gateway keeps only instances that the control plane placed on that node. It takes the organization, environment, deployment and service from that placement, never from the report. A node therefore cannot write into another organization’s logs.
Report limits. A report carries at most 2 MB of lines, counted as encoded JSON, and 5,000 lines per instance. If a service writes faster, the agent reads the rest on the next report.
Retries.
- The agent sends a report again, byte for byte, when the gateway cannot take it yet. The causes are a network error, a 5xx, or 401, 403, 408 or 429. Up to 16 MB of reports can wait.
- The agent drops a report that the gateway rejects as malformed.
- The agent skips a container whose logs it cannot read, and reads it on a later round.
- The gateway keys admission, the charge and its ClickHouse inserts on the SHA-256 of the body. A resent report is stored and charged once. A different body under the same id counts as a new report.
- The agent tracks lines that share one timestamp by count, so it never skips or sends one twice.
Output written while the agent is down is not reported.
Crashes
The agent reports two kinds of crash for each application instance:
- Restart. The process exited and Docker started the container again. A stop by Pushleft, for a deploy or a scale-down, is not a crash.
- Out of memory. The kernel stopped a process at the role’s memory limit (
resources.memory). The container can survive this when the process was not its main one.
Each crash adds an instance crashed event to the deployment, with the role, the exit code and the restart count. Pushleft also mails the owners of the organization, at most once an hour per environment and role. The mail links to the role’s metrics, and it counts the crashes in the hour before it.
Pushleft does not report a crash while the agent is down, or a restart before the agent first sees the container. A failed start during a deploy shows in the deployment’s events, as before.
Server errors
Every 5 minutes, Pushleft counts each service’s requests and 5xx answers over the last 10 minutes. It counts from the ingress access log, the same data as the Traffic page. A service alerts when both of these are true:
- It returned at least 20 server errors.
- Server errors were at least 10% of its requests.
Each alert adds an error rate high event to the deployment, with the counts and the most common status. Pushleft also mails the owners of the organization, at most once an hour per environment and role. A 502, 503 or 504 usually means that the ingress found no ready instance. Another 5xx comes from the service itself, so check its logs and errors.
Requests that Cloudflare answers itself never reach the ingress, so they do not count here. Logs → Edge shows them (edge traffic). With PUSHLEFT_WEB_ANALYTICS=0 on a node, its services send no access log and get no alert.
Limits, retention and billing
| Tier | Output per day | Kept for |
|---|---|---|
| new | 100 MB | 7 days |
| standard | 2 GB | 14 days |
| trusted | 20 GB | 30 days |
The daily limit counts bytes of output per organization per UTC day. When a report would pass it, Pushleft drops that report’s lines and all later ones until the day ends. Pushleft still accepts metrics. Pushleft also drops output while the organization is suspended or its budget is spent.
Stored output is metered as runtime_log_gb on the usage page, in GB, at the price the operator sets (billing). Metrics are not metered.
Pushleft keeps samples and lines for the tier’s retention. Each row gets its retention when it arrives, so a tier change applies to new rows only.
API
Both endpoints need the viewer role. Both take the project (slug or id) and the environment slug.
GET /v1/logs?project=&environment=[&role=][&since=][&until=][&after=] returns the newest 500 lines stored in the last day, in the order Pushleft stored them.
-
sinceanduntil(RFC 3339) keep the lines written in that window.untilneedssince. Withsince, the first page holds the oldest 500 lines from that time instead of the newest, and the window can reach back past the last day. -
Each line has
time,role,instance_id,stream,lineandcursor. The page has thecursorof its last line. -
Set
afterto a page’scursorto get the next 500 lines stored since then. The response hasmore: truewhen there may be more lines. -
Paging follows the storage order, so a report that arrives late still shows.
-
The console polls every 3 seconds, and at once while
moreis true.pl logs -fpolls every 2 seconds.
GET /v1/instances/metrics?project=&environment=[&hours=] returns about 90 points per instance over the window. The default window is 1 hour, and the maximum is 168.
- Each point has
instance_id,role,t(Unix seconds),cpu_millis(average),memory_bytes(peak),memory_limit_bytes, and therx_bytesandtx_bytescounters. - A counter drops when the container restarts.
Operating it
The gateway writes to ClickHouse: pushleft_otel.instance_samples and instance_logs, with the schema in deploy/observability/clickhouse/003_instances.sql.
- Set
PUSHLEFT_INSTANCES_CLICKHOUSE_ENDPOINT,_USERNAMEand_PASSWORDon the gateway. Use a ClickHouse user that can only insert into those two tables. - Without these settings, the gateway refuses reports with 503, the agents keep retrying, and the console says that storage is not configured.
- The API reads with the error-tracking ClickHouse user (
PUSHLEFT_ERRORS_CLICKHOUSE_*, error tracking). It needsSELECTon both tables. - The gateway’s database role needs
EXECUTEoninstance_telemetry_admit(deploy/README.md).
Related
- Errors and performance: grouped errors and request timings.
- Managed databases: insights: database metrics.
- Retention: how long Pushleft keeps history.
- Operating Pushleft: the platform’s own signals and alerts.