pushleft
Deploy your first app
Manual contents
Handbook
HandbookPushleftArchitecture
Operate
Install the control planeYour own serversFleetOperations and alertsAuthenticationDomains and DNSUsage and budgetsScheduled jobsAbuse controlsAI provider proxyCustomer mailErrors and performanceLogs and metricsRetentionObject storageContainer registryGit server
Develop
The .pl.yaml manifestOnboard with an AI agentThe pl up demoBuild executorSource ingestionAgent gatewayManaged databasesSecurity findingsTunnels and shellsDev workspacesPrivate hostnamesMesh coordinatorsDesign baseline
Runbooks
Control-plane host failoverControl-plane PostgreSQL failoverControl-plane PostgreSQL restoreCustom domains and certificatesGateway key compromiseIngress host failureNode busyNode disk fullNode drain and replacementNode lostRegistry outageHetzner Robot hostsSecret rotationSecurity incidentStaging fleet on ProxmoxStaging build cacheStuck deploymentVault/OpenBao backup and recovery
Manual · docs/errors.md

Errors and performance

Pushleft collects the errors and request timings of your applications through any Sentry SDK. The console groups errors into issues with stack traces, and charts latency, throughput and error rate per endpoint. You keep your existing Sentry code and change only the DSN.

Console Issues from any Sentry SDK, with regressions marked.
Console One issue: the stack trace in your source, the breadcrumbs and the tags.
Console One slow request, step by step, with the time each step took.

Quick start

  1. Create a DSN for the environment and store it as the secret SENTRY_DSN:

    pl errors dsn create web --production --set-secret

    The command prints a DSN such as https://[email protected]/2645624911 once. Without --production, the environment is this checkout’s preview/<branch>. Use --project and --environment to name another one.

  2. Give the secret to the application as an environment variable (manifest secrets):

    roles:
      web:
        secrets: [{ name: SENTRY_DSN, as: env }]

    Sentry SDKs read SENTRY_DSN when init has no dsn.

  3. Initialize the SDK with a release and, for performance data, a traces sample rate. Regression detection and source maps need the release.

    sentry_sdk.init(release="[email protected]", traces_sample_rate=0.2)
    Sentry.init({ dsn: process.env.SENTRY_DSN, release: '[email protected]', tracesSampleRate: 0.2 });
  4. Deploy. The next deployment gets the secret:

    pl up --production
  5. Optional: turn on alert mail for new and regressed issues, and upload the source maps of a JavaScript build:

    pl errors alerts on --production
    pl errors sourcemaps upload dist --release [email protected] --production

Errors appear on the console Errors page.

Features

  • Errors grouped into issues, with stack traces, breadcrumbs, tags and the number of affected users.
  • Resolve and ignore. A resolved issue reopens as a regression when it comes back in a newer release.
  • Alert mail to owners, admins and members, at most 10 per environment per hour.
  • Median and 95th percentile latency, throughput and error rate per endpoint, the slowest traces and a waterfall of one trace.
  • Minified JavaScript stack traces mapped back to source.
  • Error events kept for 14, 30 or 90 days, by trust tier (Storage).
  • A daily error-event quota of 5 000, 100 000 or 1 000 000, by trust tier (Limits and metering).

Pushleft does not do session replay, profiling, release health, cron monitors or attachments. The SDK can send those. Pushleft accepts and drops them, so the SDK does not retry them.

For plain stdout and stderr output, see service output.

DSNs

A DSN is not a secret in the usual sense, because browser bundles contain it. It lets its holder send events to one environment and nothing else.

Each environment is one Sentry “project” with a numeric project id: the last part of the DSN. Each DSN has its own public key. Pushleft stores only its SHA-256 and shows the key once.

CommandAPI
pl errors dsn create <name> [--production | --project P --environment E] [--set-secret]POST /v1/errors/dsns {"project", "environment", "name"} (members)
pl errors dsn listGET /v1/errors (DSNs without keys, per-environment settings, tier limits)
pl errors dsn rotate <id> [--overlap-minutes N]POST /v1/errors/dsns/{id}/rotate: a new key; the old one works for the overlap (default 10 minutes, 0 to end it now)
pl errors dsn revoke <id>DELETE /v1/errors/dsns/{id}: refused at once

A key opens only its own project id. A key of one organization or environment used with the project id of another gets 401.

Ingest

pushleft-errors serves the Sentry ingest protocol at https://errors.<base domain>:

PathBody
POST /api/<project>/envelope/A Sentry envelope: a JSON header line, then items (a JSON header line and a payload of length bytes or up to the next newline).
POST /api/<project>/store/One event as JSON (old SDKs).
  • Authentication. Server SDKs send X-Sentry-Auth: Sentry sentry_key=<key>, …. Browser and Node SDKs send the sentry_key query parameter. Tunnelled browser events carry the dsn in the envelope header.
  • Bodies. Content-Encoding can be gzip, deflate (zlib or raw) or br. The limit is 20 MiB after decompression (PUSHLEFT_ERRORS_MAX_ENVELOPE_BYTES). Pushleft drops an event or transaction item larger than 1 MiB.
  • Items. Pushleft stores event items as errors and turns transaction items into spans. It accepts and drops every other type and still answers 200. The dropped types are attachment, session, sessions, client_report, replay_event, replay_recording, profile, check_in, statsd, and types that do not exist yet.
  • Identity. The organization and environment come from the DSN only. Nothing in a payload can name another tenant. Rows in ClickHouse carry the DSN’s organization and environment. The server sets the span resource attributes pushleft.org_id and pushleft.environment_id.
  • CORS. Any origin can post, for browser SDKs. Scripts can read X-Sentry-Rate-Limits and Retry-After.
ResponseWhen
200 {"id": "<event id>"}Accepted, including envelopes with only dropped items
401Missing, unknown, revoked or expired key, or a key for another project
403The organization is suspended (abuse controls)
429 with Retry-After and X-Sentry-Rate-Limits: <s>::project:rate_limitedPer-minute limit of the organization (PUSHLEFT_ERRORS_ORG_REQUESTS_PER_MINUTE, default 6000) or environment (…_ENVIRONMENT_…, default 3000)
429 with X-Sentry-Rate-Limits: <s>:error:organization:quota_exceededThe tier’s daily error-event quota is used up and the envelope had nothing else. With transactions in it, the answer is 200 with the same header, and only the errors are dropped
429 with X-Sentry-Rate-Limits: 3600::organization:budget_exceededThe organization’s hard-stop budget is spent (budgets)

Sentry SDKs honour X-Sentry-Rate-Limits. They stop sending the named categories until the time has passed, and they do not retry.

Issues

Each error event belongs to one issue per environment. Pushleft picks the fingerprint in this order:

  1. The SDK’s fingerprint, when it sends one. {{ default }} in it stands for the default below.
  2. For exceptions: the exception types plus the five innermost in-app frames of each (module or file, and function). Line numbers, build hashes in file names, and URL hosts and queries do not count. Without in-app frames, all frames count. Without frames, the normalized exception value counts.
  3. Else the message template: logentry.message, or the message with numbers, hex values and UUIDs replaced.

The title is <type>: <value> or the first line of the message. The culprit is the transaction name or the innermost in-app frame.

StateMeaning
unresolvedOpen. New issues start here.
resolvedFixed. Pushleft remembers the last release that the issue was seen in. An event from a release first seen after that one reopens the issue as a regression. Events from that release or older ones (for example instances still running the old version) do not. If no release was known when it was resolved, any new event reopens it. Events without a release never reopen an issue that was resolved in a known release.
ignoredCounted, never reopened, no mail.

The issue API:

  • GET /v1/errors/issues?project=&environment=&status=unresolved|resolved|ignored|all lists issues. Each has the event count, the number of distinct users, first and last seen, and the first and last release. Users come from user.id, email, username or ip_address, hashed.
  • GET /v1/errors/issues/{id} adds the 20 most recent events. Each event has the stack trace, the last 100 breadcrumbs, tags, user, contexts, extra data and the request. Pushleft removes cookies and the Authorization, Cookie and X-Sentry-Auth headers from the request.
  • PATCH /v1/errors/issues/{id} with {"status": …} resolves, ignores or reopens an issue (members).

The console Errors page has the same functions.

Alert mail

Turn alert mail on with pl errors alerts on|off [--production | --project P --environment E], the checkbox on the Errors page, or PUT /v1/errors/alerts. Pushleft then mails the organization’s owners, admins and members when an issue is new or regresses in that environment.

  • Mail goes through the platform mailer (PUSHLEFT_SMTP_URL, production authentication only, platform mail).
  • At most 10 mails per environment per hour. Pushleft does not send more.
  • Alert mail is off by default.

Source maps

Upload the maps of a release before or after its errors arrive. Pushleft applies maps when you read an issue, so late uploads work.

pl errors sourcemaps upload dist --release [email protected] --production
pl errors sourcemaps list --release [email protected] --production

Names. Pushleft stores every *.map under the directory under the path of its minified file. For example, dist/static/app.min.js.map serves static/app.min.js.

  • --url-prefix adds a prefix: assets/, or ~/ to match any host.
  • A frame matches the longest stored name that its URL path ends with. For example, https://shop.test/static/app.min.js?v=3 matches static/app.min.js.

CI. Use an API token with the member role and POST /v1/errors/sourcemaps?project=&environment=&release=&name=, with the map as the body (at most 50 MiB).

Mapped frames. A mapped frame shows the original file, line, column and five lines of context from sourcesContent. The minified position stays in raw. Function names stay as the bundle had them.

  • Only flat version 3 maps work (the bundlers’ default). Pushleft refuses index maps with sections.
  • Pushleft stores maps by content hash in the BlobStore, under its artifacts use (R2 in production).
  • A new upload with the same name in the same release replaces the old map.

Grouping happens at ingest, before maps apply. Minified function names that change between builds can therefore split one error into one issue per release (gap).

Performance

Each transaction becomes one segment span (the transaction) and its child spans. Pushleft stores them in the OpenTelemetry column layout of pushleft_otel.otel_traces, plus the tenant columns. It drops a transaction that started more than 7 days ago or more than an hour in the future.

GET /v1/errors/performance?project=&environment=&hours=24 (1 to 720 hours) returns:

  • for the 100 busiest endpoints (transaction names): the request count, p50 and p95 duration and requests per minute. It also shows the error rate with the exact failed count. An error is a segment with an error status or an HTTP status of 500 or more. Add &sort=errors to get the 100 endpoints with the most failures;
  • totals over every endpoint: requests, failed, and the p95 of the whole environment;
  • series with the same three values per hour;
  • the 20 slowest traces.

GET /v1/errors/traces/{trace_id} returns every span of one trace in the organization, for the waterfall. The console shows both on the Errors page.

This view does not show spans from OpenTelemetry SDKs that go through the node collectors. They have no verified tenant identity yet (operations).

Storage

DataWhereKept
Error events (reduced payload, issue id, release, tags, hashed user)ClickHouse pushleft_otel.error_events, partitioned by the day received, ReplacingMergeTree on the event idThe tier’s error_retention_days (14 / 30 / 90), stamped per row at ingest
SpansClickHouse pushleft_otel.tenant_spansSame retention
DSN keys (SHA-256), project numbers, alert switch, issues and their state, release order, alert outbox, source map indexPostgreSQL (migration 0066), row-level security per organizationLife of the environment
Source mapsBlobStore artifacts useUntil the project is purged (deleted projects)

The schema is deploy/observability/clickhouse/002_errors.sql. The ingest service needs a ClickHouse user with INSERT on both tables, and the API needs a user with SELECT. The API binds the caller’s organization as a query parameter in every read.

Limits and metering

  • Per-minute limits per organization and environment. All ingest replicas share them (PostgreSQL counters).
  • error_events_per_day per tier, per UTC day: 5 000 (new), 100 000 (standard), 1 000 000 (trusted) (trust tiers). Transactions count against the per-minute limits only.
  • Each stored error event is one error_events unit in the usage ledger (billing). A replayed event id counts once.
  • A hard-stop budget and suspension stop ingest. Running applications keep running.

Running it

pushleft-errors is stateless. Its unit is systemd/pushleft-errors.service and its configuration is systemd/errors.env.example. It runs behind the TLS proxy as errors.<base> (deploy/README.md).

  • Its database role pushleft_errors has no table grants. It can only execute errors_admit and errors_record_event.
  • The API needs PUSHLEFT_ERRORS_PUBLIC_URL to build DSNs, PUSHLEFT_ERRORS_CLICKHOUSE_* to read, and the artifacts BlobStore for source maps.

npm run test:errors runs the official Sentry Python SDK (2.71.0) and @sentry/node (10.75.3) against a local ingest, PostgreSQL and the dev compose ClickHouse.

Gaps

  • Grouping uses minified frames. Source maps apply on read only.
  • No OTLP endpoint for OpenTelemetry SDKs yet. Only Sentry transactions feed the performance view.
  • Issues have no merge, assignment, comments, search or per-issue alert rules. Alerts do not go to Slack or webhooks.
  • The daily quota is a fixed UTC day, not a rolling 24 hours. Transactions have no daily quota.
  • Deleting a project leaves its ClickHouse rows to their retention. Its source maps go when the project is purged, 7 days after the deletion (deleted projects).
Rendered from the repository at build time. Edit docs/errors.md to change it.