Errors and performance
Pushleft collects the errors and request timings of your applications through any Sentry SDK. The console groups errors into issues with stack traces, and charts latency, throughput and error rate per endpoint. You keep your existing Sentry code and change only the DSN.
Quick start
-
Create a DSN for the environment and store it as the secret
SENTRY_DSN:pl errors dsn create web --production --set-secretThe command prints a DSN such as
https://[email protected]/2645624911once. Without--production, the environment is this checkout’spreview/<branch>. Use--projectand--environmentto name another one. -
Give the secret to the application as an environment variable (manifest secrets):
roles: web: secrets: [{ name: SENTRY_DSN, as: env }]Sentry SDKs read
SENTRY_DSNwheninithas nodsn. -
Initialize the SDK with a
releaseand, for performance data, a traces sample rate. Regression detection and source maps need the release.sentry_sdk.init(release="[email protected]", traces_sample_rate=0.2)Sentry.init({ dsn: process.env.SENTRY_DSN, release: '[email protected]', tracesSampleRate: 0.2 }); -
Deploy. The next deployment gets the secret:
pl up --production -
Optional: turn on alert mail for new and regressed issues, and upload the source maps of a JavaScript build:
pl errors alerts on --production pl errors sourcemaps upload dist --release [email protected] --production
Errors appear on the console Errors page.
Features
- Errors grouped into issues, with stack traces, breadcrumbs, tags and the number of affected users.
- Resolve and ignore. A resolved issue reopens as a regression when it comes back in a newer release.
- Alert mail to owners, admins and members, at most 10 per environment per hour.
- Median and 95th percentile latency, throughput and error rate per endpoint, the slowest traces and a waterfall of one trace.
- Minified JavaScript stack traces mapped back to source.
- Error events kept for 14, 30 or 90 days, by trust tier (Storage).
- A daily error-event quota of 5 000, 100 000 or 1 000 000, by trust tier (Limits and metering).
Pushleft does not do session replay, profiling, release health, cron monitors or attachments. The SDK can send those. Pushleft accepts and drops them, so the SDK does not retry them.
For plain stdout and stderr output, see service output.
DSNs
A DSN is not a secret in the usual sense, because browser bundles contain it. It lets its holder send events to one environment and nothing else.
Each environment is one Sentry “project” with a numeric project id: the last part of the DSN. Each DSN has its own public key. Pushleft stores only its SHA-256 and shows the key once.
| Command | API |
|---|---|
pl errors dsn create <name> [--production | --project P --environment E] [--set-secret] | POST /v1/errors/dsns {"project", "environment", "name"} (members) |
pl errors dsn list | GET /v1/errors (DSNs without keys, per-environment settings, tier limits) |
pl errors dsn rotate <id> [--overlap-minutes N] | POST /v1/errors/dsns/{id}/rotate: a new key; the old one works for the overlap (default 10 minutes, 0 to end it now) |
pl errors dsn revoke <id> | DELETE /v1/errors/dsns/{id}: refused at once |
A key opens only its own project id. A key of one organization or environment used with the project id of another gets 401.
Ingest
pushleft-errors serves the Sentry ingest protocol at https://errors.<base domain>:
| Path | Body |
|---|---|
POST /api/<project>/envelope/ | A Sentry envelope: a JSON header line, then items (a JSON header line and a payload of length bytes or up to the next newline). |
POST /api/<project>/store/ | One event as JSON (old SDKs). |
- Authentication. Server SDKs send
X-Sentry-Auth: Sentry sentry_key=<key>, …. Browser and Node SDKs send thesentry_keyquery parameter. Tunnelled browser events carry thedsnin the envelope header. - Bodies.
Content-Encodingcan begzip,deflate(zlib or raw) orbr. The limit is 20 MiB after decompression (PUSHLEFT_ERRORS_MAX_ENVELOPE_BYTES). Pushleft drops aneventortransactionitem larger than 1 MiB. - Items. Pushleft stores
eventitems as errors and turnstransactionitems into spans. It accepts and drops every other type and still answers200. The dropped types areattachment,session,sessions,client_report,replay_event,replay_recording,profile,check_in,statsd, and types that do not exist yet. - Identity. The organization and environment come from the DSN only. Nothing in a payload can name another tenant. Rows in ClickHouse carry the DSN’s organization and environment. The server sets the span resource attributes
pushleft.org_idandpushleft.environment_id. - CORS. Any origin can post, for browser SDKs. Scripts can read
X-Sentry-Rate-LimitsandRetry-After.
| Response | When |
|---|---|
200 {"id": "<event id>"} | Accepted, including envelopes with only dropped items |
401 | Missing, unknown, revoked or expired key, or a key for another project |
403 | The organization is suspended (abuse controls) |
429 with Retry-After and X-Sentry-Rate-Limits: <s>::project:rate_limited | Per-minute limit of the organization (PUSHLEFT_ERRORS_ORG_REQUESTS_PER_MINUTE, default 6000) or environment (…_ENVIRONMENT_…, default 3000) |
429 with X-Sentry-Rate-Limits: <s>:error:organization:quota_exceeded | The tier’s daily error-event quota is used up and the envelope had nothing else. With transactions in it, the answer is 200 with the same header, and only the errors are dropped |
429 with X-Sentry-Rate-Limits: 3600::organization:budget_exceeded | The organization’s hard-stop budget is spent (budgets) |
Sentry SDKs honour X-Sentry-Rate-Limits. They stop sending the named categories until the time has passed, and they do not retry.
Issues
Each error event belongs to one issue per environment. Pushleft picks the fingerprint in this order:
- The SDK’s
fingerprint, when it sends one.{{ default }}in it stands for the default below. - For exceptions: the exception types plus the five innermost in-app frames of each (module or file, and function). Line numbers, build hashes in file names, and URL hosts and queries do not count. Without in-app frames, all frames count. Without frames, the normalized exception value counts.
- Else the message template:
logentry.message, or the message with numbers, hex values and UUIDs replaced.
The title is <type>: <value> or the first line of the message. The culprit is the transaction name or the innermost in-app frame.
| State | Meaning |
|---|---|
unresolved | Open. New issues start here. |
resolved | Fixed. Pushleft remembers the last release that the issue was seen in. An event from a release first seen after that one reopens the issue as a regression. Events from that release or older ones (for example instances still running the old version) do not. If no release was known when it was resolved, any new event reopens it. Events without a release never reopen an issue that was resolved in a known release. |
ignored | Counted, never reopened, no mail. |
The issue API:
GET /v1/errors/issues?project=&environment=&status=unresolved|resolved|ignored|alllists issues. Each has the event count, the number of distinct users, first and last seen, and the first and last release. Users come fromuser.id,email,usernameorip_address, hashed.GET /v1/errors/issues/{id}adds the 20 most recent events. Each event has the stack trace, the last 100 breadcrumbs, tags, user, contexts, extra data and the request. Pushleft removes cookies and theAuthorization,CookieandX-Sentry-Authheaders from the request.PATCH /v1/errors/issues/{id}with{"status": …}resolves, ignores or reopens an issue (members).
The console Errors page has the same functions.
Alert mail
Turn alert mail on with pl errors alerts on|off [--production | --project P --environment E], the checkbox on the Errors page, or PUT /v1/errors/alerts. Pushleft then mails the organization’s owners, admins and members when an issue is new or regresses in that environment.
- Mail goes through the platform mailer (
PUSHLEFT_SMTP_URL, production authentication only, platform mail). - At most 10 mails per environment per hour. Pushleft does not send more.
- Alert mail is off by default.
Source maps
Upload the maps of a release before or after its errors arrive. Pushleft applies maps when you read an issue, so late uploads work.
pl errors sourcemaps upload dist --release [email protected] --production
pl errors sourcemaps list --release [email protected] --production
Names. Pushleft stores every *.map under the directory under the path of its minified file. For example, dist/static/app.min.js.map serves static/app.min.js.
--url-prefixadds a prefix:assets/, or~/to match any host.- A frame matches the longest stored name that its URL path ends with. For example,
https://shop.test/static/app.min.js?v=3matchesstatic/app.min.js.
CI. Use an API token with the member role and POST /v1/errors/sourcemaps?project=&environment=&release=&name=, with the map as the body (at most 50 MiB).
Mapped frames. A mapped frame shows the original file, line, column and five lines of context from sourcesContent. The minified position stays in raw. Function names stay as the bundle had them.
- Only flat version 3 maps work (the bundlers’ default). Pushleft refuses index maps with
sections. - Pushleft stores maps by content hash in the BlobStore, under its
artifactsuse (R2 in production). - A new upload with the same name in the same release replaces the old map.
Grouping happens at ingest, before maps apply. Minified function names that change between builds can therefore split one error into one issue per release (gap).
Performance
Each transaction becomes one segment span (the transaction) and its child spans. Pushleft stores them in the OpenTelemetry column layout of pushleft_otel.otel_traces, plus the tenant columns. It drops a transaction that started more than 7 days ago or more than an hour in the future.
GET /v1/errors/performance?project=&environment=&hours=24 (1 to 720 hours) returns:
- for the 100 busiest endpoints (transaction names): the request count, p50 and p95 duration and requests per minute. It also shows the error rate with the exact
failedcount. An error is a segment with an error status or an HTTP status of 500 or more. Add&sort=errorsto get the 100 endpoints with the most failures; totalsover every endpoint: requests, failed, and the p95 of the whole environment;serieswith the same three values per hour;- the 20 slowest traces.
GET /v1/errors/traces/{trace_id} returns every span of one trace in the organization, for the waterfall. The console shows both on the Errors page.
This view does not show spans from OpenTelemetry SDKs that go through the node collectors. They have no verified tenant identity yet (operations).
Storage
| Data | Where | Kept |
|---|---|---|
| Error events (reduced payload, issue id, release, tags, hashed user) | ClickHouse pushleft_otel.error_events, partitioned by the day received, ReplacingMergeTree on the event id | The tier’s error_retention_days (14 / 30 / 90), stamped per row at ingest |
| Spans | ClickHouse pushleft_otel.tenant_spans | Same retention |
| DSN keys (SHA-256), project numbers, alert switch, issues and their state, release order, alert outbox, source map index | PostgreSQL (migration 0066), row-level security per organization | Life of the environment |
| Source maps | BlobStore artifacts use | Until the project is purged (deleted projects) |
The schema is deploy/observability/clickhouse/002_errors.sql. The ingest service needs a ClickHouse user with INSERT on both tables, and the API needs a user with SELECT. The API binds the caller’s organization as a query parameter in every read.
Limits and metering
- Per-minute limits per organization and environment. All ingest replicas share them (PostgreSQL counters).
error_events_per_dayper tier, per UTC day: 5 000 (new), 100 000 (standard), 1 000 000 (trusted) (trust tiers). Transactions count against the per-minute limits only.- Each stored error event is one
error_eventsunit in the usage ledger (billing). A replayed event id counts once. - A hard-stop budget and suspension stop ingest. Running applications keep running.
Running it
pushleft-errors is stateless. Its unit is systemd/pushleft-errors.service and its configuration is systemd/errors.env.example. It runs behind the TLS proxy as errors.<base> (deploy/README.md).
- Its database role
pushleft_errorshas no table grants. It can only executeerrors_admitanderrors_record_event. - The API needs
PUSHLEFT_ERRORS_PUBLIC_URLto build DSNs,PUSHLEFT_ERRORS_CLICKHOUSE_*to read, and theartifactsBlobStore for source maps.
npm run test:errors runs the official Sentry Python SDK (2.71.0) and @sentry/node (10.75.3) against a local ingest, PostgreSQL and the dev compose ClickHouse.
Gaps
- Grouping uses minified frames. Source maps apply on read only.
- No OTLP endpoint for OpenTelemetry SDKs yet. Only Sentry transactions feed the performance view.
- Issues have no merge, assignment, comments, search or per-issue alert rules. Alerts do not go to Slack or webhooks.
- The daily quota is a fixed UTC day, not a rolling 24 hours. Transactions have no daily quota.
- Deleting a project leaves its ClickHouse rows to their retention. Its source maps go when the project is purged, 7 days after the deletion (deleted projects).
Related
- Service output and metrics: stdout, stderr and instance charts.
- The manifest: secrets: pass
SENTRY_DSNto a role. - Billing: meters and budgets.
- Abuse controls: trust tiers and suspension.