pushleft
Deploy your first app
Manual contents
Handbook
HandbookPushleftArchitecture
Operate
Install the control planeYour own serversFleetOperations and alertsAuthenticationDomains and DNSUsage and budgetsScheduled jobsAbuse controlsAI provider proxyCustomer mailErrors and performanceLogs and metricsRetentionObject storageContainer registryGit server
Develop
The .pl.yaml manifestOnboard with an AI agentThe pl up demoBuild executorSource ingestionAgent gatewayManaged databasesSecurity findingsTunnels and shellsDev workspacesPrivate hostnamesMesh coordinatorsDesign baseline
Runbooks
Control-plane host failoverControl-plane PostgreSQL failoverControl-plane PostgreSQL restoreCustom domains and certificatesGateway key compromiseIngress host failureNode busyNode disk fullNode drain and replacementNode lostRegistry outageHetzner Robot hostsSecret rotationSecurity incidentStaging fleet on ProxmoxStaging build cacheStuck deploymentVault/OpenBao backup and recovery
Manual · docs/billing.md

Usage and budgets

Pushleft measures what your organization uses each hour and prices it in EUR with your organization’s plan. Each month closes into a statement. Set a monthly budget to get email alerts, or to stop new work when the budget is spent. Nobody is charged yet: there is no payment provider, so a statement is not an invoice.

Console This month's usage per meter, the plan and the budget.
Console A closed month's statement: the base fee and one line per meter.

Quick start

  1. In the console, open Usage. It shows your plan. Select a month to see the quantity and cost per meter, and the statements of closed months.
  2. Under Monthly budget, type the amount in EUR per month.
  3. In Alert at (%), type the thresholds, for example 50, 80, 100.
  4. In When it is spent, select alert only or hard stop, then select Save budget. Only owners and admins can change the budget.

You can also set the budget with the API:

curl -X PUT https://api.pushleft.dev/v1/orgs/<org id>/budget \
  -H "Authorization: Bearer $PUSHLEFT_TOKEN" -H 'Content-Type: application/json' \
  -d '{"monthly_amount_eur": 50, "thresholds": [50, 80, 100], "mode": "hard_stop"}'

Features

  • A free trial for each new sign-up, and payment through Stripe.
  • A plan per organization: self-hosted (free), Cloud with your own servers, or Cloud (plans). The plan sets a monthly base fee and the price of each meter.
  • Hourly meters for compute, disk, stored data, builds, mail messages, AI tokens and error events (meters).
  • A statement for each closed month: the base fee plus one line per meter (statements).
  • Email alerts to owners and admins when this month’s spend reaches each threshold.
  • With hard stop, new deployments, mail sends and AI requests are refused with 402 budget_exceeded, and the error ingest drops new events (429). Running applications keep serving.
  • Spend comes from finished hours only, so a hard stop takes effect up to about 65 minutes after the usage that crossed the amount.

Internals

Pushleft keeps a ledger and freezes statements from it. There is no payment provider, so nobody is charged. When one is added, it issues the legal invoice (number, VAT, PDF) from each statement. Domain checkout debits a separate prepaid domain balance. Nothing funds that balance except an operator in the database, so a domain purchase fails with 402 until one does. Metered (resold) AI stays off until a payment provider is chosen. The “payment method” signal that could raise a trust tier is not available.

Budget and limit checks run before Pushleft accepts work (architecture invariant 7). A hard-stop budget refuses the deployment in the same transaction as the per-organization deployment quota. The budget counts the ledger and the base fee accrued so far, the same amounts the statement charges.

Meters

MeterUnitSource
compute_vcpu_hours_productionvCPU-hourcpu_millis / 1000 × hours active, of allocations in production environments
compute_vcpu_hours_previewvCPU-hourThe same, for preview (test) environments and allocations without an environment
compute_memory_gb_hoursGB-hourmemory_mb / 1024 × hours active, less the RAM the plan includes per vCPU (included_memory_gb_per_vcpu × vCPU-hours of the same hour)
database_storage_gb_hoursGB-hourdisk_gb of database_instance allocations (PostgreSQL, ClickHouse, Valkey) × hours active
volume_storage_gb_hoursGB-hourdisk_gb of volume allocations × hours active (provisioned size)
backup_storage_gb_hoursGB-hourBytes of the organization’s off-host backups on R2 (database and volume bases and WAL), from the storage sampler
dataset_storage_gb_hoursGB-hourdatabase_datasets.bytes
bucket_storage_gb_hoursGB-hourBytes in the organization’s object storage buckets, from the storage sampler
registry_storage_gb_hoursGB-hourBytes of the organization’s registry namespaces, from the storage sampler
production_environment_hoursenvironment-hourEach production environment that holds capacity at any time in the hour
build_minutesminutebuild_image jobs finished in the hour: from lease start to completion
mail_messagesmessagerecord_usage() from the customer-mail service, one per accepted recipient, through record_mail_event (mail.md)
ai_tokens_intokenrecord_usage() from the AI proxy: input tokens, cached prompt tokens included (ai-proxy.md)
ai_tokens_outtokenrecord_usage() from the AI proxy: output tokens, reasoning tokens included
error_eventseventrecord_usage() from the error ingest, one per stored error event (transactions are not metered), through errors_record_event (errors.md)
runtime_log_gbGBRuntime output admitted from your instances (logs)

Ledger rows from before migration 0151 can also hold compute_vcpu_hours (one CPU meter) and ai_tokens. The usage API lists them only for a month that has them.

Allocations. Compute and disk come from fleet allocations. An allocation counts from activated_at to released_at, or to the end of the hour while it is still active. A trigger sets activated_at at its first move to active (migration 0051). A reservation that is released before it became active is never metered.

  • The environment of an allocation comes from its application instance, database instance or volume (the view fleet_allocation_charges).
  • Not billed: the per-node environment network gateway (network_attachment), and compute and disk on a node that your organization owns (your own servers). Builds on a build node that you own are not billed either. A production environment counts for production_environment_hours on any node, your own included.

Production CPU. Nodes promise 5x their CPU threads (CPU overcommit). Every vCPU counts 1x against that capacity today (charged_cpu_millis, cpu_charge()). Counting a production vCPU 2.5x, so production is at most 2x overcommitted, returns when more hosts are added. Production application containers also get 2048 Docker CPU shares (the default is 1024). Production database containers do not yet.

Storage sampler

The API measures stored bytes outside PostgreSQL once an hour. Each source is optional. A source that fails records nothing that hour. The rollup uses each meter’s latest run of the last 26 hours, and an organization that is missing from that run holds 0 bytes.

Setting (API)Measures
PUSHLEFT_BILLING_OFFSITE_S3_{BUCKET,ENDPOINT,REGION,PREFIX,ACCESS_KEY_ID,SECRET_ACCESS_KEY_FILE}The off-host backup bucket (pushleft-offsite). Use a read-only token. Each key counts for the organization ID in its path (<host>/tenant/<org>/...).
PUSHLEFT_BILLING_REGISTRY_S3_* (same suffixes)The registry bucket. Set _PREFIX to zot’s root directory. Keys under organizations/<org>/ and proxy/<org>/ count for that organization.
The object storage settingsEach active Garage bucket through GetBucketInfo. A run where Garage does not answer for every bucket is not recorded.

The sampler writes with record_storage_samples() (SECURITY DEFINER). Runs older than 90 days are deleted.

Events. The mail and AI services call

SELECT record_usage(org uuid, meter text, quantity numeric, occurred_at timestamptz, idempotency_key text);

It returns true when it records the event and false for a replay of the same (org, meter, idempotency_key).

An event goes into the hour in which Pushleft recorded it. This way an event that is reported late still counts. occurred_at is kept for reference.

The function is SECURITY DEFINER, and PUBLIC cannot execute it. Grant EXECUTE to the service role when the service exists (database roles).

The ledger

The ledger is the record of what each organization used and what it cost. The scheduler rolls up each finished UTC hour into usage_ledger. Each row holds one organization, meter and hour, with the quantity, the unit price used and the cost in EUR (rounded to 6 decimals).

Rows are immutable. A trigger rejects UPDATE and DELETE, and tenants can read their own rows through RLS.

  • Watermark. usage_rollup_state.rolled_until is the first hour that is not yet in the ledger. Each step (usage_rollup_next) runs in one transaction. It locks the watermark row, rolls up one hour, moves the watermark forward and evaluates budgets. Two scheduler replicas cannot roll up the same hour twice.
  • Grace. The scheduler rolls up an hour 5 minutes after it ends. Usage from a transaction that was still open at the boundary commits first.
  • Catch-up. After downtime, the scheduler rolls up the missing hours oldest first, at most 168 per minute, until it reaches the present.
  • Idempotent. Rolling up an hour that is already in the ledger inserts nothing (ON CONFLICT DO NOTHING). If you move the watermark back, the scheduler re-checks the hours but does not change them.
  • Start. The watermark starts at the hour in which migration 0051 ran. Usage from before that hour is not reconstructed.
  • Prices come from the organization’s plan of that hour. Rows that already exist keep the price they were rolled up with. A plan change cannot start before the watermark, so the ledger is never repriced.
  • Month close. Rolling the first hour of a month freezes the statements of the month before it.

Plans

A plan sets a base fee per month and a price per meter. A plan never changes once it is published: a database trigger refuses the change. To change a price, add a plan with a new ID and move organizations to it. Their old months keep the old plan.

PlanBase feePrices
trial€0Every meter costs 0. A sign-up organization starts here (trial)
self-hosted€0Every meter costs 0. The signup default of a new installation
byo-2026-11 (Cloud, your own servers)€10/monthDatasets, buckets and registry images at €0.10/GB a month. Work runs only on nodes your organization owns
cloud-2026-11 (Cloud)€10/monthProduction vCPU €23 and preview vCPU €13, each with 2 GB RAM included; RAM beyond that €5/GB; storage €0.10/GB (database disk, volumes, datasets, buckets, registry images), per month

deploy/billing/plans.json holds the plans, and the pricing page reads the same file. Prices are EUR excluding VAT.

  • A meter whose name contains _hours is priced per unit-month (EUR per vCPU-month, GB-month or environment-month). The rollup divides that price by 730 hours. Other meters are priced per unit.
  • A meter that a plan does not list costs 0. An organization without a plan pays 0.
  • Included on both paid plans (metered at 0, so the usage page still shows them): backups, build minutes, mail, error events, runtime logs and production environments (migration 0155).
  • Own nodes only. A plan with own_nodes_only places its organization’s applications, databases and volumes only on nodes that it owns (node_serves_org()). Builds can still use the shared build nodes, and are billed as build_minutes.

Plan history. Each organization has one plan at a time (org_plan_periods). A new organization starts on the signup default plan. Migration 0151 put every existing organization on self-hosted.

The operator manages plans with the migration-owner DATABASE_URL:

pushleft-api billing plans                                  # every plan, as JSON lines
pushleft-api billing add-plans deploy/billing/plans.json    # add new plans; refuses changed terms
pushleft-api billing default cloud-2026-11                  # the plan new organizations start on
pushleft-api billing assign --org <uuid> --plan byo-2026-11 [--from <rfc3339>] [--reason <text>]
pushleft-api billing statements --org <uuid>
pushleft-api billing close-month 2026-11                    # normally the scheduler does this
pushleft-api billing paid --org <uuid> --month 2026-11

assign closes the current period and opens the new one at the start of the current hour, or of the hour of --from: a past hour that is not yet in the ledger. Periods start on a UTC hour because the rollup prices each hour with one plan. A change never starts in the future, because the open period is the plan that placement and the API use now.

Statements

When the scheduler rolls the first hour of a month, it freezes the month before it (billing_close_month). Each organization with a base fee or a ledger cost in that month gets one statement:

  • The base fee of each plan period in the month, prorated by hours. A month on cloud-2026-11 for half the month shows 0.5 × €10.
  • One line per meter and unit price, with the summed quantity and cost.
  • Amounts are rounded to cents. The total is the sum of the lines.

A statement and its lines never change after they are made. Only the payment state changes: open, paid or void, with payment_provider none or stripe, and the provider’s reference (payment). Closing a month again makes nothing new.

Trial

An organization from sign-up gets a free trial of 14 days (PUSHLEFT_TRIAL_DAYS). It is on the trial plan, so nothing costs money, and it has the limits of the new trust tier.

  • Three days before the end, owners and admins get a mail when no payment method is saved.
  • At the end with a payment method, the organization moves to the newest byo plan. It moves to the newest cloud plan when it uses Pushleft servers. The API checks this once a minute.
  • At the end without a payment method, the API refuses new deployments with 402 trial_ended. Running applications keep serving. Owners and admins get a mail, and so does the operator. After the owner adds a payment method, the organization moves to its paid plan within a minute.
  • After the end, new deployments need a payment method and a paid plan. If the move to the paid plan fails, the API refuses new deployments. It tries the move again at each deployment and each minute. If the owner removes the payment method, the API refuses new deployments until a new one is saved.
  • The API sends a failed mail again after 10 minutes, for up to 2 days, only to the addresses that did not get it.
  • The operator decides what happens to an organization that never pays: pushleft-api org suspend <org-id> --reason "trial ended" stops its work (suspension).

GET /v1/orgs/{id}/onboarding returns trial_ends_at, trial_ended and payment_method.

Payment

Pushleft takes payment through Stripe. Payment is off until the API has PUSHLEFT_STRIPE_SECRET_KEY; without it the two billing endpoints answer 404 and the console hides the payment panel.

VariableMeaning
PUSHLEFT_STRIPE_SECRET_KEYA Stripe secret or restricted key. It also needs PUSHLEFT_CONSOLE_URL
PUSHLEFT_STRIPE_WEBHOOK_SECRETThe signing secret (whsec_…) of the webhook endpoint. Without it the webhook answers 404
PUSHLEFT_STRIPE_AUTOMATIC_TAX=onStripe Tax works out the VAT on each invoice

Payment method. An owner selects Add payment method on the console’s Usage page.

  1. POST /v1/orgs/{id}/billing/checkout makes the Stripe customer for the organization (once) and returns the URL of a Stripe Checkout page in setup mode. The page saves a payment method and the billing address, and charges nothing. The Stripe account’s settings decide which methods the page offers.
  2. Stripe sends checkout.session.completed to POST /v1/stripe/webhook. The API makes the saved method the customer’s default and records the time. It does this only when the session’s pushleft_org, the session’s customer and the customer of the setup intent are the same organization.
  3. POST /v1/orgs/{id}/billing/portal returns the URL of Stripe’s page to change or remove the method. Stripe sends payment_method.attached or payment_method.detached, and the API reads the customer’s methods from Stripe. With no method left, the organization has no payment method and the audit log gets payment_method_removed.

Only owners can call the two endpoints. Pushleft stores the Stripe customer ID and the time, never card data.

Webhook. In Stripe, add an endpoint for https://<api>/v1/stripe/webhook with the events checkout.session.completed, payment_method.attached, payment_method.detached, invoice.paid and invoice.payment_failed. The API checks the Stripe-Signature header and refuses a signature older than 5 minutes with 401. It ignores every event without Pushleft’s metadata (pushleft_org, pushleft_statement) and every customer that no organization has. The Stripe account can serve another product too.

Invoices. After a month closes, the operator charges the statements:

pushleft-api billing invoice 2026-11

The command takes each open statement of an organization with a payment method. It makes one Stripe invoice with a line per statement line, finalizes it and charges the default method. A paid statement gets the state paid, the provider stripe and the invoice ID. A failed charge leaves the statement open; Stripe retries it and reports the result through the webhook.

A run that stops can run again. The statement stores the invoice ID when the invoice exists. Before it makes an invoice, the command looks in Stripe for one with the statement’s pushleft_statement. It adds only the lines that are missing, and finalizes the invoice only when its sum is the statement’s. A second run charges an open invoice again and invoices nothing twice. A statement that fails shows failed with the cause, and the other statements go on.

Usage API and console

Any role, viewers included, can read usage with GET /v1/orgs/{id}/usage?period=YYYY-MM. The default period is the current UTC month. The response contains:

  • The quantity and cost per meter, a base_fee line for the counted share of the month, and total_eur.
  • complete_until: the ledger watermark. Usage after it is not counted yet.
  • The budget, with this month’s spend and percentage, and spend_blocked.
  • The trust tier and its limits.
  • plan: the current plan with its base fee, prices and start.
  • The suspension and mail-pause state.

GET /v1/orgs/{id}/statements returns the current plan, the plan history and the statements with their lines, newest first.

The console shows the same data on the Usage page.

Budgets

A budget caps what an organization spends each month. Each organization has one budget with:

  • A monthly amount in EUR.

  • Alert thresholds in percent: 1 to 10 values between 1 and 1000. The default is 50, 80 and 100.

  • A mode:

    • alert_only (default): alerts only.
    • hard_stop: when this UTC month’s ledger spend reaches the amount, org_spend_blocked(org) is true.

While org_spend_blocked is true, each service refuses new work, mostly with 402 budget_exceeded:

  • The API refuses new deployments.
  • The mail service refuses sends (mail_send_allowed).
  • The AI proxy refuses every request, BYOK included.
  • The error ingest refuses new events with 429 and X-Sentry-Rate-Limits: 3600::organization:budget_exceeded (errors.md).

Running applications keep serving. The block ends when the month changes, when the amount is raised or when the mode changes to alert_only.

Owners and admins set the budget with PUT /v1/orgs/{id}/budget and the body {"monthly_amount_eur": 50, "thresholds": [50, 80, 100], "mode": "hard_stop"}. They remove it with DELETE /v1/orgs/{id}/budget. Both write a budget_changed audit event. The console has a budget form on the Usage page.

Alerts. After each hourly rollup, the scheduler checks this month’s spend against each threshold. Each threshold reached raises one budget_alerts row per organization, month and threshold. It also writes a budget_threshold_reached audit event (with spend_blocked).

When platform mail is configured (PUSHLEFT_SMTP_URL), the API claims each alert once and emails the organization’s owners and admins. A failed send is logged and not retried. The alert stays in the audit trail. Alerts older than two days are not mailed, so turning on mail later does not send old alerts.

Lag. Spend comes from finished hours only. A hard stop takes effect up to about 65 minutes after the usage that crossed the amount. The budget never refuses usage in the current hour.

Planned

  • Credit notes, and an automatic invoice run at month close. Stripe makes the legal invoice (number, VAT, PDF); the operator starts the run (payment).
  • Egress as a meter.
  • A charge for a role resource override. The periods are in org_role_resource_overrides (effective_from to effective_until, per organization), ready to join against the ledger hours (abuse.md).
  • Per-environment budgets. Only per-organization budgets exist; the AI proxy adds optional per-environment monthly token caps (ai-proxy.md).
  • A tenant export of the ledger. Use the usage API per month.
  • Datasets are measured as they are when an hour is rolled up: a catch-up after downtime misses a dataset deleted in between.
  • Abuse controls: trust tiers, suspension and mail pause.
  • Retention: how long releases, sources and events are kept.
  • AI proxy: per-environment token caps.
  • Operations: the audit trail that records budget changes.
Rendered from the repository at build time. Edit docs/billing.md to change it.