pushleft
Deploy your first app
Manual contents
Handbook
HandbookPushleftArchitecture
Operate
Install the control planeYour own serversFleetOperations and alertsAuthenticationDomains and DNSUsage and budgetsScheduled jobsAbuse controlsAI provider proxyCustomer mailErrors and performanceLogs and metricsRetentionObject storageContainer registryGit server
Develop
The .pl.yaml manifestOnboard with an AI agentThe pl up demoBuild executorSource ingestionAgent gatewayManaged databasesSecurity findingsTunnels and shellsDev workspacesPrivate hostnamesMesh coordinatorsDesign baseline
Runbooks
Control-plane host failoverControl-plane PostgreSQL failoverControl-plane PostgreSQL restoreCustom domains and certificatesGateway key compromiseIngress host failureNode busyNode disk fullNode drain and replacementNode lostRegistry outageHetzner Robot hostsSecret rotationSecurity incidentStaging fleet on ProxmoxStaging build cacheStuck deploymentVault/OpenBao backup and recovery
Manual · docs/ai-proxy.md

AI provider proxy

The Pushleft AI proxy lets your applications call large language models with your organization’s own provider keys. The application holds only a Pushleft proxy token, and the provider bills you directly. The proxy adds your stored key, checks your limits and counts the tokens.

Console Base URLs per provider, the organization's provider keys and proxy tokens.

Quick start

  1. Store your provider key on the console AI page. An admin can also call PUT /v1/ai/keys/{provider} (Provider keys). The key names are in Providers.

  2. Create a proxy token and store it as an environment secret:

    pl ai token create checkout --production | pl secrets set AI_PROXY_TOKEN --production

    The command prints the token on stdout only once. Without --production, the environment is this checkout’s preview/<branch>.

  3. Give the secret to the application in .pl.yaml (manifest secrets):

    roles:
      web:
        secrets: [{ name: AI_PROXY_TOKEN, as: env }]
  4. Point your SDK at the proxy and use the token as the API key. The console AI page shows the proxy URL, for example https://ai.<base domain>:

    • OpenAI SDKs: base URL https://ai.<base>/v1. Name models <provider>/<model>, for example anthropic/claude-haiku-4-5.
    • Anthropic SDKs: base URL https://ai.<base>/anthropic.
    • Gemini SDKs: base URL https://ai.<base>/gemini.
  5. Deploy:

    pl up --production

Features

  • One OpenAI-compatible endpoint for Anthropic, OpenAI, Gemini, OpenRouter, DeepSeek, Ollama and other OpenAI-compatible services.
  • Native Anthropic Messages and Gemini endpoints, forwarded unchanged.
  • Streaming responses as server-sent events.
  • Model allowlists, request-per-minute limits and a monthly token cap per organization or environment (Limits).
  • A daily request quota of 500, 20 000 or 200 000, by trust tier.
  • Token usage per environment and model on the console AI page (Usage).
  • No prompts, responses or keys in the proxy’s logs.

Metered mode is off. In that mode Pushleft would use its own provider accounts and resell them with a margin. It needs a payment provider (billing).

Providers

ProviderKey name in the APIEndpointFormat
Anthropicanthropichttps://api.anthropic.com (operator setting)Anthropic Messages
OpenAIopenaihttps://api.openai.com/v1OpenAI
Google Geminigeminihttps://generativelanguage.googleapis.comGemini generateContent
OpenRouteropenrouterhttps://openrouter.ai/api/v1OpenAI
DeepSeekdeepseekhttps://api.deepseek.com/v1OpenAI
Ollama (self-hosted)ollamathe customer’s base URL, for example https://ollama.example.com/v1OpenAI
Any OpenAI-compatible serviceopenai_compatiblethe customer’s base URLOpenAI

Ollama and the generic adapter need a base_url with the key: the OpenAI-compatible base, which usually ends in /v1. An Ollama server without authentication ignores the key, so store any placeholder of at least 8 characters.

Endpoints

Applications reach the proxy on a public HTTPS hostname, for example https://ai.<base domain> (Networking).

PathUse
POST /v1/chat/completionsOpenAI chat completions for every provider. The model is <provider>/<model>, for example anthropic/claude-haiku-4-5, gemini/gemini-2.5-flash, openrouter/meta-llama/llama-3.3-70b-instruct. Bare names that start with claude-, gemini-, gpt-, o1, o3 or o4 also work.
GET /v1/modelsThe allowlist when one is set, else the models that each provider with a stored key lists, as <provider>/<model>.
POST /anthropic/v1/messagesAnthropic Messages, forwarded unchanged (tools, thinking, anthropic-beta, anthropic-workspace-id and anthropic-version headers included).
POST /gemini/v1beta/models/{model}:generateContent and :streamGenerateContentGemini, forwarded unchanged. Streams always use server-sent events (alt=sse).

SDK settings:

  • OpenAI SDKs: base URL https://ai.<base>/v1, API key = the proxy token.
  • Anthropic SDKs: base URL https://ai.<base>/anthropic, API key = the proxy token (sent as x-api-key).
  • Gemini SDKs: base URL https://ai.<base>/gemini, API key = the proxy token (sent as x-goog-api-key).

The proxy accepts the token as Authorization: Bearer, x-api-key or x-goog-api-key. It never accepts a token in the URL.

Translation. /v1/chat/completions translates OpenAI requests to Anthropic Messages and Gemini generateContent. It translates the responses and stream chunks back. It translates:

  • text;
  • images as data: URLs (Anthropic also takes https: image URLs);
  • system and developer messages;
  • max_tokens and max_completion_tokens (Anthropic default 4096);
  • temperature, top_p, stop and stream.

For these two providers, it refuses the legacy functions fields and n > 1 with 400. It translates tools, tool_choice, parallel_tool_calls (Anthropic only), assistant tool_calls and tool messages, streamed and not streamed. Gemini returns a thought signature with a function call. The proxy puts it in tool_calls[].extra_content.google.thought_signature. Send that field back unchanged in the next request, or Gemini can refuse the request. Requests to OpenAI-format providers pass through unchanged, except for the model name.

Streaming. Responses stream as server-sent events. The proxy reads the usage from the stream:

  • OpenAI: the last chunk. The proxy sets stream_options.include_usage, and removes that chunk again when the client did not ask for it.
  • Anthropic: message_start and message_delta.
  • Gemini: usageMetadata.

A translated stream ends with data: [DONE] and, when the client asked for it, a usage chunk.

Tokens

A proxy token belongs to one environment of one organization. Its format is plai_<prefix>_<secret>. The database keeps only the SHA-256 of the secret, as for API tokens. Pushleft shows the secret once.

pl ai token create checkout --project shop --environment production
pl ai token list
pl ai token rotate <id> --overlap-minutes 10    # the old secret works for 10 minutes (0 = now)
pl ai token revoke <id>
APIRole
POST /v1/ai/tokens {"project", "environment", "name"}member
POST /v1/ai/tokens/{id}/rotate {"overlap_minutes": 10}member
DELETE /v1/ai/tokens/{id}member
GET /v1/ai: tokens, keys, policies, this month’s usage, the proxy URLviewer

The console AI page has the same functions. Every change writes an audit event: ai_token_created, ai_token_rotated or ai_token_revoked.

A project cannot be deleted while it has a proxy token that still works or an environment provider key. Revoke or remove them first. A deleted project’s environments take no new tokens, keys or policies (404).

Provider keys

An admin stores a key with PUT /v1/ai/keys/{provider} and {"key": "...", "project": "shop", "environment": "production", "base_url": "..."}.

  • Without project and environment, the key is for the whole organization.
  • An environment key wins over the organization key.
  • A new key for the same provider and scope replaces the old one.
  • DELETE /v1/ai/keys/{id} removes a key. The proxy stops using it at once.

How Pushleft keeps the key:

  • The API writes the key to Vault at organizations/<org>/ai/<provider>/<key id> (KV v2 mount organizations, field key), create-only. Each new key gets a new path, so a write can never replace another key.
  • PostgreSQL keeps the Vault reference, the provider, the base URL and the last 4 characters (ai_provider_keys). The API, the console and the CLI show only the last 4 characters.
  • The proxy reads keys from Vault with a read-only policy. It keeps each key in memory for at most 60 seconds.
  • A provider’s 401 or 403 becomes 502 provider_auth_failed without the provider’s message, because providers quote part of the key in it. Other provider errors pass through, with the key removed if it appears.

Gap. A removed or replaced key stays in Vault. The Vault client has no delete, and the API policy has no delete capability. Nothing reads the key after removal. If a customer asks, destroy the old versions by hand with an operator token.

Limits

The proxy calls ai_proxy_admit in PostgreSQL before every request. It refuses, in this order:

ReasonStatusRule
invalid_api_key401unknown, revoked or expired token
org_suspended403the organization is suspended (abuse controls)
budget_exceeded402a hard-stop budget is spent (org_spend_blocked, budgets)
model_not_allowed403the model is not on the organization or environment allowlist
no_provider_key400no key for this provider for the environment or the organization
rate_limited429 + Retry-Afterrequests per minute: per organization (PUSHLEFT_AI_ORG_REQUESTS_PER_MINUTE, default 600) and per environment (PUSHLEFT_AI_ENVIRONMENT_REQUESTS_PER_MINUTE, default 300); a policy can only lower them
daily_quota429 + Retry-Afterthe tier’s ai_requests_per_day (new 500, standard 20 000, trusted 200 000), per UTC day
token_cap429the policy’s monthly token cap (input plus output, UTC month)

The rate counters are the shared fixed windows of pushleft_rate_limit_hit, so all proxy replicas share them. The daily quota is a fixed UTC-day window, not a rolling 24 hours.

Policies. An admin sets the organization’s policy with PUT /v1/ai/policies and {"allowed_models": ["anthropic/claude-haiku-4-5"], "monthly_token_cap": 5000000, "requests_per_minute": 60}.

  • Add project and environment to set the policy for one environment.
  • Both policies apply.
  • A body with no limits removes the policy.
  • Changes write ai_policy_changed.

Request limits.

  • Bodies over PUSHLEFT_AI_MAX_REQUEST_BYTES (default 8 MiB) get 413.
  • The proxy reads responses that are not streamed up to 32 MiB.
  • Connections to providers time out after 10 seconds.
  • A read that waits longer than PUSHLEFT_AI_UPSTREAM_READ_TIMEOUT_SECONDS (default 120) ends the request.
  • The proxy does not follow redirects.

Usage

After every request, the proxy calls ai_proxy_record:

  • It records ai_tokens_in and ai_tokens_out with record_usage(). The idempotency keys are ai:<request id>:in and :out.
  • It adds the request to ai_usage_daily, per environment, UTC day, provider and model.
  • Input tokens include cached prompt tokens. Output tokens include reasoning tokens.
  • When a client disconnects during a stream, the proxy records the usage it saw until then.

Pushleft meters usage with your own keys (BYOK) for visibility. They cost 0: no plan prices ai_tokens_in or ai_tokens_out. The usage API and the console Usage page show both meters. The console AI page shows the month per environment and model. Every response has an x-pushleft-request-id header.

Logs. The proxy logs metadata only: request id, organization, environment, provider, model, stream flag, status, token counts and duration. It never logs or stores prompts, responses or keys.

Networking

Application containers cannot reach private address ranges or the host (architecture). Public IPv4 egress is allowed. The proxy must therefore be reachable on a public HTTPS endpoint, for example ai.<base domain>. It runs behind the control plane’s TLS proxy, in the same way as the API. Set PUSHLEFT_AI_PROXY_URL on the API to that URL, so that the console and GET /v1/ai show it.

SSRF guard. The proxy runs next to platform services, so customer base URLs (Ollama, generic) must not reach them. The proxy refuses a base URL that is an IP literal or a name that resolves to a refused address (400 upstream_refused). The API shares this filter (pushleft_core::ssrf), and also uses it for OIDC issuers, git provider URLs and registry upstreams. It refuses:

IPv4IPv6
0.0.0.0/8 (this network), 10/8, 172.16/12, 192.168/16 (private), 127/8 (loopback), 169.254/16 (link-local), 100.64/10 (shared, CGNAT), 192.0.0/24 (IETF protocol assignments), 192.0.2/24, 198.51.100/24, 203.0.113/24 (documentation), 198.18/15 (benchmarking), 224/4 (multicast), 240/4 (reserved, including 255.255.255.255):: and ::1, ::/96 (IPv4-compatible), ::ffff:0:0/96 (IPv4-mapped: the embedded address is checked), fc00::/7 (unique local), fe80::/10 (link-local), fec0::/10 (site-local), ff00::/8 (multicast), 2001:db8::/32 (documentation), 2001::/32 (Teredo), 2002::/16 (6to4: refused when the embedded IPv4 is), 64:ff9b::/32 (NAT64, including local-use 64:ff9b:1::/48), 100::/64 (discard)

The check runs in the resolver of the HTTP client, so the address that is checked is the address that is used. PUSHLEFT_AI_ALLOW_PRIVATE_UPSTREAMS=1 turns the check off. Use it for a self-hosted platform whose Ollama runs on a private network.

Getting the token into the application. Store the token as an environment secret, with pl ai token create … | pl secrets set AI_PROXY_TOKEN or the console. List it under the role’s secrets: in .pl.yaml. The application reads /run/secrets/AI_PROXY_TOKEN, or the variable with { name: AI_PROXY_TOKEN, as: env } (manifest secrets).

Operation

The proxy is pushleft-ai-proxy. Its unit is deploy/systemd/pushleft-ai-proxy.service, and its settings are in deploy/systemd/ai-proxy.env.example. GET /healthz returns 204. GET /readyz checks the database as the API does. On SIGTERM, the proxy stops accepting connections and gives open streams 30 seconds.

  • Database role. pushleft_ai_proxy, LOGIN NOBYPASSRLS, with no table grants. It can only execute ai_proxy_admit, ai_proxy_models and ai_proxy_record (deploy/README.md). The API needs DELETE on ai_provider_keys and ai_policies.
  • Vault. The proxy’s token carries deploy/vault/pushleft-ai-proxy.hcl: read on organizations/data/+/ai/*. The API’s token carries deploy/vault/pushleft-api.hcl: create only, on the same paths. Without PUSHLEFT_VAULT_ADDR on the API, storing a key answers 503.

Tests

  • cargo test -p pushleft-ai-proxy tests the translation of requests, responses and stream chunks between the three formats with usage. It also tests the SSE parser, token parsing and the headers each SDK uses. cargo test -p pushleft-core ssrf tests the SSRF filter (addresses, URLs, resolver).

  • npm run test:ai-proxy (scripts/ai-proxy-e2e.sh) runs the real API and proxy binaries. It uses PostgreSQL, a fake Vault and fake OpenAI, Anthropic and Gemini servers (scripts/fixtures/ai-fakes.py). The proxy runs as a role that has only its three functions. The test proves:

    • keys are stored and never returned, injected and never leaked to clients or logs;
    • passthrough and translation with streaming, and usage per request;
    • the per-environment rate limit, the allowlist, the hard-stop budget and suspension;
    • cross-organization isolation, rotation, revocation and pl ai token;
    • the SSRF guard and the size limit.

    With PUSHLEFT_AI_LIVE_ANTHROPIC_KEY_FILE, it also sends four small requests to the real Anthropic API.

Planned

  • Metered mode (platform provider accounts and resale), which waits for a payment provider.
  • Gemini fileData images, embeddings, the OpenAI Responses API and audio or image endpoints.
  • Usage events in ClickHouse. Usage is in the PostgreSQL ledger.
  • Deleting old key versions from Vault.
  • Prices per provider or model. The ledger has one price per meter.
Rendered from the repository at build time. Edit docs/ai-proxy.md to change it.