AI provider proxy
The Pushleft AI proxy lets your applications call large language models with your organization’s own provider keys. The application holds only a Pushleft proxy token, and the provider bills you directly. The proxy adds your stored key, checks your limits and counts the tokens.
Quick start
-
Store your provider key on the console AI page. An admin can also call
PUT /v1/ai/keys/{provider}(Provider keys). The key names are in Providers. -
Create a proxy token and store it as an environment secret:
pl ai token create checkout --production | pl secrets set AI_PROXY_TOKEN --productionThe command prints the token on stdout only once. Without
--production, the environment is this checkout’spreview/<branch>. -
Give the secret to the application in
.pl.yaml(manifest secrets):roles: web: secrets: [{ name: AI_PROXY_TOKEN, as: env }] -
Point your SDK at the proxy and use the token as the API key. The console AI page shows the proxy URL, for example
https://ai.<base domain>:- OpenAI SDKs: base URL
https://ai.<base>/v1. Name models<provider>/<model>, for exampleanthropic/claude-haiku-4-5. - Anthropic SDKs: base URL
https://ai.<base>/anthropic. - Gemini SDKs: base URL
https://ai.<base>/gemini.
- OpenAI SDKs: base URL
-
Deploy:
pl up --production
Features
- One OpenAI-compatible endpoint for Anthropic, OpenAI, Gemini, OpenRouter, DeepSeek, Ollama and other OpenAI-compatible services.
- Native Anthropic Messages and Gemini endpoints, forwarded unchanged.
- Streaming responses as server-sent events.
- Model allowlists, request-per-minute limits and a monthly token cap per organization or environment (Limits).
- A daily request quota of 500, 20 000 or 200 000, by trust tier.
- Token usage per environment and model on the console AI page (Usage).
- No prompts, responses or keys in the proxy’s logs.
Metered mode is off. In that mode Pushleft would use its own provider accounts and resell them with a margin. It needs a payment provider (billing).
Providers
| Provider | Key name in the API | Endpoint | Format |
|---|---|---|---|
| Anthropic | anthropic | https://api.anthropic.com (operator setting) | Anthropic Messages |
| OpenAI | openai | https://api.openai.com/v1 | OpenAI |
| Google Gemini | gemini | https://generativelanguage.googleapis.com | Gemini generateContent |
| OpenRouter | openrouter | https://openrouter.ai/api/v1 | OpenAI |
| DeepSeek | deepseek | https://api.deepseek.com/v1 | OpenAI |
| Ollama (self-hosted) | ollama | the customer’s base URL, for example https://ollama.example.com/v1 | OpenAI |
| Any OpenAI-compatible service | openai_compatible | the customer’s base URL | OpenAI |
Ollama and the generic adapter need a base_url with the key: the OpenAI-compatible base, which usually ends in /v1. An Ollama server without authentication ignores the key, so store any placeholder of at least 8 characters.
Endpoints
Applications reach the proxy on a public HTTPS hostname, for example https://ai.<base domain> (Networking).
| Path | Use |
|---|---|
POST /v1/chat/completions | OpenAI chat completions for every provider. The model is <provider>/<model>, for example anthropic/claude-haiku-4-5, gemini/gemini-2.5-flash, openrouter/meta-llama/llama-3.3-70b-instruct. Bare names that start with claude-, gemini-, gpt-, o1, o3 or o4 also work. |
GET /v1/models | The allowlist when one is set, else the models that each provider with a stored key lists, as <provider>/<model>. |
POST /anthropic/v1/messages | Anthropic Messages, forwarded unchanged (tools, thinking, anthropic-beta, anthropic-workspace-id and anthropic-version headers included). |
POST /gemini/v1beta/models/{model}:generateContent and :streamGenerateContent | Gemini, forwarded unchanged. Streams always use server-sent events (alt=sse). |
SDK settings:
- OpenAI SDKs: base URL
https://ai.<base>/v1, API key = the proxy token. - Anthropic SDKs: base URL
https://ai.<base>/anthropic, API key = the proxy token (sent asx-api-key). - Gemini SDKs: base URL
https://ai.<base>/gemini, API key = the proxy token (sent asx-goog-api-key).
The proxy accepts the token as Authorization: Bearer, x-api-key or x-goog-api-key. It never accepts a token in the URL.
Translation. /v1/chat/completions translates OpenAI requests to Anthropic Messages and Gemini generateContent. It translates the responses and stream chunks back. It translates:
- text;
- images as
data:URLs (Anthropic also takeshttps:image URLs); systemanddevelopermessages;max_tokensandmax_completion_tokens(Anthropic default 4096);temperature,top_p,stopandstream.
For these two providers, it refuses the legacy functions fields and n > 1 with 400. It translates tools, tool_choice, parallel_tool_calls (Anthropic only), assistant tool_calls and tool messages, streamed and not streamed. Gemini returns a thought signature with a function call. The proxy puts it in tool_calls[].extra_content.google.thought_signature. Send that field back unchanged in the next request, or Gemini can refuse the request. Requests to OpenAI-format providers pass through unchanged, except for the model name.
Streaming. Responses stream as server-sent events. The proxy reads the usage from the stream:
- OpenAI: the last chunk. The proxy sets
stream_options.include_usage, and removes that chunk again when the client did not ask for it. - Anthropic:
message_startandmessage_delta. - Gemini:
usageMetadata.
A translated stream ends with data: [DONE] and, when the client asked for it, a usage chunk.
Tokens
A proxy token belongs to one environment of one organization. Its format is plai_<prefix>_<secret>. The database keeps only the SHA-256 of the secret, as for API tokens. Pushleft shows the secret once.
pl ai token create checkout --project shop --environment production
pl ai token list
pl ai token rotate <id> --overlap-minutes 10 # the old secret works for 10 minutes (0 = now)
pl ai token revoke <id>
| API | Role |
|---|---|
POST /v1/ai/tokens {"project", "environment", "name"} | member |
POST /v1/ai/tokens/{id}/rotate {"overlap_minutes": 10} | member |
DELETE /v1/ai/tokens/{id} | member |
GET /v1/ai: tokens, keys, policies, this month’s usage, the proxy URL | viewer |
The console AI page has the same functions. Every change writes an audit event: ai_token_created, ai_token_rotated or ai_token_revoked.
A project cannot be deleted while it has a proxy token that still works or an environment provider key. Revoke or remove them first. A deleted project’s environments take no new tokens, keys or policies (404).
Provider keys
An admin stores a key with PUT /v1/ai/keys/{provider} and {"key": "...", "project": "shop", "environment": "production", "base_url": "..."}.
- Without
projectandenvironment, the key is for the whole organization. - An environment key wins over the organization key.
- A new key for the same provider and scope replaces the old one.
DELETE /v1/ai/keys/{id}removes a key. The proxy stops using it at once.
How Pushleft keeps the key:
- The API writes the key to Vault at
organizations/<org>/ai/<provider>/<key id>(KV v2 mountorganizations, fieldkey), create-only. Each new key gets a new path, so a write can never replace another key. - PostgreSQL keeps the Vault reference, the provider, the base URL and the last 4 characters (
ai_provider_keys). The API, the console and the CLI show only the last 4 characters. - The proxy reads keys from Vault with a read-only policy. It keeps each key in memory for at most 60 seconds.
- A provider’s
401or403becomes502 provider_auth_failedwithout the provider’s message, because providers quote part of the key in it. Other provider errors pass through, with the key removed if it appears.
Gap. A removed or replaced key stays in Vault. The Vault client has no delete, and the API policy has no delete capability. Nothing reads the key after removal. If a customer asks, destroy the old versions by hand with an operator token.
Limits
The proxy calls ai_proxy_admit in PostgreSQL before every request. It refuses, in this order:
| Reason | Status | Rule |
|---|---|---|
invalid_api_key | 401 | unknown, revoked or expired token |
org_suspended | 403 | the organization is suspended (abuse controls) |
budget_exceeded | 402 | a hard-stop budget is spent (org_spend_blocked, budgets) |
model_not_allowed | 403 | the model is not on the organization or environment allowlist |
no_provider_key | 400 | no key for this provider for the environment or the organization |
rate_limited | 429 + Retry-After | requests per minute: per organization (PUSHLEFT_AI_ORG_REQUESTS_PER_MINUTE, default 600) and per environment (PUSHLEFT_AI_ENVIRONMENT_REQUESTS_PER_MINUTE, default 300); a policy can only lower them |
daily_quota | 429 + Retry-After | the tier’s ai_requests_per_day (new 500, standard 20 000, trusted 200 000), per UTC day |
token_cap | 429 | the policy’s monthly token cap (input plus output, UTC month) |
The rate counters are the shared fixed windows of pushleft_rate_limit_hit, so all proxy replicas share them. The daily quota is a fixed UTC-day window, not a rolling 24 hours.
Policies. An admin sets the organization’s policy with PUT /v1/ai/policies and {"allowed_models": ["anthropic/claude-haiku-4-5"], "monthly_token_cap": 5000000, "requests_per_minute": 60}.
- Add
projectandenvironmentto set the policy for one environment. - Both policies apply.
- A body with no limits removes the policy.
- Changes write
ai_policy_changed.
Request limits.
- Bodies over
PUSHLEFT_AI_MAX_REQUEST_BYTES(default 8 MiB) get413. - The proxy reads responses that are not streamed up to 32 MiB.
- Connections to providers time out after 10 seconds.
- A read that waits longer than
PUSHLEFT_AI_UPSTREAM_READ_TIMEOUT_SECONDS(default 120) ends the request. - The proxy does not follow redirects.
Usage
After every request, the proxy calls ai_proxy_record:
- It records
ai_tokens_inandai_tokens_outwithrecord_usage(). The idempotency keys areai:<request id>:inand:out. - It adds the request to
ai_usage_daily, per environment, UTC day, provider and model. - Input tokens include cached prompt tokens. Output tokens include reasoning tokens.
- When a client disconnects during a stream, the proxy records the usage it saw until then.
Pushleft meters usage with your own keys (BYOK) for visibility. They cost 0: no plan prices ai_tokens_in or ai_tokens_out. The usage API and the console Usage page show both meters. The console AI page shows the month per environment and model. Every response has an x-pushleft-request-id header.
Logs. The proxy logs metadata only: request id, organization, environment, provider, model, stream flag, status, token counts and duration. It never logs or stores prompts, responses or keys.
Networking
Application containers cannot reach private address ranges or the host (architecture). Public IPv4 egress is allowed. The proxy must therefore be reachable on a public HTTPS endpoint, for example ai.<base domain>. It runs behind the control plane’s TLS proxy, in the same way as the API. Set PUSHLEFT_AI_PROXY_URL on the API to that URL, so that the console and GET /v1/ai show it.
SSRF guard. The proxy runs next to platform services, so customer base URLs (Ollama, generic) must not reach them. The proxy refuses a base URL that is an IP literal or a name that resolves to a refused address (400 upstream_refused). The API shares this filter (pushleft_core::ssrf), and also uses it for OIDC issuers, git provider URLs and registry upstreams. It refuses:
| IPv4 | IPv6 |
|---|---|
0.0.0.0/8 (this network), 10/8, 172.16/12, 192.168/16 (private), 127/8 (loopback), 169.254/16 (link-local), 100.64/10 (shared, CGNAT), 192.0.0/24 (IETF protocol assignments), 192.0.2/24, 198.51.100/24, 203.0.113/24 (documentation), 198.18/15 (benchmarking), 224/4 (multicast), 240/4 (reserved, including 255.255.255.255) | :: and ::1, ::/96 (IPv4-compatible), ::ffff:0:0/96 (IPv4-mapped: the embedded address is checked), fc00::/7 (unique local), fe80::/10 (link-local), fec0::/10 (site-local), ff00::/8 (multicast), 2001:db8::/32 (documentation), 2001::/32 (Teredo), 2002::/16 (6to4: refused when the embedded IPv4 is), 64:ff9b::/32 (NAT64, including local-use 64:ff9b:1::/48), 100::/64 (discard) |
The check runs in the resolver of the HTTP client, so the address that is checked is the address that is used. PUSHLEFT_AI_ALLOW_PRIVATE_UPSTREAMS=1 turns the check off. Use it for a self-hosted platform whose Ollama runs on a private network.
Getting the token into the application. Store the token as an environment secret, with pl ai token create … | pl secrets set AI_PROXY_TOKEN or the console. List it under the role’s secrets: in .pl.yaml. The application reads /run/secrets/AI_PROXY_TOKEN, or the variable with { name: AI_PROXY_TOKEN, as: env } (manifest secrets).
Operation
The proxy is pushleft-ai-proxy. Its unit is deploy/systemd/pushleft-ai-proxy.service, and its settings are in deploy/systemd/ai-proxy.env.example. GET /healthz returns 204. GET /readyz checks the database as the API does. On SIGTERM, the proxy stops accepting connections and gives open streams 30 seconds.
- Database role.
pushleft_ai_proxy,LOGIN NOBYPASSRLS, with no table grants. It can only executeai_proxy_admit,ai_proxy_modelsandai_proxy_record(deploy/README.md). The API needsDELETEonai_provider_keysandai_policies. - Vault. The proxy’s token carries
deploy/vault/pushleft-ai-proxy.hcl: read onorganizations/data/+/ai/*. The API’s token carriesdeploy/vault/pushleft-api.hcl: create only, on the same paths. WithoutPUSHLEFT_VAULT_ADDRon the API, storing a key answers503.
Tests
-
cargo test -p pushleft-ai-proxytests the translation of requests, responses and stream chunks between the three formats with usage. It also tests the SSE parser, token parsing and the headers each SDK uses.cargo test -p pushleft-core ssrftests the SSRF filter (addresses, URLs, resolver). -
npm run test:ai-proxy(scripts/ai-proxy-e2e.sh) runs the real API and proxy binaries. It uses PostgreSQL, a fake Vault and fake OpenAI, Anthropic and Gemini servers (scripts/fixtures/ai-fakes.py). The proxy runs as a role that has only its three functions. The test proves:- keys are stored and never returned, injected and never leaked to clients or logs;
- passthrough and translation with streaming, and usage per request;
- the per-environment rate limit, the allowlist, the hard-stop budget and suspension;
- cross-organization isolation, rotation, revocation and
pl ai token; - the SSRF guard and the size limit.
With
PUSHLEFT_AI_LIVE_ANTHROPIC_KEY_FILE, it also sends four small requests to the real Anthropic API.
Planned
- Metered mode (platform provider accounts and resale), which waits for a payment provider.
- Gemini
fileDataimages, embeddings, the OpenAI Responses API and audio or image endpoints. - Usage events in ClickHouse. Usage is in the PostgreSQL ledger.
- Deleting old key versions from Vault.
- Prices per provider or model. The ledger has one price per meter.
Related
- The manifest: secrets: pass
AI_PROXY_TOKENto a role. - Billing: meters, prices and budgets.
- Abuse controls: trust tiers and suspension.
- Authentication: roles and API tokens.