pushleft
Deploy your first app
Manual contents
Handbook
HandbookPushleftArchitecture
Operate
Install the control planeYour own serversFleetOperations and alertsAuthenticationDomains and DNSUsage and budgetsScheduled jobsAbuse controlsAI provider proxyCustomer mailErrors and performanceLogs and metricsRetentionObject storageContainer registryGit server
Develop
The .pl.yaml manifestOnboard with an AI agentThe pl up demoBuild executorSource ingestionAgent gatewayManaged databasesSecurity findingsTunnels and shellsDev workspacesPrivate hostnamesMesh coordinatorsDesign baseline
Runbooks
Control-plane host failoverControl-plane PostgreSQL failoverControl-plane PostgreSQL restoreCustom domains and certificatesGateway key compromiseIngress host failureNode busyNode disk fullNode drain and replacementNode lostRegistry outageHetzner Robot hostsSecret rotationSecurity incidentStaging fleet on ProxmoxStaging build cacheStuck deploymentVault/OpenBao backup and recovery
Manual · docs/private-hostnames.md

Private hostnames

Built and passing on staging (2026-11-01, staging.sh names). Decided the same day: the agent answers the names, and containers take the new DNS setting at their app’s next deploy. A role reaches another role of the same environment by a stable private name, for example http://api.billing.internal:3311, without a public hostname. First user: an app whose web role calls a private second role.

Role names

  • Every long-running role (web, private, worker) of a deployment is resolvable inside its environment as <role>.<project>.internal. <project> is the project’s hostname_label. The label is fixed when the project is created, and its preview and production hostnames use the same one. Renaming the project’s slug does not change the private name. Two projects can use the same role names without a clash.
  • The name answers with the addresses of the role’s ready instances of the environment’s serving release (the live deployment). During a rollout, the replacement rule of ingress applies: an old instance stays in the answer only until its successor is ready. A superseded release’s instances, ready or draining, are never in it. The exception is a release being rolled out: its own containers resolve its own roles first. Before promotion, and on the first deployment too, a web role’s health check that calls the new release’s private role reaches the new instances. When the new release has no ready instance of that role yet, the name falls back to the serving release. Every other container keeps resolving the serving release until promotion. The answer holds instances on every node of the environment, one A record each. The order rotates on every answer (most clients take the first). The TTL is 5 seconds.
  • Only the environment’s own containers can resolve it. Another environment, another organization and the internet never see it.
  • Each role also gets PUSHLEFT_ROLE_<ROLE>_HOST=<role>.<project>.internal for the long-running roles of its own deployment, so an app can build its URL without hard-coding the name. <ROLE> is the role name upper-cased with - turned into _ (api-worker → PUSHLEFT_ROLE_API_WORKER_HOST); role names never contain _, so the mapping is reversible.
  • The names resolve only inside the environment. From your own machine, forward a role’s port with pl tunnel <role> <port> (developer access).
  • Nothing to declare in .pl.yaml. A long-running role without port is resolvable too; the caller knows the port. release, job and cron roles get no name: they run once or on a schedule, so there is nothing to call.

Internals

What exists today, and what this uses:

  • Containers of one environment already reach each other across nodes by IP. Each node has its own /24 on the environment bridge, and the mesh gateways route between them. Applications reach their database on another node this way.
  • The agent on the node chooses an application instance’s address: a fixed address in .3–.119 (.120–.127 are kept for managed databases). No central record holds it.
  • Containers use Docker’s embedded DNS (127.0.0.11), which knows only containers on its own node and forwards other names to the host’s resolvers.
  • A container cannot open a connection to its host on any port (the environment firewall’s input chain).

The design adds four things:

  1. Recorded addresses. ensure_application reports the instance’s address in its result; the gateway stores it in application_instances.address (migration). A replacement instance reports its own.

  2. A record set per environment. The scheduler computes, per environment, <role>.<hostname_label>.internal → [addresses of the serving release's ready instances]. It applies the members rule that ingress already applies when a release replaces another. It sends the set to every node of the environment with a sync_environment_names job. The job carries the set’s generation, raised on every change. The agent applies a set only if its generation is newer than the one it holds. A retried or late job therefore never brings back a replaced instance’s address. The scheduler sends the job whenever an instance becomes ready or goes away, and every 60 seconds as a check. The mesh controller re-sends its network jobs every 30 seconds in the same way.

  3. A resolver on each node, per environment bridge. The agent answers DNS on the bridge’s gateway address (.1, port 53, UDP and TCP). It runs one listener per environment bridge, so a listener answers only for its own environment:

    • names under .internal from that environment’s record set, else NXDOMAIN. The scheduler sends one set per release in the environment (the serving one and any being rolled out). The listener finds the asking container by its source address on the bridge (Docker’s embedded DNS forwards from the container’s own namespace). It finds the container’s release from its labels on this node. It answers from that release’s set first, then from the serving release’s;
    • any other name forwarded to the host’s resolvers (as Docker’s embedded DNS does today).

    Containers are started with --dns <.1> and use that resolver alone. A fallback to the host’s resolver would carry .internal queries out of the environment when the agent is slow to answer. While the agent restarts (seconds), a container’s lookups wait. Its upstream UDP queries go out on a socket connected to the chosen resolver, and a reply must carry the query’s ID. Docker’s embedded DNS stays in front: it still resolves local container names and forwards the rest to .1. The records survive an agent restart (written beside the route fences). They go when the environment’s network leaves the node. At the next start, the agent drops a stored set that was not re-sent for a day (the scheduler re-sends live ones every minute).

  4. One firewall opening. The environment’s input chain accepts new UDP and TCP to port 53 on the bridge’s gateway address, from that bridge only, rate-limited per environment. Every other host port stays closed.

Costs and limits

  • Rollout recreates containers. --dns is part of a container’s spec, so each application container is recreated once, at its next deploy (not by the agent upgrade itself).
  • Freshness. A replaced instance’s name changes within the job’s delivery time plus the 5-second TTL. A client that caches DNS longer (some JVMs) can keep calling the old address for a while.
  • Same environment only. A role in another environment (another project’s production) is not reachable this way; the mesh does not route between environments. Cross-environment private calls (for example a shop calling a separate billing project) are a separate decision.
  • No load balancing or health beyond readiness. Round robin over ready instances; a client retries another address itself.

Alternatives considered

  • Docker network aliases per role: Docker resolves them only on the same node; roles on another node are invisible.
  • Addresses as environment variables at deploy time (as DATABASE_URL carries the database’s address): they work for one fixed instance. Replicas and replacements would need a redeploy of every caller.
  • A resolver container per environment on the bridge: it opens no host port, but adds one more container and image per environment per node to pin, scan and run.
  • Headscale/MagicDNS: the containers are not tailnet nodes; the mesh gateway is a subnet router.

Tests

  • Unit: record sets (ready long-running roles only, project-qualified names, one serving set); task validation; the resolver’s own-release-first lookup; the DNS answer, NXDOMAIN and malformed-query refusal; UDP truncation (TC) above the client’s size.
  • Staging (staging.sh names, scripts/fixtures/names-app, passed 2026-11-01): web resolves api.<label>.internal to both replicas and reaches both. The db node (a second member) answers the same. A new release moves the name to its new instances’ addresses with none of the old left. Another environment (web-only) gets NXDOMAIN, and its bridge gateway answers on 53 and refuses 22, 2019, 8080 and 9100. Replicas on two app nodes are untested (staging has one app node).
Rendered from the repository at build time. Edit docs/private-hostnames.md to change it.