pushleft
Deploy your first app
Manual contents
Handbook
HandbookPushleftArchitecture
Operate
Install the control planeYour own serversFleetOperations and alertsAuthenticationDomains and DNSUsage and budgetsScheduled jobsAbuse controlsAI provider proxyCustomer mailErrors and performanceLogs and metricsRetentionObject storageContainer registryGit server
Develop
The .pl.yaml manifestOnboard with an AI agentThe pl up demoBuild executorSource ingestionAgent gatewayManaged databasesSecurity findingsTunnels and shellsDev workspacesPrivate hostnamesMesh coordinatorsDesign baseline
Runbooks
Control-plane host failoverControl-plane PostgreSQL failoverControl-plane PostgreSQL restoreCustom domains and certificatesGateway key compromiseIngress host failureNode busyNode disk fullNode drain and replacementNode lostRegistry outageHetzner Robot hostsSecret rotationSecurity incidentStaging fleet on ProxmoxStaging build cacheStuck deploymentVault/OpenBao backup and recovery
Manual · docs/runbooks/node-capacity-high.md

Node busy

Use this when node_cpu_busy or node_memory_high fires. A node of ours used 90% or more of its CPU, or of its memory, in each of the last 5 minutes. Requests on that node slow down. At full memory, the kernel stops containers.

node_capacity_high is a ticket for reservations: a node has over 90% of its CPU or memory reserved for 20 minutes. Nodes are overcommitted on purpose, so nothing is slow. New placements there may queue. Add a node when the pool stays reserved like this, as in Fix it.

Find the cause

  1. Open the alert. It names the node and the lowest share in those 5 minutes.
  2. Open the node in the Fleet app. Check its trend and which containers use the most.
  3. A single role that suddenly uses much more than before is usually the cause. Check that role’s logs and its deploys.

Fix it

If one tenant’s role causes it, contact the tenant, or lower that role’s limits.

If the node is busy from normal work, move an environment to a node with room:

  1. Set the target node:

    pushleft-controller fleet place-environment <project> <environment> --node <node>
  2. Redeploy the environment with pl up on its branch. The command in step 1 moves nothing until the redeploy.

  3. If the environment has volumes, follow moving an environment with volumes.

  4. When the node is back under 90%, remove the preference with --clear if the environment does not need to stay there.

If every node in the pool is busy, add a node. Select Enroll host or Provision on the Nodes page of the Fleet app.

Check it worked

The alert clears at the next evaluation after one minute under 90%.

Rendered from the repository at build time. Edit docs/runbooks/node-capacity-high.md to change it.