Node busy
Use this when node_cpu_busy or node_memory_high fires. A node of ours used 90% or more of its CPU, or of its memory, in each of the last 5 minutes. Requests on that node slow down. At full memory, the kernel stops containers.
node_capacity_high is a ticket for reservations: a node has over 90% of its CPU or memory reserved for 20 minutes. Nodes are overcommitted on purpose, so nothing is slow. New placements there may queue. Add a node when the pool stays reserved like this, as in Fix it.
Find the cause
- Open the alert. It names the node and the lowest share in those 5 minutes.
- Open the node in the Fleet app. Check its trend and which containers use the most.
- A single role that suddenly uses much more than before is usually the cause. Check that role’s logs and its deploys.
Fix it
If one tenant’s role causes it, contact the tenant, or lower that role’s limits.
If the node is busy from normal work, move an environment to a node with room:
-
Set the target node:
pushleft-controller fleet place-environment <project> <environment> --node <node> -
Redeploy the environment with
pl upon its branch. The command in step 1 moves nothing until the redeploy. -
If the environment has volumes, follow moving an environment with volumes.
-
When the node is back under 90%, remove the preference with
--clearif the environment does not need to stay there.
If every node in the pool is busy, add a node. Select Enroll host or Provision on the Nodes page of the Fleet app.
Check it worked
The alert clears at the next evaluation after one minute under 90%.