Node disk full
Use this when node_data_disk_low, node_disk_high, node_disk_full, host_disk_high or host_disk_full fires. A filesystem is over 80% full, or the data disk has under 10 GB free. At 95%, backups, WAL archiving and the agent stop working.
Find the filesystem
- Open the alert. It names the node and the filesystem:
/, tenant data or backups. - Open the node in the Nodes page of the Fleet app. Compare the disks in Disk health.
Fix it
If the tenant data disk is full, move volumes to another node:
pushleft-controller fleet migrate-volumes <node>
If the whole pool is full, add a node. Select Enroll host or Provision on the Nodes page.
If the backups disk or / is full, log in to the node and find the largest directories. Check first that the off-host copy is current on the Backups page. Remove only files that you can identify.
Check it worked
The alert clears at the next evaluation after the filesystem drops under its limit.