pushleft
Deploy your first app
Manual contents
Handbook
HandbookPushleftArchitecture
Operate
Install the control planeYour own serversFleetOperations and alertsAuthenticationDomains and DNSUsage and budgetsScheduled jobsAbuse controlsAI provider proxyCustomer mailErrors and performanceLogs and metricsRetentionObject storageContainer registryGit server
Develop
The .pl.yaml manifestOnboard with an AI agentThe pl up demoBuild executorSource ingestionAgent gatewayManaged databasesSecurity findingsTunnels and shellsDev workspacesPrivate hostnamesMesh coordinatorsDesign baseline
Runbooks
Control-plane host failoverControl-plane PostgreSQL failoverControl-plane PostgreSQL restoreCustom domains and certificatesGateway key compromiseIngress host failureNode busyNode disk fullNode drain and replacementNode lostRegistry outageHetzner Robot hostsSecret rotationSecurity incidentStaging fleet on ProxmoxStaging build cacheStuck deploymentVault/OpenBao backup and recovery
Manual · docs/runbooks/node-disk-full.md

Node disk full

Use this when node_data_disk_low, node_disk_high, node_disk_full, host_disk_high or host_disk_full fires. A filesystem is over 80% full, or the data disk has under 10 GB free. At 95%, backups, WAL archiving and the agent stop working.

Find the filesystem

  1. Open the alert. It names the node and the filesystem: /, tenant data or backups.
  2. Open the node in the Nodes page of the Fleet app. Compare the disks in Disk health.

Fix it

If the tenant data disk is full, move volumes to another node:

pushleft-controller fleet migrate-volumes <node>

If the whole pool is full, add a node. Select Enroll host or Provision on the Nodes page.

If the backups disk or / is full, log in to the node and find the largest directories. Check first that the off-host copy is current on the Backups page. Remove only files that you can identify.

Check it worked

The alert clears at the next evaluation after the filesystem drops under its limit.

Rendered from the repository at build time. Edit docs/runbooks/node-disk-full.md to change it.