The kubelet starts deleting unused images at 85% root use and evicts pods at 90%. The root disk warning fired at 85% and the critical at 90%, so both arrived when the kubelet was already acting. On 2026-11-09 the warning fired for 33 hours, then an eviction stopped the tech-log pods, the image GC deleted their imagePullPolicy: Never images, and production went down. Move the warning to 80% for 10m and the critical to 85% for 5m, and add PlatformRootFilesystemFillingFast (critical, for 15m): the 6h linear trend of free bytes falls below 10% of the disk, the eviction point, within 6 hours. On 2026-11-07 the disk lost 17G in 9 hours; this forecast would have fired hours before the 80% warning. The info alert at 75% stays. Add scripts/validate/test-platform-disk-alerts.sh, promtool unit tests for the usage bands and the forecast with the pinned Prometheus image. The existing rules test needs live metric inventories, so this one runs offline. Risk: the live rules change only when the observability access apply runs; this commit does not touch the cluster. The forecast can fire on a large build that the BuildKit GC removes minutes later; the 15m hold and the 6h window damp that. Verification: the new test failed against the old rules (no alert at 81%, 86% or for a 1-per-minute fill) and passes after the change; promtool check rules finds 9 rules. test-observability-alerting.sh passes its static contract, then stops at the helm render because helm v3.19.4 is not installed on the desktop.
Observability core
This root composes the internal observability workloads deployed after the namespace, CRD, and Local PV prerequisites:
- kube-prometheus-stack without Grafana or its bundled node-exporter;
- single-binary Loki and Tempo using isolated AIStor buckets;
- Alloy for Pod logs and internal OTLP/gRPC traces;
- a standalone node-exporter;
- the exact core NetworkPolicy matrix, platform rules, scrape targets, verified dashboards, and source-only Alertmanager Slack routing.
It intentionally does not create public ingress, Grafana, OIDC, or credential
Secrets. The Slack routing source references
alertmanager-slack-webhook/url; while that Secret is absent, do not apply the
AlertmanagerConfig, its kube-prometheus-stack reference, its public HTTPS
NetworkPolicy, or alert rules. Namespace, CRD, and Local PV roots remain outside
this aggregate because their protected apply order is different.