Nothing tells the owner when monitoring itself fails. On 2026-11-11 Prometheus stored samples 9 hours ahead, then rejected every new sample as out of bounds: rules kept running on no data, and the only sign was a false KubeAPIDown. If Prometheus, Alertmanager or the node went down, no rule could fire at all. Add two rules and one route. PlatformMetricsIngestionStopped fires when no up sample from node-exporter is stored for 5 minutes; absent() still fires while the TSDB rejects samples. PlatformHeartbeat always fires, and a new platform-heartbeat route sends it every 5 minutes to an external dead man's switch through a webhook whose URL comes from the Secret alertmanager-heartbeat. When the pings stop, that service alerts, which covers Prometheus, Alertmanager, the node and the network being down. The chart Watchdog stays disabled; the platform rule keeps the route inside this repository. Rejected: an in-cluster CronJob that queries Prometheus and posts to Alertmanager. It still depends on Alertmanager and the node, and needs a new workload in the exact observability NetworkPolicy matrix. Risk: the AlertmanagerConfig now references alertmanager-heartbeat; applying it before that Secret exists keeps the operator from rendering the new Alertmanager config. The alerting README orders the steps: create the external check, create the Secret, then apply. The heartbeat needs an account on an external service, which only the owner can create. Verification: promtool tests for both rules failed against the old rules and pass now. test-observability-alerting.sh pins the new route and receiver in its exact contract and passes the static contract, the global Alertmanager render with Helm v3.19.4 and the server dry run (--server-dry-run). kubectl apply --dry-run=server accepts the updated PrometheusRule.
Observability core
This root composes the internal observability workloads deployed after the namespace, CRD, and Local PV prerequisites:
- kube-prometheus-stack without Grafana or its bundled node-exporter;
- single-binary Loki and Tempo using isolated AIStor buckets;
- Alloy for Pod logs and internal OTLP/gRPC traces;
- a standalone node-exporter;
- the exact core NetworkPolicy matrix, platform rules, scrape targets, verified dashboards, and source-only Alertmanager Slack routing.
It intentionally does not create public ingress, Grafana, OIDC, or credential
Secrets. The Slack routing source references
alertmanager-slack-webhook/url; while that Secret is absent, do not apply the
AlertmanagerConfig, its kube-prometheus-stack reference, its public HTTPS
NetworkPolicy, or alert rules. Namespace, CRD, and Local PV roots remain outside
this aggregate because their protected apply order is different.