feat(observability): alert when Prometheus itself stops working
Nothing tells the owner when monitoring itself fails. On 2026-11-11 Prometheus stored samples 9 hours ahead, then rejected every new sample as out of bounds: rules kept running on no data, and the only sign was a false KubeAPIDown. If Prometheus, Alertmanager or the node went down, no rule could fire at all. Add two rules and one route. PlatformMetricsIngestionStopped fires when no up sample from node-exporter is stored for 5 minutes; absent() still fires while the TSDB rejects samples. PlatformHeartbeat always fires, and a new platform-heartbeat route sends it every 5 minutes to an external dead man's switch through a webhook whose URL comes from the Secret alertmanager-heartbeat. When the pings stop, that service alerts, which covers Prometheus, Alertmanager, the node and the network being down. The chart Watchdog stays disabled; the platform rule keeps the route inside this repository. Rejected: an in-cluster CronJob that queries Prometheus and posts to Alertmanager. It still depends on Alertmanager and the node, and needs a new workload in the exact observability NetworkPolicy matrix. Risk: the AlertmanagerConfig now references alertmanager-heartbeat; applying it before that Secret exists keeps the operator from rendering the new Alertmanager config. The alerting README orders the steps: create the external check, create the Secret, then apply. The heartbeat needs an account on an external service, which only the owner can create. Verification: promtool tests for both rules failed against the old rules and pass now. test-observability-alerting.sh pins the new route and receiver in its exact contract and passes the static contract, the global Alertmanager render with Helm v3.19.4 and the server dry run (--server-dry-run). kubectl apply --dry-run=server accepts the updated PrometheusRule.
This commit is contained in:
@@ -87,7 +87,20 @@ if alert != {
|
||||
"value": "InfoInhibitor",
|
||||
}
|
||||
],
|
||||
}
|
||||
},
|
||||
{
|
||||
"receiver": "platform-heartbeat",
|
||||
"matchers": [
|
||||
{
|
||||
"name": "alertname",
|
||||
"matchType": "=",
|
||||
"value": "PlatformHeartbeat",
|
||||
}
|
||||
],
|
||||
"groupWait": "0s",
|
||||
"groupInterval": "1m",
|
||||
"repeatInterval": "5m",
|
||||
},
|
||||
],
|
||||
},
|
||||
"inhibitRules": [
|
||||
@@ -111,6 +124,16 @@ if alert != {
|
||||
],
|
||||
"receivers": [
|
||||
{"name": "platform-null"},
|
||||
{
|
||||
"name": "platform-heartbeat",
|
||||
"webhookConfigs": [
|
||||
{
|
||||
"urlSecret": {"name": "alertmanager-heartbeat", "key": "url"},
|
||||
"sendResolved": False,
|
||||
"maxAlerts": 1,
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
"name": "platform-slack",
|
||||
"slackConfigs": [
|
||||
@@ -156,7 +179,7 @@ if alert != {
|
||||
],
|
||||
},
|
||||
}:
|
||||
fail("platform-alertmanager must match the exact global Slack contract")
|
||||
fail("platform-alertmanager must match the exact global Slack and heartbeat contract")
|
||||
|
||||
if policy != {
|
||||
"apiVersion": "networking.k8s.io/v1",
|
||||
|
||||
Reference in New Issue
Block a user