feat(observability): alert when Prometheus itself stops working

Nothing tells the owner when monitoring itself fails. On 2026-11-11
Prometheus stored samples 9 hours ahead, then rejected every new
sample as out of bounds: rules kept running on no data, and the only
sign was a false KubeAPIDown. If Prometheus, Alertmanager or the node
went down, no rule could fire at all.

Add two rules and one route. PlatformMetricsIngestionStopped fires
when no up sample from node-exporter is stored for 5 minutes;
absent() still fires while the TSDB rejects samples. PlatformHeartbeat
always fires, and a new platform-heartbeat route sends it every 5
minutes to an external dead man's switch through a webhook whose URL
comes from the Secret alertmanager-heartbeat. When the pings stop,
that service alerts, which covers Prometheus, Alertmanager, the node
and the network being down. The chart Watchdog stays disabled; the
platform rule keeps the route inside this repository.

Rejected: an in-cluster CronJob that queries Prometheus and posts to
Alertmanager. It still depends on Alertmanager and the node, and
needs a new workload in the exact observability NetworkPolicy matrix.

Risk: the AlertmanagerConfig now references alertmanager-heartbeat;
applying it before that Secret exists keeps the operator from
rendering the new Alertmanager config. The alerting README orders the
steps: create the external check, create the Secret, then apply.
The heartbeat needs an account on an external service, which only
the owner can create.

Verification: promtool tests for both rules failed against the old
rules and pass now. test-observability-alerting.sh pins the new route
and receiver in its exact contract and passes the static contract,
the global Alertmanager render with Helm v3.19.4 and the server
dry run (--server-dry-run). kubectl apply --dry-run=server accepts
the updated PrometheusRule.
This commit is contained in:
DongHyeonka
2026-11-11 23:52:13 +09:00
parent 90ab8f4e65
commit 11c13bb87b
5 changed files with 153 additions and 4 deletions
@@ -87,7 +87,20 @@ if alert != {
"value": "InfoInhibitor",
}
],
}
},
{
"receiver": "platform-heartbeat",
"matchers": [
{
"name": "alertname",
"matchType": "=",
"value": "PlatformHeartbeat",
}
],
"groupWait": "0s",
"groupInterval": "1m",
"repeatInterval": "5m",
},
],
},
"inhibitRules": [
@@ -111,6 +124,16 @@ if alert != {
],
"receivers": [
{"name": "platform-null"},
{
"name": "platform-heartbeat",
"webhookConfigs": [
{
"urlSecret": {"name": "alertmanager-heartbeat", "key": "url"},
"sendResolved": False,
"maxAlerts": 1,
}
],
},
{
"name": "platform-slack",
"slackConfigs": [
@@ -156,7 +179,7 @@ if alert != {
],
},
}:
fail("platform-alertmanager must match the exact global Slack contract")
fail("platform-alertmanager must match the exact global Slack and heartbeat contract")
if policy != {
"apiVersion": "networking.k8s.io/v1",
+40 -2
View File
@@ -6,7 +6,8 @@
# points: warning at 80%, critical at 85%, and a forecast that fires when the
# 6h trend reaches the eviction point within 6 hours.
# PostgreSQL backups: no successful base backup for 26 hours, or WAL waiting
# to be archived.
# to be archived. Monitoring: ingestion stopped, and the always-firing
# heartbeat that an external dead man's switch watches.
#
# Needs docker, jq and kubectl (client only, no cluster access).
set -euo pipefail
@@ -199,6 +200,43 @@ ${ALERT_LABELS}
- eval_time: 20m
alertname: PlatformPostgresWALArchiveBacklog
exp_alerts: []
- name: ingestion alert fires when node-exporter samples stop
interval: 1m
input_series:
- series: 'up{job="prometheus-node-exporter",instance="10.42.0.95:9100"}'
values: '1x5 _x20'
alert_rule_test:
- eval_time: 20m
alertname: PlatformMetricsIngestionStopped
exp_alerts:
- exp_labels:
job: prometheus-node-exporter
severity: critical
exp_annotations:
summary: Prometheus에 새 지표가 저장되지 않음
description: 5분 넘게 node-exporter의 up 샘플이 저장되지 않았습니다. Prometheus가 샘플을 버리고 있으면(예 시계가 되돌아감) 다른 지표 경보도 울리지 않습니다.
runbook_url: ${RUNBOOK_URL}
- name: ingestion alert stays quiet while samples arrive
interval: 1m
input_series:
- series: 'up{job="prometheus-node-exporter",instance="10.42.0.95:9100"}'
values: '1x30'
alert_rule_test:
- eval_time: 20m
alertname: PlatformMetricsIngestionStopped
exp_alerts: []
- name: heartbeat always fires
interval: 1m
alert_rule_test:
- eval_time: 1m
alertname: PlatformHeartbeat
exp_alerts:
- exp_labels:
severity: none
exp_annotations:
summary: 경보 경로 heartbeat
description: 항상 울린다. Alertmanager가 이 경보를 외부 dead man's switch로 5분마다 보내고, 끊기면 외부 서비스가 알린다.
runbook_url: ${RUNBOOK_URL}
EOF
} >"${temp_dir}/disk.test.yaml"
@@ -213,4 +251,4 @@ docker run --rm \
--volume "${temp_dir}:/work:ro" \
--entrypoint /bin/promtool \
"$PROMETHEUS_IMAGE" test rules /work/disk.test.yaml
printf 'PASS: root filesystem and PostgreSQL backup alerts\n'
printf 'PASS: root filesystem, PostgreSQL backup, ingestion and heartbeat alerts\n'