feat(observability): alert when Prometheus itself stops working
Nothing tells the owner when monitoring itself fails. On 2026-11-11 Prometheus stored samples 9 hours ahead, then rejected every new sample as out of bounds: rules kept running on no data, and the only sign was a false KubeAPIDown. If Prometheus, Alertmanager or the node went down, no rule could fire at all. Add two rules and one route. PlatformMetricsIngestionStopped fires when no up sample from node-exporter is stored for 5 minutes; absent() still fires while the TSDB rejects samples. PlatformHeartbeat always fires, and a new platform-heartbeat route sends it every 5 minutes to an external dead man's switch through a webhook whose URL comes from the Secret alertmanager-heartbeat. When the pings stop, that service alerts, which covers Prometheus, Alertmanager, the node and the network being down. The chart Watchdog stays disabled; the platform rule keeps the route inside this repository. Rejected: an in-cluster CronJob that queries Prometheus and posts to Alertmanager. It still depends on Alertmanager and the node, and needs a new workload in the exact observability NetworkPolicy matrix. Risk: the AlertmanagerConfig now references alertmanager-heartbeat; applying it before that Secret exists keeps the operator from rendering the new Alertmanager config. The alerting README orders the steps: create the external check, create the Secret, then apply. The heartbeat needs an account on an external service, which only the owner can create. Verification: promtool tests for both rules failed against the old rules and pass now. test-observability-alerting.sh pins the new route and receiver in its exact contract and passes the static contract, the global Alertmanager render with Helm v3.19.4 and the server dry run (--server-dry-run). kubectl apply --dry-run=server accepts the updated PrometheusRule.
This commit is contained in:
@@ -87,7 +87,20 @@ if alert != {
|
||||
"value": "InfoInhibitor",
|
||||
}
|
||||
],
|
||||
}
|
||||
},
|
||||
{
|
||||
"receiver": "platform-heartbeat",
|
||||
"matchers": [
|
||||
{
|
||||
"name": "alertname",
|
||||
"matchType": "=",
|
||||
"value": "PlatformHeartbeat",
|
||||
}
|
||||
],
|
||||
"groupWait": "0s",
|
||||
"groupInterval": "1m",
|
||||
"repeatInterval": "5m",
|
||||
},
|
||||
],
|
||||
},
|
||||
"inhibitRules": [
|
||||
@@ -111,6 +124,16 @@ if alert != {
|
||||
],
|
||||
"receivers": [
|
||||
{"name": "platform-null"},
|
||||
{
|
||||
"name": "platform-heartbeat",
|
||||
"webhookConfigs": [
|
||||
{
|
||||
"urlSecret": {"name": "alertmanager-heartbeat", "key": "url"},
|
||||
"sendResolved": False,
|
||||
"maxAlerts": 1,
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
"name": "platform-slack",
|
||||
"slackConfigs": [
|
||||
@@ -156,7 +179,7 @@ if alert != {
|
||||
],
|
||||
},
|
||||
}:
|
||||
fail("platform-alertmanager must match the exact global Slack contract")
|
||||
fail("platform-alertmanager must match the exact global Slack and heartbeat contract")
|
||||
|
||||
if policy != {
|
||||
"apiVersion": "networking.k8s.io/v1",
|
||||
|
||||
@@ -6,7 +6,8 @@
|
||||
# points: warning at 80%, critical at 85%, and a forecast that fires when the
|
||||
# 6h trend reaches the eviction point within 6 hours.
|
||||
# PostgreSQL backups: no successful base backup for 26 hours, or WAL waiting
|
||||
# to be archived.
|
||||
# to be archived. Monitoring: ingestion stopped, and the always-firing
|
||||
# heartbeat that an external dead man's switch watches.
|
||||
#
|
||||
# Needs docker, jq and kubectl (client only, no cluster access).
|
||||
set -euo pipefail
|
||||
@@ -199,6 +200,43 @@ ${ALERT_LABELS}
|
||||
- eval_time: 20m
|
||||
alertname: PlatformPostgresWALArchiveBacklog
|
||||
exp_alerts: []
|
||||
- name: ingestion alert fires when node-exporter samples stop
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'up{job="prometheus-node-exporter",instance="10.42.0.95:9100"}'
|
||||
values: '1x5 _x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 20m
|
||||
alertname: PlatformMetricsIngestionStopped
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
job: prometheus-node-exporter
|
||||
severity: critical
|
||||
exp_annotations:
|
||||
summary: Prometheus에 새 지표가 저장되지 않음
|
||||
description: 5분 넘게 node-exporter의 up 샘플이 저장되지 않았습니다. Prometheus가 샘플을 버리고 있으면(예 시계가 되돌아감) 다른 지표 경보도 울리지 않습니다.
|
||||
runbook_url: ${RUNBOOK_URL}
|
||||
- name: ingestion alert stays quiet while samples arrive
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'up{job="prometheus-node-exporter",instance="10.42.0.95:9100"}'
|
||||
values: '1x30'
|
||||
alert_rule_test:
|
||||
- eval_time: 20m
|
||||
alertname: PlatformMetricsIngestionStopped
|
||||
exp_alerts: []
|
||||
- name: heartbeat always fires
|
||||
interval: 1m
|
||||
alert_rule_test:
|
||||
- eval_time: 1m
|
||||
alertname: PlatformHeartbeat
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: none
|
||||
exp_annotations:
|
||||
summary: 경보 경로 heartbeat
|
||||
description: 항상 울린다. Alertmanager가 이 경보를 외부 dead man's switch로 5분마다 보내고, 끊기면 외부 서비스가 알린다.
|
||||
runbook_url: ${RUNBOOK_URL}
|
||||
EOF
|
||||
} >"${temp_dir}/disk.test.yaml"
|
||||
|
||||
@@ -213,4 +251,4 @@ docker run --rm \
|
||||
--volume "${temp_dir}:/work:ro" \
|
||||
--entrypoint /bin/promtool \
|
||||
"$PROMETHEUS_IMAGE" test rules /work/disk.test.yaml
|
||||
printf 'PASS: root filesystem and PostgreSQL backup alerts\n'
|
||||
printf 'PASS: root filesystem, PostgreSQL backup, ingestion and heartbeat alerts\n'
|
||||
|
||||
Reference in New Issue
Block a user