feat(observability): alert when Prometheus itself stops working

Nothing tells the owner when monitoring itself fails. On 2026-11-11
Prometheus stored samples 9 hours ahead, then rejected every new
sample as out of bounds: rules kept running on no data, and the only
sign was a false KubeAPIDown. If Prometheus, Alertmanager or the node
went down, no rule could fire at all.

Add two rules and one route. PlatformMetricsIngestionStopped fires
when no up sample from node-exporter is stored for 5 minutes;
absent() still fires while the TSDB rejects samples. PlatformHeartbeat
always fires, and a new platform-heartbeat route sends it every 5
minutes to an external dead man's switch through a webhook whose URL
comes from the Secret alertmanager-heartbeat. When the pings stop,
that service alerts, which covers Prometheus, Alertmanager, the node
and the network being down. The chart Watchdog stays disabled; the
platform rule keeps the route inside this repository.

Rejected: an in-cluster CronJob that queries Prometheus and posts to
Alertmanager. It still depends on Alertmanager and the node, and
needs a new workload in the exact observability NetworkPolicy matrix.

Risk: the AlertmanagerConfig now references alertmanager-heartbeat;
applying it before that Secret exists keeps the operator from
rendering the new Alertmanager config. The alerting README orders the
steps: create the external check, create the Secret, then apply.
The heartbeat needs an account on an external service, which only
the owner can create.

Verification: promtool tests for both rules failed against the old
rules and pass now. test-observability-alerting.sh pins the new route
and receiver in its exact contract and passes the static contract,
the global Alertmanager render with Helm v3.19.4 and the server
dry run (--server-dry-run). kubectl apply --dry-run=server accepts
the updated PrometheusRule.
This commit is contained in:
DongHyeonka
2026-11-11 23:52:13 +09:00
parent 90ab8f4e65
commit 11c13bb87b
5 changed files with 153 additions and 4 deletions
@@ -87,8 +87,21 @@ if alert != {
"value": "InfoInhibitor", "value": "InfoInhibitor",
} }
], ],
},
{
"receiver": "platform-heartbeat",
"matchers": [
{
"name": "alertname",
"matchType": "=",
"value": "PlatformHeartbeat",
} }
], ],
"groupWait": "0s",
"groupInterval": "1m",
"repeatInterval": "5m",
},
],
}, },
"inhibitRules": [ "inhibitRules": [
{ {
@@ -111,6 +124,16 @@ if alert != {
], ],
"receivers": [ "receivers": [
{"name": "platform-null"}, {"name": "platform-null"},
{
"name": "platform-heartbeat",
"webhookConfigs": [
{
"urlSecret": {"name": "alertmanager-heartbeat", "key": "url"},
"sendResolved": False,
"maxAlerts": 1,
}
],
},
{ {
"name": "platform-slack", "name": "platform-slack",
"slackConfigs": [ "slackConfigs": [
@@ -156,7 +179,7 @@ if alert != {
], ],
}, },
}: }:
fail("platform-alertmanager must match the exact global Slack contract") fail("platform-alertmanager must match the exact global Slack and heartbeat contract")
if policy != { if policy != {
"apiVersion": "networking.k8s.io/v1", "apiVersion": "networking.k8s.io/v1",
+40 -2
View File
@@ -6,7 +6,8 @@
# points: warning at 80%, critical at 85%, and a forecast that fires when the # points: warning at 80%, critical at 85%, and a forecast that fires when the
# 6h trend reaches the eviction point within 6 hours. # 6h trend reaches the eviction point within 6 hours.
# PostgreSQL backups: no successful base backup for 26 hours, or WAL waiting # PostgreSQL backups: no successful base backup for 26 hours, or WAL waiting
# to be archived. # to be archived. Monitoring: ingestion stopped, and the always-firing
# heartbeat that an external dead man's switch watches.
# #
# Needs docker, jq and kubectl (client only, no cluster access). # Needs docker, jq and kubectl (client only, no cluster access).
set -euo pipefail set -euo pipefail
@@ -199,6 +200,43 @@ ${ALERT_LABELS}
- eval_time: 20m - eval_time: 20m
alertname: PlatformPostgresWALArchiveBacklog alertname: PlatformPostgresWALArchiveBacklog
exp_alerts: [] exp_alerts: []
- name: ingestion alert fires when node-exporter samples stop
interval: 1m
input_series:
- series: 'up{job="prometheus-node-exporter",instance="10.42.0.95:9100"}'
values: '1x5 _x20'
alert_rule_test:
- eval_time: 20m
alertname: PlatformMetricsIngestionStopped
exp_alerts:
- exp_labels:
job: prometheus-node-exporter
severity: critical
exp_annotations:
summary: Prometheus에 새 지표가 저장되지 않음
description: 5분 넘게 node-exporter의 up 샘플이 저장되지 않았습니다. Prometheus가 샘플을 버리고 있으면(예 시계가 되돌아감) 다른 지표 경보도 울리지 않습니다.
runbook_url: ${RUNBOOK_URL}
- name: ingestion alert stays quiet while samples arrive
interval: 1m
input_series:
- series: 'up{job="prometheus-node-exporter",instance="10.42.0.95:9100"}'
values: '1x30'
alert_rule_test:
- eval_time: 20m
alertname: PlatformMetricsIngestionStopped
exp_alerts: []
- name: heartbeat always fires
interval: 1m
alert_rule_test:
- eval_time: 1m
alertname: PlatformHeartbeat
exp_alerts:
- exp_labels:
severity: none
exp_annotations:
summary: 경보 경로 heartbeat
description: 항상 울린다. Alertmanager가 이 경보를 외부 dead man's switch로 5분마다 보내고, 끊기면 외부 서비스가 알린다.
runbook_url: ${RUNBOOK_URL}
EOF EOF
} >"${temp_dir}/disk.test.yaml" } >"${temp_dir}/disk.test.yaml"
@@ -213,4 +251,4 @@ docker run --rm \
--volume "${temp_dir}:/work:ro" \ --volume "${temp_dir}:/work:ro" \
--entrypoint /bin/promtool \ --entrypoint /bin/promtool \
"$PROMETHEUS_IMAGE" test rules /work/disk.test.yaml "$PROMETHEUS_IMAGE" test rules /work/disk.test.yaml
printf 'PASS: root filesystem and PostgreSQL backup alerts\n' printf 'PASS: root filesystem, PostgreSQL backup, ingestion and heartbeat alerts\n'
+43
View File
@@ -0,0 +1,43 @@
# Alertmanager 경로
`platform-alertmanager.yaml`은 kube-prometheus-stack이 전역
`alertmanagerConfiguration`으로 읽는 AlertmanagerConfig 하나다. 경로는 셋이다.
| 경보 | receiver | 보내는 곳 |
| --- | --- | --- |
| `InfoInhibitor` | `platform-null` | 버린다 |
| `PlatformHeartbeat` | `platform-heartbeat` | 외부 heartbeat 서비스(dead man's switch), 5분마다 |
| 나머지 | `platform-slack` | Slack, 4시간마다 다시 |
## heartbeat (dead man's switch)
`PlatformHeartbeat`(`rules/platform/core-rules.yaml`)는 항상 울린다. Alertmanager가 이것을
5분마다 외부 서비스의 ping URL로 보낸다. ping이 끊기면 그 서비스가 소유자에게 알린다.
- 막는 것: Prometheus·Alertmanager가 죽었을 때, desktop이나 인터넷 연결이 끊겼을 때.
이 경우는 클러스터 안의 어떤 규칙도 알리지 못한다.
- 막지 못하는 것: Prometheus가 돌면서 샘플만 버리는 경우(2026-11-11 시계 사고). 규칙은 돌고
heartbeat도 계속 나간다. 그 경우는 `PlatformMetricsIngestionStopped`가 Slack으로 알린다.
### 소유자가 할 일 (적용 전에 한 번)
1. 외부 heartbeat 서비스에 check를 하나 만든다. 예: healthchecks.io(무료 요금제, 오픈소스라
나중에 직접 띄울 수도 있다). 주기 5분, 유예 10분. 알림은 Slack이 아닌 곳(메일·휴대폰)도
하나 건다. Slack 경로가 죽었을 때도 알아야 하기 때문이다.
2. ping URL을 Secret으로 만든다. URL 자체가 자격 증명이라 화면에 남기지 않는다.
```bash
read -rs HEARTBEAT_URL # ping URL 붙여 넣기, 화면에 안 보임
printf '%s' "$HEARTBEAT_URL" | kubectl -n observability create secret generic alertmanager-heartbeat \
--from-file=url=/dev/stdin
unset HEARTBEAT_URL
```
3. 그 뒤에 rules·alerts를 다시 적용한다(`scripts/bootstrap/apply-observability-access.sh
--execute --rules-alerts`). **Secret보다 먼저 적용하지 않는다.** operator가 없는 Secret을
가리키는 AlertmanagerConfig로는 설정을 만들지 못한다.
4. 확인: 외부 서비스의 check가 5분 안에 「up」이 된다. Alertmanager 지표
`alertmanager_notifications_total{integration="webhook"}`가 5분마다 는다.
Alertmanager egress는 공개 IPv4 TCP/443만 허용한다(`alertmanager-public-https-egress.yaml`).
ping URL은 HTTPS여야 한다.
@@ -22,6 +22,16 @@ spec:
- name: alertname - name: alertname
matchType: "=" matchType: "="
value: InfoInhibitor value: InfoInhibitor
# PlatformHeartbeat always fires. Sending it every 5 minutes keeps the
# external dead man's switch quiet; when it stops, that service alerts.
- receiver: platform-heartbeat
matchers:
- name: alertname
matchType: "="
value: PlatformHeartbeat
groupWait: 0s
groupInterval: 1m
repeatInterval: 5m
inhibitRules: inhibitRules:
- sourceMatch: - sourceMatch:
- name: alertname - name: alertname
@@ -35,6 +45,15 @@ spec:
- namespace - namespace
receivers: receivers:
- name: platform-null - name: platform-null
- name: platform-heartbeat
webhookConfigs:
# Ping URL of the external heartbeat check. Create the Secret before
# applying this file; see services/observability/alerting/README.md.
- urlSecret:
name: alertmanager-heartbeat
key: url
sendResolved: false
maxAlerts: 1
- name: platform-slack - name: platform-slack
slackConfigs: slackConfigs:
- apiURL: - apiURL:
@@ -143,6 +143,32 @@ spec:
summary: platform-postgres의 WAL이 보관소로 올라가지 않음 summary: platform-postgres의 WAL이 보관소로 올라가지 않음
description: 보관을 기다리는 WAL 파일이 10개를 넘은 채 15분이 지났습니다. 보관이 막히면 WAL이 PostgreSQL 볼륨에 쌓여 디스크가 찹니다. description: 보관을 기다리는 WAL 파일이 10개를 넘은 채 15분이 지났습니다. 보관이 막히면 WAL이 PostgreSQL 볼륨에 쌓여 디스크가 찹니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- name: platform.monitoring
rules:
# When the TSDB rejects every sample (2026-11-11: the clock went back
# after samples were stored 9 hours ahead) rules still run, but on no
# data. absent() is one of the few expressions that still fires then.
- alert: PlatformMetricsIngestionStopped
expr: absent(up{job="prometheus-node-exporter"})
for: 5m
labels:
severity: critical
annotations:
summary: Prometheus에 새 지표가 저장되지 않음
description: 5분 넘게 node-exporter의 up 샘플이 저장되지 않았습니다. Prometheus가 샘플을 버리고 있으면(예 시계가 되돌아감) 다른 지표 경보도 울리지 않습니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
# Dead man's switch. Alertmanager forwards this to an external
# heartbeat service; when the heartbeat stops, that service alerts.
# It covers what no rule here can: Prometheus, Alertmanager or the
# whole node being down.
- alert: PlatformHeartbeat
expr: vector(1)
labels:
severity: none
annotations:
summary: 경보 경로 heartbeat
description: 항상 울린다. Alertmanager가 이 경보를 외부 dead man's switch로 5분마다 보내고, 끊기면 외부 서비스가 알린다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- name: platform.workloads - name: platform.workloads
rules: rules:
- alert: PlatformPodRestartBurst - alert: PlatformPodRestartBurst