Files
platform-core/services/observability/alerting/platform-alertmanager.yaml
T
DongHyeonka 11c13bb87b feat(observability): alert when Prometheus itself stops working
Nothing tells the owner when monitoring itself fails. On 2026-11-11
Prometheus stored samples 9 hours ahead, then rejected every new
sample as out of bounds: rules kept running on no data, and the only
sign was a false KubeAPIDown. If Prometheus, Alertmanager or the node
went down, no rule could fire at all.

Add two rules and one route. PlatformMetricsIngestionStopped fires
when no up sample from node-exporter is stored for 5 minutes;
absent() still fires while the TSDB rejects samples. PlatformHeartbeat
always fires, and a new platform-heartbeat route sends it every 5
minutes to an external dead man's switch through a webhook whose URL
comes from the Secret alertmanager-heartbeat. When the pings stop,
that service alerts, which covers Prometheus, Alertmanager, the node
and the network being down. The chart Watchdog stays disabled; the
platform rule keeps the route inside this repository.

Rejected: an in-cluster CronJob that queries Prometheus and posts to
Alertmanager. It still depends on Alertmanager and the node, and
needs a new workload in the exact observability NetworkPolicy matrix.

Risk: the AlertmanagerConfig now references alertmanager-heartbeat;
applying it before that Secret exists keeps the operator from
rendering the new Alertmanager config. The alerting README orders the
steps: create the external check, create the Secret, then apply.
The heartbeat needs an account on an external service, which only
the owner can create.

Verification: promtool tests for both rules failed against the old
rules and pass now. test-observability-alerting.sh pins the new route
and receiver in its exact contract and passes the static contract,
the global Alertmanager render with Helm v3.19.4 and the server
dry run (--server-dry-run). kubectl apply --dry-run=server accepts
the updated PrometheusRule.
2026-11-11 23:52:13 +09:00

100 lines
4.1 KiB
YAML

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: platform-alertmanager
namespace: observability
labels:
observability.hyeonworks.com/instance: home
spec:
route:
receiver: platform-slack
groupBy:
- cluster
- namespace
- alertname
- severity
groupWait: 30s
groupInterval: 5m
repeatInterval: 4h
routes:
- receiver: platform-null
matchers:
- name: alertname
matchType: "="
value: InfoInhibitor
# PlatformHeartbeat always fires. Sending it every 5 minutes keeps the
# external dead man's switch quiet; when it stops, that service alerts.
- receiver: platform-heartbeat
matchers:
- name: alertname
matchType: "="
value: PlatformHeartbeat
groupWait: 0s
groupInterval: 1m
repeatInterval: 5m
inhibitRules:
- sourceMatch:
- name: alertname
matchType: "="
value: InfoInhibitor
targetMatch:
- name: severity
matchType: "="
value: info
equal:
- namespace
receivers:
- name: platform-null
- name: platform-heartbeat
webhookConfigs:
# Ping URL of the external heartbeat check. Create the Secret before
# applying this file; see services/observability/alerting/README.md.
- urlSecret:
name: alertmanager-heartbeat
key: url
sendResolved: false
maxAlerts: 1
- name: platform-slack
slackConfigs:
- apiURL:
name: alertmanager-slack-webhook
key: url
sendResolved: true
linkNames: false
mrkdwnIn:
- text
- fields
fallback: >-
{{ if eq .Status "firing" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}
title: >-
{{ if eq .Status "firing" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}
titleLink: https://grafana.learn.hyeonworks.com/
color: >-
{{ if eq .Status "resolved" }}good{{ else if or (eq .CommonLabels.severity "emergency") (eq .CommonLabels.severity "critical") }}danger{{ else if eq .CommonLabels.severity "warning" }}warning{{ else }}#439FE0{{ end }}
fields:
- title: Status
value: '{{ .Status | toUpper }}'
short: true
- title: Severity
value: '{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}'
short: true
- title: Location
value: '{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}'
short: true
- title: Alert count
value: '{{ len .Alerts }}'
short: true
text: |-
{{ range .Alerts }}
*Alert status:* {{ .Status | toUpper }}
*Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }}
*Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }}
*Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }}
*Started:* {{ .StartsAt.Format "2006-01-02T15:04:05Z07:00" }}
{{ if eq .Status "resolved" }}*Ended:* {{ .EndsAt.Format "2006-01-02T15:04:05Z07:00" }}{{ end }}
{{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }}
{{ end }}
<https://grafana.learn.hyeonworks.com/|Grafana> · <https://grafana.learn.hyeonworks.com/explore|Explore>
footer: hyeonworks observability · Alertmanager