Nothing tells the owner when monitoring itself fails. On 2026-11-11 Prometheus stored samples 9 hours ahead, then rejected every new sample as out of bounds: rules kept running on no data, and the only sign was a false KubeAPIDown. If Prometheus, Alertmanager or the node went down, no rule could fire at all. Add two rules and one route. PlatformMetricsIngestionStopped fires when no up sample from node-exporter is stored for 5 minutes; absent() still fires while the TSDB rejects samples. PlatformHeartbeat always fires, and a new platform-heartbeat route sends it every 5 minutes to an external dead man's switch through a webhook whose URL comes from the Secret alertmanager-heartbeat. When the pings stop, that service alerts, which covers Prometheus, Alertmanager, the node and the network being down. The chart Watchdog stays disabled; the platform rule keeps the route inside this repository. Rejected: an in-cluster CronJob that queries Prometheus and posts to Alertmanager. It still depends on Alertmanager and the node, and needs a new workload in the exact observability NetworkPolicy matrix. Risk: the AlertmanagerConfig now references alertmanager-heartbeat; applying it before that Secret exists keeps the operator from rendering the new Alertmanager config. The alerting README orders the steps: create the external check, create the Secret, then apply. The heartbeat needs an account on an external service, which only the owner can create. Verification: promtool tests for both rules failed against the old rules and pass now. test-observability-alerting.sh pins the new route and receiver in its exact contract and passes the static contract, the global Alertmanager render with Helm v3.19.4 and the server dry run (--server-dry-run). kubectl apply --dry-run=server accepts the updated PrometheusRule.
100 lines
4.1 KiB
YAML
100 lines
4.1 KiB
YAML
apiVersion: monitoring.coreos.com/v1alpha1
|
|
kind: AlertmanagerConfig
|
|
metadata:
|
|
name: platform-alertmanager
|
|
namespace: observability
|
|
labels:
|
|
observability.hyeonworks.com/instance: home
|
|
spec:
|
|
route:
|
|
receiver: platform-slack
|
|
groupBy:
|
|
- cluster
|
|
- namespace
|
|
- alertname
|
|
- severity
|
|
groupWait: 30s
|
|
groupInterval: 5m
|
|
repeatInterval: 4h
|
|
routes:
|
|
- receiver: platform-null
|
|
matchers:
|
|
- name: alertname
|
|
matchType: "="
|
|
value: InfoInhibitor
|
|
# PlatformHeartbeat always fires. Sending it every 5 minutes keeps the
|
|
# external dead man's switch quiet; when it stops, that service alerts.
|
|
- receiver: platform-heartbeat
|
|
matchers:
|
|
- name: alertname
|
|
matchType: "="
|
|
value: PlatformHeartbeat
|
|
groupWait: 0s
|
|
groupInterval: 1m
|
|
repeatInterval: 5m
|
|
inhibitRules:
|
|
- sourceMatch:
|
|
- name: alertname
|
|
matchType: "="
|
|
value: InfoInhibitor
|
|
targetMatch:
|
|
- name: severity
|
|
matchType: "="
|
|
value: info
|
|
equal:
|
|
- namespace
|
|
receivers:
|
|
- name: platform-null
|
|
- name: platform-heartbeat
|
|
webhookConfigs:
|
|
# Ping URL of the external heartbeat check. Create the Secret before
|
|
# applying this file; see services/observability/alerting/README.md.
|
|
- urlSecret:
|
|
name: alertmanager-heartbeat
|
|
key: url
|
|
sendResolved: false
|
|
maxAlerts: 1
|
|
- name: platform-slack
|
|
slackConfigs:
|
|
- apiURL:
|
|
name: alertmanager-slack-webhook
|
|
key: url
|
|
sendResolved: true
|
|
linkNames: false
|
|
mrkdwnIn:
|
|
- text
|
|
- fields
|
|
fallback: >-
|
|
{{ if eq .Status "firing" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}
|
|
title: >-
|
|
{{ if eq .Status "firing" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}
|
|
titleLink: https://grafana.learn.hyeonworks.com/
|
|
color: >-
|
|
{{ if eq .Status "resolved" }}good{{ else if or (eq .CommonLabels.severity "emergency") (eq .CommonLabels.severity "critical") }}danger{{ else if eq .CommonLabels.severity "warning" }}warning{{ else }}#439FE0{{ end }}
|
|
fields:
|
|
- title: Status
|
|
value: '{{ .Status | toUpper }}'
|
|
short: true
|
|
- title: Severity
|
|
value: '{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}'
|
|
short: true
|
|
- title: Location
|
|
value: '{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}'
|
|
short: true
|
|
- title: Alert count
|
|
value: '{{ len .Alerts }}'
|
|
short: true
|
|
text: |-
|
|
{{ range .Alerts }}
|
|
*Alert status:* {{ .Status | toUpper }}
|
|
*Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }}
|
|
*Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }}
|
|
*Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }}
|
|
*Started:* {{ .StartsAt.Format "2006-01-02T15:04:05Z07:00" }}
|
|
{{ if eq .Status "resolved" }}*Ended:* {{ .EndsAt.Format "2006-01-02T15:04:05Z07:00" }}{{ end }}
|
|
{{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }}
|
|
|
|
{{ end }}
|
|
<https://grafana.learn.hyeonworks.com/|Grafana> · <https://grafana.learn.hyeonworks.com/explore|Explore>
|
|
footer: hyeonworks observability · Alertmanager
|