The observability core renderer rejects any alert or record whose name contains "backup" (no_unscoped_monitors_rules): backup monitoring is outside the observability core by design.90ab8f4put PlatformPostgresBackupMissing and PlatformPostgresWALArchiveBacklog in core-rules.yaml, so the pinned renderer failed with "out-of-scope monitor or rule is present". Move both alerts unchanged into services/platform-postgres-backup/prometheus-rule.yaml in platform-data, which Prometheus reads because the namespace has observability.hyeonworks.com/enabled=true and the rule has observability.hyeonworks.com/instance=home. They now point their runbook_url at the backup README and are applied with the ScheduledBackup. Renaming them to dodge the filter was rejected: it would keep backup rules inside a scope that excludes them on purpose. Verification: test-platform-disk-alerts.sh loads both rule files and passes; promtool checks 13 rules. render-observability-core.sh and test-render-observability-core.sh pass, and kubectl apply --dry-run=server accepts the backup kustomization with the rule. scan-platform-sensitive-source.sh stops at its observability-core renderer with RC=23 here and on an origin/main snapshot alike, so it says nothing about this change. Fixes:90ab8f4e65("feat(postgres): back up platform-postgres to AIStor every day")
47 lines
2.4 KiB
YAML
47 lines
2.4 KiB
YAML
# Backup alerts for platform-postgres. They live with the backup and not in
|
|
# the observability core, whose renderer keeps backup rules out of its scope
|
|
# (scripts/validate/render-observability-core.sh, no_unscoped_monitors_rules).
|
|
# Prometheus loads PrometheusRules from namespaces labelled
|
|
# observability.hyeonworks.com/enabled=true, which platform-data is.
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-postgres-backup
|
|
namespace: platform-data
|
|
labels:
|
|
app.kubernetes.io/name: platform-postgres
|
|
app.kubernetes.io/component: backup
|
|
observability.hyeonworks.com/instance: home
|
|
spec:
|
|
groups:
|
|
- name: platform.backup
|
|
rules:
|
|
# cnpg_collector_last_available_backup_timestamp is 0 until the first
|
|
# base backup, so a cluster without backups fires as well.
|
|
- alert: PlatformPostgresBackupMissing
|
|
expr: >-
|
|
time() - max by (job, namespace) (
|
|
cnpg_collector_last_available_backup_timestamp{job="platform-postgres",namespace="platform-data"}
|
|
) > 26 * 3600
|
|
for: 30m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: platform-postgres에 26시간 안에 끝난 백업이 없음
|
|
description: platform-postgres의 마지막 성공한 base backup이 26시간보다 오래됐거나 없습니다. 매일 03:00 KST 백업(ScheduledBackup platform-postgres-daily)을 확인하세요.
|
|
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/platform-core/src/branch/main/services/platform-postgres-backup/README.md
|
|
# A failing archive_command keeps WAL in pg_wal, which then fills the
|
|
# PostgreSQL volume.
|
|
- alert: PlatformPostgresWALArchiveBacklog
|
|
expr: >-
|
|
max by (job, namespace) (
|
|
cnpg_collector_pg_wal_archive_status{job="platform-postgres",namespace="platform-data",value="ready"}
|
|
) > 10
|
|
for: 15m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: platform-postgres의 WAL이 보관소로 올라가지 않음
|
|
description: 보관을 기다리는 WAL 파일이 10개를 넘은 채 15분이 지났습니다. 보관이 막히면 WAL이 PostgreSQL 볼륨에 쌓여 디스크가 찹니다.
|
|
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/platform-core/src/branch/main/services/platform-postgres-backup/README.md
|