# Backup alerts for platform-postgres. They live with the backup and not in # the observability core, whose renderer keeps backup rules out of its scope # (scripts/validate/render-observability-core.sh, no_unscoped_monitors_rules). # Prometheus loads PrometheusRules from namespaces labelled # observability.hyeonworks.com/enabled=true, which platform-data is. apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: platform-postgres-backup namespace: platform-data labels: app.kubernetes.io/name: platform-postgres app.kubernetes.io/component: backup observability.hyeonworks.com/instance: home spec: groups: - name: platform.backup rules: # cnpg_collector_last_available_backup_timestamp is 0 until the first # base backup, so a cluster without backups fires as well. - alert: PlatformPostgresBackupMissing expr: >- time() - max by (job, namespace) ( cnpg_collector_last_available_backup_timestamp{job="platform-postgres",namespace="platform-data"} ) > 26 * 3600 for: 30m labels: severity: critical annotations: summary: platform-postgres에 26시간 안에 끝난 백업이 없음 description: platform-postgres의 마지막 성공한 base backup이 26시간보다 오래됐거나 없습니다. 매일 03:00 KST 백업(ScheduledBackup platform-postgres-daily)을 확인하세요. runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/platform-core/src/branch/main/services/platform-postgres-backup/README.md # A failing archive_command keeps WAL in pg_wal, which then fills the # PostgreSQL volume. - alert: PlatformPostgresWALArchiveBacklog expr: >- max by (job, namespace) ( cnpg_collector_pg_wal_archive_status{job="platform-postgres",namespace="platform-data",value="ready"} ) > 10 for: 15m labels: severity: critical annotations: summary: platform-postgres의 WAL이 보관소로 올라가지 않음 description: 보관을 기다리는 WAL 파일이 10개를 넘은 채 15분이 지났습니다. 보관이 막히면 WAL이 PostgreSQL 볼륨에 쌓여 디스크가 찹니다. runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/platform-core/src/branch/main/services/platform-postgres-backup/README.md