refactor(postgres): keep the backup alerts with the backup

The observability core renderer rejects any alert or record whose
name contains "backup" (no_unscoped_monitors_rules): backup
monitoring is outside the observability core by design. f19ce54 put
PlatformPostgresBackupMissing and PlatformPostgresWALArchiveBacklog in
core-rules.yaml, so the pinned renderer failed with "out-of-scope
monitor or rule is present".

Move both alerts unchanged into
services/platform-postgres-backup/prometheus-rule.yaml in
platform-data, which Prometheus reads because the namespace has
observability.hyeonworks.com/enabled=true and the rule has
observability.hyeonworks.com/instance=home. They now point their
runbook_url at the backup README and are applied with the
ScheduledBackup. Renaming them to dodge the filter was rejected: it
would keep backup rules inside a scope that excludes them on purpose.

Verification: test-platform-disk-alerts.sh loads both rule files and
passes; promtool checks 13 rules. render-observability-core.sh and
test-render-observability-core.sh pass, and kubectl apply
--dry-run=server accepts the backup kustomization with the rule.
scan-platform-sensitive-source.sh stops at its observability-core
renderer with RC=23 here and on an origin/main snapshot alike, so it
says nothing about this change.

Fixes: f19ce54740 ("feat(postgres): back up platform-postgres to AIStor every day")
This commit is contained in:
DongHyeonka
2026-11-12 07:28:31 +09:00
parent 2b1310972c
commit d7c845ca5d
5 changed files with 64 additions and 37 deletions
@@ -113,36 +113,6 @@ spec:
summary: 루트 파일시스템 inode 사용률이 90%를 초과함
description: 노드 루트 파일시스템 inode 사용률이 심각 임계치인 90%를 15분간 초과했습니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- name: platform.backup
rules:
# cnpg_collector_last_available_backup_timestamp is 0 until the first
# base backup, so a cluster without backups fires as well.
- alert: PlatformPostgresBackupMissing
expr: >-
time() - max by (job, namespace) (
cnpg_collector_last_available_backup_timestamp{job="platform-postgres",namespace="platform-data"}
) > 26 * 3600
for: 30m
labels:
severity: critical
annotations:
summary: platform-postgres에 26시간 안에 끝난 백업이 없음
description: platform-postgres의 마지막 성공한 base backup이 26시간보다 오래됐거나 없습니다. 매일 03:00 KST 백업(ScheduledBackup platform-postgres-daily)을 확인하세요.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
# A failing archive_command keeps WAL in pg_wal, which then fills the
# PostgreSQL volume.
- alert: PlatformPostgresWALArchiveBacklog
expr: >-
max by (job, namespace) (
cnpg_collector_pg_wal_archive_status{job="platform-postgres",namespace="platform-data",value="ready"}
) > 10
for: 15m
labels:
severity: critical
annotations:
summary: platform-postgres의 WAL이 보관소로 올라가지 않음
description: 보관을 기다리는 WAL 파일이 10개를 넘은 채 15분이 지났습니다. 보관이 막히면 WAL이 PostgreSQL 볼륨에 쌓여 디스크가 찹니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- name: platform.monitoring
rules:
# When the TSDB rejects every sample (2026-11-11: the clock went back
+4 -3
View File
@@ -22,6 +22,7 @@ AIStor ObjectStore minio-aistor (HDD sdb3, aistor-local-xfs-retain)
| `scheduled-backup.yaml` | 매일 18:00 UTC(03:00 KST) base backup. 만들자마자 한 번 |
| `object-storage-counterpart.yaml` | `object-storage` 쪽 ingress: platform-data의 PostgreSQL 파드 → AIStor 9000 |
| `bucket-policy.json` | 백업 사용자가 자기 bucket만 읽고 쓰는 정책 |
| `prometheus-rule.yaml` | 백업 경보 둘(아래). 관측 core 렌더러가 백업 규칙을 core 범위 밖으로 두므로 여기 둔다 |
| `restore-drill/cluster.yaml` | 복원 리허설용 임시 Cluster |
| `../../scripts/bootstrap/create-postgres-backup-credentials.sh` | bucket·정책·사용자·Secret 생성 |
@@ -84,7 +85,7 @@ AIStor ObjectStore minio-aistor (HDD sdb3, aistor-local-xfs-retain)
4. 매일 백업을 건다. 만들자마자 첫 base backup이 돈다.
```bash
kubectl apply -k services/platform-postgres-backup
kubectl apply -k services/platform-postgres-backup # ScheduledBackup, AIStor ingress, 경보
kubectl -n platform-data get backup -w
```
@@ -93,7 +94,7 @@ AIStor ObjectStore minio-aistor (HDD sdb3, aistor-local-xfs-retain)
5. 아래 「복원 리허설」을 한 번 한다.
경보(`services/observability/rules/platform/core-rules.yaml`의 `platform.backup`):
경보(`prometheus-rule.yaml`, 4단계에서 같이 들어간다. platform-data는 Prometheus가 규칙을 읽는 namespace다):
- `PlatformPostgresBackupMissing`: 26시간 안에 끝난 base backup이 없다. 위 4단계 전까지는 계속
울린다(백업이 정말 없으니까).
@@ -104,7 +105,7 @@ AIStor ObjectStore minio-aistor (HDD sdb3, aistor-local-xfs-retain)
### 되돌리기
```bash
kubectl delete -k services/platform-postgres-backup # ScheduledBackup과 ingress
kubectl delete -k services/platform-postgres-backup # ScheduledBackup, ingress, 경보
kubectl -n platform-data patch cluster platform-postgres --type json -p '[{"op":"remove","path":"/spec/backup"}]'
```
@@ -8,6 +8,7 @@ kind: Kustomization
resources:
- object-storage-counterpart.yaml
- scheduled-backup.yaml
- prometheus-rule.yaml
labels:
- pairs:
@@ -0,0 +1,46 @@
# Backup alerts for platform-postgres. They live with the backup and not in
# the observability core, whose renderer keeps backup rules out of its scope
# (scripts/validate/render-observability-core.sh, no_unscoped_monitors_rules).
# Prometheus loads PrometheusRules from namespaces labelled
# observability.hyeonworks.com/enabled=true, which platform-data is.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-postgres-backup
namespace: platform-data
labels:
app.kubernetes.io/name: platform-postgres
app.kubernetes.io/component: backup
observability.hyeonworks.com/instance: home
spec:
groups:
- name: platform.backup
rules:
# cnpg_collector_last_available_backup_timestamp is 0 until the first
# base backup, so a cluster without backups fires as well.
- alert: PlatformPostgresBackupMissing
expr: >-
time() - max by (job, namespace) (
cnpg_collector_last_available_backup_timestamp{job="platform-postgres",namespace="platform-data"}
) > 26 * 3600
for: 30m
labels:
severity: critical
annotations:
summary: platform-postgres에 26시간 안에 끝난 백업이 없음
description: platform-postgres의 마지막 성공한 base backup이 26시간보다 오래됐거나 없습니다. 매일 03:00 KST 백업(ScheduledBackup platform-postgres-daily)을 확인하세요.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/platform-core/src/branch/main/services/platform-postgres-backup/README.md
# A failing archive_command keeps WAL in pg_wal, which then fills the
# PostgreSQL volume.
- alert: PlatformPostgresWALArchiveBacklog
expr: >-
max by (job, namespace) (
cnpg_collector_pg_wal_archive_status{job="platform-postgres",namespace="platform-data",value="ready"}
) > 10
for: 15m
labels:
severity: critical
annotations:
summary: platform-postgres의 WAL이 보관소로 올라가지 않음
description: 보관을 기다리는 WAL 파일이 10개를 넘은 채 15분이 지났습니다. 보관이 막히면 WAL이 PostgreSQL 볼륨에 쌓여 디스크가 찹니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/platform-core/src/branch/main/services/platform-postgres-backup/README.md