The one PostgreSQL instance holds the Gitea, Keycloak and tech-log
databases. It is Burstable at priority 0 with oom_score_adj 985, so
under memory pressure the kernel kills it before almost anything else,
and an eviction treats it like Grafana (k3s maturity review
2026-11-11, 7.2 items 2 and 3).
Set priorityClassName platform-critical (services/priority-classes)
and raise the requests to the existing limits, CPU 2 and memory 4Gi,
so the pod is Guaranteed and its oom_score_adj becomes -997. The
backup README now gives the restart, not pg_hba, as the reason to add
spec.backup with kubectl patch.
Risk: applying this changes the pod spec, so CloudNativePG restarts
the single instance and the three databases are down for that time.
It belongs in the k3s restart window. Node requests grow by 1.75 CPU
and 3.5Gi; peak use over the last 6 hours was 0.025 CPU and 195MB.
Verification: kubectl diff of the rendered build root against the live
cluster shows only priorityClassName, the two requests and the
DatabaseRole defaults that were already there; no pg_hba line differs.
render-phase1.sh passes.
The observability core renderer rejects any alert or record whose
name contains "backup" (no_unscoped_monitors_rules): backup
monitoring is outside the observability core by design. f19ce54 put
PlatformPostgresBackupMissing and PlatformPostgresWALArchiveBacklog in
core-rules.yaml, so the pinned renderer failed with "out-of-scope
monitor or rule is present".
Move both alerts unchanged into
services/platform-postgres-backup/prometheus-rule.yaml in
platform-data, which Prometheus reads because the namespace has
observability.hyeonworks.com/enabled=true and the rule has
observability.hyeonworks.com/instance=home. They now point their
runbook_url at the backup README and are applied with the
ScheduledBackup. Renaming them to dodge the filter was rejected: it
would keep backup rules inside a scope that excludes them on purpose.
Verification: test-platform-disk-alerts.sh loads both rule files and
passes; promtool checks 13 rules. render-observability-core.sh and
test-render-observability-core.sh pass, and kubectl apply
--dry-run=server accepts the backup kustomization with the rule.
scan-platform-sensitive-source.sh stops at its observability-core
renderer with RC=23 here and on an origin/main snapshot alike, so it
says nothing about this change.
Fixes: f19ce54740 ("feat(postgres): back up platform-postgres to AIStor every day")
platform-postgres holds the Gitea, Keycloak and tech-log databases and
had no backup: no spec.backup, no ScheduledBackup, and its only copy
is a local PV on the root SSD. A lost SSD or a bad migration had no
way back.
Add services/platform-postgres-backup: a merge patch that gives the
live Cluster barman-cloud WAL archiving and base backups to
s3://platform-postgres-backups on the AIStor ObjectStore (HDD sdb3)
with 30-day retention and gzip, a daily 03:00 KST ScheduledBackup that
also runs once at creation, the object-storage ingress for the
PostgreSQL pods, the bucket policy, and a restore drill Cluster that
only reads the bucket. scripts/bootstrap/create-postgres-backup-
credentials.sh creates the bucket (100GiB quota), policy, user and
the platform-data Secret through a restricted AIStor client Pod,
proves the user can write its bucket and cannot list loki, and prints
no key. Add PlatformPostgresBackupMissing (no base backup in 26h) and
PlatformPostgresWALArchiveBacklog (more than 10 WAL files waiting for
15m; a stuck archive fills the SSD volume).
The native barmanObjectStore is deprecated since CNPG 1.26 and the
operator warns it is removed in 1.31.0. The Barman Cloud Plugin needs
cert-manager, which this cluster does not have; installing two
operators before the first backup is slower and riskier. The bucket
layout is the same, so the plugin can take over before CNPG 1.31.
Volume snapshots do not work on the static local PV.
Risk: the live Cluster has pg_hba lines for techlog that the Phase 1
source lacks, so re-applying services/platform-postgres would cut the
tech-log backend off. The README applies only spec.backup with
kubectl patch. Backups stay on the same desktop; an off-machine copy
is not part of this change. Nothing was applied to the cluster.
Verification: kubectl apply --dry-run=server accepts the
ScheduledBackup, the NetworkPolicy and the drill Cluster; kubectl
patch --dry-run=server keeps both techlog pg_hba lines and adds
spec.backup. The row-count comparison from the drill steps ran
read-only on production (keycloak 100, techlog 44, gitea 116 tables).
mc command syntax was checked with the pinned AIStor client offline.
The two new alert tests failed against the old rules and pass now.
bash -n passes for the script; shellcheck is not installed on the
laptop or the desktop.