Commit Graph
3 Commits
Author SHA1 Message Date
DongHyeonka 90ab8f4e65 feat(postgres): back up platform-postgres to AIStor every day
platform-postgres holds the Gitea, Keycloak and tech-log databases and
had no backup: no spec.backup, no ScheduledBackup, and its only copy
is a local PV on the root SSD. A lost SSD or a bad migration had no
way back.

Add services/platform-postgres-backup: a merge patch that gives the
live Cluster barman-cloud WAL archiving and base backups to
s3://platform-postgres-backups on the AIStor ObjectStore (HDD sdb3)
with 30-day retention and gzip, a daily 03:00 KST ScheduledBackup that
also runs once at creation, the object-storage ingress for the
PostgreSQL pods, the bucket policy, and a restore drill Cluster that
only reads the bucket. scripts/bootstrap/create-postgres-backup-
credentials.sh creates the bucket (100GiB quota), policy, user and
the platform-data Secret through a restricted AIStor client Pod,
proves the user can write its bucket and cannot list loki, and prints
no key. Add PlatformPostgresBackupMissing (no base backup in 26h) and
PlatformPostgresWALArchiveBacklog (more than 10 WAL files waiting for
15m; a stuck archive fills the SSD volume).

The native barmanObjectStore is deprecated since CNPG 1.26 and the
operator warns it is removed in 1.31.0. The Barman Cloud Plugin needs
cert-manager, which this cluster does not have; installing two
operators before the first backup is slower and riskier. The bucket
layout is the same, so the plugin can take over before CNPG 1.31.
Volume snapshots do not work on the static local PV.

Risk: the live Cluster has pg_hba lines for techlog that the Phase 1
source lacks, so re-applying services/platform-postgres would cut the
tech-log backend off. The README applies only spec.backup with
kubectl patch. Backups stay on the same desktop; an off-machine copy
is not part of this change. Nothing was applied to the cluster.

Verification: kubectl apply --dry-run=server accepts the
ScheduledBackup, the NetworkPolicy and the drill Cluster; kubectl
patch --dry-run=server keeps both techlog pg_hba lines and adds
spec.backup. The row-count comparison from the drill steps ran
read-only on production (keycloak 100, techlog 44, gitea 116 tables).
mc command syntax was checked with the pinned AIStor client offline.
The two new alert tests failed against the old rules and pass now.
bash -n passes for the script; shellcheck is not installed on the
laptop or the desktop.
2026-11-11 23:20:38 +09:00
DongHyeonka 5d58160c89 feat(observability): alert on the root disk before kubelet acts
The kubelet starts deleting unused images at 85% root use and evicts
pods at 90%. The root disk warning fired at 85% and the critical at
90%, so both arrived when the kubelet was already acting. On
2026-11-09 the warning fired for 33 hours, then an eviction stopped
the tech-log pods, the image GC deleted their imagePullPolicy: Never
images, and production went down.

Move the warning to 80% for 10m and the critical to 85% for 5m, and
add PlatformRootFilesystemFillingFast (critical, for 15m): the 6h
linear trend of free bytes falls below 10% of the disk, the eviction
point, within 6 hours. On 2026-11-07 the disk lost 17G in 9 hours;
this forecast would have fired hours before the 80% warning. The
info alert at 75% stays.

Add scripts/validate/test-platform-disk-alerts.sh, promtool unit
tests for the usage bands and the forecast with the pinned Prometheus
image. The existing rules test needs live metric inventories, so this
one runs offline.

Risk: the live rules change only when the observability access apply
runs; this commit does not touch the cluster. The forecast can fire
on a large build that the BuildKit GC removes minutes later; the 15m
hold and the 6h window damp that.

Verification: the new test failed against the old rules (no alert at
81%, 86% or for a 1-per-minute fill) and passes after the change;
promtool check rules finds 9 rules. test-observability-alerting.sh
passes its static contract, then stops at the helm render because
helm v3.19.4 is not installed on the desktop.
2026-11-11 21:51:59 +09:00
donghyeon-ka 16c337bcc9 Add platform infrastructure configuration 2026-08-28 17:35:41 +09:00