Commit Graph
3 Commits
Author SHA1 Message Date
DongHyeonka ba3dddc139 feat(validate): add restore drills for the k3s and Gitea backups
A backup nobody has restored is not known to work. The 2026-08 k3s
recovery bundle never got its isolated restore drill.

k3s-datastore-restore-drill.sh downloads the latest archive, checks its
sha256 and integrity, starts the same k3s version in a network-less
throwaway container on the copy, and checks the API comes up, the
Secret count is close to live and every Secret decrypts with the
token. gitea-backup-restore-drill.sh extracts the latest dump and
checks app.ini (INTERNAL_TOKEN, SECRET_KEY), git fsck on every
repository and the repository count against live Gitea. Both accept a
local copy and write a drill timestamp metric.

Risk: read-only for the cluster. The k3s drill runs the restored
control plane with --network none, so its controllers reach nothing.

Verification (desktop):
- test-k3s-datastore-restore-drill.sh: a fresh k3s with three test
  Secrets (stored encrypted, k8s:enc:) restores with all Secrets
  decrypted in 7s; before folding the WAL the test saw 2 of 3 rows,
  which is why the real backup uses sqlite3 .backup.
- gitea-backup-restore-drill.sh on a live repository dump: 29
  repositories, 0 fsck failures, 29 live; the dump was deleted after.
2026-11-12 01:13:02 +09:00
DongHyeonka 3eafb7e1d8 feat(validate): add a live drift check against the repository renders
The 2026-11-11 review found a pg_hba rule that existed only in the
cluster by rendering every root by hand and running kubectl diff.
Nothing does that regularly, so drift is found by accident.

live-drift-check.sh renders all live roots (Phase 1 and 2, admin
services, observability core, the Traefik trust overlay, the tenant,
PriorityClass and keycloak policy roots), strips Secrets and runs
kubectl diff, which is a server-side dry run. It prints one line per
drifted object, honours a dated accepted-drift list and can write
Prometheus textfile metrics for a later alert.

Risk: read-only. It consumes the render scripts' verified-output-dir
handoff and deletes those directories on exit; the README notes this
second consumer. Secret values are not compared.

Verification (desktop, before this series is applied): 17 drifted
objects, matching the review plus the new roots; exit 1; the textfile
holds platform_live_drift_objects 17.
2026-11-12 00:29:18 +09:00
donghyeon-ka 16c337bcc9 Add platform infrastructure configuration 2026-08-28 17:35:41 +09:00