8 Commits
Author SHA1 Message Date
DongHyeonka f19ce54740 feat(postgres): back up platform-postgres to AIStor every day
platform-postgres holds the Gitea, Keycloak and tech-log databases and
had no backup: no spec.backup, no ScheduledBackup, and its only copy
is a local PV on the root SSD. A lost SSD or a bad migration had no
way back.

Add services/platform-postgres-backup: a merge patch that gives the
live Cluster barman-cloud WAL archiving and base backups to
s3://platform-postgres-backups on the AIStor ObjectStore (HDD sdb3)
with 30-day retention and gzip, a daily 03:00 KST ScheduledBackup that
also runs once at creation, the object-storage ingress for the
PostgreSQL pods, the bucket policy, and a restore drill Cluster that
only reads the bucket. scripts/bootstrap/create-postgres-backup-
credentials.sh creates the bucket (100GiB quota), policy, user and
the platform-data Secret through a restricted AIStor client Pod,
proves the user can write its bucket and cannot list loki, and prints
no key. Add PlatformPostgresBackupMissing (no base backup in 26h) and
PlatformPostgresWALArchiveBacklog (more than 10 WAL files waiting for
15m; a stuck archive fills the SSD volume).

The native barmanObjectStore is deprecated since CNPG 1.26 and the
operator warns it is removed in 1.31.0. The Barman Cloud Plugin needs
cert-manager, which this cluster does not have; installing two
operators before the first backup is slower and riskier. The bucket
layout is the same, so the plugin can take over before CNPG 1.31.
Volume snapshots do not work on the static local PV.

Risk: the live Cluster has pg_hba lines for techlog that the Phase 1
source lacks, so re-applying services/platform-postgres would cut the
tech-log backend off. The README applies only spec.backup with
kubectl patch. Backups stay on the same desktop; an off-machine copy
is not part of this change. Nothing was applied to the cluster.

Verification: kubectl apply --dry-run=server accepts the
ScheduledBackup, the NetworkPolicy and the drill Cluster; kubectl
patch --dry-run=server keeps both techlog pg_hba lines and adds
spec.backup. The row-count comparison from the drill steps ran
read-only on production (keycloak 100, techlog 44, gitea 116 tables).
mc command syntax was checked with the pinned AIStor client offline.
The two new alert tests failed against the old rules and pass now.
bash -n passes for the script; shellcheck is not installed on the
laptop or the desktop.
2026-11-12 07:28:03 +09:00
donghyeon.kang 28595c47e2 Merge pull request 'feat(keycloak): pin images and attach PriorityClasses for the restart window' (#7) from donghyeon.kang/w6-restart-window-pins into main 2026-11-11 16:48:42 +00:00
donghyeon.kang 00a6680b76 Merge pull request 'feat(backup): back up k3s and Gitea to AIStor with alerts and drills' (#6) from donghyeon.kang/w6-host-backups into main 2026-11-11 16:48:31 +00:00
DongHyeonka 9751db0e97 feat(keycloak): pin the operator source and both images by digest
The Keycloak Operator remote resource is pinned to the tag 26.7.0, and
that tag's content changed once (live build-timestamp 2026-07-09,
later renders 2026-07-23). Both images use the 26.7.0 tag with
imagePullPolicy Always, so every restart depends on quay.io and could
pull different bits.

Pin the remote resource to the tag's commit, the operator image and
RELATED_IMAGE_KEYCLOAK to the digests that were running on 2026-11-12
(pod imageID) and set the operator to IfNotPresent. The env value is
set with a strategic merge patch keyed by name; a JSON patch by list
index hit KUBERNETES_NAMESPACE instead. render-phase2.sh and
apply-keycloak.sh check the commit and the digest.

Risk: applying restarts the Keycloak operator once (restart window).
Keycloak itself keeps running.

Verification (desktop): render-phase2.sh passes; the rendered
Deployment has the pinned image, IfNotPresent and the digest in
RELATED_IMAGE_KEYCLOAK with the other env entries unchanged.
2026-11-12 01:25:24 +09:00
DongHyeonka 012452cf78 feat(observability): alert on host backups, drift and k3s serving certs
Backups, the daily drift check and the k3s serving certificates have no
alert. k3s renews certificates only when it starts within 120 days of
expiry, and the two kube-prometheus-stack kubelet certificate alerts
can never fire because the k3s kubelet exports no
kubelet_certificate_manager_* metric.

Add a PrometheusRule (platform-host-checks) for backup age, failed
runs, missing metrics, drift left for 6h, a stale drift check, and
serving certificate expiry (warning under 100 days, critical under 14).
Add the two remaining host timers (certificate expiry via openssl on
6443 and 10250; the drift check against origin/main) and an installer
that copies the scripts to a root-owned path, creates the textfile and
work directories and enables all four timers. Disable the two dead
kubelet certificate rules in kube-prometheus-stack.

Risk: the metrics need the node-exporter textfile collector, a one-line
change in W1's node-exporter values that was asked of W1. Until then
PlatformHostBackupMetricsAbsent fires after a day, which is the point.

Verification (desktop):
- test-platform-host-checks-alerts.sh: promtool check and unit tests
  pass; with the k3s threshold changed from 2 to 5 days they fail.
- render-observability-core.sh passes; the render has no Kubelet*
  certificate alerts and still has both KubeClientCertificateExpiration.
- k3s-cert-expiry.sh as the desktop user reads both certificates.
- install-host-checks.sh --check: systemd-analyze verify is clean.
2026-11-12 01:13:02 +09:00
DongHyeonka 77daa39a41 feat(backup): back up the k3s datastore and Gitea to AIStor
Nothing backs up the k3s datastore or Gitea's repositories, packages and
app.ini. W1 covers platform-postgres only. One SSD failure loses the
source, CI artefacts, published libraries and every cluster Secret.

Add two root systemd timers that write to a new AIStor bucket on the
HDD (platform-host-backups, same AIStor as W1's postgres backups):
- k3s datastore, daily: sqlite3 .backup of the live state.db plus the
  server token (k3s backup/restore docs: the token is needed to read
  it back), integrity_check, tar.gz, sha256, upload. Off-host copy is
  prepared behind two variables (gpg recipient file, rclone target) and
  stays off until the owner chooses a place.
- Gitea, weekly: gitea dump --skip-db streamed from the live container,
  listing checked for app.ini and repositories, sha256, upload. The DB
  is in the CNPG backup.
Each run writes textfile metrics for the alerts in the next commit.
create-host-backup-credentials.sh makes the bucket (50GiB quota,
lifecycle 56 days for gitea/, 30 days for k3s-datastore/), a
bucket-scoped policy and user, and a root-only env file; no key is
printed and mc gets keys only through a 0600 env file.

Rejected: in-cluster CronJobs. The datastore is a root-owned host file
and the dump needs the Gitea DB connection; a hostPath root pod or a
second Gitea-labelled pod is a larger privilege than a root timer.

Risk: not installed here (owner sudo steps). The k3s archive holds the
token; together with the datastore it decrypts every Secret. It lives
in a bucket only the backup user can read.

Verification (desktop): a dump streamed with --file - is a valid
tar.gz with app.ini; repositories appear as repos/<owner>/<name>.git
(29, matching the live count). The scripts parse with bash -n.
2026-11-12 01:13:02 +09:00
DongHyeonka 457d88e293 feat(bootstrap): apply the k3s restart bundle with one restart
The restart bundle needs root, touches k3s, kubelet and systemd files,
and must not leave the cluster half-configured. The W1 time-sync
drop-ins need a restart window too.

apply-k3s-restart-bundle.sh --check parses the files and prints the
install plan without root. --execute checks the cluster is Ready,
backs up every file it replaces plus a sqlite .backup of the datastore
and the token, installs the W6 files and W1's two time-sync drop-ins
as they are, enables systemd-time-wait-sync, restarts k3s once with
--no-block (the unit has TimeoutStartSec=0), then verifies the kubelet
configz values, the audit log, that the default namespace rejects a
privileged pod, three health URLs and the time-sync unit. Any failure
restores the backup and restarts again. Last, it creates the
local-storage .skip file and clears the local-path default annotation.

Risk: --execute was not run (owner sudo step). The API is unavailable
for one to two minutes; containers keep running (KillMode=process). The
script refuses to run until W1's host files are on main.

Verification (desktop, non-root):
- --check without the W1 files: MISSING ... (W1), exit 1.
- --check with the W1 files overlaid: parse and invariants pass, plan
  printed, exit 0.
- --execute as non-root: "--execute needs root (sudo)".
2026-11-12 00:49:58 +09:00
donghyeon-ka 16c337bcc9 Add platform infrastructure configuration 2026-08-28 17:35:41 +09:00