Commit Graph
14 Commits
Author SHA1 Message Date
DongHyeonka 050a4de235 fix(validate): render the drift check from a clean working-tree copy
In a long-lived checkout the observability renderer finds an ignored
.helm/charts/<chart>-<version> cache left by an earlier render and
refuses ("exact chart cache already exists"), so live-drift-check.sh
fails with exit 2 in /home/donghyeon/workspace/platform, the path the
runbooks use.

When run inside a git working tree, copy the tracked and untracked
(not ignored, not deleted) files to the drift work directory and
render from there. Local edits are still checked; ignored caches are
not carried over. Outside a git tree (the daily timer renders a
git archive of origin/main) nothing changes.

Also accept the one drift this exposed: the kube-prometheus-stack
PrometheusRule still has the two Kubelet certificate alerts that PR #6
disabled in values; it goes away when the observability core is
re-applied in the restart window (until 2026-11-30).

Risk: read-only, as before. The copy is removed with the work
directory.

Verification (desktop, fresh clone of main 8b019d2 with an ignored
services/observability/tempo/.helm/charts/tempo-2.2.3 created):
- old script: "REJECT: exact chart cache already exists", exit 2.
- new script as an uncommitted edit in that clone: SUMMARY drift=1
  accepted=12 (the PrometheusRule above), exit 1.
- new script from a git archive export, as the timer runs it: same
  summary.
2026-11-12 02:00:20 +09:00
donghyeon.kang 28595c47e2 Merge pull request 'feat(keycloak): pin images and attach PriorityClasses for the restart window' (#7) from donghyeon.kang/w6-restart-window-pins into main 2026-11-11 16:48:42 +00:00
donghyeon.kang 00a6680b76 Merge pull request 'feat(backup): back up k3s and Gitea to AIStor with alerts and drills' (#6) from donghyeon.kang/w6-host-backups into main 2026-11-11 16:48:31 +00:00
DongHyeonka cb78ccdb0b feat(backup): store the Gitea version next to each dump
A Gitea dump must be restored with the Gitea version that wrote it, and
a restore checks the repository count against the dump. Neither was
recorded outside the archive, so the restore runbook had nothing to
compare with.

Upload a small .meta file with each dump: creation time, the
Deployment image, the gitea --version output and the repository count.
docs/runbooks/gitea-restore.md reads it.

Risk: none for the backup itself; the file holds no secret.

Verification: bash -n; kubectl -n gitea exec deploy/gitea -c gitea --
gitea --version prints "gitea version 1.27.0 built with ..." on the
desktop, so the third field is the version.
2026-11-12 01:38:24 +09:00
DongHyeonka de600a0bb1 feat(priority): attach PriorityClasses to Keycloak, Gitea, Loki, Tempo
The PriorityClasses exist in the repository but no pod uses them, so
every platform pod still evicts at priority 0.

Attach platform-critical to Keycloak (CR spec.scheduling) and Gitea
(chart priorityClassName), and platform-low to Loki (singleBinary) and
Tempo. The Keycloak CR also gets spec.image at the running 26.7.0
digest with startOptimized: false: with spec.image the operator
otherwise assumes a pre-built optimized image, and this is the stock
one. Grafana and pgAdmin have no top-level priorityClassName in their
charts and are left for later; platform-postgres is W1's file and
tech-log belongs to its deploy/ directory.

Risk: applying restarts Keycloak, Gitea, Loki and Tempo once, so it
belongs in the k3s restart window, after the PriorityClasses exist (a
pod that names a missing class is rejected). The accepted-drift list
holds these objects until 2026-11-30.

Verification (desktop): render-phase1/2 and render-observability-core
pass with the four priorityClassName values; kubectl diff (server
dry-run) against live shows only these fields.
2026-11-12 01:25:24 +09:00
DongHyeonka 9751db0e97 feat(keycloak): pin the operator source and both images by digest
The Keycloak Operator remote resource is pinned to the tag 26.7.0, and
that tag's content changed once (live build-timestamp 2026-07-09,
later renders 2026-07-23). Both images use the 26.7.0 tag with
imagePullPolicy Always, so every restart depends on quay.io and could
pull different bits.

Pin the remote resource to the tag's commit, the operator image and
RELATED_IMAGE_KEYCLOAK to the digests that were running on 2026-11-12
(pod imageID) and set the operator to IfNotPresent. The env value is
set with a strategic merge patch keyed by name; a JSON patch by list
index hit KUBERNETES_NAMESPACE instead. render-phase2.sh and
apply-keycloak.sh check the commit and the digest.

Risk: applying restarts the Keycloak operator once (restart window).
Keycloak itself keeps running.

Verification (desktop): render-phase2.sh passes; the rendered
Deployment has the pinned image, IfNotPresent and the digest in
RELATED_IMAGE_KEYCLOAK with the other env entries unchanged.
2026-11-12 01:25:24 +09:00
DongHyeonka 9e9ea94461 feat(backup): keep k3s datastore archives outside AIStor on the HDD
AIStor runs inside the k3s cluster that the datastore archive restores.
After losing the whole cluster the AIStor copy cannot be read until k3s
is running again, which is what the archive is for.

Also keep the seven newest archives in
/srv/k3s/aistor/host-backups/k3s-datastore (root 0700) on the same HDD
filesystem (sdb3). The directory is next to AIStor's drive directory
data/, so AIStor does not list it as a bucket, and it is readable by
mounting sdb3 alone. A metric reports how many local copies exist.

Rejected: the NTFS archive partition (sdb2). It has no Unix
permissions and Windows on the same machine can read it; the archive
holds the server token.

Risk: the files live inside the AIStor PersistentVolume root, which is
mounted into the MinIO pod (uid 1000); mode 0700 root keeps the pod
from reading them.

Verification: bash -n; /srv/k3s/aistor contains only data/ (2026-11-12).
2026-11-12 01:14:45 +09:00
DongHyeonka ba3dddc139 feat(validate): add restore drills for the k3s and Gitea backups
A backup nobody has restored is not known to work. The 2026-08 k3s
recovery bundle never got its isolated restore drill.

k3s-datastore-restore-drill.sh downloads the latest archive, checks its
sha256 and integrity, starts the same k3s version in a network-less
throwaway container on the copy, and checks the API comes up, the
Secret count is close to live and every Secret decrypts with the
token. gitea-backup-restore-drill.sh extracts the latest dump and
checks app.ini (INTERNAL_TOKEN, SECRET_KEY), git fsck on every
repository and the repository count against live Gitea. Both accept a
local copy and write a drill timestamp metric.

Risk: read-only for the cluster. The k3s drill runs the restored
control plane with --network none, so its controllers reach nothing.

Verification (desktop):
- test-k3s-datastore-restore-drill.sh: a fresh k3s with three test
  Secrets (stored encrypted, k8s:enc:) restores with all Secrets
  decrypted in 7s; before folding the WAL the test saw 2 of 3 rows,
  which is why the real backup uses sqlite3 .backup.
- gitea-backup-restore-drill.sh on a live repository dump: 29
  repositories, 0 fsck failures, 29 live; the dump was deleted after.
2026-11-12 01:13:02 +09:00
DongHyeonka 012452cf78 feat(observability): alert on host backups, drift and k3s serving certs
Backups, the daily drift check and the k3s serving certificates have no
alert. k3s renews certificates only when it starts within 120 days of
expiry, and the two kube-prometheus-stack kubelet certificate alerts
can never fire because the k3s kubelet exports no
kubelet_certificate_manager_* metric.

Add a PrometheusRule (platform-host-checks) for backup age, failed
runs, missing metrics, drift left for 6h, a stale drift check, and
serving certificate expiry (warning under 100 days, critical under 14).
Add the two remaining host timers (certificate expiry via openssl on
6443 and 10250; the drift check against origin/main) and an installer
that copies the scripts to a root-owned path, creates the textfile and
work directories and enables all four timers. Disable the two dead
kubelet certificate rules in kube-prometheus-stack.

Risk: the metrics need the node-exporter textfile collector, a one-line
change in W1's node-exporter values that was asked of W1. Until then
PlatformHostBackupMetricsAbsent fires after a day, which is the point.

Verification (desktop):
- test-platform-host-checks-alerts.sh: promtool check and unit tests
  pass; with the k3s threshold changed from 2 to 5 days they fail.
- render-observability-core.sh passes; the render has no Kubelet*
  certificate alerts and still has both KubeClientCertificateExpiration.
- k3s-cert-expiry.sh as the desktop user reads both certificates.
- install-host-checks.sh --check: systemd-analyze verify is clean.
2026-11-12 01:13:02 +09:00
DongHyeonka 77daa39a41 feat(backup): back up the k3s datastore and Gitea to AIStor
Nothing backs up the k3s datastore or Gitea's repositories, packages and
app.ini. W1 covers platform-postgres only. One SSD failure loses the
source, CI artefacts, published libraries and every cluster Secret.

Add two root systemd timers that write to a new AIStor bucket on the
HDD (platform-host-backups, same AIStor as W1's postgres backups):
- k3s datastore, daily: sqlite3 .backup of the live state.db plus the
  server token (k3s backup/restore docs: the token is needed to read
  it back), integrity_check, tar.gz, sha256, upload. Off-host copy is
  prepared behind two variables (gpg recipient file, rclone target) and
  stays off until the owner chooses a place.
- Gitea, weekly: gitea dump --skip-db streamed from the live container,
  listing checked for app.ini and repositories, sha256, upload. The DB
  is in the CNPG backup.
Each run writes textfile metrics for the alerts in the next commit.
create-host-backup-credentials.sh makes the bucket (50GiB quota,
lifecycle 56 days for gitea/, 30 days for k3s-datastore/), a
bucket-scoped policy and user, and a root-only env file; no key is
printed and mc gets keys only through a 0600 env file.

Rejected: in-cluster CronJobs. The datastore is a root-owned host file
and the dump needs the Gitea DB connection; a hostPath root pod or a
second Gitea-labelled pod is a larger privilege than a root timer.

Risk: not installed here (owner sudo steps). The k3s archive holds the
token; together with the datastore it decrypts every Secret. It lives
in a bucket only the backup user can read.

Verification (desktop): a dump streamed with --file - is a valid
tar.gz with app.ini; repositories appear as repos/<owner>/<name>.git
(29, matching the live count). The scripts parse with bash -n.
2026-11-12 01:13:02 +09:00
DongHyeonka 457d88e293 feat(bootstrap): apply the k3s restart bundle with one restart
The restart bundle needs root, touches k3s, kubelet and systemd files,
and must not leave the cluster half-configured. The W1 time-sync
drop-ins need a restart window too.

apply-k3s-restart-bundle.sh --check parses the files and prints the
install plan without root. --execute checks the cluster is Ready,
backs up every file it replaces plus a sqlite .backup of the datastore
and the token, installs the W6 files and W1's two time-sync drop-ins
as they are, enables systemd-time-wait-sync, restarts k3s once with
--no-block (the unit has TimeoutStartSec=0), then verifies the kubelet
configz values, the audit log, that the default namespace rejects a
privileged pod, three health URLs and the time-sync unit. Any failure
restores the backup and restarts again. Last, it creates the
local-storage .skip file and clears the local-path default annotation.

Risk: --execute was not run (owner sudo step). The API is unavailable
for one to two minutes; containers keep running (KillMode=process). The
script refuses to run until W1's host files are on main.

Verification (desktop, non-root):
- --check without the W1 files: MISSING ... (W1), exit 1.
- --check with the W1 files overlaid: parse and invariants pass, plan
  printed, exit 0.
- --execute as non-root: "--execute needs root (sudo)".
2026-11-12 00:49:58 +09:00
DongHyeonka 3eafb7e1d8 feat(validate): add a live drift check against the repository renders
The 2026-11-11 review found a pg_hba rule that existed only in the
cluster by rendering every root by hand and running kubectl diff.
Nothing does that regularly, so drift is found by accident.

live-drift-check.sh renders all live roots (Phase 1 and 2, admin
services, observability core, the Traefik trust overlay, the tenant,
PriorityClass and keycloak policy roots), strips Secrets and runs
kubectl diff, which is a server-side dry run. It prints one line per
drifted object, honours a dated accepted-drift list and can write
Prometheus textfile metrics for a later alert.

Risk: read-only. It consumes the render scripts' verified-output-dir
handoff and deletes those directories on exit; the README notes this
second consumer. Secret values are not compared.

Verification (desktop, before this series is applied): 17 drifted
objects, matching the review plus the new roots; exit 1; the textfile
holds platform_live_drift_objects 17.
2026-11-12 00:29:18 +09:00
DongHyeonka ebdf33f072 feat(tenants): declare the tech-log objects inside platform namespaces
The tech-log tenant needs objects inside platform namespaces: its
database and role, ingress to Postgres and AIStor, blackbox probes,
alert rules, a dashboard and two observability NetworkPolicies. They
were only ever applied with kubectl and no repository holds them, so a
deleted object cannot be recreated and a platform re-apply can drop
tech-log access without anyone noticing.

Declare them in services/tenants/techlog, copied from each object's
last-applied configuration. The dashboard follows the repository's
dashboard pattern (JSON file + configMapGenerator, same name, same
labels). The hyeonworks-prod Namespace moves here as the tenant
boundary and gets PSA enforce=baseline, warn/audit=restricted.

Rejected: enforce=restricted now. A server dry-run shows all three
tech-log pods would violate it; baseline has zero violations.

Risk: the render matches the live objects exactly except the added PSA
labels. The pg_hba rules and the Secret stay out of this root (Cluster
owner and Git policy).

Verification (desktop):
- render-tenants-and-policies.sh passes.
- kubectl diff of the render shows only the Namespace PSA labels.
- kubectl label --dry-run=server enforce=baseline: no warnings.
2026-11-12 00:29:18 +09:00
donghyeon-ka 16c337bcc9 Add platform infrastructure configuration 2026-08-28 17:35:41 +09:00