In a long-lived checkout the observability renderer finds an ignored
.helm/charts/<chart>-<version> cache left by an earlier render and
refuses ("exact chart cache already exists"), so live-drift-check.sh
fails with exit 2 in /home/donghyeon/workspace/platform, the path the
runbooks use.
When run inside a git working tree, copy the tracked and untracked
(not ignored, not deleted) files to the drift work directory and
render from there. Local edits are still checked; ignored caches are
not carried over. Outside a git tree (the daily timer renders a
git archive of origin/main) nothing changes.
Also accept the one drift this exposed: the kube-prometheus-stack
PrometheusRule still has the two Kubelet certificate alerts that PR #6
disabled in values; it goes away when the observability core is
re-applied in the restart window (until 2026-11-30).
Risk: read-only, as before. The copy is removed with the work
directory.
Verification (desktop, fresh clone of main 8b019d2 with an ignored
services/observability/tempo/.helm/charts/tempo-2.2.3 created):
- old script: "REJECT: exact chart cache already exists", exit 2.
- new script as an uncommitted edit in that clone: SUMMARY drift=1
accepted=12 (the PrometheusRule above), exit 1.
- new script from a git archive export, as the timer runs it: same
summary.
The PriorityClasses exist in the repository but no pod uses them, so
every platform pod still evicts at priority 0.
Attach platform-critical to Keycloak (CR spec.scheduling) and Gitea
(chart priorityClassName), and platform-low to Loki (singleBinary) and
Tempo. The Keycloak CR also gets spec.image at the running 26.7.0
digest with startOptimized: false: with spec.image the operator
otherwise assumes a pre-built optimized image, and this is the stock
one. Grafana and pgAdmin have no top-level priorityClassName in their
charts and are left for later; platform-postgres is W1's file and
tech-log belongs to its deploy/ directory.
Risk: applying restarts Keycloak, Gitea, Loki and Tempo once, so it
belongs in the k3s restart window, after the PriorityClasses exist (a
pod that names a missing class is rejected). The accepted-drift list
holds these objects until 2026-11-30.
Verification (desktop): render-phase1/2 and render-observability-core
pass with the four priorityClassName values; kubectl diff (server
dry-run) against live shows only these fields.
The Keycloak Operator remote resource is pinned to the tag 26.7.0, and
that tag's content changed once (live build-timestamp 2026-07-09,
later renders 2026-07-23). Both images use the 26.7.0 tag with
imagePullPolicy Always, so every restart depends on quay.io and could
pull different bits.
Pin the remote resource to the tag's commit, the operator image and
RELATED_IMAGE_KEYCLOAK to the digests that were running on 2026-11-12
(pod imageID) and set the operator to IfNotPresent. The env value is
set with a strategic merge patch keyed by name; a JSON patch by list
index hit KUBERNETES_NAMESPACE instead. render-phase2.sh and
apply-keycloak.sh check the commit and the digest.
Risk: applying restarts the Keycloak operator once (restart window).
Keycloak itself keeps running.
Verification (desktop): render-phase2.sh passes; the rendered
Deployment has the pinned image, IfNotPresent and the digest in
RELATED_IMAGE_KEYCLOAK with the other env entries unchanged.
A backup nobody has restored is not known to work. The 2026-08 k3s
recovery bundle never got its isolated restore drill.
k3s-datastore-restore-drill.sh downloads the latest archive, checks its
sha256 and integrity, starts the same k3s version in a network-less
throwaway container on the copy, and checks the API comes up, the
Secret count is close to live and every Secret decrypts with the
token. gitea-backup-restore-drill.sh extracts the latest dump and
checks app.ini (INTERNAL_TOKEN, SECRET_KEY), git fsck on every
repository and the repository count against live Gitea. Both accept a
local copy and write a drill timestamp metric.
Risk: read-only for the cluster. The k3s drill runs the restored
control plane with --network none, so its controllers reach nothing.
Verification (desktop):
- test-k3s-datastore-restore-drill.sh: a fresh k3s with three test
Secrets (stored encrypted, k8s:enc:) restores with all Secrets
decrypted in 7s; before folding the WAL the test saw 2 of 3 rows,
which is why the real backup uses sqlite3 .backup.
- gitea-backup-restore-drill.sh on a live repository dump: 29
repositories, 0 fsck failures, 29 live; the dump was deleted after.
Backups, the daily drift check and the k3s serving certificates have no
alert. k3s renews certificates only when it starts within 120 days of
expiry, and the two kube-prometheus-stack kubelet certificate alerts
can never fire because the k3s kubelet exports no
kubelet_certificate_manager_* metric.
Add a PrometheusRule (platform-host-checks) for backup age, failed
runs, missing metrics, drift left for 6h, a stale drift check, and
serving certificate expiry (warning under 100 days, critical under 14).
Add the two remaining host timers (certificate expiry via openssl on
6443 and 10250; the drift check against origin/main) and an installer
that copies the scripts to a root-owned path, creates the textfile and
work directories and enables all four timers. Disable the two dead
kubelet certificate rules in kube-prometheus-stack.
Risk: the metrics need the node-exporter textfile collector, a one-line
change in W1's node-exporter values that was asked of W1. Until then
PlatformHostBackupMetricsAbsent fires after a day, which is the point.
Verification (desktop):
- test-platform-host-checks-alerts.sh: promtool check and unit tests
pass; with the k3s threshold changed from 2 to 5 days they fail.
- render-observability-core.sh passes; the render has no Kubelet*
certificate alerts and still has both KubeClientCertificateExpiration.
- k3s-cert-expiry.sh as the desktop user reads both certificates.
- install-host-checks.sh --check: systemd-analyze verify is clean.
The 2026-11-11 review found a pg_hba rule that existed only in the
cluster by rendering every root by hand and running kubectl diff.
Nothing does that regularly, so drift is found by accident.
live-drift-check.sh renders all live roots (Phase 1 and 2, admin
services, observability core, the Traefik trust overlay, the tenant,
PriorityClass and keycloak policy roots), strips Secrets and runs
kubectl diff, which is a server-side dry run. It prints one line per
drifted object, honours a dated accepted-drift list and can write
Prometheus textfile metrics for a later alert.
Risk: read-only. It consumes the render scripts' verified-output-dir
handoff and deletes those directories on exit; the README notes this
second consumer. Secret values are not compared.
Verification (desktop, before this series is applied): 17 drifted
objects, matching the review plus the new roots; exit 1; the textfile
holds platform_live_drift_objects 17.
The tech-log tenant needs objects inside platform namespaces: its
database and role, ingress to Postgres and AIStor, blackbox probes,
alert rules, a dashboard and two observability NetworkPolicies. They
were only ever applied with kubectl and no repository holds them, so a
deleted object cannot be recreated and a platform re-apply can drop
tech-log access without anyone noticing.
Declare them in services/tenants/techlog, copied from each object's
last-applied configuration. The dashboard follows the repository's
dashboard pattern (JSON file + configMapGenerator, same name, same
labels). The hyeonworks-prod Namespace moves here as the tenant
boundary and gets PSA enforce=baseline, warn/audit=restricted.
Rejected: enforce=restricted now. A server dry-run shows all three
tech-log pods would violate it; baseline has zero violations.
Risk: the render matches the live objects exactly except the added PSA
labels. The pg_hba rules and the Secret stay out of this root (Cluster
owner and Git policy).
Verification (desktop):
- render-tenants-and-policies.sh passes.
- kubectl diff of the render shows only the Namespace PSA labels.
- kubectl label --dry-run=server enforce=baseline: no warnings.