The restart bundle needs root, touches k3s, kubelet and systemd files,
and must not leave the cluster half-configured. The W1 time-sync
drop-ins need a restart window too.
apply-k3s-restart-bundle.sh --check parses the files and prints the
install plan without root. --execute checks the cluster is Ready,
backs up every file it replaces plus a sqlite .backup of the datastore
and the token, installs the W6 files and W1's two time-sync drop-ins
as they are, enables systemd-time-wait-sync, restarts k3s once with
--no-block (the unit has TimeoutStartSec=0), then verifies the kubelet
configz values, the audit log, that the default namespace rejects a
privileged pod, three health URLs and the time-sync unit. Any failure
restores the backup and restarts again. Last, it creates the
local-storage .skip file and clears the local-path default annotation.
Risk: --execute was not run (owner sudo step). The API is unavailable
for one to two minutes; containers keep running (KillMode=process). The
script refuses to run until W1's host files are on main.
Verification (desktop, non-root):
- --check without the W1 files: MISSING ... (W1), exit 1.
- --check with the W1 files overlaid: parse and invariants pass, plan
printed, exit 0.
- --execute as non-root: "--execute needs root (sudo)".
The 2026-11-11 review found a pg_hba rule that existed only in the
cluster by rendering every root by hand and running kubectl diff.
Nothing does that regularly, so drift is found by accident.
live-drift-check.sh renders all live roots (Phase 1 and 2, admin
services, observability core, the Traefik trust overlay, the tenant,
PriorityClass and keycloak policy roots), strips Secrets and runs
kubectl diff, which is a server-side dry run. It prints one line per
drifted object, honours a dated accepted-drift list and can write
Prometheus textfile metrics for a later alert.
Risk: read-only. It consumes the render scripts' verified-output-dir
handoff and deletes those directories on exit; the README notes this
second consumer. Secret values are not compared.
Verification (desktop, before this series is applied): 17 drifted
objects, matching the review plus the new roots; exit 1; the textfile
holds platform_live_drift_objects 17.
The tech-log tenant needs objects inside platform namespaces: its
database and role, ingress to Postgres and AIStor, blackbox probes,
alert rules, a dashboard and two observability NetworkPolicies. They
were only ever applied with kubectl and no repository holds them, so a
deleted object cannot be recreated and a platform re-apply can drop
tech-log access without anyone noticing.
Declare them in services/tenants/techlog, copied from each object's
last-applied configuration. The dashboard follows the repository's
dashboard pattern (JSON file + configMapGenerator, same name, same
labels). The hyeonworks-prod Namespace moves here as the tenant
boundary and gets PSA enforce=baseline, warn/audit=restricted.
Rejected: enforce=restricted now. A server dry-run shows all three
tech-log pods would violate it; baseline has zero violations.
Risk: the render matches the live objects exactly except the added PSA
labels. The pg_hba rules and the Secret stay out of this root (Cluster
owner and Git policy).
Verification (desktop):
- render-tenants-and-policies.sh passes.
- kubectl diff of the render shows only the Namespace PSA labels.
- kubectl label --dry-run=server enforce=baseline: no warnings.