docs(runbooks): add restore, rebuild, upgrade and reboot runbooks #8

Merged
donghyeon.kang merged 4 commits from donghyeon.kang/w6-runbooks AGit into main 2026-11-11 16:48:53 +00:00
4 Commits
Author SHA1 Message Date
DongHyeonka b1140efbb9 docs(reviews): record the phase-two follow-up in the maturity review
The owner decisions in section 8 were settled on 2026-11-12 and the
section 7 recommendations went into separate PRs. Readers of the
review need to know where each one landed and that one recommendation
was withdrawn.

Add a table from each 7.x item to its PR, withdraw protect-kernel-
defaults (CIS dropped it in v1.7 and its kernel.panic_on_oops=1 would
reboot this dual-boot desktop on any oops), and note that the API
server certificate is measured with openssl on the host instead of a
blackbox probe.

Risk: documentation only.

Verification: the PR numbers match the open and merged platform-core
PRs on 2026-11-12.
2026-11-12 01:38:58 +09:00
DongHyeonka ec9a118d15 docs(design): plan the separation of state data from the root SSD
All SSD Local PVs share the root filesystem with the OS, container
images and build caches; review A12 and risk 2. Owner decision 3 asks
for a plan only.

Compare four options (shrink the Windows SSD, a new SSD, the HDD, do
nothing), recommend a new SSD and otherwise the Windows SSD, and give
the migration: mount the new filesystem at the same /srv/k3s/ssd path
so no PV changes, two rsync passes around a short stop, rollback. Note
the scripts whose root-SSD findmnt checks must change in the same PR.

Risk: documentation only; status 제안.

Verification: the PV sizes and paths come from the 2026-11-11 cluster
inventory; the findmnt checks were found with git grep.
2026-11-12 01:38:58 +09:00
DongHyeonka 819f3f14f1 docs(runbooks): add restore, rebuild, upgrade and reboot runbooks
The review found no procedure for losing the host, restoring the k3s
datastore or Gitea, upgrading k3s, or checking the platform after a
reboot, and the existing runbooks live outside version control.

Add five runbooks and a restart-window procedure in docs/runbooks:
after-reboot-checks (clock, k3s and certificates, node, pods in
dependency order, public paths, Prometheus, alerts, runner),
k3s-datastore-restore, gitea-restore (file-only vs full restore, with
the warning that a PITR of the shared database also rewinds Keycloak
and tech-log), k3s-upgrade (patch within a minor), host-rebuild (order
and the easy-to-miss parts: host name for PV node affinity, Tailscale
address in the DNS config, time sync before the first k3s start), and
k3s-restart-window (what to apply in the one restart window, with the
PriorityClasses first). The docs map gains runbooks/ and design/.

Each runbook starts with its preconditions and the commands that check
them; on 2026-11-12 the backups they need are not installed yet, and
they say so.

Risk: documentation only. Restores and the rebuild were not executed.

Verification: a fresh subagent with only these files ran every command
in after-reboot-checks on the desktop and walked the others read-only;
its findings (about 60, e.g. a READY check that missed 0/1 Running
pods, `stable` pointing to another minor, a missing pgAdmin directory,
time sync after the first k3s start) are fixed here. The revised
after-reboot commands were rerun on the desktop and match the text.
Every repository path the runbooks name exists on main or in the
named PR.
2026-11-12 01:38:58 +09:00
DongHyeonka 96623200fa feat(host): keep the k3s nodeport-loopback drop-in in the repository
/etc/rancher/k3s/config.yaml.d/30-nodeport-loopback.yaml keeps Traefik's
NodePorts on 127.0.0.0/8 so the LAN cannot bypass Host Nginx. The
Traefik README and validator depend on it, but the file itself was only
on the host, so a rebuilt host would not get it.

Add the live content as is (kube-proxy-arg nodeport-addresses). The
host-rebuild and datastore-restore runbooks install it.

Risk: none; nothing installs it automatically.

Verification: the file is byte-for-byte the live drop-in
(ssh desktop cat ...); 40-secrets-encryption.yaml in the repository
already matches the host.
2026-11-12 01:38:58 +09:00