scripts/bootstrap/apply-k3s-restart-bundle.sh now installs the two
time-sync drop-ins and enables systemd-time-wait-sync together with the
kubelet drop-ins, so the owner restarts k3s once. Say so in step 2 and
keep the manual commands for installing only these two files.
The host files need root, so agent sessions do not install them. The
owner needs one ordered list with checks and rollback for each step,
including when a Docker restart is safe.
Add infrastructure/host/README.md: why the disk and clock settings
exist, the BuildKit key meanings in Docker 29, and four steps. Pin the
tech-log images against kubelet image GC until they move to the Gitea
registry, install the k3s time-sync drop-ins, set the RTC to UTC on
the Windows side, and install the host daemon.json during a window
with no CI job, no agent gate holding the heavy lock, and workmachine
not in use.
Risk: the image pin is a manual label on the current tags; a new
deployment tag needs the label again until the registry move lands.
Verification: the commands were checked on the desktop where they do
not need root (systemctl show, timedatectl, docker buildx inspect,
docker exec gitea-runner docker ps); the sudo steps were not run.
The desktop dual-boots Windows, which writes local time (KST) to the
RTC, while Ubuntu reads the RTC as UTC. On 2026-11-11 the boot started
9 hours ahead; k3s issued its certificates at that time, timesyncd
then set the clock back, the certificates were not yet valid, and
every Ingress returned 502 until k3s was restarted by hand.
Add a k3s.service drop-in with Wants= and After=time-sync.target, and
a drop-in that caps systemd-time-wait-sync.service at 10 minutes.
Enabling systemd-time-wait-sync makes time-sync.target wait for the
first NTP sync; its packaged TimeoutStartSec=infinity would keep k3s
down forever when NTP is unreachable, so after 10 minutes it fails,
the target is reached anyway and k3s starts as it does today.
Risk: the order only applies at boot, so nothing changes until the
next reboot. If NTP is down at boot, k3s starts 10 minutes later. The
Windows side still writes local time until RealTimeIsUniversal is set
there; the host guide covers that step.
Verification: systemd-analyze verify passes for k3s.service and
systemd-time-wait-sync.service with the drop-ins, and an unknown key
added to each drop-in is reported, so both files are read.
The Docker daemon inside the gitea-runner container keeps every job
image and all build cache in the gitea-runner_docker volume on the
root SSD that production k3s also uses. It grew to 16G in the two days
after the runner was installed and held 12.43G of images and 8.44G of
build cache on 2026-11-11. The README only said to look once a week
and prune by hand.
Mount daemon.json into the runner so the inner BuildKit GC keeps 2GB
to 10GB of build cache, and shrinks to 2GB while the disk has less
than 35GB free. Add gitea-runner-prune.service and .timer, a host
systemd timer for Sunday 04:30 KST that removes unused images created
more than 7 days ago and build cache older than 3 days. The service
runs as donghyeon in the docker group, so it needs root only to
install. A user timer was rejected because lingering is off for
donghyeon and user units stop with the last login session.
Risk: image prune filters on image creation time, so the pinned job
image is deleted weekly and the next job pulls about 543MB again.
Recreating the runner to pick up daemon.json cancels a running job,
so it is done only while the runner is idle.
Verification: systemd-analyze verify passes for both units, and the
calendar expression's next run is Sun 2026-11-15 04:30 KST. docker
compose config passes; dockerd --validate inside the runner (Docker
29.9.3) accepts daemon.json.
The host Docker daemon had no daemon.json. The default BuildKit policy
on this 228GiB disk keeps 21.42GiB of build cache untouched and allows
169.5GiB (docker buildx inspect default), so agent gate builds piled
up about 23G in two days before the 2026-11-09 disk incident.
Add infrastructure/host/docker/daemon.json with builder.gc
defaultReservedSpace 5GB, defaultMaxUsedSpace 15GB and
defaultMinFreeSpace 35GB. In Docker 29 the old defaultKeepStorage key
means defaultReservedSpace, a floor and not a cap, so the cap is
defaultMaxUsedSpace. 35GiB free is about 85% use, where the kubelet
starts deleting images; below it the cache shrinks to the floor first.
Moving the Docker data root to the HDD was rejected by the owner
because builds get slower.
Risk: applying needs sudo and a Docker restart, which restarts the
runner, workmachine and cloudflared and fails running CI jobs and
gates. The host apply guide added in this series says when that is
safe. dockerd --validate does not check keys inside builder.gc, so
the guide checks the live policy with docker buildx inspect.
Verification: dockerd --validate passes for the file on Docker
29.6.1; it also passes a typo inside builder.gc and rejects an unknown
top-level key, which is why the live check is in the guide.
apply-trust.sh is the observe -> trust transition. When the live
Traefik is already in the trust phase and only another value changed
(PR #3 resources), it stops at "observation phase must not trust
forwarded headers" before applying, so the change had to be applied
with a bare kubectl apply outside the guarded path.
reapply-trust.sh checks that the runtime is in the trust phase with
the reviewed CIDR and that forwarded-header trust works (a request
through Host Nginx is logged with the forwarded client, not the proxy
address), saves the live HelmChartConfig, shows the change with
kubectl diff and does nothing when there is none, asks for
"APPLY <context> RETRUST <cidr>", applies, and repeats the checks plus
the NodePort, Gitea health and site-manifest checks. Any failure after
the apply re-applies the saved HelmChartConfig, so the trusted CIDR is
never dropped.
Risk: the apply path was not run against the cluster (the live values
already match). The restore path reuses the same wait and health
helpers as apply-trust.sh.
Verification (desktop):
- Live (already trust, values match): "Forwarded-header trust in
effect (ClientHost 127.0.0.1, proxy 10.42.0.1)", then "already
matches the trust overlay. Nothing to do.", exit 0.
- Copy with memory 64Mi changed to 65Mi, answering "no": the diff
shows only that line, "ERROR: cancelled", exit 1; live still 64Mi.
- The ClientHost check of apply-trust.sh would have failed here
(127.0.0.1 is expected in the trust phase), which is why the check
is inverted.
/etc/rancher/k3s/config.yaml.d/30-nodeport-loopback.yaml keeps Traefik's
NodePorts on 127.0.0.0/8 so the LAN cannot bypass Host Nginx. The
Traefik README and validator depend on it, but the file itself was only
on the host, so a rebuilt host would not get it.
Add the live content as is (kube-proxy-arg nodeport-addresses). The
host-rebuild and datastore-restore runbooks install it.
Risk: none; nothing installs it automatically.
Verification: the file is byte-for-byte the live drop-in
(ssh desktop cat ...); 40-secrets-encryption.yaml in the repository
already matches the host.
The Keycloak Operator remote resource is pinned to the tag 26.7.0, and
that tag's content changed once (live build-timestamp 2026-07-09,
later renders 2026-07-23). Both images use the 26.7.0 tag with
imagePullPolicy Always, so every restart depends on quay.io and could
pull different bits.
Pin the remote resource to the tag's commit, the operator image and
RELATED_IMAGE_KEYCLOAK to the digests that were running on 2026-11-12
(pod imageID) and set the operator to IfNotPresent. The env value is
set with a strategic merge patch keyed by name; a JSON patch by list
index hit KUBERNETES_NAMESPACE instead. render-phase2.sh and
apply-keycloak.sh check the commit and the digest.
Risk: applying restarts the Keycloak operator once (restart window).
Keycloak itself keeps running.
Verification (desktop): render-phase2.sh passes; the rendered
Deployment has the pinned image, IfNotPresent and the digest in
RELATED_IMAGE_KEYCLOAK with the other env entries unchanged.
Backups, the daily drift check and the k3s serving certificates have no
alert. k3s renews certificates only when it starts within 120 days of
expiry, and the two kube-prometheus-stack kubelet certificate alerts
can never fire because the k3s kubelet exports no
kubelet_certificate_manager_* metric.
Add a PrometheusRule (platform-host-checks) for backup age, failed
runs, missing metrics, drift left for 6h, a stale drift check, and
serving certificate expiry (warning under 100 days, critical under 14).
Add the two remaining host timers (certificate expiry via openssl on
6443 and 10250; the drift check against origin/main) and an installer
that copies the scripts to a root-owned path, creates the textfile and
work directories and enables all four timers. Disable the two dead
kubelet certificate rules in kube-prometheus-stack.
Risk: the metrics need the node-exporter textfile collector, a one-line
change in W1's node-exporter values that was asked of W1. Until then
PlatformHostBackupMetricsAbsent fires after a day, which is the point.
Verification (desktop):
- test-platform-host-checks-alerts.sh: promtool check and unit tests
pass; with the k3s threshold changed from 2 to 5 days they fail.
- render-observability-core.sh passes; the render has no Kubelet*
certificate alerts and still has both KubeClientCertificateExpiration.
- k3s-cert-expiry.sh as the desktop user reads both certificates.
- install-host-checks.sh --check: systemd-analyze verify is clean.
Nothing backs up the k3s datastore or Gitea's repositories, packages and
app.ini. W1 covers platform-postgres only. One SSD failure loses the
source, CI artefacts, published libraries and every cluster Secret.
Add two root systemd timers that write to a new AIStor bucket on the
HDD (platform-host-backups, same AIStor as W1's postgres backups):
- k3s datastore, daily: sqlite3 .backup of the live state.db plus the
server token (k3s backup/restore docs: the token is needed to read
it back), integrity_check, tar.gz, sha256, upload. Off-host copy is
prepared behind two variables (gpg recipient file, rclone target) and
stays off until the owner chooses a place.
- Gitea, weekly: gitea dump --skip-db streamed from the live container,
listing checked for app.ini and repositories, sha256, upload. The DB
is in the CNPG backup.
Each run writes textfile metrics for the alerts in the next commit.
create-host-backup-credentials.sh makes the bucket (50GiB quota,
lifecycle 56 days for gitea/, 30 days for k3s-datastore/), a
bucket-scoped policy and user, and a root-only env file; no key is
printed and mc gets keys only through a 0600 env file.
Rejected: in-cluster CronJobs. The datastore is a root-owned host file
and the dump needs the Gitea DB connection; a hostPath root pod or a
second Gitea-labelled pod is a larger privilege than a root timer.
Risk: not installed here (owner sudo steps). The k3s archive holds the
token; together with the datastore it decrypts every Secret. It lives
in a bucket only the backup user can read.
Verification (desktop): a dump streamed with --file - is a valid
tar.gz with app.ini; repositories appear as repos/<owner>/<name>.git
(29, matching the live count). The scripts parse with bash -n.
The kubelet has no memory eviction threshold and no reservations, so
under memory pressure the kernel OOM killer picks a victim and
Postgres (oom_score_adj 985) is near the front. The API server writes
no audit log. Namespaces without labels have no Pod Security level, and
graceful node shutdown is off.
Add the files for one k3s restart: a kubelet drop-in (system 1 CPU/2Gi,
kube 0.5 CPU/1Gi, hard eviction memory 500Mi plus the two k3s disk
defaults, soft eviction 1Gi for 1m, shutdown 60s/20s), an API server
drop-in that appends the audit and admission-config flags, a Metadata
audit policy and a cluster Pod Security default (baseline, warn and
audit restricted, kube-system exempt).
Rejected: protect-kernel-defaults, recommended in review 7.3. CIS
dropped it in v1.7, and the sysctls it requires include
kernel.panic_on_oops=1, which reboots this dual-boot desktop on any
kernel oops; a reboot started the 2026-11-11 clock incident. The README
records the reason.
Risk: nothing is installed by this commit. The kubelet merges the
drop-in after k3s's 00-k3s-defaults.conf (k3s passes only --config-dir);
evictionHard lists all three keys because setting the map zeroes the
others.
Verification: the apply script's --check parses every file and asserts
the invariants (next commit).
The keycloak namespace only has the ingress policy the Keycloak
Operator creates. Neither the Keycloak pod nor the Operator pod has any
egress restriction, unlike every other platform namespace.
Add a namespace-wide default deny and allow only what is used: DNS,
platform-postgres 5432 (server and realm-import jobs), JGroups between
Keycloak pods, and for the Operator the API server and the Keycloak
management ports. Keycloak ingress stays owned by the Operator policy.
Rejected: allowing public 443 like Gitea does. Neither the repository
nor the Keycloak CR configures SMTP or an external identity provider,
and the only open TCP connections from keycloak-0 on 2026-11-12 were to
PostgreSQL. Adding SMTP or an IdP later needs a new egress rule; the
README says so.
Risk: a missed dependency would break logins for every service. The
README lists the post-apply checks (OIDC discovery, a new TCP
connection to Postgres from keycloak-0, Operator logs) and the one-line
rollback.
Verification: render and kubectl diff only add the five policies.
The k3s-bundled Traefik runs with no requests or limits, so it is a
BestEffort pod. The kubelet on this node has no memory eviction
threshold (the k3s default evictionHard only covers nodefs and
imagefs), so under memory pressure the kernel OOM killer chooses, and
BestEffort containers have oom_score_adj 1000. Every public route goes
through this one pod.
Set requests cpu 50m / memory 64Mi and a 256Mi memory limit in the base
and in both overlay patches, because the baseline and trust patches
replace valuesContent as a whole. No CPU limit, to avoid throttling.
Measured use on 2026-11-11 was about 5m CPU and 42Mi memory.
Rejected: raising Traefik to Guaranteed. It would reserve CPU the
ingress does not use, and the real fix for memory pressure is kubelet
reservation and eviction thresholds, which need a k3s restart.
Risk: applying the trust overlay makes the k3s helm-controller roll
Traefik once (RollingUpdate, a few seconds). Not applied here.
Verification (desktop):
- infrastructure/networking/traefik/scripts/validate.sh passes.
- kubectl diff of the trust overlay against the live HelmChartConfig
adds only the resources block.
- baseline and observe renders both contain the same resources.
The template repositories need a real runner for their CI, and the
owner chose one container on the desktop host (the k3s node).
Add compose.yaml and config.yaml for the official Gitea Runner 5.4.1
dind image, pinned by index digest, plus the operating guide. The
runner has capacity 1, one ubuntu-latest label pinned to
runner-images ubuntu-24.04-v26.11.03 by digest, restart: always, and
a 6 CPU / 10g cap (no more than half of the 12 cores and 31GiB).
Jobs use the dind daemon through the runner's Docker proxy for
Testcontainers; the host Docker socket is never mounted, job
containers stay unprivileged, valid_volumes is empty, and sweep
removes what a job created.
Dropped alternatives: the deprecated gitea/act_runner 0.6.1 (moved
to gitea/runner), the basic image with the host docker.sock (jobs
would own the host daemon), and dind-rootless (Testcontainers
compatibility needs separate checks).
Risk: the runner container is privileged; a job can still start a
privileged container on the inner daemon, so only trusted repository
workflows should run here. Runner jobs do not take the desktop's
heavy-work lock; the CPU and memory caps are the only guard.
Verification: config keys checked with `gitea-runner config get`,
`docker compose config` passes, the runner registered at user scope
and the Gitea API shows desktop-runner online and idle; it came back
online after `docker restart` without registering again.