/etc/rancher/k3s/config.yaml.d/30-nodeport-loopback.yaml keeps Traefik's
NodePorts on 127.0.0.0/8 so the LAN cannot bypass Host Nginx. The
Traefik README and validator depend on it, but the file itself was only
on the host, so a rebuilt host would not get it.
Add the live content as is (kube-proxy-arg nodeport-addresses). The
host-rebuild and datastore-restore runbooks install it.
Risk: none; nothing installs it automatically.
Verification: the file is byte-for-byte the live drop-in
(ssh desktop cat ...); 40-secrets-encryption.yaml in the repository
already matches the host.
Backups, the daily drift check and the k3s serving certificates have no
alert. k3s renews certificates only when it starts within 120 days of
expiry, and the two kube-prometheus-stack kubelet certificate alerts
can never fire because the k3s kubelet exports no
kubelet_certificate_manager_* metric.
Add a PrometheusRule (platform-host-checks) for backup age, failed
runs, missing metrics, drift left for 6h, a stale drift check, and
serving certificate expiry (warning under 100 days, critical under 14).
Add the two remaining host timers (certificate expiry via openssl on
6443 and 10250; the drift check against origin/main) and an installer
that copies the scripts to a root-owned path, creates the textfile and
work directories and enables all four timers. Disable the two dead
kubelet certificate rules in kube-prometheus-stack.
Risk: the metrics need the node-exporter textfile collector, a one-line
change in W1's node-exporter values that was asked of W1. Until then
PlatformHostBackupMetricsAbsent fires after a day, which is the point.
Verification (desktop):
- test-platform-host-checks-alerts.sh: promtool check and unit tests
pass; with the k3s threshold changed from 2 to 5 days they fail.
- render-observability-core.sh passes; the render has no Kubelet*
certificate alerts and still has both KubeClientCertificateExpiration.
- k3s-cert-expiry.sh as the desktop user reads both certificates.
- install-host-checks.sh --check: systemd-analyze verify is clean.
Nothing backs up the k3s datastore or Gitea's repositories, packages and
app.ini. W1 covers platform-postgres only. One SSD failure loses the
source, CI artefacts, published libraries and every cluster Secret.
Add two root systemd timers that write to a new AIStor bucket on the
HDD (platform-host-backups, same AIStor as W1's postgres backups):
- k3s datastore, daily: sqlite3 .backup of the live state.db plus the
server token (k3s backup/restore docs: the token is needed to read
it back), integrity_check, tar.gz, sha256, upload. Off-host copy is
prepared behind two variables (gpg recipient file, rclone target) and
stays off until the owner chooses a place.
- Gitea, weekly: gitea dump --skip-db streamed from the live container,
listing checked for app.ini and repositories, sha256, upload. The DB
is in the CNPG backup.
Each run writes textfile metrics for the alerts in the next commit.
create-host-backup-credentials.sh makes the bucket (50GiB quota,
lifecycle 56 days for gitea/, 30 days for k3s-datastore/), a
bucket-scoped policy and user, and a root-only env file; no key is
printed and mc gets keys only through a 0600 env file.
Rejected: in-cluster CronJobs. The datastore is a root-owned host file
and the dump needs the Gitea DB connection; a hostPath root pod or a
second Gitea-labelled pod is a larger privilege than a root timer.
Risk: not installed here (owner sudo steps). The k3s archive holds the
token; together with the datastore it decrypts every Secret. It lives
in a bucket only the backup user can read.
Verification (desktop): a dump streamed with --file - is a valid
tar.gz with app.ini; repositories appear as repos/<owner>/<name>.git
(29, matching the live count). The scripts parse with bash -n.
The kubelet has no memory eviction threshold and no reservations, so
under memory pressure the kernel OOM killer picks a victim and
Postgres (oom_score_adj 985) is near the front. The API server writes
no audit log. Namespaces without labels have no Pod Security level, and
graceful node shutdown is off.
Add the files for one k3s restart: a kubelet drop-in (system 1 CPU/2Gi,
kube 0.5 CPU/1Gi, hard eviction memory 500Mi plus the two k3s disk
defaults, soft eviction 1Gi for 1m, shutdown 60s/20s), an API server
drop-in that appends the audit and admission-config flags, a Metadata
audit policy and a cluster Pod Security default (baseline, warn and
audit restricted, kube-system exempt).
Rejected: protect-kernel-defaults, recommended in review 7.3. CIS
dropped it in v1.7, and the sysctls it requires include
kernel.panic_on_oops=1, which reboots this dual-boot desktop on any
kernel oops; a reboot started the 2026-11-11 clock incident. The README
records the reason.
Risk: nothing is installed by this commit. The kubelet merges the
drop-in after k3s's 00-k3s-defaults.conf (k3s passes only --config-dir);
evictionHard lists all three keys because setting the map zeroes the
others.
Verification: the apply script's --check parses every file and asserts
the invariants (next commit).