From e88725aab803e12a82afbbc73fa78c6c279b2d90 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Thu, 12 Nov 2026 00:49:58 +0900 Subject: [PATCH 1/2] feat(host): add the k3s kubelet, audit and pod security drop-ins The kubelet has no memory eviction threshold and no reservations, so under memory pressure the kernel OOM killer picks a victim and Postgres (oom_score_adj 985) is near the front. The API server writes no audit log. Namespaces without labels have no Pod Security level, and graceful node shutdown is off. Add the files for one k3s restart: a kubelet drop-in (system 1 CPU/2Gi, kube 0.5 CPU/1Gi, hard eviction memory 500Mi plus the two k3s disk defaults, soft eviction 1Gi for 1m, shutdown 60s/20s), an API server drop-in that appends the audit and admission-config flags, a Metadata audit policy and a cluster Pod Security default (baseline, warn and audit restricted, kube-system exempt). Rejected: protect-kernel-defaults, recommended in review 7.3. CIS dropped it in v1.7, and the sysctls it requires include kernel.panic_on_oops=1, which reboots this dual-boot desktop on any kernel oops; a reboot started the 2026-11-11 clock incident. The README records the reason. Risk: nothing is installed by this commit. The kubelet merges the drop-in after k3s's 00-k3s-defaults.conf (k3s passes only --config-dir); evictionHard lists all three keys because setting the map zeroes the others. Verification: the apply script's --check parses every file and asserts the invariants (next commit). --- infrastructure/host/k3s/README.md | 92 +++++++++++++++++++ .../config.yaml.d/60-platform-apiserver.yaml | 12 +++ .../kubelet.conf.d/50-platform-resources.conf | 29 ++++++ infrastructure/host/k3s/server/audit.yaml | 28 ++++++ infrastructure/host/k3s/server/psa.yaml | 22 +++++ 5 files changed, 183 insertions(+) create mode 100644 infrastructure/host/k3s/README.md create mode 100644 infrastructure/host/k3s/config.yaml.d/60-platform-apiserver.yaml create mode 100644 infrastructure/host/k3s/kubelet.conf.d/50-platform-resources.conf create mode 100644 infrastructure/host/k3s/server/audit.yaml create mode 100644 infrastructure/host/k3s/server/psa.yaml diff --git a/infrastructure/host/k3s/README.md b/infrastructure/host/k3s/README.md new file mode 100644 index 0000000..66bda42 --- /dev/null +++ b/infrastructure/host/k3s/README.md @@ -0,0 +1,92 @@ +# k3s 재시작 묶음 + +k3s를 **한 번만** 재시작해서 다음을 같이 넣는다. 근거는 `docs/reviews/2026-11-11-k3s-maturity-review.md` +A3·A9·A10·7.2·7.3·7.5절, 결정은 같은 문서 8절 결정 5(2026-11-12 총괄 확정)다. + +| 파일 | 설치 위치 | 하는 일 | +| --- | --- | --- | +| `kubelet.conf.d/50-platform-resources.conf` | `/var/lib/rancher/k3s/agent/etc/kubelet.conf.d/` | 예약(system 1 CPU·2Gi, kube 0.5 CPU·1Gi), 메모리 축출(hard 500Mi, soft 1Gi/1분), graceful node shutdown(60초, critical 20초) | +| `config.yaml.d/60-platform-apiserver.yaml` | `/etc/rancher/k3s/config.yaml.d/` | API 서버 감사 로그와 PodSecurity admission 설정 파일 지정 | +| `server/audit.yaml` | `/var/lib/rancher/k3s/server/audit.yaml` | 감사 정책: Metadata 수준(본문을 남기지 않아 Secret 값이 로그에 없다), 잦고 값 없는 요청은 뺌 | +| `server/psa.yaml` | `/var/lib/rancher/k3s/server/psa.yaml` | 라벨 없는 namespace의 기본값: enforce `baseline`, warn·audit `restricted`, `kube-system` 예외 | +| (W1) `../systemd/k3s.service.d/10-after-time-sync.conf` | `/etc/systemd/system/k3s.service.d/` | k3s를 시간 동기화 뒤에 시작 | +| (W1) `../systemd/systemd-time-wait-sync.service.d/10-timeout.conf` | `/etc/systemd/system/systemd-time-wait-sync.service.d/` | 동기화를 10분까지만 기다림 | + +그리고 `/var/lib/rancher/k3s/server/manifests/local-storage.yaml.skip`을 만들어 k3s가 local-path +addon을 다시 덮어쓰지 않게 한 뒤, `local-path` StorageClass의 기본 표시를 끈다. 쓰는 PVC가 없고 +(2026-11-11), 기본으로 남아 있으면 `storageClassName`을 빠뜨린 PVC가 루트 SSD에 Delete 정책으로 생긴다. +k3s 문서의 `.skip` 방식이다(packaged components). 그 뒤 local-path-provisioner는 k3s 업그레이드로 +갱신되지 않는다. + +## 값을 이렇게 정한 이유 + +- 예약: 노드 32GiB, 12 CPU. 파드 requests 합이 메모리 11Gi(34%)라 예약 3Gi를 빼도 여유가 크다. + 예약이 없으면 Allocatable = Capacity라 파드가 노드 전부를 잡을 수 있고, 같은 호스트의 Docker + 빌드·CI·데스크톱 앱과 겹치면 커널 OOM killer가 고른다. +- `evictionHard`: 메모리·디스크 셋을 모두 적는다. 이 맵을 정하면 정하지 않은 항목의 기본값이 0이 된다 + (`mergeDefaultEvictionSettings: false`). 디스크 5%는 k3s 기본값 그대로다. +- graceful shutdown: 두 값이 0이면 기능이 꺼진다. 재부팅 때 Postgres 같은 파드가 정상 종료할 시간을 + 준다. 호스트 종료가 최대 60초 늦어진다. kubelet이 logind `InhibitDelayMaxSec`(지금 30초)가 모자라면 + 스스로 늘린다. 적용 뒤 `systemd-analyze cat-config systemd/logind.conf`로 확인한다. +- PSA 기본값은 `restricted`가 아니라 `baseline`이다. 라벨 없는 namespace는 `default`·`kube-public`· + `kube-node-lease`뿐이고 파드가 없다. 플랫폼 namespace는 모두 자기 라벨이 있어 영향이 없다. + +## 넣지 않은 것: `protect-kernel-defaults` + +7.3절에서 권했지만 이 묶음에서 뺐다. + +- CIS Kubernetes Benchmark는 v1.7부터 이 항목(옛 4.2.6)을 뺐다. +- 켜려면 kubelet이 기대하는 sysctl을 먼저 맞춰야 한다(k3s hardening 가이드: `vm.overcommit_memory=1`, + `kernel.panic=10`, `kernel.panic_on_oops=1`, `vm.panic_on_oom=0`). `kernel.panic_on_oops=1`은 커널 + oops가 한 번만 나도 10초 뒤 호스트를 재부팅한다. 이 기계는 GPU 드라이버가 있는 데스크톱이고 + 이중 부팅이며, 재부팅 자체가 11-11 시계 사고의 출발점이었다. `vm.overcommit_memory=1`은 호스트 전체의 + 메모리 할당 방식을 바꾼다. +- 얻는 것(kubelet이 커널 값을 못 바꾸게 함)보다 잃는 것이 커서 뺀다. 다시 넣으려면 sysctl 변경을 따로 + 소유자가 정한다. + +## 적용 (소유자, 한 번) + +전제: + +- desktop, 이 디렉터리와 W1의 `infrastructure/host/systemd/*`가 **둘 다** 있는 platform-core `main` checkout + (W1 PR 병합 뒤). 없으면 스크립트가 `MISSING …(W1)`을 찍고 멈춘다. +- CI job과 에이전트 게이트가 없는 때. k3s 재시작 중 1~2분 API가 멈춘다. k3s는 `KillMode=process`라 + 컨테이너는 계속 돈다(파드가 재시작되지는 않는다). + +```bash +cd /home/donghyeon/workspace/platform && git pull --ff-only +bash scripts/bootstrap/apply-k3s-restart-bundle.sh --check +sudo bash scripts/bootstrap/apply-k3s-restart-bundle.sh --execute +``` + +`--execute`가 하는 일: + +1. 클러스터 Ready와 health 셋(Keycloak OIDC discovery, tech-log health, Gitea healthz)을 본다. +2. 바꿀 파일을 `/var/backups/platform-k3s-restart-bundle/<시각>/`에 복사하고, datastore를 + `sqlite3 .backup`으로 복사한다(`token`도 같이, 모드 0600, `PRAGMA integrity_check` 출력). +3. 파일을 설치하고 `systemd-time-wait-sync`를 켠 뒤 k3s를 한 번 재시작한다. +4. 확인: kubelet configz(예약·eviction·shutdown 값), 감사 로그가 쓰이는지, `default` namespace에서 + privileged 파드가 서버 dry-run으로 거절되는지, health 셋이 200인지, time-wait-sync가 enabled인지. +5. 3·4 중 하나라도 실패하면 백업한 파일을 되돌리고 새 파일을 지운 뒤 k3s를 다시 재시작한다. +6. 다 통과하면 `local-path`의 기본 표시를 끈다. + +마지막 줄이 `K3S RESTART BUNDLE APPLIED`여야 한다. + +## 같은 재시작 창에 이어서 할 것 + +PriorityClass를 파드에 붙이는 변경과 Keycloak 이미지 고정은 파드 재시작을 부르므로 이 묶음 바로 뒤에 +한다(`docs/runbooks/` 재시작 창 절차). Postgres의 `priorityClassName`·Guaranteed는 W1 범위다. + +## 되돌리기 (수동) + +```bash +B=/var/backups/platform-k3s-restart-bundle/<시각> +sudo rm -f /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/50-platform-resources.conf \ + /etc/rancher/k3s/config.yaml.d/60-platform-apiserver.yaml \ + /var/lib/rancher/k3s/server/manifests/local-storage.yaml.skip +sudo systemctl daemon-reload && sudo systemctl restart k3s +kubectl annotate storageclass local-path storageclass.kubernetes.io/is-default-class=true --overwrite +``` + +`audit.yaml`·`psa.yaml`는 참조가 없어지면 쓰이지 않으므로 남겨도 된다. W1 시간 동기화 되돌리기는 +`infrastructure/host/README.md` 2단계를 따른다. diff --git a/infrastructure/host/k3s/config.yaml.d/60-platform-apiserver.yaml b/infrastructure/host/k3s/config.yaml.d/60-platform-apiserver.yaml new file mode 100644 index 0000000..8a2a678 --- /dev/null +++ b/infrastructure/host/k3s/config.yaml.d/60-platform-apiserver.yaml @@ -0,0 +1,12 @@ +# Installed as /etc/rancher/k3s/config.yaml.d/60-platform-apiserver.yaml. +# "+" appends to kube-apiserver-arg instead of replacing it (docs.k3s.io, +# installation/configuration, multiple config files). +# Audit log: k3s hardening guide "API Server audit configuration". +# PodSecurity default: k3s hardening guide "Pod Security" (baseline here, see README). +kube-apiserver-arg+: + - "audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log" + - "audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml" + - "audit-log-maxage=30" + - "audit-log-maxbackup=10" + - "audit-log-maxsize=100" + - "admission-control-config-file=/var/lib/rancher/k3s/server/psa.yaml" diff --git a/infrastructure/host/k3s/kubelet.conf.d/50-platform-resources.conf b/infrastructure/host/k3s/kubelet.conf.d/50-platform-resources.conf new file mode 100644 index 0000000..86ef9ac --- /dev/null +++ b/infrastructure/host/k3s/kubelet.conf.d/50-platform-resources.conf @@ -0,0 +1,29 @@ +# Installed as /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/50-platform-resources.conf. +# k3s starts the kubelet with --config-dir on that directory and keeps its own +# defaults in 00-k3s-defaults.conf; this file is merged after it. +# Why: docs/reviews/2026-11-11-k3s-maturity-review.md A3 and 7.2. +apiVersion: kubelet.config.k8s.io/v1beta1 +kind: KubeletConfiguration +# Keep memory and CPU for the host (desktop, Docker builds, CI runner) and for +# k3s itself, so pods cannot take the whole node. +systemReserved: + cpu: "1" + memory: 2Gi +kubeReserved: + cpu: 500m + memory: 1Gi +# All three keys are listed on purpose: setting evictionHard replaces the +# defaults, and k3s's own default only had the two disk keys. +evictionHard: + memory.available: 500Mi + nodefs.available: "5%" + imagefs.available: "5%" +evictionSoft: + memory.available: 1Gi +evictionSoftGracePeriod: + memory.available: 1m +evictionMaxPodGracePeriod: 60 +# Graceful node shutdown: on reboot or power-off the kubelet stops normal pods +# first (40s), then critical pods (20s), instead of leaving it to systemd. +shutdownGracePeriod: 60s +shutdownGracePeriodCriticalPods: 20s diff --git a/infrastructure/host/k3s/server/audit.yaml b/infrastructure/host/k3s/server/audit.yaml new file mode 100644 index 0000000..f132d91 --- /dev/null +++ b/infrastructure/host/k3s/server/audit.yaml @@ -0,0 +1,28 @@ +# Installed as /var/lib/rancher/k3s/server/audit.yaml (mode 0600). +# Metadata level only: who did what to which object, never request or +# response bodies, so Secret values are not written to the log. +apiVersion: audit.k8s.io/v1 +kind: Policy +omitStages: + - RequestReceived +rules: + # High-volume, low-value traffic. + - level: None + resources: + - group: coordination.k8s.io + resources: ["leases"] + - level: None + resources: + - group: "" + resources: ["events"] + - level: None + nonResourceURLs: + - /healthz* + - /livez* + - /readyz* + - /version + - /metrics + - level: None + users: ["system:kube-proxy"] + verbs: ["watch"] + - level: Metadata diff --git a/infrastructure/host/k3s/server/psa.yaml b/infrastructure/host/k3s/server/psa.yaml new file mode 100644 index 0000000..66df334 --- /dev/null +++ b/infrastructure/host/k3s/server/psa.yaml @@ -0,0 +1,22 @@ +# Installed as /var/lib/rancher/k3s/server/psa.yaml (mode 0600). +# Cluster-wide Pod Security default for namespaces without their own labels. +# Namespace labels still win: every platform namespace already sets its level. +apiVersion: apiserver.config.k8s.io/v1 +kind: AdmissionConfiguration +plugins: + - name: PodSecurity + configuration: + apiVersion: pod-security.admission.config.k8s.io/v1 + kind: PodSecurityConfiguration + defaults: + enforce: "baseline" + enforce-version: "latest" + audit: "restricted" + audit-version: "latest" + warn: "restricted" + warn-version: "latest" + exemptions: + usernames: [] + runtimeClasses: [] + # k3s helm-install jobs and local-path helper pods run here. + namespaces: [kube-system] -- 2.54.0 From 457d88e2932478f802336697fd1f2c1ba46d6b8c Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Thu, 12 Nov 2026 00:49:58 +0900 Subject: [PATCH 2/2] feat(bootstrap): apply the k3s restart bundle with one restart The restart bundle needs root, touches k3s, kubelet and systemd files, and must not leave the cluster half-configured. The W1 time-sync drop-ins need a restart window too. apply-k3s-restart-bundle.sh --check parses the files and prints the install plan without root. --execute checks the cluster is Ready, backs up every file it replaces plus a sqlite .backup of the datastore and the token, installs the W6 files and W1's two time-sync drop-ins as they are, enables systemd-time-wait-sync, restarts k3s once with --no-block (the unit has TimeoutStartSec=0), then verifies the kubelet configz values, the audit log, that the default namespace rejects a privileged pod, three health URLs and the time-sync unit. Any failure restores the backup and restarts again. Last, it creates the local-storage .skip file and clears the local-path default annotation. Risk: --execute was not run (owner sudo step). The API is unavailable for one to two minutes; containers keep running (KillMode=process). The script refuses to run until W1's host files are on main. Verification (desktop, non-root): - --check without the W1 files: MISSING ... (W1), exit 1. - --check with the W1 files overlaid: parse and invariants pass, plan printed, exit 0. - --execute as non-root: "--execute needs root (sudo)". --- scripts/bootstrap/apply-k3s-restart-bundle.sh | 238 ++++++++++++++++++ 1 file changed, 238 insertions(+) create mode 100644 scripts/bootstrap/apply-k3s-restart-bundle.sh diff --git a/scripts/bootstrap/apply-k3s-restart-bundle.sh b/scripts/bootstrap/apply-k3s-restart-bundle.sh new file mode 100644 index 0000000..a5864dc --- /dev/null +++ b/scripts/bootstrap/apply-k3s-restart-bundle.sh @@ -0,0 +1,238 @@ +#!/usr/bin/env bash +# Install the k3s restart bundle and restart k3s exactly once. +# +# bash scripts/bootstrap/apply-k3s-restart-bundle.sh --check # no root, changes nothing +# sudo bash scripts/bootstrap/apply-k3s-restart-bundle.sh --execute +# +# The bundle (infrastructure/host/k3s/README.md): +# W6: kubelet reservations, memory eviction and graceful shutdown (kubelet drop-in), +# API server audit log and cluster Pod Security default (k3s drop-in + two files), +# local-path StorageClass no longer the default (manifest .skip + annotation). +# W1: k3s starts after time sync (infrastructure/host/systemd/*, installed as is). +# +# --execute backs up every file it replaces plus a datastore copy, installs, restarts +# k3s, verifies, and on any failure restores the previous files and restarts again. +set -Eeuo pipefail +umask 077 + +readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)" +readonly KCFG=/etc/rancher/k3s/k3s.yaml +readonly SERVER=/var/lib/rancher/k3s/server +readonly SKIP_FILE="$SERVER/manifests/local-storage.yaml.skip" +readonly AUDIT_DIR="$SERVER/logs" +readonly BACKUP_ROOT=/var/backups/platform-k3s-restart-bundle + +# source | destination | mode | owner of the source file +readonly -a FILES=( + "infrastructure/host/k3s/kubelet.conf.d/50-platform-resources.conf|/var/lib/rancher/k3s/agent/etc/kubelet.conf.d/50-platform-resources.conf|0600|W6" + "infrastructure/host/k3s/config.yaml.d/60-platform-apiserver.yaml|/etc/rancher/k3s/config.yaml.d/60-platform-apiserver.yaml|0644|W6" + "infrastructure/host/k3s/server/audit.yaml|$SERVER/audit.yaml|0600|W6" + "infrastructure/host/k3s/server/psa.yaml|$SERVER/psa.yaml|0600|W6" + "infrastructure/host/systemd/k3s.service.d/10-after-time-sync.conf|/etc/systemd/system/k3s.service.d/10-after-time-sync.conf|0644|W1" + "infrastructure/host/systemd/systemd-time-wait-sync.service.d/10-timeout.conf|/etc/systemd/system/systemd-time-wait-sync.service.d/10-timeout.conf|0644|W1" +) +readonly -a HEALTH_URLS=( + https://id.learn.hyeonworks.com/realms/hyeonworks/.well-known/openid-configuration + https://hyeonworks.com/api/healthcheck + https://git.learn.hyeonworks.com/api/healthz +) + +say() { printf '%s\n' "$*"; } +fail() { printf 'ERROR: %s\n' "$*" >&2; exit 1; } + +mode="${1:-}" +[[ "$#" -eq 1 && ( "$mode" == "--check" || "$mode" == "--execute" ) ]] || { + sed -n '2,15p' "${BASH_SOURCE[0]}" >&2 + exit 2 +} + +check_sources() { + local entry src dst perm owner missing=0 + for entry in "${FILES[@]}"; do + IFS='|' read -r src dst perm owner <<<"$entry" + if [[ ! -f "$ROOT/$src" ]]; then + say "MISSING $src (${owner})" + missing=1 + fi + done + (( missing == 0 )) || fail "run this from a platform-core checkout that has both the W6 and the W1 host files on main" + python3 -I - "$ROOT" <<'PY' || fail "bundle files do not parse" +import sys, pathlib, yaml +root = pathlib.Path(sys.argv[1]) / "infrastructure/host/k3s" +k = yaml.safe_load((root / "kubelet.conf.d/50-platform-resources.conf").read_text()) +assert k["kind"] == "KubeletConfiguration", "kubelet drop-in kind" +assert set(k["evictionHard"]) == {"memory.available", "nodefs.available", "imagefs.available"}, "evictionHard keys" +assert k["shutdownGracePeriod"] == "60s" and k["shutdownGracePeriodCriticalPods"] == "20s", "shutdown periods" +a = yaml.safe_load((root / "config.yaml.d/60-platform-apiserver.yaml").read_text()) +assert list(a) == ["kube-apiserver-arg+"], "apiserver drop-in must only append kube-apiserver-arg" +args = dict(x.split("=", 1) for x in a["kube-apiserver-arg+"]) +assert args["audit-policy-file"] == "/var/lib/rancher/k3s/server/audit.yaml" +assert args["admission-control-config-file"] == "/var/lib/rancher/k3s/server/psa.yaml" +p = yaml.safe_load((root / "server/audit.yaml").read_text()) +assert p["kind"] == "Policy" and p["rules"][-1] == {"level": "Metadata"}, "audit policy ends with Metadata" +assert all(r["level"] in ("None", "Metadata") for r in p["rules"]), "audit policy must not log bodies" +s = yaml.safe_load((root / "server/psa.yaml").read_text()) +d = s["plugins"][0]["configuration"]["defaults"] +assert d["enforce"] == "baseline", "PSA default enforce" +assert s["plugins"][0]["configuration"]["exemptions"]["namespaces"] == ["kube-system"], "PSA exemptions" +print("bundle files parse and keep their invariants") +PY +} + +plan() { + local entry src dst perm owner state + for entry in "${FILES[@]}"; do + IFS='|' read -r src dst perm owner <<<"$entry" + if [[ -d "$(dirname "$dst")" && ! -r "$(dirname "$dst")" ]] || + [[ ! -e "$(dirname "$dst")" && ! -r "$(dirname "$(dirname "$dst")")" ]]; then + state="unknown (needs root to read)" + elif [[ ! -e "$dst" ]]; then + state="new" + elif cmp -s "$ROOT/$src" "$dst"; then + state="same" + else + state="replace" + fi + say " ${owner} ${state} ${dst}" + done + say " W6 touch $SKIP_FILE" + say " W6 mkdir $AUDIT_DIR (0700)" + say " W1 systemctl enable systemd-time-wait-sync.service" + say " -- systemctl daemon-reload && systemctl restart k3s (once)" +} + +say "== check" +check_sources +say "== plan" +plan +[[ "$mode" == "--execute" ]] || { say "CHECK ONLY: nothing changed"; exit 0; } + +[[ "$(id -u)" -eq 0 ]] || fail "--execute needs root (sudo)" +command -v sqlite3 >/dev/null || fail "sqlite3 is required for the pre-restart datastore copy" +kubectl() { command k3s kubectl --kubeconfig "$KCFG" "$@"; } + +wait_ready() { + local i + for i in $(seq 1 60); do + if kubectl get --raw=/readyz >/dev/null 2>&1 && + [[ "$(kubectl get nodes -o jsonpath='{.items[0].status.conditions[?(@.type=="Ready")].status}' 2>/dev/null)" == "True" ]]; then + return 0 + fi + sleep 5 + done + return 1 +} + +say "== preflight" +wait_ready || fail "the cluster is not Ready before the change; not touching it" +for url in "${HEALTH_URLS[@]}"; do + code="$(curl -s -o /dev/null -m 15 -w '%{http_code}' "$url" || true)" + say " before $code $url" +done + +ts="$(date -u +%Y%m%dT%H%M%SZ)" +backup="$BACKUP_ROOT/$ts" +install -d -m 0700 "$backup" +time_wait_was_enabled="$(systemctl is-enabled systemd-time-wait-sync.service 2>/dev/null || true)" +say "== backup to $backup" +for entry in "${FILES[@]}"; do + IFS='|' read -r src dst perm owner <<<"$entry" + if [[ -e "$dst" ]]; then + install -D -m 0600 "$dst" "$backup/files$dst" + fi +done +sqlite3 "$SERVER/db/state.db" ".backup '$backup/state.db'" +install -m 0600 "$SERVER/token" "$backup/token" +say " datastore copy: $(sqlite3 "$backup/state.db" 'PRAGMA integrity_check;') $(du -h "$backup/state.db" | cut -f1)" + +rollback() { + local entry src dst perm owner + trap - ERR + say "== ROLLBACK" + set +e + for entry in "${FILES[@]}"; do + IFS='|' read -r src dst perm owner <<<"$entry" + if [[ -e "$backup/files$dst" ]]; then + install -D -m "$perm" "$backup/files$dst" "$dst" + else + rm -f -- "$dst" + fi + done + rm -f -- "$SKIP_FILE" + [[ "$time_wait_was_enabled" == "enabled" ]] || systemctl disable systemd-time-wait-sync.service + systemctl daemon-reload + systemctl restart --no-block k3s + sleep 10 + if wait_ready; then say "ROLLBACK done: previous files restored, k3s Ready"; else say "ROLLBACK: k3s still not Ready; see journalctl -u k3s"; fi + say "backup kept at $backup" + exit 1 +} +trap rollback ERR + +say "== install" +for entry in "${FILES[@]}"; do + IFS='|' read -r src dst perm owner <<<"$entry" + install -D -o root -g root -m "$perm" "$ROOT/$src" "$dst" +done +install -d -o root -g root -m 0700 "$AUDIT_DIR" +install -D -o root -g root -m 0600 /dev/null "$SKIP_FILE" +systemctl daemon-reload +systemctl enable systemd-time-wait-sync.service + +say "== restart k3s" +# k3s.service has TimeoutStartSec=0, so a blocking restart could wait forever. +systemctl restart --no-block k3s +sleep 10 +wait_ready || { say "k3s did not become Ready in 5 minutes"; false; } + +say "== verify" +node="$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')" +kubectl get --raw "/api/v1/nodes/$node/proxy/configz" | python3 -I -c ' +import json, sys +k = json.load(sys.stdin)["kubeletconfig"] +want = { + ("systemReserved", "memory"): "2Gi", ("kubeReserved", "memory"): "1Gi", + ("evictionHard", "memory.available"): "500Mi", ("evictionHard", "nodefs.available"): "5%", + ("evictionHard", "imagefs.available"): "5%", ("evictionSoft", "memory.available"): "1Gi", +} +bad = [f"{a}.{b}={k.get(a, {}).get(b)}" for (a, b), v in want.items() if k.get(a, {}).get(b) != v] +if k.get("shutdownGracePeriod") != "1m0s": bad.append(f"shutdownGracePeriod={k.get(\"shutdownGracePeriod\")}") +if bad: sys.exit("kubelet config not applied: " + ", ".join(bad)) +print(" kubelet: reservations, eviction and graceful shutdown applied") +' +for i in $(seq 1 12); do [[ -s "$AUDIT_DIR/audit.log" ]] && break; sleep 5; done +[[ -s "$AUDIT_DIR/audit.log" ]] || { say "audit log is empty"; false; } +say " audit log: $(wc -l <"$AUDIT_DIR/audit.log") lines" +psa_out="$(kubectl apply --dry-run=server -f - 2>&1 <<'YAML' || true +apiVersion: v1 +kind: Pod +metadata: + name: psa-default-probe + namespace: default +spec: + containers: + - name: probe + image: registry.k8s.io/pause:3.10 + securityContext: + privileged: true +YAML +)" +grep -q 'violates PodSecurity "baseline' <<<"$psa_out" || { say "PSA default did not reject a privileged pod: $psa_out"; false; } +say " pod security: default namespace rejects a privileged pod (baseline)" +for url in "${HEALTH_URLS[@]}"; do + ok=0 + for i in $(seq 1 30); do + [[ "$(curl -s -o /dev/null -m 15 -w '%{http_code}' "$url" || true)" == "200" ]] && { ok=1; break; } + sleep 10 + done + (( ok == 1 )) || { say "health check failed: $url"; false; } + say " after 200 $url" +done +say " time sync: $(systemctl is-enabled systemd-time-wait-sync.service), $(systemctl show systemd-time-wait-sync.service -p TimeoutStartUSec)" +trap - ERR + +say "== local-path is no longer the default StorageClass" +kubectl annotate storageclass local-path storageclass.kubernetes.io/is-default-class=false --overwrite +kubectl get storageclass local-path -o jsonpath=' local-path default={.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}{"\n"}' + +say "K3S RESTART BUNDLE APPLIED (backup: $backup)" -- 2.54.0