diff --git a/infrastructure/host/k3s/README.md b/infrastructure/host/k3s/README.md new file mode 100644 index 0000000..66bda42 --- /dev/null +++ b/infrastructure/host/k3s/README.md @@ -0,0 +1,92 @@ +# k3s 재시작 묶음 + +k3s를 **한 번만** 재시작해서 다음을 같이 넣는다. 근거는 `docs/reviews/2026-11-11-k3s-maturity-review.md` +A3·A9·A10·7.2·7.3·7.5절, 결정은 같은 문서 8절 결정 5(2026-11-12 총괄 확정)다. + +| 파일 | 설치 위치 | 하는 일 | +| --- | --- | --- | +| `kubelet.conf.d/50-platform-resources.conf` | `/var/lib/rancher/k3s/agent/etc/kubelet.conf.d/` | 예약(system 1 CPU·2Gi, kube 0.5 CPU·1Gi), 메모리 축출(hard 500Mi, soft 1Gi/1분), graceful node shutdown(60초, critical 20초) | +| `config.yaml.d/60-platform-apiserver.yaml` | `/etc/rancher/k3s/config.yaml.d/` | API 서버 감사 로그와 PodSecurity admission 설정 파일 지정 | +| `server/audit.yaml` | `/var/lib/rancher/k3s/server/audit.yaml` | 감사 정책: Metadata 수준(본문을 남기지 않아 Secret 값이 로그에 없다), 잦고 값 없는 요청은 뺌 | +| `server/psa.yaml` | `/var/lib/rancher/k3s/server/psa.yaml` | 라벨 없는 namespace의 기본값: enforce `baseline`, warn·audit `restricted`, `kube-system` 예외 | +| (W1) `../systemd/k3s.service.d/10-after-time-sync.conf` | `/etc/systemd/system/k3s.service.d/` | k3s를 시간 동기화 뒤에 시작 | +| (W1) `../systemd/systemd-time-wait-sync.service.d/10-timeout.conf` | `/etc/systemd/system/systemd-time-wait-sync.service.d/` | 동기화를 10분까지만 기다림 | + +그리고 `/var/lib/rancher/k3s/server/manifests/local-storage.yaml.skip`을 만들어 k3s가 local-path +addon을 다시 덮어쓰지 않게 한 뒤, `local-path` StorageClass의 기본 표시를 끈다. 쓰는 PVC가 없고 +(2026-11-11), 기본으로 남아 있으면 `storageClassName`을 빠뜨린 PVC가 루트 SSD에 Delete 정책으로 생긴다. +k3s 문서의 `.skip` 방식이다(packaged components). 그 뒤 local-path-provisioner는 k3s 업그레이드로 +갱신되지 않는다. + +## 값을 이렇게 정한 이유 + +- 예약: 노드 32GiB, 12 CPU. 파드 requests 합이 메모리 11Gi(34%)라 예약 3Gi를 빼도 여유가 크다. + 예약이 없으면 Allocatable = Capacity라 파드가 노드 전부를 잡을 수 있고, 같은 호스트의 Docker + 빌드·CI·데스크톱 앱과 겹치면 커널 OOM killer가 고른다. +- `evictionHard`: 메모리·디스크 셋을 모두 적는다. 이 맵을 정하면 정하지 않은 항목의 기본값이 0이 된다 + (`mergeDefaultEvictionSettings: false`). 디스크 5%는 k3s 기본값 그대로다. +- graceful shutdown: 두 값이 0이면 기능이 꺼진다. 재부팅 때 Postgres 같은 파드가 정상 종료할 시간을 + 준다. 호스트 종료가 최대 60초 늦어진다. kubelet이 logind `InhibitDelayMaxSec`(지금 30초)가 모자라면 + 스스로 늘린다. 적용 뒤 `systemd-analyze cat-config systemd/logind.conf`로 확인한다. +- PSA 기본값은 `restricted`가 아니라 `baseline`이다. 라벨 없는 namespace는 `default`·`kube-public`· + `kube-node-lease`뿐이고 파드가 없다. 플랫폼 namespace는 모두 자기 라벨이 있어 영향이 없다. + +## 넣지 않은 것: `protect-kernel-defaults` + +7.3절에서 권했지만 이 묶음에서 뺐다. + +- CIS Kubernetes Benchmark는 v1.7부터 이 항목(옛 4.2.6)을 뺐다. +- 켜려면 kubelet이 기대하는 sysctl을 먼저 맞춰야 한다(k3s hardening 가이드: `vm.overcommit_memory=1`, + `kernel.panic=10`, `kernel.panic_on_oops=1`, `vm.panic_on_oom=0`). `kernel.panic_on_oops=1`은 커널 + oops가 한 번만 나도 10초 뒤 호스트를 재부팅한다. 이 기계는 GPU 드라이버가 있는 데스크톱이고 + 이중 부팅이며, 재부팅 자체가 11-11 시계 사고의 출발점이었다. `vm.overcommit_memory=1`은 호스트 전체의 + 메모리 할당 방식을 바꾼다. +- 얻는 것(kubelet이 커널 값을 못 바꾸게 함)보다 잃는 것이 커서 뺀다. 다시 넣으려면 sysctl 변경을 따로 + 소유자가 정한다. + +## 적용 (소유자, 한 번) + +전제: + +- desktop, 이 디렉터리와 W1의 `infrastructure/host/systemd/*`가 **둘 다** 있는 platform-core `main` checkout + (W1 PR 병합 뒤). 없으면 스크립트가 `MISSING …(W1)`을 찍고 멈춘다. +- CI job과 에이전트 게이트가 없는 때. k3s 재시작 중 1~2분 API가 멈춘다. k3s는 `KillMode=process`라 + 컨테이너는 계속 돈다(파드가 재시작되지는 않는다). + +```bash +cd /home/donghyeon/workspace/platform && git pull --ff-only +bash scripts/bootstrap/apply-k3s-restart-bundle.sh --check +sudo bash scripts/bootstrap/apply-k3s-restart-bundle.sh --execute +``` + +`--execute`가 하는 일: + +1. 클러스터 Ready와 health 셋(Keycloak OIDC discovery, tech-log health, Gitea healthz)을 본다. +2. 바꿀 파일을 `/var/backups/platform-k3s-restart-bundle/<시각>/`에 복사하고, datastore를 + `sqlite3 .backup`으로 복사한다(`token`도 같이, 모드 0600, `PRAGMA integrity_check` 출력). +3. 파일을 설치하고 `systemd-time-wait-sync`를 켠 뒤 k3s를 한 번 재시작한다. +4. 확인: kubelet configz(예약·eviction·shutdown 값), 감사 로그가 쓰이는지, `default` namespace에서 + privileged 파드가 서버 dry-run으로 거절되는지, health 셋이 200인지, time-wait-sync가 enabled인지. +5. 3·4 중 하나라도 실패하면 백업한 파일을 되돌리고 새 파일을 지운 뒤 k3s를 다시 재시작한다. +6. 다 통과하면 `local-path`의 기본 표시를 끈다. + +마지막 줄이 `K3S RESTART BUNDLE APPLIED`여야 한다. + +## 같은 재시작 창에 이어서 할 것 + +PriorityClass를 파드에 붙이는 변경과 Keycloak 이미지 고정은 파드 재시작을 부르므로 이 묶음 바로 뒤에 +한다(`docs/runbooks/` 재시작 창 절차). Postgres의 `priorityClassName`·Guaranteed는 W1 범위다. + +## 되돌리기 (수동) + +```bash +B=/var/backups/platform-k3s-restart-bundle/<시각> +sudo rm -f /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/50-platform-resources.conf \ + /etc/rancher/k3s/config.yaml.d/60-platform-apiserver.yaml \ + /var/lib/rancher/k3s/server/manifests/local-storage.yaml.skip +sudo systemctl daemon-reload && sudo systemctl restart k3s +kubectl annotate storageclass local-path storageclass.kubernetes.io/is-default-class=true --overwrite +``` + +`audit.yaml`·`psa.yaml`는 참조가 없어지면 쓰이지 않으므로 남겨도 된다. W1 시간 동기화 되돌리기는 +`infrastructure/host/README.md` 2단계를 따른다. diff --git a/infrastructure/host/k3s/config.yaml.d/60-platform-apiserver.yaml b/infrastructure/host/k3s/config.yaml.d/60-platform-apiserver.yaml new file mode 100644 index 0000000..8a2a678 --- /dev/null +++ b/infrastructure/host/k3s/config.yaml.d/60-platform-apiserver.yaml @@ -0,0 +1,12 @@ +# Installed as /etc/rancher/k3s/config.yaml.d/60-platform-apiserver.yaml. +# "+" appends to kube-apiserver-arg instead of replacing it (docs.k3s.io, +# installation/configuration, multiple config files). +# Audit log: k3s hardening guide "API Server audit configuration". +# PodSecurity default: k3s hardening guide "Pod Security" (baseline here, see README). +kube-apiserver-arg+: + - "audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log" + - "audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml" + - "audit-log-maxage=30" + - "audit-log-maxbackup=10" + - "audit-log-maxsize=100" + - "admission-control-config-file=/var/lib/rancher/k3s/server/psa.yaml" diff --git a/infrastructure/host/k3s/kubelet.conf.d/50-platform-resources.conf b/infrastructure/host/k3s/kubelet.conf.d/50-platform-resources.conf new file mode 100644 index 0000000..86ef9ac --- /dev/null +++ b/infrastructure/host/k3s/kubelet.conf.d/50-platform-resources.conf @@ -0,0 +1,29 @@ +# Installed as /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/50-platform-resources.conf. +# k3s starts the kubelet with --config-dir on that directory and keeps its own +# defaults in 00-k3s-defaults.conf; this file is merged after it. +# Why: docs/reviews/2026-11-11-k3s-maturity-review.md A3 and 7.2. +apiVersion: kubelet.config.k8s.io/v1beta1 +kind: KubeletConfiguration +# Keep memory and CPU for the host (desktop, Docker builds, CI runner) and for +# k3s itself, so pods cannot take the whole node. +systemReserved: + cpu: "1" + memory: 2Gi +kubeReserved: + cpu: 500m + memory: 1Gi +# All three keys are listed on purpose: setting evictionHard replaces the +# defaults, and k3s's own default only had the two disk keys. +evictionHard: + memory.available: 500Mi + nodefs.available: "5%" + imagefs.available: "5%" +evictionSoft: + memory.available: 1Gi +evictionSoftGracePeriod: + memory.available: 1m +evictionMaxPodGracePeriod: 60 +# Graceful node shutdown: on reboot or power-off the kubelet stops normal pods +# first (40s), then critical pods (20s), instead of leaving it to systemd. +shutdownGracePeriod: 60s +shutdownGracePeriodCriticalPods: 20s diff --git a/infrastructure/host/k3s/server/audit.yaml b/infrastructure/host/k3s/server/audit.yaml new file mode 100644 index 0000000..f132d91 --- /dev/null +++ b/infrastructure/host/k3s/server/audit.yaml @@ -0,0 +1,28 @@ +# Installed as /var/lib/rancher/k3s/server/audit.yaml (mode 0600). +# Metadata level only: who did what to which object, never request or +# response bodies, so Secret values are not written to the log. +apiVersion: audit.k8s.io/v1 +kind: Policy +omitStages: + - RequestReceived +rules: + # High-volume, low-value traffic. + - level: None + resources: + - group: coordination.k8s.io + resources: ["leases"] + - level: None + resources: + - group: "" + resources: ["events"] + - level: None + nonResourceURLs: + - /healthz* + - /livez* + - /readyz* + - /version + - /metrics + - level: None + users: ["system:kube-proxy"] + verbs: ["watch"] + - level: Metadata diff --git a/infrastructure/host/k3s/server/psa.yaml b/infrastructure/host/k3s/server/psa.yaml new file mode 100644 index 0000000..66df334 --- /dev/null +++ b/infrastructure/host/k3s/server/psa.yaml @@ -0,0 +1,22 @@ +# Installed as /var/lib/rancher/k3s/server/psa.yaml (mode 0600). +# Cluster-wide Pod Security default for namespaces without their own labels. +# Namespace labels still win: every platform namespace already sets its level. +apiVersion: apiserver.config.k8s.io/v1 +kind: AdmissionConfiguration +plugins: + - name: PodSecurity + configuration: + apiVersion: pod-security.admission.config.k8s.io/v1 + kind: PodSecurityConfiguration + defaults: + enforce: "baseline" + enforce-version: "latest" + audit: "restricted" + audit-version: "latest" + warn: "restricted" + warn-version: "latest" + exemptions: + usernames: [] + runtimeClasses: [] + # k3s helm-install jobs and local-path helper pods run here. + namespaces: [kube-system] diff --git a/scripts/bootstrap/apply-k3s-restart-bundle.sh b/scripts/bootstrap/apply-k3s-restart-bundle.sh new file mode 100644 index 0000000..a5864dc --- /dev/null +++ b/scripts/bootstrap/apply-k3s-restart-bundle.sh @@ -0,0 +1,238 @@ +#!/usr/bin/env bash +# Install the k3s restart bundle and restart k3s exactly once. +# +# bash scripts/bootstrap/apply-k3s-restart-bundle.sh --check # no root, changes nothing +# sudo bash scripts/bootstrap/apply-k3s-restart-bundle.sh --execute +# +# The bundle (infrastructure/host/k3s/README.md): +# W6: kubelet reservations, memory eviction and graceful shutdown (kubelet drop-in), +# API server audit log and cluster Pod Security default (k3s drop-in + two files), +# local-path StorageClass no longer the default (manifest .skip + annotation). +# W1: k3s starts after time sync (infrastructure/host/systemd/*, installed as is). +# +# --execute backs up every file it replaces plus a datastore copy, installs, restarts +# k3s, verifies, and on any failure restores the previous files and restarts again. +set -Eeuo pipefail +umask 077 + +readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)" +readonly KCFG=/etc/rancher/k3s/k3s.yaml +readonly SERVER=/var/lib/rancher/k3s/server +readonly SKIP_FILE="$SERVER/manifests/local-storage.yaml.skip" +readonly AUDIT_DIR="$SERVER/logs" +readonly BACKUP_ROOT=/var/backups/platform-k3s-restart-bundle + +# source | destination | mode | owner of the source file +readonly -a FILES=( + "infrastructure/host/k3s/kubelet.conf.d/50-platform-resources.conf|/var/lib/rancher/k3s/agent/etc/kubelet.conf.d/50-platform-resources.conf|0600|W6" + "infrastructure/host/k3s/config.yaml.d/60-platform-apiserver.yaml|/etc/rancher/k3s/config.yaml.d/60-platform-apiserver.yaml|0644|W6" + "infrastructure/host/k3s/server/audit.yaml|$SERVER/audit.yaml|0600|W6" + "infrastructure/host/k3s/server/psa.yaml|$SERVER/psa.yaml|0600|W6" + "infrastructure/host/systemd/k3s.service.d/10-after-time-sync.conf|/etc/systemd/system/k3s.service.d/10-after-time-sync.conf|0644|W1" + "infrastructure/host/systemd/systemd-time-wait-sync.service.d/10-timeout.conf|/etc/systemd/system/systemd-time-wait-sync.service.d/10-timeout.conf|0644|W1" +) +readonly -a HEALTH_URLS=( + https://id.learn.hyeonworks.com/realms/hyeonworks/.well-known/openid-configuration + https://hyeonworks.com/api/healthcheck + https://git.learn.hyeonworks.com/api/healthz +) + +say() { printf '%s\n' "$*"; } +fail() { printf 'ERROR: %s\n' "$*" >&2; exit 1; } + +mode="${1:-}" +[[ "$#" -eq 1 && ( "$mode" == "--check" || "$mode" == "--execute" ) ]] || { + sed -n '2,15p' "${BASH_SOURCE[0]}" >&2 + exit 2 +} + +check_sources() { + local entry src dst perm owner missing=0 + for entry in "${FILES[@]}"; do + IFS='|' read -r src dst perm owner <<<"$entry" + if [[ ! -f "$ROOT/$src" ]]; then + say "MISSING $src (${owner})" + missing=1 + fi + done + (( missing == 0 )) || fail "run this from a platform-core checkout that has both the W6 and the W1 host files on main" + python3 -I - "$ROOT" <<'PY' || fail "bundle files do not parse" +import sys, pathlib, yaml +root = pathlib.Path(sys.argv[1]) / "infrastructure/host/k3s" +k = yaml.safe_load((root / "kubelet.conf.d/50-platform-resources.conf").read_text()) +assert k["kind"] == "KubeletConfiguration", "kubelet drop-in kind" +assert set(k["evictionHard"]) == {"memory.available", "nodefs.available", "imagefs.available"}, "evictionHard keys" +assert k["shutdownGracePeriod"] == "60s" and k["shutdownGracePeriodCriticalPods"] == "20s", "shutdown periods" +a = yaml.safe_load((root / "config.yaml.d/60-platform-apiserver.yaml").read_text()) +assert list(a) == ["kube-apiserver-arg+"], "apiserver drop-in must only append kube-apiserver-arg" +args = dict(x.split("=", 1) for x in a["kube-apiserver-arg+"]) +assert args["audit-policy-file"] == "/var/lib/rancher/k3s/server/audit.yaml" +assert args["admission-control-config-file"] == "/var/lib/rancher/k3s/server/psa.yaml" +p = yaml.safe_load((root / "server/audit.yaml").read_text()) +assert p["kind"] == "Policy" and p["rules"][-1] == {"level": "Metadata"}, "audit policy ends with Metadata" +assert all(r["level"] in ("None", "Metadata") for r in p["rules"]), "audit policy must not log bodies" +s = yaml.safe_load((root / "server/psa.yaml").read_text()) +d = s["plugins"][0]["configuration"]["defaults"] +assert d["enforce"] == "baseline", "PSA default enforce" +assert s["plugins"][0]["configuration"]["exemptions"]["namespaces"] == ["kube-system"], "PSA exemptions" +print("bundle files parse and keep their invariants") +PY +} + +plan() { + local entry src dst perm owner state + for entry in "${FILES[@]}"; do + IFS='|' read -r src dst perm owner <<<"$entry" + if [[ -d "$(dirname "$dst")" && ! -r "$(dirname "$dst")" ]] || + [[ ! -e "$(dirname "$dst")" && ! -r "$(dirname "$(dirname "$dst")")" ]]; then + state="unknown (needs root to read)" + elif [[ ! -e "$dst" ]]; then + state="new" + elif cmp -s "$ROOT/$src" "$dst"; then + state="same" + else + state="replace" + fi + say " ${owner} ${state} ${dst}" + done + say " W6 touch $SKIP_FILE" + say " W6 mkdir $AUDIT_DIR (0700)" + say " W1 systemctl enable systemd-time-wait-sync.service" + say " -- systemctl daemon-reload && systemctl restart k3s (once)" +} + +say "== check" +check_sources +say "== plan" +plan +[[ "$mode" == "--execute" ]] || { say "CHECK ONLY: nothing changed"; exit 0; } + +[[ "$(id -u)" -eq 0 ]] || fail "--execute needs root (sudo)" +command -v sqlite3 >/dev/null || fail "sqlite3 is required for the pre-restart datastore copy" +kubectl() { command k3s kubectl --kubeconfig "$KCFG" "$@"; } + +wait_ready() { + local i + for i in $(seq 1 60); do + if kubectl get --raw=/readyz >/dev/null 2>&1 && + [[ "$(kubectl get nodes -o jsonpath='{.items[0].status.conditions[?(@.type=="Ready")].status}' 2>/dev/null)" == "True" ]]; then + return 0 + fi + sleep 5 + done + return 1 +} + +say "== preflight" +wait_ready || fail "the cluster is not Ready before the change; not touching it" +for url in "${HEALTH_URLS[@]}"; do + code="$(curl -s -o /dev/null -m 15 -w '%{http_code}' "$url" || true)" + say " before $code $url" +done + +ts="$(date -u +%Y%m%dT%H%M%SZ)" +backup="$BACKUP_ROOT/$ts" +install -d -m 0700 "$backup" +time_wait_was_enabled="$(systemctl is-enabled systemd-time-wait-sync.service 2>/dev/null || true)" +say "== backup to $backup" +for entry in "${FILES[@]}"; do + IFS='|' read -r src dst perm owner <<<"$entry" + if [[ -e "$dst" ]]; then + install -D -m 0600 "$dst" "$backup/files$dst" + fi +done +sqlite3 "$SERVER/db/state.db" ".backup '$backup/state.db'" +install -m 0600 "$SERVER/token" "$backup/token" +say " datastore copy: $(sqlite3 "$backup/state.db" 'PRAGMA integrity_check;') $(du -h "$backup/state.db" | cut -f1)" + +rollback() { + local entry src dst perm owner + trap - ERR + say "== ROLLBACK" + set +e + for entry in "${FILES[@]}"; do + IFS='|' read -r src dst perm owner <<<"$entry" + if [[ -e "$backup/files$dst" ]]; then + install -D -m "$perm" "$backup/files$dst" "$dst" + else + rm -f -- "$dst" + fi + done + rm -f -- "$SKIP_FILE" + [[ "$time_wait_was_enabled" == "enabled" ]] || systemctl disable systemd-time-wait-sync.service + systemctl daemon-reload + systemctl restart --no-block k3s + sleep 10 + if wait_ready; then say "ROLLBACK done: previous files restored, k3s Ready"; else say "ROLLBACK: k3s still not Ready; see journalctl -u k3s"; fi + say "backup kept at $backup" + exit 1 +} +trap rollback ERR + +say "== install" +for entry in "${FILES[@]}"; do + IFS='|' read -r src dst perm owner <<<"$entry" + install -D -o root -g root -m "$perm" "$ROOT/$src" "$dst" +done +install -d -o root -g root -m 0700 "$AUDIT_DIR" +install -D -o root -g root -m 0600 /dev/null "$SKIP_FILE" +systemctl daemon-reload +systemctl enable systemd-time-wait-sync.service + +say "== restart k3s" +# k3s.service has TimeoutStartSec=0, so a blocking restart could wait forever. +systemctl restart --no-block k3s +sleep 10 +wait_ready || { say "k3s did not become Ready in 5 minutes"; false; } + +say "== verify" +node="$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')" +kubectl get --raw "/api/v1/nodes/$node/proxy/configz" | python3 -I -c ' +import json, sys +k = json.load(sys.stdin)["kubeletconfig"] +want = { + ("systemReserved", "memory"): "2Gi", ("kubeReserved", "memory"): "1Gi", + ("evictionHard", "memory.available"): "500Mi", ("evictionHard", "nodefs.available"): "5%", + ("evictionHard", "imagefs.available"): "5%", ("evictionSoft", "memory.available"): "1Gi", +} +bad = [f"{a}.{b}={k.get(a, {}).get(b)}" for (a, b), v in want.items() if k.get(a, {}).get(b) != v] +if k.get("shutdownGracePeriod") != "1m0s": bad.append(f"shutdownGracePeriod={k.get(\"shutdownGracePeriod\")}") +if bad: sys.exit("kubelet config not applied: " + ", ".join(bad)) +print(" kubelet: reservations, eviction and graceful shutdown applied") +' +for i in $(seq 1 12); do [[ -s "$AUDIT_DIR/audit.log" ]] && break; sleep 5; done +[[ -s "$AUDIT_DIR/audit.log" ]] || { say "audit log is empty"; false; } +say " audit log: $(wc -l <"$AUDIT_DIR/audit.log") lines" +psa_out="$(kubectl apply --dry-run=server -f - 2>&1 <<'YAML' || true +apiVersion: v1 +kind: Pod +metadata: + name: psa-default-probe + namespace: default +spec: + containers: + - name: probe + image: registry.k8s.io/pause:3.10 + securityContext: + privileged: true +YAML +)" +grep -q 'violates PodSecurity "baseline' <<<"$psa_out" || { say "PSA default did not reject a privileged pod: $psa_out"; false; } +say " pod security: default namespace rejects a privileged pod (baseline)" +for url in "${HEALTH_URLS[@]}"; do + ok=0 + for i in $(seq 1 30); do + [[ "$(curl -s -o /dev/null -m 15 -w '%{http_code}' "$url" || true)" == "200" ]] && { ok=1; break; } + sleep 10 + done + (( ok == 1 )) || { say "health check failed: $url"; false; } + say " after 200 $url" +done +say " time sync: $(systemctl is-enabled systemd-time-wait-sync.service), $(systemctl show systemd-time-wait-sync.service -p TimeoutStartUSec)" +trap - ERR + +say "== local-path is no longer the default StorageClass" +kubectl annotate storageclass local-path storageclass.kubernetes.io/is-default-class=false --overwrite +kubectl get storageclass local-path -o jsonpath=' local-path default={.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}{"\n"}' + +say "K3S RESTART BUNDLE APPLIED (backup: $backup)"