Files
platform-core/scripts/validate/test-k3s-datastore-restore-drill.sh
DongHyeonka ba3dddc139 feat(validate): add restore drills for the k3s and Gitea backups
A backup nobody has restored is not known to work. The 2026-08 k3s
recovery bundle never got its isolated restore drill.

k3s-datastore-restore-drill.sh downloads the latest archive, checks its
sha256 and integrity, starts the same k3s version in a network-less
throwaway container on the copy, and checks the API comes up, the
Secret count is close to live and every Secret decrypts with the
token. gitea-backup-restore-drill.sh extracts the latest dump and
checks app.ini (INTERNAL_TOKEN, SECRET_KEY), git fsck on every
repository and the repository count against live Gitea. Both accept a
local copy and write a drill timestamp metric.

Risk: read-only for the cluster. The k3s drill runs the restored
control plane with --network none, so its controllers reach nothing.

Verification (desktop):
- test-k3s-datastore-restore-drill.sh: a fresh k3s with three test
  Secrets (stored encrypted, k8s:enc:) restores with all Secrets
  decrypted in 7s; before folding the WAL the test saw 2 of 3 rows,
  which is why the real backup uses sqlite3 .backup.
- gitea-backup-restore-drill.sh on a live repository dump: 29
  repositories, 0 fsck failures, 29 live; the dump was deleted after.
2026-11-12 01:13:02 +09:00

42 lines
2.5 KiB
Bash

#!/usr/bin/env bash
# Test k3s-datastore-restore-drill.sh without production data: start a fresh,
# network-less k3s with secrets encryption, add three test Secrets, stop it,
# and run the drill on its state.db and token. Needs docker; no root.
set -Eeuo pipefail
umask 077
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
readonly K3S_IMAGE="$(sed -n 's/^readonly K3S_IMAGE="\(.*\)"$/\1/p' "$ROOT/scripts/validate/k3s-datastore-restore-drill.sh")"
[[ -n "$K3S_IMAGE" ]] || { echo "cannot read K3S_IMAGE from the drill script" >&2; exit 1; }
work="$(mktemp -d)"
seed="w6-k3s-drill-seed-$$"
cleanup() {
docker rm -f "$seed" >/dev/null 2>&1 || true
docker run --rm -v "$work:/w" --entrypoint /bin/sh "$K3S_IMAGE" -c 'rm -rf /w/*' >/dev/null 2>&1 || true
rm -rf -- "$work"
}
trap cleanup EXIT
mkdir -p "$work/server" "$work/copy"
docker run -d --name "$seed" --network none --tmpfs /run --tmpfs /var/run \
-v "$work/server:/var/lib/rancher/k3s/server" "$K3S_IMAGE" server --disable-agent --secrets-encryption \
--bind-address 127.0.0.1 --advertise-address 127.0.0.1 --node-ip 127.0.0.1 \
--disable traefik,servicelb,metrics-server,local-storage,coredns >/dev/null
for _ in $(seq 1 60); do docker exec "$seed" kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 5; done
for i in 1 2 3; do docker exec "$seed" kubectl create secret generic "drill-test-$i" --from-literal=value="test-$i" >/dev/null; done
expected="$(docker exec "$seed" kubectl get secrets -A --no-headers | wc -l)"
docker stop -t 30 "$seed" >/dev/null
docker run --rm -v "$work:/w" --entrypoint /bin/sh "$K3S_IMAGE" -c \
"cp /w/server/db/state.db* /w/server/token /w/copy/ && chown $(id -u):$(id -g) /w/copy/*"
# Fold the WAL into state.db, as `sqlite3 .backup` does in the real backup.
sqlite3 "$work/copy/state.db" 'PRAGMA wal_checkpoint(TRUNCATE);' >/dev/null
rm -f "$work/copy/state.db-wal" "$work/copy/state.db-shm"
# The test Secrets must be stored encrypted (k8s:enc:...), so the drill really
# proves that the token unlocks the encryption key.
encrypted="$(sqlite3 "$work/copy/state.db" "SELECT COUNT(*) FROM kine WHERE name LIKE '/registry/secrets/default/drill-test-%' AND CAST(value AS TEXT) LIKE 'k8s:enc:%';")"
echo "seed: $expected Secrets, encrypted test Secret rows=$encrypted"
(( encrypted >= 3 )) || { echo "test Secrets are not encrypted at rest in the seed datastore" >&2; exit 1; }
bash "$ROOT/scripts/validate/k3s-datastore-restore-drill.sh" --from-dir "$work/copy" --expect-secrets "$expected"
echo "K3S DATASTORE RESTORE DRILL TEST PASS"