A backup nobody has restored is not known to work. The 2026-08 k3s recovery bundle never got its isolated restore drill. k3s-datastore-restore-drill.sh downloads the latest archive, checks its sha256 and integrity, starts the same k3s version in a network-less throwaway container on the copy, and checks the API comes up, the Secret count is close to live and every Secret decrypts with the token. gitea-backup-restore-drill.sh extracts the latest dump and checks app.ini (INTERNAL_TOKEN, SECRET_KEY), git fsck on every repository and the repository count against live Gitea. Both accept a local copy and write a drill timestamp metric. Risk: read-only for the cluster. The k3s drill runs the restored control plane with --network none, so its controllers reach nothing. Verification (desktop): - test-k3s-datastore-restore-drill.sh: a fresh k3s with three test Secrets (stored encrypted, k8s:enc:) restores with all Secrets decrypted in 7s; before folding the WAL the test saw 2 of 3 rows, which is why the real backup uses sqlite3 .backup. - gitea-backup-restore-drill.sh on a live repository dump: 29 repositories, 0 fsck failures, 29 live; the dump was deleted after.
42 lines
2.5 KiB
Bash
42 lines
2.5 KiB
Bash
#!/usr/bin/env bash
|
|
# Test k3s-datastore-restore-drill.sh without production data: start a fresh,
|
|
# network-less k3s with secrets encryption, add three test Secrets, stop it,
|
|
# and run the drill on its state.db and token. Needs docker; no root.
|
|
set -Eeuo pipefail
|
|
umask 077
|
|
|
|
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
|
|
readonly K3S_IMAGE="$(sed -n 's/^readonly K3S_IMAGE="\(.*\)"$/\1/p' "$ROOT/scripts/validate/k3s-datastore-restore-drill.sh")"
|
|
[[ -n "$K3S_IMAGE" ]] || { echo "cannot read K3S_IMAGE from the drill script" >&2; exit 1; }
|
|
work="$(mktemp -d)"
|
|
seed="w6-k3s-drill-seed-$$"
|
|
cleanup() {
|
|
docker rm -f "$seed" >/dev/null 2>&1 || true
|
|
docker run --rm -v "$work:/w" --entrypoint /bin/sh "$K3S_IMAGE" -c 'rm -rf /w/*' >/dev/null 2>&1 || true
|
|
rm -rf -- "$work"
|
|
}
|
|
trap cleanup EXIT
|
|
mkdir -p "$work/server" "$work/copy"
|
|
|
|
docker run -d --name "$seed" --network none --tmpfs /run --tmpfs /var/run \
|
|
-v "$work/server:/var/lib/rancher/k3s/server" "$K3S_IMAGE" server --disable-agent --secrets-encryption \
|
|
--bind-address 127.0.0.1 --advertise-address 127.0.0.1 --node-ip 127.0.0.1 \
|
|
--disable traefik,servicelb,metrics-server,local-storage,coredns >/dev/null
|
|
for _ in $(seq 1 60); do docker exec "$seed" kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 5; done
|
|
for i in 1 2 3; do docker exec "$seed" kubectl create secret generic "drill-test-$i" --from-literal=value="test-$i" >/dev/null; done
|
|
expected="$(docker exec "$seed" kubectl get secrets -A --no-headers | wc -l)"
|
|
docker stop -t 30 "$seed" >/dev/null
|
|
docker run --rm -v "$work:/w" --entrypoint /bin/sh "$K3S_IMAGE" -c \
|
|
"cp /w/server/db/state.db* /w/server/token /w/copy/ && chown $(id -u):$(id -g) /w/copy/*"
|
|
# Fold the WAL into state.db, as `sqlite3 .backup` does in the real backup.
|
|
sqlite3 "$work/copy/state.db" 'PRAGMA wal_checkpoint(TRUNCATE);' >/dev/null
|
|
rm -f "$work/copy/state.db-wal" "$work/copy/state.db-shm"
|
|
# The test Secrets must be stored encrypted (k8s:enc:...), so the drill really
|
|
# proves that the token unlocks the encryption key.
|
|
encrypted="$(sqlite3 "$work/copy/state.db" "SELECT COUNT(*) FROM kine WHERE name LIKE '/registry/secrets/default/drill-test-%' AND CAST(value AS TEXT) LIKE 'k8s:enc:%';")"
|
|
echo "seed: $expected Secrets, encrypted test Secret rows=$encrypted"
|
|
(( encrypted >= 3 )) || { echo "test Secrets are not encrypted at rest in the seed datastore" >&2; exit 1; }
|
|
|
|
bash "$ROOT/scripts/validate/k3s-datastore-restore-drill.sh" --from-dir "$work/copy" --expect-secrets "$expected"
|
|
echo "K3S DATASTORE RESTORE DRILL TEST PASS"
|