feat(validate): add restore drills for the k3s and Gitea backups
A backup nobody has restored is not known to work. The 2026-08 k3s recovery bundle never got its isolated restore drill. k3s-datastore-restore-drill.sh downloads the latest archive, checks its sha256 and integrity, starts the same k3s version in a network-less throwaway container on the copy, and checks the API comes up, the Secret count is close to live and every Secret decrypts with the token. gitea-backup-restore-drill.sh extracts the latest dump and checks app.ini (INTERNAL_TOKEN, SECRET_KEY), git fsck on every repository and the repository count against live Gitea. Both accept a local copy and write a drill timestamp metric. Risk: read-only for the cluster. The k3s drill runs the restored control plane with --network none, so its controllers reach nothing. Verification (desktop): - test-k3s-datastore-restore-drill.sh: a fresh k3s with three test Secrets (stored encrypted, k8s:enc:) restores with all Secrets decrypted in 7s; before folding the WAL the test saw 2 of 3 rows, which is why the real backup uses sqlite3 .backup. - gitea-backup-restore-drill.sh on a live repository dump: 29 repositories, 0 fsck failures, 29 live; the dump was deleted after.
This commit is contained in:
@@ -596,3 +596,13 @@ PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm \
|
||||
- `--textfile FILE`: `platform_live_drift_objects`, `platform_live_drift_check_success`,
|
||||
`platform_live_drift_last_run_timestamp_seconds`를 Prometheus textfile 형식으로 쓴다.
|
||||
- Secret 값의 차이는 보지 않는다.
|
||||
|
||||
## 호스트 백업·점검 시험
|
||||
|
||||
- `test-platform-host-checks-alerts.sh`: `services/platform-host-checks/prometheus-rules.yaml`의 promtool
|
||||
단위 시험(백업 신선도, 실패, 지표 없음, drift, k3s 인증서). docker 필요, 클러스터 접근 없음.
|
||||
- `k3s-cert-expiry.sh`: 6443·10250 serving 인증서 만료를 textfile 지표로 쓴다(타이머가 매일 돌림).
|
||||
- `live-drift-check-main.sh`: checkout의 작업 트리가 아니라 `origin/main`으로 `live-drift-check.sh`를 돌린다.
|
||||
- `k3s-datastore-restore-drill.sh`, `gitea-backup-restore-drill.sh`: 복원 리허설. 인자 없이 root로 돌리면 AIStor의
|
||||
최신 백업을 쓰고, `--from-dir`·`--from-file`로 로컬 사본을 줄 수 있다.
|
||||
- `test-k3s-datastore-restore-drill.sh`: 운영 데이터 없이 k3s 복원 리허설 경로를 시험한다(일회용 k3s, 시험 Secret 셋).
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
#!/usr/bin/env bash
|
||||
# Restore drill for the Gitea backup (file level).
|
||||
#
|
||||
# sudo bash scripts/validate/gitea-backup-restore-drill.sh # latest dump in AIStor
|
||||
# bash scripts/validate/gitea-backup-restore-drill.sh --from-file DUMP.tar.gz
|
||||
#
|
||||
# Extracts the dump into a throwaway directory and checks what a restore needs:
|
||||
# app.ini is present, every repository passes `git fsck`, and the repository
|
||||
# count matches the live Gitea. The database is restored from the CloudNativePG
|
||||
# backup instead (services/platform-postgres-backup); the full restore order is
|
||||
# in docs/runbooks/gitea-restore.md. Nothing is written to the cluster.
|
||||
set -Eeuo pipefail
|
||||
umask 077
|
||||
|
||||
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
|
||||
# shellcheck source=../lib/host-backup.sh
|
||||
source "$ROOT/scripts/lib/host-backup.sh"
|
||||
readonly NAME=gitea_restore_drill
|
||||
|
||||
from_file=""
|
||||
case "${1:-}" in
|
||||
--from-file) from_file="${2:?--from-file needs a path}" ;;
|
||||
"") ;;
|
||||
*) sed -n '2,11p' "$0" >&2; exit 2 ;;
|
||||
esac
|
||||
if [[ "$(id -u)" -eq 0 ]]; then
|
||||
kubectl() { command /usr/local/bin/k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml "$@"; }
|
||||
fi
|
||||
|
||||
base="$HB_WORK_ROOT"; [[ -w "$base" ]] || base="$(mktemp -d)"
|
||||
work="$(mktemp -d "$base/gitea-drill.XXXXXX")"
|
||||
trap 'rm -rf -- "$work"' EXIT
|
||||
|
||||
if [[ -z "$from_file" ]]; then
|
||||
[[ "$(id -u)" -eq 0 ]] || hb_fail "run as root to read the backup credentials"
|
||||
hb_load_aistor_env
|
||||
latest="$(hb_mc "$work" ls "backup/$AISTOR_BUCKET/gitea/" | awk '{print $NF}' | grep -E '^gitea-dump-.*\.tar\.gz$' | sort | tail -1)"
|
||||
[[ -n "$latest" ]] || hb_fail "no Gitea dump found"
|
||||
hb_mc "$work" cp --quiet "backup/$AISTOR_BUCKET/gitea/$latest" "backup/$AISTOR_BUCKET/gitea/$latest.sha256" /work/ >/dev/null
|
||||
[[ "$(sha256sum "$work/$latest" | awk '{print $1}')" == "$(cat "$work/$latest.sha256")" ]] || hb_fail "checksum mismatch for $latest"
|
||||
from_file="$work/$latest"
|
||||
hb_log "drill on $latest"
|
||||
fi
|
||||
|
||||
mkdir "$work/restore"
|
||||
tar -C "$work/restore" -xzf "$from_file"
|
||||
[[ -s "$work/restore/app.ini" ]] || hb_fail "app.ini is missing from the dump"
|
||||
for key in INTERNAL_TOKEN SECRET_KEY; do
|
||||
grep -q "^${key}[[:space:]]*=" "$work/restore/app.ini" || hb_fail "app.ini has no ${key} (needed to restore)"
|
||||
done
|
||||
|
||||
total=0 broken=0
|
||||
while IFS= read -r -d '' repo; do
|
||||
total=$((total + 1))
|
||||
if ! git -c safe.directory='*' --git-dir="$repo" fsck --no-dangling --no-progress >/dev/null 2>"$work/fsck.err"; then
|
||||
broken=$((broken + 1))
|
||||
hb_log "fsck failed: ${repo#"$work/restore/"}: $(head -1 "$work/fsck.err")"
|
||||
fi
|
||||
done < <(find "$work/restore/repos" -mindepth 2 -maxdepth 2 -type d -name '*.git' -print0)
|
||||
hb_log "repositories in dump: $total, fsck failures: $broken"
|
||||
(( total > 0 )) || hb_fail "the dump has no repositories"
|
||||
(( broken == 0 )) || hb_fail "$broken repositories failed git fsck"
|
||||
|
||||
live="$(kubectl -n gitea exec deploy/gitea -c gitea -- sh -c 'find /data/git/gitea-repositories -mindepth 2 -maxdepth 2 -name "*.git" -type d | wc -l' 2>/dev/null || echo "")"
|
||||
if [[ -n "$live" ]]; then
|
||||
hb_log "live Gitea repositories: $live"
|
||||
(( total >= live - 2 )) || hb_fail "the dump has $total repositories, live Gitea has $live"
|
||||
fi
|
||||
|
||||
hb_write_metrics "$NAME" <<PROM
|
||||
# HELP platform_backup_restore_drill_last_success_timestamp_seconds Unix time of the last successful restore drill.
|
||||
# TYPE platform_backup_restore_drill_last_success_timestamp_seconds gauge
|
||||
platform_backup_restore_drill_last_success_timestamp_seconds{backup="gitea"} $(date +%s)
|
||||
PROM
|
||||
echo "GITEA RESTORE DRILL PASS"
|
||||
@@ -0,0 +1,99 @@
|
||||
#!/usr/bin/env bash
|
||||
# Restore drill for the k3s datastore backup.
|
||||
#
|
||||
# sudo bash scripts/validate/k3s-datastore-restore-drill.sh # latest backup in AIStor
|
||||
# bash scripts/validate/k3s-datastore-restore-drill.sh --from-dir DIR [--expect-secrets N]
|
||||
# DIR holds state.db and token (an extracted archive, or a test datastore)
|
||||
#
|
||||
# Starts a throwaway k3s server (same version as production, --network none,
|
||||
# no agent) on a copy of the backup, and checks that the API answers, that the
|
||||
# Secret count matches and that every Secret decrypts (the token unlocks the
|
||||
# encryption config stored in the datastore). Prints no Secret value. The
|
||||
# container and the copy are removed at the end.
|
||||
set -Eeuo pipefail
|
||||
umask 077
|
||||
|
||||
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
|
||||
# shellcheck source=../lib/host-backup.sh
|
||||
source "$ROOT/scripts/lib/host-backup.sh"
|
||||
# Same version as the desktop server (k3s --version: v1.36.2+k3s1).
|
||||
readonly K3S_IMAGE="docker.io/rancher/k3s:v1.36.2-k3s1@sha256:6a47cea22c4b834d4ba72c89d291696b79ebe406251f90b446e4dff03513dd87"
|
||||
readonly NAME=k3s_datastore_restore_drill
|
||||
|
||||
from_dir="" expect_secrets=""
|
||||
while (( $# > 0 )); do
|
||||
case "$1" in
|
||||
--from-dir) from_dir="${2:?}"; shift 2 ;;
|
||||
--expect-secrets) expect_secrets="${2:?}"; shift 2 ;;
|
||||
*) sed -n '2,13p' "$0" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
base="${HB_WORK_ROOT}"
|
||||
[[ -w "$base" ]] || base="$(mktemp -d)"
|
||||
work="$(mktemp -d "$base/k3s-drill.XXXXXX")"
|
||||
container="k3s-restore-drill-$$"
|
||||
cleanup() {
|
||||
docker rm -f "$container" >/dev/null 2>&1 || true
|
||||
# The drill container writes as root; remove its files with the same image.
|
||||
docker run --rm -v "$work:/w" --entrypoint /bin/sh "$K3S_IMAGE" -c 'rm -rf /w/*' >/dev/null 2>&1 || true
|
||||
rm -rf -- "$work"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
if [[ -z "$from_dir" ]]; then
|
||||
[[ "$(id -u)" -eq 0 ]] || hb_fail "run as root to read the backup credentials"
|
||||
hb_load_aistor_env
|
||||
latest="$(hb_mc "$work" ls "backup/$AISTOR_BUCKET/k3s-datastore/" | awk '{print $NF}' | grep -E '^k3s-datastore-.*\.tar\.gz$' | sort | tail -1)"
|
||||
[[ -n "$latest" ]] || hb_fail "no k3s datastore backup found"
|
||||
hb_mc "$work" cp --quiet "backup/$AISTOR_BUCKET/k3s-datastore/$latest" "backup/$AISTOR_BUCKET/k3s-datastore/$latest.sha256" /work/ >/dev/null
|
||||
[[ "$(sha256sum "$work/$latest" | awk '{print $1}')" == "$(cat "$work/$latest.sha256")" ]] || hb_fail "checksum mismatch for $latest"
|
||||
tar -C "$work" -xzf "$work/$latest"
|
||||
from_dir="$(find "$work" -mindepth 1 -maxdepth 1 -type d -name 'k3s-datastore-*' | head -1)"
|
||||
hb_log "drill on $latest"
|
||||
if [[ -z "$expect_secrets" ]]; then
|
||||
expect_secrets="$(/usr/local/bin/k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get secrets -A --no-headers | wc -l)"
|
||||
hb_log "live cluster has $expect_secrets Secrets (counts may differ by objects created since the backup)"
|
||||
fi
|
||||
fi
|
||||
[[ -f "$from_dir/state.db" && -f "$from_dir/token" ]] || hb_fail "$from_dir must contain state.db and token"
|
||||
|
||||
install -d -m 0700 "$work/server/db"
|
||||
cp "$from_dir/state.db" "$work/server/db/state.db"
|
||||
cp "$from_dir/token" "$work/server/token"
|
||||
[[ "$(sqlite3 "$work/server/db/state.db" 'PRAGMA integrity_check;')" == "ok" ]] || hb_fail "integrity_check failed"
|
||||
|
||||
started="$(date +%s)"
|
||||
docker run -d --name "$container" --network none --tmpfs /run --tmpfs /var/run \
|
||||
-v "$work/server:/var/lib/rancher/k3s/server" \
|
||||
"$K3S_IMAGE" server --disable-agent --secrets-encryption \
|
||||
--token-file /var/lib/rancher/k3s/server/token \
|
||||
--bind-address 127.0.0.1 --advertise-address 127.0.0.1 --node-ip 127.0.0.1 \
|
||||
--disable traefik,servicelb,metrics-server,local-storage,coredns >/dev/null
|
||||
|
||||
ready=0
|
||||
for _ in $(seq 1 60); do
|
||||
if docker exec "$container" kubectl get --raw /readyz >/dev/null 2>&1; then ready=1; break; fi
|
||||
sleep 5
|
||||
done
|
||||
if (( ready == 0 )); then
|
||||
docker logs --tail 20 "$container" 2>&1 | sed 's/^/ k3s: /' >&2
|
||||
hb_fail "restored k3s did not become ready in 5 minutes"
|
||||
fi
|
||||
namespaces="$(docker exec "$container" kubectl get namespaces --no-headers | wc -l)"
|
||||
secrets="$(docker exec "$container" kubectl get secrets -A --no-headers | wc -l)"
|
||||
# -o json decrypts every Secret on the server side; any failure makes kubectl exit non-zero.
|
||||
decrypted="$(docker exec "$container" kubectl get secrets -A -o json | python3 -I -c 'import json,sys; d=json.load(sys.stdin)["items"]; print(sum(1 for s in d if s.get("data")))')"
|
||||
hb_log "restored: namespaces=$namespaces secrets=$secrets with-data=$decrypted in $(( $(date +%s) - started ))s"
|
||||
if [[ -n "$expect_secrets" ]]; then
|
||||
diff=$(( secrets - expect_secrets ))
|
||||
(( ${diff#-} <= 5 )) || hb_fail "Secret count $secrets is far from the expected $expect_secrets"
|
||||
fi
|
||||
(( decrypted > 0 )) || hb_fail "no Secret carried data after restore"
|
||||
|
||||
hb_write_metrics "$NAME" <<PROM
|
||||
# HELP platform_backup_restore_drill_last_success_timestamp_seconds Unix time of the last successful restore drill.
|
||||
# TYPE platform_backup_restore_drill_last_success_timestamp_seconds gauge
|
||||
platform_backup_restore_drill_last_success_timestamp_seconds{backup="k3s-datastore"} $(date +%s)
|
||||
PROM
|
||||
echo "K3S DATASTORE RESTORE DRILL PASS"
|
||||
@@ -0,0 +1,41 @@
|
||||
#!/usr/bin/env bash
|
||||
# Test k3s-datastore-restore-drill.sh without production data: start a fresh,
|
||||
# network-less k3s with secrets encryption, add three test Secrets, stop it,
|
||||
# and run the drill on its state.db and token. Needs docker; no root.
|
||||
set -Eeuo pipefail
|
||||
umask 077
|
||||
|
||||
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
|
||||
readonly K3S_IMAGE="$(sed -n 's/^readonly K3S_IMAGE="\(.*\)"$/\1/p' "$ROOT/scripts/validate/k3s-datastore-restore-drill.sh")"
|
||||
[[ -n "$K3S_IMAGE" ]] || { echo "cannot read K3S_IMAGE from the drill script" >&2; exit 1; }
|
||||
work="$(mktemp -d)"
|
||||
seed="w6-k3s-drill-seed-$$"
|
||||
cleanup() {
|
||||
docker rm -f "$seed" >/dev/null 2>&1 || true
|
||||
docker run --rm -v "$work:/w" --entrypoint /bin/sh "$K3S_IMAGE" -c 'rm -rf /w/*' >/dev/null 2>&1 || true
|
||||
rm -rf -- "$work"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
mkdir -p "$work/server" "$work/copy"
|
||||
|
||||
docker run -d --name "$seed" --network none --tmpfs /run --tmpfs /var/run \
|
||||
-v "$work/server:/var/lib/rancher/k3s/server" "$K3S_IMAGE" server --disable-agent --secrets-encryption \
|
||||
--bind-address 127.0.0.1 --advertise-address 127.0.0.1 --node-ip 127.0.0.1 \
|
||||
--disable traefik,servicelb,metrics-server,local-storage,coredns >/dev/null
|
||||
for _ in $(seq 1 60); do docker exec "$seed" kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 5; done
|
||||
for i in 1 2 3; do docker exec "$seed" kubectl create secret generic "drill-test-$i" --from-literal=value="test-$i" >/dev/null; done
|
||||
expected="$(docker exec "$seed" kubectl get secrets -A --no-headers | wc -l)"
|
||||
docker stop -t 30 "$seed" >/dev/null
|
||||
docker run --rm -v "$work:/w" --entrypoint /bin/sh "$K3S_IMAGE" -c \
|
||||
"cp /w/server/db/state.db* /w/server/token /w/copy/ && chown $(id -u):$(id -g) /w/copy/*"
|
||||
# Fold the WAL into state.db, as `sqlite3 .backup` does in the real backup.
|
||||
sqlite3 "$work/copy/state.db" 'PRAGMA wal_checkpoint(TRUNCATE);' >/dev/null
|
||||
rm -f "$work/copy/state.db-wal" "$work/copy/state.db-shm"
|
||||
# The test Secrets must be stored encrypted (k8s:enc:...), so the drill really
|
||||
# proves that the token unlocks the encryption key.
|
||||
encrypted="$(sqlite3 "$work/copy/state.db" "SELECT COUNT(*) FROM kine WHERE name LIKE '/registry/secrets/default/drill-test-%' AND CAST(value AS TEXT) LIKE 'k8s:enc:%';")"
|
||||
echo "seed: $expected Secrets, encrypted test Secret rows=$encrypted"
|
||||
(( encrypted >= 3 )) || { echo "test Secrets are not encrypted at rest in the seed datastore" >&2; exit 1; }
|
||||
|
||||
bash "$ROOT/scripts/validate/k3s-datastore-restore-drill.sh" --from-dir "$work/copy" --expect-secrets "$expected"
|
||||
echo "K3S DATASTORE RESTORE DRILL TEST PASS"
|
||||
Reference in New Issue
Block a user