feat(validate): add restore drills for the k3s and Gitea backups
A backup nobody has restored is not known to work. The 2026-08 k3s recovery bundle never got its isolated restore drill. k3s-datastore-restore-drill.sh downloads the latest archive, checks its sha256 and integrity, starts the same k3s version in a network-less throwaway container on the copy, and checks the API comes up, the Secret count is close to live and every Secret decrypts with the token. gitea-backup-restore-drill.sh extracts the latest dump and checks app.ini (INTERNAL_TOKEN, SECRET_KEY), git fsck on every repository and the repository count against live Gitea. Both accept a local copy and write a drill timestamp metric. Risk: read-only for the cluster. The k3s drill runs the restored control plane with --network none, so its controllers reach nothing. Verification (desktop): - test-k3s-datastore-restore-drill.sh: a fresh k3s with three test Secrets (stored encrypted, k8s:enc:) restores with all Secrets decrypted in 7s; before folding the WAL the test saw 2 of 3 rows, which is why the real backup uses sqlite3 .backup. - gitea-backup-restore-drill.sh on a live repository dump: 29 repositories, 0 fsck failures, 29 live; the dump was deleted after.
This commit is contained in:
@@ -596,3 +596,13 @@ PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm \
|
||||
- `--textfile FILE`: `platform_live_drift_objects`, `platform_live_drift_check_success`,
|
||||
`platform_live_drift_last_run_timestamp_seconds`를 Prometheus textfile 형식으로 쓴다.
|
||||
- Secret 값의 차이는 보지 않는다.
|
||||
|
||||
## 호스트 백업·점검 시험
|
||||
|
||||
- `test-platform-host-checks-alerts.sh`: `services/platform-host-checks/prometheus-rules.yaml`의 promtool
|
||||
단위 시험(백업 신선도, 실패, 지표 없음, drift, k3s 인증서). docker 필요, 클러스터 접근 없음.
|
||||
- `k3s-cert-expiry.sh`: 6443·10250 serving 인증서 만료를 textfile 지표로 쓴다(타이머가 매일 돌림).
|
||||
- `live-drift-check-main.sh`: checkout의 작업 트리가 아니라 `origin/main`으로 `live-drift-check.sh`를 돌린다.
|
||||
- `k3s-datastore-restore-drill.sh`, `gitea-backup-restore-drill.sh`: 복원 리허설. 인자 없이 root로 돌리면 AIStor의
|
||||
최신 백업을 쓰고, `--from-dir`·`--from-file`로 로컬 사본을 줄 수 있다.
|
||||
- `test-k3s-datastore-restore-drill.sh`: 운영 데이터 없이 k3s 복원 리허설 경로를 시험한다(일회용 k3s, 시험 Secret 셋).
|
||||
|
||||
Reference in New Issue
Block a user