feat(observability): alert on host backups, drift and k3s serving certs

Backups, the daily drift check and the k3s serving certificates have no
alert. k3s renews certificates only when it starts within 120 days of
expiry, and the two kube-prometheus-stack kubelet certificate alerts
can never fire because the k3s kubelet exports no
kubelet_certificate_manager_* metric.

Add a PrometheusRule (platform-host-checks) for backup age, failed
runs, missing metrics, drift left for 6h, a stale drift check, and
serving certificate expiry (warning under 100 days, critical under 14).
Add the two remaining host timers (certificate expiry via openssl on
6443 and 10250; the drift check against origin/main) and an installer
that copies the scripts to a root-owned path, creates the textfile and
work directories and enables all four timers. Disable the two dead
kubelet certificate rules in kube-prometheus-stack.

Risk: the metrics need the node-exporter textfile collector, a one-line
change in W1's node-exporter values that was asked of W1. Until then
PlatformHostBackupMetricsAbsent fires after a day, which is the point.

Verification (desktop):
- test-platform-host-checks-alerts.sh: promtool check and unit tests
  pass; with the k3s threshold changed from 2 to 5 days they fail.
- render-observability-core.sh passes; the render has no Kubelet*
  certificate alerts and still has both KubeClientCertificateExpiration.
- k3s-cert-expiry.sh as the desktop user reads both certificates.
- install-host-checks.sh --check: systemd-analyze verify is clean.
This commit is contained in:
DongHyeonka
2026-11-12 01:13:02 +09:00
parent 77daa39a41
commit 012452cf78
12 changed files with 494 additions and 0 deletions
+83
View File
@@ -0,0 +1,83 @@
# 호스트 백업과 점검
desktop 호스트에서 systemd 타이머로 도는 백업과 점검이다. 결과는 node-exporter textfile 수집기
(`/var/lib/node_exporter/textfile_collector/*.prom`)로 Prometheus에 올라가고, 이 폴더의 PrometheusRule이
경보를 낸다. 근거와 결정은 `docs/reviews/2026-11-11-k3s-maturity-review.md` A1·A11·A13·A14, 8절 결정 1·6이다.
## 무엇을 어디에 얼마나
| 대상 | 방법 | 주기 | 보관 | 위치 |
| --- | --- | --- | --- | --- |
| k3s datastore + `server/token` | `sqlite3 .backup` → tar.gz + sha256 | 매일 03:30 KST | 30일 | AIStor `platform-host-backups/k3s-datastore/` (HDD sdb3) |
| Gitea (저장소·패키지·첨부·Actions·`app.ini`) | 라이브 파드에서 `gitea dump --skip-db` | 일요일 04:00 KST | 56일(8주) | AIStor `platform-host-backups/gitea/` |
| platform-postgres (Gitea·Keycloak·tech-log DB) | CloudNativePG base backup + WAL | 매일 03:00 KST | 30일 | AIStor `platform-postgres-backups/` (W1, `services/platform-postgres-backup`) |
| AIStor 데이터(Loki·Tempo) | **백업하지 않는다** | — | — | 8절 결정 6: 잃어도 되는 관측 데이터. 디스크를 잃으면 로그·추적 기록만 사라진다 |
| Prometheus·Grafana·Alertmanager PV | 백업하지 않는다 | — | — | 설정은 이 저장소가 정본이고, 지표는 다시 쌓인다 |
- Gitea DB는 dump에 없다(`--skip-db`). Gitea를 복원하려면 CNPG 복원이 먼저다(`docs/runbooks/gitea-restore.md`).
- k3s 백업에는 `token`이 들어 있다. token과 datastore가 같이 있으면 클러스터의 모든 Secret을 풀 수 있다.
Gitea dump의 `app.ini`에도 비밀 값이 있다. 그래서 둘 다 AIStor의 전용 사용자만 읽고 쓴다.
- AIStor는 같은 desktop의 HDD다. SSD가 죽으면 남지만 desktop 자체를 잃으면 같이 잃는다.
## off-host 사본 (준비만, 기본 꺼짐)
위치는 소유자가 정한다(8절 결정 1). `/etc/platform-backup/aistor.env`에 두 줄을 넣으면 k3s datastore
백업이 그 공개 키로 암호화한 사본을 `rclone`으로 보낸다.
```bash
OFFHOST_GPG_RECIPIENT_FILE=/etc/platform-backup/offhost-recipient.asc # 공개 키. 개인 키는 desktop에 두지 않는다
OFFHOST_RCLONE_TARGET=<rclone remote>:<path> # rclone 설치·설정 필요
```
지표 `platform_backup_offhost_copy{backup="k3s-datastore"}`가 1이 된다. Gitea dump의 off-host 사본은 아직 없다.
## 점검
| 타이머 | 사용자 | 하는 일 | 지표 |
| --- | --- | --- | --- |
| `platform-k3s-cert-expiry` | donghyeon | 6443·10250 serving 인증서 notAfter | `platform_k3s_certificate_not_after_seconds` |
| `platform-live-drift-check` | donghyeon | origin/main 렌더 → `kubectl diff`(서버 dry-run) | `platform_live_drift_objects` |
## 경보 (`prometheus-rules.yaml`)
| 경보 | 조건 |
| --- | --- |
| `PlatformK3sDatastoreBackupTooOld` | 마지막 성공이 2일보다 오래됨 |
| `PlatformGiteaBackupTooOld` | 8일보다 오래됨 |
| `PlatformHostBackupFailed` | 마지막 실행 실패 |
| `PlatformHostBackupMetricsAbsent` | 지표가 하루 동안 없음(타이머나 textfile 수집기가 안 돎) |
| `PlatformLiveDriftDetected` | drift가 6시간 넘게 남음 |
| `PlatformLiveDriftCheckStale` | drift 검사가 2일 넘게 완료되지 않음 |
| `PlatformK3sServingCertificateExpiry` | 100일 미만 warning, 14일 미만 critical |
| `PlatformK3sCertificateCheckStale` | 인증서 점검이 2일 넘게 안 돎 |
`scripts/validate/test-platform-host-checks-alerts.sh`가 promtool 단위 시험으로 각 경보가 울려야 할 때 울리고
아닐 때 안 울리는지 본다. 같은 PR에서 kube-prometheus-stack의 `KubeletClientCertificateExpiration`·
`KubeletServerCertificateExpiration`을 껐다. k3s kubelet은 그 지표를 내지 않아 울릴 수 없었다.
## 적용 (소유자, 순서대로)
전제: node-exporter textfile 수집기(W1, `services/observability/node-exporter/values/home.yaml`의
`--collector.textfile.directory=/host/root/var/lib/node_exporter/textfile_collector`)가 들어간 main checkout.
```bash
cd /home/donghyeon/workspace/platform && git pull --ff-only
sudo bash scripts/bootstrap/create-host-backup-credentials.sh # bucket·정책·사용자·/etc/platform-backup/aistor.env (APPLY 입력)
bash scripts/bootstrap/install-host-checks.sh --check
sudo bash scripts/bootstrap/install-host-checks.sh --execute # 타이머 설치·켜기·첫 실행
kubectl apply -k services/platform-host-checks # 경보 규칙
sudo bash scripts/validate/k3s-datastore-restore-drill.sh # 첫 백업으로 복원 리허설
sudo bash scripts/validate/gitea-backup-restore-drill.sh
```
`install-host-checks.sh --execute` 마지막에 `platform_backup_last_run_success{backup="k3s-datastore"} 1`과
`{backup="gitea"} 1`이 보여야 한다. 리허설 둘은 각각 `K3S DATASTORE RESTORE DRILL PASS`, `GITEA RESTORE DRILL PASS`로 끝나야 한다.
## 복원 리허설
- k3s: 백업을 내려받아 sha256을 맞춘 뒤, 운영과 같은 버전의 k3s를 **네트워크 없는** 일회용 컨테이너로
띄워 API가 뜨는지, Secret 수가 라이브와 비슷한지(±5), 모든 Secret이 복호화되는지 본다.
`scripts/validate/test-k3s-datastore-restore-drill.sh`는 운영 데이터 없이 같은 경로를 시험한다(새 k3s에
시험 Secret 셋 → 암호화돼 저장된 것을 확인 → 복원).
- Gitea: dump를 풀어 `app.ini`(INTERNAL_TOKEN·SECRET_KEY), 모든 저장소의 `git fsck`, 라이브와 저장소 수를 본다.
- 분기마다 한 번 돌린다. 성공하면 `platform_backup_restore_drill_last_success_timestamp_seconds`가 갱신된다.
@@ -0,0 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- prometheus-rules.yaml
@@ -0,0 +1,88 @@
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-host-checks
namespace: observability
labels:
observability.hyeonworks.com/instance: home
app.kubernetes.io/part-of: platform
spec:
groups:
- name: platform-host-backups
rules:
- alert: PlatformK3sDatastoreBackupTooOld
expr: time() - max(platform_backup_last_success_timestamp_seconds{backup="k3s-datastore"}) > 2 * 86400
for: 1h
labels:
severity: warning
annotations:
summary: k3s datastore has not been backed up for more than 2 days
description: Check `journalctl -u platform-k3s-datastore-backup` on desktop. Daily timer, 03:30 KST.
- alert: PlatformGiteaBackupTooOld
expr: time() - max(platform_backup_last_success_timestamp_seconds{backup="gitea"}) > 8 * 86400
for: 1h
labels:
severity: warning
annotations:
summary: Gitea has not been backed up for more than 8 days
description: Check `journalctl -u platform-gitea-backup` on desktop. Weekly timer, Sunday 04:00 KST.
- alert: PlatformHostBackupFailed
expr: max by (backup) (platform_backup_last_run_success) == 0
for: 10m
labels:
severity: warning
annotations:
summary: "The last {{ $labels.backup }} backup run failed"
description: "Check `journalctl -u platform-{{ $labels.backup }}-backup` on desktop."
- alert: PlatformHostBackupMetricsAbsent
expr: absent(platform_backup_last_run_timestamp_seconds{backup="k3s-datastore"}) or absent(platform_backup_last_run_timestamp_seconds{backup="gitea"})
for: 1d
labels:
severity: warning
annotations:
summary: Host backup metrics are missing
description: The backup timers or the node-exporter textfile collector (/var/lib/node_exporter/textfile_collector) are not running.
- name: platform-host-drift
rules:
- alert: PlatformLiveDriftDetected
expr: max(platform_live_drift_objects) > 0
for: 6h
labels:
severity: warning
annotations:
summary: "{{ $value }} live objects differ from platform-core origin/main"
description: Run `scripts/validate/live-drift-check.sh --accepted scripts/validate/live-drift-accepted.txt` on desktop to list them.
- alert: PlatformLiveDriftCheckStale
expr: time() - max(platform_live_drift_last_run_timestamp_seconds) > 2 * 86400 or max(platform_live_drift_check_success) == 0
for: 1h
labels:
severity: warning
annotations:
summary: The daily live drift check has not completed
description: Check `journalctl -u platform-live-drift-check` on desktop.
- name: platform-k3s-certificates
rules:
- alert: PlatformK3sServingCertificateExpiry
expr: (min by (certificate) (platform_k3s_certificate_not_after_seconds) - time()) < 100 * 86400
for: 1h
labels:
severity: warning
annotations:
summary: "k3s {{ $labels.certificate }} serving certificate expires in less than 100 days"
description: k3s renews certificates only when it starts within 120 days of expiry. Plan a k3s restart window (docs/runbooks).
- alert: PlatformK3sServingCertificateExpiry
expr: (min by (certificate) (platform_k3s_certificate_not_after_seconds) - time()) < 14 * 86400
for: 10m
labels:
severity: critical
annotations:
summary: "k3s {{ $labels.certificate }} serving certificate expires in less than 14 days"
description: Restart k3s (it renews certificates within 120 days of expiry) or run `k3s certificate rotate`.
- alert: PlatformK3sCertificateCheckStale
expr: time() - max(platform_k3s_certificate_check_timestamp_seconds) > 2 * 86400
for: 1h
labels:
severity: warning
annotations:
summary: The daily k3s certificate expiry check has not run
description: Check `journalctl -u platform-k3s-cert-expiry` on desktop.