Compare commits
39
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d0875bfe44 | ||
|
|
dd8e9140b7 | ||
|
|
5d27096ed0 | ||
|
|
ae4e256c95 | ||
|
|
27eb8c1799 | ||
|
|
149b885e35 | ||
|
|
205927f4e7 | ||
|
|
0d703ff249 | ||
|
|
c04d79481c | ||
|
|
d92dc5d163 | ||
|
|
4551a2ece6 | ||
|
|
d47d57aa14 | ||
|
|
d7c845ca5d | ||
|
|
2b1310972c | ||
|
|
bed20b456b | ||
|
|
619b90552f | ||
|
|
f19ce54740 | ||
|
|
1349ee2f25 | ||
|
|
fb5535aeb6 | ||
|
|
03ed6719b0 | ||
|
|
a5f0bff26e | ||
|
|
3bca2cf841 | ||
|
|
7fe38428e3 | ||
|
|
68a5f7d621 | ||
|
|
050a4de235 | ||
|
|
cc5decc538 | ||
|
|
8b019d26a0 | ||
|
|
28595c47e2 | ||
|
|
00a6680b76 | ||
|
|
4ab37ff18c | ||
|
|
cb78ccdb0b | ||
|
|
de600a0bb1 | ||
|
|
9751db0e97 | ||
|
|
9e9ea94461 | ||
|
|
ba3dddc139 | ||
|
|
012452cf78 | ||
|
|
77daa39a41 | ||
|
|
457d88e293 | ||
|
|
e88725aab8 |
@@ -76,44 +76,67 @@ Gradle·Docker 게이트는 `flock /tmp/claude-gates/heavy.lock`으로 서로
|
||||
## 디스크와 정리
|
||||
|
||||
안쪽 daemon의 데이터(job 이미지, Testcontainers 이미지, 빌드 cache)는 이름 있는
|
||||
볼륨 `gitea-runner_docker`에 쌓인다. desktop root 디스크는 228G 중 182G를 쓰고
|
||||
있다(2026-11-07, 여유 34G). 예상 사용량은 다음과 같다.
|
||||
볼륨 `gitea-runner_docker`에 쌓인다. 이 볼륨은 desktop 루트 SSD에 있고, 운영 k3s와
|
||||
같은 디스크다. 설치 이틀 만에 16G가 됐다(2026-11-09: 이미지 12.35G, 빌드 cache 3.4G).
|
||||
2026-11-11에는 이미지 12.43G, 빌드 cache 8.44G였다.
|
||||
|
||||
| 항목 | 크기 |
|
||||
| --- | --- |
|
||||
| runner 이미지(`gitea/runner:5.4.1-dind`) | 약 133MB |
|
||||
| job 이미지(`runner-images:ubuntu-24.04`) | amd64 압축 543MB, 안쪽 daemon에서 2.325GB(2026-11-07 실측) |
|
||||
| Testcontainers 이미지(PostgreSQL, Ryuk 등) | 수백 MB |
|
||||
| actions/cache(`/data/cache`) | 상한 5GB |
|
||||
상한은 두 가지로 건다.
|
||||
|
||||
runner는 job이 끝나면 job 컨테이너와 proxy로 만든 컨테이너·네트워크·볼륨을
|
||||
지우지만(`container.sweep: true`) 이미지는 지우지 않는다. 정리 주기는 다음과 같다.
|
||||
| 무엇 | 방법 | 값 |
|
||||
| --- | --- | --- |
|
||||
| 빌드 cache | `daemon.json`의 BuildKit GC. 안쪽 daemon이 스스로 지운다 | 최소 2GB, 상한 10GB. 디스크 여유가 35GB 아래면 2GB까지 |
|
||||
| job 이미지 | 호스트 systemd timer `gitea-runner-prune.timer`. 주 1회 | 만든 지 7일 지난 안 쓰는 이미지, 3일 지난 빌드 cache |
|
||||
| actions/cache | `config.yaml`의 `cache.size_limit` | 5GB |
|
||||
|
||||
- 매주 한 번 사용량을 본다.
|
||||
- BuildKit 키의 뜻과 값을 고른 이유는 [호스트 설정](../../host/README.md)의
|
||||
「BuildKit 상한 값」에 있다. 호스트와 같은 키에 더 작은 값을 쓴다.
|
||||
- Docker daemon은 이미지를 스스로 지우지 않는다. 그래서 이미지는 timer로 지운다.
|
||||
- `image prune --filter until=168h`는 이미지를 **만든** 시각으로 거른다. 마지막으로 쓴
|
||||
시각이 아니다. 그래서 고정한 job 이미지(`runner-images:ubuntu-24.04`, 2026-11-03 생성)도
|
||||
지워지고, 다음 job이 압축 약 543MB를 다시 받는다. 실행 중인 job 컨테이너가 쓰는
|
||||
이미지는 지워지지 않는다.
|
||||
- timer는 일요일 04:30(KST)에 돈다. 그 시각에 꺼져 있었으면 다음 부팅 때 돈다
|
||||
(`Persistent=true`). service는 `donghyeon` 사용자(docker group)로 돌아 root가 필요 없다.
|
||||
|
||||
```bash
|
||||
docker system df -v | grep gitea-runner
|
||||
docker exec gitea-runner docker system df
|
||||
```
|
||||
설치(소유자, sudo 한 번):
|
||||
|
||||
- 안쪽 사용량이 15GB를 넘거나 root 디스크 여유가 20G 아래로 내려가면 job이 없을 때
|
||||
안쪽 이미지를 정리한다. 고정된 job 이미지도 지워지므로 다음 job이 압축 약 543MB를
|
||||
다시 받는다.
|
||||
```bash
|
||||
sudo install -m 0644 infrastructure/ci/gitea-runner/gitea-runner-prune.service \
|
||||
infrastructure/ci/gitea-runner/gitea-runner-prune.timer /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now gitea-runner-prune.timer
|
||||
systemctl list-timers gitea-runner-prune.timer
|
||||
```
|
||||
|
||||
```bash
|
||||
docker exec gitea-runner docker image prune -a -f
|
||||
docker exec gitea-runner docker builder prune -f
|
||||
```
|
||||
바로 한 번 돌리려면 job이 없을 때 `sudo systemctl start gitea-runner-prune.service`를
|
||||
실행하고 `journalctl -u gitea-runner-prune.service -n 30`으로 결과를 본다.
|
||||
|
||||
- 볼륨 `gitea-runner_docker` 자체를 지우는 것은 runner를 완전히 지울 때만 한다
|
||||
(「되돌리기」). 등록 파일이 있는 `gitea-runner_data`와 섞지 않는다.
|
||||
되돌리기:
|
||||
|
||||
```bash
|
||||
sudo systemctl disable --now gitea-runner-prune.timer
|
||||
sudo rm /etc/systemd/system/gitea-runner-prune.service /etc/systemd/system/gitea-runner-prune.timer
|
||||
sudo systemctl daemon-reload
|
||||
```
|
||||
|
||||
사용량 보기:
|
||||
|
||||
```bash
|
||||
docker system df -v | grep gitea-runner
|
||||
docker exec gitea-runner docker system df
|
||||
docker exec gitea-runner docker buildx inspect default | sed -n '/GC Policy rule#1/,/Min Free/p'
|
||||
```
|
||||
|
||||
볼륨 `gitea-runner_docker` 자체를 지우는 것은 runner를 완전히 지울 때만 한다
|
||||
(「되돌리기」). 등록 파일이 있는 `gitea-runner_data`와 섞지 않는다.
|
||||
|
||||
## 파일과 위치
|
||||
|
||||
| 무엇 | 위치 |
|
||||
| --- | --- |
|
||||
| 원본 설정 | 이 디렉터리의 `compose.yaml`, `config.yaml` |
|
||||
| desktop 실행 폴더 | `/home/donghyeon/.local/share/gitea-runner/` (`compose.yaml`, `config.yaml`, `secrets/`) |
|
||||
| 원본 설정 | 이 디렉터리의 `compose.yaml`, `config.yaml`, `daemon.json` |
|
||||
| 주 1회 정리 | 이 디렉터리의 `gitea-runner-prune.service`·`.timer` → `/etc/systemd/system/` |
|
||||
| desktop 실행 폴더 | `/home/donghyeon/.local/share/gitea-runner/` (`compose.yaml`, `config.yaml`, `daemon.json`, `secrets/`) |
|
||||
| 등록 파일 `.runner` | Docker 볼륨 `gitea-runner_data`의 `/data/.runner` (runner 자격 증명, Secret으로 취급) |
|
||||
| job 이미지·안쪽 daemon 데이터 | Docker 볼륨 `gitea-runner_docker` (`/var/lib/docker`) |
|
||||
| actions/cache 저장소 | `gitea-runner_data`의 `/data/cache`, 상한 5GB |
|
||||
@@ -133,8 +156,10 @@ runner는 job이 끝나면 job 컨테이너와 proxy로 만든 컨테이너·네
|
||||
|
||||
```bash
|
||||
install -d -m 0700 ~/.local/share/gitea-runner ~/.local/share/gitea-runner/secrets
|
||||
install -m 0644 infrastructure/ci/gitea-runner/compose.yaml infrastructure/ci/gitea-runner/config.yaml ~/.local/share/gitea-runner/
|
||||
sha256sum infrastructure/ci/gitea-runner/*.yaml ~/.local/share/gitea-runner/*.yaml
|
||||
install -m 0644 infrastructure/ci/gitea-runner/compose.yaml infrastructure/ci/gitea-runner/config.yaml \
|
||||
infrastructure/ci/gitea-runner/daemon.json ~/.local/share/gitea-runner/
|
||||
sha256sum infrastructure/ci/gitea-runner/{compose.yaml,config.yaml,daemon.json} \
|
||||
~/.local/share/gitea-runner/{compose.yaml,config.yaml,daemon.json}
|
||||
```
|
||||
|
||||
2. 등록 token을 편집기로 `~/.local/share/gitea-runner/secrets/registration-token`에
|
||||
@@ -170,7 +195,7 @@ runner는 job이 끝나면 job 컨테이너와 proxy로 만든 컨테이너·네
|
||||
재생성은 실행 중인 job을 취소하므로 job이 없을 때 한다.
|
||||
- 이미지 올리기: `compose.yaml`의 tag와 digest를 함께 바꾼다. 등록 파일은 그대로 유효하다.
|
||||
major가 바뀌면 [Upgrading](https://docs.gitea.com/runner/upgrade/)의 호환 변경을 먼저 읽는다.
|
||||
- 디스크: 「디스크와 정리」의 주기를 따른다.
|
||||
- 디스크: 「디스크와 정리」의 상한과 timer가 맡는다. 매주 `list-timers`로 마지막 실행을 본다.
|
||||
|
||||
## 되돌리기
|
||||
|
||||
|
||||
@@ -22,6 +22,8 @@ services:
|
||||
CONFIG_FILE: /config.yaml
|
||||
volumes:
|
||||
- ./config.yaml:/config.yaml:ro
|
||||
# Caps the build cache of the inner Docker daemon. See README.md.
|
||||
- ./daemon.json:/etc/docker/daemon.json:ro
|
||||
- ./secrets:/run/gitea-runner:ro
|
||||
- data:/data
|
||||
- docker:/var/lib/docker
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"builder": {
|
||||
"gc": {
|
||||
"enabled": true,
|
||||
"defaultReservedSpace": "2GB",
|
||||
"defaultMaxUsedSpace": "10GB",
|
||||
"defaultMinFreeSpace": "35GB"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
# Weekly cleanup of the Docker daemon inside the gitea-runner container.
|
||||
# Installed by infrastructure/ci/gitea-runner/README.md ("디스크와 정리").
|
||||
[Unit]
|
||||
Description=Prune old images and build cache in the Gitea runner Docker
|
||||
Requires=docker.service
|
||||
After=docker.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
# The docker group is enough for docker exec. No root needed at run time.
|
||||
User=donghyeon
|
||||
Group=docker
|
||||
# image prune keeps images used by a running job container; until= filters
|
||||
# on image creation time, so the pinned job image is pulled again after a week.
|
||||
ExecStart=/usr/bin/docker exec gitea-runner docker image prune --all --force --filter until=168h
|
||||
ExecStart=/usr/bin/docker exec gitea-runner docker builder prune --all --force --filter until=72h
|
||||
ExecStartPost=/usr/bin/docker exec gitea-runner docker system df
|
||||
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=Weekly prune of the Gitea runner Docker
|
||||
|
||||
[Timer]
|
||||
# Sunday 04:30 local time (Asia/Seoul), when CI and agent sessions are quiet.
|
||||
OnCalendar=Sun *-*-* 04:30:00
|
||||
RandomizedDelaySec=30min
|
||||
# Run at the next boot if the machine was off at that time.
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,46 @@
|
||||
# CI 지표
|
||||
|
||||
desktop runner 하나가 받는 repo들의 CI 지표를 Gitea Actions API에서 뽑는다. 읽기만 하고 서버에 아무것도 쓰지 않는다. 지표의 정의와 목표는 design-docs `ci/manual.md` 10절이고, 주 1회 결과를 design-docs `ci/metrics.md`에 한 줄 남긴다.
|
||||
|
||||
## 돌리기
|
||||
|
||||
```bash
|
||||
python3 infrastructure/ci/metrics/ci_metrics.py \
|
||||
--repo clean-architecture-backend-template \
|
||||
--repo clean-architecture-frontend-template \
|
||||
--since 2026-11-07 --until 2026-11-13
|
||||
```
|
||||
|
||||
- 날짜는 KST 하루 단위이고 `--until`도 그날을 포함한다(`--utc-offset-hours`로 바꾼다).
|
||||
- 인증: `GITEA_TOKEN`(읽기 전용 토큰)이 있으면 그것을, 없으면 `git credential fill`의 그 호스트 자격 증명을 쓴다. 어느 쪽도 출력하지 않는다.
|
||||
- `--json`은 표 대신 숫자를 JSON으로 낸다.
|
||||
- 두 repo, 지난 7일이면 30초 안팎이다(run마다 jobs API를 한 번 부른다).
|
||||
|
||||
## 내는 것
|
||||
|
||||
| # | 지표 | 계산 |
|
||||
| --- | --- | --- |
|
||||
| 1 | 실행 시간 p50·p90 | 단계(`workflow:event`)별, 끝난 run(success·failure)의 job 실행 시간 합. cancelled run은 run 수에만 센다 |
|
||||
| 2 | 대기 시간 p50·p90 | run의 첫 job 시작 − run 생성 |
|
||||
| 3 | runner 사용률 | 날짜(job 시작 시각, KST)별 job 실행 시간 합 ÷ 1,440분. 준 repo를 모두 더한다(runner가 하나라서) |
|
||||
| 4 | PR당 run 수 | 기간 안에 병합된 PR마다 그 PR의 `pull_request` run 수. 병합 전 7일의 run까지 센다 |
|
||||
| 5 | main 빨강 시간 | workflow별 main push run이 처음 실패한 run의 생성부터 다음 성공 run의 끝까지. 기간 끝까지 빨강이면 그때까지 |
|
||||
|
||||
지표 6(흔들림)은 각 repo `docs/ledger/flaky-tests.jsonl`, 지표 7(캐시 적중)은 PR run 로그에서 나온다. 이 스크립트는 보지 않는다.
|
||||
|
||||
## 우리 서버(Gitea 1.27)가 API 명세와 다른 곳
|
||||
|
||||
2026-11-13 응답 한 건씩으로 확인했다. 계산은 서버를 따른다.
|
||||
|
||||
- run에 `created_at`이 없다. run의 job은 run과 함께 만들어지므로, job `created_at`의 가장 이른 값을 run 생성 시각으로 쓴다.
|
||||
- 시작하지 않은 run·job의 `started_at`·`completed_at`은 `1970-01-01T00:00:00Z`다. 없는 값으로 본다.
|
||||
- job의 `created_at`은 그 job이 실행할 수 있게 된 시각이 아니라 run의 생성 시각이다. runner 하나가 run의 job을 차례로 돌리므로 job마다 `started_at − created_at`을 대기로 세면 앞 job의 실행이 대기에 들어간다. 그래서 대기는 run 단위로 센다. `needs`가 있는 job이 늦게 풀려 대기열 뒤로 가는 시간은 이 지표에 안 보이고 실행 시간에도 안 들어간다.
|
||||
- run 목록의 run에는 `head_branch`가 없다. 단계와 ref는 `path`(`<workflow>@<ref>`)에서 읽고, PR 번호는 `pull_requests`에서 읽는다.
|
||||
|
||||
## 시험
|
||||
|
||||
```bash
|
||||
cd infrastructure/ci/metrics && python3 -m unittest -v test_ci_metrics
|
||||
```
|
||||
|
||||
서버 없이 계산만 본다. 만든 run·job 기록으로 대기·실행·빨강 시간·사용률을 확인한다.
|
||||
@@ -0,0 +1,384 @@
|
||||
#!/usr/bin/env python3
|
||||
"""CI metrics for the repositories one Gitea Actions runner serves.
|
||||
|
||||
Reads the Gitea Actions API (runs, their jobs, closed pull requests) and prints, per repository and
|
||||
stage, the CI manual's metrics 1-5: run execution time, queue wait, runner utilisation, pull
|
||||
request runs per merged pull request and time main stayed red. Read-only; it never writes to the
|
||||
server.
|
||||
|
||||
What our Gitea 1.27 returns differs from the gitea.com API reference in three places, and the
|
||||
arithmetic below follows the server:
|
||||
|
||||
* A run carries no ``created_at``. A run's jobs are all created with the run, so the earliest job
|
||||
``created_at`` is the run's creation time.
|
||||
* A run or job that has not started reports ``started_at`` and ``completed_at`` as
|
||||
``1970-01-01T00:00:00Z``. Those are treated as absent.
|
||||
* A job's ``created_at`` is the run's, not the moment it became runnable. One runner runs a run's
|
||||
jobs one after another, so ``started_at - created_at`` of a later job counts the earlier jobs'
|
||||
execution as waiting. Queue wait is therefore measured per run: first job start minus run
|
||||
creation.
|
||||
|
||||
Credentials: ``GITEA_TOKEN`` (a read-only token) if set, otherwise ``git credential fill`` for the
|
||||
server host. Neither is printed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import date, datetime, timedelta, timezone
|
||||
|
||||
EPOCH_PREFIXES = ("1970-01-01", "0001-01-01")
|
||||
FINISHED = {"success", "failure"}
|
||||
|
||||
|
||||
def parse_time(value: str | None) -> datetime | None:
|
||||
"""An API timestamp, or None for a missing or zero value."""
|
||||
if not value or value.startswith(EPOCH_PREFIXES):
|
||||
return None
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
|
||||
|
||||
def percentile(values: list[float], fraction: float) -> float | None:
|
||||
"""Nearest-rank percentile; None for no values."""
|
||||
if not values:
|
||||
return None
|
||||
ordered = sorted(values)
|
||||
rank = max(1, math.ceil(fraction * len(ordered)))
|
||||
return ordered[rank - 1]
|
||||
|
||||
|
||||
@dataclass
|
||||
class Job:
|
||||
name: str
|
||||
created: datetime | None
|
||||
started: datetime | None
|
||||
completed: datetime | None
|
||||
conclusion: str | None
|
||||
|
||||
@property
|
||||
def minutes(self) -> float:
|
||||
if self.started is None or self.completed is None:
|
||||
return 0.0
|
||||
return max(0.0, (self.completed - self.started).total_seconds() / 60)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Run:
|
||||
repo: str
|
||||
id: int
|
||||
workflow: str
|
||||
ref: str
|
||||
event: str
|
||||
status: str
|
||||
conclusion: str | None
|
||||
pull_request: int | None
|
||||
jobs: list[Job] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def stage(self) -> str:
|
||||
return f"{self.workflow}:{self.event}"
|
||||
|
||||
@property
|
||||
def created(self) -> datetime | None:
|
||||
times = [job.created for job in self.jobs if job.created is not None]
|
||||
return min(times) if times else None
|
||||
|
||||
@property
|
||||
def first_start(self) -> datetime | None:
|
||||
times = [job.started for job in self.jobs if job.started is not None]
|
||||
return min(times) if times else None
|
||||
|
||||
@property
|
||||
def completed(self) -> datetime | None:
|
||||
times = [job.completed for job in self.jobs if job.completed is not None]
|
||||
return max(times) if times and self.status == "completed" else None
|
||||
|
||||
@property
|
||||
def execution_minutes(self) -> float:
|
||||
return sum(job.minutes for job in self.jobs)
|
||||
|
||||
@property
|
||||
def queue_minutes(self) -> float | None:
|
||||
if self.created is None or self.first_start is None:
|
||||
return None
|
||||
return max(0.0, (self.first_start - self.created).total_seconds() / 60)
|
||||
|
||||
@property
|
||||
def main_push(self) -> bool:
|
||||
return self.event == "push" and self.ref == "refs/heads/main"
|
||||
|
||||
|
||||
def run_from_api(repo: str, raw: dict, jobs: list[dict]) -> Run:
|
||||
workflow, _, ref = raw.get("path", "").partition("@")
|
||||
pulls = raw.get("pull_requests") or []
|
||||
return Run(
|
||||
repo=repo,
|
||||
id=raw["id"],
|
||||
workflow=workflow,
|
||||
ref=ref,
|
||||
event=raw.get("event", ""),
|
||||
status=raw.get("status", ""),
|
||||
conclusion=raw.get("conclusion"),
|
||||
pull_request=pulls[0]["number"] if pulls else None,
|
||||
jobs=[
|
||||
Job(
|
||||
name=job.get("name", ""),
|
||||
created=parse_time(job.get("created_at")),
|
||||
started=parse_time(job.get("started_at")),
|
||||
completed=parse_time(job.get("completed_at")),
|
||||
conclusion=job.get("conclusion"),
|
||||
)
|
||||
for job in jobs
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
class Gitea:
|
||||
"""The few read-only API calls this script makes."""
|
||||
|
||||
def __init__(self, server: str):
|
||||
self.server = server.rstrip("/")
|
||||
self.header = self._authorization()
|
||||
|
||||
def _authorization(self) -> str:
|
||||
token = os.environ.get("GITEA_TOKEN")
|
||||
if token:
|
||||
return f"token {token}"
|
||||
host = urllib.parse.urlparse(self.server).netloc
|
||||
answer = subprocess.run(
|
||||
["git", "credential", "fill"],
|
||||
input=f"protocol=https\nhost={host}\n\n",
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
).stdout
|
||||
fields = dict(line.split("=", 1) for line in answer.splitlines() if "=" in line)
|
||||
pair = f"{fields['username']}:{fields['password']}".encode()
|
||||
return "Basic " + base64.b64encode(pair).decode()
|
||||
|
||||
def get(self, path: str) -> object:
|
||||
request = urllib.request.Request(
|
||||
f"{self.server}/api/v1{path}",
|
||||
headers={"Authorization": self.header, "Accept": "application/json"},
|
||||
)
|
||||
with urllib.request.urlopen(request, timeout=30) as response:
|
||||
return json.load(response)
|
||||
|
||||
|
||||
def fetch_runs(api: Gitea, owner: str, repo: str, since: datetime) -> list[Run]:
|
||||
"""Runs created at or after ``since``, newest first. Run ids grow with creation time."""
|
||||
runs: list[Run] = []
|
||||
page = 1
|
||||
while True:
|
||||
body = api.get(f"/repos/{owner}/{repo}/actions/runs?limit=50&page={page}")
|
||||
batch = body.get("workflow_runs") or []
|
||||
if not batch:
|
||||
return runs
|
||||
for raw in batch:
|
||||
jobs = api.get(f"/repos/{owner}/{repo}/actions/runs/{raw['id']}/jobs").get("jobs") or []
|
||||
run = run_from_api(repo, raw, jobs)
|
||||
if run.created is not None and run.created < since:
|
||||
return runs
|
||||
runs.append(run)
|
||||
page += 1
|
||||
|
||||
|
||||
def fetch_merged_pulls(api: Gitea, owner: str, repo: str, since: datetime, until: datetime) -> list[int]:
|
||||
numbers: list[int] = []
|
||||
page = 1
|
||||
while True:
|
||||
batch = api.get(f"/repos/{owner}/{repo}/pulls?state=closed&sort=recentupdate&limit=50&page={page}")
|
||||
if not batch:
|
||||
return numbers
|
||||
for pull in batch:
|
||||
merged = parse_time(pull.get("merged_at"))
|
||||
if pull.get("merged") and merged is not None and since <= merged < until:
|
||||
numbers.append(pull["number"])
|
||||
updated = [parse_time(pull.get("updated_at")) for pull in batch]
|
||||
if all(time is not None and time < since for time in updated):
|
||||
return numbers
|
||||
page += 1
|
||||
|
||||
|
||||
@dataclass
|
||||
class RedEpisode:
|
||||
workflow: str
|
||||
first_failed_run: int
|
||||
fixed_by_run: int | None
|
||||
hours: float
|
||||
|
||||
|
||||
def red_episodes(runs: list[Run], now: datetime) -> list[RedEpisode]:
|
||||
"""Main stays red from the first failing main push run's creation to the next green one's end."""
|
||||
episodes: list[RedEpisode] = []
|
||||
by_workflow: dict[str, list[Run]] = {}
|
||||
for run in runs:
|
||||
if run.main_push and run.status == "completed" and run.conclusion in FINISHED:
|
||||
by_workflow.setdefault(run.workflow, []).append(run)
|
||||
for workflow, history in by_workflow.items():
|
||||
history.sort(key=lambda run: run.created or now)
|
||||
start: Run | None = None
|
||||
for run in history:
|
||||
if run.conclusion == "failure" and start is None:
|
||||
start = run
|
||||
elif run.conclusion == "success" and start is not None:
|
||||
end = run.completed or now
|
||||
episodes.append(
|
||||
RedEpisode(workflow, start.id, run.id, (end - start.created).total_seconds() / 3600)
|
||||
)
|
||||
start = None
|
||||
if start is not None:
|
||||
episodes.append(RedEpisode(workflow, start.id, None, (now - start.created).total_seconds() / 3600))
|
||||
return episodes
|
||||
|
||||
|
||||
def stage_rows(runs: list[Run]) -> list[dict]:
|
||||
rows = []
|
||||
for stage in sorted({run.stage for run in runs}):
|
||||
finished = [run for run in runs if run.stage == stage and run.conclusion in FINISHED]
|
||||
waited = [run.queue_minutes for run in runs if run.stage == stage and run.queue_minutes is not None]
|
||||
executed = [run.execution_minutes for run in finished]
|
||||
rows.append(
|
||||
{
|
||||
"stage": stage,
|
||||
"runs": len([run for run in runs if run.stage == stage]),
|
||||
"finished": len(finished),
|
||||
"red": len([run for run in finished if run.conclusion == "failure"]),
|
||||
"exec_p50": percentile(executed, 0.5),
|
||||
"exec_p90": percentile(executed, 0.9),
|
||||
"wait_p50": percentile(waited, 0.5),
|
||||
"wait_p90": percentile(waited, 0.9),
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def utilisation(runs: list[Run], offset: timedelta) -> dict[date, float]:
|
||||
"""Job execution minutes per local day (job start day) across every repository given."""
|
||||
minutes: dict[date, float] = {}
|
||||
for run in runs:
|
||||
for job in run.jobs:
|
||||
if job.started is not None and job.minutes > 0:
|
||||
day = (job.started + offset).date()
|
||||
minutes[day] = minutes.get(day, 0.0) + job.minutes
|
||||
return minutes
|
||||
|
||||
|
||||
def runs_per_merged_pull(runs: list[Run], merged: list[int]) -> list[int]:
|
||||
return [len([run for run in runs if run.pull_request == number]) for number in merged]
|
||||
|
||||
|
||||
def fmt(value: float | None, digits: int = 1) -> str:
|
||||
return "-" if value is None else f"{value:.{digits}f}"
|
||||
|
||||
|
||||
def report(window: str, per_repo: dict[str, dict], days: dict[date, float]) -> str:
|
||||
lines = [f"# CI 지표 ({window})", ""]
|
||||
for repo, data in per_repo.items():
|
||||
lines += [
|
||||
f"## {repo}",
|
||||
"",
|
||||
"| 단계 | run | 끝난 run | 빨강 | 실행 p50(분) | 실행 p90(분) | 대기 p50(분) | 대기 p90(분) |",
|
||||
"|---|---|---|---|---|---|---|---|",
|
||||
]
|
||||
for row in data["stages"]:
|
||||
lines.append(
|
||||
f"| {row['stage']} | {row['runs']} | {row['finished']} | {row['red']} | {fmt(row['exec_p50'])} | "
|
||||
f"{fmt(row['exec_p90'])} | {fmt(row['wait_p50'])} | {fmt(row['wait_p90'])} |"
|
||||
)
|
||||
counts = data["runs_per_pull"]
|
||||
lines += [
|
||||
"",
|
||||
f"- 병합된 PR {len(counts)}개, PR당 run 수 p50 {fmt(percentile(counts, 0.5), 0)}, "
|
||||
f"p90 {fmt(percentile(counts, 0.9), 0)}, 최대 {max(counts) if counts else '-'}",
|
||||
]
|
||||
episodes = data["red"]
|
||||
if episodes:
|
||||
longest = max(episodes, key=lambda episode: episode.hours)
|
||||
lines.append(
|
||||
f"- main 빨강 {len(episodes)}회, 합 {sum(e.hours for e in episodes):.1f}시간, p90 "
|
||||
f"{fmt(percentile([e.hours for e in episodes], 0.9))}시간, 가장 긴 것 {longest.hours:.1f}시간"
|
||||
f"({longest.workflow} run {longest.first_failed_run} → "
|
||||
f"{longest.fixed_by_run if longest.fixed_by_run else '아직 빨강'})"
|
||||
)
|
||||
else:
|
||||
lines.append("- main 빨강 0회")
|
||||
lines.append("")
|
||||
lines += ["## runner 사용률 (위 repo 합, job 시작 날짜 기준)", "", "| 날짜 | 실행 합(분) | 사용률 |", "|---|---|---|"]
|
||||
for day in sorted(days):
|
||||
lines.append(f"| {day} | {days[day]:.0f} | {days[day] / 1440:.0%} |")
|
||||
if days:
|
||||
busiest = max(days, key=days.get)
|
||||
lines += ["", f"- 바쁜 날 {busiest}: {days[busiest] / 1440:.0%}"]
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("--server", default="https://git.learn.hyeonworks.com")
|
||||
parser.add_argument("--owner", default="donghyeon.kang")
|
||||
parser.add_argument("--repo", action="append", required=True, help="repeat for each repository")
|
||||
parser.add_argument("--since", required=True, type=date.fromisoformat, help="first local day, YYYY-MM-DD")
|
||||
parser.add_argument("--until", required=True, type=date.fromisoformat, help="last local day, inclusive")
|
||||
parser.add_argument("--utc-offset-hours", type=int, default=9, help="local day boundary (default KST)")
|
||||
parser.add_argument("--json", action="store_true", help="print the numbers as JSON instead of tables")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
offset = timedelta(hours=args.utc_offset_hours)
|
||||
since = datetime.combine(args.since, datetime.min.time(), timezone.utc) - offset
|
||||
until = datetime.combine(args.until + timedelta(days=1), datetime.min.time(), timezone.utc) - offset
|
||||
# A pull request merged in the window may have had runs in the week before it.
|
||||
lookback = since - timedelta(days=7)
|
||||
now = min(datetime.now(timezone.utc), until)
|
||||
|
||||
api = Gitea(args.server)
|
||||
per_repo: dict[str, dict] = {}
|
||||
window_runs: list[Run] = []
|
||||
for repo in args.repo:
|
||||
fetched = fetch_runs(api, args.owner, repo, lookback)
|
||||
in_window = [run for run in fetched if run.created is not None and since <= run.created < until]
|
||||
window_runs += in_window
|
||||
merged = fetch_merged_pulls(api, args.owner, repo, since, until)
|
||||
per_repo[repo] = {
|
||||
"stages": stage_rows(in_window),
|
||||
"runs_per_pull": runs_per_merged_pull(fetched, merged),
|
||||
"red": red_episodes(in_window, now),
|
||||
}
|
||||
days = utilisation(window_runs, offset)
|
||||
|
||||
window = f"{args.since} ~ {args.until}, UTC{args.utc_offset_hours:+d}"
|
||||
if args.json:
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"window": window,
|
||||
"repos": {
|
||||
repo: {
|
||||
"stages": data["stages"],
|
||||
"runs_per_pull": data["runs_per_pull"],
|
||||
"red": [episode.__dict__ for episode in data["red"]],
|
||||
}
|
||||
for repo, data in per_repo.items()
|
||||
},
|
||||
"utilisation_minutes": {str(day): round(value, 1) for day, value in sorted(days.items())},
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
else:
|
||||
print(report(window, per_repo, days), end="")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,124 @@
|
||||
"""Offline tests for ci_metrics: the arithmetic on run and job records, no server."""
|
||||
|
||||
import unittest
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import ci_metrics as m
|
||||
|
||||
T0 = datetime(2026, 11, 13, 0, 0, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def at(minutes: float) -> str:
|
||||
return (T0 + timedelta(minutes=minutes)).isoformat().replace("+00:00", "Z")
|
||||
|
||||
|
||||
def job(name, created, started=None, completed=None, conclusion="success"):
|
||||
return {
|
||||
"name": name,
|
||||
"created_at": at(created),
|
||||
"started_at": at(started) if started is not None else "1970-01-01T00:00:00Z",
|
||||
"completed_at": at(completed) if completed is not None else "1970-01-01T00:00:00Z",
|
||||
"conclusion": conclusion,
|
||||
}
|
||||
|
||||
|
||||
def run(run_id, path, event, conclusion, jobs, status="completed", pull=None):
|
||||
raw = {"id": run_id, "path": path, "event": event, "status": status, "conclusion": conclusion}
|
||||
if pull is not None:
|
||||
raw["pull_requests"] = [{"number": pull}]
|
||||
return m.run_from_api("backend", raw, jobs)
|
||||
|
||||
|
||||
class ParsingTest(unittest.TestCase):
|
||||
def test_a_zero_timestamp_from_a_job_that_has_not_started_is_absent(self):
|
||||
self.assertIsNone(m.parse_time("1970-01-01T00:00:00Z"))
|
||||
|
||||
def test_the_run_is_created_when_its_earliest_job_is(self):
|
||||
r = run(1, "pr.yml@refs/pull/7/head", "pull_request", "success", [job("done", 5, 20, 32)], pull=7)
|
||||
self.assertEqual((r.created, r.stage, r.pull_request), (T0 + timedelta(minutes=5), "pr.yml:pull_request", 7))
|
||||
|
||||
|
||||
class TimingTest(unittest.TestCase):
|
||||
def test_queue_wait_is_the_first_job_start_minus_run_creation_not_each_jobs_own_gap(self):
|
||||
# given: one runner runs the jobs one after another, so the second job's own gap is 40 minutes
|
||||
r = run(1, "main.yml@refs/heads/main", "push", "success", [job("a", 0, 10, 30), job("b", 0, 30, 40)])
|
||||
|
||||
# when
|
||||
waited = r.queue_minutes
|
||||
|
||||
# then
|
||||
self.assertEqual(waited, 10)
|
||||
|
||||
def test_execution_is_the_sum_of_job_run_times_and_a_skipped_job_adds_nothing(self):
|
||||
r = run(1, "main.yml@refs/heads/main", "push", "success",
|
||||
[job("a", 0, 10, 30), job("b", 0, 30, 40), job("c", 0, 40, 40, "skipped")])
|
||||
self.assertEqual(r.execution_minutes, 30)
|
||||
|
||||
def test_a_queued_run_has_no_wait_and_no_execution_yet(self):
|
||||
r = run(1, "pr.yml@refs/pull/7/head", "pull_request", None, [job("done", 0)], status="queued", pull=7)
|
||||
self.assertEqual((r.queue_minutes, r.execution_minutes), (None, 0.0))
|
||||
|
||||
def test_percentiles_use_the_nearest_rank(self):
|
||||
self.assertEqual((m.percentile([3, 1, 2, 10], 0.5), m.percentile([3, 1, 2, 10], 0.9)), (2, 10))
|
||||
|
||||
|
||||
class StageTest(unittest.TestCase):
|
||||
def test_a_cancelled_run_counts_as_a_run_but_not_in_execution_percentiles(self):
|
||||
# given
|
||||
runs = [
|
||||
run(1, "pr.yml@refs/pull/7/head", "pull_request", "success", [job("done", 0, 1, 13)], pull=7),
|
||||
run(2, "pr.yml@refs/pull/7/head", "pull_request", "cancelled", [job("done", 0, 1, 2)], pull=7),
|
||||
]
|
||||
|
||||
# when
|
||||
row = m.stage_rows(runs)[0]
|
||||
|
||||
# then
|
||||
self.assertEqual((row["runs"], row["finished"], row["exec_p50"], row["wait_p50"]), (2, 1, 12, 1))
|
||||
|
||||
def test_runs_per_merged_pull_request_count_every_attempt_of_that_pull_request(self):
|
||||
runs = [
|
||||
run(1, "pr.yml@refs/pull/7/head", "pull_request", "cancelled", [job("done", 0)], pull=7),
|
||||
run(2, "pr.yml@refs/pull/7/head", "pull_request", "success", [job("done", 1, 2, 14)], pull=7),
|
||||
run(3, "pr.yml@refs/pull/8/head", "pull_request", "success", [job("done", 1, 2, 14)], pull=8),
|
||||
]
|
||||
self.assertEqual(m.runs_per_merged_pull(runs, [7, 8]), [2, 1])
|
||||
|
||||
|
||||
class RedTest(unittest.TestCase):
|
||||
def test_main_is_red_from_the_first_failing_runs_creation_to_the_next_green_runs_end(self):
|
||||
# given: two failures in a row, then a green run that ends at minute 210
|
||||
runs = [
|
||||
run(1, "main.yml@refs/heads/main", "push", "failure", [job("main", 0, 5, 30, "failure")]),
|
||||
run(2, "main.yml@refs/heads/main", "push", "failure", [job("main", 60, 65, 95, "failure")]),
|
||||
run(3, "main.yml@refs/heads/main", "push", "success", [job("main", 120, 125, 210)]),
|
||||
run(4, "pr.yml@refs/pull/7/head", "pull_request", "failure", [job("done", 0, 1, 2, "failure")], pull=7),
|
||||
]
|
||||
|
||||
# when
|
||||
episodes = m.red_episodes(runs, now=T0 + timedelta(hours=10))
|
||||
|
||||
# then
|
||||
self.assertEqual([(e.first_failed_run, e.fixed_by_run, e.hours) for e in episodes], [(1, 3, 3.5)])
|
||||
|
||||
def test_main_still_red_at_the_end_of_the_window_counts_up_to_now(self):
|
||||
runs = [run(1, "main.yml@refs/heads/main", "push", "failure", [job("main", 0, 5, 30, "failure")])]
|
||||
episodes = m.red_episodes(runs, now=T0 + timedelta(hours=2))
|
||||
self.assertEqual([(e.fixed_by_run, e.hours) for e in episodes], [(None, 2.0)])
|
||||
|
||||
|
||||
class UtilisationTest(unittest.TestCase):
|
||||
def test_job_minutes_count_on_the_local_day_the_job_started(self):
|
||||
# given: 14:30 UTC is 23:30 KST on the 13th, 15:30 UTC is 00:30 KST on the 14th
|
||||
r = run(1, "main.yml@refs/heads/main", "push", "success",
|
||||
[job("a", 0, 14 * 60 + 30, 14 * 60 + 50), job("b", 0, 15 * 60 + 30, 15 * 60 + 40)])
|
||||
|
||||
# when
|
||||
days = m.utilisation([r], timedelta(hours=9))
|
||||
|
||||
# then
|
||||
self.assertEqual({str(day): value for day, value in days.items()}, {"2026-11-13": 20, "2026-11-14": 10})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -14,6 +14,11 @@ Kubernetes 리소스를 직접 참조하며 업스트림 매니페스트를 복
|
||||
- 저장소: <https://github.com/keycloak/keycloak-k8s-resources>
|
||||
- 리소스 경로: `kubernetes`
|
||||
- Keycloak 및 Operator 버전: `26.7.0`
|
||||
- 원격 ref: 태그가 아니라 그 태그의 커밋 `38fd7a0c84e15d4b66b37d1c2ea728ca2f5416a4`(2026-11-12). 같은 태그의 내용이
|
||||
한 번 바뀌었다(build-timestamp 07-09 → 07-23, 성숙도 점검 A5).
|
||||
- 이미지: Operator·Keycloak 모두 2026-11-12에 돌던 digest로 고정, Operator `imagePullPolicy: IfNotPresent`.
|
||||
Keycloak CR의 `spec.image`를 정하면 Operator가 최적화 빌드된 이미지로 보므로 `startOptimized: false`를 같이 둔다.
|
||||
올릴 때는 태그·커밋·두 digest와 `scripts/validate/render-phase2.sh`·`scripts/bootstrap/apply-keycloak.sh`의 검사 문자열을 함께 바꾼다.
|
||||
- 설치 모드: `keycloak` namespace 범위
|
||||
|
||||
업스트림 namespace 범위 Deployment는 각 컨트롤러의 감시 namespace를
|
||||
|
||||
@@ -3,8 +3,38 @@ kind: Kustomization
|
||||
|
||||
namespace: keycloak
|
||||
|
||||
# keycloak-k8s-resources tag 26.7.0 pointed at this commit on 2026-11-12. The tag
|
||||
# content changed once (live build-timestamp 2026-07-09, later renders 2026-07-23),
|
||||
# so the commit is pinned instead of the tag.
|
||||
resources:
|
||||
- github.com/keycloak/keycloak-k8s-resources/kubernetes?ref=26.7.0
|
||||
- github.com/keycloak/keycloak-k8s-resources/kubernetes?ref=38fd7a0c84e15d4b66b37d1c2ea728ca2f5416a4
|
||||
|
||||
# The 26.7.0 images that were running on 2026-11-12 (pod imageID), by digest.
|
||||
images:
|
||||
- name: quay.io/keycloak/keycloak-operator
|
||||
digest: sha256:257ee324da2c701b7fc1ed1b2ce0e03979d1bf149952383bdcf6be904ae5b4f3
|
||||
|
||||
patches:
|
||||
# Merged by container and env name (strategic merge), not by list index.
|
||||
- target:
|
||||
group: apps
|
||||
version: v1
|
||||
kind: Deployment
|
||||
name: keycloak-operator
|
||||
patch: |-
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: keycloak-operator
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: keycloak-operator
|
||||
imagePullPolicy: IfNotPresent
|
||||
env:
|
||||
- name: RELATED_IMAGE_KEYCLOAK
|
||||
value: quay.io/keycloak/keycloak@sha256:0f198be292568439d700cdbfb893e69a6009bb43a94a06a945b1d3d506c76b13
|
||||
|
||||
labels:
|
||||
- pairs:
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
# desktop 호스트 설정
|
||||
|
||||
desktop 한 대(k3s 노드 `donghyeon-system-product-name`)는 운영 k3s, Gitea
|
||||
Actions runner, 에이전트 세션의 Docker 게이트를 같이 돌린다. 루트 SSD 하나와
|
||||
시계 하나가 무너지면 운영까지 같이 내려간다. 이 폴더는 그 두 가지를 막는 호스트
|
||||
설정 조각이다. Kubernetes 리소스가 아니므로 `kustomization.yaml`이 없고, root
|
||||
권한으로 아래 경로에 설치한다.
|
||||
|
||||
| 파일 | 설치 경로 | 하는 일 |
|
||||
| --- | --- | --- |
|
||||
| `docker/daemon.json` | `/etc/docker/daemon.json` | 호스트 Docker 빌드 캐시 상한 |
|
||||
| `systemd/k3s.service.d/10-after-time-sync.conf` | `/etc/systemd/system/k3s.service.d/` | k3s를 시간 동기화 뒤에 시작 |
|
||||
| `systemd/systemd-time-wait-sync.service.d/10-timeout.conf` | `/etc/systemd/system/systemd-time-wait-sync.service.d/` | 동기화를 10분까지만 기다림 |
|
||||
|
||||
runner 안쪽 Docker 상한과 주 1회 정리는
|
||||
[`../ci/gitea-runner/README.md`](../ci/gitea-runner/README.md)의 「디스크와 정리」에 있다.
|
||||
|
||||
## 왜 바꾸나
|
||||
|
||||
### 디스크 (2026-11-09 사고)
|
||||
|
||||
- 11-07~08 이틀 동안 에이전트 게이트의 Docker 빌드 캐시가 약 23G 쌓였다. 상한이 없었다.
|
||||
- 루트 사용률 85%에서 kubelet이 안 쓰는 이미지를 지우기 시작했고, 90%(여유 10% 미만)에서
|
||||
파드를 내보냈다. 내보내진 tech-log 파드의 이미지가 「안 쓰는 이미지」가 되어 지워졌고,
|
||||
`imagePullPolicy: Never`라 다시 받지 못했다.
|
||||
- 디스크 경보(85% 경고)는 사고 전 33시간 동안 울렸지만, 이미 kubelet 이미지 GC가 시작되는
|
||||
시점이었다.
|
||||
|
||||
### BuildKit 상한 값
|
||||
|
||||
Docker 29에서 `builder.gc` 키의 뜻은 다음과 같다(moby `daemon/config/builder.go`).
|
||||
|
||||
| 키 | 뜻 | 호스트 값 |
|
||||
| --- | --- | --- |
|
||||
| `defaultReservedSpace` | 이 양 아래로는 지우지 않는다 | 5GB |
|
||||
| `defaultMaxUsedSpace` | 캐시가 이 양을 넘으면 지운다(상한) | 15GB |
|
||||
| `defaultMinFreeSpace` | 디스크 여유가 이 양 아래면 `reservedSpace`까지 지운다 | 35GB |
|
||||
|
||||
- 예전 키 `defaultKeepStorage`는 이제 `defaultReservedSpace`와 같은 뜻이다. 상한이 아니라
|
||||
「최소 보존량」이라 쓰지 않는다.
|
||||
- 설정 전 기본 정책은 `docker buildx inspect default`에서 Reserved 21.42GiB,
|
||||
Max Used 169.5GiB, Min Free 42.84GiB였다(2026-11-11). 캐시가 21GB까지는 전혀 지워지지
|
||||
않았다.
|
||||
- `35GB`는 1024 단위다. 루트 228GiB에서 여유 35GiB는 사용률 약 85%, 즉 kubelet이 이미지를
|
||||
지우기 시작하는 지점이다. 그 전에 빌드 캐시부터 줄인다.
|
||||
- Docker 저장소를 HDD(sdb)로 옮기는 안은 빌드가 느려져 하지 않는다(소유자 결정, 2026-11-11).
|
||||
|
||||
### 시계 (2026-11-11 사고)
|
||||
|
||||
- desktop은 Windows와 이중 부팅한다(`efibootmgr`: Windows Boot Manager, Ubuntu).
|
||||
- Linux는 RTC를 UTC로 읽는다(`timedatectl`: `RTC in local TZ: no`). Windows는 기본값으로
|
||||
RTC에 현지 시각(KST)을 쓴다. 그래서 Windows를 쓴 뒤 Linux로 부팅하면 시계가 9시간 앞선다.
|
||||
- 11-11 부팅이 그 상태로 시작했다(`journalctl --list-boots`의 시작 시각 11-12 04:33 KST).
|
||||
k3s가 그 시각으로 인증서를 만든 뒤 timesyncd가 시계를 되돌렸고, 인증서가 「not yet valid」가
|
||||
되어 전부 502가 됐다. `sudo systemctl restart k3s`로 복구했다.
|
||||
- Prometheus도 그 사이 9시간 앞선 시각의 샘플을 저장했다. 시계가 돌아온 뒤 새 샘플이 전부
|
||||
「out of bounds」로 버려져, 미래 시각을 따라잡을 때까지 지표와 경보가 멈춘다.
|
||||
|
||||
고치는 방법은 둘이다.
|
||||
|
||||
1. **k3s가 시간 동기화 뒤에 시작한다.** `systemd-time-wait-sync.service`는 커널 시계가
|
||||
NTP로 맞춰질 때까지 `time-sync.target`을 막는다. k3s는 `After=time-sync.target`으로 그
|
||||
뒤에 시작한다. 이 서비스는 기본으로 꺼져 있고 끝없이 기다리므로, 켜고 10분 상한을 둔다.
|
||||
10분 안에 NTP에 닿지 못하면 k3s는 지금처럼 그냥 시작한다.
|
||||
2. **Windows도 RTC를 UTC로 쓰게 한다.** 그러면 9시간 차이가 처음부터 생기지 않는다.
|
||||
|
||||
## 소유자가 실행할 명령
|
||||
|
||||
sudo가 필요해 에이전트 세션은 실행하지 않는다. platform-core checkout(이 PR이 병합된
|
||||
main)에서 위에서부터 순서대로 실행한다. 단계마다 확인 명령과 되돌리는 법이 있다.
|
||||
|
||||
### 1. 운영 이미지를 kubelet 이미지 GC에서 지킨다 (재시작 없음, 바로)
|
||||
|
||||
tech-log 이미지를 Gitea 레지스트리로 옮기기 전까지만 쓴다. pinned 라벨이 있는 이미지는
|
||||
kubelet이 지우지 않는다.
|
||||
|
||||
```bash
|
||||
sudo k3s ctr -n k8s.io images label docker.io/library/tech-log-backend:46b1cc7 io.cri-containerd.pinned=pinned
|
||||
sudo k3s ctr -n k8s.io images label docker.io/library/tech-log-frontend:7a4e201 io.cri-containerd.pinned=pinned
|
||||
sudo k3s ctr -n k8s.io images ls | grep -E 'tech-log-(backend|frontend)'
|
||||
```
|
||||
|
||||
마지막 줄의 LABELS 칸에 `io.cri-containerd.pinned=pinned`가 있어야 한다.
|
||||
배포 이미지가 바뀌면 새 tag에 다시 붙인다.
|
||||
|
||||
되돌리기: 라벨 값을 비운다.
|
||||
|
||||
```bash
|
||||
sudo k3s ctr -n k8s.io images label docker.io/library/tech-log-backend:46b1cc7 io.cri-containerd.pinned=
|
||||
sudo k3s ctr -n k8s.io images label docker.io/library/tech-log-frontend:7a4e201 io.cri-containerd.pinned=
|
||||
```
|
||||
|
||||
### 2. k3s를 시간 동기화 뒤에 시작한다 (재시작 없음, 다음 부팅부터)
|
||||
|
||||
k3s 재시작 창을 열 때는 `scripts/bootstrap/apply-k3s-restart-bundle.sh`(`infrastructure/host/k3s`의
|
||||
kubelet drop-in과 함께)가 아래 두 파일을 그대로 설치하고 `systemd-time-wait-sync`를 켠다. 그 묶음을
|
||||
쓰면 이 단계는 건너뛴다. 묶음 없이 이것만 할 때는 아래 명령을 쓴다.
|
||||
|
||||
```bash
|
||||
sudo install -D -m 0644 infrastructure/host/systemd/k3s.service.d/10-after-time-sync.conf \
|
||||
/etc/systemd/system/k3s.service.d/10-after-time-sync.conf
|
||||
sudo install -D -m 0644 infrastructure/host/systemd/systemd-time-wait-sync.service.d/10-timeout.conf \
|
||||
/etc/systemd/system/systemd-time-wait-sync.service.d/10-timeout.conf
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable systemd-time-wait-sync.service
|
||||
```
|
||||
|
||||
확인:
|
||||
|
||||
```bash
|
||||
systemctl show k3s -p After -p Wants | grep -o time-sync.target
|
||||
systemctl is-enabled systemd-time-wait-sync.service
|
||||
systemctl show systemd-time-wait-sync.service -p TimeoutStartUSec
|
||||
```
|
||||
|
||||
`time-sync.target` 두 줄, `enabled`, `TimeoutStartUSec=10min`이 나와야 한다. 순서는 부팅할
|
||||
때만 쓰이므로 지금 k3s를 다시 시작하지 않는다. 다음 재부팅 뒤
|
||||
`journalctl -b -u systemd-time-wait-sync -u k3s | head`에서 동기화가 k3s 시작보다 앞서는지 본다.
|
||||
|
||||
되돌리기:
|
||||
|
||||
```bash
|
||||
sudo systemctl disable systemd-time-wait-sync.service
|
||||
sudo rm /etc/systemd/system/k3s.service.d/10-after-time-sync.conf \
|
||||
/etc/systemd/system/systemd-time-wait-sync.service.d/10-timeout.conf
|
||||
sudo systemctl daemon-reload
|
||||
```
|
||||
|
||||
### 3. RTC를 UTC로 맞춘다
|
||||
|
||||
Linux 쪽은 이미 UTC다. 확인만 한다.
|
||||
|
||||
```bash
|
||||
timedatectl | grep -e 'RTC in local TZ' -e 'System clock synchronized'
|
||||
```
|
||||
|
||||
`RTC in local TZ: no`가 아니면 `sudo timedatectl set-local-rtc 0`을 실행한다.
|
||||
|
||||
Windows 쪽(다음에 Windows로 부팅했을 때, 관리자 명령 프롬프트):
|
||||
|
||||
```bat
|
||||
reg add "HKLM\SYSTEM\CurrentControlSet\Control\TimeZoneInformation" /v RealTimeIsUniversal /t REG_DWORD /d 1 /f
|
||||
```
|
||||
|
||||
그 뒤 Windows 설정 → 시간 및 언어 → 날짜 및 시간 → 「지금 동기화」를 누른다. Windows가
|
||||
RTC를 UTC로 다시 쓴다. Linux로 돌아와 `timedatectl`의 `RTC time`이 `Universal time`과
|
||||
같은지 본다.
|
||||
|
||||
되돌리기(Windows): `reg delete "HKLM\SYSTEM\CurrentControlSet\Control\TimeZoneInformation" /v RealTimeIsUniversal /f`
|
||||
|
||||
### 4. 호스트 Docker 빌드 캐시 상한 (Docker 재시작 필요)
|
||||
|
||||
`systemctl restart docker`는 호스트 Docker의 컨테이너를 모두 다시 시작한다. 운영 k3s는
|
||||
자기 containerd를 쓰므로 영향이 없다. 영향을 받는 것:
|
||||
|
||||
| 컨테이너 | 영향 |
|
||||
| --- | --- |
|
||||
| `gitea-runner` | 실행 중인 CI job이 실패한다. `restart: always`로 다시 뜬다 |
|
||||
| `workmachine` | cokacremote 원격 작업이 끊긴다. `unless-stopped`로 다시 뜬다 |
|
||||
| `chatgpt-cloudflared` | 터널이 잠깐 끊긴다. `unless-stopped`로 다시 뜬다 |
|
||||
| 에이전트 세션 게이트 | 돌고 있던 Testcontainers·이미지 빌드가 실패한다 |
|
||||
|
||||
**안전한 때:** 아래 세 가지가 모두 참일 때.
|
||||
|
||||
```bash
|
||||
# 1) CI job이 없다: job 컨테이너가 0개
|
||||
docker exec gitea-runner docker ps --format '{{.Names}}' | wc -l
|
||||
# 2) 에이전트 세션의 무거운 게이트가 없다: lock을 바로 잡을 수 있다
|
||||
flock -n /tmp/claude-gates/heavy.lock true && echo free
|
||||
# 3) workmachine을 지금 쓰고 있지 않다 (소유자 판단)
|
||||
```
|
||||
|
||||
적용은 lock을 잡은 채로 한다. 재시작하는 동안 새 게이트가 시작되지 않는다.
|
||||
|
||||
```bash
|
||||
sudo install -m 0644 infrastructure/host/docker/daemon.json /etc/docker/daemon.json
|
||||
sudo dockerd --validate --config-file /etc/docker/daemon.json
|
||||
flock /tmp/claude-gates/heavy.lock sudo systemctl restart docker
|
||||
docker ps --format '{{.Names}} {{.Status}}'
|
||||
docker buildx inspect default | sed -n '/GC Policy rule#1/,/Min Free/p'
|
||||
```
|
||||
|
||||
`dockerd --validate`는 맨 위 키만 검사하고 `builder.gc` 안쪽 키 오타는 잡지 못한다. 그래서
|
||||
마지막 줄로 실제 정책을 본다. `Reserved Space: 5GiB`, `Max Used Space: 15GiB`,
|
||||
`Min Free Space: 35GiB`가 나와야 한다. 세 컨테이너가 `Up`이어야 한다.
|
||||
|
||||
되돌리기: `sudo rm /etc/docker/daemon.json` 뒤 같은 조건에서 Docker를 다시 시작한다(설치 전에는
|
||||
이 파일이 없었다).
|
||||
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=Back up Gitea (gitea dump --skip-db) to AIStor (platform-host-backups)
|
||||
Documentation=file:///usr/local/lib/platform-backup/README.md
|
||||
After=k3s.service docker.service
|
||||
Requires=docker.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/bin/bash /usr/local/lib/platform-backup/backup/gitea-backup.sh
|
||||
Nice=10
|
||||
IOSchedulingClass=idle
|
||||
TimeoutStartSec=1h
|
||||
@@ -0,0 +1,10 @@
|
||||
[Unit]
|
||||
Description=Weekly Gitea backup
|
||||
|
||||
[Timer]
|
||||
OnCalendar=Sun *-*-* 04:00:00
|
||||
RandomizedDelaySec=10min
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,8 @@
|
||||
[Unit]
|
||||
Description=Record the k3s API server and kubelet serving certificate expiry
|
||||
After=k3s.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=donghyeon
|
||||
ExecStart=/bin/bash /usr/local/lib/platform-backup/validate/k3s-cert-expiry.sh
|
||||
@@ -0,0 +1,9 @@
|
||||
[Unit]
|
||||
Description=Daily k3s certificate expiry check
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 05:00:00
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,11 @@
|
||||
[Unit]
|
||||
Description=Back up the k3s datastore and token to AIStor (platform-host-backups)
|
||||
Documentation=file:///usr/local/lib/platform-backup/README.md
|
||||
After=k3s.service docker.service
|
||||
Requires=docker.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/bin/bash /usr/local/lib/platform-backup/backup/k3s-datastore-backup.sh
|
||||
Nice=10
|
||||
IOSchedulingClass=idle
|
||||
@@ -0,0 +1,10 @@
|
||||
[Unit]
|
||||
Description=Daily k3s datastore backup
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 03:30:00
|
||||
RandomizedDelaySec=10min
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,16 @@
|
||||
[Unit]
|
||||
Description=Compare the cluster with platform-core origin/main (kubectl diff, server dry-run)
|
||||
After=k3s.service network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=donghyeon
|
||||
Environment=PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm
|
||||
Environment=PLATFORM_CHECKOUT=/home/donghyeon/workspace/platform
|
||||
ExecStart=/bin/bash /usr/local/lib/platform-backup/validate/live-drift-check-main.sh
|
||||
# Exit 1 means "drift found"; the metric and its alert report that. Only 2 (the
|
||||
# check itself failed) marks the unit failed.
|
||||
SuccessExitStatus=1
|
||||
Nice=10
|
||||
TimeoutStartSec=30min
|
||||
@@ -0,0 +1,10 @@
|
||||
[Unit]
|
||||
Description=Daily live drift check
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 05:15:00
|
||||
RandomizedDelaySec=10min
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"builder": {
|
||||
"gc": {
|
||||
"enabled": true,
|
||||
"defaultReservedSpace": "5GB",
|
||||
"defaultMaxUsedSpace": "15GB",
|
||||
"defaultMinFreeSpace": "35GB"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
# k3s 재시작 묶음
|
||||
|
||||
k3s를 **한 번만** 재시작해서 다음을 같이 넣는다. 근거는 `docs/reviews/2026-11-11-k3s-maturity-review.md`
|
||||
A3·A9·A10·7.2·7.3·7.5절, 결정은 같은 문서 8절 결정 5(2026-11-12 총괄 확정)다.
|
||||
|
||||
| 파일 | 설치 위치 | 하는 일 |
|
||||
| --- | --- | --- |
|
||||
| `kubelet.conf.d/50-platform-resources.conf` | `/var/lib/rancher/k3s/agent/etc/kubelet.conf.d/` | 예약(system 1 CPU·2Gi, kube 0.5 CPU·1Gi), 메모리 축출(hard 500Mi, soft 1Gi/1분), graceful node shutdown(60초, critical 20초) |
|
||||
| `config.yaml.d/60-platform-apiserver.yaml` | `/etc/rancher/k3s/config.yaml.d/` | API 서버 감사 로그와 PodSecurity admission 설정 파일 지정 |
|
||||
| `server/audit.yaml` | `/var/lib/rancher/k3s/server/audit.yaml` | 감사 정책: Metadata 수준(본문을 남기지 않아 Secret 값이 로그에 없다), 잦고 값 없는 요청은 뺌 |
|
||||
| `server/psa.yaml` | `/var/lib/rancher/k3s/server/psa.yaml` | 라벨 없는 namespace의 기본값: enforce `baseline`, warn·audit `restricted`, `kube-system` 예외 |
|
||||
| (W1) `../systemd/k3s.service.d/10-after-time-sync.conf` | `/etc/systemd/system/k3s.service.d/` | k3s를 시간 동기화 뒤에 시작 |
|
||||
| (W1) `../systemd/systemd-time-wait-sync.service.d/10-timeout.conf` | `/etc/systemd/system/systemd-time-wait-sync.service.d/` | 동기화를 10분까지만 기다림 |
|
||||
|
||||
그리고 `/var/lib/rancher/k3s/server/manifests/local-storage.yaml.skip`을 만들어 k3s가 local-path
|
||||
addon을 다시 덮어쓰지 않게 한 뒤, `local-path` StorageClass의 기본 표시를 끈다. 쓰는 PVC가 없고
|
||||
(2026-11-11), 기본으로 남아 있으면 `storageClassName`을 빠뜨린 PVC가 루트 SSD에 Delete 정책으로 생긴다.
|
||||
k3s 문서의 `.skip` 방식이다(packaged components). 그 뒤 local-path-provisioner는 k3s 업그레이드로
|
||||
갱신되지 않는다.
|
||||
|
||||
## 값을 이렇게 정한 이유
|
||||
|
||||
- 예약: 노드 32GiB, 12 CPU. 파드 requests 합이 메모리 11Gi(34%)라 예약 3Gi를 빼도 여유가 크다.
|
||||
예약이 없으면 Allocatable = Capacity라 파드가 노드 전부를 잡을 수 있고, 같은 호스트의 Docker
|
||||
빌드·CI·데스크톱 앱과 겹치면 커널 OOM killer가 고른다.
|
||||
- `evictionHard`: 메모리·디스크 셋을 모두 적는다. 이 맵을 정하면 정하지 않은 항목의 기본값이 0이 된다
|
||||
(`mergeDefaultEvictionSettings: false`). 디스크 5%는 k3s 기본값 그대로다.
|
||||
- graceful shutdown: 두 값이 0이면 기능이 꺼진다. 재부팅 때 Postgres 같은 파드가 정상 종료할 시간을
|
||||
준다. 호스트 종료가 최대 60초 늦어진다. kubelet이 logind `InhibitDelayMaxSec`(지금 30초)가 모자라면
|
||||
스스로 늘린다. 적용 뒤 `systemd-analyze cat-config systemd/logind.conf`로 확인한다.
|
||||
- PSA 기본값은 `restricted`가 아니라 `baseline`이다. 라벨 없는 namespace는 `default`·`kube-public`·
|
||||
`kube-node-lease`뿐이고 파드가 없다. 플랫폼 namespace는 모두 자기 라벨이 있어 영향이 없다.
|
||||
|
||||
## 넣지 않은 것: `protect-kernel-defaults`
|
||||
|
||||
7.3절에서 권했지만 이 묶음에서 뺐다.
|
||||
|
||||
- CIS Kubernetes Benchmark는 v1.7부터 이 항목(옛 4.2.6)을 뺐다.
|
||||
- 켜려면 kubelet이 기대하는 sysctl을 먼저 맞춰야 한다(k3s hardening 가이드: `vm.overcommit_memory=1`,
|
||||
`kernel.panic=10`, `kernel.panic_on_oops=1`, `vm.panic_on_oom=0`). `kernel.panic_on_oops=1`은 커널
|
||||
oops가 한 번만 나도 10초 뒤 호스트를 재부팅한다. 이 기계는 GPU 드라이버가 있는 데스크톱이고
|
||||
이중 부팅이며, 재부팅 자체가 11-11 시계 사고의 출발점이었다. `vm.overcommit_memory=1`은 호스트 전체의
|
||||
메모리 할당 방식을 바꾼다.
|
||||
- 얻는 것(kubelet이 커널 값을 못 바꾸게 함)보다 잃는 것이 커서 뺀다. 다시 넣으려면 sysctl 변경을 따로
|
||||
소유자가 정한다.
|
||||
|
||||
## 적용 (소유자, 한 번)
|
||||
|
||||
전제:
|
||||
|
||||
- desktop, 이 디렉터리와 W1의 `infrastructure/host/systemd/*`가 **둘 다** 있는 platform-core `main` checkout
|
||||
(W1 PR 병합 뒤). 없으면 스크립트가 `MISSING …(W1)`을 찍고 멈춘다.
|
||||
- CI job과 에이전트 게이트가 없는 때. k3s 재시작 중 1~2분 API가 멈춘다. k3s는 `KillMode=process`라
|
||||
컨테이너는 계속 돈다(파드가 재시작되지는 않는다).
|
||||
|
||||
```bash
|
||||
cd /home/donghyeon/workspace/platform && git pull --ff-only
|
||||
bash scripts/bootstrap/apply-k3s-restart-bundle.sh --check
|
||||
sudo bash scripts/bootstrap/apply-k3s-restart-bundle.sh --execute
|
||||
```
|
||||
|
||||
`--execute`가 하는 일:
|
||||
|
||||
1. 클러스터 Ready와 health 셋(Keycloak OIDC discovery, tech-log health, Gitea healthz)을 본다.
|
||||
2. 바꿀 파일을 `/var/backups/platform-k3s-restart-bundle/<시각>/`에 복사하고, datastore를
|
||||
`sqlite3 .backup`으로 복사한다(`token`도 같이, 모드 0600, `PRAGMA integrity_check` 출력).
|
||||
3. 파일을 설치하고 `systemd-time-wait-sync`를 켠 뒤 k3s를 한 번 재시작한다.
|
||||
4. 확인: kubelet configz(예약·eviction·shutdown 값), 감사 로그가 쓰이는지, `default` namespace에서
|
||||
privileged 파드가 서버 dry-run으로 거절되는지, health 셋이 200인지, time-wait-sync가 enabled인지.
|
||||
5. 3·4 중 하나라도 실패하면 백업한 파일을 되돌리고 새 파일을 지운 뒤 k3s를 다시 재시작한다.
|
||||
6. 다 통과하면 `local-path`의 기본 표시를 끈다.
|
||||
|
||||
마지막 줄이 `K3S RESTART BUNDLE APPLIED`여야 한다.
|
||||
|
||||
## 같은 재시작 창에 이어서 할 것
|
||||
|
||||
PriorityClass를 파드에 붙이는 변경과 Keycloak 이미지 고정은 파드 재시작을 부르므로 이 묶음 바로 뒤에
|
||||
한다(`docs/runbooks/` 재시작 창 절차). Postgres의 `priorityClassName`·Guaranteed는 W1 범위다.
|
||||
|
||||
## 되돌리기 (수동)
|
||||
|
||||
```bash
|
||||
B=/var/backups/platform-k3s-restart-bundle/<시각>
|
||||
sudo rm -f /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/50-platform-resources.conf \
|
||||
/etc/rancher/k3s/config.yaml.d/60-platform-apiserver.yaml \
|
||||
/var/lib/rancher/k3s/server/manifests/local-storage.yaml.skip
|
||||
sudo systemctl daemon-reload && sudo systemctl restart k3s
|
||||
kubectl annotate storageclass local-path storageclass.kubernetes.io/is-default-class=true --overwrite
|
||||
```
|
||||
|
||||
`audit.yaml`·`psa.yaml`는 참조가 없어지면 쓰이지 않으므로 남겨도 된다. W1 시간 동기화 되돌리기는
|
||||
`infrastructure/host/README.md` 2단계를 따른다.
|
||||
@@ -0,0 +1,12 @@
|
||||
# Installed as /etc/rancher/k3s/config.yaml.d/60-platform-apiserver.yaml.
|
||||
# "+" appends to kube-apiserver-arg instead of replacing it (docs.k3s.io,
|
||||
# installation/configuration, multiple config files).
|
||||
# Audit log: k3s hardening guide "API Server audit configuration".
|
||||
# PodSecurity default: k3s hardening guide "Pod Security" (baseline here, see README).
|
||||
kube-apiserver-arg+:
|
||||
- "audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log"
|
||||
- "audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml"
|
||||
- "audit-log-maxage=30"
|
||||
- "audit-log-maxbackup=10"
|
||||
- "audit-log-maxsize=100"
|
||||
- "admission-control-config-file=/var/lib/rancher/k3s/server/psa.yaml"
|
||||
@@ -0,0 +1,29 @@
|
||||
# Installed as /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/50-platform-resources.conf.
|
||||
# k3s starts the kubelet with --config-dir on that directory and keeps its own
|
||||
# defaults in 00-k3s-defaults.conf; this file is merged after it.
|
||||
# Why: docs/reviews/2026-11-11-k3s-maturity-review.md A3 and 7.2.
|
||||
apiVersion: kubelet.config.k8s.io/v1beta1
|
||||
kind: KubeletConfiguration
|
||||
# Keep memory and CPU for the host (desktop, Docker builds, CI runner) and for
|
||||
# k3s itself, so pods cannot take the whole node.
|
||||
systemReserved:
|
||||
cpu: "1"
|
||||
memory: 2Gi
|
||||
kubeReserved:
|
||||
cpu: 500m
|
||||
memory: 1Gi
|
||||
# All three keys are listed on purpose: setting evictionHard replaces the
|
||||
# defaults, and k3s's own default only had the two disk keys.
|
||||
evictionHard:
|
||||
memory.available: 500Mi
|
||||
nodefs.available: "5%"
|
||||
imagefs.available: "5%"
|
||||
evictionSoft:
|
||||
memory.available: 1Gi
|
||||
evictionSoftGracePeriod:
|
||||
memory.available: 1m
|
||||
evictionMaxPodGracePeriod: 60
|
||||
# Graceful node shutdown: on reboot or power-off the kubelet stops normal pods
|
||||
# first (40s), then critical pods (20s), instead of leaving it to systemd.
|
||||
shutdownGracePeriod: 60s
|
||||
shutdownGracePeriodCriticalPods: 20s
|
||||
@@ -0,0 +1,28 @@
|
||||
# Installed as /var/lib/rancher/k3s/server/audit.yaml (mode 0600).
|
||||
# Metadata level only: who did what to which object, never request or
|
||||
# response bodies, so Secret values are not written to the log.
|
||||
apiVersion: audit.k8s.io/v1
|
||||
kind: Policy
|
||||
omitStages:
|
||||
- RequestReceived
|
||||
rules:
|
||||
# High-volume, low-value traffic.
|
||||
- level: None
|
||||
resources:
|
||||
- group: coordination.k8s.io
|
||||
resources: ["leases"]
|
||||
- level: None
|
||||
resources:
|
||||
- group: ""
|
||||
resources: ["events"]
|
||||
- level: None
|
||||
nonResourceURLs:
|
||||
- /healthz*
|
||||
- /livez*
|
||||
- /readyz*
|
||||
- /version
|
||||
- /metrics
|
||||
- level: None
|
||||
users: ["system:kube-proxy"]
|
||||
verbs: ["watch"]
|
||||
- level: Metadata
|
||||
@@ -0,0 +1,22 @@
|
||||
# Installed as /var/lib/rancher/k3s/server/psa.yaml (mode 0600).
|
||||
# Cluster-wide Pod Security default for namespaces without their own labels.
|
||||
# Namespace labels still win: every platform namespace already sets its level.
|
||||
apiVersion: apiserver.config.k8s.io/v1
|
||||
kind: AdmissionConfiguration
|
||||
plugins:
|
||||
- name: PodSecurity
|
||||
configuration:
|
||||
apiVersion: pod-security.admission.config.k8s.io/v1
|
||||
kind: PodSecurityConfiguration
|
||||
defaults:
|
||||
enforce: "baseline"
|
||||
enforce-version: "latest"
|
||||
audit: "restricted"
|
||||
audit-version: "latest"
|
||||
warn: "restricted"
|
||||
warn-version: "latest"
|
||||
exemptions:
|
||||
usernames: []
|
||||
runtimeClasses: []
|
||||
# k3s helm-install jobs and local-path helper pods run here.
|
||||
namespaces: [kube-system]
|
||||
@@ -0,0 +1,9 @@
|
||||
# Start k3s only after the system clock is synchronized.
|
||||
# k3s writes certificates with the current time at start. If the clock is
|
||||
# wrong at boot (RTC read 9 hours ahead on 2026-11-11) and NTP corrects it
|
||||
# later, those certificates are "not yet valid" and every Ingress returns 502.
|
||||
# time-sync.target is reached only when systemd-time-wait-sync.service is
|
||||
# enabled. See infrastructure/host/README.md.
|
||||
[Unit]
|
||||
Wants=time-sync.target
|
||||
After=time-sync.target
|
||||
@@ -0,0 +1,5 @@
|
||||
# The packaged unit waits forever (TimeoutStartSec=infinity). If NTP is not
|
||||
# reachable at boot, k3s would never start. After 10 minutes this unit fails,
|
||||
# time-sync.target is reached anyway, and k3s starts with the clock it has.
|
||||
[Service]
|
||||
TimeoutStartSec=10min
|
||||
@@ -94,6 +94,7 @@ infrastructure/networking/traefik/
|
||||
├── apply-observe.sh
|
||||
├── observe-client-host.sh
|
||||
├── apply-trust.sh
|
||||
├── reapply-trust.sh
|
||||
├── rollback-to-observe.sh
|
||||
└── validate.sh
|
||||
```
|
||||
@@ -238,6 +239,20 @@ bash infrastructure/networking/traefik/scripts/observe-client-host.sh
|
||||
`apply-trust.sh`를 실행한다. Chart, 이미지, context, API server, NodePort 경계 또는
|
||||
재관측 값이 기대와 다르면 스크립트가 적용을 중단한다.
|
||||
|
||||
이미 trust인 상태에서 trust overlay의 다른 값(예: `resources`)만 바꿨을 때는 `apply-trust.sh`가
|
||||
「observation phase must not trust forwarded headers」로 멈춘다(observe → trust 전환 전용이다). 그때는
|
||||
`reapply-trust.sh`를 쓴다.
|
||||
|
||||
```sh
|
||||
bash infrastructure/networking/traefik/scripts/reapply-trust.sh --observed-client-host 10.42.0.1 --execute
|
||||
# 승인: APPLY <현재-context> RETRUST 10.42.0.1/32
|
||||
```
|
||||
|
||||
- 적용 전에 runtime이 trust인지, 전달 헤더 신뢰가 실제로 작동하는지(Host Nginx 경유 요청의 ClientHost가
|
||||
프록시 주소가 아님)를 본다. 바꿀 것을 `kubectl diff`로 보여 주고, 차이가 없으면 아무것도 하지 않는다.
|
||||
- 적용 뒤 같은 검사를 다시 하고, 하나라도 실패하면 직전에 라이브였던 HelmChartConfig를 다시 적용한다.
|
||||
trusted CIDR이 빠지는 일은 없다.
|
||||
|
||||
### 롤백
|
||||
|
||||
trust만 제거하고 JSON access log를 남기는 표준 롤백은 observe overlay를 적용한다.
|
||||
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
#!/usr/bin/env bash
|
||||
# Re-apply the trust overlay when the live Traefik is already in the trust phase,
|
||||
# for example after a values change such as resources. apply-trust.sh is only for
|
||||
# the observe -> trust transition and stops when the runtime already trusts the
|
||||
# CIDR ("observation phase must not trust forwarded headers").
|
||||
#
|
||||
# bash infrastructure/networking/traefik/scripts/reapply-trust.sh \
|
||||
# --observed-client-host 10.42.0.1 --execute
|
||||
#
|
||||
# On any failure after the apply it restores the HelmChartConfig that was live
|
||||
# before, so the trusted CIDR is never dropped.
|
||||
set -Eeuo pipefail
|
||||
set +x
|
||||
umask 077
|
||||
|
||||
readonly SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)"
|
||||
# shellcheck source=common.sh
|
||||
source "${SCRIPT_DIR}/common.sh"
|
||||
|
||||
[[ "$#" -eq 3 && "$1" == "--observed-client-host" && "$3" == "--execute" ]] || {
|
||||
sed -n '2,11p' "${BASH_SOURCE[0]}" >&2
|
||||
exit 2
|
||||
}
|
||||
readonly REVIEWED_CLIENT_HOST="$2"
|
||||
|
||||
require_commands kubectl jq rg curl nc python3 awk sort date cmp find bash
|
||||
reviewed_cidr="$(host_to_exact_cidr "$REVIEWED_CLIENT_HOST")"
|
||||
declared_cidr="$(normalize_exact_host_cidr "$(source_trusted_proxy_cidr)")"
|
||||
[[ "$declared_cidr" != "$SENTINEL_TRUSTED_PROXY_CIDR" ]] || \
|
||||
fail "trust overlay still contains the non-routable sentinel CIDR"
|
||||
[[ "$declared_cidr" == "$reviewed_cidr" ]] || \
|
||||
fail "declared CIDR ${declared_cidr} does not match ClientHost ${REVIEWED_CLIENT_HOST}"
|
||||
|
||||
bash "${SCRIPT_DIR}/validate.sh"
|
||||
|
||||
work_dir="$(mktemp -d "${TMPDIR:-/tmp}/platform-traefik-retrust.XXXXXX")"
|
||||
restore_required=false
|
||||
selected_context=""
|
||||
cleanup() {
|
||||
local exit_code=$?
|
||||
trap - EXIT
|
||||
if [[ "$exit_code" -ne 0 && "$restore_required" == "true" ]]; then
|
||||
printf '\nRESTORE: re-applying the HelmChartConfig that was live before.\n' >&2
|
||||
set +e
|
||||
kubectl --context "$selected_context" apply --filename "${work_dir}/previous.yaml"
|
||||
wait_for_runtime trust "$declared_cidr"
|
||||
kubectl --context "$selected_context" --namespace kube-system \
|
||||
rollout status deployment/traefik --timeout=5m
|
||||
wait_for_nodeport_boundary_and_health
|
||||
printf 'RESTORE complete. The previous trust values are live again.\n' >&2
|
||||
set -e
|
||||
fi
|
||||
case "$work_dir" in
|
||||
/tmp/platform-traefik-retrust.*|"${TMPDIR:-/tmp}"/platform-traefik-retrust.*) rm -rf -- "$work_dir" ;;
|
||||
esac
|
||||
exit "$exit_code"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
trap 'exit 130' INT
|
||||
trap 'exit 143' TERM
|
||||
|
||||
trust_render="${work_dir}/trust.yaml"
|
||||
render_overlay "$TRUST_OVERLAY" "$trust_render"
|
||||
|
||||
assert_live_baseline
|
||||
assert_runtime trust "$declared_cidr"
|
||||
assert_nodeport_boundary_and_health
|
||||
|
||||
# In the trust phase Traefik logs the forwarded client, not the proxy. Seeing the
|
||||
# proxy address (the reviewed ClientHost) would mean the trust is not working.
|
||||
assert_trust_effective() {
|
||||
local seen
|
||||
seen="$(observe_host_nginx_client_host)"
|
||||
[[ "$seen" != "$REVIEWED_CLIENT_HOST" ]] || \
|
||||
fail "Traefik logged the proxy ${seen} as ClientHost; forwarded-header trust is not in effect"
|
||||
printf 'Forwarded-header trust in effect (ClientHost %s, proxy %s)\n' "$seen" "$REVIEWED_CLIENT_HOST"
|
||||
}
|
||||
assert_trust_effective
|
||||
|
||||
kubectl --namespace kube-system get helmchartconfig traefik --output=json |
|
||||
python3 -I -c '
|
||||
import json, sys
|
||||
o = json.load(sys.stdin)
|
||||
md = o["metadata"]
|
||||
keep = {k: md[k] for k in ("name", "namespace", "labels", "annotations") if k in md}
|
||||
keep.get("annotations", {}).pop("kubectl.kubernetes.io/last-applied-configuration", None)
|
||||
print(json.dumps({"apiVersion": o["apiVersion"], "kind": o["kind"], "metadata": keep, "spec": o["spec"]}))
|
||||
' >"${work_dir}/previous.yaml"
|
||||
|
||||
printf '\nChange to apply (kubectl diff, server dry-run):\n'
|
||||
set +e
|
||||
KUBECTL_EXTERNAL_DIFF="diff -u -N" kubectl diff --filename "$trust_render" | grep -E '^[-+] ' | grep -v -E '^[-+] generation:'
|
||||
diff_rc=${PIPESTATUS[0]}
|
||||
set -e
|
||||
if [[ "$diff_rc" -eq 0 ]]; then
|
||||
printf 'Live HelmChartConfig already matches the trust overlay. Nothing to do.\n'
|
||||
exit 0
|
||||
fi
|
||||
(( diff_rc == 1 )) || fail "kubectl diff failed"
|
||||
|
||||
selected_context="$(current_context)"
|
||||
selected_api_server="$(current_api_server)"
|
||||
printf '\nKubernetes context: %s\nAPI server: %s\nTrusted CIDR (unchanged): %s\n' \
|
||||
"$selected_context" "$selected_api_server" "$declared_cidr"
|
||||
printf 'Type APPLY %s RETRUST %s to continue: ' "$selected_context" "$declared_cidr"
|
||||
read -r confirmation
|
||||
[[ "$confirmation" == "APPLY ${selected_context} RETRUST ${declared_cidr}" ]] || fail "cancelled"
|
||||
[[ "$(current_context)" == "$selected_context" ]] || fail "kubectl context changed after confirmation"
|
||||
[[ "$(current_api_server)" == "$selected_api_server" ]] || fail "Kubernetes API server changed after confirmation"
|
||||
assert_runtime trust "$declared_cidr"
|
||||
|
||||
restore_required=true
|
||||
kubectl apply --filename "$trust_render"
|
||||
wait_for_runtime trust "$declared_cidr"
|
||||
kubectl --namespace kube-system rollout status deployment/traefik --timeout=5m
|
||||
assert_runtime trust "$declared_cidr"
|
||||
assert_live_hcc_matches_overlay "$TRUST_OVERLAY" "${work_dir}/post-apply-trust.yaml"
|
||||
assert_live_baseline
|
||||
wait_for_nodeport_boundary_and_health
|
||||
assert_manifest_https
|
||||
assert_trust_effective
|
||||
restore_required=false
|
||||
|
||||
printf '\nTRUST PHASE RE-APPLIED\n'
|
||||
printf 'Traefik web trusted CIDR: %s\n' "$declared_cidr"
|
||||
printf 'Gitea health and HTTPS site-manifest checks: PASS\n'
|
||||
@@ -0,0 +1,68 @@
|
||||
#!/usr/bin/env bash
|
||||
# Back up Gitea (repositories, packages, attachments, Actions data, app.ini) to
|
||||
# AIStor (HDD sdb3). Root only. Run by platform-gitea-backup.timer.
|
||||
#
|
||||
# `gitea dump --skip-db` runs inside the live Gitea container and streams a
|
||||
# tar.gz to stdout (logs go to stderr). The database is backed up separately by
|
||||
# CloudNativePG (services/platform-postgres-backup). The dump contains app.ini,
|
||||
# whose secrets are needed to restore, so the archive is sensitive.
|
||||
set -Eeuo pipefail
|
||||
umask 077
|
||||
# shellcheck source=../lib/host-backup.sh
|
||||
source "$(dirname -- "${BASH_SOURCE[0]}")/../lib/host-backup.sh"
|
||||
|
||||
readonly NAME=gitea_backup
|
||||
readonly KCFG=/etc/rancher/k3s/k3s.yaml
|
||||
[[ "$(id -u)" -eq 0 ]] || hb_fail "run as root"
|
||||
hb_load_aistor_env
|
||||
kubectl() { command /usr/local/bin/k3s kubectl --kubeconfig "$KCFG" "$@"; }
|
||||
|
||||
started="$(date +%s)"
|
||||
previous_success="$(awk '/^platform_backup_last_success_timestamp_seconds/ {print $2}' "$TEXTFILE_DIR/$NAME.prom" 2>/dev/null || echo 0)"
|
||||
on_error() {
|
||||
hb_write_metrics "$NAME" <<PROM
|
||||
platform_backup_last_run_success{backup="gitea"} 0
|
||||
platform_backup_last_run_timestamp_seconds{backup="gitea"} $(date +%s)
|
||||
platform_backup_last_success_timestamp_seconds{backup="gitea"} ${previous_success:-0}
|
||||
PROM
|
||||
}
|
||||
trap on_error ERR
|
||||
|
||||
work="$(mktemp -d "$HB_WORK_ROOT/gitea.XXXXXX")"
|
||||
trap 'rm -rf -- "$work"' EXIT
|
||||
ts="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
archive="gitea-dump-$ts.tar.gz"
|
||||
|
||||
kubectl -n gitea exec deploy/gitea -c gitea -- \
|
||||
gitea dump --skip-db --type tar.gz --file - >"$work/$archive" 2>"$work/dump.log"
|
||||
tar -tzf "$work/$archive" >"$work/list.txt"
|
||||
grep -qx 'app.ini' "$work/list.txt" || hb_fail "the dump has no app.ini"
|
||||
repos="$(grep -c -E '^repos/[^/]+/[^/]+\.git/HEAD$' "$work/list.txt" || true)"
|
||||
(( repos > 0 )) || hb_fail "the dump has no repositories"
|
||||
sha256sum "$work/$archive" | awk '{print $1}' >"$work/$archive.sha256"
|
||||
size="$(stat -c %s "$work/$archive")"
|
||||
# What a restore must match: restore with the same Gitea version (docs/runbooks/gitea-restore.md).
|
||||
{
|
||||
echo "created=$ts"
|
||||
echo "gitea_image=$(kubectl -n gitea get deploy gitea -o jsonpath='{.spec.template.spec.containers[0].image}')"
|
||||
echo "gitea_version=$(kubectl -n gitea exec deploy/gitea -c gitea -- gitea --version 2>/dev/null | awk '{print $3}')"
|
||||
echo "repositories=$repos"
|
||||
} >"$work/$archive.meta"
|
||||
|
||||
hb_mc "$work" cp --quiet "/work/$archive" "/work/$archive.sha256" "/work/$archive.meta" "backup/$AISTOR_BUCKET/gitea/" >/dev/null
|
||||
hb_mc "$work" stat "backup/$AISTOR_BUCKET/gitea/$archive" >/dev/null
|
||||
|
||||
trap - ERR
|
||||
hb_write_metrics "$NAME" <<PROM
|
||||
# HELP platform_backup_last_success_timestamp_seconds Unix time of the last successful backup.
|
||||
# TYPE platform_backup_last_success_timestamp_seconds gauge
|
||||
platform_backup_last_success_timestamp_seconds{backup="gitea"} $(date +%s)
|
||||
platform_backup_last_run_success{backup="gitea"} 1
|
||||
platform_backup_last_run_timestamp_seconds{backup="gitea"} $(date +%s)
|
||||
platform_backup_last_size_bytes{backup="gitea"} $size
|
||||
platform_backup_last_duration_seconds{backup="gitea"} $(( $(date +%s) - started ))
|
||||
# HELP platform_backup_gitea_repositories Repositories in the last successful Gitea dump.
|
||||
# TYPE platform_backup_gitea_repositories gauge
|
||||
platform_backup_gitea_repositories $repos
|
||||
PROM
|
||||
hb_log "gitea backup $archive ($size bytes, $repos repositories) uploaded"
|
||||
@@ -0,0 +1,99 @@
|
||||
#!/usr/bin/env bash
|
||||
# Back up the k3s datastore (sqlite) and the server token to AIStor (HDD sdb3).
|
||||
# Root only. Run by platform-k3s-datastore-backup.timer (infrastructure/host/backup).
|
||||
#
|
||||
# k3s docs (datastore/backup-restore, SQLite): copy server/db and server/token;
|
||||
# the token is required to read the datastore back. `sqlite3 .backup` takes a
|
||||
# consistent copy of the live database.
|
||||
#
|
||||
# Off-host copy (prepared, off by default): set in /etc/platform-backup/aistor.env
|
||||
# OFFHOST_GPG_RECIPIENT_FILE=/etc/platform-backup/offhost-recipient.asc
|
||||
# OFFHOST_RCLONE_TARGET=<rclone remote:path>
|
||||
# The archive is then encrypted to that public key and copied with rclone.
|
||||
set -Eeuo pipefail
|
||||
umask 077
|
||||
# shellcheck source=../lib/host-backup.sh
|
||||
source "$(dirname -- "${BASH_SOURCE[0]}")/../lib/host-backup.sh"
|
||||
|
||||
readonly SERVER=/var/lib/rancher/k3s/server
|
||||
readonly NAME=k3s_datastore_backup
|
||||
[[ "$(id -u)" -eq 0 ]] || hb_fail "run as root"
|
||||
command -v sqlite3 >/dev/null || hb_fail "sqlite3 is required"
|
||||
hb_load_aistor_env
|
||||
|
||||
started="$(date +%s)"
|
||||
on_error() {
|
||||
hb_write_metrics "$NAME" <<PROM
|
||||
# HELP platform_backup_last_run_success 1 when the last run of this backup succeeded.
|
||||
# TYPE platform_backup_last_run_success gauge
|
||||
platform_backup_last_run_success{backup="k3s-datastore"} 0
|
||||
platform_backup_last_run_timestamp_seconds{backup="k3s-datastore"} $(date +%s)
|
||||
platform_backup_last_success_timestamp_seconds{backup="k3s-datastore"} ${previous_success:-0}
|
||||
PROM
|
||||
}
|
||||
previous_success="$(awk '/^platform_backup_last_success_timestamp_seconds/ {print $2}' "$TEXTFILE_DIR/$NAME.prom" 2>/dev/null || echo 0)"
|
||||
trap on_error ERR
|
||||
|
||||
work="$(mktemp -d "$HB_WORK_ROOT/k3s.XXXXXX")"
|
||||
trap 'rm -rf -- "$work"' EXIT
|
||||
ts="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
stage="$work/k3s-datastore-$ts"
|
||||
mkdir -m 0700 "$stage"
|
||||
|
||||
sqlite3 "$SERVER/db/state.db" ".backup '$stage/state.db'"
|
||||
check="$(sqlite3 "$stage/state.db" 'PRAGMA integrity_check;')"
|
||||
[[ "$check" == "ok" ]] || hb_fail "integrity_check on the copy returned: $check"
|
||||
install -m 0600 "$SERVER/token" "$stage/token"
|
||||
{
|
||||
echo "created=$ts"
|
||||
echo "k3s_version=$(/usr/local/bin/k3s --version | head -1)"
|
||||
echo "host=$(hostname)"
|
||||
echo "kine_rows=$(sqlite3 "$stage/state.db" 'SELECT COUNT(*) FROM kine;')"
|
||||
} >"$stage/META"
|
||||
tar -C "$work" -czf "$work/k3s-datastore-$ts.tar.gz" "k3s-datastore-$ts"
|
||||
sha256sum "$work/k3s-datastore-$ts.tar.gz" | awk '{print $1}' >"$work/k3s-datastore-$ts.tar.gz.sha256"
|
||||
size="$(stat -c %s "$work/k3s-datastore-$ts.tar.gz")"
|
||||
|
||||
hb_mc "$work" cp --quiet "/work/k3s-datastore-$ts.tar.gz" "/work/k3s-datastore-$ts.tar.gz.sha256" "backup/$AISTOR_BUCKET/k3s-datastore/" >/dev/null
|
||||
hb_mc "$work" stat "backup/$AISTOR_BUCKET/k3s-datastore/k3s-datastore-$ts.tar.gz" >/dev/null
|
||||
|
||||
# Local copy outside AIStor on the same HDD (sdb3). AIStor runs inside the
|
||||
# cluster this archive restores, so after losing the cluster the AIStor copy
|
||||
# cannot be read until k3s is back; this copy can. The directory sits next to
|
||||
# AIStor's drive directory (data/), so AIStor does not see it as a bucket.
|
||||
readonly LOCAL_DIR=/srv/k3s/aistor/host-backups/k3s-datastore
|
||||
install -d -m 0700 -o root -g root "$(dirname "$LOCAL_DIR")" "$LOCAL_DIR"
|
||||
install -m 0600 "$work/k3s-datastore-$ts.tar.gz" "$work/k3s-datastore-$ts.tar.gz.sha256" "$LOCAL_DIR/"
|
||||
ls -1t "$LOCAL_DIR"/k3s-datastore-*.tar.gz | tail -n +8 | while read -r old; do rm -f -- "$old" "$old.sha256"; done
|
||||
local_copies="$(ls -1 "$LOCAL_DIR"/k3s-datastore-*.tar.gz | wc -l)"
|
||||
|
||||
offhost=0
|
||||
if [[ -n "${OFFHOST_GPG_RECIPIENT_FILE:-}" && -n "${OFFHOST_RCLONE_TARGET:-}" ]]; then
|
||||
command -v rclone >/dev/null || hb_fail "OFFHOST_RCLONE_TARGET is set but rclone is not installed"
|
||||
gpg --batch --yes --trust-model always --recipient-file "$OFFHOST_GPG_RECIPIENT_FILE" \
|
||||
--output "$work/k3s-datastore-$ts.tar.gz.gpg" --encrypt "$work/k3s-datastore-$ts.tar.gz"
|
||||
rclone copyto "$work/k3s-datastore-$ts.tar.gz.gpg" "$OFFHOST_RCLONE_TARGET/k3s-datastore-$ts.tar.gz.gpg"
|
||||
offhost=1
|
||||
fi
|
||||
|
||||
trap - ERR
|
||||
hb_write_metrics "$NAME" <<PROM
|
||||
# HELP platform_backup_last_success_timestamp_seconds Unix time of the last successful backup.
|
||||
# TYPE platform_backup_last_success_timestamp_seconds gauge
|
||||
platform_backup_last_success_timestamp_seconds{backup="k3s-datastore"} $(date +%s)
|
||||
# HELP platform_backup_last_run_success 1 when the last run of this backup succeeded.
|
||||
# TYPE platform_backup_last_run_success gauge
|
||||
platform_backup_last_run_success{backup="k3s-datastore"} 1
|
||||
platform_backup_last_run_timestamp_seconds{backup="k3s-datastore"} $(date +%s)
|
||||
# HELP platform_backup_last_size_bytes Size of the last successful backup archive.
|
||||
# TYPE platform_backup_last_size_bytes gauge
|
||||
platform_backup_last_size_bytes{backup="k3s-datastore"} $size
|
||||
platform_backup_last_duration_seconds{backup="k3s-datastore"} $(( $(date +%s) - started ))
|
||||
# HELP platform_backup_local_copies Archives kept outside AIStor on the HDD.
|
||||
# TYPE platform_backup_local_copies gauge
|
||||
platform_backup_local_copies{backup="k3s-datastore"} $local_copies
|
||||
# HELP platform_backup_offhost_copy 1 when the last run also wrote the encrypted off-host copy.
|
||||
# TYPE platform_backup_offhost_copy gauge
|
||||
platform_backup_offhost_copy{backup="k3s-datastore"} $offhost
|
||||
PROM
|
||||
hb_log "k3s datastore backup k3s-datastore-$ts.tar.gz ($size bytes) uploaded; off-host=$offhost"
|
||||
@@ -0,0 +1,238 @@
|
||||
#!/usr/bin/env bash
|
||||
# Install the k3s restart bundle and restart k3s exactly once.
|
||||
#
|
||||
# bash scripts/bootstrap/apply-k3s-restart-bundle.sh --check # no root, changes nothing
|
||||
# sudo bash scripts/bootstrap/apply-k3s-restart-bundle.sh --execute
|
||||
#
|
||||
# The bundle (infrastructure/host/k3s/README.md):
|
||||
# W6: kubelet reservations, memory eviction and graceful shutdown (kubelet drop-in),
|
||||
# API server audit log and cluster Pod Security default (k3s drop-in + two files),
|
||||
# local-path StorageClass no longer the default (manifest .skip + annotation).
|
||||
# W1: k3s starts after time sync (infrastructure/host/systemd/*, installed as is).
|
||||
#
|
||||
# --execute backs up every file it replaces plus a datastore copy, installs, restarts
|
||||
# k3s, verifies, and on any failure restores the previous files and restarts again.
|
||||
set -Eeuo pipefail
|
||||
umask 077
|
||||
|
||||
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
|
||||
readonly KCFG=/etc/rancher/k3s/k3s.yaml
|
||||
readonly SERVER=/var/lib/rancher/k3s/server
|
||||
readonly SKIP_FILE="$SERVER/manifests/local-storage.yaml.skip"
|
||||
readonly AUDIT_DIR="$SERVER/logs"
|
||||
readonly BACKUP_ROOT=/var/backups/platform-k3s-restart-bundle
|
||||
|
||||
# source | destination | mode | owner of the source file
|
||||
readonly -a FILES=(
|
||||
"infrastructure/host/k3s/kubelet.conf.d/50-platform-resources.conf|/var/lib/rancher/k3s/agent/etc/kubelet.conf.d/50-platform-resources.conf|0600|W6"
|
||||
"infrastructure/host/k3s/config.yaml.d/60-platform-apiserver.yaml|/etc/rancher/k3s/config.yaml.d/60-platform-apiserver.yaml|0644|W6"
|
||||
"infrastructure/host/k3s/server/audit.yaml|$SERVER/audit.yaml|0600|W6"
|
||||
"infrastructure/host/k3s/server/psa.yaml|$SERVER/psa.yaml|0600|W6"
|
||||
"infrastructure/host/systemd/k3s.service.d/10-after-time-sync.conf|/etc/systemd/system/k3s.service.d/10-after-time-sync.conf|0644|W1"
|
||||
"infrastructure/host/systemd/systemd-time-wait-sync.service.d/10-timeout.conf|/etc/systemd/system/systemd-time-wait-sync.service.d/10-timeout.conf|0644|W1"
|
||||
)
|
||||
readonly -a HEALTH_URLS=(
|
||||
https://id.learn.hyeonworks.com/realms/hyeonworks/.well-known/openid-configuration
|
||||
https://hyeonworks.com/api/healthcheck
|
||||
https://git.learn.hyeonworks.com/api/healthz
|
||||
)
|
||||
|
||||
say() { printf '%s\n' "$*"; }
|
||||
fail() { printf 'ERROR: %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
mode="${1:-}"
|
||||
[[ "$#" -eq 1 && ( "$mode" == "--check" || "$mode" == "--execute" ) ]] || {
|
||||
sed -n '2,15p' "${BASH_SOURCE[0]}" >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
check_sources() {
|
||||
local entry src dst perm owner missing=0
|
||||
for entry in "${FILES[@]}"; do
|
||||
IFS='|' read -r src dst perm owner <<<"$entry"
|
||||
if [[ ! -f "$ROOT/$src" ]]; then
|
||||
say "MISSING $src (${owner})"
|
||||
missing=1
|
||||
fi
|
||||
done
|
||||
(( missing == 0 )) || fail "run this from a platform-core checkout that has both the W6 and the W1 host files on main"
|
||||
python3 -I - "$ROOT" <<'PY' || fail "bundle files do not parse"
|
||||
import sys, pathlib, yaml
|
||||
root = pathlib.Path(sys.argv[1]) / "infrastructure/host/k3s"
|
||||
k = yaml.safe_load((root / "kubelet.conf.d/50-platform-resources.conf").read_text())
|
||||
assert k["kind"] == "KubeletConfiguration", "kubelet drop-in kind"
|
||||
assert set(k["evictionHard"]) == {"memory.available", "nodefs.available", "imagefs.available"}, "evictionHard keys"
|
||||
assert k["shutdownGracePeriod"] == "60s" and k["shutdownGracePeriodCriticalPods"] == "20s", "shutdown periods"
|
||||
a = yaml.safe_load((root / "config.yaml.d/60-platform-apiserver.yaml").read_text())
|
||||
assert list(a) == ["kube-apiserver-arg+"], "apiserver drop-in must only append kube-apiserver-arg"
|
||||
args = dict(x.split("=", 1) for x in a["kube-apiserver-arg+"])
|
||||
assert args["audit-policy-file"] == "/var/lib/rancher/k3s/server/audit.yaml"
|
||||
assert args["admission-control-config-file"] == "/var/lib/rancher/k3s/server/psa.yaml"
|
||||
p = yaml.safe_load((root / "server/audit.yaml").read_text())
|
||||
assert p["kind"] == "Policy" and p["rules"][-1] == {"level": "Metadata"}, "audit policy ends with Metadata"
|
||||
assert all(r["level"] in ("None", "Metadata") for r in p["rules"]), "audit policy must not log bodies"
|
||||
s = yaml.safe_load((root / "server/psa.yaml").read_text())
|
||||
d = s["plugins"][0]["configuration"]["defaults"]
|
||||
assert d["enforce"] == "baseline", "PSA default enforce"
|
||||
assert s["plugins"][0]["configuration"]["exemptions"]["namespaces"] == ["kube-system"], "PSA exemptions"
|
||||
print("bundle files parse and keep their invariants")
|
||||
PY
|
||||
}
|
||||
|
||||
plan() {
|
||||
local entry src dst perm owner state
|
||||
for entry in "${FILES[@]}"; do
|
||||
IFS='|' read -r src dst perm owner <<<"$entry"
|
||||
if [[ -d "$(dirname "$dst")" && ! -r "$(dirname "$dst")" ]] ||
|
||||
[[ ! -e "$(dirname "$dst")" && ! -r "$(dirname "$(dirname "$dst")")" ]]; then
|
||||
state="unknown (needs root to read)"
|
||||
elif [[ ! -e "$dst" ]]; then
|
||||
state="new"
|
||||
elif cmp -s "$ROOT/$src" "$dst"; then
|
||||
state="same"
|
||||
else
|
||||
state="replace"
|
||||
fi
|
||||
say " ${owner} ${state} ${dst}"
|
||||
done
|
||||
say " W6 touch $SKIP_FILE"
|
||||
say " W6 mkdir $AUDIT_DIR (0700)"
|
||||
say " W1 systemctl enable systemd-time-wait-sync.service"
|
||||
say " -- systemctl daemon-reload && systemctl restart k3s (once)"
|
||||
}
|
||||
|
||||
say "== check"
|
||||
check_sources
|
||||
say "== plan"
|
||||
plan
|
||||
[[ "$mode" == "--execute" ]] || { say "CHECK ONLY: nothing changed"; exit 0; }
|
||||
|
||||
[[ "$(id -u)" -eq 0 ]] || fail "--execute needs root (sudo)"
|
||||
command -v sqlite3 >/dev/null || fail "sqlite3 is required for the pre-restart datastore copy"
|
||||
kubectl() { command k3s kubectl --kubeconfig "$KCFG" "$@"; }
|
||||
|
||||
wait_ready() {
|
||||
local i
|
||||
for i in $(seq 1 60); do
|
||||
if kubectl get --raw=/readyz >/dev/null 2>&1 &&
|
||||
[[ "$(kubectl get nodes -o jsonpath='{.items[0].status.conditions[?(@.type=="Ready")].status}' 2>/dev/null)" == "True" ]]; then
|
||||
return 0
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
say "== preflight"
|
||||
wait_ready || fail "the cluster is not Ready before the change; not touching it"
|
||||
for url in "${HEALTH_URLS[@]}"; do
|
||||
code="$(curl -s -o /dev/null -m 15 -w '%{http_code}' "$url" || true)"
|
||||
say " before $code $url"
|
||||
done
|
||||
|
||||
ts="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
backup="$BACKUP_ROOT/$ts"
|
||||
install -d -m 0700 "$backup"
|
||||
time_wait_was_enabled="$(systemctl is-enabled systemd-time-wait-sync.service 2>/dev/null || true)"
|
||||
say "== backup to $backup"
|
||||
for entry in "${FILES[@]}"; do
|
||||
IFS='|' read -r src dst perm owner <<<"$entry"
|
||||
if [[ -e "$dst" ]]; then
|
||||
install -D -m 0600 "$dst" "$backup/files$dst"
|
||||
fi
|
||||
done
|
||||
sqlite3 "$SERVER/db/state.db" ".backup '$backup/state.db'"
|
||||
install -m 0600 "$SERVER/token" "$backup/token"
|
||||
say " datastore copy: $(sqlite3 "$backup/state.db" 'PRAGMA integrity_check;') $(du -h "$backup/state.db" | cut -f1)"
|
||||
|
||||
rollback() {
|
||||
local entry src dst perm owner
|
||||
trap - ERR
|
||||
say "== ROLLBACK"
|
||||
set +e
|
||||
for entry in "${FILES[@]}"; do
|
||||
IFS='|' read -r src dst perm owner <<<"$entry"
|
||||
if [[ -e "$backup/files$dst" ]]; then
|
||||
install -D -m "$perm" "$backup/files$dst" "$dst"
|
||||
else
|
||||
rm -f -- "$dst"
|
||||
fi
|
||||
done
|
||||
rm -f -- "$SKIP_FILE"
|
||||
[[ "$time_wait_was_enabled" == "enabled" ]] || systemctl disable systemd-time-wait-sync.service
|
||||
systemctl daemon-reload
|
||||
systemctl restart --no-block k3s
|
||||
sleep 10
|
||||
if wait_ready; then say "ROLLBACK done: previous files restored, k3s Ready"; else say "ROLLBACK: k3s still not Ready; see journalctl -u k3s"; fi
|
||||
say "backup kept at $backup"
|
||||
exit 1
|
||||
}
|
||||
trap rollback ERR
|
||||
|
||||
say "== install"
|
||||
for entry in "${FILES[@]}"; do
|
||||
IFS='|' read -r src dst perm owner <<<"$entry"
|
||||
install -D -o root -g root -m "$perm" "$ROOT/$src" "$dst"
|
||||
done
|
||||
install -d -o root -g root -m 0700 "$AUDIT_DIR"
|
||||
install -D -o root -g root -m 0600 /dev/null "$SKIP_FILE"
|
||||
systemctl daemon-reload
|
||||
systemctl enable systemd-time-wait-sync.service
|
||||
|
||||
say "== restart k3s"
|
||||
# k3s.service has TimeoutStartSec=0, so a blocking restart could wait forever.
|
||||
systemctl restart --no-block k3s
|
||||
sleep 10
|
||||
wait_ready || { say "k3s did not become Ready in 5 minutes"; false; }
|
||||
|
||||
say "== verify"
|
||||
node="$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')"
|
||||
kubectl get --raw "/api/v1/nodes/$node/proxy/configz" | python3 -I -c '
|
||||
import json, sys
|
||||
k = json.load(sys.stdin)["kubeletconfig"]
|
||||
want = {
|
||||
("systemReserved", "memory"): "2Gi", ("kubeReserved", "memory"): "1Gi",
|
||||
("evictionHard", "memory.available"): "500Mi", ("evictionHard", "nodefs.available"): "5%",
|
||||
("evictionHard", "imagefs.available"): "5%", ("evictionSoft", "memory.available"): "1Gi",
|
||||
}
|
||||
bad = [f"{a}.{b}={k.get(a, {}).get(b)}" for (a, b), v in want.items() if k.get(a, {}).get(b) != v]
|
||||
if k.get("shutdownGracePeriod") != "1m0s": bad.append(f"shutdownGracePeriod={k.get(\"shutdownGracePeriod\")}")
|
||||
if bad: sys.exit("kubelet config not applied: " + ", ".join(bad))
|
||||
print(" kubelet: reservations, eviction and graceful shutdown applied")
|
||||
'
|
||||
for i in $(seq 1 12); do [[ -s "$AUDIT_DIR/audit.log" ]] && break; sleep 5; done
|
||||
[[ -s "$AUDIT_DIR/audit.log" ]] || { say "audit log is empty"; false; }
|
||||
say " audit log: $(wc -l <"$AUDIT_DIR/audit.log") lines"
|
||||
psa_out="$(kubectl apply --dry-run=server -f - 2>&1 <<'YAML' || true
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: psa-default-probe
|
||||
namespace: default
|
||||
spec:
|
||||
containers:
|
||||
- name: probe
|
||||
image: registry.k8s.io/pause:3.10
|
||||
securityContext:
|
||||
privileged: true
|
||||
YAML
|
||||
)"
|
||||
grep -q 'violates PodSecurity "baseline' <<<"$psa_out" || { say "PSA default did not reject a privileged pod: $psa_out"; false; }
|
||||
say " pod security: default namespace rejects a privileged pod (baseline)"
|
||||
for url in "${HEALTH_URLS[@]}"; do
|
||||
ok=0
|
||||
for i in $(seq 1 30); do
|
||||
[[ "$(curl -s -o /dev/null -m 15 -w '%{http_code}' "$url" || true)" == "200" ]] && { ok=1; break; }
|
||||
sleep 10
|
||||
done
|
||||
(( ok == 1 )) || { say "health check failed: $url"; false; }
|
||||
say " after 200 $url"
|
||||
done
|
||||
say " time sync: $(systemctl is-enabled systemd-time-wait-sync.service), $(systemctl show systemd-time-wait-sync.service -p TimeoutStartUSec)"
|
||||
trap - ERR
|
||||
|
||||
say "== local-path is no longer the default StorageClass"
|
||||
kubectl annotate storageclass local-path storageclass.kubernetes.io/is-default-class=false --overwrite
|
||||
kubectl get storageclass local-path -o jsonpath=' local-path default={.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}{"\n"}'
|
||||
|
||||
say "K3S RESTART BUNDLE APPLIED (backup: $backup)"
|
||||
@@ -189,8 +189,8 @@ assert_regex_count \
|
||||
|
||||
assert_regex_count \
|
||||
"${REPOSITORY_ROOT}/infrastructure/controllers/keycloak-operator/kustomization.yaml" \
|
||||
'github\.com/keycloak/keycloak-k8s-resources/kubernetes\?ref=26\.7\.0' 1 \
|
||||
"Keycloak Operator 26.7.0 source pin"
|
||||
'github\.com/keycloak/keycloak-k8s-resources/kubernetes\?ref=38fd7a0c84e15d4b66b37d1c2ea728ca2f5416a4' 1 \
|
||||
"Keycloak Operator 26.7.0 commit pin"
|
||||
assert_regex_count \
|
||||
"${render_temp_dir}/keycloak-operator.yaml" \
|
||||
'^kind:[[:space:]]CustomResourceDefinition$' 4 \
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
#!/usr/bin/env bash
|
||||
# Create the AIStor bucket, lifecycle rules, policy and user that the host backup
|
||||
# timers use, and the root-only /etc/platform-backup/aistor.env.
|
||||
#
|
||||
# sudo bash scripts/bootstrap/create-host-backup-credentials.sh
|
||||
#
|
||||
# Run once, by the owner, on the desktop. Refuses to run when the env file exists;
|
||||
# never rotates or deletes a bucket, policy, user or object. No key is printed.
|
||||
set -Eeuo pipefail
|
||||
set +x
|
||||
umask 077
|
||||
|
||||
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
|
||||
# shellcheck source=../lib/host-backup.sh
|
||||
source "$ROOT/scripts/lib/host-backup.sh"
|
||||
readonly BUCKET=platform-host-backups
|
||||
readonly POLICY=platform-host-backups
|
||||
readonly QUOTA=50GiB
|
||||
readonly POLICY_FILE="$ROOT/services/platform-host-checks/bucket-policy.json"
|
||||
readonly PASS_LINE=PLATFORM_HOST_BACKUP_BOOTSTRAP_PASS
|
||||
kubectl() { command /usr/local/bin/k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml "$@"; }
|
||||
|
||||
[[ "$(id -u)" -eq 0 ]] || hb_fail "run with sudo"
|
||||
for c in docker jq openssl; do command -v "$c" >/dev/null || hb_fail "missing command: $c"; done
|
||||
jq -e . "$POLICY_FILE" >/dev/null || hb_fail "bucket policy is not valid JSON"
|
||||
[[ ! -e "$HOST_BACKUP_ENV" ]] || hb_fail "$HOST_BACKUP_ENV already exists; this script does not rotate it"
|
||||
|
||||
ip="$(kubectl -n object-storage get service minio -o jsonpath='{.spec.clusterIP}')"
|
||||
[[ -n "$ip" ]] || hb_fail "object-storage/minio Service not found"
|
||||
endpoint="http://${ip}:80"
|
||||
|
||||
work="$(mktemp -d)"
|
||||
trap 'rm -rf -- "$work"' EXIT
|
||||
kubectl -n object-storage get secret aistor-root-configuration -o jsonpath='{.data.config\.env}' | base64 -d >"$work/root.env"
|
||||
grep -q '^export MINIO_ROOT_USER=' "$work/root.env" || hb_fail "aistor-root-configuration has an unexpected format"
|
||||
printf 'pcbackup%s' "$(openssl rand -hex 8)" >"$work/ACCESS_KEY"
|
||||
openssl rand -hex 32 | tr -d '\n' >"$work/SECRET_KEY"
|
||||
cp "$POLICY_FILE" "$work/policy.json"
|
||||
|
||||
printf '%s\n' \
|
||||
"Bucket: $BUCKET (quota $QUOTA; expire gitea/ after 56 days, k3s-datastore/ after 30 days) on $endpoint" \
|
||||
"Policy: $POLICY (${POLICY_FILE#"$ROOT/"})" \
|
||||
"User: a new generated access key attached to $POLICY" \
|
||||
"File: $HOST_BACKUP_ENV (root, 0600)" \
|
||||
'No key value will be printed.'
|
||||
printf 'Type APPLY to create them: '
|
||||
IFS= read -r confirmation
|
||||
[[ "$confirmation" == "APPLY" ]] || hb_fail "confirmation did not match APPLY"
|
||||
|
||||
# Every mc call is silenced so no key reaches the terminal.
|
||||
docker run --rm --network host -v "$work:/run/b:ro" --tmpfs /tmp -e MC_CONFIG_DIR=/tmp/mc \
|
||||
--entrypoint /bin/sh "$MC_IMAGE" -ec '
|
||||
. /run/b/root.env
|
||||
ak="$(cat /run/b/ACCESS_KEY)"; sk="$(cat /run/b/SECRET_KEY)"
|
||||
mc alias set root "'"$endpoint"'" "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" --api S3v4 >/dev/null
|
||||
mc mb --ignore-existing "root/'"$BUCKET"'" >/dev/null
|
||||
mc quota set "root/'"$BUCKET"'" --size "'"$QUOTA"'" >/dev/null
|
||||
mc ilm rule add --prefix gitea/ --expire-days 56 "root/'"$BUCKET"'" >/dev/null
|
||||
mc ilm rule add --prefix k3s-datastore/ --expire-days 30 "root/'"$BUCKET"'" >/dev/null
|
||||
mc admin policy create root "'"$POLICY"'" /run/b/policy.json >/dev/null
|
||||
mc admin user add root "$ak" "$sk" >/dev/null
|
||||
mc admin policy attach root "'"$POLICY"'" --user "$ak" >/dev/null
|
||||
mc alias set backup "'"$endpoint"'" "$ak" "$sk" --api S3v4 >/dev/null
|
||||
probe=".bootstrap-probe-$(date +%s)"
|
||||
printf probe | mc pipe "backup/'"$BUCKET"'/$probe" >/dev/null
|
||||
[ "$(mc cat "backup/'"$BUCKET"'/$probe")" = probe ]
|
||||
mc rm "backup/'"$BUCKET"'/$probe" >/dev/null
|
||||
if mc ls backup/loki >/dev/null 2>&1; then echo "the backup user can list another bucket" >&2; exit 1; fi
|
||||
echo "'"$PASS_LINE"'"' | grep -qx "$PASS_LINE" || hb_fail "bootstrap did not finish; $HOST_BACKUP_ENV was not written"
|
||||
|
||||
install -d -m 0700 -o root -g root "$(dirname "$HOST_BACKUP_ENV")"
|
||||
{
|
||||
echo "AISTOR_BUCKET=$BUCKET"
|
||||
echo "AISTOR_ACCESS_KEY=$(cat "$work/ACCESS_KEY")"
|
||||
echo "AISTOR_SECRET_KEY=$(cat "$work/SECRET_KEY")"
|
||||
echo "# Off-host encrypted copy of the k3s datastore archive (off until both are set):"
|
||||
echo "# OFFHOST_GPG_RECIPIENT_FILE=/etc/platform-backup/offhost-recipient.asc"
|
||||
echo "# OFFHOST_RCLONE_TARGET=remote:path"
|
||||
} >"$HOST_BACKUP_ENV"
|
||||
chmod 0600 "$HOST_BACKUP_ENV"
|
||||
printf '%s\n' "$PASS_LINE" "Created bucket $BUCKET, policy $POLICY, one user and $HOST_BACKUP_ENV."
|
||||
+170
@@ -0,0 +1,170 @@
|
||||
#!/usr/bin/env bash
|
||||
# Create the AIStor bucket, policy and user that CloudNativePG uses to back up
|
||||
# platform-postgres, and the platform-data/platform-postgres-backup-s3 Secret.
|
||||
#
|
||||
# Run once, by the owner, from a platform-core checkout on the desktop. The
|
||||
# script refuses to run when the Secret already exists; it never rotates or
|
||||
# deletes a bucket, policy, user or object. No credential value is printed.
|
||||
set -Eeuo pipefail
|
||||
set +x
|
||||
umask 077
|
||||
|
||||
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
|
||||
readonly EXPECTED_CONTEXT=default
|
||||
readonly OS_NAMESPACE=object-storage
|
||||
readonly PG_NAMESPACE=platform-data
|
||||
readonly ROOT_SECRET=aistor-root-configuration
|
||||
readonly TARGET_SECRET=platform-postgres-backup-s3
|
||||
readonly BUCKET=platform-postgres-backups
|
||||
readonly POLICY=platform-postgres-backups
|
||||
readonly QUOTA=100GiB
|
||||
readonly S3_ENDPOINT=http://minio.object-storage.svc.cluster.local
|
||||
# Same pinned AIStor client as scripts/validate/aistor-s3-smoke.sh.
|
||||
readonly MC_IMAGE="quay.io/minio/aistor/mc@sha256:c5ec777c080fd6292b7529309a0f311c58c3a92e268c9c57c0901da86881949b"
|
||||
readonly POLICY_FILE="${ROOT}/services/platform-postgres-backup/bucket-policy.json"
|
||||
readonly PASS_LINE=PLATFORM_POSTGRES_BACKUP_BOOTSTRAP_PASS
|
||||
|
||||
fail() {
|
||||
printf 'FAIL: %s\n' "$*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
for command in kubectl jq openssl; do
|
||||
command -v "$command" >/dev/null || fail "missing command: ${command}"
|
||||
done
|
||||
[[ -f "$POLICY_FILE" ]] || fail "missing ${POLICY_FILE}"
|
||||
jq -e . "$POLICY_FILE" >/dev/null || fail "bucket policy is not valid JSON"
|
||||
|
||||
[[ "$(kubectl config current-context)" == "$EXPECTED_CONTEXT" ]] ||
|
||||
fail "kube context must be ${EXPECTED_CONTEXT}"
|
||||
if kubectl --namespace "$PG_NAMESPACE" get secret "$TARGET_SECRET" >/dev/null 2>&1; then
|
||||
fail "${PG_NAMESPACE}/${TARGET_SECRET} already exists; this script does not rotate it"
|
||||
fi
|
||||
kubectl --namespace "$OS_NAMESPACE" get secret "$ROOT_SECRET" >/dev/null ||
|
||||
fail "${OS_NAMESPACE}/${ROOT_SECRET} is missing"
|
||||
|
||||
work="$(mktemp -d)"
|
||||
pod="pg-backup-bootstrap-$(date +%H%M%S)-$$"
|
||||
handoff_secret="${pod}-keys"
|
||||
policy_map="${pod}-policy"
|
||||
|
||||
cleanup() {
|
||||
kubectl --namespace "$OS_NAMESPACE" delete pod "$pod" \
|
||||
--ignore-not-found --wait=false >/dev/null 2>&1 || true
|
||||
kubectl --namespace "$OS_NAMESPACE" delete secret "$handoff_secret" \
|
||||
--ignore-not-found >/dev/null 2>&1 || true
|
||||
kubectl --namespace "$OS_NAMESPACE" delete configmap "$policy_map" \
|
||||
--ignore-not-found >/dev/null 2>&1 || true
|
||||
rm -rf -- "$work"
|
||||
}
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
printf 'pgbackup%s' "$(openssl rand -hex 8)" >"${work}/ACCESS_KEY_ID"
|
||||
openssl rand -hex 32 | tr -d '\n' >"${work}/ACCESS_SECRET_KEY"
|
||||
|
||||
printf '%s\n' \
|
||||
"Bucket: ${BUCKET} (quota ${QUOTA}) on AIStor ${S3_ENDPOINT}" \
|
||||
"Policy: ${POLICY} (${POLICY_FILE#"${ROOT}/"})" \
|
||||
"User: a new generated access key attached to ${POLICY}" \
|
||||
"Secret: ${PG_NAMESPACE}/${TARGET_SECRET} (ACCESS_KEY_ID, ACCESS_SECRET_KEY)" \
|
||||
"Temporary: one restricted client Pod, one Secret and one ConfigMap in ${OS_NAMESPACE}" \
|
||||
'No credential value will be printed.'
|
||||
printf 'Type APPLY to create them: '
|
||||
IFS= read -r confirmation
|
||||
[[ "$confirmation" == "APPLY" ]] || fail "confirmation did not match APPLY"
|
||||
|
||||
kubectl --namespace "$OS_NAMESPACE" create secret generic "$handoff_secret" \
|
||||
--from-file=ACCESS_KEY_ID="${work}/ACCESS_KEY_ID" \
|
||||
--from-file=ACCESS_SECRET_KEY="${work}/ACCESS_SECRET_KEY" >/dev/null
|
||||
kubectl --namespace "$OS_NAMESPACE" create configmap "$policy_map" \
|
||||
--from-file=policy.json="$POLICY_FILE" >/dev/null
|
||||
|
||||
# Every mc call is silenced so neither key reaches the Pod log.
|
||||
bootstrap_command='set -eu
|
||||
. /run/aistor-root/config.env
|
||||
access_key="$(cat /run/backup/ACCESS_KEY_ID)"
|
||||
secret_key="$(cat /run/backup/ACCESS_SECRET_KEY)"
|
||||
mc alias set root "'"$S3_ENDPOINT"'" "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" --api S3v4 >/dev/null
|
||||
mc mb --ignore-existing "root/'"$BUCKET"'" >/dev/null
|
||||
mc quota set "root/'"$BUCKET"'" --size "'"$QUOTA"'" >/dev/null
|
||||
mc admin policy create root "'"$POLICY"'" /run/policy/policy.json >/dev/null
|
||||
mc admin user add root "$access_key" "$secret_key" >/dev/null
|
||||
mc admin policy attach root "'"$POLICY"'" --user "$access_key" >/dev/null
|
||||
mc alias set backup "'"$S3_ENDPOINT"'" "$access_key" "$secret_key" --api S3v4 >/dev/null
|
||||
probe=".bootstrap-probe-$(date +%s)"
|
||||
printf "probe" | mc pipe "backup/'"$BUCKET"'/$probe" >/dev/null
|
||||
[ "$(mc cat "backup/'"$BUCKET"'/$probe")" = "probe" ]
|
||||
mc rm "backup/'"$BUCKET"'/$probe" >/dev/null
|
||||
if mc ls backup/loki >/dev/null 2>&1; then
|
||||
echo "the backup user can list another bucket" >&2
|
||||
exit 1
|
||||
fi
|
||||
printf "%s\n" "'"$PASS_LINE"'"'
|
||||
|
||||
overrides="$(
|
||||
jq -nc --arg command "$bootstrap_command" --arg image "$MC_IMAGE" \
|
||||
--arg root "$ROOT_SECRET" --arg keys "$handoff_secret" --arg policy "$policy_map" '{
|
||||
spec: {
|
||||
automountServiceAccountToken: false,
|
||||
securityContext: {
|
||||
runAsNonRoot: true,
|
||||
runAsUser: 1000,
|
||||
runAsGroup: 1000,
|
||||
fsGroup: 1000,
|
||||
seccompProfile: {type: "RuntimeDefault"}
|
||||
},
|
||||
containers: [{
|
||||
name: "pg-backup-bootstrap",
|
||||
image: $image,
|
||||
imagePullPolicy: "IfNotPresent",
|
||||
command: ["/bin/sh", "-ec"],
|
||||
args: [$command],
|
||||
securityContext: {
|
||||
allowPrivilegeEscalation: false,
|
||||
capabilities: {drop: ["ALL"]},
|
||||
readOnlyRootFilesystem: true
|
||||
},
|
||||
env: [{name: "MC_CONFIG_DIR", value: "/tmp/mc"}],
|
||||
volumeMounts: [
|
||||
{name: "root-config", mountPath: "/run/aistor-root", readOnly: true},
|
||||
{name: "backup-keys", mountPath: "/run/backup", readOnly: true},
|
||||
{name: "policy", mountPath: "/run/policy", readOnly: true},
|
||||
{name: "tmp", mountPath: "/tmp"}
|
||||
]
|
||||
}],
|
||||
volumes: [
|
||||
{name: "root-config", secret: {secretName: $root, defaultMode: 256}},
|
||||
{name: "backup-keys", secret: {secretName: $keys, defaultMode: 256}},
|
||||
{name: "policy", configMap: {name: $policy}},
|
||||
{name: "tmp", emptyDir: {}}
|
||||
]
|
||||
}
|
||||
}'
|
||||
)"
|
||||
|
||||
kubectl --namespace "$OS_NAMESPACE" run "$pod" \
|
||||
--image="$MC_IMAGE" \
|
||||
--restart=Never \
|
||||
--labels='platform.hyeonworks.com/aistor-client=true,app.kubernetes.io/name=pg-backup-bootstrap' \
|
||||
--overrides="$overrides" >/dev/null
|
||||
|
||||
phase=''
|
||||
for _ in $(seq 1 90); do
|
||||
phase="$(kubectl --namespace "$OS_NAMESPACE" get pod "$pod" --output=jsonpath='{.status.phase}')"
|
||||
[[ "$phase" == "Succeeded" || "$phase" == "Failed" ]] && break
|
||||
sleep 2
|
||||
done
|
||||
logs="$(kubectl --namespace "$OS_NAMESPACE" logs "$pod" 2>&1 || true)"
|
||||
if [[ "$phase" != "Succeeded" || "$logs" != *"$PASS_LINE"* ]]; then
|
||||
printf '%s\n' "$logs" | tail -20 >&2
|
||||
fail "bootstrap Pod ended in phase '${phase}'; the platform-data Secret was not created"
|
||||
fi
|
||||
|
||||
kubectl --namespace "$PG_NAMESPACE" create secret generic "$TARGET_SECRET" \
|
||||
--from-file=ACCESS_KEY_ID="${work}/ACCESS_KEY_ID" \
|
||||
--from-file=ACCESS_SECRET_KEY="${work}/ACCESS_SECRET_KEY" >/dev/null
|
||||
kubectl --namespace "$PG_NAMESPACE" label secret "$TARGET_SECRET" \
|
||||
app.kubernetes.io/part-of=platform app.kubernetes.io/component=platform-postgres-backup >/dev/null
|
||||
|
||||
printf '%s\n' "$PASS_LINE" \
|
||||
"Created bucket ${BUCKET}, policy ${POLICY}, one user and ${PG_NAMESPACE}/${TARGET_SECRET}."
|
||||
@@ -0,0 +1,78 @@
|
||||
#!/usr/bin/env bash
|
||||
# Install the host backup and check timers (infrastructure/host/backup).
|
||||
#
|
||||
# bash scripts/bootstrap/install-host-checks.sh --check # no root, changes nothing
|
||||
# sudo bash scripts/bootstrap/install-host-checks.sh --execute # install, enable, first runs
|
||||
#
|
||||
# Scripts are copied to /usr/local/lib/platform-backup (root-owned) so the root
|
||||
# timers never execute files the desktop user can edit.
|
||||
set -Eeuo pipefail
|
||||
umask 022
|
||||
|
||||
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
|
||||
readonly LIB=/usr/local/lib/platform-backup
|
||||
readonly UNIT_DIR=/etc/systemd/system
|
||||
readonly TEXTFILE_DIR=/var/lib/node_exporter/textfile_collector
|
||||
readonly WORK_ROOT=/var/lib/platform-backup/work
|
||||
readonly USER_NAME=donghyeon
|
||||
readonly -a SCRIPTS=(
|
||||
scripts/lib/host-backup.sh:lib/host-backup.sh
|
||||
scripts/backup/k3s-datastore-backup.sh:backup/k3s-datastore-backup.sh
|
||||
scripts/backup/gitea-backup.sh:backup/gitea-backup.sh
|
||||
scripts/validate/live-drift-check-main.sh:validate/live-drift-check-main.sh
|
||||
scripts/validate/k3s-cert-expiry.sh:validate/k3s-cert-expiry.sh
|
||||
services/platform-host-checks/README.md:README.md
|
||||
)
|
||||
readonly -a TIMERS=(platform-k3s-datastore-backup platform-gitea-backup platform-live-drift-check platform-k3s-cert-expiry)
|
||||
|
||||
fail() { printf 'ERROR: %s\n' "$*" >&2; exit 1; }
|
||||
mode="${1:-}"
|
||||
[[ "$#" -eq 1 && ( "$mode" == "--check" || "$mode" == "--execute" ) ]] || { sed -n '2,8p' "$0" >&2; exit 2; }
|
||||
|
||||
echo "== check"
|
||||
for pair in "${SCRIPTS[@]}"; do
|
||||
src="${pair%%:*}"
|
||||
[[ -f "$ROOT/$src" ]] || fail "missing $src"
|
||||
[[ "$src" != *.sh ]] || bash -n "$ROOT/$src" || fail "syntax error in $src"
|
||||
done
|
||||
for t in "${TIMERS[@]}"; do
|
||||
for u in service timer; do [[ -f "$ROOT/infrastructure/host/backup/$t.$u" ]] || fail "missing $t.$u"; done
|
||||
done
|
||||
if command -v systemd-analyze >/dev/null; then
|
||||
verify_dir="$(mktemp -d)"
|
||||
trap 'rm -rf -- "$verify_dir"' EXIT
|
||||
cp "$ROOT"/infrastructure/host/backup/*.service "$ROOT"/infrastructure/host/backup/*.timer "$verify_dir/"
|
||||
# ExecStart paths do not exist before installation; only syntax and keys are checked.
|
||||
out="$(cd "$verify_dir" && systemd-analyze verify ./*.timer ./*.service 2>&1 | grep -v -E 'is not executable|No such file|Command .* is not executable' || true)"
|
||||
[[ -z "$out" ]] || { printf '%s\n' "$out" >&2; fail "systemd-analyze verify reported problems"; }
|
||||
echo " units: systemd-analyze verify clean"
|
||||
fi
|
||||
echo "== plan"
|
||||
for pair in "${SCRIPTS[@]}"; do echo " install ${pair%%:*} -> $LIB/${pair##*:}"; done
|
||||
for t in "${TIMERS[@]}"; do echo " install $UNIT_DIR/$t.service, $UNIT_DIR/$t.timer; enable --now $t.timer"; done
|
||||
echo " mkdir $TEXTFILE_DIR (root:$USER_NAME 0775), $WORK_ROOT (root 0700)"
|
||||
echo " first runs: k3s-cert-expiry, k3s-datastore-backup, gitea-backup, live-drift-check"
|
||||
[[ "$mode" == "--execute" ]] || { echo "CHECK ONLY: nothing changed"; exit 0; }
|
||||
|
||||
[[ "$(id -u)" -eq 0 ]] || fail "--execute needs root (sudo)"
|
||||
[[ -f /etc/platform-backup/aistor.env ]] || fail "run scripts/bootstrap/create-host-backup-credentials.sh first"
|
||||
echo "== install"
|
||||
for pair in "${SCRIPTS[@]}"; do
|
||||
install -D -o root -g root -m 0755 "$ROOT/${pair%%:*}" "$LIB/${pair##*:}"
|
||||
done
|
||||
install -d -o root -g "$USER_NAME" -m 0775 "$TEXTFILE_DIR"
|
||||
install -d -o root -g root -m 0700 "$WORK_ROOT"
|
||||
for t in "${TIMERS[@]}"; do
|
||||
install -o root -g root -m 0644 "$ROOT/infrastructure/host/backup/$t.service" "$ROOT/infrastructure/host/backup/$t.timer" "$UNIT_DIR/"
|
||||
done
|
||||
systemctl daemon-reload
|
||||
for t in "${TIMERS[@]}"; do systemctl enable --now "$t.timer"; done
|
||||
|
||||
echo "== first runs"
|
||||
for t in platform-k3s-cert-expiry platform-k3s-datastore-backup platform-gitea-backup platform-live-drift-check; do
|
||||
if systemctl start "$t.service"; then echo " $t: ok"; else echo " $t: FAILED (journalctl -u $t)"; fi
|
||||
done
|
||||
echo "== metrics"
|
||||
grep -h -E '^platform_' "$TEXTFILE_DIR"/*.prom || echo " no metrics yet"
|
||||
systemctl list-timers 'platform-*' --no-pager
|
||||
echo "HOST CHECKS INSTALLED"
|
||||
@@ -0,0 +1,55 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared helpers for the host backup and host check scripts. Source only.
|
||||
# shellcheck shell=bash
|
||||
|
||||
readonly HOST_BACKUP_ENV=/etc/platform-backup/aistor.env
|
||||
readonly TEXTFILE_DIR=/var/lib/node_exporter/textfile_collector
|
||||
# Backup staging (root 0700, same SSD). Not /var/tmp: dockerd must see the path.
|
||||
readonly HB_WORK_ROOT=/var/lib/platform-backup/work
|
||||
# Same pinned AIStor client as scripts/validate/aistor-s3-smoke.sh.
|
||||
readonly MC_IMAGE="quay.io/minio/aistor/mc@sha256:c5ec777c080fd6292b7529309a0f311c58c3a92e268c9c57c0901da86881949b"
|
||||
|
||||
hb_log() { printf '%s %s\n' "$(date -u +%FT%TZ)" "$*"; }
|
||||
hb_fail() { hb_log "FAIL: $*" >&2; exit 1; }
|
||||
|
||||
# Write Prometheus textfile metrics atomically: hb_write_metrics NAME <<'PROM' ... PROM
|
||||
hb_write_metrics() {
|
||||
local name="$1" tmp
|
||||
[[ -d "$TEXTFILE_DIR" ]] || { hb_log "textfile dir $TEXTFILE_DIR missing; metrics not written"; cat >/dev/null; return 0; }
|
||||
tmp="$(mktemp "$TEXTFILE_DIR/.${name}.XXXXXX")"
|
||||
cat >"$tmp"
|
||||
chmod 0644 "$tmp"
|
||||
mv -f -- "$tmp" "$TEXTFILE_DIR/${name}.prom"
|
||||
}
|
||||
|
||||
# Load the AIStor endpoint and the backup user's keys (root-only file).
|
||||
hb_load_aistor_env() {
|
||||
[[ -f "$HOST_BACKUP_ENV" ]] || hb_fail "$HOST_BACKUP_ENV is missing; run scripts/bootstrap/create-host-backup-credentials.sh"
|
||||
[[ "$(stat -c '%u %a' "$HOST_BACKUP_ENV")" == "0 600" ]] || hb_fail "$HOST_BACKUP_ENV must be owned by root with mode 0600"
|
||||
# shellcheck disable=SC1090
|
||||
. "$HOST_BACKUP_ENV"
|
||||
: "${AISTOR_BUCKET:?}" "${AISTOR_ACCESS_KEY:?}" "${AISTOR_SECRET_KEY:?}"
|
||||
# The host cannot resolve cluster DNS, so use the Service ClusterIP (port 80 -> 9000).
|
||||
if [[ -z "${AISTOR_ENDPOINT:-}" ]]; then
|
||||
local ip
|
||||
ip="$(/usr/local/bin/k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml -n object-storage \
|
||||
get service minio -o jsonpath='{.spec.clusterIP}')"
|
||||
[[ -n "$ip" ]] || hb_fail "cannot find the object-storage/minio Service"
|
||||
AISTOR_ENDPOINT="http://${ip}:80"
|
||||
fi
|
||||
}
|
||||
|
||||
# Run the pinned mc client on the host network with the backup user's keys.
|
||||
# The keys go through a 0600 env file, never the command line (ps would show it).
|
||||
# Usage: hb_mc <dir mounted at /work> <mc args...>
|
||||
hb_mc() {
|
||||
local work="$1"; shift
|
||||
local envfile="$work/.mc.env"
|
||||
if [[ ! -f "$envfile" ]]; then
|
||||
( umask 077; printf 'MC_HOST_backup=%s\nMC_CONFIG_DIR=/tmp/mc\n' \
|
||||
"${AISTOR_ENDPOINT/:\/\//://${AISTOR_ACCESS_KEY}:${AISTOR_SECRET_KEY}@}" >"$envfile" )
|
||||
fi
|
||||
docker run --rm --network host --user "$(id -u):$(id -g)" \
|
||||
--env-file "$envfile" -v "$work:/work" --tmpfs /tmp \
|
||||
"$MC_IMAGE" "$@"
|
||||
}
|
||||
@@ -596,3 +596,13 @@ PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm \
|
||||
- `--textfile FILE`: `platform_live_drift_objects`, `platform_live_drift_check_success`,
|
||||
`platform_live_drift_last_run_timestamp_seconds`를 Prometheus textfile 형식으로 쓴다.
|
||||
- Secret 값의 차이는 보지 않는다.
|
||||
|
||||
## 호스트 백업·점검 시험
|
||||
|
||||
- `test-platform-host-checks-alerts.sh`: `services/platform-host-checks/prometheus-rules.yaml`의 promtool
|
||||
단위 시험(백업 신선도, 실패, 지표 없음, drift, k3s 인증서). docker 필요, 클러스터 접근 없음.
|
||||
- `k3s-cert-expiry.sh`: 6443·10250 serving 인증서 만료를 textfile 지표로 쓴다(타이머가 매일 돌림).
|
||||
- `live-drift-check-main.sh`: checkout의 작업 트리가 아니라 `origin/main`으로 `live-drift-check.sh`를 돌린다.
|
||||
- `k3s-datastore-restore-drill.sh`, `gitea-backup-restore-drill.sh`: 복원 리허설. 인자 없이 root로 돌리면 AIStor의
|
||||
최신 백업을 쓰고, `--from-dir`·`--from-file`로 로컬 사본을 줄 수 있다.
|
||||
- `test-k3s-datastore-restore-drill.sh`: 운영 데이터 없이 k3s 복원 리허설 경로를 시험한다(일회용 k3s, 시험 Secret 셋).
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
#!/usr/bin/env bash
|
||||
# Restore drill for the Gitea backup (file level).
|
||||
#
|
||||
# sudo bash scripts/validate/gitea-backup-restore-drill.sh # latest dump in AIStor
|
||||
# bash scripts/validate/gitea-backup-restore-drill.sh --from-file DUMP.tar.gz
|
||||
#
|
||||
# Extracts the dump into a throwaway directory and checks what a restore needs:
|
||||
# app.ini is present, every repository passes `git fsck`, and the repository
|
||||
# count matches the live Gitea. The database is restored from the CloudNativePG
|
||||
# backup instead (services/platform-postgres-backup); the full restore order is
|
||||
# in docs/runbooks/gitea-restore.md. Nothing is written to the cluster.
|
||||
set -Eeuo pipefail
|
||||
umask 077
|
||||
|
||||
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
|
||||
# shellcheck source=../lib/host-backup.sh
|
||||
source "$ROOT/scripts/lib/host-backup.sh"
|
||||
readonly NAME=gitea_restore_drill
|
||||
|
||||
from_file=""
|
||||
case "${1:-}" in
|
||||
--from-file) from_file="${2:?--from-file needs a path}" ;;
|
||||
"") ;;
|
||||
*) sed -n '2,11p' "$0" >&2; exit 2 ;;
|
||||
esac
|
||||
if [[ "$(id -u)" -eq 0 ]]; then
|
||||
kubectl() { command /usr/local/bin/k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml "$@"; }
|
||||
fi
|
||||
|
||||
base="$HB_WORK_ROOT"; [[ -w "$base" ]] || base="$(mktemp -d)"
|
||||
work="$(mktemp -d "$base/gitea-drill.XXXXXX")"
|
||||
trap 'rm -rf -- "$work"' EXIT
|
||||
|
||||
if [[ -z "$from_file" ]]; then
|
||||
[[ "$(id -u)" -eq 0 ]] || hb_fail "run as root to read the backup credentials"
|
||||
hb_load_aistor_env
|
||||
latest="$(hb_mc "$work" ls "backup/$AISTOR_BUCKET/gitea/" | awk '{print $NF}' | grep -E '^gitea-dump-.*\.tar\.gz$' | sort | tail -1)"
|
||||
[[ -n "$latest" ]] || hb_fail "no Gitea dump found"
|
||||
hb_mc "$work" cp --quiet "backup/$AISTOR_BUCKET/gitea/$latest" "backup/$AISTOR_BUCKET/gitea/$latest.sha256" /work/ >/dev/null
|
||||
[[ "$(sha256sum "$work/$latest" | awk '{print $1}')" == "$(cat "$work/$latest.sha256")" ]] || hb_fail "checksum mismatch for $latest"
|
||||
from_file="$work/$latest"
|
||||
hb_log "drill on $latest"
|
||||
fi
|
||||
|
||||
mkdir "$work/restore"
|
||||
tar -C "$work/restore" -xzf "$from_file"
|
||||
[[ -s "$work/restore/app.ini" ]] || hb_fail "app.ini is missing from the dump"
|
||||
for key in INTERNAL_TOKEN SECRET_KEY; do
|
||||
grep -q "^${key}[[:space:]]*=" "$work/restore/app.ini" || hb_fail "app.ini has no ${key} (needed to restore)"
|
||||
done
|
||||
|
||||
total=0 broken=0
|
||||
while IFS= read -r -d '' repo; do
|
||||
total=$((total + 1))
|
||||
if ! git -c safe.directory='*' --git-dir="$repo" fsck --no-dangling --no-progress >/dev/null 2>"$work/fsck.err"; then
|
||||
broken=$((broken + 1))
|
||||
hb_log "fsck failed: ${repo#"$work/restore/"}: $(head -1 "$work/fsck.err")"
|
||||
fi
|
||||
done < <(find "$work/restore/repos" -mindepth 2 -maxdepth 2 -type d -name '*.git' -print0)
|
||||
hb_log "repositories in dump: $total, fsck failures: $broken"
|
||||
(( total > 0 )) || hb_fail "the dump has no repositories"
|
||||
(( broken == 0 )) || hb_fail "$broken repositories failed git fsck"
|
||||
|
||||
live="$(kubectl -n gitea exec deploy/gitea -c gitea -- sh -c 'find /data/git/gitea-repositories -mindepth 2 -maxdepth 2 -name "*.git" -type d | wc -l' 2>/dev/null || echo "")"
|
||||
if [[ -n "$live" ]]; then
|
||||
hb_log "live Gitea repositories: $live"
|
||||
(( total >= live - 2 )) || hb_fail "the dump has $total repositories, live Gitea has $live"
|
||||
fi
|
||||
|
||||
hb_write_metrics "$NAME" <<PROM
|
||||
# HELP platform_backup_restore_drill_last_success_timestamp_seconds Unix time of the last successful restore drill.
|
||||
# TYPE platform_backup_restore_drill_last_success_timestamp_seconds gauge
|
||||
platform_backup_restore_drill_last_success_timestamp_seconds{backup="gitea"} $(date +%s)
|
||||
PROM
|
||||
echo "GITEA RESTORE DRILL PASS"
|
||||
@@ -0,0 +1,25 @@
|
||||
#!/usr/bin/env bash
|
||||
# Write the notAfter time of the k3s API server (6443) and kubelet (10250)
|
||||
# serving certificates as Prometheus textfile metrics. No root needed.
|
||||
# k3s renews these only when it starts and they are within 120 days of expiry
|
||||
# (docs.k3s.io cli/certificate), and the kubelet exposes no certificate metric,
|
||||
# so nothing else in the cluster watches them.
|
||||
set -Eeuo pipefail
|
||||
# shellcheck source=../lib/host-backup.sh
|
||||
source "$(dirname -- "${BASH_SOURCE[0]}")/../lib/host-backup.sh"
|
||||
|
||||
lines=""
|
||||
for target in apiserver:6443 kubelet:10250; do
|
||||
name="${target%%:*}" port="${target##*:}"
|
||||
end="$(echo | timeout 10 openssl s_client -connect "127.0.0.1:$port" 2>/dev/null | openssl x509 -noout -enddate 2>/dev/null | sed 's/^notAfter=//')" || true
|
||||
[[ -n "$end" ]] || hb_fail "could not read the $name certificate on 127.0.0.1:$port"
|
||||
lines+="platform_k3s_certificate_not_after_seconds{certificate=\"$name\"} $(date -d "$end" +%s)"$'\n'
|
||||
hb_log "$name certificate notAfter $end"
|
||||
done
|
||||
hb_write_metrics platform_k3s_certificates <<PROM
|
||||
# HELP platform_k3s_certificate_not_after_seconds notAfter of a k3s serving certificate (Unix time).
|
||||
# TYPE platform_k3s_certificate_not_after_seconds gauge
|
||||
${lines}# HELP platform_k3s_certificate_check_timestamp_seconds Unix time of the last certificate check.
|
||||
# TYPE platform_k3s_certificate_check_timestamp_seconds gauge
|
||||
platform_k3s_certificate_check_timestamp_seconds $(date +%s)
|
||||
PROM
|
||||
@@ -0,0 +1,99 @@
|
||||
#!/usr/bin/env bash
|
||||
# Restore drill for the k3s datastore backup.
|
||||
#
|
||||
# sudo bash scripts/validate/k3s-datastore-restore-drill.sh # latest backup in AIStor
|
||||
# bash scripts/validate/k3s-datastore-restore-drill.sh --from-dir DIR [--expect-secrets N]
|
||||
# DIR holds state.db and token (an extracted archive, or a test datastore)
|
||||
#
|
||||
# Starts a throwaway k3s server (same version as production, --network none,
|
||||
# no agent) on a copy of the backup, and checks that the API answers, that the
|
||||
# Secret count matches and that every Secret decrypts (the token unlocks the
|
||||
# encryption config stored in the datastore). Prints no Secret value. The
|
||||
# container and the copy are removed at the end.
|
||||
set -Eeuo pipefail
|
||||
umask 077
|
||||
|
||||
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
|
||||
# shellcheck source=../lib/host-backup.sh
|
||||
source "$ROOT/scripts/lib/host-backup.sh"
|
||||
# Same version as the desktop server (k3s --version: v1.36.2+k3s1).
|
||||
readonly K3S_IMAGE="docker.io/rancher/k3s:v1.36.2-k3s1@sha256:6a47cea22c4b834d4ba72c89d291696b79ebe406251f90b446e4dff03513dd87"
|
||||
readonly NAME=k3s_datastore_restore_drill
|
||||
|
||||
from_dir="" expect_secrets=""
|
||||
while (( $# > 0 )); do
|
||||
case "$1" in
|
||||
--from-dir) from_dir="${2:?}"; shift 2 ;;
|
||||
--expect-secrets) expect_secrets="${2:?}"; shift 2 ;;
|
||||
*) sed -n '2,13p' "$0" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
base="${HB_WORK_ROOT}"
|
||||
[[ -w "$base" ]] || base="$(mktemp -d)"
|
||||
work="$(mktemp -d "$base/k3s-drill.XXXXXX")"
|
||||
container="k3s-restore-drill-$$"
|
||||
cleanup() {
|
||||
docker rm -f "$container" >/dev/null 2>&1 || true
|
||||
# The drill container writes as root; remove its files with the same image.
|
||||
docker run --rm -v "$work:/w" --entrypoint /bin/sh "$K3S_IMAGE" -c 'rm -rf /w/*' >/dev/null 2>&1 || true
|
||||
rm -rf -- "$work"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
if [[ -z "$from_dir" ]]; then
|
||||
[[ "$(id -u)" -eq 0 ]] || hb_fail "run as root to read the backup credentials"
|
||||
hb_load_aistor_env
|
||||
latest="$(hb_mc "$work" ls "backup/$AISTOR_BUCKET/k3s-datastore/" | awk '{print $NF}' | grep -E '^k3s-datastore-.*\.tar\.gz$' | sort | tail -1)"
|
||||
[[ -n "$latest" ]] || hb_fail "no k3s datastore backup found"
|
||||
hb_mc "$work" cp --quiet "backup/$AISTOR_BUCKET/k3s-datastore/$latest" "backup/$AISTOR_BUCKET/k3s-datastore/$latest.sha256" /work/ >/dev/null
|
||||
[[ "$(sha256sum "$work/$latest" | awk '{print $1}')" == "$(cat "$work/$latest.sha256")" ]] || hb_fail "checksum mismatch for $latest"
|
||||
tar -C "$work" -xzf "$work/$latest"
|
||||
from_dir="$(find "$work" -mindepth 1 -maxdepth 1 -type d -name 'k3s-datastore-*' | head -1)"
|
||||
hb_log "drill on $latest"
|
||||
if [[ -z "$expect_secrets" ]]; then
|
||||
expect_secrets="$(/usr/local/bin/k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get secrets -A --no-headers | wc -l)"
|
||||
hb_log "live cluster has $expect_secrets Secrets (counts may differ by objects created since the backup)"
|
||||
fi
|
||||
fi
|
||||
[[ -f "$from_dir/state.db" && -f "$from_dir/token" ]] || hb_fail "$from_dir must contain state.db and token"
|
||||
|
||||
install -d -m 0700 "$work/server/db"
|
||||
cp "$from_dir/state.db" "$work/server/db/state.db"
|
||||
cp "$from_dir/token" "$work/server/token"
|
||||
[[ "$(sqlite3 "$work/server/db/state.db" 'PRAGMA integrity_check;')" == "ok" ]] || hb_fail "integrity_check failed"
|
||||
|
||||
started="$(date +%s)"
|
||||
docker run -d --name "$container" --network none --tmpfs /run --tmpfs /var/run \
|
||||
-v "$work/server:/var/lib/rancher/k3s/server" \
|
||||
"$K3S_IMAGE" server --disable-agent --secrets-encryption \
|
||||
--token-file /var/lib/rancher/k3s/server/token \
|
||||
--bind-address 127.0.0.1 --advertise-address 127.0.0.1 --node-ip 127.0.0.1 \
|
||||
--disable traefik,servicelb,metrics-server,local-storage,coredns >/dev/null
|
||||
|
||||
ready=0
|
||||
for _ in $(seq 1 60); do
|
||||
if docker exec "$container" kubectl get --raw /readyz >/dev/null 2>&1; then ready=1; break; fi
|
||||
sleep 5
|
||||
done
|
||||
if (( ready == 0 )); then
|
||||
docker logs --tail 20 "$container" 2>&1 | sed 's/^/ k3s: /' >&2
|
||||
hb_fail "restored k3s did not become ready in 5 minutes"
|
||||
fi
|
||||
namespaces="$(docker exec "$container" kubectl get namespaces --no-headers | wc -l)"
|
||||
secrets="$(docker exec "$container" kubectl get secrets -A --no-headers | wc -l)"
|
||||
# -o json decrypts every Secret on the server side; any failure makes kubectl exit non-zero.
|
||||
decrypted="$(docker exec "$container" kubectl get secrets -A -o json | python3 -I -c 'import json,sys; d=json.load(sys.stdin)["items"]; print(sum(1 for s in d if s.get("data")))')"
|
||||
hb_log "restored: namespaces=$namespaces secrets=$secrets with-data=$decrypted in $(( $(date +%s) - started ))s"
|
||||
if [[ -n "$expect_secrets" ]]; then
|
||||
diff=$(( secrets - expect_secrets ))
|
||||
(( ${diff#-} <= 5 )) || hb_fail "Secret count $secrets is far from the expected $expect_secrets"
|
||||
fi
|
||||
(( decrypted > 0 )) || hb_fail "no Secret carried data after restore"
|
||||
|
||||
hb_write_metrics "$NAME" <<PROM
|
||||
# HELP platform_backup_restore_drill_last_success_timestamp_seconds Unix time of the last successful restore drill.
|
||||
# TYPE platform_backup_restore_drill_last_success_timestamp_seconds gauge
|
||||
platform_backup_restore_drill_last_success_timestamp_seconds{backup="k3s-datastore"} $(date +%s)
|
||||
PROM
|
||||
echo "K3S DATASTORE RESTORE DRILL PASS"
|
||||
@@ -3,8 +3,13 @@
|
||||
Cluster platform-data platform-postgres 2026-11-30 pg_hba의 techlog 두 줄이 repo에 없다. w1이 CNPG 백업과 함께 넣는다(1c4ef9a).
|
||||
DatabaseRole platform-data platform-postgres-gitea 2026-11-30 렌더가 false 기본값 필드를 명시할 뿐 값은 같다. 다음 Phase 1 적용 때 사라진다.
|
||||
DatabaseRole platform-data platform-postgres-keycloak 2026-11-30 렌더가 false 기본값 필드를 명시할 뿐 값은 같다. 다음 Phase 2 적용 때 사라진다.
|
||||
Deployment keycloak keycloak-operator 2026-11-30 원격 ref 26.7.0 태그 내용이 바뀌어 build-timestamp 라벨만 다르다. 재시작 창에 커밋 SHA 고정과 함께 적용한다.
|
||||
Deployment keycloak keycloak-operator 2026-11-30 커밋 SHA·이미지 digest·IfNotPresent 고정(재시작). 재시작 창에 apply-keycloak.sh로 적용한다.
|
||||
Service keycloak keycloak-operator 2026-11-30 위와 같은 build-timestamp 라벨.
|
||||
ServiceAccount keycloak keycloak-operator 2026-11-30 위와 같은 build-timestamp 라벨.
|
||||
Deployment observability observability-core-kube-pr-operator 2026-11-30 이미지 문자열이 tag@digest와 digest로 다르고 digest는 같다. 적용하면 재시작되므로 재시작 창에.
|
||||
Deployment observability observability-core-kube-state-metrics 2026-11-30 위와 같은 이미지 문자열 차이.
|
||||
Keycloak keycloak keycloak 2026-11-30 이미지 digest·startOptimized false·platform-critical(Keycloak 재시작). 재시작 창에 PriorityClass를 먼저 만든 뒤 적용한다.
|
||||
Deployment gitea gitea 2026-11-30 priorityClassName platform-critical(Gitea 재시작). 재시작 창에 apply-gitea-oidc.sh로 적용한다.
|
||||
StatefulSet observability loki 2026-11-30 priorityClassName platform-low(Loki 재시작). 재시작 창에 관측 core 적용으로.
|
||||
StatefulSet observability tempo 2026-11-30 priorityClassName platform-low(Tempo 재시작). 재시작 창에 관측 core 적용으로.
|
||||
PrometheusRule observability observability-core-kube-pr-kubernetes-system-kubelet 2026-11-30 Kubelet 인증서 경보 둘을 values에서 껐다(PR #6). 관측 core를 다시 적용하면 사라진다(재시작 창 2단계).
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
#!/usr/bin/env bash
|
||||
# Daily drift check against origin/main (not the working tree of the checkout),
|
||||
# so local edits or a stale checkout do not hide or invent drift.
|
||||
# Run by platform-live-drift-check.timer as the desktop user.
|
||||
set -Eeuo pipefail
|
||||
readonly CHECKOUT="${PLATFORM_CHECKOUT:-/home/donghyeon/workspace/platform}"
|
||||
readonly TEXTFILE=/var/lib/node_exporter/textfile_collector/platform_live_drift.prom
|
||||
|
||||
git -C "$CHECKOUT" fetch --quiet origin main
|
||||
work="$(mktemp -d /tmp/platform-drift-main.XXXXXX)"
|
||||
trap 'rm -rf -- "$work"' EXIT
|
||||
git -C "$CHECKOUT" archive origin/main | tar -x -C "$work"
|
||||
printf 'origin/main %s\n' "$(git -C "$CHECKOUT" rev-parse --short origin/main)"
|
||||
args=(--accepted "$work/scripts/validate/live-drift-accepted.txt")
|
||||
[[ -d "$(dirname "$TEXTFILE")" ]] && args+=(--textfile "$TEXTFILE")
|
||||
bash "$work/scripts/validate/live-drift-check.sh" "${args[@]}"
|
||||
@@ -64,10 +64,29 @@ die() {
|
||||
work="$(mktemp -d /tmp/platform-drift.XXXXXX)"; made_dirs+=("$work")
|
||||
mkdir -p "$work/in" "$work/out"
|
||||
|
||||
# Render from a clean copy when run inside a git working tree. The observability
|
||||
# renderer refuses to run when an ignored chart cache (.helm/) is left over from an
|
||||
# earlier render, which happens in long-lived checkouts. The copy keeps tracked and
|
||||
# untracked-but-not-ignored files, so local edits are still checked.
|
||||
SOURCE_ROOT="$REPOSITORY_ROOT"
|
||||
if git -C "$REPOSITORY_ROOT" rev-parse --is-inside-work-tree >/dev/null 2>&1; then
|
||||
SOURCE_ROOT="$work/src"
|
||||
mkdir -p "$SOURCE_ROOT"
|
||||
python3 -I - "$REPOSITORY_ROOT" <<'PY' | tar -C "$REPOSITORY_ROOT" --null -T - -cf - | tar -x -C "$SOURCE_ROOT" || die "could not copy the working tree"
|
||||
import subprocess, sys
|
||||
root = sys.argv[1]
|
||||
def ls(*args):
|
||||
out = subprocess.run(["git", "-C", root, "ls-files", "-z", *args], check=True, capture_output=True).stdout
|
||||
return [p for p in out.decode().split("\0") if p]
|
||||
keep = set(ls("--cached", "--others", "--exclude-standard")) - set(ls("--deleted"))
|
||||
sys.stdout.write("\0".join(sorted(keep)) + "\0")
|
||||
PY
|
||||
fi
|
||||
|
||||
render_root() {
|
||||
local script="$1" prefix="$2" dir
|
||||
dir="$(mktemp -d "/tmp/${prefix}.XXXXXX")"; made_dirs+=("$dir")
|
||||
bash "$REPOSITORY_ROOT/scripts/validate/${script}" --verified-output-dir "$dir" \
|
||||
bash "$SOURCE_ROOT/scripts/validate/${script}" --verified-output-dir "$dir" \
|
||||
>"$work/${script}.log" 2>&1 || { tail -5 "$work/${script}.log" >&2; die "${script} failed"; }
|
||||
printf '%s\n' "$dir"
|
||||
}
|
||||
@@ -86,10 +105,10 @@ declare -a inputs=(
|
||||
for f in "$phase2"/*.yaml "$admin"/*.yaml; do inputs+=("$f"); done
|
||||
|
||||
# Live Traefik is the trust overlay (infrastructure/networking/traefik/README.md).
|
||||
kubectl kustomize "$REPOSITORY_ROOT/infrastructure/networking/traefik/overlays/trust" >"$work/traefik-trust.yaml" || die "traefik render failed"
|
||||
kubectl kustomize "$REPOSITORY_ROOT/services/tenants/techlog" >"$work/tenant-techlog.yaml" || die "tenant render failed"
|
||||
kubectl kustomize "$REPOSITORY_ROOT/services/priority-classes" >"$work/priority-classes.yaml" || die "priority class render failed"
|
||||
kubectl kustomize "$REPOSITORY_ROOT/infrastructure/networking/keycloak" >"$work/keycloak-network-policies.yaml" || die "keycloak policy render failed"
|
||||
kubectl kustomize "$SOURCE_ROOT/infrastructure/networking/traefik/overlays/trust" >"$work/traefik-trust.yaml" || die "traefik render failed"
|
||||
kubectl kustomize "$SOURCE_ROOT/services/tenants/techlog" >"$work/tenant-techlog.yaml" || die "tenant render failed"
|
||||
kubectl kustomize "$SOURCE_ROOT/services/priority-classes" >"$work/priority-classes.yaml" || die "priority class render failed"
|
||||
kubectl kustomize "$SOURCE_ROOT/infrastructure/networking/keycloak" >"$work/keycloak-network-policies.yaml" || die "keycloak policy render failed"
|
||||
inputs+=("$work/traefik-trust.yaml" "$work/tenant-techlog.yaml" "$work/priority-classes.yaml" "$work/keycloak-network-policies.yaml")
|
||||
|
||||
n=0
|
||||
|
||||
@@ -412,9 +412,9 @@ elif mode == "alertmanager_routing":
|
||||
if item.get("kind") == "AlertmanagerConfig"
|
||||
and (item.get("metadata") or {}).get("name") == "platform-alertmanager"
|
||||
]
|
||||
expected_config = json.loads(r'''{"apiVersion":"monitoring.coreos.com/v1alpha1","kind":"AlertmanagerConfig","metadata":{"name":"platform-alertmanager","namespace":"observability","labels":{"observability.hyeonworks.com/instance":"home"}},"spec":{"route":{"receiver":"platform-slack","groupBy":["cluster","namespace","alertname","severity"],"groupWait":"30s","groupInterval":"5m","repeatInterval":"4h","routes":[{"receiver":"platform-null","matchers":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}]}]},"inhibitRules":[{"sourceMatch":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}],"targetMatch":[{"name":"severity","matchType":"=","value":"info"}],"equal":["namespace"]}],"receivers":[{"name":"platform-null"},{"name":"platform-slack","slackConfigs":[{"apiURL":{"name":"alertmanager-slack-webhook","key":"url"},"sendResolved":true,"linkNames":false,"mrkdwnIn":["text","fields"],"fallback":"{{ if eq .Status \"firing\" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","title":"{{ if eq .Status \"firing\" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","titleLink":"https://grafana.learn.hyeonworks.com/","color":"{{ if eq .Status \"resolved\" }}good{{ else if or (eq .CommonLabels.severity \"emergency\") (eq .CommonLabels.severity \"critical\") }}danger{{ else if eq .CommonLabels.severity \"warning\" }}warning{{ else }}#439FE0{{ end }}","fields":[{"title":"Status","value":"{{ .Status | toUpper }}","short":true},{"title":"Severity","value":"{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}","short":true},{"title":"Location","value":"{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}","short":true},{"title":"Alert count","value":"{{ len .Alerts }}","short":true}],"text":"{{ range .Alerts }}\n*Alert status:* {{ .Status | toUpper }}\n*Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }}\n*Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }}\n*Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }}\n*Started:* {{ .StartsAt.Format \"2006-01-02T15:04:05Z07:00\" }}\n{{ if eq .Status \"resolved\" }}*Ended:* {{ .EndsAt.Format \"2006-01-02T15:04:05Z07:00\" }}{{ end }}\n{{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }}\n\n{{ end }}\n<https://grafana.learn.hyeonworks.com/|Grafana> · <https://grafana.learn.hyeonworks.com/explore|Explore>","footer":"hyeonworks observability · Alertmanager"}]}]}}''')
|
||||
expected_config = json.loads(r'''{"apiVersion":"monitoring.coreos.com/v1alpha1","kind":"AlertmanagerConfig","metadata":{"name":"platform-alertmanager","namespace":"observability","labels":{"observability.hyeonworks.com/instance":"home"}},"spec":{"route":{"receiver":"platform-slack","groupBy":["cluster","namespace","alertname","severity"],"groupWait":"30s","groupInterval":"5m","repeatInterval":"4h","routes":[{"receiver":"platform-null","matchers":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}]},{"receiver":"platform-heartbeat","matchers":[{"name":"alertname","matchType":"=","value":"PlatformHeartbeat"}],"groupWait":"0s","groupInterval":"1m","repeatInterval":"5m"}]},"inhibitRules":[{"sourceMatch":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}],"targetMatch":[{"name":"severity","matchType":"=","value":"info"}],"equal":["namespace"]}],"receivers":[{"name":"platform-null"},{"name":"platform-heartbeat","webhookConfigs":[{"urlSecret":{"name":"alertmanager-heartbeat","key":"url"},"sendResolved":false,"maxAlerts":1}]},{"name":"platform-slack","slackConfigs":[{"apiURL":{"name":"alertmanager-slack-webhook","key":"url"},"sendResolved":true,"linkNames":false,"mrkdwnIn":["text","fields"],"fallback":"{{ if eq .Status \"firing\" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","title":"{{ if eq .Status \"firing\" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","titleLink":"https://grafana.learn.hyeonworks.com/","color":"{{ if eq .Status \"resolved\" }}good{{ else if or (eq .CommonLabels.severity \"emergency\") (eq .CommonLabels.severity \"critical\") }}danger{{ else if eq .CommonLabels.severity \"warning\" }}warning{{ else }}#439FE0{{ end }}","fields":[{"title":"Status","value":"{{ .Status | toUpper }}","short":true},{"title":"Severity","value":"{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}","short":true},{"title":"Location","value":"{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}","short":true},{"title":"Alert count","value":"{{ len .Alerts }}","short":true}],"text":"{{ range .Alerts }}\n*Alert status:* {{ .Status | toUpper }}\n*Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }}\n*Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }}\n*Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }}\n*Started:* {{ .StartsAt.Format \"2006-01-02T15:04:05Z07:00\" }}\n{{ if eq .Status \"resolved\" }}*Ended:* {{ .EndsAt.Format \"2006-01-02T15:04:05Z07:00\" }}{{ end }}\n{{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }}\n\n{{ end }}\n<https://grafana.learn.hyeonworks.com/|Grafana> · <https://grafana.learn.hyeonworks.com/explore|Explore>","footer":"hyeonworks observability · Alertmanager"}]}]}}''')
|
||||
if configs != [expected_config]:
|
||||
reject("platform AlertmanagerConfig is not the exact global Slack contract")
|
||||
reject("platform AlertmanagerConfig is not the exact global Slack and heartbeat contract")
|
||||
|
||||
policies = [
|
||||
item
|
||||
|
||||
@@ -402,6 +402,10 @@ fi
|
||||
fail "the Keycloak role must authenticate only to the Keycloak database"
|
||||
[[ "$(rg --count -- '^[[:space:]]*-[[:space:]]host all keycloak all reject[[:space:]]*$' "${render_dir}/platform-postgres.yaml" || true)" == "1" ]] || \
|
||||
fail "the Keycloak role must be rejected from every other database"
|
||||
[[ "$(rg --count -- '^[[:space:]]*-[[:space:]]host techlog techlog all scram-sha-256[[:space:]]*$' "${render_dir}/platform-postgres.yaml" || true)" == "1" ]] || \
|
||||
fail "the tech-log role must authenticate only to the tech-log database"
|
||||
[[ "$(rg --count -- '^[[:space:]]*-[[:space:]]host all techlog all reject[[:space:]]*$' "${render_dir}/platform-postgres.yaml" || true)" == "1" ]] || \
|
||||
fail "the tech-log role must be rejected from every other database"
|
||||
|
||||
|
||||
rg --quiet '^kind: CustomResourceDefinition$' "${render_dir}/cnpg-operator.yaml" || \
|
||||
|
||||
@@ -266,7 +266,8 @@ aistor_objectstore_kustomization="${REPOSITORY_ROOT}/services/minio-aistor/base/
|
||||
aistor_operator_values="${REPOSITORY_ROOT}/infrastructure/controllers/aistor-operator/values/home.yaml"
|
||||
aistor_objectstore_values="${REPOSITORY_ROOT}/services/minio-aistor/base/deployment/values/home.yaml"
|
||||
|
||||
assert_regex_count "$keycloak_operator_kustomization" '^[[:space:]]*-[[:space:]]github\.com/keycloak/keycloak-k8s-resources/kubernetes\?ref=26\.7\.0[[:space:]]*$' 1 "Keycloak Operator remote ref must be 26.7.0"
|
||||
assert_regex_count "$keycloak_operator_kustomization" '^[[:space:]]*-[[:space:]]github\.com/keycloak/keycloak-k8s-resources/kubernetes\?ref=38fd7a0c84e15d4b66b37d1c2ea728ca2f5416a4[[:space:]]*$' 1 "Keycloak Operator remote ref must be the 26.7.0 commit"
|
||||
assert_regex_count "$keycloak_operator_kustomization" '^[[:space:]]*digest:[[:space:]]sha256:257ee324da2c701b7fc1ed1b2ce0e03979d1bf149952383bdcf6be904ae5b4f3[[:space:]]*$' 1 "Keycloak Operator image must be pinned by digest"
|
||||
assert_regex_count "$keycloak_operator_kustomization" 'github\.com/keycloak/keycloak-k8s-resources/kubernetes' 1 "exactly one Keycloak Operator remote resource is allowed"
|
||||
assert_regex_count "$keycloak_operator_kustomization" '\?ref=' 1 "exactly one pinned remote ref is allowed in the Keycloak Operator root"
|
||||
|
||||
@@ -342,6 +343,9 @@ assert_regex_count "${render_dir}/keycloak.yaml" '^kind: Ingress$' 1 "Keycloak I
|
||||
assert_regex_count "${render_dir}/minio-aistor.yaml" '^kind: ObjectStore$' 1 "AIStor ObjectStore count"
|
||||
|
||||
assert_regex_count "${render_dir}/keycloak.yaml" '^[[:space:]]*instances:[[:space:]]1[[:space:]]*$' 1 "Keycloak instance count field"
|
||||
assert_regex_count "${render_dir}/keycloak.yaml" '^[[:space:]]*image:[[:space:]]quay\.io/keycloak/keycloak@sha256:[0-9a-f]{64}[[:space:]]*$' 1 "Keycloak server image must be pinned by digest"
|
||||
assert_regex_count "${render_dir}/keycloak.yaml" '^[[:space:]]*startOptimized:[[:space:]]false[[:space:]]*$' 1 "the stock Keycloak image must not start optimized"
|
||||
assert_regex_count "${render_dir}/keycloak.yaml" '^[[:space:]]*priorityClassName:[[:space:]]platform-critical[[:space:]]*$' 1 "Keycloak priority class"
|
||||
assert_regex_count "${render_dir}/keycloak.yaml" '^[[:space:]]*hostname:[[:space:]]https://id\.learn\.hyeonworks\.com[[:space:]]*$' 1 "Keycloak external hostname"
|
||||
assert_regex_count "${render_dir}/keycloak.yaml" '^[[:space:]]*strict:[[:space:]]true[[:space:]]*$' 1 "Keycloak strict hostname mode"
|
||||
assert_regex_count "${render_dir}/keycloak.yaml" '^[[:space:]]*httpEnabled:[[:space:]]true[[:space:]]*$' 1 "Keycloak internal HTTP mode"
|
||||
|
||||
@@ -2,9 +2,13 @@
|
||||
|
||||
set -o pipefail
|
||||
|
||||
readonly PLATFORM_SENSITIVE_SOURCE_EXACT_PLATFORM_ROOT=/home/donghyeon/workspace/platform
|
||||
readonly PLATFORM_SENSITIVE_SOURCE_EXACT_DOCS_ROOT=/home/donghyeon/workspace/docs/platform
|
||||
readonly PLATFORM_SENSITIVE_SOURCE_EXACT_SCANNER="$PLATFORM_SENSITIVE_SOURCE_EXACT_PLATFORM_ROOT/scripts/validate/scan-platform-sensitive-source.sh"
|
||||
# The platform root is the checkout this scanner lives in, so a temporary copy
|
||||
# or a CI runner scans itself. The central documentation root stays on the
|
||||
# desktop host unless --docs-root or --platform-only says otherwise.
|
||||
PLATFORM_SENSITIVE_SOURCE_SELF_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
|
||||
readonly PLATFORM_SENSITIVE_SOURCE_SELF_ROOT
|
||||
readonly PLATFORM_SENSITIVE_SOURCE_SELF="$PLATFORM_SENSITIVE_SOURCE_SELF_ROOT/scripts/validate/scan-platform-sensitive-source.sh"
|
||||
readonly PLATFORM_SENSITIVE_SOURCE_DEFAULT_DOCS_ROOT=/home/donghyeon/workspace/docs/platform
|
||||
|
||||
_sensitive_source_fail() {
|
||||
printf 'ERROR: %s\n' "$*" >&2
|
||||
@@ -14,17 +18,22 @@ _sensitive_source_fail() {
|
||||
_sensitive_source_usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
bash scripts/validate/scan-platform-sensitive-source.sh
|
||||
bash scripts/validate/scan-platform-sensitive-source.sh [--docs-root DIR | --platform-only]
|
||||
|
||||
Scans the complete platform source and central platform documentation trees.
|
||||
Only .git and .helm directory contents, plus the exact platform-root
|
||||
.superpowers/sdd scratch subtree, are excluded. Diagnostics contain file names
|
||||
and finding classes only; matched values and lines are never printed.
|
||||
Scans the platform checkout that contains this script and the central platform
|
||||
documentation tree (default /home/donghyeon/workspace/docs/platform, or
|
||||
--docs-root DIR). --platform-only skips the documentation tree, for a
|
||||
temporary copy or a CI runner that has none. Only .git and .helm directory
|
||||
contents, plus the platform-root .superpowers/sdd scratch subtree, are
|
||||
excluded. Diagnostics contain file names and finding classes only; matched
|
||||
values and lines are never printed.
|
||||
|
||||
The production run invokes all five renderer entrypoints in private temporary
|
||||
contexts. Four output-producing renderers publish private handoffs. The access
|
||||
renderer uses its output-free Grafana verifier because it does not require
|
||||
historical metric inventories; its exact two-line success contract is checked.
|
||||
contexts. In a git checkout they run from a private copy of the tracked and
|
||||
not-ignored files, so an ignored chart cache cannot stop them. Four
|
||||
output-producing renderers publish private handoffs. The access renderer uses
|
||||
its output-free Grafana verifier because it does not require historical metric
|
||||
inventories; its exact two-line success contract is checked.
|
||||
It still traverses the pinned core renderer, whose separately published handoff
|
||||
contains the only allowlisted credential-free Secret and is decoded here.
|
||||
USAGE
|
||||
@@ -43,7 +52,7 @@ _sensitive_source_report_path() {
|
||||
local path=$1 platform_root=$2 docs_root=$3 relative
|
||||
if [[ "$path" == "$platform_root"/* ]]; then
|
||||
relative="platform/${path#"$platform_root"/}"
|
||||
elif [[ "$path" == "$docs_root"/* ]]; then
|
||||
elif [[ -n "$docs_root" && "$path" == "$docs_root"/* ]]; then
|
||||
relative="docs/${path#"$docs_root"/}"
|
||||
else
|
||||
relative="$(basename -- "$path")"
|
||||
@@ -82,13 +91,19 @@ structured_source = suffix in (".yaml", ".yml", ".json")
|
||||
python_source = suffix == ".py"
|
||||
shell_source = suffix in (".sh", ".bash", ".zsh") or not suffix
|
||||
prose_source = suffix in (".md", ".markdown", ".txt", ".rst")
|
||||
embedded_python_source = path in (
|
||||
"/home/donghyeon/workspace/platform/scripts/validate/observability-smoke.sh",
|
||||
"/home/donghyeon/workspace/platform/scripts/validate/test-scan-platform-sensitive-source.sh",
|
||||
"/home/donghyeon/workspace/platform/scripts/validate/test-configure-keycloak-grafana-oidc.sh",
|
||||
"/home/donghyeon/workspace/platform/scripts/bootstrap/create-observability-secrets.sh",
|
||||
"/home/donghyeon/workspace/platform/scripts/bootstrap/configure-keycloak-grafana-oidc.sh",
|
||||
)
|
||||
# Shell files with embedded Python, relative to the checkout that holds this
|
||||
# scanner (scanner_source is <root>/scripts/validate/<scanner>).
|
||||
scanner_root = Path(scanner_source).parents[2]
|
||||
embedded_python_source = path in {
|
||||
str(scanner_root / relative)
|
||||
for relative in (
|
||||
"scripts/validate/observability-smoke.sh",
|
||||
"scripts/validate/test-scan-platform-sensitive-source.sh",
|
||||
"scripts/validate/test-configure-keycloak-grafana-oidc.sh",
|
||||
"scripts/bootstrap/create-observability-secrets.sh",
|
||||
"scripts/bootstrap/configure-keycloak-grafana-oidc.sh",
|
||||
)
|
||||
}
|
||||
|
||||
try:
|
||||
raw = open(path, "rb").read()
|
||||
@@ -717,6 +732,19 @@ def permitted_reference_container(key, value):
|
||||
)
|
||||
|
||||
|
||||
def permitted_s3_credentials_reference(path, key, value):
|
||||
# CloudNativePG barmanObjectStore.s3Credentials: each entry is a
|
||||
# SecretKeySelector (name and key only), never the value itself.
|
||||
return (
|
||||
len(path) >= 1
|
||||
and path[-1] == "s3Credentials"
|
||||
and key in ("accessKeyId", "secretAccessKey")
|
||||
and isinstance(value, dict)
|
||||
and set(value) == {"name", "key"}
|
||||
and all(isinstance(value[field], str) and value[field] for field in value)
|
||||
)
|
||||
|
||||
|
||||
def permitted_webhook_feature(path, key, value):
|
||||
return (
|
||||
tuple(str(part) for part in path) == ("operators", "object-store")
|
||||
@@ -752,6 +780,8 @@ def walk(value, path=()):
|
||||
pass
|
||||
elif permitted_reference_container(key, child):
|
||||
pass
|
||||
elif permitted_s3_credentials_reference(path, key, child):
|
||||
pass
|
||||
elif not isinstance(child, str) or unsafe_scalar(child):
|
||||
return True
|
||||
if walk(child, path + (key,)):
|
||||
@@ -1382,9 +1412,13 @@ scan_platform_sensitive_roots() (
|
||||
return 1
|
||||
}
|
||||
done
|
||||
[[ "$platform_root" == /* && "$docs_root" == /* && "$platform_root" != "$docs_root" ]] ||
|
||||
# An empty docs root means a platform-only scan.
|
||||
local -a scan_roots=("$platform_root")
|
||||
[[ -z "$docs_root" ]] || scan_roots+=("$docs_root")
|
||||
[[ "$platform_root" == /* ]] && { [[ -z "$docs_root" ]] ||
|
||||
[[ "$docs_root" == /* && "$platform_root" != "$docs_root" ]]; } ||
|
||||
_sensitive_source_fail 'scan roots must be distinct absolute paths' || return
|
||||
for argument in "$platform_root" "$docs_root"; do
|
||||
for argument in "${scan_roots[@]}"; do
|
||||
[[ -d "$argument" && ! -L "$argument" && "$(cd -- "$argument" && pwd -P)" == "$argument" ]] ||
|
||||
_sensitive_source_fail "scan root is not a physical directory: $argument" || return
|
||||
done
|
||||
@@ -1397,7 +1431,7 @@ scan_platform_sensitive_roots() (
|
||||
printf 'SCANNER_ERROR: file inventory hardening failed\n' >&2
|
||||
return 1
|
||||
}
|
||||
if ! find "$platform_root" "$docs_root" \
|
||||
if ! find "${scan_roots[@]}" \
|
||||
\( -type d -path "$platform_root/.superpowers/sdd" -prune \) -o \
|
||||
\( -type d \( -name .git -o -name .helm \) -prune \) -o \
|
||||
\( -type f -o -type l \) -print0 >"$list_file"; then
|
||||
@@ -1432,7 +1466,7 @@ scan_platform_sensitive_roots() (
|
||||
fi
|
||||
|
||||
finding="$(_sensitive_source_python_scan \
|
||||
"$file" "$test_file" "$PLATFORM_SENSITIVE_SOURCE_EXACT_SCANNER")" || {
|
||||
"$file" "$test_file" "$PLATFORM_SENSITIVE_SOURCE_SELF")" || {
|
||||
printf 'SCANNER_ERROR: content scan failed for FILE=%s\n' "$report" >&2
|
||||
return 1
|
||||
}
|
||||
@@ -1505,7 +1539,7 @@ scan_platform_sensitive_roots() (
|
||||
scan_platform_renderer_secrets() (
|
||||
set -o pipefail
|
||||
local platform_root=$1 renderer label output_prefix publishes log file results line rc entry artifact
|
||||
local helm_bin list_file=''
|
||||
local helm_bin list_file='' render_root source_copy=''
|
||||
local failed=false
|
||||
local -a renderers=(
|
||||
'phase1|render-phase1.sh|platform-phase1-apply|yes'
|
||||
@@ -1546,6 +1580,11 @@ scan_platform_renderer_secrets() (
|
||||
rm -f -- "$list_file"
|
||||
;;
|
||||
esac
|
||||
case "$source_copy" in
|
||||
/tmp/platform-sensitive-source-render-src.??????)
|
||||
rm -rf -- "$source_copy"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
trap cleanup_renderer_scan EXIT HUP INT TERM
|
||||
|
||||
@@ -1570,9 +1609,56 @@ scan_platform_renderer_secrets() (
|
||||
return 1
|
||||
}
|
||||
|
||||
# In a git checkout, render from a private copy of the tracked and not-ignored
|
||||
# files (minus files deleted from the working tree). Ignored state such as a
|
||||
# leftover .helm chart cache would otherwise make the renderers refuse to run.
|
||||
render_root=$platform_root
|
||||
if [[ -e "$platform_root/.git" ]]; then
|
||||
for label in git tar; do
|
||||
command -v "$label" >/dev/null 2>&1 || {
|
||||
printf 'SCANNER_ERROR: renderer command unavailable: %s\n' "$label" >&2
|
||||
return 1
|
||||
}
|
||||
done
|
||||
[[ "$(git -C "$platform_root" rev-parse --show-toplevel 2>/dev/null)" == "$platform_root" ]] || {
|
||||
printf 'SCANNER_ERROR: renderer root is not the top of its git checkout\n' >&2
|
||||
return 1
|
||||
}
|
||||
source_copy="$(mktemp -d /tmp/platform-sensitive-source-render-src.XXXXXX)" || {
|
||||
printf 'SCANNER_ERROR: renderer source copy allocation failed\n' >&2
|
||||
return 1
|
||||
}
|
||||
chmod 0700 "$source_copy" || return 1
|
||||
if ! python3 -I - "$platform_root" <<'PY' |
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
root = sys.argv[1]
|
||||
|
||||
|
||||
def listed(*options):
|
||||
result = subprocess.run(
|
||||
["git", "-C", root, "ls-files", "-z", *options],
|
||||
check=True,
|
||||
stdout=subprocess.PIPE,
|
||||
)
|
||||
return [entry for entry in result.stdout.split(b"\0") if entry]
|
||||
|
||||
|
||||
keep = set(listed("--cached", "--others", "--exclude-standard")) - set(listed("--deleted"))
|
||||
for entry in sorted(keep):
|
||||
sys.stdout.buffer.write(entry + b"\0")
|
||||
PY
|
||||
tar -C "$platform_root" --null -T - -cf - | tar -x -C "$source_copy"; then
|
||||
printf 'SCANNER_ERROR: renderer source copy failed\n' >&2
|
||||
return 1
|
||||
fi
|
||||
render_root=$source_copy
|
||||
fi
|
||||
|
||||
for renderer in "${renderers[@]}"; do
|
||||
IFS='|' read -r label renderer output_prefix publishes <<<"$renderer"
|
||||
renderer="$platform_root/scripts/validate/$renderer"
|
||||
renderer="$render_root/scripts/validate/$renderer"
|
||||
[[ -f "$renderer" && ! -L "$renderer" ]] || {
|
||||
printf 'SCANNER_ERROR: required renderer is unsafe: %s\n' "$label" >&2
|
||||
return 1
|
||||
@@ -1706,14 +1792,23 @@ scan_platform_renderer_secrets() (
|
||||
)
|
||||
|
||||
scan_platform_sensitive_source_main() {
|
||||
(( $# == 0 )) || {
|
||||
local docs_root=$PLATFORM_SENSITIVE_SOURCE_DEFAULT_DOCS_ROOT
|
||||
if (( $# == 1 )) && [[ "$1" == --platform-only ]]; then
|
||||
docs_root=''
|
||||
elif (( $# == 2 )) && [[ "$1" == --docs-root && "$2" == /* ]]; then
|
||||
docs_root=$2
|
||||
elif (( $# != 0 )); then
|
||||
_sensitive_source_usage >&2
|
||||
return 2
|
||||
}
|
||||
fi
|
||||
if [[ -n "$docs_root" && ! -d "$docs_root" ]]; then
|
||||
printf 'ERROR: documentation root %s does not exist; pass --docs-root DIR or --platform-only\n' \
|
||||
"$docs_root" >&2
|
||||
return 2
|
||||
fi
|
||||
scan_platform_sensitive_roots \
|
||||
"$PLATFORM_SENSITIVE_SOURCE_EXACT_PLATFORM_ROOT" \
|
||||
"$PLATFORM_SENSITIVE_SOURCE_EXACT_DOCS_ROOT" >/dev/null || return
|
||||
scan_platform_renderer_secrets "$PLATFORM_SENSITIVE_SOURCE_EXACT_PLATFORM_ROOT" || return
|
||||
"$PLATFORM_SENSITIVE_SOURCE_SELF_ROOT" "$docs_root" >/dev/null || return
|
||||
scan_platform_renderer_secrets "$PLATFORM_SENSITIVE_SOURCE_SELF_ROOT" || return
|
||||
printf 'PLATFORM SENSITIVE SOURCE SCAN PASS\n'
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
#!/usr/bin/env bash
|
||||
# Test k3s-datastore-restore-drill.sh without production data: start a fresh,
|
||||
# network-less k3s with secrets encryption, add three test Secrets, stop it,
|
||||
# and run the drill on its state.db and token. Needs docker; no root.
|
||||
set -Eeuo pipefail
|
||||
umask 077
|
||||
|
||||
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
|
||||
readonly K3S_IMAGE="$(sed -n 's/^readonly K3S_IMAGE="\(.*\)"$/\1/p' "$ROOT/scripts/validate/k3s-datastore-restore-drill.sh")"
|
||||
[[ -n "$K3S_IMAGE" ]] || { echo "cannot read K3S_IMAGE from the drill script" >&2; exit 1; }
|
||||
work="$(mktemp -d)"
|
||||
seed="w6-k3s-drill-seed-$$"
|
||||
cleanup() {
|
||||
docker rm -f "$seed" >/dev/null 2>&1 || true
|
||||
docker run --rm -v "$work:/w" --entrypoint /bin/sh "$K3S_IMAGE" -c 'rm -rf /w/*' >/dev/null 2>&1 || true
|
||||
rm -rf -- "$work"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
mkdir -p "$work/server" "$work/copy"
|
||||
|
||||
docker run -d --name "$seed" --network none --tmpfs /run --tmpfs /var/run \
|
||||
-v "$work/server:/var/lib/rancher/k3s/server" "$K3S_IMAGE" server --disable-agent --secrets-encryption \
|
||||
--bind-address 127.0.0.1 --advertise-address 127.0.0.1 --node-ip 127.0.0.1 \
|
||||
--disable traefik,servicelb,metrics-server,local-storage,coredns >/dev/null
|
||||
for _ in $(seq 1 60); do docker exec "$seed" kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 5; done
|
||||
for i in 1 2 3; do docker exec "$seed" kubectl create secret generic "drill-test-$i" --from-literal=value="test-$i" >/dev/null; done
|
||||
expected="$(docker exec "$seed" kubectl get secrets -A --no-headers | wc -l)"
|
||||
docker stop -t 30 "$seed" >/dev/null
|
||||
docker run --rm -v "$work:/w" --entrypoint /bin/sh "$K3S_IMAGE" -c \
|
||||
"cp /w/server/db/state.db* /w/server/token /w/copy/ && chown $(id -u):$(id -g) /w/copy/*"
|
||||
# Fold the WAL into state.db, as `sqlite3 .backup` does in the real backup.
|
||||
sqlite3 "$work/copy/state.db" 'PRAGMA wal_checkpoint(TRUNCATE);' >/dev/null
|
||||
rm -f "$work/copy/state.db-wal" "$work/copy/state.db-shm"
|
||||
# The test Secrets must be stored encrypted (k8s:enc:...), so the drill really
|
||||
# proves that the token unlocks the encryption key.
|
||||
encrypted="$(sqlite3 "$work/copy/state.db" "SELECT COUNT(*) FROM kine WHERE name LIKE '/registry/secrets/default/drill-test-%' AND CAST(value AS TEXT) LIKE 'k8s:enc:%';")"
|
||||
echo "seed: $expected Secrets, encrypted test Secret rows=$encrypted"
|
||||
(( encrypted >= 3 )) || { echo "test Secrets are not encrypted at rest in the seed datastore" >&2; exit 1; }
|
||||
|
||||
bash "$ROOT/scripts/validate/k3s-datastore-restore-drill.sh" --from-dir "$work/copy" --expect-secrets "$expected"
|
||||
echo "K3S DATASTORE RESTORE DRILL TEST PASS"
|
||||
@@ -87,7 +87,20 @@ if alert != {
|
||||
"value": "InfoInhibitor",
|
||||
}
|
||||
],
|
||||
}
|
||||
},
|
||||
{
|
||||
"receiver": "platform-heartbeat",
|
||||
"matchers": [
|
||||
{
|
||||
"name": "alertname",
|
||||
"matchType": "=",
|
||||
"value": "PlatformHeartbeat",
|
||||
}
|
||||
],
|
||||
"groupWait": "0s",
|
||||
"groupInterval": "1m",
|
||||
"repeatInterval": "5m",
|
||||
},
|
||||
],
|
||||
},
|
||||
"inhibitRules": [
|
||||
@@ -111,6 +124,16 @@ if alert != {
|
||||
],
|
||||
"receivers": [
|
||||
{"name": "platform-null"},
|
||||
{
|
||||
"name": "platform-heartbeat",
|
||||
"webhookConfigs": [
|
||||
{
|
||||
"urlSecret": {"name": "alertmanager-heartbeat", "key": "url"},
|
||||
"sendResolved": False,
|
||||
"maxAlerts": 1,
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
"name": "platform-slack",
|
||||
"slackConfigs": [
|
||||
@@ -156,7 +179,7 @@ if alert != {
|
||||
],
|
||||
},
|
||||
}:
|
||||
fail("platform-alertmanager must match the exact global Slack contract")
|
||||
fail("platform-alertmanager must match the exact global Slack and heartbeat contract")
|
||||
|
||||
if policy != {
|
||||
"apiVersion": "networking.k8s.io/v1",
|
||||
|
||||
+263
@@ -0,0 +1,263 @@
|
||||
#!/usr/bin/env bash
|
||||
# Unit tests for the alerts that guard the desktop host: core-rules.yaml and
|
||||
# services/platform-postgres-backup/prometheus-rule.yaml.
|
||||
#
|
||||
# Root disk: the kubelet starts deleting unused images at 85% use and evicts
|
||||
# pods at 90% (nodefs.available < 10%). The alerts sit in front of those two
|
||||
# points: warning at 80%, critical at 85%, and a forecast that fires when the
|
||||
# 6h trend reaches the eviction point within 6 hours.
|
||||
# PostgreSQL backups: no successful base backup for 26 hours, or WAL waiting
|
||||
# to be archived. Monitoring: ingestion stopped, and the always-firing
|
||||
# heartbeat that an external dead man's switch watches.
|
||||
#
|
||||
# Needs docker, jq and kubectl (client only, no cluster access).
|
||||
set -euo pipefail
|
||||
|
||||
readonly ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
readonly RULE_FILE="${ROOT_DIR}/services/observability/rules/platform/core-rules.yaml"
|
||||
readonly BACKUP_RULE_FILE="${ROOT_DIR}/services/platform-postgres-backup/prometheus-rule.yaml"
|
||||
# Same pinned image as test-platform-observability-rules.sh.
|
||||
readonly PROMETHEUS_IMAGE="quay.io/prometheus/prometheus:v3.13.2-distroless@sha256:ce95cfa77eff5aad28bd7a65aff19868cf78d9e17e4c254da7dfe22ade78318b"
|
||||
readonly RUNBOOK_URL="https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md"
|
||||
readonly BACKUP_RUNBOOK_URL="https://git.learn.hyeonworks.com/donghyeon.kang/platform-core/src/branch/main/services/platform-postgres-backup/README.md"
|
||||
|
||||
temp_dir=''
|
||||
cleanup() {
|
||||
if [[ -n "$temp_dir" && -d "$temp_dir" ]]; then
|
||||
rm -rf -- "$temp_dir"
|
||||
fi
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
fail() {
|
||||
printf 'FAIL: %s\n' "$*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
for command in docker jq kubectl; do
|
||||
command -v "$command" >/dev/null || fail "missing command: ${command}"
|
||||
done
|
||||
|
||||
temp_dir="$(mktemp -d)"
|
||||
chmod 0755 "$temp_dir"
|
||||
|
||||
kubectl create --dry-run=client --validate=false -f "$RULE_FILE" -o json |
|
||||
jq '.spec' >"${temp_dir}/core.rules.json"
|
||||
kubectl create --dry-run=client --validate=false -f "$BACKUP_RULE_FILE" -o json |
|
||||
jq '.spec' >"${temp_dir}/backup.rules.json"
|
||||
|
||||
# Every core alert keeps the runbook the other platform rules use.
|
||||
jq -e --arg url "$RUNBOOK_URL" '
|
||||
all(.groups[].rules[] | select(has("alert")); .annotations.runbook_url == $url)
|
||||
' "${temp_dir}/core.rules.json" >/dev/null || fail "core alert without the shared runbook_url"
|
||||
jq -e --arg url "$BACKUP_RUNBOOK_URL" '
|
||||
all(.groups[].rules[] | select(has("alert")); .annotations.runbook_url == $url)
|
||||
' "${temp_dir}/backup.rules.json" >/dev/null || fail "backup alert without the backup README runbook_url"
|
||||
|
||||
# node-exporter labels as scraped by serviceMonitor/observability-agent/node-exporter.
|
||||
readonly SERIES_LABELS='job="prometheus-node-exporter",instance="10.42.0.95:9100",device="/dev/sdc1",fstype="ext4",mountpoint="/"'
|
||||
readonly PG_LABELS='job="platform-postgres",namespace="platform-data",pod="platform-postgres-1",instance="10.42.0.130:9187"'
|
||||
readonly ALERT_LABELS=' job: prometheus-node-exporter
|
||||
instance: 10.42.0.95:9100
|
||||
device: /dev/sdc1
|
||||
fstype: ext4
|
||||
mountpoint: /'
|
||||
|
||||
usage_case() {
|
||||
# name, used percent, eval minute, expected alerts (YAML list body or "[]")
|
||||
local name=$1 used=$2 eval_minute=$3 expected=$4
|
||||
local avail=$((100 - used))
|
||||
cat <<EOF
|
||||
- name: ${name}
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'node_filesystem_avail_bytes{${SERIES_LABELS}}'
|
||||
values: '${avail}+0x40'
|
||||
- series: 'node_filesystem_size_bytes{${SERIES_LABELS}}'
|
||||
values: '100+0x40'
|
||||
alert_rule_test:
|
||||
- eval_time: ${eval_minute}m
|
||||
alertname: PlatformRootFilesystemUsage
|
||||
exp_alerts:${expected}
|
||||
EOF
|
||||
}
|
||||
|
||||
usage_alert() {
|
||||
local severity=$1 threshold=$2 summary=$3 description=$4
|
||||
cat <<EOF
|
||||
|
||||
- exp_labels:
|
||||
${ALERT_LABELS}
|
||||
severity: ${severity}
|
||||
threshold: "${threshold}"
|
||||
exp_annotations:
|
||||
summary: ${summary}
|
||||
description: ${description}
|
||||
runbook_url: ${RUNBOOK_URL}
|
||||
EOF
|
||||
}
|
||||
|
||||
readonly WARNING_ALERT="$(usage_alert warning 80 \
|
||||
'루트 파일시스템 사용률이 80%를 초과함' \
|
||||
'노드 루트 파일시스템 사용률이 80%를 10분간 초과했습니다. 85%에서 kubelet이 이미지를 지우기 시작합니다.')"
|
||||
readonly CRITICAL_ALERT="$(usage_alert critical 85 \
|
||||
'루트 파일시스템 사용률이 85%를 초과함' \
|
||||
'노드 루트 파일시스템 사용률이 85%를 5분간 초과했습니다. kubelet이 안 쓰는 이미지를 지우는 중이고 90%에서 파드를 내보냅니다.')"
|
||||
|
||||
forecast_case() {
|
||||
# name, starting avail, avail change per minute, eval minute, expected alerts
|
||||
local name=$1 start=$2 step=$3 eval_minute=$4 expected=$5
|
||||
cat <<EOF
|
||||
- name: ${name}
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'node_filesystem_avail_bytes{${SERIES_LABELS}}'
|
||||
values: '${start}${step}x60'
|
||||
- series: 'node_filesystem_size_bytes{${SERIES_LABELS}}'
|
||||
values: '1000+0x60'
|
||||
alert_rule_test:
|
||||
- eval_time: ${eval_minute}m
|
||||
alertname: PlatformRootFilesystemFillingFast
|
||||
exp_alerts:${expected}
|
||||
EOF
|
||||
}
|
||||
|
||||
readonly FORECAST_ALERT="
|
||||
- exp_labels:
|
||||
${ALERT_LABELS}
|
||||
severity: critical
|
||||
exp_annotations:
|
||||
summary: 루트 파일시스템이 6시간 안에 축출 문턱에 닿을 것으로 예측됨
|
||||
description: 최근 6시간 추세대로면 루트 파일시스템 여유가 6시간 안에 10% 아래로 내려갑니다. 10% 아래에서 kubelet이 파드를 내보냅니다.
|
||||
runbook_url: ${RUNBOOK_URL}"
|
||||
|
||||
{
|
||||
printf '%s\n' \
|
||||
'rule_files:' \
|
||||
' - /work/core.rules.json' \
|
||||
' - /work/backup.rules.json' \
|
||||
'evaluation_interval: 1m' \
|
||||
'tests:'
|
||||
usage_case 'warning stays quiet at 79 percent' 79 11 ' []'
|
||||
usage_case 'warning fires at 81 percent after 10 minutes' 81 11 "$WARNING_ALERT"
|
||||
usage_case 'warning is still pending at 81 percent after 9 minutes' 81 9 ' []'
|
||||
usage_case 'critical fires at 86 percent after 5 minutes' 86 6 "$CRITICAL_ALERT"
|
||||
usage_case 'warning and critical both fire at 86 percent after 10 minutes' 86 11 "${WARNING_ALERT}${CRITICAL_ALERT}"
|
||||
# 1000 bytes, 400 free, losing 1 per minute: the 6h trend reaches the
|
||||
# 100-byte eviction point in 300 minutes.
|
||||
forecast_case 'forecast fires for a fast fill after 15 minutes' 400 '-1' 16 "$FORECAST_ALERT"
|
||||
forecast_case 'forecast is still pending before 15 minutes' 400 '-1' 14 ' []'
|
||||
# Losing 0.1 per minute takes far longer than 6 hours to reach 100.
|
||||
forecast_case 'forecast stays quiet for a slow fill' 400 '-0.1' 30 ' []'
|
||||
forecast_case 'forecast stays quiet for flat usage' 400 '+0' 30 ' []'
|
||||
|
||||
# PostgreSQL backups. Timestamps are seconds since the test epoch.
|
||||
cat <<EOF
|
||||
# The test clock starts at 0, so "27.8 hours ago" is a negative timestamp.
|
||||
# In production a cluster without any backup reports 0 and fires the same way.
|
||||
- name: backup alert fires when the last backup is older than 26 hours
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'cnpg_collector_last_available_backup_timestamp{${PG_LABELS}}'
|
||||
values: '-100000+0x40'
|
||||
alert_rule_test:
|
||||
- eval_time: 31m
|
||||
alertname: PlatformPostgresBackupMissing
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
job: platform-postgres
|
||||
namespace: platform-data
|
||||
severity: critical
|
||||
exp_annotations:
|
||||
summary: platform-postgres에 26시간 안에 끝난 백업이 없음
|
||||
description: platform-postgres의 마지막 성공한 base backup이 26시간보다 오래됐거나 없습니다. 매일 03:00 KST 백업(ScheduledBackup platform-postgres-daily)을 확인하세요.
|
||||
runbook_url: ${BACKUP_RUNBOOK_URL}
|
||||
- name: backup alert stays quiet after a fresh backup
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'cnpg_collector_last_available_backup_timestamp{${PG_LABELS}}'
|
||||
values: '60+0x40'
|
||||
alert_rule_test:
|
||||
- eval_time: 31m
|
||||
alertname: PlatformPostgresBackupMissing
|
||||
exp_alerts: []
|
||||
- name: WAL backlog fires after 15 minutes
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'cnpg_collector_pg_wal_archive_status{${PG_LABELS},value="ready"}'
|
||||
values: '11+0x30'
|
||||
- series: 'cnpg_collector_pg_wal_archive_status{${PG_LABELS},value="done"}'
|
||||
values: '500+0x30'
|
||||
alert_rule_test:
|
||||
- eval_time: 16m
|
||||
alertname: PlatformPostgresWALArchiveBacklog
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
job: platform-postgres
|
||||
namespace: platform-data
|
||||
severity: critical
|
||||
exp_annotations:
|
||||
summary: platform-postgres의 WAL이 보관소로 올라가지 않음
|
||||
description: 보관을 기다리는 WAL 파일이 10개를 넘은 채 15분이 지났습니다. 보관이 막히면 WAL이 PostgreSQL 볼륨에 쌓여 디스크가 찹니다.
|
||||
runbook_url: ${BACKUP_RUNBOOK_URL}
|
||||
- name: WAL backlog stays quiet at 10 ready files
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'cnpg_collector_pg_wal_archive_status{${PG_LABELS},value="ready"}'
|
||||
values: '10+0x30'
|
||||
alert_rule_test:
|
||||
- eval_time: 20m
|
||||
alertname: PlatformPostgresWALArchiveBacklog
|
||||
exp_alerts: []
|
||||
- name: ingestion alert fires when node-exporter samples stop
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'up{job="prometheus-node-exporter",instance="10.42.0.95:9100"}'
|
||||
values: '1x5 _x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 20m
|
||||
alertname: PlatformMetricsIngestionStopped
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
job: prometheus-node-exporter
|
||||
severity: critical
|
||||
exp_annotations:
|
||||
summary: Prometheus에 새 지표가 저장되지 않음
|
||||
description: 5분 넘게 node-exporter의 up 샘플이 저장되지 않았습니다. Prometheus가 샘플을 버리고 있으면(예 시계가 되돌아감) 다른 지표 경보도 울리지 않습니다.
|
||||
runbook_url: ${RUNBOOK_URL}
|
||||
- name: ingestion alert stays quiet while samples arrive
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'up{job="prometheus-node-exporter",instance="10.42.0.95:9100"}'
|
||||
values: '1x30'
|
||||
alert_rule_test:
|
||||
- eval_time: 20m
|
||||
alertname: PlatformMetricsIngestionStopped
|
||||
exp_alerts: []
|
||||
- name: heartbeat always fires
|
||||
interval: 1m
|
||||
alert_rule_test:
|
||||
- eval_time: 1m
|
||||
alertname: PlatformHeartbeat
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: none
|
||||
exp_annotations:
|
||||
summary: 경보 경로 heartbeat
|
||||
description: 항상 울린다. Alertmanager가 이 경보를 외부 dead man's switch로 5분마다 보내고, 끊기면 외부 서비스가 알린다.
|
||||
runbook_url: ${RUNBOOK_URL}
|
||||
EOF
|
||||
} >"${temp_dir}/disk.test.yaml"
|
||||
|
||||
chmod 0444 "${temp_dir}"/*
|
||||
docker run --rm \
|
||||
--network none \
|
||||
--volume "${temp_dir}:/work:ro" \
|
||||
--entrypoint /bin/promtool \
|
||||
"$PROMETHEUS_IMAGE" check rules /work/core.rules.json /work/backup.rules.json
|
||||
docker run --rm \
|
||||
--network none \
|
||||
--volume "${temp_dir}:/work:ro" \
|
||||
--entrypoint /bin/promtool \
|
||||
"$PROMETHEUS_IMAGE" test rules /work/disk.test.yaml
|
||||
printf 'PASS: root filesystem, PostgreSQL backup, ingestion and heartbeat alerts\n'
|
||||
@@ -0,0 +1,151 @@
|
||||
#!/usr/bin/env bash
|
||||
# promtool unit tests for services/platform-host-checks/prometheus-rules.yaml.
|
||||
# Needs docker and python3 (PyYAML). No cluster access.
|
||||
set -Eeuo pipefail
|
||||
|
||||
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
|
||||
# Same pinned image as test-platform-observability-rules.sh.
|
||||
readonly PROMETHEUS_IMAGE="quay.io/prometheus/prometheus:v3.13.2-distroless@sha256:ce95cfa77eff5aad28bd7a65aff19868cf78d9e17e4c254da7dfe22ade78318b"
|
||||
|
||||
work="$(mktemp -d)"
|
||||
trap 'rm -rf -- "$work"' EXIT
|
||||
python3 -I - "$ROOT/services/platform-host-checks/prometheus-rules.yaml" "$work/rules.yaml" <<'PY'
|
||||
import sys, yaml
|
||||
rule = yaml.safe_load(open(sys.argv[1]))
|
||||
yaml.safe_dump({"groups": rule["spec"]["groups"]}, open(sys.argv[2], "w"), sort_keys=False)
|
||||
PY
|
||||
|
||||
# promtool evaluates from Unix time 0, so "now" in each test is the eval_time.
|
||||
cat >"$work/tests.yaml" <<'EOF'
|
||||
rule_files:
|
||||
- rules.yaml
|
||||
evaluation_interval: 1m
|
||||
tests:
|
||||
# k3s datastore: last success at t=0; stale after 2 days, alert after 1h more.
|
||||
- interval: 1m
|
||||
input_series:
|
||||
- series: 'platform_backup_last_success_timestamp_seconds{backup="k3s-datastore"}'
|
||||
values: '0x5500'
|
||||
- series: 'platform_backup_last_run_timestamp_seconds{backup="k3s-datastore"}'
|
||||
values: '0x5500'
|
||||
- series: 'platform_backup_last_run_timestamp_seconds{backup="gitea"}'
|
||||
values: '0x5500'
|
||||
- series: 'platform_backup_last_success_timestamp_seconds{backup="gitea"}'
|
||||
values: '0x5500'
|
||||
alert_rule_test:
|
||||
- eval_time: 24h
|
||||
alertname: PlatformK3sDatastoreBackupTooOld
|
||||
exp_alerts: []
|
||||
- eval_time: 50h
|
||||
alertname: PlatformK3sDatastoreBackupTooOld
|
||||
exp_alerts:
|
||||
- exp_labels: {severity: warning}
|
||||
exp_annotations:
|
||||
summary: k3s datastore has not been backed up for more than 2 days
|
||||
description: Check `journalctl -u platform-k3s-datastore-backup` on desktop. Daily timer, 03:30 KST.
|
||||
- eval_time: 90h
|
||||
alertname: PlatformGiteaBackupTooOld
|
||||
exp_alerts: []
|
||||
- eval_time: 90h
|
||||
alertname: PlatformHostBackupMetricsAbsent
|
||||
exp_alerts: []
|
||||
- interval: 1m
|
||||
input_series:
|
||||
- series: 'platform_backup_last_success_timestamp_seconds{backup="gitea"}'
|
||||
values: '0x11700'
|
||||
alert_rule_test:
|
||||
- eval_time: 194h
|
||||
alertname: PlatformGiteaBackupTooOld
|
||||
exp_alerts:
|
||||
- exp_labels: {severity: warning}
|
||||
exp_annotations:
|
||||
summary: Gitea has not been backed up for more than 8 days
|
||||
description: Check `journalctl -u platform-gitea-backup` on desktop. Weekly timer, Sunday 04:00 KST.
|
||||
# A failed run is reported with its backup label.
|
||||
- interval: 1m
|
||||
input_series:
|
||||
- series: 'platform_backup_last_run_success{backup="gitea",instance="desktop"}'
|
||||
values: '0x30'
|
||||
- series: 'platform_backup_last_run_success{backup="k3s-datastore",instance="desktop"}'
|
||||
values: '1x30'
|
||||
alert_rule_test:
|
||||
- eval_time: 15m
|
||||
alertname: PlatformHostBackupFailed
|
||||
exp_alerts:
|
||||
- exp_labels: {severity: warning, backup: gitea}
|
||||
exp_annotations:
|
||||
summary: "The last gitea backup run failed"
|
||||
description: "Check `journalctl -u platform-gitea-backup` on desktop."
|
||||
# Metrics never written (timers or textfile collector missing).
|
||||
- interval: 1h
|
||||
input_series:
|
||||
- series: 'up{job="prometheus-node-exporter"}'
|
||||
values: '1x30'
|
||||
alert_rule_test:
|
||||
- eval_time: 25h
|
||||
alertname: PlatformHostBackupMetricsAbsent
|
||||
exp_alerts:
|
||||
- exp_labels: {severity: warning, backup: k3s-datastore}
|
||||
exp_annotations:
|
||||
summary: Host backup metrics are missing
|
||||
description: The backup timers or the node-exporter textfile collector (/var/lib/node_exporter/textfile_collector) are not running.
|
||||
- exp_labels: {severity: warning, backup: gitea}
|
||||
exp_annotations:
|
||||
summary: Host backup metrics are missing
|
||||
description: The backup timers or the node-exporter textfile collector (/var/lib/node_exporter/textfile_collector) are not running.
|
||||
# Drift for 7h fires; drift 0 does not.
|
||||
- interval: 1m
|
||||
input_series:
|
||||
- series: 'platform_live_drift_objects'
|
||||
values: '3x500'
|
||||
- series: 'platform_live_drift_last_run_timestamp_seconds'
|
||||
values: '0x500'
|
||||
- series: 'platform_live_drift_check_success'
|
||||
values: '1x500'
|
||||
alert_rule_test:
|
||||
- eval_time: 5h
|
||||
alertname: PlatformLiveDriftDetected
|
||||
exp_alerts: []
|
||||
- eval_time: 7h
|
||||
alertname: PlatformLiveDriftDetected
|
||||
exp_alerts:
|
||||
- exp_labels: {severity: warning}
|
||||
exp_annotations:
|
||||
summary: "3 live objects differ from platform-core origin/main"
|
||||
description: Run `scripts/validate/live-drift-check.sh --accepted scripts/validate/live-drift-accepted.txt` on desktop to list them.
|
||||
- eval_time: 7h
|
||||
alertname: PlatformLiveDriftCheckStale
|
||||
exp_alerts: []
|
||||
# Certificates: 50 days left -> warning only; 10 days left -> warning and critical.
|
||||
- interval: 1m
|
||||
input_series:
|
||||
- series: 'platform_k3s_certificate_not_after_seconds{certificate="apiserver"}'
|
||||
values: '4320000x200'
|
||||
- series: 'platform_k3s_certificate_not_after_seconds{certificate="kubelet"}'
|
||||
values: '864000x200'
|
||||
- series: 'platform_k3s_certificate_check_timestamp_seconds'
|
||||
values: '0x200'
|
||||
alert_rule_test:
|
||||
- eval_time: 2h
|
||||
alertname: PlatformK3sServingCertificateExpiry
|
||||
exp_alerts:
|
||||
- exp_labels: {severity: warning, certificate: apiserver}
|
||||
exp_annotations:
|
||||
summary: "k3s apiserver serving certificate expires in less than 100 days"
|
||||
description: k3s renews certificates only when it starts within 120 days of expiry. Plan a k3s restart window (docs/runbooks).
|
||||
- exp_labels: {severity: warning, certificate: kubelet}
|
||||
exp_annotations:
|
||||
summary: "k3s kubelet serving certificate expires in less than 100 days"
|
||||
description: k3s renews certificates only when it starts within 120 days of expiry. Plan a k3s restart window (docs/runbooks).
|
||||
- exp_labels: {severity: critical, certificate: kubelet}
|
||||
exp_annotations:
|
||||
summary: "k3s kubelet serving certificate expires in less than 14 days"
|
||||
description: Restart k3s (it renews certificates within 120 days of expiry) or run `k3s certificate rotate`.
|
||||
- eval_time: 2h
|
||||
alertname: PlatformK3sCertificateCheckStale
|
||||
exp_alerts: []
|
||||
EOF
|
||||
|
||||
docker run --rm --user "$(id -u):$(id -g)" -v "$work:/work:ro" -w /work --entrypoint /bin/promtool "$PROMETHEUS_IMAGE" check rules rules.yaml
|
||||
docker run --rm --user "$(id -u):$(id -g)" -v "$work:/work:ro" -w /work --entrypoint /bin/promtool "$PROMETHEUS_IMAGE" test rules tests.yaml
|
||||
echo "PLATFORM HOST CHECK ALERTS PASS"
|
||||
@@ -197,7 +197,7 @@ write_file "$alertmanager_fixture" \
|
||||
' alertmanagerConfiguration:' \
|
||||
' name: platform-alertmanager'
|
||||
alerting_fixture="$fixture_dir/alerting.yaml"
|
||||
write_file "$alerting_fixture" '{"apiVersion":"monitoring.coreos.com/v1alpha1","kind":"AlertmanagerConfig","metadata":{"name":"platform-alertmanager","namespace":"observability","labels":{"observability.hyeonworks.com/instance":"home"}},"spec":{"route":{"receiver":"platform-slack","groupBy":["cluster","namespace","alertname","severity"],"groupWait":"30s","groupInterval":"5m","repeatInterval":"4h","routes":[{"receiver":"platform-null","matchers":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}]}]},"inhibitRules":[{"sourceMatch":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}],"targetMatch":[{"name":"severity","matchType":"=","value":"info"}],"equal":["namespace"]}],"receivers":[{"name":"platform-null"},{"name":"platform-slack","slackConfigs":[{"apiURL":{"name":"alertmanager-slack-webhook","key":"url"},"sendResolved":true,"linkNames":false,"mrkdwnIn":["text","fields"],"fallback":"{{ if eq .Status \"firing\" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","title":"{{ if eq .Status \"firing\" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","titleLink":"https://grafana.learn.hyeonworks.com/","color":"{{ if eq .Status \"resolved\" }}good{{ else if or (eq .CommonLabels.severity \"emergency\") (eq .CommonLabels.severity \"critical\") }}danger{{ else if eq .CommonLabels.severity \"warning\" }}warning{{ else }}#439FE0{{ end }}","fields":[{"title":"Status","value":"{{ .Status | toUpper }}","short":true},{"title":"Severity","value":"{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}","short":true},{"title":"Location","value":"{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}","short":true},{"title":"Alert count","value":"{{ len .Alerts }}","short":true}],"text":"{{ range .Alerts }}\n*Alert status:* {{ .Status | toUpper }}\n*Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }}\n*Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }}\n*Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }}\n*Started:* {{ .StartsAt.Format \"2006-01-02T15:04:05Z07:00\" }}\n{{ if eq .Status \"resolved\" }}*Ended:* {{ .EndsAt.Format \"2006-01-02T15:04:05Z07:00\" }}{{ end }}\n{{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }}\n\n{{ end }}\n<https://grafana.learn.hyeonworks.com/|Grafana> · <https://grafana.learn.hyeonworks.com/explore|Explore>","footer":"hyeonworks observability · Alertmanager"}]}]}}' \
|
||||
write_file "$alerting_fixture" '{"apiVersion":"monitoring.coreos.com/v1alpha1","kind":"AlertmanagerConfig","metadata":{"name":"platform-alertmanager","namespace":"observability","labels":{"observability.hyeonworks.com/instance":"home"}},"spec":{"route":{"receiver":"platform-slack","groupBy":["cluster","namespace","alertname","severity"],"groupWait":"30s","groupInterval":"5m","repeatInterval":"4h","routes":[{"receiver":"platform-null","matchers":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}]},{"receiver":"platform-heartbeat","matchers":[{"name":"alertname","matchType":"=","value":"PlatformHeartbeat"}],"groupWait":"0s","groupInterval":"1m","repeatInterval":"5m"}]},"inhibitRules":[{"sourceMatch":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}],"targetMatch":[{"name":"severity","matchType":"=","value":"info"}],"equal":["namespace"]}],"receivers":[{"name":"platform-null"},{"name":"platform-heartbeat","webhookConfigs":[{"urlSecret":{"name":"alertmanager-heartbeat","key":"url"},"sendResolved":false,"maxAlerts":1}]},{"name":"platform-slack","slackConfigs":[{"apiURL":{"name":"alertmanager-slack-webhook","key":"url"},"sendResolved":true,"linkNames":false,"mrkdwnIn":["text","fields"],"fallback":"{{ if eq .Status \"firing\" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","title":"{{ if eq .Status \"firing\" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","titleLink":"https://grafana.learn.hyeonworks.com/","color":"{{ if eq .Status \"resolved\" }}good{{ else if or (eq .CommonLabels.severity \"emergency\") (eq .CommonLabels.severity \"critical\") }}danger{{ else if eq .CommonLabels.severity \"warning\" }}warning{{ else }}#439FE0{{ end }}","fields":[{"title":"Status","value":"{{ .Status | toUpper }}","short":true},{"title":"Severity","value":"{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}","short":true},{"title":"Location","value":"{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}","short":true},{"title":"Alert count","value":"{{ len .Alerts }}","short":true}],"text":"{{ range .Alerts }}\n*Alert status:* {{ .Status | toUpper }}\n*Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }}\n*Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }}\n*Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }}\n*Started:* {{ .StartsAt.Format \"2006-01-02T15:04:05Z07:00\" }}\n{{ if eq .Status \"resolved\" }}*Ended:* {{ .EndsAt.Format \"2006-01-02T15:04:05Z07:00\" }}{{ end }}\n{{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }}\n\n{{ end }}\n<https://grafana.learn.hyeonworks.com/|Grafana> · <https://grafana.learn.hyeonworks.com/explore|Explore>","footer":"hyeonworks observability · Alertmanager"}]}]}}' \
|
||||
'---' \
|
||||
'apiVersion: networking.k8s.io/v1' \
|
||||
'kind: NetworkPolicy' \
|
||||
|
||||
@@ -378,6 +378,44 @@ envValueFrom:
|
||||
YAML
|
||||
expect_reject 'a secretKeyRef container with a literal child fails closed' \
|
||||
'platform/bootstrap/reference-container.yaml'
|
||||
rm -f -- "$WORK/platform/bootstrap/reference-container.yaml"
|
||||
|
||||
cat >"$WORK/platform/bootstrap/s3-credentials.yaml" <<'YAML'
|
||||
spec:
|
||||
backup:
|
||||
barmanObjectStore:
|
||||
s3Credentials:
|
||||
accessKeyId:
|
||||
name: platform-postgres-backup-s3
|
||||
key: ACCESS_KEY_ID
|
||||
secretAccessKey:
|
||||
name: platform-postgres-backup-s3
|
||||
key: ACCESS_SECRET_KEY
|
||||
YAML
|
||||
expect_clean 'a CloudNativePG s3Credentials Secret reference remains a reference'
|
||||
cat >"$WORK/platform/bootstrap/s3-credentials.yaml" <<'YAML'
|
||||
spec:
|
||||
s3Credentials:
|
||||
secretAccessKey:
|
||||
name: platform-postgres-backup-s3
|
||||
key: ACCESS_SECRET_KEY
|
||||
value: should-never-be-printed-s3-reference
|
||||
YAML
|
||||
expect_reject 'an s3Credentials reference with an extra literal child fails closed' \
|
||||
'platform/bootstrap/s3-credentials.yaml'
|
||||
cat >"$WORK/platform/bootstrap/s3-credentials.yaml" <<'YAML'
|
||||
spec:
|
||||
storage:
|
||||
secretAccessKey:
|
||||
name: platform-postgres-backup-s3
|
||||
key: ACCESS_SECRET_KEY
|
||||
YAML
|
||||
expect_reject 'a name and key container outside s3Credentials still fails closed' \
|
||||
'platform/bootstrap/s3-credentials.yaml'
|
||||
printf 'spec:\n s3Credentials:\n secretAccess%sKey: Actual%s\n' '' 'ProductionSecretValue9283' \
|
||||
>"$WORK/platform/bootstrap/s3-credentials.yaml"
|
||||
expect_reject 'a literal s3Credentials secret access key is rejected' \
|
||||
'platform/bootstrap/s3-credentials.yaml'
|
||||
|
||||
new_fixture
|
||||
cat >"$WORK/platform/bootstrap/webhook-feature.yaml" <<'YAML'
|
||||
@@ -967,6 +1005,16 @@ label="$(basename -- "$0")"
|
||||
handoff=no
|
||||
[[ -z "$out" ]] || handoff=yes
|
||||
printf '%s\n' "$label${component:+:$component}:handoff=$handoff" >>"$PLATFORM_SENSITIVE_SOURCE_RENDER_LOG"
|
||||
root="$(cd -- "$(dirname -- "$0")/../.." && pwd -P)"
|
||||
if [[ "${PLATFORM_SENSITIVE_SOURCE_FAKE_REJECT_CACHE:-0}" != 0 ]] && \
|
||||
[[ -n "$(find "$root" -name .helm -type d -print -quit)" ]]; then
|
||||
printf 'REJECT: exact chart cache already exists\n' >&2
|
||||
exit 23
|
||||
fi
|
||||
if [[ -n "${PLATFORM_SENSITIVE_SOURCE_FAKE_REQUIRE_INPUT:-}" && \
|
||||
! -f "$root/$PLATFORM_SENSITIVE_SOURCE_FAKE_REQUIRE_INPUT" ]]; then
|
||||
exit 42
|
||||
fi
|
||||
|
||||
emit_phase1_log() {
|
||||
local publish=$1 name
|
||||
@@ -1438,6 +1486,41 @@ for injected_renderer in \
|
||||
fail "$injected_renderer unexpected stdout leaked content"
|
||||
pass "$injected_renderer stdout is restricted to its bounded grammar"
|
||||
done
|
||||
|
||||
# A git checkout renders from a private copy of its tracked and not-ignored files,
|
||||
# so an ignored chart cache left in the checkout cannot stop the renderers (the
|
||||
# real renderers refuse a pre-existing exact cache with RC=23).
|
||||
command -v git >/dev/null 2>&1 || fail 'git is required for the clean render copy test'
|
||||
printf '.helm/\n' >"$WORK/platform/.gitignore"
|
||||
git -C "$WORK/platform" init -q
|
||||
git -C "$WORK/platform" add -A
|
||||
git -C "$WORK/platform" -c user.name=fixture -c user.email=fixture@example.invalid \
|
||||
commit -q -m fixture
|
||||
mkdir -p "$WORK/platform/services/observability/.helm/charts/stale-1.0.0"
|
||||
printf 'untracked\n' >"$WORK/platform/services/observability/untracked-input.yaml"
|
||||
: >"$WORK/renderer.log"
|
||||
renderer_rc=0
|
||||
renderer_output="$(PLATFORM_SENSITIVE_SOURCE_FAKE_REJECT_CACHE=1 \
|
||||
PLATFORM_SENSITIVE_SOURCE_FAKE_REQUIRE_INPUT=services/observability/untracked-input.yaml \
|
||||
scan_platform_renderer_secrets "$WORK/platform" 2>&1)" || renderer_rc=$?
|
||||
(( renderer_rc == 0 )) || {
|
||||
printf '%s\n' "$renderer_output" >&2
|
||||
fail 'an ignored chart cache in a git checkout blocked the renderers'
|
||||
}
|
||||
[[ -d "$WORK/platform/services/observability/.helm/charts/stale-1.0.0" ]] || \
|
||||
fail 'the renderer scan changed the ignored cache in the scanned checkout'
|
||||
pass 'a git checkout renders from a clean copy, so an ignored chart cache cannot block it'
|
||||
pass 'the clean render copy keeps untracked files that git does not ignore'
|
||||
rm -f -- "$WORK/platform/services/observability/untracked-input.yaml"
|
||||
rm -f -- "$WORK/platform/scripts/validate/render-phase2.sh"
|
||||
renderer_rc=0
|
||||
renderer_output="$(scan_platform_renderer_secrets "$WORK/platform" 2>&1)" || renderer_rc=$?
|
||||
(( renderer_rc != 0 )) || fail 'a renderer deleted from the working tree was still run'
|
||||
[[ "$renderer_output" == *'required renderer is unsafe: phase2'* ]] || {
|
||||
printf '%s\n' "$renderer_output" >&2
|
||||
fail 'a renderer deleted from the working tree did not fail at the renderer check'
|
||||
}
|
||||
pass 'the clean render copy leaves out files deleted from the working tree'
|
||||
unset PLATFORM_SENSITIVE_SOURCE_RENDER_LOG
|
||||
|
||||
new_fixture
|
||||
@@ -1468,6 +1551,39 @@ output="$(run_scanner 2>&1)" || scanner_error_rc=$?
|
||||
fail 'scanner error did not fail closed'
|
||||
}
|
||||
pass 'rg errors fail closed'
|
||||
PLATFORM_SENSITIVE_SOURCE_TEST_PATH=''
|
||||
|
||||
[[ "$PLATFORM_SENSITIVE_SOURCE_SELF_ROOT" == "$ROOT" ]] || \
|
||||
fail 'the scanner does not scan the checkout it lives in'
|
||||
pass 'the scanner scans the checkout it lives in, not a fixed path'
|
||||
main_rc=0
|
||||
main_output="$(scan_platform_sensitive_source_main --unknown 2>&1)" || main_rc=$?
|
||||
(( main_rc == 2 )) || fail 'an unknown entrypoint argument was accepted'
|
||||
main_rc=0
|
||||
main_output="$(scan_platform_sensitive_source_main --docs-root relative/docs 2>&1)" || main_rc=$?
|
||||
(( main_rc == 2 )) || fail 'a relative docs root was accepted'
|
||||
new_fixture
|
||||
main_rc=0
|
||||
main_output="$(scan_platform_sensitive_source_main --docs-root "$WORK/missing-docs" 2>&1)" || main_rc=$?
|
||||
(( main_rc == 2 )) || fail 'a missing docs root was accepted'
|
||||
[[ "$main_output" == *'--platform-only'* ]] || \
|
||||
fail 'a missing docs root did not name --platform-only'
|
||||
pass 'the entrypoint rejects unknown, relative and missing docs roots before scanning'
|
||||
printf 'ordinary input\n' >"$WORK/platform/bootstrap/ordinary.md"
|
||||
platform_only_output="$(scan_platform_sensitive_roots "$WORK/platform" '' 2>&1)" || {
|
||||
printf '%s\n' "$platform_only_output" >&2
|
||||
fail 'a platform-only root scan failed'
|
||||
}
|
||||
[[ "$platform_only_output" == *'PLATFORM SENSITIVE SOURCE SCAN PASS'* ]] || \
|
||||
fail 'a platform-only root scan did not emit the PASS marker'
|
||||
printf 'Slack: https://hooks.%s/services/T1/B2/C3\n' 'slack.com' \
|
||||
>"$WORK/platform/bootstrap/live.env"
|
||||
platform_only_rc=0
|
||||
platform_only_output="$(scan_platform_sensitive_roots "$WORK/platform" '' 2>&1)" || platform_only_rc=$?
|
||||
(( platform_only_rc != 0 )) || fail 'a platform-only root scan accepted a Slack webhook'
|
||||
[[ "$platform_only_output" == *'platform/bootstrap/live.env'* ]] || \
|
||||
fail 'a platform-only root scan did not report the filename'
|
||||
pass 'a platform-only root scan still scans the platform root'
|
||||
|
||||
cleanup_before="$(
|
||||
find /tmp -maxdepth 1 -type d -name 'platform-sensitive-source-test.??????' \
|
||||
|
||||
@@ -2,6 +2,9 @@ namespace: gitea
|
||||
|
||||
replicaCount: 1
|
||||
|
||||
# services/priority-classes: evicted after everything that is not critical.
|
||||
priorityClassName: platform-critical
|
||||
|
||||
image:
|
||||
registry: docker.gitea.com
|
||||
repository: gitea
|
||||
|
||||
@@ -9,6 +9,15 @@ metadata:
|
||||
spec:
|
||||
instances: 1
|
||||
|
||||
# The 26.7.0 server image running on 2026-11-12, by digest. With spec.image set
|
||||
# the operator assumes a pre-built (optimized) image; this is the stock image,
|
||||
# so startOptimized stays false and Keycloak builds at start as before.
|
||||
image: quay.io/keycloak/keycloak@sha256:0f198be292568439d700cdbfb893e69a6009bb43a94a06a945b1d3d506c76b13
|
||||
startOptimized: false
|
||||
|
||||
scheduling:
|
||||
priorityClassName: platform-critical
|
||||
|
||||
db:
|
||||
vendor: postgres
|
||||
host: platform-postgres-rw.platform-data.svc.cluster.local
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
# Alertmanager 경로
|
||||
|
||||
`platform-alertmanager.yaml`은 kube-prometheus-stack이 전역
|
||||
`alertmanagerConfiguration`으로 읽는 AlertmanagerConfig 하나다. 경로는 셋이다.
|
||||
|
||||
| 경보 | receiver | 보내는 곳 |
|
||||
| --- | --- | --- |
|
||||
| `InfoInhibitor` | `platform-null` | 버린다 |
|
||||
| `PlatformHeartbeat` | `platform-heartbeat` | 외부 heartbeat 서비스(dead man's switch), 5분마다 |
|
||||
| 나머지 | `platform-slack` | Slack, 4시간마다 다시 |
|
||||
|
||||
## heartbeat (dead man's switch)
|
||||
|
||||
`PlatformHeartbeat`(`rules/platform/core-rules.yaml`)는 항상 울린다. Alertmanager가 이것을
|
||||
5분마다 외부 서비스의 ping URL로 보낸다. ping이 끊기면 그 서비스가 소유자에게 알린다.
|
||||
|
||||
- 막는 것: Prometheus·Alertmanager가 죽었을 때, desktop이나 인터넷 연결이 끊겼을 때.
|
||||
이 경우는 클러스터 안의 어떤 규칙도 알리지 못한다.
|
||||
- 막지 못하는 것: Prometheus가 돌면서 샘플만 버리는 경우(2026-11-11 시계 사고). 규칙은 돌고
|
||||
heartbeat도 계속 나간다. 그 경우는 `PlatformMetricsIngestionStopped`가 Slack으로 알린다.
|
||||
|
||||
### 소유자가 할 일 (적용 전에 한 번)
|
||||
|
||||
1. 외부 heartbeat 서비스에 check를 하나 만든다. 예: healthchecks.io(무료 요금제, 오픈소스라
|
||||
나중에 직접 띄울 수도 있다). 주기 5분, 유예 10분. 알림은 Slack이 아닌 곳(메일·휴대폰)도
|
||||
하나 건다. Slack 경로가 죽었을 때도 알아야 하기 때문이다.
|
||||
2. ping URL을 Secret으로 만든다. URL 자체가 자격 증명이라 화면에 남기지 않는다.
|
||||
|
||||
```bash
|
||||
read -rs HEARTBEAT_URL # ping URL 붙여 넣기, 화면에 안 보임
|
||||
printf '%s' "$HEARTBEAT_URL" | kubectl -n observability create secret generic alertmanager-heartbeat \
|
||||
--from-file=url=/dev/stdin
|
||||
unset HEARTBEAT_URL
|
||||
```
|
||||
|
||||
3. 그 뒤에 rules·alerts를 다시 적용한다(`scripts/bootstrap/apply-observability-access.sh
|
||||
--execute --rules-alerts`). **Secret보다 먼저 적용하지 않는다.** operator가 없는 Secret을
|
||||
가리키는 AlertmanagerConfig로는 설정을 만들지 못한다.
|
||||
4. 확인: 외부 서비스의 check가 5분 안에 「up」이 된다. Alertmanager 지표
|
||||
`alertmanager_notifications_total{integration="webhook"}`가 5분마다 는다.
|
||||
|
||||
Alertmanager egress는 공개 IPv4 TCP/443만 허용한다(`alertmanager-public-https-egress.yaml`).
|
||||
ping URL은 HTTPS여야 한다.
|
||||
@@ -22,6 +22,16 @@ spec:
|
||||
- name: alertname
|
||||
matchType: "="
|
||||
value: InfoInhibitor
|
||||
# PlatformHeartbeat always fires. Sending it every 5 minutes keeps the
|
||||
# external dead man's switch quiet; when it stops, that service alerts.
|
||||
- receiver: platform-heartbeat
|
||||
matchers:
|
||||
- name: alertname
|
||||
matchType: "="
|
||||
value: PlatformHeartbeat
|
||||
groupWait: 0s
|
||||
groupInterval: 1m
|
||||
repeatInterval: 5m
|
||||
inhibitRules:
|
||||
- sourceMatch:
|
||||
- name: alertname
|
||||
@@ -35,6 +45,15 @@ spec:
|
||||
- namespace
|
||||
receivers:
|
||||
- name: platform-null
|
||||
- name: platform-heartbeat
|
||||
webhookConfigs:
|
||||
# Ping URL of the external heartbeat check. Create the Secret before
|
||||
# applying this file; see services/observability/alerting/README.md.
|
||||
- urlSecret:
|
||||
name: alertmanager-heartbeat
|
||||
key: url
|
||||
sendResolved: false
|
||||
maxAlerts: 1
|
||||
- name: platform-slack
|
||||
slackConfigs:
|
||||
- apiURL:
|
||||
|
||||
@@ -34,6 +34,11 @@ kubeProxy:
|
||||
defaultRules:
|
||||
disabled:
|
||||
Watchdog: true
|
||||
# k3s kubelet exposes no kubelet_certificate_manager_* metrics, so these two
|
||||
# can never fire. Serving certificates are watched by platform-host-checks
|
||||
# (PlatformK3sServingCertificateExpiry) instead. Review 2026-11-11 A11.
|
||||
KubeletClientCertificateExpiration: true
|
||||
KubeletServerCertificateExpiration: true
|
||||
rules:
|
||||
etcd: false
|
||||
kubeControllerManager: false
|
||||
|
||||
@@ -80,6 +80,8 @@ loki:
|
||||
max_compaction_parallelism: 1
|
||||
|
||||
singleBinary:
|
||||
# services/priority-classes: logs can be lost first under node pressure.
|
||||
priorityClassName: platform-low
|
||||
replicas: 1
|
||||
sidecar: false
|
||||
podLabels:
|
||||
|
||||
@@ -32,7 +32,11 @@ resources:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 250m
|
||||
# Average use is about 2m, but each scrape runs the collectors in
|
||||
# parallel. At 250m (25ms per 100ms CFS period) 26% of periods were
|
||||
# throttled on 2026-11-11. One core removes the throttling and still caps
|
||||
# a runaway collector.
|
||||
cpu: "1"
|
||||
memory: 256Mi
|
||||
|
||||
serviceAccount:
|
||||
@@ -62,6 +66,14 @@ hostNetwork: false
|
||||
hostPID: false
|
||||
hostIPC: false
|
||||
|
||||
# Host timers (infrastructure/host/backup, owned by the k3s maturity work)
|
||||
# write *.prom files here: last successful Gitea dump and k3s datastore
|
||||
# backup, live drift count, API server and kubelet certificate expiry. The
|
||||
# host root is already mounted read-only at /host/root. A missing directory
|
||||
# only produces an error metric.
|
||||
extraArgs:
|
||||
- --collector.textfile.directory=/host/root/var/lib/node_exporter/textfile_collector
|
||||
|
||||
hostRootFsMount:
|
||||
enabled: true
|
||||
mountPropagation: None
|
||||
|
||||
@@ -41,14 +41,14 @@ spec:
|
||||
node_filesystem_avail_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
|
||||
/
|
||||
node_filesystem_size_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
|
||||
) > 85
|
||||
for: 15m
|
||||
) > 80
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
threshold: "85"
|
||||
threshold: "80"
|
||||
annotations:
|
||||
summary: 루트 파일시스템 사용률이 85%를 초과함
|
||||
description: 노드 루트 파일시스템 사용률이 경고 임계치인 85%를 15분간 초과했습니다.
|
||||
summary: 루트 파일시스템 사용률이 80%를 초과함
|
||||
description: 노드 루트 파일시스템 사용률이 80%를 10분간 초과했습니다. 85%에서 kubelet이 이미지를 지우기 시작합니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- alert: PlatformRootFilesystemUsage
|
||||
expr: >-
|
||||
@@ -56,14 +56,32 @@ spec:
|
||||
node_filesystem_avail_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
|
||||
/
|
||||
node_filesystem_size_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
|
||||
) > 90
|
||||
) > 85
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
threshold: "90"
|
||||
threshold: "85"
|
||||
annotations:
|
||||
summary: 루트 파일시스템 사용률이 90%를 초과함
|
||||
description: 노드 루트 파일시스템 사용률이 심각 임계치인 90%를 5분간 초과했습니다.
|
||||
summary: 루트 파일시스템 사용률이 85%를 초과함
|
||||
description: 노드 루트 파일시스템 사용률이 85%를 5분간 초과했습니다. kubelet이 안 쓰는 이미지를 지우는 중이고 90%에서 파드를 내보냅니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
# kubelet evicts pods when nodefs.available drops below 10%. Fire when
|
||||
# the 6h trend says that happens within the next 6 hours, so a fast
|
||||
# fill (17G in 9h on 2026-11-07) is caught before the 80% warning.
|
||||
- alert: PlatformRootFilesystemFillingFast
|
||||
expr: >-
|
||||
predict_linear(
|
||||
node_filesystem_avail_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}[6h],
|
||||
6 * 3600
|
||||
)
|
||||
<
|
||||
0.10 * node_filesystem_size_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: 루트 파일시스템이 6시간 안에 축출 문턱에 닿을 것으로 예측됨
|
||||
description: 최근 6시간 추세대로면 루트 파일시스템 여유가 6시간 안에 10% 아래로 내려갑니다. 10% 아래에서 kubelet이 파드를 내보냅니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- alert: PlatformRootFilesystemInodes
|
||||
expr: >-
|
||||
@@ -95,6 +113,32 @@ spec:
|
||||
summary: 루트 파일시스템 inode 사용률이 90%를 초과함
|
||||
description: 노드 루트 파일시스템 inode 사용률이 심각 임계치인 90%를 15분간 초과했습니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- name: platform.monitoring
|
||||
rules:
|
||||
# When the TSDB rejects every sample (2026-11-11: the clock went back
|
||||
# after samples were stored 9 hours ahead) rules still run, but on no
|
||||
# data. absent() is one of the few expressions that still fires then.
|
||||
- alert: PlatformMetricsIngestionStopped
|
||||
expr: absent(up{job="prometheus-node-exporter"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: Prometheus에 새 지표가 저장되지 않음
|
||||
description: 5분 넘게 node-exporter의 up 샘플이 저장되지 않았습니다. Prometheus가 샘플을 버리고 있으면(예 시계가 되돌아감) 다른 지표 경보도 울리지 않습니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
# Dead man's switch. Alertmanager forwards this to an external
|
||||
# heartbeat service; when the heartbeat stops, that service alerts.
|
||||
# It covers what no rule here can: Prometheus, Alertmanager or the
|
||||
# whole node being down.
|
||||
- alert: PlatformHeartbeat
|
||||
expr: vector(1)
|
||||
labels:
|
||||
severity: none
|
||||
annotations:
|
||||
summary: 경보 경로 heartbeat
|
||||
description: 항상 울린다. Alertmanager가 이 경보를 외부 dead man's switch로 5분마다 보내고, 끊기면 외부 서비스가 알린다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- name: platform.workloads
|
||||
rules:
|
||||
- alert: PlatformPodRestartBurst
|
||||
|
||||
@@ -4,6 +4,9 @@ global:
|
||||
|
||||
replicas: 1
|
||||
|
||||
# services/priority-classes: traces can be lost first under node pressure.
|
||||
priorityClassName: platform-low
|
||||
|
||||
config: |
|
||||
stream_over_http_enabled: false
|
||||
memberlist:
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
# 호스트 백업과 점검
|
||||
|
||||
desktop 호스트에서 systemd 타이머로 도는 백업과 점검이다. 결과는 node-exporter textfile 수집기
|
||||
(`/var/lib/node_exporter/textfile_collector/*.prom`)로 Prometheus에 올라가고, 이 폴더의 PrometheusRule이
|
||||
경보를 낸다. 근거와 결정은 `docs/reviews/2026-11-11-k3s-maturity-review.md` A1·A11·A13·A14, 8절 결정 1·6이다.
|
||||
|
||||
## 무엇을 어디에 얼마나
|
||||
|
||||
| 대상 | 방법 | 주기 | 보관 | 위치 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| k3s datastore + `server/token` | `sqlite3 .backup` → tar.gz + sha256 | 매일 03:30 KST | AIStor 30일 + 로컬 최근 7개 | AIStor `platform-host-backups/k3s-datastore/`와 `/srv/k3s/aistor/host-backups/k3s-datastore/`(root 0700). 둘 다 HDD sdb3 |
|
||||
| Gitea (저장소·패키지·첨부·Actions·`app.ini`) | 라이브 파드에서 `gitea dump --skip-db` | 일요일 04:00 KST | 56일(8주) | AIStor `platform-host-backups/gitea/` |
|
||||
| platform-postgres (Gitea·Keycloak·tech-log DB) | CloudNativePG base backup + WAL | 매일 03:00 KST | 30일 | AIStor `platform-postgres-backups/` (W1, `services/platform-postgres-backup`) |
|
||||
| AIStor 데이터(Loki·Tempo) | **백업하지 않는다** | — | — | 8절 결정 6: 잃어도 되는 관측 데이터. 디스크를 잃으면 로그·추적 기록만 사라진다 |
|
||||
| Prometheus·Grafana·Alertmanager PV | 백업하지 않는다 | — | — | 설정은 이 저장소가 정본이고, 지표는 다시 쌓인다 |
|
||||
|
||||
- Gitea DB는 dump에 없다(`--skip-db`). Gitea를 복원하려면 CNPG 복원이 먼저다(`docs/runbooks/gitea-restore.md`).
|
||||
- k3s 백업에는 `token`이 들어 있다. token과 datastore가 같이 있으면 클러스터의 모든 Secret을 풀 수 있다.
|
||||
Gitea dump의 `app.ini`에도 비밀 값이 있다. 그래서 둘 다 AIStor의 전용 사용자만 읽고 쓴다.
|
||||
- AIStor는 같은 desktop의 HDD다. SSD가 죽으면 남지만 desktop 자체를 잃으면 같이 잃는다.
|
||||
- k3s 백업을 AIStor 밖에도 두는 이유: AIStor는 바로 그 k3s 위에서 돈다. 클러스터를 통째로 잃으면 AIStor를
|
||||
다시 띄우기 전에는 AIStor 안의 사본을 꺼낼 수 없다. 로컬 사본은 sdb3를 mount하기만 하면 읽힌다
|
||||
(`docs/runbooks/k3s-datastore-restore.md`). 폴더는 AIStor 드라이브 폴더(`data/`) 옆이라 bucket으로 보이지 않는다.
|
||||
|
||||
## off-host 사본 (준비만, 기본 꺼짐)
|
||||
|
||||
위치는 소유자가 정한다(8절 결정 1). `/etc/platform-backup/aistor.env`에 두 줄을 넣으면 k3s datastore
|
||||
백업이 그 공개 키로 암호화한 사본을 `rclone`으로 보낸다.
|
||||
|
||||
```bash
|
||||
OFFHOST_GPG_RECIPIENT_FILE=/etc/platform-backup/offhost-recipient.asc # 공개 키. 개인 키는 desktop에 두지 않는다
|
||||
OFFHOST_RCLONE_TARGET=<rclone remote>:<path> # rclone 설치·설정 필요
|
||||
```
|
||||
|
||||
지표 `platform_backup_offhost_copy{backup="k3s-datastore"}`가 1이 된다. Gitea dump의 off-host 사본은 아직 없다.
|
||||
|
||||
## 점검
|
||||
|
||||
| 타이머 | 사용자 | 하는 일 | 지표 |
|
||||
| --- | --- | --- | --- |
|
||||
| `platform-k3s-cert-expiry` | donghyeon | 6443·10250 serving 인증서 notAfter | `platform_k3s_certificate_not_after_seconds` |
|
||||
| `platform-live-drift-check` | donghyeon | origin/main 렌더 → `kubectl diff`(서버 dry-run) | `platform_live_drift_objects` |
|
||||
|
||||
## 경보 (`prometheus-rules.yaml`)
|
||||
|
||||
| 경보 | 조건 |
|
||||
| --- | --- |
|
||||
| `PlatformK3sDatastoreBackupTooOld` | 마지막 성공이 2일보다 오래됨 |
|
||||
| `PlatformGiteaBackupTooOld` | 8일보다 오래됨 |
|
||||
| `PlatformHostBackupFailed` | 마지막 실행 실패 |
|
||||
| `PlatformHostBackupMetricsAbsent` | 지표가 하루 동안 없음(타이머나 textfile 수집기가 안 돎) |
|
||||
| `PlatformLiveDriftDetected` | drift가 6시간 넘게 남음 |
|
||||
| `PlatformLiveDriftCheckStale` | drift 검사가 2일 넘게 완료되지 않음 |
|
||||
| `PlatformK3sServingCertificateExpiry` | 100일 미만 warning, 14일 미만 critical |
|
||||
| `PlatformK3sCertificateCheckStale` | 인증서 점검이 2일 넘게 안 돎 |
|
||||
|
||||
`scripts/validate/test-platform-host-checks-alerts.sh`가 promtool 단위 시험으로 각 경보가 울려야 할 때 울리고
|
||||
아닐 때 안 울리는지 본다. 같은 PR에서 kube-prometheus-stack의 `KubeletClientCertificateExpiration`·
|
||||
`KubeletServerCertificateExpiration`을 껐다. k3s kubelet은 그 지표를 내지 않아 울릴 수 없었다.
|
||||
|
||||
## 적용 (소유자, 순서대로)
|
||||
|
||||
전제: node-exporter textfile 수집기(W1, `services/observability/node-exporter/values/home.yaml`의
|
||||
`--collector.textfile.directory=/host/root/var/lib/node_exporter/textfile_collector`)가 들어간 main checkout.
|
||||
|
||||
```bash
|
||||
cd /home/donghyeon/workspace/platform && git pull --ff-only
|
||||
sudo bash scripts/bootstrap/create-host-backup-credentials.sh # bucket·정책·사용자·/etc/platform-backup/aistor.env (APPLY 입력)
|
||||
bash scripts/bootstrap/install-host-checks.sh --check
|
||||
sudo bash scripts/bootstrap/install-host-checks.sh --execute # 타이머 설치·켜기·첫 실행
|
||||
kubectl apply -k services/platform-host-checks # 경보 규칙
|
||||
sudo bash scripts/validate/k3s-datastore-restore-drill.sh # 첫 백업으로 복원 리허설
|
||||
sudo bash scripts/validate/gitea-backup-restore-drill.sh
|
||||
```
|
||||
|
||||
`install-host-checks.sh --execute` 마지막에 `platform_backup_last_run_success{backup="k3s-datastore"} 1`과
|
||||
`{backup="gitea"} 1`이 보여야 한다. 리허설 둘은 각각 `K3S DATASTORE RESTORE DRILL PASS`, `GITEA RESTORE DRILL PASS`로 끝나야 한다.
|
||||
|
||||
## 복원 리허설
|
||||
|
||||
- k3s: 백업을 내려받아 sha256을 맞춘 뒤, 운영과 같은 버전의 k3s를 **네트워크 없는** 일회용 컨테이너로
|
||||
띄워 API가 뜨는지, Secret 수가 라이브와 비슷한지(±5), 모든 Secret이 복호화되는지 본다.
|
||||
`scripts/validate/test-k3s-datastore-restore-drill.sh`는 운영 데이터 없이 같은 경로를 시험한다(새 k3s에
|
||||
시험 Secret 셋 → 암호화돼 저장된 것을 확인 → 복원).
|
||||
- Gitea: dump를 풀어 `app.ini`(INTERNAL_TOKEN·SECRET_KEY), 모든 저장소의 `git fsck`, 라이브와 저장소 수를 본다.
|
||||
- 분기마다 한 번 돌린다. 성공하면 `platform_backup_restore_drill_last_success_timestamp_seconds`가 갱신된다.
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["s3:GetBucketLocation", "s3:ListBucket"],
|
||||
"Resource": ["arn:aws:s3:::platform-host-backups"]
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
|
||||
"Resource": ["arn:aws:s3:::platform-host-backups/*"]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- prometheus-rules.yaml
|
||||
@@ -0,0 +1,88 @@
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: platform-host-checks
|
||||
namespace: observability
|
||||
labels:
|
||||
observability.hyeonworks.com/instance: home
|
||||
app.kubernetes.io/part-of: platform
|
||||
spec:
|
||||
groups:
|
||||
- name: platform-host-backups
|
||||
rules:
|
||||
- alert: PlatformK3sDatastoreBackupTooOld
|
||||
expr: time() - max(platform_backup_last_success_timestamp_seconds{backup="k3s-datastore"}) > 2 * 86400
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: k3s datastore has not been backed up for more than 2 days
|
||||
description: Check `journalctl -u platform-k3s-datastore-backup` on desktop. Daily timer, 03:30 KST.
|
||||
- alert: PlatformGiteaBackupTooOld
|
||||
expr: time() - max(platform_backup_last_success_timestamp_seconds{backup="gitea"}) > 8 * 86400
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: Gitea has not been backed up for more than 8 days
|
||||
description: Check `journalctl -u platform-gitea-backup` on desktop. Weekly timer, Sunday 04:00 KST.
|
||||
- alert: PlatformHostBackupFailed
|
||||
expr: max by (backup) (platform_backup_last_run_success) == 0
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "The last {{ $labels.backup }} backup run failed"
|
||||
description: "Check `journalctl -u platform-{{ $labels.backup }}-backup` on desktop."
|
||||
- alert: PlatformHostBackupMetricsAbsent
|
||||
expr: absent(platform_backup_last_run_timestamp_seconds{backup="k3s-datastore"}) or absent(platform_backup_last_run_timestamp_seconds{backup="gitea"})
|
||||
for: 1d
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: Host backup metrics are missing
|
||||
description: The backup timers or the node-exporter textfile collector (/var/lib/node_exporter/textfile_collector) are not running.
|
||||
- name: platform-host-drift
|
||||
rules:
|
||||
- alert: PlatformLiveDriftDetected
|
||||
expr: max(platform_live_drift_objects) > 0
|
||||
for: 6h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "{{ $value }} live objects differ from platform-core origin/main"
|
||||
description: Run `scripts/validate/live-drift-check.sh --accepted scripts/validate/live-drift-accepted.txt` on desktop to list them.
|
||||
- alert: PlatformLiveDriftCheckStale
|
||||
expr: time() - max(platform_live_drift_last_run_timestamp_seconds) > 2 * 86400 or max(platform_live_drift_check_success) == 0
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: The daily live drift check has not completed
|
||||
description: Check `journalctl -u platform-live-drift-check` on desktop.
|
||||
- name: platform-k3s-certificates
|
||||
rules:
|
||||
- alert: PlatformK3sServingCertificateExpiry
|
||||
expr: (min by (certificate) (platform_k3s_certificate_not_after_seconds) - time()) < 100 * 86400
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "k3s {{ $labels.certificate }} serving certificate expires in less than 100 days"
|
||||
description: k3s renews certificates only when it starts within 120 days of expiry. Plan a k3s restart window (docs/runbooks).
|
||||
- alert: PlatformK3sServingCertificateExpiry
|
||||
expr: (min by (certificate) (platform_k3s_certificate_not_after_seconds) - time()) < 14 * 86400
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "k3s {{ $labels.certificate }} serving certificate expires in less than 14 days"
|
||||
description: Restart k3s (it renews certificates within 120 days of expiry) or run `k3s certificate rotate`.
|
||||
- alert: PlatformK3sCertificateCheckStale
|
||||
expr: time() - max(platform_k3s_certificate_check_timestamp_seconds) > 2 * 86400
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: The daily k3s certificate expiry check has not run
|
||||
description: Check `journalctl -u platform-k3s-cert-expiry` on desktop.
|
||||
@@ -0,0 +1,176 @@
|
||||
# 플랫폼 PostgreSQL 백업
|
||||
|
||||
`platform-data/platform-postgres` 하나에 Gitea, Keycloak, tech-log의 DB가 같이 있다
|
||||
(2026-11-11: gitea 608MB, keycloak 14MB, techlog 14MB). 2026-11-11까지 백업 설정도,
|
||||
ScheduledBackup도 없었다. 그 PV는 루트 SSD 하나에만 있었다.
|
||||
|
||||
이 폴더는 매일 base backup과 WAL 연속 보관을 AIStor(HDD `sdb3`)에 두고, 30일 안의
|
||||
아무 시점으로나 되돌릴 수 있게 한다.
|
||||
|
||||
## 무엇이 어디로 가나
|
||||
|
||||
```text
|
||||
platform-postgres-1 (SSD)
|
||||
├─ WAL 파일이 찰 때마다, 그리고 5분마다 ─▶ s3://platform-postgres-backups/platform-postgres/wals/
|
||||
└─ 매일 03:00 KST base backup ─────────▶ s3://platform-postgres-backups/platform-postgres/base/
|
||||
AIStor ObjectStore minio-aistor (HDD sdb3, aistor-local-xfs-retain)
|
||||
```
|
||||
|
||||
| 파일 | 하는 일 |
|
||||
| --- | --- |
|
||||
| `cluster-backup-patch.yaml` | live Cluster에 `spec.backup`(보관소, 30일 보존, gzip)을 더한다 |
|
||||
| `scheduled-backup.yaml` | 매일 18:00 UTC(03:00 KST) base backup. 만들자마자 한 번 |
|
||||
| `object-storage-counterpart.yaml` | `object-storage` 쪽 ingress: platform-data의 PostgreSQL 파드 → AIStor 9000 |
|
||||
| `bucket-policy.json` | 백업 사용자가 자기 bucket만 읽고 쓰는 정책 |
|
||||
| `prometheus-rule.yaml` | 백업 경보 둘(아래). 관측 core 렌더러가 백업 규칙을 core 범위 밖으로 두므로 여기 둔다 |
|
||||
| `restore-drill/cluster.yaml` | 복원 리허설용 임시 Cluster |
|
||||
| `../../scripts/bootstrap/create-postgres-backup-credentials.sh` | bucket·정책·사용자·Secret 생성 |
|
||||
|
||||
## 고른 방법
|
||||
|
||||
- **내장 `barmanObjectStore`.** CloudNativePG 1.26부터 deprecated이고 Barman Cloud Plugin이
|
||||
권장이다. 하지만 plugin은 cert-manager가 필요한데 이 클러스터에는 없다(2026-11-11 CRD 없음).
|
||||
백업이 하나도 없는 지금은 새 operator 두 개(cert-manager, plugin)를 먼저 세우는 것보다 지금
|
||||
operator(1.30.0)가 바로 지원하는 쪽이 빠르고 위험이 작다. bucket 구조가 같아 plugin으로
|
||||
옮겨도 백업을 그대로 쓴다.
|
||||
- **기한:** operator 경고는 「1.31.0에서 완전히 제거」라고 한다(문서는 1.32.0). 그래서
|
||||
**CNPG를 1.31로 올리기 전에** Barman Cloud Plugin으로 옮긴다.
|
||||
- **백업은 HDD에만 둔다(소유자 지시 2026-11-12: 「운영 db 백업은 SSD 말고 HDD쪽에」).** 보관소는
|
||||
AIStor다. AIStor의 데이터 볼륨은 하나뿐이고, PV `aistor-data-local-pv` → `/srv/k3s/aistor` →
|
||||
`/dev/sdb3`(xfs, WDC WD20EZAZ 2TB HDD, `lsblk` `ROTA=1`)이다. 루트 SSD(`/dev/sdc1`, Crucial MX500)에는
|
||||
백업이 가지 않는다. barman-cloud는 WAL과 base backup을 SSD에 따로 모으지 않고 곧바로 올린다.
|
||||
- 버린 대안: HDD 보관소 `/mnt/hdd-archive`(`/dev/sdb2`, NTFS). POSIX 권한이 없어 PostgreSQL 파일을
|
||||
그대로 두기 어렵고, barman-cloud가 쓰는 S3 API도 없다. 거기에 `pg_dump`를 두면 매일 한 시점만
|
||||
남아 30일 안의 아무 시점으로 되돌릴 수 없다. AIStor는 같은 HDD 위에서 bucket별 사용자·정책과
|
||||
quota를 준다.
|
||||
- SSD를 잃어도 백업은 남는다. desktop 자체를 잃으면 백업도 같이 잃는다. 다른 기계나 외부로 복사하는
|
||||
일은 남은 과제다.
|
||||
- 복원 리허설의 임시 Cluster는 local-path(루트 SSD) 5Gi를 잠깐 쓰고 바로 지운다. 백업 보관이 아니라
|
||||
복원 결과를 확인하는 자리다. HDD에는 동적으로 볼륨을 만드는 StorageClass가 없고, `sdb3`는 통째로
|
||||
AIStor가 쓴다.
|
||||
- **보존 30일, quota 100GiB.** 압축한 base backup이 하루 수백 MB, WAL은 5분마다 한 파일
|
||||
(`archive_timeout` 기본값)이라 30일에 수십 GB 안쪽이다.
|
||||
- **볼륨 스냅숏은 쓰지 않는다.** `ssd-local-postgres-retain`은 정적 local PV라 CSI 스냅숏이 없다.
|
||||
|
||||
## 적용 (소유자, 순서대로)
|
||||
|
||||
전제: 이 PR이 병합된 platform-core checkout, desktop의 kube context `default`.
|
||||
|
||||
**백업 설정은 `kubectl patch`로만 더한다.** `services/platform-postgres` 렌더를 다시 적용하면
|
||||
`priorityClassName`과 Guaranteed resources도 같이 들어가 PostgreSQL이 재시작된다. 그것은
|
||||
k3s 재시작 창(`scripts/bootstrap/apply-k3s-restart-bundle.sh`)에서 한다. 2026-11-11까지는
|
||||
live `pg_hba`의 `techlog` 두 줄이 원본에 없어서 다시 적용하면 tech-log 백엔드가 끊겼다.
|
||||
이 브랜치에서 원본에 넣었으므로(`fix(platform-postgres): keep the tech-log pg_hba rules`)
|
||||
지금 렌더의 `kubectl diff`에는 pg_hba 차이가 없다.
|
||||
|
||||
1. bucket, 정책, 사용자, Secret을 만든다. 화면에 자격 증명이 나오지 않는다.
|
||||
|
||||
```bash
|
||||
bash scripts/bootstrap/create-postgres-backup-credentials.sh
|
||||
```
|
||||
|
||||
마지막 줄에 `PLATFORM_POSTGRES_BACKUP_BOOTSTRAP_PASS`가 나와야 한다. 새 사용자가 자기
|
||||
bucket에 쓰고 읽고 지울 수 있고, `loki` bucket은 못 보는 것까지 확인한다.
|
||||
|
||||
2. AIStor 쪽 ingress를 연다. 그 전에는 PostgreSQL이 9000에 닿지 못한다.
|
||||
|
||||
```bash
|
||||
kubectl apply -f services/platform-postgres-backup/object-storage-counterpart.yaml
|
||||
```
|
||||
|
||||
3. live Cluster에 백업 설정을 더한다. PostgreSQL은 재시작하지 않는다.
|
||||
|
||||
```bash
|
||||
kubectl -n platform-data patch cluster platform-postgres --type merge \
|
||||
--patch-file services/platform-postgres-backup/cluster-backup-patch.yaml --dry-run=server -o name
|
||||
kubectl -n platform-data patch cluster platform-postgres --type merge \
|
||||
--patch-file services/platform-postgres-backup/cluster-backup-patch.yaml
|
||||
kubectl -n platform-data get cluster platform-postgres -o jsonpath='{.spec.postgresql.pg_hba}{"\n"}'
|
||||
```
|
||||
|
||||
마지막 줄에 `techlog` 두 줄이 있어야 한다. 몇 분 뒤 WAL이 올라가는지 본다.
|
||||
|
||||
```bash
|
||||
kubectl -n platform-data get cluster platform-postgres \
|
||||
-o jsonpath='{.status.conditions[?(@.type=="ContinuousArchiving")].status}{" "}{.status.firstRecoverabilityPoint}{"\n"}'
|
||||
```
|
||||
|
||||
4. 매일 백업을 건다. 만들자마자 첫 base backup이 돈다.
|
||||
|
||||
```bash
|
||||
kubectl apply -k services/platform-postgres-backup # ScheduledBackup, AIStor ingress, 경보
|
||||
kubectl -n platform-data get backup -w
|
||||
```
|
||||
|
||||
`PHASE`가 `completed`가 되면 된다. `failed`면
|
||||
`kubectl -n platform-data describe backup <이름>`의 마지막 줄을 본다.
|
||||
|
||||
5. 아래 「복원 리허설」을 한 번 한다.
|
||||
|
||||
경보(`prometheus-rule.yaml`, 4단계에서 같이 들어간다. platform-data는 Prometheus가 규칙을 읽는 namespace다):
|
||||
|
||||
- `PlatformPostgresBackupMissing`: 26시간 안에 끝난 base backup이 없다. 위 4단계 전까지는 계속
|
||||
울린다(백업이 정말 없으니까).
|
||||
- `PlatformPostgresWALArchiveBacklog`: 보관을 기다리는 WAL이 10개를 넘은 채 15분.
|
||||
보관이 막히면 WAL이 SSD 볼륨에 쌓여 PostgreSQL이 멈춘다. 2·3단계의 순서가 바뀌었거나
|
||||
Secret이 틀리면 이것이 울린다.
|
||||
|
||||
### 되돌리기
|
||||
|
||||
```bash
|
||||
kubectl delete -k services/platform-postgres-backup # ScheduledBackup, ingress, 경보
|
||||
kubectl -n platform-data patch cluster platform-postgres --type json -p '[{"op":"remove","path":"/spec/backup"}]'
|
||||
```
|
||||
|
||||
bucket, 사용자, Secret, 이미 올린 백업은 지우지 않는다. 지울 때는 소유자가 `mc`로 직접 한다.
|
||||
|
||||
## 복원 리허설
|
||||
|
||||
백업이 실제로 복원되는지는 복원해 봐야 안다. 첫 base backup이 끝난 뒤 한 번, 그 뒤로는 CNPG나
|
||||
PostgreSQL을 올릴 때마다 한다. 운영 Cluster는 건드리지 않는다. 임시 Cluster는 bucket을 읽기만 한다
|
||||
(`backup` 칸이 없다).
|
||||
|
||||
1. 임시 Cluster를 만든다. local-path 5Gi(루트 SSD)를 잠깐 쓴다.
|
||||
|
||||
```bash
|
||||
kubectl apply -f services/platform-postgres-backup/restore-drill/cluster.yaml
|
||||
kubectl -n platform-data wait cluster/platform-postgres-restore-drill \
|
||||
--for=condition=Ready --timeout=20m
|
||||
```
|
||||
|
||||
2. 두 쪽의 표마다 행 수를 비교한다. 운영은 그 사이에도 쓰이므로 활동이 많은 표(Gitea의
|
||||
`action`, `notice` 등)는 조금 다를 수 있다. 표가 없거나 크게 다르면 실패다.
|
||||
|
||||
```bash
|
||||
count_rows() { # $1 = cluster, $2 = database
|
||||
kubectl -n platform-data exec "$1-1" -c postgres -- psql -At -d "$2" -c \
|
||||
"select string_agg(format('select %L, count(*) from %I.%I', schemaname||'.'||relname, schemaname, relname), ' union all ')
|
||||
from pg_stat_user_tables" |
|
||||
kubectl -n platform-data exec -i "$1-1" -c postgres -- psql -At -d "$2" | sort
|
||||
}
|
||||
for db in gitea keycloak techlog; do
|
||||
diff <(count_rows platform-postgres "$db") <(count_rows platform-postgres-restore-drill "$db") \
|
||||
&& echo "$db: same" || echo "$db: differs (see above)"
|
||||
done
|
||||
```
|
||||
|
||||
3. 복원된 시점을 기록한다.
|
||||
|
||||
```bash
|
||||
kubectl -n platform-data exec platform-postgres-restore-drill-1 -c postgres -- \
|
||||
psql -At -c "select now(), pg_last_xact_replay_timestamp()"
|
||||
```
|
||||
|
||||
4. 임시 Cluster를 지운다. PVC도 같이 지워진다.
|
||||
|
||||
```bash
|
||||
kubectl -n platform-data delete cluster platform-postgres-restore-drill
|
||||
```
|
||||
|
||||
결과(날짜, 복원된 시점, 표 비교, 걸린 시간)는 아래 「기록」에 남긴다.
|
||||
|
||||
## 기록
|
||||
|
||||
- 2026-11-11: 설계와 원본만. bucket·Secret·적용·리허설은 하지 않았다(소유자 단계).
|
||||
`kubectl apply --dry-run=server`로 ScheduledBackup, ingress, 리허설 Cluster가 admission을
|
||||
통과했고, `kubectl patch --dry-run=server`로 patch 뒤에도 `techlog` pg_hba 두 줄이 남는 것을 봤다.
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:GetBucketLocation",
|
||||
"s3:ListBucket",
|
||||
"s3:ListBucketMultipartUploads"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::platform-postgres-backups"
|
||||
]
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:GetObject",
|
||||
"s3:PutObject",
|
||||
"s3:DeleteObject",
|
||||
"s3:AbortMultipartUpload",
|
||||
"s3:ListMultipartUploadParts"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::platform-postgres-backups/*"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
# Merge patch for the live Cluster. Apply it with kubectl patch, not by
|
||||
# re-applying services/platform-postgres: the live Cluster carries pg_hba lines
|
||||
# for techlog that the Phase 1 source does not have (README.md, "적용").
|
||||
spec:
|
||||
backup:
|
||||
# Base backups and WAL older than the newest point that still lets a
|
||||
# restore reach 30 days back are deleted by the operator.
|
||||
retentionPolicy: "30d"
|
||||
barmanObjectStore:
|
||||
# AIStor ObjectStore on the HDD (sdb3, aistor-local-xfs-retain), not the
|
||||
# root SSD that holds the database itself.
|
||||
destinationPath: s3://platform-postgres-backups/
|
||||
endpointURL: http://minio.object-storage.svc.cluster.local
|
||||
s3Credentials:
|
||||
accessKeyId:
|
||||
name: platform-postgres-backup-s3
|
||||
key: ACCESS_KEY_ID
|
||||
secretAccessKey:
|
||||
name: platform-postgres-backup-s3
|
||||
key: ACCESS_SECRET_KEY
|
||||
wal:
|
||||
compression: gzip
|
||||
maxParallel: 2
|
||||
data:
|
||||
compression: gzip
|
||||
jobs: 2
|
||||
@@ -0,0 +1,17 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
# Applied after platform-postgres (Phase 1) and AIStor (Phase 2) exist and
|
||||
# after scripts/bootstrap/create-postgres-backup-credentials.sh has created
|
||||
# the bucket and the platform-data/platform-postgres-backup-s3 Secret.
|
||||
# cluster-backup-patch.yaml and restore-drill/ are applied by hand; see README.md.
|
||||
resources:
|
||||
- object-storage-counterpart.yaml
|
||||
- scheduled-backup.yaml
|
||||
- prometheus-rule.yaml
|
||||
|
||||
labels:
|
||||
- pairs:
|
||||
app.kubernetes.io/part-of: platform
|
||||
app.kubernetes.io/managed-by: kustomize
|
||||
includeSelectors: false
|
||||
@@ -0,0 +1,30 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: object-storage-allow-platform-postgres-backup
|
||||
namespace: object-storage
|
||||
labels:
|
||||
app.kubernetes.io/component: platform-postgres-backup
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
aistor.min.io/objectStore: minio-aistor
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
# barman-cloud in the PostgreSQL pods writes base backups and WAL; the
|
||||
# restore drill cluster reads them.
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: platform-data
|
||||
podSelector:
|
||||
matchExpressions:
|
||||
- key: cnpg.io/cluster
|
||||
operator: In
|
||||
values:
|
||||
- platform-postgres
|
||||
- platform-postgres-restore-drill
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9000
|
||||
@@ -0,0 +1,46 @@
|
||||
# Backup alerts for platform-postgres. They live with the backup and not in
|
||||
# the observability core, whose renderer keeps backup rules out of its scope
|
||||
# (scripts/validate/render-observability-core.sh, no_unscoped_monitors_rules).
|
||||
# Prometheus loads PrometheusRules from namespaces labelled
|
||||
# observability.hyeonworks.com/enabled=true, which platform-data is.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: platform-postgres-backup
|
||||
namespace: platform-data
|
||||
labels:
|
||||
app.kubernetes.io/name: platform-postgres
|
||||
app.kubernetes.io/component: backup
|
||||
observability.hyeonworks.com/instance: home
|
||||
spec:
|
||||
groups:
|
||||
- name: platform.backup
|
||||
rules:
|
||||
# cnpg_collector_last_available_backup_timestamp is 0 until the first
|
||||
# base backup, so a cluster without backups fires as well.
|
||||
- alert: PlatformPostgresBackupMissing
|
||||
expr: >-
|
||||
time() - max by (job, namespace) (
|
||||
cnpg_collector_last_available_backup_timestamp{job="platform-postgres",namespace="platform-data"}
|
||||
) > 26 * 3600
|
||||
for: 30m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: platform-postgres에 26시간 안에 끝난 백업이 없음
|
||||
description: platform-postgres의 마지막 성공한 base backup이 26시간보다 오래됐거나 없습니다. 매일 03:00 KST 백업(ScheduledBackup platform-postgres-daily)을 확인하세요.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/platform-core/src/branch/main/services/platform-postgres-backup/README.md
|
||||
# A failing archive_command keeps WAL in pg_wal, which then fills the
|
||||
# PostgreSQL volume.
|
||||
- alert: PlatformPostgresWALArchiveBacklog
|
||||
expr: >-
|
||||
max by (job, namespace) (
|
||||
cnpg_collector_pg_wal_archive_status{job="platform-postgres",namespace="platform-data",value="ready"}
|
||||
) > 10
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: platform-postgres의 WAL이 보관소로 올라가지 않음
|
||||
description: 보관을 기다리는 WAL 파일이 10개를 넘은 채 15분이 지났습니다. 보관이 막히면 WAL이 PostgreSQL 볼륨에 쌓여 디스크가 찹니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/platform-core/src/branch/main/services/platform-postgres-backup/README.md
|
||||
@@ -0,0 +1,49 @@
|
||||
# Restore drill: a throwaway Cluster bootstrapped from the platform-postgres
|
||||
# backups. It has no backup section, so it never writes to the bucket.
|
||||
# Apply, compare, delete; see ../README.md ("복원 리허설").
|
||||
apiVersion: postgresql.cnpg.io/v1
|
||||
kind: Cluster
|
||||
metadata:
|
||||
name: platform-postgres-restore-drill
|
||||
namespace: platform-data
|
||||
labels:
|
||||
app.kubernetes.io/name: platform-postgres-restore-drill
|
||||
app.kubernetes.io/component: restore-drill
|
||||
spec:
|
||||
instances: 1
|
||||
imageName: ghcr.io/cloudnative-pg/postgresql:17.9-standard-trixie
|
||||
storage:
|
||||
# Dynamic local-path volume on the root SSD, deleted with the Cluster.
|
||||
storageClass: local-path
|
||||
size: 5Gi
|
||||
bootstrap:
|
||||
recovery:
|
||||
source: platform-postgres
|
||||
externalClusters:
|
||||
- name: platform-postgres
|
||||
barmanObjectStore:
|
||||
destinationPath: s3://platform-postgres-backups/
|
||||
endpointURL: http://minio.object-storage.svc.cluster.local
|
||||
serverName: platform-postgres
|
||||
s3Credentials:
|
||||
accessKeyId:
|
||||
name: platform-postgres-backup-s3
|
||||
key: ACCESS_KEY_ID
|
||||
secretAccessKey:
|
||||
name: platform-postgres-backup-s3
|
||||
key: ACCESS_SECRET_KEY
|
||||
wal:
|
||||
maxParallel: 2
|
||||
managed:
|
||||
services:
|
||||
disabledDefaultServices:
|
||||
- ro
|
||||
- r
|
||||
affinity:
|
||||
enablePodAntiAffinity: false
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 1Gi
|
||||
@@ -0,0 +1,17 @@
|
||||
apiVersion: postgresql.cnpg.io/v1
|
||||
kind: ScheduledBackup
|
||||
metadata:
|
||||
name: platform-postgres-daily
|
||||
namespace: platform-data
|
||||
labels:
|
||||
app.kubernetes.io/name: platform-postgres
|
||||
app.kubernetes.io/component: backup
|
||||
spec:
|
||||
# Seconds field first. The operator runs in UTC: 18:00 UTC is 03:00 KST.
|
||||
schedule: "0 0 18 * * *"
|
||||
# Take the first base backup as soon as this object is created.
|
||||
immediate: true
|
||||
backupOwnerReference: self
|
||||
method: barmanObjectStore
|
||||
cluster:
|
||||
name: platform-postgres
|
||||
@@ -21,9 +21,16 @@ platform-postgres-rw.platform-data.svc.cluster.local:5432
|
||||
업그레이드한다.
|
||||
|
||||
Cluster의 사용자 정의 `pg_hba` 규칙은 `gitea`가 `gitea` Database에만,
|
||||
`keycloak`이 `keycloak` Database에만 인증하도록 허용한 뒤 각 Role의 다른 모든
|
||||
Database 접근을 거부한다. 이 규칙들은 CloudNativePG의 기본 fallback 인증 규칙보다
|
||||
앞에 배치된다.
|
||||
`keycloak`이 `keycloak` Database에만, `techlog`가 `techlog` Database에만 인증하도록
|
||||
허용한 뒤 각 Role의 다른 모든 Database 접근을 거부한다. 이 규칙들은
|
||||
CloudNativePG의 기본 fallback 인증 규칙보다 앞에 배치된다.
|
||||
|
||||
`techlog` 규칙 쌍은 운영 tech-log(hyeonworks.com)가 쓰는 것으로, 클러스터에는
|
||||
이 저장소 밖에서 먼저 들어갔다(2026-11-11 점검에서 확인). 이 규칙 쌍이 없는 렌더를
|
||||
적용하면 운영 backend가 DB에 접속하지 못하므로 공통 Cluster에 둔다. `techlog`
|
||||
`DatabaseRole`·`Database`, `platform-data/techlog-db-credentials`와 tech-log용
|
||||
NetworkPolicy는 아직 어느 저장소에도 없다. 둘 곳은 소유자가 정한다
|
||||
(`docs/reviews/2026-11-11-k3s-maturity-review.md` 4절).
|
||||
|
||||
## Phase 1 Secret 계약
|
||||
|
||||
|
||||
@@ -24,6 +24,8 @@ spec:
|
||||
- host all gitea all reject
|
||||
- host keycloak keycloak all scram-sha-256
|
||||
- host all keycloak all reject
|
||||
- host techlog techlog all scram-sha-256
|
||||
- host all techlog all reject
|
||||
|
||||
storage:
|
||||
size: 20Gi
|
||||
@@ -37,10 +39,15 @@ spec:
|
||||
affinity:
|
||||
enablePodAntiAffinity: false
|
||||
|
||||
# The DB is the last thing that should go when the node runs short.
|
||||
# platform-critical comes from services/priority-classes. requests equal
|
||||
# limits so the pod is Guaranteed QoS and its oom_score_adj drops from 985
|
||||
# to -997 (k3s maturity review 2026-11-11, 7.2 items 2 and 3).
|
||||
priorityClassName: platform-critical
|
||||
resources:
|
||||
requests:
|
||||
cpu: 250m
|
||||
memory: 512Mi
|
||||
cpu: "2"
|
||||
memory: 4Gi
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: 4Gi
|
||||
|
||||
@@ -6,13 +6,14 @@
|
||||
|
||||
| 이름 | 값 | 선점 | 붙일 곳 |
|
||||
| --- | --- | --- | --- |
|
||||
| `platform-critical` | 1000000 | 낮은 것을 선점 | platform-postgres, Gitea, Keycloak, tech-log |
|
||||
| `platform-low` | -1000 | 선점 안 함 | Grafana, pgAdmin, Loki, Tempo |
|
||||
| `platform-critical` | 1000000 | 낮은 것을 선점 | Gitea(`services/gitea` values), Keycloak(CR `spec.scheduling`). platform-postgres는 W1, tech-log는 tech-log `deploy/` |
|
||||
| `platform-low` | -1000 | 선점 안 함 | Loki(`singleBinary`), Tempo. Grafana·pgAdmin은 chart에 최상위 키가 없어 아직 안 붙였다 |
|
||||
|
||||
- 아무것도 안 붙인 파드는 기본값 0이다. 시스템 클래스(`system-*`, 20억)는 그대로 가장 높다.
|
||||
- `globalDefault`는 둘 다 끈다. 새 파드가 모르게 우선순위를 얻지 않게 하려는 것이다.
|
||||
- 클래스를 만드는 것은 아무 파드도 바꾸지 않는다. **붙이는 것은 파드 재시작을 부르므로**
|
||||
k3s 재시작 창에 한다(`infrastructure/host/k3s/README.md`).
|
||||
k3s 재시작 창에 한다(`docs/runbooks/k3s-restart-window.md`).
|
||||
- **클래스를 먼저 만든다.** 없는 클래스를 가리키는 파드는 생성이 거절돼 Deployment가 새 파드를 못 만든다.
|
||||
|
||||
```sh
|
||||
kubectl apply -k services/priority-classes
|
||||
|
||||
Reference in New Issue
Block a user