From 7fe38428e332c22441d168fb0933930d8779df35 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Wed, 11 Nov 2026 21:51:59 +0900 Subject: [PATCH 01/16] feat(observability): alert on the root disk before kubelet acts The kubelet starts deleting unused images at 85% root use and evicts pods at 90%. The root disk warning fired at 85% and the critical at 90%, so both arrived when the kubelet was already acting. On 2026-11-09 the warning fired for 33 hours, then an eviction stopped the tech-log pods, the image GC deleted their imagePullPolicy: Never images, and production went down. Move the warning to 80% for 10m and the critical to 85% for 5m, and add PlatformRootFilesystemFillingFast (critical, for 15m): the 6h linear trend of free bytes falls below 10% of the disk, the eviction point, within 6 hours. On 2026-11-07 the disk lost 17G in 9 hours; this forecast would have fired hours before the 80% warning. The info alert at 75% stays. Add scripts/validate/test-platform-disk-alerts.sh, promtool unit tests for the usage bands and the forecast with the pinned Prometheus image. The existing rules test needs live metric inventories, so this one runs offline. Risk: the live rules change only when the observability access apply runs; this commit does not touch the cluster. The forecast can fire on a large build that the BuildKit GC removes minutes later; the 15m hold and the 6h window damp that. Verification: the new test failed against the old rules (no alert at 81%, 86% or for a 1-per-minute fill) and passes after the change; promtool check rules finds 9 rules. test-observability-alerting.sh passes its static contract, then stops at the helm render because helm v3.19.4 is not installed on the desktop. --- scripts/validate/test-platform-disk-alerts.sh | 153 ++++++++++++++++++ .../rules/platform/core-rules.yaml | 36 +++-- 2 files changed, 180 insertions(+), 9 deletions(-) create mode 100755 scripts/validate/test-platform-disk-alerts.sh diff --git a/scripts/validate/test-platform-disk-alerts.sh b/scripts/validate/test-platform-disk-alerts.sh new file mode 100755 index 0000000..ee20253 --- /dev/null +++ b/scripts/validate/test-platform-disk-alerts.sh @@ -0,0 +1,153 @@ +#!/usr/bin/env bash +# Unit tests for the root filesystem alerts in core-rules.yaml. +# +# The kubelet starts deleting unused images at 85% use and evicts pods at +# 90% (nodefs.available < 10%). These tests pin the alerts in front of those +# two points: warning at 80%, critical at 85%, and a forecast that fires when +# the 6h trend reaches the eviction point within 6 hours. +# +# Needs docker, jq and kubectl (client only, no cluster access). +set -euo pipefail + +readonly ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +readonly RULE_FILE="${ROOT_DIR}/services/observability/rules/platform/core-rules.yaml" +# Same pinned image as test-platform-observability-rules.sh. +readonly PROMETHEUS_IMAGE="quay.io/prometheus/prometheus:v3.13.2-distroless@sha256:ce95cfa77eff5aad28bd7a65aff19868cf78d9e17e4c254da7dfe22ade78318b" +readonly RUNBOOK_URL="https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md" + +temp_dir='' +cleanup() { + if [[ -n "$temp_dir" && -d "$temp_dir" ]]; then + rm -rf -- "$temp_dir" + fi +} +trap cleanup EXIT + +fail() { + printf 'FAIL: %s\n' "$*" >&2 + exit 1 +} + +for command in docker jq kubectl; do + command -v "$command" >/dev/null || fail "missing command: ${command}" +done + +temp_dir="$(mktemp -d)" +chmod 0755 "$temp_dir" + +kubectl create --dry-run=client --validate=false -f "$RULE_FILE" -o json | + jq '.spec' >"${temp_dir}/core.rules.json" + +# Every core alert keeps the runbook the other platform rules use. +jq -e --arg url "$RUNBOOK_URL" ' + all(.groups[].rules[] | select(has("alert")); .annotations.runbook_url == $url) +' "${temp_dir}/core.rules.json" >/dev/null || fail "core alert without the shared runbook_url" + +# node-exporter labels as scraped by serviceMonitor/observability-agent/node-exporter. +readonly SERIES_LABELS='job="prometheus-node-exporter",instance="10.42.0.95:9100",device="/dev/sdc1",fstype="ext4",mountpoint="/"' +readonly ALERT_LABELS=' job: prometheus-node-exporter + instance: 10.42.0.95:9100 + device: /dev/sdc1 + fstype: ext4 + mountpoint: /' + +usage_case() { + # name, used percent, eval minute, expected alerts (YAML list body or "[]") + local name=$1 used=$2 eval_minute=$3 expected=$4 + local avail=$((100 - used)) + cat <"${temp_dir}/disk.test.yaml" + +chmod 0444 "${temp_dir}"/* +docker run --rm \ + --network none \ + --volume "${temp_dir}:/work:ro" \ + --entrypoint /bin/promtool \ + "$PROMETHEUS_IMAGE" check rules /work/core.rules.json +docker run --rm \ + --network none \ + --volume "${temp_dir}:/work:ro" \ + --entrypoint /bin/promtool \ + "$PROMETHEUS_IMAGE" test rules /work/disk.test.yaml +printf 'PASS: root filesystem usage and forecast alerts\n' diff --git a/services/observability/rules/platform/core-rules.yaml b/services/observability/rules/platform/core-rules.yaml index 846b484..0a363e2 100644 --- a/services/observability/rules/platform/core-rules.yaml +++ b/services/observability/rules/platform/core-rules.yaml @@ -41,14 +41,14 @@ spec: node_filesystem_avail_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"} / node_filesystem_size_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"} - ) > 85 - for: 15m + ) > 80 + for: 10m labels: severity: warning - threshold: "85" + threshold: "80" annotations: - summary: 루트 파일시스템 사용률이 85%를 초과함 - description: 노드 루트 파일시스템 사용률이 경고 임계치인 85%를 15분간 초과했습니다. + summary: 루트 파일시스템 사용률이 80%를 초과함 + description: 노드 루트 파일시스템 사용률이 80%를 10분간 초과했습니다. 85%에서 kubelet이 이미지를 지우기 시작합니다. runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md - alert: PlatformRootFilesystemUsage expr: >- @@ -56,14 +56,32 @@ spec: node_filesystem_avail_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"} / node_filesystem_size_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"} - ) > 90 + ) > 85 for: 5m labels: severity: critical - threshold: "90" + threshold: "85" annotations: - summary: 루트 파일시스템 사용률이 90%를 초과함 - description: 노드 루트 파일시스템 사용률이 심각 임계치인 90%를 5분간 초과했습니다. + summary: 루트 파일시스템 사용률이 85%를 초과함 + description: 노드 루트 파일시스템 사용률이 85%를 5분간 초과했습니다. kubelet이 안 쓰는 이미지를 지우는 중이고 90%에서 파드를 내보냅니다. + runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md + # kubelet evicts pods when nodefs.available drops below 10%. Fire when + # the 6h trend says that happens within the next 6 hours, so a fast + # fill (17G in 9h on 2026-11-07) is caught before the 80% warning. + - alert: PlatformRootFilesystemFillingFast + expr: >- + predict_linear( + node_filesystem_avail_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}[6h], + 6 * 3600 + ) + < + 0.10 * node_filesystem_size_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"} + for: 15m + labels: + severity: critical + annotations: + summary: 루트 파일시스템이 6시간 안에 축출 문턱에 닿을 것으로 예측됨 + description: 최근 6시간 추세대로면 루트 파일시스템 여유가 6시간 안에 10% 아래로 내려갑니다. 10% 아래에서 kubelet이 파드를 내보냅니다. runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md - alert: PlatformRootFilesystemInodes expr: >- From 3bca2cf841bfb8992a0e513a4545c58d175266d2 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Wed, 11 Nov 2026 21:51:59 +0900 Subject: [PATCH 02/16] feat(host): cap the host Docker build cache The host Docker daemon had no daemon.json. The default BuildKit policy on this 228GiB disk keeps 21.42GiB of build cache untouched and allows 169.5GiB (docker buildx inspect default), so agent gate builds piled up about 23G in two days before the 2026-11-09 disk incident. Add infrastructure/host/docker/daemon.json with builder.gc defaultReservedSpace 5GB, defaultMaxUsedSpace 15GB and defaultMinFreeSpace 35GB. In Docker 29 the old defaultKeepStorage key means defaultReservedSpace, a floor and not a cap, so the cap is defaultMaxUsedSpace. 35GiB free is about 85% use, where the kubelet starts deleting images; below it the cache shrinks to the floor first. Moving the Docker data root to the HDD was rejected by the owner because builds get slower. Risk: applying needs sudo and a Docker restart, which restarts the runner, workmachine and cloudflared and fails running CI jobs and gates. The host apply guide added in this series says when that is safe. dockerd --validate does not check keys inside builder.gc, so the guide checks the live policy with docker buildx inspect. Verification: dockerd --validate passes for the file on Docker 29.6.1; it also passes a typo inside builder.gc and rejects an unknown top-level key, which is why the live check is in the guide. --- infrastructure/host/docker/daemon.json | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 infrastructure/host/docker/daemon.json diff --git a/infrastructure/host/docker/daemon.json b/infrastructure/host/docker/daemon.json new file mode 100644 index 0000000..8c52b50 --- /dev/null +++ b/infrastructure/host/docker/daemon.json @@ -0,0 +1,10 @@ +{ + "builder": { + "gc": { + "enabled": true, + "defaultReservedSpace": "5GB", + "defaultMaxUsedSpace": "15GB", + "defaultMinFreeSpace": "35GB" + } + } +} From a5f0bff26e61b390eed9a73032dc4b2c76fa1bc4 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Wed, 11 Nov 2026 21:52:39 +0900 Subject: [PATCH 03/16] feat(ci): cap the runner Docker and prune it weekly The Docker daemon inside the gitea-runner container keeps every job image and all build cache in the gitea-runner_docker volume on the root SSD that production k3s also uses. It grew to 16G in the two days after the runner was installed and held 12.43G of images and 8.44G of build cache on 2026-11-11. The README only said to look once a week and prune by hand. Mount daemon.json into the runner so the inner BuildKit GC keeps 2GB to 10GB of build cache, and shrinks to 2GB while the disk has less than 35GB free. Add gitea-runner-prune.service and .timer, a host systemd timer for Sunday 04:30 KST that removes unused images created more than 7 days ago and build cache older than 3 days. The service runs as donghyeon in the docker group, so it needs root only to install. A user timer was rejected because lingering is off for donghyeon and user units stop with the last login session. Risk: image prune filters on image creation time, so the pinned job image is deleted weekly and the next job pulls about 543MB again. Recreating the runner to pick up daemon.json cancels a running job, so it is done only while the runner is idle. Verification: systemd-analyze verify passes for both units, and the calendar expression's next run is Sun 2026-11-15 04:30 KST. docker compose config passes; dockerd --validate inside the runner (Docker 29.9.3) accepts daemon.json. --- infrastructure/ci/gitea-runner/README.md | 81 ++++++++++++------- infrastructure/ci/gitea-runner/compose.yaml | 2 + infrastructure/ci/gitea-runner/daemon.json | 10 +++ .../gitea-runner/gitea-runner-prune.service | 17 ++++ .../ci/gitea-runner/gitea-runner-prune.timer | 12 +++ 5 files changed, 94 insertions(+), 28 deletions(-) create mode 100644 infrastructure/ci/gitea-runner/daemon.json create mode 100644 infrastructure/ci/gitea-runner/gitea-runner-prune.service create mode 100644 infrastructure/ci/gitea-runner/gitea-runner-prune.timer diff --git a/infrastructure/ci/gitea-runner/README.md b/infrastructure/ci/gitea-runner/README.md index cf44fff..eba2561 100644 --- a/infrastructure/ci/gitea-runner/README.md +++ b/infrastructure/ci/gitea-runner/README.md @@ -76,43 +76,66 @@ Gradle·Docker 게이트는 `flock /tmp/claude-gates/heavy.lock`으로 서로 ## 디스크와 정리 안쪽 daemon의 데이터(job 이미지, Testcontainers 이미지, 빌드 cache)는 이름 있는 -볼륨 `gitea-runner_docker`에 쌓인다. desktop root 디스크는 228G 중 182G를 쓰고 -있다(2026-11-07, 여유 34G). 예상 사용량은 다음과 같다. +볼륨 `gitea-runner_docker`에 쌓인다. 이 볼륨은 desktop 루트 SSD에 있고, 운영 k3s와 +같은 디스크다. 설치 이틀 만에 16G가 됐다(2026-11-09: 이미지 12.35G, 빌드 cache 3.4G). +2026-11-11에는 이미지 12.43G, 빌드 cache 8.44G였다. -| 항목 | 크기 | -| --- | --- | -| runner 이미지(`gitea/runner:5.4.1-dind`) | 약 133MB | -| job 이미지(`runner-images:ubuntu-24.04`) | amd64 압축 543MB, 안쪽 daemon에서 2.325GB(2026-11-07 실측) | -| Testcontainers 이미지(PostgreSQL, Ryuk 등) | 수백 MB | -| actions/cache(`/data/cache`) | 상한 5GB | +상한은 두 가지로 건다. -runner는 job이 끝나면 job 컨테이너와 proxy로 만든 컨테이너·네트워크·볼륨을 -지우지만(`container.sweep: true`) 이미지는 지우지 않는다. 정리 주기는 다음과 같다. +| 무엇 | 방법 | 값 | +| --- | --- | --- | +| 빌드 cache | `daemon.json`의 BuildKit GC. 안쪽 daemon이 스스로 지운다 | 최소 2GB, 상한 10GB. 디스크 여유가 35GB 아래면 2GB까지 | +| job 이미지 | 호스트 systemd timer `gitea-runner-prune.timer`. 주 1회 | 만든 지 7일 지난 안 쓰는 이미지, 3일 지난 빌드 cache | +| actions/cache | `config.yaml`의 `cache.size_limit` | 5GB | -- 매주 한 번 사용량을 본다. +- BuildKit 키의 뜻과 값을 고른 이유는 [호스트 설정](../../host/README.md)의 + 「BuildKit 상한 값」에 있다. 호스트와 같은 키에 더 작은 값을 쓴다. +- Docker daemon은 이미지를 스스로 지우지 않는다. 그래서 이미지는 timer로 지운다. +- `image prune --filter until=168h`는 이미지를 **만든** 시각으로 거른다. 마지막으로 쓴 + 시각이 아니다. 그래서 고정한 job 이미지(`runner-images:ubuntu-24.04`, 2026-11-03 생성)도 + 지워지고, 다음 job이 압축 약 543MB를 다시 받는다. 실행 중인 job 컨테이너가 쓰는 + 이미지는 지워지지 않는다. +- timer는 일요일 04:30(KST)에 돈다. 그 시각에 꺼져 있었으면 다음 부팅 때 돈다 + (`Persistent=true`). service는 `donghyeon` 사용자(docker group)로 돌아 root가 필요 없다. - ```bash - docker system df -v | grep gitea-runner - docker exec gitea-runner docker system df - ``` +설치(소유자, sudo 한 번): -- 안쪽 사용량이 15GB를 넘거나 root 디스크 여유가 20G 아래로 내려가면 job이 없을 때 - 안쪽 이미지를 정리한다. 고정된 job 이미지도 지워지므로 다음 job이 압축 약 543MB를 - 다시 받는다. +```bash +sudo install -m 0644 infrastructure/ci/gitea-runner/gitea-runner-prune.service \ + infrastructure/ci/gitea-runner/gitea-runner-prune.timer /etc/systemd/system/ +sudo systemctl daemon-reload +sudo systemctl enable --now gitea-runner-prune.timer +systemctl list-timers gitea-runner-prune.timer +``` - ```bash - docker exec gitea-runner docker image prune -a -f - docker exec gitea-runner docker builder prune -f - ``` +바로 한 번 돌리려면 job이 없을 때 `sudo systemctl start gitea-runner-prune.service`를 +실행하고 `journalctl -u gitea-runner-prune.service -n 30`으로 결과를 본다. -- 볼륨 `gitea-runner_docker` 자체를 지우는 것은 runner를 완전히 지울 때만 한다 - (「되돌리기」). 등록 파일이 있는 `gitea-runner_data`와 섞지 않는다. +되돌리기: + +```bash +sudo systemctl disable --now gitea-runner-prune.timer +sudo rm /etc/systemd/system/gitea-runner-prune.service /etc/systemd/system/gitea-runner-prune.timer +sudo systemctl daemon-reload +``` + +사용량 보기: + +```bash +docker system df -v | grep gitea-runner +docker exec gitea-runner docker system df +docker exec gitea-runner docker buildx inspect default | sed -n '/GC Policy rule#1/,/Min Free/p' +``` + +볼륨 `gitea-runner_docker` 자체를 지우는 것은 runner를 완전히 지울 때만 한다 +(「되돌리기」). 등록 파일이 있는 `gitea-runner_data`와 섞지 않는다. ## 파일과 위치 | 무엇 | 위치 | | --- | --- | -| 원본 설정 | 이 디렉터리의 `compose.yaml`, `config.yaml` | +| 원본 설정 | 이 디렉터리의 `compose.yaml`, `config.yaml`, `daemon.json` | +| 주 1회 정리 | 이 디렉터리의 `gitea-runner-prune.service`·`.timer` → `/etc/systemd/system/` | | desktop 실행 폴더 | `/home/donghyeon/.local/share/gitea-runner/` (`compose.yaml`, `config.yaml`, `secrets/`) | | 등록 파일 `.runner` | Docker 볼륨 `gitea-runner_data`의 `/data/.runner` (runner 자격 증명, Secret으로 취급) | | job 이미지·안쪽 daemon 데이터 | Docker 볼륨 `gitea-runner_docker` (`/var/lib/docker`) | @@ -133,8 +156,10 @@ runner는 job이 끝나면 job 컨테이너와 proxy로 만든 컨테이너·네 ```bash install -d -m 0700 ~/.local/share/gitea-runner ~/.local/share/gitea-runner/secrets - install -m 0644 infrastructure/ci/gitea-runner/compose.yaml infrastructure/ci/gitea-runner/config.yaml ~/.local/share/gitea-runner/ - sha256sum infrastructure/ci/gitea-runner/*.yaml ~/.local/share/gitea-runner/*.yaml + install -m 0644 infrastructure/ci/gitea-runner/compose.yaml infrastructure/ci/gitea-runner/config.yaml \ + infrastructure/ci/gitea-runner/daemon.json ~/.local/share/gitea-runner/ + sha256sum infrastructure/ci/gitea-runner/{compose.yaml,config.yaml,daemon.json} \ + ~/.local/share/gitea-runner/{compose.yaml,config.yaml,daemon.json} ``` 2. 등록 token을 편집기로 `~/.local/share/gitea-runner/secrets/registration-token`에 @@ -170,7 +195,7 @@ runner는 job이 끝나면 job 컨테이너와 proxy로 만든 컨테이너·네 재생성은 실행 중인 job을 취소하므로 job이 없을 때 한다. - 이미지 올리기: `compose.yaml`의 tag와 digest를 함께 바꾼다. 등록 파일은 그대로 유효하다. major가 바뀌면 [Upgrading](https://docs.gitea.com/runner/upgrade/)의 호환 변경을 먼저 읽는다. -- 디스크: 「디스크와 정리」의 주기를 따른다. +- 디스크: 「디스크와 정리」의 상한과 timer가 맡는다. 매주 `list-timers`로 마지막 실행을 본다. ## 되돌리기 diff --git a/infrastructure/ci/gitea-runner/compose.yaml b/infrastructure/ci/gitea-runner/compose.yaml index 8973b99..848bcb5 100644 --- a/infrastructure/ci/gitea-runner/compose.yaml +++ b/infrastructure/ci/gitea-runner/compose.yaml @@ -22,6 +22,8 @@ services: CONFIG_FILE: /config.yaml volumes: - ./config.yaml:/config.yaml:ro + # Caps the build cache of the inner Docker daemon. See README.md. + - ./daemon.json:/etc/docker/daemon.json:ro - ./secrets:/run/gitea-runner:ro - data:/data - docker:/var/lib/docker diff --git a/infrastructure/ci/gitea-runner/daemon.json b/infrastructure/ci/gitea-runner/daemon.json new file mode 100644 index 0000000..6777ff6 --- /dev/null +++ b/infrastructure/ci/gitea-runner/daemon.json @@ -0,0 +1,10 @@ +{ + "builder": { + "gc": { + "enabled": true, + "defaultReservedSpace": "2GB", + "defaultMaxUsedSpace": "10GB", + "defaultMinFreeSpace": "35GB" + } + } +} diff --git a/infrastructure/ci/gitea-runner/gitea-runner-prune.service b/infrastructure/ci/gitea-runner/gitea-runner-prune.service new file mode 100644 index 0000000..ac87845 --- /dev/null +++ b/infrastructure/ci/gitea-runner/gitea-runner-prune.service @@ -0,0 +1,17 @@ +# Weekly cleanup of the Docker daemon inside the gitea-runner container. +# Installed by infrastructure/ci/gitea-runner/README.md ("디스크와 정리"). +[Unit] +Description=Prune old images and build cache in the Gitea runner Docker +Requires=docker.service +After=docker.service + +[Service] +Type=oneshot +# The docker group is enough for docker exec. No root needed at run time. +User=donghyeon +Group=docker +# image prune keeps images used by a running job container; until= filters +# on image creation time, so the pinned job image is pulled again after a week. +ExecStart=/usr/bin/docker exec gitea-runner docker image prune --all --force --filter until=168h +ExecStart=/usr/bin/docker exec gitea-runner docker builder prune --all --force --filter until=72h +ExecStartPost=/usr/bin/docker exec gitea-runner docker system df diff --git a/infrastructure/ci/gitea-runner/gitea-runner-prune.timer b/infrastructure/ci/gitea-runner/gitea-runner-prune.timer new file mode 100644 index 0000000..9cab1ad --- /dev/null +++ b/infrastructure/ci/gitea-runner/gitea-runner-prune.timer @@ -0,0 +1,12 @@ +[Unit] +Description=Weekly prune of the Gitea runner Docker + +[Timer] +# Sunday 04:30 local time (Asia/Seoul), when CI and agent sessions are quiet. +OnCalendar=Sun *-*-* 04:30:00 +RandomizedDelaySec=30min +# Run at the next boot if the machine was off at that time. +Persistent=true + +[Install] +WantedBy=timers.target From 03ed6719b057421ab6e46c407c7b13c23faa6d4e Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Wed, 11 Nov 2026 21:52:39 +0900 Subject: [PATCH 04/16] feat(host): start k3s only after the clock is synchronized The desktop dual-boots Windows, which writes local time (KST) to the RTC, while Ubuntu reads the RTC as UTC. On 2026-11-11 the boot started 9 hours ahead; k3s issued its certificates at that time, timesyncd then set the clock back, the certificates were not yet valid, and every Ingress returned 502 until k3s was restarted by hand. Add a k3s.service drop-in with Wants= and After=time-sync.target, and a drop-in that caps systemd-time-wait-sync.service at 10 minutes. Enabling systemd-time-wait-sync makes time-sync.target wait for the first NTP sync; its packaged TimeoutStartSec=infinity would keep k3s down forever when NTP is unreachable, so after 10 minutes it fails, the target is reached anyway and k3s starts as it does today. Risk: the order only applies at boot, so nothing changes until the next reboot. If NTP is down at boot, k3s starts 10 minutes later. The Windows side still writes local time until RealTimeIsUniversal is set there; the host guide covers that step. Verification: systemd-analyze verify passes for k3s.service and systemd-time-wait-sync.service with the drop-ins, and an unknown key added to each drop-in is reported, so both files are read. --- .../host/systemd/k3s.service.d/10-after-time-sync.conf | 9 +++++++++ .../systemd-time-wait-sync.service.d/10-timeout.conf | 5 +++++ 2 files changed, 14 insertions(+) create mode 100644 infrastructure/host/systemd/k3s.service.d/10-after-time-sync.conf create mode 100644 infrastructure/host/systemd/systemd-time-wait-sync.service.d/10-timeout.conf diff --git a/infrastructure/host/systemd/k3s.service.d/10-after-time-sync.conf b/infrastructure/host/systemd/k3s.service.d/10-after-time-sync.conf new file mode 100644 index 0000000..05b4891 --- /dev/null +++ b/infrastructure/host/systemd/k3s.service.d/10-after-time-sync.conf @@ -0,0 +1,9 @@ +# Start k3s only after the system clock is synchronized. +# k3s writes certificates with the current time at start. If the clock is +# wrong at boot (RTC read 9 hours ahead on 2026-11-11) and NTP corrects it +# later, those certificates are "not yet valid" and every Ingress returns 502. +# time-sync.target is reached only when systemd-time-wait-sync.service is +# enabled. See infrastructure/host/README.md. +[Unit] +Wants=time-sync.target +After=time-sync.target diff --git a/infrastructure/host/systemd/systemd-time-wait-sync.service.d/10-timeout.conf b/infrastructure/host/systemd/systemd-time-wait-sync.service.d/10-timeout.conf new file mode 100644 index 0000000..3d1993d --- /dev/null +++ b/infrastructure/host/systemd/systemd-time-wait-sync.service.d/10-timeout.conf @@ -0,0 +1,5 @@ +# The packaged unit waits forever (TimeoutStartSec=infinity). If NTP is not +# reachable at boot, k3s would never start. After 10 minutes this unit fails, +# time-sync.target is reached anyway, and k3s starts with the clock it has. +[Service] +TimeoutStartSec=10min From fb5535aeb68c0d47a2aff1dc33c9217ec8371f1d Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Wed, 11 Nov 2026 21:52:39 +0900 Subject: [PATCH 05/16] docs(host): add the owner steps for the desktop host settings The host files need root, so agent sessions do not install them. The owner needs one ordered list with checks and rollback for each step, including when a Docker restart is safe. Add infrastructure/host/README.md: why the disk and clock settings exist, the BuildKit key meanings in Docker 29, and four steps. Pin the tech-log images against kubelet image GC until they move to the Gitea registry, install the k3s time-sync drop-ins, set the RTC to UTC on the Windows side, and install the host daemon.json during a window with no CI job, no agent gate holding the heavy lock, and workmachine not in use. Risk: the image pin is a manual label on the current tags; a new deployment tag needs the label again until the registry move lands. Verification: the commands were checked on the desktop where they do not need root (systemctl show, timedatectl, docker buildx inspect, docker exec gitea-runner docker ps); the sudo steps were not run. --- infrastructure/host/README.md | 184 ++++++++++++++++++++++++++++++++++ 1 file changed, 184 insertions(+) create mode 100644 infrastructure/host/README.md diff --git a/infrastructure/host/README.md b/infrastructure/host/README.md new file mode 100644 index 0000000..ba8a22a --- /dev/null +++ b/infrastructure/host/README.md @@ -0,0 +1,184 @@ +# desktop 호스트 설정 + +desktop 한 대(k3s 노드 `donghyeon-system-product-name`)는 운영 k3s, Gitea +Actions runner, 에이전트 세션의 Docker 게이트를 같이 돌린다. 루트 SSD 하나와 +시계 하나가 무너지면 운영까지 같이 내려간다. 이 폴더는 그 두 가지를 막는 호스트 +설정 조각이다. Kubernetes 리소스가 아니므로 `kustomization.yaml`이 없고, root +권한으로 아래 경로에 설치한다. + +| 파일 | 설치 경로 | 하는 일 | +| --- | --- | --- | +| `docker/daemon.json` | `/etc/docker/daemon.json` | 호스트 Docker 빌드 캐시 상한 | +| `systemd/k3s.service.d/10-after-time-sync.conf` | `/etc/systemd/system/k3s.service.d/` | k3s를 시간 동기화 뒤에 시작 | +| `systemd/systemd-time-wait-sync.service.d/10-timeout.conf` | `/etc/systemd/system/systemd-time-wait-sync.service.d/` | 동기화를 10분까지만 기다림 | + +runner 안쪽 Docker 상한과 주 1회 정리는 +[`../ci/gitea-runner/README.md`](../ci/gitea-runner/README.md)의 「디스크와 정리」에 있다. + +## 왜 바꾸나 + +### 디스크 (2026-11-09 사고) + +- 11-07~08 이틀 동안 에이전트 게이트의 Docker 빌드 캐시가 약 23G 쌓였다. 상한이 없었다. +- 루트 사용률 85%에서 kubelet이 안 쓰는 이미지를 지우기 시작했고, 90%(여유 10% 미만)에서 + 파드를 내보냈다. 내보내진 tech-log 파드의 이미지가 「안 쓰는 이미지」가 되어 지워졌고, + `imagePullPolicy: Never`라 다시 받지 못했다. +- 디스크 경보(85% 경고)는 사고 전 33시간 동안 울렸지만, 이미 kubelet 이미지 GC가 시작되는 + 시점이었다. + +### BuildKit 상한 값 + +Docker 29에서 `builder.gc` 키의 뜻은 다음과 같다(moby `daemon/config/builder.go`). + +| 키 | 뜻 | 호스트 값 | +| --- | --- | --- | +| `defaultReservedSpace` | 이 양 아래로는 지우지 않는다 | 5GB | +| `defaultMaxUsedSpace` | 캐시가 이 양을 넘으면 지운다(상한) | 15GB | +| `defaultMinFreeSpace` | 디스크 여유가 이 양 아래면 `reservedSpace`까지 지운다 | 35GB | + +- 예전 키 `defaultKeepStorage`는 이제 `defaultReservedSpace`와 같은 뜻이다. 상한이 아니라 + 「최소 보존량」이라 쓰지 않는다. +- 설정 전 기본 정책은 `docker buildx inspect default`에서 Reserved 21.42GiB, + Max Used 169.5GiB, Min Free 42.84GiB였다(2026-11-11). 캐시가 21GB까지는 전혀 지워지지 + 않았다. +- `35GB`는 1024 단위다. 루트 228GiB에서 여유 35GiB는 사용률 약 85%, 즉 kubelet이 이미지를 + 지우기 시작하는 지점이다. 그 전에 빌드 캐시부터 줄인다. +- Docker 저장소를 HDD(sdb)로 옮기는 안은 빌드가 느려져 하지 않는다(소유자 결정, 2026-11-11). + +### 시계 (2026-11-11 사고) + +- desktop은 Windows와 이중 부팅한다(`efibootmgr`: Windows Boot Manager, Ubuntu). +- Linux는 RTC를 UTC로 읽는다(`timedatectl`: `RTC in local TZ: no`). Windows는 기본값으로 + RTC에 현지 시각(KST)을 쓴다. 그래서 Windows를 쓴 뒤 Linux로 부팅하면 시계가 9시간 앞선다. +- 11-11 부팅이 그 상태로 시작했다(`journalctl --list-boots`의 시작 시각 11-12 04:33 KST). + k3s가 그 시각으로 인증서를 만든 뒤 timesyncd가 시계를 되돌렸고, 인증서가 「not yet valid」가 + 되어 전부 502가 됐다. `sudo systemctl restart k3s`로 복구했다. +- Prometheus도 그 사이 9시간 앞선 시각의 샘플을 저장했다. 시계가 돌아온 뒤 새 샘플이 전부 + 「out of bounds」로 버려져, 미래 시각을 따라잡을 때까지 지표와 경보가 멈춘다. + +고치는 방법은 둘이다. + +1. **k3s가 시간 동기화 뒤에 시작한다.** `systemd-time-wait-sync.service`는 커널 시계가 + NTP로 맞춰질 때까지 `time-sync.target`을 막는다. k3s는 `After=time-sync.target`으로 그 + 뒤에 시작한다. 이 서비스는 기본으로 꺼져 있고 끝없이 기다리므로, 켜고 10분 상한을 둔다. + 10분 안에 NTP에 닿지 못하면 k3s는 지금처럼 그냥 시작한다. +2. **Windows도 RTC를 UTC로 쓰게 한다.** 그러면 9시간 차이가 처음부터 생기지 않는다. + +## 소유자가 실행할 명령 + +sudo가 필요해 에이전트 세션은 실행하지 않는다. platform-core checkout(이 PR이 병합된 +main)에서 위에서부터 순서대로 실행한다. 단계마다 확인 명령과 되돌리는 법이 있다. + +### 1. 운영 이미지를 kubelet 이미지 GC에서 지킨다 (재시작 없음, 바로) + +tech-log 이미지를 Gitea 레지스트리로 옮기기 전까지만 쓴다. pinned 라벨이 있는 이미지는 +kubelet이 지우지 않는다. + +```bash +sudo k3s ctr -n k8s.io images label docker.io/library/tech-log-backend:46b1cc7 io.cri-containerd.pinned=pinned +sudo k3s ctr -n k8s.io images label docker.io/library/tech-log-frontend:7a4e201 io.cri-containerd.pinned=pinned +sudo k3s ctr -n k8s.io images ls | grep -E 'tech-log-(backend|frontend)' +``` + +마지막 줄의 LABELS 칸에 `io.cri-containerd.pinned=pinned`가 있어야 한다. +배포 이미지가 바뀌면 새 tag에 다시 붙인다. + +되돌리기: 라벨 값을 비운다. + +```bash +sudo k3s ctr -n k8s.io images label docker.io/library/tech-log-backend:46b1cc7 io.cri-containerd.pinned= +sudo k3s ctr -n k8s.io images label docker.io/library/tech-log-frontend:7a4e201 io.cri-containerd.pinned= +``` + +### 2. k3s를 시간 동기화 뒤에 시작한다 (재시작 없음, 다음 부팅부터) + +```bash +sudo install -D -m 0644 infrastructure/host/systemd/k3s.service.d/10-after-time-sync.conf \ + /etc/systemd/system/k3s.service.d/10-after-time-sync.conf +sudo install -D -m 0644 infrastructure/host/systemd/systemd-time-wait-sync.service.d/10-timeout.conf \ + /etc/systemd/system/systemd-time-wait-sync.service.d/10-timeout.conf +sudo systemctl daemon-reload +sudo systemctl enable systemd-time-wait-sync.service +``` + +확인: + +```bash +systemctl show k3s -p After -p Wants | grep -o time-sync.target +systemctl is-enabled systemd-time-wait-sync.service +systemctl show systemd-time-wait-sync.service -p TimeoutStartUSec +``` + +`time-sync.target` 두 줄, `enabled`, `TimeoutStartUSec=10min`이 나와야 한다. 순서는 부팅할 +때만 쓰이므로 지금 k3s를 다시 시작하지 않는다. 다음 재부팅 뒤 +`journalctl -b -u systemd-time-wait-sync -u k3s | head`에서 동기화가 k3s 시작보다 앞서는지 본다. + +되돌리기: + +```bash +sudo systemctl disable systemd-time-wait-sync.service +sudo rm /etc/systemd/system/k3s.service.d/10-after-time-sync.conf \ + /etc/systemd/system/systemd-time-wait-sync.service.d/10-timeout.conf +sudo systemctl daemon-reload +``` + +### 3. RTC를 UTC로 맞춘다 + +Linux 쪽은 이미 UTC다. 확인만 한다. + +```bash +timedatectl | grep -e 'RTC in local TZ' -e 'System clock synchronized' +``` + +`RTC in local TZ: no`가 아니면 `sudo timedatectl set-local-rtc 0`을 실행한다. + +Windows 쪽(다음에 Windows로 부팅했을 때, 관리자 명령 프롬프트): + +```bat +reg add "HKLM\SYSTEM\CurrentControlSet\Control\TimeZoneInformation" /v RealTimeIsUniversal /t REG_DWORD /d 1 /f +``` + +그 뒤 Windows 설정 → 시간 및 언어 → 날짜 및 시간 → 「지금 동기화」를 누른다. Windows가 +RTC를 UTC로 다시 쓴다. Linux로 돌아와 `timedatectl`의 `RTC time`이 `Universal time`과 +같은지 본다. + +되돌리기(Windows): `reg delete "HKLM\SYSTEM\CurrentControlSet\Control\TimeZoneInformation" /v RealTimeIsUniversal /f` + +### 4. 호스트 Docker 빌드 캐시 상한 (Docker 재시작 필요) + +`systemctl restart docker`는 호스트 Docker의 컨테이너를 모두 다시 시작한다. 운영 k3s는 +자기 containerd를 쓰므로 영향이 없다. 영향을 받는 것: + +| 컨테이너 | 영향 | +| --- | --- | +| `gitea-runner` | 실행 중인 CI job이 실패한다. `restart: always`로 다시 뜬다 | +| `workmachine` | cokacremote 원격 작업이 끊긴다. `unless-stopped`로 다시 뜬다 | +| `chatgpt-cloudflared` | 터널이 잠깐 끊긴다. `unless-stopped`로 다시 뜬다 | +| 에이전트 세션 게이트 | 돌고 있던 Testcontainers·이미지 빌드가 실패한다 | + +**안전한 때:** 아래 세 가지가 모두 참일 때. + +```bash +# 1) CI job이 없다: job 컨테이너가 0개 +docker exec gitea-runner docker ps --format '{{.Names}}' | wc -l +# 2) 에이전트 세션의 무거운 게이트가 없다: lock을 바로 잡을 수 있다 +flock -n /tmp/claude-gates/heavy.lock true && echo free +# 3) workmachine을 지금 쓰고 있지 않다 (소유자 판단) +``` + +적용은 lock을 잡은 채로 한다. 재시작하는 동안 새 게이트가 시작되지 않는다. + +```bash +sudo install -m 0644 infrastructure/host/docker/daemon.json /etc/docker/daemon.json +sudo dockerd --validate --config-file /etc/docker/daemon.json +flock /tmp/claude-gates/heavy.lock sudo systemctl restart docker +docker ps --format '{{.Names}} {{.Status}}' +docker buildx inspect default | sed -n '/GC Policy rule#1/,/Min Free/p' +``` + +`dockerd --validate`는 맨 위 키만 검사하고 `builder.gc` 안쪽 키 오타는 잡지 못한다. 그래서 +마지막 줄로 실제 정책을 본다. `Reserved Space: 5GiB`, `Max Used Space: 15GiB`, +`Min Free Space: 35GiB`가 나와야 한다. 세 컨테이너가 `Up`이어야 한다. + +되돌리기: `sudo rm /etc/docker/daemon.json` 뒤 같은 조건에서 Docker를 다시 시작한다(설치 전에는 +이 파일이 없었다). From 1349ee2f257b9208e420edee984099b4176152ea Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Wed, 11 Nov 2026 22:00:51 +0900 Subject: [PATCH 06/16] docs(ci): list daemon.json in the runner runtime folder --- infrastructure/ci/gitea-runner/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/infrastructure/ci/gitea-runner/README.md b/infrastructure/ci/gitea-runner/README.md index eba2561..0c2cdae 100644 --- a/infrastructure/ci/gitea-runner/README.md +++ b/infrastructure/ci/gitea-runner/README.md @@ -136,7 +136,7 @@ docker exec gitea-runner docker buildx inspect default | sed -n '/GC Policy rule | --- | --- | | 원본 설정 | 이 디렉터리의 `compose.yaml`, `config.yaml`, `daemon.json` | | 주 1회 정리 | 이 디렉터리의 `gitea-runner-prune.service`·`.timer` → `/etc/systemd/system/` | -| desktop 실행 폴더 | `/home/donghyeon/.local/share/gitea-runner/` (`compose.yaml`, `config.yaml`, `secrets/`) | +| desktop 실행 폴더 | `/home/donghyeon/.local/share/gitea-runner/` (`compose.yaml`, `config.yaml`, `daemon.json`, `secrets/`) | | 등록 파일 `.runner` | Docker 볼륨 `gitea-runner_data`의 `/data/.runner` (runner 자격 증명, Secret으로 취급) | | job 이미지·안쪽 daemon 데이터 | Docker 볼륨 `gitea-runner_docker` (`/var/lib/docker`) | | actions/cache 저장소 | `gitea-runner_data`의 `/data/cache`, 상한 5GB | From f19ce547406b53d87c03fb3315adafbe9d1c3612 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Wed, 11 Nov 2026 23:20:38 +0900 Subject: [PATCH 07/16] feat(postgres): back up platform-postgres to AIStor every day platform-postgres holds the Gitea, Keycloak and tech-log databases and had no backup: no spec.backup, no ScheduledBackup, and its only copy is a local PV on the root SSD. A lost SSD or a bad migration had no way back. Add services/platform-postgres-backup: a merge patch that gives the live Cluster barman-cloud WAL archiving and base backups to s3://platform-postgres-backups on the AIStor ObjectStore (HDD sdb3) with 30-day retention and gzip, a daily 03:00 KST ScheduledBackup that also runs once at creation, the object-storage ingress for the PostgreSQL pods, the bucket policy, and a restore drill Cluster that only reads the bucket. scripts/bootstrap/create-postgres-backup- credentials.sh creates the bucket (100GiB quota), policy, user and the platform-data Secret through a restricted AIStor client Pod, proves the user can write its bucket and cannot list loki, and prints no key. Add PlatformPostgresBackupMissing (no base backup in 26h) and PlatformPostgresWALArchiveBacklog (more than 10 WAL files waiting for 15m; a stuck archive fills the SSD volume). The native barmanObjectStore is deprecated since CNPG 1.26 and the operator warns it is removed in 1.31.0. The Barman Cloud Plugin needs cert-manager, which this cluster does not have; installing two operators before the first backup is slower and riskier. The bucket layout is the same, so the plugin can take over before CNPG 1.31. Volume snapshots do not work on the static local PV. Risk: the live Cluster has pg_hba lines for techlog that the Phase 1 source lacks, so re-applying services/platform-postgres would cut the tech-log backend off. The README applies only spec.backup with kubectl patch. Backups stay on the same desktop; an off-machine copy is not part of this change. Nothing was applied to the cluster. Verification: kubectl apply --dry-run=server accepts the ScheduledBackup, the NetworkPolicy and the drill Cluster; kubectl patch --dry-run=server keeps both techlog pg_hba lines and adds spec.backup. The row-count comparison from the drill steps ran read-only on production (keycloak 100, techlog 44, gitea 116 tables). mc command syntax was checked with the pinned AIStor client offline. The two new alert tests failed against the old rules and pass now. bash -n passes for the script; shellcheck is not installed on the laptop or the desktop. --- .../create-postgres-backup-credentials.sh | 170 ++++++++++++++++++ scripts/validate/test-platform-disk-alerts.sh | 75 +++++++- .../rules/platform/core-rules.yaml | 30 ++++ services/platform-postgres-backup/README.md | 162 +++++++++++++++++ .../bucket-policy.json | 29 +++ .../cluster-backup-patch.yaml | 26 +++ .../kustomization.yaml | 16 ++ .../object-storage-counterpart.yaml | 30 ++++ .../restore-drill/cluster.yaml | 49 +++++ .../scheduled-backup.yaml | 17 ++ 10 files changed, 598 insertions(+), 6 deletions(-) create mode 100755 scripts/bootstrap/create-postgres-backup-credentials.sh create mode 100644 services/platform-postgres-backup/README.md create mode 100644 services/platform-postgres-backup/bucket-policy.json create mode 100644 services/platform-postgres-backup/cluster-backup-patch.yaml create mode 100644 services/platform-postgres-backup/kustomization.yaml create mode 100644 services/platform-postgres-backup/object-storage-counterpart.yaml create mode 100644 services/platform-postgres-backup/restore-drill/cluster.yaml create mode 100644 services/platform-postgres-backup/scheduled-backup.yaml diff --git a/scripts/bootstrap/create-postgres-backup-credentials.sh b/scripts/bootstrap/create-postgres-backup-credentials.sh new file mode 100755 index 0000000..67adb92 --- /dev/null +++ b/scripts/bootstrap/create-postgres-backup-credentials.sh @@ -0,0 +1,170 @@ +#!/usr/bin/env bash +# Create the AIStor bucket, policy and user that CloudNativePG uses to back up +# platform-postgres, and the platform-data/platform-postgres-backup-s3 Secret. +# +# Run once, by the owner, from a platform-core checkout on the desktop. The +# script refuses to run when the Secret already exists; it never rotates or +# deletes a bucket, policy, user or object. No credential value is printed. +set -Eeuo pipefail +set +x +umask 077 + +readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)" +readonly EXPECTED_CONTEXT=default +readonly OS_NAMESPACE=object-storage +readonly PG_NAMESPACE=platform-data +readonly ROOT_SECRET=aistor-root-configuration +readonly TARGET_SECRET=platform-postgres-backup-s3 +readonly BUCKET=platform-postgres-backups +readonly POLICY=platform-postgres-backups +readonly QUOTA=100GiB +readonly S3_ENDPOINT=http://minio.object-storage.svc.cluster.local +# Same pinned AIStor client as scripts/validate/aistor-s3-smoke.sh. +readonly MC_IMAGE="quay.io/minio/aistor/mc@sha256:c5ec777c080fd6292b7529309a0f311c58c3a92e268c9c57c0901da86881949b" +readonly POLICY_FILE="${ROOT}/services/platform-postgres-backup/bucket-policy.json" +readonly PASS_LINE=PLATFORM_POSTGRES_BACKUP_BOOTSTRAP_PASS + +fail() { + printf 'FAIL: %s\n' "$*" >&2 + exit 1 +} + +for command in kubectl jq openssl; do + command -v "$command" >/dev/null || fail "missing command: ${command}" +done +[[ -f "$POLICY_FILE" ]] || fail "missing ${POLICY_FILE}" +jq -e . "$POLICY_FILE" >/dev/null || fail "bucket policy is not valid JSON" + +[[ "$(kubectl config current-context)" == "$EXPECTED_CONTEXT" ]] || + fail "kube context must be ${EXPECTED_CONTEXT}" +if kubectl --namespace "$PG_NAMESPACE" get secret "$TARGET_SECRET" >/dev/null 2>&1; then + fail "${PG_NAMESPACE}/${TARGET_SECRET} already exists; this script does not rotate it" +fi +kubectl --namespace "$OS_NAMESPACE" get secret "$ROOT_SECRET" >/dev/null || + fail "${OS_NAMESPACE}/${ROOT_SECRET} is missing" + +work="$(mktemp -d)" +pod="pg-backup-bootstrap-$(date +%H%M%S)-$$" +handoff_secret="${pod}-keys" +policy_map="${pod}-policy" + +cleanup() { + kubectl --namespace "$OS_NAMESPACE" delete pod "$pod" \ + --ignore-not-found --wait=false >/dev/null 2>&1 || true + kubectl --namespace "$OS_NAMESPACE" delete secret "$handoff_secret" \ + --ignore-not-found >/dev/null 2>&1 || true + kubectl --namespace "$OS_NAMESPACE" delete configmap "$policy_map" \ + --ignore-not-found >/dev/null 2>&1 || true + rm -rf -- "$work" +} +trap cleanup EXIT INT TERM + +printf 'pgbackup%s' "$(openssl rand -hex 8)" >"${work}/ACCESS_KEY_ID" +openssl rand -hex 32 | tr -d '\n' >"${work}/ACCESS_SECRET_KEY" + +printf '%s\n' \ + "Bucket: ${BUCKET} (quota ${QUOTA}) on AIStor ${S3_ENDPOINT}" \ + "Policy: ${POLICY} (${POLICY_FILE#"${ROOT}/"})" \ + "User: a new generated access key attached to ${POLICY}" \ + "Secret: ${PG_NAMESPACE}/${TARGET_SECRET} (ACCESS_KEY_ID, ACCESS_SECRET_KEY)" \ + "Temporary: one restricted client Pod, one Secret and one ConfigMap in ${OS_NAMESPACE}" \ + 'No credential value will be printed.' +printf 'Type APPLY to create them: ' +IFS= read -r confirmation +[[ "$confirmation" == "APPLY" ]] || fail "confirmation did not match APPLY" + +kubectl --namespace "$OS_NAMESPACE" create secret generic "$handoff_secret" \ + --from-file=ACCESS_KEY_ID="${work}/ACCESS_KEY_ID" \ + --from-file=ACCESS_SECRET_KEY="${work}/ACCESS_SECRET_KEY" >/dev/null +kubectl --namespace "$OS_NAMESPACE" create configmap "$policy_map" \ + --from-file=policy.json="$POLICY_FILE" >/dev/null + +# Every mc call is silenced so neither key reaches the Pod log. +bootstrap_command='set -eu +. /run/aistor-root/config.env +access_key="$(cat /run/backup/ACCESS_KEY_ID)" +secret_key="$(cat /run/backup/ACCESS_SECRET_KEY)" +mc alias set root "'"$S3_ENDPOINT"'" "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" --api S3v4 >/dev/null +mc mb --ignore-existing "root/'"$BUCKET"'" >/dev/null +mc quota set "root/'"$BUCKET"'" --size "'"$QUOTA"'" >/dev/null +mc admin policy create root "'"$POLICY"'" /run/policy/policy.json >/dev/null +mc admin user add root "$access_key" "$secret_key" >/dev/null +mc admin policy attach root "'"$POLICY"'" --user "$access_key" >/dev/null +mc alias set backup "'"$S3_ENDPOINT"'" "$access_key" "$secret_key" --api S3v4 >/dev/null +probe=".bootstrap-probe-$(date +%s)" +printf "probe" | mc pipe "backup/'"$BUCKET"'/$probe" >/dev/null +[ "$(mc cat "backup/'"$BUCKET"'/$probe")" = "probe" ] +mc rm "backup/'"$BUCKET"'/$probe" >/dev/null +if mc ls backup/loki >/dev/null 2>&1; then + echo "the backup user can list another bucket" >&2 + exit 1 +fi +printf "%s\n" "'"$PASS_LINE"'"' + +overrides="$( + jq -nc --arg command "$bootstrap_command" --arg image "$MC_IMAGE" \ + --arg root "$ROOT_SECRET" --arg keys "$handoff_secret" --arg policy "$policy_map" '{ + spec: { + automountServiceAccountToken: false, + securityContext: { + runAsNonRoot: true, + runAsUser: 1000, + runAsGroup: 1000, + fsGroup: 1000, + seccompProfile: {type: "RuntimeDefault"} + }, + containers: [{ + name: "pg-backup-bootstrap", + image: $image, + imagePullPolicy: "IfNotPresent", + command: ["/bin/sh", "-ec"], + args: [$command], + securityContext: { + allowPrivilegeEscalation: false, + capabilities: {drop: ["ALL"]}, + readOnlyRootFilesystem: true + }, + env: [{name: "MC_CONFIG_DIR", value: "/tmp/mc"}], + volumeMounts: [ + {name: "root-config", mountPath: "/run/aistor-root", readOnly: true}, + {name: "backup-keys", mountPath: "/run/backup", readOnly: true}, + {name: "policy", mountPath: "/run/policy", readOnly: true}, + {name: "tmp", mountPath: "/tmp"} + ] + }], + volumes: [ + {name: "root-config", secret: {secretName: $root, defaultMode: 256}}, + {name: "backup-keys", secret: {secretName: $keys, defaultMode: 256}}, + {name: "policy", configMap: {name: $policy}}, + {name: "tmp", emptyDir: {}} + ] + } + }' +)" + +kubectl --namespace "$OS_NAMESPACE" run "$pod" \ + --image="$MC_IMAGE" \ + --restart=Never \ + --labels='platform.hyeonworks.com/aistor-client=true,app.kubernetes.io/name=pg-backup-bootstrap' \ + --overrides="$overrides" >/dev/null + +phase='' +for _ in $(seq 1 90); do + phase="$(kubectl --namespace "$OS_NAMESPACE" get pod "$pod" --output=jsonpath='{.status.phase}')" + [[ "$phase" == "Succeeded" || "$phase" == "Failed" ]] && break + sleep 2 +done +logs="$(kubectl --namespace "$OS_NAMESPACE" logs "$pod" 2>&1 || true)" +if [[ "$phase" != "Succeeded" || "$logs" != *"$PASS_LINE"* ]]; then + printf '%s\n' "$logs" | tail -20 >&2 + fail "bootstrap Pod ended in phase '${phase}'; the platform-data Secret was not created" +fi + +kubectl --namespace "$PG_NAMESPACE" create secret generic "$TARGET_SECRET" \ + --from-file=ACCESS_KEY_ID="${work}/ACCESS_KEY_ID" \ + --from-file=ACCESS_SECRET_KEY="${work}/ACCESS_SECRET_KEY" >/dev/null +kubectl --namespace "$PG_NAMESPACE" label secret "$TARGET_SECRET" \ + app.kubernetes.io/part-of=platform app.kubernetes.io/component=platform-postgres-backup >/dev/null + +printf '%s\n' "$PASS_LINE" \ + "Created bucket ${BUCKET}, policy ${POLICY}, one user and ${PG_NAMESPACE}/${TARGET_SECRET}." diff --git a/scripts/validate/test-platform-disk-alerts.sh b/scripts/validate/test-platform-disk-alerts.sh index ee20253..c192247 100755 --- a/scripts/validate/test-platform-disk-alerts.sh +++ b/scripts/validate/test-platform-disk-alerts.sh @@ -1,10 +1,12 @@ #!/usr/bin/env bash -# Unit tests for the root filesystem alerts in core-rules.yaml. +# Unit tests for core-rules.yaml alerts that guard the desktop host. # -# The kubelet starts deleting unused images at 85% use and evicts pods at -# 90% (nodefs.available < 10%). These tests pin the alerts in front of those -# two points: warning at 80%, critical at 85%, and a forecast that fires when -# the 6h trend reaches the eviction point within 6 hours. +# Root disk: the kubelet starts deleting unused images at 85% use and evicts +# pods at 90% (nodefs.available < 10%). The alerts sit in front of those two +# points: warning at 80%, critical at 85%, and a forecast that fires when the +# 6h trend reaches the eviction point within 6 hours. +# PostgreSQL backups: no successful base backup for 26 hours, or WAL waiting +# to be archived. # # Needs docker, jq and kubectl (client only, no cluster access). set -euo pipefail @@ -45,6 +47,7 @@ jq -e --arg url "$RUNBOOK_URL" ' # node-exporter labels as scraped by serviceMonitor/observability-agent/node-exporter. readonly SERIES_LABELS='job="prometheus-node-exporter",instance="10.42.0.95:9100",device="/dev/sdc1",fstype="ext4",mountpoint="/"' +readonly PG_LABELS='job="platform-postgres",namespace="platform-data",pod="platform-postgres-1",instance="10.42.0.130:9187"' readonly ALERT_LABELS=' job: prometheus-node-exporter instance: 10.42.0.95:9100 device: /dev/sdc1 @@ -137,6 +140,66 @@ ${ALERT_LABELS} # Losing 0.1 per minute takes far longer than 6 hours to reach 100. forecast_case 'forecast stays quiet for a slow fill' 400 '-0.1' 30 ' []' forecast_case 'forecast stays quiet for flat usage' 400 '+0' 30 ' []' + + # PostgreSQL backups. Timestamps are seconds since the test epoch. + cat <"${temp_dir}/disk.test.yaml" chmod 0444 "${temp_dir}"/* @@ -150,4 +213,4 @@ docker run --rm \ --volume "${temp_dir}:/work:ro" \ --entrypoint /bin/promtool \ "$PROMETHEUS_IMAGE" test rules /work/disk.test.yaml -printf 'PASS: root filesystem usage and forecast alerts\n' +printf 'PASS: root filesystem and PostgreSQL backup alerts\n' diff --git a/services/observability/rules/platform/core-rules.yaml b/services/observability/rules/platform/core-rules.yaml index 0a363e2..0f37b15 100644 --- a/services/observability/rules/platform/core-rules.yaml +++ b/services/observability/rules/platform/core-rules.yaml @@ -113,6 +113,36 @@ spec: summary: 루트 파일시스템 inode 사용률이 90%를 초과함 description: 노드 루트 파일시스템 inode 사용률이 심각 임계치인 90%를 15분간 초과했습니다. runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md + - name: platform.backup + rules: + # cnpg_collector_last_available_backup_timestamp is 0 until the first + # base backup, so a cluster without backups fires as well. + - alert: PlatformPostgresBackupMissing + expr: >- + time() - max by (job, namespace) ( + cnpg_collector_last_available_backup_timestamp{job="platform-postgres",namespace="platform-data"} + ) > 26 * 3600 + for: 30m + labels: + severity: critical + annotations: + summary: platform-postgres에 26시간 안에 끝난 백업이 없음 + description: platform-postgres의 마지막 성공한 base backup이 26시간보다 오래됐거나 없습니다. 매일 03:00 KST 백업(ScheduledBackup platform-postgres-daily)을 확인하세요. + runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md + # A failing archive_command keeps WAL in pg_wal, which then fills the + # PostgreSQL volume. + - alert: PlatformPostgresWALArchiveBacklog + expr: >- + max by (job, namespace) ( + cnpg_collector_pg_wal_archive_status{job="platform-postgres",namespace="platform-data",value="ready"} + ) > 10 + for: 15m + labels: + severity: critical + annotations: + summary: platform-postgres의 WAL이 보관소로 올라가지 않음 + description: 보관을 기다리는 WAL 파일이 10개를 넘은 채 15분이 지났습니다. 보관이 막히면 WAL이 PostgreSQL 볼륨에 쌓여 디스크가 찹니다. + runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md - name: platform.workloads rules: - alert: PlatformPodRestartBurst diff --git a/services/platform-postgres-backup/README.md b/services/platform-postgres-backup/README.md new file mode 100644 index 0000000..0901aa5 --- /dev/null +++ b/services/platform-postgres-backup/README.md @@ -0,0 +1,162 @@ +# 플랫폼 PostgreSQL 백업 + +`platform-data/platform-postgres` 하나에 Gitea, Keycloak, tech-log의 DB가 같이 있다 +(2026-11-11: gitea 608MB, keycloak 14MB, techlog 14MB). 2026-11-11까지 백업 설정도, +ScheduledBackup도 없었다. 그 PV는 루트 SSD 하나에만 있었다. + +이 폴더는 매일 base backup과 WAL 연속 보관을 AIStor(HDD `sdb3`)에 두고, 30일 안의 +아무 시점으로나 되돌릴 수 있게 한다. + +## 무엇이 어디로 가나 + +```text +platform-postgres-1 (SSD) + ├─ WAL 파일이 찰 때마다, 그리고 5분마다 ─▶ s3://platform-postgres-backups/platform-postgres/wals/ + └─ 매일 03:00 KST base backup ─────────▶ s3://platform-postgres-backups/platform-postgres/base/ +AIStor ObjectStore minio-aistor (HDD sdb3, aistor-local-xfs-retain) +``` + +| 파일 | 하는 일 | +| --- | --- | +| `cluster-backup-patch.yaml` | live Cluster에 `spec.backup`(보관소, 30일 보존, gzip)을 더한다 | +| `scheduled-backup.yaml` | 매일 18:00 UTC(03:00 KST) base backup. 만들자마자 한 번 | +| `object-storage-counterpart.yaml` | `object-storage` 쪽 ingress: platform-data의 PostgreSQL 파드 → AIStor 9000 | +| `bucket-policy.json` | 백업 사용자가 자기 bucket만 읽고 쓰는 정책 | +| `restore-drill/cluster.yaml` | 복원 리허설용 임시 Cluster | +| `../../scripts/bootstrap/create-postgres-backup-credentials.sh` | bucket·정책·사용자·Secret 생성 | + +## 고른 방법 + +- **내장 `barmanObjectStore`.** CloudNativePG 1.26부터 deprecated이고 Barman Cloud Plugin이 + 권장이다. 하지만 plugin은 cert-manager가 필요한데 이 클러스터에는 없다(2026-11-11 CRD 없음). + 백업이 하나도 없는 지금은 새 operator 두 개(cert-manager, plugin)를 먼저 세우는 것보다 지금 + operator(1.30.0)가 바로 지원하는 쪽이 빠르고 위험이 작다. bucket 구조가 같아 plugin으로 + 옮겨도 백업을 그대로 쓴다. + - **기한:** operator 경고는 「1.31.0에서 완전히 제거」라고 한다(문서는 1.32.0). 그래서 + **CNPG를 1.31로 올리기 전에** Barman Cloud Plugin으로 옮긴다. +- **보관소는 AIStor.** 같은 desktop이지만 SSD가 아닌 HDD(`sdb3`, 900Gi)다. SSD가 죽어도 + 남는다. desktop 자체가 죽으면 같이 잃는다. 다른 기계·외부로 복사하는 일은 남은 과제다. +- **보존 30일, quota 100GiB.** 압축한 base backup이 하루 수백 MB, WAL은 5분마다 한 파일 + (`archive_timeout` 기본값)이라 30일에 수십 GB 안쪽이다. +- **볼륨 스냅숏은 쓰지 않는다.** `ssd-local-postgres-retain`은 정적 local PV라 CSI 스냅숏이 없다. + +## 적용 (소유자, 순서대로) + +전제: 이 PR이 병합된 platform-core checkout, desktop의 kube context `default`. + +**주의: `services/platform-postgres`를 통째로 다시 적용하지 않는다.** live Cluster의 +`pg_hba`에는 Phase 1 원본에 없는 `host techlog techlog all scram-sha-256`과 +`host all techlog all reject`가 있다. 원본을 다시 적용하면 이 두 줄이 지워져 tech-log 운영 +백엔드가 DB에 붙지 못한다. 그래서 백업 설정만 `kubectl patch`로 더한다. + +1. bucket, 정책, 사용자, Secret을 만든다. 화면에 자격 증명이 나오지 않는다. + + ```bash + bash scripts/bootstrap/create-postgres-backup-credentials.sh + ``` + + 마지막 줄에 `PLATFORM_POSTGRES_BACKUP_BOOTSTRAP_PASS`가 나와야 한다. 새 사용자가 자기 + bucket에 쓰고 읽고 지울 수 있고, `loki` bucket은 못 보는 것까지 확인한다. + +2. AIStor 쪽 ingress를 연다. 그 전에는 PostgreSQL이 9000에 닿지 못한다. + + ```bash + kubectl apply -f services/platform-postgres-backup/object-storage-counterpart.yaml + ``` + +3. live Cluster에 백업 설정을 더한다. PostgreSQL은 재시작하지 않는다. + + ```bash + kubectl -n platform-data patch cluster platform-postgres --type merge \ + --patch-file services/platform-postgres-backup/cluster-backup-patch.yaml --dry-run=server -o name + kubectl -n platform-data patch cluster platform-postgres --type merge \ + --patch-file services/platform-postgres-backup/cluster-backup-patch.yaml + kubectl -n platform-data get cluster platform-postgres -o jsonpath='{.spec.postgresql.pg_hba}{"\n"}' + ``` + + 마지막 줄에 `techlog` 두 줄이 그대로 있어야 한다. 몇 분 뒤 WAL이 올라가는지 본다. + + ```bash + kubectl -n platform-data get cluster platform-postgres \ + -o jsonpath='{.status.conditions[?(@.type=="ContinuousArchiving")].status}{" "}{.status.firstRecoverabilityPoint}{"\n"}' + ``` + +4. 매일 백업을 건다. 만들자마자 첫 base backup이 돈다. + + ```bash + kubectl apply -k services/platform-postgres-backup + kubectl -n platform-data get backup -w + ``` + + `PHASE`가 `completed`가 되면 된다. `failed`면 + `kubectl -n platform-data describe backup <이름>`의 마지막 줄을 본다. + +5. 아래 「복원 리허설」을 한 번 한다. + +경보(`services/observability/rules/platform/core-rules.yaml`의 `platform.backup`): + +- `PlatformPostgresBackupMissing`: 26시간 안에 끝난 base backup이 없다. 위 4단계 전까지는 계속 + 울린다(백업이 정말 없으니까). +- `PlatformPostgresWALArchiveBacklog`: 보관을 기다리는 WAL이 10개를 넘은 채 15분. + 보관이 막히면 WAL이 SSD 볼륨에 쌓여 PostgreSQL이 멈춘다. 2·3단계의 순서가 바뀌었거나 + Secret이 틀리면 이것이 울린다. + +### 되돌리기 + +```bash +kubectl delete -k services/platform-postgres-backup # ScheduledBackup과 ingress +kubectl -n platform-data patch cluster platform-postgres --type json -p '[{"op":"remove","path":"/spec/backup"}]' +``` + +bucket, 사용자, Secret, 이미 올린 백업은 지우지 않는다. 지울 때는 소유자가 `mc`로 직접 한다. + +## 복원 리허설 + +백업이 실제로 복원되는지는 복원해 봐야 안다. 첫 base backup이 끝난 뒤 한 번, 그 뒤로는 CNPG나 +PostgreSQL을 올릴 때마다 한다. 운영 Cluster는 건드리지 않는다. 임시 Cluster는 bucket을 읽기만 한다 +(`backup` 칸이 없다). + +1. 임시 Cluster를 만든다. local-path 5Gi(루트 SSD)를 잠깐 쓴다. + + ```bash + kubectl apply -f services/platform-postgres-backup/restore-drill/cluster.yaml + kubectl -n platform-data wait cluster/platform-postgres-restore-drill \ + --for=condition=Ready --timeout=20m + ``` + +2. 두 쪽의 표마다 행 수를 비교한다. 운영은 그 사이에도 쓰이므로 활동이 많은 표(Gitea의 + `action`, `notice` 등)는 조금 다를 수 있다. 표가 없거나 크게 다르면 실패다. + + ```bash + count_rows() { # $1 = cluster, $2 = database + kubectl -n platform-data exec "$1-1" -c postgres -- psql -At -d "$2" -c \ + "select string_agg(format('select %L, count(*) from %I.%I', schemaname||'.'||relname, schemaname, relname), ' union all ') + from pg_stat_user_tables" | + kubectl -n platform-data exec -i "$1-1" -c postgres -- psql -At -d "$2" | sort + } + for db in gitea keycloak techlog; do + diff <(count_rows platform-postgres "$db") <(count_rows platform-postgres-restore-drill "$db") \ + && echo "$db: same" || echo "$db: differs (see above)" + done + ``` + +3. 복원된 시점을 기록한다. + + ```bash + kubectl -n platform-data exec platform-postgres-restore-drill-1 -c postgres -- \ + psql -At -c "select now(), pg_last_xact_replay_timestamp()" + ``` + +4. 임시 Cluster를 지운다. PVC도 같이 지워진다. + + ```bash + kubectl -n platform-data delete cluster platform-postgres-restore-drill + ``` + +결과(날짜, 복원된 시점, 표 비교, 걸린 시간)는 아래 「기록」에 남긴다. + +## 기록 + +- 2026-11-11: 설계와 원본만. bucket·Secret·적용·리허설은 하지 않았다(소유자 단계). + `kubectl apply --dry-run=server`로 ScheduledBackup, ingress, 리허설 Cluster가 admission을 + 통과했고, `kubectl patch --dry-run=server`로 patch 뒤에도 `techlog` pg_hba 두 줄이 남는 것을 봤다. diff --git a/services/platform-postgres-backup/bucket-policy.json b/services/platform-postgres-backup/bucket-policy.json new file mode 100644 index 0000000..c446d6f --- /dev/null +++ b/services/platform-postgres-backup/bucket-policy.json @@ -0,0 +1,29 @@ +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:GetBucketLocation", + "s3:ListBucket", + "s3:ListBucketMultipartUploads" + ], + "Resource": [ + "arn:aws:s3:::platform-postgres-backups" + ] + }, + { + "Effect": "Allow", + "Action": [ + "s3:GetObject", + "s3:PutObject", + "s3:DeleteObject", + "s3:AbortMultipartUpload", + "s3:ListMultipartUploadParts" + ], + "Resource": [ + "arn:aws:s3:::platform-postgres-backups/*" + ] + } + ] +} diff --git a/services/platform-postgres-backup/cluster-backup-patch.yaml b/services/platform-postgres-backup/cluster-backup-patch.yaml new file mode 100644 index 0000000..a8d7d54 --- /dev/null +++ b/services/platform-postgres-backup/cluster-backup-patch.yaml @@ -0,0 +1,26 @@ +# Merge patch for the live Cluster. Apply it with kubectl patch, not by +# re-applying services/platform-postgres: the live Cluster carries pg_hba lines +# for techlog that the Phase 1 source does not have (README.md, "적용"). +spec: + backup: + # Base backups and WAL older than the newest point that still lets a + # restore reach 30 days back are deleted by the operator. + retentionPolicy: "30d" + barmanObjectStore: + # AIStor ObjectStore on the HDD (sdb3, aistor-local-xfs-retain), not the + # root SSD that holds the database itself. + destinationPath: s3://platform-postgres-backups/ + endpointURL: http://minio.object-storage.svc.cluster.local + s3Credentials: + accessKeyId: + name: platform-postgres-backup-s3 + key: ACCESS_KEY_ID + secretAccessKey: + name: platform-postgres-backup-s3 + key: ACCESS_SECRET_KEY + wal: + compression: gzip + maxParallel: 2 + data: + compression: gzip + jobs: 2 diff --git a/services/platform-postgres-backup/kustomization.yaml b/services/platform-postgres-backup/kustomization.yaml new file mode 100644 index 0000000..0a85c50 --- /dev/null +++ b/services/platform-postgres-backup/kustomization.yaml @@ -0,0 +1,16 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +# Applied after platform-postgres (Phase 1) and AIStor (Phase 2) exist and +# after scripts/bootstrap/create-postgres-backup-credentials.sh has created +# the bucket and the platform-data/platform-postgres-backup-s3 Secret. +# cluster-backup-patch.yaml and restore-drill/ are applied by hand; see README.md. +resources: + - object-storage-counterpart.yaml + - scheduled-backup.yaml + +labels: + - pairs: + app.kubernetes.io/part-of: platform + app.kubernetes.io/managed-by: kustomize + includeSelectors: false diff --git a/services/platform-postgres-backup/object-storage-counterpart.yaml b/services/platform-postgres-backup/object-storage-counterpart.yaml new file mode 100644 index 0000000..4e94191 --- /dev/null +++ b/services/platform-postgres-backup/object-storage-counterpart.yaml @@ -0,0 +1,30 @@ +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: object-storage-allow-platform-postgres-backup + namespace: object-storage + labels: + app.kubernetes.io/component: platform-postgres-backup +spec: + podSelector: + matchLabels: + aistor.min.io/objectStore: minio-aistor + policyTypes: + - Ingress + ingress: + # barman-cloud in the PostgreSQL pods writes base backups and WAL; the + # restore drill cluster reads them. + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: platform-data + podSelector: + matchExpressions: + - key: cnpg.io/cluster + operator: In + values: + - platform-postgres + - platform-postgres-restore-drill + ports: + - protocol: TCP + port: 9000 diff --git a/services/platform-postgres-backup/restore-drill/cluster.yaml b/services/platform-postgres-backup/restore-drill/cluster.yaml new file mode 100644 index 0000000..6e13de3 --- /dev/null +++ b/services/platform-postgres-backup/restore-drill/cluster.yaml @@ -0,0 +1,49 @@ +# Restore drill: a throwaway Cluster bootstrapped from the platform-postgres +# backups. It has no backup section, so it never writes to the bucket. +# Apply, compare, delete; see ../README.md ("복원 리허설"). +apiVersion: postgresql.cnpg.io/v1 +kind: Cluster +metadata: + name: platform-postgres-restore-drill + namespace: platform-data + labels: + app.kubernetes.io/name: platform-postgres-restore-drill + app.kubernetes.io/component: restore-drill +spec: + instances: 1 + imageName: ghcr.io/cloudnative-pg/postgresql:17.9-standard-trixie + storage: + # Dynamic local-path volume on the root SSD, deleted with the Cluster. + storageClass: local-path + size: 5Gi + bootstrap: + recovery: + source: platform-postgres + externalClusters: + - name: platform-postgres + barmanObjectStore: + destinationPath: s3://platform-postgres-backups/ + endpointURL: http://minio.object-storage.svc.cluster.local + serverName: platform-postgres + s3Credentials: + accessKeyId: + name: platform-postgres-backup-s3 + key: ACCESS_KEY_ID + secretAccessKey: + name: platform-postgres-backup-s3 + key: ACCESS_SECRET_KEY + wal: + maxParallel: 2 + managed: + services: + disabledDefaultServices: + - ro + - r + affinity: + enablePodAntiAffinity: false + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + memory: 1Gi diff --git a/services/platform-postgres-backup/scheduled-backup.yaml b/services/platform-postgres-backup/scheduled-backup.yaml new file mode 100644 index 0000000..8a71acb --- /dev/null +++ b/services/platform-postgres-backup/scheduled-backup.yaml @@ -0,0 +1,17 @@ +apiVersion: postgresql.cnpg.io/v1 +kind: ScheduledBackup +metadata: + name: platform-postgres-daily + namespace: platform-data + labels: + app.kubernetes.io/name: platform-postgres + app.kubernetes.io/component: backup +spec: + # Seconds field first. The operator runs in UTC: 18:00 UTC is 03:00 KST. + schedule: "0 0 18 * * *" + # Take the first base backup as soon as this object is created. + immediate: true + backupOwnerReference: self + method: barmanObjectStore + cluster: + name: platform-postgres From 619b90552f226ee52309d121a6cfa27e8aea1bbc Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Wed, 11 Nov 2026 23:52:13 +0900 Subject: [PATCH 08/16] feat(observability): alert when Prometheus itself stops working Nothing tells the owner when monitoring itself fails. On 2026-11-11 Prometheus stored samples 9 hours ahead, then rejected every new sample as out of bounds: rules kept running on no data, and the only sign was a false KubeAPIDown. If Prometheus, Alertmanager or the node went down, no rule could fire at all. Add two rules and one route. PlatformMetricsIngestionStopped fires when no up sample from node-exporter is stored for 5 minutes; absent() still fires while the TSDB rejects samples. PlatformHeartbeat always fires, and a new platform-heartbeat route sends it every 5 minutes to an external dead man's switch through a webhook whose URL comes from the Secret alertmanager-heartbeat. When the pings stop, that service alerts, which covers Prometheus, Alertmanager, the node and the network being down. The chart Watchdog stays disabled; the platform rule keeps the route inside this repository. Rejected: an in-cluster CronJob that queries Prometheus and posts to Alertmanager. It still depends on Alertmanager and the node, and needs a new workload in the exact observability NetworkPolicy matrix. Risk: the AlertmanagerConfig now references alertmanager-heartbeat; applying it before that Secret exists keeps the operator from rendering the new Alertmanager config. The alerting README orders the steps: create the external check, create the Secret, then apply. The heartbeat needs an account on an external service, which only the owner can create. Verification: promtool tests for both rules failed against the old rules and pass now. test-observability-alerting.sh pins the new route and receiver in its exact contract and passes the static contract, the global Alertmanager render with Helm v3.19.4 and the server dry run (--server-dry-run). kubectl apply --dry-run=server accepts the updated PrometheusRule. --- .../validate/test-observability-alerting.sh | 27 +++++++++++- scripts/validate/test-platform-disk-alerts.sh | 42 +++++++++++++++++- services/observability/alerting/README.md | 43 +++++++++++++++++++ .../alerting/platform-alertmanager.yaml | 19 ++++++++ .../rules/platform/core-rules.yaml | 26 +++++++++++ 5 files changed, 153 insertions(+), 4 deletions(-) create mode 100644 services/observability/alerting/README.md diff --git a/scripts/validate/test-observability-alerting.sh b/scripts/validate/test-observability-alerting.sh index 11a858a..4b41ef1 100755 --- a/scripts/validate/test-observability-alerting.sh +++ b/scripts/validate/test-observability-alerting.sh @@ -87,7 +87,20 @@ if alert != { "value": "InfoInhibitor", } ], - } + }, + { + "receiver": "platform-heartbeat", + "matchers": [ + { + "name": "alertname", + "matchType": "=", + "value": "PlatformHeartbeat", + } + ], + "groupWait": "0s", + "groupInterval": "1m", + "repeatInterval": "5m", + }, ], }, "inhibitRules": [ @@ -111,6 +124,16 @@ if alert != { ], "receivers": [ {"name": "platform-null"}, + { + "name": "platform-heartbeat", + "webhookConfigs": [ + { + "urlSecret": {"name": "alertmanager-heartbeat", "key": "url"}, + "sendResolved": False, + "maxAlerts": 1, + } + ], + }, { "name": "platform-slack", "slackConfigs": [ @@ -156,7 +179,7 @@ if alert != { ], }, }: - fail("platform-alertmanager must match the exact global Slack contract") + fail("platform-alertmanager must match the exact global Slack and heartbeat contract") if policy != { "apiVersion": "networking.k8s.io/v1", diff --git a/scripts/validate/test-platform-disk-alerts.sh b/scripts/validate/test-platform-disk-alerts.sh index c192247..b2c95e5 100755 --- a/scripts/validate/test-platform-disk-alerts.sh +++ b/scripts/validate/test-platform-disk-alerts.sh @@ -6,7 +6,8 @@ # points: warning at 80%, critical at 85%, and a forecast that fires when the # 6h trend reaches the eviction point within 6 hours. # PostgreSQL backups: no successful base backup for 26 hours, or WAL waiting -# to be archived. +# to be archived. Monitoring: ingestion stopped, and the always-firing +# heartbeat that an external dead man's switch watches. # # Needs docker, jq and kubectl (client only, no cluster access). set -euo pipefail @@ -199,6 +200,43 @@ ${ALERT_LABELS} - eval_time: 20m alertname: PlatformPostgresWALArchiveBacklog exp_alerts: [] + - name: ingestion alert fires when node-exporter samples stop + interval: 1m + input_series: + - series: 'up{job="prometheus-node-exporter",instance="10.42.0.95:9100"}' + values: '1x5 _x20' + alert_rule_test: + - eval_time: 20m + alertname: PlatformMetricsIngestionStopped + exp_alerts: + - exp_labels: + job: prometheus-node-exporter + severity: critical + exp_annotations: + summary: Prometheus에 새 지표가 저장되지 않음 + description: 5분 넘게 node-exporter의 up 샘플이 저장되지 않았습니다. Prometheus가 샘플을 버리고 있으면(예 시계가 되돌아감) 다른 지표 경보도 울리지 않습니다. + runbook_url: ${RUNBOOK_URL} + - name: ingestion alert stays quiet while samples arrive + interval: 1m + input_series: + - series: 'up{job="prometheus-node-exporter",instance="10.42.0.95:9100"}' + values: '1x30' + alert_rule_test: + - eval_time: 20m + alertname: PlatformMetricsIngestionStopped + exp_alerts: [] + - name: heartbeat always fires + interval: 1m + alert_rule_test: + - eval_time: 1m + alertname: PlatformHeartbeat + exp_alerts: + - exp_labels: + severity: none + exp_annotations: + summary: 경보 경로 heartbeat + description: 항상 울린다. Alertmanager가 이 경보를 외부 dead man's switch로 5분마다 보내고, 끊기면 외부 서비스가 알린다. + runbook_url: ${RUNBOOK_URL} EOF } >"${temp_dir}/disk.test.yaml" @@ -213,4 +251,4 @@ docker run --rm \ --volume "${temp_dir}:/work:ro" \ --entrypoint /bin/promtool \ "$PROMETHEUS_IMAGE" test rules /work/disk.test.yaml -printf 'PASS: root filesystem and PostgreSQL backup alerts\n' +printf 'PASS: root filesystem, PostgreSQL backup, ingestion and heartbeat alerts\n' diff --git a/services/observability/alerting/README.md b/services/observability/alerting/README.md new file mode 100644 index 0000000..8444a36 --- /dev/null +++ b/services/observability/alerting/README.md @@ -0,0 +1,43 @@ +# Alertmanager 경로 + +`platform-alertmanager.yaml`은 kube-prometheus-stack이 전역 +`alertmanagerConfiguration`으로 읽는 AlertmanagerConfig 하나다. 경로는 셋이다. + +| 경보 | receiver | 보내는 곳 | +| --- | --- | --- | +| `InfoInhibitor` | `platform-null` | 버린다 | +| `PlatformHeartbeat` | `platform-heartbeat` | 외부 heartbeat 서비스(dead man's switch), 5분마다 | +| 나머지 | `platform-slack` | Slack, 4시간마다 다시 | + +## heartbeat (dead man's switch) + +`PlatformHeartbeat`(`rules/platform/core-rules.yaml`)는 항상 울린다. Alertmanager가 이것을 +5분마다 외부 서비스의 ping URL로 보낸다. ping이 끊기면 그 서비스가 소유자에게 알린다. + +- 막는 것: Prometheus·Alertmanager가 죽었을 때, desktop이나 인터넷 연결이 끊겼을 때. + 이 경우는 클러스터 안의 어떤 규칙도 알리지 못한다. +- 막지 못하는 것: Prometheus가 돌면서 샘플만 버리는 경우(2026-11-11 시계 사고). 규칙은 돌고 + heartbeat도 계속 나간다. 그 경우는 `PlatformMetricsIngestionStopped`가 Slack으로 알린다. + +### 소유자가 할 일 (적용 전에 한 번) + +1. 외부 heartbeat 서비스에 check를 하나 만든다. 예: healthchecks.io(무료 요금제, 오픈소스라 + 나중에 직접 띄울 수도 있다). 주기 5분, 유예 10분. 알림은 Slack이 아닌 곳(메일·휴대폰)도 + 하나 건다. Slack 경로가 죽었을 때도 알아야 하기 때문이다. +2. ping URL을 Secret으로 만든다. URL 자체가 자격 증명이라 화면에 남기지 않는다. + + ```bash + read -rs HEARTBEAT_URL # ping URL 붙여 넣기, 화면에 안 보임 + printf '%s' "$HEARTBEAT_URL" | kubectl -n observability create secret generic alertmanager-heartbeat \ + --from-file=url=/dev/stdin + unset HEARTBEAT_URL + ``` + +3. 그 뒤에 rules·alerts를 다시 적용한다(`scripts/bootstrap/apply-observability-access.sh + --execute --rules-alerts`). **Secret보다 먼저 적용하지 않는다.** operator가 없는 Secret을 + 가리키는 AlertmanagerConfig로는 설정을 만들지 못한다. +4. 확인: 외부 서비스의 check가 5분 안에 「up」이 된다. Alertmanager 지표 + `alertmanager_notifications_total{integration="webhook"}`가 5분마다 는다. + +Alertmanager egress는 공개 IPv4 TCP/443만 허용한다(`alertmanager-public-https-egress.yaml`). +ping URL은 HTTPS여야 한다. diff --git a/services/observability/alerting/platform-alertmanager.yaml b/services/observability/alerting/platform-alertmanager.yaml index fe46c64..cabce86 100644 --- a/services/observability/alerting/platform-alertmanager.yaml +++ b/services/observability/alerting/platform-alertmanager.yaml @@ -22,6 +22,16 @@ spec: - name: alertname matchType: "=" value: InfoInhibitor + # PlatformHeartbeat always fires. Sending it every 5 minutes keeps the + # external dead man's switch quiet; when it stops, that service alerts. + - receiver: platform-heartbeat + matchers: + - name: alertname + matchType: "=" + value: PlatformHeartbeat + groupWait: 0s + groupInterval: 1m + repeatInterval: 5m inhibitRules: - sourceMatch: - name: alertname @@ -35,6 +45,15 @@ spec: - namespace receivers: - name: platform-null + - name: platform-heartbeat + webhookConfigs: + # Ping URL of the external heartbeat check. Create the Secret before + # applying this file; see services/observability/alerting/README.md. + - urlSecret: + name: alertmanager-heartbeat + key: url + sendResolved: false + maxAlerts: 1 - name: platform-slack slackConfigs: - apiURL: diff --git a/services/observability/rules/platform/core-rules.yaml b/services/observability/rules/platform/core-rules.yaml index 0f37b15..0370606 100644 --- a/services/observability/rules/platform/core-rules.yaml +++ b/services/observability/rules/platform/core-rules.yaml @@ -143,6 +143,32 @@ spec: summary: platform-postgres의 WAL이 보관소로 올라가지 않음 description: 보관을 기다리는 WAL 파일이 10개를 넘은 채 15분이 지났습니다. 보관이 막히면 WAL이 PostgreSQL 볼륨에 쌓여 디스크가 찹니다. runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md + - name: platform.monitoring + rules: + # When the TSDB rejects every sample (2026-11-11: the clock went back + # after samples were stored 9 hours ahead) rules still run, but on no + # data. absent() is one of the few expressions that still fires then. + - alert: PlatformMetricsIngestionStopped + expr: absent(up{job="prometheus-node-exporter"}) + for: 5m + labels: + severity: critical + annotations: + summary: Prometheus에 새 지표가 저장되지 않음 + description: 5분 넘게 node-exporter의 up 샘플이 저장되지 않았습니다. Prometheus가 샘플을 버리고 있으면(예 시계가 되돌아감) 다른 지표 경보도 울리지 않습니다. + runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md + # Dead man's switch. Alertmanager forwards this to an external + # heartbeat service; when the heartbeat stops, that service alerts. + # It covers what no rule here can: Prometheus, Alertmanager or the + # whole node being down. + - alert: PlatformHeartbeat + expr: vector(1) + labels: + severity: none + annotations: + summary: 경보 경로 heartbeat + description: 항상 울린다. Alertmanager가 이 경보를 외부 dead man's switch로 5분마다 보내고, 끊기면 외부 서비스가 알린다. + runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md - name: platform.workloads rules: - alert: PlatformPodRestartBurst From bed20b456b2558b4856c19f54f7989f5a0d33cd2 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Wed, 11 Nov 2026 23:52:13 +0900 Subject: [PATCH 09/16] fix(observability): stop throttling node-exporter node-exporter averages about 2m CPU, but each scrape runs its collectors in parallel. With a 250m limit (25ms per 100ms CFS period) 26% of its CFS periods were throttled over 30 minutes on 2026-11-11 (container_cpu_cfs_throttled_periods_total / cfs_periods_total), which delays scrapes and can make them time out. Raise the CPU limit to one core. The request stays 100m, so scheduling does not change, and a runaway collector is still capped. Removing the limit was rejected because every other workload in this repository sets one. Risk: none beyond one core of headroom on a 12-core node. The live DaemonSet changes only when the observability core is applied again. Verification: the throttling ratio and usage came from live Prometheus queries. helm template of prometheus-node-exporter 4.56.1 with these values renders limits.cpu "1". The effect after rollout is not measured yet. --- services/observability/node-exporter/values/home.yaml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/services/observability/node-exporter/values/home.yaml b/services/observability/node-exporter/values/home.yaml index d994b14..c6df077 100644 --- a/services/observability/node-exporter/values/home.yaml +++ b/services/observability/node-exporter/values/home.yaml @@ -32,7 +32,11 @@ resources: cpu: 100m memory: 128Mi limits: - cpu: 250m + # Average use is about 2m, but each scrape runs the collectors in + # parallel. At 250m (25ms per 100ms CFS period) 26% of periods were + # throttled on 2026-11-11. One core removes the throttling and still caps + # a runaway collector. + cpu: "1" memory: 256Mi serviceAccount: From 2b1310972c56e6a6af4de1004acd8323d3ddf632 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Thu, 12 Nov 2026 07:28:31 +0900 Subject: [PATCH 10/16] fix(observability): pin the heartbeat route in the core renderer render-observability-core.sh keeps its own exact copy of the AlertmanagerConfig, and test-render-observability-core.sh feeds it the same copy as a fixture. 619b905 added the platform-heartbeat route and receiver to the source and to test-observability-alerting.sh but not to these two, so the pinned renderer rejected the source with "platform AlertmanagerConfig is not the exact global Slack contract". Add the route and receiver to both copies and name the heartbeat in the reject message. Verification: render-observability-core.sh (Helm v3.19.4, kustomize v5.8.1) prints OBSERVABILITY CORE STATIC RENDER PASS, also with --verified-output-dir; test-render-observability-core.sh passes. Fixes: 619b90552f22 ("feat(observability): alert when Prometheus itself stops working") --- scripts/validate/render-observability-core.sh | 4 ++-- scripts/validate/test-render-observability-core.sh | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/scripts/validate/render-observability-core.sh b/scripts/validate/render-observability-core.sh index ea93f07..cce4635 100755 --- a/scripts/validate/render-observability-core.sh +++ b/scripts/validate/render-observability-core.sh @@ -412,9 +412,9 @@ elif mode == "alertmanager_routing": if item.get("kind") == "AlertmanagerConfig" and (item.get("metadata") or {}).get("name") == "platform-alertmanager" ] - expected_config = json.loads(r'''{"apiVersion":"monitoring.coreos.com/v1alpha1","kind":"AlertmanagerConfig","metadata":{"name":"platform-alertmanager","namespace":"observability","labels":{"observability.hyeonworks.com/instance":"home"}},"spec":{"route":{"receiver":"platform-slack","groupBy":["cluster","namespace","alertname","severity"],"groupWait":"30s","groupInterval":"5m","repeatInterval":"4h","routes":[{"receiver":"platform-null","matchers":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}]}]},"inhibitRules":[{"sourceMatch":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}],"targetMatch":[{"name":"severity","matchType":"=","value":"info"}],"equal":["namespace"]}],"receivers":[{"name":"platform-null"},{"name":"platform-slack","slackConfigs":[{"apiURL":{"name":"alertmanager-slack-webhook","key":"url"},"sendResolved":true,"linkNames":false,"mrkdwnIn":["text","fields"],"fallback":"{{ if eq .Status \"firing\" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","title":"{{ if eq .Status \"firing\" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","titleLink":"https://grafana.learn.hyeonworks.com/","color":"{{ if eq .Status \"resolved\" }}good{{ else if or (eq .CommonLabels.severity \"emergency\") (eq .CommonLabels.severity \"critical\") }}danger{{ else if eq .CommonLabels.severity \"warning\" }}warning{{ else }}#439FE0{{ end }}","fields":[{"title":"Status","value":"{{ .Status | toUpper }}","short":true},{"title":"Severity","value":"{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}","short":true},{"title":"Location","value":"{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}","short":true},{"title":"Alert count","value":"{{ len .Alerts }}","short":true}],"text":"{{ range .Alerts }}\n*Alert status:* {{ .Status | toUpper }}\n*Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }}\n*Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }}\n*Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }}\n*Started:* {{ .StartsAt.Format \"2006-01-02T15:04:05Z07:00\" }}\n{{ if eq .Status \"resolved\" }}*Ended:* {{ .EndsAt.Format \"2006-01-02T15:04:05Z07:00\" }}{{ end }}\n{{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }}\n\n{{ end }}\n · ","footer":"hyeonworks observability · Alertmanager"}]}]}}''') + expected_config = json.loads(r'''{"apiVersion":"monitoring.coreos.com/v1alpha1","kind":"AlertmanagerConfig","metadata":{"name":"platform-alertmanager","namespace":"observability","labels":{"observability.hyeonworks.com/instance":"home"}},"spec":{"route":{"receiver":"platform-slack","groupBy":["cluster","namespace","alertname","severity"],"groupWait":"30s","groupInterval":"5m","repeatInterval":"4h","routes":[{"receiver":"platform-null","matchers":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}]},{"receiver":"platform-heartbeat","matchers":[{"name":"alertname","matchType":"=","value":"PlatformHeartbeat"}],"groupWait":"0s","groupInterval":"1m","repeatInterval":"5m"}]},"inhibitRules":[{"sourceMatch":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}],"targetMatch":[{"name":"severity","matchType":"=","value":"info"}],"equal":["namespace"]}],"receivers":[{"name":"platform-null"},{"name":"platform-heartbeat","webhookConfigs":[{"urlSecret":{"name":"alertmanager-heartbeat","key":"url"},"sendResolved":false,"maxAlerts":1}]},{"name":"platform-slack","slackConfigs":[{"apiURL":{"name":"alertmanager-slack-webhook","key":"url"},"sendResolved":true,"linkNames":false,"mrkdwnIn":["text","fields"],"fallback":"{{ if eq .Status \"firing\" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","title":"{{ if eq .Status \"firing\" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","titleLink":"https://grafana.learn.hyeonworks.com/","color":"{{ if eq .Status \"resolved\" }}good{{ else if or (eq .CommonLabels.severity \"emergency\") (eq .CommonLabels.severity \"critical\") }}danger{{ else if eq .CommonLabels.severity \"warning\" }}warning{{ else }}#439FE0{{ end }}","fields":[{"title":"Status","value":"{{ .Status | toUpper }}","short":true},{"title":"Severity","value":"{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}","short":true},{"title":"Location","value":"{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}","short":true},{"title":"Alert count","value":"{{ len .Alerts }}","short":true}],"text":"{{ range .Alerts }}\n*Alert status:* {{ .Status | toUpper }}\n*Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }}\n*Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }}\n*Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }}\n*Started:* {{ .StartsAt.Format \"2006-01-02T15:04:05Z07:00\" }}\n{{ if eq .Status \"resolved\" }}*Ended:* {{ .EndsAt.Format \"2006-01-02T15:04:05Z07:00\" }}{{ end }}\n{{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }}\n\n{{ end }}\n · ","footer":"hyeonworks observability · Alertmanager"}]}]}}''') if configs != [expected_config]: - reject("platform AlertmanagerConfig is not the exact global Slack contract") + reject("platform AlertmanagerConfig is not the exact global Slack and heartbeat contract") policies = [ item diff --git a/scripts/validate/test-render-observability-core.sh b/scripts/validate/test-render-observability-core.sh index 97dce8b..ce9c9b2 100755 --- a/scripts/validate/test-render-observability-core.sh +++ b/scripts/validate/test-render-observability-core.sh @@ -197,7 +197,7 @@ write_file "$alertmanager_fixture" \ ' alertmanagerConfiguration:' \ ' name: platform-alertmanager' alerting_fixture="$fixture_dir/alerting.yaml" -write_file "$alerting_fixture" '{"apiVersion":"monitoring.coreos.com/v1alpha1","kind":"AlertmanagerConfig","metadata":{"name":"platform-alertmanager","namespace":"observability","labels":{"observability.hyeonworks.com/instance":"home"}},"spec":{"route":{"receiver":"platform-slack","groupBy":["cluster","namespace","alertname","severity"],"groupWait":"30s","groupInterval":"5m","repeatInterval":"4h","routes":[{"receiver":"platform-null","matchers":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}]}]},"inhibitRules":[{"sourceMatch":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}],"targetMatch":[{"name":"severity","matchType":"=","value":"info"}],"equal":["namespace"]}],"receivers":[{"name":"platform-null"},{"name":"platform-slack","slackConfigs":[{"apiURL":{"name":"alertmanager-slack-webhook","key":"url"},"sendResolved":true,"linkNames":false,"mrkdwnIn":["text","fields"],"fallback":"{{ if eq .Status \"firing\" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","title":"{{ if eq .Status \"firing\" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","titleLink":"https://grafana.learn.hyeonworks.com/","color":"{{ if eq .Status \"resolved\" }}good{{ else if or (eq .CommonLabels.severity \"emergency\") (eq .CommonLabels.severity \"critical\") }}danger{{ else if eq .CommonLabels.severity \"warning\" }}warning{{ else }}#439FE0{{ end }}","fields":[{"title":"Status","value":"{{ .Status | toUpper }}","short":true},{"title":"Severity","value":"{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}","short":true},{"title":"Location","value":"{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}","short":true},{"title":"Alert count","value":"{{ len .Alerts }}","short":true}],"text":"{{ range .Alerts }}\n*Alert status:* {{ .Status | toUpper }}\n*Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }}\n*Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }}\n*Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }}\n*Started:* {{ .StartsAt.Format \"2006-01-02T15:04:05Z07:00\" }}\n{{ if eq .Status \"resolved\" }}*Ended:* {{ .EndsAt.Format \"2006-01-02T15:04:05Z07:00\" }}{{ end }}\n{{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }}\n\n{{ end }}\n · ","footer":"hyeonworks observability · Alertmanager"}]}]}}' \ +write_file "$alerting_fixture" '{"apiVersion":"monitoring.coreos.com/v1alpha1","kind":"AlertmanagerConfig","metadata":{"name":"platform-alertmanager","namespace":"observability","labels":{"observability.hyeonworks.com/instance":"home"}},"spec":{"route":{"receiver":"platform-slack","groupBy":["cluster","namespace","alertname","severity"],"groupWait":"30s","groupInterval":"5m","repeatInterval":"4h","routes":[{"receiver":"platform-null","matchers":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}]},{"receiver":"platform-heartbeat","matchers":[{"name":"alertname","matchType":"=","value":"PlatformHeartbeat"}],"groupWait":"0s","groupInterval":"1m","repeatInterval":"5m"}]},"inhibitRules":[{"sourceMatch":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}],"targetMatch":[{"name":"severity","matchType":"=","value":"info"}],"equal":["namespace"]}],"receivers":[{"name":"platform-null"},{"name":"platform-heartbeat","webhookConfigs":[{"urlSecret":{"name":"alertmanager-heartbeat","key":"url"},"sendResolved":false,"maxAlerts":1}]},{"name":"platform-slack","slackConfigs":[{"apiURL":{"name":"alertmanager-slack-webhook","key":"url"},"sendResolved":true,"linkNames":false,"mrkdwnIn":["text","fields"],"fallback":"{{ if eq .Status \"firing\" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","title":"{{ if eq .Status \"firing\" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","titleLink":"https://grafana.learn.hyeonworks.com/","color":"{{ if eq .Status \"resolved\" }}good{{ else if or (eq .CommonLabels.severity \"emergency\") (eq .CommonLabels.severity \"critical\") }}danger{{ else if eq .CommonLabels.severity \"warning\" }}warning{{ else }}#439FE0{{ end }}","fields":[{"title":"Status","value":"{{ .Status | toUpper }}","short":true},{"title":"Severity","value":"{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}","short":true},{"title":"Location","value":"{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}","short":true},{"title":"Alert count","value":"{{ len .Alerts }}","short":true}],"text":"{{ range .Alerts }}\n*Alert status:* {{ .Status | toUpper }}\n*Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }}\n*Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }}\n*Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }}\n*Started:* {{ .StartsAt.Format \"2006-01-02T15:04:05Z07:00\" }}\n{{ if eq .Status \"resolved\" }}*Ended:* {{ .EndsAt.Format \"2006-01-02T15:04:05Z07:00\" }}{{ end }}\n{{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }}\n\n{{ end }}\n · ","footer":"hyeonworks observability · Alertmanager"}]}]}}' \ '---' \ 'apiVersion: networking.k8s.io/v1' \ 'kind: NetworkPolicy' \ From d7c845ca5d63fb9a2afa0ff1bcb43dd92681a873 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Thu, 12 Nov 2026 07:28:31 +0900 Subject: [PATCH 11/16] refactor(postgres): keep the backup alerts with the backup The observability core renderer rejects any alert or record whose name contains "backup" (no_unscoped_monitors_rules): backup monitoring is outside the observability core by design. f19ce54 put PlatformPostgresBackupMissing and PlatformPostgresWALArchiveBacklog in core-rules.yaml, so the pinned renderer failed with "out-of-scope monitor or rule is present". Move both alerts unchanged into services/platform-postgres-backup/prometheus-rule.yaml in platform-data, which Prometheus reads because the namespace has observability.hyeonworks.com/enabled=true and the rule has observability.hyeonworks.com/instance=home. They now point their runbook_url at the backup README and are applied with the ScheduledBackup. Renaming them to dodge the filter was rejected: it would keep backup rules inside a scope that excludes them on purpose. Verification: test-platform-disk-alerts.sh loads both rule files and passes; promtool checks 13 rules. render-observability-core.sh and test-render-observability-core.sh pass, and kubectl apply --dry-run=server accepts the backup kustomization with the rule. scan-platform-sensitive-source.sh stops at its observability-core renderer with RC=23 here and on an origin/main snapshot alike, so it says nothing about this change. Fixes: f19ce547406b ("feat(postgres): back up platform-postgres to AIStor every day") --- scripts/validate/test-platform-disk-alerts.sh | 17 +++++-- .../rules/platform/core-rules.yaml | 30 ------------ services/platform-postgres-backup/README.md | 7 +-- .../kustomization.yaml | 1 + .../prometheus-rule.yaml | 46 +++++++++++++++++++ 5 files changed, 64 insertions(+), 37 deletions(-) create mode 100644 services/platform-postgres-backup/prometheus-rule.yaml diff --git a/scripts/validate/test-platform-disk-alerts.sh b/scripts/validate/test-platform-disk-alerts.sh index b2c95e5..184b5dc 100755 --- a/scripts/validate/test-platform-disk-alerts.sh +++ b/scripts/validate/test-platform-disk-alerts.sh @@ -1,5 +1,6 @@ #!/usr/bin/env bash -# Unit tests for core-rules.yaml alerts that guard the desktop host. +# Unit tests for the alerts that guard the desktop host: core-rules.yaml and +# services/platform-postgres-backup/prometheus-rule.yaml. # # Root disk: the kubelet starts deleting unused images at 85% use and evicts # pods at 90% (nodefs.available < 10%). The alerts sit in front of those two @@ -14,9 +15,11 @@ set -euo pipefail readonly ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" readonly RULE_FILE="${ROOT_DIR}/services/observability/rules/platform/core-rules.yaml" +readonly BACKUP_RULE_FILE="${ROOT_DIR}/services/platform-postgres-backup/prometheus-rule.yaml" # Same pinned image as test-platform-observability-rules.sh. readonly PROMETHEUS_IMAGE="quay.io/prometheus/prometheus:v3.13.2-distroless@sha256:ce95cfa77eff5aad28bd7a65aff19868cf78d9e17e4c254da7dfe22ade78318b" readonly RUNBOOK_URL="https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md" +readonly BACKUP_RUNBOOK_URL="https://git.learn.hyeonworks.com/donghyeon.kang/platform-core/src/branch/main/services/platform-postgres-backup/README.md" temp_dir='' cleanup() { @@ -40,11 +43,16 @@ chmod 0755 "$temp_dir" kubectl create --dry-run=client --validate=false -f "$RULE_FILE" -o json | jq '.spec' >"${temp_dir}/core.rules.json" +kubectl create --dry-run=client --validate=false -f "$BACKUP_RULE_FILE" -o json | + jq '.spec' >"${temp_dir}/backup.rules.json" # Every core alert keeps the runbook the other platform rules use. jq -e --arg url "$RUNBOOK_URL" ' all(.groups[].rules[] | select(has("alert")); .annotations.runbook_url == $url) ' "${temp_dir}/core.rules.json" >/dev/null || fail "core alert without the shared runbook_url" +jq -e --arg url "$BACKUP_RUNBOOK_URL" ' + all(.groups[].rules[] | select(has("alert")); .annotations.runbook_url == $url) +' "${temp_dir}/backup.rules.json" >/dev/null || fail "backup alert without the backup README runbook_url" # node-exporter labels as scraped by serviceMonitor/observability-agent/node-exporter. readonly SERIES_LABELS='job="prometheus-node-exporter",instance="10.42.0.95:9100",device="/dev/sdc1",fstype="ext4",mountpoint="/"' @@ -127,6 +135,7 @@ ${ALERT_LABELS} printf '%s\n' \ 'rule_files:' \ ' - /work/core.rules.json' \ + ' - /work/backup.rules.json' \ 'evaluation_interval: 1m' \ 'tests:' usage_case 'warning stays quiet at 79 percent' 79 11 ' []' @@ -162,7 +171,7 @@ ${ALERT_LABELS} exp_annotations: summary: platform-postgres에 26시간 안에 끝난 백업이 없음 description: platform-postgres의 마지막 성공한 base backup이 26시간보다 오래됐거나 없습니다. 매일 03:00 KST 백업(ScheduledBackup platform-postgres-daily)을 확인하세요. - runbook_url: ${RUNBOOK_URL} + runbook_url: ${BACKUP_RUNBOOK_URL} - name: backup alert stays quiet after a fresh backup interval: 1m input_series: @@ -190,7 +199,7 @@ ${ALERT_LABELS} exp_annotations: summary: platform-postgres의 WAL이 보관소로 올라가지 않음 description: 보관을 기다리는 WAL 파일이 10개를 넘은 채 15분이 지났습니다. 보관이 막히면 WAL이 PostgreSQL 볼륨에 쌓여 디스크가 찹니다. - runbook_url: ${RUNBOOK_URL} + runbook_url: ${BACKUP_RUNBOOK_URL} - name: WAL backlog stays quiet at 10 ready files interval: 1m input_series: @@ -245,7 +254,7 @@ docker run --rm \ --network none \ --volume "${temp_dir}:/work:ro" \ --entrypoint /bin/promtool \ - "$PROMETHEUS_IMAGE" check rules /work/core.rules.json + "$PROMETHEUS_IMAGE" check rules /work/core.rules.json /work/backup.rules.json docker run --rm \ --network none \ --volume "${temp_dir}:/work:ro" \ diff --git a/services/observability/rules/platform/core-rules.yaml b/services/observability/rules/platform/core-rules.yaml index 0370606..f56f76a 100644 --- a/services/observability/rules/platform/core-rules.yaml +++ b/services/observability/rules/platform/core-rules.yaml @@ -113,36 +113,6 @@ spec: summary: 루트 파일시스템 inode 사용률이 90%를 초과함 description: 노드 루트 파일시스템 inode 사용률이 심각 임계치인 90%를 15분간 초과했습니다. runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md - - name: platform.backup - rules: - # cnpg_collector_last_available_backup_timestamp is 0 until the first - # base backup, so a cluster without backups fires as well. - - alert: PlatformPostgresBackupMissing - expr: >- - time() - max by (job, namespace) ( - cnpg_collector_last_available_backup_timestamp{job="platform-postgres",namespace="platform-data"} - ) > 26 * 3600 - for: 30m - labels: - severity: critical - annotations: - summary: platform-postgres에 26시간 안에 끝난 백업이 없음 - description: platform-postgres의 마지막 성공한 base backup이 26시간보다 오래됐거나 없습니다. 매일 03:00 KST 백업(ScheduledBackup platform-postgres-daily)을 확인하세요. - runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md - # A failing archive_command keeps WAL in pg_wal, which then fills the - # PostgreSQL volume. - - alert: PlatformPostgresWALArchiveBacklog - expr: >- - max by (job, namespace) ( - cnpg_collector_pg_wal_archive_status{job="platform-postgres",namespace="platform-data",value="ready"} - ) > 10 - for: 15m - labels: - severity: critical - annotations: - summary: platform-postgres의 WAL이 보관소로 올라가지 않음 - description: 보관을 기다리는 WAL 파일이 10개를 넘은 채 15분이 지났습니다. 보관이 막히면 WAL이 PostgreSQL 볼륨에 쌓여 디스크가 찹니다. - runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md - name: platform.monitoring rules: # When the TSDB rejects every sample (2026-11-11: the clock went back diff --git a/services/platform-postgres-backup/README.md b/services/platform-postgres-backup/README.md index 0901aa5..71621d0 100644 --- a/services/platform-postgres-backup/README.md +++ b/services/platform-postgres-backup/README.md @@ -22,6 +22,7 @@ AIStor ObjectStore minio-aistor (HDD sdb3, aistor-local-xfs-retain) | `scheduled-backup.yaml` | 매일 18:00 UTC(03:00 KST) base backup. 만들자마자 한 번 | | `object-storage-counterpart.yaml` | `object-storage` 쪽 ingress: platform-data의 PostgreSQL 파드 → AIStor 9000 | | `bucket-policy.json` | 백업 사용자가 자기 bucket만 읽고 쓰는 정책 | +| `prometheus-rule.yaml` | 백업 경보 둘(아래). 관측 core 렌더러가 백업 규칙을 core 범위 밖으로 두므로 여기 둔다 | | `restore-drill/cluster.yaml` | 복원 리허설용 임시 Cluster | | `../../scripts/bootstrap/create-postgres-backup-credentials.sh` | bucket·정책·사용자·Secret 생성 | @@ -84,7 +85,7 @@ AIStor ObjectStore minio-aistor (HDD sdb3, aistor-local-xfs-retain) 4. 매일 백업을 건다. 만들자마자 첫 base backup이 돈다. ```bash - kubectl apply -k services/platform-postgres-backup + kubectl apply -k services/platform-postgres-backup # ScheduledBackup, AIStor ingress, 경보 kubectl -n platform-data get backup -w ``` @@ -93,7 +94,7 @@ AIStor ObjectStore minio-aistor (HDD sdb3, aistor-local-xfs-retain) 5. 아래 「복원 리허설」을 한 번 한다. -경보(`services/observability/rules/platform/core-rules.yaml`의 `platform.backup`): +경보(`prometheus-rule.yaml`, 4단계에서 같이 들어간다. platform-data는 Prometheus가 규칙을 읽는 namespace다): - `PlatformPostgresBackupMissing`: 26시간 안에 끝난 base backup이 없다. 위 4단계 전까지는 계속 울린다(백업이 정말 없으니까). @@ -104,7 +105,7 @@ AIStor ObjectStore minio-aistor (HDD sdb3, aistor-local-xfs-retain) ### 되돌리기 ```bash -kubectl delete -k services/platform-postgres-backup # ScheduledBackup과 ingress +kubectl delete -k services/platform-postgres-backup # ScheduledBackup, ingress, 경보 kubectl -n platform-data patch cluster platform-postgres --type json -p '[{"op":"remove","path":"/spec/backup"}]' ``` diff --git a/services/platform-postgres-backup/kustomization.yaml b/services/platform-postgres-backup/kustomization.yaml index 0a85c50..ac4ed59 100644 --- a/services/platform-postgres-backup/kustomization.yaml +++ b/services/platform-postgres-backup/kustomization.yaml @@ -8,6 +8,7 @@ kind: Kustomization resources: - object-storage-counterpart.yaml - scheduled-backup.yaml + - prometheus-rule.yaml labels: - pairs: diff --git a/services/platform-postgres-backup/prometheus-rule.yaml b/services/platform-postgres-backup/prometheus-rule.yaml new file mode 100644 index 0000000..5ab183d --- /dev/null +++ b/services/platform-postgres-backup/prometheus-rule.yaml @@ -0,0 +1,46 @@ +# Backup alerts for platform-postgres. They live with the backup and not in +# the observability core, whose renderer keeps backup rules out of its scope +# (scripts/validate/render-observability-core.sh, no_unscoped_monitors_rules). +# Prometheus loads PrometheusRules from namespaces labelled +# observability.hyeonworks.com/enabled=true, which platform-data is. +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: platform-postgres-backup + namespace: platform-data + labels: + app.kubernetes.io/name: platform-postgres + app.kubernetes.io/component: backup + observability.hyeonworks.com/instance: home +spec: + groups: + - name: platform.backup + rules: + # cnpg_collector_last_available_backup_timestamp is 0 until the first + # base backup, so a cluster without backups fires as well. + - alert: PlatformPostgresBackupMissing + expr: >- + time() - max by (job, namespace) ( + cnpg_collector_last_available_backup_timestamp{job="platform-postgres",namespace="platform-data"} + ) > 26 * 3600 + for: 30m + labels: + severity: critical + annotations: + summary: platform-postgres에 26시간 안에 끝난 백업이 없음 + description: platform-postgres의 마지막 성공한 base backup이 26시간보다 오래됐거나 없습니다. 매일 03:00 KST 백업(ScheduledBackup platform-postgres-daily)을 확인하세요. + runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/platform-core/src/branch/main/services/platform-postgres-backup/README.md + # A failing archive_command keeps WAL in pg_wal, which then fills the + # PostgreSQL volume. + - alert: PlatformPostgresWALArchiveBacklog + expr: >- + max by (job, namespace) ( + cnpg_collector_pg_wal_archive_status{job="platform-postgres",namespace="platform-data",value="ready"} + ) > 10 + for: 15m + labels: + severity: critical + annotations: + summary: platform-postgres의 WAL이 보관소로 올라가지 않음 + description: 보관을 기다리는 WAL 파일이 10개를 넘은 채 15분이 지났습니다. 보관이 막히면 WAL이 PostgreSQL 볼륨에 쌓여 디스크가 찹니다. + runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/platform-core/src/branch/main/services/platform-postgres-backup/README.md From d47d57aa14789c6750e5349b026093c9a578df57 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Wed, 11 Nov 2026 23:39:12 +0900 Subject: [PATCH 12/16] fix(platform-postgres): keep the tech-log pg_hba rules the cluster uses The live platform-postgres Cluster has two pg_hba rules for the techlog role that the repository does not declare. They were added outside the repository for hyeonworks.com. Applying the current render would remove them and the production backend could no longer reach its database. Declare the same two rules, in the same position after the Keycloak pair, and pin them with render invariants like the Gitea and Keycloak pairs. The techlog DatabaseRole, Database, Secret and NetworkPolicies stay outside this build root; where they belong is an owner decision recorded in the 2026-11-11 maturity review. Risk: none for the live cluster. The rendered Cluster now matches the live object. A fresh install without the tech-log tenant gets two rules for a role that does not exist, which PostgreSQL ignores. Verification (desktop, Helm 3.19.4, Kustomize 5.8.1): - render-phase1.sh with the old cluster.yaml and the new invariants fails: "the tech-log role must authenticate only to the tech-log database"; with the new cluster.yaml it passes. - kubectl diff of the new render against the live cluster shows no Cluster change; only the pre-existing DatabaseRole default fields. --- scripts/validate/render-phase1.sh | 4 ++++ services/platform-postgres/README.md | 13 ++++++++++--- services/platform-postgres/base/cluster.yaml | 2 ++ 3 files changed, 16 insertions(+), 3 deletions(-) diff --git a/scripts/validate/render-phase1.sh b/scripts/validate/render-phase1.sh index 1292a45..6093bcc 100755 --- a/scripts/validate/render-phase1.sh +++ b/scripts/validate/render-phase1.sh @@ -402,6 +402,10 @@ fi fail "the Keycloak role must authenticate only to the Keycloak database" [[ "$(rg --count -- '^[[:space:]]*-[[:space:]]host all keycloak all reject[[:space:]]*$' "${render_dir}/platform-postgres.yaml" || true)" == "1" ]] || \ fail "the Keycloak role must be rejected from every other database" +[[ "$(rg --count -- '^[[:space:]]*-[[:space:]]host techlog techlog all scram-sha-256[[:space:]]*$' "${render_dir}/platform-postgres.yaml" || true)" == "1" ]] || \ + fail "the tech-log role must authenticate only to the tech-log database" +[[ "$(rg --count -- '^[[:space:]]*-[[:space:]]host all techlog all reject[[:space:]]*$' "${render_dir}/platform-postgres.yaml" || true)" == "1" ]] || \ + fail "the tech-log role must be rejected from every other database" rg --quiet '^kind: CustomResourceDefinition$' "${render_dir}/cnpg-operator.yaml" || \ diff --git a/services/platform-postgres/README.md b/services/platform-postgres/README.md index ebb0f2e..0293d15 100644 --- a/services/platform-postgres/README.md +++ b/services/platform-postgres/README.md @@ -21,9 +21,16 @@ platform-postgres-rw.platform-data.svc.cluster.local:5432 업그레이드한다. Cluster의 사용자 정의 `pg_hba` 규칙은 `gitea`가 `gitea` Database에만, -`keycloak`이 `keycloak` Database에만 인증하도록 허용한 뒤 각 Role의 다른 모든 -Database 접근을 거부한다. 이 규칙들은 CloudNativePG의 기본 fallback 인증 규칙보다 -앞에 배치된다. +`keycloak`이 `keycloak` Database에만, `techlog`가 `techlog` Database에만 인증하도록 +허용한 뒤 각 Role의 다른 모든 Database 접근을 거부한다. 이 규칙들은 +CloudNativePG의 기본 fallback 인증 규칙보다 앞에 배치된다. + +`techlog` 규칙 쌍은 운영 tech-log(hyeonworks.com)가 쓰는 것으로, 클러스터에는 +이 저장소 밖에서 먼저 들어갔다(2026-11-11 점검에서 확인). 이 규칙 쌍이 없는 렌더를 +적용하면 운영 backend가 DB에 접속하지 못하므로 공통 Cluster에 둔다. `techlog` +`DatabaseRole`·`Database`, `platform-data/techlog-db-credentials`와 tech-log용 +NetworkPolicy는 아직 어느 저장소에도 없다. 둘 곳은 소유자가 정한다 +(`docs/reviews/2026-11-11-k3s-maturity-review.md` 4절). ## Phase 1 Secret 계약 diff --git a/services/platform-postgres/base/cluster.yaml b/services/platform-postgres/base/cluster.yaml index 070617d..c280342 100644 --- a/services/platform-postgres/base/cluster.yaml +++ b/services/platform-postgres/base/cluster.yaml @@ -24,6 +24,8 @@ spec: - host all gitea all reject - host keycloak keycloak all scram-sha-256 - host all keycloak all reject + - host techlog techlog all scram-sha-256 + - host all techlog all reject storage: size: 20Gi From 4551a2ece60e847df98bcd16ad2b2ec728971c31 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Thu, 12 Nov 2026 07:29:48 +0900 Subject: [PATCH 13/16] feat(platform-postgres): run the shared database as Guaranteed QoS The one PostgreSQL instance holds the Gitea, Keycloak and tech-log databases. It is Burstable at priority 0 with oom_score_adj 985, so under memory pressure the kernel kills it before almost anything else, and an eviction treats it like Grafana (k3s maturity review 2026-11-11, 7.2 items 2 and 3). Set priorityClassName platform-critical (services/priority-classes) and raise the requests to the existing limits, CPU 2 and memory 4Gi, so the pod is Guaranteed and its oom_score_adj becomes -997. The backup README now gives the restart, not pg_hba, as the reason to add spec.backup with kubectl patch. Risk: applying this changes the pod spec, so CloudNativePG restarts the single instance and the three databases are down for that time. It belongs in the k3s restart window. Node requests grow by 1.75 CPU and 3.5Gi; peak use over the last 6 hours was 0.025 CPU and 195MB. Verification: kubectl diff of the rendered build root against the live cluster shows only priorityClassName, the two requests and the DatabaseRole defaults that were already there; no pg_hba line differs. render-phase1.sh passes. --- services/platform-postgres-backup/README.md | 12 +++++++----- services/platform-postgres/base/cluster.yaml | 9 +++++++-- 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/services/platform-postgres-backup/README.md b/services/platform-postgres-backup/README.md index 71621d0..fc99d81 100644 --- a/services/platform-postgres-backup/README.md +++ b/services/platform-postgres-backup/README.md @@ -45,10 +45,12 @@ AIStor ObjectStore minio-aistor (HDD sdb3, aistor-local-xfs-retain) 전제: 이 PR이 병합된 platform-core checkout, desktop의 kube context `default`. -**주의: `services/platform-postgres`를 통째로 다시 적용하지 않는다.** live Cluster의 -`pg_hba`에는 Phase 1 원본에 없는 `host techlog techlog all scram-sha-256`과 -`host all techlog all reject`가 있다. 원본을 다시 적용하면 이 두 줄이 지워져 tech-log 운영 -백엔드가 DB에 붙지 못한다. 그래서 백업 설정만 `kubectl patch`로 더한다. +**백업 설정은 `kubectl patch`로만 더한다.** `services/platform-postgres` 렌더를 다시 적용하면 +`priorityClassName`과 Guaranteed resources도 같이 들어가 PostgreSQL이 재시작된다. 그것은 +k3s 재시작 창(`scripts/bootstrap/apply-k3s-restart-bundle.sh`)에서 한다. 2026-11-11까지는 +live `pg_hba`의 `techlog` 두 줄이 원본에 없어서 다시 적용하면 tech-log 백엔드가 끊겼다. +이 브랜치에서 원본에 넣었으므로(`fix(platform-postgres): keep the tech-log pg_hba rules`) +지금 렌더의 `kubectl diff`에는 pg_hba 차이가 없다. 1. bucket, 정책, 사용자, Secret을 만든다. 화면에 자격 증명이 나오지 않는다. @@ -75,7 +77,7 @@ AIStor ObjectStore minio-aistor (HDD sdb3, aistor-local-xfs-retain) kubectl -n platform-data get cluster platform-postgres -o jsonpath='{.spec.postgresql.pg_hba}{"\n"}' ``` - 마지막 줄에 `techlog` 두 줄이 그대로 있어야 한다. 몇 분 뒤 WAL이 올라가는지 본다. + 마지막 줄에 `techlog` 두 줄이 있어야 한다. 몇 분 뒤 WAL이 올라가는지 본다. ```bash kubectl -n platform-data get cluster platform-postgres \ diff --git a/services/platform-postgres/base/cluster.yaml b/services/platform-postgres/base/cluster.yaml index c280342..a4fff51 100644 --- a/services/platform-postgres/base/cluster.yaml +++ b/services/platform-postgres/base/cluster.yaml @@ -39,10 +39,15 @@ spec: affinity: enablePodAntiAffinity: false + # The DB is the last thing that should go when the node runs short. + # platform-critical comes from services/priority-classes. requests equal + # limits so the pod is Guaranteed QoS and its oom_score_adj drops from 985 + # to -997 (k3s maturity review 2026-11-11, 7.2 items 2 and 3). + priorityClassName: platform-critical resources: requests: - cpu: 250m - memory: 512Mi + cpu: "2" + memory: 4Gi limits: cpu: "2" memory: 4Gi From d92dc5d16367c73558357b910ee45d50a9943a30 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Thu, 12 Nov 2026 07:29:48 +0900 Subject: [PATCH 14/16] feat(observability): read node-exporter textfile metrics The host timers from the k3s maturity work (last Gitea dump and k3s datastore backup, live drift count, API server and kubelet certificate expiry) write their results as *.prom files. node-exporter does not read any textfile directory, so Prometheus never sees them. Pass --collector.textfile.directory pointing at /var/lib/node_exporter/textfile_collector under the read-only host root mount the chart already has. No volume is added. The host installer creates the directory; until then the collector only reports an error metric. Verification: helm template of prometheus-node-exporter 4.56.1 with these values renders the argument next to --path.rootfs=/host/root. --- services/observability/node-exporter/values/home.yaml | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/services/observability/node-exporter/values/home.yaml b/services/observability/node-exporter/values/home.yaml index c6df077..a7a6adc 100644 --- a/services/observability/node-exporter/values/home.yaml +++ b/services/observability/node-exporter/values/home.yaml @@ -66,6 +66,14 @@ hostNetwork: false hostPID: false hostIPC: false +# Host timers (infrastructure/host/backup, owned by the k3s maturity work) +# write *.prom files here: last successful Gitea dump and k3s datastore +# backup, live drift count, API server and kubelet certificate expiry. The +# host root is already mounted read-only at /host/root. A missing directory +# only produces an error metric. +extraArgs: + - --collector.textfile.directory=/host/root/var/lib/node_exporter/textfile_collector + hostRootFsMount: enabled: true mountPropagation: None From c04d79481c75e017e4101eb01625f61294b2800b Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Thu, 12 Nov 2026 07:36:11 +0900 Subject: [PATCH 15/16] docs(host): point the time-sync step at the k3s restart bundle scripts/bootstrap/apply-k3s-restart-bundle.sh now installs the two time-sync drop-ins and enables systemd-time-wait-sync together with the kubelet drop-ins, so the owner restarts k3s once. Say so in step 2 and keep the manual commands for installing only these two files. --- infrastructure/host/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/infrastructure/host/README.md b/infrastructure/host/README.md index ba8a22a..2477207 100644 --- a/infrastructure/host/README.md +++ b/infrastructure/host/README.md @@ -92,6 +92,10 @@ sudo k3s ctr -n k8s.io images label docker.io/library/tech-log-frontend:7a4e201 ### 2. k3s를 시간 동기화 뒤에 시작한다 (재시작 없음, 다음 부팅부터) +k3s 재시작 창을 열 때는 `scripts/bootstrap/apply-k3s-restart-bundle.sh`(`infrastructure/host/k3s`의 +kubelet drop-in과 함께)가 아래 두 파일을 그대로 설치하고 `systemd-time-wait-sync`를 켠다. 그 묶음을 +쓰면 이 단계는 건너뛴다. 묶음 없이 이것만 할 때는 아래 명령을 쓴다. + ```bash sudo install -D -m 0644 infrastructure/host/systemd/k3s.service.d/10-after-time-sync.conf \ /etc/systemd/system/k3s.service.d/10-after-time-sync.conf From 0d703ff2497a440c392079b363eca553d80dbe04 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Thu, 12 Nov 2026 07:41:43 +0900 Subject: [PATCH 16/16] docs(postgres): record why the backups go to the HDD The owner asked on 2026-11-12 that production database backups stay off the SSD. State the evidence that the AIStor volume is the HDD (aistor-data-local-pv -> /srv/k3s/aistor -> /dev/sdb3, WD20EZAZ, ROTA=1), why the NTFS archive on sdb2 was not used (no POSIX permissions, no S3 API, a daily dump gives no point-in-time restore), and that the restore drill uses a short-lived local-path volume on the SSD because the HDD has no dynamic StorageClass. --- services/platform-postgres-backup/README.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/services/platform-postgres-backup/README.md b/services/platform-postgres-backup/README.md index fc99d81..c781f49 100644 --- a/services/platform-postgres-backup/README.md +++ b/services/platform-postgres-backup/README.md @@ -35,8 +35,19 @@ AIStor ObjectStore minio-aistor (HDD sdb3, aistor-local-xfs-retain) 옮겨도 백업을 그대로 쓴다. - **기한:** operator 경고는 「1.31.0에서 완전히 제거」라고 한다(문서는 1.32.0). 그래서 **CNPG를 1.31로 올리기 전에** Barman Cloud Plugin으로 옮긴다. -- **보관소는 AIStor.** 같은 desktop이지만 SSD가 아닌 HDD(`sdb3`, 900Gi)다. SSD가 죽어도 - 남는다. desktop 자체가 죽으면 같이 잃는다. 다른 기계·외부로 복사하는 일은 남은 과제다. +- **백업은 HDD에만 둔다(소유자 지시 2026-11-12: 「운영 db 백업은 SSD 말고 HDD쪽에」).** 보관소는 + AIStor다. AIStor의 데이터 볼륨은 하나뿐이고, PV `aistor-data-local-pv` → `/srv/k3s/aistor` → + `/dev/sdb3`(xfs, WDC WD20EZAZ 2TB HDD, `lsblk` `ROTA=1`)이다. 루트 SSD(`/dev/sdc1`, Crucial MX500)에는 + 백업이 가지 않는다. barman-cloud는 WAL과 base backup을 SSD에 따로 모으지 않고 곧바로 올린다. + - 버린 대안: HDD 보관소 `/mnt/hdd-archive`(`/dev/sdb2`, NTFS). POSIX 권한이 없어 PostgreSQL 파일을 + 그대로 두기 어렵고, barman-cloud가 쓰는 S3 API도 없다. 거기에 `pg_dump`를 두면 매일 한 시점만 + 남아 30일 안의 아무 시점으로 되돌릴 수 없다. AIStor는 같은 HDD 위에서 bucket별 사용자·정책과 + quota를 준다. + - SSD를 잃어도 백업은 남는다. desktop 자체를 잃으면 백업도 같이 잃는다. 다른 기계나 외부로 복사하는 + 일은 남은 과제다. + - 복원 리허설의 임시 Cluster는 local-path(루트 SSD) 5Gi를 잠깐 쓰고 바로 지운다. 백업 보관이 아니라 + 복원 결과를 확인하는 자리다. HDD에는 동적으로 볼륨을 만드는 StorageClass가 없고, `sdb3`는 통째로 + AIStor가 쓴다. - **보존 30일, quota 100GiB.** 압축한 base backup이 하루 수백 MB, WAL은 5분마다 한 파일 (`archive_timeout` 기본값)이라 30일에 수십 GB 안쪽이다. - **볼륨 스냅숏은 쓰지 않는다.** `ssd-local-postgres-retain`은 정적 local PV라 CSI 스냅숏이 없다.