From c2a3f75bc369206e27091bbf39f74cc184f0ba4a Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Thu, 12 Nov 2026 00:28:57 +0900 Subject: [PATCH 1/4] feat(priority): add platform-critical and platform-low PriorityClasses Every platform pod runs at priority 0. During the 2026-11-09 disk eviction the tech-log production pods were ranked with Grafana and kube-state-metrics. The kubelet ranks evictions by usage over requests and then by priority, so without classes the database and the production app have no head start. Add two classes and attach nothing yet: platform-critical (1000000, may preempt) for platform-postgres, Gitea, Keycloak and tech-log, and platform-low (-1000, never preempts) for Grafana, pgAdmin, Loki and Tempo. Neither is the global default, so no pod changes silently. Risk: creating the classes changes no pod. Attaching them restarts the pods, so that is left for the k3s restart window. Verification: render-tenants-and-policies.sh (added later in this series) checks the values, the preemption policy and globalDefault. --- services/priority-classes/README.md | 20 +++++++++++++++ services/priority-classes/kustomization.yaml | 5 ++++ .../priority-classes/priority-classes.yaml | 25 +++++++++++++++++++ 3 files changed, 50 insertions(+) create mode 100644 services/priority-classes/README.md create mode 100644 services/priority-classes/kustomization.yaml create mode 100644 services/priority-classes/priority-classes.yaml diff --git a/services/priority-classes/README.md b/services/priority-classes/README.md new file mode 100644 index 0000000..26eb2dc --- /dev/null +++ b/services/priority-classes/README.md @@ -0,0 +1,20 @@ +# 플랫폼 PriorityClass + +노드가 한 대라 자원이 모자라면 kubelet이 파드를 내보낸다. 내보낼 순서는 requests를 넘었는지, +그다음 우선순위로 정한다. 2026-11-09 디스크 축출 때 플랫폼 파드는 모두 우선순위 0이라 tech-log +운영 파드가 Grafana·kube-state-metrics와 같은 순위로 나갔다. + +| 이름 | 값 | 선점 | 붙일 곳 | +| --- | --- | --- | --- | +| `platform-critical` | 1000000 | 낮은 것을 선점 | platform-postgres, Gitea, Keycloak, tech-log | +| `platform-low` | -1000 | 선점 안 함 | Grafana, pgAdmin, Loki, Tempo | + +- 아무것도 안 붙인 파드는 기본값 0이다. 시스템 클래스(`system-*`, 20억)는 그대로 가장 높다. +- `globalDefault`는 둘 다 끈다. 새 파드가 모르게 우선순위를 얻지 않게 하려는 것이다. +- 클래스를 만드는 것은 아무 파드도 바꾸지 않는다. **붙이는 것은 파드 재시작을 부르므로** + k3s 재시작 창에 한다(`infrastructure/host/k3s/README.md`). + +```sh +kubectl apply -k services/priority-classes +kubectl get priorityclass platform-critical platform-low +``` diff --git a/services/priority-classes/kustomization.yaml b/services/priority-classes/kustomization.yaml new file mode 100644 index 0000000..8ba8781 --- /dev/null +++ b/services/priority-classes/kustomization.yaml @@ -0,0 +1,5 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - priority-classes.yaml diff --git a/services/priority-classes/priority-classes.yaml b/services/priority-classes/priority-classes.yaml new file mode 100644 index 0000000..38f87ea --- /dev/null +++ b/services/priority-classes/priority-classes.yaml @@ -0,0 +1,25 @@ +apiVersion: scheduling.k8s.io/v1 +kind: PriorityClass +metadata: + name: platform-critical + labels: + app.kubernetes.io/part-of: platform +value: 1000000 +globalDefault: false +preemptionPolicy: PreemptLowerPriority +description: >- + Stateful platform services and production workloads that other services depend on + (platform-postgres, Gitea, Keycloak, tech-log). Evicted after everything else. +--- +apiVersion: scheduling.k8s.io/v1 +kind: PriorityClass +metadata: + name: platform-low + labels: + app.kubernetes.io/part-of: platform +value: -1000 +globalDefault: false +preemptionPolicy: Never +description: >- + Tools whose loss does not stop a user-facing service (Grafana, pgAdmin, Loki, Tempo). + Evicted first and never preempts another pod. From c0749ec8a449a9dd39cbc6470131097915ca1e54 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Thu, 12 Nov 2026 00:29:18 +0900 Subject: [PATCH 2/4] feat(keycloak): default-deny the keycloak namespace egress The keycloak namespace only has the ingress policy the Keycloak Operator creates. Neither the Keycloak pod nor the Operator pod has any egress restriction, unlike every other platform namespace. Add a namespace-wide default deny and allow only what is used: DNS, platform-postgres 5432 (server and realm-import jobs), JGroups between Keycloak pods, and for the Operator the API server and the Keycloak management ports. Keycloak ingress stays owned by the Operator policy. Rejected: allowing public 443 like Gitea does. Neither the repository nor the Keycloak CR configures SMTP or an external identity provider, and the only open TCP connections from keycloak-0 on 2026-11-12 were to PostgreSQL. Adding SMTP or an IdP later needs a new egress rule; the README says so. Risk: a missed dependency would break logins for every service. The README lists the post-apply checks (OIDC discovery, a new TCP connection to Postgres from keycloak-0, Operator logs) and the one-line rollback. Verification: render and kubectl diff only add the five policies. --- infrastructure/networking/keycloak/README.md | 38 ++++++ .../networking/keycloak/kustomization.yaml | 12 ++ .../networking/keycloak/network-policies.yaml | 117 ++++++++++++++++++ 3 files changed, 167 insertions(+) create mode 100644 infrastructure/networking/keycloak/README.md create mode 100644 infrastructure/networking/keycloak/kustomization.yaml create mode 100644 infrastructure/networking/keycloak/network-policies.yaml diff --git a/infrastructure/networking/keycloak/README.md b/infrastructure/networking/keycloak/README.md new file mode 100644 index 0000000..08df1c3 --- /dev/null +++ b/infrastructure/networking/keycloak/README.md @@ -0,0 +1,38 @@ +# keycloak namespace NetworkPolicy + +2026-11-11 점검 전 `keycloak` namespace에는 Keycloak Operator가 만든 ingress 정책 하나뿐이었다. +Keycloak 파드와 Operator 파드의 egress는 막혀 있지 않았다. 다른 플랫폼 namespace처럼 기본 차단에 +필요한 것만 연다. + +| 정책 | 대상 | 허용 | +| --- | --- | --- | +| `keycloak-default-deny` | 모든 파드 | ingress·egress 기본 차단 | +| `keycloak-allow-dns` | 모든 파드 | kube-dns 53 | +| `keycloak-allow-platform-postgres` | 모든 파드(서버와 realm import Job) | `platform-data` Postgres 5432 | +| `keycloak-allow-cluster-peers` | Keycloak 파드 | 같은 Keycloak 파드 JGroups 7800·57800 | +| `keycloak-operator-allow-egress` | Operator | API 서버(`10.43.0.1:443`, `192.168.0.107:6443`), Keycloak 8080·9000 | + +- Keycloak 파드의 **ingress**는 계속 Keycloak CR `spec.networkPolicy`로 Operator가 만드는 + `keycloak-network-policy`가 소유한다(Traefik 8080, 관리 9000, JGroups). +- 바깥(인터넷)으로 나가는 길은 없다. 2026-11-12에 repo와 Keycloak CR에 SMTP·외부 IdP 설정이 + 없고, Keycloak 파드의 열린 TCP 연결이 Postgres뿐인 것을 확인했다. SMTP나 외부 IdP를 붙이면 + 여기에 egress를 더해야 한다(안 더하면 메일·외부 로그인만 조용히 실패한다). +- kubelet probe는 노드에서 오므로 막히지 않는다. + +## 적용과 확인 + +```sh +bash scripts/validate/render-tenants-and-policies.sh +kubectl kustomize infrastructure/networking/keycloak | kubectl diff -f - +kubectl apply -k infrastructure/networking/keycloak +``` + +적용 뒤 확인(모두 통과해야 한다): + +```sh +kubectl -n keycloak get pods +curl -fsS -o /dev/null -w '%{http_code}\n' https://id.learn.hyeonworks.com/realms/hyeonworks/.well-known/openid-configuration +kubectl -n keycloak logs deploy/keycloak-operator --since=5m | grep -i -E 'error|exception|timeout' || echo no-operator-errors +``` + +되돌리기: `kubectl delete -k infrastructure/networking/keycloak`(Operator 정책은 남는다). diff --git a/infrastructure/networking/keycloak/kustomization.yaml b/infrastructure/networking/keycloak/kustomization.yaml new file mode 100644 index 0000000..6dabb5c --- /dev/null +++ b/infrastructure/networking/keycloak/kustomization.yaml @@ -0,0 +1,12 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: keycloak + +labels: + - pairs: + app.kubernetes.io/part-of: platform + includeSelectors: false + +resources: + - network-policies.yaml diff --git a/infrastructure/networking/keycloak/network-policies.yaml b/infrastructure/networking/keycloak/network-policies.yaml new file mode 100644 index 0000000..6808550 --- /dev/null +++ b/infrastructure/networking/keycloak/network-policies.yaml @@ -0,0 +1,117 @@ +# keycloak namespace: 기본 차단 + 필요한 것만 허용. +# Keycloak 파드의 ingress는 Keycloak CR(spec.networkPolicy)로 Operator가 만드는 +# keycloak-network-policy가 계속 소유한다. 여기서는 기본 차단과 egress만 정한다. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: keycloak-default-deny +spec: + podSelector: {} + policyTypes: + - Ingress + - Egress +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: keycloak-allow-dns +spec: + podSelector: {} + policyTypes: + - Egress + egress: + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + podSelector: + matchLabels: + k8s-app: kube-dns + ports: + - port: 53 + protocol: UDP + - port: 53 + protocol: TCP +--- +# Keycloak 서버와 realm import Job이 공용 PostgreSQL에 붙는다. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: keycloak-allow-platform-postgres +spec: + podSelector: {} + policyTypes: + - Egress + egress: + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: platform-data + podSelector: + matchLabels: + cnpg.io/cluster: platform-postgres + ports: + - port: 5432 + protocol: TCP +--- +# 같은 Keycloak 인스턴스끼리의 JGroups(7800, 57800). Operator ingress 정책과 짝이다. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: keycloak-allow-cluster-peers +spec: + podSelector: + matchLabels: + app: keycloak + app.kubernetes.io/instance: keycloak + app.kubernetes.io/managed-by: keycloak-operator + policyTypes: + - Egress + egress: + - to: + - podSelector: + matchLabels: + app: keycloak + app.kubernetes.io/instance: keycloak + app.kubernetes.io/managed-by: keycloak-operator + ports: + - port: 7800 + protocol: TCP + - port: 57800 + protocol: TCP +--- +# Operator: Kubernetes API(ClusterIP와 실제 API 서버 주소)와 Keycloak 관리 포트. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: keycloak-operator-allow-egress +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: keycloak-operator + policyTypes: + - Egress + egress: + - to: + - ipBlock: + cidr: 10.43.0.1/32 + ports: + - port: 443 + protocol: TCP + - to: + - ipBlock: + cidr: 192.168.0.107/32 + ports: + - port: 6443 + protocol: TCP + - to: + - podSelector: + matchLabels: + app: keycloak + app.kubernetes.io/instance: keycloak + app.kubernetes.io/managed-by: keycloak-operator + ports: + - port: 8080 + protocol: TCP + - port: 9000 + protocol: TCP From ebdf33f07282ad4109dae643719b829b2ca63e0a Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Thu, 12 Nov 2026 00:29:18 +0900 Subject: [PATCH 3/4] feat(tenants): declare the tech-log objects inside platform namespaces The tech-log tenant needs objects inside platform namespaces: its database and role, ingress to Postgres and AIStor, blackbox probes, alert rules, a dashboard and two observability NetworkPolicies. They were only ever applied with kubectl and no repository holds them, so a deleted object cannot be recreated and a platform re-apply can drop tech-log access without anyone noticing. Declare them in services/tenants/techlog, copied from each object's last-applied configuration. The dashboard follows the repository's dashboard pattern (JSON file + configMapGenerator, same name, same labels). The hyeonworks-prod Namespace moves here as the tenant boundary and gets PSA enforce=baseline, warn/audit=restricted. Rejected: enforce=restricted now. A server dry-run shows all three tech-log pods would violate it; baseline has zero violations. Risk: the render matches the live objects exactly except the added PSA labels. The pg_hba rules and the Secret stay out of this root (Cluster owner and Git policy). Verification (desktop): - render-tenants-and-policies.sh passes. - kubectl diff of the render shows only the Namespace PSA labels. - kubectl label --dry-run=server enforce=baseline: no warnings. --- .../validate/render-tenants-and-policies.sh | 126 ++++++++++++++++++ services/tenants/techlog/README.md | 44 ++++++ .../dashboards/techlog-operations.json | 1 + services/tenants/techlog/kustomization.yaml | 23 ++++ services/tenants/techlog/namespace.yaml | 12 ++ services/tenants/techlog/object-storage.yaml | 22 +++ .../observability-network-policies.yaml | 45 +++++++ .../tenants/techlog/observability-probes.yaml | 46 +++++++ .../tenants/techlog/observability-rules.yaml | 67 ++++++++++ services/tenants/techlog/platform-data.yaml | 55 ++++++++ 10 files changed, 441 insertions(+) create mode 100644 scripts/validate/render-tenants-and-policies.sh create mode 100644 services/tenants/techlog/README.md create mode 100644 services/tenants/techlog/dashboards/techlog-operations.json create mode 100644 services/tenants/techlog/kustomization.yaml create mode 100644 services/tenants/techlog/namespace.yaml create mode 100644 services/tenants/techlog/object-storage.yaml create mode 100644 services/tenants/techlog/observability-network-policies.yaml create mode 100644 services/tenants/techlog/observability-probes.yaml create mode 100644 services/tenants/techlog/observability-rules.yaml create mode 100644 services/tenants/techlog/platform-data.yaml diff --git a/scripts/validate/render-tenants-and-policies.sh b/scripts/validate/render-tenants-and-policies.sh new file mode 100644 index 0000000..49bd860 --- /dev/null +++ b/scripts/validate/render-tenants-and-policies.sh @@ -0,0 +1,126 @@ +#!/usr/bin/env bash +# Render the tech-log tenant root, the platform PriorityClasses and the keycloak +# namespace NetworkPolicies, and check their invariants. Static: no cluster access. +set -Eeuo pipefail + +readonly EXPECTED_KUSTOMIZE_VERSION="v5.8.1" +readonly REPOSITORY_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)" + +fail() { + printf 'ERROR: %s\n' "$*" >&2 + exit 1 +} + +kustomize_version="$(kubectl version --client --output=yaml | sed -n 's/^kustomizeVersion: //p')" +[[ "$kustomize_version" == "$EXPECTED_KUSTOMIZE_VERSION" ]] || \ + fail "expected Kustomize ${EXPECTED_KUSTOMIZE_VERSION}, found ${kustomize_version:-unknown}" + +work="$(mktemp -d "${TMPDIR:-/tmp}/platform-tenants-render.XXXXXX")" +trap 'rm -rf -- "$work"' EXIT + +kubectl kustomize "$REPOSITORY_ROOT/services/tenants/techlog" >"$work/techlog.yaml" +kubectl kustomize "$REPOSITORY_ROOT/services/priority-classes" >"$work/priority-classes.yaml" +kubectl kustomize "$REPOSITORY_ROOT/infrastructure/networking/keycloak" >"$work/keycloak-network-policies.yaml" + +python3 -I - "$work" <<'PY' +import sys, pathlib, yaml + +work = pathlib.Path(sys.argv[1]) +errors = [] + +def load(name): + return [d for d in yaml.safe_load_all((work / name).read_text()) if d] + +def keyset(docs): + return sorted((d["kind"], d["metadata"].get("namespace", ""), d["metadata"]["name"]) for d in docs) + +for name in ("techlog.yaml", "priority-classes.yaml", "keycloak-network-policies.yaml"): + for d in load(name): + if d["kind"] == "Secret": + errors.append(f"{name}: Secret objects must not be rendered") + +techlog = load("techlog.yaml") +expected = sorted([ + ("Namespace", "", "hyeonworks-prod"), + ("Database", "platform-data", "platform-postgres-techlog"), + ("DatabaseRole", "platform-data", "platform-postgres-techlog"), + ("NetworkPolicy", "platform-data", "allow-techlog-backend"), + ("NetworkPolicy", "object-storage", "allow-techlog-backend"), + ("Probe", "observability", "techlog-edge"), + ("Probe", "observability", "techlog-readiness"), + ("PrometheusRule", "observability", "techlog"), + ("ConfigMap", "observability", "grafana-dashboard-techlog"), + ("NetworkPolicy", "observability", "blackbox-allow-techlog-health"), + ("NetworkPolicy", "observability", "observability-allow-techlog-scrape"), +]) +if keyset(techlog) != expected: + errors.append(f"tech-log tenant objects differ: {keyset(techlog)}") +for d in techlog: + kind = d["kind"] + if kind == "Namespace": + labels = d["metadata"].get("labels", {}) + want = { + "pod-security.kubernetes.io/enforce": "baseline", + "pod-security.kubernetes.io/warn": "restricted", + "pod-security.kubernetes.io/audit": "restricted", + "observability.hyeonworks.com/enabled": "true", + } + for k, v in want.items(): + if labels.get(k) != v: + errors.append(f"hyeonworks-prod label {k} must be {v}") + if kind == "DatabaseRole": + spec = d["spec"] + if spec.get("passwordSecret", {}).get("name") != "techlog-db-credentials": + errors.append("techlog DatabaseRole must reference techlog-db-credentials") + for flag in ("superuser", "createdb", "createrole", "replication", "bypassrls"): + if spec.get(flag) is not False: + errors.append(f"techlog DatabaseRole {flag} must be false") + if kind in ("Database", "DatabaseRole"): + policy = d["spec"].get("databaseReclaimPolicy") or d["spec"].get("databaseRoleReclaimPolicy") + if policy != "retain": + errors.append(f"{kind} must keep the retain reclaim policy") + if kind == "ConfigMap": + labels = d["metadata"].get("labels", {}) + if labels.get("grafana_dashboard") != "1": + errors.append("tech-log dashboard ConfigMap needs grafana_dashboard=1") + if d["metadata"]["name"] != "grafana-dashboard-techlog": + errors.append("tech-log dashboard ConfigMap must keep its name (no hash suffix)") + +classes = {d["metadata"]["name"]: d for d in load("priority-classes.yaml")} +if sorted(classes) != ["platform-critical", "platform-low"]: + errors.append(f"PriorityClasses differ: {sorted(classes)}") +else: + crit, low = classes["platform-critical"], classes["platform-low"] + if not (0 < crit["value"] < 1_000_000_000): + errors.append("platform-critical must be above 0 and below the system classes") + if not low["value"] < 0: + errors.append("platform-low must be below the default priority 0") + if low.get("preemptionPolicy") != "Never": + errors.append("platform-low must never preempt") + for d in classes.values(): + if d.get("globalDefault") is not False: + errors.append(f"{d['metadata']['name']} must not be the global default") + +policies = {d["metadata"]["name"]: d for d in load("keycloak-network-policies.yaml")} +deny = policies.get("keycloak-default-deny") +if not deny or deny["spec"].get("podSelector") != {} or sorted(deny["spec"].get("policyTypes", [])) != ["Egress", "Ingress"]: + errors.append("keycloak needs a namespace-wide default deny for ingress and egress") +for d in policies.values(): + if d["metadata"].get("namespace") != "keycloak": + errors.append(f"{d['metadata']['name']} must live in the keycloak namespace") + for rule in d["spec"].get("egress", []) or []: + for peer in rule.get("to", []): + cidr = peer.get("ipBlock", {}).get("cidr") + if cidr and cidr not in ("10.43.0.1/32", "192.168.0.107/32"): + errors.append(f"{d['metadata']['name']}: unexpected egress CIDR {cidr}") + if not rule.get("ports"): + errors.append(f"{d['metadata']['name']}: every egress rule must name its ports") + +if errors: + for e in errors: + print(f"ERROR: {e}", file=sys.stderr) + sys.exit(1) +print(f"tech-log tenant: {len(techlog)} objects; PriorityClasses: {len(classes)}; keycloak NetworkPolicies: {len(policies)}") +PY + +echo "TENANTS AND POLICIES STATIC RENDER PASS" diff --git a/services/tenants/techlog/README.md b/services/tenants/techlog/README.md new file mode 100644 index 0000000..3eca649 --- /dev/null +++ b/services/tenants/techlog/README.md @@ -0,0 +1,44 @@ +# tech-log 테넌트 — 플랫폼 쪽 객체 + +tech-log(hyeonworks.com 운영)가 플랫폼을 쓰려면 플랫폼 namespace 안에 허용 객체가 있어야 한다. +이 빌드 루트가 그 객체를 소유한다. 앱 자체(Deployment·Service·Ingress·ServiceMonitor·앱 +namespace 안의 NetworkPolicy)는 tech-log 저장소의 `deploy/`가 소유한다. + +2026-11-11 점검(`docs/reviews/2026-11-11-k3s-maturity-review.md` 4.3절) 전까지 이 객체들은 +`kubectl apply`로만 들어가 있었고 어느 저장소에도 없었다. 처음 내용은 라이브 객체의 +`last-applied-configuration`을 그대로 옮겼다. 라이브와 다른 것은 `hyeonworks-prod`의 PSA 라벨뿐이다. + +| 파일 | 객체 | +| --- | --- | +| `namespace.yaml` | Namespace `hyeonworks-prod`: 관측 대상 라벨, PSA `enforce=baseline`·`warn=restricted`·`audit=restricted` | +| `platform-data.yaml` | `techlog` Database와 DatabaseRole(로그인만, 연결 30), Postgres로 들어오는 ingress | +| `object-storage.yaml` | AIStor 9000으로 들어오는 ingress | +| `observability-probes.yaml` | blackbox Probe 둘(공개 HTTPS, 내부 readiness) | +| `observability-rules.yaml` | PrometheusRule `techlog` 일곱 개 | +| `dashboards/techlog-operations.json` | Grafana 대시보드(`grafana-dashboard-techlog` ConfigMap으로 생성) | +| `observability-network-policies.yaml` | blackbox → readiness, Prometheus → management scrape egress | + +## 여기에 없는 것 + +- `platform-data/platform-postgres`의 `pg_hba` techlog 두 줄: Cluster 객체는 + `services/platform-postgres`가 소유한다. +- Secret: `platform-data/techlog-db-credentials`(`kubernetes.io/basic-auth`, `username`은 정확히 + `techlog`)는 DatabaseRole이 참조한다. 앱 쪽 `hyeonworks-prod/hyeonworks-secrets`는 tech-log가 + 소유한다. 둘 다 Git에 두지 않는다. + +## PSA + +`enforce=baseline`은 2026-11-12에 서버 dry-run(`kubectl label --dry-run=server`)으로 위반 0을 +확인하고 붙였다. `restricted`로 강제하면 지금 파드 셋(redis, backend, frontend)이 모두 걸린다 +(`allowPrivilegeEscalation`, capabilities, `runAsNonRoot`, seccomp). tech-log 쪽 securityContext가 +고쳐진 뒤 같은 dry-run이 경고 없이 통과하면 `enforce`를 `restricted`로 올린다. + +## 렌더·적용 + +```sh +bash scripts/validate/render-tenants-and-policies.sh +kubectl kustomize services/tenants/techlog | kubectl diff -f - +kubectl apply -k services/tenants/techlog +``` + +새로 설치할 때는 이 루트를 tech-log `deploy/`보다 먼저 적용한다(namespace를 여기서 만든다). diff --git a/services/tenants/techlog/dashboards/techlog-operations.json b/services/tenants/techlog/dashboards/techlog-operations.json new file mode 100644 index 0000000..8dd5929 --- /dev/null +++ b/services/tenants/techlog/dashboards/techlog-operations.json @@ -0,0 +1 @@ +{"uid": "techlog-operations", "title": "TechLog · 운영 상태", "tags": ["techlog", "home", "application"], "timezone": "browser", "schemaVersion": 39, "version": 1, "refresh": "30s", "time": {"from": "now-1h", "to": "now"}, "panels": [{"id": 1, "title": "API 메트릭 수집", "type": "stat", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 0, "y": 0, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "up{namespace=\"hyeonworks-prod\",service=\"techlog-management\"}", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "short"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 2, "title": "의존성 readiness", "type": "stat", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 12, "y": 0, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "probe_success{job=\"techlog-readiness\"}", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "short"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 3, "title": "웹·API HTTPS 상태", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 0, "y": 7, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "probe_success{job=\"techlog-edge\"}", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "short"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 4, "title": "인증서 남은 기간", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 12, "y": 7, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "(probe_ssl_earliest_cert_expiry{job=\"techlog-edge\"}-time())/86400", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "d"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 5, "title": "HTTP 요청률", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 0, "y": 14, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "sum by(uri,status)(rate(http_server_requests_seconds_count{namespace=\"hyeonworks-prod\",uri!~\"/actuator.*\"}[$__rate_interval]))", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "reqps"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 6, "title": "HTTP p95·p99 지연", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 12, "y": 14, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "histogram_quantile(0.99,sum by(le)(rate(http_server_requests_seconds_bucket{namespace=\"hyeonworks-prod\",uri!~\"/actuator.*\"}[$__rate_interval])))", "legendFormat": "p99"}, {"refId": "B", "expr": "histogram_quantile(0.95,sum by(le)(rate(http_server_requests_seconds_bucket{namespace=\"hyeonworks-prod\",uri!~\"/actuator.*\"}[$__rate_interval])))", "legendFormat": "p95"}], "fieldConfig": {"defaults": {"unit": "s"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 7, "title": "HTTP 상태별 응답", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 0, "y": 21, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "sum by(status)(rate(http_server_requests_seconds_count{namespace=\"hyeonworks-prod\"}[$__rate_interval]))", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "reqps"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 8, "title": "JVM 메모리", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 12, "y": 21, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "sum by(area)(jvm_memory_used_bytes{namespace=\"hyeonworks-prod\"})", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "bytes"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 9, "title": "JVM GC 시간", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 0, "y": 28, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "sum by(action)(rate(jvm_gc_pause_seconds_sum{namespace=\"hyeonworks-prod\"}[$__rate_interval]))", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "s"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 10, "title": "프로세스 CPU", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 12, "y": 28, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "process_cpu_usage{namespace=\"hyeonworks-prod\"}", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "percentunit"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 11, "title": "DB 연결 풀 사용·대기", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 0, "y": 35, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "hikaricp_connections_active{namespace=\"hyeonworks-prod\"}", "legendFormat": "active"}, {"refId": "B", "expr": "hikaricp_connections_pending{namespace=\"hyeonworks-prod\"}", "legendFormat": "pending"}], "fieldConfig": {"defaults": {"unit": "short"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 12, "title": "DB 연결 획득 지연", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 12, "y": 35, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "rate(hikaricp_connections_acquire_seconds_sum{namespace=\"hyeonworks-prod\"}[$__rate_interval])/clamp_min(rate(hikaricp_connections_acquire_seconds_count{namespace=\"hyeonworks-prod\"}[$__rate_interval]),0.001)", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "s"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 13, "title": "Pod 재시작", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 0, "y": 42, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "kube_pod_container_status_restarts_total{namespace=\"hyeonworks-prod\"}", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "short"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 14, "title": "분산 추적 샘플링 비율", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 12, "y": 42, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "tracing_sampling_rate{namespace=\"hyeonworks-prod\"}", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "percentunit"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 15, "title": "TechLog 구조화 로그 · 요청 ID / 추적 ID", "type": "logs", "datasource": {"type": "loki", "uid": "loki"}, "gridPos": {"x": 0, "y": 49, "w": 24, "h": 12}, "targets": [{"refId": "A", "expr": "{namespace=\"hyeonworks-prod\",app=\"techlog-backend\"}", "queryType": "range"}], "options": {"showTime": true, "showLabels": false, "showCommonLabels": false, "wrapLogMessage": true, "prettifyLogMessage": true, "sortOrder": "Descending", "enableLogDetails": true}}, {"id": 16, "type": "text", "title": "추적 수집 상태", "gridPos": {"x": 0, "y": 61, "w": 24, "h": 4}, "options": {"mode": "markdown", "content": "요청 ID와 추적 ID는 JSON 로그에서 조회할 수 있습니다. **Tempo span 저장은 현재 이미지에서 검증되지 않았습니다.** 메트릭·로그·의존성 health 수집은 동작합니다."}}], "links": []} \ No newline at end of file diff --git a/services/tenants/techlog/kustomization.yaml b/services/tenants/techlog/kustomization.yaml new file mode 100644 index 0000000..6ede4d1 --- /dev/null +++ b/services/tenants/techlog/kustomization.yaml @@ -0,0 +1,23 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +# tech-log(hyeonworks.com) 테넌트의 플랫폼 쪽 객체. 앱 쪽 매니페스트는 tech-log repo가 소유한다. +resources: + - namespace.yaml + - platform-data.yaml + - object-storage.yaml + - observability-probes.yaml + - observability-rules.yaml + - observability-network-policies.yaml + +generatorOptions: + disableNameSuffixHash: true + labels: + app.kubernetes.io/part-of: techlog + grafana_dashboard: "1" + +configMapGenerator: + - name: grafana-dashboard-techlog + namespace: observability + files: + - dashboards/techlog-operations.json diff --git a/services/tenants/techlog/namespace.yaml b/services/tenants/techlog/namespace.yaml new file mode 100644 index 0000000..7374c1b --- /dev/null +++ b/services/tenants/techlog/namespace.yaml @@ -0,0 +1,12 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: hyeonworks-prod + labels: + observability.hyeonworks.com/enabled: 'true' + pod-security.kubernetes.io/enforce: baseline + pod-security.kubernetes.io/enforce-version: latest + pod-security.kubernetes.io/warn: restricted + pod-security.kubernetes.io/warn-version: latest + pod-security.kubernetes.io/audit: restricted + pod-security.kubernetes.io/audit-version: latest diff --git a/services/tenants/techlog/object-storage.yaml b/services/tenants/techlog/object-storage.yaml new file mode 100644 index 0000000..8774f13 --- /dev/null +++ b/services/tenants/techlog/object-storage.yaml @@ -0,0 +1,22 @@ +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-techlog-backend + namespace: object-storage +spec: + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: hyeonworks-prod + podSelector: + matchLabels: + app: techlog-backend + ports: + - port: 9000 + protocol: TCP + podSelector: + matchLabels: + aistor.min.io/objectStore: minio-aistor + policyTypes: + - Ingress diff --git a/services/tenants/techlog/observability-network-policies.yaml b/services/tenants/techlog/observability-network-policies.yaml new file mode 100644 index 0000000..bc4e88f --- /dev/null +++ b/services/tenants/techlog/observability-network-policies.yaml @@ -0,0 +1,45 @@ +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: blackbox-allow-techlog-health + namespace: observability +spec: + egress: + - ports: + - port: 9001 + protocol: TCP + to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: hyeonworks-prod + podSelector: + matchLabels: + app: techlog-backend + podSelector: + matchLabels: + app.kubernetes.io/name: prometheus-blackbox-exporter + policyTypes: + - Egress +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: observability-allow-techlog-scrape + namespace: observability +spec: + egress: + - ports: + - port: 9001 + protocol: TCP + to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: hyeonworks-prod + podSelector: + matchLabels: + app: techlog-backend + podSelector: + matchLabels: + app.kubernetes.io/name: prometheus + policyTypes: + - Egress diff --git a/services/tenants/techlog/observability-probes.yaml b/services/tenants/techlog/observability-probes.yaml new file mode 100644 index 0000000..71e0bcd --- /dev/null +++ b/services/tenants/techlog/observability-probes.yaml @@ -0,0 +1,46 @@ +apiVersion: monitoring.coreos.com/v1 +kind: Probe +metadata: + labels: + observability.hyeonworks.com/instance: home + name: techlog-edge + namespace: observability +spec: + interval: 30s + jobName: techlog-edge + module: http_2xx + prober: + path: /probe + scheme: http + url: blackbox-exporter.observability.svc.cluster.local:9115 + scrapeTimeout: 15s + targets: + staticConfig: + labels: + application: techlog + static: + - https://hyeonworks.com/ + - https://hyeonworks.com/api/healthcheck +--- +apiVersion: monitoring.coreos.com/v1 +kind: Probe +metadata: + labels: + observability.hyeonworks.com/instance: home + name: techlog-readiness + namespace: observability +spec: + interval: 30s + jobName: techlog-readiness + module: http_2xx + prober: + path: /probe + scheme: http + url: blackbox-exporter.observability.svc.cluster.local:9115 + scrapeTimeout: 15s + targets: + staticConfig: + labels: + application: techlog + static: + - http://techlog-management.hyeonworks-prod.svc.cluster.local:9001/actuator/health/readiness diff --git a/services/tenants/techlog/observability-rules.yaml b/services/tenants/techlog/observability-rules.yaml new file mode 100644 index 0000000..89eec74 --- /dev/null +++ b/services/tenants/techlog/observability-rules.yaml @@ -0,0 +1,67 @@ +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + labels: + observability.hyeonworks.com/instance: home + name: techlog + namespace: observability +spec: + groups: + - name: techlog + rules: + - alert: TechLogMetricsDown + annotations: + summary: TechLog metrics target is unreachable + expr: up{namespace="hyeonworks-prod",service="techlog-management"} == 0 + for: 5m + labels: + application: techlog + severity: warning + - alert: TechLogReadinessDown + annotations: + summary: TechLog dependency readiness is DOWN + expr: probe_success{job="techlog-readiness"} == 0 + for: 3m + labels: + application: techlog + severity: critical + - alert: TechLogHTTPSDown + annotations: + summary: TechLog HTTPS or API health is failing + expr: probe_success{job="techlog-edge"} == 0 + for: 3m + labels: + application: techlog + severity: critical + - alert: TechLogHTTP5xx + annotations: + summary: TechLog HTTP 5xx exceeds 5% + expr: (sum(rate(http_server_requests_seconds_count{namespace="hyeonworks-prod",status=~"5.."}[5m])) / clamp_min(sum(rate(http_server_requests_seconds_count{namespace="hyeonworks-prod"}[5m])),0.001)) > 0.05 and sum(increase(http_server_requests_seconds_count{namespace="hyeonworks-prod"}[5m])) > 20 + for: 5m + labels: + application: techlog + severity: warning + - alert: TechLogLatencyHigh + annotations: + summary: TechLog p99 latency exceeds 2 seconds + expr: histogram_quantile(0.99,sum by(le)(rate(http_server_requests_seconds_bucket{namespace="hyeonworks-prod",uri!~"/actuator.*"}[5m]))) > 2 + for: 10m + labels: + application: techlog + severity: warning + - alert: TechLogHeapPressure + annotations: + summary: TechLog heap usage exceeds 85% + expr: sum(jvm_memory_used_bytes{namespace="hyeonworks-prod",area="heap"}) / sum(jvm_memory_max_bytes{namespace="hyeonworks-prod",area="heap"}) > 0.85 + for: 10m + labels: + application: techlog + severity: warning + - alert: TechLogDatabasePoolWaiting + annotations: + summary: TechLog database connections are queued + expr: hikaricp_connections_pending{namespace="hyeonworks-prod"} > 0 + for: 5m + labels: + application: techlog + severity: warning diff --git a/services/tenants/techlog/platform-data.yaml b/services/tenants/techlog/platform-data.yaml new file mode 100644 index 0000000..4b1bc54 --- /dev/null +++ b/services/tenants/techlog/platform-data.yaml @@ -0,0 +1,55 @@ +apiVersion: postgresql.cnpg.io/v1 +kind: Database +metadata: + name: platform-postgres-techlog + namespace: platform-data +spec: + cluster: + name: platform-postgres + databaseReclaimPolicy: retain + ensure: present + name: techlog + owner: techlog +--- +apiVersion: postgresql.cnpg.io/v1 +kind: DatabaseRole +metadata: + name: platform-postgres-techlog + namespace: platform-data +spec: + bypassrls: false + cluster: + name: platform-postgres + connectionLimit: 30 + createdb: false + createrole: false + databaseRoleReclaimPolicy: retain + login: true + name: techlog + passwordSecret: + name: techlog-db-credentials + replication: false + superuser: false +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-techlog-backend + namespace: platform-data +spec: + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: hyeonworks-prod + podSelector: + matchLabels: + app: techlog-backend + ports: + - port: 5432 + protocol: TCP + podSelector: + matchLabels: + cnpg.io/cluster: platform-postgres + policyTypes: + - Ingress From 3eafb7e1d896d7512c732e577858ca7db96c478c Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Thu, 12 Nov 2026 00:29:18 +0900 Subject: [PATCH 4/4] feat(validate): add a live drift check against the repository renders The 2026-11-11 review found a pg_hba rule that existed only in the cluster by rendering every root by hand and running kubectl diff. Nothing does that regularly, so drift is found by accident. live-drift-check.sh renders all live roots (Phase 1 and 2, admin services, observability core, the Traefik trust overlay, the tenant, PriorityClass and keycloak policy roots), strips Secrets and runs kubectl diff, which is a server-side dry run. It prints one line per drifted object, honours a dated accepted-drift list and can write Prometheus textfile metrics for a later alert. Risk: read-only. It consumes the render scripts' verified-output-dir handoff and deletes those directories on exit; the README notes this second consumer. Secret values are not compared. Verification (desktop, before this series is applied): 17 drifted objects, matching the review plus the new roots; exit 1; the textfile holds platform_live_drift_objects 17. --- scripts/validate/README.md | 37 +++++ scripts/validate/live-drift-accepted.txt | 10 ++ scripts/validate/live-drift-check.sh | 173 +++++++++++++++++++++++ 3 files changed, 220 insertions(+) create mode 100644 scripts/validate/live-drift-accepted.txt create mode 100644 scripts/validate/live-drift-check.sh diff --git a/scripts/validate/README.md b/scripts/validate/README.md index 6c6df6e..b30e14a 100644 --- a/scripts/validate/README.md +++ b/scripts/validate/README.md @@ -559,3 +559,40 @@ parser와 세 mode의 fixture 회귀는 live cluster에 연결하지 않고 다 ```sh bash scripts/validate/test-k3s-secret-encryption-status.sh ``` + +## 테넌트·우선순위·keycloak 정책 렌더 검사 + +`render-tenants-and-policies.sh`는 클러스터에 닿지 않고 세 빌드 루트를 렌더해 불변식을 +확인한다. + +- `services/tenants/techlog`: tech-log 테넌트의 플랫폼 쪽 객체 11개(목록이 정확히 같아야 + 한다), `hyeonworks-prod` PSA 라벨, techlog DatabaseRole 권한 플래그, 대시보드 이름 +- `services/priority-classes`: `platform-critical`(0보다 크고 시스템 클래스보다 작다), + `platform-low`(0보다 작고 선점하지 않는다), 둘 다 global default가 아니다 +- `infrastructure/networking/keycloak`: namespace 전체 기본 차단, egress 규칙마다 포트, + 허용 CIDR은 API 서버 둘뿐 + +```sh +bash scripts/validate/render-tenants-and-policies.sh +``` + +## 라이브 drift 검사 + +`live-drift-check.sh`는 이 저장소의 라이브 빌드 루트를 모두 렌더하고(Phase 1·2, 관리 도구, +관측 core, Traefik trust overlay, 위 세 루트) Secret을 뺀 뒤 `kubectl diff`(서버 dry-run, +저장하지 않음)로 클러스터와 비교한다. 렌더에는 위 render 스크립트의 `--verified-output-dir` +출력을 읽기만 하는 용도로 쓴다(그 디렉터리는 스크립트가 끝날 때 지운다). + +```sh +PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm \ + bash scripts/validate/live-drift-check.sh \ + --accepted scripts/validate/live-drift-accepted.txt +``` + +- 종료 코드: 0 차이 없음, 1 차이 있음, 2 검사 자체 실패. +- `DRIFT / (+추가 -삭제) [입력 파일]`로 객체마다 한 줄을 쓴다. + `generation` 줄만 다른 것은 세지 않는다. +- `--accepted`: 알고 있는 차이를 기한과 이유를 달아 적는다. 기한이 지나면 다시 drift로 센다. +- `--textfile FILE`: `platform_live_drift_objects`, `platform_live_drift_check_success`, + `platform_live_drift_last_run_timestamp_seconds`를 Prometheus textfile 형식으로 쓴다. +- Secret 값의 차이는 보지 않는다. diff --git a/scripts/validate/live-drift-accepted.txt b/scripts/validate/live-drift-accepted.txt new file mode 100644 index 0000000..762302b --- /dev/null +++ b/scripts/validate/live-drift-accepted.txt @@ -0,0 +1,10 @@ +# 알고 있는 라이브 drift. 형식: <기한 YYYY-MM-DD> <이유> +# 기한이 지나면 live-drift-check.sh가 다시 drift로 센다. 해소하면 줄을 지운다. +Cluster platform-data platform-postgres 2026-11-30 pg_hba의 techlog 두 줄이 repo에 없다. w1이 CNPG 백업과 함께 넣는다(1c4ef9a). +DatabaseRole platform-data platform-postgres-gitea 2026-11-30 렌더가 false 기본값 필드를 명시할 뿐 값은 같다. 다음 Phase 1 적용 때 사라진다. +DatabaseRole platform-data platform-postgres-keycloak 2026-11-30 렌더가 false 기본값 필드를 명시할 뿐 값은 같다. 다음 Phase 2 적용 때 사라진다. +Deployment keycloak keycloak-operator 2026-11-30 원격 ref 26.7.0 태그 내용이 바뀌어 build-timestamp 라벨만 다르다. 재시작 창에 커밋 SHA 고정과 함께 적용한다. +Service keycloak keycloak-operator 2026-11-30 위와 같은 build-timestamp 라벨. +ServiceAccount keycloak keycloak-operator 2026-11-30 위와 같은 build-timestamp 라벨. +Deployment observability observability-core-kube-pr-operator 2026-11-30 이미지 문자열이 tag@digest와 digest로 다르고 digest는 같다. 적용하면 재시작되므로 재시작 창에. +Deployment observability observability-core-kube-state-metrics 2026-11-30 위와 같은 이미지 문자열 차이. diff --git a/scripts/validate/live-drift-check.sh b/scripts/validate/live-drift-check.sh new file mode 100644 index 0000000..4e85a2d --- /dev/null +++ b/scripts/validate/live-drift-check.sh @@ -0,0 +1,173 @@ +#!/usr/bin/env bash +# Render every live build root of this repository and compare it with the +# cluster using `kubectl diff` (server-side dry-run: nothing is persisted). +# +# Usage: +# PLATFORM_HELM_BIN=/abs/helm bash scripts/validate/live-drift-check.sh \ +# [--accepted FILE] [--textfile FILE] +# +# --accepted FILE known differences, one per line: +# ` ` +# Expired entries count as drift again. +# --textfile FILE also write Prometheus textfile metrics (atomic rename). +# +# Exit status: 0 no drift, 1 drift found, 2 the check itself failed. +# Secrets are removed from the renders before diffing; Secret drift is out of scope. +set -Eeuo pipefail + +readonly REPOSITORY_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)" +accepted_file="" +textfile="" +while (( $# > 0 )); do + case "$1" in + --accepted) accepted_file="${2:?--accepted needs a file}"; shift 2 ;; + --textfile) textfile="${2:?--textfile needs a file}"; shift 2 ;; + -h|--help) sed -n '2,16p' "${BASH_SOURCE[0]}"; exit 0 ;; + *) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;; + esac +done + + +declare -a made_dirs=() +cleanup() { + local d + for d in "${made_dirs[@]}"; do + case "$d" in /tmp/platform-*) rm -rf -- "$d" ;; esac + done +} +trap cleanup EXIT + +write_textfile() { + local drift="$1" ok="$2" + [[ -n "$textfile" ]] || return 0 + local tmp="${textfile}.tmp.$$" + { + echo '# HELP platform_live_drift_objects Objects whose live state differs from the repository render.' + echo '# TYPE platform_live_drift_objects gauge' + echo "platform_live_drift_objects ${drift}" + echo '# HELP platform_live_drift_check_success 1 when the drift check itself completed.' + echo '# TYPE platform_live_drift_check_success gauge' + echo "platform_live_drift_check_success ${ok}" + echo '# HELP platform_live_drift_last_run_timestamp_seconds Unix time the drift check finished.' + echo '# TYPE platform_live_drift_last_run_timestamp_seconds gauge' + echo "platform_live_drift_last_run_timestamp_seconds $(date +%s)" + } >"$tmp" + mv -f -- "$tmp" "$textfile" +} + +die() { + printf 'DRIFT CHECK ERROR: %s\n' "$*" >&2 + write_textfile 0 0 + exit 2 +} + +work="$(mktemp -d /tmp/platform-drift.XXXXXX)"; made_dirs+=("$work") +mkdir -p "$work/in" "$work/out" + +render_root() { + local script="$1" prefix="$2" dir + dir="$(mktemp -d "/tmp/${prefix}.XXXXXX")"; made_dirs+=("$dir") + bash "$REPOSITORY_ROOT/scripts/validate/${script}" --verified-output-dir "$dir" \ + >"$work/${script}.log" 2>&1 || { tail -5 "$work/${script}.log" >&2; die "${script} failed"; } + printf '%s\n' "$dir" +} + +phase1="$(render_root render-phase1.sh platform-phase1-apply)" +phase2="$(render_root render-phase2.sh platform-phase2-apply)" +admin="$(render_root render-admin-services.sh platform-admin-apply)" +observability="$(render_root render-observability-core.sh platform-observability-core-apply)" + +# The live Gitea is the OIDC profile; gitea.yaml is the fresh-install baseline only. +declare -a inputs=( + "$phase1/namespaces.yaml" "$phase1/ssd-local-pv.yaml" "$phase1/cnpg-operator.yaml" + "$phase1/platform-postgres.yaml" "$phase1/gitea-oidc.yaml" + "$observability/aggregate.yaml" +) +for f in "$phase2"/*.yaml "$admin"/*.yaml; do inputs+=("$f"); done + +# Live Traefik is the trust overlay (infrastructure/networking/traefik/README.md). +kubectl kustomize "$REPOSITORY_ROOT/infrastructure/networking/traefik/overlays/trust" >"$work/traefik-trust.yaml" || die "traefik render failed" +kubectl kustomize "$REPOSITORY_ROOT/services/tenants/techlog" >"$work/tenant-techlog.yaml" || die "tenant render failed" +kubectl kustomize "$REPOSITORY_ROOT/services/priority-classes" >"$work/priority-classes.yaml" || die "priority class render failed" +kubectl kustomize "$REPOSITORY_ROOT/infrastructure/networking/keycloak" >"$work/keycloak-network-policies.yaml" || die "keycloak policy render failed" +inputs+=("$work/traefik-trust.yaml" "$work/tenant-techlog.yaml" "$work/priority-classes.yaml" "$work/keycloak-network-policies.yaml") + +n=0 +for f in "${inputs[@]}"; do + n=$((n + 1)) + python3 -I - "$f" "$work/in/$(printf '%02d' "$n")-$(basename "$f")" <<'PY' || die "could not strip Secrets from $f" +import sys, yaml +yaml.SafeLoader.add_constructor("tag:yaml.org,2002:value", lambda l, n: l.construct_scalar(n)) +docs = [d for d in yaml.safe_load_all(open(sys.argv[1])) if isinstance(d, dict) and d.get("kind") not in (None, "Secret")] +yaml.safe_dump_all(docs, open(sys.argv[2], "w"), sort_keys=False) +PY +done + +for f in "$work"/in/*.yaml; do + set +e + KUBECTL_EXTERNAL_DIFF="diff -u -N" kubectl diff -f "$f" >"$work/out/$(basename "$f").diff" 2>"$work/out/$(basename "$f").err" + rc=$? + set -e + if (( rc > 1 )); then + sed -n '1,5p' "$work/out/$(basename "$f").err" >&2 + die "kubectl diff failed for $(basename "$f")" + fi +done + +set +e +python3 -I - "$work/out" "${accepted_file:-}" "$work/drift-count" <<'PY' +import sys, pathlib, re, datetime +out, accepted_path, count_path = pathlib.Path(sys.argv[1]), sys.argv[2], pathlib.Path(sys.argv[3]) +today = datetime.date.today() +accepted, expired = {}, set() +if accepted_path: + for line in pathlib.Path(accepted_path).read_text().splitlines(): + line = line.strip() + if not line or line.startswith("#"): + continue + kind, ns, name, until, *reason = line.split() + key = (kind, "" if ns == "-" else ns, name) + if datetime.date.fromisoformat(until) < today: + expired.add(key) + else: + accepted[key] = " ".join(reason) +version = re.compile(r"^v\d+((alpha|beta)\d+)?$") +objects = {} +for diff in sorted(out.glob("*.diff")): + current = None + for line in diff.read_text().splitlines(): + if line.startswith("diff -u -N "): + parts = line.split()[3].rsplit("/", 1)[-1].split(".") + i = next(i for i, p in enumerate(parts) if version.match(p)) + current = (parts[i + 1], parts[i + 2], ".".join(parts[i + 3:])) + objects.setdefault(current, [0, 0, diff.name]) + elif current and line[:1] in "+-" and not line.startswith(("+++", "---")): + if re.match(r"^[+-]\s+generation:", line): + continue + objects[current][0 if line[0] == "+" else 1] += 1 +drift = accepted_seen = 0 +for key, (plus, minus, src) in sorted(objects.items()): + if plus == 0 and minus == 0: + continue + kind, ns, name = key + label = f"{kind} {ns or '-'}/{name} (+{plus} -{minus}) [{src}]" + if key in accepted: + accepted_seen += 1 + print(f"ACCEPTED {label}: {accepted[key]}") + continue + drift += 1 + print(f"DRIFT {label}" + (" (accepted entry expired)" if key in expired else "")) +print(f"SUMMARY drift={drift} accepted={accepted_seen}") +count_path.write_text(str(drift)) +PY +rc=$? +set -e +(( rc == 0 )) || die "could not summarize the diffs" +drift_count="$(cat "$work/drift-count")" +write_textfile "$drift_count" 1 +if (( drift_count == 0 )); then + echo "LIVE DRIFT CHECK PASS" + exit 0 +fi +echo "LIVE DRIFT CHECK FOUND DRIFT" +exit 1