diff --git a/infrastructure/networking/keycloak/README.md b/infrastructure/networking/keycloak/README.md new file mode 100644 index 0000000..08df1c3 --- /dev/null +++ b/infrastructure/networking/keycloak/README.md @@ -0,0 +1,38 @@ +# keycloak namespace NetworkPolicy + +2026-11-11 점검 전 `keycloak` namespace에는 Keycloak Operator가 만든 ingress 정책 하나뿐이었다. +Keycloak 파드와 Operator 파드의 egress는 막혀 있지 않았다. 다른 플랫폼 namespace처럼 기본 차단에 +필요한 것만 연다. + +| 정책 | 대상 | 허용 | +| --- | --- | --- | +| `keycloak-default-deny` | 모든 파드 | ingress·egress 기본 차단 | +| `keycloak-allow-dns` | 모든 파드 | kube-dns 53 | +| `keycloak-allow-platform-postgres` | 모든 파드(서버와 realm import Job) | `platform-data` Postgres 5432 | +| `keycloak-allow-cluster-peers` | Keycloak 파드 | 같은 Keycloak 파드 JGroups 7800·57800 | +| `keycloak-operator-allow-egress` | Operator | API 서버(`10.43.0.1:443`, `192.168.0.107:6443`), Keycloak 8080·9000 | + +- Keycloak 파드의 **ingress**는 계속 Keycloak CR `spec.networkPolicy`로 Operator가 만드는 + `keycloak-network-policy`가 소유한다(Traefik 8080, 관리 9000, JGroups). +- 바깥(인터넷)으로 나가는 길은 없다. 2026-11-12에 repo와 Keycloak CR에 SMTP·외부 IdP 설정이 + 없고, Keycloak 파드의 열린 TCP 연결이 Postgres뿐인 것을 확인했다. SMTP나 외부 IdP를 붙이면 + 여기에 egress를 더해야 한다(안 더하면 메일·외부 로그인만 조용히 실패한다). +- kubelet probe는 노드에서 오므로 막히지 않는다. + +## 적용과 확인 + +```sh +bash scripts/validate/render-tenants-and-policies.sh +kubectl kustomize infrastructure/networking/keycloak | kubectl diff -f - +kubectl apply -k infrastructure/networking/keycloak +``` + +적용 뒤 확인(모두 통과해야 한다): + +```sh +kubectl -n keycloak get pods +curl -fsS -o /dev/null -w '%{http_code}\n' https://id.learn.hyeonworks.com/realms/hyeonworks/.well-known/openid-configuration +kubectl -n keycloak logs deploy/keycloak-operator --since=5m | grep -i -E 'error|exception|timeout' || echo no-operator-errors +``` + +되돌리기: `kubectl delete -k infrastructure/networking/keycloak`(Operator 정책은 남는다). diff --git a/infrastructure/networking/keycloak/kustomization.yaml b/infrastructure/networking/keycloak/kustomization.yaml new file mode 100644 index 0000000..6dabb5c --- /dev/null +++ b/infrastructure/networking/keycloak/kustomization.yaml @@ -0,0 +1,12 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: keycloak + +labels: + - pairs: + app.kubernetes.io/part-of: platform + includeSelectors: false + +resources: + - network-policies.yaml diff --git a/infrastructure/networking/keycloak/network-policies.yaml b/infrastructure/networking/keycloak/network-policies.yaml new file mode 100644 index 0000000..6808550 --- /dev/null +++ b/infrastructure/networking/keycloak/network-policies.yaml @@ -0,0 +1,117 @@ +# keycloak namespace: 기본 차단 + 필요한 것만 허용. +# Keycloak 파드의 ingress는 Keycloak CR(spec.networkPolicy)로 Operator가 만드는 +# keycloak-network-policy가 계속 소유한다. 여기서는 기본 차단과 egress만 정한다. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: keycloak-default-deny +spec: + podSelector: {} + policyTypes: + - Ingress + - Egress +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: keycloak-allow-dns +spec: + podSelector: {} + policyTypes: + - Egress + egress: + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + podSelector: + matchLabels: + k8s-app: kube-dns + ports: + - port: 53 + protocol: UDP + - port: 53 + protocol: TCP +--- +# Keycloak 서버와 realm import Job이 공용 PostgreSQL에 붙는다. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: keycloak-allow-platform-postgres +spec: + podSelector: {} + policyTypes: + - Egress + egress: + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: platform-data + podSelector: + matchLabels: + cnpg.io/cluster: platform-postgres + ports: + - port: 5432 + protocol: TCP +--- +# 같은 Keycloak 인스턴스끼리의 JGroups(7800, 57800). Operator ingress 정책과 짝이다. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: keycloak-allow-cluster-peers +spec: + podSelector: + matchLabels: + app: keycloak + app.kubernetes.io/instance: keycloak + app.kubernetes.io/managed-by: keycloak-operator + policyTypes: + - Egress + egress: + - to: + - podSelector: + matchLabels: + app: keycloak + app.kubernetes.io/instance: keycloak + app.kubernetes.io/managed-by: keycloak-operator + ports: + - port: 7800 + protocol: TCP + - port: 57800 + protocol: TCP +--- +# Operator: Kubernetes API(ClusterIP와 실제 API 서버 주소)와 Keycloak 관리 포트. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: keycloak-operator-allow-egress +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: keycloak-operator + policyTypes: + - Egress + egress: + - to: + - ipBlock: + cidr: 10.43.0.1/32 + ports: + - port: 443 + protocol: TCP + - to: + - ipBlock: + cidr: 192.168.0.107/32 + ports: + - port: 6443 + protocol: TCP + - to: + - podSelector: + matchLabels: + app: keycloak + app.kubernetes.io/instance: keycloak + app.kubernetes.io/managed-by: keycloak-operator + ports: + - port: 8080 + protocol: TCP + - port: 9000 + protocol: TCP diff --git a/scripts/validate/README.md b/scripts/validate/README.md index 6c6df6e..b30e14a 100644 --- a/scripts/validate/README.md +++ b/scripts/validate/README.md @@ -559,3 +559,40 @@ parser와 세 mode의 fixture 회귀는 live cluster에 연결하지 않고 다 ```sh bash scripts/validate/test-k3s-secret-encryption-status.sh ``` + +## 테넌트·우선순위·keycloak 정책 렌더 검사 + +`render-tenants-and-policies.sh`는 클러스터에 닿지 않고 세 빌드 루트를 렌더해 불변식을 +확인한다. + +- `services/tenants/techlog`: tech-log 테넌트의 플랫폼 쪽 객체 11개(목록이 정확히 같아야 + 한다), `hyeonworks-prod` PSA 라벨, techlog DatabaseRole 권한 플래그, 대시보드 이름 +- `services/priority-classes`: `platform-critical`(0보다 크고 시스템 클래스보다 작다), + `platform-low`(0보다 작고 선점하지 않는다), 둘 다 global default가 아니다 +- `infrastructure/networking/keycloak`: namespace 전체 기본 차단, egress 규칙마다 포트, + 허용 CIDR은 API 서버 둘뿐 + +```sh +bash scripts/validate/render-tenants-and-policies.sh +``` + +## 라이브 drift 검사 + +`live-drift-check.sh`는 이 저장소의 라이브 빌드 루트를 모두 렌더하고(Phase 1·2, 관리 도구, +관측 core, Traefik trust overlay, 위 세 루트) Secret을 뺀 뒤 `kubectl diff`(서버 dry-run, +저장하지 않음)로 클러스터와 비교한다. 렌더에는 위 render 스크립트의 `--verified-output-dir` +출력을 읽기만 하는 용도로 쓴다(그 디렉터리는 스크립트가 끝날 때 지운다). + +```sh +PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm \ + bash scripts/validate/live-drift-check.sh \ + --accepted scripts/validate/live-drift-accepted.txt +``` + +- 종료 코드: 0 차이 없음, 1 차이 있음, 2 검사 자체 실패. +- `DRIFT / (+추가 -삭제) [입력 파일]`로 객체마다 한 줄을 쓴다. + `generation` 줄만 다른 것은 세지 않는다. +- `--accepted`: 알고 있는 차이를 기한과 이유를 달아 적는다. 기한이 지나면 다시 drift로 센다. +- `--textfile FILE`: `platform_live_drift_objects`, `platform_live_drift_check_success`, + `platform_live_drift_last_run_timestamp_seconds`를 Prometheus textfile 형식으로 쓴다. +- Secret 값의 차이는 보지 않는다. diff --git a/scripts/validate/live-drift-accepted.txt b/scripts/validate/live-drift-accepted.txt new file mode 100644 index 0000000..762302b --- /dev/null +++ b/scripts/validate/live-drift-accepted.txt @@ -0,0 +1,10 @@ +# 알고 있는 라이브 drift. 형식: <기한 YYYY-MM-DD> <이유> +# 기한이 지나면 live-drift-check.sh가 다시 drift로 센다. 해소하면 줄을 지운다. +Cluster platform-data platform-postgres 2026-11-30 pg_hba의 techlog 두 줄이 repo에 없다. w1이 CNPG 백업과 함께 넣는다(1c4ef9a). +DatabaseRole platform-data platform-postgres-gitea 2026-11-30 렌더가 false 기본값 필드를 명시할 뿐 값은 같다. 다음 Phase 1 적용 때 사라진다. +DatabaseRole platform-data platform-postgres-keycloak 2026-11-30 렌더가 false 기본값 필드를 명시할 뿐 값은 같다. 다음 Phase 2 적용 때 사라진다. +Deployment keycloak keycloak-operator 2026-11-30 원격 ref 26.7.0 태그 내용이 바뀌어 build-timestamp 라벨만 다르다. 재시작 창에 커밋 SHA 고정과 함께 적용한다. +Service keycloak keycloak-operator 2026-11-30 위와 같은 build-timestamp 라벨. +ServiceAccount keycloak keycloak-operator 2026-11-30 위와 같은 build-timestamp 라벨. +Deployment observability observability-core-kube-pr-operator 2026-11-30 이미지 문자열이 tag@digest와 digest로 다르고 digest는 같다. 적용하면 재시작되므로 재시작 창에. +Deployment observability observability-core-kube-state-metrics 2026-11-30 위와 같은 이미지 문자열 차이. diff --git a/scripts/validate/live-drift-check.sh b/scripts/validate/live-drift-check.sh new file mode 100644 index 0000000..4e85a2d --- /dev/null +++ b/scripts/validate/live-drift-check.sh @@ -0,0 +1,173 @@ +#!/usr/bin/env bash +# Render every live build root of this repository and compare it with the +# cluster using `kubectl diff` (server-side dry-run: nothing is persisted). +# +# Usage: +# PLATFORM_HELM_BIN=/abs/helm bash scripts/validate/live-drift-check.sh \ +# [--accepted FILE] [--textfile FILE] +# +# --accepted FILE known differences, one per line: +# ` ` +# Expired entries count as drift again. +# --textfile FILE also write Prometheus textfile metrics (atomic rename). +# +# Exit status: 0 no drift, 1 drift found, 2 the check itself failed. +# Secrets are removed from the renders before diffing; Secret drift is out of scope. +set -Eeuo pipefail + +readonly REPOSITORY_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)" +accepted_file="" +textfile="" +while (( $# > 0 )); do + case "$1" in + --accepted) accepted_file="${2:?--accepted needs a file}"; shift 2 ;; + --textfile) textfile="${2:?--textfile needs a file}"; shift 2 ;; + -h|--help) sed -n '2,16p' "${BASH_SOURCE[0]}"; exit 0 ;; + *) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;; + esac +done + + +declare -a made_dirs=() +cleanup() { + local d + for d in "${made_dirs[@]}"; do + case "$d" in /tmp/platform-*) rm -rf -- "$d" ;; esac + done +} +trap cleanup EXIT + +write_textfile() { + local drift="$1" ok="$2" + [[ -n "$textfile" ]] || return 0 + local tmp="${textfile}.tmp.$$" + { + echo '# HELP platform_live_drift_objects Objects whose live state differs from the repository render.' + echo '# TYPE platform_live_drift_objects gauge' + echo "platform_live_drift_objects ${drift}" + echo '# HELP platform_live_drift_check_success 1 when the drift check itself completed.' + echo '# TYPE platform_live_drift_check_success gauge' + echo "platform_live_drift_check_success ${ok}" + echo '# HELP platform_live_drift_last_run_timestamp_seconds Unix time the drift check finished.' + echo '# TYPE platform_live_drift_last_run_timestamp_seconds gauge' + echo "platform_live_drift_last_run_timestamp_seconds $(date +%s)" + } >"$tmp" + mv -f -- "$tmp" "$textfile" +} + +die() { + printf 'DRIFT CHECK ERROR: %s\n' "$*" >&2 + write_textfile 0 0 + exit 2 +} + +work="$(mktemp -d /tmp/platform-drift.XXXXXX)"; made_dirs+=("$work") +mkdir -p "$work/in" "$work/out" + +render_root() { + local script="$1" prefix="$2" dir + dir="$(mktemp -d "/tmp/${prefix}.XXXXXX")"; made_dirs+=("$dir") + bash "$REPOSITORY_ROOT/scripts/validate/${script}" --verified-output-dir "$dir" \ + >"$work/${script}.log" 2>&1 || { tail -5 "$work/${script}.log" >&2; die "${script} failed"; } + printf '%s\n' "$dir" +} + +phase1="$(render_root render-phase1.sh platform-phase1-apply)" +phase2="$(render_root render-phase2.sh platform-phase2-apply)" +admin="$(render_root render-admin-services.sh platform-admin-apply)" +observability="$(render_root render-observability-core.sh platform-observability-core-apply)" + +# The live Gitea is the OIDC profile; gitea.yaml is the fresh-install baseline only. +declare -a inputs=( + "$phase1/namespaces.yaml" "$phase1/ssd-local-pv.yaml" "$phase1/cnpg-operator.yaml" + "$phase1/platform-postgres.yaml" "$phase1/gitea-oidc.yaml" + "$observability/aggregate.yaml" +) +for f in "$phase2"/*.yaml "$admin"/*.yaml; do inputs+=("$f"); done + +# Live Traefik is the trust overlay (infrastructure/networking/traefik/README.md). +kubectl kustomize "$REPOSITORY_ROOT/infrastructure/networking/traefik/overlays/trust" >"$work/traefik-trust.yaml" || die "traefik render failed" +kubectl kustomize "$REPOSITORY_ROOT/services/tenants/techlog" >"$work/tenant-techlog.yaml" || die "tenant render failed" +kubectl kustomize "$REPOSITORY_ROOT/services/priority-classes" >"$work/priority-classes.yaml" || die "priority class render failed" +kubectl kustomize "$REPOSITORY_ROOT/infrastructure/networking/keycloak" >"$work/keycloak-network-policies.yaml" || die "keycloak policy render failed" +inputs+=("$work/traefik-trust.yaml" "$work/tenant-techlog.yaml" "$work/priority-classes.yaml" "$work/keycloak-network-policies.yaml") + +n=0 +for f in "${inputs[@]}"; do + n=$((n + 1)) + python3 -I - "$f" "$work/in/$(printf '%02d' "$n")-$(basename "$f")" <<'PY' || die "could not strip Secrets from $f" +import sys, yaml +yaml.SafeLoader.add_constructor("tag:yaml.org,2002:value", lambda l, n: l.construct_scalar(n)) +docs = [d for d in yaml.safe_load_all(open(sys.argv[1])) if isinstance(d, dict) and d.get("kind") not in (None, "Secret")] +yaml.safe_dump_all(docs, open(sys.argv[2], "w"), sort_keys=False) +PY +done + +for f in "$work"/in/*.yaml; do + set +e + KUBECTL_EXTERNAL_DIFF="diff -u -N" kubectl diff -f "$f" >"$work/out/$(basename "$f").diff" 2>"$work/out/$(basename "$f").err" + rc=$? + set -e + if (( rc > 1 )); then + sed -n '1,5p' "$work/out/$(basename "$f").err" >&2 + die "kubectl diff failed for $(basename "$f")" + fi +done + +set +e +python3 -I - "$work/out" "${accepted_file:-}" "$work/drift-count" <<'PY' +import sys, pathlib, re, datetime +out, accepted_path, count_path = pathlib.Path(sys.argv[1]), sys.argv[2], pathlib.Path(sys.argv[3]) +today = datetime.date.today() +accepted, expired = {}, set() +if accepted_path: + for line in pathlib.Path(accepted_path).read_text().splitlines(): + line = line.strip() + if not line or line.startswith("#"): + continue + kind, ns, name, until, *reason = line.split() + key = (kind, "" if ns == "-" else ns, name) + if datetime.date.fromisoformat(until) < today: + expired.add(key) + else: + accepted[key] = " ".join(reason) +version = re.compile(r"^v\d+((alpha|beta)\d+)?$") +objects = {} +for diff in sorted(out.glob("*.diff")): + current = None + for line in diff.read_text().splitlines(): + if line.startswith("diff -u -N "): + parts = line.split()[3].rsplit("/", 1)[-1].split(".") + i = next(i for i, p in enumerate(parts) if version.match(p)) + current = (parts[i + 1], parts[i + 2], ".".join(parts[i + 3:])) + objects.setdefault(current, [0, 0, diff.name]) + elif current and line[:1] in "+-" and not line.startswith(("+++", "---")): + if re.match(r"^[+-]\s+generation:", line): + continue + objects[current][0 if line[0] == "+" else 1] += 1 +drift = accepted_seen = 0 +for key, (plus, minus, src) in sorted(objects.items()): + if plus == 0 and minus == 0: + continue + kind, ns, name = key + label = f"{kind} {ns or '-'}/{name} (+{plus} -{minus}) [{src}]" + if key in accepted: + accepted_seen += 1 + print(f"ACCEPTED {label}: {accepted[key]}") + continue + drift += 1 + print(f"DRIFT {label}" + (" (accepted entry expired)" if key in expired else "")) +print(f"SUMMARY drift={drift} accepted={accepted_seen}") +count_path.write_text(str(drift)) +PY +rc=$? +set -e +(( rc == 0 )) || die "could not summarize the diffs" +drift_count="$(cat "$work/drift-count")" +write_textfile "$drift_count" 1 +if (( drift_count == 0 )); then + echo "LIVE DRIFT CHECK PASS" + exit 0 +fi +echo "LIVE DRIFT CHECK FOUND DRIFT" +exit 1 diff --git a/scripts/validate/render-tenants-and-policies.sh b/scripts/validate/render-tenants-and-policies.sh new file mode 100644 index 0000000..49bd860 --- /dev/null +++ b/scripts/validate/render-tenants-and-policies.sh @@ -0,0 +1,126 @@ +#!/usr/bin/env bash +# Render the tech-log tenant root, the platform PriorityClasses and the keycloak +# namespace NetworkPolicies, and check their invariants. Static: no cluster access. +set -Eeuo pipefail + +readonly EXPECTED_KUSTOMIZE_VERSION="v5.8.1" +readonly REPOSITORY_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)" + +fail() { + printf 'ERROR: %s\n' "$*" >&2 + exit 1 +} + +kustomize_version="$(kubectl version --client --output=yaml | sed -n 's/^kustomizeVersion: //p')" +[[ "$kustomize_version" == "$EXPECTED_KUSTOMIZE_VERSION" ]] || \ + fail "expected Kustomize ${EXPECTED_KUSTOMIZE_VERSION}, found ${kustomize_version:-unknown}" + +work="$(mktemp -d "${TMPDIR:-/tmp}/platform-tenants-render.XXXXXX")" +trap 'rm -rf -- "$work"' EXIT + +kubectl kustomize "$REPOSITORY_ROOT/services/tenants/techlog" >"$work/techlog.yaml" +kubectl kustomize "$REPOSITORY_ROOT/services/priority-classes" >"$work/priority-classes.yaml" +kubectl kustomize "$REPOSITORY_ROOT/infrastructure/networking/keycloak" >"$work/keycloak-network-policies.yaml" + +python3 -I - "$work" <<'PY' +import sys, pathlib, yaml + +work = pathlib.Path(sys.argv[1]) +errors = [] + +def load(name): + return [d for d in yaml.safe_load_all((work / name).read_text()) if d] + +def keyset(docs): + return sorted((d["kind"], d["metadata"].get("namespace", ""), d["metadata"]["name"]) for d in docs) + +for name in ("techlog.yaml", "priority-classes.yaml", "keycloak-network-policies.yaml"): + for d in load(name): + if d["kind"] == "Secret": + errors.append(f"{name}: Secret objects must not be rendered") + +techlog = load("techlog.yaml") +expected = sorted([ + ("Namespace", "", "hyeonworks-prod"), + ("Database", "platform-data", "platform-postgres-techlog"), + ("DatabaseRole", "platform-data", "platform-postgres-techlog"), + ("NetworkPolicy", "platform-data", "allow-techlog-backend"), + ("NetworkPolicy", "object-storage", "allow-techlog-backend"), + ("Probe", "observability", "techlog-edge"), + ("Probe", "observability", "techlog-readiness"), + ("PrometheusRule", "observability", "techlog"), + ("ConfigMap", "observability", "grafana-dashboard-techlog"), + ("NetworkPolicy", "observability", "blackbox-allow-techlog-health"), + ("NetworkPolicy", "observability", "observability-allow-techlog-scrape"), +]) +if keyset(techlog) != expected: + errors.append(f"tech-log tenant objects differ: {keyset(techlog)}") +for d in techlog: + kind = d["kind"] + if kind == "Namespace": + labels = d["metadata"].get("labels", {}) + want = { + "pod-security.kubernetes.io/enforce": "baseline", + "pod-security.kubernetes.io/warn": "restricted", + "pod-security.kubernetes.io/audit": "restricted", + "observability.hyeonworks.com/enabled": "true", + } + for k, v in want.items(): + if labels.get(k) != v: + errors.append(f"hyeonworks-prod label {k} must be {v}") + if kind == "DatabaseRole": + spec = d["spec"] + if spec.get("passwordSecret", {}).get("name") != "techlog-db-credentials": + errors.append("techlog DatabaseRole must reference techlog-db-credentials") + for flag in ("superuser", "createdb", "createrole", "replication", "bypassrls"): + if spec.get(flag) is not False: + errors.append(f"techlog DatabaseRole {flag} must be false") + if kind in ("Database", "DatabaseRole"): + policy = d["spec"].get("databaseReclaimPolicy") or d["spec"].get("databaseRoleReclaimPolicy") + if policy != "retain": + errors.append(f"{kind} must keep the retain reclaim policy") + if kind == "ConfigMap": + labels = d["metadata"].get("labels", {}) + if labels.get("grafana_dashboard") != "1": + errors.append("tech-log dashboard ConfigMap needs grafana_dashboard=1") + if d["metadata"]["name"] != "grafana-dashboard-techlog": + errors.append("tech-log dashboard ConfigMap must keep its name (no hash suffix)") + +classes = {d["metadata"]["name"]: d for d in load("priority-classes.yaml")} +if sorted(classes) != ["platform-critical", "platform-low"]: + errors.append(f"PriorityClasses differ: {sorted(classes)}") +else: + crit, low = classes["platform-critical"], classes["platform-low"] + if not (0 < crit["value"] < 1_000_000_000): + errors.append("platform-critical must be above 0 and below the system classes") + if not low["value"] < 0: + errors.append("platform-low must be below the default priority 0") + if low.get("preemptionPolicy") != "Never": + errors.append("platform-low must never preempt") + for d in classes.values(): + if d.get("globalDefault") is not False: + errors.append(f"{d['metadata']['name']} must not be the global default") + +policies = {d["metadata"]["name"]: d for d in load("keycloak-network-policies.yaml")} +deny = policies.get("keycloak-default-deny") +if not deny or deny["spec"].get("podSelector") != {} or sorted(deny["spec"].get("policyTypes", [])) != ["Egress", "Ingress"]: + errors.append("keycloak needs a namespace-wide default deny for ingress and egress") +for d in policies.values(): + if d["metadata"].get("namespace") != "keycloak": + errors.append(f"{d['metadata']['name']} must live in the keycloak namespace") + for rule in d["spec"].get("egress", []) or []: + for peer in rule.get("to", []): + cidr = peer.get("ipBlock", {}).get("cidr") + if cidr and cidr not in ("10.43.0.1/32", "192.168.0.107/32"): + errors.append(f"{d['metadata']['name']}: unexpected egress CIDR {cidr}") + if not rule.get("ports"): + errors.append(f"{d['metadata']['name']}: every egress rule must name its ports") + +if errors: + for e in errors: + print(f"ERROR: {e}", file=sys.stderr) + sys.exit(1) +print(f"tech-log tenant: {len(techlog)} objects; PriorityClasses: {len(classes)}; keycloak NetworkPolicies: {len(policies)}") +PY + +echo "TENANTS AND POLICIES STATIC RENDER PASS" diff --git a/services/priority-classes/README.md b/services/priority-classes/README.md new file mode 100644 index 0000000..26eb2dc --- /dev/null +++ b/services/priority-classes/README.md @@ -0,0 +1,20 @@ +# 플랫폼 PriorityClass + +노드가 한 대라 자원이 모자라면 kubelet이 파드를 내보낸다. 내보낼 순서는 requests를 넘었는지, +그다음 우선순위로 정한다. 2026-11-09 디스크 축출 때 플랫폼 파드는 모두 우선순위 0이라 tech-log +운영 파드가 Grafana·kube-state-metrics와 같은 순위로 나갔다. + +| 이름 | 값 | 선점 | 붙일 곳 | +| --- | --- | --- | --- | +| `platform-critical` | 1000000 | 낮은 것을 선점 | platform-postgres, Gitea, Keycloak, tech-log | +| `platform-low` | -1000 | 선점 안 함 | Grafana, pgAdmin, Loki, Tempo | + +- 아무것도 안 붙인 파드는 기본값 0이다. 시스템 클래스(`system-*`, 20억)는 그대로 가장 높다. +- `globalDefault`는 둘 다 끈다. 새 파드가 모르게 우선순위를 얻지 않게 하려는 것이다. +- 클래스를 만드는 것은 아무 파드도 바꾸지 않는다. **붙이는 것은 파드 재시작을 부르므로** + k3s 재시작 창에 한다(`infrastructure/host/k3s/README.md`). + +```sh +kubectl apply -k services/priority-classes +kubectl get priorityclass platform-critical platform-low +``` diff --git a/services/priority-classes/kustomization.yaml b/services/priority-classes/kustomization.yaml new file mode 100644 index 0000000..8ba8781 --- /dev/null +++ b/services/priority-classes/kustomization.yaml @@ -0,0 +1,5 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - priority-classes.yaml diff --git a/services/priority-classes/priority-classes.yaml b/services/priority-classes/priority-classes.yaml new file mode 100644 index 0000000..38f87ea --- /dev/null +++ b/services/priority-classes/priority-classes.yaml @@ -0,0 +1,25 @@ +apiVersion: scheduling.k8s.io/v1 +kind: PriorityClass +metadata: + name: platform-critical + labels: + app.kubernetes.io/part-of: platform +value: 1000000 +globalDefault: false +preemptionPolicy: PreemptLowerPriority +description: >- + Stateful platform services and production workloads that other services depend on + (platform-postgres, Gitea, Keycloak, tech-log). Evicted after everything else. +--- +apiVersion: scheduling.k8s.io/v1 +kind: PriorityClass +metadata: + name: platform-low + labels: + app.kubernetes.io/part-of: platform +value: -1000 +globalDefault: false +preemptionPolicy: Never +description: >- + Tools whose loss does not stop a user-facing service (Grafana, pgAdmin, Loki, Tempo). + Evicted first and never preempts another pod. diff --git a/services/tenants/techlog/README.md b/services/tenants/techlog/README.md new file mode 100644 index 0000000..3eca649 --- /dev/null +++ b/services/tenants/techlog/README.md @@ -0,0 +1,44 @@ +# tech-log 테넌트 — 플랫폼 쪽 객체 + +tech-log(hyeonworks.com 운영)가 플랫폼을 쓰려면 플랫폼 namespace 안에 허용 객체가 있어야 한다. +이 빌드 루트가 그 객체를 소유한다. 앱 자체(Deployment·Service·Ingress·ServiceMonitor·앱 +namespace 안의 NetworkPolicy)는 tech-log 저장소의 `deploy/`가 소유한다. + +2026-11-11 점검(`docs/reviews/2026-11-11-k3s-maturity-review.md` 4.3절) 전까지 이 객체들은 +`kubectl apply`로만 들어가 있었고 어느 저장소에도 없었다. 처음 내용은 라이브 객체의 +`last-applied-configuration`을 그대로 옮겼다. 라이브와 다른 것은 `hyeonworks-prod`의 PSA 라벨뿐이다. + +| 파일 | 객체 | +| --- | --- | +| `namespace.yaml` | Namespace `hyeonworks-prod`: 관측 대상 라벨, PSA `enforce=baseline`·`warn=restricted`·`audit=restricted` | +| `platform-data.yaml` | `techlog` Database와 DatabaseRole(로그인만, 연결 30), Postgres로 들어오는 ingress | +| `object-storage.yaml` | AIStor 9000으로 들어오는 ingress | +| `observability-probes.yaml` | blackbox Probe 둘(공개 HTTPS, 내부 readiness) | +| `observability-rules.yaml` | PrometheusRule `techlog` 일곱 개 | +| `dashboards/techlog-operations.json` | Grafana 대시보드(`grafana-dashboard-techlog` ConfigMap으로 생성) | +| `observability-network-policies.yaml` | blackbox → readiness, Prometheus → management scrape egress | + +## 여기에 없는 것 + +- `platform-data/platform-postgres`의 `pg_hba` techlog 두 줄: Cluster 객체는 + `services/platform-postgres`가 소유한다. +- Secret: `platform-data/techlog-db-credentials`(`kubernetes.io/basic-auth`, `username`은 정확히 + `techlog`)는 DatabaseRole이 참조한다. 앱 쪽 `hyeonworks-prod/hyeonworks-secrets`는 tech-log가 + 소유한다. 둘 다 Git에 두지 않는다. + +## PSA + +`enforce=baseline`은 2026-11-12에 서버 dry-run(`kubectl label --dry-run=server`)으로 위반 0을 +확인하고 붙였다. `restricted`로 강제하면 지금 파드 셋(redis, backend, frontend)이 모두 걸린다 +(`allowPrivilegeEscalation`, capabilities, `runAsNonRoot`, seccomp). tech-log 쪽 securityContext가 +고쳐진 뒤 같은 dry-run이 경고 없이 통과하면 `enforce`를 `restricted`로 올린다. + +## 렌더·적용 + +```sh +bash scripts/validate/render-tenants-and-policies.sh +kubectl kustomize services/tenants/techlog | kubectl diff -f - +kubectl apply -k services/tenants/techlog +``` + +새로 설치할 때는 이 루트를 tech-log `deploy/`보다 먼저 적용한다(namespace를 여기서 만든다). diff --git a/services/tenants/techlog/dashboards/techlog-operations.json b/services/tenants/techlog/dashboards/techlog-operations.json new file mode 100644 index 0000000..8dd5929 --- /dev/null +++ b/services/tenants/techlog/dashboards/techlog-operations.json @@ -0,0 +1 @@ +{"uid": "techlog-operations", "title": "TechLog · 운영 상태", "tags": ["techlog", "home", "application"], "timezone": "browser", "schemaVersion": 39, "version": 1, "refresh": "30s", "time": {"from": "now-1h", "to": "now"}, "panels": [{"id": 1, "title": "API 메트릭 수집", "type": "stat", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 0, "y": 0, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "up{namespace=\"hyeonworks-prod\",service=\"techlog-management\"}", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "short"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 2, "title": "의존성 readiness", "type": "stat", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 12, "y": 0, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "probe_success{job=\"techlog-readiness\"}", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "short"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 3, "title": "웹·API HTTPS 상태", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 0, "y": 7, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "probe_success{job=\"techlog-edge\"}", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "short"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 4, "title": "인증서 남은 기간", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 12, "y": 7, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "(probe_ssl_earliest_cert_expiry{job=\"techlog-edge\"}-time())/86400", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "d"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 5, "title": "HTTP 요청률", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 0, "y": 14, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "sum by(uri,status)(rate(http_server_requests_seconds_count{namespace=\"hyeonworks-prod\",uri!~\"/actuator.*\"}[$__rate_interval]))", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "reqps"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 6, "title": "HTTP p95·p99 지연", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 12, "y": 14, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "histogram_quantile(0.99,sum by(le)(rate(http_server_requests_seconds_bucket{namespace=\"hyeonworks-prod\",uri!~\"/actuator.*\"}[$__rate_interval])))", "legendFormat": "p99"}, {"refId": "B", "expr": "histogram_quantile(0.95,sum by(le)(rate(http_server_requests_seconds_bucket{namespace=\"hyeonworks-prod\",uri!~\"/actuator.*\"}[$__rate_interval])))", "legendFormat": "p95"}], "fieldConfig": {"defaults": {"unit": "s"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 7, "title": "HTTP 상태별 응답", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 0, "y": 21, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "sum by(status)(rate(http_server_requests_seconds_count{namespace=\"hyeonworks-prod\"}[$__rate_interval]))", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "reqps"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 8, "title": "JVM 메모리", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 12, "y": 21, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "sum by(area)(jvm_memory_used_bytes{namespace=\"hyeonworks-prod\"})", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "bytes"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 9, "title": "JVM GC 시간", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 0, "y": 28, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "sum by(action)(rate(jvm_gc_pause_seconds_sum{namespace=\"hyeonworks-prod\"}[$__rate_interval]))", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "s"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 10, "title": "프로세스 CPU", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 12, "y": 28, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "process_cpu_usage{namespace=\"hyeonworks-prod\"}", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "percentunit"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 11, "title": "DB 연결 풀 사용·대기", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 0, "y": 35, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "hikaricp_connections_active{namespace=\"hyeonworks-prod\"}", "legendFormat": "active"}, {"refId": "B", "expr": "hikaricp_connections_pending{namespace=\"hyeonworks-prod\"}", "legendFormat": "pending"}], "fieldConfig": {"defaults": {"unit": "short"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 12, "title": "DB 연결 획득 지연", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 12, "y": 35, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "rate(hikaricp_connections_acquire_seconds_sum{namespace=\"hyeonworks-prod\"}[$__rate_interval])/clamp_min(rate(hikaricp_connections_acquire_seconds_count{namespace=\"hyeonworks-prod\"}[$__rate_interval]),0.001)", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "s"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 13, "title": "Pod 재시작", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 0, "y": 42, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "kube_pod_container_status_restarts_total{namespace=\"hyeonworks-prod\"}", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "short"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 14, "title": "분산 추적 샘플링 비율", "type": "timeseries", "datasource": {"type": "prometheus", "uid": "prometheus"}, "gridPos": {"x": 12, "y": 42, "w": 12, "h": 7}, "targets": [{"refId": "A", "expr": "tracing_sampling_rate{namespace=\"hyeonworks-prod\"}", "legendFormat": "{{uri}} {{status}} {{pool}} {{id}}"}], "fieldConfig": {"defaults": {"unit": "percentunit"}, "overrides": []}, "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}}}, {"id": 15, "title": "TechLog 구조화 로그 · 요청 ID / 추적 ID", "type": "logs", "datasource": {"type": "loki", "uid": "loki"}, "gridPos": {"x": 0, "y": 49, "w": 24, "h": 12}, "targets": [{"refId": "A", "expr": "{namespace=\"hyeonworks-prod\",app=\"techlog-backend\"}", "queryType": "range"}], "options": {"showTime": true, "showLabels": false, "showCommonLabels": false, "wrapLogMessage": true, "prettifyLogMessage": true, "sortOrder": "Descending", "enableLogDetails": true}}, {"id": 16, "type": "text", "title": "추적 수집 상태", "gridPos": {"x": 0, "y": 61, "w": 24, "h": 4}, "options": {"mode": "markdown", "content": "요청 ID와 추적 ID는 JSON 로그에서 조회할 수 있습니다. **Tempo span 저장은 현재 이미지에서 검증되지 않았습니다.** 메트릭·로그·의존성 health 수집은 동작합니다."}}], "links": []} \ No newline at end of file diff --git a/services/tenants/techlog/kustomization.yaml b/services/tenants/techlog/kustomization.yaml new file mode 100644 index 0000000..6ede4d1 --- /dev/null +++ b/services/tenants/techlog/kustomization.yaml @@ -0,0 +1,23 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +# tech-log(hyeonworks.com) 테넌트의 플랫폼 쪽 객체. 앱 쪽 매니페스트는 tech-log repo가 소유한다. +resources: + - namespace.yaml + - platform-data.yaml + - object-storage.yaml + - observability-probes.yaml + - observability-rules.yaml + - observability-network-policies.yaml + +generatorOptions: + disableNameSuffixHash: true + labels: + app.kubernetes.io/part-of: techlog + grafana_dashboard: "1" + +configMapGenerator: + - name: grafana-dashboard-techlog + namespace: observability + files: + - dashboards/techlog-operations.json diff --git a/services/tenants/techlog/namespace.yaml b/services/tenants/techlog/namespace.yaml new file mode 100644 index 0000000..7374c1b --- /dev/null +++ b/services/tenants/techlog/namespace.yaml @@ -0,0 +1,12 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: hyeonworks-prod + labels: + observability.hyeonworks.com/enabled: 'true' + pod-security.kubernetes.io/enforce: baseline + pod-security.kubernetes.io/enforce-version: latest + pod-security.kubernetes.io/warn: restricted + pod-security.kubernetes.io/warn-version: latest + pod-security.kubernetes.io/audit: restricted + pod-security.kubernetes.io/audit-version: latest diff --git a/services/tenants/techlog/object-storage.yaml b/services/tenants/techlog/object-storage.yaml new file mode 100644 index 0000000..8774f13 --- /dev/null +++ b/services/tenants/techlog/object-storage.yaml @@ -0,0 +1,22 @@ +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-techlog-backend + namespace: object-storage +spec: + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: hyeonworks-prod + podSelector: + matchLabels: + app: techlog-backend + ports: + - port: 9000 + protocol: TCP + podSelector: + matchLabels: + aistor.min.io/objectStore: minio-aistor + policyTypes: + - Ingress diff --git a/services/tenants/techlog/observability-network-policies.yaml b/services/tenants/techlog/observability-network-policies.yaml new file mode 100644 index 0000000..bc4e88f --- /dev/null +++ b/services/tenants/techlog/observability-network-policies.yaml @@ -0,0 +1,45 @@ +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: blackbox-allow-techlog-health + namespace: observability +spec: + egress: + - ports: + - port: 9001 + protocol: TCP + to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: hyeonworks-prod + podSelector: + matchLabels: + app: techlog-backend + podSelector: + matchLabels: + app.kubernetes.io/name: prometheus-blackbox-exporter + policyTypes: + - Egress +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: observability-allow-techlog-scrape + namespace: observability +spec: + egress: + - ports: + - port: 9001 + protocol: TCP + to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: hyeonworks-prod + podSelector: + matchLabels: + app: techlog-backend + podSelector: + matchLabels: + app.kubernetes.io/name: prometheus + policyTypes: + - Egress diff --git a/services/tenants/techlog/observability-probes.yaml b/services/tenants/techlog/observability-probes.yaml new file mode 100644 index 0000000..71e0bcd --- /dev/null +++ b/services/tenants/techlog/observability-probes.yaml @@ -0,0 +1,46 @@ +apiVersion: monitoring.coreos.com/v1 +kind: Probe +metadata: + labels: + observability.hyeonworks.com/instance: home + name: techlog-edge + namespace: observability +spec: + interval: 30s + jobName: techlog-edge + module: http_2xx + prober: + path: /probe + scheme: http + url: blackbox-exporter.observability.svc.cluster.local:9115 + scrapeTimeout: 15s + targets: + staticConfig: + labels: + application: techlog + static: + - https://hyeonworks.com/ + - https://hyeonworks.com/api/healthcheck +--- +apiVersion: monitoring.coreos.com/v1 +kind: Probe +metadata: + labels: + observability.hyeonworks.com/instance: home + name: techlog-readiness + namespace: observability +spec: + interval: 30s + jobName: techlog-readiness + module: http_2xx + prober: + path: /probe + scheme: http + url: blackbox-exporter.observability.svc.cluster.local:9115 + scrapeTimeout: 15s + targets: + staticConfig: + labels: + application: techlog + static: + - http://techlog-management.hyeonworks-prod.svc.cluster.local:9001/actuator/health/readiness diff --git a/services/tenants/techlog/observability-rules.yaml b/services/tenants/techlog/observability-rules.yaml new file mode 100644 index 0000000..89eec74 --- /dev/null +++ b/services/tenants/techlog/observability-rules.yaml @@ -0,0 +1,67 @@ +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + labels: + observability.hyeonworks.com/instance: home + name: techlog + namespace: observability +spec: + groups: + - name: techlog + rules: + - alert: TechLogMetricsDown + annotations: + summary: TechLog metrics target is unreachable + expr: up{namespace="hyeonworks-prod",service="techlog-management"} == 0 + for: 5m + labels: + application: techlog + severity: warning + - alert: TechLogReadinessDown + annotations: + summary: TechLog dependency readiness is DOWN + expr: probe_success{job="techlog-readiness"} == 0 + for: 3m + labels: + application: techlog + severity: critical + - alert: TechLogHTTPSDown + annotations: + summary: TechLog HTTPS or API health is failing + expr: probe_success{job="techlog-edge"} == 0 + for: 3m + labels: + application: techlog + severity: critical + - alert: TechLogHTTP5xx + annotations: + summary: TechLog HTTP 5xx exceeds 5% + expr: (sum(rate(http_server_requests_seconds_count{namespace="hyeonworks-prod",status=~"5.."}[5m])) / clamp_min(sum(rate(http_server_requests_seconds_count{namespace="hyeonworks-prod"}[5m])),0.001)) > 0.05 and sum(increase(http_server_requests_seconds_count{namespace="hyeonworks-prod"}[5m])) > 20 + for: 5m + labels: + application: techlog + severity: warning + - alert: TechLogLatencyHigh + annotations: + summary: TechLog p99 latency exceeds 2 seconds + expr: histogram_quantile(0.99,sum by(le)(rate(http_server_requests_seconds_bucket{namespace="hyeonworks-prod",uri!~"/actuator.*"}[5m]))) > 2 + for: 10m + labels: + application: techlog + severity: warning + - alert: TechLogHeapPressure + annotations: + summary: TechLog heap usage exceeds 85% + expr: sum(jvm_memory_used_bytes{namespace="hyeonworks-prod",area="heap"}) / sum(jvm_memory_max_bytes{namespace="hyeonworks-prod",area="heap"}) > 0.85 + for: 10m + labels: + application: techlog + severity: warning + - alert: TechLogDatabasePoolWaiting + annotations: + summary: TechLog database connections are queued + expr: hikaricp_connections_pending{namespace="hyeonworks-prod"} > 0 + for: 5m + labels: + application: techlog + severity: warning diff --git a/services/tenants/techlog/platform-data.yaml b/services/tenants/techlog/platform-data.yaml new file mode 100644 index 0000000..4b1bc54 --- /dev/null +++ b/services/tenants/techlog/platform-data.yaml @@ -0,0 +1,55 @@ +apiVersion: postgresql.cnpg.io/v1 +kind: Database +metadata: + name: platform-postgres-techlog + namespace: platform-data +spec: + cluster: + name: platform-postgres + databaseReclaimPolicy: retain + ensure: present + name: techlog + owner: techlog +--- +apiVersion: postgresql.cnpg.io/v1 +kind: DatabaseRole +metadata: + name: platform-postgres-techlog + namespace: platform-data +spec: + bypassrls: false + cluster: + name: platform-postgres + connectionLimit: 30 + createdb: false + createrole: false + databaseRoleReclaimPolicy: retain + login: true + name: techlog + passwordSecret: + name: techlog-db-credentials + replication: false + superuser: false +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-techlog-backend + namespace: platform-data +spec: + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: hyeonworks-prod + podSelector: + matchLabels: + app: techlog-backend + ports: + - port: 5432 + protocol: TCP + podSelector: + matchLabels: + cnpg.io/cluster: platform-postgres + policyTypes: + - Ingress