feat(tenants): declare tech-log platform objects and keycloak egress #4

Merged
donghyeon.kang merged 4 commits from donghyeon.kang/w6-tenant-and-policies AGit into main 2026-11-11 15:43:43 +00:00
19 changed files with 878 additions and 0 deletions
@@ -0,0 +1,38 @@
# keycloak namespace NetworkPolicy
2026-11-11 점검 전 `keycloak` namespace에는 Keycloak Operator가 만든 ingress 정책 하나뿐이었다.
Keycloak 파드와 Operator 파드의 egress는 막혀 있지 않았다. 다른 플랫폼 namespace처럼 기본 차단에
필요한 것만 연다.
| 정책 | 대상 | 허용 |
| --- | --- | --- |
| `keycloak-default-deny` | 모든 파드 | ingress·egress 기본 차단 |
| `keycloak-allow-dns` | 모든 파드 | kube-dns 53 |
| `keycloak-allow-platform-postgres` | 모든 파드(서버와 realm import Job) | `platform-data` Postgres 5432 |
| `keycloak-allow-cluster-peers` | Keycloak 파드 | 같은 Keycloak 파드 JGroups 7800·57800 |
| `keycloak-operator-allow-egress` | Operator | API 서버(`10.43.0.1:443`, `192.168.0.107:6443`), Keycloak 8080·9000 |
- Keycloak 파드의 **ingress**는 계속 Keycloak CR `spec.networkPolicy`로 Operator가 만드는
`keycloak-network-policy`가 소유한다(Traefik 8080, 관리 9000, JGroups).
- 바깥(인터넷)으로 나가는 길은 없다. 2026-11-12에 repo와 Keycloak CR에 SMTP·외부 IdP 설정이
없고, Keycloak 파드의 열린 TCP 연결이 Postgres뿐인 것을 확인했다. SMTP나 외부 IdP를 붙이면
여기에 egress를 더해야 한다(안 더하면 메일·외부 로그인만 조용히 실패한다).
- kubelet probe는 노드에서 오므로 막히지 않는다.
## 적용과 확인
```sh
bash scripts/validate/render-tenants-and-policies.sh
kubectl kustomize infrastructure/networking/keycloak | kubectl diff -f -
kubectl apply -k infrastructure/networking/keycloak
```
적용 뒤 확인(모두 통과해야 한다):
```sh
kubectl -n keycloak get pods
curl -fsS -o /dev/null -w '%{http_code}\n' https://id.learn.hyeonworks.com/realms/hyeonworks/.well-known/openid-configuration
kubectl -n keycloak logs deploy/keycloak-operator --since=5m | grep -i -E 'error|exception|timeout' || echo no-operator-errors
```
되돌리기: `kubectl delete -k infrastructure/networking/keycloak`(Operator 정책은 남는다).
@@ -0,0 +1,12 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: keycloak
labels:
- pairs:
app.kubernetes.io/part-of: platform
includeSelectors: false
resources:
- network-policies.yaml
@@ -0,0 +1,117 @@
# keycloak namespace: 기본 차단 + 필요한 것만 허용.
# Keycloak 파드의 ingress는 Keycloak CR(spec.networkPolicy)로 Operator가 만드는
# keycloak-network-policy가 계속 소유한다. 여기서는 기본 차단과 egress만 정한다.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: keycloak-default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: keycloak-allow-dns
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP
---
# Keycloak 서버와 realm import Job이 공용 PostgreSQL에 붙는다.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: keycloak-allow-platform-postgres
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: platform-data
podSelector:
matchLabels:
cnpg.io/cluster: platform-postgres
ports:
- port: 5432
protocol: TCP
---
# 같은 Keycloak 인스턴스끼리의 JGroups(7800, 57800). Operator ingress 정책과 짝이다.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: keycloak-allow-cluster-peers
spec:
podSelector:
matchLabels:
app: keycloak
app.kubernetes.io/instance: keycloak
app.kubernetes.io/managed-by: keycloak-operator
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
app: keycloak
app.kubernetes.io/instance: keycloak
app.kubernetes.io/managed-by: keycloak-operator
ports:
- port: 7800
protocol: TCP
- port: 57800
protocol: TCP
---
# Operator: Kubernetes API(ClusterIP와 실제 API 서버 주소)와 Keycloak 관리 포트.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: keycloak-operator-allow-egress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: keycloak-operator
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 10.43.0.1/32
ports:
- port: 443
protocol: TCP
- to:
- ipBlock:
cidr: 192.168.0.107/32
ports:
- port: 6443
protocol: TCP
- to:
- podSelector:
matchLabels:
app: keycloak
app.kubernetes.io/instance: keycloak
app.kubernetes.io/managed-by: keycloak-operator
ports:
- port: 8080
protocol: TCP
- port: 9000
protocol: TCP
+37
View File
@@ -559,3 +559,40 @@ parser와 세 mode의 fixture 회귀는 live cluster에 연결하지 않고 다
```sh
bash scripts/validate/test-k3s-secret-encryption-status.sh
```
## 테넌트·우선순위·keycloak 정책 렌더 검사
`render-tenants-and-policies.sh`는 클러스터에 닿지 않고 세 빌드 루트를 렌더해 불변식을
확인한다.
- `services/tenants/techlog`: tech-log 테넌트의 플랫폼 쪽 객체 11개(목록이 정확히 같아야
한다), `hyeonworks-prod` PSA 라벨, techlog DatabaseRole 권한 플래그, 대시보드 이름
- `services/priority-classes`: `platform-critical`(0보다 크고 시스템 클래스보다 작다),
`platform-low`(0보다 작고 선점하지 않는다), 둘 다 global default가 아니다
- `infrastructure/networking/keycloak`: namespace 전체 기본 차단, egress 규칙마다 포트,
허용 CIDR은 API 서버 둘뿐
```sh
bash scripts/validate/render-tenants-and-policies.sh
```
## 라이브 drift 검사
`live-drift-check.sh`는 이 저장소의 라이브 빌드 루트를 모두 렌더하고(Phase 1·2, 관리 도구,
관측 core, Traefik trust overlay, 위 세 루트) Secret을 뺀 뒤 `kubectl diff`(서버 dry-run,
저장하지 않음)로 클러스터와 비교한다. 렌더에는 위 render 스크립트의 `--verified-output-dir`
출력을 읽기만 하는 용도로 쓴다(그 디렉터리는 스크립트가 끝날 때 지운다).
```sh
PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm \
bash scripts/validate/live-drift-check.sh \
--accepted scripts/validate/live-drift-accepted.txt
```
- 종료 코드: 0 차이 없음, 1 차이 있음, 2 검사 자체 실패.
- `DRIFT <Kind> <ns>/<name> (+추가 -삭제) [입력 파일]`로 객체마다 한 줄을 쓴다.
`generation` 줄만 다른 것은 세지 않는다.
- `--accepted`: 알고 있는 차이를 기한과 이유를 달아 적는다. 기한이 지나면 다시 drift로 센다.
- `--textfile FILE`: `platform_live_drift_objects`, `platform_live_drift_check_success`,
`platform_live_drift_last_run_timestamp_seconds`를 Prometheus textfile 형식으로 쓴다.
- Secret 값의 차이는 보지 않는다.
+10
View File
@@ -0,0 +1,10 @@
# 알고 있는 라이브 drift. 형식: <Kind> <namespace 또는 -> <name> <기한 YYYY-MM-DD> <이유>
# 기한이 지나면 live-drift-check.sh가 다시 drift로 센다. 해소하면 줄을 지운다.
Cluster platform-data platform-postgres 2026-11-30 pg_hba의 techlog 두 줄이 repo에 없다. w1이 CNPG 백업과 함께 넣는다(1c4ef9a).
DatabaseRole platform-data platform-postgres-gitea 2026-11-30 렌더가 false 기본값 필드를 명시할 뿐 값은 같다. 다음 Phase 1 적용 때 사라진다.
DatabaseRole platform-data platform-postgres-keycloak 2026-11-30 렌더가 false 기본값 필드를 명시할 뿐 값은 같다. 다음 Phase 2 적용 때 사라진다.
Deployment keycloak keycloak-operator 2026-11-30 원격 ref 26.7.0 태그 내용이 바뀌어 build-timestamp 라벨만 다르다. 재시작 창에 커밋 SHA 고정과 함께 적용한다.
Service keycloak keycloak-operator 2026-11-30 위와 같은 build-timestamp 라벨.
ServiceAccount keycloak keycloak-operator 2026-11-30 위와 같은 build-timestamp 라벨.
Deployment observability observability-core-kube-pr-operator 2026-11-30 이미지 문자열이 tag@digest와 digest로 다르고 digest는 같다. 적용하면 재시작되므로 재시작 창에.
Deployment observability observability-core-kube-state-metrics 2026-11-30 위와 같은 이미지 문자열 차이.
+173
View File
@@ -0,0 +1,173 @@
#!/usr/bin/env bash
# Render every live build root of this repository and compare it with the
# cluster using `kubectl diff` (server-side dry-run: nothing is persisted).
#
# Usage:
# PLATFORM_HELM_BIN=/abs/helm bash scripts/validate/live-drift-check.sh \
# [--accepted FILE] [--textfile FILE]
#
# --accepted FILE known differences, one per line:
# `<Kind> <namespace or -> <name> <until YYYY-MM-DD> <reason...>`
# Expired entries count as drift again.
# --textfile FILE also write Prometheus textfile metrics (atomic rename).
#
# Exit status: 0 no drift, 1 drift found, 2 the check itself failed.
# Secrets are removed from the renders before diffing; Secret drift is out of scope.
set -Eeuo pipefail
readonly REPOSITORY_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
accepted_file=""
textfile=""
while (( $# > 0 )); do
case "$1" in
--accepted) accepted_file="${2:?--accepted needs a file}"; shift 2 ;;
--textfile) textfile="${2:?--textfile needs a file}"; shift 2 ;;
-h|--help) sed -n '2,16p' "${BASH_SOURCE[0]}"; exit 0 ;;
*) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
esac
done
declare -a made_dirs=()
cleanup() {
local d
for d in "${made_dirs[@]}"; do
case "$d" in /tmp/platform-*) rm -rf -- "$d" ;; esac
done
}
trap cleanup EXIT
write_textfile() {
local drift="$1" ok="$2"
[[ -n "$textfile" ]] || return 0
local tmp="${textfile}.tmp.$$"
{
echo '# HELP platform_live_drift_objects Objects whose live state differs from the repository render.'
echo '# TYPE platform_live_drift_objects gauge'
echo "platform_live_drift_objects ${drift}"
echo '# HELP platform_live_drift_check_success 1 when the drift check itself completed.'
echo '# TYPE platform_live_drift_check_success gauge'
echo "platform_live_drift_check_success ${ok}"
echo '# HELP platform_live_drift_last_run_timestamp_seconds Unix time the drift check finished.'
echo '# TYPE platform_live_drift_last_run_timestamp_seconds gauge'
echo "platform_live_drift_last_run_timestamp_seconds $(date +%s)"
} >"$tmp"
mv -f -- "$tmp" "$textfile"
}
die() {
printf 'DRIFT CHECK ERROR: %s\n' "$*" >&2
write_textfile 0 0
exit 2
}
work="$(mktemp -d /tmp/platform-drift.XXXXXX)"; made_dirs+=("$work")
mkdir -p "$work/in" "$work/out"
render_root() {
local script="$1" prefix="$2" dir
dir="$(mktemp -d "/tmp/${prefix}.XXXXXX")"; made_dirs+=("$dir")
bash "$REPOSITORY_ROOT/scripts/validate/${script}" --verified-output-dir "$dir" \
>"$work/${script}.log" 2>&1 || { tail -5 "$work/${script}.log" >&2; die "${script} failed"; }
printf '%s\n' "$dir"
}
phase1="$(render_root render-phase1.sh platform-phase1-apply)"
phase2="$(render_root render-phase2.sh platform-phase2-apply)"
admin="$(render_root render-admin-services.sh platform-admin-apply)"
observability="$(render_root render-observability-core.sh platform-observability-core-apply)"
# The live Gitea is the OIDC profile; gitea.yaml is the fresh-install baseline only.
declare -a inputs=(
"$phase1/namespaces.yaml" "$phase1/ssd-local-pv.yaml" "$phase1/cnpg-operator.yaml"
"$phase1/platform-postgres.yaml" "$phase1/gitea-oidc.yaml"
"$observability/aggregate.yaml"
)
for f in "$phase2"/*.yaml "$admin"/*.yaml; do inputs+=("$f"); done
# Live Traefik is the trust overlay (infrastructure/networking/traefik/README.md).
kubectl kustomize "$REPOSITORY_ROOT/infrastructure/networking/traefik/overlays/trust" >"$work/traefik-trust.yaml" || die "traefik render failed"
kubectl kustomize "$REPOSITORY_ROOT/services/tenants/techlog" >"$work/tenant-techlog.yaml" || die "tenant render failed"
kubectl kustomize "$REPOSITORY_ROOT/services/priority-classes" >"$work/priority-classes.yaml" || die "priority class render failed"
kubectl kustomize "$REPOSITORY_ROOT/infrastructure/networking/keycloak" >"$work/keycloak-network-policies.yaml" || die "keycloak policy render failed"
inputs+=("$work/traefik-trust.yaml" "$work/tenant-techlog.yaml" "$work/priority-classes.yaml" "$work/keycloak-network-policies.yaml")
n=0
for f in "${inputs[@]}"; do
n=$((n + 1))
python3 -I - "$f" "$work/in/$(printf '%02d' "$n")-$(basename "$f")" <<'PY' || die "could not strip Secrets from $f"
import sys, yaml
yaml.SafeLoader.add_constructor("tag:yaml.org,2002:value", lambda l, n: l.construct_scalar(n))
docs = [d for d in yaml.safe_load_all(open(sys.argv[1])) if isinstance(d, dict) and d.get("kind") not in (None, "Secret")]
yaml.safe_dump_all(docs, open(sys.argv[2], "w"), sort_keys=False)
PY
done
for f in "$work"/in/*.yaml; do
set +e
KUBECTL_EXTERNAL_DIFF="diff -u -N" kubectl diff -f "$f" >"$work/out/$(basename "$f").diff" 2>"$work/out/$(basename "$f").err"
rc=$?
set -e
if (( rc > 1 )); then
sed -n '1,5p' "$work/out/$(basename "$f").err" >&2
die "kubectl diff failed for $(basename "$f")"
fi
done
set +e
python3 -I - "$work/out" "${accepted_file:-}" "$work/drift-count" <<'PY'
import sys, pathlib, re, datetime
out, accepted_path, count_path = pathlib.Path(sys.argv[1]), sys.argv[2], pathlib.Path(sys.argv[3])
today = datetime.date.today()
accepted, expired = {}, set()
if accepted_path:
for line in pathlib.Path(accepted_path).read_text().splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
kind, ns, name, until, *reason = line.split()
key = (kind, "" if ns == "-" else ns, name)
if datetime.date.fromisoformat(until) < today:
expired.add(key)
else:
accepted[key] = " ".join(reason)
version = re.compile(r"^v\d+((alpha|beta)\d+)?$")
objects = {}
for diff in sorted(out.glob("*.diff")):
current = None
for line in diff.read_text().splitlines():
if line.startswith("diff -u -N "):
parts = line.split()[3].rsplit("/", 1)[-1].split(".")
i = next(i for i, p in enumerate(parts) if version.match(p))
current = (parts[i + 1], parts[i + 2], ".".join(parts[i + 3:]))
objects.setdefault(current, [0, 0, diff.name])
elif current and line[:1] in "+-" and not line.startswith(("+++", "---")):
if re.match(r"^[+-]\s+generation:", line):
continue
objects[current][0 if line[0] == "+" else 1] += 1
drift = accepted_seen = 0
for key, (plus, minus, src) in sorted(objects.items()):
if plus == 0 and minus == 0:
continue
kind, ns, name = key
label = f"{kind} {ns or '-'}/{name} (+{plus} -{minus}) [{src}]"
if key in accepted:
accepted_seen += 1
print(f"ACCEPTED {label}: {accepted[key]}")
continue
drift += 1
print(f"DRIFT {label}" + (" (accepted entry expired)" if key in expired else ""))
print(f"SUMMARY drift={drift} accepted={accepted_seen}")
count_path.write_text(str(drift))
PY
rc=$?
set -e
(( rc == 0 )) || die "could not summarize the diffs"
drift_count="$(cat "$work/drift-count")"
write_textfile "$drift_count" 1
if (( drift_count == 0 )); then
echo "LIVE DRIFT CHECK PASS"
exit 0
fi
echo "LIVE DRIFT CHECK FOUND DRIFT"
exit 1
@@ -0,0 +1,126 @@
#!/usr/bin/env bash
# Render the tech-log tenant root, the platform PriorityClasses and the keycloak
# namespace NetworkPolicies, and check their invariants. Static: no cluster access.
set -Eeuo pipefail
readonly EXPECTED_KUSTOMIZE_VERSION="v5.8.1"
readonly REPOSITORY_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
fail() {
printf 'ERROR: %s\n' "$*" >&2
exit 1
}
kustomize_version="$(kubectl version --client --output=yaml | sed -n 's/^kustomizeVersion: //p')"
[[ "$kustomize_version" == "$EXPECTED_KUSTOMIZE_VERSION" ]] || \
fail "expected Kustomize ${EXPECTED_KUSTOMIZE_VERSION}, found ${kustomize_version:-unknown}"
work="$(mktemp -d "${TMPDIR:-/tmp}/platform-tenants-render.XXXXXX")"
trap 'rm -rf -- "$work"' EXIT
kubectl kustomize "$REPOSITORY_ROOT/services/tenants/techlog" >"$work/techlog.yaml"
kubectl kustomize "$REPOSITORY_ROOT/services/priority-classes" >"$work/priority-classes.yaml"
kubectl kustomize "$REPOSITORY_ROOT/infrastructure/networking/keycloak" >"$work/keycloak-network-policies.yaml"
python3 -I - "$work" <<'PY'
import sys, pathlib, yaml
work = pathlib.Path(sys.argv[1])
errors = []
def load(name):
return [d for d in yaml.safe_load_all((work / name).read_text()) if d]
def keyset(docs):
return sorted((d["kind"], d["metadata"].get("namespace", ""), d["metadata"]["name"]) for d in docs)
for name in ("techlog.yaml", "priority-classes.yaml", "keycloak-network-policies.yaml"):
for d in load(name):
if d["kind"] == "Secret":
errors.append(f"{name}: Secret objects must not be rendered")
techlog = load("techlog.yaml")
expected = sorted([
("Namespace", "", "hyeonworks-prod"),
("Database", "platform-data", "platform-postgres-techlog"),
("DatabaseRole", "platform-data", "platform-postgres-techlog"),
("NetworkPolicy", "platform-data", "allow-techlog-backend"),
("NetworkPolicy", "object-storage", "allow-techlog-backend"),
("Probe", "observability", "techlog-edge"),
("Probe", "observability", "techlog-readiness"),
("PrometheusRule", "observability", "techlog"),
("ConfigMap", "observability", "grafana-dashboard-techlog"),
("NetworkPolicy", "observability", "blackbox-allow-techlog-health"),
("NetworkPolicy", "observability", "observability-allow-techlog-scrape"),
])
if keyset(techlog) != expected:
errors.append(f"tech-log tenant objects differ: {keyset(techlog)}")
for d in techlog:
kind = d["kind"]
if kind == "Namespace":
labels = d["metadata"].get("labels", {})
want = {
"pod-security.kubernetes.io/enforce": "baseline",
"pod-security.kubernetes.io/warn": "restricted",
"pod-security.kubernetes.io/audit": "restricted",
"observability.hyeonworks.com/enabled": "true",
}
for k, v in want.items():
if labels.get(k) != v:
errors.append(f"hyeonworks-prod label {k} must be {v}")
if kind == "DatabaseRole":
spec = d["spec"]
if spec.get("passwordSecret", {}).get("name") != "techlog-db-credentials":
errors.append("techlog DatabaseRole must reference techlog-db-credentials")
for flag in ("superuser", "createdb", "createrole", "replication", "bypassrls"):
if spec.get(flag) is not False:
errors.append(f"techlog DatabaseRole {flag} must be false")
if kind in ("Database", "DatabaseRole"):
policy = d["spec"].get("databaseReclaimPolicy") or d["spec"].get("databaseRoleReclaimPolicy")
if policy != "retain":
errors.append(f"{kind} must keep the retain reclaim policy")
if kind == "ConfigMap":
labels = d["metadata"].get("labels", {})
if labels.get("grafana_dashboard") != "1":
errors.append("tech-log dashboard ConfigMap needs grafana_dashboard=1")
if d["metadata"]["name"] != "grafana-dashboard-techlog":
errors.append("tech-log dashboard ConfigMap must keep its name (no hash suffix)")
classes = {d["metadata"]["name"]: d for d in load("priority-classes.yaml")}
if sorted(classes) != ["platform-critical", "platform-low"]:
errors.append(f"PriorityClasses differ: {sorted(classes)}")
else:
crit, low = classes["platform-critical"], classes["platform-low"]
if not (0 < crit["value"] < 1_000_000_000):
errors.append("platform-critical must be above 0 and below the system classes")
if not low["value"] < 0:
errors.append("platform-low must be below the default priority 0")
if low.get("preemptionPolicy") != "Never":
errors.append("platform-low must never preempt")
for d in classes.values():
if d.get("globalDefault") is not False:
errors.append(f"{d['metadata']['name']} must not be the global default")
policies = {d["metadata"]["name"]: d for d in load("keycloak-network-policies.yaml")}
deny = policies.get("keycloak-default-deny")
if not deny or deny["spec"].get("podSelector") != {} or sorted(deny["spec"].get("policyTypes", [])) != ["Egress", "Ingress"]:
errors.append("keycloak needs a namespace-wide default deny for ingress and egress")
for d in policies.values():
if d["metadata"].get("namespace") != "keycloak":
errors.append(f"{d['metadata']['name']} must live in the keycloak namespace")
for rule in d["spec"].get("egress", []) or []:
for peer in rule.get("to", []):
cidr = peer.get("ipBlock", {}).get("cidr")
if cidr and cidr not in ("10.43.0.1/32", "192.168.0.107/32"):
errors.append(f"{d['metadata']['name']}: unexpected egress CIDR {cidr}")
if not rule.get("ports"):
errors.append(f"{d['metadata']['name']}: every egress rule must name its ports")
if errors:
for e in errors:
print(f"ERROR: {e}", file=sys.stderr)
sys.exit(1)
print(f"tech-log tenant: {len(techlog)} objects; PriorityClasses: {len(classes)}; keycloak NetworkPolicies: {len(policies)}")
PY
echo "TENANTS AND POLICIES STATIC RENDER PASS"
+20
View File
@@ -0,0 +1,20 @@
# 플랫폼 PriorityClass
노드가 한 대라 자원이 모자라면 kubelet이 파드를 내보낸다. 내보낼 순서는 requests를 넘었는지,
그다음 우선순위로 정한다. 2026-11-09 디스크 축출 때 플랫폼 파드는 모두 우선순위 0이라 tech-log
운영 파드가 Grafana·kube-state-metrics와 같은 순위로 나갔다.
| 이름 | 값 | 선점 | 붙일 곳 |
| --- | --- | --- | --- |
| `platform-critical` | 1000000 | 낮은 것을 선점 | platform-postgres, Gitea, Keycloak, tech-log |
| `platform-low` | -1000 | 선점 안 함 | Grafana, pgAdmin, Loki, Tempo |
- 아무것도 안 붙인 파드는 기본값 0이다. 시스템 클래스(`system-*`, 20억)는 그대로 가장 높다.
- `globalDefault`는 둘 다 끈다. 새 파드가 모르게 우선순위를 얻지 않게 하려는 것이다.
- 클래스를 만드는 것은 아무 파드도 바꾸지 않는다. **붙이는 것은 파드 재시작을 부르므로**
k3s 재시작 창에 한다(`infrastructure/host/k3s/README.md`).
```sh
kubectl apply -k services/priority-classes
kubectl get priorityclass platform-critical platform-low
```
@@ -0,0 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- priority-classes.yaml
@@ -0,0 +1,25 @@
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: platform-critical
labels:
app.kubernetes.io/part-of: platform
value: 1000000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: >-
Stateful platform services and production workloads that other services depend on
(platform-postgres, Gitea, Keycloak, tech-log). Evicted after everything else.
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: platform-low
labels:
app.kubernetes.io/part-of: platform
value: -1000
globalDefault: false
preemptionPolicy: Never
description: >-
Tools whose loss does not stop a user-facing service (Grafana, pgAdmin, Loki, Tempo).
Evicted first and never preempts another pod.
+44
View File
@@ -0,0 +1,44 @@
# tech-log 테넌트 — 플랫폼 쪽 객체
tech-log(hyeonworks.com 운영)가 플랫폼을 쓰려면 플랫폼 namespace 안에 허용 객체가 있어야 한다.
이 빌드 루트가 그 객체를 소유한다. 앱 자체(Deployment·Service·Ingress·ServiceMonitor·앱
namespace 안의 NetworkPolicy)는 tech-log 저장소의 `deploy/`가 소유한다.
2026-11-11 점검(`docs/reviews/2026-11-11-k3s-maturity-review.md` 4.3절) 전까지 이 객체들은
`kubectl apply`로만 들어가 있었고 어느 저장소에도 없었다. 처음 내용은 라이브 객체의
`last-applied-configuration`을 그대로 옮겼다. 라이브와 다른 것은 `hyeonworks-prod`의 PSA 라벨뿐이다.
| 파일 | 객체 |
| --- | --- |
| `namespace.yaml` | Namespace `hyeonworks-prod`: 관측 대상 라벨, PSA `enforce=baseline`·`warn=restricted`·`audit=restricted` |
| `platform-data.yaml` | `techlog` Database와 DatabaseRole(로그인만, 연결 30), Postgres로 들어오는 ingress |
| `object-storage.yaml` | AIStor 9000으로 들어오는 ingress |
| `observability-probes.yaml` | blackbox Probe 둘(공개 HTTPS, 내부 readiness) |
| `observability-rules.yaml` | PrometheusRule `techlog` 일곱 개 |
| `dashboards/techlog-operations.json` | Grafana 대시보드(`grafana-dashboard-techlog` ConfigMap으로 생성) |
| `observability-network-policies.yaml` | blackbox → readiness, Prometheus → management scrape egress |
## 여기에 없는 것
- `platform-data/platform-postgres`의 `pg_hba` techlog 두 줄: Cluster 객체는
`services/platform-postgres`가 소유한다.
- Secret: `platform-data/techlog-db-credentials`(`kubernetes.io/basic-auth`, `username`은 정확히
`techlog`)는 DatabaseRole이 참조한다. 앱 쪽 `hyeonworks-prod/hyeonworks-secrets`는 tech-log가
소유한다. 둘 다 Git에 두지 않는다.
## PSA
`enforce=baseline`은 2026-11-12에 서버 dry-run(`kubectl label --dry-run=server`)으로 위반 0을
확인하고 붙였다. `restricted`로 강제하면 지금 파드 셋(redis, backend, frontend)이 모두 걸린다
(`allowPrivilegeEscalation`, capabilities, `runAsNonRoot`, seccomp). tech-log 쪽 securityContext가
고쳐진 뒤 같은 dry-run이 경고 없이 통과하면 `enforce`를 `restricted`로 올린다.
## 렌더·적용
```sh
bash scripts/validate/render-tenants-and-policies.sh
kubectl kustomize services/tenants/techlog | kubectl diff -f -
kubectl apply -k services/tenants/techlog
```
새로 설치할 때는 이 루트를 tech-log `deploy/`보다 먼저 적용한다(namespace를 여기서 만든다).
File diff suppressed because one or more lines are too long
@@ -0,0 +1,23 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# tech-log(hyeonworks.com) 테넌트의 플랫폼 쪽 객체. 앱 쪽 매니페스트는 tech-log repo가 소유한다.
resources:
- namespace.yaml
- platform-data.yaml
- object-storage.yaml
- observability-probes.yaml
- observability-rules.yaml
- observability-network-policies.yaml
generatorOptions:
disableNameSuffixHash: true
labels:
app.kubernetes.io/part-of: techlog
grafana_dashboard: "1"
configMapGenerator:
- name: grafana-dashboard-techlog
namespace: observability
files:
- dashboards/techlog-operations.json
+12
View File
@@ -0,0 +1,12 @@
apiVersion: v1
kind: Namespace
metadata:
name: hyeonworks-prod
labels:
observability.hyeonworks.com/enabled: 'true'
pod-security.kubernetes.io/enforce: baseline
pod-security.kubernetes.io/enforce-version: latest
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: latest
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/audit-version: latest
@@ -0,0 +1,22 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-techlog-backend
namespace: object-storage
spec:
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: hyeonworks-prod
podSelector:
matchLabels:
app: techlog-backend
ports:
- port: 9000
protocol: TCP
podSelector:
matchLabels:
aistor.min.io/objectStore: minio-aistor
policyTypes:
- Ingress
@@ -0,0 +1,45 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: blackbox-allow-techlog-health
namespace: observability
spec:
egress:
- ports:
- port: 9001
protocol: TCP
to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: hyeonworks-prod
podSelector:
matchLabels:
app: techlog-backend
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus-blackbox-exporter
policyTypes:
- Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-techlog-scrape
namespace: observability
spec:
egress:
- ports:
- port: 9001
protocol: TCP
to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: hyeonworks-prod
podSelector:
matchLabels:
app: techlog-backend
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
policyTypes:
- Egress
@@ -0,0 +1,46 @@
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
labels:
observability.hyeonworks.com/instance: home
name: techlog-edge
namespace: observability
spec:
interval: 30s
jobName: techlog-edge
module: http_2xx
prober:
path: /probe
scheme: http
url: blackbox-exporter.observability.svc.cluster.local:9115
scrapeTimeout: 15s
targets:
staticConfig:
labels:
application: techlog
static:
- https://hyeonworks.com/
- https://hyeonworks.com/api/healthcheck
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
labels:
observability.hyeonworks.com/instance: home
name: techlog-readiness
namespace: observability
spec:
interval: 30s
jobName: techlog-readiness
module: http_2xx
prober:
path: /probe
scheme: http
url: blackbox-exporter.observability.svc.cluster.local:9115
scrapeTimeout: 15s
targets:
staticConfig:
labels:
application: techlog
static:
- http://techlog-management.hyeonworks-prod.svc.cluster.local:9001/actuator/health/readiness
@@ -0,0 +1,67 @@
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
labels:
observability.hyeonworks.com/instance: home
name: techlog
namespace: observability
spec:
groups:
- name: techlog
rules:
- alert: TechLogMetricsDown
annotations:
summary: TechLog metrics target is unreachable
expr: up{namespace="hyeonworks-prod",service="techlog-management"} == 0
for: 5m
labels:
application: techlog
severity: warning
- alert: TechLogReadinessDown
annotations:
summary: TechLog dependency readiness is DOWN
expr: probe_success{job="techlog-readiness"} == 0
for: 3m
labels:
application: techlog
severity: critical
- alert: TechLogHTTPSDown
annotations:
summary: TechLog HTTPS or API health is failing
expr: probe_success{job="techlog-edge"} == 0
for: 3m
labels:
application: techlog
severity: critical
- alert: TechLogHTTP5xx
annotations:
summary: TechLog HTTP 5xx exceeds 5%
expr: (sum(rate(http_server_requests_seconds_count{namespace="hyeonworks-prod",status=~"5.."}[5m])) / clamp_min(sum(rate(http_server_requests_seconds_count{namespace="hyeonworks-prod"}[5m])),0.001)) > 0.05 and sum(increase(http_server_requests_seconds_count{namespace="hyeonworks-prod"}[5m])) > 20
for: 5m
labels:
application: techlog
severity: warning
- alert: TechLogLatencyHigh
annotations:
summary: TechLog p99 latency exceeds 2 seconds
expr: histogram_quantile(0.99,sum by(le)(rate(http_server_requests_seconds_bucket{namespace="hyeonworks-prod",uri!~"/actuator.*"}[5m]))) > 2
for: 10m
labels:
application: techlog
severity: warning
- alert: TechLogHeapPressure
annotations:
summary: TechLog heap usage exceeds 85%
expr: sum(jvm_memory_used_bytes{namespace="hyeonworks-prod",area="heap"}) / sum(jvm_memory_max_bytes{namespace="hyeonworks-prod",area="heap"}) > 0.85
for: 10m
labels:
application: techlog
severity: warning
- alert: TechLogDatabasePoolWaiting
annotations:
summary: TechLog database connections are queued
expr: hikaricp_connections_pending{namespace="hyeonworks-prod"} > 0
for: 5m
labels:
application: techlog
severity: warning
@@ -0,0 +1,55 @@
apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata:
name: platform-postgres-techlog
namespace: platform-data
spec:
cluster:
name: platform-postgres
databaseReclaimPolicy: retain
ensure: present
name: techlog
owner: techlog
---
apiVersion: postgresql.cnpg.io/v1
kind: DatabaseRole
metadata:
name: platform-postgres-techlog
namespace: platform-data
spec:
bypassrls: false
cluster:
name: platform-postgres
connectionLimit: 30
createdb: false
createrole: false
databaseRoleReclaimPolicy: retain
login: true
name: techlog
passwordSecret:
name: techlog-db-credentials
replication: false
superuser: false
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-techlog-backend
namespace: platform-data
spec:
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: hyeonworks-prod
podSelector:
matchLabels:
app: techlog-backend
ports:
- port: 5432
protocol: TCP
podSelector:
matchLabels:
cnpg.io/cluster: platform-postgres
policyTypes:
- Ingress