Author SHA1 Message Date
DongHyeonka 3eafb7e1d8 feat(validate): add a live drift check against the repository renders
The 2026-11-11 review found a pg_hba rule that existed only in the
cluster by rendering every root by hand and running kubectl diff.
Nothing does that regularly, so drift is found by accident.

live-drift-check.sh renders all live roots (Phase 1 and 2, admin
services, observability core, the Traefik trust overlay, the tenant,
PriorityClass and keycloak policy roots), strips Secrets and runs
kubectl diff, which is a server-side dry run. It prints one line per
drifted object, honours a dated accepted-drift list and can write
Prometheus textfile metrics for a later alert.

Risk: read-only. It consumes the render scripts' verified-output-dir
handoff and deletes those directories on exit; the README notes this
second consumer. Secret values are not compared.

Verification (desktop, before this series is applied): 17 drifted
objects, matching the review plus the new roots; exit 1; the textfile
holds platform_live_drift_objects 17.
2026-11-12 00:29:18 +09:00
DongHyeonka ebdf33f072 feat(tenants): declare the tech-log objects inside platform namespaces
The tech-log tenant needs objects inside platform namespaces: its
database and role, ingress to Postgres and AIStor, blackbox probes,
alert rules, a dashboard and two observability NetworkPolicies. They
were only ever applied with kubectl and no repository holds them, so a
deleted object cannot be recreated and a platform re-apply can drop
tech-log access without anyone noticing.

Declare them in services/tenants/techlog, copied from each object's
last-applied configuration. The dashboard follows the repository's
dashboard pattern (JSON file + configMapGenerator, same name, same
labels). The hyeonworks-prod Namespace moves here as the tenant
boundary and gets PSA enforce=baseline, warn/audit=restricted.

Rejected: enforce=restricted now. A server dry-run shows all three
tech-log pods would violate it; baseline has zero violations.

Risk: the render matches the live objects exactly except the added PSA
labels. The pg_hba rules and the Secret stay out of this root (Cluster
owner and Git policy).

Verification (desktop):
- render-tenants-and-policies.sh passes.
- kubectl diff of the render shows only the Namespace PSA labels.
- kubectl label --dry-run=server enforce=baseline: no warnings.
2026-11-12 00:29:18 +09:00
DongHyeonka c0749ec8a4 feat(keycloak): default-deny the keycloak namespace egress
The keycloak namespace only has the ingress policy the Keycloak
Operator creates. Neither the Keycloak pod nor the Operator pod has any
egress restriction, unlike every other platform namespace.

Add a namespace-wide default deny and allow only what is used: DNS,
platform-postgres 5432 (server and realm-import jobs), JGroups between
Keycloak pods, and for the Operator the API server and the Keycloak
management ports. Keycloak ingress stays owned by the Operator policy.

Rejected: allowing public 443 like Gitea does. Neither the repository
nor the Keycloak CR configures SMTP or an external identity provider,
and the only open TCP connections from keycloak-0 on 2026-11-12 were to
PostgreSQL. Adding SMTP or an IdP later needs a new egress rule; the
README says so.

Risk: a missed dependency would break logins for every service. The
README lists the post-apply checks (OIDC discovery, a new TCP
connection to Postgres from keycloak-0, Operator logs) and the one-line
rollback.

Verification: render and kubectl diff only add the five policies.
2026-11-12 00:29:18 +09:00
DongHyeonka c2a3f75bc3 feat(priority): add platform-critical and platform-low PriorityClasses
Every platform pod runs at priority 0. During the 2026-11-09 disk
eviction the tech-log production pods were ranked with Grafana and
kube-state-metrics. The kubelet ranks evictions by usage over requests
and then by priority, so without classes the database and the
production app have no head start.

Add two classes and attach nothing yet: platform-critical (1000000,
may preempt) for platform-postgres, Gitea, Keycloak and tech-log, and
platform-low (-1000, never preempts) for Grafana, pgAdmin, Loki and
Tempo. Neither is the global default, so no pod changes silently.

Risk: creating the classes changes no pod. Attaching them restarts the
pods, so that is left for the k3s restart window.

Verification: render-tenants-and-policies.sh (added later in this
series) checks the values, the preemption policy and globalDefault.
2026-11-12 00:28:57 +09:00
donghyeon.kang 7580eb7f75 Merge pull request 'docs(reviews): add the k3s maturity review and set Traefik resources' (#3) from donghyeon.kang/k3s-maturity-review into main 2026-11-11 15:14:16 +00:00
19 changed files with 878 additions and 0 deletions
@@ -0,0 +1,38 @@
# keycloak namespace NetworkPolicy
2026-11-11 점검 전 `keycloak` namespace에는 Keycloak Operator가 만든 ingress 정책 하나뿐이었다.
Keycloak 파드와 Operator 파드의 egress는 막혀 있지 않았다. 다른 플랫폼 namespace처럼 기본 차단에
필요한 것만 연다.
| 정책 | 대상 | 허용 |
| --- | --- | --- |
| `keycloak-default-deny` | 모든 파드 | ingress·egress 기본 차단 |
| `keycloak-allow-dns` | 모든 파드 | kube-dns 53 |
| `keycloak-allow-platform-postgres` | 모든 파드(서버와 realm import Job) | `platform-data` Postgres 5432 |
| `keycloak-allow-cluster-peers` | Keycloak 파드 | 같은 Keycloak 파드 JGroups 7800·57800 |
| `keycloak-operator-allow-egress` | Operator | API 서버(`10.43.0.1:443`, `192.168.0.107:6443`), Keycloak 8080·9000 |
- Keycloak 파드의 **ingress**는 계속 Keycloak CR `spec.networkPolicy`로 Operator가 만드는
`keycloak-network-policy`가 소유한다(Traefik 8080, 관리 9000, JGroups).
- 바깥(인터넷)으로 나가는 길은 없다. 2026-11-12에 repo와 Keycloak CR에 SMTP·외부 IdP 설정이
없고, Keycloak 파드의 열린 TCP 연결이 Postgres뿐인 것을 확인했다. SMTP나 외부 IdP를 붙이면
여기에 egress를 더해야 한다(안 더하면 메일·외부 로그인만 조용히 실패한다).
- kubelet probe는 노드에서 오므로 막히지 않는다.
## 적용과 확인
```sh
bash scripts/validate/render-tenants-and-policies.sh
kubectl kustomize infrastructure/networking/keycloak | kubectl diff -f -
kubectl apply -k infrastructure/networking/keycloak
```
적용 뒤 확인(모두 통과해야 한다):
```sh
kubectl -n keycloak get pods
curl -fsS -o /dev/null -w '%{http_code}\n' https://id.learn.hyeonworks.com/realms/hyeonworks/.well-known/openid-configuration
kubectl -n keycloak logs deploy/keycloak-operator --since=5m | grep -i -E 'error|exception|timeout' || echo no-operator-errors
```
되돌리기: `kubectl delete -k infrastructure/networking/keycloak`(Operator 정책은 남는다).
@@ -0,0 +1,12 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: keycloak
labels:
- pairs:
app.kubernetes.io/part-of: platform
includeSelectors: false
resources:
- network-policies.yaml
@@ -0,0 +1,117 @@
# keycloak namespace: 기본 차단 + 필요한 것만 허용.
# Keycloak 파드의 ingress는 Keycloak CR(spec.networkPolicy)로 Operator가 만드는
# keycloak-network-policy가 계속 소유한다. 여기서는 기본 차단과 egress만 정한다.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: keycloak-default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: keycloak-allow-dns
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP
---
# Keycloak 서버와 realm import Job이 공용 PostgreSQL에 붙는다.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: keycloak-allow-platform-postgres
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: platform-data
podSelector:
matchLabels:
cnpg.io/cluster: platform-postgres
ports:
- port: 5432
protocol: TCP
---
# 같은 Keycloak 인스턴스끼리의 JGroups(7800, 57800). Operator ingress 정책과 짝이다.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: keycloak-allow-cluster-peers
spec:
podSelector:
matchLabels:
app: keycloak
app.kubernetes.io/instance: keycloak
app.kubernetes.io/managed-by: keycloak-operator
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
app: keycloak
app.kubernetes.io/instance: keycloak
app.kubernetes.io/managed-by: keycloak-operator
ports:
- port: 7800
protocol: TCP
- port: 57800
protocol: TCP
---
# Operator: Kubernetes API(ClusterIP와 실제 API 서버 주소)와 Keycloak 관리 포트.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: keycloak-operator-allow-egress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: keycloak-operator
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 10.43.0.1/32
ports:
- port: 443
protocol: TCP
- to:
- ipBlock:
cidr: 192.168.0.107/32
ports:
- port: 6443
protocol: TCP
- to:
- podSelector:
matchLabels:
app: keycloak
app.kubernetes.io/instance: keycloak
app.kubernetes.io/managed-by: keycloak-operator
ports:
- port: 8080
protocol: TCP
- port: 9000
protocol: TCP
+37
View File
@@ -559,3 +559,40 @@ parser와 세 mode의 fixture 회귀는 live cluster에 연결하지 않고 다
```sh
bash scripts/validate/test-k3s-secret-encryption-status.sh
```
## 테넌트·우선순위·keycloak 정책 렌더 검사
`render-tenants-and-policies.sh`는 클러스터에 닿지 않고 세 빌드 루트를 렌더해 불변식을
확인한다.
- `services/tenants/techlog`: tech-log 테넌트의 플랫폼 쪽 객체 11개(목록이 정확히 같아야
한다), `hyeonworks-prod` PSA 라벨, techlog DatabaseRole 권한 플래그, 대시보드 이름
- `services/priority-classes`: `platform-critical`(0보다 크고 시스템 클래스보다 작다),
`platform-low`(0보다 작고 선점하지 않는다), 둘 다 global default가 아니다
- `infrastructure/networking/keycloak`: namespace 전체 기본 차단, egress 규칙마다 포트,
허용 CIDR은 API 서버 둘뿐
```sh
bash scripts/validate/render-tenants-and-policies.sh
```
## 라이브 drift 검사
`live-drift-check.sh`는 이 저장소의 라이브 빌드 루트를 모두 렌더하고(Phase 1·2, 관리 도구,
관측 core, Traefik trust overlay, 위 세 루트) Secret을 뺀 뒤 `kubectl diff`(서버 dry-run,
저장하지 않음)로 클러스터와 비교한다. 렌더에는 위 render 스크립트의 `--verified-output-dir`
출력을 읽기만 하는 용도로 쓴다(그 디렉터리는 스크립트가 끝날 때 지운다).
```sh
PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm \
bash scripts/validate/live-drift-check.sh \
--accepted scripts/validate/live-drift-accepted.txt
```
- 종료 코드: 0 차이 없음, 1 차이 있음, 2 검사 자체 실패.
- `DRIFT <Kind> <ns>/<name> (+추가 -삭제) [입력 파일]`로 객체마다 한 줄을 쓴다.
`generation` 줄만 다른 것은 세지 않는다.
- `--accepted`: 알고 있는 차이를 기한과 이유를 달아 적는다. 기한이 지나면 다시 drift로 센다.
- `--textfile FILE`: `platform_live_drift_objects`, `platform_live_drift_check_success`,
`platform_live_drift_last_run_timestamp_seconds`를 Prometheus textfile 형식으로 쓴다.
- Secret 값의 차이는 보지 않는다.
+10
View File
@@ -0,0 +1,10 @@
# 알고 있는 라이브 drift. 형식: <Kind> <namespace 또는 -> <name> <기한 YYYY-MM-DD> <이유>
# 기한이 지나면 live-drift-check.sh가 다시 drift로 센다. 해소하면 줄을 지운다.
Cluster platform-data platform-postgres 2026-11-30 pg_hba의 techlog 두 줄이 repo에 없다. w1이 CNPG 백업과 함께 넣는다(1c4ef9a).
DatabaseRole platform-data platform-postgres-gitea 2026-11-30 렌더가 false 기본값 필드를 명시할 뿐 값은 같다. 다음 Phase 1 적용 때 사라진다.
DatabaseRole platform-data platform-postgres-keycloak 2026-11-30 렌더가 false 기본값 필드를 명시할 뿐 값은 같다. 다음 Phase 2 적용 때 사라진다.
Deployment keycloak keycloak-operator 2026-11-30 원격 ref 26.7.0 태그 내용이 바뀌어 build-timestamp 라벨만 다르다. 재시작 창에 커밋 SHA 고정과 함께 적용한다.
Service keycloak keycloak-operator 2026-11-30 위와 같은 build-timestamp 라벨.
ServiceAccount keycloak keycloak-operator 2026-11-30 위와 같은 build-timestamp 라벨.
Deployment observability observability-core-kube-pr-operator 2026-11-30 이미지 문자열이 tag@digest와 digest로 다르고 digest는 같다. 적용하면 재시작되므로 재시작 창에.
Deployment observability observability-core-kube-state-metrics 2026-11-30 위와 같은 이미지 문자열 차이.
+173
View File
@@ -0,0 +1,173 @@
#!/usr/bin/env bash
# Render every live build root of this repository and compare it with the
# cluster using `kubectl diff` (server-side dry-run: nothing is persisted).
#
# Usage:
# PLATFORM_HELM_BIN=/abs/helm bash scripts/validate/live-drift-check.sh \
# [--accepted FILE] [--textfile FILE]
#
# --accepted FILE known differences, one per line:
# `<Kind> <namespace or -> <name> <until YYYY-MM-DD> <reason...>`
# Expired entries count as drift again.
# --textfile FILE also write Prometheus textfile metrics (atomic rename).
#
# Exit status: 0 no drift, 1 drift found, 2 the check itself failed.
# Secrets are removed from the renders before diffing; Secret drift is out of scope.
set -Eeuo pipefail
readonly REPOSITORY_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
accepted_file=""
textfile=""
while (( $# > 0 )); do
case "$1" in
--accepted) accepted_file="${2:?--accepted needs a file}"; shift 2 ;;
--textfile) textfile="${2:?--textfile needs a file}"; shift 2 ;;
-h|--help) sed -n '2,16p' "${BASH_SOURCE[0]}"; exit 0 ;;
*) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
esac
done
declare -a made_dirs=()
cleanup() {
local d
for d in "${made_dirs[@]}"; do
case "$d" in /tmp/platform-*) rm -rf -- "$d" ;; esac
done
}
trap cleanup EXIT
write_textfile() {
local drift="$1" ok="$2"
[[ -n "$textfile" ]] || return 0
local tmp="${textfile}.tmp.$$"
{
echo '# HELP platform_live_drift_objects Objects whose live state differs from the repository render.'
echo '# TYPE platform_live_drift_objects gauge'
echo "platform_live_drift_objects ${drift}"
echo '# HELP platform_live_drift_check_success 1 when the drift check itself completed.'
echo '# TYPE platform_live_drift_check_success gauge'
echo "platform_live_drift_check_success ${ok}"
echo '# HELP platform_live_drift_last_run_timestamp_seconds Unix time the drift check finished.'
echo '# TYPE platform_live_drift_last_run_timestamp_seconds gauge'
echo "platform_live_drift_last_run_timestamp_seconds $(date +%s)"
} >"$tmp"
mv -f -- "$tmp" "$textfile"
}
die() {
printf 'DRIFT CHECK ERROR: %s\n' "$*" >&2
write_textfile 0 0
exit 2
}
work="$(mktemp -d /tmp/platform-drift.XXXXXX)"; made_dirs+=("$work")
mkdir -p "$work/in" "$work/out"
render_root() {
local script="$1" prefix="$2" dir
dir="$(mktemp -d "/tmp/${prefix}.XXXXXX")"; made_dirs+=("$dir")
bash "$REPOSITORY_ROOT/scripts/validate/${script}" --verified-output-dir "$dir" \
>"$work/${script}.log" 2>&1 || { tail -5 "$work/${script}.log" >&2; die "${script} failed"; }
printf '%s\n' "$dir"
}
phase1="$(render_root render-phase1.sh platform-phase1-apply)"
phase2="$(render_root render-phase2.sh platform-phase2-apply)"
admin="$(render_root render-admin-services.sh platform-admin-apply)"
observability="$(render_root render-observability-core.sh platform-observability-core-apply)"
# The live Gitea is the OIDC profile; gitea.yaml is the fresh-install baseline only.
declare -a inputs=(
"$phase1/namespaces.yaml" "$phase1/ssd-local-pv.yaml" "$phase1/cnpg-operator.yaml"
"$phase1/platform-postgres.yaml" "$phase1/gitea-oidc.yaml"
"$observability/aggregate.yaml"
)
for f in "$phase2"/*.yaml "$admin"/*.yaml; do inputs+=("$f"); done
# Live Traefik is the trust overlay (infrastructure/networking/traefik/README.md).
kubectl kustomize "$REPOSITORY_ROOT/infrastructure/networking/traefik/overlays/trust" >"$work/traefik-trust.yaml" || die "traefik render failed"
kubectl kustomize "$REPOSITORY_ROOT/services/tenants/techlog" >"$work/tenant-techlog.yaml" || die "tenant render failed"
kubectl kustomize "$REPOSITORY_ROOT/services/priority-classes" >"$work/priority-classes.yaml" || die "priority class render failed"
kubectl kustomize "$REPOSITORY_ROOT/infrastructure/networking/keycloak" >"$work/keycloak-network-policies.yaml" || die "keycloak policy render failed"
inputs+=("$work/traefik-trust.yaml" "$work/tenant-techlog.yaml" "$work/priority-classes.yaml" "$work/keycloak-network-policies.yaml")
n=0
for f in "${inputs[@]}"; do
n=$((n + 1))
python3 -I - "$f" "$work/in/$(printf '%02d' "$n")-$(basename "$f")" <<'PY' || die "could not strip Secrets from $f"
import sys, yaml
yaml.SafeLoader.add_constructor("tag:yaml.org,2002:value", lambda l, n: l.construct_scalar(n))
docs = [d for d in yaml.safe_load_all(open(sys.argv[1])) if isinstance(d, dict) and d.get("kind") not in (None, "Secret")]
yaml.safe_dump_all(docs, open(sys.argv[2], "w"), sort_keys=False)
PY
done
for f in "$work"/in/*.yaml; do
set +e
KUBECTL_EXTERNAL_DIFF="diff -u -N" kubectl diff -f "$f" >"$work/out/$(basename "$f").diff" 2>"$work/out/$(basename "$f").err"
rc=$?
set -e
if (( rc > 1 )); then
sed -n '1,5p' "$work/out/$(basename "$f").err" >&2
die "kubectl diff failed for $(basename "$f")"
fi
done
set +e
python3 -I - "$work/out" "${accepted_file:-}" "$work/drift-count" <<'PY'
import sys, pathlib, re, datetime
out, accepted_path, count_path = pathlib.Path(sys.argv[1]), sys.argv[2], pathlib.Path(sys.argv[3])
today = datetime.date.today()
accepted, expired = {}, set()
if accepted_path:
for line in pathlib.Path(accepted_path).read_text().splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
kind, ns, name, until, *reason = line.split()
key = (kind, "" if ns == "-" else ns, name)
if datetime.date.fromisoformat(until) < today:
expired.add(key)
else:
accepted[key] = " ".join(reason)
version = re.compile(r"^v\d+((alpha|beta)\d+)?$")
objects = {}
for diff in sorted(out.glob("*.diff")):
current = None
for line in diff.read_text().splitlines():
if line.startswith("diff -u -N "):
parts = line.split()[3].rsplit("/", 1)[-1].split(".")
i = next(i for i, p in enumerate(parts) if version.match(p))
current = (parts[i + 1], parts[i + 2], ".".join(parts[i + 3:]))
objects.setdefault(current, [0, 0, diff.name])
elif current and line[:1] in "+-" and not line.startswith(("+++", "---")):
if re.match(r"^[+-]\s+generation:", line):
continue
objects[current][0 if line[0] == "+" else 1] += 1
drift = accepted_seen = 0
for key, (plus, minus, src) in sorted(objects.items()):
if plus == 0 and minus == 0:
continue
kind, ns, name = key
label = f"{kind} {ns or '-'}/{name} (+{plus} -{minus}) [{src}]"
if key in accepted:
accepted_seen += 1
print(f"ACCEPTED {label}: {accepted[key]}")
continue
drift += 1
print(f"DRIFT {label}" + (" (accepted entry expired)" if key in expired else ""))
print(f"SUMMARY drift={drift} accepted={accepted_seen}")
count_path.write_text(str(drift))
PY
rc=$?
set -e
(( rc == 0 )) || die "could not summarize the diffs"
drift_count="$(cat "$work/drift-count")"
write_textfile "$drift_count" 1
if (( drift_count == 0 )); then
echo "LIVE DRIFT CHECK PASS"
exit 0
fi
echo "LIVE DRIFT CHECK FOUND DRIFT"
exit 1
@@ -0,0 +1,126 @@
#!/usr/bin/env bash
# Render the tech-log tenant root, the platform PriorityClasses and the keycloak
# namespace NetworkPolicies, and check their invariants. Static: no cluster access.
set -Eeuo pipefail
readonly EXPECTED_KUSTOMIZE_VERSION="v5.8.1"
readonly REPOSITORY_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
fail() {
printf 'ERROR: %s\n' "$*" >&2
exit 1
}
kustomize_version="$(kubectl version --client --output=yaml | sed -n 's/^kustomizeVersion: //p')"
[[ "$kustomize_version" == "$EXPECTED_KUSTOMIZE_VERSION" ]] || \
fail "expected Kustomize ${EXPECTED_KUSTOMIZE_VERSION}, found ${kustomize_version:-unknown}"
work="$(mktemp -d "${TMPDIR:-/tmp}/platform-tenants-render.XXXXXX")"
trap 'rm -rf -- "$work"' EXIT
kubectl kustomize "$REPOSITORY_ROOT/services/tenants/techlog" >"$work/techlog.yaml"
kubectl kustomize "$REPOSITORY_ROOT/services/priority-classes" >"$work/priority-classes.yaml"
kubectl kustomize "$REPOSITORY_ROOT/infrastructure/networking/keycloak" >"$work/keycloak-network-policies.yaml"
python3 -I - "$work" <<'PY'
import sys, pathlib, yaml
work = pathlib.Path(sys.argv[1])
errors = []
def load(name):
return [d for d in yaml.safe_load_all((work / name).read_text()) if d]
def keyset(docs):
return sorted((d["kind"], d["metadata"].get("namespace", ""), d["metadata"]["name"]) for d in docs)
for name in ("techlog.yaml", "priority-classes.yaml", "keycloak-network-policies.yaml"):
for d in load(name):
if d["kind"] == "Secret":
errors.append(f"{name}: Secret objects must not be rendered")
techlog = load("techlog.yaml")
expected = sorted([
("Namespace", "", "hyeonworks-prod"),
("Database", "platform-data", "platform-postgres-techlog"),
("DatabaseRole", "platform-data", "platform-postgres-techlog"),
("NetworkPolicy", "platform-data", "allow-techlog-backend"),
("NetworkPolicy", "object-storage", "allow-techlog-backend"),
("Probe", "observability", "techlog-edge"),
("Probe", "observability", "techlog-readiness"),
("PrometheusRule", "observability", "techlog"),
("ConfigMap", "observability", "grafana-dashboard-techlog"),
("NetworkPolicy", "observability", "blackbox-allow-techlog-health"),
("NetworkPolicy", "observability", "observability-allow-techlog-scrape"),
])
if keyset(techlog) != expected:
errors.append(f"tech-log tenant objects differ: {keyset(techlog)}")
for d in techlog:
kind = d["kind"]
if kind == "Namespace":
labels = d["metadata"].get("labels", {})
want = {
"pod-security.kubernetes.io/enforce": "baseline",
"pod-security.kubernetes.io/warn": "restricted",
"pod-security.kubernetes.io/audit": "restricted",
"observability.hyeonworks.com/enabled": "true",
}
for k, v in want.items():
if labels.get(k) != v:
errors.append(f"hyeonworks-prod label {k} must be {v}")
if kind == "DatabaseRole":
spec = d["spec"]
if spec.get("passwordSecret", {}).get("name") != "techlog-db-credentials":
errors.append("techlog DatabaseRole must reference techlog-db-credentials")
for flag in ("superuser", "createdb", "createrole", "replication", "bypassrls"):
if spec.get(flag) is not False:
errors.append(f"techlog DatabaseRole {flag} must be false")
if kind in ("Database", "DatabaseRole"):
policy = d["spec"].get("databaseReclaimPolicy") or d["spec"].get("databaseRoleReclaimPolicy")
if policy != "retain":
errors.append(f"{kind} must keep the retain reclaim policy")
if kind == "ConfigMap":
labels = d["metadata"].get("labels", {})
if labels.get("grafana_dashboard") != "1":
errors.append("tech-log dashboard ConfigMap needs grafana_dashboard=1")
if d["metadata"]["name"] != "grafana-dashboard-techlog":
errors.append("tech-log dashboard ConfigMap must keep its name (no hash suffix)")
classes = {d["metadata"]["name"]: d for d in load("priority-classes.yaml")}
if sorted(classes) != ["platform-critical", "platform-low"]:
errors.append(f"PriorityClasses differ: {sorted(classes)}")
else:
crit, low = classes["platform-critical"], classes["platform-low"]
if not (0 < crit["value"] < 1_000_000_000):
errors.append("platform-critical must be above 0 and below the system classes")
if not low["value"] < 0:
errors.append("platform-low must be below the default priority 0")
if low.get("preemptionPolicy") != "Never":
errors.append("platform-low must never preempt")
for d in classes.values():
if d.get("globalDefault") is not False:
errors.append(f"{d['metadata']['name']} must not be the global default")
policies = {d["metadata"]["name"]: d for d in load("keycloak-network-policies.yaml")}
deny = policies.get("keycloak-default-deny")
if not deny or deny["spec"].get("podSelector") != {} or sorted(deny["spec"].get("policyTypes", [])) != ["Egress", "Ingress"]:
errors.append("keycloak needs a namespace-wide default deny for ingress and egress")
for d in policies.values():
if d["metadata"].get("namespace") != "keycloak":
errors.append(f"{d['metadata']['name']} must live in the keycloak namespace")
for rule in d["spec"].get("egress", []) or []:
for peer in rule.get("to", []):
cidr = peer.get("ipBlock", {}).get("cidr")
if cidr and cidr not in ("10.43.0.1/32", "192.168.0.107/32"):
errors.append(f"{d['metadata']['name']}: unexpected egress CIDR {cidr}")
if not rule.get("ports"):
errors.append(f"{d['metadata']['name']}: every egress rule must name its ports")
if errors:
for e in errors:
print(f"ERROR: {e}", file=sys.stderr)
sys.exit(1)
print(f"tech-log tenant: {len(techlog)} objects; PriorityClasses: {len(classes)}; keycloak NetworkPolicies: {len(policies)}")
PY
echo "TENANTS AND POLICIES STATIC RENDER PASS"
+20
View File
@@ -0,0 +1,20 @@
# 플랫폼 PriorityClass
노드가 한 대라 자원이 모자라면 kubelet이 파드를 내보낸다. 내보낼 순서는 requests를 넘었는지,
그다음 우선순위로 정한다. 2026-11-09 디스크 축출 때 플랫폼 파드는 모두 우선순위 0이라 tech-log
운영 파드가 Grafana·kube-state-metrics와 같은 순위로 나갔다.
| 이름 | 값 | 선점 | 붙일 곳 |
| --- | --- | --- | --- |
| `platform-critical` | 1000000 | 낮은 것을 선점 | platform-postgres, Gitea, Keycloak, tech-log |
| `platform-low` | -1000 | 선점 안 함 | Grafana, pgAdmin, Loki, Tempo |
- 아무것도 안 붙인 파드는 기본값 0이다. 시스템 클래스(`system-*`, 20억)는 그대로 가장 높다.
- `globalDefault`는 둘 다 끈다. 새 파드가 모르게 우선순위를 얻지 않게 하려는 것이다.
- 클래스를 만드는 것은 아무 파드도 바꾸지 않는다. **붙이는 것은 파드 재시작을 부르므로**
k3s 재시작 창에 한다(`infrastructure/host/k3s/README.md`).
```sh
kubectl apply -k services/priority-classes
kubectl get priorityclass platform-critical platform-low
```
@@ -0,0 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- priority-classes.yaml
@@ -0,0 +1,25 @@
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: platform-critical
labels:
app.kubernetes.io/part-of: platform
value: 1000000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: >-
Stateful platform services and production workloads that other services depend on
(platform-postgres, Gitea, Keycloak, tech-log). Evicted after everything else.
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: platform-low
labels:
app.kubernetes.io/part-of: platform
value: -1000
globalDefault: false
preemptionPolicy: Never
description: >-
Tools whose loss does not stop a user-facing service (Grafana, pgAdmin, Loki, Tempo).
Evicted first and never preempts another pod.
+44
View File
@@ -0,0 +1,44 @@
# tech-log 테넌트 — 플랫폼 쪽 객체
tech-log(hyeonworks.com 운영)가 플랫폼을 쓰려면 플랫폼 namespace 안에 허용 객체가 있어야 한다.
이 빌드 루트가 그 객체를 소유한다. 앱 자체(Deployment·Service·Ingress·ServiceMonitor·앱
namespace 안의 NetworkPolicy)는 tech-log 저장소의 `deploy/`가 소유한다.
2026-11-11 점검(`docs/reviews/2026-11-11-k3s-maturity-review.md` 4.3절) 전까지 이 객체들은
`kubectl apply`로만 들어가 있었고 어느 저장소에도 없었다. 처음 내용은 라이브 객체의
`last-applied-configuration`을 그대로 옮겼다. 라이브와 다른 것은 `hyeonworks-prod`의 PSA 라벨뿐이다.
| 파일 | 객체 |
| --- | --- |
| `namespace.yaml` | Namespace `hyeonworks-prod`: 관측 대상 라벨, PSA `enforce=baseline`·`warn=restricted`·`audit=restricted` |
| `platform-data.yaml` | `techlog` Database와 DatabaseRole(로그인만, 연결 30), Postgres로 들어오는 ingress |
| `object-storage.yaml` | AIStor 9000으로 들어오는 ingress |
| `observability-probes.yaml` | blackbox Probe 둘(공개 HTTPS, 내부 readiness) |
| `observability-rules.yaml` | PrometheusRule `techlog` 일곱 개 |
| `dashboards/techlog-operations.json` | Grafana 대시보드(`grafana-dashboard-techlog` ConfigMap으로 생성) |
| `observability-network-policies.yaml` | blackbox → readiness, Prometheus → management scrape egress |
## 여기에 없는 것
- `platform-data/platform-postgres`의 `pg_hba` techlog 두 줄: Cluster 객체는
`services/platform-postgres`가 소유한다.
- Secret: `platform-data/techlog-db-credentials`(`kubernetes.io/basic-auth`, `username`은 정확히
`techlog`)는 DatabaseRole이 참조한다. 앱 쪽 `hyeonworks-prod/hyeonworks-secrets`는 tech-log가
소유한다. 둘 다 Git에 두지 않는다.
## PSA
`enforce=baseline`은 2026-11-12에 서버 dry-run(`kubectl label --dry-run=server`)으로 위반 0을
확인하고 붙였다. `restricted`로 강제하면 지금 파드 셋(redis, backend, frontend)이 모두 걸린다
(`allowPrivilegeEscalation`, capabilities, `runAsNonRoot`, seccomp). tech-log 쪽 securityContext가
고쳐진 뒤 같은 dry-run이 경고 없이 통과하면 `enforce`를 `restricted`로 올린다.
## 렌더·적용
```sh
bash scripts/validate/render-tenants-and-policies.sh
kubectl kustomize services/tenants/techlog | kubectl diff -f -
kubectl apply -k services/tenants/techlog
```
새로 설치할 때는 이 루트를 tech-log `deploy/`보다 먼저 적용한다(namespace를 여기서 만든다).
File diff suppressed because one or more lines are too long
@@ -0,0 +1,23 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# tech-log(hyeonworks.com) 테넌트의 플랫폼 쪽 객체. 앱 쪽 매니페스트는 tech-log repo가 소유한다.
resources:
- namespace.yaml
- platform-data.yaml
- object-storage.yaml
- observability-probes.yaml
- observability-rules.yaml
- observability-network-policies.yaml
generatorOptions:
disableNameSuffixHash: true
labels:
app.kubernetes.io/part-of: techlog
grafana_dashboard: "1"
configMapGenerator:
- name: grafana-dashboard-techlog
namespace: observability
files:
- dashboards/techlog-operations.json
+12
View File
@@ -0,0 +1,12 @@
apiVersion: v1
kind: Namespace
metadata:
name: hyeonworks-prod
labels:
observability.hyeonworks.com/enabled: 'true'
pod-security.kubernetes.io/enforce: baseline
pod-security.kubernetes.io/enforce-version: latest
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: latest
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/audit-version: latest
@@ -0,0 +1,22 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-techlog-backend
namespace: object-storage
spec:
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: hyeonworks-prod
podSelector:
matchLabels:
app: techlog-backend
ports:
- port: 9000
protocol: TCP
podSelector:
matchLabels:
aistor.min.io/objectStore: minio-aistor
policyTypes:
- Ingress
@@ -0,0 +1,45 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: blackbox-allow-techlog-health
namespace: observability
spec:
egress:
- ports:
- port: 9001
protocol: TCP
to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: hyeonworks-prod
podSelector:
matchLabels:
app: techlog-backend
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus-blackbox-exporter
policyTypes:
- Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-techlog-scrape
namespace: observability
spec:
egress:
- ports:
- port: 9001
protocol: TCP
to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: hyeonworks-prod
podSelector:
matchLabels:
app: techlog-backend
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
policyTypes:
- Egress
@@ -0,0 +1,46 @@
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
labels:
observability.hyeonworks.com/instance: home
name: techlog-edge
namespace: observability
spec:
interval: 30s
jobName: techlog-edge
module: http_2xx
prober:
path: /probe
scheme: http
url: blackbox-exporter.observability.svc.cluster.local:9115
scrapeTimeout: 15s
targets:
staticConfig:
labels:
application: techlog
static:
- https://hyeonworks.com/
- https://hyeonworks.com/api/healthcheck
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
labels:
observability.hyeonworks.com/instance: home
name: techlog-readiness
namespace: observability
spec:
interval: 30s
jobName: techlog-readiness
module: http_2xx
prober:
path: /probe
scheme: http
url: blackbox-exporter.observability.svc.cluster.local:9115
scrapeTimeout: 15s
targets:
staticConfig:
labels:
application: techlog
static:
- http://techlog-management.hyeonworks-prod.svc.cluster.local:9001/actuator/health/readiness
@@ -0,0 +1,67 @@
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
labels:
observability.hyeonworks.com/instance: home
name: techlog
namespace: observability
spec:
groups:
- name: techlog
rules:
- alert: TechLogMetricsDown
annotations:
summary: TechLog metrics target is unreachable
expr: up{namespace="hyeonworks-prod",service="techlog-management"} == 0
for: 5m
labels:
application: techlog
severity: warning
- alert: TechLogReadinessDown
annotations:
summary: TechLog dependency readiness is DOWN
expr: probe_success{job="techlog-readiness"} == 0
for: 3m
labels:
application: techlog
severity: critical
- alert: TechLogHTTPSDown
annotations:
summary: TechLog HTTPS or API health is failing
expr: probe_success{job="techlog-edge"} == 0
for: 3m
labels:
application: techlog
severity: critical
- alert: TechLogHTTP5xx
annotations:
summary: TechLog HTTP 5xx exceeds 5%
expr: (sum(rate(http_server_requests_seconds_count{namespace="hyeonworks-prod",status=~"5.."}[5m])) / clamp_min(sum(rate(http_server_requests_seconds_count{namespace="hyeonworks-prod"}[5m])),0.001)) > 0.05 and sum(increase(http_server_requests_seconds_count{namespace="hyeonworks-prod"}[5m])) > 20
for: 5m
labels:
application: techlog
severity: warning
- alert: TechLogLatencyHigh
annotations:
summary: TechLog p99 latency exceeds 2 seconds
expr: histogram_quantile(0.99,sum by(le)(rate(http_server_requests_seconds_bucket{namespace="hyeonworks-prod",uri!~"/actuator.*"}[5m]))) > 2
for: 10m
labels:
application: techlog
severity: warning
- alert: TechLogHeapPressure
annotations:
summary: TechLog heap usage exceeds 85%
expr: sum(jvm_memory_used_bytes{namespace="hyeonworks-prod",area="heap"}) / sum(jvm_memory_max_bytes{namespace="hyeonworks-prod",area="heap"}) > 0.85
for: 10m
labels:
application: techlog
severity: warning
- alert: TechLogDatabasePoolWaiting
annotations:
summary: TechLog database connections are queued
expr: hikaricp_connections_pending{namespace="hyeonworks-prod"} > 0
for: 5m
labels:
application: techlog
severity: warning
@@ -0,0 +1,55 @@
apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata:
name: platform-postgres-techlog
namespace: platform-data
spec:
cluster:
name: platform-postgres
databaseReclaimPolicy: retain
ensure: present
name: techlog
owner: techlog
---
apiVersion: postgresql.cnpg.io/v1
kind: DatabaseRole
metadata:
name: platform-postgres-techlog
namespace: platform-data
spec:
bypassrls: false
cluster:
name: platform-postgres
connectionLimit: 30
createdb: false
createrole: false
databaseRoleReclaimPolicy: retain
login: true
name: techlog
passwordSecret:
name: techlog-db-credentials
replication: false
superuser: false
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-techlog-backend
namespace: platform-data
spec:
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: hyeonworks-prod
podSelector:
matchLabels:
app: techlog-backend
ports:
- port: 5432
protocol: TCP
podSelector:
matchLabels:
cnpg.io/cluster: platform-postgres
policyTypes:
- Ingress