docs(keycloak-session-store): import the session-storage lab as a new project
The keycloak project ended with four open questions that design could not
settle. A two-VM lab was built to answer them by measurement, and this is
that material: 26 experiments, 125 raw command outputs, 22 browser captures.
Follows the import procedure in README.md.
source/ the originating repository verbatim — 78 documents, 28 SVGs,
8 manifests, plus .source-revision recording the commit
final/ the SSOT
document.md 729 lines written from the 29 experiment documents, not
concatenated: what was predicted, what was measured, and
where the measurement itself was wrong
evidence/raw 125 outputs, flattened to <experiment>__<file> because
the originals collided (01-baseline.txt appeared three
times) and the audit only globs the top level
evidence/meta one per raw file; command and exitCode are null and the
README says why rather than inventing them
evidence/browser 22 captures
assets/ three diagrams through techviz
.techviz/ their VizSpecs
A separate project rather than an addition to keycloak: the B-layer answers
that project's four questions, but the A, C and D layers are about cluster
failure, SSO and operations, and one document.md should hold one subject.
The four question records there can point here through 관계.
Recorded rather than papered over: only three of the 28 diagrams were
remade. The repository forbids hand-drawn SVG and forbids titles inside the
canvas; all 28 originals carry both, so converting them is redrawing, not
reformatting. They stay in source/ and the gap is written into the document.
verify-pipeline.py passes. audit-records.py reports no issues.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
DongHyeonka
co-authored by
Claude Opus 5
Claude Opus 5
parent
43bccd08a8
commit
b2963105a8
@@ -0,0 +1,46 @@
|
||||
# Experiment A-1 — cut the JGroups transport (TCP 7800) while leaving discovery alone.
|
||||
#
|
||||
# The point is to separate two things that are easy to conflate:
|
||||
#
|
||||
# discovery how the nodes FIND each other -> PostgreSQL JGROUPS_PING table
|
||||
# transport how they actually TALK -> TCP 7800
|
||||
#
|
||||
# Blocking only the transport produces a state that cannot happen on a single
|
||||
# node: both members stay registered in the database, so each believes the other
|
||||
# exists, yet no message gets through.
|
||||
#
|
||||
# kubectl apply -f deploy/lab/k8s/a1-block-jgroups-transport.yaml
|
||||
# kubectl -n keycloak-lab delete networkpolicy a1-block-jgroups-transport
|
||||
#
|
||||
# NetworkPolicy is an ALLOWLIST, not a firewall with deny rules. There is no way
|
||||
# to write "deny 7800". The moment a pod is selected by a policy carrying
|
||||
# policyTypes: [Ingress], every inbound port is denied unless a rule permits it.
|
||||
# So 7800 is blocked by *omission*: 8080 and 9000 are listed, 7800 is not.
|
||||
#
|
||||
# That makes the two allow rules load-bearing — get them wrong and the experiment
|
||||
# measures a dead Keycloak instead of a partitioned cluster:
|
||||
#
|
||||
# 8080 the HTTP endpoint. Traefik, the other pod's REST calls, and the probe
|
||||
# traffic all arrive here.
|
||||
# 9000 the management port: /health/started, /health/ready, /health/live and
|
||||
# /metrics. Losing it means the kubelet fails the readiness probe and
|
||||
# kills the pod — the cluster would break for the wrong reason.
|
||||
#
|
||||
# Both rules deliberately omit `from:`, which allows those ports from any source.
|
||||
# Narrowing the source is not the subject here; the 2-hop experiment already
|
||||
# established how to do that by label when it matters.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: a1-block-jgroups-transport
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: keycloak
|
||||
policyTypes: [Ingress]
|
||||
ingress:
|
||||
- ports:
|
||||
- { port: 8080, protocol: TCP } # HTTP — must stay open
|
||||
- { port: 9000, protocol: TCP } # health + metrics — must stay open
|
||||
# 7800 is absent on purpose. That is the whole experiment.
|
||||
@@ -0,0 +1,128 @@
|
||||
# Experiment B-7 — oauth2-proxy, to measure how replicas share a cookie secret
|
||||
# and what happens when it is rotated (Q1, unknown 7).
|
||||
#
|
||||
# This is a different shape of problem from the BFF. The BFF keeps state on the
|
||||
# server, so the question was "which store". oauth2-proxy keeps no server state
|
||||
# at all: the whole session rides in a cookie that is signed and encrypted with
|
||||
# --cookie-secret. So there is nothing to share and nothing to lose on restart —
|
||||
# instead, every replica must hold the *same* secret, and changing it invalidates
|
||||
# every cookie at once.
|
||||
#
|
||||
# kubectl apply -f deploy/lab/k8s/b7-oauth2-proxy.yaml
|
||||
#
|
||||
# app2.hyeonworks.com is borrowed from Grafana for the duration of this
|
||||
# experiment; the certificate only covers auth / app1 / app2, so a fourth name
|
||||
# is not available. Grafana's Ingress is restored afterwards.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: oauth2-proxy-secrets
|
||||
namespace: keycloak-lab
|
||||
type: Opaque
|
||||
stringData:
|
||||
# oauth2-proxy requires exactly 16, 24 or 32 bytes. This is the value whose
|
||||
# rotation the experiment is about.
|
||||
COOKIE_SECRET_A: "lab-cookie-secret-aaaaaaaaaaaaaa"
|
||||
COOKIE_SECRET_B: "lab-cookie-secret-bbbbbbbbbbbbbb"
|
||||
CLIENT_SECRET: proxy-lab-secret
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
# Two replicas is the point: Q1 asks how they share the secret.
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels: { app: oauth2-proxy }
|
||||
template:
|
||||
metadata:
|
||||
labels: { app: oauth2-proxy }
|
||||
spec:
|
||||
# See B-1: Kubernetes injects <SVCNAME>_PORT as a tcp:// URL and it
|
||||
# collides with ordinary configuration names.
|
||||
enableServiceLinks: false
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1
|
||||
topologyKey: kubernetes.io/hostname
|
||||
whenUnsatisfiable: ScheduleAnyway
|
||||
labelSelector:
|
||||
matchLabels: { app: oauth2-proxy }
|
||||
containers:
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.7.1
|
||||
args:
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://auth.hyeonworks.com/realms/keycloak-patterns
|
||||
- --client-id=oauth2-proxy
|
||||
- --redirect-url=https://app2.hyeonworks.com/oauth2/callback
|
||||
- --email-domain=*
|
||||
- --http-address=0.0.0.0:4180
|
||||
# The upstream is the same echo app the B-4 header experiment used,
|
||||
# so what the proxy forwards can be read straight off the response.
|
||||
- --upstream=http://echo.header-lab.svc:8081
|
||||
# ★ 이 옵션을 켜면 세션(=쿠키)에 access token 이 들어간다.
|
||||
# 그러면 Set-Cookie 가 커져 프록시 앞단에서 502 가 났다.
|
||||
# B-4 에서 본 헤더 크기 절벽이 이번에는 응답 쪽에서 나타난 것이다.
|
||||
# - --pass-authorization-header=true
|
||||
- --set-xauthrequest=true
|
||||
- --reverse-proxy=true
|
||||
- --cookie-secure=true
|
||||
# One hour, matching the value Q1 records for the current setup.
|
||||
- --cookie-expire=1h
|
||||
- --skip-provider-button=true
|
||||
# ★ 쿠키에 세션 전체를 담으면 Set-Cookie 가 커지고, 그 응답이
|
||||
# 앞단 nginx 의 proxy_buffer 를 넘겨 502 가 났다(측정됨).
|
||||
# Redis 로 옮기면 쿠키에는 티켓만 남는다 — 그리고 그 순간
|
||||
# "replica 가 secret 을 공유해야 한다"는 문제의 성격도 바뀐다.
|
||||
- --session-store-type=redis
|
||||
- --redis-connection-url=redis://redis.keycloak-lab.svc:6379
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef: { name: oauth2-proxy-secrets, key: CLIENT_SECRET }
|
||||
# Which of the two secrets is in use is switched here. Both replicas
|
||||
# read the same key, which is exactly the sharing Q1 asks about.
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef: { name: oauth2-proxy-secrets, key: COOKIE_SECRET_A }
|
||||
ports:
|
||||
- containerPort: 4180
|
||||
name: http
|
||||
readinessProbe:
|
||||
httpGet: { path: /ping, port: http }
|
||||
initialDelaySeconds: 5
|
||||
resources:
|
||||
requests: { memory: 32Mi, cpu: 20m }
|
||||
limits: { memory: 128Mi }
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
selector: { app: oauth2-proxy }
|
||||
ports:
|
||||
- port: 4180
|
||||
targetPort: http
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: app2.hyeonworks.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: oauth2-proxy
|
||||
port:
|
||||
number: 4180
|
||||
@@ -0,0 +1,214 @@
|
||||
# BFF (2 replicas) + Redis, for the B-layer experiments.
|
||||
#
|
||||
# The BFF is deployed FIRST WITHOUT any session store wiring. That is deliberate:
|
||||
# B-0 asks what Spring Boot's autoconfiguration actually picks when nothing is
|
||||
# configured, and the only honest way to answer is to look at a running instance
|
||||
# that has been given nothing. Redis is deployed alongside but left unused until
|
||||
# B-1 turns it on.
|
||||
#
|
||||
# kubectl apply -f deploy/lab/k8s/bff-redis.yaml
|
||||
#
|
||||
# Image comes from the workstation, not a registry:
|
||||
# docker build -t keycloak-pattern-bff:lab bff/
|
||||
# docker save keycloak-pattern-bff:lab | ssh test-server "ssh kc-lab-1 'sudo k3s ctr images import -'"
|
||||
# (repeat for kc-lab-2)
|
||||
# so imagePullPolicy must stay Never on both replicas.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: bff-secrets
|
||||
namespace: keycloak-lab
|
||||
type: Opaque
|
||||
stringData:
|
||||
# Matches the client created with kcadm in the keycloak-patterns realm.
|
||||
# Base64 in etcd is not encryption — see D-3.
|
||||
KEYCLOAK_CLIENT_SECRET: bff-lab-secret
|
||||
---
|
||||
# Redis. B-5 measured that turning on AOF with `redis-cli config set` changes
|
||||
# nothing here, because /data is the container filesystem and dies with the
|
||||
# container — the appendonlydir was created and then thrown away. Persistence
|
||||
# configuration without a volume is decoration.
|
||||
#
|
||||
# So the volume comes first, and only then does `--appendonly yes` mean anything.
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: redis-data
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: local-path
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: redis
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels: { app: redis }
|
||||
template:
|
||||
metadata:
|
||||
labels: { app: redis }
|
||||
spec:
|
||||
# Same node as postgres so a node-loss experiment takes both stores at
|
||||
# once, matching how A-4 was set up.
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: kc-lab-2
|
||||
containers:
|
||||
- name: redis
|
||||
image: redis:7.4-alpine
|
||||
# appendfsync everysec 이 기본값이다 — 1초 분량을 잃을 수 있다.
|
||||
# Keycloak 의 synchronous_commit OFF(A-3)와 같은 모양의 트레이드오프다.
|
||||
args: ["redis-server", "--appendonly", "yes", "--dir", "/data"]
|
||||
ports:
|
||||
- containerPort: 6379
|
||||
name: redis
|
||||
readinessProbe:
|
||||
exec: { command: ["redis-cli", "ping"] }
|
||||
initialDelaySeconds: 3
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
resources:
|
||||
requests: { memory: 32Mi, cpu: 20m }
|
||||
limits: { memory: 128Mi }
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: redis-data
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: redis
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
selector: { app: redis }
|
||||
ports:
|
||||
- port: 6379
|
||||
targetPort: redis
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: bff
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
# Two replicas is the whole point: Q1 and Q2 only exist because a request can
|
||||
# land on an instance that did not handle the login.
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels: { app: bff }
|
||||
template:
|
||||
metadata:
|
||||
labels: { app: bff }
|
||||
spec:
|
||||
# Spread across both nodes so "the other instance" is genuinely another
|
||||
# machine, not another process on the same kernel.
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1
|
||||
topologyKey: kubernetes.io/hostname
|
||||
whenUnsatisfiable: ScheduleAnyway
|
||||
labelSelector:
|
||||
matchLabels: { app: bff }
|
||||
# 쿠버네티스는 같은 네임스페이스의 Service 마다 Docker link 시절의
|
||||
# 환경변수를 자동 주입한다: REDIS_PORT=tcp://10.43.57.116:6379.
|
||||
# 그것이 application.yml 의 ${REDIS_PORT:6379} 를 덮어써서 기동이 실패했다.
|
||||
# Failed to bind properties under 'spring.data.redis.port' to int:
|
||||
# Value: "tcp://10.43.57.116:6379"
|
||||
# 이 주입 자체를 끄는 것이 근본 처방이다. 이름을 바꿔 피하면 다음 사람이
|
||||
# 같은 함정에 다시 빠진다.
|
||||
enableServiceLinks: false
|
||||
containers:
|
||||
- name: bff
|
||||
image: keycloak-pattern-bff:lab
|
||||
imagePullPolicy: Never
|
||||
ports:
|
||||
- containerPort: 8083
|
||||
name: http
|
||||
env:
|
||||
# The browser is redirected to the public name; the BFF calls the
|
||||
# token endpoint over the cluster network. Getting these two the same
|
||||
# way round is what the 2-hop header experiment was about.
|
||||
- name: KC_ISSUER_EXTERNAL
|
||||
value: https://auth.hyeonworks.com/realms/keycloak-patterns
|
||||
- name: KC_ISSUER_INTERNAL
|
||||
value: http://keycloak.keycloak-lab.svc:8080/realms/keycloak-patterns
|
||||
# echo 는 header-lab 네임스페이스의 8081 이다. 다른 네임스페이스의
|
||||
# 서비스는 <svc>.<ns>.svc 로 부른다. 이름을 틀리면 500 이 나는데
|
||||
# 원인은 UnresolvedAddressException 이지 토큰 문제가 아니다.
|
||||
- name: RESOURCE_API_BASE_URL
|
||||
value: http://echo.header-lab.svc:8081
|
||||
- name: KEYCLOAK_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef: { name: bff-secrets, key: KEYCLOAK_CLIENT_SECRET }
|
||||
# Spring needs to know it is behind TLS termination, for the same
|
||||
# reason Keycloak needs KC_PROXY_HEADERS. Without it the redirect_uri
|
||||
# it builds comes back as http:// and Keycloak rejects it.
|
||||
- name: SERVER_FORWARD_HEADERS_STRATEGY
|
||||
value: native
|
||||
# B-1: Application Session 을 Redis 로 옮긴다.
|
||||
# OAuth2AuthorizedClient 는 이것으로 옮겨지지 않는다 — 조회 키가
|
||||
# 다르기 때문이며, B-0 에서 확인한 사실이다.
|
||||
- name: SPRING_SESSION_STORE_TYPE
|
||||
value: redis
|
||||
- name: REDIS_HOST
|
||||
value: redis.keycloak-lab.svc
|
||||
- name: REDIS_PORT
|
||||
value: "6379"
|
||||
# B-2: authorized client 는 PostgreSQL 로. 세션(Redis)과 다른
|
||||
# 저장소를 쓰는 것이 Q3 가 말한 "각각 설계한다"의 실물이다.
|
||||
- name: BFF_DB_URL
|
||||
value: jdbc:postgresql://postgres.keycloak-lab.svc:5432/keycloak
|
||||
- name: BFF_DB_USER
|
||||
value: keycloak
|
||||
- name: BFF_DB_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef: { name: keycloak-lab-secrets, key: POSTGRES_PASSWORD }
|
||||
- name: JAVA_TOOL_OPTIONS
|
||||
value: "-Xms128m -Xmx320m"
|
||||
readinessProbe:
|
||||
httpGet: { path: /actuator/health/readiness, port: http }
|
||||
initialDelaySeconds: 20
|
||||
failureThreshold: 30
|
||||
livenessProbe:
|
||||
httpGet: { path: /actuator/health/liveness, port: http }
|
||||
initialDelaySeconds: 60
|
||||
resources:
|
||||
requests: { memory: 320Mi, cpu: 100m }
|
||||
limits: { memory: 512Mi }
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: bff
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
selector: { app: bff }
|
||||
ports:
|
||||
- port: 8083
|
||||
targetPort: http
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: bff
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: app1.hyeonworks.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: bff
|
||||
port:
|
||||
number: 8083
|
||||
@@ -0,0 +1,62 @@
|
||||
# Restrict who may reach the echo pods.
|
||||
#
|
||||
# Traefik is configured to trust X-Forwarded-* from the whole pod CIDR, and the
|
||||
# app's Tomcat valve trusts every private range by default. Both are IP-range
|
||||
# decisions, so any pod in the cluster can forge those headers by talking to the
|
||||
# Service directly and bypassing Traefik entirely. Measured, not hypothetical:
|
||||
#
|
||||
# kubectl -n header-lab run t --rm -i --restart=Never --image=curlimages/curl -- \
|
||||
# curl -s http://echo:8081/api/echo -H 'X-Forwarded-Host: evil.example.com'
|
||||
# → serverName evil.example.com, remoteAddr 1.2.3.4
|
||||
#
|
||||
# A NetworkPolicy closes that path. It selects by label rather than IP, so it
|
||||
# survives pod restarts and rescheduling — unlike the trustedIPs list, which
|
||||
# could not name Traefik because its IP changes.
|
||||
#
|
||||
# "Trusting forwarded headers" and "guaranteeing a proxy sits in front" are a
|
||||
# pair. Doing only the first leaves this hole.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: echo-allow-traefik-only
|
||||
namespace: header-lab
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: echo
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
# The proxy itself. namespaceSelector and podSelector in one list item are
|
||||
# ANDed, so this is "traefik pods in kube-system" and nothing else.
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: traefik
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8081
|
||||
|
||||
# kubelet readiness/liveness probes originate from the node, not from a pod,
|
||||
# so they need their own rule. Without it the probes fail and the pods are
|
||||
# restarted in a loop.
|
||||
#
|
||||
# The probe's source address is the node's flannel bridge (cni0), which
|
||||
# holds the first address of that node's /24:
|
||||
# kc-lab-1 10.42.0.1 kc-lab-2 10.42.1.1
|
||||
# Listing them as /32 keeps this rule from re-admitting arbitrary pods,
|
||||
# which a broader 10.42.0.0/16 block would do and would undo the policy.
|
||||
#
|
||||
# Adding a node means adding its gateway here. Verify with:
|
||||
# kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{end}'
|
||||
- from:
|
||||
- ipBlock:
|
||||
cidr: 10.42.0.1/32
|
||||
- ipBlock:
|
||||
cidr: 10.42.1.1/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8081
|
||||
@@ -0,0 +1,113 @@
|
||||
# Header echo workload for the two-hop proxy contract measurement.
|
||||
#
|
||||
# browser -> host nginx (TLS termination) -> Traefik -> this pod
|
||||
#
|
||||
# The image is built from backend/ and imported straight into each node's
|
||||
# containerd, so imagePullPolicy must stay Never. See scripts/build-and-import.sh.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: header-lab
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: echo
|
||||
namespace: header-lab
|
||||
spec:
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app: echo
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: echo
|
||||
spec:
|
||||
# One replica per node so the sticky-session switch on the host nginx
|
||||
# upstream has something observable to route between.
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1
|
||||
topologyKey: kubernetes.io/hostname
|
||||
whenUnsatisfiable: ScheduleAnyway
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app: echo
|
||||
containers:
|
||||
- name: echo
|
||||
image: keycloak-pattern-api:lab
|
||||
imagePullPolicy: Never
|
||||
ports:
|
||||
- containerPort: 8081
|
||||
name: http
|
||||
env:
|
||||
- name: SERVER_PORT
|
||||
value: "8081"
|
||||
# "none" makes the app report the raw connection, so scheme/secure/
|
||||
# requestUrl show what arrives without any forwarded-header handling.
|
||||
# Set to "native" and redeploy to see the same request interpreted
|
||||
# with X-Forwarded-* honoured. Keycloak's KC_PROXY_HEADERS is the
|
||||
# same opt-in, which is why measuring both sides matters here.
|
||||
- name: SERVER_FORWARD_HEADERS_STRATEGY
|
||||
value: "native"
|
||||
# The JVM sizes its heap from the container limit, not the host.
|
||||
- name: JAVA_TOOL_OPTIONS
|
||||
value: "-XX:MaxRAMPercentage=70"
|
||||
# /api/echo is permitAll, so the JWT decoder is never exercised.
|
||||
# These stay pointed at the future Keycloak service name.
|
||||
- name: SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI
|
||||
value: "https://auth.hyeonworks.com/realms/keycloak-patterns"
|
||||
- name: SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI
|
||||
value: "https://auth.hyeonworks.com/realms/keycloak-patterns/protocol/openid-connect/certs"
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /actuator/health/readiness
|
||||
port: http
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 5
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /actuator/health/liveness
|
||||
port: http
|
||||
initialDelaySeconds: 45
|
||||
periodSeconds: 15
|
||||
resources:
|
||||
requests:
|
||||
memory: 320Mi
|
||||
cpu: 100m
|
||||
limits:
|
||||
memory: 512Mi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: echo
|
||||
namespace: header-lab
|
||||
spec:
|
||||
selector:
|
||||
app: echo
|
||||
ports:
|
||||
- port: 8081
|
||||
targetPort: http
|
||||
name: http
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: echo
|
||||
namespace: header-lab
|
||||
spec:
|
||||
# k3s ships Traefik as the default ingress controller. Keeping it is what
|
||||
# makes this lab a faithful two-hop replica.
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: app1.hyeonworks.com
|
||||
http:
|
||||
paths:
|
||||
- path: /api
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: echo
|
||||
port:
|
||||
number: 8081
|
||||
@@ -0,0 +1,277 @@
|
||||
# Keycloak multi-node cluster with PostgreSQL.
|
||||
#
|
||||
# Goal of this manifest: two Keycloak pods on two different nodes must discover
|
||||
# each other and form one Infinispan cluster. Keycloak 26 discovers peers through
|
||||
# the database (jdbc-ping) rather than multicast, writing to a JGROUPS_PING table,
|
||||
# but the cluster traffic itself runs over TCP 7800 between the pods. Those are
|
||||
# two separate mechanisms, which is why "registered in the DB but not clustered"
|
||||
# is a real failure mode — and one that a single node cannot reproduce.
|
||||
#
|
||||
# kubectl apply -f deploy/lab/k8s/keycloak-cluster.yaml
|
||||
# kubectl -n keycloak-lab rollout status statefulset/keycloak --timeout=600s
|
||||
#
|
||||
# Secrets are plain here. Proper secret handling is roadmap item 11; keeping it
|
||||
# visible for now is deliberate so the gap is obvious rather than forgotten.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: keycloak-lab
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: keycloak-lab-secrets
|
||||
namespace: keycloak-lab
|
||||
type: Opaque
|
||||
stringData:
|
||||
POSTGRES_PASSWORD: lab-postgres-change-me
|
||||
KC_BOOTSTRAP_ADMIN_PASSWORD: lab-admin-change-me
|
||||
---
|
||||
# PostgreSQL. local-path binds the volume to whichever node the pod lands on, so
|
||||
# the database is effectively pinned to one node. That is not a flaw here: it is
|
||||
# what makes "the database node dies" a meaningful experiment later.
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: postgres-data
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: local-path
|
||||
resources:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: postgres
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate # RWO volume cannot be mounted by two pods at once
|
||||
selector:
|
||||
matchLabels:
|
||||
app: postgres
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: postgres
|
||||
spec:
|
||||
containers:
|
||||
- name: postgres
|
||||
image: postgres:16-alpine
|
||||
ports:
|
||||
- containerPort: 5432
|
||||
name: postgres
|
||||
env:
|
||||
- name: POSTGRES_DB
|
||||
value: keycloak
|
||||
- name: POSTGRES_USER
|
||||
value: keycloak
|
||||
- name: POSTGRES_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: keycloak-lab-secrets
|
||||
key: POSTGRES_PASSWORD
|
||||
# The image refuses to initialise into a non-empty mount, and
|
||||
# local-path volumes are clean, but this keeps the data one level
|
||||
# down so a lost+found or similar never blocks initdb.
|
||||
- name: PGDATA
|
||||
value: /var/lib/postgresql/data/pgdata
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /var/lib/postgresql/data
|
||||
readinessProbe:
|
||||
exec:
|
||||
command: ["sh", "-c", "pg_isready -U keycloak -d keycloak"]
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 5
|
||||
resources:
|
||||
requests:
|
||||
memory: 192Mi
|
||||
cpu: 50m
|
||||
limits:
|
||||
memory: 512Mi
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: postgres-data
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: postgres
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
selector:
|
||||
app: postgres
|
||||
ports:
|
||||
- port: 5432
|
||||
targetPort: postgres
|
||||
---
|
||||
# Keycloak. A StatefulSet rather than a Deployment so each pod keeps a stable
|
||||
# name (keycloak-0, keycloak-1); cluster membership is far easier to read in
|
||||
# logs and in the JGROUPS_PING table when the identities do not churn.
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: keycloak
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
serviceName: keycloak-headless
|
||||
replicas: 2
|
||||
podManagementPolicy: Parallel # both pods start together, so they race to
|
||||
# register — which is the interesting case
|
||||
selector:
|
||||
matchLabels:
|
||||
app: keycloak
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: keycloak
|
||||
spec:
|
||||
# One pod per node. Two pods on one node would share a kernel and make the
|
||||
# 7800 blocking experiment meaningless.
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1
|
||||
topologyKey: kubernetes.io/hostname
|
||||
whenUnsatisfiable: ScheduleAnyway
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app: keycloak
|
||||
containers:
|
||||
- name: keycloak
|
||||
image: quay.io/keycloak/keycloak:26.7.0
|
||||
# "start", not "start-dev". Dev mode forces cache=local and there is
|
||||
# no cluster to form at all.
|
||||
args: ["start"]
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
name: http
|
||||
- containerPort: 9000
|
||||
name: management
|
||||
- containerPort: 7800
|
||||
name: jgroups
|
||||
env:
|
||||
- name: KC_DB
|
||||
value: postgres
|
||||
- name: KC_DB_URL
|
||||
value: jdbc:postgresql://postgres:5432/keycloak
|
||||
- name: KC_DB_USERNAME
|
||||
value: keycloak
|
||||
- name: KC_DB_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: keycloak-lab-secrets
|
||||
key: POSTGRES_PASSWORD
|
||||
|
||||
# Settings confirmed by the two-hop header measurement.
|
||||
# KC_HOSTNAME carries the full external URL, which pins scheme and
|
||||
# host for issuer and redirect URLs regardless of headers.
|
||||
# KC_PROXY_HEADERS is the separate opt-in that lets the forwarded
|
||||
# client address through — the same kind of switch as Spring's
|
||||
# forward-headers-strategy. See docs/two-hop-proxy-header-contract.md.
|
||||
- name: KC_HOSTNAME
|
||||
value: https://auth.hyeonworks.com
|
||||
- name: KC_HOSTNAME_STRICT
|
||||
value: "true"
|
||||
- name: KC_PROXY_HEADERS
|
||||
value: xforwarded
|
||||
- name: KC_HTTP_ENABLED
|
||||
value: "true"
|
||||
|
||||
- name: KC_HEALTH_ENABLED
|
||||
value: "true"
|
||||
- name: KC_METRICS_ENABLED
|
||||
value: "true"
|
||||
|
||||
# Without an explicit cap the JVM sizes its heap from the container
|
||||
# limit and this lab has roughly 3.8GB of guest headroom in total.
|
||||
- name: JAVA_OPTS_KC_HEAP
|
||||
value: "-Xms256m -Xmx512m"
|
||||
|
||||
- name: KC_BOOTSTRAP_ADMIN_USERNAME
|
||||
value: admin
|
||||
- name: KC_BOOTSTRAP_ADMIN_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: keycloak-lab-secrets
|
||||
key: KC_BOOTSTRAP_ADMIN_PASSWORD
|
||||
|
||||
# Keycloak serves health and metrics on the management port (9000),
|
||||
# not on 8080, since version 25.
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /health/started
|
||||
port: management
|
||||
periodSeconds: 10
|
||||
failureThreshold: 60 # first boot runs an implicit build
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health/ready
|
||||
port: management
|
||||
periodSeconds: 10
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health/live
|
||||
port: management
|
||||
periodSeconds: 30
|
||||
resources:
|
||||
requests:
|
||||
memory: 640Mi
|
||||
cpu: 100m
|
||||
limits:
|
||||
memory: 900Mi
|
||||
---
|
||||
# Headless service. Not required for jdbc-ping discovery, which goes through the
|
||||
# database, but it gives each pod a stable DNS name for direct inspection.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: keycloak-headless
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
clusterIP: None
|
||||
selector:
|
||||
app: keycloak
|
||||
ports:
|
||||
- port: 8080
|
||||
targetPort: http
|
||||
name: http
|
||||
- port: 9000
|
||||
targetPort: management
|
||||
name: management
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: keycloak
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
selector:
|
||||
app: keycloak
|
||||
ports:
|
||||
- port: 8080
|
||||
targetPort: http
|
||||
name: http
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: keycloak
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: auth.hyeonworks.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: keycloak
|
||||
port:
|
||||
number: 8080
|
||||
@@ -0,0 +1,373 @@
|
||||
# Prometheus + node-exporter + Grafana.
|
||||
#
|
||||
# Purpose: during a fault-injection experiment, know *which signal moved first*.
|
||||
# Without a metrics store the only record is whatever scrolled past in a terminal,
|
||||
# and "the cluster recovered in about a minute" is not a measurement.
|
||||
#
|
||||
# kubectl apply -f deploy/lab/k8s/observability.yaml
|
||||
# kubectl -n observability rollout status deployment/prometheus --timeout=300s
|
||||
#
|
||||
# Placement decision — Prometheus and Grafana are pinned to the control-plane
|
||||
# node (kc-lab-1). An observability stack must not share a failure domain with
|
||||
# the thing it observes. With only two nodes that cannot be fully avoided, so the
|
||||
# rule here is: the node that gets killed in experiments is the *agent*
|
||||
# (kc-lab-2, holding keycloak-0 and postgres), and everything needed to watch
|
||||
# that happen lives on the server node.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: observability
|
||||
---
|
||||
# Prometheus discovers scrape targets by querying the Kubernetes API, so it
|
||||
# needs read access to nodes, services, endpoints and pods. Without this the
|
||||
# kubernetes_sd_configs below silently return no targets.
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: prometheus
|
||||
namespace: observability
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: prometheus
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
# nodes/proxy is required in addition to nodes/metrics: the kubelet job
|
||||
# reaches each node through the API server's proxy subresource
|
||||
# (/api/v1/nodes/<name>/proxy/metrics). Without it every kubelet target
|
||||
# fails with 403 Forbidden while the other jobs stay green — a partial
|
||||
# failure that is easy to miss unless the target list is checked.
|
||||
resources: [nodes, nodes/metrics, nodes/proxy, services, endpoints, pods]
|
||||
verbs: [get, list, watch]
|
||||
- nonResourceURLs: ["/metrics"]
|
||||
verbs: [get]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: prometheus
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: prometheus
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: prometheus
|
||||
namespace: observability
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: prometheus-config
|
||||
namespace: observability
|
||||
data:
|
||||
prometheus.yml: |
|
||||
global:
|
||||
# 15s is short for production but right here: a node loss should show up
|
||||
# within a couple of samples, not a minute later.
|
||||
scrape_interval: 15s
|
||||
evaluation_interval: 15s
|
||||
|
||||
scrape_configs:
|
||||
# Prometheus scraping itself. Useful as a control: if this target is down,
|
||||
# the problem is Prometheus, not the thing being measured.
|
||||
- job_name: prometheus
|
||||
static_configs:
|
||||
- targets: ['localhost:9090']
|
||||
|
||||
# Keycloak. Metrics live on the management port 9000, not 8080 — the same
|
||||
# split that the health probes use. KC_METRICS_ENABLED=true is already set
|
||||
# on the StatefulSet.
|
||||
#
|
||||
# Discovery is by endpoints rather than a static list because pod IPs
|
||||
# change on every restart; that was observed directly when the lab was
|
||||
# power-cycled and every pod came back with a new address.
|
||||
- job_name: keycloak
|
||||
kubernetes_sd_configs:
|
||||
- role: endpoints
|
||||
namespaces:
|
||||
names: [keycloak-lab]
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
|
||||
action: keep
|
||||
regex: keycloak-headless;management
|
||||
- source_labels: [__meta_kubernetes_pod_name]
|
||||
target_label: pod
|
||||
- source_labels: [__meta_kubernetes_pod_node_name]
|
||||
target_label: node
|
||||
|
||||
# node-exporter, one per node via DaemonSet. This is what answers
|
||||
# "did the machine die or did the process die".
|
||||
- job_name: node-exporter
|
||||
kubernetes_sd_configs:
|
||||
- role: endpoints
|
||||
namespaces:
|
||||
names: [observability]
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_service_name]
|
||||
action: keep
|
||||
regex: node-exporter
|
||||
- source_labels: [__meta_kubernetes_pod_node_name]
|
||||
target_label: node
|
||||
|
||||
# The kubelet's own metrics, reached through the API server proxy so no
|
||||
# extra port needs opening.
|
||||
- job_name: kubelet
|
||||
scheme: https
|
||||
tls_config:
|
||||
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
||||
insecure_skip_verify: true
|
||||
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
||||
kubernetes_sd_configs:
|
||||
- role: node
|
||||
relabel_configs:
|
||||
- action: labelmap
|
||||
regex: __meta_kubernetes_node_label_(.+)
|
||||
- target_label: __address__
|
||||
replacement: kubernetes.default.svc:443
|
||||
- source_labels: [__meta_kubernetes_node_name]
|
||||
regex: (.+)
|
||||
target_label: __metrics_path__
|
||||
replacement: /api/v1/nodes/${1}/proxy/metrics
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: prometheus-data
|
||||
namespace: observability
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: local-path
|
||||
resources:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: prometheus
|
||||
namespace: observability
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate # RWO volume; two pods cannot mount it at once
|
||||
selector:
|
||||
matchLabels:
|
||||
app: prometheus
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: prometheus
|
||||
spec:
|
||||
serviceAccountName: prometheus
|
||||
# See the placement note at the top of this file.
|
||||
nodeSelector:
|
||||
node-role.kubernetes.io/control-plane: "true"
|
||||
securityContext:
|
||||
fsGroup: 65534 # the image runs as nobody and must own the volume
|
||||
containers:
|
||||
- name: prometheus
|
||||
image: prom/prometheus:v3.1.0
|
||||
args:
|
||||
- --config.file=/etc/prometheus/prometheus.yml
|
||||
- --storage.tsdb.path=/prometheus
|
||||
# 7 days is far more than an experiment needs and keeps the volume
|
||||
# small enough that it never becomes the reason a node fills up.
|
||||
- --storage.tsdb.retention.time=7d
|
||||
- --web.enable-lifecycle
|
||||
ports:
|
||||
- containerPort: 9090
|
||||
name: http
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/prometheus
|
||||
- name: data
|
||||
mountPath: /prometheus
|
||||
readinessProbe:
|
||||
httpGet: { path: /-/ready, port: http }
|
||||
initialDelaySeconds: 10
|
||||
livenessProbe:
|
||||
httpGet: { path: /-/healthy, port: http }
|
||||
initialDelaySeconds: 30
|
||||
resources:
|
||||
requests: { memory: 256Mi, cpu: 50m }
|
||||
limits: { memory: 640Mi }
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: prometheus-config
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: prometheus-data
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: prometheus
|
||||
namespace: observability
|
||||
spec:
|
||||
selector:
|
||||
app: prometheus
|
||||
ports:
|
||||
- port: 9090
|
||||
targetPort: http
|
||||
---
|
||||
# node-exporter. A DaemonSet so every node reports, including one that is about
|
||||
# to be killed — the last samples before it goes silent are the interesting part.
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: node-exporter
|
||||
namespace: observability
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: node-exporter
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: node-exporter
|
||||
spec:
|
||||
# Host namespaces: the point is to measure the machine, not the container.
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
tolerations:
|
||||
- operator: Exists # must also run on tainted nodes
|
||||
containers:
|
||||
- name: node-exporter
|
||||
image: prom/node-exporter:v1.8.2
|
||||
args:
|
||||
- --path.procfs=/host/proc
|
||||
- --path.sysfs=/host/sys
|
||||
- --path.rootfs=/host/root
|
||||
- --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
|
||||
ports:
|
||||
- containerPort: 9100
|
||||
name: metrics
|
||||
hostPort: 9100
|
||||
volumeMounts:
|
||||
- { name: proc, mountPath: /host/proc, readOnly: true }
|
||||
- { name: sys, mountPath: /host/sys, readOnly: true }
|
||||
- { name: rootfs, mountPath: /host/root, readOnly: true, mountPropagation: HostToContainer }
|
||||
resources:
|
||||
requests: { memory: 32Mi, cpu: 20m }
|
||||
limits: { memory: 96Mi }
|
||||
volumes:
|
||||
- { name: proc, hostPath: { path: /proc } }
|
||||
- { name: sys, hostPath: { path: /sys } }
|
||||
- { name: rootfs, hostPath: { path: / } }
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: node-exporter
|
||||
namespace: observability
|
||||
spec:
|
||||
clusterIP: None # headless: Prometheus wants each pod, not a VIP
|
||||
selector:
|
||||
app: node-exporter
|
||||
ports:
|
||||
- port: 9100
|
||||
targetPort: metrics
|
||||
name: metrics
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: grafana
|
||||
namespace: observability
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: grafana
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: grafana
|
||||
spec:
|
||||
nodeSelector:
|
||||
node-role.kubernetes.io/control-plane: "true"
|
||||
containers:
|
||||
- name: grafana
|
||||
image: grafana/grafana:11.4.0
|
||||
ports:
|
||||
- containerPort: 3000
|
||||
name: http
|
||||
env:
|
||||
- name: GF_SECURITY_ADMIN_USER
|
||||
value: admin
|
||||
- name: GF_SECURITY_ADMIN_PASSWORD
|
||||
value: lab-grafana-change-me
|
||||
# Grafana builds absolute URLs for redirects and asset paths. Behind
|
||||
# the nginx -> Traefik chain it must be told the external address,
|
||||
# for exactly the reason Keycloak needs KC_HOSTNAME. Without it,
|
||||
# login redirects come back as http://<pod-ip>:3000.
|
||||
- name: GF_SERVER_ROOT_URL
|
||||
value: https://app2.hyeonworks.com
|
||||
volumeMounts:
|
||||
- name: datasources
|
||||
mountPath: /etc/grafana/provisioning/datasources
|
||||
readinessProbe:
|
||||
httpGet: { path: /api/health, port: http }
|
||||
initialDelaySeconds: 15
|
||||
resources:
|
||||
requests: { memory: 128Mi, cpu: 50m }
|
||||
limits: { memory: 320Mi }
|
||||
volumes:
|
||||
- name: datasources
|
||||
configMap:
|
||||
name: grafana-datasources
|
||||
---
|
||||
# Provisioning the datasource as a file means Grafana comes up already wired to
|
||||
# Prometheus. Clicking through the UI would leave the configuration only in
|
||||
# Grafana's own database, which is emptyDir here and disappears on restart.
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: grafana-datasources
|
||||
namespace: observability
|
||||
data:
|
||||
prometheus.yaml: |
|
||||
apiVersion: 1
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://prometheus.observability.svc:9090
|
||||
isDefault: true
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: grafana
|
||||
namespace: observability
|
||||
spec:
|
||||
selector:
|
||||
app: grafana
|
||||
ports:
|
||||
- port: 3000
|
||||
targetPort: http
|
||||
---
|
||||
# Grafana is published on app2.hyeonworks.com because that name is already in
|
||||
# the wildcard-free certificate (auth / app1 / app2) and is otherwise unused.
|
||||
# It moves when app2 is needed for the SSO experiment.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: grafana
|
||||
namespace: observability
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: app2.hyeonworks.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: grafana
|
||||
port:
|
||||
number: 3000
|
||||
@@ -0,0 +1,43 @@
|
||||
# Make Traefik trust the X-Forwarded-* headers that the host nginx sets.
|
||||
#
|
||||
# Without this, Traefik rewrites every forwarded header from its own connection,
|
||||
# which is plain HTTP on port 80. The application then sees scheme=http even
|
||||
# though the browser connected over TLS. See docs/two-hop-proxy-header-contract.md.
|
||||
#
|
||||
# k3s installs Traefik through its bundled HelmChart, so values are overridden
|
||||
# with a HelmChartConfig rather than by editing the deployment. k3s reconciles
|
||||
# the chart and recreates the Traefik pod.
|
||||
#
|
||||
# kubectl apply -f deploy/lab/k8s/traefik-forwarded-headers.yaml
|
||||
# kubectl -n kube-system rollout status deploy/traefik --timeout=180s
|
||||
apiVersion: helm.cattle.io/v1
|
||||
kind: HelmChartConfig
|
||||
metadata:
|
||||
name: traefik
|
||||
namespace: kube-system
|
||||
spec:
|
||||
valuesContent: |-
|
||||
ports:
|
||||
web:
|
||||
forwardedHeaders:
|
||||
# Requests arriving from these sources keep their existing
|
||||
# X-Forwarded-* values instead of having them rewritten.
|
||||
#
|
||||
# 10.42.0.0/16 is the pod CIDR. It is required because the traefik
|
||||
# Service uses externalTrafficPolicy: Cluster, so svclb SNATs the
|
||||
# traffic and Traefik sees a pod-network address rather than the
|
||||
# host nginx address.
|
||||
#
|
||||
# The node/host range is deliberately absent. Because svclb SNATs,
|
||||
# the host nginx address never reaches Traefik — measured, not assumed.
|
||||
# Trusting a range that cannot appear only widens the surface.
|
||||
#
|
||||
# Trusting the whole pod CIDR still means any pod in the cluster could
|
||||
# forge these headers, which is why echo-network-policy.yaml restricts
|
||||
# who may reach the application at all.
|
||||
trustedIPs:
|
||||
- 10.42.0.0/16
|
||||
websecure:
|
||||
forwardedHeaders:
|
||||
trustedIPs:
|
||||
- 10.42.0.0/16
|
||||
Reference in New Issue
Block a user