docs(keycloak-session-store): import the session-storage lab as a new project
The keycloak project ended with four open questions that design could not
settle. A two-VM lab was built to answer them by measurement, and this is
that material: 26 experiments, 125 raw command outputs, 22 browser captures.
Follows the import procedure in README.md.
source/ the originating repository verbatim — 78 documents, 28 SVGs,
8 manifests, plus .source-revision recording the commit
final/ the SSOT
document.md 729 lines written from the 29 experiment documents, not
concatenated: what was predicted, what was measured, and
where the measurement itself was wrong
evidence/raw 125 outputs, flattened to <experiment>__<file> because
the originals collided (01-baseline.txt appeared three
times) and the audit only globs the top level
evidence/meta one per raw file; command and exitCode are null and the
README says why rather than inventing them
evidence/browser 22 captures
assets/ three diagrams through techviz
.techviz/ their VizSpecs
A separate project rather than an addition to keycloak: the B-layer answers
that project's four questions, but the A, C and D layers are about cluster
failure, SSO and operations, and one document.md should hold one subject.
The four question records there can point here through 관계.
Recorded rather than papered over: only three of the 28 diagrams were
remade. The repository forbids hand-drawn SVG and forbids titles inside the
canvas; all 28 originals carry both, so converting them is redrawing, not
reformatting. They stay in source/ and the gap is written into the document.
verify-pipeline.py passes. audit-records.py reports no issues.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
DongHyeonka
co-authored by
Claude Opus 5
Claude Opus 5
parent
43bccd08a8
commit
b2963105a8
@@ -0,0 +1 @@
|
||||
cdac9b8178391311d8eca1ebc6cac15bb62d79af
|
||||
@@ -0,0 +1,136 @@
|
||||
# Session store lab
|
||||
|
||||
세션 저장소·refresh token 경쟁·장애 복구를 검증하는 2노드 k3s 실험대.
|
||||
네 인증 패턴(AP1~AP4)을 가로지르는 공통층이므로 별도 축으로 관리한다.
|
||||
|
||||
이 문서는 **절차**만 담는다.
|
||||
|
||||
| 문서 | 내용 |
|
||||
|---|---|
|
||||
| [`docs/session-store-lab-roadmap.md`](../../docs/session-store-lab-roadmap.md) | 이 축의 계획과 진행 상황 |
|
||||
| [`docs/session-lab-concepts.md`](../../docs/session-lab-concepts.md) | 등장 개념 전체 |
|
||||
| [`docs/session-lab-operations.md`](../../docs/session-lab-operations.md) | 관측 도구 · 자주 쓰는 명령 · 훈련 |
|
||||
| [`docs/two-hop-proxy-header-contract.md`](../../docs/two-hop-proxy-header-contract.md) | 첫 실험 결과 |
|
||||
|
||||
## 토폴로지
|
||||
|
||||
```
|
||||
브라우저 / SSH (tailnet)
|
||||
│ https://{auth,app1,app2}.hyeonworks.com → 100.83.212.4
|
||||
▼
|
||||
lab host ── nginx :443 TLS 종료 · X-Forwarded-* 주입
|
||||
│ nginx :80 301 → https
|
||||
│
|
||||
│ virbr0 192.168.122.0/24 (libvirt NAT)
|
||||
├──▶ kc-lab-1 .11 k3s server Traefik :80
|
||||
└──▶ kc-lab-2 .12 k3s agent Traefik :80
|
||||
└──▶ Pod
|
||||
```
|
||||
|
||||
`nginx → Traefik` **2홉**이 운영 구조와 같다는 점이 이 배치의 핵심이다.
|
||||
L7 프록시가 두 겹인 이유는 역할이 다르기 때문이다 — nginx는 바깥세상과의
|
||||
접점(TLS·인증서·헤더)을, Traefik은 클러스터 내부의 동적 라우팅을 맡는다.
|
||||
|
||||
## 구성 요소
|
||||
|
||||
| 경로 | 역할 |
|
||||
|---|---|
|
||||
| `cloud-init/kc-lab.yaml.example` | 게스트 부트스트랩 템플릿 |
|
||||
| `host/nginx-keycloak-lab.conf` | lab host의 `sites-available/keycloak-lab` |
|
||||
| `k8s/echo.yaml` | 2홉 헤더 계약 측정용 워크로드 |
|
||||
| `scripts/rebuild-seed.sh` | cloud-init 시드 ISO 재생성 + 풀 업로드 |
|
||||
| `scripts/build-and-import.sh` | 이미지 빌드 → 각 노드 containerd 반입 |
|
||||
| `scripts/measure-proxy-headers.sh` | 헤더 계약 실측 |
|
||||
| `scripts/verify-lab.sh` | 인프라 상태 점검 |
|
||||
|
||||
## 상태 점검
|
||||
|
||||
```bash
|
||||
./deploy/lab/scripts/verify-lab.sh # lab host 에서
|
||||
```
|
||||
|
||||
**`404`가 성공 신호다.** TLS가 종료되고 Traefik까지 도달했으나 매칭되는
|
||||
Ingress 규칙이 없다는 뜻이다. `502`나 연결 거부면 체인이 끊긴 것이다.
|
||||
|
||||
## 첫 실험 — 2홉 헤더 계약
|
||||
|
||||
[`docs/reverse-proxy-headers.md`](../../docs/reverse-proxy-headers.md)의 계약은
|
||||
nginx **1홉**을 가정하고 쓰였다. 실제 배치는 2홉이므로, nginx가 세팅한
|
||||
`X-Forwarded-*`를 Traefik이 그대로 넘기는지 덮어쓰는지 **측정해서 확인한다.**
|
||||
|
||||
이 결론이 뒤의 모든 실험에 깔린다. Keycloak의 `iss` 클레임, redirect URL,
|
||||
쿠키 도메인 검증이 전부 이 헤더에 의존하기 때문이다.
|
||||
|
||||
```bash
|
||||
# 워크스테이션: 이미지 빌드 후 두 노드에 반입
|
||||
./deploy/lab/scripts/build-and-import.sh
|
||||
|
||||
# lab host: 배포
|
||||
kubectl apply -f deploy/lab/k8s/echo.yaml
|
||||
kubectl -n header-lab rollout status deployment/echo
|
||||
|
||||
# 어디서든: 실측
|
||||
./deploy/lab/scripts/measure-proxy-headers.sh
|
||||
```
|
||||
|
||||
관측 대상은 넷이다.
|
||||
|
||||
1. `X-Forwarded-For` — Traefik이 **덧붙이는가 덮어쓰는가**
|
||||
2. `X-Forwarded-Proto` / `-Host` / `-Port` — 그대로 전달되는가
|
||||
3. **위조 내성** — 클라이언트가 직접 넣은 `X-Forwarded-*`가 앱까지 도달하는가
|
||||
4. `scheme` / `secure` / `requestUrl` — Keycloak이 URL을 만들 때 쓰는 값
|
||||
|
||||
3번이 신뢰 경계의 핵심이다. 이 헤더들은 누구나 위조할 수 있는 평범한 HTTP
|
||||
헤더이므로, 신뢰 경계에 선 프록시가 **반드시 덮어써야** 한다.
|
||||
|
||||
## 이미지 배포 경로
|
||||
|
||||
k3s는 containerd를 쓰고 이 실험대에는 레지스트리가 없다.
|
||||
|
||||
```
|
||||
워크스테이션 docker build → docker save
|
||||
│ ssh (lab host 경유)
|
||||
▼
|
||||
게스트 sudo k3s ctr images import
|
||||
매니페스트 imagePullPolicy: Never
|
||||
```
|
||||
|
||||
**두 노드 모두에 반입해야 한다.** 스케줄러가 어느 노드에 배치할지 모른다.
|
||||
Keycloak·PostgreSQL·Redis는 공식 이미지를 그대로 당겨오므로 이 경로가
|
||||
필요한 것은 자체 빌드 이미지뿐이다.
|
||||
|
||||
**lab host에 Docker를 설치하지 않는다.** k3s의 containerd와 이미지 저장소가
|
||||
갈려서 `docker build`한 이미지를 k3s가 보지 못하게 된다.
|
||||
|
||||
## 게스트 재생성
|
||||
|
||||
파괴적 실험 후 초기화하는 경로다.
|
||||
|
||||
```bash
|
||||
virsh destroy kc-lab-1
|
||||
virsh undefine kc-lab-1 # --remove-all-storage 는 시드 ISO 까지 지운다
|
||||
virsh vol-delete --pool default kc-lab-1.qcow2
|
||||
|
||||
./deploy/lab/scripts/rebuild-seed.sh 1 # user-data 를 고쳤을 때만
|
||||
|
||||
virt-install --name kc-lab-1 --memory 3584 --vcpus 2 \
|
||||
--disk size=20,backing_store=/var/lib/libvirt/images/base.qcow2 \
|
||||
--disk vol=default/seed-kc-lab-1.iso,device=disk,bus=virtio,readonly=on \
|
||||
--network network=default,mac=52:54:00:aa:bb:11 \
|
||||
--import --os-variant debian12 --noautoconsole
|
||||
```
|
||||
|
||||
시드는 **virtio 디스크**로 붙인다. `virt-install --cloud-init`은 시드를 SATA
|
||||
CD-ROM으로 붙이는데, Debian `genericcloud` 이미지는 크기를 줄이려고 물리
|
||||
하드웨어 드라이버를 제외해서 **AHCI 장치를 보지 못한다.** 그러면 cloud-init이
|
||||
데이터소스를 찾지 못하고 아무 오류도 남기지 않은 채 종료한다. 증상은
|
||||
hostname이 `localhost`로 남고 SSH가 `Permission denied (publickey)`로 거부되는
|
||||
것뿐이다.
|
||||
|
||||
게스트에 들어갈 수 없을 때는 화면을 직접 뜬다.
|
||||
|
||||
```bash
|
||||
virsh screenshot kc-lab-1 /tmp/kc1.ppm # 확장자와 무관하게 PNG 로 저장된다
|
||||
```
|
||||
|
||||
`localhost login:`이면 cloud-init 미실행, `kc-lab-1 login:`이면 실행된 것이다.
|
||||
@@ -0,0 +1,37 @@
|
||||
#cloud-config
|
||||
# Template for both lab guests. scripts/rebuild-seed.sh substitutes __NODE__
|
||||
# and bakes this into a CIDATA seed image.
|
||||
#
|
||||
# Copy to kc-lab.yaml and fill the two placeholders. The real file is ignored by
|
||||
# git because plain_text_passwd is a credential, however disposable.
|
||||
#
|
||||
# Indentation is spaces only. YAML forbids tabs, and cloud-init fails silently
|
||||
# on a parse error: the guest boots as "localhost" with no user and no way in.
|
||||
hostname: kc-lab-__NODE__
|
||||
fqdn: kc-lab-__NODE__
|
||||
manage_etc_hosts: true
|
||||
|
||||
users:
|
||||
- name: donghyeon
|
||||
groups: [sudo]
|
||||
shell: /bin/bash
|
||||
# NOPASSWD is required: the k3s installer and the fault-injection scripts
|
||||
# run non-interactively and would block on a password prompt.
|
||||
sudo: ['ALL=(ALL) NOPASSWD:ALL']
|
||||
# Console-only escape hatch. Without it, a cloud-init failure leaves a guest
|
||||
# that cannot be logged into at all, so its own failure log is unreadable.
|
||||
# ssh_pwauth stays false, so this never widens SSH exposure.
|
||||
lock_passwd: false
|
||||
plain_text_passwd: CHANGE_ME
|
||||
ssh_authorized_keys:
|
||||
# Lab host key: needed because automation runs from the lab host, where
|
||||
# agent forwarding is not available.
|
||||
- CHANGE_ME_LAB_HOST_PUBLIC_KEY
|
||||
# Workstation key: lets ProxyJump reach the guest directly.
|
||||
- CHANGE_ME_WORKSTATION_PUBLIC_KEY
|
||||
|
||||
ssh_pwauth: false
|
||||
package_update: true
|
||||
packages:
|
||||
- curl
|
||||
- nftables
|
||||
@@ -0,0 +1,56 @@
|
||||
# Lab entry point. Deployed on the lab host as
|
||||
# /etc/nginx/sites-available/keycloak-lab
|
||||
# and symlinked from sites-enabled/.
|
||||
#
|
||||
# Arch does not ship the Debian sites-available convention, so nginx.conf needs
|
||||
# include /etc/nginx/sites-enabled/*;
|
||||
# inside its http { } block before this file has any effect.
|
||||
#
|
||||
# This is the outer of two L7 hops. It terminates TLS and hands plain HTTP to
|
||||
# the Traefik instance running on each k3s node.
|
||||
|
||||
upstream k3s_traefik {
|
||||
# Sticky-session switch. Keycloak recommends affinity on AUTH_SESSION_ID;
|
||||
# ip_hash is the cheap stand-in for a single-browser lab. Leaving it off is
|
||||
# the interesting case: Infinispan still routes correctly, only slower.
|
||||
# ip_hash;
|
||||
server 192.168.122.11:80;
|
||||
server 192.168.122.12:80;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 80 default_server;
|
||||
server_name _;
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl default_server;
|
||||
http2 on;
|
||||
server_name _;
|
||||
|
||||
# fullchain.pem, never cert.pem: omitting the intermediates passes on
|
||||
# desktop browsers and fails on mobile and curl.
|
||||
ssl_certificate /etc/letsencrypt/live/auth.hyeonworks.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/auth.hyeonworks.com/privkey.pem;
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
|
||||
location / {
|
||||
proxy_pass http://k3s_traefik;
|
||||
proxy_http_version 1.1;
|
||||
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-Host $host;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
proxy_set_header X-Forwarded-Port 443;
|
||||
|
||||
# $remote_addr, not $proxy_add_x_forwarded_for. This is the trust
|
||||
# boundary: a client-supplied X-Forwarded-For must be discarded, not
|
||||
# extended, or nothing downstream can rely on the value.
|
||||
proxy_set_header X-Forwarded-For $remote_addr;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_send_timeout 3600s;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
# Experiment A-1 — cut the JGroups transport (TCP 7800) while leaving discovery alone.
|
||||
#
|
||||
# The point is to separate two things that are easy to conflate:
|
||||
#
|
||||
# discovery how the nodes FIND each other -> PostgreSQL JGROUPS_PING table
|
||||
# transport how they actually TALK -> TCP 7800
|
||||
#
|
||||
# Blocking only the transport produces a state that cannot happen on a single
|
||||
# node: both members stay registered in the database, so each believes the other
|
||||
# exists, yet no message gets through.
|
||||
#
|
||||
# kubectl apply -f deploy/lab/k8s/a1-block-jgroups-transport.yaml
|
||||
# kubectl -n keycloak-lab delete networkpolicy a1-block-jgroups-transport
|
||||
#
|
||||
# NetworkPolicy is an ALLOWLIST, not a firewall with deny rules. There is no way
|
||||
# to write "deny 7800". The moment a pod is selected by a policy carrying
|
||||
# policyTypes: [Ingress], every inbound port is denied unless a rule permits it.
|
||||
# So 7800 is blocked by *omission*: 8080 and 9000 are listed, 7800 is not.
|
||||
#
|
||||
# That makes the two allow rules load-bearing — get them wrong and the experiment
|
||||
# measures a dead Keycloak instead of a partitioned cluster:
|
||||
#
|
||||
# 8080 the HTTP endpoint. Traefik, the other pod's REST calls, and the probe
|
||||
# traffic all arrive here.
|
||||
# 9000 the management port: /health/started, /health/ready, /health/live and
|
||||
# /metrics. Losing it means the kubelet fails the readiness probe and
|
||||
# kills the pod — the cluster would break for the wrong reason.
|
||||
#
|
||||
# Both rules deliberately omit `from:`, which allows those ports from any source.
|
||||
# Narrowing the source is not the subject here; the 2-hop experiment already
|
||||
# established how to do that by label when it matters.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: a1-block-jgroups-transport
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: keycloak
|
||||
policyTypes: [Ingress]
|
||||
ingress:
|
||||
- ports:
|
||||
- { port: 8080, protocol: TCP } # HTTP — must stay open
|
||||
- { port: 9000, protocol: TCP } # health + metrics — must stay open
|
||||
# 7800 is absent on purpose. That is the whole experiment.
|
||||
@@ -0,0 +1,128 @@
|
||||
# Experiment B-7 — oauth2-proxy, to measure how replicas share a cookie secret
|
||||
# and what happens when it is rotated (Q1, unknown 7).
|
||||
#
|
||||
# This is a different shape of problem from the BFF. The BFF keeps state on the
|
||||
# server, so the question was "which store". oauth2-proxy keeps no server state
|
||||
# at all: the whole session rides in a cookie that is signed and encrypted with
|
||||
# --cookie-secret. So there is nothing to share and nothing to lose on restart —
|
||||
# instead, every replica must hold the *same* secret, and changing it invalidates
|
||||
# every cookie at once.
|
||||
#
|
||||
# kubectl apply -f deploy/lab/k8s/b7-oauth2-proxy.yaml
|
||||
#
|
||||
# app2.hyeonworks.com is borrowed from Grafana for the duration of this
|
||||
# experiment; the certificate only covers auth / app1 / app2, so a fourth name
|
||||
# is not available. Grafana's Ingress is restored afterwards.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: oauth2-proxy-secrets
|
||||
namespace: keycloak-lab
|
||||
type: Opaque
|
||||
stringData:
|
||||
# oauth2-proxy requires exactly 16, 24 or 32 bytes. This is the value whose
|
||||
# rotation the experiment is about.
|
||||
COOKIE_SECRET_A: "lab-cookie-secret-aaaaaaaaaaaaaa"
|
||||
COOKIE_SECRET_B: "lab-cookie-secret-bbbbbbbbbbbbbb"
|
||||
CLIENT_SECRET: proxy-lab-secret
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
# Two replicas is the point: Q1 asks how they share the secret.
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels: { app: oauth2-proxy }
|
||||
template:
|
||||
metadata:
|
||||
labels: { app: oauth2-proxy }
|
||||
spec:
|
||||
# See B-1: Kubernetes injects <SVCNAME>_PORT as a tcp:// URL and it
|
||||
# collides with ordinary configuration names.
|
||||
enableServiceLinks: false
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1
|
||||
topologyKey: kubernetes.io/hostname
|
||||
whenUnsatisfiable: ScheduleAnyway
|
||||
labelSelector:
|
||||
matchLabels: { app: oauth2-proxy }
|
||||
containers:
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.7.1
|
||||
args:
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://auth.hyeonworks.com/realms/keycloak-patterns
|
||||
- --client-id=oauth2-proxy
|
||||
- --redirect-url=https://app2.hyeonworks.com/oauth2/callback
|
||||
- --email-domain=*
|
||||
- --http-address=0.0.0.0:4180
|
||||
# The upstream is the same echo app the B-4 header experiment used,
|
||||
# so what the proxy forwards can be read straight off the response.
|
||||
- --upstream=http://echo.header-lab.svc:8081
|
||||
# ★ 이 옵션을 켜면 세션(=쿠키)에 access token 이 들어간다.
|
||||
# 그러면 Set-Cookie 가 커져 프록시 앞단에서 502 가 났다.
|
||||
# B-4 에서 본 헤더 크기 절벽이 이번에는 응답 쪽에서 나타난 것이다.
|
||||
# - --pass-authorization-header=true
|
||||
- --set-xauthrequest=true
|
||||
- --reverse-proxy=true
|
||||
- --cookie-secure=true
|
||||
# One hour, matching the value Q1 records for the current setup.
|
||||
- --cookie-expire=1h
|
||||
- --skip-provider-button=true
|
||||
# ★ 쿠키에 세션 전체를 담으면 Set-Cookie 가 커지고, 그 응답이
|
||||
# 앞단 nginx 의 proxy_buffer 를 넘겨 502 가 났다(측정됨).
|
||||
# Redis 로 옮기면 쿠키에는 티켓만 남는다 — 그리고 그 순간
|
||||
# "replica 가 secret 을 공유해야 한다"는 문제의 성격도 바뀐다.
|
||||
- --session-store-type=redis
|
||||
- --redis-connection-url=redis://redis.keycloak-lab.svc:6379
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef: { name: oauth2-proxy-secrets, key: CLIENT_SECRET }
|
||||
# Which of the two secrets is in use is switched here. Both replicas
|
||||
# read the same key, which is exactly the sharing Q1 asks about.
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef: { name: oauth2-proxy-secrets, key: COOKIE_SECRET_A }
|
||||
ports:
|
||||
- containerPort: 4180
|
||||
name: http
|
||||
readinessProbe:
|
||||
httpGet: { path: /ping, port: http }
|
||||
initialDelaySeconds: 5
|
||||
resources:
|
||||
requests: { memory: 32Mi, cpu: 20m }
|
||||
limits: { memory: 128Mi }
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
selector: { app: oauth2-proxy }
|
||||
ports:
|
||||
- port: 4180
|
||||
targetPort: http
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: app2.hyeonworks.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: oauth2-proxy
|
||||
port:
|
||||
number: 4180
|
||||
@@ -0,0 +1,214 @@
|
||||
# BFF (2 replicas) + Redis, for the B-layer experiments.
|
||||
#
|
||||
# The BFF is deployed FIRST WITHOUT any session store wiring. That is deliberate:
|
||||
# B-0 asks what Spring Boot's autoconfiguration actually picks when nothing is
|
||||
# configured, and the only honest way to answer is to look at a running instance
|
||||
# that has been given nothing. Redis is deployed alongside but left unused until
|
||||
# B-1 turns it on.
|
||||
#
|
||||
# kubectl apply -f deploy/lab/k8s/bff-redis.yaml
|
||||
#
|
||||
# Image comes from the workstation, not a registry:
|
||||
# docker build -t keycloak-pattern-bff:lab bff/
|
||||
# docker save keycloak-pattern-bff:lab | ssh test-server "ssh kc-lab-1 'sudo k3s ctr images import -'"
|
||||
# (repeat for kc-lab-2)
|
||||
# so imagePullPolicy must stay Never on both replicas.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: bff-secrets
|
||||
namespace: keycloak-lab
|
||||
type: Opaque
|
||||
stringData:
|
||||
# Matches the client created with kcadm in the keycloak-patterns realm.
|
||||
# Base64 in etcd is not encryption — see D-3.
|
||||
KEYCLOAK_CLIENT_SECRET: bff-lab-secret
|
||||
---
|
||||
# Redis. B-5 measured that turning on AOF with `redis-cli config set` changes
|
||||
# nothing here, because /data is the container filesystem and dies with the
|
||||
# container — the appendonlydir was created and then thrown away. Persistence
|
||||
# configuration without a volume is decoration.
|
||||
#
|
||||
# So the volume comes first, and only then does `--appendonly yes` mean anything.
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: redis-data
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: local-path
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: redis
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels: { app: redis }
|
||||
template:
|
||||
metadata:
|
||||
labels: { app: redis }
|
||||
spec:
|
||||
# Same node as postgres so a node-loss experiment takes both stores at
|
||||
# once, matching how A-4 was set up.
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: kc-lab-2
|
||||
containers:
|
||||
- name: redis
|
||||
image: redis:7.4-alpine
|
||||
# appendfsync everysec 이 기본값이다 — 1초 분량을 잃을 수 있다.
|
||||
# Keycloak 의 synchronous_commit OFF(A-3)와 같은 모양의 트레이드오프다.
|
||||
args: ["redis-server", "--appendonly", "yes", "--dir", "/data"]
|
||||
ports:
|
||||
- containerPort: 6379
|
||||
name: redis
|
||||
readinessProbe:
|
||||
exec: { command: ["redis-cli", "ping"] }
|
||||
initialDelaySeconds: 3
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
resources:
|
||||
requests: { memory: 32Mi, cpu: 20m }
|
||||
limits: { memory: 128Mi }
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: redis-data
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: redis
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
selector: { app: redis }
|
||||
ports:
|
||||
- port: 6379
|
||||
targetPort: redis
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: bff
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
# Two replicas is the whole point: Q1 and Q2 only exist because a request can
|
||||
# land on an instance that did not handle the login.
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels: { app: bff }
|
||||
template:
|
||||
metadata:
|
||||
labels: { app: bff }
|
||||
spec:
|
||||
# Spread across both nodes so "the other instance" is genuinely another
|
||||
# machine, not another process on the same kernel.
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1
|
||||
topologyKey: kubernetes.io/hostname
|
||||
whenUnsatisfiable: ScheduleAnyway
|
||||
labelSelector:
|
||||
matchLabels: { app: bff }
|
||||
# 쿠버네티스는 같은 네임스페이스의 Service 마다 Docker link 시절의
|
||||
# 환경변수를 자동 주입한다: REDIS_PORT=tcp://10.43.57.116:6379.
|
||||
# 그것이 application.yml 의 ${REDIS_PORT:6379} 를 덮어써서 기동이 실패했다.
|
||||
# Failed to bind properties under 'spring.data.redis.port' to int:
|
||||
# Value: "tcp://10.43.57.116:6379"
|
||||
# 이 주입 자체를 끄는 것이 근본 처방이다. 이름을 바꿔 피하면 다음 사람이
|
||||
# 같은 함정에 다시 빠진다.
|
||||
enableServiceLinks: false
|
||||
containers:
|
||||
- name: bff
|
||||
image: keycloak-pattern-bff:lab
|
||||
imagePullPolicy: Never
|
||||
ports:
|
||||
- containerPort: 8083
|
||||
name: http
|
||||
env:
|
||||
# The browser is redirected to the public name; the BFF calls the
|
||||
# token endpoint over the cluster network. Getting these two the same
|
||||
# way round is what the 2-hop header experiment was about.
|
||||
- name: KC_ISSUER_EXTERNAL
|
||||
value: https://auth.hyeonworks.com/realms/keycloak-patterns
|
||||
- name: KC_ISSUER_INTERNAL
|
||||
value: http://keycloak.keycloak-lab.svc:8080/realms/keycloak-patterns
|
||||
# echo 는 header-lab 네임스페이스의 8081 이다. 다른 네임스페이스의
|
||||
# 서비스는 <svc>.<ns>.svc 로 부른다. 이름을 틀리면 500 이 나는데
|
||||
# 원인은 UnresolvedAddressException 이지 토큰 문제가 아니다.
|
||||
- name: RESOURCE_API_BASE_URL
|
||||
value: http://echo.header-lab.svc:8081
|
||||
- name: KEYCLOAK_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef: { name: bff-secrets, key: KEYCLOAK_CLIENT_SECRET }
|
||||
# Spring needs to know it is behind TLS termination, for the same
|
||||
# reason Keycloak needs KC_PROXY_HEADERS. Without it the redirect_uri
|
||||
# it builds comes back as http:// and Keycloak rejects it.
|
||||
- name: SERVER_FORWARD_HEADERS_STRATEGY
|
||||
value: native
|
||||
# B-1: Application Session 을 Redis 로 옮긴다.
|
||||
# OAuth2AuthorizedClient 는 이것으로 옮겨지지 않는다 — 조회 키가
|
||||
# 다르기 때문이며, B-0 에서 확인한 사실이다.
|
||||
- name: SPRING_SESSION_STORE_TYPE
|
||||
value: redis
|
||||
- name: REDIS_HOST
|
||||
value: redis.keycloak-lab.svc
|
||||
- name: REDIS_PORT
|
||||
value: "6379"
|
||||
# B-2: authorized client 는 PostgreSQL 로. 세션(Redis)과 다른
|
||||
# 저장소를 쓰는 것이 Q3 가 말한 "각각 설계한다"의 실물이다.
|
||||
- name: BFF_DB_URL
|
||||
value: jdbc:postgresql://postgres.keycloak-lab.svc:5432/keycloak
|
||||
- name: BFF_DB_USER
|
||||
value: keycloak
|
||||
- name: BFF_DB_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef: { name: keycloak-lab-secrets, key: POSTGRES_PASSWORD }
|
||||
- name: JAVA_TOOL_OPTIONS
|
||||
value: "-Xms128m -Xmx320m"
|
||||
readinessProbe:
|
||||
httpGet: { path: /actuator/health/readiness, port: http }
|
||||
initialDelaySeconds: 20
|
||||
failureThreshold: 30
|
||||
livenessProbe:
|
||||
httpGet: { path: /actuator/health/liveness, port: http }
|
||||
initialDelaySeconds: 60
|
||||
resources:
|
||||
requests: { memory: 320Mi, cpu: 100m }
|
||||
limits: { memory: 512Mi }
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: bff
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
selector: { app: bff }
|
||||
ports:
|
||||
- port: 8083
|
||||
targetPort: http
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: bff
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: app1.hyeonworks.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: bff
|
||||
port:
|
||||
number: 8083
|
||||
@@ -0,0 +1,62 @@
|
||||
# Restrict who may reach the echo pods.
|
||||
#
|
||||
# Traefik is configured to trust X-Forwarded-* from the whole pod CIDR, and the
|
||||
# app's Tomcat valve trusts every private range by default. Both are IP-range
|
||||
# decisions, so any pod in the cluster can forge those headers by talking to the
|
||||
# Service directly and bypassing Traefik entirely. Measured, not hypothetical:
|
||||
#
|
||||
# kubectl -n header-lab run t --rm -i --restart=Never --image=curlimages/curl -- \
|
||||
# curl -s http://echo:8081/api/echo -H 'X-Forwarded-Host: evil.example.com'
|
||||
# → serverName evil.example.com, remoteAddr 1.2.3.4
|
||||
#
|
||||
# A NetworkPolicy closes that path. It selects by label rather than IP, so it
|
||||
# survives pod restarts and rescheduling — unlike the trustedIPs list, which
|
||||
# could not name Traefik because its IP changes.
|
||||
#
|
||||
# "Trusting forwarded headers" and "guaranteeing a proxy sits in front" are a
|
||||
# pair. Doing only the first leaves this hole.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: echo-allow-traefik-only
|
||||
namespace: header-lab
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: echo
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
# The proxy itself. namespaceSelector and podSelector in one list item are
|
||||
# ANDed, so this is "traefik pods in kube-system" and nothing else.
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: traefik
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8081
|
||||
|
||||
# kubelet readiness/liveness probes originate from the node, not from a pod,
|
||||
# so they need their own rule. Without it the probes fail and the pods are
|
||||
# restarted in a loop.
|
||||
#
|
||||
# The probe's source address is the node's flannel bridge (cni0), which
|
||||
# holds the first address of that node's /24:
|
||||
# kc-lab-1 10.42.0.1 kc-lab-2 10.42.1.1
|
||||
# Listing them as /32 keeps this rule from re-admitting arbitrary pods,
|
||||
# which a broader 10.42.0.0/16 block would do and would undo the policy.
|
||||
#
|
||||
# Adding a node means adding its gateway here. Verify with:
|
||||
# kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{end}'
|
||||
- from:
|
||||
- ipBlock:
|
||||
cidr: 10.42.0.1/32
|
||||
- ipBlock:
|
||||
cidr: 10.42.1.1/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8081
|
||||
@@ -0,0 +1,113 @@
|
||||
# Header echo workload for the two-hop proxy contract measurement.
|
||||
#
|
||||
# browser -> host nginx (TLS termination) -> Traefik -> this pod
|
||||
#
|
||||
# The image is built from backend/ and imported straight into each node's
|
||||
# containerd, so imagePullPolicy must stay Never. See scripts/build-and-import.sh.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: header-lab
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: echo
|
||||
namespace: header-lab
|
||||
spec:
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app: echo
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: echo
|
||||
spec:
|
||||
# One replica per node so the sticky-session switch on the host nginx
|
||||
# upstream has something observable to route between.
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1
|
||||
topologyKey: kubernetes.io/hostname
|
||||
whenUnsatisfiable: ScheduleAnyway
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app: echo
|
||||
containers:
|
||||
- name: echo
|
||||
image: keycloak-pattern-api:lab
|
||||
imagePullPolicy: Never
|
||||
ports:
|
||||
- containerPort: 8081
|
||||
name: http
|
||||
env:
|
||||
- name: SERVER_PORT
|
||||
value: "8081"
|
||||
# "none" makes the app report the raw connection, so scheme/secure/
|
||||
# requestUrl show what arrives without any forwarded-header handling.
|
||||
# Set to "native" and redeploy to see the same request interpreted
|
||||
# with X-Forwarded-* honoured. Keycloak's KC_PROXY_HEADERS is the
|
||||
# same opt-in, which is why measuring both sides matters here.
|
||||
- name: SERVER_FORWARD_HEADERS_STRATEGY
|
||||
value: "native"
|
||||
# The JVM sizes its heap from the container limit, not the host.
|
||||
- name: JAVA_TOOL_OPTIONS
|
||||
value: "-XX:MaxRAMPercentage=70"
|
||||
# /api/echo is permitAll, so the JWT decoder is never exercised.
|
||||
# These stay pointed at the future Keycloak service name.
|
||||
- name: SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI
|
||||
value: "https://auth.hyeonworks.com/realms/keycloak-patterns"
|
||||
- name: SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI
|
||||
value: "https://auth.hyeonworks.com/realms/keycloak-patterns/protocol/openid-connect/certs"
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /actuator/health/readiness
|
||||
port: http
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 5
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /actuator/health/liveness
|
||||
port: http
|
||||
initialDelaySeconds: 45
|
||||
periodSeconds: 15
|
||||
resources:
|
||||
requests:
|
||||
memory: 320Mi
|
||||
cpu: 100m
|
||||
limits:
|
||||
memory: 512Mi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: echo
|
||||
namespace: header-lab
|
||||
spec:
|
||||
selector:
|
||||
app: echo
|
||||
ports:
|
||||
- port: 8081
|
||||
targetPort: http
|
||||
name: http
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: echo
|
||||
namespace: header-lab
|
||||
spec:
|
||||
# k3s ships Traefik as the default ingress controller. Keeping it is what
|
||||
# makes this lab a faithful two-hop replica.
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: app1.hyeonworks.com
|
||||
http:
|
||||
paths:
|
||||
- path: /api
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: echo
|
||||
port:
|
||||
number: 8081
|
||||
@@ -0,0 +1,277 @@
|
||||
# Keycloak multi-node cluster with PostgreSQL.
|
||||
#
|
||||
# Goal of this manifest: two Keycloak pods on two different nodes must discover
|
||||
# each other and form one Infinispan cluster. Keycloak 26 discovers peers through
|
||||
# the database (jdbc-ping) rather than multicast, writing to a JGROUPS_PING table,
|
||||
# but the cluster traffic itself runs over TCP 7800 between the pods. Those are
|
||||
# two separate mechanisms, which is why "registered in the DB but not clustered"
|
||||
# is a real failure mode — and one that a single node cannot reproduce.
|
||||
#
|
||||
# kubectl apply -f deploy/lab/k8s/keycloak-cluster.yaml
|
||||
# kubectl -n keycloak-lab rollout status statefulset/keycloak --timeout=600s
|
||||
#
|
||||
# Secrets are plain here. Proper secret handling is roadmap item 11; keeping it
|
||||
# visible for now is deliberate so the gap is obvious rather than forgotten.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: keycloak-lab
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: keycloak-lab-secrets
|
||||
namespace: keycloak-lab
|
||||
type: Opaque
|
||||
stringData:
|
||||
POSTGRES_PASSWORD: lab-postgres-change-me
|
||||
KC_BOOTSTRAP_ADMIN_PASSWORD: lab-admin-change-me
|
||||
---
|
||||
# PostgreSQL. local-path binds the volume to whichever node the pod lands on, so
|
||||
# the database is effectively pinned to one node. That is not a flaw here: it is
|
||||
# what makes "the database node dies" a meaningful experiment later.
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: postgres-data
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: local-path
|
||||
resources:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: postgres
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate # RWO volume cannot be mounted by two pods at once
|
||||
selector:
|
||||
matchLabels:
|
||||
app: postgres
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: postgres
|
||||
spec:
|
||||
containers:
|
||||
- name: postgres
|
||||
image: postgres:16-alpine
|
||||
ports:
|
||||
- containerPort: 5432
|
||||
name: postgres
|
||||
env:
|
||||
- name: POSTGRES_DB
|
||||
value: keycloak
|
||||
- name: POSTGRES_USER
|
||||
value: keycloak
|
||||
- name: POSTGRES_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: keycloak-lab-secrets
|
||||
key: POSTGRES_PASSWORD
|
||||
# The image refuses to initialise into a non-empty mount, and
|
||||
# local-path volumes are clean, but this keeps the data one level
|
||||
# down so a lost+found or similar never blocks initdb.
|
||||
- name: PGDATA
|
||||
value: /var/lib/postgresql/data/pgdata
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /var/lib/postgresql/data
|
||||
readinessProbe:
|
||||
exec:
|
||||
command: ["sh", "-c", "pg_isready -U keycloak -d keycloak"]
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 5
|
||||
resources:
|
||||
requests:
|
||||
memory: 192Mi
|
||||
cpu: 50m
|
||||
limits:
|
||||
memory: 512Mi
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: postgres-data
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: postgres
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
selector:
|
||||
app: postgres
|
||||
ports:
|
||||
- port: 5432
|
||||
targetPort: postgres
|
||||
---
|
||||
# Keycloak. A StatefulSet rather than a Deployment so each pod keeps a stable
|
||||
# name (keycloak-0, keycloak-1); cluster membership is far easier to read in
|
||||
# logs and in the JGROUPS_PING table when the identities do not churn.
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: keycloak
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
serviceName: keycloak-headless
|
||||
replicas: 2
|
||||
podManagementPolicy: Parallel # both pods start together, so they race to
|
||||
# register — which is the interesting case
|
||||
selector:
|
||||
matchLabels:
|
||||
app: keycloak
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: keycloak
|
||||
spec:
|
||||
# One pod per node. Two pods on one node would share a kernel and make the
|
||||
# 7800 blocking experiment meaningless.
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1
|
||||
topologyKey: kubernetes.io/hostname
|
||||
whenUnsatisfiable: ScheduleAnyway
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app: keycloak
|
||||
containers:
|
||||
- name: keycloak
|
||||
image: quay.io/keycloak/keycloak:26.7.0
|
||||
# "start", not "start-dev". Dev mode forces cache=local and there is
|
||||
# no cluster to form at all.
|
||||
args: ["start"]
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
name: http
|
||||
- containerPort: 9000
|
||||
name: management
|
||||
- containerPort: 7800
|
||||
name: jgroups
|
||||
env:
|
||||
- name: KC_DB
|
||||
value: postgres
|
||||
- name: KC_DB_URL
|
||||
value: jdbc:postgresql://postgres:5432/keycloak
|
||||
- name: KC_DB_USERNAME
|
||||
value: keycloak
|
||||
- name: KC_DB_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: keycloak-lab-secrets
|
||||
key: POSTGRES_PASSWORD
|
||||
|
||||
# Settings confirmed by the two-hop header measurement.
|
||||
# KC_HOSTNAME carries the full external URL, which pins scheme and
|
||||
# host for issuer and redirect URLs regardless of headers.
|
||||
# KC_PROXY_HEADERS is the separate opt-in that lets the forwarded
|
||||
# client address through — the same kind of switch as Spring's
|
||||
# forward-headers-strategy. See docs/two-hop-proxy-header-contract.md.
|
||||
- name: KC_HOSTNAME
|
||||
value: https://auth.hyeonworks.com
|
||||
- name: KC_HOSTNAME_STRICT
|
||||
value: "true"
|
||||
- name: KC_PROXY_HEADERS
|
||||
value: xforwarded
|
||||
- name: KC_HTTP_ENABLED
|
||||
value: "true"
|
||||
|
||||
- name: KC_HEALTH_ENABLED
|
||||
value: "true"
|
||||
- name: KC_METRICS_ENABLED
|
||||
value: "true"
|
||||
|
||||
# Without an explicit cap the JVM sizes its heap from the container
|
||||
# limit and this lab has roughly 3.8GB of guest headroom in total.
|
||||
- name: JAVA_OPTS_KC_HEAP
|
||||
value: "-Xms256m -Xmx512m"
|
||||
|
||||
- name: KC_BOOTSTRAP_ADMIN_USERNAME
|
||||
value: admin
|
||||
- name: KC_BOOTSTRAP_ADMIN_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: keycloak-lab-secrets
|
||||
key: KC_BOOTSTRAP_ADMIN_PASSWORD
|
||||
|
||||
# Keycloak serves health and metrics on the management port (9000),
|
||||
# not on 8080, since version 25.
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /health/started
|
||||
port: management
|
||||
periodSeconds: 10
|
||||
failureThreshold: 60 # first boot runs an implicit build
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health/ready
|
||||
port: management
|
||||
periodSeconds: 10
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health/live
|
||||
port: management
|
||||
periodSeconds: 30
|
||||
resources:
|
||||
requests:
|
||||
memory: 640Mi
|
||||
cpu: 100m
|
||||
limits:
|
||||
memory: 900Mi
|
||||
---
|
||||
# Headless service. Not required for jdbc-ping discovery, which goes through the
|
||||
# database, but it gives each pod a stable DNS name for direct inspection.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: keycloak-headless
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
clusterIP: None
|
||||
selector:
|
||||
app: keycloak
|
||||
ports:
|
||||
- port: 8080
|
||||
targetPort: http
|
||||
name: http
|
||||
- port: 9000
|
||||
targetPort: management
|
||||
name: management
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: keycloak
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
selector:
|
||||
app: keycloak
|
||||
ports:
|
||||
- port: 8080
|
||||
targetPort: http
|
||||
name: http
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: keycloak
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: auth.hyeonworks.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: keycloak
|
||||
port:
|
||||
number: 8080
|
||||
@@ -0,0 +1,373 @@
|
||||
# Prometheus + node-exporter + Grafana.
|
||||
#
|
||||
# Purpose: during a fault-injection experiment, know *which signal moved first*.
|
||||
# Without a metrics store the only record is whatever scrolled past in a terminal,
|
||||
# and "the cluster recovered in about a minute" is not a measurement.
|
||||
#
|
||||
# kubectl apply -f deploy/lab/k8s/observability.yaml
|
||||
# kubectl -n observability rollout status deployment/prometheus --timeout=300s
|
||||
#
|
||||
# Placement decision — Prometheus and Grafana are pinned to the control-plane
|
||||
# node (kc-lab-1). An observability stack must not share a failure domain with
|
||||
# the thing it observes. With only two nodes that cannot be fully avoided, so the
|
||||
# rule here is: the node that gets killed in experiments is the *agent*
|
||||
# (kc-lab-2, holding keycloak-0 and postgres), and everything needed to watch
|
||||
# that happen lives on the server node.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: observability
|
||||
---
|
||||
# Prometheus discovers scrape targets by querying the Kubernetes API, so it
|
||||
# needs read access to nodes, services, endpoints and pods. Without this the
|
||||
# kubernetes_sd_configs below silently return no targets.
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: prometheus
|
||||
namespace: observability
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: prometheus
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
# nodes/proxy is required in addition to nodes/metrics: the kubelet job
|
||||
# reaches each node through the API server's proxy subresource
|
||||
# (/api/v1/nodes/<name>/proxy/metrics). Without it every kubelet target
|
||||
# fails with 403 Forbidden while the other jobs stay green — a partial
|
||||
# failure that is easy to miss unless the target list is checked.
|
||||
resources: [nodes, nodes/metrics, nodes/proxy, services, endpoints, pods]
|
||||
verbs: [get, list, watch]
|
||||
- nonResourceURLs: ["/metrics"]
|
||||
verbs: [get]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: prometheus
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: prometheus
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: prometheus
|
||||
namespace: observability
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: prometheus-config
|
||||
namespace: observability
|
||||
data:
|
||||
prometheus.yml: |
|
||||
global:
|
||||
# 15s is short for production but right here: a node loss should show up
|
||||
# within a couple of samples, not a minute later.
|
||||
scrape_interval: 15s
|
||||
evaluation_interval: 15s
|
||||
|
||||
scrape_configs:
|
||||
# Prometheus scraping itself. Useful as a control: if this target is down,
|
||||
# the problem is Prometheus, not the thing being measured.
|
||||
- job_name: prometheus
|
||||
static_configs:
|
||||
- targets: ['localhost:9090']
|
||||
|
||||
# Keycloak. Metrics live on the management port 9000, not 8080 — the same
|
||||
# split that the health probes use. KC_METRICS_ENABLED=true is already set
|
||||
# on the StatefulSet.
|
||||
#
|
||||
# Discovery is by endpoints rather than a static list because pod IPs
|
||||
# change on every restart; that was observed directly when the lab was
|
||||
# power-cycled and every pod came back with a new address.
|
||||
- job_name: keycloak
|
||||
kubernetes_sd_configs:
|
||||
- role: endpoints
|
||||
namespaces:
|
||||
names: [keycloak-lab]
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
|
||||
action: keep
|
||||
regex: keycloak-headless;management
|
||||
- source_labels: [__meta_kubernetes_pod_name]
|
||||
target_label: pod
|
||||
- source_labels: [__meta_kubernetes_pod_node_name]
|
||||
target_label: node
|
||||
|
||||
# node-exporter, one per node via DaemonSet. This is what answers
|
||||
# "did the machine die or did the process die".
|
||||
- job_name: node-exporter
|
||||
kubernetes_sd_configs:
|
||||
- role: endpoints
|
||||
namespaces:
|
||||
names: [observability]
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_service_name]
|
||||
action: keep
|
||||
regex: node-exporter
|
||||
- source_labels: [__meta_kubernetes_pod_node_name]
|
||||
target_label: node
|
||||
|
||||
# The kubelet's own metrics, reached through the API server proxy so no
|
||||
# extra port needs opening.
|
||||
- job_name: kubelet
|
||||
scheme: https
|
||||
tls_config:
|
||||
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
||||
insecure_skip_verify: true
|
||||
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
||||
kubernetes_sd_configs:
|
||||
- role: node
|
||||
relabel_configs:
|
||||
- action: labelmap
|
||||
regex: __meta_kubernetes_node_label_(.+)
|
||||
- target_label: __address__
|
||||
replacement: kubernetes.default.svc:443
|
||||
- source_labels: [__meta_kubernetes_node_name]
|
||||
regex: (.+)
|
||||
target_label: __metrics_path__
|
||||
replacement: /api/v1/nodes/${1}/proxy/metrics
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: prometheus-data
|
||||
namespace: observability
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: local-path
|
||||
resources:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: prometheus
|
||||
namespace: observability
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate # RWO volume; two pods cannot mount it at once
|
||||
selector:
|
||||
matchLabels:
|
||||
app: prometheus
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: prometheus
|
||||
spec:
|
||||
serviceAccountName: prometheus
|
||||
# See the placement note at the top of this file.
|
||||
nodeSelector:
|
||||
node-role.kubernetes.io/control-plane: "true"
|
||||
securityContext:
|
||||
fsGroup: 65534 # the image runs as nobody and must own the volume
|
||||
containers:
|
||||
- name: prometheus
|
||||
image: prom/prometheus:v3.1.0
|
||||
args:
|
||||
- --config.file=/etc/prometheus/prometheus.yml
|
||||
- --storage.tsdb.path=/prometheus
|
||||
# 7 days is far more than an experiment needs and keeps the volume
|
||||
# small enough that it never becomes the reason a node fills up.
|
||||
- --storage.tsdb.retention.time=7d
|
||||
- --web.enable-lifecycle
|
||||
ports:
|
||||
- containerPort: 9090
|
||||
name: http
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/prometheus
|
||||
- name: data
|
||||
mountPath: /prometheus
|
||||
readinessProbe:
|
||||
httpGet: { path: /-/ready, port: http }
|
||||
initialDelaySeconds: 10
|
||||
livenessProbe:
|
||||
httpGet: { path: /-/healthy, port: http }
|
||||
initialDelaySeconds: 30
|
||||
resources:
|
||||
requests: { memory: 256Mi, cpu: 50m }
|
||||
limits: { memory: 640Mi }
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: prometheus-config
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: prometheus-data
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: prometheus
|
||||
namespace: observability
|
||||
spec:
|
||||
selector:
|
||||
app: prometheus
|
||||
ports:
|
||||
- port: 9090
|
||||
targetPort: http
|
||||
---
|
||||
# node-exporter. A DaemonSet so every node reports, including one that is about
|
||||
# to be killed — the last samples before it goes silent are the interesting part.
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: node-exporter
|
||||
namespace: observability
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: node-exporter
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: node-exporter
|
||||
spec:
|
||||
# Host namespaces: the point is to measure the machine, not the container.
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
tolerations:
|
||||
- operator: Exists # must also run on tainted nodes
|
||||
containers:
|
||||
- name: node-exporter
|
||||
image: prom/node-exporter:v1.8.2
|
||||
args:
|
||||
- --path.procfs=/host/proc
|
||||
- --path.sysfs=/host/sys
|
||||
- --path.rootfs=/host/root
|
||||
- --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
|
||||
ports:
|
||||
- containerPort: 9100
|
||||
name: metrics
|
||||
hostPort: 9100
|
||||
volumeMounts:
|
||||
- { name: proc, mountPath: /host/proc, readOnly: true }
|
||||
- { name: sys, mountPath: /host/sys, readOnly: true }
|
||||
- { name: rootfs, mountPath: /host/root, readOnly: true, mountPropagation: HostToContainer }
|
||||
resources:
|
||||
requests: { memory: 32Mi, cpu: 20m }
|
||||
limits: { memory: 96Mi }
|
||||
volumes:
|
||||
- { name: proc, hostPath: { path: /proc } }
|
||||
- { name: sys, hostPath: { path: /sys } }
|
||||
- { name: rootfs, hostPath: { path: / } }
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: node-exporter
|
||||
namespace: observability
|
||||
spec:
|
||||
clusterIP: None # headless: Prometheus wants each pod, not a VIP
|
||||
selector:
|
||||
app: node-exporter
|
||||
ports:
|
||||
- port: 9100
|
||||
targetPort: metrics
|
||||
name: metrics
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: grafana
|
||||
namespace: observability
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: grafana
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: grafana
|
||||
spec:
|
||||
nodeSelector:
|
||||
node-role.kubernetes.io/control-plane: "true"
|
||||
containers:
|
||||
- name: grafana
|
||||
image: grafana/grafana:11.4.0
|
||||
ports:
|
||||
- containerPort: 3000
|
||||
name: http
|
||||
env:
|
||||
- name: GF_SECURITY_ADMIN_USER
|
||||
value: admin
|
||||
- name: GF_SECURITY_ADMIN_PASSWORD
|
||||
value: lab-grafana-change-me
|
||||
# Grafana builds absolute URLs for redirects and asset paths. Behind
|
||||
# the nginx -> Traefik chain it must be told the external address,
|
||||
# for exactly the reason Keycloak needs KC_HOSTNAME. Without it,
|
||||
# login redirects come back as http://<pod-ip>:3000.
|
||||
- name: GF_SERVER_ROOT_URL
|
||||
value: https://app2.hyeonworks.com
|
||||
volumeMounts:
|
||||
- name: datasources
|
||||
mountPath: /etc/grafana/provisioning/datasources
|
||||
readinessProbe:
|
||||
httpGet: { path: /api/health, port: http }
|
||||
initialDelaySeconds: 15
|
||||
resources:
|
||||
requests: { memory: 128Mi, cpu: 50m }
|
||||
limits: { memory: 320Mi }
|
||||
volumes:
|
||||
- name: datasources
|
||||
configMap:
|
||||
name: grafana-datasources
|
||||
---
|
||||
# Provisioning the datasource as a file means Grafana comes up already wired to
|
||||
# Prometheus. Clicking through the UI would leave the configuration only in
|
||||
# Grafana's own database, which is emptyDir here and disappears on restart.
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: grafana-datasources
|
||||
namespace: observability
|
||||
data:
|
||||
prometheus.yaml: |
|
||||
apiVersion: 1
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://prometheus.observability.svc:9090
|
||||
isDefault: true
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: grafana
|
||||
namespace: observability
|
||||
spec:
|
||||
selector:
|
||||
app: grafana
|
||||
ports:
|
||||
- port: 3000
|
||||
targetPort: http
|
||||
---
|
||||
# Grafana is published on app2.hyeonworks.com because that name is already in
|
||||
# the wildcard-free certificate (auth / app1 / app2) and is otherwise unused.
|
||||
# It moves when app2 is needed for the SSO experiment.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: grafana
|
||||
namespace: observability
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: app2.hyeonworks.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: grafana
|
||||
port:
|
||||
number: 3000
|
||||
@@ -0,0 +1,43 @@
|
||||
# Make Traefik trust the X-Forwarded-* headers that the host nginx sets.
|
||||
#
|
||||
# Without this, Traefik rewrites every forwarded header from its own connection,
|
||||
# which is plain HTTP on port 80. The application then sees scheme=http even
|
||||
# though the browser connected over TLS. See docs/two-hop-proxy-header-contract.md.
|
||||
#
|
||||
# k3s installs Traefik through its bundled HelmChart, so values are overridden
|
||||
# with a HelmChartConfig rather than by editing the deployment. k3s reconciles
|
||||
# the chart and recreates the Traefik pod.
|
||||
#
|
||||
# kubectl apply -f deploy/lab/k8s/traefik-forwarded-headers.yaml
|
||||
# kubectl -n kube-system rollout status deploy/traefik --timeout=180s
|
||||
apiVersion: helm.cattle.io/v1
|
||||
kind: HelmChartConfig
|
||||
metadata:
|
||||
name: traefik
|
||||
namespace: kube-system
|
||||
spec:
|
||||
valuesContent: |-
|
||||
ports:
|
||||
web:
|
||||
forwardedHeaders:
|
||||
# Requests arriving from these sources keep their existing
|
||||
# X-Forwarded-* values instead of having them rewritten.
|
||||
#
|
||||
# 10.42.0.0/16 is the pod CIDR. It is required because the traefik
|
||||
# Service uses externalTrafficPolicy: Cluster, so svclb SNATs the
|
||||
# traffic and Traefik sees a pod-network address rather than the
|
||||
# host nginx address.
|
||||
#
|
||||
# The node/host range is deliberately absent. Because svclb SNATs,
|
||||
# the host nginx address never reaches Traefik — measured, not assumed.
|
||||
# Trusting a range that cannot appear only widens the surface.
|
||||
#
|
||||
# Trusting the whole pod CIDR still means any pod in the cluster could
|
||||
# forge these headers, which is why echo-network-policy.yaml restricts
|
||||
# who may reach the application at all.
|
||||
trustedIPs:
|
||||
- 10.42.0.0/16
|
||||
websecure:
|
||||
forwardedHeaders:
|
||||
trustedIPs:
|
||||
- 10.42.0.0/16
|
||||
@@ -0,0 +1,42 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build the API image on this workstation and import it into each lab node's
|
||||
# containerd.
|
||||
#
|
||||
# k3s does not run Docker and the lab has no registry, so images are shipped as
|
||||
# a stream: docker save -> ssh through the lab host -> k3s ctr images import.
|
||||
# Every node needs its own copy because the scheduler may place the pod anywhere.
|
||||
#
|
||||
# ./deploy/lab/scripts/build-and-import.sh
|
||||
# IMAGE=keycloak-pattern-api:lab NODES="kc-lab-1" ./deploy/lab/scripts/build-and-import.sh
|
||||
set -euo pipefail
|
||||
|
||||
IMAGE="${IMAGE:-keycloak-pattern-api:lab}"
|
||||
NODES="${NODES:-kc-lab-1 kc-lab-2}"
|
||||
LAB_HOST="${LAB_HOST:-test-server}"
|
||||
CONTEXT="${CONTEXT:-backend}"
|
||||
|
||||
repo_root="$(git rev-parse --show-toplevel)"
|
||||
cd "$repo_root"
|
||||
|
||||
echo "==> building ${IMAGE} from ${CONTEXT}/"
|
||||
docker build -t "$IMAGE" "$CONTEXT"
|
||||
|
||||
for node in $NODES; do
|
||||
echo "==> importing into ${node}"
|
||||
# Nested ssh: the workstation cannot reach the guests directly because they
|
||||
# sit behind the lab host's libvirt NAT. The lab host's ~/.ssh/config holds
|
||||
# the kc-lab-* aliases.
|
||||
docker save "$IMAGE" \
|
||||
| ssh "$LAB_HOST" "ssh ${node} 'sudo k3s ctr images import -'"
|
||||
done
|
||||
|
||||
echo "==> verifying"
|
||||
for node in $NODES; do
|
||||
printf ' %-10s ' "$node"
|
||||
ssh "$LAB_HOST" "ssh ${node} 'sudo k3s ctr images ls -q'" \
|
||||
| grep -c "$IMAGE" \
|
||||
| xargs -I{} echo "{} match(es)"
|
||||
done
|
||||
|
||||
echo
|
||||
echo "next: kubectl rollout restart -n header-lab deployment/echo"
|
||||
+55
@@ -0,0 +1,55 @@
|
||||
#!/usr/bin/env bash
|
||||
# Experiment 0c — where does a session entry actually live?
|
||||
#
|
||||
# Experiment 0b showed keycloak-1's session cache never moved when keycloak-0
|
||||
# handled a login. That leaves two explanations:
|
||||
#
|
||||
# (a) a DISTRIBUTED cache with owners=1 — entries are spread across nodes by
|
||||
# consistent hashing, and this one happened to land on keycloak-0;
|
||||
# (b) a LOCAL cache — each node only ever caches what it handled itself.
|
||||
#
|
||||
# They are distinguished by driving logins at the OTHER node. Under (a) the
|
||||
# entries would keep landing on both nodes regardless of who was asked. Under
|
||||
# (b) the count rises only on the node that received the request.
|
||||
set -uo pipefail
|
||||
|
||||
NS="${NS:-keycloak-lab}"
|
||||
N="${N:-5}"
|
||||
K0_IP=$(kubectl -n "$NS" get pod keycloak-0 -o jsonpath='{.status.podIP}')
|
||||
K1_IP=$(kubectl -n "$NS" get pod keycloak-1 -o jsonpath='{.status.podIP}')
|
||||
ADMIN_PW=$(kubectl -n "$NS" get secret keycloak-lab-secrets \
|
||||
-o jsonpath='{.data.KC_BOOTSTRAP_ADMIN_PASSWORD}' | base64 -d)
|
||||
|
||||
echo "수집 시각: $(date '+%Y-%m-%d %H:%M:%S %Z')"
|
||||
echo " keycloak-0 = $K0_IP ($(kubectl -n "$NS" get pod keycloak-0 -o jsonpath='{.spec.nodeName}'))"
|
||||
echo " keycloak-1 = $K1_IP ($(kubectl -n "$NS" get pod keycloak-1 -o jsonpath='{.spec.nodeName}'))"
|
||||
echo
|
||||
|
||||
kubectl -n "$NS" run kc-own --rm -i --restart=Never \
|
||||
--image=curlimages/curl:8.11.1 --quiet --command -- sh -c "
|
||||
O=/tmp/o; : > \$O
|
||||
ent() {
|
||||
curl -s --retry 3 --max-time 20 http://\$1:9000/metrics \
|
||||
| grep -E '^vendor_statistics_approximate_entries_unique.cache=.sessions' \
|
||||
| awk '{print \$NF}'
|
||||
}
|
||||
login() { i=0; while [ \$i -lt $N ]; do
|
||||
curl -s -o /dev/null -X POST http://\$1:8080/realms/master/protocol/openid-connect/token \
|
||||
-d grant_type=password -d client_id=admin-cli \
|
||||
-d username=admin -d 'password=$ADMIN_PW'
|
||||
i=\$((i+1)); done; sleep 5; }
|
||||
{
|
||||
printf '%-32s %12s %12s\n' '단계' 'k0 entries' 'k1 entries'
|
||||
printf '%-32s %12s %12s\n' '시작' \"\$(ent $K0_IP)\" \"\$(ent $K1_IP)\"
|
||||
login $K1_IP
|
||||
printf '%-32s %12s %12s\n' 'keycloak-1 에 로그인 ${N}회' \"\$(ent $K0_IP)\" \"\$(ent $K1_IP)\"
|
||||
login $K0_IP
|
||||
printf '%-32s %12s %12s\n' 'keycloak-0 에 로그인 ${N}회' \"\$(ent $K0_IP)\" \"\$(ent $K1_IP)\"
|
||||
} >> \$O
|
||||
cat \$O
|
||||
" 2>&1 | grep -v '^pod .* deleted$'
|
||||
|
||||
echo
|
||||
echo "=== 대조: PostgreSQL 에는 몇 건인가 ==="
|
||||
kubectl -n "$NS" exec deploy/postgres -- psql -U keycloak -d keycloak -tAc \
|
||||
"select count(*) from offline_user_session where offline_flag='0'" 2>/dev/null | sed 's/^/ online 세션 /'
|
||||
Executable
+83
@@ -0,0 +1,83 @@
|
||||
#!/usr/bin/env bash
|
||||
# Experiment 0b — does the Infinispan cache itself replicate, or do both nodes
|
||||
# merely agree because they read the same database?
|
||||
#
|
||||
# Experiment 0 proved the two nodes give the same answers. That alone does NOT
|
||||
# prove Infinispan replicated anything: with persistent-user-sessions (the
|
||||
# Keycloak 26 default) the session is written to PostgreSQL, so two nodes reading
|
||||
# one database would agree even with the cache disabled entirely.
|
||||
#
|
||||
# This script separates the two by measuring the cache counters on BOTH nodes
|
||||
# around a single login. If the write on keycloak-0 shows up as cache activity
|
||||
# on keycloak-1, the replication is real and not a database artifact.
|
||||
set -uo pipefail
|
||||
|
||||
NS="${NS:-keycloak-lab}"
|
||||
K0_IP=$(kubectl -n "$NS" get pod keycloak-0 -o jsonpath='{.status.podIP}')
|
||||
K1_IP=$(kubectl -n "$NS" get pod keycloak-1 -o jsonpath='{.status.podIP}')
|
||||
ADMIN_PW=$(kubectl -n "$NS" get secret keycloak-lab-secrets \
|
||||
-o jsonpath='{.data.KC_BOOTSTRAP_ADMIN_PASSWORD}' | base64 -d)
|
||||
|
||||
echo "수집 시각: $(date '+%Y-%m-%d %H:%M:%S %Z')"
|
||||
echo
|
||||
|
||||
# 파드 출력을 스트리밍으로 받으면 조각이 유실된다. 실제로 첫 시도에서
|
||||
# keycloak-1 의 스냅샷과 그 다음 마커가 통째로 사라져 델타가 0 으로 보였다.
|
||||
# 파드 안에서 파일로 모았다가 마지막에 한 번만 내보낸다.
|
||||
kubectl -n "$NS" run kc-delta --rm -i --restart=Never \
|
||||
--image=curlimages/curl:8.11.1 --quiet --command -- sh -c "
|
||||
set -u
|
||||
K0='http://$K0_IP'; K1='http://$K1_IP'
|
||||
O=/tmp/o.txt; : > \$O
|
||||
snap() {
|
||||
curl -s --retry 3 --retry-connrefused --max-time 20 \$1:9000/metrics \
|
||||
| grep -E '^vendor_(statistics_(stores|hits|misses|approximate_entries_unique)|rpc_manager_replication_count)\{cache=\"(sessions|clientSessions)\"' \
|
||||
| sed 's/,cache_manager=\"keycloak\"//; s/,node=\"[^\"]*\"//' >> \$O
|
||||
}
|
||||
echo '###BEFORE_K0' >> \$O; snap \$K0
|
||||
echo '###BEFORE_K1' >> \$O; snap \$K1
|
||||
echo '###LOGIN' >> \$O
|
||||
curl -s -o /dev/null -w 'http_code=%{http_code}\n' -X POST \
|
||||
\"\$K0:8080/realms/master/protocol/openid-connect/token\" \
|
||||
-d grant_type=password -d client_id=admin-cli \
|
||||
-d username=admin -d 'password=$ADMIN_PW' >> \$O
|
||||
sleep 5
|
||||
echo '###AFTER_K0' >> \$O; snap \$K0
|
||||
echo '###AFTER_K1' >> \$O; snap \$K1
|
||||
echo '###END' >> \$O
|
||||
cat \$O
|
||||
" 2>&1 | grep -v '^pod .* deleted$' > /tmp/cache-delta.txt
|
||||
|
||||
python3 - /tmp/cache-delta.txt <<'PY'
|
||||
import re, sys
|
||||
raw = open(sys.argv[1]).read()
|
||||
blocks, cur = {}, None
|
||||
for line in raw.splitlines():
|
||||
if line.startswith('###'):
|
||||
cur = line[3:]; blocks[cur] = {}
|
||||
elif cur and '{' in line:
|
||||
m = re.match(r'(\S+?)\{cache="(\w+)"\}\s+(\S+)', line)
|
||||
if m:
|
||||
blocks[cur][(m.group(1), m.group(2))] = float(m.group(3))
|
||||
|
||||
print('=== 로그인은 keycloak-0 에만 보냈다 ===')
|
||||
code = [l for l in raw.splitlines() if l.startswith('http_code=')]
|
||||
print(' 로그인 응답: ' + (code[0] if code else '없음'))
|
||||
for n in ('BEFORE_K0','BEFORE_K1','AFTER_K0','AFTER_K1'):
|
||||
if not blocks.get(n):
|
||||
print(f' !! {n} 스냅샷이 비었다 — 델타를 신뢰할 수 없다')
|
||||
print()
|
||||
hdr = f" {'계수기':<42} {'캐시':<15} {'전':>8} {'후':>8} {'증가':>7}"
|
||||
for node in ('K0', 'K1'):
|
||||
who = 'keycloak-0 (로그인을 받은 노드)' if node == 'K0' else 'keycloak-1 (아무 요청도 받지 않은 노드)'
|
||||
print(f'=== {who} ===')
|
||||
print(hdr)
|
||||
b, a = blocks.get(f'BEFORE_{node}', {}), blocks.get(f'AFTER_{node}', {})
|
||||
for k in sorted(set(b) | set(a)):
|
||||
before, after = b.get(k[0:2], 0.0), a.get(k[0:2], 0.0)
|
||||
d = after - before
|
||||
mark = ' ←' if d else ''
|
||||
name = k[0].replace('vendor_statistics_', '').replace('vendor_rpc_manager_', 'rpc.')
|
||||
print(f" {name:<42} {k[1]:<15} {before:>8.0f} {after:>8.0f} {d:>+7.0f}{mark}")
|
||||
print()
|
||||
PY
|
||||
+108
@@ -0,0 +1,108 @@
|
||||
#!/usr/bin/env bash
|
||||
# Experiment 0d — capture the actual SQL that the OTHER node runs.
|
||||
#
|
||||
# Experiments 0b/0c showed that session entries never appear in keycloak-1's
|
||||
# memory, yet keycloak-1 can use a session keycloak-0 created. The conclusion
|
||||
# "keycloak-1 reads it from PostgreSQL" was an inference, not an observation.
|
||||
#
|
||||
# This script turns on statement logging in PostgreSQL for a few seconds, sends
|
||||
# ONE refresh request to keycloak-1 for a session born on keycloak-0, and greps
|
||||
# the database log for that session id. If the inference is right, the SQL is
|
||||
# there, issued from keycloak-1's pod IP.
|
||||
#
|
||||
# It also checks whether serving that request makes keycloak-1 cache the session
|
||||
# — which sharpens "each node caches what it handled" from "what it logged in"
|
||||
# to "what it touched".
|
||||
set -uo pipefail
|
||||
|
||||
NS="${NS:-keycloak-lab}"
|
||||
PSQL="kubectl -n $NS exec deploy/postgres -- psql -U keycloak -d keycloak -tAc"
|
||||
|
||||
K0_IP=$(kubectl -n "$NS" get pod keycloak-0 -o jsonpath='{.status.podIP}')
|
||||
K1_IP=$(kubectl -n "$NS" get pod keycloak-1 -o jsonpath='{.status.podIP}')
|
||||
ADMIN_PW=$(kubectl -n "$NS" get secret keycloak-lab-secrets \
|
||||
-o jsonpath='{.data.KC_BOOTSTRAP_ADMIN_PASSWORD}' | base64 -d)
|
||||
|
||||
echo "수집 시각: $(date '+%Y-%m-%d %H:%M:%S %Z')"
|
||||
echo " keycloak-0 = $K0_IP (세션을 만드는 노드)"
|
||||
echo " keycloak-1 = $K1_IP (읽기만 하는 노드)"
|
||||
echo
|
||||
|
||||
# %h 를 넣어야 어느 파드가 보낸 질의인지 로그에서 구분된다.
|
||||
echo "=== PostgreSQL 문장 로깅을 켠다 ==="
|
||||
$PSQL "alter system set log_statement='all'" >/dev/null 2>&1
|
||||
$PSQL "alter system set log_line_prefix='%m [%p] %h '" >/dev/null 2>&1
|
||||
$PSQL "select pg_reload_conf()" >/dev/null 2>&1
|
||||
echo " log_statement = $($PSQL 'show log_statement' 2>/dev/null)"
|
||||
echo " log_line_prefix = $($PSQL 'show log_line_prefix' 2>/dev/null)"
|
||||
echo
|
||||
|
||||
# 로그 커서를 잡아둔다. 이 줄 수 이후만 본다.
|
||||
LOG_BEFORE=$(kubectl -n "$NS" logs deploy/postgres --tail=-1 2>/dev/null | wc -l)
|
||||
|
||||
RESULT=$(kubectl -n "$NS" run kc-readpath --rm -i --restart=Never \
|
||||
--image=curlimages/curl:8.11.1 --quiet --command -- sh -c "
|
||||
O=/tmp/o; : > \$O
|
||||
TOKEN_EP='/realms/master/protocol/openid-connect/token'
|
||||
jget() { sed -n \"s/.*\\\"\$1\\\":\\\"\\([^\\\"]*\\)\\\".*/\\1/p\"; }
|
||||
ent() {
|
||||
curl -s --retry 3 --max-time 20 http://\$1:9000/metrics \
|
||||
| grep -E '^vendor_statistics_approximate_entries_unique.cache=.sessions' | awk '{print \$NF}'
|
||||
}
|
||||
# keycloak-0 에서 로그인한다
|
||||
L=\$(curl -s -X POST \"http://$K0_IP:8080\$TOKEN_EP\" -d grant_type=password \
|
||||
-d client_id=admin-cli -d username=admin -d 'password=$ADMIN_PW')
|
||||
SID=\$(echo \"\$L\" | jget access_token | cut -d. -f2 | sed 's/\$/==/' | base64 -d 2>/dev/null | jget sid)
|
||||
RT=\$(echo \"\$L\" | jget refresh_token)
|
||||
echo \"SID=\$SID\" >> \$O
|
||||
echo \"K1_ENTRIES_BEFORE=\$(ent $K1_IP)\" >> \$O
|
||||
sleep 2
|
||||
# 반대편 노드에 refresh 를 딱 한 번 보낸다
|
||||
# 인용을 한 겹 더 쌓으면 curl 이 URL 을 통째로 못 읽는다. 실제로 000 이 나왔다.
|
||||
CODE=\$(curl -s -o /dev/null -w '%{http_code}' -X POST \
|
||||
\"http://$K1_IP:8080\$TOKEN_EP\" \
|
||||
-d grant_type=refresh_token -d client_id=admin-cli -d \"refresh_token=\$RT\")
|
||||
echo \"REFRESH_ON_K1=\$CODE\" >> \$O
|
||||
sleep 3
|
||||
echo \"K1_ENTRIES_AFTER=\$(ent $K1_IP)\" >> \$O
|
||||
cat \$O
|
||||
" 2>&1 | grep -v '^pod .* deleted$')
|
||||
|
||||
echo "=== 요청 ==="
|
||||
echo "$RESULT" | sed 's/^/ /'
|
||||
SID=$(echo "$RESULT" | sed -n 's/^SID=//p')
|
||||
|
||||
echo
|
||||
echo "=== PostgreSQL 문장 로깅을 끈다 ==="
|
||||
$PSQL "alter system reset log_statement" >/dev/null 2>&1
|
||||
$PSQL "alter system reset log_line_prefix" >/dev/null 2>&1
|
||||
$PSQL "select pg_reload_conf()" >/dev/null 2>&1
|
||||
echo " log_statement = $($PSQL 'show log_statement' 2>/dev/null)"
|
||||
|
||||
echo
|
||||
echo "=== keycloak-1 이 실제로 보낸 SQL 문장 ==="
|
||||
echo " (파라미터가 \$1 로 묶여 있어, sid 는 바로 아래 DETAIL 줄에 있다)"
|
||||
echo
|
||||
kubectl -n "$NS" logs deploy/postgres --tail=-1 2>/dev/null \
|
||||
| tail -n +$((LOG_BEFORE + 1)) \
|
||||
| grep -F "$K1_IP" | grep -E "LOG: execute" \
|
||||
| sed 's/.*execute [^:]*: //' | sed 's/^/ /' | head -12
|
||||
echo
|
||||
echo "=== 그 sid 를 언급한 SQL — 누가 보냈는가 ==="
|
||||
echo " 찾는 sid: $SID"
|
||||
echo
|
||||
kubectl -n "$NS" logs deploy/postgres --tail=-1 2>/dev/null \
|
||||
| tail -n +$((LOG_BEFORE + 1)) \
|
||||
| grep -F "$SID" \
|
||||
| sed -e "s/$K0_IP/[keycloak-0]/g" -e "s/$K1_IP/[keycloak-1]/g" \
|
||||
| cut -c1-220 \
|
||||
| head -20
|
||||
|
||||
echo
|
||||
echo "=== 요약: 파드별 질의 건수 ==="
|
||||
kubectl -n "$NS" logs deploy/postgres --tail=-1 2>/dev/null \
|
||||
| tail -n +$((LOG_BEFORE + 1)) \
|
||||
| grep -F "$SID" \
|
||||
| grep -oE "^[0-9-]+ [0-9:.]+ [A-Z]+ \[[0-9]+\] [0-9.]+" \
|
||||
| awk '{print $NF}' | sort | uniq -c \
|
||||
| sed -e "s/$K0_IP/[keycloak-0]/" -e "s/$K1_IP/[keycloak-1]/" -e 's/^/ /'
|
||||
+207
@@ -0,0 +1,207 @@
|
||||
#!/usr/bin/env bash
|
||||
# Experiment 0 — is a session created on one Keycloak node usable on the other?
|
||||
#
|
||||
# Forming a cluster is not the same as sharing session state. The Infinispan log
|
||||
# says "cluster view (2)", but that only proves the members found each other.
|
||||
#
|
||||
# Design notes, learned the hard way:
|
||||
#
|
||||
# * Every probe has a CONTROL. A result from the far node means nothing unless
|
||||
# the same call against the issuing node is also measured. The first version
|
||||
# of this script reported "403 on keycloak-1" as if it were a replication
|
||||
# failure; the issuing node returned 403 too, and the cause was a missing
|
||||
# openid scope. Measure both, always.
|
||||
#
|
||||
# * Sessions are tracked by SID, not by count. Both the test login and the
|
||||
# admin API calls create sessions for the same user, so counts are noisy.
|
||||
# A specific session id either appears in a node's answer or it does not.
|
||||
#
|
||||
# * The probe is the REFRESH TOKEN grant, not userinfo. userinfo only validates
|
||||
# a signature and can succeed on a node that knows nothing about the session.
|
||||
# Refreshing requires the node to find the session, check it is alive, and
|
||||
# write back a new refresh time — it actually touches the session store.
|
||||
#
|
||||
# Talks to pod IPs directly: going through nginx/Traefik would hide which node
|
||||
# handled each request, which is the entire question.
|
||||
#
|
||||
# ./deploy/lab/scripts/experiment-session-replication.sh
|
||||
set -uo pipefail
|
||||
|
||||
NS="${NS:-keycloak-lab}"
|
||||
OUT="${OUT:-/tmp/session-replication}"
|
||||
mkdir -p "$OUT"
|
||||
|
||||
PSQL="kubectl -n $NS exec deploy/postgres -- psql -U keycloak -d keycloak -tAc"
|
||||
|
||||
echo "수집 시각: $(date '+%Y-%m-%d %H:%M:%S %Z')"
|
||||
echo
|
||||
|
||||
K0_IP=$(kubectl -n "$NS" get pod keycloak-0 -o jsonpath='{.status.podIP}')
|
||||
K1_IP=$(kubectl -n "$NS" get pod keycloak-1 -o jsonpath='{.status.podIP}')
|
||||
K0_NODE=$(kubectl -n "$NS" get pod keycloak-0 -o jsonpath='{.spec.nodeName}')
|
||||
K1_NODE=$(kubectl -n "$NS" get pod keycloak-1 -o jsonpath='{.spec.nodeName}')
|
||||
ADMIN_PW=$(kubectl -n "$NS" get secret keycloak-lab-secrets \
|
||||
-o jsonpath='{.data.KC_BOOTSTRAP_ADMIN_PASSWORD}' | base64 -d)
|
||||
|
||||
echo "=== 대상 ==="
|
||||
printf ' keycloak-0 %-14s %s\n' "$K0_IP" "$K0_NODE"
|
||||
printf ' keycloak-1 %-14s %s\n' "$K1_IP" "$K1_NODE"
|
||||
echo
|
||||
|
||||
echo "=== [0] 실험 전 DB 세션 ==="
|
||||
$PSQL "select offline_flag, count(*) from offline_user_session group by offline_flag" 2>/dev/null \
|
||||
| sed 's/^/ offline_flag=/' || echo " (없음)"
|
||||
echo
|
||||
|
||||
# 파드 하나 안에서 전 단계를 실행한다. 단계마다 파드를 새로 띄우면 토큰을
|
||||
# 단계 사이로 넘길 수 없다.
|
||||
kubectl -n "$NS" run kc-probe --rm -i --restart=Never \
|
||||
--image=curlimages/curl:8.11.1 --quiet --command -- sh -c "
|
||||
set -u
|
||||
K0='http://$K0_IP:8080'; K1='http://$K1_IP:8080'
|
||||
TOKEN_EP='/realms/master/protocol/openid-connect/token'
|
||||
jget() { sed -n \"s/.*\\\"\$1\\\":\\\"\\([^\\\"]*\\)\\\".*/\\1/p\"; }
|
||||
|
||||
# ── [1] keycloak-0 에서 로그인. 이 노드가 세션의 출생지다 ──────────────────
|
||||
LOGIN=\$(curl -s -X POST \"\$K0\$TOKEN_EP\" \
|
||||
-d grant_type=password -d client_id=admin-cli \
|
||||
-d username=admin -d 'password=$ADMIN_PW')
|
||||
echo '###STEP1_LOGIN'; echo \"\$LOGIN\"
|
||||
|
||||
AT=\$(echo \"\$LOGIN\" | jget access_token)
|
||||
RT=\$(echo \"\$LOGIN\" | jget refresh_token)
|
||||
|
||||
# ── [2] 관리 API 조회용 토큰. 세션 오염을 피하려고 따로 하나만 더 만든다 ──
|
||||
ADMTOK=\$(curl -s -X POST \"\$K0\$TOKEN_EP\" \
|
||||
-d grant_type=password -d client_id=admin-cli \
|
||||
-d username=admin -d 'password=$ADMIN_PW' | jget access_token)
|
||||
CID=\$(curl -s -H \"Authorization: Bearer \$ADMTOK\" \
|
||||
\"\$K0/admin/realms/master/clients?clientId=admin-cli\" | jget id | head -1)
|
||||
|
||||
# ── [3] 두 노드에 같은 질문을 한다: admin-cli 의 세션 목록 ────────────────
|
||||
echo '###STEP3_SESSIONS_K0'
|
||||
curl -s -H \"Authorization: Bearer \$ADMTOK\" \
|
||||
\"\$K0/admin/realms/master/clients/\$CID/user-sessions?max=100\"
|
||||
echo
|
||||
echo '###STEP3_SESSIONS_K1'
|
||||
curl -s -H \"Authorization: Bearer \$ADMTOK\" \
|
||||
\"\$K1/admin/realms/master/clients/\$CID/user-sessions?max=100\"
|
||||
echo
|
||||
|
||||
# ── [4] 대조군: keycloak-0 이 발급한 refresh token 을 keycloak-0 에 쓴다 ──
|
||||
# 먼저 반대편에 써야 하므로 여기서는 쓰지 않고, 순서를 [5] 뒤로 미룬다.
|
||||
# refresh token 은 회전(rotation)되므로 한 번 쓰면 옛 것이 무효가 된다.
|
||||
# 따라서 '반대편 먼저'가 유일하게 의미 있는 순서다.
|
||||
|
||||
# ── [5] 시험군: keycloak-0 이 발급한 refresh token 을 keycloak-1 에 쓴다 ──
|
||||
echo '###STEP5_REFRESH_ON_K1'
|
||||
curl -s -w '\nhttp_code=%{http_code}\n' -X POST \"\$K1\$TOKEN_EP\" \
|
||||
-d grant_type=refresh_token -d client_id=admin-cli -d \"refresh_token=\$RT\"
|
||||
|
||||
RT2=\$(curl -s -X POST \"\$K1\$TOKEN_EP\" \
|
||||
-d grant_type=refresh_token -d client_id=admin-cli -d \"refresh_token=\$RT\" \
|
||||
| jget refresh_token)
|
||||
|
||||
# ── [6] 무효화가 반대 방향으로도 전파되는가 ───────────────────────────────
|
||||
# keycloak-1 에서 로그아웃시키고, keycloak-0 에서 갱신을 시도한다.
|
||||
echo '###STEP6_LOGOUT_VIA_K1'
|
||||
curl -s -o /dev/null -w 'http_code=%{http_code}\n' -X POST \"\$K1/realms/master/protocol/openid-connect/logout\" \
|
||||
-d client_id=admin-cli -d \"refresh_token=\$RT2\"
|
||||
|
||||
echo '###STEP7_REFRESH_ON_K0_AFTER_LOGOUT'
|
||||
curl -s -w '\nhttp_code=%{http_code}\n' -X POST \"\$K0\$TOKEN_EP\" \
|
||||
-d grant_type=refresh_token -d client_id=admin-cli -d \"refresh_token=\$RT2\"
|
||||
echo '###END'
|
||||
" > "$OUT/raw.txt" 2>&1
|
||||
|
||||
sed -i '/^pod .* deleted$/d' "$OUT/raw.txt"
|
||||
|
||||
python3 - "$OUT/raw.txt" <<'PY' | tee "$OUT/report.txt"
|
||||
import base64, json, sys
|
||||
|
||||
raw = open(sys.argv[1]).read()
|
||||
blocks, cur = {}, None
|
||||
for line in raw.splitlines():
|
||||
if line.startswith('###'):
|
||||
cur = line[3:]; blocks[cur] = []
|
||||
elif cur is not None:
|
||||
blocks[cur].append(line)
|
||||
get = lambda k: '\n'.join(blocks.get(k, [])).strip()
|
||||
|
||||
def j(s):
|
||||
try: return json.JSONDecoder().raw_decode(s.strip())[0]
|
||||
except Exception: return None
|
||||
|
||||
def claims(tok):
|
||||
p = tok.split('.')[1]; p += '=' * (-len(p) % 4)
|
||||
return json.loads(base64.urlsafe_b64decode(p))
|
||||
|
||||
login = j(get('STEP1_LOGIN'))
|
||||
if not login or 'access_token' not in login:
|
||||
print('로그인 실패:', get('STEP1_LOGIN')[:300]); sys.exit(1)
|
||||
|
||||
ac = claims(login['access_token'])
|
||||
rc = claims(login['refresh_token'])
|
||||
SID = ac['sid']
|
||||
print('=== [1] keycloak-0 에서 로그인 ===')
|
||||
print(f" sid {SID}")
|
||||
print(f" sub {ac.get('sub')}")
|
||||
print(f" iss {ac.get('iss')}")
|
||||
print(f" access 수명 {ac['exp']-ac['iat']}초")
|
||||
print(f" refresh 수명 {rc['exp']-rc['iat']}초 typ={rc.get('typ')}")
|
||||
print(f" refresh jti {rc.get('jti')}")
|
||||
|
||||
print()
|
||||
print('=== [3] 같은 sid 가 두 노드 모두에서 보이는가 ===')
|
||||
for step, who in (('STEP3_SESSIONS_K0', 'keycloak-0 (발급 노드)'),
|
||||
('STEP3_SESSIONS_K1', 'keycloak-1 (반대편)')):
|
||||
d = j(get(step))
|
||||
if d is None:
|
||||
print(f' {who:24} 파싱 실패: {get(step)[:120]}'); continue
|
||||
ids = [s.get('id') for s in d]
|
||||
mark = '보임 ✔' if SID in ids else '없음 ✘'
|
||||
print(f' {who:24} 세션 {len(ids)}개 중 대상 sid → {mark}')
|
||||
for s in d:
|
||||
if s.get('id') == SID:
|
||||
print(f" ipAddress={s.get('ipAddress')} start={s.get('start')} lastAccess={s.get('lastAccess')}")
|
||||
|
||||
def show(step, title, expect):
|
||||
print(); print(f'=== {title} ===')
|
||||
body = get(step)
|
||||
code = [l for l in body.splitlines() if l.startswith('http_code=')]
|
||||
code = code[0].split('=')[1] if code else '?'
|
||||
d = j(body)
|
||||
ok = '기대대로' if code == expect else f'기대({expect})와 다름'
|
||||
print(f' HTTP {code} ← {ok}')
|
||||
if d and 'access_token' in d:
|
||||
c = claims(d['access_token'])
|
||||
same = '동일 ✔' if c.get('sid') == SID else f"다름 ✘ ({c.get('sid')})"
|
||||
print(f' 새 토큰의 sid → {same}')
|
||||
elif d:
|
||||
print(f" error {d.get('error')}")
|
||||
print(f" error_description {d.get('error_description')}")
|
||||
|
||||
show('STEP5_REFRESH_ON_K1',
|
||||
'[5] keycloak-0 이 발급한 refresh token 을 keycloak-1 에 사용', '200')
|
||||
|
||||
print(); print('=== [6] keycloak-1 을 통해 로그아웃 ===')
|
||||
print(' ' + get('STEP6_LOGOUT_VIA_K1').strip())
|
||||
|
||||
show('STEP7_REFRESH_ON_K0_AFTER_LOGOUT',
|
||||
'[7] 로그아웃 후 keycloak-0 에서 갱신 시도 (무효화 전파)', '400')
|
||||
|
||||
open('/tmp/session-replication/sid.txt','w').write(SID)
|
||||
PY
|
||||
|
||||
SID=$(cat /tmp/session-replication/sid.txt 2>/dev/null)
|
||||
echo
|
||||
echo "=== [8] PostgreSQL 에서 그 sid 를 직접 확인 ==="
|
||||
echo " 대상 sid: $SID"
|
||||
$PSQL "select user_session_id, offline_flag, created_on, last_session_refresh
|
||||
from offline_user_session where user_session_id='$SID'" 2>/dev/null \
|
||||
| sed 's/^/ /' | grep -q . \
|
||||
&& $PSQL "select user_session_id||' | flag='||offline_flag||' | created='||created_on||' | refresh='||last_session_refresh
|
||||
from offline_user_session where user_session_id='$SID'" 2>/dev/null | sed 's/^/ /' \
|
||||
|| echo " 행 없음 — 로그아웃으로 삭제되었다"
|
||||
echo
|
||||
echo " 전체 세션 수: $($PSQL 'select count(*) from offline_user_session' 2>/dev/null)"
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
#!/usr/bin/env bash
|
||||
# Measure what the nginx -> Traefik chain actually delivers to the application.
|
||||
#
|
||||
# docs/reverse-proxy-headers.md documents a single-hop nginx contract. The lab
|
||||
# runs two hops, so the forwarded headers are measured rather than assumed.
|
||||
# Run from anywhere that can resolve the lab hostnames.
|
||||
#
|
||||
# ./deploy/lab/scripts/measure-proxy-headers.sh
|
||||
set -euo pipefail
|
||||
|
||||
HOST="${HOST:-app1.hyeonworks.com}"
|
||||
URL="https://${HOST}/api/echo"
|
||||
|
||||
jqf() {
|
||||
if command -v jq >/dev/null 2>&1; then jq "$@"; else python3 -m json.tool; fi
|
||||
}
|
||||
|
||||
echo "=== 1. baseline: what the app sees for a normal request ==="
|
||||
curl -s "$URL" | jqf '{
|
||||
scheme, secure, serverName, serverPort, requestUrl, remoteAddr,
|
||||
forwarded: .headers | with_entries(select(.key | startswith("x-forwarded") or . == "x-real-ip" or . == "forwarded"))
|
||||
}' 2>/dev/null || curl -s "$URL"
|
||||
|
||||
echo
|
||||
echo "=== 2. spoof test: client sends its own X-Forwarded-* ==="
|
||||
echo " a trusted boundary must overwrite these, not append to them"
|
||||
curl -s "$URL" \
|
||||
-H 'X-Forwarded-For: 1.2.3.4' \
|
||||
-H 'X-Forwarded-Proto: http' \
|
||||
-H 'X-Forwarded-Host: evil.example.com' \
|
||||
-H 'X-Real-IP: 1.2.3.4' \
|
||||
| jqf '.headers | with_entries(select(.key | startswith("x-forwarded") or . == "x-real-ip"))' 2>/dev/null
|
||||
|
||||
echo
|
||||
echo "=== 3. which pod answered (host nginx upstream distribution) ==="
|
||||
for _ in 1 2 3 4; do
|
||||
curl -s "$URL" | jqf -r '.headers["x-forwarded-server"] // "n/a"' 2>/dev/null
|
||||
done
|
||||
|
||||
echo
|
||||
echo "=== 4. plain HTTP is redirected, not proxied ==="
|
||||
curl -s -o /dev/null -w ' http -> %{http_code} %{redirect_url}\n' "http://${HOST}/api/echo"
|
||||
@@ -0,0 +1,47 @@
|
||||
#!/usr/bin/env bash
|
||||
# Rebuild a guest's cloud-init seed image and publish it into the libvirt pool.
|
||||
# Run on the lab host.
|
||||
#
|
||||
# ./rebuild-seed.sh 1
|
||||
#
|
||||
# The same content lives in three places: the source YAML, the ISO, and the
|
||||
# uploaded pool volume. Editing the YAML alone changes nothing, which is why
|
||||
# this is a script and not a set of remembered commands.
|
||||
#
|
||||
# A rebuilt seed only takes effect on a freshly created VM. cloud-init runs its
|
||||
# per-instance modules once per instance-id, so an existing guest ignores it.
|
||||
set -euo pipefail
|
||||
|
||||
N="${1:?usage: rebuild-seed.sh <1|2>}"
|
||||
CLOUD_DIR="${CLOUD_DIR:-$HOME/workspace/cloud}"
|
||||
POOL="${POOL:-default}"
|
||||
export LIBVIRT_DEFAULT_URI="${LIBVIRT_DEFAULT_URI:-qemu:///system}"
|
||||
|
||||
cd "$CLOUD_DIR"
|
||||
src="kc-lab-${N}.yaml"
|
||||
iso="seed-kc-lab-${N}.iso"
|
||||
meta="meta-kc-lab-${N}"
|
||||
|
||||
[ -f "$src" ] || { echo "missing $CLOUD_DIR/$src" >&2; exit 1; }
|
||||
|
||||
# A fresh instance-id makes cloud-init treat the guest as new and re-run the
|
||||
# per-instance modules.
|
||||
printf 'instance-id: kc-lab-%s-%s\nlocal-hostname: kc-lab-%s\n' \
|
||||
"$N" "$(date +%s)" "$N" > "$meta"
|
||||
|
||||
# NoCloud looks for a volume labelled cidata holding files named exactly
|
||||
# user-data and meta-data. -graft-points renames them inside the image so no
|
||||
# staging directory is needed.
|
||||
xorrisofs -quiet -output "$iso" -volid CIDATA -joliet -rock -graft-points \
|
||||
"/user-data=${src}" "/meta-data=${meta}"
|
||||
|
||||
size="$(stat -c%s "$iso")"
|
||||
virsh vol-delete --pool "$POOL" "$iso" >/dev/null 2>&1 || true
|
||||
virsh vol-create-as "$POOL" "$iso" "$size" --format raw >/dev/null
|
||||
virsh vol-upload --pool "$POOL" "$iso" "$iso"
|
||||
|
||||
echo "$iso published to pool '$POOL' ($size bytes)"
|
||||
echo "attach it as a virtio disk, not a SATA cdrom:"
|
||||
echo " --disk vol=${POOL}/${iso},device=disk,bus=virtio,readonly=on"
|
||||
echo "Debian genericcloud images carry no AHCI driver, so a SATA cdrom is invisible"
|
||||
echo "to the guest and cloud-init fails with no error anywhere."
|
||||
@@ -0,0 +1,47 @@
|
||||
#!/usr/bin/env bash
|
||||
# Confirm the lab infrastructure is intact. Run on the lab host.
|
||||
#
|
||||
# A 404 from the HTTPS entry point is the success signal: TLS terminated and the
|
||||
# request reached Traefik, which simply had no matching ingress rule. A 502 or a
|
||||
# refused connection means the chain is broken somewhere.
|
||||
set -uo pipefail
|
||||
|
||||
export LIBVIRT_DEFAULT_URI="${LIBVIRT_DEFAULT_URI:-qemu:///system}"
|
||||
HOSTS="${HOSTS:-auth.hyeonworks.com app1.hyeonworks.com app2.hyeonworks.com}"
|
||||
NODE_IPS="${NODE_IPS:-192.168.122.11 192.168.122.12}"
|
||||
fail=0
|
||||
|
||||
check() { # description, expected, actual
|
||||
if [ "$2" = "$3" ]; then printf ' ok %-34s %s\n' "$1" "$3"
|
||||
else printf ' FAIL %-34s got %s, want %s\n' "$1" "$3" "$2"; fail=1; fi
|
||||
}
|
||||
|
||||
echo "== guests =="
|
||||
for name in kc-lab-1 kc-lab-2; do
|
||||
check "$name" running "$(virsh domstate "$name" 2>/dev/null || echo absent)"
|
||||
done
|
||||
|
||||
echo "== k3s =="
|
||||
ready="$(kubectl get nodes --no-headers 2>/dev/null | grep -c ' Ready ')"
|
||||
check "nodes Ready" 2 "$ready"
|
||||
lb="$(kubectl -n kube-system get svc traefik \
|
||||
-o jsonpath='{.status.loadBalancer.ingress[*].ip}' 2>/dev/null | wc -w)"
|
||||
check "traefik node IPs" 2 "$lb"
|
||||
|
||||
echo "== host nginx =="
|
||||
check "service" active "$(systemctl is-active nginx)"
|
||||
check "cert renew timer" active "$(systemctl is-active certbot-renew.timer)"
|
||||
for ip in $NODE_IPS; do
|
||||
check "traefik $ip" 404 "$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "http://${ip}/")"
|
||||
done
|
||||
|
||||
echo "== public entry point =="
|
||||
for h in $HOSTS; do
|
||||
check "https://$h" 404 "$(curl -s -o /dev/null -w '%{http_code}' --max-time 8 "https://${h}/")"
|
||||
check "tls verify $h" 0 "$(curl -s -o /dev/null -w '%{ssl_verify_result}' --max-time 8 "https://${h}/")"
|
||||
done
|
||||
check "http redirect" 301 "$(curl -s -o /dev/null -w '%{http_code}' --max-time 8 "http://${HOSTS%% *}/")"
|
||||
|
||||
echo
|
||||
[ "$fail" -eq 0 ] && echo "lab is healthy" || echo "lab has failures"
|
||||
exit "$fail"
|
||||
@@ -0,0 +1,6 @@
|
||||
# Keycloak receives HTTP only from the trusted reverse proxy.
|
||||
KC_HTTP_ENABLED=true
|
||||
KC_PROXY_HEADERS=xforwarded
|
||||
KC_HOSTNAME=https://auth.example.test
|
||||
KC_HOSTNAME_STRICT=true
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
server {
|
||||
listen 8080;
|
||||
server_name auth.example.test;
|
||||
|
||||
location / {
|
||||
proxy_pass http://keycloak:8080;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-Host $host;
|
||||
proxy_set_header X-Forwarded-Port 443;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
auth.example.test {
|
||||
tls /etc/tls/tls.crt /etc/tls/tls.key
|
||||
|
||||
reverse_proxy keycloak:8080 {
|
||||
header_up Host {host}
|
||||
header_up X-Forwarded-Host {host}
|
||||
header_up X-Forwarded-Port 443
|
||||
header_up X-Forwarded-Proto https
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
events {}
|
||||
|
||||
http {
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name auth.example.test;
|
||||
|
||||
ssl_certificate /etc/tls/tls.crt;
|
||||
ssl_certificate_key /etc/tls/tls.key;
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
|
||||
location / {
|
||||
proxy_pass http://keycloak:8080;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-Host $host;
|
||||
proxy_set_header X-Forwarded-Port 443;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
tunnel: 00000000-0000-0000-0000-000000000000
|
||||
credentials-file: /etc/cloudflared/00000000-0000-0000-0000-000000000000.json
|
||||
|
||||
ingress:
|
||||
- hostname: auth.example.test
|
||||
service: http://reverse-proxy:8080
|
||||
- service: http_status:404
|
||||
@@ -0,0 +1,18 @@
|
||||
# Federated account key: `sub`, not email
|
||||
|
||||
외부 IdP의 email은 표시·연락 속성이지 계정 식별자나 자동 연결 증명이 아니다.
|
||||
Keycloak의 federated identity는 provider alias와 provider user ID(`sub`)를
|
||||
로컬 사용자에 연결한다.
|
||||
|
||||
정책:
|
||||
|
||||
- 신규 identity의 email이 기존 로컬 계정과 충돌하면 기존 계정의 인증을 다시
|
||||
요구하는 기본 First Broker Login flow를 사용한다.
|
||||
- `Automatically Set Existing User`를 production flow에 넣지 않는다.
|
||||
- upstream email 변경은 같은 `sub`의 계정 귀속을 바꾸지 않는다.
|
||||
- 마지막 로그인 수단을 unlink하는 UI에서는 먼저 다른 인증 수단을 등록하도록
|
||||
안내한다.
|
||||
|
||||
`verify-account-linking-sub-vs-email.sh`는 mock IdP 사용자의 email을 실제로
|
||||
변경하고 다시 로그인한다. 로컬 사용자 ID가 유지되고 federated `userId`가
|
||||
upstream `sub`와 같은지 확인한 후 원래 email을 복구한다.
|
||||
@@ -0,0 +1,12 @@
|
||||
# 다이어그램 규약
|
||||
|
||||
| 표현 | 뜻 |
|
||||
|---|---|
|
||||
| 실선 상자 | 살아 있는 구성 요소 |
|
||||
| 붉은 점선 상자 | 이 실험에서 죽이거나 막은 것 |
|
||||
| ✂ 붉은 X | 주입 지점 |
|
||||
| 실선 화살표 | 정상 경로 |
|
||||
| 붉은 점선 화살표 | 실험에서 깨진 경로 |
|
||||
| 회색 글씨 | 측정값 |
|
||||
|
||||
SVG 는 GitHub 에서 그대로 렌더링되며 외부 폰트를 쓰지 않는다.
|
||||
@@ -0,0 +1,48 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 400" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
|
||||
<style>
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
|
||||
.t{font-size:12px;fill:#1f2328}
|
||||
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
|
||||
.s{font-size:10.5px;fill:#656d76}
|
||||
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
|
||||
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
|
||||
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
|
||||
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
|
||||
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
|
||||
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
|
||||
</defs>
|
||||
<text x="20" y="26" class="ttl">A-0 — 세션을 공유하는 것은 Infinispan 이 아니라 PostgreSQL 이다</text>
|
||||
<text x="20" y="44" class="s">클러스터가 형성됐다는 것과 세션이 복제된다는 것은 다른 얘기였다</text>
|
||||
<rect x="30" y="76" width="330" height="120" class="box"/>
|
||||
<text x="46" y="98" class="tb">keycloak-0 (kc-lab-2)</text>
|
||||
<rect x="46" y="112" width="298" height="34" class="box"/>
|
||||
<text x="58" y="134" class="t">Infinispan sessions 캐시</text>
|
||||
<text x="46" y="166" class="s">자기가 처리한 로그인만 들어 있다</text>
|
||||
<text x="46" y="184" class="s">entries_unique 는 노드마다 다르다</text>
|
||||
<rect x="500" y="76" width="330" height="120" class="box"/>
|
||||
<text x="516" y="98" class="tb">keycloak-1 (kc-lab-1)</text>
|
||||
<rect x="516" y="112" width="298" height="34" class="box"/>
|
||||
<text x="528" y="134" class="t">Infinispan sessions 캐시</text>
|
||||
<text x="516" y="166" class="s">여기에도 자기 것만 있다</text>
|
||||
<text x="516" y="184" class="s">상대 세션은 들어오지 않는다</text>
|
||||
<path d="M362 130 L498 130" class="r" stroke-dasharray="5 3" marker-end="url(#ar)"/>
|
||||
<path d="M498 146 L362 146" class="r" stroke-dasharray="5 3" marker-end="url(#ar)"/>
|
||||
<text x="430" y="122" class="tb" text-anchor="middle" fill="#cf222e">복제 없음</text>
|
||||
<text x="430" y="170" class="s" text-anchor="middle" fill="#cf222e">세션 엔트리는 노드 사이를 건너가지 않는다</text>
|
||||
<rect x="300" y="250" width="260" height="76" class="ok"/>
|
||||
<text x="316" y="272" class="tb">PostgreSQL</text>
|
||||
<text x="316" y="292" class="s">OFFLINE_USER_SESSION (offline_flag='0')</text>
|
||||
<text x="316" y="310" class="s">두 노드가 같은 행을 본다</text>
|
||||
<path d="M150 198 L330 246" class="g" marker-end="url(#ag)"/>
|
||||
<path d="M710 198 L530 246" class="g" marker-end="url(#ag)"/>
|
||||
<text x="180" y="232" class="s">SELECT / INSERT</text>
|
||||
<text x="600" y="232" class="s">SELECT / INSERT</text>
|
||||
<rect x="30" y="346" width="800" height="40" class="box"/>
|
||||
<text x="46" y="370" class="s">근거 — 노드 A 로 로그인하고 노드 B 로 refresh 했을 때, 반대편 노드가 날린 SQL 을 문장 로깅으로 직접 잡았다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.2 KiB |
@@ -0,0 +1,47 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 430" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
|
||||
<style>
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
|
||||
.t{font-size:12px;fill:#1f2328}
|
||||
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
|
||||
.s{font-size:10.5px;fill:#656d76}
|
||||
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
|
||||
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
|
||||
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
|
||||
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
|
||||
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
|
||||
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
|
||||
</defs>
|
||||
<text x="20" y="26" class="ttl">A-1 — 7800 을 막아도 세션 공유는 안 깨진다. 깨지는 것은 로그아웃 전파다</text>
|
||||
<text x="20" y="44" class="s">예측 하나가 빗나갔고, 예상 못 한 것이 둘 나왔다</text>
|
||||
<rect x="30" y="72" width="300" height="104" class="box"/>
|
||||
<text x="46" y="94" class="tb">keycloak-0</text>
|
||||
<text x="46" y="116" class="s">로그인 처리 · 세션을 DB 에 쓴다</text>
|
||||
<rect x="530" y="72" width="300" height="104" class="box"/>
|
||||
<text x="546" y="94" class="tb">keycloak-1</text>
|
||||
<text x="546" y="116" class="s">DB 를 읽어 같은 세션을 안다</text>
|
||||
<rect x="352" y="86" width="156" height="34" class="dead"/>
|
||||
<text x="430" y="107" class="t" text-anchor="middle" fill="#cf222e">TCP 7800 차단</text>
|
||||
<text x="430" y="140" class="s" text-anchor="middle">NetworkPolicy 는 허용목록이다</text>
|
||||
<text x="430" y="156" class="s" text-anchor="middle">8080·9000 만 열고 7800 은 누락시킨다</text>
|
||||
<rect x="300" y="206" width="260" height="64" class="ok"/>
|
||||
<text x="316" y="228" class="tb">PostgreSQL</text>
|
||||
<text x="316" y="248" class="s">세션은 여기 있다 → 교차 노드 refresh 200</text>
|
||||
<path d="M150 178 L320 202" class="g" marker-end="url(#ag)"/>
|
||||
<path d="M710 178 L540 202" class="g" marker-end="url(#ag)"/>
|
||||
<rect x="30" y="290" width="390" height="76" class="dead"/>
|
||||
<text x="46" y="312" class="tb">깨진 것 — 로그아웃 전파</text>
|
||||
<text x="46" y="332" class="s">무효화 통지가 7800 을 탄다</text>
|
||||
<text x="46" y="350" class="s">400 이어야 할 refresh 가 200 을 반환했다</text>
|
||||
<rect x="440" y="290" width="390" height="76" class="warn"/>
|
||||
<text x="456" y="312" class="tb">★ 주입이 먹지 않았다 — conntrack</text>
|
||||
<text x="456" y="332" class="s">ESTABLISHED 연결은 규칙 평가를 건너뛴다</text>
|
||||
<text x="456" y="350" class="s">cluster_size 가 25분간 2 로 남았다 → 삭제 후에야 갈렸다</text>
|
||||
<text x="30" y="392" class="s">그리고 예상 못 한 둘째 — 분단된 노드가 readiness 실패로 스스로 로드밸런서에서 빠진다</text>
|
||||
<text x="30" y="410" class="s">덕분에 외부에서는 장애가 보이지 않는다. 관측 지점을 밖에만 두면 이 실험은 「아무 일도 없음」으로 보인다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.3 KiB |
@@ -0,0 +1,26 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 300" width="700" height="300" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
|
||||
<style>
|
||||
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
|
||||
.r{font-size:12px;fill:#cf222e;font-weight:600}
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
|
||||
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
|
||||
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
|
||||
</defs>
|
||||
<text class="h" x="16" y="24">A-2 · PostgreSQL 정지 — 살아남는 노드가 없다</text>
|
||||
<rect class="box" x="30" y="46" width="180" height="52"/><text class="t" x="120" y="68" text-anchor="middle">keycloak-1</text><text class="s" x="120" y="86" text-anchor="middle">캐시: 세션 N개</text>
|
||||
<rect class="box" x="480" y="46" width="180" height="52"/><text class="t" x="570" y="68" text-anchor="middle">keycloak-0</text><text class="s" x="570" y="86" text-anchor="middle">캐시: 세션 M개</text>
|
||||
<path class="ln" d="M210,72 L480,72"/><text class="s" x="345" y="66" text-anchor="middle">7800 · 살아 있다</text>
|
||||
<path class="bad" d="M120,100 L300,150"/><path class="bad" d="M570,100 L400,150"/>
|
||||
<rect class="dead" x="270" y="156" width="160" height="52"/>
|
||||
<text class="r" x="350" y="178" text-anchor="middle">postgres ✗</text><text class="s" x="350" y="196" text-anchor="middle">replicas=0</text>
|
||||
<rect class="dead" x="30" y="228" width="290" height="52"/>
|
||||
<text class="r" x="175" y="250" text-anchor="middle">양쪽 모두 NotReady</text><text class="s" x="175" y="268" text-anchor="middle">ready 주소 = [] · 외부 503</text>
|
||||
<rect class="box" x="370" y="228" width="290" height="52"/>
|
||||
<text class="t" x="515" y="250" text-anchor="middle">up{job="keycloak"} = 1</text><text class="s" x="515" y="268" text-anchor="middle">프로세스는 살아 있다 — up 은 못 잡는다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.4 KiB |
@@ -0,0 +1,55 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 400" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
|
||||
<style>
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
|
||||
.t{font-size:12px;fill:#1f2328}
|
||||
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
|
||||
.s{font-size:10.5px;fill:#656d76}
|
||||
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
|
||||
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
|
||||
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
|
||||
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
|
||||
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
|
||||
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
|
||||
</defs>
|
||||
<text x="20" y="26" class="ttl">A-3 — 200 을 받은 로그인 153건 중 4건이 DB 에 없다</text>
|
||||
<text x="20" y="44" class="s">버그가 아니라 synchronous_commit OFF 의 대가를 실측한 것이다</text>
|
||||
<rect x="30" y="74" width="180" height="70" class="box"/>
|
||||
<text x="46" y="96" class="tb">클라이언트</text>
|
||||
<text x="46" y="116" class="s">로그인 요청</text>
|
||||
<text x="46" y="134" class="s">200 + 토큰 수신</text>
|
||||
<rect x="250" y="74" width="200" height="70" class="box"/>
|
||||
<text x="266" y="96" class="tb">Keycloak</text>
|
||||
<text x="266" y="116" class="s">트랜잭션마다</text>
|
||||
<text x="266" y="134" class="s">SET LOCAL synchronous_commit OFF</text>
|
||||
<rect x="490" y="74" width="180" height="70" class="warn"/>
|
||||
<text x="506" y="96" class="tb">PostgreSQL</text>
|
||||
<text x="506" y="116" class="s">COMMIT 을 즉시 반환</text>
|
||||
<text x="506" y="134" class="s">WAL 은 아직 메모리에</text>
|
||||
<rect x="710" y="74" width="120" height="70" class="box"/>
|
||||
<text x="726" y="96" class="tb">디스크</text>
|
||||
<text x="726" y="116" class="s">wal_writer_delay</text>
|
||||
<text x="726" y="134" class="s">200ms 뒤 기록</text>
|
||||
<path d="M212 108 L246 108" class="k" marker-end="url(#a)"/>
|
||||
<path d="M452 108 L486 108" class="k" marker-end="url(#a)"/>
|
||||
<path d="M672 108 L706 108" class="r" stroke-dasharray="4 3" marker-end="url(#ar)"/>
|
||||
<text x="676" y="164" class="s" fill="#cf222e">이 구간이 비어 있다</text>
|
||||
<rect x="620" y="180" width="210" height="40" class="dead"/>
|
||||
<text x="636" y="205" class="t" fill="#cf222e">✂ 여기서 강제 종료</text>
|
||||
<rect x="30" y="250" width="380" height="96" class="dead"/>
|
||||
<text x="46" y="272" class="tb">결과 — RPO 는 0 이 아니다</text>
|
||||
<text x="46" y="296" class="t">클라이언트가 200 받은 로그인 153 건</text>
|
||||
<text x="46" y="316" class="t">그중 DB 에 실제로 존재 149 건</text>
|
||||
<text x="46" y="336" class="t" fill="#cf222e">★ 유실 4 건</text>
|
||||
<rect x="430" y="250" width="400" height="96" class="box"/>
|
||||
<text x="446" y="272" class="tb">주입도 두 번 실패했다</text>
|
||||
<text x="446" y="294" class="s">kubectl delete --force 는 크래시가 아니다 — 런타임이 SIGTERM 을 보내</text>
|
||||
<text x="446" y="310" class="s">PostgreSQL 이 정상 플러시했고 유실이 0 이었다</text>
|
||||
<text x="446" y="332" class="s">kill -9 1 도 무시된다 — PID 1 은 자기 네임스페이스의 SIGKILL 을 받지 않는다</text>
|
||||
<text x="30" y="372" class="s">백엔드 프로세스를 죽여 postmaster 가 reinitialize 하게 만들자 비로소 "not properly shut down / redo starts" 가 찍혔다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.7 KiB |
@@ -0,0 +1,52 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 420" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
|
||||
<style>
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
|
||||
.t{font-size:12px;fill:#1f2328}
|
||||
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
|
||||
.s{font-size:10.5px;fill:#656d76}
|
||||
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
|
||||
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
|
||||
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
|
||||
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
|
||||
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
|
||||
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
|
||||
</defs>
|
||||
<text x="20" y="26" class="ttl">A-4 — 둘 다 전면 장애지만 이유가 다르다</text>
|
||||
<text x="20" y="44" class="s">그리고 죽은 파드가 산 파드보다 건강해 보인다</text>
|
||||
<rect x="30" y="70" width="390" height="150" class="box"/>
|
||||
<text x="46" y="92" class="tb">4a — 워커(kc-lab-2) 상실</text>
|
||||
<rect x="46" y="104" width="170" height="46" class="dead"/>
|
||||
<text x="58" y="124" class="t">keycloak-0</text>
|
||||
<text x="58" y="142" class="s">PostgreSQL 도 여기 있었다</text>
|
||||
<rect x="236" y="104" width="170" height="46" class="warn"/>
|
||||
<text x="248" y="124" class="t">keycloak-1 살아있음</text>
|
||||
<text x="248" y="142" class="s">그러나 DB 가 없다</text>
|
||||
<text x="46" y="172" class="t">외부 응답 <tspan fill="#cf222e" font-weight="600">503</tspan> · kubectl 정상</text>
|
||||
<text x="46" y="194" class="s">DB 가 같이 죽어서 장애다. 남은 파드는 돌지만 아무것도 못 한다</text>
|
||||
<text x="46" y="212" class="s">PVC 가 local-path 라 다른 노드로 재배치되지 않는다</text>
|
||||
<rect x="440" y="70" width="390" height="150" class="box"/>
|
||||
<text x="456" y="92" class="tb">4b — 컨트롤 플레인(kc-lab-1) 상실</text>
|
||||
<rect x="456" y="104" width="170" height="46" class="ok"/>
|
||||
<text x="468" y="124" class="t">keycloak-0</text>
|
||||
<text x="468" y="142" class="s">★ 계속 돌고 있다</text>
|
||||
<rect x="646" y="104" width="170" height="46" class="dead"/>
|
||||
<text x="658" y="124" class="t">API 서버 · traefik</text>
|
||||
<text x="658" y="142" class="s">들어갈 길이 없다</text>
|
||||
<text x="456" y="172" class="t">외부 응답 <tspan fill="#cf222e" font-weight="600">000</tspan> · kubectl <tspan fill="#cf222e">불통</tspan></text>
|
||||
<text x="456" y="194" class="s">워크로드는 멀쩡한데 도달할 수 없어 장애다</text>
|
||||
<text x="456" y="212" class="s">진입점이 단일 노드에 있으면 워크로드 이중화는 의미가 없다</text>
|
||||
<rect x="30" y="242" width="800" height="86" class="warn"/>
|
||||
<text x="46" y="264" class="tb">★ 예상하지 못한 것 셋</text>
|
||||
<text x="46" y="286" class="s">1. 죽은 파드가 산 파드보다 건강해 보인다 — kubelet 이 사라져 상태가 갱신되지 않으니 Running 으로 남는다</text>
|
||||
<text x="46" y="304" class="s">2. StatefulSet 은 Terminating 파드의 대체를 만들지 않는다 — 이름이 같아야 하므로 지워지기를 기다린다</text>
|
||||
<text x="46" y="322" class="s">3. node-monitor-grace-period 40초 + tolerationSeconds 300초 = 축출까지 5분 40초</text>
|
||||
<rect x="30" y="348" width="800" height="46" class="ok"/>
|
||||
<text x="46" y="370" class="tb">복구는 둘 다 virsh start 이후 60초</text>
|
||||
<text x="46" y="388" class="s">장애 시간의 대부분은 복구가 아니라 "누가 죽은 것을 알아채는 데" 걸린 시간이다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.9 KiB |
@@ -0,0 +1,49 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 420" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
|
||||
<style>
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
|
||||
.t{font-size:12px;fill:#1f2328}
|
||||
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
|
||||
.s{font-size:10.5px;fill:#656d76}
|
||||
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
|
||||
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
|
||||
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
|
||||
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
|
||||
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
|
||||
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
|
||||
</defs>
|
||||
<text x="20" y="26" class="ttl">A-5 — 비대칭 차단은 클러스터를 가르지 못한다. 그리고 갈라도 서비스는 계속된다</text>
|
||||
<text x="20" y="44" class="s">주입을 세 번 실패했고, 세 번 모두 「아무 일도 없었다」로 보였다</text>
|
||||
<rect x="30" y="72" width="390" height="112" class="box"/>
|
||||
<text x="46" y="94" class="tb">① 한 방향만 차단</text>
|
||||
<rect x="46" y="106" width="130" height="34" class="box"/>
|
||||
<text x="58" y="128" class="t">keycloak-0</text>
|
||||
<rect x="290" y="106" width="116" height="34" class="box"/>
|
||||
<text x="302" y="128" class="t">keycloak-1</text>
|
||||
<path d="M178 116 L286 116" class="r" stroke-dasharray="4 3" marker-end="url(#ar)"/>
|
||||
<path d="M286 132 L178 132" class="g" marker-end="url(#ag)"/>
|
||||
<text x="46" y="164" class="s" fill="#1a7f37">열린 방향으로 재연결한다 → cluster_size 2 유지. 가르지 못한다</text>
|
||||
<rect x="440" y="72" width="390" height="112" class="box"/>
|
||||
<text x="456" y="94" class="tb">② 양방향 완전 차단</text>
|
||||
<rect x="456" y="106" width="130" height="34" class="dead"/>
|
||||
<text x="468" y="128" class="t">keycloak-0</text>
|
||||
<rect x="700" y="106" width="116" height="34" class="ok"/>
|
||||
<text x="712" y="128" class="t">keycloak-1</text>
|
||||
<path d="M588 116 L696 116" class="r" stroke-dasharray="4 3" marker-end="url(#ar)"/>
|
||||
<path d="M696 132 L588 132" class="r" stroke-dasharray="4 3" marker-end="url(#ar)"/>
|
||||
<text x="456" y="164" class="s">양쪽 모두 멤버 1개. 그런데 <tspan fill="#cf222e" font-weight="600">한쪽만 DOWN</tspan> 이 된다 — 코디네이터 쪽이 살아남는다</text>
|
||||
<rect x="30" y="204" width="800" height="48" class="ok"/>
|
||||
<text x="46" y="226" class="tb">그래서 전면 장애 경로가 없다 — 외부 응답 200 유지</text>
|
||||
<text x="46" y="244" class="s">분단된 쪽이 스스로 readiness 를 떨어뜨려 로드밸런서에서 빠지고, 남은 쪽이 계속 응답한다</text>
|
||||
<rect x="30" y="272" width="800" height="130" class="warn"/>
|
||||
<text x="46" y="294" class="tb">★ 세 번의 주입 실패 — 전부 「조용히」 실패했다</text>
|
||||
<text x="46" y="318" class="s">1. iptables -I FORWARD 1 이 무시됐다 — kube-router 가 자기 체인을 FORWARD 맨 위에 다시 끼워 넣는다 (패킷 0)</text>
|
||||
<text x="46" y="338" class="s">2. 방향이 뒤집혀 있었다 — JGroups 의 client/server 역할은 재시작마다 바뀐다. raw 규칙이 엉뚱한 노드에 걸렸다</text>
|
||||
<text x="46" y="358" class="s">3. dev eth0 이 없다 — Debian 은 enp1s0 이고, 게다가 flannel VXLAN 이 이미 캡슐화해 파드 IP 가 안 보인다</text>
|
||||
<text x="46" y="382" class="s" fill="#cf222e">해결: raw 테이블 PREROUTING(conntrack 보다 먼저) + flannel.1 인터페이스에서 필터</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.8 KiB |
@@ -0,0 +1,51 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 400" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
|
||||
<style>
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
|
||||
.t{font-size:12px;fill:#1f2328}
|
||||
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
|
||||
.s{font-size:10.5px;fill:#656d76}
|
||||
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
|
||||
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
|
||||
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
|
||||
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
|
||||
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
|
||||
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
|
||||
</defs>
|
||||
<text x="20" y="26" class="ttl">A-6 — 200밀리초가 22초가 된다</text>
|
||||
<text x="20" y="44" class="s">지연은 왕복 횟수만큼 곱해지고, 커넥션 풀에서 한 번 더 곱해진다</text>
|
||||
<rect x="30" y="76" width="180" height="60" class="ok"/>
|
||||
<text x="46" y="98" class="tb">대조군 keycloak-0</text>
|
||||
<text x="46" y="120" class="s">DB 와 같은 노드 · 66ms</text>
|
||||
<rect x="250" y="76" width="180" height="60" class="dead"/>
|
||||
<text x="266" y="98" class="tb">시험군 keycloak-1</text>
|
||||
<text x="266" y="120" class="s">tc netem delay 200ms</text>
|
||||
<text x="230" y="112" class="tb" fill="#cf222e">✂</text>
|
||||
<rect x="470" y="76" width="180" height="60" class="warn"/>
|
||||
<text x="486" y="98" class="tb">단일 요청</text>
|
||||
<text x="486" y="120" class="t">1,872 ms <tspan class="s">(28배)</tspan></text>
|
||||
<rect x="690" y="76" width="140" height="60" class="dead"/>
|
||||
<text x="706" y="98" class="tb">동시 20건</text>
|
||||
<text x="706" y="120" class="t" fill="#cf222e">22.2 초</text>
|
||||
<path d="M432 106 L466 106" class="k" marker-end="url(#a)"/>
|
||||
<path d="M652 106 L686 106" class="r" marker-end="url(#ar)"/>
|
||||
<text x="20" y="176" class="tb">왜 200ms 가 1,872ms 가 되는가 — 왕복 횟수</text>
|
||||
<rect x="30" y="190" width="800" height="46" class="box"/>
|
||||
<text x="46" y="210" class="s">로그인 한 번은 DB 왕복을 여러 번 한다: 클라이언트 조회 · 사용자 조회 · 세션 INSERT · 커밋 …</text>
|
||||
<text x="46" y="228" class="s">왕복마다 200ms 가 더해진다. 애플리케이션은 아무것도 잘못하지 않았다</text>
|
||||
<text x="20" y="266" class="tb">왜 1,872ms 가 22.2초가 되는가 — 커넥션 풀</text>
|
||||
<rect x="30" y="280" width="390" height="76" class="dead"/>
|
||||
<text x="46" y="302" class="s">요청이 커넥션을 오래 붙들고 있으면</text>
|
||||
<text x="46" y="320" class="s">뒤의 요청은 풀에서 대기한다</text>
|
||||
<text x="46" y="342" class="t" fill="#cf222e">agroal 커넥션 획득 대기 최대 20,000 ms</text>
|
||||
<rect x="440" y="280" width="390" height="76" class="warn"/>
|
||||
<text x="456" y="302" class="tb">그리고 파드가 죽는다</text>
|
||||
<text x="456" y="322" class="s">readiness 프로브가 타임아웃으로 실패 →</text>
|
||||
<text x="456" y="342" class="s">느린 노드가 로드밸런서에서 빠진다. 느림이 장애로 승격된다</text>
|
||||
<text x="30" y="384" class="s">측정 장치 주의 — 동시 20건을 kubectl run --rm -i 로 돌리면 출력 스트림이 유실된다. 상주 탐침에 파일로 모아야 한다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.6 KiB |
@@ -0,0 +1,36 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 720 330" width="720" height="330" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
|
||||
<style>
|
||||
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
|
||||
.r{font-size:12px;fill:#cf222e;font-weight:600}.g{font-size:12px;fill:#1a7f37;font-weight:600}
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
|
||||
</style>
|
||||
<defs><marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker></defs>
|
||||
<text class="h" x="16" y="24">A-7 · 같은 주입, 같은 관측, 정반대 결과</text>
|
||||
|
||||
<text class="h" x="180" y="52" text-anchor="middle">persistent (KC 26 기본)</text>
|
||||
<rect class="box" x="30" y="62" width="300" height="54"/>
|
||||
<text class="t" x="180" y="84" text-anchor="middle">keycloak ×2 — 로컬 캐시</text>
|
||||
<path class="ln" d="M180,116 L180,140"/>
|
||||
<rect class="ok" x="70" y="142" width="220" height="40"/>
|
||||
<text class="g" x="180" y="167" text-anchor="middle">PostgreSQL — 진실의 원천</text>
|
||||
|
||||
<text class="h" x="540" y="52" text-anchor="middle">volatile (KC 24 이전 방식)</text>
|
||||
<rect class="box" x="390" y="62" width="300" height="54"/>
|
||||
<text class="t" x="540" y="84" text-anchor="middle">keycloak ×2 — 캐시가 곧 진실</text>
|
||||
<path class="ln" d="M470,116 L470,140"/><path class="ln" d="M610,140 L610,116"/>
|
||||
<rect class="ok" x="430" y="142" width="220" height="40"/>
|
||||
<text class="g" x="540" y="167" text-anchor="middle">클러스터 복제 (7800)</text>
|
||||
|
||||
<rect class="box" x="30" y="202" width="660" height="112"/>
|
||||
<text class="t" x="360" y="224" text-anchor="middle">뒤집힌 세 결과</text>
|
||||
<text class="s" x="200" y="248" text-anchor="middle">A-1 7800 차단 후 교차 refresh</text>
|
||||
<text class="g" x="430" y="248" text-anchor="middle">200</text><text class="r" x="560" y="248" text-anchor="middle">400 Session not active</text>
|
||||
<text class="s" x="200" y="272" text-anchor="middle">A-8 롤링 재시작 후 refresh</text>
|
||||
<text class="g" x="430" y="272" text-anchor="middle">200</text><text class="r" x="560" y="272" text-anchor="middle">400 Session not active</text>
|
||||
<text class="s" x="200" y="296" text-anchor="middle">A-2 DB 정지 중 새 로그인</text>
|
||||
<text class="r" x="430" y="296" text-anchor="middle">500</text><text class="g" x="560" y="296" text-anchor="middle">200</text>
|
||||
<text class="s" x="430" y="230" text-anchor="middle">persistent</text><text class="s" x="560" y="230" text-anchor="middle">volatile</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.8 KiB |
@@ -0,0 +1,70 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 470" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,sans-serif">
|
||||
<style>
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
|
||||
.t{font-size:12px;fill:#1f2328}
|
||||
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
|
||||
.s{font-size:10.5px;fill:#656d76}
|
||||
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
|
||||
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
|
||||
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
|
||||
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
|
||||
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
|
||||
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
|
||||
</defs>
|
||||
<text x="20" y="26" class="ttl">A-7a — volatile + DB 정지의 결과는 「무엇을 하느냐」가 아니라 「캐시가 더운가」로 갈린다</text>
|
||||
<text x="20" y="44" class="s">문장 로깅(log_statement=all)으로 측정 · 세 상태 모두 재현</text>
|
||||
|
||||
<rect x="20" y="66" width="380" height="120" class="box"/>
|
||||
<text x="34" y="88" class="tb">로그인이 쏘는 SQL</text>
|
||||
<rect x="36" y="100" width="348" height="34" class="ok"/>
|
||||
<text x="48" y="122" class="t">0개 — realm·사용자·클라이언트가 전부 캐시</text>
|
||||
<text x="36" y="152" class="s">MARK_LOGIN_START / MARK_LOGIN_END 사이가 비어 있다</text>
|
||||
<text x="36" y="170" class="s">volatile 이라 세션 쓰기도 없다</text>
|
||||
|
||||
<rect x="440" y="66" width="400" height="120" class="box"/>
|
||||
<text x="454" y="88" class="tb">refresh 가 쏘는 SQL</text>
|
||||
<rect x="456" y="100" width="368" height="34" class="warn"/>
|
||||
<text x="466" y="115" class="t">select cscme1_0.SCOPE_ID from CLIENT_SCOPE_CLIENT</text>
|
||||
<text x="466" y="129" class="s">where CLIENT_ID=$1 and DEFAULT_SCOPE=$2 ('f' = optional)</text>
|
||||
<text x="454" y="152" class="s">★ REVOKED_TOKEN 은 한 번도 나오지 않는다 — A-7 의 가설은 틀렸다</text>
|
||||
<text x="454" y="170" class="s">그리고 첫 refresh 만 쏜다. 이후 3회는 SQL 0건 = 캐시됨</text>
|
||||
|
||||
<text x="20" y="222" class="tb">그래서 같은 설정이 세 가지 답을 낸다 — PostgreSQL 정지 시</text>
|
||||
|
||||
<rect x="20" y="240" width="266" height="120" class="dead"/>
|
||||
<text x="34" y="262" class="tb">① 완전 냉시동</text>
|
||||
<text x="34" y="282" class="s">재시작 직후, 아무것도 안 함</text>
|
||||
<text x="34" y="306" class="t">로그인 <tspan fill="#cf222e" font-weight="600">400</tspan> refresh 400</text>
|
||||
<text x="34" y="328" class="s">select ce1_0.ID from CLIENT</text>
|
||||
<text x="34" y="344" class="s">클라이언트 조회조차 캐시에 없다</text>
|
||||
|
||||
<rect x="298" y="240" width="266" height="120" class="warn"/>
|
||||
<text x="312" y="262" class="tb">② CLIENT 만 더움</text>
|
||||
<text x="312" y="282" class="s">로그인 1회 뒤 · ← A-7 이 본 상태</text>
|
||||
<text x="312" y="306" class="t">로그인 200 refresh <tspan fill="#bf8700" font-weight="600">500</tspan></text>
|
||||
<text x="312" y="328" class="s">CLIENT_SCOPE_CLIENT 조회 실패</text>
|
||||
<text x="312" y="344" class="s">Keycloak 로그가 이 SQL 을 직접 지목</text>
|
||||
|
||||
<rect x="576" y="240" width="264" height="120" class="ok"/>
|
||||
<text x="590" y="262" class="tb">③ 완전히 더움</text>
|
||||
<text x="590" y="282" class="s">refresh 3회로 캐시를 채운 뒤</text>
|
||||
<text x="590" y="306" class="t">로그인 200 refresh <tspan fill="#1a7f37" font-weight="600">200</tspan></text>
|
||||
<text x="590" y="328" class="s">SQL 0건 — DB 를 아예 안 본다</text>
|
||||
<text x="590" y="344" class="s">A-7 의 표와 정반대</text>
|
||||
|
||||
<path d="M286 300 L294 300" class="k" marker-end="url(#a)"/>
|
||||
<path d="M564 300 L572 300" class="k" marker-end="url(#a)"/>
|
||||
<text x="292" y="290" class="s">로그인 1회</text>
|
||||
<text x="570" y="290" class="s">refresh 3회</text>
|
||||
|
||||
<rect x="20" y="384" width="820" height="66" class="box"/>
|
||||
<text x="34" y="406" class="tb">남기는 것</text>
|
||||
<text x="34" y="426" class="s">· A-7 의 표에는 조건이 빠져 있었다 — 「volatile 이면 DB 없이 로그인된다」도 냉시동에서는 거짓이다</text>
|
||||
<text x="34" y="442" class="s">· 상태가 결과를 바꾸는데 그 상태가 안 보인다. A-1 의 conntrack 과 같은 계열의 함정이다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.5 KiB |
@@ -0,0 +1,47 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 380" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
|
||||
<style>
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
|
||||
.t{font-size:12px;fill:#1f2328}
|
||||
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
|
||||
.s{font-size:10.5px;fill:#656d76}
|
||||
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
|
||||
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
|
||||
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
|
||||
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
|
||||
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
|
||||
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
|
||||
</defs>
|
||||
<text x="20" y="26" class="ttl">A-8 — 세션은 살아남고 캐시만 사라진다</text>
|
||||
<text x="20" y="44" class="s">이것이 persistent-user-sessions 를 켜는 진짜 이유다</text>
|
||||
<rect x="30" y="74" width="800" height="30" class="box"/>
|
||||
<text x="46" y="94" class="tb">kubectl rollout restart statefulset/keycloak — 한 번에 한 파드씩</text>
|
||||
<rect x="30" y="124" width="250" height="110" class="dead"/>
|
||||
<text x="46" y="146" class="tb">사라지는 것 — 메모리</text>
|
||||
<text x="46" y="168" class="s">Infinispan sessions 캐시</text>
|
||||
<text x="46" y="188" class="t">entries_unique → <tspan fill="#cf222e" font-weight="600">0</tspan></text>
|
||||
<text x="46" y="212" class="s">파드와 함께 없어진다</text>
|
||||
<rect x="305" y="124" width="250" height="110" class="ok"/>
|
||||
<text x="321" y="146" class="tb">남는 것 — PostgreSQL</text>
|
||||
<text x="321" y="168" class="s">OFFLINE_USER_SESSION</text>
|
||||
<text x="321" y="188" class="t">151 → <tspan fill="#1a7f37" font-weight="600">151</tspan> 그대로</text>
|
||||
<text x="321" y="212" class="s">재시작과 무관하다</text>
|
||||
<rect x="580" y="124" width="250" height="110" class="ok"/>
|
||||
<text x="596" y="146" class="tb">그래서 사용자는</text>
|
||||
<text x="596" y="168" class="s">재시작 전 발급한 refresh token</text>
|
||||
<text x="596" y="188" class="t">여전히 <tspan fill="#1a7f37" font-weight="600">200</tspan></text>
|
||||
<text x="596" y="212" class="s">전 구간 중단 없음</text>
|
||||
<path d="M282 178 L301 178" class="k" marker-end="url(#a)"/>
|
||||
<path d="M557 178 L576 178" class="k" marker-end="url(#a)"/>
|
||||
<rect x="30" y="254" width="800" height="52" class="box"/>
|
||||
<text x="46" y="276" class="tb">클러스터는 자동 재형성된다</text>
|
||||
<text x="46" y="296" class="s">JGROUPS_PING 테이블에 다시 등록되고 cluster_size 가 2 로 돌아온다 — 사람이 할 일이 없다</text>
|
||||
<rect x="30" y="322" width="800" height="44" class="warn"/>
|
||||
<text x="46" y="344" class="s">★ A-7 과 대조 — volatile 이면 같은 재시작에서 refresh 가 400 Session not active 가 된다.</text>
|
||||
<text x="46" y="360" class="s">세션이 메모리에만 있었으므로 캐시가 사라지는 순간 세션도 사라진다. 이 실험의 결론은 버전 조건부다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.3 KiB |
@@ -0,0 +1,45 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 400" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
|
||||
<style>
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
|
||||
.t{font-size:12px;fill:#1f2328}
|
||||
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
|
||||
.s{font-size:10.5px;fill:#656d76}
|
||||
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
|
||||
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
|
||||
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
|
||||
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
|
||||
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
|
||||
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
|
||||
</defs>
|
||||
<text x="20" y="26" class="ttl">B-0 — 아무것도 설정하지 않으면 Spring 이 무엇을 고르는가</text>
|
||||
<text x="20" y="44" class="s">추측이 맞았지만, 추측으로 두면 안 되는 이유가 둘째 줄에 있다</text>
|
||||
<rect x="30" y="72" width="800" height="118" class="box"/>
|
||||
<text x="46" y="94" class="tb">/actuator/beans 가 실제로 답한 것</text>
|
||||
<text x="46" y="118" class="t">authorizedClientService → InMemoryOAuth2AuthorizedClientService</text>
|
||||
<text x="46" y="138" class="t">authorizedClientRepository → AuthenticatedPrincipalOAuth2AuthorizedClientRepository</text>
|
||||
<text x="46" y="158" class="t">SessionRepository → 없음 (서블릿 컨테이너 in-memory)</text>
|
||||
<text x="46" y="178" class="t" fill="#cf222e">Redis / Spring Session → 없음</text>
|
||||
<rect x="30" y="212" width="390" height="150" class="warn"/>
|
||||
<text x="46" y="234" class="tb">★ 이름이 곧 설명이다</text>
|
||||
<text x="46" y="256" class="s">AuthenticatedPrincipal…Repository 는</text>
|
||||
<text x="46" y="276" class="t">principal 이름으로 찾는다</text>
|
||||
<rect x="46" y="288" width="358" height="30" class="dead"/>
|
||||
<text x="58" y="308" class="s">조회 키에 session id 가 없다</text>
|
||||
<text x="46" y="336" class="s">그래서 세션 저장소를 Redis 로 옮겨도</text>
|
||||
<text x="46" y="354" class="s">토큰은 따라오지 않는다 — B-1 이 겪는 문제의 뿌리</text>
|
||||
<rect x="440" y="212" width="390" height="150" class="box"/>
|
||||
<text x="456" y="234" class="tb">두 개가 서로 다른 것을 저장한다</text>
|
||||
<rect x="456" y="248" width="358" height="40" class="box"/>
|
||||
<text x="468" y="266" class="t">Application Session</text>
|
||||
<text x="468" y="282" class="s">누가 로그인했는지 · 세션 id 로 찾는다</text>
|
||||
<rect x="456" y="298" width="358" height="40" class="box"/>
|
||||
<text x="468" y="316" class="t">OAuth2AuthorizedClient</text>
|
||||
<text x="468" y="332" class="s">access / refresh token · principal 이름으로 찾는다</text>
|
||||
<text x="456" y="356" class="s">이 둘을 하나로 생각하면 B-1·B-2 의 결과를 해석할 수 없다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.2 KiB |
@@ -0,0 +1,33 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 330" width="700" height="330" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
|
||||
<style>
|
||||
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
|
||||
.r{font-size:12px;fill:#cf222e;font-weight:600}
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
|
||||
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
|
||||
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
|
||||
</defs>
|
||||
<text class="h" x="16" y="24">B-1 · Redis 는 세션만 옮기고 토큰은 두고 간다</text>
|
||||
<rect class="box" x="30" y="46" width="140" height="46"/><text class="t" x="100" y="66" text-anchor="middle">bff-0</text><text class="s" x="100" y="82" text-anchor="middle">kc-lab-1</text>
|
||||
<rect class="box" x="30" y="104" width="140" height="46"/><text class="t" x="100" y="124" text-anchor="middle">bff-1</text><text class="s" x="100" y="140" text-anchor="middle">kc-lab-2</text>
|
||||
<rect class="ok" x="330" y="46" width="330" height="60"/>
|
||||
<text class="t" x="495" y="68" text-anchor="middle">Redis — Application Session</text>
|
||||
<text class="s" x="495" y="86" text-anchor="middle">sessionRepository → RedisSessionRepository ✔ 옮겨졌다</text>
|
||||
<text class="s" x="495" y="100" text-anchor="middle">필드: SPRING_SECURITY_CONTEXT · TTL 1772초</text>
|
||||
<rect class="dead" x="330" y="122" width="330" height="60"/>
|
||||
<text class="r" x="495" y="144" text-anchor="middle">프로세스 메모리 — OAuth2AuthorizedClient</text>
|
||||
<text class="s" x="495" y="162" text-anchor="middle">InMemoryOAuth2AuthorizedClientService ✗ 그대로</text>
|
||||
<text class="s" x="495" y="176" text-anchor="middle">access token · refresh token 이 여기 있다</text>
|
||||
<path class="ln" d="M170,69 L330,69"/><path class="ln" d="M170,127 L330,80"/>
|
||||
<path class="bad" d="M170,140 L330,150"/>
|
||||
<rect class="box" x="30" y="210" width="630" height="90"/>
|
||||
<text class="t" x="345" y="234" text-anchor="middle">그 결과 사용자에게 보이는 것</text>
|
||||
<text class="s" x="345" y="256" text-anchor="middle">principal: labuser ← 로그인은 되어 있다</text>
|
||||
<text class="s" x="345" y="272" text-anchor="middle">accessTokenStoredOnServer: false ← 토큰이 없다</text>
|
||||
<text class="r" x="345" y="292" text-anchor="middle">완전히 로그아웃되는 편이 차라리 낫다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.8 KiB |
@@ -0,0 +1,55 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 440" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
|
||||
<style>
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
|
||||
.t{font-size:12px;fill:#1f2328}
|
||||
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
|
||||
.s{font-size:10.5px;fill:#656d76}
|
||||
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
|
||||
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
|
||||
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
|
||||
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
|
||||
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
|
||||
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
|
||||
</defs>
|
||||
<text x="20" y="26" class="ttl">B-2 — 저장소를 나눠 문제를 풀자 다른 두 문제가 남았다</text>
|
||||
<text x="20" y="44" class="s">Q1 의 네 항목 중 둘이 통과, 둘이 실패</text>
|
||||
<rect x="30" y="72" width="240" height="90" class="box"/>
|
||||
<text x="46" y="94" class="tb">bff 인스턴스 2개</text>
|
||||
<text x="46" y="116" class="s">어느 쪽으로 요청이 가도 된다</text>
|
||||
<text x="46" y="136" class="s">enableServiceLinks: false</text>
|
||||
<text x="46" y="154" class="s">(REDIS_PORT 충돌 회피)</text>
|
||||
<rect x="330" y="72" width="230" height="90" class="ok"/>
|
||||
<text x="346" y="94" class="tb">Redis</text>
|
||||
<text x="346" y="116" class="s">Application Session</text>
|
||||
<text x="346" y="136" class="s">세션 id 로 조회</text>
|
||||
<text x="346" y="154" class="s">로그아웃 시 정리됨</text>
|
||||
<rect x="600" y="72" width="230" height="90" class="warn"/>
|
||||
<text x="616" y="94" class="tb">PostgreSQL</text>
|
||||
<text x="616" y="116" class="s">OAuth2AuthorizedClient</text>
|
||||
<text x="616" y="136" class="s">principal 이름으로 조회</text>
|
||||
<text x="616" y="154" class="s">로그아웃해도 남는다</text>
|
||||
<path d="M272 108 L326 108" class="g" marker-end="url(#ag)"/>
|
||||
<path d="M562 108 L596 108" class="g" marker-end="url(#ag)"/>
|
||||
<rect x="30" y="188" width="390" height="100" class="ok"/>
|
||||
<text x="46" y="210" class="tb">통과</text>
|
||||
<text x="46" y="234" class="t">① 다른 인스턴스로 요청해도 된다</text>
|
||||
<text x="46" y="256" class="t">② 재시작 후에도 로그인 유지</text>
|
||||
<text x="46" y="278" class="s">저장소를 밖으로 뺐으니 당연한 결과다</text>
|
||||
<rect x="440" y="188" width="390" height="100" class="dead"/>
|
||||
<text x="456" y="210" class="tb">실패</text>
|
||||
<text x="456" y="234" class="t" fill="#cf222e">③ 같은 사용자의 다른 브라우저가 덮어쓴다</text>
|
||||
<text x="456" y="256" class="t" fill="#cf222e">④ 로그아웃해도 한쪽만 정리된다</text>
|
||||
<text x="456" y="278" class="s">둘 다 저장소 선택의 문제가 아니라 스키마의 문제다</text>
|
||||
<rect x="30" y="308" width="800" height="70" class="warn"/>
|
||||
<text x="46" y="330" class="tb">★ 뿌리는 DDL 한 줄이다</text>
|
||||
<text x="46" y="352" class="t">PRIMARY KEY (client_registration_id, principal_name)</text>
|
||||
<text x="46" y="370" class="s">세션 id 가 키에 없다 → 같은 사용자의 두 세션이 같은 행을 쓴다 → 나중 로그인이 앞의 토큰을 덮어쓴다</text>
|
||||
<rect x="30" y="392" width="800" height="40" class="dead"/>
|
||||
<text x="46" y="416" class="s">로그아웃 후: Redis 세션 0 키 (정리됨) · PostgreSQL 토큰 1 행 — 평문 refresh token 이 그대로 남는다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.7 KiB |
@@ -0,0 +1,29 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 300" width="700" height="300" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
|
||||
<style>
|
||||
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
|
||||
.r{font-size:12px;fill:#cf222e;font-weight:600}
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
|
||||
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
|
||||
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
|
||||
</defs>
|
||||
<text class="h" x="16" y="24">B-3 · 동시 refresh — 경쟁이 아니라 세션 파괴</text>
|
||||
<rect class="box" x="30" y="46" width="150" height="40"/><text class="t" x="105" y="71" text-anchor="middle">같은 refresh token ×5</text>
|
||||
<path class="ln" d="M180,66 L280,66"/>
|
||||
<rect class="box" x="280" y="42" width="180" height="48"/><text class="t" x="370" y="62" text-anchor="middle">Keycloak</text>
|
||||
<text class="s" x="370" y="80" text-anchor="middle">rotation ON · maxReuse=0</text>
|
||||
<path class="ln" d="M460,58 L560,58"/><rect class="box" x="560" y="42" width="110" height="24"/><text class="s" x="615" y="58" text-anchor="middle">1× HTTP 200</text>
|
||||
<path class="bad" d="M460,78 L560,78"/><rect class="dead" x="560" y="66" width="110" height="24"/><text class="s" x="615" y="82" text-anchor="middle">4× HTTP 400</text>
|
||||
<rect class="dead" x="120" y="120" width="460" height="70"/>
|
||||
<text class="r" x="350" y="144" text-anchor="middle">재사용 탐지가 client session 을 제거한다</text>
|
||||
<text class="s" x="350" y="164" text-anchor="middle">user_session 은 남고 client_session = 0 (정상 세션은 1)</text>
|
||||
<text class="s" x="350" y="180" text-anchor="middle">그래서 오류가 "Session doesn't have required client"</text>
|
||||
<rect class="dead" x="120" y="210" width="460" height="60"/>
|
||||
<text class="r" x="350" y="234" text-anchor="middle">★ 이긴 요청의 새 토큰도 곧바로 400</text>
|
||||
<text class="s" x="350" y="254" text-anchor="middle">재시도로 회복 불가 → Q2 의 판정은 lock</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.5 KiB |
@@ -0,0 +1,37 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 290" width="700" height="290" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
|
||||
<style>
|
||||
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
|
||||
.r{font-size:12px;fill:#cf222e;font-weight:600}.g{font-size:12px;fill:#1a7f37;font-weight:600}
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
|
||||
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
|
||||
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
|
||||
</defs>
|
||||
<text class="h" x="16" y="24">B-4 · edge 가 설정하지 않은 헤더는 그대로 통과한다</text>
|
||||
<rect class="box" x="30" y="46" width="130" height="60"/><text class="t" x="95" y="68" text-anchor="middle">공격자</text>
|
||||
<text class="s" x="95" y="86" text-anchor="middle">X-Auth-Request-</text><text class="s" x="95" y="100" text-anchor="middle">Roles: admin</text>
|
||||
<path class="ln" d="M160,76 L250,76"/>
|
||||
<rect class="box" x="250" y="40" width="180" height="72"/><text class="t" x="340" y="62" text-anchor="middle">nginx</text>
|
||||
<text class="s" x="340" y="82" text-anchor="middle">proxy_set_header 한 것만 덮어쓴다</text>
|
||||
<text class="r" x="340" y="100" text-anchor="middle">X-Auth-Request-* 는 설정이 없다</text>
|
||||
<path class="bad" d="M430,76 L520,76"/>
|
||||
<rect class="dead" x="520" y="46" width="150" height="60"/>
|
||||
<text class="r" x="595" y="68" text-anchor="middle">upstream</text>
|
||||
<text class="s" x="595" y="86" text-anchor="middle">['viewer','admin']</text><text class="s" x="595" y="100" text-anchor="middle">둘 다 도착 · 검증 없음</text>
|
||||
<rect class="box" x="30" y="134" width="310" height="66"/>
|
||||
<text class="t" x="185" y="156" text-anchor="middle">구분자 문제</text>
|
||||
<text class="s" x="185" y="176" text-anchor="middle">"admin,editor" 와 "role-with,comma" 가</text>
|
||||
<text class="s" x="185" y="192" text-anchor="middle">도착 시점에 구별되지 않는다</text>
|
||||
<rect class="box" x="360" y="134" width="310" height="66"/>
|
||||
<text class="t" x="515" y="156" text-anchor="middle">크기는 절벽이다</text>
|
||||
<text class="s" x="515" y="176" text-anchor="middle">4KB 통과 · 8KB → Tomcat 400</text>
|
||||
<text class="s" x="515" y="192" text-anchor="middle">16KB → 연결 끊김 (nginx)</text>
|
||||
<rect class="dead" x="30" y="222" width="640" height="50"/>
|
||||
<text class="r" x="350" y="244" text-anchor="middle">헤더가 인가 근거가 되면 위조 가능성이 곧 권한 상승이다</text>
|
||||
<text class="s" x="350" y="262" text-anchor="middle">Q4 의 5문항 중 2·4번 해당 → Q4 자신의 기준으로 BFF 구조</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.1 KiB |
@@ -0,0 +1,35 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 300" width="700" height="300" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
|
||||
<style>
|
||||
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
|
||||
.r{font-size:12px;fill:#cf222e;font-weight:600}.g{font-size:12px;fill:#1a7f37;font-weight:600}
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
|
||||
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
|
||||
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
|
||||
</defs>
|
||||
<text class="h" x="16" y="24">B-5 · 파드가 Ready 인 채로 계속 실패한다</text>
|
||||
<rect class="box" x="30" y="46" width="180" height="76"/>
|
||||
<text class="t" x="120" y="68" text-anchor="middle">bff ×2</text>
|
||||
<text class="g" x="120" y="90" text-anchor="middle">Ready = true</text>
|
||||
<text class="s" x="120" y="110" text-anchor="middle">Service 에 그대로 남는다</text>
|
||||
<path class="bad" d="M210,84 L330,84"/>
|
||||
<rect class="dead" x="330" y="58" width="160" height="52"/>
|
||||
<text class="r" x="410" y="80" text-anchor="middle">redis ✗</text><text class="s" x="410" y="98" text-anchor="middle">replicas=0</text>
|
||||
<rect class="box" x="30" y="146" width="310" height="110"/>
|
||||
<text class="t" x="185" y="168" text-anchor="middle">health group 이 갈랐다</text>
|
||||
<text class="r" x="185" y="192" text-anchor="middle">/actuator/health → 503</text>
|
||||
<text class="g" x="185" y="214" text-anchor="middle">/actuator/health/readiness → 200 UP</text>
|
||||
<text class="s" x="185" y="238" text-anchor="middle">redis 지표가 readiness 그룹에 없다</text>
|
||||
<rect class="box" x="360" y="146" width="310" height="110"/>
|
||||
<text class="t" x="515" y="168" text-anchor="middle">A-2 와 정반대</text>
|
||||
<text class="s" x="515" y="192" text-anchor="middle">A-2 Keycloak: DB 검사가 readiness 에</text>
|
||||
<text class="s" x="515" y="208" text-anchor="middle">→ NotReady → 503 (명확)</text>
|
||||
<text class="s" x="515" y="230" text-anchor="middle">B-5 BFF: 없음 → Ready 유지</text>
|
||||
<text class="r" x="515" y="248" text-anchor="middle">→ HTTP 000 (멈춤)</text>
|
||||
<text class="s" x="16" y="284">영속화: 볼륨 없이 AOF 만 켜면 appendonlydir 은 생기지만 파드 삭제로 전부 사라진다 — 볼륨이 먼저다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.8 KiB |
@@ -0,0 +1,34 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 260" width="700" height="260" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
|
||||
<style>
|
||||
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
|
||||
.r{font-size:12px;fill:#cf222e;font-weight:600}.g{font-size:12px;fill:#1a7f37;font-weight:600}
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
|
||||
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
|
||||
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
|
||||
</defs>
|
||||
<text class="h" x="16" y="24">B-6 · 회전은 안전하고 옛 키를 버리는 순간이 위험하다</text>
|
||||
<rect class="ok" x="30" y="46" width="190" height="76"/>
|
||||
<text class="t" x="125" y="68" text-anchor="middle">t0 — 키 A 만</text>
|
||||
<text class="s" x="125" y="88" text-anchor="middle">발급 A · 검증 A</text>
|
||||
<text class="s" x="125" y="108" text-anchor="middle">JWKS RS256 1개</text>
|
||||
<path class="ln" d="M220,84 L255,84"/>
|
||||
<rect class="ok" x="255" y="46" width="190" height="76"/>
|
||||
<text class="t" x="350" y="68" text-anchor="middle">t1 — B 추가 (priority 200)</text>
|
||||
<text class="g" x="350" y="88" text-anchor="middle">발급 B · 검증 A+B</text>
|
||||
<text class="s" x="350" y="108" text-anchor="middle">옛 토큰 200 · 새 토큰 200</text>
|
||||
<path class="bad" d="M445,84 L480,84"/>
|
||||
<rect class="dead" x="480" y="46" width="190" height="76"/>
|
||||
<text class="t" x="575" y="68" text-anchor="middle">t2 — A 제거</text>
|
||||
<text class="r" x="575" y="88" text-anchor="middle">옛 토큰 즉시 401</text>
|
||||
<text class="s" x="575" y="108" text-anchor="middle">캐시가 유예를 주지 않는다</text>
|
||||
<rect class="box" x="30" y="146" width="640" height="60"/>
|
||||
<text class="t" x="350" y="168" text-anchor="middle">겹침 구간(t1~t2)의 최소 길이 = 옛 키로 서명된 것 중 가장 오래 사는 것의 수명</text>
|
||||
<text class="s" x="350" y="190" text-anchor="middle">access token 60초 · refresh token 1800초 → 최소 30분</text>
|
||||
<text class="s" x="16" y="234">모르는 kid 를 만나면 JWKS 를 다시 받으므로 제거가 즉시 반영된다. 유예는 옛 키를 남겨두는 기간으로 만든다.</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.7 KiB |
@@ -0,0 +1,52 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 420" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
|
||||
<style>
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
|
||||
.t{font-size:12px;fill:#1f2328}
|
||||
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
|
||||
.s{font-size:10.5px;fill:#656d76}
|
||||
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
|
||||
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
|
||||
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
|
||||
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
|
||||
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
|
||||
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
|
||||
</defs>
|
||||
<text x="20" y="26" class="ttl">B-7 — BFF 와 정반대의 성질: 공유할 상태가 없고, 대신 겹침 구간도 없다</text>
|
||||
<text x="20" y="44" class="s">oauth2-proxy 는 세션을 쿠키에 담는다</text>
|
||||
<rect x="30" y="72" width="380" height="120" class="box"/>
|
||||
<text x="46" y="94" class="tb">replica 는 무엇을 공유하나</text>
|
||||
<rect x="46" y="106" width="160" height="34" class="box"/>
|
||||
<text x="58" y="128" class="t">oauth2-proxy A</text>
|
||||
<rect x="250" y="106" width="146" height="34" class="box"/>
|
||||
<text x="262" y="128" class="t">oauth2-proxy B</text>
|
||||
<rect x="106" y="152" width="230" height="30" class="ok"/>
|
||||
<text x="118" y="172" class="s">같은 k8s Secret 을 읽는다</text>
|
||||
<path d="M126 142 L180 150" class="g"/>
|
||||
<path d="M316 142 L262 150" class="g"/>
|
||||
<text x="46" y="196" class="s"> </text>
|
||||
<rect x="440" y="72" width="390" height="120" class="ok"/>
|
||||
<text x="456" y="94" class="tb">그래서 얻는 것</text>
|
||||
<text x="456" y="118" class="s">공유할 서버 상태가 없다 → 콜백이 다른 replica 로 가도 된다</text>
|
||||
<text x="456" y="140" class="s">replica 를 늘려도 세션 저장소 고민이 없다</text>
|
||||
<text x="456" y="164" class="s">BFF 가 Redis·PostgreSQL 로 풀던 문제가 아예 생기지 않는다</text>
|
||||
<rect x="30" y="212" width="380" height="106" class="dead"/>
|
||||
<text x="46" y="234" class="tb">★ 대신 겹침 구간을 만들 수 없다</text>
|
||||
<text x="46" y="258" class="t">--cookie-secret 은 단수다</text>
|
||||
<text x="46" y="280" class="s">「옛 secret 도 당분간 받아준다」가 불가능하다</text>
|
||||
<text x="46" y="300" class="s">교체하는 순간 모든 쿠키가 한꺼번에 무효가 된다</text>
|
||||
<rect x="440" y="212" width="390" height="106" class="warn"/>
|
||||
<text x="456" y="234" class="tb">교체하면 벌어지는 일</text>
|
||||
<text x="456" y="258" class="s">옛 쿠키 → session ticket cookie failed validation</text>
|
||||
<text x="456" y="278" class="s">Keycloak SSO 가 살아 있으면 로그인 화면 없이 조용히 재인증</text>
|
||||
<text x="456" y="300" class="s" fill="#cf222e">서버 쪽 세션은 고아로 남는다 — 티켓을 못 풀어 지우지도 못한다</text>
|
||||
<rect x="30" y="338" width="800" height="66" class="ok"/>
|
||||
<text x="46" y="360" class="tb">→ B-7a 가 이어받았다</text>
|
||||
<text x="46" y="382" class="s">「지울 수 없다」는 oauth2-proxy 의 한계일 뿐이었다. 운영자는 지울 수 있고, TTL 역산으로 고아만 골라낼 수 있다</text>
|
||||
<text x="46" y="398" class="s">그리고 고아는 생성 후 정확히 1시간에 사라진다 — TTL 이 요청으로 갱신되지 않기 때문이다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.7 KiB |
@@ -0,0 +1,74 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 500" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,sans-serif">
|
||||
<style>
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
|
||||
.t{font-size:12px;fill:#1f2328}
|
||||
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
|
||||
.s{font-size:10.5px;fill:#656d76}
|
||||
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
|
||||
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
|
||||
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
|
||||
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
|
||||
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
|
||||
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
|
||||
</defs>
|
||||
<text x="20" y="26" class="ttl">B-7a — 고아 세션은 지울 수 있다. oauth2-proxy 가 못 할 뿐이다</text>
|
||||
<text x="20" y="44" class="s">회전 2회 측정 · Redis + oauth2-proxy 로그 + 브라우저</text>
|
||||
|
||||
<text x="20" y="76" class="tb">왜 프록시는 못 지우는가 — 티켓의 구조</text>
|
||||
<rect x="20" y="88" width="400" height="86" class="box"/>
|
||||
<text x="34" y="110" class="t">쿠키에 담기는 것 = 티켓</text>
|
||||
<rect x="36" y="120" width="180" height="30" class="box"/>
|
||||
<text x="48" y="140" class="s">세션 ID → Redis 키 이름</text>
|
||||
<rect x="224" y="120" width="180" height="30" class="box"/>
|
||||
<text x="236" y="140" class="s">암호화 키 → 값 복호화</text>
|
||||
<text x="34" y="166" class="s">티켓 전체가 cookie secret 으로 암호화되어 있다</text>
|
||||
|
||||
<rect x="444" y="88" width="396" height="86" class="dead"/>
|
||||
<text x="458" y="110" class="tb">secret 을 바꾸면</text>
|
||||
<text x="458" y="130" class="s">티켓을 못 연다 → 세션 ID 조차 못 읽는다</text>
|
||||
<text x="458" y="148" class="s">→ 어느 Redis 키를 지울지 모른다</text>
|
||||
<text x="458" y="166" class="s">Error removing session: error decoding ticket to clear session</text>
|
||||
<path d="M424 131 L440 131" class="r" marker-end="url(#ar)"/>
|
||||
|
||||
<text x="20" y="208" class="tb">회전할 때마다 누적한다</text>
|
||||
<line x1="70" y1="256" x2="800" y2="256" class="k"/>
|
||||
<line x1="130" y1="244" x2="130" y2="268" stroke="#cf222e" stroke-width="2"/>
|
||||
<line x1="430" y1="244" x2="430" y2="268" stroke="#cf222e" stroke-width="2"/>
|
||||
<text x="92" y="288" class="s">1차 회전 11:29:56</text>
|
||||
<text x="392" y="288" class="s">2차 회전 11:33:27</text>
|
||||
|
||||
<rect x="70" y="224" width="60" height="16" class="ok"/>
|
||||
<text x="76" y="236" class="s">세션 A</text>
|
||||
<rect x="130" y="224" width="300" height="16" class="dead"/>
|
||||
<text x="150" y="236" class="s" fill="#cf222e">A 는 고아 · TTL 만료까지 남는다</text>
|
||||
<rect x="130" y="224" width="0" height="0"/>
|
||||
<rect x="150" y="304" width="280" height="16" class="ok"/>
|
||||
<text x="156" y="316" class="s">세션 B (1차 회전 후 생성)</text>
|
||||
<rect x="430" y="304" width="370" height="16" class="dead"/>
|
||||
<text x="450" y="316" class="s" fill="#cf222e">B 도 고아가 된다</text>
|
||||
<rect x="450" y="224" width="350" height="16" class="ok"/>
|
||||
<text x="456" y="236" class="s">세션 C (2차 회전 후 생성)</text>
|
||||
<text x="70" y="338" class="s">회전 1회 = 그 시점 로그인 사용자 수만큼의 고아</text>
|
||||
|
||||
<rect x="20" y="356" width="400" height="126" class="dead"/>
|
||||
<text x="34" y="378" class="tb">Redis 만 보고는 구분할 수 없다</text>
|
||||
<text x="34" y="400" class="s">이름 _oauth2_proxy-<불투명한 32자 hex> (동일 형식)</text>
|
||||
<text x="34" y="418" class="s">type string (동일)</text>
|
||||
<text x="34" y="436" class="s">크기 3510 바이트 (바이트까지 동일)</text>
|
||||
<text x="34" y="454" class="s">값 암호화됨 (읽을 수 없다)</text>
|
||||
<text x="34" y="474" class="s" fill="#cf222e">→ 다른 것은 TTL 하나뿐이다</text>
|
||||
|
||||
<rect x="444" y="356" width="396" height="126" class="ok"/>
|
||||
<text x="458" y="378" class="tb">그래서 TTL 이 정리 규칙이 된다</text>
|
||||
<text x="458" y="398" class="s">--cookie-refresh 가 없어 TTL 이 갱신되지 않는다 (refresh:disabled)</text>
|
||||
<text x="458" y="416" class="s">→ TTL 은 생성 시각의 정확한 함수다</text>
|
||||
<rect x="460" y="424" width="364" height="26" class="box"/>
|
||||
<text x="470" y="442" class="t">생성시각 = 지금 − (cookie-expire − TTL)</text>
|
||||
<text x="458" y="466" class="s" fill="#1a7f37">이 값이 회전 시각보다 이르면 고아. 검증: 역산 11:30:26 vs 로그 11:30:27 — 1초 오차</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.8 KiB |
@@ -0,0 +1,31 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 300" width="700" height="300" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
|
||||
<style>
|
||||
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
|
||||
.r{font-size:12px;fill:#cf222e;font-weight:600}.g{font-size:12px;fill:#1a7f37;font-weight:600}
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
|
||||
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
|
||||
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
|
||||
</defs>
|
||||
<text class="h" x="16" y="24">C-1 · SSO 의 구조와 IdP 로그아웃의 한계</text>
|
||||
<rect class="box" x="220" y="42" width="260" height="52"/>
|
||||
<text class="t" x="350" y="64" text-anchor="middle">Keycloak user session ×1</text>
|
||||
<text class="s" x="350" y="82" text-anchor="middle">oqOjHekin4JU-BZjgQLjUByW</text>
|
||||
<path class="ln" d="M300,96 L180,130"/><path class="ln" d="M400,96 L520,130"/>
|
||||
<rect class="box" x="60" y="134" width="240" height="52"/>
|
||||
<text class="t" x="180" y="156" text-anchor="middle">client session — bff-confidential</text>
|
||||
<text class="s" x="180" y="174" text-anchor="middle">app1 · Redis 세션 + PostgreSQL 토큰</text>
|
||||
<rect class="box" x="400" y="134" width="240" height="52"/>
|
||||
<text class="t" x="520" y="156" text-anchor="middle">client session — oauth2-proxy</text>
|
||||
<text class="s" x="520" y="174" text-anchor="middle">app2 · 쿠키 티켓 + Redis 세션</text>
|
||||
<rect class="dead" x="220" y="206" width="260" height="40"/>
|
||||
<text class="r" x="350" y="231" text-anchor="middle">IdP 세션 삭제 ✗</text>
|
||||
<path class="bad" d="M300,246 L200,262"/><path class="bad" d="M400,246 L500,262"/>
|
||||
<rect class="ok" x="60" y="256" width="240" height="34"/><text class="g" x="180" y="278" text-anchor="middle">app1 그대로 동작</text>
|
||||
<rect class="ok" x="400" y="256" width="240" height="34"/><text class="g" x="520" y="278" text-anchor="middle">app2 그대로 동작</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.4 KiB |
@@ -0,0 +1,44 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 400" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
|
||||
<style>
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
|
||||
.t{font-size:12px;fill:#1f2328}
|
||||
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
|
||||
.s{font-size:10.5px;fill:#656d76}
|
||||
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
|
||||
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
|
||||
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
|
||||
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
|
||||
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
|
||||
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
|
||||
</defs>
|
||||
<text x="20" y="26" class="ttl">C-2 — 로그아웃이 안 퍼지는 이유는 단순했다. 아무도 구현하지 않았다</text>
|
||||
<text x="20" y="44" class="s">C-1 이 관측한 현상의 원인</text>
|
||||
<text x="20" y="76" class="tb">백채널 로그아웃이 동작하려면 양쪽이 다 있어야 한다</text>
|
||||
<rect x="30" y="94" width="380" height="110" class="dead"/>
|
||||
<text x="46" y="116" class="tb">① IdP 쪽 — Keycloak 클라이언트 설정</text>
|
||||
<text x="46" y="140" class="t">backchannelLogoutUrl</text>
|
||||
<text x="46" y="162" class="t" fill="#cf222e">두 클라이언트 모두 비어 있다</text>
|
||||
<text x="46" y="186" class="s">Keycloak 은 부를 주소를 모른다</text>
|
||||
<rect x="440" y="94" width="390" height="110" class="dead"/>
|
||||
<text x="456" y="116" class="tb">② 앱 쪽 — 수신 엔드포인트</text>
|
||||
<text x="456" y="140" class="t">/logout/connect/back-channel/{registrationId}</text>
|
||||
<text x="456" y="162" class="t" fill="#cf222e">BFF 소스에 oidcLogout 설정이 없다</text>
|
||||
<text x="456" y="186" class="s">주소를 알려줘도 받을 곳이 없다</text>
|
||||
<rect x="30" y="228" width="800" height="60" class="warn"/>
|
||||
<text x="46" y="250" class="tb">★ 그래서 IdP 쪽만 설정하면 되는 줄 알면 틀린다</text>
|
||||
<text x="46" y="272" class="s">backchannelLogoutUrl 을 넣어 봤지만 앱 세션은 그대로 남았다. 받는 쪽이 없기 때문이다</text>
|
||||
<rect x="30" y="308" width="380" height="72" class="ok"/>
|
||||
<text x="46" y="330" class="tb">네트워크 문제가 아니라는 확인</text>
|
||||
<text x="46" y="352" class="s">Keycloak 파드에서 앱 URL 로 요청 → HTTP 200</text>
|
||||
<text x="46" y="372" class="s">닿기는 한다. 닿아도 처리할 코드가 없을 뿐이다</text>
|
||||
<rect x="440" y="308" width="390" height="72" class="box"/>
|
||||
<text x="456" y="330" class="tb">남기는 것</text>
|
||||
<text x="456" y="352" class="s">「설정이 빠졌다」와 「기능이 없다」는 다르게 고쳐야 한다</text>
|
||||
<text x="456" y="372" class="s">여기는 둘 다였고, 확인 순서를 바꿨다면 한쪽만 고치고 끝냈을 것이다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.2 KiB |
@@ -0,0 +1,30 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 270" width="700" height="270" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
|
||||
<style>
|
||||
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
|
||||
.r{font-size:12px;fill:#cf222e;font-weight:600}.g{font-size:12px;fill:#1a7f37;font-weight:600}
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
|
||||
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
|
||||
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
|
||||
</defs>
|
||||
<text class="h" x="16" y="24">D-1 · 빈 데이터베이스가 200 을 냈다</text>
|
||||
<rect class="box" x="30" y="46" width="180" height="46"/><text class="t" x="120" y="66" text-anchor="middle">pg_dump</text><text class="s" x="120" y="82" text-anchor="middle">395KB · 101 테이블 · 세션 포함</text>
|
||||
<path class="ln" d="M210,69 L270,69"/>
|
||||
<rect class="dead" x="270" y="42" width="180" height="54"/>
|
||||
<text class="r" x="360" y="64" text-anchor="middle">DROP SCHEMA CASCADE</text><text class="s" x="360" y="84" text-anchor="middle">남은 테이블 0</text>
|
||||
<path class="ln" d="M450,69 L510,69"/>
|
||||
<rect class="ok" x="510" y="42" width="160" height="54"/>
|
||||
<text class="g" x="590" y="64" text-anchor="middle">복구 1초</text><text class="s" x="590" y="84" text-anchor="middle">오류 0건 · 재시작 0회</text>
|
||||
<rect class="box" x="30" y="118" width="640" height="76"/>
|
||||
<text class="t" x="350" y="140" text-anchor="middle">테이블이 0개일 때 무엇이 깨졌는가 — 전부가 아니다</text>
|
||||
<text class="g" x="350" y="162" text-anchor="middle">/protocol/openid-connect/certs → 200 (realm 키가 캐시에 있다)</text>
|
||||
<text class="r" x="350" y="182" text-anchor="middle">/.well-known → 500 토큰 발급 → 400</text>
|
||||
<rect class="dead" x="30" y="210" width="640" height="46"/>
|
||||
<text class="r" x="350" y="232" text-anchor="middle">헬스체크는 "DB 가 살아 있다"만 보고 "데이터가 있다"는 안 본다</text>
|
||||
<text class="s" x="350" y="250" text-anchor="middle">RPO = 백업 주기 + A-3 의 synchronous_commit 손실 · 덤프는 같은 호스트 /tmp 에 있었다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.6 KiB |
@@ -0,0 +1,48 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 420" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,sans-serif">
|
||||
<style>
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
|
||||
.t{font-size:12px;fill:#1f2328}
|
||||
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
|
||||
.s{font-size:10.5px;fill:#656d76}
|
||||
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
|
||||
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
|
||||
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
|
||||
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
|
||||
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
|
||||
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
|
||||
</defs>
|
||||
<text x="20" y="26" class="ttl">D-2 — 롤백은 안 된다. 다만 롤링 업데이트가 사고를 절반에서 멈춰줬다</text>
|
||||
<text x="20" y="44" class="s">그리고 이 결론은 조건부였다 — 스키마가 안 바뀌면 롤백된다(후속에서 정밀화)</text>
|
||||
<rect x="30" y="72" width="380" height="126" class="box"/>
|
||||
<text x="46" y="94" class="tb">앞으로 (26.7.0 → 26.7.3)</text>
|
||||
<text x="46" y="118" class="t" fill="#1a7f37">된다. 무중단</text>
|
||||
<text x="46" y="140" class="s">87회 요청 전부 200</text>
|
||||
<text x="46" y="160" class="s">databasechangelog 210 → 210 (스키마 변경 없음)</text>
|
||||
<text x="46" y="180" class="s">세션 유지 · Infinispan 16.0.12 → 16.0.14</text>
|
||||
<rect x="440" y="72" width="390" height="126" class="dead"/>
|
||||
<text x="456" y="94" class="tb">뒤로 (스키마가 이미 움직였을 때)</text>
|
||||
<text x="456" y="118" class="t" fill="#cf222e">안 된다</text>
|
||||
<text x="456" y="140" class="s">liquibase ValidationFailedException:</text>
|
||||
<text x="456" y="158" class="s">1 changesets check sum</text>
|
||||
<text x="456" y="180" class="s">새 버전이 남긴 체크섬을 옛 버전이 거부한다</text>
|
||||
<rect x="30" y="222" width="800" height="72" class="ok"/>
|
||||
<text x="46" y="244" class="tb">★ 그런데 서비스는 살아 있었다 — StatefulSet 롤링 업데이트 덕분이다</text>
|
||||
<text x="46" y="266" class="s">한 번에 한 파드씩 바꾼다. 첫 파드가 기동에 실패하면 거기서 멈추고 나머지는 건드리지 않는다</text>
|
||||
<text x="46" y="286" class="s">그래서 남은 파드가 외부 200 을 계속 냈다. 「롤백 계획」이 없어도 사고가 전면화되지 않았다</text>
|
||||
<rect x="30" y="314" width="380" height="90" class="warn"/>
|
||||
<text x="46" y="336" class="tb">그러면 무엇으로 판단하나</text>
|
||||
<text x="46" y="358" class="t">select count(*) from databasechangelog</text>
|
||||
<text x="46" y="380" class="s">업그레이드 전후 이 수가 같으면 롤백 가능</text>
|
||||
<text x="46" y="398" class="s">늘었으면 스키마가 움직였다 = 롤백 불가</text>
|
||||
<rect x="440" y="314" width="390" height="90" class="box"/>
|
||||
<text x="456" y="336" class="tb">전제 — 백업이 먼저다 (D-1)</text>
|
||||
<text x="456" y="358" class="s">롤백이 막히는 상황에서 되돌릴 방법은 덤프 복원뿐이다</text>
|
||||
<text x="456" y="380" class="s">D-1 이 그 절차를 재고, D-2 가 그 절차가 필요한 이유를 만든다</text>
|
||||
<text x="456" y="398" class="s">순서를 바꾸면 되돌릴 수 없는 상태에서 백업을 배우게 된다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.6 KiB |
@@ -0,0 +1,31 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 280" width="700" height="280" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
|
||||
<style>
|
||||
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
|
||||
.r{font-size:12px;fill:#cf222e;font-weight:600}.g{font-size:12px;fill:#1a7f37;font-weight:600}
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
|
||||
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
|
||||
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
|
||||
</defs>
|
||||
<text class="h" x="16" y="24">D-3 · 네 경로 중 RBAC 만 실제로 감춘다</text>
|
||||
<rect class="dead" x="30" y="46" width="310" height="60"/>
|
||||
<text class="r" x="185" y="68" text-anchor="middle">kubectl get -o jsonpath | base64 -d</text>
|
||||
<text class="s" x="185" y="88" text-anchor="middle">POSTGRES_PASSWORD = lab-postgres-change-me</text>
|
||||
<rect class="dead" x="360" y="46" width="310" height="60"/>
|
||||
<text class="r" x="515" y="68" text-anchor="middle">저장소 (at rest)</text>
|
||||
<text class="s" x="515" y="88" text-anchor="middle">Encryption Disabled · state.db 에 평문</text>
|
||||
<rect class="dead" x="30" y="120" width="310" height="60"/>
|
||||
<text class="r" x="185" y="142" text-anchor="middle">파드 안</text>
|
||||
<text class="s" x="185" y="162" text-anchor="middle">KEYCLOAK_CLIENT_SECRET=... 환경변수</text>
|
||||
<rect class="ok" x="360" y="120" width="310" height="60"/>
|
||||
<text class="g" x="515" y="142" text-anchor="middle">RBAC</text>
|
||||
<text class="s" x="515" y="162" text-anchor="middle">default SA 는 get secrets 불가</text>
|
||||
<rect class="box" x="30" y="200" width="640" height="60"/>
|
||||
<text class="t" x="350" y="222" text-anchor="middle">describe 는 "14 bytes" 만 보여줘 감춰졌다는 착각을 준다</text>
|
||||
<text class="s" x="350" y="244" text-anchor="middle">base64 는 감추기 위한 것이 아니라 YAML 에 임의 바이트를 담기 위한 인코딩이다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.4 KiB |
@@ -0,0 +1,30 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 700 250" width="700" height="250" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
|
||||
<style>
|
||||
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
|
||||
.r{font-size:12px;fill:#cf222e;font-weight:600}.g{font-size:12px;fill:#1a7f37;font-weight:600}
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
|
||||
.bad{stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;fill:none;marker-end:url(#b)}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker>
|
||||
<marker id="b" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#cf222e"/></marker>
|
||||
</defs>
|
||||
<text class="h" x="16" y="24">D-4 · 인증서 체인과 SAN 제약</text>
|
||||
<rect class="ok" x="30" y="46" width="300" height="104"/>
|
||||
<text class="t" x="180" y="68" text-anchor="middle">체인 4단계 · Verify return code: 0</text>
|
||||
<text class="s" x="180" y="90" text-anchor="middle">0 CN=auth.hyeonworks.com</text>
|
||||
<text class="s" x="180" y="106" text-anchor="middle">1 Let's Encrypt YE2</text>
|
||||
<text class="s" x="180" y="122" text-anchor="middle">2 ISRG Root YE</text>
|
||||
<text class="s" x="180" y="138" text-anchor="middle">3 ISRG Root X2</text>
|
||||
<rect class="box" x="360" y="46" width="310" height="104"/>
|
||||
<text class="t" x="515" y="68" text-anchor="middle">SAN 3개 · 와일드카드 아님</text>
|
||||
<text class="s" x="515" y="90" text-anchor="middle">auth · app1 · app2</text>
|
||||
<text class="r" x="515" y="114" text-anchor="middle">네 번째 이름이 없다</text>
|
||||
<text class="s" x="515" y="134" text-anchor="middle">B-7 에서 Grafana 의 app2 를 빌려야 했다</text>
|
||||
<rect class="box" x="30" y="168" width="640" height="60"/>
|
||||
<text class="t" x="350" y="190" text-anchor="middle">단계가 1개면 cert.pem, 2개 이상이면 fullchain.pem 이다</text>
|
||||
<text class="s" x="350" y="212" text-anchor="middle">브라우저는 중간 인증서를 캐시하므로 cert.pem 실수는 캐시 없는 클라이언트에서만 드러난다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.4 KiB |
@@ -0,0 +1,94 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 520" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,sans-serif">
|
||||
<style>
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.t{font-size:12px;fill:#1f2328}
|
||||
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
|
||||
.s{font-size:10.5px;fill:#656d76}
|
||||
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
|
||||
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
|
||||
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
|
||||
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
|
||||
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
|
||||
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
|
||||
</defs>
|
||||
|
||||
<text x="20" y="26" class="ttl">D-4 — 「갱신 성공」과 「새 인증서 서빙」 사이의 38분 25초</text>
|
||||
<text x="20" y="44" class="s">측정: 2026-09-04 · 일련번호 5초 간격 564표본 · 새 연결 0.2초 간격 8856건</text>
|
||||
|
||||
<!-- 좌: 디스크 -->
|
||||
<rect x="20" y="70" width="250" height="196" class="box"/>
|
||||
<text x="34" y="92" class="tb">디스크 — certbot</text>
|
||||
<rect x="36" y="104" width="218" height="46" class="box"/>
|
||||
<text x="48" y="122" class="t">cert1.pem</text>
|
||||
<text x="48" y="138" class="s">09-03 10:47:40 · notAfter Dec 2</text>
|
||||
<rect x="36" y="158" width="218" height="46" class="ok"/>
|
||||
<text x="48" y="176" class="t">cert2.pem</text>
|
||||
<text x="48" y="192" class="s">09-04 17:22:13 · notAfter Dec 3</text>
|
||||
<text x="36" y="224" class="s">certbot renew --force-renewal</text>
|
||||
<text x="36" y="240" class="s">→ 성공. live/ 링크가 2번을 가리킨다</text>
|
||||
<text x="36" y="256" class="s">경로는 그대로, 내용만 바뀐다</text>
|
||||
|
||||
<!-- 중: 끊긴 고리 -->
|
||||
<rect x="300" y="96" width="260" height="144" class="dead"/>
|
||||
<text x="316" y="118" class="tb">★ 잇는 것이 없다</text>
|
||||
<text x="316" y="140" class="t">1. ExecStartPost</text>
|
||||
<text x="470" y="140" class="t" fill="#cf222e">없음</text>
|
||||
<text x="316" y="160" class="t">2. renewal-hooks/deploy/</text>
|
||||
<text x="470" y="160" class="t" fill="#cf222e">비었음</text>
|
||||
<text x="316" y="180" class="t">3. certbot nginx 플러그인</text>
|
||||
<text x="470" y="180" class="t" fill="#cf222e">없음</text>
|
||||
<text x="316" y="206" class="s">post/ · pre/ 도 비어 있다 (total 8)</text>
|
||||
<text x="316" y="222" class="s">셋 다 비면 아무도 reload 를 부르지 않는다</text>
|
||||
<path d="M270 168 L296 168" class="r" marker-end="url(#ar)"/>
|
||||
<path d="M300 230 L560 108" stroke="#cf222e" stroke-width="1.2" stroke-dasharray="4 3"/>
|
||||
<path d="M300 108 L560 230" stroke="#cf222e" stroke-width="1.2" stroke-dasharray="4 3"/>
|
||||
|
||||
<!-- 우: nginx -->
|
||||
<rect x="590" y="70" width="250" height="196" class="box"/>
|
||||
<text x="604" y="92" class="tb">nginx — 메모리</text>
|
||||
<rect x="606" y="104" width="218" height="60" class="dead"/>
|
||||
<text x="618" y="124" class="t">기동 시 읽은 cert1</text>
|
||||
<text x="618" y="140" class="s">master 585 / worker 586</text>
|
||||
<text x="618" y="156" class="s">둘 다 09-03 19:00:39, 22.4시간째</text>
|
||||
<text x="606" y="184" class="s">reload 없이는 디스크를 다시 안 본다.</text>
|
||||
<text x="606" y="200" class="s">reload 하면 마스터는 남고</text>
|
||||
<text x="606" y="216" class="s">워커만 새로 뜬다 → 워커 PID 로 확인</text>
|
||||
<text x="606" y="240" class="s">ssl_certificate .../fullchain.pem</text>
|
||||
<path d="M564 168 L586 168" class="k" marker-end="url(#a)"/>
|
||||
|
||||
<!-- 타임라인 -->
|
||||
<text x="20" y="306" class="tb">밖에서 본 것 — 일련번호 (5초 간격)</text>
|
||||
<line x1="60" y1="356" x2="800" y2="356" class="k"/>
|
||||
<line x1="60" y1="350" x2="60" y2="362" class="k"/>
|
||||
<line x1="300" y1="350" x2="300" y2="362" class="k"/>
|
||||
<line x1="660" y1="344" x2="660" y2="368" stroke="#1a7f37" stroke-width="2"/>
|
||||
<line x1="800" y1="350" x2="800" y2="362" class="k"/>
|
||||
<text x="46" y="382" class="s">08:10:51</text>
|
||||
<text x="278" y="382" class="s">08:22:13</text>
|
||||
<text x="620" y="382" class="s">08:58:52</text>
|
||||
<text x="770" y="382" class="s">09:02</text>
|
||||
<text x="278" y="398" class="s">디스크 기록</text>
|
||||
<text x="618" y="398" class="s">nginx -s reload</text>
|
||||
|
||||
<rect x="60" y="330" width="600" height="16" class="dead"/>
|
||||
<text x="300" y="342" class="s" text-anchor="middle" fill="#cf222e">0520BB…B853 (옛 인증서) — 이 구간에서 428회 관측</text>
|
||||
<rect x="660" y="330" width="140" height="16" class="ok"/>
|
||||
<text x="730" y="342" class="s" text-anchor="middle" fill="#1a7f37">06C7CB…EA1D</text>
|
||||
<path d="M300 416 L660 416" class="r" marker-end="url(#ar)"/>
|
||||
<path d="M660 416 L300 416" class="r" marker-end="url(#ar)"/>
|
||||
<text x="480" y="410" class="tb" text-anchor="middle" fill="#cf222e">38분 25초</text>
|
||||
<text x="480" y="434" class="s" text-anchor="middle">사람이 reload 했기에 36분이다. 아무도 안 했다면 다음 재시작까지 — 무기한</text>
|
||||
|
||||
<!-- reload 판정 -->
|
||||
<rect x="20" y="452" width="400" height="52" class="ok"/>
|
||||
<text x="34" y="472" class="tb">reload 는 무중단이었다</text>
|
||||
<text x="34" y="490" class="s">새 연결 8856건 전부 200 · p95 205.7 → 204.3ms (변화 없음)</text>
|
||||
<rect x="440" y="452" width="400" height="52" class="ok"/>
|
||||
<text x="454" y="472" class="tb">진행 중이던 요청도 살아남았다</text>
|
||||
<text x="454" y="490" class="s">전송 12초째에 reload · 845361바이트 전량 · 연결수 1</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 5.6 KiB |
@@ -0,0 +1,57 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 430" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,sans-serif">
|
||||
<style>
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.dead{fill:#fff5f5;stroke:#cf222e;stroke-width:1.6;stroke-dasharray:5 3;rx:6}
|
||||
.ok{fill:#f6fdf6;stroke:#1a7f37;stroke-width:1.6;rx:6}
|
||||
.warn{fill:#fff8f0;stroke:#bf8700;stroke-width:1.5;rx:6}
|
||||
.t{font-size:12px;fill:#1f2328}
|
||||
.tb{font-size:12.5px;fill:#1f2328;font-weight:600}
|
||||
.s{font-size:10.5px;fill:#656d76}
|
||||
.ttl{font-size:14px;fill:#1f2328;font-weight:600}
|
||||
.r{stroke:#cf222e;stroke-width:1.6;fill:none}
|
||||
.g{stroke:#1a7f37;stroke-width:1.6;fill:none}
|
||||
.k{stroke:#8c959f;stroke-width:1.3;fill:none}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="a" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#8c959f"/></marker>
|
||||
<marker id="ar" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#cf222e"/></marker>
|
||||
<marker id="ag" markerWidth="8" markerHeight="8" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 z" fill="#1a7f37"/></marker>
|
||||
</defs>
|
||||
<text x="20" y="26" class="ttl">D-4a — deploy 훅 하나로 38분 25초가 1~2초가 된다</text>
|
||||
<text x="20" y="44" class="s">시각은 전부 실제 UTC (test-server 시계에서 106초 보정)</text>
|
||||
|
||||
<rect x="20" y="68" width="820" height="112" class="dead"/>
|
||||
<text x="34" y="90" class="tb">D-4 — 훅이 없을 때</text>
|
||||
<line x1="60" y1="130" x2="800" y2="130" class="k"/>
|
||||
<line x1="90" y1="120" x2="90" y2="140" stroke="#cf222e" stroke-width="2"/>
|
||||
<line x1="740" y1="120" x2="740" y2="140" stroke="#1a7f37" stroke-width="2"/>
|
||||
<rect x="90" y="122" width="650" height="16" class="dead"/>
|
||||
<text x="415" y="134" class="s" text-anchor="middle" fill="#cf222e">옛 인증서를 계속 서빙 — 428회 관측</text>
|
||||
<text x="62" y="158" class="s">08:20:27 디스크 기록</text>
|
||||
<text x="660" y="158" class="s">08:58:52 사람이 nginx -s reload</text>
|
||||
<text x="415" y="112" class="tb" text-anchor="middle" fill="#cf222e">2305초 = 38분 25초</text>
|
||||
<text x="34" y="174" class="s">아무도 reload 하지 않았다면 다음 nginx 재시작까지 = 사실상 무기한</text>
|
||||
|
||||
<rect x="20" y="196" width="820" height="112" class="ok"/>
|
||||
<text x="34" y="218" class="tb">D-4a — deploy 훅을 넣었을 때</text>
|
||||
<line x1="60" y1="258" x2="800" y2="258" class="k"/>
|
||||
<line x1="90" y1="248" x2="90" y2="268" stroke="#1a7f37" stroke-width="2"/>
|
||||
<line x1="104" y1="248" x2="104" y2="268" stroke="#1a7f37" stroke-width="2"/>
|
||||
<rect x="90" y="250" width="14" height="16" class="ok"/>
|
||||
<text x="62" y="286" class="s">12:27:49.05 발급 (SCT — CT 로그의 독립 시계)</text>
|
||||
<text x="360" y="286" class="s">12:27:50 nginx -t · 새 워커 37252 · 12:27:51 reload</text>
|
||||
<text x="150" y="244" class="tb" fill="#1a7f37">1~2초, 자동</text>
|
||||
<text x="34" y="302" class="s">certbot 이 파일을 쓴 직후 같은 실행 안에서 deploy 훅을 부른다 — 사람이 개입할 자리가 없다</text>
|
||||
|
||||
<rect x="20" y="324" width="400" height="92" class="box"/>
|
||||
<text x="34" y="346" class="tb">판정은 문구가 아니라 워커 PID 로</text>
|
||||
<text x="34" y="366" class="s">master 585 → 585 (유지)</text>
|
||||
<text x="34" y="384" class="s">worker 28829 → 37252 (교체 = reload 됨)</text>
|
||||
<text x="34" y="404" class="s">nginx 는 인증서를 기동 시 읽어 메모리에 든다</text>
|
||||
|
||||
<rect x="444" y="324" width="396" height="92" class="warn"/>
|
||||
<text x="458" y="346" class="tb">★ 함정 — 성공을 실패로 오독한다</text>
|
||||
<text x="458" y="366" class="s">Hook 'deploy-hook' ran with error output:</text>
|
||||
<text x="458" y="384" class="s"> [warn] could not build optimal types_hash …</text>
|
||||
<text x="458" y="402" class="s">nginx 경고가 stderr 로 나갔을 뿐. 내용은 test is successful</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.8 KiB |
@@ -0,0 +1,57 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 470" width="760" height="470" font-family="system-ui,-apple-system,Segoe UI,Roboto,sans-serif">
|
||||
<style>
|
||||
.t{font-size:13px;fill:#1f2328}.s{font-size:11px;fill:#59636e}.h{font-size:14px;font-weight:600;fill:#1f2328}
|
||||
.box{fill:#fff;stroke:#8c959f;stroke-width:1.4;rx:6}
|
||||
.host{fill:#f6f8fa;stroke:#59636e;stroke-width:1.6;rx:8}
|
||||
.node{fill:#fff;stroke:#0969da;stroke-width:1.6;rx:8}
|
||||
.ln{stroke:#59636e;stroke-width:1.4;fill:none;marker-end:url(#a)}
|
||||
</style>
|
||||
<defs><marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto"><path d="M0,0 L9,4.5 L0,9 z" fill="#59636e"/></marker></defs>
|
||||
|
||||
<rect class="box" x="290" y="12" width="180" height="38"/>
|
||||
<text class="t" x="380" y="30" text-anchor="middle">개발 노트북</text>
|
||||
<text class="s" x="380" y="44" text-anchor="middle">브라우저 · kubectl · Playwright</text>
|
||||
<path class="ln" d="M380,52 L380,80"/>
|
||||
<text class="s" x="392" y="70">https · tailnet 100.x · split DNS</text>
|
||||
|
||||
<rect class="host" x="40" y="84" width="680" height="66"/>
|
||||
<text class="h" x="56" y="106">test-server</text>
|
||||
<text class="s" x="56" y="122">Arch Linux · 12GB · WiFi only · sudo 는 비밀번호 필요</text>
|
||||
<rect class="box" x="430" y="94" width="270" height="46"/>
|
||||
<text class="t" x="565" y="112" text-anchor="middle">nginx :443 — TLS 종료</text>
|
||||
<text class="s" x="565" y="128" text-anchor="middle">auth / app1 / app2 (SAN 3개, 와일드카드 아님)</text>
|
||||
|
||||
<path class="ln" d="M240,152 L200,186"/><path class="ln" d="M520,152 L560,186"/>
|
||||
<text class="s" x="330" y="172" text-anchor="middle">http · libvirt NAT (virbr0)</text>
|
||||
|
||||
<rect class="node" x="40" y="190" width="320" height="250"/>
|
||||
<text class="h" x="56" y="212">kc-lab-1 · 5120MB</text>
|
||||
<text class="s" x="56" y="228">k3s server · 10.42.0.0/24</text>
|
||||
<rect class="box" x="56" y="238" width="130" height="26"/><text class="t" x="121" y="255" text-anchor="middle">traefik ×1</text>
|
||||
<rect class="box" x="196" y="238" width="148" height="26"/><text class="t" x="270" y="255" text-anchor="middle">coredns</text>
|
||||
<rect class="box" x="56" y="272" width="130" height="26"/><text class="t" x="121" y="289" text-anchor="middle">keycloak-1</text>
|
||||
<rect class="box" x="196" y="272" width="148" height="26"/><text class="t" x="270" y="289" text-anchor="middle">bff (1/2)</text>
|
||||
<rect class="box" x="56" y="306" width="288" height="26"/><text class="t" x="200" y="323" text-anchor="middle">oauth2-proxy (1/2)</text>
|
||||
<rect class="box" x="56" y="340" width="288" height="46"/>
|
||||
<text class="t" x="200" y="358" text-anchor="middle">prometheus (PVC) · grafana</text>
|
||||
<text class="s" x="200" y="374" text-anchor="middle">관측 스택은 여기 고정 — 죽이지 않는다</text>
|
||||
|
||||
<rect class="node" x="400" y="190" width="320" height="250"/>
|
||||
<text class="h" x="416" y="212">kc-lab-2 · 4096MB</text>
|
||||
<text class="s" x="416" y="228">k3s agent · 10.42.1.0/24</text>
|
||||
<rect class="box" x="416" y="238" width="288" height="26"/><text class="t" x="560" y="255" text-anchor="middle">keycloak-0</text>
|
||||
<rect class="box" x="416" y="272" width="140" height="26"/><text class="t" x="486" y="289" text-anchor="middle">bff (2/2)</text>
|
||||
<rect class="box" x="566" y="272" width="138" height="26"/><text class="t" x="635" y="289" text-anchor="middle">oauth2-proxy</text>
|
||||
<rect class="box" x="416" y="306" width="140" height="46"/>
|
||||
<text class="t" x="486" y="324" text-anchor="middle">postgres</text><text class="s" x="486" y="340" text-anchor="middle">PVC (노드 고정)</text>
|
||||
<rect class="box" x="566" y="306" width="138" height="46"/>
|
||||
<text class="t" x="635" y="324" text-anchor="middle">redis</text><text class="s" x="635" y="340" text-anchor="middle">PVC + AOF</text>
|
||||
<text class="s" x="560" y="374" text-anchor="middle">장애 주입은 여기</text>
|
||||
|
||||
<path d="M360,290 L400,290" stroke="#0969da" stroke-width="1.6" fill="none" marker-end="url(#a)"/>
|
||||
<path d="M400,300 L360,300" stroke="#0969da" stroke-width="1.6" fill="none" marker-end="url(#a)"/>
|
||||
<text class="s" x="380" y="284" text-anchor="middle">7800</text>
|
||||
<text class="s" x="380" y="318" text-anchor="middle">JGroups</text>
|
||||
|
||||
<text class="s" x="40" y="460">A-0 에서 확인: 세션은 이 7800 이 아니라 postgres 를 통해 공유된다</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.3 KiB |
+13
@@ -0,0 +1,13 @@
|
||||
=== [기준선 1] 클러스터 뷰 — 양쪽 파드의 마지막 ISPN000094 ===
|
||||
keycloak-0: [keycloak-1-48749(v=16.0.12)|5] (2) [keycloak-1-48749(v=16.0.12), keycloak-0-30843(v=16.0.12)]
|
||||
keycloak-1: [keycloak-1-48749(v=16.0.12)|5] (2) [keycloak-1-48749(v=16.0.12), keycloak-0-30843(v=16.0.12)]
|
||||
|
||||
=== [기준선 2] JGROUPS_PING — 디스커버리 등록 ===
|
||||
name | ip | coord | coordinated_by
|
||||
------------------+-----------------+-------+---------------------------------------------
|
||||
keycloak-0-30843 | 10.42.1.43:7800 | f | uuid://00000000-0000-0000-0000-000000000007
|
||||
keycloak-1-48749 | 10.42.0.35:7800 | t | uuid://00000000-0000-0000-0000-000000000007
|
||||
(2 rows)
|
||||
|
||||
=== [기준선 3] 기존 NetworkPolicy ===
|
||||
No resources found in keycloak-lab namespace.
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
=== [대조군] 차단 전 — keycloak-0 로그인 → keycloak-1 에서 refresh ===
|
||||
sid tAWs2gCPr6SOcD4jDR9-_CzB
|
||||
keycloak-1 에서 refresh: 200
|
||||
|
||||
=== [기준선 4] JGroups 지표 — 양쪽 노드 ===
|
||||
--- K0 (10.42.1.43) ---
|
||||
vendor_jgroups_stats_bytes_sent_total 31476.0
|
||||
vendor_jgroups_merge3_get_num_merge_events 0.0
|
||||
vendor_jgroups_merge3_get_views 0.0
|
||||
vendor_jgroups_fd_sock2_get_num_suspected_members 0.0
|
||||
vendor_jgroups_nakack2_get_xmit_table_missing_messages 0.0
|
||||
--- K1 (10.42.0.35) ---
|
||||
vendor_jgroups_merge3_get_views 0.0
|
||||
vendor_jgroups_stats_bytes_sent_total 126765.0
|
||||
vendor_jgroups_nakack2_get_xmit_table_missing_messages 0.0
|
||||
vendor_jgroups_fd_sock2_get_num_suspected_members 0.0
|
||||
vendor_jgroups_merge3_get_num_merge_events 0.0
|
||||
+8
@@ -0,0 +1,8 @@
|
||||
=== 차단 적용 ===
|
||||
networkpolicy.networking.k8s.io/a1-block-jgroups-transport created
|
||||
a1-block-jgroups-transport map[app:keycloak]
|
||||
|
||||
적용 시각: 11:38:08
|
||||
=== FD_SOCK2 가 상대를 의심하기까지 기다린다 (15초 간격, 최대 3분) ===
|
||||
+15초 suspected(k0 k1) =
|
||||
→ 변화 감지
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
차단 경과: 11:38:51 (적용 11:38:08)
|
||||
|
||||
=== [차단 후 1] JGroups 지표 ===
|
||||
--- keycloak-0 ---
|
||||
vendor_jgroups_stats_bytes_sent_total 32857.0
|
||||
vendor_jgroups_merge3_get_num_merge_events 0.0
|
||||
vendor_jgroups_merge3_get_views 0.0
|
||||
vendor_jgroups_fd_sock2_get_num_suspected_members 0.0
|
||||
vendor_jgroups_nakack2_get_xmit_table_missing_messages 0.0
|
||||
--- keycloak-1 ---
|
||||
vendor_jgroups_merge3_get_views 0.0
|
||||
vendor_jgroups_stats_bytes_sent_total 129103.0
|
||||
vendor_jgroups_nakack2_get_xmit_table_missing_messages 0.0
|
||||
vendor_jgroups_fd_sock2_get_num_suspected_members 0.0
|
||||
vendor_jgroups_merge3_get_num_merge_events 0.0
|
||||
|
||||
=== [차단 후 2] 클러스터 뷰 — 갈라졌는가 ===
|
||||
keycloak-0:
|
||||
keycloak-1:
|
||||
=== [차단 후 3] JGROUPS_PING — 디스커버리는 살아 있는가 ===
|
||||
name | ip | coord
|
||||
------------------+-----------------+-------
|
||||
keycloak-0-30843 | 10.42.1.43:7800 | f
|
||||
keycloak-1-48749 | 10.42.0.35:7800 | t
|
||||
(2 rows)
|
||||
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
=== [문제 확정] NetworkPolicy 적용 후에도 기존 연결이 conntrack 에 살아 있다 ===
|
||||
--- kc-lab-1 ---
|
||||
tcp 6 86398 ESTABLISHED src=10.42.0.35 dst=10.42.1.43 sport=40023 dport=7800 src=10.42.1.43 dst=10.42.0.35 sport=7800 dport=40023 [ASSURED] mark=0 use=1
|
||||
tcp 6 79982 ESTABLISHED src=10.42.0.35 dst=10.42.1.43 sport=50477 dport=57800 src=10.42.1.43 dst=10.42.0.35 sport=57800 dport=50477 [ASSURED] mark=0 use=1
|
||||
--- kc-lab-2 ---
|
||||
tcp 6 86398 ESTABLISHED src=10.42.0.35 dst=10.42.1.43 sport=40023 dport=7800 src=10.42.1.43 dst=10.42.0.35 sport=7800 dport=40023 [ASSURED] mark=0 use=1
|
||||
tcp 6 33 SYN_SENT src=10.42.1.58 dst=10.42.0.35 sport=34824 dport=7800 [UNREPLIED] src=10.42.0.35 dst=10.42.1.58 sport=7800 dport=34824 mark=0 use=1
|
||||
tcp 6 79982 ESTABLISHED src=10.42.0.35 dst=10.42.1.43 sport=50477 dport=57800 src=10.42.1.43 dst=10.42.0.35 sport=57800 dport=50477 [ASSURED] mark=0 use=1
|
||||
|
||||
=== [조치] 7800 흐름의 conntrack 항목을 지운다 → 다음 패킷이 정책을 다시 탄다 ===
|
||||
kc-lab-1: tcp 6 86398 ESTABLISHED src=10.42.0.35 dst=10.42.1.43 sport=40023 dport=7800 src=10.42.1.43 dst=10.42.0.35 sport=7800 dport=40023 [ASSURED] mark=0 use=1 conntrack v1.4.7 (conntrack-tools): 0 flow entries have been deleted.
|
||||
kc-lab-2: tcp 6 33 SYN_SENT src=10.42.1.58 dst=10.42.0.35 sport=34824 dport=7800 [UNREPLIED] src=10.42.0.35 dst=10.42.1.58 sport=7800 dport=34824 mark=0 use=1 conntrack v1.4.7 (conntrack-tools): 0 flow entries have been deleted.
|
||||
|
||||
=== 삭제 후 7800 conntrack ===
|
||||
kc-lab-1: 2 건
|
||||
kc-lab-2: 2 건
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
관찰 시작: 11:42:03
|
||||
+20초 suspected(k0 k1) = []
|
||||
+40초 suspected(k0 k1) = []
|
||||
+60초 suspected(k0 k1) = [0.0 0.0 0.0 0.0 ]
|
||||
+80초 suspected(k0 k1) = []
|
||||
+100초 suspected(k0 k1) = []
|
||||
+120초 suspected(k0 k1) = []
|
||||
+140초 suspected(k0 k1) = [0.0 ]
|
||||
+160초 suspected(k0 k1) = [0.0 0.0 0.0 0.0 ]
|
||||
|
||||
=== 클러스터 뷰 변화 (최근 8분) ===
|
||||
--- keycloak-0 ---
|
||||
--- keycloak-1 ---
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
=== vendor_cluster_size — 지난 25분 (차단 11:38:08, conntrack 삭제 11:41) ===
|
||||
keycloak-0:
|
||||
11:20=2 11:21=2 11:22=2 11:23=2 11:24=2 11:25=2 11:26=2 11:27=2 11:28=2 11:29=2 11:30=2 11:31=2 11:32=2 11:33=2 11:34=2 11:35=2 11:36=2 11:37=2 11:38=2 11:39=2 11:40=2 11:41=2 11:42=2 11:43=2 11:44=2 11:45=2
|
||||
keycloak-1:
|
||||
11:20=2 11:21=2 11:22=2 11:23=2 11:24=2 11:25=2 11:26=2 11:27=2 11:28=2 11:29=2 11:30=2 11:31=2 11:32=2 11:33=2 11:34=2 11:35=2 11:36=2 11:37=2 11:38=2 11:39=2 11:40=2 11:41=2 11:42=2 11:43=2 11:44=2 11:45=2
|
||||
|
||||
=== 현재 값 ===
|
||||
keycloak-1 = 2 멤버
|
||||
keycloak-0 = 2 멤버
|
||||
|
||||
=== 7800 소켓 상태 (파드 내부) ===
|
||||
keycloak-0 2
|
||||
keycloak-1 2
|
||||
=== conntrack ===
|
||||
kc-lab-1 1 건
|
||||
kc-lab-2 1 건
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
=== 정책이 걸린 상태에서 keycloak-0 을 재시작한다 → 재연결이 막힌다 ===
|
||||
재시작 시각: 11:46:07
|
||||
pod "keycloak-0" deleted from keycloak-lab namespace
|
||||
keycloak-0 false 10.42.1.67 2026-09-04T02:44:23Z
|
||||
|
||||
=== cluster_size 추이 ===
|
||||
keycloak-0: 11:45:27=1 11:45:57=1 11:46:27=1 11:46:57=1 11:47:27=1
|
||||
keycloak-0: 11:40:57=2 11:41:27=2 11:41:57=2 11:42:27=2 11:42:57=2 11:43:27=2 11:43:57=2
|
||||
keycloak-1: 11:40:57=2 11:41:27=2 11:41:57=2 11:42:27=2 11:42:57=2 11:43:27=2 11:43:57=2 11:44:27=1 11:44:57=1 11:45:27=1 11:45:57=1 11:46:27=1 11:46:57=1 11:47:27=1
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
=== keycloak-0 헬스 상태 ===
|
||||
keycloak-0 = 10.42.1.67 keycloak-1 = 10.42.0.35
|
||||
PodReadyToStartContainers=True
|
||||
Initialized=True
|
||||
Ready=False ContainersNotReady
|
||||
ContainersReady=False ContainersNotReady
|
||||
PodScheduled=True
|
||||
|
||||
=== ★ 본 시험 — 분단 상태에서 교차 노드 세션이 되는가 ===
|
||||
[1] keycloak-0 로그인 sid=nShl5TaBrZnKStDqaspjgmJB
|
||||
[2] keycloak-1 에서 refresh HTTP 200
|
||||
[3] keycloak-1 에서 로그아웃 HTTP 204
|
||||
[4] keycloak-0 에서 재갱신 시도 HTTP 200
|
||||
(400 이면 무효화가 전파된 것)
|
||||
|
||||
=== DB 세션 수 ===
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
=== 그 sid 가 DB 에 남아 있는가 ===
|
||||
user_session_id | offline_flag | last_session_refresh
|
||||
-----------------+--------------+----------------------
|
||||
(0 rows)
|
||||
|
||||
=== 전체 온라인 세션 수 ===
|
||||
1
|
||||
|
||||
=== 노드별 세션 캐시 엔트리 (Prometheus) ===
|
||||
keycloak-1 kc-lab-1 = 0
|
||||
keycloak-0 kc-lab-2 = 1
|
||||
|
||||
=== keycloak-0 이 Ready 가 아닌 이유 — 헬스 응답 ===
|
||||
{
|
||||
"status": "DOWN",
|
||||
"checks": [
|
||||
{
|
||||
"name": "Graceful Shutdown",
|
||||
"status": "UP"
|
||||
},
|
||||
{
|
||||
"name": "Keycloak cluster health check",
|
||||
"status": "DOWN",
|
||||
"data": {
|
||||
"Failing since": "2026-09-04 02:45:14,251"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Keycloak database connections async health check",
|
||||
"status": "UP"
|
||||
},
|
||||
{
|
||||
"name": "Keycloak Initialized",
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
=== 양쪽 노드의 readiness — 둘 다 DOWN 이면 전면 장애다 ===
|
||||
Traceback (most recent call last):
|
||||
File "<string>", line 3, in <module>
|
||||
d=json.load(sys.stdin)
|
||||
File "/usr/lib/python3.14/json/__init__.py", line 298, in load
|
||||
return loads(fp.read(),
|
||||
cls=cls, object_hook=object_hook,
|
||||
parse_float=parse_float, parse_int=parse_int,
|
||||
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
|
||||
File "/usr/lib/python3.14/json/__init__.py", line 352, in loads
|
||||
return _default_decoder.decode(s)
|
||||
~~~~~~~~~~~~~~~~~~~~~~~^^^
|
||||
File "/usr/lib/python3.14/json/decoder.py", line 348, in decode
|
||||
raise JSONDecodeError("Extra data", s, end)
|
||||
json.decoder.JSONDecodeError: Extra data: line 21 column 2 (char 446)
|
||||
|
||||
=== 파드 Ready 상태 ===
|
||||
keycloak-0 false 0
|
||||
keycloak-1 true 0
|
||||
|
||||
=== ★ Service 엔드포인트 — 트래픽을 받는 파드가 남아 있는가 ===
|
||||
ready 주소: [10.42.0.35] notReady : [10.42.1.67]
|
||||
=== ★ 외부 진입점으로 실제 로그인이 되는가 (nginx→Traefik→Service) ===
|
||||
https://auth.hyeonworks.com/realms/master HTTP 200
|
||||
토큰 발급 HTTP 200
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
=== 차단 해제 ===
|
||||
해제 시각: 11:49:58
|
||||
networkpolicy.networking.k8s.io "a1-block-jgroups-transport" deleted from keycloak-lab namespace
|
||||
|
||||
=== 자동으로 다시 붙는가 (30초 간격, 최대 4분) ===
|
||||
+30초 keycloak-0=1 keycloak-1=1 | Ready 파드 2 개
|
||||
+60초 keycloak-0=1 keycloak-1=1 | Ready 파드 2 개
|
||||
+90초 keycloak-0=2 keycloak-1=2 | Ready 파드 3 개
|
||||
→ 클러스터 재형성
|
||||
|
||||
=== 복구 로그 ===
|
||||
keycloak-0: [keycloak-0-26403(v=16.0.12)|0] (1) [keycloak-0-26403(v=16.0.12)]
|
||||
keycloak-1: [keycloak-1-48749(v=16.0.12)|6] (1) [keycloak-1-48749(v=16.0.12)]
|
||||
|
||||
=== MERGE3 가 합쳤는가 ===
|
||||
merge_events keycloak-1 = 1
|
||||
merge_events keycloak-0 = 1
|
||||
=== JGROUPS_PING — 코디네이터가 하나로 돌아왔는가 ===
|
||||
name | ip | coord
|
||||
------------------+-----------------+-------
|
||||
keycloak-0-26403 | 10.42.1.67:7800 | t
|
||||
keycloak-1-48749 | 10.42.0.35:7800 | f
|
||||
(2 rows)
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
# A-1 — JGroups 트랜스포트(7800) 차단 증거
|
||||
|
||||
2026-09-04 11:38–11:52 KST · Keycloak 26.7.0 / Infinispan 16.0.12
|
||||
해설: [`docs/experiment-a1-jgroups-transport-block.md`](../../experiment-a1-jgroups-transport-block.md)
|
||||
|
||||
| 파일 | 무엇을 보여주는가 |
|
||||
|---|---|
|
||||
| `01-baseline-cluster.txt` | 차단 전 — 양쪽이 뷰 ID 5·멤버 2로 일치, `JGROUPS_PING` 코디네이터 1명 |
|
||||
| `02-control-before-block.txt` | **대조군** — 차단 전 교차 노드 refresh `200`, JGroups 지표 전부 0 |
|
||||
| `03-block-applied.txt` | NetworkPolicy 적용. **빈 측정값을 "변화 감지"로 오판한 기록** |
|
||||
| `04-after-block-state.txt` | 차단 43초 후 — 지표 무변화, `JGROUPS_PING` 그대로 |
|
||||
| `05-conntrack-problem.txt` | **핵심 문제** — `ESTABLISHED [ASSURED]` 로 기존 연결이 살아 있음. FD_SOCK2 의 **57800** 포트도 함께 드러남 |
|
||||
| `06-partition-observed.txt` | 임시 curl 파드 폴링의 실패 — 빈 값·개수 불일치 |
|
||||
| `07-cluster-size.txt` | **`vendor_cluster_size` 가 25분 내내 2** — 분단이 일어나지 않았다는 결정적 증거 |
|
||||
| `08-restart-forced-partition.txt` | 재연결 강제 후 `2 → 1` |
|
||||
| `09-cross-node-under-partition.txt` | **본 시험** — 교차 refresh `200`(예측 적중), **로그아웃 후 재갱신 `200`(예측 빗나감)** |
|
||||
| `10-logout-not-propagated.txt` | 기제 확정 — **DB 행 0건인데 keycloak-0 캐시에 1건**, 헬스체크 `cluster health: DOWN` |
|
||||
| `11-service-impact.txt` | **분단 노드가 Service 에서 빠짐.** `ready=[10.42.0.35] notReady=[10.42.1.67]`, 외부 로그인 `200` |
|
||||
| `12-recovery.txt` | 90초 만에 자동 재형성, `merge3_get_num_merge_events = 1`, 코디네이터 재선출 |
|
||||
| `a1-cluster-size-partition-recovery.png` | Grafana — `vendor_cluster_size` 가 `2 → 1 → 2` 로 움직이는 전 구간 |
|
||||
|
||||
## 핵심 세 줄
|
||||
|
||||
1. **NetworkPolicy 만으로는 이미 붙어 있는 클러스터를 못 끊는다.** conntrack 의 ESTABLISHED 가 먼저 통과시킨다.
|
||||
2. **세션 공유는 분단을 견딘다(200).** 통념이 틀렸고 A-0 모델이 맞다.
|
||||
3. **로그아웃 무효화는 7800 을 탄다.** DB 행이 지워져도 반대편은 낡은 캐시로 200 을 준다 — A-0 의 인과 해석을 정정한다.
|
||||
BIN
Binary file not shown.
|
After Width: | Height: | Size: 66 KiB |
@@ -0,0 +1,14 @@
|
||||
=== A-2 기준선 — 클러스터가 정상으로 돌아왔는가 ===
|
||||
keycloak-0 true 10.42.1.67 kc-lab-2
|
||||
keycloak-1 true 10.42.0.35 kc-lab-1
|
||||
postgres-7b474b88c8-sn9ff true 10.42.1.24 kc-lab-2
|
||||
|
||||
cluster_size keycloak-1 = 2
|
||||
cluster_size keycloak-0 = 2
|
||||
|
||||
=== 노드별 세션 캐시 (실험 설계에 필요) ===
|
||||
keycloak-1 kc-lab-1 = 0 건
|
||||
keycloak-0 kc-lab-2 = 0 건
|
||||
|
||||
=== DB 온라인 세션 ===
|
||||
2
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
pod/a2-probe condition met
|
||||
keycloak-0=10.42.1.67 keycloak-1=10.42.0.35
|
||||
|
||||
=== [준비] 양쪽 노드에 세션을 하나씩 만든다 ===
|
||||
keycloak-0 에서 로그인 sid=EAXV5HcG2J1BZ3vnwONf64AQ 토큰길이=613
|
||||
keycloak-1 에서 로그인 sid=McyTj5lj3n_JqApCXeuAHExc 토큰길이=613
|
||||
|
||||
=== [확인] 세션이 각자 노드에만 캐시되었는가 ===
|
||||
keycloak-1 = 0 건
|
||||
keycloak-0 = 1 건
|
||||
@@ -0,0 +1,16 @@
|
||||
=== [1] 토큰을 새로 발급 (access 수명 60초) ===
|
||||
발급 완료 sid=RKXQGAgkuLtouFMVPTFmp_0_
|
||||
|
||||
=== [2] PostgreSQL 정지 ===
|
||||
정지 시각: 11:56:04
|
||||
deployment.apps/postgres scaled
|
||||
pod/postgres-7b474b88c8-sn9ff condition met
|
||||
삭제 완료: 11:56:04
|
||||
|
||||
=== [3] 네 경로를 즉시 시험 ===
|
||||
④ 이미 발급된 access token 으로 관리 API HTTP 000000{"error":"HTTP 401 Unauthorized"}401
|
||||
① 캐시를 가진 노드(keycloak-0)에서 refresh HTTP 500
|
||||
② 캐시가 없는 노드(keycloak-1)에서 refresh HTTP 500
|
||||
③ 새 로그인 HTTP 500
|
||||
--- 오류 본문 (새 로그인) ---
|
||||
{"error":"unknown_error","error_description":"For more on this error consult the server log."}
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
=== 파드 Ready 상태 — DB 가 없으면 어떻게 되는가 ===
|
||||
keycloak-0 false 0
|
||||
keycloak-1 false 0
|
||||
|
||||
=== Service 엔드포인트 ===
|
||||
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
|
||||
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
|
||||
notReady: [10.42.0.35 10.42.1.67]
|
||||
=== health/ready 상세 ===
|
||||
전체: DOWN
|
||||
Graceful Shutdown UP
|
||||
Keycloak cluster health check UP
|
||||
Keycloak database connections async health check DOWN
|
||||
Keycloak Initialized UP
|
||||
|
||||
=== ④ 다시 — 서명 검증만 필요한 경로는 살아 있는가 ===
|
||||
JWKS 엔드포인트(realm 공개키) HTTP 200
|
||||
realm 메타데이터(.well-known) HTTP 200
|
||||
관리 API(세션 조회 필요) HTTP 500
|
||||
|
||||
=== 외부 진입점 ===
|
||||
https://auth.hyeonworks.com/realms/master HTTP 503
|
||||
|
||||
=== Keycloak 로그 — 실제 오류 ===
|
||||
at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:664)
|
||||
at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:645)
|
||||
Caused by: java.net.ConnectException: Connection refused
|
||||
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:219)
|
||||
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:365)
|
||||
@@ -0,0 +1,21 @@
|
||||
=== ★ up 지표는 무엇을 말하는가 (프로세스는 살아 있다) ===
|
||||
up{pod=keycloak-1} = 1 ← 1 인데 서비스는 503 이다
|
||||
up{pod=keycloak-0} = 1 ← 1 인데 서비스는 503 이다
|
||||
|
||||
=== 복구 — PostgreSQL 재기동 ===
|
||||
재기동 시각: 11:57:09
|
||||
deployment.apps/postgres scaled
|
||||
Waiting for deployment "postgres" rollout to finish: 0 out of 1 new replicas have been updated...
|
||||
Waiting for deployment "postgres" rollout to finish: 0 of 1 updated replicas are available...
|
||||
deployment "postgres" successfully rolled out
|
||||
|
||||
=== Keycloak 이 스스로 회복하는가 (재시작 없이) ===
|
||||
+15초 keycloak-0 true keycloak-1 true | 외부 HTTP 200
|
||||
→ 서비스 복귀
|
||||
|
||||
=== 재시작 횟수 — 파드가 죽었다 살아난 것인가, 그대로 회복한 것인가 ===
|
||||
keycloak-0 0
|
||||
keycloak-1 0
|
||||
|
||||
=== 정지 전 세션이 살아남았는가 ===
|
||||
online 세션 5
|
||||
@@ -0,0 +1,19 @@
|
||||
# A-2 — PostgreSQL 정지 증거
|
||||
|
||||
2026-09-04 11:56–11:58 KST · Keycloak 26.7.0
|
||||
해설: [`docs/experiment-a2-database-loss.md`](../../experiment-a2-database-loss.md)
|
||||
|
||||
| 파일 | 무엇을 보여주는가 |
|
||||
|---|---|
|
||||
| `01-baseline.txt` | 정지 전 — 양쪽 Ready, `cluster_size=2` |
|
||||
| `02-setup-sessions.txt` | 양쪽 노드에 세션 하나씩. 캐시는 각자 노드에만 |
|
||||
| `03-four-paths.txt` | **네 경로 전부 `500`.** 캐시를 가진 노드도 실패 — refresh 는 쓰기다 |
|
||||
| `04-health-and-service.txt` | **전면 장애 증거** — Ready 파드 0개, `ready 주소=[]`, 외부 **503**, `database connections: DOWN`. JWKS·.well-known 은 `200` |
|
||||
| `05-recovery.txt` | **`up=1` 인 채로 503.** DB 복귀 15초 후 재시작 0회로 자동 회복, 세션 5건 생존 |
|
||||
| `a2-up-stayed-1-during-outage.png` | Grafana — `up{job="keycloak"}` 이 전면 장애 내내 **1에 평평** |
|
||||
|
||||
## 핵심 세 줄
|
||||
|
||||
1. **DB 는 단일 장애점이다.** Keycloak 을 몇 대로 늘려도 같이 죽는다 — Ready 파드 0개, 외부 503.
|
||||
2. **캐시는 읽기를 대신할 뿐 쓰기를 못 한다.** refresh 는 `UPDATE LAST_SESSION_REFRESH` 를 하므로 캐시가 있어도 실패한다.
|
||||
3. **`up` 은 이 장애를 못 잡는다.** 알림은 readiness 와 외부 응답 코드에 걸어야 한다.
|
||||
BIN
Binary file not shown.
|
After Width: | Height: | Size: 60 KiB |
+19
@@ -0,0 +1,19 @@
|
||||
=== [준비] 손실 측정 설계 확인 ===
|
||||
LAST_SESSION_REFRESH 는 integer(초) — 200ms 손실은 보이지 않는다
|
||||
created_on | integer | | not null |
|
||||
last_session_refresh | integer | | not null | 0
|
||||
"idx_user_session_expiration_created" btree (realm_id, offline_flag, remember_me, created_on, user_session_id, user_id)
|
||||
"idx_user_session_expiration_last_refresh" btree (realm_id, offline_flag, remember_me, last_session_refresh, user_session_id, user_id)
|
||||
→ 대신 행 존재 여부로 잰다. 로그인 하나 = 행 하나 = 이진 판정
|
||||
|
||||
전역 synchronous_commit: on
|
||||
|
||||
=== [1] 빠른 연속 로그인을 백그라운드로 시작 ===
|
||||
루프 시작
|
||||
6초 경과 — 지금까지 성공한 로그인: 0
|
||||
|
||||
=== [2] PostgreSQL 강제 종료 (SIGKILL) ===
|
||||
종료 시각: 12:00:26.511
|
||||
pod "postgres-7b474b88c8-xc2vt" force deleted from keycloak-lab namespace
|
||||
삭제 반환: 12:00:26.586
|
||||
클라이언트가 200 을 받은 로그인 수: 0
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
deployment "postgres" successfully rolled out
|
||||
|
||||
=== crash recovery 가 실행되었는가 (강제 종료의 흔적) ===
|
||||
2026-09-04 02:58:41.036 UTC [1] LOG: database system is ready to accept connections
|
||||
|
||||
=== [설계 확인] 로그인 트랜잭션도 synchronous_commit 을 끄는가 ===
|
||||
--- 로그인 트랜잭션 (INSERT 가 있는 것) ---
|
||||
2:BEGIN
|
||||
5:COMMIT
|
||||
6:BEGIN
|
||||
9:insert into OFFLINE_USER_SESSION (BROKER_SESSION_ID,CREATED_ON,DATA,LAST_SESSION_REFRESH,REALM_ID,REMEMBER_ME,USER_ID,VERSION,OFFLINE_FLAG,USER_SESSION_ID) values ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10)
|
||||
10:insert into OFFLINE_CLIENT_SESSION (DATA,REALM_ID,TIMESTAMP,VERSION,CLIENT_ID,CLIENT_STORAGE_PROVIDER,EXTERNAL_CLIENT_ID,OFFLINE_FLAG,USER_SESSION_ID) values ($1,$2,$3,$4,$5,$6,$7,$8,$9)
|
||||
11:SET LOCAL synchronous_commit TO OFF
|
||||
12:COMMIT
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
=== [1] 로그인 루프 시작 (호스트에서 백그라운드로 exec — 세션이 살아 있어야 한다) ===
|
||||
8초 동안 클라이언트가 200 을 받은 로그인: 106 건
|
||||
|
||||
=== [2] SIGKILL ===
|
||||
종료: 12:01:32.981
|
||||
반환: 12:01:33.236
|
||||
최종 성공 로그인 수: 110 건
|
||||
마지막 sid: FimM-krSybBACP2qIvshLWwU
|
||||
마지막 sid: EwFfFwOIfqiv8N5GQ5OjtsVq
|
||||
마지막 sid: CJX-PxFkS7rUc_9FQgB7iw1f
|
||||
마지막 sid: 1EFK7SgUA4M7tq_SkC_BD2er
|
||||
마지막 sid: _LiqTczuyxlpOs3T3xs25SLv
|
||||
|
||||
=== [3] PostgreSQL 재기동 후 crash recovery 확인 ===
|
||||
Waiting for deployment "postgres" rollout to finish: 0 of 1 updated replicas are available...
|
||||
deployment "postgres" successfully rolled out
|
||||
2026-09-04 02:59:48.427 UTC [1] LOG: database system is ready to accept connections
|
||||
@@ -0,0 +1,27 @@
|
||||
=== [4] 클라이언트가 받은 sid 가 DB 에 있는가 ===
|
||||
클라이언트가 200 을 받은 sid: 291 건
|
||||
DB 온라인 세션 총계: 375
|
||||
|
||||
--- 마지막 15건을 하나씩 조회 ---
|
||||
TCCOYnVlN30Y2fGyJEsVJ_Lq 있음
|
||||
vBcllIkKWovh-FXN9tSzmxAe 있음
|
||||
eMpN_ywUdRTks9uBE9aTumPK 있음
|
||||
aPC_T0yrlMjrlskcLAp0AvY6 있음
|
||||
UBPxmduB-ahGHBg636sY3AEz 있음
|
||||
HLnBloNX9R3qQJSkpOnkNqL4 있음
|
||||
_KPJS30IAqVhkTJGHcCxKvrM 있음
|
||||
rq3caZ9MkyMYlFSSzRjQLygD 있음
|
||||
ivvQm70hjl55DPpF7vpYmz_E 있음
|
||||
81mx-rmi-tAeogHx3-su3z2q 있음
|
||||
WnvNDH93uzcz1XNFbaMSXk1A 있음
|
||||
DXPAIhjO5sGpCUlcD8IEoS8L 있음
|
||||
aZMvl4IwPdK-rC_7bG005Z5C 있음
|
||||
eIuBCprfWA5x0glcgSKYrrX0 있음
|
||||
ozES5kEeu2IFf_cfcC_jFlbF 있음
|
||||
|
||||
마지막 15건 중 유실: 0 건
|
||||
|
||||
=== [5] 전체 대조 — 몇 건이나 사라졌는가 ===
|
||||
클라이언트 성공: 291 건
|
||||
DB 에 존재: 291 건
|
||||
★ 유실: 0 건
|
||||
@@ -0,0 +1,12 @@
|
||||
=== [정리] 세션 테이블 비우고 루프 잔여 확인 ===
|
||||
DELETE 375
|
||||
남은 세션: 0
|
||||
|
||||
=== [재주입] postmaster(PID 1)에 SIGKILL — 진짜 크래시 ===
|
||||
8초 후 성공 로그인: 110 건
|
||||
SIGKILL: 12:03:21.441
|
||||
최종 성공 로그인: 139 건
|
||||
|
||||
=== [검증] 이번엔 crash recovery 가 돌았는가 ===
|
||||
deployment "postgres" successfully rolled out
|
||||
2026-09-04 02:59:48.427 UTC [1] LOG: database system is ready to accept connections
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
=== 로그인 루프 시작 ===
|
||||
8초 후: 112 건
|
||||
|
||||
=== 백엔드 프로세스에 SIGKILL → postmaster 가 재초기화한다 ===
|
||||
시각: 12:04:22.063
|
||||
최종 성공 로그인: 153 건
|
||||
|
||||
=== [검증] crash recovery 가 돌았는가 ===
|
||||
2026-09-04 02:59:48.427 UTC [1] LOG: database system is ready to accept connections
|
||||
2026-09-04 03:02:35.807 UTC [1] LOG: server process (PID 40) was terminated by signal 9: Killed
|
||||
2026-09-04 03:02:35.807 UTC [1] LOG: terminating any other active server processes
|
||||
2026-09-04 03:02:35.814 UTC [1] LOG: all server processes terminated; reinitializing
|
||||
2026-09-04 03:02:35.896 UTC [2585] LOG: database system was not properly shut down; automatic recovery in progress
|
||||
2026-09-04 03:02:35.899 UTC [2585] LOG: redo starts at 0/23CAB68
|
||||
2026-09-04 03:02:35.904 UTC [2585] LOG: redo done at 0/2529E40 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
|
||||
2026-09-04 03:02:35.923 UTC [2586] LOG: checkpoint complete: wrote 113 buffers (0.7%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.004 s, sync=0.004 s, total=0.015 s; sync files=27, longest=0.003 s, average=0.001 s; distance=1405 kB, estimate=1405 kB; lsn=0/252A048, redo lsn=0/252A048
|
||||
2026-09-04 03:02:35.926 UTC [1] LOG: database system is ready to accept connections
|
||||
@@ -0,0 +1,19 @@
|
||||
=== 크래시 전후 대조 ===
|
||||
클라이언트가 200 과 토큰을 받은 로그인 : 153 건
|
||||
그중 DB 에 실제로 존재 : 149 건
|
||||
★ 유실 : 4 건
|
||||
DB 전체 온라인 세션 : 150 건
|
||||
|
||||
=== 유실된 sid 목록 ===
|
||||
★ CQUfg9HLH29xvhiu6pVlfWOo ← 토큰은 발급됐는데 세션이 없다
|
||||
★ 5gLP4fqmpZBbjhH_d-0TPMMr ← 토큰은 발급됐는데 세션이 없다
|
||||
★ hkcOv1QskUFmYveMLB6Hljra ← 토큰은 발급됐는데 세션이 없다
|
||||
★ p5XybeQIYmAs818gO4Vl_5ea ← 토큰은 발급됐는데 세션이 없다
|
||||
|
||||
=== 그 토큰이 지금 실제로 쓰이는가 (마지막 sid 로 확인) ===
|
||||
마지막 sid: 8do0Bw6tkVLDVxgxotE7GosH
|
||||
user_session_id | created_on | last_session_refresh
|
||||
--------------------------+------------+----------------------
|
||||
8do0Bw6tkVLDVxgxotE7GosH | 1788490958 | 1788490958
|
||||
(1 row)
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
=== A-3 이 가정만 하고 재지 않은 값 ===
|
||||
name | setting | unit | source
|
||||
------------------------+---------+------+---------
|
||||
commit_delay | 0 | | default
|
||||
synchronous_commit | on | | default
|
||||
wal_writer_delay | 200 | ms | default
|
||||
wal_writer_flush_after | 128 | 8kB | default
|
||||
(4 rows)
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
# A-3 — DB 강제 종료와 데이터 손실 증거
|
||||
|
||||
2026-09-04 12:00–12:05 KST · Keycloak 26.7.0 / PostgreSQL 16
|
||||
해설: [`docs/experiment-a3-database-crash.md`](../../experiment-a3-database-crash.md)
|
||||
|
||||
| 파일 | 무엇을 보여주는가 |
|
||||
|---|---|
|
||||
| `01-crash-injection.txt` | 첫 시도 실패 — 파드 안 백그라운드 루프가 `exec` 종료와 함께 죽어 0건 수집 |
|
||||
| `02-design-check.txt` | **핵심 설계 확인** — 로그인 트랜잭션도 `SET LOCAL synchronous_commit TO OFF` 로 커밋한다 |
|
||||
| `03-loss-measurement.txt` | `--grace-period=0 --force` 주입 |
|
||||
| `04-comparison.txt` | **유실 0건** — 그러나 crash recovery 가 안 돌았다. 죽인 적이 없는 것 |
|
||||
| `05-true-crash.txt` | `kill -9 1` 시도 — **컨테이너 안에서 PID 1 은 SIGKILL 을 무시한다** |
|
||||
| `06-backend-kill-crash.txt` | **성공한 주입** — 백엔드에 SIGKILL → `not properly shut down` / `redo starts` / `redo done` |
|
||||
| `07-loss-result.txt` | **결과: 153건 중 4건 유실.** 토큰은 발급됐는데 세션 행이 없는 sid 목록 |
|
||||
|
||||
## 핵심 세 줄
|
||||
|
||||
1. **로그인도 비동기 커밋이다.** refresh 시각뿐 아니라 **로그인 자체**가 사라질 수 있다.
|
||||
2. **153건 중 4건(약 2.6%) 유실** — 초당 19건 기준 마지막 0.2초 분량, `wal_writer_delay` 기본값과 일치.
|
||||
3. **주입을 세 번 시도해 세 번째에 성공했다.** 앞의 둘은 "손실 0"으로 보였지만 실제로는 크래시가 아니었다.
|
||||
@@ -0,0 +1,20 @@
|
||||
=== A-4 기준선 ===
|
||||
kc-lab-1 Ready true
|
||||
kc-lab-2 Ready <none>
|
||||
|
||||
a2-probe true kc-lab-2
|
||||
keycloak-0 true kc-lab-2
|
||||
keycloak-1 true kc-lab-1
|
||||
postgres-7b474b88c8-2gf27 true kc-lab-2
|
||||
|
||||
=== PVC 가 어느 노드에 묶여 있는가 (재배치 가능성) ===
|
||||
persistentvolumeclaim/postgres-data → kc-lab-2
|
||||
|
||||
=== 서비스 정상 확인 ===
|
||||
https://auth.hyeonworks.com/realms/master HTTP 200
|
||||
|
||||
=== VM 상태 ===
|
||||
--------------------------
|
||||
1 kc-lab-1 running
|
||||
2 kc-lab-2 running
|
||||
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
=== 워커 노드(kc-lab-2) 전원 차단 — virsh destroy 는 종료 신호가 없다 ===
|
||||
차단 시각: 12:07:43
|
||||
Domain 'kc-lab-2' destroyed
|
||||
|
||||
|
||||
+15초 node=Ready | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 000
|
||||
+30초 node=Ready | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 000
|
||||
+45초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+60초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+75초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+90초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+105초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+120초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+135초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+150초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+165초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+180초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
=== 파드 상태의 진실 — Running 인데 노드가 없다 ===
|
||||
a2-probe Running true kc-lab-2 <none>
|
||||
keycloak-0 Running true kc-lab-2 <none>
|
||||
keycloak-1 Running false kc-lab-1 <none>
|
||||
postgres-7b474b88c8-2gf27 Running true kc-lab-2 <none>
|
||||
|
||||
=== 재배치가 시도되었는가 ===
|
||||
10m Warning Unhealthy pod/keycloak-0 Readiness probe failed: Get "http://10.42.1.67:9000/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
|
||||
3m15s Warning NodeNotReady pod/postgres-7b474b88c8-2gf27 Node is not ready
|
||||
3m15s Warning NodeNotReady pod/keycloak-0 Node is not ready
|
||||
3m15s Warning NodeNotReady pod/a2-probe Node is not ready
|
||||
2m27s Warning Unhealthy pod/keycloak-1 Readiness probe failed: Get "http://10.42.0.35:9000/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
|
||||
2s Warning Unhealthy pod/keycloak-1 Readiness probe failed: HTTP probe failed with statuscode: 503
|
||||
|
||||
=== 노드 taint — 쿠버네티스가 붙인 것 ===
|
||||
node.kubernetes.io/unreachable=:NoSchedule
|
||||
node.kubernetes.io/unreachable=:NoExecute
|
||||
|
||||
=== Prometheus 가 본 것 (kc-lab-1 에 있어 살아남았다) ===
|
||||
up{job=keycloak pod=keycloak-1 } = 1
|
||||
up{job=keycloak pod=keycloak-0 } = 0
|
||||
up{job=kubelet pod=- } = 1
|
||||
up{job=kubelet pod=- } = 0
|
||||
up{job=node-exporter pod=kc-lab-1 } = 1
|
||||
up{job=node-exporter pod=kc-lab-2 } = 0
|
||||
up{job=prometheus pod=- } = 1
|
||||
|
||||
=== 진입점이 처음 40초간 000 이었던 이유 — nginx upstream ===
|
||||
@@ -0,0 +1,15 @@
|
||||
=== nginx 설정 위치 찾기 ===
|
||||
|
||||
=== NoExecute taint 의 tolerationSeconds — 언제 축출되는가 ===
|
||||
node.kubernetes.io/not-ready NoExecute tolerationSeconds=300
|
||||
node.kubernetes.io/unreachable NoExecute tolerationSeconds=300
|
||||
|
||||
=== 5분 축출 시점까지 관찰 ===
|
||||
+210초 a2-probe:Running keycloak-0:Running keycloak-1:Running postgres-7b474b88c8-2gf27:Running
|
||||
+240초 a2-probe:Running keycloak-0:Running keycloak-1:Running postgres-7b474b88c8-2gf27:Running
|
||||
+270초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending
|
||||
+300초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending
|
||||
+330초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending
|
||||
+360초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending
|
||||
+390초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending
|
||||
+420초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending
|
||||
@@ -0,0 +1,18 @@
|
||||
=== 새 postgres 가 Pending 인 이유 ===
|
||||
Events:
|
||||
Type Reason Age From Message
|
||||
---- ------ ---- ---- -------
|
||||
Warning FailedScheduling 4m45s default-scheduler 0/2 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
|
||||
|
||||
=== keycloak-0 대체 파드가 안 생기는 이유 (StatefulSet) ===
|
||||
keycloak 2 <none> 1
|
||||
keycloak-0 1/1 Terminating 0 30m
|
||||
keycloak-1 0/1 Running 0 143m
|
||||
|
||||
=== 복구 — 노드 재기동 ===
|
||||
재기동 시각: 12:16:31
|
||||
Domain 'kc-lab-2' started
|
||||
|
||||
+30초 node=Ready | Running 파드 3 개 | 외부 HTTP 503
|
||||
+60초 node=Ready | Running 파드 3 개 | 외부 HTTP 200
|
||||
→ 서비스 복귀
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
=== 복구 확인 ===
|
||||
keycloak-0 1/1 Running 0 68s
|
||||
keycloak-1 1/1 Running 0 144m
|
||||
postgres-7b474b88c8-9cmsv 1/1 Running 0 4m20s
|
||||
|
||||
=== kc-lab-1(k3s server)에 무엇이 있는가 — 이게 곧 영향 범위다 ===
|
||||
keycloak-lab keycloak-1
|
||||
kube-system coredns-54996dc9b4-8k8fj
|
||||
kube-system helm-install-traefik-crd-q29b5
|
||||
kube-system local-path-provisioner-77b9867795-g27z8
|
||||
kube-system metrics-server-6dc596dfb8-7xxq4
|
||||
kube-system svclb-traefik-5eb6a9a1-qwwk5
|
||||
kube-system traefik-5d6fcf895-wpfhr
|
||||
observability grafana-845b5678cf-b6gvc
|
||||
observability node-exporter-9qk9w
|
||||
observability prometheus-6774f94f7c-pzr2t
|
||||
|
||||
=== Traefik replica 수 (진입점의 단일 장애점인가) ===
|
||||
traefik 1 1
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
=== 컨트롤 플레인 노드(kc-lab-1) 전원 차단 ===
|
||||
차단 시각: 12:18:08
|
||||
Domain 'kc-lab-1' destroyed
|
||||
|
||||
|
||||
+20초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
+40초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
+60초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
+80초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
+100초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
+120초 외부 auth=000 grafana=502 | kubectl: Unable to connect to the server: dial tcp
|
||||
+140초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
+160초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
|
||||
=== 살아 있는 노드에서 직접 확인 — 워크로드는 도는가 ===
|
||||
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
|
||||
e5f777900b762 60e153026e8f5 4 minutes ago Running keycloak 0 640d4dafaefb3 keycloak-0 keycloak-lab
|
||||
|
||||
6
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
=== 컨트롤 플레인 노드 복구 ===
|
||||
재기동: 12:23:39
|
||||
Domain 'kc-lab-1' started
|
||||
|
||||
+30초 외부=502 | kc-lab-1=Ready kc-lab-2=Ready
|
||||
+60초 외부=200 | kc-lab-1=Ready kc-lab-2=Ready
|
||||
→ 서비스 복귀 (총 60초)
|
||||
|
||||
=== 최종 상태 ===
|
||||
keycloak-0 1/1 Running 0 7m57s
|
||||
keycloak-1 1/1 Running 1 (<invalid> ago) 151m
|
||||
postgres-7b474b88c8-9cmsv 1/1 Running 0 11m
|
||||
(prometheus port-forward 재연결 필요)
|
||||
@@ -0,0 +1,23 @@
|
||||
# A-4 — 노드 전원 차단 증거
|
||||
|
||||
2026-09-04 12:07–12:24 KST
|
||||
해설: [`docs/experiment-a4-node-loss.md`](../../experiment-a4-node-loss.md)
|
||||
|
||||
| 파일 | 무엇을 보여주는가 |
|
||||
|---|---|
|
||||
| `01-baseline.txt` | 차단 전 — 양쪽 Ready, **PVC 가 kc-lab-2 에 못박혀 있음**(재배치 불가의 원인), 외부 200 |
|
||||
| `02-worker-node-killed.txt` | `virsh destroy kc-lab-2` — **40초간 노드가 Ready 로 남아 있고** 외부는 이미 `000`. 이후 `503` |
|
||||
| `03-state-during-loss.txt` | **죽은 파드가 `ready=true`, 산 파드가 `ready=false`.** `up` 은 정확히 0. `unreachable` taint |
|
||||
| `04-eviction-timing.txt` | `tolerationSeconds=300` — **5분 뒤** 축출, 새 postgres 는 `Pending` |
|
||||
| `05-recovery.txt` | `FailedScheduling: didn't match PersistentVolume's node affinity`, StatefulSet `DESIRED=2 CURRENT=1`. 노드 복귀 후 **60초** |
|
||||
| `06-control-plane-inventory.txt` | kc-lab-1 에 있는 것 목록 — **Traefik `replicas=1`** |
|
||||
| `07-control-plane-loss.txt` | `virsh destroy kc-lab-1` — 외부 `000`, `kubectl` 불통. **그런데 `crictl ps` 로 보면 keycloak-0 은 Running** |
|
||||
| `08-control-plane-recovery.txt` | 60초 만에 복귀 |
|
||||
| `a4-up-dropped-per-node.png` | Grafana — `up` 이 노드별로 0 으로 떨어지는 구간. 12:18–12:23 은 **0 이 아니라 데이터 없음**(관측자가 같이 죽음) |
|
||||
|
||||
## 핵심 네 줄
|
||||
|
||||
1. **쿠버네티스는 40초 동안 노드가 살아 있다고 말한다.** 사용자는 이미 장애를 겪는 중이다.
|
||||
2. **죽은 파드의 상태는 화석이다.** `ready=true` 인 파드가 꺼진 기계 위에 있다.
|
||||
3. **StatefulSet 은 대체 파드를 만들지 않고, PVC 는 재배치를 막는다.** 사람이 개입해야 한다.
|
||||
4. **컨트롤 플레인 상실 ≠ 워크로드 상실.** 컨테이너는 계속 돌고, 들어갈 문만 사라진다.
|
||||
BIN
Binary file not shown.
|
After Width: | Height: | Size: 79 KiB |
+42
@@ -0,0 +1,42 @@
|
||||
=== A-5 기준선 ===
|
||||
keycloak-0=10.42.1.77 (kc-lab-2) keycloak-1=10.42.0.42 (kc-lab-1)
|
||||
keycloak-0 1/1 Running 0 11m
|
||||
keycloak-1 1/1 Running 1 (2m48s ago) 155m
|
||||
postgres-7b474b88c8-9cmsv 1/1 Running 0 14m
|
||||
Traceback (most recent call last):
|
||||
File "<string>", line 3, in <module>
|
||||
for r in json.load(sys.stdin)["data"]["result"]: print(f" cluster_size {r["metric"].get("pod"):12} = {r["value"][1]}")
|
||||
~~~~~~~~~^^^^^^^^^^^
|
||||
File "/usr/lib/python3.14/json/__init__.py", line 298, in load
|
||||
return loads(fp.read(),
|
||||
cls=cls, object_hook=object_hook,
|
||||
parse_float=parse_float, parse_int=parse_int,
|
||||
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
|
||||
File "/usr/lib/python3.14/json/__init__.py", line 352, in loads
|
||||
return _default_decoder.decode(s)
|
||||
~~~~~~~~~~~~~~~~~~~~~~~^^^
|
||||
File "/usr/lib/python3.14/json/decoder.py", line 345, in decode
|
||||
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
|
||||
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/lib/python3.14/json/decoder.py", line 363, in raw_decode
|
||||
raise JSONDecodeError("Expecting value", s, err.value) from None
|
||||
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
|
||||
|
||||
=== 주입: keycloak-0 으로 들어가는 7800/57800 만 DROP (한 방향) ===
|
||||
A-1 의 NetworkPolicy 는 conntrack ESTABLISHED 에 막혀 기존 연결을 못 끊었다.
|
||||
FORWARD 최상단에 넣으면 conntrack 승인보다 먼저 평가된다.
|
||||
Chain FORWARD (policy ACCEPT)
|
||||
num target prot opt source destination
|
||||
1 DROP 6 -- 0.0.0.0/0 10.42.1.77 tcp dpt:57800
|
||||
2 DROP 6 -- 0.0.0.0/0 10.42.1.77 tcp dpt:7800
|
||||
주입 시각: 12:28:23
|
||||
|
||||
=== 관찰 — 어느 쪽이 먼저 상대를 의심하는가 ===
|
||||
+25초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+50초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+75초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+100초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+125초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+150초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+175초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+200초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
# 주의 — 이 파일은 원 실험 시점에 0바이트로 저장됐다.
|
||||
# 리다이렉션이 stdout 만 받았는데 출력이 stderr 로 갔거나 tee 앞 파이프가
|
||||
# 비어 있었던 것으로 보인다. README 는 그 사이 파일 내용을 서술하고 있었는데,
|
||||
# 그것은 화면에서 본 것을 적은 것이지 이 파일에서 온 것이 아니었다.
|
||||
#
|
||||
# 아래는 사후에 다시 수집한 것이며, 원 시점의 DROP 규칙(0 패킷)은 이미
|
||||
# 제거되어 재현되지 않는다. 구조적 사실(kube-router 가 자기 체인을 FORWARD
|
||||
# 최상단에 유지한다)만 확인할 수 있다.
|
||||
# 원 실험의 결정적 증거는 04-correct-direction.txt 의 패킷 카운터 19/21 이다.
|
||||
|
||||
=== A-5 재수집 — filter 테이블 규칙이 CNI 체인에 밀리는 것 ===
|
||||
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
|
||||
num pkts bytes target prot opt in out source destination
|
||||
1 1690 3386K KUBE-ROUTER-FORWARD 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-router netpol - TEMCG2JMHZYE7H7T */
|
||||
2 7 612 KUBE-PROXY-FIREWALL 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes load balancer firewall */
|
||||
3 40 13196 KUBE-FORWARD 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */
|
||||
|
||||
(원 실험 시점의 규칙은 이미 제거됐다. 아래는 kube-router 가 자기 체인을
|
||||
FORWARD 최상단에 유지한다는 구조적 사실만 보여준다 — 그것이 실패 원인이었다.)
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
=== 실패한 규칙 제거 ===
|
||||
제거완료
|
||||
|
||||
=== raw 테이블 PREROUTING 에 넣는다 — conntrack 보다 먼저 실행된다 ===
|
||||
netfilter 순서: raw PREROUTING → conntrack → mangle → nat → filter
|
||||
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
|
||||
num pkts bytes target prot opt in out source destination
|
||||
1 0 0 DROP 6 -- * * 0.0.0.0/0 10.42.1.77 tcp dpt:57800
|
||||
2 0 0 DROP 6 -- * * 0.0.0.0/0 10.42.1.77 tcp dpt:7800
|
||||
주입: 12:32:35
|
||||
|
||||
=== [검증] 이번엔 패킷이 걸렸는가 ===
|
||||
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
|
||||
pkts bytes target prot opt in out source destination
|
||||
0 0 DROP 6 -- * * 0.0.0.0/0 10.42.1.77 tcp dpt:57800
|
||||
0 0 DROP 6 -- * * 0.0.0.0/0 10.42.1.77 tcp dpt:7800
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
=== keycloak-1(수신측)으로 들어가는 7800/57800 만 DROP — kc-lab-1 에 넣는다 ===
|
||||
주입: 12:33:58
|
||||
|
||||
=== [검증] 패킷이 걸리는가 ===
|
||||
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
|
||||
pkts bytes target prot opt in out source destination
|
||||
0 0 DROP 6 -- * * 0.0.0.0/0 10.42.0.42 tcp dpt:57800
|
||||
19 2938 DROP 6 -- * * 0.0.0.0/0 10.42.0.42 tcp dpt:7800
|
||||
|
||||
=== 관찰 ===
|
||||
+25초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+50초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+75초 - | keycloak-0:1/1 keycloak-1:0/1 | 외부 200
|
||||
+100초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+125초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+150초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+175초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+200초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
=== 지금 연결 방향은? (차단은 → 10.42.0.42:7800 만) ===
|
||||
tcp 6 299 ESTABLISHED src=10.42.0.42 dst=10.42.1.77 sport=48473 dport=7800 src=10.42.1.77 dst=10.42.0.42 sport=7800 dport=48473
|
||||
tcp 6 86232 ESTABLISHED src=10.42.0.42 dst=10.42.1.77 sport=44205 dport=57800 src=10.42.1.77 dst=10.42.0.42 sport=57800 dport=44205
|
||||
|
||||
=== 차단 규칙 누적 카운터 ===
|
||||
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
|
||||
pkts bytes target prot opt in out source destination
|
||||
19 1096 DROP 6 -- * * 0.0.0.0/0 10.42.0.42 tcp dpt:57800
|
||||
21 3058 DROP 6 -- * * 0.0.0.0/0 10.42.0.42 tcp dpt:7800
|
||||
|
||||
=== 클러스터 멤버 (파드에서 직접) ===
|
||||
keycloak-0 cluster_size=2.0
|
||||
keycloak-1 cluster_size=2.0
|
||||
|
||||
=== 최근 뷰 로그 ===
|
||||
keycloak-0: [keycloak-0-24309(v=16.0.12)|12] (1) [keycloak-0-24309(v=16.0.12)]
|
||||
|
||||
keycloak-1: [keycloak-1-45480(v=16.0.12)|12] (1) [keycloak-1-45480(v=16.0.12)]
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user