docs: A-8 — a rolling restart keeps every session and drops only the cache

Nine samples through the restart all returned 200, the refresh token issued beforehand still works, and the session count is unchanged at 151 while both caches reset to zero. The updated last_session_refresh proves the write path recovered too, not just the response code.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
DongHyeonka
2026-09-04 13:20:56 +09:00
co-authored by Claude Opus 5
parent dba0c3975c
commit 114d21aebe
8 changed files with 302 additions and 0 deletions
@@ -10,3 +10,11 @@
[ 52154ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 [ 52154ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 67608ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 [ 67608ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 86965ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 [ 86965ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 108668ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 129158ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 137544ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 152699ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 159898ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 173491ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 187925ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 197824ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,11 @@
[ 635ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22vendor_statistics_approximate_entries_unique%7Bcache%3D%5C%22sessions%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%2C%7B%22refId%22%3A%22B%22%2C%22expr%22%3A%22vendor_cluster_size%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22cluster_size%20%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-15m%22%2C%22to%22%3A%22now%22%7D%7D%7D:0
[ 686ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1451ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 3030ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 4308ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 5296ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 7691ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 18337ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 20184ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 32473ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 50697ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,56 @@
- generic [ref=f18e4]:
- link "Skip to main content" [ref=f18e5] [cursor=pointer]:
- /url: "#pageContent"
- banner [ref=f18e7]:
- generic [ref=f18e8]:
- link [ref=f18e10] [cursor=pointer]:
- /url: /
- img "Grafana" [ref=f18e11]
- generic [ref=f18e14]:
- button "Search or jump to..." [ref=f18e18] [cursor=pointer]
- generic [ref=f18e19]: ctrl+k
- generic [ref=f18e23]:
- button "New" [ref=f18e24] [cursor=pointer]
- button "Help" [ref=f18e30] [cursor=pointer]
- button "News" [ref=f18e33] [cursor=pointer]
- button "Profile" [ref=f18e36] [cursor=pointer]:
- img "User avatar" [ref=f18e37]
- generic [ref=f18e38]:
- button "Open menu" [ref=f18e40] [cursor=pointer]
- navigation "Breadcrumbs" [ref=f18e43]:
- list [ref=f18e44]:
- listitem [ref=f18e45]:
- link "Home" [ref=f18e46] [cursor=pointer]:
- /url: /
- listitem [ref=f18e50]:
- link "Explore" [ref=f18e51] [cursor=pointer]:
- /url: /explore
- listitem [ref=f18e55]:
- generic "Prometheus" [ref=f18e56]
- generic [ref=f18e57]:
- button "Show more items" [ref=f18e60] [cursor=pointer]
- button "Toggle top search bar" [ref=f18e64] [cursor=pointer]
- main [ref=f18e70]:
- generic [ref=f18e72]:
- heading "Explore" [level=1] [ref=f18e73]
- generic [ref=f18e78]:
- navigation "Explore toolbar" [ref=f18e80]:
- navigation "Search links" [ref=f18e82]:
- generic [ref=f18e83]:
- button "Content outline" [expanded] [ref=f18e85] [cursor=pointer]:
- generic [ref=f18e88]: Outline
- generic [ref=f18e93] [cursor=pointer]:
- img "Prometheus logo" [ref=f18e95]
- textbox "Select a data source" [ref=f18e96]:
- /placeholder: ""
- button "Show more items" [ref=f18e102] [cursor=pointer]
- generic [ref=f18e106]:
- button "Collapse outline" [expanded] [ref=f18e112] [cursor=pointer]:
- img "arrow-from-right" [ref=f18e113]
- generic [ref=f18e120]:
- generic [ref=f18e123]:
- button "Add query" [ref=f18e124] [cursor=pointer]
- button "Query history" [ref=f18e128] [cursor=pointer]
- button "Query inspector" [ref=f18e132] [cursor=pointer]
- generic:
- main
@@ -0,0 +1,17 @@
=== [1] 재시작 전 로그인 — 토큰을 파드 안에 보관 ===
sid = XLcgQWRiJrTkuNZcJsNeT_2j
DB 세션 수: 151
=== [2] 롤링 재시작 중 가용성 — 5초 간격으로 외부 진입점 확인 ===
statefulset.apps/keycloak restarted
200 Waiting for partitioned roll out to finish: 0 out of 2 new pods have been updated...
Waiting for 1 pods to be ready...
Waiting for 1 pods to be ready...
Waiting for 1 pods to be ready...
200 200 200 200 Waiting for partitioned roll out to finish: 1 out of 2 new pods have been updated...
Waiting for 1 pods to be ready...
Waiting for 1 pods to be ready...
Waiting for 1 pods to be ready...
200 200 200 200 partitioned roll out complete: 2 new pods have been updated...
(위 숫자열이 재시작 중 외부 응답 코드의 시계열)
@@ -0,0 +1,19 @@
=== [3] 재시작 전 발급한 refresh token 이 아직 통하는가 ===
대상 sid: XLcgQWRiJrTkuNZcJsNeT_2j
keycloak-0 에서 refresh HTTP 200
=== [4] DB 에 그 세션이 남아 있는가 ===
user_session_id | created_on | last_session_refresh
--------------------------+------------+----------------------
XLcgQWRiJrTkuNZcJsNeT_2j | 1788495513 | 1788495577
(1 row)
전체 온라인 세션: 151 (재시작 전 151)
=== [5] 캐시는 어떻게 되었는가 ===
keycloak-0 sessions 캐시 0.0 건 / cluster_size 2.0
keycloak-1 sessions 캐시 1.0 건 / cluster_size 2.0
=== [6] 파드 나이 — 정말 재시작되었나 ===
keycloak-0 1/1 Running 0 44s
keycloak-1 1/1 Running 0 66s
@@ -0,0 +1,16 @@
# A-8 — 롤링 재시작 증거
2026-09-04 13:3813:41 KST
해설: [`docs/experiment-a8-rolling-restart.md`](../../experiment-a8-rolling-restart.md)
| 파일 | 무엇을 보여주는가 |
|---|---|
| `01-restart-availability.txt` | 재시작 전 로그인(sid 기록), 롤링 재시작 중 외부 진입점 **9회 모두 `200`** |
| `02-session-survival.txt` | 재시작 전 토큰으로 refresh **`200`**, DB 행 생존(`last_session_refresh` 갱신 확인), **세션 수 151 → 151**, 캐시 0으로 초기화, 파드 나이 44초/66초 |
| `a8-cache-reset-cluster-reformed.png` | Grafana — 세션 캐시가 0 으로 떨어지고 `cluster_size` 가 다시 2 가 되는 구간 |
## 핵심 세 줄
1. **무중단이었다.** 한 번에 하나씩 내리고 readiness 가 전환을 맞춰준다 — replica ≥ 2 가 전제.
2. **세션은 살아남고 캐시만 사라진다.** DB 151건 그대로, 재시작 전 토큰이 그대로 통한다.
3. **이것이 `persistent-user-sessions` 를 켜는 진짜 이유다.** A-7(volatile)에서 정반대가 나와야 한다.
Binary file not shown.

After

Width:  |  Height:  |  Size: 82 KiB

+175
View File
@@ -0,0 +1,175 @@
# A-8 — 배포할 때마다 로그아웃되는가
브랜치 `feature/keycloak-a8-rolling-restart` ·
증거 [`docs/evidence/a8-rolling-restart/`](evidence/a8-rolling-restart/) ·
2026-09-04 13:3813:41 KST
**운영에서 가장 자주 겪는 일이다.** 장애가 아니라 정상 작업인데도
사용자가 로그아웃되면 그건 사고다.
---
## 0. 결론부터
| 확인 | 결과 |
|---|---|
| 재시작 중 서비스 중단 | **없음.** 전 구간 `200` |
| 재시작 전 발급한 refresh token | **여전히 통한다** (`200`) |
| DB 세션 수 | **151 → 151** 그대로 |
| 세션 캐시 | **0 으로 초기화** |
| 클러스터 | 자동 재형성 (`cluster_size = 2`) |
**세션은 살아남고 캐시만 사라진다.** 이것이 `persistent-user-sessions`
켜는 진짜 이유다.
---
## 1. 방법
```bash
# 1. 재시작 전 로그인 — 토큰을 상주 파드 안에 보관한다
kubectl -n keycloak-lab run a8-probe --image=curlimages/curl:8.11.1 \
--restart=Never --command -- sleep 3600
kubectl -n keycloak-lab exec a8-probe -- sh -c '<로그인 후 /tmp/rt, /tmp/sid 에 저장>'
# 2. 재시작하면서 5초 간격으로 외부 진입점을 찍는다
kubectl -n keycloak-lab rollout restart statefulset/keycloak
( for i in $(seq 1 48); do
curl -s -o /dev/null -w '%{http_code} ' --max-time 4 https://auth.hyeonworks.com/realms/master
sleep 5
done ) &
kubectl -n keycloak-lab rollout status statefulset/keycloak --timeout=420s
```
**탐침 파드가 StatefulSet 밖에 있어야** 재시작을 넘어 토큰을 들고 있을 수 있다.
---
## 2. 가용성 — 무중단이었다
```
statefulset.apps/keycloak restarted
200 Waiting for partitioned roll out to finish: 0 out of 2 new pods have been updated...
Waiting for 1 pods to be ready...
200 200 200 200 Waiting for partitioned roll out to finish: 1 out of 2 new pods have been updated...
Waiting for 1 pods to be ready...
200 200 200 200 partitioned roll out complete: 2 new pods have been updated...
```
**9번 찍어서 9번 다 `200`.** 한 번도 끊기지 않았다.
### 왜 무중단이 되는가
```
StatefulSet 롤링 재시작
├─ keycloak-1 종료 → Service 엔드포인트에서 빠짐
│ └─ 이 동안 keycloak-0 이 전부 받는다
├─ keycloak-1 기동 → readiness UP → 엔드포인트 복귀
└─ keycloak-0 종료 → ... (반복)
```
**한 번에 하나씩** 내리므로 항상 최소 하나는 Ready 다.
readiness 프로브가 이 전환을 정확히 맞춰준다 — A-2 에서 본 그 메커니즘이
여기서는 **정상 작업을 안전하게** 만든다.
> 다만 이 실험대는 **파드가 2개**다. replica 1 이면 반드시 끊긴다.
> 무중단은 공짜가 아니라 **replica ≥ 2 와 readiness 의 조합**이다.
---
## 3. 세션 생존
```
=== 재시작 전 발급한 refresh token 이 아직 통하는가 ===
대상 sid: XLcgQWRiJrTkuNZcJsNeT_2j
keycloak-0 에서 refresh HTTP 200
=== DB 에 그 세션이 남아 있는가 ===
user_session_id | created_on | last_session_refresh
--------------------------+------------+----------------------
XLcgQWRiJrTkuNZcJsNeT_2j | 1788495513 | 1788495577
전체 온라인 세션: 151 (재시작 전 151)
```
**`last_session_refresh``created_on` 보다 64초 뒤**다. 재시작 후의 refresh
가 **실제로 DB 에 기록**되었다는 뜻이다 — 응답 코드만 200 인 게 아니라
쓰기까지 정상이다.
**파드가 통째로 바뀌었는데(44초/66초 나이) 세션은 그대로다.**
---
## 4. 캐시는 사라진다
```
keycloak-0 sessions 캐시 0.0 건 / cluster_size 2.0
keycloak-1 sessions 캐시 1.0 건 / cluster_size 2.0
```
![캐시 초기화와 클러스터 재형성](evidence/a8-rolling-restart/a8-cache-reset-cluster-reformed.png)
**캐시는 프로세스 메모리이므로 재시작에 사라진다.** `keycloak-1` 의 1건은
방금 refresh 를 처리하며 새로 담은 것이다.
```
재시작 전: 캐시 N건 + DB 151건
재시작 후: 캐시 0건 + DB 151건 ← 진실은 DB 에 있다
```
**A-0 의 모델이 그대로 확인된다.** 캐시가 통째로 날아가도 정확성은 유지되고
**첫 접근만 느려진다** (룩어사이드 캐시의 성질).
---
## 5. 이것이 `persistent-user-sessions` 를 켜는 진짜 이유다
| | persistent (KC 26 기본) | volatile (KC 24 이전 방식) |
|---|---|---|
| 롤링 재시작 후 | **세션 유지** | **전원 로그아웃** |
| 배포 빈도 | 자유롭다 | 배포가 곧 사고다 |
| 대가 | DB 쓰기 (A-6 에서 본 지연) | 없음 |
**A-7 에서 volatile 로 바꿔 같은 실험을 반복하면 여기가 정반대가 될 것이다.**
그 비교가 이 실험의 짝이다.
---
## 6. 재현 절차 (명령어)
```bash
# 상주 탐침 (StatefulSet 밖에 있어야 한다)
kubectl -n keycloak-lab run a8-probe --image=curlimages/curl:8.11.1 \
--restart=Never --command -- sleep 3600
kubectl -n keycloak-lab wait --for=condition=Ready pod/a8-probe --timeout=120s
# 로그인하고 토큰 보관
kubectl -n keycloak-lab exec a8-probe -- sh -c \
'curl -s -X POST http://<pod>:8080/realms/master/protocol/openid-connect/token \
-d grant_type=password -d client_id=admin-cli -d username=admin -d password=<pw> > /tmp/tok'
# 재시작 + 가용성 감시
kubectl -n keycloak-lab rollout restart statefulset/keycloak
kubectl -n keycloak-lab rollout status statefulset/keycloak --timeout=420s
# 세션 생존 확인
kubectl -n keycloak-lab exec a8-probe -- sh -c \
'curl -s -o /dev/null -w "%{http_code}\n" -X POST http://<pod>:8080/realms/master/protocol/openid-connect/token \
-d grant_type=refresh_token -d client_id=admin-cli -d refresh_token=$(cat /tmp/rt)'
# DB 대조
kubectl -n keycloak-lab exec deploy/postgres -- psql -U keycloak -d keycloak \
-c "select user_session_id, created_on, last_session_refresh from offline_user_session where user_session_id='<sid>'"
```
---
## 7. 다음 실험에 남기는 것
| 실험 | 이 실험이 준 것 |
|---|---|
| **A-7** volatile 비교 | **이 실험을 그대로 반복하면 정반대 결과가 나와야 한다** |
| **D-2** 버전 업그레이드 | 롤링 재시작이 안전하다는 것이 업그레이드의 전제 |
| 구성 | 무중단은 **replica ≥ 2 + readiness** 의 조합이다 |