docs: A-7 — three results invert when persistent sessions are turned off

Disabling persistent-user-sessions moves the session from PostgreSQL into the cluster, and the A-1 and A-8 outcomes flip to 400 Session not active while a new login during database loss starts working. The control group in each case still returns 200, so the injections cut only what they were meant to cut. This is the pair that makes the A layer legible: the conventional wisdom that sessions ride TCP 7800 is correct for Keycloak 24 and earlier, and the mistake is applying it to 26 without checking the version. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
docs: A-8 — a rolling restart keeps every session and drops only the cache
2026-09-04 13:33:33 +09:00 · 2026-09-04 13:20:56 +09:00 · 2026-09-04 13:18:10 +09:00 · 2026-09-04 13:12:04 +09:00 · 2026-09-04 12:25:30 +09:00 · 2026-09-04 12:05:08 +09:00
62 changed files with 3540 additions and 0 deletions
@@ -14,3 +14,116 @@
 [   75046ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   75890ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   96067ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  113467ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  121972ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  139681ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  150971ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  161211ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  180744ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  191401ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  210668ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  229082ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  239345ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  252119ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  266045ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  276705ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  289911ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  296875ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  315099ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  325431ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  328418ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  343774ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  347054ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  352405ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  361291ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  370639ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  374286ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  377413ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  394362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  405423ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  425476ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  434299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  441775ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  450786ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  456829ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  468812ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  478913ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  491241ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  495228ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  502295ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  511625ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  524619ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  533050ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  553189ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  571699ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  575413ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  588005ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  599161ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  608816ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  619031ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  626612ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  639003ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  655077ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  674532ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  677197ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  696027ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  709512ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  727505ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  747139ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  754565ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  773682ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  784674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  787399ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  802845ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  825589ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  832768ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  842043ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  852717ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  871748ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  876571ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  894491ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  895515ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  904310ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  916681ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  934752ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  941175ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  946004ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  960849ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  962180ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  984446ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  997579ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1010105ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1027190ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1043388ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1061311ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1066040ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1080349ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1092336ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1104312ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1109362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1127047ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1134728ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1154899ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1161253ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1174971ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1184491ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1185340ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1189111ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1196984ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1204794ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1247961ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1286322ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1319329ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1362771ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1396881ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1434184ms] [ERROR] Failed to load resource: the server responded with a status of 502 () @ https://app2.hyeonworks.com/api/user/auth-tokens/rotate:0
 [ 1443506ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1465016ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1513337ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1556906ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1562963ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1572896ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1576390ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 503 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1593794ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1600651ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1616424ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,125 @@
 [     268ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/login:0
 [     379ms] [VERBOSE] [DOM] Input elements should have autocomplete attributes (suggested: "username"): (More info: https://goo.gl/9p2vKq) %o @ https://app2.hyeonworks.com/login:0
 [   10453ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   10532ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&panes=%7B%22ich%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22up%7Bjob%3D%7E%5C%22keycloak%7Cnode-exporter%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bjob%7D%7D+%E2%80%94+%7B%7Bpod%7D%7D%7B%7Bnode%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-55m%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1:0
 [   11687ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   13737ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   15113ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   20185ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   27259ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   40154ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   41870ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   50493ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   54082ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   60121ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   74362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   83269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   95349ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   98008ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  104458ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  123816ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  142764ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  157198ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  175220ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  182187ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  183411ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  190581ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  199499ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  209995ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  214849ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  233084ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  242910ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  257975ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  277220ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  290806ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  292674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  300013ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  309267ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  323998ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  328620ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  339787ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  355959ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  361079ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  369272ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  381866ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  397120ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  400712ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  412579ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  431630ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  440746ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  447809ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  461741ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  475459ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  482723ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  494299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  513174ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  521541ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  532145ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  551129ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  556250ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  574381ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  586190ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  595782ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  610555ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  630695ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  633257ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  652194ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  671361ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  677606ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  692444ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  696949ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  700122ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  709954ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  723992ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  743173ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  745075ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  753299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  766683ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  767855ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  787261ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  789725ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  800680ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  811324ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  814096ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  829045ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  848506ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  857716ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  866212ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  873174ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  876033ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  895198ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  898884ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  918642ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  930133ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  934008ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  948541ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  956860ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  959397ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  968406ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  971788ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  982746ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  997394ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1005078ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1009269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1021453ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1026674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1042137ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1057497ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1059851ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1061855ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1071317ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1078179ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1084940ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1093950ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1112279ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1124360ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1139110ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1157981ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1163071ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1179032ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1181195ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1191955ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1201472ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1218261ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1221169ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1229430ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,50 @@
 [     199ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [     297ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22vendor_cluster_size%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-25m%22%2C%22to%22%3A%22now%22%7D%7D%7D:0
 [    1000ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [    2023ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [    3559ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [    7749ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   16364ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   25781ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   42158ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   59672ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   77090ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   87426ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   89952ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  104743ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error in connection establishment: net::ERR_NAME_NOT_RESOLVED @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1573121ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1584038ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1601626ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1613280ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1630173ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1634776ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1654205ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1661576ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1665269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1682669ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1689435ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1704385ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1706314ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1724551ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1743701ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1758339ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1769299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1789478ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1795415ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1815382ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1820392ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1822445ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1839035ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1840162ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1847739ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1861419ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1880381ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1893315ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1911434ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1922701ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1933868ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1944951ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1948001ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1952503ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1953320ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1955369ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,20 @@
 [     585ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [    1132ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&panes=%7B%22o68%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22agroal_blocking_time_max_milliseconds%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22blocking+max+-+%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-30m%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1:0
 [    1555ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [    3604ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [    6373ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [    9135ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   19066ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   24595ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   47125ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   52154ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   67608ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   86965ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  108668ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  129158ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  137544ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  152699ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  159898ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  173491ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  187925ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  197824ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,84 @@
 [     635ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22vendor_statistics_approximate_entries_unique%7Bcache%3D%5C%22sessions%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%2C%7B%22refId%22%3A%22B%22%2C%22expr%22%3A%22vendor_cluster_size%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22cluster_size%20%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-15m%22%2C%22to%22%3A%22now%22%7D%7D%7D:0
 [     686ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [    1451ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [    3030ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [    4308ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [    5296ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [    7691ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   18337ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   20184ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   32473ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   50697ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   65961ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   85590ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   99644ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  115524ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  130773ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  134354ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  142549ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  151361ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  155246ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  165901ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  169379ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  180952ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  190372ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  193657ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  205737ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  220582ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  225496ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  240339ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  244029ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  264211ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  273424ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  284893ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  292571ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  305778ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  306804ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  324378ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  335169ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  337968ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  352967ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  364258ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  379409ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  399590ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  403681ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  406340ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  421033ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  428667ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  429483ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  437573ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  442286ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  445865ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  456312ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  464043ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  480382ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  491438ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  494608ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  508231ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  528305ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  529632ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  538033ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  550079ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  566347ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  586157ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  603877ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  615779ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  624657ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  629879ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  637255ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  643298ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  650047ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  668080ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  669816ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  675422ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  677001ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  686611ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  692655ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  713114ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  723787ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  730443ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  751427ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  757583ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  776007ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  792298ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  803509ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,44 @@
 - main [ref=f6e7]:
  - generic [ref=f6e9]:
    - generic [ref=f6e11]:
      - generic [ref=f6e12]:
        - img "Grafana" [ref=f6e13]
        - heading "Welcome to Grafana" [level=1] [ref=f6e15]
      - generic [ref=f6e19]:
        - generic [ref=f6e20]:
          - generic [ref=f6e21]: Email or username
          - textbox "Email or username" [active] [ref=f6e28]:
            - /placeholder: email or username
        - generic [ref=f6e29]:
          - generic [ref=f6e30]: Password
          - generic [ref=f6e36]:
            - textbox "Password" [ref=f6e37]:
              - /placeholder: password
            - switch "Show password" [ref=f6e39] [cursor=pointer]
        - button "Log in" [ref=f6e42] [cursor=pointer]
        - link "Forgot your password?" [ref=f6e45] [cursor=pointer]:
          - /url: /user/password/send-reset-email
    - list [ref=f6e49]:
      - listitem [ref=f6e50]:
        - link "Documentation" [ref=f6e53] [cursor=pointer]:
          - /url: https://grafana.com/docs/grafana/latest/?utm_source=grafana_footer
        - text: "|"
      - listitem [ref=f6e54]:
        - link "Support" [ref=f6e57] [cursor=pointer]:
          - /url: https://grafana.com/products/enterprise/?utm_source=grafana_footer
        - text: "|"
      - listitem [ref=f6e58]:
        - link "Community" [ref=f6e61] [cursor=pointer]:
          - /url: https://community.grafana.com/?utm_source=grafana_footer
        - text: "|"
      - listitem [ref=f6e62]:
        - link "Open Source" [ref=f6e63] [cursor=pointer]:
          - /url: https://grafana.com/oss/grafana?utm_source=grafana_footer
        - text: "|"
      - listitem [ref=f6e64]:
        - link "Grafana v11.4.0 (b58701869e)" [ref=f6e65] [cursor=pointer]:
          - /url: https://github.com/grafana/grafana/blob/main/CHANGELOG.md
        - text: "|"
      - listitem [ref=f6e66]:
        - link "New version available!" [ref=f6e69] [cursor=pointer]:
          - /url: https://grafana.com/grafana/download?utm_source=grafana_footer
@@ -0,0 +1,159 @@
 - generic [active] [ref=f9e1]:
  - generic [ref=f9e4]:
    - link "Skip to main content" [ref=f9e5] [cursor=pointer]:
      - /url: "#pageContent"
    - banner [ref=f9e7]:
      - generic [ref=f9e8]:
        - link [ref=f9e10] [cursor=pointer]:
          - /url: /
          - img "Grafana" [ref=f9e11]
        - generic [ref=f9e14]:
          - button "Search or jump to..." [ref=f9e18] [cursor=pointer]
          - generic [ref=f9e19]: ctrl+k
        - generic [ref=f9e23]:
          - button "New" [ref=f9e24] [cursor=pointer]
          - button "Help" [ref=f9e30] [cursor=pointer]
          - button "News" [ref=f9e33] [cursor=pointer]
          - button "Profile" [ref=f9e36] [cursor=pointer]:
            - img "User avatar" [ref=f9e37]
      - generic [ref=f9e38]:
        - button "Open menu" [ref=f9e40] [cursor=pointer]
        - navigation "Breadcrumbs" [ref=f9e43]:
          - list [ref=f9e44]:
            - listitem [ref=f9e45]:
              - link "Home" [ref=f9e46] [cursor=pointer]:
                - /url: /
            - listitem [ref=f9e50]:
              - link "Explore" [ref=f9e51] [cursor=pointer]:
                - /url: /explore
            - listitem [ref=f9e55]:
              - generic "Prometheus" [ref=f9e56]
        - generic [ref=f9e57]:
          - generic [ref=f9e60]:
            - button "Copy shortened URL" [ref=f9e61] [cursor=pointer]
            - button "Open copy link options" [ref=f9e64] [cursor=pointer]
          - button "Toggle top search bar" [ref=f9e68] [cursor=pointer]
    - main [ref=f9e74]:
      - generic [ref=f9e76]:
        - heading "Explore" [level=1] [ref=f9e77]
        - generic [ref=f9e82]:
          - navigation "Explore toolbar" [ref=f9e84]:
            - navigation "Search links" [ref=f9e86]:
              - generic [ref=f9e87]:
                - button "Content outline" [expanded] [ref=f9e89] [cursor=pointer]:
                  - generic [ref=f9e92]: Outline
                - generic [ref=f9e97] [cursor=pointer]:
                  - img "Prometheus logo" [ref=f9e99]
                  - textbox "Select a data source" [ref=f9e100]:
                    - /placeholder: ""
            - generic [ref=f9e104]:
              - button "Split the pane" [ref=f9e106] [cursor=pointer]:
                - generic [ref=f9e109]: Split
              - button "Add" [ref=f9e111] [cursor=pointer]
              - generic [ref=f9e116]:
                - 'button "Time range selected: Last 55 minutes" [ref=f9e117] [cursor=pointer]'
                - button "Zoom out time range" [ref=f9e122] [cursor=pointer]
              - generic [ref=f9e126]:
                - button "Run query" [ref=f9e127] [cursor=pointer]
                - button "Auto refresh turned off. Choose refresh time interval" [ref=f9e131] [cursor=pointer]
          - generic [ref=f9e135]:
            - generic [ref=f9e139]:
              - button "Collapse outline" [expanded] [ref=f9e141] [cursor=pointer]:
                - img "arrow-from-right" [ref=f9e142]
              - button "Queries" [ref=f9e145] [cursor=pointer]:
                - img "arrow" [ref=f9e146]
              - button "Graph" [ref=f9e150] [cursor=pointer]:
                - img "graph-bar" [ref=f9e151]
            - generic [ref=f9e158]:
              - generic [ref=f9e160]:
                - generic "Query editor row" [ref=f9e163]:
                  - generic [ref=f9e164]:
                    - generic [ref=f9e166]:
                      - generic [ref=f9e167]:
                        - button "Collapse query row" [expanded] [ref=f9e168] [cursor=pointer]
                        - generic [ref=f9e171]:
                          - button "Query editor row title A" [ref=f9e172] [cursor=pointer]:
                            - generic [ref=f9e173]: A
                          - emphasis [ref=f9e174]: (Prometheus)
                      - generic [ref=f9e175]:
                        - button "Show data source help" [ref=f9e177] [cursor=pointer]
                        - button "Duplicate query" [ref=f9e181] [cursor=pointer]
                        - button "Hide response" [ref=f9e185] [cursor=pointer]
                        - button "Remove query" [ref=f9e189] [cursor=pointer]
                        - button "Drag and drop to reorder" [ref=f9e192]:
                          - img "Drag and drop to reorder" [ref=f9e193]
                    - generic [ref=f9e196]:
                      - generic [ref=f9e197]:
                        - button "Kick start your query" [ref=f9e198] [cursor=pointer]
                        - generic [ref=f9e201]:
                          - generic [ref=f9e202] [cursor=pointer]: Explain
                          - generic [ref=f9e203]:
                            - checkbox "Explain Toggle switch" [ref=f9e204]
                            - generic "Toggle switch" [ref=f9e205] [cursor=pointer]
                        - radiogroup [ref=f9e210]:
                          - generic [ref=f9e211]:
                            - radio "Builder" [ref=f9e212] [cursor=pointer]
                            - generic [ref=f9e213] [cursor=pointer]: Builder
                          - generic [ref=f9e214]:
                            - radio "Code" [checked] [ref=f9e215] [cursor=pointer]
                            - generic [ref=f9e216] [cursor=pointer]: Code
                      - generic [ref=f9e218]:
                        - generic [ref=f9e220]:
                          - button "Metrics browser" [ref=f9e221] [cursor=pointer]
                          - code [ref=f9e228]:
                            - generic [ref=f9e229]:
                              - generic [ref=f9e234]: "up{job=~\"keycloak|node-exporter\"}"
                              - textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=f9e239]: "up{job=~\"keycloak|node-exporter\"}"
                        - 'button "Options Legend: {{job}} — {{pod}}{{node}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f9e245] [cursor=pointer]':
                          - generic [ref=f9e249]:
                            - heading "Options" [level=6] [ref=f9e250]
                            - generic [ref=f9e251]:
                              - generic [ref=f9e252]: "Legend: {{job}} — {{pod}}{{node}}"
                              - generic [ref=f9e253]: "Format: Time series"
                              - generic [ref=f9e254]: "Step: auto"
                              - generic [ref=f9e255]: "Type: Range"
                              - generic [ref=f9e256]: "Exemplars: false"
                - generic [ref=f9e257]:
                  - button "Add query" [ref=f9e258] [cursor=pointer]
                  - button "Query history" [ref=f9e262] [cursor=pointer]
                  - button "Query inspector" [ref=f9e266] [cursor=pointer]
              - main [ref=f9e270]:
                - region [ref=f9e272]:
                  - generic [ref=f9e273]:
                    - heading "Graph" [level=2] [ref=f9e275]
                    - radiogroup [ref=f9e278]:
                      - generic [ref=f9e279]:
                        - radio "Lines" [checked] [ref=f9e280] [cursor=pointer]
                        - generic [ref=f9e281] [cursor=pointer]: Lines
                      - generic [ref=f9e282]:
                        - radio "Bars" [ref=f9e283] [cursor=pointer]
                        - generic [ref=f9e284] [cursor=pointer]: Bars
                      - generic [ref=f9e285]:
                        - radio "Points" [ref=f9e286] [cursor=pointer]
                        - generic [ref=f9e287] [cursor=pointer]: Points
                      - generic [ref=f9e288]:
                        - radio "Stacked lines" [ref=f9e289] [cursor=pointer]
                        - generic [ref=f9e290] [cursor=pointer]: Stacked lines
                      - generic [ref=f9e291]:
                        - radio "Stacked bars" [ref=f9e292] [cursor=pointer]
                        - generic [ref=f9e293] [cursor=pointer]: Stacked bars
                  - list [ref=f9e311]:
                    - listitem [ref=f9e312]:
                      - button "keycloak — keycloak-1kc-lab-1" [ref=f9e316] [cursor=pointer]
                    - listitem [ref=f9e317]:
                      - button "keycloak — keycloak-1kc-lab-1" [ref=f9e321] [cursor=pointer]
                    - listitem [ref=f9e322]:
                      - button "keycloak — keycloak-0kc-lab-2" [ref=f9e326] [cursor=pointer]
                    - listitem [ref=f9e327]:
                      - button "keycloak — keycloak-0kc-lab-2" [ref=f9e331] [cursor=pointer]
                    - listitem [ref=f9e332]:
                      - button "keycloak — keycloak-0kc-lab-2" [ref=f9e336] [cursor=pointer]
                    - listitem [ref=f9e337]:
                      - button "node-exporter — kc-lab-1" [ref=f9e341] [cursor=pointer]
                    - listitem [ref=f9e342]:
                      - button "node-exporter — kc-lab-2" [ref=f9e346] [cursor=pointer]
  - generic [ref=f9e351]:
    - alert
    - alert
    - complementary
    - complementary
@@ -0,0 +1,104 @@
 - generic [ref=f12e4]:
  - link "Skip to main content" [ref=f12e5] [cursor=pointer]:
    - /url: "#pageContent"
  - banner [ref=f12e7]:
    - generic [ref=f12e8]:
      - link [ref=f12e10] [cursor=pointer]:
        - /url: /
        - img "Grafana" [ref=f12e11]
      - generic [ref=f12e14]:
        - button "Search or jump to..." [ref=f12e18] [cursor=pointer]
        - generic [ref=f12e19]: ctrl+k
      - generic [ref=f12e23]:
        - button "New" [ref=f12e24] [cursor=pointer]
        - button "Help" [ref=f12e30] [cursor=pointer]
        - button "News" [ref=f12e33] [cursor=pointer]
        - button "Profile" [ref=f12e36] [cursor=pointer]:
          - img "User avatar" [ref=f12e37]
    - generic [ref=f12e38]:
      - button "Open menu" [ref=f12e40] [cursor=pointer]
      - navigation "Breadcrumbs" [ref=f12e43]:
        - list [ref=f12e44]:
          - listitem [ref=f12e45]:
            - link "Home" [ref=f12e46] [cursor=pointer]:
              - /url: /
          - listitem [ref=f12e50]:
            - link "Explore" [ref=f12e51] [cursor=pointer]:
              - /url: /explore
          - listitem [ref=f12e55]:
            - generic "Prometheus" [ref=f12e56]
      - generic [ref=f12e57]:
        - button "Show more items" [ref=f12e60] [cursor=pointer]
        - button "Toggle top search bar" [ref=f12e64] [cursor=pointer]
  - main [ref=f12e70]:
    - generic [ref=f12e72]:
      - heading "Explore" [level=1] [ref=f12e73]
      - generic [ref=f12e78]:
        - navigation "Explore toolbar" [ref=f12e80]:
          - navigation "Search links" [ref=f12e82]:
            - generic [ref=f12e83]:
              - button "Content outline" [expanded] [ref=f12e85] [cursor=pointer]:
                - generic [ref=f12e88]: Outline
              - generic [ref=f12e93] [cursor=pointer]:
                - img "Prometheus logo" [ref=f12e95]
                - textbox "Select a data source" [ref=f12e96]:
                  - /placeholder: ""
          - button "Show more items" [ref=f12e102] [cursor=pointer]
        - generic [ref=f12e106]:
          - generic [ref=f12e110]:
            - button "Collapse outline" [expanded] [ref=f12e112] [cursor=pointer]:
              - img "arrow-from-right" [ref=f12e113]
            - button "Queries" [ref=f12e116] [cursor=pointer]:
              - img "arrow" [ref=f12e117]
          - generic [ref=f12e124]:
            - generic [ref=f12e126]:
              - generic "Query editor row" [ref=f12e129]:
                - generic [ref=f12e130]:
                  - generic [ref=f12e132]:
                    - generic [ref=f12e133]:
                      - button "Collapse query row" [expanded] [ref=f12e134] [cursor=pointer]
                      - generic [ref=f12e137]:
                        - button "Query editor row title A" [ref=f12e138] [cursor=pointer]:
                          - generic [ref=f12e139]: A
                        - emphasis [ref=f12e140]: (Prometheus)
                    - generic [ref=f12e141]:
                      - button "Show data source help" [ref=f12e143] [cursor=pointer]
                      - button "Duplicate query" [ref=f12e147] [cursor=pointer]
                      - button "Hide response" [ref=f12e151] [cursor=pointer]
                      - button "Remove query" [ref=f12e155] [cursor=pointer]
                      - button "Drag and drop to reorder" [ref=f12e158]:
                        - img "Drag and drop to reorder" [ref=f12e159]
                  - generic [ref=f12e162]:
                    - generic [ref=f12e163]:
                      - button "Kick start your query" [ref=f12e164] [cursor=pointer]
                      - generic [ref=f12e167]:
                        - generic [ref=f12e168] [cursor=pointer]: Explain
                        - generic [ref=f12e169]:
                          - checkbox "Explain Toggle switch" [ref=f12e170]
                          - generic "Toggle switch" [ref=f12e171] [cursor=pointer]
                      - radiogroup [ref=f12e176]:
                        - generic [ref=f12e177]:
                          - radio "Builder" [ref=f12e178] [cursor=pointer]
                          - generic [ref=f12e179] [cursor=pointer]: Builder
                        - generic [ref=f12e180]:
                          - radio "Code" [checked] [ref=f12e181] [cursor=pointer]
                          - generic [ref=f12e182] [cursor=pointer]: Code
                    - generic [ref=f12e184]:
                      - generic [ref=f12e186]:
                        - button "Loading metrics..." [disabled] [ref=f12e187] [cursor=pointer]
                        - generic [ref=f12e190]: Loading editor
                      - 'button "Options Legend: {{pod}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f12e198] [cursor=pointer]':
                        - generic [ref=f12e202]:
                          - heading "Options" [level=6] [ref=f12e203]
                          - generic [ref=f12e204]:
                            - generic [ref=f12e205]: "Legend: {{pod}}"
                            - generic [ref=f12e206]: "Format: Time series"
                            - generic [ref=f12e207]: "Step: auto"
                            - generic [ref=f12e208]: "Type: Range"
                            - generic [ref=f12e209]: "Exemplars: false"
              - generic [ref=f12e210]:
                - button "Add query" [ref=f12e211] [cursor=pointer]
                - button "Query history" [ref=f12e215] [cursor=pointer]
                - button "Query inspector" [ref=f12e219] [cursor=pointer]
            - generic:
              - main
@@ -0,0 +1,145 @@
 - generic [active] [ref=f15e1]:
  - generic [ref=f15e4]:
    - link "Skip to main content" [ref=f15e5] [cursor=pointer]:
      - /url: "#pageContent"
    - banner [ref=f15e7]:
      - generic [ref=f15e8]:
        - link [ref=f15e10] [cursor=pointer]:
          - /url: /
          - img "Grafana" [ref=f15e11]
        - generic [ref=f15e14]:
          - button "Search or jump to..." [ref=f15e18] [cursor=pointer]
          - generic [ref=f15e19]: ctrl+k
        - generic [ref=f15e23]:
          - button "New" [ref=f15e24] [cursor=pointer]
          - button "Help" [ref=f15e30] [cursor=pointer]
          - button "News" [ref=f15e33] [cursor=pointer]
          - button "Profile" [ref=f15e36] [cursor=pointer]:
            - img "User avatar" [ref=f15e37]
      - generic [ref=f15e38]:
        - button "Open menu" [ref=f15e40] [cursor=pointer]
        - navigation "Breadcrumbs" [ref=f15e43]:
          - list [ref=f15e44]:
            - listitem [ref=f15e45]:
              - link "Home" [ref=f15e46] [cursor=pointer]:
                - /url: /
            - listitem [ref=f15e50]:
              - link "Explore" [ref=f15e51] [cursor=pointer]:
                - /url: /explore
            - listitem [ref=f15e55]:
              - generic "Prometheus" [ref=f15e56]
        - generic [ref=f15e57]:
          - generic [ref=f15e60]:
            - button "Copy shortened URL" [ref=f15e61] [cursor=pointer]
            - button "Open copy link options" [ref=f15e64] [cursor=pointer]
          - button "Toggle top search bar" [ref=f15e68] [cursor=pointer]
    - main [ref=f15e74]:
      - generic [ref=f15e76]:
        - heading "Explore" [level=1] [ref=f15e77]
        - generic [ref=f15e82]:
          - navigation "Explore toolbar" [ref=f15e84]:
            - navigation "Search links" [ref=f15e86]:
              - generic [ref=f15e87]:
                - button "Content outline" [expanded] [ref=f15e89] [cursor=pointer]:
                  - generic [ref=f15e92]: Outline
                - generic [ref=f15e97] [cursor=pointer]:
                  - img "Prometheus logo" [ref=f15e99]
                  - textbox "Select a data source" [ref=f15e100]:
                    - /placeholder: ""
            - generic [ref=f15e104]:
              - button "Split the pane" [ref=f15e106] [cursor=pointer]:
                - generic [ref=f15e109]: Split
              - button "Add" [ref=f15e111] [cursor=pointer]
              - generic [ref=f15e116]:
                - 'button "Time range selected: Last 30 minutes" [ref=f15e117] [cursor=pointer]'
                - button "Zoom out time range" [ref=f15e122] [cursor=pointer]
              - generic [ref=f15e126]:
                - button "Run query" [ref=f15e127] [cursor=pointer]
                - button "Auto refresh turned off. Choose refresh time interval" [ref=f15e131] [cursor=pointer]
          - generic [ref=f15e135]:
            - generic [ref=f15e139]:
              - button "Collapse outline" [expanded] [ref=f15e141] [cursor=pointer]:
                - img "arrow-from-right" [ref=f15e142]
              - button "Queries" [ref=f15e145] [cursor=pointer]:
                - img "arrow" [ref=f15e146]
              - button "Graph" [ref=f15e150] [cursor=pointer]:
                - img "graph-bar" [ref=f15e151]
            - generic [ref=f15e158]:
              - generic [ref=f15e160]:
                - generic "Query editor row" [ref=f15e163]:
                  - generic [ref=f15e164]:
                    - generic [ref=f15e166]:
                      - generic [ref=f15e167]:
                        - button "Collapse query row" [expanded] [ref=f15e168] [cursor=pointer]
                        - generic [ref=f15e171]:
                          - button "Query editor row title A" [ref=f15e172] [cursor=pointer]:
                            - generic [ref=f15e173]: A
                          - emphasis [ref=f15e174]: (Prometheus)
                      - generic [ref=f15e175]:
                        - button "Show data source help" [ref=f15e177] [cursor=pointer]
                        - button "Duplicate query" [ref=f15e181] [cursor=pointer]
                        - button "Hide response" [ref=f15e185] [cursor=pointer]
                        - button "Remove query" [ref=f15e189] [cursor=pointer]
                        - button "Drag and drop to reorder" [ref=f15e192]:
                          - img "Drag and drop to reorder" [ref=f15e193]
                    - generic [ref=f15e196]:
                      - generic [ref=f15e197]:
                        - button "Kick start your query" [ref=f15e198] [cursor=pointer]
                        - generic [ref=f15e201]:
                          - generic [ref=f15e202] [cursor=pointer]: Explain
                          - generic [ref=f15e203]:
                            - checkbox "Explain Toggle switch" [ref=f15e204]
                            - generic "Toggle switch" [ref=f15e205] [cursor=pointer]
                        - radiogroup [ref=f15e210]:
                          - generic [ref=f15e211]:
                            - radio "Builder" [ref=f15e212] [cursor=pointer]
                            - generic [ref=f15e213] [cursor=pointer]: Builder
                          - generic [ref=f15e214]:
                            - radio "Code" [checked] [ref=f15e215] [cursor=pointer]
                            - generic [ref=f15e216] [cursor=pointer]: Code
                      - generic [ref=f15e218]:
                        - generic [ref=f15e220]:
                          - button "Loading metrics..." [disabled] [ref=f15e221] [cursor=pointer]
                          - code [ref=f15e228]:
                            - generic [ref=f15e229]:
                              - generic [ref=f15e234]: agroal_blocking_time_max_milliseconds
                              - textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=f15e239]: agroal_blocking_time_max_milliseconds
                        - 'button "Options Legend: blocking max - {{pod}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f15e245] [cursor=pointer]':
                          - generic [ref=f15e249]:
                            - heading "Options" [level=6] [ref=f15e250]
                            - generic [ref=f15e251]:
                              - generic [ref=f15e252]: "Legend: blocking max - {{pod}}"
                              - generic [ref=f15e253]: "Format: Time series"
                              - generic [ref=f15e254]: "Step: auto"
                              - generic [ref=f15e255]: "Type: Range"
                              - generic [ref=f15e256]: "Exemplars: false"
                - generic [ref=f15e257]:
                  - button "Add query" [ref=f15e258] [cursor=pointer]
                  - button "Query history" [ref=f15e262] [cursor=pointer]
                  - button "Query inspector" [ref=f15e266] [cursor=pointer]
              - main [ref=f15e270]:
                - region [ref=f15e272]:
                  - generic [ref=f15e273]:
                    - heading "Graph" [level=2] [ref=f15e275]
                    - radiogroup [ref=f15e278]:
                      - generic [ref=f15e279]:
                        - radio "Lines" [checked] [ref=f15e280] [cursor=pointer]
                        - generic [ref=f15e281] [cursor=pointer]: Lines
                      - generic [ref=f15e282]:
                        - radio "Bars" [ref=f15e283] [cursor=pointer]
                        - generic [ref=f15e284] [cursor=pointer]: Bars
                      - generic [ref=f15e285]:
                        - radio "Points" [ref=f15e286] [cursor=pointer]
                        - generic [ref=f15e287] [cursor=pointer]: Points
                      - generic [ref=f15e288]:
                        - radio "Stacked lines" [ref=f15e289] [cursor=pointer]
                        - generic [ref=f15e290] [cursor=pointer]: Stacked lines
                      - generic [ref=f15e291]:
                        - radio "Stacked bars" [ref=f15e292] [cursor=pointer]
                        - generic [ref=f15e293] [cursor=pointer]: Stacked bars
                  - generic [ref=f15e294]: Loading plugin panel...
  - generic [ref=f15e299]:
    - alert
    - alert
    - complementary
    - complementary
@@ -0,0 +1,56 @@
 - generic [ref=f18e4]:
  - link "Skip to main content" [ref=f18e5] [cursor=pointer]:
    - /url: "#pageContent"
  - banner [ref=f18e7]:
    - generic [ref=f18e8]:
      - link [ref=f18e10] [cursor=pointer]:
        - /url: /
        - img "Grafana" [ref=f18e11]
      - generic [ref=f18e14]:
        - button "Search or jump to..." [ref=f18e18] [cursor=pointer]
        - generic [ref=f18e19]: ctrl+k
      - generic [ref=f18e23]:
        - button "New" [ref=f18e24] [cursor=pointer]
        - button "Help" [ref=f18e30] [cursor=pointer]
        - button "News" [ref=f18e33] [cursor=pointer]
        - button "Profile" [ref=f18e36] [cursor=pointer]:
          - img "User avatar" [ref=f18e37]
    - generic [ref=f18e38]:
      - button "Open menu" [ref=f18e40] [cursor=pointer]
      - navigation "Breadcrumbs" [ref=f18e43]:
        - list [ref=f18e44]:
          - listitem [ref=f18e45]:
            - link "Home" [ref=f18e46] [cursor=pointer]:
              - /url: /
          - listitem [ref=f18e50]:
            - link "Explore" [ref=f18e51] [cursor=pointer]:
              - /url: /explore
          - listitem [ref=f18e55]:
            - generic "Prometheus" [ref=f18e56]
      - generic [ref=f18e57]:
        - button "Show more items" [ref=f18e60] [cursor=pointer]
        - button "Toggle top search bar" [ref=f18e64] [cursor=pointer]
  - main [ref=f18e70]:
    - generic [ref=f18e72]:
      - heading "Explore" [level=1] [ref=f18e73]
      - generic [ref=f18e78]:
        - navigation "Explore toolbar" [ref=f18e80]:
          - navigation "Search links" [ref=f18e82]:
            - generic [ref=f18e83]:
              - button "Content outline" [expanded] [ref=f18e85] [cursor=pointer]:
                - generic [ref=f18e88]: Outline
              - generic [ref=f18e93] [cursor=pointer]:
                - img "Prometheus logo" [ref=f18e95]
                - textbox "Select a data source" [ref=f18e96]:
                  - /placeholder: ""
          - button "Show more items" [ref=f18e102] [cursor=pointer]
        - generic [ref=f18e106]:
          - button "Collapse outline" [expanded] [ref=f18e112] [cursor=pointer]:
            - img "arrow-from-right" [ref=f18e113]
          - generic [ref=f18e120]:
            - generic [ref=f18e123]:
              - button "Add query" [ref=f18e124] [cursor=pointer]
              - button "Query history" [ref=f18e128] [cursor=pointer]
              - button "Query inspector" [ref=f18e132] [cursor=pointer]
            - generic:
              - main
@@ -0,0 +1,19 @@
 === [준비] 손실 측정 설계 확인 ===
  LAST_SESSION_REFRESH 는 integer(초) — 200ms 손실은 보이지 않는다
 created_on           | integer                 |           | not null | 
 last_session_refresh | integer                 |           | not null | 0
    "idx_user_session_expiration_created" btree (realm_id, offline_flag, remember_me, created_on, user_session_id, user_id)
    "idx_user_session_expiration_last_refresh" btree (realm_id, offline_flag, remember_me, last_session_refresh, user_session_id, user_id)
  → 대신 행 존재 여부로 잰다. 로그인 하나 = 행 하나 = 이진 판정
  전역 synchronous_commit: on
 === [1] 빠른 연속 로그인을 백그라운드로 시작 ===
  루프 시작
  6초 경과 — 지금까지 성공한 로그인: 0
 === [2] PostgreSQL 강제 종료 (SIGKILL) ===
  종료 시각: 12:00:26.511
 pod "postgres-7b474b88c8-xc2vt" force deleted from keycloak-lab namespace
  삭제 반환: 12:00:26.586
  클라이언트가 200 을 받은 로그인 수: 0
@@ -0,0 +1,14 @@
 deployment "postgres" successfully rolled out
 === crash recovery 가 실행되었는가 (강제 종료의 흔적) ===
 2026-09-04 02:58:41.036 UTC [1] LOG:  database system is ready to accept connections
 === [설계 확인] 로그인 트랜잭션도 synchronous_commit 을 끄는가 ===
  --- 로그인 트랜잭션 (INSERT 가 있는 것) ---
 2:BEGIN
 5:COMMIT
 6:BEGIN
 9:insert into OFFLINE_USER_SESSION (BROKER_SESSION_ID,CREATED_ON,DATA,LAST_SESSION_REFRESH,REALM_ID,REMEMBER_ME,USER_ID,VERSION,OFFLINE_FLAG,USER_SESSION_ID) values ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10)
 10:insert into OFFLINE_CLIENT_SESSION (DATA,REALM_ID,TIMESTAMP,VERSION,CLIENT_ID,CLIENT_STORAGE_PROVIDER,EXTERNAL_CLIENT_ID,OFFLINE_FLAG,USER_SESSION_ID) values ($1,$2,$3,$4,$5,$6,$7,$8,$9)
 11:SET LOCAL synchronous_commit TO OFF
 12:COMMIT
@@ -0,0 +1,17 @@
 === [1] 로그인 루프 시작 (호스트에서 백그라운드로 exec — 세션이 살아 있어야 한다) ===
  8초 동안 클라이언트가 200 을 받은 로그인: 106 건
 === [2] SIGKILL ===
  종료: 12:01:32.981
  반환: 12:01:33.236
  최종 성공 로그인 수: 110 건
    마지막 sid: FimM-krSybBACP2qIvshLWwU
    마지막 sid: EwFfFwOIfqiv8N5GQ5OjtsVq
    마지막 sid: CJX-PxFkS7rUc_9FQgB7iw1f
    마지막 sid: 1EFK7SgUA4M7tq_SkC_BD2er
    마지막 sid: _LiqTczuyxlpOs3T3xs25SLv
 === [3] PostgreSQL 재기동 후 crash recovery 확인 ===
 Waiting for deployment "postgres" rollout to finish: 0 of 1 updated replicas are available...
 deployment "postgres" successfully rolled out
  2026-09-04 02:59:48.427 UTC [1] LOG:  database system is ready to accept connections
@@ -0,0 +1,27 @@
 === [4] 클라이언트가 받은 sid 가 DB 에 있는가 ===
  클라이언트가 200 을 받은 sid: 291 건
  DB 온라인 세션 총계: 375
  --- 마지막 15건을 하나씩 조회 ---
    TCCOYnVlN30Y2fGyJEsVJ_Lq   있음
    vBcllIkKWovh-FXN9tSzmxAe   있음
    eMpN_ywUdRTks9uBE9aTumPK   있음
    aPC_T0yrlMjrlskcLAp0AvY6   있음
    UBPxmduB-ahGHBg636sY3AEz   있음
    HLnBloNX9R3qQJSkpOnkNqL4   있음
    _KPJS30IAqVhkTJGHcCxKvrM   있음
    rq3caZ9MkyMYlFSSzRjQLygD   있음
    ivvQm70hjl55DPpF7vpYmz_E   있음
    81mx-rmi-tAeogHx3-su3z2q   있음
    WnvNDH93uzcz1XNFbaMSXk1A   있음
    DXPAIhjO5sGpCUlcD8IEoS8L   있음
    aZMvl4IwPdK-rC_7bG005Z5C   있음
    eIuBCprfWA5x0glcgSKYrrX0   있음
    ozES5kEeu2IFf_cfcC_jFlbF   있음
  마지막 15건 중 유실: 0 건
 === [5] 전체 대조 — 몇 건이나 사라졌는가 ===
  클라이언트 성공: 291 건
  DB 에 존재:      291 건
  ★ 유실:          0 건
@@ -0,0 +1,12 @@
 === [정리] 세션 테이블 비우고 루프 잔여 확인 ===
 DELETE 375
  남은 세션: 0
 === [재주입] postmaster(PID 1)에 SIGKILL — 진짜 크래시 ===
  8초 후 성공 로그인: 110 건
  SIGKILL: 12:03:21.441
  최종 성공 로그인: 139 건
 === [검증] 이번엔 crash recovery 가 돌았는가 ===
 deployment "postgres" successfully rolled out
  2026-09-04 02:59:48.427 UTC [1] LOG:  database system is ready to accept connections
@@ -0,0 +1,17 @@
 === 로그인 루프 시작 ===
  8초 후: 112 건
 === 백엔드 프로세스에 SIGKILL → postmaster 가 재초기화한다 ===
  시각: 12:04:22.063
  최종 성공 로그인: 153 건
 === [검증] crash recovery 가 돌았는가 ===
  2026-09-04 02:59:48.427 UTC [1] LOG:  database system is ready to accept connections
  2026-09-04 03:02:35.807 UTC [1] LOG:  server process (PID 40) was terminated by signal 9: Killed
  2026-09-04 03:02:35.807 UTC [1] LOG:  terminating any other active server processes
  2026-09-04 03:02:35.814 UTC [1] LOG:  all server processes terminated; reinitializing
  2026-09-04 03:02:35.896 UTC [2585] LOG:  database system was not properly shut down; automatic recovery in progress
  2026-09-04 03:02:35.899 UTC [2585] LOG:  redo starts at 0/23CAB68
  2026-09-04 03:02:35.904 UTC [2585] LOG:  redo done at 0/2529E40 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
  2026-09-04 03:02:35.923 UTC [2586] LOG:  checkpoint complete: wrote 113 buffers (0.7%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.004 s, sync=0.004 s, total=0.015 s; sync files=27, longest=0.003 s, average=0.001 s; distance=1405 kB, estimate=1405 kB; lsn=0/252A048, redo lsn=0/252A048
  2026-09-04 03:02:35.926 UTC [1] LOG:  database system is ready to accept connections
@@ -0,0 +1,19 @@
 === 크래시 전후 대조 ===
  클라이언트가 200 과 토큰을 받은 로그인 : 153 건
  그중 DB 에 실제로 존재                 : 149 건
  ★ 유실                                 : 4 건
  DB 전체 온라인 세션                    : 150 건
 === 유실된 sid 목록 ===
    ★ CQUfg9HLH29xvhiu6pVlfWOo  ← 토큰은 발급됐는데 세션이 없다
    ★ 5gLP4fqmpZBbjhH_d-0TPMMr  ← 토큰은 발급됐는데 세션이 없다
    ★ hkcOv1QskUFmYveMLB6Hljra  ← 토큰은 발급됐는데 세션이 없다
    ★ p5XybeQIYmAs818gO4Vl_5ea  ← 토큰은 발급됐는데 세션이 없다
 === 그 토큰이 지금 실제로 쓰이는가 (마지막 sid 로 확인) ===
  마지막 sid: 8do0Bw6tkVLDVxgxotE7GosH
     user_session_id      | created_on | last_session_refresh 
 --------------------------+------------+----------------------
 8do0Bw6tkVLDVxgxotE7GosH | 1788490958 |           1788490958
 (1 row)
@@ -0,0 +1,20 @@
 # A-3 — DB 강제 종료와 데이터 손실 증거
 2026-09-04 12:00–12:05 KST · Keycloak 26.7.0 / PostgreSQL 16
 해설: [`docs/experiment-a3-database-crash.md`](../../experiment-a3-database-crash.md)
 | 파일 | 무엇을 보여주는가 |
 |---|---|
 | `01-crash-injection.txt` | 첫 시도 실패 — 파드 안 백그라운드 루프가 `exec` 종료와 함께 죽어 0건 수집 |
 | `02-design-check.txt` | **핵심 설계 확인** — 로그인 트랜잭션도 `SET LOCAL synchronous_commit TO OFF` 로 커밋한다 |
 | `03-loss-measurement.txt` | `--grace-period=0 --force` 주입 |
 | `04-comparison.txt` | **유실 0건** — 그러나 crash recovery 가 안 돌았다. 죽인 적이 없는 것 |
 | `05-true-crash.txt` | `kill -9 1` 시도 — **컨테이너 안에서 PID 1 은 SIGKILL 을 무시한다** |
 | `06-backend-kill-crash.txt` | **성공한 주입** — 백엔드에 SIGKILL → `not properly shut down` / `redo starts` / `redo done` |
 | `07-loss-result.txt` | **결과: 153건 중 4건 유실.** 토큰은 발급됐는데 세션 행이 없는 sid 목록 |
 ## 핵심 세 줄
 1. **로그인도 비동기 커밋이다.** refresh 시각뿐 아니라 **로그인 자체**가 사라질 수 있다.
 2. **153건 중 4건(약 2.6%) 유실** — 초당 19건 기준 마지막 0.2초 분량, `wal_writer_delay` 기본값과 일치.
 3. **주입을 세 번 시도해 세 번째에 성공했다.** 앞의 둘은 "손실 0"으로 보였지만 실제로는 크래시가 아니었다.
@@ -0,0 +1,20 @@
 === A-4 기준선 ===
 kc-lab-1   Ready   true
 kc-lab-2   Ready   <none>
 a2-probe                    true   kc-lab-2
 keycloak-0                  true   kc-lab-2
 keycloak-1                  true   kc-lab-1
 postgres-7b474b88c8-2gf27   true   kc-lab-2
 === PVC 가 어느 노드에 묶여 있는가 (재배치 가능성) ===
  persistentvolumeclaim/postgres-data → kc-lab-2
 === 서비스 정상 확인 ===
  https://auth.hyeonworks.com/realms/master  HTTP 200
 === VM 상태 ===
 --------------------------
 1    kc-lab-1   running
 2    kc-lab-2   running
@@ -0,0 +1,17 @@
 === 워커 노드(kc-lab-2) 전원 차단 — virsh destroy 는 종료 신호가 없다 ===
 차단 시각: 12:07:43
 Domain 'kc-lab-2' destroyed
  +15초  node=Ready | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 000
  +30초  node=Ready | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 000
  +45초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
  +60초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
  +75초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
  +90초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
  +105초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
  +120초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
  +135초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
  +150초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
  +165초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
  +180초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
@@ -0,0 +1,28 @@
 === 파드 상태의 진실 — Running 인데 노드가 없다 ===
 a2-probe                    Running   true    kc-lab-2   <none>
 keycloak-0                  Running   true    kc-lab-2   <none>
 keycloak-1                  Running   false   kc-lab-1   <none>
 postgres-7b474b88c8-2gf27   Running   true    kc-lab-2   <none>
 === 재배치가 시도되었는가 ===
 10m         Warning   Unhealthy           pod/keycloak-0                   Readiness probe failed: Get "http://10.42.1.67:9000/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
 3m15s       Warning   NodeNotReady        pod/postgres-7b474b88c8-2gf27    Node is not ready
 3m15s       Warning   NodeNotReady        pod/keycloak-0                   Node is not ready
 3m15s       Warning   NodeNotReady        pod/a2-probe                     Node is not ready
 2m27s       Warning   Unhealthy           pod/keycloak-1                   Readiness probe failed: Get "http://10.42.0.35:9000/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
 2s          Warning   Unhealthy           pod/keycloak-1                   Readiness probe failed: HTTP probe failed with statuscode: 503
 === 노드 taint — 쿠버네티스가 붙인 것 ===
  node.kubernetes.io/unreachable=:NoSchedule
  node.kubernetes.io/unreachable=:NoExecute
 === Prometheus 가 본 것 (kc-lab-1 에 있어 살아남았다) ===
  up{job=keycloak       pod=keycloak-1                } = 1
  up{job=keycloak       pod=keycloak-0                } = 0
  up{job=kubelet        pod=-                         } = 1
  up{job=kubelet        pod=-                         } = 0
  up{job=node-exporter  pod=kc-lab-1                  } = 1
  up{job=node-exporter  pod=kc-lab-2                  } = 0
  up{job=prometheus     pod=-                         } = 1
 === 진입점이 처음 40초간 000 이었던 이유 — nginx upstream ===
@@ -0,0 +1,15 @@
 === nginx 설정 위치 찾기 ===
 === NoExecute taint 의 tolerationSeconds — 언제 축출되는가 ===
  node.kubernetes.io/not-ready NoExecute tolerationSeconds=300
  node.kubernetes.io/unreachable NoExecute tolerationSeconds=300
 === 5분 축출 시점까지 관찰 ===
  +210초  a2-probe:Running keycloak-0:Running keycloak-1:Running postgres-7b474b88c8-2gf27:Running 
  +240초  a2-probe:Running keycloak-0:Running keycloak-1:Running postgres-7b474b88c8-2gf27:Running 
  +270초  a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending 
  +300초  a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending 
  +330초  a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending 
  +360초  a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending 
  +390초  a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending 
  +420초  a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending 
@@ -0,0 +1,18 @@
 === 새 postgres 가 Pending 인 이유 ===
 Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  4m45s  default-scheduler  0/2 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
 === keycloak-0 대체 파드가 안 생기는 이유 (StatefulSet) ===
 keycloak   2     <none>   1
 keycloak-0                  1/1   Terminating   0     30m
 keycloak-1                  0/1   Running       0     143m
 === 복구 — 노드 재기동 ===
 재기동 시각: 12:16:31
 Domain 'kc-lab-2' started
  +30초  node=Ready | Running 파드 3 개 | 외부 HTTP 503
  +60초  node=Ready | Running 파드 3 개 | 외부 HTTP 200
  → 서비스 복귀
@@ -0,0 +1,19 @@
 === 복구 확인 ===
 keycloak-0                  1/1   Running   0     68s
 keycloak-1                  1/1   Running   0     144m
 postgres-7b474b88c8-9cmsv   1/1   Running   0     4m20s
 === kc-lab-1(k3s server)에 무엇이 있는가 — 이게 곧 영향 범위다 ===
  keycloak-lab     keycloak-1
  kube-system      coredns-54996dc9b4-8k8fj
  kube-system      helm-install-traefik-crd-q29b5
  kube-system      local-path-provisioner-77b9867795-g27z8
  kube-system      metrics-server-6dc596dfb8-7xxq4
  kube-system      svclb-traefik-5eb6a9a1-qwwk5
  kube-system      traefik-5d6fcf895-wpfhr
  observability    grafana-845b5678cf-b6gvc
  observability    node-exporter-9qk9w
  observability    prometheus-6774f94f7c-pzr2t
 === Traefik replica 수 (진입점의 단일 장애점인가) ===
 traefik   1     1
@@ -0,0 +1,19 @@
 === 컨트롤 플레인 노드(kc-lab-1) 전원 차단 ===
 차단 시각: 12:18:08
 Domain 'kc-lab-1' destroyed
  +20초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
  +40초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
  +60초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
  +80초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
  +100초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
  +120초  외부 auth=000 grafana=502 | kubectl: Unable to connect to the server: dial tcp 
  +140초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
  +160초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
 === 살아 있는 노드에서 직접 확인 — 워크로드는 도는가 ===
  CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD                 NAMESPACE
  e5f777900b762       60e153026e8f5       4 minutes ago       Running             keycloak            0                   640d4dafaefb3       keycloak-0          keycloak-lab
  6
@@ -0,0 +1,13 @@
 === 컨트롤 플레인 노드 복구 ===
 재기동: 12:23:39
 Domain 'kc-lab-1' started
  +30초  외부=502 | kc-lab-1=Ready kc-lab-2=Ready 
  +60초  외부=200 | kc-lab-1=Ready kc-lab-2=Ready 
  → 서비스 복귀 (총 60초)
 === 최종 상태 ===
 keycloak-0                  1/1   Running   0                   7m57s
 keycloak-1                  1/1   Running   1 (<invalid> ago)   151m
 postgres-7b474b88c8-9cmsv   1/1   Running   0                   11m
  (prometheus port-forward 재연결 필요)
@@ -0,0 +1,23 @@
 # A-4 — 노드 전원 차단 증거
 2026-09-04 12:07–12:24 KST
 해설: [`docs/experiment-a4-node-loss.md`](../../experiment-a4-node-loss.md)
 | 파일 | 무엇을 보여주는가 |
 |---|---|
 | `01-baseline.txt` | 차단 전 — 양쪽 Ready, **PVC 가 kc-lab-2 에 못박혀 있음**(재배치 불가의 원인), 외부 200 |
 | `02-worker-node-killed.txt` | `virsh destroy kc-lab-2` — **40초간 노드가 Ready 로 남아 있고** 외부는 이미 `000`. 이후 `503` |
 | `03-state-during-loss.txt` | **죽은 파드가 `ready=true`, 산 파드가 `ready=false`.** `up` 은 정확히 0. `unreachable` taint |
 | `04-eviction-timing.txt` | `tolerationSeconds=300` — **5분 뒤** 축출, 새 postgres 는 `Pending` |
 | `05-recovery.txt` | `FailedScheduling: didn't match PersistentVolume's node affinity`, StatefulSet `DESIRED=2 CURRENT=1`. 노드 복귀 후 **60초** |
 | `06-control-plane-inventory.txt` | kc-lab-1 에 있는 것 목록 — **Traefik `replicas=1`** |
 | `07-control-plane-loss.txt` | `virsh destroy kc-lab-1` — 외부 `000`, `kubectl` 불통. **그런데 `crictl ps` 로 보면 keycloak-0 은 Running** |
 | `08-control-plane-recovery.txt` | 60초 만에 복귀 |
 | `a4-up-dropped-per-node.png` | Grafana — `up` 이 노드별로 0 으로 떨어지는 구간. 12:18–12:23 은 **0 이 아니라 데이터 없음**(관측자가 같이 죽음) |
 ## 핵심 네 줄
 1. **쿠버네티스는 40초 동안 노드가 살아 있다고 말한다.** 사용자는 이미 장애를 겪는 중이다.
 2. **죽은 파드의 상태는 화석이다.** `ready=true` 인 파드가 꺼진 기계 위에 있다.
 3. **StatefulSet 은 대체 파드를 만들지 않고, PVC 는 재배치를 막는다.** 사람이 개입해야 한다.
 4. **컨트롤 플레인 상실 ≠ 워크로드 상실.** 컨테이너는 계속 돌고, 들어갈 문만 사라진다.
@@ -0,0 +1,42 @@
 === A-5 기준선 ===
  keycloak-0=10.42.1.77 (kc-lab-2)   keycloak-1=10.42.0.42 (kc-lab-1)
 keycloak-0                  1/1   Running   0               11m
 keycloak-1                  1/1   Running   1 (2m48s ago)   155m
 postgres-7b474b88c8-9cmsv   1/1   Running   0               14m
 Traceback (most recent call last):
  File "<string>", line 3, in <module>
    for r in json.load(sys.stdin)["data"]["result"]: print(f"  cluster_size {r["metric"].get("pod"):12} = {r["value"][1]}")
             ~~~~~~~~~^^^^^^^^^^^
  File "/usr/lib/python3.14/json/__init__.py", line 298, in load
    return loads(fp.read(),
        cls=cls, object_hook=object_hook,
        parse_float=parse_float, parse_int=parse_int,
        parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib/python3.14/json/__init__.py", line 352, in loads
    return _default_decoder.decode(s)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^
  File "/usr/lib/python3.14/json/decoder.py", line 345, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.14/json/decoder.py", line 363, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
 json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
 === 주입: keycloak-0 으로 들어가는 7800/57800 만 DROP (한 방향) ===
  A-1 의 NetworkPolicy 는 conntrack ESTABLISHED 에 막혀 기존 연결을 못 끊었다.
  FORWARD 최상단에 넣으면 conntrack 승인보다 먼저 평가된다.
 Chain FORWARD (policy ACCEPT)
 num  target     prot opt source               destination         
 1    DROP       6    --  0.0.0.0/0            10.42.1.77           tcp dpt:57800
 2    DROP       6    --  0.0.0.0/0            10.42.1.77           tcp dpt:7800
  주입 시각: 12:28:23
 === 관찰 — 어느 쪽이 먼저 상대를 의심하는가 ===
  +25초   | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
  +50초   | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
  +75초   | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
  +100초   | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
  +125초   | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
  +150초   | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
  +175초   | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
  +200초   | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
@@ -0,0 +1,16 @@
 === 실패한 규칙 제거 ===
 제거완료
 === raw 테이블 PREROUTING 에 넣는다 — conntrack 보다 먼저 실행된다 ===
  netfilter 순서: raw PREROUTING → conntrack → mangle → nat → filter
 Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 num   pkts bytes target     prot opt in     out     source               destination         
 1        0     0 DROP       6    --  *      *       0.0.0.0/0            10.42.1.77           tcp dpt:57800
 2        0     0 DROP       6    --  *      *       0.0.0.0/0            10.42.1.77           tcp dpt:7800
  주입: 12:32:35
 === [검증] 이번엔 패킷이 걸렸는가 ===
  Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
   pkts bytes target     prot opt in     out     source               destination         
      0     0 DROP       6    --  *      *       0.0.0.0/0            10.42.1.77           tcp dpt:57800
      0     0 DROP       6    --  *      *       0.0.0.0/0            10.42.1.77           tcp dpt:7800
@@ -0,0 +1,18 @@
 === keycloak-1(수신측)으로 들어가는 7800/57800 만 DROP — kc-lab-1 에 넣는다 ===
  주입: 12:33:58
 === [검증] 패킷이 걸리는가 ===
  Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
   pkts bytes target     prot opt in     out     source               destination         
      0     0 DROP       6    --  *      *       0.0.0.0/0            10.42.0.42           tcp dpt:57800
     19  2938 DROP       6    --  *      *       0.0.0.0/0            10.42.0.42           tcp dpt:7800
 === 관찰 ===
  +25초  - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
  +50초  - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
  +75초  - | keycloak-0:1/1 keycloak-1:0/1 | 외부 200
  +100초  - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
  +125초  - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
  +150초  - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
  +175초  - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
  +200초  - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
@@ -0,0 +1,19 @@
 === 지금 연결 방향은? (차단은 → 10.42.0.42:7800 만) ===
  tcp      6 299 ESTABLISHED src=10.42.0.42 dst=10.42.1.77 sport=48473 dport=7800 src=10.42.1.77 dst=10.42.0.42 sport=7800 dport=48473
  tcp      6 86232 ESTABLISHED src=10.42.0.42 dst=10.42.1.77 sport=44205 dport=57800 src=10.42.1.77 dst=10.42.0.42 sport=57800 dport=44205
 === 차단 규칙 누적 카운터 ===
  Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
   pkts bytes target     prot opt in     out     source               destination         
     19  1096 DROP       6    --  *      *       0.0.0.0/0            10.42.0.42           tcp dpt:57800
     21  3058 DROP       6    --  *      *       0.0.0.0/0            10.42.0.42           tcp dpt:7800
 === 클러스터 멤버 (파드에서 직접) ===
  keycloak-0  cluster_size=2.0
  keycloak-1  cluster_size=2.0
 === 최근 뷰 로그 ===
  keycloak-0: [keycloak-0-24309(v=16.0.12)|12] (1) [keycloak-0-24309(v=16.0.12)]
  keycloak-1: [keycloak-1-45480(v=16.0.12)|12] (1) [keycloak-1-45480(v=16.0.12)]
@@ -0,0 +1,28 @@
 === 뷰 변화 전체 이력 (차단 12:33:58 전후) ===
 --- keycloak-0 ---
  2026-09-04 03:33:49 | during merge: CacheTopology{id=37, phase=NO_REBALANCE, rebalanceId=11, currentCH=DefaultConsistentHash{ns=25
  2026-09-04 03:33:49,398 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=offlineSessions] ISPN100007: After me
  2026-09-04 03:33:49 | during merge: CacheTopology{id=37, phase=NO_REBALANCE, rebalanceId=11, currentCH=DefaultConsistentHash{ns=25
  2026-09-04 03:33:49,399 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=loginFailures] ISPN100007: After merg
  2026-09-04 03:33:49 | during merge: CacheTopology{id=37, phase=NO_REBALANCE, rebalanceId=11, currentCH=DefaultConsistentHash{ns=25
  2026-09-04 03:33:49,402 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=actionTokens] ISPN100007: After merge
 --- keycloak-1 ---
  2026-09-04 03:32:59,874 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p2-t2) [Context=work] ISPN100007: After merge (or coo
  2026-09-04 03:32:59,879 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p2-t2) [Context=clientSessions] ISPN100007: After mer
  2026-09-04 03:32:59,879 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=offlineSessions] ISPN100007: After me
  2026-09-04 03:32:59,883 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=loginFailures] ISPN100007: After merg
  2026-09-04 03:32:59,884 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p2-t2) [Context=actionTokens] ISPN100007: After merge
  2026-09-04 03:33:49 | MergeView::[keycloak-0-24309(v=16.0.12)|13] (2) [keycloak-0-24309(v=16.0.12), keycloak-1-45480(v=16.0.12)], 
 === MERGE3 병합 이벤트 ===
  keycloak-0  merge_events=1.0  suspected=0.0
  keycloak-1  merge_events=1.0  suspected=0.0
 === 차단 해제 ===
 해제완료
  keycloak-0  cluster_size=2.0
 warning: couldn't attach to pod/a5-f, falling back to streaming logs: unable to upgrade connection: container a5-f not found in pod a5-f_keycloak-lab
  keycloak-0  cluster_size=2.0
  keycloak-1  cluster_size=2.0
 keycloak-0                  1/1   Running   0             22m
 keycloak-1                  1/1   Running   1 (13m ago)   166m
@@ -0,0 +1,17 @@
 === 양방향 차단 — 두 노드 모두에 raw DROP ===
  주입: 12:40:25
  +25초  keycloak-0:1/1 keycloak-1:1/1 | ready=[10.42.0.42 10.42.1.77] 외부 200
  +50초  keycloak-0:1/1 keycloak-1:1/1 | ready=[10.42.0.42 10.42.1.77] 외부 200
  +75초  keycloak-0:1/1 keycloak-1:1/1 | ready=[10.42.0.42 10.42.1.77] 외부 200
  +100초  keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
  +125초  keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
  +150초  keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
  +175초  keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
  +200초  keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
  +225초  keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
 === 뷰 상태 ===
  keycloak-0  [keycloak-0-24309(v=16.0.12)|14] (1) [keycloak-0-24309(v=16.0.12)]
  keycloak-1  [keycloak-1-45480(v=16.0.12)|14] (1) [keycloak-1-45480(v=16.0.12)]
@@ -0,0 +1,39 @@
 === 왜 한쪽만 DOWN 인가 — 코디네이터 여부 확인 ===
       name       |       ip        | coord 
 ------------------+-----------------+-------
 keycloak-0-24309 | 10.42.1.77:7800 | t
 keycloak-1-45480 | 10.42.0.42:7800 | t
 (2 rows)
 === 양쪽 헬스체크 상세 ===
 --- keycloak-0 ---
 {"status":"UP","checks":[{"name":"GracefulShutdown","status":"UP"}
 {"name":"KeycloakInitialized","status":"UP"}
 {"name":"Keycloakclusterhealthcheck","status":"UP"}
 --- keycloak-1 ---
 {"status":"DOWN","checks":[{"name":"GracefulShutdown","status":"UP"}
 {"name":"Keycloakdatabaseconnectionsasynchealthcheck","status":"UP"}
 {"name":"KeycloakInitialized","status":"UP"}
 warning: couldn't attach to pod/a5-h, falling back to streaming logs: Internal error occurred: Internal error occurred: error attaching to container: container is in CONTAINER_EXITED state
 --- keycloak-0 ---
 {"status":"UP","checks":[{"name":"GracefulShutdown","status":"UP"}
 {"name":"KeycloakInitialized","status":"UP"}
 {"name":"Keycloakclusterhealthcheck","status":"UP"}
 --- keycloak-1 ---
 {"status":"DOWN","checks":[{"name":"GracefulShutdown","status":"UP"}
 {"name":"Keycloakdatabaseconnectionsasynchealthcheck","status":"UP"}
 {"name":"KeycloakInitialized","status":"UP"}
 === 차단 해제 ===
  해제: 12:44:37
  +25초  keycloak-0:1/1 keycloak-1:0/1 
  +50초  keycloak-0:1/1 keycloak-1:1/1 
  → 복구 완료
  keycloak-0  MergeView::[keycloak-0-24309(v=16.0.12)|15] (2) [keycloak-0-24309(v=16.0.12), keycloak-1-45480(
  keycloak-1  MergeView::[keycloak-0-24309(v=16.0.12)|15] (2) [keycloak-0-24309(v=16.0.12), keycloak-1-45480(
@@ -0,0 +1,23 @@
 # A-5 — 비대칭 파티션 증거
 2026-09-04 12:28–12:46 KST
 해설: [`docs/experiment-a5-asymmetric-partition.md`](../../experiment-a5-asymmetric-partition.md)
 | 파일 | 무엇을 보여주는가 |
 |---|---|
 | `01-injection.txt` | 첫 시도 — `iptables -I FORWARD 1` |
 | `02-injection-verify.txt` | **실패 확인** — kube-router 가 자기 체인을 위로 재삽입, DROP 규칙 **0 패킷** |
 | `03-raw-table-injection.txt` | `raw` 테이블로 이동. 그래도 0 패킷 |
 | `04-correct-direction.txt` | **연결 방향이 반대였다.** 수신측 노드로 옮기니 **19/21 패킷 차단** |
 | `05-reconnect-observed.txt` | **핵심** — 연결이 **열린 방향으로 뒤집혀 재연결**. 뷰 변화 없음, `suspected=0` |
 | `06-view-history-and-cleanup.txt` | 뷰 ID 이력. 주입 이후 변화 없음 확인 |
 | `07-bidirectional-block.txt` | 양방향 차단 → **뷰 14, 멤버 1개씩.** 그런데 `ready=[10.42.1.77]` 로 **한쪽은 남고 외부 200** |
 | `08-coordinator-and-recovery.txt` | **`coord=t` 가 둘**(split brain). keycloak-0 `UP` / keycloak-1 `DOWN`. 해제 후 50초, `MergeView |15] (2)` |
 | `a5-cluster-size-bidirectional-block.png` | Grafana — `vendor_cluster_size` 가 갈라졌다 합쳐지는 구간 |
 ## 핵심 네 줄
 1. **주입을 세 번 실패했다** — CNI 체인 경쟁, 연결 방향 오판, conntrack. 셋 다 "아무 일 없음"으로 보였다.
 2. **단방향 차단으로는 못 가른다.** JGroups 가 열린 방향으로 재연결한다 (자가 치유).
 3. **양방향이면 갈라진다** — `coord=t` 가 둘.
 4. **그래도 전면 장애가 아니다.** 코디네이터 쪽만 UP 을 유지해 서비스가 계속된다 — A-1 의 열린 질문에 대한 답.
@@ -0,0 +1,22 @@
 === 배치 — 왜 A/B 가 되는가 ===
  postgres  10.42.1.76  (kc-lab-2)
  keycloak-0 10.42.1.77 (kc-lab-2) → DB 와 같은 노드, cni0 로 직행
  keycloak-1 10.42.0.42 (kc-lab-1) → DB 와 다른 노드, VXLAN 을 건넌다  ← 여기에 지연을 건다
 === 사용 가능한 커넥션 풀 지표 ===
 agroal_acquire_count_total
 agroal_active_count
 agroal_available_count
 agroal_awaiting_count
 agroal_blocking_time_average_milliseconds
 agroal_blocking_time_max_milliseconds
 agroal_blocking_time_total_milliseconds
 agroal_creation_count_total
 agroal_creation_time_average_milliseconds
 agroal_creation_time_max_milliseconds
 agroal_creation_time_total_milliseconds
 agroal_destroy_count_total
 === 기준선 지연 — 각 노드에서 로그인 20회 ===
  keycloak-0  평균 70 ms
  keycloak-1  평균 66 ms
@@ -0,0 +1,25 @@
 === 주입: postgres(10.42.1.76) 가 보내는 패킷만 200ms 지연 (kc-lab-2 eth0) ===
  prio qdisc 로 밴드를 나누고, u32 필터로 출발지 IP 가 postgres 인 것만 3번 밴드로 보낸다
 Cannot find device "eth0"
 Cannot find device "eth0"
 적용완료
 Cannot find device "eth0"
 Cannot find device "eth0"
  주입: 13:14:55
 === [검증] 지연이 실제로 걸렸는가 — 두 노드 비교 ===
  keycloak-0  평균 43 ms   최대 64 ms
  keycloak-1  평균 47 ms   최대 70 ms
 === 커넥션 풀 상태 ===
 --- keycloak-0 ---
  agroal_blocking_time_max_milliseconds 102.0
  agroal_active_count 0.0
  agroal_awaiting_count 0.0
  agroal_blocking_time_average_milliseconds 0.0
  agroal_available_count 2.0
  agroal_blocking_time_max_milliseconds 164.0
  agroal_active_count 0.0
  agroal_awaiting_count 0.0
  agroal_blocking_time_average_milliseconds 0.0
  agroal_available_count 2.0
@@ -0,0 +1,20 @@
 === 오버레이 인터페이스 확인 ===
 flannel.1        UNKNOWN        a6:b2:62:04:c1:a4 <BROADCAST,MULTICAST,UP,LOWER_UP> 
 cni0             UP             5a:77:1a:e2:b0:a4 <BROADCAST,MULTICAST,UP,LOWER_UP> 
 === flannel.1 에 주입 — 여기서는 파드 IP 가 보인다 (캡슐화 전) ===
 qdisc prio 1: root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
 qdisc netem 30: parent 1:3 limit 1000 delay 200ms
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
 === [검증] 필터에 패킷이 걸리는가 ===
  qdisc netem 30: parent 1:3 limit 1000 delay 200ms
   Sent 18388 bytes 150 pkt (dropped 0, overlimits 0 requeues 0) 
   backlog 0b 0p requeues 0
 === 두 노드 지연 비교 (기준선: k0=70ms k1=66ms) ===
  keycloak-0  평균 41 ms   최대 57 ms
  keycloak-1  평균 1872 ms   최대 1887 ms
@@ -0,0 +1,37 @@
 === 동시 부하 20건을 keycloak-1 에 — 커넥션 풀이 견디는가 ===
        1 200 1.911191
        1 200 1.913766
        1 200 1.958374
        1 200 1.981620
        1 200 10.539402
        1 200 11.951943
        1 200 13.351102
        1 200 14.785832
        1 200 16.189533
        1 200 17.625166
        1 200 19.053724
        1 200 20.495883
        1 200 21.905932
        1 200 22.228466
        1 200 22.230871
        1 200 3.441366
        1 200 4.841075
        1 200 6.257489
        1 200 7.704608
        1 200 9.104792
 === 부하 직후 커넥션 풀 ===
  agroal_blocking_time_max_milliseconds 20000.0
  agroal_max_used_count 19.0
  agroal_acquire_count_total 672.0
  agroal_active_count 0.0
  agroal_awaiting_count 0.0
  agroal_blocking_time_average_milliseconds 281.0
  agroal_available_count 19.0
 === readiness 가 흔들렸는가 ===
 keycloak-0                  1/1   Running   0             60m
 keycloak-1                  1/1   Running   1 (51m ago)   3h24m
 52m         Normal    TaintManagerEviction   pod/keycloak-1                   Cancelling deletion of Pod keycloak-lab/keycloak-1
 32m         Warning   Unhealthy              pod/keycloak-1                   Readiness probe failed: HTTP probe failed with statuscode: 503
 89s         Warning   Unhealthy              pod/keycloak-1                   Readiness probe failed: Get "http://10.42.0.42:9000/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
@@ -0,0 +1,12 @@
 === 지연 해제 ===
 해제완료
 qdisc noqueue 0: root refcnt 2 
 === 회복 확인 ===
  keycloak-0  평균 43 ms
  keycloak-1  평균 51 ms
 keycloak-0                  1/1   Running   0             61m
 keycloak-1                  1/1   Running   1 (52m ago)   3h24m
 === 낙관적 락 충돌이 늘었는가 — 지연 중 로그 ===
  관련 로그 줄수: 0
@@ -0,0 +1,20 @@
 # A-6 — 지연 주입 증거
 2026-09-04 13:10–13:35 KST
 해설: [`docs/experiment-a6-latency-injection.md`](../../experiment-a6-latency-injection.md)
 | 파일 | 무엇을 보여주는가 |
 |---|---|
 | `01-baseline.txt` | 배치 설명(A/B 가 되는 이유), `agroal_*` 지표 목록, **기준선 70ms / 66ms** |
 | `02-delay-injected.txt` | 첫 시도 실패 — **`Cannot find device "eth0"`** (Debian 은 `enp1s0`) |
 | `03-flannel-injection.txt` | **성공** — `flannel.1` 에 걸어야 파드 IP 가 보인다. netem `Sent 150 pkt` 로 검증. **k0=41ms vs k1=1872ms** |
 | `04-pool-under-load.txt` | 동시 20건 — 응답이 1.9초에서 **22.2초**까지 계단. `blocking_time_max=20000ms`, `max_used_count=19`, **readiness 프로브 타임아웃** |
 | `05-recovery.txt` | 해제 즉시 43ms / 51ms 회복. **낙관적 락 충돌 0건**(예측 빗나감) |
 | `a6-connection-pool-blocking.png` | Grafana — `agroal_blocking_time_max_milliseconds` |
 ## 핵심 네 줄
 1. **200ms 가 1,872ms 가 된다.** 로그인 트랜잭션의 왕복이 9번이라 지연이 곱해진다.
 2. **동시 부하에서 22초까지 늘어난다.** 커넥션 풀 큐잉으로 한 번 더 곱해진다.
 3. **헬스체크도 같은 줄에 선다** — 프로브가 타임아웃되어 노드가 로드밸런서에서 빠지고, 남은 노드로 부하가 몰린다.
 4. **낙관적 락 충돌은 없었다.** 로그인은 새 행을 만들 뿐 같은 행을 다투지 않는다 — B-3 의 영역.
@@ -0,0 +1,18 @@
 === 비교를 위해 세션을 비운다 ===
 DELETE 151
 === volatile 모드로 전환 ===
 namespace/keycloak-lab unchanged
 secret/keycloak-lab-secrets configured
 persistentvolumeclaim/postgres-data unchanged
 deployment.apps/postgres unchanged
 service/postgres unchanged
 statefulset.apps/keycloak configured
 service/keycloak-headless unchanged
 service/keycloak unchanged
 ingress.networking.k8s.io/keycloak unchanged
 Waiting for 1 pods to be ready...
 partitioned roll out complete: 2 new pods have been updated...
 === [검증] 정말 꺼졌는가 ===
 ["start","--features-disabled=persistent-user-sessions"]
@@ -0,0 +1,15 @@
  keycloak-0=10.42.1.94  keycloak-1=10.42.0.45
 === [A-0 재실행] keycloak-0 에만 로그인 5회 → 캐시가 어디에 담기는가 ===
 로그인완료
  keycloak-0  sessions 캐시 5.0 건
  keycloak-1  sessions 캐시 0.0 건
 === DB 에는 들어갔는가 (persistent 였을 때는 5건이 들어갔다) ===
 offline_flag | count 
 --------------+-------
 (0 rows)
 === 교차 노드 세션은 되는가 ===
  keycloak-0 로그인 → keycloak-1 에서 refresh  HTTP 200
@@ -0,0 +1,13 @@
 === [A-8 재실행] 재시작 전 로그인 ===
  sid = aVwYnzKZFFvMqD3bpSeiILuM
 === 롤링 재시작 ===
 statefulset.apps/keycloak restarted
 partitioned roll out complete: 2 new pods have been updated...
 === ★ 재시작 전 토큰이 아직 통하는가 (persistent 였을 때는 200) ===
  keycloak-0 에서 refresh  HTTP 400
  --- 오류 본문 ---
 {"error":"invalid_grant","error_description":"Session not active"}
 === 캐시 상태 ===
  keycloak-1  sessions 캐시 1.0 건
@@ -0,0 +1,21 @@
 === [A-1 재실행] volatile 에서 7800 을 막으면 ===
  keycloak-0=10.42.1.99  keycloak-1=10.42.0.46
  [대조군] 차단 전 교차 노드 refresh
    keycloak-1 에서 refresh  HTTP 200
  차단 적용 (A-5 에서 확인한 raw 테이블 방식, 양방향)
  분단이 성립할 때까지 대기...
    +25초  cluster_size(k0 k1) = [2.0 2.0 ]
    +50초  cluster_size(k0 k1) = [1.0 ]
    +75초  cluster_size(k0 k1) = [1.0 ]
    +100초  cluster_size(k0 k1) = []
    +125초  cluster_size(k0 k1) = [1.0 ]
    +150초  cluster_size(k0 k1) = [1.0 ]
    +175초  cluster_size(k0 k1) = [1.0 ]
    +200초  cluster_size(k0 k1) = []
 === ★ 분단 상태에서 교차 노드 세션 (persistent 였을 때는 200) ===
  keycloak-0 로그인 → keycloak-0 에서 refresh  HTTP 200  ← 대조군
  keycloak-0 로그인 → keycloak-1 에서 refresh  HTTP 400  ← 시험군
  --- 시험군 오류 본문 ---
 {"error":"invalid_grant","error_description":"Session not active"}
@@ -0,0 +1,12 @@
 차단 해제, 클러스터 재형성 대기...
 === [A-2 재실행] volatile 에서 DB 를 내리면 — 세션이 메모리에 있으니 살아남는가? ===
  DB 정지 전 로그인 완료
 deployment.apps/postgres scaled
  postgres 정지
  ① 캐시를 가진 노드에서 refresh  HTTP 500
  ② 새 로그인                     HTTP 200
 === DB 복구 후 원복 ===
 deployment.apps/postgres scaled
 deployment "postgres" successfully rolled out
@@ -0,0 +1,12 @@
 === persistent 모드로 원복 ===
 statefulset.apps/keycloak configured
 partitioned roll out complete: 2 new pods have been updated...
 === [검증] persistent 로 돌아왔는가 — 로그인 후 DB 에 행이 생기는가 ===
 ["start"]
 로그인
  DB 온라인 세션: 1 건  (1 이면 persistent 복귀)
 keycloak-0                  1/1   Running   0     67s
 keycloak-1                  1/1   Running   0     89s
 postgres-7b474b88c8-t6rrf   1/1   Running   0     2m8s
  외부 진입점 HTTP 200
@@ -0,0 +1,23 @@
 # A-7 — volatile 모드 비교 증거
 2026-09-04 13:45–14:15 KST
 해설: [`docs/experiment-a7-volatile-comparison.md`](../../experiment-a7-volatile-comparison.md)
 | 파일 | 무엇을 보여주는가 |
 |---|---|
 | `01-switch-to-volatile.txt` | `--features-disabled=persistent-user-sessions` 적용, args 확인 |
 | `02-a0-rerun.txt` | **DB 0건**인데 교차 노드 refresh `200` — 경로가 DB 에서 클러스터로 바뀌었다 |
 | `03-a8-rerun-restart.txt` | **롤링 재시작 후 `400 Session not active`** — persistent 에서는 `200` 이었다 |
 | `04-a1-rerun-partition.txt` | **7800 차단 시 교차 노드 `400`** — persistent 에서는 `200`. 대조군(같은 노드)은 `200` 유지 |
 | `05-a2-rerun-db-loss.txt` | DB 정지 중 **새 로그인 `200`**(persistent 에서는 500), refresh 는 `500` |
 | `06-restore-persistent.txt` | 원복 확인 — `args: ["start"]`, 로그인 후 DB 1건, 외부 200 |
 ## 뒤집힌 결과
 | 실험 | persistent | volatile |
 |---|---|---|
 | A-1 7800 차단 후 교차 refresh | `200` | **`400`** |
 | A-8 롤링 재시작 후 refresh | `200` | **`400`** |
 | A-2 DB 정지 중 새 로그인 | `500` | **`200`** |
 **같은 주입, 같은 관측, 정반대 결과.** A층 전체가 버전 조건부임을 보여주는 대조군이다.
@@ -0,0 +1,17 @@
 === [1] 재시작 전 로그인 — 토큰을 파드 안에 보관 ===
  sid = XLcgQWRiJrTkuNZcJsNeT_2j
  DB 세션 수: 151
 === [2] 롤링 재시작 중 가용성 — 5초 간격으로 외부 진입점 확인 ===
 statefulset.apps/keycloak restarted
 200 Waiting for partitioned roll out to finish: 0 out of 2 new pods have been updated...
 Waiting for 1 pods to be ready...
 Waiting for 1 pods to be ready...
 Waiting for 1 pods to be ready...
 200 200 200 200 Waiting for partitioned roll out to finish: 1 out of 2 new pods have been updated...
 Waiting for 1 pods to be ready...
 Waiting for 1 pods to be ready...
 Waiting for 1 pods to be ready...
 200 200 200 200 partitioned roll out complete: 2 new pods have been updated...
  (위 숫자열이 재시작 중 외부 응답 코드의 시계열)
@@ -0,0 +1,19 @@
 === [3] 재시작 전 발급한 refresh token 이 아직 통하는가 ===
  대상 sid: XLcgQWRiJrTkuNZcJsNeT_2j
  keycloak-0 에서 refresh  HTTP 200
 === [4] DB 에 그 세션이 남아 있는가 ===
     user_session_id      | created_on | last_session_refresh 
 --------------------------+------------+----------------------
 XLcgQWRiJrTkuNZcJsNeT_2j | 1788495513 |           1788495577
 (1 row)
  전체 온라인 세션: 151 (재시작 전 151)
 === [5] 캐시는 어떻게 되었는가 ===
  keycloak-0  sessions 캐시 0.0 건 / cluster_size 2.0
  keycloak-1  sessions 캐시 1.0 건 / cluster_size 2.0
 === [6] 파드 나이 — 정말 재시작되었나 ===
 keycloak-0                  1/1   Running   0     44s
 keycloak-1                  1/1   Running   0     66s
@@ -0,0 +1,16 @@
 # A-8 — 롤링 재시작 증거
 2026-09-04 13:38–13:41 KST
 해설: [`docs/experiment-a8-rolling-restart.md`](../../experiment-a8-rolling-restart.md)
 | 파일 | 무엇을 보여주는가 |
 |---|---|
 | `01-restart-availability.txt` | 재시작 전 로그인(sid 기록), 롤링 재시작 중 외부 진입점 **9회 모두 `200`** |
 | `02-session-survival.txt` | 재시작 전 토큰으로 refresh **`200`**, DB 행 생존(`last_session_refresh` 갱신 확인), **세션 수 151 → 151**, 캐시 0으로 초기화, 파드 나이 44초/66초 |
 | `a8-cache-reset-cluster-reformed.png` | Grafana — 세션 캐시가 0 으로 떨어지고 `cluster_size` 가 다시 2 가 되는 구간 |
 ## 핵심 세 줄
 1. **무중단이었다.** 한 번에 하나씩 내리고 readiness 가 전환을 맞춰준다 — replica ≥ 2 가 전제.
 2. **세션은 살아남고 캐시만 사라진다.** DB 151건 그대로, 재시작 전 토큰이 그대로 통한다.
 3. **이것이 `persistent-user-sessions` 를 켜는 진짜 이유다.** A-7(volatile)에서 정반대가 나와야 한다.
@@ -0,0 +1,323 @@
 # A-3 — DB 를 강제로 죽이면 무엇을 잃는가 (RPO)
 브랜치 `feature/keycloak-a3-database-crash` ·
 증거 [`docs/evidence/a3-database-crash/`](evidence/a3-database-crash/) ·
 2026-09-04 12:00–12:05 KST · Keycloak 26.7.0 / PostgreSQL 16
 선행: [`A-0`](experiment-00-session-replication.md) ·
 [`A-2`](experiment-a2-database-loss.md)
 ---
 ## 0. 결론부터
 ```
  클라이언트가 200 과 토큰을 받은 로그인 : 153 건
  그중 DB 에 실제로 존재                 : 149 건
  ★ 유실                                 :   4 건
 ```
 **로그인이 성공했다고 응답받았는데 세션이 존재하지 않는다.**
 A-0 에서 발견한 `SET LOCAL synchronous_commit TO OFF` 의 대가를 실측했다.
 버그가 아니라 **의도된 설계**이며, 그 비용이 얼마인지를 숫자로 확인한 것이다.
 ---
 ## 1. 설계 — 무엇을 재야 손실이 보이는가
 ### 1-1. `LAST_SESSION_REFRESH` 로는 못 잰다
 처음 계획은 "세션 갱신 시각이 되감기는지" 보는 것이었다. 스키마를 보고 접었다.
 ```
 created_on           | integer
 last_session_refresh | integer      ← 초 단위
 ```
 **손실 창은 수백 밀리초**인데 눈금이 **1초**다. 보일 리가 없다.
 ### 1-2. 행 존재 여부로 잰다 — 이진 판정
 ```
   로그인 1회  =  OFFLINE_USER_SESSION 행 1개
   클라이언트가 sid 를 받았다  =  서버가 COMMIT 했다고 응답했다
   크래시 후 그 sid 가 없다    =  잃은 것
 ```
 **있거나 없거나**이므로 눈금 문제가 없다.
 ### 1-3. 그런데 로그인도 비동기 커밋인가 — **먼저 확인해야 한다**
 A-0 에서 잡은 것은 **refresh** 트랜잭션이었다. 로그인(INSERT)도 그런지는
 확인하지 않았다. 아니라면 이 측정 설계 자체가 성립하지 않는다.
 ```bash
 kubectl -n keycloak-lab exec deploy/postgres -- \
  psql -U keycloak -d keycloak -c "alter system set log_statement='all'"
 kubectl -n keycloak-lab exec deploy/postgres -- \
  psql -U keycloak -d keycloak -c "select pg_reload_conf()"
 ```
 ```
 BEGIN
 insert into OFFLINE_USER_SESSION (...) values (...)
 insert into OFFLINE_CLIENT_SESSION (...) values (...)
 SET LOCAL synchronous_commit TO OFF          ← 로그인도 비동기 커밋이다
 COMMIT
 ```
 **확인됐고, 함의가 refresh 보다 훨씬 무겁다.**
 | | 잃으면 |
 |---|---|
 | refresh 갱신 시각 | 세션 수명이 조금 짧아진다. 사용자는 모른다 |
 | **로그인 자체** | **토큰은 손에 있는데 세션이 없다.** 다음 요청부터 실패 |
 ---
 ## 2. 실패한 주입 ① — `--grace-period=0 --force` 는 크래시가 아니다
 ```bash
 kubectl -n keycloak-lab delete pod -l app=postgres --grace-period=0 --force
 ```
 ```
  클라이언트 성공: 291 건
  DB 에 존재:      291 건
  ★ 유실:            0 건
 ```
 **0건.** 그런데 이건 "안 잃었다"가 아니라 **죽인 적이 없는 것**이다.
 ```
 === 재기동 로그 ===
  database system is ready to accept connections
  (그뿐. "not properly shut down" 이 없다)
 ```
 **crash recovery 가 돌지 않았다 = 깨끗하게 내려갔다.**
 | 신호 | PostgreSQL 의 반응 |
 |---|---|
 | **SIGTERM** | **fast shutdown** — 진행 중 트랜잭션을 롤백하고 **WAL 을 플러시**한 뒤 종료 |
 | SIGINT | smart shutdown — 연결이 끊기길 기다린다 |
 | **SIGKILL** | **즉사** — 플러시 없음. 다음 기동에 crash recovery |
 `--force --grace-period=0` 는 API 오브젝트를 즉시 지우지만 컨테이너 런타임은
 여전히 정상 종료 절차를 밟는다. **PostgreSQL 은 SIGTERM 을 받고 얌전히
 플러시했다.**
 > **A-1 에서 배운 것이 또 나왔다** — 주입이 실제로 걸렸는지 먼저 확인하지
 > 않으면 **"아무 일도 없었다"를 결과로 착각한다.**
 > 여기서는 **crash recovery 메시지가 그 확인 수단**이다.
 ---
 ## 3. 실패한 주입 ② — 컨테이너 안에서 PID 1 은 SIGKILL 을 받지 않는다
 ```bash
 kubectl -n keycloak-lab exec deploy/postgres -- kill -9 1
 ```
 **아무 일도 일어나지 않았다.** 파드는 재시작하지 않았고 로그 시각도 그대로였다.
 ### 개념 — PID 1 의 시그널 보호
 리눅스 커널은 **PID 1 을 특별 취급**한다. 자기 PID 네임스페이스 안에서 온
 시그널은 **핸들러가 등록된 것만** 전달된다. **SIGKILL 도 예외가 아니다.**
 ```
   같은 네임스페이스 안에서  →  PID 1 은 등록하지 않은 시그널을 무시한다
   조상 네임스페이스에서     →  전달된다 (노드에서 kill -9 하면 죽는다)
 ```
 부팅 초기에 init 을 실수로 죽여 시스템이 멈추는 것을 막기 위한 장치인데,
 컨테이너에서는 **"안에서는 PID 1 을 못 죽인다"** 로 나타난다.
 ---
 ## 4. 성공한 주입 — 백엔드 프로세스를 죽인다
 PostgreSQL 은 **postmaster(부모) + 연결마다 백엔드(자식)** 구조다.
 자식 하나가 비정상 종료하면 **postmaster 는 공유 메모리가 오염됐다고 보고
 전체를 재초기화**한다. 그게 곧 crash recovery 다.
 ```bash
 kubectl -n keycloak-lab exec deploy/postgres -- \
  sh -c 'kill -9 $(pgrep -f "postgres: keycloak keycloak" | head -1)'
 ```
 ```
 server process (PID 40) was terminated by signal 9: Killed
 terminating any other active server processes
 all server processes terminated; reinitializing
 database system was not properly shut down; automatic recovery in progress
 redo starts at 0/23CAB68
 redo done at 0/2529E40
 checkpoint complete: wrote 113 buffers ...
 database system is ready to accept connections
 ```
 **이번엔 주입이 걸렸다.** `not properly shut down` + `redo` 가 증거다.
 파드는 재시작하지 않는다 (`restarts=0`) — 컨테이너의 PID 1 인 postmaster 는
 살아 있고, 자식만 갈아치운 것이다. **데이터 관점에서는 전원이 나간 것과 같다.**
 ---
 ## 5. 결과
 ```
 === 크래시 전후 대조 ===
  클라이언트가 200 과 토큰을 받은 로그인 : 153 건
  그중 DB 에 실제로 존재                 : 149 건
  ★ 유실                                 :   4 건
 === 유실된 sid ===
  ★ CQUfg9HLH29xvhiu6pVlfWOo  ← 토큰은 발급됐는데 세션이 없다
  ★ 5gLP4fqmpZBbjhH_d-0TPMMr
  ★ hkcOv1QskUFmYveMLB6Hljra
  ★ p5XybeQIYmAs818gO4Vl_5ea
 ```
 **약 2.6% 유실.** 초당 19건 정도 로그인하던 중이었으므로
 **대략 마지막 0.2초 분량**이다 — `wal_writer_delay` 기본값(200ms)과 맞는다.
 ### 사용자에게 어떻게 보이는가
 ```
   로그인 성공  →  access token + refresh token 을 받음
        │
        │  (크래시)
        ▼
   다음 요청     →  access token 은 60초간 통한다
        │            (서명만 보는 경로라면)
        ▼
   60초 후 refresh  →  "Session not active"  →  다시 로그인
 ```
 **즉시 드러나지 않는다.** access token 수명 동안은 정상으로 보이다가
 갱신 시점에 끊긴다. 장애와 증상 사이에 **최대 60초의 시차**가 있다.
 ---
 ## 6. 개념
 ### WAL 과 `synchronous_commit`
 ```
   COMMIT
     │
     ├─ WAL 버퍼(메모리)에 기록          ← 항상 한다
     │
     ├─ synchronous_commit = on  : 디스크 플러시를 기다렸다가 응답
     └─ synchronous_commit = off : 기다리지 않고 즉시 응답   ← Keycloak
                                     │
                                     └─ 크래시 시 이 구간이 사라진다
 ```
 | 설정 | 응답 속도 | 잃는 것 |
 |---|---|---|
 | `on` (PostgreSQL 기본) | 느리다 (디스크 대기) | 없다 |
 | **`off`** | 빠르다 | **최대 `wal_writer_delay` × 3 분량** |
 **전역 설정은 `on` 이었다.**
 ```
 전역 synchronous_commit: on
 ```
 **Keycloak 이 자기 트랜잭션에만 `SET LOCAL` 로 끈다.** DBA 가 서버 설정만
 보고 "우리는 동기 커밋"이라 믿으면 틀린다. **애플리케이션이 트랜잭션 단위로
 뒤집을 수 있다.**
 ### crash recovery
 ```
   기동 시 pg_control 을 읽는다
     └─ "깨끗하게 종료됨" 표시가 없다
          └─ "database system was not properly shut down"
               └─ 마지막 체크포인트부터 WAL 을 재생(redo)
                    └─ 디스크에 안 내려간 커밋은 복구할 수 없다  ← 손실
 ```
 `redo starts at 0/23CAB68` → `redo done at 0/2529E40` 사이가 재생된 구간이다.
 **WAL 에 없는 것은 재생할 수도 없다.**
 ### 이 손실이 "허용된" 이유
 Keycloak 의 판단은 이렇게 읽힌다.
 | | |
 |---|---|
 | 세션 쓰기는 **매우 잦다** | 로그인마다, refresh 마다 |
 | 잃어도 **회복 가능하다** | 사용자가 다시 로그인하면 된다 |
 | 동기 커밋의 비용은 **모든 요청에 붙는다** | 크래시는 드물다 |
 **드문 사고의 비용을 상시 지연으로 지불하지 않겠다는 선택**이다.
 합리적이지만, **선택했다는 사실을 알고 있어야 한다.**
 ---
 ## 7. 운영에 주는 것
 | 알게 된 것 | 함의 |
 |---|---|
 | 로그인도 비동기 커밋 | **RPO 가 0 이 아니다.** 크래시 시 마지막 수백 ms 로그인은 사라진다 |
 | 전역 `on` 인데 세션만 `off` | **서버 설정으로 판단하면 안 된다.** 애플리케이션이 뒤집는다 |
 | 손실이 즉시 안 보인다 | access token 수명만큼 시차. **모니터링은 갱신 실패율을 봐야 한다** |
 | `--grace-period=0` 은 크래시가 아니다 | **장애 훈련이 훈련이 안 될 수 있다** |
 | 컨테이너 안에서 PID 1 을 못 죽인다 | 크래시 재현은 **자식 프로세스**나 **노드에서** |
 ### 바꿀 수 있는가
 ```sql
 -- 세션 트랜잭션까지 동기 커밋으로 강제하려면 (지연 대가를 치른다)
 ALTER DATABASE keycloak SET synchronous_commit = on;   -- SET LOCAL 이 이깁니다
 ```
 **`SET LOCAL` 이 우선하므로 이것으로는 못 막는다.** Keycloak 설정이나
 소스 수준의 문제이며, **RPO 0 이 필요하면 복제(streaming replication)로
 푸는 것이 맞다** — 동기 스탠바이가 있으면 `synchronous_commit` 의 의미가
 달라진다.
 ---
 ## 8. 재현 절차 (명령어)
 ```bash
 # 0. 설계 확인 — 로그인도 비동기 커밋인지 먼저 본다
 kubectl -n keycloak-lab exec deploy/postgres -- psql -U keycloak -d keycloak \
  -c "alter system set log_statement='all'" -c "select pg_reload_conf()"
 #   → 로그인 1회 후 로그에서 "SET LOCAL synchronous_commit TO OFF" 확인
 # 1. 세션 테이블 비우기
 kubectl -n keycloak-lab exec deploy/postgres -- psql -U keycloak -d keycloak \
  -c "delete from offline_user_session"
 # 2. 로그인 루프 (호스트에서 백그라운드 exec — 파드 안 & 는 exec 종료와 함께 죽는다)
 kubectl -n keycloak-lab exec a2-probe -- sh -c '<로그인 반복, sid 를 /tmp/sids 에>' &
 # 3. 진짜 크래시 — 백엔드 프로세스에 SIGKILL
 kubectl -n keycloak-lab exec deploy/postgres -- \
  sh -c 'kill -9 $(pgrep -f "postgres: keycloak keycloak" | head -1)'
 # 4. 주입이 걸렸는지 확인 — 이게 없으면 결과를 해석하지 않는다
 kubectl -n keycloak-lab logs deploy/postgres | grep -E "not properly shut down|redo"
 # 5. 대조
 kubectl -n keycloak-lab exec deploy/postgres -- psql -U keycloak -d keycloak -tAc \
  "select count(*) from offline_user_session where user_session_id in (<sid 목록>)"
 ```
 ---
 ## 9. 다음 실험에 남기는 것
 | 실험 | 이 실험이 준 것 |
 |---|---|
 | **D-1** 백업·복구 | RPO 는 **백업 주기 + 이 손실**이다. 둘을 더해야 진짜 RPO |
 | **B-6** Redis 영속화 | `appendfsync everysec` 은 **같은 모양의 트레이드오프** |
 | **A-4** 노드 상실 | 노드가 죽으면 이것도 함께 일어난다 (postgres 가 kc-lab-2) |
 | 전체 | **주입 성공 신호를 미리 정한다.** 여기서는 crash recovery 로그 |
@@ -0,0 +1,397 @@
 # A-4 — 노드가 통째로 죽으면 어떻게 되는가
 브랜치 `feature/keycloak-a4-node-loss` ·
 증거 [`docs/evidence/a4-node-loss/`](evidence/a4-node-loss/) ·
 2026-09-04 12:07–12:24 KST
 선행: [`A-2`](experiment-a2-database-loss.md) · [`A-3`](experiment-a3-database-crash.md)
 **두 판본으로 나눠 실행했다.**
 | | 죽인 노드 | 성격 |
 |---|---|---|
 | **4a** | `kc-lab-2` (워커) | Keycloak + **DB 동시 상실** |
 | **4b** | `kc-lab-1` (k3s 서버) | **컨트롤 플레인 + 진입점 상실** |
 `virsh destroy` 는 종료 신호를 보내지 않는다. **전원을 뽑는 것과 같다.**
 ---
 ## 0. 결론부터
 | | 4a 워커 상실 | 4b 컨트롤 플레인 상실 |
 |---|---|---|
 | 외부 응답 | **503** | **000** (연결 자체가 안 됨) |
 | `kubectl` | 정상 | **불통** |
 | 살아 있는 워크로드 | keycloak-1 (하지만 DB 없음) | **keycloak-0 은 계속 돌고 있다** |
 | 복구 시간 | **60초** | **60초** |
 **둘 다 전면 장애**지만 이유가 다르다. 4a 는 **DB 가 같이 죽어서**, 4b 는
 **들어갈 길이 없어서**다.
 그리고 예상하지 못한 것 셋 —
 **죽은 파드가 산 파드보다 건강해 보이고**, **StatefulSet 은 대체 파드를 만들지
 않으며**, **PVC 때문에 재배치가 불가능하다.**
 ---
 ## 4a. 워커 노드 상실 (`kc-lab-2`)
 ### 기준선
 ```
 kc-lab-1   Ready   control-plane=true
 kc-lab-2   Ready   <none>
 keycloak-0   ready=true   kc-lab-2
 keycloak-1   ready=true   kc-lab-1
 postgres     ready=true   kc-lab-2
 PVC postgres-data → kc-lab-2        ← 재배치 가능성을 여기서 이미 알 수 있다
 외부 HTTP 200
 ```
 ### 주입
 ```bash
 virsh destroy kc-lab-2
 ```
 ### 관찰
 ```
  +15초  node=Ready     | keycloak-0=Running postgres=Running | 외부 HTTP 000
  +30초  node=Ready     | keycloak-0=Running postgres=Running | 외부 HTTP 000
  +45초  node=NotReady  | keycloak-0=Running postgres=Running | 외부 HTTP 503
  ...
  +180초 node=NotReady  | keycloak-0=Running postgres=Running | 외부 HTTP 503
 ```
 ### 발견 ① — 쿠버네티스가 알아채는 데 **40초** 걸린다
 `+15초`, `+30초` 에도 노드는 여전히 `Ready` 다. **기계는 이미 없는데.**
 kube-controller-manager 의 `node-monitor-grace-period` (기본 40초) 동안
 kubelet 의 하트비트가 없어야 `NotReady` 로 바꾼다.
 > **그 40초 동안 쿠버네티스는 거짓말을 한다.** 사용자는 이미 장애를 겪고 있다
 > (`000`). **노드 상태를 알림 근거로 삼으면 항상 늦는다.**
 ### 발견 ② — **죽은 파드가 산 파드보다 건강해 보인다**
 ```
 POD          PHASE     READY   NODE
 keycloak-0   Running   true    kc-lab-2      ← 기계가 꺼져 있다
 keycloak-1   Running   false   kc-lab-1      ← 살아 있다
 ```
 **`keycloak-0` 은 `ready=true`, `keycloak-1` 은 `ready=false`.**
 | 왜 | |
 |---|---|
 | keycloak-0 | kubelet 이 없어 **상태를 갱신할 수 없다.** 마지막으로 보고한 값이 그대로 얼어 있다 |
 | keycloak-1 | 살아서 **정직하게 보고한다** — DB 가 없으니 readiness 실패 |
 > **파드 상태는 "지금 어떤가"가 아니라 "마지막으로 그렇게 들었다"이다.**
 > 노드가 죽으면 그 노드 파드의 상태는 **화석**이 된다.
 Prometheus 는 정확했다.
 ```
  up{job=keycloak      pod=keycloak-1} = 1
  up{job=keycloak      pod=keycloak-0} = 0     ← 긁기에 실패했다 = 사실
  up{job=kubelet                     } = 0
  up{job=node-exporter node=kc-lab-2 } = 0
 ```
 **A-2 와 정반대다.** A-2 에서는 `up=1` 인데 서비스가 죽었고, 여기서는 `up=0`
 이 정확했다.
 | | `up` 이 잡는가 |
 |---|---|
 | **대상이 사라짐** (노드 상실) | **잡는다** |
 | **대상이 살아서 못 씀** (DB 상실) | **못 잡는다** |
 ![up 이 노드별로 0 으로 떨어졌다](evidence/a4-node-loss/a4-up-dropped-per-node.png)
 ### 발견 ③ — 축출은 **5분** 뒤에 시작된다
 쿠버네티스가 노드에 taint 를 붙인다.
 ```
  node.kubernetes.io/unreachable=:NoSchedule
  node.kubernetes.io/unreachable=:NoExecute
 ```
 그런데 모든 파드에는 기본 관용이 붙어 있다.
 ```
  node.kubernetes.io/not-ready    NoExecute  tolerationSeconds=300
  node.kubernetes.io/unreachable  NoExecute  tolerationSeconds=300
 ```
 ```
  +240초  전부 Running
  +270초  a2-probe:Terminating keycloak-0:Terminating postgres:Terminating
          postgres-...-9cmsv:Pending          ← 새 파드가 생겼다
 ```
 **노드가 죽고 약 5분이 지나야 파드가 축출된다.** 잠깐 끊긴 노드가 돌아올 수도
 있으니 성급하게 옮기지 않겠다는 설계다. 대신 **그동안은 아무 일도 일어나지
 않는다.**
 ### 발견 ④ — 축출돼도 **갈 곳이 없다**
 ```
 Warning  FailedScheduling  default-scheduler
  0/2 nodes are available:
    1 node(s) didn't match PersistentVolume's node affinity,
    1 node(s) had untolerated taint(s).
 ```
 ```
   kc-lab-2  →  taint 때문에 못 간다 (죽은 노드)
   kc-lab-1  →  PVC 가 kc-lab-2 에 묶여 있어 못 간다
 ```
 **`local-path` PVC 는 그 노드의 로컬 디스크 디렉터리다.** 노드가 죽으면
 볼륨도 죽는다. 새 파드는 **영원히 Pending** 이다.
 > 결함이 아니라 **조건**이다. 이 실험대는 그걸 알고 `local-path` 를 골랐다.
 > 운영이라면 네트워크 스토리지나 DB 복제가 이 자리를 메워야 한다.
 ### 발견 ⑤ — StatefulSet 은 **대체 파드를 만들지 않는다**
 ```
 NAME       DESIRED   READY     CURRENT
 keycloak   2         <none>    1
 keycloak-0   1/1   Terminating   30m       ← 30분째
 keycloak-1   0/1   Running
 ```
 **`keycloak-0` 이 `Terminating` 에서 영원히 멈춰 있고, 대체 파드가 안 생긴다.**
 | 왜 | |
 |---|---|
 | StatefulSet 의 계약 | **같은 이름의 파드는 클러스터에 하나뿐**이어야 한다 |
 | 컨트롤 플레인이 아는 것 | 노드가 안 보인다 = **파드가 죽었는지 확신할 수 없다** |
 | 그래서 | 옛 파드를 확실히 지우기 전엔 새 `keycloak-0` 을 못 만든다 |
 **Deployment 였다면 즉시 새 파드를 만든다.** 이름이 아무래도 되기 때문이다.
 StatefulSet 의 "안정된 이름"이라는 이득의 **반대편 비용**이 여기다.
 강제로 진행시키려면 사람이 개입해야 한다.
 ```bash
 kubectl -n keycloak-lab delete pod keycloak-0 --grace-period=0 --force
 ```
 **위험한 명령이다.** 노드가 사실은 살아 있고 네트워크만 끊긴 것이라면
 **같은 이름의 파드 둘이 동시에 존재**하게 된다 (split brain).
 ### 복구
 ```bash
 virsh start kc-lab-2
 ```
 ```
  +30초  node=Ready | Running 파드 3 개 | 외부 HTTP 503
  +60초  node=Ready | Running 파드 3 개 | 외부 HTTP 200
  → 서비스 복귀
 ```
 **60초.** 사람 개입 없이 전부 제자리로 돌아왔다.
 ---
 ## 4b. 컨트롤 플레인 노드 상실 (`kc-lab-1`)
 ### 이 노드에 무엇이 있었나 — 그게 곧 영향 범위다
 ```
 keycloak-lab    keycloak-1
 kube-system     coredns
 kube-system     local-path-provisioner
 kube-system     metrics-server
 kube-system     svclb-traefik
 kube-system     traefik                    ← replicas=1
 observability   grafana
 observability   prometheus
 observability   node-exporter
 ```
 ```
 NAME      REPLICAS   READY
 traefik   1          1        ← 진입점이 한 개다
 ```
 ### 주입과 관찰
 ```bash
 virsh destroy kc-lab-1
 ```
 ```
  +20초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server
  +60초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server
  +120초 외부 auth=000 grafana=502 | kubectl: Unable to connect to the server
  +160초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server
 ```
 ### 발견 ⑥ — 워크로드는 살아 있는데 **길이 없다**
 살아남은 노드에서 직접 확인했다.
 ```bash
 ssh kc-lab-2 'sudo crictl ps --name keycloak'
 ```
 ```
 CONTAINER      STATE     NAME       POD          NAMESPACE
 e5f777900b76   Running   keycloak   keycloak-0   keycloak-lab
 ```
 **`keycloak-0` 은 멀쩡히 돌고 있다.** 컨테이너 런타임(containerd)은 API 서버
 없이도 이미 떠 있는 파드를 계속 굴린다.
 ```
   죽은 것:  API 서버 · 스케줄러 · coredns · Traefik · Prometheus · Grafana
   산 것:    keycloak-0 · postgres · containerd
   문제:     들어갈 문(Traefik)이 없다
 ```
 > **컨트롤 플레인 상실은 워크로드 상실이 아니다.**
 > 이미 떠 있는 것은 계속 돈다. **새로 뜨거나 옮기거나 고치는 것이 안 될 뿐.**
 ### 발견 ⑦ — `000` 과 `503` 의 차이
 | 코드 | 뜻 |
 |---|---|
 | **`503`** (4a) | nginx·Traefik 은 살아 있고 **뒤에 보낼 파드가 없다** |
 | **`000`** (4b) | curl 이 응답 자체를 못 받았다 — **Traefik 이 없어 nginx 가 죽은 주소를 기다리다 타임아웃** |
 `grafana=502` 가 한 번 찍힌 것이 증거다 — nginx 는 살아서 502 를 만들 수
 있었지만, 대부분은 타임아웃(`--max-time 8`)에 걸렸다.
 **두 코드가 다른 층의 고장을 가리킨다.**
 ### 발견 ⑧ — 관측 시스템이 같이 죽으면 **0 이 아니라 구멍이 남는다**
 Prometheus 가 `kc-lab-1` 에 있었다. 그 노드가 죽은 12:18–12:23 구간은
 그래프에 **0 이 아니라 데이터 없음**으로 남는다.
 ```
   대상이 죽음        →  up = 0        →  "언제 죽었는지" 알 수 있다
   관측자가 죽음      →  데이터 없음    →  "그때 무슨 일이 있었는지" 모른다
 ```
 **계획 단계에서 "관측 스택은 죽이지 않는 노드에 둔다"고 정했던 이유**를
 실제로 확인한 셈이다. 다만 노드가 둘뿐이라 완전히 피할 수는 없다.
 ### 복구
 ```bash
 virsh start kc-lab-1
 ```
 ```
  +30초  외부=502 | kc-lab-1=Ready kc-lab-2=Ready
  +60초  외부=200 | kc-lab-1=Ready kc-lab-2=Ready
  → 서비스 복귀 (총 60초)
 ```
 **Grafana 는 로그인 세션을 잃었다** — 데이터가 `emptyDir` 이라 파드 재시작에
 사라진다. Prometheus 는 PVC 라 지표가 남았다. **의도한 설계대로 동작했다.**
 ---
 ## 5. 개념
 ### `node-monitor-grace-period` 와 `tolerationSeconds`
 ```
   기계 정지
      │
      │  40초   node-monitor-grace-period      → 노드 NotReady
      │
      │  +300초 tolerationSeconds (NoExecute)  → 파드 축출 시작
      │
      ▼
   총 약 5분 40초 동안 쿠버네티스는 아무것도 하지 않는다
 ```
 **빠른 장애 대응이 필요하면 이 값을 줄여야 하지만**, 짧게 하면 일시적
 네트워크 끊김에도 파드를 옮기게 되어 **불안정해진다.** 맞바꿈이다.
 ### `Terminating` 이 안 끝나는 이유
 ```
   파드 삭제 요청
     └─ kubelet 이 컨테이너를 멈추고 "지웠다"고 보고해야 끝난다
          └─ kubelet 이 없다 → 보고가 없다 → 영원히 Terminating
 ```
 `--grace-period=0 --force` 는 **보고를 기다리지 않고 API 에서 지우는 것**이다.
 컨테이너가 실제로 죽었는지는 **모른 채** 진행한다.
 ### StatefulSet vs Deployment — 노드 상실에서
 | | Deployment | StatefulSet |
 |---|---|---|
 | 대체 파드 | **즉시 만든다** (이름 무관) | **안 만든다** (이름이 유일해야) |
 | 노드 상실 시 | 자동 회복 | **사람이 강제 삭제해야** |
 | 위험 | 없음 | 강제 삭제 시 **중복 실행 가능** |
 **A-0 에서 "로그 대조가 쉬워서" StatefulSet 을 골랐는데, 그 대가가 여기 있다.**
 ### `local-path` PVC 와 재배치
 ```
   PVC → PV → nodeAffinity: kubernetes.io/hostname = kc-lab-2
                              └─ 그 노드의 /var/lib/rancher/k3s/storage/... 디렉터리
 ```
 **볼륨이 노드에 못박히면 파드도 못박힌다.** 노드가 죽으면 데이터도 함께
 접근 불가가 된다.
 ---
 ## 6. 재현 절차 (명령어)
 ```bash
 export LIBVIRT_DEFAULT_URI=qemu:///system
 # 사전 — PVC 가 어디 묶여 있는지 먼저 본다 (재배치 가능성이 여기서 정해진다)
 kubectl -n keycloak-lab get pvc -o name | while read p; do
  V=$(kubectl -n keycloak-lab get $p -o jsonpath='{.spec.volumeName}')
  kubectl get pv $V -o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]}'
 done
 # 4a 워커 상실
 virsh destroy kc-lab-2
 kubectl get node kc-lab-2                        # 40초 뒤 NotReady
 kubectl -n keycloak-lab get pods -o wide         # Running 인 채로 얼어 있다
 kubectl get node kc-lab-2 -o jsonpath='{.spec.taints}'
 #   5분 뒤 Terminating + 새 파드 Pending
 kubectl -n keycloak-lab describe pod <new-pod> | grep -A4 Events
 # 4b 컨트롤 플레인 상실 — kubectl 이 죽으므로 노드에서 직접 본다
 virsh destroy kc-lab-1
 ssh kc-lab-2 'sudo crictl ps'                    # 워크로드는 살아 있다
 # 복구
 virsh start kc-lab-2 && virsh start kc-lab-1
 ```
 ---
 ## 7. 다음 실험에 남기는 것
 | 실험 | 이 실험이 준 것 |
 |---|---|
 | **A-5** 비대칭 파티션 | 여기서는 노드가 **완전히** 사라졌다. 부분 단절은 더 고약하다 |
 | **D-1** 백업·복구 | **PVC 가 노드에 묶여 있다** — 노드가 영영 안 돌아오면 백업이 유일한 길 |
 | 구성 개선 | **Traefik `replicas=1` 은 진입점 단일 장애점** — 2로 늘리거나 DaemonSet 으로 |
 | 구성 개선 | **Grafana 가 `emptyDir`** — 재시작마다 로그인 세션이 사라진다 |
@@ -0,0 +1,319 @@
 # A-5 — 한쪽 방향만 끊기면 어떻게 되는가 (split brain)
 브랜치 `feature/keycloak-a5-asymmetric-partition` ·
 증거 [`docs/evidence/a5-asymmetric-partition/`](evidence/a5-asymmetric-partition/) ·
 2026-09-04 12:28–12:46 KST
 선행: [`A-1`](experiment-a1-jgroups-transport-block.md) — 이 실험은 A-1 이
 열어둔 질문에 답한다.
 > **A-1 의 열린 질문** — *"양쪽이 동시에 NotReady 가 되는 경로가 있다면
 > 전면 장애다. A-5 에서 이어서 본다."*
 ---
 ## 0. 결론부터
 | 물음 | 답 |
 |---|---|
 | 비대칭(한 방향) 차단이 클러스터를 가르는가 | **아니다.** 열린 방향으로 재연결한다 |
 | 양방향 완전 차단은 가르는가 | **그렇다.** 양쪽 모두 멤버 1개 |
 | **그때 양쪽이 다 NotReady 가 되는가** | **아니다. 한쪽만 DOWN 이다** |
 | 그래서 서비스는 | **계속된다.** 외부 200 유지 |
 **전면 장애 경로는 없었다.** 코디네이터 쪽이 항상 살아남는다.
 그리고 주입을 **세 번 실패**했다. 세 번 모두 다른 이유였고, 셋 다
 **"아무 일도 없었다"로 보였다.**
 ---
 ## 1. 주입을 세 번 실패했다
 A-1 에서 NetworkPolicy 가 conntrack 의 `ESTABLISHED` 승인에 막혀 기존 연결을
 못 끊는다는 것을 배웠다. 그래서 이번에는 iptables 로 갔는데, **또 다른 벽이
 세 개** 있었다.
 ### 실패 ① — `iptables -I FORWARD 1` 은 최상단에 남지 않는다
 ```bash
 ssh kc-lab-2 'sudo iptables -I FORWARD 1 -p tcp -d 10.42.1.77 --dport 7800 -j DROP'
 ```
 ```
 num   pkts bytes target
 1      232  377K KUBE-ROUTER-FORWARD  /* kube-router netpol */   ← 다시 1번이 되었다
 2        0     0 DROP  tcp dpt:57800
 3        0     0 DROP  tcp dpt:7800                              ← 0 패킷
 ```
 **kube-router 가 주기적으로 자기 체인을 최상단에 다시 삽입한다.**
 내가 1번에 넣어도 곧 2번, 3번으로 밀려난다.
 > **직접 넣은 iptables 규칙은 CNI 가 관리하는 체인과 경쟁한다.**
 > 넣는 것으로 끝이 아니라 **패킷 카운터로 확인해야 한다.**
 ### 실패 ② — 그래도 0 패킷: **연결 방향을 잘못 짚었다**
 `raw` 테이블로 옮겼는데도 0 패킷이었다.
 ```bash
 ssh kc-lab-2 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d 10.42.1.77 --dport 7800 -j DROP'
 ```
 실제 연결을 봤더니
 ```
 ESTABLISHED src=10.42.1.77 dst=10.42.0.42 sport=60485 dport=7800
            ────────────            ────────────────────
            keycloak-0 가 클라이언트   keycloak-1 이 서버
 ```
 **A-1 때와 방향이 반대였다.** 파드가 재시작되면서 누가 먼저 연결을 걸었는지가
 바뀌었다.
 > **JGroups 의 TCP 연결 방향은 고정이 아니다.** 먼저 뜬 쪽, 먼저 JOIN 을 건
 > 쪽에 따라 달라진다. **가정하지 말고 `conntrack -L` 로 봐야 한다.**
 ### 성공 — `raw` 테이블 PREROUTING, 수신측 노드에
 ```bash
 ssh kc-lab-1 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d 10.42.0.42 --dport 7800  -j DROP
              sudo iptables -t raw -I PREROUTING 1 -p tcp -d 10.42.0.42 --dport 57800 -j DROP'
 ```
 ```
   pkts bytes target
     19  1096 DROP  tcp dpt:57800
     21  3058 DROP  tcp dpt:7800     ← 걸린다
 ```
 ### 개념 — netfilter 처리 순서
 ```
   패킷 도착
     │
     ├─▶ raw     PREROUTING     ← conntrack 보다 먼저. NOTRACK·DROP 용
     │
     ├─▶ conntrack 조회/생성    ← 여기서 ESTABLISHED 가 결정된다
     │
     ├─▶ mangle  PREROUTING
     ├─▶ nat     PREROUTING
     ├─▶ filter  FORWARD        ← NetworkPolicy·kube-router 가 여기 있다
     └─▶ 목적지 파드
 ```
 | 어디에 넣는가 | 기존 연결을 끊는가 | CNI 와 경쟁하는가 |
 |---|---|---|
 | NetworkPolicy (filter) | **못 끊는다** — conntrack 이 먼저 통과시킨다 | 없음 |
 | filter FORWARD 직접 | 순서에 따라 | **경쟁한다** (kube-router 가 밀어낸다) |
 | **raw PREROUTING** | **끊는다** | **없다** — CNI 가 안 쓰는 테이블 |
 **진짜 네트워크 분단을 흉내내려면 `raw` 테이블이 맞다.**
 ---
 ## 2. 비대칭 차단 — 클러스터가 갈라지지 않았다
 한 방향(`→ keycloak-1:7800`)만 막고 관찰했다.
 ```
  +25초  keycloak-0:1/1 keycloak-1:1/1 | 외부 200
  +75초  keycloak-0:1/1 keycloak-1:0/1 | 외부 200      ← 잠깐 흔들림
  +100초 keycloak-0:1/1 keycloak-1:1/1 | 외부 200      ← 회복
  ...
  +200초 keycloak-0:1/1 keycloak-1:1/1 | 외부 200
 ```
 **주입 이후 뷰 변화가 하나도 없었다.**
 ```bash
 kubectl -n keycloak-lab logs keycloak-0 --since=20m | grep ISPN000094 | awk '$2 >= "03:33:58"'
 # (아무것도 안 나온다)
 ```
 ```
  뷰 ID 이력:  |13] (2)  ← 주입 전부터 지금까지 그대로
  suspected  = 0
 ```
 ### 왜 안 갈라졌는가 — **연결 방향이 뒤집혔다**
 ```
 차단 전:  keycloak-0 → keycloak-1:7800     ← 내가 막은 방향
 차단 후:  keycloak-1 → keycloak-0:7800     ← 열린 방향으로 다시 붙었다
 ```
 ```
 ESTABLISHED src=10.42.0.42 dst=10.42.1.77 sport=48473 dport=7800
            ────────────            ─────────────
            keycloak-1 이 클라이언트   keycloak-0 이 서버
 ```
 **JGroups 는 막힌 연결이 죽자 반대 방향으로 새로 연결했고, 그 사이 FD_SOCK2 가
 상대를 의심하기 전에 복구가 끝났다.** `suspected = 0` 이 그 증거다.
 > **한 방향만 막는 것으로는 JGroups 를 가를 수 없다.**
 > 두 노드는 서로에게 연결을 걸 수 있으므로, **한쪽 길이 막히면 다른 길로 간다.**
 >
 > 운영적으로는 좋은 소식이다 — 단방향 방화벽 오설정은 **자가 치유**된다.
 > 반대로 **분단을 재현하려는 실험자에게는 함정**이다.
 ---
 ## 3. 양방향 차단 — 갈라졌지만 전면 장애는 아니다
 양쪽 노드 모두에 규칙을 넣었다.
 ```
  +75초   keycloak-0:1/1 keycloak-1:1/1 | ready=[10.42.0.42 10.42.1.77] 외부 200
  +100초  keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77]            외부 200
  ...
  +225초  keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77]            외부 200
 ```
 ```
 === 뷰 ===
  keycloak-0  [keycloak-0-24309|14] (1) [keycloak-0-24309]
  keycloak-1  [keycloak-1-45480|14] (1) [keycloak-1-45480]
 === JGROUPS_PING ===
 keycloak-0-24309 | 10.42.1.77:7800 | t     ← 코디네이터
 keycloak-1-45480 | 10.42.0.42:7800 | t     ← 코디네이터
 ```
 **`coord = t` 가 둘.** 완전한 split brain 이다.
 ![cluster_size 가 갈라졌다 합쳐진다](evidence/a5-asymmetric-partition/a5-cluster-size-bidirectional-block.png)
 ### 그런데 **한쪽만 DOWN 이다**
 ```
 --- keycloak-0 ---
 {"status":"UP",   ... {"name":"Keycloak cluster health check","status":"UP"}
 --- keycloak-1 ---
 {"status":"DOWN", ... (cluster health check 가 DOWN)
 ```
 | | keycloak-0 | keycloak-1 |
 |---|---|---|
 | 분단 전 역할 | **코디네이터** (뷰 13 의 발행자) | 일반 멤버 |
 | 분단 후 자기 인식 | "멤버가 하나 나갔다" — **정상 사건** | "코디네이터를 잃었다" — **비정상** |
 | 헬스체크 | **UP** | **DOWN** |
 | Service 엔드포인트 | **남는다** | 빠진다 |
 > **A-1 의 열린 질문에 대한 답 — 양쪽이 동시에 NotReady 가 되는 경로는 없었다.**
 >
 > Keycloak 의 클러스터 헬스체크는 **비대칭**이다. 코디네이터였던 쪽은
 > 자기가 정상이라고 보고, 잃은 쪽만 DOWN 이 된다. 그래서 완전 분단조차
 > **용량 저하로 끝나고 전면 장애가 되지 않는다.**
 >
 > A-2(DB 상실)에서 양쪽이 동시에 DOWN 이 된 것과 대조된다. **DB 는 모두가
 > 의존하는 하나지만, 클러스터 멤버십은 서로 상대적**이기 때문이다.
 ---
 ## 4. 복구
 ```bash
 ssh kc-lab-1 'sudo iptables -t raw -F PREROUTING'
 ssh kc-lab-2 'sudo iptables -t raw -F PREROUTING'
 ```
 ```
  +25초  keycloak-0:1/1 keycloak-1:0/1
  +50초  keycloak-0:1/1 keycloak-1:1/1
  → 복구 완료
 MergeView::[keycloak-0-24309|15] (2) [keycloak-0-24309, keycloak-1-45480]
 ```
 **50초, 사람 개입 없음.** `MergeView` 로 뷰 15 가 발행되며 병합됐다.
 각 캐시마다 재분배 로그가 남는다.
 ```
 [Context=work]            ISPN100007: After merge (or coordinator change) ...
 [Context=clientSessions]  ISPN100007: After merge ...
 [Context=offlineSessions] ISPN100007: After merge ...
 [Context=loginFailures]   ISPN100007: After merge ...
 [Context=actionTokens]    ISPN100007: After merge ...
 ```
 **`ISPN100007` 은 병합 후 캐시별 토폴로지 재계산**이다. 캐시가 여럿이므로
 로그도 캐시 수만큼 나온다.
 ---
 ## 5. 개념
 ### `MergeView` 와 뷰 ID
 ```
 [keycloak-0-24309|13] (2)   ← 정상
 [keycloak-0-24309|14] (1)   ← 분단. 양쪽이 각자 14 를 발행
 MergeView::[...|15] (2)     ← 병합. 뷰 ID 는 계속 증가한다
 ```
 **뷰 ID 는 단조 증가**하므로 "언제 몇 번 갈라졌는지"를 로그만으로 셀 수 있다.
 ### 코디네이터의 비대칭성
 JGroups 코디네이터는 **가장 오래된 멤버**다. 분단이 나면
 ```
   코디네이터 쪽:  "멤버가 나갔다"        → 정상 처리, 계속 코디네이터
   나머지 쪽:      "코디네이터가 사라졌다"  → 새 코디네이터를 자기로 선출
 ```
 **둘 다 자기가 코디네이터라고 믿는 상태**가 split brain 이고,
 `JGROUPS_PING.coord` 컬럼에 그대로 드러난다.
 ### 실험자를 위한 규칙
 | 상황 | 확인 방법 |
 |---|---|
 | 규칙을 넣었는데 안 걸린다 | `iptables -L -n -v` 의 **패킷 카운터** |
 | 방향을 모르겠다 | `conntrack -L \| grep <포트>` |
 | CNI 가 밀어낸다 | `raw` 테이블을 쓴다 |
 | 갈라졌는지 알고 싶다 | `vendor_cluster_size`, `ISPN000094`, `JGROUPS_PING.coord` |
 ---
 ## 6. 재현 절차 (명령어)
 ```bash
 # 1. 지금 연결이 어느 방향인지 먼저 본다 — 가정하면 실패한다
 ssh kc-lab-1 'sudo conntrack -L | grep 7800'
 # 2. 수신측 노드의 raw PREROUTING 에 넣는다 (filter 는 CNI 와 경쟁한다)
 ssh kc-lab-1 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d <수신 파드IP> --dport 7800  -j DROP'
 ssh kc-lab-1 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d <수신 파드IP> --dport 57800 -j DROP'
 # 3. 걸렸는지 카운터로 확인 — 0 이면 해석 금지
 ssh kc-lab-1 'sudo iptables -t raw -L PREROUTING -n -v'
 # 4. 양방향으로 하려면 반대 노드에도 (한 방향만으로는 자가 치유된다)
 ssh kc-lab-2 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d <반대 파드IP> --dport 7800 -j DROP'
 # 5. 분단 확인
 kubectl -n keycloak-lab exec deploy/postgres -- psql -U keycloak -d keycloak \
  -c "select name, coord from jgroups_ping"          # coord=t 가 둘이면 split brain
 kubectl -n keycloak-lab get endpoints keycloak -o jsonpath='{.subsets[*].addresses[*].ip}'
 # 6. 해제
 ssh kc-lab-1 'sudo iptables -t raw -F PREROUTING'
 ssh kc-lab-2 'sudo iptables -t raw -F PREROUTING'
 ```
 ---
 ## 7. 다음 실험에 남기는 것
 | 실험 | 이 실험이 준 것 |
 |---|---|
 | **A-6** 지연 주입 | `tc netem` 도 같은 함정 — **주입이 걸렸는지 먼저 확인** |
 | **A-7** volatile 비교 | 여기서는 분단에도 서비스가 계속됐다. volatile 이면 **세션이 갈라진다** |
 | 운영 | **단방향 방화벽 오설정은 자가 치유된다.** 양방향이어야 사고가 된다 |
 | 운영 | **완전 분단조차 전면 장애가 아니다** — 코디네이터 쪽이 살아남는다 |
@@ -0,0 +1,338 @@
 # A-6 — 끊기지 않고 느려지기만 하면 어떻게 되는가
 브랜치 `feature/keycloak-a6-latency-injection` ·
 증거 [`docs/evidence/a6-latency-injection/`](evidence/a6-latency-injection/) ·
 2026-09-04 13:10–13:35 KST
 **실제 장애의 대부분은 완전 사망이 아니라 느려짐이다.** 그리고 느려짐은
 사망보다 진단하기 어렵다 — 헬스체크가 통과하기 때문이다.
 ---
 ## 0. 결론부터
 | 측정 | 값 |
 |---|---|
 | 주입한 네트워크 지연 | **200 ms** |
 | 로그인 응답 시간 | **66 ms → 1,872 ms (28배)** |
 | 동시 20건에서 최대 응답 | **22.2 초** |
 | 커넥션 획득 대기 최대 | **20,000 ms** |
 | readiness 프로브 | **타임아웃으로 실패** |
 **200밀리초가 22초가 됐다.** 지연은 **왕복 횟수만큼 곱해지고**, 커넥션 풀에서
 **한 번 더 곱해진다.**
 ---
 ## 1. 설계 — 한 실험에서 A/B 가 되는 배치
 ```
   kc-lab-2                        kc-lab-1
   ┌──────────────────┐            ┌──────────────────┐
   │ postgres         │            │ keycloak-1       │
   │ keycloak-0       │            │                  │
   │   └─ cni0 로 직행 │◀─ VXLAN ──▶│   └─ 오버레이 경유 │
   └──────────────────┘            └──────────────────┘
        지연 없음                        여기만 느려진다
 ```
 **postgres 가 보내는 패킷만** 지연시키면 `keycloak-1` 의 DB 접근만 느려지고
 `keycloak-0` 은 그대로다. **대조군이 같은 실험 안에 있다.**
 기준선은 거의 같았다.
 ```
  keycloak-0  평균 70 ms
  keycloak-1  평균 66 ms
 ```
 ---
 ## 2. 주입을 두 번 실패했다
 ### 실패 ① — `eth0` 이라는 인터페이스가 없다
 ```bash
 ssh kc-lab-2 'sudo tc qdisc add dev eth0 root handle 1: prio'
 ```
 ```
 Cannot find device "eth0"
 ```
 Debian 클라우드 이미지는 **예측 가능한 인터페이스 이름**을 쓴다.
 ```
 enp1s0    UP    52:54:00:aa:bb:12
 ```
 `en` (ethernet) + `p1` (PCI bus 1) + `s0` (slot 0). 이름이 하드웨어 위치에서
 나오므로 **NIC 순서가 바뀌어도 이름이 안 바뀐다.**
 ### 실패 ② — `enp1s0` 에서는 **파드 IP 가 보이지 않는다**
 여기가 핵심이다. 노드 간 파드 통신은 **flannel VXLAN 으로 캡슐화**된다.
 ```
   원래 패킷:   src=10.42.1.76(postgres)  dst=10.42.0.42(keycloak-1)
        │
        ▼  flannel.1 에서 캡슐화
   실제 패킷:   src=192.168.122.12(노드)   dst=192.168.122.11(노드)  UDP 8472
                └─ 안쪽에 원래 패킷이 통째로 들어 있다
        │
        ▼
   enp1s0 로 나간다
 ```
 **`enp1s0` 에서 `match ip src 10.42.1.76` 은 절대 일치하지 않는다.**
 그 IP 는 페이로드 안에 있고, 헤더에는 노드 IP 만 있다.
 ### 성공 — `flannel.1` 에 건다
 ```bash
 ssh kc-lab-2 '
  sudo tc qdisc add dev flannel.1 root handle 1: prio
  sudo tc qdisc add dev flannel.1 parent 1:3 handle 30: netem delay 200ms
  sudo tc filter add dev flannel.1 protocol ip parent 1:0 prio 3 \
       u32 match ip src 10.42.1.76/32 flowid 1:3'
 ```
 **`flannel.1` 은 캡슐화 직전 단계**이므로 여기서는 파드 IP 가 보인다.
 ### 검증 — 카운터로 확인한다
 ```
 qdisc netem 30: parent 1:3 limit 1000 delay 200ms
 Sent 18388 bytes 150 pkt (dropped 0, overlimits 0 requeues 0)
                  ───────
                  실제로 지연 밴드를 통과했다
 ```
 > **A-1·A-5 와 같은 교훈이 세 번째로 나왔다.**
 > 주입을 넣은 것과 걸린 것은 다르다. **카운터를 봐야 한다.**
 ### 개념 — `tc` 의 계층 구조
 ```
   qdisc (큐 규율)     인터페이스에 붙는 패킷 스케줄러
     ├─ prio           우선순위 밴드 3개로 나눈다
     │    ├─ 1:1 (기본)
     │    ├─ 1:2 (기본)
     │    └─ 1:3  ← 여기에 netem 을 붙인다
     └─ filter         어떤 패킷을 어느 밴드로 보낼지
 ```
 `netem` 을 root 에 바로 붙이면 **모든 트래픽**이 느려진다.
 `prio` + `filter` 를 쓰면 **고른 트래픽만** 느려진다.
 ---
 ## 3. 결과 — 지연은 곱해진다
 ```
 === 두 노드 지연 비교 (기준선: k0=70ms k1=66ms) ===
  keycloak-0  평균   41 ms   최대   57 ms      ← 영향 없음
  keycloak-1  평균 1872 ms   최대 1887 ms      ← 28배
 ```
 ### 왜 200ms 가 1,872ms 가 되는가
 A-0 에서 잡은 로그인 트랜잭션의 SQL 이 답이다.
 ```
 BEGIN
 select ... from OFFLINE_USER_SESSION ...
 select VERSION ... for no key update skip locked
 select ... from OFFLINE_CLIENT_SESSION ...
 select VERSION ... for no key update skip locked
 insert into OFFLINE_USER_SESSION ...
 insert into OFFLINE_CLIENT_SESSION ...
 SET LOCAL synchronous_commit TO OFF
 COMMIT
 ```
 **왕복이 아홉 번이다.**
 ```
   200 ms × 9 왕복 ≈ 1,800 ms      실측 1,872 ms
 ```
 > **네트워크 지연은 왕복 횟수만큼 증폭된다.**
 > "DB 가 200ms 느려졌다"는 "애플리케이션이 200ms 느려졌다"가 아니다.
 > **쿼리 수를 줄이는 것이 지연 환경에서 결정적인 이유**가 이것이다.
 ---
 ## 4. 동시 부하 — 여기서 진짜 고장이 난다
 동시 20건을 `keycloak-1` 에 보냈다.
 ```
  200  1.911    200  1.913    200  1.958    200  1.981
  200  3.441    200  4.841    200  6.257    200  7.704
  200  9.104    200 10.539    200 11.951    200 13.351
  200 14.785    200 16.189    200 17.625    200 19.053
  200 20.495    200 21.905    200 22.228    200 22.230
 ```
 **전부 성공(200)했지만 응답 시간이 1.9초에서 22.2초까지 계단으로 늘어난다.**
 ```
   1.9 → 3.4 → 4.8 → 6.2 → 7.7 → 9.1 → 10.5 → ...
        ────  ────  ────  ────
        약 1.4초 간격 — 앞 요청이 커넥션을 놓아줄 때까지 줄을 선다
 ```
 **전형적인 큐잉이다.** 커넥션 수는 유한하고, 각 요청이 커넥션을 1.9초씩
 붙잡고 있으므로 뒤에 온 요청은 그만큼 기다린다.
 ### 커넥션 풀 지표가 증언한다
 ```
  agroal_blocking_time_max_milliseconds   20000.0    ← 20초를 기다린 요청이 있다
  agroal_blocking_time_average_milliseconds  281.0
  agroal_max_used_count                      19.0    ← 풀이 19개까지 늘었다
  agroal_acquire_count_total                672.0
  agroal_active_count                         0.0    ← 부하가 끝나 지금은 0
 ```
 ![커넥션 획득 대기 시간](evidence/a6-latency-injection/a6-connection-pool-blocking.png)
 ### 그리고 헬스체크가 무너진다
 ```
 Warning  Unhealthy  pod/keycloak-1  Readiness probe failed:
  Get "http://10.42.0.42:9000/health/ready": context deadline exceeded
 ```
 **readiness 프로브 자체가 타임아웃됐다.**
 ### 연쇄 고장의 모양
 ```
   DB 가 느려진다
        ↓
   요청이 커넥션을 오래 붙잡는다
        ↓
   커넥션 풀이 고갈된다
        ↓
   새 요청이 줄을 선다 (최대 20초)
        ↓
   헬스체크도 줄에 선다 → 타임아웃 → NotReady
        ↓
   그 노드가 로드밸런서에서 빠진다
        ↓
   ★ 남은 노드로 트래픽이 몰린다 → 그 노드도 같은 길을 간다
 ```
 **마지막 화살표가 무서운 부분이다.** 느려짐은 **전파된다.**
 A-2(DB 완전 정지)는 즉시 503 으로 드러나 오히려 명확했지만,
 **느려짐은 살아 있는 노드를 하나씩 무너뜨린다.**
 ---
 ## 5. 빗나간 예측 — 낙관적 락 충돌은 늘지 않았다
 계획서에 이렇게 적었다.
 > **낙관적 락 충돌 증가** — 트랜잭션이 길어져 `VERSION` 충돌이 늘어야 한다
 ```
  관련 로그 줄수: 0
 ```
 **하나도 없었다.** 이유가 명확하다.
 ```
   로그인   →  매번 새 세션 행을 INSERT      →  다툴 상대가 없다
   refresh  →  같은 세션 행을 UPDATE         →  여기서 다툰다
 ```
 **충돌은 같은 행을 동시에 고칠 때만 일어난다.** 로그인 부하로는 재현되지
 않는다. 이건 **B-3(refresh 토큰 경쟁)의 영역**이며, 거기서 지연을 함께 주면
 충돌률이 올라갈 것이다.
 > 예측을 적어두지 않았다면 "충돌이 없네" 하고 넘어갔을 것이다.
 > **빗나간 예측이 다음 실험의 설계를 정해준다.**
 ---
 ## 6. 복구
 ```bash
 ssh kc-lab-2 'sudo tc qdisc del dev flannel.1 root'
 ```
 ```
  keycloak-0  평균 43 ms
  keycloak-1  평균 51 ms      ← 즉시 정상
 ```
 **파드 재시작 없이 즉시 회복.** 커넥션 풀도 스스로 정상화됐다.
 ---
 ## 7. 운영에 주는 것
 | 알게 된 것 | 함의 |
 |---|---|
 | 지연은 **왕복 횟수만큼 곱해진다** | DB 지연 대책은 "쿼리 수 줄이기"가 먼저다 |
 | 커넥션 풀에서 **한 번 더 곱해진다** | 풀 크기와 타임아웃이 장애 반경을 정한다 |
 | **헬스체크도 줄에 선다** | 프로브 타임아웃이 풀 대기보다 짧아야 격리가 제때 된다 |
 | 느려짐은 **전파된다** | 노드를 빼면 남은 노드가 더 빨리 무너진다 |
 | `up` 도 readiness 도 **늦게 반응** | **응답 시간 분포(p95/p99)를 봐야 한다** |
 ### 이 실험대에 없는 알림
 지금 관측 스택에는 **지연 분포 지표가 없다.** `agroal_blocking_time_*` 은
 있지만 히스토그램이 아니라 평균/최대뿐이다.
 ```promql
 # 있으면 좋았을 것
 histogram_quantile(0.99, rate(http_server_requests_seconds_bucket[5m]))
 ```
 **A-2 에서 `kube-state-metrics` 가 빠진 것을 찾았고, 여기서는 응답 시간
 히스토그램이 빠진 것을 찾았다.** 둘 다 보완 항목이다.
 ---
 ## 8. 재현 절차 (명령어)
 ```bash
 # 1. 인터페이스 이름 확인 — eth0 이 아니다
 ssh kc-lab-2 'ip -brief link show | grep -E "flannel|cni|enp"'
 # 2. 오버레이 인터페이스에 건다 (enp1s0 에서는 파드 IP 가 안 보인다)
 ssh kc-lab-2 '
  sudo tc qdisc add dev flannel.1 root handle 1: prio
  sudo tc qdisc add dev flannel.1 parent 1:3 handle 30: netem delay 200ms
  sudo tc filter add dev flannel.1 protocol ip parent 1:0 prio 3 \
       u32 match ip src <postgres 파드IP>/32 flowid 1:3'
 # 3. 걸렸는지 카운터로 확인 — Sent 가 0 이면 해석 금지
 ssh kc-lab-2 'sudo tc -s qdisc show dev flannel.1 | grep -A2 netem'
 # 4. 단일 요청 지연 (대조군은 같은 노드의 keycloak-0)
 kubectl -n keycloak-lab run t --rm -i --restart=Never --image=curlimages/curl:8.11.1 \
  --command -- curl -s -o /dev/null -w '%{time_total}\n' -X POST http://<pod>:8080/realms/master/protocol/openid-connect/token ...
 # 5. 동시 부하로 풀 고갈 재현
 #    ( curl ... ) & 를 20개 띄우고 wait
 # 6. 풀 지표
 curl -s http://<pod>:9000/metrics | grep -E '^agroal_(blocking_time|max_used|awaiting)'
 # 7. 해제
 ssh kc-lab-2 'sudo tc qdisc del dev flannel.1 root'
 ```
 ---
 ## 9. 다음 실험에 남기는 것
 | 실험 | 이 실험이 준 것 |
 |---|---|
 | **B-3** refresh 경쟁 | **지연을 함께 주면 낙관적 락 충돌이 재현될 것** — 여기서는 안 됐다 |
 | **B-1** 저장소 지연 (Q3 제약) | 같은 기법을 Redis 앞에 쓴다 |
 | 관측 보완 | **응답 시간 히스토그램**이 없다 |
 | 구성 | **프로브 타임아웃 < 커넥션 풀 대기**여야 격리가 제때 된다 |
@@ -0,0 +1,267 @@
 # A-7 — 옛 방식(volatile)이었다면 무엇이 달라지는가
 브랜치 `feature/keycloak-a7-volatile-comparison` ·
 증거 [`docs/evidence/a7-volatile-comparison/`](evidence/a7-volatile-comparison/) ·
 2026-09-04 13:45–14:15 KST
 **A층의 결론 전체가 "Keycloak 26 기본값"이라는 전제 위에 있다.**
 전제를 뒤집어 같은 실험을 반복한 것이 이 실험이다.
 ---
 ## 0. 결론부터 — 비교표
 | 실험 | persistent (KC 26 기본) | **volatile (KC 24 이전 방식)** |
 |---|---|---|
 | **A-0** 세션 저장 위치 | DB 에 행이 생긴다 | **DB 0건.** 메모리에만 |
 | **A-1** 7800 차단 후 교차 노드 refresh | **`200`** — 안 깨진다 | **`400 Session not active`** — 깨진다 |
 | **A-8** 롤링 재시작 후 refresh | **`200`** — 세션 생존 | **`400 Session not active`** — 전원 로그아웃 |
 | **A-2** DB 정지 중 새 로그인 | `500` | **`200`** — 된다 |
 | A-2 DB 정지 중 refresh | `500` | `500` |
 **세 개가 정반대로 뒤집혔다.** 예측한 그대로다.
 > **"세션 공유는 7800 을 안 탄다"는 A-1 의 결론은 버전에 달린 사실이다.**
 > 인터넷 자료 대부분이 24 이전 기준이므로 **거기서는 통념이 맞다.**
 > 틀린 것은 자료가 아니라 **버전을 확인하지 않고 적용하는 것**이다.
 ---
 ## 1. 전환 방법
 ```bash
 # 이 버전에서 끌 수 있는지부터 확인한다
 kubectl -n keycloak-lab exec keycloak-0 -- /opt/keycloak/bin/kc.sh build --help-all \
  | tr ',' '\n' | grep -i persistent
 #   persistent-user-sessions[:v1]      ← 목록에 있다
 ```
 ```yaml
 # deploy/lab/k8s/keycloak-cluster.yaml
 args: ["start", "--features-disabled=persistent-user-sessions"]
 ```
 ```bash
 kubectl apply -f deploy/lab/k8s/keycloak-cluster.yaml
 kubectl -n keycloak-lab rollout status statefulset/keycloak --timeout=500s
 ```
 **빌드 옵션이므로 기동 시 재빌드가 일어나 평소보다 오래 걸린다.**
 ---
 ## 2. A-0 재실행 — 세션이 DB 에 안 들어간다
 ```
 === keycloak-0 에만 로그인 5회 ===
  keycloak-0  sessions 캐시 5.0 건
  keycloak-1  sessions 캐시 0.0 건
 === DB 에는 들어갔는가 (persistent 였을 때는 5건) ===
 offline_flag | count
 --------------+-------
 (0 rows)                    ← 0 건
 === 교차 노드 세션은 되는가 ===
  keycloak-0 로그인 → keycloak-1 에서 refresh  HTTP 200
 ```
 **DB 는 비어 있는데 교차 노드가 된다.** persistent 때와 겉보기 결과가 같지만
 **경로가 완전히 다르다** — 이제는 DB 가 아니라 **클러스터를 타고** 있다.
 > 캐시 엔트리 수가 `5 / 0` 인 것은 persistent 때와 같다.
 > `approximate_entries_unique` 는 **그 노드가 소유한 엔트리**만 세므로,
 > 백업본을 들고 있어도 0 으로 보인다. **이 지표만으로는 두 모드를 구분할 수
 > 없다** — 구분하려면 7800 을 끊어봐야 한다. 그게 다음 절이다.
 ---
 ## 3. A-1 재실행 — **정반대다**
 A-5 에서 확립한 `raw` 테이블 양방향 차단을 그대로 썼다.
 ```
  [대조군] 차단 전 교차 노드 refresh   HTTP 200
  차단 적용 → 분단 성립
    +25초  cluster_size(k0 k1) = [2.0 2.0]
    +50초  cluster_size(k0 k1) = [1.0 ...]     ← 갈라졌다
 === 분단 상태에서 ===
  keycloak-0 로그인 → keycloak-0 에서 refresh  HTTP 200  ← 대조군
  keycloak-0 로그인 → keycloak-1 에서 refresh  HTTP 400  ← 시험군
  {"error":"invalid_grant","error_description":"Session not active"}
 ```
 **대조군이 200 인 것이 중요하다.** 차단이 모든 것을 망가뜨린 게 아니라
 **교차 노드만** 끊었다는 증거다.
 ```
   persistent :  세션 ── PostgreSQL ──▶ 양쪽이 본다      7800 무관
   volatile   :  세션 ── 클러스터(7800) ─▶ 상대에게 간다   7800 필수
 ```
 **같은 주입, 같은 관측, 정반대 결과.** 이 한 쌍이 A층 전체의 근거다.
 ---
 ## 4. A-8 재실행 — 배포가 곧 로그아웃
 ```
 === 재시작 전 로그인 ===
  sid = aVwYnzKZFFvMqD3bpSeiILuM
 === 롤링 재시작 ===
 partitioned roll out complete: 2 new pods have been updated...
 === ★ 재시작 전 토큰이 아직 통하는가 (persistent 였을 때는 200) ===
  keycloak-0 에서 refresh  HTTP 400
  {"error":"invalid_grant","error_description":"Session not active"}
 ```
 **배포할 때마다 전원 로그아웃된다.**
 | | persistent | volatile |
 |---|---|---|
 | 배포 | 자유롭다 | **모든 사용자가 다시 로그인** |
 | 파드 재시작 (OOM, 노드 교체) | 무해 | **그 노드가 처리하던 세션 소멸** |
 | 무중단 여부 | 무중단 (A-8) | 접속은 되지만 **로그인 상태가 사라진다** |
 > **A-8 에서 "이것이 persistent 를 켜는 진짜 이유"라고 썼는데, 여기서 증명된다.**
 > 24 이전 버전을 쓰는 곳에서 "배포하면 로그아웃된다"가 당연하게 여겨졌던 이유다.
 ---
 ## 5. A-2 재실행 — 예상 못 한 비대칭
 ```
  ① 캐시를 가진 노드에서 refresh   HTTP 500
  ② 새 로그인                      HTTP 200      ← persistent 에서는 500 이었다
 ```
 **새 로그인은 되는데 refresh 가 안 된다.** 순서가 거꾸로다.
 ### 왜 새 로그인이 되는가
 ```
   로그인에 필요한 것
     ├─ realm 설정   → Infinispan `realms` 캐시에 있다
     ├─ 사용자 자격  → `users` 캐시에 있다
     └─ 세션 저장    → volatile 이므로 메모리
   → DB 없이 완결된다
 ```
 **A-2 에서 persistent 로 했을 때 로그인이 실패한 이유는 "세션을 DB 에 써야
 해서"였다.** 그 쓰기가 없어지니 로그인이 통과한다.
 ### refresh 가 500 인 이유 — 가설
 **측정은 확실하지만 원인은 확정하지 못했다.** 유력한 후보는
 `REVOKED_TOKEN` 테이블이다 — refresh token 회전에서 **이미 쓴 토큰인지**
 확인하려면 그 테이블을 봐야 하고, 그 경로는 캐시되지 않는다.
 ```
 kubectl -n keycloak-lab exec deploy/postgres -- psql -U keycloak -d keycloak -c "\dt" | grep revoked
 public | revoked_token | table | keycloak
 ```
 **확정하려면 A-3 에서 쓴 문장 로깅을 켜고 다시 재현해야 한다.** 여기서는
 **가설로 남긴다.**
 > **volatile 이 "DB 없이 돌아간다"는 뜻은 아니다.**
 > realm·사용자·클라이언트·취소 토큰은 **여전히 DB 에 있다.**
 > 세션만 메모리로 옮긴 것이다.
 ---
 ## 6. 원복
 ```bash
 # args 를 되돌린다
 args: ["start"]
 kubectl apply -f deploy/lab/k8s/keycloak-cluster.yaml
 ```
 ```
 === [검증] persistent 로 돌아왔는가 ===
 ["start"]
  DB 온라인 세션: 1 건  (로그인 1회 후 → persistent 복귀 확인)
  외부 진입점 HTTP 200
 ```
 **전환 자체는 설정 한 줄이고 되돌리기도 한 줄이다.** 다만 전환 시점에
 **기존 세션은 전부 사라진다** (저장 위치가 바뀌므로).
 ---
 ## 7. 개념
 ### 두 모드의 데이터 흐름
 ```
   persistent (KC 25+, 26 기본)
      로그인 ──▶ PostgreSQL (진실)  +  로컬 캐시 (사본)
      조회   ──▶ 캐시에 없으면 DB
      공유   ──▶ 같은 DB 를 보는 것
   volatile (KC 24 이전)
      로그인 ──▶ Infinispan (진실)
      조회   ──▶ 클러스터에서 찾는다
      공유   ──▶ 7800 을 통한 복제
 ```
 ### 무엇을 맞바꾸는가
 | | persistent | volatile |
 |---|---|---|
 | 재시작 내구성 | **있다** | 없다 |
 | 7800 의존 | 낮다 (무효화만) | **높다 (세션 자체)** |
 | DB 부하 | **로그인·refresh 마다 쓰기** | 세션 관련 없음 |
 | 노드 확장 | DB 가 병목 | **복제 트래픽이 N² 로 증가** |
 | 지연 민감도 | **DB 왕복에 민감** (A-6) | 클러스터 왕복에 민감 |
 **26 이 기본을 바꾼 이유가 이 표에 있다** — 운영에서 가장 아픈 것이
 "배포하면 로그아웃"이었기 때문이다.
 ---
 ## 8. 재현 절차 (명령어)
 ```bash
 # 1. 끌 수 있는지 확인
 kubectl -n keycloak-lab exec keycloak-0 -- /opt/keycloak/bin/kc.sh build --help-all \
  | tr ',' '\n' | grep -i persistent
 # 2. 세션을 비우고 전환 (비교 기준을 맞추기 위해)
 kubectl -n keycloak-lab exec deploy/postgres -- psql -U keycloak -d keycloak \
  -c "delete from offline_user_session"
 #   args: ["start", "--features-disabled=persistent-user-sessions"]
 kubectl apply -f deploy/lab/k8s/keycloak-cluster.yaml
 kubectl -n keycloak-lab rollout status statefulset/keycloak --timeout=500s
 # 3. volatile 확인 — 로그인 후 DB 가 비어 있어야 한다
 kubectl -n keycloak-lab exec deploy/postgres -- psql -U keycloak -d keycloak \
  -c "select offline_flag, count(*) from offline_user_session group by offline_flag"
 # 4. A-1 / A-8 을 그대로 반복한다 (증거 파일 참조)
 # 5. 원복
 #   args: ["start"]
 kubectl apply -f deploy/lab/k8s/keycloak-cluster.yaml
 ```
 ---
 ## 9. 이 실험이 A층에 남기는 것
 | | |
 |---|---|
 | **A-0~A-6 의 결론은 모두 "26 기본값" 조건부다** | 버전이 다르면 답이 다르다 |
 | **A-1 이 통념과 어긋난 이유가 확정됐다** | 통념은 24 이전에서 맞다 |
 | **버전 확인이 1순위** | `kc.sh --version` 을 먼저 본다 |
 | **volatile 이 DB 독립을 뜻하지 않는다** | realm·사용자·취소 토큰은 여전히 DB |
 ### 미해결로 남긴 것
 **volatile 에서 refresh 만 500 이 되는 이유** — `REVOKED_TOKEN` 조회 가설을
 세웠지만 확정하지 못했다. A-3 의 문장 로깅 기법으로 재현하면 답이 나온다.
@@ -0,0 +1,175 @@
 # A-8 — 배포할 때마다 로그아웃되는가
 브랜치 `feature/keycloak-a8-rolling-restart` ·
 증거 [`docs/evidence/a8-rolling-restart/`](evidence/a8-rolling-restart/) ·
 2026-09-04 13:38–13:41 KST
 **운영에서 가장 자주 겪는 일이다.** 장애가 아니라 정상 작업인데도
 사용자가 로그아웃되면 그건 사고다.
 ---
 ## 0. 결론부터
 | 확인 | 결과 |
 |---|---|
 | 재시작 중 서비스 중단 | **없음.** 전 구간 `200` |
 | 재시작 전 발급한 refresh token | **여전히 통한다** (`200`) |
 | DB 세션 수 | **151 → 151** 그대로 |
 | 세션 캐시 | **0 으로 초기화** |
 | 클러스터 | 자동 재형성 (`cluster_size = 2`) |
 **세션은 살아남고 캐시만 사라진다.** 이것이 `persistent-user-sessions` 를
 켜는 진짜 이유다.
 ---
 ## 1. 방법
 ```bash
 # 1. 재시작 전 로그인 — 토큰을 상주 파드 안에 보관한다
 kubectl -n keycloak-lab run a8-probe --image=curlimages/curl:8.11.1 \
  --restart=Never --command -- sleep 3600
 kubectl -n keycloak-lab exec a8-probe -- sh -c '<로그인 후 /tmp/rt, /tmp/sid 에 저장>'
 # 2. 재시작하면서 5초 간격으로 외부 진입점을 찍는다
 kubectl -n keycloak-lab rollout restart statefulset/keycloak
 ( for i in $(seq 1 48); do
    curl -s -o /dev/null -w '%{http_code} ' --max-time 4 https://auth.hyeonworks.com/realms/master
    sleep 5
  done ) &
 kubectl -n keycloak-lab rollout status statefulset/keycloak --timeout=420s
 ```
 **탐침 파드가 StatefulSet 밖에 있어야** 재시작을 넘어 토큰을 들고 있을 수 있다.
 ---
 ## 2. 가용성 — 무중단이었다
 ```
 statefulset.apps/keycloak restarted
 200 Waiting for partitioned roll out to finish: 0 out of 2 new pods have been updated...
    Waiting for 1 pods to be ready...
 200 200 200 200 Waiting for partitioned roll out to finish: 1 out of 2 new pods have been updated...
    Waiting for 1 pods to be ready...
 200 200 200 200 partitioned roll out complete: 2 new pods have been updated...
 ```
 **9번 찍어서 9번 다 `200`.** 한 번도 끊기지 않았다.
 ### 왜 무중단이 되는가
 ```
   StatefulSet 롤링 재시작
     │
     ├─ keycloak-1 종료 → Service 엔드포인트에서 빠짐
     │     └─ 이 동안 keycloak-0 이 전부 받는다
     ├─ keycloak-1 기동 → readiness UP → 엔드포인트 복귀
     │
     └─ keycloak-0 종료 → ... (반복)
 ```
 **한 번에 하나씩** 내리므로 항상 최소 하나는 Ready 다.
 readiness 프로브가 이 전환을 정확히 맞춰준다 — A-2 에서 본 그 메커니즘이
 여기서는 **정상 작업을 안전하게** 만든다.
 > 다만 이 실험대는 **파드가 2개**다. replica 1 이면 반드시 끊긴다.
 > 무중단은 공짜가 아니라 **replica ≥ 2 와 readiness 의 조합**이다.
 ---
 ## 3. 세션 생존
 ```
 === 재시작 전 발급한 refresh token 이 아직 통하는가 ===
  대상 sid: XLcgQWRiJrTkuNZcJsNeT_2j
  keycloak-0 에서 refresh  HTTP 200
 === DB 에 그 세션이 남아 있는가 ===
     user_session_id      | created_on | last_session_refresh
 --------------------------+------------+----------------------
 XLcgQWRiJrTkuNZcJsNeT_2j | 1788495513 |           1788495577
  전체 온라인 세션: 151 (재시작 전 151)
 ```
 **`last_session_refresh` 가 `created_on` 보다 64초 뒤**다. 재시작 후의 refresh
 가 **실제로 DB 에 기록**되었다는 뜻이다 — 응답 코드만 200 인 게 아니라
 쓰기까지 정상이다.
 **파드가 통째로 바뀌었는데(44초/66초 나이) 세션은 그대로다.**
 ---
 ## 4. 캐시는 사라진다
 ```
  keycloak-0  sessions 캐시 0.0 건 / cluster_size 2.0
  keycloak-1  sessions 캐시 1.0 건 / cluster_size 2.0
 ```
 ![캐시 초기화와 클러스터 재형성](evidence/a8-rolling-restart/a8-cache-reset-cluster-reformed.png)
 **캐시는 프로세스 메모리이므로 재시작에 사라진다.** `keycloak-1` 의 1건은
 방금 refresh 를 처리하며 새로 담은 것이다.
 ```
   재시작 전:  캐시 N건  +  DB 151건
   재시작 후:  캐시 0건  +  DB 151건      ← 진실은 DB 에 있다
 ```
 **A-0 의 모델이 그대로 확인된다.** 캐시가 통째로 날아가도 정확성은 유지되고
 **첫 접근만 느려진다** (룩어사이드 캐시의 성질).
 ---
 ## 5. 이것이 `persistent-user-sessions` 를 켜는 진짜 이유다
 | | persistent (KC 26 기본) | volatile (KC 24 이전 방식) |
 |---|---|---|
 | 롤링 재시작 후 | **세션 유지** | **전원 로그아웃** |
 | 배포 빈도 | 자유롭다 | 배포가 곧 사고다 |
 | 대가 | DB 쓰기 (A-6 에서 본 지연) | 없음 |
 **A-7 에서 volatile 로 바꿔 같은 실험을 반복하면 여기가 정반대가 될 것이다.**
 그 비교가 이 실험의 짝이다.
 ---
 ## 6. 재현 절차 (명령어)
 ```bash
 # 상주 탐침 (StatefulSet 밖에 있어야 한다)
 kubectl -n keycloak-lab run a8-probe --image=curlimages/curl:8.11.1 \
  --restart=Never --command -- sleep 3600
 kubectl -n keycloak-lab wait --for=condition=Ready pod/a8-probe --timeout=120s
 # 로그인하고 토큰 보관
 kubectl -n keycloak-lab exec a8-probe -- sh -c \
  'curl -s -X POST http://<pod>:8080/realms/master/protocol/openid-connect/token \
     -d grant_type=password -d client_id=admin-cli -d username=admin -d password=<pw> > /tmp/tok'
 # 재시작 + 가용성 감시
 kubectl -n keycloak-lab rollout restart statefulset/keycloak
 kubectl -n keycloak-lab rollout status statefulset/keycloak --timeout=420s
 # 세션 생존 확인
 kubectl -n keycloak-lab exec a8-probe -- sh -c \
  'curl -s -o /dev/null -w "%{http_code}\n" -X POST http://<pod>:8080/realms/master/protocol/openid-connect/token \
     -d grant_type=refresh_token -d client_id=admin-cli -d refresh_token=$(cat /tmp/rt)'
 # DB 대조
 kubectl -n keycloak-lab exec deploy/postgres -- psql -U keycloak -d keycloak \
  -c "select user_session_id, created_on, last_session_refresh from offline_user_session where user_session_id='<sid>'"
 ```
 ---
 ## 7. 다음 실험에 남기는 것
 | 실험 | 이 실험이 준 것 |
 |---|---|
 | **A-7** volatile 비교 | **이 실험을 그대로 반복하면 정반대 결과가 나와야 한다** |
 | **D-2** 버전 업그레이드 | 롤링 재시작이 안전하다는 것이 업그레이드의 전제 |
 | 구성 | 무중단은 **replica ≥ 2 + readiness** 의 조합이다 |
Author	SHA1	Message	Date
DongHyeonkaandClaude Opus 5	8f6d67df35	docs: A-7 — three results invert when persistent sessions are turned off Disabling persistent-user-sessions moves the session from PostgreSQL into the cluster, and the A-1 and A-8 outcomes flip to 400 Session not active while a new login during database loss starts working. The control group in each case still returns 200, so the injections cut only what they were meant to cut. This is the pair that makes the A layer legible: the conventional wisdom that sessions ride TCP 7800 is correct for Keycloak 24 and earlier, and the mistake is applying it to 26 without checking the version. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 13:33:33 +09:00
DongHyeonkaandClaude Opus 5	114d21aebe	docs: A-8 — a rolling restart keeps every session and drops only the cache Nine samples through the restart all returned 200, the refresh token issued beforehand still works, and the session count is unchanged at 151 while both caches reset to zero. The updated last_session_refresh proves the write path recovered too, not just the response code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 13:20:56 +09:00
DongHyeonkaandClaude Opus 5	dba0c3975c	docs: A-6 — 200ms of network delay becomes 22 seconds of user latency Nine database round trips per login multiply the injected delay to 1.9 seconds, and connection pool queueing multiplies it again under twenty concurrent requests. The readiness probe joins the same queue and times out, so the node leaves the load balancer and pushes its load onto the one still standing. Two injections missed first: the guest interface is enp1s0 rather than eth0, and a filter on it can never match a pod IP because flannel has already encapsulated the packet. The delay has to go on flannel.1, before encapsulation. The predicted rise in optimistic lock conflicts did not happen, because logins insert new rows rather than contending for one. That belongs to B-3. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 13:18:10 +09:00
DongHyeonkaandClaude Opus 5	2fa77a899e	docs: A-5 — a one-way block heals itself and a full split still leaves one node serving Three injections failed first: kube-router keeps reinserting its chain above a hand-placed FORWARD rule, the JGroups connection direction had reversed since A-1, and only the raw table runs ahead of conntrack. Each failure looked like nothing happening. Blocking one direction never partitioned the cluster because JGroups reconnected the other way before failure detection fired. Blocking both produced a real split brain with two coordinators in JGROUPS_PING, yet only the non-coordinator node reported DOWN, so the Service kept an endpoint and the front door stayed at 200. That answers the question A-1 left open. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 13:12:04 +09:00
DongHyeonkaandClaude Opus 5	d0666c5ba0	docs: A-4 — a dead pod reports healthier than a live one Kubernetes keeps calling the node Ready for forty seconds while users already see failures, and the pod on the powered-off machine stays ready=true because its kubelet can no longer contradict itself. Eviction waits another five minutes, then the StatefulSet refuses to recreate its pod and the replacement Deployment pod cannot schedule because the local-path volume is pinned to the dead node. Killing the server node instead shows the opposite shape: containerd keeps the workload running while the API server, Traefik and the observability stack disappear, so the outage is the missing path rather than the missing application. Traefik at one replica is the ingress single point of failure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 12:25:30 +09:00
DongHyeonkaandClaude Opus 5	99b689e715	docs: A-3 — four logins returned tokens for sessions the crash erased Keycloak commits the login INSERT with synchronous_commit off, so a crash loses whole sessions and not just refresh timestamps. Measured 4 of 153 lost, matching the default wal_writer_delay window. Two injections failed silently first: --grace-period=0 --force lets the container runtime send SIGTERM so PostgreSQL flushes and shuts down cleanly, and SIGKILL to PID 1 from inside its own namespace is ignored by the kernel. Killing a backend makes the postmaster reinitialize, which is a real crash recovery. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 12:05:08 +09:00