docs: B-6 — rotation is safe, retiring the old key is not

Adding a higher-priority RSA provider leaves both kids in JWKS, so tokens signed before and after the rotation both validate. Deleting the old provider makes its tokens 401 immediately, and the resource server's JWKS cache does not buy a grace period because an unknown kid triggers a refetch. The encryption key Q3 asks about does not exist yet, since B-2 showed the tokens are stored as plaintext JWTs, so the measured signing-key rotation is what its design has to copy: write with one key, read with several, and keep the overlap longer than the lifetime of anything signed with the old one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
docs: B-5 — the pod stays Ready while every request hangs
2026-09-04 14:33:42 +09:00 · 2026-09-04 14:29:17 +09:00 · 2026-09-04 14:23:41 +09:00 · 2026-09-04 14:20:04 +09:00 · 2026-09-04 14:15:05 +09:00 · 2026-09-04 14:05:02 +09:00
109 changed files with 5027 additions and 0 deletions
@@ -22,3 +22,29 @@
 [ 1661576ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1665269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [ 1682669ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1689435ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1704385ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1706314ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1724551ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1743701ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1758339ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1769299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1789478ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1795415ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1815382ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1820392ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1822445ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1839035ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1840162ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1847739ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1861419ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1880381ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1893315ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1911434ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1922701ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1933868ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1944951ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1948001ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1952503ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1953320ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1955369ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,20 @@
+[     585ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    1132ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&panes=%7B%22o68%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22agroal_blocking_time_max_milliseconds%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22blocking+max+-+%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-30m%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1:0
+[    1555ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    3604ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    6373ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    9135ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   19066ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   24595ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   47125ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   52154ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   67608ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   86965ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  108668ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  129158ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  137544ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  152699ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  159898ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  173491ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  187925ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  197824ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,143 @@
+[     635ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22vendor_statistics_approximate_entries_unique%7Bcache%3D%5C%22sessions%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%2C%7B%22refId%22%3A%22B%22%2C%22expr%22%3A%22vendor_cluster_size%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22cluster_size%20%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-15m%22%2C%22to%22%3A%22now%22%7D%7D%7D:0
+[     686ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    1451ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    3030ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    4308ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    5296ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    7691ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   18337ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   20184ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   32473ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   50697ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   65961ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   85590ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   99644ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  115524ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  130773ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  134354ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  142549ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  151361ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  155246ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  165901ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  169379ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  180952ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  190372ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  193657ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  205737ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  220582ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  225496ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  240339ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  244029ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  264211ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  273424ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  284893ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  292571ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  305778ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  306804ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  324378ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  335169ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  337968ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  352967ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  364258ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  379409ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  399590ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  403681ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  406340ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  421033ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  428667ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  429483ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  437573ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  442286ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  445865ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  456312ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  464043ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  480382ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  491438ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  494608ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  508231ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  528305ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  529632ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  538033ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  550079ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  566347ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  586157ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  603877ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  615779ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  624657ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  629879ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  637255ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  643298ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  650047ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  668080ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  669816ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  675422ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  677001ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  686611ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  692655ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  713114ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  723787ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  730443ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  751427ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  757583ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  776007ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  792298ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  803509ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  822917ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  827374ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  847278ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  867460ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  875338ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  886296ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  900938ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  912511ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  917067ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  924405ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  930435ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  942211ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  953146ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  957153ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  969140ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  984088ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1003544ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1014908ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1020851ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1034668ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1040406ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1056887ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1062828ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1073375ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1085052ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1102031ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1110039ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1129341ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1132300ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1133670ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1142705ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1160906ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1172400ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1190482ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1196673ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1213666ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1225136ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1228310ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1242751ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1243874ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1256126ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1274444ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1288526ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1307058ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1320986ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1325134ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1335974ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1344832ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1354780ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1364165ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1383346ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1399019ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1400207ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1413875ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1423809ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1427502ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1430773ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1436408ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1447464ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1 @@
+[     207ms] [ERROR] Failed to load resource: the server responded with a status of 404 () @ https://app1.hyeonworks.com/favicon.ico:0
@@ -0,0 +1 @@
+[     236ms] [ERROR] Failed to load resource: the server responded with a status of 500 () @ https://app1.hyeonworks.com/bff/api/me:0
@@ -0,0 +1,2 @@
+[    7292ms] [ERROR] Access to fetch at 'https://auth.hyeonworks.com/realms/keycloak-patterns/protocol/openid-connect/auth?response_type=code&client_id=bff-confidential&scope=openid%20profile%20email&state=WWc76H7TY73Fbsdc41B2nRD5exkXqHcohLh4WdJB4AA%3D&redirect_uri=https://app1.hyeonworks.com/login/oauth2/code/keycloak&nonce=A4TXweuKS4Y5HdZ63rLJUez1ZOyI2em6zs3OIfTXLFo&code_challenge=wPr8PXG0lcUvie7Wo91YrVMhOUYq0KtEU4PxVJ0_CWA&code_challenge_method=S256' (redirected from 'https://app1.hyeonworks.com/bff/api/me') from origin 'https://app1.hyeonworks.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. @ https://app1.hyeonworks.com/bff/token-boundary:0
+[    7293ms] [ERROR] Failed to load resource: net::ERR_FAILED @ https://auth.hyeonworks.com/realms/keycloak-patterns/protocol/openid-connect/auth?response_type=code&client_id=bff-confidential&scope=openid%20profile%20email&state=WWc76H7TY73Fbsdc41B2nRD5exkXqHcohLh4WdJB4AA%3D&redirect_uri=https://app1.hyeonworks.com/login/oauth2/code/keycloak&nonce=A4TXweuKS4Y5HdZ63rLJUez1ZOyI2em6zs3OIfTXLFo&code_challenge=wPr8PXG0lcUvie7Wo91YrVMhOUYq0KtEU4PxVJ0_CWA&code_challenge_method=S256:0
@@ -0,0 +1,2 @@
+[    6726ms] [ERROR] Access to fetch at 'https://auth.hyeonworks.com/realms/keycloak-patterns/protocol/openid-connect/auth?response_type=code&client_id=bff-confidential&scope=openid%20profile%20email&state=GGupuPr3ZklKp8ah99r7h7mNHEq9yTsEWZr85WyXevE%3D&redirect_uri=https://app1.hyeonworks.com/login/oauth2/code/keycloak&nonce=rPVvEOvG7rzssAjR7pP67qNvoY2W6ZVpIpKYz1LGoU8&code_challenge=qoKRLRrzB7z9CU_rlaAxJ7UYcRZswyqmDi8PgxmaWM0&code_challenge_method=S256' (redirected from 'https://app1.hyeonworks.com/bff/api/me') from origin 'https://app1.hyeonworks.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. @ https://app1.hyeonworks.com/:0
+[    6726ms] [ERROR] Failed to load resource: net::ERR_FAILED @ https://auth.hyeonworks.com/realms/keycloak-patterns/protocol/openid-connect/auth?response_type=code&client_id=bff-confidential&scope=openid%20profile%20email&state=GGupuPr3ZklKp8ah99r7h7mNHEq9yTsEWZr85WyXevE%3D&redirect_uri=https://app1.hyeonworks.com/login/oauth2/code/keycloak&nonce=rPVvEOvG7rzssAjR7pP67qNvoY2W6ZVpIpKYz1LGoU8&code_challenge=qoKRLRrzB7z9CU_rlaAxJ7UYcRZswyqmDi8PgxmaWM0&code_challenge_method=S256:0
@@ -0,0 +1,2 @@
+[    6898ms] [ERROR] Failed to load resource: the server responded with a status of 403 () @ https://app1.hyeonworks.com/logout:0
+[   23950ms] [ERROR] Failed to load resource: the server responded with a status of 403 () @ https://app1.hyeonworks.com/logout:0
@@ -0,0 +1,145 @@
+- generic [active] [ref=f15e1]:
+  - generic [ref=f15e4]:
+    - link "Skip to main content" [ref=f15e5] [cursor=pointer]:
+      - /url: "#pageContent"
+    - banner [ref=f15e7]:
+      - generic [ref=f15e8]:
+        - link [ref=f15e10] [cursor=pointer]:
+          - /url: /
+          - img "Grafana" [ref=f15e11]
+        - generic [ref=f15e14]:
+          - button "Search or jump to..." [ref=f15e18] [cursor=pointer]
+          - generic [ref=f15e19]: ctrl+k
+        - generic [ref=f15e23]:
+          - button "New" [ref=f15e24] [cursor=pointer]
+          - button "Help" [ref=f15e30] [cursor=pointer]
+          - button "News" [ref=f15e33] [cursor=pointer]
+          - button "Profile" [ref=f15e36] [cursor=pointer]:
+            - img "User avatar" [ref=f15e37]
+      - generic [ref=f15e38]:
+        - button "Open menu" [ref=f15e40] [cursor=pointer]
+        - navigation "Breadcrumbs" [ref=f15e43]:
+          - list [ref=f15e44]:
+            - listitem [ref=f15e45]:
+              - link "Home" [ref=f15e46] [cursor=pointer]:
+                - /url: /
+            - listitem [ref=f15e50]:
+              - link "Explore" [ref=f15e51] [cursor=pointer]:
+                - /url: /explore
+            - listitem [ref=f15e55]:
+              - generic "Prometheus" [ref=f15e56]
+        - generic [ref=f15e57]:
+          - generic [ref=f15e60]:
+            - button "Copy shortened URL" [ref=f15e61] [cursor=pointer]
+            - button "Open copy link options" [ref=f15e64] [cursor=pointer]
+          - button "Toggle top search bar" [ref=f15e68] [cursor=pointer]
+    - main [ref=f15e74]:
+      - generic [ref=f15e76]:
+        - heading "Explore" [level=1] [ref=f15e77]
+        - generic [ref=f15e82]:
+          - navigation "Explore toolbar" [ref=f15e84]:
+            - navigation "Search links" [ref=f15e86]:
+              - generic [ref=f15e87]:
+                - button "Content outline" [expanded] [ref=f15e89] [cursor=pointer]:
+                  - generic [ref=f15e92]: Outline
+                - generic [ref=f15e97] [cursor=pointer]:
+                  - img "Prometheus logo" [ref=f15e99]
+                  - textbox "Select a data source" [ref=f15e100]:
+                    - /placeholder: ""
+            - generic [ref=f15e104]:
+              - button "Split the pane" [ref=f15e106] [cursor=pointer]:
+                - generic [ref=f15e109]: Split
+              - button "Add" [ref=f15e111] [cursor=pointer]
+              - generic [ref=f15e116]:
+                - 'button "Time range selected: Last 30 minutes" [ref=f15e117] [cursor=pointer]'
+                - button "Zoom out time range" [ref=f15e122] [cursor=pointer]
+              - generic [ref=f15e126]:
+                - button "Run query" [ref=f15e127] [cursor=pointer]
+                - button "Auto refresh turned off. Choose refresh time interval" [ref=f15e131] [cursor=pointer]
+          - generic [ref=f15e135]:
+            - generic [ref=f15e139]:
+              - button "Collapse outline" [expanded] [ref=f15e141] [cursor=pointer]:
+                - img "arrow-from-right" [ref=f15e142]
+              - button "Queries" [ref=f15e145] [cursor=pointer]:
+                - img "arrow" [ref=f15e146]
+              - button "Graph" [ref=f15e150] [cursor=pointer]:
+                - img "graph-bar" [ref=f15e151]
+            - generic [ref=f15e158]:
+              - generic [ref=f15e160]:
+                - generic "Query editor row" [ref=f15e163]:
+                  - generic [ref=f15e164]:
+                    - generic [ref=f15e166]:
+                      - generic [ref=f15e167]:
+                        - button "Collapse query row" [expanded] [ref=f15e168] [cursor=pointer]
+                        - generic [ref=f15e171]:
+                          - button "Query editor row title A" [ref=f15e172] [cursor=pointer]:
+                            - generic [ref=f15e173]: A
+                          - emphasis [ref=f15e174]: (Prometheus)
+                      - generic [ref=f15e175]:
+                        - button "Show data source help" [ref=f15e177] [cursor=pointer]
+                        - button "Duplicate query" [ref=f15e181] [cursor=pointer]
+                        - button "Hide response" [ref=f15e185] [cursor=pointer]
+                        - button "Remove query" [ref=f15e189] [cursor=pointer]
+                        - button "Drag and drop to reorder" [ref=f15e192]:
+                          - img "Drag and drop to reorder" [ref=f15e193]
+                    - generic [ref=f15e196]:
+                      - generic [ref=f15e197]:
+                        - button "Kick start your query" [ref=f15e198] [cursor=pointer]
+                        - generic [ref=f15e201]:
+                          - generic [ref=f15e202] [cursor=pointer]: Explain
+                          - generic [ref=f15e203]:
+                            - checkbox "Explain Toggle switch" [ref=f15e204]
+                            - generic "Toggle switch" [ref=f15e205] [cursor=pointer]
+                        - radiogroup [ref=f15e210]:
+                          - generic [ref=f15e211]:
+                            - radio "Builder" [ref=f15e212] [cursor=pointer]
+                            - generic [ref=f15e213] [cursor=pointer]: Builder
+                          - generic [ref=f15e214]:
+                            - radio "Code" [checked] [ref=f15e215] [cursor=pointer]
+                            - generic [ref=f15e216] [cursor=pointer]: Code
+                      - generic [ref=f15e218]:
+                        - generic [ref=f15e220]:
+                          - button "Loading metrics..." [disabled] [ref=f15e221] [cursor=pointer]
+                          - code [ref=f15e228]:
+                            - generic [ref=f15e229]:
+                              - generic [ref=f15e234]: agroal_blocking_time_max_milliseconds
+                              - textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=f15e239]: agroal_blocking_time_max_milliseconds
+                        - 'button "Options Legend: blocking max - {{pod}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f15e245] [cursor=pointer]':
+                          - generic [ref=f15e249]:
+                            - heading "Options" [level=6] [ref=f15e250]
+                            - generic [ref=f15e251]:
+                              - generic [ref=f15e252]: "Legend: blocking max - {{pod}}"
+                              - generic [ref=f15e253]: "Format: Time series"
+                              - generic [ref=f15e254]: "Step: auto"
+                              - generic [ref=f15e255]: "Type: Range"
+                              - generic [ref=f15e256]: "Exemplars: false"
+                - generic [ref=f15e257]:
+                  - button "Add query" [ref=f15e258] [cursor=pointer]
+                  - button "Query history" [ref=f15e262] [cursor=pointer]
+                  - button "Query inspector" [ref=f15e266] [cursor=pointer]
+              - main [ref=f15e270]:
+                - region [ref=f15e272]:
+                  - generic [ref=f15e273]:
+                    - heading "Graph" [level=2] [ref=f15e275]
+                    - radiogroup [ref=f15e278]:
+                      - generic [ref=f15e279]:
+                        - radio "Lines" [checked] [ref=f15e280] [cursor=pointer]
+                        - generic [ref=f15e281] [cursor=pointer]: Lines
+                      - generic [ref=f15e282]:
+                        - radio "Bars" [ref=f15e283] [cursor=pointer]
+                        - generic [ref=f15e284] [cursor=pointer]: Bars
+                      - generic [ref=f15e285]:
+                        - radio "Points" [ref=f15e286] [cursor=pointer]
+                        - generic [ref=f15e287] [cursor=pointer]: Points
+                      - generic [ref=f15e288]:
+                        - radio "Stacked lines" [ref=f15e289] [cursor=pointer]
+                        - generic [ref=f15e290] [cursor=pointer]: Stacked lines
+                      - generic [ref=f15e291]:
+                        - radio "Stacked bars" [ref=f15e292] [cursor=pointer]
+                        - generic [ref=f15e293] [cursor=pointer]: Stacked bars
+                  - generic [ref=f15e294]: Loading plugin panel...
+  - generic [ref=f15e299]:
+    - alert
+    - alert
+    - complementary
+    - complementary
@@ -0,0 +1,56 @@
+- generic [ref=f18e4]:
+  - link "Skip to main content" [ref=f18e5] [cursor=pointer]:
+    - /url: "#pageContent"
+  - banner [ref=f18e7]:
+    - generic [ref=f18e8]:
+      - link [ref=f18e10] [cursor=pointer]:
+        - /url: /
+        - img "Grafana" [ref=f18e11]
+      - generic [ref=f18e14]:
+        - button "Search or jump to..." [ref=f18e18] [cursor=pointer]
+        - generic [ref=f18e19]: ctrl+k
+      - generic [ref=f18e23]:
+        - button "New" [ref=f18e24] [cursor=pointer]
+        - button "Help" [ref=f18e30] [cursor=pointer]
+        - button "News" [ref=f18e33] [cursor=pointer]
+        - button "Profile" [ref=f18e36] [cursor=pointer]:
+          - img "User avatar" [ref=f18e37]
+    - generic [ref=f18e38]:
+      - button "Open menu" [ref=f18e40] [cursor=pointer]
+      - navigation "Breadcrumbs" [ref=f18e43]:
+        - list [ref=f18e44]:
+          - listitem [ref=f18e45]:
+            - link "Home" [ref=f18e46] [cursor=pointer]:
+              - /url: /
+          - listitem [ref=f18e50]:
+            - link "Explore" [ref=f18e51] [cursor=pointer]:
+              - /url: /explore
+          - listitem [ref=f18e55]:
+            - generic "Prometheus" [ref=f18e56]
+      - generic [ref=f18e57]:
+        - button "Show more items" [ref=f18e60] [cursor=pointer]
+        - button "Toggle top search bar" [ref=f18e64] [cursor=pointer]
+  - main [ref=f18e70]:
+    - generic [ref=f18e72]:
+      - heading "Explore" [level=1] [ref=f18e73]
+      - generic [ref=f18e78]:
+        - navigation "Explore toolbar" [ref=f18e80]:
+          - navigation "Search links" [ref=f18e82]:
+            - generic [ref=f18e83]:
+              - button "Content outline" [expanded] [ref=f18e85] [cursor=pointer]:
+                - generic [ref=f18e88]: Outline
+              - generic [ref=f18e93] [cursor=pointer]:
+                - img "Prometheus logo" [ref=f18e95]
+                - textbox "Select a data source" [ref=f18e96]:
+                  - /placeholder: ""
+          - button "Show more items" [ref=f18e102] [cursor=pointer]
+        - generic [ref=f18e106]:
+          - button "Collapse outline" [expanded] [ref=f18e112] [cursor=pointer]:
+            - img "arrow-from-right" [ref=f18e113]
+          - generic [ref=f18e120]:
+            - generic [ref=f18e123]:
+              - button "Add query" [ref=f18e124] [cursor=pointer]
+              - button "Query history" [ref=f18e128] [cursor=pointer]
+              - button "Query inspector" [ref=f18e132] [cursor=pointer]
+            - generic:
+              - main
@@ -0,0 +1,7 @@
+- main [ref=f21e2]:
+  - heading "AP3 · Backend-for-Frontend" [level=1] [ref=f21e3]
+  - paragraph [ref=f21e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
+  - button "Keycloak 로그인" [ref=f21e5] [cursor=pointer]
+  - button "token 경계 확인" [ref=f21e6] [cursor=pointer]
+  - button "BFF 경유 API 호출" [ref=f21e7] [cursor=pointer]
+  - button "CSRF token으로 상태 변경" [ref=f21e8] [cursor=pointer]
@@ -0,0 +1,16 @@
+- generic [ref=f22e3]:
+  - banner [ref=f22e4]:
+    - generic [ref=f22e5]: keycloak-patterns
+  - main [ref=f22e6]:
+    - heading "Sign in to your account" [level=1] [ref=f22e8]
+    - generic [ref=f22e12]:
+      - generic [ref=f22e13]:
+        - generic [ref=f22e14]: Username or email
+        - textbox "Username or email" [active] [ref=f22e17]
+      - generic [ref=f22e18]:
+        - generic [ref=f22e19]: Password
+        - generic [ref=f22e21]:
+          - textbox "Password" [ref=f22e24]
+          - button "Show password" [ref=f22e26] [cursor=pointer]:
+            - generic [aria-hidden] [ref=f22e27]: 
+      - button "Sign In" [ref=f22e30] [cursor=pointer]
@@ -0,0 +1,20 @@
+- generic [ref=f23e3]:
+  - banner [ref=f23e4]:
+    - generic [ref=f23e5]: keycloak-patterns
+  - main [ref=f23e6]:
+    - heading "Update Account Information" [level=1] [ref=f23e8]
+    - generic [ref=f23e9]:
+      - generic [ref=f23e10]: "* Required fields"
+      - generic [ref=f23e13]:
+        - generic [ref=f23e14]:
+          - generic [ref=f23e15]: Email *
+          - textbox "Email" [ref=f23e19]: labuser@example.com
+        - generic [ref=f23e20]:
+          - generic [ref=f23e21]: First name *
+          - textbox "First name" [invalid] [ref=f23e25]
+          - generic [ref=f23e26]: Please specify this field.
+        - generic [ref=f23e31]:
+          - generic [ref=f23e32]: Last name *
+          - textbox "Last name" [invalid] [ref=f23e36]
+          - generic [ref=f23e37]: Please specify this field.
+        - button "Submit" [ref=f23e44]
@@ -0,0 +1,20 @@
+- generic [ref=f23e3]:
+  - banner [ref=f23e4]:
+    - generic [ref=f23e5]: keycloak-patterns
+  - main [ref=f23e6]:
+    - heading "Update Account Information" [level=1] [ref=f23e8]
+    - generic [ref=f23e9]:
+      - generic [ref=f23e10]: "* Required fields"
+      - generic [ref=f23e13]:
+        - generic [ref=f23e14]:
+          - generic [ref=f23e15]: Email *
+          - textbox "Email" [ref=f23e19]: labuser@example.com
+        - generic [ref=f23e20]:
+          - generic [ref=f23e21]: First name *
+          - textbox "First name" [invalid] [ref=f23e25]: Lab
+          - generic [ref=f23e26]: Please specify this field.
+        - generic [ref=f23e31]:
+          - generic [ref=f23e32]: Last name *
+          - textbox "Last name" [active] [invalid] [ref=f23e36]: User
+          - generic [ref=f23e37]: Please specify this field.
+        - button "Submit" [ref=f23e44]
@@ -0,0 +1,7 @@
+- main [ref=f24e2]:
+  - heading "AP3 · Backend-for-Frontend" [level=1] [ref=f24e3]
+  - paragraph [ref=f24e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
+  - button "Keycloak 로그인" [ref=f24e5] [cursor=pointer]
+  - button "token 경계 확인" [ref=f24e6] [cursor=pointer]
+  - button "BFF 경유 API 호출" [ref=f24e7] [cursor=pointer]
+  - button "CSRF token으로 상태 변경" [ref=f24e8] [cursor=pointer]
@@ -0,0 +1,16 @@
+- generic [ref=f25e3]:
+  - banner [ref=f25e4]:
+    - generic [ref=f25e5]: keycloak-patterns
+  - main [ref=f25e6]:
+    - heading "Sign in to your account" [level=1] [ref=f25e8]
+    - generic [ref=f25e12]:
+      - generic [ref=f25e13]:
+        - generic [ref=f25e14]: Username or email
+        - textbox "Username or email" [active] [ref=f25e17]
+      - generic [ref=f25e18]:
+        - generic [ref=f25e19]: Password
+        - generic [ref=f25e21]:
+          - textbox "Password" [ref=f25e24]
+          - button "Show password" [ref=f25e26] [cursor=pointer]:
+            - generic [aria-hidden] [ref=f25e27]: 
+      - button "Sign In" [ref=f25e30] [cursor=pointer]
@@ -0,0 +1,9 @@
+- generic [ref=f26e2]:
+  - heading "Login with OAuth 2.0" [level=2] [ref=f26e3]
+  - alert [ref=f26e4]: Invalid credentials
+  - table [ref=f26e5]:
+    - rowgroup [ref=f26e6]:
+      - row [ref=f26e7]:
+        - cell [ref=f26e8]:
+          - link "keycloak" [ref=f26e9] [cursor=pointer]:
+            - /url: /oauth2/authorization/keycloak
@@ -0,0 +1,7 @@
+- main [ref=f27e2]:
+  - heading "AP3 · Backend-for-Frontend" [level=1] [ref=f27e3]
+  - paragraph [ref=f27e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
+  - button "Keycloak 로그인" [ref=f27e5] [cursor=pointer]
+  - button "token 경계 확인" [ref=f27e6] [cursor=pointer]
+  - button "BFF 경유 API 호출" [ref=f27e7] [cursor=pointer]
+  - button "CSRF token으로 상태 변경" [ref=f27e8] [cursor=pointer]
@@ -0,0 +1,7 @@
+- main [ref=f27e2]:
+  - heading "AP3 · Backend-for-Frontend" [level=1] [ref=f27e3]
+  - paragraph [ref=f27e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
+  - button "Keycloak 로그인" [ref=f27e5] [cursor=pointer]
+  - button "token 경계 확인" [ref=f27e6] [cursor=pointer]
+  - button "BFF 경유 API 호출" [ref=f27e7] [cursor=pointer]
+  - button "CSRF token으로 상태 변경" [ref=f27e8] [cursor=pointer]
@@ -0,0 +1,7 @@
+- main [ref=f27e2]:
+  - heading "AP3 · Backend-for-Frontend" [level=1] [ref=f27e3]
+  - paragraph [ref=f27e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
+  - button "Keycloak 로그인" [ref=f27e5] [cursor=pointer]
+  - button "token 경계 확인" [ref=f27e6] [cursor=pointer]
+  - button "BFF 경유 API 호출" [ref=f27e7] [cursor=pointer]
+  - button "CSRF token으로 상태 변경" [ref=f27e8] [cursor=pointer]
@@ -0,0 +1 @@
+- generic [active] [ref=f28e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":true,\"refreshTokenStoredOnServer\":true,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1,7 @@
+- main [ref=f29e2]:
+  - heading "AP3 · Backend-for-Frontend" [level=1] [ref=f29e3]
+  - paragraph [ref=f29e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
+  - button "Keycloak 로그인" [ref=f29e5] [cursor=pointer]
+  - button "token 경계 확인" [ref=f29e6] [cursor=pointer]
+  - button "BFF 경유 API 호출" [ref=f29e7] [cursor=pointer]
+  - button "CSRF token으로 상태 변경" [ref=f29e8] [cursor=pointer]
@@ -0,0 +1 @@
+- generic [active] [ref=f30e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1,5 @@
+- generic [active] [ref=f31e1]:
+  - heading "Whitelabel Error Page" [level=1] [ref=f31e2]
+  - paragraph [ref=f31e3]: This application has no explicit mapping for /error, so you are seeing this as a fallback.
+  - generic [ref=f31e4]: Fri Sep 04 05:00:51 GMT 2026
+  - generic [ref=f31e5]: There was an unexpected error (type=Internal Server Error, status=500).
@@ -0,0 +1 @@
+- generic [active] [ref=f32e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1,7 @@
+- main [ref=f33e2]:
+  - heading "AP3 · Backend-for-Frontend" [level=1] [ref=f33e3]
+  - paragraph [ref=f33e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
+  - button "Keycloak 로그인" [ref=f33e5] [cursor=pointer]
+  - button "token 경계 확인" [ref=f33e6] [cursor=pointer]
+  - button "BFF 경유 API 호출" [ref=f33e7] [cursor=pointer]
+  - button "CSRF token으로 상태 변경" [ref=f33e8] [cursor=pointer]
@@ -0,0 +1 @@
+- generic [active] [ref=f34e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1 @@
+- generic [active] [ref=f35e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1,7 @@
+- main [ref=f36e2]:
+  - heading "AP3 · Backend-for-Frontend" [level=1] [ref=f36e3]
+  - paragraph [ref=f36e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
+  - button "Keycloak 로그인" [ref=f36e5] [cursor=pointer]
+  - button "token 경계 확인" [ref=f36e6] [cursor=pointer]
+  - button "BFF 경유 API 호출" [ref=f36e7] [cursor=pointer]
+  - button "CSRF token으로 상태 변경" [ref=f36e8] [cursor=pointer]
@@ -0,0 +1 @@
+- generic [active] [ref=f37e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":true,\"refreshTokenStoredOnServer\":true,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1,7 @@
+- main [ref=f38e2]:
+  - heading "AP3 · Backend-for-Frontend" [level=1] [ref=f38e3]
+  - paragraph [ref=f38e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
+  - button "Keycloak 로그인" [ref=f38e5] [cursor=pointer]
+  - button "token 경계 확인" [ref=f38e6] [cursor=pointer]
+  - button "BFF 경유 API 호출" [ref=f38e7] [cursor=pointer]
+  - button "CSRF token으로 상태 변경" [ref=f38e8] [cursor=pointer]
@@ -0,0 +1,7 @@
+- main [ref=f39e2]:
+  - heading "AP3 · Backend-for-Frontend" [level=1] [ref=f39e3]
+  - paragraph [ref=f39e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
+  - button "Keycloak 로그인" [ref=f39e5] [cursor=pointer]
+  - button "token 경계 확인" [ref=f39e6] [cursor=pointer]
+  - button "BFF 경유 API 호출" [ref=f39e7] [cursor=pointer]
+  - button "CSRF token으로 상태 변경" [ref=f39e8] [cursor=pointer]
@@ -0,0 +1 @@
+target/
@@ -0,0 +1,14 @@
+FROM maven:3.9.11-eclipse-temurin-21-alpine AS build
+WORKDIR /workspace
+COPY pom.xml .
+RUN mvn --batch-mode dependency:go-offline
+COPY src src
+RUN mvn --batch-mode verify
+
+FROM eclipse-temurin:21-jre-alpine
+RUN addgroup -S spring && adduser -S spring -G spring
+WORKDIR /app
+COPY --from=build /workspace/target/keycloak-bff.jar app.jar
+USER spring:spring
+EXPOSE 8083
+ENTRYPOINT ["java", "-jar", "/app/app.jar"]
@@ -0,0 +1,90 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+
+    <parent>
+        <groupId>org.springframework.boot</groupId>
+        <artifactId>spring-boot-starter-parent</artifactId>
+        <version>3.5.16</version>
+        <relativePath/>
+    </parent>
+
+    <groupId>com.example</groupId>
+    <artifactId>keycloak-bff</artifactId>
+    <version>0.0.1-SNAPSHOT</version>
+    <name>keycloak-bff</name>
+
+    <properties>
+        <java.version>21</java.version>
+    </properties>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.springframework.boot</groupId>
+            <artifactId>spring-boot-starter-actuator</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.springframework.boot</groupId>
+            <artifactId>spring-boot-starter-oauth2-client</artifactId>
+        </dependency>
+
+        <!-- B-1: Application Session 을 Redis 로 옮긴다.
+             spring-session-data-redis 가 SessionRepository 를 갈아끼우고,
+             spring-boot-starter-data-redis 가 연결(Lettuce)을 제공한다.
+             둘 다 있어야 자동구성이 걸린다 — 하나만 넣으면 조용히 in-memory 로 남는다. -->
+        <dependency>
+            <groupId>org.springframework.session</groupId>
+            <artifactId>spring-session-data-redis</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.springframework.boot</groupId>
+            <artifactId>spring-boot-starter-data-redis</artifactId>
+        </dependency>
+
+        <!-- B-2: OAuth2AuthorizedClient 를 PostgreSQL 로 옮긴다.
+             Q3 가 후보로 든 "Redis 와 JDBC 중 무엇" 에서 JDBC 쪽이며,
+             JdbcOAuth2AuthorizedClientService 는 같은 인터페이스라
+             컨트롤러를 바꾸지 않아도 된다. -->
+        <dependency>
+            <groupId>org.springframework.boot</groupId>
+            <artifactId>spring-boot-starter-jdbc</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.postgresql</groupId>
+            <artifactId>postgresql</artifactId>
+            <scope>runtime</scope>
+        </dependency>
+        <dependency>
+            <groupId>com.h2database</groupId>
+            <artifactId>h2</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.springframework.boot</groupId>
+            <artifactId>spring-boot-starter-web</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>org.springframework.boot</groupId>
+            <artifactId>spring-boot-starter-test</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.springframework.security</groupId>
+            <artifactId>spring-security-test</artifactId>
+            <scope>test</scope>
+        </dependency>
+    </dependencies>
+
+    <build>
+        <finalName>keycloak-bff</finalName>
+        <plugins>
+            <plugin>
+                <groupId>org.springframework.boot</groupId>
+                <artifactId>spring-boot-maven-plugin</artifactId>
+            </plugin>
+        </plugins>
+    </build>
+</project>
@@ -0,0 +1,12 @@
+package com.example.keycloakpattern.bff;
+
+import org.springframework.boot.SpringApplication;
+import org.springframework.boot.autoconfigure.SpringBootApplication;
+
+@SpringBootApplication
+public class BffApplication {
+
+    public static void main(String[] args) {
+        SpringApplication.run(BffApplication.class, args);
+    }
+}
@@ -0,0 +1,112 @@
+package com.example.keycloakpattern.bff;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.springframework.beans.factory.annotation.Value;
+import org.springframework.http.CacheControl;
+import org.springframework.http.HttpHeaders;
+import org.springframework.http.ResponseEntity;
+import org.springframework.security.core.Authentication;
+import org.springframework.security.oauth2.client.OAuth2AuthorizeRequest;
+import org.springframework.security.oauth2.client.OAuth2AuthorizedClient;
+import org.springframework.security.oauth2.client.OAuth2AuthorizedClientManager;
+import org.springframework.security.oauth2.client.OAuth2AuthorizedClientService;
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.PostMapping;
+import org.springframework.web.bind.annotation.RequestParam;
+import org.springframework.web.bind.annotation.RestController;
+import org.springframework.web.client.RestClient;
+import org.springframework.web.server.ResponseStatusException;
+
+import static org.springframework.http.HttpStatus.UNAUTHORIZED;
+
+@RestController
+public class BffController {
+
+    private final OAuth2AuthorizedClientService authorizedClientService;
+    private final OAuth2AuthorizedClientManager authorizedClientManager;
+    private final RestClient resourceApi;
+    private final AtomicReference<String> theme = new AtomicReference<>("system");
+
+    public BffController(
+        OAuth2AuthorizedClientService authorizedClientService,
+        OAuth2AuthorizedClientManager authorizedClientManager,
+        RestClient.Builder restClientBuilder,
+        @Value("${resource-api.base-url}") String resourceApiBaseUrl
+    ) {
+        this.authorizedClientService = authorizedClientService;
+        this.authorizedClientManager = authorizedClientManager;
+        this.resourceApi = restClientBuilder.baseUrl(resourceApiBaseUrl).build();
+    }
+
+    @GetMapping("/bff/token-boundary")
+    ResponseEntity<Map<String, Object>> tokenBoundary(Authentication authentication) {
+        OAuth2AuthorizedClient client = authorizedClientService.loadAuthorizedClient(
+            "keycloak",
+            authentication.getName()
+        );
+
+        Map<String, Object> response = new LinkedHashMap<>();
+        response.put("pattern", "AP3-backend-for-frontend");
+        response.put("principal", authentication.getName());
+        response.put("accessTokenStoredOnServer", client != null
+            && client.getAccessToken() != null);
+        response.put("refreshTokenStoredOnServer", client != null
+            && client.getRefreshToken() != null);
+        response.put("browserTokenCount", 0);
+        response.put("csrfProtectionEnabled", true);
+
+        return ResponseEntity.ok()
+            .cacheControl(CacheControl.noStore())
+            .header("Pragma", "no-cache")
+            .body(response);
+    }
+
+    @GetMapping("/bff/api/me")
+    ResponseEntity<?> currentUser(Authentication authentication) {
+        OAuth2AuthorizedClient client = authorizedClient(authentication);
+        return resourceApi.get()
+            .uri("/api/me")
+            .header(
+                HttpHeaders.AUTHORIZATION,
+                "Bearer " + client.getAccessToken().getTokenValue()
+            )
+            .retrieve()
+            .toEntity(Map.class);
+    }
+
+    @PostMapping("/bff/api/preferences")
+    Map<String, Object> updatePreference(
+        Authentication authentication,
+        @RequestParam(defaultValue = "system") String theme
+    ) {
+        this.theme.set(theme);
+        return Map.of(
+            "updated", true,
+            "theme", this.theme.get(),
+            "principal", authentication.getName()
+        );
+    }
+
+    @GetMapping("/bff/api/preferences")
+    Map<String, String> preference() {
+        return Map.of("theme", theme.get());
+    }
+
+    private OAuth2AuthorizedClient authorizedClient(Authentication authentication) {
+        OAuth2AuthorizeRequest request = OAuth2AuthorizeRequest
+            .withClientRegistrationId("keycloak")
+            .principal(authentication)
+            .build();
+        OAuth2AuthorizedClient client = authorizedClientManager.authorize(request);
+        if (client == null || client.getAccessToken() == null) {
+            throw new ResponseStatusException(
+                UNAUTHORIZED,
+                "No authorized Keycloak client is available"
+            );
+        }
+        return client;
+    }
+}
@@ -0,0 +1,25 @@
+package com.example.keycloakpattern.bff;
+
+import java.util.Map;
+
+import org.springframework.http.CacheControl;
+import org.springframework.http.ResponseEntity;
+import org.springframework.security.web.csrf.CsrfToken;
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.RestController;
+
+@RestController
+public class CsrfController {
+
+    @GetMapping("/bff/csrf")
+    ResponseEntity<Map<String, String>> csrf(CsrfToken csrfToken) {
+        return ResponseEntity.ok()
+            .cacheControl(CacheControl.noStore())
+            .header("Pragma", "no-cache")
+            .body(Map.of(
+                "headerName", csrfToken.getHeaderName(),
+                "parameterName", csrfToken.getParameterName(),
+                "token", csrfToken.getToken()
+            ));
+    }
+}
@@ -0,0 +1,108 @@
+package com.example.keycloakpattern.bff;
+
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+import org.springframework.security.config.annotation.web.builders.HttpSecurity;
+import org.springframework.security.oauth2.client.AuthorizedClientServiceOAuth2AuthorizedClientManager;
+import org.springframework.security.oauth2.client.OAuth2AuthorizedClientManager;
+import org.springframework.security.oauth2.client.OAuth2AuthorizedClientProvider;
+import org.springframework.security.oauth2.client.OAuth2AuthorizedClientProviderBuilder;
+import org.springframework.security.oauth2.client.OAuth2AuthorizedClientService;
+import org.springframework.security.oauth2.client.JdbcOAuth2AuthorizedClientService;
+import org.springframework.security.oauth2.client.registration.ClientRegistrationRepository;
+import org.springframework.security.oauth2.client.web.DefaultOAuth2AuthorizationRequestResolver;
+import org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestCustomizers;
+import org.springframework.security.web.SecurityFilterChain;
+import org.springframework.security.web.csrf.CookieCsrfTokenRepository;
+import org.springframework.jdbc.core.JdbcOperations;
+
+@Configuration
+public class SecurityConfig {
+
+    /**
+     * B-2 — authorized client 를 프로세스 메모리에서 PostgreSQL 로 옮긴다.
+     *
+     * B-1 에서 Application Session 만 Redis 로 옮겼더니, 사용자는 로그인
+     * 상태로 보이는데 BFF 에는 access token 이 없는 상태가 만들어졌다.
+     * 두 상태의 저장소를 **각각** 정해야 한다는 Q3 의 지적이 그대로 나타난 것이다.
+     *
+     * 주의 — 이것이 고치는 것과 고치지 못하는 것이 다르다.
+     *   고친다   : 인스턴스 간 공유. 어느 replica 로 가도 같은 토큰을 본다.
+     *   못 고친다: 조회 키. JdbcOAuth2AuthorizedClientService 도
+     *              (clientRegistrationId, principalName) 으로 찾으므로
+     *              같은 사용자의 두 브라우저는 여전히 한 항목을 공유한다.
+     */
+    @Bean
+    OAuth2AuthorizedClientService authorizedClientService(
+        JdbcOperations jdbcOperations,
+        ClientRegistrationRepository clientRegistrationRepository
+    ) {
+        return new JdbcOAuth2AuthorizedClientService(jdbcOperations, clientRegistrationRepository);
+    }
+
+    @Bean
+    SecurityFilterChain bffSecurity(
+        HttpSecurity http,
+        ClientRegistrationRepository clientRegistrationRepository
+    ) throws Exception {
+        DefaultOAuth2AuthorizationRequestResolver authorizationRequestResolver =
+            new DefaultOAuth2AuthorizationRequestResolver(
+                clientRegistrationRepository,
+                "/oauth2/authorization"
+            );
+        authorizationRequestResolver.setAuthorizationRequestCustomizer(
+            OAuth2AuthorizationRequestCustomizers.withPkce()
+        );
+
+        CookieCsrfTokenRepository csrfTokenRepository =
+            CookieCsrfTokenRepository.withHttpOnlyFalse();
+        csrfTokenRepository.setCookiePath("/");
+
+        return http
+            .csrf(csrf -> csrf
+                .csrfTokenRepository(csrfTokenRepository)
+                .csrfTokenRequestHandler(new SpaCsrfTokenRequestHandler()))
+            .authorizeHttpRequests(authorize -> authorize
+                .requestMatchers(
+                    "/",
+                    "/index.html",
+                    "/app.js",
+                    "/favicon.ico",
+                    "/actuator/health",
+                    "/actuator/health/**",
+                    // 실험대 전용 — B-0 은 "자동구성이 실제로 무엇을 골랐는가"를
+                    // 밖에서 읽어야 답할 수 있다. 운영에서는 절대 열지 않는다:
+                    // /actuator/beans 와 /actuator/env 는 내부 구조와 설정값을
+                    // 그대로 드러낸다.
+                    "/actuator/**"
+                )
+                .permitAll()
+                .anyRequest()
+                .authenticated())
+            .oauth2Login(oauth2 -> oauth2
+                .authorizationEndpoint(endpoint -> endpoint
+                    .authorizationRequestResolver(authorizationRequestResolver))
+                .defaultSuccessUrl("/", true))
+            .build();
+    }
+
+    @Bean
+    OAuth2AuthorizedClientManager authorizedClientManager(
+        ClientRegistrationRepository clientRegistrationRepository,
+        OAuth2AuthorizedClientService authorizedClientService
+    ) {
+        OAuth2AuthorizedClientProvider authorizedClientProvider =
+            OAuth2AuthorizedClientProviderBuilder.builder()
+                .authorizationCode()
+                .refreshToken()
+                .build();
+
+        AuthorizedClientServiceOAuth2AuthorizedClientManager manager =
+            new AuthorizedClientServiceOAuth2AuthorizedClientManager(
+                clientRegistrationRepository,
+                authorizedClientService
+            );
+        manager.setAuthorizedClientProvider(authorizedClientProvider);
+        return manager;
+    }
+}
@@ -0,0 +1,40 @@
+package com.example.keycloakpattern.bff;
+
+import java.util.function.Supplier;
+
+import jakarta.servlet.http.HttpServletRequest;
+import jakarta.servlet.http.HttpServletResponse;
+
+import org.springframework.security.web.csrf.CsrfToken;
+import org.springframework.security.web.csrf.CsrfTokenRequestAttributeHandler;
+import org.springframework.security.web.csrf.CsrfTokenRequestHandler;
+import org.springframework.security.web.csrf.XorCsrfTokenRequestAttributeHandler;
+import org.springframework.util.StringUtils;
+
+final class SpaCsrfTokenRequestHandler implements CsrfTokenRequestHandler {
+
+    private final CsrfTokenRequestHandler plain =
+        new CsrfTokenRequestAttributeHandler();
+    private final CsrfTokenRequestHandler xor =
+        new XorCsrfTokenRequestAttributeHandler();
+
+    @Override
+    public void handle(
+        HttpServletRequest request,
+        HttpServletResponse response,
+        Supplier<CsrfToken> deferredCsrfToken
+    ) {
+        xor.handle(request, response, deferredCsrfToken);
+    }
+
+    @Override
+    public String resolveCsrfTokenValue(
+        HttpServletRequest request,
+        CsrfToken csrfToken
+    ) {
+        if (StringUtils.hasText(request.getHeader(csrfToken.getHeaderName()))) {
+            return plain.resolveCsrfTokenValue(request, csrfToken);
+        }
+        return xor.resolveCsrfTokenValue(request, csrfToken);
+    }
+}
@@ -0,0 +1,80 @@
+server:
+  port: ${SERVER_PORT:8083}
+  servlet:
+    session:
+      cookie:
+        name: AP3_SESSION
+        http-only: true
+        same-site: lax
+
+spring:
+  application:
+    name: keycloak-bff
+  datasource:
+    # B-2: authorized client 전용. Keycloak 과 같은 PostgreSQL 인스턴스지만
+    # 테이블이 다르다(oauth2_authorized_client). 운영이라면 분리를 검토한다.
+    url: ${BFF_DB_URL:jdbc:postgresql://localhost:5432/keycloak}
+    username: ${BFF_DB_USER:keycloak}
+    password: ${BFF_DB_PASSWORD:keycloak}
+  sql:
+    init:
+      # Spring Security 가 제공하는 DDL 을 그대로 쓴다.
+      # always 로 두면 매 기동마다 실행되므로 CREATE TABLE IF NOT EXISTS 가 아닌
+      # 스크립트에서는 실패한다 → continue-on-error 로 넘긴다.
+      mode: ${SPRING_SQL_INIT_MODE:always}
+      # ★ PostgreSQL 은 -postgres 판본을 써야 한다. 기본 판본은 `blob` 타입을
+      #   쓰는데 PostgreSQL 에는 그 타입이 없다(`bytea` 다). continue-on-error 가
+      #   그 실패를 삼켜서 "테이블이 조용히 안 생기는" 상태가 됐었다.
+      schema-locations: classpath:org/springframework/security/oauth2/client/oauth2-client-schema-postgres.sql
+      continue-on-error: true
+  data:
+    redis:
+      host: ${REDIS_HOST:localhost}
+      port: ${REDIS_PORT:6379}
+  session:
+    # Application Session 만 Redis 로 간다. OAuth2AuthorizedClient 는
+    # 이 설정과 무관하며 여전히 InMemory 다 — 조회 키가 다르기 때문이다(B-0).
+    store-type: ${SPRING_SESSION_STORE_TYPE:redis}
+    timeout: ${SPRING_SESSION_TIMEOUT:30m}
+    redis:
+      namespace: bff:session
+  security:
+    oauth2:
+      client:
+        registration:
+          keycloak:
+            provider: keycloak
+            client-id: bff-confidential
+            client-secret: ${KEYCLOAK_CLIENT_SECRET}
+            client-authentication-method: client_secret_basic
+            authorization-grant-type: authorization_code
+            redirect-uri: "{baseUrl}/login/oauth2/code/{registrationId}"
+            scope:
+              - openid
+              - profile
+              - email
+        provider:
+          keycloak:
+            # 브라우저가 리다이렉트되는 주소와 BFF 가 서버끼리 부르는 주소는 다르다.
+            # 앞의 것은 외부에서 닿는 이름이어야 하고, 뒤의 것은 클러스터 안 주소여도 된다.
+            authorization-uri: ${KC_ISSUER_EXTERNAL:http://localhost:8080/realms/keycloak-patterns}/protocol/openid-connect/auth
+            token-uri:         ${KC_ISSUER_INTERNAL:http://keycloak:8080/realms/keycloak-patterns}/protocol/openid-connect/token
+            jwk-set-uri:       ${KC_ISSUER_INTERNAL:http://keycloak:8080/realms/keycloak-patterns}/protocol/openid-connect/certs
+            user-info-uri:     ${KC_ISSUER_INTERNAL:http://keycloak:8080/realms/keycloak-patterns}/protocol/openid-connect/userinfo
+            user-name-attribute: preferred_username
+
+resource-api:
+  base-url: ${RESOURCE_API_BASE_URL:http://localhost:8081}
+
+management:
+  endpoint:
+    health:
+      probes:
+        enabled: true
+      show-details: always
+  endpoints:
+    web:
+      exposure:
+        # beans / conditions 는 B-0 에서 "자동구성이 실제로 무엇을 골랐는가"를
+        # 보기 위해 연다. 운영에 그대로 두면 내부 구조가 노출된다.
+        include: health,info,beans,conditions,env
@@ -0,0 +1,59 @@
+const result = document.querySelector("#result");
+
+function render(value) {
+  result.textContent = JSON.stringify(value, null, 2);
+}
+
+function readCookie(name) {
+  const prefix = `${encodeURIComponent(name)}=`;
+  const value = document.cookie
+    .split("; ")
+    .find((cookie) => cookie.startsWith(prefix));
+  return value ? decodeURIComponent(value.slice(prefix.length)) : null;
+}
+
+async function request(path, options = {}) {
+  const response = await fetch(path, {
+    ...options,
+    headers: { Accept: "application/json", ...options.headers },
+  });
+  if (response.redirected || response.status === 401) {
+    window.location.assign("/oauth2/authorization/keycloak");
+    return null;
+  }
+  const body = await response.json();
+  render({ status: response.status, ...body });
+  return { response, body };
+}
+
+document.querySelector("#login").addEventListener("click", () => {
+  window.location.assign("/oauth2/authorization/keycloak");
+});
+
+document.querySelector("#inspect").addEventListener("click", () => {
+  void request("/bff/token-boundary");
+});
+
+document.querySelector("#call-bff").addEventListener("click", () => {
+  void request("/bff/api/me");
+});
+
+document.querySelector("#change-with-csrf").addEventListener("click", async () => {
+  const csrfResponse = await fetch("/bff/csrf", {
+    headers: { Accept: "application/json" },
+  });
+  const csrf = await csrfResponse.json();
+  const csrfToken = readCookie("XSRF-TOKEN");
+  if (!csrfToken) {
+    render({ status: 500, error: "XSRF-TOKEN cookie was not created" });
+    return;
+  }
+  await request("/bff/api/preferences", {
+    method: "POST",
+    body: new URLSearchParams({ theme: "dark" }),
+    headers: {
+      "Content-Type": "application/x-www-form-urlencoded",
+      [csrf.headerName]: csrfToken,
+    },
+  });
+});
@@ -0,0 +1,31 @@
+<!doctype html>
+<html lang="ko">
+<head>
+  <meta charset="utf-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1">
+  <title>AP3 · Backend-for-Frontend</title>
+  <style>
+    :root { color-scheme: light dark; font-family: system-ui, sans-serif; }
+    body { max-width: 58rem; margin: 6vh auto; padding: 0 1.5rem; line-height: 1.6; }
+    button { margin: 0 0.5rem 0.5rem 0; padding: 0.6rem 0.9rem; cursor: pointer; }
+    pre { min-height: 9rem; padding: 1rem; border-radius: 0.4rem;
+      background: color-mix(in srgb, CanvasText 9%, Canvas); white-space: pre-wrap; }
+  </style>
+</head>
+<body>
+  <main>
+    <h1>AP3 · Backend-for-Frontend</h1>
+    <p>
+      브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session
+      cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource
+      Server 요청에 붙입니다.
+    </p>
+    <button id="login" type="button">Keycloak 로그인</button>
+    <button id="inspect" type="button">token 경계 확인</button>
+    <button id="call-bff" type="button">BFF 경유 API 호출</button>
+    <button id="change-with-csrf" type="button">CSRF token으로 상태 변경</button>
+    <pre id="result" aria-live="polite"></pre>
+  </main>
+  <script type="module" src="/app.js"></script>
+</body>
+</html>
@@ -0,0 +1,99 @@
+package com.example.keycloakpattern.bff;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.springframework.security.test.web.servlet.request.SecurityMockMvcRequestPostProcessors.oidcLogin;
+import static org.springframework.security.test.web.servlet.request.SecurityMockMvcRequestPostProcessors.csrf;
+import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
+import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post;
+import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.header;
+import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
+import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
+
+import org.junit.jupiter.api.Test;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
+import org.springframework.boot.test.context.SpringBootTest;
+import org.springframework.security.oauth2.client.OAuth2AuthorizedClient;
+import org.springframework.security.oauth2.client.OAuth2AuthorizedClientManager;
+import org.springframework.security.oauth2.client.OAuth2AuthorizedClientService;
+import org.springframework.security.oauth2.core.OAuth2AccessToken;
+import org.springframework.security.oauth2.core.OAuth2RefreshToken;
+import org.springframework.test.context.bean.override.mockito.MockitoBean;
+import org.springframework.test.web.servlet.MockMvc;
+
+@SpringBootTest(properties = {
+    "KEYCLOAK_CLIENT_SECRET=test-only-secret",
+    // 테스트는 Redis 를 띄우지 않는다. store-type=none 이면 자동구성이
+    // 서블릿 컨테이너 기본 세션으로 되돌아가 컨텍스트가 뜬다.
+    "spring.session.store-type=none",
+    // 테스트에는 PostgreSQL 이 없다. H2 로 대신하고 Spring Security 의
+    // DDL 을 그대로 태워 JdbcOAuth2AuthorizedClientService 가 뜨게 한다.
+    "spring.datasource.url=jdbc:h2:mem:bfftest;DB_CLOSE_DELAY=-1",
+    "spring.datasource.username=sa",
+    "spring.datasource.password=",
+    "spring.sql.init.mode=always",
+    "resource-api.base-url=http://127.0.0.1:9"
+})
+@AutoConfigureMockMvc
+class BffControllerTest {
+
+    @Autowired
+    private MockMvc mockMvc;
+
+    @MockitoBean
+    private OAuth2AuthorizedClientService authorizedClientService;
+
+    @MockitoBean
+    private OAuth2AuthorizedClientManager authorizedClientManager;
+
+    @Test
+    void reportsServerTokenCustodyWithoutReturningTokens() throws Exception {
+        OAuth2AuthorizedClient client = mock(OAuth2AuthorizedClient.class);
+        when(client.getAccessToken()).thenReturn(mock(OAuth2AccessToken.class));
+        when(client.getRefreshToken()).thenReturn(mock(OAuth2RefreshToken.class));
+        when(authorizedClientService.loadAuthorizedClient("keycloak", "test-subject"))
+            .thenReturn(client);
+
+        mockMvc.perform(get("/bff/token-boundary").with(oidcLogin()
+                .idToken(token -> token.subject("test-subject"))))
+            .andExpect(status().isOk())
+            .andExpect(header().string("Cache-Control", "no-store"))
+            .andExpect(jsonPath("$.accessTokenStoredOnServer").value(true))
+            .andExpect(jsonPath("$.refreshTokenStoredOnServer").value(true))
+            .andExpect(jsonPath("$.browserTokenCount").value(0))
+            .andExpect(jsonPath("$.csrfProtectionEnabled").value(true))
+            .andExpect(jsonPath("$.access_token").doesNotExist())
+            .andExpect(jsonPath("$.refresh_token").doesNotExist());
+    }
+
+    @Test
+    void rejectsStateChangeWithoutCsrfToken() throws Exception {
+        mockMvc.perform(post("/bff/api/preferences")
+                .param("theme", "attacker")
+                .with(oidcLogin().idToken(token -> token.subject("test-subject"))))
+            .andExpect(status().isForbidden());
+    }
+
+    @Test
+    void acceptsStateChangeWithCsrfToken() throws Exception {
+        mockMvc.perform(post("/bff/api/preferences")
+                .param("theme", "dark")
+                .with(oidcLogin().idToken(token -> token.subject("test-subject")))
+                .with(csrf()))
+            .andExpect(status().isOk())
+            .andExpect(jsonPath("$.updated").value(true))
+            .andExpect(jsonPath("$.theme").value("dark"));
+    }
+
+    @Test
+    void exposesSpaCsrfTokenWithoutCaching() throws Exception {
+        mockMvc.perform(get("/bff/csrf").with(oidcLogin()
+                .idToken(token -> token.subject("test-subject"))))
+            .andExpect(status().isOk())
+            .andExpect(header().string("Cache-Control", "no-store"))
+            .andExpect(header().exists("Set-Cookie"))
+            .andExpect(jsonPath("$.headerName").value("X-XSRF-TOKEN"))
+            .andExpect(jsonPath("$.token").isNotEmpty());
+    }
+}
@@ -0,0 +1,214 @@
+# BFF (2 replicas) + Redis, for the B-layer experiments.
+#
+# The BFF is deployed FIRST WITHOUT any session store wiring. That is deliberate:
+# B-0 asks what Spring Boot's autoconfiguration actually picks when nothing is
+# configured, and the only honest way to answer is to look at a running instance
+# that has been given nothing. Redis is deployed alongside but left unused until
+# B-1 turns it on.
+#
+#   kubectl apply -f deploy/lab/k8s/bff-redis.yaml
+#
+# Image comes from the workstation, not a registry:
+#   docker build -t keycloak-pattern-bff:lab bff/
+#   docker save keycloak-pattern-bff:lab | ssh test-server "ssh kc-lab-1 'sudo k3s ctr images import -'"
+#   (repeat for kc-lab-2)
+# so imagePullPolicy must stay Never on both replicas.
+apiVersion: v1
+kind: Secret
+metadata:
+  name: bff-secrets
+  namespace: keycloak-lab
+type: Opaque
+stringData:
+  # Matches the client created with kcadm in the keycloak-patterns realm.
+  # Base64 in etcd is not encryption — see D-3.
+  KEYCLOAK_CLIENT_SECRET: bff-lab-secret
+---
+# Redis. B-5 measured that turning on AOF with `redis-cli config set` changes
+# nothing here, because /data is the container filesystem and dies with the
+# container — the appendonlydir was created and then thrown away. Persistence
+# configuration without a volume is decoration.
+#
+# So the volume comes first, and only then does `--appendonly yes` mean anything.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: redis-data
+  namespace: keycloak-lab
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: local-path
+  resources:
+    requests:
+      storage: 1Gi
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: redis
+  namespace: keycloak-lab
+spec:
+  replicas: 1
+  selector:
+    matchLabels: { app: redis }
+  template:
+    metadata:
+      labels: { app: redis }
+    spec:
+      # Same node as postgres so a node-loss experiment takes both stores at
+      # once, matching how A-4 was set up.
+      nodeSelector:
+        kubernetes.io/hostname: kc-lab-2
+      containers:
+        - name: redis
+          image: redis:7.4-alpine
+          # appendfsync everysec 이 기본값이다 — 1초 분량을 잃을 수 있다.
+          # Keycloak 의 synchronous_commit OFF(A-3)와 같은 모양의 트레이드오프다.
+          args: ["redis-server", "--appendonly", "yes", "--dir", "/data"]
+          ports:
+            - containerPort: 6379
+              name: redis
+          readinessProbe:
+            exec: { command: ["redis-cli", "ping"] }
+            initialDelaySeconds: 3
+          volumeMounts:
+            - name: data
+              mountPath: /data
+          resources:
+            requests: { memory: 32Mi, cpu: 20m }
+            limits:   { memory: 128Mi }
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: redis-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: redis
+  namespace: keycloak-lab
+spec:
+  selector: { app: redis }
+  ports:
+    - port: 6379
+      targetPort: redis
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: bff
+  namespace: keycloak-lab
+spec:
+  # Two replicas is the whole point: Q1 and Q2 only exist because a request can
+  # land on an instance that did not handle the login.
+  replicas: 2
+  selector:
+    matchLabels: { app: bff }
+  template:
+    metadata:
+      labels: { app: bff }
+    spec:
+      # Spread across both nodes so "the other instance" is genuinely another
+      # machine, not another process on the same kernel.
+      topologySpreadConstraints:
+        - maxSkew: 1
+          topologyKey: kubernetes.io/hostname
+          whenUnsatisfiable: ScheduleAnyway
+          labelSelector:
+            matchLabels: { app: bff }
+      # 쿠버네티스는 같은 네임스페이스의 Service 마다 Docker link 시절의
+      # 환경변수를 자동 주입한다: REDIS_PORT=tcp://10.43.57.116:6379.
+      # 그것이 application.yml 의 ${REDIS_PORT:6379} 를 덮어써서 기동이 실패했다.
+      #   Failed to bind properties under 'spring.data.redis.port' to int:
+      #     Value: "tcp://10.43.57.116:6379"
+      # 이 주입 자체를 끄는 것이 근본 처방이다. 이름을 바꿔 피하면 다음 사람이
+      # 같은 함정에 다시 빠진다.
+      enableServiceLinks: false
+      containers:
+        - name: bff
+          image: keycloak-pattern-bff:lab
+          imagePullPolicy: Never
+          ports:
+            - containerPort: 8083
+              name: http
+          env:
+            # The browser is redirected to the public name; the BFF calls the
+            # token endpoint over the cluster network. Getting these two the same
+            # way round is what the 2-hop header experiment was about.
+            - name: KC_ISSUER_EXTERNAL
+              value: https://auth.hyeonworks.com/realms/keycloak-patterns
+            - name: KC_ISSUER_INTERNAL
+              value: http://keycloak.keycloak-lab.svc:8080/realms/keycloak-patterns
+            # echo 는 header-lab 네임스페이스의 8081 이다. 다른 네임스페이스의
+            # 서비스는 <svc>.<ns>.svc 로 부른다. 이름을 틀리면 500 이 나는데
+            # 원인은 UnresolvedAddressException 이지 토큰 문제가 아니다.
+            - name: RESOURCE_API_BASE_URL
+              value: http://echo.header-lab.svc:8081
+            - name: KEYCLOAK_CLIENT_SECRET
+              valueFrom:
+                secretKeyRef: { name: bff-secrets, key: KEYCLOAK_CLIENT_SECRET }
+            # Spring needs to know it is behind TLS termination, for the same
+            # reason Keycloak needs KC_PROXY_HEADERS. Without it the redirect_uri
+            # it builds comes back as http:// and Keycloak rejects it.
+            - name: SERVER_FORWARD_HEADERS_STRATEGY
+              value: native
+            # B-1: Application Session 을 Redis 로 옮긴다.
+            # OAuth2AuthorizedClient 는 이것으로 옮겨지지 않는다 — 조회 키가
+            # 다르기 때문이며, B-0 에서 확인한 사실이다.
+            - name: SPRING_SESSION_STORE_TYPE
+              value: redis
+            - name: REDIS_HOST
+              value: redis.keycloak-lab.svc
+            - name: REDIS_PORT
+              value: "6379"
+            # B-2: authorized client 는 PostgreSQL 로. 세션(Redis)과 다른
+            # 저장소를 쓰는 것이 Q3 가 말한 "각각 설계한다"의 실물이다.
+            - name: BFF_DB_URL
+              value: jdbc:postgresql://postgres.keycloak-lab.svc:5432/keycloak
+            - name: BFF_DB_USER
+              value: keycloak
+            - name: BFF_DB_PASSWORD
+              valueFrom:
+                secretKeyRef: { name: keycloak-lab-secrets, key: POSTGRES_PASSWORD }
+            - name: JAVA_TOOL_OPTIONS
+              value: "-Xms128m -Xmx320m"
+          readinessProbe:
+            httpGet: { path: /actuator/health/readiness, port: http }
+            initialDelaySeconds: 20
+            failureThreshold: 30
+          livenessProbe:
+            httpGet: { path: /actuator/health/liveness, port: http }
+            initialDelaySeconds: 60
+          resources:
+            requests: { memory: 320Mi, cpu: 100m }
+            limits:   { memory: 512Mi }
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: bff
+  namespace: keycloak-lab
+spec:
+  selector: { app: bff }
+  ports:
+    - port: 8083
+      targetPort: http
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: bff
+  namespace: keycloak-lab
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: app1.hyeonworks.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: bff
+                port:
+                  number: 8083
@@ -0,0 +1,22 @@
+=== 배치 — 왜 A/B 가 되는가 ===
+  postgres  10.42.1.76  (kc-lab-2)
+  keycloak-0 10.42.1.77 (kc-lab-2) → DB 와 같은 노드, cni0 로 직행
+  keycloak-1 10.42.0.42 (kc-lab-1) → DB 와 다른 노드, VXLAN 을 건넌다  ← 여기에 지연을 건다
+
+=== 사용 가능한 커넥션 풀 지표 ===
+agroal_acquire_count_total
+agroal_active_count
+agroal_available_count
+agroal_awaiting_count
+agroal_blocking_time_average_milliseconds
+agroal_blocking_time_max_milliseconds
+agroal_blocking_time_total_milliseconds
+agroal_creation_count_total
+agroal_creation_time_average_milliseconds
+agroal_creation_time_max_milliseconds
+agroal_creation_time_total_milliseconds
+agroal_destroy_count_total
+
+=== 기준선 지연 — 각 노드에서 로그인 20회 ===
+  keycloak-0  평균 70 ms
+  keycloak-1  평균 66 ms
@@ -0,0 +1,25 @@
+=== 주입: postgres(10.42.1.76) 가 보내는 패킷만 200ms 지연 (kc-lab-2 eth0) ===
+  prio qdisc 로 밴드를 나누고, u32 필터로 출발지 IP 가 postgres 인 것만 3번 밴드로 보낸다
+Cannot find device "eth0"
+Cannot find device "eth0"
+적용완료
+Cannot find device "eth0"
+Cannot find device "eth0"
+  주입: 13:14:55
+
+=== [검증] 지연이 실제로 걸렸는가 — 두 노드 비교 ===
+  keycloak-0  평균 43 ms   최대 64 ms
+  keycloak-1  평균 47 ms   최대 70 ms
+
+=== 커넥션 풀 상태 ===
+--- keycloak-0 ---
+  agroal_blocking_time_max_milliseconds 102.0
+  agroal_active_count 0.0
+  agroal_awaiting_count 0.0
+  agroal_blocking_time_average_milliseconds 0.0
+  agroal_available_count 2.0
+  agroal_blocking_time_max_milliseconds 164.0
+  agroal_active_count 0.0
+  agroal_awaiting_count 0.0
+  agroal_blocking_time_average_milliseconds 0.0
+  agroal_available_count 2.0
@@ -0,0 +1,20 @@
+=== 오버레이 인터페이스 확인 ===
+flannel.1        UNKNOWN        a6:b2:62:04:c1:a4 <BROADCAST,MULTICAST,UP,LOWER_UP> 
+cni0             UP             5a:77:1a:e2:b0:a4 <BROADCAST,MULTICAST,UP,LOWER_UP> 
+
+=== flannel.1 에 주입 — 여기서는 파드 IP 가 보인다 (캡슐화 전) ===
+qdisc prio 1: root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
+ Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
+ backlog 0b 0p requeues 0
+qdisc netem 30: parent 1:3 limit 1000 delay 200ms
+ Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
+ backlog 0b 0p requeues 0
+
+=== [검증] 필터에 패킷이 걸리는가 ===
+  qdisc netem 30: parent 1:3 limit 1000 delay 200ms
+   Sent 18388 bytes 150 pkt (dropped 0, overlimits 0 requeues 0) 
+   backlog 0b 0p requeues 0
+
+=== 두 노드 지연 비교 (기준선: k0=70ms k1=66ms) ===
+  keycloak-0  평균 41 ms   최대 57 ms
+  keycloak-1  평균 1872 ms   최대 1887 ms
@@ -0,0 +1,37 @@
+=== 동시 부하 20건을 keycloak-1 에 — 커넥션 풀이 견디는가 ===
+        1 200 1.911191
+        1 200 1.913766
+        1 200 1.958374
+        1 200 1.981620
+        1 200 10.539402
+        1 200 11.951943
+        1 200 13.351102
+        1 200 14.785832
+        1 200 16.189533
+        1 200 17.625166
+        1 200 19.053724
+        1 200 20.495883
+        1 200 21.905932
+        1 200 22.228466
+        1 200 22.230871
+        1 200 3.441366
+        1 200 4.841075
+        1 200 6.257489
+        1 200 7.704608
+        1 200 9.104792
+
+=== 부하 직후 커넥션 풀 ===
+  agroal_blocking_time_max_milliseconds 20000.0
+  agroal_max_used_count 19.0
+  agroal_acquire_count_total 672.0
+  agroal_active_count 0.0
+  agroal_awaiting_count 0.0
+  agroal_blocking_time_average_milliseconds 281.0
+  agroal_available_count 19.0
+
+=== readiness 가 흔들렸는가 ===
+keycloak-0                  1/1   Running   0             60m
+keycloak-1                  1/1   Running   1 (51m ago)   3h24m
+52m         Normal    TaintManagerEviction   pod/keycloak-1                   Cancelling deletion of Pod keycloak-lab/keycloak-1
+32m         Warning   Unhealthy              pod/keycloak-1                   Readiness probe failed: HTTP probe failed with statuscode: 503
+89s         Warning   Unhealthy              pod/keycloak-1                   Readiness probe failed: Get "http://10.42.0.42:9000/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
@@ -0,0 +1,12 @@
+=== 지연 해제 ===
+해제완료
+qdisc noqueue 0: root refcnt 2 
+
+=== 회복 확인 ===
+  keycloak-0  평균 43 ms
+  keycloak-1  평균 51 ms
+keycloak-0                  1/1   Running   0             61m
+keycloak-1                  1/1   Running   1 (52m ago)   3h24m
+
+=== 낙관적 락 충돌이 늘었는가 — 지연 중 로그 ===
+  관련 로그 줄수: 0
@@ -0,0 +1,20 @@
+# A-6 — 지연 주입 증거
+
+2026-09-04 13:10–13:35 KST
+해설: [`docs/experiment-a6-latency-injection.md`](../../experiment-a6-latency-injection.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-baseline.txt` | 배치 설명(A/B 가 되는 이유), `agroal_*` 지표 목록, **기준선 70ms / 66ms** |
+| `02-delay-injected.txt` | 첫 시도 실패 — **`Cannot find device "eth0"`** (Debian 은 `enp1s0`) |
+| `03-flannel-injection.txt` | **성공** — `flannel.1` 에 걸어야 파드 IP 가 보인다. netem `Sent 150 pkt` 로 검증. **k0=41ms vs k1=1872ms** |
+| `04-pool-under-load.txt` | 동시 20건 — 응답이 1.9초에서 **22.2초**까지 계단. `blocking_time_max=20000ms`, `max_used_count=19`, **readiness 프로브 타임아웃** |
+| `05-recovery.txt` | 해제 즉시 43ms / 51ms 회복. **낙관적 락 충돌 0건**(예측 빗나감) |
+| `a6-connection-pool-blocking.png` | Grafana — `agroal_blocking_time_max_milliseconds` |
+
+## 핵심 네 줄
+
+1. **200ms 가 1,872ms 가 된다.** 로그인 트랜잭션의 왕복이 9번이라 지연이 곱해진다.
+2. **동시 부하에서 22초까지 늘어난다.** 커넥션 풀 큐잉으로 한 번 더 곱해진다.
+3. **헬스체크도 같은 줄에 선다** — 프로브가 타임아웃되어 노드가 로드밸런서에서 빠지고, 남은 노드로 부하가 몰린다.
+4. **낙관적 락 충돌은 없었다.** 로그인은 새 행을 만들 뿐 같은 행을 다투지 않는다 — B-3 의 영역.
@@ -0,0 +1,18 @@
+=== 비교를 위해 세션을 비운다 ===
+DELETE 151
+
+=== volatile 모드로 전환 ===
+namespace/keycloak-lab unchanged
+secret/keycloak-lab-secrets configured
+persistentvolumeclaim/postgres-data unchanged
+deployment.apps/postgres unchanged
+service/postgres unchanged
+statefulset.apps/keycloak configured
+service/keycloak-headless unchanged
+service/keycloak unchanged
+ingress.networking.k8s.io/keycloak unchanged
+Waiting for 1 pods to be ready...
+partitioned roll out complete: 2 new pods have been updated...
+
+=== [검증] 정말 꺼졌는가 ===
+["start","--features-disabled=persistent-user-sessions"]
@@ -0,0 +1,15 @@
+  keycloak-0=10.42.1.94  keycloak-1=10.42.0.45
+
+=== [A-0 재실행] keycloak-0 에만 로그인 5회 → 캐시가 어디에 담기는가 ===
+로그인완료
+  keycloak-0  sessions 캐시 5.0 건
+  keycloak-1  sessions 캐시 0.0 건
+
+=== DB 에는 들어갔는가 (persistent 였을 때는 5건이 들어갔다) ===
+ offline_flag | count 
+--------------+-------
+(0 rows)
+
+
+=== 교차 노드 세션은 되는가 ===
+  keycloak-0 로그인 → keycloak-1 에서 refresh  HTTP 200
@@ -0,0 +1,13 @@
+=== [A-8 재실행] 재시작 전 로그인 ===
+  sid = aVwYnzKZFFvMqD3bpSeiILuM
+
+=== 롤링 재시작 ===
+statefulset.apps/keycloak restarted
+partitioned roll out complete: 2 new pods have been updated...
+
+=== ★ 재시작 전 토큰이 아직 통하는가 (persistent 였을 때는 200) ===
+  keycloak-0 에서 refresh  HTTP 400
+  --- 오류 본문 ---
+{"error":"invalid_grant","error_description":"Session not active"}
+=== 캐시 상태 ===
+  keycloak-1  sessions 캐시 1.0 건
@@ -0,0 +1,21 @@
+=== [A-1 재실행] volatile 에서 7800 을 막으면 ===
+  keycloak-0=10.42.1.99  keycloak-1=10.42.0.46
+  [대조군] 차단 전 교차 노드 refresh
+    keycloak-1 에서 refresh  HTTP 200
+
+  차단 적용 (A-5 에서 확인한 raw 테이블 방식, 양방향)
+  분단이 성립할 때까지 대기...
+    +25초  cluster_size(k0 k1) = [2.0 2.0 ]
+    +50초  cluster_size(k0 k1) = [1.0 ]
+    +75초  cluster_size(k0 k1) = [1.0 ]
+    +100초  cluster_size(k0 k1) = []
+    +125초  cluster_size(k0 k1) = [1.0 ]
+    +150초  cluster_size(k0 k1) = [1.0 ]
+    +175초  cluster_size(k0 k1) = [1.0 ]
+    +200초  cluster_size(k0 k1) = []
+
+=== ★ 분단 상태에서 교차 노드 세션 (persistent 였을 때는 200) ===
+  keycloak-0 로그인 → keycloak-0 에서 refresh  HTTP 200  ← 대조군
+  keycloak-0 로그인 → keycloak-1 에서 refresh  HTTP 400  ← 시험군
+  --- 시험군 오류 본문 ---
+{"error":"invalid_grant","error_description":"Session not active"}
@@ -0,0 +1,12 @@
+차단 해제, 클러스터 재형성 대기...
+
+=== [A-2 재실행] volatile 에서 DB 를 내리면 — 세션이 메모리에 있으니 살아남는가? ===
+  DB 정지 전 로그인 완료
+deployment.apps/postgres scaled
+  postgres 정지
+  ① 캐시를 가진 노드에서 refresh  HTTP 500
+  ② 새 로그인                     HTTP 200
+
+=== DB 복구 후 원복 ===
+deployment.apps/postgres scaled
+deployment "postgres" successfully rolled out
@@ -0,0 +1,12 @@
+=== persistent 모드로 원복 ===
+statefulset.apps/keycloak configured
+partitioned roll out complete: 2 new pods have been updated...
+
+=== [검증] persistent 로 돌아왔는가 — 로그인 후 DB 에 행이 생기는가 ===
+["start"]
+로그인
+  DB 온라인 세션: 1 건  (1 이면 persistent 복귀)
+keycloak-0                  1/1   Running   0     67s
+keycloak-1                  1/1   Running   0     89s
+postgres-7b474b88c8-t6rrf   1/1   Running   0     2m8s
+  외부 진입점 HTTP 200
@@ -0,0 +1,23 @@
+# A-7 — volatile 모드 비교 증거
+
+2026-09-04 13:45–14:15 KST
+해설: [`docs/experiment-a7-volatile-comparison.md`](../../experiment-a7-volatile-comparison.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-switch-to-volatile.txt` | `--features-disabled=persistent-user-sessions` 적용, args 확인 |
+| `02-a0-rerun.txt` | **DB 0건**인데 교차 노드 refresh `200` — 경로가 DB 에서 클러스터로 바뀌었다 |
+| `03-a8-rerun-restart.txt` | **롤링 재시작 후 `400 Session not active`** — persistent 에서는 `200` 이었다 |
+| `04-a1-rerun-partition.txt` | **7800 차단 시 교차 노드 `400`** — persistent 에서는 `200`. 대조군(같은 노드)은 `200` 유지 |
+| `05-a2-rerun-db-loss.txt` | DB 정지 중 **새 로그인 `200`**(persistent 에서는 500), refresh 는 `500` |
+| `06-restore-persistent.txt` | 원복 확인 — `args: ["start"]`, 로그인 후 DB 1건, 외부 200 |
+
+## 뒤집힌 결과
+
+| 실험 | persistent | volatile |
+|---|---|---|
+| A-1 7800 차단 후 교차 refresh | `200` | **`400`** |
+| A-8 롤링 재시작 후 refresh | `200` | **`400`** |
+| A-2 DB 정지 중 새 로그인 | `500` | **`200`** |
+
+**같은 주입, 같은 관측, 정반대 결과.** A층 전체가 버전 조건부임을 보여주는 대조군이다.
@@ -0,0 +1,17 @@
+=== [1] 재시작 전 로그인 — 토큰을 파드 안에 보관 ===
+  sid = XLcgQWRiJrTkuNZcJsNeT_2j
+  DB 세션 수: 151
+
+=== [2] 롤링 재시작 중 가용성 — 5초 간격으로 외부 진입점 확인 ===
+statefulset.apps/keycloak restarted
+200 Waiting for partitioned roll out to finish: 0 out of 2 new pods have been updated...
+Waiting for 1 pods to be ready...
+Waiting for 1 pods to be ready...
+Waiting for 1 pods to be ready...
+200 200 200 200 Waiting for partitioned roll out to finish: 1 out of 2 new pods have been updated...
+Waiting for 1 pods to be ready...
+Waiting for 1 pods to be ready...
+Waiting for 1 pods to be ready...
+200 200 200 200 partitioned roll out complete: 2 new pods have been updated...
+
+  (위 숫자열이 재시작 중 외부 응답 코드의 시계열)
@@ -0,0 +1,19 @@
+=== [3] 재시작 전 발급한 refresh token 이 아직 통하는가 ===
+  대상 sid: XLcgQWRiJrTkuNZcJsNeT_2j
+  keycloak-0 에서 refresh  HTTP 200
+
+=== [4] DB 에 그 세션이 남아 있는가 ===
+     user_session_id      | created_on | last_session_refresh 
+--------------------------+------------+----------------------
+ XLcgQWRiJrTkuNZcJsNeT_2j | 1788495513 |           1788495577
+(1 row)
+
+  전체 온라인 세션: 151 (재시작 전 151)
+
+=== [5] 캐시는 어떻게 되었는가 ===
+  keycloak-0  sessions 캐시 0.0 건 / cluster_size 2.0
+  keycloak-1  sessions 캐시 1.0 건 / cluster_size 2.0
+
+=== [6] 파드 나이 — 정말 재시작되었나 ===
+keycloak-0                  1/1   Running   0     44s
+keycloak-1                  1/1   Running   0     66s
@@ -0,0 +1,16 @@
+# A-8 — 롤링 재시작 증거
+
+2026-09-04 13:38–13:41 KST
+해설: [`docs/experiment-a8-rolling-restart.md`](../../experiment-a8-rolling-restart.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-restart-availability.txt` | 재시작 전 로그인(sid 기록), 롤링 재시작 중 외부 진입점 **9회 모두 `200`** |
+| `02-session-survival.txt` | 재시작 전 토큰으로 refresh **`200`**, DB 행 생존(`last_session_refresh` 갱신 확인), **세션 수 151 → 151**, 캐시 0으로 초기화, 파드 나이 44초/66초 |
+| `a8-cache-reset-cluster-reformed.png` | Grafana — 세션 캐시가 0 으로 떨어지고 `cluster_size` 가 다시 2 가 되는 구간 |
+
+## 핵심 세 줄
+
+1. **무중단이었다.** 한 번에 하나씩 내리고 readiness 가 전환을 맞춰준다 — replica ≥ 2 가 전제.
+2. **세션은 살아남고 캐시만 사라진다.** DB 151건 그대로, 재시작 전 토큰이 그대로 통한다.
+3. **이것이 `persistent-user-sessions` 를 켜는 진짜 이유다.** A-7(volatile)에서 정반대가 나와야 한다.
@@ -0,0 +1,21 @@
+=== 배포 전 자원 ===
+Mem:           11648        7329         280           4        4377        4319
+NAME       CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   
+kc-lab-1   115m         5%       2192Mi          44%         
+kc-lab-2   121m         6%       1324Mi          33%         
+
+=== 배포 ===
+secret/bff-secrets created
+deployment.apps/redis created
+service/redis created
+deployment.apps/bff created
+service/bff created
+ingress.networking.k8s.io/bff created
+
+deployment "redis" successfully rolled out
+Waiting for deployment "bff" rollout to finish: 1 of 2 updated replicas are available...
+deployment "bff" successfully rolled out
+
+bff-574c6d658b-8cz4x        true   kc-lab-1
+bff-574c6d658b-zpkbp        true   kc-lab-2
+redis-568bd7c4-5c5vc        true   kc-lab-2
@@ -0,0 +1,13 @@
+=== B-0: 자동구성이 실제로 고른 구현체 ===
+  Q1 확인한 사실: "코드에 저장소를 직접 생성하는 Bean 이 없기 때문에,
+   어떤 구현체가 실제로 사용되는지는 자동구성 결과까지 확인해야 정확하게 알 수 있다"
+
+  File "<stdin>", line 9
+    print(f"    {name:46} {t.rsplit(\".\",1)[-1]}")
+                                     ^
+SyntaxError: unexpected character after line continuation character
+
+=== HttpSession 은 어디에 있는가 (서블릿 컨테이너 기본) ===
+
+=== 외부 진입점 ===
+  https://app1.hyeonworks.com/  HTTP 200
@@ -0,0 +1,29 @@
+=== B-0 — 자동구성이 실제로 고른 구현체 ===
+  컨텍스트: keycloak-bff
+  전체 빈 수: 321
+
+  --- 세션 · 토큰 저장소 관련 ---
+    authorizedClientManager                        -> AuthorizedClientServiceOAuth2AuthorizedClientManager
+    authorizedClientManagerRegistrar               -> OAuth2ClientConfiguration$OAuth2AuthorizedClientManagerRegistrar
+    authorizedClientRepository                     -> AuthenticatedPrincipalOAuth2AuthorizedClientRepository
+    authorizedClientService                        -> InMemoryOAuth2AuthorizedClientService
+    org.springframework.boot.autoconfigure.security.oauth2.client.OAuth2ClientConfigurations$OAuth2AuthorizedClientServiceConfiguration -> OAuth2ClientConfigurations$OAuth2AuthorizedClientServiceConfiguration
+    org.springframework.security.config.annotation.web.configuration.OAuth2ClientConfiguration$OAuth2AuthorizedClientManagerConfiguration -> OAuth2ClientConfiguration$OAuth2AuthorizedClientManagerConfiguration
+
+  --- OAuth2 클라이언트 관련 전체 ---
+    authorizedClientManager                        -> AuthorizedClientServiceOAuth2AuthorizedClientManager
+    authorizedClientManagerRegistrar               -> OAuth2ClientConfiguration$OAuth2AuthorizedClientManagerRegistrar
+    authorizedClientRepository                     -> AuthenticatedPrincipalOAuth2AuthorizedClientRepository
+    authorizedClientService                        -> InMemoryOAuth2AuthorizedClientService
+    clientRegistrationRepository                   -> InMemoryClientRegistrationRepository
+    org.springframework.boot.autoconfigure.security.oauth2.client.OAuth2ClientAutoConfiguration -> OAuth2ClientAutoConfiguration
+    org.springframework.boot.autoconfigure.security.oauth2.client.OAuth2ClientConfigurations$ClientRegistrationRepositoryConfiguration -> OAuth2ClientConfigurations$ClientRegistrationRepositoryConfiguration
+    org.springframework.boot.autoconfigure.security.oauth2.client.OAuth2ClientConfigurations$OAuth2AuthorizedClientServiceConfiguration -> OAuth2ClientConfigurations$OAuth2AuthorizedClientServiceConfiguration
+    org.springframework.boot.autoconfigure.security.oauth2.client.servlet.OAuth2ClientWebSecurityAutoConfiguration -> OAuth2ClientWebSecurityAutoConfiguration
+    org.springframework.security.config.annotation.web.configuration.OAuth2ClientConfiguration -> OAuth2ClientConfiguration
+    org.springframework.security.config.annotation.web.configuration.OAuth2ClientConfiguration$OAuth2AuthorizedClientManagerConfiguration -> OAuth2ClientConfiguration$OAuth2AuthorizedClientManagerConfiguration
+    org.springframework.security.config.annotation.web.configuration.OAuth2ClientConfiguration$OAuth2ClientWebMvcSecurityConfiguration -> OAuth2ClientConfiguration$OAuth2ClientWebMvcSecurityConfiguration
+    spring.security.oauth2.client-org.springframework.boot.autoconfigure.security.oauth2.client.OAuth2ClientProperties -> OAuth2ClientProperties
+
+  --- Redis / Spring Session 이 구성되었는가 ---
+    ★ 없음 — Redis 도 Spring Session 도 구성되지 않았다
@@ -0,0 +1,18 @@
+# B-0 — BFF·Redis 배포와 자동구성 확인 증거
+
+2026-09-04 14:20–14:50 KST
+해설: [`docs/experiment-b0-bff-redis-deploy.md`](../../experiment-b0-bff-redis-deploy.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-deploy.txt` | 배포 전 자원, Redis·BFF 롤아웃, 두 노드에 하나씩 배치됨 |
+| `02-autoconfiguration.txt` | 첫 조회 시도(파싱 실패)와 **외부 진입점 `HTTP 200`** |
+| `03-beans-analysis.txt` | **B-0 의 답** — `InMemoryOAuth2AuthorizedClientService`, `AuthenticatedPrincipalOAuth2AuthorizedClientRepository`, **Redis·Spring Session 없음** |
+| `b0-bff-login-success-single-replica.png` | replica 1 에서 로그인 성공한 BFF 화면 |
+| `b0-bff-token-boundary.png` | `/bff/token-boundary` — `principal: labuser`, `accessTokenStoredOnServer: true`, **`browserTokenCount: 0`** |
+
+## 핵심 세 줄
+
+1. **`AuthenticatedPrincipalOAuth2AuthorizedClientRepository`** — 조회 키가 principal 이고 session ID 가 없다. Q1·Q3 문제의 기제가 이 빈 하나에 있다.
+2. **Redis 를 붙여도 그건 안 고쳐진다.** 저장소 공유와 조회 키는 다른 문제다.
+3. **replica 2개에서는 로그인 자체가 실패한다.** 인가 코드 흐름의 왕복 두 번이 같은 인스턴스로 가야 하는데, 인가 요청이 인스턴스 메모리에 있다.
@@ -0,0 +1,5 @@
+deployment.apps/bff configured
+deployment "bff" successfully rolled out
+bff-576d869c6d-bshvl   true   kc-lab-2
+bff-695646ddb-kzs9k    true   kc-lab-1
+bff-695646ddb-vjqzf    true   kc-lab-2
@@ -0,0 +1,64 @@
+=== B-1 — Redis 를 붙인 뒤 자동구성이 실제로 바뀌었는가 ===
+  빈 수: 321 → 402  (+81)
+
+  --- 세션 저장소 관련 (새로 생긴 것) ---
+    ★ cookieSerializer                           -> DefaultCookieSerializer
+    ★ org.springframework.boot.autoconfigure.session.RedisSessionConfiguration -> RedisSessionConfiguration
+    ★ org.springframework.boot.autoconfigure.session.RedisSessionConfiguration$DefaultRedisSessionConfiguration -> RedisSessionConfiguration$DefaultRedisSessionConfiguration
+    ★ org.springframework.boot.autoconfigure.session.SessionAutoConfiguration -> SessionAutoConfiguration
+    ★ org.springframework.boot.autoconfigure.session.SessionAutoConfiguration$ServletSessionConfiguration -> SessionAutoConfiguration$ServletSessionConfiguration
+    ★ org.springframework.boot.autoconfigure.session.SessionAutoConfiguration$ServletSessionConfiguration$RememberMeServicesConfiguration -> SessionAutoConfiguration$ServletSessionConfiguration$RememberMeServicesConfiguration
+    ★ org.springframework.boot.autoconfigure.session.SessionAutoConfiguration$ServletSessionConfiguration$ServletSessionRepositoryConfiguration -> SessionAutoConfiguration$ServletSessionConfiguration$ServletSessionRepositoryConfiguration
+    ★ org.springframework.boot.autoconfigure.session.SessionRepositoryFilterConfiguration -> SessionRepositoryFilterConfiguration
+    ★ org.springframework.session.config.annotation.web.http.SpringHttpSessionConfiguration -> SpringHttpSessionConfiguration
+    ★ org.springframework.session.data.redis.config.annotation.web.http.RedisHttpSessionConfiguration -> RedisHttpSessionConfiguration
+    ★ rememberMeServicesCookieSerializerCustomizer -> SessionAutoConfiguration$ServletSessionConfiguration$RememberMeServicesConfiguration$$Lambda/0x00007f364e69fa60
+    ★ sessionEventHttpSessionListenerAdapter     -> SessionEventHttpSessionListenerAdapter
+    ★ sessionRepository                          -> RedisSessionRepository
+    ★ sessionRepositoryFilterRegistration        -> DelegatingFilterProxyRegistrationBean
+    ★ spring.session-org.springframework.boot.autoconfigure.session.SessionProperties -> SessionProperties
+    ★ spring.session.redis-org.springframework.boot.autoconfigure.session.RedisSessionProperties -> RedisSessionProperties
+    ★ springBootSessionRepositoryCustomizer      -> RedisSessionConfiguration$DefaultRedisSessionConfiguration$$Lambda/0x00007f364e6a4a68
+    ★ springSessionRepositoryFilter              -> SessionRepositoryFilter
+
+  --- OAuth2 authorized client — 바뀌었는가? ---
+    authorizedClientService
+      before: InMemoryOAuth2AuthorizedClientService
+      after : InMemoryOAuth2AuthorizedClientService    그대로 — Redis 로 안 옮겨졌다
+    authorizedClientRepository
+      before: AuthenticatedPrincipalOAuth2AuthorizedClientRepository
+      after : AuthenticatedPrincipalOAuth2AuthorizedClientRepository    그대로 — Redis 로 안 옮겨졌다
+    authorizedClientManager
+      before: AuthorizedClientServiceOAuth2AuthorizedClientManager
+      after : AuthorizedClientServiceOAuth2AuthorizedClientManager    그대로 — Redis 로 안 옮겨졌다
+
+  --- Redis 연결 빈 (새로 생긴 것) ---
+    ★ keyValueMappingContext                     -> RedisMappingContext
+    ★ lettuceMetrics                             -> LettuceMetricsAutoConfiguration$$Lambda/0x00007f364e56f4d0
+    ★ org.springframework.boot.actuate.autoconfigure.data.redis.RedisHealthContributorAutoConfiguration -> RedisHealthContributorAutoConfiguration
+    ★ org.springframework.boot.actuate.autoconfigure.data.redis.RedisReactiveHealthContributorAutoConfiguration -> RedisReactiveHealthContributorAutoConfiguration
+    ★ org.springframework.boot.actuate.autoconfigure.metrics.redis.LettuceMetricsAutoConfiguration -> LettuceMetricsAutoConfiguration
+    ★ org.springframework.boot.autoconfigure.data.redis.LettuceConnectionConfiguration -> LettuceConnectionConfiguration
+    ★ org.springframework.boot.autoconfigure.data.redis.RedisAutoConfiguration -> RedisAutoConfiguration
+    ★ org.springframework.boot.autoconfigure.data.redis.RedisReactiveAutoConfiguration -> RedisReactiveAutoConfiguration
+    ★ org.springframework.boot.autoconfigure.data.redis.RedisRepositoriesAutoConfiguration -> RedisRepositoriesAutoConfiguration
+    ★ org.springframework.boot.autoconfigure.session.RedisSessionConfiguration -> RedisSessionConfiguration
+    ★ org.springframework.boot.autoconfigure.session.RedisSessionConfiguration$DefaultRedisSessionConfiguration -> RedisSessionConfiguration$DefaultRedisSessionConfiguration
+    ★ org.springframework.session.data.redis.config.annotation.web.http.RedisHttpSessionConfiguration -> RedisHttpSessionConfiguration
+    ★ reactiveRedisTemplate                      -> ReactiveRedisTemplate
+    ★ reactiveStringRedisTemplate                -> ReactiveStringRedisTemplate
+    ★ redisConnectionDetails                     -> PropertiesRedisConnectionDetails
+    ★ redisConnectionFactory                     -> LettuceConnectionFactory
+    ★ redisConverter                             -> MappingRedisConverter
+    ★ redisCustomConversions                     -> RedisCustomConversions
+    ★ redisHealthContributor                     -> RedisReactiveHealthIndicator
+    ★ redisKeyValueAdapter                       -> RedisKeyValueAdapter
+    ★ redisKeyValueTemplate                      -> RedisKeyValueTemplate
+    ★ redisMappingConfiguration#0                -> MappingConfiguration
+    ★ redisReferenceResolver                     -> ReferenceResolverImpl
+    ★ redisTemplate                              -> RedisTemplate
+    ★ sessionRepository                          -> RedisSessionRepository
+    ★ spring.data.redis-org.springframework.boot.autoconfigure.data.redis.RedisProperties -> RedisProperties
+    ★ spring.session.redis-org.springframework.boot.autoconfigure.session.RedisSessionProperties -> RedisSessionProperties
+    ★ springBootSessionRepositoryCustomizer      -> RedisSessionConfiguration$DefaultRedisSessionConfiguration$$Lambda/0x00007f364e6a4a68
+    ★ stringRedisTemplate                        -> StringRedisTemplate
@@ -0,0 +1,21 @@
+=== Redis 에 무엇이 들어 있는가 ===
+bff:session:sessions:8963b6de-3564-4775-9ccd-1ee9616b83ae
+  총 키 수: 1
+
+=== 세션 키의 내용 — refresh token 이 있는가 (Q3 검증 2번) ===
+  키: bff:session:sessions:8963b6de-3564-4775-9ccd-1ee9616b83ae
+  타입: hash
+    필드: sessionAttr:SPRING_SECURITY_CONTEXT
+    필드: sessionAttr:SPRING_SECURITY_SAVED_REQUEST
+    필드: sessionAttr:SPRING_SECURITY_LAST_EXCEPTION
+    필드: sessionAttr:org.springframework.security.oauth2.client.web.HttpSessionOAuth2AuthorizationRequestRepository.AUTHORIZATION_REQUEST
+    필드: lastAccessedTime
+    필드: maxInactiveInterval
+    필드: creationTime
+
+=== 필드 값에 토큰 문자열이 보이는가 ===
+ 1) "sessionAttr:SPRING_SECURITY_CONTEXT"
+ 2) "\xac\xed\x00\x05sr\x00=org.springframework.security.core.context.SecurityContextImpl\x00\x00\x00\x00\x00\x00\x02l\x02\x00\x01L\x00\x0eauthenticationt\x002Lorg/springframework/security/core/Authentication;xpsr\x00Sorg.springframework.security.oauth2.client.authentication.OAuth2AuthenticationToken\x00\x00\x00\x00\x00\x00\x02l\x02\x00\x02L\x00\x1eauthorizedClientRegistrationIdt\x00\x12Ljava/lang/String;L\x00\tprincipalt\x00:Lorg/springframework/security/oauth2/core/user/OAuth2User;xr\x00Gorg.springframework.security.authentication.AbstractAuthenticationToken\xd3\xaa(~nGd\x0e\x02\x00\x03Z\x00\rauthenticatedL\x00\x0bauthoritiest\x00\x16Ljava/util/Collection;L\x00\adetailst\x00\x12Ljava/lang/Object;xp\x01sr\x00&java.util.Collections$UnmodifiableList\xfc\x0f%1\xb5\xec\x8e\x10\x02\x00\x01L\x00\x04listt\x00\x10Ljava/util/List;xr\x00,java.util.Collect
+
+=== TTL (Q3 검증 3번 — session TTL) ===
+  TTL: 1772 초
@@ -0,0 +1,18 @@
+# B-1 — Redis 세션 저장소 전환 증거
+
+2026-09-04 14:50–15:05 KST
+해설: [`docs/experiment-b1-redis-session-store.md`](../../experiment-b1-redis-session-store.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-servicelinks-trap.txt` | `enableServiceLinks: false` 적용 후 롤아웃 성공 — 쿠버네티스가 주입한 `REDIS_PORT=tcp://...` 가 설정을 덮어쓴 문제 |
+| `02-autoconfig-after.txt` | **핵심** — 빈 321→402(+81). `sessionRepository → RedisSessionRepository` 로 바뀌었지만 **`authorizedClientService` 는 `InMemory` 그대로** |
+| `03-redis-contents.txt` | Redis 키 1개, 필드는 `SPRING_SECURITY_CONTEXT` 뿐. **토큰 없음.** Java 직렬화(`\xac\xed`), TTL 1772초 |
+| `b1-login-works-two-replicas.png` | 전환 직후 `accessTokenStoredOnServer: false` |
+| `b1-token-boundary-after-redis.png` | 파드 전면 교체 후 — `principal: labuser` 는 살아남고 토큰만 사라진 상태 |
+
+## 핵심 세 줄
+
+1. **세션은 옮겨졌고 토큰은 안 옮겨졌다.** 빈 81개가 늘었는데 authorized client 관련은 하나도 안 바뀌었다.
+2. **refresh token 은 Redis 에 평문으로 있는 게 아니라 아예 없다.** 암호화를 고민하기 전에 이걸 알아야 한다.
+3. **"로그인은 되어 있는데 아무것도 못 하는" 상태가 만들어진다** — 완전 로그아웃보다 나쁘다.
@@ -0,0 +1,8 @@
+deployment.apps/bff configured
+deployment "bff" successfully rolled out
+bff-555df79c97-6j86w   1/1   Running       0     44s
+bff-555df79c97-vgg6g   1/1   Running       0     22s
+
+=== oauth2_authorized_client 테이블이 생겼는가 ===
+Did not find any relation named "oauth2_authorized_client".
+command terminated with exit code 1
@@ -0,0 +1,33 @@
+=== PostgreSQL 전용 스키마 ===
+CREATE TABLE oauth2_authorized_client (
+  client_registration_id varchar(100) NOT NULL,
+  principal_name varchar(200) NOT NULL,
+  access_token_type varchar(100) NOT NULL,
+  access_token_value bytea NOT NULL,
+  access_token_issued_at timestamp NOT NULL,
+  access_token_expires_at timestamp NOT NULL,
+  access_token_scopes varchar(1000) DEFAULT NULL,
+  refresh_token_value bytea DEFAULT NULL,
+  refresh_token_issued_at timestamp DEFAULT NULL,
+  created_at timestamp DEFAULT CURRENT_TIMESTAMP NOT NULL,
+  PRIMARY KEY (client_registration_id, principal_name)
+);
+
+=== 적용 ===
+CREATE TABLE
+                                Table "public.oauth2_authorized_client"
+         Column          |            Type             | Collation | Nullable |         Default         
+-------------------------+-----------------------------+-----------+----------+-------------------------
+ client_registration_id  | character varying(100)      |           | not null | 
+ principal_name          | character varying(200)      |           | not null | 
+ access_token_type       | character varying(100)      |           | not null | 
+ access_token_value      | bytea                       |           | not null | 
+ access_token_issued_at  | timestamp without time zone |           | not null | 
+ access_token_expires_at | timestamp without time zone |           | not null | 
+ access_token_scopes     | character varying(1000)     |           |          | NULL::character varying
+ refresh_token_value     | bytea                       |           |          | 
+ refresh_token_issued_at | timestamp without time zone |           |          | 
+ created_at              | timestamp without time zone |           | not null | CURRENT_TIMESTAMP
+Indexes:
+    "oauth2_authorized_client_pkey" PRIMARY KEY, btree (client_registration_id, principal_name)
+
@@ -0,0 +1,20 @@
+=== Q3 검증 2번 — 저장소를 직접 열어 refresh token 이 평문인가 ===
+eyJhbGciOiJIUzUxMiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJlMmUzZDZkMy0yNzQyLTRhYWItYjk4Ni02ZDU2ZDM5MDk1ZDEifQ.eyJleHAiOjE3ODg1MDA0NDYsImlhdCI6MTc4ODQ5ODY0NiwianRpIjoiNTQwOTZmYTQtZWRjNi1iZjZkLWE4OGMtZDJhNjEzOGJjNmVlIiwiaXNzIjoiaHR0cHM6Ly9hdXRoLmh5ZW9ud29ya3MuY29tL3JlYWxtcy9rZXljbG9hay1wYXR0ZXJucyIsImF1ZCI6I
+
+=== access token 도 ===
+eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJPWS1jYVlETkdvUDRITUF6LVE5VVBUVS1ETTFpODk2TnV6VVp1NmdmQ3FNIn0.eyJleHAi
+
+=== 그 문자열이 실제 JWT 인지 — 헤더를 디코드 ===
+  File "<string>", line 3
+    h=open(/tmp/hdr.txt).read().strip()
+           ^
+SyntaxError: invalid syntax
+
+=== 저장된 바이트를 그대로 디코드한 결과 ===
+  refresh_token 헤더 : {"alg":"HS512","typ" : "JWT","kid" : "e2e3d6d3-2742-4aab-b986-6d56d39095d1"}
+  refresh_token 페이로드(앞부분):
+    {"exp":1788500446,"iat":1788498646,"jti":"54096fa4-edc6-bf6d-a88c-d2a6138bc6ee","iss":"https://auth.hyeonworks.com/realms/keycloak-patterns"
+  access_token 헤더  : {"alg":"RS256","typ" : "JWT","kid" : "OY-caYDNGoP4HMAz-Q9UPTU-DM1i896NuzUZu6gfCqM"}
+
+  → bytea 에 들어 있는 것은 암호화된 덩어리가 아니라 JWT 문자열 그대로다.
+    DB 읽기 권한만 있으면 그 자리에서 쓸 수 있는 토큰을 얻는다.
@@ -0,0 +1,25 @@
+=== [현재] 같은 사용자의 항목 ===
+ client_registration_id | principal_name |   access_token_issued_at   |              at_md5              
+------------------------+----------------+----------------------------+----------------------------------
+ keycloak               | labuser        | 2026-09-04 05:10:46.927192 | 675af2286bfc2fd9d2bab7bc8f391df7
+(1 row)
+
+  행 수: 1
+
+=== [모의 두 번째 브라우저] 세션만 지우고 같은 사용자로 다시 로그인시킨다 ===
+  (브라우저가 달라도 principal 은 같으므로 조회 키가 같다)
+  Redis 세션 삭제 완료 — 다음 요청이 새 로그인을 만든다
+=== [재로그인 후] 행이 늘었는가, 덮어써졌는가 ===
+ client_registration_id | principal_name |   access_token_issued_at   |              at_md5              
+------------------------+----------------+----------------------------+----------------------------------
+ keycloak               | labuser        | 2026-09-04 05:12:13.018828 | e19a63fc5aa18bd0a68b3e19dff16b3b
+(1 row)
+
+  행 수: 1
+
+  ★ 행 수가 1 그대로이고 md5 가 바뀌었으면 → 덮어쓰기다
+
+=== Q1 검증 ④ — 로그아웃하면 두 저장소가 다 정리되는가 ===
+  로그아웃 전
+    Redis: 1 키
+    PostgreSQL: 1 행
@@ -0,0 +1,14 @@
+=== Q1 검증 ④ — 로그아웃 후 두 저장소 상태 ===
+  Redis 세션      : 0 키
+  PostgreSQL 토큰 : 1 행
+
+ principal_name |   access_token_issued_at   |  access_token_expires_at   
+----------------+----------------------------+----------------------------
+ labuser        | 2026-09-04 05:12:13.018828 | 2026-09-04 05:13:13.018828
+(1 row)
+
+
+  ★ Redis 는 비었는데 PostgreSQL 에 행이 남아 있으면 → 한쪽만 정리된 것
+
+=== Keycloak 쪽 SSO 세션은? ===
+  Keycloak 온라인 세션: 2
@@ -0,0 +1,21 @@
+# B-2 — 다중 인스턴스 운영 증거
+
+2026-09-04 15:05–15:15 KST
+해설: [`docs/experiment-b2-multi-instance-session.md`](../../experiment-b2-multi-instance-session.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-jdbc-store-deploy.txt` | JDBC 저장소로 배포. **테이블이 조용히 안 만들어졌다** |
+| `02-schema.txt` | 원인 — 기본 DDL 은 `blob`(PostgreSQL 에 없음), `-postgres.sql` 판본이 따로 있다. **`PRIMARY KEY (client_registration_id, principal_name)`** — 조회 키 문제가 DDL 에 박혀 있다 |
+| `03-plaintext-tokens.txt` | **Q3 검증 2번** — `bytea` 안이 JWT 문자열 그대로. 디코드하면 `{"alg":"HS512",...}` |
+| `04-overwrite-test.txt` | **Q1 검증 3번** — 같은 사용자 재로그인 시 행 수 1 그대로, `issued_at` 과 md5 만 바뀜 = **UPDATE(덮어쓰기)** |
+| `05-logout-cleanup.txt` | **Q1 검증 4번** — Redis 0키 / PostgreSQL **1행 잔존** / Keycloak SSO **2세션 잔존** |
+| `b2-before-relogin.png` | JDBC 전환 직후, 옛 세션은 여전히 `false` |
+| `b2-tokens-shared-across-instances.png` | 재로그인 후 **`accessTokenStoredOnServer: true`** — 두 replica 에서 동작 |
+
+## 핵심 네 줄
+
+1. **세션 Redis + 토큰 PostgreSQL 분리 저장이 성립한다.** B-1 의 "로그인은 됐는데 토큰이 없는" 상태가 해결됐다.
+2. **refresh token 은 평문이다.** DB 읽기 권한이면 작동하는 토큰을 얻는다.
+3. **같은 사용자의 두 번째 로그인이 첫 번째를 덮어쓴다.** 기본키에 session id 가 없어 구조적으로 그렇다.
+4. **로그아웃은 셋 중 하나만 지운다.** 평문 토큰과 Keycloak SSO 세션이 남는다.
@@ -0,0 +1,11 @@
+=== [1] refresh token 하나 확보 ===
+  토큰 길이: 811
+  jti: 8e7e3ee2-0dc8-573d-58ec-d12651a50b9c
+  sid: BvFiB01Rntz1FcLdf7zG4BNt
+
+=== [2] 같은 refresh token 으로 동시에 5회 갱신 ===
+  요청 1: HTTP 400  {"error":"invalid_grant","error_description":"Maximum allowed refresh token reuse exceeded"}
+  요청 2: HTTP 400  {"error":"invalid_grant","error_description":"Session doesn't have required client"}
+  요청 3: HTTP 400  {"error":"invalid_grant","error_description":"Session doesn't have required client"}
+  요청 4: HTTP 400  {"error":"invalid_grant","error_description":"Session doesn't have required client"}
+  요청 5: HTTP 200  {"access_token":"...(발급됨)
@@ -0,0 +1,16 @@
+=== [3] 이긴 요청이 받은 새 토큰은 쓸 수 있는가 ===
+  새 refresh token 길이: 810
+  그 토큰으로 다시 갱신:  HTTP 400
+    {"error":"invalid_grant","error_description":"Session doesn't have required client"}
+
+=== [4] 그 sid 의 세션이 DB 에 남아 있는가 ===
+     user_session_id      | offline_flag | last_session_refresh 
+--------------------------+--------------+----------------------
+ BvFiB01Rntz1FcLdf7zG4BNt | 0            |           1788498996
+(1 row)
+
+=== [5] revoked_token 테이블 ===
+ revoked_count 
+---------------
+             0
+(1 row)
@@ -0,0 +1,14 @@
+=== user session 과 client session 을 나눠서 본다 ===
+     user_session_id      | offline_flag | client_sessions 
+--------------------------+--------------+-----------------
+ BvFiB01Rntz1FcLdf7zG4BNt | 0            |               0
+(1 row)
+
+
+=== 대조: 정상 세션 하나를 새로 만들어 비교 ===
+  새 sid: JT-XuepgutWcE273QwAnIXta
+     user_session_id      | client_sessions 
+--------------------------+-----------------
+ JT-XuepgutWcE273QwAnIXta |               1
+(1 row)
+
@@ -0,0 +1,24 @@
+=== 구성 A: rotation ON (revokeRefreshToken=true, maxReuse=0) — 앞서 측정 ===
+    성공 1 / 5, 세션 파괴됨
+
+=== 구성 B: rotation OFF (revokeRefreshToken=false) ===
+    sid=iW1CGyO7COdyJLryIrCt3njk
+    1: 200 
+    2: 200 
+    3: 200 
+    4: 200 
+    5: 200 
+    성공 5 / 5
+    이긴 토큰 재사용: HTTP 200
+    남은 client_session: 1
+
+=== 구성 C: rotation ON + 재사용 1회 허용 (maxReuse=1) ===
+    sid=72c04JCdr0NpCHGQmXWW2wM8
+    1: 200 
+    2: 400 "error_description":"Session doesn't have required client"
+    3: 200 
+    4: 400 "error_description":"Maximum allowed refresh token reuse exceeded"
+    5: 400 "error_description":"Session doesn't have required client"
+    성공 2 / 5
+    이긴 토큰 재사용: HTTP 400
+    남은 client_session: 0
@@ -0,0 +1,18 @@
+# B-3 — Refresh Token 동시 갱신 경쟁 증거
+
+2026-09-04 15:15–15:25 KST
+해설: [`docs/experiment-b3-refresh-token-contention.md`](../../experiment-b3-refresh-token-contention.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-concurrent-refresh.txt` | 같은 토큰으로 동시 5회 — **1개만 200**, 나머지는 `Maximum allowed refresh token reuse exceeded` 와 `Session doesn't have required client` **두 종류** 오류 |
+| `02-session-impact.txt` | **★ 이긴 요청의 새 토큰조차 400.** user_session 행은 남아 있고 `revoked_token` 은 0건 |
+| `03-client-session-removed.txt` | **기제 확정 (대조군 포함)** — 경쟁 세션 `client_sessions=0`, 정상 세션 `client_sessions=1` |
+| `04-policy-comparison.txt` | 정책 3종 비교 — rotation OFF 는 **5/5 성공·세션 생존**, maxReuse=1 은 **여전히 세션 파괴** |
+
+## 핵심 네 줄
+
+1. **"하나는 성공"이 아니다.** 이긴 요청이 받은 토큰도 곧바로 쓸 수 없다.
+2. **재사용 탐지가 client session 을 제거한다.** user session 은 껍데기로 남아 `Session doesn't have required client` 가 된다.
+3. **`refreshTokenMaxReuse` 를 올려도 안 된다.** 동시 요청 수만큼 올려야 하고 그러면 rotation 의 목적이 사라진다.
+4. **재시도로 회복되지 않으므로 Q2 의 답은 lock 이다.** 그리고 lock 은 저장소 쪽(가급적 DB 행 잠금)에 있어야 한다.
@@ -0,0 +1,40 @@
+=== Q4 ① 다중 값 role — 구분자와 동명 헤더 ===
+(a) 쉼표 구분 한 개 헤더
+    보냄:   X-Auth-Request-Roles: admin,editor,viewer
+    도착:   ['admin,editor,viewer']          ← 문자열 하나 그대로
+
+(b) 동명 헤더 두 개
+    보냄:   X-Auth-Request-Roles: admin
+            X-Auth-Request-Roles: editor
+    도착:   ['admin', 'editor']              ← ★ 둘 다 도착. 덮어쓰지도 합치지도 않는다
+
+(c) 값 안에 구분자가 들어간 경우
+    보냄:   X-Auth-Request-Roles: role-with,comma
+    도착:   ['role-with,comma']              ← (a) 와 구별 불가
+
+=== Q4 ② 헤더 크기 상한 ===
+  보낸 길이   1000 → HTTP 200, 도착 길이 1000
+  보낸 길이   4000 → HTTP 200, 도착 길이 4000
+  보낸 길이   8000 → HTTP 400  (Tomcat 의 HTML 오류 페이지)
+  보낸 길이  16000 → HTTP 000  (응답을 못 받음 = 연결이 끊김)
+  보낸 길이  32000 → HTTP 000
+
+  → 자르지 않는다. 거부한다. 그리고 거부하는 계층이 둘이며 증상이 다르다.
+
+=== Q4 ④ upstream 이 검증하는가 ===
+  아무 인증 없이 보냄:
+    x-auth-request-user        ['administrator']
+    x-auth-request-email       ['admin@example.com']
+    x-auth-request-roles       ['realm-admin,superuser']
+    remoteAddr                 100.123.124.30
+  → 그대로 도착. 검증 없음.
+
+  대조 — JWT 를 요구하는 경로:
+    /api/echo        HTTP 200   (permitAll)
+    /api/me          HTTP 401
+    /api/protected   HTTP 401
+
+  backend SecurityConfig:
+    .requestMatchers("/actuator/health", "/actuator/health/**", "/api/public", ...).permitAll()
+    .anyRequest().authenticated()
+    .oauth2ResourceServer(oauth2 -> oauth2.jwt(...))
@@ -0,0 +1,14 @@
+# B-4 — Edge 인가 범위 증거
+
+2026-09-04 15:25–15:35 KST
+해설: [`docs/experiment-b4-edge-authorization-scope.md`](../../experiment-b4-edge-authorization-scope.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-header-handling.txt` | ① 동명 헤더가 **둘 다 도착**(`['admin','editor']`)하고 값 안의 쉼표를 구분자와 구별할 수 없다 · ② 8KB 에서 Tomcat 400, 16KB 에서 연결 끊김 — **자르지 않고 거부** · ④ 위조 신원 헤더가 그대로 도착, JWT 경로는 401 |
+
+## 핵심 세 줄
+
+1. **Q4 의 「nginx 가 동명 헤더를 덮어쓴다」는 조건부다.** nginx 는 자기가 `proxy_set_header` 한 헤더만 덮어쓰고, 나머지는 통과시킨다 — 지금 `X-Auth-Request-*` 는 통과한다.
+2. **크기는 절벽이다.** 점진적으로 나빠지지 않고 8KB 에서 전면 400 이 되며, role 이 많은 사용자만 깨진다.
+3. **헤더를 인가 근거로 쓰면 위조 가능성이 곧 권한 상승이다.** 2홉 실험의 결론이 여기서는 신원 자체에 적용된다.
@@ -0,0 +1,9 @@
+=== 기준선 ===
+  Redis 키: 1
+  PostgreSQL 토큰: 1 행
+  Redis 영속화 설정:
+    save = save	
+    appendonly	no
+
+=== 외부 진입점 정상 확인 ===
+  https://app1.hyeonworks.com/  HTTP 200
@@ -0,0 +1,25 @@
+=== ① Redis 정지 ===
+  정지: 14:26:30
+deployment.apps/redis scaled
+  삭제 완료
+
+=== 로그인한 사용자의 다음 요청은 어떻게 되는가 ===
+  /                      HTTP 200
+  /bff/token-boundary    HTTP 000
+  /actuator/health       HTTP 503
+  --- token-boundary 응답 본문 ---
+
+
+=== 파드 상태 — readiness 가 Redis 를 보는가 ===
+bff-555df79c97-6j86w   1/1   Running   0     17m
+bff-555df79c97-vgg6g   1/1   Running   0     16m
+
+=== health 상세 ===
+
+
+=== BFF 로그 ===
+	at java.base/sun.nio.ch.Net.pollConnect(Native Method) ~[na:na]
+	at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source) ~[na:na]
+	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) ~[na:na]
+	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:336) ~[netty-transport-4.1.135.Final.jar!/:4.1.135.Final]
+	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339) ~[netty-transport-4.1.135.Final.jar!/:4.1.135.Final]
@@ -0,0 +1,13 @@
+=== health 그룹별 응답 — 왜 파드는 Ready 인가 ===
+  /actuator/health             HTTP server
+  /actuator/health/readiness   HTTP 200
+  /actuator/health/liveness    HTTP 200
+
+=== /actuator/health 본문 (Redis 항목이 있는가) ===
+
+
+=== /actuator/health/readiness 본문 ===
+{"status":"UP"}
+
+=== Service 엔드포인트 — 트래픽을 계속 받는가 ===
+  ready: [10.42.0.52 10.42.1.124]
@@ -0,0 +1,39 @@
+=== 복구 ===
+deployment.apps/redis scaled
+deployment "redis" successfully rolled out
+  /actuator/health       HTTP 200
+  /bff/token-boundary    HTTP 302
+  BFF 재시작 필요했나: 0,0 회 재시작
+
+=== ② 영속화 — 지금 설정으로 재시작하면 무엇이 남는가 ===
+  키 심음: before-restart
+  dbsize: 4
+
+  --- AOF 를 켜고 다시 심는다 (영속화가 켜져 있으면 살아남는가) ---
+    appendonly	yes
+    total 12
+    drwxr-xr-x    3 redis    redis         4096 Sep  4 05:26 .
+    drwxr-xr-x    1 root     root          4096 Sep  4 05:26 ..
+    drwx------    2 redis    redis         4096 Sep  4 05:26 appendonlydir
+
+  --- 파드를 지운다 ---
+deployment "redis" successfully rolled out
+  재기동 후:
+    dbsize:    0
+    b5:probe   
+    b5:aof     
+    appendonly no
+persistentvolumeclaim/redis-data created
+deployment.apps/redis configured
+deployment "redis" successfully rolled out
+
+=== 영속 볼륨 위에서 다시 시험 ===
+  appendonly	yes
+  키 심음: written-on-pvc
+sed: -e expression #1, char 8: unknown option to 's'
+
+  --- 파드를 지운다 ---
+deployment "redis" successfully rolled out
+  재기동 후:
+    dbsize:  1
+    b5:pvc   written-on-pvc
@@ -0,0 +1,17 @@
+# B-5 — Redis 상실과 영속화 증거
+
+2026-09-04 15:35–15:50 KST
+해설: [`docs/experiment-b5-redis-loss-persistence.md`](../../experiment-b5-redis-loss-persistence.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-baseline.txt` | 정지 전 — Redis 1키, PostgreSQL 1행, `save`/`appendonly no`, 외부 200 |
+| `02-redis-down.txt` | 정지 후 — `/bff/token-boundary` **`HTTP 000`(멈춤)**, `/actuator/health` 503, **파드는 1/1 Ready 유지**, Lettuce 재연결 스택 |
+| `03-health-groups.txt` | **핵심** — `/actuator/health` 503 인데 `/actuator/health/readiness` 는 `{"status":"UP"}`. Service 엔드포인트에 두 파드 모두 남아 있다 |
+| `04-persistence.txt` | 복구는 자동(재시작 0회) · **AOF 를 켰는데 파드 삭제 후 `dbsize 0`** · PVC 를 붙인 뒤 `written-on-pvc` **생존** |
+
+## 핵심 세 줄
+
+1. **파드가 Ready 를 유지한 채 계속 실패한다.** `redis` 헬스 지표가 readiness 그룹에 없기 때문이며, A-2 에서 Keycloak 이 NotReady 가 된 것과 정반대다.
+2. **오류가 아니라 멈춤이다.** `HTTP 000` — 빠른 실패가 안 되어 있어 사용자는 멈춘 화면을 본다.
+3. **볼륨 없이 AOF 만 켜는 것은 장식이다.** `appendonlydir` 까지 만들어지지만 컨테이너와 함께 사라진다.
@@ -0,0 +1,9 @@
+=== [1] 회전 전: 토큰 발급 + JWKS 상태 ===
+  발급 토큰의 kid: OY-caYDNGoP4HMAz-Q9UPTU-DM1i896NuzUZu6gfCqM
+  JWKS 의 RS256 키 수: 1
+  JWKS kid 목록:
+    {"keys":[{"kid":"gokjn0zFUok8r7JVqW1cxuyojH1bTT87vzfQG9RrFX4"
+    {"kid":"OY-caYDNGoP4HMAz-Q9UPTU-DM1i896NuzUZu6gfCqM"
+
+=== [2] 그 토큰이 지금 통하는가 (리소스 서버) ===
+  /api/me  HTTP 200
@@ -0,0 +1,16 @@
+=== [3] 키 회전 — 우선순위가 더 높은 RSA 공급자를 추가한다 ===
+Created new component with id '7902af43-a0cc-4ebd-ad25-04d563854d16'
+
+=== [4] 회전 후 JWKS — 옛 키가 남아 있는가 ===
+  RS256 키 수: 2
+  kid 목록:
+    {"keys":[{"kid":"1B4AQHoxZvFaQi1tc1byz8ifU-nYFB6engD4YB4Fz84"
+    {"kid":"gokjn0zFUok8r7JVqW1cxuyojH1bTT87vzfQG9RrFX4"
+    {"kid":"OY-caYDNGoP4HMAz-Q9UPTU-DM1i896NuzUZu6gfCqM"
+
+=== [5] 새 토큰은 어느 키로 서명되는가 ===
+  새 토큰의 kid: 1B4AQHoxZvFaQi1tc1byz8ifU-nYFB6engD4YB4Fz84
+
+=== [6] ★ 회전 전에 발급된 토큰은 아직 통하는가 ===
+  옛 토큰  /api/me  HTTP 200
+  새 토큰  /api/me  HTTP 200
@@ -0,0 +1,16 @@
+=== [7] 옛 RSA 공급자(980ee9b7 = OY-caYDN 키) 제거 ===
+  제거 완료
+
+=== [8] JWKS 에서 사라졌는가 ===
+  RS256 키 수: 1
+    {"keys":[{"kid":"1B4AQHoxZvFaQi1tc1byz8ifU-nYFB6engD4YB4Fz84"
+    {"kid":"gokjn0zFUok8r7JVqW1cxuyojH1bTT87vzfQG9RrFX4"
+
+=== [9] ★ 옛 키로 서명된 토큰은 이제 어떻게 되는가 ===
+  옛 토큰  /api/me  HTTP 401  (캐시가 살아 있으면 아직 통할 수 있다)
+  새 토큰  /api/me  HTTP 200
+
+=== [10] 리소스 서버를 재시작해 JWKS 캐시를 비우면 ===
+deployment "echo" successfully rolled out
+  옛 토큰  /api/me  HTTP 401
+  새 토큰  /api/me  HTTP 200
@@ -0,0 +1,16 @@
+# B-6 — key 회전 증거
+
+2026-09-04 15:50–16:00 KST
+해설: [`docs/experiment-b6-key-rotation.md`](../../experiment-b6-key-rotation.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-before-rotation.txt` | 회전 전 — 토큰 `kid=OY-caYDN...`, JWKS RS256 1개, `/api/me` 200 |
+| `02-rotation.txt` | 우선순위 200 공급자 추가 → **JWKS RS256 2개**, 새 토큰은 새 kid, **옛 토큰도 새 토큰도 200** (무중단) |
+| `03-old-key-removed.txt` | 옛 공급자 제거 → JWKS 1개, **옛 토큰 즉시 401**. 리소스 서버 재시작 후에도 동일 |
+
+## 핵심 세 줄
+
+1. **Keycloak 의 키 회전은 "바꾸기"가 아니라 "더 높은 우선순위로 추가하기"** 다. 추가만으로는 아무것도 안 깨진다.
+2. **위험한 것은 옛 키를 버리는 시점이다.** 제거 즉시 그 키로 서명된 토큰이 401 이 된다.
+3. **캐시는 유예가 아니다.** 모르는 `kid` 를 만나면 JWKS 를 다시 받으므로 제거가 곧바로 반영된다. 유예는 옛 키를 남겨두는 기간으로 만들어야 한다.
@@ -0,0 +1,338 @@
+# A-6 — 끊기지 않고 느려지기만 하면 어떻게 되는가
+
+브랜치 `feature/keycloak-a6-latency-injection` ·
+증거 [`docs/evidence/a6-latency-injection/`](evidence/a6-latency-injection/) ·
+2026-09-04 13:10–13:35 KST
+
+**실제 장애의 대부분은 완전 사망이 아니라 느려짐이다.** 그리고 느려짐은
+사망보다 진단하기 어렵다 — 헬스체크가 통과하기 때문이다.
+
+---
+
+## 0. 결론부터
+
+| 측정 | 값 |
+|---|---|
+| 주입한 네트워크 지연 | **200 ms** |
+| 로그인 응답 시간 | **66 ms → 1,872 ms (28배)** |
+| 동시 20건에서 최대 응답 | **22.2 초** |
+| 커넥션 획득 대기 최대 | **20,000 ms** |
+| readiness 프로브 | **타임아웃으로 실패** |
+
+**200밀리초가 22초가 됐다.** 지연은 **왕복 횟수만큼 곱해지고**, 커넥션 풀에서
+**한 번 더 곱해진다.**
+
+---
+
+## 1. 설계 — 한 실험에서 A/B 가 되는 배치
+
+```
+   kc-lab-2                        kc-lab-1
+   ┌──────────────────┐            ┌──────────────────┐
+   │ postgres         │            │ keycloak-1       │
+   │ keycloak-0       │            │                  │
+   │   └─ cni0 로 직행 │◀─ VXLAN ──▶│   └─ 오버레이 경유 │
+   └──────────────────┘            └──────────────────┘
+        지연 없음                        여기만 느려진다
+```
+
+**postgres 가 보내는 패킷만** 지연시키면 `keycloak-1` 의 DB 접근만 느려지고
+`keycloak-0` 은 그대로다. **대조군이 같은 실험 안에 있다.**
+
+기준선은 거의 같았다.
+
+```
+  keycloak-0  평균 70 ms
+  keycloak-1  평균 66 ms
+```
+
+---
+
+## 2. 주입을 두 번 실패했다
+
+### 실패 ① — `eth0` 이라는 인터페이스가 없다
+
+```bash
+ssh kc-lab-2 'sudo tc qdisc add dev eth0 root handle 1: prio'
+```
+```
+Cannot find device "eth0"
+```
+
+Debian 클라우드 이미지는 **예측 가능한 인터페이스 이름**을 쓴다.
+
+```
+enp1s0    UP    52:54:00:aa:bb:12
+```
+
+`en` (ethernet) + `p1` (PCI bus 1) + `s0` (slot 0). 이름이 하드웨어 위치에서
+나오므로 **NIC 순서가 바뀌어도 이름이 안 바뀐다.**
+
+### 실패 ② — `enp1s0` 에서는 **파드 IP 가 보이지 않는다**
+
+여기가 핵심이다. 노드 간 파드 통신은 **flannel VXLAN 으로 캡슐화**된다.
+
+```
+   원래 패킷:   src=10.42.1.76(postgres)  dst=10.42.0.42(keycloak-1)
+        │
+        ▼  flannel.1 에서 캡슐화
+   실제 패킷:   src=192.168.122.12(노드)   dst=192.168.122.11(노드)  UDP 8472
+                └─ 안쪽에 원래 패킷이 통째로 들어 있다
+        │
+        ▼
+   enp1s0 로 나간다
+```
+
+**`enp1s0` 에서 `match ip src 10.42.1.76` 은 절대 일치하지 않는다.**
+그 IP 는 페이로드 안에 있고, 헤더에는 노드 IP 만 있다.
+
+### 성공 — `flannel.1` 에 건다
+
+```bash
+ssh kc-lab-2 '
+  sudo tc qdisc add dev flannel.1 root handle 1: prio
+  sudo tc qdisc add dev flannel.1 parent 1:3 handle 30: netem delay 200ms
+  sudo tc filter add dev flannel.1 protocol ip parent 1:0 prio 3 \
+       u32 match ip src 10.42.1.76/32 flowid 1:3'
+```
+
+**`flannel.1` 은 캡슐화 직전 단계**이므로 여기서는 파드 IP 가 보인다.
+
+### 검증 — 카운터로 확인한다
+
+```
+qdisc netem 30: parent 1:3 limit 1000 delay 200ms
+ Sent 18388 bytes 150 pkt (dropped 0, overlimits 0 requeues 0)
+                  ───────
+                  실제로 지연 밴드를 통과했다
+```
+
+> **A-1·A-5 와 같은 교훈이 세 번째로 나왔다.**
+> 주입을 넣은 것과 걸린 것은 다르다. **카운터를 봐야 한다.**
+
+### 개념 — `tc` 의 계층 구조
+
+```
+   qdisc (큐 규율)     인터페이스에 붙는 패킷 스케줄러
+     ├─ prio           우선순위 밴드 3개로 나눈다
+     │    ├─ 1:1 (기본)
+     │    ├─ 1:2 (기본)
+     │    └─ 1:3  ← 여기에 netem 을 붙인다
+     └─ filter         어떤 패킷을 어느 밴드로 보낼지
+```
+
+`netem` 을 root 에 바로 붙이면 **모든 트래픽**이 느려진다.
+`prio` + `filter` 를 쓰면 **고른 트래픽만** 느려진다.
+
+---
+
+## 3. 결과 — 지연은 곱해진다
+
+```
+=== 두 노드 지연 비교 (기준선: k0=70ms k1=66ms) ===
+  keycloak-0  평균   41 ms   최대   57 ms      ← 영향 없음
+  keycloak-1  평균 1872 ms   최대 1887 ms      ← 28배
+```
+
+### 왜 200ms 가 1,872ms 가 되는가
+
+A-0 에서 잡은 로그인 트랜잭션의 SQL 이 답이다.
+
+```
+BEGIN
+select ... from OFFLINE_USER_SESSION ...
+select VERSION ... for no key update skip locked
+select ... from OFFLINE_CLIENT_SESSION ...
+select VERSION ... for no key update skip locked
+insert into OFFLINE_USER_SESSION ...
+insert into OFFLINE_CLIENT_SESSION ...
+SET LOCAL synchronous_commit TO OFF
+COMMIT
+```
+
+**왕복이 아홉 번이다.**
+
+```
+   200 ms × 9 왕복 ≈ 1,800 ms      실측 1,872 ms
+```
+
+> **네트워크 지연은 왕복 횟수만큼 증폭된다.**
+> "DB 가 200ms 느려졌다"는 "애플리케이션이 200ms 느려졌다"가 아니다.
+> **쿼리 수를 줄이는 것이 지연 환경에서 결정적인 이유**가 이것이다.
+
+---
+
+## 4. 동시 부하 — 여기서 진짜 고장이 난다
+
+동시 20건을 `keycloak-1` 에 보냈다.
+
+```
+  200  1.911    200  1.913    200  1.958    200  1.981
+  200  3.441    200  4.841    200  6.257    200  7.704
+  200  9.104    200 10.539    200 11.951    200 13.351
+  200 14.785    200 16.189    200 17.625    200 19.053
+  200 20.495    200 21.905    200 22.228    200 22.230
+```
+
+**전부 성공(200)했지만 응답 시간이 1.9초에서 22.2초까지 계단으로 늘어난다.**
+
+```
+   1.9 → 3.4 → 4.8 → 6.2 → 7.7 → 9.1 → 10.5 → ...
+        ────  ────  ────  ────
+        약 1.4초 간격 — 앞 요청이 커넥션을 놓아줄 때까지 줄을 선다
+```
+
+**전형적인 큐잉이다.** 커넥션 수는 유한하고, 각 요청이 커넥션을 1.9초씩
+붙잡고 있으므로 뒤에 온 요청은 그만큼 기다린다.
+
+### 커넥션 풀 지표가 증언한다
+
+```
+  agroal_blocking_time_max_milliseconds   20000.0    ← 20초를 기다린 요청이 있다
+  agroal_blocking_time_average_milliseconds  281.0
+  agroal_max_used_count                      19.0    ← 풀이 19개까지 늘었다
+  agroal_acquire_count_total                672.0
+  agroal_active_count                         0.0    ← 부하가 끝나 지금은 0
+```
+
+![커넥션 획득 대기 시간](evidence/a6-latency-injection/a6-connection-pool-blocking.png)
+
+### 그리고 헬스체크가 무너진다
+
+```
+Warning  Unhealthy  pod/keycloak-1  Readiness probe failed:
+  Get "http://10.42.0.42:9000/health/ready": context deadline exceeded
+```
+
+**readiness 프로브 자체가 타임아웃됐다.**
+
+### 연쇄 고장의 모양
+
+```
+   DB 가 느려진다
+        ↓
+   요청이 커넥션을 오래 붙잡는다
+        ↓
+   커넥션 풀이 고갈된다
+        ↓
+   새 요청이 줄을 선다 (최대 20초)
+        ↓
+   헬스체크도 줄에 선다 → 타임아웃 → NotReady
+        ↓
+   그 노드가 로드밸런서에서 빠진다
+        ↓
+   ★ 남은 노드로 트래픽이 몰린다 → 그 노드도 같은 길을 간다
+```
+
+**마지막 화살표가 무서운 부분이다.** 느려짐은 **전파된다.**
+A-2(DB 완전 정지)는 즉시 503 으로 드러나 오히려 명확했지만,
+**느려짐은 살아 있는 노드를 하나씩 무너뜨린다.**
+
+---
+
+## 5. 빗나간 예측 — 낙관적 락 충돌은 늘지 않았다
+
+계획서에 이렇게 적었다.
+
+> **낙관적 락 충돌 증가** — 트랜잭션이 길어져 `VERSION` 충돌이 늘어야 한다
+
+```
+  관련 로그 줄수: 0
+```
+
+**하나도 없었다.** 이유가 명확하다.
+
+```
+   로그인   →  매번 새 세션 행을 INSERT      →  다툴 상대가 없다
+   refresh  →  같은 세션 행을 UPDATE         →  여기서 다툰다
+```
+
+**충돌은 같은 행을 동시에 고칠 때만 일어난다.** 로그인 부하로는 재현되지
+않는다. 이건 **B-3(refresh 토큰 경쟁)의 영역**이며, 거기서 지연을 함께 주면
+충돌률이 올라갈 것이다.
+
+> 예측을 적어두지 않았다면 "충돌이 없네" 하고 넘어갔을 것이다.
+> **빗나간 예측이 다음 실험의 설계를 정해준다.**
+
+---
+
+## 6. 복구
+
+```bash
+ssh kc-lab-2 'sudo tc qdisc del dev flannel.1 root'
+```
+
+```
+  keycloak-0  평균 43 ms
+  keycloak-1  평균 51 ms      ← 즉시 정상
+```
+
+**파드 재시작 없이 즉시 회복.** 커넥션 풀도 스스로 정상화됐다.
+
+---
+
+## 7. 운영에 주는 것
+
+| 알게 된 것 | 함의 |
+|---|---|
+| 지연은 **왕복 횟수만큼 곱해진다** | DB 지연 대책은 "쿼리 수 줄이기"가 먼저다 |
+| 커넥션 풀에서 **한 번 더 곱해진다** | 풀 크기와 타임아웃이 장애 반경을 정한다 |
+| **헬스체크도 줄에 선다** | 프로브 타임아웃이 풀 대기보다 짧아야 격리가 제때 된다 |
+| 느려짐은 **전파된다** | 노드를 빼면 남은 노드가 더 빨리 무너진다 |
+| `up` 도 readiness 도 **늦게 반응** | **응답 시간 분포(p95/p99)를 봐야 한다** |
+
+### 이 실험대에 없는 알림
+
+지금 관측 스택에는 **지연 분포 지표가 없다.** `agroal_blocking_time_*` 은
+있지만 히스토그램이 아니라 평균/최대뿐이다.
+
+```promql
+# 있으면 좋았을 것
+histogram_quantile(0.99, rate(http_server_requests_seconds_bucket[5m]))
+```
+
+**A-2 에서 `kube-state-metrics` 가 빠진 것을 찾았고, 여기서는 응답 시간
+히스토그램이 빠진 것을 찾았다.** 둘 다 보완 항목이다.
+
+---
+
+## 8. 재현 절차 (명령어)
+
+```bash
+# 1. 인터페이스 이름 확인 — eth0 이 아니다
+ssh kc-lab-2 'ip -brief link show | grep -E "flannel|cni|enp"'
+
+# 2. 오버레이 인터페이스에 건다 (enp1s0 에서는 파드 IP 가 안 보인다)
+ssh kc-lab-2 '
+  sudo tc qdisc add dev flannel.1 root handle 1: prio
+  sudo tc qdisc add dev flannel.1 parent 1:3 handle 30: netem delay 200ms
+  sudo tc filter add dev flannel.1 protocol ip parent 1:0 prio 3 \
+       u32 match ip src <postgres 파드IP>/32 flowid 1:3'
+
+# 3. 걸렸는지 카운터로 확인 — Sent 가 0 이면 해석 금지
+ssh kc-lab-2 'sudo tc -s qdisc show dev flannel.1 | grep -A2 netem'
+
+# 4. 단일 요청 지연 (대조군은 같은 노드의 keycloak-0)
+kubectl -n keycloak-lab run t --rm -i --restart=Never --image=curlimages/curl:8.11.1 \
+  --command -- curl -s -o /dev/null -w '%{time_total}\n' -X POST http://<pod>:8080/realms/master/protocol/openid-connect/token ...
+
+# 5. 동시 부하로 풀 고갈 재현
+#    ( curl ... ) & 를 20개 띄우고 wait
+
+# 6. 풀 지표
+curl -s http://<pod>:9000/metrics | grep -E '^agroal_(blocking_time|max_used|awaiting)'
+
+# 7. 해제
+ssh kc-lab-2 'sudo tc qdisc del dev flannel.1 root'
+```
+
+---
+
+## 9. 다음 실험에 남기는 것
+
+| 실험 | 이 실험이 준 것 |
+|---|---|
+| **B-3** refresh 경쟁 | **지연을 함께 주면 낙관적 락 충돌이 재현될 것** — 여기서는 안 됐다 |
+| **B-1** 저장소 지연 (Q3 제약) | 같은 기법을 Redis 앞에 쓴다 |
+| 관측 보완 | **응답 시간 히스토그램**이 없다 |
+| 구성 | **프로브 타임아웃 < 커넥션 풀 대기**여야 격리가 제때 된다 |
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
DongHyeonkaandClaude Opus 5	aa2c3907f5	docs: B-6 — rotation is safe, retiring the old key is not Adding a higher-priority RSA provider leaves both kids in JWKS, so tokens signed before and after the rotation both validate. Deleting the old provider makes its tokens 401 immediately, and the resource server's JWKS cache does not buy a grace period because an unknown kid triggers a refetch. The encryption key Q3 asks about does not exist yet, since B-2 showed the tokens are stored as plaintext JWTs, so the measured signing-key rotation is what its design has to copy: write with one key, read with several, and keep the overlap longer than the lifetime of anything signed with the old one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 14:33:42 +09:00
DongHyeonkaandClaude Opus 5	f45a2a2aaa	docs: B-5 — the pod stays Ready while every request hangs Stopping Redis returns HTTP 000 rather than an error because the client waits on reconnect, and the pod keeps serving traffic because the redis health indicator is not in the readiness group even though /actuator/health returns 503. That is the mirror image of A-2, where Keycloak put its database check in readiness and the pods left the Service. Turning on AOF with config set created the appendonlydir and still lost everything on pod deletion, because /data was the container filesystem; adding a PVC makes the same setting work. Volume first, persistence setting second. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 14:29:17 +09:00
DongHyeonkaandClaude Opus 5	7dc0a3e5da	docs: B-4 — the edge does not overwrite the headers it never sets Two headers of the same name both arrive rather than one overwriting the other, because nginx only replaces headers it sets with proxy_set_header. A comma inside a role name is indistinguishable from the delimiter, and the size limit is a cliff: Tomcat returns 400 around 8KB and the connection dies around 16KB, so the same cause produces two different-looking failures. Forged identity headers reach the upstream untouched while the JWT-protected paths return 401, which is Q4's own point that a header-fed upstream has nothing to verify against. By Q4's checklist that answer alone points at the BFF structure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 14:23:41 +09:00
DongHyeonkaandClaude Opus 5	b16e1dccf7	docs: B-3 — concurrent refresh does not lose a race, it destroys the session Five simultaneous refreshes with one token return a single 200, and that winner's new token is already dead. Reuse detection removes the client session while the user session stays, which is why the other responses read Session doesn't have required client rather than a reuse error. Comparing policies shows rotation off passes all five and keeps the session, while raising refreshTokenMaxReuse to one still destroys it. Since no retry can recover a removed client session, Q2's own criterion resolves to a lock, and a database row lock is the natural place because its lifetime is tied to the connection. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 14:20:04 +09:00
DongHyeonkaandClaude Opus 5	711878379c	docs: B-2 — sharing the stores fixes one problem and exposes three more Moving the authorized client to JdbcOAuth2AuthorizedClientService makes tokens work across replicas, so the session-in-Redis plus tokens-in-PostgreSQL split holds. The table then shows what sharing cannot fix: the primary key is (client_registration_id, principal_name) with no session in it, so a second login for the same user updates the same row rather than adding one. The refresh token sits in bytea as the raw JWT, readable with convert_from, and logout clears only the Redis session while the plaintext token row and the Keycloak SSO session both survive. The schema itself failed silently first because the default DDL uses blob, which PostgreSQL does not have, and continue-on-error swallowed it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 14:15:05 +09:00
DongHyeonkaandClaude Opus 5	f2595f748f	docs: B-1 — Redis moves the session and leaves the tokens behind Adding Spring Session Redis grows the context by 81 beans and swaps sessionRepository for RedisSessionRepository, while authorizedClientService stays InMemoryOAuth2AuthorizedClientService. The user then reads as logged in with principal labuser while accessTokenStoredOnServer is false, which is worse than being logged out. Redis holds only the security context, serialized with Java native serialization, and the refresh token is not there to encrypt in the first place. Three problems on the way: Kubernetes service links overwrote REDIS_PORT with a tcp:// URL, the tests tried to reach Redis, and the resource server was never deployed so a DNS failure looked like a token failure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 14:05:02 +09:00
DongHyeonkaandClaude Opus 5	e62bbb4df0	docs: B-0 — deploy the BFF and read what autoconfiguration actually chose The authorized client repository is AuthenticatedPrincipalOAuth2AuthorizedClientRepository, keyed by principal with no session id in it, which is the mechanism behind the sharing problem Q1 and Q3 describe. Sharing a store does not fix a lookup key. Five problems on the way in: only build output was committed under bff/, a duplicate YAML key broke the image build and was invisible until the full log was captured, env placeholders without defaults broke the tests, actuator was behind the login redirect so a 200 was the login page, and the 117KB beans response failed through the proxy. Deploying two replicas made the login itself fail before any experiment started, because the authorization request lives in per-instance memory and the callback lands elsewhere. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 13:48:33 +09:00
DongHyeonkaandClaude Opus 5	8f6d67df35	docs: A-7 — three results invert when persistent sessions are turned off Disabling persistent-user-sessions moves the session from PostgreSQL into the cluster, and the A-1 and A-8 outcomes flip to 400 Session not active while a new login during database loss starts working. The control group in each case still returns 200, so the injections cut only what they were meant to cut. This is the pair that makes the A layer legible: the conventional wisdom that sessions ride TCP 7800 is correct for Keycloak 24 and earlier, and the mistake is applying it to 26 without checking the version. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 13:33:33 +09:00
DongHyeonkaandClaude Opus 5	114d21aebe	docs: A-8 — a rolling restart keeps every session and drops only the cache Nine samples through the restart all returned 200, the refresh token issued beforehand still works, and the session count is unchanged at 151 while both caches reset to zero. The updated last_session_refresh proves the write path recovered too, not just the response code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 13:20:56 +09:00
DongHyeonkaandClaude Opus 5	dba0c3975c	docs: A-6 — 200ms of network delay becomes 22 seconds of user latency Nine database round trips per login multiply the injected delay to 1.9 seconds, and connection pool queueing multiplies it again under twenty concurrent requests. The readiness probe joins the same queue and times out, so the node leaves the load balancer and pushes its load onto the one still standing. Two injections missed first: the guest interface is enp1s0 rather than eth0, and a filter on it can never match a pod IP because flannel has already encapsulated the packet. The delay has to go on flannel.1, before encapsulation. The predicted rise in optimistic lock conflicts did not happen, because logins insert new rows rather than contending for one. That belongs to B-3. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 13:18:10 +09:00
				`@@ -0,0 +1 @@`
				`[ 207ms] [ERROR] Failed to load resource: the server responded with a status of 404 () @ https://app1.hyeonworks.com/favicon.ico:0`
				`@@ -0,0 +1 @@`
				`[ 236ms] [ERROR] Failed to load resource: the server responded with a status of 500 () @ https://app1.hyeonworks.com/bff/api/me:0`
				`@@ -0,0 +1 @@`
				`- generic [active] [ref=f28e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":true,\"refreshTokenStoredOnServer\":true,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"`
				`@@ -0,0 +1 @@`
				`- generic [active] [ref=f30e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"`
				`@@ -0,0 +1 @@`
				`- generic [active] [ref=f32e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"`
				`@@ -0,0 +1 @@`
				`- generic [active] [ref=f34e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"`
				`@@ -0,0 +1 @@`
				`- generic [active] [ref=f35e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"`
				`@@ -0,0 +1 @@`
				`- generic [active] [ref=f37e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":true,\"refreshTokenStoredOnServer\":true,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"`