docs: D-3 — only RBAC actually hides anything

Every secret in the lab prints in four commands, while kubectl describe shows just a byte count and creates the impression that something is hidden. k3s reports encryption at rest disabled and the plaintext password is present in state.db, so one node disk carries the whole cluster's secrets, and inside the pod they are ordinary environment variables visible to exec, /proc and crash dumps. The default service account cannot read secrets, which makes RBAC the one control doing real work here and the thing worth tightening. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
docs: D-2 — rolling back the image does not roll back the schema
2026-09-04 15:07:35 +09:00 · 2026-09-04 15:05:36 +09:00 · 2026-09-04 14:59:56 +09:00 · 2026-09-04 14:56:58 +09:00 · 2026-09-04 14:49:36 +09:00 · 2026-09-04 14:44:00 +09:00
117 changed files with 4475 additions and 5 deletions
@@ -0,0 +1 @@
+[     236ms] [ERROR] Failed to load resource: the server responded with a status of 500 () @ https://app1.hyeonworks.com/bff/api/me:0
@@ -0,0 +1,2 @@
+[    7292ms] [ERROR] Access to fetch at 'https://auth.hyeonworks.com/realms/keycloak-patterns/protocol/openid-connect/auth?response_type=code&client_id=bff-confidential&scope=openid%20profile%20email&state=WWc76H7TY73Fbsdc41B2nRD5exkXqHcohLh4WdJB4AA%3D&redirect_uri=https://app1.hyeonworks.com/login/oauth2/code/keycloak&nonce=A4TXweuKS4Y5HdZ63rLJUez1ZOyI2em6zs3OIfTXLFo&code_challenge=wPr8PXG0lcUvie7Wo91YrVMhOUYq0KtEU4PxVJ0_CWA&code_challenge_method=S256' (redirected from 'https://app1.hyeonworks.com/bff/api/me') from origin 'https://app1.hyeonworks.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. @ https://app1.hyeonworks.com/bff/token-boundary:0
+[    7293ms] [ERROR] Failed to load resource: net::ERR_FAILED @ https://auth.hyeonworks.com/realms/keycloak-patterns/protocol/openid-connect/auth?response_type=code&client_id=bff-confidential&scope=openid%20profile%20email&state=WWc76H7TY73Fbsdc41B2nRD5exkXqHcohLh4WdJB4AA%3D&redirect_uri=https://app1.hyeonworks.com/login/oauth2/code/keycloak&nonce=A4TXweuKS4Y5HdZ63rLJUez1ZOyI2em6zs3OIfTXLFo&code_challenge=wPr8PXG0lcUvie7Wo91YrVMhOUYq0KtEU4PxVJ0_CWA&code_challenge_method=S256:0
@@ -0,0 +1,2 @@
+[    6726ms] [ERROR] Access to fetch at 'https://auth.hyeonworks.com/realms/keycloak-patterns/protocol/openid-connect/auth?response_type=code&client_id=bff-confidential&scope=openid%20profile%20email&state=GGupuPr3ZklKp8ah99r7h7mNHEq9yTsEWZr85WyXevE%3D&redirect_uri=https://app1.hyeonworks.com/login/oauth2/code/keycloak&nonce=rPVvEOvG7rzssAjR7pP67qNvoY2W6ZVpIpKYz1LGoU8&code_challenge=qoKRLRrzB7z9CU_rlaAxJ7UYcRZswyqmDi8PgxmaWM0&code_challenge_method=S256' (redirected from 'https://app1.hyeonworks.com/bff/api/me') from origin 'https://app1.hyeonworks.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. @ https://app1.hyeonworks.com/:0
+[    6726ms] [ERROR] Failed to load resource: net::ERR_FAILED @ https://auth.hyeonworks.com/realms/keycloak-patterns/protocol/openid-connect/auth?response_type=code&client_id=bff-confidential&scope=openid%20profile%20email&state=GGupuPr3ZklKp8ah99r7h7mNHEq9yTsEWZr85WyXevE%3D&redirect_uri=https://app1.hyeonworks.com/login/oauth2/code/keycloak&nonce=rPVvEOvG7rzssAjR7pP67qNvoY2W6ZVpIpKYz1LGoU8&code_challenge=qoKRLRrzB7z9CU_rlaAxJ7UYcRZswyqmDi8PgxmaWM0&code_challenge_method=S256:0
@@ -0,0 +1,2 @@
+[    6898ms] [ERROR] Failed to load resource: the server responded with a status of 403 () @ https://app1.hyeonworks.com/logout:0
+[   23950ms] [ERROR] Failed to load resource: the server responded with a status of 403 () @ https://app1.hyeonworks.com/logout:0
@@ -0,0 +1,2 @@
+[     275ms] [ERROR] Failed to load resource: the server responded with a status of 502 () @ https://app2.hyeonworks.com/oauth2/callback?state=G2-BDWkehNWO7hGwhYCxXBVRKZ6AomLIrSLBtIXr0Gw%3A%2Fapi%2Fecho&session_state=vsW8xDlLJN3DUl-0B-X1WL7Q&iss=https%3A%2F%2Fauth.hyeonworks.com%2Frealms%2Fkeycloak-patterns&code=4155f58e-6a58-e47b-93bd-7e2b625c2b91.vsW8xDlLJN3DUl-0B-X1WL7Q.80431dbc-af81-4673-9790-ad06d1570b2e:0
+[     408ms] [ERROR] Failed to load resource: the server responded with a status of 502 () @ https://app2.hyeonworks.com/oauth2/callback?state=Da-7OcMB4f7vyHgr-6CqrTtJpmj1R_SfRfcE3CUJNzE%3A%2Ffavicon.ico&session_state=vsW8xDlLJN3DUl-0B-X1WL7Q&iss=https%3A%2F%2Fauth.hyeonworks.com%2Frealms%2Fkeycloak-patterns&code=96d66247-91b0-0cc1-89dc-e527c2b69bf2.vsW8xDlLJN3DUl-0B-X1WL7Q.80431dbc-af81-4673-9790-ad06d1570b2e:0
@@ -0,0 +1,2 @@
+[     266ms] [ERROR] Failed to load resource: the server responded with a status of 502 () @ https://app2.hyeonworks.com/oauth2/callback?state=TZnQvjIWCEySrf4PMg2WLVFoOfkyOJWB58f2LGjVfpo%3A%2Fapi%2Fecho&session_state=vsW8xDlLJN3DUl-0B-X1WL7Q&iss=https%3A%2F%2Fauth.hyeonworks.com%2Frealms%2Fkeycloak-patterns&code=8ae913a1-2647-0ed9-625e-3d80e1565024.vsW8xDlLJN3DUl-0B-X1WL7Q.80431dbc-af81-4673-9790-ad06d1570b2e:0
+[     416ms] [ERROR] Failed to load resource: the server responded with a status of 502 () @ https://app2.hyeonworks.com/oauth2/callback?state=uAYwZp59ncz95XjkJXAlIgsT7oksBQBViiQS07t2eow%3A%2Ffavicon.ico&session_state=vsW8xDlLJN3DUl-0B-X1WL7Q&iss=https%3A%2F%2Fauth.hyeonworks.com%2Frealms%2Fkeycloak-patterns&code=7b7fc816-0b27-93a0-83b8-656488813253.vsW8xDlLJN3DUl-0B-X1WL7Q.80431dbc-af81-4673-9790-ad06d1570b2e:0
@@ -0,0 +1,2 @@
+[     287ms] [ERROR] Failed to load resource: the server responded with a status of 502 () @ https://app2.hyeonworks.com/oauth2/callback?state=0EOFj1PoLyPil0dgukpi7zKW4JKnGZTP9Wj6EhTR-lw%3A%2Fapi%2Fecho&session_state=vsW8xDlLJN3DUl-0B-X1WL7Q&iss=https%3A%2F%2Fauth.hyeonworks.com%2Frealms%2Fkeycloak-patterns&code=d13cc206-133f-00a9-9908-599988c4d7cf.vsW8xDlLJN3DUl-0B-X1WL7Q.80431dbc-af81-4673-9790-ad06d1570b2e:0
+[     457ms] [ERROR] Failed to load resource: the server responded with a status of 502 () @ https://app2.hyeonworks.com/oauth2/callback?state=kbYC5O4_ELsoQFw85vyYfgWEqlI2yWImB4rmelbmSTM%3A%2Ffavicon.ico&session_state=vsW8xDlLJN3DUl-0B-X1WL7Q&iss=https%3A%2F%2Fauth.hyeonworks.com%2Frealms%2Fkeycloak-patterns&code=91f9fde9-f376-b6f3-4622-a1ba674cc4fd.vsW8xDlLJN3DUl-0B-X1WL7Q.80431dbc-af81-4673-9790-ad06d1570b2e:0
@@ -0,0 +1 @@
+[     307ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ https://app2.hyeonworks.com/favicon.ico:0
@@ -0,0 +1 @@
+[     261ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ https://app2.hyeonworks.com/favicon.ico:0
@@ -0,0 +1 @@
+[     882ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ https://app2.hyeonworks.com/favicon.ico:0
@@ -0,0 +1 @@
+[     178ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ https://app2.hyeonworks.com/favicon.ico:0
@@ -0,0 +1,7 @@
+- main [ref=f29e2]:
+  - heading "AP3 · Backend-for-Frontend" [level=1] [ref=f29e3]
+  - paragraph [ref=f29e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
+  - button "Keycloak 로그인" [ref=f29e5] [cursor=pointer]
+  - button "token 경계 확인" [ref=f29e6] [cursor=pointer]
+  - button "BFF 경유 API 호출" [ref=f29e7] [cursor=pointer]
+  - button "CSRF token으로 상태 변경" [ref=f29e8] [cursor=pointer]
@@ -0,0 +1 @@
+- generic [active] [ref=f30e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1,5 @@
+- generic [active] [ref=f31e1]:
+  - heading "Whitelabel Error Page" [level=1] [ref=f31e2]
+  - paragraph [ref=f31e3]: This application has no explicit mapping for /error, so you are seeing this as a fallback.
+  - generic [ref=f31e4]: Fri Sep 04 05:00:51 GMT 2026
+  - generic [ref=f31e5]: There was an unexpected error (type=Internal Server Error, status=500).
@@ -0,0 +1 @@
+- generic [active] [ref=f32e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1,7 @@
+- main [ref=f33e2]:
+  - heading "AP3 · Backend-for-Frontend" [level=1] [ref=f33e3]
+  - paragraph [ref=f33e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
+  - button "Keycloak 로그인" [ref=f33e5] [cursor=pointer]
+  - button "token 경계 확인" [ref=f33e6] [cursor=pointer]
+  - button "BFF 경유 API 호출" [ref=f33e7] [cursor=pointer]
+  - button "CSRF token으로 상태 변경" [ref=f33e8] [cursor=pointer]
@@ -0,0 +1 @@
+- generic [active] [ref=f34e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1 @@
+- generic [active] [ref=f35e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1,7 @@
+- main [ref=f36e2]:
+  - heading "AP3 · Backend-for-Frontend" [level=1] [ref=f36e3]
+  - paragraph [ref=f36e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
+  - button "Keycloak 로그인" [ref=f36e5] [cursor=pointer]
+  - button "token 경계 확인" [ref=f36e6] [cursor=pointer]
+  - button "BFF 경유 API 호출" [ref=f36e7] [cursor=pointer]
+  - button "CSRF token으로 상태 변경" [ref=f36e8] [cursor=pointer]
@@ -0,0 +1 @@
+- generic [active] [ref=f37e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":true,\"refreshTokenStoredOnServer\":true,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1,7 @@
+- main [ref=f38e2]:
+  - heading "AP3 · Backend-for-Frontend" [level=1] [ref=f38e3]
+  - paragraph [ref=f38e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
+  - button "Keycloak 로그인" [ref=f38e5] [cursor=pointer]
+  - button "token 경계 확인" [ref=f38e6] [cursor=pointer]
+  - button "BFF 경유 API 호출" [ref=f38e7] [cursor=pointer]
+  - button "CSRF token으로 상태 변경" [ref=f38e8] [cursor=pointer]
@@ -0,0 +1,7 @@
+- main [ref=f39e2]:
+  - heading "AP3 · Backend-for-Frontend" [level=1] [ref=f39e3]
+  - paragraph [ref=f39e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
+  - button "Keycloak 로그인" [ref=f39e5] [cursor=pointer]
+  - button "token 경계 확인" [ref=f39e6] [cursor=pointer]
+  - button "BFF 경유 API 호출" [ref=f39e7] [cursor=pointer]
+  - button "CSRF token으로 상태 변경" [ref=f39e8] [cursor=pointer]
@@ -0,0 +1,4 @@
+- generic [active] [ref=f40e1]:
+  - heading "502 Bad Gateway" [level=1] [ref=f40e3]
+  - separator [ref=f40e4]
+  - generic [ref=f40e5]: nginx/1.30.4
@@ -0,0 +1,4 @@
+- generic [active] [ref=f41e1]:
+  - heading "502 Bad Gateway" [level=1] [ref=f41e3]
+  - separator [ref=f41e4]
+  - generic [ref=f41e5]: nginx/1.30.4
@@ -0,0 +1,4 @@
+- generic [active] [ref=f42e1]:
+  - heading "502 Bad Gateway" [level=1] [ref=f42e3]
+  - separator [ref=f42e4]
+  - generic [ref=f42e5]: nginx/1.30.4
@@ -0,0 +1 @@
+- generic [active] [ref=f43e1]: "{ \"headers\" : { \"host\" : [ \"app2.hyeonworks.com\" ], \"user-agent\" : [ \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36\" ], \"accept\" : [ \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7\" ], \"accept-encoding\" : [ \"gzip, deflate, br, zstd\" ], \"accept-language\" : [ \"en-US,en;q=0.9\" ], \"cookie\" : [ \"grafana_session=60c3e7ae41ffc00665f4a2c377def399; grafana_session_expiry=1788497124; _oauth2_proxy=djIuWDI5aGRYUm9NbDl3Y205NGVTMWlNall4TVRGbVltUXhabVJoWWpOaFpUSXhPREpsTWpnM01EQXhZakF5WVEuTk81VE82RHRod2NWZWstaHpPZVg1Zw==|1788500470|iPSRUlwHDB0XgC6sUdU4dq1EHq9WQDPYrDoezajKVUA=\" ], \"priority\" : [ \"u=0, i\" ], \"sec-ch-ua\" : [ \"\\\"Chromium\\\";v=\\\"152\\\", \\\"Not?A_Brand\\\";v=\\\"24\\\", \\\"Google Chrome\\\";v=\\\"152\\\"\" ], \"sec-ch-ua-mobile\" : [ \"?0\" ], \"sec-ch-ua-platform\" : [ \"\\\"Linux\\\"\" ], \"sec-fetch-dest\" : [ \"document\" ], \"sec-fetch-mode\" : [ \"navigate\" ], \"sec-fetch-site\" : [ \"none\" ], \"sec-fetch-user\" : [ \"?1\" ], \"upgrade-insecure-requests\" : [ \"1\" ], \"x-forwarded-email\" : [ \"labuser@example.com\" ], \"x-forwarded-host\" : [ \"app2.hyeonworks.com\" ], \"x-forwarded-port\" : [ \"443\" ], \"x-forwarded-preferred-username\" : [ \"labuser\" ], \"x-forwarded-proto\" : [ \"https\" ], \"x-forwarded-server\" : [ \"traefik-5d6fcf895-wpfhr\" ], \"x-forwarded-user\" : [ \"27df5ea9-8703-4ec5-badd-d972c583e1ff\" ], \"x-real-ip\" : [ \"100.123.124.30\" ] }, \"remoteAddr\" : \"100.123.124.30\", \"localAddr\" : \"10.42.0.53\", \"scheme\" : \"https\", \"secure\" : true, \"serverName\" : \"app2.hyeonworks.com\", \"serverPort\" : 443, \"requestUrl\" : \"https://app2.hyeonworks.com/api/echo\" }"
@@ -0,0 +1 @@
+- generic [active] [ref=f44e1]: "{ \"headers\" : { \"host\" : [ \"app2.hyeonworks.com\" ], \"user-agent\" : [ \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36\" ], \"accept\" : [ \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7\" ], \"accept-encoding\" : [ \"gzip, deflate, br, zstd\" ], \"accept-language\" : [ \"en-US,en;q=0.9\" ], \"cookie\" : [ \"grafana_session=60c3e7ae41ffc00665f4a2c377def399; grafana_session_expiry=1788497124; _oauth2_proxy=djIuWDI5aGRYUm9NbDl3Y205NGVTMDVOemhrWm1GbFptSmtZV1JqWTJJNU5tTTNZbVV4TmpJMVpHSmhOVFl4TmcucmoxSnJPYjJKOW1ZV191aXVWa2FCZw==|1788500538|RoiStOeQcIDldxB3cckyO-OAiMgBjBfw5gOSvUsgTFU=\" ], \"priority\" : [ \"u=0, i\" ], \"sec-ch-ua\" : [ \"\\\"Chromium\\\";v=\\\"152\\\", \\\"Not?A_Brand\\\";v=\\\"24\\\", \\\"Google Chrome\\\";v=\\\"152\\\"\" ], \"sec-ch-ua-mobile\" : [ \"?0\" ], \"sec-ch-ua-platform\" : [ \"\\\"Linux\\\"\" ], \"sec-fetch-dest\" : [ \"document\" ], \"sec-fetch-mode\" : [ \"navigate\" ], \"sec-fetch-site\" : [ \"none\" ], \"sec-fetch-user\" : [ \"?1\" ], \"upgrade-insecure-requests\" : [ \"1\" ], \"x-forwarded-email\" : [ \"labuser@example.com\" ], \"x-forwarded-host\" : [ \"app2.hyeonworks.com\" ], \"x-forwarded-port\" : [ \"443\" ], \"x-forwarded-preferred-username\" : [ \"labuser\" ], \"x-forwarded-proto\" : [ \"https\" ], \"x-forwarded-server\" : [ \"traefik-5d6fcf895-wpfhr\" ], \"x-forwarded-user\" : [ \"27df5ea9-8703-4ec5-badd-d972c583e1ff\" ], \"x-real-ip\" : [ \"100.123.124.30\" ] }, \"remoteAddr\" : \"100.123.124.30\", \"localAddr\" : \"10.42.1.132\", \"scheme\" : \"https\", \"secure\" : true, \"serverName\" : \"app2.hyeonworks.com\", \"serverPort\" : 443, \"requestUrl\" : \"https://app2.hyeonworks.com/api/echo\" }"
@@ -0,0 +1,16 @@
+- generic [ref=f45e3]:
+  - banner [ref=f45e4]:
+    - generic [ref=f45e5]: keycloak-patterns
+  - main [ref=f45e6]:
+    - heading "Sign in to your account" [level=1] [ref=f45e8]
+    - generic [ref=f45e12]:
+      - generic [ref=f45e13]:
+        - generic [ref=f45e14]: Username or email
+        - textbox "Username or email" [active] [ref=f45e17]
+      - generic [ref=f45e18]:
+        - generic [ref=f45e19]: Password
+        - generic [ref=f45e21]:
+          - textbox "Password" [ref=f45e24]
+          - button "Show password" [ref=f45e26] [cursor=pointer]:
+            - generic [aria-hidden] [ref=f45e27]: 
+      - button "Sign In" [ref=f45e30] [cursor=pointer]
@@ -0,0 +1,7 @@
+- main [ref=f46e2]:
+  - heading "AP3 · Backend-for-Frontend" [level=1] [ref=f46e3]
+  - paragraph [ref=f46e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
+  - button "Keycloak 로그인" [ref=f46e5] [cursor=pointer]
+  - button "token 경계 확인" [ref=f46e6] [cursor=pointer]
+  - button "BFF 경유 API 호출" [ref=f46e7] [cursor=pointer]
+  - button "CSRF token으로 상태 변경" [ref=f46e8] [cursor=pointer]
@@ -0,0 +1 @@
+- generic [active] [ref=f47e1]: "{ \"headers\" : { \"host\" : [ \"app2.hyeonworks.com\" ], \"user-agent\" : [ \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36\" ], \"accept\" : [ \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7\" ], \"accept-encoding\" : [ \"gzip, deflate, br, zstd\" ], \"accept-language\" : [ \"en-US,en;q=0.9\" ], \"cookie\" : [ \"grafana_session=60c3e7ae41ffc00665f4a2c377def399; grafana_session_expiry=1788497124; _oauth2_proxy=djIuWDI5aGRYUm9NbDl3Y205NGVTMDJZakF5T0dFM01HWTJPV000WmpCa1lUazVOalpsWWpNMk9UY3laR1ptTWcuWmpUamNTbGxVSHV1RmJjQ2ZRd2lsUQ==|1788500797|17Hbo3RDtzOldnLZx2xOt3e34lHo_v0yqRNdqdKUx-g=\" ], \"priority\" : [ \"u=0, i\" ], \"sec-ch-ua\" : [ \"\\\"Chromium\\\";v=\\\"152\\\", \\\"Not?A_Brand\\\";v=\\\"24\\\", \\\"Google Chrome\\\";v=\\\"152\\\"\" ], \"sec-ch-ua-mobile\" : [ \"?0\" ], \"sec-ch-ua-platform\" : [ \"\\\"Linux\\\"\" ], \"sec-fetch-dest\" : [ \"document\" ], \"sec-fetch-mode\" : [ \"navigate\" ], \"sec-fetch-site\" : [ \"none\" ], \"sec-fetch-user\" : [ \"?1\" ], \"upgrade-insecure-requests\" : [ \"1\" ], \"x-forwarded-email\" : [ \"labuser@example.com\" ], \"x-forwarded-host\" : [ \"app2.hyeonworks.com\" ], \"x-forwarded-port\" : [ \"443\" ], \"x-forwarded-preferred-username\" : [ \"labuser\" ], \"x-forwarded-proto\" : [ \"https\" ], \"x-forwarded-server\" : [ \"traefik-5d6fcf895-wpfhr\" ], \"x-forwarded-user\" : [ \"27df5ea9-8703-4ec5-badd-d972c583e1ff\" ], \"x-real-ip\" : [ \"100.123.124.30\" ] }, \"remoteAddr\" : \"100.123.124.30\", \"localAddr\" : \"10.42.0.53\", \"scheme\" : \"https\", \"secure\" : true, \"serverName\" : \"app2.hyeonworks.com\", \"serverPort\" : 443, \"requestUrl\" : \"https://app2.hyeonworks.com/api/echo\" }"
@@ -0,0 +1 @@
+- generic [active] [ref=f48e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":true,\"refreshTokenStoredOnServer\":true,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1 @@
+- generic [active] [ref=f49e1]: "{ \"headers\" : { \"host\" : [ \"app2.hyeonworks.com\" ], \"user-agent\" : [ \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36\" ], \"accept\" : [ \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7\" ], \"accept-encoding\" : [ \"gzip, deflate, br, zstd\" ], \"accept-language\" : [ \"en-US,en;q=0.9\" ], \"cookie\" : [ \"grafana_session=60c3e7ae41ffc00665f4a2c377def399; grafana_session_expiry=1788497124; _oauth2_proxy=djIuWDI5aGRYUm9NbDl3Y205NGVTMDJZakF5T0dFM01HWTJPV000WmpCa1lUazVOalpsWWpNMk9UY3laR1ptTWcuWmpUamNTbGxVSHV1RmJjQ2ZRd2lsUQ==|1788500797|17Hbo3RDtzOldnLZx2xOt3e34lHo_v0yqRNdqdKUx-g=\" ], \"priority\" : [ \"u=0, i\" ], \"sec-ch-ua\" : [ \"\\\"Chromium\\\";v=\\\"152\\\", \\\"Not?A_Brand\\\";v=\\\"24\\\", \\\"Google Chrome\\\";v=\\\"152\\\"\" ], \"sec-ch-ua-mobile\" : [ \"?0\" ], \"sec-ch-ua-platform\" : [ \"\\\"Linux\\\"\" ], \"sec-fetch-dest\" : [ \"document\" ], \"sec-fetch-mode\" : [ \"navigate\" ], \"sec-fetch-site\" : [ \"none\" ], \"sec-fetch-user\" : [ \"?1\" ], \"upgrade-insecure-requests\" : [ \"1\" ], \"x-forwarded-email\" : [ \"labuser@example.com\" ], \"x-forwarded-host\" : [ \"app2.hyeonworks.com\" ], \"x-forwarded-port\" : [ \"443\" ], \"x-forwarded-preferred-username\" : [ \"labuser\" ], \"x-forwarded-proto\" : [ \"https\" ], \"x-forwarded-server\" : [ \"traefik-5d6fcf895-wpfhr\" ], \"x-forwarded-user\" : [ \"27df5ea9-8703-4ec5-badd-d972c583e1ff\" ], \"x-real-ip\" : [ \"100.123.124.30\" ] }, \"remoteAddr\" : \"100.123.124.30\", \"localAddr\" : \"10.42.0.53\", \"scheme\" : \"https\", \"secure\" : true, \"serverName\" : \"app2.hyeonworks.com\", \"serverPort\" : 443, \"requestUrl\" : \"https://app2.hyeonworks.com/api/echo\" }"
@@ -0,0 +1 @@
+- generic [active] [ref=f50e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":true,\"refreshTokenStoredOnServer\":true,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1,16 @@
+- generic [ref=f51e3]:
+  - banner [ref=f51e4]:
+    - generic [ref=f51e5]: keycloak-patterns
+  - main [ref=f51e6]:
+    - heading "Sign in to your account" [level=1] [ref=f51e8]
+    - generic [ref=f51e12]:
+      - generic [ref=f51e13]:
+        - generic [ref=f51e14]: Username or email
+        - textbox "Username or email" [active] [ref=f51e17]
+      - generic [ref=f51e18]:
+        - generic [ref=f51e19]: Password
+        - generic [ref=f51e21]:
+          - textbox "Password" [ref=f51e24]
+          - button "Show password" [ref=f51e26] [cursor=pointer]:
+            - generic [aria-hidden] [ref=f51e27]: 
+      - button "Sign In" [ref=f51e30] [cursor=pointer]
@@ -0,0 +1,7 @@
+- main [ref=f52e2]:
+  - heading "AP3 · Backend-for-Frontend" [level=1] [ref=f52e3]
+  - paragraph [ref=f52e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
+  - button "Keycloak 로그인" [ref=f52e5] [cursor=pointer]
+  - button "token 경계 확인" [ref=f52e6] [cursor=pointer]
+  - button "BFF 경유 API 호출" [ref=f52e7] [cursor=pointer]
+  - button "CSRF token으로 상태 변경" [ref=f52e8] [cursor=pointer]
@@ -0,0 +1,16 @@
+- generic [ref=f53e3]:
+  - banner [ref=f53e4]:
+    - generic [ref=f53e5]: keycloak-patterns
+  - main [ref=f53e6]:
+    - heading "Sign in to your account" [level=1] [ref=f53e8]
+    - generic [ref=f53e12]:
+      - generic [ref=f53e13]:
+        - generic [ref=f53e14]: Username or email
+        - textbox "Username or email" [ref=f53e17]
+      - generic [ref=f53e18]:
+        - generic [ref=f53e19]: Password
+        - generic [ref=f53e21]:
+          - textbox "Password" [ref=f53e24]
+          - button "Show password" [ref=f53e26] [cursor=pointer]:
+            - generic [aria-hidden] [ref=f53e27]: 
+      - button "Sign In" [ref=f53e30] [cursor=pointer]
@@ -0,0 +1,16 @@
+- generic [ref=f53e3]:
+  - banner [ref=f53e4]:
+    - generic [ref=f53e5]: keycloak-patterns
+  - main [ref=f53e6]:
+    - heading "Sign in to your account" [level=1] [ref=f53e8]
+    - generic [ref=f53e12]:
+      - generic [ref=f53e13]:
+        - generic [ref=f53e14]: Username or email
+        - textbox "Username or email" [ref=f53e17]: labuser
+      - generic [ref=f53e18]:
+        - generic [ref=f53e19]: Password
+        - generic [ref=f53e21]:
+          - textbox "Password" [active] [ref=f53e24]: labpass
+          - button "Show password" [ref=f53e26] [cursor=pointer]:
+            - generic [aria-hidden] [ref=f53e27]: 
+      - button "Sign In" [ref=f53e30] [cursor=pointer]
@@ -0,0 +1,16 @@
+- generic [ref=f53e3]:
+  - banner [ref=f53e4]:
+    - generic [ref=f53e5]: keycloak-patterns
+  - main [ref=f53e6]:
+    - heading "Sign in to your account" [level=1] [ref=f53e8]
+    - generic [ref=f53e12]:
+      - generic [ref=f53e13]:
+        - generic [ref=f53e14]: Username or email
+        - textbox "Username or email" [ref=f53e17]: labuser
+      - generic [ref=f53e18]:
+        - generic [ref=f53e19]: Password
+        - generic [ref=f53e21]:
+          - textbox "Password" [active] [ref=f53e24]: labpass
+          - button "Show password" [ref=f53e26] [cursor=pointer]:
+            - generic [aria-hidden] [ref=f53e27]: 
+      - button "Sign In" [ref=f53e30] [cursor=pointer]
@@ -0,0 +1,16 @@
+- generic [ref=f53e3]:
+  - banner [ref=f53e4]:
+    - generic [ref=f53e5]: keycloak-patterns
+  - main [ref=f53e6]:
+    - heading "Sign in to your account" [level=1] [ref=f53e8]
+    - generic [ref=f53e12]:
+      - generic [ref=f53e13]:
+        - generic [ref=f53e14]: Username or email
+        - textbox "Username or email" [ref=f53e17]: labuser
+      - generic [ref=f53e18]:
+        - generic [ref=f53e19]: Password
+        - generic [ref=f53e21]:
+          - textbox "Password" [active] [ref=f53e24]: labpass
+          - button "Show password" [ref=f53e26] [cursor=pointer]:
+            - generic [aria-hidden] [ref=f53e27]: 
+      - button "Sign In" [ref=f53e30] [cursor=pointer]
@@ -0,0 +1,16 @@
+- generic [ref=f54e3]:
+  - banner [ref=f54e4]:
+    - generic [ref=f54e5]: keycloak-patterns
+  - main [ref=f54e6]:
+    - heading "Sign in to your account" [level=1] [ref=f54e8]
+    - generic [ref=f54e12]:
+      - generic [ref=f54e13]:
+        - generic [ref=f54e14]: Username or email
+        - textbox "Username or email" [ref=f54e17]
+      - generic [ref=f54e18]:
+        - generic [ref=f54e19]: Password
+        - generic [ref=f54e21]:
+          - textbox "Password" [ref=f54e24]
+          - button "Show password" [ref=f54e26] [cursor=pointer]:
+            - generic [aria-hidden] [ref=f54e27]: 
+      - button "Sign In" [ref=f54e30] [cursor=pointer]
@@ -0,0 +1,16 @@
+- generic [ref=f54e3]:
+  - banner [ref=f54e4]:
+    - generic [ref=f54e5]: keycloak-patterns
+  - main [ref=f54e6]:
+    - heading "Sign in to your account" [level=1] [ref=f54e8]
+    - generic [ref=f54e12]:
+      - generic [ref=f54e13]:
+        - generic [ref=f54e14]: Username or email
+        - textbox "Username or email" [ref=f54e17]: labuser
+      - generic [ref=f54e18]:
+        - generic [ref=f54e19]: Password
+        - generic [ref=f54e21]:
+          - textbox "Password" [active] [ref=f54e24]: labpass
+          - button "Show password" [ref=f54e26] [cursor=pointer]:
+            - generic [aria-hidden] [ref=f54e27]: 
+      - button "Sign In" [ref=f54e30] [cursor=pointer]
@@ -29,6 +29,38 @@
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-oauth2-client</artifactId>
        </dependency>
+
+        <!-- B-1: Application Session 을 Redis 로 옮긴다.
+             spring-session-data-redis 가 SessionRepository 를 갈아끼우고,
+             spring-boot-starter-data-redis 가 연결(Lettuce)을 제공한다.
+             둘 다 있어야 자동구성이 걸린다 — 하나만 넣으면 조용히 in-memory 로 남는다. -->
+        <dependency>
+            <groupId>org.springframework.session</groupId>
+            <artifactId>spring-session-data-redis</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.springframework.boot</groupId>
+            <artifactId>spring-boot-starter-data-redis</artifactId>
+        </dependency>
+
+        <!-- B-2: OAuth2AuthorizedClient 를 PostgreSQL 로 옮긴다.
+             Q3 가 후보로 든 "Redis 와 JDBC 중 무엇" 에서 JDBC 쪽이며,
+             JdbcOAuth2AuthorizedClientService 는 같은 인터페이스라
+             컨트롤러를 바꾸지 않아도 된다. -->
+        <dependency>
+            <groupId>org.springframework.boot</groupId>
+            <artifactId>spring-boot-starter-jdbc</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.postgresql</groupId>
+            <artifactId>postgresql</artifactId>
+            <scope>runtime</scope>
+        </dependency>
+        <dependency>
+            <groupId>com.h2database</groupId>
+            <artifactId>h2</artifactId>
+            <scope>test</scope>
+        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
@@ -8,15 +8,38 @@ import org.springframework.security.oauth2.client.OAuth2AuthorizedClientManager;
 import org.springframework.security.oauth2.client.OAuth2AuthorizedClientProvider;
 import org.springframework.security.oauth2.client.OAuth2AuthorizedClientProviderBuilder;
 import org.springframework.security.oauth2.client.OAuth2AuthorizedClientService;
+import org.springframework.security.oauth2.client.JdbcOAuth2AuthorizedClientService;
 import org.springframework.security.oauth2.client.registration.ClientRegistrationRepository;
 import org.springframework.security.oauth2.client.web.DefaultOAuth2AuthorizationRequestResolver;
 import org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestCustomizers;
 import org.springframework.security.web.SecurityFilterChain;
 import org.springframework.security.web.csrf.CookieCsrfTokenRepository;
+import org.springframework.jdbc.core.JdbcOperations;

@Configuration
 public class SecurityConfig {

+    /**
+     * B-2 — authorized client 를 프로세스 메모리에서 PostgreSQL 로 옮긴다.
+     *
+     * B-1 에서 Application Session 만 Redis 로 옮겼더니, 사용자는 로그인
+     * 상태로 보이는데 BFF 에는 access token 이 없는 상태가 만들어졌다.
+     * 두 상태의 저장소를 **각각** 정해야 한다는 Q3 의 지적이 그대로 나타난 것이다.
+     *
+     * 주의 — 이것이 고치는 것과 고치지 못하는 것이 다르다.
+     *   고친다   : 인스턴스 간 공유. 어느 replica 로 가도 같은 토큰을 본다.
+     *   못 고친다: 조회 키. JdbcOAuth2AuthorizedClientService 도
+     *              (clientRegistrationId, principalName) 으로 찾으므로
+     *              같은 사용자의 두 브라우저는 여전히 한 항목을 공유한다.
+     */
+    @Bean
+    OAuth2AuthorizedClientService authorizedClientService(
+        JdbcOperations jdbcOperations,
+        ClientRegistrationRepository clientRegistrationRepository
+    ) {
+        return new JdbcOAuth2AuthorizedClientService(jdbcOperations, clientRegistrationRepository);
+    }
+
    @Bean
    SecurityFilterChain bffSecurity(
        HttpSecurity http,
@@ -10,6 +10,34 @@ server:
 spring:
  application:
    name: keycloak-bff
+  datasource:
+    # B-2: authorized client 전용. Keycloak 과 같은 PostgreSQL 인스턴스지만
+    # 테이블이 다르다(oauth2_authorized_client). 운영이라면 분리를 검토한다.
+    url: ${BFF_DB_URL:jdbc:postgresql://localhost:5432/keycloak}
+    username: ${BFF_DB_USER:keycloak}
+    password: ${BFF_DB_PASSWORD:keycloak}
+  sql:
+    init:
+      # Spring Security 가 제공하는 DDL 을 그대로 쓴다.
+      # always 로 두면 매 기동마다 실행되므로 CREATE TABLE IF NOT EXISTS 가 아닌
+      # 스크립트에서는 실패한다 → continue-on-error 로 넘긴다.
+      mode: ${SPRING_SQL_INIT_MODE:always}
+      # ★ PostgreSQL 은 -postgres 판본을 써야 한다. 기본 판본은 `blob` 타입을
+      #   쓰는데 PostgreSQL 에는 그 타입이 없다(`bytea` 다). continue-on-error 가
+      #   그 실패를 삼켜서 "테이블이 조용히 안 생기는" 상태가 됐었다.
+      schema-locations: classpath:org/springframework/security/oauth2/client/oauth2-client-schema-postgres.sql
+      continue-on-error: true
+  data:
+    redis:
+      host: ${REDIS_HOST:localhost}
+      port: ${REDIS_PORT:6379}
+  session:
+    # Application Session 만 Redis 로 간다. OAuth2AuthorizedClient 는
+    # 이 설정과 무관하며 여전히 InMemory 다 — 조회 키가 다르기 때문이다(B-0).
+    store-type: ${SPRING_SESSION_STORE_TYPE:redis}
+    timeout: ${SPRING_SESSION_TIMEOUT:30m}
+    redis:
+      namespace: bff:session
  security:
    oauth2:
      client:
@@ -24,6 +24,15 @@ import org.springframework.test.web.servlet.MockMvc;

@SpringBootTest(properties = {
    "KEYCLOAK_CLIENT_SECRET=test-only-secret",
+    // 테스트는 Redis 를 띄우지 않는다. store-type=none 이면 자동구성이
+    // 서블릿 컨테이너 기본 세션으로 되돌아가 컨텍스트가 뜬다.
+    "spring.session.store-type=none",
+    // 테스트에는 PostgreSQL 이 없다. H2 로 대신하고 Spring Security 의
+    // DDL 을 그대로 태워 JdbcOAuth2AuthorizedClientService 가 뜨게 한다.
+    "spring.datasource.url=jdbc:h2:mem:bfftest;DB_CLOSE_DELAY=-1",
+    "spring.datasource.username=sa",
+    "spring.datasource.password=",
+    "spring.sql.init.mode=always",
    "resource-api.base-url=http://127.0.0.1:9"
 })
@AutoConfigureMockMvc
@@ -0,0 +1,128 @@
+# Experiment B-7 — oauth2-proxy, to measure how replicas share a cookie secret
+# and what happens when it is rotated (Q1, unknown 7).
+#
+# This is a different shape of problem from the BFF. The BFF keeps state on the
+# server, so the question was "which store". oauth2-proxy keeps no server state
+# at all: the whole session rides in a cookie that is signed and encrypted with
+# --cookie-secret. So there is nothing to share and nothing to lose on restart —
+# instead, every replica must hold the *same* secret, and changing it invalidates
+# every cookie at once.
+#
+#   kubectl apply -f deploy/lab/k8s/b7-oauth2-proxy.yaml
+#
+# app2.hyeonworks.com is borrowed from Grafana for the duration of this
+# experiment; the certificate only covers auth / app1 / app2, so a fourth name
+# is not available. Grafana's Ingress is restored afterwards.
+apiVersion: v1
+kind: Secret
+metadata:
+  name: oauth2-proxy-secrets
+  namespace: keycloak-lab
+type: Opaque
+stringData:
+  # oauth2-proxy requires exactly 16, 24 or 32 bytes. This is the value whose
+  # rotation the experiment is about.
+  COOKIE_SECRET_A: "lab-cookie-secret-aaaaaaaaaaaaaa"
+  COOKIE_SECRET_B: "lab-cookie-secret-bbbbbbbbbbbbbb"
+  CLIENT_SECRET: proxy-lab-secret
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: oauth2-proxy
+  namespace: keycloak-lab
+spec:
+  # Two replicas is the point: Q1 asks how they share the secret.
+  replicas: 2
+  selector:
+    matchLabels: { app: oauth2-proxy }
+  template:
+    metadata:
+      labels: { app: oauth2-proxy }
+    spec:
+      # See B-1: Kubernetes injects <SVCNAME>_PORT as a tcp:// URL and it
+      # collides with ordinary configuration names.
+      enableServiceLinks: false
+      topologySpreadConstraints:
+        - maxSkew: 1
+          topologyKey: kubernetes.io/hostname
+          whenUnsatisfiable: ScheduleAnyway
+          labelSelector:
+            matchLabels: { app: oauth2-proxy }
+      containers:
+        - name: oauth2-proxy
+          image: quay.io/oauth2-proxy/oauth2-proxy:v7.7.1
+          args:
+            - --provider=oidc
+            - --oidc-issuer-url=https://auth.hyeonworks.com/realms/keycloak-patterns
+            - --client-id=oauth2-proxy
+            - --redirect-url=https://app2.hyeonworks.com/oauth2/callback
+            - --email-domain=*
+            - --http-address=0.0.0.0:4180
+            # The upstream is the same echo app the B-4 header experiment used,
+            # so what the proxy forwards can be read straight off the response.
+            - --upstream=http://echo.header-lab.svc:8081
+            # ★ 이 옵션을 켜면 세션(=쿠키)에 access token 이 들어간다.
+            # 그러면 Set-Cookie 가 커져 프록시 앞단에서 502 가 났다.
+            # B-4 에서 본 헤더 크기 절벽이 이번에는 응답 쪽에서 나타난 것이다.
+            # - --pass-authorization-header=true
+            - --set-xauthrequest=true
+            - --reverse-proxy=true
+            - --cookie-secure=true
+            # One hour, matching the value Q1 records for the current setup.
+            - --cookie-expire=1h
+            - --skip-provider-button=true
+            # ★ 쿠키에 세션 전체를 담으면 Set-Cookie 가 커지고, 그 응답이
+            # 앞단 nginx 의 proxy_buffer 를 넘겨 502 가 났다(측정됨).
+            # Redis 로 옮기면 쿠키에는 티켓만 남는다 — 그리고 그 순간
+            # "replica 가 secret 을 공유해야 한다"는 문제의 성격도 바뀐다.
+            - --session-store-type=redis
+            - --redis-connection-url=redis://redis.keycloak-lab.svc:6379
+          env:
+            - name: OAUTH2_PROXY_CLIENT_SECRET
+              valueFrom:
+                secretKeyRef: { name: oauth2-proxy-secrets, key: CLIENT_SECRET }
+            # Which of the two secrets is in use is switched here. Both replicas
+            # read the same key, which is exactly the sharing Q1 asks about.
+            - name: OAUTH2_PROXY_COOKIE_SECRET
+              valueFrom:
+                secretKeyRef: { name: oauth2-proxy-secrets, key: COOKIE_SECRET_A }
+          ports:
+            - containerPort: 4180
+              name: http
+          readinessProbe:
+            httpGet: { path: /ping, port: http }
+            initialDelaySeconds: 5
+          resources:
+            requests: { memory: 32Mi, cpu: 20m }
+            limits:   { memory: 128Mi }
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: oauth2-proxy
+  namespace: keycloak-lab
+spec:
+  selector: { app: oauth2-proxy }
+  ports:
+    - port: 4180
+      targetPort: http
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: oauth2-proxy
+  namespace: keycloak-lab
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: app2.hyeonworks.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: oauth2-proxy
+                port:
+                  number: 4180
@@ -24,9 +24,24 @@ stringData:
  # Base64 in etcd is not encryption — see D-3.
  KEYCLOAK_CLIENT_SECRET: bff-lab-secret
 ---
-# Redis. No persistence yet: `--save ""` and no appendonly, so a restart loses
-# everything. B-5 and B-6 compare that against RDB and AOF, which is easier to
-# reason about when the starting point is "nothing survives".
+# Redis. B-5 measured that turning on AOF with `redis-cli config set` changes
+# nothing here, because /data is the container filesystem and dies with the
+# container — the appendonlydir was created and then thrown away. Persistence
+# configuration without a volume is decoration.
+#
+# So the volume comes first, and only then does `--appendonly yes` mean anything.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: redis-data
+  namespace: keycloak-lab
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: local-path
+  resources:
+    requests:
+      storage: 1Gi
+---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -47,16 +62,25 @@ spec:
      containers:
        - name: redis
          image: redis:7.4-alpine
-          args: ["redis-server", "--save", "", "--appendonly", "no"]
+          # appendfsync everysec 이 기본값이다 — 1초 분량을 잃을 수 있다.
+          # Keycloak 의 synchronous_commit OFF(A-3)와 같은 모양의 트레이드오프다.
+          args: ["redis-server", "--appendonly", "yes", "--dir", "/data"]
          ports:
            - containerPort: 6379
              name: redis
          readinessProbe:
            exec: { command: ["redis-cli", "ping"] }
            initialDelaySeconds: 3
+          volumeMounts:
+            - name: data
+              mountPath: /data
          resources:
            requests: { memory: 32Mi, cpu: 20m }
            limits:   { memory: 128Mi }
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: redis-data
 ---
 apiVersion: v1
 kind: Service
@@ -92,6 +116,14 @@ spec:
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: { app: bff }
+      # 쿠버네티스는 같은 네임스페이스의 Service 마다 Docker link 시절의
+      # 환경변수를 자동 주입한다: REDIS_PORT=tcp://10.43.57.116:6379.
+      # 그것이 application.yml 의 ${REDIS_PORT:6379} 를 덮어써서 기동이 실패했다.
+      #   Failed to bind properties under 'spring.data.redis.port' to int:
+      #     Value: "tcp://10.43.57.116:6379"
+      # 이 주입 자체를 끄는 것이 근본 처방이다. 이름을 바꿔 피하면 다음 사람이
+      # 같은 함정에 다시 빠진다.
+      enableServiceLinks: false
      containers:
        - name: bff
          image: keycloak-pattern-bff:lab
@@ -107,8 +139,11 @@ spec:
              value: https://auth.hyeonworks.com/realms/keycloak-patterns
            - name: KC_ISSUER_INTERNAL
              value: http://keycloak.keycloak-lab.svc:8080/realms/keycloak-patterns
+            # echo 는 header-lab 네임스페이스의 8081 이다. 다른 네임스페이스의
+            # 서비스는 <svc>.<ns>.svc 로 부른다. 이름을 틀리면 500 이 나는데
+            # 원인은 UnresolvedAddressException 이지 토큰 문제가 아니다.
            - name: RESOURCE_API_BASE_URL
-              value: http://echo.keycloak-lab.svc:8080
+              value: http://echo.header-lab.svc:8081
            - name: KEYCLOAK_CLIENT_SECRET
              valueFrom:
                secretKeyRef: { name: bff-secrets, key: KEYCLOAK_CLIENT_SECRET }
@@ -117,6 +152,24 @@ spec:
            # it builds comes back as http:// and Keycloak rejects it.
            - name: SERVER_FORWARD_HEADERS_STRATEGY
              value: native
+            # B-1: Application Session 을 Redis 로 옮긴다.
+            # OAuth2AuthorizedClient 는 이것으로 옮겨지지 않는다 — 조회 키가
+            # 다르기 때문이며, B-0 에서 확인한 사실이다.
+            - name: SPRING_SESSION_STORE_TYPE
+              value: redis
+            - name: REDIS_HOST
+              value: redis.keycloak-lab.svc
+            - name: REDIS_PORT
+              value: "6379"
+            # B-2: authorized client 는 PostgreSQL 로. 세션(Redis)과 다른
+            # 저장소를 쓰는 것이 Q3 가 말한 "각각 설계한다"의 실물이다.
+            - name: BFF_DB_URL
+              value: jdbc:postgresql://postgres.keycloak-lab.svc:5432/keycloak
+            - name: BFF_DB_USER
+              value: keycloak
+            - name: BFF_DB_PASSWORD
+              valueFrom:
+                secretKeyRef: { name: keycloak-lab-secrets, key: POSTGRES_PASSWORD }
            - name: JAVA_TOOL_OPTIONS
              value: "-Xms128m -Xmx320m"
          readinessProbe:
@@ -0,0 +1,5 @@
+deployment.apps/bff configured
+deployment "bff" successfully rolled out
+bff-576d869c6d-bshvl   true   kc-lab-2
+bff-695646ddb-kzs9k    true   kc-lab-1
+bff-695646ddb-vjqzf    true   kc-lab-2
@@ -0,0 +1,64 @@
+=== B-1 — Redis 를 붙인 뒤 자동구성이 실제로 바뀌었는가 ===
+  빈 수: 321 → 402  (+81)
+
+  --- 세션 저장소 관련 (새로 생긴 것) ---
+    ★ cookieSerializer                           -> DefaultCookieSerializer
+    ★ org.springframework.boot.autoconfigure.session.RedisSessionConfiguration -> RedisSessionConfiguration
+    ★ org.springframework.boot.autoconfigure.session.RedisSessionConfiguration$DefaultRedisSessionConfiguration -> RedisSessionConfiguration$DefaultRedisSessionConfiguration
+    ★ org.springframework.boot.autoconfigure.session.SessionAutoConfiguration -> SessionAutoConfiguration
+    ★ org.springframework.boot.autoconfigure.session.SessionAutoConfiguration$ServletSessionConfiguration -> SessionAutoConfiguration$ServletSessionConfiguration
+    ★ org.springframework.boot.autoconfigure.session.SessionAutoConfiguration$ServletSessionConfiguration$RememberMeServicesConfiguration -> SessionAutoConfiguration$ServletSessionConfiguration$RememberMeServicesConfiguration
+    ★ org.springframework.boot.autoconfigure.session.SessionAutoConfiguration$ServletSessionConfiguration$ServletSessionRepositoryConfiguration -> SessionAutoConfiguration$ServletSessionConfiguration$ServletSessionRepositoryConfiguration
+    ★ org.springframework.boot.autoconfigure.session.SessionRepositoryFilterConfiguration -> SessionRepositoryFilterConfiguration
+    ★ org.springframework.session.config.annotation.web.http.SpringHttpSessionConfiguration -> SpringHttpSessionConfiguration
+    ★ org.springframework.session.data.redis.config.annotation.web.http.RedisHttpSessionConfiguration -> RedisHttpSessionConfiguration
+    ★ rememberMeServicesCookieSerializerCustomizer -> SessionAutoConfiguration$ServletSessionConfiguration$RememberMeServicesConfiguration$$Lambda/0x00007f364e69fa60
+    ★ sessionEventHttpSessionListenerAdapter     -> SessionEventHttpSessionListenerAdapter
+    ★ sessionRepository                          -> RedisSessionRepository
+    ★ sessionRepositoryFilterRegistration        -> DelegatingFilterProxyRegistrationBean
+    ★ spring.session-org.springframework.boot.autoconfigure.session.SessionProperties -> SessionProperties
+    ★ spring.session.redis-org.springframework.boot.autoconfigure.session.RedisSessionProperties -> RedisSessionProperties
+    ★ springBootSessionRepositoryCustomizer      -> RedisSessionConfiguration$DefaultRedisSessionConfiguration$$Lambda/0x00007f364e6a4a68
+    ★ springSessionRepositoryFilter              -> SessionRepositoryFilter
+
+  --- OAuth2 authorized client — 바뀌었는가? ---
+    authorizedClientService
+      before: InMemoryOAuth2AuthorizedClientService
+      after : InMemoryOAuth2AuthorizedClientService    그대로 — Redis 로 안 옮겨졌다
+    authorizedClientRepository
+      before: AuthenticatedPrincipalOAuth2AuthorizedClientRepository
+      after : AuthenticatedPrincipalOAuth2AuthorizedClientRepository    그대로 — Redis 로 안 옮겨졌다
+    authorizedClientManager
+      before: AuthorizedClientServiceOAuth2AuthorizedClientManager
+      after : AuthorizedClientServiceOAuth2AuthorizedClientManager    그대로 — Redis 로 안 옮겨졌다
+
+  --- Redis 연결 빈 (새로 생긴 것) ---
+    ★ keyValueMappingContext                     -> RedisMappingContext
+    ★ lettuceMetrics                             -> LettuceMetricsAutoConfiguration$$Lambda/0x00007f364e56f4d0
+    ★ org.springframework.boot.actuate.autoconfigure.data.redis.RedisHealthContributorAutoConfiguration -> RedisHealthContributorAutoConfiguration
+    ★ org.springframework.boot.actuate.autoconfigure.data.redis.RedisReactiveHealthContributorAutoConfiguration -> RedisReactiveHealthContributorAutoConfiguration
+    ★ org.springframework.boot.actuate.autoconfigure.metrics.redis.LettuceMetricsAutoConfiguration -> LettuceMetricsAutoConfiguration
+    ★ org.springframework.boot.autoconfigure.data.redis.LettuceConnectionConfiguration -> LettuceConnectionConfiguration
+    ★ org.springframework.boot.autoconfigure.data.redis.RedisAutoConfiguration -> RedisAutoConfiguration
+    ★ org.springframework.boot.autoconfigure.data.redis.RedisReactiveAutoConfiguration -> RedisReactiveAutoConfiguration
+    ★ org.springframework.boot.autoconfigure.data.redis.RedisRepositoriesAutoConfiguration -> RedisRepositoriesAutoConfiguration
+    ★ org.springframework.boot.autoconfigure.session.RedisSessionConfiguration -> RedisSessionConfiguration
+    ★ org.springframework.boot.autoconfigure.session.RedisSessionConfiguration$DefaultRedisSessionConfiguration -> RedisSessionConfiguration$DefaultRedisSessionConfiguration
+    ★ org.springframework.session.data.redis.config.annotation.web.http.RedisHttpSessionConfiguration -> RedisHttpSessionConfiguration
+    ★ reactiveRedisTemplate                      -> ReactiveRedisTemplate
+    ★ reactiveStringRedisTemplate                -> ReactiveStringRedisTemplate
+    ★ redisConnectionDetails                     -> PropertiesRedisConnectionDetails
+    ★ redisConnectionFactory                     -> LettuceConnectionFactory
+    ★ redisConverter                             -> MappingRedisConverter
+    ★ redisCustomConversions                     -> RedisCustomConversions
+    ★ redisHealthContributor                     -> RedisReactiveHealthIndicator
+    ★ redisKeyValueAdapter                       -> RedisKeyValueAdapter
+    ★ redisKeyValueTemplate                      -> RedisKeyValueTemplate
+    ★ redisMappingConfiguration#0                -> MappingConfiguration
+    ★ redisReferenceResolver                     -> ReferenceResolverImpl
+    ★ redisTemplate                              -> RedisTemplate
+    ★ sessionRepository                          -> RedisSessionRepository
+    ★ spring.data.redis-org.springframework.boot.autoconfigure.data.redis.RedisProperties -> RedisProperties
+    ★ spring.session.redis-org.springframework.boot.autoconfigure.session.RedisSessionProperties -> RedisSessionProperties
+    ★ springBootSessionRepositoryCustomizer      -> RedisSessionConfiguration$DefaultRedisSessionConfiguration$$Lambda/0x00007f364e6a4a68
+    ★ stringRedisTemplate                        -> StringRedisTemplate
@@ -0,0 +1,21 @@
+=== Redis 에 무엇이 들어 있는가 ===
+bff:session:sessions:8963b6de-3564-4775-9ccd-1ee9616b83ae
+  총 키 수: 1
+
+=== 세션 키의 내용 — refresh token 이 있는가 (Q3 검증 2번) ===
+  키: bff:session:sessions:8963b6de-3564-4775-9ccd-1ee9616b83ae
+  타입: hash
+    필드: sessionAttr:SPRING_SECURITY_CONTEXT
+    필드: sessionAttr:SPRING_SECURITY_SAVED_REQUEST
+    필드: sessionAttr:SPRING_SECURITY_LAST_EXCEPTION
+    필드: sessionAttr:org.springframework.security.oauth2.client.web.HttpSessionOAuth2AuthorizationRequestRepository.AUTHORIZATION_REQUEST
+    필드: lastAccessedTime
+    필드: maxInactiveInterval
+    필드: creationTime
+
+=== 필드 값에 토큰 문자열이 보이는가 ===
+ 1) "sessionAttr:SPRING_SECURITY_CONTEXT"
+ 2) "\xac\xed\x00\x05sr\x00=org.springframework.security.core.context.SecurityContextImpl\x00\x00\x00\x00\x00\x00\x02l\x02\x00\x01L\x00\x0eauthenticationt\x002Lorg/springframework/security/core/Authentication;xpsr\x00Sorg.springframework.security.oauth2.client.authentication.OAuth2AuthenticationToken\x00\x00\x00\x00\x00\x00\x02l\x02\x00\x02L\x00\x1eauthorizedClientRegistrationIdt\x00\x12Ljava/lang/String;L\x00\tprincipalt\x00:Lorg/springframework/security/oauth2/core/user/OAuth2User;xr\x00Gorg.springframework.security.authentication.AbstractAuthenticationToken\xd3\xaa(~nGd\x0e\x02\x00\x03Z\x00\rauthenticatedL\x00\x0bauthoritiest\x00\x16Ljava/util/Collection;L\x00\adetailst\x00\x12Ljava/lang/Object;xp\x01sr\x00&java.util.Collections$UnmodifiableList\xfc\x0f%1\xb5\xec\x8e\x10\x02\x00\x01L\x00\x04listt\x00\x10Ljava/util/List;xr\x00,java.util.Collect
+
+=== TTL (Q3 검증 3번 — session TTL) ===
+  TTL: 1772 초
@@ -0,0 +1,18 @@
+# B-1 — Redis 세션 저장소 전환 증거
+
+2026-09-04 14:50–15:05 KST
+해설: [`docs/experiment-b1-redis-session-store.md`](../../experiment-b1-redis-session-store.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-servicelinks-trap.txt` | `enableServiceLinks: false` 적용 후 롤아웃 성공 — 쿠버네티스가 주입한 `REDIS_PORT=tcp://...` 가 설정을 덮어쓴 문제 |
+| `02-autoconfig-after.txt` | **핵심** — 빈 321→402(+81). `sessionRepository → RedisSessionRepository` 로 바뀌었지만 **`authorizedClientService` 는 `InMemory` 그대로** |
+| `03-redis-contents.txt` | Redis 키 1개, 필드는 `SPRING_SECURITY_CONTEXT` 뿐. **토큰 없음.** Java 직렬화(`\xac\xed`), TTL 1772초 |
+| `b1-login-works-two-replicas.png` | 전환 직후 `accessTokenStoredOnServer: false` |
+| `b1-token-boundary-after-redis.png` | 파드 전면 교체 후 — `principal: labuser` 는 살아남고 토큰만 사라진 상태 |
+
+## 핵심 세 줄
+
+1. **세션은 옮겨졌고 토큰은 안 옮겨졌다.** 빈 81개가 늘었는데 authorized client 관련은 하나도 안 바뀌었다.
+2. **refresh token 은 Redis 에 평문으로 있는 게 아니라 아예 없다.** 암호화를 고민하기 전에 이걸 알아야 한다.
+3. **"로그인은 되어 있는데 아무것도 못 하는" 상태가 만들어진다** — 완전 로그아웃보다 나쁘다.
@@ -0,0 +1,8 @@
+deployment.apps/bff configured
+deployment "bff" successfully rolled out
+bff-555df79c97-6j86w   1/1   Running       0     44s
+bff-555df79c97-vgg6g   1/1   Running       0     22s
+
+=== oauth2_authorized_client 테이블이 생겼는가 ===
+Did not find any relation named "oauth2_authorized_client".
+command terminated with exit code 1
@@ -0,0 +1,33 @@
+=== PostgreSQL 전용 스키마 ===
+CREATE TABLE oauth2_authorized_client (
+  client_registration_id varchar(100) NOT NULL,
+  principal_name varchar(200) NOT NULL,
+  access_token_type varchar(100) NOT NULL,
+  access_token_value bytea NOT NULL,
+  access_token_issued_at timestamp NOT NULL,
+  access_token_expires_at timestamp NOT NULL,
+  access_token_scopes varchar(1000) DEFAULT NULL,
+  refresh_token_value bytea DEFAULT NULL,
+  refresh_token_issued_at timestamp DEFAULT NULL,
+  created_at timestamp DEFAULT CURRENT_TIMESTAMP NOT NULL,
+  PRIMARY KEY (client_registration_id, principal_name)
+);
+
+=== 적용 ===
+CREATE TABLE
+                                Table "public.oauth2_authorized_client"
+         Column          |            Type             | Collation | Nullable |         Default         
+-------------------------+-----------------------------+-----------+----------+-------------------------
+ client_registration_id  | character varying(100)      |           | not null | 
+ principal_name          | character varying(200)      |           | not null | 
+ access_token_type       | character varying(100)      |           | not null | 
+ access_token_value      | bytea                       |           | not null | 
+ access_token_issued_at  | timestamp without time zone |           | not null | 
+ access_token_expires_at | timestamp without time zone |           | not null | 
+ access_token_scopes     | character varying(1000)     |           |          | NULL::character varying
+ refresh_token_value     | bytea                       |           |          | 
+ refresh_token_issued_at | timestamp without time zone |           |          | 
+ created_at              | timestamp without time zone |           | not null | CURRENT_TIMESTAMP
+Indexes:
+    "oauth2_authorized_client_pkey" PRIMARY KEY, btree (client_registration_id, principal_name)
+
@@ -0,0 +1,20 @@
+=== Q3 검증 2번 — 저장소를 직접 열어 refresh token 이 평문인가 ===
+eyJhbGciOiJIUzUxMiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJlMmUzZDZkMy0yNzQyLTRhYWItYjk4Ni02ZDU2ZDM5MDk1ZDEifQ.eyJleHAiOjE3ODg1MDA0NDYsImlhdCI6MTc4ODQ5ODY0NiwianRpIjoiNTQwOTZmYTQtZWRjNi1iZjZkLWE4OGMtZDJhNjEzOGJjNmVlIiwiaXNzIjoiaHR0cHM6Ly9hdXRoLmh5ZW9ud29ya3MuY29tL3JlYWxtcy9rZXljbG9hay1wYXR0ZXJucyIsImF1ZCI6I
+
+=== access token 도 ===
+eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJPWS1jYVlETkdvUDRITUF6LVE5VVBUVS1ETTFpODk2TnV6VVp1NmdmQ3FNIn0.eyJleHAi
+
+=== 그 문자열이 실제 JWT 인지 — 헤더를 디코드 ===
+  File "<string>", line 3
+    h=open(/tmp/hdr.txt).read().strip()
+           ^
+SyntaxError: invalid syntax
+
+=== 저장된 바이트를 그대로 디코드한 결과 ===
+  refresh_token 헤더 : {"alg":"HS512","typ" : "JWT","kid" : "e2e3d6d3-2742-4aab-b986-6d56d39095d1"}
+  refresh_token 페이로드(앞부분):
+    {"exp":1788500446,"iat":1788498646,"jti":"54096fa4-edc6-bf6d-a88c-d2a6138bc6ee","iss":"https://auth.hyeonworks.com/realms/keycloak-patterns"
+  access_token 헤더  : {"alg":"RS256","typ" : "JWT","kid" : "OY-caYDNGoP4HMAz-Q9UPTU-DM1i896NuzUZu6gfCqM"}
+
+  → bytea 에 들어 있는 것은 암호화된 덩어리가 아니라 JWT 문자열 그대로다.
+    DB 읽기 권한만 있으면 그 자리에서 쓸 수 있는 토큰을 얻는다.
@@ -0,0 +1,25 @@
+=== [현재] 같은 사용자의 항목 ===
+ client_registration_id | principal_name |   access_token_issued_at   |              at_md5              
+------------------------+----------------+----------------------------+----------------------------------
+ keycloak               | labuser        | 2026-09-04 05:10:46.927192 | 675af2286bfc2fd9d2bab7bc8f391df7
+(1 row)
+
+  행 수: 1
+
+=== [모의 두 번째 브라우저] 세션만 지우고 같은 사용자로 다시 로그인시킨다 ===
+  (브라우저가 달라도 principal 은 같으므로 조회 키가 같다)
+  Redis 세션 삭제 완료 — 다음 요청이 새 로그인을 만든다
+=== [재로그인 후] 행이 늘었는가, 덮어써졌는가 ===
+ client_registration_id | principal_name |   access_token_issued_at   |              at_md5              
+------------------------+----------------+----------------------------+----------------------------------
+ keycloak               | labuser        | 2026-09-04 05:12:13.018828 | e19a63fc5aa18bd0a68b3e19dff16b3b
+(1 row)
+
+  행 수: 1
+
+  ★ 행 수가 1 그대로이고 md5 가 바뀌었으면 → 덮어쓰기다
+
+=== Q1 검증 ④ — 로그아웃하면 두 저장소가 다 정리되는가 ===
+  로그아웃 전
+    Redis: 1 키
+    PostgreSQL: 1 행
@@ -0,0 +1,14 @@
+=== Q1 검증 ④ — 로그아웃 후 두 저장소 상태 ===
+  Redis 세션      : 0 키
+  PostgreSQL 토큰 : 1 행
+
+ principal_name |   access_token_issued_at   |  access_token_expires_at   
+----------------+----------------------------+----------------------------
+ labuser        | 2026-09-04 05:12:13.018828 | 2026-09-04 05:13:13.018828
+(1 row)
+
+
+  ★ Redis 는 비었는데 PostgreSQL 에 행이 남아 있으면 → 한쪽만 정리된 것
+
+=== Keycloak 쪽 SSO 세션은? ===
+  Keycloak 온라인 세션: 2
@@ -0,0 +1,21 @@
+# B-2 — 다중 인스턴스 운영 증거
+
+2026-09-04 15:05–15:15 KST
+해설: [`docs/experiment-b2-multi-instance-session.md`](../../experiment-b2-multi-instance-session.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-jdbc-store-deploy.txt` | JDBC 저장소로 배포. **테이블이 조용히 안 만들어졌다** |
+| `02-schema.txt` | 원인 — 기본 DDL 은 `blob`(PostgreSQL 에 없음), `-postgres.sql` 판본이 따로 있다. **`PRIMARY KEY (client_registration_id, principal_name)`** — 조회 키 문제가 DDL 에 박혀 있다 |
+| `03-plaintext-tokens.txt` | **Q3 검증 2번** — `bytea` 안이 JWT 문자열 그대로. 디코드하면 `{"alg":"HS512",...}` |
+| `04-overwrite-test.txt` | **Q1 검증 3번** — 같은 사용자 재로그인 시 행 수 1 그대로, `issued_at` 과 md5 만 바뀜 = **UPDATE(덮어쓰기)** |
+| `05-logout-cleanup.txt` | **Q1 검증 4번** — Redis 0키 / PostgreSQL **1행 잔존** / Keycloak SSO **2세션 잔존** |
+| `b2-before-relogin.png` | JDBC 전환 직후, 옛 세션은 여전히 `false` |
+| `b2-tokens-shared-across-instances.png` | 재로그인 후 **`accessTokenStoredOnServer: true`** — 두 replica 에서 동작 |
+
+## 핵심 네 줄
+
+1. **세션 Redis + 토큰 PostgreSQL 분리 저장이 성립한다.** B-1 의 "로그인은 됐는데 토큰이 없는" 상태가 해결됐다.
+2. **refresh token 은 평문이다.** DB 읽기 권한이면 작동하는 토큰을 얻는다.
+3. **같은 사용자의 두 번째 로그인이 첫 번째를 덮어쓴다.** 기본키에 session id 가 없어 구조적으로 그렇다.
+4. **로그아웃은 셋 중 하나만 지운다.** 평문 토큰과 Keycloak SSO 세션이 남는다.
@@ -0,0 +1,11 @@
+=== [1] refresh token 하나 확보 ===
+  토큰 길이: 811
+  jti: 8e7e3ee2-0dc8-573d-58ec-d12651a50b9c
+  sid: BvFiB01Rntz1FcLdf7zG4BNt
+
+=== [2] 같은 refresh token 으로 동시에 5회 갱신 ===
+  요청 1: HTTP 400  {"error":"invalid_grant","error_description":"Maximum allowed refresh token reuse exceeded"}
+  요청 2: HTTP 400  {"error":"invalid_grant","error_description":"Session doesn't have required client"}
+  요청 3: HTTP 400  {"error":"invalid_grant","error_description":"Session doesn't have required client"}
+  요청 4: HTTP 400  {"error":"invalid_grant","error_description":"Session doesn't have required client"}
+  요청 5: HTTP 200  {"access_token":"...(발급됨)
@@ -0,0 +1,16 @@
+=== [3] 이긴 요청이 받은 새 토큰은 쓸 수 있는가 ===
+  새 refresh token 길이: 810
+  그 토큰으로 다시 갱신:  HTTP 400
+    {"error":"invalid_grant","error_description":"Session doesn't have required client"}
+
+=== [4] 그 sid 의 세션이 DB 에 남아 있는가 ===
+     user_session_id      | offline_flag | last_session_refresh 
+--------------------------+--------------+----------------------
+ BvFiB01Rntz1FcLdf7zG4BNt | 0            |           1788498996
+(1 row)
+
+=== [5] revoked_token 테이블 ===
+ revoked_count 
+---------------
+             0
+(1 row)
@@ -0,0 +1,14 @@
+=== user session 과 client session 을 나눠서 본다 ===
+     user_session_id      | offline_flag | client_sessions 
+--------------------------+--------------+-----------------
+ BvFiB01Rntz1FcLdf7zG4BNt | 0            |               0
+(1 row)
+
+
+=== 대조: 정상 세션 하나를 새로 만들어 비교 ===
+  새 sid: JT-XuepgutWcE273QwAnIXta
+     user_session_id      | client_sessions 
+--------------------------+-----------------
+ JT-XuepgutWcE273QwAnIXta |               1
+(1 row)
+
@@ -0,0 +1,24 @@
+=== 구성 A: rotation ON (revokeRefreshToken=true, maxReuse=0) — 앞서 측정 ===
+    성공 1 / 5, 세션 파괴됨
+
+=== 구성 B: rotation OFF (revokeRefreshToken=false) ===
+    sid=iW1CGyO7COdyJLryIrCt3njk
+    1: 200 
+    2: 200 
+    3: 200 
+    4: 200 
+    5: 200 
+    성공 5 / 5
+    이긴 토큰 재사용: HTTP 200
+    남은 client_session: 1
+
+=== 구성 C: rotation ON + 재사용 1회 허용 (maxReuse=1) ===
+    sid=72c04JCdr0NpCHGQmXWW2wM8
+    1: 200 
+    2: 400 "error_description":"Session doesn't have required client"
+    3: 200 
+    4: 400 "error_description":"Maximum allowed refresh token reuse exceeded"
+    5: 400 "error_description":"Session doesn't have required client"
+    성공 2 / 5
+    이긴 토큰 재사용: HTTP 400
+    남은 client_session: 0
@@ -0,0 +1,18 @@
+# B-3 — Refresh Token 동시 갱신 경쟁 증거
+
+2026-09-04 15:15–15:25 KST
+해설: [`docs/experiment-b3-refresh-token-contention.md`](../../experiment-b3-refresh-token-contention.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-concurrent-refresh.txt` | 같은 토큰으로 동시 5회 — **1개만 200**, 나머지는 `Maximum allowed refresh token reuse exceeded` 와 `Session doesn't have required client` **두 종류** 오류 |
+| `02-session-impact.txt` | **★ 이긴 요청의 새 토큰조차 400.** user_session 행은 남아 있고 `revoked_token` 은 0건 |
+| `03-client-session-removed.txt` | **기제 확정 (대조군 포함)** — 경쟁 세션 `client_sessions=0`, 정상 세션 `client_sessions=1` |
+| `04-policy-comparison.txt` | 정책 3종 비교 — rotation OFF 는 **5/5 성공·세션 생존**, maxReuse=1 은 **여전히 세션 파괴** |
+
+## 핵심 네 줄
+
+1. **"하나는 성공"이 아니다.** 이긴 요청이 받은 토큰도 곧바로 쓸 수 없다.
+2. **재사용 탐지가 client session 을 제거한다.** user session 은 껍데기로 남아 `Session doesn't have required client` 가 된다.
+3. **`refreshTokenMaxReuse` 를 올려도 안 된다.** 동시 요청 수만큼 올려야 하고 그러면 rotation 의 목적이 사라진다.
+4. **재시도로 회복되지 않으므로 Q2 의 답은 lock 이다.** 그리고 lock 은 저장소 쪽(가급적 DB 행 잠금)에 있어야 한다.
@@ -0,0 +1,40 @@
+=== Q4 ① 다중 값 role — 구분자와 동명 헤더 ===
+(a) 쉼표 구분 한 개 헤더
+    보냄:   X-Auth-Request-Roles: admin,editor,viewer
+    도착:   ['admin,editor,viewer']          ← 문자열 하나 그대로
+
+(b) 동명 헤더 두 개
+    보냄:   X-Auth-Request-Roles: admin
+            X-Auth-Request-Roles: editor
+    도착:   ['admin', 'editor']              ← ★ 둘 다 도착. 덮어쓰지도 합치지도 않는다
+
+(c) 값 안에 구분자가 들어간 경우
+    보냄:   X-Auth-Request-Roles: role-with,comma
+    도착:   ['role-with,comma']              ← (a) 와 구별 불가
+
+=== Q4 ② 헤더 크기 상한 ===
+  보낸 길이   1000 → HTTP 200, 도착 길이 1000
+  보낸 길이   4000 → HTTP 200, 도착 길이 4000
+  보낸 길이   8000 → HTTP 400  (Tomcat 의 HTML 오류 페이지)
+  보낸 길이  16000 → HTTP 000  (응답을 못 받음 = 연결이 끊김)
+  보낸 길이  32000 → HTTP 000
+
+  → 자르지 않는다. 거부한다. 그리고 거부하는 계층이 둘이며 증상이 다르다.
+
+=== Q4 ④ upstream 이 검증하는가 ===
+  아무 인증 없이 보냄:
+    x-auth-request-user        ['administrator']
+    x-auth-request-email       ['admin@example.com']
+    x-auth-request-roles       ['realm-admin,superuser']
+    remoteAddr                 100.123.124.30
+  → 그대로 도착. 검증 없음.
+
+  대조 — JWT 를 요구하는 경로:
+    /api/echo        HTTP 200   (permitAll)
+    /api/me          HTTP 401
+    /api/protected   HTTP 401
+
+  backend SecurityConfig:
+    .requestMatchers("/actuator/health", "/actuator/health/**", "/api/public", ...).permitAll()
+    .anyRequest().authenticated()
+    .oauth2ResourceServer(oauth2 -> oauth2.jwt(...))
@@ -0,0 +1,14 @@
+# B-4 — Edge 인가 범위 증거
+
+2026-09-04 15:25–15:35 KST
+해설: [`docs/experiment-b4-edge-authorization-scope.md`](../../experiment-b4-edge-authorization-scope.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-header-handling.txt` | ① 동명 헤더가 **둘 다 도착**(`['admin','editor']`)하고 값 안의 쉼표를 구분자와 구별할 수 없다 · ② 8KB 에서 Tomcat 400, 16KB 에서 연결 끊김 — **자르지 않고 거부** · ④ 위조 신원 헤더가 그대로 도착, JWT 경로는 401 |
+
+## 핵심 세 줄
+
+1. **Q4 의 「nginx 가 동명 헤더를 덮어쓴다」는 조건부다.** nginx 는 자기가 `proxy_set_header` 한 헤더만 덮어쓰고, 나머지는 통과시킨다 — 지금 `X-Auth-Request-*` 는 통과한다.
+2. **크기는 절벽이다.** 점진적으로 나빠지지 않고 8KB 에서 전면 400 이 되며, role 이 많은 사용자만 깨진다.
+3. **헤더를 인가 근거로 쓰면 위조 가능성이 곧 권한 상승이다.** 2홉 실험의 결론이 여기서는 신원 자체에 적용된다.
@@ -0,0 +1,9 @@
+=== 기준선 ===
+  Redis 키: 1
+  PostgreSQL 토큰: 1 행
+  Redis 영속화 설정:
+    save = save	
+    appendonly	no
+
+=== 외부 진입점 정상 확인 ===
+  https://app1.hyeonworks.com/  HTTP 200
@@ -0,0 +1,25 @@
+=== ① Redis 정지 ===
+  정지: 14:26:30
+deployment.apps/redis scaled
+  삭제 완료
+
+=== 로그인한 사용자의 다음 요청은 어떻게 되는가 ===
+  /                      HTTP 200
+  /bff/token-boundary    HTTP 000
+  /actuator/health       HTTP 503
+  --- token-boundary 응답 본문 ---
+
+
+=== 파드 상태 — readiness 가 Redis 를 보는가 ===
+bff-555df79c97-6j86w   1/1   Running   0     17m
+bff-555df79c97-vgg6g   1/1   Running   0     16m
+
+=== health 상세 ===
+
+
+=== BFF 로그 ===
+	at java.base/sun.nio.ch.Net.pollConnect(Native Method) ~[na:na]
+	at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source) ~[na:na]
+	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) ~[na:na]
+	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:336) ~[netty-transport-4.1.135.Final.jar!/:4.1.135.Final]
+	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339) ~[netty-transport-4.1.135.Final.jar!/:4.1.135.Final]
@@ -0,0 +1,13 @@
+=== health 그룹별 응답 — 왜 파드는 Ready 인가 ===
+  /actuator/health             HTTP server
+  /actuator/health/readiness   HTTP 200
+  /actuator/health/liveness    HTTP 200
+
+=== /actuator/health 본문 (Redis 항목이 있는가) ===
+
+
+=== /actuator/health/readiness 본문 ===
+{"status":"UP"}
+
+=== Service 엔드포인트 — 트래픽을 계속 받는가 ===
+  ready: [10.42.0.52 10.42.1.124]
@@ -0,0 +1,39 @@
+=== 복구 ===
+deployment.apps/redis scaled
+deployment "redis" successfully rolled out
+  /actuator/health       HTTP 200
+  /bff/token-boundary    HTTP 302
+  BFF 재시작 필요했나: 0,0 회 재시작
+
+=== ② 영속화 — 지금 설정으로 재시작하면 무엇이 남는가 ===
+  키 심음: before-restart
+  dbsize: 4
+
+  --- AOF 를 켜고 다시 심는다 (영속화가 켜져 있으면 살아남는가) ---
+    appendonly	yes
+    total 12
+    drwxr-xr-x    3 redis    redis         4096 Sep  4 05:26 .
+    drwxr-xr-x    1 root     root          4096 Sep  4 05:26 ..
+    drwx------    2 redis    redis         4096 Sep  4 05:26 appendonlydir
+
+  --- 파드를 지운다 ---
+deployment "redis" successfully rolled out
+  재기동 후:
+    dbsize:    0
+    b5:probe   
+    b5:aof     
+    appendonly no
+persistentvolumeclaim/redis-data created
+deployment.apps/redis configured
+deployment "redis" successfully rolled out
+
+=== 영속 볼륨 위에서 다시 시험 ===
+  appendonly	yes
+  키 심음: written-on-pvc
+sed: -e expression #1, char 8: unknown option to 's'
+
+  --- 파드를 지운다 ---
+deployment "redis" successfully rolled out
+  재기동 후:
+    dbsize:  1
+    b5:pvc   written-on-pvc
@@ -0,0 +1,17 @@
+# B-5 — Redis 상실과 영속화 증거
+
+2026-09-04 15:35–15:50 KST
+해설: [`docs/experiment-b5-redis-loss-persistence.md`](../../experiment-b5-redis-loss-persistence.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-baseline.txt` | 정지 전 — Redis 1키, PostgreSQL 1행, `save`/`appendonly no`, 외부 200 |
+| `02-redis-down.txt` | 정지 후 — `/bff/token-boundary` **`HTTP 000`(멈춤)**, `/actuator/health` 503, **파드는 1/1 Ready 유지**, Lettuce 재연결 스택 |
+| `03-health-groups.txt` | **핵심** — `/actuator/health` 503 인데 `/actuator/health/readiness` 는 `{"status":"UP"}`. Service 엔드포인트에 두 파드 모두 남아 있다 |
+| `04-persistence.txt` | 복구는 자동(재시작 0회) · **AOF 를 켰는데 파드 삭제 후 `dbsize 0`** · PVC 를 붙인 뒤 `written-on-pvc` **생존** |
+
+## 핵심 세 줄
+
+1. **파드가 Ready 를 유지한 채 계속 실패한다.** `redis` 헬스 지표가 readiness 그룹에 없기 때문이며, A-2 에서 Keycloak 이 NotReady 가 된 것과 정반대다.
+2. **오류가 아니라 멈춤이다.** `HTTP 000` — 빠른 실패가 안 되어 있어 사용자는 멈춘 화면을 본다.
+3. **볼륨 없이 AOF 만 켜는 것은 장식이다.** `appendonlydir` 까지 만들어지지만 컨테이너와 함께 사라진다.
@@ -0,0 +1,9 @@
+=== [1] 회전 전: 토큰 발급 + JWKS 상태 ===
+  발급 토큰의 kid: OY-caYDNGoP4HMAz-Q9UPTU-DM1i896NuzUZu6gfCqM
+  JWKS 의 RS256 키 수: 1
+  JWKS kid 목록:
+    {"keys":[{"kid":"gokjn0zFUok8r7JVqW1cxuyojH1bTT87vzfQG9RrFX4"
+    {"kid":"OY-caYDNGoP4HMAz-Q9UPTU-DM1i896NuzUZu6gfCqM"
+
+=== [2] 그 토큰이 지금 통하는가 (리소스 서버) ===
+  /api/me  HTTP 200
@@ -0,0 +1,16 @@
+=== [3] 키 회전 — 우선순위가 더 높은 RSA 공급자를 추가한다 ===
+Created new component with id '7902af43-a0cc-4ebd-ad25-04d563854d16'
+
+=== [4] 회전 후 JWKS — 옛 키가 남아 있는가 ===
+  RS256 키 수: 2
+  kid 목록:
+    {"keys":[{"kid":"1B4AQHoxZvFaQi1tc1byz8ifU-nYFB6engD4YB4Fz84"
+    {"kid":"gokjn0zFUok8r7JVqW1cxuyojH1bTT87vzfQG9RrFX4"
+    {"kid":"OY-caYDNGoP4HMAz-Q9UPTU-DM1i896NuzUZu6gfCqM"
+
+=== [5] 새 토큰은 어느 키로 서명되는가 ===
+  새 토큰의 kid: 1B4AQHoxZvFaQi1tc1byz8ifU-nYFB6engD4YB4Fz84
+
+=== [6] ★ 회전 전에 발급된 토큰은 아직 통하는가 ===
+  옛 토큰  /api/me  HTTP 200
+  새 토큰  /api/me  HTTP 200
@@ -0,0 +1,16 @@
+=== [7] 옛 RSA 공급자(980ee9b7 = OY-caYDN 키) 제거 ===
+  제거 완료
+
+=== [8] JWKS 에서 사라졌는가 ===
+  RS256 키 수: 1
+    {"keys":[{"kid":"1B4AQHoxZvFaQi1tc1byz8ifU-nYFB6engD4YB4Fz84"
+    {"kid":"gokjn0zFUok8r7JVqW1cxuyojH1bTT87vzfQG9RrFX4"
+
+=== [9] ★ 옛 키로 서명된 토큰은 이제 어떻게 되는가 ===
+  옛 토큰  /api/me  HTTP 401  (캐시가 살아 있으면 아직 통할 수 있다)
+  새 토큰  /api/me  HTTP 200
+
+=== [10] 리소스 서버를 재시작해 JWKS 캐시를 비우면 ===
+deployment "echo" successfully rolled out
+  옛 토큰  /api/me  HTTP 401
+  새 토큰  /api/me  HTTP 200
@@ -0,0 +1,16 @@
+# B-6 — key 회전 증거
+
+2026-09-04 15:50–16:00 KST
+해설: [`docs/experiment-b6-key-rotation.md`](../../experiment-b6-key-rotation.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-before-rotation.txt` | 회전 전 — 토큰 `kid=OY-caYDN...`, JWKS RS256 1개, `/api/me` 200 |
+| `02-rotation.txt` | 우선순위 200 공급자 추가 → **JWKS RS256 2개**, 새 토큰은 새 kid, **옛 토큰도 새 토큰도 200** (무중단) |
+| `03-old-key-removed.txt` | 옛 공급자 제거 → JWKS 1개, **옛 토큰 즉시 401**. 리소스 서버 재시작 후에도 동일 |
+
+## 핵심 세 줄
+
+1. **Keycloak 의 키 회전은 "바꾸기"가 아니라 "더 높은 우선순위로 추가하기"** 다. 추가만으로는 아무것도 안 깨진다.
+2. **위험한 것은 옛 키를 버리는 시점이다.** 제거 즉시 그 키로 서명된 토큰이 401 이 된다.
+3. **캐시는 유예가 아니다.** 모르는 `kid` 를 만나면 JWKS 를 다시 받으므로 제거가 곧바로 반영된다. 유예는 옛 키를 남겨두는 기간으로 만들어야 한다.
@@ -0,0 +1,13 @@
+=== Grafana ingress 를 잠시 내린다 (app2 를 빌린다) ===
+  grafana ingress 삭제
+secret/oauth2-proxy-secrets created
+deployment.apps/oauth2-proxy created
+service/oauth2-proxy created
+ingress.networking.k8s.io/oauth2-proxy created
+deployment "oauth2-proxy" successfully rolled out
+oauth2-proxy-c76b49c59-8p5hl   true   kc-lab-1
+oauth2-proxy-c76b49c59-b9928   true   kc-lab-2
+
+=== 진입점 확인 ===
+  https://app2.hyeonworks.com/  HTTP 302
+  /ping                         HTTP 200
@@ -0,0 +1,9 @@
+=== curl 로 OAuth 흐름을 완주한다 (nginx 우회, Traefik 직접) ===
+  로그인 폼 action: https://auth.hyeonworks.com/realms/keycloak-patterns/login-actions/authenticate?session_co...
+  쿠키 항아리:
+                                 len=0
+    KC_AUTH_SESSION_HASH         len=64
+
+=== 두 replica 모두 이 쿠키를 받아들이는가 ===
+  10.42.1.135    /oauth2/auth  HTTP 000
+  10.42.1.134    /oauth2/auth  HTTP 000
@@ -0,0 +1,31 @@
+=== 세션이 Redis 에 들어갔는가 ===
+b5:pvc
+_oauth2_proxy-b26111fbd1fdab3ae2182e287001b02a
+  dbsize: 2
+
+=== oauth2-proxy 가 cookie secret 을 여러 개 받는가 ===
+        --cookie-secret string                                the seed string for secure cookies (optionally base64 encoded)
+
+=== ★ secret 을 A → B 로 교체한다 ===
+deployment.apps/oauth2-proxy patched
+deployment "oauth2-proxy" successfully rolled out
+  현재 secret 키: COOKIE_SECRET_B
+  Redis 세션은 그대로인가: 2 키
+=== secret 교체 후 oauth2-proxy 로그 — 옛 쿠키를 어떻게 처리했나 ===
+[2026/09/04 05:41:46] [oauthproxy.go:178] Cookie settings: name:_oauth2_proxy secure(https):true httponly:true expiry:1h0m0s domains: path:/ samesite: refresh:disabled
+[2026/09/04 05:42:18] [oauthproxy.go:1024] No valid authentication in request. Initiating login.
+100.123.124.30 - cb8c0ec1-1d87-479c-9aef-e1d9158a5829 - - [2026/09/04 05:42:18] app2.hyeonworks.com GET - "/api/echo" HTTP/1.1 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, lik
+[2026/09/04 05:42:18] [stored_session.go:94] Error loading cookied session: session ticket cookie failed validation: <nil>, removing session
+[2026/09/04 05:42:18] [stored_session.go:97] Error removing session: error decoding ticket to clear session: session ticket cookie failed validation: <nil>
+100.123.124.30 - 28af938f-08b5-4e15-9094-8d9591a18a3f - labuser@example.com [2026/09/04 05:42:18] app2.hyeonworks.com GET / "/api/echo" HTTP/1.1 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/
+[2026/09/04 05:41:58] [providers.go:146] Warning: Your provider supports PKCE methods ["plain" "S256"], but you have not enabled one with --code-challenge-method
+[2026/09/04 05:41:58] [oauthproxy.go:172] OAuthProxy configured for OpenID Connect Client ID: oauth2-proxy
+[2026/09/04 05:41:58] [oauthproxy.go:178] Cookie settings: name:_oauth2_proxy secure(https):true httponly:true expiry:1h0m0s domains: path:/ samesite: refresh:disabled
+100.123.124.30 - 5a08219f-60e0-4c97-bfee-78cae8891ca8 - labuser@example.com [2026/09/04 05:42:18] [AuthSuccess] Authenticated via OAuth2: Session{email:labuser@example.com user:27df5ea9-8703
+100.123.124.30 - 5a08219f-60e0-4c97-bfee-78cae8891ca8 - - [2026/09/04 05:42:18] app2.hyeonworks.com GET - "/oauth2/callback?state=j7eKInWCrYqRyi5LVDGDjtLrIBJCpwdkmzoJVdhJUc0%3A%2Fapi%2Fecho&
+100.123.124.30 - a3074807-5143-49b9-b77c-e7e2eb90ac24 - labuser@example.com [2026/09/04 05:42:18] app2.hyeonworks.com GET / "/favicon.ico" HTTP/1.1 "Mozilla/5.0 (X11; Linux x86_64) AppleWebK
+
+=== Redis 세션 수 (옛 세션이 남아 있는가) ===
+  _oauth2_proxy-978dfaefbdadccb96c7be1625dba5616
+  _oauth2_proxy-b26111fbd1fdab3ae2182e287001b02a
+  총: 2 개
@@ -0,0 +1,18 @@
+# B-7 — oauth2-proxy cookie secret 교체 증거
+
+2026-09-04 16:00–16:15 KST
+해설: [`docs/experiment-b7-cookie-secret-rotation.md`](../../experiment-b7-cookie-secret-rotation.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-deploy.txt` | 양 노드에 replica 하나씩. `/` 302, `/ping` 200 |
+| `02-cookie-portability.txt` | curl 로 흐름을 완주하려던 시도 — 파드 IP 는 호스트에서 안 닿는다 |
+| `03-rotation.txt` | **`--cookie-secret string` 단수 확인** · 교체 후 `session ticket cookie failed validation` · **`Error removing session`** · Redis 에 **고아 세션 2개** |
+| `b7-oauth2proxy-login-success.png` | Redis 세션 전환 후 성공한 Forward-Auth — `x-forwarded-user/email/preferred-username` 과 **티켓 형태 쿠키** |
+
+## 핵심 네 줄
+
+1. **BFF 와 정반대다.** 인가 요청이 쿠키에 있어 **콜백이 다른 replica 로 가도 성공**한다 — B-0 에서 BFF 가 실패한 바로 그 지점.
+2. **502 의 원인은 큰 쿠키였다.** Traefik 직접은 정상이고 nginx 만 502 — B-4 의 헤더 절벽이 응답 쪽에서 재현됐다.
+3. **겹침 구간을 만들 수 없다.** `--cookie-secret` 이 단수라 B-6 의 무중단 회전이 불가능하다.
+4. **교체하면 서버 세션이 고아로 남는다.** 티켓을 못 푸니 지울 수도 없다.
@@ -0,0 +1,11 @@
+=== 깨끗한 상태로 초기화 ===
+DELETE 1
+
+=== 기준선 ===
+  Keycloak 온라인 세션: 4
+  Redis 키: 0
+
+=== 두 앱의 구조 ===
+  app1.hyeonworks.com  →  BFF        (서버 세션: Redis + PostgreSQL)
+  app2.hyeonworks.com  →  oauth2-proxy (쿠키 티켓 + Redis)
+  둘 다 realm keycloak-patterns 를 쓴다
@@ -0,0 +1,9 @@
+=== app1 로그인 직후 Keycloak 세션 ===
+     user_session_id      | client_sessions 
+--------------------------+-----------------
+ oqOjHekin4JU-BZjgQLjUByW |               1
+(1 row)
+
+  Redis 키: 1
+    bff:session:sessions:6e0d9af4-2c8f-47d2-bf83-8b1e9670c679
+  PostgreSQL authorized client: 1 행
@@ -0,0 +1,20 @@
+=== app2 방문 후 — 로그인 화면 없이 통과했는가 ===
+     user_session_id      | client_sessions 
+--------------------------+-----------------
+ oqOjHekin4JU-BZjgQLjUByW |               2
+(1 row)
+
+
+=== 어느 클라이언트가 붙었는가 ===
+              client_id               |       name       
+--------------------------------------+------------------
+ 9055fa46-6abb-4d6d-a339-8a9183bbf26d | bff-confidential
+ 80431dbc-af81-4673-9790-ad06d1570b2e | oauth2-proxy
+(2 rows)
+
+
+=== 저장소 상태 ===
+  Redis 키:
+    _oauth2_proxy-6b028a70f69c8f0da9966eb36972dff2
+    bff:session:sessions:6e0d9af4-2c8f-47d2-bf83-8b1e9670c679
+  PostgreSQL authorized client: 1 행
@@ -0,0 +1,25 @@
+=== ★ Keycloak 의 SSO 세션 하나를 죽인다 ===
+  남은 Keycloak 세션: 1
+
+=== 두 앱의 애플리케이션 세션은 그대로인가 ===
+  _oauth2_proxy-6b028a70f69c8f0da9966eb36972dff2
+  bff:session:sessions:6e0d9af4-2c8f-47d2-bf83-8b1e9670c679
+  PostgreSQL authorized client: 1 행
+
+  → IdP 세션은 없어졌는데 앱 세션은 남아 있다면, 두 계층의 수명이 어긋난 것이다
+=== 사용자 단위 로그아웃 (IdP 세션만 끊는다) ===
+  남은 Keycloak 세션: 1
+
+=== 앱 세션은 남아 있는가 ===
+  _oauth2_proxy-6b028a70f69c8f0da9966eb36972dff2
+  bff:session:sessions:6e0d9af4-2c8f-47d2-bf83-8b1e9670c679
+  PostgreSQL authorized client: 1 행
+=== 남은 세션의 realm 과 client ===
+     user_session_id      | realm  | clients 
+--------------------------+--------+---------
+ E1q5xI7tt4U_WhZpW7rEPIF2 | master |       1
+(1 row)
+
+
+=== 브라우저에서 두 앱을 다시 열면 어떻게 되는가 ===
+  (IdP 세션이 사라졌으면 재로그인이 필요해야 한다)
@@ -0,0 +1,19 @@
+# C-1 — 다중 앱 SSO 증거
+
+2026-09-04 16:15–16:30 KST
+해설: [`docs/experiment-c1-multi-app-sso.md`](../../experiment-c1-multi-app-sso.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-baseline.txt` | 초기화 시도 — `logout-all` 이 안 먹어 세션 4개가 남았다 |
+| `02-after-app1-login.txt` | app1 로그인 후 — user session 1 · client session 1 · Redis 1 · DB 1행 |
+| `03-after-app2-visit.txt` | **app2 방문 후 client session 1 → 2**, `bff-confidential` 과 `oauth2-proxy` 가 같은 user session 에 붙음. Redis 에 두 종류 세션 |
+| `04-sso-session-killed.txt` | IdP 세션 삭제 후 — **앱 세션 셋 다 남아 있다**. realm 을 join 해 보고서야 남은 것이 master 세션임을 확인 |
+| `c1-sso-app2-no-login-screen.png` | app2 가 로그인 화면 없이 열린 화면 |
+| `c1-apps-alive-after-idp-logout.png` | **IdP 세션을 죽인 뒤에도 그대로 열리는 화면** |
+
+## 핵심 세 줄
+
+1. **SSO 는 user session 1개에 client session N개** 구조다 — A-3(전체 소실)과 B-3(client 만 제거)의 차이가 여기서 의미를 갖는다.
+2. **IdP 세션을 죽여도 두 앱은 계속 동작한다.** 세 층(IdP·앱·토큰)의 수명이 각자이기 때문이다.
+3. **IdP 는 "로그인 경로"의 단일 장애점이지 "이미 로그인한 사용자"의 단일 장애점이 아니다.** 장애는 앱 세션 수명만큼 지연되어 몰려온다.
@@ -0,0 +1,12 @@
+=== 현재 클라이언트의 백채널 로그아웃 설정 ===
+--- bff-confidential ---
+    "frontchannelLogout" : false,
+--- oauth2-proxy ---
+    "frontchannelLogout" : false,
+
+=== BFF 가 백채널 로그아웃 엔드포인트를 갖고 있는가 ===
+
+=== 실제로 그 경로가 있는가 ===
+  /logout/connect/back-channel/keycloak      HTTP 302
+  /backchannel-logout                        HTTP 302
+  /oauth2/sign_out                           HTTP 302
@@ -0,0 +1,8 @@
+=== IdP 쪽에만 백채널 로그아웃 URL 을 설정한다 ===
+  client id: 9055fa46-6abb-4d6d-a339-8a9183bbf26d
+command terminated with exit code 1
+
+=== 로그인 상태를 만든다 ===
+  (브라우저에 이미 세션이 있다)
+  Keycloak 세션: 2
+  Redis: 2 키
@@ -0,0 +1,28 @@
+=== 로그아웃 전 상태 ===
+  Redis: 2 키
+  keycloak-patterns 세션: 0
+
+=== ★ IdP 로그아웃 — Keycloak 이 백채널 알림을 보낼 것이다 ===
+  시각: 14:53:29
+
+=== Keycloak 로그 — 백채널 요청을 보냈는가, 결과는 ===
+
+=== BFF 로그 — 백채널 요청이 도착했는가 ===
+
+=== 앱 세션이 정리되었는가 ===
+  Redis: 2 키
+    _oauth2_proxy-6b028a70f69c8f0da9966eb36972dff2
+    bff:session:sessions:6e0d9af4-2c8f-47d2-bf83-8b1e9670c679
+=== 로그아웃 전 — 실제 세션이 있는가 ===
+  keycloak-patterns 세션: 1
+  Redis: 1 키
+
+=== ★ IdP 로그아웃 → 백채널 알림 ===
+  시각: 14:54:21
+
+=== Keycloak 로그 ===
+
+=== BFF 로그 — 요청이 왔는가 ===
+
+=== 앱 세션 ===
+  Redis: 1 키
@@ -0,0 +1,15 @@
+=== IdP 세션은 실제로 끊겼는가 ===
+  keycloak-patterns 세션: 0
+
+=== ★ Keycloak 파드가 app1.hyeonworks.com 에 닿는가 ===
+  DNS 해석:
+    Address: 100.83.212.4
+    
+    Non-authoritative answer:
+    
+  HTTPS 도달:
+    HTTP 200  (0 이면 못 닿음)
+
+=== Keycloak 로그 전체에서 backchannel 흔적 ===
+  keycloak-0: 0 줄
+  keycloak-1: 0 줄
@@ -0,0 +1,17 @@
+# C-2 — 백채널 로그아웃 증거
+
+2026-09-04 16:30–16:55 KST
+해설: [`docs/experiment-c2-backchannel-logout.md`](../../experiment-c2-backchannel-logout.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-current-state.txt` | 두 클라이언트 모두 `backchannelLogoutUrl` 없음 · BFF 소스에 `oidcLogout` 없음 · 후보 경로 셋 다 **302**(핸들러 없음) |
+| `02-configure-idp.txt` | IdP 쪽에만 `backchannel.logout.url` 설정 (점 표기는 실패, JSON 으로 성공) |
+| `03-logout-attempt.txt` | **살아 있는 세션(1)에 로그아웃 → IdP 세션 0, Redis 세션은 1 그대로.** Keycloak·BFF 로그에 흔적 없음 |
+| `04-reachability.txt` | **Keycloak 파드가 `app1.hyeonworks.com` 에 `HTTP 200` 으로 닿는다** — 네트워크 문제가 아님 |
+
+## 핵심 세 줄
+
+1. **백채널 로그아웃은 어느 쪽에도 구현되어 있지 않았다.** C-1 이 관측한 "전파 안 됨"의 원인이다.
+2. **IdP 쪽만 설정해도 소용없다.** 받을 엔드포인트와 `sid → 세션` 역인덱스가 앱에 있어야 한다.
+3. **도달성이 숨은 전제다.** 이 실험대는 닿지만, 앱이 사설망에 있으면 설정해도 조용히 실패한다.
@@ -0,0 +1,15 @@
+=== 백업 전 상태 ===
+  realms|clients|users|sessions|authclients = 2|15|2|3|1
+
+=== pg_dump — 전체 덤프 ===
+  시작: 14:59:30
+  완료: 14:59:30
+  크기: 394945 bytes  (6956 줄)
+  포함된 테이블 수: 101
+
+=== 덤프에 세션이 들어 있는가 ===
+  offline_user_session 언급: 13
+    COPY public.offline_user_session (user_session_id, user_id, realm_id, created_on, offline_flag, data, last_session_refre
+    E1q5xI7tt4U_WhZpW7rEPIF2	48b37d33-8419-49aa-9b5b-7731975be50c	7845f394-723a-4d07-b530-c7416b2e1d31	1788500836	0	{"ipAddr
+    2ap3DyRiBF8OdMiqCodsJ0mp	48b37d33-8419-49aa-9b5b-7731975be50c	7845f394-723a-4d07-b530-c7416b2e1d31	1788501029	0	{"ipAddr
+    Zsk4QcgXf_qgyMKzde5AG-Fz	48b37d33-8419-49aa-9b5b-7731975be50c	7845f394-723a-4d07-b530-c7416b2e1d31	1788501263	0	{"ipAddr
@@ -0,0 +1,17 @@
+=== ★ 파괴 — 스키마를 통째로 지운다 ===
+  시각: 14:59:47
+DROP SCHEMA
+CREATE SCHEMA
+  남은 테이블: 0
+
+=== 서비스 영향 ===
+  https://auth.hyeonworks.com/realms/master   HTTP 200
+  https://app1.hyeonworks.com/                HTTP 200
+bff-555df79c97-6j86w           1/1   Running   0     49m
+bff-555df79c97-vgg6g           1/1   Running   0     49m
+keycloak-0                     1/1   Running   0     4m15s
+keycloak-1                     1/1   Running   0     4m38s
+
+=== Keycloak 이 무엇을 말하는가 ===
+  2026-09-04 05:58:02,598 WARN  [org.keycloak.jgroups.protocol.KEYCLOAK_JDBC_PING2] (blocking-thread--p3-t2) Failed to fetch the cluster members from the database.: org.postgresql.ut
+  	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2904)
@@ -0,0 +1,29 @@
+=== 무엇이 실제로 깨지는가 ===
+  /.well-known/openid-configuration      HTTP 500
+  /protocol/openid-connect/certs         HTTP 200
+  토큰 발급 (DB 쓰기 필요)       HTTP 400
+
+=== ★ 복구 — 덤프에서 되돌린다 ===
+  시작: 15:00:12
+  완료: 15:00:13
+  오류 줄: 0
+
+=== 복구 후 데이터 ===
+  realms|clients|users|sessions|authclients = 2|15|2|3|1
+=== 복구 직후 — 재시작 없이 되는가 ===
+  +15초  well-known=200  토큰발급=200
+  → 재시작 없이 회복
+
+=== 복구 전 세션이 살아났는가 ===
+     user_session_id      |       realm       
+--------------------------+-------------------
+ E1q5xI7tt4U_WhZpW7rEPIF2 | master
+ 2ap3DyRiBF8OdMiqCodsJ0mp | master
+ Zsk4QcgXf_qgyMKzde5AG-Fz | master
+ vsDgCVo12-qX0CC63ZmYzbYF | keycloak-patterns
+(4 rows)
+
+
+=== 파드 재시작 횟수 ===
+  keycloak-0 restarts=0
+  keycloak-1 restarts=0
@@ -0,0 +1,16 @@
+# D-1 — 백업·복구 리허설 증거
+
+2026-09-04 16:55–17:05 KST
+해설: [`docs/experiment-d1-backup-restore.md`](../../experiment-d1-backup-restore.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-backup.txt` | `pg_dump --clean --if-exists` — 395KB · 101 테이블 · **세션 데이터 포함** |
+| `02-destruction.txt` | `DROP SCHEMA public CASCADE` → 테이블 0개. **그런데 외부는 `HTTP 200`** — Keycloak 이 realm 캐시로 서빙한다 |
+| `03-restore.txt` | 깨지는 것과 안 깨지는 것(`certs` 200 / `well-known` 500 / 토큰 400) · **복구 1초 · 오류 0건 · 데이터 완전 일치 · 재시작 0회** |
+
+## 핵심 세 줄
+
+1. **데이터베이스를 통째로 비웠는데 서비스가 200 을 냈다.** 헬스체크는 "DB 가 살아 있다"만 보고 "데이터가 있다"는 안 본다.
+2. **복구는 1초, 오류 0건, 재시작 불필요.** 절차가 맞다는 것은 확인됐다.
+3. **RPO 는 두 겹이다** — 백업 주기 + A-3 에서 측정한 `synchronous_commit OFF` 손실. 그리고 이번 덤프는 호스트의 `/tmp` 에 있어 **같은 장애 도메인**이다.
@@ -0,0 +1,9 @@
+=== D-1 의 교훈: 업그레이드 전에 백업한다 ===
+  백업: 396333 bytes
+
+=== 현재 버전과 스키마 상태 ===
+quay.io/keycloak/keycloak:26.7.0
+  총 마이그레이션 수: 210
+
+=== 로그인 상태 만들기 (업그레이드 후 살아남는지 볼 것) ===
+  현재 세션: 4
@@ -0,0 +1,17 @@
+=== ★ 롤백 시도: 26.7.0 → 26.0 ===
+  시각: 15:02:20
+statefulset.apps/keycloak image updated
+  +20초  keycloak-0:Running(1/1) keycloak-1:Running(0/1) 
+  +40초  keycloak-0:Running(1/1) keycloak-1:Running(0/1) 
+  +60초  keycloak-0:Running(1/1) keycloak-1:Running(0/1) 
+  +80초  keycloak-0:Running(1/1) keycloak-1:Error(0/1) 
+  +100초  keycloak-0:Running(1/1) keycloak-1:Running(0/1) 
+  +120초  keycloak-0:Running(1/1) keycloak-1:Error(0/1) 
+  +140초  keycloak-0:Running(1/1) keycloak-1:CrashLoopBackOff(0/1) 
+  +160초  keycloak-0:Running(1/1) keycloak-1:Running(0/1) 
+
+=== 새 파드가 무엇을 말하는가 ===
+2026-09-04 06:03:25,877 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (production) mode
+2026-09-04 06:03:25,877 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: liquibase.exception.ValidationFailedException: Validation Failed:
+2026-09-04 06:03:25,877 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Validation Failed:
+2026-09-04 06:03:25,877 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the detai
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
DongHyeonkaandClaude Opus 5	027c24ee27	docs: D-3 — only RBAC actually hides anything Every secret in the lab prints in four commands, while kubectl describe shows just a byte count and creates the impression that something is hidden. k3s reports encryption at rest disabled and the plaintext password is present in state.db, so one node disk carries the whole cluster's secrets, and inside the pod they are ordinary environment variables visible to exec, /proc and crash dumps. The default service account cannot read secrets, which makes RBAC the one control doing real work here and the thing worth tightening. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 15:07:35 +09:00
DongHyeonkaandClaude Opus 5	df140ab218	docs: D-2 — rolling back the image does not roll back the schema Downgrading from 26.7.0 to 26.0 fails with liquibase ValidationFailedException on a changeset checksum, which is stricter than an unknown migration: the old version knows the changeset but its definition differs. The pod goes CrashLoopBackOff and never starts. The StatefulSet stopped the rollout at the first pod, so the other kept serving and the front door stayed at 200, which replica 1 would not have done. The failed start never touched the schema, so restoring the image was enough; had the migration already applied, the D-1 database restore would have been the only way back. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 15:05:36 +09:00
DongHyeonkaandClaude Opus 5	df5af95cb3	docs: D-1 — an empty database still answered 200 Dropping the schema left Keycloak serving realm metadata and JWKS from its Infinispan cache, so the front door stayed at 200 while only the paths that read the database failed. That is a different shape from A-2, where the connection itself broke and readiness pulled the pods out of the Service; here the connection is fine and the tables are simply gone, which the health check does not notice. Restoring the pg_dump took one second with zero errors and no pod restart, and the row counts matched the backup exactly, sessions included. The real RPO is the backup interval plus the synchronous_commit loss measured in A-3, and this dump sits in the host's /tmp, which is the same failure domain as the thing it protects. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 14:59:56 +09:00
DongHyeonkaandClaude Opus 5	6c310c93b7	docs: C-2 — nothing propagates because nobody implemented the receiving end Neither client had a backchannel logout URL and the BFF has no oidcLogout configuration, so the three candidate paths all answer 302, which is the authentication redirect rather than a handler. Setting the URL on the identity provider alone changed nothing: with a live session, logging the user out emptied the Keycloak side and left the Redis session untouched. Reachability is not the blocker here, since a Keycloak pod fetches the app's public URL with a 200, but that is a property of this tailnet split-DNS lab and is the assumption most likely to fail in production, where it fails silently. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 14:56:58 +09:00
DongHyeonkaandClaude Opus 5	e856e7af4d	docs: C-1 — killing the SSO session logs nobody out One user session carries a client session per application, so visiting the second app skips the login screen. Deleting the identity provider session leaves both application sessions untouched and both apps keep serving, because the identity provider, the application session and the access token each have their own lifetime. That inverts the B-2 finding: there the app session was cleared and the surviving SSO session let the user straight back in. Either way, clearing one side leaves the other. It also means an identity provider outage is a single point of failure for logging in, not for already-authenticated users, and the failure arrives late and all at once. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 14:49:36 +09:00
DongHyeonkaandClaude Opus 5	bcb563a04e	docs: B-7 — the cookie secret has no overlap window and rotation orphans sessions oauth2-proxy carries the authorization request in a signed cookie, so the callback can land on a different replica and still succeed, which is the opposite of the BFF failure in B-0. Sharing is therefore just sharing one Secret. Rotating it is all-or-nothing: --cookie-secret is singular, so there is no second key to read old tickets with, and the log shows both the validation failure and Error removing session, leaving the Redis session orphaned because the key cannot be derived from a ticket that will not decode. Getting there required two diagnoses: the callback 502 came from the full session riding in Set-Cookie past nginx's buffer, and every earlier attempt to read nginx config returned nothing because sudo on the host asks for a password while the guests do not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 14:44:00 +09:00
DongHyeonkaandClaude Opus 5	aa2c3907f5	docs: B-6 — rotation is safe, retiring the old key is not Adding a higher-priority RSA provider leaves both kids in JWKS, so tokens signed before and after the rotation both validate. Deleting the old provider makes its tokens 401 immediately, and the resource server's JWKS cache does not buy a grace period because an unknown kid triggers a refetch. The encryption key Q3 asks about does not exist yet, since B-2 showed the tokens are stored as plaintext JWTs, so the measured signing-key rotation is what its design has to copy: write with one key, read with several, and keep the overlap longer than the lifetime of anything signed with the old one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 14:33:42 +09:00
DongHyeonkaandClaude Opus 5	f45a2a2aaa	docs: B-5 — the pod stays Ready while every request hangs Stopping Redis returns HTTP 000 rather than an error because the client waits on reconnect, and the pod keeps serving traffic because the redis health indicator is not in the readiness group even though /actuator/health returns 503. That is the mirror image of A-2, where Keycloak put its database check in readiness and the pods left the Service. Turning on AOF with config set created the appendonlydir and still lost everything on pod deletion, because /data was the container filesystem; adding a PVC makes the same setting work. Volume first, persistence setting second. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 14:29:17 +09:00
DongHyeonkaandClaude Opus 5	7dc0a3e5da	docs: B-4 — the edge does not overwrite the headers it never sets Two headers of the same name both arrive rather than one overwriting the other, because nginx only replaces headers it sets with proxy_set_header. A comma inside a role name is indistinguishable from the delimiter, and the size limit is a cliff: Tomcat returns 400 around 8KB and the connection dies around 16KB, so the same cause produces two different-looking failures. Forged identity headers reach the upstream untouched while the JWT-protected paths return 401, which is Q4's own point that a header-fed upstream has nothing to verify against. By Q4's checklist that answer alone points at the BFF structure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 14:23:41 +09:00
DongHyeonkaandClaude Opus 5	b16e1dccf7	docs: B-3 — concurrent refresh does not lose a race, it destroys the session Five simultaneous refreshes with one token return a single 200, and that winner's new token is already dead. Reuse detection removes the client session while the user session stays, which is why the other responses read Session doesn't have required client rather than a reuse error. Comparing policies shows rotation off passes all five and keeps the session, while raising refreshTokenMaxReuse to one still destroys it. Since no retry can recover a removed client session, Q2's own criterion resolves to a lock, and a database row lock is the natural place because its lifetime is tied to the connection. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 14:20:04 +09:00
DongHyeonkaandClaude Opus 5	711878379c	docs: B-2 — sharing the stores fixes one problem and exposes three more Moving the authorized client to JdbcOAuth2AuthorizedClientService makes tokens work across replicas, so the session-in-Redis plus tokens-in-PostgreSQL split holds. The table then shows what sharing cannot fix: the primary key is (client_registration_id, principal_name) with no session in it, so a second login for the same user updates the same row rather than adding one. The refresh token sits in bytea as the raw JWT, readable with convert_from, and logout clears only the Redis session while the plaintext token row and the Keycloak SSO session both survive. The schema itself failed silently first because the default DDL uses blob, which PostgreSQL does not have, and continue-on-error swallowed it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 14:15:05 +09:00
DongHyeonkaandClaude Opus 5	f2595f748f	docs: B-1 — Redis moves the session and leaves the tokens behind Adding Spring Session Redis grows the context by 81 beans and swaps sessionRepository for RedisSessionRepository, while authorizedClientService stays InMemoryOAuth2AuthorizedClientService. The user then reads as logged in with principal labuser while accessTokenStoredOnServer is false, which is worse than being logged out. Redis holds only the security context, serialized with Java native serialization, and the refresh token is not there to encrypt in the first place. Three problems on the way: Kubernetes service links overwrote REDIS_PORT with a tcp:// URL, the tests tried to reach Redis, and the resource server was never deployed so a DNS failure looked like a token failure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 14:05:02 +09:00
				`@@ -0,0 +1 @@`
				`[ 236ms] [ERROR] Failed to load resource: the server responded with a status of 500 () @ https://app1.hyeonworks.com/bff/api/me:0`
				`@@ -0,0 +1 @@`
				`[ 307ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ https://app2.hyeonworks.com/favicon.ico:0`
				`@@ -0,0 +1 @@`
				`[ 261ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ https://app2.hyeonworks.com/favicon.ico:0`
				`@@ -0,0 +1 @@`
				`[ 882ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ https://app2.hyeonworks.com/favicon.ico:0`
				`@@ -0,0 +1 @@`
				`[ 178ms] [ERROR] Failed to load resource: the server responded with a status of 401 () @ https://app2.hyeonworks.com/favicon.ico:0`
				`@@ -0,0 +1 @@`
				`- generic [active] [ref=f30e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"`
				`@@ -0,0 +1 @@`
				`- generic [active] [ref=f32e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"`
				`@@ -0,0 +1 @@`
				`- generic [active] [ref=f34e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"`
				`@@ -0,0 +1 @@`
				`- generic [active] [ref=f35e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"`