docs: A-6 — 200ms of network delay becomes 22 seconds of user latency

Nine database round trips per login multiply the injected delay to 1.9 seconds, and connection pool queueing multiplies it again under twenty concurrent requests. The readiness probe joins the same queue and times out, so the node leaves the load balancer and pushes its load onto the one still standing.

Two injections missed first: the guest interface is enp1s0 rather than eth0, and a filter on it can never match a pod IP because flannel has already encapsulated the packet. The delay has to go on flannel.1, before encapsulation.

The predicted rise in optimistic lock conflicts did not happen, because logins insert new rows rather than contending for one. That belongs to B-3.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.playwright-mcp/console-2026-09-04T02-50-02-217Z.log

@@ -19,3 +19,23 @@
 [  137390ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  157071ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [  166091ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  182011ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  188613ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  206228ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  218561ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  229607ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  235818ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  237158ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  252095ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  261118ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  273809ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  280227ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  292650ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  306477ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  309957ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  311984ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  325683ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  344586ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  357576ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  359294ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  364740ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362

.playwright-mcp/console-2026-09-04T02-56-16-987Z.log

@@ -0,0 +1,129 @@
+[     615ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&panes=%7B%22te0%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22up%7Bjob%3D%5C%22keycloak%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22up+%E2%80%94+%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-20m%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1:0
+[     929ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    2566ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    5541ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    7990ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   12402ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   17640ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   20285ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   28277ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   34341ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   34985ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   50081ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   65649ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   75046ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   75890ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   96067ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  113467ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  121972ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  139681ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  150971ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  161211ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  180744ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  191401ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  210668ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  229082ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  239345ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  252119ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  266045ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  276705ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  289911ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  296875ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  315099ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  325431ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  328418ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  343774ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  347054ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  352405ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  361291ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  370639ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  374286ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  377413ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  394362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  405423ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  425476ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  434299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  441775ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  450786ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  456829ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  468812ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  478913ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  491241ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  495228ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  502295ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  511625ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  524619ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  533050ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  553189ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  571699ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  575413ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  588005ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  599161ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  608816ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  619031ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  626612ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  639003ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  655077ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  674532ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  677197ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  696027ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  709512ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  727505ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  747139ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  754565ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  773682ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  784674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  787399ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  802845ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  825589ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  832768ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  842043ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  852717ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  871748ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  876571ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  894491ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  895515ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  904310ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  916681ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  934752ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  941175ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  946004ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  960849ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  962180ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  984446ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  997579ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1010105ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1027190ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1043388ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1061311ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1066040ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1080349ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1092336ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1104312ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1109362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1127047ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1134728ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1154899ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1161253ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1174971ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1184491ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1185340ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1189111ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1196984ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1204794ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1247961ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1286322ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1319329ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1362771ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1396881ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1434184ms] [ERROR] Failed to load resource: the server responded with a status of 502 () @ https://app2.hyeonworks.com/api/user/auth-tokens/rotate:0
+[ 1443506ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1465016ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1513337ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1556906ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1562963ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1572896ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1576390ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 503 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1593794ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1600651ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1616424ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362

.playwright-mcp/console-2026-09-04T03-23-15-961Z.log

@@ -0,0 +1,125 @@
+[     268ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/login:0
+[     379ms] [VERBOSE] [DOM] Input elements should have autocomplete attributes (suggested: "username"): (More info: https://goo.gl/9p2vKq) %o @ https://app2.hyeonworks.com/login:0
+[   10453ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   10532ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&panes=%7B%22ich%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22up%7Bjob%3D%7E%5C%22keycloak%7Cnode-exporter%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bjob%7D%7D+%E2%80%94+%7B%7Bpod%7D%7D%7B%7Bnode%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-55m%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1:0
+[   11687ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   13737ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   15113ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   20185ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   27259ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   40154ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   41870ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   50493ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   54082ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   60121ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   74362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   83269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   95349ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   98008ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  104458ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  123816ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  142764ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  157198ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  175220ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  182187ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  183411ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  190581ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  199499ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  209995ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  214849ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  233084ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  242910ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  257975ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  277220ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  290806ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  292674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  300013ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  309267ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  323998ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  328620ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  339787ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  355959ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  361079ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  369272ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  381866ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  397120ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  400712ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  412579ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  431630ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  440746ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  447809ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  461741ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  475459ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  482723ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  494299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  513174ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  521541ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  532145ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  551129ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  556250ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  574381ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  586190ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  595782ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  610555ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  630695ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  633257ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  652194ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  671361ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  677606ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  692444ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  696949ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  700122ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  709954ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  723992ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  743173ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  745075ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  753299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  766683ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  767855ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  787261ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  789725ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  800680ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  811324ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  814096ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  829045ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  848506ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  857716ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  866212ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  873174ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  876033ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  895198ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  898884ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  918642ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  930133ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  934008ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  948541ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  956860ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  959397ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  968406ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  971788ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  982746ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  997394ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1005078ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1009269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1021453ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1026674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1042137ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1057497ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1059851ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1061855ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1071317ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1078179ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1084940ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1093950ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1112279ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1124360ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1139110ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1157981ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1163071ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1179032ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1181195ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1191955ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1201472ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1218261ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1221169ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1229430ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362

.playwright-mcp/console-2026-09-04T03-43-55-141Z.log

@@ -0,0 +1,50 @@
+[     199ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[     297ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22vendor_cluster_size%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-25m%22%2C%22to%22%3A%22now%22%7D%7D%7D:0
+[    1000ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    2023ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    3559ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    7749ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   16364ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   25781ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   42158ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   59672ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   77090ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   87426ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   89952ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  104743ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error in connection establishment: net::ERR_NAME_NOT_RESOLVED @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1573121ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1584038ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1601626ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1613280ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1630173ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1634776ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1654205ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1661576ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1665269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1682669ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1689435ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1704385ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1706314ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1724551ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1743701ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1758339ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1769299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1789478ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1795415ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1815382ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1820392ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1822445ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1839035ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1840162ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1847739ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1861419ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1880381ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1893315ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1911434ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1922701ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1933868ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1944951ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1948001ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1952503ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1953320ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1955369ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362

.playwright-mcp/console-2026-09-04T04-16-32-330Z.log

@@ -0,0 +1,12 @@
+[     585ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    1132ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&panes=%7B%22o68%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22agroal_blocking_time_max_milliseconds%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22blocking+max+-+%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-30m%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1:0
+[    1555ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    3604ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    6373ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[    9135ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   19066ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   24595ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   47125ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   52154ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   67608ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   86965ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362

.playwright-mcp/page-2026-09-04T02-56-17-648Z.yml

@@ -0,0 +1,145 @@
+- generic [active] [ref=f3e1]:
+  - generic [ref=f3e4]:
+    - link "Skip to main content" [ref=f3e5] [cursor=pointer]:
+      - /url: "#pageContent"
+    - banner [ref=f3e7]:
+      - generic [ref=f3e8]:
+        - link [ref=f3e10] [cursor=pointer]:
+          - /url: /
+          - img "Grafana" [ref=f3e11]
+        - generic [ref=f3e14]:
+          - button "Search or jump to..." [ref=f3e18] [cursor=pointer]
+          - generic [ref=f3e19]: ctrl+k
+        - generic [ref=f3e23]:
+          - button "New" [ref=f3e24] [cursor=pointer]
+          - button "Help" [ref=f3e30] [cursor=pointer]
+          - button "News" [ref=f3e33] [cursor=pointer]
+          - button "Profile" [ref=f3e36] [cursor=pointer]:
+            - img "User avatar" [ref=f3e37]
+      - generic [ref=f3e38]:
+        - button "Open menu" [ref=f3e40] [cursor=pointer]
+        - navigation "Breadcrumbs" [ref=f3e43]:
+          - list [ref=f3e44]:
+            - listitem [ref=f3e45]:
+              - link "Home" [ref=f3e46] [cursor=pointer]:
+                - /url: /
+            - listitem [ref=f3e50]:
+              - link "Explore" [ref=f3e51] [cursor=pointer]:
+                - /url: /explore
+            - listitem [ref=f3e55]:
+              - generic "Prometheus" [ref=f3e56]
+        - generic [ref=f3e57]:
+          - generic [ref=f3e60]:
+            - button "Copy shortened URL" [ref=f3e61] [cursor=pointer]
+            - button "Open copy link options" [ref=f3e64] [cursor=pointer]
+          - button "Toggle top search bar" [ref=f3e68] [cursor=pointer]
+    - main [ref=f3e74]:
+      - generic [ref=f3e76]:
+        - heading "Explore" [level=1] [ref=f3e77]
+        - generic [ref=f3e82]:
+          - navigation "Explore toolbar" [ref=f3e84]:
+            - navigation "Search links" [ref=f3e86]:
+              - generic [ref=f3e87]:
+                - button "Content outline" [expanded] [ref=f3e89] [cursor=pointer]:
+                  - generic [ref=f3e92]: Outline
+                - generic [ref=f3e97] [cursor=pointer]:
+                  - img "Prometheus logo" [ref=f3e99]
+                  - textbox "Select a data source" [ref=f3e100]:
+                    - /placeholder: ""
+            - generic [ref=f3e104]:
+              - button "Split the pane" [ref=f3e106] [cursor=pointer]:
+                - generic [ref=f3e109]: Split
+              - button "Add" [ref=f3e111] [cursor=pointer]
+              - generic [ref=f3e116]:
+                - 'button "Time range selected: Last 20 minutes" [ref=f3e117] [cursor=pointer]'
+                - button "Zoom out time range" [ref=f3e122] [cursor=pointer]
+              - generic [ref=f3e126]:
+                - button "Run query" [ref=f3e127] [cursor=pointer]
+                - button "Auto refresh turned off. Choose refresh time interval" [ref=f3e131] [cursor=pointer]
+          - generic [ref=f3e135]:
+            - generic [ref=f3e139]:
+              - button "Collapse outline" [expanded] [ref=f3e141] [cursor=pointer]:
+                - img "arrow-from-right" [ref=f3e142]
+              - button "Queries" [ref=f3e145] [cursor=pointer]:
+                - img "arrow" [ref=f3e146]
+              - button "Graph" [ref=f3e150] [cursor=pointer]:
+                - img "graph-bar" [ref=f3e151]
+            - generic [ref=f3e158]:
+              - generic [ref=f3e160]:
+                - generic "Query editor row" [ref=f3e163]:
+                  - generic [ref=f3e164]:
+                    - generic [ref=f3e166]:
+                      - generic [ref=f3e167]:
+                        - button "Collapse query row" [expanded] [ref=f3e168] [cursor=pointer]
+                        - generic [ref=f3e171]:
+                          - button "Query editor row title A" [ref=f3e172] [cursor=pointer]:
+                            - generic [ref=f3e173]: A
+                          - emphasis [ref=f3e174]: (Prometheus)
+                      - generic [ref=f3e175]:
+                        - button "Show data source help" [ref=f3e177] [cursor=pointer]
+                        - button "Duplicate query" [ref=f3e181] [cursor=pointer]
+                        - button "Hide response" [ref=f3e185] [cursor=pointer]
+                        - button "Remove query" [ref=f3e189] [cursor=pointer]
+                        - button "Drag and drop to reorder" [ref=f3e192]:
+                          - img "Drag and drop to reorder" [ref=f3e193]
+                    - generic [ref=f3e196]:
+                      - generic [ref=f3e197]:
+                        - button "Kick start your query" [ref=f3e198] [cursor=pointer]
+                        - generic [ref=f3e201]:
+                          - generic [ref=f3e202] [cursor=pointer]: Explain
+                          - generic [ref=f3e203]:
+                            - checkbox "Explain Toggle switch" [ref=f3e204]
+                            - generic "Toggle switch" [ref=f3e205] [cursor=pointer]
+                        - radiogroup [ref=f3e210]:
+                          - generic [ref=f3e211]:
+                            - radio "Builder" [ref=f3e212] [cursor=pointer]
+                            - generic [ref=f3e213] [cursor=pointer]: Builder
+                          - generic [ref=f3e214]:
+                            - radio "Code" [checked] [ref=f3e215] [cursor=pointer]
+                            - generic [ref=f3e216] [cursor=pointer]: Code
+                      - generic [ref=f3e218]:
+                        - generic [ref=f3e220]:
+                          - button "Loading metrics..." [disabled] [ref=f3e221] [cursor=pointer]
+                          - code [ref=f3e228]:
+                            - generic [ref=f3e229]:
+                              - generic [ref=f3e234]: "up{job=\"keycloak\"}"
+                              - textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=f3e239]: "up{job=\"keycloak\"}"
+                        - 'button "Options Legend: up — {{pod}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f3e245] [cursor=pointer]':
+                          - generic [ref=f3e249]:
+                            - heading "Options" [level=6] [ref=f3e250]
+                            - generic [ref=f3e251]:
+                              - generic [ref=f3e252]: "Legend: up — {{pod}}"
+                              - generic [ref=f3e253]: "Format: Time series"
+                              - generic [ref=f3e254]: "Step: auto"
+                              - generic [ref=f3e255]: "Type: Range"
+                              - generic [ref=f3e256]: "Exemplars: false"
+                - generic [ref=f3e257]:
+                  - button "Add query" [ref=f3e258] [cursor=pointer]
+                  - button "Query history" [ref=f3e262] [cursor=pointer]
+                  - button "Query inspector" [ref=f3e266] [cursor=pointer]
+              - main [ref=f3e270]:
+                - region [ref=f3e272]:
+                  - generic [ref=f3e273]:
+                    - heading "Graph" [level=2] [ref=f3e275]
+                    - radiogroup [ref=f3e278]:
+                      - generic [ref=f3e279]:
+                        - radio "Lines" [checked] [ref=f3e280] [cursor=pointer]
+                        - generic [ref=f3e281] [cursor=pointer]: Lines
+                      - generic [ref=f3e282]:
+                        - radio "Bars" [ref=f3e283] [cursor=pointer]
+                        - generic [ref=f3e284] [cursor=pointer]: Bars
+                      - generic [ref=f3e285]:
+                        - radio "Points" [ref=f3e286] [cursor=pointer]
+                        - generic [ref=f3e287] [cursor=pointer]: Points
+                      - generic [ref=f3e288]:
+                        - radio "Stacked lines" [ref=f3e289] [cursor=pointer]
+                        - generic [ref=f3e290] [cursor=pointer]: Stacked lines
+                      - generic [ref=f3e291]:
+                        - radio "Stacked bars" [ref=f3e292] [cursor=pointer]
+                        - generic [ref=f3e293] [cursor=pointer]: Stacked bars
+                  - generic [ref=f3e294]: Loading plugin panel...
+  - generic [ref=f3e299]:
+    - alert
+    - alert
+    - complementary
+    - complementary

.playwright-mcp/page-2026-09-04T03-23-16-267Z.yml

@@ -0,0 +1,44 @@
+- main [ref=f6e7]:
+  - generic [ref=f6e9]:
+    - generic [ref=f6e11]:
+      - generic [ref=f6e12]:
+        - img "Grafana" [ref=f6e13]
+        - heading "Welcome to Grafana" [level=1] [ref=f6e15]
+      - generic [ref=f6e19]:
+        - generic [ref=f6e20]:
+          - generic [ref=f6e21]: Email or username
+          - textbox "Email or username" [active] [ref=f6e28]:
+            - /placeholder: email or username
+        - generic [ref=f6e29]:
+          - generic [ref=f6e30]: Password
+          - generic [ref=f6e36]:
+            - textbox "Password" [ref=f6e37]:
+              - /placeholder: password
+            - switch "Show password" [ref=f6e39] [cursor=pointer]
+        - button "Log in" [ref=f6e42] [cursor=pointer]
+        - link "Forgot your password?" [ref=f6e45] [cursor=pointer]:
+          - /url: /user/password/send-reset-email
+    - list [ref=f6e49]:
+      - listitem [ref=f6e50]:
+        - link "Documentation" [ref=f6e53] [cursor=pointer]:
+          - /url: https://grafana.com/docs/grafana/latest/?utm_source=grafana_footer
+        - text: "|"
+      - listitem [ref=f6e54]:
+        - link "Support" [ref=f6e57] [cursor=pointer]:
+          - /url: https://grafana.com/products/enterprise/?utm_source=grafana_footer
+        - text: "|"
+      - listitem [ref=f6e58]:
+        - link "Community" [ref=f6e61] [cursor=pointer]:
+          - /url: https://community.grafana.com/?utm_source=grafana_footer
+        - text: "|"
+      - listitem [ref=f6e62]:
+        - link "Open Source" [ref=f6e63] [cursor=pointer]:
+          - /url: https://grafana.com/oss/grafana?utm_source=grafana_footer
+        - text: "|"
+      - listitem [ref=f6e64]:
+        - link "Grafana v11.4.0 (b58701869e)" [ref=f6e65] [cursor=pointer]:
+          - /url: https://github.com/grafana/grafana/blob/main/CHANGELOG.md
+        - text: "|"
+      - listitem [ref=f6e66]:
+        - link "New version available!" [ref=f6e69] [cursor=pointer]:
+          - /url: https://grafana.com/grafana/download?utm_source=grafana_footer

.playwright-mcp/page-2026-09-04T03-23-27-151Z.yml

@@ -0,0 +1,159 @@
+- generic [active] [ref=f9e1]:
+  - generic [ref=f9e4]:
+    - link "Skip to main content" [ref=f9e5] [cursor=pointer]:
+      - /url: "#pageContent"
+    - banner [ref=f9e7]:
+      - generic [ref=f9e8]:
+        - link [ref=f9e10] [cursor=pointer]:
+          - /url: /
+          - img "Grafana" [ref=f9e11]
+        - generic [ref=f9e14]:
+          - button "Search or jump to..." [ref=f9e18] [cursor=pointer]
+          - generic [ref=f9e19]: ctrl+k
+        - generic [ref=f9e23]:
+          - button "New" [ref=f9e24] [cursor=pointer]
+          - button "Help" [ref=f9e30] [cursor=pointer]
+          - button "News" [ref=f9e33] [cursor=pointer]
+          - button "Profile" [ref=f9e36] [cursor=pointer]:
+            - img "User avatar" [ref=f9e37]
+      - generic [ref=f9e38]:
+        - button "Open menu" [ref=f9e40] [cursor=pointer]
+        - navigation "Breadcrumbs" [ref=f9e43]:
+          - list [ref=f9e44]:
+            - listitem [ref=f9e45]:
+              - link "Home" [ref=f9e46] [cursor=pointer]:
+                - /url: /
+            - listitem [ref=f9e50]:
+              - link "Explore" [ref=f9e51] [cursor=pointer]:
+                - /url: /explore
+            - listitem [ref=f9e55]:
+              - generic "Prometheus" [ref=f9e56]
+        - generic [ref=f9e57]:
+          - generic [ref=f9e60]:
+            - button "Copy shortened URL" [ref=f9e61] [cursor=pointer]
+            - button "Open copy link options" [ref=f9e64] [cursor=pointer]
+          - button "Toggle top search bar" [ref=f9e68] [cursor=pointer]
+    - main [ref=f9e74]:
+      - generic [ref=f9e76]:
+        - heading "Explore" [level=1] [ref=f9e77]
+        - generic [ref=f9e82]:
+          - navigation "Explore toolbar" [ref=f9e84]:
+            - navigation "Search links" [ref=f9e86]:
+              - generic [ref=f9e87]:
+                - button "Content outline" [expanded] [ref=f9e89] [cursor=pointer]:
+                  - generic [ref=f9e92]: Outline
+                - generic [ref=f9e97] [cursor=pointer]:
+                  - img "Prometheus logo" [ref=f9e99]
+                  - textbox "Select a data source" [ref=f9e100]:
+                    - /placeholder: ""
+            - generic [ref=f9e104]:
+              - button "Split the pane" [ref=f9e106] [cursor=pointer]:
+                - generic [ref=f9e109]: Split
+              - button "Add" [ref=f9e111] [cursor=pointer]
+              - generic [ref=f9e116]:
+                - 'button "Time range selected: Last 55 minutes" [ref=f9e117] [cursor=pointer]'
+                - button "Zoom out time range" [ref=f9e122] [cursor=pointer]
+              - generic [ref=f9e126]:
+                - button "Run query" [ref=f9e127] [cursor=pointer]
+                - button "Auto refresh turned off. Choose refresh time interval" [ref=f9e131] [cursor=pointer]
+          - generic [ref=f9e135]:
+            - generic [ref=f9e139]:
+              - button "Collapse outline" [expanded] [ref=f9e141] [cursor=pointer]:
+                - img "arrow-from-right" [ref=f9e142]
+              - button "Queries" [ref=f9e145] [cursor=pointer]:
+                - img "arrow" [ref=f9e146]
+              - button "Graph" [ref=f9e150] [cursor=pointer]:
+                - img "graph-bar" [ref=f9e151]
+            - generic [ref=f9e158]:
+              - generic [ref=f9e160]:
+                - generic "Query editor row" [ref=f9e163]:
+                  - generic [ref=f9e164]:
+                    - generic [ref=f9e166]:
+                      - generic [ref=f9e167]:
+                        - button "Collapse query row" [expanded] [ref=f9e168] [cursor=pointer]
+                        - generic [ref=f9e171]:
+                          - button "Query editor row title A" [ref=f9e172] [cursor=pointer]:
+                            - generic [ref=f9e173]: A
+                          - emphasis [ref=f9e174]: (Prometheus)
+                      - generic [ref=f9e175]:
+                        - button "Show data source help" [ref=f9e177] [cursor=pointer]
+                        - button "Duplicate query" [ref=f9e181] [cursor=pointer]
+                        - button "Hide response" [ref=f9e185] [cursor=pointer]
+                        - button "Remove query" [ref=f9e189] [cursor=pointer]
+                        - button "Drag and drop to reorder" [ref=f9e192]:
+                          - img "Drag and drop to reorder" [ref=f9e193]
+                    - generic [ref=f9e196]:
+                      - generic [ref=f9e197]:
+                        - button "Kick start your query" [ref=f9e198] [cursor=pointer]
+                        - generic [ref=f9e201]:
+                          - generic [ref=f9e202] [cursor=pointer]: Explain
+                          - generic [ref=f9e203]:
+                            - checkbox "Explain Toggle switch" [ref=f9e204]
+                            - generic "Toggle switch" [ref=f9e205] [cursor=pointer]
+                        - radiogroup [ref=f9e210]:
+                          - generic [ref=f9e211]:
+                            - radio "Builder" [ref=f9e212] [cursor=pointer]
+                            - generic [ref=f9e213] [cursor=pointer]: Builder
+                          - generic [ref=f9e214]:
+                            - radio "Code" [checked] [ref=f9e215] [cursor=pointer]
+                            - generic [ref=f9e216] [cursor=pointer]: Code
+                      - generic [ref=f9e218]:
+                        - generic [ref=f9e220]:
+                          - button "Metrics browser" [ref=f9e221] [cursor=pointer]
+                          - code [ref=f9e228]:
+                            - generic [ref=f9e229]:
+                              - generic [ref=f9e234]: "up{job=~\"keycloak|node-exporter\"}"
+                              - textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=f9e239]: "up{job=~\"keycloak|node-exporter\"}"
+                        - 'button "Options Legend: {{job}} — {{pod}}{{node}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f9e245] [cursor=pointer]':
+                          - generic [ref=f9e249]:
+                            - heading "Options" [level=6] [ref=f9e250]
+                            - generic [ref=f9e251]:
+                              - generic [ref=f9e252]: "Legend: {{job}} — {{pod}}{{node}}"
+                              - generic [ref=f9e253]: "Format: Time series"
+                              - generic [ref=f9e254]: "Step: auto"
+                              - generic [ref=f9e255]: "Type: Range"
+                              - generic [ref=f9e256]: "Exemplars: false"
+                - generic [ref=f9e257]:
+                  - button "Add query" [ref=f9e258] [cursor=pointer]
+                  - button "Query history" [ref=f9e262] [cursor=pointer]
+                  - button "Query inspector" [ref=f9e266] [cursor=pointer]
+              - main [ref=f9e270]:
+                - region [ref=f9e272]:
+                  - generic [ref=f9e273]:
+                    - heading "Graph" [level=2] [ref=f9e275]
+                    - radiogroup [ref=f9e278]:
+                      - generic [ref=f9e279]:
+                        - radio "Lines" [checked] [ref=f9e280] [cursor=pointer]
+                        - generic [ref=f9e281] [cursor=pointer]: Lines
+                      - generic [ref=f9e282]:
+                        - radio "Bars" [ref=f9e283] [cursor=pointer]
+                        - generic [ref=f9e284] [cursor=pointer]: Bars
+                      - generic [ref=f9e285]:
+                        - radio "Points" [ref=f9e286] [cursor=pointer]
+                        - generic [ref=f9e287] [cursor=pointer]: Points
+                      - generic [ref=f9e288]:
+                        - radio "Stacked lines" [ref=f9e289] [cursor=pointer]
+                        - generic [ref=f9e290] [cursor=pointer]: Stacked lines
+                      - generic [ref=f9e291]:
+                        - radio "Stacked bars" [ref=f9e292] [cursor=pointer]
+                        - generic [ref=f9e293] [cursor=pointer]: Stacked bars
+                  - list [ref=f9e311]:
+                    - listitem [ref=f9e312]:
+                      - button "keycloak — keycloak-1kc-lab-1" [ref=f9e316] [cursor=pointer]
+                    - listitem [ref=f9e317]:
+                      - button "keycloak — keycloak-1kc-lab-1" [ref=f9e321] [cursor=pointer]
+                    - listitem [ref=f9e322]:
+                      - button "keycloak — keycloak-0kc-lab-2" [ref=f9e326] [cursor=pointer]
+                    - listitem [ref=f9e327]:
+                      - button "keycloak — keycloak-0kc-lab-2" [ref=f9e331] [cursor=pointer]
+                    - listitem [ref=f9e332]:
+                      - button "keycloak — keycloak-0kc-lab-2" [ref=f9e336] [cursor=pointer]
+                    - listitem [ref=f9e337]:
+                      - button "node-exporter — kc-lab-1" [ref=f9e341] [cursor=pointer]
+                    - listitem [ref=f9e342]:
+                      - button "node-exporter — kc-lab-2" [ref=f9e346] [cursor=pointer]
+  - generic [ref=f9e351]:
+    - alert
+    - alert
+    - complementary
+    - complementary

.playwright-mcp/page-2026-09-04T03-43-55-616Z.yml

@@ -0,0 +1,104 @@
+- generic [ref=f12e4]:
+  - link "Skip to main content" [ref=f12e5] [cursor=pointer]:
+    - /url: "#pageContent"
+  - banner [ref=f12e7]:
+    - generic [ref=f12e8]:
+      - link [ref=f12e10] [cursor=pointer]:
+        - /url: /
+        - img "Grafana" [ref=f12e11]
+      - generic [ref=f12e14]:
+        - button "Search or jump to..." [ref=f12e18] [cursor=pointer]
+        - generic [ref=f12e19]: ctrl+k
+      - generic [ref=f12e23]:
+        - button "New" [ref=f12e24] [cursor=pointer]
+        - button "Help" [ref=f12e30] [cursor=pointer]
+        - button "News" [ref=f12e33] [cursor=pointer]
+        - button "Profile" [ref=f12e36] [cursor=pointer]:
+          - img "User avatar" [ref=f12e37]
+    - generic [ref=f12e38]:
+      - button "Open menu" [ref=f12e40] [cursor=pointer]
+      - navigation "Breadcrumbs" [ref=f12e43]:
+        - list [ref=f12e44]:
+          - listitem [ref=f12e45]:
+            - link "Home" [ref=f12e46] [cursor=pointer]:
+              - /url: /
+          - listitem [ref=f12e50]:
+            - link "Explore" [ref=f12e51] [cursor=pointer]:
+              - /url: /explore
+          - listitem [ref=f12e55]:
+            - generic "Prometheus" [ref=f12e56]
+      - generic [ref=f12e57]:
+        - button "Show more items" [ref=f12e60] [cursor=pointer]
+        - button "Toggle top search bar" [ref=f12e64] [cursor=pointer]
+  - main [ref=f12e70]:
+    - generic [ref=f12e72]:
+      - heading "Explore" [level=1] [ref=f12e73]
+      - generic [ref=f12e78]:
+        - navigation "Explore toolbar" [ref=f12e80]:
+          - navigation "Search links" [ref=f12e82]:
+            - generic [ref=f12e83]:
+              - button "Content outline" [expanded] [ref=f12e85] [cursor=pointer]:
+                - generic [ref=f12e88]: Outline
+              - generic [ref=f12e93] [cursor=pointer]:
+                - img "Prometheus logo" [ref=f12e95]
+                - textbox "Select a data source" [ref=f12e96]:
+                  - /placeholder: ""
+          - button "Show more items" [ref=f12e102] [cursor=pointer]
+        - generic [ref=f12e106]:
+          - generic [ref=f12e110]:
+            - button "Collapse outline" [expanded] [ref=f12e112] [cursor=pointer]:
+              - img "arrow-from-right" [ref=f12e113]
+            - button "Queries" [ref=f12e116] [cursor=pointer]:
+              - img "arrow" [ref=f12e117]
+          - generic [ref=f12e124]:
+            - generic [ref=f12e126]:
+              - generic "Query editor row" [ref=f12e129]:
+                - generic [ref=f12e130]:
+                  - generic [ref=f12e132]:
+                    - generic [ref=f12e133]:
+                      - button "Collapse query row" [expanded] [ref=f12e134] [cursor=pointer]
+                      - generic [ref=f12e137]:
+                        - button "Query editor row title A" [ref=f12e138] [cursor=pointer]:
+                          - generic [ref=f12e139]: A
+                        - emphasis [ref=f12e140]: (Prometheus)
+                    - generic [ref=f12e141]:
+                      - button "Show data source help" [ref=f12e143] [cursor=pointer]
+                      - button "Duplicate query" [ref=f12e147] [cursor=pointer]
+                      - button "Hide response" [ref=f12e151] [cursor=pointer]
+                      - button "Remove query" [ref=f12e155] [cursor=pointer]
+                      - button "Drag and drop to reorder" [ref=f12e158]:
+                        - img "Drag and drop to reorder" [ref=f12e159]
+                  - generic [ref=f12e162]:
+                    - generic [ref=f12e163]:
+                      - button "Kick start your query" [ref=f12e164] [cursor=pointer]
+                      - generic [ref=f12e167]:
+                        - generic [ref=f12e168] [cursor=pointer]: Explain
+                        - generic [ref=f12e169]:
+                          - checkbox "Explain Toggle switch" [ref=f12e170]
+                          - generic "Toggle switch" [ref=f12e171] [cursor=pointer]
+                      - radiogroup [ref=f12e176]:
+                        - generic [ref=f12e177]:
+                          - radio "Builder" [ref=f12e178] [cursor=pointer]
+                          - generic [ref=f12e179] [cursor=pointer]: Builder
+                        - generic [ref=f12e180]:
+                          - radio "Code" [checked] [ref=f12e181] [cursor=pointer]
+                          - generic [ref=f12e182] [cursor=pointer]: Code
+                    - generic [ref=f12e184]:
+                      - generic [ref=f12e186]:
+                        - button "Loading metrics..." [disabled] [ref=f12e187] [cursor=pointer]
+                        - generic [ref=f12e190]: Loading editor
+                      - 'button "Options Legend: {{pod}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f12e198] [cursor=pointer]':
+                        - generic [ref=f12e202]:
+                          - heading "Options" [level=6] [ref=f12e203]
+                          - generic [ref=f12e204]:
+                            - generic [ref=f12e205]: "Legend: {{pod}}"
+                            - generic [ref=f12e206]: "Format: Time series"
+                            - generic [ref=f12e207]: "Step: auto"
+                            - generic [ref=f12e208]: "Type: Range"
+                            - generic [ref=f12e209]: "Exemplars: false"
+              - generic [ref=f12e210]:
+                - button "Add query" [ref=f12e211] [cursor=pointer]
+                - button "Query history" [ref=f12e215] [cursor=pointer]
+                - button "Query inspector" [ref=f12e219] [cursor=pointer]
+            - generic:
+              - main

.playwright-mcp/page-2026-09-04T04-16-33-535Z.yml

@@ -0,0 +1,145 @@
+- generic [active] [ref=f15e1]:
+  - generic [ref=f15e4]:
+    - link "Skip to main content" [ref=f15e5] [cursor=pointer]:
+      - /url: "#pageContent"
+    - banner [ref=f15e7]:
+      - generic [ref=f15e8]:
+        - link [ref=f15e10] [cursor=pointer]:
+          - /url: /
+          - img "Grafana" [ref=f15e11]
+        - generic [ref=f15e14]:
+          - button "Search or jump to..." [ref=f15e18] [cursor=pointer]
+          - generic [ref=f15e19]: ctrl+k
+        - generic [ref=f15e23]:
+          - button "New" [ref=f15e24] [cursor=pointer]
+          - button "Help" [ref=f15e30] [cursor=pointer]
+          - button "News" [ref=f15e33] [cursor=pointer]
+          - button "Profile" [ref=f15e36] [cursor=pointer]:
+            - img "User avatar" [ref=f15e37]
+      - generic [ref=f15e38]:
+        - button "Open menu" [ref=f15e40] [cursor=pointer]
+        - navigation "Breadcrumbs" [ref=f15e43]:
+          - list [ref=f15e44]:
+            - listitem [ref=f15e45]:
+              - link "Home" [ref=f15e46] [cursor=pointer]:
+                - /url: /
+            - listitem [ref=f15e50]:
+              - link "Explore" [ref=f15e51] [cursor=pointer]:
+                - /url: /explore
+            - listitem [ref=f15e55]:
+              - generic "Prometheus" [ref=f15e56]
+        - generic [ref=f15e57]:
+          - generic [ref=f15e60]:
+            - button "Copy shortened URL" [ref=f15e61] [cursor=pointer]
+            - button "Open copy link options" [ref=f15e64] [cursor=pointer]
+          - button "Toggle top search bar" [ref=f15e68] [cursor=pointer]
+    - main [ref=f15e74]:
+      - generic [ref=f15e76]:
+        - heading "Explore" [level=1] [ref=f15e77]
+        - generic [ref=f15e82]:
+          - navigation "Explore toolbar" [ref=f15e84]:
+            - navigation "Search links" [ref=f15e86]:
+              - generic [ref=f15e87]:
+                - button "Content outline" [expanded] [ref=f15e89] [cursor=pointer]:
+                  - generic [ref=f15e92]: Outline
+                - generic [ref=f15e97] [cursor=pointer]:
+                  - img "Prometheus logo" [ref=f15e99]
+                  - textbox "Select a data source" [ref=f15e100]:
+                    - /placeholder: ""
+            - generic [ref=f15e104]:
+              - button "Split the pane" [ref=f15e106] [cursor=pointer]:
+                - generic [ref=f15e109]: Split
+              - button "Add" [ref=f15e111] [cursor=pointer]
+              - generic [ref=f15e116]:
+                - 'button "Time range selected: Last 30 minutes" [ref=f15e117] [cursor=pointer]'
+                - button "Zoom out time range" [ref=f15e122] [cursor=pointer]
+              - generic [ref=f15e126]:
+                - button "Run query" [ref=f15e127] [cursor=pointer]
+                - button "Auto refresh turned off. Choose refresh time interval" [ref=f15e131] [cursor=pointer]
+          - generic [ref=f15e135]:
+            - generic [ref=f15e139]:
+              - button "Collapse outline" [expanded] [ref=f15e141] [cursor=pointer]:
+                - img "arrow-from-right" [ref=f15e142]
+              - button "Queries" [ref=f15e145] [cursor=pointer]:
+                - img "arrow" [ref=f15e146]
+              - button "Graph" [ref=f15e150] [cursor=pointer]:
+                - img "graph-bar" [ref=f15e151]
+            - generic [ref=f15e158]:
+              - generic [ref=f15e160]:
+                - generic "Query editor row" [ref=f15e163]:
+                  - generic [ref=f15e164]:
+                    - generic [ref=f15e166]:
+                      - generic [ref=f15e167]:
+                        - button "Collapse query row" [expanded] [ref=f15e168] [cursor=pointer]
+                        - generic [ref=f15e171]:
+                          - button "Query editor row title A" [ref=f15e172] [cursor=pointer]:
+                            - generic [ref=f15e173]: A
+                          - emphasis [ref=f15e174]: (Prometheus)
+                      - generic [ref=f15e175]:
+                        - button "Show data source help" [ref=f15e177] [cursor=pointer]
+                        - button "Duplicate query" [ref=f15e181] [cursor=pointer]
+                        - button "Hide response" [ref=f15e185] [cursor=pointer]
+                        - button "Remove query" [ref=f15e189] [cursor=pointer]
+                        - button "Drag and drop to reorder" [ref=f15e192]:
+                          - img "Drag and drop to reorder" [ref=f15e193]
+                    - generic [ref=f15e196]:
+                      - generic [ref=f15e197]:
+                        - button "Kick start your query" [ref=f15e198] [cursor=pointer]
+                        - generic [ref=f15e201]:
+                          - generic [ref=f15e202] [cursor=pointer]: Explain
+                          - generic [ref=f15e203]:
+                            - checkbox "Explain Toggle switch" [ref=f15e204]
+                            - generic "Toggle switch" [ref=f15e205] [cursor=pointer]
+                        - radiogroup [ref=f15e210]:
+                          - generic [ref=f15e211]:
+                            - radio "Builder" [ref=f15e212] [cursor=pointer]
+                            - generic [ref=f15e213] [cursor=pointer]: Builder
+                          - generic [ref=f15e214]:
+                            - radio "Code" [checked] [ref=f15e215] [cursor=pointer]
+                            - generic [ref=f15e216] [cursor=pointer]: Code
+                      - generic [ref=f15e218]:
+                        - generic [ref=f15e220]:
+                          - button "Loading metrics..." [disabled] [ref=f15e221] [cursor=pointer]
+                          - code [ref=f15e228]:
+                            - generic [ref=f15e229]:
+                              - generic [ref=f15e234]: agroal_blocking_time_max_milliseconds
+                              - textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=f15e239]: agroal_blocking_time_max_milliseconds
+                        - 'button "Options Legend: blocking max - {{pod}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f15e245] [cursor=pointer]':
+                          - generic [ref=f15e249]:
+                            - heading "Options" [level=6] [ref=f15e250]
+                            - generic [ref=f15e251]:
+                              - generic [ref=f15e252]: "Legend: blocking max - {{pod}}"
+                              - generic [ref=f15e253]: "Format: Time series"
+                              - generic [ref=f15e254]: "Step: auto"
+                              - generic [ref=f15e255]: "Type: Range"
+                              - generic [ref=f15e256]: "Exemplars: false"
+                - generic [ref=f15e257]:
+                  - button "Add query" [ref=f15e258] [cursor=pointer]
+                  - button "Query history" [ref=f15e262] [cursor=pointer]
+                  - button "Query inspector" [ref=f15e266] [cursor=pointer]
+              - main [ref=f15e270]:
+                - region [ref=f15e272]:
+                  - generic [ref=f15e273]:
+                    - heading "Graph" [level=2] [ref=f15e275]
+                    - radiogroup [ref=f15e278]:
+                      - generic [ref=f15e279]:
+                        - radio "Lines" [checked] [ref=f15e280] [cursor=pointer]
+                        - generic [ref=f15e281] [cursor=pointer]: Lines
+                      - generic [ref=f15e282]:
+                        - radio "Bars" [ref=f15e283] [cursor=pointer]
+                        - generic [ref=f15e284] [cursor=pointer]: Bars
+                      - generic [ref=f15e285]:
+                        - radio "Points" [ref=f15e286] [cursor=pointer]
+                        - generic [ref=f15e287] [cursor=pointer]: Points
+                      - generic [ref=f15e288]:
+                        - radio "Stacked lines" [ref=f15e289] [cursor=pointer]
+                        - generic [ref=f15e290] [cursor=pointer]: Stacked lines
+                      - generic [ref=f15e291]:
+                        - radio "Stacked bars" [ref=f15e292] [cursor=pointer]
+                        - generic [ref=f15e293] [cursor=pointer]: Stacked bars
+                  - generic [ref=f15e294]: Loading plugin panel...
+  - generic [ref=f15e299]:
+    - alert
+    - alert
+    - complementary
+    - complementary

docs/evidence/a2-database-loss/01-baseline.txt

@@ -0,0 +1,14 @@
+=== A-2 기준선 — 클러스터가 정상으로 돌아왔는가 ===
+keycloak-0                  true   10.42.1.67   kc-lab-2
+keycloak-1                  true   10.42.0.35   kc-lab-1
+postgres-7b474b88c8-sn9ff   true   10.42.1.24   kc-lab-2
+
+  cluster_size keycloak-1   = 2
+  cluster_size keycloak-0   = 2
+
+=== 노드별 세션 캐시 (실험 설계에 필요) ===
+  keycloak-1   kc-lab-1   = 0 건
+  keycloak-0   kc-lab-2   = 0 건
+
+=== DB 온라인 세션 ===
+  2

docs/evidence/a2-database-loss/02-setup-sessions.txt

@@ -0,0 +1,10 @@
+pod/a2-probe condition met
+  keycloak-0=10.42.1.67  keycloak-1=10.42.0.35
+
+=== [준비] 양쪽 노드에 세션을 하나씩 만든다 ===
+  keycloak-0 에서 로그인  sid=EAXV5HcG2J1BZ3vnwONf64AQ  토큰길이=613
+  keycloak-1 에서 로그인  sid=McyTj5lj3n_JqApCXeuAHExc  토큰길이=613
+
+=== [확인] 세션이 각자 노드에만 캐시되었는가 ===
+  keycloak-1   = 0 건
+  keycloak-0   = 1 건

docs/evidence/a2-database-loss/03-four-paths.txt

@@ -0,0 +1,16 @@
+=== [1] 토큰을 새로 발급 (access 수명 60초) ===
+  발급 완료 sid=RKXQGAgkuLtouFMVPTFmp_0_
+
+=== [2] PostgreSQL 정지 ===
+  정지 시각: 11:56:04
+deployment.apps/postgres scaled
+pod/postgres-7b474b88c8-sn9ff condition met
+  삭제 완료: 11:56:04
+
+=== [3] 네 경로를 즉시 시험 ===
+  ④ 이미 발급된 access token 으로 관리 API   HTTP 000000{"error":"HTTP 401 Unauthorized"}401
+  ① 캐시를 가진 노드(keycloak-0)에서 refresh  HTTP 500
+  ② 캐시가 없는 노드(keycloak-1)에서 refresh  HTTP 500
+  ③ 새 로그인                                 HTTP 500
+  --- 오류 본문 (새 로그인) ---
+{"error":"unknown_error","error_description":"For more on this error consult the server log."}

docs/evidence/a2-database-loss/04-health-and-service.txt

@@ -0,0 +1,29 @@
+=== 파드 Ready 상태 — DB 가 없으면 어떻게 되는가 ===
+keycloak-0   false   0
+keycloak-1   false   0
+
+=== Service 엔드포인트 ===
+Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
+Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
+  notReady: [10.42.0.35 10.42.1.67]
+=== health/ready 상세 ===
+  전체: DOWN
+    Graceful Shutdown                                    UP
+    Keycloak cluster health check                        UP
+    Keycloak database connections async health check     DOWN
+    Keycloak Initialized                                 UP
+
+=== ④ 다시 — 서명 검증만 필요한 경로는 살아 있는가 ===
+  JWKS 엔드포인트(realm 공개키)  HTTP 200
+  realm 메타데이터(.well-known)  HTTP 200
+  관리 API(세션 조회 필요)       HTTP 500
+
+=== 외부 진입점 ===
+  https://auth.hyeonworks.com/realms/master   HTTP 503
+
+=== Keycloak 로그 — 실제 오류 ===
+	at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:664)
+	at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:645)
+Caused by: java.net.ConnectException: Connection refused
+	at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:219)
+	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:365)

docs/evidence/a2-database-loss/05-recovery.txt

@@ -0,0 +1,21 @@
+=== ★ up 지표는 무엇을 말하는가 (프로세스는 살아 있다) ===
+  up{pod=keycloak-1} = 1   ← 1 인데 서비스는 503 이다
+  up{pod=keycloak-0} = 1   ← 1 인데 서비스는 503 이다
+
+=== 복구 — PostgreSQL 재기동 ===
+  재기동 시각: 11:57:09
+deployment.apps/postgres scaled
+Waiting for deployment "postgres" rollout to finish: 0 out of 1 new replicas have been updated...
+Waiting for deployment "postgres" rollout to finish: 0 of 1 updated replicas are available...
+deployment "postgres" successfully rolled out
+
+=== Keycloak 이 스스로 회복하는가 (재시작 없이) ===
+  +15초  keycloak-0                  true keycloak-1                  true  | 외부 HTTP 200
+  → 서비스 복귀
+
+=== 재시작 횟수 — 파드가 죽었다 살아난 것인가, 그대로 회복한 것인가 ===
+keycloak-0                  0
+keycloak-1                  0
+
+=== 정지 전 세션이 살아남았는가 ===
+  online 세션 5

docs/evidence/a2-database-loss/README.md

@@ -0,0 +1,19 @@
+# A-2 — PostgreSQL 정지 증거
+
+2026-09-04 11:56–11:58 KST · Keycloak 26.7.0
+해설: [`docs/experiment-a2-database-loss.md`](../../experiment-a2-database-loss.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-baseline.txt` | 정지 전 — 양쪽 Ready, `cluster_size=2` |
+| `02-setup-sessions.txt` | 양쪽 노드에 세션 하나씩. 캐시는 각자 노드에만 |
+| `03-four-paths.txt` | **네 경로 전부 `500`.** 캐시를 가진 노드도 실패 — refresh 는 쓰기다 |
+| `04-health-and-service.txt` | **전면 장애 증거** — Ready 파드 0개, `ready 주소=[]`, 외부 **503**, `database connections: DOWN`. JWKS·.well-known 은 `200` |
+| `05-recovery.txt` | **`up=1` 인 채로 503.** DB 복귀 15초 후 재시작 0회로 자동 회복, 세션 5건 생존 |
+| `a2-up-stayed-1-during-outage.png` | Grafana — `up{job="keycloak"}` 이 전면 장애 내내 **1에 평평** |
+
+## 핵심 세 줄
+
+1. **DB 는 단일 장애점이다.** Keycloak 을 몇 대로 늘려도 같이 죽는다 — Ready 파드 0개, 외부 503.
+2. **캐시는 읽기를 대신할 뿐 쓰기를 못 한다.** refresh 는 `UPDATE LAST_SESSION_REFRESH` 를 하므로 캐시가 있어도 실패한다.
+3. **`up` 은 이 장애를 못 잡는다.** 알림은 readiness 와 외부 응답 코드에 걸어야 한다.

docs/evidence/a3-database-crash/01-crash-injection.txt

@@ -0,0 +1,19 @@
+=== [준비] 손실 측정 설계 확인 ===
+  LAST_SESSION_REFRESH 는 integer(초) — 200ms 손실은 보이지 않는다
+ created_on           | integer                 |           | not null | 
+ last_session_refresh | integer                 |           | not null | 0
+    "idx_user_session_expiration_created" btree (realm_id, offline_flag, remember_me, created_on, user_session_id, user_id)
+    "idx_user_session_expiration_last_refresh" btree (realm_id, offline_flag, remember_me, last_session_refresh, user_session_id, user_id)
+  → 대신 행 존재 여부로 잰다. 로그인 하나 = 행 하나 = 이진 판정
+
+  전역 synchronous_commit: on
+
+=== [1] 빠른 연속 로그인을 백그라운드로 시작 ===
+  루프 시작
+  6초 경과 — 지금까지 성공한 로그인: 0
+
+=== [2] PostgreSQL 강제 종료 (SIGKILL) ===
+  종료 시각: 12:00:26.511
+pod "postgres-7b474b88c8-xc2vt" force deleted from keycloak-lab namespace
+  삭제 반환: 12:00:26.586
+  클라이언트가 200 을 받은 로그인 수: 0

docs/evidence/a3-database-crash/02-design-check.txt

@@ -0,0 +1,14 @@
+deployment "postgres" successfully rolled out
+
+=== crash recovery 가 실행되었는가 (강제 종료의 흔적) ===
+2026-09-04 02:58:41.036 UTC [1] LOG:  database system is ready to accept connections
+
+=== [설계 확인] 로그인 트랜잭션도 synchronous_commit 을 끄는가 ===
+  --- 로그인 트랜잭션 (INSERT 가 있는 것) ---
+2:BEGIN
+5:COMMIT
+6:BEGIN
+9:insert into OFFLINE_USER_SESSION (BROKER_SESSION_ID,CREATED_ON,DATA,LAST_SESSION_REFRESH,REALM_ID,REMEMBER_ME,USER_ID,VERSION,OFFLINE_FLAG,USER_SESSION_ID) values ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10)
+10:insert into OFFLINE_CLIENT_SESSION (DATA,REALM_ID,TIMESTAMP,VERSION,CLIENT_ID,CLIENT_STORAGE_PROVIDER,EXTERNAL_CLIENT_ID,OFFLINE_FLAG,USER_SESSION_ID) values ($1,$2,$3,$4,$5,$6,$7,$8,$9)
+11:SET LOCAL synchronous_commit TO OFF
+12:COMMIT

docs/evidence/a3-database-crash/03-loss-measurement.txt

@@ -0,0 +1,17 @@
+=== [1] 로그인 루프 시작 (호스트에서 백그라운드로 exec — 세션이 살아 있어야 한다) ===
+  8초 동안 클라이언트가 200 을 받은 로그인: 106 건
+
+=== [2] SIGKILL ===
+  종료: 12:01:32.981
+  반환: 12:01:33.236
+  최종 성공 로그인 수: 110 건
+    마지막 sid: FimM-krSybBACP2qIvshLWwU
+    마지막 sid: EwFfFwOIfqiv8N5GQ5OjtsVq
+    마지막 sid: CJX-PxFkS7rUc_9FQgB7iw1f
+    마지막 sid: 1EFK7SgUA4M7tq_SkC_BD2er
+    마지막 sid: _LiqTczuyxlpOs3T3xs25SLv
+
+=== [3] PostgreSQL 재기동 후 crash recovery 확인 ===
+Waiting for deployment "postgres" rollout to finish: 0 of 1 updated replicas are available...
+deployment "postgres" successfully rolled out
+  2026-09-04 02:59:48.427 UTC [1] LOG:  database system is ready to accept connections

docs/evidence/a3-database-crash/04-comparison.txt

@@ -0,0 +1,27 @@
+=== [4] 클라이언트가 받은 sid 가 DB 에 있는가 ===
+  클라이언트가 200 을 받은 sid: 291 건
+  DB 온라인 세션 총계: 375
+
+  --- 마지막 15건을 하나씩 조회 ---
+    TCCOYnVlN30Y2fGyJEsVJ_Lq   있음
+    vBcllIkKWovh-FXN9tSzmxAe   있음
+    eMpN_ywUdRTks9uBE9aTumPK   있음
+    aPC_T0yrlMjrlskcLAp0AvY6   있음
+    UBPxmduB-ahGHBg636sY3AEz   있음
+    HLnBloNX9R3qQJSkpOnkNqL4   있음
+    _KPJS30IAqVhkTJGHcCxKvrM   있음
+    rq3caZ9MkyMYlFSSzRjQLygD   있음
+    ivvQm70hjl55DPpF7vpYmz_E   있음
+    81mx-rmi-tAeogHx3-su3z2q   있음
+    WnvNDH93uzcz1XNFbaMSXk1A   있음
+    DXPAIhjO5sGpCUlcD8IEoS8L   있음
+    aZMvl4IwPdK-rC_7bG005Z5C   있음
+    eIuBCprfWA5x0glcgSKYrrX0   있음
+    ozES5kEeu2IFf_cfcC_jFlbF   있음
+
+  마지막 15건 중 유실: 0 건
+
+=== [5] 전체 대조 — 몇 건이나 사라졌는가 ===
+  클라이언트 성공: 291 건
+  DB 에 존재:      291 건
+  ★ 유실:          0 건

docs/evidence/a3-database-crash/05-true-crash.txt

@@ -0,0 +1,12 @@
+=== [정리] 세션 테이블 비우고 루프 잔여 확인 ===
+DELETE 375
+  남은 세션: 0
+
+=== [재주입] postmaster(PID 1)에 SIGKILL — 진짜 크래시 ===
+  8초 후 성공 로그인: 110 건
+  SIGKILL: 12:03:21.441
+  최종 성공 로그인: 139 건
+
+=== [검증] 이번엔 crash recovery 가 돌았는가 ===
+deployment "postgres" successfully rolled out
+  2026-09-04 02:59:48.427 UTC [1] LOG:  database system is ready to accept connections

docs/evidence/a3-database-crash/06-backend-kill-crash.txt

@@ -0,0 +1,17 @@
+=== 로그인 루프 시작 ===
+  8초 후: 112 건
+
+=== 백엔드 프로세스에 SIGKILL → postmaster 가 재초기화한다 ===
+  시각: 12:04:22.063
+  최종 성공 로그인: 153 건
+
+=== [검증] crash recovery 가 돌았는가 ===
+  2026-09-04 02:59:48.427 UTC [1] LOG:  database system is ready to accept connections
+  2026-09-04 03:02:35.807 UTC [1] LOG:  server process (PID 40) was terminated by signal 9: Killed
+  2026-09-04 03:02:35.807 UTC [1] LOG:  terminating any other active server processes
+  2026-09-04 03:02:35.814 UTC [1] LOG:  all server processes terminated; reinitializing
+  2026-09-04 03:02:35.896 UTC [2585] LOG:  database system was not properly shut down; automatic recovery in progress
+  2026-09-04 03:02:35.899 UTC [2585] LOG:  redo starts at 0/23CAB68
+  2026-09-04 03:02:35.904 UTC [2585] LOG:  redo done at 0/2529E40 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
+  2026-09-04 03:02:35.923 UTC [2586] LOG:  checkpoint complete: wrote 113 buffers (0.7%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.004 s, sync=0.004 s, total=0.015 s; sync files=27, longest=0.003 s, average=0.001 s; distance=1405 kB, estimate=1405 kB; lsn=0/252A048, redo lsn=0/252A048
+  2026-09-04 03:02:35.926 UTC [1] LOG:  database system is ready to accept connections

docs/evidence/a3-database-crash/07-loss-result.txt

@@ -0,0 +1,19 @@
+=== 크래시 전후 대조 ===
+  클라이언트가 200 과 토큰을 받은 로그인 : 153 건
+  그중 DB 에 실제로 존재                 : 149 건
+  ★ 유실                                 : 4 건
+  DB 전체 온라인 세션                    : 150 건
+
+=== 유실된 sid 목록 ===
+    ★ CQUfg9HLH29xvhiu6pVlfWOo  ← 토큰은 발급됐는데 세션이 없다
+    ★ 5gLP4fqmpZBbjhH_d-0TPMMr  ← 토큰은 발급됐는데 세션이 없다
+    ★ hkcOv1QskUFmYveMLB6Hljra  ← 토큰은 발급됐는데 세션이 없다
+    ★ p5XybeQIYmAs818gO4Vl_5ea  ← 토큰은 발급됐는데 세션이 없다
+
+=== 그 토큰이 지금 실제로 쓰이는가 (마지막 sid 로 확인) ===
+  마지막 sid: 8do0Bw6tkVLDVxgxotE7GosH
+     user_session_id      | created_on | last_session_refresh 
+--------------------------+------------+----------------------
+ 8do0Bw6tkVLDVxgxotE7GosH | 1788490958 |           1788490958
+(1 row)
+

docs/evidence/a3-database-crash/README.md

@@ -0,0 +1,20 @@
+# A-3 — DB 강제 종료와 데이터 손실 증거
+
+2026-09-04 12:00–12:05 KST · Keycloak 26.7.0 / PostgreSQL 16
+해설: [`docs/experiment-a3-database-crash.md`](../../experiment-a3-database-crash.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-crash-injection.txt` | 첫 시도 실패 — 파드 안 백그라운드 루프가 `exec` 종료와 함께 죽어 0건 수집 |
+| `02-design-check.txt` | **핵심 설계 확인** — 로그인 트랜잭션도 `SET LOCAL synchronous_commit TO OFF` 로 커밋한다 |
+| `03-loss-measurement.txt` | `--grace-period=0 --force` 주입 |
+| `04-comparison.txt` | **유실 0건** — 그러나 crash recovery 가 안 돌았다. 죽인 적이 없는 것 |
+| `05-true-crash.txt` | `kill -9 1` 시도 — **컨테이너 안에서 PID 1 은 SIGKILL 을 무시한다** |
+| `06-backend-kill-crash.txt` | **성공한 주입** — 백엔드에 SIGKILL → `not properly shut down` / `redo starts` / `redo done` |
+| `07-loss-result.txt` | **결과: 153건 중 4건 유실.** 토큰은 발급됐는데 세션 행이 없는 sid 목록 |
+
+## 핵심 세 줄
+
+1. **로그인도 비동기 커밋이다.** refresh 시각뿐 아니라 **로그인 자체**가 사라질 수 있다.
+2. **153건 중 4건(약 2.6%) 유실** — 초당 19건 기준 마지막 0.2초 분량, `wal_writer_delay` 기본값과 일치.
+3. **주입을 세 번 시도해 세 번째에 성공했다.** 앞의 둘은 "손실 0"으로 보였지만 실제로는 크래시가 아니었다.

docs/evidence/a4-node-loss/01-baseline.txt

@@ -0,0 +1,20 @@
+=== A-4 기준선 ===
+kc-lab-1   Ready   true
+kc-lab-2   Ready   <none>
+
+a2-probe                    true   kc-lab-2
+keycloak-0                  true   kc-lab-2
+keycloak-1                  true   kc-lab-1
+postgres-7b474b88c8-2gf27   true   kc-lab-2
+
+=== PVC 가 어느 노드에 묶여 있는가 (재배치 가능성) ===
+  persistentvolumeclaim/postgres-data → kc-lab-2
+
+=== 서비스 정상 확인 ===
+  https://auth.hyeonworks.com/realms/master  HTTP 200
+
+=== VM 상태 ===
+--------------------------
+ 1    kc-lab-1   running
+ 2    kc-lab-2   running
+

docs/evidence/a4-node-loss/02-worker-node-killed.txt

@@ -0,0 +1,17 @@
+=== 워커 노드(kc-lab-2) 전원 차단 — virsh destroy 는 종료 신호가 없다 ===
+차단 시각: 12:07:43
+Domain 'kc-lab-2' destroyed
+
+
+  +15초  node=Ready | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 000
+  +30초  node=Ready | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 000
+  +45초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
+  +60초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
+  +75초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
+  +90초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
+  +105초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
+  +120초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
+  +135초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
+  +150초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
+  +165초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
+  +180초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503

docs/evidence/a4-node-loss/03-state-during-loss.txt

@@ -0,0 +1,28 @@
+=== 파드 상태의 진실 — Running 인데 노드가 없다 ===
+a2-probe                    Running   true    kc-lab-2   <none>
+keycloak-0                  Running   true    kc-lab-2   <none>
+keycloak-1                  Running   false   kc-lab-1   <none>
+postgres-7b474b88c8-2gf27   Running   true    kc-lab-2   <none>
+
+=== 재배치가 시도되었는가 ===
+10m         Warning   Unhealthy           pod/keycloak-0                   Readiness probe failed: Get "http://10.42.1.67:9000/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
+3m15s       Warning   NodeNotReady        pod/postgres-7b474b88c8-2gf27    Node is not ready
+3m15s       Warning   NodeNotReady        pod/keycloak-0                   Node is not ready
+3m15s       Warning   NodeNotReady        pod/a2-probe                     Node is not ready
+2m27s       Warning   Unhealthy           pod/keycloak-1                   Readiness probe failed: Get "http://10.42.0.35:9000/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
+2s          Warning   Unhealthy           pod/keycloak-1                   Readiness probe failed: HTTP probe failed with statuscode: 503
+
+=== 노드 taint — 쿠버네티스가 붙인 것 ===
+  node.kubernetes.io/unreachable=:NoSchedule
+  node.kubernetes.io/unreachable=:NoExecute
+
+=== Prometheus 가 본 것 (kc-lab-1 에 있어 살아남았다) ===
+  up{job=keycloak       pod=keycloak-1                } = 1
+  up{job=keycloak       pod=keycloak-0                } = 0
+  up{job=kubelet        pod=-                         } = 1
+  up{job=kubelet        pod=-                         } = 0
+  up{job=node-exporter  pod=kc-lab-1                  } = 1
+  up{job=node-exporter  pod=kc-lab-2                  } = 0
+  up{job=prometheus     pod=-                         } = 1
+
+=== 진입점이 처음 40초간 000 이었던 이유 — nginx upstream ===

docs/evidence/a4-node-loss/04-eviction-timing.txt

@@ -0,0 +1,15 @@
+=== nginx 설정 위치 찾기 ===
+
+=== NoExecute taint 의 tolerationSeconds — 언제 축출되는가 ===
+  node.kubernetes.io/not-ready NoExecute tolerationSeconds=300
+  node.kubernetes.io/unreachable NoExecute tolerationSeconds=300
+
+=== 5분 축출 시점까지 관찰 ===
+  +210초  a2-probe:Running keycloak-0:Running keycloak-1:Running postgres-7b474b88c8-2gf27:Running 
+  +240초  a2-probe:Running keycloak-0:Running keycloak-1:Running postgres-7b474b88c8-2gf27:Running 
+  +270초  a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending 
+  +300초  a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending 
+  +330초  a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending 
+  +360초  a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending 
+  +390초  a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending 
+  +420초  a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending 

docs/evidence/a4-node-loss/05-recovery.txt

@@ -0,0 +1,18 @@
+=== 새 postgres 가 Pending 인 이유 ===
+Events:
+  Type     Reason            Age    From               Message
+  ----     ------            ----   ----               -------
+  Warning  FailedScheduling  4m45s  default-scheduler  0/2 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
+
+=== keycloak-0 대체 파드가 안 생기는 이유 (StatefulSet) ===
+keycloak   2     <none>   1
+keycloak-0                  1/1   Terminating   0     30m
+keycloak-1                  0/1   Running       0     143m
+
+=== 복구 — 노드 재기동 ===
+재기동 시각: 12:16:31
+Domain 'kc-lab-2' started
+
+  +30초  node=Ready | Running 파드 3 개 | 외부 HTTP 503
+  +60초  node=Ready | Running 파드 3 개 | 외부 HTTP 200
+  → 서비스 복귀

docs/evidence/a4-node-loss/06-control-plane-inventory.txt

@@ -0,0 +1,19 @@
+=== 복구 확인 ===
+keycloak-0                  1/1   Running   0     68s
+keycloak-1                  1/1   Running   0     144m
+postgres-7b474b88c8-9cmsv   1/1   Running   0     4m20s
+
+=== kc-lab-1(k3s server)에 무엇이 있는가 — 이게 곧 영향 범위다 ===
+  keycloak-lab     keycloak-1
+  kube-system      coredns-54996dc9b4-8k8fj
+  kube-system      helm-install-traefik-crd-q29b5
+  kube-system      local-path-provisioner-77b9867795-g27z8
+  kube-system      metrics-server-6dc596dfb8-7xxq4
+  kube-system      svclb-traefik-5eb6a9a1-qwwk5
+  kube-system      traefik-5d6fcf895-wpfhr
+  observability    grafana-845b5678cf-b6gvc
+  observability    node-exporter-9qk9w
+  observability    prometheus-6774f94f7c-pzr2t
+
+=== Traefik replica 수 (진입점의 단일 장애점인가) ===
+traefik   1     1

docs/evidence/a4-node-loss/07-control-plane-loss.txt

@@ -0,0 +1,19 @@
+=== 컨트롤 플레인 노드(kc-lab-1) 전원 차단 ===
+차단 시각: 12:18:08
+Domain 'kc-lab-1' destroyed
+
+
+  +20초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
+  +40초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
+  +60초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
+  +80초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
+  +100초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
+  +120초  외부 auth=000 grafana=502 | kubectl: Unable to connect to the server: dial tcp 
+  +140초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
+  +160초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
+
+=== 살아 있는 노드에서 직접 확인 — 워크로드는 도는가 ===
+  CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD                 NAMESPACE
+  e5f777900b762       60e153026e8f5       4 minutes ago       Running             keycloak            0                   640d4dafaefb3       keycloak-0          keycloak-lab
+  
+  6

docs/evidence/a4-node-loss/08-control-plane-recovery.txt

@@ -0,0 +1,13 @@
+=== 컨트롤 플레인 노드 복구 ===
+재기동: 12:23:39
+Domain 'kc-lab-1' started
+
+  +30초  외부=502 | kc-lab-1=Ready kc-lab-2=Ready 
+  +60초  외부=200 | kc-lab-1=Ready kc-lab-2=Ready 
+  → 서비스 복귀 (총 60초)
+
+=== 최종 상태 ===
+keycloak-0                  1/1   Running   0                   7m57s
+keycloak-1                  1/1   Running   1 (<invalid> ago)   151m
+postgres-7b474b88c8-9cmsv   1/1   Running   0                   11m
+  (prometheus port-forward 재연결 필요)

docs/evidence/a4-node-loss/README.md

@@ -0,0 +1,23 @@
+# A-4 — 노드 전원 차단 증거
+
+2026-09-04 12:07–12:24 KST
+해설: [`docs/experiment-a4-node-loss.md`](../../experiment-a4-node-loss.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-baseline.txt` | 차단 전 — 양쪽 Ready, **PVC 가 kc-lab-2 에 못박혀 있음**(재배치 불가의 원인), 외부 200 |
+| `02-worker-node-killed.txt` | `virsh destroy kc-lab-2` — **40초간 노드가 Ready 로 남아 있고** 외부는 이미 `000`. 이후 `503` |
+| `03-state-during-loss.txt` | **죽은 파드가 `ready=true`, 산 파드가 `ready=false`.** `up` 은 정확히 0. `unreachable` taint |
+| `04-eviction-timing.txt` | `tolerationSeconds=300` — **5분 뒤** 축출, 새 postgres 는 `Pending` |
+| `05-recovery.txt` | `FailedScheduling: didn't match PersistentVolume's node affinity`, StatefulSet `DESIRED=2 CURRENT=1`. 노드 복귀 후 **60초** |
+| `06-control-plane-inventory.txt` | kc-lab-1 에 있는 것 목록 — **Traefik `replicas=1`** |
+| `07-control-plane-loss.txt` | `virsh destroy kc-lab-1` — 외부 `000`, `kubectl` 불통. **그런데 `crictl ps` 로 보면 keycloak-0 은 Running** |
+| `08-control-plane-recovery.txt` | 60초 만에 복귀 |
+| `a4-up-dropped-per-node.png` | Grafana — `up` 이 노드별로 0 으로 떨어지는 구간. 12:18–12:23 은 **0 이 아니라 데이터 없음**(관측자가 같이 죽음) |
+
+## 핵심 네 줄
+
+1. **쿠버네티스는 40초 동안 노드가 살아 있다고 말한다.** 사용자는 이미 장애를 겪는 중이다.
+2. **죽은 파드의 상태는 화석이다.** `ready=true` 인 파드가 꺼진 기계 위에 있다.
+3. **StatefulSet 은 대체 파드를 만들지 않고, PVC 는 재배치를 막는다.** 사람이 개입해야 한다.
+4. **컨트롤 플레인 상실 ≠ 워크로드 상실.** 컨테이너는 계속 돌고, 들어갈 문만 사라진다.

docs/evidence/a5-asymmetric-partition/01-injection.txt

@@ -0,0 +1,42 @@
+=== A-5 기준선 ===
+  keycloak-0=10.42.1.77 (kc-lab-2)   keycloak-1=10.42.0.42 (kc-lab-1)
+keycloak-0                  1/1   Running   0               11m
+keycloak-1                  1/1   Running   1 (2m48s ago)   155m
+postgres-7b474b88c8-9cmsv   1/1   Running   0               14m
+Traceback (most recent call last):
+  File "<string>", line 3, in <module>
+    for r in json.load(sys.stdin)["data"]["result"]: print(f"  cluster_size {r["metric"].get("pod"):12} = {r["value"][1]}")
+             ~~~~~~~~~^^^^^^^^^^^
+  File "/usr/lib/python3.14/json/__init__.py", line 298, in load
+    return loads(fp.read(),
+        cls=cls, object_hook=object_hook,
+        parse_float=parse_float, parse_int=parse_int,
+        parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
+  File "/usr/lib/python3.14/json/__init__.py", line 352, in loads
+    return _default_decoder.decode(s)
+           ~~~~~~~~~~~~~~~~~~~~~~~^^^
+  File "/usr/lib/python3.14/json/decoder.py", line 345, in decode
+    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
+               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.14/json/decoder.py", line 363, in raw_decode
+    raise JSONDecodeError("Expecting value", s, err.value) from None
+json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
+
+=== 주입: keycloak-0 으로 들어가는 7800/57800 만 DROP (한 방향) ===
+  A-1 의 NetworkPolicy 는 conntrack ESTABLISHED 에 막혀 기존 연결을 못 끊었다.
+  FORWARD 최상단에 넣으면 conntrack 승인보다 먼저 평가된다.
+Chain FORWARD (policy ACCEPT)
+num  target     prot opt source               destination         
+1    DROP       6    --  0.0.0.0/0            10.42.1.77           tcp dpt:57800
+2    DROP       6    --  0.0.0.0/0            10.42.1.77           tcp dpt:7800
+  주입 시각: 12:28:23
+
+=== 관찰 — 어느 쪽이 먼저 상대를 의심하는가 ===
+  +25초   | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
+  +50초   | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
+  +75초   | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
+  +100초   | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
+  +125초   | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
+  +150초   | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
+  +175초   | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
+  +200초   | keycloak-0:1/1 keycloak-1:1/1 | 외부 200

docs/evidence/a5-asymmetric-partition/03-raw-table-injection.txt

@@ -0,0 +1,16 @@
+=== 실패한 규칙 제거 ===
+제거완료
+
+=== raw 테이블 PREROUTING 에 넣는다 — conntrack 보다 먼저 실행된다 ===
+  netfilter 순서: raw PREROUTING → conntrack → mangle → nat → filter
+Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
+num   pkts bytes target     prot opt in     out     source               destination         
+1        0     0 DROP       6    --  *      *       0.0.0.0/0            10.42.1.77           tcp dpt:57800
+2        0     0 DROP       6    --  *      *       0.0.0.0/0            10.42.1.77           tcp dpt:7800
+  주입: 12:32:35
+
+=== [검증] 이번엔 패킷이 걸렸는가 ===
+  Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
+   pkts bytes target     prot opt in     out     source               destination         
+      0     0 DROP       6    --  *      *       0.0.0.0/0            10.42.1.77           tcp dpt:57800
+      0     0 DROP       6    --  *      *       0.0.0.0/0            10.42.1.77           tcp dpt:7800

docs/evidence/a5-asymmetric-partition/04-correct-direction.txt

@@ -0,0 +1,18 @@
+=== keycloak-1(수신측)으로 들어가는 7800/57800 만 DROP — kc-lab-1 에 넣는다 ===
+  주입: 12:33:58
+
+=== [검증] 패킷이 걸리는가 ===
+  Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
+   pkts bytes target     prot opt in     out     source               destination         
+      0     0 DROP       6    --  *      *       0.0.0.0/0            10.42.0.42           tcp dpt:57800
+     19  2938 DROP       6    --  *      *       0.0.0.0/0            10.42.0.42           tcp dpt:7800
+
+=== 관찰 ===
+  +25초  - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
+  +50초  - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
+  +75초  - | keycloak-0:1/1 keycloak-1:0/1 | 외부 200
+  +100초  - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
+  +125초  - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
+  +150초  - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
+  +175초  - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
+  +200초  - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200

docs/evidence/a5-asymmetric-partition/05-reconnect-observed.txt

@@ -0,0 +1,19 @@
+=== 지금 연결 방향은? (차단은 → 10.42.0.42:7800 만) ===
+  tcp      6 299 ESTABLISHED src=10.42.0.42 dst=10.42.1.77 sport=48473 dport=7800 src=10.42.1.77 dst=10.42.0.42 sport=7800 dport=48473
+  tcp      6 86232 ESTABLISHED src=10.42.0.42 dst=10.42.1.77 sport=44205 dport=57800 src=10.42.1.77 dst=10.42.0.42 sport=57800 dport=44205
+
+=== 차단 규칙 누적 카운터 ===
+  Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
+   pkts bytes target     prot opt in     out     source               destination         
+     19  1096 DROP       6    --  *      *       0.0.0.0/0            10.42.0.42           tcp dpt:57800
+     21  3058 DROP       6    --  *      *       0.0.0.0/0            10.42.0.42           tcp dpt:7800
+
+=== 클러스터 멤버 (파드에서 직접) ===
+  keycloak-0  cluster_size=2.0
+  keycloak-1  cluster_size=2.0
+
+=== 최근 뷰 로그 ===
+  keycloak-0: [keycloak-0-24309(v=16.0.12)|12] (1) [keycloak-0-24309(v=16.0.12)]
+
+  keycloak-1: [keycloak-1-45480(v=16.0.12)|12] (1) [keycloak-1-45480(v=16.0.12)]
+

docs/evidence/a5-asymmetric-partition/06-view-history-and-cleanup.txt

@@ -0,0 +1,28 @@
+=== 뷰 변화 전체 이력 (차단 12:33:58 전후) ===
+--- keycloak-0 ---
+  2026-09-04 03:33:49 | during merge: CacheTopology{id=37, phase=NO_REBALANCE, rebalanceId=11, currentCH=DefaultConsistentHash{ns=25
+  2026-09-04 03:33:49,398 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=offlineSessions] ISPN100007: After me
+  2026-09-04 03:33:49 | during merge: CacheTopology{id=37, phase=NO_REBALANCE, rebalanceId=11, currentCH=DefaultConsistentHash{ns=25
+  2026-09-04 03:33:49,399 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=loginFailures] ISPN100007: After merg
+  2026-09-04 03:33:49 | during merge: CacheTopology{id=37, phase=NO_REBALANCE, rebalanceId=11, currentCH=DefaultConsistentHash{ns=25
+  2026-09-04 03:33:49,402 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=actionTokens] ISPN100007: After merge
+--- keycloak-1 ---
+  2026-09-04 03:32:59,874 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p2-t2) [Context=work] ISPN100007: After merge (or coo
+  2026-09-04 03:32:59,879 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p2-t2) [Context=clientSessions] ISPN100007: After mer
+  2026-09-04 03:32:59,879 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=offlineSessions] ISPN100007: After me
+  2026-09-04 03:32:59,883 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=loginFailures] ISPN100007: After merg
+  2026-09-04 03:32:59,884 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p2-t2) [Context=actionTokens] ISPN100007: After merge
+  2026-09-04 03:33:49 | MergeView::[keycloak-0-24309(v=16.0.12)|13] (2) [keycloak-0-24309(v=16.0.12), keycloak-1-45480(v=16.0.12)], 
+
+=== MERGE3 병합 이벤트 ===
+  keycloak-0  merge_events=1.0  suspected=0.0
+  keycloak-1  merge_events=1.0  suspected=0.0
+
+=== 차단 해제 ===
+해제완료
+  keycloak-0  cluster_size=2.0
+warning: couldn't attach to pod/a5-f, falling back to streaming logs: unable to upgrade connection: container a5-f not found in pod a5-f_keycloak-lab
+  keycloak-0  cluster_size=2.0
+  keycloak-1  cluster_size=2.0
+keycloak-0                  1/1   Running   0             22m
+keycloak-1                  1/1   Running   1 (13m ago)   166m

docs/evidence/a5-asymmetric-partition/07-bidirectional-block.txt

@@ -0,0 +1,17 @@
+=== 양방향 차단 — 두 노드 모두에 raw DROP ===
+  주입: 12:40:25
+  +25초  keycloak-0:1/1 keycloak-1:1/1 | ready=[10.42.0.42 10.42.1.77] 외부 200
+  +50초  keycloak-0:1/1 keycloak-1:1/1 | ready=[10.42.0.42 10.42.1.77] 외부 200
+  +75초  keycloak-0:1/1 keycloak-1:1/1 | ready=[10.42.0.42 10.42.1.77] 외부 200
+  +100초  keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
+  +125초  keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
+  +150초  keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
+  +175초  keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
+  +200초  keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
+  +225초  keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
+
+=== 뷰 상태 ===
+  keycloak-0  [keycloak-0-24309(v=16.0.12)|14] (1) [keycloak-0-24309(v=16.0.12)]
+
+  keycloak-1  [keycloak-1-45480(v=16.0.12)|14] (1) [keycloak-1-45480(v=16.0.12)]
+

docs/evidence/a5-asymmetric-partition/08-coordinator-and-recovery.txt

@@ -0,0 +1,39 @@
+=== 왜 한쪽만 DOWN 인가 — 코디네이터 여부 확인 ===
+       name       |       ip        | coord 
+------------------+-----------------+-------
+ keycloak-0-24309 | 10.42.1.77:7800 | t
+ keycloak-1-45480 | 10.42.0.42:7800 | t
+(2 rows)
+
+=== 양쪽 헬스체크 상세 ===
+--- keycloak-0 ---
+{"status":"UP","checks":[{"name":"GracefulShutdown","status":"UP"}
+{"name":"KeycloakInitialized","status":"UP"}
+{"name":"Keycloakclusterhealthcheck","status":"UP"}
+
+--- keycloak-1 ---
+{"status":"DOWN","checks":[{"name":"GracefulShutdown","status":"UP"}
+{"name":"Keycloakdatabaseconnectionsasynchealthcheck","status":"UP"}
+{"name":"KeycloakInitialized","status":"UP"}
+
+warning: couldn't attach to pod/a5-h, falling back to streaming logs: Internal error occurred: Internal error occurred: error attaching to container: container is in CONTAINER_EXITED state
+--- keycloak-0 ---
+{"status":"UP","checks":[{"name":"GracefulShutdown","status":"UP"}
+{"name":"KeycloakInitialized","status":"UP"}
+{"name":"Keycloakclusterhealthcheck","status":"UP"}
+
+--- keycloak-1 ---
+{"status":"DOWN","checks":[{"name":"GracefulShutdown","status":"UP"}
+{"name":"Keycloakdatabaseconnectionsasynchealthcheck","status":"UP"}
+{"name":"KeycloakInitialized","status":"UP"}
+
+
+=== 차단 해제 ===
+  해제: 12:44:37
+  +25초  keycloak-0:1/1 keycloak-1:0/1 
+  +50초  keycloak-0:1/1 keycloak-1:1/1 
+  → 복구 완료
+  keycloak-0  MergeView::[keycloak-0-24309(v=16.0.12)|15] (2) [keycloak-0-24309(v=16.0.12), keycloak-1-45480(
+
+  keycloak-1  MergeView::[keycloak-0-24309(v=16.0.12)|15] (2) [keycloak-0-24309(v=16.0.12), keycloak-1-45480(
+

docs/evidence/a5-asymmetric-partition/README.md

@@ -0,0 +1,23 @@
+# A-5 — 비대칭 파티션 증거
+
+2026-09-04 12:28–12:46 KST
+해설: [`docs/experiment-a5-asymmetric-partition.md`](../../experiment-a5-asymmetric-partition.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-injection.txt` | 첫 시도 — `iptables -I FORWARD 1` |
+| `02-injection-verify.txt` | **실패 확인** — kube-router 가 자기 체인을 위로 재삽입, DROP 규칙 **0 패킷** |
+| `03-raw-table-injection.txt` | `raw` 테이블로 이동. 그래도 0 패킷 |
+| `04-correct-direction.txt` | **연결 방향이 반대였다.** 수신측 노드로 옮기니 **19/21 패킷 차단** |
+| `05-reconnect-observed.txt` | **핵심** — 연결이 **열린 방향으로 뒤집혀 재연결**. 뷰 변화 없음, `suspected=0` |
+| `06-view-history-and-cleanup.txt` | 뷰 ID 이력. 주입 이후 변화 없음 확인 |
+| `07-bidirectional-block.txt` | 양방향 차단 → **뷰 14, 멤버 1개씩.** 그런데 `ready=[10.42.1.77]` 로 **한쪽은 남고 외부 200** |
+| `08-coordinator-and-recovery.txt` | **`coord=t` 가 둘**(split brain). keycloak-0 `UP` / keycloak-1 `DOWN`. 해제 후 50초, `MergeView |15] (2)` |
+| `a5-cluster-size-bidirectional-block.png` | Grafana — `vendor_cluster_size` 가 갈라졌다 합쳐지는 구간 |
+
+## 핵심 네 줄
+
+1. **주입을 세 번 실패했다** — CNI 체인 경쟁, 연결 방향 오판, conntrack. 셋 다 "아무 일 없음"으로 보였다.
+2. **단방향 차단으로는 못 가른다.** JGroups 가 열린 방향으로 재연결한다 (자가 치유).
+3. **양방향이면 갈라진다** — `coord=t` 가 둘.
+4. **그래도 전면 장애가 아니다.** 코디네이터 쪽만 UP 을 유지해 서비스가 계속된다 — A-1 의 열린 질문에 대한 답.

docs/evidence/a6-latency-injection/01-baseline.txt

@@ -0,0 +1,22 @@
+=== 배치 — 왜 A/B 가 되는가 ===
+  postgres  10.42.1.76  (kc-lab-2)
+  keycloak-0 10.42.1.77 (kc-lab-2) → DB 와 같은 노드, cni0 로 직행
+  keycloak-1 10.42.0.42 (kc-lab-1) → DB 와 다른 노드, VXLAN 을 건넌다  ← 여기에 지연을 건다
+
+=== 사용 가능한 커넥션 풀 지표 ===
+agroal_acquire_count_total
+agroal_active_count
+agroal_available_count
+agroal_awaiting_count
+agroal_blocking_time_average_milliseconds
+agroal_blocking_time_max_milliseconds
+agroal_blocking_time_total_milliseconds
+agroal_creation_count_total
+agroal_creation_time_average_milliseconds
+agroal_creation_time_max_milliseconds
+agroal_creation_time_total_milliseconds
+agroal_destroy_count_total
+
+=== 기준선 지연 — 각 노드에서 로그인 20회 ===
+  keycloak-0  평균 70 ms
+  keycloak-1  평균 66 ms

docs/evidence/a6-latency-injection/02-delay-injected.txt

@@ -0,0 +1,25 @@
+=== 주입: postgres(10.42.1.76) 가 보내는 패킷만 200ms 지연 (kc-lab-2 eth0) ===
+  prio qdisc 로 밴드를 나누고, u32 필터로 출발지 IP 가 postgres 인 것만 3번 밴드로 보낸다
+Cannot find device "eth0"
+Cannot find device "eth0"
+적용완료
+Cannot find device "eth0"
+Cannot find device "eth0"
+  주입: 13:14:55
+
+=== [검증] 지연이 실제로 걸렸는가 — 두 노드 비교 ===
+  keycloak-0  평균 43 ms   최대 64 ms
+  keycloak-1  평균 47 ms   최대 70 ms
+
+=== 커넥션 풀 상태 ===
+--- keycloak-0 ---
+  agroal_blocking_time_max_milliseconds 102.0
+  agroal_active_count 0.0
+  agroal_awaiting_count 0.0
+  agroal_blocking_time_average_milliseconds 0.0
+  agroal_available_count 2.0
+  agroal_blocking_time_max_milliseconds 164.0
+  agroal_active_count 0.0
+  agroal_awaiting_count 0.0
+  agroal_blocking_time_average_milliseconds 0.0
+  agroal_available_count 2.0

docs/evidence/a6-latency-injection/03-flannel-injection.txt

@@ -0,0 +1,20 @@
+=== 오버레이 인터페이스 확인 ===
+flannel.1        UNKNOWN        a6:b2:62:04:c1:a4 <BROADCAST,MULTICAST,UP,LOWER_UP> 
+cni0             UP             5a:77:1a:e2:b0:a4 <BROADCAST,MULTICAST,UP,LOWER_UP> 
+
+=== flannel.1 에 주입 — 여기서는 파드 IP 가 보인다 (캡슐화 전) ===
+qdisc prio 1: root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
+ Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
+ backlog 0b 0p requeues 0
+qdisc netem 30: parent 1:3 limit 1000 delay 200ms
+ Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
+ backlog 0b 0p requeues 0
+
+=== [검증] 필터에 패킷이 걸리는가 ===
+  qdisc netem 30: parent 1:3 limit 1000 delay 200ms
+   Sent 18388 bytes 150 pkt (dropped 0, overlimits 0 requeues 0) 
+   backlog 0b 0p requeues 0
+
+=== 두 노드 지연 비교 (기준선: k0=70ms k1=66ms) ===
+  keycloak-0  평균 41 ms   최대 57 ms
+  keycloak-1  평균 1872 ms   최대 1887 ms

docs/evidence/a6-latency-injection/04-pool-under-load.txt

@@ -0,0 +1,37 @@
+=== 동시 부하 20건을 keycloak-1 에 — 커넥션 풀이 견디는가 ===
+        1 200 1.911191
+        1 200 1.913766
+        1 200 1.958374
+        1 200 1.981620
+        1 200 10.539402
+        1 200 11.951943
+        1 200 13.351102
+        1 200 14.785832
+        1 200 16.189533
+        1 200 17.625166
+        1 200 19.053724
+        1 200 20.495883
+        1 200 21.905932
+        1 200 22.228466
+        1 200 22.230871
+        1 200 3.441366
+        1 200 4.841075
+        1 200 6.257489
+        1 200 7.704608
+        1 200 9.104792
+
+=== 부하 직후 커넥션 풀 ===
+  agroal_blocking_time_max_milliseconds 20000.0
+  agroal_max_used_count 19.0
+  agroal_acquire_count_total 672.0
+  agroal_active_count 0.0
+  agroal_awaiting_count 0.0
+  agroal_blocking_time_average_milliseconds 281.0
+  agroal_available_count 19.0
+
+=== readiness 가 흔들렸는가 ===
+keycloak-0                  1/1   Running   0             60m
+keycloak-1                  1/1   Running   1 (51m ago)   3h24m
+52m         Normal    TaintManagerEviction   pod/keycloak-1                   Cancelling deletion of Pod keycloak-lab/keycloak-1
+32m         Warning   Unhealthy              pod/keycloak-1                   Readiness probe failed: HTTP probe failed with statuscode: 503
+89s         Warning   Unhealthy              pod/keycloak-1                   Readiness probe failed: Get "http://10.42.0.42:9000/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

docs/evidence/a6-latency-injection/05-recovery.txt

@@ -0,0 +1,12 @@
+=== 지연 해제 ===
+해제완료
+qdisc noqueue 0: root refcnt 2 
+
+=== 회복 확인 ===
+  keycloak-0  평균 43 ms
+  keycloak-1  평균 51 ms
+keycloak-0                  1/1   Running   0             61m
+keycloak-1                  1/1   Running   1 (52m ago)   3h24m
+
+=== 낙관적 락 충돌이 늘었는가 — 지연 중 로그 ===
+  관련 로그 줄수: 0

docs/evidence/a6-latency-injection/README.md

@@ -0,0 +1,20 @@
+# A-6 — 지연 주입 증거
+
+2026-09-04 13:10–13:35 KST
+해설: [`docs/experiment-a6-latency-injection.md`](../../experiment-a6-latency-injection.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-baseline.txt` | 배치 설명(A/B 가 되는 이유), `agroal_*` 지표 목록, **기준선 70ms / 66ms** |
+| `02-delay-injected.txt` | 첫 시도 실패 — **`Cannot find device "eth0"`** (Debian 은 `enp1s0`) |
+| `03-flannel-injection.txt` | **성공** — `flannel.1` 에 걸어야 파드 IP 가 보인다. netem `Sent 150 pkt` 로 검증. **k0=41ms vs k1=1872ms** |
+| `04-pool-under-load.txt` | 동시 20건 — 응답이 1.9초에서 **22.2초**까지 계단. `blocking_time_max=20000ms`, `max_used_count=19`, **readiness 프로브 타임아웃** |
+| `05-recovery.txt` | 해제 즉시 43ms / 51ms 회복. **낙관적 락 충돌 0건**(예측 빗나감) |
+| `a6-connection-pool-blocking.png` | Grafana — `agroal_blocking_time_max_milliseconds` |
+
+## 핵심 네 줄
+
+1. **200ms 가 1,872ms 가 된다.** 로그인 트랜잭션의 왕복이 9번이라 지연이 곱해진다.
+2. **동시 부하에서 22초까지 늘어난다.** 커넥션 풀 큐잉으로 한 번 더 곱해진다.
+3. **헬스체크도 같은 줄에 선다** — 프로브가 타임아웃되어 노드가 로드밸런서에서 빠지고, 남은 노드로 부하가 몰린다.
+4. **낙관적 락 충돌은 없었다.** 로그인은 새 행을 만들 뿐 같은 행을 다투지 않는다 — B-3 의 영역.

docs/experiment-a2-database-loss.md

@@ -0,0 +1,308 @@
+# A-2 — PostgreSQL 이 죽으면 어떻게 되는가
+
+브랜치 `feature/keycloak-a2-database-loss` ·
+증거 [`docs/evidence/a2-database-loss/`](evidence/a2-database-loss/) ·
+2026-09-04 11:56–11:58 KST · Keycloak 26.7.0
+
+선행: [`A-0`](experiment-00-session-replication.md) ·
+[`A-1`](experiment-a1-jgroups-transport-block.md)
+
+---
+
+## 0. 결론부터
+
+| 예측 | 결과 |
+|---|---|
+| 즉시 전면 장애 | **맞다.** 외부 진입점 **503**, 양쪽 노드 NotReady |
+| 캐시에 있어도 못 쓴다 | **맞다.** 캐시를 가진 노드도 `500` |
+| — | **`up = 1` 인 채로 전면 장애가 났다** (관측의 함정) |
+| — | **DB 복귀 15초 만에 재시작 없이 자동 회복** |
+
+**A-1 과 정반대다.** A-1 은 한쪽만 빠지고 서비스가 계속됐지만,
+A-2 는 **살아남는 노드가 없다.**
+
+---
+
+## 1. 설계 — 네 경로를 구분해서 본다
+
+A-1 에서 **"룩어사이드 캐시는 읽을 때 DB 와 대조하지 않는다"** 를 확인했다.
+그렇다면 캐시를 가진 노드는 DB 없이도 버틸지 모른다. 그 가설을 가른다.
+
+| # | 경로 | 무엇을 보는가 |
+|---|---|---|
+| ① | **캐시를 가진 노드**에서 refresh | 캐시가 DB 를 대신할 수 있는가 |
+| ② | 캐시가 없는 노드에서 refresh | 완전한 DB 의존 |
+| ③ | 새 로그인 | 쓰기 경로 |
+| ④ | 이미 발급된 토큰으로 조회 | 서명만으로 되는 경로 |
+
+**access token 수명이 60초**이므로, 토큰 발급 → DB 정지 → 시험을 그 안에
+끝내야 한다.
+
+### 계측 도구를 바꿨다
+
+A-1 에서 임시 curl 파드가 형편없는 계측 도구임을 확인했다. 여기서는
+**상주 탐침 파드**를 하나 띄우고 `exec` 로 단계를 이어간다. 토큰을 파드 안
+파일에 남겨 **DB 정지 전후로 같은 토큰**을 쓸 수 있다.
+
+```bash
+kubectl -n keycloak-lab run a2-probe --image=curlimages/curl:8.11.1 \
+  --restart=Never --command -- sleep 7200
+kubectl -n keycloak-lab wait --for=condition=Ready pod/a2-probe --timeout=120s
+```
+
+---
+
+## 2. 기준선
+
+```
+keycloak-0   ready=true   10.42.1.67   kc-lab-2
+keycloak-1   ready=true   10.42.0.35   kc-lab-1
+postgres     ready=true   10.42.1.24   kc-lab-2
+
+cluster_size keycloak-0 = 2
+cluster_size keycloak-1 = 2
+```
+
+세션을 양쪽에 하나씩 만들고, A-0 대로 **각자 자기 노드에만 캐시**되는 것을
+확인했다.
+
+```
+  keycloak-0 에서 로그인  sid=EAXV5HcG2J1BZ3vnwONf64AQ
+  keycloak-1 에서 로그인  sid=McyTj5lj3n_JqApCXeuAHExc
+  → 캐시  keycloak-0 = 1 건 / keycloak-1 = 0 건 (스크레이프 지연)
+```
+
+---
+
+## 3. 주입
+
+```bash
+kubectl -n keycloak-lab scale deployment/postgres --replicas=0
+kubectl -n keycloak-lab wait --for=delete pod -l app=postgres --timeout=90s
+```
+
+```
+정지 시각: 11:56:04
+삭제 완료: 11:56:04     ← 즉시
+```
+
+---
+
+## 4. 결과 — 네 경로
+
+```
+  ① 캐시를 가진 노드(keycloak-0)에서 refresh   HTTP 500
+  ② 캐시가 없는 노드(keycloak-1)에서 refresh   HTTP 500
+  ③ 새 로그인                                  HTTP 500
+  ④ 관리 API (세션 조회 필요)                   HTTP 500
+
+  --- 오류 본문 ---
+  {"error":"unknown_error","error_description":"For more on this error consult the server log."}
+```
+
+### ① 이 500 인 것이 중요하다
+
+**캐시에 세션을 들고 있어도 refresh 는 실패한다.**
+
+A-1 에서는 로그아웃된 세션을 캐시로 `200` 을 줬다. 왜 여기서는 안 되는가.
+
+```
+   refresh 처리
+     ├── 세션이 존재하는가        →  캐시로 답할 수 있다
+     └── LAST_SESSION_REFRESH 갱신 →  DB 쓰기가 필요하다   ← 여기서 죽는다
+```
+
+A-0 에서 잡은 SQL 그대로다.
+
+```sql
+update OFFLINE_USER_SESSION set LAST_SESSION_REFRESH=$1, VERSION=$2 where ...
+```
+
+> **캐시는 읽기를 대신할 뿐, 쓰기를 대신하지 못한다.**
+> refresh 는 이름과 달리 **쓰기 연산**이다.
+
+### 로그가 말하는 원인
+
+```
+Caused by: java.net.ConnectException: Connection refused
+  at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect
+  at io.agroal.pool.ConnectionPool$CreateConnectionTask.call
+```
+
+`agroal` 은 Quarkus 의 커넥션 풀이다. 풀이 새 커넥션을 만들지 못한다.
+
+---
+
+## 5. 살아남은 것 — 상태가 필요 없는 경로
+
+```
+  JWKS 엔드포인트(realm 공개키)   HTTP 200
+  realm 메타데이터(.well-known)   HTTP 200
+  관리 API (세션 조회 필요)        HTTP 500
+```
+
+**realm 공개키와 메타데이터는 메모리에 있으므로 DB 없이도 응답한다.**
+
+이론적으로는 **이미 JWKS 를 캐시한 리소스 서버는 토큰 검증을 계속할 수 있다**는
+뜻이다. 다만 이 실험대에는 독립 리소스 서버가 아직 없으므로 **여기까지가
+말할 수 있는 범위**다 — B층에서 확인한다.
+
+> **그런데 정문으로는 이것도 못 쓴다.** 아래 6절 때문이다.
+
+---
+
+## 6. 전면 장애 — 살아남는 노드가 없다
+
+```
+=== 파드 Ready ===
+keycloak-0   false   restarts=0
+keycloak-1   false   restarts=0
+
+=== Service 엔드포인트 ===
+  ready   : []                              ← 비었다
+  notReady: [10.42.0.35 10.42.1.67]
+
+=== 외부 진입점 ===
+  https://auth.hyeonworks.com/realms/master   HTTP 503
+```
+
+```json
+{ "status": "DOWN",
+  "checks": [
+    { "name": "Keycloak cluster health check",                    "status": "UP"   },
+    { "name": "Keycloak database connections async health check", "status": "DOWN" },
+    { "name": "Keycloak Initialized",                             "status": "UP"   } ] }
+```
+
+**`cluster health` 는 UP 인데 `database connections` 가 DOWN 이라 전체가 DOWN 이다.**
+헬스체크는 **모든 항목이 UP 이어야 UP** 이다.
+
+### A-1 과의 대비가 이 실험의 핵심이다
+
+| | A-1 (7800 차단) | **A-2 (DB 정지)** |
+|---|---|---|
+| Ready 인 파드 | keycloak-1 **1개 생존** | **0개** |
+| Service `ready` | `[10.42.0.35]` | **`[]`** |
+| 외부 응답 | **200** | **503** |
+| 성격 | 용량 저하 | **전면 장애** |
+
+**노드를 몇 대로 늘려도 DB 가 죽으면 전부 같이 죽는다.**
+Keycloak 의 대수는 DB 장애에 아무 도움이 되지 않는다.
+
+> 원래 질문 *"Redis 또는 DB가 뒤질 경우 어떻게 복구를 해야 되는지"* 에 대한
+> 첫 번째 답 — **복구 이전에, DB 이중화가 Keycloak 대수보다 우선한다.**
+
+---
+
+## 7. 관측의 함정 — `up = 1` 인 채로 전면 장애
+
+```
+  up{pod=keycloak-1} = 1
+  up{pod=keycloak-0} = 1     ← 서비스는 503 인데
+```
+
+![up 은 움직이지 않았다](evidence/a2-database-loss/a2-up-stayed-1-during-outage.png)
+
+**전 구간 평평하다.** (11:44 의 짧은 골은 A-1 에서 파드를 교체한 자국이다.)
+
+`up` 은 **Prometheus 가 `/metrics` 를 긁는 데 성공했는가**만 말한다.
+프로세스는 멀쩡히 살아 메트릭을 내놓고 있었다. **기능은 전멸했는데.**
+
+| 지표 | 이 장애에서 |
+|---|---|
+| `up` | **1 — 아무것도 알려주지 않는다** |
+| 파드 `Ready` | **false — 여기서 드러난다** |
+| 외부 HTTP 코드 | **503 — 사용자가 겪는 것** |
+
+> **A-0 에서 나는 `up` 을 "가장 중요한 합성 지표"라고 썼다.**
+> 절반만 맞다. `up` 은 **대상이 사라진 것**을 잡지만 **대상이 살아서 못 쓰는 것**은
+> 못 잡는다. 후자가 운영에서 훨씬 흔하다.
+>
+> **알림은 `up` 이 아니라 readiness 와 외부 응답 코드에 걸어야 한다.**
+
+이 실험대에는 아직 `kube-state-metrics` 가 없어 파드 readiness 가 지표로
+남지 않는다. **관측 스택에 빠진 것을 이 실험이 찾아냈다** — 보완 항목이다.
+
+---
+
+## 8. 복구 — 자동이었다
+
+```bash
+kubectl -n keycloak-lab scale deployment/postgres --replicas=1
+```
+
+```
+  재기동 시각: 11:57:09
+  +15초  keycloak-0 true  keycloak-1 true  | 외부 HTTP 200
+  → 서비스 복귀
+
+  재시작 횟수:  keycloak-0 = 0,  keycloak-1 = 0
+  정지 전 세션: online 세션 5 건 살아남음
+```
+
+| | |
+|---|---|
+| 회복 시간 | **약 15초** (DB Ready 이후) |
+| 사람 개입 | **없음** |
+| Keycloak 재시작 | **불필요** — `restarts=0` |
+| 세션 | **살아남음** — DB 에 있으므로 |
+
+**커넥션 풀이 스스로 재연결하고 readiness 가 다시 UP 이 되면서 Service 에
+복귀했다.** `readiness` 를 쓴 설계의 이득이 여기서 나온다 — `liveness` 였다면
+파드가 재시작되어 캐시까지 날아갔을 것이다.
+
+### 개념 — readiness 와 liveness 를 가르는 기준
+
+| | 실패하면 | 언제 쓰나 |
+|---|---|---|
+| **liveness** | **재시작** | 재시작하면 나아지는 문제 (교착, 메모리 누수) |
+| **readiness** | **트래픽에서 격리** | 재시작해도 안 나아지는 문제 (**의존 대상이 죽음**) |
+
+**DB 장애에 liveness 를 걸면 재앙이다.** 모든 파드가 무한 재시작하고,
+DB 가 돌아와도 CrashLoopBackOff 의 백오프 때문에 회복이 늦어진다.
+
+---
+
+## 9. 재현 절차 (명령어)
+
+```bash
+# 0. 상주 탐침 (임시 파드는 계측에 부적합 — A-1 참조)
+kubectl -n keycloak-lab run a2-probe --image=curlimages/curl:8.11.1 \
+  --restart=Never --command -- sleep 7200
+kubectl -n keycloak-lab wait --for=condition=Ready pod/a2-probe --timeout=120s
+
+# 1. 토큰 발급 (access 60초 안에 시험을 끝내야 한다)
+kubectl -n keycloak-lab exec a2-probe -- sh -c \
+  'curl -s -X POST http://<k0>:8080/realms/master/protocol/openid-connect/token \
+     -d grant_type=password -d client_id=admin-cli \
+     -d username=admin -d password=<pw> > /tmp/tok.json'
+
+# 2. DB 정지
+kubectl -n keycloak-lab scale deployment/postgres --replicas=0
+kubectl -n keycloak-lab wait --for=delete pod -l app=postgres --timeout=90s
+
+# 3. 네 경로
+kubectl -n keycloak-lab exec a2-probe -- curl -s -o /dev/null -w '%{http_code}\n' ...
+
+# 4. 영향 범위
+kubectl -n keycloak-lab get endpoints keycloak \
+  -o jsonpath='{.subsets[*].addresses[*].ip}'          # 비어 있으면 전면 장애
+curl -s -o /dev/null -w '%{http_code}\n' https://auth.hyeonworks.com/realms/master
+
+# 5. up 이 거짓말하는 것을 확인
+curl -s "http://localhost:19090/api/v1/query?query=up%7Bjob=%22keycloak%22%7D"
+
+# 6. 복구
+kubectl -n keycloak-lab scale deployment/postgres --replicas=1
+```
+
+---
+
+## 10. 다음 실험에 남기는 것
+
+| 실험 | 이 실험이 준 것 |
+|---|---|
+| **A-3** DB 강제 종료 | 정상 정지는 데이터를 안 잃었다. **강제 종료는?** (`synchronous_commit OFF`) |
+| **A-4** 노드 상실 | postgres 가 kc-lab-2 에 있으므로 그 노드를 죽이면 **A-2 가 함께 일어난다** |
+| **D-1** 백업·복구 | 여기서는 DB 가 되살아났다. **데이터가 사라졌다면?** |
+| 관측 스택 | **`kube-state-metrics` 가 없어 파드 readiness 가 지표로 안 남는다** — 보완 필요 |

docs/experiment-a3-database-crash.md

@@ -0,0 +1,323 @@
+# A-3 — DB 를 강제로 죽이면 무엇을 잃는가 (RPO)
+
+브랜치 `feature/keycloak-a3-database-crash` ·
+증거 [`docs/evidence/a3-database-crash/`](evidence/a3-database-crash/) ·
+2026-09-04 12:00–12:05 KST · Keycloak 26.7.0 / PostgreSQL 16
+
+선행: [`A-0`](experiment-00-session-replication.md) ·
+[`A-2`](experiment-a2-database-loss.md)
+
+---
+
+## 0. 결론부터
+
+```
+  클라이언트가 200 과 토큰을 받은 로그인 : 153 건
+  그중 DB 에 실제로 존재                 : 149 건
+  ★ 유실                                 :   4 건
+```
+
+**로그인이 성공했다고 응답받았는데 세션이 존재하지 않는다.**
+
+A-0 에서 발견한 `SET LOCAL synchronous_commit TO OFF` 의 대가를 실측했다.
+버그가 아니라 **의도된 설계**이며, 그 비용이 얼마인지를 숫자로 확인한 것이다.
+
+---
+
+## 1. 설계 — 무엇을 재야 손실이 보이는가
+
+### 1-1. `LAST_SESSION_REFRESH` 로는 못 잰다
+
+처음 계획은 "세션 갱신 시각이 되감기는지" 보는 것이었다. 스키마를 보고 접었다.
+
+```
+ created_on           | integer
+ last_session_refresh | integer      ← 초 단위
+```
+
+**손실 창은 수백 밀리초**인데 눈금이 **1초**다. 보일 리가 없다.
+
+### 1-2. 행 존재 여부로 잰다 — 이진 판정
+
+```
+   로그인 1회  =  OFFLINE_USER_SESSION 행 1개
+   클라이언트가 sid 를 받았다  =  서버가 COMMIT 했다고 응답했다
+   크래시 후 그 sid 가 없다    =  잃은 것
+```
+
+**있거나 없거나**이므로 눈금 문제가 없다.
+
+### 1-3. 그런데 로그인도 비동기 커밋인가 — **먼저 확인해야 한다**
+
+A-0 에서 잡은 것은 **refresh** 트랜잭션이었다. 로그인(INSERT)도 그런지는
+확인하지 않았다. 아니라면 이 측정 설계 자체가 성립하지 않는다.
+
+```bash
+kubectl -n keycloak-lab exec deploy/postgres -- \
+  psql -U keycloak -d keycloak -c "alter system set log_statement='all'"
+kubectl -n keycloak-lab exec deploy/postgres -- \
+  psql -U keycloak -d keycloak -c "select pg_reload_conf()"
+```
+
+```
+BEGIN
+insert into OFFLINE_USER_SESSION (...) values (...)
+insert into OFFLINE_CLIENT_SESSION (...) values (...)
+SET LOCAL synchronous_commit TO OFF          ← 로그인도 비동기 커밋이다
+COMMIT
+```
+
+**확인됐고, 함의가 refresh 보다 훨씬 무겁다.**
+
+| | 잃으면 |
+|---|---|
+| refresh 갱신 시각 | 세션 수명이 조금 짧아진다. 사용자는 모른다 |
+| **로그인 자체** | **토큰은 손에 있는데 세션이 없다.** 다음 요청부터 실패 |
+
+---
+
+## 2. 실패한 주입 ① — `--grace-period=0 --force` 는 크래시가 아니다
+
+```bash
+kubectl -n keycloak-lab delete pod -l app=postgres --grace-period=0 --force
+```
+
+```
+  클라이언트 성공: 291 건
+  DB 에 존재:      291 건
+  ★ 유실:            0 건
+```
+
+**0건.** 그런데 이건 "안 잃었다"가 아니라 **죽인 적이 없는 것**이다.
+
+```
+=== 재기동 로그 ===
+  database system is ready to accept connections
+  (그뿐. "not properly shut down" 이 없다)
+```
+
+**crash recovery 가 돌지 않았다 = 깨끗하게 내려갔다.**
+
+| 신호 | PostgreSQL 의 반응 |
+|---|---|
+| **SIGTERM** | **fast shutdown** — 진행 중 트랜잭션을 롤백하고 **WAL 을 플러시**한 뒤 종료 |
+| SIGINT | smart shutdown — 연결이 끊기길 기다린다 |
+| **SIGKILL** | **즉사** — 플러시 없음. 다음 기동에 crash recovery |
+
+`--force --grace-period=0` 는 API 오브젝트를 즉시 지우지만 컨테이너 런타임은
+여전히 정상 종료 절차를 밟는다. **PostgreSQL 은 SIGTERM 을 받고 얌전히
+플러시했다.**
+
+> **A-1 에서 배운 것이 또 나왔다** — 주입이 실제로 걸렸는지 먼저 확인하지
+> 않으면 **"아무 일도 없었다"를 결과로 착각한다.**
+> 여기서는 **crash recovery 메시지가 그 확인 수단**이다.
+
+---
+
+## 3. 실패한 주입 ② — 컨테이너 안에서 PID 1 은 SIGKILL 을 받지 않는다
+
+```bash
+kubectl -n keycloak-lab exec deploy/postgres -- kill -9 1
+```
+
+**아무 일도 일어나지 않았다.** 파드는 재시작하지 않았고 로그 시각도 그대로였다.
+
+### 개념 — PID 1 의 시그널 보호
+
+리눅스 커널은 **PID 1 을 특별 취급**한다. 자기 PID 네임스페이스 안에서 온
+시그널은 **핸들러가 등록된 것만** 전달된다. **SIGKILL 도 예외가 아니다.**
+
+```
+   같은 네임스페이스 안에서  →  PID 1 은 등록하지 않은 시그널을 무시한다
+   조상 네임스페이스에서     →  전달된다 (노드에서 kill -9 하면 죽는다)
+```
+
+부팅 초기에 init 을 실수로 죽여 시스템이 멈추는 것을 막기 위한 장치인데,
+컨테이너에서는 **"안에서는 PID 1 을 못 죽인다"** 로 나타난다.
+
+---
+
+## 4. 성공한 주입 — 백엔드 프로세스를 죽인다
+
+PostgreSQL 은 **postmaster(부모) + 연결마다 백엔드(자식)** 구조다.
+자식 하나가 비정상 종료하면 **postmaster 는 공유 메모리가 오염됐다고 보고
+전체를 재초기화**한다. 그게 곧 crash recovery 다.
+
+```bash
+kubectl -n keycloak-lab exec deploy/postgres -- \
+  sh -c 'kill -9 $(pgrep -f "postgres: keycloak keycloak" | head -1)'
+```
+
+```
+server process (PID 40) was terminated by signal 9: Killed
+terminating any other active server processes
+all server processes terminated; reinitializing
+database system was not properly shut down; automatic recovery in progress
+redo starts at 0/23CAB68
+redo done at 0/2529E40
+checkpoint complete: wrote 113 buffers ...
+database system is ready to accept connections
+```
+
+**이번엔 주입이 걸렸다.** `not properly shut down` + `redo` 가 증거다.
+
+파드는 재시작하지 않는다 (`restarts=0`) — 컨테이너의 PID 1 인 postmaster 는
+살아 있고, 자식만 갈아치운 것이다. **데이터 관점에서는 전원이 나간 것과 같다.**
+
+---
+
+## 5. 결과
+
+```
+=== 크래시 전후 대조 ===
+  클라이언트가 200 과 토큰을 받은 로그인 : 153 건
+  그중 DB 에 실제로 존재                 : 149 건
+  ★ 유실                                 :   4 건
+
+=== 유실된 sid ===
+  ★ CQUfg9HLH29xvhiu6pVlfWOo  ← 토큰은 발급됐는데 세션이 없다
+  ★ 5gLP4fqmpZBbjhH_d-0TPMMr
+  ★ hkcOv1QskUFmYveMLB6Hljra
+  ★ p5XybeQIYmAs818gO4Vl_5ea
+```
+
+**약 2.6% 유실.** 초당 19건 정도 로그인하던 중이었으므로
+**대략 마지막 0.2초 분량**이다 — `wal_writer_delay` 기본값(200ms)과 맞는다.
+
+### 사용자에게 어떻게 보이는가
+
+```
+   로그인 성공  →  access token + refresh token 을 받음
+        │
+        │  (크래시)
+        ▼
+   다음 요청     →  access token 은 60초간 통한다
+        │            (서명만 보는 경로라면)
+        ▼
+   60초 후 refresh  →  "Session not active"  →  다시 로그인
+```
+
+**즉시 드러나지 않는다.** access token 수명 동안은 정상으로 보이다가
+갱신 시점에 끊긴다. 장애와 증상 사이에 **최대 60초의 시차**가 있다.
+
+---
+
+## 6. 개념
+
+### WAL 과 `synchronous_commit`
+
+```
+   COMMIT
+     │
+     ├─ WAL 버퍼(메모리)에 기록          ← 항상 한다
+     │
+     ├─ synchronous_commit = on  : 디스크 플러시를 기다렸다가 응답
+     └─ synchronous_commit = off : 기다리지 않고 즉시 응답   ← Keycloak
+                                     │
+                                     └─ 크래시 시 이 구간이 사라진다
+```
+
+| 설정 | 응답 속도 | 잃는 것 |
+|---|---|---|
+| `on` (PostgreSQL 기본) | 느리다 (디스크 대기) | 없다 |
+| **`off`** | 빠르다 | **최대 `wal_writer_delay` × 3 분량** |
+
+**전역 설정은 `on` 이었다.**
+
+```
+전역 synchronous_commit: on
+```
+
+**Keycloak 이 자기 트랜잭션에만 `SET LOCAL` 로 끈다.** DBA 가 서버 설정만
+보고 "우리는 동기 커밋"이라 믿으면 틀린다. **애플리케이션이 트랜잭션 단위로
+뒤집을 수 있다.**
+
+### crash recovery
+
+```
+   기동 시 pg_control 을 읽는다
+     └─ "깨끗하게 종료됨" 표시가 없다
+          └─ "database system was not properly shut down"
+               └─ 마지막 체크포인트부터 WAL 을 재생(redo)
+                    └─ 디스크에 안 내려간 커밋은 복구할 수 없다  ← 손실
+```
+
+`redo starts at 0/23CAB68` → `redo done at 0/2529E40` 사이가 재생된 구간이다.
+**WAL 에 없는 것은 재생할 수도 없다.**
+
+### 이 손실이 "허용된" 이유
+
+Keycloak 의 판단은 이렇게 읽힌다.
+
+| | |
+|---|---|
+| 세션 쓰기는 **매우 잦다** | 로그인마다, refresh 마다 |
+| 잃어도 **회복 가능하다** | 사용자가 다시 로그인하면 된다 |
+| 동기 커밋의 비용은 **모든 요청에 붙는다** | 크래시는 드물다 |
+
+**드문 사고의 비용을 상시 지연으로 지불하지 않겠다는 선택**이다.
+합리적이지만, **선택했다는 사실을 알고 있어야 한다.**
+
+---
+
+## 7. 운영에 주는 것
+
+| 알게 된 것 | 함의 |
+|---|---|
+| 로그인도 비동기 커밋 | **RPO 가 0 이 아니다.** 크래시 시 마지막 수백 ms 로그인은 사라진다 |
+| 전역 `on` 인데 세션만 `off` | **서버 설정으로 판단하면 안 된다.** 애플리케이션이 뒤집는다 |
+| 손실이 즉시 안 보인다 | access token 수명만큼 시차. **모니터링은 갱신 실패율을 봐야 한다** |
+| `--grace-period=0` 은 크래시가 아니다 | **장애 훈련이 훈련이 안 될 수 있다** |
+| 컨테이너 안에서 PID 1 을 못 죽인다 | 크래시 재현은 **자식 프로세스**나 **노드에서** |
+
+### 바꿀 수 있는가
+
+```sql
+-- 세션 트랜잭션까지 동기 커밋으로 강제하려면 (지연 대가를 치른다)
+ALTER DATABASE keycloak SET synchronous_commit = on;   -- SET LOCAL 이 이깁니다
+```
+
+**`SET LOCAL` 이 우선하므로 이것으로는 못 막는다.** Keycloak 설정이나
+소스 수준의 문제이며, **RPO 0 이 필요하면 복제(streaming replication)로
+푸는 것이 맞다** — 동기 스탠바이가 있으면 `synchronous_commit` 의 의미가
+달라진다.
+
+---
+
+## 8. 재현 절차 (명령어)
+
+```bash
+# 0. 설계 확인 — 로그인도 비동기 커밋인지 먼저 본다
+kubectl -n keycloak-lab exec deploy/postgres -- psql -U keycloak -d keycloak \
+  -c "alter system set log_statement='all'" -c "select pg_reload_conf()"
+#   → 로그인 1회 후 로그에서 "SET LOCAL synchronous_commit TO OFF" 확인
+
+# 1. 세션 테이블 비우기
+kubectl -n keycloak-lab exec deploy/postgres -- psql -U keycloak -d keycloak \
+  -c "delete from offline_user_session"
+
+# 2. 로그인 루프 (호스트에서 백그라운드 exec — 파드 안 & 는 exec 종료와 함께 죽는다)
+kubectl -n keycloak-lab exec a2-probe -- sh -c '<로그인 반복, sid 를 /tmp/sids 에>' &
+
+# 3. 진짜 크래시 — 백엔드 프로세스에 SIGKILL
+kubectl -n keycloak-lab exec deploy/postgres -- \
+  sh -c 'kill -9 $(pgrep -f "postgres: keycloak keycloak" | head -1)'
+
+# 4. 주입이 걸렸는지 확인 — 이게 없으면 결과를 해석하지 않는다
+kubectl -n keycloak-lab logs deploy/postgres | grep -E "not properly shut down|redo"
+
+# 5. 대조
+kubectl -n keycloak-lab exec deploy/postgres -- psql -U keycloak -d keycloak -tAc \
+  "select count(*) from offline_user_session where user_session_id in (<sid 목록>)"
+```
+
+---
+
+## 9. 다음 실험에 남기는 것
+
+| 실험 | 이 실험이 준 것 |
+|---|---|
+| **D-1** 백업·복구 | RPO 는 **백업 주기 + 이 손실**이다. 둘을 더해야 진짜 RPO |
+| **B-6** Redis 영속화 | `appendfsync everysec` 은 **같은 모양의 트레이드오프** |
+| **A-4** 노드 상실 | 노드가 죽으면 이것도 함께 일어난다 (postgres 가 kc-lab-2) |
+| 전체 | **주입 성공 신호를 미리 정한다.** 여기서는 crash recovery 로그 |

docs/experiment-a4-node-loss.md

@@ -0,0 +1,397 @@
+# A-4 — 노드가 통째로 죽으면 어떻게 되는가
+
+브랜치 `feature/keycloak-a4-node-loss` ·
+증거 [`docs/evidence/a4-node-loss/`](evidence/a4-node-loss/) ·
+2026-09-04 12:07–12:24 KST
+
+선행: [`A-2`](experiment-a2-database-loss.md) · [`A-3`](experiment-a3-database-crash.md)
+
+**두 판본으로 나눠 실행했다.**
+
+| | 죽인 노드 | 성격 |
+|---|---|---|
+| **4a** | `kc-lab-2` (워커) | Keycloak + **DB 동시 상실** |
+| **4b** | `kc-lab-1` (k3s 서버) | **컨트롤 플레인 + 진입점 상실** |
+
+`virsh destroy` 는 종료 신호를 보내지 않는다. **전원을 뽑는 것과 같다.**
+
+---
+
+## 0. 결론부터
+
+| | 4a 워커 상실 | 4b 컨트롤 플레인 상실 |
+|---|---|---|
+| 외부 응답 | **503** | **000** (연결 자체가 안 됨) |
+| `kubectl` | 정상 | **불통** |
+| 살아 있는 워크로드 | keycloak-1 (하지만 DB 없음) | **keycloak-0 은 계속 돌고 있다** |
+| 복구 시간 | **60초** | **60초** |
+
+**둘 다 전면 장애**지만 이유가 다르다. 4a 는 **DB 가 같이 죽어서**, 4b 는
+**들어갈 길이 없어서**다.
+
+그리고 예상하지 못한 것 셋 —
+**죽은 파드가 산 파드보다 건강해 보이고**, **StatefulSet 은 대체 파드를 만들지
+않으며**, **PVC 때문에 재배치가 불가능하다.**
+
+---
+
+## 4a. 워커 노드 상실 (`kc-lab-2`)
+
+### 기준선
+
+```
+kc-lab-1   Ready   control-plane=true
+kc-lab-2   Ready   <none>
+
+keycloak-0   ready=true   kc-lab-2
+keycloak-1   ready=true   kc-lab-1
+postgres     ready=true   kc-lab-2
+
+PVC postgres-data → kc-lab-2        ← 재배치 가능성을 여기서 이미 알 수 있다
+외부 HTTP 200
+```
+
+### 주입
+
+```bash
+virsh destroy kc-lab-2
+```
+
+### 관찰
+
+```
+  +15초  node=Ready     | keycloak-0=Running postgres=Running | 외부 HTTP 000
+  +30초  node=Ready     | keycloak-0=Running postgres=Running | 외부 HTTP 000
+  +45초  node=NotReady  | keycloak-0=Running postgres=Running | 외부 HTTP 503
+  ...
+  +180초 node=NotReady  | keycloak-0=Running postgres=Running | 외부 HTTP 503
+```
+
+### 발견 ① — 쿠버네티스가 알아채는 데 **40초** 걸린다
+
+`+15초`, `+30초` 에도 노드는 여전히 `Ready` 다. **기계는 이미 없는데.**
+
+kube-controller-manager 의 `node-monitor-grace-period` (기본 40초) 동안
+kubelet 의 하트비트가 없어야 `NotReady` 로 바꾼다.
+
+> **그 40초 동안 쿠버네티스는 거짓말을 한다.** 사용자는 이미 장애를 겪고 있다
+> (`000`). **노드 상태를 알림 근거로 삼으면 항상 늦는다.**
+
+### 발견 ② — **죽은 파드가 산 파드보다 건강해 보인다**
+
+```
+POD          PHASE     READY   NODE
+keycloak-0   Running   true    kc-lab-2      ← 기계가 꺼져 있다
+keycloak-1   Running   false   kc-lab-1      ← 살아 있다
+```
+
+**`keycloak-0` 은 `ready=true`, `keycloak-1` 은 `ready=false`.**
+
+| 왜 | |
+|---|---|
+| keycloak-0 | kubelet 이 없어 **상태를 갱신할 수 없다.** 마지막으로 보고한 값이 그대로 얼어 있다 |
+| keycloak-1 | 살아서 **정직하게 보고한다** — DB 가 없으니 readiness 실패 |
+
+> **파드 상태는 "지금 어떤가"가 아니라 "마지막으로 그렇게 들었다"이다.**
+> 노드가 죽으면 그 노드 파드의 상태는 **화석**이 된다.
+
+Prometheus 는 정확했다.
+
+```
+  up{job=keycloak      pod=keycloak-1} = 1
+  up{job=keycloak      pod=keycloak-0} = 0     ← 긁기에 실패했다 = 사실
+  up{job=kubelet                     } = 0
+  up{job=node-exporter node=kc-lab-2 } = 0
+```
+
+**A-2 와 정반대다.** A-2 에서는 `up=1` 인데 서비스가 죽었고, 여기서는 `up=0`
+이 정확했다.
+
+| | `up` 이 잡는가 |
+|---|---|
+| **대상이 사라짐** (노드 상실) | **잡는다** |
+| **대상이 살아서 못 씀** (DB 상실) | **못 잡는다** |
+
+![up 이 노드별로 0 으로 떨어졌다](evidence/a4-node-loss/a4-up-dropped-per-node.png)
+
+### 발견 ③ — 축출은 **5분** 뒤에 시작된다
+
+쿠버네티스가 노드에 taint 를 붙인다.
+
+```
+  node.kubernetes.io/unreachable=:NoSchedule
+  node.kubernetes.io/unreachable=:NoExecute
+```
+
+그런데 모든 파드에는 기본 관용이 붙어 있다.
+
+```
+  node.kubernetes.io/not-ready    NoExecute  tolerationSeconds=300
+  node.kubernetes.io/unreachable  NoExecute  tolerationSeconds=300
+```
+
+```
+  +240초  전부 Running
+  +270초  a2-probe:Terminating keycloak-0:Terminating postgres:Terminating
+          postgres-...-9cmsv:Pending          ← 새 파드가 생겼다
+```
+
+**노드가 죽고 약 5분이 지나야 파드가 축출된다.** 잠깐 끊긴 노드가 돌아올 수도
+있으니 성급하게 옮기지 않겠다는 설계다. 대신 **그동안은 아무 일도 일어나지
+않는다.**
+
+### 발견 ④ — 축출돼도 **갈 곳이 없다**
+
+```
+Warning  FailedScheduling  default-scheduler
+  0/2 nodes are available:
+    1 node(s) didn't match PersistentVolume's node affinity,
+    1 node(s) had untolerated taint(s).
+```
+
+```
+   kc-lab-2  →  taint 때문에 못 간다 (죽은 노드)
+   kc-lab-1  →  PVC 가 kc-lab-2 에 묶여 있어 못 간다
+```
+
+**`local-path` PVC 는 그 노드의 로컬 디스크 디렉터리다.** 노드가 죽으면
+볼륨도 죽는다. 새 파드는 **영원히 Pending** 이다.
+
+> 결함이 아니라 **조건**이다. 이 실험대는 그걸 알고 `local-path` 를 골랐다.
+> 운영이라면 네트워크 스토리지나 DB 복제가 이 자리를 메워야 한다.
+
+### 발견 ⑤ — StatefulSet 은 **대체 파드를 만들지 않는다**
+
+```
+NAME       DESIRED   READY     CURRENT
+keycloak   2         <none>    1
+
+keycloak-0   1/1   Terminating   30m       ← 30분째
+keycloak-1   0/1   Running
+```
+
+**`keycloak-0` 이 `Terminating` 에서 영원히 멈춰 있고, 대체 파드가 안 생긴다.**
+
+| 왜 | |
+|---|---|
+| StatefulSet 의 계약 | **같은 이름의 파드는 클러스터에 하나뿐**이어야 한다 |
+| 컨트롤 플레인이 아는 것 | 노드가 안 보인다 = **파드가 죽었는지 확신할 수 없다** |
+| 그래서 | 옛 파드를 확실히 지우기 전엔 새 `keycloak-0` 을 못 만든다 |
+
+**Deployment 였다면 즉시 새 파드를 만든다.** 이름이 아무래도 되기 때문이다.
+StatefulSet 의 "안정된 이름"이라는 이득의 **반대편 비용**이 여기다.
+
+강제로 진행시키려면 사람이 개입해야 한다.
+
+```bash
+kubectl -n keycloak-lab delete pod keycloak-0 --grace-period=0 --force
+```
+
+**위험한 명령이다.** 노드가 사실은 살아 있고 네트워크만 끊긴 것이라면
+**같은 이름의 파드 둘이 동시에 존재**하게 된다 (split brain).
+
+### 복구
+
+```bash
+virsh start kc-lab-2
+```
+
+```
+  +30초  node=Ready | Running 파드 3 개 | 외부 HTTP 503
+  +60초  node=Ready | Running 파드 3 개 | 외부 HTTP 200
+  → 서비스 복귀
+```
+
+**60초.** 사람 개입 없이 전부 제자리로 돌아왔다.
+
+---
+
+## 4b. 컨트롤 플레인 노드 상실 (`kc-lab-1`)
+
+### 이 노드에 무엇이 있었나 — 그게 곧 영향 범위다
+
+```
+keycloak-lab    keycloak-1
+kube-system     coredns
+kube-system     local-path-provisioner
+kube-system     metrics-server
+kube-system     svclb-traefik
+kube-system     traefik                    ← replicas=1
+observability   grafana
+observability   prometheus
+observability   node-exporter
+```
+
+```
+NAME      REPLICAS   READY
+traefik   1          1        ← 진입점이 한 개다
+```
+
+### 주입과 관찰
+
+```bash
+virsh destroy kc-lab-1
+```
+
+```
+  +20초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server
+  +60초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server
+  +120초 외부 auth=000 grafana=502 | kubectl: Unable to connect to the server
+  +160초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server
+```
+
+### 발견 ⑥ — 워크로드는 살아 있는데 **길이 없다**
+
+살아남은 노드에서 직접 확인했다.
+
+```bash
+ssh kc-lab-2 'sudo crictl ps --name keycloak'
+```
+
+```
+CONTAINER      STATE     NAME       POD          NAMESPACE
+e5f777900b76   Running   keycloak   keycloak-0   keycloak-lab
+```
+
+**`keycloak-0` 은 멀쩡히 돌고 있다.** 컨테이너 런타임(containerd)은 API 서버
+없이도 이미 떠 있는 파드를 계속 굴린다.
+
+```
+   죽은 것:  API 서버 · 스케줄러 · coredns · Traefik · Prometheus · Grafana
+   산 것:    keycloak-0 · postgres · containerd
+   문제:     들어갈 문(Traefik)이 없다
+```
+
+> **컨트롤 플레인 상실은 워크로드 상실이 아니다.**
+> 이미 떠 있는 것은 계속 돈다. **새로 뜨거나 옮기거나 고치는 것이 안 될 뿐.**
+
+### 발견 ⑦ — `000` 과 `503` 의 차이
+
+| 코드 | 뜻 |
+|---|---|
+| **`503`** (4a) | nginx·Traefik 은 살아 있고 **뒤에 보낼 파드가 없다** |
+| **`000`** (4b) | curl 이 응답 자체를 못 받았다 — **Traefik 이 없어 nginx 가 죽은 주소를 기다리다 타임아웃** |
+
+`grafana=502` 가 한 번 찍힌 것이 증거다 — nginx 는 살아서 502 를 만들 수
+있었지만, 대부분은 타임아웃(`--max-time 8`)에 걸렸다.
+
+**두 코드가 다른 층의 고장을 가리킨다.**
+
+### 발견 ⑧ — 관측 시스템이 같이 죽으면 **0 이 아니라 구멍이 남는다**
+
+Prometheus 가 `kc-lab-1` 에 있었다. 그 노드가 죽은 12:18–12:23 구간은
+그래프에 **0 이 아니라 데이터 없음**으로 남는다.
+
+```
+   대상이 죽음        →  up = 0        →  "언제 죽었는지" 알 수 있다
+   관측자가 죽음      →  데이터 없음    →  "그때 무슨 일이 있었는지" 모른다
+```
+
+**계획 단계에서 "관측 스택은 죽이지 않는 노드에 둔다"고 정했던 이유**를
+실제로 확인한 셈이다. 다만 노드가 둘뿐이라 완전히 피할 수는 없다.
+
+### 복구
+
+```bash
+virsh start kc-lab-1
+```
+
+```
+  +30초  외부=502 | kc-lab-1=Ready kc-lab-2=Ready
+  +60초  외부=200 | kc-lab-1=Ready kc-lab-2=Ready
+  → 서비스 복귀 (총 60초)
+```
+
+**Grafana 는 로그인 세션을 잃었다** — 데이터가 `emptyDir` 이라 파드 재시작에
+사라진다. Prometheus 는 PVC 라 지표가 남았다. **의도한 설계대로 동작했다.**
+
+---
+
+## 5. 개념
+
+### `node-monitor-grace-period` 와 `tolerationSeconds`
+
+```
+   기계 정지
+      │
+      │  40초   node-monitor-grace-period      → 노드 NotReady
+      │
+      │  +300초 tolerationSeconds (NoExecute)  → 파드 축출 시작
+      │
+      ▼
+   총 약 5분 40초 동안 쿠버네티스는 아무것도 하지 않는다
+```
+
+**빠른 장애 대응이 필요하면 이 값을 줄여야 하지만**, 짧게 하면 일시적
+네트워크 끊김에도 파드를 옮기게 되어 **불안정해진다.** 맞바꿈이다.
+
+### `Terminating` 이 안 끝나는 이유
+
+```
+   파드 삭제 요청
+     └─ kubelet 이 컨테이너를 멈추고 "지웠다"고 보고해야 끝난다
+          └─ kubelet 이 없다 → 보고가 없다 → 영원히 Terminating
+```
+
+`--grace-period=0 --force` 는 **보고를 기다리지 않고 API 에서 지우는 것**이다.
+컨테이너가 실제로 죽었는지는 **모른 채** 진행한다.
+
+### StatefulSet vs Deployment — 노드 상실에서
+
+| | Deployment | StatefulSet |
+|---|---|---|
+| 대체 파드 | **즉시 만든다** (이름 무관) | **안 만든다** (이름이 유일해야) |
+| 노드 상실 시 | 자동 회복 | **사람이 강제 삭제해야** |
+| 위험 | 없음 | 강제 삭제 시 **중복 실행 가능** |
+
+**A-0 에서 "로그 대조가 쉬워서" StatefulSet 을 골랐는데, 그 대가가 여기 있다.**
+
+### `local-path` PVC 와 재배치
+
+```
+   PVC → PV → nodeAffinity: kubernetes.io/hostname = kc-lab-2
+                              └─ 그 노드의 /var/lib/rancher/k3s/storage/... 디렉터리
+```
+
+**볼륨이 노드에 못박히면 파드도 못박힌다.** 노드가 죽으면 데이터도 함께
+접근 불가가 된다.
+
+---
+
+## 6. 재현 절차 (명령어)
+
+```bash
+export LIBVIRT_DEFAULT_URI=qemu:///system
+
+# 사전 — PVC 가 어디 묶여 있는지 먼저 본다 (재배치 가능성이 여기서 정해진다)
+kubectl -n keycloak-lab get pvc -o name | while read p; do
+  V=$(kubectl -n keycloak-lab get $p -o jsonpath='{.spec.volumeName}')
+  kubectl get pv $V -o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]}'
+done
+
+# 4a 워커 상실
+virsh destroy kc-lab-2
+kubectl get node kc-lab-2                        # 40초 뒤 NotReady
+kubectl -n keycloak-lab get pods -o wide         # Running 인 채로 얼어 있다
+kubectl get node kc-lab-2 -o jsonpath='{.spec.taints}'
+#   5분 뒤 Terminating + 새 파드 Pending
+kubectl -n keycloak-lab describe pod <new-pod> | grep -A4 Events
+
+# 4b 컨트롤 플레인 상실 — kubectl 이 죽으므로 노드에서 직접 본다
+virsh destroy kc-lab-1
+ssh kc-lab-2 'sudo crictl ps'                    # 워크로드는 살아 있다
+
+# 복구
+virsh start kc-lab-2 && virsh start kc-lab-1
+```
+
+---
+
+## 7. 다음 실험에 남기는 것
+
+| 실험 | 이 실험이 준 것 |
+|---|---|
+| **A-5** 비대칭 파티션 | 여기서는 노드가 **완전히** 사라졌다. 부분 단절은 더 고약하다 |
+| **D-1** 백업·복구 | **PVC 가 노드에 묶여 있다** — 노드가 영영 안 돌아오면 백업이 유일한 길 |
+| 구성 개선 | **Traefik `replicas=1` 은 진입점 단일 장애점** — 2로 늘리거나 DaemonSet 으로 |
+| 구성 개선 | **Grafana 가 `emptyDir`** — 재시작마다 로그인 세션이 사라진다 |

docs/experiment-a5-asymmetric-partition.md

@@ -0,0 +1,319 @@
+# A-5 — 한쪽 방향만 끊기면 어떻게 되는가 (split brain)
+
+브랜치 `feature/keycloak-a5-asymmetric-partition` ·
+증거 [`docs/evidence/a5-asymmetric-partition/`](evidence/a5-asymmetric-partition/) ·
+2026-09-04 12:28–12:46 KST
+
+선행: [`A-1`](experiment-a1-jgroups-transport-block.md) — 이 실험은 A-1 이
+열어둔 질문에 답한다.
+
+> **A-1 의 열린 질문** — *"양쪽이 동시에 NotReady 가 되는 경로가 있다면
+> 전면 장애다. A-5 에서 이어서 본다."*
+
+---
+
+## 0. 결론부터
+
+| 물음 | 답 |
+|---|---|
+| 비대칭(한 방향) 차단이 클러스터를 가르는가 | **아니다.** 열린 방향으로 재연결한다 |
+| 양방향 완전 차단은 가르는가 | **그렇다.** 양쪽 모두 멤버 1개 |
+| **그때 양쪽이 다 NotReady 가 되는가** | **아니다. 한쪽만 DOWN 이다** |
+| 그래서 서비스는 | **계속된다.** 외부 200 유지 |
+
+**전면 장애 경로는 없었다.** 코디네이터 쪽이 항상 살아남는다.
+
+그리고 주입을 **세 번 실패**했다. 세 번 모두 다른 이유였고, 셋 다
+**"아무 일도 없었다"로 보였다.**
+
+---
+
+## 1. 주입을 세 번 실패했다
+
+A-1 에서 NetworkPolicy 가 conntrack 의 `ESTABLISHED` 승인에 막혀 기존 연결을
+못 끊는다는 것을 배웠다. 그래서 이번에는 iptables 로 갔는데, **또 다른 벽이
+세 개** 있었다.
+
+### 실패 ① — `iptables -I FORWARD 1` 은 최상단에 남지 않는다
+
+```bash
+ssh kc-lab-2 'sudo iptables -I FORWARD 1 -p tcp -d 10.42.1.77 --dport 7800 -j DROP'
+```
+
+```
+num   pkts bytes target
+1      232  377K KUBE-ROUTER-FORWARD  /* kube-router netpol */   ← 다시 1번이 되었다
+2        0     0 DROP  tcp dpt:57800
+3        0     0 DROP  tcp dpt:7800                              ← 0 패킷
+```
+
+**kube-router 가 주기적으로 자기 체인을 최상단에 다시 삽입한다.**
+내가 1번에 넣어도 곧 2번, 3번으로 밀려난다.
+
+> **직접 넣은 iptables 규칙은 CNI 가 관리하는 체인과 경쟁한다.**
+> 넣는 것으로 끝이 아니라 **패킷 카운터로 확인해야 한다.**
+
+### 실패 ② — 그래도 0 패킷: **연결 방향을 잘못 짚었다**
+
+`raw` 테이블로 옮겼는데도 0 패킷이었다.
+
+```bash
+ssh kc-lab-2 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d 10.42.1.77 --dport 7800 -j DROP'
+```
+
+실제 연결을 봤더니
+
+```
+ESTABLISHED src=10.42.1.77 dst=10.42.0.42 sport=60485 dport=7800
+            ────────────            ────────────────────
+            keycloak-0 가 클라이언트   keycloak-1 이 서버
+```
+
+**A-1 때와 방향이 반대였다.** 파드가 재시작되면서 누가 먼저 연결을 걸었는지가
+바뀌었다.
+
+> **JGroups 의 TCP 연결 방향은 고정이 아니다.** 먼저 뜬 쪽, 먼저 JOIN 을 건
+> 쪽에 따라 달라진다. **가정하지 말고 `conntrack -L` 로 봐야 한다.**
+
+### 성공 — `raw` 테이블 PREROUTING, 수신측 노드에
+
+```bash
+ssh kc-lab-1 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d 10.42.0.42 --dport 7800  -j DROP
+              sudo iptables -t raw -I PREROUTING 1 -p tcp -d 10.42.0.42 --dport 57800 -j DROP'
+```
+
+```
+   pkts bytes target
+     19  1096 DROP  tcp dpt:57800
+     21  3058 DROP  tcp dpt:7800     ← 걸린다
+```
+
+### 개념 — netfilter 처리 순서
+
+```
+   패킷 도착
+     │
+     ├─▶ raw     PREROUTING     ← conntrack 보다 먼저. NOTRACK·DROP 용
+     │
+     ├─▶ conntrack 조회/생성    ← 여기서 ESTABLISHED 가 결정된다
+     │
+     ├─▶ mangle  PREROUTING
+     ├─▶ nat     PREROUTING
+     ├─▶ filter  FORWARD        ← NetworkPolicy·kube-router 가 여기 있다
+     └─▶ 목적지 파드
+```
+
+| 어디에 넣는가 | 기존 연결을 끊는가 | CNI 와 경쟁하는가 |
+|---|---|---|
+| NetworkPolicy (filter) | **못 끊는다** — conntrack 이 먼저 통과시킨다 | 없음 |
+| filter FORWARD 직접 | 순서에 따라 | **경쟁한다** (kube-router 가 밀어낸다) |
+| **raw PREROUTING** | **끊는다** | **없다** — CNI 가 안 쓰는 테이블 |
+
+**진짜 네트워크 분단을 흉내내려면 `raw` 테이블이 맞다.**
+
+---
+
+## 2. 비대칭 차단 — 클러스터가 갈라지지 않았다
+
+한 방향(`→ keycloak-1:7800`)만 막고 관찰했다.
+
+```
+  +25초  keycloak-0:1/1 keycloak-1:1/1 | 외부 200
+  +75초  keycloak-0:1/1 keycloak-1:0/1 | 외부 200      ← 잠깐 흔들림
+  +100초 keycloak-0:1/1 keycloak-1:1/1 | 외부 200      ← 회복
+  ...
+  +200초 keycloak-0:1/1 keycloak-1:1/1 | 외부 200
+```
+
+**주입 이후 뷰 변화가 하나도 없었다.**
+
+```bash
+kubectl -n keycloak-lab logs keycloak-0 --since=20m | grep ISPN000094 | awk '$2 >= "03:33:58"'
+# (아무것도 안 나온다)
+```
+
+```
+  뷰 ID 이력:  |13] (2)  ← 주입 전부터 지금까지 그대로
+  suspected  = 0
+```
+
+### 왜 안 갈라졌는가 — **연결 방향이 뒤집혔다**
+
+```
+차단 전:  keycloak-0 → keycloak-1:7800     ← 내가 막은 방향
+차단 후:  keycloak-1 → keycloak-0:7800     ← 열린 방향으로 다시 붙었다
+```
+
+```
+ESTABLISHED src=10.42.0.42 dst=10.42.1.77 sport=48473 dport=7800
+            ────────────            ─────────────
+            keycloak-1 이 클라이언트   keycloak-0 이 서버
+```
+
+**JGroups 는 막힌 연결이 죽자 반대 방향으로 새로 연결했고, 그 사이 FD_SOCK2 가
+상대를 의심하기 전에 복구가 끝났다.** `suspected = 0` 이 그 증거다.
+
+> **한 방향만 막는 것으로는 JGroups 를 가를 수 없다.**
+> 두 노드는 서로에게 연결을 걸 수 있으므로, **한쪽 길이 막히면 다른 길로 간다.**
+>
+> 운영적으로는 좋은 소식이다 — 단방향 방화벽 오설정은 **자가 치유**된다.
+> 반대로 **분단을 재현하려는 실험자에게는 함정**이다.
+
+---
+
+## 3. 양방향 차단 — 갈라졌지만 전면 장애는 아니다
+
+양쪽 노드 모두에 규칙을 넣었다.
+
+```
+  +75초   keycloak-0:1/1 keycloak-1:1/1 | ready=[10.42.0.42 10.42.1.77] 외부 200
+  +100초  keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77]            외부 200
+  ...
+  +225초  keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77]            외부 200
+```
+
+```
+=== 뷰 ===
+  keycloak-0  [keycloak-0-24309|14] (1) [keycloak-0-24309]
+  keycloak-1  [keycloak-1-45480|14] (1) [keycloak-1-45480]
+
+=== JGROUPS_PING ===
+ keycloak-0-24309 | 10.42.1.77:7800 | t     ← 코디네이터
+ keycloak-1-45480 | 10.42.0.42:7800 | t     ← 코디네이터
+```
+
+**`coord = t` 가 둘.** 완전한 split brain 이다.
+
+![cluster_size 가 갈라졌다 합쳐진다](evidence/a5-asymmetric-partition/a5-cluster-size-bidirectional-block.png)
+
+### 그런데 **한쪽만 DOWN 이다**
+
+```
+--- keycloak-0 ---
+{"status":"UP",   ... {"name":"Keycloak cluster health check","status":"UP"}
+
+--- keycloak-1 ---
+{"status":"DOWN", ... (cluster health check 가 DOWN)
+```
+
+| | keycloak-0 | keycloak-1 |
+|---|---|---|
+| 분단 전 역할 | **코디네이터** (뷰 13 의 발행자) | 일반 멤버 |
+| 분단 후 자기 인식 | "멤버가 하나 나갔다" — **정상 사건** | "코디네이터를 잃었다" — **비정상** |
+| 헬스체크 | **UP** | **DOWN** |
+| Service 엔드포인트 | **남는다** | 빠진다 |
+
+> **A-1 의 열린 질문에 대한 답 — 양쪽이 동시에 NotReady 가 되는 경로는 없었다.**
+>
+> Keycloak 의 클러스터 헬스체크는 **비대칭**이다. 코디네이터였던 쪽은
+> 자기가 정상이라고 보고, 잃은 쪽만 DOWN 이 된다. 그래서 완전 분단조차
+> **용량 저하로 끝나고 전면 장애가 되지 않는다.**
+>
+> A-2(DB 상실)에서 양쪽이 동시에 DOWN 이 된 것과 대조된다. **DB 는 모두가
+> 의존하는 하나지만, 클러스터 멤버십은 서로 상대적**이기 때문이다.
+
+---
+
+## 4. 복구
+
+```bash
+ssh kc-lab-1 'sudo iptables -t raw -F PREROUTING'
+ssh kc-lab-2 'sudo iptables -t raw -F PREROUTING'
+```
+
+```
+  +25초  keycloak-0:1/1 keycloak-1:0/1
+  +50초  keycloak-0:1/1 keycloak-1:1/1
+  → 복구 완료
+
+MergeView::[keycloak-0-24309|15] (2) [keycloak-0-24309, keycloak-1-45480]
+```
+
+**50초, 사람 개입 없음.** `MergeView` 로 뷰 15 가 발행되며 병합됐다.
+
+각 캐시마다 재분배 로그가 남는다.
+
+```
+[Context=work]            ISPN100007: After merge (or coordinator change) ...
+[Context=clientSessions]  ISPN100007: After merge ...
+[Context=offlineSessions] ISPN100007: After merge ...
+[Context=loginFailures]   ISPN100007: After merge ...
+[Context=actionTokens]    ISPN100007: After merge ...
+```
+
+**`ISPN100007` 은 병합 후 캐시별 토폴로지 재계산**이다. 캐시가 여럿이므로
+로그도 캐시 수만큼 나온다.
+
+---
+
+## 5. 개념
+
+### `MergeView` 와 뷰 ID
+
+```
+[keycloak-0-24309|13] (2)   ← 정상
+[keycloak-0-24309|14] (1)   ← 분단. 양쪽이 각자 14 를 발행
+MergeView::[...|15] (2)     ← 병합. 뷰 ID 는 계속 증가한다
+```
+
+**뷰 ID 는 단조 증가**하므로 "언제 몇 번 갈라졌는지"를 로그만으로 셀 수 있다.
+
+### 코디네이터의 비대칭성
+
+JGroups 코디네이터는 **가장 오래된 멤버**다. 분단이 나면
+
+```
+   코디네이터 쪽:  "멤버가 나갔다"        → 정상 처리, 계속 코디네이터
+   나머지 쪽:      "코디네이터가 사라졌다"  → 새 코디네이터를 자기로 선출
+```
+
+**둘 다 자기가 코디네이터라고 믿는 상태**가 split brain 이고,
+`JGROUPS_PING.coord` 컬럼에 그대로 드러난다.
+
+### 실험자를 위한 규칙
+
+| 상황 | 확인 방법 |
+|---|---|
+| 규칙을 넣었는데 안 걸린다 | `iptables -L -n -v` 의 **패킷 카운터** |
+| 방향을 모르겠다 | `conntrack -L \| grep <포트>` |
+| CNI 가 밀어낸다 | `raw` 테이블을 쓴다 |
+| 갈라졌는지 알고 싶다 | `vendor_cluster_size`, `ISPN000094`, `JGROUPS_PING.coord` |
+
+---
+
+## 6. 재현 절차 (명령어)
+
+```bash
+# 1. 지금 연결이 어느 방향인지 먼저 본다 — 가정하면 실패한다
+ssh kc-lab-1 'sudo conntrack -L | grep 7800'
+
+# 2. 수신측 노드의 raw PREROUTING 에 넣는다 (filter 는 CNI 와 경쟁한다)
+ssh kc-lab-1 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d <수신 파드IP> --dport 7800  -j DROP'
+ssh kc-lab-1 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d <수신 파드IP> --dport 57800 -j DROP'
+
+# 3. 걸렸는지 카운터로 확인 — 0 이면 해석 금지
+ssh kc-lab-1 'sudo iptables -t raw -L PREROUTING -n -v'
+
+# 4. 양방향으로 하려면 반대 노드에도 (한 방향만으로는 자가 치유된다)
+ssh kc-lab-2 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d <반대 파드IP> --dport 7800 -j DROP'
+
+# 5. 분단 확인
+kubectl -n keycloak-lab exec deploy/postgres -- psql -U keycloak -d keycloak \
+  -c "select name, coord from jgroups_ping"          # coord=t 가 둘이면 split brain
+kubectl -n keycloak-lab get endpoints keycloak -o jsonpath='{.subsets[*].addresses[*].ip}'
+
+# 6. 해제
+ssh kc-lab-1 'sudo iptables -t raw -F PREROUTING'
+ssh kc-lab-2 'sudo iptables -t raw -F PREROUTING'
+```
+
+---
+
+## 7. 다음 실험에 남기는 것
+
+| 실험 | 이 실험이 준 것 |
+|---|---|
+| **A-6** 지연 주입 | `tc netem` 도 같은 함정 — **주입이 걸렸는지 먼저 확인** |
+| **A-7** volatile 비교 | 여기서는 분단에도 서비스가 계속됐다. volatile 이면 **세션이 갈라진다** |
+| 운영 | **단방향 방화벽 오설정은 자가 치유된다.** 양방향이어야 사고가 된다 |
+| 운영 | **완전 분단조차 전면 장애가 아니다** — 코디네이터 쪽이 살아남는다 |

docs/experiment-a6-latency-injection.md

@@ -0,0 +1,338 @@
+# A-6 — 끊기지 않고 느려지기만 하면 어떻게 되는가
+
+브랜치 `feature/keycloak-a6-latency-injection` ·
+증거 [`docs/evidence/a6-latency-injection/`](evidence/a6-latency-injection/) ·
+2026-09-04 13:10–13:35 KST
+
+**실제 장애의 대부분은 완전 사망이 아니라 느려짐이다.** 그리고 느려짐은
+사망보다 진단하기 어렵다 — 헬스체크가 통과하기 때문이다.
+
+---
+
+## 0. 결론부터
+
+| 측정 | 값 |
+|---|---|
+| 주입한 네트워크 지연 | **200 ms** |
+| 로그인 응답 시간 | **66 ms → 1,872 ms (28배)** |
+| 동시 20건에서 최대 응답 | **22.2 초** |
+| 커넥션 획득 대기 최대 | **20,000 ms** |
+| readiness 프로브 | **타임아웃으로 실패** |
+
+**200밀리초가 22초가 됐다.** 지연은 **왕복 횟수만큼 곱해지고**, 커넥션 풀에서
+**한 번 더 곱해진다.**
+
+---
+
+## 1. 설계 — 한 실험에서 A/B 가 되는 배치
+
+```
+   kc-lab-2                        kc-lab-1
+   ┌──────────────────┐            ┌──────────────────┐
+   │ postgres         │            │ keycloak-1       │
+   │ keycloak-0       │            │                  │
+   │   └─ cni0 로 직행 │◀─ VXLAN ──▶│   └─ 오버레이 경유 │
+   └──────────────────┘            └──────────────────┘
+        지연 없음                        여기만 느려진다
+```
+
+**postgres 가 보내는 패킷만** 지연시키면 `keycloak-1` 의 DB 접근만 느려지고
+`keycloak-0` 은 그대로다. **대조군이 같은 실험 안에 있다.**
+
+기준선은 거의 같았다.
+
+```
+  keycloak-0  평균 70 ms
+  keycloak-1  평균 66 ms
+```
+
+---
+
+## 2. 주입을 두 번 실패했다
+
+### 실패 ① — `eth0` 이라는 인터페이스가 없다
+
+```bash
+ssh kc-lab-2 'sudo tc qdisc add dev eth0 root handle 1: prio'
+```
+```
+Cannot find device "eth0"
+```
+
+Debian 클라우드 이미지는 **예측 가능한 인터페이스 이름**을 쓴다.
+
+```
+enp1s0    UP    52:54:00:aa:bb:12
+```
+
+`en` (ethernet) + `p1` (PCI bus 1) + `s0` (slot 0). 이름이 하드웨어 위치에서
+나오므로 **NIC 순서가 바뀌어도 이름이 안 바뀐다.**
+
+### 실패 ② — `enp1s0` 에서는 **파드 IP 가 보이지 않는다**
+
+여기가 핵심이다. 노드 간 파드 통신은 **flannel VXLAN 으로 캡슐화**된다.
+
+```
+   원래 패킷:   src=10.42.1.76(postgres)  dst=10.42.0.42(keycloak-1)
+        │
+        ▼  flannel.1 에서 캡슐화
+   실제 패킷:   src=192.168.122.12(노드)   dst=192.168.122.11(노드)  UDP 8472
+                └─ 안쪽에 원래 패킷이 통째로 들어 있다
+        │
+        ▼
+   enp1s0 로 나간다
+```
+
+**`enp1s0` 에서 `match ip src 10.42.1.76` 은 절대 일치하지 않는다.**
+그 IP 는 페이로드 안에 있고, 헤더에는 노드 IP 만 있다.
+
+### 성공 — `flannel.1` 에 건다
+
+```bash
+ssh kc-lab-2 '
+  sudo tc qdisc add dev flannel.1 root handle 1: prio
+  sudo tc qdisc add dev flannel.1 parent 1:3 handle 30: netem delay 200ms
+  sudo tc filter add dev flannel.1 protocol ip parent 1:0 prio 3 \
+       u32 match ip src 10.42.1.76/32 flowid 1:3'
+```
+
+**`flannel.1` 은 캡슐화 직전 단계**이므로 여기서는 파드 IP 가 보인다.
+
+### 검증 — 카운터로 확인한다
+
+```
+qdisc netem 30: parent 1:3 limit 1000 delay 200ms
+ Sent 18388 bytes 150 pkt (dropped 0, overlimits 0 requeues 0)
+                  ───────
+                  실제로 지연 밴드를 통과했다
+```
+
+> **A-1·A-5 와 같은 교훈이 세 번째로 나왔다.**
+> 주입을 넣은 것과 걸린 것은 다르다. **카운터를 봐야 한다.**
+
+### 개념 — `tc` 의 계층 구조
+
+```
+   qdisc (큐 규율)     인터페이스에 붙는 패킷 스케줄러
+     ├─ prio           우선순위 밴드 3개로 나눈다
+     │    ├─ 1:1 (기본)
+     │    ├─ 1:2 (기본)
+     │    └─ 1:3  ← 여기에 netem 을 붙인다
+     └─ filter         어떤 패킷을 어느 밴드로 보낼지
+```
+
+`netem` 을 root 에 바로 붙이면 **모든 트래픽**이 느려진다.
+`prio` + `filter` 를 쓰면 **고른 트래픽만** 느려진다.
+
+---
+
+## 3. 결과 — 지연은 곱해진다
+
+```
+=== 두 노드 지연 비교 (기준선: k0=70ms k1=66ms) ===
+  keycloak-0  평균   41 ms   최대   57 ms      ← 영향 없음
+  keycloak-1  평균 1872 ms   최대 1887 ms      ← 28배
+```
+
+### 왜 200ms 가 1,872ms 가 되는가
+
+A-0 에서 잡은 로그인 트랜잭션의 SQL 이 답이다.
+
+```
+BEGIN
+select ... from OFFLINE_USER_SESSION ...
+select VERSION ... for no key update skip locked
+select ... from OFFLINE_CLIENT_SESSION ...
+select VERSION ... for no key update skip locked
+insert into OFFLINE_USER_SESSION ...
+insert into OFFLINE_CLIENT_SESSION ...
+SET LOCAL synchronous_commit TO OFF
+COMMIT
+```
+
+**왕복이 아홉 번이다.**
+
+```
+   200 ms × 9 왕복 ≈ 1,800 ms      실측 1,872 ms
+```
+
+> **네트워크 지연은 왕복 횟수만큼 증폭된다.**
+> "DB 가 200ms 느려졌다"는 "애플리케이션이 200ms 느려졌다"가 아니다.
+> **쿼리 수를 줄이는 것이 지연 환경에서 결정적인 이유**가 이것이다.
+
+---
+
+## 4. 동시 부하 — 여기서 진짜 고장이 난다
+
+동시 20건을 `keycloak-1` 에 보냈다.
+
+```
+  200  1.911    200  1.913    200  1.958    200  1.981
+  200  3.441    200  4.841    200  6.257    200  7.704
+  200  9.104    200 10.539    200 11.951    200 13.351
+  200 14.785    200 16.189    200 17.625    200 19.053
+  200 20.495    200 21.905    200 22.228    200 22.230
+```
+
+**전부 성공(200)했지만 응답 시간이 1.9초에서 22.2초까지 계단으로 늘어난다.**
+
+```
+   1.9 → 3.4 → 4.8 → 6.2 → 7.7 → 9.1 → 10.5 → ...
+        ────  ────  ────  ────
+        약 1.4초 간격 — 앞 요청이 커넥션을 놓아줄 때까지 줄을 선다
+```
+
+**전형적인 큐잉이다.** 커넥션 수는 유한하고, 각 요청이 커넥션을 1.9초씩
+붙잡고 있으므로 뒤에 온 요청은 그만큼 기다린다.
+
+### 커넥션 풀 지표가 증언한다
+
+```
+  agroal_blocking_time_max_milliseconds   20000.0    ← 20초를 기다린 요청이 있다
+  agroal_blocking_time_average_milliseconds  281.0
+  agroal_max_used_count                      19.0    ← 풀이 19개까지 늘었다
+  agroal_acquire_count_total                672.0
+  agroal_active_count                         0.0    ← 부하가 끝나 지금은 0
+```
+
+![커넥션 획득 대기 시간](evidence/a6-latency-injection/a6-connection-pool-blocking.png)
+
+### 그리고 헬스체크가 무너진다
+
+```
+Warning  Unhealthy  pod/keycloak-1  Readiness probe failed:
+  Get "http://10.42.0.42:9000/health/ready": context deadline exceeded
+```
+
+**readiness 프로브 자체가 타임아웃됐다.**
+
+### 연쇄 고장의 모양
+
+```
+   DB 가 느려진다
+        ↓
+   요청이 커넥션을 오래 붙잡는다
+        ↓
+   커넥션 풀이 고갈된다
+        ↓
+   새 요청이 줄을 선다 (최대 20초)
+        ↓
+   헬스체크도 줄에 선다 → 타임아웃 → NotReady
+        ↓
+   그 노드가 로드밸런서에서 빠진다
+        ↓
+   ★ 남은 노드로 트래픽이 몰린다 → 그 노드도 같은 길을 간다
+```
+
+**마지막 화살표가 무서운 부분이다.** 느려짐은 **전파된다.**
+A-2(DB 완전 정지)는 즉시 503 으로 드러나 오히려 명확했지만,
+**느려짐은 살아 있는 노드를 하나씩 무너뜨린다.**
+
+---
+
+## 5. 빗나간 예측 — 낙관적 락 충돌은 늘지 않았다
+
+계획서에 이렇게 적었다.
+
+> **낙관적 락 충돌 증가** — 트랜잭션이 길어져 `VERSION` 충돌이 늘어야 한다
+
+```
+  관련 로그 줄수: 0
+```
+
+**하나도 없었다.** 이유가 명확하다.
+
+```
+   로그인   →  매번 새 세션 행을 INSERT      →  다툴 상대가 없다
+   refresh  →  같은 세션 행을 UPDATE         →  여기서 다툰다
+```
+
+**충돌은 같은 행을 동시에 고칠 때만 일어난다.** 로그인 부하로는 재현되지
+않는다. 이건 **B-3(refresh 토큰 경쟁)의 영역**이며, 거기서 지연을 함께 주면
+충돌률이 올라갈 것이다.
+
+> 예측을 적어두지 않았다면 "충돌이 없네" 하고 넘어갔을 것이다.
+> **빗나간 예측이 다음 실험의 설계를 정해준다.**
+
+---
+
+## 6. 복구
+
+```bash
+ssh kc-lab-2 'sudo tc qdisc del dev flannel.1 root'
+```
+
+```
+  keycloak-0  평균 43 ms
+  keycloak-1  평균 51 ms      ← 즉시 정상
+```
+
+**파드 재시작 없이 즉시 회복.** 커넥션 풀도 스스로 정상화됐다.
+
+---
+
+## 7. 운영에 주는 것
+
+| 알게 된 것 | 함의 |
+|---|---|
+| 지연은 **왕복 횟수만큼 곱해진다** | DB 지연 대책은 "쿼리 수 줄이기"가 먼저다 |
+| 커넥션 풀에서 **한 번 더 곱해진다** | 풀 크기와 타임아웃이 장애 반경을 정한다 |
+| **헬스체크도 줄에 선다** | 프로브 타임아웃이 풀 대기보다 짧아야 격리가 제때 된다 |
+| 느려짐은 **전파된다** | 노드를 빼면 남은 노드가 더 빨리 무너진다 |
+| `up` 도 readiness 도 **늦게 반응** | **응답 시간 분포(p95/p99)를 봐야 한다** |
+
+### 이 실험대에 없는 알림
+
+지금 관측 스택에는 **지연 분포 지표가 없다.** `agroal_blocking_time_*` 은
+있지만 히스토그램이 아니라 평균/최대뿐이다.
+
+```promql
+# 있으면 좋았을 것
+histogram_quantile(0.99, rate(http_server_requests_seconds_bucket[5m]))
+```
+
+**A-2 에서 `kube-state-metrics` 가 빠진 것을 찾았고, 여기서는 응답 시간
+히스토그램이 빠진 것을 찾았다.** 둘 다 보완 항목이다.
+
+---
+
+## 8. 재현 절차 (명령어)
+
+```bash
+# 1. 인터페이스 이름 확인 — eth0 이 아니다
+ssh kc-lab-2 'ip -brief link show | grep -E "flannel|cni|enp"'
+
+# 2. 오버레이 인터페이스에 건다 (enp1s0 에서는 파드 IP 가 안 보인다)
+ssh kc-lab-2 '
+  sudo tc qdisc add dev flannel.1 root handle 1: prio
+  sudo tc qdisc add dev flannel.1 parent 1:3 handle 30: netem delay 200ms
+  sudo tc filter add dev flannel.1 protocol ip parent 1:0 prio 3 \
+       u32 match ip src <postgres 파드IP>/32 flowid 1:3'
+
+# 3. 걸렸는지 카운터로 확인 — Sent 가 0 이면 해석 금지
+ssh kc-lab-2 'sudo tc -s qdisc show dev flannel.1 | grep -A2 netem'
+
+# 4. 단일 요청 지연 (대조군은 같은 노드의 keycloak-0)
+kubectl -n keycloak-lab run t --rm -i --restart=Never --image=curlimages/curl:8.11.1 \
+  --command -- curl -s -o /dev/null -w '%{time_total}\n' -X POST http://<pod>:8080/realms/master/protocol/openid-connect/token ...
+
+# 5. 동시 부하로 풀 고갈 재현
+#    ( curl ... ) & 를 20개 띄우고 wait
+
+# 6. 풀 지표
+curl -s http://<pod>:9000/metrics | grep -E '^agroal_(blocking_time|max_used|awaiting)'
+
+# 7. 해제
+ssh kc-lab-2 'sudo tc qdisc del dev flannel.1 root'
+```
+
+---
+
+## 9. 다음 실험에 남기는 것
+
+| 실험 | 이 실험이 준 것 |
+|---|---|
+| **B-3** refresh 경쟁 | **지연을 함께 주면 낙관적 락 충돌이 재현될 것** — 여기서는 안 됐다 |
+| **B-1** 저장소 지연 (Q3 제약) | 같은 기법을 Redis 앞에 쓴다 |
+| 관측 보완 | **응답 시간 히스토그램**이 없다 |
+| 구성 | **프로브 타임아웃 < 커넥션 풀 대기**여야 격리가 제때 된다 |