docs: A-4 — a dead pod reports healthier than a live one

Kubernetes keeps calling the node Ready for forty seconds while users already see failures, and the pod on the powered-off machine stays ready=true because its kubelet can no longer contradict itself. Eviction waits another five minutes, then the StatefulSet refuses to recreate its pod and the replacement Deployment pod cannot schedule because the local-path volume is pinned to the dead node.

Killing the server node instead shows the opposite shape: containerd keeps the workload running while the API server, Traefik and the observability stack disappear, so the outage is the missing path rather than the missing application. Traefik at one replica is the ingress single point of failure.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.playwright-mcp/console-2026-09-04T02-56-16-987Z.log

@@ -14,3 +14,116 @@
 [   75046ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   75890ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
 [   96067ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  113467ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  121972ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  139681ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  150971ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  161211ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  180744ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  191401ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  210668ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  229082ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  239345ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  252119ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  266045ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  276705ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  289911ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  296875ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  315099ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  325431ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  328418ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  343774ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  347054ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  352405ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  361291ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  370639ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  374286ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  377413ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  394362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  405423ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  425476ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  434299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  441775ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  450786ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  456829ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  468812ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  478913ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  491241ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  495228ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  502295ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  511625ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  524619ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  533050ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  553189ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  571699ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  575413ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  588005ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  599161ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  608816ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  619031ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  626612ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  639003ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  655077ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  674532ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  677197ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  696027ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  709512ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  727505ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  747139ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  754565ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  773682ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  784674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  787399ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  802845ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  825589ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  832768ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  842043ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  852717ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  871748ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  876571ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  894491ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  895515ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  904310ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  916681ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  934752ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  941175ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  946004ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  960849ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  962180ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  984446ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  997579ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1010105ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1027190ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1043388ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1061311ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1066040ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1080349ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1092336ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1104312ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1109362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1127047ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1134728ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1154899ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1161253ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1174971ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1184491ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1185340ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1189111ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1196984ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1204794ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1247961ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1286322ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1319329ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1362771ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1396881ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1434184ms] [ERROR] Failed to load resource: the server responded with a status of 502 () @ https://app2.hyeonworks.com/api/user/auth-tokens/rotate:0
+[ 1443506ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1465016ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1513337ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1556906ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1562963ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1572896ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1576390ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 503 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1593794ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1600651ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[ 1616424ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362

.playwright-mcp/console-2026-09-04T03-23-15-961Z.log

@@ -0,0 +1,20 @@
+[     268ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/login:0
+[     379ms] [VERBOSE] [DOM] Input elements should have autocomplete attributes (suggested: "username"): (More info: https://goo.gl/9p2vKq) %o @ https://app2.hyeonworks.com/login:0
+[   10453ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   10532ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&panes=%7B%22ich%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22up%7Bjob%3D%7E%5C%22keycloak%7Cnode-exporter%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bjob%7D%7D+%E2%80%94+%7B%7Bpod%7D%7D%7B%7Bnode%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-55m%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1:0
+[   11687ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   13737ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   15113ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   20185ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   27259ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   40154ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   41870ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   50493ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   54082ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   60121ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   74362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   83269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   95349ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[   98008ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  104458ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
+[  123816ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362

.playwright-mcp/page-2026-09-04T03-23-16-267Z.yml

@@ -0,0 +1,44 @@
+- main [ref=f6e7]:
+  - generic [ref=f6e9]:
+    - generic [ref=f6e11]:
+      - generic [ref=f6e12]:
+        - img "Grafana" [ref=f6e13]
+        - heading "Welcome to Grafana" [level=1] [ref=f6e15]
+      - generic [ref=f6e19]:
+        - generic [ref=f6e20]:
+          - generic [ref=f6e21]: Email or username
+          - textbox "Email or username" [active] [ref=f6e28]:
+            - /placeholder: email or username
+        - generic [ref=f6e29]:
+          - generic [ref=f6e30]: Password
+          - generic [ref=f6e36]:
+            - textbox "Password" [ref=f6e37]:
+              - /placeholder: password
+            - switch "Show password" [ref=f6e39] [cursor=pointer]
+        - button "Log in" [ref=f6e42] [cursor=pointer]
+        - link "Forgot your password?" [ref=f6e45] [cursor=pointer]:
+          - /url: /user/password/send-reset-email
+    - list [ref=f6e49]:
+      - listitem [ref=f6e50]:
+        - link "Documentation" [ref=f6e53] [cursor=pointer]:
+          - /url: https://grafana.com/docs/grafana/latest/?utm_source=grafana_footer
+        - text: "|"
+      - listitem [ref=f6e54]:
+        - link "Support" [ref=f6e57] [cursor=pointer]:
+          - /url: https://grafana.com/products/enterprise/?utm_source=grafana_footer
+        - text: "|"
+      - listitem [ref=f6e58]:
+        - link "Community" [ref=f6e61] [cursor=pointer]:
+          - /url: https://community.grafana.com/?utm_source=grafana_footer
+        - text: "|"
+      - listitem [ref=f6e62]:
+        - link "Open Source" [ref=f6e63] [cursor=pointer]:
+          - /url: https://grafana.com/oss/grafana?utm_source=grafana_footer
+        - text: "|"
+      - listitem [ref=f6e64]:
+        - link "Grafana v11.4.0 (b58701869e)" [ref=f6e65] [cursor=pointer]:
+          - /url: https://github.com/grafana/grafana/blob/main/CHANGELOG.md
+        - text: "|"
+      - listitem [ref=f6e66]:
+        - link "New version available!" [ref=f6e69] [cursor=pointer]:
+          - /url: https://grafana.com/grafana/download?utm_source=grafana_footer

.playwright-mcp/page-2026-09-04T03-23-27-151Z.yml

@@ -0,0 +1,159 @@
+- generic [active] [ref=f9e1]:
+  - generic [ref=f9e4]:
+    - link "Skip to main content" [ref=f9e5] [cursor=pointer]:
+      - /url: "#pageContent"
+    - banner [ref=f9e7]:
+      - generic [ref=f9e8]:
+        - link [ref=f9e10] [cursor=pointer]:
+          - /url: /
+          - img "Grafana" [ref=f9e11]
+        - generic [ref=f9e14]:
+          - button "Search or jump to..." [ref=f9e18] [cursor=pointer]
+          - generic [ref=f9e19]: ctrl+k
+        - generic [ref=f9e23]:
+          - button "New" [ref=f9e24] [cursor=pointer]
+          - button "Help" [ref=f9e30] [cursor=pointer]
+          - button "News" [ref=f9e33] [cursor=pointer]
+          - button "Profile" [ref=f9e36] [cursor=pointer]:
+            - img "User avatar" [ref=f9e37]
+      - generic [ref=f9e38]:
+        - button "Open menu" [ref=f9e40] [cursor=pointer]
+        - navigation "Breadcrumbs" [ref=f9e43]:
+          - list [ref=f9e44]:
+            - listitem [ref=f9e45]:
+              - link "Home" [ref=f9e46] [cursor=pointer]:
+                - /url: /
+            - listitem [ref=f9e50]:
+              - link "Explore" [ref=f9e51] [cursor=pointer]:
+                - /url: /explore
+            - listitem [ref=f9e55]:
+              - generic "Prometheus" [ref=f9e56]
+        - generic [ref=f9e57]:
+          - generic [ref=f9e60]:
+            - button "Copy shortened URL" [ref=f9e61] [cursor=pointer]
+            - button "Open copy link options" [ref=f9e64] [cursor=pointer]
+          - button "Toggle top search bar" [ref=f9e68] [cursor=pointer]
+    - main [ref=f9e74]:
+      - generic [ref=f9e76]:
+        - heading "Explore" [level=1] [ref=f9e77]
+        - generic [ref=f9e82]:
+          - navigation "Explore toolbar" [ref=f9e84]:
+            - navigation "Search links" [ref=f9e86]:
+              - generic [ref=f9e87]:
+                - button "Content outline" [expanded] [ref=f9e89] [cursor=pointer]:
+                  - generic [ref=f9e92]: Outline
+                - generic [ref=f9e97] [cursor=pointer]:
+                  - img "Prometheus logo" [ref=f9e99]
+                  - textbox "Select a data source" [ref=f9e100]:
+                    - /placeholder: ""
+            - generic [ref=f9e104]:
+              - button "Split the pane" [ref=f9e106] [cursor=pointer]:
+                - generic [ref=f9e109]: Split
+              - button "Add" [ref=f9e111] [cursor=pointer]
+              - generic [ref=f9e116]:
+                - 'button "Time range selected: Last 55 minutes" [ref=f9e117] [cursor=pointer]'
+                - button "Zoom out time range" [ref=f9e122] [cursor=pointer]
+              - generic [ref=f9e126]:
+                - button "Run query" [ref=f9e127] [cursor=pointer]
+                - button "Auto refresh turned off. Choose refresh time interval" [ref=f9e131] [cursor=pointer]
+          - generic [ref=f9e135]:
+            - generic [ref=f9e139]:
+              - button "Collapse outline" [expanded] [ref=f9e141] [cursor=pointer]:
+                - img "arrow-from-right" [ref=f9e142]
+              - button "Queries" [ref=f9e145] [cursor=pointer]:
+                - img "arrow" [ref=f9e146]
+              - button "Graph" [ref=f9e150] [cursor=pointer]:
+                - img "graph-bar" [ref=f9e151]
+            - generic [ref=f9e158]:
+              - generic [ref=f9e160]:
+                - generic "Query editor row" [ref=f9e163]:
+                  - generic [ref=f9e164]:
+                    - generic [ref=f9e166]:
+                      - generic [ref=f9e167]:
+                        - button "Collapse query row" [expanded] [ref=f9e168] [cursor=pointer]
+                        - generic [ref=f9e171]:
+                          - button "Query editor row title A" [ref=f9e172] [cursor=pointer]:
+                            - generic [ref=f9e173]: A
+                          - emphasis [ref=f9e174]: (Prometheus)
+                      - generic [ref=f9e175]:
+                        - button "Show data source help" [ref=f9e177] [cursor=pointer]
+                        - button "Duplicate query" [ref=f9e181] [cursor=pointer]
+                        - button "Hide response" [ref=f9e185] [cursor=pointer]
+                        - button "Remove query" [ref=f9e189] [cursor=pointer]
+                        - button "Drag and drop to reorder" [ref=f9e192]:
+                          - img "Drag and drop to reorder" [ref=f9e193]
+                    - generic [ref=f9e196]:
+                      - generic [ref=f9e197]:
+                        - button "Kick start your query" [ref=f9e198] [cursor=pointer]
+                        - generic [ref=f9e201]:
+                          - generic [ref=f9e202] [cursor=pointer]: Explain
+                          - generic [ref=f9e203]:
+                            - checkbox "Explain Toggle switch" [ref=f9e204]
+                            - generic "Toggle switch" [ref=f9e205] [cursor=pointer]
+                        - radiogroup [ref=f9e210]:
+                          - generic [ref=f9e211]:
+                            - radio "Builder" [ref=f9e212] [cursor=pointer]
+                            - generic [ref=f9e213] [cursor=pointer]: Builder
+                          - generic [ref=f9e214]:
+                            - radio "Code" [checked] [ref=f9e215] [cursor=pointer]
+                            - generic [ref=f9e216] [cursor=pointer]: Code
+                      - generic [ref=f9e218]:
+                        - generic [ref=f9e220]:
+                          - button "Metrics browser" [ref=f9e221] [cursor=pointer]
+                          - code [ref=f9e228]:
+                            - generic [ref=f9e229]:
+                              - generic [ref=f9e234]: "up{job=~\"keycloak|node-exporter\"}"
+                              - textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=f9e239]: "up{job=~\"keycloak|node-exporter\"}"
+                        - 'button "Options Legend: {{job}} — {{pod}}{{node}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f9e245] [cursor=pointer]':
+                          - generic [ref=f9e249]:
+                            - heading "Options" [level=6] [ref=f9e250]
+                            - generic [ref=f9e251]:
+                              - generic [ref=f9e252]: "Legend: {{job}} — {{pod}}{{node}}"
+                              - generic [ref=f9e253]: "Format: Time series"
+                              - generic [ref=f9e254]: "Step: auto"
+                              - generic [ref=f9e255]: "Type: Range"
+                              - generic [ref=f9e256]: "Exemplars: false"
+                - generic [ref=f9e257]:
+                  - button "Add query" [ref=f9e258] [cursor=pointer]
+                  - button "Query history" [ref=f9e262] [cursor=pointer]
+                  - button "Query inspector" [ref=f9e266] [cursor=pointer]
+              - main [ref=f9e270]:
+                - region [ref=f9e272]:
+                  - generic [ref=f9e273]:
+                    - heading "Graph" [level=2] [ref=f9e275]
+                    - radiogroup [ref=f9e278]:
+                      - generic [ref=f9e279]:
+                        - radio "Lines" [checked] [ref=f9e280] [cursor=pointer]
+                        - generic [ref=f9e281] [cursor=pointer]: Lines
+                      - generic [ref=f9e282]:
+                        - radio "Bars" [ref=f9e283] [cursor=pointer]
+                        - generic [ref=f9e284] [cursor=pointer]: Bars
+                      - generic [ref=f9e285]:
+                        - radio "Points" [ref=f9e286] [cursor=pointer]
+                        - generic [ref=f9e287] [cursor=pointer]: Points
+                      - generic [ref=f9e288]:
+                        - radio "Stacked lines" [ref=f9e289] [cursor=pointer]
+                        - generic [ref=f9e290] [cursor=pointer]: Stacked lines
+                      - generic [ref=f9e291]:
+                        - radio "Stacked bars" [ref=f9e292] [cursor=pointer]
+                        - generic [ref=f9e293] [cursor=pointer]: Stacked bars
+                  - list [ref=f9e311]:
+                    - listitem [ref=f9e312]:
+                      - button "keycloak — keycloak-1kc-lab-1" [ref=f9e316] [cursor=pointer]
+                    - listitem [ref=f9e317]:
+                      - button "keycloak — keycloak-1kc-lab-1" [ref=f9e321] [cursor=pointer]
+                    - listitem [ref=f9e322]:
+                      - button "keycloak — keycloak-0kc-lab-2" [ref=f9e326] [cursor=pointer]
+                    - listitem [ref=f9e327]:
+                      - button "keycloak — keycloak-0kc-lab-2" [ref=f9e331] [cursor=pointer]
+                    - listitem [ref=f9e332]:
+                      - button "keycloak — keycloak-0kc-lab-2" [ref=f9e336] [cursor=pointer]
+                    - listitem [ref=f9e337]:
+                      - button "node-exporter — kc-lab-1" [ref=f9e341] [cursor=pointer]
+                    - listitem [ref=f9e342]:
+                      - button "node-exporter — kc-lab-2" [ref=f9e346] [cursor=pointer]
+  - generic [ref=f9e351]:
+    - alert
+    - alert
+    - complementary
+    - complementary

docs/evidence/a3-database-crash/01-crash-injection.txt

@@ -0,0 +1,19 @@
+=== [준비] 손실 측정 설계 확인 ===
+  LAST_SESSION_REFRESH 는 integer(초) — 200ms 손실은 보이지 않는다
+ created_on           | integer                 |           | not null | 
+ last_session_refresh | integer                 |           | not null | 0
+    "idx_user_session_expiration_created" btree (realm_id, offline_flag, remember_me, created_on, user_session_id, user_id)
+    "idx_user_session_expiration_last_refresh" btree (realm_id, offline_flag, remember_me, last_session_refresh, user_session_id, user_id)
+  → 대신 행 존재 여부로 잰다. 로그인 하나 = 행 하나 = 이진 판정
+
+  전역 synchronous_commit: on
+
+=== [1] 빠른 연속 로그인을 백그라운드로 시작 ===
+  루프 시작
+  6초 경과 — 지금까지 성공한 로그인: 0
+
+=== [2] PostgreSQL 강제 종료 (SIGKILL) ===
+  종료 시각: 12:00:26.511
+pod "postgres-7b474b88c8-xc2vt" force deleted from keycloak-lab namespace
+  삭제 반환: 12:00:26.586
+  클라이언트가 200 을 받은 로그인 수: 0

docs/evidence/a3-database-crash/02-design-check.txt

@@ -0,0 +1,14 @@
+deployment "postgres" successfully rolled out
+
+=== crash recovery 가 실행되었는가 (강제 종료의 흔적) ===
+2026-09-04 02:58:41.036 UTC [1] LOG:  database system is ready to accept connections
+
+=== [설계 확인] 로그인 트랜잭션도 synchronous_commit 을 끄는가 ===
+  --- 로그인 트랜잭션 (INSERT 가 있는 것) ---
+2:BEGIN
+5:COMMIT
+6:BEGIN
+9:insert into OFFLINE_USER_SESSION (BROKER_SESSION_ID,CREATED_ON,DATA,LAST_SESSION_REFRESH,REALM_ID,REMEMBER_ME,USER_ID,VERSION,OFFLINE_FLAG,USER_SESSION_ID) values ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10)
+10:insert into OFFLINE_CLIENT_SESSION (DATA,REALM_ID,TIMESTAMP,VERSION,CLIENT_ID,CLIENT_STORAGE_PROVIDER,EXTERNAL_CLIENT_ID,OFFLINE_FLAG,USER_SESSION_ID) values ($1,$2,$3,$4,$5,$6,$7,$8,$9)
+11:SET LOCAL synchronous_commit TO OFF
+12:COMMIT

docs/evidence/a3-database-crash/03-loss-measurement.txt

@@ -0,0 +1,17 @@
+=== [1] 로그인 루프 시작 (호스트에서 백그라운드로 exec — 세션이 살아 있어야 한다) ===
+  8초 동안 클라이언트가 200 을 받은 로그인: 106 건
+
+=== [2] SIGKILL ===
+  종료: 12:01:32.981
+  반환: 12:01:33.236
+  최종 성공 로그인 수: 110 건
+    마지막 sid: FimM-krSybBACP2qIvshLWwU
+    마지막 sid: EwFfFwOIfqiv8N5GQ5OjtsVq
+    마지막 sid: CJX-PxFkS7rUc_9FQgB7iw1f
+    마지막 sid: 1EFK7SgUA4M7tq_SkC_BD2er
+    마지막 sid: _LiqTczuyxlpOs3T3xs25SLv
+
+=== [3] PostgreSQL 재기동 후 crash recovery 확인 ===
+Waiting for deployment "postgres" rollout to finish: 0 of 1 updated replicas are available...
+deployment "postgres" successfully rolled out
+  2026-09-04 02:59:48.427 UTC [1] LOG:  database system is ready to accept connections

docs/evidence/a3-database-crash/04-comparison.txt

@@ -0,0 +1,27 @@
+=== [4] 클라이언트가 받은 sid 가 DB 에 있는가 ===
+  클라이언트가 200 을 받은 sid: 291 건
+  DB 온라인 세션 총계: 375
+
+  --- 마지막 15건을 하나씩 조회 ---
+    TCCOYnVlN30Y2fGyJEsVJ_Lq   있음
+    vBcllIkKWovh-FXN9tSzmxAe   있음
+    eMpN_ywUdRTks9uBE9aTumPK   있음
+    aPC_T0yrlMjrlskcLAp0AvY6   있음
+    UBPxmduB-ahGHBg636sY3AEz   있음
+    HLnBloNX9R3qQJSkpOnkNqL4   있음
+    _KPJS30IAqVhkTJGHcCxKvrM   있음
+    rq3caZ9MkyMYlFSSzRjQLygD   있음
+    ivvQm70hjl55DPpF7vpYmz_E   있음
+    81mx-rmi-tAeogHx3-su3z2q   있음
+    WnvNDH93uzcz1XNFbaMSXk1A   있음
+    DXPAIhjO5sGpCUlcD8IEoS8L   있음
+    aZMvl4IwPdK-rC_7bG005Z5C   있음
+    eIuBCprfWA5x0glcgSKYrrX0   있음
+    ozES5kEeu2IFf_cfcC_jFlbF   있음
+
+  마지막 15건 중 유실: 0 건
+
+=== [5] 전체 대조 — 몇 건이나 사라졌는가 ===
+  클라이언트 성공: 291 건
+  DB 에 존재:      291 건
+  ★ 유실:          0 건

docs/evidence/a3-database-crash/05-true-crash.txt

@@ -0,0 +1,12 @@
+=== [정리] 세션 테이블 비우고 루프 잔여 확인 ===
+DELETE 375
+  남은 세션: 0
+
+=== [재주입] postmaster(PID 1)에 SIGKILL — 진짜 크래시 ===
+  8초 후 성공 로그인: 110 건
+  SIGKILL: 12:03:21.441
+  최종 성공 로그인: 139 건
+
+=== [검증] 이번엔 crash recovery 가 돌았는가 ===
+deployment "postgres" successfully rolled out
+  2026-09-04 02:59:48.427 UTC [1] LOG:  database system is ready to accept connections

docs/evidence/a3-database-crash/06-backend-kill-crash.txt

@@ -0,0 +1,17 @@
+=== 로그인 루프 시작 ===
+  8초 후: 112 건
+
+=== 백엔드 프로세스에 SIGKILL → postmaster 가 재초기화한다 ===
+  시각: 12:04:22.063
+  최종 성공 로그인: 153 건
+
+=== [검증] crash recovery 가 돌았는가 ===
+  2026-09-04 02:59:48.427 UTC [1] LOG:  database system is ready to accept connections
+  2026-09-04 03:02:35.807 UTC [1] LOG:  server process (PID 40) was terminated by signal 9: Killed
+  2026-09-04 03:02:35.807 UTC [1] LOG:  terminating any other active server processes
+  2026-09-04 03:02:35.814 UTC [1] LOG:  all server processes terminated; reinitializing
+  2026-09-04 03:02:35.896 UTC [2585] LOG:  database system was not properly shut down; automatic recovery in progress
+  2026-09-04 03:02:35.899 UTC [2585] LOG:  redo starts at 0/23CAB68
+  2026-09-04 03:02:35.904 UTC [2585] LOG:  redo done at 0/2529E40 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
+  2026-09-04 03:02:35.923 UTC [2586] LOG:  checkpoint complete: wrote 113 buffers (0.7%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.004 s, sync=0.004 s, total=0.015 s; sync files=27, longest=0.003 s, average=0.001 s; distance=1405 kB, estimate=1405 kB; lsn=0/252A048, redo lsn=0/252A048
+  2026-09-04 03:02:35.926 UTC [1] LOG:  database system is ready to accept connections

docs/evidence/a3-database-crash/07-loss-result.txt

@@ -0,0 +1,19 @@
+=== 크래시 전후 대조 ===
+  클라이언트가 200 과 토큰을 받은 로그인 : 153 건
+  그중 DB 에 실제로 존재                 : 149 건
+  ★ 유실                                 : 4 건
+  DB 전체 온라인 세션                    : 150 건
+
+=== 유실된 sid 목록 ===
+    ★ CQUfg9HLH29xvhiu6pVlfWOo  ← 토큰은 발급됐는데 세션이 없다
+    ★ 5gLP4fqmpZBbjhH_d-0TPMMr  ← 토큰은 발급됐는데 세션이 없다
+    ★ hkcOv1QskUFmYveMLB6Hljra  ← 토큰은 발급됐는데 세션이 없다
+    ★ p5XybeQIYmAs818gO4Vl_5ea  ← 토큰은 발급됐는데 세션이 없다
+
+=== 그 토큰이 지금 실제로 쓰이는가 (마지막 sid 로 확인) ===
+  마지막 sid: 8do0Bw6tkVLDVxgxotE7GosH
+     user_session_id      | created_on | last_session_refresh 
+--------------------------+------------+----------------------
+ 8do0Bw6tkVLDVxgxotE7GosH | 1788490958 |           1788490958
+(1 row)
+

docs/evidence/a3-database-crash/README.md

@@ -0,0 +1,20 @@
+# A-3 — DB 강제 종료와 데이터 손실 증거
+
+2026-09-04 12:00–12:05 KST · Keycloak 26.7.0 / PostgreSQL 16
+해설: [`docs/experiment-a3-database-crash.md`](../../experiment-a3-database-crash.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-crash-injection.txt` | 첫 시도 실패 — 파드 안 백그라운드 루프가 `exec` 종료와 함께 죽어 0건 수집 |
+| `02-design-check.txt` | **핵심 설계 확인** — 로그인 트랜잭션도 `SET LOCAL synchronous_commit TO OFF` 로 커밋한다 |
+| `03-loss-measurement.txt` | `--grace-period=0 --force` 주입 |
+| `04-comparison.txt` | **유실 0건** — 그러나 crash recovery 가 안 돌았다. 죽인 적이 없는 것 |
+| `05-true-crash.txt` | `kill -9 1` 시도 — **컨테이너 안에서 PID 1 은 SIGKILL 을 무시한다** |
+| `06-backend-kill-crash.txt` | **성공한 주입** — 백엔드에 SIGKILL → `not properly shut down` / `redo starts` / `redo done` |
+| `07-loss-result.txt` | **결과: 153건 중 4건 유실.** 토큰은 발급됐는데 세션 행이 없는 sid 목록 |
+
+## 핵심 세 줄
+
+1. **로그인도 비동기 커밋이다.** refresh 시각뿐 아니라 **로그인 자체**가 사라질 수 있다.
+2. **153건 중 4건(약 2.6%) 유실** — 초당 19건 기준 마지막 0.2초 분량, `wal_writer_delay` 기본값과 일치.
+3. **주입을 세 번 시도해 세 번째에 성공했다.** 앞의 둘은 "손실 0"으로 보였지만 실제로는 크래시가 아니었다.

docs/evidence/a4-node-loss/01-baseline.txt

@@ -0,0 +1,20 @@
+=== A-4 기준선 ===
+kc-lab-1   Ready   true
+kc-lab-2   Ready   <none>
+
+a2-probe                    true   kc-lab-2
+keycloak-0                  true   kc-lab-2
+keycloak-1                  true   kc-lab-1
+postgres-7b474b88c8-2gf27   true   kc-lab-2
+
+=== PVC 가 어느 노드에 묶여 있는가 (재배치 가능성) ===
+  persistentvolumeclaim/postgres-data → kc-lab-2
+
+=== 서비스 정상 확인 ===
+  https://auth.hyeonworks.com/realms/master  HTTP 200
+
+=== VM 상태 ===
+--------------------------
+ 1    kc-lab-1   running
+ 2    kc-lab-2   running
+

docs/evidence/a4-node-loss/02-worker-node-killed.txt

@@ -0,0 +1,17 @@
+=== 워커 노드(kc-lab-2) 전원 차단 — virsh destroy 는 종료 신호가 없다 ===
+차단 시각: 12:07:43
+Domain 'kc-lab-2' destroyed
+
+
+  +15초  node=Ready | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 000
+  +30초  node=Ready | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 000
+  +45초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
+  +60초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
+  +75초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
+  +90초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
+  +105초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
+  +120초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
+  +135초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
+  +150초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
+  +165초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
+  +180초  node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503

docs/evidence/a4-node-loss/03-state-during-loss.txt

@@ -0,0 +1,28 @@
+=== 파드 상태의 진실 — Running 인데 노드가 없다 ===
+a2-probe                    Running   true    kc-lab-2   <none>
+keycloak-0                  Running   true    kc-lab-2   <none>
+keycloak-1                  Running   false   kc-lab-1   <none>
+postgres-7b474b88c8-2gf27   Running   true    kc-lab-2   <none>
+
+=== 재배치가 시도되었는가 ===
+10m         Warning   Unhealthy           pod/keycloak-0                   Readiness probe failed: Get "http://10.42.1.67:9000/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
+3m15s       Warning   NodeNotReady        pod/postgres-7b474b88c8-2gf27    Node is not ready
+3m15s       Warning   NodeNotReady        pod/keycloak-0                   Node is not ready
+3m15s       Warning   NodeNotReady        pod/a2-probe                     Node is not ready
+2m27s       Warning   Unhealthy           pod/keycloak-1                   Readiness probe failed: Get "http://10.42.0.35:9000/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
+2s          Warning   Unhealthy           pod/keycloak-1                   Readiness probe failed: HTTP probe failed with statuscode: 503
+
+=== 노드 taint — 쿠버네티스가 붙인 것 ===
+  node.kubernetes.io/unreachable=:NoSchedule
+  node.kubernetes.io/unreachable=:NoExecute
+
+=== Prometheus 가 본 것 (kc-lab-1 에 있어 살아남았다) ===
+  up{job=keycloak       pod=keycloak-1                } = 1
+  up{job=keycloak       pod=keycloak-0                } = 0
+  up{job=kubelet        pod=-                         } = 1
+  up{job=kubelet        pod=-                         } = 0
+  up{job=node-exporter  pod=kc-lab-1                  } = 1
+  up{job=node-exporter  pod=kc-lab-2                  } = 0
+  up{job=prometheus     pod=-                         } = 1
+
+=== 진입점이 처음 40초간 000 이었던 이유 — nginx upstream ===

docs/evidence/a4-node-loss/04-eviction-timing.txt

@@ -0,0 +1,15 @@
+=== nginx 설정 위치 찾기 ===
+
+=== NoExecute taint 의 tolerationSeconds — 언제 축출되는가 ===
+  node.kubernetes.io/not-ready NoExecute tolerationSeconds=300
+  node.kubernetes.io/unreachable NoExecute tolerationSeconds=300
+
+=== 5분 축출 시점까지 관찰 ===
+  +210초  a2-probe:Running keycloak-0:Running keycloak-1:Running postgres-7b474b88c8-2gf27:Running 
+  +240초  a2-probe:Running keycloak-0:Running keycloak-1:Running postgres-7b474b88c8-2gf27:Running 
+  +270초  a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending 
+  +300초  a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending 
+  +330초  a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending 
+  +360초  a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending 
+  +390초  a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending 
+  +420초  a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending 

docs/evidence/a4-node-loss/05-recovery.txt

@@ -0,0 +1,18 @@
+=== 새 postgres 가 Pending 인 이유 ===
+Events:
+  Type     Reason            Age    From               Message
+  ----     ------            ----   ----               -------
+  Warning  FailedScheduling  4m45s  default-scheduler  0/2 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
+
+=== keycloak-0 대체 파드가 안 생기는 이유 (StatefulSet) ===
+keycloak   2     <none>   1
+keycloak-0                  1/1   Terminating   0     30m
+keycloak-1                  0/1   Running       0     143m
+
+=== 복구 — 노드 재기동 ===
+재기동 시각: 12:16:31
+Domain 'kc-lab-2' started
+
+  +30초  node=Ready | Running 파드 3 개 | 외부 HTTP 503
+  +60초  node=Ready | Running 파드 3 개 | 외부 HTTP 200
+  → 서비스 복귀

docs/evidence/a4-node-loss/06-control-plane-inventory.txt

@@ -0,0 +1,19 @@
+=== 복구 확인 ===
+keycloak-0                  1/1   Running   0     68s
+keycloak-1                  1/1   Running   0     144m
+postgres-7b474b88c8-9cmsv   1/1   Running   0     4m20s
+
+=== kc-lab-1(k3s server)에 무엇이 있는가 — 이게 곧 영향 범위다 ===
+  keycloak-lab     keycloak-1
+  kube-system      coredns-54996dc9b4-8k8fj
+  kube-system      helm-install-traefik-crd-q29b5
+  kube-system      local-path-provisioner-77b9867795-g27z8
+  kube-system      metrics-server-6dc596dfb8-7xxq4
+  kube-system      svclb-traefik-5eb6a9a1-qwwk5
+  kube-system      traefik-5d6fcf895-wpfhr
+  observability    grafana-845b5678cf-b6gvc
+  observability    node-exporter-9qk9w
+  observability    prometheus-6774f94f7c-pzr2t
+
+=== Traefik replica 수 (진입점의 단일 장애점인가) ===
+traefik   1     1

docs/evidence/a4-node-loss/07-control-plane-loss.txt

@@ -0,0 +1,19 @@
+=== 컨트롤 플레인 노드(kc-lab-1) 전원 차단 ===
+차단 시각: 12:18:08
+Domain 'kc-lab-1' destroyed
+
+
+  +20초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
+  +40초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
+  +60초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
+  +80초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
+  +100초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
+  +120초  외부 auth=000 grafana=502 | kubectl: Unable to connect to the server: dial tcp 
+  +140초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
+  +160초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp 
+
+=== 살아 있는 노드에서 직접 확인 — 워크로드는 도는가 ===
+  CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD                 NAMESPACE
+  e5f777900b762       60e153026e8f5       4 minutes ago       Running             keycloak            0                   640d4dafaefb3       keycloak-0          keycloak-lab
+  
+  6

docs/evidence/a4-node-loss/08-control-plane-recovery.txt

@@ -0,0 +1,13 @@
+=== 컨트롤 플레인 노드 복구 ===
+재기동: 12:23:39
+Domain 'kc-lab-1' started
+
+  +30초  외부=502 | kc-lab-1=Ready kc-lab-2=Ready 
+  +60초  외부=200 | kc-lab-1=Ready kc-lab-2=Ready 
+  → 서비스 복귀 (총 60초)
+
+=== 최종 상태 ===
+keycloak-0                  1/1   Running   0                   7m57s
+keycloak-1                  1/1   Running   1 (<invalid> ago)   151m
+postgres-7b474b88c8-9cmsv   1/1   Running   0                   11m
+  (prometheus port-forward 재연결 필요)

docs/evidence/a4-node-loss/README.md

@@ -0,0 +1,23 @@
+# A-4 — 노드 전원 차단 증거
+
+2026-09-04 12:07–12:24 KST
+해설: [`docs/experiment-a4-node-loss.md`](../../experiment-a4-node-loss.md)
+
+| 파일 | 무엇을 보여주는가 |
+|---|---|
+| `01-baseline.txt` | 차단 전 — 양쪽 Ready, **PVC 가 kc-lab-2 에 못박혀 있음**(재배치 불가의 원인), 외부 200 |
+| `02-worker-node-killed.txt` | `virsh destroy kc-lab-2` — **40초간 노드가 Ready 로 남아 있고** 외부는 이미 `000`. 이후 `503` |
+| `03-state-during-loss.txt` | **죽은 파드가 `ready=true`, 산 파드가 `ready=false`.** `up` 은 정확히 0. `unreachable` taint |
+| `04-eviction-timing.txt` | `tolerationSeconds=300` — **5분 뒤** 축출, 새 postgres 는 `Pending` |
+| `05-recovery.txt` | `FailedScheduling: didn't match PersistentVolume's node affinity`, StatefulSet `DESIRED=2 CURRENT=1`. 노드 복귀 후 **60초** |
+| `06-control-plane-inventory.txt` | kc-lab-1 에 있는 것 목록 — **Traefik `replicas=1`** |
+| `07-control-plane-loss.txt` | `virsh destroy kc-lab-1` — 외부 `000`, `kubectl` 불통. **그런데 `crictl ps` 로 보면 keycloak-0 은 Running** |
+| `08-control-plane-recovery.txt` | 60초 만에 복귀 |
+| `a4-up-dropped-per-node.png` | Grafana — `up` 이 노드별로 0 으로 떨어지는 구간. 12:18–12:23 은 **0 이 아니라 데이터 없음**(관측자가 같이 죽음) |
+
+## 핵심 네 줄
+
+1. **쿠버네티스는 40초 동안 노드가 살아 있다고 말한다.** 사용자는 이미 장애를 겪는 중이다.
+2. **죽은 파드의 상태는 화석이다.** `ready=true` 인 파드가 꺼진 기계 위에 있다.
+3. **StatefulSet 은 대체 파드를 만들지 않고, PVC 는 재배치를 막는다.** 사람이 개입해야 한다.
+4. **컨트롤 플레인 상실 ≠ 워크로드 상실.** 컨테이너는 계속 돌고, 들어갈 문만 사라진다.

docs/experiment-a3-database-crash.md

@@ -0,0 +1,323 @@
+# A-3 — DB 를 강제로 죽이면 무엇을 잃는가 (RPO)
+
+브랜치 `feature/keycloak-a3-database-crash` ·
+증거 [`docs/evidence/a3-database-crash/`](evidence/a3-database-crash/) ·
+2026-09-04 12:00–12:05 KST · Keycloak 26.7.0 / PostgreSQL 16
+
+선행: [`A-0`](experiment-00-session-replication.md) ·
+[`A-2`](experiment-a2-database-loss.md)
+
+---
+
+## 0. 결론부터
+
+```
+  클라이언트가 200 과 토큰을 받은 로그인 : 153 건
+  그중 DB 에 실제로 존재                 : 149 건
+  ★ 유실                                 :   4 건
+```
+
+**로그인이 성공했다고 응답받았는데 세션이 존재하지 않는다.**
+
+A-0 에서 발견한 `SET LOCAL synchronous_commit TO OFF` 의 대가를 실측했다.
+버그가 아니라 **의도된 설계**이며, 그 비용이 얼마인지를 숫자로 확인한 것이다.
+
+---
+
+## 1. 설계 — 무엇을 재야 손실이 보이는가
+
+### 1-1. `LAST_SESSION_REFRESH` 로는 못 잰다
+
+처음 계획은 "세션 갱신 시각이 되감기는지" 보는 것이었다. 스키마를 보고 접었다.
+
+```
+ created_on           | integer
+ last_session_refresh | integer      ← 초 단위
+```
+
+**손실 창은 수백 밀리초**인데 눈금이 **1초**다. 보일 리가 없다.
+
+### 1-2. 행 존재 여부로 잰다 — 이진 판정
+
+```
+   로그인 1회  =  OFFLINE_USER_SESSION 행 1개
+   클라이언트가 sid 를 받았다  =  서버가 COMMIT 했다고 응답했다
+   크래시 후 그 sid 가 없다    =  잃은 것
+```
+
+**있거나 없거나**이므로 눈금 문제가 없다.
+
+### 1-3. 그런데 로그인도 비동기 커밋인가 — **먼저 확인해야 한다**
+
+A-0 에서 잡은 것은 **refresh** 트랜잭션이었다. 로그인(INSERT)도 그런지는
+확인하지 않았다. 아니라면 이 측정 설계 자체가 성립하지 않는다.
+
+```bash
+kubectl -n keycloak-lab exec deploy/postgres -- \
+  psql -U keycloak -d keycloak -c "alter system set log_statement='all'"
+kubectl -n keycloak-lab exec deploy/postgres -- \
+  psql -U keycloak -d keycloak -c "select pg_reload_conf()"
+```
+
+```
+BEGIN
+insert into OFFLINE_USER_SESSION (...) values (...)
+insert into OFFLINE_CLIENT_SESSION (...) values (...)
+SET LOCAL synchronous_commit TO OFF          ← 로그인도 비동기 커밋이다
+COMMIT
+```
+
+**확인됐고, 함의가 refresh 보다 훨씬 무겁다.**
+
+| | 잃으면 |
+|---|---|
+| refresh 갱신 시각 | 세션 수명이 조금 짧아진다. 사용자는 모른다 |
+| **로그인 자체** | **토큰은 손에 있는데 세션이 없다.** 다음 요청부터 실패 |
+
+---
+
+## 2. 실패한 주입 ① — `--grace-period=0 --force` 는 크래시가 아니다
+
+```bash
+kubectl -n keycloak-lab delete pod -l app=postgres --grace-period=0 --force
+```
+
+```
+  클라이언트 성공: 291 건
+  DB 에 존재:      291 건
+  ★ 유실:            0 건
+```
+
+**0건.** 그런데 이건 "안 잃었다"가 아니라 **죽인 적이 없는 것**이다.
+
+```
+=== 재기동 로그 ===
+  database system is ready to accept connections
+  (그뿐. "not properly shut down" 이 없다)
+```
+
+**crash recovery 가 돌지 않았다 = 깨끗하게 내려갔다.**
+
+| 신호 | PostgreSQL 의 반응 |
+|---|---|
+| **SIGTERM** | **fast shutdown** — 진행 중 트랜잭션을 롤백하고 **WAL 을 플러시**한 뒤 종료 |
+| SIGINT | smart shutdown — 연결이 끊기길 기다린다 |
+| **SIGKILL** | **즉사** — 플러시 없음. 다음 기동에 crash recovery |
+
+`--force --grace-period=0` 는 API 오브젝트를 즉시 지우지만 컨테이너 런타임은
+여전히 정상 종료 절차를 밟는다. **PostgreSQL 은 SIGTERM 을 받고 얌전히
+플러시했다.**
+
+> **A-1 에서 배운 것이 또 나왔다** — 주입이 실제로 걸렸는지 먼저 확인하지
+> 않으면 **"아무 일도 없었다"를 결과로 착각한다.**
+> 여기서는 **crash recovery 메시지가 그 확인 수단**이다.
+
+---
+
+## 3. 실패한 주입 ② — 컨테이너 안에서 PID 1 은 SIGKILL 을 받지 않는다
+
+```bash
+kubectl -n keycloak-lab exec deploy/postgres -- kill -9 1
+```
+
+**아무 일도 일어나지 않았다.** 파드는 재시작하지 않았고 로그 시각도 그대로였다.
+
+### 개념 — PID 1 의 시그널 보호
+
+리눅스 커널은 **PID 1 을 특별 취급**한다. 자기 PID 네임스페이스 안에서 온
+시그널은 **핸들러가 등록된 것만** 전달된다. **SIGKILL 도 예외가 아니다.**
+
+```
+   같은 네임스페이스 안에서  →  PID 1 은 등록하지 않은 시그널을 무시한다
+   조상 네임스페이스에서     →  전달된다 (노드에서 kill -9 하면 죽는다)
+```
+
+부팅 초기에 init 을 실수로 죽여 시스템이 멈추는 것을 막기 위한 장치인데,
+컨테이너에서는 **"안에서는 PID 1 을 못 죽인다"** 로 나타난다.
+
+---
+
+## 4. 성공한 주입 — 백엔드 프로세스를 죽인다
+
+PostgreSQL 은 **postmaster(부모) + 연결마다 백엔드(자식)** 구조다.
+자식 하나가 비정상 종료하면 **postmaster 는 공유 메모리가 오염됐다고 보고
+전체를 재초기화**한다. 그게 곧 crash recovery 다.
+
+```bash
+kubectl -n keycloak-lab exec deploy/postgres -- \
+  sh -c 'kill -9 $(pgrep -f "postgres: keycloak keycloak" | head -1)'
+```
+
+```
+server process (PID 40) was terminated by signal 9: Killed
+terminating any other active server processes
+all server processes terminated; reinitializing
+database system was not properly shut down; automatic recovery in progress
+redo starts at 0/23CAB68
+redo done at 0/2529E40
+checkpoint complete: wrote 113 buffers ...
+database system is ready to accept connections
+```
+
+**이번엔 주입이 걸렸다.** `not properly shut down` + `redo` 가 증거다.
+
+파드는 재시작하지 않는다 (`restarts=0`) — 컨테이너의 PID 1 인 postmaster 는
+살아 있고, 자식만 갈아치운 것이다. **데이터 관점에서는 전원이 나간 것과 같다.**
+
+---
+
+## 5. 결과
+
+```
+=== 크래시 전후 대조 ===
+  클라이언트가 200 과 토큰을 받은 로그인 : 153 건
+  그중 DB 에 실제로 존재                 : 149 건
+  ★ 유실                                 :   4 건
+
+=== 유실된 sid ===
+  ★ CQUfg9HLH29xvhiu6pVlfWOo  ← 토큰은 발급됐는데 세션이 없다
+  ★ 5gLP4fqmpZBbjhH_d-0TPMMr
+  ★ hkcOv1QskUFmYveMLB6Hljra
+  ★ p5XybeQIYmAs818gO4Vl_5ea
+```
+
+**약 2.6% 유실.** 초당 19건 정도 로그인하던 중이었으므로
+**대략 마지막 0.2초 분량**이다 — `wal_writer_delay` 기본값(200ms)과 맞는다.
+
+### 사용자에게 어떻게 보이는가
+
+```
+   로그인 성공  →  access token + refresh token 을 받음
+        │
+        │  (크래시)
+        ▼
+   다음 요청     →  access token 은 60초간 통한다
+        │            (서명만 보는 경로라면)
+        ▼
+   60초 후 refresh  →  "Session not active"  →  다시 로그인
+```
+
+**즉시 드러나지 않는다.** access token 수명 동안은 정상으로 보이다가
+갱신 시점에 끊긴다. 장애와 증상 사이에 **최대 60초의 시차**가 있다.
+
+---
+
+## 6. 개념
+
+### WAL 과 `synchronous_commit`
+
+```
+   COMMIT
+     │
+     ├─ WAL 버퍼(메모리)에 기록          ← 항상 한다
+     │
+     ├─ synchronous_commit = on  : 디스크 플러시를 기다렸다가 응답
+     └─ synchronous_commit = off : 기다리지 않고 즉시 응답   ← Keycloak
+                                     │
+                                     └─ 크래시 시 이 구간이 사라진다
+```
+
+| 설정 | 응답 속도 | 잃는 것 |
+|---|---|---|
+| `on` (PostgreSQL 기본) | 느리다 (디스크 대기) | 없다 |
+| **`off`** | 빠르다 | **최대 `wal_writer_delay` × 3 분량** |
+
+**전역 설정은 `on` 이었다.**
+
+```
+전역 synchronous_commit: on
+```
+
+**Keycloak 이 자기 트랜잭션에만 `SET LOCAL` 로 끈다.** DBA 가 서버 설정만
+보고 "우리는 동기 커밋"이라 믿으면 틀린다. **애플리케이션이 트랜잭션 단위로
+뒤집을 수 있다.**
+
+### crash recovery
+
+```
+   기동 시 pg_control 을 읽는다
+     └─ "깨끗하게 종료됨" 표시가 없다
+          └─ "database system was not properly shut down"
+               └─ 마지막 체크포인트부터 WAL 을 재생(redo)
+                    └─ 디스크에 안 내려간 커밋은 복구할 수 없다  ← 손실
+```
+
+`redo starts at 0/23CAB68` → `redo done at 0/2529E40` 사이가 재생된 구간이다.
+**WAL 에 없는 것은 재생할 수도 없다.**
+
+### 이 손실이 "허용된" 이유
+
+Keycloak 의 판단은 이렇게 읽힌다.
+
+| | |
+|---|---|
+| 세션 쓰기는 **매우 잦다** | 로그인마다, refresh 마다 |
+| 잃어도 **회복 가능하다** | 사용자가 다시 로그인하면 된다 |
+| 동기 커밋의 비용은 **모든 요청에 붙는다** | 크래시는 드물다 |
+
+**드문 사고의 비용을 상시 지연으로 지불하지 않겠다는 선택**이다.
+합리적이지만, **선택했다는 사실을 알고 있어야 한다.**
+
+---
+
+## 7. 운영에 주는 것
+
+| 알게 된 것 | 함의 |
+|---|---|
+| 로그인도 비동기 커밋 | **RPO 가 0 이 아니다.** 크래시 시 마지막 수백 ms 로그인은 사라진다 |
+| 전역 `on` 인데 세션만 `off` | **서버 설정으로 판단하면 안 된다.** 애플리케이션이 뒤집는다 |
+| 손실이 즉시 안 보인다 | access token 수명만큼 시차. **모니터링은 갱신 실패율을 봐야 한다** |
+| `--grace-period=0` 은 크래시가 아니다 | **장애 훈련이 훈련이 안 될 수 있다** |
+| 컨테이너 안에서 PID 1 을 못 죽인다 | 크래시 재현은 **자식 프로세스**나 **노드에서** |
+
+### 바꿀 수 있는가
+
+```sql
+-- 세션 트랜잭션까지 동기 커밋으로 강제하려면 (지연 대가를 치른다)
+ALTER DATABASE keycloak SET synchronous_commit = on;   -- SET LOCAL 이 이깁니다
+```
+
+**`SET LOCAL` 이 우선하므로 이것으로는 못 막는다.** Keycloak 설정이나
+소스 수준의 문제이며, **RPO 0 이 필요하면 복제(streaming replication)로
+푸는 것이 맞다** — 동기 스탠바이가 있으면 `synchronous_commit` 의 의미가
+달라진다.
+
+---
+
+## 8. 재현 절차 (명령어)
+
+```bash
+# 0. 설계 확인 — 로그인도 비동기 커밋인지 먼저 본다
+kubectl -n keycloak-lab exec deploy/postgres -- psql -U keycloak -d keycloak \
+  -c "alter system set log_statement='all'" -c "select pg_reload_conf()"
+#   → 로그인 1회 후 로그에서 "SET LOCAL synchronous_commit TO OFF" 확인
+
+# 1. 세션 테이블 비우기
+kubectl -n keycloak-lab exec deploy/postgres -- psql -U keycloak -d keycloak \
+  -c "delete from offline_user_session"
+
+# 2. 로그인 루프 (호스트에서 백그라운드 exec — 파드 안 & 는 exec 종료와 함께 죽는다)
+kubectl -n keycloak-lab exec a2-probe -- sh -c '<로그인 반복, sid 를 /tmp/sids 에>' &
+
+# 3. 진짜 크래시 — 백엔드 프로세스에 SIGKILL
+kubectl -n keycloak-lab exec deploy/postgres -- \
+  sh -c 'kill -9 $(pgrep -f "postgres: keycloak keycloak" | head -1)'
+
+# 4. 주입이 걸렸는지 확인 — 이게 없으면 결과를 해석하지 않는다
+kubectl -n keycloak-lab logs deploy/postgres | grep -E "not properly shut down|redo"
+
+# 5. 대조
+kubectl -n keycloak-lab exec deploy/postgres -- psql -U keycloak -d keycloak -tAc \
+  "select count(*) from offline_user_session where user_session_id in (<sid 목록>)"
+```
+
+---
+
+## 9. 다음 실험에 남기는 것
+
+| 실험 | 이 실험이 준 것 |
+|---|---|
+| **D-1** 백업·복구 | RPO 는 **백업 주기 + 이 손실**이다. 둘을 더해야 진짜 RPO |
+| **B-6** Redis 영속화 | `appendfsync everysec` 은 **같은 모양의 트레이드오프** |
+| **A-4** 노드 상실 | 노드가 죽으면 이것도 함께 일어난다 (postgres 가 kc-lab-2) |
+| 전체 | **주입 성공 신호를 미리 정한다.** 여기서는 crash recovery 로그 |

docs/experiment-a4-node-loss.md

@@ -0,0 +1,397 @@
+# A-4 — 노드가 통째로 죽으면 어떻게 되는가
+
+브랜치 `feature/keycloak-a4-node-loss` ·
+증거 [`docs/evidence/a4-node-loss/`](evidence/a4-node-loss/) ·
+2026-09-04 12:07–12:24 KST
+
+선행: [`A-2`](experiment-a2-database-loss.md) · [`A-3`](experiment-a3-database-crash.md)
+
+**두 판본으로 나눠 실행했다.**
+
+| | 죽인 노드 | 성격 |
+|---|---|---|
+| **4a** | `kc-lab-2` (워커) | Keycloak + **DB 동시 상실** |
+| **4b** | `kc-lab-1` (k3s 서버) | **컨트롤 플레인 + 진입점 상실** |
+
+`virsh destroy` 는 종료 신호를 보내지 않는다. **전원을 뽑는 것과 같다.**
+
+---
+
+## 0. 결론부터
+
+| | 4a 워커 상실 | 4b 컨트롤 플레인 상실 |
+|---|---|---|
+| 외부 응답 | **503** | **000** (연결 자체가 안 됨) |
+| `kubectl` | 정상 | **불통** |
+| 살아 있는 워크로드 | keycloak-1 (하지만 DB 없음) | **keycloak-0 은 계속 돌고 있다** |
+| 복구 시간 | **60초** | **60초** |
+
+**둘 다 전면 장애**지만 이유가 다르다. 4a 는 **DB 가 같이 죽어서**, 4b 는
+**들어갈 길이 없어서**다.
+
+그리고 예상하지 못한 것 셋 —
+**죽은 파드가 산 파드보다 건강해 보이고**, **StatefulSet 은 대체 파드를 만들지
+않으며**, **PVC 때문에 재배치가 불가능하다.**
+
+---
+
+## 4a. 워커 노드 상실 (`kc-lab-2`)
+
+### 기준선
+
+```
+kc-lab-1   Ready   control-plane=true
+kc-lab-2   Ready   <none>
+
+keycloak-0   ready=true   kc-lab-2
+keycloak-1   ready=true   kc-lab-1
+postgres     ready=true   kc-lab-2
+
+PVC postgres-data → kc-lab-2        ← 재배치 가능성을 여기서 이미 알 수 있다
+외부 HTTP 200
+```
+
+### 주입
+
+```bash
+virsh destroy kc-lab-2
+```
+
+### 관찰
+
+```
+  +15초  node=Ready     | keycloak-0=Running postgres=Running | 외부 HTTP 000
+  +30초  node=Ready     | keycloak-0=Running postgres=Running | 외부 HTTP 000
+  +45초  node=NotReady  | keycloak-0=Running postgres=Running | 외부 HTTP 503
+  ...
+  +180초 node=NotReady  | keycloak-0=Running postgres=Running | 외부 HTTP 503
+```
+
+### 발견 ① — 쿠버네티스가 알아채는 데 **40초** 걸린다
+
+`+15초`, `+30초` 에도 노드는 여전히 `Ready` 다. **기계는 이미 없는데.**
+
+kube-controller-manager 의 `node-monitor-grace-period` (기본 40초) 동안
+kubelet 의 하트비트가 없어야 `NotReady` 로 바꾼다.
+
+> **그 40초 동안 쿠버네티스는 거짓말을 한다.** 사용자는 이미 장애를 겪고 있다
+> (`000`). **노드 상태를 알림 근거로 삼으면 항상 늦는다.**
+
+### 발견 ② — **죽은 파드가 산 파드보다 건강해 보인다**
+
+```
+POD          PHASE     READY   NODE
+keycloak-0   Running   true    kc-lab-2      ← 기계가 꺼져 있다
+keycloak-1   Running   false   kc-lab-1      ← 살아 있다
+```
+
+**`keycloak-0` 은 `ready=true`, `keycloak-1` 은 `ready=false`.**
+
+| 왜 | |
+|---|---|
+| keycloak-0 | kubelet 이 없어 **상태를 갱신할 수 없다.** 마지막으로 보고한 값이 그대로 얼어 있다 |
+| keycloak-1 | 살아서 **정직하게 보고한다** — DB 가 없으니 readiness 실패 |
+
+> **파드 상태는 "지금 어떤가"가 아니라 "마지막으로 그렇게 들었다"이다.**
+> 노드가 죽으면 그 노드 파드의 상태는 **화석**이 된다.
+
+Prometheus 는 정확했다.
+
+```
+  up{job=keycloak      pod=keycloak-1} = 1
+  up{job=keycloak      pod=keycloak-0} = 0     ← 긁기에 실패했다 = 사실
+  up{job=kubelet                     } = 0
+  up{job=node-exporter node=kc-lab-2 } = 0
+```
+
+**A-2 와 정반대다.** A-2 에서는 `up=1` 인데 서비스가 죽었고, 여기서는 `up=0`
+이 정확했다.
+
+| | `up` 이 잡는가 |
+|---|---|
+| **대상이 사라짐** (노드 상실) | **잡는다** |
+| **대상이 살아서 못 씀** (DB 상실) | **못 잡는다** |
+
+![up 이 노드별로 0 으로 떨어졌다](evidence/a4-node-loss/a4-up-dropped-per-node.png)
+
+### 발견 ③ — 축출은 **5분** 뒤에 시작된다
+
+쿠버네티스가 노드에 taint 를 붙인다.
+
+```
+  node.kubernetes.io/unreachable=:NoSchedule
+  node.kubernetes.io/unreachable=:NoExecute
+```
+
+그런데 모든 파드에는 기본 관용이 붙어 있다.
+
+```
+  node.kubernetes.io/not-ready    NoExecute  tolerationSeconds=300
+  node.kubernetes.io/unreachable  NoExecute  tolerationSeconds=300
+```
+
+```
+  +240초  전부 Running
+  +270초  a2-probe:Terminating keycloak-0:Terminating postgres:Terminating
+          postgres-...-9cmsv:Pending          ← 새 파드가 생겼다
+```
+
+**노드가 죽고 약 5분이 지나야 파드가 축출된다.** 잠깐 끊긴 노드가 돌아올 수도
+있으니 성급하게 옮기지 않겠다는 설계다. 대신 **그동안은 아무 일도 일어나지
+않는다.**
+
+### 발견 ④ — 축출돼도 **갈 곳이 없다**
+
+```
+Warning  FailedScheduling  default-scheduler
+  0/2 nodes are available:
+    1 node(s) didn't match PersistentVolume's node affinity,
+    1 node(s) had untolerated taint(s).
+```
+
+```
+   kc-lab-2  →  taint 때문에 못 간다 (죽은 노드)
+   kc-lab-1  →  PVC 가 kc-lab-2 에 묶여 있어 못 간다
+```
+
+**`local-path` PVC 는 그 노드의 로컬 디스크 디렉터리다.** 노드가 죽으면
+볼륨도 죽는다. 새 파드는 **영원히 Pending** 이다.
+
+> 결함이 아니라 **조건**이다. 이 실험대는 그걸 알고 `local-path` 를 골랐다.
+> 운영이라면 네트워크 스토리지나 DB 복제가 이 자리를 메워야 한다.
+
+### 발견 ⑤ — StatefulSet 은 **대체 파드를 만들지 않는다**
+
+```
+NAME       DESIRED   READY     CURRENT
+keycloak   2         <none>    1
+
+keycloak-0   1/1   Terminating   30m       ← 30분째
+keycloak-1   0/1   Running
+```
+
+**`keycloak-0` 이 `Terminating` 에서 영원히 멈춰 있고, 대체 파드가 안 생긴다.**
+
+| 왜 | |
+|---|---|
+| StatefulSet 의 계약 | **같은 이름의 파드는 클러스터에 하나뿐**이어야 한다 |
+| 컨트롤 플레인이 아는 것 | 노드가 안 보인다 = **파드가 죽었는지 확신할 수 없다** |
+| 그래서 | 옛 파드를 확실히 지우기 전엔 새 `keycloak-0` 을 못 만든다 |
+
+**Deployment 였다면 즉시 새 파드를 만든다.** 이름이 아무래도 되기 때문이다.
+StatefulSet 의 "안정된 이름"이라는 이득의 **반대편 비용**이 여기다.
+
+강제로 진행시키려면 사람이 개입해야 한다.
+
+```bash
+kubectl -n keycloak-lab delete pod keycloak-0 --grace-period=0 --force
+```
+
+**위험한 명령이다.** 노드가 사실은 살아 있고 네트워크만 끊긴 것이라면
+**같은 이름의 파드 둘이 동시에 존재**하게 된다 (split brain).
+
+### 복구
+
+```bash
+virsh start kc-lab-2
+```
+
+```
+  +30초  node=Ready | Running 파드 3 개 | 외부 HTTP 503
+  +60초  node=Ready | Running 파드 3 개 | 외부 HTTP 200
+  → 서비스 복귀
+```
+
+**60초.** 사람 개입 없이 전부 제자리로 돌아왔다.
+
+---
+
+## 4b. 컨트롤 플레인 노드 상실 (`kc-lab-1`)
+
+### 이 노드에 무엇이 있었나 — 그게 곧 영향 범위다
+
+```
+keycloak-lab    keycloak-1
+kube-system     coredns
+kube-system     local-path-provisioner
+kube-system     metrics-server
+kube-system     svclb-traefik
+kube-system     traefik                    ← replicas=1
+observability   grafana
+observability   prometheus
+observability   node-exporter
+```
+
+```
+NAME      REPLICAS   READY
+traefik   1          1        ← 진입점이 한 개다
+```
+
+### 주입과 관찰
+
+```bash
+virsh destroy kc-lab-1
+```
+
+```
+  +20초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server
+  +60초  외부 auth=000 grafana=000 | kubectl: Unable to connect to the server
+  +120초 외부 auth=000 grafana=502 | kubectl: Unable to connect to the server
+  +160초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server
+```
+
+### 발견 ⑥ — 워크로드는 살아 있는데 **길이 없다**
+
+살아남은 노드에서 직접 확인했다.
+
+```bash
+ssh kc-lab-2 'sudo crictl ps --name keycloak'
+```
+
+```
+CONTAINER      STATE     NAME       POD          NAMESPACE
+e5f777900b76   Running   keycloak   keycloak-0   keycloak-lab
+```
+
+**`keycloak-0` 은 멀쩡히 돌고 있다.** 컨테이너 런타임(containerd)은 API 서버
+없이도 이미 떠 있는 파드를 계속 굴린다.
+
+```
+   죽은 것:  API 서버 · 스케줄러 · coredns · Traefik · Prometheus · Grafana
+   산 것:    keycloak-0 · postgres · containerd
+   문제:     들어갈 문(Traefik)이 없다
+```
+
+> **컨트롤 플레인 상실은 워크로드 상실이 아니다.**
+> 이미 떠 있는 것은 계속 돈다. **새로 뜨거나 옮기거나 고치는 것이 안 될 뿐.**
+
+### 발견 ⑦ — `000` 과 `503` 의 차이
+
+| 코드 | 뜻 |
+|---|---|
+| **`503`** (4a) | nginx·Traefik 은 살아 있고 **뒤에 보낼 파드가 없다** |
+| **`000`** (4b) | curl 이 응답 자체를 못 받았다 — **Traefik 이 없어 nginx 가 죽은 주소를 기다리다 타임아웃** |
+
+`grafana=502` 가 한 번 찍힌 것이 증거다 — nginx 는 살아서 502 를 만들 수
+있었지만, 대부분은 타임아웃(`--max-time 8`)에 걸렸다.
+
+**두 코드가 다른 층의 고장을 가리킨다.**
+
+### 발견 ⑧ — 관측 시스템이 같이 죽으면 **0 이 아니라 구멍이 남는다**
+
+Prometheus 가 `kc-lab-1` 에 있었다. 그 노드가 죽은 12:18–12:23 구간은
+그래프에 **0 이 아니라 데이터 없음**으로 남는다.
+
+```
+   대상이 죽음        →  up = 0        →  "언제 죽었는지" 알 수 있다
+   관측자가 죽음      →  데이터 없음    →  "그때 무슨 일이 있었는지" 모른다
+```
+
+**계획 단계에서 "관측 스택은 죽이지 않는 노드에 둔다"고 정했던 이유**를
+실제로 확인한 셈이다. 다만 노드가 둘뿐이라 완전히 피할 수는 없다.
+
+### 복구
+
+```bash
+virsh start kc-lab-1
+```
+
+```
+  +30초  외부=502 | kc-lab-1=Ready kc-lab-2=Ready
+  +60초  외부=200 | kc-lab-1=Ready kc-lab-2=Ready
+  → 서비스 복귀 (총 60초)
+```
+
+**Grafana 는 로그인 세션을 잃었다** — 데이터가 `emptyDir` 이라 파드 재시작에
+사라진다. Prometheus 는 PVC 라 지표가 남았다. **의도한 설계대로 동작했다.**
+
+---
+
+## 5. 개념
+
+### `node-monitor-grace-period` 와 `tolerationSeconds`
+
+```
+   기계 정지
+      │
+      │  40초   node-monitor-grace-period      → 노드 NotReady
+      │
+      │  +300초 tolerationSeconds (NoExecute)  → 파드 축출 시작
+      │
+      ▼
+   총 약 5분 40초 동안 쿠버네티스는 아무것도 하지 않는다
+```
+
+**빠른 장애 대응이 필요하면 이 값을 줄여야 하지만**, 짧게 하면 일시적
+네트워크 끊김에도 파드를 옮기게 되어 **불안정해진다.** 맞바꿈이다.
+
+### `Terminating` 이 안 끝나는 이유
+
+```
+   파드 삭제 요청
+     └─ kubelet 이 컨테이너를 멈추고 "지웠다"고 보고해야 끝난다
+          └─ kubelet 이 없다 → 보고가 없다 → 영원히 Terminating
+```
+
+`--grace-period=0 --force` 는 **보고를 기다리지 않고 API 에서 지우는 것**이다.
+컨테이너가 실제로 죽었는지는 **모른 채** 진행한다.
+
+### StatefulSet vs Deployment — 노드 상실에서
+
+| | Deployment | StatefulSet |
+|---|---|---|
+| 대체 파드 | **즉시 만든다** (이름 무관) | **안 만든다** (이름이 유일해야) |
+| 노드 상실 시 | 자동 회복 | **사람이 강제 삭제해야** |
+| 위험 | 없음 | 강제 삭제 시 **중복 실행 가능** |
+
+**A-0 에서 "로그 대조가 쉬워서" StatefulSet 을 골랐는데, 그 대가가 여기 있다.**
+
+### `local-path` PVC 와 재배치
+
+```
+   PVC → PV → nodeAffinity: kubernetes.io/hostname = kc-lab-2
+                              └─ 그 노드의 /var/lib/rancher/k3s/storage/... 디렉터리
+```
+
+**볼륨이 노드에 못박히면 파드도 못박힌다.** 노드가 죽으면 데이터도 함께
+접근 불가가 된다.
+
+---
+
+## 6. 재현 절차 (명령어)
+
+```bash
+export LIBVIRT_DEFAULT_URI=qemu:///system
+
+# 사전 — PVC 가 어디 묶여 있는지 먼저 본다 (재배치 가능성이 여기서 정해진다)
+kubectl -n keycloak-lab get pvc -o name | while read p; do
+  V=$(kubectl -n keycloak-lab get $p -o jsonpath='{.spec.volumeName}')
+  kubectl get pv $V -o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]}'
+done
+
+# 4a 워커 상실
+virsh destroy kc-lab-2
+kubectl get node kc-lab-2                        # 40초 뒤 NotReady
+kubectl -n keycloak-lab get pods -o wide         # Running 인 채로 얼어 있다
+kubectl get node kc-lab-2 -o jsonpath='{.spec.taints}'
+#   5분 뒤 Terminating + 새 파드 Pending
+kubectl -n keycloak-lab describe pod <new-pod> | grep -A4 Events
+
+# 4b 컨트롤 플레인 상실 — kubectl 이 죽으므로 노드에서 직접 본다
+virsh destroy kc-lab-1
+ssh kc-lab-2 'sudo crictl ps'                    # 워크로드는 살아 있다
+
+# 복구
+virsh start kc-lab-2 && virsh start kc-lab-1
+```
+
+---
+
+## 7. 다음 실험에 남기는 것
+
+| 실험 | 이 실험이 준 것 |
+|---|---|
+| **A-5** 비대칭 파티션 | 여기서는 노드가 **완전히** 사라졌다. 부분 단절은 더 고약하다 |
+| **D-1** 백업·복구 | **PVC 가 노드에 묶여 있다** — 노드가 영영 안 돌아오면 백업이 유일한 길 |
+| 구성 개선 | **Traefik `replicas=1` 은 진입점 단일 장애점** — 2로 늘리거나 DaemonSet 으로 |
+| 구성 개선 | **Grafana 가 `emptyDir`** — 재시작마다 로그인 세션이 사라진다 |