diff --git a/.playwright-mcp/console-2026-09-04T02-56-16-987Z.log b/.playwright-mcp/console-2026-09-04T02-56-16-987Z.log index 499e5a5..fb86694 100644 --- a/.playwright-mcp/console-2026-09-04T02-56-16-987Z.log +++ b/.playwright-mcp/console-2026-09-04T02-56-16-987Z.log @@ -53,3 +53,77 @@ [ 502295ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 [ 511625ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 [ 524619ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 533050ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 553189ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 571699ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 575413ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 588005ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 599161ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 608816ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 619031ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 626612ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 639003ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 655077ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 674532ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 677197ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 696027ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 709512ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 727505ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 747139ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 754565ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 773682ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 784674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 787399ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 802845ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 825589ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 832768ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 842043ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 852717ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 871748ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 876571ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 894491ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 895515ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 904310ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 916681ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 934752ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 941175ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 946004ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 960849ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 962180ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 984446ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 997579ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1010105ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1027190ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1043388ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1061311ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1066040ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1080349ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1092336ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1104312ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1109362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1127047ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1134728ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1154899ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1161253ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1174971ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1184491ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1185340ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1189111ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1196984ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1204794ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1247961ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1286322ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1319329ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1362771ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1396881ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1434184ms] [ERROR] Failed to load resource: the server responded with a status of 502 () @ https://app2.hyeonworks.com/api/user/auth-tokens/rotate:0 +[ 1443506ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1465016ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1513337ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1556906ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1562963ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1572896ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1576390ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 503 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1593794ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1600651ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 1616424ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 diff --git a/.playwright-mcp/console-2026-09-04T03-23-15-961Z.log b/.playwright-mcp/console-2026-09-04T03-23-15-961Z.log new file mode 100644 index 0000000..102ab83 --- /dev/null +++ b/.playwright-mcp/console-2026-09-04T03-23-15-961Z.log @@ -0,0 +1,20 @@ +[ 268ms] [WARNING] is deprecated. Please include @ https://app2.hyeonworks.com/login:0 +[ 379ms] [VERBOSE] [DOM] Input elements should have autocomplete attributes (suggested: "username"): (More info: https://goo.gl/9p2vKq) %o @ https://app2.hyeonworks.com/login:0 +[ 10453ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 10532ms] [WARNING] is deprecated. Please include @ https://app2.hyeonworks.com/explore?schemaVersion=1&panes=%7B%22ich%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22up%7Bjob%3D%7E%5C%22keycloak%7Cnode-exporter%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bjob%7D%7D+%E2%80%94+%7B%7Bpod%7D%7D%7B%7Bnode%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-55m%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1:0 +[ 11687ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 13737ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 15113ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 20185ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 27259ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 40154ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 41870ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 50493ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 54082ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 60121ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 74362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 83269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 95349ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 98008ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 104458ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 +[ 123816ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362 diff --git a/.playwright-mcp/page-2026-09-04T03-23-16-267Z.yml b/.playwright-mcp/page-2026-09-04T03-23-16-267Z.yml new file mode 100644 index 0000000..9ec7240 --- /dev/null +++ b/.playwright-mcp/page-2026-09-04T03-23-16-267Z.yml @@ -0,0 +1,44 @@ +- main [ref=f6e7]: + - generic [ref=f6e9]: + - generic [ref=f6e11]: + - generic [ref=f6e12]: + - img "Grafana" [ref=f6e13] + - heading "Welcome to Grafana" [level=1] [ref=f6e15] + - generic [ref=f6e19]: + - generic [ref=f6e20]: + - generic [ref=f6e21]: Email or username + - textbox "Email or username" [active] [ref=f6e28]: + - /placeholder: email or username + - generic [ref=f6e29]: + - generic [ref=f6e30]: Password + - generic [ref=f6e36]: + - textbox "Password" [ref=f6e37]: + - /placeholder: password + - switch "Show password" [ref=f6e39] [cursor=pointer] + - button "Log in" [ref=f6e42] [cursor=pointer] + - link "Forgot your password?" [ref=f6e45] [cursor=pointer]: + - /url: /user/password/send-reset-email + - list [ref=f6e49]: + - listitem [ref=f6e50]: + - link "Documentation" [ref=f6e53] [cursor=pointer]: + - /url: https://grafana.com/docs/grafana/latest/?utm_source=grafana_footer + - text: "|" + - listitem [ref=f6e54]: + - link "Support" [ref=f6e57] [cursor=pointer]: + - /url: https://grafana.com/products/enterprise/?utm_source=grafana_footer + - text: "|" + - listitem [ref=f6e58]: + - link "Community" [ref=f6e61] [cursor=pointer]: + - /url: https://community.grafana.com/?utm_source=grafana_footer + - text: "|" + - listitem [ref=f6e62]: + - link "Open Source" [ref=f6e63] [cursor=pointer]: + - /url: https://grafana.com/oss/grafana?utm_source=grafana_footer + - text: "|" + - listitem [ref=f6e64]: + - link "Grafana v11.4.0 (b58701869e)" [ref=f6e65] [cursor=pointer]: + - /url: https://github.com/grafana/grafana/blob/main/CHANGELOG.md + - text: "|" + - listitem [ref=f6e66]: + - link "New version available!" [ref=f6e69] [cursor=pointer]: + - /url: https://grafana.com/grafana/download?utm_source=grafana_footer \ No newline at end of file diff --git a/.playwright-mcp/page-2026-09-04T03-23-27-151Z.yml b/.playwright-mcp/page-2026-09-04T03-23-27-151Z.yml new file mode 100644 index 0000000..943ed02 --- /dev/null +++ b/.playwright-mcp/page-2026-09-04T03-23-27-151Z.yml @@ -0,0 +1,159 @@ +- generic [active] [ref=f9e1]: + - generic [ref=f9e4]: + - link "Skip to main content" [ref=f9e5] [cursor=pointer]: + - /url: "#pageContent" + - banner [ref=f9e7]: + - generic [ref=f9e8]: + - link [ref=f9e10] [cursor=pointer]: + - /url: / + - img "Grafana" [ref=f9e11] + - generic [ref=f9e14]: + - button "Search or jump to..." [ref=f9e18] [cursor=pointer] + - generic [ref=f9e19]: ctrl+k + - generic [ref=f9e23]: + - button "New" [ref=f9e24] [cursor=pointer] + - button "Help" [ref=f9e30] [cursor=pointer] + - button "News" [ref=f9e33] [cursor=pointer] + - button "Profile" [ref=f9e36] [cursor=pointer]: + - img "User avatar" [ref=f9e37] + - generic [ref=f9e38]: + - button "Open menu" [ref=f9e40] [cursor=pointer] + - navigation "Breadcrumbs" [ref=f9e43]: + - list [ref=f9e44]: + - listitem [ref=f9e45]: + - link "Home" [ref=f9e46] [cursor=pointer]: + - /url: / + - listitem [ref=f9e50]: + - link "Explore" [ref=f9e51] [cursor=pointer]: + - /url: /explore + - listitem [ref=f9e55]: + - generic "Prometheus" [ref=f9e56] + - generic [ref=f9e57]: + - generic [ref=f9e60]: + - button "Copy shortened URL" [ref=f9e61] [cursor=pointer] + - button "Open copy link options" [ref=f9e64] [cursor=pointer] + - button "Toggle top search bar" [ref=f9e68] [cursor=pointer] + - main [ref=f9e74]: + - generic [ref=f9e76]: + - heading "Explore" [level=1] [ref=f9e77] + - generic [ref=f9e82]: + - navigation "Explore toolbar" [ref=f9e84]: + - navigation "Search links" [ref=f9e86]: + - generic [ref=f9e87]: + - button "Content outline" [expanded] [ref=f9e89] [cursor=pointer]: + - generic [ref=f9e92]: Outline + - generic [ref=f9e97] [cursor=pointer]: + - img "Prometheus logo" [ref=f9e99] + - textbox "Select a data source" [ref=f9e100]: + - /placeholder: "" + - generic [ref=f9e104]: + - button "Split the pane" [ref=f9e106] [cursor=pointer]: + - generic [ref=f9e109]: Split + - button "Add" [ref=f9e111] [cursor=pointer] + - generic [ref=f9e116]: + - 'button "Time range selected: Last 55 minutes" [ref=f9e117] [cursor=pointer]' + - button "Zoom out time range" [ref=f9e122] [cursor=pointer] + - generic [ref=f9e126]: + - button "Run query" [ref=f9e127] [cursor=pointer] + - button "Auto refresh turned off. Choose refresh time interval" [ref=f9e131] [cursor=pointer] + - generic [ref=f9e135]: + - generic [ref=f9e139]: + - button "Collapse outline" [expanded] [ref=f9e141] [cursor=pointer]: + - img "arrow-from-right" [ref=f9e142] + - button "Queries" [ref=f9e145] [cursor=pointer]: + - img "arrow" [ref=f9e146] + - button "Graph" [ref=f9e150] [cursor=pointer]: + - img "graph-bar" [ref=f9e151] + - generic [ref=f9e158]: + - generic [ref=f9e160]: + - generic "Query editor row" [ref=f9e163]: + - generic [ref=f9e164]: + - generic [ref=f9e166]: + - generic [ref=f9e167]: + - button "Collapse query row" [expanded] [ref=f9e168] [cursor=pointer] + - generic [ref=f9e171]: + - button "Query editor row title A" [ref=f9e172] [cursor=pointer]: + - generic [ref=f9e173]: A + - emphasis [ref=f9e174]: (Prometheus) + - generic [ref=f9e175]: + - button "Show data source help" [ref=f9e177] [cursor=pointer] + - button "Duplicate query" [ref=f9e181] [cursor=pointer] + - button "Hide response" [ref=f9e185] [cursor=pointer] + - button "Remove query" [ref=f9e189] [cursor=pointer] + - button "Drag and drop to reorder" [ref=f9e192]: + - img "Drag and drop to reorder" [ref=f9e193] + - generic [ref=f9e196]: + - generic [ref=f9e197]: + - button "Kick start your query" [ref=f9e198] [cursor=pointer] + - generic [ref=f9e201]: + - generic [ref=f9e202] [cursor=pointer]: Explain + - generic [ref=f9e203]: + - checkbox "Explain Toggle switch" [ref=f9e204] + - generic "Toggle switch" [ref=f9e205] [cursor=pointer] + - radiogroup [ref=f9e210]: + - generic [ref=f9e211]: + - radio "Builder" [ref=f9e212] [cursor=pointer] + - generic [ref=f9e213] [cursor=pointer]: Builder + - generic [ref=f9e214]: + - radio "Code" [checked] [ref=f9e215] [cursor=pointer] + - generic [ref=f9e216] [cursor=pointer]: Code + - generic [ref=f9e218]: + - generic [ref=f9e220]: + - button "Metrics browser" [ref=f9e221] [cursor=pointer] + - code [ref=f9e228]: + - generic [ref=f9e229]: + - generic [ref=f9e234]: "up{job=~\"keycloak|node-exporter\"}" + - textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=f9e239]: "up{job=~\"keycloak|node-exporter\"}" + - 'button "Options Legend: {{job}} — {{pod}}{{node}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f9e245] [cursor=pointer]': + - generic [ref=f9e249]: + - heading "Options" [level=6] [ref=f9e250] + - generic [ref=f9e251]: + - generic [ref=f9e252]: "Legend: {{job}} — {{pod}}{{node}}" + - generic [ref=f9e253]: "Format: Time series" + - generic [ref=f9e254]: "Step: auto" + - generic [ref=f9e255]: "Type: Range" + - generic [ref=f9e256]: "Exemplars: false" + - generic [ref=f9e257]: + - button "Add query" [ref=f9e258] [cursor=pointer] + - button "Query history" [ref=f9e262] [cursor=pointer] + - button "Query inspector" [ref=f9e266] [cursor=pointer] + - main [ref=f9e270]: + - region [ref=f9e272]: + - generic [ref=f9e273]: + - heading "Graph" [level=2] [ref=f9e275] + - radiogroup [ref=f9e278]: + - generic [ref=f9e279]: + - radio "Lines" [checked] [ref=f9e280] [cursor=pointer] + - generic [ref=f9e281] [cursor=pointer]: Lines + - generic [ref=f9e282]: + - radio "Bars" [ref=f9e283] [cursor=pointer] + - generic [ref=f9e284] [cursor=pointer]: Bars + - generic [ref=f9e285]: + - radio "Points" [ref=f9e286] [cursor=pointer] + - generic [ref=f9e287] [cursor=pointer]: Points + - generic [ref=f9e288]: + - radio "Stacked lines" [ref=f9e289] [cursor=pointer] + - generic [ref=f9e290] [cursor=pointer]: Stacked lines + - generic [ref=f9e291]: + - radio "Stacked bars" [ref=f9e292] [cursor=pointer] + - generic [ref=f9e293] [cursor=pointer]: Stacked bars + - list [ref=f9e311]: + - listitem [ref=f9e312]: + - button "keycloak — keycloak-1kc-lab-1" [ref=f9e316] [cursor=pointer] + - listitem [ref=f9e317]: + - button "keycloak — keycloak-1kc-lab-1" [ref=f9e321] [cursor=pointer] + - listitem [ref=f9e322]: + - button "keycloak — keycloak-0kc-lab-2" [ref=f9e326] [cursor=pointer] + - listitem [ref=f9e327]: + - button "keycloak — keycloak-0kc-lab-2" [ref=f9e331] [cursor=pointer] + - listitem [ref=f9e332]: + - button "keycloak — keycloak-0kc-lab-2" [ref=f9e336] [cursor=pointer] + - listitem [ref=f9e337]: + - button "node-exporter — kc-lab-1" [ref=f9e341] [cursor=pointer] + - listitem [ref=f9e342]: + - button "node-exporter — kc-lab-2" [ref=f9e346] [cursor=pointer] + - generic [ref=f9e351]: + - alert + - alert + - complementary + - complementary \ No newline at end of file diff --git a/docs/evidence/a4-node-loss/01-baseline.txt b/docs/evidence/a4-node-loss/01-baseline.txt new file mode 100644 index 0000000..ea9de8e --- /dev/null +++ b/docs/evidence/a4-node-loss/01-baseline.txt @@ -0,0 +1,20 @@ +=== A-4 기준선 === +kc-lab-1 Ready true +kc-lab-2 Ready + +a2-probe true kc-lab-2 +keycloak-0 true kc-lab-2 +keycloak-1 true kc-lab-1 +postgres-7b474b88c8-2gf27 true kc-lab-2 + +=== PVC 가 어느 노드에 묶여 있는가 (재배치 가능성) === + persistentvolumeclaim/postgres-data → kc-lab-2 + +=== 서비스 정상 확인 === + https://auth.hyeonworks.com/realms/master HTTP 200 + +=== VM 상태 === +-------------------------- + 1 kc-lab-1 running + 2 kc-lab-2 running + diff --git a/docs/evidence/a4-node-loss/02-worker-node-killed.txt b/docs/evidence/a4-node-loss/02-worker-node-killed.txt new file mode 100644 index 0000000..a8ab55d --- /dev/null +++ b/docs/evidence/a4-node-loss/02-worker-node-killed.txt @@ -0,0 +1,17 @@ +=== 워커 노드(kc-lab-2) 전원 차단 — virsh destroy 는 종료 신호가 없다 === +차단 시각: 12:07:43 +Domain 'kc-lab-2' destroyed + + + +15초 node=Ready | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 000 + +30초 node=Ready | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 000 + +45초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503 + +60초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503 + +75초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503 + +90초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503 + +105초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503 + +120초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503 + +135초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503 + +150초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503 + +165초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503 + +180초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503 diff --git a/docs/evidence/a4-node-loss/03-state-during-loss.txt b/docs/evidence/a4-node-loss/03-state-during-loss.txt new file mode 100644 index 0000000..a5d5557 --- /dev/null +++ b/docs/evidence/a4-node-loss/03-state-during-loss.txt @@ -0,0 +1,28 @@ +=== 파드 상태의 진실 — Running 인데 노드가 없다 === +a2-probe Running true kc-lab-2 +keycloak-0 Running true kc-lab-2 +keycloak-1 Running false kc-lab-1 +postgres-7b474b88c8-2gf27 Running true kc-lab-2 + +=== 재배치가 시도되었는가 === +10m Warning Unhealthy pod/keycloak-0 Readiness probe failed: Get "http://10.42.1.67:9000/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) +3m15s Warning NodeNotReady pod/postgres-7b474b88c8-2gf27 Node is not ready +3m15s Warning NodeNotReady pod/keycloak-0 Node is not ready +3m15s Warning NodeNotReady pod/a2-probe Node is not ready +2m27s Warning Unhealthy pod/keycloak-1 Readiness probe failed: Get "http://10.42.0.35:9000/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) +2s Warning Unhealthy pod/keycloak-1 Readiness probe failed: HTTP probe failed with statuscode: 503 + +=== 노드 taint — 쿠버네티스가 붙인 것 === + node.kubernetes.io/unreachable=:NoSchedule + node.kubernetes.io/unreachable=:NoExecute + +=== Prometheus 가 본 것 (kc-lab-1 에 있어 살아남았다) === + up{job=keycloak pod=keycloak-1 } = 1 + up{job=keycloak pod=keycloak-0 } = 0 + up{job=kubelet pod=- } = 1 + up{job=kubelet pod=- } = 0 + up{job=node-exporter pod=kc-lab-1 } = 1 + up{job=node-exporter pod=kc-lab-2 } = 0 + up{job=prometheus pod=- } = 1 + +=== 진입점이 처음 40초간 000 이었던 이유 — nginx upstream === diff --git a/docs/evidence/a4-node-loss/04-eviction-timing.txt b/docs/evidence/a4-node-loss/04-eviction-timing.txt new file mode 100644 index 0000000..03de449 --- /dev/null +++ b/docs/evidence/a4-node-loss/04-eviction-timing.txt @@ -0,0 +1,15 @@ +=== nginx 설정 위치 찾기 === + +=== NoExecute taint 의 tolerationSeconds — 언제 축출되는가 === + node.kubernetes.io/not-ready NoExecute tolerationSeconds=300 + node.kubernetes.io/unreachable NoExecute tolerationSeconds=300 + +=== 5분 축출 시점까지 관찰 === + +210초 a2-probe:Running keycloak-0:Running keycloak-1:Running postgres-7b474b88c8-2gf27:Running + +240초 a2-probe:Running keycloak-0:Running keycloak-1:Running postgres-7b474b88c8-2gf27:Running + +270초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending + +300초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending + +330초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending + +360초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending + +390초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending + +420초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending diff --git a/docs/evidence/a4-node-loss/05-recovery.txt b/docs/evidence/a4-node-loss/05-recovery.txt new file mode 100644 index 0000000..d70cff4 --- /dev/null +++ b/docs/evidence/a4-node-loss/05-recovery.txt @@ -0,0 +1,18 @@ +=== 새 postgres 가 Pending 인 이유 === +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning FailedScheduling 4m45s default-scheduler 0/2 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. + +=== keycloak-0 대체 파드가 안 생기는 이유 (StatefulSet) === +keycloak 2 1 +keycloak-0 1/1 Terminating 0 30m +keycloak-1 0/1 Running 0 143m + +=== 복구 — 노드 재기동 === +재기동 시각: 12:16:31 +Domain 'kc-lab-2' started + + +30초 node=Ready | Running 파드 3 개 | 외부 HTTP 503 + +60초 node=Ready | Running 파드 3 개 | 외부 HTTP 200 + → 서비스 복귀 diff --git a/docs/evidence/a4-node-loss/06-control-plane-inventory.txt b/docs/evidence/a4-node-loss/06-control-plane-inventory.txt new file mode 100644 index 0000000..022af30 --- /dev/null +++ b/docs/evidence/a4-node-loss/06-control-plane-inventory.txt @@ -0,0 +1,19 @@ +=== 복구 확인 === +keycloak-0 1/1 Running 0 68s +keycloak-1 1/1 Running 0 144m +postgres-7b474b88c8-9cmsv 1/1 Running 0 4m20s + +=== kc-lab-1(k3s server)에 무엇이 있는가 — 이게 곧 영향 범위다 === + keycloak-lab keycloak-1 + kube-system coredns-54996dc9b4-8k8fj + kube-system helm-install-traefik-crd-q29b5 + kube-system local-path-provisioner-77b9867795-g27z8 + kube-system metrics-server-6dc596dfb8-7xxq4 + kube-system svclb-traefik-5eb6a9a1-qwwk5 + kube-system traefik-5d6fcf895-wpfhr + observability grafana-845b5678cf-b6gvc + observability node-exporter-9qk9w + observability prometheus-6774f94f7c-pzr2t + +=== Traefik replica 수 (진입점의 단일 장애점인가) === +traefik 1 1 diff --git a/docs/evidence/a4-node-loss/07-control-plane-loss.txt b/docs/evidence/a4-node-loss/07-control-plane-loss.txt new file mode 100644 index 0000000..348eceb --- /dev/null +++ b/docs/evidence/a4-node-loss/07-control-plane-loss.txt @@ -0,0 +1,19 @@ +=== 컨트롤 플레인 노드(kc-lab-1) 전원 차단 === +차단 시각: 12:18:08 +Domain 'kc-lab-1' destroyed + + + +20초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp + +40초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp + +60초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp + +80초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp + +100초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp + +120초 외부 auth=000 grafana=502 | kubectl: Unable to connect to the server: dial tcp + +140초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp + +160초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp + +=== 살아 있는 노드에서 직접 확인 — 워크로드는 도는가 === + CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE + e5f777900b762 60e153026e8f5 4 minutes ago Running keycloak 0 640d4dafaefb3 keycloak-0 keycloak-lab + + 6 diff --git a/docs/evidence/a4-node-loss/08-control-plane-recovery.txt b/docs/evidence/a4-node-loss/08-control-plane-recovery.txt new file mode 100644 index 0000000..0ed0e7b --- /dev/null +++ b/docs/evidence/a4-node-loss/08-control-plane-recovery.txt @@ -0,0 +1,13 @@ +=== 컨트롤 플레인 노드 복구 === +재기동: 12:23:39 +Domain 'kc-lab-1' started + + +30초 외부=502 | kc-lab-1=Ready kc-lab-2=Ready + +60초 외부=200 | kc-lab-1=Ready kc-lab-2=Ready + → 서비스 복귀 (총 60초) + +=== 최종 상태 === +keycloak-0 1/1 Running 0 7m57s +keycloak-1 1/1 Running 1 ( ago) 151m +postgres-7b474b88c8-9cmsv 1/1 Running 0 11m + (prometheus port-forward 재연결 필요) diff --git a/docs/evidence/a4-node-loss/README.md b/docs/evidence/a4-node-loss/README.md new file mode 100644 index 0000000..db38b7d --- /dev/null +++ b/docs/evidence/a4-node-loss/README.md @@ -0,0 +1,23 @@ +# A-4 — 노드 전원 차단 증거 + +2026-09-04 12:07–12:24 KST +해설: [`docs/experiment-a4-node-loss.md`](../../experiment-a4-node-loss.md) + +| 파일 | 무엇을 보여주는가 | +|---|---| +| `01-baseline.txt` | 차단 전 — 양쪽 Ready, **PVC 가 kc-lab-2 에 못박혀 있음**(재배치 불가의 원인), 외부 200 | +| `02-worker-node-killed.txt` | `virsh destroy kc-lab-2` — **40초간 노드가 Ready 로 남아 있고** 외부는 이미 `000`. 이후 `503` | +| `03-state-during-loss.txt` | **죽은 파드가 `ready=true`, 산 파드가 `ready=false`.** `up` 은 정확히 0. `unreachable` taint | +| `04-eviction-timing.txt` | `tolerationSeconds=300` — **5분 뒤** 축출, 새 postgres 는 `Pending` | +| `05-recovery.txt` | `FailedScheduling: didn't match PersistentVolume's node affinity`, StatefulSet `DESIRED=2 CURRENT=1`. 노드 복귀 후 **60초** | +| `06-control-plane-inventory.txt` | kc-lab-1 에 있는 것 목록 — **Traefik `replicas=1`** | +| `07-control-plane-loss.txt` | `virsh destroy kc-lab-1` — 외부 `000`, `kubectl` 불통. **그런데 `crictl ps` 로 보면 keycloak-0 은 Running** | +| `08-control-plane-recovery.txt` | 60초 만에 복귀 | +| `a4-up-dropped-per-node.png` | Grafana — `up` 이 노드별로 0 으로 떨어지는 구간. 12:18–12:23 은 **0 이 아니라 데이터 없음**(관측자가 같이 죽음) | + +## 핵심 네 줄 + +1. **쿠버네티스는 40초 동안 노드가 살아 있다고 말한다.** 사용자는 이미 장애를 겪는 중이다. +2. **죽은 파드의 상태는 화석이다.** `ready=true` 인 파드가 꺼진 기계 위에 있다. +3. **StatefulSet 은 대체 파드를 만들지 않고, PVC 는 재배치를 막는다.** 사람이 개입해야 한다. +4. **컨트롤 플레인 상실 ≠ 워크로드 상실.** 컨테이너는 계속 돌고, 들어갈 문만 사라진다. diff --git a/docs/evidence/a4-node-loss/a4-up-dropped-per-node.png b/docs/evidence/a4-node-loss/a4-up-dropped-per-node.png new file mode 100644 index 0000000..606cc6f Binary files /dev/null and b/docs/evidence/a4-node-loss/a4-up-dropped-per-node.png differ diff --git a/docs/experiment-a4-node-loss.md b/docs/experiment-a4-node-loss.md new file mode 100644 index 0000000..c619478 --- /dev/null +++ b/docs/experiment-a4-node-loss.md @@ -0,0 +1,397 @@ +# A-4 — 노드가 통째로 죽으면 어떻게 되는가 + +브랜치 `feature/keycloak-a4-node-loss` · +증거 [`docs/evidence/a4-node-loss/`](evidence/a4-node-loss/) · +2026-09-04 12:07–12:24 KST + +선행: [`A-2`](experiment-a2-database-loss.md) · [`A-3`](experiment-a3-database-crash.md) + +**두 판본으로 나눠 실행했다.** + +| | 죽인 노드 | 성격 | +|---|---|---| +| **4a** | `kc-lab-2` (워커) | Keycloak + **DB 동시 상실** | +| **4b** | `kc-lab-1` (k3s 서버) | **컨트롤 플레인 + 진입점 상실** | + +`virsh destroy` 는 종료 신호를 보내지 않는다. **전원을 뽑는 것과 같다.** + +--- + +## 0. 결론부터 + +| | 4a 워커 상실 | 4b 컨트롤 플레인 상실 | +|---|---|---| +| 외부 응답 | **503** | **000** (연결 자체가 안 됨) | +| `kubectl` | 정상 | **불통** | +| 살아 있는 워크로드 | keycloak-1 (하지만 DB 없음) | **keycloak-0 은 계속 돌고 있다** | +| 복구 시간 | **60초** | **60초** | + +**둘 다 전면 장애**지만 이유가 다르다. 4a 는 **DB 가 같이 죽어서**, 4b 는 +**들어갈 길이 없어서**다. + +그리고 예상하지 못한 것 셋 — +**죽은 파드가 산 파드보다 건강해 보이고**, **StatefulSet 은 대체 파드를 만들지 +않으며**, **PVC 때문에 재배치가 불가능하다.** + +--- + +## 4a. 워커 노드 상실 (`kc-lab-2`) + +### 기준선 + +``` +kc-lab-1 Ready control-plane=true +kc-lab-2 Ready + +keycloak-0 ready=true kc-lab-2 +keycloak-1 ready=true kc-lab-1 +postgres ready=true kc-lab-2 + +PVC postgres-data → kc-lab-2 ← 재배치 가능성을 여기서 이미 알 수 있다 +외부 HTTP 200 +``` + +### 주입 + +```bash +virsh destroy kc-lab-2 +``` + +### 관찰 + +``` + +15초 node=Ready | keycloak-0=Running postgres=Running | 외부 HTTP 000 + +30초 node=Ready | keycloak-0=Running postgres=Running | 외부 HTTP 000 + +45초 node=NotReady | keycloak-0=Running postgres=Running | 외부 HTTP 503 + ... + +180초 node=NotReady | keycloak-0=Running postgres=Running | 외부 HTTP 503 +``` + +### 발견 ① — 쿠버네티스가 알아채는 데 **40초** 걸린다 + +`+15초`, `+30초` 에도 노드는 여전히 `Ready` 다. **기계는 이미 없는데.** + +kube-controller-manager 의 `node-monitor-grace-period` (기본 40초) 동안 +kubelet 의 하트비트가 없어야 `NotReady` 로 바꾼다. + +> **그 40초 동안 쿠버네티스는 거짓말을 한다.** 사용자는 이미 장애를 겪고 있다 +> (`000`). **노드 상태를 알림 근거로 삼으면 항상 늦는다.** + +### 발견 ② — **죽은 파드가 산 파드보다 건강해 보인다** + +``` +POD PHASE READY NODE +keycloak-0 Running true kc-lab-2 ← 기계가 꺼져 있다 +keycloak-1 Running false kc-lab-1 ← 살아 있다 +``` + +**`keycloak-0` 은 `ready=true`, `keycloak-1` 은 `ready=false`.** + +| 왜 | | +|---|---| +| keycloak-0 | kubelet 이 없어 **상태를 갱신할 수 없다.** 마지막으로 보고한 값이 그대로 얼어 있다 | +| keycloak-1 | 살아서 **정직하게 보고한다** — DB 가 없으니 readiness 실패 | + +> **파드 상태는 "지금 어떤가"가 아니라 "마지막으로 그렇게 들었다"이다.** +> 노드가 죽으면 그 노드 파드의 상태는 **화석**이 된다. + +Prometheus 는 정확했다. + +``` + up{job=keycloak pod=keycloak-1} = 1 + up{job=keycloak pod=keycloak-0} = 0 ← 긁기에 실패했다 = 사실 + up{job=kubelet } = 0 + up{job=node-exporter node=kc-lab-2 } = 0 +``` + +**A-2 와 정반대다.** A-2 에서는 `up=1` 인데 서비스가 죽었고, 여기서는 `up=0` +이 정확했다. + +| | `up` 이 잡는가 | +|---|---| +| **대상이 사라짐** (노드 상실) | **잡는다** | +| **대상이 살아서 못 씀** (DB 상실) | **못 잡는다** | + +![up 이 노드별로 0 으로 떨어졌다](evidence/a4-node-loss/a4-up-dropped-per-node.png) + +### 발견 ③ — 축출은 **5분** 뒤에 시작된다 + +쿠버네티스가 노드에 taint 를 붙인다. + +``` + node.kubernetes.io/unreachable=:NoSchedule + node.kubernetes.io/unreachable=:NoExecute +``` + +그런데 모든 파드에는 기본 관용이 붙어 있다. + +``` + node.kubernetes.io/not-ready NoExecute tolerationSeconds=300 + node.kubernetes.io/unreachable NoExecute tolerationSeconds=300 +``` + +``` + +240초 전부 Running + +270초 a2-probe:Terminating keycloak-0:Terminating postgres:Terminating + postgres-...-9cmsv:Pending ← 새 파드가 생겼다 +``` + +**노드가 죽고 약 5분이 지나야 파드가 축출된다.** 잠깐 끊긴 노드가 돌아올 수도 +있으니 성급하게 옮기지 않겠다는 설계다. 대신 **그동안은 아무 일도 일어나지 +않는다.** + +### 발견 ④ — 축출돼도 **갈 곳이 없다** + +``` +Warning FailedScheduling default-scheduler + 0/2 nodes are available: + 1 node(s) didn't match PersistentVolume's node affinity, + 1 node(s) had untolerated taint(s). +``` + +``` + kc-lab-2 → taint 때문에 못 간다 (죽은 노드) + kc-lab-1 → PVC 가 kc-lab-2 에 묶여 있어 못 간다 +``` + +**`local-path` PVC 는 그 노드의 로컬 디스크 디렉터리다.** 노드가 죽으면 +볼륨도 죽는다. 새 파드는 **영원히 Pending** 이다. + +> 결함이 아니라 **조건**이다. 이 실험대는 그걸 알고 `local-path` 를 골랐다. +> 운영이라면 네트워크 스토리지나 DB 복제가 이 자리를 메워야 한다. + +### 발견 ⑤ — StatefulSet 은 **대체 파드를 만들지 않는다** + +``` +NAME DESIRED READY CURRENT +keycloak 2 1 + +keycloak-0 1/1 Terminating 30m ← 30분째 +keycloak-1 0/1 Running +``` + +**`keycloak-0` 이 `Terminating` 에서 영원히 멈춰 있고, 대체 파드가 안 생긴다.** + +| 왜 | | +|---|---| +| StatefulSet 의 계약 | **같은 이름의 파드는 클러스터에 하나뿐**이어야 한다 | +| 컨트롤 플레인이 아는 것 | 노드가 안 보인다 = **파드가 죽었는지 확신할 수 없다** | +| 그래서 | 옛 파드를 확실히 지우기 전엔 새 `keycloak-0` 을 못 만든다 | + +**Deployment 였다면 즉시 새 파드를 만든다.** 이름이 아무래도 되기 때문이다. +StatefulSet 의 "안정된 이름"이라는 이득의 **반대편 비용**이 여기다. + +강제로 진행시키려면 사람이 개입해야 한다. + +```bash +kubectl -n keycloak-lab delete pod keycloak-0 --grace-period=0 --force +``` + +**위험한 명령이다.** 노드가 사실은 살아 있고 네트워크만 끊긴 것이라면 +**같은 이름의 파드 둘이 동시에 존재**하게 된다 (split brain). + +### 복구 + +```bash +virsh start kc-lab-2 +``` + +``` + +30초 node=Ready | Running 파드 3 개 | 외부 HTTP 503 + +60초 node=Ready | Running 파드 3 개 | 외부 HTTP 200 + → 서비스 복귀 +``` + +**60초.** 사람 개입 없이 전부 제자리로 돌아왔다. + +--- + +## 4b. 컨트롤 플레인 노드 상실 (`kc-lab-1`) + +### 이 노드에 무엇이 있었나 — 그게 곧 영향 범위다 + +``` +keycloak-lab keycloak-1 +kube-system coredns +kube-system local-path-provisioner +kube-system metrics-server +kube-system svclb-traefik +kube-system traefik ← replicas=1 +observability grafana +observability prometheus +observability node-exporter +``` + +``` +NAME REPLICAS READY +traefik 1 1 ← 진입점이 한 개다 +``` + +### 주입과 관찰 + +```bash +virsh destroy kc-lab-1 +``` + +``` + +20초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server + +60초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server + +120초 외부 auth=000 grafana=502 | kubectl: Unable to connect to the server + +160초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server +``` + +### 발견 ⑥ — 워크로드는 살아 있는데 **길이 없다** + +살아남은 노드에서 직접 확인했다. + +```bash +ssh kc-lab-2 'sudo crictl ps --name keycloak' +``` + +``` +CONTAINER STATE NAME POD NAMESPACE +e5f777900b76 Running keycloak keycloak-0 keycloak-lab +``` + +**`keycloak-0` 은 멀쩡히 돌고 있다.** 컨테이너 런타임(containerd)은 API 서버 +없이도 이미 떠 있는 파드를 계속 굴린다. + +``` + 죽은 것: API 서버 · 스케줄러 · coredns · Traefik · Prometheus · Grafana + 산 것: keycloak-0 · postgres · containerd + 문제: 들어갈 문(Traefik)이 없다 +``` + +> **컨트롤 플레인 상실은 워크로드 상실이 아니다.** +> 이미 떠 있는 것은 계속 돈다. **새로 뜨거나 옮기거나 고치는 것이 안 될 뿐.** + +### 발견 ⑦ — `000` 과 `503` 의 차이 + +| 코드 | 뜻 | +|---|---| +| **`503`** (4a) | nginx·Traefik 은 살아 있고 **뒤에 보낼 파드가 없다** | +| **`000`** (4b) | curl 이 응답 자체를 못 받았다 — **Traefik 이 없어 nginx 가 죽은 주소를 기다리다 타임아웃** | + +`grafana=502` 가 한 번 찍힌 것이 증거다 — nginx 는 살아서 502 를 만들 수 +있었지만, 대부분은 타임아웃(`--max-time 8`)에 걸렸다. + +**두 코드가 다른 층의 고장을 가리킨다.** + +### 발견 ⑧ — 관측 시스템이 같이 죽으면 **0 이 아니라 구멍이 남는다** + +Prometheus 가 `kc-lab-1` 에 있었다. 그 노드가 죽은 12:18–12:23 구간은 +그래프에 **0 이 아니라 데이터 없음**으로 남는다. + +``` + 대상이 죽음 → up = 0 → "언제 죽었는지" 알 수 있다 + 관측자가 죽음 → 데이터 없음 → "그때 무슨 일이 있었는지" 모른다 +``` + +**계획 단계에서 "관측 스택은 죽이지 않는 노드에 둔다"고 정했던 이유**를 +실제로 확인한 셈이다. 다만 노드가 둘뿐이라 완전히 피할 수는 없다. + +### 복구 + +```bash +virsh start kc-lab-1 +``` + +``` + +30초 외부=502 | kc-lab-1=Ready kc-lab-2=Ready + +60초 외부=200 | kc-lab-1=Ready kc-lab-2=Ready + → 서비스 복귀 (총 60초) +``` + +**Grafana 는 로그인 세션을 잃었다** — 데이터가 `emptyDir` 이라 파드 재시작에 +사라진다. Prometheus 는 PVC 라 지표가 남았다. **의도한 설계대로 동작했다.** + +--- + +## 5. 개념 + +### `node-monitor-grace-period` 와 `tolerationSeconds` + +``` + 기계 정지 + │ + │ 40초 node-monitor-grace-period → 노드 NotReady + │ + │ +300초 tolerationSeconds (NoExecute) → 파드 축출 시작 + │ + ▼ + 총 약 5분 40초 동안 쿠버네티스는 아무것도 하지 않는다 +``` + +**빠른 장애 대응이 필요하면 이 값을 줄여야 하지만**, 짧게 하면 일시적 +네트워크 끊김에도 파드를 옮기게 되어 **불안정해진다.** 맞바꿈이다. + +### `Terminating` 이 안 끝나는 이유 + +``` + 파드 삭제 요청 + └─ kubelet 이 컨테이너를 멈추고 "지웠다"고 보고해야 끝난다 + └─ kubelet 이 없다 → 보고가 없다 → 영원히 Terminating +``` + +`--grace-period=0 --force` 는 **보고를 기다리지 않고 API 에서 지우는 것**이다. +컨테이너가 실제로 죽었는지는 **모른 채** 진행한다. + +### StatefulSet vs Deployment — 노드 상실에서 + +| | Deployment | StatefulSet | +|---|---|---| +| 대체 파드 | **즉시 만든다** (이름 무관) | **안 만든다** (이름이 유일해야) | +| 노드 상실 시 | 자동 회복 | **사람이 강제 삭제해야** | +| 위험 | 없음 | 강제 삭제 시 **중복 실행 가능** | + +**A-0 에서 "로그 대조가 쉬워서" StatefulSet 을 골랐는데, 그 대가가 여기 있다.** + +### `local-path` PVC 와 재배치 + +``` + PVC → PV → nodeAffinity: kubernetes.io/hostname = kc-lab-2 + └─ 그 노드의 /var/lib/rancher/k3s/storage/... 디렉터리 +``` + +**볼륨이 노드에 못박히면 파드도 못박힌다.** 노드가 죽으면 데이터도 함께 +접근 불가가 된다. + +--- + +## 6. 재현 절차 (명령어) + +```bash +export LIBVIRT_DEFAULT_URI=qemu:///system + +# 사전 — PVC 가 어디 묶여 있는지 먼저 본다 (재배치 가능성이 여기서 정해진다) +kubectl -n keycloak-lab get pvc -o name | while read p; do + V=$(kubectl -n keycloak-lab get $p -o jsonpath='{.spec.volumeName}') + kubectl get pv $V -o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]}' +done + +# 4a 워커 상실 +virsh destroy kc-lab-2 +kubectl get node kc-lab-2 # 40초 뒤 NotReady +kubectl -n keycloak-lab get pods -o wide # Running 인 채로 얼어 있다 +kubectl get node kc-lab-2 -o jsonpath='{.spec.taints}' +# 5분 뒤 Terminating + 새 파드 Pending +kubectl -n keycloak-lab describe pod | grep -A4 Events + +# 4b 컨트롤 플레인 상실 — kubectl 이 죽으므로 노드에서 직접 본다 +virsh destroy kc-lab-1 +ssh kc-lab-2 'sudo crictl ps' # 워크로드는 살아 있다 + +# 복구 +virsh start kc-lab-2 && virsh start kc-lab-1 +``` + +--- + +## 7. 다음 실험에 남기는 것 + +| 실험 | 이 실험이 준 것 | +|---|---| +| **A-5** 비대칭 파티션 | 여기서는 노드가 **완전히** 사라졌다. 부분 단절은 더 고약하다 | +| **D-1** 백업·복구 | **PVC 가 노드에 묶여 있다** — 노드가 영영 안 돌아오면 백업이 유일한 길 | +| 구성 개선 | **Traefik `replicas=1` 은 진입점 단일 장애점** — 2로 늘리거나 DaemonSet 으로 | +| 구성 개선 | **Grafana 가 `emptyDir`** — 재시작마다 로그인 세션이 사라진다 |