Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 DongHyeonkaandClaude Opus 5
|
2fa77a899e | ||
|
DongHyeonkaandClaude Opus 5
|
d0666c5ba0 |
@@ -53,3 +53,77 @@
|
||||
[ 502295ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 511625ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 524619ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 533050ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 553189ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 571699ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 575413ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 588005ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 599161ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 608816ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 619031ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 626612ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 639003ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 655077ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 674532ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 677197ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 696027ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 709512ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 727505ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 747139ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 754565ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 773682ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 784674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 787399ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 802845ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 825589ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 832768ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 842043ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 852717ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 871748ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 876571ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 894491ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 895515ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 904310ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 916681ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 934752ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 941175ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 946004ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 960849ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 962180ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 984446ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 997579ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1010105ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1027190ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1043388ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1061311ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1066040ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1080349ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1092336ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1104312ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1109362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1127047ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1134728ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1154899ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1161253ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1174971ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1184491ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1185340ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1189111ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1196984ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1204794ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1247961ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1286322ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1319329ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1362771ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1396881ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1434184ms] [ERROR] Failed to load resource: the server responded with a status of 502 () @ https://app2.hyeonworks.com/api/user/auth-tokens/rotate:0
|
||||
[ 1443506ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1465016ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1513337ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1556906ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1562963ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1572896ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1576390ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 503 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1593794ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1600651ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1616424ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
[ 268ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/login:0
|
||||
[ 379ms] [VERBOSE] [DOM] Input elements should have autocomplete attributes (suggested: "username"): (More info: https://goo.gl/9p2vKq) %o @ https://app2.hyeonworks.com/login:0
|
||||
[ 10453ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 10532ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&panes=%7B%22ich%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22up%7Bjob%3D%7E%5C%22keycloak%7Cnode-exporter%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bjob%7D%7D+%E2%80%94+%7B%7Bpod%7D%7D%7B%7Bnode%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-55m%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1:0
|
||||
[ 11687ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 13737ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 15113ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 20185ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 27259ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 40154ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 41870ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 50493ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 54082ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 60121ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 74362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 83269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 95349ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 98008ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 104458ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 123816ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 142764ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 157198ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 175220ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 182187ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 183411ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 190581ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 199499ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 209995ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 214849ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 233084ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 242910ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 257975ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 277220ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 290806ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 292674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 300013ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 309267ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 323998ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 328620ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 339787ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 355959ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 361079ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 369272ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 381866ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 397120ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 400712ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 412579ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 431630ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 440746ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 447809ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 461741ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 475459ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 482723ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 494299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 513174ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 521541ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 532145ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 551129ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 556250ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 574381ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 586190ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 595782ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 610555ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 630695ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 633257ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 652194ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 671361ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 677606ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 692444ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 696949ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 700122ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 709954ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 723992ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 743173ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 745075ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 753299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 766683ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 767855ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 787261ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 789725ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 800680ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 811324ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 814096ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 829045ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 848506ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 857716ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 866212ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 873174ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 876033ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 895198ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 898884ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 918642ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 930133ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 934008ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 948541ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 956860ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 959397ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 968406ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 971788ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 982746ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 997394ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1005078ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1009269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1021453ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1026674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1042137ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1057497ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1059851ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1061855ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1071317ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1078179ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1084940ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1093950ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1112279ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1124360ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1139110ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1157981ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1163071ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1179032ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1181195ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1191955ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1201472ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1218261ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1221169ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1229430ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
@@ -0,0 +1,24 @@
|
||||
[ 199ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 297ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22vendor_cluster_size%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-25m%22%2C%22to%22%3A%22now%22%7D%7D%7D:0
|
||||
[ 1000ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 2023ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 3559ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 7749ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 16364ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 25781ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 42158ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 59672ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 77090ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 87426ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 89952ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 104743ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error in connection establishment: net::ERR_NAME_NOT_RESOLVED @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1573121ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1584038ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1601626ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1613280ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1630173ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1634776ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1654205ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1661576ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1665269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1682669ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
@@ -0,0 +1,44 @@
|
||||
- main [ref=f6e7]:
|
||||
- generic [ref=f6e9]:
|
||||
- generic [ref=f6e11]:
|
||||
- generic [ref=f6e12]:
|
||||
- img "Grafana" [ref=f6e13]
|
||||
- heading "Welcome to Grafana" [level=1] [ref=f6e15]
|
||||
- generic [ref=f6e19]:
|
||||
- generic [ref=f6e20]:
|
||||
- generic [ref=f6e21]: Email or username
|
||||
- textbox "Email or username" [active] [ref=f6e28]:
|
||||
- /placeholder: email or username
|
||||
- generic [ref=f6e29]:
|
||||
- generic [ref=f6e30]: Password
|
||||
- generic [ref=f6e36]:
|
||||
- textbox "Password" [ref=f6e37]:
|
||||
- /placeholder: password
|
||||
- switch "Show password" [ref=f6e39] [cursor=pointer]
|
||||
- button "Log in" [ref=f6e42] [cursor=pointer]
|
||||
- link "Forgot your password?" [ref=f6e45] [cursor=pointer]:
|
||||
- /url: /user/password/send-reset-email
|
||||
- list [ref=f6e49]:
|
||||
- listitem [ref=f6e50]:
|
||||
- link "Documentation" [ref=f6e53] [cursor=pointer]:
|
||||
- /url: https://grafana.com/docs/grafana/latest/?utm_source=grafana_footer
|
||||
- text: "|"
|
||||
- listitem [ref=f6e54]:
|
||||
- link "Support" [ref=f6e57] [cursor=pointer]:
|
||||
- /url: https://grafana.com/products/enterprise/?utm_source=grafana_footer
|
||||
- text: "|"
|
||||
- listitem [ref=f6e58]:
|
||||
- link "Community" [ref=f6e61] [cursor=pointer]:
|
||||
- /url: https://community.grafana.com/?utm_source=grafana_footer
|
||||
- text: "|"
|
||||
- listitem [ref=f6e62]:
|
||||
- link "Open Source" [ref=f6e63] [cursor=pointer]:
|
||||
- /url: https://grafana.com/oss/grafana?utm_source=grafana_footer
|
||||
- text: "|"
|
||||
- listitem [ref=f6e64]:
|
||||
- link "Grafana v11.4.0 (b58701869e)" [ref=f6e65] [cursor=pointer]:
|
||||
- /url: https://github.com/grafana/grafana/blob/main/CHANGELOG.md
|
||||
- text: "|"
|
||||
- listitem [ref=f6e66]:
|
||||
- link "New version available!" [ref=f6e69] [cursor=pointer]:
|
||||
- /url: https://grafana.com/grafana/download?utm_source=grafana_footer
|
||||
@@ -0,0 +1,159 @@
|
||||
- generic [active] [ref=f9e1]:
|
||||
- generic [ref=f9e4]:
|
||||
- link "Skip to main content" [ref=f9e5] [cursor=pointer]:
|
||||
- /url: "#pageContent"
|
||||
- banner [ref=f9e7]:
|
||||
- generic [ref=f9e8]:
|
||||
- link [ref=f9e10] [cursor=pointer]:
|
||||
- /url: /
|
||||
- img "Grafana" [ref=f9e11]
|
||||
- generic [ref=f9e14]:
|
||||
- button "Search or jump to..." [ref=f9e18] [cursor=pointer]
|
||||
- generic [ref=f9e19]: ctrl+k
|
||||
- generic [ref=f9e23]:
|
||||
- button "New" [ref=f9e24] [cursor=pointer]
|
||||
- button "Help" [ref=f9e30] [cursor=pointer]
|
||||
- button "News" [ref=f9e33] [cursor=pointer]
|
||||
- button "Profile" [ref=f9e36] [cursor=pointer]:
|
||||
- img "User avatar" [ref=f9e37]
|
||||
- generic [ref=f9e38]:
|
||||
- button "Open menu" [ref=f9e40] [cursor=pointer]
|
||||
- navigation "Breadcrumbs" [ref=f9e43]:
|
||||
- list [ref=f9e44]:
|
||||
- listitem [ref=f9e45]:
|
||||
- link "Home" [ref=f9e46] [cursor=pointer]:
|
||||
- /url: /
|
||||
- listitem [ref=f9e50]:
|
||||
- link "Explore" [ref=f9e51] [cursor=pointer]:
|
||||
- /url: /explore
|
||||
- listitem [ref=f9e55]:
|
||||
- generic "Prometheus" [ref=f9e56]
|
||||
- generic [ref=f9e57]:
|
||||
- generic [ref=f9e60]:
|
||||
- button "Copy shortened URL" [ref=f9e61] [cursor=pointer]
|
||||
- button "Open copy link options" [ref=f9e64] [cursor=pointer]
|
||||
- button "Toggle top search bar" [ref=f9e68] [cursor=pointer]
|
||||
- main [ref=f9e74]:
|
||||
- generic [ref=f9e76]:
|
||||
- heading "Explore" [level=1] [ref=f9e77]
|
||||
- generic [ref=f9e82]:
|
||||
- navigation "Explore toolbar" [ref=f9e84]:
|
||||
- navigation "Search links" [ref=f9e86]:
|
||||
- generic [ref=f9e87]:
|
||||
- button "Content outline" [expanded] [ref=f9e89] [cursor=pointer]:
|
||||
- generic [ref=f9e92]: Outline
|
||||
- generic [ref=f9e97] [cursor=pointer]:
|
||||
- img "Prometheus logo" [ref=f9e99]
|
||||
- textbox "Select a data source" [ref=f9e100]:
|
||||
- /placeholder: ""
|
||||
- generic [ref=f9e104]:
|
||||
- button "Split the pane" [ref=f9e106] [cursor=pointer]:
|
||||
- generic [ref=f9e109]: Split
|
||||
- button "Add" [ref=f9e111] [cursor=pointer]
|
||||
- generic [ref=f9e116]:
|
||||
- 'button "Time range selected: Last 55 minutes" [ref=f9e117] [cursor=pointer]'
|
||||
- button "Zoom out time range" [ref=f9e122] [cursor=pointer]
|
||||
- generic [ref=f9e126]:
|
||||
- button "Run query" [ref=f9e127] [cursor=pointer]
|
||||
- button "Auto refresh turned off. Choose refresh time interval" [ref=f9e131] [cursor=pointer]
|
||||
- generic [ref=f9e135]:
|
||||
- generic [ref=f9e139]:
|
||||
- button "Collapse outline" [expanded] [ref=f9e141] [cursor=pointer]:
|
||||
- img "arrow-from-right" [ref=f9e142]
|
||||
- button "Queries" [ref=f9e145] [cursor=pointer]:
|
||||
- img "arrow" [ref=f9e146]
|
||||
- button "Graph" [ref=f9e150] [cursor=pointer]:
|
||||
- img "graph-bar" [ref=f9e151]
|
||||
- generic [ref=f9e158]:
|
||||
- generic [ref=f9e160]:
|
||||
- generic "Query editor row" [ref=f9e163]:
|
||||
- generic [ref=f9e164]:
|
||||
- generic [ref=f9e166]:
|
||||
- generic [ref=f9e167]:
|
||||
- button "Collapse query row" [expanded] [ref=f9e168] [cursor=pointer]
|
||||
- generic [ref=f9e171]:
|
||||
- button "Query editor row title A" [ref=f9e172] [cursor=pointer]:
|
||||
- generic [ref=f9e173]: A
|
||||
- emphasis [ref=f9e174]: (Prometheus)
|
||||
- generic [ref=f9e175]:
|
||||
- button "Show data source help" [ref=f9e177] [cursor=pointer]
|
||||
- button "Duplicate query" [ref=f9e181] [cursor=pointer]
|
||||
- button "Hide response" [ref=f9e185] [cursor=pointer]
|
||||
- button "Remove query" [ref=f9e189] [cursor=pointer]
|
||||
- button "Drag and drop to reorder" [ref=f9e192]:
|
||||
- img "Drag and drop to reorder" [ref=f9e193]
|
||||
- generic [ref=f9e196]:
|
||||
- generic [ref=f9e197]:
|
||||
- button "Kick start your query" [ref=f9e198] [cursor=pointer]
|
||||
- generic [ref=f9e201]:
|
||||
- generic [ref=f9e202] [cursor=pointer]: Explain
|
||||
- generic [ref=f9e203]:
|
||||
- checkbox "Explain Toggle switch" [ref=f9e204]
|
||||
- generic "Toggle switch" [ref=f9e205] [cursor=pointer]
|
||||
- radiogroup [ref=f9e210]:
|
||||
- generic [ref=f9e211]:
|
||||
- radio "Builder" [ref=f9e212] [cursor=pointer]
|
||||
- generic [ref=f9e213] [cursor=pointer]: Builder
|
||||
- generic [ref=f9e214]:
|
||||
- radio "Code" [checked] [ref=f9e215] [cursor=pointer]
|
||||
- generic [ref=f9e216] [cursor=pointer]: Code
|
||||
- generic [ref=f9e218]:
|
||||
- generic [ref=f9e220]:
|
||||
- button "Metrics browser" [ref=f9e221] [cursor=pointer]
|
||||
- code [ref=f9e228]:
|
||||
- generic [ref=f9e229]:
|
||||
- generic [ref=f9e234]: "up{job=~\"keycloak|node-exporter\"}"
|
||||
- textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=f9e239]: "up{job=~\"keycloak|node-exporter\"}"
|
||||
- 'button "Options Legend: {{job}} — {{pod}}{{node}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f9e245] [cursor=pointer]':
|
||||
- generic [ref=f9e249]:
|
||||
- heading "Options" [level=6] [ref=f9e250]
|
||||
- generic [ref=f9e251]:
|
||||
- generic [ref=f9e252]: "Legend: {{job}} — {{pod}}{{node}}"
|
||||
- generic [ref=f9e253]: "Format: Time series"
|
||||
- generic [ref=f9e254]: "Step: auto"
|
||||
- generic [ref=f9e255]: "Type: Range"
|
||||
- generic [ref=f9e256]: "Exemplars: false"
|
||||
- generic [ref=f9e257]:
|
||||
- button "Add query" [ref=f9e258] [cursor=pointer]
|
||||
- button "Query history" [ref=f9e262] [cursor=pointer]
|
||||
- button "Query inspector" [ref=f9e266] [cursor=pointer]
|
||||
- main [ref=f9e270]:
|
||||
- region [ref=f9e272]:
|
||||
- generic [ref=f9e273]:
|
||||
- heading "Graph" [level=2] [ref=f9e275]
|
||||
- radiogroup [ref=f9e278]:
|
||||
- generic [ref=f9e279]:
|
||||
- radio "Lines" [checked] [ref=f9e280] [cursor=pointer]
|
||||
- generic [ref=f9e281] [cursor=pointer]: Lines
|
||||
- generic [ref=f9e282]:
|
||||
- radio "Bars" [ref=f9e283] [cursor=pointer]
|
||||
- generic [ref=f9e284] [cursor=pointer]: Bars
|
||||
- generic [ref=f9e285]:
|
||||
- radio "Points" [ref=f9e286] [cursor=pointer]
|
||||
- generic [ref=f9e287] [cursor=pointer]: Points
|
||||
- generic [ref=f9e288]:
|
||||
- radio "Stacked lines" [ref=f9e289] [cursor=pointer]
|
||||
- generic [ref=f9e290] [cursor=pointer]: Stacked lines
|
||||
- generic [ref=f9e291]:
|
||||
- radio "Stacked bars" [ref=f9e292] [cursor=pointer]
|
||||
- generic [ref=f9e293] [cursor=pointer]: Stacked bars
|
||||
- list [ref=f9e311]:
|
||||
- listitem [ref=f9e312]:
|
||||
- button "keycloak — keycloak-1kc-lab-1" [ref=f9e316] [cursor=pointer]
|
||||
- listitem [ref=f9e317]:
|
||||
- button "keycloak — keycloak-1kc-lab-1" [ref=f9e321] [cursor=pointer]
|
||||
- listitem [ref=f9e322]:
|
||||
- button "keycloak — keycloak-0kc-lab-2" [ref=f9e326] [cursor=pointer]
|
||||
- listitem [ref=f9e327]:
|
||||
- button "keycloak — keycloak-0kc-lab-2" [ref=f9e331] [cursor=pointer]
|
||||
- listitem [ref=f9e332]:
|
||||
- button "keycloak — keycloak-0kc-lab-2" [ref=f9e336] [cursor=pointer]
|
||||
- listitem [ref=f9e337]:
|
||||
- button "node-exporter — kc-lab-1" [ref=f9e341] [cursor=pointer]
|
||||
- listitem [ref=f9e342]:
|
||||
- button "node-exporter — kc-lab-2" [ref=f9e346] [cursor=pointer]
|
||||
- generic [ref=f9e351]:
|
||||
- alert
|
||||
- alert
|
||||
- complementary
|
||||
- complementary
|
||||
@@ -0,0 +1,104 @@
|
||||
- generic [ref=f12e4]:
|
||||
- link "Skip to main content" [ref=f12e5] [cursor=pointer]:
|
||||
- /url: "#pageContent"
|
||||
- banner [ref=f12e7]:
|
||||
- generic [ref=f12e8]:
|
||||
- link [ref=f12e10] [cursor=pointer]:
|
||||
- /url: /
|
||||
- img "Grafana" [ref=f12e11]
|
||||
- generic [ref=f12e14]:
|
||||
- button "Search or jump to..." [ref=f12e18] [cursor=pointer]
|
||||
- generic [ref=f12e19]: ctrl+k
|
||||
- generic [ref=f12e23]:
|
||||
- button "New" [ref=f12e24] [cursor=pointer]
|
||||
- button "Help" [ref=f12e30] [cursor=pointer]
|
||||
- button "News" [ref=f12e33] [cursor=pointer]
|
||||
- button "Profile" [ref=f12e36] [cursor=pointer]:
|
||||
- img "User avatar" [ref=f12e37]
|
||||
- generic [ref=f12e38]:
|
||||
- button "Open menu" [ref=f12e40] [cursor=pointer]
|
||||
- navigation "Breadcrumbs" [ref=f12e43]:
|
||||
- list [ref=f12e44]:
|
||||
- listitem [ref=f12e45]:
|
||||
- link "Home" [ref=f12e46] [cursor=pointer]:
|
||||
- /url: /
|
||||
- listitem [ref=f12e50]:
|
||||
- link "Explore" [ref=f12e51] [cursor=pointer]:
|
||||
- /url: /explore
|
||||
- listitem [ref=f12e55]:
|
||||
- generic "Prometheus" [ref=f12e56]
|
||||
- generic [ref=f12e57]:
|
||||
- button "Show more items" [ref=f12e60] [cursor=pointer]
|
||||
- button "Toggle top search bar" [ref=f12e64] [cursor=pointer]
|
||||
- main [ref=f12e70]:
|
||||
- generic [ref=f12e72]:
|
||||
- heading "Explore" [level=1] [ref=f12e73]
|
||||
- generic [ref=f12e78]:
|
||||
- navigation "Explore toolbar" [ref=f12e80]:
|
||||
- navigation "Search links" [ref=f12e82]:
|
||||
- generic [ref=f12e83]:
|
||||
- button "Content outline" [expanded] [ref=f12e85] [cursor=pointer]:
|
||||
- generic [ref=f12e88]: Outline
|
||||
- generic [ref=f12e93] [cursor=pointer]:
|
||||
- img "Prometheus logo" [ref=f12e95]
|
||||
- textbox "Select a data source" [ref=f12e96]:
|
||||
- /placeholder: ""
|
||||
- button "Show more items" [ref=f12e102] [cursor=pointer]
|
||||
- generic [ref=f12e106]:
|
||||
- generic [ref=f12e110]:
|
||||
- button "Collapse outline" [expanded] [ref=f12e112] [cursor=pointer]:
|
||||
- img "arrow-from-right" [ref=f12e113]
|
||||
- button "Queries" [ref=f12e116] [cursor=pointer]:
|
||||
- img "arrow" [ref=f12e117]
|
||||
- generic [ref=f12e124]:
|
||||
- generic [ref=f12e126]:
|
||||
- generic "Query editor row" [ref=f12e129]:
|
||||
- generic [ref=f12e130]:
|
||||
- generic [ref=f12e132]:
|
||||
- generic [ref=f12e133]:
|
||||
- button "Collapse query row" [expanded] [ref=f12e134] [cursor=pointer]
|
||||
- generic [ref=f12e137]:
|
||||
- button "Query editor row title A" [ref=f12e138] [cursor=pointer]:
|
||||
- generic [ref=f12e139]: A
|
||||
- emphasis [ref=f12e140]: (Prometheus)
|
||||
- generic [ref=f12e141]:
|
||||
- button "Show data source help" [ref=f12e143] [cursor=pointer]
|
||||
- button "Duplicate query" [ref=f12e147] [cursor=pointer]
|
||||
- button "Hide response" [ref=f12e151] [cursor=pointer]
|
||||
- button "Remove query" [ref=f12e155] [cursor=pointer]
|
||||
- button "Drag and drop to reorder" [ref=f12e158]:
|
||||
- img "Drag and drop to reorder" [ref=f12e159]
|
||||
- generic [ref=f12e162]:
|
||||
- generic [ref=f12e163]:
|
||||
- button "Kick start your query" [ref=f12e164] [cursor=pointer]
|
||||
- generic [ref=f12e167]:
|
||||
- generic [ref=f12e168] [cursor=pointer]: Explain
|
||||
- generic [ref=f12e169]:
|
||||
- checkbox "Explain Toggle switch" [ref=f12e170]
|
||||
- generic "Toggle switch" [ref=f12e171] [cursor=pointer]
|
||||
- radiogroup [ref=f12e176]:
|
||||
- generic [ref=f12e177]:
|
||||
- radio "Builder" [ref=f12e178] [cursor=pointer]
|
||||
- generic [ref=f12e179] [cursor=pointer]: Builder
|
||||
- generic [ref=f12e180]:
|
||||
- radio "Code" [checked] [ref=f12e181] [cursor=pointer]
|
||||
- generic [ref=f12e182] [cursor=pointer]: Code
|
||||
- generic [ref=f12e184]:
|
||||
- generic [ref=f12e186]:
|
||||
- button "Loading metrics..." [disabled] [ref=f12e187] [cursor=pointer]
|
||||
- generic [ref=f12e190]: Loading editor
|
||||
- 'button "Options Legend: {{pod}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f12e198] [cursor=pointer]':
|
||||
- generic [ref=f12e202]:
|
||||
- heading "Options" [level=6] [ref=f12e203]
|
||||
- generic [ref=f12e204]:
|
||||
- generic [ref=f12e205]: "Legend: {{pod}}"
|
||||
- generic [ref=f12e206]: "Format: Time series"
|
||||
- generic [ref=f12e207]: "Step: auto"
|
||||
- generic [ref=f12e208]: "Type: Range"
|
||||
- generic [ref=f12e209]: "Exemplars: false"
|
||||
- generic [ref=f12e210]:
|
||||
- button "Add query" [ref=f12e211] [cursor=pointer]
|
||||
- button "Query history" [ref=f12e215] [cursor=pointer]
|
||||
- button "Query inspector" [ref=f12e219] [cursor=pointer]
|
||||
- generic:
|
||||
- main
|
||||
@@ -0,0 +1,20 @@
|
||||
=== A-4 기준선 ===
|
||||
kc-lab-1 Ready true
|
||||
kc-lab-2 Ready <none>
|
||||
|
||||
a2-probe true kc-lab-2
|
||||
keycloak-0 true kc-lab-2
|
||||
keycloak-1 true kc-lab-1
|
||||
postgres-7b474b88c8-2gf27 true kc-lab-2
|
||||
|
||||
=== PVC 가 어느 노드에 묶여 있는가 (재배치 가능성) ===
|
||||
persistentvolumeclaim/postgres-data → kc-lab-2
|
||||
|
||||
=== 서비스 정상 확인 ===
|
||||
https://auth.hyeonworks.com/realms/master HTTP 200
|
||||
|
||||
=== VM 상태 ===
|
||||
--------------------------
|
||||
1 kc-lab-1 running
|
||||
2 kc-lab-2 running
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
=== 워커 노드(kc-lab-2) 전원 차단 — virsh destroy 는 종료 신호가 없다 ===
|
||||
차단 시각: 12:07:43
|
||||
Domain 'kc-lab-2' destroyed
|
||||
|
||||
|
||||
+15초 node=Ready | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 000
|
||||
+30초 node=Ready | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 000
|
||||
+45초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+60초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+75초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+90초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+105초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+120초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+135초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+150초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+165초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+180초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
@@ -0,0 +1,28 @@
|
||||
=== 파드 상태의 진실 — Running 인데 노드가 없다 ===
|
||||
a2-probe Running true kc-lab-2 <none>
|
||||
keycloak-0 Running true kc-lab-2 <none>
|
||||
keycloak-1 Running false kc-lab-1 <none>
|
||||
postgres-7b474b88c8-2gf27 Running true kc-lab-2 <none>
|
||||
|
||||
=== 재배치가 시도되었는가 ===
|
||||
10m Warning Unhealthy pod/keycloak-0 Readiness probe failed: Get "http://10.42.1.67:9000/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
|
||||
3m15s Warning NodeNotReady pod/postgres-7b474b88c8-2gf27 Node is not ready
|
||||
3m15s Warning NodeNotReady pod/keycloak-0 Node is not ready
|
||||
3m15s Warning NodeNotReady pod/a2-probe Node is not ready
|
||||
2m27s Warning Unhealthy pod/keycloak-1 Readiness probe failed: Get "http://10.42.0.35:9000/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
|
||||
2s Warning Unhealthy pod/keycloak-1 Readiness probe failed: HTTP probe failed with statuscode: 503
|
||||
|
||||
=== 노드 taint — 쿠버네티스가 붙인 것 ===
|
||||
node.kubernetes.io/unreachable=:NoSchedule
|
||||
node.kubernetes.io/unreachable=:NoExecute
|
||||
|
||||
=== Prometheus 가 본 것 (kc-lab-1 에 있어 살아남았다) ===
|
||||
up{job=keycloak pod=keycloak-1 } = 1
|
||||
up{job=keycloak pod=keycloak-0 } = 0
|
||||
up{job=kubelet pod=- } = 1
|
||||
up{job=kubelet pod=- } = 0
|
||||
up{job=node-exporter pod=kc-lab-1 } = 1
|
||||
up{job=node-exporter pod=kc-lab-2 } = 0
|
||||
up{job=prometheus pod=- } = 1
|
||||
|
||||
=== 진입점이 처음 40초간 000 이었던 이유 — nginx upstream ===
|
||||
@@ -0,0 +1,15 @@
|
||||
=== nginx 설정 위치 찾기 ===
|
||||
|
||||
=== NoExecute taint 의 tolerationSeconds — 언제 축출되는가 ===
|
||||
node.kubernetes.io/not-ready NoExecute tolerationSeconds=300
|
||||
node.kubernetes.io/unreachable NoExecute tolerationSeconds=300
|
||||
|
||||
=== 5분 축출 시점까지 관찰 ===
|
||||
+210초 a2-probe:Running keycloak-0:Running keycloak-1:Running postgres-7b474b88c8-2gf27:Running
|
||||
+240초 a2-probe:Running keycloak-0:Running keycloak-1:Running postgres-7b474b88c8-2gf27:Running
|
||||
+270초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending
|
||||
+300초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending
|
||||
+330초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending
|
||||
+360초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending
|
||||
+390초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending
|
||||
+420초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending
|
||||
@@ -0,0 +1,18 @@
|
||||
=== 새 postgres 가 Pending 인 이유 ===
|
||||
Events:
|
||||
Type Reason Age From Message
|
||||
---- ------ ---- ---- -------
|
||||
Warning FailedScheduling 4m45s default-scheduler 0/2 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
|
||||
|
||||
=== keycloak-0 대체 파드가 안 생기는 이유 (StatefulSet) ===
|
||||
keycloak 2 <none> 1
|
||||
keycloak-0 1/1 Terminating 0 30m
|
||||
keycloak-1 0/1 Running 0 143m
|
||||
|
||||
=== 복구 — 노드 재기동 ===
|
||||
재기동 시각: 12:16:31
|
||||
Domain 'kc-lab-2' started
|
||||
|
||||
+30초 node=Ready | Running 파드 3 개 | 외부 HTTP 503
|
||||
+60초 node=Ready | Running 파드 3 개 | 외부 HTTP 200
|
||||
→ 서비스 복귀
|
||||
@@ -0,0 +1,19 @@
|
||||
=== 복구 확인 ===
|
||||
keycloak-0 1/1 Running 0 68s
|
||||
keycloak-1 1/1 Running 0 144m
|
||||
postgres-7b474b88c8-9cmsv 1/1 Running 0 4m20s
|
||||
|
||||
=== kc-lab-1(k3s server)에 무엇이 있는가 — 이게 곧 영향 범위다 ===
|
||||
keycloak-lab keycloak-1
|
||||
kube-system coredns-54996dc9b4-8k8fj
|
||||
kube-system helm-install-traefik-crd-q29b5
|
||||
kube-system local-path-provisioner-77b9867795-g27z8
|
||||
kube-system metrics-server-6dc596dfb8-7xxq4
|
||||
kube-system svclb-traefik-5eb6a9a1-qwwk5
|
||||
kube-system traefik-5d6fcf895-wpfhr
|
||||
observability grafana-845b5678cf-b6gvc
|
||||
observability node-exporter-9qk9w
|
||||
observability prometheus-6774f94f7c-pzr2t
|
||||
|
||||
=== Traefik replica 수 (진입점의 단일 장애점인가) ===
|
||||
traefik 1 1
|
||||
@@ -0,0 +1,19 @@
|
||||
=== 컨트롤 플레인 노드(kc-lab-1) 전원 차단 ===
|
||||
차단 시각: 12:18:08
|
||||
Domain 'kc-lab-1' destroyed
|
||||
|
||||
|
||||
+20초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
+40초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
+60초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
+80초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
+100초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
+120초 외부 auth=000 grafana=502 | kubectl: Unable to connect to the server: dial tcp
|
||||
+140초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
+160초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
|
||||
=== 살아 있는 노드에서 직접 확인 — 워크로드는 도는가 ===
|
||||
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
|
||||
e5f777900b762 60e153026e8f5 4 minutes ago Running keycloak 0 640d4dafaefb3 keycloak-0 keycloak-lab
|
||||
|
||||
6
|
||||
@@ -0,0 +1,13 @@
|
||||
=== 컨트롤 플레인 노드 복구 ===
|
||||
재기동: 12:23:39
|
||||
Domain 'kc-lab-1' started
|
||||
|
||||
+30초 외부=502 | kc-lab-1=Ready kc-lab-2=Ready
|
||||
+60초 외부=200 | kc-lab-1=Ready kc-lab-2=Ready
|
||||
→ 서비스 복귀 (총 60초)
|
||||
|
||||
=== 최종 상태 ===
|
||||
keycloak-0 1/1 Running 0 7m57s
|
||||
keycloak-1 1/1 Running 1 (<invalid> ago) 151m
|
||||
postgres-7b474b88c8-9cmsv 1/1 Running 0 11m
|
||||
(prometheus port-forward 재연결 필요)
|
||||
@@ -0,0 +1,23 @@
|
||||
# A-4 — 노드 전원 차단 증거
|
||||
|
||||
2026-09-04 12:07–12:24 KST
|
||||
해설: [`docs/experiment-a4-node-loss.md`](../../experiment-a4-node-loss.md)
|
||||
|
||||
| 파일 | 무엇을 보여주는가 |
|
||||
|---|---|
|
||||
| `01-baseline.txt` | 차단 전 — 양쪽 Ready, **PVC 가 kc-lab-2 에 못박혀 있음**(재배치 불가의 원인), 외부 200 |
|
||||
| `02-worker-node-killed.txt` | `virsh destroy kc-lab-2` — **40초간 노드가 Ready 로 남아 있고** 외부는 이미 `000`. 이후 `503` |
|
||||
| `03-state-during-loss.txt` | **죽은 파드가 `ready=true`, 산 파드가 `ready=false`.** `up` 은 정확히 0. `unreachable` taint |
|
||||
| `04-eviction-timing.txt` | `tolerationSeconds=300` — **5분 뒤** 축출, 새 postgres 는 `Pending` |
|
||||
| `05-recovery.txt` | `FailedScheduling: didn't match PersistentVolume's node affinity`, StatefulSet `DESIRED=2 CURRENT=1`. 노드 복귀 후 **60초** |
|
||||
| `06-control-plane-inventory.txt` | kc-lab-1 에 있는 것 목록 — **Traefik `replicas=1`** |
|
||||
| `07-control-plane-loss.txt` | `virsh destroy kc-lab-1` — 외부 `000`, `kubectl` 불통. **그런데 `crictl ps` 로 보면 keycloak-0 은 Running** |
|
||||
| `08-control-plane-recovery.txt` | 60초 만에 복귀 |
|
||||
| `a4-up-dropped-per-node.png` | Grafana — `up` 이 노드별로 0 으로 떨어지는 구간. 12:18–12:23 은 **0 이 아니라 데이터 없음**(관측자가 같이 죽음) |
|
||||
|
||||
## 핵심 네 줄
|
||||
|
||||
1. **쿠버네티스는 40초 동안 노드가 살아 있다고 말한다.** 사용자는 이미 장애를 겪는 중이다.
|
||||
2. **죽은 파드의 상태는 화석이다.** `ready=true` 인 파드가 꺼진 기계 위에 있다.
|
||||
3. **StatefulSet 은 대체 파드를 만들지 않고, PVC 는 재배치를 막는다.** 사람이 개입해야 한다.
|
||||
4. **컨트롤 플레인 상실 ≠ 워크로드 상실.** 컨테이너는 계속 돌고, 들어갈 문만 사라진다.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 79 KiB |
@@ -0,0 +1,42 @@
|
||||
=== A-5 기준선 ===
|
||||
keycloak-0=10.42.1.77 (kc-lab-2) keycloak-1=10.42.0.42 (kc-lab-1)
|
||||
keycloak-0 1/1 Running 0 11m
|
||||
keycloak-1 1/1 Running 1 (2m48s ago) 155m
|
||||
postgres-7b474b88c8-9cmsv 1/1 Running 0 14m
|
||||
Traceback (most recent call last):
|
||||
File "<string>", line 3, in <module>
|
||||
for r in json.load(sys.stdin)["data"]["result"]: print(f" cluster_size {r["metric"].get("pod"):12} = {r["value"][1]}")
|
||||
~~~~~~~~~^^^^^^^^^^^
|
||||
File "/usr/lib/python3.14/json/__init__.py", line 298, in load
|
||||
return loads(fp.read(),
|
||||
cls=cls, object_hook=object_hook,
|
||||
parse_float=parse_float, parse_int=parse_int,
|
||||
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
|
||||
File "/usr/lib/python3.14/json/__init__.py", line 352, in loads
|
||||
return _default_decoder.decode(s)
|
||||
~~~~~~~~~~~~~~~~~~~~~~~^^^
|
||||
File "/usr/lib/python3.14/json/decoder.py", line 345, in decode
|
||||
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
|
||||
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/lib/python3.14/json/decoder.py", line 363, in raw_decode
|
||||
raise JSONDecodeError("Expecting value", s, err.value) from None
|
||||
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
|
||||
|
||||
=== 주입: keycloak-0 으로 들어가는 7800/57800 만 DROP (한 방향) ===
|
||||
A-1 의 NetworkPolicy 는 conntrack ESTABLISHED 에 막혀 기존 연결을 못 끊었다.
|
||||
FORWARD 최상단에 넣으면 conntrack 승인보다 먼저 평가된다.
|
||||
Chain FORWARD (policy ACCEPT)
|
||||
num target prot opt source destination
|
||||
1 DROP 6 -- 0.0.0.0/0 10.42.1.77 tcp dpt:57800
|
||||
2 DROP 6 -- 0.0.0.0/0 10.42.1.77 tcp dpt:7800
|
||||
주입 시각: 12:28:23
|
||||
|
||||
=== 관찰 — 어느 쪽이 먼저 상대를 의심하는가 ===
|
||||
+25초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+50초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+75초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+100초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+125초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+150초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+175초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+200초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
@@ -0,0 +1,16 @@
|
||||
=== 실패한 규칙 제거 ===
|
||||
제거완료
|
||||
|
||||
=== raw 테이블 PREROUTING 에 넣는다 — conntrack 보다 먼저 실행된다 ===
|
||||
netfilter 순서: raw PREROUTING → conntrack → mangle → nat → filter
|
||||
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
|
||||
num pkts bytes target prot opt in out source destination
|
||||
1 0 0 DROP 6 -- * * 0.0.0.0/0 10.42.1.77 tcp dpt:57800
|
||||
2 0 0 DROP 6 -- * * 0.0.0.0/0 10.42.1.77 tcp dpt:7800
|
||||
주입: 12:32:35
|
||||
|
||||
=== [검증] 이번엔 패킷이 걸렸는가 ===
|
||||
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
|
||||
pkts bytes target prot opt in out source destination
|
||||
0 0 DROP 6 -- * * 0.0.0.0/0 10.42.1.77 tcp dpt:57800
|
||||
0 0 DROP 6 -- * * 0.0.0.0/0 10.42.1.77 tcp dpt:7800
|
||||
@@ -0,0 +1,18 @@
|
||||
=== keycloak-1(수신측)으로 들어가는 7800/57800 만 DROP — kc-lab-1 에 넣는다 ===
|
||||
주입: 12:33:58
|
||||
|
||||
=== [검증] 패킷이 걸리는가 ===
|
||||
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
|
||||
pkts bytes target prot opt in out source destination
|
||||
0 0 DROP 6 -- * * 0.0.0.0/0 10.42.0.42 tcp dpt:57800
|
||||
19 2938 DROP 6 -- * * 0.0.0.0/0 10.42.0.42 tcp dpt:7800
|
||||
|
||||
=== 관찰 ===
|
||||
+25초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+50초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+75초 - | keycloak-0:1/1 keycloak-1:0/1 | 외부 200
|
||||
+100초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+125초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+150초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+175초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+200초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
@@ -0,0 +1,19 @@
|
||||
=== 지금 연결 방향은? (차단은 → 10.42.0.42:7800 만) ===
|
||||
tcp 6 299 ESTABLISHED src=10.42.0.42 dst=10.42.1.77 sport=48473 dport=7800 src=10.42.1.77 dst=10.42.0.42 sport=7800 dport=48473
|
||||
tcp 6 86232 ESTABLISHED src=10.42.0.42 dst=10.42.1.77 sport=44205 dport=57800 src=10.42.1.77 dst=10.42.0.42 sport=57800 dport=44205
|
||||
|
||||
=== 차단 규칙 누적 카운터 ===
|
||||
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
|
||||
pkts bytes target prot opt in out source destination
|
||||
19 1096 DROP 6 -- * * 0.0.0.0/0 10.42.0.42 tcp dpt:57800
|
||||
21 3058 DROP 6 -- * * 0.0.0.0/0 10.42.0.42 tcp dpt:7800
|
||||
|
||||
=== 클러스터 멤버 (파드에서 직접) ===
|
||||
keycloak-0 cluster_size=2.0
|
||||
keycloak-1 cluster_size=2.0
|
||||
|
||||
=== 최근 뷰 로그 ===
|
||||
keycloak-0: [keycloak-0-24309(v=16.0.12)|12] (1) [keycloak-0-24309(v=16.0.12)]
|
||||
|
||||
keycloak-1: [keycloak-1-45480(v=16.0.12)|12] (1) [keycloak-1-45480(v=16.0.12)]
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
=== 뷰 변화 전체 이력 (차단 12:33:58 전후) ===
|
||||
--- keycloak-0 ---
|
||||
2026-09-04 03:33:49 | during merge: CacheTopology{id=37, phase=NO_REBALANCE, rebalanceId=11, currentCH=DefaultConsistentHash{ns=25
|
||||
2026-09-04 03:33:49,398 INFO [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=offlineSessions] ISPN100007: After me
|
||||
2026-09-04 03:33:49 | during merge: CacheTopology{id=37, phase=NO_REBALANCE, rebalanceId=11, currentCH=DefaultConsistentHash{ns=25
|
||||
2026-09-04 03:33:49,399 INFO [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=loginFailures] ISPN100007: After merg
|
||||
2026-09-04 03:33:49 | during merge: CacheTopology{id=37, phase=NO_REBALANCE, rebalanceId=11, currentCH=DefaultConsistentHash{ns=25
|
||||
2026-09-04 03:33:49,402 INFO [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=actionTokens] ISPN100007: After merge
|
||||
--- keycloak-1 ---
|
||||
2026-09-04 03:32:59,874 INFO [org.infinispan.CLUSTER] (non-blocking-thread--p2-t2) [Context=work] ISPN100007: After merge (or coo
|
||||
2026-09-04 03:32:59,879 INFO [org.infinispan.CLUSTER] (non-blocking-thread--p2-t2) [Context=clientSessions] ISPN100007: After mer
|
||||
2026-09-04 03:32:59,879 INFO [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=offlineSessions] ISPN100007: After me
|
||||
2026-09-04 03:32:59,883 INFO [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=loginFailures] ISPN100007: After merg
|
||||
2026-09-04 03:32:59,884 INFO [org.infinispan.CLUSTER] (non-blocking-thread--p2-t2) [Context=actionTokens] ISPN100007: After merge
|
||||
2026-09-04 03:33:49 | MergeView::[keycloak-0-24309(v=16.0.12)|13] (2) [keycloak-0-24309(v=16.0.12), keycloak-1-45480(v=16.0.12)],
|
||||
|
||||
=== MERGE3 병합 이벤트 ===
|
||||
keycloak-0 merge_events=1.0 suspected=0.0
|
||||
keycloak-1 merge_events=1.0 suspected=0.0
|
||||
|
||||
=== 차단 해제 ===
|
||||
해제완료
|
||||
keycloak-0 cluster_size=2.0
|
||||
warning: couldn't attach to pod/a5-f, falling back to streaming logs: unable to upgrade connection: container a5-f not found in pod a5-f_keycloak-lab
|
||||
keycloak-0 cluster_size=2.0
|
||||
keycloak-1 cluster_size=2.0
|
||||
keycloak-0 1/1 Running 0 22m
|
||||
keycloak-1 1/1 Running 1 (13m ago) 166m
|
||||
@@ -0,0 +1,17 @@
|
||||
=== 양방향 차단 — 두 노드 모두에 raw DROP ===
|
||||
주입: 12:40:25
|
||||
+25초 keycloak-0:1/1 keycloak-1:1/1 | ready=[10.42.0.42 10.42.1.77] 외부 200
|
||||
+50초 keycloak-0:1/1 keycloak-1:1/1 | ready=[10.42.0.42 10.42.1.77] 외부 200
|
||||
+75초 keycloak-0:1/1 keycloak-1:1/1 | ready=[10.42.0.42 10.42.1.77] 외부 200
|
||||
+100초 keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
|
||||
+125초 keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
|
||||
+150초 keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
|
||||
+175초 keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
|
||||
+200초 keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
|
||||
+225초 keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
|
||||
|
||||
=== 뷰 상태 ===
|
||||
keycloak-0 [keycloak-0-24309(v=16.0.12)|14] (1) [keycloak-0-24309(v=16.0.12)]
|
||||
|
||||
keycloak-1 [keycloak-1-45480(v=16.0.12)|14] (1) [keycloak-1-45480(v=16.0.12)]
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
=== 왜 한쪽만 DOWN 인가 — 코디네이터 여부 확인 ===
|
||||
name | ip | coord
|
||||
------------------+-----------------+-------
|
||||
keycloak-0-24309 | 10.42.1.77:7800 | t
|
||||
keycloak-1-45480 | 10.42.0.42:7800 | t
|
||||
(2 rows)
|
||||
|
||||
=== 양쪽 헬스체크 상세 ===
|
||||
--- keycloak-0 ---
|
||||
{"status":"UP","checks":[{"name":"GracefulShutdown","status":"UP"}
|
||||
{"name":"KeycloakInitialized","status":"UP"}
|
||||
{"name":"Keycloakclusterhealthcheck","status":"UP"}
|
||||
|
||||
--- keycloak-1 ---
|
||||
{"status":"DOWN","checks":[{"name":"GracefulShutdown","status":"UP"}
|
||||
{"name":"Keycloakdatabaseconnectionsasynchealthcheck","status":"UP"}
|
||||
{"name":"KeycloakInitialized","status":"UP"}
|
||||
|
||||
warning: couldn't attach to pod/a5-h, falling back to streaming logs: Internal error occurred: Internal error occurred: error attaching to container: container is in CONTAINER_EXITED state
|
||||
--- keycloak-0 ---
|
||||
{"status":"UP","checks":[{"name":"GracefulShutdown","status":"UP"}
|
||||
{"name":"KeycloakInitialized","status":"UP"}
|
||||
{"name":"Keycloakclusterhealthcheck","status":"UP"}
|
||||
|
||||
--- keycloak-1 ---
|
||||
{"status":"DOWN","checks":[{"name":"GracefulShutdown","status":"UP"}
|
||||
{"name":"Keycloakdatabaseconnectionsasynchealthcheck","status":"UP"}
|
||||
{"name":"KeycloakInitialized","status":"UP"}
|
||||
|
||||
|
||||
=== 차단 해제 ===
|
||||
해제: 12:44:37
|
||||
+25초 keycloak-0:1/1 keycloak-1:0/1
|
||||
+50초 keycloak-0:1/1 keycloak-1:1/1
|
||||
→ 복구 완료
|
||||
keycloak-0 MergeView::[keycloak-0-24309(v=16.0.12)|15] (2) [keycloak-0-24309(v=16.0.12), keycloak-1-45480(
|
||||
|
||||
keycloak-1 MergeView::[keycloak-0-24309(v=16.0.12)|15] (2) [keycloak-0-24309(v=16.0.12), keycloak-1-45480(
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
# A-5 — 비대칭 파티션 증거
|
||||
|
||||
2026-09-04 12:28–12:46 KST
|
||||
해설: [`docs/experiment-a5-asymmetric-partition.md`](../../experiment-a5-asymmetric-partition.md)
|
||||
|
||||
| 파일 | 무엇을 보여주는가 |
|
||||
|---|---|
|
||||
| `01-injection.txt` | 첫 시도 — `iptables -I FORWARD 1` |
|
||||
| `02-injection-verify.txt` | **실패 확인** — kube-router 가 자기 체인을 위로 재삽입, DROP 규칙 **0 패킷** |
|
||||
| `03-raw-table-injection.txt` | `raw` 테이블로 이동. 그래도 0 패킷 |
|
||||
| `04-correct-direction.txt` | **연결 방향이 반대였다.** 수신측 노드로 옮기니 **19/21 패킷 차단** |
|
||||
| `05-reconnect-observed.txt` | **핵심** — 연결이 **열린 방향으로 뒤집혀 재연결**. 뷰 변화 없음, `suspected=0` |
|
||||
| `06-view-history-and-cleanup.txt` | 뷰 ID 이력. 주입 이후 변화 없음 확인 |
|
||||
| `07-bidirectional-block.txt` | 양방향 차단 → **뷰 14, 멤버 1개씩.** 그런데 `ready=[10.42.1.77]` 로 **한쪽은 남고 외부 200** |
|
||||
| `08-coordinator-and-recovery.txt` | **`coord=t` 가 둘**(split brain). keycloak-0 `UP` / keycloak-1 `DOWN`. 해제 후 50초, `MergeView |15] (2)` |
|
||||
| `a5-cluster-size-bidirectional-block.png` | Grafana — `vendor_cluster_size` 가 갈라졌다 합쳐지는 구간 |
|
||||
|
||||
## 핵심 네 줄
|
||||
|
||||
1. **주입을 세 번 실패했다** — CNI 체인 경쟁, 연결 방향 오판, conntrack. 셋 다 "아무 일 없음"으로 보였다.
|
||||
2. **단방향 차단으로는 못 가른다.** JGroups 가 열린 방향으로 재연결한다 (자가 치유).
|
||||
3. **양방향이면 갈라진다** — `coord=t` 가 둘.
|
||||
4. **그래도 전면 장애가 아니다.** 코디네이터 쪽만 UP 을 유지해 서비스가 계속된다 — A-1 의 열린 질문에 대한 답.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 65 KiB |
@@ -0,0 +1,397 @@
|
||||
# A-4 — 노드가 통째로 죽으면 어떻게 되는가
|
||||
|
||||
브랜치 `feature/keycloak-a4-node-loss` ·
|
||||
증거 [`docs/evidence/a4-node-loss/`](evidence/a4-node-loss/) ·
|
||||
2026-09-04 12:07–12:24 KST
|
||||
|
||||
선행: [`A-2`](experiment-a2-database-loss.md) · [`A-3`](experiment-a3-database-crash.md)
|
||||
|
||||
**두 판본으로 나눠 실행했다.**
|
||||
|
||||
| | 죽인 노드 | 성격 |
|
||||
|---|---|---|
|
||||
| **4a** | `kc-lab-2` (워커) | Keycloak + **DB 동시 상실** |
|
||||
| **4b** | `kc-lab-1` (k3s 서버) | **컨트롤 플레인 + 진입점 상실** |
|
||||
|
||||
`virsh destroy` 는 종료 신호를 보내지 않는다. **전원을 뽑는 것과 같다.**
|
||||
|
||||
---
|
||||
|
||||
## 0. 결론부터
|
||||
|
||||
| | 4a 워커 상실 | 4b 컨트롤 플레인 상실 |
|
||||
|---|---|---|
|
||||
| 외부 응답 | **503** | **000** (연결 자체가 안 됨) |
|
||||
| `kubectl` | 정상 | **불통** |
|
||||
| 살아 있는 워크로드 | keycloak-1 (하지만 DB 없음) | **keycloak-0 은 계속 돌고 있다** |
|
||||
| 복구 시간 | **60초** | **60초** |
|
||||
|
||||
**둘 다 전면 장애**지만 이유가 다르다. 4a 는 **DB 가 같이 죽어서**, 4b 는
|
||||
**들어갈 길이 없어서**다.
|
||||
|
||||
그리고 예상하지 못한 것 셋 —
|
||||
**죽은 파드가 산 파드보다 건강해 보이고**, **StatefulSet 은 대체 파드를 만들지
|
||||
않으며**, **PVC 때문에 재배치가 불가능하다.**
|
||||
|
||||
---
|
||||
|
||||
## 4a. 워커 노드 상실 (`kc-lab-2`)
|
||||
|
||||
### 기준선
|
||||
|
||||
```
|
||||
kc-lab-1 Ready control-plane=true
|
||||
kc-lab-2 Ready <none>
|
||||
|
||||
keycloak-0 ready=true kc-lab-2
|
||||
keycloak-1 ready=true kc-lab-1
|
||||
postgres ready=true kc-lab-2
|
||||
|
||||
PVC postgres-data → kc-lab-2 ← 재배치 가능성을 여기서 이미 알 수 있다
|
||||
외부 HTTP 200
|
||||
```
|
||||
|
||||
### 주입
|
||||
|
||||
```bash
|
||||
virsh destroy kc-lab-2
|
||||
```
|
||||
|
||||
### 관찰
|
||||
|
||||
```
|
||||
+15초 node=Ready | keycloak-0=Running postgres=Running | 외부 HTTP 000
|
||||
+30초 node=Ready | keycloak-0=Running postgres=Running | 외부 HTTP 000
|
||||
+45초 node=NotReady | keycloak-0=Running postgres=Running | 외부 HTTP 503
|
||||
...
|
||||
+180초 node=NotReady | keycloak-0=Running postgres=Running | 외부 HTTP 503
|
||||
```
|
||||
|
||||
### 발견 ① — 쿠버네티스가 알아채는 데 **40초** 걸린다
|
||||
|
||||
`+15초`, `+30초` 에도 노드는 여전히 `Ready` 다. **기계는 이미 없는데.**
|
||||
|
||||
kube-controller-manager 의 `node-monitor-grace-period` (기본 40초) 동안
|
||||
kubelet 의 하트비트가 없어야 `NotReady` 로 바꾼다.
|
||||
|
||||
> **그 40초 동안 쿠버네티스는 거짓말을 한다.** 사용자는 이미 장애를 겪고 있다
|
||||
> (`000`). **노드 상태를 알림 근거로 삼으면 항상 늦는다.**
|
||||
|
||||
### 발견 ② — **죽은 파드가 산 파드보다 건강해 보인다**
|
||||
|
||||
```
|
||||
POD PHASE READY NODE
|
||||
keycloak-0 Running true kc-lab-2 ← 기계가 꺼져 있다
|
||||
keycloak-1 Running false kc-lab-1 ← 살아 있다
|
||||
```
|
||||
|
||||
**`keycloak-0` 은 `ready=true`, `keycloak-1` 은 `ready=false`.**
|
||||
|
||||
| 왜 | |
|
||||
|---|---|
|
||||
| keycloak-0 | kubelet 이 없어 **상태를 갱신할 수 없다.** 마지막으로 보고한 값이 그대로 얼어 있다 |
|
||||
| keycloak-1 | 살아서 **정직하게 보고한다** — DB 가 없으니 readiness 실패 |
|
||||
|
||||
> **파드 상태는 "지금 어떤가"가 아니라 "마지막으로 그렇게 들었다"이다.**
|
||||
> 노드가 죽으면 그 노드 파드의 상태는 **화석**이 된다.
|
||||
|
||||
Prometheus 는 정확했다.
|
||||
|
||||
```
|
||||
up{job=keycloak pod=keycloak-1} = 1
|
||||
up{job=keycloak pod=keycloak-0} = 0 ← 긁기에 실패했다 = 사실
|
||||
up{job=kubelet } = 0
|
||||
up{job=node-exporter node=kc-lab-2 } = 0
|
||||
```
|
||||
|
||||
**A-2 와 정반대다.** A-2 에서는 `up=1` 인데 서비스가 죽었고, 여기서는 `up=0`
|
||||
이 정확했다.
|
||||
|
||||
| | `up` 이 잡는가 |
|
||||
|---|---|
|
||||
| **대상이 사라짐** (노드 상실) | **잡는다** |
|
||||
| **대상이 살아서 못 씀** (DB 상실) | **못 잡는다** |
|
||||
|
||||
|
||||
|
||||
### 발견 ③ — 축출은 **5분** 뒤에 시작된다
|
||||
|
||||
쿠버네티스가 노드에 taint 를 붙인다.
|
||||
|
||||
```
|
||||
node.kubernetes.io/unreachable=:NoSchedule
|
||||
node.kubernetes.io/unreachable=:NoExecute
|
||||
```
|
||||
|
||||
그런데 모든 파드에는 기본 관용이 붙어 있다.
|
||||
|
||||
```
|
||||
node.kubernetes.io/not-ready NoExecute tolerationSeconds=300
|
||||
node.kubernetes.io/unreachable NoExecute tolerationSeconds=300
|
||||
```
|
||||
|
||||
```
|
||||
+240초 전부 Running
|
||||
+270초 a2-probe:Terminating keycloak-0:Terminating postgres:Terminating
|
||||
postgres-...-9cmsv:Pending ← 새 파드가 생겼다
|
||||
```
|
||||
|
||||
**노드가 죽고 약 5분이 지나야 파드가 축출된다.** 잠깐 끊긴 노드가 돌아올 수도
|
||||
있으니 성급하게 옮기지 않겠다는 설계다. 대신 **그동안은 아무 일도 일어나지
|
||||
않는다.**
|
||||
|
||||
### 발견 ④ — 축출돼도 **갈 곳이 없다**
|
||||
|
||||
```
|
||||
Warning FailedScheduling default-scheduler
|
||||
0/2 nodes are available:
|
||||
1 node(s) didn't match PersistentVolume's node affinity,
|
||||
1 node(s) had untolerated taint(s).
|
||||
```
|
||||
|
||||
```
|
||||
kc-lab-2 → taint 때문에 못 간다 (죽은 노드)
|
||||
kc-lab-1 → PVC 가 kc-lab-2 에 묶여 있어 못 간다
|
||||
```
|
||||
|
||||
**`local-path` PVC 는 그 노드의 로컬 디스크 디렉터리다.** 노드가 죽으면
|
||||
볼륨도 죽는다. 새 파드는 **영원히 Pending** 이다.
|
||||
|
||||
> 결함이 아니라 **조건**이다. 이 실험대는 그걸 알고 `local-path` 를 골랐다.
|
||||
> 운영이라면 네트워크 스토리지나 DB 복제가 이 자리를 메워야 한다.
|
||||
|
||||
### 발견 ⑤ — StatefulSet 은 **대체 파드를 만들지 않는다**
|
||||
|
||||
```
|
||||
NAME DESIRED READY CURRENT
|
||||
keycloak 2 <none> 1
|
||||
|
||||
keycloak-0 1/1 Terminating 30m ← 30분째
|
||||
keycloak-1 0/1 Running
|
||||
```
|
||||
|
||||
**`keycloak-0` 이 `Terminating` 에서 영원히 멈춰 있고, 대체 파드가 안 생긴다.**
|
||||
|
||||
| 왜 | |
|
||||
|---|---|
|
||||
| StatefulSet 의 계약 | **같은 이름의 파드는 클러스터에 하나뿐**이어야 한다 |
|
||||
| 컨트롤 플레인이 아는 것 | 노드가 안 보인다 = **파드가 죽었는지 확신할 수 없다** |
|
||||
| 그래서 | 옛 파드를 확실히 지우기 전엔 새 `keycloak-0` 을 못 만든다 |
|
||||
|
||||
**Deployment 였다면 즉시 새 파드를 만든다.** 이름이 아무래도 되기 때문이다.
|
||||
StatefulSet 의 "안정된 이름"이라는 이득의 **반대편 비용**이 여기다.
|
||||
|
||||
강제로 진행시키려면 사람이 개입해야 한다.
|
||||
|
||||
```bash
|
||||
kubectl -n keycloak-lab delete pod keycloak-0 --grace-period=0 --force
|
||||
```
|
||||
|
||||
**위험한 명령이다.** 노드가 사실은 살아 있고 네트워크만 끊긴 것이라면
|
||||
**같은 이름의 파드 둘이 동시에 존재**하게 된다 (split brain).
|
||||
|
||||
### 복구
|
||||
|
||||
```bash
|
||||
virsh start kc-lab-2
|
||||
```
|
||||
|
||||
```
|
||||
+30초 node=Ready | Running 파드 3 개 | 외부 HTTP 503
|
||||
+60초 node=Ready | Running 파드 3 개 | 외부 HTTP 200
|
||||
→ 서비스 복귀
|
||||
```
|
||||
|
||||
**60초.** 사람 개입 없이 전부 제자리로 돌아왔다.
|
||||
|
||||
---
|
||||
|
||||
## 4b. 컨트롤 플레인 노드 상실 (`kc-lab-1`)
|
||||
|
||||
### 이 노드에 무엇이 있었나 — 그게 곧 영향 범위다
|
||||
|
||||
```
|
||||
keycloak-lab keycloak-1
|
||||
kube-system coredns
|
||||
kube-system local-path-provisioner
|
||||
kube-system metrics-server
|
||||
kube-system svclb-traefik
|
||||
kube-system traefik ← replicas=1
|
||||
observability grafana
|
||||
observability prometheus
|
||||
observability node-exporter
|
||||
```
|
||||
|
||||
```
|
||||
NAME REPLICAS READY
|
||||
traefik 1 1 ← 진입점이 한 개다
|
||||
```
|
||||
|
||||
### 주입과 관찰
|
||||
|
||||
```bash
|
||||
virsh destroy kc-lab-1
|
||||
```
|
||||
|
||||
```
|
||||
+20초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server
|
||||
+60초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server
|
||||
+120초 외부 auth=000 grafana=502 | kubectl: Unable to connect to the server
|
||||
+160초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server
|
||||
```
|
||||
|
||||
### 발견 ⑥ — 워크로드는 살아 있는데 **길이 없다**
|
||||
|
||||
살아남은 노드에서 직접 확인했다.
|
||||
|
||||
```bash
|
||||
ssh kc-lab-2 'sudo crictl ps --name keycloak'
|
||||
```
|
||||
|
||||
```
|
||||
CONTAINER STATE NAME POD NAMESPACE
|
||||
e5f777900b76 Running keycloak keycloak-0 keycloak-lab
|
||||
```
|
||||
|
||||
**`keycloak-0` 은 멀쩡히 돌고 있다.** 컨테이너 런타임(containerd)은 API 서버
|
||||
없이도 이미 떠 있는 파드를 계속 굴린다.
|
||||
|
||||
```
|
||||
죽은 것: API 서버 · 스케줄러 · coredns · Traefik · Prometheus · Grafana
|
||||
산 것: keycloak-0 · postgres · containerd
|
||||
문제: 들어갈 문(Traefik)이 없다
|
||||
```
|
||||
|
||||
> **컨트롤 플레인 상실은 워크로드 상실이 아니다.**
|
||||
> 이미 떠 있는 것은 계속 돈다. **새로 뜨거나 옮기거나 고치는 것이 안 될 뿐.**
|
||||
|
||||
### 발견 ⑦ — `000` 과 `503` 의 차이
|
||||
|
||||
| 코드 | 뜻 |
|
||||
|---|---|
|
||||
| **`503`** (4a) | nginx·Traefik 은 살아 있고 **뒤에 보낼 파드가 없다** |
|
||||
| **`000`** (4b) | curl 이 응답 자체를 못 받았다 — **Traefik 이 없어 nginx 가 죽은 주소를 기다리다 타임아웃** |
|
||||
|
||||
`grafana=502` 가 한 번 찍힌 것이 증거다 — nginx 는 살아서 502 를 만들 수
|
||||
있었지만, 대부분은 타임아웃(`--max-time 8`)에 걸렸다.
|
||||
|
||||
**두 코드가 다른 층의 고장을 가리킨다.**
|
||||
|
||||
### 발견 ⑧ — 관측 시스템이 같이 죽으면 **0 이 아니라 구멍이 남는다**
|
||||
|
||||
Prometheus 가 `kc-lab-1` 에 있었다. 그 노드가 죽은 12:18–12:23 구간은
|
||||
그래프에 **0 이 아니라 데이터 없음**으로 남는다.
|
||||
|
||||
```
|
||||
대상이 죽음 → up = 0 → "언제 죽었는지" 알 수 있다
|
||||
관측자가 죽음 → 데이터 없음 → "그때 무슨 일이 있었는지" 모른다
|
||||
```
|
||||
|
||||
**계획 단계에서 "관측 스택은 죽이지 않는 노드에 둔다"고 정했던 이유**를
|
||||
실제로 확인한 셈이다. 다만 노드가 둘뿐이라 완전히 피할 수는 없다.
|
||||
|
||||
### 복구
|
||||
|
||||
```bash
|
||||
virsh start kc-lab-1
|
||||
```
|
||||
|
||||
```
|
||||
+30초 외부=502 | kc-lab-1=Ready kc-lab-2=Ready
|
||||
+60초 외부=200 | kc-lab-1=Ready kc-lab-2=Ready
|
||||
→ 서비스 복귀 (총 60초)
|
||||
```
|
||||
|
||||
**Grafana 는 로그인 세션을 잃었다** — 데이터가 `emptyDir` 이라 파드 재시작에
|
||||
사라진다. Prometheus 는 PVC 라 지표가 남았다. **의도한 설계대로 동작했다.**
|
||||
|
||||
---
|
||||
|
||||
## 5. 개념
|
||||
|
||||
### `node-monitor-grace-period` 와 `tolerationSeconds`
|
||||
|
||||
```
|
||||
기계 정지
|
||||
│
|
||||
│ 40초 node-monitor-grace-period → 노드 NotReady
|
||||
│
|
||||
│ +300초 tolerationSeconds (NoExecute) → 파드 축출 시작
|
||||
│
|
||||
▼
|
||||
총 약 5분 40초 동안 쿠버네티스는 아무것도 하지 않는다
|
||||
```
|
||||
|
||||
**빠른 장애 대응이 필요하면 이 값을 줄여야 하지만**, 짧게 하면 일시적
|
||||
네트워크 끊김에도 파드를 옮기게 되어 **불안정해진다.** 맞바꿈이다.
|
||||
|
||||
### `Terminating` 이 안 끝나는 이유
|
||||
|
||||
```
|
||||
파드 삭제 요청
|
||||
└─ kubelet 이 컨테이너를 멈추고 "지웠다"고 보고해야 끝난다
|
||||
└─ kubelet 이 없다 → 보고가 없다 → 영원히 Terminating
|
||||
```
|
||||
|
||||
`--grace-period=0 --force` 는 **보고를 기다리지 않고 API 에서 지우는 것**이다.
|
||||
컨테이너가 실제로 죽었는지는 **모른 채** 진행한다.
|
||||
|
||||
### StatefulSet vs Deployment — 노드 상실에서
|
||||
|
||||
| | Deployment | StatefulSet |
|
||||
|---|---|---|
|
||||
| 대체 파드 | **즉시 만든다** (이름 무관) | **안 만든다** (이름이 유일해야) |
|
||||
| 노드 상실 시 | 자동 회복 | **사람이 강제 삭제해야** |
|
||||
| 위험 | 없음 | 강제 삭제 시 **중복 실행 가능** |
|
||||
|
||||
**A-0 에서 "로그 대조가 쉬워서" StatefulSet 을 골랐는데, 그 대가가 여기 있다.**
|
||||
|
||||
### `local-path` PVC 와 재배치
|
||||
|
||||
```
|
||||
PVC → PV → nodeAffinity: kubernetes.io/hostname = kc-lab-2
|
||||
└─ 그 노드의 /var/lib/rancher/k3s/storage/... 디렉터리
|
||||
```
|
||||
|
||||
**볼륨이 노드에 못박히면 파드도 못박힌다.** 노드가 죽으면 데이터도 함께
|
||||
접근 불가가 된다.
|
||||
|
||||
---
|
||||
|
||||
## 6. 재현 절차 (명령어)
|
||||
|
||||
```bash
|
||||
export LIBVIRT_DEFAULT_URI=qemu:///system
|
||||
|
||||
# 사전 — PVC 가 어디 묶여 있는지 먼저 본다 (재배치 가능성이 여기서 정해진다)
|
||||
kubectl -n keycloak-lab get pvc -o name | while read p; do
|
||||
V=$(kubectl -n keycloak-lab get $p -o jsonpath='{.spec.volumeName}')
|
||||
kubectl get pv $V -o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]}'
|
||||
done
|
||||
|
||||
# 4a 워커 상실
|
||||
virsh destroy kc-lab-2
|
||||
kubectl get node kc-lab-2 # 40초 뒤 NotReady
|
||||
kubectl -n keycloak-lab get pods -o wide # Running 인 채로 얼어 있다
|
||||
kubectl get node kc-lab-2 -o jsonpath='{.spec.taints}'
|
||||
# 5분 뒤 Terminating + 새 파드 Pending
|
||||
kubectl -n keycloak-lab describe pod <new-pod> | grep -A4 Events
|
||||
|
||||
# 4b 컨트롤 플레인 상실 — kubectl 이 죽으므로 노드에서 직접 본다
|
||||
virsh destroy kc-lab-1
|
||||
ssh kc-lab-2 'sudo crictl ps' # 워크로드는 살아 있다
|
||||
|
||||
# 복구
|
||||
virsh start kc-lab-2 && virsh start kc-lab-1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. 다음 실험에 남기는 것
|
||||
|
||||
| 실험 | 이 실험이 준 것 |
|
||||
|---|---|
|
||||
| **A-5** 비대칭 파티션 | 여기서는 노드가 **완전히** 사라졌다. 부분 단절은 더 고약하다 |
|
||||
| **D-1** 백업·복구 | **PVC 가 노드에 묶여 있다** — 노드가 영영 안 돌아오면 백업이 유일한 길 |
|
||||
| 구성 개선 | **Traefik `replicas=1` 은 진입점 단일 장애점** — 2로 늘리거나 DaemonSet 으로 |
|
||||
| 구성 개선 | **Grafana 가 `emptyDir`** — 재시작마다 로그인 세션이 사라진다 |
|
||||
@@ -0,0 +1,319 @@
|
||||
# A-5 — 한쪽 방향만 끊기면 어떻게 되는가 (split brain)
|
||||
|
||||
브랜치 `feature/keycloak-a5-asymmetric-partition` ·
|
||||
증거 [`docs/evidence/a5-asymmetric-partition/`](evidence/a5-asymmetric-partition/) ·
|
||||
2026-09-04 12:28–12:46 KST
|
||||
|
||||
선행: [`A-1`](experiment-a1-jgroups-transport-block.md) — 이 실험은 A-1 이
|
||||
열어둔 질문에 답한다.
|
||||
|
||||
> **A-1 의 열린 질문** — *"양쪽이 동시에 NotReady 가 되는 경로가 있다면
|
||||
> 전면 장애다. A-5 에서 이어서 본다."*
|
||||
|
||||
---
|
||||
|
||||
## 0. 결론부터
|
||||
|
||||
| 물음 | 답 |
|
||||
|---|---|
|
||||
| 비대칭(한 방향) 차단이 클러스터를 가르는가 | **아니다.** 열린 방향으로 재연결한다 |
|
||||
| 양방향 완전 차단은 가르는가 | **그렇다.** 양쪽 모두 멤버 1개 |
|
||||
| **그때 양쪽이 다 NotReady 가 되는가** | **아니다. 한쪽만 DOWN 이다** |
|
||||
| 그래서 서비스는 | **계속된다.** 외부 200 유지 |
|
||||
|
||||
**전면 장애 경로는 없었다.** 코디네이터 쪽이 항상 살아남는다.
|
||||
|
||||
그리고 주입을 **세 번 실패**했다. 세 번 모두 다른 이유였고, 셋 다
|
||||
**"아무 일도 없었다"로 보였다.**
|
||||
|
||||
---
|
||||
|
||||
## 1. 주입을 세 번 실패했다
|
||||
|
||||
A-1 에서 NetworkPolicy 가 conntrack 의 `ESTABLISHED` 승인에 막혀 기존 연결을
|
||||
못 끊는다는 것을 배웠다. 그래서 이번에는 iptables 로 갔는데, **또 다른 벽이
|
||||
세 개** 있었다.
|
||||
|
||||
### 실패 ① — `iptables -I FORWARD 1` 은 최상단에 남지 않는다
|
||||
|
||||
```bash
|
||||
ssh kc-lab-2 'sudo iptables -I FORWARD 1 -p tcp -d 10.42.1.77 --dport 7800 -j DROP'
|
||||
```
|
||||
|
||||
```
|
||||
num pkts bytes target
|
||||
1 232 377K KUBE-ROUTER-FORWARD /* kube-router netpol */ ← 다시 1번이 되었다
|
||||
2 0 0 DROP tcp dpt:57800
|
||||
3 0 0 DROP tcp dpt:7800 ← 0 패킷
|
||||
```
|
||||
|
||||
**kube-router 가 주기적으로 자기 체인을 최상단에 다시 삽입한다.**
|
||||
내가 1번에 넣어도 곧 2번, 3번으로 밀려난다.
|
||||
|
||||
> **직접 넣은 iptables 규칙은 CNI 가 관리하는 체인과 경쟁한다.**
|
||||
> 넣는 것으로 끝이 아니라 **패킷 카운터로 확인해야 한다.**
|
||||
|
||||
### 실패 ② — 그래도 0 패킷: **연결 방향을 잘못 짚었다**
|
||||
|
||||
`raw` 테이블로 옮겼는데도 0 패킷이었다.
|
||||
|
||||
```bash
|
||||
ssh kc-lab-2 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d 10.42.1.77 --dport 7800 -j DROP'
|
||||
```
|
||||
|
||||
실제 연결을 봤더니
|
||||
|
||||
```
|
||||
ESTABLISHED src=10.42.1.77 dst=10.42.0.42 sport=60485 dport=7800
|
||||
──────────── ────────────────────
|
||||
keycloak-0 가 클라이언트 keycloak-1 이 서버
|
||||
```
|
||||
|
||||
**A-1 때와 방향이 반대였다.** 파드가 재시작되면서 누가 먼저 연결을 걸었는지가
|
||||
바뀌었다.
|
||||
|
||||
> **JGroups 의 TCP 연결 방향은 고정이 아니다.** 먼저 뜬 쪽, 먼저 JOIN 을 건
|
||||
> 쪽에 따라 달라진다. **가정하지 말고 `conntrack -L` 로 봐야 한다.**
|
||||
|
||||
### 성공 — `raw` 테이블 PREROUTING, 수신측 노드에
|
||||
|
||||
```bash
|
||||
ssh kc-lab-1 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d 10.42.0.42 --dport 7800 -j DROP
|
||||
sudo iptables -t raw -I PREROUTING 1 -p tcp -d 10.42.0.42 --dport 57800 -j DROP'
|
||||
```
|
||||
|
||||
```
|
||||
pkts bytes target
|
||||
19 1096 DROP tcp dpt:57800
|
||||
21 3058 DROP tcp dpt:7800 ← 걸린다
|
||||
```
|
||||
|
||||
### 개념 — netfilter 처리 순서
|
||||
|
||||
```
|
||||
패킷 도착
|
||||
│
|
||||
├─▶ raw PREROUTING ← conntrack 보다 먼저. NOTRACK·DROP 용
|
||||
│
|
||||
├─▶ conntrack 조회/생성 ← 여기서 ESTABLISHED 가 결정된다
|
||||
│
|
||||
├─▶ mangle PREROUTING
|
||||
├─▶ nat PREROUTING
|
||||
├─▶ filter FORWARD ← NetworkPolicy·kube-router 가 여기 있다
|
||||
└─▶ 목적지 파드
|
||||
```
|
||||
|
||||
| 어디에 넣는가 | 기존 연결을 끊는가 | CNI 와 경쟁하는가 |
|
||||
|---|---|---|
|
||||
| NetworkPolicy (filter) | **못 끊는다** — conntrack 이 먼저 통과시킨다 | 없음 |
|
||||
| filter FORWARD 직접 | 순서에 따라 | **경쟁한다** (kube-router 가 밀어낸다) |
|
||||
| **raw PREROUTING** | **끊는다** | **없다** — CNI 가 안 쓰는 테이블 |
|
||||
|
||||
**진짜 네트워크 분단을 흉내내려면 `raw` 테이블이 맞다.**
|
||||
|
||||
---
|
||||
|
||||
## 2. 비대칭 차단 — 클러스터가 갈라지지 않았다
|
||||
|
||||
한 방향(`→ keycloak-1:7800`)만 막고 관찰했다.
|
||||
|
||||
```
|
||||
+25초 keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+75초 keycloak-0:1/1 keycloak-1:0/1 | 외부 200 ← 잠깐 흔들림
|
||||
+100초 keycloak-0:1/1 keycloak-1:1/1 | 외부 200 ← 회복
|
||||
...
|
||||
+200초 keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
```
|
||||
|
||||
**주입 이후 뷰 변화가 하나도 없었다.**
|
||||
|
||||
```bash
|
||||
kubectl -n keycloak-lab logs keycloak-0 --since=20m | grep ISPN000094 | awk '$2 >= "03:33:58"'
|
||||
# (아무것도 안 나온다)
|
||||
```
|
||||
|
||||
```
|
||||
뷰 ID 이력: |13] (2) ← 주입 전부터 지금까지 그대로
|
||||
suspected = 0
|
||||
```
|
||||
|
||||
### 왜 안 갈라졌는가 — **연결 방향이 뒤집혔다**
|
||||
|
||||
```
|
||||
차단 전: keycloak-0 → keycloak-1:7800 ← 내가 막은 방향
|
||||
차단 후: keycloak-1 → keycloak-0:7800 ← 열린 방향으로 다시 붙었다
|
||||
```
|
||||
|
||||
```
|
||||
ESTABLISHED src=10.42.0.42 dst=10.42.1.77 sport=48473 dport=7800
|
||||
──────────── ─────────────
|
||||
keycloak-1 이 클라이언트 keycloak-0 이 서버
|
||||
```
|
||||
|
||||
**JGroups 는 막힌 연결이 죽자 반대 방향으로 새로 연결했고, 그 사이 FD_SOCK2 가
|
||||
상대를 의심하기 전에 복구가 끝났다.** `suspected = 0` 이 그 증거다.
|
||||
|
||||
> **한 방향만 막는 것으로는 JGroups 를 가를 수 없다.**
|
||||
> 두 노드는 서로에게 연결을 걸 수 있으므로, **한쪽 길이 막히면 다른 길로 간다.**
|
||||
>
|
||||
> 운영적으로는 좋은 소식이다 — 단방향 방화벽 오설정은 **자가 치유**된다.
|
||||
> 반대로 **분단을 재현하려는 실험자에게는 함정**이다.
|
||||
|
||||
---
|
||||
|
||||
## 3. 양방향 차단 — 갈라졌지만 전면 장애는 아니다
|
||||
|
||||
양쪽 노드 모두에 규칙을 넣었다.
|
||||
|
||||
```
|
||||
+75초 keycloak-0:1/1 keycloak-1:1/1 | ready=[10.42.0.42 10.42.1.77] 외부 200
|
||||
+100초 keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
|
||||
...
|
||||
+225초 keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
|
||||
```
|
||||
|
||||
```
|
||||
=== 뷰 ===
|
||||
keycloak-0 [keycloak-0-24309|14] (1) [keycloak-0-24309]
|
||||
keycloak-1 [keycloak-1-45480|14] (1) [keycloak-1-45480]
|
||||
|
||||
=== JGROUPS_PING ===
|
||||
keycloak-0-24309 | 10.42.1.77:7800 | t ← 코디네이터
|
||||
keycloak-1-45480 | 10.42.0.42:7800 | t ← 코디네이터
|
||||
```
|
||||
|
||||
**`coord = t` 가 둘.** 완전한 split brain 이다.
|
||||
|
||||
|
||||
|
||||
### 그런데 **한쪽만 DOWN 이다**
|
||||
|
||||
```
|
||||
--- keycloak-0 ---
|
||||
{"status":"UP", ... {"name":"Keycloak cluster health check","status":"UP"}
|
||||
|
||||
--- keycloak-1 ---
|
||||
{"status":"DOWN", ... (cluster health check 가 DOWN)
|
||||
```
|
||||
|
||||
| | keycloak-0 | keycloak-1 |
|
||||
|---|---|---|
|
||||
| 분단 전 역할 | **코디네이터** (뷰 13 의 발행자) | 일반 멤버 |
|
||||
| 분단 후 자기 인식 | "멤버가 하나 나갔다" — **정상 사건** | "코디네이터를 잃었다" — **비정상** |
|
||||
| 헬스체크 | **UP** | **DOWN** |
|
||||
| Service 엔드포인트 | **남는다** | 빠진다 |
|
||||
|
||||
> **A-1 의 열린 질문에 대한 답 — 양쪽이 동시에 NotReady 가 되는 경로는 없었다.**
|
||||
>
|
||||
> Keycloak 의 클러스터 헬스체크는 **비대칭**이다. 코디네이터였던 쪽은
|
||||
> 자기가 정상이라고 보고, 잃은 쪽만 DOWN 이 된다. 그래서 완전 분단조차
|
||||
> **용량 저하로 끝나고 전면 장애가 되지 않는다.**
|
||||
>
|
||||
> A-2(DB 상실)에서 양쪽이 동시에 DOWN 이 된 것과 대조된다. **DB 는 모두가
|
||||
> 의존하는 하나지만, 클러스터 멤버십은 서로 상대적**이기 때문이다.
|
||||
|
||||
---
|
||||
|
||||
## 4. 복구
|
||||
|
||||
```bash
|
||||
ssh kc-lab-1 'sudo iptables -t raw -F PREROUTING'
|
||||
ssh kc-lab-2 'sudo iptables -t raw -F PREROUTING'
|
||||
```
|
||||
|
||||
```
|
||||
+25초 keycloak-0:1/1 keycloak-1:0/1
|
||||
+50초 keycloak-0:1/1 keycloak-1:1/1
|
||||
→ 복구 완료
|
||||
|
||||
MergeView::[keycloak-0-24309|15] (2) [keycloak-0-24309, keycloak-1-45480]
|
||||
```
|
||||
|
||||
**50초, 사람 개입 없음.** `MergeView` 로 뷰 15 가 발행되며 병합됐다.
|
||||
|
||||
각 캐시마다 재분배 로그가 남는다.
|
||||
|
||||
```
|
||||
[Context=work] ISPN100007: After merge (or coordinator change) ...
|
||||
[Context=clientSessions] ISPN100007: After merge ...
|
||||
[Context=offlineSessions] ISPN100007: After merge ...
|
||||
[Context=loginFailures] ISPN100007: After merge ...
|
||||
[Context=actionTokens] ISPN100007: After merge ...
|
||||
```
|
||||
|
||||
**`ISPN100007` 은 병합 후 캐시별 토폴로지 재계산**이다. 캐시가 여럿이므로
|
||||
로그도 캐시 수만큼 나온다.
|
||||
|
||||
---
|
||||
|
||||
## 5. 개념
|
||||
|
||||
### `MergeView` 와 뷰 ID
|
||||
|
||||
```
|
||||
[keycloak-0-24309|13] (2) ← 정상
|
||||
[keycloak-0-24309|14] (1) ← 분단. 양쪽이 각자 14 를 발행
|
||||
MergeView::[...|15] (2) ← 병합. 뷰 ID 는 계속 증가한다
|
||||
```
|
||||
|
||||
**뷰 ID 는 단조 증가**하므로 "언제 몇 번 갈라졌는지"를 로그만으로 셀 수 있다.
|
||||
|
||||
### 코디네이터의 비대칭성
|
||||
|
||||
JGroups 코디네이터는 **가장 오래된 멤버**다. 분단이 나면
|
||||
|
||||
```
|
||||
코디네이터 쪽: "멤버가 나갔다" → 정상 처리, 계속 코디네이터
|
||||
나머지 쪽: "코디네이터가 사라졌다" → 새 코디네이터를 자기로 선출
|
||||
```
|
||||
|
||||
**둘 다 자기가 코디네이터라고 믿는 상태**가 split brain 이고,
|
||||
`JGROUPS_PING.coord` 컬럼에 그대로 드러난다.
|
||||
|
||||
### 실험자를 위한 규칙
|
||||
|
||||
| 상황 | 확인 방법 |
|
||||
|---|---|
|
||||
| 규칙을 넣었는데 안 걸린다 | `iptables -L -n -v` 의 **패킷 카운터** |
|
||||
| 방향을 모르겠다 | `conntrack -L \| grep <포트>` |
|
||||
| CNI 가 밀어낸다 | `raw` 테이블을 쓴다 |
|
||||
| 갈라졌는지 알고 싶다 | `vendor_cluster_size`, `ISPN000094`, `JGROUPS_PING.coord` |
|
||||
|
||||
---
|
||||
|
||||
## 6. 재현 절차 (명령어)
|
||||
|
||||
```bash
|
||||
# 1. 지금 연결이 어느 방향인지 먼저 본다 — 가정하면 실패한다
|
||||
ssh kc-lab-1 'sudo conntrack -L | grep 7800'
|
||||
|
||||
# 2. 수신측 노드의 raw PREROUTING 에 넣는다 (filter 는 CNI 와 경쟁한다)
|
||||
ssh kc-lab-1 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d <수신 파드IP> --dport 7800 -j DROP'
|
||||
ssh kc-lab-1 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d <수신 파드IP> --dport 57800 -j DROP'
|
||||
|
||||
# 3. 걸렸는지 카운터로 확인 — 0 이면 해석 금지
|
||||
ssh kc-lab-1 'sudo iptables -t raw -L PREROUTING -n -v'
|
||||
|
||||
# 4. 양방향으로 하려면 반대 노드에도 (한 방향만으로는 자가 치유된다)
|
||||
ssh kc-lab-2 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d <반대 파드IP> --dport 7800 -j DROP'
|
||||
|
||||
# 5. 분단 확인
|
||||
kubectl -n keycloak-lab exec deploy/postgres -- psql -U keycloak -d keycloak \
|
||||
-c "select name, coord from jgroups_ping" # coord=t 가 둘이면 split brain
|
||||
kubectl -n keycloak-lab get endpoints keycloak -o jsonpath='{.subsets[*].addresses[*].ip}'
|
||||
|
||||
# 6. 해제
|
||||
ssh kc-lab-1 'sudo iptables -t raw -F PREROUTING'
|
||||
ssh kc-lab-2 'sudo iptables -t raw -F PREROUTING'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. 다음 실험에 남기는 것
|
||||
|
||||
| 실험 | 이 실험이 준 것 |
|
||||
|---|---|
|
||||
| **A-6** 지연 주입 | `tc netem` 도 같은 함정 — **주입이 걸렸는지 먼저 확인** |
|
||||
| **A-7** volatile 비교 | 여기서는 분단에도 서비스가 계속됐다. volatile 이면 **세션이 갈라진다** |
|
||||
| 운영 | **단방향 방화벽 오설정은 자가 치유된다.** 양방향이어야 사고가 된다 |
|
||||
| 운영 | **완전 분단조차 전면 장애가 아니다** — 코디네이터 쪽이 살아남는다 |
|
||||
Reference in New Issue
Block a user