Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 DongHyeonkaandClaude Opus 5
|
dba0c3975c | ||
|
DongHyeonkaandClaude Opus 5
|
2fa77a899e |
@@ -18,3 +18,108 @@
|
||||
[ 98008ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 104458ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 123816ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 142764ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 157198ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 175220ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 182187ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 183411ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 190581ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 199499ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 209995ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 214849ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 233084ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 242910ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 257975ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 277220ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 290806ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 292674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 300013ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 309267ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 323998ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 328620ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 339787ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 355959ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 361079ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 369272ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 381866ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 397120ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 400712ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 412579ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 431630ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 440746ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 447809ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 461741ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 475459ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 482723ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 494299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 513174ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 521541ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 532145ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 551129ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 556250ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 574381ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 586190ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 595782ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 610555ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 630695ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 633257ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 652194ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 671361ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 677606ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 692444ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 696949ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 700122ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 709954ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 723992ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 743173ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 745075ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 753299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 766683ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 767855ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 787261ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 789725ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 800680ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 811324ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 814096ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 829045ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 848506ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 857716ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 866212ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 873174ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 876033ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 895198ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 898884ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 918642ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 930133ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 934008ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 948541ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 956860ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 959397ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 968406ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 971788ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 982746ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 997394ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1005078ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1009269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1021453ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1026674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1042137ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1057497ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1059851ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1061855ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1071317ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1078179ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1084940ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1093950ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1112279ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1124360ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1139110ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1157981ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1163071ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1179032ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1181195ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1191955ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1201472ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1218261ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1221169ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1229430ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
[ 199ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 297ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22vendor_cluster_size%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-25m%22%2C%22to%22%3A%22now%22%7D%7D%7D:0
|
||||
[ 1000ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 2023ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 3559ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 7749ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 16364ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 25781ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 42158ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 59672ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 77090ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 87426ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 89952ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 104743ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error in connection establishment: net::ERR_NAME_NOT_RESOLVED @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1573121ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1584038ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1601626ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1613280ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1630173ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1634776ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1654205ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1661576ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1665269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1682669ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1689435ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1704385ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1706314ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1724551ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1743701ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1758339ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1769299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1789478ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1795415ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1815382ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1820392ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1822445ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1839035ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1840162ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1847739ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1861419ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1880381ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1893315ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1911434ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1922701ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1933868ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1944951ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1948001ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1952503ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1953320ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1955369ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
@@ -0,0 +1,12 @@
|
||||
[ 585ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1132ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&panes=%7B%22o68%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22agroal_blocking_time_max_milliseconds%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22blocking+max+-+%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-30m%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1:0
|
||||
[ 1555ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 3604ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 6373ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 9135ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 19066ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 24595ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 47125ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 52154ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 67608ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 86965ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
@@ -0,0 +1,104 @@
|
||||
- generic [ref=f12e4]:
|
||||
- link "Skip to main content" [ref=f12e5] [cursor=pointer]:
|
||||
- /url: "#pageContent"
|
||||
- banner [ref=f12e7]:
|
||||
- generic [ref=f12e8]:
|
||||
- link [ref=f12e10] [cursor=pointer]:
|
||||
- /url: /
|
||||
- img "Grafana" [ref=f12e11]
|
||||
- generic [ref=f12e14]:
|
||||
- button "Search or jump to..." [ref=f12e18] [cursor=pointer]
|
||||
- generic [ref=f12e19]: ctrl+k
|
||||
- generic [ref=f12e23]:
|
||||
- button "New" [ref=f12e24] [cursor=pointer]
|
||||
- button "Help" [ref=f12e30] [cursor=pointer]
|
||||
- button "News" [ref=f12e33] [cursor=pointer]
|
||||
- button "Profile" [ref=f12e36] [cursor=pointer]:
|
||||
- img "User avatar" [ref=f12e37]
|
||||
- generic [ref=f12e38]:
|
||||
- button "Open menu" [ref=f12e40] [cursor=pointer]
|
||||
- navigation "Breadcrumbs" [ref=f12e43]:
|
||||
- list [ref=f12e44]:
|
||||
- listitem [ref=f12e45]:
|
||||
- link "Home" [ref=f12e46] [cursor=pointer]:
|
||||
- /url: /
|
||||
- listitem [ref=f12e50]:
|
||||
- link "Explore" [ref=f12e51] [cursor=pointer]:
|
||||
- /url: /explore
|
||||
- listitem [ref=f12e55]:
|
||||
- generic "Prometheus" [ref=f12e56]
|
||||
- generic [ref=f12e57]:
|
||||
- button "Show more items" [ref=f12e60] [cursor=pointer]
|
||||
- button "Toggle top search bar" [ref=f12e64] [cursor=pointer]
|
||||
- main [ref=f12e70]:
|
||||
- generic [ref=f12e72]:
|
||||
- heading "Explore" [level=1] [ref=f12e73]
|
||||
- generic [ref=f12e78]:
|
||||
- navigation "Explore toolbar" [ref=f12e80]:
|
||||
- navigation "Search links" [ref=f12e82]:
|
||||
- generic [ref=f12e83]:
|
||||
- button "Content outline" [expanded] [ref=f12e85] [cursor=pointer]:
|
||||
- generic [ref=f12e88]: Outline
|
||||
- generic [ref=f12e93] [cursor=pointer]:
|
||||
- img "Prometheus logo" [ref=f12e95]
|
||||
- textbox "Select a data source" [ref=f12e96]:
|
||||
- /placeholder: ""
|
||||
- button "Show more items" [ref=f12e102] [cursor=pointer]
|
||||
- generic [ref=f12e106]:
|
||||
- generic [ref=f12e110]:
|
||||
- button "Collapse outline" [expanded] [ref=f12e112] [cursor=pointer]:
|
||||
- img "arrow-from-right" [ref=f12e113]
|
||||
- button "Queries" [ref=f12e116] [cursor=pointer]:
|
||||
- img "arrow" [ref=f12e117]
|
||||
- generic [ref=f12e124]:
|
||||
- generic [ref=f12e126]:
|
||||
- generic "Query editor row" [ref=f12e129]:
|
||||
- generic [ref=f12e130]:
|
||||
- generic [ref=f12e132]:
|
||||
- generic [ref=f12e133]:
|
||||
- button "Collapse query row" [expanded] [ref=f12e134] [cursor=pointer]
|
||||
- generic [ref=f12e137]:
|
||||
- button "Query editor row title A" [ref=f12e138] [cursor=pointer]:
|
||||
- generic [ref=f12e139]: A
|
||||
- emphasis [ref=f12e140]: (Prometheus)
|
||||
- generic [ref=f12e141]:
|
||||
- button "Show data source help" [ref=f12e143] [cursor=pointer]
|
||||
- button "Duplicate query" [ref=f12e147] [cursor=pointer]
|
||||
- button "Hide response" [ref=f12e151] [cursor=pointer]
|
||||
- button "Remove query" [ref=f12e155] [cursor=pointer]
|
||||
- button "Drag and drop to reorder" [ref=f12e158]:
|
||||
- img "Drag and drop to reorder" [ref=f12e159]
|
||||
- generic [ref=f12e162]:
|
||||
- generic [ref=f12e163]:
|
||||
- button "Kick start your query" [ref=f12e164] [cursor=pointer]
|
||||
- generic [ref=f12e167]:
|
||||
- generic [ref=f12e168] [cursor=pointer]: Explain
|
||||
- generic [ref=f12e169]:
|
||||
- checkbox "Explain Toggle switch" [ref=f12e170]
|
||||
- generic "Toggle switch" [ref=f12e171] [cursor=pointer]
|
||||
- radiogroup [ref=f12e176]:
|
||||
- generic [ref=f12e177]:
|
||||
- radio "Builder" [ref=f12e178] [cursor=pointer]
|
||||
- generic [ref=f12e179] [cursor=pointer]: Builder
|
||||
- generic [ref=f12e180]:
|
||||
- radio "Code" [checked] [ref=f12e181] [cursor=pointer]
|
||||
- generic [ref=f12e182] [cursor=pointer]: Code
|
||||
- generic [ref=f12e184]:
|
||||
- generic [ref=f12e186]:
|
||||
- button "Loading metrics..." [disabled] [ref=f12e187] [cursor=pointer]
|
||||
- generic [ref=f12e190]: Loading editor
|
||||
- 'button "Options Legend: {{pod}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f12e198] [cursor=pointer]':
|
||||
- generic [ref=f12e202]:
|
||||
- heading "Options" [level=6] [ref=f12e203]
|
||||
- generic [ref=f12e204]:
|
||||
- generic [ref=f12e205]: "Legend: {{pod}}"
|
||||
- generic [ref=f12e206]: "Format: Time series"
|
||||
- generic [ref=f12e207]: "Step: auto"
|
||||
- generic [ref=f12e208]: "Type: Range"
|
||||
- generic [ref=f12e209]: "Exemplars: false"
|
||||
- generic [ref=f12e210]:
|
||||
- button "Add query" [ref=f12e211] [cursor=pointer]
|
||||
- button "Query history" [ref=f12e215] [cursor=pointer]
|
||||
- button "Query inspector" [ref=f12e219] [cursor=pointer]
|
||||
- generic:
|
||||
- main
|
||||
@@ -0,0 +1,145 @@
|
||||
- generic [active] [ref=f15e1]:
|
||||
- generic [ref=f15e4]:
|
||||
- link "Skip to main content" [ref=f15e5] [cursor=pointer]:
|
||||
- /url: "#pageContent"
|
||||
- banner [ref=f15e7]:
|
||||
- generic [ref=f15e8]:
|
||||
- link [ref=f15e10] [cursor=pointer]:
|
||||
- /url: /
|
||||
- img "Grafana" [ref=f15e11]
|
||||
- generic [ref=f15e14]:
|
||||
- button "Search or jump to..." [ref=f15e18] [cursor=pointer]
|
||||
- generic [ref=f15e19]: ctrl+k
|
||||
- generic [ref=f15e23]:
|
||||
- button "New" [ref=f15e24] [cursor=pointer]
|
||||
- button "Help" [ref=f15e30] [cursor=pointer]
|
||||
- button "News" [ref=f15e33] [cursor=pointer]
|
||||
- button "Profile" [ref=f15e36] [cursor=pointer]:
|
||||
- img "User avatar" [ref=f15e37]
|
||||
- generic [ref=f15e38]:
|
||||
- button "Open menu" [ref=f15e40] [cursor=pointer]
|
||||
- navigation "Breadcrumbs" [ref=f15e43]:
|
||||
- list [ref=f15e44]:
|
||||
- listitem [ref=f15e45]:
|
||||
- link "Home" [ref=f15e46] [cursor=pointer]:
|
||||
- /url: /
|
||||
- listitem [ref=f15e50]:
|
||||
- link "Explore" [ref=f15e51] [cursor=pointer]:
|
||||
- /url: /explore
|
||||
- listitem [ref=f15e55]:
|
||||
- generic "Prometheus" [ref=f15e56]
|
||||
- generic [ref=f15e57]:
|
||||
- generic [ref=f15e60]:
|
||||
- button "Copy shortened URL" [ref=f15e61] [cursor=pointer]
|
||||
- button "Open copy link options" [ref=f15e64] [cursor=pointer]
|
||||
- button "Toggle top search bar" [ref=f15e68] [cursor=pointer]
|
||||
- main [ref=f15e74]:
|
||||
- generic [ref=f15e76]:
|
||||
- heading "Explore" [level=1] [ref=f15e77]
|
||||
- generic [ref=f15e82]:
|
||||
- navigation "Explore toolbar" [ref=f15e84]:
|
||||
- navigation "Search links" [ref=f15e86]:
|
||||
- generic [ref=f15e87]:
|
||||
- button "Content outline" [expanded] [ref=f15e89] [cursor=pointer]:
|
||||
- generic [ref=f15e92]: Outline
|
||||
- generic [ref=f15e97] [cursor=pointer]:
|
||||
- img "Prometheus logo" [ref=f15e99]
|
||||
- textbox "Select a data source" [ref=f15e100]:
|
||||
- /placeholder: ""
|
||||
- generic [ref=f15e104]:
|
||||
- button "Split the pane" [ref=f15e106] [cursor=pointer]:
|
||||
- generic [ref=f15e109]: Split
|
||||
- button "Add" [ref=f15e111] [cursor=pointer]
|
||||
- generic [ref=f15e116]:
|
||||
- 'button "Time range selected: Last 30 minutes" [ref=f15e117] [cursor=pointer]'
|
||||
- button "Zoom out time range" [ref=f15e122] [cursor=pointer]
|
||||
- generic [ref=f15e126]:
|
||||
- button "Run query" [ref=f15e127] [cursor=pointer]
|
||||
- button "Auto refresh turned off. Choose refresh time interval" [ref=f15e131] [cursor=pointer]
|
||||
- generic [ref=f15e135]:
|
||||
- generic [ref=f15e139]:
|
||||
- button "Collapse outline" [expanded] [ref=f15e141] [cursor=pointer]:
|
||||
- img "arrow-from-right" [ref=f15e142]
|
||||
- button "Queries" [ref=f15e145] [cursor=pointer]:
|
||||
- img "arrow" [ref=f15e146]
|
||||
- button "Graph" [ref=f15e150] [cursor=pointer]:
|
||||
- img "graph-bar" [ref=f15e151]
|
||||
- generic [ref=f15e158]:
|
||||
- generic [ref=f15e160]:
|
||||
- generic "Query editor row" [ref=f15e163]:
|
||||
- generic [ref=f15e164]:
|
||||
- generic [ref=f15e166]:
|
||||
- generic [ref=f15e167]:
|
||||
- button "Collapse query row" [expanded] [ref=f15e168] [cursor=pointer]
|
||||
- generic [ref=f15e171]:
|
||||
- button "Query editor row title A" [ref=f15e172] [cursor=pointer]:
|
||||
- generic [ref=f15e173]: A
|
||||
- emphasis [ref=f15e174]: (Prometheus)
|
||||
- generic [ref=f15e175]:
|
||||
- button "Show data source help" [ref=f15e177] [cursor=pointer]
|
||||
- button "Duplicate query" [ref=f15e181] [cursor=pointer]
|
||||
- button "Hide response" [ref=f15e185] [cursor=pointer]
|
||||
- button "Remove query" [ref=f15e189] [cursor=pointer]
|
||||
- button "Drag and drop to reorder" [ref=f15e192]:
|
||||
- img "Drag and drop to reorder" [ref=f15e193]
|
||||
- generic [ref=f15e196]:
|
||||
- generic [ref=f15e197]:
|
||||
- button "Kick start your query" [ref=f15e198] [cursor=pointer]
|
||||
- generic [ref=f15e201]:
|
||||
- generic [ref=f15e202] [cursor=pointer]: Explain
|
||||
- generic [ref=f15e203]:
|
||||
- checkbox "Explain Toggle switch" [ref=f15e204]
|
||||
- generic "Toggle switch" [ref=f15e205] [cursor=pointer]
|
||||
- radiogroup [ref=f15e210]:
|
||||
- generic [ref=f15e211]:
|
||||
- radio "Builder" [ref=f15e212] [cursor=pointer]
|
||||
- generic [ref=f15e213] [cursor=pointer]: Builder
|
||||
- generic [ref=f15e214]:
|
||||
- radio "Code" [checked] [ref=f15e215] [cursor=pointer]
|
||||
- generic [ref=f15e216] [cursor=pointer]: Code
|
||||
- generic [ref=f15e218]:
|
||||
- generic [ref=f15e220]:
|
||||
- button "Loading metrics..." [disabled] [ref=f15e221] [cursor=pointer]
|
||||
- code [ref=f15e228]:
|
||||
- generic [ref=f15e229]:
|
||||
- generic [ref=f15e234]: agroal_blocking_time_max_milliseconds
|
||||
- textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=f15e239]: agroal_blocking_time_max_milliseconds
|
||||
- 'button "Options Legend: blocking max - {{pod}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f15e245] [cursor=pointer]':
|
||||
- generic [ref=f15e249]:
|
||||
- heading "Options" [level=6] [ref=f15e250]
|
||||
- generic [ref=f15e251]:
|
||||
- generic [ref=f15e252]: "Legend: blocking max - {{pod}}"
|
||||
- generic [ref=f15e253]: "Format: Time series"
|
||||
- generic [ref=f15e254]: "Step: auto"
|
||||
- generic [ref=f15e255]: "Type: Range"
|
||||
- generic [ref=f15e256]: "Exemplars: false"
|
||||
- generic [ref=f15e257]:
|
||||
- button "Add query" [ref=f15e258] [cursor=pointer]
|
||||
- button "Query history" [ref=f15e262] [cursor=pointer]
|
||||
- button "Query inspector" [ref=f15e266] [cursor=pointer]
|
||||
- main [ref=f15e270]:
|
||||
- region [ref=f15e272]:
|
||||
- generic [ref=f15e273]:
|
||||
- heading "Graph" [level=2] [ref=f15e275]
|
||||
- radiogroup [ref=f15e278]:
|
||||
- generic [ref=f15e279]:
|
||||
- radio "Lines" [checked] [ref=f15e280] [cursor=pointer]
|
||||
- generic [ref=f15e281] [cursor=pointer]: Lines
|
||||
- generic [ref=f15e282]:
|
||||
- radio "Bars" [ref=f15e283] [cursor=pointer]
|
||||
- generic [ref=f15e284] [cursor=pointer]: Bars
|
||||
- generic [ref=f15e285]:
|
||||
- radio "Points" [ref=f15e286] [cursor=pointer]
|
||||
- generic [ref=f15e287] [cursor=pointer]: Points
|
||||
- generic [ref=f15e288]:
|
||||
- radio "Stacked lines" [ref=f15e289] [cursor=pointer]
|
||||
- generic [ref=f15e290] [cursor=pointer]: Stacked lines
|
||||
- generic [ref=f15e291]:
|
||||
- radio "Stacked bars" [ref=f15e292] [cursor=pointer]
|
||||
- generic [ref=f15e293] [cursor=pointer]: Stacked bars
|
||||
- generic [ref=f15e294]: Loading plugin panel...
|
||||
- generic [ref=f15e299]:
|
||||
- alert
|
||||
- alert
|
||||
- complementary
|
||||
- complementary
|
||||
@@ -0,0 +1,42 @@
|
||||
=== A-5 기준선 ===
|
||||
keycloak-0=10.42.1.77 (kc-lab-2) keycloak-1=10.42.0.42 (kc-lab-1)
|
||||
keycloak-0 1/1 Running 0 11m
|
||||
keycloak-1 1/1 Running 1 (2m48s ago) 155m
|
||||
postgres-7b474b88c8-9cmsv 1/1 Running 0 14m
|
||||
Traceback (most recent call last):
|
||||
File "<string>", line 3, in <module>
|
||||
for r in json.load(sys.stdin)["data"]["result"]: print(f" cluster_size {r["metric"].get("pod"):12} = {r["value"][1]}")
|
||||
~~~~~~~~~^^^^^^^^^^^
|
||||
File "/usr/lib/python3.14/json/__init__.py", line 298, in load
|
||||
return loads(fp.read(),
|
||||
cls=cls, object_hook=object_hook,
|
||||
parse_float=parse_float, parse_int=parse_int,
|
||||
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
|
||||
File "/usr/lib/python3.14/json/__init__.py", line 352, in loads
|
||||
return _default_decoder.decode(s)
|
||||
~~~~~~~~~~~~~~~~~~~~~~~^^^
|
||||
File "/usr/lib/python3.14/json/decoder.py", line 345, in decode
|
||||
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
|
||||
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/lib/python3.14/json/decoder.py", line 363, in raw_decode
|
||||
raise JSONDecodeError("Expecting value", s, err.value) from None
|
||||
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
|
||||
|
||||
=== 주입: keycloak-0 으로 들어가는 7800/57800 만 DROP (한 방향) ===
|
||||
A-1 의 NetworkPolicy 는 conntrack ESTABLISHED 에 막혀 기존 연결을 못 끊었다.
|
||||
FORWARD 최상단에 넣으면 conntrack 승인보다 먼저 평가된다.
|
||||
Chain FORWARD (policy ACCEPT)
|
||||
num target prot opt source destination
|
||||
1 DROP 6 -- 0.0.0.0/0 10.42.1.77 tcp dpt:57800
|
||||
2 DROP 6 -- 0.0.0.0/0 10.42.1.77 tcp dpt:7800
|
||||
주입 시각: 12:28:23
|
||||
|
||||
=== 관찰 — 어느 쪽이 먼저 상대를 의심하는가 ===
|
||||
+25초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+50초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+75초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+100초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+125초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+150초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+175초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+200초 | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
@@ -0,0 +1,16 @@
|
||||
=== 실패한 규칙 제거 ===
|
||||
제거완료
|
||||
|
||||
=== raw 테이블 PREROUTING 에 넣는다 — conntrack 보다 먼저 실행된다 ===
|
||||
netfilter 순서: raw PREROUTING → conntrack → mangle → nat → filter
|
||||
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
|
||||
num pkts bytes target prot opt in out source destination
|
||||
1 0 0 DROP 6 -- * * 0.0.0.0/0 10.42.1.77 tcp dpt:57800
|
||||
2 0 0 DROP 6 -- * * 0.0.0.0/0 10.42.1.77 tcp dpt:7800
|
||||
주입: 12:32:35
|
||||
|
||||
=== [검증] 이번엔 패킷이 걸렸는가 ===
|
||||
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
|
||||
pkts bytes target prot opt in out source destination
|
||||
0 0 DROP 6 -- * * 0.0.0.0/0 10.42.1.77 tcp dpt:57800
|
||||
0 0 DROP 6 -- * * 0.0.0.0/0 10.42.1.77 tcp dpt:7800
|
||||
@@ -0,0 +1,18 @@
|
||||
=== keycloak-1(수신측)으로 들어가는 7800/57800 만 DROP — kc-lab-1 에 넣는다 ===
|
||||
주입: 12:33:58
|
||||
|
||||
=== [검증] 패킷이 걸리는가 ===
|
||||
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
|
||||
pkts bytes target prot opt in out source destination
|
||||
0 0 DROP 6 -- * * 0.0.0.0/0 10.42.0.42 tcp dpt:57800
|
||||
19 2938 DROP 6 -- * * 0.0.0.0/0 10.42.0.42 tcp dpt:7800
|
||||
|
||||
=== 관찰 ===
|
||||
+25초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+50초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+75초 - | keycloak-0:1/1 keycloak-1:0/1 | 외부 200
|
||||
+100초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+125초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+150초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+175초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+200초 - | keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
@@ -0,0 +1,19 @@
|
||||
=== 지금 연결 방향은? (차단은 → 10.42.0.42:7800 만) ===
|
||||
tcp 6 299 ESTABLISHED src=10.42.0.42 dst=10.42.1.77 sport=48473 dport=7800 src=10.42.1.77 dst=10.42.0.42 sport=7800 dport=48473
|
||||
tcp 6 86232 ESTABLISHED src=10.42.0.42 dst=10.42.1.77 sport=44205 dport=57800 src=10.42.1.77 dst=10.42.0.42 sport=57800 dport=44205
|
||||
|
||||
=== 차단 규칙 누적 카운터 ===
|
||||
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
|
||||
pkts bytes target prot opt in out source destination
|
||||
19 1096 DROP 6 -- * * 0.0.0.0/0 10.42.0.42 tcp dpt:57800
|
||||
21 3058 DROP 6 -- * * 0.0.0.0/0 10.42.0.42 tcp dpt:7800
|
||||
|
||||
=== 클러스터 멤버 (파드에서 직접) ===
|
||||
keycloak-0 cluster_size=2.0
|
||||
keycloak-1 cluster_size=2.0
|
||||
|
||||
=== 최근 뷰 로그 ===
|
||||
keycloak-0: [keycloak-0-24309(v=16.0.12)|12] (1) [keycloak-0-24309(v=16.0.12)]
|
||||
|
||||
keycloak-1: [keycloak-1-45480(v=16.0.12)|12] (1) [keycloak-1-45480(v=16.0.12)]
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
=== 뷰 변화 전체 이력 (차단 12:33:58 전후) ===
|
||||
--- keycloak-0 ---
|
||||
2026-09-04 03:33:49 | during merge: CacheTopology{id=37, phase=NO_REBALANCE, rebalanceId=11, currentCH=DefaultConsistentHash{ns=25
|
||||
2026-09-04 03:33:49,398 INFO [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=offlineSessions] ISPN100007: After me
|
||||
2026-09-04 03:33:49 | during merge: CacheTopology{id=37, phase=NO_REBALANCE, rebalanceId=11, currentCH=DefaultConsistentHash{ns=25
|
||||
2026-09-04 03:33:49,399 INFO [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=loginFailures] ISPN100007: After merg
|
||||
2026-09-04 03:33:49 | during merge: CacheTopology{id=37, phase=NO_REBALANCE, rebalanceId=11, currentCH=DefaultConsistentHash{ns=25
|
||||
2026-09-04 03:33:49,402 INFO [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=actionTokens] ISPN100007: After merge
|
||||
--- keycloak-1 ---
|
||||
2026-09-04 03:32:59,874 INFO [org.infinispan.CLUSTER] (non-blocking-thread--p2-t2) [Context=work] ISPN100007: After merge (or coo
|
||||
2026-09-04 03:32:59,879 INFO [org.infinispan.CLUSTER] (non-blocking-thread--p2-t2) [Context=clientSessions] ISPN100007: After mer
|
||||
2026-09-04 03:32:59,879 INFO [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=offlineSessions] ISPN100007: After me
|
||||
2026-09-04 03:32:59,883 INFO [org.infinispan.CLUSTER] (non-blocking-thread--p2-t1) [Context=loginFailures] ISPN100007: After merg
|
||||
2026-09-04 03:32:59,884 INFO [org.infinispan.CLUSTER] (non-blocking-thread--p2-t2) [Context=actionTokens] ISPN100007: After merge
|
||||
2026-09-04 03:33:49 | MergeView::[keycloak-0-24309(v=16.0.12)|13] (2) [keycloak-0-24309(v=16.0.12), keycloak-1-45480(v=16.0.12)],
|
||||
|
||||
=== MERGE3 병합 이벤트 ===
|
||||
keycloak-0 merge_events=1.0 suspected=0.0
|
||||
keycloak-1 merge_events=1.0 suspected=0.0
|
||||
|
||||
=== 차단 해제 ===
|
||||
해제완료
|
||||
keycloak-0 cluster_size=2.0
|
||||
warning: couldn't attach to pod/a5-f, falling back to streaming logs: unable to upgrade connection: container a5-f not found in pod a5-f_keycloak-lab
|
||||
keycloak-0 cluster_size=2.0
|
||||
keycloak-1 cluster_size=2.0
|
||||
keycloak-0 1/1 Running 0 22m
|
||||
keycloak-1 1/1 Running 1 (13m ago) 166m
|
||||
@@ -0,0 +1,17 @@
|
||||
=== 양방향 차단 — 두 노드 모두에 raw DROP ===
|
||||
주입: 12:40:25
|
||||
+25초 keycloak-0:1/1 keycloak-1:1/1 | ready=[10.42.0.42 10.42.1.77] 외부 200
|
||||
+50초 keycloak-0:1/1 keycloak-1:1/1 | ready=[10.42.0.42 10.42.1.77] 외부 200
|
||||
+75초 keycloak-0:1/1 keycloak-1:1/1 | ready=[10.42.0.42 10.42.1.77] 외부 200
|
||||
+100초 keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
|
||||
+125초 keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
|
||||
+150초 keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
|
||||
+175초 keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
|
||||
+200초 keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
|
||||
+225초 keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
|
||||
|
||||
=== 뷰 상태 ===
|
||||
keycloak-0 [keycloak-0-24309(v=16.0.12)|14] (1) [keycloak-0-24309(v=16.0.12)]
|
||||
|
||||
keycloak-1 [keycloak-1-45480(v=16.0.12)|14] (1) [keycloak-1-45480(v=16.0.12)]
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
=== 왜 한쪽만 DOWN 인가 — 코디네이터 여부 확인 ===
|
||||
name | ip | coord
|
||||
------------------+-----------------+-------
|
||||
keycloak-0-24309 | 10.42.1.77:7800 | t
|
||||
keycloak-1-45480 | 10.42.0.42:7800 | t
|
||||
(2 rows)
|
||||
|
||||
=== 양쪽 헬스체크 상세 ===
|
||||
--- keycloak-0 ---
|
||||
{"status":"UP","checks":[{"name":"GracefulShutdown","status":"UP"}
|
||||
{"name":"KeycloakInitialized","status":"UP"}
|
||||
{"name":"Keycloakclusterhealthcheck","status":"UP"}
|
||||
|
||||
--- keycloak-1 ---
|
||||
{"status":"DOWN","checks":[{"name":"GracefulShutdown","status":"UP"}
|
||||
{"name":"Keycloakdatabaseconnectionsasynchealthcheck","status":"UP"}
|
||||
{"name":"KeycloakInitialized","status":"UP"}
|
||||
|
||||
warning: couldn't attach to pod/a5-h, falling back to streaming logs: Internal error occurred: Internal error occurred: error attaching to container: container is in CONTAINER_EXITED state
|
||||
--- keycloak-0 ---
|
||||
{"status":"UP","checks":[{"name":"GracefulShutdown","status":"UP"}
|
||||
{"name":"KeycloakInitialized","status":"UP"}
|
||||
{"name":"Keycloakclusterhealthcheck","status":"UP"}
|
||||
|
||||
--- keycloak-1 ---
|
||||
{"status":"DOWN","checks":[{"name":"GracefulShutdown","status":"UP"}
|
||||
{"name":"Keycloakdatabaseconnectionsasynchealthcheck","status":"UP"}
|
||||
{"name":"KeycloakInitialized","status":"UP"}
|
||||
|
||||
|
||||
=== 차단 해제 ===
|
||||
해제: 12:44:37
|
||||
+25초 keycloak-0:1/1 keycloak-1:0/1
|
||||
+50초 keycloak-0:1/1 keycloak-1:1/1
|
||||
→ 복구 완료
|
||||
keycloak-0 MergeView::[keycloak-0-24309(v=16.0.12)|15] (2) [keycloak-0-24309(v=16.0.12), keycloak-1-45480(
|
||||
|
||||
keycloak-1 MergeView::[keycloak-0-24309(v=16.0.12)|15] (2) [keycloak-0-24309(v=16.0.12), keycloak-1-45480(
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
# A-5 — 비대칭 파티션 증거
|
||||
|
||||
2026-09-04 12:28–12:46 KST
|
||||
해설: [`docs/experiment-a5-asymmetric-partition.md`](../../experiment-a5-asymmetric-partition.md)
|
||||
|
||||
| 파일 | 무엇을 보여주는가 |
|
||||
|---|---|
|
||||
| `01-injection.txt` | 첫 시도 — `iptables -I FORWARD 1` |
|
||||
| `02-injection-verify.txt` | **실패 확인** — kube-router 가 자기 체인을 위로 재삽입, DROP 규칙 **0 패킷** |
|
||||
| `03-raw-table-injection.txt` | `raw` 테이블로 이동. 그래도 0 패킷 |
|
||||
| `04-correct-direction.txt` | **연결 방향이 반대였다.** 수신측 노드로 옮기니 **19/21 패킷 차단** |
|
||||
| `05-reconnect-observed.txt` | **핵심** — 연결이 **열린 방향으로 뒤집혀 재연결**. 뷰 변화 없음, `suspected=0` |
|
||||
| `06-view-history-and-cleanup.txt` | 뷰 ID 이력. 주입 이후 변화 없음 확인 |
|
||||
| `07-bidirectional-block.txt` | 양방향 차단 → **뷰 14, 멤버 1개씩.** 그런데 `ready=[10.42.1.77]` 로 **한쪽은 남고 외부 200** |
|
||||
| `08-coordinator-and-recovery.txt` | **`coord=t` 가 둘**(split brain). keycloak-0 `UP` / keycloak-1 `DOWN`. 해제 후 50초, `MergeView |15] (2)` |
|
||||
| `a5-cluster-size-bidirectional-block.png` | Grafana — `vendor_cluster_size` 가 갈라졌다 합쳐지는 구간 |
|
||||
|
||||
## 핵심 네 줄
|
||||
|
||||
1. **주입을 세 번 실패했다** — CNI 체인 경쟁, 연결 방향 오판, conntrack. 셋 다 "아무 일 없음"으로 보였다.
|
||||
2. **단방향 차단으로는 못 가른다.** JGroups 가 열린 방향으로 재연결한다 (자가 치유).
|
||||
3. **양방향이면 갈라진다** — `coord=t` 가 둘.
|
||||
4. **그래도 전면 장애가 아니다.** 코디네이터 쪽만 UP 을 유지해 서비스가 계속된다 — A-1 의 열린 질문에 대한 답.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 65 KiB |
@@ -0,0 +1,22 @@
|
||||
=== 배치 — 왜 A/B 가 되는가 ===
|
||||
postgres 10.42.1.76 (kc-lab-2)
|
||||
keycloak-0 10.42.1.77 (kc-lab-2) → DB 와 같은 노드, cni0 로 직행
|
||||
keycloak-1 10.42.0.42 (kc-lab-1) → DB 와 다른 노드, VXLAN 을 건넌다 ← 여기에 지연을 건다
|
||||
|
||||
=== 사용 가능한 커넥션 풀 지표 ===
|
||||
agroal_acquire_count_total
|
||||
agroal_active_count
|
||||
agroal_available_count
|
||||
agroal_awaiting_count
|
||||
agroal_blocking_time_average_milliseconds
|
||||
agroal_blocking_time_max_milliseconds
|
||||
agroal_blocking_time_total_milliseconds
|
||||
agroal_creation_count_total
|
||||
agroal_creation_time_average_milliseconds
|
||||
agroal_creation_time_max_milliseconds
|
||||
agroal_creation_time_total_milliseconds
|
||||
agroal_destroy_count_total
|
||||
|
||||
=== 기준선 지연 — 각 노드에서 로그인 20회 ===
|
||||
keycloak-0 평균 70 ms
|
||||
keycloak-1 평균 66 ms
|
||||
@@ -0,0 +1,25 @@
|
||||
=== 주입: postgres(10.42.1.76) 가 보내는 패킷만 200ms 지연 (kc-lab-2 eth0) ===
|
||||
prio qdisc 로 밴드를 나누고, u32 필터로 출발지 IP 가 postgres 인 것만 3번 밴드로 보낸다
|
||||
Cannot find device "eth0"
|
||||
Cannot find device "eth0"
|
||||
적용완료
|
||||
Cannot find device "eth0"
|
||||
Cannot find device "eth0"
|
||||
주입: 13:14:55
|
||||
|
||||
=== [검증] 지연이 실제로 걸렸는가 — 두 노드 비교 ===
|
||||
keycloak-0 평균 43 ms 최대 64 ms
|
||||
keycloak-1 평균 47 ms 최대 70 ms
|
||||
|
||||
=== 커넥션 풀 상태 ===
|
||||
--- keycloak-0 ---
|
||||
agroal_blocking_time_max_milliseconds 102.0
|
||||
agroal_active_count 0.0
|
||||
agroal_awaiting_count 0.0
|
||||
agroal_blocking_time_average_milliseconds 0.0
|
||||
agroal_available_count 2.0
|
||||
agroal_blocking_time_max_milliseconds 164.0
|
||||
agroal_active_count 0.0
|
||||
agroal_awaiting_count 0.0
|
||||
agroal_blocking_time_average_milliseconds 0.0
|
||||
agroal_available_count 2.0
|
||||
@@ -0,0 +1,20 @@
|
||||
=== 오버레이 인터페이스 확인 ===
|
||||
flannel.1 UNKNOWN a6:b2:62:04:c1:a4 <BROADCAST,MULTICAST,UP,LOWER_UP>
|
||||
cni0 UP 5a:77:1a:e2:b0:a4 <BROADCAST,MULTICAST,UP,LOWER_UP>
|
||||
|
||||
=== flannel.1 에 주입 — 여기서는 파드 IP 가 보인다 (캡슐화 전) ===
|
||||
qdisc prio 1: root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
|
||||
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
|
||||
backlog 0b 0p requeues 0
|
||||
qdisc netem 30: parent 1:3 limit 1000 delay 200ms
|
||||
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
|
||||
backlog 0b 0p requeues 0
|
||||
|
||||
=== [검증] 필터에 패킷이 걸리는가 ===
|
||||
qdisc netem 30: parent 1:3 limit 1000 delay 200ms
|
||||
Sent 18388 bytes 150 pkt (dropped 0, overlimits 0 requeues 0)
|
||||
backlog 0b 0p requeues 0
|
||||
|
||||
=== 두 노드 지연 비교 (기준선: k0=70ms k1=66ms) ===
|
||||
keycloak-0 평균 41 ms 최대 57 ms
|
||||
keycloak-1 평균 1872 ms 최대 1887 ms
|
||||
@@ -0,0 +1,37 @@
|
||||
=== 동시 부하 20건을 keycloak-1 에 — 커넥션 풀이 견디는가 ===
|
||||
1 200 1.911191
|
||||
1 200 1.913766
|
||||
1 200 1.958374
|
||||
1 200 1.981620
|
||||
1 200 10.539402
|
||||
1 200 11.951943
|
||||
1 200 13.351102
|
||||
1 200 14.785832
|
||||
1 200 16.189533
|
||||
1 200 17.625166
|
||||
1 200 19.053724
|
||||
1 200 20.495883
|
||||
1 200 21.905932
|
||||
1 200 22.228466
|
||||
1 200 22.230871
|
||||
1 200 3.441366
|
||||
1 200 4.841075
|
||||
1 200 6.257489
|
||||
1 200 7.704608
|
||||
1 200 9.104792
|
||||
|
||||
=== 부하 직후 커넥션 풀 ===
|
||||
agroal_blocking_time_max_milliseconds 20000.0
|
||||
agroal_max_used_count 19.0
|
||||
agroal_acquire_count_total 672.0
|
||||
agroal_active_count 0.0
|
||||
agroal_awaiting_count 0.0
|
||||
agroal_blocking_time_average_milliseconds 281.0
|
||||
agroal_available_count 19.0
|
||||
|
||||
=== readiness 가 흔들렸는가 ===
|
||||
keycloak-0 1/1 Running 0 60m
|
||||
keycloak-1 1/1 Running 1 (51m ago) 3h24m
|
||||
52m Normal TaintManagerEviction pod/keycloak-1 Cancelling deletion of Pod keycloak-lab/keycloak-1
|
||||
32m Warning Unhealthy pod/keycloak-1 Readiness probe failed: HTTP probe failed with statuscode: 503
|
||||
89s Warning Unhealthy pod/keycloak-1 Readiness probe failed: Get "http://10.42.0.42:9000/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
|
||||
@@ -0,0 +1,12 @@
|
||||
=== 지연 해제 ===
|
||||
해제완료
|
||||
qdisc noqueue 0: root refcnt 2
|
||||
|
||||
=== 회복 확인 ===
|
||||
keycloak-0 평균 43 ms
|
||||
keycloak-1 평균 51 ms
|
||||
keycloak-0 1/1 Running 0 61m
|
||||
keycloak-1 1/1 Running 1 (52m ago) 3h24m
|
||||
|
||||
=== 낙관적 락 충돌이 늘었는가 — 지연 중 로그 ===
|
||||
관련 로그 줄수: 0
|
||||
@@ -0,0 +1,20 @@
|
||||
# A-6 — 지연 주입 증거
|
||||
|
||||
2026-09-04 13:10–13:35 KST
|
||||
해설: [`docs/experiment-a6-latency-injection.md`](../../experiment-a6-latency-injection.md)
|
||||
|
||||
| 파일 | 무엇을 보여주는가 |
|
||||
|---|---|
|
||||
| `01-baseline.txt` | 배치 설명(A/B 가 되는 이유), `agroal_*` 지표 목록, **기준선 70ms / 66ms** |
|
||||
| `02-delay-injected.txt` | 첫 시도 실패 — **`Cannot find device "eth0"`** (Debian 은 `enp1s0`) |
|
||||
| `03-flannel-injection.txt` | **성공** — `flannel.1` 에 걸어야 파드 IP 가 보인다. netem `Sent 150 pkt` 로 검증. **k0=41ms vs k1=1872ms** |
|
||||
| `04-pool-under-load.txt` | 동시 20건 — 응답이 1.9초에서 **22.2초**까지 계단. `blocking_time_max=20000ms`, `max_used_count=19`, **readiness 프로브 타임아웃** |
|
||||
| `05-recovery.txt` | 해제 즉시 43ms / 51ms 회복. **낙관적 락 충돌 0건**(예측 빗나감) |
|
||||
| `a6-connection-pool-blocking.png` | Grafana — `agroal_blocking_time_max_milliseconds` |
|
||||
|
||||
## 핵심 네 줄
|
||||
|
||||
1. **200ms 가 1,872ms 가 된다.** 로그인 트랜잭션의 왕복이 9번이라 지연이 곱해진다.
|
||||
2. **동시 부하에서 22초까지 늘어난다.** 커넥션 풀 큐잉으로 한 번 더 곱해진다.
|
||||
3. **헬스체크도 같은 줄에 선다** — 프로브가 타임아웃되어 노드가 로드밸런서에서 빠지고, 남은 노드로 부하가 몰린다.
|
||||
4. **낙관적 락 충돌은 없었다.** 로그인은 새 행을 만들 뿐 같은 행을 다투지 않는다 — B-3 의 영역.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 69 KiB |
@@ -0,0 +1,319 @@
|
||||
# A-5 — 한쪽 방향만 끊기면 어떻게 되는가 (split brain)
|
||||
|
||||
브랜치 `feature/keycloak-a5-asymmetric-partition` ·
|
||||
증거 [`docs/evidence/a5-asymmetric-partition/`](evidence/a5-asymmetric-partition/) ·
|
||||
2026-09-04 12:28–12:46 KST
|
||||
|
||||
선행: [`A-1`](experiment-a1-jgroups-transport-block.md) — 이 실험은 A-1 이
|
||||
열어둔 질문에 답한다.
|
||||
|
||||
> **A-1 의 열린 질문** — *"양쪽이 동시에 NotReady 가 되는 경로가 있다면
|
||||
> 전면 장애다. A-5 에서 이어서 본다."*
|
||||
|
||||
---
|
||||
|
||||
## 0. 결론부터
|
||||
|
||||
| 물음 | 답 |
|
||||
|---|---|
|
||||
| 비대칭(한 방향) 차단이 클러스터를 가르는가 | **아니다.** 열린 방향으로 재연결한다 |
|
||||
| 양방향 완전 차단은 가르는가 | **그렇다.** 양쪽 모두 멤버 1개 |
|
||||
| **그때 양쪽이 다 NotReady 가 되는가** | **아니다. 한쪽만 DOWN 이다** |
|
||||
| 그래서 서비스는 | **계속된다.** 외부 200 유지 |
|
||||
|
||||
**전면 장애 경로는 없었다.** 코디네이터 쪽이 항상 살아남는다.
|
||||
|
||||
그리고 주입을 **세 번 실패**했다. 세 번 모두 다른 이유였고, 셋 다
|
||||
**"아무 일도 없었다"로 보였다.**
|
||||
|
||||
---
|
||||
|
||||
## 1. 주입을 세 번 실패했다
|
||||
|
||||
A-1 에서 NetworkPolicy 가 conntrack 의 `ESTABLISHED` 승인에 막혀 기존 연결을
|
||||
못 끊는다는 것을 배웠다. 그래서 이번에는 iptables 로 갔는데, **또 다른 벽이
|
||||
세 개** 있었다.
|
||||
|
||||
### 실패 ① — `iptables -I FORWARD 1` 은 최상단에 남지 않는다
|
||||
|
||||
```bash
|
||||
ssh kc-lab-2 'sudo iptables -I FORWARD 1 -p tcp -d 10.42.1.77 --dport 7800 -j DROP'
|
||||
```
|
||||
|
||||
```
|
||||
num pkts bytes target
|
||||
1 232 377K KUBE-ROUTER-FORWARD /* kube-router netpol */ ← 다시 1번이 되었다
|
||||
2 0 0 DROP tcp dpt:57800
|
||||
3 0 0 DROP tcp dpt:7800 ← 0 패킷
|
||||
```
|
||||
|
||||
**kube-router 가 주기적으로 자기 체인을 최상단에 다시 삽입한다.**
|
||||
내가 1번에 넣어도 곧 2번, 3번으로 밀려난다.
|
||||
|
||||
> **직접 넣은 iptables 규칙은 CNI 가 관리하는 체인과 경쟁한다.**
|
||||
> 넣는 것으로 끝이 아니라 **패킷 카운터로 확인해야 한다.**
|
||||
|
||||
### 실패 ② — 그래도 0 패킷: **연결 방향을 잘못 짚었다**
|
||||
|
||||
`raw` 테이블로 옮겼는데도 0 패킷이었다.
|
||||
|
||||
```bash
|
||||
ssh kc-lab-2 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d 10.42.1.77 --dport 7800 -j DROP'
|
||||
```
|
||||
|
||||
실제 연결을 봤더니
|
||||
|
||||
```
|
||||
ESTABLISHED src=10.42.1.77 dst=10.42.0.42 sport=60485 dport=7800
|
||||
──────────── ────────────────────
|
||||
keycloak-0 가 클라이언트 keycloak-1 이 서버
|
||||
```
|
||||
|
||||
**A-1 때와 방향이 반대였다.** 파드가 재시작되면서 누가 먼저 연결을 걸었는지가
|
||||
바뀌었다.
|
||||
|
||||
> **JGroups 의 TCP 연결 방향은 고정이 아니다.** 먼저 뜬 쪽, 먼저 JOIN 을 건
|
||||
> 쪽에 따라 달라진다. **가정하지 말고 `conntrack -L` 로 봐야 한다.**
|
||||
|
||||
### 성공 — `raw` 테이블 PREROUTING, 수신측 노드에
|
||||
|
||||
```bash
|
||||
ssh kc-lab-1 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d 10.42.0.42 --dport 7800 -j DROP
|
||||
sudo iptables -t raw -I PREROUTING 1 -p tcp -d 10.42.0.42 --dport 57800 -j DROP'
|
||||
```
|
||||
|
||||
```
|
||||
pkts bytes target
|
||||
19 1096 DROP tcp dpt:57800
|
||||
21 3058 DROP tcp dpt:7800 ← 걸린다
|
||||
```
|
||||
|
||||
### 개념 — netfilter 처리 순서
|
||||
|
||||
```
|
||||
패킷 도착
|
||||
│
|
||||
├─▶ raw PREROUTING ← conntrack 보다 먼저. NOTRACK·DROP 용
|
||||
│
|
||||
├─▶ conntrack 조회/생성 ← 여기서 ESTABLISHED 가 결정된다
|
||||
│
|
||||
├─▶ mangle PREROUTING
|
||||
├─▶ nat PREROUTING
|
||||
├─▶ filter FORWARD ← NetworkPolicy·kube-router 가 여기 있다
|
||||
└─▶ 목적지 파드
|
||||
```
|
||||
|
||||
| 어디에 넣는가 | 기존 연결을 끊는가 | CNI 와 경쟁하는가 |
|
||||
|---|---|---|
|
||||
| NetworkPolicy (filter) | **못 끊는다** — conntrack 이 먼저 통과시킨다 | 없음 |
|
||||
| filter FORWARD 직접 | 순서에 따라 | **경쟁한다** (kube-router 가 밀어낸다) |
|
||||
| **raw PREROUTING** | **끊는다** | **없다** — CNI 가 안 쓰는 테이블 |
|
||||
|
||||
**진짜 네트워크 분단을 흉내내려면 `raw` 테이블이 맞다.**
|
||||
|
||||
---
|
||||
|
||||
## 2. 비대칭 차단 — 클러스터가 갈라지지 않았다
|
||||
|
||||
한 방향(`→ keycloak-1:7800`)만 막고 관찰했다.
|
||||
|
||||
```
|
||||
+25초 keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
+75초 keycloak-0:1/1 keycloak-1:0/1 | 외부 200 ← 잠깐 흔들림
|
||||
+100초 keycloak-0:1/1 keycloak-1:1/1 | 외부 200 ← 회복
|
||||
...
|
||||
+200초 keycloak-0:1/1 keycloak-1:1/1 | 외부 200
|
||||
```
|
||||
|
||||
**주입 이후 뷰 변화가 하나도 없었다.**
|
||||
|
||||
```bash
|
||||
kubectl -n keycloak-lab logs keycloak-0 --since=20m | grep ISPN000094 | awk '$2 >= "03:33:58"'
|
||||
# (아무것도 안 나온다)
|
||||
```
|
||||
|
||||
```
|
||||
뷰 ID 이력: |13] (2) ← 주입 전부터 지금까지 그대로
|
||||
suspected = 0
|
||||
```
|
||||
|
||||
### 왜 안 갈라졌는가 — **연결 방향이 뒤집혔다**
|
||||
|
||||
```
|
||||
차단 전: keycloak-0 → keycloak-1:7800 ← 내가 막은 방향
|
||||
차단 후: keycloak-1 → keycloak-0:7800 ← 열린 방향으로 다시 붙었다
|
||||
```
|
||||
|
||||
```
|
||||
ESTABLISHED src=10.42.0.42 dst=10.42.1.77 sport=48473 dport=7800
|
||||
──────────── ─────────────
|
||||
keycloak-1 이 클라이언트 keycloak-0 이 서버
|
||||
```
|
||||
|
||||
**JGroups 는 막힌 연결이 죽자 반대 방향으로 새로 연결했고, 그 사이 FD_SOCK2 가
|
||||
상대를 의심하기 전에 복구가 끝났다.** `suspected = 0` 이 그 증거다.
|
||||
|
||||
> **한 방향만 막는 것으로는 JGroups 를 가를 수 없다.**
|
||||
> 두 노드는 서로에게 연결을 걸 수 있으므로, **한쪽 길이 막히면 다른 길로 간다.**
|
||||
>
|
||||
> 운영적으로는 좋은 소식이다 — 단방향 방화벽 오설정은 **자가 치유**된다.
|
||||
> 반대로 **분단을 재현하려는 실험자에게는 함정**이다.
|
||||
|
||||
---
|
||||
|
||||
## 3. 양방향 차단 — 갈라졌지만 전면 장애는 아니다
|
||||
|
||||
양쪽 노드 모두에 규칙을 넣었다.
|
||||
|
||||
```
|
||||
+75초 keycloak-0:1/1 keycloak-1:1/1 | ready=[10.42.0.42 10.42.1.77] 외부 200
|
||||
+100초 keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
|
||||
...
|
||||
+225초 keycloak-0:1/1 keycloak-1:0/1 | ready=[10.42.1.77] 외부 200
|
||||
```
|
||||
|
||||
```
|
||||
=== 뷰 ===
|
||||
keycloak-0 [keycloak-0-24309|14] (1) [keycloak-0-24309]
|
||||
keycloak-1 [keycloak-1-45480|14] (1) [keycloak-1-45480]
|
||||
|
||||
=== JGROUPS_PING ===
|
||||
keycloak-0-24309 | 10.42.1.77:7800 | t ← 코디네이터
|
||||
keycloak-1-45480 | 10.42.0.42:7800 | t ← 코디네이터
|
||||
```
|
||||
|
||||
**`coord = t` 가 둘.** 완전한 split brain 이다.
|
||||
|
||||
|
||||
|
||||
### 그런데 **한쪽만 DOWN 이다**
|
||||
|
||||
```
|
||||
--- keycloak-0 ---
|
||||
{"status":"UP", ... {"name":"Keycloak cluster health check","status":"UP"}
|
||||
|
||||
--- keycloak-1 ---
|
||||
{"status":"DOWN", ... (cluster health check 가 DOWN)
|
||||
```
|
||||
|
||||
| | keycloak-0 | keycloak-1 |
|
||||
|---|---|---|
|
||||
| 분단 전 역할 | **코디네이터** (뷰 13 의 발행자) | 일반 멤버 |
|
||||
| 분단 후 자기 인식 | "멤버가 하나 나갔다" — **정상 사건** | "코디네이터를 잃었다" — **비정상** |
|
||||
| 헬스체크 | **UP** | **DOWN** |
|
||||
| Service 엔드포인트 | **남는다** | 빠진다 |
|
||||
|
||||
> **A-1 의 열린 질문에 대한 답 — 양쪽이 동시에 NotReady 가 되는 경로는 없었다.**
|
||||
>
|
||||
> Keycloak 의 클러스터 헬스체크는 **비대칭**이다. 코디네이터였던 쪽은
|
||||
> 자기가 정상이라고 보고, 잃은 쪽만 DOWN 이 된다. 그래서 완전 분단조차
|
||||
> **용량 저하로 끝나고 전면 장애가 되지 않는다.**
|
||||
>
|
||||
> A-2(DB 상실)에서 양쪽이 동시에 DOWN 이 된 것과 대조된다. **DB 는 모두가
|
||||
> 의존하는 하나지만, 클러스터 멤버십은 서로 상대적**이기 때문이다.
|
||||
|
||||
---
|
||||
|
||||
## 4. 복구
|
||||
|
||||
```bash
|
||||
ssh kc-lab-1 'sudo iptables -t raw -F PREROUTING'
|
||||
ssh kc-lab-2 'sudo iptables -t raw -F PREROUTING'
|
||||
```
|
||||
|
||||
```
|
||||
+25초 keycloak-0:1/1 keycloak-1:0/1
|
||||
+50초 keycloak-0:1/1 keycloak-1:1/1
|
||||
→ 복구 완료
|
||||
|
||||
MergeView::[keycloak-0-24309|15] (2) [keycloak-0-24309, keycloak-1-45480]
|
||||
```
|
||||
|
||||
**50초, 사람 개입 없음.** `MergeView` 로 뷰 15 가 발행되며 병합됐다.
|
||||
|
||||
각 캐시마다 재분배 로그가 남는다.
|
||||
|
||||
```
|
||||
[Context=work] ISPN100007: After merge (or coordinator change) ...
|
||||
[Context=clientSessions] ISPN100007: After merge ...
|
||||
[Context=offlineSessions] ISPN100007: After merge ...
|
||||
[Context=loginFailures] ISPN100007: After merge ...
|
||||
[Context=actionTokens] ISPN100007: After merge ...
|
||||
```
|
||||
|
||||
**`ISPN100007` 은 병합 후 캐시별 토폴로지 재계산**이다. 캐시가 여럿이므로
|
||||
로그도 캐시 수만큼 나온다.
|
||||
|
||||
---
|
||||
|
||||
## 5. 개념
|
||||
|
||||
### `MergeView` 와 뷰 ID
|
||||
|
||||
```
|
||||
[keycloak-0-24309|13] (2) ← 정상
|
||||
[keycloak-0-24309|14] (1) ← 분단. 양쪽이 각자 14 를 발행
|
||||
MergeView::[...|15] (2) ← 병합. 뷰 ID 는 계속 증가한다
|
||||
```
|
||||
|
||||
**뷰 ID 는 단조 증가**하므로 "언제 몇 번 갈라졌는지"를 로그만으로 셀 수 있다.
|
||||
|
||||
### 코디네이터의 비대칭성
|
||||
|
||||
JGroups 코디네이터는 **가장 오래된 멤버**다. 분단이 나면
|
||||
|
||||
```
|
||||
코디네이터 쪽: "멤버가 나갔다" → 정상 처리, 계속 코디네이터
|
||||
나머지 쪽: "코디네이터가 사라졌다" → 새 코디네이터를 자기로 선출
|
||||
```
|
||||
|
||||
**둘 다 자기가 코디네이터라고 믿는 상태**가 split brain 이고,
|
||||
`JGROUPS_PING.coord` 컬럼에 그대로 드러난다.
|
||||
|
||||
### 실험자를 위한 규칙
|
||||
|
||||
| 상황 | 확인 방법 |
|
||||
|---|---|
|
||||
| 규칙을 넣었는데 안 걸린다 | `iptables -L -n -v` 의 **패킷 카운터** |
|
||||
| 방향을 모르겠다 | `conntrack -L \| grep <포트>` |
|
||||
| CNI 가 밀어낸다 | `raw` 테이블을 쓴다 |
|
||||
| 갈라졌는지 알고 싶다 | `vendor_cluster_size`, `ISPN000094`, `JGROUPS_PING.coord` |
|
||||
|
||||
---
|
||||
|
||||
## 6. 재현 절차 (명령어)
|
||||
|
||||
```bash
|
||||
# 1. 지금 연결이 어느 방향인지 먼저 본다 — 가정하면 실패한다
|
||||
ssh kc-lab-1 'sudo conntrack -L | grep 7800'
|
||||
|
||||
# 2. 수신측 노드의 raw PREROUTING 에 넣는다 (filter 는 CNI 와 경쟁한다)
|
||||
ssh kc-lab-1 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d <수신 파드IP> --dport 7800 -j DROP'
|
||||
ssh kc-lab-1 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d <수신 파드IP> --dport 57800 -j DROP'
|
||||
|
||||
# 3. 걸렸는지 카운터로 확인 — 0 이면 해석 금지
|
||||
ssh kc-lab-1 'sudo iptables -t raw -L PREROUTING -n -v'
|
||||
|
||||
# 4. 양방향으로 하려면 반대 노드에도 (한 방향만으로는 자가 치유된다)
|
||||
ssh kc-lab-2 'sudo iptables -t raw -I PREROUTING 1 -p tcp -d <반대 파드IP> --dport 7800 -j DROP'
|
||||
|
||||
# 5. 분단 확인
|
||||
kubectl -n keycloak-lab exec deploy/postgres -- psql -U keycloak -d keycloak \
|
||||
-c "select name, coord from jgroups_ping" # coord=t 가 둘이면 split brain
|
||||
kubectl -n keycloak-lab get endpoints keycloak -o jsonpath='{.subsets[*].addresses[*].ip}'
|
||||
|
||||
# 6. 해제
|
||||
ssh kc-lab-1 'sudo iptables -t raw -F PREROUTING'
|
||||
ssh kc-lab-2 'sudo iptables -t raw -F PREROUTING'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. 다음 실험에 남기는 것
|
||||
|
||||
| 실험 | 이 실험이 준 것 |
|
||||
|---|---|
|
||||
| **A-6** 지연 주입 | `tc netem` 도 같은 함정 — **주입이 걸렸는지 먼저 확인** |
|
||||
| **A-7** volatile 비교 | 여기서는 분단에도 서비스가 계속됐다. volatile 이면 **세션이 갈라진다** |
|
||||
| 운영 | **단방향 방화벽 오설정은 자가 치유된다.** 양방향이어야 사고가 된다 |
|
||||
| 운영 | **완전 분단조차 전면 장애가 아니다** — 코디네이터 쪽이 살아남는다 |
|
||||
@@ -0,0 +1,338 @@
|
||||
# A-6 — 끊기지 않고 느려지기만 하면 어떻게 되는가
|
||||
|
||||
브랜치 `feature/keycloak-a6-latency-injection` ·
|
||||
증거 [`docs/evidence/a6-latency-injection/`](evidence/a6-latency-injection/) ·
|
||||
2026-09-04 13:10–13:35 KST
|
||||
|
||||
**실제 장애의 대부분은 완전 사망이 아니라 느려짐이다.** 그리고 느려짐은
|
||||
사망보다 진단하기 어렵다 — 헬스체크가 통과하기 때문이다.
|
||||
|
||||
---
|
||||
|
||||
## 0. 결론부터
|
||||
|
||||
| 측정 | 값 |
|
||||
|---|---|
|
||||
| 주입한 네트워크 지연 | **200 ms** |
|
||||
| 로그인 응답 시간 | **66 ms → 1,872 ms (28배)** |
|
||||
| 동시 20건에서 최대 응답 | **22.2 초** |
|
||||
| 커넥션 획득 대기 최대 | **20,000 ms** |
|
||||
| readiness 프로브 | **타임아웃으로 실패** |
|
||||
|
||||
**200밀리초가 22초가 됐다.** 지연은 **왕복 횟수만큼 곱해지고**, 커넥션 풀에서
|
||||
**한 번 더 곱해진다.**
|
||||
|
||||
---
|
||||
|
||||
## 1. 설계 — 한 실험에서 A/B 가 되는 배치
|
||||
|
||||
```
|
||||
kc-lab-2 kc-lab-1
|
||||
┌──────────────────┐ ┌──────────────────┐
|
||||
│ postgres │ │ keycloak-1 │
|
||||
│ keycloak-0 │ │ │
|
||||
│ └─ cni0 로 직행 │◀─ VXLAN ──▶│ └─ 오버레이 경유 │
|
||||
└──────────────────┘ └──────────────────┘
|
||||
지연 없음 여기만 느려진다
|
||||
```
|
||||
|
||||
**postgres 가 보내는 패킷만** 지연시키면 `keycloak-1` 의 DB 접근만 느려지고
|
||||
`keycloak-0` 은 그대로다. **대조군이 같은 실험 안에 있다.**
|
||||
|
||||
기준선은 거의 같았다.
|
||||
|
||||
```
|
||||
keycloak-0 평균 70 ms
|
||||
keycloak-1 평균 66 ms
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. 주입을 두 번 실패했다
|
||||
|
||||
### 실패 ① — `eth0` 이라는 인터페이스가 없다
|
||||
|
||||
```bash
|
||||
ssh kc-lab-2 'sudo tc qdisc add dev eth0 root handle 1: prio'
|
||||
```
|
||||
```
|
||||
Cannot find device "eth0"
|
||||
```
|
||||
|
||||
Debian 클라우드 이미지는 **예측 가능한 인터페이스 이름**을 쓴다.
|
||||
|
||||
```
|
||||
enp1s0 UP 52:54:00:aa:bb:12
|
||||
```
|
||||
|
||||
`en` (ethernet) + `p1` (PCI bus 1) + `s0` (slot 0). 이름이 하드웨어 위치에서
|
||||
나오므로 **NIC 순서가 바뀌어도 이름이 안 바뀐다.**
|
||||
|
||||
### 실패 ② — `enp1s0` 에서는 **파드 IP 가 보이지 않는다**
|
||||
|
||||
여기가 핵심이다. 노드 간 파드 통신은 **flannel VXLAN 으로 캡슐화**된다.
|
||||
|
||||
```
|
||||
원래 패킷: src=10.42.1.76(postgres) dst=10.42.0.42(keycloak-1)
|
||||
│
|
||||
▼ flannel.1 에서 캡슐화
|
||||
실제 패킷: src=192.168.122.12(노드) dst=192.168.122.11(노드) UDP 8472
|
||||
└─ 안쪽에 원래 패킷이 통째로 들어 있다
|
||||
│
|
||||
▼
|
||||
enp1s0 로 나간다
|
||||
```
|
||||
|
||||
**`enp1s0` 에서 `match ip src 10.42.1.76` 은 절대 일치하지 않는다.**
|
||||
그 IP 는 페이로드 안에 있고, 헤더에는 노드 IP 만 있다.
|
||||
|
||||
### 성공 — `flannel.1` 에 건다
|
||||
|
||||
```bash
|
||||
ssh kc-lab-2 '
|
||||
sudo tc qdisc add dev flannel.1 root handle 1: prio
|
||||
sudo tc qdisc add dev flannel.1 parent 1:3 handle 30: netem delay 200ms
|
||||
sudo tc filter add dev flannel.1 protocol ip parent 1:0 prio 3 \
|
||||
u32 match ip src 10.42.1.76/32 flowid 1:3'
|
||||
```
|
||||
|
||||
**`flannel.1` 은 캡슐화 직전 단계**이므로 여기서는 파드 IP 가 보인다.
|
||||
|
||||
### 검증 — 카운터로 확인한다
|
||||
|
||||
```
|
||||
qdisc netem 30: parent 1:3 limit 1000 delay 200ms
|
||||
Sent 18388 bytes 150 pkt (dropped 0, overlimits 0 requeues 0)
|
||||
───────
|
||||
실제로 지연 밴드를 통과했다
|
||||
```
|
||||
|
||||
> **A-1·A-5 와 같은 교훈이 세 번째로 나왔다.**
|
||||
> 주입을 넣은 것과 걸린 것은 다르다. **카운터를 봐야 한다.**
|
||||
|
||||
### 개념 — `tc` 의 계층 구조
|
||||
|
||||
```
|
||||
qdisc (큐 규율) 인터페이스에 붙는 패킷 스케줄러
|
||||
├─ prio 우선순위 밴드 3개로 나눈다
|
||||
│ ├─ 1:1 (기본)
|
||||
│ ├─ 1:2 (기본)
|
||||
│ └─ 1:3 ← 여기에 netem 을 붙인다
|
||||
└─ filter 어떤 패킷을 어느 밴드로 보낼지
|
||||
```
|
||||
|
||||
`netem` 을 root 에 바로 붙이면 **모든 트래픽**이 느려진다.
|
||||
`prio` + `filter` 를 쓰면 **고른 트래픽만** 느려진다.
|
||||
|
||||
---
|
||||
|
||||
## 3. 결과 — 지연은 곱해진다
|
||||
|
||||
```
|
||||
=== 두 노드 지연 비교 (기준선: k0=70ms k1=66ms) ===
|
||||
keycloak-0 평균 41 ms 최대 57 ms ← 영향 없음
|
||||
keycloak-1 평균 1872 ms 최대 1887 ms ← 28배
|
||||
```
|
||||
|
||||
### 왜 200ms 가 1,872ms 가 되는가
|
||||
|
||||
A-0 에서 잡은 로그인 트랜잭션의 SQL 이 답이다.
|
||||
|
||||
```
|
||||
BEGIN
|
||||
select ... from OFFLINE_USER_SESSION ...
|
||||
select VERSION ... for no key update skip locked
|
||||
select ... from OFFLINE_CLIENT_SESSION ...
|
||||
select VERSION ... for no key update skip locked
|
||||
insert into OFFLINE_USER_SESSION ...
|
||||
insert into OFFLINE_CLIENT_SESSION ...
|
||||
SET LOCAL synchronous_commit TO OFF
|
||||
COMMIT
|
||||
```
|
||||
|
||||
**왕복이 아홉 번이다.**
|
||||
|
||||
```
|
||||
200 ms × 9 왕복 ≈ 1,800 ms 실측 1,872 ms
|
||||
```
|
||||
|
||||
> **네트워크 지연은 왕복 횟수만큼 증폭된다.**
|
||||
> "DB 가 200ms 느려졌다"는 "애플리케이션이 200ms 느려졌다"가 아니다.
|
||||
> **쿼리 수를 줄이는 것이 지연 환경에서 결정적인 이유**가 이것이다.
|
||||
|
||||
---
|
||||
|
||||
## 4. 동시 부하 — 여기서 진짜 고장이 난다
|
||||
|
||||
동시 20건을 `keycloak-1` 에 보냈다.
|
||||
|
||||
```
|
||||
200 1.911 200 1.913 200 1.958 200 1.981
|
||||
200 3.441 200 4.841 200 6.257 200 7.704
|
||||
200 9.104 200 10.539 200 11.951 200 13.351
|
||||
200 14.785 200 16.189 200 17.625 200 19.053
|
||||
200 20.495 200 21.905 200 22.228 200 22.230
|
||||
```
|
||||
|
||||
**전부 성공(200)했지만 응답 시간이 1.9초에서 22.2초까지 계단으로 늘어난다.**
|
||||
|
||||
```
|
||||
1.9 → 3.4 → 4.8 → 6.2 → 7.7 → 9.1 → 10.5 → ...
|
||||
──── ──── ──── ────
|
||||
약 1.4초 간격 — 앞 요청이 커넥션을 놓아줄 때까지 줄을 선다
|
||||
```
|
||||
|
||||
**전형적인 큐잉이다.** 커넥션 수는 유한하고, 각 요청이 커넥션을 1.9초씩
|
||||
붙잡고 있으므로 뒤에 온 요청은 그만큼 기다린다.
|
||||
|
||||
### 커넥션 풀 지표가 증언한다
|
||||
|
||||
```
|
||||
agroal_blocking_time_max_milliseconds 20000.0 ← 20초를 기다린 요청이 있다
|
||||
agroal_blocking_time_average_milliseconds 281.0
|
||||
agroal_max_used_count 19.0 ← 풀이 19개까지 늘었다
|
||||
agroal_acquire_count_total 672.0
|
||||
agroal_active_count 0.0 ← 부하가 끝나 지금은 0
|
||||
```
|
||||
|
||||
|
||||
|
||||
### 그리고 헬스체크가 무너진다
|
||||
|
||||
```
|
||||
Warning Unhealthy pod/keycloak-1 Readiness probe failed:
|
||||
Get "http://10.42.0.42:9000/health/ready": context deadline exceeded
|
||||
```
|
||||
|
||||
**readiness 프로브 자체가 타임아웃됐다.**
|
||||
|
||||
### 연쇄 고장의 모양
|
||||
|
||||
```
|
||||
DB 가 느려진다
|
||||
↓
|
||||
요청이 커넥션을 오래 붙잡는다
|
||||
↓
|
||||
커넥션 풀이 고갈된다
|
||||
↓
|
||||
새 요청이 줄을 선다 (최대 20초)
|
||||
↓
|
||||
헬스체크도 줄에 선다 → 타임아웃 → NotReady
|
||||
↓
|
||||
그 노드가 로드밸런서에서 빠진다
|
||||
↓
|
||||
★ 남은 노드로 트래픽이 몰린다 → 그 노드도 같은 길을 간다
|
||||
```
|
||||
|
||||
**마지막 화살표가 무서운 부분이다.** 느려짐은 **전파된다.**
|
||||
A-2(DB 완전 정지)는 즉시 503 으로 드러나 오히려 명확했지만,
|
||||
**느려짐은 살아 있는 노드를 하나씩 무너뜨린다.**
|
||||
|
||||
---
|
||||
|
||||
## 5. 빗나간 예측 — 낙관적 락 충돌은 늘지 않았다
|
||||
|
||||
계획서에 이렇게 적었다.
|
||||
|
||||
> **낙관적 락 충돌 증가** — 트랜잭션이 길어져 `VERSION` 충돌이 늘어야 한다
|
||||
|
||||
```
|
||||
관련 로그 줄수: 0
|
||||
```
|
||||
|
||||
**하나도 없었다.** 이유가 명확하다.
|
||||
|
||||
```
|
||||
로그인 → 매번 새 세션 행을 INSERT → 다툴 상대가 없다
|
||||
refresh → 같은 세션 행을 UPDATE → 여기서 다툰다
|
||||
```
|
||||
|
||||
**충돌은 같은 행을 동시에 고칠 때만 일어난다.** 로그인 부하로는 재현되지
|
||||
않는다. 이건 **B-3(refresh 토큰 경쟁)의 영역**이며, 거기서 지연을 함께 주면
|
||||
충돌률이 올라갈 것이다.
|
||||
|
||||
> 예측을 적어두지 않았다면 "충돌이 없네" 하고 넘어갔을 것이다.
|
||||
> **빗나간 예측이 다음 실험의 설계를 정해준다.**
|
||||
|
||||
---
|
||||
|
||||
## 6. 복구
|
||||
|
||||
```bash
|
||||
ssh kc-lab-2 'sudo tc qdisc del dev flannel.1 root'
|
||||
```
|
||||
|
||||
```
|
||||
keycloak-0 평균 43 ms
|
||||
keycloak-1 평균 51 ms ← 즉시 정상
|
||||
```
|
||||
|
||||
**파드 재시작 없이 즉시 회복.** 커넥션 풀도 스스로 정상화됐다.
|
||||
|
||||
---
|
||||
|
||||
## 7. 운영에 주는 것
|
||||
|
||||
| 알게 된 것 | 함의 |
|
||||
|---|---|
|
||||
| 지연은 **왕복 횟수만큼 곱해진다** | DB 지연 대책은 "쿼리 수 줄이기"가 먼저다 |
|
||||
| 커넥션 풀에서 **한 번 더 곱해진다** | 풀 크기와 타임아웃이 장애 반경을 정한다 |
|
||||
| **헬스체크도 줄에 선다** | 프로브 타임아웃이 풀 대기보다 짧아야 격리가 제때 된다 |
|
||||
| 느려짐은 **전파된다** | 노드를 빼면 남은 노드가 더 빨리 무너진다 |
|
||||
| `up` 도 readiness 도 **늦게 반응** | **응답 시간 분포(p95/p99)를 봐야 한다** |
|
||||
|
||||
### 이 실험대에 없는 알림
|
||||
|
||||
지금 관측 스택에는 **지연 분포 지표가 없다.** `agroal_blocking_time_*` 은
|
||||
있지만 히스토그램이 아니라 평균/최대뿐이다.
|
||||
|
||||
```promql
|
||||
# 있으면 좋았을 것
|
||||
histogram_quantile(0.99, rate(http_server_requests_seconds_bucket[5m]))
|
||||
```
|
||||
|
||||
**A-2 에서 `kube-state-metrics` 가 빠진 것을 찾았고, 여기서는 응답 시간
|
||||
히스토그램이 빠진 것을 찾았다.** 둘 다 보완 항목이다.
|
||||
|
||||
---
|
||||
|
||||
## 8. 재현 절차 (명령어)
|
||||
|
||||
```bash
|
||||
# 1. 인터페이스 이름 확인 — eth0 이 아니다
|
||||
ssh kc-lab-2 'ip -brief link show | grep -E "flannel|cni|enp"'
|
||||
|
||||
# 2. 오버레이 인터페이스에 건다 (enp1s0 에서는 파드 IP 가 안 보인다)
|
||||
ssh kc-lab-2 '
|
||||
sudo tc qdisc add dev flannel.1 root handle 1: prio
|
||||
sudo tc qdisc add dev flannel.1 parent 1:3 handle 30: netem delay 200ms
|
||||
sudo tc filter add dev flannel.1 protocol ip parent 1:0 prio 3 \
|
||||
u32 match ip src <postgres 파드IP>/32 flowid 1:3'
|
||||
|
||||
# 3. 걸렸는지 카운터로 확인 — Sent 가 0 이면 해석 금지
|
||||
ssh kc-lab-2 'sudo tc -s qdisc show dev flannel.1 | grep -A2 netem'
|
||||
|
||||
# 4. 단일 요청 지연 (대조군은 같은 노드의 keycloak-0)
|
||||
kubectl -n keycloak-lab run t --rm -i --restart=Never --image=curlimages/curl:8.11.1 \
|
||||
--command -- curl -s -o /dev/null -w '%{time_total}\n' -X POST http://<pod>:8080/realms/master/protocol/openid-connect/token ...
|
||||
|
||||
# 5. 동시 부하로 풀 고갈 재현
|
||||
# ( curl ... ) & 를 20개 띄우고 wait
|
||||
|
||||
# 6. 풀 지표
|
||||
curl -s http://<pod>:9000/metrics | grep -E '^agroal_(blocking_time|max_used|awaiting)'
|
||||
|
||||
# 7. 해제
|
||||
ssh kc-lab-2 'sudo tc qdisc del dev flannel.1 root'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. 다음 실험에 남기는 것
|
||||
|
||||
| 실험 | 이 실험이 준 것 |
|
||||
|---|---|
|
||||
| **B-3** refresh 경쟁 | **지연을 함께 주면 낙관적 락 충돌이 재현될 것** — 여기서는 안 됐다 |
|
||||
| **B-1** 저장소 지연 (Q3 제약) | 같은 기법을 Redis 앞에 쓴다 |
|
||||
| 관측 보완 | **응답 시간 히스토그램**이 없다 |
|
||||
| 구성 | **프로브 타임아웃 < 커넥션 풀 대기**여야 격리가 제때 된다 |
|
||||
Reference in New Issue
Block a user