Compare commits
12
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 donghyeon-ka
|
48bdf0b5ff | ||
|
donghyeon-ka
|
e03ec4eaed | ||
|
donghyeon-ka
|
87edb6634d | ||
|
donghyeon-ka
|
76c852e947 | ||
|
donghyeon-ka
|
84966750f6 | ||
|
donghyeon-ka
|
e682777fc5 | ||
|
donghyeon-ka
|
f8544d5bec | ||
|
donghyeon-ka
|
2d58dea5e1 | ||
|
donghyeon-ka
|
452aa4808a | ||
|
donghyeon-ka
|
357b7f927b | ||
|
donghyeon-ka
|
5ce47689a9 | ||
|
donghyeon-ka
|
4ac0133586 |
@@ -10,6 +10,8 @@ POSTGRES_PASSWORD=change-me-postgres-password
|
||||
TOKEN_MEDIATING_CLIENT_SECRET=change-me-token-mediating-client-secret
|
||||
BFF_CLIENT_SECRET=change-me-bff-client-secret
|
||||
EDGE_PROXY_CLIENT_SECRET=change-me-edge-proxy-client-secret
|
||||
OAUTH2_PROXY_COOKIE_SECRET=generate-a-base64-encoded-32-byte-secret
|
||||
INTERNAL_AUTH_TOKEN=generate-a-long-random-edge-to-backend-token
|
||||
MOCK_GOOGLE_BROKER_CLIENT_SECRET=change-me-mock-google-broker-client-secret
|
||||
ADMIN_USER_PASSWORD=change-me-admin-user-password
|
||||
REGULAR_USER_PASSWORD=change-me-regular-user-password
|
||||
|
||||
+2
-7
@@ -4,15 +4,10 @@
|
||||
*.iml
|
||||
|
||||
backend/target/
|
||||
**/node_modules/
|
||||
frontend/dist/
|
||||
build/
|
||||
e2e/node_modules/
|
||||
google-e2e/node_modules/
|
||||
frontend/node_modules/
|
||||
frontend/dist/
|
||||
|
||||
bff/target
|
||||
token-mediator/target
|
||||
|
||||
# lab cloud-init contains a console password; keep the filled copy local
|
||||
deploy/lab/cloud-init/kc-lab.yaml
|
||||
deploy/lab/cloud-init/kc-lab-*.yaml
|
||||
|
||||
@@ -1,22 +0,0 @@
|
||||
[ 1289ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/login:0
|
||||
[ 1417ms] [VERBOSE] [DOM] Input elements should have autocomplete attributes (suggested: "username"): (More info: https://goo.gl/9p2vKq) %o @ https://app2.hyeonworks.com/login:0
|
||||
[ 9024ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 9130ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/:0
|
||||
[ 9456ms] [WARNING] Deprecation warning: value provided is not in a recognized RFC2822 or ISO format. moment construction falls back to js Date(), which is not reliable across all browsers and versions. Non RFC2822/ISO date formats are discouraged. Please refer to http://momentjs.com/guides/#/warnings/js-date/ for more info.
|
||||
Arguments:
|
||||
[0] _isAMomentObject: true, _isUTC: false, _useUTC: false, _l: undefined, _i: Thu, 27 Aug 2026 13:03:49, _f: undefined, _strict: undefined, _locale: [object Object]
|
||||
Error
|
||||
at a.createFromInputFallback (https://app2.hyeonworks.com/public/build/6029.0549a3fcb50e73c4b256.js:624:3)
|
||||
at an (https://app2.hyeonworks.com/public/build/6029.0549a3fcb50e73c4b256.js:624:25647)
|
||||
at un (https://app2.hyeonworks.com/public/build/6029.0549a3fcb50e73c4b256.js:624:29355)
|
||||
at aa (https://app2.hyeonworks.com/public/build/6029.0549a3fcb50e73c4b256.js:624:29221)
|
||||
at on (https://app2.hyeonworks.com/public/build/6029.0549a3fcb50e73c4b256.js:624:28938)
|
||||
at sa (https://app2.hyeonworks.com/public/build/6029.0549a3fcb50e73c4b256.js:624:29715)
|
||||
at A (https://app2.hyeonworks.com/public/build/6029.0549a3fcb50e73c4b256.js:624:29748)
|
||||
at a (https://app2.hyeonworks.com/public/build/6029.0549a3fcb50e73c4b256.js:621:89)
|
||||
at f (https://app2.hyeonworks.com/public/build/3719.c065b2e146c4c8347d51.js:1:4635)
|
||||
at u (https://app2.hyeonworks.com/public/build/322.177b4bb01c5d74f9b28f.js:2473:47448) @ https://app2.hyeonworks.com/public/build/6029.0549a3fcb50e73c4b256.js:620
|
||||
[ 9605ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 10620ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 11527ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 13875ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
@@ -1,7 +0,0 @@
|
||||
[ 144ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 153ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore:0
|
||||
[ 1077ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 2101ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 2922ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 7323ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 9370ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
@@ -1,8 +0,0 @@
|
||||
[ 271ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22vendor_statistics_approximate_entries_unique%7Bcache%3D%5C%22sessions%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bpod%7D%7D%20on%20%7B%7Bnode%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-15m%22%2C%22to%22%3A%22now%22%7D%7D%7D:0
|
||||
[ 346ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1512ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 2433ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 6941ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 13188ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 21578ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 25998ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
@@ -1,7 +0,0 @@
|
||||
[ 766ms] [WARNING] An iframe which has both allow-scripts and allow-same-origin for its sandbox attribute can escape its sandboxing. @ https://auth.hyeonworks.com/realms/master/protocol/openid-connect/3p-cookies/step1.html:0
|
||||
[ 781ms] [WARNING] An iframe which has both allow-scripts and allow-same-origin for its sandbox attribute can escape its sandboxing. @ https://auth.hyeonworks.com/realms/master/protocol/openid-connect/3p-cookies/step2.html:0
|
||||
[ 17929ms] [WARNING] An iframe which has both allow-scripts and allow-same-origin for its sandbox attribute can escape its sandboxing. @ https://auth.hyeonworks.com/realms/master/protocol/openid-connect/3p-cookies/step1.html:0
|
||||
[ 17949ms] [WARNING] An iframe which has both allow-scripts and allow-same-origin for its sandbox attribute can escape its sandboxing. @ https://auth.hyeonworks.com/realms/master/protocol/openid-connect/3p-cookies/step2.html:0
|
||||
[ 17981ms] [WARNING] An iframe which has both allow-scripts and allow-same-origin for its sandbox attribute can escape its sandboxing. @ https://auth.hyeonworks.com/realms/master/protocol/openid-connect/login-status-iframe.html:0
|
||||
[ 18447ms] [WARNING] For accessibility reasons an aria-label should be specified on nav groups if a title isn't @ https://auth.hyeonworks.com/resources/9v5yc/admin/keycloak.v2/assets/main-BbID33M6.js:7
|
||||
[ 18462ms] [WARNING] For accessibility reasons an aria-label should be specified on nav groups if a title isn't @ https://auth.hyeonworks.com/resources/9v5yc/admin/keycloak.v2/assets/main-BbID33M6.js:7
|
||||
@@ -1,2 +0,0 @@
|
||||
[ 75ms] [ERROR] Failed to load resource: the server responded with a status of 404 (Not Found) @ https://hyeonworks.com/questions:0
|
||||
[ 100ms] [ERROR] Failed to load resource: the server responded with a status of 404 (Not Found) @ https://hyeonworks.com/favicon.ico:0
|
||||
@@ -1 +0,0 @@
|
||||
[ 149ms] [ERROR] Failed to load resource: the server responded with a status of 401 (Unauthorized) @ https://hyeonworks.com/api/v1/studio/session:0
|
||||
@@ -1 +0,0 @@
|
||||
[ 103ms] [ERROR] Failed to load resource: the server responded with a status of 401 (Unauthorized) @ https://hyeonworks.com/api/v1/studio/session:0
|
||||
@@ -1 +0,0 @@
|
||||
[ 95ms] [ERROR] Failed to load resource: the server responded with a status of 401 (Unauthorized) @ https://hyeonworks.com/api/v1/studio/session:0
|
||||
@@ -1 +0,0 @@
|
||||
[ 132ms] [ERROR] Failed to load resource: the server responded with a status of 401 (Unauthorized) @ https://hyeonworks.com/api/v1/studio/session:0
|
||||
@@ -1 +0,0 @@
|
||||
[ 239ms] [ERROR] Failed to load resource: the server responded with a status of 401 (Unauthorized) @ https://hyeonworks.com/api/v1/studio/session:0
|
||||
@@ -1 +0,0 @@
|
||||
[ 105ms] [ERROR] Failed to load resource: the server responded with a status of 401 (Unauthorized) @ https://hyeonworks.com/api/v1/studio/session:0
|
||||
@@ -1 +0,0 @@
|
||||
[ 108ms] [ERROR] Failed to load resource: the server responded with a status of 401 (Unauthorized) @ https://hyeonworks.com/api/v1/studio/session:0
|
||||
@@ -1,41 +0,0 @@
|
||||
[ 344ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 869ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&panes=%7B%22cf1%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22vendor_cluster_size%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bpod%7D%7D+on+%7B%7Bnode%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-45m%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1:0
|
||||
[ 1014ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 2039ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 4085ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 7289ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 15556ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 26818ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 32659ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 35930ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 53849ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 61722ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 79450ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 84571ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 87233ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 92665ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 113244ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 119181ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 137390ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 157071ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 166091ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 182011ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 188613ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 206228ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 218561ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 229607ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 235818ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 237158ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 252095ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 261118ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 273809ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 280227ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 292650ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 306477ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 309957ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 311984ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 325683ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 344586ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 357576ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 359294ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 364740ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
@@ -1,129 +0,0 @@
|
||||
[ 615ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&panes=%7B%22te0%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22up%7Bjob%3D%5C%22keycloak%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22up+%E2%80%94+%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-20m%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1:0
|
||||
[ 929ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 2566ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 5541ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 7990ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 12402ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 17640ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 20285ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 28277ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 34341ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 34985ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 50081ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 65649ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 75046ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 75890ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 96067ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 113467ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 121972ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 139681ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 150971ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 161211ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 180744ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 191401ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 210668ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 229082ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 239345ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 252119ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 266045ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 276705ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 289911ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 296875ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 315099ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 325431ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 328418ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 343774ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 347054ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 352405ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 361291ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 370639ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 374286ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 377413ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 394362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 405423ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 425476ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 434299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 441775ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 450786ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 456829ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 468812ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 478913ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 491241ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 495228ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 502295ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 511625ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 524619ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 533050ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 553189ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 571699ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 575413ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 588005ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 599161ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 608816ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 619031ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 626612ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 639003ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 655077ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 674532ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 677197ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 696027ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 709512ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 727505ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 747139ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 754565ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 773682ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 784674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 787399ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 802845ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 825589ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 832768ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 842043ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 852717ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 871748ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 876571ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 894491ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 895515ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 904310ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 916681ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 934752ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 941175ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 946004ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 960849ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 962180ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 984446ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 997579ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1010105ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1027190ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1043388ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1061311ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1066040ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1080349ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1092336ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1104312ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1109362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1127047ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1134728ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1154899ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1161253ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1174971ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1184491ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1185340ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1189111ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1196984ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1204794ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1247961ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1286322ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1319329ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1362771ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1396881ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1434184ms] [ERROR] Failed to load resource: the server responded with a status of 502 () @ https://app2.hyeonworks.com/api/user/auth-tokens/rotate:0
|
||||
[ 1443506ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1465016ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1513337ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1556906ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1562963ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1572896ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1576390ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 503 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1593794ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1600651ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1616424ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
@@ -1,125 +0,0 @@
|
||||
[ 268ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/login:0
|
||||
[ 379ms] [VERBOSE] [DOM] Input elements should have autocomplete attributes (suggested: "username"): (More info: https://goo.gl/9p2vKq) %o @ https://app2.hyeonworks.com/login:0
|
||||
[ 10453ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 10532ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&panes=%7B%22ich%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22up%7Bjob%3D%7E%5C%22keycloak%7Cnode-exporter%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bjob%7D%7D+%E2%80%94+%7B%7Bpod%7D%7D%7B%7Bnode%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-55m%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1:0
|
||||
[ 11687ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 13737ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 15113ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 20185ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 27259ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 40154ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 41870ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 50493ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 54082ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 60121ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 74362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 83269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 95349ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 98008ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 104458ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 123816ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 142764ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 157198ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 175220ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 182187ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 183411ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 190581ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 199499ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 209995ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 214849ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 233084ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 242910ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 257975ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 277220ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 290806ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 292674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 300013ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 309267ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 323998ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 328620ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 339787ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 355959ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 361079ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 369272ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 381866ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 397120ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 400712ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 412579ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 431630ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 440746ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 447809ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 461741ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 475459ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 482723ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 494299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 513174ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 521541ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 532145ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 551129ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 556250ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 574381ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 586190ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 595782ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 610555ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 630695ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 633257ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 652194ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 671361ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 677606ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 692444ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 696949ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 700122ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 709954ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 723992ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 743173ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 745075ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 753299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 766683ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 767855ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 787261ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 789725ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 800680ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 811324ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 814096ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 829045ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 848506ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 857716ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 866212ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 873174ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 876033ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 895198ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 898884ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 918642ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 930133ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 934008ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 948541ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 956860ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 959397ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 968406ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 971788ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 982746ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 997394ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1005078ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1009269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1021453ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1026674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1042137ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1057497ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1059851ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1061855ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1071317ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1078179ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1084940ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1093950ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1112279ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1124360ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1139110ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1157981ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1163071ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1179032ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1181195ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1191955ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1201472ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1218261ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1221169ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1229430ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
@@ -1,24 +0,0 @@
|
||||
[ 199ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 297ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22vendor_cluster_size%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-25m%22%2C%22to%22%3A%22now%22%7D%7D%7D:0
|
||||
[ 1000ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 2023ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 3559ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 7749ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 16364ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 25781ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 42158ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 59672ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 77090ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 87426ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 89952ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 104743ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error in connection establishment: net::ERR_NAME_NOT_RESOLVED @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1573121ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1584038ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1601626ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1613280ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1630173ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1634776ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1654205ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1661576ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1665269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
[ 1682669ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
|
||||
@@ -1,2 +0,0 @@
|
||||
- main [ref=e7]:
|
||||
- status "Loading" [ref=e10]
|
||||
@@ -1 +0,0 @@
|
||||
- main [ref=f3e7]
|
||||
@@ -1,2 +0,0 @@
|
||||
- main [ref=f6e7]:
|
||||
- status "Loading" [ref=f6e10]
|
||||
@@ -1,104 +0,0 @@
|
||||
- generic [ref=f9e4]:
|
||||
- link "Skip to main content" [ref=f9e5] [cursor=pointer]:
|
||||
- /url: "#pageContent"
|
||||
- banner [ref=f9e7]:
|
||||
- generic [ref=f9e8]:
|
||||
- link [ref=f9e10] [cursor=pointer]:
|
||||
- /url: /
|
||||
- img "Grafana" [ref=f9e11]
|
||||
- generic [ref=f9e14]:
|
||||
- button "Search or jump to..." [ref=f9e18] [cursor=pointer]
|
||||
- generic [ref=f9e19]: ctrl+k
|
||||
- generic [ref=f9e23]:
|
||||
- button "New" [ref=f9e24] [cursor=pointer]
|
||||
- button "Help" [ref=f9e30] [cursor=pointer]
|
||||
- button "News" [ref=f9e33] [cursor=pointer]
|
||||
- button "Profile" [ref=f9e36] [cursor=pointer]:
|
||||
- img "User avatar" [ref=f9e37]
|
||||
- generic [ref=f9e38]:
|
||||
- button "Open menu" [ref=f9e40] [cursor=pointer]
|
||||
- navigation "Breadcrumbs" [ref=f9e43]:
|
||||
- list [ref=f9e44]:
|
||||
- listitem [ref=f9e45]:
|
||||
- link "Home" [ref=f9e46] [cursor=pointer]:
|
||||
- /url: /
|
||||
- listitem [ref=f9e50]:
|
||||
- link "Explore" [ref=f9e51] [cursor=pointer]:
|
||||
- /url: /explore
|
||||
- listitem [ref=f9e55]:
|
||||
- generic "Prometheus" [ref=f9e56]
|
||||
- generic [ref=f9e57]:
|
||||
- button "Show more items" [ref=f9e60] [cursor=pointer]
|
||||
- button "Toggle top search bar" [ref=f9e64] [cursor=pointer]
|
||||
- main [ref=f9e70]:
|
||||
- generic [ref=f9e72]:
|
||||
- heading "Explore" [level=1] [ref=f9e73]
|
||||
- generic [ref=f9e78]:
|
||||
- navigation "Explore toolbar" [ref=f9e80]:
|
||||
- navigation "Search links" [ref=f9e82]:
|
||||
- generic [ref=f9e83]:
|
||||
- button "Content outline" [expanded] [ref=f9e85] [cursor=pointer]:
|
||||
- generic [ref=f9e88]: Outline
|
||||
- generic [ref=f9e93] [cursor=pointer]:
|
||||
- img "Prometheus logo" [ref=f9e95]
|
||||
- textbox "Select a data source" [ref=f9e96]:
|
||||
- /placeholder: ""
|
||||
- button "Show more items" [ref=f9e102] [cursor=pointer]
|
||||
- generic [ref=f9e106]:
|
||||
- generic [ref=f9e110]:
|
||||
- button "Collapse outline" [expanded] [ref=f9e112] [cursor=pointer]:
|
||||
- img "arrow-from-right" [ref=f9e113]
|
||||
- button "Queries" [ref=f9e116] [cursor=pointer]:
|
||||
- img "arrow" [ref=f9e117]
|
||||
- generic [ref=f9e124]:
|
||||
- generic [ref=f9e126]:
|
||||
- generic "Query editor row" [ref=f9e129]:
|
||||
- generic [ref=f9e130]:
|
||||
- generic [ref=f9e132]:
|
||||
- generic [ref=f9e133]:
|
||||
- button "Collapse query row" [expanded] [ref=f9e134] [cursor=pointer]
|
||||
- generic [ref=f9e137]:
|
||||
- button "Query editor row title A" [ref=f9e138] [cursor=pointer]:
|
||||
- generic [ref=f9e139]: A
|
||||
- emphasis [ref=f9e140]: (Prometheus)
|
||||
- generic [ref=f9e141]:
|
||||
- button "Show data source help" [ref=f9e143] [cursor=pointer]
|
||||
- button "Duplicate query" [ref=f9e147] [cursor=pointer]
|
||||
- button "Hide response" [ref=f9e151] [cursor=pointer]
|
||||
- button "Remove query" [ref=f9e155] [cursor=pointer]
|
||||
- button "Drag and drop to reorder" [ref=f9e158]:
|
||||
- img "Drag and drop to reorder" [ref=f9e159]
|
||||
- generic [ref=f9e162]:
|
||||
- generic [ref=f9e163]:
|
||||
- button "Kick start your query" [ref=f9e164] [cursor=pointer]
|
||||
- generic [ref=f9e167]:
|
||||
- generic [ref=f9e168] [cursor=pointer]: Explain
|
||||
- generic [ref=f9e169]:
|
||||
- checkbox "Explain Toggle switch" [ref=f9e170]
|
||||
- generic "Toggle switch" [ref=f9e171] [cursor=pointer]
|
||||
- radiogroup [ref=f9e176]:
|
||||
- generic [ref=f9e177]:
|
||||
- radio "Builder" [ref=f9e178] [cursor=pointer]
|
||||
- generic [ref=f9e179] [cursor=pointer]: Builder
|
||||
- generic [ref=f9e180]:
|
||||
- radio "Code" [checked] [ref=f9e181] [cursor=pointer]
|
||||
- generic [ref=f9e182] [cursor=pointer]: Code
|
||||
- generic [ref=f9e184]:
|
||||
- generic [ref=f9e186]:
|
||||
- button "Loading metrics..." [disabled] [ref=f9e187] [cursor=pointer]
|
||||
- generic [ref=f9e190]: Loading editor
|
||||
- 'button "Options Legend: {{pod}} on {{node}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f9e198] [cursor=pointer]':
|
||||
- generic [ref=f9e202]:
|
||||
- heading "Options" [level=6] [ref=f9e203]
|
||||
- generic [ref=f9e204]:
|
||||
- generic [ref=f9e205]: "Legend: {{pod}} on {{node}}"
|
||||
- generic [ref=f9e206]: "Format: Time series"
|
||||
- generic [ref=f9e207]: "Step: auto"
|
||||
- generic [ref=f9e208]: "Type: Range"
|
||||
- generic [ref=f9e209]: "Exemplars: false"
|
||||
- generic [ref=f9e210]:
|
||||
- button "Add query" [ref=f9e211] [cursor=pointer]
|
||||
- button "Query history" [ref=f9e215] [cursor=pointer]
|
||||
- button "Query inspector" [ref=f9e219] [cursor=pointer]
|
||||
- generic:
|
||||
- main
|
||||
@@ -1,4 +0,0 @@
|
||||
- main [ref=f12e3]:
|
||||
- generic [ref=f12e4]:
|
||||
- progressbar "Contents" [ref=f12e5]
|
||||
- paragraph [ref=f12e8]: Loading the Administration Console
|
||||
@@ -1,4 +0,0 @@
|
||||
- generic [active] [ref=f15e1]:
|
||||
- progressbar "Loading" [ref=f15e4]
|
||||
- generic:
|
||||
- list
|
||||
@@ -1 +0,0 @@
|
||||
- generic [active] [ref=f18e1]: Not Found
|
||||
@@ -1,151 +0,0 @@
|
||||
- generic [active] [ref=e1]:
|
||||
- generic [ref=e4]:
|
||||
- link "Skip to main content" [ref=e5] [cursor=pointer]:
|
||||
- /url: "#pageContent"
|
||||
- banner [ref=e7]:
|
||||
- generic [ref=e8]:
|
||||
- link [ref=e10] [cursor=pointer]:
|
||||
- /url: /
|
||||
- img "Grafana" [ref=e11]
|
||||
- generic [ref=e14]:
|
||||
- button "Search or jump to..." [ref=e18] [cursor=pointer]
|
||||
- generic [ref=e19]: ctrl+k
|
||||
- generic [ref=e23]:
|
||||
- button "New" [ref=e24] [cursor=pointer]
|
||||
- button "Help" [ref=e30] [cursor=pointer]
|
||||
- button "News" [ref=e33] [cursor=pointer]
|
||||
- button "Profile" [ref=e36] [cursor=pointer]:
|
||||
- img "User avatar" [ref=e37]
|
||||
- generic [ref=e38]:
|
||||
- button "Open menu" [ref=e40] [cursor=pointer]
|
||||
- navigation "Breadcrumbs" [ref=e43]:
|
||||
- list [ref=e44]:
|
||||
- listitem [ref=e45]:
|
||||
- link "Home" [ref=e46] [cursor=pointer]:
|
||||
- /url: /
|
||||
- listitem [ref=e50]:
|
||||
- link "Explore" [ref=e51] [cursor=pointer]:
|
||||
- /url: /explore
|
||||
- listitem [ref=e55]:
|
||||
- generic "Prometheus" [ref=e56]
|
||||
- generic [ref=e57]:
|
||||
- generic [ref=e60]:
|
||||
- button "Copy shortened URL" [ref=e61] [cursor=pointer]
|
||||
- button "Open copy link options" [ref=e64] [cursor=pointer]
|
||||
- button "Toggle top search bar" [ref=e68] [cursor=pointer]
|
||||
- main [ref=e74]:
|
||||
- generic [ref=e76]:
|
||||
- heading "Explore" [level=1] [ref=e77]
|
||||
- generic [ref=e82]:
|
||||
- navigation "Explore toolbar" [ref=e84]:
|
||||
- navigation "Search links" [ref=e86]:
|
||||
- generic [ref=e87]:
|
||||
- button "Content outline" [expanded] [ref=e89] [cursor=pointer]:
|
||||
- generic [ref=e92]: Outline
|
||||
- generic [ref=e97] [cursor=pointer]:
|
||||
- img "Prometheus logo" [ref=e99]
|
||||
- textbox "Select a data source" [ref=e100]:
|
||||
- /placeholder: ""
|
||||
- generic [ref=e104]:
|
||||
- button "Split the pane" [ref=e106] [cursor=pointer]:
|
||||
- generic [ref=e109]: Split
|
||||
- button "Add" [ref=e111] [cursor=pointer]
|
||||
- generic [ref=e116]:
|
||||
- 'button "Time range selected: Last 45 minutes" [ref=e117] [cursor=pointer]'
|
||||
- button "Zoom out time range" [ref=e122] [cursor=pointer]
|
||||
- generic [ref=e126]:
|
||||
- button "Run query" [ref=e127] [cursor=pointer]
|
||||
- button "Auto refresh turned off. Choose refresh time interval" [ref=e131] [cursor=pointer]
|
||||
- generic [ref=e135]:
|
||||
- generic [ref=e139]:
|
||||
- button "Collapse outline" [expanded] [ref=e141] [cursor=pointer]:
|
||||
- img "arrow-from-right" [ref=e142]
|
||||
- button "Queries" [ref=e145] [cursor=pointer]:
|
||||
- img "arrow" [ref=e146]
|
||||
- button "Graph" [ref=e150] [cursor=pointer]:
|
||||
- img "graph-bar" [ref=e151]
|
||||
- generic [ref=e158]:
|
||||
- generic [ref=e160]:
|
||||
- generic "Query editor row" [ref=e163]:
|
||||
- generic [ref=e164]:
|
||||
- generic [ref=e166]:
|
||||
- generic [ref=e167]:
|
||||
- button "Collapse query row" [expanded] [ref=e168] [cursor=pointer]
|
||||
- generic [ref=e171]:
|
||||
- button "Query editor row title A" [ref=e172] [cursor=pointer]:
|
||||
- generic [ref=e173]: A
|
||||
- emphasis [ref=e174]: (Prometheus)
|
||||
- generic [ref=e175]:
|
||||
- button "Show data source help" [ref=e177] [cursor=pointer]
|
||||
- button "Duplicate query" [ref=e181] [cursor=pointer]
|
||||
- button "Hide response" [ref=e185] [cursor=pointer]
|
||||
- button "Remove query" [ref=e189] [cursor=pointer]
|
||||
- button "Drag and drop to reorder" [ref=e192]:
|
||||
- img "Drag and drop to reorder" [ref=e193]
|
||||
- generic [ref=e196]:
|
||||
- generic [ref=e197]:
|
||||
- button "Kick start your query" [ref=e198] [cursor=pointer]
|
||||
- generic [ref=e201]:
|
||||
- generic [ref=e202] [cursor=pointer]: Explain
|
||||
- generic [ref=e203]:
|
||||
- checkbox "Explain Toggle switch" [ref=e204]
|
||||
- generic "Toggle switch" [ref=e205] [cursor=pointer]
|
||||
- radiogroup [ref=e210]:
|
||||
- generic [ref=e211]:
|
||||
- radio "Builder" [ref=e212] [cursor=pointer]
|
||||
- generic [ref=e213] [cursor=pointer]: Builder
|
||||
- generic [ref=e214]:
|
||||
- radio "Code" [checked] [ref=e215] [cursor=pointer]
|
||||
- generic [ref=e216] [cursor=pointer]: Code
|
||||
- generic [ref=e218]:
|
||||
- generic [ref=e220]:
|
||||
- button "Metrics browser" [ref=e221] [cursor=pointer]
|
||||
- code [ref=e228]:
|
||||
- generic [ref=e229]:
|
||||
- generic [ref=e234]: vendor_cluster_size
|
||||
- textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=e239]: vendor_cluster_size
|
||||
- 'button "Options Legend: {{pod}} on {{node}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=e245] [cursor=pointer]':
|
||||
- generic [ref=e249]:
|
||||
- heading "Options" [level=6] [ref=e250]
|
||||
- generic [ref=e251]:
|
||||
- generic [ref=e252]: "Legend: {{pod}} on {{node}}"
|
||||
- generic [ref=e253]: "Format: Time series"
|
||||
- generic [ref=e254]: "Step: auto"
|
||||
- generic [ref=e255]: "Type: Range"
|
||||
- generic [ref=e256]: "Exemplars: false"
|
||||
- generic [ref=e257]:
|
||||
- button "Add query" [ref=e258] [cursor=pointer]
|
||||
- button "Query history" [ref=e262] [cursor=pointer]
|
||||
- button "Query inspector" [ref=e266] [cursor=pointer]
|
||||
- main [ref=e270]:
|
||||
- region [ref=e272]:
|
||||
- generic [ref=e273]:
|
||||
- heading "Graph" [level=2] [ref=e275]
|
||||
- radiogroup [ref=e278]:
|
||||
- generic [ref=e279]:
|
||||
- radio "Lines" [checked] [ref=e280] [cursor=pointer]
|
||||
- generic [ref=e281] [cursor=pointer]: Lines
|
||||
- generic [ref=e282]:
|
||||
- radio "Bars" [ref=e283] [cursor=pointer]
|
||||
- generic [ref=e284] [cursor=pointer]: Bars
|
||||
- generic [ref=e285]:
|
||||
- radio "Points" [ref=e286] [cursor=pointer]
|
||||
- generic [ref=e287] [cursor=pointer]: Points
|
||||
- generic [ref=e288]:
|
||||
- radio "Stacked lines" [ref=e289] [cursor=pointer]
|
||||
- generic [ref=e290] [cursor=pointer]: Stacked lines
|
||||
- generic [ref=e291]:
|
||||
- radio "Stacked bars" [ref=e292] [cursor=pointer]
|
||||
- generic [ref=e293] [cursor=pointer]: Stacked bars
|
||||
- list [ref=e302]:
|
||||
- listitem [ref=e303]:
|
||||
- button "keycloak-0 on kc-lab-2" [ref=e307] [cursor=pointer]
|
||||
- listitem [ref=e308]:
|
||||
- button "keycloak-0 on kc-lab-2" [ref=e312] [cursor=pointer]
|
||||
- listitem [ref=e313]:
|
||||
- button "keycloak-1 on kc-lab-1" [ref=e317] [cursor=pointer]
|
||||
- generic [ref=e322]:
|
||||
- alert
|
||||
- alert
|
||||
- complementary
|
||||
- complementary
|
||||
@@ -1,145 +0,0 @@
|
||||
- generic [active] [ref=f3e1]:
|
||||
- generic [ref=f3e4]:
|
||||
- link "Skip to main content" [ref=f3e5] [cursor=pointer]:
|
||||
- /url: "#pageContent"
|
||||
- banner [ref=f3e7]:
|
||||
- generic [ref=f3e8]:
|
||||
- link [ref=f3e10] [cursor=pointer]:
|
||||
- /url: /
|
||||
- img "Grafana" [ref=f3e11]
|
||||
- generic [ref=f3e14]:
|
||||
- button "Search or jump to..." [ref=f3e18] [cursor=pointer]
|
||||
- generic [ref=f3e19]: ctrl+k
|
||||
- generic [ref=f3e23]:
|
||||
- button "New" [ref=f3e24] [cursor=pointer]
|
||||
- button "Help" [ref=f3e30] [cursor=pointer]
|
||||
- button "News" [ref=f3e33] [cursor=pointer]
|
||||
- button "Profile" [ref=f3e36] [cursor=pointer]:
|
||||
- img "User avatar" [ref=f3e37]
|
||||
- generic [ref=f3e38]:
|
||||
- button "Open menu" [ref=f3e40] [cursor=pointer]
|
||||
- navigation "Breadcrumbs" [ref=f3e43]:
|
||||
- list [ref=f3e44]:
|
||||
- listitem [ref=f3e45]:
|
||||
- link "Home" [ref=f3e46] [cursor=pointer]:
|
||||
- /url: /
|
||||
- listitem [ref=f3e50]:
|
||||
- link "Explore" [ref=f3e51] [cursor=pointer]:
|
||||
- /url: /explore
|
||||
- listitem [ref=f3e55]:
|
||||
- generic "Prometheus" [ref=f3e56]
|
||||
- generic [ref=f3e57]:
|
||||
- generic [ref=f3e60]:
|
||||
- button "Copy shortened URL" [ref=f3e61] [cursor=pointer]
|
||||
- button "Open copy link options" [ref=f3e64] [cursor=pointer]
|
||||
- button "Toggle top search bar" [ref=f3e68] [cursor=pointer]
|
||||
- main [ref=f3e74]:
|
||||
- generic [ref=f3e76]:
|
||||
- heading "Explore" [level=1] [ref=f3e77]
|
||||
- generic [ref=f3e82]:
|
||||
- navigation "Explore toolbar" [ref=f3e84]:
|
||||
- navigation "Search links" [ref=f3e86]:
|
||||
- generic [ref=f3e87]:
|
||||
- button "Content outline" [expanded] [ref=f3e89] [cursor=pointer]:
|
||||
- generic [ref=f3e92]: Outline
|
||||
- generic [ref=f3e97] [cursor=pointer]:
|
||||
- img "Prometheus logo" [ref=f3e99]
|
||||
- textbox "Select a data source" [ref=f3e100]:
|
||||
- /placeholder: ""
|
||||
- generic [ref=f3e104]:
|
||||
- button "Split the pane" [ref=f3e106] [cursor=pointer]:
|
||||
- generic [ref=f3e109]: Split
|
||||
- button "Add" [ref=f3e111] [cursor=pointer]
|
||||
- generic [ref=f3e116]:
|
||||
- 'button "Time range selected: Last 20 minutes" [ref=f3e117] [cursor=pointer]'
|
||||
- button "Zoom out time range" [ref=f3e122] [cursor=pointer]
|
||||
- generic [ref=f3e126]:
|
||||
- button "Run query" [ref=f3e127] [cursor=pointer]
|
||||
- button "Auto refresh turned off. Choose refresh time interval" [ref=f3e131] [cursor=pointer]
|
||||
- generic [ref=f3e135]:
|
||||
- generic [ref=f3e139]:
|
||||
- button "Collapse outline" [expanded] [ref=f3e141] [cursor=pointer]:
|
||||
- img "arrow-from-right" [ref=f3e142]
|
||||
- button "Queries" [ref=f3e145] [cursor=pointer]:
|
||||
- img "arrow" [ref=f3e146]
|
||||
- button "Graph" [ref=f3e150] [cursor=pointer]:
|
||||
- img "graph-bar" [ref=f3e151]
|
||||
- generic [ref=f3e158]:
|
||||
- generic [ref=f3e160]:
|
||||
- generic "Query editor row" [ref=f3e163]:
|
||||
- generic [ref=f3e164]:
|
||||
- generic [ref=f3e166]:
|
||||
- generic [ref=f3e167]:
|
||||
- button "Collapse query row" [expanded] [ref=f3e168] [cursor=pointer]
|
||||
- generic [ref=f3e171]:
|
||||
- button "Query editor row title A" [ref=f3e172] [cursor=pointer]:
|
||||
- generic [ref=f3e173]: A
|
||||
- emphasis [ref=f3e174]: (Prometheus)
|
||||
- generic [ref=f3e175]:
|
||||
- button "Show data source help" [ref=f3e177] [cursor=pointer]
|
||||
- button "Duplicate query" [ref=f3e181] [cursor=pointer]
|
||||
- button "Hide response" [ref=f3e185] [cursor=pointer]
|
||||
- button "Remove query" [ref=f3e189] [cursor=pointer]
|
||||
- button "Drag and drop to reorder" [ref=f3e192]:
|
||||
- img "Drag and drop to reorder" [ref=f3e193]
|
||||
- generic [ref=f3e196]:
|
||||
- generic [ref=f3e197]:
|
||||
- button "Kick start your query" [ref=f3e198] [cursor=pointer]
|
||||
- generic [ref=f3e201]:
|
||||
- generic [ref=f3e202] [cursor=pointer]: Explain
|
||||
- generic [ref=f3e203]:
|
||||
- checkbox "Explain Toggle switch" [ref=f3e204]
|
||||
- generic "Toggle switch" [ref=f3e205] [cursor=pointer]
|
||||
- radiogroup [ref=f3e210]:
|
||||
- generic [ref=f3e211]:
|
||||
- radio "Builder" [ref=f3e212] [cursor=pointer]
|
||||
- generic [ref=f3e213] [cursor=pointer]: Builder
|
||||
- generic [ref=f3e214]:
|
||||
- radio "Code" [checked] [ref=f3e215] [cursor=pointer]
|
||||
- generic [ref=f3e216] [cursor=pointer]: Code
|
||||
- generic [ref=f3e218]:
|
||||
- generic [ref=f3e220]:
|
||||
- button "Loading metrics..." [disabled] [ref=f3e221] [cursor=pointer]
|
||||
- code [ref=f3e228]:
|
||||
- generic [ref=f3e229]:
|
||||
- generic [ref=f3e234]: "up{job=\"keycloak\"}"
|
||||
- textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=f3e239]: "up{job=\"keycloak\"}"
|
||||
- 'button "Options Legend: up — {{pod}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f3e245] [cursor=pointer]':
|
||||
- generic [ref=f3e249]:
|
||||
- heading "Options" [level=6] [ref=f3e250]
|
||||
- generic [ref=f3e251]:
|
||||
- generic [ref=f3e252]: "Legend: up — {{pod}}"
|
||||
- generic [ref=f3e253]: "Format: Time series"
|
||||
- generic [ref=f3e254]: "Step: auto"
|
||||
- generic [ref=f3e255]: "Type: Range"
|
||||
- generic [ref=f3e256]: "Exemplars: false"
|
||||
- generic [ref=f3e257]:
|
||||
- button "Add query" [ref=f3e258] [cursor=pointer]
|
||||
- button "Query history" [ref=f3e262] [cursor=pointer]
|
||||
- button "Query inspector" [ref=f3e266] [cursor=pointer]
|
||||
- main [ref=f3e270]:
|
||||
- region [ref=f3e272]:
|
||||
- generic [ref=f3e273]:
|
||||
- heading "Graph" [level=2] [ref=f3e275]
|
||||
- radiogroup [ref=f3e278]:
|
||||
- generic [ref=f3e279]:
|
||||
- radio "Lines" [checked] [ref=f3e280] [cursor=pointer]
|
||||
- generic [ref=f3e281] [cursor=pointer]: Lines
|
||||
- generic [ref=f3e282]:
|
||||
- radio "Bars" [ref=f3e283] [cursor=pointer]
|
||||
- generic [ref=f3e284] [cursor=pointer]: Bars
|
||||
- generic [ref=f3e285]:
|
||||
- radio "Points" [ref=f3e286] [cursor=pointer]
|
||||
- generic [ref=f3e287] [cursor=pointer]: Points
|
||||
- generic [ref=f3e288]:
|
||||
- radio "Stacked lines" [ref=f3e289] [cursor=pointer]
|
||||
- generic [ref=f3e290] [cursor=pointer]: Stacked lines
|
||||
- generic [ref=f3e291]:
|
||||
- radio "Stacked bars" [ref=f3e292] [cursor=pointer]
|
||||
- generic [ref=f3e293] [cursor=pointer]: Stacked bars
|
||||
- generic [ref=f3e294]: Loading plugin panel...
|
||||
- generic [ref=f3e299]:
|
||||
- alert
|
||||
- alert
|
||||
- complementary
|
||||
- complementary
|
||||
@@ -1,44 +0,0 @@
|
||||
- main [ref=f6e7]:
|
||||
- generic [ref=f6e9]:
|
||||
- generic [ref=f6e11]:
|
||||
- generic [ref=f6e12]:
|
||||
- img "Grafana" [ref=f6e13]
|
||||
- heading "Welcome to Grafana" [level=1] [ref=f6e15]
|
||||
- generic [ref=f6e19]:
|
||||
- generic [ref=f6e20]:
|
||||
- generic [ref=f6e21]: Email or username
|
||||
- textbox "Email or username" [active] [ref=f6e28]:
|
||||
- /placeholder: email or username
|
||||
- generic [ref=f6e29]:
|
||||
- generic [ref=f6e30]: Password
|
||||
- generic [ref=f6e36]:
|
||||
- textbox "Password" [ref=f6e37]:
|
||||
- /placeholder: password
|
||||
- switch "Show password" [ref=f6e39] [cursor=pointer]
|
||||
- button "Log in" [ref=f6e42] [cursor=pointer]
|
||||
- link "Forgot your password?" [ref=f6e45] [cursor=pointer]:
|
||||
- /url: /user/password/send-reset-email
|
||||
- list [ref=f6e49]:
|
||||
- listitem [ref=f6e50]:
|
||||
- link "Documentation" [ref=f6e53] [cursor=pointer]:
|
||||
- /url: https://grafana.com/docs/grafana/latest/?utm_source=grafana_footer
|
||||
- text: "|"
|
||||
- listitem [ref=f6e54]:
|
||||
- link "Support" [ref=f6e57] [cursor=pointer]:
|
||||
- /url: https://grafana.com/products/enterprise/?utm_source=grafana_footer
|
||||
- text: "|"
|
||||
- listitem [ref=f6e58]:
|
||||
- link "Community" [ref=f6e61] [cursor=pointer]:
|
||||
- /url: https://community.grafana.com/?utm_source=grafana_footer
|
||||
- text: "|"
|
||||
- listitem [ref=f6e62]:
|
||||
- link "Open Source" [ref=f6e63] [cursor=pointer]:
|
||||
- /url: https://grafana.com/oss/grafana?utm_source=grafana_footer
|
||||
- text: "|"
|
||||
- listitem [ref=f6e64]:
|
||||
- link "Grafana v11.4.0 (b58701869e)" [ref=f6e65] [cursor=pointer]:
|
||||
- /url: https://github.com/grafana/grafana/blob/main/CHANGELOG.md
|
||||
- text: "|"
|
||||
- listitem [ref=f6e66]:
|
||||
- link "New version available!" [ref=f6e69] [cursor=pointer]:
|
||||
- /url: https://grafana.com/grafana/download?utm_source=grafana_footer
|
||||
@@ -1,159 +0,0 @@
|
||||
- generic [active] [ref=f9e1]:
|
||||
- generic [ref=f9e4]:
|
||||
- link "Skip to main content" [ref=f9e5] [cursor=pointer]:
|
||||
- /url: "#pageContent"
|
||||
- banner [ref=f9e7]:
|
||||
- generic [ref=f9e8]:
|
||||
- link [ref=f9e10] [cursor=pointer]:
|
||||
- /url: /
|
||||
- img "Grafana" [ref=f9e11]
|
||||
- generic [ref=f9e14]:
|
||||
- button "Search or jump to..." [ref=f9e18] [cursor=pointer]
|
||||
- generic [ref=f9e19]: ctrl+k
|
||||
- generic [ref=f9e23]:
|
||||
- button "New" [ref=f9e24] [cursor=pointer]
|
||||
- button "Help" [ref=f9e30] [cursor=pointer]
|
||||
- button "News" [ref=f9e33] [cursor=pointer]
|
||||
- button "Profile" [ref=f9e36] [cursor=pointer]:
|
||||
- img "User avatar" [ref=f9e37]
|
||||
- generic [ref=f9e38]:
|
||||
- button "Open menu" [ref=f9e40] [cursor=pointer]
|
||||
- navigation "Breadcrumbs" [ref=f9e43]:
|
||||
- list [ref=f9e44]:
|
||||
- listitem [ref=f9e45]:
|
||||
- link "Home" [ref=f9e46] [cursor=pointer]:
|
||||
- /url: /
|
||||
- listitem [ref=f9e50]:
|
||||
- link "Explore" [ref=f9e51] [cursor=pointer]:
|
||||
- /url: /explore
|
||||
- listitem [ref=f9e55]:
|
||||
- generic "Prometheus" [ref=f9e56]
|
||||
- generic [ref=f9e57]:
|
||||
- generic [ref=f9e60]:
|
||||
- button "Copy shortened URL" [ref=f9e61] [cursor=pointer]
|
||||
- button "Open copy link options" [ref=f9e64] [cursor=pointer]
|
||||
- button "Toggle top search bar" [ref=f9e68] [cursor=pointer]
|
||||
- main [ref=f9e74]:
|
||||
- generic [ref=f9e76]:
|
||||
- heading "Explore" [level=1] [ref=f9e77]
|
||||
- generic [ref=f9e82]:
|
||||
- navigation "Explore toolbar" [ref=f9e84]:
|
||||
- navigation "Search links" [ref=f9e86]:
|
||||
- generic [ref=f9e87]:
|
||||
- button "Content outline" [expanded] [ref=f9e89] [cursor=pointer]:
|
||||
- generic [ref=f9e92]: Outline
|
||||
- generic [ref=f9e97] [cursor=pointer]:
|
||||
- img "Prometheus logo" [ref=f9e99]
|
||||
- textbox "Select a data source" [ref=f9e100]:
|
||||
- /placeholder: ""
|
||||
- generic [ref=f9e104]:
|
||||
- button "Split the pane" [ref=f9e106] [cursor=pointer]:
|
||||
- generic [ref=f9e109]: Split
|
||||
- button "Add" [ref=f9e111] [cursor=pointer]
|
||||
- generic [ref=f9e116]:
|
||||
- 'button "Time range selected: Last 55 minutes" [ref=f9e117] [cursor=pointer]'
|
||||
- button "Zoom out time range" [ref=f9e122] [cursor=pointer]
|
||||
- generic [ref=f9e126]:
|
||||
- button "Run query" [ref=f9e127] [cursor=pointer]
|
||||
- button "Auto refresh turned off. Choose refresh time interval" [ref=f9e131] [cursor=pointer]
|
||||
- generic [ref=f9e135]:
|
||||
- generic [ref=f9e139]:
|
||||
- button "Collapse outline" [expanded] [ref=f9e141] [cursor=pointer]:
|
||||
- img "arrow-from-right" [ref=f9e142]
|
||||
- button "Queries" [ref=f9e145] [cursor=pointer]:
|
||||
- img "arrow" [ref=f9e146]
|
||||
- button "Graph" [ref=f9e150] [cursor=pointer]:
|
||||
- img "graph-bar" [ref=f9e151]
|
||||
- generic [ref=f9e158]:
|
||||
- generic [ref=f9e160]:
|
||||
- generic "Query editor row" [ref=f9e163]:
|
||||
- generic [ref=f9e164]:
|
||||
- generic [ref=f9e166]:
|
||||
- generic [ref=f9e167]:
|
||||
- button "Collapse query row" [expanded] [ref=f9e168] [cursor=pointer]
|
||||
- generic [ref=f9e171]:
|
||||
- button "Query editor row title A" [ref=f9e172] [cursor=pointer]:
|
||||
- generic [ref=f9e173]: A
|
||||
- emphasis [ref=f9e174]: (Prometheus)
|
||||
- generic [ref=f9e175]:
|
||||
- button "Show data source help" [ref=f9e177] [cursor=pointer]
|
||||
- button "Duplicate query" [ref=f9e181] [cursor=pointer]
|
||||
- button "Hide response" [ref=f9e185] [cursor=pointer]
|
||||
- button "Remove query" [ref=f9e189] [cursor=pointer]
|
||||
- button "Drag and drop to reorder" [ref=f9e192]:
|
||||
- img "Drag and drop to reorder" [ref=f9e193]
|
||||
- generic [ref=f9e196]:
|
||||
- generic [ref=f9e197]:
|
||||
- button "Kick start your query" [ref=f9e198] [cursor=pointer]
|
||||
- generic [ref=f9e201]:
|
||||
- generic [ref=f9e202] [cursor=pointer]: Explain
|
||||
- generic [ref=f9e203]:
|
||||
- checkbox "Explain Toggle switch" [ref=f9e204]
|
||||
- generic "Toggle switch" [ref=f9e205] [cursor=pointer]
|
||||
- radiogroup [ref=f9e210]:
|
||||
- generic [ref=f9e211]:
|
||||
- radio "Builder" [ref=f9e212] [cursor=pointer]
|
||||
- generic [ref=f9e213] [cursor=pointer]: Builder
|
||||
- generic [ref=f9e214]:
|
||||
- radio "Code" [checked] [ref=f9e215] [cursor=pointer]
|
||||
- generic [ref=f9e216] [cursor=pointer]: Code
|
||||
- generic [ref=f9e218]:
|
||||
- generic [ref=f9e220]:
|
||||
- button "Metrics browser" [ref=f9e221] [cursor=pointer]
|
||||
- code [ref=f9e228]:
|
||||
- generic [ref=f9e229]:
|
||||
- generic [ref=f9e234]: "up{job=~\"keycloak|node-exporter\"}"
|
||||
- textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=f9e239]: "up{job=~\"keycloak|node-exporter\"}"
|
||||
- 'button "Options Legend: {{job}} — {{pod}}{{node}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f9e245] [cursor=pointer]':
|
||||
- generic [ref=f9e249]:
|
||||
- heading "Options" [level=6] [ref=f9e250]
|
||||
- generic [ref=f9e251]:
|
||||
- generic [ref=f9e252]: "Legend: {{job}} — {{pod}}{{node}}"
|
||||
- generic [ref=f9e253]: "Format: Time series"
|
||||
- generic [ref=f9e254]: "Step: auto"
|
||||
- generic [ref=f9e255]: "Type: Range"
|
||||
- generic [ref=f9e256]: "Exemplars: false"
|
||||
- generic [ref=f9e257]:
|
||||
- button "Add query" [ref=f9e258] [cursor=pointer]
|
||||
- button "Query history" [ref=f9e262] [cursor=pointer]
|
||||
- button "Query inspector" [ref=f9e266] [cursor=pointer]
|
||||
- main [ref=f9e270]:
|
||||
- region [ref=f9e272]:
|
||||
- generic [ref=f9e273]:
|
||||
- heading "Graph" [level=2] [ref=f9e275]
|
||||
- radiogroup [ref=f9e278]:
|
||||
- generic [ref=f9e279]:
|
||||
- radio "Lines" [checked] [ref=f9e280] [cursor=pointer]
|
||||
- generic [ref=f9e281] [cursor=pointer]: Lines
|
||||
- generic [ref=f9e282]:
|
||||
- radio "Bars" [ref=f9e283] [cursor=pointer]
|
||||
- generic [ref=f9e284] [cursor=pointer]: Bars
|
||||
- generic [ref=f9e285]:
|
||||
- radio "Points" [ref=f9e286] [cursor=pointer]
|
||||
- generic [ref=f9e287] [cursor=pointer]: Points
|
||||
- generic [ref=f9e288]:
|
||||
- radio "Stacked lines" [ref=f9e289] [cursor=pointer]
|
||||
- generic [ref=f9e290] [cursor=pointer]: Stacked lines
|
||||
- generic [ref=f9e291]:
|
||||
- radio "Stacked bars" [ref=f9e292] [cursor=pointer]
|
||||
- generic [ref=f9e293] [cursor=pointer]: Stacked bars
|
||||
- list [ref=f9e311]:
|
||||
- listitem [ref=f9e312]:
|
||||
- button "keycloak — keycloak-1kc-lab-1" [ref=f9e316] [cursor=pointer]
|
||||
- listitem [ref=f9e317]:
|
||||
- button "keycloak — keycloak-1kc-lab-1" [ref=f9e321] [cursor=pointer]
|
||||
- listitem [ref=f9e322]:
|
||||
- button "keycloak — keycloak-0kc-lab-2" [ref=f9e326] [cursor=pointer]
|
||||
- listitem [ref=f9e327]:
|
||||
- button "keycloak — keycloak-0kc-lab-2" [ref=f9e331] [cursor=pointer]
|
||||
- listitem [ref=f9e332]:
|
||||
- button "keycloak — keycloak-0kc-lab-2" [ref=f9e336] [cursor=pointer]
|
||||
- listitem [ref=f9e337]:
|
||||
- button "node-exporter — kc-lab-1" [ref=f9e341] [cursor=pointer]
|
||||
- listitem [ref=f9e342]:
|
||||
- button "node-exporter — kc-lab-2" [ref=f9e346] [cursor=pointer]
|
||||
- generic [ref=f9e351]:
|
||||
- alert
|
||||
- alert
|
||||
- complementary
|
||||
- complementary
|
||||
@@ -1,104 +0,0 @@
|
||||
- generic [ref=f12e4]:
|
||||
- link "Skip to main content" [ref=f12e5] [cursor=pointer]:
|
||||
- /url: "#pageContent"
|
||||
- banner [ref=f12e7]:
|
||||
- generic [ref=f12e8]:
|
||||
- link [ref=f12e10] [cursor=pointer]:
|
||||
- /url: /
|
||||
- img "Grafana" [ref=f12e11]
|
||||
- generic [ref=f12e14]:
|
||||
- button "Search or jump to..." [ref=f12e18] [cursor=pointer]
|
||||
- generic [ref=f12e19]: ctrl+k
|
||||
- generic [ref=f12e23]:
|
||||
- button "New" [ref=f12e24] [cursor=pointer]
|
||||
- button "Help" [ref=f12e30] [cursor=pointer]
|
||||
- button "News" [ref=f12e33] [cursor=pointer]
|
||||
- button "Profile" [ref=f12e36] [cursor=pointer]:
|
||||
- img "User avatar" [ref=f12e37]
|
||||
- generic [ref=f12e38]:
|
||||
- button "Open menu" [ref=f12e40] [cursor=pointer]
|
||||
- navigation "Breadcrumbs" [ref=f12e43]:
|
||||
- list [ref=f12e44]:
|
||||
- listitem [ref=f12e45]:
|
||||
- link "Home" [ref=f12e46] [cursor=pointer]:
|
||||
- /url: /
|
||||
- listitem [ref=f12e50]:
|
||||
- link "Explore" [ref=f12e51] [cursor=pointer]:
|
||||
- /url: /explore
|
||||
- listitem [ref=f12e55]:
|
||||
- generic "Prometheus" [ref=f12e56]
|
||||
- generic [ref=f12e57]:
|
||||
- button "Show more items" [ref=f12e60] [cursor=pointer]
|
||||
- button "Toggle top search bar" [ref=f12e64] [cursor=pointer]
|
||||
- main [ref=f12e70]:
|
||||
- generic [ref=f12e72]:
|
||||
- heading "Explore" [level=1] [ref=f12e73]
|
||||
- generic [ref=f12e78]:
|
||||
- navigation "Explore toolbar" [ref=f12e80]:
|
||||
- navigation "Search links" [ref=f12e82]:
|
||||
- generic [ref=f12e83]:
|
||||
- button "Content outline" [expanded] [ref=f12e85] [cursor=pointer]:
|
||||
- generic [ref=f12e88]: Outline
|
||||
- generic [ref=f12e93] [cursor=pointer]:
|
||||
- img "Prometheus logo" [ref=f12e95]
|
||||
- textbox "Select a data source" [ref=f12e96]:
|
||||
- /placeholder: ""
|
||||
- button "Show more items" [ref=f12e102] [cursor=pointer]
|
||||
- generic [ref=f12e106]:
|
||||
- generic [ref=f12e110]:
|
||||
- button "Collapse outline" [expanded] [ref=f12e112] [cursor=pointer]:
|
||||
- img "arrow-from-right" [ref=f12e113]
|
||||
- button "Queries" [ref=f12e116] [cursor=pointer]:
|
||||
- img "arrow" [ref=f12e117]
|
||||
- generic [ref=f12e124]:
|
||||
- generic [ref=f12e126]:
|
||||
- generic "Query editor row" [ref=f12e129]:
|
||||
- generic [ref=f12e130]:
|
||||
- generic [ref=f12e132]:
|
||||
- generic [ref=f12e133]:
|
||||
- button "Collapse query row" [expanded] [ref=f12e134] [cursor=pointer]
|
||||
- generic [ref=f12e137]:
|
||||
- button "Query editor row title A" [ref=f12e138] [cursor=pointer]:
|
||||
- generic [ref=f12e139]: A
|
||||
- emphasis [ref=f12e140]: (Prometheus)
|
||||
- generic [ref=f12e141]:
|
||||
- button "Show data source help" [ref=f12e143] [cursor=pointer]
|
||||
- button "Duplicate query" [ref=f12e147] [cursor=pointer]
|
||||
- button "Hide response" [ref=f12e151] [cursor=pointer]
|
||||
- button "Remove query" [ref=f12e155] [cursor=pointer]
|
||||
- button "Drag and drop to reorder" [ref=f12e158]:
|
||||
- img "Drag and drop to reorder" [ref=f12e159]
|
||||
- generic [ref=f12e162]:
|
||||
- generic [ref=f12e163]:
|
||||
- button "Kick start your query" [ref=f12e164] [cursor=pointer]
|
||||
- generic [ref=f12e167]:
|
||||
- generic [ref=f12e168] [cursor=pointer]: Explain
|
||||
- generic [ref=f12e169]:
|
||||
- checkbox "Explain Toggle switch" [ref=f12e170]
|
||||
- generic "Toggle switch" [ref=f12e171] [cursor=pointer]
|
||||
- radiogroup [ref=f12e176]:
|
||||
- generic [ref=f12e177]:
|
||||
- radio "Builder" [ref=f12e178] [cursor=pointer]
|
||||
- generic [ref=f12e179] [cursor=pointer]: Builder
|
||||
- generic [ref=f12e180]:
|
||||
- radio "Code" [checked] [ref=f12e181] [cursor=pointer]
|
||||
- generic [ref=f12e182] [cursor=pointer]: Code
|
||||
- generic [ref=f12e184]:
|
||||
- generic [ref=f12e186]:
|
||||
- button "Loading metrics..." [disabled] [ref=f12e187] [cursor=pointer]
|
||||
- generic [ref=f12e190]: Loading editor
|
||||
- 'button "Options Legend: {{pod}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f12e198] [cursor=pointer]':
|
||||
- generic [ref=f12e202]:
|
||||
- heading "Options" [level=6] [ref=f12e203]
|
||||
- generic [ref=f12e204]:
|
||||
- generic [ref=f12e205]: "Legend: {{pod}}"
|
||||
- generic [ref=f12e206]: "Format: Time series"
|
||||
- generic [ref=f12e207]: "Step: auto"
|
||||
- generic [ref=f12e208]: "Type: Range"
|
||||
- generic [ref=f12e209]: "Exemplars: false"
|
||||
- generic [ref=f12e210]:
|
||||
- button "Add query" [ref=f12e211] [cursor=pointer]
|
||||
- button "Query history" [ref=f12e215] [cursor=pointer]
|
||||
- button "Query inspector" [ref=f12e219] [cursor=pointer]
|
||||
- generic:
|
||||
- main
|
||||
@@ -15,11 +15,6 @@ Keycloak을 중심으로 네 가지 브라우저 인증 통합 패턴을 같은
|
||||
- AP3: Backend-for-Frontend (BFF)
|
||||
- AP4: Edge forward-auth
|
||||
|
||||
세션 저장소·refresh token 경쟁·장애 복구는 네 패턴을 가로지르는 별도 축으로
|
||||
`develop-keycloak-session-store` 브랜치에서 진행합니다. 계획과 진행 상황은
|
||||
[`docs/session-store-lab-roadmap.md`](docs/session-store-lab-roadmap.md)에
|
||||
있습니다.
|
||||
|
||||
현재 `develop`의 공통 baseline은 Keycloak, PostgreSQL, Spring Boot API,
|
||||
nginx를 Docker Compose로 실행하는 토대입니다. 패턴별 구현은 이 baseline
|
||||
위에서 별도 브랜치로 진행합니다.
|
||||
@@ -108,3 +103,22 @@ Keycloak을 잠시 중지하고 export한 뒤 자동으로 다시 올립니다.
|
||||
|
||||
runtime export에는 실제 client secret과 credential hash가 포함될 수 있어
|
||||
gitignored `build/keycloak-export/`에 권한 `0600`으로만 저장됩니다.
|
||||
|
||||
## AP4: oauth2-proxy Edge Forward Auth
|
||||
|
||||
`develop-keycloak-pattern4`는 oauth2-proxy와 Nginx `auth_request`가
|
||||
인증을 edge에서 강제하는 패턴입니다.
|
||||
|
||||
```bash
|
||||
./scripts/verify-pattern4.sh
|
||||
```
|
||||
|
||||
첫 feature에서는 oauth2-proxy를 `http://localhost:4180`에 직접 노출해
|
||||
OIDC redirect/PKCE/callback과 forwarded-user를 분리 확인합니다. 두 번째
|
||||
feature부터 `http://localhost:8088` Nginx가 단일 진입점이며, 내부
|
||||
`auth_request`는 브라우저 요청을 login 302로, API 요청을 JSON 401로
|
||||
구분합니다. 최종 feature에서는 backend와 oauth2-proxy의 호스트 노출을
|
||||
제거하고 Nginx 헤더 덮어쓰기와 내부 토큰 검증으로 spoofing을 막습니다.
|
||||
자세한 내용은
|
||||
[`docs/ap4-edge-forward-auth.md`](docs/ap4-edge-forward-auth.md)를
|
||||
참고하세요.
|
||||
|
||||
@@ -1,8 +1,6 @@
|
||||
package com.example.keycloakpattern;
|
||||
|
||||
import java.util.Collections;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import org.springframework.security.core.annotation.AuthenticationPrincipal;
|
||||
@@ -11,8 +9,6 @@ import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
|
||||
import jakarta.servlet.http.HttpServletRequest;
|
||||
|
||||
@RestController
|
||||
@RequestMapping("/api")
|
||||
public class ApiController {
|
||||
@@ -22,38 +18,6 @@ public class ApiController {
|
||||
return Map.of("status", "ok", "service", "keycloak-pattern-api");
|
||||
}
|
||||
|
||||
/**
|
||||
* Reflects what actually reached the application after the proxy chain.
|
||||
*
|
||||
* <p>The reverse proxy contract is defined in {@code docs/reverse-proxy-headers.md}
|
||||
* for a single nginx hop. The lab runs {@code nginx -> Traefik -> pod}, so this
|
||||
* endpoint exists to measure the two-hop result instead of assuming it.
|
||||
*
|
||||
* <p>{@code scheme}, {@code secure} and {@code requestUrl} are the values Keycloak
|
||||
* uses to build the {@code iss} claim and redirect URLs. If forwarded headers are
|
||||
* lost or rewritten, the mismatch shows up here first.
|
||||
*/
|
||||
@GetMapping("/echo")
|
||||
public Map<String, Object> echo(HttpServletRequest request) {
|
||||
Map<String, List<String>> headers = new LinkedHashMap<>();
|
||||
for (String name : Collections.list(request.getHeaderNames())) {
|
||||
headers.put(name.toLowerCase(), Collections.list(request.getHeaders(name)));
|
||||
}
|
||||
|
||||
Map<String, Object> response = new LinkedHashMap<>();
|
||||
response.put("headers", headers);
|
||||
response.put("remoteAddr", request.getRemoteAddr());
|
||||
// Pod IP. Identifies which replica answered, which is what makes the
|
||||
// host nginx upstream distribution and the sticky-session switch observable.
|
||||
response.put("localAddr", request.getLocalAddr());
|
||||
response.put("scheme", request.getScheme());
|
||||
response.put("secure", request.isSecure());
|
||||
response.put("serverName", request.getServerName());
|
||||
response.put("serverPort", request.getServerPort());
|
||||
response.put("requestUrl", request.getRequestURL().toString());
|
||||
return response;
|
||||
}
|
||||
|
||||
@GetMapping("/me")
|
||||
public Map<String, Object> currentUser(@AuthenticationPrincipal Jwt jwt) {
|
||||
Map<String, Object> response = new LinkedHashMap<>();
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
package com.example.keycloakpattern;
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.MessageDigest;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Map;
|
||||
|
||||
import jakarta.servlet.http.HttpServletRequest;
|
||||
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
|
||||
@RestController
|
||||
public class EdgeIdentityController {
|
||||
|
||||
private final byte[] internalAuthToken;
|
||||
|
||||
EdgeIdentityController(@Value("${edge.internal-auth-token}") String internalAuthToken) {
|
||||
if (!hasText(internalAuthToken)) {
|
||||
throw new IllegalStateException("edge.internal-auth-token must be configured");
|
||||
}
|
||||
this.internalAuthToken = internalAuthToken.getBytes(StandardCharsets.UTF_8);
|
||||
}
|
||||
|
||||
@GetMapping("/edge/me")
|
||||
ResponseEntity<Map<String, Object>> currentUser(HttpServletRequest request) {
|
||||
String authRequestUser = request.getHeader("X-Auth-Request-User");
|
||||
if (!hasText(authRequestUser) || !hasValidInternalToken(request)) {
|
||||
return ResponseEntity.status(401).body(Map.of(
|
||||
"error",
|
||||
"trusted edge authentication is required"
|
||||
));
|
||||
}
|
||||
|
||||
Map<String, Object> response = new LinkedHashMap<>();
|
||||
response.put("pattern", "AP4-edge-forward-auth");
|
||||
response.put("user", authRequestUser);
|
||||
response.put("email", request.getHeader("X-Auth-Request-Email"));
|
||||
response.put("identityHeader", "X-Auth-Request-User");
|
||||
return ResponseEntity.ok(response);
|
||||
}
|
||||
|
||||
private boolean hasValidInternalToken(HttpServletRequest request) {
|
||||
String suppliedToken = request.getHeader("X-Internal-Auth-Token");
|
||||
if (!hasText(suppliedToken)) {
|
||||
return false;
|
||||
}
|
||||
return MessageDigest.isEqual(
|
||||
internalAuthToken,
|
||||
suppliedToken.getBytes(StandardCharsets.UTF_8)
|
||||
);
|
||||
}
|
||||
|
||||
private static boolean hasText(String value) {
|
||||
return value != null && !value.isBlank();
|
||||
}
|
||||
}
|
||||
@@ -17,8 +17,12 @@ public class SecurityConfig {
|
||||
.sessionManagement(session ->
|
||||
session.sessionCreationPolicy(SessionCreationPolicy.STATELESS))
|
||||
.authorizeHttpRequests(authorize -> authorize
|
||||
.requestMatchers("/actuator/health", "/actuator/health/**", "/api/public",
|
||||
"/api/echo")
|
||||
.requestMatchers(
|
||||
"/actuator/health",
|
||||
"/actuator/health/**",
|
||||
"/api/public",
|
||||
"/edge/**"
|
||||
)
|
||||
.permitAll()
|
||||
.anyRequest()
|
||||
.authenticated())
|
||||
|
||||
@@ -1,19 +1,12 @@
|
||||
server:
|
||||
port: ${SERVER_PORT:8081}
|
||||
# Spring ignores X-Forwarded-* unless this is set, so scheme/secure/requestUrl
|
||||
# report the raw connection by default. Keycloak has the same opt-in as
|
||||
# KC_PROXY_HEADERS. Flipping this to "native" is what the two-hop measurement
|
||||
# compares against.
|
||||
forward-headers-strategy: ${SERVER_FORWARD_HEADERS_STRATEGY:none}
|
||||
|
||||
edge:
|
||||
internal-auth-token: ${EDGE_INTERNAL_AUTH_TOKEN:}
|
||||
|
||||
spring:
|
||||
application:
|
||||
name: keycloak-pattern-api
|
||||
jackson:
|
||||
serialization:
|
||||
# /api/echo is read by humans and captured as evidence screenshots, so the
|
||||
# response is indented rather than relying on a browser's JSON viewer.
|
||||
indent-output: true
|
||||
security:
|
||||
oauth2:
|
||||
resourceserver:
|
||||
|
||||
@@ -11,7 +11,7 @@ import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMock
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.test.web.servlet.MockMvc;
|
||||
|
||||
@SpringBootTest
|
||||
@SpringBootTest(properties = "edge.internal-auth-token=test-internal-edge-token")
|
||||
@AutoConfigureMockMvc
|
||||
class ApiSecurityTest {
|
||||
|
||||
@@ -25,18 +25,6 @@ class ApiSecurityTest {
|
||||
.andExpect(jsonPath("$.status").value("ok"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void echoEndpointReflectsForwardedHeadersWithoutAuthentication() throws Exception {
|
||||
mockMvc.perform(get("/api/echo")
|
||||
.header("X-Forwarded-Proto", "https")
|
||||
.header("X-Forwarded-Host", "app1.example.test"))
|
||||
.andExpect(status().isOk())
|
||||
.andExpect(jsonPath("$.headers['x-forwarded-proto'][0]").value("https"))
|
||||
.andExpect(jsonPath("$.headers['x-forwarded-host'][0]").value("app1.example.test"))
|
||||
.andExpect(jsonPath("$.requestUrl").exists())
|
||||
.andExpect(jsonPath("$.remoteAddr").exists());
|
||||
}
|
||||
|
||||
@Test
|
||||
void protectedEndpointRejectsAnonymousRequests() throws Exception {
|
||||
mockMvc.perform(get("/api/me"))
|
||||
@@ -52,4 +40,36 @@ class ApiSecurityTest {
|
||||
.andExpect(jsonPath("$.subject").value("test-subject"))
|
||||
.andExpect(jsonPath("$.username").value("regular-user"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void edgeEndpointRejectsMissingTrustedHeaders() throws Exception {
|
||||
mockMvc.perform(get("/edge/me"))
|
||||
.andExpect(status().isUnauthorized());
|
||||
}
|
||||
|
||||
@Test
|
||||
void edgeEndpointRejectsForgedIdentityWithoutInternalToken() throws Exception {
|
||||
mockMvc.perform(get("/edge/me")
|
||||
.header("X-Auth-Request-User", "spoofed-admin"))
|
||||
.andExpect(status().isUnauthorized());
|
||||
}
|
||||
|
||||
@Test
|
||||
void edgeEndpointRejectsWrongInternalToken() throws Exception {
|
||||
mockMvc.perform(get("/edge/me")
|
||||
.header("X-Auth-Request-User", "spoofed-admin")
|
||||
.header("X-Internal-Auth-Token", "wrong-token"))
|
||||
.andExpect(status().isUnauthorized());
|
||||
}
|
||||
|
||||
@Test
|
||||
void edgeEndpointAcceptsIdentityFromTrustedEdge() throws Exception {
|
||||
mockMvc.perform(get("/edge/me")
|
||||
.header("X-Auth-Request-User", "regular-user")
|
||||
.header("X-Auth-Request-Email", "regular-user@example.test")
|
||||
.header("X-Internal-Auth-Token", "test-internal-edge-token"))
|
||||
.andExpect(status().isOk())
|
||||
.andExpect(jsonPath("$.user").value("regular-user"))
|
||||
.andExpect(jsonPath("$.identityHeader").value("X-Auth-Request-User"));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,136 +0,0 @@
|
||||
# Session store lab
|
||||
|
||||
세션 저장소·refresh token 경쟁·장애 복구를 검증하는 2노드 k3s 실험대.
|
||||
네 인증 패턴(AP1~AP4)을 가로지르는 공통층이므로 별도 축으로 관리한다.
|
||||
|
||||
이 문서는 **절차**만 담는다.
|
||||
|
||||
| 문서 | 내용 |
|
||||
|---|---|
|
||||
| [`docs/session-store-lab-roadmap.md`](../../docs/session-store-lab-roadmap.md) | 이 축의 계획과 진행 상황 |
|
||||
| [`docs/session-lab-concepts.md`](../../docs/session-lab-concepts.md) | 등장 개념 전체 |
|
||||
| [`docs/session-lab-operations.md`](../../docs/session-lab-operations.md) | 관측 도구 · 자주 쓰는 명령 · 훈련 |
|
||||
| [`docs/two-hop-proxy-header-contract.md`](../../docs/two-hop-proxy-header-contract.md) | 첫 실험 결과 |
|
||||
|
||||
## 토폴로지
|
||||
|
||||
```
|
||||
브라우저 / SSH (tailnet)
|
||||
│ https://{auth,app1,app2}.hyeonworks.com → 100.83.212.4
|
||||
▼
|
||||
lab host ── nginx :443 TLS 종료 · X-Forwarded-* 주입
|
||||
│ nginx :80 301 → https
|
||||
│
|
||||
│ virbr0 192.168.122.0/24 (libvirt NAT)
|
||||
├──▶ kc-lab-1 .11 k3s server Traefik :80
|
||||
└──▶ kc-lab-2 .12 k3s agent Traefik :80
|
||||
└──▶ Pod
|
||||
```
|
||||
|
||||
`nginx → Traefik` **2홉**이 운영 구조와 같다는 점이 이 배치의 핵심이다.
|
||||
L7 프록시가 두 겹인 이유는 역할이 다르기 때문이다 — nginx는 바깥세상과의
|
||||
접점(TLS·인증서·헤더)을, Traefik은 클러스터 내부의 동적 라우팅을 맡는다.
|
||||
|
||||
## 구성 요소
|
||||
|
||||
| 경로 | 역할 |
|
||||
|---|---|
|
||||
| `cloud-init/kc-lab.yaml.example` | 게스트 부트스트랩 템플릿 |
|
||||
| `host/nginx-keycloak-lab.conf` | lab host의 `sites-available/keycloak-lab` |
|
||||
| `k8s/echo.yaml` | 2홉 헤더 계약 측정용 워크로드 |
|
||||
| `scripts/rebuild-seed.sh` | cloud-init 시드 ISO 재생성 + 풀 업로드 |
|
||||
| `scripts/build-and-import.sh` | 이미지 빌드 → 각 노드 containerd 반입 |
|
||||
| `scripts/measure-proxy-headers.sh` | 헤더 계약 실측 |
|
||||
| `scripts/verify-lab.sh` | 인프라 상태 점검 |
|
||||
|
||||
## 상태 점검
|
||||
|
||||
```bash
|
||||
./deploy/lab/scripts/verify-lab.sh # lab host 에서
|
||||
```
|
||||
|
||||
**`404`가 성공 신호다.** TLS가 종료되고 Traefik까지 도달했으나 매칭되는
|
||||
Ingress 규칙이 없다는 뜻이다. `502`나 연결 거부면 체인이 끊긴 것이다.
|
||||
|
||||
## 첫 실험 — 2홉 헤더 계약
|
||||
|
||||
[`docs/reverse-proxy-headers.md`](../../docs/reverse-proxy-headers.md)의 계약은
|
||||
nginx **1홉**을 가정하고 쓰였다. 실제 배치는 2홉이므로, nginx가 세팅한
|
||||
`X-Forwarded-*`를 Traefik이 그대로 넘기는지 덮어쓰는지 **측정해서 확인한다.**
|
||||
|
||||
이 결론이 뒤의 모든 실험에 깔린다. Keycloak의 `iss` 클레임, redirect URL,
|
||||
쿠키 도메인 검증이 전부 이 헤더에 의존하기 때문이다.
|
||||
|
||||
```bash
|
||||
# 워크스테이션: 이미지 빌드 후 두 노드에 반입
|
||||
./deploy/lab/scripts/build-and-import.sh
|
||||
|
||||
# lab host: 배포
|
||||
kubectl apply -f deploy/lab/k8s/echo.yaml
|
||||
kubectl -n header-lab rollout status deployment/echo
|
||||
|
||||
# 어디서든: 실측
|
||||
./deploy/lab/scripts/measure-proxy-headers.sh
|
||||
```
|
||||
|
||||
관측 대상은 넷이다.
|
||||
|
||||
1. `X-Forwarded-For` — Traefik이 **덧붙이는가 덮어쓰는가**
|
||||
2. `X-Forwarded-Proto` / `-Host` / `-Port` — 그대로 전달되는가
|
||||
3. **위조 내성** — 클라이언트가 직접 넣은 `X-Forwarded-*`가 앱까지 도달하는가
|
||||
4. `scheme` / `secure` / `requestUrl` — Keycloak이 URL을 만들 때 쓰는 값
|
||||
|
||||
3번이 신뢰 경계의 핵심이다. 이 헤더들은 누구나 위조할 수 있는 평범한 HTTP
|
||||
헤더이므로, 신뢰 경계에 선 프록시가 **반드시 덮어써야** 한다.
|
||||
|
||||
## 이미지 배포 경로
|
||||
|
||||
k3s는 containerd를 쓰고 이 실험대에는 레지스트리가 없다.
|
||||
|
||||
```
|
||||
워크스테이션 docker build → docker save
|
||||
│ ssh (lab host 경유)
|
||||
▼
|
||||
게스트 sudo k3s ctr images import
|
||||
매니페스트 imagePullPolicy: Never
|
||||
```
|
||||
|
||||
**두 노드 모두에 반입해야 한다.** 스케줄러가 어느 노드에 배치할지 모른다.
|
||||
Keycloak·PostgreSQL·Redis는 공식 이미지를 그대로 당겨오므로 이 경로가
|
||||
필요한 것은 자체 빌드 이미지뿐이다.
|
||||
|
||||
**lab host에 Docker를 설치하지 않는다.** k3s의 containerd와 이미지 저장소가
|
||||
갈려서 `docker build`한 이미지를 k3s가 보지 못하게 된다.
|
||||
|
||||
## 게스트 재생성
|
||||
|
||||
파괴적 실험 후 초기화하는 경로다.
|
||||
|
||||
```bash
|
||||
virsh destroy kc-lab-1
|
||||
virsh undefine kc-lab-1 # --remove-all-storage 는 시드 ISO 까지 지운다
|
||||
virsh vol-delete --pool default kc-lab-1.qcow2
|
||||
|
||||
./deploy/lab/scripts/rebuild-seed.sh 1 # user-data 를 고쳤을 때만
|
||||
|
||||
virt-install --name kc-lab-1 --memory 3584 --vcpus 2 \
|
||||
--disk size=20,backing_store=/var/lib/libvirt/images/base.qcow2 \
|
||||
--disk vol=default/seed-kc-lab-1.iso,device=disk,bus=virtio,readonly=on \
|
||||
--network network=default,mac=52:54:00:aa:bb:11 \
|
||||
--import --os-variant debian12 --noautoconsole
|
||||
```
|
||||
|
||||
시드는 **virtio 디스크**로 붙인다. `virt-install --cloud-init`은 시드를 SATA
|
||||
CD-ROM으로 붙이는데, Debian `genericcloud` 이미지는 크기를 줄이려고 물리
|
||||
하드웨어 드라이버를 제외해서 **AHCI 장치를 보지 못한다.** 그러면 cloud-init이
|
||||
데이터소스를 찾지 못하고 아무 오류도 남기지 않은 채 종료한다. 증상은
|
||||
hostname이 `localhost`로 남고 SSH가 `Permission denied (publickey)`로 거부되는
|
||||
것뿐이다.
|
||||
|
||||
게스트에 들어갈 수 없을 때는 화면을 직접 뜬다.
|
||||
|
||||
```bash
|
||||
virsh screenshot kc-lab-1 /tmp/kc1.ppm # 확장자와 무관하게 PNG 로 저장된다
|
||||
```
|
||||
|
||||
`localhost login:`이면 cloud-init 미실행, `kc-lab-1 login:`이면 실행된 것이다.
|
||||
@@ -1,37 +0,0 @@
|
||||
#cloud-config
|
||||
# Template for both lab guests. scripts/rebuild-seed.sh substitutes __NODE__
|
||||
# and bakes this into a CIDATA seed image.
|
||||
#
|
||||
# Copy to kc-lab.yaml and fill the two placeholders. The real file is ignored by
|
||||
# git because plain_text_passwd is a credential, however disposable.
|
||||
#
|
||||
# Indentation is spaces only. YAML forbids tabs, and cloud-init fails silently
|
||||
# on a parse error: the guest boots as "localhost" with no user and no way in.
|
||||
hostname: kc-lab-__NODE__
|
||||
fqdn: kc-lab-__NODE__
|
||||
manage_etc_hosts: true
|
||||
|
||||
users:
|
||||
- name: donghyeon
|
||||
groups: [sudo]
|
||||
shell: /bin/bash
|
||||
# NOPASSWD is required: the k3s installer and the fault-injection scripts
|
||||
# run non-interactively and would block on a password prompt.
|
||||
sudo: ['ALL=(ALL) NOPASSWD:ALL']
|
||||
# Console-only escape hatch. Without it, a cloud-init failure leaves a guest
|
||||
# that cannot be logged into at all, so its own failure log is unreadable.
|
||||
# ssh_pwauth stays false, so this never widens SSH exposure.
|
||||
lock_passwd: false
|
||||
plain_text_passwd: CHANGE_ME
|
||||
ssh_authorized_keys:
|
||||
# Lab host key: needed because automation runs from the lab host, where
|
||||
# agent forwarding is not available.
|
||||
- CHANGE_ME_LAB_HOST_PUBLIC_KEY
|
||||
# Workstation key: lets ProxyJump reach the guest directly.
|
||||
- CHANGE_ME_WORKSTATION_PUBLIC_KEY
|
||||
|
||||
ssh_pwauth: false
|
||||
package_update: true
|
||||
packages:
|
||||
- curl
|
||||
- nftables
|
||||
@@ -1,56 +0,0 @@
|
||||
# Lab entry point. Deployed on the lab host as
|
||||
# /etc/nginx/sites-available/keycloak-lab
|
||||
# and symlinked from sites-enabled/.
|
||||
#
|
||||
# Arch does not ship the Debian sites-available convention, so nginx.conf needs
|
||||
# include /etc/nginx/sites-enabled/*;
|
||||
# inside its http { } block before this file has any effect.
|
||||
#
|
||||
# This is the outer of two L7 hops. It terminates TLS and hands plain HTTP to
|
||||
# the Traefik instance running on each k3s node.
|
||||
|
||||
upstream k3s_traefik {
|
||||
# Sticky-session switch. Keycloak recommends affinity on AUTH_SESSION_ID;
|
||||
# ip_hash is the cheap stand-in for a single-browser lab. Leaving it off is
|
||||
# the interesting case: Infinispan still routes correctly, only slower.
|
||||
# ip_hash;
|
||||
server 192.168.122.11:80;
|
||||
server 192.168.122.12:80;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 80 default_server;
|
||||
server_name _;
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl default_server;
|
||||
http2 on;
|
||||
server_name _;
|
||||
|
||||
# fullchain.pem, never cert.pem: omitting the intermediates passes on
|
||||
# desktop browsers and fails on mobile and curl.
|
||||
ssl_certificate /etc/letsencrypt/live/auth.hyeonworks.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/auth.hyeonworks.com/privkey.pem;
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
|
||||
location / {
|
||||
proxy_pass http://k3s_traefik;
|
||||
proxy_http_version 1.1;
|
||||
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-Host $host;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
proxy_set_header X-Forwarded-Port 443;
|
||||
|
||||
# $remote_addr, not $proxy_add_x_forwarded_for. This is the trust
|
||||
# boundary: a client-supplied X-Forwarded-For must be discarded, not
|
||||
# extended, or nothing downstream can rely on the value.
|
||||
proxy_set_header X-Forwarded-For $remote_addr;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_send_timeout 3600s;
|
||||
}
|
||||
}
|
||||
@@ -1,46 +0,0 @@
|
||||
# Experiment A-1 — cut the JGroups transport (TCP 7800) while leaving discovery alone.
|
||||
#
|
||||
# The point is to separate two things that are easy to conflate:
|
||||
#
|
||||
# discovery how the nodes FIND each other -> PostgreSQL JGROUPS_PING table
|
||||
# transport how they actually TALK -> TCP 7800
|
||||
#
|
||||
# Blocking only the transport produces a state that cannot happen on a single
|
||||
# node: both members stay registered in the database, so each believes the other
|
||||
# exists, yet no message gets through.
|
||||
#
|
||||
# kubectl apply -f deploy/lab/k8s/a1-block-jgroups-transport.yaml
|
||||
# kubectl -n keycloak-lab delete networkpolicy a1-block-jgroups-transport
|
||||
#
|
||||
# NetworkPolicy is an ALLOWLIST, not a firewall with deny rules. There is no way
|
||||
# to write "deny 7800". The moment a pod is selected by a policy carrying
|
||||
# policyTypes: [Ingress], every inbound port is denied unless a rule permits it.
|
||||
# So 7800 is blocked by *omission*: 8080 and 9000 are listed, 7800 is not.
|
||||
#
|
||||
# That makes the two allow rules load-bearing — get them wrong and the experiment
|
||||
# measures a dead Keycloak instead of a partitioned cluster:
|
||||
#
|
||||
# 8080 the HTTP endpoint. Traefik, the other pod's REST calls, and the probe
|
||||
# traffic all arrive here.
|
||||
# 9000 the management port: /health/started, /health/ready, /health/live and
|
||||
# /metrics. Losing it means the kubelet fails the readiness probe and
|
||||
# kills the pod — the cluster would break for the wrong reason.
|
||||
#
|
||||
# Both rules deliberately omit `from:`, which allows those ports from any source.
|
||||
# Narrowing the source is not the subject here; the 2-hop experiment already
|
||||
# established how to do that by label when it matters.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: a1-block-jgroups-transport
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: keycloak
|
||||
policyTypes: [Ingress]
|
||||
ingress:
|
||||
- ports:
|
||||
- { port: 8080, protocol: TCP } # HTTP — must stay open
|
||||
- { port: 9000, protocol: TCP } # health + metrics — must stay open
|
||||
# 7800 is absent on purpose. That is the whole experiment.
|
||||
@@ -1,62 +0,0 @@
|
||||
# Restrict who may reach the echo pods.
|
||||
#
|
||||
# Traefik is configured to trust X-Forwarded-* from the whole pod CIDR, and the
|
||||
# app's Tomcat valve trusts every private range by default. Both are IP-range
|
||||
# decisions, so any pod in the cluster can forge those headers by talking to the
|
||||
# Service directly and bypassing Traefik entirely. Measured, not hypothetical:
|
||||
#
|
||||
# kubectl -n header-lab run t --rm -i --restart=Never --image=curlimages/curl -- \
|
||||
# curl -s http://echo:8081/api/echo -H 'X-Forwarded-Host: evil.example.com'
|
||||
# → serverName evil.example.com, remoteAddr 1.2.3.4
|
||||
#
|
||||
# A NetworkPolicy closes that path. It selects by label rather than IP, so it
|
||||
# survives pod restarts and rescheduling — unlike the trustedIPs list, which
|
||||
# could not name Traefik because its IP changes.
|
||||
#
|
||||
# "Trusting forwarded headers" and "guaranteeing a proxy sits in front" are a
|
||||
# pair. Doing only the first leaves this hole.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: echo-allow-traefik-only
|
||||
namespace: header-lab
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: echo
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
# The proxy itself. namespaceSelector and podSelector in one list item are
|
||||
# ANDed, so this is "traefik pods in kube-system" and nothing else.
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: traefik
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8081
|
||||
|
||||
# kubelet readiness/liveness probes originate from the node, not from a pod,
|
||||
# so they need their own rule. Without it the probes fail and the pods are
|
||||
# restarted in a loop.
|
||||
#
|
||||
# The probe's source address is the node's flannel bridge (cni0), which
|
||||
# holds the first address of that node's /24:
|
||||
# kc-lab-1 10.42.0.1 kc-lab-2 10.42.1.1
|
||||
# Listing them as /32 keeps this rule from re-admitting arbitrary pods,
|
||||
# which a broader 10.42.0.0/16 block would do and would undo the policy.
|
||||
#
|
||||
# Adding a node means adding its gateway here. Verify with:
|
||||
# kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{end}'
|
||||
- from:
|
||||
- ipBlock:
|
||||
cidr: 10.42.0.1/32
|
||||
- ipBlock:
|
||||
cidr: 10.42.1.1/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8081
|
||||
@@ -1,113 +0,0 @@
|
||||
# Header echo workload for the two-hop proxy contract measurement.
|
||||
#
|
||||
# browser -> host nginx (TLS termination) -> Traefik -> this pod
|
||||
#
|
||||
# The image is built from backend/ and imported straight into each node's
|
||||
# containerd, so imagePullPolicy must stay Never. See scripts/build-and-import.sh.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: header-lab
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: echo
|
||||
namespace: header-lab
|
||||
spec:
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app: echo
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: echo
|
||||
spec:
|
||||
# One replica per node so the sticky-session switch on the host nginx
|
||||
# upstream has something observable to route between.
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1
|
||||
topologyKey: kubernetes.io/hostname
|
||||
whenUnsatisfiable: ScheduleAnyway
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app: echo
|
||||
containers:
|
||||
- name: echo
|
||||
image: keycloak-pattern-api:lab
|
||||
imagePullPolicy: Never
|
||||
ports:
|
||||
- containerPort: 8081
|
||||
name: http
|
||||
env:
|
||||
- name: SERVER_PORT
|
||||
value: "8081"
|
||||
# "none" makes the app report the raw connection, so scheme/secure/
|
||||
# requestUrl show what arrives without any forwarded-header handling.
|
||||
# Set to "native" and redeploy to see the same request interpreted
|
||||
# with X-Forwarded-* honoured. Keycloak's KC_PROXY_HEADERS is the
|
||||
# same opt-in, which is why measuring both sides matters here.
|
||||
- name: SERVER_FORWARD_HEADERS_STRATEGY
|
||||
value: "native"
|
||||
# The JVM sizes its heap from the container limit, not the host.
|
||||
- name: JAVA_TOOL_OPTIONS
|
||||
value: "-XX:MaxRAMPercentage=70"
|
||||
# /api/echo is permitAll, so the JWT decoder is never exercised.
|
||||
# These stay pointed at the future Keycloak service name.
|
||||
- name: SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI
|
||||
value: "https://auth.hyeonworks.com/realms/keycloak-patterns"
|
||||
- name: SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI
|
||||
value: "https://auth.hyeonworks.com/realms/keycloak-patterns/protocol/openid-connect/certs"
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /actuator/health/readiness
|
||||
port: http
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 5
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /actuator/health/liveness
|
||||
port: http
|
||||
initialDelaySeconds: 45
|
||||
periodSeconds: 15
|
||||
resources:
|
||||
requests:
|
||||
memory: 320Mi
|
||||
cpu: 100m
|
||||
limits:
|
||||
memory: 512Mi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: echo
|
||||
namespace: header-lab
|
||||
spec:
|
||||
selector:
|
||||
app: echo
|
||||
ports:
|
||||
- port: 8081
|
||||
targetPort: http
|
||||
name: http
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: echo
|
||||
namespace: header-lab
|
||||
spec:
|
||||
# k3s ships Traefik as the default ingress controller. Keeping it is what
|
||||
# makes this lab a faithful two-hop replica.
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: app1.hyeonworks.com
|
||||
http:
|
||||
paths:
|
||||
- path: /api
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: echo
|
||||
port:
|
||||
number: 8081
|
||||
@@ -1,277 +0,0 @@
|
||||
# Keycloak multi-node cluster with PostgreSQL.
|
||||
#
|
||||
# Goal of this manifest: two Keycloak pods on two different nodes must discover
|
||||
# each other and form one Infinispan cluster. Keycloak 26 discovers peers through
|
||||
# the database (jdbc-ping) rather than multicast, writing to a JGROUPS_PING table,
|
||||
# but the cluster traffic itself runs over TCP 7800 between the pods. Those are
|
||||
# two separate mechanisms, which is why "registered in the DB but not clustered"
|
||||
# is a real failure mode — and one that a single node cannot reproduce.
|
||||
#
|
||||
# kubectl apply -f deploy/lab/k8s/keycloak-cluster.yaml
|
||||
# kubectl -n keycloak-lab rollout status statefulset/keycloak --timeout=600s
|
||||
#
|
||||
# Secrets are plain here. Proper secret handling is roadmap item 11; keeping it
|
||||
# visible for now is deliberate so the gap is obvious rather than forgotten.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: keycloak-lab
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: keycloak-lab-secrets
|
||||
namespace: keycloak-lab
|
||||
type: Opaque
|
||||
stringData:
|
||||
POSTGRES_PASSWORD: lab-postgres-change-me
|
||||
KC_BOOTSTRAP_ADMIN_PASSWORD: lab-admin-change-me
|
||||
---
|
||||
# PostgreSQL. local-path binds the volume to whichever node the pod lands on, so
|
||||
# the database is effectively pinned to one node. That is not a flaw here: it is
|
||||
# what makes "the database node dies" a meaningful experiment later.
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: postgres-data
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: local-path
|
||||
resources:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: postgres
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate # RWO volume cannot be mounted by two pods at once
|
||||
selector:
|
||||
matchLabels:
|
||||
app: postgres
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: postgres
|
||||
spec:
|
||||
containers:
|
||||
- name: postgres
|
||||
image: postgres:16-alpine
|
||||
ports:
|
||||
- containerPort: 5432
|
||||
name: postgres
|
||||
env:
|
||||
- name: POSTGRES_DB
|
||||
value: keycloak
|
||||
- name: POSTGRES_USER
|
||||
value: keycloak
|
||||
- name: POSTGRES_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: keycloak-lab-secrets
|
||||
key: POSTGRES_PASSWORD
|
||||
# The image refuses to initialise into a non-empty mount, and
|
||||
# local-path volumes are clean, but this keeps the data one level
|
||||
# down so a lost+found or similar never blocks initdb.
|
||||
- name: PGDATA
|
||||
value: /var/lib/postgresql/data/pgdata
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /var/lib/postgresql/data
|
||||
readinessProbe:
|
||||
exec:
|
||||
command: ["sh", "-c", "pg_isready -U keycloak -d keycloak"]
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 5
|
||||
resources:
|
||||
requests:
|
||||
memory: 192Mi
|
||||
cpu: 50m
|
||||
limits:
|
||||
memory: 512Mi
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: postgres-data
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: postgres
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
selector:
|
||||
app: postgres
|
||||
ports:
|
||||
- port: 5432
|
||||
targetPort: postgres
|
||||
---
|
||||
# Keycloak. A StatefulSet rather than a Deployment so each pod keeps a stable
|
||||
# name (keycloak-0, keycloak-1); cluster membership is far easier to read in
|
||||
# logs and in the JGROUPS_PING table when the identities do not churn.
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: keycloak
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
serviceName: keycloak-headless
|
||||
replicas: 2
|
||||
podManagementPolicy: Parallel # both pods start together, so they race to
|
||||
# register — which is the interesting case
|
||||
selector:
|
||||
matchLabels:
|
||||
app: keycloak
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: keycloak
|
||||
spec:
|
||||
# One pod per node. Two pods on one node would share a kernel and make the
|
||||
# 7800 blocking experiment meaningless.
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1
|
||||
topologyKey: kubernetes.io/hostname
|
||||
whenUnsatisfiable: ScheduleAnyway
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app: keycloak
|
||||
containers:
|
||||
- name: keycloak
|
||||
image: quay.io/keycloak/keycloak:26.7.0
|
||||
# "start", not "start-dev". Dev mode forces cache=local and there is
|
||||
# no cluster to form at all.
|
||||
args: ["start"]
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
name: http
|
||||
- containerPort: 9000
|
||||
name: management
|
||||
- containerPort: 7800
|
||||
name: jgroups
|
||||
env:
|
||||
- name: KC_DB
|
||||
value: postgres
|
||||
- name: KC_DB_URL
|
||||
value: jdbc:postgresql://postgres:5432/keycloak
|
||||
- name: KC_DB_USERNAME
|
||||
value: keycloak
|
||||
- name: KC_DB_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: keycloak-lab-secrets
|
||||
key: POSTGRES_PASSWORD
|
||||
|
||||
# Settings confirmed by the two-hop header measurement.
|
||||
# KC_HOSTNAME carries the full external URL, which pins scheme and
|
||||
# host for issuer and redirect URLs regardless of headers.
|
||||
# KC_PROXY_HEADERS is the separate opt-in that lets the forwarded
|
||||
# client address through — the same kind of switch as Spring's
|
||||
# forward-headers-strategy. See docs/two-hop-proxy-header-contract.md.
|
||||
- name: KC_HOSTNAME
|
||||
value: https://auth.hyeonworks.com
|
||||
- name: KC_HOSTNAME_STRICT
|
||||
value: "true"
|
||||
- name: KC_PROXY_HEADERS
|
||||
value: xforwarded
|
||||
- name: KC_HTTP_ENABLED
|
||||
value: "true"
|
||||
|
||||
- name: KC_HEALTH_ENABLED
|
||||
value: "true"
|
||||
- name: KC_METRICS_ENABLED
|
||||
value: "true"
|
||||
|
||||
# Without an explicit cap the JVM sizes its heap from the container
|
||||
# limit and this lab has roughly 3.8GB of guest headroom in total.
|
||||
- name: JAVA_OPTS_KC_HEAP
|
||||
value: "-Xms256m -Xmx512m"
|
||||
|
||||
- name: KC_BOOTSTRAP_ADMIN_USERNAME
|
||||
value: admin
|
||||
- name: KC_BOOTSTRAP_ADMIN_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: keycloak-lab-secrets
|
||||
key: KC_BOOTSTRAP_ADMIN_PASSWORD
|
||||
|
||||
# Keycloak serves health and metrics on the management port (9000),
|
||||
# not on 8080, since version 25.
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /health/started
|
||||
port: management
|
||||
periodSeconds: 10
|
||||
failureThreshold: 60 # first boot runs an implicit build
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health/ready
|
||||
port: management
|
||||
periodSeconds: 10
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health/live
|
||||
port: management
|
||||
periodSeconds: 30
|
||||
resources:
|
||||
requests:
|
||||
memory: 640Mi
|
||||
cpu: 100m
|
||||
limits:
|
||||
memory: 900Mi
|
||||
---
|
||||
# Headless service. Not required for jdbc-ping discovery, which goes through the
|
||||
# database, but it gives each pod a stable DNS name for direct inspection.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: keycloak-headless
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
clusterIP: None
|
||||
selector:
|
||||
app: keycloak
|
||||
ports:
|
||||
- port: 8080
|
||||
targetPort: http
|
||||
name: http
|
||||
- port: 9000
|
||||
targetPort: management
|
||||
name: management
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: keycloak
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
selector:
|
||||
app: keycloak
|
||||
ports:
|
||||
- port: 8080
|
||||
targetPort: http
|
||||
name: http
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: keycloak
|
||||
namespace: keycloak-lab
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: auth.hyeonworks.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: keycloak
|
||||
port:
|
||||
number: 8080
|
||||
@@ -1,373 +0,0 @@
|
||||
# Prometheus + node-exporter + Grafana.
|
||||
#
|
||||
# Purpose: during a fault-injection experiment, know *which signal moved first*.
|
||||
# Without a metrics store the only record is whatever scrolled past in a terminal,
|
||||
# and "the cluster recovered in about a minute" is not a measurement.
|
||||
#
|
||||
# kubectl apply -f deploy/lab/k8s/observability.yaml
|
||||
# kubectl -n observability rollout status deployment/prometheus --timeout=300s
|
||||
#
|
||||
# Placement decision — Prometheus and Grafana are pinned to the control-plane
|
||||
# node (kc-lab-1). An observability stack must not share a failure domain with
|
||||
# the thing it observes. With only two nodes that cannot be fully avoided, so the
|
||||
# rule here is: the node that gets killed in experiments is the *agent*
|
||||
# (kc-lab-2, holding keycloak-0 and postgres), and everything needed to watch
|
||||
# that happen lives on the server node.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: observability
|
||||
---
|
||||
# Prometheus discovers scrape targets by querying the Kubernetes API, so it
|
||||
# needs read access to nodes, services, endpoints and pods. Without this the
|
||||
# kubernetes_sd_configs below silently return no targets.
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: prometheus
|
||||
namespace: observability
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: prometheus
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
# nodes/proxy is required in addition to nodes/metrics: the kubelet job
|
||||
# reaches each node through the API server's proxy subresource
|
||||
# (/api/v1/nodes/<name>/proxy/metrics). Without it every kubelet target
|
||||
# fails with 403 Forbidden while the other jobs stay green — a partial
|
||||
# failure that is easy to miss unless the target list is checked.
|
||||
resources: [nodes, nodes/metrics, nodes/proxy, services, endpoints, pods]
|
||||
verbs: [get, list, watch]
|
||||
- nonResourceURLs: ["/metrics"]
|
||||
verbs: [get]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: prometheus
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: prometheus
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: prometheus
|
||||
namespace: observability
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: prometheus-config
|
||||
namespace: observability
|
||||
data:
|
||||
prometheus.yml: |
|
||||
global:
|
||||
# 15s is short for production but right here: a node loss should show up
|
||||
# within a couple of samples, not a minute later.
|
||||
scrape_interval: 15s
|
||||
evaluation_interval: 15s
|
||||
|
||||
scrape_configs:
|
||||
# Prometheus scraping itself. Useful as a control: if this target is down,
|
||||
# the problem is Prometheus, not the thing being measured.
|
||||
- job_name: prometheus
|
||||
static_configs:
|
||||
- targets: ['localhost:9090']
|
||||
|
||||
# Keycloak. Metrics live on the management port 9000, not 8080 — the same
|
||||
# split that the health probes use. KC_METRICS_ENABLED=true is already set
|
||||
# on the StatefulSet.
|
||||
#
|
||||
# Discovery is by endpoints rather than a static list because pod IPs
|
||||
# change on every restart; that was observed directly when the lab was
|
||||
# power-cycled and every pod came back with a new address.
|
||||
- job_name: keycloak
|
||||
kubernetes_sd_configs:
|
||||
- role: endpoints
|
||||
namespaces:
|
||||
names: [keycloak-lab]
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
|
||||
action: keep
|
||||
regex: keycloak-headless;management
|
||||
- source_labels: [__meta_kubernetes_pod_name]
|
||||
target_label: pod
|
||||
- source_labels: [__meta_kubernetes_pod_node_name]
|
||||
target_label: node
|
||||
|
||||
# node-exporter, one per node via DaemonSet. This is what answers
|
||||
# "did the machine die or did the process die".
|
||||
- job_name: node-exporter
|
||||
kubernetes_sd_configs:
|
||||
- role: endpoints
|
||||
namespaces:
|
||||
names: [observability]
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_service_name]
|
||||
action: keep
|
||||
regex: node-exporter
|
||||
- source_labels: [__meta_kubernetes_pod_node_name]
|
||||
target_label: node
|
||||
|
||||
# The kubelet's own metrics, reached through the API server proxy so no
|
||||
# extra port needs opening.
|
||||
- job_name: kubelet
|
||||
scheme: https
|
||||
tls_config:
|
||||
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
||||
insecure_skip_verify: true
|
||||
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
||||
kubernetes_sd_configs:
|
||||
- role: node
|
||||
relabel_configs:
|
||||
- action: labelmap
|
||||
regex: __meta_kubernetes_node_label_(.+)
|
||||
- target_label: __address__
|
||||
replacement: kubernetes.default.svc:443
|
||||
- source_labels: [__meta_kubernetes_node_name]
|
||||
regex: (.+)
|
||||
target_label: __metrics_path__
|
||||
replacement: /api/v1/nodes/${1}/proxy/metrics
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: prometheus-data
|
||||
namespace: observability
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: local-path
|
||||
resources:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: prometheus
|
||||
namespace: observability
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate # RWO volume; two pods cannot mount it at once
|
||||
selector:
|
||||
matchLabels:
|
||||
app: prometheus
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: prometheus
|
||||
spec:
|
||||
serviceAccountName: prometheus
|
||||
# See the placement note at the top of this file.
|
||||
nodeSelector:
|
||||
node-role.kubernetes.io/control-plane: "true"
|
||||
securityContext:
|
||||
fsGroup: 65534 # the image runs as nobody and must own the volume
|
||||
containers:
|
||||
- name: prometheus
|
||||
image: prom/prometheus:v3.1.0
|
||||
args:
|
||||
- --config.file=/etc/prometheus/prometheus.yml
|
||||
- --storage.tsdb.path=/prometheus
|
||||
# 7 days is far more than an experiment needs and keeps the volume
|
||||
# small enough that it never becomes the reason a node fills up.
|
||||
- --storage.tsdb.retention.time=7d
|
||||
- --web.enable-lifecycle
|
||||
ports:
|
||||
- containerPort: 9090
|
||||
name: http
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/prometheus
|
||||
- name: data
|
||||
mountPath: /prometheus
|
||||
readinessProbe:
|
||||
httpGet: { path: /-/ready, port: http }
|
||||
initialDelaySeconds: 10
|
||||
livenessProbe:
|
||||
httpGet: { path: /-/healthy, port: http }
|
||||
initialDelaySeconds: 30
|
||||
resources:
|
||||
requests: { memory: 256Mi, cpu: 50m }
|
||||
limits: { memory: 640Mi }
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: prometheus-config
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: prometheus-data
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: prometheus
|
||||
namespace: observability
|
||||
spec:
|
||||
selector:
|
||||
app: prometheus
|
||||
ports:
|
||||
- port: 9090
|
||||
targetPort: http
|
||||
---
|
||||
# node-exporter. A DaemonSet so every node reports, including one that is about
|
||||
# to be killed — the last samples before it goes silent are the interesting part.
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: node-exporter
|
||||
namespace: observability
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: node-exporter
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: node-exporter
|
||||
spec:
|
||||
# Host namespaces: the point is to measure the machine, not the container.
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
tolerations:
|
||||
- operator: Exists # must also run on tainted nodes
|
||||
containers:
|
||||
- name: node-exporter
|
||||
image: prom/node-exporter:v1.8.2
|
||||
args:
|
||||
- --path.procfs=/host/proc
|
||||
- --path.sysfs=/host/sys
|
||||
- --path.rootfs=/host/root
|
||||
- --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
|
||||
ports:
|
||||
- containerPort: 9100
|
||||
name: metrics
|
||||
hostPort: 9100
|
||||
volumeMounts:
|
||||
- { name: proc, mountPath: /host/proc, readOnly: true }
|
||||
- { name: sys, mountPath: /host/sys, readOnly: true }
|
||||
- { name: rootfs, mountPath: /host/root, readOnly: true, mountPropagation: HostToContainer }
|
||||
resources:
|
||||
requests: { memory: 32Mi, cpu: 20m }
|
||||
limits: { memory: 96Mi }
|
||||
volumes:
|
||||
- { name: proc, hostPath: { path: /proc } }
|
||||
- { name: sys, hostPath: { path: /sys } }
|
||||
- { name: rootfs, hostPath: { path: / } }
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: node-exporter
|
||||
namespace: observability
|
||||
spec:
|
||||
clusterIP: None # headless: Prometheus wants each pod, not a VIP
|
||||
selector:
|
||||
app: node-exporter
|
||||
ports:
|
||||
- port: 9100
|
||||
targetPort: metrics
|
||||
name: metrics
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: grafana
|
||||
namespace: observability
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: grafana
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: grafana
|
||||
spec:
|
||||
nodeSelector:
|
||||
node-role.kubernetes.io/control-plane: "true"
|
||||
containers:
|
||||
- name: grafana
|
||||
image: grafana/grafana:11.4.0
|
||||
ports:
|
||||
- containerPort: 3000
|
||||
name: http
|
||||
env:
|
||||
- name: GF_SECURITY_ADMIN_USER
|
||||
value: admin
|
||||
- name: GF_SECURITY_ADMIN_PASSWORD
|
||||
value: lab-grafana-change-me
|
||||
# Grafana builds absolute URLs for redirects and asset paths. Behind
|
||||
# the nginx -> Traefik chain it must be told the external address,
|
||||
# for exactly the reason Keycloak needs KC_HOSTNAME. Without it,
|
||||
# login redirects come back as http://<pod-ip>:3000.
|
||||
- name: GF_SERVER_ROOT_URL
|
||||
value: https://app2.hyeonworks.com
|
||||
volumeMounts:
|
||||
- name: datasources
|
||||
mountPath: /etc/grafana/provisioning/datasources
|
||||
readinessProbe:
|
||||
httpGet: { path: /api/health, port: http }
|
||||
initialDelaySeconds: 15
|
||||
resources:
|
||||
requests: { memory: 128Mi, cpu: 50m }
|
||||
limits: { memory: 320Mi }
|
||||
volumes:
|
||||
- name: datasources
|
||||
configMap:
|
||||
name: grafana-datasources
|
||||
---
|
||||
# Provisioning the datasource as a file means Grafana comes up already wired to
|
||||
# Prometheus. Clicking through the UI would leave the configuration only in
|
||||
# Grafana's own database, which is emptyDir here and disappears on restart.
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: grafana-datasources
|
||||
namespace: observability
|
||||
data:
|
||||
prometheus.yaml: |
|
||||
apiVersion: 1
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://prometheus.observability.svc:9090
|
||||
isDefault: true
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: grafana
|
||||
namespace: observability
|
||||
spec:
|
||||
selector:
|
||||
app: grafana
|
||||
ports:
|
||||
- port: 3000
|
||||
targetPort: http
|
||||
---
|
||||
# Grafana is published on app2.hyeonworks.com because that name is already in
|
||||
# the wildcard-free certificate (auth / app1 / app2) and is otherwise unused.
|
||||
# It moves when app2 is needed for the SSO experiment.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: grafana
|
||||
namespace: observability
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: app2.hyeonworks.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: grafana
|
||||
port:
|
||||
number: 3000
|
||||
@@ -1,43 +0,0 @@
|
||||
# Make Traefik trust the X-Forwarded-* headers that the host nginx sets.
|
||||
#
|
||||
# Without this, Traefik rewrites every forwarded header from its own connection,
|
||||
# which is plain HTTP on port 80. The application then sees scheme=http even
|
||||
# though the browser connected over TLS. See docs/two-hop-proxy-header-contract.md.
|
||||
#
|
||||
# k3s installs Traefik through its bundled HelmChart, so values are overridden
|
||||
# with a HelmChartConfig rather than by editing the deployment. k3s reconciles
|
||||
# the chart and recreates the Traefik pod.
|
||||
#
|
||||
# kubectl apply -f deploy/lab/k8s/traefik-forwarded-headers.yaml
|
||||
# kubectl -n kube-system rollout status deploy/traefik --timeout=180s
|
||||
apiVersion: helm.cattle.io/v1
|
||||
kind: HelmChartConfig
|
||||
metadata:
|
||||
name: traefik
|
||||
namespace: kube-system
|
||||
spec:
|
||||
valuesContent: |-
|
||||
ports:
|
||||
web:
|
||||
forwardedHeaders:
|
||||
# Requests arriving from these sources keep their existing
|
||||
# X-Forwarded-* values instead of having them rewritten.
|
||||
#
|
||||
# 10.42.0.0/16 is the pod CIDR. It is required because the traefik
|
||||
# Service uses externalTrafficPolicy: Cluster, so svclb SNATs the
|
||||
# traffic and Traefik sees a pod-network address rather than the
|
||||
# host nginx address.
|
||||
#
|
||||
# The node/host range is deliberately absent. Because svclb SNATs,
|
||||
# the host nginx address never reaches Traefik — measured, not assumed.
|
||||
# Trusting a range that cannot appear only widens the surface.
|
||||
#
|
||||
# Trusting the whole pod CIDR still means any pod in the cluster could
|
||||
# forge these headers, which is why echo-network-policy.yaml restricts
|
||||
# who may reach the application at all.
|
||||
trustedIPs:
|
||||
- 10.42.0.0/16
|
||||
websecure:
|
||||
forwardedHeaders:
|
||||
trustedIPs:
|
||||
- 10.42.0.0/16
|
||||
@@ -1,42 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build the API image on this workstation and import it into each lab node's
|
||||
# containerd.
|
||||
#
|
||||
# k3s does not run Docker and the lab has no registry, so images are shipped as
|
||||
# a stream: docker save -> ssh through the lab host -> k3s ctr images import.
|
||||
# Every node needs its own copy because the scheduler may place the pod anywhere.
|
||||
#
|
||||
# ./deploy/lab/scripts/build-and-import.sh
|
||||
# IMAGE=keycloak-pattern-api:lab NODES="kc-lab-1" ./deploy/lab/scripts/build-and-import.sh
|
||||
set -euo pipefail
|
||||
|
||||
IMAGE="${IMAGE:-keycloak-pattern-api:lab}"
|
||||
NODES="${NODES:-kc-lab-1 kc-lab-2}"
|
||||
LAB_HOST="${LAB_HOST:-test-server}"
|
||||
CONTEXT="${CONTEXT:-backend}"
|
||||
|
||||
repo_root="$(git rev-parse --show-toplevel)"
|
||||
cd "$repo_root"
|
||||
|
||||
echo "==> building ${IMAGE} from ${CONTEXT}/"
|
||||
docker build -t "$IMAGE" "$CONTEXT"
|
||||
|
||||
for node in $NODES; do
|
||||
echo "==> importing into ${node}"
|
||||
# Nested ssh: the workstation cannot reach the guests directly because they
|
||||
# sit behind the lab host's libvirt NAT. The lab host's ~/.ssh/config holds
|
||||
# the kc-lab-* aliases.
|
||||
docker save "$IMAGE" \
|
||||
| ssh "$LAB_HOST" "ssh ${node} 'sudo k3s ctr images import -'"
|
||||
done
|
||||
|
||||
echo "==> verifying"
|
||||
for node in $NODES; do
|
||||
printf ' %-10s ' "$node"
|
||||
ssh "$LAB_HOST" "ssh ${node} 'sudo k3s ctr images ls -q'" \
|
||||
| grep -c "$IMAGE" \
|
||||
| xargs -I{} echo "{} match(es)"
|
||||
done
|
||||
|
||||
echo
|
||||
echo "next: kubectl rollout restart -n header-lab deployment/echo"
|
||||
@@ -1,55 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Experiment 0c — where does a session entry actually live?
|
||||
#
|
||||
# Experiment 0b showed keycloak-1's session cache never moved when keycloak-0
|
||||
# handled a login. That leaves two explanations:
|
||||
#
|
||||
# (a) a DISTRIBUTED cache with owners=1 — entries are spread across nodes by
|
||||
# consistent hashing, and this one happened to land on keycloak-0;
|
||||
# (b) a LOCAL cache — each node only ever caches what it handled itself.
|
||||
#
|
||||
# They are distinguished by driving logins at the OTHER node. Under (a) the
|
||||
# entries would keep landing on both nodes regardless of who was asked. Under
|
||||
# (b) the count rises only on the node that received the request.
|
||||
set -uo pipefail
|
||||
|
||||
NS="${NS:-keycloak-lab}"
|
||||
N="${N:-5}"
|
||||
K0_IP=$(kubectl -n "$NS" get pod keycloak-0 -o jsonpath='{.status.podIP}')
|
||||
K1_IP=$(kubectl -n "$NS" get pod keycloak-1 -o jsonpath='{.status.podIP}')
|
||||
ADMIN_PW=$(kubectl -n "$NS" get secret keycloak-lab-secrets \
|
||||
-o jsonpath='{.data.KC_BOOTSTRAP_ADMIN_PASSWORD}' | base64 -d)
|
||||
|
||||
echo "수집 시각: $(date '+%Y-%m-%d %H:%M:%S %Z')"
|
||||
echo " keycloak-0 = $K0_IP ($(kubectl -n "$NS" get pod keycloak-0 -o jsonpath='{.spec.nodeName}'))"
|
||||
echo " keycloak-1 = $K1_IP ($(kubectl -n "$NS" get pod keycloak-1 -o jsonpath='{.spec.nodeName}'))"
|
||||
echo
|
||||
|
||||
kubectl -n "$NS" run kc-own --rm -i --restart=Never \
|
||||
--image=curlimages/curl:8.11.1 --quiet --command -- sh -c "
|
||||
O=/tmp/o; : > \$O
|
||||
ent() {
|
||||
curl -s --retry 3 --max-time 20 http://\$1:9000/metrics \
|
||||
| grep -E '^vendor_statistics_approximate_entries_unique.cache=.sessions' \
|
||||
| awk '{print \$NF}'
|
||||
}
|
||||
login() { i=0; while [ \$i -lt $N ]; do
|
||||
curl -s -o /dev/null -X POST http://\$1:8080/realms/master/protocol/openid-connect/token \
|
||||
-d grant_type=password -d client_id=admin-cli \
|
||||
-d username=admin -d 'password=$ADMIN_PW'
|
||||
i=\$((i+1)); done; sleep 5; }
|
||||
{
|
||||
printf '%-32s %12s %12s\n' '단계' 'k0 entries' 'k1 entries'
|
||||
printf '%-32s %12s %12s\n' '시작' \"\$(ent $K0_IP)\" \"\$(ent $K1_IP)\"
|
||||
login $K1_IP
|
||||
printf '%-32s %12s %12s\n' 'keycloak-1 에 로그인 ${N}회' \"\$(ent $K0_IP)\" \"\$(ent $K1_IP)\"
|
||||
login $K0_IP
|
||||
printf '%-32s %12s %12s\n' 'keycloak-0 에 로그인 ${N}회' \"\$(ent $K0_IP)\" \"\$(ent $K1_IP)\"
|
||||
} >> \$O
|
||||
cat \$O
|
||||
" 2>&1 | grep -v '^pod .* deleted$'
|
||||
|
||||
echo
|
||||
echo "=== 대조: PostgreSQL 에는 몇 건인가 ==="
|
||||
kubectl -n "$NS" exec deploy/postgres -- psql -U keycloak -d keycloak -tAc \
|
||||
"select count(*) from offline_user_session where offline_flag='0'" 2>/dev/null | sed 's/^/ online 세션 /'
|
||||
@@ -1,83 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Experiment 0b — does the Infinispan cache itself replicate, or do both nodes
|
||||
# merely agree because they read the same database?
|
||||
#
|
||||
# Experiment 0 proved the two nodes give the same answers. That alone does NOT
|
||||
# prove Infinispan replicated anything: with persistent-user-sessions (the
|
||||
# Keycloak 26 default) the session is written to PostgreSQL, so two nodes reading
|
||||
# one database would agree even with the cache disabled entirely.
|
||||
#
|
||||
# This script separates the two by measuring the cache counters on BOTH nodes
|
||||
# around a single login. If the write on keycloak-0 shows up as cache activity
|
||||
# on keycloak-1, the replication is real and not a database artifact.
|
||||
set -uo pipefail
|
||||
|
||||
NS="${NS:-keycloak-lab}"
|
||||
K0_IP=$(kubectl -n "$NS" get pod keycloak-0 -o jsonpath='{.status.podIP}')
|
||||
K1_IP=$(kubectl -n "$NS" get pod keycloak-1 -o jsonpath='{.status.podIP}')
|
||||
ADMIN_PW=$(kubectl -n "$NS" get secret keycloak-lab-secrets \
|
||||
-o jsonpath='{.data.KC_BOOTSTRAP_ADMIN_PASSWORD}' | base64 -d)
|
||||
|
||||
echo "수집 시각: $(date '+%Y-%m-%d %H:%M:%S %Z')"
|
||||
echo
|
||||
|
||||
# 파드 출력을 스트리밍으로 받으면 조각이 유실된다. 실제로 첫 시도에서
|
||||
# keycloak-1 의 스냅샷과 그 다음 마커가 통째로 사라져 델타가 0 으로 보였다.
|
||||
# 파드 안에서 파일로 모았다가 마지막에 한 번만 내보낸다.
|
||||
kubectl -n "$NS" run kc-delta --rm -i --restart=Never \
|
||||
--image=curlimages/curl:8.11.1 --quiet --command -- sh -c "
|
||||
set -u
|
||||
K0='http://$K0_IP'; K1='http://$K1_IP'
|
||||
O=/tmp/o.txt; : > \$O
|
||||
snap() {
|
||||
curl -s --retry 3 --retry-connrefused --max-time 20 \$1:9000/metrics \
|
||||
| grep -E '^vendor_(statistics_(stores|hits|misses|approximate_entries_unique)|rpc_manager_replication_count)\{cache=\"(sessions|clientSessions)\"' \
|
||||
| sed 's/,cache_manager=\"keycloak\"//; s/,node=\"[^\"]*\"//' >> \$O
|
||||
}
|
||||
echo '###BEFORE_K0' >> \$O; snap \$K0
|
||||
echo '###BEFORE_K1' >> \$O; snap \$K1
|
||||
echo '###LOGIN' >> \$O
|
||||
curl -s -o /dev/null -w 'http_code=%{http_code}\n' -X POST \
|
||||
\"\$K0:8080/realms/master/protocol/openid-connect/token\" \
|
||||
-d grant_type=password -d client_id=admin-cli \
|
||||
-d username=admin -d 'password=$ADMIN_PW' >> \$O
|
||||
sleep 5
|
||||
echo '###AFTER_K0' >> \$O; snap \$K0
|
||||
echo '###AFTER_K1' >> \$O; snap \$K1
|
||||
echo '###END' >> \$O
|
||||
cat \$O
|
||||
" 2>&1 | grep -v '^pod .* deleted$' > /tmp/cache-delta.txt
|
||||
|
||||
python3 - /tmp/cache-delta.txt <<'PY'
|
||||
import re, sys
|
||||
raw = open(sys.argv[1]).read()
|
||||
blocks, cur = {}, None
|
||||
for line in raw.splitlines():
|
||||
if line.startswith('###'):
|
||||
cur = line[3:]; blocks[cur] = {}
|
||||
elif cur and '{' in line:
|
||||
m = re.match(r'(\S+?)\{cache="(\w+)"\}\s+(\S+)', line)
|
||||
if m:
|
||||
blocks[cur][(m.group(1), m.group(2))] = float(m.group(3))
|
||||
|
||||
print('=== 로그인은 keycloak-0 에만 보냈다 ===')
|
||||
code = [l for l in raw.splitlines() if l.startswith('http_code=')]
|
||||
print(' 로그인 응답: ' + (code[0] if code else '없음'))
|
||||
for n in ('BEFORE_K0','BEFORE_K1','AFTER_K0','AFTER_K1'):
|
||||
if not blocks.get(n):
|
||||
print(f' !! {n} 스냅샷이 비었다 — 델타를 신뢰할 수 없다')
|
||||
print()
|
||||
hdr = f" {'계수기':<42} {'캐시':<15} {'전':>8} {'후':>8} {'증가':>7}"
|
||||
for node in ('K0', 'K1'):
|
||||
who = 'keycloak-0 (로그인을 받은 노드)' if node == 'K0' else 'keycloak-1 (아무 요청도 받지 않은 노드)'
|
||||
print(f'=== {who} ===')
|
||||
print(hdr)
|
||||
b, a = blocks.get(f'BEFORE_{node}', {}), blocks.get(f'AFTER_{node}', {})
|
||||
for k in sorted(set(b) | set(a)):
|
||||
before, after = b.get(k[0:2], 0.0), a.get(k[0:2], 0.0)
|
||||
d = after - before
|
||||
mark = ' ←' if d else ''
|
||||
name = k[0].replace('vendor_statistics_', '').replace('vendor_rpc_manager_', 'rpc.')
|
||||
print(f" {name:<42} {k[1]:<15} {before:>8.0f} {after:>8.0f} {d:>+7.0f}{mark}")
|
||||
print()
|
||||
PY
|
||||
@@ -1,108 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Experiment 0d — capture the actual SQL that the OTHER node runs.
|
||||
#
|
||||
# Experiments 0b/0c showed that session entries never appear in keycloak-1's
|
||||
# memory, yet keycloak-1 can use a session keycloak-0 created. The conclusion
|
||||
# "keycloak-1 reads it from PostgreSQL" was an inference, not an observation.
|
||||
#
|
||||
# This script turns on statement logging in PostgreSQL for a few seconds, sends
|
||||
# ONE refresh request to keycloak-1 for a session born on keycloak-0, and greps
|
||||
# the database log for that session id. If the inference is right, the SQL is
|
||||
# there, issued from keycloak-1's pod IP.
|
||||
#
|
||||
# It also checks whether serving that request makes keycloak-1 cache the session
|
||||
# — which sharpens "each node caches what it handled" from "what it logged in"
|
||||
# to "what it touched".
|
||||
set -uo pipefail
|
||||
|
||||
NS="${NS:-keycloak-lab}"
|
||||
PSQL="kubectl -n $NS exec deploy/postgres -- psql -U keycloak -d keycloak -tAc"
|
||||
|
||||
K0_IP=$(kubectl -n "$NS" get pod keycloak-0 -o jsonpath='{.status.podIP}')
|
||||
K1_IP=$(kubectl -n "$NS" get pod keycloak-1 -o jsonpath='{.status.podIP}')
|
||||
ADMIN_PW=$(kubectl -n "$NS" get secret keycloak-lab-secrets \
|
||||
-o jsonpath='{.data.KC_BOOTSTRAP_ADMIN_PASSWORD}' | base64 -d)
|
||||
|
||||
echo "수집 시각: $(date '+%Y-%m-%d %H:%M:%S %Z')"
|
||||
echo " keycloak-0 = $K0_IP (세션을 만드는 노드)"
|
||||
echo " keycloak-1 = $K1_IP (읽기만 하는 노드)"
|
||||
echo
|
||||
|
||||
# %h 를 넣어야 어느 파드가 보낸 질의인지 로그에서 구분된다.
|
||||
echo "=== PostgreSQL 문장 로깅을 켠다 ==="
|
||||
$PSQL "alter system set log_statement='all'" >/dev/null 2>&1
|
||||
$PSQL "alter system set log_line_prefix='%m [%p] %h '" >/dev/null 2>&1
|
||||
$PSQL "select pg_reload_conf()" >/dev/null 2>&1
|
||||
echo " log_statement = $($PSQL 'show log_statement' 2>/dev/null)"
|
||||
echo " log_line_prefix = $($PSQL 'show log_line_prefix' 2>/dev/null)"
|
||||
echo
|
||||
|
||||
# 로그 커서를 잡아둔다. 이 줄 수 이후만 본다.
|
||||
LOG_BEFORE=$(kubectl -n "$NS" logs deploy/postgres --tail=-1 2>/dev/null | wc -l)
|
||||
|
||||
RESULT=$(kubectl -n "$NS" run kc-readpath --rm -i --restart=Never \
|
||||
--image=curlimages/curl:8.11.1 --quiet --command -- sh -c "
|
||||
O=/tmp/o; : > \$O
|
||||
TOKEN_EP='/realms/master/protocol/openid-connect/token'
|
||||
jget() { sed -n \"s/.*\\\"\$1\\\":\\\"\\([^\\\"]*\\)\\\".*/\\1/p\"; }
|
||||
ent() {
|
||||
curl -s --retry 3 --max-time 20 http://\$1:9000/metrics \
|
||||
| grep -E '^vendor_statistics_approximate_entries_unique.cache=.sessions' | awk '{print \$NF}'
|
||||
}
|
||||
# keycloak-0 에서 로그인한다
|
||||
L=\$(curl -s -X POST \"http://$K0_IP:8080\$TOKEN_EP\" -d grant_type=password \
|
||||
-d client_id=admin-cli -d username=admin -d 'password=$ADMIN_PW')
|
||||
SID=\$(echo \"\$L\" | jget access_token | cut -d. -f2 | sed 's/\$/==/' | base64 -d 2>/dev/null | jget sid)
|
||||
RT=\$(echo \"\$L\" | jget refresh_token)
|
||||
echo \"SID=\$SID\" >> \$O
|
||||
echo \"K1_ENTRIES_BEFORE=\$(ent $K1_IP)\" >> \$O
|
||||
sleep 2
|
||||
# 반대편 노드에 refresh 를 딱 한 번 보낸다
|
||||
# 인용을 한 겹 더 쌓으면 curl 이 URL 을 통째로 못 읽는다. 실제로 000 이 나왔다.
|
||||
CODE=\$(curl -s -o /dev/null -w '%{http_code}' -X POST \
|
||||
\"http://$K1_IP:8080\$TOKEN_EP\" \
|
||||
-d grant_type=refresh_token -d client_id=admin-cli -d \"refresh_token=\$RT\")
|
||||
echo \"REFRESH_ON_K1=\$CODE\" >> \$O
|
||||
sleep 3
|
||||
echo \"K1_ENTRIES_AFTER=\$(ent $K1_IP)\" >> \$O
|
||||
cat \$O
|
||||
" 2>&1 | grep -v '^pod .* deleted$')
|
||||
|
||||
echo "=== 요청 ==="
|
||||
echo "$RESULT" | sed 's/^/ /'
|
||||
SID=$(echo "$RESULT" | sed -n 's/^SID=//p')
|
||||
|
||||
echo
|
||||
echo "=== PostgreSQL 문장 로깅을 끈다 ==="
|
||||
$PSQL "alter system reset log_statement" >/dev/null 2>&1
|
||||
$PSQL "alter system reset log_line_prefix" >/dev/null 2>&1
|
||||
$PSQL "select pg_reload_conf()" >/dev/null 2>&1
|
||||
echo " log_statement = $($PSQL 'show log_statement' 2>/dev/null)"
|
||||
|
||||
echo
|
||||
echo "=== keycloak-1 이 실제로 보낸 SQL 문장 ==="
|
||||
echo " (파라미터가 \$1 로 묶여 있어, sid 는 바로 아래 DETAIL 줄에 있다)"
|
||||
echo
|
||||
kubectl -n "$NS" logs deploy/postgres --tail=-1 2>/dev/null \
|
||||
| tail -n +$((LOG_BEFORE + 1)) \
|
||||
| grep -F "$K1_IP" | grep -E "LOG: execute" \
|
||||
| sed 's/.*execute [^:]*: //' | sed 's/^/ /' | head -12
|
||||
echo
|
||||
echo "=== 그 sid 를 언급한 SQL — 누가 보냈는가 ==="
|
||||
echo " 찾는 sid: $SID"
|
||||
echo
|
||||
kubectl -n "$NS" logs deploy/postgres --tail=-1 2>/dev/null \
|
||||
| tail -n +$((LOG_BEFORE + 1)) \
|
||||
| grep -F "$SID" \
|
||||
| sed -e "s/$K0_IP/[keycloak-0]/g" -e "s/$K1_IP/[keycloak-1]/g" \
|
||||
| cut -c1-220 \
|
||||
| head -20
|
||||
|
||||
echo
|
||||
echo "=== 요약: 파드별 질의 건수 ==="
|
||||
kubectl -n "$NS" logs deploy/postgres --tail=-1 2>/dev/null \
|
||||
| tail -n +$((LOG_BEFORE + 1)) \
|
||||
| grep -F "$SID" \
|
||||
| grep -oE "^[0-9-]+ [0-9:.]+ [A-Z]+ \[[0-9]+\] [0-9.]+" \
|
||||
| awk '{print $NF}' | sort | uniq -c \
|
||||
| sed -e "s/$K0_IP/[keycloak-0]/" -e "s/$K1_IP/[keycloak-1]/" -e 's/^/ /'
|
||||
@@ -1,207 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Experiment 0 — is a session created on one Keycloak node usable on the other?
|
||||
#
|
||||
# Forming a cluster is not the same as sharing session state. The Infinispan log
|
||||
# says "cluster view (2)", but that only proves the members found each other.
|
||||
#
|
||||
# Design notes, learned the hard way:
|
||||
#
|
||||
# * Every probe has a CONTROL. A result from the far node means nothing unless
|
||||
# the same call against the issuing node is also measured. The first version
|
||||
# of this script reported "403 on keycloak-1" as if it were a replication
|
||||
# failure; the issuing node returned 403 too, and the cause was a missing
|
||||
# openid scope. Measure both, always.
|
||||
#
|
||||
# * Sessions are tracked by SID, not by count. Both the test login and the
|
||||
# admin API calls create sessions for the same user, so counts are noisy.
|
||||
# A specific session id either appears in a node's answer or it does not.
|
||||
#
|
||||
# * The probe is the REFRESH TOKEN grant, not userinfo. userinfo only validates
|
||||
# a signature and can succeed on a node that knows nothing about the session.
|
||||
# Refreshing requires the node to find the session, check it is alive, and
|
||||
# write back a new refresh time — it actually touches the session store.
|
||||
#
|
||||
# Talks to pod IPs directly: going through nginx/Traefik would hide which node
|
||||
# handled each request, which is the entire question.
|
||||
#
|
||||
# ./deploy/lab/scripts/experiment-session-replication.sh
|
||||
set -uo pipefail
|
||||
|
||||
NS="${NS:-keycloak-lab}"
|
||||
OUT="${OUT:-/tmp/session-replication}"
|
||||
mkdir -p "$OUT"
|
||||
|
||||
PSQL="kubectl -n $NS exec deploy/postgres -- psql -U keycloak -d keycloak -tAc"
|
||||
|
||||
echo "수집 시각: $(date '+%Y-%m-%d %H:%M:%S %Z')"
|
||||
echo
|
||||
|
||||
K0_IP=$(kubectl -n "$NS" get pod keycloak-0 -o jsonpath='{.status.podIP}')
|
||||
K1_IP=$(kubectl -n "$NS" get pod keycloak-1 -o jsonpath='{.status.podIP}')
|
||||
K0_NODE=$(kubectl -n "$NS" get pod keycloak-0 -o jsonpath='{.spec.nodeName}')
|
||||
K1_NODE=$(kubectl -n "$NS" get pod keycloak-1 -o jsonpath='{.spec.nodeName}')
|
||||
ADMIN_PW=$(kubectl -n "$NS" get secret keycloak-lab-secrets \
|
||||
-o jsonpath='{.data.KC_BOOTSTRAP_ADMIN_PASSWORD}' | base64 -d)
|
||||
|
||||
echo "=== 대상 ==="
|
||||
printf ' keycloak-0 %-14s %s\n' "$K0_IP" "$K0_NODE"
|
||||
printf ' keycloak-1 %-14s %s\n' "$K1_IP" "$K1_NODE"
|
||||
echo
|
||||
|
||||
echo "=== [0] 실험 전 DB 세션 ==="
|
||||
$PSQL "select offline_flag, count(*) from offline_user_session group by offline_flag" 2>/dev/null \
|
||||
| sed 's/^/ offline_flag=/' || echo " (없음)"
|
||||
echo
|
||||
|
||||
# 파드 하나 안에서 전 단계를 실행한다. 단계마다 파드를 새로 띄우면 토큰을
|
||||
# 단계 사이로 넘길 수 없다.
|
||||
kubectl -n "$NS" run kc-probe --rm -i --restart=Never \
|
||||
--image=curlimages/curl:8.11.1 --quiet --command -- sh -c "
|
||||
set -u
|
||||
K0='http://$K0_IP:8080'; K1='http://$K1_IP:8080'
|
||||
TOKEN_EP='/realms/master/protocol/openid-connect/token'
|
||||
jget() { sed -n \"s/.*\\\"\$1\\\":\\\"\\([^\\\"]*\\)\\\".*/\\1/p\"; }
|
||||
|
||||
# ── [1] keycloak-0 에서 로그인. 이 노드가 세션의 출생지다 ──────────────────
|
||||
LOGIN=\$(curl -s -X POST \"\$K0\$TOKEN_EP\" \
|
||||
-d grant_type=password -d client_id=admin-cli \
|
||||
-d username=admin -d 'password=$ADMIN_PW')
|
||||
echo '###STEP1_LOGIN'; echo \"\$LOGIN\"
|
||||
|
||||
AT=\$(echo \"\$LOGIN\" | jget access_token)
|
||||
RT=\$(echo \"\$LOGIN\" | jget refresh_token)
|
||||
|
||||
# ── [2] 관리 API 조회용 토큰. 세션 오염을 피하려고 따로 하나만 더 만든다 ──
|
||||
ADMTOK=\$(curl -s -X POST \"\$K0\$TOKEN_EP\" \
|
||||
-d grant_type=password -d client_id=admin-cli \
|
||||
-d username=admin -d 'password=$ADMIN_PW' | jget access_token)
|
||||
CID=\$(curl -s -H \"Authorization: Bearer \$ADMTOK\" \
|
||||
\"\$K0/admin/realms/master/clients?clientId=admin-cli\" | jget id | head -1)
|
||||
|
||||
# ── [3] 두 노드에 같은 질문을 한다: admin-cli 의 세션 목록 ────────────────
|
||||
echo '###STEP3_SESSIONS_K0'
|
||||
curl -s -H \"Authorization: Bearer \$ADMTOK\" \
|
||||
\"\$K0/admin/realms/master/clients/\$CID/user-sessions?max=100\"
|
||||
echo
|
||||
echo '###STEP3_SESSIONS_K1'
|
||||
curl -s -H \"Authorization: Bearer \$ADMTOK\" \
|
||||
\"\$K1/admin/realms/master/clients/\$CID/user-sessions?max=100\"
|
||||
echo
|
||||
|
||||
# ── [4] 대조군: keycloak-0 이 발급한 refresh token 을 keycloak-0 에 쓴다 ──
|
||||
# 먼저 반대편에 써야 하므로 여기서는 쓰지 않고, 순서를 [5] 뒤로 미룬다.
|
||||
# refresh token 은 회전(rotation)되므로 한 번 쓰면 옛 것이 무효가 된다.
|
||||
# 따라서 '반대편 먼저'가 유일하게 의미 있는 순서다.
|
||||
|
||||
# ── [5] 시험군: keycloak-0 이 발급한 refresh token 을 keycloak-1 에 쓴다 ──
|
||||
echo '###STEP5_REFRESH_ON_K1'
|
||||
curl -s -w '\nhttp_code=%{http_code}\n' -X POST \"\$K1\$TOKEN_EP\" \
|
||||
-d grant_type=refresh_token -d client_id=admin-cli -d \"refresh_token=\$RT\"
|
||||
|
||||
RT2=\$(curl -s -X POST \"\$K1\$TOKEN_EP\" \
|
||||
-d grant_type=refresh_token -d client_id=admin-cli -d \"refresh_token=\$RT\" \
|
||||
| jget refresh_token)
|
||||
|
||||
# ── [6] 무효화가 반대 방향으로도 전파되는가 ───────────────────────────────
|
||||
# keycloak-1 에서 로그아웃시키고, keycloak-0 에서 갱신을 시도한다.
|
||||
echo '###STEP6_LOGOUT_VIA_K1'
|
||||
curl -s -o /dev/null -w 'http_code=%{http_code}\n' -X POST \"\$K1/realms/master/protocol/openid-connect/logout\" \
|
||||
-d client_id=admin-cli -d \"refresh_token=\$RT2\"
|
||||
|
||||
echo '###STEP7_REFRESH_ON_K0_AFTER_LOGOUT'
|
||||
curl -s -w '\nhttp_code=%{http_code}\n' -X POST \"\$K0\$TOKEN_EP\" \
|
||||
-d grant_type=refresh_token -d client_id=admin-cli -d \"refresh_token=\$RT2\"
|
||||
echo '###END'
|
||||
" > "$OUT/raw.txt" 2>&1
|
||||
|
||||
sed -i '/^pod .* deleted$/d' "$OUT/raw.txt"
|
||||
|
||||
python3 - "$OUT/raw.txt" <<'PY' | tee "$OUT/report.txt"
|
||||
import base64, json, sys
|
||||
|
||||
raw = open(sys.argv[1]).read()
|
||||
blocks, cur = {}, None
|
||||
for line in raw.splitlines():
|
||||
if line.startswith('###'):
|
||||
cur = line[3:]; blocks[cur] = []
|
||||
elif cur is not None:
|
||||
blocks[cur].append(line)
|
||||
get = lambda k: '\n'.join(blocks.get(k, [])).strip()
|
||||
|
||||
def j(s):
|
||||
try: return json.JSONDecoder().raw_decode(s.strip())[0]
|
||||
except Exception: return None
|
||||
|
||||
def claims(tok):
|
||||
p = tok.split('.')[1]; p += '=' * (-len(p) % 4)
|
||||
return json.loads(base64.urlsafe_b64decode(p))
|
||||
|
||||
login = j(get('STEP1_LOGIN'))
|
||||
if not login or 'access_token' not in login:
|
||||
print('로그인 실패:', get('STEP1_LOGIN')[:300]); sys.exit(1)
|
||||
|
||||
ac = claims(login['access_token'])
|
||||
rc = claims(login['refresh_token'])
|
||||
SID = ac['sid']
|
||||
print('=== [1] keycloak-0 에서 로그인 ===')
|
||||
print(f" sid {SID}")
|
||||
print(f" sub {ac.get('sub')}")
|
||||
print(f" iss {ac.get('iss')}")
|
||||
print(f" access 수명 {ac['exp']-ac['iat']}초")
|
||||
print(f" refresh 수명 {rc['exp']-rc['iat']}초 typ={rc.get('typ')}")
|
||||
print(f" refresh jti {rc.get('jti')}")
|
||||
|
||||
print()
|
||||
print('=== [3] 같은 sid 가 두 노드 모두에서 보이는가 ===')
|
||||
for step, who in (('STEP3_SESSIONS_K0', 'keycloak-0 (발급 노드)'),
|
||||
('STEP3_SESSIONS_K1', 'keycloak-1 (반대편)')):
|
||||
d = j(get(step))
|
||||
if d is None:
|
||||
print(f' {who:24} 파싱 실패: {get(step)[:120]}'); continue
|
||||
ids = [s.get('id') for s in d]
|
||||
mark = '보임 ✔' if SID in ids else '없음 ✘'
|
||||
print(f' {who:24} 세션 {len(ids)}개 중 대상 sid → {mark}')
|
||||
for s in d:
|
||||
if s.get('id') == SID:
|
||||
print(f" ipAddress={s.get('ipAddress')} start={s.get('start')} lastAccess={s.get('lastAccess')}")
|
||||
|
||||
def show(step, title, expect):
|
||||
print(); print(f'=== {title} ===')
|
||||
body = get(step)
|
||||
code = [l for l in body.splitlines() if l.startswith('http_code=')]
|
||||
code = code[0].split('=')[1] if code else '?'
|
||||
d = j(body)
|
||||
ok = '기대대로' if code == expect else f'기대({expect})와 다름'
|
||||
print(f' HTTP {code} ← {ok}')
|
||||
if d and 'access_token' in d:
|
||||
c = claims(d['access_token'])
|
||||
same = '동일 ✔' if c.get('sid') == SID else f"다름 ✘ ({c.get('sid')})"
|
||||
print(f' 새 토큰의 sid → {same}')
|
||||
elif d:
|
||||
print(f" error {d.get('error')}")
|
||||
print(f" error_description {d.get('error_description')}")
|
||||
|
||||
show('STEP5_REFRESH_ON_K1',
|
||||
'[5] keycloak-0 이 발급한 refresh token 을 keycloak-1 에 사용', '200')
|
||||
|
||||
print(); print('=== [6] keycloak-1 을 통해 로그아웃 ===')
|
||||
print(' ' + get('STEP6_LOGOUT_VIA_K1').strip())
|
||||
|
||||
show('STEP7_REFRESH_ON_K0_AFTER_LOGOUT',
|
||||
'[7] 로그아웃 후 keycloak-0 에서 갱신 시도 (무효화 전파)', '400')
|
||||
|
||||
open('/tmp/session-replication/sid.txt','w').write(SID)
|
||||
PY
|
||||
|
||||
SID=$(cat /tmp/session-replication/sid.txt 2>/dev/null)
|
||||
echo
|
||||
echo "=== [8] PostgreSQL 에서 그 sid 를 직접 확인 ==="
|
||||
echo " 대상 sid: $SID"
|
||||
$PSQL "select user_session_id, offline_flag, created_on, last_session_refresh
|
||||
from offline_user_session where user_session_id='$SID'" 2>/dev/null \
|
||||
| sed 's/^/ /' | grep -q . \
|
||||
&& $PSQL "select user_session_id||' | flag='||offline_flag||' | created='||created_on||' | refresh='||last_session_refresh
|
||||
from offline_user_session where user_session_id='$SID'" 2>/dev/null | sed 's/^/ /' \
|
||||
|| echo " 행 없음 — 로그아웃으로 삭제되었다"
|
||||
echo
|
||||
echo " 전체 세션 수: $($PSQL 'select count(*) from offline_user_session' 2>/dev/null)"
|
||||
@@ -1,42 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Measure what the nginx -> Traefik chain actually delivers to the application.
|
||||
#
|
||||
# docs/reverse-proxy-headers.md documents a single-hop nginx contract. The lab
|
||||
# runs two hops, so the forwarded headers are measured rather than assumed.
|
||||
# Run from anywhere that can resolve the lab hostnames.
|
||||
#
|
||||
# ./deploy/lab/scripts/measure-proxy-headers.sh
|
||||
set -euo pipefail
|
||||
|
||||
HOST="${HOST:-app1.hyeonworks.com}"
|
||||
URL="https://${HOST}/api/echo"
|
||||
|
||||
jqf() {
|
||||
if command -v jq >/dev/null 2>&1; then jq "$@"; else python3 -m json.tool; fi
|
||||
}
|
||||
|
||||
echo "=== 1. baseline: what the app sees for a normal request ==="
|
||||
curl -s "$URL" | jqf '{
|
||||
scheme, secure, serverName, serverPort, requestUrl, remoteAddr,
|
||||
forwarded: .headers | with_entries(select(.key | startswith("x-forwarded") or . == "x-real-ip" or . == "forwarded"))
|
||||
}' 2>/dev/null || curl -s "$URL"
|
||||
|
||||
echo
|
||||
echo "=== 2. spoof test: client sends its own X-Forwarded-* ==="
|
||||
echo " a trusted boundary must overwrite these, not append to them"
|
||||
curl -s "$URL" \
|
||||
-H 'X-Forwarded-For: 1.2.3.4' \
|
||||
-H 'X-Forwarded-Proto: http' \
|
||||
-H 'X-Forwarded-Host: evil.example.com' \
|
||||
-H 'X-Real-IP: 1.2.3.4' \
|
||||
| jqf '.headers | with_entries(select(.key | startswith("x-forwarded") or . == "x-real-ip"))' 2>/dev/null
|
||||
|
||||
echo
|
||||
echo "=== 3. which pod answered (host nginx upstream distribution) ==="
|
||||
for _ in 1 2 3 4; do
|
||||
curl -s "$URL" | jqf -r '.headers["x-forwarded-server"] // "n/a"' 2>/dev/null
|
||||
done
|
||||
|
||||
echo
|
||||
echo "=== 4. plain HTTP is redirected, not proxied ==="
|
||||
curl -s -o /dev/null -w ' http -> %{http_code} %{redirect_url}\n' "http://${HOST}/api/echo"
|
||||
@@ -1,47 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Rebuild a guest's cloud-init seed image and publish it into the libvirt pool.
|
||||
# Run on the lab host.
|
||||
#
|
||||
# ./rebuild-seed.sh 1
|
||||
#
|
||||
# The same content lives in three places: the source YAML, the ISO, and the
|
||||
# uploaded pool volume. Editing the YAML alone changes nothing, which is why
|
||||
# this is a script and not a set of remembered commands.
|
||||
#
|
||||
# A rebuilt seed only takes effect on a freshly created VM. cloud-init runs its
|
||||
# per-instance modules once per instance-id, so an existing guest ignores it.
|
||||
set -euo pipefail
|
||||
|
||||
N="${1:?usage: rebuild-seed.sh <1|2>}"
|
||||
CLOUD_DIR="${CLOUD_DIR:-$HOME/workspace/cloud}"
|
||||
POOL="${POOL:-default}"
|
||||
export LIBVIRT_DEFAULT_URI="${LIBVIRT_DEFAULT_URI:-qemu:///system}"
|
||||
|
||||
cd "$CLOUD_DIR"
|
||||
src="kc-lab-${N}.yaml"
|
||||
iso="seed-kc-lab-${N}.iso"
|
||||
meta="meta-kc-lab-${N}"
|
||||
|
||||
[ -f "$src" ] || { echo "missing $CLOUD_DIR/$src" >&2; exit 1; }
|
||||
|
||||
# A fresh instance-id makes cloud-init treat the guest as new and re-run the
|
||||
# per-instance modules.
|
||||
printf 'instance-id: kc-lab-%s-%s\nlocal-hostname: kc-lab-%s\n' \
|
||||
"$N" "$(date +%s)" "$N" > "$meta"
|
||||
|
||||
# NoCloud looks for a volume labelled cidata holding files named exactly
|
||||
# user-data and meta-data. -graft-points renames them inside the image so no
|
||||
# staging directory is needed.
|
||||
xorrisofs -quiet -output "$iso" -volid CIDATA -joliet -rock -graft-points \
|
||||
"/user-data=${src}" "/meta-data=${meta}"
|
||||
|
||||
size="$(stat -c%s "$iso")"
|
||||
virsh vol-delete --pool "$POOL" "$iso" >/dev/null 2>&1 || true
|
||||
virsh vol-create-as "$POOL" "$iso" "$size" --format raw >/dev/null
|
||||
virsh vol-upload --pool "$POOL" "$iso" "$iso"
|
||||
|
||||
echo "$iso published to pool '$POOL' ($size bytes)"
|
||||
echo "attach it as a virtio disk, not a SATA cdrom:"
|
||||
echo " --disk vol=${POOL}/${iso},device=disk,bus=virtio,readonly=on"
|
||||
echo "Debian genericcloud images carry no AHCI driver, so a SATA cdrom is invisible"
|
||||
echo "to the guest and cloud-init fails with no error anywhere."
|
||||
@@ -1,47 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Confirm the lab infrastructure is intact. Run on the lab host.
|
||||
#
|
||||
# A 404 from the HTTPS entry point is the success signal: TLS terminated and the
|
||||
# request reached Traefik, which simply had no matching ingress rule. A 502 or a
|
||||
# refused connection means the chain is broken somewhere.
|
||||
set -uo pipefail
|
||||
|
||||
export LIBVIRT_DEFAULT_URI="${LIBVIRT_DEFAULT_URI:-qemu:///system}"
|
||||
HOSTS="${HOSTS:-auth.hyeonworks.com app1.hyeonworks.com app2.hyeonworks.com}"
|
||||
NODE_IPS="${NODE_IPS:-192.168.122.11 192.168.122.12}"
|
||||
fail=0
|
||||
|
||||
check() { # description, expected, actual
|
||||
if [ "$2" = "$3" ]; then printf ' ok %-34s %s\n' "$1" "$3"
|
||||
else printf ' FAIL %-34s got %s, want %s\n' "$1" "$3" "$2"; fail=1; fi
|
||||
}
|
||||
|
||||
echo "== guests =="
|
||||
for name in kc-lab-1 kc-lab-2; do
|
||||
check "$name" running "$(virsh domstate "$name" 2>/dev/null || echo absent)"
|
||||
done
|
||||
|
||||
echo "== k3s =="
|
||||
ready="$(kubectl get nodes --no-headers 2>/dev/null | grep -c ' Ready ')"
|
||||
check "nodes Ready" 2 "$ready"
|
||||
lb="$(kubectl -n kube-system get svc traefik \
|
||||
-o jsonpath='{.status.loadBalancer.ingress[*].ip}' 2>/dev/null | wc -w)"
|
||||
check "traefik node IPs" 2 "$lb"
|
||||
|
||||
echo "== host nginx =="
|
||||
check "service" active "$(systemctl is-active nginx)"
|
||||
check "cert renew timer" active "$(systemctl is-active certbot-renew.timer)"
|
||||
for ip in $NODE_IPS; do
|
||||
check "traefik $ip" 404 "$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "http://${ip}/")"
|
||||
done
|
||||
|
||||
echo "== public entry point =="
|
||||
for h in $HOSTS; do
|
||||
check "https://$h" 404 "$(curl -s -o /dev/null -w '%{http_code}' --max-time 8 "https://${h}/")"
|
||||
check "tls verify $h" 0 "$(curl -s -o /dev/null -w '%{ssl_verify_result}' --max-time 8 "https://${h}/")"
|
||||
done
|
||||
check "http redirect" 301 "$(curl -s -o /dev/null -w '%{http_code}' --max-time 8 "http://${HOSTS%% *}/")"
|
||||
|
||||
echo
|
||||
[ "$fail" -eq 0 ] && echo "lab is healthy" || echo "lab has failures"
|
||||
exit "$fail"
|
||||
+64
-4
@@ -71,8 +71,9 @@ services:
|
||||
SERVER_PORT: "8081"
|
||||
SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI: http://localhost:8080/realms/keycloak-patterns
|
||||
SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI: http://keycloak:8080/realms/keycloak-patterns/protocol/openid-connect/certs
|
||||
ports:
|
||||
- "127.0.0.1:8081:8081"
|
||||
EDGE_INTERNAL_AUTH_TOKEN: ${INTERNAL_AUTH_TOKEN:?set INTERNAL_AUTH_TOKEN in .env}
|
||||
expose:
|
||||
- "8081"
|
||||
depends_on:
|
||||
keycloak:
|
||||
condition: service_healthy
|
||||
@@ -88,13 +89,68 @@ services:
|
||||
- keycloak-net
|
||||
restart: unless-stopped
|
||||
|
||||
oauth2-proxy:
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.2
|
||||
command:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --provider=keycloak-oidc
|
||||
- --oidc-issuer-url=http://localhost:8080/realms/keycloak-patterns
|
||||
- --skip-oidc-discovery=true
|
||||
- --login-url=http://localhost:8080/realms/keycloak-patterns/protocol/openid-connect/auth
|
||||
- --redeem-url=http://keycloak:8080/realms/keycloak-patterns/protocol/openid-connect/token
|
||||
- --oidc-jwks-url=http://keycloak:8080/realms/keycloak-patterns/protocol/openid-connect/certs
|
||||
- --profile-url=http://keycloak:8080/realms/keycloak-patterns/protocol/openid-connect/userinfo
|
||||
- --validate-url=http://keycloak:8080/realms/keycloak-patterns/protocol/openid-connect/userinfo
|
||||
- --redirect-url=http://localhost:8088/oauth2/callback
|
||||
- --upstream=http://app:8081
|
||||
- --email-domain=*
|
||||
- --scope=openid profile email
|
||||
- --code-challenge-method=S256
|
||||
- --reverse-proxy=true
|
||||
- --trusted-proxy-ip=172.30.40.10/32
|
||||
- --cookie-name=AP4_SESSION
|
||||
- --cookie-secure=false
|
||||
- --cookie-samesite=lax
|
||||
- --cookie-expire=1h
|
||||
- --session-cookie-minimal=true
|
||||
- --skip-provider-button=true
|
||||
- --set-xauthrequest=true
|
||||
- --pass-user-headers=true
|
||||
- --whitelist-domain=localhost:8088
|
||||
- --whitelist-domain=localhost:8080
|
||||
environment:
|
||||
OAUTH2_PROXY_CLIENT_ID: edge-proxy
|
||||
OAUTH2_PROXY_CLIENT_SECRET: ${EDGE_PROXY_CLIENT_SECRET:?set EDGE_PROXY_CLIENT_SECRET in .env}
|
||||
OAUTH2_PROXY_COOKIE_SECRET: ${OAUTH2_PROXY_COOKIE_SECRET:?set OAUTH2_PROXY_COOKIE_SECRET in .env}
|
||||
expose:
|
||||
- "4180"
|
||||
depends_on:
|
||||
keycloak:
|
||||
condition: service_healthy
|
||||
app:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD
|
||||
- /bin/oauth2-proxy
|
||||
- --version
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 5s
|
||||
networks:
|
||||
- keycloak-net
|
||||
restart: unless-stopped
|
||||
|
||||
nginx:
|
||||
build:
|
||||
context: ./frontend
|
||||
environment:
|
||||
INTERNAL_AUTH_TOKEN: ${INTERNAL_AUTH_TOKEN:?set INTERNAL_AUTH_TOKEN in .env}
|
||||
ports:
|
||||
- "127.0.0.1:${NGINX_PORT:-8088}:80"
|
||||
depends_on:
|
||||
app:
|
||||
oauth2-proxy:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test:
|
||||
@@ -104,7 +160,8 @@ services:
|
||||
timeout: 5s
|
||||
retries: 12
|
||||
networks:
|
||||
- keycloak-net
|
||||
keycloak-net:
|
||||
ipv4_address: 172.30.40.10
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
@@ -114,3 +171,6 @@ volumes:
|
||||
networks:
|
||||
keycloak-net:
|
||||
driver: bridge
|
||||
ipam:
|
||||
config:
|
||||
- subnet: 172.30.40.0/24
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
# AP4 · oauth2-proxy Edge Forward Auth
|
||||
|
||||
## 첫 단계: oauth2-proxy 자체 OIDC 흐름
|
||||
|
||||
`feature/keycloak-oauth2-proxy-oidc-flow`에서는 oauth2-proxy를
|
||||
`http://localhost:4180`에 직접 노출해 구성 요소를 분리해서 확인합니다.
|
||||
|
||||
1. `/edge/me` 미인증 요청이 Keycloak로 redirect됩니다.
|
||||
2. oauth2-proxy는 confidential `edge-proxy` client와 PKCE S256을 사용합니다.
|
||||
3. callback에서 code/token 교환과 ID/access token 검증은 서버끼리
|
||||
수행합니다.
|
||||
4. 브라우저에는 HttpOnly `AP4_SESSION` cookie만 남습니다.
|
||||
5. oauth2-proxy가 backend 요청에 `X-Forwarded-User`를 붙여 200을 받습니다.
|
||||
|
||||
Keycloak이 발급하는 issuer는 브라우저 기준
|
||||
`http://localhost:8080/realms/keycloak-patterns`입니다. 컨테이너 내부의
|
||||
`localhost`는 oauth2-proxy 자신이므로 discovery endpoint에 도달할 수
|
||||
없습니다. 그래서 이 로컬 Compose 구성은 issuer 검증값은 외부 URL로
|
||||
유지하되, login URL은 브라우저용 외부 주소, token/JWKS/userinfo는
|
||||
`http://keycloak:8080` 내부 주소로 각각 명시합니다.
|
||||
|
||||
HTTP 로컬 시연이라 `cookie-secure=false`를 사용합니다. 운영 HTTPS에서는
|
||||
반드시 secure cookie로 되돌려야 합니다.
|
||||
|
||||
## 다음 단계의 보안 전제
|
||||
|
||||
이 첫 feature의 backend는 전달된 사용자 헤더를 신뢰하며 8081도
|
||||
loopback에 publish되어 있습니다. 따라서 로컬에서 직접
|
||||
`X-Forwarded-User: spoofed-admin`을 보내면 우회가 재현됩니다. 이후
|
||||
Nginx `auth_request` 통합을 거쳐 최종 feature에서 backend no-publish와
|
||||
내부 shared-secret 검증을 함께 적용합니다.
|
||||
|
||||
## 두 번째 단계: Nginx `auth_request`
|
||||
|
||||
`feature/keycloak-nginx-auth-request-integration`부터 외부 진입점은
|
||||
`http://localhost:8088` Nginx 하나입니다. oauth2-proxy의 4180 포트는
|
||||
Compose 네트워크에만 expose됩니다.
|
||||
|
||||
- Nginx의 정확 일치 `location = /oauth2/auth`는 `internal`이라 외부에서
|
||||
직접 호출할 수 없습니다.
|
||||
- 인증 서브리퀘스트에는 본문을 보내지 않고 `Content-Length`도
|
||||
비웁니다.
|
||||
- 일반 브라우저 요청의 401은 `/oauth2/start` 302로 변환합니다.
|
||||
- API 요청 `/api/edge`는 redirect하지 않고 JSON 401을 반환합니다.
|
||||
- 인증 성공 시 oauth2-proxy의 `X-Auth-Request-User`와 email만 backend로
|
||||
전달합니다.
|
||||
|
||||
Nginx 컨테이너 IP를 전용 Compose subnet에서 고정하고 oauth2-proxy의
|
||||
trusted proxy를 그 단일 IP로 제한합니다. 다만 이 단계에서는 backend
|
||||
8081이 로컬 호스트에 열려 있어 신뢰 헤더를 직접 위조할 수 있습니다.
|
||||
그 재현 조건은 마지막 feature에서 제거합니다.
|
||||
|
||||
## 마지막 단계: 신뢰 경계와 헤더 스푸핑 방어
|
||||
|
||||
`feature/keycloak-header-spoofing-defense`에서는 신뢰 경계를 실제
|
||||
네트워크와 application 양쪽에서 강제합니다.
|
||||
|
||||
1. backend 8081과 oauth2-proxy 4180은 host에 publish하지 않습니다.
|
||||
브라우저가 접근 가능한 application 포트는 Nginx 8088뿐입니다.
|
||||
2. Nginx는 client가 보낸 `X-Auth-Request-User`, email, 내부 토큰을
|
||||
그대로 전달하지 않고 oauth2-proxy 결과와 server-side 토큰으로
|
||||
항상 덮어씁니다.
|
||||
3. backend는 `X-Auth-Request-User`와 `X-Internal-Auth-Token`이 모두
|
||||
유효할 때만 edge identity를 받아들이며 token은 constant-time으로
|
||||
비교합니다.
|
||||
|
||||
shared token은 방어 심층화 수단입니다. 운영에서는 Secret Manager나
|
||||
orchestrator secret으로 주입하고 주기적으로 교체해야 합니다. 서비스
|
||||
간 mTLS 또는 service mesh identity를 사용할 수 있다면 단순 shared
|
||||
token보다 강한 workload identity로 대체하는 편이 좋습니다.
|
||||
@@ -0,0 +1,26 @@
|
||||
# AP4 edge forward-auth with Google federation
|
||||
|
||||
Google federation은 AP4의 edge contract를 바꾸지 않는다.
|
||||
|
||||
```text
|
||||
Browser -> nginx -> oauth2-proxy -> Keycloak -> Google
|
||||
Browser <- AP4_SESSION <- oauth2-proxy <- Keycloak
|
||||
nginx -> trusted identity headers -> upstream app
|
||||
```
|
||||
|
||||
oauth2-proxy가 신뢰하는 issuer는 Google이 아니라 Keycloak이다. Google ID
|
||||
token은 Keycloak broker 경계 안에서 검증되고, oauth2-proxy는 Keycloak
|
||||
authorization code/token과 session cookie만 다룬다. upstream 앱도
|
||||
broker 여부와 무관하게 동일한 trusted headers를 받는다.
|
||||
|
||||
`verify-edge-google-federation.sh`는 mock Google 로그인, confidential
|
||||
server-side token 교환(브라우저에 token 요청 없음), 미검증 broker email 거부,
|
||||
Keycloak email verification 완료 후 HttpOnly edge cookie와 brokered
|
||||
subject/email header를 실제 컨테이너와 브라우저로 검증한다. minimal session의
|
||||
`X-Auth-Request-User`는 표시용 username이 아니라 Keycloak의 안정적인 local
|
||||
subject UUID이며, 화면 이름이 필요하면 별도 허용 header를 명시한다.
|
||||
|
||||
brokered token/claims가 client-side session cookie의 4KB 한계를 넘지 않도록
|
||||
oauth2-proxy에는 `session-cookie-minimal=true`를 적용한다. AP4 upstream은
|
||||
token forwarding이 아니라 trusted identity headers만 사용하므로 cookie에
|
||||
access/refresh/ID token을 보관할 필요가 없다.
|
||||
@@ -0,0 +1,20 @@
|
||||
# AP4 edge forward-auth: local identity profile
|
||||
|
||||
nginx의 `auth_request`가 oauth2-proxy `/oauth2/auth`를 호출하고, 미인증
|
||||
브라우저만 Keycloak로 redirect한다. oauth2-proxy는 confidential
|
||||
`edge-proxy` client로 code를 교환하며 browser에는 HttpOnly session cookie만
|
||||
남긴다.
|
||||
|
||||
Google이 없어도 이 경계는 완전히 동작한다. 사용자는 Keycloak local
|
||||
credential로 로그인하고 upstream 애플리케이션은 OAuth/OIDC를 몰라도 된다.
|
||||
|
||||
보안 경계:
|
||||
|
||||
- backend와 oauth2-proxy 포트를 host에 publish하지 않는다.
|
||||
- 외부에서 받은 identity header를 nginx가 제거·덮어쓴다.
|
||||
- backend는 trusted nginx가 주입하는 별도 internal token도 확인한다.
|
||||
- 브라우저가 직접 호출한 `/oauth2/auth`는 공개하지 않는다.
|
||||
|
||||
`verify-edge-no-google-profile.sh`는 Compose와 nginx 계약을 검사하고,
|
||||
`verify-pattern4.sh`는 실제 local-user 브라우저 흐름과 spoofing 방어를
|
||||
검증한다.
|
||||
@@ -1,13 +0,0 @@
|
||||
=== [기준선 1] 클러스터 뷰 — 양쪽 파드의 마지막 ISPN000094 ===
|
||||
keycloak-0: [keycloak-1-48749(v=16.0.12)|5] (2) [keycloak-1-48749(v=16.0.12), keycloak-0-30843(v=16.0.12)]
|
||||
keycloak-1: [keycloak-1-48749(v=16.0.12)|5] (2) [keycloak-1-48749(v=16.0.12), keycloak-0-30843(v=16.0.12)]
|
||||
|
||||
=== [기준선 2] JGROUPS_PING — 디스커버리 등록 ===
|
||||
name | ip | coord | coordinated_by
|
||||
------------------+-----------------+-------+---------------------------------------------
|
||||
keycloak-0-30843 | 10.42.1.43:7800 | f | uuid://00000000-0000-0000-0000-000000000007
|
||||
keycloak-1-48749 | 10.42.0.35:7800 | t | uuid://00000000-0000-0000-0000-000000000007
|
||||
(2 rows)
|
||||
|
||||
=== [기준선 3] 기존 NetworkPolicy ===
|
||||
No resources found in keycloak-lab namespace.
|
||||
@@ -1,17 +0,0 @@
|
||||
=== [대조군] 차단 전 — keycloak-0 로그인 → keycloak-1 에서 refresh ===
|
||||
sid tAWs2gCPr6SOcD4jDR9-_CzB
|
||||
keycloak-1 에서 refresh: 200
|
||||
|
||||
=== [기준선 4] JGroups 지표 — 양쪽 노드 ===
|
||||
--- K0 (10.42.1.43) ---
|
||||
vendor_jgroups_stats_bytes_sent_total 31476.0
|
||||
vendor_jgroups_merge3_get_num_merge_events 0.0
|
||||
vendor_jgroups_merge3_get_views 0.0
|
||||
vendor_jgroups_fd_sock2_get_num_suspected_members 0.0
|
||||
vendor_jgroups_nakack2_get_xmit_table_missing_messages 0.0
|
||||
--- K1 (10.42.0.35) ---
|
||||
vendor_jgroups_merge3_get_views 0.0
|
||||
vendor_jgroups_stats_bytes_sent_total 126765.0
|
||||
vendor_jgroups_nakack2_get_xmit_table_missing_messages 0.0
|
||||
vendor_jgroups_fd_sock2_get_num_suspected_members 0.0
|
||||
vendor_jgroups_merge3_get_num_merge_events 0.0
|
||||
@@ -1,8 +0,0 @@
|
||||
=== 차단 적용 ===
|
||||
networkpolicy.networking.k8s.io/a1-block-jgroups-transport created
|
||||
a1-block-jgroups-transport map[app:keycloak]
|
||||
|
||||
적용 시각: 11:38:08
|
||||
=== FD_SOCK2 가 상대를 의심하기까지 기다린다 (15초 간격, 최대 3분) ===
|
||||
+15초 suspected(k0 k1) =
|
||||
→ 변화 감지
|
||||
@@ -1,26 +0,0 @@
|
||||
차단 경과: 11:38:51 (적용 11:38:08)
|
||||
|
||||
=== [차단 후 1] JGroups 지표 ===
|
||||
--- keycloak-0 ---
|
||||
vendor_jgroups_stats_bytes_sent_total 32857.0
|
||||
vendor_jgroups_merge3_get_num_merge_events 0.0
|
||||
vendor_jgroups_merge3_get_views 0.0
|
||||
vendor_jgroups_fd_sock2_get_num_suspected_members 0.0
|
||||
vendor_jgroups_nakack2_get_xmit_table_missing_messages 0.0
|
||||
--- keycloak-1 ---
|
||||
vendor_jgroups_merge3_get_views 0.0
|
||||
vendor_jgroups_stats_bytes_sent_total 129103.0
|
||||
vendor_jgroups_nakack2_get_xmit_table_missing_messages 0.0
|
||||
vendor_jgroups_fd_sock2_get_num_suspected_members 0.0
|
||||
vendor_jgroups_merge3_get_num_merge_events 0.0
|
||||
|
||||
=== [차단 후 2] 클러스터 뷰 — 갈라졌는가 ===
|
||||
keycloak-0:
|
||||
keycloak-1:
|
||||
=== [차단 후 3] JGROUPS_PING — 디스커버리는 살아 있는가 ===
|
||||
name | ip | coord
|
||||
------------------+-----------------+-------
|
||||
keycloak-0-30843 | 10.42.1.43:7800 | f
|
||||
keycloak-1-48749 | 10.42.0.35:7800 | t
|
||||
(2 rows)
|
||||
|
||||
@@ -1,16 +0,0 @@
|
||||
=== [문제 확정] NetworkPolicy 적용 후에도 기존 연결이 conntrack 에 살아 있다 ===
|
||||
--- kc-lab-1 ---
|
||||
tcp 6 86398 ESTABLISHED src=10.42.0.35 dst=10.42.1.43 sport=40023 dport=7800 src=10.42.1.43 dst=10.42.0.35 sport=7800 dport=40023 [ASSURED] mark=0 use=1
|
||||
tcp 6 79982 ESTABLISHED src=10.42.0.35 dst=10.42.1.43 sport=50477 dport=57800 src=10.42.1.43 dst=10.42.0.35 sport=57800 dport=50477 [ASSURED] mark=0 use=1
|
||||
--- kc-lab-2 ---
|
||||
tcp 6 86398 ESTABLISHED src=10.42.0.35 dst=10.42.1.43 sport=40023 dport=7800 src=10.42.1.43 dst=10.42.0.35 sport=7800 dport=40023 [ASSURED] mark=0 use=1
|
||||
tcp 6 33 SYN_SENT src=10.42.1.58 dst=10.42.0.35 sport=34824 dport=7800 [UNREPLIED] src=10.42.0.35 dst=10.42.1.58 sport=7800 dport=34824 mark=0 use=1
|
||||
tcp 6 79982 ESTABLISHED src=10.42.0.35 dst=10.42.1.43 sport=50477 dport=57800 src=10.42.1.43 dst=10.42.0.35 sport=57800 dport=50477 [ASSURED] mark=0 use=1
|
||||
|
||||
=== [조치] 7800 흐름의 conntrack 항목을 지운다 → 다음 패킷이 정책을 다시 탄다 ===
|
||||
kc-lab-1: tcp 6 86398 ESTABLISHED src=10.42.0.35 dst=10.42.1.43 sport=40023 dport=7800 src=10.42.1.43 dst=10.42.0.35 sport=7800 dport=40023 [ASSURED] mark=0 use=1 conntrack v1.4.7 (conntrack-tools): 0 flow entries have been deleted.
|
||||
kc-lab-2: tcp 6 33 SYN_SENT src=10.42.1.58 dst=10.42.0.35 sport=34824 dport=7800 [UNREPLIED] src=10.42.0.35 dst=10.42.1.58 sport=7800 dport=34824 mark=0 use=1 conntrack v1.4.7 (conntrack-tools): 0 flow entries have been deleted.
|
||||
|
||||
=== 삭제 후 7800 conntrack ===
|
||||
kc-lab-1: 2 건
|
||||
kc-lab-2: 2 건
|
||||
@@ -1,13 +0,0 @@
|
||||
관찰 시작: 11:42:03
|
||||
+20초 suspected(k0 k1) = []
|
||||
+40초 suspected(k0 k1) = []
|
||||
+60초 suspected(k0 k1) = [0.0 0.0 0.0 0.0 ]
|
||||
+80초 suspected(k0 k1) = []
|
||||
+100초 suspected(k0 k1) = []
|
||||
+120초 suspected(k0 k1) = []
|
||||
+140초 suspected(k0 k1) = [0.0 ]
|
||||
+160초 suspected(k0 k1) = [0.0 0.0 0.0 0.0 ]
|
||||
|
||||
=== 클러스터 뷰 변화 (최근 8분) ===
|
||||
--- keycloak-0 ---
|
||||
--- keycloak-1 ---
|
||||
@@ -1,16 +0,0 @@
|
||||
=== vendor_cluster_size — 지난 25분 (차단 11:38:08, conntrack 삭제 11:41) ===
|
||||
keycloak-0:
|
||||
11:20=2 11:21=2 11:22=2 11:23=2 11:24=2 11:25=2 11:26=2 11:27=2 11:28=2 11:29=2 11:30=2 11:31=2 11:32=2 11:33=2 11:34=2 11:35=2 11:36=2 11:37=2 11:38=2 11:39=2 11:40=2 11:41=2 11:42=2 11:43=2 11:44=2 11:45=2
|
||||
keycloak-1:
|
||||
11:20=2 11:21=2 11:22=2 11:23=2 11:24=2 11:25=2 11:26=2 11:27=2 11:28=2 11:29=2 11:30=2 11:31=2 11:32=2 11:33=2 11:34=2 11:35=2 11:36=2 11:37=2 11:38=2 11:39=2 11:40=2 11:41=2 11:42=2 11:43=2 11:44=2 11:45=2
|
||||
|
||||
=== 현재 값 ===
|
||||
keycloak-1 = 2 멤버
|
||||
keycloak-0 = 2 멤버
|
||||
|
||||
=== 7800 소켓 상태 (파드 내부) ===
|
||||
keycloak-0 2
|
||||
keycloak-1 2
|
||||
=== conntrack ===
|
||||
kc-lab-1 1 건
|
||||
kc-lab-2 1 건
|
||||
@@ -1,9 +0,0 @@
|
||||
=== 정책이 걸린 상태에서 keycloak-0 을 재시작한다 → 재연결이 막힌다 ===
|
||||
재시작 시각: 11:46:07
|
||||
pod "keycloak-0" deleted from keycloak-lab namespace
|
||||
keycloak-0 false 10.42.1.67 2026-09-04T02:44:23Z
|
||||
|
||||
=== cluster_size 추이 ===
|
||||
keycloak-0: 11:45:27=1 11:45:57=1 11:46:27=1 11:46:57=1 11:47:27=1
|
||||
keycloak-0: 11:40:57=2 11:41:27=2 11:41:57=2 11:42:27=2 11:42:57=2 11:43:27=2 11:43:57=2
|
||||
keycloak-1: 11:40:57=2 11:41:27=2 11:41:57=2 11:42:27=2 11:42:57=2 11:43:27=2 11:43:57=2 11:44:27=1 11:44:57=1 11:45:27=1 11:45:57=1 11:46:27=1 11:46:57=1 11:47:27=1
|
||||
@@ -1,16 +0,0 @@
|
||||
=== keycloak-0 헬스 상태 ===
|
||||
keycloak-0 = 10.42.1.67 keycloak-1 = 10.42.0.35
|
||||
PodReadyToStartContainers=True
|
||||
Initialized=True
|
||||
Ready=False ContainersNotReady
|
||||
ContainersReady=False ContainersNotReady
|
||||
PodScheduled=True
|
||||
|
||||
=== ★ 본 시험 — 분단 상태에서 교차 노드 세션이 되는가 ===
|
||||
[1] keycloak-0 로그인 sid=nShl5TaBrZnKStDqaspjgmJB
|
||||
[2] keycloak-1 에서 refresh HTTP 200
|
||||
[3] keycloak-1 에서 로그아웃 HTTP 204
|
||||
[4] keycloak-0 에서 재갱신 시도 HTTP 200
|
||||
(400 이면 무효화가 전파된 것)
|
||||
|
||||
=== DB 세션 수 ===
|
||||
@@ -1,33 +0,0 @@
|
||||
=== 그 sid 가 DB 에 남아 있는가 ===
|
||||
user_session_id | offline_flag | last_session_refresh
|
||||
-----------------+--------------+----------------------
|
||||
(0 rows)
|
||||
|
||||
=== 전체 온라인 세션 수 ===
|
||||
1
|
||||
|
||||
=== 노드별 세션 캐시 엔트리 (Prometheus) ===
|
||||
keycloak-1 kc-lab-1 = 0
|
||||
keycloak-0 kc-lab-2 = 1
|
||||
|
||||
=== keycloak-0 이 Ready 가 아닌 이유 — 헬스 응답 ===
|
||||
{
|
||||
"status": "DOWN",
|
||||
"checks": [
|
||||
{
|
||||
"name": "Graceful Shutdown",
|
||||
"status": "UP"
|
||||
},
|
||||
{
|
||||
"name": "Keycloak cluster health check",
|
||||
"status": "DOWN",
|
||||
"data": {
|
||||
"Failing since": "2026-09-04 02:45:14,251"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Keycloak database connections async health check",
|
||||
"status": "UP"
|
||||
},
|
||||
{
|
||||
"name": "Keycloak Initialized",
|
||||
@@ -1,25 +0,0 @@
|
||||
=== 양쪽 노드의 readiness — 둘 다 DOWN 이면 전면 장애다 ===
|
||||
Traceback (most recent call last):
|
||||
File "<string>", line 3, in <module>
|
||||
d=json.load(sys.stdin)
|
||||
File "/usr/lib/python3.14/json/__init__.py", line 298, in load
|
||||
return loads(fp.read(),
|
||||
cls=cls, object_hook=object_hook,
|
||||
parse_float=parse_float, parse_int=parse_int,
|
||||
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
|
||||
File "/usr/lib/python3.14/json/__init__.py", line 352, in loads
|
||||
return _default_decoder.decode(s)
|
||||
~~~~~~~~~~~~~~~~~~~~~~~^^^
|
||||
File "/usr/lib/python3.14/json/decoder.py", line 348, in decode
|
||||
raise JSONDecodeError("Extra data", s, end)
|
||||
json.decoder.JSONDecodeError: Extra data: line 21 column 2 (char 446)
|
||||
|
||||
=== 파드 Ready 상태 ===
|
||||
keycloak-0 false 0
|
||||
keycloak-1 true 0
|
||||
|
||||
=== ★ Service 엔드포인트 — 트래픽을 받는 파드가 남아 있는가 ===
|
||||
ready 주소: [10.42.0.35] notReady : [10.42.1.67]
|
||||
=== ★ 외부 진입점으로 실제 로그인이 되는가 (nginx→Traefik→Service) ===
|
||||
https://auth.hyeonworks.com/realms/master HTTP 200
|
||||
토큰 발급 HTTP 200
|
||||
@@ -1,24 +0,0 @@
|
||||
=== 차단 해제 ===
|
||||
해제 시각: 11:49:58
|
||||
networkpolicy.networking.k8s.io "a1-block-jgroups-transport" deleted from keycloak-lab namespace
|
||||
|
||||
=== 자동으로 다시 붙는가 (30초 간격, 최대 4분) ===
|
||||
+30초 keycloak-0=1 keycloak-1=1 | Ready 파드 2 개
|
||||
+60초 keycloak-0=1 keycloak-1=1 | Ready 파드 2 개
|
||||
+90초 keycloak-0=2 keycloak-1=2 | Ready 파드 3 개
|
||||
→ 클러스터 재형성
|
||||
|
||||
=== 복구 로그 ===
|
||||
keycloak-0: [keycloak-0-26403(v=16.0.12)|0] (1) [keycloak-0-26403(v=16.0.12)]
|
||||
keycloak-1: [keycloak-1-48749(v=16.0.12)|6] (1) [keycloak-1-48749(v=16.0.12)]
|
||||
|
||||
=== MERGE3 가 합쳤는가 ===
|
||||
merge_events keycloak-1 = 1
|
||||
merge_events keycloak-0 = 1
|
||||
=== JGROUPS_PING — 코디네이터가 하나로 돌아왔는가 ===
|
||||
name | ip | coord
|
||||
------------------+-----------------+-------
|
||||
keycloak-0-26403 | 10.42.1.67:7800 | t
|
||||
keycloak-1-48749 | 10.42.0.35:7800 | f
|
||||
(2 rows)
|
||||
|
||||
@@ -1,26 +0,0 @@
|
||||
# A-1 — JGroups 트랜스포트(7800) 차단 증거
|
||||
|
||||
2026-09-04 11:38–11:52 KST · Keycloak 26.7.0 / Infinispan 16.0.12
|
||||
해설: [`docs/experiment-a1-jgroups-transport-block.md`](../../experiment-a1-jgroups-transport-block.md)
|
||||
|
||||
| 파일 | 무엇을 보여주는가 |
|
||||
|---|---|
|
||||
| `01-baseline-cluster.txt` | 차단 전 — 양쪽이 뷰 ID 5·멤버 2로 일치, `JGROUPS_PING` 코디네이터 1명 |
|
||||
| `02-control-before-block.txt` | **대조군** — 차단 전 교차 노드 refresh `200`, JGroups 지표 전부 0 |
|
||||
| `03-block-applied.txt` | NetworkPolicy 적용. **빈 측정값을 "변화 감지"로 오판한 기록** |
|
||||
| `04-after-block-state.txt` | 차단 43초 후 — 지표 무변화, `JGROUPS_PING` 그대로 |
|
||||
| `05-conntrack-problem.txt` | **핵심 문제** — `ESTABLISHED [ASSURED]` 로 기존 연결이 살아 있음. FD_SOCK2 의 **57800** 포트도 함께 드러남 |
|
||||
| `06-partition-observed.txt` | 임시 curl 파드 폴링의 실패 — 빈 값·개수 불일치 |
|
||||
| `07-cluster-size.txt` | **`vendor_cluster_size` 가 25분 내내 2** — 분단이 일어나지 않았다는 결정적 증거 |
|
||||
| `08-restart-forced-partition.txt` | 재연결 강제 후 `2 → 1` |
|
||||
| `09-cross-node-under-partition.txt` | **본 시험** — 교차 refresh `200`(예측 적중), **로그아웃 후 재갱신 `200`(예측 빗나감)** |
|
||||
| `10-logout-not-propagated.txt` | 기제 확정 — **DB 행 0건인데 keycloak-0 캐시에 1건**, 헬스체크 `cluster health: DOWN` |
|
||||
| `11-service-impact.txt` | **분단 노드가 Service 에서 빠짐.** `ready=[10.42.0.35] notReady=[10.42.1.67]`, 외부 로그인 `200` |
|
||||
| `12-recovery.txt` | 90초 만에 자동 재형성, `merge3_get_num_merge_events = 1`, 코디네이터 재선출 |
|
||||
| `a1-cluster-size-partition-recovery.png` | Grafana — `vendor_cluster_size` 가 `2 → 1 → 2` 로 움직이는 전 구간 |
|
||||
|
||||
## 핵심 세 줄
|
||||
|
||||
1. **NetworkPolicy 만으로는 이미 붙어 있는 클러스터를 못 끊는다.** conntrack 의 ESTABLISHED 가 먼저 통과시킨다.
|
||||
2. **세션 공유는 분단을 견딘다(200).** 통념이 틀렸고 A-0 모델이 맞다.
|
||||
3. **로그아웃 무효화는 7800 을 탄다.** DB 행이 지워져도 반대편은 낡은 캐시로 200 을 준다 — A-0 의 인과 해석을 정정한다.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 66 KiB |
@@ -1,14 +0,0 @@
|
||||
=== A-2 기준선 — 클러스터가 정상으로 돌아왔는가 ===
|
||||
keycloak-0 true 10.42.1.67 kc-lab-2
|
||||
keycloak-1 true 10.42.0.35 kc-lab-1
|
||||
postgres-7b474b88c8-sn9ff true 10.42.1.24 kc-lab-2
|
||||
|
||||
cluster_size keycloak-1 = 2
|
||||
cluster_size keycloak-0 = 2
|
||||
|
||||
=== 노드별 세션 캐시 (실험 설계에 필요) ===
|
||||
keycloak-1 kc-lab-1 = 0 건
|
||||
keycloak-0 kc-lab-2 = 0 건
|
||||
|
||||
=== DB 온라인 세션 ===
|
||||
2
|
||||
@@ -1,10 +0,0 @@
|
||||
pod/a2-probe condition met
|
||||
keycloak-0=10.42.1.67 keycloak-1=10.42.0.35
|
||||
|
||||
=== [준비] 양쪽 노드에 세션을 하나씩 만든다 ===
|
||||
keycloak-0 에서 로그인 sid=EAXV5HcG2J1BZ3vnwONf64AQ 토큰길이=613
|
||||
keycloak-1 에서 로그인 sid=McyTj5lj3n_JqApCXeuAHExc 토큰길이=613
|
||||
|
||||
=== [확인] 세션이 각자 노드에만 캐시되었는가 ===
|
||||
keycloak-1 = 0 건
|
||||
keycloak-0 = 1 건
|
||||
@@ -1,16 +0,0 @@
|
||||
=== [1] 토큰을 새로 발급 (access 수명 60초) ===
|
||||
발급 완료 sid=RKXQGAgkuLtouFMVPTFmp_0_
|
||||
|
||||
=== [2] PostgreSQL 정지 ===
|
||||
정지 시각: 11:56:04
|
||||
deployment.apps/postgres scaled
|
||||
pod/postgres-7b474b88c8-sn9ff condition met
|
||||
삭제 완료: 11:56:04
|
||||
|
||||
=== [3] 네 경로를 즉시 시험 ===
|
||||
④ 이미 발급된 access token 으로 관리 API HTTP 000000{"error":"HTTP 401 Unauthorized"}401
|
||||
① 캐시를 가진 노드(keycloak-0)에서 refresh HTTP 500
|
||||
② 캐시가 없는 노드(keycloak-1)에서 refresh HTTP 500
|
||||
③ 새 로그인 HTTP 500
|
||||
--- 오류 본문 (새 로그인) ---
|
||||
{"error":"unknown_error","error_description":"For more on this error consult the server log."}
|
||||
@@ -1,29 +0,0 @@
|
||||
=== 파드 Ready 상태 — DB 가 없으면 어떻게 되는가 ===
|
||||
keycloak-0 false 0
|
||||
keycloak-1 false 0
|
||||
|
||||
=== Service 엔드포인트 ===
|
||||
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
|
||||
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
|
||||
notReady: [10.42.0.35 10.42.1.67]
|
||||
=== health/ready 상세 ===
|
||||
전체: DOWN
|
||||
Graceful Shutdown UP
|
||||
Keycloak cluster health check UP
|
||||
Keycloak database connections async health check DOWN
|
||||
Keycloak Initialized UP
|
||||
|
||||
=== ④ 다시 — 서명 검증만 필요한 경로는 살아 있는가 ===
|
||||
JWKS 엔드포인트(realm 공개키) HTTP 200
|
||||
realm 메타데이터(.well-known) HTTP 200
|
||||
관리 API(세션 조회 필요) HTTP 500
|
||||
|
||||
=== 외부 진입점 ===
|
||||
https://auth.hyeonworks.com/realms/master HTTP 503
|
||||
|
||||
=== Keycloak 로그 — 실제 오류 ===
|
||||
at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:664)
|
||||
at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:645)
|
||||
Caused by: java.net.ConnectException: Connection refused
|
||||
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:219)
|
||||
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:365)
|
||||
@@ -1,21 +0,0 @@
|
||||
=== ★ up 지표는 무엇을 말하는가 (프로세스는 살아 있다) ===
|
||||
up{pod=keycloak-1} = 1 ← 1 인데 서비스는 503 이다
|
||||
up{pod=keycloak-0} = 1 ← 1 인데 서비스는 503 이다
|
||||
|
||||
=== 복구 — PostgreSQL 재기동 ===
|
||||
재기동 시각: 11:57:09
|
||||
deployment.apps/postgres scaled
|
||||
Waiting for deployment "postgres" rollout to finish: 0 out of 1 new replicas have been updated...
|
||||
Waiting for deployment "postgres" rollout to finish: 0 of 1 updated replicas are available...
|
||||
deployment "postgres" successfully rolled out
|
||||
|
||||
=== Keycloak 이 스스로 회복하는가 (재시작 없이) ===
|
||||
+15초 keycloak-0 true keycloak-1 true | 외부 HTTP 200
|
||||
→ 서비스 복귀
|
||||
|
||||
=== 재시작 횟수 — 파드가 죽었다 살아난 것인가, 그대로 회복한 것인가 ===
|
||||
keycloak-0 0
|
||||
keycloak-1 0
|
||||
|
||||
=== 정지 전 세션이 살아남았는가 ===
|
||||
online 세션 5
|
||||
@@ -1,19 +0,0 @@
|
||||
# A-2 — PostgreSQL 정지 증거
|
||||
|
||||
2026-09-04 11:56–11:58 KST · Keycloak 26.7.0
|
||||
해설: [`docs/experiment-a2-database-loss.md`](../../experiment-a2-database-loss.md)
|
||||
|
||||
| 파일 | 무엇을 보여주는가 |
|
||||
|---|---|
|
||||
| `01-baseline.txt` | 정지 전 — 양쪽 Ready, `cluster_size=2` |
|
||||
| `02-setup-sessions.txt` | 양쪽 노드에 세션 하나씩. 캐시는 각자 노드에만 |
|
||||
| `03-four-paths.txt` | **네 경로 전부 `500`.** 캐시를 가진 노드도 실패 — refresh 는 쓰기다 |
|
||||
| `04-health-and-service.txt` | **전면 장애 증거** — Ready 파드 0개, `ready 주소=[]`, 외부 **503**, `database connections: DOWN`. JWKS·.well-known 은 `200` |
|
||||
| `05-recovery.txt` | **`up=1` 인 채로 503.** DB 복귀 15초 후 재시작 0회로 자동 회복, 세션 5건 생존 |
|
||||
| `a2-up-stayed-1-during-outage.png` | Grafana — `up{job="keycloak"}` 이 전면 장애 내내 **1에 평평** |
|
||||
|
||||
## 핵심 세 줄
|
||||
|
||||
1. **DB 는 단일 장애점이다.** Keycloak 을 몇 대로 늘려도 같이 죽는다 — Ready 파드 0개, 외부 503.
|
||||
2. **캐시는 읽기를 대신할 뿐 쓰기를 못 한다.** refresh 는 `UPDATE LAST_SESSION_REFRESH` 를 하므로 캐시가 있어도 실패한다.
|
||||
3. **`up` 은 이 장애를 못 잡는다.** 알림은 readiness 와 외부 응답 코드에 걸어야 한다.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 60 KiB |
@@ -1,19 +0,0 @@
|
||||
=== [준비] 손실 측정 설계 확인 ===
|
||||
LAST_SESSION_REFRESH 는 integer(초) — 200ms 손실은 보이지 않는다
|
||||
created_on | integer | | not null |
|
||||
last_session_refresh | integer | | not null | 0
|
||||
"idx_user_session_expiration_created" btree (realm_id, offline_flag, remember_me, created_on, user_session_id, user_id)
|
||||
"idx_user_session_expiration_last_refresh" btree (realm_id, offline_flag, remember_me, last_session_refresh, user_session_id, user_id)
|
||||
→ 대신 행 존재 여부로 잰다. 로그인 하나 = 행 하나 = 이진 판정
|
||||
|
||||
전역 synchronous_commit: on
|
||||
|
||||
=== [1] 빠른 연속 로그인을 백그라운드로 시작 ===
|
||||
루프 시작
|
||||
6초 경과 — 지금까지 성공한 로그인: 0
|
||||
|
||||
=== [2] PostgreSQL 강제 종료 (SIGKILL) ===
|
||||
종료 시각: 12:00:26.511
|
||||
pod "postgres-7b474b88c8-xc2vt" force deleted from keycloak-lab namespace
|
||||
삭제 반환: 12:00:26.586
|
||||
클라이언트가 200 을 받은 로그인 수: 0
|
||||
@@ -1,14 +0,0 @@
|
||||
deployment "postgres" successfully rolled out
|
||||
|
||||
=== crash recovery 가 실행되었는가 (강제 종료의 흔적) ===
|
||||
2026-09-04 02:58:41.036 UTC [1] LOG: database system is ready to accept connections
|
||||
|
||||
=== [설계 확인] 로그인 트랜잭션도 synchronous_commit 을 끄는가 ===
|
||||
--- 로그인 트랜잭션 (INSERT 가 있는 것) ---
|
||||
2:BEGIN
|
||||
5:COMMIT
|
||||
6:BEGIN
|
||||
9:insert into OFFLINE_USER_SESSION (BROKER_SESSION_ID,CREATED_ON,DATA,LAST_SESSION_REFRESH,REALM_ID,REMEMBER_ME,USER_ID,VERSION,OFFLINE_FLAG,USER_SESSION_ID) values ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10)
|
||||
10:insert into OFFLINE_CLIENT_SESSION (DATA,REALM_ID,TIMESTAMP,VERSION,CLIENT_ID,CLIENT_STORAGE_PROVIDER,EXTERNAL_CLIENT_ID,OFFLINE_FLAG,USER_SESSION_ID) values ($1,$2,$3,$4,$5,$6,$7,$8,$9)
|
||||
11:SET LOCAL synchronous_commit TO OFF
|
||||
12:COMMIT
|
||||
@@ -1,17 +0,0 @@
|
||||
=== [1] 로그인 루프 시작 (호스트에서 백그라운드로 exec — 세션이 살아 있어야 한다) ===
|
||||
8초 동안 클라이언트가 200 을 받은 로그인: 106 건
|
||||
|
||||
=== [2] SIGKILL ===
|
||||
종료: 12:01:32.981
|
||||
반환: 12:01:33.236
|
||||
최종 성공 로그인 수: 110 건
|
||||
마지막 sid: FimM-krSybBACP2qIvshLWwU
|
||||
마지막 sid: EwFfFwOIfqiv8N5GQ5OjtsVq
|
||||
마지막 sid: CJX-PxFkS7rUc_9FQgB7iw1f
|
||||
마지막 sid: 1EFK7SgUA4M7tq_SkC_BD2er
|
||||
마지막 sid: _LiqTczuyxlpOs3T3xs25SLv
|
||||
|
||||
=== [3] PostgreSQL 재기동 후 crash recovery 확인 ===
|
||||
Waiting for deployment "postgres" rollout to finish: 0 of 1 updated replicas are available...
|
||||
deployment "postgres" successfully rolled out
|
||||
2026-09-04 02:59:48.427 UTC [1] LOG: database system is ready to accept connections
|
||||
@@ -1,27 +0,0 @@
|
||||
=== [4] 클라이언트가 받은 sid 가 DB 에 있는가 ===
|
||||
클라이언트가 200 을 받은 sid: 291 건
|
||||
DB 온라인 세션 총계: 375
|
||||
|
||||
--- 마지막 15건을 하나씩 조회 ---
|
||||
TCCOYnVlN30Y2fGyJEsVJ_Lq 있음
|
||||
vBcllIkKWovh-FXN9tSzmxAe 있음
|
||||
eMpN_ywUdRTks9uBE9aTumPK 있음
|
||||
aPC_T0yrlMjrlskcLAp0AvY6 있음
|
||||
UBPxmduB-ahGHBg636sY3AEz 있음
|
||||
HLnBloNX9R3qQJSkpOnkNqL4 있음
|
||||
_KPJS30IAqVhkTJGHcCxKvrM 있음
|
||||
rq3caZ9MkyMYlFSSzRjQLygD 있음
|
||||
ivvQm70hjl55DPpF7vpYmz_E 있음
|
||||
81mx-rmi-tAeogHx3-su3z2q 있음
|
||||
WnvNDH93uzcz1XNFbaMSXk1A 있음
|
||||
DXPAIhjO5sGpCUlcD8IEoS8L 있음
|
||||
aZMvl4IwPdK-rC_7bG005Z5C 있음
|
||||
eIuBCprfWA5x0glcgSKYrrX0 있음
|
||||
ozES5kEeu2IFf_cfcC_jFlbF 있음
|
||||
|
||||
마지막 15건 중 유실: 0 건
|
||||
|
||||
=== [5] 전체 대조 — 몇 건이나 사라졌는가 ===
|
||||
클라이언트 성공: 291 건
|
||||
DB 에 존재: 291 건
|
||||
★ 유실: 0 건
|
||||
@@ -1,12 +0,0 @@
|
||||
=== [정리] 세션 테이블 비우고 루프 잔여 확인 ===
|
||||
DELETE 375
|
||||
남은 세션: 0
|
||||
|
||||
=== [재주입] postmaster(PID 1)에 SIGKILL — 진짜 크래시 ===
|
||||
8초 후 성공 로그인: 110 건
|
||||
SIGKILL: 12:03:21.441
|
||||
최종 성공 로그인: 139 건
|
||||
|
||||
=== [검증] 이번엔 crash recovery 가 돌았는가 ===
|
||||
deployment "postgres" successfully rolled out
|
||||
2026-09-04 02:59:48.427 UTC [1] LOG: database system is ready to accept connections
|
||||
@@ -1,17 +0,0 @@
|
||||
=== 로그인 루프 시작 ===
|
||||
8초 후: 112 건
|
||||
|
||||
=== 백엔드 프로세스에 SIGKILL → postmaster 가 재초기화한다 ===
|
||||
시각: 12:04:22.063
|
||||
최종 성공 로그인: 153 건
|
||||
|
||||
=== [검증] crash recovery 가 돌았는가 ===
|
||||
2026-09-04 02:59:48.427 UTC [1] LOG: database system is ready to accept connections
|
||||
2026-09-04 03:02:35.807 UTC [1] LOG: server process (PID 40) was terminated by signal 9: Killed
|
||||
2026-09-04 03:02:35.807 UTC [1] LOG: terminating any other active server processes
|
||||
2026-09-04 03:02:35.814 UTC [1] LOG: all server processes terminated; reinitializing
|
||||
2026-09-04 03:02:35.896 UTC [2585] LOG: database system was not properly shut down; automatic recovery in progress
|
||||
2026-09-04 03:02:35.899 UTC [2585] LOG: redo starts at 0/23CAB68
|
||||
2026-09-04 03:02:35.904 UTC [2585] LOG: redo done at 0/2529E40 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
|
||||
2026-09-04 03:02:35.923 UTC [2586] LOG: checkpoint complete: wrote 113 buffers (0.7%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.004 s, sync=0.004 s, total=0.015 s; sync files=27, longest=0.003 s, average=0.001 s; distance=1405 kB, estimate=1405 kB; lsn=0/252A048, redo lsn=0/252A048
|
||||
2026-09-04 03:02:35.926 UTC [1] LOG: database system is ready to accept connections
|
||||
@@ -1,19 +0,0 @@
|
||||
=== 크래시 전후 대조 ===
|
||||
클라이언트가 200 과 토큰을 받은 로그인 : 153 건
|
||||
그중 DB 에 실제로 존재 : 149 건
|
||||
★ 유실 : 4 건
|
||||
DB 전체 온라인 세션 : 150 건
|
||||
|
||||
=== 유실된 sid 목록 ===
|
||||
★ CQUfg9HLH29xvhiu6pVlfWOo ← 토큰은 발급됐는데 세션이 없다
|
||||
★ 5gLP4fqmpZBbjhH_d-0TPMMr ← 토큰은 발급됐는데 세션이 없다
|
||||
★ hkcOv1QskUFmYveMLB6Hljra ← 토큰은 발급됐는데 세션이 없다
|
||||
★ p5XybeQIYmAs818gO4Vl_5ea ← 토큰은 발급됐는데 세션이 없다
|
||||
|
||||
=== 그 토큰이 지금 실제로 쓰이는가 (마지막 sid 로 확인) ===
|
||||
마지막 sid: 8do0Bw6tkVLDVxgxotE7GosH
|
||||
user_session_id | created_on | last_session_refresh
|
||||
--------------------------+------------+----------------------
|
||||
8do0Bw6tkVLDVxgxotE7GosH | 1788490958 | 1788490958
|
||||
(1 row)
|
||||
|
||||
@@ -1,20 +0,0 @@
|
||||
# A-3 — DB 강제 종료와 데이터 손실 증거
|
||||
|
||||
2026-09-04 12:00–12:05 KST · Keycloak 26.7.0 / PostgreSQL 16
|
||||
해설: [`docs/experiment-a3-database-crash.md`](../../experiment-a3-database-crash.md)
|
||||
|
||||
| 파일 | 무엇을 보여주는가 |
|
||||
|---|---|
|
||||
| `01-crash-injection.txt` | 첫 시도 실패 — 파드 안 백그라운드 루프가 `exec` 종료와 함께 죽어 0건 수집 |
|
||||
| `02-design-check.txt` | **핵심 설계 확인** — 로그인 트랜잭션도 `SET LOCAL synchronous_commit TO OFF` 로 커밋한다 |
|
||||
| `03-loss-measurement.txt` | `--grace-period=0 --force` 주입 |
|
||||
| `04-comparison.txt` | **유실 0건** — 그러나 crash recovery 가 안 돌았다. 죽인 적이 없는 것 |
|
||||
| `05-true-crash.txt` | `kill -9 1` 시도 — **컨테이너 안에서 PID 1 은 SIGKILL 을 무시한다** |
|
||||
| `06-backend-kill-crash.txt` | **성공한 주입** — 백엔드에 SIGKILL → `not properly shut down` / `redo starts` / `redo done` |
|
||||
| `07-loss-result.txt` | **결과: 153건 중 4건 유실.** 토큰은 발급됐는데 세션 행이 없는 sid 목록 |
|
||||
|
||||
## 핵심 세 줄
|
||||
|
||||
1. **로그인도 비동기 커밋이다.** refresh 시각뿐 아니라 **로그인 자체**가 사라질 수 있다.
|
||||
2. **153건 중 4건(약 2.6%) 유실** — 초당 19건 기준 마지막 0.2초 분량, `wal_writer_delay` 기본값과 일치.
|
||||
3. **주입을 세 번 시도해 세 번째에 성공했다.** 앞의 둘은 "손실 0"으로 보였지만 실제로는 크래시가 아니었다.
|
||||
@@ -1,20 +0,0 @@
|
||||
=== A-4 기준선 ===
|
||||
kc-lab-1 Ready true
|
||||
kc-lab-2 Ready <none>
|
||||
|
||||
a2-probe true kc-lab-2
|
||||
keycloak-0 true kc-lab-2
|
||||
keycloak-1 true kc-lab-1
|
||||
postgres-7b474b88c8-2gf27 true kc-lab-2
|
||||
|
||||
=== PVC 가 어느 노드에 묶여 있는가 (재배치 가능성) ===
|
||||
persistentvolumeclaim/postgres-data → kc-lab-2
|
||||
|
||||
=== 서비스 정상 확인 ===
|
||||
https://auth.hyeonworks.com/realms/master HTTP 200
|
||||
|
||||
=== VM 상태 ===
|
||||
--------------------------
|
||||
1 kc-lab-1 running
|
||||
2 kc-lab-2 running
|
||||
|
||||
@@ -1,17 +0,0 @@
|
||||
=== 워커 노드(kc-lab-2) 전원 차단 — virsh destroy 는 종료 신호가 없다 ===
|
||||
차단 시각: 12:07:43
|
||||
Domain 'kc-lab-2' destroyed
|
||||
|
||||
|
||||
+15초 node=Ready | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 000
|
||||
+30초 node=Ready | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 000
|
||||
+45초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+60초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+75초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+90초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+105초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+120초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+135초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+150초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+165초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
+180초 node=NotReady | keycloak-0=Running postgres-7b474b88c8-2gf27=Running | 외부 HTTP 503
|
||||
@@ -1,28 +0,0 @@
|
||||
=== 파드 상태의 진실 — Running 인데 노드가 없다 ===
|
||||
a2-probe Running true kc-lab-2 <none>
|
||||
keycloak-0 Running true kc-lab-2 <none>
|
||||
keycloak-1 Running false kc-lab-1 <none>
|
||||
postgres-7b474b88c8-2gf27 Running true kc-lab-2 <none>
|
||||
|
||||
=== 재배치가 시도되었는가 ===
|
||||
10m Warning Unhealthy pod/keycloak-0 Readiness probe failed: Get "http://10.42.1.67:9000/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
|
||||
3m15s Warning NodeNotReady pod/postgres-7b474b88c8-2gf27 Node is not ready
|
||||
3m15s Warning NodeNotReady pod/keycloak-0 Node is not ready
|
||||
3m15s Warning NodeNotReady pod/a2-probe Node is not ready
|
||||
2m27s Warning Unhealthy pod/keycloak-1 Readiness probe failed: Get "http://10.42.0.35:9000/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
|
||||
2s Warning Unhealthy pod/keycloak-1 Readiness probe failed: HTTP probe failed with statuscode: 503
|
||||
|
||||
=== 노드 taint — 쿠버네티스가 붙인 것 ===
|
||||
node.kubernetes.io/unreachable=:NoSchedule
|
||||
node.kubernetes.io/unreachable=:NoExecute
|
||||
|
||||
=== Prometheus 가 본 것 (kc-lab-1 에 있어 살아남았다) ===
|
||||
up{job=keycloak pod=keycloak-1 } = 1
|
||||
up{job=keycloak pod=keycloak-0 } = 0
|
||||
up{job=kubelet pod=- } = 1
|
||||
up{job=kubelet pod=- } = 0
|
||||
up{job=node-exporter pod=kc-lab-1 } = 1
|
||||
up{job=node-exporter pod=kc-lab-2 } = 0
|
||||
up{job=prometheus pod=- } = 1
|
||||
|
||||
=== 진입점이 처음 40초간 000 이었던 이유 — nginx upstream ===
|
||||
@@ -1,15 +0,0 @@
|
||||
=== nginx 설정 위치 찾기 ===
|
||||
|
||||
=== NoExecute taint 의 tolerationSeconds — 언제 축출되는가 ===
|
||||
node.kubernetes.io/not-ready NoExecute tolerationSeconds=300
|
||||
node.kubernetes.io/unreachable NoExecute tolerationSeconds=300
|
||||
|
||||
=== 5분 축출 시점까지 관찰 ===
|
||||
+210초 a2-probe:Running keycloak-0:Running keycloak-1:Running postgres-7b474b88c8-2gf27:Running
|
||||
+240초 a2-probe:Running keycloak-0:Running keycloak-1:Running postgres-7b474b88c8-2gf27:Running
|
||||
+270초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending
|
||||
+300초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending
|
||||
+330초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending
|
||||
+360초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending
|
||||
+390초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending
|
||||
+420초 a2-probe:Terminating keycloak-0:Terminating keycloak-1:Running postgres-7b474b88c8-2gf27:Terminating postgres-7b474b88c8-9cmsv:Pending
|
||||
@@ -1,18 +0,0 @@
|
||||
=== 새 postgres 가 Pending 인 이유 ===
|
||||
Events:
|
||||
Type Reason Age From Message
|
||||
---- ------ ---- ---- -------
|
||||
Warning FailedScheduling 4m45s default-scheduler 0/2 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
|
||||
|
||||
=== keycloak-0 대체 파드가 안 생기는 이유 (StatefulSet) ===
|
||||
keycloak 2 <none> 1
|
||||
keycloak-0 1/1 Terminating 0 30m
|
||||
keycloak-1 0/1 Running 0 143m
|
||||
|
||||
=== 복구 — 노드 재기동 ===
|
||||
재기동 시각: 12:16:31
|
||||
Domain 'kc-lab-2' started
|
||||
|
||||
+30초 node=Ready | Running 파드 3 개 | 외부 HTTP 503
|
||||
+60초 node=Ready | Running 파드 3 개 | 외부 HTTP 200
|
||||
→ 서비스 복귀
|
||||
@@ -1,19 +0,0 @@
|
||||
=== 복구 확인 ===
|
||||
keycloak-0 1/1 Running 0 68s
|
||||
keycloak-1 1/1 Running 0 144m
|
||||
postgres-7b474b88c8-9cmsv 1/1 Running 0 4m20s
|
||||
|
||||
=== kc-lab-1(k3s server)에 무엇이 있는가 — 이게 곧 영향 범위다 ===
|
||||
keycloak-lab keycloak-1
|
||||
kube-system coredns-54996dc9b4-8k8fj
|
||||
kube-system helm-install-traefik-crd-q29b5
|
||||
kube-system local-path-provisioner-77b9867795-g27z8
|
||||
kube-system metrics-server-6dc596dfb8-7xxq4
|
||||
kube-system svclb-traefik-5eb6a9a1-qwwk5
|
||||
kube-system traefik-5d6fcf895-wpfhr
|
||||
observability grafana-845b5678cf-b6gvc
|
||||
observability node-exporter-9qk9w
|
||||
observability prometheus-6774f94f7c-pzr2t
|
||||
|
||||
=== Traefik replica 수 (진입점의 단일 장애점인가) ===
|
||||
traefik 1 1
|
||||
@@ -1,19 +0,0 @@
|
||||
=== 컨트롤 플레인 노드(kc-lab-1) 전원 차단 ===
|
||||
차단 시각: 12:18:08
|
||||
Domain 'kc-lab-1' destroyed
|
||||
|
||||
|
||||
+20초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
+40초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
+60초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
+80초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
+100초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
+120초 외부 auth=000 grafana=502 | kubectl: Unable to connect to the server: dial tcp
|
||||
+140초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
+160초 외부 auth=000 grafana=000 | kubectl: Unable to connect to the server: dial tcp
|
||||
|
||||
=== 살아 있는 노드에서 직접 확인 — 워크로드는 도는가 ===
|
||||
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
|
||||
e5f777900b762 60e153026e8f5 4 minutes ago Running keycloak 0 640d4dafaefb3 keycloak-0 keycloak-lab
|
||||
|
||||
6
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user