Compare commits

..
Author SHA1 Message Date
DongHyeonkaandClaude Opus 5 f45a2a2aaa docs: B-5 — the pod stays Ready while every request hangs
Stopping Redis returns HTTP 000 rather than an error because the client waits on reconnect, and the pod keeps serving traffic because the redis health indicator is not in the readiness group even though /actuator/health returns 503. That is the mirror image of A-2, where Keycloak put its database check in readiness and the pods left the Service.

Turning on AOF with config set created the appendonlydir and still lost everything on pod deletion, because /data was the container filesystem; adding a PVC makes the same setting work. Volume first, persistence setting second.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 14:29:17 +09:00
DongHyeonkaandClaude Opus 5 7dc0a3e5da docs: B-4 — the edge does not overwrite the headers it never sets
Two headers of the same name both arrive rather than one overwriting the other, because nginx only replaces headers it sets with proxy_set_header. A comma inside a role name is indistinguishable from the delimiter, and the size limit is a cliff: Tomcat returns 400 around 8KB and the connection dies around 16KB, so the same cause produces two different-looking failures.

Forged identity headers reach the upstream untouched while the JWT-protected paths return 401, which is Q4's own point that a header-fed upstream has nothing to verify against. By Q4's checklist that answer alone points at the BFF structure.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 14:23:41 +09:00
DongHyeonkaandClaude Opus 5 b16e1dccf7 docs: B-3 — concurrent refresh does not lose a race, it destroys the session
Five simultaneous refreshes with one token return a single 200, and that winner's new token is already dead. Reuse detection removes the client session while the user session stays, which is why the other responses read Session doesn't have required client rather than a reuse error.

Comparing policies shows rotation off passes all five and keeps the session, while raising refreshTokenMaxReuse to one still destroys it. Since no retry can recover a removed client session, Q2's own criterion resolves to a lock, and a database row lock is the natural place because its lifetime is tied to the connection.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 14:20:04 +09:00
DongHyeonkaandClaude Opus 5 711878379c docs: B-2 — sharing the stores fixes one problem and exposes three more
Moving the authorized client to JdbcOAuth2AuthorizedClientService makes tokens work across replicas, so the session-in-Redis plus tokens-in-PostgreSQL split holds. The table then shows what sharing cannot fix: the primary key is (client_registration_id, principal_name) with no session in it, so a second login for the same user updates the same row rather than adding one.

The refresh token sits in bytea as the raw JWT, readable with convert_from, and logout clears only the Redis session while the plaintext token row and the Keycloak SSO session both survive. The schema itself failed silently first because the default DDL uses blob, which PostgreSQL does not have, and continue-on-error swallowed it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 14:15:05 +09:00
DongHyeonkaandClaude Opus 5 f2595f748f docs: B-1 — Redis moves the session and leaves the tokens behind
Adding Spring Session Redis grows the context by 81 beans and swaps sessionRepository for RedisSessionRepository, while authorizedClientService stays InMemoryOAuth2AuthorizedClientService. The user then reads as logged in with principal labuser while accessTokenStoredOnServer is false, which is worse than being logged out.

Redis holds only the security context, serialized with Java native serialization, and the refresh token is not there to encrypt in the first place. Three problems on the way: Kubernetes service links overwrote REDIS_PORT with a tcp:// URL, the tests tried to reach Redis, and the resource server was never deployed so a DNS failure looked like a token failure.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 14:05:02 +09:00
DongHyeonkaandClaude Opus 5 e62bbb4df0 docs: B-0 — deploy the BFF and read what autoconfiguration actually chose
The authorized client repository is AuthenticatedPrincipalOAuth2AuthorizedClientRepository, keyed by principal with no session id in it, which is the mechanism behind the sharing problem Q1 and Q3 describe. Sharing a store does not fix a lookup key.

Five problems on the way in: only build output was committed under bff/, a duplicate YAML key broke the image build and was invisible until the full log was captured, env placeholders without defaults broke the tests, actuator was behind the login redirect so a 200 was the login page, and the 117KB beans response failed through the proxy.

Deploying two replicas made the login itself fail before any experiment started, because the authorization request lives in per-instance memory and the callback lands elsewhere.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 13:48:33 +09:00
DongHyeonkaandClaude Opus 5 8f6d67df35 docs: A-7 — three results invert when persistent sessions are turned off
Disabling persistent-user-sessions moves the session from PostgreSQL into the cluster, and the A-1 and A-8 outcomes flip to 400 Session not active while a new login during database loss starts working. The control group in each case still returns 200, so the injections cut only what they were meant to cut.

This is the pair that makes the A layer legible: the conventional wisdom that sessions ride TCP 7800 is correct for Keycloak 24 and earlier, and the mistake is applying it to 26 without checking the version.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 13:33:33 +09:00
DongHyeonkaandClaude Opus 5 114d21aebe docs: A-8 — a rolling restart keeps every session and drops only the cache
Nine samples through the restart all returned 200, the refresh token issued beforehand still works, and the session count is unchanged at 151 while both caches reset to zero. The updated last_session_refresh proves the write path recovered too, not just the response code.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 13:20:56 +09:00
DongHyeonkaandClaude Opus 5 dba0c3975c docs: A-6 — 200ms of network delay becomes 22 seconds of user latency
Nine database round trips per login multiply the injected delay to 1.9 seconds, and connection pool queueing multiplies it again under twenty concurrent requests. The readiness probe joins the same queue and times out, so the node leaves the load balancer and pushes its load onto the one still standing.

Two injections missed first: the guest interface is enp1s0 rather than eth0, and a filter on it can never match a pod IP because flannel has already encapsulated the packet. The delay has to go on flannel.1, before encapsulation.

The predicted rise in optimistic lock conflicts did not happen, because logins insert new rows rather than contending for one. That belongs to B-3.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 13:18:10 +09:00
DongHyeonkaandClaude Opus 5 2fa77a899e docs: A-5 — a one-way block heals itself and a full split still leaves one node serving
Three injections failed first: kube-router keeps reinserting its chain above a hand-placed FORWARD rule, the JGroups connection direction had reversed since A-1, and only the raw table runs ahead of conntrack. Each failure looked like nothing happening.

Blocking one direction never partitioned the cluster because JGroups reconnected the other way before failure detection fired. Blocking both produced a real split brain with two coordinators in JGROUPS_PING, yet only the non-coordinator node reported DOWN, so the Service kept an endpoint and the front door stayed at 200. That answers the question A-1 left open.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 13:12:04 +09:00
DongHyeonkaandClaude Opus 5 d0666c5ba0 docs: A-4 — a dead pod reports healthier than a live one
Kubernetes keeps calling the node Ready for forty seconds while users already see failures, and the pod on the powered-off machine stays ready=true because its kubelet can no longer contradict itself. Eviction waits another five minutes, then the StatefulSet refuses to recreate its pod and the replacement Deployment pod cannot schedule because the local-path volume is pinned to the dead node.

Killing the server node instead shows the opposite shape: containerd keeps the workload running while the API server, Traefik and the observability stack disappear, so the outage is the missing path rather than the missing application. Traefik at one replica is the ingress single point of failure.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 12:25:30 +09:00
DongHyeonkaandClaude Opus 5 99b689e715 docs: A-3 — four logins returned tokens for sessions the crash erased
Keycloak commits the login INSERT with synchronous_commit off, so a crash loses whole sessions and not just refresh timestamps. Measured 4 of 153 lost, matching the default wal_writer_delay window.

Two injections failed silently first: --grace-period=0 --force lets the container runtime send SIGTERM so PostgreSQL flushes and shuts down cleanly, and SIGKILL to PID 1 from inside its own namespace is ignored by the kernel. Killing a backend makes the postmaster reinitialize, which is a real crash recovery.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 12:05:08 +09:00
DongHyeonkaandClaude Opus 5 4177fb6a48 docs: A-2 — losing the database takes every node down while up stays 1
Both pods go NotReady, the Service endpoint list empties and the front door returns 503, so adding Keycloak replicas buys nothing against database loss. The node holding the session in cache fails too, because a refresh writes LAST_SESSION_REFRESH. Recovery was automatic in about fifteen seconds with no restart, which is what readiness rather than liveness buys.

The observability finding matters as much: up stayed at 1 through a total outage, so alerting on it would have caught nothing. kube-state-metrics is missing and pod readiness is therefore not recorded as a metric.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 11:57:55 +09:00
DongHyeonkaandClaude Opus 5 2a98ef1090 docs: A-1 — sessions survive a JGroups partition but logout invalidation does not
Cutting TCP 7800 leaves cross-node refresh working (200), confirming sessions travel through PostgreSQL rather than the cluster transport. Logout is the opposite: the database row is deleted but the other node answers from its stale local cache, so the A-0 conclusion that invalidation rides the database is corrected here.

Two things the plan did not anticipate: a NetworkPolicy cannot sever an established connection because conntrack accepts it before policy evaluation, and Keycloak reports the partition through its readiness probe so the split node removes itself from the Service.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 11:52:49 +09:00
DongHyeonkaandClaude Opus 5 9c3cde457e docs: close nine gaps found by re-reading every open question in full
Cross-checked each question's 남은 미지수, 다음 검증 and 제약 against the plan item by item. Adds B-0 (autoconfiguration actually chosen), B-6 (encryption key rotation) and B-7 (oauth2-proxy cookie secret rotation) as new experiments, plus lock-holder death, rotation-disabled comparison, partial-logout recovery, store latency and the Q4 design checklist. Restores the Redis persistence comparison and records the correct index URL.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 11:16:59 +09:00
DongHyeonkaandClaude Opus 5 f1e8c35805 docs: plan all remaining experiments with architecture, injection points and predictions
Twenty experiments across four layers, each with a topology diagram marking where the fault goes in, the metrics to watch, a falsifiable prediction written before the run, and a pass/fail rule.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 10:50:23 +09:00
DongHyeonkaandClaude Opus 5 1de6108157 docs: add the prerequisite knowledge this lab assumes
Builds up from HTTP statelessness to why session storage location determines the operational response, so the measurements have context to land in.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 10:24:21 +09:00
DongHyeonkaandClaude Opus 5 22d873eb4f docs: capture the SQL the other node actually runs, and correct the replication claim
PostgreSQL statement logging shows keycloak-1 reading and updating the session created on keycloak-0. The same transaction reveals optimistic locking via VERSION, SKIP LOCKED, and synchronous_commit turned off. Fixes the earlier concept note that credited Infinispan with cross-node propagation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 10:14:45 +09:00
DongHyeonkaandClaude Opus 5 e5ebaeb623 docs: prove sessions are shared by PostgreSQL, not Infinispan replication
Experiment 0 with three probes: cross-node refresh/logout, cache counter deltas around a single login, and cache entry ownership. Each node caches only what it handled; cache totals sum exactly to the database count.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 09:58:10 +09:00
DongHyeonkaandClaude Opus 5 006da7d490 docs: record observability setup and add concept layers 10-13
Adds Kubernetes resources (StatefulSet, PVC, Secret, RBAC, placement), Keycloak clustering internals (Infinispan, JGroups), Prometheus concepts and virtualization operations.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 09:21:45 +09:00
DongHyeonkaandClaude Opus 5 d5cc2b55a9 fix: grant nodes/proxy so kubelet metrics can be scraped
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 09:14:51 +09:00
DongHyeonkaandClaude Opus 5 0bb0e0ac49 feat: add Prometheus, node-exporter and Grafana for fault-injection observability
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 09:13:41 +09:00
DongHyeonkaandClaude Opus 5 33878e8880 docs: record multi-node cluster setup, rationale and formation evidence
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 17:26:45 +09:00
DongHyeonkaandClaude Opus 5 6dce35ec83 feat: deploy Keycloak multi-node cluster with PostgreSQL
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 17:17:25 +09:00
DongHyeonkaandClaude Opus 5 001efd624a docs: correct the memory analysis to distinguish host and guest headroom
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 17:02:11 +09:00
DongHyeonkaandClaude Opus 5 d6f8b9f8b3 docs: restructure roadmap into A/B/operations layers and add five operational items
Adds backup rehearsal, version upgrade, observability, secret management and certificate renewal. Corrects the experiment order so the refresh-token contention test runs after the shared store exists.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 16:56:13 +09:00
DongHyeonkaandClaude Opus 5 df4d3b4345 docs: map published open questions to lab coverage and fix experiment order
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 16:44:12 +09:00
DongHyeonkaandClaude Opus 5 61ba5db259 docs: add lab operations guide with tooling, commands and drills
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 16:39:45 +09:00
DongHyeonkaandClaude Opus 5 bc784fcd6e docs: surface the spoofing bypass finding in the roadmap
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 16:24:52 +09:00
DongHyeonkaandClaude Opus 5 ddcb1c08e6 docs: record the session store lab roadmap and progress
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 16:20:38 +09:00
DongHyeonkaandClaude Opus 5 e1ba9c5626 docs: record proxy-bypass closure with before and after evidence
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 16:17:24 +09:00
DongHyeonkaandClaude Opus 5 3af52bb66a feat: narrow Traefik trusted range and restrict echo ingress to Traefik
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 16:15:28 +09:00
DongHyeonkaandClaude Opus 5 98874b0c6c docs: retake staged evidence screenshots with indented output
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 15:52:47 +09:00
DongHyeonkaandClaude Opus 5 b708c8d503 feat: indent echo responses for readable evidence
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 15:48:57 +09:00
DongHyeonkaandClaude Opus 5 7737787937 docs: record the staged fix and post-fix evidence
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 15:33:40 +09:00
DongHyeonkaandClaude Opus 5 1c1b86e849 feat: let the app interpret forwarded headers
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 15:30:31 +09:00
DongHyeonkaandClaude Opus 5 2294c52095 feat: make Traefik trust forwarded headers from the host nginx
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 15:29:18 +09:00
DongHyeonkaandClaude Opus 5 69d4502757 docs: collect evidence for the two-hop header contract failure
Control experiment isolates two independent causes: the nginx 443 block still emits X-Forwarded-Proto http, and Traefik rewrites forwarded headers regardless.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 15:04:38 +09:00
DongHyeonkaandClaude Opus 5 ae1f391598 docs: expand two-hop header contract with switch locations and per-pattern impact
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 14:57:08 +09:00
DongHyeonkaandClaude Opus 5 844d6f1d33 docs: measure and diagnose the two-hop proxy header contract
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 14:35:27 +09:00
DongHyeonkaandClaude Opus 5 a831792c5c docs: map deploy/ assets and record why unapplied configs are kept
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 14:18:08 +09:00
DongHyeonkaandClaude Opus 5 bcfdeb93ee docs: explain every setting used in the echo manifest
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 14:11:18 +09:00
DongHyeonkaandClaude Opus 5 deae8966b8 docs: record why Docker must not be installed on the lab host
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 13:53:36 +09:00
DongHyeonkaandClaude Opus 5 6c90468c5a feat: report pod address from echo endpoint
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 13:44:29 +09:00
DongHyeonka 51bb055d61 feat: 2홉 구성 진행 2026-09-03 11:46:56 +09:00
DongHyeonka c22b217fbd chore: 메타데이터 gitignore 2026-09-02 14:44:06 +09:00
259 changed files with 19485 additions and 734 deletions
-2
View File
@@ -10,8 +10,6 @@ POSTGRES_PASSWORD=change-me-postgres-password
TOKEN_MEDIATING_CLIENT_SECRET=change-me-token-mediating-client-secret
BFF_CLIENT_SECRET=change-me-bff-client-secret
EDGE_PROXY_CLIENT_SECRET=change-me-edge-proxy-client-secret
OAUTH2_PROXY_COOKIE_SECRET=generate-a-base64-encoded-32-byte-secret
INTERNAL_AUTH_TOKEN=generate-a-long-random-edge-to-backend-token
MOCK_GOOGLE_BROKER_CLIENT_SECRET=change-me-mock-google-broker-client-secret
ADMIN_USER_PASSWORD=change-me-admin-user-password
REGULAR_USER_PASSWORD=change-me-regular-user-password
+7 -2
View File
@@ -4,10 +4,15 @@
*.iml
backend/target/
**/node_modules/
frontend/dist/
build/
e2e/node_modules/
google-e2e/node_modules/
frontend/node_modules/
frontend/dist/
bff/target
token-mediator/target
# lab cloud-init contains a console password; keep the filled copy local
deploy/lab/cloud-init/kc-lab.yaml
deploy/lab/cloud-init/kc-lab-*.yaml
@@ -0,0 +1,22 @@
[ 1289ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/login:0
[ 1417ms] [VERBOSE] [DOM] Input elements should have autocomplete attributes (suggested: "username"): (More info: https://goo.gl/9p2vKq) %o @ https://app2.hyeonworks.com/login:0
[ 9024ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 9130ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/:0
[ 9456ms] [WARNING] Deprecation warning: value provided is not in a recognized RFC2822 or ISO format. moment construction falls back to js Date(), which is not reliable across all browsers and versions. Non RFC2822/ISO date formats are discouraged. Please refer to http://momentjs.com/guides/#/warnings/js-date/ for more info.
Arguments:
[0] _isAMomentObject: true, _isUTC: false, _useUTC: false, _l: undefined, _i: Thu, 27 Aug 2026 13:03:49, _f: undefined, _strict: undefined, _locale: [object Object]
Error
at a.createFromInputFallback (https://app2.hyeonworks.com/public/build/6029.0549a3fcb50e73c4b256.js:624:3)
at an (https://app2.hyeonworks.com/public/build/6029.0549a3fcb50e73c4b256.js:624:25647)
at un (https://app2.hyeonworks.com/public/build/6029.0549a3fcb50e73c4b256.js:624:29355)
at aa (https://app2.hyeonworks.com/public/build/6029.0549a3fcb50e73c4b256.js:624:29221)
at on (https://app2.hyeonworks.com/public/build/6029.0549a3fcb50e73c4b256.js:624:28938)
at sa (https://app2.hyeonworks.com/public/build/6029.0549a3fcb50e73c4b256.js:624:29715)
at A (https://app2.hyeonworks.com/public/build/6029.0549a3fcb50e73c4b256.js:624:29748)
at a (https://app2.hyeonworks.com/public/build/6029.0549a3fcb50e73c4b256.js:621:89)
at f (https://app2.hyeonworks.com/public/build/3719.c065b2e146c4c8347d51.js:1:4635)
at u (https://app2.hyeonworks.com/public/build/322.177b4bb01c5d74f9b28f.js:2473:47448) @ https://app2.hyeonworks.com/public/build/6029.0549a3fcb50e73c4b256.js:620
[ 9605ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 10620ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 11527ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 13875ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,7 @@
[ 144ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 153ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore:0
[ 1077ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 2101ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 2922ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 7323ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 9370ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,8 @@
[ 271ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22vendor_statistics_approximate_entries_unique%7Bcache%3D%5C%22sessions%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bpod%7D%7D%20on%20%7B%7Bnode%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-15m%22%2C%22to%22%3A%22now%22%7D%7D%7D:0
[ 346ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1512ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 2433ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 6941ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 13188ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 21578ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 25998ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,7 @@
[ 766ms] [WARNING] An iframe which has both allow-scripts and allow-same-origin for its sandbox attribute can escape its sandboxing. @ https://auth.hyeonworks.com/realms/master/protocol/openid-connect/3p-cookies/step1.html:0
[ 781ms] [WARNING] An iframe which has both allow-scripts and allow-same-origin for its sandbox attribute can escape its sandboxing. @ https://auth.hyeonworks.com/realms/master/protocol/openid-connect/3p-cookies/step2.html:0
[ 17929ms] [WARNING] An iframe which has both allow-scripts and allow-same-origin for its sandbox attribute can escape its sandboxing. @ https://auth.hyeonworks.com/realms/master/protocol/openid-connect/3p-cookies/step1.html:0
[ 17949ms] [WARNING] An iframe which has both allow-scripts and allow-same-origin for its sandbox attribute can escape its sandboxing. @ https://auth.hyeonworks.com/realms/master/protocol/openid-connect/3p-cookies/step2.html:0
[ 17981ms] [WARNING] An iframe which has both allow-scripts and allow-same-origin for its sandbox attribute can escape its sandboxing. @ https://auth.hyeonworks.com/realms/master/protocol/openid-connect/login-status-iframe.html:0
[ 18447ms] [WARNING] For accessibility reasons an aria-label should be specified on nav groups if a title isn't @ https://auth.hyeonworks.com/resources/9v5yc/admin/keycloak.v2/assets/main-BbID33M6.js:7
[ 18462ms] [WARNING] For accessibility reasons an aria-label should be specified on nav groups if a title isn't @ https://auth.hyeonworks.com/resources/9v5yc/admin/keycloak.v2/assets/main-BbID33M6.js:7
@@ -0,0 +1,2 @@
[ 75ms] [ERROR] Failed to load resource: the server responded with a status of 404 (Not Found) @ https://hyeonworks.com/questions:0
[ 100ms] [ERROR] Failed to load resource: the server responded with a status of 404 (Not Found) @ https://hyeonworks.com/favicon.ico:0
@@ -0,0 +1 @@
[ 149ms] [ERROR] Failed to load resource: the server responded with a status of 401 (Unauthorized) @ https://hyeonworks.com/api/v1/studio/session:0
@@ -0,0 +1 @@
[ 103ms] [ERROR] Failed to load resource: the server responded with a status of 401 (Unauthorized) @ https://hyeonworks.com/api/v1/studio/session:0
@@ -0,0 +1 @@
[ 95ms] [ERROR] Failed to load resource: the server responded with a status of 401 (Unauthorized) @ https://hyeonworks.com/api/v1/studio/session:0
@@ -0,0 +1 @@
[ 132ms] [ERROR] Failed to load resource: the server responded with a status of 401 (Unauthorized) @ https://hyeonworks.com/api/v1/studio/session:0
@@ -0,0 +1 @@
[ 239ms] [ERROR] Failed to load resource: the server responded with a status of 401 (Unauthorized) @ https://hyeonworks.com/api/v1/studio/session:0
@@ -0,0 +1 @@
[ 105ms] [ERROR] Failed to load resource: the server responded with a status of 401 (Unauthorized) @ https://hyeonworks.com/api/v1/studio/session:0
@@ -0,0 +1 @@
[ 108ms] [ERROR] Failed to load resource: the server responded with a status of 401 (Unauthorized) @ https://hyeonworks.com/api/v1/studio/session:0
@@ -0,0 +1,41 @@
[ 344ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 869ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&panes=%7B%22cf1%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22vendor_cluster_size%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bpod%7D%7D+on+%7B%7Bnode%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-45m%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1:0
[ 1014ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 2039ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 4085ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 7289ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 15556ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 26818ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 32659ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 35930ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 53849ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 61722ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 79450ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 84571ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 87233ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 92665ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 113244ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 119181ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 137390ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 157071ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 166091ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 182011ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 188613ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 206228ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 218561ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 229607ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 235818ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 237158ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 252095ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 261118ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 273809ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 280227ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 292650ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 306477ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 309957ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 311984ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 325683ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 344586ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 357576ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 359294ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 364740ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,129 @@
[ 615ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&panes=%7B%22te0%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22up%7Bjob%3D%5C%22keycloak%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22up+%E2%80%94+%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-20m%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1:0
[ 929ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 2566ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 5541ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 7990ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 12402ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 17640ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 20285ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 28277ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 34341ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 34985ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 50081ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 65649ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 75046ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 75890ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 96067ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 113467ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 121972ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 139681ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 150971ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 161211ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 180744ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 191401ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 210668ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 229082ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 239345ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 252119ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 266045ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 276705ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 289911ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 296875ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 315099ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 325431ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 328418ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 343774ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 347054ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 352405ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 361291ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 370639ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 374286ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 377413ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 394362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 405423ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 425476ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 434299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 441775ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 450786ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 456829ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 468812ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 478913ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 491241ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 495228ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 502295ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 511625ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 524619ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 533050ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 553189ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 571699ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 575413ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 588005ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 599161ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 608816ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 619031ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 626612ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 639003ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 655077ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 674532ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 677197ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 696027ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 709512ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 727505ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 747139ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 754565ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 773682ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 784674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 787399ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 802845ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 825589ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 832768ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 842043ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 852717ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 871748ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 876571ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 894491ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 895515ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 904310ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 916681ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 934752ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 941175ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 946004ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 960849ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 962180ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 984446ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 997579ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1010105ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1027190ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1043388ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1061311ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1066040ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1080349ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1092336ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1104312ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1109362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1127047ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1134728ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1154899ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1161253ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1174971ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1184491ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1185340ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1189111ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1196984ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1204794ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1247961ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1286322ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1319329ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1362771ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1396881ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1434184ms] [ERROR] Failed to load resource: the server responded with a status of 502 () @ https://app2.hyeonworks.com/api/user/auth-tokens/rotate:0
[ 1443506ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1465016ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1513337ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1556906ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1562963ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1572896ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 502 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1576390ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 503 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1593794ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1600651ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1616424ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,125 @@
[ 268ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/login:0
[ 379ms] [VERBOSE] [DOM] Input elements should have autocomplete attributes (suggested: "username"): (More info: https://goo.gl/9p2vKq) %o @ https://app2.hyeonworks.com/login:0
[ 10453ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 10532ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&panes=%7B%22ich%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22up%7Bjob%3D%7E%5C%22keycloak%7Cnode-exporter%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bjob%7D%7D+%E2%80%94+%7B%7Bpod%7D%7D%7B%7Bnode%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-55m%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1:0
[ 11687ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 13737ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 15113ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 20185ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 27259ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 40154ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 41870ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 50493ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 54082ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 60121ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 74362ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 83269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 95349ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 98008ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 104458ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 123816ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 142764ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 157198ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 175220ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 182187ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 183411ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 190581ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 199499ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 209995ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 214849ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 233084ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 242910ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 257975ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 277220ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 290806ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 292674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 300013ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 309267ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 323998ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 328620ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 339787ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 355959ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 361079ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 369272ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 381866ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 397120ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 400712ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 412579ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 431630ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 440746ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 447809ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 461741ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 475459ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 482723ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 494299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 513174ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 521541ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 532145ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 551129ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 556250ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 574381ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 586190ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 595782ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 610555ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 630695ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 633257ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 652194ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 671361ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 677606ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 692444ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 696949ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 700122ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 709954ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 723992ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 743173ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 745075ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 753299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 766683ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 767855ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 787261ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 789725ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 800680ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 811324ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 814096ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 829045ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 848506ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 857716ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 866212ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 873174ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 876033ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 895198ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 898884ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 918642ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 930133ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 934008ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 948541ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 956860ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 959397ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 968406ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 971788ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 982746ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 997394ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1005078ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1009269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1021453ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1026674ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1042137ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1057497ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1059851ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1061855ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1071317ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1078179ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1084940ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1093950ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1112279ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1124360ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1139110ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1157981ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1163071ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1179032ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1181195ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1191955ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1201472ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1218261ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1221169ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1229430ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,50 @@
[ 199ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 297ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22vendor_cluster_size%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-25m%22%2C%22to%22%3A%22now%22%7D%7D%7D:0
[ 1000ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 2023ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 3559ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 7749ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 16364ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 25781ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 42158ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 59672ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 77090ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 87426ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 89952ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 104743ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error in connection establishment: net::ERR_NAME_NOT_RESOLVED @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1573121ms] [WARNING] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: WebSocket is closed before the connection is established. @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1584038ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1601626ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1613280ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1630173ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1634776ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1654205ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1661576ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1665269ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1682669ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1689435ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1704385ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1706314ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1724551ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1743701ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1758339ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1769299ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1789478ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1795415ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1815382ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1820392ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1822445ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1839035ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1840162ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1847739ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1861419ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1880381ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1893315ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1911434ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1922701ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1933868ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: HTTP Authentication failed; no valid credentials available @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1944951ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1948001ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1952503ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1953320ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1955369ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,20 @@
[ 585ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1132ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&panes=%7B%22o68%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22agroal_blocking_time_max_milliseconds%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22blocking+max+-+%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-30m%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1:0
[ 1555ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 3604ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 6373ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 9135ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 19066ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 24595ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 47125ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 52154ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 67608ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 86965ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 108668ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 129158ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 137544ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 152699ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 159898ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 173491ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 187925ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 197824ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1,143 @@
[ 635ms] [WARNING] <meta name="apple-mobile-web-app-capable" content="yes"> is deprecated. Please include <meta name="mobile-web-app-capable" content="yes"> @ https://app2.hyeonworks.com/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22PBFA97CFB590B2093%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22vendor_statistics_approximate_entries_unique%7Bcache%3D%5C%22sessions%5C%22%7D%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%2C%7B%22refId%22%3A%22B%22%2C%22expr%22%3A%22vendor_cluster_size%22%2C%22range%22%3Atrue%2C%22instant%22%3Afalse%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22cluster_size%20%7B%7Bpod%7D%7D%22%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22PBFA97CFB590B2093%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-15m%22%2C%22to%22%3A%22now%22%7D%7D%7D:0
[ 686ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1451ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 3030ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 4308ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 5296ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 7691ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 18337ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 20184ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 32473ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 50697ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 65961ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 85590ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 99644ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 115524ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 130773ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 134354ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 142549ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 151361ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 155246ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 165901ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 169379ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 180952ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 190372ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 193657ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 205737ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 220582ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 225496ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 240339ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 244029ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 264211ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 273424ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 284893ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 292571ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 305778ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 306804ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 324378ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 335169ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 337968ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 352967ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 364258ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 379409ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 399590ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 403681ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 406340ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 421033ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 428667ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 429483ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 437573ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 442286ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 445865ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 456312ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 464043ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 480382ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 491438ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 494608ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 508231ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 528305ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 529632ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 538033ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 550079ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 566347ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 586157ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 603877ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 615779ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 624657ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 629879ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 637255ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 643298ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 650047ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 668080ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 669816ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 675422ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 677001ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 686611ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 692655ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 713114ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 723787ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 730443ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 751427ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 757583ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 776007ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 792298ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 803509ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 822917ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 827374ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 847278ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 867460ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 875338ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 886296ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 900938ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 912511ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 917067ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 924405ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 930435ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 942211ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 953146ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 957153ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 969140ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 984088ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1003544ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1014908ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1020851ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1034668ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1040406ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1056887ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1062828ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1073375ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1085052ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1102031ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1110039ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1129341ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1132300ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1133670ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1142705ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1160906ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1172400ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1190482ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1196673ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1213666ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1225136ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1228310ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1242751ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1243874ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1256126ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1274444ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1288526ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1307058ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1320986ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1325134ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1335974ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1344832ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1354780ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1364165ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1383346ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1399019ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1400207ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1413875ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1423809ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1427502ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1430773ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1436408ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
[ 1447464ms] [ERROR] WebSocket connection to 'wss://app2.hyeonworks.com/api/live/ws' failed: Error during WebSocket handshake: Unexpected response code: 400 @ https://app2.hyeonworks.com/public/build/1518.a3f1f690c084a37f01c7.js:362
@@ -0,0 +1 @@
[ 207ms] [ERROR] Failed to load resource: the server responded with a status of 404 () @ https://app1.hyeonworks.com/favicon.ico:0
@@ -0,0 +1 @@
[ 236ms] [ERROR] Failed to load resource: the server responded with a status of 500 () @ https://app1.hyeonworks.com/bff/api/me:0
@@ -0,0 +1,2 @@
[ 7292ms] [ERROR] Access to fetch at 'https://auth.hyeonworks.com/realms/keycloak-patterns/protocol/openid-connect/auth?response_type=code&client_id=bff-confidential&scope=openid%20profile%20email&state=WWc76H7TY73Fbsdc41B2nRD5exkXqHcohLh4WdJB4AA%3D&redirect_uri=https://app1.hyeonworks.com/login/oauth2/code/keycloak&nonce=A4TXweuKS4Y5HdZ63rLJUez1ZOyI2em6zs3OIfTXLFo&code_challenge=wPr8PXG0lcUvie7Wo91YrVMhOUYq0KtEU4PxVJ0_CWA&code_challenge_method=S256' (redirected from 'https://app1.hyeonworks.com/bff/api/me') from origin 'https://app1.hyeonworks.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. @ https://app1.hyeonworks.com/bff/token-boundary:0
[ 7293ms] [ERROR] Failed to load resource: net::ERR_FAILED @ https://auth.hyeonworks.com/realms/keycloak-patterns/protocol/openid-connect/auth?response_type=code&client_id=bff-confidential&scope=openid%20profile%20email&state=WWc76H7TY73Fbsdc41B2nRD5exkXqHcohLh4WdJB4AA%3D&redirect_uri=https://app1.hyeonworks.com/login/oauth2/code/keycloak&nonce=A4TXweuKS4Y5HdZ63rLJUez1ZOyI2em6zs3OIfTXLFo&code_challenge=wPr8PXG0lcUvie7Wo91YrVMhOUYq0KtEU4PxVJ0_CWA&code_challenge_method=S256:0
@@ -0,0 +1,2 @@
[ 6726ms] [ERROR] Access to fetch at 'https://auth.hyeonworks.com/realms/keycloak-patterns/protocol/openid-connect/auth?response_type=code&client_id=bff-confidential&scope=openid%20profile%20email&state=GGupuPr3ZklKp8ah99r7h7mNHEq9yTsEWZr85WyXevE%3D&redirect_uri=https://app1.hyeonworks.com/login/oauth2/code/keycloak&nonce=rPVvEOvG7rzssAjR7pP67qNvoY2W6ZVpIpKYz1LGoU8&code_challenge=qoKRLRrzB7z9CU_rlaAxJ7UYcRZswyqmDi8PgxmaWM0&code_challenge_method=S256' (redirected from 'https://app1.hyeonworks.com/bff/api/me') from origin 'https://app1.hyeonworks.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. @ https://app1.hyeonworks.com/:0
[ 6726ms] [ERROR] Failed to load resource: net::ERR_FAILED @ https://auth.hyeonworks.com/realms/keycloak-patterns/protocol/openid-connect/auth?response_type=code&client_id=bff-confidential&scope=openid%20profile%20email&state=GGupuPr3ZklKp8ah99r7h7mNHEq9yTsEWZr85WyXevE%3D&redirect_uri=https://app1.hyeonworks.com/login/oauth2/code/keycloak&nonce=rPVvEOvG7rzssAjR7pP67qNvoY2W6ZVpIpKYz1LGoU8&code_challenge=qoKRLRrzB7z9CU_rlaAxJ7UYcRZswyqmDi8PgxmaWM0&code_challenge_method=S256:0
@@ -0,0 +1,2 @@
[ 6898ms] [ERROR] Failed to load resource: the server responded with a status of 403 () @ https://app1.hyeonworks.com/logout:0
[ 23950ms] [ERROR] Failed to load resource: the server responded with a status of 403 () @ https://app1.hyeonworks.com/logout:0
@@ -0,0 +1,2 @@
- main [ref=e7]:
- status "Loading" [ref=e10]
@@ -0,0 +1 @@
- main [ref=f3e7]
@@ -0,0 +1,2 @@
- main [ref=f6e7]:
- status "Loading" [ref=f6e10]
@@ -0,0 +1,104 @@
- generic [ref=f9e4]:
- link "Skip to main content" [ref=f9e5] [cursor=pointer]:
- /url: "#pageContent"
- banner [ref=f9e7]:
- generic [ref=f9e8]:
- link [ref=f9e10] [cursor=pointer]:
- /url: /
- img "Grafana" [ref=f9e11]
- generic [ref=f9e14]:
- button "Search or jump to..." [ref=f9e18] [cursor=pointer]
- generic [ref=f9e19]: ctrl+k
- generic [ref=f9e23]:
- button "New" [ref=f9e24] [cursor=pointer]
- button "Help" [ref=f9e30] [cursor=pointer]
- button "News" [ref=f9e33] [cursor=pointer]
- button "Profile" [ref=f9e36] [cursor=pointer]:
- img "User avatar" [ref=f9e37]
- generic [ref=f9e38]:
- button "Open menu" [ref=f9e40] [cursor=pointer]
- navigation "Breadcrumbs" [ref=f9e43]:
- list [ref=f9e44]:
- listitem [ref=f9e45]:
- link "Home" [ref=f9e46] [cursor=pointer]:
- /url: /
- listitem [ref=f9e50]:
- link "Explore" [ref=f9e51] [cursor=pointer]:
- /url: /explore
- listitem [ref=f9e55]:
- generic "Prometheus" [ref=f9e56]
- generic [ref=f9e57]:
- button "Show more items" [ref=f9e60] [cursor=pointer]
- button "Toggle top search bar" [ref=f9e64] [cursor=pointer]
- main [ref=f9e70]:
- generic [ref=f9e72]:
- heading "Explore" [level=1] [ref=f9e73]
- generic [ref=f9e78]:
- navigation "Explore toolbar" [ref=f9e80]:
- navigation "Search links" [ref=f9e82]:
- generic [ref=f9e83]:
- button "Content outline" [expanded] [ref=f9e85] [cursor=pointer]:
- generic [ref=f9e88]: Outline
- generic [ref=f9e93] [cursor=pointer]:
- img "Prometheus logo" [ref=f9e95]
- textbox "Select a data source" [ref=f9e96]:
- /placeholder: ""
- button "Show more items" [ref=f9e102] [cursor=pointer]
- generic [ref=f9e106]:
- generic [ref=f9e110]:
- button "Collapse outline" [expanded] [ref=f9e112] [cursor=pointer]:
- img "arrow-from-right" [ref=f9e113]
- button "Queries" [ref=f9e116] [cursor=pointer]:
- img "arrow" [ref=f9e117]
- generic [ref=f9e124]:
- generic [ref=f9e126]:
- generic "Query editor row" [ref=f9e129]:
- generic [ref=f9e130]:
- generic [ref=f9e132]:
- generic [ref=f9e133]:
- button "Collapse query row" [expanded] [ref=f9e134] [cursor=pointer]
- generic [ref=f9e137]:
- button "Query editor row title A" [ref=f9e138] [cursor=pointer]:
- generic [ref=f9e139]: A
- emphasis [ref=f9e140]: (Prometheus)
- generic [ref=f9e141]:
- button "Show data source help" [ref=f9e143] [cursor=pointer]
- button "Duplicate query" [ref=f9e147] [cursor=pointer]
- button "Hide response" [ref=f9e151] [cursor=pointer]
- button "Remove query" [ref=f9e155] [cursor=pointer]
- button "Drag and drop to reorder" [ref=f9e158]:
- img "Drag and drop to reorder" [ref=f9e159]
- generic [ref=f9e162]:
- generic [ref=f9e163]:
- button "Kick start your query" [ref=f9e164] [cursor=pointer]
- generic [ref=f9e167]:
- generic [ref=f9e168] [cursor=pointer]: Explain
- generic [ref=f9e169]:
- checkbox "Explain Toggle switch" [ref=f9e170]
- generic "Toggle switch" [ref=f9e171] [cursor=pointer]
- radiogroup [ref=f9e176]:
- generic [ref=f9e177]:
- radio "Builder" [ref=f9e178] [cursor=pointer]
- generic [ref=f9e179] [cursor=pointer]: Builder
- generic [ref=f9e180]:
- radio "Code" [checked] [ref=f9e181] [cursor=pointer]
- generic [ref=f9e182] [cursor=pointer]: Code
- generic [ref=f9e184]:
- generic [ref=f9e186]:
- button "Loading metrics..." [disabled] [ref=f9e187] [cursor=pointer]
- generic [ref=f9e190]: Loading editor
- 'button "Options Legend: {{pod}} on {{node}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f9e198] [cursor=pointer]':
- generic [ref=f9e202]:
- heading "Options" [level=6] [ref=f9e203]
- generic [ref=f9e204]:
- generic [ref=f9e205]: "Legend: {{pod}} on {{node}}"
- generic [ref=f9e206]: "Format: Time series"
- generic [ref=f9e207]: "Step: auto"
- generic [ref=f9e208]: "Type: Range"
- generic [ref=f9e209]: "Exemplars: false"
- generic [ref=f9e210]:
- button "Add query" [ref=f9e211] [cursor=pointer]
- button "Query history" [ref=f9e215] [cursor=pointer]
- button "Query inspector" [ref=f9e219] [cursor=pointer]
- generic:
- main
@@ -0,0 +1,4 @@
- main [ref=f12e3]:
- generic [ref=f12e4]:
- progressbar "Contents" [ref=f12e5]
- paragraph [ref=f12e8]: Loading the Administration Console
@@ -0,0 +1,4 @@
- generic [active] [ref=f15e1]:
- progressbar "Loading" [ref=f15e4]
- generic:
- list
@@ -0,0 +1 @@
- generic [active] [ref=f18e1]: Not Found
@@ -0,0 +1,151 @@
- generic [active] [ref=e1]:
- generic [ref=e4]:
- link "Skip to main content" [ref=e5] [cursor=pointer]:
- /url: "#pageContent"
- banner [ref=e7]:
- generic [ref=e8]:
- link [ref=e10] [cursor=pointer]:
- /url: /
- img "Grafana" [ref=e11]
- generic [ref=e14]:
- button "Search or jump to..." [ref=e18] [cursor=pointer]
- generic [ref=e19]: ctrl+k
- generic [ref=e23]:
- button "New" [ref=e24] [cursor=pointer]
- button "Help" [ref=e30] [cursor=pointer]
- button "News" [ref=e33] [cursor=pointer]
- button "Profile" [ref=e36] [cursor=pointer]:
- img "User avatar" [ref=e37]
- generic [ref=e38]:
- button "Open menu" [ref=e40] [cursor=pointer]
- navigation "Breadcrumbs" [ref=e43]:
- list [ref=e44]:
- listitem [ref=e45]:
- link "Home" [ref=e46] [cursor=pointer]:
- /url: /
- listitem [ref=e50]:
- link "Explore" [ref=e51] [cursor=pointer]:
- /url: /explore
- listitem [ref=e55]:
- generic "Prometheus" [ref=e56]
- generic [ref=e57]:
- generic [ref=e60]:
- button "Copy shortened URL" [ref=e61] [cursor=pointer]
- button "Open copy link options" [ref=e64] [cursor=pointer]
- button "Toggle top search bar" [ref=e68] [cursor=pointer]
- main [ref=e74]:
- generic [ref=e76]:
- heading "Explore" [level=1] [ref=e77]
- generic [ref=e82]:
- navigation "Explore toolbar" [ref=e84]:
- navigation "Search links" [ref=e86]:
- generic [ref=e87]:
- button "Content outline" [expanded] [ref=e89] [cursor=pointer]:
- generic [ref=e92]: Outline
- generic [ref=e97] [cursor=pointer]:
- img "Prometheus logo" [ref=e99]
- textbox "Select a data source" [ref=e100]:
- /placeholder: ""
- generic [ref=e104]:
- button "Split the pane" [ref=e106] [cursor=pointer]:
- generic [ref=e109]: Split
- button "Add" [ref=e111] [cursor=pointer]
- generic [ref=e116]:
- 'button "Time range selected: Last 45 minutes" [ref=e117] [cursor=pointer]'
- button "Zoom out time range" [ref=e122] [cursor=pointer]
- generic [ref=e126]:
- button "Run query" [ref=e127] [cursor=pointer]
- button "Auto refresh turned off. Choose refresh time interval" [ref=e131] [cursor=pointer]
- generic [ref=e135]:
- generic [ref=e139]:
- button "Collapse outline" [expanded] [ref=e141] [cursor=pointer]:
- img "arrow-from-right" [ref=e142]
- button "Queries" [ref=e145] [cursor=pointer]:
- img "arrow" [ref=e146]
- button "Graph" [ref=e150] [cursor=pointer]:
- img "graph-bar" [ref=e151]
- generic [ref=e158]:
- generic [ref=e160]:
- generic "Query editor row" [ref=e163]:
- generic [ref=e164]:
- generic [ref=e166]:
- generic [ref=e167]:
- button "Collapse query row" [expanded] [ref=e168] [cursor=pointer]
- generic [ref=e171]:
- button "Query editor row title A" [ref=e172] [cursor=pointer]:
- generic [ref=e173]: A
- emphasis [ref=e174]: (Prometheus)
- generic [ref=e175]:
- button "Show data source help" [ref=e177] [cursor=pointer]
- button "Duplicate query" [ref=e181] [cursor=pointer]
- button "Hide response" [ref=e185] [cursor=pointer]
- button "Remove query" [ref=e189] [cursor=pointer]
- button "Drag and drop to reorder" [ref=e192]:
- img "Drag and drop to reorder" [ref=e193]
- generic [ref=e196]:
- generic [ref=e197]:
- button "Kick start your query" [ref=e198] [cursor=pointer]
- generic [ref=e201]:
- generic [ref=e202] [cursor=pointer]: Explain
- generic [ref=e203]:
- checkbox "Explain Toggle switch" [ref=e204]
- generic "Toggle switch" [ref=e205] [cursor=pointer]
- radiogroup [ref=e210]:
- generic [ref=e211]:
- radio "Builder" [ref=e212] [cursor=pointer]
- generic [ref=e213] [cursor=pointer]: Builder
- generic [ref=e214]:
- radio "Code" [checked] [ref=e215] [cursor=pointer]
- generic [ref=e216] [cursor=pointer]: Code
- generic [ref=e218]:
- generic [ref=e220]:
- button "Metrics browser" [ref=e221] [cursor=pointer]
- code [ref=e228]:
- generic [ref=e229]:
- generic [ref=e234]: vendor_cluster_size
- textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=e239]: vendor_cluster_size
- 'button "Options Legend: {{pod}} on {{node}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=e245] [cursor=pointer]':
- generic [ref=e249]:
- heading "Options" [level=6] [ref=e250]
- generic [ref=e251]:
- generic [ref=e252]: "Legend: {{pod}} on {{node}}"
- generic [ref=e253]: "Format: Time series"
- generic [ref=e254]: "Step: auto"
- generic [ref=e255]: "Type: Range"
- generic [ref=e256]: "Exemplars: false"
- generic [ref=e257]:
- button "Add query" [ref=e258] [cursor=pointer]
- button "Query history" [ref=e262] [cursor=pointer]
- button "Query inspector" [ref=e266] [cursor=pointer]
- main [ref=e270]:
- region [ref=e272]:
- generic [ref=e273]:
- heading "Graph" [level=2] [ref=e275]
- radiogroup [ref=e278]:
- generic [ref=e279]:
- radio "Lines" [checked] [ref=e280] [cursor=pointer]
- generic [ref=e281] [cursor=pointer]: Lines
- generic [ref=e282]:
- radio "Bars" [ref=e283] [cursor=pointer]
- generic [ref=e284] [cursor=pointer]: Bars
- generic [ref=e285]:
- radio "Points" [ref=e286] [cursor=pointer]
- generic [ref=e287] [cursor=pointer]: Points
- generic [ref=e288]:
- radio "Stacked lines" [ref=e289] [cursor=pointer]
- generic [ref=e290] [cursor=pointer]: Stacked lines
- generic [ref=e291]:
- radio "Stacked bars" [ref=e292] [cursor=pointer]
- generic [ref=e293] [cursor=pointer]: Stacked bars
- list [ref=e302]:
- listitem [ref=e303]:
- button "keycloak-0 on kc-lab-2" [ref=e307] [cursor=pointer]
- listitem [ref=e308]:
- button "keycloak-0 on kc-lab-2" [ref=e312] [cursor=pointer]
- listitem [ref=e313]:
- button "keycloak-1 on kc-lab-1" [ref=e317] [cursor=pointer]
- generic [ref=e322]:
- alert
- alert
- complementary
- complementary
@@ -0,0 +1,145 @@
- generic [active] [ref=f3e1]:
- generic [ref=f3e4]:
- link "Skip to main content" [ref=f3e5] [cursor=pointer]:
- /url: "#pageContent"
- banner [ref=f3e7]:
- generic [ref=f3e8]:
- link [ref=f3e10] [cursor=pointer]:
- /url: /
- img "Grafana" [ref=f3e11]
- generic [ref=f3e14]:
- button "Search or jump to..." [ref=f3e18] [cursor=pointer]
- generic [ref=f3e19]: ctrl+k
- generic [ref=f3e23]:
- button "New" [ref=f3e24] [cursor=pointer]
- button "Help" [ref=f3e30] [cursor=pointer]
- button "News" [ref=f3e33] [cursor=pointer]
- button "Profile" [ref=f3e36] [cursor=pointer]:
- img "User avatar" [ref=f3e37]
- generic [ref=f3e38]:
- button "Open menu" [ref=f3e40] [cursor=pointer]
- navigation "Breadcrumbs" [ref=f3e43]:
- list [ref=f3e44]:
- listitem [ref=f3e45]:
- link "Home" [ref=f3e46] [cursor=pointer]:
- /url: /
- listitem [ref=f3e50]:
- link "Explore" [ref=f3e51] [cursor=pointer]:
- /url: /explore
- listitem [ref=f3e55]:
- generic "Prometheus" [ref=f3e56]
- generic [ref=f3e57]:
- generic [ref=f3e60]:
- button "Copy shortened URL" [ref=f3e61] [cursor=pointer]
- button "Open copy link options" [ref=f3e64] [cursor=pointer]
- button "Toggle top search bar" [ref=f3e68] [cursor=pointer]
- main [ref=f3e74]:
- generic [ref=f3e76]:
- heading "Explore" [level=1] [ref=f3e77]
- generic [ref=f3e82]:
- navigation "Explore toolbar" [ref=f3e84]:
- navigation "Search links" [ref=f3e86]:
- generic [ref=f3e87]:
- button "Content outline" [expanded] [ref=f3e89] [cursor=pointer]:
- generic [ref=f3e92]: Outline
- generic [ref=f3e97] [cursor=pointer]:
- img "Prometheus logo" [ref=f3e99]
- textbox "Select a data source" [ref=f3e100]:
- /placeholder: ""
- generic [ref=f3e104]:
- button "Split the pane" [ref=f3e106] [cursor=pointer]:
- generic [ref=f3e109]: Split
- button "Add" [ref=f3e111] [cursor=pointer]
- generic [ref=f3e116]:
- 'button "Time range selected: Last 20 minutes" [ref=f3e117] [cursor=pointer]'
- button "Zoom out time range" [ref=f3e122] [cursor=pointer]
- generic [ref=f3e126]:
- button "Run query" [ref=f3e127] [cursor=pointer]
- button "Auto refresh turned off. Choose refresh time interval" [ref=f3e131] [cursor=pointer]
- generic [ref=f3e135]:
- generic [ref=f3e139]:
- button "Collapse outline" [expanded] [ref=f3e141] [cursor=pointer]:
- img "arrow-from-right" [ref=f3e142]
- button "Queries" [ref=f3e145] [cursor=pointer]:
- img "arrow" [ref=f3e146]
- button "Graph" [ref=f3e150] [cursor=pointer]:
- img "graph-bar" [ref=f3e151]
- generic [ref=f3e158]:
- generic [ref=f3e160]:
- generic "Query editor row" [ref=f3e163]:
- generic [ref=f3e164]:
- generic [ref=f3e166]:
- generic [ref=f3e167]:
- button "Collapse query row" [expanded] [ref=f3e168] [cursor=pointer]
- generic [ref=f3e171]:
- button "Query editor row title A" [ref=f3e172] [cursor=pointer]:
- generic [ref=f3e173]: A
- emphasis [ref=f3e174]: (Prometheus)
- generic [ref=f3e175]:
- button "Show data source help" [ref=f3e177] [cursor=pointer]
- button "Duplicate query" [ref=f3e181] [cursor=pointer]
- button "Hide response" [ref=f3e185] [cursor=pointer]
- button "Remove query" [ref=f3e189] [cursor=pointer]
- button "Drag and drop to reorder" [ref=f3e192]:
- img "Drag and drop to reorder" [ref=f3e193]
- generic [ref=f3e196]:
- generic [ref=f3e197]:
- button "Kick start your query" [ref=f3e198] [cursor=pointer]
- generic [ref=f3e201]:
- generic [ref=f3e202] [cursor=pointer]: Explain
- generic [ref=f3e203]:
- checkbox "Explain Toggle switch" [ref=f3e204]
- generic "Toggle switch" [ref=f3e205] [cursor=pointer]
- radiogroup [ref=f3e210]:
- generic [ref=f3e211]:
- radio "Builder" [ref=f3e212] [cursor=pointer]
- generic [ref=f3e213] [cursor=pointer]: Builder
- generic [ref=f3e214]:
- radio "Code" [checked] [ref=f3e215] [cursor=pointer]
- generic [ref=f3e216] [cursor=pointer]: Code
- generic [ref=f3e218]:
- generic [ref=f3e220]:
- button "Loading metrics..." [disabled] [ref=f3e221] [cursor=pointer]
- code [ref=f3e228]:
- generic [ref=f3e229]:
- generic [ref=f3e234]: "up{job=\"keycloak\"}"
- textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=f3e239]: "up{job=\"keycloak\"}"
- 'button "Options Legend: up — {{pod}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f3e245] [cursor=pointer]':
- generic [ref=f3e249]:
- heading "Options" [level=6] [ref=f3e250]
- generic [ref=f3e251]:
- generic [ref=f3e252]: "Legend: up — {{pod}}"
- generic [ref=f3e253]: "Format: Time series"
- generic [ref=f3e254]: "Step: auto"
- generic [ref=f3e255]: "Type: Range"
- generic [ref=f3e256]: "Exemplars: false"
- generic [ref=f3e257]:
- button "Add query" [ref=f3e258] [cursor=pointer]
- button "Query history" [ref=f3e262] [cursor=pointer]
- button "Query inspector" [ref=f3e266] [cursor=pointer]
- main [ref=f3e270]:
- region [ref=f3e272]:
- generic [ref=f3e273]:
- heading "Graph" [level=2] [ref=f3e275]
- radiogroup [ref=f3e278]:
- generic [ref=f3e279]:
- radio "Lines" [checked] [ref=f3e280] [cursor=pointer]
- generic [ref=f3e281] [cursor=pointer]: Lines
- generic [ref=f3e282]:
- radio "Bars" [ref=f3e283] [cursor=pointer]
- generic [ref=f3e284] [cursor=pointer]: Bars
- generic [ref=f3e285]:
- radio "Points" [ref=f3e286] [cursor=pointer]
- generic [ref=f3e287] [cursor=pointer]: Points
- generic [ref=f3e288]:
- radio "Stacked lines" [ref=f3e289] [cursor=pointer]
- generic [ref=f3e290] [cursor=pointer]: Stacked lines
- generic [ref=f3e291]:
- radio "Stacked bars" [ref=f3e292] [cursor=pointer]
- generic [ref=f3e293] [cursor=pointer]: Stacked bars
- generic [ref=f3e294]: Loading plugin panel...
- generic [ref=f3e299]:
- alert
- alert
- complementary
- complementary
@@ -0,0 +1,44 @@
- main [ref=f6e7]:
- generic [ref=f6e9]:
- generic [ref=f6e11]:
- generic [ref=f6e12]:
- img "Grafana" [ref=f6e13]
- heading "Welcome to Grafana" [level=1] [ref=f6e15]
- generic [ref=f6e19]:
- generic [ref=f6e20]:
- generic [ref=f6e21]: Email or username
- textbox "Email or username" [active] [ref=f6e28]:
- /placeholder: email or username
- generic [ref=f6e29]:
- generic [ref=f6e30]: Password
- generic [ref=f6e36]:
- textbox "Password" [ref=f6e37]:
- /placeholder: password
- switch "Show password" [ref=f6e39] [cursor=pointer]
- button "Log in" [ref=f6e42] [cursor=pointer]
- link "Forgot your password?" [ref=f6e45] [cursor=pointer]:
- /url: /user/password/send-reset-email
- list [ref=f6e49]:
- listitem [ref=f6e50]:
- link "Documentation" [ref=f6e53] [cursor=pointer]:
- /url: https://grafana.com/docs/grafana/latest/?utm_source=grafana_footer
- text: "|"
- listitem [ref=f6e54]:
- link "Support" [ref=f6e57] [cursor=pointer]:
- /url: https://grafana.com/products/enterprise/?utm_source=grafana_footer
- text: "|"
- listitem [ref=f6e58]:
- link "Community" [ref=f6e61] [cursor=pointer]:
- /url: https://community.grafana.com/?utm_source=grafana_footer
- text: "|"
- listitem [ref=f6e62]:
- link "Open Source" [ref=f6e63] [cursor=pointer]:
- /url: https://grafana.com/oss/grafana?utm_source=grafana_footer
- text: "|"
- listitem [ref=f6e64]:
- link "Grafana v11.4.0 (b58701869e)" [ref=f6e65] [cursor=pointer]:
- /url: https://github.com/grafana/grafana/blob/main/CHANGELOG.md
- text: "|"
- listitem [ref=f6e66]:
- link "New version available!" [ref=f6e69] [cursor=pointer]:
- /url: https://grafana.com/grafana/download?utm_source=grafana_footer
@@ -0,0 +1,159 @@
- generic [active] [ref=f9e1]:
- generic [ref=f9e4]:
- link "Skip to main content" [ref=f9e5] [cursor=pointer]:
- /url: "#pageContent"
- banner [ref=f9e7]:
- generic [ref=f9e8]:
- link [ref=f9e10] [cursor=pointer]:
- /url: /
- img "Grafana" [ref=f9e11]
- generic [ref=f9e14]:
- button "Search or jump to..." [ref=f9e18] [cursor=pointer]
- generic [ref=f9e19]: ctrl+k
- generic [ref=f9e23]:
- button "New" [ref=f9e24] [cursor=pointer]
- button "Help" [ref=f9e30] [cursor=pointer]
- button "News" [ref=f9e33] [cursor=pointer]
- button "Profile" [ref=f9e36] [cursor=pointer]:
- img "User avatar" [ref=f9e37]
- generic [ref=f9e38]:
- button "Open menu" [ref=f9e40] [cursor=pointer]
- navigation "Breadcrumbs" [ref=f9e43]:
- list [ref=f9e44]:
- listitem [ref=f9e45]:
- link "Home" [ref=f9e46] [cursor=pointer]:
- /url: /
- listitem [ref=f9e50]:
- link "Explore" [ref=f9e51] [cursor=pointer]:
- /url: /explore
- listitem [ref=f9e55]:
- generic "Prometheus" [ref=f9e56]
- generic [ref=f9e57]:
- generic [ref=f9e60]:
- button "Copy shortened URL" [ref=f9e61] [cursor=pointer]
- button "Open copy link options" [ref=f9e64] [cursor=pointer]
- button "Toggle top search bar" [ref=f9e68] [cursor=pointer]
- main [ref=f9e74]:
- generic [ref=f9e76]:
- heading "Explore" [level=1] [ref=f9e77]
- generic [ref=f9e82]:
- navigation "Explore toolbar" [ref=f9e84]:
- navigation "Search links" [ref=f9e86]:
- generic [ref=f9e87]:
- button "Content outline" [expanded] [ref=f9e89] [cursor=pointer]:
- generic [ref=f9e92]: Outline
- generic [ref=f9e97] [cursor=pointer]:
- img "Prometheus logo" [ref=f9e99]
- textbox "Select a data source" [ref=f9e100]:
- /placeholder: ""
- generic [ref=f9e104]:
- button "Split the pane" [ref=f9e106] [cursor=pointer]:
- generic [ref=f9e109]: Split
- button "Add" [ref=f9e111] [cursor=pointer]
- generic [ref=f9e116]:
- 'button "Time range selected: Last 55 minutes" [ref=f9e117] [cursor=pointer]'
- button "Zoom out time range" [ref=f9e122] [cursor=pointer]
- generic [ref=f9e126]:
- button "Run query" [ref=f9e127] [cursor=pointer]
- button "Auto refresh turned off. Choose refresh time interval" [ref=f9e131] [cursor=pointer]
- generic [ref=f9e135]:
- generic [ref=f9e139]:
- button "Collapse outline" [expanded] [ref=f9e141] [cursor=pointer]:
- img "arrow-from-right" [ref=f9e142]
- button "Queries" [ref=f9e145] [cursor=pointer]:
- img "arrow" [ref=f9e146]
- button "Graph" [ref=f9e150] [cursor=pointer]:
- img "graph-bar" [ref=f9e151]
- generic [ref=f9e158]:
- generic [ref=f9e160]:
- generic "Query editor row" [ref=f9e163]:
- generic [ref=f9e164]:
- generic [ref=f9e166]:
- generic [ref=f9e167]:
- button "Collapse query row" [expanded] [ref=f9e168] [cursor=pointer]
- generic [ref=f9e171]:
- button "Query editor row title A" [ref=f9e172] [cursor=pointer]:
- generic [ref=f9e173]: A
- emphasis [ref=f9e174]: (Prometheus)
- generic [ref=f9e175]:
- button "Show data source help" [ref=f9e177] [cursor=pointer]
- button "Duplicate query" [ref=f9e181] [cursor=pointer]
- button "Hide response" [ref=f9e185] [cursor=pointer]
- button "Remove query" [ref=f9e189] [cursor=pointer]
- button "Drag and drop to reorder" [ref=f9e192]:
- img "Drag and drop to reorder" [ref=f9e193]
- generic [ref=f9e196]:
- generic [ref=f9e197]:
- button "Kick start your query" [ref=f9e198] [cursor=pointer]
- generic [ref=f9e201]:
- generic [ref=f9e202] [cursor=pointer]: Explain
- generic [ref=f9e203]:
- checkbox "Explain Toggle switch" [ref=f9e204]
- generic "Toggle switch" [ref=f9e205] [cursor=pointer]
- radiogroup [ref=f9e210]:
- generic [ref=f9e211]:
- radio "Builder" [ref=f9e212] [cursor=pointer]
- generic [ref=f9e213] [cursor=pointer]: Builder
- generic [ref=f9e214]:
- radio "Code" [checked] [ref=f9e215] [cursor=pointer]
- generic [ref=f9e216] [cursor=pointer]: Code
- generic [ref=f9e218]:
- generic [ref=f9e220]:
- button "Metrics browser" [ref=f9e221] [cursor=pointer]
- code [ref=f9e228]:
- generic [ref=f9e229]:
- generic [ref=f9e234]: "up{job=~\"keycloak|node-exporter\"}"
- textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=f9e239]: "up{job=~\"keycloak|node-exporter\"}"
- 'button "Options Legend: {{job}} — {{pod}}{{node}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f9e245] [cursor=pointer]':
- generic [ref=f9e249]:
- heading "Options" [level=6] [ref=f9e250]
- generic [ref=f9e251]:
- generic [ref=f9e252]: "Legend: {{job}} — {{pod}}{{node}}"
- generic [ref=f9e253]: "Format: Time series"
- generic [ref=f9e254]: "Step: auto"
- generic [ref=f9e255]: "Type: Range"
- generic [ref=f9e256]: "Exemplars: false"
- generic [ref=f9e257]:
- button "Add query" [ref=f9e258] [cursor=pointer]
- button "Query history" [ref=f9e262] [cursor=pointer]
- button "Query inspector" [ref=f9e266] [cursor=pointer]
- main [ref=f9e270]:
- region [ref=f9e272]:
- generic [ref=f9e273]:
- heading "Graph" [level=2] [ref=f9e275]
- radiogroup [ref=f9e278]:
- generic [ref=f9e279]:
- radio "Lines" [checked] [ref=f9e280] [cursor=pointer]
- generic [ref=f9e281] [cursor=pointer]: Lines
- generic [ref=f9e282]:
- radio "Bars" [ref=f9e283] [cursor=pointer]
- generic [ref=f9e284] [cursor=pointer]: Bars
- generic [ref=f9e285]:
- radio "Points" [ref=f9e286] [cursor=pointer]
- generic [ref=f9e287] [cursor=pointer]: Points
- generic [ref=f9e288]:
- radio "Stacked lines" [ref=f9e289] [cursor=pointer]
- generic [ref=f9e290] [cursor=pointer]: Stacked lines
- generic [ref=f9e291]:
- radio "Stacked bars" [ref=f9e292] [cursor=pointer]
- generic [ref=f9e293] [cursor=pointer]: Stacked bars
- list [ref=f9e311]:
- listitem [ref=f9e312]:
- button "keycloak — keycloak-1kc-lab-1" [ref=f9e316] [cursor=pointer]
- listitem [ref=f9e317]:
- button "keycloak — keycloak-1kc-lab-1" [ref=f9e321] [cursor=pointer]
- listitem [ref=f9e322]:
- button "keycloak — keycloak-0kc-lab-2" [ref=f9e326] [cursor=pointer]
- listitem [ref=f9e327]:
- button "keycloak — keycloak-0kc-lab-2" [ref=f9e331] [cursor=pointer]
- listitem [ref=f9e332]:
- button "keycloak — keycloak-0kc-lab-2" [ref=f9e336] [cursor=pointer]
- listitem [ref=f9e337]:
- button "node-exporter — kc-lab-1" [ref=f9e341] [cursor=pointer]
- listitem [ref=f9e342]:
- button "node-exporter — kc-lab-2" [ref=f9e346] [cursor=pointer]
- generic [ref=f9e351]:
- alert
- alert
- complementary
- complementary
@@ -0,0 +1,104 @@
- generic [ref=f12e4]:
- link "Skip to main content" [ref=f12e5] [cursor=pointer]:
- /url: "#pageContent"
- banner [ref=f12e7]:
- generic [ref=f12e8]:
- link [ref=f12e10] [cursor=pointer]:
- /url: /
- img "Grafana" [ref=f12e11]
- generic [ref=f12e14]:
- button "Search or jump to..." [ref=f12e18] [cursor=pointer]
- generic [ref=f12e19]: ctrl+k
- generic [ref=f12e23]:
- button "New" [ref=f12e24] [cursor=pointer]
- button "Help" [ref=f12e30] [cursor=pointer]
- button "News" [ref=f12e33] [cursor=pointer]
- button "Profile" [ref=f12e36] [cursor=pointer]:
- img "User avatar" [ref=f12e37]
- generic [ref=f12e38]:
- button "Open menu" [ref=f12e40] [cursor=pointer]
- navigation "Breadcrumbs" [ref=f12e43]:
- list [ref=f12e44]:
- listitem [ref=f12e45]:
- link "Home" [ref=f12e46] [cursor=pointer]:
- /url: /
- listitem [ref=f12e50]:
- link "Explore" [ref=f12e51] [cursor=pointer]:
- /url: /explore
- listitem [ref=f12e55]:
- generic "Prometheus" [ref=f12e56]
- generic [ref=f12e57]:
- button "Show more items" [ref=f12e60] [cursor=pointer]
- button "Toggle top search bar" [ref=f12e64] [cursor=pointer]
- main [ref=f12e70]:
- generic [ref=f12e72]:
- heading "Explore" [level=1] [ref=f12e73]
- generic [ref=f12e78]:
- navigation "Explore toolbar" [ref=f12e80]:
- navigation "Search links" [ref=f12e82]:
- generic [ref=f12e83]:
- button "Content outline" [expanded] [ref=f12e85] [cursor=pointer]:
- generic [ref=f12e88]: Outline
- generic [ref=f12e93] [cursor=pointer]:
- img "Prometheus logo" [ref=f12e95]
- textbox "Select a data source" [ref=f12e96]:
- /placeholder: ""
- button "Show more items" [ref=f12e102] [cursor=pointer]
- generic [ref=f12e106]:
- generic [ref=f12e110]:
- button "Collapse outline" [expanded] [ref=f12e112] [cursor=pointer]:
- img "arrow-from-right" [ref=f12e113]
- button "Queries" [ref=f12e116] [cursor=pointer]:
- img "arrow" [ref=f12e117]
- generic [ref=f12e124]:
- generic [ref=f12e126]:
- generic "Query editor row" [ref=f12e129]:
- generic [ref=f12e130]:
- generic [ref=f12e132]:
- generic [ref=f12e133]:
- button "Collapse query row" [expanded] [ref=f12e134] [cursor=pointer]
- generic [ref=f12e137]:
- button "Query editor row title A" [ref=f12e138] [cursor=pointer]:
- generic [ref=f12e139]: A
- emphasis [ref=f12e140]: (Prometheus)
- generic [ref=f12e141]:
- button "Show data source help" [ref=f12e143] [cursor=pointer]
- button "Duplicate query" [ref=f12e147] [cursor=pointer]
- button "Hide response" [ref=f12e151] [cursor=pointer]
- button "Remove query" [ref=f12e155] [cursor=pointer]
- button "Drag and drop to reorder" [ref=f12e158]:
- img "Drag and drop to reorder" [ref=f12e159]
- generic [ref=f12e162]:
- generic [ref=f12e163]:
- button "Kick start your query" [ref=f12e164] [cursor=pointer]
- generic [ref=f12e167]:
- generic [ref=f12e168] [cursor=pointer]: Explain
- generic [ref=f12e169]:
- checkbox "Explain Toggle switch" [ref=f12e170]
- generic "Toggle switch" [ref=f12e171] [cursor=pointer]
- radiogroup [ref=f12e176]:
- generic [ref=f12e177]:
- radio "Builder" [ref=f12e178] [cursor=pointer]
- generic [ref=f12e179] [cursor=pointer]: Builder
- generic [ref=f12e180]:
- radio "Code" [checked] [ref=f12e181] [cursor=pointer]
- generic [ref=f12e182] [cursor=pointer]: Code
- generic [ref=f12e184]:
- generic [ref=f12e186]:
- button "Loading metrics..." [disabled] [ref=f12e187] [cursor=pointer]
- generic [ref=f12e190]: Loading editor
- 'button "Options Legend: {{pod}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f12e198] [cursor=pointer]':
- generic [ref=f12e202]:
- heading "Options" [level=6] [ref=f12e203]
- generic [ref=f12e204]:
- generic [ref=f12e205]: "Legend: {{pod}}"
- generic [ref=f12e206]: "Format: Time series"
- generic [ref=f12e207]: "Step: auto"
- generic [ref=f12e208]: "Type: Range"
- generic [ref=f12e209]: "Exemplars: false"
- generic [ref=f12e210]:
- button "Add query" [ref=f12e211] [cursor=pointer]
- button "Query history" [ref=f12e215] [cursor=pointer]
- button "Query inspector" [ref=f12e219] [cursor=pointer]
- generic:
- main
@@ -0,0 +1,145 @@
- generic [active] [ref=f15e1]:
- generic [ref=f15e4]:
- link "Skip to main content" [ref=f15e5] [cursor=pointer]:
- /url: "#pageContent"
- banner [ref=f15e7]:
- generic [ref=f15e8]:
- link [ref=f15e10] [cursor=pointer]:
- /url: /
- img "Grafana" [ref=f15e11]
- generic [ref=f15e14]:
- button "Search or jump to..." [ref=f15e18] [cursor=pointer]
- generic [ref=f15e19]: ctrl+k
- generic [ref=f15e23]:
- button "New" [ref=f15e24] [cursor=pointer]
- button "Help" [ref=f15e30] [cursor=pointer]
- button "News" [ref=f15e33] [cursor=pointer]
- button "Profile" [ref=f15e36] [cursor=pointer]:
- img "User avatar" [ref=f15e37]
- generic [ref=f15e38]:
- button "Open menu" [ref=f15e40] [cursor=pointer]
- navigation "Breadcrumbs" [ref=f15e43]:
- list [ref=f15e44]:
- listitem [ref=f15e45]:
- link "Home" [ref=f15e46] [cursor=pointer]:
- /url: /
- listitem [ref=f15e50]:
- link "Explore" [ref=f15e51] [cursor=pointer]:
- /url: /explore
- listitem [ref=f15e55]:
- generic "Prometheus" [ref=f15e56]
- generic [ref=f15e57]:
- generic [ref=f15e60]:
- button "Copy shortened URL" [ref=f15e61] [cursor=pointer]
- button "Open copy link options" [ref=f15e64] [cursor=pointer]
- button "Toggle top search bar" [ref=f15e68] [cursor=pointer]
- main [ref=f15e74]:
- generic [ref=f15e76]:
- heading "Explore" [level=1] [ref=f15e77]
- generic [ref=f15e82]:
- navigation "Explore toolbar" [ref=f15e84]:
- navigation "Search links" [ref=f15e86]:
- generic [ref=f15e87]:
- button "Content outline" [expanded] [ref=f15e89] [cursor=pointer]:
- generic [ref=f15e92]: Outline
- generic [ref=f15e97] [cursor=pointer]:
- img "Prometheus logo" [ref=f15e99]
- textbox "Select a data source" [ref=f15e100]:
- /placeholder: ""
- generic [ref=f15e104]:
- button "Split the pane" [ref=f15e106] [cursor=pointer]:
- generic [ref=f15e109]: Split
- button "Add" [ref=f15e111] [cursor=pointer]
- generic [ref=f15e116]:
- 'button "Time range selected: Last 30 minutes" [ref=f15e117] [cursor=pointer]'
- button "Zoom out time range" [ref=f15e122] [cursor=pointer]
- generic [ref=f15e126]:
- button "Run query" [ref=f15e127] [cursor=pointer]
- button "Auto refresh turned off. Choose refresh time interval" [ref=f15e131] [cursor=pointer]
- generic [ref=f15e135]:
- generic [ref=f15e139]:
- button "Collapse outline" [expanded] [ref=f15e141] [cursor=pointer]:
- img "arrow-from-right" [ref=f15e142]
- button "Queries" [ref=f15e145] [cursor=pointer]:
- img "arrow" [ref=f15e146]
- button "Graph" [ref=f15e150] [cursor=pointer]:
- img "graph-bar" [ref=f15e151]
- generic [ref=f15e158]:
- generic [ref=f15e160]:
- generic "Query editor row" [ref=f15e163]:
- generic [ref=f15e164]:
- generic [ref=f15e166]:
- generic [ref=f15e167]:
- button "Collapse query row" [expanded] [ref=f15e168] [cursor=pointer]
- generic [ref=f15e171]:
- button "Query editor row title A" [ref=f15e172] [cursor=pointer]:
- generic [ref=f15e173]: A
- emphasis [ref=f15e174]: (Prometheus)
- generic [ref=f15e175]:
- button "Show data source help" [ref=f15e177] [cursor=pointer]
- button "Duplicate query" [ref=f15e181] [cursor=pointer]
- button "Hide response" [ref=f15e185] [cursor=pointer]
- button "Remove query" [ref=f15e189] [cursor=pointer]
- button "Drag and drop to reorder" [ref=f15e192]:
- img "Drag and drop to reorder" [ref=f15e193]
- generic [ref=f15e196]:
- generic [ref=f15e197]:
- button "Kick start your query" [ref=f15e198] [cursor=pointer]
- generic [ref=f15e201]:
- generic [ref=f15e202] [cursor=pointer]: Explain
- generic [ref=f15e203]:
- checkbox "Explain Toggle switch" [ref=f15e204]
- generic "Toggle switch" [ref=f15e205] [cursor=pointer]
- radiogroup [ref=f15e210]:
- generic [ref=f15e211]:
- radio "Builder" [ref=f15e212] [cursor=pointer]
- generic [ref=f15e213] [cursor=pointer]: Builder
- generic [ref=f15e214]:
- radio "Code" [checked] [ref=f15e215] [cursor=pointer]
- generic [ref=f15e216] [cursor=pointer]: Code
- generic [ref=f15e218]:
- generic [ref=f15e220]:
- button "Loading metrics..." [disabled] [ref=f15e221] [cursor=pointer]
- code [ref=f15e228]:
- generic [ref=f15e229]:
- generic [ref=f15e234]: agroal_blocking_time_max_milliseconds
- textbox "Editor content;Press Alt+F1 for Accessibility Options." [ref=f15e239]: agroal_blocking_time_max_milliseconds
- 'button "Options Legend: blocking max - {{pod}} Format: Time series Step: auto Type: Range Exemplars: false" [ref=f15e245] [cursor=pointer]':
- generic [ref=f15e249]:
- heading "Options" [level=6] [ref=f15e250]
- generic [ref=f15e251]:
- generic [ref=f15e252]: "Legend: blocking max - {{pod}}"
- generic [ref=f15e253]: "Format: Time series"
- generic [ref=f15e254]: "Step: auto"
- generic [ref=f15e255]: "Type: Range"
- generic [ref=f15e256]: "Exemplars: false"
- generic [ref=f15e257]:
- button "Add query" [ref=f15e258] [cursor=pointer]
- button "Query history" [ref=f15e262] [cursor=pointer]
- button "Query inspector" [ref=f15e266] [cursor=pointer]
- main [ref=f15e270]:
- region [ref=f15e272]:
- generic [ref=f15e273]:
- heading "Graph" [level=2] [ref=f15e275]
- radiogroup [ref=f15e278]:
- generic [ref=f15e279]:
- radio "Lines" [checked] [ref=f15e280] [cursor=pointer]
- generic [ref=f15e281] [cursor=pointer]: Lines
- generic [ref=f15e282]:
- radio "Bars" [ref=f15e283] [cursor=pointer]
- generic [ref=f15e284] [cursor=pointer]: Bars
- generic [ref=f15e285]:
- radio "Points" [ref=f15e286] [cursor=pointer]
- generic [ref=f15e287] [cursor=pointer]: Points
- generic [ref=f15e288]:
- radio "Stacked lines" [ref=f15e289] [cursor=pointer]
- generic [ref=f15e290] [cursor=pointer]: Stacked lines
- generic [ref=f15e291]:
- radio "Stacked bars" [ref=f15e292] [cursor=pointer]
- generic [ref=f15e293] [cursor=pointer]: Stacked bars
- generic [ref=f15e294]: Loading plugin panel...
- generic [ref=f15e299]:
- alert
- alert
- complementary
- complementary
@@ -0,0 +1,56 @@
- generic [ref=f18e4]:
- link "Skip to main content" [ref=f18e5] [cursor=pointer]:
- /url: "#pageContent"
- banner [ref=f18e7]:
- generic [ref=f18e8]:
- link [ref=f18e10] [cursor=pointer]:
- /url: /
- img "Grafana" [ref=f18e11]
- generic [ref=f18e14]:
- button "Search or jump to..." [ref=f18e18] [cursor=pointer]
- generic [ref=f18e19]: ctrl+k
- generic [ref=f18e23]:
- button "New" [ref=f18e24] [cursor=pointer]
- button "Help" [ref=f18e30] [cursor=pointer]
- button "News" [ref=f18e33] [cursor=pointer]
- button "Profile" [ref=f18e36] [cursor=pointer]:
- img "User avatar" [ref=f18e37]
- generic [ref=f18e38]:
- button "Open menu" [ref=f18e40] [cursor=pointer]
- navigation "Breadcrumbs" [ref=f18e43]:
- list [ref=f18e44]:
- listitem [ref=f18e45]:
- link "Home" [ref=f18e46] [cursor=pointer]:
- /url: /
- listitem [ref=f18e50]:
- link "Explore" [ref=f18e51] [cursor=pointer]:
- /url: /explore
- listitem [ref=f18e55]:
- generic "Prometheus" [ref=f18e56]
- generic [ref=f18e57]:
- button "Show more items" [ref=f18e60] [cursor=pointer]
- button "Toggle top search bar" [ref=f18e64] [cursor=pointer]
- main [ref=f18e70]:
- generic [ref=f18e72]:
- heading "Explore" [level=1] [ref=f18e73]
- generic [ref=f18e78]:
- navigation "Explore toolbar" [ref=f18e80]:
- navigation "Search links" [ref=f18e82]:
- generic [ref=f18e83]:
- button "Content outline" [expanded] [ref=f18e85] [cursor=pointer]:
- generic [ref=f18e88]: Outline
- generic [ref=f18e93] [cursor=pointer]:
- img "Prometheus logo" [ref=f18e95]
- textbox "Select a data source" [ref=f18e96]:
- /placeholder: ""
- button "Show more items" [ref=f18e102] [cursor=pointer]
- generic [ref=f18e106]:
- button "Collapse outline" [expanded] [ref=f18e112] [cursor=pointer]:
- img "arrow-from-right" [ref=f18e113]
- generic [ref=f18e120]:
- generic [ref=f18e123]:
- button "Add query" [ref=f18e124] [cursor=pointer]
- button "Query history" [ref=f18e128] [cursor=pointer]
- button "Query inspector" [ref=f18e132] [cursor=pointer]
- generic:
- main
@@ -0,0 +1,7 @@
- main [ref=f21e2]:
- heading "AP3 · Backend-for-Frontend" [level=1] [ref=f21e3]
- paragraph [ref=f21e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
- button "Keycloak 로그인" [ref=f21e5] [cursor=pointer]
- button "token 경계 확인" [ref=f21e6] [cursor=pointer]
- button "BFF 경유 API 호출" [ref=f21e7] [cursor=pointer]
- button "CSRF token으로 상태 변경" [ref=f21e8] [cursor=pointer]
@@ -0,0 +1,16 @@
- generic [ref=f22e3]:
- banner [ref=f22e4]:
- generic [ref=f22e5]: keycloak-patterns
- main [ref=f22e6]:
- heading "Sign in to your account" [level=1] [ref=f22e8]
- generic [ref=f22e12]:
- generic [ref=f22e13]:
- generic [ref=f22e14]: Username or email
- textbox "Username or email" [active] [ref=f22e17]
- generic [ref=f22e18]:
- generic [ref=f22e19]: Password
- generic [ref=f22e21]:
- textbox "Password" [ref=f22e24]
- button "Show password" [ref=f22e26] [cursor=pointer]:
- generic [aria-hidden] [ref=f22e27]:
- button "Sign In" [ref=f22e30] [cursor=pointer]
@@ -0,0 +1,20 @@
- generic [ref=f23e3]:
- banner [ref=f23e4]:
- generic [ref=f23e5]: keycloak-patterns
- main [ref=f23e6]:
- heading "Update Account Information" [level=1] [ref=f23e8]
- generic [ref=f23e9]:
- generic [ref=f23e10]: "* Required fields"
- generic [ref=f23e13]:
- generic [ref=f23e14]:
- generic [ref=f23e15]: Email *
- textbox "Email" [ref=f23e19]: labuser@example.com
- generic [ref=f23e20]:
- generic [ref=f23e21]: First name *
- textbox "First name" [invalid] [ref=f23e25]
- generic [ref=f23e26]: Please specify this field.
- generic [ref=f23e31]:
- generic [ref=f23e32]: Last name *
- textbox "Last name" [invalid] [ref=f23e36]
- generic [ref=f23e37]: Please specify this field.
- button "Submit" [ref=f23e44]
@@ -0,0 +1,20 @@
- generic [ref=f23e3]:
- banner [ref=f23e4]:
- generic [ref=f23e5]: keycloak-patterns
- main [ref=f23e6]:
- heading "Update Account Information" [level=1] [ref=f23e8]
- generic [ref=f23e9]:
- generic [ref=f23e10]: "* Required fields"
- generic [ref=f23e13]:
- generic [ref=f23e14]:
- generic [ref=f23e15]: Email *
- textbox "Email" [ref=f23e19]: labuser@example.com
- generic [ref=f23e20]:
- generic [ref=f23e21]: First name *
- textbox "First name" [invalid] [ref=f23e25]: Lab
- generic [ref=f23e26]: Please specify this field.
- generic [ref=f23e31]:
- generic [ref=f23e32]: Last name *
- textbox "Last name" [active] [invalid] [ref=f23e36]: User
- generic [ref=f23e37]: Please specify this field.
- button "Submit" [ref=f23e44]
@@ -0,0 +1,7 @@
- main [ref=f24e2]:
- heading "AP3 · Backend-for-Frontend" [level=1] [ref=f24e3]
- paragraph [ref=f24e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
- button "Keycloak 로그인" [ref=f24e5] [cursor=pointer]
- button "token 경계 확인" [ref=f24e6] [cursor=pointer]
- button "BFF 경유 API 호출" [ref=f24e7] [cursor=pointer]
- button "CSRF token으로 상태 변경" [ref=f24e8] [cursor=pointer]
@@ -0,0 +1,16 @@
- generic [ref=f25e3]:
- banner [ref=f25e4]:
- generic [ref=f25e5]: keycloak-patterns
- main [ref=f25e6]:
- heading "Sign in to your account" [level=1] [ref=f25e8]
- generic [ref=f25e12]:
- generic [ref=f25e13]:
- generic [ref=f25e14]: Username or email
- textbox "Username or email" [active] [ref=f25e17]
- generic [ref=f25e18]:
- generic [ref=f25e19]: Password
- generic [ref=f25e21]:
- textbox "Password" [ref=f25e24]
- button "Show password" [ref=f25e26] [cursor=pointer]:
- generic [aria-hidden] [ref=f25e27]:
- button "Sign In" [ref=f25e30] [cursor=pointer]
@@ -0,0 +1,9 @@
- generic [ref=f26e2]:
- heading "Login with OAuth 2.0" [level=2] [ref=f26e3]
- alert [ref=f26e4]: Invalid credentials
- table [ref=f26e5]:
- rowgroup [ref=f26e6]:
- row [ref=f26e7]:
- cell [ref=f26e8]:
- link "keycloak" [ref=f26e9] [cursor=pointer]:
- /url: /oauth2/authorization/keycloak
@@ -0,0 +1,7 @@
- main [ref=f27e2]:
- heading "AP3 · Backend-for-Frontend" [level=1] [ref=f27e3]
- paragraph [ref=f27e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
- button "Keycloak 로그인" [ref=f27e5] [cursor=pointer]
- button "token 경계 확인" [ref=f27e6] [cursor=pointer]
- button "BFF 경유 API 호출" [ref=f27e7] [cursor=pointer]
- button "CSRF token으로 상태 변경" [ref=f27e8] [cursor=pointer]
@@ -0,0 +1,7 @@
- main [ref=f27e2]:
- heading "AP3 · Backend-for-Frontend" [level=1] [ref=f27e3]
- paragraph [ref=f27e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
- button "Keycloak 로그인" [ref=f27e5] [cursor=pointer]
- button "token 경계 확인" [ref=f27e6] [cursor=pointer]
- button "BFF 경유 API 호출" [ref=f27e7] [cursor=pointer]
- button "CSRF token으로 상태 변경" [ref=f27e8] [cursor=pointer]
@@ -0,0 +1,7 @@
- main [ref=f27e2]:
- heading "AP3 · Backend-for-Frontend" [level=1] [ref=f27e3]
- paragraph [ref=f27e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
- button "Keycloak 로그인" [ref=f27e5] [cursor=pointer]
- button "token 경계 확인" [ref=f27e6] [cursor=pointer]
- button "BFF 경유 API 호출" [ref=f27e7] [cursor=pointer]
- button "CSRF token으로 상태 변경" [ref=f27e8] [cursor=pointer]
@@ -0,0 +1 @@
- generic [active] [ref=f28e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":true,\"refreshTokenStoredOnServer\":true,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1,7 @@
- main [ref=f29e2]:
- heading "AP3 · Backend-for-Frontend" [level=1] [ref=f29e3]
- paragraph [ref=f29e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
- button "Keycloak 로그인" [ref=f29e5] [cursor=pointer]
- button "token 경계 확인" [ref=f29e6] [cursor=pointer]
- button "BFF 경유 API 호출" [ref=f29e7] [cursor=pointer]
- button "CSRF token으로 상태 변경" [ref=f29e8] [cursor=pointer]
@@ -0,0 +1 @@
- generic [active] [ref=f30e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1,5 @@
- generic [active] [ref=f31e1]:
- heading "Whitelabel Error Page" [level=1] [ref=f31e2]
- paragraph [ref=f31e3]: This application has no explicit mapping for /error, so you are seeing this as a fallback.
- generic [ref=f31e4]: Fri Sep 04 05:00:51 GMT 2026
- generic [ref=f31e5]: There was an unexpected error (type=Internal Server Error, status=500).
@@ -0,0 +1 @@
- generic [active] [ref=f32e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1,7 @@
- main [ref=f33e2]:
- heading "AP3 · Backend-for-Frontend" [level=1] [ref=f33e3]
- paragraph [ref=f33e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
- button "Keycloak 로그인" [ref=f33e5] [cursor=pointer]
- button "token 경계 확인" [ref=f33e6] [cursor=pointer]
- button "BFF 경유 API 호출" [ref=f33e7] [cursor=pointer]
- button "CSRF token으로 상태 변경" [ref=f33e8] [cursor=pointer]
@@ -0,0 +1 @@
- generic [active] [ref=f34e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1 @@
- generic [active] [ref=f35e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":false,\"refreshTokenStoredOnServer\":false,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1,7 @@
- main [ref=f36e2]:
- heading "AP3 · Backend-for-Frontend" [level=1] [ref=f36e3]
- paragraph [ref=f36e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
- button "Keycloak 로그인" [ref=f36e5] [cursor=pointer]
- button "token 경계 확인" [ref=f36e6] [cursor=pointer]
- button "BFF 경유 API 호출" [ref=f36e7] [cursor=pointer]
- button "CSRF token으로 상태 변경" [ref=f36e8] [cursor=pointer]
@@ -0,0 +1 @@
- generic [active] [ref=f37e1]: "{\"pattern\":\"AP3-backend-for-frontend\",\"principal\":\"labuser\",\"accessTokenStoredOnServer\":true,\"refreshTokenStoredOnServer\":true,\"browserTokenCount\":0,\"csrfProtectionEnabled\":true}"
@@ -0,0 +1,7 @@
- main [ref=f38e2]:
- heading "AP3 · Backend-for-Frontend" [level=1] [ref=f38e3]
- paragraph [ref=f38e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
- button "Keycloak 로그인" [ref=f38e5] [cursor=pointer]
- button "token 경계 확인" [ref=f38e6] [cursor=pointer]
- button "BFF 경유 API 호출" [ref=f38e7] [cursor=pointer]
- button "CSRF token으로 상태 변경" [ref=f38e8] [cursor=pointer]
@@ -0,0 +1,7 @@
- main [ref=f39e2]:
- heading "AP3 · Backend-for-Frontend" [level=1] [ref=f39e3]
- paragraph [ref=f39e4]: 브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource Server 요청에 붙입니다.
- button "Keycloak 로그인" [ref=f39e5] [cursor=pointer]
- button "token 경계 확인" [ref=f39e6] [cursor=pointer]
- button "BFF 경유 API 호출" [ref=f39e7] [cursor=pointer]
- button "CSRF token으로 상태 변경" [ref=f39e8] [cursor=pointer]
+5 -19
View File
@@ -15,6 +15,11 @@ Keycloak을 중심으로 네 가지 브라우저 인증 통합 패턴을 같은
- AP3: Backend-for-Frontend (BFF)
- AP4: Edge forward-auth
세션 저장소·refresh token 경쟁·장애 복구는 네 패턴을 가로지르는 별도 축으로
`develop-keycloak-session-store` 브랜치에서 진행합니다. 계획과 진행 상황은
[`docs/session-store-lab-roadmap.md`](docs/session-store-lab-roadmap.md)에
있습니다.
현재 `develop`의 공통 baseline은 Keycloak, PostgreSQL, Spring Boot API,
nginx를 Docker Compose로 실행하는 토대입니다. 패턴별 구현은 이 baseline
위에서 별도 브랜치로 진행합니다.
@@ -103,22 +108,3 @@ Keycloak을 잠시 중지하고 export한 뒤 자동으로 다시 올립니다.
runtime export에는 실제 client secret과 credential hash가 포함될 수 있어
gitignored `build/keycloak-export/`에 권한 `0600`으로만 저장됩니다.
## AP4: oauth2-proxy Edge Forward Auth
`develop-keycloak-pattern4`는 oauth2-proxy와 Nginx `auth_request`
인증을 edge에서 강제하는 패턴입니다.
```bash
./scripts/verify-pattern4.sh
```
첫 feature에서는 oauth2-proxy를 `http://localhost:4180`에 직접 노출해
OIDC redirect/PKCE/callback과 forwarded-user를 분리 확인합니다. 두 번째
feature부터 `http://localhost:8088` Nginx가 단일 진입점이며, 내부
`auth_request`는 브라우저 요청을 login 302로, API 요청을 JSON 401로
구분합니다. 최종 feature에서는 backend와 oauth2-proxy의 호스트 노출을
제거하고 Nginx 헤더 덮어쓰기와 내부 토큰 검증으로 spoofing을 막습니다.
자세한 내용은
[`docs/ap4-edge-forward-auth.md`](docs/ap4-edge-forward-auth.md)를
참고하세요.
@@ -1,6 +1,8 @@
package com.example.keycloakpattern;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import org.springframework.security.core.annotation.AuthenticationPrincipal;
@@ -9,6 +11,8 @@ import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import jakarta.servlet.http.HttpServletRequest;
@RestController
@RequestMapping("/api")
public class ApiController {
@@ -18,6 +22,38 @@ public class ApiController {
return Map.of("status", "ok", "service", "keycloak-pattern-api");
}
/**
* Reflects what actually reached the application after the proxy chain.
*
* <p>The reverse proxy contract is defined in {@code docs/reverse-proxy-headers.md}
* for a single nginx hop. The lab runs {@code nginx -> Traefik -> pod}, so this
* endpoint exists to measure the two-hop result instead of assuming it.
*
* <p>{@code scheme}, {@code secure} and {@code requestUrl} are the values Keycloak
* uses to build the {@code iss} claim and redirect URLs. If forwarded headers are
* lost or rewritten, the mismatch shows up here first.
*/
@GetMapping("/echo")
public Map<String, Object> echo(HttpServletRequest request) {
Map<String, List<String>> headers = new LinkedHashMap<>();
for (String name : Collections.list(request.getHeaderNames())) {
headers.put(name.toLowerCase(), Collections.list(request.getHeaders(name)));
}
Map<String, Object> response = new LinkedHashMap<>();
response.put("headers", headers);
response.put("remoteAddr", request.getRemoteAddr());
// Pod IP. Identifies which replica answered, which is what makes the
// host nginx upstream distribution and the sticky-session switch observable.
response.put("localAddr", request.getLocalAddr());
response.put("scheme", request.getScheme());
response.put("secure", request.isSecure());
response.put("serverName", request.getServerName());
response.put("serverPort", request.getServerPort());
response.put("requestUrl", request.getRequestURL().toString());
return response;
}
@GetMapping("/me")
public Map<String, Object> currentUser(@AuthenticationPrincipal Jwt jwt) {
Map<String, Object> response = new LinkedHashMap<>();
@@ -1,59 +0,0 @@
package com.example.keycloakpattern;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.util.LinkedHashMap;
import java.util.Map;
import jakarta.servlet.http.HttpServletRequest;
import org.springframework.http.ResponseEntity;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
@RestController
public class EdgeIdentityController {
private final byte[] internalAuthToken;
EdgeIdentityController(@Value("${edge.internal-auth-token}") String internalAuthToken) {
if (!hasText(internalAuthToken)) {
throw new IllegalStateException("edge.internal-auth-token must be configured");
}
this.internalAuthToken = internalAuthToken.getBytes(StandardCharsets.UTF_8);
}
@GetMapping("/edge/me")
ResponseEntity<Map<String, Object>> currentUser(HttpServletRequest request) {
String authRequestUser = request.getHeader("X-Auth-Request-User");
if (!hasText(authRequestUser) || !hasValidInternalToken(request)) {
return ResponseEntity.status(401).body(Map.of(
"error",
"trusted edge authentication is required"
));
}
Map<String, Object> response = new LinkedHashMap<>();
response.put("pattern", "AP4-edge-forward-auth");
response.put("user", authRequestUser);
response.put("email", request.getHeader("X-Auth-Request-Email"));
response.put("identityHeader", "X-Auth-Request-User");
return ResponseEntity.ok(response);
}
private boolean hasValidInternalToken(HttpServletRequest request) {
String suppliedToken = request.getHeader("X-Internal-Auth-Token");
if (!hasText(suppliedToken)) {
return false;
}
return MessageDigest.isEqual(
internalAuthToken,
suppliedToken.getBytes(StandardCharsets.UTF_8)
);
}
private static boolean hasText(String value) {
return value != null && !value.isBlank();
}
}
@@ -17,12 +17,8 @@ public class SecurityConfig {
.sessionManagement(session ->
session.sessionCreationPolicy(SessionCreationPolicy.STATELESS))
.authorizeHttpRequests(authorize -> authorize
.requestMatchers(
"/actuator/health",
"/actuator/health/**",
"/api/public",
"/edge/**"
)
.requestMatchers("/actuator/health", "/actuator/health/**", "/api/public",
"/api/echo")
.permitAll()
.anyRequest()
.authenticated())
+10 -3
View File
@@ -1,12 +1,19 @@
server:
port: ${SERVER_PORT:8081}
edge:
internal-auth-token: ${EDGE_INTERNAL_AUTH_TOKEN:}
# Spring ignores X-Forwarded-* unless this is set, so scheme/secure/requestUrl
# report the raw connection by default. Keycloak has the same opt-in as
# KC_PROXY_HEADERS. Flipping this to "native" is what the two-hop measurement
# compares against.
forward-headers-strategy: ${SERVER_FORWARD_HEADERS_STRATEGY:none}
spring:
application:
name: keycloak-pattern-api
jackson:
serialization:
# /api/echo is read by humans and captured as evidence screenshots, so the
# response is indented rather than relying on a browser's JSON viewer.
indent-output: true
security:
oauth2:
resourceserver:
@@ -11,7 +11,7 @@ import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMock
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.web.servlet.MockMvc;
@SpringBootTest(properties = "edge.internal-auth-token=test-internal-edge-token")
@SpringBootTest
@AutoConfigureMockMvc
class ApiSecurityTest {
@@ -25,6 +25,18 @@ class ApiSecurityTest {
.andExpect(jsonPath("$.status").value("ok"));
}
@Test
void echoEndpointReflectsForwardedHeadersWithoutAuthentication() throws Exception {
mockMvc.perform(get("/api/echo")
.header("X-Forwarded-Proto", "https")
.header("X-Forwarded-Host", "app1.example.test"))
.andExpect(status().isOk())
.andExpect(jsonPath("$.headers['x-forwarded-proto'][0]").value("https"))
.andExpect(jsonPath("$.headers['x-forwarded-host'][0]").value("app1.example.test"))
.andExpect(jsonPath("$.requestUrl").exists())
.andExpect(jsonPath("$.remoteAddr").exists());
}
@Test
void protectedEndpointRejectsAnonymousRequests() throws Exception {
mockMvc.perform(get("/api/me"))
@@ -40,36 +52,4 @@ class ApiSecurityTest {
.andExpect(jsonPath("$.subject").value("test-subject"))
.andExpect(jsonPath("$.username").value("regular-user"));
}
@Test
void edgeEndpointRejectsMissingTrustedHeaders() throws Exception {
mockMvc.perform(get("/edge/me"))
.andExpect(status().isUnauthorized());
}
@Test
void edgeEndpointRejectsForgedIdentityWithoutInternalToken() throws Exception {
mockMvc.perform(get("/edge/me")
.header("X-Auth-Request-User", "spoofed-admin"))
.andExpect(status().isUnauthorized());
}
@Test
void edgeEndpointRejectsWrongInternalToken() throws Exception {
mockMvc.perform(get("/edge/me")
.header("X-Auth-Request-User", "spoofed-admin")
.header("X-Internal-Auth-Token", "wrong-token"))
.andExpect(status().isUnauthorized());
}
@Test
void edgeEndpointAcceptsIdentityFromTrustedEdge() throws Exception {
mockMvc.perform(get("/edge/me")
.header("X-Auth-Request-User", "regular-user")
.header("X-Auth-Request-Email", "regular-user@example.test")
.header("X-Internal-Auth-Token", "test-internal-edge-token"))
.andExpect(status().isOk())
.andExpect(jsonPath("$.user").value("regular-user"))
.andExpect(jsonPath("$.identityHeader").value("X-Auth-Request-User"));
}
}
+1
View File
@@ -0,0 +1 @@
target/
+14
View File
@@ -0,0 +1,14 @@
FROM maven:3.9.11-eclipse-temurin-21-alpine AS build
WORKDIR /workspace
COPY pom.xml .
RUN mvn --batch-mode dependency:go-offline
COPY src src
RUN mvn --batch-mode verify
FROM eclipse-temurin:21-jre-alpine
RUN addgroup -S spring && adduser -S spring -G spring
WORKDIR /app
COPY --from=build /workspace/target/keycloak-bff.jar app.jar
USER spring:spring
EXPOSE 8083
ENTRYPOINT ["java", "-jar", "/app/app.jar"]
+90
View File
@@ -0,0 +1,90 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>3.5.16</version>
<relativePath/>
</parent>
<groupId>com.example</groupId>
<artifactId>keycloak-bff</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>keycloak-bff</name>
<properties>
<java.version>21</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-oauth2-client</artifactId>
</dependency>
<!-- B-1: Application Session 을 Redis 로 옮긴다.
spring-session-data-redis 가 SessionRepository 를 갈아끼우고,
spring-boot-starter-data-redis 가 연결(Lettuce)을 제공한다.
둘 다 있어야 자동구성이 걸린다 — 하나만 넣으면 조용히 in-memory 로 남는다. -->
<dependency>
<groupId>org.springframework.session</groupId>
<artifactId>spring-session-data-redis</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<!-- B-2: OAuth2AuthorizedClient 를 PostgreSQL 로 옮긴다.
Q3 가 후보로 든 "Redis 와 JDBC 중 무엇" 에서 JDBC 쪽이며,
JdbcOAuth2AuthorizedClientService 는 같은 인터페이스라
컨트롤러를 바꾸지 않아도 된다. -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<finalName>keycloak-bff</finalName>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
@@ -0,0 +1,12 @@
package com.example.keycloakpattern.bff;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class BffApplication {
public static void main(String[] args) {
SpringApplication.run(BffApplication.class, args);
}
}
@@ -0,0 +1,112 @@
package com.example.keycloakpattern.bff;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.concurrent.atomic.AtomicReference;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.CacheControl;
import org.springframework.http.HttpHeaders;
import org.springframework.http.ResponseEntity;
import org.springframework.security.core.Authentication;
import org.springframework.security.oauth2.client.OAuth2AuthorizeRequest;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClient;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClientManager;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClientService;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestClient;
import org.springframework.web.server.ResponseStatusException;
import static org.springframework.http.HttpStatus.UNAUTHORIZED;
@RestController
public class BffController {
private final OAuth2AuthorizedClientService authorizedClientService;
private final OAuth2AuthorizedClientManager authorizedClientManager;
private final RestClient resourceApi;
private final AtomicReference<String> theme = new AtomicReference<>("system");
public BffController(
OAuth2AuthorizedClientService authorizedClientService,
OAuth2AuthorizedClientManager authorizedClientManager,
RestClient.Builder restClientBuilder,
@Value("${resource-api.base-url}") String resourceApiBaseUrl
) {
this.authorizedClientService = authorizedClientService;
this.authorizedClientManager = authorizedClientManager;
this.resourceApi = restClientBuilder.baseUrl(resourceApiBaseUrl).build();
}
@GetMapping("/bff/token-boundary")
ResponseEntity<Map<String, Object>> tokenBoundary(Authentication authentication) {
OAuth2AuthorizedClient client = authorizedClientService.loadAuthorizedClient(
"keycloak",
authentication.getName()
);
Map<String, Object> response = new LinkedHashMap<>();
response.put("pattern", "AP3-backend-for-frontend");
response.put("principal", authentication.getName());
response.put("accessTokenStoredOnServer", client != null
&& client.getAccessToken() != null);
response.put("refreshTokenStoredOnServer", client != null
&& client.getRefreshToken() != null);
response.put("browserTokenCount", 0);
response.put("csrfProtectionEnabled", true);
return ResponseEntity.ok()
.cacheControl(CacheControl.noStore())
.header("Pragma", "no-cache")
.body(response);
}
@GetMapping("/bff/api/me")
ResponseEntity<?> currentUser(Authentication authentication) {
OAuth2AuthorizedClient client = authorizedClient(authentication);
return resourceApi.get()
.uri("/api/me")
.header(
HttpHeaders.AUTHORIZATION,
"Bearer " + client.getAccessToken().getTokenValue()
)
.retrieve()
.toEntity(Map.class);
}
@PostMapping("/bff/api/preferences")
Map<String, Object> updatePreference(
Authentication authentication,
@RequestParam(defaultValue = "system") String theme
) {
this.theme.set(theme);
return Map.of(
"updated", true,
"theme", this.theme.get(),
"principal", authentication.getName()
);
}
@GetMapping("/bff/api/preferences")
Map<String, String> preference() {
return Map.of("theme", theme.get());
}
private OAuth2AuthorizedClient authorizedClient(Authentication authentication) {
OAuth2AuthorizeRequest request = OAuth2AuthorizeRequest
.withClientRegistrationId("keycloak")
.principal(authentication)
.build();
OAuth2AuthorizedClient client = authorizedClientManager.authorize(request);
if (client == null || client.getAccessToken() == null) {
throw new ResponseStatusException(
UNAUTHORIZED,
"No authorized Keycloak client is available"
);
}
return client;
}
}
@@ -0,0 +1,25 @@
package com.example.keycloakpattern.bff;
import java.util.Map;
import org.springframework.http.CacheControl;
import org.springframework.http.ResponseEntity;
import org.springframework.security.web.csrf.CsrfToken;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
@RestController
public class CsrfController {
@GetMapping("/bff/csrf")
ResponseEntity<Map<String, String>> csrf(CsrfToken csrfToken) {
return ResponseEntity.ok()
.cacheControl(CacheControl.noStore())
.header("Pragma", "no-cache")
.body(Map.of(
"headerName", csrfToken.getHeaderName(),
"parameterName", csrfToken.getParameterName(),
"token", csrfToken.getToken()
));
}
}
@@ -0,0 +1,108 @@
package com.example.keycloakpattern.bff;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.oauth2.client.AuthorizedClientServiceOAuth2AuthorizedClientManager;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClientManager;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClientProvider;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClientProviderBuilder;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClientService;
import org.springframework.security.oauth2.client.JdbcOAuth2AuthorizedClientService;
import org.springframework.security.oauth2.client.registration.ClientRegistrationRepository;
import org.springframework.security.oauth2.client.web.DefaultOAuth2AuthorizationRequestResolver;
import org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestCustomizers;
import org.springframework.security.web.SecurityFilterChain;
import org.springframework.security.web.csrf.CookieCsrfTokenRepository;
import org.springframework.jdbc.core.JdbcOperations;
@Configuration
public class SecurityConfig {
/**
* B-2 — authorized client 를 프로세스 메모리에서 PostgreSQL 로 옮긴다.
*
* B-1 에서 Application Session 만 Redis 로 옮겼더니, 사용자는 로그인
* 상태로 보이는데 BFF 에는 access token 이 없는 상태가 만들어졌다.
* 두 상태의 저장소를 **각각** 정해야 한다는 Q3 의 지적이 그대로 나타난 것이다.
*
* 주의 — 이것이 고치는 것과 고치지 못하는 것이 다르다.
* 고친다 : 인스턴스 간 공유. 어느 replica 로 가도 같은 토큰을 본다.
* 못 고친다: 조회 키. JdbcOAuth2AuthorizedClientService 도
* (clientRegistrationId, principalName) 으로 찾으므로
* 같은 사용자의 두 브라우저는 여전히 한 항목을 공유한다.
*/
@Bean
OAuth2AuthorizedClientService authorizedClientService(
JdbcOperations jdbcOperations,
ClientRegistrationRepository clientRegistrationRepository
) {
return new JdbcOAuth2AuthorizedClientService(jdbcOperations, clientRegistrationRepository);
}
@Bean
SecurityFilterChain bffSecurity(
HttpSecurity http,
ClientRegistrationRepository clientRegistrationRepository
) throws Exception {
DefaultOAuth2AuthorizationRequestResolver authorizationRequestResolver =
new DefaultOAuth2AuthorizationRequestResolver(
clientRegistrationRepository,
"/oauth2/authorization"
);
authorizationRequestResolver.setAuthorizationRequestCustomizer(
OAuth2AuthorizationRequestCustomizers.withPkce()
);
CookieCsrfTokenRepository csrfTokenRepository =
CookieCsrfTokenRepository.withHttpOnlyFalse();
csrfTokenRepository.setCookiePath("/");
return http
.csrf(csrf -> csrf
.csrfTokenRepository(csrfTokenRepository)
.csrfTokenRequestHandler(new SpaCsrfTokenRequestHandler()))
.authorizeHttpRequests(authorize -> authorize
.requestMatchers(
"/",
"/index.html",
"/app.js",
"/favicon.ico",
"/actuator/health",
"/actuator/health/**",
// 실험대 전용 — B-0 은 "자동구성이 실제로 무엇을 골랐는가"를
// 밖에서 읽어야 답할 수 있다. 운영에서는 절대 열지 않는다:
// /actuator/beans 와 /actuator/env 는 내부 구조와 설정값을
// 그대로 드러낸다.
"/actuator/**"
)
.permitAll()
.anyRequest()
.authenticated())
.oauth2Login(oauth2 -> oauth2
.authorizationEndpoint(endpoint -> endpoint
.authorizationRequestResolver(authorizationRequestResolver))
.defaultSuccessUrl("/", true))
.build();
}
@Bean
OAuth2AuthorizedClientManager authorizedClientManager(
ClientRegistrationRepository clientRegistrationRepository,
OAuth2AuthorizedClientService authorizedClientService
) {
OAuth2AuthorizedClientProvider authorizedClientProvider =
OAuth2AuthorizedClientProviderBuilder.builder()
.authorizationCode()
.refreshToken()
.build();
AuthorizedClientServiceOAuth2AuthorizedClientManager manager =
new AuthorizedClientServiceOAuth2AuthorizedClientManager(
clientRegistrationRepository,
authorizedClientService
);
manager.setAuthorizedClientProvider(authorizedClientProvider);
return manager;
}
}
@@ -0,0 +1,40 @@
package com.example.keycloakpattern.bff;
import java.util.function.Supplier;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.springframework.security.web.csrf.CsrfToken;
import org.springframework.security.web.csrf.CsrfTokenRequestAttributeHandler;
import org.springframework.security.web.csrf.CsrfTokenRequestHandler;
import org.springframework.security.web.csrf.XorCsrfTokenRequestAttributeHandler;
import org.springframework.util.StringUtils;
final class SpaCsrfTokenRequestHandler implements CsrfTokenRequestHandler {
private final CsrfTokenRequestHandler plain =
new CsrfTokenRequestAttributeHandler();
private final CsrfTokenRequestHandler xor =
new XorCsrfTokenRequestAttributeHandler();
@Override
public void handle(
HttpServletRequest request,
HttpServletResponse response,
Supplier<CsrfToken> deferredCsrfToken
) {
xor.handle(request, response, deferredCsrfToken);
}
@Override
public String resolveCsrfTokenValue(
HttpServletRequest request,
CsrfToken csrfToken
) {
if (StringUtils.hasText(request.getHeader(csrfToken.getHeaderName()))) {
return plain.resolveCsrfTokenValue(request, csrfToken);
}
return xor.resolveCsrfTokenValue(request, csrfToken);
}
}
+80
View File
@@ -0,0 +1,80 @@
server:
port: ${SERVER_PORT:8083}
servlet:
session:
cookie:
name: AP3_SESSION
http-only: true
same-site: lax
spring:
application:
name: keycloak-bff
datasource:
# B-2: authorized client 전용. Keycloak 과 같은 PostgreSQL 인스턴스지만
# 테이블이 다르다(oauth2_authorized_client). 운영이라면 분리를 검토한다.
url: ${BFF_DB_URL:jdbc:postgresql://localhost:5432/keycloak}
username: ${BFF_DB_USER:keycloak}
password: ${BFF_DB_PASSWORD:keycloak}
sql:
init:
# Spring Security 가 제공하는 DDL 을 그대로 쓴다.
# always 로 두면 매 기동마다 실행되므로 CREATE TABLE IF NOT EXISTS 가 아닌
# 스크립트에서는 실패한다 → continue-on-error 로 넘긴다.
mode: ${SPRING_SQL_INIT_MODE:always}
# ★ PostgreSQL 은 -postgres 판본을 써야 한다. 기본 판본은 `blob` 타입을
# 쓰는데 PostgreSQL 에는 그 타입이 없다(`bytea` 다). continue-on-error 가
# 그 실패를 삼켜서 "테이블이 조용히 안 생기는" 상태가 됐었다.
schema-locations: classpath:org/springframework/security/oauth2/client/oauth2-client-schema-postgres.sql
continue-on-error: true
data:
redis:
host: ${REDIS_HOST:localhost}
port: ${REDIS_PORT:6379}
session:
# Application Session 만 Redis 로 간다. OAuth2AuthorizedClient 는
# 이 설정과 무관하며 여전히 InMemory 다 — 조회 키가 다르기 때문이다(B-0).
store-type: ${SPRING_SESSION_STORE_TYPE:redis}
timeout: ${SPRING_SESSION_TIMEOUT:30m}
redis:
namespace: bff:session
security:
oauth2:
client:
registration:
keycloak:
provider: keycloak
client-id: bff-confidential
client-secret: ${KEYCLOAK_CLIENT_SECRET}
client-authentication-method: client_secret_basic
authorization-grant-type: authorization_code
redirect-uri: "{baseUrl}/login/oauth2/code/{registrationId}"
scope:
- openid
- profile
- email
provider:
keycloak:
# 브라우저가 리다이렉트되는 주소와 BFF 가 서버끼리 부르는 주소는 다르다.
# 앞의 것은 외부에서 닿는 이름이어야 하고, 뒤의 것은 클러스터 안 주소여도 된다.
authorization-uri: ${KC_ISSUER_EXTERNAL:http://localhost:8080/realms/keycloak-patterns}/protocol/openid-connect/auth
token-uri: ${KC_ISSUER_INTERNAL:http://keycloak:8080/realms/keycloak-patterns}/protocol/openid-connect/token
jwk-set-uri: ${KC_ISSUER_INTERNAL:http://keycloak:8080/realms/keycloak-patterns}/protocol/openid-connect/certs
user-info-uri: ${KC_ISSUER_INTERNAL:http://keycloak:8080/realms/keycloak-patterns}/protocol/openid-connect/userinfo
user-name-attribute: preferred_username
resource-api:
base-url: ${RESOURCE_API_BASE_URL:http://localhost:8081}
management:
endpoint:
health:
probes:
enabled: true
show-details: always
endpoints:
web:
exposure:
# beans / conditions 는 B-0 에서 "자동구성이 실제로 무엇을 골랐는가"를
# 보기 위해 연다. 운영에 그대로 두면 내부 구조가 노출된다.
include: health,info,beans,conditions,env
+59
View File
@@ -0,0 +1,59 @@
const result = document.querySelector("#result");
function render(value) {
result.textContent = JSON.stringify(value, null, 2);
}
function readCookie(name) {
const prefix = `${encodeURIComponent(name)}=`;
const value = document.cookie
.split("; ")
.find((cookie) => cookie.startsWith(prefix));
return value ? decodeURIComponent(value.slice(prefix.length)) : null;
}
async function request(path, options = {}) {
const response = await fetch(path, {
...options,
headers: { Accept: "application/json", ...options.headers },
});
if (response.redirected || response.status === 401) {
window.location.assign("/oauth2/authorization/keycloak");
return null;
}
const body = await response.json();
render({ status: response.status, ...body });
return { response, body };
}
document.querySelector("#login").addEventListener("click", () => {
window.location.assign("/oauth2/authorization/keycloak");
});
document.querySelector("#inspect").addEventListener("click", () => {
void request("/bff/token-boundary");
});
document.querySelector("#call-bff").addEventListener("click", () => {
void request("/bff/api/me");
});
document.querySelector("#change-with-csrf").addEventListener("click", async () => {
const csrfResponse = await fetch("/bff/csrf", {
headers: { Accept: "application/json" },
});
const csrf = await csrfResponse.json();
const csrfToken = readCookie("XSRF-TOKEN");
if (!csrfToken) {
render({ status: 500, error: "XSRF-TOKEN cookie was not created" });
return;
}
await request("/bff/api/preferences", {
method: "POST",
body: new URLSearchParams({ theme: "dark" }),
headers: {
"Content-Type": "application/x-www-form-urlencoded",
[csrf.headerName]: csrfToken,
},
});
});
+31
View File
@@ -0,0 +1,31 @@
<!doctype html>
<html lang="ko">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>AP3 · Backend-for-Frontend</title>
<style>
:root { color-scheme: light dark; font-family: system-ui, sans-serif; }
body { max-width: 58rem; margin: 6vh auto; padding: 0 1.5rem; line-height: 1.6; }
button { margin: 0 0.5rem 0.5rem 0; padding: 0.6rem 0.9rem; cursor: pointer; }
pre { min-height: 9rem; padding: 1rem; border-radius: 0.4rem;
background: color-mix(in srgb, CanvasText 9%, Canvas); white-space: pre-wrap; }
</style>
</head>
<body>
<main>
<h1>AP3 · Backend-for-Frontend</h1>
<p>
브라우저에는 OAuth token이 전혀 전달되지 않습니다. HttpOnly session
cookie로 BFF만 호출하고, BFF가 서버 보관 access token을 Resource
Server 요청에 붙입니다.
</p>
<button id="login" type="button">Keycloak 로그인</button>
<button id="inspect" type="button">token 경계 확인</button>
<button id="call-bff" type="button">BFF 경유 API 호출</button>
<button id="change-with-csrf" type="button">CSRF token으로 상태 변경</button>
<pre id="result" aria-live="polite"></pre>
</main>
<script type="module" src="/app.js"></script>
</body>
</html>
@@ -0,0 +1,99 @@
package com.example.keycloakpattern.bff;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
import static org.springframework.security.test.web.servlet.request.SecurityMockMvcRequestPostProcessors.oidcLogin;
import static org.springframework.security.test.web.servlet.request.SecurityMockMvcRequestPostProcessors.csrf;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.header;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClient;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClientManager;
import org.springframework.security.oauth2.client.OAuth2AuthorizedClientService;
import org.springframework.security.oauth2.core.OAuth2AccessToken;
import org.springframework.security.oauth2.core.OAuth2RefreshToken;
import org.springframework.test.context.bean.override.mockito.MockitoBean;
import org.springframework.test.web.servlet.MockMvc;
@SpringBootTest(properties = {
"KEYCLOAK_CLIENT_SECRET=test-only-secret",
// 테스트는 Redis 를 띄우지 않는다. store-type=none 이면 자동구성이
// 서블릿 컨테이너 기본 세션으로 되돌아가 컨텍스트가 뜬다.
"spring.session.store-type=none",
// 테스트에는 PostgreSQL 이 없다. H2 로 대신하고 Spring Security 의
// DDL 을 그대로 태워 JdbcOAuth2AuthorizedClientService 가 뜨게 한다.
"spring.datasource.url=jdbc:h2:mem:bfftest;DB_CLOSE_DELAY=-1",
"spring.datasource.username=sa",
"spring.datasource.password=",
"spring.sql.init.mode=always",
"resource-api.base-url=http://127.0.0.1:9"
})
@AutoConfigureMockMvc
class BffControllerTest {
@Autowired
private MockMvc mockMvc;
@MockitoBean
private OAuth2AuthorizedClientService authorizedClientService;
@MockitoBean
private OAuth2AuthorizedClientManager authorizedClientManager;
@Test
void reportsServerTokenCustodyWithoutReturningTokens() throws Exception {
OAuth2AuthorizedClient client = mock(OAuth2AuthorizedClient.class);
when(client.getAccessToken()).thenReturn(mock(OAuth2AccessToken.class));
when(client.getRefreshToken()).thenReturn(mock(OAuth2RefreshToken.class));
when(authorizedClientService.loadAuthorizedClient("keycloak", "test-subject"))
.thenReturn(client);
mockMvc.perform(get("/bff/token-boundary").with(oidcLogin()
.idToken(token -> token.subject("test-subject"))))
.andExpect(status().isOk())
.andExpect(header().string("Cache-Control", "no-store"))
.andExpect(jsonPath("$.accessTokenStoredOnServer").value(true))
.andExpect(jsonPath("$.refreshTokenStoredOnServer").value(true))
.andExpect(jsonPath("$.browserTokenCount").value(0))
.andExpect(jsonPath("$.csrfProtectionEnabled").value(true))
.andExpect(jsonPath("$.access_token").doesNotExist())
.andExpect(jsonPath("$.refresh_token").doesNotExist());
}
@Test
void rejectsStateChangeWithoutCsrfToken() throws Exception {
mockMvc.perform(post("/bff/api/preferences")
.param("theme", "attacker")
.with(oidcLogin().idToken(token -> token.subject("test-subject"))))
.andExpect(status().isForbidden());
}
@Test
void acceptsStateChangeWithCsrfToken() throws Exception {
mockMvc.perform(post("/bff/api/preferences")
.param("theme", "dark")
.with(oidcLogin().idToken(token -> token.subject("test-subject")))
.with(csrf()))
.andExpect(status().isOk())
.andExpect(jsonPath("$.updated").value(true))
.andExpect(jsonPath("$.theme").value("dark"));
}
@Test
void exposesSpaCsrfTokenWithoutCaching() throws Exception {
mockMvc.perform(get("/bff/csrf").with(oidcLogin()
.idToken(token -> token.subject("test-subject"))))
.andExpect(status().isOk())
.andExpect(header().string("Cache-Control", "no-store"))
.andExpect(header().exists("Set-Cookie"))
.andExpect(jsonPath("$.headerName").value("X-XSRF-TOKEN"))
.andExpect(jsonPath("$.token").isNotEmpty());
}
}
+136
View File
@@ -0,0 +1,136 @@
# Session store lab
세션 저장소·refresh token 경쟁·장애 복구를 검증하는 2노드 k3s 실험대.
네 인증 패턴(AP1~AP4)을 가로지르는 공통층이므로 별도 축으로 관리한다.
이 문서는 **절차**만 담는다.
| 문서 | 내용 |
|---|---|
| [`docs/session-store-lab-roadmap.md`](../../docs/session-store-lab-roadmap.md) | 이 축의 계획과 진행 상황 |
| [`docs/session-lab-concepts.md`](../../docs/session-lab-concepts.md) | 등장 개념 전체 |
| [`docs/session-lab-operations.md`](../../docs/session-lab-operations.md) | 관측 도구 · 자주 쓰는 명령 · 훈련 |
| [`docs/two-hop-proxy-header-contract.md`](../../docs/two-hop-proxy-header-contract.md) | 첫 실험 결과 |
## 토폴로지
```
브라우저 / SSH (tailnet)
│ https://{auth,app1,app2}.hyeonworks.com → 100.83.212.4
lab host ── nginx :443 TLS 종료 · X-Forwarded-* 주입
│ nginx :80 301 → https
│ virbr0 192.168.122.0/24 (libvirt NAT)
├──▶ kc-lab-1 .11 k3s server Traefik :80
└──▶ kc-lab-2 .12 k3s agent Traefik :80
└──▶ Pod
```
`nginx → Traefik` **2홉**이 운영 구조와 같다는 점이 이 배치의 핵심이다.
L7 프록시가 두 겹인 이유는 역할이 다르기 때문이다 — nginx는 바깥세상과의
접점(TLS·인증서·헤더)을, Traefik은 클러스터 내부의 동적 라우팅을 맡는다.
## 구성 요소
| 경로 | 역할 |
|---|---|
| `cloud-init/kc-lab.yaml.example` | 게스트 부트스트랩 템플릿 |
| `host/nginx-keycloak-lab.conf` | lab host의 `sites-available/keycloak-lab` |
| `k8s/echo.yaml` | 2홉 헤더 계약 측정용 워크로드 |
| `scripts/rebuild-seed.sh` | cloud-init 시드 ISO 재생성 + 풀 업로드 |
| `scripts/build-and-import.sh` | 이미지 빌드 → 각 노드 containerd 반입 |
| `scripts/measure-proxy-headers.sh` | 헤더 계약 실측 |
| `scripts/verify-lab.sh` | 인프라 상태 점검 |
## 상태 점검
```bash
./deploy/lab/scripts/verify-lab.sh # lab host 에서
```
**`404`가 성공 신호다.** TLS가 종료되고 Traefik까지 도달했으나 매칭되는
Ingress 규칙이 없다는 뜻이다. `502`나 연결 거부면 체인이 끊긴 것이다.
## 첫 실험 — 2홉 헤더 계약
[`docs/reverse-proxy-headers.md`](../../docs/reverse-proxy-headers.md)의 계약은
nginx **1홉**을 가정하고 쓰였다. 실제 배치는 2홉이므로, nginx가 세팅한
`X-Forwarded-*`를 Traefik이 그대로 넘기는지 덮어쓰는지 **측정해서 확인한다.**
이 결론이 뒤의 모든 실험에 깔린다. Keycloak의 `iss` 클레임, redirect URL,
쿠키 도메인 검증이 전부 이 헤더에 의존하기 때문이다.
```bash
# 워크스테이션: 이미지 빌드 후 두 노드에 반입
./deploy/lab/scripts/build-and-import.sh
# lab host: 배포
kubectl apply -f deploy/lab/k8s/echo.yaml
kubectl -n header-lab rollout status deployment/echo
# 어디서든: 실측
./deploy/lab/scripts/measure-proxy-headers.sh
```
관측 대상은 넷이다.
1. `X-Forwarded-For` — Traefik이 **덧붙이는가 덮어쓰는가**
2. `X-Forwarded-Proto` / `-Host` / `-Port` — 그대로 전달되는가
3. **위조 내성** — 클라이언트가 직접 넣은 `X-Forwarded-*`가 앱까지 도달하는가
4. `scheme` / `secure` / `requestUrl` — Keycloak이 URL을 만들 때 쓰는 값
3번이 신뢰 경계의 핵심이다. 이 헤더들은 누구나 위조할 수 있는 평범한 HTTP
헤더이므로, 신뢰 경계에 선 프록시가 **반드시 덮어써야** 한다.
## 이미지 배포 경로
k3s는 containerd를 쓰고 이 실험대에는 레지스트리가 없다.
```
워크스테이션 docker build → docker save
│ ssh (lab host 경유)
게스트 sudo k3s ctr images import
매니페스트 imagePullPolicy: Never
```
**두 노드 모두에 반입해야 한다.** 스케줄러가 어느 노드에 배치할지 모른다.
Keycloak·PostgreSQL·Redis는 공식 이미지를 그대로 당겨오므로 이 경로가
필요한 것은 자체 빌드 이미지뿐이다.
**lab host에 Docker를 설치하지 않는다.** k3s의 containerd와 이미지 저장소가
갈려서 `docker build`한 이미지를 k3s가 보지 못하게 된다.
## 게스트 재생성
파괴적 실험 후 초기화하는 경로다.
```bash
virsh destroy kc-lab-1
virsh undefine kc-lab-1 # --remove-all-storage 는 시드 ISO 까지 지운다
virsh vol-delete --pool default kc-lab-1.qcow2
./deploy/lab/scripts/rebuild-seed.sh 1 # user-data 를 고쳤을 때만
virt-install --name kc-lab-1 --memory 3584 --vcpus 2 \
--disk size=20,backing_store=/var/lib/libvirt/images/base.qcow2 \
--disk vol=default/seed-kc-lab-1.iso,device=disk,bus=virtio,readonly=on \
--network network=default,mac=52:54:00:aa:bb:11 \
--import --os-variant debian12 --noautoconsole
```
시드는 **virtio 디스크**로 붙인다. `virt-install --cloud-init`은 시드를 SATA
CD-ROM으로 붙이는데, Debian `genericcloud` 이미지는 크기를 줄이려고 물리
하드웨어 드라이버를 제외해서 **AHCI 장치를 보지 못한다.** 그러면 cloud-init이
데이터소스를 찾지 못하고 아무 오류도 남기지 않은 채 종료한다. 증상은
hostname이 `localhost`로 남고 SSH가 `Permission denied (publickey)`로 거부되는
것뿐이다.
게스트에 들어갈 수 없을 때는 화면을 직접 뜬다.
```bash
virsh screenshot kc-lab-1 /tmp/kc1.ppm # 확장자와 무관하게 PNG 로 저장된다
```
`localhost login:`이면 cloud-init 미실행, `kc-lab-1 login:`이면 실행된 것이다.
+37
View File
@@ -0,0 +1,37 @@
#cloud-config
# Template for both lab guests. scripts/rebuild-seed.sh substitutes __NODE__
# and bakes this into a CIDATA seed image.
#
# Copy to kc-lab.yaml and fill the two placeholders. The real file is ignored by
# git because plain_text_passwd is a credential, however disposable.
#
# Indentation is spaces only. YAML forbids tabs, and cloud-init fails silently
# on a parse error: the guest boots as "localhost" with no user and no way in.
hostname: kc-lab-__NODE__
fqdn: kc-lab-__NODE__
manage_etc_hosts: true
users:
- name: donghyeon
groups: [sudo]
shell: /bin/bash
# NOPASSWD is required: the k3s installer and the fault-injection scripts
# run non-interactively and would block on a password prompt.
sudo: ['ALL=(ALL) NOPASSWD:ALL']
# Console-only escape hatch. Without it, a cloud-init failure leaves a guest
# that cannot be logged into at all, so its own failure log is unreadable.
# ssh_pwauth stays false, so this never widens SSH exposure.
lock_passwd: false
plain_text_passwd: CHANGE_ME
ssh_authorized_keys:
# Lab host key: needed because automation runs from the lab host, where
# agent forwarding is not available.
- CHANGE_ME_LAB_HOST_PUBLIC_KEY
# Workstation key: lets ProxyJump reach the guest directly.
- CHANGE_ME_WORKSTATION_PUBLIC_KEY
ssh_pwauth: false
package_update: true
packages:
- curl
- nftables
+56
View File
@@ -0,0 +1,56 @@
# Lab entry point. Deployed on the lab host as
# /etc/nginx/sites-available/keycloak-lab
# and symlinked from sites-enabled/.
#
# Arch does not ship the Debian sites-available convention, so nginx.conf needs
# include /etc/nginx/sites-enabled/*;
# inside its http { } block before this file has any effect.
#
# This is the outer of two L7 hops. It terminates TLS and hands plain HTTP to
# the Traefik instance running on each k3s node.
upstream k3s_traefik {
# Sticky-session switch. Keycloak recommends affinity on AUTH_SESSION_ID;
# ip_hash is the cheap stand-in for a single-browser lab. Leaving it off is
# the interesting case: Infinispan still routes correctly, only slower.
# ip_hash;
server 192.168.122.11:80;
server 192.168.122.12:80;
}
server {
listen 80 default_server;
server_name _;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl default_server;
http2 on;
server_name _;
# fullchain.pem, never cert.pem: omitting the intermediates passes on
# desktop browsers and fails on mobile and curl.
ssl_certificate /etc/letsencrypt/live/auth.hyeonworks.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/auth.hyeonworks.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
location / {
proxy_pass http://k3s_traefik;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Port 443;
# $remote_addr, not $proxy_add_x_forwarded_for. This is the trust
# boundary: a client-supplied X-Forwarded-For must be discarded, not
# extended, or nothing downstream can rely on the value.
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Real-IP $remote_addr;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
}
}
@@ -0,0 +1,46 @@
# Experiment A-1 — cut the JGroups transport (TCP 7800) while leaving discovery alone.
#
# The point is to separate two things that are easy to conflate:
#
# discovery how the nodes FIND each other -> PostgreSQL JGROUPS_PING table
# transport how they actually TALK -> TCP 7800
#
# Blocking only the transport produces a state that cannot happen on a single
# node: both members stay registered in the database, so each believes the other
# exists, yet no message gets through.
#
# kubectl apply -f deploy/lab/k8s/a1-block-jgroups-transport.yaml
# kubectl -n keycloak-lab delete networkpolicy a1-block-jgroups-transport
#
# NetworkPolicy is an ALLOWLIST, not a firewall with deny rules. There is no way
# to write "deny 7800". The moment a pod is selected by a policy carrying
# policyTypes: [Ingress], every inbound port is denied unless a rule permits it.
# So 7800 is blocked by *omission*: 8080 and 9000 are listed, 7800 is not.
#
# That makes the two allow rules load-bearing — get them wrong and the experiment
# measures a dead Keycloak instead of a partitioned cluster:
#
# 8080 the HTTP endpoint. Traefik, the other pod's REST calls, and the probe
# traffic all arrive here.
# 9000 the management port: /health/started, /health/ready, /health/live and
# /metrics. Losing it means the kubelet fails the readiness probe and
# kills the pod — the cluster would break for the wrong reason.
#
# Both rules deliberately omit `from:`, which allows those ports from any source.
# Narrowing the source is not the subject here; the 2-hop experiment already
# established how to do that by label when it matters.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: a1-block-jgroups-transport
namespace: keycloak-lab
spec:
podSelector:
matchLabels:
app: keycloak
policyTypes: [Ingress]
ingress:
- ports:
- { port: 8080, protocol: TCP } # HTTP — must stay open
- { port: 9000, protocol: TCP } # health + metrics — must stay open
# 7800 is absent on purpose. That is the whole experiment.
+214
View File
@@ -0,0 +1,214 @@
# BFF (2 replicas) + Redis, for the B-layer experiments.
#
# The BFF is deployed FIRST WITHOUT any session store wiring. That is deliberate:
# B-0 asks what Spring Boot's autoconfiguration actually picks when nothing is
# configured, and the only honest way to answer is to look at a running instance
# that has been given nothing. Redis is deployed alongside but left unused until
# B-1 turns it on.
#
# kubectl apply -f deploy/lab/k8s/bff-redis.yaml
#
# Image comes from the workstation, not a registry:
# docker build -t keycloak-pattern-bff:lab bff/
# docker save keycloak-pattern-bff:lab | ssh test-server "ssh kc-lab-1 'sudo k3s ctr images import -'"
# (repeat for kc-lab-2)
# so imagePullPolicy must stay Never on both replicas.
apiVersion: v1
kind: Secret
metadata:
name: bff-secrets
namespace: keycloak-lab
type: Opaque
stringData:
# Matches the client created with kcadm in the keycloak-patterns realm.
# Base64 in etcd is not encryption — see D-3.
KEYCLOAK_CLIENT_SECRET: bff-lab-secret
---
# Redis. B-5 measured that turning on AOF with `redis-cli config set` changes
# nothing here, because /data is the container filesystem and dies with the
# container — the appendonlydir was created and then thrown away. Persistence
# configuration without a volume is decoration.
#
# So the volume comes first, and only then does `--appendonly yes` mean anything.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data
namespace: keycloak-lab
spec:
accessModes: [ReadWriteOnce]
storageClassName: local-path
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis
namespace: keycloak-lab
spec:
replicas: 1
selector:
matchLabels: { app: redis }
template:
metadata:
labels: { app: redis }
spec:
# Same node as postgres so a node-loss experiment takes both stores at
# once, matching how A-4 was set up.
nodeSelector:
kubernetes.io/hostname: kc-lab-2
containers:
- name: redis
image: redis:7.4-alpine
# appendfsync everysec 이 기본값이다 — 1초 분량을 잃을 수 있다.
# Keycloak 의 synchronous_commit OFF(A-3)와 같은 모양의 트레이드오프다.
args: ["redis-server", "--appendonly", "yes", "--dir", "/data"]
ports:
- containerPort: 6379
name: redis
readinessProbe:
exec: { command: ["redis-cli", "ping"] }
initialDelaySeconds: 3
volumeMounts:
- name: data
mountPath: /data
resources:
requests: { memory: 32Mi, cpu: 20m }
limits: { memory: 128Mi }
volumes:
- name: data
persistentVolumeClaim:
claimName: redis-data
---
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: keycloak-lab
spec:
selector: { app: redis }
ports:
- port: 6379
targetPort: redis
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: bff
namespace: keycloak-lab
spec:
# Two replicas is the whole point: Q1 and Q2 only exist because a request can
# land on an instance that did not handle the login.
replicas: 2
selector:
matchLabels: { app: bff }
template:
metadata:
labels: { app: bff }
spec:
# Spread across both nodes so "the other instance" is genuinely another
# machine, not another process on the same kernel.
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels: { app: bff }
# 쿠버네티스는 같은 네임스페이스의 Service 마다 Docker link 시절의
# 환경변수를 자동 주입한다: REDIS_PORT=tcp://10.43.57.116:6379.
# 그것이 application.yml 의 ${REDIS_PORT:6379} 를 덮어써서 기동이 실패했다.
# Failed to bind properties under 'spring.data.redis.port' to int:
# Value: "tcp://10.43.57.116:6379"
# 이 주입 자체를 끄는 것이 근본 처방이다. 이름을 바꿔 피하면 다음 사람이
# 같은 함정에 다시 빠진다.
enableServiceLinks: false
containers:
- name: bff
image: keycloak-pattern-bff:lab
imagePullPolicy: Never
ports:
- containerPort: 8083
name: http
env:
# The browser is redirected to the public name; the BFF calls the
# token endpoint over the cluster network. Getting these two the same
# way round is what the 2-hop header experiment was about.
- name: KC_ISSUER_EXTERNAL
value: https://auth.hyeonworks.com/realms/keycloak-patterns
- name: KC_ISSUER_INTERNAL
value: http://keycloak.keycloak-lab.svc:8080/realms/keycloak-patterns
# echo 는 header-lab 네임스페이스의 8081 이다. 다른 네임스페이스의
# 서비스는 <svc>.<ns>.svc 로 부른다. 이름을 틀리면 500 이 나는데
# 원인은 UnresolvedAddressException 이지 토큰 문제가 아니다.
- name: RESOURCE_API_BASE_URL
value: http://echo.header-lab.svc:8081
- name: KEYCLOAK_CLIENT_SECRET
valueFrom:
secretKeyRef: { name: bff-secrets, key: KEYCLOAK_CLIENT_SECRET }
# Spring needs to know it is behind TLS termination, for the same
# reason Keycloak needs KC_PROXY_HEADERS. Without it the redirect_uri
# it builds comes back as http:// and Keycloak rejects it.
- name: SERVER_FORWARD_HEADERS_STRATEGY
value: native
# B-1: Application Session 을 Redis 로 옮긴다.
# OAuth2AuthorizedClient 는 이것으로 옮겨지지 않는다 — 조회 키가
# 다르기 때문이며, B-0 에서 확인한 사실이다.
- name: SPRING_SESSION_STORE_TYPE
value: redis
- name: REDIS_HOST
value: redis.keycloak-lab.svc
- name: REDIS_PORT
value: "6379"
# B-2: authorized client 는 PostgreSQL 로. 세션(Redis)과 다른
# 저장소를 쓰는 것이 Q3 가 말한 "각각 설계한다"의 실물이다.
- name: BFF_DB_URL
value: jdbc:postgresql://postgres.keycloak-lab.svc:5432/keycloak
- name: BFF_DB_USER
value: keycloak
- name: BFF_DB_PASSWORD
valueFrom:
secretKeyRef: { name: keycloak-lab-secrets, key: POSTGRES_PASSWORD }
- name: JAVA_TOOL_OPTIONS
value: "-Xms128m -Xmx320m"
readinessProbe:
httpGet: { path: /actuator/health/readiness, port: http }
initialDelaySeconds: 20
failureThreshold: 30
livenessProbe:
httpGet: { path: /actuator/health/liveness, port: http }
initialDelaySeconds: 60
resources:
requests: { memory: 320Mi, cpu: 100m }
limits: { memory: 512Mi }
---
apiVersion: v1
kind: Service
metadata:
name: bff
namespace: keycloak-lab
spec:
selector: { app: bff }
ports:
- port: 8083
targetPort: http
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: bff
namespace: keycloak-lab
spec:
ingressClassName: traefik
rules:
- host: app1.hyeonworks.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: bff
port:
number: 8083
+62
View File
@@ -0,0 +1,62 @@
# Restrict who may reach the echo pods.
#
# Traefik is configured to trust X-Forwarded-* from the whole pod CIDR, and the
# app's Tomcat valve trusts every private range by default. Both are IP-range
# decisions, so any pod in the cluster can forge those headers by talking to the
# Service directly and bypassing Traefik entirely. Measured, not hypothetical:
#
# kubectl -n header-lab run t --rm -i --restart=Never --image=curlimages/curl -- \
# curl -s http://echo:8081/api/echo -H 'X-Forwarded-Host: evil.example.com'
# → serverName evil.example.com, remoteAddr 1.2.3.4
#
# A NetworkPolicy closes that path. It selects by label rather than IP, so it
# survives pod restarts and rescheduling — unlike the trustedIPs list, which
# could not name Traefik because its IP changes.
#
# "Trusting forwarded headers" and "guaranteeing a proxy sits in front" are a
# pair. Doing only the first leaves this hole.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: echo-allow-traefik-only
namespace: header-lab
spec:
podSelector:
matchLabels:
app: echo
policyTypes:
- Ingress
ingress:
# The proxy itself. namespaceSelector and podSelector in one list item are
# ANDed, so this is "traefik pods in kube-system" and nothing else.
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ports:
- protocol: TCP
port: 8081
# kubelet readiness/liveness probes originate from the node, not from a pod,
# so they need their own rule. Without it the probes fail and the pods are
# restarted in a loop.
#
# The probe's source address is the node's flannel bridge (cni0), which
# holds the first address of that node's /24:
# kc-lab-1 10.42.0.1 kc-lab-2 10.42.1.1
# Listing them as /32 keeps this rule from re-admitting arbitrary pods,
# which a broader 10.42.0.0/16 block would do and would undo the policy.
#
# Adding a node means adding its gateway here. Verify with:
# kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{end}'
- from:
- ipBlock:
cidr: 10.42.0.1/32
- ipBlock:
cidr: 10.42.1.1/32
ports:
- protocol: TCP
port: 8081
+113
View File
@@ -0,0 +1,113 @@
# Header echo workload for the two-hop proxy contract measurement.
#
# browser -> host nginx (TLS termination) -> Traefik -> this pod
#
# The image is built from backend/ and imported straight into each node's
# containerd, so imagePullPolicy must stay Never. See scripts/build-and-import.sh.
apiVersion: v1
kind: Namespace
metadata:
name: header-lab
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: echo
namespace: header-lab
spec:
replicas: 2
selector:
matchLabels:
app: echo
template:
metadata:
labels:
app: echo
spec:
# One replica per node so the sticky-session switch on the host nginx
# upstream has something observable to route between.
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: echo
containers:
- name: echo
image: keycloak-pattern-api:lab
imagePullPolicy: Never
ports:
- containerPort: 8081
name: http
env:
- name: SERVER_PORT
value: "8081"
# "none" makes the app report the raw connection, so scheme/secure/
# requestUrl show what arrives without any forwarded-header handling.
# Set to "native" and redeploy to see the same request interpreted
# with X-Forwarded-* honoured. Keycloak's KC_PROXY_HEADERS is the
# same opt-in, which is why measuring both sides matters here.
- name: SERVER_FORWARD_HEADERS_STRATEGY
value: "native"
# The JVM sizes its heap from the container limit, not the host.
- name: JAVA_TOOL_OPTIONS
value: "-XX:MaxRAMPercentage=70"
# /api/echo is permitAll, so the JWT decoder is never exercised.
# These stay pointed at the future Keycloak service name.
- name: SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI
value: "https://auth.hyeonworks.com/realms/keycloak-patterns"
- name: SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI
value: "https://auth.hyeonworks.com/realms/keycloak-patterns/protocol/openid-connect/certs"
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: http
initialDelaySeconds: 15
periodSeconds: 5
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: http
initialDelaySeconds: 45
periodSeconds: 15
resources:
requests:
memory: 320Mi
cpu: 100m
limits:
memory: 512Mi
---
apiVersion: v1
kind: Service
metadata:
name: echo
namespace: header-lab
spec:
selector:
app: echo
ports:
- port: 8081
targetPort: http
name: http
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: echo
namespace: header-lab
spec:
# k3s ships Traefik as the default ingress controller. Keeping it is what
# makes this lab a faithful two-hop replica.
ingressClassName: traefik
rules:
- host: app1.hyeonworks.com
http:
paths:
- path: /api
pathType: Prefix
backend:
service:
name: echo
port:
number: 8081
+277
View File
@@ -0,0 +1,277 @@
# Keycloak multi-node cluster with PostgreSQL.
#
# Goal of this manifest: two Keycloak pods on two different nodes must discover
# each other and form one Infinispan cluster. Keycloak 26 discovers peers through
# the database (jdbc-ping) rather than multicast, writing to a JGROUPS_PING table,
# but the cluster traffic itself runs over TCP 7800 between the pods. Those are
# two separate mechanisms, which is why "registered in the DB but not clustered"
# is a real failure mode — and one that a single node cannot reproduce.
#
# kubectl apply -f deploy/lab/k8s/keycloak-cluster.yaml
# kubectl -n keycloak-lab rollout status statefulset/keycloak --timeout=600s
#
# Secrets are plain here. Proper secret handling is roadmap item 11; keeping it
# visible for now is deliberate so the gap is obvious rather than forgotten.
apiVersion: v1
kind: Namespace
metadata:
name: keycloak-lab
---
apiVersion: v1
kind: Secret
metadata:
name: keycloak-lab-secrets
namespace: keycloak-lab
type: Opaque
stringData:
POSTGRES_PASSWORD: lab-postgres-change-me
KC_BOOTSTRAP_ADMIN_PASSWORD: lab-admin-change-me
---
# PostgreSQL. local-path binds the volume to whichever node the pod lands on, so
# the database is effectively pinned to one node. That is not a flaw here: it is
# what makes "the database node dies" a meaningful experiment later.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgres-data
namespace: keycloak-lab
spec:
accessModes: [ReadWriteOnce]
storageClassName: local-path
resources:
requests:
storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
namespace: keycloak-lab
spec:
replicas: 1
strategy:
type: Recreate # RWO volume cannot be mounted by two pods at once
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:16-alpine
ports:
- containerPort: 5432
name: postgres
env:
- name: POSTGRES_DB
value: keycloak
- name: POSTGRES_USER
value: keycloak
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: keycloak-lab-secrets
key: POSTGRES_PASSWORD
# The image refuses to initialise into a non-empty mount, and
# local-path volumes are clean, but this keeps the data one level
# down so a lost+found or similar never blocks initdb.
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
readinessProbe:
exec:
command: ["sh", "-c", "pg_isready -U keycloak -d keycloak"]
initialDelaySeconds: 10
periodSeconds: 5
resources:
requests:
memory: 192Mi
cpu: 50m
limits:
memory: 512Mi
volumes:
- name: data
persistentVolumeClaim:
claimName: postgres-data
---
apiVersion: v1
kind: Service
metadata:
name: postgres
namespace: keycloak-lab
spec:
selector:
app: postgres
ports:
- port: 5432
targetPort: postgres
---
# Keycloak. A StatefulSet rather than a Deployment so each pod keeps a stable
# name (keycloak-0, keycloak-1); cluster membership is far easier to read in
# logs and in the JGROUPS_PING table when the identities do not churn.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: keycloak
namespace: keycloak-lab
spec:
serviceName: keycloak-headless
replicas: 2
podManagementPolicy: Parallel # both pods start together, so they race to
# register — which is the interesting case
selector:
matchLabels:
app: keycloak
template:
metadata:
labels:
app: keycloak
spec:
# One pod per node. Two pods on one node would share a kernel and make the
# 7800 blocking experiment meaningless.
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: keycloak
containers:
- name: keycloak
image: quay.io/keycloak/keycloak:26.7.0
# "start", not "start-dev". Dev mode forces cache=local and there is
# no cluster to form at all.
args: ["start"]
ports:
- containerPort: 8080
name: http
- containerPort: 9000
name: management
- containerPort: 7800
name: jgroups
env:
- name: KC_DB
value: postgres
- name: KC_DB_URL
value: jdbc:postgresql://postgres:5432/keycloak
- name: KC_DB_USERNAME
value: keycloak
- name: KC_DB_PASSWORD
valueFrom:
secretKeyRef:
name: keycloak-lab-secrets
key: POSTGRES_PASSWORD
# Settings confirmed by the two-hop header measurement.
# KC_HOSTNAME carries the full external URL, which pins scheme and
# host for issuer and redirect URLs regardless of headers.
# KC_PROXY_HEADERS is the separate opt-in that lets the forwarded
# client address through — the same kind of switch as Spring's
# forward-headers-strategy. See docs/two-hop-proxy-header-contract.md.
- name: KC_HOSTNAME
value: https://auth.hyeonworks.com
- name: KC_HOSTNAME_STRICT
value: "true"
- name: KC_PROXY_HEADERS
value: xforwarded
- name: KC_HTTP_ENABLED
value: "true"
- name: KC_HEALTH_ENABLED
value: "true"
- name: KC_METRICS_ENABLED
value: "true"
# Without an explicit cap the JVM sizes its heap from the container
# limit and this lab has roughly 3.8GB of guest headroom in total.
- name: JAVA_OPTS_KC_HEAP
value: "-Xms256m -Xmx512m"
- name: KC_BOOTSTRAP_ADMIN_USERNAME
value: admin
- name: KC_BOOTSTRAP_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: keycloak-lab-secrets
key: KC_BOOTSTRAP_ADMIN_PASSWORD
# Keycloak serves health and metrics on the management port (9000),
# not on 8080, since version 25.
startupProbe:
httpGet:
path: /health/started
port: management
periodSeconds: 10
failureThreshold: 60 # first boot runs an implicit build
readinessProbe:
httpGet:
path: /health/ready
port: management
periodSeconds: 10
livenessProbe:
httpGet:
path: /health/live
port: management
periodSeconds: 30
resources:
requests:
memory: 640Mi
cpu: 100m
limits:
memory: 900Mi
---
# Headless service. Not required for jdbc-ping discovery, which goes through the
# database, but it gives each pod a stable DNS name for direct inspection.
apiVersion: v1
kind: Service
metadata:
name: keycloak-headless
namespace: keycloak-lab
spec:
clusterIP: None
selector:
app: keycloak
ports:
- port: 8080
targetPort: http
name: http
- port: 9000
targetPort: management
name: management
---
apiVersion: v1
kind: Service
metadata:
name: keycloak
namespace: keycloak-lab
spec:
selector:
app: keycloak
ports:
- port: 8080
targetPort: http
name: http
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: keycloak
namespace: keycloak-lab
spec:
ingressClassName: traefik
rules:
- host: auth.hyeonworks.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: keycloak
port:
number: 8080
+373
View File
@@ -0,0 +1,373 @@
# Prometheus + node-exporter + Grafana.
#
# Purpose: during a fault-injection experiment, know *which signal moved first*.
# Without a metrics store the only record is whatever scrolled past in a terminal,
# and "the cluster recovered in about a minute" is not a measurement.
#
# kubectl apply -f deploy/lab/k8s/observability.yaml
# kubectl -n observability rollout status deployment/prometheus --timeout=300s
#
# Placement decision — Prometheus and Grafana are pinned to the control-plane
# node (kc-lab-1). An observability stack must not share a failure domain with
# the thing it observes. With only two nodes that cannot be fully avoided, so the
# rule here is: the node that gets killed in experiments is the *agent*
# (kc-lab-2, holding keycloak-0 and postgres), and everything needed to watch
# that happen lives on the server node.
apiVersion: v1
kind: Namespace
metadata:
name: observability
---
# Prometheus discovers scrape targets by querying the Kubernetes API, so it
# needs read access to nodes, services, endpoints and pods. Without this the
# kubernetes_sd_configs below silently return no targets.
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: observability
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups: [""]
# nodes/proxy is required in addition to nodes/metrics: the kubelet job
# reaches each node through the API server's proxy subresource
# (/api/v1/nodes/<name>/proxy/metrics). Without it every kubelet target
# fails with 403 Forbidden while the other jobs stay green — a partial
# failure that is easy to miss unless the target list is checked.
resources: [nodes, nodes/metrics, nodes/proxy, services, endpoints, pods]
verbs: [get, list, watch]
- nonResourceURLs: ["/metrics"]
verbs: [get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: observability
---
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
namespace: observability
data:
prometheus.yml: |
global:
# 15s is short for production but right here: a node loss should show up
# within a couple of samples, not a minute later.
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
# Prometheus scraping itself. Useful as a control: if this target is down,
# the problem is Prometheus, not the thing being measured.
- job_name: prometheus
static_configs:
- targets: ['localhost:9090']
# Keycloak. Metrics live on the management port 9000, not 8080 — the same
# split that the health probes use. KC_METRICS_ENABLED=true is already set
# on the StatefulSet.
#
# Discovery is by endpoints rather than a static list because pod IPs
# change on every restart; that was observed directly when the lab was
# power-cycled and every pod came back with a new address.
- job_name: keycloak
kubernetes_sd_configs:
- role: endpoints
namespaces:
names: [keycloak-lab]
relabel_configs:
- source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: keycloak-headless;management
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
- source_labels: [__meta_kubernetes_pod_node_name]
target_label: node
# node-exporter, one per node via DaemonSet. This is what answers
# "did the machine die or did the process die".
- job_name: node-exporter
kubernetes_sd_configs:
- role: endpoints
namespaces:
names: [observability]
relabel_configs:
- source_labels: [__meta_kubernetes_service_name]
action: keep
regex: node-exporter
- source_labels: [__meta_kubernetes_pod_node_name]
target_label: node
# The kubelet's own metrics, reached through the API server proxy so no
# extra port needs opening.
- job_name: kubelet
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: prometheus-data
namespace: observability
spec:
accessModes: [ReadWriteOnce]
storageClassName: local-path
resources:
requests:
storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus
namespace: observability
spec:
replicas: 1
strategy:
type: Recreate # RWO volume; two pods cannot mount it at once
selector:
matchLabels:
app: prometheus
template:
metadata:
labels:
app: prometheus
spec:
serviceAccountName: prometheus
# See the placement note at the top of this file.
nodeSelector:
node-role.kubernetes.io/control-plane: "true"
securityContext:
fsGroup: 65534 # the image runs as nobody and must own the volume
containers:
- name: prometheus
image: prom/prometheus:v3.1.0
args:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.path=/prometheus
# 7 days is far more than an experiment needs and keeps the volume
# small enough that it never becomes the reason a node fills up.
- --storage.tsdb.retention.time=7d
- --web.enable-lifecycle
ports:
- containerPort: 9090
name: http
volumeMounts:
- name: config
mountPath: /etc/prometheus
- name: data
mountPath: /prometheus
readinessProbe:
httpGet: { path: /-/ready, port: http }
initialDelaySeconds: 10
livenessProbe:
httpGet: { path: /-/healthy, port: http }
initialDelaySeconds: 30
resources:
requests: { memory: 256Mi, cpu: 50m }
limits: { memory: 640Mi }
volumes:
- name: config
configMap:
name: prometheus-config
- name: data
persistentVolumeClaim:
claimName: prometheus-data
---
apiVersion: v1
kind: Service
metadata:
name: prometheus
namespace: observability
spec:
selector:
app: prometheus
ports:
- port: 9090
targetPort: http
---
# node-exporter. A DaemonSet so every node reports, including one that is about
# to be killed — the last samples before it goes silent are the interesting part.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: node-exporter
namespace: observability
spec:
selector:
matchLabels:
app: node-exporter
template:
metadata:
labels:
app: node-exporter
spec:
# Host namespaces: the point is to measure the machine, not the container.
hostNetwork: true
hostPID: true
tolerations:
- operator: Exists # must also run on tainted nodes
containers:
- name: node-exporter
image: prom/node-exporter:v1.8.2
args:
- --path.procfs=/host/proc
- --path.sysfs=/host/sys
- --path.rootfs=/host/root
- --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
ports:
- containerPort: 9100
name: metrics
hostPort: 9100
volumeMounts:
- { name: proc, mountPath: /host/proc, readOnly: true }
- { name: sys, mountPath: /host/sys, readOnly: true }
- { name: rootfs, mountPath: /host/root, readOnly: true, mountPropagation: HostToContainer }
resources:
requests: { memory: 32Mi, cpu: 20m }
limits: { memory: 96Mi }
volumes:
- { name: proc, hostPath: { path: /proc } }
- { name: sys, hostPath: { path: /sys } }
- { name: rootfs, hostPath: { path: / } }
---
apiVersion: v1
kind: Service
metadata:
name: node-exporter
namespace: observability
spec:
clusterIP: None # headless: Prometheus wants each pod, not a VIP
selector:
app: node-exporter
ports:
- port: 9100
targetPort: metrics
name: metrics
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana
namespace: observability
spec:
replicas: 1
selector:
matchLabels:
app: grafana
template:
metadata:
labels:
app: grafana
spec:
nodeSelector:
node-role.kubernetes.io/control-plane: "true"
containers:
- name: grafana
image: grafana/grafana:11.4.0
ports:
- containerPort: 3000
name: http
env:
- name: GF_SECURITY_ADMIN_USER
value: admin
- name: GF_SECURITY_ADMIN_PASSWORD
value: lab-grafana-change-me
# Grafana builds absolute URLs for redirects and asset paths. Behind
# the nginx -> Traefik chain it must be told the external address,
# for exactly the reason Keycloak needs KC_HOSTNAME. Without it,
# login redirects come back as http://<pod-ip>:3000.
- name: GF_SERVER_ROOT_URL
value: https://app2.hyeonworks.com
volumeMounts:
- name: datasources
mountPath: /etc/grafana/provisioning/datasources
readinessProbe:
httpGet: { path: /api/health, port: http }
initialDelaySeconds: 15
resources:
requests: { memory: 128Mi, cpu: 50m }
limits: { memory: 320Mi }
volumes:
- name: datasources
configMap:
name: grafana-datasources
---
# Provisioning the datasource as a file means Grafana comes up already wired to
# Prometheus. Clicking through the UI would leave the configuration only in
# Grafana's own database, which is emptyDir here and disappears on restart.
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-datasources
namespace: observability
data:
prometheus.yaml: |
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus.observability.svc:9090
isDefault: true
---
apiVersion: v1
kind: Service
metadata:
name: grafana
namespace: observability
spec:
selector:
app: grafana
ports:
- port: 3000
targetPort: http
---
# Grafana is published on app2.hyeonworks.com because that name is already in
# the wildcard-free certificate (auth / app1 / app2) and is otherwise unused.
# It moves when app2 is needed for the SSO experiment.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: grafana
namespace: observability
spec:
ingressClassName: traefik
rules:
- host: app2.hyeonworks.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: grafana
port:
number: 3000
@@ -0,0 +1,43 @@
# Make Traefik trust the X-Forwarded-* headers that the host nginx sets.
#
# Without this, Traefik rewrites every forwarded header from its own connection,
# which is plain HTTP on port 80. The application then sees scheme=http even
# though the browser connected over TLS. See docs/two-hop-proxy-header-contract.md.
#
# k3s installs Traefik through its bundled HelmChart, so values are overridden
# with a HelmChartConfig rather than by editing the deployment. k3s reconciles
# the chart and recreates the Traefik pod.
#
# kubectl apply -f deploy/lab/k8s/traefik-forwarded-headers.yaml
# kubectl -n kube-system rollout status deploy/traefik --timeout=180s
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: traefik
namespace: kube-system
spec:
valuesContent: |-
ports:
web:
forwardedHeaders:
# Requests arriving from these sources keep their existing
# X-Forwarded-* values instead of having them rewritten.
#
# 10.42.0.0/16 is the pod CIDR. It is required because the traefik
# Service uses externalTrafficPolicy: Cluster, so svclb SNATs the
# traffic and Traefik sees a pod-network address rather than the
# host nginx address.
#
# The node/host range is deliberately absent. Because svclb SNATs,
# the host nginx address never reaches Traefik — measured, not assumed.
# Trusting a range that cannot appear only widens the surface.
#
# Trusting the whole pod CIDR still means any pod in the cluster could
# forge these headers, which is why echo-network-policy.yaml restricts
# who may reach the application at all.
trustedIPs:
- 10.42.0.0/16
websecure:
forwardedHeaders:
trustedIPs:
- 10.42.0.0/16
+42
View File
@@ -0,0 +1,42 @@
#!/usr/bin/env bash
# Build the API image on this workstation and import it into each lab node's
# containerd.
#
# k3s does not run Docker and the lab has no registry, so images are shipped as
# a stream: docker save -> ssh through the lab host -> k3s ctr images import.
# Every node needs its own copy because the scheduler may place the pod anywhere.
#
# ./deploy/lab/scripts/build-and-import.sh
# IMAGE=keycloak-pattern-api:lab NODES="kc-lab-1" ./deploy/lab/scripts/build-and-import.sh
set -euo pipefail
IMAGE="${IMAGE:-keycloak-pattern-api:lab}"
NODES="${NODES:-kc-lab-1 kc-lab-2}"
LAB_HOST="${LAB_HOST:-test-server}"
CONTEXT="${CONTEXT:-backend}"
repo_root="$(git rev-parse --show-toplevel)"
cd "$repo_root"
echo "==> building ${IMAGE} from ${CONTEXT}/"
docker build -t "$IMAGE" "$CONTEXT"
for node in $NODES; do
echo "==> importing into ${node}"
# Nested ssh: the workstation cannot reach the guests directly because they
# sit behind the lab host's libvirt NAT. The lab host's ~/.ssh/config holds
# the kc-lab-* aliases.
docker save "$IMAGE" \
| ssh "$LAB_HOST" "ssh ${node} 'sudo k3s ctr images import -'"
done
echo "==> verifying"
for node in $NODES; do
printf ' %-10s ' "$node"
ssh "$LAB_HOST" "ssh ${node} 'sudo k3s ctr images ls -q'" \
| grep -c "$IMAGE" \
| xargs -I{} echo "{} match(es)"
done
echo
echo "next: kubectl rollout restart -n header-lab deployment/echo"
+55
View File
@@ -0,0 +1,55 @@
#!/usr/bin/env bash
# Experiment 0c — where does a session entry actually live?
#
# Experiment 0b showed keycloak-1's session cache never moved when keycloak-0
# handled a login. That leaves two explanations:
#
# (a) a DISTRIBUTED cache with owners=1 — entries are spread across nodes by
# consistent hashing, and this one happened to land on keycloak-0;
# (b) a LOCAL cache — each node only ever caches what it handled itself.
#
# They are distinguished by driving logins at the OTHER node. Under (a) the
# entries would keep landing on both nodes regardless of who was asked. Under
# (b) the count rises only on the node that received the request.
set -uo pipefail
NS="${NS:-keycloak-lab}"
N="${N:-5}"
K0_IP=$(kubectl -n "$NS" get pod keycloak-0 -o jsonpath='{.status.podIP}')
K1_IP=$(kubectl -n "$NS" get pod keycloak-1 -o jsonpath='{.status.podIP}')
ADMIN_PW=$(kubectl -n "$NS" get secret keycloak-lab-secrets \
-o jsonpath='{.data.KC_BOOTSTRAP_ADMIN_PASSWORD}' | base64 -d)
echo "수집 시각: $(date '+%Y-%m-%d %H:%M:%S %Z')"
echo " keycloak-0 = $K0_IP ($(kubectl -n "$NS" get pod keycloak-0 -o jsonpath='{.spec.nodeName}'))"
echo " keycloak-1 = $K1_IP ($(kubectl -n "$NS" get pod keycloak-1 -o jsonpath='{.spec.nodeName}'))"
echo
kubectl -n "$NS" run kc-own --rm -i --restart=Never \
--image=curlimages/curl:8.11.1 --quiet --command -- sh -c "
O=/tmp/o; : > \$O
ent() {
curl -s --retry 3 --max-time 20 http://\$1:9000/metrics \
| grep -E '^vendor_statistics_approximate_entries_unique.cache=.sessions' \
| awk '{print \$NF}'
}
login() { i=0; while [ \$i -lt $N ]; do
curl -s -o /dev/null -X POST http://\$1:8080/realms/master/protocol/openid-connect/token \
-d grant_type=password -d client_id=admin-cli \
-d username=admin -d 'password=$ADMIN_PW'
i=\$((i+1)); done; sleep 5; }
{
printf '%-32s %12s %12s\n' '단계' 'k0 entries' 'k1 entries'
printf '%-32s %12s %12s\n' '시작' \"\$(ent $K0_IP)\" \"\$(ent $K1_IP)\"
login $K1_IP
printf '%-32s %12s %12s\n' 'keycloak-1 에 로그인 ${N}회' \"\$(ent $K0_IP)\" \"\$(ent $K1_IP)\"
login $K0_IP
printf '%-32s %12s %12s\n' 'keycloak-0 에 로그인 ${N}회' \"\$(ent $K0_IP)\" \"\$(ent $K1_IP)\"
} >> \$O
cat \$O
" 2>&1 | grep -v '^pod .* deleted$'
echo
echo "=== 대조: PostgreSQL 에는 몇 건인가 ==="
kubectl -n "$NS" exec deploy/postgres -- psql -U keycloak -d keycloak -tAc \
"select count(*) from offline_user_session where offline_flag='0'" 2>/dev/null | sed 's/^/ online 세션 /'
+83
View File
@@ -0,0 +1,83 @@
#!/usr/bin/env bash
# Experiment 0b — does the Infinispan cache itself replicate, or do both nodes
# merely agree because they read the same database?
#
# Experiment 0 proved the two nodes give the same answers. That alone does NOT
# prove Infinispan replicated anything: with persistent-user-sessions (the
# Keycloak 26 default) the session is written to PostgreSQL, so two nodes reading
# one database would agree even with the cache disabled entirely.
#
# This script separates the two by measuring the cache counters on BOTH nodes
# around a single login. If the write on keycloak-0 shows up as cache activity
# on keycloak-1, the replication is real and not a database artifact.
set -uo pipefail
NS="${NS:-keycloak-lab}"
K0_IP=$(kubectl -n "$NS" get pod keycloak-0 -o jsonpath='{.status.podIP}')
K1_IP=$(kubectl -n "$NS" get pod keycloak-1 -o jsonpath='{.status.podIP}')
ADMIN_PW=$(kubectl -n "$NS" get secret keycloak-lab-secrets \
-o jsonpath='{.data.KC_BOOTSTRAP_ADMIN_PASSWORD}' | base64 -d)
echo "수집 시각: $(date '+%Y-%m-%d %H:%M:%S %Z')"
echo
# 파드 출력을 스트리밍으로 받으면 조각이 유실된다. 실제로 첫 시도에서
# keycloak-1 의 스냅샷과 그 다음 마커가 통째로 사라져 델타가 0 으로 보였다.
# 파드 안에서 파일로 모았다가 마지막에 한 번만 내보낸다.
kubectl -n "$NS" run kc-delta --rm -i --restart=Never \
--image=curlimages/curl:8.11.1 --quiet --command -- sh -c "
set -u
K0='http://$K0_IP'; K1='http://$K1_IP'
O=/tmp/o.txt; : > \$O
snap() {
curl -s --retry 3 --retry-connrefused --max-time 20 \$1:9000/metrics \
| grep -E '^vendor_(statistics_(stores|hits|misses|approximate_entries_unique)|rpc_manager_replication_count)\{cache=\"(sessions|clientSessions)\"' \
| sed 's/,cache_manager=\"keycloak\"//; s/,node=\"[^\"]*\"//' >> \$O
}
echo '###BEFORE_K0' >> \$O; snap \$K0
echo '###BEFORE_K1' >> \$O; snap \$K1
echo '###LOGIN' >> \$O
curl -s -o /dev/null -w 'http_code=%{http_code}\n' -X POST \
\"\$K0:8080/realms/master/protocol/openid-connect/token\" \
-d grant_type=password -d client_id=admin-cli \
-d username=admin -d 'password=$ADMIN_PW' >> \$O
sleep 5
echo '###AFTER_K0' >> \$O; snap \$K0
echo '###AFTER_K1' >> \$O; snap \$K1
echo '###END' >> \$O
cat \$O
" 2>&1 | grep -v '^pod .* deleted$' > /tmp/cache-delta.txt
python3 - /tmp/cache-delta.txt <<'PY'
import re, sys
raw = open(sys.argv[1]).read()
blocks, cur = {}, None
for line in raw.splitlines():
if line.startswith('###'):
cur = line[3:]; blocks[cur] = {}
elif cur and '{' in line:
m = re.match(r'(\S+?)\{cache="(\w+)"\}\s+(\S+)', line)
if m:
blocks[cur][(m.group(1), m.group(2))] = float(m.group(3))
print('=== 로그인은 keycloak-0 에만 보냈다 ===')
code = [l for l in raw.splitlines() if l.startswith('http_code=')]
print(' 로그인 응답: ' + (code[0] if code else '없음'))
for n in ('BEFORE_K0','BEFORE_K1','AFTER_K0','AFTER_K1'):
if not blocks.get(n):
print(f' !! {n} 스냅샷이 비었다 — 델타를 신뢰할 수 없다')
print()
hdr = f" {'계수기':<42} {'캐시':<15} {'전':>8} {'후':>8} {'증가':>7}"
for node in ('K0', 'K1'):
who = 'keycloak-0 (로그인을 받은 노드)' if node == 'K0' else 'keycloak-1 (아무 요청도 받지 않은 노드)'
print(f'=== {who} ===')
print(hdr)
b, a = blocks.get(f'BEFORE_{node}', {}), blocks.get(f'AFTER_{node}', {})
for k in sorted(set(b) | set(a)):
before, after = b.get(k[0:2], 0.0), a.get(k[0:2], 0.0)
d = after - before
mark = ' ←' if d else ''
name = k[0].replace('vendor_statistics_', '').replace('vendor_rpc_manager_', 'rpc.')
print(f" {name:<42} {k[1]:<15} {before:>8.0f} {after:>8.0f} {d:>+7.0f}{mark}")
print()
PY
+108
View File
@@ -0,0 +1,108 @@
#!/usr/bin/env bash
# Experiment 0d — capture the actual SQL that the OTHER node runs.
#
# Experiments 0b/0c showed that session entries never appear in keycloak-1's
# memory, yet keycloak-1 can use a session keycloak-0 created. The conclusion
# "keycloak-1 reads it from PostgreSQL" was an inference, not an observation.
#
# This script turns on statement logging in PostgreSQL for a few seconds, sends
# ONE refresh request to keycloak-1 for a session born on keycloak-0, and greps
# the database log for that session id. If the inference is right, the SQL is
# there, issued from keycloak-1's pod IP.
#
# It also checks whether serving that request makes keycloak-1 cache the session
# — which sharpens "each node caches what it handled" from "what it logged in"
# to "what it touched".
set -uo pipefail
NS="${NS:-keycloak-lab}"
PSQL="kubectl -n $NS exec deploy/postgres -- psql -U keycloak -d keycloak -tAc"
K0_IP=$(kubectl -n "$NS" get pod keycloak-0 -o jsonpath='{.status.podIP}')
K1_IP=$(kubectl -n "$NS" get pod keycloak-1 -o jsonpath='{.status.podIP}')
ADMIN_PW=$(kubectl -n "$NS" get secret keycloak-lab-secrets \
-o jsonpath='{.data.KC_BOOTSTRAP_ADMIN_PASSWORD}' | base64 -d)
echo "수집 시각: $(date '+%Y-%m-%d %H:%M:%S %Z')"
echo " keycloak-0 = $K0_IP (세션을 만드는 노드)"
echo " keycloak-1 = $K1_IP (읽기만 하는 노드)"
echo
# %h 를 넣어야 어느 파드가 보낸 질의인지 로그에서 구분된다.
echo "=== PostgreSQL 문장 로깅을 켠다 ==="
$PSQL "alter system set log_statement='all'" >/dev/null 2>&1
$PSQL "alter system set log_line_prefix='%m [%p] %h '" >/dev/null 2>&1
$PSQL "select pg_reload_conf()" >/dev/null 2>&1
echo " log_statement = $($PSQL 'show log_statement' 2>/dev/null)"
echo " log_line_prefix = $($PSQL 'show log_line_prefix' 2>/dev/null)"
echo
# 로그 커서를 잡아둔다. 이 줄 수 이후만 본다.
LOG_BEFORE=$(kubectl -n "$NS" logs deploy/postgres --tail=-1 2>/dev/null | wc -l)
RESULT=$(kubectl -n "$NS" run kc-readpath --rm -i --restart=Never \
--image=curlimages/curl:8.11.1 --quiet --command -- sh -c "
O=/tmp/o; : > \$O
TOKEN_EP='/realms/master/protocol/openid-connect/token'
jget() { sed -n \"s/.*\\\"\$1\\\":\\\"\\([^\\\"]*\\)\\\".*/\\1/p\"; }
ent() {
curl -s --retry 3 --max-time 20 http://\$1:9000/metrics \
| grep -E '^vendor_statistics_approximate_entries_unique.cache=.sessions' | awk '{print \$NF}'
}
# keycloak-0 에서 로그인한다
L=\$(curl -s -X POST \"http://$K0_IP:8080\$TOKEN_EP\" -d grant_type=password \
-d client_id=admin-cli -d username=admin -d 'password=$ADMIN_PW')
SID=\$(echo \"\$L\" | jget access_token | cut -d. -f2 | sed 's/\$/==/' | base64 -d 2>/dev/null | jget sid)
RT=\$(echo \"\$L\" | jget refresh_token)
echo \"SID=\$SID\" >> \$O
echo \"K1_ENTRIES_BEFORE=\$(ent $K1_IP)\" >> \$O
sleep 2
# 반대편 노드에 refresh 를 딱 한 번 보낸다
# 인용을 한 겹 더 쌓으면 curl 이 URL 을 통째로 못 읽는다. 실제로 000 이 나왔다.
CODE=\$(curl -s -o /dev/null -w '%{http_code}' -X POST \
\"http://$K1_IP:8080\$TOKEN_EP\" \
-d grant_type=refresh_token -d client_id=admin-cli -d \"refresh_token=\$RT\")
echo \"REFRESH_ON_K1=\$CODE\" >> \$O
sleep 3
echo \"K1_ENTRIES_AFTER=\$(ent $K1_IP)\" >> \$O
cat \$O
" 2>&1 | grep -v '^pod .* deleted$')
echo "=== 요청 ==="
echo "$RESULT" | sed 's/^/ /'
SID=$(echo "$RESULT" | sed -n 's/^SID=//p')
echo
echo "=== PostgreSQL 문장 로깅을 끈다 ==="
$PSQL "alter system reset log_statement" >/dev/null 2>&1
$PSQL "alter system reset log_line_prefix" >/dev/null 2>&1
$PSQL "select pg_reload_conf()" >/dev/null 2>&1
echo " log_statement = $($PSQL 'show log_statement' 2>/dev/null)"
echo
echo "=== keycloak-1 이 실제로 보낸 SQL 문장 ==="
echo " (파라미터가 \$1 로 묶여 있어, sid 는 바로 아래 DETAIL 줄에 있다)"
echo
kubectl -n "$NS" logs deploy/postgres --tail=-1 2>/dev/null \
| tail -n +$((LOG_BEFORE + 1)) \
| grep -F "$K1_IP" | grep -E "LOG: execute" \
| sed 's/.*execute [^:]*: //' | sed 's/^/ /' | head -12
echo
echo "=== 그 sid 를 언급한 SQL — 누가 보냈는가 ==="
echo " 찾는 sid: $SID"
echo
kubectl -n "$NS" logs deploy/postgres --tail=-1 2>/dev/null \
| tail -n +$((LOG_BEFORE + 1)) \
| grep -F "$SID" \
| sed -e "s/$K0_IP/[keycloak-0]/g" -e "s/$K1_IP/[keycloak-1]/g" \
| cut -c1-220 \
| head -20
echo
echo "=== 요약: 파드별 질의 건수 ==="
kubectl -n "$NS" logs deploy/postgres --tail=-1 2>/dev/null \
| tail -n +$((LOG_BEFORE + 1)) \
| grep -F "$SID" \
| grep -oE "^[0-9-]+ [0-9:.]+ [A-Z]+ \[[0-9]+\] [0-9.]+" \
| awk '{print $NF}' | sort | uniq -c \
| sed -e "s/$K0_IP/[keycloak-0]/" -e "s/$K1_IP/[keycloak-1]/" -e 's/^/ /'

Some files were not shown because too many files have changed in this diff Show More