feat: web, websocket 어댑터 추가 구현
This commit is contained in:
@@ -0,0 +1,107 @@
|
||||
# WebSocket platform runbook
|
||||
|
||||
Organised by what you observe, because that is what you have during an incident.
|
||||
|
||||
## Clients report 1006 — "closed abnormally, no reason"
|
||||
|
||||
The most important symptom here, because 1006 is indistinguishable from a network failure and
|
||||
drives every client into its most aggressive reconnect path.
|
||||
|
||||
- **During a deploy** — the platform did not close its connections. `PlatformWebSocketHandler`
|
||||
implements `SmartLifecycle` and closes with **1001 going away**; a container's own graceful
|
||||
shutdown does **not** do this, because it waits for in-flight *requests* and an established
|
||||
WebSocket is not a request. Check the handler bean is registered and that its phase runs before
|
||||
the web server stops.
|
||||
- **Behind a proxy** — the upgrade is mishandled, so the close frame never becomes a close frame.
|
||||
See the next section.
|
||||
- **Neither** — something threw while closing. A close reason longer than 123 bytes makes the
|
||||
container throw mid-close and turns a clean refusal into a 1006; the platform truncates for that
|
||||
reason.
|
||||
|
||||
## Connections work locally and not behind the load balancer
|
||||
|
||||
Three nginx directives, each producing a different failure:
|
||||
|
||||
| Missing | Symptom |
|
||||
| --- | --- |
|
||||
| `proxy_http_version 1.1` | Handshake answered 400, or simply not upgraded. HTTP/1.0 has no Upgrade mechanism. |
|
||||
| `Upgrade` / `Connection` forwarded | Upstream sees an ordinary GET and answers 404 or 200 for a route that works when tested directly. They are hop-by-hop headers, so a proxy is *required* to drop them. |
|
||||
| `proxy_read_timeout` raised | Every connection quieter than 60s is killed with no close frame. A WebSocket is idle by nature. |
|
||||
|
||||
`websocketNginxTest` asserts all three against a real Nginx, and carries a deliberately broken
|
||||
configuration so it proves its own assertions can fail.
|
||||
|
||||
## Connections vanish with no close frame and no error
|
||||
|
||||
Almost always an intermediary's idle timeout, not the application. TCP does not report a departed
|
||||
peer — a closed laptop lid, a phone switching to cellular and a NAT forgetting its mapping all
|
||||
produce no FIN.
|
||||
|
||||
- Check `HeartbeatPolicy`: the ping interval must be under ~20s, because intermediaries commonly
|
||||
drop idle connections at 30–60s and say nothing.
|
||||
- The idle timeout must be at least two ping intervals. Below that, one dropped ping on a congested
|
||||
network closes a healthy connection, and the reconnect storm makes the congestion worse.
|
||||
|
||||
## Memory grows and nothing is failing
|
||||
|
||||
A slow consumer. It does not error — it reads more slowly than the server writes and the difference
|
||||
accumulates. Arranging it requires no tooling: reading slowly is enough.
|
||||
|
||||
- **Check first:** `WebSocketNodeSnapshot.droppedOutboundMessages`. Connection counts and buffered
|
||||
bytes both look healthy while data is being lost; the drop count is the only number that says so.
|
||||
- **Bounds:** per connection (`maxBufferedOutboundBytes`) *and* node-wide (`GlobalBufferBudget`).
|
||||
The second is not redundant — a megabyte each is fine at a hundred connections and is the whole
|
||||
heap at fifty thousand.
|
||||
- **If drops are zero and memory still grows:** check that the queue lock and the socket lock are
|
||||
separate. When they were one lock, a stalled peer blocked every producer, the queue never filled,
|
||||
and the shedding bound never fired. That is fixed, and it is the shape to look for if it recurs.
|
||||
|
||||
## A command ran twice
|
||||
|
||||
The reconnect is what makes this a WebSocket problem: a client replays everything it never saw an
|
||||
answer for, on a new connection, all at once.
|
||||
|
||||
1. **Key scoped to the connection?** It must not be. `WebSocketCommandKey` is endpoint + actor +
|
||||
the client's own message id, so it survives the reconnect.
|
||||
2. **Ledger not transactional?** The only implementation that works writes in the application's own
|
||||
transaction. Anything else has a window between the business commit and the ledger write.
|
||||
3. **Outcome `UNKNOWN`?** That is the window. Neither replaying nor re-running is safe — replaying
|
||||
invents a result, re-running duplicates a committed write. Only `CommandReconciliation` reading
|
||||
the business data can settle it, and `INDETERMINATE` is a real answer to act on.
|
||||
|
||||
## A client cannot connect and the endpoint is definitely there
|
||||
|
||||
- **403** — origin. The same-origin policy does **not** protect a WebSocket handshake and there is
|
||||
no preflight, so the server's `Origin` check is the entire defence and it is strict: exact match,
|
||||
lowercase, no path, no wildcard.
|
||||
- **401** — the ticket was spent, expired (30s cap), or minted for another endpoint. All three are
|
||||
deliberate; a longer-lived or unbound ticket is a bearer token in a query string again.
|
||||
- **400** — no offered subprotocol is supported. Production refuses a client that offers none.
|
||||
- **503** — the endpoint is at its connection cap.
|
||||
|
||||
Order matters and is asserted: route → origin → capacity → credential → subprotocol. Origin is
|
||||
checked before authentication because a cookie handshake from an attacker's page authenticates
|
||||
perfectly; the credential is valid and the page is not allowed to use it.
|
||||
|
||||
## Endpoints never answer and the app reports healthy
|
||||
|
||||
Both runtimes on one classpath. Spring Boot deduces one application type and picks the servlet one,
|
||||
so declared reactive endpoints are simply never served — no error, no warning.
|
||||
`WebSocketStackExclusivity` fails startup on this; if a running instance has it, the check is not
|
||||
wired.
|
||||
|
||||
## Metrics stopped arriving
|
||||
|
||||
A high-cardinality tag, and the risk is worse than for HTTP: a request produces one observation,
|
||||
a connection produces them for hours, so the series outlive the connections and only accumulate.
|
||||
`WebSocketMetricTags` allows eight names; `connectionId`, `sessionId` and `actor` are not among
|
||||
them. `nodeId` is, because a fleet has a knowable number of nodes.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
cd src
|
||||
./gradlew :adapter:inbound:websocket:test # unit, boundary, architecture, Tomcat runtime
|
||||
./gradlew :adapter:inbound:websocket:websocketJettyTest # second servlet container
|
||||
./gradlew :adapter:inbound:websocket:websocketNginxTest # real Nginx; needs Docker, fails without it
|
||||
```
|
||||
Reference in New Issue
Block a user