clean-architecture-backend-template/docs/websocket/runbook.md

WebSocket platform runbook

Organised by what you observe, because that is what you have during an incident.

Clients report 1006 — "closed abnormally, no reason"

The most important symptom here, because 1006 is indistinguishable from a network failure and drives every client into its most aggressive reconnect path.

• During a deploy — the platform did not close its connections. PlatformWebSocketHandler implements SmartLifecycle and closes with 1001 going away; a container's own graceful shutdown does not do this, because it waits for in-flight requests and an established WebSocket is not a request. Check the handler bean is registered and that its phase runs before the web server stops.
• Behind a proxy — the upgrade is mishandled, so the close frame never becomes a close frame. See the next section.
• Neither — something threw while closing. A close reason longer than 123 bytes makes the container throw mid-close and turns a clean refusal into a 1006; the platform truncates for that reason.

Connections work locally and not behind the load balancer

Three nginx directives, each producing a different failure:

Missing	Symptom
proxy_http_version 1.1	Handshake answered 400, or simply not upgraded. HTTP/1.0 has no Upgrade mechanism.
Upgrade / Connection forwarded	Upstream sees an ordinary GET and answers 404 or 200 for a route that works when tested directly. They are hop-by-hop headers, so a proxy is required to drop them.
proxy_read_timeout raised	Every connection quieter than 60s is killed with no close frame. A WebSocket is idle by nature.

websocketNginxTest asserts all three against a real Nginx, and carries a deliberately broken configuration so it proves its own assertions can fail.

Connections vanish with no close frame and no error

Almost always an intermediary's idle timeout, not the application. TCP does not report a departed peer — a closed laptop lid, a phone switching to cellular and a NAT forgetting its mapping all produce no FIN.

• Check HeartbeatPolicy: the ping interval must be under ~20s, because intermediaries commonly drop idle connections at 30–60s and say nothing.
• The idle timeout must be at least two ping intervals. Below that, one dropped ping on a congested network closes a healthy connection, and the reconnect storm makes the congestion worse.

Memory grows and nothing is failing

A slow consumer. It does not error — it reads more slowly than the server writes and the difference accumulates. Arranging it requires no tooling: reading slowly is enough.

• Check first: WebSocketNodeSnapshot.droppedOutboundMessages. Connection counts and buffered bytes both look healthy while data is being lost; the drop count is the only number that says so.
• Bounds: per connection (maxBufferedOutboundBytes) and node-wide (GlobalBufferBudget). The second is not redundant — a megabyte each is fine at a hundred connections and is the whole heap at fifty thousand.
• If drops are zero and memory still grows: check that the queue lock and the socket lock are separate. When they were one lock, a stalled peer blocked every producer, the queue never filled, and the shedding bound never fired. That is fixed, and it is the shape to look for if it recurs.

A command ran twice

The reconnect is what makes this a WebSocket problem: a client replays everything it never saw an answer for, on a new connection, all at once.

1. Key scoped to the connection? It must not be. WebSocketCommandKey is endpoint + actor + the client's own message id, so it survives the reconnect.
2. Ledger not transactional? The only implementation that works writes in the application's own transaction. Anything else has a window between the business commit and the ledger write.
3. Outcome UNKNOWN? That is the window. Neither replaying nor re-running is safe — replaying invents a result, re-running duplicates a committed write. Only CommandReconciliation reading the business data can settle it, and INDETERMINATE is a real answer to act on.

A client cannot connect and the endpoint is definitely there

• 403 — origin. The same-origin policy does not protect a WebSocket handshake and there is no preflight, so the server's Origin check is the entire defence and it is strict: exact match, lowercase, no path, no wildcard.
• 401 — the ticket was spent, expired (30s cap), or minted for another endpoint. All three are deliberate; a longer-lived or unbound ticket is a bearer token in a query string again.
• 400 — no offered subprotocol is supported. Production refuses a client that offers none.
• 503 — the endpoint is at its connection cap.

Order matters and is asserted: route → origin → capacity → credential → subprotocol. Origin is checked before authentication because a cookie handshake from an attacker's page authenticates perfectly; the credential is valid and the page is not allowed to use it.

Endpoints never answer and the app reports healthy

Both runtimes on one classpath. Spring Boot deduces one application type and picks the servlet one, so declared reactive endpoints are simply never served — no error, no warning. WebSocketStackExclusivity fails startup on this; if a running instance has it, the check is not wired.

Metrics stopped arriving

A high-cardinality tag, and the risk is worse than for HTTP: a request produces one observation, a connection produces them for hours, so the series outlive the connections and only accumulate. WebSocketMetricTags allows eight names; connectionId, sessionId and actor are not among them. nodeId is, because a fleet has a knowable number of nodes.

Verification

cd src
./gradlew :adapter:inbound:websocket:test                # unit, boundary, architecture, Tomcat runtime
./gradlew :adapter:inbound:websocket:websocketJettyTest  # second servlet container
./gradlew :adapter:inbound:websocket:websocketNginxTest  # real Nginx; needs Docker, fails without it