# WebSocket platform runbook

Organised by what you observe, because that is what you have during an incident.

## Clients report 1006 — "closed abnormally, no reason"

The most important symptom here, because 1006 is indistinguishable from a network failure and
drives every client into its most aggressive reconnect path.

- **During a deploy** — the platform did not close its connections. `PlatformWebSocketHandler`
  implements `SmartLifecycle` and closes with **1001 going away**; a container's own graceful
  shutdown does **not** do this, because it waits for in-flight *requests* and an established
  WebSocket is not a request. Check the handler bean is registered and that its phase runs before
  the web server stops.
- **Behind a proxy** — the upgrade is mishandled, so the close frame never becomes a close frame.
  See the next section.
- **Neither** — something threw while closing. A close reason longer than 123 bytes makes the
  container throw mid-close and turns a clean refusal into a 1006; the platform truncates for that
  reason.

## Connections work locally and not behind the load balancer

Three nginx directives, each producing a different failure:

| Missing | Symptom |
| --- | --- |
| `proxy_http_version 1.1` | Handshake answered 400, or simply not upgraded. HTTP/1.0 has no Upgrade mechanism. |
| `Upgrade` / `Connection` forwarded | Upstream sees an ordinary GET and answers 404 or 200 for a route that works when tested directly. They are hop-by-hop headers, so a proxy is *required* to drop them. |
| `proxy_read_timeout` raised | Every connection quieter than 60s is killed with no close frame. A WebSocket is idle by nature. |

`websocketNginxTest` asserts all three against a real Nginx, and carries a deliberately broken
configuration so it proves its own assertions can fail.

## Connections vanish with no close frame and no error

Almost always an intermediary's idle timeout, not the application. TCP does not report a departed
peer — a closed laptop lid, a phone switching to cellular and a NAT forgetting its mapping all
produce no FIN.

- Check `HeartbeatPolicy`: the ping interval must be under ~20s, because intermediaries commonly
  drop idle connections at 30–60s and say nothing.
- The idle timeout must be at least two ping intervals. Below that, one dropped ping on a congested
  network closes a healthy connection, and the reconnect storm makes the congestion worse.

## Memory grows and nothing is failing

A slow consumer. It does not error — it reads more slowly than the server writes and the difference
accumulates. Arranging it requires no tooling: reading slowly is enough.

- **Check first:** `WebSocketNodeSnapshot.droppedOutboundMessages`. Connection counts and buffered
  bytes both look healthy while data is being lost; the drop count is the only number that says so.
- **Bounds:** per connection (`maxBufferedOutboundBytes`) *and* node-wide (`GlobalBufferBudget`).
  The second is not redundant — a megabyte each is fine at a hundred connections and is the whole
  heap at fifty thousand.
- **If drops are zero and memory still grows:** check that the queue lock and the socket lock are
  separate. When they were one lock, a stalled peer blocked every producer, the queue never filled,
  and the shedding bound never fired. That is fixed, and it is the shape to look for if it recurs.

## A command ran twice

The reconnect is what makes this a WebSocket problem: a client replays everything it never saw an
answer for, on a new connection, all at once.

1. **Key scoped to the connection?** It must not be. `WebSocketCommandKey` is endpoint + actor +
   the client's own message id, so it survives the reconnect.
2. **Ledger not transactional?** The only implementation that works writes in the application's own
   transaction. Anything else has a window between the business commit and the ledger write.
3. **Outcome `UNKNOWN`?** That is the window. Neither replaying nor re-running is safe — replaying
   invents a result, re-running duplicates a committed write. Only `CommandReconciliation` reading
   the business data can settle it, and `INDETERMINATE` is a real answer to act on.

## A client cannot connect and the endpoint is definitely there

- **403** — origin. The same-origin policy does **not** protect a WebSocket handshake and there is
  no preflight, so the server's `Origin` check is the entire defence and it is strict: exact match,
  lowercase, no path, no wildcard.
- **401** — the ticket was spent, expired (30s cap), or minted for another endpoint. All three are
  deliberate; a longer-lived or unbound ticket is a bearer token in a query string again.
- **400** — no offered subprotocol is supported. Production refuses a client that offers none.
- **503** — the endpoint is at its connection cap.

Order matters and is asserted: route → origin → capacity → credential → subprotocol. Origin is
checked before authentication because a cookie handshake from an attacker's page authenticates
perfectly; the credential is valid and the page is not allowed to use it.

## Endpoints never answer and the app reports healthy

Both runtimes on one classpath. Spring Boot deduces one application type and picks the servlet one,
so declared reactive endpoints are simply never served — no error, no warning.
`WebSocketStackExclusivity` fails startup on this; if a running instance has it, the check is not
wired.

## Metrics stopped arriving

A high-cardinality tag, and the risk is worse than for HTTP: a request produces one observation,
a connection produces them for hours, so the series outlive the connections and only accumulate.
`WebSocketMetricTags` allows eight names; `connectionId`, `sessionId` and `actor` are not among
them. `nodeId` is, because a fleet has a knowable number of nodes.

## Verification

```bash
cd src
./gradlew :adapter:inbound:websocket:test                # unit, boundary, architecture, Tomcat runtime
./gradlew :adapter:inbound:websocket:websocketJettyTest  # second servlet container
./gradlew :adapter:inbound:websocket:websocketNginxTest  # real Nginx; needs Docker, fails without it
```