feat: web, websocket 어댑터 추가 구현
This commit is contained in:
DongHyeonka
@@ -0,0 +1,47 @@
|
||||
# WebSocket Advanced: support matrix
|
||||
|
||||
Every capability is off unless named. This table is what each one costs and what has to be true
|
||||
before it is promoted. `AdvancedPromotionGate.forCapability` is the machine-checked form of the last
|
||||
two columns; if they disagree, the code wins and this table is stale.
|
||||
|
||||
| Capability | Flag | Adds | Required suites | Soak |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `RESUME` | `…advanced.resume.enabled` | A signed token and a replay store | `websocket:test`, `websocketJettyTest`, `resume-history-loss`, `resume-replay` | 8h |
|
||||
| `CLUSTER_REDIS` | `…advanced.cluster-redis.enabled` | A Redis dependency on the routing path | `websocket:test`, `multi-node-fanout`, `node-loss`, `index-partition` | 24h |
|
||||
| `CLUSTER_MESSAGING` | `…advanced.cluster-messaging.enabled` | A broker dependency on the delivery path | as `CLUSTER_REDIS` | 24h |
|
||||
| `PRESENCE` | `…advanced.presence.enabled` | A read model over the cluster index | as `CLUSTER_REDIS` | 24h |
|
||||
| `STOMP` | `…advanced.stomp.enabled` | A second protocol parser, pre-authentication | `websocket:test`, `broker-outage`, `broker-reconnect`, `user-destination` | 8h |
|
||||
| `BROKER_RELAY_RABBIT` | `…advanced.stomp.relay.enabled` | A TCP dependency on an external broker | as `STOMP` | 8h |
|
||||
| `CODEC_PROTOBUF` | `…advanced.codec-protobuf.enabled` | A second decode path | `websocket:test`, `websocketJettyTest` | 8h |
|
||||
| `CODEC_CBOR` | `…advanced.codec-cbor.enabled` | A second decode path | `websocket:test`, `websocketJettyTest` | 8h |
|
||||
| `COMPRESSION` | `…advanced.compression.enabled` | Per-connection memory, and a length side channel | `websocket:test`, `decompression-bound`, `memory-under-load` | 24h |
|
||||
| `OUTBOUND_CLIENT` | `…advanced.outbound-client.enabled` | Long-lived connections this service initiates | `websocket:test`, `websocketJettyTest` | 8h |
|
||||
| `SOCKJS_COMPAT` | `…advanced.sockjs.enabled` | Credentialed cross-origin HTTP, so CSRF | `websocket:test`, `websocketJettyTest` | 8h |
|
||||
| `HTTP2_COMPAT` | `…advanced.http2.enabled` | RFC 8441 extended CONNECT | `websocket:test`, `websocketNginxTest`, `proxy-matrix`, `classic-upgrade-fallback` | 7d |
|
||||
| `HTTP3_EXPERIMENTAL` | `…advanced.http3.enabled` | QUIC, and every hop problem HTTP/2 has | as `HTTP2_COMPAT` | 7d |
|
||||
| `GRAPHQL_TRANSPORT` | `…advanced.graphql-transport.enabled` | A subprotocol on the Stable runtime | `websocket:test`, `websocketJettyTest` | 8h |
|
||||
|
||||
## What is *not* supported, and why the row is here
|
||||
|
||||
| Not supported | Reason |
|
||||
| --- | --- |
|
||||
| Simple broker in a multi-node deployment | It does not error. It delivers to whichever fraction of users is on the publishing node, which reads as intermittent loss. `SimpleBrokerProfile.activatableUnder` refuses it. |
|
||||
| `permessage-deflate` on an endpoint mixing a secret with attacker-influenced content | The CRIME/BREACH shape. No parameter combination makes it safe; the leak is in the compressed length. `CompressionPolicy.mayCompress` refuses it. |
|
||||
| JSONP polling on a sensitive endpoint | JSONP executes server-supplied script in the page. |
|
||||
| Protobuf payloads above 256KB | The broker, the replay store and every in-memory queue would each hold the message whole. |
|
||||
| Extended CONNECT on an untested hop | RFC 8441 fails by killing the connection, not by negotiating a fallback. |
|
||||
| HTTP/3 as stable support | `Http3ExperimentalProfile.stableSupport()` returns false, always. |
|
||||
| Both STOMP channels at once | The broker that results is whichever configurer ran last, with no error. `StompBrokerExclusivity` refuses it. |
|
||||
|
||||
## Promotion
|
||||
|
||||
Each capability is promoted on its own evidence. They share a feature-flag mechanism and nothing
|
||||
else, so promoting them together means the evidence for the cheapest is treated as evidence for the
|
||||
most dangerous.
|
||||
|
||||
Two conditions apply to all of them and are not waivable:
|
||||
|
||||
- **Rollback exercised.** A flag nobody has turned off is not known to turn off.
|
||||
- **Stable artifact unchanged.** If enabling the capability changed Stable's wire contract or
|
||||
dependency graph, Stable was never independent of it, and deployments that did not enable it are
|
||||
affected anyway.
|
||||
@@ -0,0 +1,180 @@
|
||||
# WebSocket platform: how the design maps onto this repository
|
||||
|
||||
The design models the platform as eighteen Gradle modules under `modules/websocket`. This
|
||||
repository's fail-closed registry (`src/config/architecture/modules.json`) owns the leaf list, so
|
||||
those modules are packages inside the registered `:adapter:inbound:websocket` leaf — the same
|
||||
resolution the JPA, GraphQL and web platforms reached.
|
||||
|
||||
That is only honest if the boundaries are machine-checked, so `WebSocketStableModule` declares each
|
||||
module's package, its purity grade and its exact allowed edges, and `WebSocketModuleBoundaryTest`
|
||||
scans the source tree and fails when the two disagree in either direction. Promoting a package to
|
||||
its own Gradle leaf later is a registry edit rather than an archaeology exercise.
|
||||
|
||||
## Where the module map deviates from the design, and why
|
||||
|
||||
| Design places it in | Here | Reason |
|
||||
| --- | --- | --- |
|
||||
| `WebSocketSubprotocolName` in `websocket-protocol` | `core` | The connection context must name the negotiated token, and the context is core — leaving the type in `protocol` made `core` depend on `protocol` while `protocol` already depended on `core`. The boundary test refused the cycle. The negotiation *policy* stays in `protocol`. |
|
||||
| `StrictWebSocketJsonCodec` in `websocket-protocol` | `codec` (its own FRAMEWORK_BOUND module) | `protocol` is CORE here, and a Jackson import in a CORE module is refused. Splitting is better than relaxing the rule: the envelope's field rules stay testable with no mapper, and everything that touches a parser sits in one package a reviewer can read end to end. |
|
||||
| `budget -> core` | `core -> budget` | `budget` imports nothing from `core` — numbers depend on nothing. The endpoint profile, which is core, has to name a budget. The declared direction was simply backwards. |
|
||||
| — | `stomp` module | The pre-existing STOMP-over-SockJS channel predates this platform and still ships. Declared so the boundary is complete rather than excused; it has no edge to any platform module and none to it. |
|
||||
|
||||
## What the design's rules actually prevent
|
||||
|
||||
A few of the design's requirements read as style and are not. These are the ones worth keeping.
|
||||
|
||||
**`websocket-core-api` names no framework.** Stricter here than in the HTTP platform, because a
|
||||
connection is a long-lived object owned by a container and reaching for the container's own session
|
||||
type is tempting from everywhere. A CORE module has no `WebSocketSession`, so the same decision
|
||||
serves both runtimes and is testable without a server. The detector's framework list was missing
|
||||
`tools.jackson` (this repo runs Jackson 3, not 2) — a CORE module could have imported a mapper
|
||||
unnoticed. Fixed in both this leaf and the web leaf.
|
||||
|
||||
**No Java class name on the wire.** A FQCN publishes the package layout, breaks every client on a
|
||||
rename, and makes the receiver's type resolution an attack surface. `WebSocketMessageType` refuses
|
||||
anything that looks like one; the manifest binds published names to records, and records cannot run
|
||||
code while being populated.
|
||||
|
||||
**The payload is an encoded string, not a `Map`.** A map accepts any shape, defers validation to
|
||||
whichever handler reaches for a missing key, makes an entity trivially serializable onto the wire,
|
||||
and brings unbounded nesting with it.
|
||||
|
||||
**Handlers cannot write.** `WebSocketHandlerContext` has no session and no write method. This is
|
||||
what makes ordering, backpressure and the drain sequence guarantees rather than conventions — a
|
||||
handler that could write directly would bypass the queue, and every promise would hold only for the
|
||||
handlers that cooperated.
|
||||
|
||||
## Two findings from building it
|
||||
|
||||
**Tomcat's graceful shutdown does not close WebSocket connections.** It waits for in-flight
|
||||
*requests*, and an established WebSocket is not a request — so the shutdown completes with the
|
||||
connections still open and they die when the socket is torn down. The client sees a **1006**,
|
||||
indistinguishable from a network failure, which sends it into its most aggressive reconnect path at
|
||||
exactly the moment the fleet is restarting. `PlatformWebSocketHandler` therefore implements
|
||||
`SmartLifecycle` and closes its own connections with **1001 going away**, at a phase that runs
|
||||
before the web server stops. Found by `TomcatWebSocketAbuseIT`, which failed on its first run.
|
||||
|
||||
**Both runtimes on one classpath is a silent outage.** Spring Boot deduces one application type
|
||||
from what is present, and picks the servlet one. A deployment that declared reactive endpoints and
|
||||
shipped both starts, reports healthy, and never answers. `WebSocketStackExclusivity` reads what the
|
||||
classpath will actually produce and fails startup with a sentence explaining it.
|
||||
|
||||
## Lanes
|
||||
|
||||
```bash
|
||||
cd src
|
||||
./gradlew :adapter:inbound:websocket:test # unit, boundary, architecture, Tomcat runtime
|
||||
./gradlew :adapter:inbound:websocket:websocketJettyTest # the second servlet container
|
||||
./gradlew :adapter:inbound:websocket:websocketNginxTest # real Nginx; needs Docker, fails without it
|
||||
```
|
||||
|
||||
The Nginx lane carries a deliberately broken configuration alongside the correct one
|
||||
(`nginx-no-upgrade.conf`) so the lane proves its own assertions can fail. A contract that only ever
|
||||
runs against a correct configuration cannot tell whether it is checking anything.
|
||||
|
||||
## The Stable behaviours that had no code
|
||||
|
||||
A late audit compared every named type in the Stable plan against the source tree rather than
|
||||
against memory, and found six that nothing implemented. They are listed because the way they were
|
||||
missed is more useful than the fact that they were: each is a *behaviour under failure*, and the
|
||||
plan named it inside a task whose other half was already built — so the task read as done.
|
||||
|
||||
| Behaviour | Where it lives now |
|
||||
| --- | --- |
|
||||
| A queue observation an operator can read, separating a configured drop from a lossless overflow | `outbound/OutboundQueueSnapshot` + `OutboundQueue.snapshot()` |
|
||||
| A timed-out correlation id that is remembered, so a late answer is not attached to a reused id | `handler/LateResponseTombstone` |
|
||||
| Pooled-buffer retention on the reactive stack, bounded and counted | `webflux/WebSocketDataBufferPolicy` + `WebSocketDataBufferLifecycle` |
|
||||
| What a proxy in front of this platform has to do | `release/WebSocketNginxProxyProfile` |
|
||||
| What a rolling restart has to demonstrate, in order | `release/WebSocketRollingRestartScenario` |
|
||||
| What the platform must show before promotion | `release/WebSocketStableReleaseGate` |
|
||||
|
||||
The three `release` types share a module identity (`RELEASE`, CORE) that names no other module.
|
||||
That is deliberate: a gate importing the parts it gates would be satisfiable by construction — the
|
||||
evidence and the checklist would come from the same source. They are predicates over facts a release
|
||||
engineer supplies, so a missing runtime stays a missing runtime.
|
||||
|
||||
`WebSocketStableReleaseGate` overlaps `advanced/release/AdvancedPromotionGate` in shape and differs
|
||||
in one condition that matters. Advanced capabilities are off unless a deployment names them, so a
|
||||
broken one affects whoever enabled it; Stable is what every deployment gets, so its evidence has to
|
||||
cover every runtime it claims — Tomcat, Jetty, Reactor Netty and Nginx — rather than the one the
|
||||
author happened to test. It also refuses to promote an artifact containing an Advanced type. WS-ARCH-6
|
||||
already refuses that as a source edge; nothing in it notices a type that arrived through packaging,
|
||||
and the effect is identical — Stable that does not build without Advanced is a naming convention.
|
||||
|
||||
## Not yet implemented
|
||||
|
||||
**Tasks 47–48, the browser matrix.** The design asks for a protocol test client driven through
|
||||
Chromium, Firefox and WebKit. Not built: it needs Playwright and a browser download per engine,
|
||||
which is a dependency and a network requirement this template does not otherwise carry, and a
|
||||
browser lane that silently skips when the browsers are absent is worth less than no lane. The
|
||||
properties it would cover that the Java client does not — that a browser cannot set headers on the
|
||||
`WebSocket` constructor, and that it surfaces close codes to page script — are the reasons
|
||||
`ONE_TIME_TICKET` and the standard close codes exist, and both are asserted at the unit level.
|
||||
|
||||
## Advanced capabilities
|
||||
|
||||
The Advanced expansion plan asks for fifteen Gradle modules under `modules/websocket-advanced/`.
|
||||
They are packages under `advanced.**` in this leaf, for the same reason the Stable platform is —
|
||||
`src/config/architecture/modules.json` is fail-closed and owns the leaf list, and a fifteen-leaf
|
||||
addition to satisfy a directory layout is a change to the registry, not to the architecture. The
|
||||
separation the design wanted is enforced by `WebSocketStableModule` and by WS-ARCH-6, which fails
|
||||
the build when a Stable class names an Advanced one. A feature flag decides whether a bean exists;
|
||||
it does nothing about a compile-time edge, and ArchUnit does.
|
||||
|
||||
Three of the fifteen needed their own module identity rather than sharing `advanced`:
|
||||
|
||||
- **`advanced-stomp`** (`advanced.stomp`) is `FRAMEWORK_BOUND`. STOMP here *is* Spring Messaging, and
|
||||
folding it into `advanced` would have relaxed that module's purity for every capability in it.
|
||||
- **`advanced-stomp-rabbit`** (`advanced.stomp.rabbit`) is separate again. The adapter parses a
|
||||
protocol; the relay opens a TCP connection to somebody else's broker and makes every delivery
|
||||
depend on it. Different blast radius, so a deployment can refuse one and keep the other.
|
||||
- Everything else resolves to `advanced` by longest-prefix, which is what lets `advanced.codec.cbor`
|
||||
exist without its own edge set.
|
||||
|
||||
### What was adapted rather than copied
|
||||
|
||||
**Task 8's presence record shape.** The design specifies
|
||||
`(actorFingerprint, activeConnectionCount, state, lastObservedAt)` with a four-state observation
|
||||
model. Implemented with `WebSocketActorReference` in place of a bare fingerprint string — it carries
|
||||
the fingerprint and refuses to be constructed from raw identity, which is the property the design
|
||||
was buying with the field name. The four states are implemented as specified; `PresenceState.STALE`
|
||||
and `OFFLINE` are distinct because collapsing them reports every user as disconnected during a Redis
|
||||
partition, when the connections are fine and the index went dark.
|
||||
|
||||
**Task 14's Protobuf codec.** The descriptor compatibility gate and the profile are implemented and
|
||||
tested. The encode/decode path is not: a Protobuf codec without generated message classes has
|
||||
nothing to encode, and generating them requires a `.proto` source this template does not have and
|
||||
should not invent. `DescriptorCompatibilityGate` is the part with the failure mode worth guarding —
|
||||
a changed or reused field number reinterprets bytes already on the wire, and during a rolling deploy
|
||||
both descriptor versions are live, so the receiver reads the wrong field without erroring.
|
||||
|
||||
**Task 15's CBOR codec.** Same shape, same reason: `CborCodecProfile` fixes canonical encoding and
|
||||
the duplicate-key policy, which are the two settings that decide whether two systems reading the
|
||||
same bytes agree. `WebSocketCborCodec` implements the encode/decode path on top of it, and
|
||||
`SchemaParity` is what stops it publishing a different catalog than JSON.
|
||||
|
||||
`jackson-dataformat-cbor` and `protobuf-java` are `compileOnly` plus `testImplementation`, not
|
||||
`implementation`. Both jars change an adopter's behaviour by their mere presence — Spring Boot
|
||||
registers a `cborMapper` bean for the first and Spring registers a Protobuf message converter for
|
||||
the second — so an adopting composition root would acquire both without enabling either capability.
|
||||
The sibling web leaf shipped exactly that mistake and it broke the composition root outright; see
|
||||
`docs/web/repository-adaptation.md`. `WebSocketBinaryCodecBackend` turns an absent backend into a
|
||||
sentence naming the missing coordinate, and `BinaryCodecBackendScopeTest` reads `build.gradle` and
|
||||
fails if either returns to `implementation`.
|
||||
|
||||
**Task 12's relay configuration.** `RabbitBrokerRelayConfiguration` contributes only the broker, not
|
||||
the destination prefixes. Two `WebSocketMessageBrokerConfigurer` beans each setting the application
|
||||
prefix produce whichever ran last, silently — the same failure `StompBrokerExclusivity` exists to
|
||||
catch between the two STOMP channels.
|
||||
|
||||
### Verification
|
||||
|
||||
```bash
|
||||
cd src
|
||||
./gradlew :adapter:inbound:websocket:test # 509 tests, Advanced included
|
||||
./gradlew :adapter:inbound:websocket:websocketJettyTest
|
||||
./gradlew :adapter:inbound:websocket:websocketNginxTest
|
||||
```
|
||||
|
||||
See `docs/adr/ADR-WS-002-resume-and-cluster.md`, `docs/adr/ADR-WS-003-stomp-and-broker-relay.md`,
|
||||
`docs/websocket/advanced-support-matrix.md` and `docs/websocket/runbooks/`.
|
||||
@@ -0,0 +1,107 @@
|
||||
# WebSocket platform runbook
|
||||
|
||||
Organised by what you observe, because that is what you have during an incident.
|
||||
|
||||
## Clients report 1006 — "closed abnormally, no reason"
|
||||
|
||||
The most important symptom here, because 1006 is indistinguishable from a network failure and
|
||||
drives every client into its most aggressive reconnect path.
|
||||
|
||||
- **During a deploy** — the platform did not close its connections. `PlatformWebSocketHandler`
|
||||
implements `SmartLifecycle` and closes with **1001 going away**; a container's own graceful
|
||||
shutdown does **not** do this, because it waits for in-flight *requests* and an established
|
||||
WebSocket is not a request. Check the handler bean is registered and that its phase runs before
|
||||
the web server stops.
|
||||
- **Behind a proxy** — the upgrade is mishandled, so the close frame never becomes a close frame.
|
||||
See the next section.
|
||||
- **Neither** — something threw while closing. A close reason longer than 123 bytes makes the
|
||||
container throw mid-close and turns a clean refusal into a 1006; the platform truncates for that
|
||||
reason.
|
||||
|
||||
## Connections work locally and not behind the load balancer
|
||||
|
||||
Three nginx directives, each producing a different failure:
|
||||
|
||||
| Missing | Symptom |
|
||||
| --- | --- |
|
||||
| `proxy_http_version 1.1` | Handshake answered 400, or simply not upgraded. HTTP/1.0 has no Upgrade mechanism. |
|
||||
| `Upgrade` / `Connection` forwarded | Upstream sees an ordinary GET and answers 404 or 200 for a route that works when tested directly. They are hop-by-hop headers, so a proxy is *required* to drop them. |
|
||||
| `proxy_read_timeout` raised | Every connection quieter than 60s is killed with no close frame. A WebSocket is idle by nature. |
|
||||
|
||||
`websocketNginxTest` asserts all three against a real Nginx, and carries a deliberately broken
|
||||
configuration so it proves its own assertions can fail.
|
||||
|
||||
## Connections vanish with no close frame and no error
|
||||
|
||||
Almost always an intermediary's idle timeout, not the application. TCP does not report a departed
|
||||
peer — a closed laptop lid, a phone switching to cellular and a NAT forgetting its mapping all
|
||||
produce no FIN.
|
||||
|
||||
- Check `HeartbeatPolicy`: the ping interval must be under ~20s, because intermediaries commonly
|
||||
drop idle connections at 30–60s and say nothing.
|
||||
- The idle timeout must be at least two ping intervals. Below that, one dropped ping on a congested
|
||||
network closes a healthy connection, and the reconnect storm makes the congestion worse.
|
||||
|
||||
## Memory grows and nothing is failing
|
||||
|
||||
A slow consumer. It does not error — it reads more slowly than the server writes and the difference
|
||||
accumulates. Arranging it requires no tooling: reading slowly is enough.
|
||||
|
||||
- **Check first:** `WebSocketNodeSnapshot.droppedOutboundMessages`. Connection counts and buffered
|
||||
bytes both look healthy while data is being lost; the drop count is the only number that says so.
|
||||
- **Bounds:** per connection (`maxBufferedOutboundBytes`) *and* node-wide (`GlobalBufferBudget`).
|
||||
The second is not redundant — a megabyte each is fine at a hundred connections and is the whole
|
||||
heap at fifty thousand.
|
||||
- **If drops are zero and memory still grows:** check that the queue lock and the socket lock are
|
||||
separate. When they were one lock, a stalled peer blocked every producer, the queue never filled,
|
||||
and the shedding bound never fired. That is fixed, and it is the shape to look for if it recurs.
|
||||
|
||||
## A command ran twice
|
||||
|
||||
The reconnect is what makes this a WebSocket problem: a client replays everything it never saw an
|
||||
answer for, on a new connection, all at once.
|
||||
|
||||
1. **Key scoped to the connection?** It must not be. `WebSocketCommandKey` is endpoint + actor +
|
||||
the client's own message id, so it survives the reconnect.
|
||||
2. **Ledger not transactional?** The only implementation that works writes in the application's own
|
||||
transaction. Anything else has a window between the business commit and the ledger write.
|
||||
3. **Outcome `UNKNOWN`?** That is the window. Neither replaying nor re-running is safe — replaying
|
||||
invents a result, re-running duplicates a committed write. Only `CommandReconciliation` reading
|
||||
the business data can settle it, and `INDETERMINATE` is a real answer to act on.
|
||||
|
||||
## A client cannot connect and the endpoint is definitely there
|
||||
|
||||
- **403** — origin. The same-origin policy does **not** protect a WebSocket handshake and there is
|
||||
no preflight, so the server's `Origin` check is the entire defence and it is strict: exact match,
|
||||
lowercase, no path, no wildcard.
|
||||
- **401** — the ticket was spent, expired (30s cap), or minted for another endpoint. All three are
|
||||
deliberate; a longer-lived or unbound ticket is a bearer token in a query string again.
|
||||
- **400** — no offered subprotocol is supported. Production refuses a client that offers none.
|
||||
- **503** — the endpoint is at its connection cap.
|
||||
|
||||
Order matters and is asserted: route → origin → capacity → credential → subprotocol. Origin is
|
||||
checked before authentication because a cookie handshake from an attacker's page authenticates
|
||||
perfectly; the credential is valid and the page is not allowed to use it.
|
||||
|
||||
## Endpoints never answer and the app reports healthy
|
||||
|
||||
Both runtimes on one classpath. Spring Boot deduces one application type and picks the servlet one,
|
||||
so declared reactive endpoints are simply never served — no error, no warning.
|
||||
`WebSocketStackExclusivity` fails startup on this; if a running instance has it, the check is not
|
||||
wired.
|
||||
|
||||
## Metrics stopped arriving
|
||||
|
||||
A high-cardinality tag, and the risk is worse than for HTTP: a request produces one observation,
|
||||
a connection produces them for hours, so the series outlive the connections and only accumulate.
|
||||
`WebSocketMetricTags` allows eight names; `connectionId`, `sessionId` and `actor` are not among
|
||||
them. `nodeId` is, because a fleet has a knowable number of nodes.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
cd src
|
||||
./gradlew :adapter:inbound:websocket:test # unit, boundary, architecture, Tomcat runtime
|
||||
./gradlew :adapter:inbound:websocket:websocketJettyTest # second servlet container
|
||||
./gradlew :adapter:inbound:websocket:websocketNginxTest # real Nginx; needs Docker, fails without it
|
||||
```
|
||||
@@ -0,0 +1,55 @@
|
||||
# Runbook: STOMP broker outage
|
||||
|
||||
Applies when `advanced.stomp.relay` is enabled and the RabbitMQ broker becomes unreachable.
|
||||
|
||||
## What the symptom looks like
|
||||
|
||||
Not an error rate. The relay holds one TCP connection to the broker plus one per authenticated
|
||||
session, and a broker that stops responding without closing leaves all of them **open**. The
|
||||
platform's own health check stays green; connections stay established; publishes are accepted.
|
||||
|
||||
The first real signal is one of:
|
||||
|
||||
- subscriptions established but no `MESSAGE` frames arriving, for everyone at once;
|
||||
- the relay's system heartbeat failing to receive (`systemHeartbeatReceiveInterval` elapsed);
|
||||
- new sessions failing to subscribe while existing ones appear fine — this is the broker's
|
||||
connection limit, not the outage itself.
|
||||
|
||||
If the heartbeat is not configured, none of the above fires and the first signal is a user report.
|
||||
`RabbitBrokerRelayProfile` refuses a zero heartbeat for exactly this reason.
|
||||
|
||||
## Triage
|
||||
|
||||
1. **Confirm the direction.** From a platform node, open a STOMP connection to the broker's host and
|
||||
port directly. If that succeeds, the problem is the relay's connection state, not the broker.
|
||||
2. **Check the connection count against the broker's limit.** `brokerConnectionsFor(sessions)` is
|
||||
`sessions + 1`. A broker at its limit refuses new connections and serves existing ones, which
|
||||
produces the "new users cannot subscribe" shape.
|
||||
3. **Check whether messages are being accepted.** A half-open connection accepts every publish
|
||||
silently. Publishes succeeding is not evidence the broker is alive.
|
||||
|
||||
## Recovery
|
||||
|
||||
- **Broker restarted, relay did not reconnect.** The relay reconnects on its own; if it has not
|
||||
within two heartbeat intervals, restart the platform nodes one at a time. Do not restart them all
|
||||
at once — every session reconnects simultaneously and the broker meets its whole client population
|
||||
in one instant.
|
||||
- **Broker at its connection limit.** Raise the limit or shed sessions. Shedding is the faster of
|
||||
the two and the connection count falls with the sessions.
|
||||
- **Broker gone and not coming back.** There is no safe fallback to the simple broker in a
|
||||
multi-node deployment: it delivers to whichever fraction of users is on the publishing node.
|
||||
`SimpleBrokerProfile.activatableUnder` refuses it outside local/test, and that refusal should not
|
||||
be overridden during an incident. Scale to a single node first if the simple broker is the only
|
||||
option.
|
||||
|
||||
## What is lost
|
||||
|
||||
Anything the broker held and did not persist. `StompEvidence.BROKER_ACK` is only durable if the
|
||||
broker is durable, and RabbitMQ's durability is a property of the queue topology, not of the relay.
|
||||
Messages acknowledged at `PROTOCOL_RECEIPT` were never in the broker at all.
|
||||
|
||||
## Afterwards
|
||||
|
||||
- If the heartbeat did not fire first, that is the finding. Fix it before the postmortem closes.
|
||||
- If the connection limit was reached, record the session count that reached it. It is a hard
|
||||
ceiling on the deployment and it is not otherwise written down anywhere.
|
||||
@@ -0,0 +1,56 @@
|
||||
# Runbook: resume history loss
|
||||
|
||||
Applies when `advanced.resume` is enabled and clients present resume tokens the replay store can no
|
||||
longer satisfy.
|
||||
|
||||
## What the symptom looks like
|
||||
|
||||
Clients reconnecting and resynchronising rather than resuming. This is the **designed** behaviour,
|
||||
not a fault — `ResumeCoordinator` consults `ReplayAvailability` and refuses to honour a position the
|
||||
store has evicted, because delivering a stream with a hole in it is worse than an explicit
|
||||
resynchronise.
|
||||
|
||||
It becomes an incident when the resynchronise rate is high enough to matter:
|
||||
|
||||
- a resynchronise means the client re-reads its whole state, so a spike is a load spike on whatever
|
||||
serves that state;
|
||||
- for a client that cannot resynchronise cheaply, it is user-visible as a stall.
|
||||
|
||||
## Triage
|
||||
|
||||
1. **Establish which of the three causes it is.**
|
||||
- *Store eviction under load.* The replay store's retention is shorter than the disconnect
|
||||
durations being seen. Look at retention against reconnect latency, not against a nominal
|
||||
figure.
|
||||
- *Store restarted or partitioned.* Availability drops to nothing and every token fails at once.
|
||||
- *Key rotation.* `ResumeTokenKeyRing` verifies against retired keys as well as the current one;
|
||||
if a key was removed rather than retired, every token minted under it fails to verify. This
|
||||
produces the same symptom and a different fix.
|
||||
|
||||
`ResumeTokenOutcome` distinguishes these. A token that fails verification is not the same as one
|
||||
that verifies and names an evicted position.
|
||||
|
||||
2. **Check whether the resynchronise is succeeding.** A high resynchronise rate that completes is a
|
||||
capacity problem. One that fails is a correctness problem and is more urgent.
|
||||
|
||||
## Recovery
|
||||
|
||||
- **Eviction under load.** Raise retention if the store can hold it. Retention is bounded by memory,
|
||||
so this trades against the store's own stability — do not raise it past what the store survives.
|
||||
- **Store restarted.** Nothing to recover; the tokens are genuinely unsatisfiable. Let clients
|
||||
resynchronise. If the resynchronise load is the problem, shed connections so they arrive in
|
||||
batches rather than all at once.
|
||||
- **Key removed rather than retired.** Restore the key to the ring as a verify-only entry. Minting
|
||||
continues under the current key.
|
||||
|
||||
## What is lost
|
||||
|
||||
Nothing that was acknowledged. Resume is an optimisation over resynchronise; the client's ability to
|
||||
rebuild its state from the authoritative source is the actual guarantee, and it is unaffected.
|
||||
|
||||
## Afterwards
|
||||
|
||||
- If retention was the cause, record the disconnect duration distribution that exceeded it. The
|
||||
nominal retention figure is meaningless without it.
|
||||
- If a key was removed, that is a process finding, not a platform one. Keys are retired, never
|
||||
deleted, and the ring is the place that is enforced.
|
||||
Reference in New Issue
Block a user