feat: web, websocket 어댑터 추가 구현

This commit is contained in:
DongHyeonka
2026-08-28 17:01:27 +09:00
parent 0137263441
commit a24ece9cf7
883 changed files with 100584 additions and 2623 deletions
+55
View File
@@ -0,0 +1,55 @@
# Runbook: STOMP broker outage
Applies when `advanced.stomp.relay` is enabled and the RabbitMQ broker becomes unreachable.
## What the symptom looks like
Not an error rate. The relay holds one TCP connection to the broker plus one per authenticated
session, and a broker that stops responding without closing leaves all of them **open**. The
platform's own health check stays green; connections stay established; publishes are accepted.
The first real signal is one of:
- subscriptions established but no `MESSAGE` frames arriving, for everyone at once;
- the relay's system heartbeat failing to receive (`systemHeartbeatReceiveInterval` elapsed);
- new sessions failing to subscribe while existing ones appear fine — this is the broker's
connection limit, not the outage itself.
If the heartbeat is not configured, none of the above fires and the first signal is a user report.
`RabbitBrokerRelayProfile` refuses a zero heartbeat for exactly this reason.
## Triage
1. **Confirm the direction.** From a platform node, open a STOMP connection to the broker's host and
port directly. If that succeeds, the problem is the relay's connection state, not the broker.
2. **Check the connection count against the broker's limit.** `brokerConnectionsFor(sessions)` is
`sessions + 1`. A broker at its limit refuses new connections and serves existing ones, which
produces the "new users cannot subscribe" shape.
3. **Check whether messages are being accepted.** A half-open connection accepts every publish
silently. Publishes succeeding is not evidence the broker is alive.
## Recovery
- **Broker restarted, relay did not reconnect.** The relay reconnects on its own; if it has not
within two heartbeat intervals, restart the platform nodes one at a time. Do not restart them all
at once — every session reconnects simultaneously and the broker meets its whole client population
in one instant.
- **Broker at its connection limit.** Raise the limit or shed sessions. Shedding is the faster of
the two and the connection count falls with the sessions.
- **Broker gone and not coming back.** There is no safe fallback to the simple broker in a
multi-node deployment: it delivers to whichever fraction of users is on the publishing node.
`SimpleBrokerProfile.activatableUnder` refuses it outside local/test, and that refusal should not
be overridden during an incident. Scale to a single node first if the simple broker is the only
option.
## What is lost
Anything the broker held and did not persist. `StompEvidence.BROKER_ACK` is only durable if the
broker is durable, and RabbitMQ's durability is a property of the queue topology, not of the relay.
Messages acknowledged at `PROTOCOL_RECEIPT` were never in the broker at all.
## Afterwards
- If the heartbeat did not fire first, that is the finding. Fix it before the postmortem closes.
- If the connection limit was reached, record the session count that reached it. It is a hard
ceiling on the deployment and it is not otherwise written down anywhere.
@@ -0,0 +1,56 @@
# Runbook: resume history loss
Applies when `advanced.resume` is enabled and clients present resume tokens the replay store can no
longer satisfy.
## What the symptom looks like
Clients reconnecting and resynchronising rather than resuming. This is the **designed** behaviour,
not a fault — `ResumeCoordinator` consults `ReplayAvailability` and refuses to honour a position the
store has evicted, because delivering a stream with a hole in it is worse than an explicit
resynchronise.
It becomes an incident when the resynchronise rate is high enough to matter:
- a resynchronise means the client re-reads its whole state, so a spike is a load spike on whatever
serves that state;
- for a client that cannot resynchronise cheaply, it is user-visible as a stall.
## Triage
1. **Establish which of the three causes it is.**
- *Store eviction under load.* The replay store's retention is shorter than the disconnect
durations being seen. Look at retention against reconnect latency, not against a nominal
figure.
- *Store restarted or partitioned.* Availability drops to nothing and every token fails at once.
- *Key rotation.* `ResumeTokenKeyRing` verifies against retired keys as well as the current one;
if a key was removed rather than retired, every token minted under it fails to verify. This
produces the same symptom and a different fix.
`ResumeTokenOutcome` distinguishes these. A token that fails verification is not the same as one
that verifies and names an evicted position.
2. **Check whether the resynchronise is succeeding.** A high resynchronise rate that completes is a
capacity problem. One that fails is a correctness problem and is more urgent.
## Recovery
- **Eviction under load.** Raise retention if the store can hold it. Retention is bounded by memory,
so this trades against the store's own stability — do not raise it past what the store survives.
- **Store restarted.** Nothing to recover; the tokens are genuinely unsatisfiable. Let clients
resynchronise. If the resynchronise load is the problem, shed connections so they arrive in
batches rather than all at once.
- **Key removed rather than retired.** Restore the key to the ring as a verify-only entry. Minting
continues under the current key.
## What is lost
Nothing that was acknowledged. Resume is an optimisation over resynchronise; the client's ability to
rebuild its state from the authoritative source is the actual guarantee, and it is unaffected.
## Afterwards
- If retention was the cause, record the disconnect duration distribution that exceeded it. The
nominal retention figure is meaningless without it.
- If a key was removed, that is a process finding, not a platform one. Keys are retired, never
deleted, and the ring is the place that is enforced.