feat: web, websocket 어댑터 추가 구현
This commit is contained in:
@@ -0,0 +1,55 @@
|
||||
# Runbook: STOMP broker outage
|
||||
|
||||
Applies when `advanced.stomp.relay` is enabled and the RabbitMQ broker becomes unreachable.
|
||||
|
||||
## What the symptom looks like
|
||||
|
||||
Not an error rate. The relay holds one TCP connection to the broker plus one per authenticated
|
||||
session, and a broker that stops responding without closing leaves all of them **open**. The
|
||||
platform's own health check stays green; connections stay established; publishes are accepted.
|
||||
|
||||
The first real signal is one of:
|
||||
|
||||
- subscriptions established but no `MESSAGE` frames arriving, for everyone at once;
|
||||
- the relay's system heartbeat failing to receive (`systemHeartbeatReceiveInterval` elapsed);
|
||||
- new sessions failing to subscribe while existing ones appear fine — this is the broker's
|
||||
connection limit, not the outage itself.
|
||||
|
||||
If the heartbeat is not configured, none of the above fires and the first signal is a user report.
|
||||
`RabbitBrokerRelayProfile` refuses a zero heartbeat for exactly this reason.
|
||||
|
||||
## Triage
|
||||
|
||||
1. **Confirm the direction.** From a platform node, open a STOMP connection to the broker's host and
|
||||
port directly. If that succeeds, the problem is the relay's connection state, not the broker.
|
||||
2. **Check the connection count against the broker's limit.** `brokerConnectionsFor(sessions)` is
|
||||
`sessions + 1`. A broker at its limit refuses new connections and serves existing ones, which
|
||||
produces the "new users cannot subscribe" shape.
|
||||
3. **Check whether messages are being accepted.** A half-open connection accepts every publish
|
||||
silently. Publishes succeeding is not evidence the broker is alive.
|
||||
|
||||
## Recovery
|
||||
|
||||
- **Broker restarted, relay did not reconnect.** The relay reconnects on its own; if it has not
|
||||
within two heartbeat intervals, restart the platform nodes one at a time. Do not restart them all
|
||||
at once — every session reconnects simultaneously and the broker meets its whole client population
|
||||
in one instant.
|
||||
- **Broker at its connection limit.** Raise the limit or shed sessions. Shedding is the faster of
|
||||
the two and the connection count falls with the sessions.
|
||||
- **Broker gone and not coming back.** There is no safe fallback to the simple broker in a
|
||||
multi-node deployment: it delivers to whichever fraction of users is on the publishing node.
|
||||
`SimpleBrokerProfile.activatableUnder` refuses it outside local/test, and that refusal should not
|
||||
be overridden during an incident. Scale to a single node first if the simple broker is the only
|
||||
option.
|
||||
|
||||
## What is lost
|
||||
|
||||
Anything the broker held and did not persist. `StompEvidence.BROKER_ACK` is only durable if the
|
||||
broker is durable, and RabbitMQ's durability is a property of the queue topology, not of the relay.
|
||||
Messages acknowledged at `PROTOCOL_RECEIPT` were never in the broker at all.
|
||||
|
||||
## Afterwards
|
||||
|
||||
- If the heartbeat did not fire first, that is the finding. Fix it before the postmortem closes.
|
||||
- If the connection limit was reached, record the session count that reached it. It is a hard
|
||||
ceiling on the deployment and it is not otherwise written down anywhere.
|
||||
@@ -0,0 +1,56 @@
|
||||
# Runbook: resume history loss
|
||||
|
||||
Applies when `advanced.resume` is enabled and clients present resume tokens the replay store can no
|
||||
longer satisfy.
|
||||
|
||||
## What the symptom looks like
|
||||
|
||||
Clients reconnecting and resynchronising rather than resuming. This is the **designed** behaviour,
|
||||
not a fault — `ResumeCoordinator` consults `ReplayAvailability` and refuses to honour a position the
|
||||
store has evicted, because delivering a stream with a hole in it is worse than an explicit
|
||||
resynchronise.
|
||||
|
||||
It becomes an incident when the resynchronise rate is high enough to matter:
|
||||
|
||||
- a resynchronise means the client re-reads its whole state, so a spike is a load spike on whatever
|
||||
serves that state;
|
||||
- for a client that cannot resynchronise cheaply, it is user-visible as a stall.
|
||||
|
||||
## Triage
|
||||
|
||||
1. **Establish which of the three causes it is.**
|
||||
- *Store eviction under load.* The replay store's retention is shorter than the disconnect
|
||||
durations being seen. Look at retention against reconnect latency, not against a nominal
|
||||
figure.
|
||||
- *Store restarted or partitioned.* Availability drops to nothing and every token fails at once.
|
||||
- *Key rotation.* `ResumeTokenKeyRing` verifies against retired keys as well as the current one;
|
||||
if a key was removed rather than retired, every token minted under it fails to verify. This
|
||||
produces the same symptom and a different fix.
|
||||
|
||||
`ResumeTokenOutcome` distinguishes these. A token that fails verification is not the same as one
|
||||
that verifies and names an evicted position.
|
||||
|
||||
2. **Check whether the resynchronise is succeeding.** A high resynchronise rate that completes is a
|
||||
capacity problem. One that fails is a correctness problem and is more urgent.
|
||||
|
||||
## Recovery
|
||||
|
||||
- **Eviction under load.** Raise retention if the store can hold it. Retention is bounded by memory,
|
||||
so this trades against the store's own stability — do not raise it past what the store survives.
|
||||
- **Store restarted.** Nothing to recover; the tokens are genuinely unsatisfiable. Let clients
|
||||
resynchronise. If the resynchronise load is the problem, shed connections so they arrive in
|
||||
batches rather than all at once.
|
||||
- **Key removed rather than retired.** Restore the key to the ring as a verify-only entry. Minting
|
||||
continues under the current key.
|
||||
|
||||
## What is lost
|
||||
|
||||
Nothing that was acknowledged. Resume is an optimisation over resynchronise; the client's ability to
|
||||
rebuild its state from the authoritative source is the actual guarantee, and it is unaffected.
|
||||
|
||||
## Afterwards
|
||||
|
||||
- If retention was the cause, record the disconnect duration distribution that exceeded it. The
|
||||
nominal retention figure is meaningless without it.
|
||||
- If a key was removed, that is a process finding, not a platform one. Keys are retired, never
|
||||
deleted, and the ring is the place that is enforced.
|
||||
Reference in New Issue
Block a user