feat: web, websocket 어댑터 추가 구현
This commit is contained in:
@@ -0,0 +1,56 @@
|
||||
# Runbook: resume history loss
|
||||
|
||||
Applies when `advanced.resume` is enabled and clients present resume tokens the replay store can no
|
||||
longer satisfy.
|
||||
|
||||
## What the symptom looks like
|
||||
|
||||
Clients reconnecting and resynchronising rather than resuming. This is the **designed** behaviour,
|
||||
not a fault — `ResumeCoordinator` consults `ReplayAvailability` and refuses to honour a position the
|
||||
store has evicted, because delivering a stream with a hole in it is worse than an explicit
|
||||
resynchronise.
|
||||
|
||||
It becomes an incident when the resynchronise rate is high enough to matter:
|
||||
|
||||
- a resynchronise means the client re-reads its whole state, so a spike is a load spike on whatever
|
||||
serves that state;
|
||||
- for a client that cannot resynchronise cheaply, it is user-visible as a stall.
|
||||
|
||||
## Triage
|
||||
|
||||
1. **Establish which of the three causes it is.**
|
||||
- *Store eviction under load.* The replay store's retention is shorter than the disconnect
|
||||
durations being seen. Look at retention against reconnect latency, not against a nominal
|
||||
figure.
|
||||
- *Store restarted or partitioned.* Availability drops to nothing and every token fails at once.
|
||||
- *Key rotation.* `ResumeTokenKeyRing` verifies against retired keys as well as the current one;
|
||||
if a key was removed rather than retired, every token minted under it fails to verify. This
|
||||
produces the same symptom and a different fix.
|
||||
|
||||
`ResumeTokenOutcome` distinguishes these. A token that fails verification is not the same as one
|
||||
that verifies and names an evicted position.
|
||||
|
||||
2. **Check whether the resynchronise is succeeding.** A high resynchronise rate that completes is a
|
||||
capacity problem. One that fails is a correctness problem and is more urgent.
|
||||
|
||||
## Recovery
|
||||
|
||||
- **Eviction under load.** Raise retention if the store can hold it. Retention is bounded by memory,
|
||||
so this trades against the store's own stability — do not raise it past what the store survives.
|
||||
- **Store restarted.** Nothing to recover; the tokens are genuinely unsatisfiable. Let clients
|
||||
resynchronise. If the resynchronise load is the problem, shed connections so they arrive in
|
||||
batches rather than all at once.
|
||||
- **Key removed rather than retired.** Restore the key to the ring as a verify-only entry. Minting
|
||||
continues under the current key.
|
||||
|
||||
## What is lost
|
||||
|
||||
Nothing that was acknowledged. Resume is an optimisation over resynchronise; the client's ability to
|
||||
rebuild its state from the authoritative source is the actual guarantee, and it is unaffected.
|
||||
|
||||
## Afterwards
|
||||
|
||||
- If retention was the cause, record the disconnect duration distribution that exceeded it. The
|
||||
nominal retention figure is meaningless without it.
|
||||
- If a key was removed, that is a process finding, not a platform one. Keys are retired, never
|
||||
deleted, and the ring is the place that is enforced.
|
||||
Reference in New Issue
Block a user