57 lines
2.8 KiB
Markdown
57 lines
2.8 KiB
Markdown
# Runbook: resume history loss
|
|
|
|
Applies when `advanced.resume` is enabled and clients present resume tokens the replay store can no
|
|
longer satisfy.
|
|
|
|
## What the symptom looks like
|
|
|
|
Clients reconnecting and resynchronising rather than resuming. This is the **designed** behaviour,
|
|
not a fault — `ResumeCoordinator` consults `ReplayAvailability` and refuses to honour a position the
|
|
store has evicted, because delivering a stream with a hole in it is worse than an explicit
|
|
resynchronise.
|
|
|
|
It becomes an incident when the resynchronise rate is high enough to matter:
|
|
|
|
- a resynchronise means the client re-reads its whole state, so a spike is a load spike on whatever
|
|
serves that state;
|
|
- for a client that cannot resynchronise cheaply, it is user-visible as a stall.
|
|
|
|
## Triage
|
|
|
|
1. **Establish which of the three causes it is.**
|
|
- *Store eviction under load.* The replay store's retention is shorter than the disconnect
|
|
durations being seen. Look at retention against reconnect latency, not against a nominal
|
|
figure.
|
|
- *Store restarted or partitioned.* Availability drops to nothing and every token fails at once.
|
|
- *Key rotation.* `ResumeTokenKeyRing` verifies against retired keys as well as the current one;
|
|
if a key was removed rather than retired, every token minted under it fails to verify. This
|
|
produces the same symptom and a different fix.
|
|
|
|
`ResumeTokenOutcome` distinguishes these. A token that fails verification is not the same as one
|
|
that verifies and names an evicted position.
|
|
|
|
2. **Check whether the resynchronise is succeeding.** A high resynchronise rate that completes is a
|
|
capacity problem. One that fails is a correctness problem and is more urgent.
|
|
|
|
## Recovery
|
|
|
|
- **Eviction under load.** Raise retention if the store can hold it. Retention is bounded by memory,
|
|
so this trades against the store's own stability — do not raise it past what the store survives.
|
|
- **Store restarted.** Nothing to recover; the tokens are genuinely unsatisfiable. Let clients
|
|
resynchronise. If the resynchronise load is the problem, shed connections so they arrive in
|
|
batches rather than all at once.
|
|
- **Key removed rather than retired.** Restore the key to the ring as a verify-only entry. Minting
|
|
continues under the current key.
|
|
|
|
## What is lost
|
|
|
|
Nothing that was acknowledged. Resume is an optimisation over resynchronise; the client's ability to
|
|
rebuild its state from the authoritative source is the actual guarantee, and it is unaffected.
|
|
|
|
## Afterwards
|
|
|
|
- If retention was the cause, record the disconnect duration distribution that exceeded it. The
|
|
nominal retention figure is meaningless without it.
|
|
- If a key was removed, that is a process finding, not a platform one. Keys are retired, never
|
|
deleted, and the ring is the place that is enforced.
|