Runbook: resume history loss

Applies when advanced.resume is enabled and clients present resume tokens the replay store can no longer satisfy.

What the symptom looks like

Clients reconnecting and resynchronising rather than resuming. This is the designed behaviour, not a fault — ResumeCoordinator consults ReplayAvailability and refuses to honour a position the store has evicted, because delivering a stream with a hole in it is worse than an explicit resynchronise.

It becomes an incident when the resynchronise rate is high enough to matter:

a resynchronise means the client re-reads its whole state, so a spike is a load spike on whatever serves that state;
for a client that cannot resynchronise cheaply, it is user-visible as a stall.

Triage

Establish which of the three causes it is.
- Store eviction under load. The replay store's retention is shorter than the disconnect durations being seen. Look at retention against reconnect latency, not against a nominal figure.
- Store restarted or partitioned. Availability drops to nothing and every token fails at once.
- Key rotation. ResumeTokenKeyRing verifies against retired keys as well as the current one; if a key was removed rather than retired, every token minted under it fails to verify. This produces the same symptom and a different fix.
ResumeTokenOutcome distinguishes these. A token that fails verification is not the same as one that verifies and names an evicted position.
Check whether the resynchronise is succeeding. A high resynchronise rate that completes is a capacity problem. One that fails is a correctness problem and is more urgent.

Recovery

Eviction under load. Raise retention if the store can hold it. Retention is bounded by memory, so this trades against the store's own stability — do not raise it past what the store survives.
Store restarted. Nothing to recover; the tokens are genuinely unsatisfiable. Let clients resynchronise. If the resynchronise load is the problem, shed connections so they arrive in batches rather than all at once.
Key removed rather than retired. Restore the key to the ring as a verify-only entry. Minting continues under the current key.

What is lost

Nothing that was acknowledged. Resume is an optimisation over resynchronise; the client's ability to rebuild its state from the authoritative source is the actual guarantee, and it is unaffected.

Afterwards

If retention was the cause, record the disconnect duration distribution that exceeded it. The nominal retention figure is meaningless without it.
If a key was removed, that is a process finding, not a platform one. Keys are retired, never deleted, and the ring is the place that is enforced.

2.8 KiB Raw Blame History