# Runbook: resume history loss

Applies when `advanced.resume` is enabled and clients present resume tokens the replay store can no
longer satisfy.

## What the symptom looks like

Clients reconnecting and resynchronising rather than resuming. This is the **designed** behaviour,
not a fault — `ResumeCoordinator` consults `ReplayAvailability` and refuses to honour a position the
store has evicted, because delivering a stream with a hole in it is worse than an explicit
resynchronise.

It becomes an incident when the resynchronise rate is high enough to matter:

- a resynchronise means the client re-reads its whole state, so a spike is a load spike on whatever
  serves that state;
- for a client that cannot resynchronise cheaply, it is user-visible as a stall.

## Triage

1. **Establish which of the three causes it is.**
   - *Store eviction under load.* The replay store's retention is shorter than the disconnect
     durations being seen. Look at retention against reconnect latency, not against a nominal
     figure.
   - *Store restarted or partitioned.* Availability drops to nothing and every token fails at once.
   - *Key rotation.* `ResumeTokenKeyRing` verifies against retired keys as well as the current one;
     if a key was removed rather than retired, every token minted under it fails to verify. This
     produces the same symptom and a different fix.

   `ResumeTokenOutcome` distinguishes these. A token that fails verification is not the same as one
   that verifies and names an evicted position.

2. **Check whether the resynchronise is succeeding.** A high resynchronise rate that completes is a
   capacity problem. One that fails is a correctness problem and is more urgent.

## Recovery

- **Eviction under load.** Raise retention if the store can hold it. Retention is bounded by memory,
  so this trades against the store's own stability — do not raise it past what the store survives.
- **Store restarted.** Nothing to recover; the tokens are genuinely unsatisfiable. Let clients
  resynchronise. If the resynchronise load is the problem, shed connections so they arrive in
  batches rather than all at once.
- **Key removed rather than retired.** Restore the key to the ring as a verify-only entry. Minting
  continues under the current key.

## What is lost

Nothing that was acknowledged. Resume is an optimisation over resynchronise; the client's ability to
rebuild its state from the authoritative source is the actual guarantee, and it is unaffected.

## Afterwards

- If retention was the cause, record the disconnect duration distribution that exceeded it. The
  nominal retention figure is meaningless without it.
- If a key was removed, that is a process finding, not a platform one. Keys are retired, never
  deleted, and the ring is the place that is enforced.