---
title: Runbook — MongoDB change stream history lost
category: mongodb
severity: P1
owner: oncall
last_updated: 2026-08-13
status: active
---

# Runbook: change stream history lost

Design §20.3, scenarios `OPLOG_HISTORY_LOSS`, `RESUME_TOKEN_LOSS`.

The stored resume token predates the oldest entry in the oplog. The events between the checkpoint and
now are gone from the server; no retry recovers them. `MongoChangeStreamRecoveryPolicy` returns
`halt(HISTORY_LOST, "history-lost")` and the runner stops.

**The platform will not silently restart from "now".** That looks like a recovery and is actually a
permanent, unreported gap in the projection.

## Symptoms

- `MongoChangeHistoryLostException`.
- `MongoChangeStreamState.HISTORY_LOST`; the consumer is stopped, not looping.
- Precedes it: consumer lag approaching the oplog window, or a consumer that was down for a long
  period (a deploy that failed, a scaled-to-zero worker, a long outage).

## Diagnosis

1. **Determine the gap.** The checkpoint's cluster time is the start; the oldest oplog entry is the
   end of what is unrecoverable. Everything in between was never processed.
2. **Determine the oplog window.** `rs.printReplicationInfo()` on the primary gives the first and last
   oplog timestamps. If the window is materially smaller than it was, the write rate rose or the
   oplog was resized — the consumer may be fine and the server changed.
3. **Determine what the projection is missing.** Which collections and which operations does this
   projector consume? The gap is bounded by that, not by everything that happened.
4. **Check for a second consumer.** If another projector on the same collection is healthy, its
   checkpoint tells you whether the problem is this consumer or the oplog.

## Action

Resuming is not an option. The choices are:

**Rebuild from source.** If the projection is derivable from the current state of the source
collections, rebuild it: stop the consumer, rebuild the projection, then start the stream from the
cluster time at which the rebuild snapshot was taken. This is the correct answer whenever the
projection is a materialised view rather than an event log, and it is the reason a projection should
be derivable.

**Backfill the gap.** If the source documents carry a timestamp covering the gap, run a bounded
backfill for that window through the migration runner (checkpointed, resumable — see
[schema-index-migration-guide.md](../schema-index-migration-guide.md) §5), then resume from the
current cluster time.

**Accept the gap explicitly.** Only when the projection is advisory and the business owner says so.
Record the window in the incident log and reset the checkpoint. This is a decision someone signs, not
a default.

Never: reset the checkpoint to "now" and restart quietly. That converts a visible P1 into an
invisible data-quality defect that surfaces months later as "the report has been wrong since March".

## Prevention

- **Alert on lag against the oplog window, not wall-clock.** "Consumer is 30 minutes behind" is fine
  with a 24-hour oplog and an emergency with a 45-minute one. The threshold that matters is
  `lag / oplogWindow`.
- **Size the oplog for the longest tolerable consumer outage**, including a failed deploy discovered
  the next morning.
- **Checkpoint after processing, never on receipt** — see
  [change-stream-guide.md](../change-stream-guide.md) §3.
- **Back up the checkpoint store.** `RESUME_TOKEN_LOSS` is the same incident reached from the other
  direction: the oplog is fine, the checkpoint is gone.
- **Make the projection rebuildable.** A projection that can only be built by replaying every event
  has no recovery path once the oplog rolls.

## Escalation

- P1 on detection. The consumer is stopped, so lag grows for as long as this is unresolved.
- Page the service owner for the rebuild decision, and the database owner if the oplog window shrank
  unexpectedly.

## Verification

```bash
cd src
./gradlew :adapter:outbound:persistence-mongo:mongoFailoverTest --console=plain
```

`MongoFailoverScenario.OPLOG_HISTORY_LOSS` and `RESUME_TOKEN_LOSS` assert the runner halts and names
this runbook rather than restarting from the current position.