DongHyeonkaandClaude Opus 5
d57d2f62a0
Implements the mongodb-superpowers-package design: Stable Tasks 1-50 and Advanced Tasks 1-15. The design assumes 19 Stable + 12 Advanced Gradle projects under modules/mongodb*. This repository's fail-closed registry declares exactly 19 leaf identities, so those modules become package boundaries inside the registered leaf :adapter:outbound:persistence-mongo, with the design's module dependency table enforced by ten ArchUnit rules. The mapping and every deviation are recorded in docs/mongodb/repository-adaptation.md. Contract highlights, all enforced by tests rather than convention: - Transaction body retry and commit retry are separate loops. A new session per body attempt; commit-only retry on an unknown commit. The body is never replayed after a commit ambiguity, so a failover cannot become a duplicate. - MongoExecutionOutcome keeps both ambiguous outcomes distinct from success and failure, and MongoFailureContext records only the design-permitted fields. - Failure classification reads server error labels before numeric codes. - BSON representations come from a pinned manifest, never a library default, and a golden type-signature gate fails on any drift. - Index and validator changes go through the manifest and the admin plane; metadata ownership gates every drop. - Every Advanced capability refuses construction unless its flag is enabled. Verified against real servers, not only unit tests. Running the lanes for the first time exposed four defects that a green `check` had hidden: - Four release lanes passed while executing zero tests; the gate now counts executed tests per lane and fails on zero. - The "single replica set" fixture was a standalone, because Testcontainers 2.x needs withReplicaSet(); its test only asserted a connection string. - The three-node fixture was three independent clusters, so no election could occur, and awaitNewPrimary() compared against the post-stop primary. - The migration lease checked modifiedCount, so a same-millisecond refresh read as a lost lease. scripts/verify-mongodb-platform.sh now reports: 9 lanes, 0 skipped, 0 failed, every evidence category produced. scripts/verify-mongodb-advanced.sh reports NOT PROMOTABLE: actual-topology evidence (real sharded cluster, real KMS, real target deployment) is unobtainable here, so it is named rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
91 lines
4.1 KiB
Markdown
91 lines
4.1 KiB
Markdown
---
|
|
title: Runbook — MongoDB change stream history lost
|
|
category: mongodb
|
|
severity: P1
|
|
owner: oncall
|
|
last_updated: 2026-08-13
|
|
status: active
|
|
---
|
|
|
|
# Runbook: change stream history lost
|
|
|
|
Design §20.3, scenarios `OPLOG_HISTORY_LOSS`, `RESUME_TOKEN_LOSS`.
|
|
|
|
The stored resume token predates the oldest entry in the oplog. The events between the checkpoint and
|
|
now are gone from the server; no retry recovers them. `MongoChangeStreamRecoveryPolicy` returns
|
|
`halt(HISTORY_LOST, "history-lost")` and the runner stops.
|
|
|
|
**The platform will not silently restart from "now".** That looks like a recovery and is actually a
|
|
permanent, unreported gap in the projection.
|
|
|
|
## Symptoms
|
|
|
|
- `MongoChangeHistoryLostException`.
|
|
- `MongoChangeStreamState.HISTORY_LOST`; the consumer is stopped, not looping.
|
|
- Precedes it: consumer lag approaching the oplog window, or a consumer that was down for a long
|
|
period (a deploy that failed, a scaled-to-zero worker, a long outage).
|
|
|
|
## Diagnosis
|
|
|
|
1. **Determine the gap.** The checkpoint's cluster time is the start; the oldest oplog entry is the
|
|
end of what is unrecoverable. Everything in between was never processed.
|
|
2. **Determine the oplog window.** `rs.printReplicationInfo()` on the primary gives the first and last
|
|
oplog timestamps. If the window is materially smaller than it was, the write rate rose or the
|
|
oplog was resized — the consumer may be fine and the server changed.
|
|
3. **Determine what the projection is missing.** Which collections and which operations does this
|
|
projector consume? The gap is bounded by that, not by everything that happened.
|
|
4. **Check for a second consumer.** If another projector on the same collection is healthy, its
|
|
checkpoint tells you whether the problem is this consumer or the oplog.
|
|
|
|
## Action
|
|
|
|
Resuming is not an option. The choices are:
|
|
|
|
**Rebuild from source.** If the projection is derivable from the current state of the source
|
|
collections, rebuild it: stop the consumer, rebuild the projection, then start the stream from the
|
|
cluster time at which the rebuild snapshot was taken. This is the correct answer whenever the
|
|
projection is a materialised view rather than an event log, and it is the reason a projection should
|
|
be derivable.
|
|
|
|
**Backfill the gap.** If the source documents carry a timestamp covering the gap, run a bounded
|
|
backfill for that window through the migration runner (checkpointed, resumable — see
|
|
[schema-index-migration-guide.md](../schema-index-migration-guide.md) §5), then resume from the
|
|
current cluster time.
|
|
|
|
**Accept the gap explicitly.** Only when the projection is advisory and the business owner says so.
|
|
Record the window in the incident log and reset the checkpoint. This is a decision someone signs, not
|
|
a default.
|
|
|
|
Never: reset the checkpoint to "now" and restart quietly. That converts a visible P1 into an
|
|
invisible data-quality defect that surfaces months later as "the report has been wrong since March".
|
|
|
|
## Prevention
|
|
|
|
- **Alert on lag against the oplog window, not wall-clock.** "Consumer is 30 minutes behind" is fine
|
|
with a 24-hour oplog and an emergency with a 45-minute one. The threshold that matters is
|
|
`lag / oplogWindow`.
|
|
- **Size the oplog for the longest tolerable consumer outage**, including a failed deploy discovered
|
|
the next morning.
|
|
- **Checkpoint after processing, never on receipt** — see
|
|
[change-stream-guide.md](../change-stream-guide.md) §3.
|
|
- **Back up the checkpoint store.** `RESUME_TOKEN_LOSS` is the same incident reached from the other
|
|
direction: the oplog is fine, the checkpoint is gone.
|
|
- **Make the projection rebuildable.** A projection that can only be built by replaying every event
|
|
has no recovery path once the oplog rolls.
|
|
|
|
## Escalation
|
|
|
|
- P1 on detection. The consumer is stopped, so lag grows for as long as this is unresolved.
|
|
- Page the service owner for the rebuild decision, and the database owner if the oplog window shrank
|
|
unexpectedly.
|
|
|
|
## Verification
|
|
|
|
```bash
|
|
cd src
|
|
./gradlew :adapter:outbound:persistence-mongo:mongoFailoverTest --console=plain
|
|
```
|
|
|
|
`MongoFailoverScenario.OPLOG_HISTORY_LOSS` and `RESUME_TOKEN_LOSS` assert the runner halts and names
|
|
this runbook rather than restarting from the current position.
|