DongHyeonkaandClaude Opus 5
d57d2f62a0
Implements the mongodb-superpowers-package design: Stable Tasks 1-50 and Advanced Tasks 1-15. The design assumes 19 Stable + 12 Advanced Gradle projects under modules/mongodb*. This repository's fail-closed registry declares exactly 19 leaf identities, so those modules become package boundaries inside the registered leaf :adapter:outbound:persistence-mongo, with the design's module dependency table enforced by ten ArchUnit rules. The mapping and every deviation are recorded in docs/mongodb/repository-adaptation.md. Contract highlights, all enforced by tests rather than convention: - Transaction body retry and commit retry are separate loops. A new session per body attempt; commit-only retry on an unknown commit. The body is never replayed after a commit ambiguity, so a failover cannot become a duplicate. - MongoExecutionOutcome keeps both ambiguous outcomes distinct from success and failure, and MongoFailureContext records only the design-permitted fields. - Failure classification reads server error labels before numeric codes. - BSON representations come from a pinned manifest, never a library default, and a golden type-signature gate fails on any drift. - Index and validator changes go through the manifest and the admin plane; metadata ownership gates every drop. - Every Advanced capability refuses construction unless its flag is enabled. Verified against real servers, not only unit tests. Running the lanes for the first time exposed four defects that a green `check` had hidden: - Four release lanes passed while executing zero tests; the gate now counts executed tests per lane and fails on zero. - The "single replica set" fixture was a standalone, because Testcontainers 2.x needs withReplicaSet(); its test only asserted a connection string. - The three-node fixture was three independent clusters, so no election could occur, and awaitNewPrimary() compared against the post-stop primary. - The migration lease checked modifiedCount, so a same-millisecond refresh read as a lost lease. scripts/verify-mongodb-platform.sh now reports: 9 lanes, 0 skipped, 0 failed, every evidence category produced. scripts/verify-mongodb-advanced.sh reports NOT PROMOTABLE: actual-topology evidence (real sharded cluster, real KMS, real target deployment) is unobtainable here, so it is named rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4.1 KiB
title, category, severity, owner, last_updated, status
| title | category | severity | owner | last_updated | status |
|---|---|---|---|---|---|
| Runbook — MongoDB change stream history lost | mongodb | P1 | oncall | 2026-08-13 | active |
Runbook: change stream history lost
Design §20.3, scenarios OPLOG_HISTORY_LOSS, RESUME_TOKEN_LOSS.
The stored resume token predates the oldest entry in the oplog. The events between the checkpoint and
now are gone from the server; no retry recovers them. MongoChangeStreamRecoveryPolicy returns
halt(HISTORY_LOST, "history-lost") and the runner stops.
The platform will not silently restart from "now". That looks like a recovery and is actually a permanent, unreported gap in the projection.
Symptoms
MongoChangeHistoryLostException.MongoChangeStreamState.HISTORY_LOST; the consumer is stopped, not looping.- Precedes it: consumer lag approaching the oplog window, or a consumer that was down for a long period (a deploy that failed, a scaled-to-zero worker, a long outage).
Diagnosis
- Determine the gap. The checkpoint's cluster time is the start; the oldest oplog entry is the end of what is unrecoverable. Everything in between was never processed.
- Determine the oplog window.
rs.printReplicationInfo()on the primary gives the first and last oplog timestamps. If the window is materially smaller than it was, the write rate rose or the oplog was resized — the consumer may be fine and the server changed. - Determine what the projection is missing. Which collections and which operations does this projector consume? The gap is bounded by that, not by everything that happened.
- Check for a second consumer. If another projector on the same collection is healthy, its checkpoint tells you whether the problem is this consumer or the oplog.
Action
Resuming is not an option. The choices are:
Rebuild from source. If the projection is derivable from the current state of the source collections, rebuild it: stop the consumer, rebuild the projection, then start the stream from the cluster time at which the rebuild snapshot was taken. This is the correct answer whenever the projection is a materialised view rather than an event log, and it is the reason a projection should be derivable.
Backfill the gap. If the source documents carry a timestamp covering the gap, run a bounded backfill for that window through the migration runner (checkpointed, resumable — see schema-index-migration-guide.md §5), then resume from the current cluster time.
Accept the gap explicitly. Only when the projection is advisory and the business owner says so. Record the window in the incident log and reset the checkpoint. This is a decision someone signs, not a default.
Never: reset the checkpoint to "now" and restart quietly. That converts a visible P1 into an invisible data-quality defect that surfaces months later as "the report has been wrong since March".
Prevention
- Alert on lag against the oplog window, not wall-clock. "Consumer is 30 minutes behind" is fine
with a 24-hour oplog and an emergency with a 45-minute one. The threshold that matters is
lag / oplogWindow. - Size the oplog for the longest tolerable consumer outage, including a failed deploy discovered the next morning.
- Checkpoint after processing, never on receipt — see change-stream-guide.md §3.
- Back up the checkpoint store.
RESUME_TOKEN_LOSSis the same incident reached from the other direction: the oplog is fine, the checkpoint is gone. - Make the projection rebuildable. A projection that can only be built by replaying every event has no recovery path once the oplog rolls.
Escalation
- P1 on detection. The consumer is stopped, so lag grows for as long as this is unresolved.
- Page the service owner for the rebuild decision, and the database owner if the oplog window shrank unexpectedly.
Verification
cd src
./gradlew :adapter:outbound:persistence-mongo:mongoFailoverTest --console=plain
MongoFailoverScenario.OPLOG_HISTORY_LOSS and RESUME_TOKEN_LOSS assert the runner halts and names
this runbook rather than restarting from the current position.