DongHyeonkaandClaude Opus 5
d57d2f62a0
Implements the mongodb-superpowers-package design: Stable Tasks 1-50 and Advanced Tasks 1-15. The design assumes 19 Stable + 12 Advanced Gradle projects under modules/mongodb*. This repository's fail-closed registry declares exactly 19 leaf identities, so those modules become package boundaries inside the registered leaf :adapter:outbound:persistence-mongo, with the design's module dependency table enforced by ten ArchUnit rules. The mapping and every deviation are recorded in docs/mongodb/repository-adaptation.md. Contract highlights, all enforced by tests rather than convention: - Transaction body retry and commit retry are separate loops. A new session per body attempt; commit-only retry on an unknown commit. The body is never replayed after a commit ambiguity, so a failover cannot become a duplicate. - MongoExecutionOutcome keeps both ambiguous outcomes distinct from success and failure, and MongoFailureContext records only the design-permitted fields. - Failure classification reads server error labels before numeric codes. - BSON representations come from a pinned manifest, never a library default, and a golden type-signature gate fails on any drift. - Index and validator changes go through the manifest and the admin plane; metadata ownership gates every drop. - Every Advanced capability refuses construction unless its flag is enabled. Verified against real servers, not only unit tests. Running the lanes for the first time exposed four defects that a green `check` had hidden: - Four release lanes passed while executing zero tests; the gate now counts executed tests per lane and fails on zero. - The "single replica set" fixture was a standalone, because Testcontainers 2.x needs withReplicaSet(); its test only asserted a connection string. - The three-node fixture was three independent clusters, so no election could occur, and awaitNewPrimary() compared against the post-stop primary. - The migration lease checked modifiedCount, so a same-millisecond refresh read as a lost lease. scripts/verify-mongodb-platform.sh now reports: 9 lanes, 0 skipped, 0 failed, every evidence category produced. scripts/verify-mongodb-advanced.sh reports NOT PROMOTABLE: actual-topology evidence (real sharded cluster, real KMS, real target deployment) is unobtainable here, so it is named rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
99 lines
4.7 KiB
Markdown
99 lines
4.7 KiB
Markdown
---
|
|
title: Runbook — MongoDB primary failover
|
|
category: mongodb
|
|
severity: P2
|
|
owner: oncall
|
|
last_updated: 2026-08-13
|
|
status: active
|
|
---
|
|
|
|
# Runbook: MongoDB primary failover
|
|
|
|
Design §29, scenarios `PRIMARY_KILL`, `NETWORK_PARTITION`, `SERVER_SELECTION_TIMEOUT`,
|
|
`WRITE_RESPONSE_LOSS`.
|
|
|
|
## Symptoms
|
|
|
|
- `MongoServerSelectionException` / `MongoConnectionException` spike, then recovery within seconds.
|
|
- `MongoSdamObservationListener` reports a topology change (primary removed, new primary elected).
|
|
- `MongoPoolObservationListener` shows checkout wait times rising while server-side command duration
|
|
stays flat — the wait is topology, not query cost.
|
|
- Latency spike on writes with no corresponding rise in read latency.
|
|
|
|
A failover that resolves in under ~15 s and produces no `WRITE_RESULT_UNKNOWN` is normal replica-set
|
|
behaviour and needs no action beyond confirming it self-healed.
|
|
|
|
## Diagnosis
|
|
|
|
1. Confirm an election actually happened. SDAM events distinguish an election from "the database got
|
|
slow"; without them the two are indistinguishable in application metrics.
|
|
2. Split the failure categories. Metric tag `failureCategory`:
|
|
- `SERVER_SELECTION` / `CONNECTION` → the driver could not reach a primary. `NOT_SENT`; safe.
|
|
- `TIMEOUT` with outcome `WRITE_RESULT_UNKNOWN` → a write may have applied. Not safe; see below.
|
|
- `TRANSACTION_COMMIT_UNKNOWN` → go to [unknown-commit.md](unknown-commit.md) instead.
|
|
3. Check the election duration against `MongoRetryBudget`. If the election outlasted the budget, the
|
|
retries were exhausted before a primary existed and callers saw errors that a longer budget would
|
|
have absorbed.
|
|
4. Check whether the new primary is in the expected region/AZ. A failover to a distant node changes
|
|
write latency permanently, not transiently.
|
|
|
|
## Action
|
|
|
|
**Self-healed (the common case).**
|
|
Confirm outcome distribution contains no `WRITE_RESULT_UNKNOWN`, record the election in the incident
|
|
log, and close. Nothing to replay.
|
|
|
|
**Writes with `WRITE_RESULT_UNKNOWN`.**
|
|
These writes may or may not have applied. Do not blind-retry.
|
|
- Idempotent operation (registered `MongoUpdateOperator` with an `AtomicFilter` precondition): retry.
|
|
The precondition makes the second application a no-op.
|
|
- Non-idempotent operation: reconcile by reading the target document and comparing against the
|
|
intended post-state. Retry only if it does not reflect the write.
|
|
|
|
**Server selection never recovers.**
|
|
The set has lost quorum — two of three nodes are down or partitioned. No client-side action fixes
|
|
this; escalate to the database owner to restore a majority. The application should be failing closed,
|
|
not queueing.
|
|
|
|
**Elections are frequent (more than one a day, unprompted).**
|
|
This is an infrastructure symptom, not an application one: check node resource saturation, disk
|
|
latency on the primary, and network stability between members. Repeated elections cause repeated
|
|
unknown-outcome windows.
|
|
|
|
## Escalation
|
|
|
|
- P2 → P1 if server selection has failed for more than 2 minutes, or if any non-idempotent write
|
|
returned `WRITE_RESULT_UNKNOWN` and cannot be reconciled.
|
|
- Page the database owner for quorum loss, and the service owner for reconciliation of ambiguous
|
|
writes.
|
|
|
|
## Verification
|
|
|
|
The failover lane reproduces this deliberately:
|
|
|
|
```bash
|
|
cd src
|
|
./gradlew :adapter:outbound:persistence-mongo:mongoFailoverTest --console=plain
|
|
```
|
|
|
|
It starts a real three-node set (`MongoThreeNodeReplicaSet`), stops the primary
|
|
(`MongoPrimaryController`), and injects network faults through Toxiproxy
|
|
(`ToxiproxyMongoNetworkFaultController`). A single-node set is not sufficient for the election: it
|
|
never holds one, so every guarantee that depends on a primary change goes untested.
|
|
|
|
The network faults need their own fixture (`MongoProxiedReplicaSetNode`) because a stopped container
|
|
cannot produce them. Stopping a node tells the client the write did not happen; cutting the *path*
|
|
while the server keeps running produces a client that cannot tell. `MongoNetworkFaultLaneTest`
|
|
asserts the difference by reaching the same server twice — once through the proxy, once directly:
|
|
|
|
- **Partition**: the proxied client fails, the direct client finds the server healthy and the earlier
|
|
write intact. The path was cut, not the server.
|
|
- **Response loss**: the proxied client fails, and the direct client then finds the document
|
|
*present*. The write applied and only the acknowledgement was lost —
|
|
`DefaultMongoFailureClassifier` returns `WRITE_RESULT_UNKNOWN`, and a retry would have inserted a
|
|
second document.
|
|
|
|
One detail the lane depends on: the connection is warmed before the toxic is applied. On a cold
|
|
connection it is the driver's handshake whose response is dropped, so the write is never transmitted
|
|
— `NOT_SENT`, the opposite of the ambiguity being tested.
|