clean-architecture-backend-template/docs/mongodb/runbooks/failover.md

title, category, severity, owner, last_updated, status

title	category	severity	owner	last_updated	status
Runbook — MongoDB primary failover	mongodb	P2	oncall	2026-08-13	active

Runbook: MongoDB primary failover

Design §29, scenarios PRIMARY_KILL, NETWORK_PARTITION, SERVER_SELECTION_TIMEOUT, WRITE_RESPONSE_LOSS.

Symptoms

• MongoServerSelectionException / MongoConnectionException spike, then recovery within seconds.
• MongoSdamObservationListener reports a topology change (primary removed, new primary elected).
• MongoPoolObservationListener shows checkout wait times rising while server-side command duration stays flat — the wait is topology, not query cost.
• Latency spike on writes with no corresponding rise in read latency.

A failover that resolves in under ~15 s and produces no WRITE_RESULT_UNKNOWN is normal replica-set behaviour and needs no action beyond confirming it self-healed.

Diagnosis

1. Confirm an election actually happened. SDAM events distinguish an election from "the database got slow"; without them the two are indistinguishable in application metrics.
2. Split the failure categories. Metric tag failureCategory:
   • SERVER_SELECTION / CONNECTION → the driver could not reach a primary. NOT_SENT; safe.
   • TIMEOUT with outcome WRITE_RESULT_UNKNOWN → a write may have applied. Not safe; see below.
   • TRANSACTION_COMMIT_UNKNOWN → go to unknown-commit.md instead.
3. Check the election duration against MongoRetryBudget. If the election outlasted the budget, the retries were exhausted before a primary existed and callers saw errors that a longer budget would have absorbed.
4. Check whether the new primary is in the expected region/AZ. A failover to a distant node changes write latency permanently, not transiently.

Action

Self-healed (the common case). Confirm outcome distribution contains no WRITE_RESULT_UNKNOWN, record the election in the incident log, and close. Nothing to replay.

Writes with WRITE_RESULT_UNKNOWN. These writes may or may not have applied. Do not blind-retry.

• Idempotent operation (registered MongoUpdateOperator with an AtomicFilter precondition): retry. The precondition makes the second application a no-op.
• Non-idempotent operation: reconcile by reading the target document and comparing against the intended post-state. Retry only if it does not reflect the write.

Server selection never recovers. The set has lost quorum — two of three nodes are down or partitioned. No client-side action fixes this; escalate to the database owner to restore a majority. The application should be failing closed, not queueing.

Elections are frequent (more than one a day, unprompted). This is an infrastructure symptom, not an application one: check node resource saturation, disk latency on the primary, and network stability between members. Repeated elections cause repeated unknown-outcome windows.

Escalation

• P2 → P1 if server selection has failed for more than 2 minutes, or if any non-idempotent write returned WRITE_RESULT_UNKNOWN and cannot be reconciled.
• Page the database owner for quorum loss, and the service owner for reconciliation of ambiguous writes.

Verification

The failover lane reproduces this deliberately:

cd src
./gradlew :adapter:outbound:persistence-mongo:mongoFailoverTest --console=plain

It starts a real three-node set (MongoThreeNodeReplicaSet), stops the primary (MongoPrimaryController), and injects network faults through Toxiproxy (ToxiproxyMongoNetworkFaultController). A single-node set is not sufficient for the election: it never holds one, so every guarantee that depends on a primary change goes untested.

The network faults need their own fixture (MongoProxiedReplicaSetNode) because a stopped container cannot produce them. Stopping a node tells the client the write did not happen; cutting the path while the server keeps running produces a client that cannot tell. MongoNetworkFaultLaneTest asserts the difference by reaching the same server twice — once through the proxy, once directly:

• Partition: the proxied client fails, the direct client finds the server healthy and the earlier write intact. The path was cut, not the server.
• Response loss: the proxied client fails, and the direct client then finds the document present. The write applied and only the acknowledgement was lost — DefaultMongoFailureClassifier returns WRITE_RESULT_UNKNOWN, and a retry would have inserted a second document.

One detail the lane depends on: the connection is warmed before the toxic is applied. On a cold connection it is the driver's handshake whose response is dropped, so the write is never transmitted — NOT_SENT, the opposite of the ambiguity being tested.