---
title: Runbook — MongoDB primary failover
category: mongodb
severity: P2
owner: oncall
last_updated: 2026-08-13
status: active
---

# Runbook: MongoDB primary failover

Design §29, scenarios `PRIMARY_KILL`, `NETWORK_PARTITION`, `SERVER_SELECTION_TIMEOUT`,
`WRITE_RESPONSE_LOSS`.

## Symptoms

- `MongoServerSelectionException` / `MongoConnectionException` spike, then recovery within seconds.
- `MongoSdamObservationListener` reports a topology change (primary removed, new primary elected).
- `MongoPoolObservationListener` shows checkout wait times rising while server-side command duration
  stays flat — the wait is topology, not query cost.
- Latency spike on writes with no corresponding rise in read latency.

A failover that resolves in under ~15 s and produces no `WRITE_RESULT_UNKNOWN` is normal replica-set
behaviour and needs no action beyond confirming it self-healed.

## Diagnosis

1. Confirm an election actually happened. SDAM events distinguish an election from "the database got
   slow"; without them the two are indistinguishable in application metrics.
2. Split the failure categories. Metric tag `failureCategory`:
   - `SERVER_SELECTION` / `CONNECTION` → the driver could not reach a primary. `NOT_SENT`; safe.
   - `TIMEOUT` with outcome `WRITE_RESULT_UNKNOWN` → a write may have applied. Not safe; see below.
   - `TRANSACTION_COMMIT_UNKNOWN` → go to [unknown-commit.md](unknown-commit.md) instead.
3. Check the election duration against `MongoRetryBudget`. If the election outlasted the budget, the
   retries were exhausted before a primary existed and callers saw errors that a longer budget would
   have absorbed.
4. Check whether the new primary is in the expected region/AZ. A failover to a distant node changes
   write latency permanently, not transiently.

## Action

**Self-healed (the common case).**
Confirm outcome distribution contains no `WRITE_RESULT_UNKNOWN`, record the election in the incident
log, and close. Nothing to replay.

**Writes with `WRITE_RESULT_UNKNOWN`.**
These writes may or may not have applied. Do not blind-retry.
- Idempotent operation (registered `MongoUpdateOperator` with an `AtomicFilter` precondition): retry.
  The precondition makes the second application a no-op.
- Non-idempotent operation: reconcile by reading the target document and comparing against the
  intended post-state. Retry only if it does not reflect the write.

**Server selection never recovers.**
The set has lost quorum — two of three nodes are down or partitioned. No client-side action fixes
this; escalate to the database owner to restore a majority. The application should be failing closed,
not queueing.

**Elections are frequent (more than one a day, unprompted).**
This is an infrastructure symptom, not an application one: check node resource saturation, disk
latency on the primary, and network stability between members. Repeated elections cause repeated
unknown-outcome windows.

## Escalation

- P2 → P1 if server selection has failed for more than 2 minutes, or if any non-idempotent write
  returned `WRITE_RESULT_UNKNOWN` and cannot be reconciled.
- Page the database owner for quorum loss, and the service owner for reconciliation of ambiguous
  writes.

## Verification

The failover lane reproduces this deliberately:

```bash
cd src
./gradlew :adapter:outbound:persistence-mongo:mongoFailoverTest --console=plain
```

It starts a real three-node set (`MongoThreeNodeReplicaSet`), stops the primary
(`MongoPrimaryController`), and injects network faults through Toxiproxy
(`ToxiproxyMongoNetworkFaultController`). A single-node set is not sufficient for the election: it
never holds one, so every guarantee that depends on a primary change goes untested.

The network faults need their own fixture (`MongoProxiedReplicaSetNode`) because a stopped container
cannot produce them. Stopping a node tells the client the write did not happen; cutting the *path*
while the server keeps running produces a client that cannot tell. `MongoNetworkFaultLaneTest`
asserts the difference by reaching the same server twice — once through the proxy, once directly:

- **Partition**: the proxied client fails, the direct client finds the server healthy and the earlier
  write intact. The path was cut, not the server.
- **Response loss**: the proxied client fails, and the direct client then finds the document
  *present*. The write applied and only the acknowledgement was lost —
  `DefaultMongoFailureClassifier` returns `WRITE_RESULT_UNKNOWN`, and a retry would have inserted a
  second document.

One detail the lane depends on: the connection is warmed before the toxic is applied. On a cold
connection it is the driver's handshake whose response is dropped, so the write is never transmitted
— `NOT_SENT`, the opposite of the ambiguity being tested.