DongHyeonkaandClaude Opus 5
d57d2f62a0
Implements the mongodb-superpowers-package design: Stable Tasks 1-50 and Advanced Tasks 1-15. The design assumes 19 Stable + 12 Advanced Gradle projects under modules/mongodb*. This repository's fail-closed registry declares exactly 19 leaf identities, so those modules become package boundaries inside the registered leaf :adapter:outbound:persistence-mongo, with the design's module dependency table enforced by ten ArchUnit rules. The mapping and every deviation are recorded in docs/mongodb/repository-adaptation.md. Contract highlights, all enforced by tests rather than convention: - Transaction body retry and commit retry are separate loops. A new session per body attempt; commit-only retry on an unknown commit. The body is never replayed after a commit ambiguity, so a failover cannot become a duplicate. - MongoExecutionOutcome keeps both ambiguous outcomes distinct from success and failure, and MongoFailureContext records only the design-permitted fields. - Failure classification reads server error labels before numeric codes. - BSON representations come from a pinned manifest, never a library default, and a golden type-signature gate fails on any drift. - Index and validator changes go through the manifest and the admin plane; metadata ownership gates every drop. - Every Advanced capability refuses construction unless its flag is enabled. Verified against real servers, not only unit tests. Running the lanes for the first time exposed four defects that a green `check` had hidden: - Four release lanes passed while executing zero tests; the gate now counts executed tests per lane and fails on zero. - The "single replica set" fixture was a standalone, because Testcontainers 2.x needs withReplicaSet(); its test only asserted a connection string. - The three-node fixture was three independent clusters, so no election could occur, and awaitNewPrimary() compared against the post-stop primary. - The migration lease checked modifiedCount, so a same-millisecond refresh read as a lost lease. scripts/verify-mongodb-platform.sh now reports: 9 lanes, 0 skipped, 0 failed, every evidence category produced. scripts/verify-mongodb-advanced.sh reports NOT PROMOTABLE: actual-topology evidence (real sharded cluster, real KMS, real target deployment) is unobtainable here, so it is named rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
88 lines
4.1 KiB
Markdown
88 lines
4.1 KiB
Markdown
---
|
|
title: Runbook — MongoDB unknown transaction commit result
|
|
category: mongodb
|
|
severity: P1
|
|
owner: oncall
|
|
last_updated: 2026-08-13
|
|
status: active
|
|
---
|
|
|
|
# Runbook: unknown transaction commit result
|
|
|
|
Design §16, decision D-10, scenario `UNKNOWN_TRANSACTION_COMMIT_RESULT`.
|
|
|
|
`MongoExecutionOutcome.TRANSACTION_COMMIT_UNKNOWN` means the commit **may have applied**. It is not a
|
|
failure and must never be reported to a caller as one. The single worst response is to re-run the
|
|
transaction body: if the commit did apply, the body applies a second time.
|
|
|
|
## Symptoms
|
|
|
|
- `MongoTransactionCommitUnknownException` in logs.
|
|
- Metric `failureCategory=TRANSACTION_COMMIT_UNKNOWN`.
|
|
- Usually accompanies a primary election — see [failover.md](failover.md).
|
|
- Downstream reports of duplicated effects (double charge, double increment) are the symptom of this
|
|
being handled wrongly, not of the condition itself.
|
|
|
|
## Diagnosis
|
|
|
|
1. **Confirm the platform did the right thing automatically.**
|
|
`MongoTransactionRetryCoordinator` retries the *commit only*, on the same session, within
|
|
`MongoRetryBudget`. A commit retry against an already-committed transaction is a no-op by design.
|
|
Most occurrences resolve here and never reach a human.
|
|
|
|
2. **If the budget was exhausted, determine the actual state.** The commit either applied or it did
|
|
not; you must find out which, not guess.
|
|
- If the transaction body wrote a deterministic marker (an idempotency key, a business id, a
|
|
revision), read it back. That is exactly what `MongoCommitReconciler` does, and it is the
|
|
reason the design requires transactions to write one.
|
|
- If there is no marker: reconstruct from a downstream artefact — an outbox row, an audit record,
|
|
an external side effect. If nothing exists to compare against, the transaction was not
|
|
designed to be reconcilable and that is the finding to record.
|
|
|
|
3. **Check whether the body was replayed.** Grep for a second execution with the same operation name
|
|
and correlation id. If the body ran twice, the effects need reversing, and the code path that
|
|
replayed it is a defect: an ambiguous commit is `COMMIT_ONLY` scope
|
|
(`MongoRetryScope.COMMIT_ONLY`), never `BODY`.
|
|
|
|
## Action
|
|
|
|
**Commit applied.** Nothing to do. Record the reconciliation.
|
|
|
|
**Commit did not apply.** Re-run the whole operation from the top — a new session, a new body
|
|
attempt. This is safe precisely because you established the previous attempt left no trace.
|
|
|
|
**Cannot determine.** Do not retry. Escalate. A blind retry here is a coin flip between "no effect"
|
|
and "duplicate effect", and duplicates in a financial or notification path are worse than a delay.
|
|
Freeze the affected entity if the domain supports it, and hand off with: operation name, correlation
|
|
id, document id, the time window, and what you checked.
|
|
|
|
**Recurring.** More than one an hour means the commit path is racing something structural — a
|
|
`maxCommitTime` shorter than the observed election duration, an oversized transaction, or an
|
|
undersized retry budget. Fix the budget or the transaction shape; do not raise the retry count and
|
|
call it resolved.
|
|
|
|
## Prevention
|
|
|
|
- Every transaction body writes a deterministic marker that identifies its own commit.
|
|
- `maxCommitTime` exceeds the observed p99 election duration.
|
|
- Callers surface the ambiguity to their own callers rather than mapping it to a generic 500 — an
|
|
ambiguous outcome reported as a failure invites the caller to retry, which is the one thing that
|
|
must not happen.
|
|
- Prefer a single-document atomic operation (D-09). A transaction that exists only to wrap one
|
|
document write has invented this failure mode for nothing.
|
|
|
|
## Escalation
|
|
|
|
- Always P1 when the state cannot be determined and the operation has an external effect.
|
|
- Page the service owner immediately; the database owner only if elections are the trigger.
|
|
|
|
## Verification
|
|
|
|
```bash
|
|
cd src
|
|
./gradlew :adapter:outbound:persistence-mongo:mongoFailoverTest --console=plain
|
|
```
|
|
|
|
`MongoFailoverScenario.UNKNOWN_TRANSACTION_COMMIT_RESULT` runs this path against a real three-node
|
|
set, and the coordinator test asserts the body is never replayed after a commit ambiguity.
|