feat(mongodb): implement the MongoDB document persistence platform
Implements the mongodb-superpowers-package design: Stable Tasks 1-50 and Advanced Tasks 1-15. The design assumes 19 Stable + 12 Advanced Gradle projects under modules/mongodb*. This repository's fail-closed registry declares exactly 19 leaf identities, so those modules become package boundaries inside the registered leaf :adapter:outbound:persistence-mongo, with the design's module dependency table enforced by ten ArchUnit rules. The mapping and every deviation are recorded in docs/mongodb/repository-adaptation.md. Contract highlights, all enforced by tests rather than convention: - Transaction body retry and commit retry are separate loops. A new session per body attempt; commit-only retry on an unknown commit. The body is never replayed after a commit ambiguity, so a failover cannot become a duplicate. - MongoExecutionOutcome keeps both ambiguous outcomes distinct from success and failure, and MongoFailureContext records only the design-permitted fields. - Failure classification reads server error labels before numeric codes. - BSON representations come from a pinned manifest, never a library default, and a golden type-signature gate fails on any drift. - Index and validator changes go through the manifest and the admin plane; metadata ownership gates every drop. - Every Advanced capability refuses construction unless its flag is enabled. Verified against real servers, not only unit tests. Running the lanes for the first time exposed four defects that a green `check` had hidden: - Four release lanes passed while executing zero tests; the gate now counts executed tests per lane and fails on zero. - The "single replica set" fixture was a standalone, because Testcontainers 2.x needs withReplicaSet(); its test only asserted a connection string. - The three-node fixture was three independent clusters, so no election could occur, and awaitNewPrimary() compared against the post-stop primary. - The migration lease checked modifiedCount, so a same-millisecond refresh read as a lost lease. scripts/verify-mongodb-platform.sh now reports: 9 lanes, 0 skipped, 0 failed, every evidence category produced. scripts/verify-mongodb-advanced.sh reports NOT PROMOTABLE: actual-topology evidence (real sharded cluster, real KMS, real target deployment) is unobtainable here, so it is named rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
3b5aee50e3
commit
d57d2f62a0
@@ -0,0 +1,137 @@
|
||||
# Schema, Index and Migration Guide
|
||||
|
||||
Design §21–§25, decision D-13. Indexes and validators are **declared** in a manifest and **applied**
|
||||
by an explicit plane. Automatic index creation in production is explicitly unsupported (§3.4): an
|
||||
index build on a large collection is a capacity event, and discovering it because a deployment
|
||||
started one is not an operating model.
|
||||
|
||||
## 1. The manifest is the source of truth
|
||||
|
||||
`MongoManifestRegistry` holds one `MongoCollectionManifest` per collection, containing:
|
||||
|
||||
- `MongoIndexManifest` — the declared indexes (`MongoIndexKey`, `MongoIndexDirection`, uniqueness,
|
||||
partial filter, collation)
|
||||
- `MongoSchemaManifest` — the declared `$jsonSchema` validator
|
||||
- `MongoMetadataOwnership` — who owns each observed object
|
||||
|
||||
Ownership is the field that makes drift handling safe:
|
||||
|
||||
| Ownership | Owner | Droppable on drift |
|
||||
|---|---|---|
|
||||
| `APPLICATION_MANAGED` | this manifest | yes |
|
||||
| `SEARCH_MANAGED` | the search service | no |
|
||||
| `ENCRYPTION_MANAGED` | Queryable Encryption | no |
|
||||
| `EXTERNAL` | someone else (a DBA, another service) | no |
|
||||
|
||||
A diff engine that does not know about ownership eventually proposes dropping
|
||||
`enxcol_.customers.esc` or a search index, and "the drift tool cleaned it up" is a very bad incident
|
||||
summary.
|
||||
|
||||
## 2. Index diff and apply
|
||||
|
||||
`MongoIndexDiffEngine` compares the manifest against `MongoIndexDescriptorView` observations and
|
||||
produces a `MongoIndexDiff`: missing, extra, and *changed* (same name, different definition —
|
||||
MongoDB will not silently rebuild these, so they must be reported rather than re-issued).
|
||||
|
||||
`MongoIndexApplyPolicy` decides what happens with a diff:
|
||||
|
||||
| Policy | Behaviour | Environment |
|
||||
|---|---|---|
|
||||
| `APPLY` | create what is missing | local / test |
|
||||
| `APPLY_WITH_DIFF` | create what is missing and report the rest | staging |
|
||||
| `DIFF_WITH_APPROVED_APPLY` | apply only what a human approved | production |
|
||||
| `REPORT_ONLY` | never write | audit |
|
||||
|
||||
Dropping is never implicit. `MongoIndexRetirementPlan` moves an index through
|
||||
`MongoIndexRetirementState` — declared → hidden → observed-unused → droppable — and each transition
|
||||
is a separate deployment. Hiding an index makes the planner ignore it while keeping it maintained, so
|
||||
an unexpected regression is one command to undo. Dropping it is not.
|
||||
|
||||
## 3. Validators
|
||||
|
||||
`MongoValidatorDescriptor` carries the `$jsonSchema`, a `MongoValidationLevel`
|
||||
(`OFF` / `MODERATE` / `STRICT`) and a `MongoValidationAction`.
|
||||
|
||||
**Stable validation actions are `error` and `warn` only.** `errorAndLog` is not part of the Stable
|
||||
contract on MongoDB 7.0 or 8.0 and the descriptor refuses it.
|
||||
|
||||
`MongoValidatorDiffEngine` produces a `MongoValidatorDiff`; `MongoValidatorApplyPolicy` gates the
|
||||
apply. Tightening a validator on a collection with existing data is the dangerous direction: introduce
|
||||
it as `warn` + `MODERATE`, confirm the warning count is zero, then promote to `error` + `STRICT` in a
|
||||
second deployment.
|
||||
|
||||
## 4. TTL
|
||||
|
||||
D-13: TTL is **physical cleanup**, nothing else.
|
||||
|
||||
`MongoTtlIndexDescriptor` declares the field and `expireAfterSeconds`. `MongoTtlPolicyValidator`
|
||||
enforces what `MongoTtlPolicy` allows, and `MongoExpirationAccessPolicy` states the rule that matters:
|
||||
|
||||
> A document's presence is not authorization, and its absence is not a deadline.
|
||||
|
||||
The TTL monitor runs about once a minute and deletes in batches, so a document can outlive its
|
||||
expiry by minutes to hours under load. Consequences:
|
||||
|
||||
- Access control must check the expiry field, not the document's existence. A still-present expired
|
||||
session is a valid document and an invalid session.
|
||||
- Business scheduling must not be built on TTL. If something must happen at a time, schedule it.
|
||||
- A TTL field must be a BSON date. A TTL index on a string silently never deletes anything.
|
||||
|
||||
## 5. Migrations
|
||||
|
||||
`MongoMigrationRunner` executes `MongoMigration` units with:
|
||||
|
||||
- `MongoMigrationId` — ordered, unique
|
||||
- `MongoMigrationChecksum` — content hash; a changed checksum for an applied id is a hard failure, not
|
||||
a re-run. Editing an applied migration means two environments ran different code under the same id.
|
||||
- `MongoMigrationLedger` — what has been applied
|
||||
- `MongoMigrationLock` — one runner at a time; a rolling deploy starts several instances at once
|
||||
- `MongoMigrationPrecondition` / `MongoMigrationPostcondition` — checked before and after; a migration
|
||||
that cannot verify its own result is a migration whose failure is discovered by a customer
|
||||
- `MongoMigrationCheckpoint` — a resumable position for a backfill
|
||||
|
||||
`MongoMigrationResult` reports applied / incomplete / dry-run with the reason. `INCOMPLETE` is not a
|
||||
failure: a rate-limited backfill that ran out of its time budget has done real work and stored a
|
||||
checkpoint, and reporting it as failed would send the next run back to the beginning.
|
||||
|
||||
`MongoCollectionMigrationLedger` and `MongoCollectionMigrationLock` are the MongoDB-backed
|
||||
implementations. Two details are load-bearing and only exist on a server:
|
||||
|
||||
- The ledger's **unique index** on the migration id, created by `ensureIndexes()`. Without it, two
|
||||
runners that both pass the "not applied yet" read both insert, and the ledger then reports one
|
||||
migration applied twice with two checksums — indistinguishable from tampering.
|
||||
- The lease is taken with **one conditional update**, not read-then-write. A filter matching only a
|
||||
free or expired lease lets the server pick the winner; two runners that each read "free" and then
|
||||
write would both believe they hold it.
|
||||
|
||||
The lease expires so a runner killed mid-migration does not block every future deployment, and
|
||||
`refresh` between batches is what proves the holder is still alive.
|
||||
|
||||
### Backfills restart, they do not restart-from-zero
|
||||
|
||||
A long backfill will be interrupted — a deploy, an OOM, a node replacement. The checkpoint records
|
||||
the last completed key so the restart continues rather than re-processing from the beginning.
|
||||
`MongoBackfillRestartFixture` in the testkit asserts exactly this: kill mid-run, restart, and the
|
||||
result is identical to the uninterrupted run and does not re-apply completed work.
|
||||
|
||||
### Flamingock
|
||||
|
||||
`FlamingockMongoMigrationAdapter` bridges to Flamingock through the platform-owned
|
||||
`FlamingockChangeUnitView`, with `FlamingockLedgerAdapter` and `FlamingockLockAdapter` mapping the
|
||||
ledger and lock. The public contract does not reference Flamingock types, so the provider can be
|
||||
replaced without touching a migration.
|
||||
|
||||
## 6. Ordering with deployments
|
||||
|
||||
```
|
||||
1. Add the index (hidden if it is large) -> deployment N
|
||||
2. Unhide / verify usage -> deployment N+1
|
||||
3. Ship code that depends on the index -> deployment N+1
|
||||
4. Backfill data -> migration, resumable
|
||||
5. Tighten the validator from warn to error -> deployment N+2
|
||||
6. Retire the old index through the retirement states -> deployments N+3…
|
||||
```
|
||||
|
||||
Each step is independently reversible. A deployment that adds an index and the code that requires it
|
||||
at the same time has no safe rollback: rolling back the code leaves the index build running, and
|
||||
rolling back the index breaks the code that is still live on half the fleet.
|
||||
Reference in New Issue
Block a user