DongHyeonkaandClaude Opus 5
d57d2f62a0
Implements the mongodb-superpowers-package design: Stable Tasks 1-50 and Advanced Tasks 1-15. The design assumes 19 Stable + 12 Advanced Gradle projects under modules/mongodb*. This repository's fail-closed registry declares exactly 19 leaf identities, so those modules become package boundaries inside the registered leaf :adapter:outbound:persistence-mongo, with the design's module dependency table enforced by ten ArchUnit rules. The mapping and every deviation are recorded in docs/mongodb/repository-adaptation.md. Contract highlights, all enforced by tests rather than convention: - Transaction body retry and commit retry are separate loops. A new session per body attempt; commit-only retry on an unknown commit. The body is never replayed after a commit ambiguity, so a failover cannot become a duplicate. - MongoExecutionOutcome keeps both ambiguous outcomes distinct from success and failure, and MongoFailureContext records only the design-permitted fields. - Failure classification reads server error labels before numeric codes. - BSON representations come from a pinned manifest, never a library default, and a golden type-signature gate fails on any drift. - Index and validator changes go through the manifest and the admin plane; metadata ownership gates every drop. - Every Advanced capability refuses construction unless its flag is enabled. Verified against real servers, not only unit tests. Running the lanes for the first time exposed four defects that a green `check` had hidden: - Four release lanes passed while executing zero tests; the gate now counts executed tests per lane and fails on zero. - The "single replica set" fixture was a standalone, because Testcontainers 2.x needs withReplicaSet(); its test only asserted a connection string. - The three-node fixture was three independent clusters, so no election could occur, and awaitNewPrimary() compared against the post-stop primary. - The migration lease checked modifiedCount, so a same-millisecond refresh read as a lost lease. scripts/verify-mongodb-platform.sh now reports: 9 lanes, 0 skipped, 0 failed, every evidence category produced. scripts/verify-mongodb-advanced.sh reports NOT PROMOTABLE: actual-topology evidence (real sharded cluster, real KMS, real target deployment) is unobtainable here, so it is named rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
138 lines
7.0 KiB
Markdown
138 lines
7.0 KiB
Markdown
# Schema, Index and Migration Guide
|
|
|
|
Design §21–§25, decision D-13. Indexes and validators are **declared** in a manifest and **applied**
|
|
by an explicit plane. Automatic index creation in production is explicitly unsupported (§3.4): an
|
|
index build on a large collection is a capacity event, and discovering it because a deployment
|
|
started one is not an operating model.
|
|
|
|
## 1. The manifest is the source of truth
|
|
|
|
`MongoManifestRegistry` holds one `MongoCollectionManifest` per collection, containing:
|
|
|
|
- `MongoIndexManifest` — the declared indexes (`MongoIndexKey`, `MongoIndexDirection`, uniqueness,
|
|
partial filter, collation)
|
|
- `MongoSchemaManifest` — the declared `$jsonSchema` validator
|
|
- `MongoMetadataOwnership` — who owns each observed object
|
|
|
|
Ownership is the field that makes drift handling safe:
|
|
|
|
| Ownership | Owner | Droppable on drift |
|
|
|---|---|---|
|
|
| `APPLICATION_MANAGED` | this manifest | yes |
|
|
| `SEARCH_MANAGED` | the search service | no |
|
|
| `ENCRYPTION_MANAGED` | Queryable Encryption | no |
|
|
| `EXTERNAL` | someone else (a DBA, another service) | no |
|
|
|
|
A diff engine that does not know about ownership eventually proposes dropping
|
|
`enxcol_.customers.esc` or a search index, and "the drift tool cleaned it up" is a very bad incident
|
|
summary.
|
|
|
|
## 2. Index diff and apply
|
|
|
|
`MongoIndexDiffEngine` compares the manifest against `MongoIndexDescriptorView` observations and
|
|
produces a `MongoIndexDiff`: missing, extra, and *changed* (same name, different definition —
|
|
MongoDB will not silently rebuild these, so they must be reported rather than re-issued).
|
|
|
|
`MongoIndexApplyPolicy` decides what happens with a diff:
|
|
|
|
| Policy | Behaviour | Environment |
|
|
|---|---|---|
|
|
| `APPLY` | create what is missing | local / test |
|
|
| `APPLY_WITH_DIFF` | create what is missing and report the rest | staging |
|
|
| `DIFF_WITH_APPROVED_APPLY` | apply only what a human approved | production |
|
|
| `REPORT_ONLY` | never write | audit |
|
|
|
|
Dropping is never implicit. `MongoIndexRetirementPlan` moves an index through
|
|
`MongoIndexRetirementState` — declared → hidden → observed-unused → droppable — and each transition
|
|
is a separate deployment. Hiding an index makes the planner ignore it while keeping it maintained, so
|
|
an unexpected regression is one command to undo. Dropping it is not.
|
|
|
|
## 3. Validators
|
|
|
|
`MongoValidatorDescriptor` carries the `$jsonSchema`, a `MongoValidationLevel`
|
|
(`OFF` / `MODERATE` / `STRICT`) and a `MongoValidationAction`.
|
|
|
|
**Stable validation actions are `error` and `warn` only.** `errorAndLog` is not part of the Stable
|
|
contract on MongoDB 7.0 or 8.0 and the descriptor refuses it.
|
|
|
|
`MongoValidatorDiffEngine` produces a `MongoValidatorDiff`; `MongoValidatorApplyPolicy` gates the
|
|
apply. Tightening a validator on a collection with existing data is the dangerous direction: introduce
|
|
it as `warn` + `MODERATE`, confirm the warning count is zero, then promote to `error` + `STRICT` in a
|
|
second deployment.
|
|
|
|
## 4. TTL
|
|
|
|
D-13: TTL is **physical cleanup**, nothing else.
|
|
|
|
`MongoTtlIndexDescriptor` declares the field and `expireAfterSeconds`. `MongoTtlPolicyValidator`
|
|
enforces what `MongoTtlPolicy` allows, and `MongoExpirationAccessPolicy` states the rule that matters:
|
|
|
|
> A document's presence is not authorization, and its absence is not a deadline.
|
|
|
|
The TTL monitor runs about once a minute and deletes in batches, so a document can outlive its
|
|
expiry by minutes to hours under load. Consequences:
|
|
|
|
- Access control must check the expiry field, not the document's existence. A still-present expired
|
|
session is a valid document and an invalid session.
|
|
- Business scheduling must not be built on TTL. If something must happen at a time, schedule it.
|
|
- A TTL field must be a BSON date. A TTL index on a string silently never deletes anything.
|
|
|
|
## 5. Migrations
|
|
|
|
`MongoMigrationRunner` executes `MongoMigration` units with:
|
|
|
|
- `MongoMigrationId` — ordered, unique
|
|
- `MongoMigrationChecksum` — content hash; a changed checksum for an applied id is a hard failure, not
|
|
a re-run. Editing an applied migration means two environments ran different code under the same id.
|
|
- `MongoMigrationLedger` — what has been applied
|
|
- `MongoMigrationLock` — one runner at a time; a rolling deploy starts several instances at once
|
|
- `MongoMigrationPrecondition` / `MongoMigrationPostcondition` — checked before and after; a migration
|
|
that cannot verify its own result is a migration whose failure is discovered by a customer
|
|
- `MongoMigrationCheckpoint` — a resumable position for a backfill
|
|
|
|
`MongoMigrationResult` reports applied / incomplete / dry-run with the reason. `INCOMPLETE` is not a
|
|
failure: a rate-limited backfill that ran out of its time budget has done real work and stored a
|
|
checkpoint, and reporting it as failed would send the next run back to the beginning.
|
|
|
|
`MongoCollectionMigrationLedger` and `MongoCollectionMigrationLock` are the MongoDB-backed
|
|
implementations. Two details are load-bearing and only exist on a server:
|
|
|
|
- The ledger's **unique index** on the migration id, created by `ensureIndexes()`. Without it, two
|
|
runners that both pass the "not applied yet" read both insert, and the ledger then reports one
|
|
migration applied twice with two checksums — indistinguishable from tampering.
|
|
- The lease is taken with **one conditional update**, not read-then-write. A filter matching only a
|
|
free or expired lease lets the server pick the winner; two runners that each read "free" and then
|
|
write would both believe they hold it.
|
|
|
|
The lease expires so a runner killed mid-migration does not block every future deployment, and
|
|
`refresh` between batches is what proves the holder is still alive.
|
|
|
|
### Backfills restart, they do not restart-from-zero
|
|
|
|
A long backfill will be interrupted — a deploy, an OOM, a node replacement. The checkpoint records
|
|
the last completed key so the restart continues rather than re-processing from the beginning.
|
|
`MongoBackfillRestartFixture` in the testkit asserts exactly this: kill mid-run, restart, and the
|
|
result is identical to the uninterrupted run and does not re-apply completed work.
|
|
|
|
### Flamingock
|
|
|
|
`FlamingockMongoMigrationAdapter` bridges to Flamingock through the platform-owned
|
|
`FlamingockChangeUnitView`, with `FlamingockLedgerAdapter` and `FlamingockLockAdapter` mapping the
|
|
ledger and lock. The public contract does not reference Flamingock types, so the provider can be
|
|
replaced without touching a migration.
|
|
|
|
## 6. Ordering with deployments
|
|
|
|
```
|
|
1. Add the index (hidden if it is large) -> deployment N
|
|
2. Unhide / verify usage -> deployment N+1
|
|
3. Ship code that depends on the index -> deployment N+1
|
|
4. Backfill data -> migration, resumable
|
|
5. Tighten the validator from warn to error -> deployment N+2
|
|
6. Retire the old index through the retirement states -> deployments N+3…
|
|
```
|
|
|
|
Each step is independently reversible. A deployment that adds an index and the code that requires it
|
|
at the same time has no safe rollback: rolling back the code leaves the index build running, and
|
|
rolling back the index breaks the code that is still live on half the fleet.
|