clean-architecture-backend-template/docs/mongodb/schema-index-migration-guide.md

Schema, Index and Migration Guide

Design §21–§25, decision D-13. Indexes and validators are declared in a manifest and applied by an explicit plane. Automatic index creation in production is explicitly unsupported (§3.4): an index build on a large collection is a capacity event, and discovering it because a deployment started one is not an operating model.

1. The manifest is the source of truth

MongoManifestRegistry holds one MongoCollectionManifest per collection, containing:

• MongoIndexManifest — the declared indexes (MongoIndexKey, MongoIndexDirection, uniqueness, partial filter, collation)
• MongoSchemaManifest — the declared $jsonSchema validator
• MongoMetadataOwnership — who owns each observed object

Ownership is the field that makes drift handling safe:

Ownership	Owner	Droppable on drift
APPLICATION_MANAGED	this manifest	yes
SEARCH_MANAGED	the search service	no
ENCRYPTION_MANAGED	Queryable Encryption	no
EXTERNAL	someone else (a DBA, another service)	no

A diff engine that does not know about ownership eventually proposes dropping enxcol_.customers.esc or a search index, and "the drift tool cleaned it up" is a very bad incident summary.

2. Index diff and apply

MongoIndexDiffEngine compares the manifest against MongoIndexDescriptorView observations and produces a MongoIndexDiff: missing, extra, and changed (same name, different definition — MongoDB will not silently rebuild these, so they must be reported rather than re-issued).

MongoIndexApplyPolicy decides what happens with a diff:

Policy	Behaviour	Environment
APPLY	create what is missing	local / test
APPLY_WITH_DIFF	create what is missing and report the rest	staging
DIFF_WITH_APPROVED_APPLY	apply only what a human approved	production
REPORT_ONLY	never write	audit

Dropping is never implicit. MongoIndexRetirementPlan moves an index through MongoIndexRetirementState — declared → hidden → observed-unused → droppable — and each transition is a separate deployment. Hiding an index makes the planner ignore it while keeping it maintained, so an unexpected regression is one command to undo. Dropping it is not.

3. Validators

MongoValidatorDescriptor carries the $jsonSchema, a MongoValidationLevel (OFF / MODERATE / STRICT) and a MongoValidationAction.

Stable validation actions are error and warn only. errorAndLog is not part of the Stable contract on MongoDB 7.0 or 8.0 and the descriptor refuses it.

MongoValidatorDiffEngine produces a MongoValidatorDiff; MongoValidatorApplyPolicy gates the apply. Tightening a validator on a collection with existing data is the dangerous direction: introduce it as warn + MODERATE, confirm the warning count is zero, then promote to error + STRICT in a second deployment.

4. TTL

D-13: TTL is physical cleanup, nothing else.

MongoTtlIndexDescriptor declares the field and expireAfterSeconds. MongoTtlPolicyValidator enforces what MongoTtlPolicy allows, and MongoExpirationAccessPolicy states the rule that matters:

A document's presence is not authorization, and its absence is not a deadline.

The TTL monitor runs about once a minute and deletes in batches, so a document can outlive its expiry by minutes to hours under load. Consequences:

• Access control must check the expiry field, not the document's existence. A still-present expired session is a valid document and an invalid session.
• Business scheduling must not be built on TTL. If something must happen at a time, schedule it.
• A TTL field must be a BSON date. A TTL index on a string silently never deletes anything.

5. Migrations

MongoMigrationRunner executes MongoMigration units with:

• MongoMigrationId — ordered, unique
• MongoMigrationChecksum — content hash; a changed checksum for an applied id is a hard failure, not a re-run. Editing an applied migration means two environments ran different code under the same id.
• MongoMigrationLedger — what has been applied
• MongoMigrationLock — one runner at a time; a rolling deploy starts several instances at once
• MongoMigrationPrecondition / MongoMigrationPostcondition — checked before and after; a migration that cannot verify its own result is a migration whose failure is discovered by a customer
• MongoMigrationCheckpoint — a resumable position for a backfill

MongoMigrationResult reports applied / incomplete / dry-run with the reason. INCOMPLETE is not a failure: a rate-limited backfill that ran out of its time budget has done real work and stored a checkpoint, and reporting it as failed would send the next run back to the beginning.

MongoCollectionMigrationLedger and MongoCollectionMigrationLock are the MongoDB-backed implementations. Two details are load-bearing and only exist on a server:

• The ledger's unique index on the migration id, created by ensureIndexes(). Without it, two runners that both pass the "not applied yet" read both insert, and the ledger then reports one migration applied twice with two checksums — indistinguishable from tampering.
• The lease is taken with one conditional update, not read-then-write. A filter matching only a free or expired lease lets the server pick the winner; two runners that each read "free" and then write would both believe they hold it.

The lease expires so a runner killed mid-migration does not block every future deployment, and refresh between batches is what proves the holder is still alive.

Backfills restart, they do not restart-from-zero

A long backfill will be interrupted — a deploy, an OOM, a node replacement. The checkpoint records the last completed key so the restart continues rather than re-processing from the beginning. MongoBackfillRestartFixture in the testkit asserts exactly this: kill mid-run, restart, and the result is identical to the uninterrupted run and does not re-apply completed work.

Flamingock

FlamingockMongoMigrationAdapter bridges to Flamingock through the platform-owned FlamingockChangeUnitView, with FlamingockLedgerAdapter and FlamingockLockAdapter mapping the ledger and lock. The public contract does not reference Flamingock types, so the provider can be replaced without touching a migration.

6. Ordering with deployments

1. Add the index (hidden if it is large)              -> deployment N
2. Unhide / verify usage                              -> deployment N+1
3. Ship code that depends on the index                -> deployment N+1
4. Backfill data                                      -> migration, resumable
5. Tighten the validator from warn to error           -> deployment N+2
6. Retire the old index through the retirement states -> deployments N+3…

Each step is independently reversible. A deployment that adds an index and the code that requires it at the same time has no safe rollback: rolling back the code leaves the index build running, and rolling back the index breaks the code that is still live on half the fleet.