# Schema, Index and Migration Guide

Design §21–§25, decision D-13. Indexes and validators are **declared** in a manifest and **applied**
by an explicit plane. Automatic index creation in production is explicitly unsupported (§3.4): an
index build on a large collection is a capacity event, and discovering it because a deployment
started one is not an operating model.

## 1. The manifest is the source of truth

`MongoManifestRegistry` holds one `MongoCollectionManifest` per collection, containing:

- `MongoIndexManifest` — the declared indexes (`MongoIndexKey`, `MongoIndexDirection`, uniqueness,
  partial filter, collation)
- `MongoSchemaManifest` — the declared `$jsonSchema` validator
- `MongoMetadataOwnership` — who owns each observed object

Ownership is the field that makes drift handling safe:

| Ownership | Owner | Droppable on drift |
|---|---|---|
| `APPLICATION_MANAGED` | this manifest | yes |
| `SEARCH_MANAGED` | the search service | no |
| `ENCRYPTION_MANAGED` | Queryable Encryption | no |
| `EXTERNAL` | someone else (a DBA, another service) | no |

A diff engine that does not know about ownership eventually proposes dropping
`enxcol_.customers.esc` or a search index, and "the drift tool cleaned it up" is a very bad incident
summary.

## 2. Index diff and apply

`MongoIndexDiffEngine` compares the manifest against `MongoIndexDescriptorView` observations and
produces a `MongoIndexDiff`: missing, extra, and *changed* (same name, different definition —
MongoDB will not silently rebuild these, so they must be reported rather than re-issued).

`MongoIndexApplyPolicy` decides what happens with a diff:

| Policy | Behaviour | Environment |
|---|---|---|
| `APPLY` | create what is missing | local / test |
| `APPLY_WITH_DIFF` | create what is missing and report the rest | staging |
| `DIFF_WITH_APPROVED_APPLY` | apply only what a human approved | production |
| `REPORT_ONLY` | never write | audit |

Dropping is never implicit. `MongoIndexRetirementPlan` moves an index through
`MongoIndexRetirementState` — declared → hidden → observed-unused → droppable — and each transition
is a separate deployment. Hiding an index makes the planner ignore it while keeping it maintained, so
an unexpected regression is one command to undo. Dropping it is not.

## 3. Validators

`MongoValidatorDescriptor` carries the `$jsonSchema`, a `MongoValidationLevel`
(`OFF` / `MODERATE` / `STRICT`) and a `MongoValidationAction`.

**Stable validation actions are `error` and `warn` only.** `errorAndLog` is not part of the Stable
contract on MongoDB 7.0 or 8.0 and the descriptor refuses it.

`MongoValidatorDiffEngine` produces a `MongoValidatorDiff`; `MongoValidatorApplyPolicy` gates the
apply. Tightening a validator on a collection with existing data is the dangerous direction: introduce
it as `warn` + `MODERATE`, confirm the warning count is zero, then promote to `error` + `STRICT` in a
second deployment.

## 4. TTL

D-13: TTL is **physical cleanup**, nothing else.

`MongoTtlIndexDescriptor` declares the field and `expireAfterSeconds`. `MongoTtlPolicyValidator`
enforces what `MongoTtlPolicy` allows, and `MongoExpirationAccessPolicy` states the rule that matters:

> A document's presence is not authorization, and its absence is not a deadline.

The TTL monitor runs about once a minute and deletes in batches, so a document can outlive its
expiry by minutes to hours under load. Consequences:

- Access control must check the expiry field, not the document's existence. A still-present expired
  session is a valid document and an invalid session.
- Business scheduling must not be built on TTL. If something must happen at a time, schedule it.
- A TTL field must be a BSON date. A TTL index on a string silently never deletes anything.

## 5. Migrations

`MongoMigrationRunner` executes `MongoMigration` units with:

- `MongoMigrationId` — ordered, unique
- `MongoMigrationChecksum` — content hash; a changed checksum for an applied id is a hard failure, not
  a re-run. Editing an applied migration means two environments ran different code under the same id.
- `MongoMigrationLedger` — what has been applied
- `MongoMigrationLock` — one runner at a time; a rolling deploy starts several instances at once
- `MongoMigrationPrecondition` / `MongoMigrationPostcondition` — checked before and after; a migration
  that cannot verify its own result is a migration whose failure is discovered by a customer
- `MongoMigrationCheckpoint` — a resumable position for a backfill

`MongoMigrationResult` reports applied / incomplete / dry-run with the reason. `INCOMPLETE` is not a
failure: a rate-limited backfill that ran out of its time budget has done real work and stored a
checkpoint, and reporting it as failed would send the next run back to the beginning.

`MongoCollectionMigrationLedger` and `MongoCollectionMigrationLock` are the MongoDB-backed
implementations. Two details are load-bearing and only exist on a server:

- The ledger's **unique index** on the migration id, created by `ensureIndexes()`. Without it, two
  runners that both pass the "not applied yet" read both insert, and the ledger then reports one
  migration applied twice with two checksums — indistinguishable from tampering.
- The lease is taken with **one conditional update**, not read-then-write. A filter matching only a
  free or expired lease lets the server pick the winner; two runners that each read "free" and then
  write would both believe they hold it.

The lease expires so a runner killed mid-migration does not block every future deployment, and
`refresh` between batches is what proves the holder is still alive.

### Backfills restart, they do not restart-from-zero

A long backfill will be interrupted — a deploy, an OOM, a node replacement. The checkpoint records
the last completed key so the restart continues rather than re-processing from the beginning.
`MongoBackfillRestartFixture` in the testkit asserts exactly this: kill mid-run, restart, and the
result is identical to the uninterrupted run and does not re-apply completed work.

### Flamingock

`FlamingockMongoMigrationAdapter` bridges to Flamingock through the platform-owned
`FlamingockChangeUnitView`, with `FlamingockLedgerAdapter` and `FlamingockLockAdapter` mapping the
ledger and lock. The public contract does not reference Flamingock types, so the provider can be
replaced without touching a migration.

## 6. Ordering with deployments

```
1. Add the index (hidden if it is large)              -> deployment N
2. Unhide / verify usage                              -> deployment N+1
3. Ship code that depends on the index                -> deployment N+1
4. Backfill data                                      -> migration, resumable
5. Tighten the validator from warn to error           -> deployment N+2
6. Retire the old index through the retirement states -> deployments N+3…
```

Each step is independently reversible. A deployment that adds an index and the code that requires it
at the same time has no safe rollback: rolling back the code leaves the index build running, and
rolling back the index breaks the code that is still live on half the fleet.