feat(mongodb): implement the MongoDB document persistence platform
Implements the mongodb-superpowers-package design: Stable Tasks 1-50 and Advanced Tasks 1-15. The design assumes 19 Stable + 12 Advanced Gradle projects under modules/mongodb*. This repository's fail-closed registry declares exactly 19 leaf identities, so those modules become package boundaries inside the registered leaf :adapter:outbound:persistence-mongo, with the design's module dependency table enforced by ten ArchUnit rules. The mapping and every deviation are recorded in docs/mongodb/repository-adaptation.md. Contract highlights, all enforced by tests rather than convention: - Transaction body retry and commit retry are separate loops. A new session per body attempt; commit-only retry on an unknown commit. The body is never replayed after a commit ambiguity, so a failover cannot become a duplicate. - MongoExecutionOutcome keeps both ambiguous outcomes distinct from success and failure, and MongoFailureContext records only the design-permitted fields. - Failure classification reads server error labels before numeric codes. - BSON representations come from a pinned manifest, never a library default, and a golden type-signature gate fails on any drift. - Index and validator changes go through the manifest and the admin plane; metadata ownership gates every drop. - Every Advanced capability refuses construction unless its flag is enabled. Verified against real servers, not only unit tests. Running the lanes for the first time exposed four defects that a green `check` had hidden: - Four release lanes passed while executing zero tests; the gate now counts executed tests per lane and fails on zero. - The "single replica set" fixture was a standalone, because Testcontainers 2.x needs withReplicaSet(); its test only asserted a connection string. - The three-node fixture was three independent clusters, so no election could occur, and awaitNewPrimary() compared against the post-stop primary. - The migration lease checked modifiedCount, so a same-millisecond refresh read as a lost lease. scripts/verify-mongodb-platform.sh now reports: 9 lanes, 0 skipped, 0 failed, every evidence category produced. scripts/verify-mongodb-advanced.sh reports NOT PROMOTABLE: actual-topology evidence (real sharded cluster, real KMS, real target deployment) is unobtainable here, so it is named rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
DongHyeonka
co-authored by
Claude Opus 5
Claude Opus 5
parent
3b5aee50e3
commit
d57d2f62a0
@@ -0,0 +1,91 @@
|
||||
# Advanced — CSFLE and Queryable Encryption
|
||||
|
||||
**Capabilities:** `MongoCapability.CSFLE`, `MongoCapability.QUERYABLE_ENCRYPTION`
|
||||
**Properties:** `ca-skeleton.persistence-mongo.advanced.csfle.enabled`,
|
||||
`ca-skeleton.persistence-mongo.advanced.queryable-encryption.enabled`
|
||||
**Status:** Advanced.
|
||||
|
||||
## Requirements
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Topology | Replica set or sharded cluster. |
|
||||
| Server | MongoDB 7.0 or 8.0 (see §4 for the 8.0 query-type limits). |
|
||||
| Privilege | `MongoPrincipalRole.ENCRYPTION_ADMIN` for the key vault; the application role never holds it. |
|
||||
| Environment | A real KMS and key vault. A local key provider does not exercise any of the failure modes that matter. |
|
||||
|
||||
## 1. CSFLE
|
||||
|
||||
`MongoCsfleProfile` binds a collection to its `MongoCsfleFieldPolicy` list, a key vault
|
||||
`MongoCredentialReference` and the key vault namespace. `MongoCsfleClientFactory` builds the encrypted
|
||||
client; `MongoDataKeyResolver` resolves data keys.
|
||||
|
||||
`MongoCsfleMode`:
|
||||
|
||||
| Mode | Queryable | Trade-off |
|
||||
|---|---|---|
|
||||
| `RANDOMIZED` | no | Same plaintext encrypts differently each time. The safe default. |
|
||||
| `DETERMINISTIC` | equality only | Same plaintext always yields the same ciphertext, so equality works — and so does frequency analysis. |
|
||||
| `UNINDEXED` | no | Stored encrypted, excluded from any index. |
|
||||
|
||||
`MongoCsfleFieldPolicy.forPii(field, queryable)` defaults to `RANDOMIZED` when the field is not
|
||||
queried. Deterministic encryption requires a written `equalityQueryJustification`; the constructor
|
||||
refuses a blank one, naming frequency analysis. A low-cardinality deterministic field (a status, a
|
||||
country, a boolean) leaks its distribution to anyone who can read the collection, which is the party
|
||||
encryption was protecting against.
|
||||
|
||||
## 2. Queryable Encryption
|
||||
|
||||
`MongoQueryableEncryptionProfile` binds a collection to `MongoEncryptedFieldDescriptor` entries.
|
||||
`MongoQueryableEncryptionQueryType` has exactly two values:
|
||||
|
||||
- `EQUALITY`
|
||||
- `RANGE` — must declare its domain (`min`, `max`). The constructor refuses a range field without one,
|
||||
because changing the domain later means re-encrypting the field.
|
||||
|
||||
`MongoQueryableEncryptionCollectionManager` owns the collection's lifecycle, because a QE collection
|
||||
is not just a collection: it carries metadata collections.
|
||||
|
||||
## 3. Metadata ownership
|
||||
|
||||
`MongoEncryptionMetadataOwnership` maps `customers` to `enxcol_.customers.esc` and
|
||||
`enxcol_.customers.ecoc`, and reports `__safeContent__`-prefixed indexes as
|
||||
`MongoMetadataOwnership.ENCRYPTION_MANAGED`.
|
||||
|
||||
These are never application-owned and never droppable by drift reconciliation. A drift tool that
|
||||
drops `enxcol_.customers.ecoc` corrupts the collection's queryability. This is the single most
|
||||
important integration point between encryption and
|
||||
[ADR-MONGO-004](../../adr/ADR-MONGO-004-index-schema-admin-plane.md).
|
||||
|
||||
## 4. Unsupported combinations
|
||||
|
||||
Refused at declaration, not discovered at runtime:
|
||||
|
||||
| Combination | Why |
|
||||
|---|---|
|
||||
| CSFLE **and** QE on the same collection | Two incompatible encryption schemes over one namespace. Both profile constructors refuse it. |
|
||||
| CSFLE on a time series collection | `requireNotTimeSeries(true)` raises `MongoOperationRejectedException`. |
|
||||
| QE `prefix` / `suffix` / `substring` | Not available on the platform's 8.0 baseline. The factory methods throw `UnsupportedOperationException` rather than returning a profile that fails later. |
|
||||
| Deterministic CSFLE without a justification | `IllegalArgumentException` naming frequency analysis. |
|
||||
| Range QE without a declared domain | `IllegalArgumentException` naming re-encryption. |
|
||||
|
||||
## 5. Failure recovery
|
||||
|
||||
| Symptom | Cause | Action |
|
||||
|---|---|---|
|
||||
| `MongoEncryptionException` on read | Wrong data key, or the key vault is unreachable | Check KMS reachability and the key vault credential. Data is intact; the client cannot decrypt it. |
|
||||
| `MongoEncryptionException` on write | KMS permission revoked mid-operation | Restore the grant. Writes fail closed — nothing was written in plaintext. |
|
||||
| Queries return nothing on a deterministic field | The field was re-keyed | Equality matching is over ciphertext; a new key produces different ciphertext. Re-encrypt the field. |
|
||||
| QE queries fail after a drift reconciliation | A metadata collection was dropped | Restore from backup. This is why ownership gates drops. |
|
||||
|
||||
**Key rotation.** Rotating the customer master key re-wraps the data keys and does not require
|
||||
re-encrypting documents. Rotating a *data* key does require re-encrypting every document that used
|
||||
it. These are different operations with different costs, and confusing them is how a rotation becomes
|
||||
an outage.
|
||||
|
||||
## 6. Promotion evidence
|
||||
|
||||
Per [ADR-MONGO-ADV-001](../../adr/ADR-MONGO-ADV-001-capability-promotion.md), promotion requires the
|
||||
real KMS and key vault, plus negative cases that fail closed: wrong key, missing permission, rotation
|
||||
mid-operation (`MongoAtlasCapabilityContractSuite.kmsFailureModes`). A local key provider certifies
|
||||
none of these — it never rejects anything.
|
||||
@@ -0,0 +1,63 @@
|
||||
# Advanced — GridFS compatibility and migration
|
||||
|
||||
**Capability:** `MongoCapability.GRIDFS_COMPATIBILITY`
|
||||
**Property:** `ca-skeleton.persistence-mongo.advanced.gridfs-compatibility.enabled`
|
||||
**Status:** Advanced, compatibility only. Decision D-14.
|
||||
|
||||
## Position
|
||||
|
||||
GridFS is a **compatibility adapter for files that already exist there**. New files use the existing
|
||||
Fileserver / Object Storage adapter, which is the source of truth for binary content.
|
||||
|
||||
The reason is not preference. GridFS stores file chunks in the same collections, on the same replica
|
||||
set, competing for the same working set as your documents. A large file read evicts document pages
|
||||
from cache, and file storage growth becomes replica-set growth — which means it becomes oplog
|
||||
pressure, backup duration and failover time. Object storage was built for this and MongoDB was not.
|
||||
|
||||
## Reading legacy files
|
||||
|
||||
`MongoGridFsCompatibilityReader` reads existing GridFS content as
|
||||
`GridFsLegacyContent(legacyId, filename, sizeBytes, checksum, stream)`. It reads; it does not write.
|
||||
|
||||
## Migration
|
||||
|
||||
`MongoGridFsMigrationJob` moves a file to object storage in a fixed order:
|
||||
|
||||
```
|
||||
read legacy content
|
||||
→ write to object storage
|
||||
→ verify the target checksum matches the source
|
||||
→ switch the reference
|
||||
→ (later, separately) delete the source
|
||||
```
|
||||
|
||||
Three properties, each of which exists because of a specific way this goes wrong:
|
||||
|
||||
1. **Verify before switching.** `MongoGridFsObjectReference` requires a non-blank checksum, and the
|
||||
job returns empty and writes no reference when the target checksum does not match the source. A
|
||||
migration that switches the reference on a successful *write* rather than a verified *copy*
|
||||
silently points at a truncated object.
|
||||
2. **The source is never deleted here.** Deletion is a separate, later decision after the new
|
||||
location has been serving reads long enough to be trusted. A migration that deletes as it goes has
|
||||
no rollback.
|
||||
3. **The checkpoint separates migrated from failed.** `MongoGridFsMigrationCheckpoint` tracks
|
||||
`migratedCount()`, `failedCount()`, `clean()` and `lastMigratedLegacyId()`, so a restart continues
|
||||
from the last completed file rather than starting over, and a partially failed run is visible as
|
||||
partial rather than as "done".
|
||||
|
||||
## Failure recovery
|
||||
|
||||
| Symptom | Cause | Action |
|
||||
|---|---|---|
|
||||
| `migrate` returns empty | Checksum mismatch | The copy is bad. Investigate before retrying; do not force the reference. |
|
||||
| `IllegalArgumentException` on the reference | Missing checksum | A reference without a checksum cannot be verified and is refused. |
|
||||
| Checkpoint not `clean()` | Some files failed | Re-run for the failed ids only; the checkpoint names the last successful one. |
|
||||
| Reference switched but content missing | Source deleted too early | Restore from backup. This is what rule 2 prevents. |
|
||||
|
||||
## Promotion evidence
|
||||
|
||||
Actual-topology evidence against the real object storage backend, a security review of the storage
|
||||
credential, the migration path above, failure cases (checksum mismatch refused, missing checksum
|
||||
refused, restart resumes), and this document as the runbook.
|
||||
|
||||
New file storage does not go through here at all — see the fileserver adapter.
|
||||
@@ -0,0 +1,90 @@
|
||||
# Advanced — Multi-tenancy
|
||||
|
||||
**Capabilities:** `MongoCapability.SHARED_COLLECTION_TENANCY` (Advanced),
|
||||
`MongoCapability.DATABASE_PER_TENANT` (Experimental)
|
||||
**Properties:** `ca-skeleton.persistence-mongo.advanced.shared-collection-tenancy.enabled`,
|
||||
`ca-skeleton.persistence-mongo.advanced.database-per-tenant.enabled`
|
||||
|
||||
## 1. Shared collection
|
||||
|
||||
Every tenant's documents live in one collection, discriminated by a tenant field.
|
||||
|
||||
`MongoTenantContext` carries the tenant. `MongoTenantPredicateInjector` adds the tenant predicate to
|
||||
every query, every atomic filter and the **first** aggregation stage. `TenantScopedMongoOperations`
|
||||
is the entry point, so a caller cannot construct an unscoped operation by forgetting.
|
||||
|
||||
Three details are load-bearing:
|
||||
|
||||
- **Injection, not convention.** A tenant predicate that each query is expected to add itself is a
|
||||
cross-tenant leak waiting for one missed `where(...)`. The injector adds it structurally.
|
||||
- **First aggregation stage.** `firstStageMatch(...)` places the tenant `$match` before anything else.
|
||||
A `$lookup` or `$group` that runs before the tenant filter has already crossed the boundary, even if
|
||||
a later stage filters the output.
|
||||
- **An absent tenant is not "all tenants".** The injector takes an `Optional<MongoTenantContext>` so
|
||||
the missing case is a decision the policy makes explicitly, not a predicate that quietly disappears.
|
||||
|
||||
`MongoTenantManifestValidator.validate(manifest, tenantScopedUniqueIndexes)` checks that every unique
|
||||
index that should be per-tenant actually includes the tenant field. A unique index on `email` alone in
|
||||
a shared collection makes an email globally unique across tenants — tenant B cannot register an
|
||||
address tenant A already used, which is both a bug and an information leak.
|
||||
|
||||
`requireShardKeyAnalysed(...)` requires a shard-key readiness report before a shared-collection tenant
|
||||
model is sharded: tenant id as a shard key prefix concentrates the largest tenant on one shard.
|
||||
|
||||
### Observability
|
||||
|
||||
`tenantId` and `rawTenantId` are on `MongoObservationConvention`'s forbidden tag list. Cardinality
|
||||
grows with the customer list, and the tag ships tenant identity into the metrics backend.
|
||||
|
||||
## 2. Database per tenant (Experimental)
|
||||
|
||||
`MongoTenantDatabaseResolver` maps a tenant to its database; `MongoTenantClientRegistry` holds the
|
||||
clients.
|
||||
|
||||
Experimental for a specific reason: it is correct in the small and unbounded in the large. Each tenant
|
||||
database costs connections, file handles and monitoring cardinality. It works beautifully at 20
|
||||
tenants and falls over at 2,000, and nothing in a functional test distinguishes the two. Promotion
|
||||
requires operational scale evidence.
|
||||
|
||||
`MongoTenantLifecyclePolicy`:
|
||||
|
||||
- `requireActivationReady(tenantKey, schemaAndIndexesValidated)` — a tenant is not activated until its
|
||||
schema and indexes are validated. Activating first means the first customer request is the migration
|
||||
test.
|
||||
- `requireDeleteAllowed(...)` — deletion requires an explicit retention decision. Dropping a tenant
|
||||
database is irreversible and takes the backup surface with it.
|
||||
|
||||
`MongoTenantMigrationCoordinator` runs a migration across tenant databases with per-tenant results.
|
||||
Partial failure is normal and must be reported per tenant: "migration failed" across 500 databases is
|
||||
not a report anyone can act on.
|
||||
|
||||
## 3. Choosing
|
||||
|
||||
| | Shared collection | Database per tenant |
|
||||
|---|---|---|
|
||||
| Isolation | Logical, enforced by injection | Physical |
|
||||
| Tenant count | Unbounded | Bounded by connections and file handles |
|
||||
| Per-tenant restore | Hard | Natural |
|
||||
| Noisy neighbour | Shared resources | Isolated |
|
||||
| Migration | One collection | N databases, partial failures |
|
||||
| Cross-tenant query | Possible (and must be forbidden) | Structurally impossible |
|
||||
|
||||
Shared collection is the default. Database-per-tenant is for a small number of tenants with a
|
||||
contractual isolation or per-tenant-restore requirement.
|
||||
|
||||
## 4. Failure recovery
|
||||
|
||||
| Symptom | Cause | Action |
|
||||
|---|---|---|
|
||||
| Cross-tenant data visible | An operation bypassed `TenantScopedMongoOperations` | Treat as a security incident. Find the path, close it, audit access. |
|
||||
| Unique constraint fires across tenants | Unique index missing the tenant field | Rebuild the index with the tenant field as prefix; the validator catches this before it ships. |
|
||||
| One shard holds most data | Tenant id as shard-key prefix with a dominant tenant | Refine the shard key with a high-cardinality suffix. |
|
||||
| Connection exhaustion | Database-per-tenant beyond the connection budget | The scale limit. Consolidate or move to shared collections. |
|
||||
| Migration partially applied across tenants | Normal | `MongoTenantMigrationCoordinator` reports per tenant; re-run for the failures only. |
|
||||
|
||||
## 5. Promotion evidence
|
||||
|
||||
Shared-collection tenancy: actual-topology evidence, a security review covering cross-tenant access,
|
||||
a migration path, failure cases (injection proven on query, atomic filter and first aggregation
|
||||
stage), this runbook. Database-per-tenant additionally requires **operational scale evidence** and
|
||||
stays Experimental until it exists.
|
||||
@@ -0,0 +1,81 @@
|
||||
# Advanced — Search and Vector Search
|
||||
|
||||
**Capabilities:** `MongoCapability.SEARCH`, `MongoCapability.VECTOR_SEARCH`
|
||||
**Properties:** `ca-skeleton.persistence-mongo.advanced.search.enabled`,
|
||||
`ca-skeleton.persistence-mongo.advanced.vector-search.enabled`
|
||||
**Status:** Experimental (design §3.3). Hybrid search likewise.
|
||||
|
||||
## Requirements
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Topology | A deployment with the search service. Atlas Local in a container is a pull-request convenience and is **not** release evidence. |
|
||||
| Privilege | `MongoPrincipalRole.SEARCH_ADMIN` for index management; the application role queries only. |
|
||||
| Gate | `MongoAtlasCapabilityContractSuite` on the actual target deployment. |
|
||||
|
||||
## 1. Created is not ready
|
||||
|
||||
`MongoSearchIndexState`: `CREATED` → `BUILDING` → `READY`, plus `FAILED` and `DELETING`.
|
||||
|
||||
`MongoSearchReadinessGate.requireReady(state)` refuses anything but `READY`. A search index is built
|
||||
asynchronously: the create call returns immediately and the index answers queries with *partial*
|
||||
results while building. Not an error, not empty — partial. A deployment that creates an index and
|
||||
starts querying serves incomplete results for as long as the build takes, and nothing reports it.
|
||||
|
||||
## 2. Search indexes are not application-owned
|
||||
|
||||
`MongoSearchIndexDescriptor.metadataOwnership()` is `MongoMetadataOwnership.SEARCH_MANAGED`, and
|
||||
`droppableByApplicationDrift()` is false. The index reconciliation described in
|
||||
[ADR-MONGO-004](../../adr/ADR-MONGO-004-index-schema-admin-plane.md) must not drop it.
|
||||
|
||||
## 3. Query guardrails
|
||||
|
||||
`MongoSearchQuery` binds an index, an allowlist of paths, the search text and a result limit.
|
||||
|
||||
- `requireAllowedPaths(allowed)` raises `MongoOperationRejectedException` on a path outside the
|
||||
allowlist. Without it, a caller can search any indexed field, including ones indexed for a
|
||||
different purpose.
|
||||
- Search text length and result count are bounded at construction. An unbounded search text is a
|
||||
cost multiplier on someone else's service.
|
||||
|
||||
## 4. Vector search
|
||||
|
||||
`MongoVectorIndexDescriptor.cosine(path, dimensions)` declares the index.
|
||||
`MongoEmbedding.forIndex(index, values)` binds an embedding to it and **rejects a dimension
|
||||
mismatch** — a 1536-dimension embedding against a 768-dimension index is not a runtime degradation,
|
||||
it is a category error, and catching it at construction beats catching it as a confusing server
|
||||
message.
|
||||
|
||||
`MongoEmbedding` copies its backing array in and out. A vector that shares an array with its caller
|
||||
can be mutated after the query is built, which produces a query nobody wrote.
|
||||
|
||||
`MongoVectorQuery` requires `numCandidates > limit` — searching 10 candidates to return 10 results is
|
||||
an exhaustive scan wearing an ANN index's name. `MongoVectorQuery.nearest(embedding, 10)` uses the
|
||||
standard 20× ratio (200 candidates for 10 results).
|
||||
|
||||
## 5. Relevance is the gate, not functionality
|
||||
|
||||
`MongoVectorSearchBenchmarkGate.standard()` requires **recall** alongside latency and index size.
|
||||
`requiredEvidence()` names recall explicitly.
|
||||
|
||||
This is the difference between search and everything else in the platform. A vector index can be
|
||||
functionally perfect — it accepts the index, accepts the query, returns k results, within the latency
|
||||
budget — and return the wrong k. A gate that measures only latency certifies a fast wrong answer.
|
||||
`failures(recall, latencyMs, indexMb)` reports which dimension failed so the finding is actionable.
|
||||
|
||||
## 6. Failure recovery
|
||||
|
||||
| Symptom | Cause | Action |
|
||||
|---|---|---|
|
||||
| Incomplete results after a deploy | Queried a `BUILDING` index | Wait for `READY`. The gate prevents this; if it fired, something bypassed it. |
|
||||
| `MongoOperationRejectedException` on a path | Path not in the allowlist | Add it deliberately, or fix the caller. |
|
||||
| Dimension mismatch | Model changed | A new model means a new index. Build alongside, cut over, then retire. |
|
||||
| Recall dropped without a code change | The index was rebuilt with different parameters, or the data distribution shifted | Re-run the benchmark gate; treat a recall regression like a failing test. |
|
||||
| Index `FAILED` | Build error on the search service | Search-side diagnosis; the application must not fall back to a scan silently. |
|
||||
|
||||
## 7. Promotion evidence
|
||||
|
||||
Per [ADR-MONGO-ADV-001](../../adr/ADR-MONGO-ADV-001-capability-promotion.md): the actual target
|
||||
deployment (not Atlas Local), security review of `SEARCH_ADMIN`, a rebuild path, failure cases
|
||||
(non-ready index refused, disallowed path refused, dimension mismatch refused), this document as the
|
||||
runbook, **and** relevance evidence. Search and vector search do not promote on functional success.
|
||||
@@ -0,0 +1,82 @@
|
||||
# Advanced — Sharding
|
||||
|
||||
**Capability:** `MongoCapability.SHARDING`
|
||||
**Property:** `ca-skeleton.persistence-mongo.advanced.sharding.enabled`
|
||||
**Status:** Advanced. Reshard orchestration remains Experimental.
|
||||
|
||||
## Requirements
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Topology | A real sharded cluster. A replica set cannot exercise routing. |
|
||||
| Server | MongoDB 7.0 or 8.0. |
|
||||
| Privilege | `MongoPrincipalRole.SHARD_ADMIN` for the admin plane; the application role is unchanged. |
|
||||
| Gate | `mongoShardedTest` lane with `MongoShardingContractSuite`. |
|
||||
|
||||
## Shard key
|
||||
|
||||
`ShardKeyDescriptor` declares the key as an ordered list of `ShardKeyPart` plus a `ShardStrategy`:
|
||||
|
||||
| Strategy | Distributes | Cost |
|
||||
|---|---|---|
|
||||
| `RANGE` | by value ranges | Range queries stay targeted; a monotonic key (a timestamp, an `ObjectId`) sends every insert to one shard. |
|
||||
| `HASHED` | by hash of the key | Inserts spread evenly; every range query becomes scatter-gather. |
|
||||
|
||||
There is no strategy that is good at both, which is why the choice is a declaration rather than a
|
||||
default.
|
||||
|
||||
## Routing classification
|
||||
|
||||
`ShardAwareQueryValidator` classifies each query before execution:
|
||||
|
||||
| `MongoRoutingClassification` | Meaning |
|
||||
|---|---|
|
||||
| `TARGETED` | The full shard key is present. One shard answers. |
|
||||
| `PREFIX_TARGETED` | A prefix of a compound key is present. A subset of shards answers. |
|
||||
| `SCATTER_GATHER` | No shard-key predicate. Every shard answers. |
|
||||
| `REJECTED` | Scatter-gather where the profile forbids it. |
|
||||
|
||||
A scatter-gather query is not an error — some queries legitimately need every shard — but it must be
|
||||
declared. Undeclared scatter-gather raises `MongoShardRoutingException`. The reason is that
|
||||
scatter-gather passes every test on a single-shard development cluster and only degrades once the
|
||||
cluster grows, at which point the query is already in production and the fix is a schema change.
|
||||
|
||||
## Unsupported combinations
|
||||
|
||||
- Unique index on a field that is not a prefix of the shard key. MongoDB cannot enforce it across
|
||||
shards, and it fails at index creation, not at query time.
|
||||
- Transactions that touch documents on multiple shards remain supported but cost a cross-shard
|
||||
two-phase commit. Prefer a shard key that keeps a transaction's documents co-located.
|
||||
- CSFLE on a sharded collection: see [encryption.md](encryption.md) for the combinations that are
|
||||
refused.
|
||||
|
||||
## Admin plane
|
||||
|
||||
`MongoShardingAdminGateway` (D4, `SHARD_ADMIN` credential) covers shard-collection, refine-shard-key
|
||||
and reshard.
|
||||
|
||||
`ShardKeyAnalyzer` produces a `ShardKeyReadinessReport` before sharding a collection: cardinality,
|
||||
frequency skew and monotonicity. A key with low cardinality creates jumbo chunks that cannot be split;
|
||||
a monotonic key creates a hot shard. Both are visible in the report and invisible in a functional
|
||||
test.
|
||||
|
||||
`ReshardApproval` is required for a reshard — a named approver and a stated window. Resharding
|
||||
rewrites the collection: it duplicates the data during the operation and saturates IO. It is not a
|
||||
runtime operation and the type refuses to pretend otherwise.
|
||||
|
||||
## Failure recovery
|
||||
|
||||
| Symptom | Cause | Action |
|
||||
|---|---|---|
|
||||
| `MongoShardRoutingException` | Undeclared scatter-gather | Add the shard key to the predicate, or declare the query as scatter-gather in its profile after review. |
|
||||
| Jumbo chunks | Low-cardinality shard key | Refine the shard key (adds a suffix, non-destructive) before considering a reshard. |
|
||||
| One hot shard | Monotonic range key | Refine with a high-cardinality prefix, or reshard to hashed if range queries are not needed. |
|
||||
| Balancer never converges | Chunk migration blocked by long-running operations | Check for long transactions and cursors; the balancer waits on them. |
|
||||
|
||||
## Promotion evidence
|
||||
|
||||
Per [ADR-MONGO-ADV-001](../../adr/ADR-MONGO-ADV-001-capability-promotion.md): actual sharded-cluster
|
||||
evidence, security review of the `SHARD_ADMIN` role, a migration path for an existing unsharded
|
||||
collection, failure cases (undeclared scatter-gather refused, jumbo chunk detected), and this
|
||||
document as the runbook. Reshard orchestration stays Experimental until operational scale evidence
|
||||
exists.
|
||||
@@ -0,0 +1,73 @@
|
||||
# Advanced — Time Series
|
||||
|
||||
**Capability:** `MongoCapability.TIME_SERIES`
|
||||
**Property:** `ca-skeleton.persistence-mongo.advanced.time-series.enabled`
|
||||
**Status:** Advanced.
|
||||
|
||||
## Requirements
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Topology | Replica set or sharded cluster. |
|
||||
| Server | MongoDB 7.0 or 8.0. |
|
||||
| Privilege | Standard application role; collection creation goes through the admin plane. |
|
||||
|
||||
## Descriptor
|
||||
|
||||
`MongoTimeSeriesDescriptor` declares:
|
||||
|
||||
- **timeField** — required, a BSON date. This is the bucketing axis.
|
||||
- **metaField** — optional but nearly always wanted: the series identity (device id, tenant, sensor).
|
||||
Documents sharing a `metaField` value bucket together, which is where the compression comes from.
|
||||
- **granularity** — `MongoTimeSeriesGranularity`:
|
||||
|
||||
| Granularity | Bucket span | Use for |
|
||||
|---|---|---|
|
||||
| `SECONDS` | 1 hour | Sub-second to per-second ingest. |
|
||||
| `MINUTES` | 24 hours | Per-minute metrics. |
|
||||
| `HOURS` | 30 days | Hourly rollups. |
|
||||
|
||||
Granularity that is too fine produces many small buckets and loses the compression; too coarse
|
||||
produces oversized buckets that must be read whole to answer a narrow query.
|
||||
|
||||
## What a time series collection is not
|
||||
|
||||
`MongoTimeSeriesCapabilityValidator` refuses the operations the collection type does not support, at
|
||||
declaration time rather than at first use:
|
||||
|
||||
- **No arbitrary updates.** Time series data is append-mostly. Delete and limited update support
|
||||
exists on recent servers but is not part of this platform's contract.
|
||||
- **No unique index on the measurement.** There is no `_id` to be unique on in the usual sense.
|
||||
- **No CSFLE.** Refused — see [encryption.md](encryption.md).
|
||||
- **No change stream on the raw buckets** as a business event source. The bucket documents are a
|
||||
storage representation, not your measurements.
|
||||
|
||||
Converting an existing regular collection to a time series collection is a copy, not an alter. Plan
|
||||
it as a migration with a dual-write window.
|
||||
|
||||
## TTL
|
||||
|
||||
Time series collections use `expireAfterSeconds` on the collection rather than a TTL index on a
|
||||
field. The [TTL rules](../schema-index-migration-guide.md#4-ttl) still apply: expiry is physical
|
||||
cleanup on a bucket boundary, so a measurement can outlive its expiry by up to a bucket span plus the
|
||||
monitor interval. Do not treat absence as a deadline.
|
||||
|
||||
## Operations
|
||||
|
||||
`MongoTimeSeriesOperations` is the port for insert and windowed read. Reads are bounded by the same
|
||||
`MongoOperationBudget` as everything else: an unbounded time-range query on a time series collection
|
||||
is the fastest way to read a year of data into heap.
|
||||
|
||||
## Failure recovery
|
||||
|
||||
| Symptom | Cause | Action |
|
||||
|---|---|---|
|
||||
| Writes rejected with an unsupported-operation error | An update or unique-index expectation | The collection type does not support it; change the access pattern. |
|
||||
| Poor compression / large storage | Missing `metaField`, or granularity too fine | Both require a rebuild; measure on a copy before committing. |
|
||||
| Slow range queries | Granularity too coarse for the query window | Same: rebuild with the granularity matched to the dominant query. |
|
||||
|
||||
## Promotion evidence
|
||||
|
||||
Actual-topology evidence on the target deployment, a migration path from the existing collection,
|
||||
failure cases (unsupported update refused, CSFLE combination refused), and this document as the
|
||||
runbook.
|
||||
Reference in New Issue
Block a user