clean-architecture-backend-template/docs/mongodb/change-stream-guide.md

Change Stream Guide

Design §20, decision D-12. A change stream is an at-least-once projector, not an event bus.

1. What a change stream is not

D-12 is explicit: a physical change event is not a business integration event. The two differ in ways that matter to every consumer:

Change event	Integration event
Emitted per document write	Emitted per business fact
Shape follows the storage schema	Shape is a published contract
A refactor of the document changes it	A refactor of the document does not change it
Replayed on resume, duplicated on retry	Versioned and deliberately evolved

Publishing raw change events externally makes your storage schema a public API, and the first time someone renames a field the downstream consumers break. If you need to bridge to messaging, use the Advanced bridge, which maps to an owned envelope (advanced/multi-tenancy.md is separate; the bridge is described in §7 below).

2. Subscription and resume

MongoChangeStreamSubscription declares the collection, pipeline and consistency. MongoResumePosition is either a resume token or a cluster time; MongoResumeCheckpoint is what gets persisted and MongoResumeCheckpointStore persists it.

The checkpoint stores the token as a Base64 encodedToken string rather than a byte array — a record with an array component has broken equality, and a checkpoint that does not compare correctly is a checkpoint that silently fails its own dedup test.

3. Checkpoint after processing, not after receiving

The ordering rule that makes at-least-once actually hold:

receive event
→ process it (idempotently)
→ persist the checkpoint

Checkpointing on receipt turns the delivery guarantee into at-most-once, and the events lost are exactly the ones the process died while handling.

4. Idempotency

MongoChangeEventIdentity is the dedup key: (resumeToken, documentKey, clusterTime, operationType). MongoChangeDeduplicationStore records what has been applied. Duplicates are not an edge case — every resume after any interruption replays at least one event, so a projector that is not idempotent is wrong on its first restart, not on some rare day.

MongoChangeProjector returns a MongoChangeProjectionResult so the runner can distinguish applied from skipped-as-duplicate, and the skip count is worth a metric: a sudden rise means something is looping.

5. States and recovery

MongoChangeStreamState: STARTING, RUNNING, RESUMING, STOPPED, HISTORY_LOST.

MongoChangeStreamRecoveryPolicy returns a MongoChangeStreamRecoveryDecision, which is either resume() (auto-resume from the checkpoint) or halt(state, runbook). A halting decision must name a runbook — a decision that only says "stopped" leaves the on-call engineer to work out from scratch whether the projection can be rebuilt and from what.

Situation	Decision
Transient network error, token still valid	resume()
Primary failover	resume() — the token survives an election
invalidate (collection dropped/renamed)	halt(STOPPED, …) → MongoInvalidateRecovery
Token no longer in the oplog	halt(HISTORY_LOST, "history-lost") → MongoChangeHistoryLostException

6. History lost

MongoChangeHistoryLostException is raised when the resume token predates the oldest oplog entry. The stream cannot be resumed: the events between the checkpoint and now are gone from the server, and no amount of retrying brings them back.

What the platform will not do is silently restart from "now". That looks like a recovery and is actually a silent gap in the projection — the worst possible outcome, because nothing reports it. The runner halts and requires an operator decision. See runbooks/history-lost.md.

7. Bridging to messaging (Advanced)

MongoChangeMessagingBridge is opt-in behind MongoCapability.CHANGE_STREAM plus the bridge's own flag. It maps a change event to a platform-owned MongoIntegrationEventEnvelope through MongoChangeToIntegrationEventMapper and hands it to a MongoIntegrationEventPublisher port.

The port is defined in the bridge package rather than imported from the messaging adapter because the architecture registry forbids adapter-to-adapter dependencies; the composition root supplies the implementation.

MongoBridgeOutboxPolicy and MongoBridgeCheckpointPolicy state the delivery contract: publish then checkpoint, at-least-once, consumers must dedup on the envelope's event id.

8. Operating notes

• Change streams require a replica set. MongoStartupValidator refuses a change-stream profile on STANDALONE.
• The change-stream principal is its own role (MongoPrincipalRole.CHANGE_STREAM) with changeStream and find — not the application write credential.
• Oplog window is the recovery budget. If the oplog holds four hours, a consumer that is down for five hours needs a rebuild, not a resume. Alert on consumer lag against the oplog window, not against wall-clock.