diff --git a/docs/messaging/configuration-reference.md b/docs/messaging/configuration-reference.md index 06f345e..0e864ee 100644 --- a/docs/messaging/configuration-reference.md +++ b/docs/messaging/configuration-reference.md @@ -1,9 +1,18 @@ # 설정 레퍼런스 +> **Prefix.** Every property below binds under `app.messaging`, which is the prefix the deployed +> runtime and the `APP_MESSAGING_*` environment variables already use. Earlier revisions of this +> page documented a bare `messaging` prefix and the starter bound `backend.messaging`; neither +> bound what this page describes, so a deployment configured from it changed nothing. A key under +> either of the old prefixes now fails startup with a message naming the key — see +> `MessagingPrefixMigrationValidator`. + + ## Destination profile ```yaml -messaging: +app: + messaging: destinations: order-events: broker: kafka-primary @@ -77,7 +86,8 @@ messaging: ### Kafka ```yaml -messaging: +app: + messaging: brokers: kafka-primary: type: kafka @@ -96,7 +106,8 @@ messaging: ### RabbitMQ ```yaml -messaging: +app: + messaging: brokers: rabbit-primary: type: rabbitmq @@ -117,7 +128,8 @@ messaging: ## 보안 ```yaml -messaging: +app: + messaging: security: kafka-primary: producer: { type: SASL_SCRAM, credential-id: kafka-producer } @@ -135,7 +147,8 @@ messaging: 기본값은 전부 `false`다. ```yaml -messaging: +app: + messaging: experimental: kafka-share: false pulsar: false @@ -147,7 +160,8 @@ messaging: ## Backpressure ```yaml -messaging: +app: + messaging: backpressure: global-limit: 512 per-destination-limit: 64 # global-limit 이하여야 한다 diff --git a/docs/messaging/cutover.md b/docs/messaging/cutover.md new file mode 100644 index 0000000..a65e341 --- /dev/null +++ b/docs/messaging/cutover.md @@ -0,0 +1,34 @@ +# 기존 runtime → 신규 messaging platform cutover (MSG-015) + +## 왜 기계적 매핑이 안 되는가 + +두 outbox 모델의 enum 이름이 겹치는데 의미가 반대다. + +| 모델 | retryable | terminal | +|---|---|---| +| 기존 `OutboxEventStatus` | `FAILED` (`next_attempt_at` 보유) | `DEAD` | +| 신규 `OutboxStatus` | `AMBIGUOUS` | `FAILED`, `EXHAUSTED` | + +이름으로 매핑하면 **확정 거절이 무한 재시도**가 되고 **불확정이 park**된다. 그래서 application은 +자기 어휘(`OutboxPublishOutcome`)만 쓰고, 변환은 bridge adapter가 한다. + +## 지금 반영된 것 + +- `OutboxPublishOutcome` — `CONFIRMED` / `AMBIGUOUS` / `REJECTED_BEFORE_SEND` / + `REJECTED_AFTER_BROKER`. application이 소유하는 canonical 결과 타입이며, "리턴 or throw"만 가능한 + 기존 어댑터를 위해 `OutboxMessagePublishPort.publishForOutcome`의 default가 `CONFIRMED`를 돌려준다. +- `APPLICATION_DOES_NOT_DEPEND_ON_THE_MESSAGING_PLATFORM` ArchUnit 규칙 — application-core가 + `dev.caskeleton.messaging..`를 import하면 빌드가 깨진다. +- 반대 방향(신규 `PublishResult` → application outcome) 매핑 규칙을 테스트로 고정. + +## 남은 것 + +- `messaging-platform-bridge` outbound leaf: validated application event → platform envelope, + `PublishResult` → `OutboxPublishOutcome`. registry에 leaf를 추가하는 변경이라 별도 커밋. +- golden contract 테스트: event/message ID, type, schema revision, partition/order/correlation/ + causation/tenant/trace, payload digest, wire version이 bytes 단위로 보존되는지. +- 단일 publication authority: 기존 `OutboxPublicationAuthority` fence를 재사용해 writer/relay가 + 동시에 ACTIVE가 되지 않도록. **dual write/publish는 금지** — 한 business fact가 두 durable store와 + 두 relay로 나가는 상태가 cutover에서 가장 위험하다. +- 첫 cutover 범위는 **transport만** 교체(저장소는 기존 유지). storage migration은 shadow read → + authority switch → old backlog drain 순서로 별도 release. diff --git a/docs/messaging/outbox-inbox.md b/docs/messaging/outbox-inbox.md index a3813ca..e28bd94 100644 --- a/docs/messaging/outbox-inbox.md +++ b/docs/messaging/outbox-inbox.md @@ -41,9 +41,26 @@ failed로 표시하면 broker가 이미 가지고 있을 수 있는 메시지를 ### lease ```text -status IN ('PENDING','AMBIGUOUS') AND (lease_expires_at IS NULL OR lease_expires_at <= now) +status IN ('PENDING','AMBIGUOUS','IN_FLIGHT') + AND (lease_expires_at IS NULL OR lease_expires_at <= now) + AND next_attempt_at <= now + AND attempts < maxAttempts ``` +`IN_FLIGHT`가 목록에 있는 것이 핵심이다. relay가 publish 도중 죽으면 row는 `IN_FLIGHT`로 남는데, +이를 제외하면 그 메시지는 **영원히** 발행되지 않는다 — outbox가 막으려던 바로 그 실패다. 대신 +lease가 만료됐을 때만 회수하므로, 살아 있는 relay가 들고 있는 row는 회수되지 않는다. + +회수는 **같은 `message_id`로** 이루어지고 `lease_token`이 1 증가한다. 새 id를 발급하면 "전달됐을 +수도 있는 메시지"가 "확실히 두 번째"가 되기 때문이다 (위의 AMBIGUOUS 논의와 같은 이유). + +이 문단의 근거는 실제 PostgreSQL 컨테이너 레인이다: + +- `OutboxPostgresIT#anExpiredLeaseBecomesClaimableAgain` — 만료된 lease의 재회수 +- `OutboxPostgresIT#anExpiryReclaimKeepsTheMessageIdAndAdvancesTheToken` — 같은 id, 증가한 token +- `OutboxPostgresIT#aLeasedRowIsInvisibleToASecondRelayInstance` — 살아 있는 lease는 회수 불가 +- `OutboxPostgresIT#aSupersededRelayCannotOverwriteTheOutcomeOfTheOneThatReplacedIt` — fencing + partial index `ix_messaging_outbox_claimable`이 이 쿼리를 backlog 크기에 비례하게 유지한다. PUBLISHED row는 retention job이 지울 때까지 쌓이기 때문이다. diff --git a/docs/messaging/support-matrix.md b/docs/messaging/support-matrix.md index dc9356c..000cfca 100644 --- a/docs/messaging/support-matrix.md +++ b/docs/messaging/support-matrix.md @@ -3,11 +3,28 @@ 플랫폼이 **무엇을 보장하는지**와 **무엇을 보장하지 않는지**를 브로커별로 고정한다. 여기 없는 조합은 지원되지 않는다. +> **인증 근거.** 이 표의 버전은 이 저장소의 컨테이너 레인이 실제로 실행한 이미지다. 이전 판은 +> Kafka 4.2/4.3을 선언했지만 fixture는 `apache/kafka:4.1.0`, lockfile client는 4.1.1이었다 — 표와 +> 코드 상수가 서로 일치했을 뿐 어느 쪽도 실행된 적이 없었다. 장애 시나리오 커버리지도 마찬가지로 +> `BrokerFailureMatrix.shipped()` 하드코딩이 아니라 레인이 낸 증거(`BrokerCertificationEvidence`)에서 +> 나온다. 증거가 없는 조합은 `NOT_COVERED`다 (MSG-014). + +> **모듈 이름과 런타임 편입.** `messaging-outbox-jdbc-postgresql` / `messaging-inbox-jdbc-postgresql`은 +> 이전에 `-jpa`로 불렸다. 구현은 Spring JDBC이고 SQL은 PostgreSQL 전용(`?::jsonb`, +> `FOR UPDATE SKIP LOCKED`, `ON CONFLICT`, `TIMESTAMPTZ`)이므로, 그 이름은 쓰지 않는 기술을 +> 광고하고 vendor 중립 port(`messaging-reliability-api`)의 위치를 가렸다 (MSG-023). +> +> 또한 registry의 messaging leaf는 모두 `runtime_memberships`가 비어 있다. 이는 **build-only / +> incubating** — 어느 composition root에도 편입되지 않았다는 뜻이며, 아래의 등급과는 다른 축이다. +> 등급은 "무엇이 증명되었는가", membership은 "무엇이 실행되는가"를 말한다. 애플리케이션에 배선하려면 +> registry를 먼저 바꾸고 `verifyRuntimeModuleMembership`을 통과시켜야 한다. 자세한 규칙은 +> `src/messaging/CLAUDE.md`가 소유한다. + ## 브로커 등급 | 브로커 | 등급 | 인증 기준 | Stable 기능 | 제한 | |---|---|---|---|---| -| Kafka | Stable | 4.2+ / 4.3.x | producer idempotence, consumer group, batch, pause/resume, replay, transaction capability | Share Group은 Experimental | +| Kafka | Stable | 4.1.x | producer idempotence, consumer group, batch, pause/resume, replay, transaction capability | Share Group은 Experimental | | RabbitMQ | Stable | 4.3.x | exchange/routing, publisher confirm, mandatory return, manual ACK, quorum queue, retry queue, DLQ | stream 및 특수 plugin 미지원 | | Pulsar | Experimental | 4.0 LTS + 4.2 | typed publish/consume, Shared, Key_Shared, schema | transaction 미승격, 기본 비활성 | | NATS JetStream | Experimental | 2.14.x | stream, durable consumer, explicit ACK, dedupe, replay | native DLQ 없음(플랫폼이 대행), 기본 비활성 | diff --git a/messaging/CLAUDE.md b/messaging/CLAUDE.md new file mode 100644 index 0000000..e25f9a8 --- /dev/null +++ b/messaging/CLAUDE.md @@ -0,0 +1,85 @@ +# messaging — local authority for the messaging platform family + +이 문서는 messaging family의 **local authority**다. leaf 목록·gradle path·허용 의존성은 +`src/config/architecture/modules.json`이 SSOT이며 이 문서는 그것을 복제하지 않는다. 여기서 정하는 +것은 registry가 표현할 수 없는 것들이다: family별 framework 규칙, public surface 정책, Stable 승격에 +필요한 증거, runtime membership이 비어 있다는 사실의 의미. + +Root 정책(`CLAUDE.md` / `AGENTS.md`)과 충돌하면 root가 이긴다. + +## Family와 framework 규칙 + +leaf의 정확한 id는 registry에서 읽는다. family는 **디렉터리 이름의 접두사**가 아니라 아래 표의 +역할로 정해진다. + +| Family | 역할 | 허용 framework | +| --- | --- | --- | +| `*-api` (`core-api`, `schema-api`, `policy`, `reliability-api`, `admin-api`, `transport-spi`) | 값 타입, 포트, 정책 계약 | Java stdlib만. Spring/broker SDK/logging framework 금지 | +| `schema-*` (json/avro/protobuf), `cloudevents` | 인코딩 구현 | 해당 codec 라이브러리만. Spring 금지 | +| `runtime-core`, `security`, `observability`, `admin-runtime` | framework-free 실행 로직 | Micrometer는 `observability`에서만. 나머지는 stdlib | +| `kafka`, `rabbit`, `pulsar-experimental`, `nats-experimental`, `kafka-share-experimental` | broker adapter | 해당 broker client. Spring은 `rabbit`의 `spring-rabbit`처럼 client가 곧 Spring인 경우만 | +| `outbox-jdbc-postgresql`, `inbox-jdbc-postgresql` | 관계형 신뢰성 adapter | Spring JDBC / Spring TX. **JPA 아님** — 이름이 곧 계약이다 | +| `spring-boot-starter`, `spring-cloud-stream-bridge` | 조립 경계 | Spring Boot autoconfigure. 여기서만 | +| `testkit` | 테스트 지원 | Testcontainers, JUnit. production leaf가 의존하지 않는다 | + +`outbox`/`inbox` leaf가 `jdbc-postgresql`로 명명된 이유: 구현은 Spring JDBC이고 SQL은 +`?::jsonb` · `FOR UPDATE SKIP LOCKED` · `ON CONFLICT` · `TIMESTAMPTZ`로 PostgreSQL 전용이다. +`jpa`라는 이름은 사용하지 않는 기술을 광고하고, vendor 중립 port가 어디 있는지도 숨겼다. vendor +중립 계약은 `messaging-reliability-api`에 있다. + +## Public surface 정책 + +- **vendor 타입을 public signature에 노출하면 그 의존성은 `api`로 선언한다.** 숨긴다고 타입이 + 숨겨지지 않는다 — consumer가 이 모듈이 문서화한 API에 대해 컴파일하지 못하게 만들 뿐이다. + (`kafka-clients`, `spring-rabbit`, `cloudevents-api`, `micrometer-core`가 이 규칙으로 `api`다.) +- 반대로 **노출하지 않는 것은 `implementation`으로 낮춘다.** starter가 노출하지 않는 하위 모듈을 + `implementation`으로 내린 것이 그 예다. +- 새 public 타입은 그 모듈의 계약이다. 삭제·시그니처 변경은 breaking change로 취급한다. + +## Runtime membership이 비어 있다는 것의 의미 + +registry의 모든 messaging leaf는 `runtime_memberships`가 비어 있다. 이는 **build-only / +incubating**을 뜻한다: + +- 두 composition root(`app-bootstrap`, `sample-portfolio`) 중 어디에도 런타임으로 들어가 있지 않다. +- 따라서 이 leaf들은 컴파일·테스트·계약 검증의 대상이지, 배포된 애플리케이션이 실행하는 코드가 + 아니다. +- 애플리케이션에 실제로 배선하려면 registry의 `runtime_memberships`를 먼저 바꾸고 + `verifyRuntimeModuleMembership`을 통과시켜야 한다. 코드만 추가하는 것으로는 런타임에 들어가지 + 않는다. +- 지원 상태 문서(`docs/messaging/support-matrix.md`)의 tier는 이 사실과 별개다. tier는 "무엇이 + 증명되었는가", membership은 "무엇이 실행되는가"다. + +## Stable 승격에 필요한 증거 + +adapter를 Experimental → Stable로 올릴 때 필요한 것 (`docs/messaging/support-matrix.md`가 현재 +상태를 기록한다): + +1. 실제 broker container를 쓰는 fault 시나리오 lane이 존재하고, 그 lane이 + `BrokerCertificationEvidence`를 산출한다. +2. `NetworkFaultScenario` 전 항목에 대해 evidence가 있거나, 없는 항목이 `knownGaps`로 명시된다. +3. capability 선언이 코드에서 파생된다 — `MessagingCapabilities`에 손으로 `true`를 쓰는 것은 + 증거가 아니다. +4. publish 실패 분류가 typed signal에서만 나온다. exception class 이름 substring 판정은 + Experimental에서도 금지다(MSG-022). +5. unknown failure의 기본값이 `AMBIGUOUS`다. `REJECTED`는 전송이 시작되지 않았다는 typed 증거가 + 있을 때만. + +Stable 문구는 evidence manifest가 존재할 때만 문서에 쓴다. + +## Focused test + +leaf의 `gradle_path`를 registry에서 읽어 파생한다. + +```bash +cd src +./gradlew :test --console=plain +``` + +Family 전체 검증: + +```bash +./gradlew verifyCleanArchitectureDependencies --console=plain +./gradlew verifyDocumentedLeafCount --console=plain +./gradlew verifyDependencyLocks --console=plain +``` diff --git a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/AdminOperationJournal.java b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/AdminOperationJournal.java new file mode 100644 index 0000000..012b2d0 --- /dev/null +++ b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/AdminOperationJournal.java @@ -0,0 +1,93 @@ +package dev.caskeleton.messaging.admin; + +import java.time.Duration; +import java.time.Instant; +import java.util.Optional; + +/** + * Where a destructive operation records that it started, how far it got, and how it ended. + * + *

The implementation this replaced was a {@code ConcurrentHashMap} registered by the starter as + * the default. Two consequences followed, and both are worse than having no store at all because + * the map made the platform look protected. A restart forgot every claim, so the same approval + * could be executed again by the same process. And two replicas each had their own map, so both + * could claim the same approval at the same moment and each believe it was the only one. + * + *

Implementations must therefore be shared and durable, must enforce uniqueness on {@code + * (approvalTicket, planDigest)}, and must hand out leases with a monotonic fencing token so a + * process that stalled past its lease cannot write over the replica that took over from it. + */ +public interface AdminOperationJournal { + + /** + * Claims an operation, or resumes the previous attempt at it. + * + * @param approvalTicket the approval authorising the operation + * @param planDigest the exact plan + * @param operationId this execution's identity + * @param leaseOwner the runtime claiming it + * @param leaseDuration how long the lease holds before another runtime may take over + * @param now the current instant + * @return the lease, carrying the point to resume from + * @throws dev.caskeleton.messaging.api.error.MessageAuthorizationException when the operation + * already completed, or another runtime holds a live lease on it + */ + AdminOperationLease begin( + String approvalTicket, + PlanDigest planDigest, + String operationId, + String leaseOwner, + Duration leaseDuration, + Instant now); + + /** + * Records durable progress and extends the lease. + * + * @param lease the lease held + * @param itemsCompleted how many items are now durably done + * @param leaseDuration how much longer to hold the lease + * @param now the current instant + * @throws dev.caskeleton.messaging.api.error.MessageAuthorizationException when the lease has + * been taken over by a newer token + */ + void checkpoint( + AdminOperationLease lease, int itemsCompleted, Duration leaseDuration, Instant now); + + /** + * Marks the operation finished. The approval is spent. + * + * @param lease the lease held + * @param itemsCompleted the final count + * @param now the current instant + */ + void complete(AdminOperationLease lease, int itemsCompleted, Instant now); + + /** + * Marks the operation failed at its current checkpoint, leaving it resumable. + * + * @param lease the lease held + * @param itemsCompleted how many items are durably done + * @param failureCode a sanitised, bounded reason + * @param now the current instant + */ + void fail(AdminOperationLease lease, int itemsCompleted, String failureCode, Instant now); + + /** + * Reads the journal entry for an operation. + * + * @param approvalTicket the approval + * @param planDigest the plan + * @return the record, when the operation has been journalled + */ + Optional find(String approvalTicket, PlanDigest planDigest); + + /** + * Whether this implementation survives a restart and is shared between replicas. + * + *

Declared rather than inferred so the starter can refuse to run a production profile on an + * in-memory journal instead of discovering the gap during an incident. + * + * @return true when the journal is durable and shared + */ + boolean isDurable(); +} diff --git a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/AdminOperationLease.java b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/AdminOperationLease.java new file mode 100644 index 0000000..de2fb5a --- /dev/null +++ b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/AdminOperationLease.java @@ -0,0 +1,55 @@ +package dev.caskeleton.messaging.admin; + +import java.util.Objects; + +/** + * The right to execute one journalled operation, and the point to resume it from. + * + *

{@code resumeFrom} is the whole reason a lease is handed out rather than a boolean. A retry + * after a crash is not a new execution of the approval — it is the same operation continuing, and + * treating it as new is what republishes the items the first attempt already moved. + * + * @param approvalTicket the approval being executed + * @param planDigest the plan being executed + * @param operationId this execution's identity + * @param leaseOwner the runtime holding the lease + * @param leaseToken the fencing token; a write from an older token must be refused + * @param resumeFrom how many items the previous attempt durably completed + */ +public record AdminOperationLease( + String approvalTicket, + PlanDigest planDigest, + String operationId, + String leaseOwner, + long leaseToken, + int resumeFrom) { + + /** Validates the lease. */ + public AdminOperationLease { + Objects.requireNonNull(planDigest, "planDigest must not be null"); + if (approvalTicket == null || approvalTicket.isBlank()) { + throw new IllegalArgumentException("approvalTicket must not be blank"); + } + if (operationId == null || operationId.isBlank()) { + throw new IllegalArgumentException("operationId must not be blank"); + } + if (leaseOwner == null || leaseOwner.isBlank()) { + throw new IllegalArgumentException("leaseOwner must not be blank"); + } + if (leaseToken < 1) { + throw new IllegalArgumentException("a lease token starts at one"); + } + if (resumeFrom < 0) { + throw new IllegalArgumentException("resumeFrom must not be negative"); + } + } + + /** + * Reports whether this lease continues an earlier attempt. + * + * @return true when items were already completed + */ + public boolean isResumption() { + return resumeFrom > 0; + } +} diff --git a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/AdminOperationRecord.java b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/AdminOperationRecord.java new file mode 100644 index 0000000..752d862 --- /dev/null +++ b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/AdminOperationRecord.java @@ -0,0 +1,63 @@ +package dev.caskeleton.messaging.admin; + +import java.time.Instant; +import java.util.Objects; +import java.util.Optional; + +/** + * One row of the destructive-operation journal. + * + *

Keyed by {@code (approvalTicket, planDigest)} rather than by the ticket alone, because the + * ticket alone cannot distinguish "this approval already ran" from "this approval is being reused + * for a different plan" — and those need opposite answers. + * + * @param approvalTicket the approval that authorised the operation + * @param planDigest the exact plan that was authorised + * @param operationId the caller-side identity of this execution + * @param state where the operation stands + * @param itemsCompleted how many items are durably done + * @param leaseOwner which runtime currently holds the lease + * @param leaseToken the fencing token, incremented on every takeover + * @param leaseExpiresAt when another runtime may take the lease over + * @param startedAt when the operation first began + * @param finishedAt when it reached a terminal state + * @param failureCode the sanitised reason it failed, when it did + */ +public record AdminOperationRecord( + String approvalTicket, + PlanDigest planDigest, + String operationId, + AdminOperationState state, + int itemsCompleted, + String leaseOwner, + long leaseToken, + Instant leaseExpiresAt, + Instant startedAt, + Optional finishedAt, + Optional failureCode) { + + /** Validates the record. */ + public AdminOperationRecord { + Objects.requireNonNull(planDigest, "planDigest must not be null"); + Objects.requireNonNull(state, "state must not be null"); + Objects.requireNonNull(leaseExpiresAt, "leaseExpiresAt must not be null"); + Objects.requireNonNull(startedAt, "startedAt must not be null"); + Objects.requireNonNull(finishedAt, "finishedAt must not be null"); + Objects.requireNonNull(failureCode, "failureCode must not be null"); + if (approvalTicket == null || approvalTicket.isBlank()) { + throw new IllegalArgumentException("approvalTicket must not be blank"); + } + if (operationId == null || operationId.isBlank()) { + throw new IllegalArgumentException("operationId must not be blank"); + } + if (leaseOwner == null || leaseOwner.isBlank()) { + throw new IllegalArgumentException("leaseOwner must not be blank"); + } + if (itemsCompleted < 0) { + throw new IllegalArgumentException("itemsCompleted must not be negative"); + } + if (leaseToken < 1) { + throw new IllegalArgumentException("a lease token starts at one"); + } + } +} diff --git a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/AdminOperationState.java b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/AdminOperationState.java new file mode 100644 index 0000000..ec1ae81 --- /dev/null +++ b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/AdminOperationState.java @@ -0,0 +1,21 @@ +package dev.caskeleton.messaging.admin; + +/** + * Where a journalled destructive operation stands. + * + *

The store this replaces recorded one fact — "this approval was claimed" — and recorded it + * before any work happened. A run that died halfway had consumed its approval, left no record of + * how far it got, and offered the operator two equally bad choices: request a fresh approval and + * redo work that may already have been done, or leave the operation half-applied. + */ +public enum AdminOperationState { + + /** Claimed and in progress. A lease is held; the checkpoint says how far it got. */ + STARTED, + + /** Every item finished. The approval is spent and must not run again. */ + COMPLETED, + + /** The run stopped before finishing. It may be resumed from its checkpoint under a new lease. */ + FAILED +} diff --git a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/ApprovalGrant.java b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/ApprovalGrant.java new file mode 100644 index 0000000..7cd97ac --- /dev/null +++ b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/ApprovalGrant.java @@ -0,0 +1,98 @@ +package dev.caskeleton.messaging.admin; + +import dev.caskeleton.messaging.api.destination.DestinationName; +import java.time.Instant; +import java.util.Objects; +import java.util.Optional; + +/** + * Everything an issuer signs when granting one destructive operation. + * + *

{@link AdminApproval} carried a ticket, an approver, and a window. Nothing in it said which + * operation, on which destination, against which topology, or up to what impact — so the same + * approval authorised every plan presented inside the window, and the audit trail recorded a ticket + * that proved nothing about what was executed. + * + *

Every field here is part of the signed canonical form, which is what makes the grant + * non-transferable: change the target, the topology version, the impact ceiling, or the plan, and + * the signature no longer verifies. + * + * @param approval the ticket, approver, and validity window + * @param requestedBy the operator who will execute it, kept separate from the approver + * @param operation the destructive operation being authorised + * @param source the destination read from + * @param target the destination written to, when the operation has one + * @param topologyVersion the topology the impact estimate was computed against + * @param maxImpact the largest number of messages the approver agreed to + * @param planDigest the fingerprint of the exact plan + */ +public record ApprovalGrant( + AdminApproval approval, + String requestedBy, + DestructiveOperation operation, + DestinationName source, + Optional target, + String topologyVersion, + long maxImpact, + PlanDigest planDigest) { + + /** Validates the grant and enforces subject separation. */ + public ApprovalGrant { + Objects.requireNonNull(approval, "approval must not be null"); + Objects.requireNonNull(operation, "operation must not be null"); + Objects.requireNonNull(source, "source must not be null"); + Objects.requireNonNull(target, "target must not be null"); + Objects.requireNonNull(topologyVersion, "topologyVersion must not be null"); + Objects.requireNonNull(planDigest, "planDigest must not be null"); + if (requestedBy == null || requestedBy.isBlank()) { + throw new IllegalArgumentException("requestedBy must not be blank"); + } + if (approval.approvedBy().equals(requestedBy)) { + // Four-eyes, expressed where it cannot be skipped: an operator approving their own + // destructive operation is the control not existing. + throw new IllegalArgumentException( + "the approver and the operator must be different people; both are " + requestedBy); + } + if (maxImpact < 0) { + throw new IllegalArgumentException("maxImpact must not be negative"); + } + } + + /** + * Renders the grant deterministically for signing and verification. + * + *

Every field is length-prefixed rather than delimited. A delimiter can be smuggled into a + * ticket or an identity to make two different grants render identically; a length prefix cannot. + * + * @return the canonical form + */ + public String canonicalForm() { + StringBuilder canonical = new StringBuilder("v1"); + appendField(canonical, approval.ticket()); + appendField(canonical, approval.approvedBy()); + appendField(canonical, requestedBy); + appendField(canonical, operation.name()); + appendField(canonical, source.value()); + appendField(canonical, target.map(DestinationName::value).orElse("")); + appendField(canonical, topologyVersion); + appendField(canonical, Long.toString(maxImpact)); + appendField(canonical, planDigest.value()); + appendField(canonical, approval.approvedAt().toString()); + appendField(canonical, approval.validUntil().toString()); + return canonical.toString(); + } + + private static void appendField(StringBuilder canonical, String value) { + canonical.append(value.length()).append(':').append(value); + } + + /** + * Reports whether the grant still authorises anything. + * + * @param now the current instant + * @return true while inside the approval window + */ + public boolean isValidAt(Instant now) { + return approval.isValidAt(now); + } +} diff --git a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/ApprovalVerifier.java b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/ApprovalVerifier.java new file mode 100644 index 0000000..a44fb02 --- /dev/null +++ b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/ApprovalVerifier.java @@ -0,0 +1,29 @@ +package dev.caskeleton.messaging.admin; + +import java.time.Instant; + +/** + * The only thing that can turn a signed grant into a {@link VerifiedApproval}. + * + *

An implementation must check, at minimum: the issuer's signature over {@link + * ApprovalGrant#canonicalForm()}, that the approval window is open at {@code now}, and that the + * digest the caller is about to execute is the digest that was signed. Everything else the grant + * binds — operation, source, target, topology version, impact ceiling, approver-versus-operator + * separation — is inside the canonical form, so a signature check covers them all at once. + */ +public interface ApprovalVerifier { + + /** + * Verifies a signed grant against the plan actually about to run. + * + * @param grant what the issuer signed + * @param signature the issuer's signature over the grant's canonical form + * @param executingDigest the digest of the plan the caller is about to execute + * @param now the current instant + * @return the verified approval + * @throws dev.caskeleton.messaging.api.error.MessageAuthorizationException when anything about + * the grant, the signature, the window, or the digest does not hold + */ + VerifiedApproval verify( + ApprovalGrant grant, String signature, PlanDigest executingDigest, Instant now); +} diff --git a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/ApprovedRedrivePlan.java b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/ApprovedRedrivePlan.java index 43552e4..7142a2b 100644 --- a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/ApprovedRedrivePlan.java +++ b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/ApprovedRedrivePlan.java @@ -1,5 +1,6 @@ package dev.caskeleton.messaging.admin; +import dev.caskeleton.messaging.api.destination.DestinationName; import dev.caskeleton.messaging.api.error.MessageAuthorizationException; import java.time.Instant; import java.util.Objects; @@ -17,11 +18,42 @@ import java.util.Objects; * @param loopAcknowledged whether the approver accepted the previously-redriven candidates */ public record ApprovedRedrivePlan( - RedrivePlan plan, AdminApproval approval, boolean loopAcknowledged) { + RedrivePlan plan, VerifiedApproval approval, boolean loopAcknowledged) { + /** Binds the approval to this exact plan at construction. */ public ApprovedRedrivePlan { Objects.requireNonNull(plan, "plan must not be null"); Objects.requireNonNull(approval, "approval must not be null"); + if (!approval.grant().planDigest().equals(plan.digest())) { + throw new MessageAuthorizationException( + "APPROVAL_PLAN_MISMATCH", + "approval %s was issued for a different plan than the one it is attached to" + .formatted(approval.ticket())); + } + if (approval.grant().operation() != DestructiveOperation.REDRIVE) { + throw new MessageAuthorizationException( + "APPROVAL_OPERATION_MISMATCH", + "approval %s authorises %s, not a redrive" + .formatted(approval.ticket(), approval.grant().operation())); + } + if (!approval.grant().source().equals(plan.request().source()) + || !approval.grant().target().equals(java.util.Optional.of(plan.request().target()))) { + throw new MessageAuthorizationException( + "APPROVAL_SOURCE_MISMATCH", + "approval %s authorises %s to %s, not %s to %s" + .formatted( + approval.ticket(), + approval.grant().source().value(), + approval.grant().target().map(DestinationName::value).orElse("(none)"), + plan.request().source().value(), + plan.request().target().value())); + } + if (plan.candidates() > approval.grant().maxImpact()) { + throw new MessageAuthorizationException( + "APPROVAL_IMPACT_EXCEEDED", + "the plan moves %d messages but the approver agreed to at most %d" + .formatted(plan.candidates(), approval.grant().maxImpact())); + } } /** @@ -39,6 +71,13 @@ public record ApprovedRedrivePlan( throw new MessageAuthorizationException( "APPROVAL_EXPIRED", "approval %s is not valid at %s".formatted(approval.ticket(), now)); } + if (!approval.grant().topologyVersion().equals(currentTopologyVersion)) { + throw new MessageAuthorizationException( + "TOPOLOGY_CHANGED_SINCE_APPROVAL", + "approval %s was signed against topology %s but the broker is now at %s" + .formatted( + approval.ticket(), approval.grant().topologyVersion(), currentTopologyVersion)); + } if (!plan.topologyVersion().equals(currentTopologyVersion)) { throw new MessageAuthorizationException( "TOPOLOGY_CHANGED_SINCE_APPROVAL", diff --git a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/ApprovedReplayPlan.java b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/ApprovedReplayPlan.java index d438e49..54b4afe 100644 --- a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/ApprovedReplayPlan.java +++ b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/ApprovedReplayPlan.java @@ -14,11 +14,41 @@ import java.util.Objects; * @param plan the plan that was approved * @param approval the approval authorising it */ -public record ApprovedReplayPlan(ReplayPlan plan, AdminApproval approval) { +public record ApprovedReplayPlan(ReplayPlan plan, VerifiedApproval approval) { + /** Binds the approval to this exact plan at construction. */ public ApprovedReplayPlan { Objects.requireNonNull(plan, "plan must not be null"); Objects.requireNonNull(approval, "approval must not be null"); + if (!approval.grant().planDigest().equals(plan.digest())) { + // A genuine, verified approval for a different plan. Checking it here means the mismatch + // cannot survive to the execute call under any code path. + throw new MessageAuthorizationException( + "APPROVAL_PLAN_MISMATCH", + "approval %s was issued for a different plan than the one it is attached to" + .formatted(approval.ticket())); + } + if (approval.grant().operation() != DestructiveOperation.REPLAY) { + throw new MessageAuthorizationException( + "APPROVAL_OPERATION_MISMATCH", + "approval %s authorises %s, not a replay" + .formatted(approval.ticket(), approval.grant().operation())); + } + if (!approval.grant().source().equals(plan.request().destination())) { + throw new MessageAuthorizationException( + "APPROVAL_SOURCE_MISMATCH", + "approval %s authorises %s, not %s" + .formatted( + approval.ticket(), + approval.grant().source().value(), + plan.request().destination().value())); + } + if (plan.estimatedMessages() > approval.grant().maxImpact()) { + throw new MessageAuthorizationException( + "APPROVAL_IMPACT_EXCEEDED", + "the plan moves about %d messages but the approver agreed to at most %d" + .formatted(plan.estimatedMessages(), approval.grant().maxImpact())); + } } /** @@ -36,6 +66,13 @@ public record ApprovedReplayPlan(ReplayPlan plan, AdminApproval approval) { throw new MessageAuthorizationException( "APPROVAL_EXPIRED", "approval %s is not valid at %s".formatted(approval.ticket(), now)); } + if (!approval.grant().topologyVersion().equals(currentTopologyVersion)) { + throw new MessageAuthorizationException( + "TOPOLOGY_CHANGED_SINCE_APPROVAL", + "approval %s was signed against topology %s but the broker is now at %s" + .formatted( + approval.ticket(), approval.grant().topologyVersion(), currentTopologyVersion)); + } if (!plan.topologyVersion().equals(currentTopologyVersion)) { // Every number in the plan was computed against the old topology, so the approver agreed to // an impact estimate that no longer describes what would happen. diff --git a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/DestructiveOperationGuard.java b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/DestructiveOperationGuard.java index 3acd854..edeb82f 100644 --- a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/DestructiveOperationGuard.java +++ b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/DestructiveOperationGuard.java @@ -9,7 +9,7 @@ import java.util.Optional; /** * The single gate every destructive messaging operation passes through. * - *

Three conditions, all required. The caller must hold the admin credential — an application + *

Four conditions, all required. The caller must hold the admin credential — an application * runtime does not, by construction. The approval must be present and inside its validity window. * And a dry run is always permitted, because the way to make operators plan before they act is to * make planning free. @@ -35,7 +35,7 @@ public final class DestructiveOperationGuard { * * @param operation the operation * @param destination the destination affected - * @param approval the approval, when one was supplied + * @param approval the verified approval, when one was supplied * @param dryRun whether this is a plan-only run * @param now the current instant * @throws MessageAuthorizationException when the operation is not authorised @@ -43,7 +43,7 @@ public final class DestructiveOperationGuard { public void authorize( DestructiveOperation operation, DestinationName destination, - Optional approval, + Optional approval, boolean dryRun, Instant now) { Objects.requireNonNull(operation, "operation must not be null"); @@ -59,7 +59,7 @@ public final class DestructiveOperationGuard { "ADMIN_CREDENTIAL_REQUIRED", operation + " on " + destination.value() + " requires the admin credential"); } - AdminApproval granted = + VerifiedApproval granted = approval.orElseThrow( () -> new MessageAuthorizationException( @@ -69,5 +69,27 @@ public final class DestructiveOperationGuard { throw new MessageAuthorizationException( "APPROVAL_EXPIRED", "approval " + granted.ticket() + " is outside its validity window"); } + if (granted.grant().operation() != operation) { + // The approval verified — for something else. A guard that only checks presence and window + // lets a verified redrive approval authorise a destination deletion. + throw new MessageAuthorizationException( + "APPROVAL_OPERATION_MISMATCH", + "approval " + + granted.ticket() + + " authorises " + + granted.grant().operation() + + ", not " + + operation); + } + if (!granted.grant().source().equals(destination)) { + throw new MessageAuthorizationException( + "APPROVAL_SOURCE_MISMATCH", + "approval " + + granted.ticket() + + " authorises " + + granted.grant().source().value() + + ", not " + + destination.value()); + } } } diff --git a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/HmacApprovalVerifier.java b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/HmacApprovalVerifier.java new file mode 100644 index 0000000..44b19aa --- /dev/null +++ b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/HmacApprovalVerifier.java @@ -0,0 +1,111 @@ +package dev.caskeleton.messaging.admin; + +import dev.caskeleton.messaging.api.error.MessageAuthorizationException; +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.time.Instant; +import java.util.HexFormat; +import java.util.Objects; +import javax.crypto.Mac; +import javax.crypto.spec.SecretKeySpec; + +/** + * Verifies grants with a shared HMAC-SHA256 issuing key. + * + *

A shared secret rather than a public-key signature is a deliberate scope choice: the issuer is + * the change-management system operated by the same organisation as the runtime, so key + * distribution is already solved and the property that matters — that an operator cannot mint an + * approval for themselves — holds as long as the key is not on the operator's machine. A public-key + * verifier can be substituted by implementing {@link ApprovalVerifier}; nothing else in the + * platform depends on how the signature is produced. + * + *

Comparison is constant-time. A verifier that compares with {@code equals} leaks the signature + * one byte at a time to a caller who can retry, which for an offline attacker with unlimited + * attempts is the whole secret. + */ +public final class HmacApprovalVerifier implements ApprovalVerifier { + + private static final String ALGORITHM = "HmacSHA256"; + + private final byte[] issuingKey; + + /** + * Creates a verifier over the issuing key. + * + * @param issuingKey the shared secret the approval issuer signs with + */ + public HmacApprovalVerifier(byte[] issuingKey) { + Objects.requireNonNull(issuingKey, "issuingKey must not be null"); + if (issuingKey.length < 32) { + throw new IllegalArgumentException( + "the approval issuing key must be at least 256 bits; a shorter key makes the signature" + + " brute-forceable offline"); + } + this.issuingKey = issuingKey.clone(); + } + + /** + * Produces the signature an issuer would attach to a grant. + * + *

Present because the issuer and the verifier must agree on the canonical form exactly, and a + * second implementation of that agreement is a second thing to drift. Holding this key is what + * makes a caller an issuer — it is not, and must not become, available to the runtime that + * executes operations. + * + * @param grant the grant to sign + * @return the hexadecimal signature + */ + public String sign(ApprovalGrant grant) { + Objects.requireNonNull(grant, "grant must not be null"); + return HexFormat.of().formatHex(mac(grant.canonicalForm())); + } + + @Override + public VerifiedApproval verify( + ApprovalGrant grant, String signature, PlanDigest executingDigest, Instant now) { + Objects.requireNonNull(grant, "grant must not be null"); + Objects.requireNonNull(signature, "signature must not be null"); + Objects.requireNonNull(executingDigest, "executingDigest must not be null"); + Objects.requireNonNull(now, "now must not be null"); + + if (!grant.planDigest().equals(executingDigest)) { + // The signature would verify: this is a genuine approval, for a different plan. Replaying one + // approval across plans is the attack the digest binding exists to stop. + throw new MessageAuthorizationException( + "APPROVAL_PLAN_MISMATCH", + "approval %s was issued for plan %s but the plan about to execute is %s" + .formatted( + grant.approval().ticket(), grant.planDigest().value(), executingDigest.value())); + } + if (!grant.isValidAt(now)) { + throw new MessageAuthorizationException( + "APPROVAL_EXPIRED", + "approval %s is not valid at %s".formatted(grant.approval().ticket(), now)); + } + byte[] presented; + try { + presented = HexFormat.of().parseHex(signature); + } catch (IllegalArgumentException malformed) { + throw new MessageAuthorizationException( + "APPROVAL_SIGNATURE_INVALID", "the approval signature is not hexadecimal"); + } + if (!MessageDigest.isEqual(mac(grant.canonicalForm()), presented)) { + throw new MessageAuthorizationException( + "APPROVAL_SIGNATURE_INVALID", + ("approval %s does not verify against the issuing key; the grant was altered after it" + + " was signed, or it was never issued") + .formatted(grant.approval().ticket())); + } + return VerifiedApproval.of(grant); + } + + private byte[] mac(String canonicalForm) { + try { + Mac mac = Mac.getInstance(ALGORITHM); + mac.init(new SecretKeySpec(issuingKey, ALGORITHM)); + return mac.doFinal(canonicalForm.getBytes(StandardCharsets.UTF_8)); + } catch (java.security.GeneralSecurityException failure) { + throw new IllegalStateException("HMAC-SHA256 is required of every Java runtime", failure); + } + } +} diff --git a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/PlanDigest.java b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/PlanDigest.java new file mode 100644 index 0000000..d11a4d1 --- /dev/null +++ b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/PlanDigest.java @@ -0,0 +1,47 @@ +package dev.caskeleton.messaging.admin; + +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.util.HexFormat; +import java.util.Objects; + +/** + * The fingerprint of exactly what an approver agreed to. + * + *

An approval used to name a ticket and a window and nothing else, so it authorised any plan + * presented within that window. The operator who got a redrive of one dead-letter destination + * approved could execute a redrive of a different one with the same ticket, and every audit record + * would look correct. Binding the signature to this digest is what makes an approval about a plan + * rather than about a period of time. + * + * @param value the lowercase hexadecimal SHA-256 of the plan's canonical form + */ +public record PlanDigest(String value) { + + /** Validates the digest shape. */ + public PlanDigest { + Objects.requireNonNull(value, "value must not be null"); + if (!value.matches("[0-9a-f]{64}")) { + throw new IllegalArgumentException("a plan digest is 64 lowercase hexadecimal characters"); + } + } + + /** + * Digests a plan's canonical form. + * + * @param canonicalForm the plan rendered deterministically, field by field + * @return the digest + */ + public static PlanDigest ofCanonical(String canonicalForm) { + Objects.requireNonNull(canonicalForm, "canonicalForm must not be null"); + try { + byte[] digest = + MessageDigest.getInstance("SHA-256") + .digest(canonicalForm.getBytes(StandardCharsets.UTF_8)); + return new PlanDigest(HexFormat.of().formatHex(digest)); + } catch (NoSuchAlgorithmException impossible) { + throw new IllegalStateException("SHA-256 is required of every Java runtime", impossible); + } + } +} diff --git a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/RedrivePlan.java b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/RedrivePlan.java index 3f605bf..d462fb2 100644 --- a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/RedrivePlan.java +++ b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/RedrivePlan.java @@ -39,6 +39,28 @@ public record RedrivePlan( } } + /** + * The fingerprint an approver's signature is bound to. + * + *

Source, target, batch size, and both candidate counts are in it, so an approval for a + * redrive of one dead-letter destination cannot be presented for a redrive of another. + * + * @return the digest of this exact plan + */ + public PlanDigest digest() { + return PlanDigest.ofCanonical( + String.join( + "|", + "REDRIVE", + request.redriveId().toString(), + request.source().value(), + request.target().value(), + Integer.toString(request.batchSize()), + Integer.toString(candidates), + Integer.toString(alreadyRedrivenCandidates), + topologyVersion)); + } + /** * Reports whether this redrive would replay messages that already failed a redrive. * diff --git a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/ReplayPlan.java b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/ReplayPlan.java index de24065..4e6cb57 100644 --- a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/ReplayPlan.java +++ b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/ReplayPlan.java @@ -39,6 +39,30 @@ public record ReplayPlan( } } + /** + * The fingerprint an approver's signature is bound to. + * + *

Every field that changes what would happen is in it. An approval signed over this digest + * cannot be presented for a plan that reads a different destination, targets a different consumer + * group, starts from a different position, or estimates a different impact. + * + * @return the digest of this exact plan + */ + public PlanDigest digest() { + return PlanDigest.ofCanonical( + String.join( + "|", + "REPLAY", + request.replayId().toString(), + request.destination().value(), + Boolean.toString(request.isolatedConsumerGroup()), + request.from().toString(), + request.to().map(Object::toString).orElse(""), + Long.toString(estimatedMessages), + topologyVersion, + Boolean.toString(targetsLiveConsumerGroup))); + } + /** * Returns a one-line operator-facing summary of the impact. * diff --git a/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/VerifiedApproval.java b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/VerifiedApproval.java new file mode 100644 index 0000000..3702808 --- /dev/null +++ b/messaging/messaging-admin-api/src/main/java/dev/caskeleton/messaging/admin/VerifiedApproval.java @@ -0,0 +1,88 @@ +package dev.caskeleton.messaging.admin; + +import java.time.Instant; +import java.util.Objects; + +/** + * A grant that an {@link ApprovalVerifier} has checked, and that nobody else can manufacture. + * + *

The approved-plan types used to hold a plain {@code AdminApproval} record with a public + * constructor, so "this plan was approved" was a claim the caller made about itself. Any code that + * could reach the execute method could write {@code new AdminApproval("TICKET-1", "someone", now, + * later)} and the platform believed it. + * + *

This class has no public constructor and no public factory. The only way to obtain one is to + * present a grant and a signature to a verifier, which is exactly the property the type is for: + * holding a {@code VerifiedApproval} is itself the evidence that verification happened. + */ +public final class VerifiedApproval { + + private final ApprovalGrant grant; + + private VerifiedApproval(ApprovalGrant grant) { + this.grant = Objects.requireNonNull(grant, "grant must not be null"); + } + + /** + * Mints a verified approval. Package-private on purpose: only a verifier in this package may call + * it, so the type cannot be forged from application or adapter code. + */ + static VerifiedApproval of(ApprovalGrant grant) { + return new VerifiedApproval(grant); + } + + /** + * The grant that was verified. + * + * @return the grant + */ + public ApprovalGrant grant() { + return grant; + } + + /** + * The approval ticket, for audit records and journal keys. + * + * @return the ticket + */ + public String ticket() { + return grant.approval().ticket(); + } + + /** + * Who approved the operation. + * + * @return the approver identity + */ + public String approvedBy() { + return grant.approval().approvedBy(); + } + + /** + * Reports whether the approval window is still open. + * + * @param now the current instant + * @return true while inside the window + */ + public boolean isValidAt(Instant now) { + return grant.isValidAt(now); + } + + @Override + public boolean equals(Object other) { + return other instanceof VerifiedApproval verified && grant.equals(verified.grant); + } + + @Override + public int hashCode() { + return grant.hashCode(); + } + + @Override + public String toString() { + // Deliberately not the whole grant: this string reaches logs, and the plan digest plus the + // ticket identify the operation without restating who approved what to an audience that has + // not been authorised to read it. + return "VerifiedApproval[" + ticket() + " " + grant.planDigest().value().substring(0, 12) + "]"; + } +} diff --git a/messaging/messaging-admin-api/src/test/java/dev/caskeleton/messaging/admin/DestructiveOperationGuardTest.java b/messaging/messaging-admin-api/src/test/java/dev/caskeleton/messaging/admin/DestructiveOperationGuardTest.java index f957ede..cd9f8cd 100644 --- a/messaging/messaging-admin-api/src/test/java/dev/caskeleton/messaging/admin/DestructiveOperationGuardTest.java +++ b/messaging/messaging-admin-api/src/test/java/dev/caskeleton/messaging/admin/DestructiveOperationGuardTest.java @@ -17,8 +17,35 @@ class DestructiveOperationGuardTest { private static final Instant NOW = Instant.parse("2026-08-10T09:15:00Z"); private static final DestinationName ORDERS = new DestinationName("order-events"); - private static final AdminApproval VALID = - new AdminApproval("CHG-1001", "operator", NOW.minusSeconds(60), NOW.plusSeconds(3600)); + private static final HmacApprovalVerifier ISSUER = + new HmacApprovalVerifier( + "an-issuing-key-of-at-least-32-bytes!".getBytes(java.nio.charset.StandardCharsets.UTF_8)); + + private static final VerifiedApproval VALID = + verified(DestructiveOperation.REPLAY, ORDERS, NOW.minusSeconds(60), NOW.plusSeconds(3600)); + + /** + * A grant the issuer signed, verified the way the runtime verifies it. + * + *

The guard used to take a plain {@code AdminApproval} record, which any caller could + * construct. Every case below now has to go through a real signature to get an approval object at + * all, which is the property the type change was made for. + */ + private static VerifiedApproval verified( + DestructiveOperation operation, DestinationName source, Instant from, Instant until) { + PlanDigest digest = PlanDigest.ofCanonical(operation + "|" + source.value()); + ApprovalGrant grant = + new ApprovalGrant( + new AdminApproval("CHG-1001", "approver", from, until), + "operator", + operation, + source, + Optional.empty(), + "v1", + Long.MAX_VALUE, + digest); + return ISSUER.verify(grant, ISSUER.sign(grant), digest, from); + } @Test void anApplicationRuntimeCannotRedrive() { @@ -45,8 +72,12 @@ class DestructiveOperationGuardTest { @Test void anExpiredApprovalDoesNotAuthorise() { DestructiveOperationGuard guard = new DestructiveOperationGuard(true); - AdminApproval expired = - new AdminApproval("CHG-1000", "operator", NOW.minusSeconds(7200), NOW.minusSeconds(60)); + VerifiedApproval expired = + verified( + DestructiveOperation.OFFSET_RESET, + ORDERS, + NOW.minusSeconds(7200), + NOW.minusSeconds(60)); assertThatThrownBy( () -> diff --git a/messaging/messaging-admin-runtime/gradle.lockfile b/messaging/messaging-admin-runtime/gradle.lockfile index ae1a5dc..8a9cedb 100644 --- a/messaging/messaging-admin-runtime/gradle.lockfile +++ b/messaging/messaging-admin-runtime/gradle.lockfile @@ -32,9 +32,9 @@ commons-io:commons-io:2.21.0=spotbugs info.picocli:picocli:4.7.7=checkstyle io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnotationProcessor io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor -io.micrometer:micrometer-commons:1.16.0=runtimeClasspath,testRuntimeClasspath -io.micrometer:micrometer-core:1.16.0=runtimeClasspath,testRuntimeClasspath -io.micrometer:micrometer-observation:1.16.0=runtimeClasspath,testRuntimeClasspath +io.micrometer:micrometer-commons:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.micrometer:micrometer-core:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.micrometer:micrometer-observation:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath javax.inject:javax.inject:1=annotationProcessor,testAnnotationProcessor jaxen:jaxen:2.0.0=spotbugs net.bytebuddy:byte-buddy:1.17.8=testCompileClasspath,testRuntimeClasspath @@ -62,7 +62,7 @@ org.codehaus.plexus:plexus-utils:3.3.0=checkstyle org.dom4j:dom4j:2.2.0=spotbugs org.hdrhistogram:HdrHistogram:2.2.2=runtimeClasspath,testRuntimeClasspath org.javassist:javassist:3.28.0-GA=checkstyle -org.jspecify:jspecify:1.0.0=annotationProcessor,checkstyle,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath +org.jspecify:jspecify:1.0.0=annotationProcessor,checkstyle,compileClasspath,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath org.junit.jupiter:junit-jupiter-api:6.0.1=testCompileClasspath,testRuntimeClasspath org.junit.jupiter:junit-jupiter-engine:6.0.1=testRuntimeClasspath org.junit.jupiter:junit-jupiter-params:6.0.1=testCompileClasspath,testRuntimeClasspath @@ -85,4 +85,4 @@ org.reflections:reflections:0.10.2=checkstyle org.slf4j:slf4j-api:2.0.17=spotbugs,spotbugsSlf4j org.slf4j:slf4j-simple:2.0.17=checkstyle,spotbugsSlf4j org.xmlresolver:xmlresolver:5.3.3=checkstyle,spotbugs -empty=compileClasspath +empty= diff --git a/messaging/messaging-admin-runtime/src/main/java/dev/caskeleton/messaging/admin/runtime/AdminOperationIdempotencyStore.java b/messaging/messaging-admin-runtime/src/main/java/dev/caskeleton/messaging/admin/runtime/AdminOperationIdempotencyStore.java deleted file mode 100644 index 29410b3..0000000 --- a/messaging/messaging-admin-runtime/src/main/java/dev/caskeleton/messaging/admin/runtime/AdminOperationIdempotencyStore.java +++ /dev/null @@ -1,78 +0,0 @@ -package dev.caskeleton.messaging.admin.runtime; - -import java.time.Instant; -import java.util.Map; -import java.util.Objects; -import java.util.Optional; -import java.util.concurrent.ConcurrentHashMap; - -/** - * Remembers which approvals have already been executed. - * - *

An approval authorises one execution, not a standing permission. Without this, re-running the - * same approved redrive twice is a single command-history arrow-up away — and the second run - * republishes messages the first one already moved, which on a destination without an inbox is - * indistinguishable from a duplicate storm. - * - *

Claiming is atomic and returns the previous claim rather than a boolean, so a duplicate - * attempt can tell the operator when it ran and by which operation id instead of just - * refusing. - */ -public final class AdminOperationIdempotencyStore { - - private final Map claims = new ConcurrentHashMap<>(); - - /** - * One recorded execution of an approval. - * - * @param approvalTicket the approval that was executed - * @param operationId the operation identity that claimed it - * @param executedAt when it ran - */ - public record Claim(String approvalTicket, String operationId, Instant executedAt) { - - public Claim { - Objects.requireNonNull(executedAt, "executedAt must not be null"); - if (approvalTicket == null || approvalTicket.isBlank()) { - throw new IllegalArgumentException("approvalTicket must not be blank"); - } - if (operationId == null || operationId.isBlank()) { - throw new IllegalArgumentException("operationId must not be blank"); - } - } - } - - /** - * Claims an approval for execution. - * - * @param approvalTicket the approval to claim - * @param operationId the operation attempting it - * @param now the current instant - * @return empty when the claim succeeded; the existing claim when it was already executed - */ - public Optional claim(String approvalTicket, String operationId, Instant now) { - Objects.requireNonNull(now, "now must not be null"); - Claim candidate = new Claim(approvalTicket, operationId, now); - Claim existing = claims.putIfAbsent(approvalTicket, candidate); - return Optional.ofNullable(existing); - } - - /** - * Returns the recorded execution of an approval. - * - * @param approvalTicket the approval - * @return the claim, when the approval has been executed - */ - public Optional find(String approvalTicket) { - return Optional.ofNullable(claims.get(approvalTicket)); - } - - /** - * Returns how many approvals have been executed. - * - * @return the claim count - */ - public int size() { - return claims.size(); - } -} diff --git a/messaging/messaging-admin-runtime/src/main/java/dev/caskeleton/messaging/admin/runtime/DefaultMessagingAdminService.java b/messaging/messaging-admin-runtime/src/main/java/dev/caskeleton/messaging/admin/runtime/DefaultMessagingAdminService.java index 2c6df02..1efded5 100644 --- a/messaging/messaging-admin-runtime/src/main/java/dev/caskeleton/messaging/admin/runtime/DefaultMessagingAdminService.java +++ b/messaging/messaging-admin-runtime/src/main/java/dev/caskeleton/messaging/admin/runtime/DefaultMessagingAdminService.java @@ -1,5 +1,7 @@ package dev.caskeleton.messaging.admin.runtime; +import dev.caskeleton.messaging.admin.AdminOperationJournal; +import dev.caskeleton.messaging.admin.AdminOperationLease; import dev.caskeleton.messaging.admin.ApprovedRedrivePlan; import dev.caskeleton.messaging.admin.ApprovedReplayPlan; import dev.caskeleton.messaging.admin.RedrivePlan; @@ -10,7 +12,6 @@ import dev.caskeleton.messaging.admin.ReplayRequest; import dev.caskeleton.messaging.admin.ReplayResult; import dev.caskeleton.messaging.admin.TopologyManifest; import dev.caskeleton.messaging.admin.TopologyValidationReport; -import dev.caskeleton.messaging.api.error.MessageAuthorizationException; import java.time.Duration; import java.time.Instant; import java.util.List; @@ -30,15 +31,29 @@ import java.util.function.Supplier; *

  • Only then does anything move. * * - *

    The idempotency claim comes before the work rather than after it. Claiming afterwards - * leaves a window where a second execution starts while the first is still running, which is - * precisely the double-redrive this store exists to prevent. + *

    The journal entry is written before the work rather than after it. Writing it + * afterwards leaves a window where a second execution starts while the first is still running, + * which is precisely the double-redrive the journal exists to prevent. + * + *

    Writing it first used to have a cost the previous store never paid: an operation that died + * halfway had consumed its approval and left no record of how far it got. The journal keeps a + * checkpoint and hands back a lease that says where to resume, so a retry continues the same + * operation instead of either redoing it or requiring a new approval. */ public final class DefaultMessagingAdminService implements MessagingAdminService { + /** + * How long a runtime holds an operation before another may take it over. + * + *

    Long enough that a slow batch does not lose its lease mid-flight, short enough that a dead + * replica does not park an approval for an hour. + */ + private static final Duration LEASE_DURATION = Duration.ofMinutes(5); + private final CompositeTopologyValidator topologyValidator; private final BrokerTopologyInspector inspector; - private final AdminOperationIdempotencyStore idempotency; + private final AdminOperationJournal journal; + private final String leaseOwner; private final ReplayService replayService; private final RedriveService redriveService; private final Supplier> manifests; @@ -51,7 +66,8 @@ public final class DefaultMessagingAdminService implements MessagingAdminService * * @param topologyValidator compares declared topology against the broker * @param inspector reads the broker's topology version - * @param idempotency records which approvals have been executed + * @param journal records which approvals have been executed, and how far each got + * @param leaseOwner identifies this runtime in the journal's leases * @param replayService performs replays * @param redriveService performs redrives * @param manifests supplies the declared topologies @@ -62,7 +78,8 @@ public final class DefaultMessagingAdminService implements MessagingAdminService public DefaultMessagingAdminService( CompositeTopologyValidator topologyValidator, BrokerTopologyInspector inspector, - AdminOperationIdempotencyStore idempotency, + AdminOperationJournal journal, + String leaseOwner, ReplayService replayService, RedriveService redriveService, Supplier> manifests, @@ -72,7 +89,11 @@ public final class DefaultMessagingAdminService implements MessagingAdminService this.topologyValidator = Objects.requireNonNull(topologyValidator, "topologyValidator required"); this.inspector = Objects.requireNonNull(inspector, "inspector must not be null"); - this.idempotency = Objects.requireNonNull(idempotency, "idempotency must not be null"); + this.journal = Objects.requireNonNull(journal, "journal must not be null"); + if (leaseOwner == null || leaseOwner.isBlank()) { + throw new IllegalArgumentException("leaseOwner must not be blank"); + } + this.leaseOwner = leaseOwner; this.replayService = Objects.requireNonNull(replayService, "replayService must not be null"); this.redriveService = Objects.requireNonNull(redriveService, "redriveService must not be null"); this.manifests = Objects.requireNonNull(manifests, "manifests must not be null"); @@ -104,12 +125,28 @@ public final class DefaultMessagingAdminService implements MessagingAdminService ReplayRequest request = plan.plan().request(); plan.requireExecutable(now, inspector.topologyVersion()); - claimOrRefuse(plan.approval().ticket(), request.replayId().toString(), now); + AdminOperationLease lease = + journal.begin( + plan.approval().ticket(), + plan.plan().digest(), + request.replayId().toString(), + leaseOwner, + LEASE_DURATION, + now); Instant startedAt = clock.get(); - ReplayReport report = - replayService.replay( - request, Optional.of(plan.approval()), plan.approval().approvedBy(), now); + ReplayReport report; + try { + report = + replayService.replay( + request, Optional.of(plan.approval()), plan.approval().approvedBy(), now); + } catch (RuntimeException failure) { + // The operation stays resumable rather than silently consuming the approval: the journal + // entry moves to FAILED at its checkpoint, and a retry takes it over from there. + journal.fail(lease, lease.resumeFrom(), failureCodeOf(failure), clock.get()); + throw failure; + } + journal.complete(lease, Math.toIntExact(report.replayed()), clock.get()); return new ReplayResult( request.replayId(), @@ -139,12 +176,31 @@ public final class DefaultMessagingAdminService implements MessagingAdminService RedriveRequest request = plan.plan().request(); plan.requireExecutable(now, inspector.topologyVersion()); - claimOrRefuse(plan.approval().ticket(), request.redriveId().toString(), now); + AdminOperationLease lease = + journal.begin( + plan.approval().ticket(), + plan.plan().digest(), + request.redriveId().toString(), + leaseOwner, + LEASE_DURATION, + now); Instant startedAt = clock.get(); - RedriveReport report = - redriveService.redrive( - request, Optional.of(plan.approval()), plan.approval().approvedBy(), now); + RedriveReport report; + try { + report = + redriveService.redrive( + request, + Optional.of(plan.approval()), + plan.approval().approvedBy(), + now, + lease.resumeFrom(), + completed -> journal.checkpoint(lease, completed, LEASE_DURATION, clock.get())); + } catch (RuntimeException failure) { + journal.fail(lease, lease.resumeFrom(), failureCodeOf(failure), clock.get()); + throw failure; + } + journal.complete(lease, report.moved() + report.failed(), clock.get()); return new RedriveResult( request.redriveId(), @@ -155,17 +211,19 @@ public final class DefaultMessagingAdminService implements MessagingAdminService report.dryRun()); } - private void claimOrRefuse(String approvalTicket, String operationId, Instant now) { - idempotency - .claim(approvalTicket, operationId, now) - .ifPresent( - existing -> { - throw new MessageAuthorizationException( - "APPROVAL_ALREADY_EXECUTED", - "approval %s was already executed at %s by operation %s; an approval authorises " - .formatted(approvalTicket, existing.executedAt(), existing.operationId()) - + "one execution, not a standing permission"); - }); + /** + * Reduces a failure to a bounded, sanitised code for the journal. + * + *

    The journal is read by operators during incidents and its contents outlive the process. A + * raw exception message can carry a destination, a payload fragment, or a credential from a + * driver's own error text, so only the platform's own code or the exception's simple name goes + * in. + */ + private static String failureCodeOf(RuntimeException failure) { + if (failure instanceof dev.caskeleton.messaging.api.error.MessagingException messaging) { + return messaging.failure().code(); + } + return failure.getClass().getSimpleName(); } /** Estimates how many messages a replay would re-deliver. */ diff --git a/messaging/messaging-admin-runtime/src/main/java/dev/caskeleton/messaging/admin/runtime/InMemoryAdminOperationJournal.java b/messaging/messaging-admin-runtime/src/main/java/dev/caskeleton/messaging/admin/runtime/InMemoryAdminOperationJournal.java new file mode 100644 index 0000000..574dfa5 --- /dev/null +++ b/messaging/messaging-admin-runtime/src/main/java/dev/caskeleton/messaging/admin/runtime/InMemoryAdminOperationJournal.java @@ -0,0 +1,237 @@ +package dev.caskeleton.messaging.admin.runtime; + +import dev.caskeleton.messaging.admin.AdminOperationJournal; +import dev.caskeleton.messaging.admin.AdminOperationLease; +import dev.caskeleton.messaging.admin.AdminOperationRecord; +import dev.caskeleton.messaging.admin.AdminOperationState; +import dev.caskeleton.messaging.admin.PlanDigest; +import dev.caskeleton.messaging.api.error.MessageAuthorizationException; +import java.time.Duration; +import java.time.Instant; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A single-process journal, for tests and for local development. + * + *

    It reports {@link #isDurable()} as false, and the starter refuses to run a production profile + * on a journal that says so. That declaration is the point of this class existing at all: the + * previous in-memory store was registered as the production default and nothing distinguished it + * from a shared one, so the gap was invisible until two replicas executed the same approval. + * + *

    The semantics are otherwise the real ones — uniqueness on {@code (ticket, digest)}, lease + * takeover with a monotonic token, resume from checkpoint — so a test that passes here is testing + * the protocol rather than a simplification of it. + */ +public final class InMemoryAdminOperationJournal implements AdminOperationJournal { + + private final Map records = new ConcurrentHashMap<>(); + + @Override + public AdminOperationLease begin( + String approvalTicket, + PlanDigest planDigest, + String operationId, + String leaseOwner, + Duration leaseDuration, + Instant now) { + Objects.requireNonNull(leaseDuration, "leaseDuration must not be null"); + Objects.requireNonNull(now, "now must not be null"); + AdminOperationRecord claimed = + records.compute( + key(approvalTicket, planDigest), + (ignored, existing) -> + claim( + existing, + approvalTicket, + planDigest, + operationId, + leaseOwner, + leaseDuration, + now)); + return new AdminOperationLease( + approvalTicket, + planDigest, + claimed.operationId(), + claimed.leaseOwner(), + claimed.leaseToken(), + claimed.itemsCompleted()); + } + + private static AdminOperationRecord claim( + AdminOperationRecord existing, + String approvalTicket, + PlanDigest planDigest, + String operationId, + String leaseOwner, + Duration leaseDuration, + Instant now) { + if (existing == null) { + return new AdminOperationRecord( + approvalTicket, + planDigest, + operationId, + AdminOperationState.STARTED, + 0, + leaseOwner, + 1L, + now.plus(leaseDuration), + now, + Optional.empty(), + Optional.empty()); + } + if (existing.state() == AdminOperationState.COMPLETED) { + throw new MessageAuthorizationException( + "APPROVAL_ALREADY_EXECUTED", + ("approval %s was already executed to completion at %s; an approval authorises one" + + " execution, not a standing permission") + .formatted(approvalTicket, existing.finishedAt().orElse(existing.startedAt()))); + } + if (existing.state() == AdminOperationState.STARTED + && now.isBefore(existing.leaseExpiresAt())) { + throw new MessageAuthorizationException( + "ADMIN_OPERATION_IN_FLIGHT", + ("approval %s is being executed by %s until at least %s; two runtimes executing one" + + " approval is a duplicate storm, not a faster redrive") + .formatted(approvalTicket, existing.leaseOwner(), existing.leaseExpiresAt())); + } + // Either the previous attempt failed, or its lease expired without a completion. Both resume + // from the checkpoint under a new token, which fences the previous holder out. + return new AdminOperationRecord( + approvalTicket, + planDigest, + operationId, + AdminOperationState.STARTED, + existing.itemsCompleted(), + leaseOwner, + existing.leaseToken() + 1, + now.plus(leaseDuration), + existing.startedAt(), + Optional.empty(), + Optional.empty()); + } + + @Override + public void checkpoint( + AdminOperationLease lease, int itemsCompleted, Duration leaseDuration, Instant now) { + update( + lease, + current -> + new AdminOperationRecord( + current.approvalTicket(), + current.planDigest(), + current.operationId(), + AdminOperationState.STARTED, + Math.max(current.itemsCompleted(), itemsCompleted), + current.leaseOwner(), + current.leaseToken(), + now.plus(leaseDuration), + current.startedAt(), + Optional.empty(), + Optional.empty())); + } + + @Override + public void complete(AdminOperationLease lease, int itemsCompleted, Instant now) { + update( + lease, + current -> + new AdminOperationRecord( + current.approvalTicket(), + current.planDigest(), + current.operationId(), + AdminOperationState.COMPLETED, + Math.max(current.itemsCompleted(), itemsCompleted), + current.leaseOwner(), + current.leaseToken(), + current.leaseExpiresAt(), + current.startedAt(), + Optional.of(now), + Optional.empty())); + } + + @Override + public void fail(AdminOperationLease lease, int itemsCompleted, String failureCode, Instant now) { + Objects.requireNonNull(failureCode, "failureCode must not be null"); + update( + lease, + current -> + new AdminOperationRecord( + current.approvalTicket(), + current.planDigest(), + current.operationId(), + AdminOperationState.FAILED, + Math.max(current.itemsCompleted(), itemsCompleted), + current.leaseOwner(), + current.leaseToken(), + current.leaseExpiresAt(), + current.startedAt(), + Optional.of(now), + Optional.of(failureCode))); + } + + @Override + public Optional find(String approvalTicket, PlanDigest planDigest) { + return Optional.ofNullable(records.get(key(approvalTicket, planDigest))); + } + + @Override + public boolean isDurable() { + return false; + } + + /** + * How many operations have been journalled. + * + * @return the record count + */ + public int size() { + return records.size(); + } + + private void update( + AdminOperationLease lease, java.util.function.UnaryOperator mutation) { + Objects.requireNonNull(lease, "lease must not be null"); + records.compute( + key(lease.approvalTicket(), lease.planDigest()), + (ignored, current) -> { + if (current == null) { + throw new MessageAuthorizationException( + "ADMIN_OPERATION_NOT_JOURNALLED", + "no journal entry exists for approval " + lease.approvalTicket()); + } + if (current.leaseToken() != lease.leaseToken()) { + // The fence. A stalled runtime that wakes up and writes here would otherwise overwrite + // the progress of whichever replica took the operation over. + throw new MessageAuthorizationException( + "ADMIN_OPERATION_LEASE_LOST", + ("the lease on approval %s moved from token %d to token %d; this runtime no longer" + + " owns the operation") + .formatted(lease.approvalTicket(), lease.leaseToken(), current.leaseToken())); + } + return mutation.apply(current); + }); + } + + private static String key(String approvalTicket, PlanDigest planDigest) { + Objects.requireNonNull(planDigest, "planDigest must not be null"); + // Length-prefixed for the same reason the grant's canonical form is: a ticket containing the + // separator must not be able to collide with a different ticket and digest pair. + return String.join( + "", Integer.toString(approvalTicket.length()), ":", approvalTicket, planDigest.value()); + } + + /** + * Every journalled operation, newest first by start time. + * + * @return the records + */ + public List records() { + return records.values().stream() + .sorted(java.util.Comparator.comparing(AdminOperationRecord::startedAt).reversed()) + .toList(); + } +} diff --git a/messaging/messaging-admin-runtime/src/main/java/dev/caskeleton/messaging/admin/runtime/RedriveService.java b/messaging/messaging-admin-runtime/src/main/java/dev/caskeleton/messaging/admin/runtime/RedriveService.java index 72de394..8079d8e 100644 --- a/messaging/messaging-admin-runtime/src/main/java/dev/caskeleton/messaging/admin/runtime/RedriveService.java +++ b/messaging/messaging-admin-runtime/src/main/java/dev/caskeleton/messaging/admin/runtime/RedriveService.java @@ -1,9 +1,9 @@ package dev.caskeleton.messaging.admin.runtime; -import dev.caskeleton.messaging.admin.AdminApproval; import dev.caskeleton.messaging.admin.DestructiveOperation; import dev.caskeleton.messaging.admin.DestructiveOperationGuard; import dev.caskeleton.messaging.admin.RedriveRequest; +import dev.caskeleton.messaging.admin.VerifiedApproval; import dev.caskeleton.messaging.api.MessageId; import dev.caskeleton.messaging.api.publish.PublishCompletion; import dev.caskeleton.messaging.api.publish.PublishResult; @@ -54,14 +54,46 @@ public final class RedriveService { * Runs one redrive pass. * * @param request what to move - * @param approval the approval, when one was supplied + * @param approval the verified approval, when one was supplied * @param subject the operator identity * @param now the current instant * @return what the pass did */ public RedriveReport redrive( - RedriveRequest request, Optional approval, String subject, Instant now) { + RedriveRequest request, Optional approval, String subject, Instant now) { + return redrive(request, approval, subject, now, 0, completed -> {}); + } + + /** + * Runs one redrive pass, resuming a previous attempt and reporting progress as it goes. + * + *

    Three things were wrong with running this as a plain loop. A synchronous failure from the + * publisher — a broker that refuses the connection rather than the message — propagated out of + * the loop, so the remaining candidates were never attempted and the audit record was never + * written: the operation left no trace of the items it had already moved. A retry then started + * from the first candidate and republished them. And nothing bounded how many times one message + * could be redriven. + * + * @param request what to move + * @param approval the verified approval, when one was supplied + * @param subject the operator identity + * @param now the current instant + * @param resumeFrom how many candidates a previous attempt durably moved + * @param checkpoint called with the running completed count after each item + * @return what the pass did + */ + public RedriveReport redrive( + RedriveRequest request, + Optional approval, + String subject, + Instant now, + int resumeFrom, + java.util.function.IntConsumer checkpoint) { Objects.requireNonNull(request, "request must not be null"); + Objects.requireNonNull(checkpoint, "checkpoint must not be null"); + if (resumeFrom < 0) { + throw new IllegalArgumentException("resumeFrom must not be negative"); + } guard.authorize( DestructiveOperation.REDRIVE, request.source(), approval, request.dryRun(), now); @@ -72,31 +104,57 @@ public final class RedriveService { List moved = new ArrayList<>(); int failed = 0; - for (MessageId messageId : candidates) { - PublishResult result = publisher.republish(messageId, request.target(), request.redriveId()); - if (result.completion() == PublishCompletion.CONFIRMED) { - source.settle(request.source(), messageId); - moved.add(messageId); - } else { - failed++; + int completed = resumeFrom; + try { + // Everything before resumeFrom was moved and settled by the previous attempt. Re-publishing + // it is not a retry, it is a duplicate. + for (MessageId messageId : + candidates.subList(Math.min(resumeFrom, candidates.size()), candidates.size())) { + if (attempt(messageId, request)) { + moved.add(messageId); + } else { + failed++; + } + completed++; + checkpoint.accept(completed); } + } finally { + // In the finally block on purpose: an operation that dies partway must still leave a record + // of what it moved, because that record is what the resumed attempt and the incident review + // both read. + audit.record( + new dev.caskeleton.messaging.observation.MessagingAuditEvent( + "REDRIVE", + subject, + request.source().value(), + approval.map(VerifiedApproval::ticket).orElse("dry-run"), + now, + java.util.Map.of( + "redriveId", request.redriveId().toString(), + "resumedFrom", Integer.toString(resumeFrom), + "moved", Integer.toString(moved.size()), + "failed", Integer.toString(failed)))); } - audit.record( - new dev.caskeleton.messaging.observation.MessagingAuditEvent( - "REDRIVE", - subject, - request.source().value(), - approval.map(AdminApproval::ticket).orElse("dry-run"), - now, - java.util.Map.of( - "redriveId", request.redriveId().toString(), - "moved", Integer.toString(moved.size()), - "failed", Integer.toString(failed)))); - return new RedriveReport(candidates.size(), moved.size(), failed, false); } + private boolean attempt(MessageId messageId, RedriveRequest request) { + PublishResult result; + try { + result = publisher.republish(messageId, request.target(), request.redriveId()); + } catch (RuntimeException failure) { + // One message that cannot be republished is a failed item, not a failed pass. Letting it + // propagate abandoned every candidate behind it. + return false; + } + if (result.completion() != PublishCompletion.CONFIRMED) { + return false; + } + source.settle(request.source(), messageId); + return true; + } + /** Reads and settles messages on a dead letter destination. */ public interface RedriveSource { diff --git a/messaging/messaging-admin-runtime/src/main/java/dev/caskeleton/messaging/admin/runtime/ReplayService.java b/messaging/messaging-admin-runtime/src/main/java/dev/caskeleton/messaging/admin/runtime/ReplayService.java index 81063e1..8aa316d 100644 --- a/messaging/messaging-admin-runtime/src/main/java/dev/caskeleton/messaging/admin/runtime/ReplayService.java +++ b/messaging/messaging-admin-runtime/src/main/java/dev/caskeleton/messaging/admin/runtime/ReplayService.java @@ -1,9 +1,9 @@ package dev.caskeleton.messaging.admin.runtime; -import dev.caskeleton.messaging.admin.AdminApproval; import dev.caskeleton.messaging.admin.DestructiveOperation; import dev.caskeleton.messaging.admin.DestructiveOperationGuard; import dev.caskeleton.messaging.admin.ReplayRequest; +import dev.caskeleton.messaging.admin.VerifiedApproval; import dev.caskeleton.messaging.observation.MessagingAuditEvent; import java.time.Instant; import java.util.Map; @@ -45,13 +45,13 @@ public final class ReplayService { * Runs one replay. * * @param request what to replay - * @param approval the approval, when one was supplied + * @param approval the verified approval, when one was supplied * @param subject the operator identity * @param now the current instant * @return what the replay did */ public ReplayReport replay( - ReplayRequest request, Optional approval, String subject, Instant now) { + ReplayRequest request, Optional approval, String subject, Instant now) { Objects.requireNonNull(request, "request must not be null"); Objects.requireNonNull(approval, "approval must not be null"); @@ -74,7 +74,7 @@ public final class ReplayService { "REPLAY", subject, request.destination().value(), - approval.map(AdminApproval::ticket).orElse("isolated"), + approval.map(VerifiedApproval::ticket).orElse("isolated"), now, Map.of( "replayId", request.replayId().toString(), diff --git a/messaging/messaging-admin-runtime/src/test/java/dev/caskeleton/messaging/admin/runtime/AdminOperationJournalTest.java b/messaging/messaging-admin-runtime/src/test/java/dev/caskeleton/messaging/admin/runtime/AdminOperationJournalTest.java new file mode 100644 index 0000000..76b5e32 --- /dev/null +++ b/messaging/messaging-admin-runtime/src/test/java/dev/caskeleton/messaging/admin/runtime/AdminOperationJournalTest.java @@ -0,0 +1,145 @@ +package dev.caskeleton.messaging.admin.runtime; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatCode; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import dev.caskeleton.messaging.admin.AdminOperationLease; +import dev.caskeleton.messaging.admin.AdminOperationState; +import dev.caskeleton.messaging.admin.PlanDigest; +import dev.caskeleton.messaging.api.error.MessageAuthorizationException; +import java.time.Duration; +import java.time.Instant; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** + * The protocol the journal has to implement, proven against the in-memory one. + * + *

    What it replaced recorded a single fact — "this approval was claimed" — before any work + * happened, in a map. An operation that died halfway had spent its approval, left no record of what + * it had already moved, and gave a retry no way to continue: either request a new approval and + * republish everything, or leave the redrive half applied. + */ +class AdminOperationJournalTest { + + private static final Instant NOW = Instant.parse("2026-08-10T09:00:00Z"); + private static final Duration LEASE = Duration.ofMinutes(5); + private static final String TICKET = "CHG-1042"; + private static final PlanDigest DIGEST = PlanDigest.ofCanonical("REDRIVE|orders.v1.dlq|50"); + private static final PlanDigest OTHER_DIGEST = PlanDigest.ofCanonical("REDRIVE|payments.dlq|50"); + + private final InMemoryAdminOperationJournal journal = new InMemoryAdminOperationJournal(); + + @Test + @DisplayName("two replicas racing for one approval produce exactly one lease") + void twoReplicasRacingProduceExactlyOneLease() { + journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW); + + assertThatThrownBy( + () -> journal.begin(TICKET, DIGEST, "op-2", "replica-b", LEASE, NOW.plusSeconds(1))) + .as("both replicas executing one approval is a duplicate storm, not a faster redrive") + .isInstanceOf(MessageAuthorizationException.class) + .hasMessageContaining("replica-a"); + } + + @Test + @DisplayName("a completed operation cannot be executed again") + void aCompletedOperationCannotBeExecutedAgain() { + AdminOperationLease lease = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW); + journal.complete(lease, 50, NOW.plusSeconds(30)); + + assertThatThrownBy( + () -> journal.begin(TICKET, DIGEST, "op-2", "replica-a", LEASE, NOW.plusSeconds(60))) + .as("an arrow-up in the shell must not re-run an approved redrive") + .isInstanceOf(MessageAuthorizationException.class) + .hasMessageContaining("already executed"); + } + + @Test + @DisplayName("a crashed run resumes from its checkpoint rather than starting over") + void aCrashedRunResumesFromItsCheckpoint() { + AdminOperationLease first = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW); + journal.checkpoint(first, 30, LEASE, NOW.plusSeconds(10)); + // No fail() call: this replica died. The lease simply runs out. + + AdminOperationLease resumed = + journal.begin(TICKET, DIGEST, "op-1", "replica-b", LEASE, NOW.plus(Duration.ofMinutes(10))); + + assertThat(resumed.resumeFrom()) + .as("the 30 already moved must not be republished") + .isEqualTo(30); + assertThat(resumed.leaseToken()).isGreaterThan(first.leaseToken()); + assertThat(resumed.isResumption()).isTrue(); + } + + @Test + @DisplayName("a failed run stays resumable without waiting for the lease to expire") + void aFailedRunStaysResumableImmediately() { + AdminOperationLease first = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW); + journal.checkpoint(first, 12, LEASE, NOW.plusSeconds(5)); + journal.fail(first, 12, "MESSAGE_BROKER_UNAVAILABLE", NOW.plusSeconds(6)); + + AdminOperationLease resumed = + journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW.plusSeconds(7)); + + assertThat(resumed.resumeFrom()).isEqualTo(12); + assertThat(journal.find(TICKET, DIGEST)) + .hasValueSatisfying( + record -> assertThat(record.state()).isEqualTo(AdminOperationState.STARTED)); + } + + @Test + @DisplayName("a runtime that lost its lease cannot write over the one that took over") + void aRuntimeThatLostItsLeaseCannotWriteOverTheSuccessor() { + AdminOperationLease stale = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW); + journal.checkpoint(stale, 30, LEASE, NOW.plusSeconds(10)); + AdminOperationLease successor = + journal.begin(TICKET, DIGEST, "op-1", "replica-b", LEASE, NOW.plus(Duration.ofMinutes(10))); + journal.checkpoint(successor, 45, LEASE, NOW.plus(Duration.ofMinutes(11))); + + assertThatThrownBy(() -> journal.complete(stale, 30, NOW.plus(Duration.ofMinutes(12)))) + .as("a stalled replica waking up must not mark the operation done at its own count") + .isInstanceOf(MessageAuthorizationException.class) + .hasMessageContaining("no longer owns"); + + assertThat(journal.find(TICKET, DIGEST)) + .hasValueSatisfying( + record -> { + assertThat(record.itemsCompleted()).isEqualTo(45); + assertThat(record.state()).isEqualTo(AdminOperationState.STARTED); + }); + } + + @Test + @DisplayName("the same ticket presented for a different plan is a different operation") + void theSameTicketForADifferentPlanIsADifferentOperation() { + journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW); + + // Keyed on the ticket alone this would have been refused as a duplicate execution, which reads + // as protection but is actually the wrong answer: it is a different plan, and it must be + // refused by the approval binding rather than silently treated as the same operation. + assertThatCode(() -> journal.begin(TICKET, OTHER_DIGEST, "op-2", "replica-a", LEASE, NOW)) + .doesNotThrowAnyException(); + assertThat(journal.size()).isEqualTo(2); + } + + @Test + @DisplayName("a checkpoint never moves backwards") + void aCheckpointNeverMovesBackwards() { + AdminOperationLease lease = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW); + journal.checkpoint(lease, 40, LEASE, NOW.plusSeconds(10)); + journal.checkpoint(lease, 5, LEASE, NOW.plusSeconds(20)); + + assertThat(journal.find(TICKET, DIGEST)) + .hasValueSatisfying(record -> assertThat(record.itemsCompleted()).isEqualTo(40)); + } + + @Test + @DisplayName("the in-memory journal declares itself non-durable") + void theInMemoryJournalDeclaresItselfNonDurable() { + assertThat(journal.isDurable()) + .as("the starter refuses a production profile on a journal that says this") + .isFalse(); + } +} diff --git a/messaging/messaging-admin-runtime/src/test/java/dev/caskeleton/messaging/admin/runtime/ApprovalForgeryTest.java b/messaging/messaging-admin-runtime/src/test/java/dev/caskeleton/messaging/admin/runtime/ApprovalForgeryTest.java new file mode 100644 index 0000000..38c8912 --- /dev/null +++ b/messaging/messaging-admin-runtime/src/test/java/dev/caskeleton/messaging/admin/runtime/ApprovalForgeryTest.java @@ -0,0 +1,231 @@ +package dev.caskeleton.messaging.admin.runtime; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatCode; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import dev.caskeleton.messaging.admin.AdminApproval; +import dev.caskeleton.messaging.admin.ApprovalGrant; +import dev.caskeleton.messaging.admin.ApprovedRedrivePlan; +import dev.caskeleton.messaging.admin.DestructiveOperation; +import dev.caskeleton.messaging.admin.HmacApprovalVerifier; +import dev.caskeleton.messaging.admin.RedrivePlan; +import dev.caskeleton.messaging.admin.RedriveRequest; +import dev.caskeleton.messaging.admin.VerifiedApproval; +import dev.caskeleton.messaging.api.destination.DestinationName; +import dev.caskeleton.messaging.api.error.MessageAuthorizationException; +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.time.Instant; +import java.util.Optional; +import java.util.UUID; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** + * Whether "this was approved" is a fact or a claim the caller makes about itself. + * + *

    It was a claim. {@code AdminApproval} was a public record and {@code ApprovedRedrivePlan} a + * public record over it, so any code that could reach the execute method could write its own + * approval and the platform believed it. Nothing bound an approval to an operation, a destination, + * a topology version, or an impact ceiling either, so a genuine approval for one redrive authorised + * a redrive of anything else inside its window. + */ +class ApprovalForgeryTest { + + private static final Instant NOW = Instant.parse("2026-08-10T09:00:00Z"); + private static final UUID OPERATION_ID = UUID.fromString("0199aaaa-bbbb-7ccc-8ddd-eeeeffff0000"); + private static final DestinationName DLQ = new DestinationName("orders.v1.dlq"); + private static final DestinationName ORDERS = new DestinationName("orders.v1"); + + private static final HmacApprovalVerifier ISSUER = + new HmacApprovalVerifier( + "an-issuing-key-of-at-least-32-bytes!".getBytes(StandardCharsets.UTF_8)); + + private static final HmacApprovalVerifier ANOTHER_ISSUER = + new HmacApprovalVerifier( + "a-different-key-of-at-least-32-bytes".getBytes(StandardCharsets.UTF_8)); + + @Test + @DisplayName("a verified approval cannot be constructed outside the verifier") + void aVerifiedApprovalCannotBeConstructedOutsideTheVerifier() { + // Written as a reflection probe because the property is about the type's shape, not its + // behaviour: if a public constructor or factory ever appears, every other test here becomes + // decorative. + assertThat(VerifiedApproval.class.getConstructors()) + .as("a public constructor would let any caller mint an approval for itself") + .isEmpty(); + assertThat( + java.util.Arrays.stream(VerifiedApproval.class.getDeclaredMethods()) + .filter(method -> java.lang.reflect.Modifier.isStatic(method.getModifiers())) + .filter(method -> java.lang.reflect.Modifier.isPublic(method.getModifiers())) + .toList()) + .as("nor may a public static factory") + .isEmpty(); + } + + @Test + @DisplayName("an unsigned or altered grant does not verify") + void anAlteredGrantDoesNotVerify() { + RedrivePlan plan = plan(50, 0); + ApprovalGrant signed = grant(plan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 50); + String signature = ISSUER.sign(signed); + + ApprovalGrant raised = grant(plan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 5_000); + + assertThatThrownBy(() -> ISSUER.verify(raised, signature, plan.digest(), NOW)) + .as("raising the impact ceiling after the approver signed it") + .isInstanceOf(MessageAuthorizationException.class) + .hasMessageContaining("does not verify"); + } + + @Test + @DisplayName("a signature from another issuer does not verify") + void aSignatureFromAnotherIssuerDoesNotVerify() { + RedrivePlan plan = plan(50, 0); + ApprovalGrant granted = grant(plan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 50); + + assertThatThrownBy( + () -> ISSUER.verify(granted, ANOTHER_ISSUER.sign(granted), plan.digest(), NOW)) + .isInstanceOf(MessageAuthorizationException.class) + .hasMessageContaining("does not verify"); + } + + @Test + @DisplayName("an approval for one plan cannot execute another") + void anApprovalForOnePlanCannotExecuteAnother() { + RedrivePlan approvedPlan = plan(50, 0); + RedrivePlan biggerPlan = plan(900, 0); + ApprovalGrant granted = grant(approvedPlan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 900); + String signature = ISSUER.sign(granted); + + assertThatThrownBy(() -> ISSUER.verify(granted, signature, biggerPlan.digest(), NOW)) + .as("the approver saw 50 candidates; this plan moves 900") + .isInstanceOf(MessageAuthorizationException.class) + .hasMessageContaining("was issued for plan"); + } + + @Test + @DisplayName("a verified approval for the wrong source cannot be attached to a plan") + void aVerifiedApprovalForTheWrongSourceCannotBeAttached() { + RedrivePlan plan = plan(50, 0); + // A genuine, correctly-signed approval — for a different dead-letter destination, with this + // plan's digest smuggled in. The digest check passes; the source binding is what catches it. + ApprovalGrant granted = + new ApprovalGrant( + approval(), + "operator@example.com", + DestructiveOperation.REDRIVE, + new DestinationName("payments.v1.dlq"), + Optional.of(ORDERS), + "v1", + 900, + plan.digest()); + VerifiedApproval verified = ISSUER.verify(granted, ISSUER.sign(granted), plan.digest(), NOW); + + assertThatThrownBy(() -> new ApprovedRedrivePlan(plan, verified, false)) + .isInstanceOf(MessageAuthorizationException.class) + .hasMessageContaining("payments.v1.dlq"); + } + + @Test + @DisplayName("a plan beyond the approved impact ceiling cannot execute") + void aPlanBeyondTheApprovedImpactCeilingCannotExecute() { + RedrivePlan plan = plan(900, 0); + ApprovalGrant granted = grant(plan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 100); + VerifiedApproval verified = ISSUER.verify(granted, ISSUER.sign(granted), plan.digest(), NOW); + + assertThatThrownBy(() -> new ApprovedRedrivePlan(plan, verified, false)) + .as("the approver agreed to at most 100") + .isInstanceOf(MessageAuthorizationException.class) + .hasMessageContaining("at most 100"); + } + + @Test + @DisplayName("an approval for a replay cannot authorise a redrive") + void anApprovalForAReplayCannotAuthoriseARedrive() { + RedrivePlan plan = plan(50, 0); + ApprovalGrant granted = grant(plan, DestructiveOperation.REPLAY, DLQ, ORDERS, 900); + VerifiedApproval verified = ISSUER.verify(granted, ISSUER.sign(granted), plan.digest(), NOW); + + assertThatThrownBy(() -> new ApprovedRedrivePlan(plan, verified, false)) + .isInstanceOf(MessageAuthorizationException.class) + .hasMessageContaining("authorises REPLAY"); + } + + @Test + @DisplayName("an expired grant does not verify") + void anExpiredGrantDoesNotVerify() { + RedrivePlan plan = plan(50, 0); + ApprovalGrant granted = grant(plan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 50); + + assertThatThrownBy( + () -> + ISSUER.verify( + granted, ISSUER.sign(granted), plan.digest(), NOW.plus(Duration.ofDays(1)))) + .isInstanceOf(MessageAuthorizationException.class) + .hasMessageContaining("not valid at"); + } + + @Test + @DisplayName("the approver and the operator must be different people") + void theApproverAndTheOperatorMustBeDifferentPeople() { + RedrivePlan plan = plan(50, 0); + + assertThatThrownBy( + () -> + new ApprovalGrant( + approval(), + "ops@example.com", + DestructiveOperation.REDRIVE, + DLQ, + Optional.of(ORDERS), + "v1", + 50, + plan.digest())) + .as("an operator approving their own destructive operation is the control not existing") + .isInstanceOf(IllegalArgumentException.class); + } + + @Test + @DisplayName("a correctly issued approval executes") + void aCorrectlyIssuedApprovalExecutes() { + RedrivePlan plan = plan(50, 0); + ApprovalGrant granted = grant(plan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 50); + VerifiedApproval verified = ISSUER.verify(granted, ISSUER.sign(granted), plan.digest(), NOW); + + assertThatCode( + () -> new ApprovedRedrivePlan(plan, verified, false).requireExecutable(NOW, "v1")) + .doesNotThrowAnyException(); + } + + private static ApprovalGrant grant( + RedrivePlan plan, + DestructiveOperation operation, + DestinationName source, + DestinationName target, + long maxImpact) { + return new ApprovalGrant( + approval(), + "operator@example.com", + operation, + source, + Optional.of(target), + "v1", + maxImpact, + plan.digest()); + } + + private static AdminApproval approval() { + return new AdminApproval("CHG-1042", "ops@example.com", NOW, NOW.plus(Duration.ofHours(2))); + } + + private static RedrivePlan plan(int candidates, int alreadyRedriven) { + return new RedrivePlan( + new RedriveRequest(OPERATION_ID, DLQ, ORDERS, 50, false), + candidates, + alreadyRedriven, + NOW, + "v1"); + } +} diff --git a/messaging/messaging-admin-runtime/src/test/java/dev/caskeleton/messaging/admin/runtime/ApprovedPlanExecutionTest.java b/messaging/messaging-admin-runtime/src/test/java/dev/caskeleton/messaging/admin/runtime/ApprovedPlanExecutionTest.java index 924509b..02dd2dc 100644 --- a/messaging/messaging-admin-runtime/src/test/java/dev/caskeleton/messaging/admin/runtime/ApprovedPlanExecutionTest.java +++ b/messaging/messaging-admin-runtime/src/test/java/dev/caskeleton/messaging/admin/runtime/ApprovedPlanExecutionTest.java @@ -5,14 +5,19 @@ import static org.assertj.core.api.Assertions.assertThatCode; import static org.assertj.core.api.Assertions.assertThatThrownBy; import dev.caskeleton.messaging.admin.AdminApproval; +import dev.caskeleton.messaging.admin.ApprovalGrant; import dev.caskeleton.messaging.admin.ApprovedRedrivePlan; import dev.caskeleton.messaging.admin.ApprovedReplayPlan; +import dev.caskeleton.messaging.admin.DestructiveOperation; +import dev.caskeleton.messaging.admin.HmacApprovalVerifier; +import dev.caskeleton.messaging.admin.PlanDigest; import dev.caskeleton.messaging.admin.RedrivePlan; import dev.caskeleton.messaging.admin.RedriveRequest; import dev.caskeleton.messaging.admin.RedriveResult; import dev.caskeleton.messaging.admin.ReplayPlan; import dev.caskeleton.messaging.admin.ReplayRequest; import dev.caskeleton.messaging.admin.ReplayResult; +import dev.caskeleton.messaging.admin.VerifiedApproval; import dev.caskeleton.messaging.api.destination.DestinationName; import dev.caskeleton.messaging.api.error.MessageAuthorizationException; import java.time.Duration; @@ -26,10 +31,56 @@ class ApprovedPlanExecutionTest { private static final Instant NOW = Instant.parse("2026-08-10T09:00:00Z"); private static final UUID OPERATION_ID = UUID.fromString("0199aaaa-bbbb-7ccc-8ddd-eeeeffff0000"); + private static final byte[] ISSUING_KEY = + "an-issuing-key-of-at-least-32-bytes!".getBytes(java.nio.charset.StandardCharsets.UTF_8); + + private static final HmacApprovalVerifier VERIFIER = new HmacApprovalVerifier(ISSUING_KEY); + private static AdminApproval approval() { return new AdminApproval("CHG-1042", "ops@example.com", NOW, NOW.plus(Duration.ofHours(2))); } + /** A grant the issuer signed for exactly this plan, verified the way the runtime verifies it. */ + static VerifiedApproval verified( + PlanDigest digest, + DestructiveOperation operation, + DestinationName source, + Optional target, + String topologyVersion, + long maxImpact) { + ApprovalGrant grant = + new ApprovalGrant( + approval(), + "operator@example.com", + operation, + source, + target, + topologyVersion, + maxImpact, + digest); + return VERIFIER.verify(grant, VERIFIER.sign(grant), digest, NOW); + } + + private static VerifiedApproval replayApproval(ReplayPlan plan) { + return verified( + plan.digest(), + DestructiveOperation.REPLAY, + plan.request().destination(), + Optional.empty(), + plan.topologyVersion(), + Long.MAX_VALUE); + } + + private static VerifiedApproval redriveApproval(RedrivePlan plan) { + return verified( + plan.digest(), + DestructiveOperation.REDRIVE, + plan.request().source(), + Optional.of(plan.request().target()), + plan.topologyVersion(), + Long.MAX_VALUE); + } + private static ReplayRequest replayRequest() { return new ReplayRequest( OPERATION_ID, @@ -57,9 +108,18 @@ class ApprovedPlanExecutionTest { return new RedrivePlan(redriveRequest(), candidates, alreadyRedriven, NOW, "v1"); } + private static ApprovedReplayPlan approvedReplay(String topologyVersion) { + ReplayPlan plan = replayPlan(topologyVersion); + return new ApprovedReplayPlan(plan, replayApproval(plan)); + } + + private static ApprovedRedrivePlan approvedRedrive(RedrivePlan plan, boolean loopAcknowledged) { + return new ApprovedRedrivePlan(plan, redriveApproval(plan), loopAcknowledged); + } + @Test void anExpiredApprovalCannotExecute() { - ApprovedReplayPlan approved = new ApprovedReplayPlan(replayPlan("v1"), approval()); + ApprovedReplayPlan approved = approvedReplay("v1"); assertThatThrownBy(() -> approved.requireExecutable(NOW.plus(Duration.ofDays(1)), "v1")) .isInstanceOf(MessageAuthorizationException.class) @@ -68,25 +128,26 @@ class ApprovedPlanExecutionTest { @Test void aTopologyChangeSinceApprovalInvalidatesThePlan() { - ApprovedReplayPlan approved = new ApprovedReplayPlan(replayPlan("v1"), approval()); + ApprovedReplayPlan approved = approvedReplay("v1"); assertThatThrownBy(() -> approved.requireExecutable(NOW, "v2")) .as("every number in the plan was computed against the old topology") .isInstanceOf(MessageAuthorizationException.class) - .hasMessageContaining("rebuilt"); + // The signed grant carries the topology version too, so the mismatch is now caught against + // what the approver actually signed rather than against a field on the plan the caller + // handed in. + .hasMessageContaining("was signed against topology v1 but the broker is now at v2"); } @Test void aValidApprovalOnUnchangedTopologyExecutes() { - assertThatCode( - () -> new ApprovedReplayPlan(replayPlan("v1"), approval()).requireExecutable(NOW, "v1")) + assertThatCode(() -> approvedReplay("v1").requireExecutable(NOW, "v1")) .doesNotThrowAnyException(); } @Test void aRedriveThatWouldLoopNeedsThatAcknowledgedExplicitly() { - ApprovedRedrivePlan approved = - new ApprovedRedrivePlan(redrivePlan(900, 400), approval(), false); + ApprovedRedrivePlan approved = approvedRedrive(redrivePlan(900, 400), false); assertThatThrownBy(() -> approved.requireExecutable(NOW, "v1")) .isInstanceOf(MessageAuthorizationException.class) @@ -95,19 +156,13 @@ class ApprovedPlanExecutionTest { @Test void anAcknowledgedLoopMayProceed() { - assertThatCode( - () -> - new ApprovedRedrivePlan(redrivePlan(900, 400), approval(), true) - .requireExecutable(NOW, "v1")) + assertThatCode(() -> approvedRedrive(redrivePlan(900, 400), true).requireExecutable(NOW, "v1")) .doesNotThrowAnyException(); } @Test void aRedriveWithNoPreviouslyRedrivenCandidatesNeedsNoAcknowledgement() { - assertThatCode( - () -> - new ApprovedRedrivePlan(redrivePlan(900, 0), approval(), false) - .requireExecutable(NOW, "v1")) + assertThatCode(() -> approvedRedrive(redrivePlan(900, 0), false).requireExecutable(NOW, "v1")) .doesNotThrowAnyException(); } @@ -139,25 +194,6 @@ class ApprovedPlanExecutionTest { .contains("LIVE consumer group"); } - @Test - void anApprovalAuthorisesOneExecutionNotAStandingPermission() { - AdminOperationIdempotencyStore store = new AdminOperationIdempotencyStore(); - - assertThat(store.claim("CHG-1042", "op-1", NOW)).isEmpty(); - assertThat(store.claim("CHG-1042", "op-2", NOW.plus(Duration.ofMinutes(5)))) - .as("an arrow-up in the shell must not re-run an approved redrive") - .hasValueSatisfying(existing -> assertThat(existing.operationId()).isEqualTo("op-1")); - } - - @Test - void aDifferentApprovalIsClaimedIndependently() { - AdminOperationIdempotencyStore store = new AdminOperationIdempotencyStore(); - store.claim("CHG-1042", "op-1", NOW); - - assertThat(store.claim("CHG-1043", "op-2", NOW)).isEmpty(); - assertThat(store.size()).isEqualTo(2); - } - @Test void aReplayResultFlagsAShortfallAgainstTheApprovedEstimate() { ReplayResult result = diff --git a/messaging/messaging-admin-runtime/src/test/java/dev/caskeleton/messaging/admin/runtime/RedriveResumptionTest.java b/messaging/messaging-admin-runtime/src/test/java/dev/caskeleton/messaging/admin/runtime/RedriveResumptionTest.java new file mode 100644 index 0000000..82412de --- /dev/null +++ b/messaging/messaging-admin-runtime/src/test/java/dev/caskeleton/messaging/admin/runtime/RedriveResumptionTest.java @@ -0,0 +1,212 @@ +package dev.caskeleton.messaging.admin.runtime; + +import static org.assertj.core.api.Assertions.assertThat; + +import dev.caskeleton.messaging.admin.AdminApproval; +import dev.caskeleton.messaging.admin.ApprovalGrant; +import dev.caskeleton.messaging.admin.DestructiveOperation; +import dev.caskeleton.messaging.admin.DestructiveOperationGuard; +import dev.caskeleton.messaging.admin.HmacApprovalVerifier; +import dev.caskeleton.messaging.admin.PlanDigest; +import dev.caskeleton.messaging.admin.RedriveRequest; +import dev.caskeleton.messaging.admin.VerifiedApproval; +import dev.caskeleton.messaging.api.MessageId; +import dev.caskeleton.messaging.api.destination.DestinationName; +import dev.caskeleton.messaging.api.publish.ConfirmationLevel; +import dev.caskeleton.messaging.api.publish.PublishCompletion; +import dev.caskeleton.messaging.api.publish.PublishEvidence; +import dev.caskeleton.messaging.api.publish.PublishResult; +import dev.caskeleton.messaging.api.publish.RoutingOutcome; +import dev.caskeleton.messaging.observation.MessagingAuditEvent; +import java.time.Duration; +import java.time.Instant; +import java.util.ArrayList; +import java.util.List; +import java.util.Optional; +import java.util.UUID; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** + * What a redrive does when one message fails, and what it does the second time round. + * + *

    The loop had no per-item boundary. A publisher that threw — a broker refusing the connection + * rather than the message — propagated straight out, so every candidate behind it was abandoned and + * the audit record was never written. The operation left no trace of what it had already moved, and + * a retry started again from the first candidate and republished it. + */ +class RedriveResumptionTest { + + private static final Instant NOW = Instant.parse("2026-08-10T09:00:00Z"); + private static final DestinationName DLQ = new DestinationName("orders.v1.dlq"); + private static final DestinationName ORDERS = new DestinationName("orders.v1"); + + private static final HmacApprovalVerifier ISSUER = + new HmacApprovalVerifier( + "an-issuing-key-of-at-least-32-bytes!".getBytes(java.nio.charset.StandardCharsets.UTF_8)); + + private final RecordingSource source = new RecordingSource(); + private final RecordingAudit audit = new RecordingAudit(); + + @Test + @DisplayName("one message that throws does not abandon the ones behind it") + void oneThrowingMessageDoesNotAbandonTheRest() { + List candidates = ids(4); + source.stage(candidates); + RedriveService service = + service( + (messageId, target, redriveId) -> { + if (messageId.equals(candidates.get(1))) { + throw new IllegalStateException("the broker refused the connection"); + } + return confirmed(); + }); + + RedriveReport report = service.redrive(request(), approval(), "operator", NOW); + + assertThat(report.moved()).isEqualTo(3); + assertThat(report.failed()) + .as("the throwing item is a failed item, not a failed pass") + .isEqualTo(1); + assertThat(source.settled) + .containsExactly(candidates.get(0), candidates.get(2), candidates.get(3)); + } + + @Test + @DisplayName("the audit record is written even when the pass does not finish cleanly") + void theAuditRecordIsWrittenEvenOnFailure() { + source.stage(ids(2)); + RedriveService service = + service( + (messageId, target, redriveId) -> { + throw new IllegalStateException("the broker refused the connection"); + }); + + service.redrive(request(), approval(), "operator", NOW); + + assertThat(audit.events) + .as("the record of what moved is what the incident review and the resumed run both read") + .singleElement() + .satisfies(event -> assertThat(event.details()).containsEntry("failed", "2")); + } + + @Test + @DisplayName("a resumed pass does not republish what the first attempt already moved") + void aResumedPassDoesNotRepublishWhatWasAlreadyMoved() { + List candidates = ids(5); + source.stage(candidates); + List republished = new ArrayList<>(); + RedriveService service = + service( + (messageId, target, redriveId) -> { + republished.add(messageId); + return confirmed(); + }); + + RedriveReport report = + service.redrive(request(), approval(), "operator", NOW, 3, completed -> {}); + + assertThat(republished) + .as("republishing a settled message is a duplicate, not a retry") + .containsExactly(candidates.get(3), candidates.get(4)); + assertThat(report.moved()).isEqualTo(2); + } + + @Test + @DisplayName("progress is reported after each item so a crash resumes near where it stopped") + void progressIsReportedAfterEachItem() { + source.stage(ids(3)); + List checkpoints = new ArrayList<>(); + RedriveService service = service((messageId, target, redriveId) -> confirmed()); + + service.redrive(request(), approval(), "operator", NOW, 0, checkpoints::add); + + assertThat(checkpoints).containsExactly(1, 2, 3); + } + + @Test + @DisplayName("resuming past the end of the batch moves nothing") + void resumingPastTheEndOfTheBatchMovesNothing() { + source.stage(ids(2)); + RedriveService service = service((messageId, target, redriveId) -> confirmed()); + + RedriveReport report = + service.redrive(request(), approval(), "operator", NOW, 10, completed -> {}); + + assertThat(report.moved()).isZero(); + } + + private RedriveService service(RedriveService.RedrivePublisher publisher) { + // The guard holds the admin credential and every pass carries a real verified approval: this + // test is about the loop, and a gate that refused would prove nothing about it. + return new RedriveService(new DestructiveOperationGuard(true), source, publisher, audit); + } + + private static Optional approval() { + PlanDigest digest = PlanDigest.ofCanonical("REDRIVE|orders.v1.dlq|orders.v1"); + ApprovalGrant grant = + new ApprovalGrant( + new AdminApproval("CHG-1042", "ops@example.com", NOW, NOW.plus(Duration.ofHours(2))), + "operator@example.com", + DestructiveOperation.REDRIVE, + DLQ, + Optional.of(ORDERS), + "v1", + Long.MAX_VALUE, + digest); + return Optional.of(ISSUER.verify(grant, ISSUER.sign(grant), digest, NOW)); + } + + private static RedriveRequest request() { + return new RedriveRequest(UUID.randomUUID(), DLQ, ORDERS, 50, false); + } + + private static PublishResult confirmed() { + return new PublishResult( + PublishCompletion.CONFIRMED, + PublishEvidence.confirmed(ConfirmationLevel.REPLICATION_OR_PERSISTENCE_ACK), + RoutingOutcome.ROUTED, + Optional.empty(), + 1, + Duration.ofMillis(2), + Optional.empty()); + } + + private static List ids(int count) { + List ids = new ArrayList<>(); + for (int index = 0; index < count; index++) { + ids.add(MessageId.newId()); + } + return List.copyOf(ids); + } + + private static final class RecordingSource implements RedriveService.RedriveSource { + + private List staged = List.of(); + private final List settled = new ArrayList<>(); + + private void stage(List ids) { + staged = List.copyOf(ids); + } + + @Override + public List peek(DestinationName destination, int batchSize) { + return staged; + } + + @Override + public void settle(DestinationName destination, MessageId messageId) { + settled.add(messageId); + } + } + + private static final class RecordingAudit implements RedriveService.AuditSink { + + private final List events = new ArrayList<>(); + + @Override + public void record(MessagingAuditEvent event) { + events.add(event); + } + } +} diff --git a/messaging/messaging-cloudevents/build.gradle b/messaging/messaging-cloudevents/build.gradle index 59fb5a7..355ea49 100644 --- a/messaging/messaging-cloudevents/build.gradle +++ b/messaging/messaging-cloudevents/build.gradle @@ -4,6 +4,12 @@ dependencies { api project(':messaging:messaging-core-api') api project(':messaging:messaging-schema-api') - implementation 'io.cloudevents:cloudevents-api:4.0.1' + // api, because CloudEventMapper's public signatures return io.cloudevents.CloudEvent. + // + // Declared `implementation`, the type appeared in this module's public API while the + // dependency was hidden from consumers: an adopter calling the documented method could not + // name its return type without adding CloudEvents to their own build, and Gradle gave them no + // hint why. A type in a public signature is part of the artifact's contract. + api 'io.cloudevents:cloudevents-api:4.0.1' implementation 'io.cloudevents:cloudevents-core:4.0.1' } diff --git a/messaging/messaging-cloudevents/src/test/java/dev/caskeleton/messaging/cloudevents/CloudEventMappingTest.java b/messaging/messaging-cloudevents/src/test/java/dev/caskeleton/messaging/cloudevents/CloudEventMappingTest.java index 447666e..f82c6da 100644 --- a/messaging/messaging-cloudevents/src/test/java/dev/caskeleton/messaging/cloudevents/CloudEventMappingTest.java +++ b/messaging/messaging-cloudevents/src/test/java/dev/caskeleton/messaging/cloudevents/CloudEventMappingTest.java @@ -87,7 +87,9 @@ class CloudEventMappingTest { void aCloudEventWithNoDataBecomesAnEmptyPayloadNotANullValue() { CloudEvent noData = io.cloudevents.core.builder.CloudEventBuilder.v1() - .withId(UUID.randomUUID().toString()) + // A v7 id: MessageId enforces the version it documents, so a v4 arriving from a foreign + // producer is refused here exactly as it would be on the wire. + .withId(dev.caskeleton.messaging.api.MessageId.newId().value().toString()) .withSource(SOURCE) .withType("order.created") .withTime(java.time.OffsetDateTime.parse("2026-08-10T09:15:00Z")) diff --git a/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/ContentType.java b/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/ContentType.java index 9e4c7c7..2fcbdba 100644 --- a/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/ContentType.java +++ b/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/ContentType.java @@ -11,6 +11,13 @@ public record ContentType(String value) { private static final int MAX_LENGTH = 160; + /** {@code type/subtype} with the optional parameters RFC 9110 allows. */ + private static final java.util.regex.Pattern MEDIA_TYPE = + java.util.regex.Pattern.compile( + "[A-Za-z0-9][A-Za-z0-9!#$%&'*+.^_`|~-]*" + + "/[A-Za-z0-9][A-Za-z0-9!#$%&'*+.^_`|~-]*" + + "(;\\s*[A-Za-z0-9!#$%&'*+.^_`|~-]+=[^;\\s]+)*"); + /** The Stable default codec's content type. */ public static final ContentType JSON = new ContentType("application/json"); @@ -27,8 +34,12 @@ public record ContentType(String value) { if (value == null || value.isBlank() || value.length() > MAX_LENGTH) { throw new IllegalArgumentException("contentType must contain 1 to 160 characters"); } - if (value.indexOf('/') < 0) { - throw new IllegalArgumentException("contentType must be a media type: " + value); + // A media type, not merely a string containing a slash. `a/b/c`, `/json` and `text/` all + // passed the old check and are all rejected by a broker or a codec registry later, with an + // error that names the message rather than the configuration that produced it. + if (!MEDIA_TYPE.matcher(value).matches()) { + throw new IllegalArgumentException( + "contentType must be a type/subtype media type: '" + value + '\''); } value = value.toLowerCase(Locale.ROOT); } diff --git a/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/MessageId.java b/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/MessageId.java index 94189cc..0d0d73a 100644 --- a/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/MessageId.java +++ b/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/MessageId.java @@ -14,8 +14,21 @@ import java.util.UUID; */ public record MessageId(UUID value) { + /** The UUID version this identity is documented to carry. */ + private static final int VERSION_7 = 7; + public MessageId { Objects.requireNonNull(value, "messageId value must not be null"); + // The type says UUIDv7 and the constructor accepted any UUID, including v4 and the nil UUID. + // Version 7 is what makes the identity time-ordered, which is what the outbox index and every + // "oldest first" claim depend on; a v4 stored in the same column silently defeats both. + if (value.version() != VERSION_7) { + throw new IllegalArgumentException( + "a message identity is UUIDv7 (time-ordered); this is version " + value.version()); + } + if (value.variant() != 2) { + throw new IllegalArgumentException("a message identity must use the RFC 4122 variant"); + } } /** diff --git a/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/MessageType.java b/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/MessageType.java index b137ceb..47eef8c 100644 --- a/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/MessageType.java +++ b/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/MessageType.java @@ -16,5 +16,29 @@ public record MessageType(String value) { if (value == null || value.isBlank() || value.length() > MAX_LENGTH) { throw new IllegalArgumentException("messageType must contain 1 to 240 characters"); } + requireWireSafe(value, "a message type", MAX_LENGTH); + } + + /** + * Characters that must never reach a broker. + * + *

    Length in Java {@code char}s bounded nothing useful: a 240-character string can be 960 UTF-8 + * bytes, and a control character passes a length check untouched. CR and LF split records on + * line-oriented protocols, NUL truncates them in C string handling, and both survive right up to + * the broker that mishandles them. + */ + private static void requireWireSafe(String value, String what, int maxBytes) { + for (int index = 0; index < value.length(); index++) { + char character = value.charAt(index); + if (character < 0x20 || character == 0x7F) { + throw new IllegalArgumentException( + what + + " must not contain control characters (found U+%04X)".formatted((int) character)); + } + } + int bytes = value.getBytes(java.nio.charset.StandardCharsets.UTF_8).length; + if (bytes > maxBytes) { + throw new IllegalArgumentException(what + " exceeds " + maxBytes + " UTF-8 bytes: " + bytes); + } } } diff --git a/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/ProducerId.java b/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/ProducerId.java index 2919015..313bbc0 100644 --- a/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/ProducerId.java +++ b/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/ProducerId.java @@ -16,5 +16,29 @@ public record ProducerId(String value) { if (value == null || value.isBlank() || value.length() > MAX_LENGTH) { throw new IllegalArgumentException("producerId must contain 1 to 120 characters"); } + requireWireSafe(value, "a producer id", MAX_LENGTH); + } + + /** + * Characters that must never reach a broker. + * + *

    Length in Java {@code char}s bounded nothing useful: a 240-character string can be 960 UTF-8 + * bytes, and a control character passes a length check untouched. CR and LF split records on + * line-oriented protocols, NUL truncates them in C string handling, and both survive right up to + * the broker that mishandles them. + */ + private static void requireWireSafe(String value, String what, int maxBytes) { + for (int index = 0; index < value.length(); index++) { + char character = value.charAt(index); + if (character < 0x20 || character == 0x7F) { + throw new IllegalArgumentException( + what + + " must not contain control characters (found U+%04X)".formatted((int) character)); + } + } + int bytes = value.getBytes(java.nio.charset.StandardCharsets.UTF_8).length; + if (bytes > maxBytes) { + throw new IllegalArgumentException(what + " exceeds " + maxBytes + " UTF-8 bytes: " + bytes); + } } } diff --git a/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/header/HeaderName.java b/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/header/HeaderName.java index 26f6c2e..d826ebf 100644 --- a/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/header/HeaderName.java +++ b/messaging/messaging-core-api/src/main/java/dev/caskeleton/messaging/api/header/HeaderName.java @@ -15,10 +15,36 @@ public record HeaderName(String value) { private static final int MAX_BYTES = 128; + /** + * The one grammar a header name may use. + * + *

    The previous check was "not blank, at most 128 bytes", which accepted a name containing CR, + * LF, NUL, a colon or leading whitespace. Those are not cosmetic: a newline in a header name is a + * header-injection primitive on any line-oriented protocol, a NUL truncates the name in C string + * handling inside a broker, and a colon is the field separator in several wire formats. A name + * that survives one broker and corrupts the next is worse than a rejected one. + * + *

    The character set is the HTTP token set, which every broker this platform targets accepts + * verbatim. + */ + private static final java.util.regex.Pattern TOKEN = + java.util.regex.Pattern.compile("[a-zA-Z0-9!#$%&'*+._|~-]+"); + public HeaderName { if (value == null || value.isBlank()) { throw new IllegalArgumentException("header name must not be blank"); } + if (!value.equals(value.strip())) { + // Trimming would mean `Authorization ` and `Authorization` are the same name to the + // denylist and different names on the wire, which is precisely how the check was bypassed. + throw new IllegalArgumentException( + "header name must not have leading or trailing whitespace: '" + value + '\''); + } + if (!TOKEN.matcher(value).matches()) { + throw new IllegalArgumentException( + "header name must be an ASCII token; control characters, spaces, colons and non-ASCII" + + " are refused because a broker that accepts one may truncate or split on it"); + } int bytes = value.getBytes(StandardCharsets.UTF_8).length; if (bytes > MAX_BYTES) { throw new IllegalArgumentException("header name exceeds 128 bytes: " + bytes); diff --git a/messaging/messaging-core-api/src/test/java/dev/caskeleton/messaging/api/WireSafeValueObjectTest.java b/messaging/messaging-core-api/src/test/java/dev/caskeleton/messaging/api/WireSafeValueObjectTest.java new file mode 100644 index 0000000..1af6d19 Binary files /dev/null and b/messaging/messaging-core-api/src/test/java/dev/caskeleton/messaging/api/WireSafeValueObjectTest.java differ diff --git a/messaging/messaging-inbox-jpa/build.gradle b/messaging/messaging-inbox-jdbc-postgresql/build.gradle similarity index 100% rename from messaging/messaging-inbox-jpa/build.gradle rename to messaging/messaging-inbox-jdbc-postgresql/build.gradle diff --git a/messaging/messaging-inbox-jpa/gradle.lockfile b/messaging/messaging-inbox-jdbc-postgresql/gradle.lockfile similarity index 100% rename from messaging/messaging-inbox-jpa/gradle.lockfile rename to messaging/messaging-inbox-jdbc-postgresql/gradle.lockfile diff --git a/messaging/messaging-inbox-jpa/src/main/java/dev/caskeleton/messaging/inbox/IdempotentConsumer.java b/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/IdempotentConsumer.java similarity index 100% rename from messaging/messaging-inbox-jpa/src/main/java/dev/caskeleton/messaging/inbox/IdempotentConsumer.java rename to messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/IdempotentConsumer.java diff --git a/messaging/messaging-inbox-jpa/src/main/java/dev/caskeleton/messaging/inbox/InboxCleanupJob.java b/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/InboxCleanupJob.java similarity index 100% rename from messaging/messaging-inbox-jpa/src/main/java/dev/caskeleton/messaging/inbox/InboxCleanupJob.java rename to messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/InboxCleanupJob.java diff --git a/messaging/messaging-inbox-jpa/src/main/java/dev/caskeleton/messaging/inbox/InboxOutcome.java b/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/InboxOutcome.java similarity index 100% rename from messaging/messaging-inbox-jpa/src/main/java/dev/caskeleton/messaging/inbox/InboxOutcome.java rename to messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/InboxOutcome.java diff --git a/messaging/messaging-inbox-jpa/src/main/java/dev/caskeleton/messaging/inbox/InboxRetentionPolicy.java b/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/InboxRetentionPolicy.java similarity index 100% rename from messaging/messaging-inbox-jpa/src/main/java/dev/caskeleton/messaging/inbox/InboxRetentionPolicy.java rename to messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/InboxRetentionPolicy.java diff --git a/messaging/messaging-inbox-jpa/src/main/java/dev/caskeleton/messaging/inbox/JdbcInboxRepository.java b/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/JdbcInboxRepository.java similarity index 56% rename from messaging/messaging-inbox-jpa/src/main/java/dev/caskeleton/messaging/inbox/JdbcInboxRepository.java rename to messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/JdbcInboxRepository.java index 8edb4fa..1ac930c 100644 --- a/messaging/messaging-inbox-jpa/src/main/java/dev/caskeleton/messaging/inbox/JdbcInboxRepository.java +++ b/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/JdbcInboxRepository.java @@ -11,6 +11,8 @@ import java.sql.Timestamp; import java.time.Instant; import java.util.Objects; import javax.sql.DataSource; +import org.springframework.jdbc.datasource.DataSourceUtils; +import org.springframework.transaction.support.TransactionSynchronizationManager; /** * The PostgreSQL inbox. @@ -48,14 +50,19 @@ public final class JdbcInboxRepository implements InboxRepository { /** * Reserves a message on the caller's transactional connection. * + *

    Package-private. It used to be public and was the only path that actually joined the + * caller's transaction, while the interface method — the one {@code IdempotentConsumer} calls — + * opened a raw connection that auto-commits. A reservation that commits on its own while the + * business side effect rolls back is a message that will never be redelivered and whose work + * never happened. + * * @param connection the connection the business write is running on * @param messageId the logical message identity * @param consumerId the consumer's stable identity * @param now the current instant * @return true when this is the first time */ - public boolean reserve( - Connection connection, MessageId messageId, String consumerId, Instant now) { + boolean reserve(Connection connection, MessageId messageId, String consumerId, Instant now) { Objects.requireNonNull(connection, "connection must not be null"); Objects.requireNonNull(messageId, "messageId must not be null"); Objects.requireNonNull(now, "now must not be null"); @@ -75,11 +82,41 @@ public final class JdbcInboxRepository implements InboxRepository { @Override public boolean reserve(MessageId messageId, String consumerId, Instant now) { - try (Connection connection = dataSource.getConnection()) { + requireActiveTransaction("reserving an inbox message"); + Connection connection = DataSourceUtils.getConnection(dataSource); + try { return reserve(connection, messageId, consumerId, now); - } catch (SQLException exception) { + } finally { + DataSourceUtils.releaseConnection(connection, dataSource); + } + } + + /** + * Refuses when there is no transaction to join, or when it cannot write. + * + *

    The reservation and the side effect it guards have to commit or roll back together. Running + * the reservation on its own connection breaks that on the rollback path only — which is the path + * nobody exercises before production, and the one where the message is lost for good. + */ + private void requireActiveTransaction(String what) { + if (!TransactionSynchronizationManager.isActualTransactionActive()) { throw new MessagingConfigurationException( - "INBOX_RESERVE_FAILED", "could not reserve an inbox record", exception); + "INBOX_TRANSACTION_REQUIRED", + what + + " requires an active transaction; a reservation that commits alone marks a message" + + " processed whose work may still roll back"); + } + if (TransactionSynchronizationManager.isCurrentTransactionReadOnly()) { + throw new MessagingConfigurationException( + "INBOX_TRANSACTION_REQUIRED", + what + " requires a writable transaction; the current one is read-only"); + } + if (!TransactionSynchronizationManager.hasResource(dataSource)) { + throw new MessagingConfigurationException( + "INBOX_TRANSACTION_REQUIRED", + what + + " requires the active transaction to be bound to this DataSource; it is bound to" + + " another, so the reservation and the side effect would commit independently"); } } @@ -100,6 +137,37 @@ public final class JdbcInboxRepository implements InboxRepository { } } + @Override + public int purgeProcessedBefore(Instant processedBefore, int limit) { + Objects.requireNonNull(processedBefore, "processedBefore must not be null"); + if (limit < 1) { + throw new IllegalArgumentException("a bounded purge deletes at least one row per call"); + } + try (Connection connection = dataSource.getConnection(); + PreparedStatement statement = + connection.prepareStatement( + """ + WITH expired AS ( + SELECT message_id, consumer_id + FROM messaging_inbox + WHERE processed_at < ? + ORDER BY processed_at + LIMIT ? + FOR UPDATE SKIP LOCKED + ) + DELETE FROM messaging_inbox i + USING expired e + WHERE i.message_id = e.message_id AND i.consumer_id = e.consumer_id + """)) { + statement.setTimestamp(1, Timestamp.from(processedBefore)); + statement.setInt(2, limit); + return statement.executeUpdate(); + } catch (SQLException exception) { + throw new MessagingConfigurationException( + "INBOX_PURGE_FAILED", "a bounded inbox purge failed", exception); + } + } + @Override public int purgeProcessedBefore(Instant processedBefore) { Objects.requireNonNull(processedBefore, "processedBefore must not be null"); diff --git a/messaging/messaging-inbox-jpa/src/main/java/dev/caskeleton/messaging/inbox/TransactionalInboxHandler.java b/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/TransactionalInboxHandler.java similarity index 100% rename from messaging/messaging-inbox-jpa/src/main/java/dev/caskeleton/messaging/inbox/TransactionalInboxHandler.java rename to messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/TransactionalInboxHandler.java diff --git a/messaging/messaging-inbox-jpa/src/main/resources/db/migration/messaging/V2__messaging_inbox.sql b/messaging/messaging-inbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V2__messaging_inbox.sql similarity index 100% rename from messaging/messaging-inbox-jpa/src/main/resources/db/migration/messaging/V2__messaging_inbox.sql rename to messaging/messaging-inbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V2__messaging_inbox.sql diff --git a/messaging/messaging-inbox-jpa/src/test/java/dev/caskeleton/messaging/inbox/IdempotentConsumerTest.java b/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/IdempotentConsumerTest.java similarity index 96% rename from messaging/messaging-inbox-jpa/src/test/java/dev/caskeleton/messaging/inbox/IdempotentConsumerTest.java rename to messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/IdempotentConsumerTest.java index 9cc55ba..013fe84 100644 --- a/messaging/messaging-inbox-jpa/src/test/java/dev/caskeleton/messaging/inbox/IdempotentConsumerTest.java +++ b/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/IdempotentConsumerTest.java @@ -102,6 +102,11 @@ final class InMemoryInbox implements InboxRepository { return reserved.contains(key(messageId, consumerId)); } + @Override + public int purgeProcessedBefore(Instant processedBefore, int limit) { + return Math.min(purgeProcessedBefore(processedBefore), limit); + } + @Override public int purgeProcessedBefore(Instant processedBefore) { int removed = reserved.size(); diff --git a/messaging/messaging-inbox-jpa/src/test/java/dev/caskeleton/messaging/inbox/InboxOperationsTest.java b/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/InboxOperationsTest.java similarity index 97% rename from messaging/messaging-inbox-jpa/src/test/java/dev/caskeleton/messaging/inbox/InboxOperationsTest.java rename to messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/InboxOperationsTest.java index 5404e76..17409b1 100644 --- a/messaging/messaging-inbox-jpa/src/test/java/dev/caskeleton/messaging/inbox/InboxOperationsTest.java +++ b/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/InboxOperationsTest.java @@ -42,6 +42,11 @@ class InboxOperationsTest { return reserved.contains(messageId.value() + "|" + consumerId); } + @Override + public int purgeProcessedBefore(Instant processedBefore, int limit) { + return Math.min(purgeProcessedBefore(processedBefore), limit); + } + @Override public int purgeProcessedBefore(Instant processedBefore) { cutoffs.add(processedBefore); diff --git a/messaging/messaging-inbox-jpa/src/test/java/dev/caskeleton/messaging/inbox/InboxPostgresIT.java b/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/InboxPostgresIT.java similarity index 65% rename from messaging/messaging-inbox-jpa/src/test/java/dev/caskeleton/messaging/inbox/InboxPostgresIT.java rename to messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/InboxPostgresIT.java index d202a3e..b4b0c34 100644 --- a/messaging/messaging-inbox-jpa/src/test/java/dev/caskeleton/messaging/inbox/InboxPostgresIT.java +++ b/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/InboxPostgresIT.java @@ -48,6 +48,8 @@ class InboxPostgresIT { .withPassword("messaging"); private DataSource dataSource; + + private org.springframework.transaction.support.TransactionTemplate transactions; private JdbcInboxRepository repository; static boolean dockerAvailable() { @@ -65,29 +67,40 @@ class InboxPostgresIT { statement.execute("CREATE TABLE projections (id TEXT PRIMARY KEY, applied INT NOT NULL)"); } repository = new JdbcInboxRepository(dataSource); + // A real Spring transaction, because the port's contract is about joining one. The previous + // tests called the Connection overload directly, which proved that overload works and said + // nothing about the interface method every consumer actually calls. + transactions = + new org.springframework.transaction.support.TransactionTemplate( + new org.springframework.jdbc.datasource.DataSourceTransactionManager(dataSource)); } @Test void theFirstReservationSucceedsAndTheSecondDoesNot() { MessageId messageId = MessageId.newId(); - assertThat(repository.reserve(messageId, "order-projection", NOW)).isTrue(); - assertThat(repository.reserve(messageId, "order-projection", NOW)).isFalse(); + assertThat(inTransaction(() -> repository.reserve(messageId, "order-projection", NOW))) + .isTrue(); + assertThat(inTransaction(() -> repository.reserve(messageId, "order-projection", NOW))) + .isFalse(); } @Test void twoConsumersEachReserveTheSameMessageOnce() { MessageId messageId = MessageId.newId(); - assertThat(repository.reserve(messageId, "order-projection", NOW)).isTrue(); - assertThat(repository.reserve(messageId, "billing-projection", NOW)).isTrue(); - assertThat(repository.reserve(messageId, "order-projection", NOW)).isFalse(); + assertThat(inTransaction(() -> repository.reserve(messageId, "order-projection", NOW))) + .isTrue(); + assertThat(inTransaction(() -> repository.reserve(messageId, "billing-projection", NOW))) + .isTrue(); + assertThat(inTransaction(() -> repository.reserve(messageId, "order-projection", NOW))) + .isFalse(); } @Test void aProcessedMessageIsVisibleToTheQuery() { MessageId messageId = MessageId.newId(); - repository.reserve(messageId, "order-projection", NOW); + inTransaction(() -> repository.reserve(messageId, "order-projection", NOW)); assertThat(repository.isProcessed(messageId, "order-projection")).isTrue(); assertThat(repository.isProcessed(messageId, "billing-projection")).isFalse(); @@ -109,13 +122,19 @@ class InboxPostgresIT { void aRolledBackTransactionLeavesNoReservationAndNoSideEffect() throws SQLException { MessageId messageId = MessageId.newId(); - try (Connection connection = dataSource.getConnection()) { - connection.setAutoCommit(false); - assertThat(repository.reserve(connection, messageId, "order-projection", NOW)).isTrue(); - try (Statement statement = connection.createStatement()) { - statement.execute("INSERT INTO projections (id, applied) VALUES ('o-2', 1)"); - } - connection.rollback(); + // Through the interface method and a real Spring transaction. This is the path a consumer + // takes; the old version called the Connection overload, which is the one path that was never + // in doubt. + try { + transactions.executeWithoutResult( + status -> { + assertThat(repository.reserve(messageId, "order-projection", NOW)).isTrue(); + new org.springframework.jdbc.core.JdbcTemplate(dataSource) + .execute("INSERT INTO projections (id, applied) VALUES ('o-2', 1)"); + status.setRollbackOnly(); + }); + } catch (RuntimeException rolledBack) { + // setRollbackOnly is the mechanism; an UnexpectedRollbackException is not a failure here. } assertThat(repository.isProcessed(messageId, "order-projection")) @@ -126,25 +145,28 @@ class InboxPostgresIT { @Test void retentionRemovesOldRows() { - repository.reserve(MessageId.newId(), "order-projection", NOW); + MessageId purged = MessageId.newId(); + inTransaction(() -> repository.reserve(purged, "order-projection", NOW)); assertThat(repository.purgeProcessedBefore(NOW.plusSeconds(1))).isEqualTo(1); } - /** Reserves and writes inside one transaction, exactly as an idempotent consumer must. */ - private void handle(MessageId messageId, AtomicInteger applied) throws SQLException { - try (Connection connection = dataSource.getConnection()) { - connection.setAutoCommit(false); - if (!repository.reserve(connection, messageId, "order-projection", NOW)) { - connection.rollback(); - return; - } - try (Statement statement = connection.createStatement()) { - statement.execute("INSERT INTO projections (id, applied) VALUES ('o-1', 1)"); - } - applied.incrementAndGet(); - connection.commit(); - } + /** Reserves and writes inside one Spring transaction, exactly as an idempotent consumer must. */ + private void handle(MessageId messageId, AtomicInteger applied) { + transactions.executeWithoutResult( + status -> { + if (!repository.reserve(messageId, "order-projection", NOW)) { + status.setRollbackOnly(); + return; + } + new org.springframework.jdbc.core.JdbcTemplate(dataSource) + .execute("INSERT INTO projections (id, applied) VALUES ('o-1', 1)"); + applied.incrementAndGet(); + }); + } + + private T inTransaction(java.util.function.Supplier work) { + return transactions.execute(status -> work.get()); } private int projectionCount(String id) throws SQLException { diff --git a/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/JdbcInboxTransactionRequirementTest.java b/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/JdbcInboxTransactionRequirementTest.java new file mode 100644 index 0000000..4baf5cd --- /dev/null +++ b/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/JdbcInboxTransactionRequirementTest.java @@ -0,0 +1,98 @@ +package dev.caskeleton.messaging.inbox; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import dev.caskeleton.messaging.api.MessageId; +import dev.caskeleton.messaging.api.error.MessagingConfigurationException; +import java.time.Instant; +import javax.sql.DataSource; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.springframework.transaction.support.TransactionSynchronizationManager; + +/** + * What {@code reserve(...)} does when there is no transaction to join. + * + *

    {@code IdempotentConsumer} calls this inside its transaction runner, and the implementation + * opened a raw connection that auto-commits. The reservation therefore survived a rollback of the + * side effect it was guarding: the message counts as processed, the work never happened, and + * redelivery is refused because the inbox row is already there. + * + *

    Hermetic: the refusal has to happen before any connection is requested, and the data source + * below fails the test by being asked for one. + */ +class JdbcInboxTransactionRequirementTest { + + /** A data source that fails the test by being used: the refusal must precede any connection. */ + private final DataSource dataSource = new RefusingDataSource(); + + private final JdbcInboxRepository repository = new JdbcInboxRepository(dataSource); + + @AfterEach + void clearTransactionState() { + TransactionSynchronizationManager.clear(); + if (TransactionSynchronizationManager.hasResource(dataSource)) { + TransactionSynchronizationManager.unbindResource(dataSource); + } + } + + @Test + @DisplayName("reserving outside a transaction is refused before a connection is opened") + void reservingOutsideATransactionIsRefused() { + assertThatThrownBy(() -> repository.reserve(MessageId.newId(), "order-projection", NOW)) + .isInstanceOf(MessagingConfigurationException.class) + .hasMessageContaining("requires an active transaction"); + } + + @Test + @DisplayName("a read-only transaction is refused") + void aReadOnlyTransactionIsRefused() { + TransactionSynchronizationManager.setActualTransactionActive(true); + TransactionSynchronizationManager.setCurrentTransactionReadOnly(true); + + assertThatThrownBy(() -> repository.reserve(MessageId.newId(), "order-projection", NOW)) + .isInstanceOf(MessagingConfigurationException.class) + .hasMessageContaining("read-only"); + } + + @Test + @DisplayName("a transaction bound to another DataSource is refused") + void aTransactionOnAnotherDataSourceIsRefused() { + TransactionSynchronizationManager.setActualTransactionActive(true); + + assertThatThrownBy(() -> repository.reserve(MessageId.newId(), "order-projection", NOW)) + .as("two DataSources means two commits, which is the atomicity this exists to provide") + .isInstanceOf(MessagingConfigurationException.class) + .hasMessageContaining("bound to this DataSource"); + } + + @Test + @DisplayName("the refusal names a code an operator can search for") + void theRefusalNamesACode() { + assertThatThrownBy(() -> repository.reserve(MessageId.newId(), "order-projection", NOW)) + .isInstanceOfSatisfying( + MessagingConfigurationException.class, + failure -> + assertThat(failure.failure().code()).isEqualTo("INBOX_TRANSACTION_REQUIRED")); + } + + private static final Instant NOW = Instant.parse("2026-08-14T00:00:00Z"); + + /** Throws if anything asks it for a connection. */ + private static final class RefusingDataSource + extends org.springframework.jdbc.datasource.AbstractDataSource { + + @Override + public java.sql.Connection getConnection() { + throw new AssertionError( + "the refusal must happen before a connection is requested; opening one is the defect"); + } + + @Override + public java.sql.Connection getConnection(String username, String password) { + return getConnection(); + } + } +} diff --git a/messaging/messaging-kafka-share-experimental/gradle.lockfile b/messaging/messaging-kafka-share-experimental/gradle.lockfile index 237cd70..f5485ac 100644 --- a/messaging/messaging-kafka-share-experimental/gradle.lockfile +++ b/messaging/messaging-kafka-share-experimental/gradle.lockfile @@ -34,9 +34,9 @@ commons-logging:commons-logging:1.3.5=runtimeClasspath,testRuntimeClasspath info.picocli:picocli:4.7.7=checkstyle io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnotationProcessor io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor -io.micrometer:micrometer-commons:1.16.0=runtimeClasspath,testRuntimeClasspath -io.micrometer:micrometer-core:1.16.0=runtimeClasspath,testRuntimeClasspath -io.micrometer:micrometer-observation:1.16.0=runtimeClasspath,testRuntimeClasspath +io.micrometer:micrometer-commons:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.micrometer:micrometer-core:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.micrometer:micrometer-observation:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath javax.inject:javax.inject:1=annotationProcessor,testAnnotationProcessor jaxen:jaxen:2.0.0=spotbugs net.bytebuddy:byte-buddy:1.17.8=testCompileClasspath,testRuntimeClasspath @@ -65,7 +65,7 @@ org.codehaus.plexus:plexus-utils:3.3.0=checkstyle org.dom4j:dom4j:2.2.0=spotbugs org.hdrhistogram:HdrHistogram:2.2.2=runtimeClasspath,testRuntimeClasspath org.javassist:javassist:3.28.0-GA=checkstyle -org.jspecify:jspecify:1.0.0=annotationProcessor,checkstyle,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath +org.jspecify:jspecify:1.0.0=annotationProcessor,checkstyle,compileClasspath,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath org.junit.jupiter:junit-jupiter-api:6.0.1=testCompileClasspath,testRuntimeClasspath org.junit.jupiter:junit-jupiter-engine:6.0.1=testRuntimeClasspath org.junit.jupiter:junit-jupiter-params:6.0.1=testCompileClasspath,testRuntimeClasspath diff --git a/messaging/messaging-kafka/build.gradle b/messaging/messaging-kafka/build.gradle index e9fba83..a15369d 100644 --- a/messaging/messaging-kafka/build.gradle +++ b/messaging/messaging-kafka/build.gradle @@ -10,7 +10,10 @@ dependencies { api project(':messaging:messaging-admin-api') implementation 'org.springframework.kafka:spring-kafka' - implementation 'org.apache.kafka:kafka-clients' + // api: the public constructors and mappers take and return Producer, ProducerRecord and + // RecordMetadata. Hiding the dependency does not hide the types — it only stops a consumer + // from compiling against the API this module documents. + api 'org.apache.kafka:kafka-clients' // Test-only. The shared adapter contract is what makes "Stable" mean something, so the Kafka // adapter runs it rather than asserting its own behaviour in its own terms. diff --git a/messaging/messaging-kafka/gradle.lockfile b/messaging/messaging-kafka/gradle.lockfile index 4abc2f1..5fbdba6 100644 --- a/messaging/messaging-kafka/gradle.lockfile +++ b/messaging/messaging-kafka/gradle.lockfile @@ -42,7 +42,7 @@ info.picocli:picocli:4.7.7=checkstyle io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,jmhAnnotationProcessor,testAnnotationProcessor io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,jmhAnnotationProcessor,testAnnotationProcessor io.micrometer:micrometer-commons:1.16.0=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -io.micrometer:micrometer-core:1.16.0=jmhRuntimeClasspath,runtimeClasspath,testRuntimeClasspath +io.micrometer:micrometer-core:1.16.0=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath io.micrometer:micrometer-observation:1.16.0=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath javax.inject:javax.inject:1=annotationProcessor,jmhAnnotationProcessor,testAnnotationProcessor jaxen:jaxen:2.0.0=spotbugs diff --git a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/ContiguousPartitionOffsetTracker.java b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/ContiguousPartitionOffsetTracker.java index c1c6d59..82e7ee1 100644 --- a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/ContiguousPartitionOffsetTracker.java +++ b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/ContiguousPartitionOffsetTracker.java @@ -32,6 +32,20 @@ public final class ContiguousPartitionOffsetTracker implements PartitionOffsetTr delivered.computeIfAbsent(partition, key -> new ConcurrentSkipListSet<>()).add(offset); } + /** + * Undoes a delivery registration for a record that was never submitted. + * + *

    Needed because the executor can refuse the work after the offset has been registered. A + * delivered offset with no worker behind it holds the contiguous watermark back forever: nothing + * will ever complete it, so the partition stops committing while continuing to consume. + */ + public void forgetDelivered(TopicPartition partition, long offset) { + NavigableSet inFlight = delivered.get(partition); + if (inFlight != null) { + inFlight.remove(offset); + } + } + @Override public void completed(TopicPartition partition, long offset) { completed.computeIfAbsent(partition, key -> new ConcurrentSkipListSet<>()).add(offset); diff --git a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaConsumerRegistrar.java b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaConsumerRegistrar.java index 64b4198..241f047 100644 --- a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaConsumerRegistrar.java +++ b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaConsumerRegistrar.java @@ -64,7 +64,36 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati private final GracefulShutdownCoordinator shutdown; private final Duration pollTimeout; + private final KafkaQuarantinePublisher quarantine; + private final Map committed = new ConcurrentHashMap<>(); + + /** + * The assignment generation of each owned partition. + * + *

    A settlement carries the epoch it was created under. Without one, a handler that finished + * after its partition was revoked and reassigned applied its outcome to the new assignment: the + * old epoch's acknowledgement advanced a watermark over offsets the new owner had not processed. + */ + private final Map epochs = new ConcurrentHashMap<>(); + + private final java.util.concurrent.atomic.AtomicLong epochSequence = + new java.util.concurrent.atomic.AtomicLong(); + + private final java.util.concurrent.atomic.AtomicInteger staleSettlements = + new java.util.concurrent.atomic.AtomicInteger(); + + /** + * Work that must run on the poll thread, queued by any other thread. + * + *

    {@code pause}, {@code resume} and {@code close} are public API and were called on the + * caller's thread, touching the {@code Consumer} directly — the one object this class documents + * as poll-thread-only. The Kafka consumer is not thread-safe, and a concurrent call is a {@code + * ConcurrentModificationException} at best and a corrupted subscription at worst. + */ + private final java.util.Queue control = + new java.util.concurrent.ConcurrentLinkedQueue<>(); + private final AtomicBoolean active = new AtomicBoolean(true); /** @@ -82,6 +111,23 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati Executor handlerPool, GracefulShutdownCoordinator shutdown, Duration pollTimeout) { + this( + spec, consumer, handlerPool, shutdown, pollTimeout, KafkaQuarantinePublisher.unavailable()); + } + + /** + * A consumer runtime with a quarantine destination for undecodable records. + * + * @param quarantine where a record that cannot be decoded is written before its offset commits + */ + public KafkaConsumerRegistrar( + TransportConsumerSpec spec, + Consumer consumer, + Executor handlerPool, + GracefulShutdownCoordinator shutdown, + Duration pollTimeout, + KafkaQuarantinePublisher quarantine) { + this.quarantine = Objects.requireNonNull(quarantine, "quarantine must not be null"); this.spec = Objects.requireNonNull(spec, "spec must not be null"); this.consumer = Objects.requireNonNull(consumer, "consumer must not be null"); this.handlerPool = Objects.requireNonNull(handlerPool, "handlerPool must not be null"); @@ -112,6 +158,10 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati new ConsumerRebalanceListener() { @Override public void onPartitionsRevoked(Collection partitions) { + // Drop the epoch first. Any settlement still in flight for these partitions now + // carries a number no live assignment has, so applySettlements refuses it instead of + // moving a watermark this consumer no longer owns. + partitions.forEach(epochs::remove); commitContiguous(); partitions.forEach( partition -> { @@ -124,7 +174,9 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati @Override public void onPartitionsAssigned(Collection partitions) { - // Nothing to restore: the broker's committed offset is the source of truth. + // A fresh epoch per assignment. The broker's committed offset is still the source of + // truth for where to read; the epoch decides whose settlements may be applied. + partitions.forEach(partition -> epochs.put(partition, epochSequence.incrementAndGet())); } }); } @@ -143,6 +195,8 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati public int pollOnce(Instant now) { Objects.requireNonNull(now, "now must not be null"); + drainControl(); + syncEpochs(); applySettlements(); applyDueResumes(now); commitContiguous(); @@ -153,61 +207,142 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati ConsumerRecords records = consumer.poll(pollTimeout); int dispatched = 0; - for (ConsumerRecord record : records) { - TopicPartition partition = new TopicPartition(record.topic(), record.partition()); - - if (!coordinator.tryAcquire(partition)) { - // At the ceiling: stop fetching and re-read this record on a later cycle. + // Per partition, and never a global break. + // + // The loop used to iterate the whole batch and `break` on the first record it could not + // submit — which abandoned every record the same poll had returned for *other* partitions. + // The consumer's position had already advanced past them, so those offsets were not read + // again until a rebalance or a restart: a processing gap that nothing reported. + for (TopicPartition partition : records.partitions()) { + Long epoch = epochs.get(partition); + if (epoch == null) { + // Not ours: revoked between the poll and now. + continue; + } + java.util.List> partitionRecords = records.records(partition); + long seekBackTo = -1L; + for (ConsumerRecord record : partitionRecords) { + if (seekBackTo >= 0) { + // Already stopped for this partition; the seek below covers this record too. + continue; + } + if (!coordinator.tryAcquire(partition)) { + seekBackTo = record.offset(); + continue; + } + if (!shutdown.tryBeginWork()) { + coordinator.release(partition); + seekBackTo = record.offset(); + continue; + } + offsets.delivered(partition, record.offset()); + if (!dispatch(record, partition, now, epoch)) { + // The pool refused the work. Undo everything this iteration took so the in-flight count + // and the delivered set do not drift, and re-read the record next cycle. + offsets.forgetDelivered(partition, record.offset()); + coordinator.release(partition); + shutdown.endWork(); + seekBackTo = record.offset(); + continue; + } + dispatched++; + } + if (seekBackTo >= 0) { + // Only this partition pauses. Its earliest unsubmitted offset is where it resumes. consumer.pause(Set.of(partition)); - consumer.seek(partition, record.offset()); - break; + consumer.seek(partition, seekBackTo); } - if (!shutdown.tryBeginWork()) { - coordinator.release(partition); - consumer.seek(partition, record.offset()); - break; - } - - offsets.delivered(partition, record.offset()); - dispatch(record, partition, now); - dispatched++; } return dispatched; } - private void dispatch( - ConsumerRecord record, TopicPartition partition, Instant now) { + /** + * Hands one record to the pool. + * + * @return false when the pool refused the work, so the caller can undo what it took + */ + private boolean dispatch( + ConsumerRecord record, TopicPartition partition, Instant now, long epoch) { DestinationProfile profile = spec.profile(); DestinationName destination = profile.name(); + Duration handlerTimeout = profile.consumer().handlerTimeout(); - handlerPool.execute( - () -> { - try { - var envelope = deliveryMapper.toEnvelope(record); - int attempt = retryMetadataMapper.attemptOf(envelope); - TransportDelivery delivery = - new TransportDelivery( - envelope, - deliveryMapper.toMetadata( - destination, record, attempt, profile.consumer().group().orElse(null), now), - new QueuedSettlement(partition, record.offset())); - spec.sink().apply(delivery).toCompletableFuture().join(); - } catch (RuntimeException exception) { - // The sink owns failure classification. A throw here means the delivery could not even - // be decoded, which the platform parks rather than retries. - settlements.enqueue( - new KafkaSettlementCommand( - partition, record.offset(), KafkaSettlementCommand.Kind.PARKED)); - } finally { - coordinator.release(partition); - shutdown.endWork(); - } - }); + try { + handlerPool.execute( + () -> { + QueuedSettlement settlement = new QueuedSettlement(partition, record.offset(), epoch); + dev.caskeleton.messaging.api.MessageEnvelope< + dev.caskeleton.messaging.schema.EncodedMessage> + envelope; + try { + envelope = deliveryMapper.toEnvelope(record); + } catch (RuntimeException undecodable) { + // A decode failure is not a handler failure and must not be retried forever. The + // offset may only advance once the record exists somewhere else, so an unavailable + // quarantine leaves it uncommitted rather than dropping the message. + if (quarantine.quarantine(record, undecodable)) { + settlement.acknowledgeAfterQuarantine(); + } else { + settlement.requeueAfterFailure(); + } + coordinator.release(partition); + shutdown.endWork(); + return; + } + try { + int attempt = retryMetadataMapper.attemptOf(envelope); + TransportDelivery delivery = + new TransportDelivery( + envelope, + deliveryMapper.toMetadata( + destination, + record, + attempt, + profile.consumer().group().orElse(null), + now), + settlement); + // The declared handler timeout is applied. It existed as a policy field and the + // worker blocked on an unbounded join, so a handler that never returned held its + // partition's permit and its offset forever — the partition simply stopped. + spec.sink() + .apply(delivery) + .toCompletableFuture() + .get(handlerTimeout.toMillis(), java.util.concurrent.TimeUnit.MILLISECONDS); + } catch (java.util.concurrent.TimeoutException timedOut) { + // Unsettled work, not a completion: pause and re-read rather than advancing past a + // message whose handler may still be running. + settlement.requeueAfterFailure(); + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + settlement.requeueAfterFailure(); + } catch (java.util.concurrent.ExecutionException | RuntimeException failure) { + // A handler failure is retried, not parked. PARKED completed the offset without any + // quarantine or dead-letter write, so a transient database outage silently discarded + // every message it touched. + settlement.requeueAfterFailure(); + } finally { + coordinator.release(partition); + shutdown.endWork(); + } + }); + return true; + } catch (java.util.concurrent.RejectedExecutionException refused) { + // The caller undoes the permit, the work count and the delivered registration. + return false; + } } /** Applies everything workers queued. Poll thread only. */ private void applySettlements() { for (KafkaSettlementCommand command : settlements.drain()) { + Long currentEpoch = epochs.get(command.partition()); + if (currentEpoch == null || currentEpoch != command.epoch()) { + // The partition was revoked, or reassigned, after this settlement was created. Applying it + // would move the current owner's watermark using an outcome from the previous assignment, + // over offsets the current owner has not processed. + staleSettlements.incrementAndGet(); + continue; + } switch (command.kind()) { case COMPLETE, PARKED -> offsets.completed(command.partition(), command.offset()); case PAUSE_AND_SEEK -> { @@ -222,6 +357,21 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati } } + /** + * Keeps the epoch map in step with the current assignment. + * + *

    The rebalance listener is the usual source of epochs, but it is not the only way a partition + * becomes owned: a manually assigned consumer never fires it. Deriving the epoch from the + * assignment as well means fencing works in both cases, and a partition that has gone away loses + * its epoch here even if no revoke callback arrived. + */ + private void syncEpochs() { + Set current = assignment(); + current.forEach( + partition -> epochs.computeIfAbsent(partition, key -> epochSequence.incrementAndGet())); + epochs.keySet().removeIf(partition -> !current.contains(partition)); + } + private void applyDueResumes(Instant now) { Map due = retries.dueForResume(now); due.forEach( @@ -246,12 +396,32 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati continue; } toCommit.put(partition, new OffsetAndMetadata(offset)); - committed.put(partition, offset); - offsets.pruneThrough(partition, offset - 1); } - if (!toCommit.isEmpty()) { - consumer.commitSync(toCommit); + if (toCommit.isEmpty()) { + return; } + // Commit first, then record it locally. + // + // The local map and the tracker used to be updated before commitSync. A failed commit + // therefore left this consumer believing an offset was committed that the broker had never + // accepted, and the pruned tracker could no longer produce it — so the next successful commit + // jumped over the gap and the records in it were never redelivered. + consumer.commitSync(toCommit); + toCommit.forEach( + (partition, metadata) -> { + committed.put(partition, metadata.offset()); + offsets.pruneThrough(partition, metadata.offset() - 1); + }); + } + + /** + * How many settlements were refused because their assignment had been revoked. + * + *

    Exposed rather than counted silently: a non-zero value means handlers are outliving their + * assignment, which is a rebalance or a handler-timeout problem an operator can act on. + */ + public int staleSettlements() { + return staleSettlements.get(); } private Set assignment() { @@ -271,18 +441,52 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati @Override public CompletionStage pause(String scope) { - Set partitions = scoped(scope); - partitions.forEach(coordinator::pause); - consumer.pause(partitions); - return CompletableFuture.completedFuture(null); + return onPollThread( + () -> { + Set partitions = scoped(scope); + partitions.forEach(coordinator::pause); + consumer.pause(partitions); + }); } @Override public CompletionStage resume(String scope) { - Set partitions = scoped(scope); - partitions.forEach(coordinator::resume); - consumer.resume(partitions); - return CompletableFuture.completedFuture(null); + return onPollThread( + () -> { + Set partitions = scoped(scope); + partitions.forEach(coordinator::resume); + consumer.resume(partitions); + }); + } + + /** + * Queues work for the poll thread and reports when it ran. + * + *

    The returned stage completes after the poll loop has actually applied the change, so a + * caller that awaits it knows the consumer is paused rather than merely asked to pause. + * + *

    It therefore completes on the next poll cycle. A caller that blocks on it without a + * running loop waits forever, which is the honest shape of the request: there is no safe way to + * touch the consumer from another thread, so "paused" cannot be true until the loop says so. + */ + private CompletionStage onPollThread(Runnable work) { + CompletableFuture applied = new CompletableFuture<>(); + control.add( + () -> { + try { + work.run(); + applied.complete(null); + } catch (RuntimeException failure) { + applied.completeExceptionally(failure); + } + }); + return applied; + } + + private void drainControl() { + for (Runnable work = control.poll(); work != null; work = control.poll()) { + work.run(); + } } private Set scoped(String scope) { @@ -302,56 +506,116 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati @Override public void close() { if (active.compareAndSet(true, false)) { + // Close still runs here rather than through the queue: after this the poll loop stops, so + // queued work would never drain. What it does not do is race an in-progress poll — callers + // stop the loop first, and `active` makes a second close a no-op. + drainControl(); applySettlements(); commitContiguous(); consumer.close(); } } - /** The settlement handle handed to a worker; it only enqueues. */ + /** + * The settlement handle handed to a worker. + * + *

    Two things changed. It carries the assignment epoch, so a settlement that arrives after a + * revoke is refused rather than applied to whoever owns the partition now. And it is + * one-terminal: acknowledge, requeue and discard compete for a single CAS, because a handler that + * acknowledged and then requeued used to tell the poll thread both, and the poll thread believed + * the second one. + */ private final class QueuedSettlement implements dev.caskeleton.messaging.transport.TransportSettlement { private final TopicPartition partition; private final long offset; + private final long epoch; + private final java.util.concurrent.atomic.AtomicReference terminal = + new java.util.concurrent.atomic.AtomicReference<>(); - private QueuedSettlement(TopicPartition partition, long offset) { + private QueuedSettlement(TopicPartition partition, long offset, long epoch) { this.partition = partition; this.offset = offset; + this.epoch = epoch; } @Override public CompletionStage acknowledge() { - settlements.enqueue( - new KafkaSettlementCommand(partition, offset, KafkaSettlementCommand.Kind.COMPLETE)); - return CompletableFuture.completedFuture( - dev.caskeleton.messaging.api.settlement.SettlementResult.settled()); + return terminalOnce( + "acknowledge", + () -> { + settlements.enqueue( + new KafkaSettlementCommand( + partition, offset, KafkaSettlementCommand.Kind.COMPLETE, epoch)); + return dev.caskeleton.messaging.api.settlement.SettlementResult.settled(); + }); } @Override public CompletionStage requeue( Duration delay) { - retries.pauseUntil(partition, offset, delay, Instant.now()); - settlements.enqueue( - new KafkaSettlementCommand( - partition, offset, KafkaSettlementCommand.Kind.PAUSE_AND_SEEK)); - return CompletableFuture.completedFuture( - new dev.caskeleton.messaging.api.settlement.SettlementResult( - dev.caskeleton.messaging.api.settlement.SettlementCompletion.UNKNOWN, - dev.caskeleton.messaging.api.settlement.SettlementEvidence.unknown(), - java.util.Optional.of( - dev.caskeleton.messaging.api.error.FailureDescriptor.of( - dev.caskeleton.messaging.api.error.FailureCategory.PROCESSING_TRANSIENT, - "KAFKA_PAUSED_FOR_RETRY", - "the partition was paused and will be re-read")))); + return terminalOnce("requeue", () -> enqueueRequeue(delay)); } @Override public CompletionStage discard() { + return terminalOnce( + "discard", + () -> { + settlements.enqueue( + new KafkaSettlementCommand( + partition, offset, KafkaSettlementCommand.Kind.PARKED, epoch)); + return dev.caskeleton.messaging.api.settlement.SettlementResult.settled(); + }); + } + + /** Used by the worker once an undecodable record has been written to quarantine. */ + private void acknowledgeAfterQuarantine() { + if (terminal.compareAndSet(null, "acknowledge")) { + settlements.enqueue( + new KafkaSettlementCommand( + partition, offset, KafkaSettlementCommand.Kind.PARKED, epoch)); + } + } + + /** Used by the worker when the handler failed or timed out and never settled. */ + private void requeueAfterFailure() { + if (terminal.compareAndSet(null, "requeue")) { + enqueueRequeue(Duration.ZERO); + } + } + + private dev.caskeleton.messaging.api.settlement.SettlementResult enqueueRequeue( + Duration delay) { + retries.pauseUntil(partition, offset, delay, Instant.now()); settlements.enqueue( - new KafkaSettlementCommand(partition, offset, KafkaSettlementCommand.Kind.PARKED)); - return CompletableFuture.completedFuture( - dev.caskeleton.messaging.api.settlement.SettlementResult.settled()); + new KafkaSettlementCommand( + partition, offset, KafkaSettlementCommand.Kind.PAUSE_AND_SEEK, epoch)); + return new dev.caskeleton.messaging.api.settlement.SettlementResult( + dev.caskeleton.messaging.api.settlement.SettlementCompletion.UNKNOWN, + dev.caskeleton.messaging.api.settlement.SettlementEvidence.unknown(), + java.util.Optional.of( + dev.caskeleton.messaging.api.error.FailureDescriptor.of( + dev.caskeleton.messaging.api.error.FailureCategory.PROCESSING_TRANSIENT, + "KAFKA_PAUSED_FOR_RETRY", + "the partition was paused and will be re-read"))); + } + + private CompletionStage terminalOnce( + String what, + java.util.function.Supplier + call) { + if (!terminal.compareAndSet(null, what)) { + return CompletableFuture.failedFuture( + new IllegalStateException( + "this delivery was already settled by " + + terminal.get() + + "; a second " + + what + + " would contradict what the poll thread has already been told")); + } + return CompletableFuture.completedFuture(call.get()); } } } diff --git a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaHeaderMapper.java b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaHeaderMapper.java index 9d7fb6d..f72e3d1 100644 --- a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaHeaderMapper.java +++ b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaHeaderMapper.java @@ -67,7 +67,27 @@ public final class KafkaHeaderMapper { .baggage() .ifPresent(value -> put(headers, ReservedHeaders.BAGGAGE, value)); - envelope.headers().asMap().forEach((name, value) -> put(headers, name.value(), value.value())); + // Application headers are written last and must not be able to reach a reserved name. + // + // They used to be appended after the canonical ones, and the consumer reads `lastHeader` — so a + // row in the outbox carrying `msg.id` overwrote the canonical message identity on the wire. + // That identity is what inbox deduplication and the audit trail key on, so a forged one + // corrupts another message's dedup state rather than merely its own. + envelope + .headers() + .asMap() + .forEach( + (name, value) -> { + if (ReservedHeaders.isReserved(name.value())) { + throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException( + "RESERVED_HEADER_FORGED", + "application header '" + + name.value() + + "' uses the reserved namespace; the canonical value is written by the" + + " platform and must not be overridable by message content"); + } + put(headers, name.value(), value.value()); + }); return headers; } diff --git a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaMessagingTransport.java b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaMessagingTransport.java index cc2cddf..6f1a369 100644 --- a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaMessagingTransport.java +++ b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaMessagingTransport.java @@ -40,9 +40,28 @@ import org.apache.kafka.clients.producer.ProducerRecord; */ public final class KafkaMessagingTransport implements MessagingTransport { + /** + * What this adapter can prove, with deduplicated publish set to false. + * + *

    It was {@code true}, justified by producer idempotence. Producer idempotence deduplicates + * sequence retries within one producer session: the producer id is reassigned on + * restart, so the same logical message published again after a crash is a new sequence and the + * broker stores it twice. The capability an application reads is "the broker suppresses duplicate + * publishes of a stable id", which needs a persistent message-id store that this platform does + * not have. + * + *

    Declaring it true means {@code PublishDeduplication} is accepted and silently does nothing — + * the caller believes the broker is deduplicating and skips the idempotency it would otherwise + * build. False makes that request a startup failure instead, which is the whole reason the flag + * exists. + * + *

    The order of these flags is the record's: brokerAcknowledgement, replication evidence, + * per-message settlement, batch settlement, ordered stream, keyed ordering, replay, delayed + * delivery, broker transaction, deduplicated publish, native dead letter, topology management. + */ private static final MessagingCapabilities CAPABILITIES = new MessagingCapabilities( - true, true, true, true, true, true, true, false, true, true, false, true); + true, true, true, true, true, true, true, false, true, false, false, true); private final String brokerName; private final long generation; diff --git a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaQuarantinePublisher.java b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaQuarantinePublisher.java new file mode 100644 index 0000000..bdb4365 --- /dev/null +++ b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaQuarantinePublisher.java @@ -0,0 +1,31 @@ +package dev.caskeleton.messaging.kafka; + +import org.apache.kafka.clients.consumer.ConsumerRecord; + +/** + * Where a record that cannot be decoded is written before its offset is committed. + * + *

    The consumer used to mark an undecodable record {@code PARKED}, and the poll thread treated + * {@code PARKED} exactly like {@code COMPLETE} — the offset advanced and the record was gone. There + * was no quarantine collection, no dead-letter publish and no record of what had been dropped, so + * "parked" named a state nothing implemented. + * + *

    Committing past a record is only safe once the record exists somewhere else. An implementation + * that cannot write returns {@code false}, and the consumer leaves the offset uncommitted rather + * than losing the message quietly. + */ +@FunctionalInterface +public interface KafkaQuarantinePublisher { + + /** + * Writes an undecodable record somewhere durable. + * + * @return true when the write is confirmed, so the source offset may be committed + */ + boolean quarantine(ConsumerRecord record, Throwable reason); + + /** The default: nothing is configured, so nothing may be committed past. */ + static KafkaQuarantinePublisher unavailable() { + return (record, reason) -> false; + } +} diff --git a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaRetryMetadataMapper.java b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaRetryMetadataMapper.java index 4bde99c..f04c616 100644 --- a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaRetryMetadataMapper.java +++ b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaRetryMetadataMapper.java @@ -72,9 +72,22 @@ public final class KafkaRetryMetadataMapper { return 1; } try { - return Math.max(1, Integer.parseInt(value.orElseThrow().value())); - } catch (NumberFormatException exception) { - return 1; + int attempt = Integer.parseInt(value.orElseThrow().value()); + if (attempt < 1) { + throw new NumberFormatException("attempt " + attempt); + } + return attempt; + } catch (NumberFormatException malformed) { + // Fail closed rather than resetting to 1. + // + // Returning 1 for an unreadable header restarts the retry budget on every redelivery, so a + // message that should have been dead-lettered after N attempts is retried forever — and the + // header is caller-influenced, which makes "unreadable" a way to defeat the cap rather than + // an accident. + throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException( + "RETRY_ATTEMPT_MALFORMED", + "the retry attempt header is not a positive integer; the message is quarantined rather" + + " than restarting its retry budget"); } } } diff --git a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaSecurityConfigurer.java b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaSecurityConfigurer.java index 04255c2..2e1d672 100644 --- a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaSecurityConfigurer.java +++ b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaSecurityConfigurer.java @@ -93,11 +93,15 @@ public final class KafkaSecurityConfigurer { properties.put(SASL_JAAS_CONFIG, scramJaas(scram.credentialId(), resolved)); } case BrokerCredentialProfile.OAuth2 oauth -> { - credentials.resolve(oauth.credentialId(), now); - properties.put(SASL_MECHANISM, "OAUTHBEARER"); - properties.put( - SASL_JAAS_CONFIG, - "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;"); + // Refused rather than half-configured. Setting the mechanism name without a callback + // handler produces a client that authenticates with nothing and fails at connect time, + // which is later and harder to attribute than failing here. + throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException( + "SASL_OAUTH_NOT_CONFIGURED", + "OAuth bearer needs a login callback handler and a token endpoint; this platform" + + " configures the mechanism name only, so the profile is refused rather than" + + " producing a client that cannot authenticate: " + + oauth.credentialId()); } case BrokerCredentialProfile.MutualTls ignored -> properties.put(SASL_MECHANISM, "NONE"); case BrokerCredentialProfile.UsernamePassword plain -> { @@ -122,14 +126,48 @@ public final class KafkaSecurityConfigurer { } private static String scramJaas(String credentialId, CredentialRuntime resolved) { - return "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"%s\" " - .formatted(credentialId) - + "password=\"%s\";".formatted(new String(resolved.material())); + return jaas("org.apache.kafka.common.security.scram.ScramLoginModule", credentialId, resolved); } private static String plainJaas(String credentialId, CredentialRuntime resolved) { - return "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"%s\" " - .formatted(credentialId) - + "password=\"%s\";".formatted(new String(resolved.material())); + return jaas("org.apache.kafka.common.security.plain.PlainLoginModule", credentialId, resolved); + } + + /** + * Builds a JAAS entry with both values escaped. + * + *

    The password was interpolated into a quoted string unescaped. A perfectly ordinary generated + * secret containing a quote or a backslash then produced a JAAS config that either fails to parse + * — an authentication error that names nothing useful — or, with a quote and a semicolon, + * terminates the option early and injects another one. A secret is attacker-influenced in exactly + * one sense that matters here: nobody reads it before it is used. + */ + private static String jaas(String loginModule, String credentialId, CredentialRuntime resolved) { + return loginModule + + " required username=\"" + + escapeJaas(credentialId) + + "\" password=\"" + + escapeJaas(new String(resolved.material())) + + "\";"; + } + + /** + * Escapes a JAAS quoted-string value. + * + *

    Backslash first, then quote — the other order double-escapes the backslash it just added. + * Control characters are refused rather than escaped: JAAS has no escape for them, and a secret + * containing one cannot be expressed in this format at all. + */ + private static String escapeJaas(String value) { + for (int index = 0; index < value.length(); index++) { + char character = value.charAt(index); + if (character < 0x20 || character == 0x7F) { + throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException( + "JAAS_VALUE_UNREPRESENTABLE", + "a credential containing a control character cannot be expressed in a JAAS entry;" + + " configure the client with a callback handler instead"); + } + } + return value.replace("\\", "\\\\").replace("\"", "\\\""); } } diff --git a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaSettlementCommand.java b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaSettlementCommand.java index a90e296..26c50a4 100644 --- a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaSettlementCommand.java +++ b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaSettlementCommand.java @@ -15,7 +15,18 @@ import org.apache.kafka.common.TopicPartition; * @param offset the record offset * @param kind what to do */ -public record KafkaSettlementCommand(TopicPartition partition, long offset, Kind kind) { +public record KafkaSettlementCommand(TopicPartition partition, long offset, Kind kind, long epoch) { + + /** + * A command without an epoch, for callers that predate assignment fencing. + * + *

    Epoch 0 never matches a live assignment, so such a command is always stale. That is + * deliberate: a settlement that cannot say which assignment it belongs to cannot be shown to + * belong to the current one. + */ + public KafkaSettlementCommand(TopicPartition partition, long offset, Kind kind) { + this(partition, offset, kind, 0L); + } /** What a queued settlement instruction asks the poll thread to do. */ public enum Kind { diff --git a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaTransactionalPublisher.java b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaTransactionalPublisher.java index 3b83b8e..18ff368 100644 --- a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaTransactionalPublisher.java +++ b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaTransactionalPublisher.java @@ -78,6 +78,67 @@ public final class KafkaTransactionalPublisher { * @param profiles the destination profile for each output destination * @param inputOffsets the input offsets to commit with the outputs */ + /** + * Begins a transaction, runs the caller's work inside it, then sends outputs and offsets. + * + *

    The processor used to run the handler and only afterwards hand the delivery here — so {@code + * beginTransaction} happened *after* the handler had already finished. Anything the handler + * produced itself, and any external effect it triggered, was therefore outside the transaction + * the documentation promised it was in: a handler that succeeded and a commit that then failed + * left the handler's work applied with its input offsets unsent. + * + *

    The body runs between begin and commit, and every failure path aborts. + * + * @param body the handler, executed inside the transaction + * @return whatever the body produced + */ + public T inTransaction( + KafkaTransactionalDelivery delivery, + Map profiles, + Map inputOffsets, + java.util.function.Supplier body) { + Objects.requireNonNull(delivery, "delivery must not be null"); + Objects.requireNonNull(profiles, "profiles must not be null"); + Objects.requireNonNull(inputOffsets, "inputOffsets must not be null"); + Objects.requireNonNull(body, "body must not be null"); + + producer.beginTransaction(); + try { + T value = body.get(); + sendOutputsAndOffsets(delivery, profiles, inputOffsets); + producer.commitTransaction(); + return value; + } catch (RuntimeException failure) { + // Handler failure, send failure, offset failure and commit failure all land here. Under + // read_committed a consumer sees none of the outputs, and the input offsets are unsent, so + // the source is redelivered. + producer.abortTransaction(); + throw failure; + } + } + + @SuppressWarnings("FutureReturnValueIgnored") + private void sendOutputsAndOffsets( + KafkaTransactionalDelivery delivery, + Map profiles, + Map inputOffsets) { + for (KafkaTransactionalOutput output : delivery.outputs()) { + DestinationProfile profile = profiles.get(output.destination().value()); + if (profile == null) { + throw new MessagingConfigurationException( + "DESTINATION_NOT_REGISTERED", + "no destination profile is registered for " + output.destination().value()); + } + producer.send( + mapper.toRecord( + new dev.caskeleton.messaging.transport.TransportPublishRequest( + profile, + output.envelope(), + dev.caskeleton.messaging.api.publish.PublishOptions.defaults()))); + } + producer.sendOffsetsToTransaction(inputOffsets, groupMetadata.get()); + } + @SuppressWarnings("FutureReturnValueIgnored") public void sendInTransaction( KafkaTransactionalDelivery delivery, diff --git a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/SpringKafkaTransactionalProcessor.java b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/SpringKafkaTransactionalProcessor.java index 84f080b..bdd6d3d 100644 --- a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/SpringKafkaTransactionalProcessor.java +++ b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/SpringKafkaTransactionalProcessor.java @@ -51,8 +51,15 @@ public final class SpringKafkaTransactionalProcessor implements KafkaTransaction Objects.requireNonNull(handler, "handler must not be null"); try { - T result = handler.apply(delivery); - publisher.sendInTransaction(delivery, profiles, inputOffsets.apply(delivery)); + // The handler runs between beginTransaction and commitTransaction. + // + // It used to run first, with the transaction opened afterwards inside sendInTransaction — so + // the contract this interface documents ("the callback executes inside the Kafka + // transaction") was false for every output the handler produced itself and every external + // effect it triggered. + T result = + publisher.inTransaction( + delivery, profiles, inputOffsets.apply(delivery), () -> handler.apply(delivery)); return CompletableFuture.completedFuture(result); } catch (RuntimeException exception) { return CompletableFuture.failedFuture(exception); diff --git a/messaging/messaging-kafka/src/test/java/dev/caskeleton/messaging/kafka/KafkaConsumerRegistrarTest.java b/messaging/messaging-kafka/src/test/java/dev/caskeleton/messaging/kafka/KafkaConsumerRegistrarTest.java index 94324e3..f8e30bd 100644 --- a/messaging/messaging-kafka/src/test/java/dev/caskeleton/messaging/kafka/KafkaConsumerRegistrarTest.java +++ b/messaging/messaging-kafka/src/test/java/dev/caskeleton/messaging/kafka/KafkaConsumerRegistrarTest.java @@ -2,6 +2,7 @@ package dev.caskeleton.messaging.kafka; import static org.assertj.core.api.Assertions.assertThat; +import dev.caskeleton.messaging.api.MessageId; import dev.caskeleton.messaging.api.header.ReservedHeaders; import dev.caskeleton.messaging.transport.GracefulShutdownCoordinator; import dev.caskeleton.messaging.transport.TransportConsumerSpec; @@ -14,7 +15,6 @@ import java.util.ArrayList; import java.util.Deque; import java.util.List; import java.util.Map; -import java.util.UUID; import java.util.concurrent.CompletableFuture; import java.util.concurrent.CompletionStage; import java.util.concurrent.Executor; @@ -22,6 +22,7 @@ import org.apache.kafka.clients.consumer.ConsumerRecord; import org.apache.kafka.clients.consumer.MockConsumer; import org.apache.kafka.common.TopicPartition; import org.apache.kafka.common.header.internals.RecordHeaders; +import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; /** @@ -157,10 +158,18 @@ class KafkaConsumerRegistrarTest { KafkaConsumerRegistrar registrar = registrar(this::settleImmediately); assign(); - registrar.pause("").toCompletableFuture().join(); + // pause/resume are queued for the poll thread rather than touching the Consumer from the + // caller's thread — the Kafka consumer is not thread-safe, and this class documents itself as + // poll-thread-only. The stage therefore completes on the next poll cycle, which is what + // "applied" means for a runtime whose loop is running. + java.util.concurrent.CompletionStage paused = registrar.pause(""); + registrar.pollOnce(NOW); + paused.toCompletableFuture().join(); assertThat(consumer.paused()).contains(PARTITION); - registrar.resume("").toCompletableFuture().join(); + java.util.concurrent.CompletionStage resumed = registrar.resume(""); + registrar.pollOnce(NOW); + resumed.toCompletableFuture().join(); assertThat(consumer.paused()).doesNotContain(PARTITION); } @@ -175,9 +184,174 @@ class KafkaConsumerRegistrarTest { registrar.pollOnce(NOW); assertThat(handled).isEmpty(); + assertThat(registrar.committedOffset(PARTITION)) + .as("PARKED completed the offset with no quarantine write, so the record was simply lost") + .isEmpty(); + } + + @Test + @DisplayName("an undecodable record commits only once quarantine confirms the write") + void anUndecodableRecordCommitsAfterQuarantineConfirms() { + java.util.List quarantined = new java.util.ArrayList<>(); + KafkaConsumerRegistrar registrar = + new KafkaConsumerRegistrar( + new TransportConsumerSpec(KafkaContractHarness.profile(), this::settleImmediately), + consumer, + handlers, + new GracefulShutdownCoordinator(Duration.ofSeconds(30)), + Duration.ofMillis(1), + (record, reason) -> { + quarantined.add(record.offset()); + return true; + }); + assign(); + addRecordWithoutIdentityHeaders(10); + + registrar.pollOnce(NOW); + handlers.runAll(); + registrar.pollOnce(NOW); + + assertThat(quarantined).containsExactly(10L); assertThat(registrar.committedOffset(PARTITION)).hasValue(11L); } + @Test + @DisplayName("one partition hitting its ceiling does not abandon the other's records") + void oneSaturatedPartitionDoesNotAbandonAnother() { + org.apache.kafka.common.TopicPartition second = + new org.apache.kafka.common.TopicPartition(PARTITION.topic(), PARTITION.partition() + 1); + consumer.assign(List.of(PARTITION, second)); + consumer.updateBeginningOffsets(Map.of(PARTITION, 10L, second, 10L)); + consumer.seek(PARTITION, 10L); + consumer.seek(second, 10L); + KafkaConsumerRegistrar registrar = registrar(this::settleImmediately); + // Two records on the first partition (its ordering unit allows one in flight) and one on the + // second. The old loop broke out of the whole batch at the first record it could not submit. + addRecord(10); + addRecord(11); + addRecordOn(second, 10); + + int dispatched = registrar.pollOnce(NOW); + + assertThat(dispatched) + .as("the second partition's record was returned by the same poll and must not be dropped") + .isEqualTo(2); + } + + @Test + @DisplayName("a rejected executor submission leaves nothing in flight and re-reads the record") + void aRejectedSubmissionIsUndone() { + KafkaConsumerRegistrar registrar = + new KafkaConsumerRegistrar( + new TransportConsumerSpec(KafkaContractHarness.profile(), this::settleImmediately), + consumer, + command -> { + throw new java.util.concurrent.RejectedExecutionException("pool is shutting down"); + }, + new GracefulShutdownCoordinator(Duration.ofSeconds(30)), + Duration.ofMillis(1)); + assign(); + addRecord(10); + + int dispatched = registrar.pollOnce(NOW); + + assertThat(dispatched).isZero(); + assertThat(registrar.committedOffset(PARTITION)) + .as("an offset registered as delivered with no worker behind it never completes") + .isEmpty(); + assertThat(consumer.position(PARTITION)) + .as("the record must be re-read next cycle") + .isEqualTo(10L); + } + + @Test + @DisplayName("a settlement from a revoked assignment is refused") + void aStaleSettlementIsRefused() { + KafkaConsumerRegistrar registrar = registrar(this::deferSettlement); + assign(); + addRecord(10); + registrar.pollOnce(NOW); + + // The partition goes away and comes back: a new assignment, a new epoch. + consumer.assign(List.of()); + registrar.pollOnce(NOW); + assign(); + registrar.pollOnce(NOW); + + // Now the old handler finally acknowledges. + handlers.runAll(); + registrar.pollOnce(NOW); + + assertThat(registrar.staleSettlements()) + .as("an acknowledgement from the previous assignment must not move this one's watermark") + .isPositive(); + assertThat(registrar.committedOffset(PARTITION)).isEmpty(); + } + + @Test + @DisplayName("a second terminal call on one delivery is refused") + void aSecondTerminalCallIsRefused() { + java.util.List refusals = new java.util.ArrayList<>(); + KafkaConsumerRegistrar registrar = + registrar( + delivery -> { + delivery.settlement().acknowledge().toCompletableFuture().join(); + delivery + .settlement() + .requeue(Duration.ZERO) + .toCompletableFuture() + .handle( + (result, failure) -> { + if (failure != null) { + refusals.add(failure); + } + return null; + }) + .join(); + return java.util.concurrent.CompletableFuture.completedFuture(null); + }); + assign(); + addRecord(10); + + registrar.pollOnce(NOW); + handlers.runAll(); + + assertThat(refusals) + .as("telling the poll thread both 'done' and 're-read this' lets the second one win") + .hasSize(1); + } + + private void addRecordOn(org.apache.kafka.common.TopicPartition partition, long offset) { + RecordHeaders headers = new RecordHeaders(); + headers.add( + ReservedHeaders.MESSAGE_ID, + // A v7 identity: MessageId now enforces the version it documents, and a v4 here would be + // refused by the envelope mapper exactly as a foreign producer's would. + MessageId.newId().value().toString().getBytes(StandardCharsets.UTF_8)); + headers.add(ReservedHeaders.MESSAGE_TYPE, "order.created".getBytes(StandardCharsets.UTF_8)); + headers.add(ReservedHeaders.SCHEMA_VERSION, "1".getBytes(StandardCharsets.UTF_8)); + headers.add(ReservedHeaders.CONTENT_TYPE, "application/json".getBytes(StandardCharsets.UTF_8)); + consumer.addRecord( + new ConsumerRecord<>( + partition.topic(), + partition.partition(), + offset, + NOW.toEpochMilli(), + org.apache.kafka.common.record.TimestampType.CREATE_TIME, + 0, + 0, + null, + "{}".getBytes(StandardCharsets.UTF_8), + headers, + java.util.Optional.empty())); + } + + /** Never settles inside the handler, so the settlement arrives later. */ + private CompletionStage deferSettlement(TransportDelivery delivery) { + delivery.settlement().acknowledge(); + return java.util.concurrent.CompletableFuture.completedFuture(null); + } + private void assign() { consumer.assign(List.of(PARTITION)); consumer.updateBeginningOffsets(Map.of(PARTITION, 10L)); @@ -187,7 +361,10 @@ class KafkaConsumerRegistrarTest { private void addRecord(long offset) { RecordHeaders headers = new RecordHeaders(); headers.add( - ReservedHeaders.MESSAGE_ID, UUID.randomUUID().toString().getBytes(StandardCharsets.UTF_8)); + ReservedHeaders.MESSAGE_ID, + // A v7 identity: MessageId now enforces the version it documents, and a v4 here would be + // refused by the envelope mapper exactly as a foreign producer's would. + MessageId.newId().value().toString().getBytes(StandardCharsets.UTF_8)); headers.add(ReservedHeaders.MESSAGE_TYPE, "order.created".getBytes(StandardCharsets.UTF_8)); headers.add(ReservedHeaders.SCHEMA_VERSION, "1".getBytes(StandardCharsets.UTF_8)); headers.add(ReservedHeaders.CONTENT_TYPE, "application/json".getBytes(StandardCharsets.UTF_8)); diff --git a/messaging/messaging-kafka/src/test/java/dev/caskeleton/messaging/kafka/KafkaSecurityConfigurerTest.java b/messaging/messaging-kafka/src/test/java/dev/caskeleton/messaging/kafka/KafkaSecurityConfigurerTest.java new file mode 100644 index 0000000..e0aea61 --- /dev/null +++ b/messaging/messaging-kafka/src/test/java/dev/caskeleton/messaging/kafka/KafkaSecurityConfigurerTest.java @@ -0,0 +1,105 @@ +package dev.caskeleton.messaging.kafka; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import dev.caskeleton.messaging.api.error.MessagingConfigurationException; +import dev.caskeleton.messaging.security.BrokerCredentialProfile; +import dev.caskeleton.messaging.security.BrokerSecurityProfile; +import dev.caskeleton.messaging.security.CredentialProvider; +import dev.caskeleton.messaging.security.CredentialRuntimeRegistry; +import java.time.Duration; +import java.time.Instant; +import java.util.Map; +import java.util.Optional; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** + * What reaches a JAAS entry, and what is refused outright. + * + *

    The password was interpolated into a quoted string with no escaping. A generated secret + * containing a quote or a backslash produced a config that fails to parse — an authentication error + * naming nothing useful — and one containing a quote and a semicolon terminated the option early + * and injected another. + */ +class KafkaSecurityConfigurerTest { + + private static final Instant NOW = Instant.parse("2026-08-14T00:00:00Z"); + + @Test + @DisplayName("a secret containing a quote and a backslash is escaped, not injected") + void aSecretWithQuotesIsEscaped() { + Map properties = configure("pa\"ss\\word;"); + + String jaas = String.valueOf(properties.get("sasl.jaas.config")); + assertThat(jaas) + .as("an unescaped quote closes the option and everything after it becomes new options") + .contains("password=\"pa\\\"ss\\\\word;\";"); + assertThat(jaas) + .as("the raw form would close the password option and make the rest new options") + .doesNotContain("password=\"pa\""); + } + + @Test + @DisplayName("a secret containing a control character is refused") + void aSecretWithAControlCharacterIsRefused() { + assertThatThrownBy(() -> configure("pass\nword")) + .isInstanceOf(MessagingConfigurationException.class) + .hasMessageContaining("control character"); + } + + @Test + @DisplayName("an OAuth profile is refused rather than half-configured") + void anOAuthProfileIsRefused() { + CredentialRuntimeRegistry credentials = registry("secret"); + + BrokerCredentialProfile oauth = new BrokerCredentialProfile.OAuth2("kafka-oauth"); + assertThatThrownBy( + () -> + new KafkaSecurityConfigurer( + credentials, new dev.caskeleton.messaging.security.BrokerTlsPolicy()) + .configure(profile(oauth), oauth, java.util.List.of("TLSv1.3"), NOW)) + .as("naming the mechanism without a callback handler fails at connect time instead") + .isInstanceOf(MessagingConfigurationException.class) + .hasMessageContaining("callback handler"); + } + + private static Map configure(String secret) { + return new KafkaSecurityConfigurer( + registry(secret), new dev.caskeleton.messaging.security.BrokerTlsPolicy()) + .configure( + profile(new BrokerCredentialProfile.SaslScram("kafka-app")), + new BrokerCredentialProfile.SaslScram("kafka-app"), + java.util.List.of("TLSv1.3"), + NOW); + } + + private static BrokerSecurityProfile profile(BrokerCredentialProfile credential) { + return new BrokerSecurityProfile( + "kafka-primary", + true, + true, + true, + credential, + credential, + Optional.empty(), + dev.caskeleton.messaging.security.DestinationAccessPolicy.denyAll()); + } + + private static CredentialRuntimeRegistry registry(String secret) { + return new CredentialRuntimeRegistry( + new CredentialProvider() { + @Override + public char[] resolve(String credentialId) { + return secret.toCharArray(); + } + + @Override + public Optional expiresAt(String credentialId) { + return Optional.of(NOW.plus(Duration.ofHours(1))); + } + }, + Duration.ofMinutes(5)); + } +} diff --git a/messaging/messaging-kafka/src/test/java/dev/caskeleton/messaging/kafka/KafkaTransactionOrderingTest.java b/messaging/messaging-kafka/src/test/java/dev/caskeleton/messaging/kafka/KafkaTransactionOrderingTest.java new file mode 100644 index 0000000..c12f63f --- /dev/null +++ b/messaging/messaging-kafka/src/test/java/dev/caskeleton/messaging/kafka/KafkaTransactionOrderingTest.java @@ -0,0 +1,192 @@ +package dev.caskeleton.messaging.kafka; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import dev.caskeleton.messaging.api.ContentType; +import dev.caskeleton.messaging.api.MessageEnvelope; +import dev.caskeleton.messaging.api.MessageId; +import dev.caskeleton.messaging.api.MessageType; +import dev.caskeleton.messaging.api.ProducerId; +import dev.caskeleton.messaging.api.SchemaVersion; +import dev.caskeleton.messaging.api.TraceContext; +import dev.caskeleton.messaging.api.header.MessageHeaders; +import dev.caskeleton.messaging.policy.DestinationProfile; +import dev.caskeleton.messaging.schema.EncodedMessage; +import java.nio.charset.StandardCharsets; +import java.time.Instant; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.atomic.AtomicBoolean; +import org.apache.kafka.clients.consumer.ConsumerGroupMetadata; +import org.apache.kafka.clients.producer.MockProducer; +import org.apache.kafka.common.serialization.ByteArraySerializer; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** + * Where the handler runs relative to {@code beginTransaction}. + * + *

    {@code KafkaTransactionalProcessor} documents that the callback executes inside the Kafka + * transaction. It did not: the processor ran the handler and only afterwards called {@code + * sendInTransaction}, which is where {@code beginTransaction} lived. Everything the handler + * produced itself — and every external effect it triggered — was outside the transaction, so a + * handler that succeeded followed by a failed commit left its work applied with the input offsets + * unsent. + */ +class KafkaTransactionOrderingTest { + + private final RecordingProducer producer = new RecordingProducer(); + + private final KafkaTransactionalPublisher publisher = + new KafkaTransactionalPublisher( + producer, () -> new ConsumerGroupMetadata("order-projection")); + + @Test + @DisplayName("the transaction is open while the handler runs") + void theTransactionIsOpenWhileTheHandlerRuns() { + AtomicBoolean openDuringHandler = new AtomicBoolean(); + SpringKafkaTransactionalProcessor processor = processor(); + + processor + .processInTransaction( + delivery(), + ignored -> { + // The real publisher has already begun the transaction by the time the handler runs. + openDuringHandler.set(producer.calls().equals(List.of("begin"))); + return "handled"; + }) + .toCompletableFuture() + .join(); + + assertThat(openDuringHandler) + .as("the handler used to run before beginTransaction was ever called") + .isTrue(); + assertThat(producer.calls()).containsExactly("begin", "offsets", "commit"); + } + + @Test + @DisplayName("a handler failure aborts and never commits") + void aHandlerFailureAbortsAndNeverCommits() { + SpringKafkaTransactionalProcessor processor = processor(); + + assertThatThrownBy( + () -> + processor + .processInTransaction( + delivery(), + ignored -> { + throw new IllegalStateException("the projection is down"); + }) + .toCompletableFuture() + .join()) + .hasCauseInstanceOf(IllegalStateException.class); + + assertThat(producer.calls()) + .as("under read_committed nothing may become visible, and the offsets stay unsent") + .containsExactly("begin", "abort"); + } + + @Test + @DisplayName("a commit failure aborts") + void aCommitFailureAborts() { + producer.failCommit(); + SpringKafkaTransactionalProcessor processor = processor(); + + assertThatThrownBy( + () -> + processor + .processInTransaction(delivery(), ignored -> "handled") + .toCompletableFuture() + .join()) + .hasCauseInstanceOf(IllegalStateException.class); + + assertThat(producer.calls()).endsWith("abort"); + } + + /** A MockProducer that records the transaction calls in the order they happen. */ + private static final class RecordingProducer extends MockProducer { + + private final List calls = new java.util.ArrayList<>(); + + private boolean commitFails; + + private RecordingProducer() { + // The four-arg constructor with a null partitioner: this producer never partitions, because + // the assertions are about transaction call order rather than record placement. + super(true, null, new ByteArraySerializer(), new ByteArraySerializer()); + initTransactions(); + } + + private void failCommit() { + commitFails = true; + } + + private List calls() { + return List.copyOf(calls); + } + + @Override + public void beginTransaction() { + calls.add("begin"); + super.beginTransaction(); + } + + @Override + public void sendOffsetsToTransaction( + Map< + org.apache.kafka.common.TopicPartition, + org.apache.kafka.clients.consumer.OffsetAndMetadata> + offsets, + ConsumerGroupMetadata groupMetadata) { + calls.add("offsets"); + super.sendOffsetsToTransaction(offsets, groupMetadata); + } + + @Override + public void commitTransaction() { + if (commitFails) { + throw new IllegalStateException("commit failed"); + } + calls.add("commit"); + super.commitTransaction(); + } + + @Override + public void abortTransaction() { + calls.add("abort"); + super.abortTransaction(); + } + } + + private SpringKafkaTransactionalProcessor processor() { + DestinationProfile profile = KafkaContractHarness.profile(); + return new SpringKafkaTransactionalProcessor( + publisher, Map.of(profile.name().value(), profile), ignored -> Map.of()); + } + + private static KafkaTransactionalDelivery delivery() { + return new KafkaTransactionalDelivery(envelope(), List.of()); + } + + private static MessageEnvelope envelope() { + return new MessageEnvelope<>( + MessageId.newId(), + new MessageType("order.created"), + new SchemaVersion(1), + Instant.parse("2026-08-15T00:00:00Z"), + Optional.empty(), + new ProducerId("orders"), + Optional.empty(), + Optional.empty(), + ContentType.JSON, + Optional.empty(), + Optional.empty(), + Optional.empty(), + TraceContext.none(), + MessageHeaders.empty(), + new EncodedMessage( + "{}".getBytes(StandardCharsets.UTF_8), ContentType.JSON, Optional.empty())); + } +} diff --git a/messaging/messaging-kafka/src/test/java/dev/caskeleton/messaging/kafka/ReservedHeaderForgeryTest.java b/messaging/messaging-kafka/src/test/java/dev/caskeleton/messaging/kafka/ReservedHeaderForgeryTest.java new file mode 100644 index 0000000..b60045b --- /dev/null +++ b/messaging/messaging-kafka/src/test/java/dev/caskeleton/messaging/kafka/ReservedHeaderForgeryTest.java @@ -0,0 +1,123 @@ +package dev.caskeleton.messaging.kafka; + +import static org.assertj.core.api.Assertions.assertThatCode; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import dev.caskeleton.messaging.api.ContentType; +import dev.caskeleton.messaging.api.MessageEnvelope; +import dev.caskeleton.messaging.api.MessageId; +import dev.caskeleton.messaging.api.MessageType; +import dev.caskeleton.messaging.api.ProducerId; +import dev.caskeleton.messaging.api.SchemaVersion; +import dev.caskeleton.messaging.api.TraceContext; +import dev.caskeleton.messaging.api.error.MessagingConfigurationException; +import dev.caskeleton.messaging.api.header.HeaderName; +import dev.caskeleton.messaging.api.header.HeaderValue; +import dev.caskeleton.messaging.api.header.MessageHeaders; +import dev.caskeleton.messaging.api.header.ReservedHeaders; +import dev.caskeleton.messaging.schema.EncodedMessage; +import java.nio.charset.StandardCharsets; +import java.time.Instant; +import java.util.Map; +import java.util.Optional; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** + * What happens when message content claims a reserved header. + * + *

    The canonical headers were written first and the envelope's own headers appended afterwards, + * and the consumer reads {@code lastHeader}. A row in the outbox carrying {@code msg.id} therefore + * overwrote the canonical identity on the wire — and that identity is what inbox deduplication and + * the audit trail key on, so a forged one corrupts a *different* message's dedup state. + */ +class ReservedHeaderForgeryTest { + + private final KafkaHeaderMapper mapper = new KafkaHeaderMapper(); + + @Test + @DisplayName("an application header in the reserved namespace is refused") + void aForgedReservedHeaderIsRefused() { + assertThatThrownBy( + () -> mapper.toKafkaHeaders(envelopeWithHeader(ReservedHeaders.MESSAGE_ID, "forged"))) + .isInstanceOf(MessagingConfigurationException.class) + .hasMessageContaining("reserved namespace"); + } + + @Test + @DisplayName("a forged message type is refused too") + void aForgedMessageTypeIsRefused() { + assertThatThrownBy( + () -> + mapper.toKafkaHeaders( + envelopeWithHeader(ReservedHeaders.MESSAGE_TYPE, "other.type"))) + .isInstanceOf(MessagingConfigurationException.class); + } + + @Test + @DisplayName("an ordinary application header still travels") + void anOrdinaryApplicationHeaderStillTravels() { + assertThatCode(() -> mapper.toKafkaHeaders(envelopeWithHeader("x-tenant-hint", "acme"))) + .doesNotThrowAnyException(); + } + + @Test + @DisplayName("a malformed retry attempt is quarantined, not reset to one") + void aMalformedRetryAttemptIsQuarantined() { + assertThatThrownBy( + () -> + new KafkaRetryMetadataMapper() + .attemptOf( + envelopeWithPlatformHeader(ReservedHeaders.RETRY_ATTEMPT, "not-a-number"))) + .as("resetting to 1 restarts the budget on every redelivery, so the cap never applies") + .isInstanceOf(MessagingConfigurationException.class) + .hasMessageContaining("retry attempt"); + } + + @Test + @DisplayName("a negative retry attempt is quarantined") + void aNegativeRetryAttemptIsQuarantined() { + assertThatThrownBy( + () -> + new KafkaRetryMetadataMapper() + .attemptOf(envelopeWithPlatformHeader(ReservedHeaders.RETRY_ATTEMPT, "-3"))) + .isInstanceOf(MessagingConfigurationException.class); + } + + /** + * Headers built through the platform factory, which is the path an outbox row takes. + * + *

    {@code MessageHeaders.application} already refuses reserved names, so business code could + * never forge one. The gap was the platform factory: {@code OutboxEnvelopeFactory} promoted a + * row's arbitrary {@code Map} straight through it, and the mapper then appended + * the result after the canonical headers. + */ + private static MessageEnvelope envelopeWithHeader(String name, String value) { + return envelope(MessageHeaders.platform(Map.of(new HeaderName(name), new HeaderValue(value)))); + } + + private static MessageEnvelope envelopeWithPlatformHeader( + String name, String value) { + return envelope(MessageHeaders.platform(Map.of(new HeaderName(name), new HeaderValue(value)))); + } + + private static MessageEnvelope envelope(MessageHeaders headers) { + return new MessageEnvelope<>( + MessageId.newId(), + new MessageType("order.created"), + new SchemaVersion(1), + Instant.parse("2026-08-15T00:00:00Z"), + Optional.empty(), + new ProducerId("orders"), + Optional.empty(), + Optional.empty(), + ContentType.JSON, + Optional.empty(), + Optional.empty(), + Optional.empty(), + TraceContext.none(), + headers, + new EncodedMessage( + "{}".getBytes(StandardCharsets.UTF_8), ContentType.JSON, Optional.empty())); + } +} diff --git a/messaging/messaging-nats-experimental/gradle.lockfile b/messaging/messaging-nats-experimental/gradle.lockfile index 2000bd2..328f63a 100644 --- a/messaging/messaging-nats-experimental/gradle.lockfile +++ b/messaging/messaging-nats-experimental/gradle.lockfile @@ -32,9 +32,9 @@ commons-io:commons-io:2.21.0=spotbugs info.picocli:picocli:4.7.7=checkstyle io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnotationProcessor io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor -io.micrometer:micrometer-commons:1.16.0=runtimeClasspath,testRuntimeClasspath -io.micrometer:micrometer-core:1.16.0=runtimeClasspath,testRuntimeClasspath -io.micrometer:micrometer-observation:1.16.0=runtimeClasspath,testRuntimeClasspath +io.micrometer:micrometer-commons:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.micrometer:micrometer-core:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.micrometer:micrometer-observation:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath io.nats:jnats:2.26.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath javax.inject:javax.inject:1=annotationProcessor,testAnnotationProcessor jaxen:jaxen:2.0.0=spotbugs diff --git a/messaging/messaging-nats-experimental/src/main/java/dev/caskeleton/messaging/nats/NatsJetStreamProfileValidator.java b/messaging/messaging-nats-experimental/src/main/java/dev/caskeleton/messaging/nats/NatsJetStreamProfileValidator.java index ca16a57..e5aaf17 100644 --- a/messaging/messaging-nats-experimental/src/main/java/dev/caskeleton/messaging/nats/NatsJetStreamProfileValidator.java +++ b/messaging/messaging-nats-experimental/src/main/java/dev/caskeleton/messaging/nats/NatsJetStreamProfileValidator.java @@ -41,7 +41,7 @@ public final class NatsJetStreamProfileValidator { throw new MessagingCapabilityUnavailableException( "NATS_DISABLED", "the NATS JetStream adapter is experimental and disabled unless " - + "backend.messaging.experimental.nats=true"); + + "app.messaging.experimental.nats=true"); } if (profile.deliveryGuarantee() == DeliveryGuarantee.AT_LEAST_ONCE && !jetStreamEnabled) { throw new IllegalArgumentException( diff --git a/messaging/messaging-nats-experimental/src/main/java/dev/caskeleton/messaging/nats/NatsJetStreamTransport.java b/messaging/messaging-nats-experimental/src/main/java/dev/caskeleton/messaging/nats/NatsJetStreamTransport.java index e14e647..bf8f5c8 100644 --- a/messaging/messaging-nats-experimental/src/main/java/dev/caskeleton/messaging/nats/NatsJetStreamTransport.java +++ b/messaging/messaging-nats-experimental/src/main/java/dev/caskeleton/messaging/nats/NatsJetStreamTransport.java @@ -39,6 +39,13 @@ import java.util.function.Function; * stored it and lost only the acknowledgement, and the deduplication window is what makes retrying * it safe when the profile enables one. * + *

    This is an Experimental contract seam, not a Stable adapter. It exercises the transport SPI + * against a publish operation the application supplies; it does not ship a NATS client bridge, + * connection management, or reconnection. Only a failure the operation reports as a {@link + * NatsPreSendRejection} is treated as definitely not transmitted — every other failure, recognised + * or not, is ambiguous, because a caller that reads {@code REJECTED} may republish under a new + * identity and duplicate a message the server already stored. + * *

    The adapter reports {@code nativeDeadLetter} as false. JetStream terminates a message at its * delivery limit rather than routing it anywhere, so the platform parks it via {@link * NatsMaxDeliverParkingWorkflow}; claiming a native dead-letter would send operators looking for a @@ -119,9 +126,20 @@ public final class NatsJetStreamTransport implements MessagingTransport { "PAYLOAD_TOO_LARGE", "encoded payload is " + size + " bytes, limit is " + limit)); } + // The caller's own deadline, honoured rather than ignored. Without it a stalled JetStream + // publish holds the caller for as long as the SDK's default takes, and the timeout the caller + // set to bound its own latency budget means nothing. + long startedAt = System.nanoTime(); return publish .publish(profile.subject(), deduplicationId(request), request) - .handle((position, failure) -> failure == null ? confirmed(position) : classify(failure)) + .toCompletableFuture() + .orTimeout( + request.options().timeout().toMillis(), java.util.concurrent.TimeUnit.MILLISECONDS) + .handle( + (position, failure) -> + failure == null + ? confirmed(position, elapsedSince(startedAt)) + : classify(failure, elapsedSince(startedAt))) .thenApply(TransportPublishResult::new); } @@ -180,7 +198,11 @@ public final class NatsJetStreamTransport implements MessagingTransport { .map(window -> request.envelope().messageId().value().toString()); } - private static PublishResult confirmed(NatsStreamPosition position) { + private static Duration elapsedSince(long startedAtNanos) { + return Duration.ofNanos(System.nanoTime() - startedAtNanos); + } + + private static PublishResult confirmed(NatsStreamPosition position, Duration elapsed) { return new PublishResult( PublishCompletion.CONFIRMED, // The publish acknowledgement names the stream and sequence the message landed in, which is @@ -193,45 +215,49 @@ public final class NatsJetStreamTransport implements MessagingTransport { RoutingOutcome.ROUTED, Optional.of(position), 1, - Duration.ZERO, + elapsed, Optional.empty()); } - private static PublishResult classify(Throwable failure) { + /** + * Decides what a publish failure means, from typed signals only. + * + *

    Only {@link NatsPreSendRejection} — thrown by the publish operation when it knows the send + * never left the client — reports {@code REJECTED / NOT_TRANSMITTED}. Everything else is {@code + * AMBIGUOUS}, including failures this adapter does not recognise, because the alternative is + * telling a caller nothing was transmitted when the server may already hold the message. + */ + private static PublishResult classify(Throwable failure, Duration elapsed) { Throwable cause = failure instanceof CompletionException ? failure.getCause() : failure; - String name = cause == null ? "" : cause.getClass().getSimpleName(); - - if (name.contains("Timeout")) { + if (cause instanceof NatsPreSendRejection rejection) { return new PublishResult( - PublishCompletion.AMBIGUOUS, - new PublishEvidence( - true, TransmissionEvidence.MAY_HAVE_BEEN_TRANSMITTED, false, ConfirmationLevel.NONE), - RoutingOutcome.UNKNOWN, + PublishCompletion.REJECTED, + PublishEvidence.notTransmitted(), + RoutingOutcome.NOT_APPLICABLE, Optional.empty(), 1, - Duration.ZERO, + elapsed, Optional.of( FailureDescriptor.of( - FailureCategory.TRANSIENT_INFRASTRUCTURE, - "NATS_PUBLISH_TIMEOUT", - "the publish acknowledgement did not arrive; the stream may already hold it"))); + FailureCategory.CONFIGURATION, rejection.code(), rejection.getMessage()))); } - // "no responders" means no JetStream server is listening on the subject at all. That is a - // configuration fault, not a transient one: retrying it will fail identically until the stream - // is declared. - boolean noStream = name.contains("NoRespond") || name.contains("Jetstream"); + boolean timedOut = cause instanceof java.util.concurrent.TimeoutException; return new PublishResult( - PublishCompletion.REJECTED, - PublishEvidence.notTransmitted(), - RoutingOutcome.NOT_APPLICABLE, + PublishCompletion.AMBIGUOUS, + new PublishEvidence( + true, TransmissionEvidence.MAY_HAVE_BEEN_TRANSMITTED, false, ConfirmationLevel.NONE), + RoutingOutcome.UNKNOWN, Optional.empty(), 1, - Duration.ZERO, + elapsed, Optional.of( FailureDescriptor.of( - noStream ? FailureCategory.CONFIGURATION : FailureCategory.PERMANENT_BUSINESS, - noStream ? "NATS_STREAM_NOT_DECLARED" : "NATS_PUBLISH_REJECTED", - "the JetStream publish was rejected: " + name))); + FailureCategory.TRANSIENT_INFRASTRUCTURE, + timedOut ? "NATS_PUBLISH_TIMEOUT" : "NATS_PUBLISH_OUTCOME_UNKNOWN", + timedOut + ? "the publish acknowledgement did not arrive; the stream may already hold it" + : "the publish failed in a way this adapter does not classify; the stream may" + + " already hold the message"))); } private static TransportPublishResult rejectedLocally(String code, String message) { @@ -242,6 +268,7 @@ public final class NatsJetStreamTransport implements MessagingTransport { RoutingOutcome.NOT_APPLICABLE, Optional.empty(), 1, + // A local refusal: nothing was attempted, so no time was spent on the wire. Duration.ZERO, Optional.of(FailureDescriptor.of(FailureCategory.PERMANENT_BUSINESS, code, message)))); } diff --git a/messaging/messaging-nats-experimental/src/main/java/dev/caskeleton/messaging/nats/NatsPreSendRejection.java b/messaging/messaging-nats-experimental/src/main/java/dev/caskeleton/messaging/nats/NatsPreSendRejection.java new file mode 100644 index 0000000..039addc --- /dev/null +++ b/messaging/messaging-nats-experimental/src/main/java/dev/caskeleton/messaging/nats/NatsPreSendRejection.java @@ -0,0 +1,65 @@ +package dev.caskeleton.messaging.nats; + +import java.util.Objects; + +/** + * The one signal that lets this adapter report {@code REJECTED / NOT_TRANSMITTED}. + * + *

    Classification used to work by reading the exception's class simple name and matching + * substrings — {@code "Timeout"} meant ambiguous, anything else meant rejected. Two things are + * wrong with that. A class name is not part of any SDK's contract: it changes between versions, and + * an unrelated exception whose name happens to contain the substring is misread. And defaulting the + * unknown case to {@code REJECTED} tells the caller nothing was transmitted, so a caller that + * republishes under a new identity duplicates a message the server may well have stored. + * + *

    The publish operation throws this only when it knows the send never left the client: an + * argument the SDK refused, a connection that was already closed, a subject the client could not + * resolve. Everything else — including anything the adapter does not recognise — is ambiguous, and + * ambiguity is the honest answer when the write may have landed. + */ +public final class NatsPreSendRejection extends RuntimeException { + + private static final long serialVersionUID = 1L; + + private final String code; + + /** + * Creates a pre-send rejection. + * + * @param code the sanitised platform code for the refusal + * @param message a sanitised description + */ + public NatsPreSendRejection(String code, String message) { + super(message); + this.code = requireCode(code); + } + + /** + * Creates a pre-send rejection wrapping the SDK's own exception. + * + * @param code the sanitised platform code for the refusal + * @param message a sanitised description + * @param cause the SDK exception that proves the send never started + */ + public NatsPreSendRejection(String code, String message, Throwable cause) { + super(message, cause); + this.code = requireCode(code); + } + + private static String requireCode(String code) { + Objects.requireNonNull(code, "code must not be null"); + if (code.isBlank()) { + throw new IllegalArgumentException("code must not be blank"); + } + return code; + } + + /** + * The sanitised code reported to the caller. + * + * @return the code + */ + public String code() { + return code; + } +} diff --git a/messaging/messaging-nats-experimental/src/test/java/dev/caskeleton/messaging/nats/NatsAdapterContractTest.java b/messaging/messaging-nats-experimental/src/test/java/dev/caskeleton/messaging/nats/NatsAdapterContractTest.java index 706cd17..1c0edc5 100644 --- a/messaging/messaging-nats-experimental/src/test/java/dev/caskeleton/messaging/nats/NatsAdapterContractTest.java +++ b/messaging/messaging-nats-experimental/src/test/java/dev/caskeleton/messaging/nats/NatsAdapterContractTest.java @@ -102,14 +102,76 @@ class NatsAdapterContractTest { } @Test - void aMissingStreamIsAConfigurationFaultNotATransientOne() { + void aFailureNamedLikeAKnownOneIsStillAmbiguous() { + // The class name used to be the classifier: an exception whose simple name contained + // "NoRespond" was reported REJECTED / NOT_TRANSMITTED. A name is not an SDK contract, and a + // caller that reads NOT_TRANSMITTED republishes under a new identity. PublishResult result = await(failingWith(new IllegalStateException("NoRespondersException")).publish(request(64))); - assertThat(result.completion()).isEqualTo(PublishCompletion.REJECTED); + assertThat(result.completion()).isEqualTo(PublishCompletion.AMBIGUOUS); assertThat(result.failure()) .hasValueSatisfying( - failure -> assertThat(failure.code()).isEqualTo("NATS_PUBLISH_REJECTED")); + failure -> assertThat(failure.code()).isEqualTo("NATS_PUBLISH_OUTCOME_UNKNOWN")); + } + + @Test + void onlyATypedPreSendRejectionReportsNothingTransmitted() { + PublishResult result = + await( + failingWith( + new NatsPreSendRejection( + "NATS_CONNECTION_CLOSED", "the client was closed before the send")) + .publish(request(64))); + + assertThat(result.completion()).isEqualTo(PublishCompletion.REJECTED); + assertThat(result.evidence().transmission()) + .isEqualTo(dev.caskeleton.messaging.api.publish.TransmissionEvidence.NOT_TRANSMITTED); + assertThat(result.failure()) + .hasValueSatisfying( + failure -> assertThat(failure.code()).isEqualTo("NATS_CONNECTION_CLOSED")); + } + + @Test + void aWrappedUnknownFailureIsAmbiguousRatherThanRejected() { + PublishResult result = + await( + failingWith( + new java.util.concurrent.CompletionException( + new RuntimeException("something the adapter has never seen"))) + .publish(request(64))); + + assertThat(result.completion()).isEqualTo(PublishCompletion.AMBIGUOUS); + } + + @Test + void aPublishThatNeverCompletesIsBoundedByTheCallersTimeout() { + NatsJetStreamTransport transport = + new NatsJetStreamTransport( + "nats-primary", + 1L, + profile(Optional.empty()), + (subject, deduplicationId, request) -> new java.util.concurrent.CompletableFuture<>()); + + PublishResult result = await(transport.publish(requestWithTimeout(Duration.ofMillis(50)))); + + assertThat(result.completion()) + .as("the caller's deadline used to be ignored entirely") + .isEqualTo(PublishCompletion.AMBIGUOUS); + assertThat(result.failure()) + .hasValueSatisfying( + failure -> assertThat(failure.code()).isEqualTo("NATS_PUBLISH_TIMEOUT")); + } + + @Test + void theReportedElapsedTimeIsMeasuredRatherThanZero() { + PublishResult result = + await( + failingWith(new java.util.concurrent.TimeoutException("no ack")).publish(request(64))); + + assertThat(result.elapsed()) + .as("every outcome reported Duration.ZERO, so latency evidence was fiction") + .isGreaterThanOrEqualTo(Duration.ZERO); } @Test @@ -190,14 +252,16 @@ class NatsAdapterContractTest { } @Test - void aFailureCategoryIsAlwaysAttachedToARejection() { + void aFailureCategoryIsAlwaysAttachedToAFailure() { PublishResult result = await(failingWith(new IllegalStateException("boom")).publish(request(64))); assertThat(result.failure()) .hasValueSatisfying( failure -> - assertThat(failure.category()).isEqualTo(FailureCategory.PERMANENT_BUSINESS)); + assertThat(failure.category()) + .as("an unclassified failure is transient, not a permanent business rejection") + .isEqualTo(FailureCategory.TRANSIENT_INFRASTRUCTURE)); } private static NatsJetStreamProfile profile(Optional deduplicationWindow) { @@ -216,6 +280,15 @@ class NatsAdapterContractTest { destinationProfile(), envelope(payloadBytes), PublishOptions.defaults()); } + private static TransportPublishRequest requestWithTimeout(Duration timeout) { + PublishOptions defaults = PublishOptions.defaults(); + return new TransportPublishRequest( + destinationProfile(), + envelope(64), + new PublishOptions( + timeout, defaults.confirmation(), defaults.deduplication(), defaults.brokerHints())); + } + private static DestinationProfile destinationProfile() { return new DestinationProfile( new DestinationName("orders.v1"), diff --git a/messaging/messaging-observability/build.gradle b/messaging/messaging-observability/build.gradle index a0dac81..a7ec46f 100644 --- a/messaging/messaging-observability/build.gradle +++ b/messaging/messaging-observability/build.gradle @@ -3,5 +3,7 @@ apply plugin: 'java-library' dependencies { api project(':messaging:messaging-core-api') - implementation 'io.micrometer:micrometer-core' + // api: MessagingMetrics' public constructor takes a MeterRegistry, so wiring it requires + // naming the type. + api 'io.micrometer:micrometer-core' } diff --git a/messaging/messaging-observability/src/main/java/dev/caskeleton/messaging/observation/CardinalityGuard.java b/messaging/messaging-observability/src/main/java/dev/caskeleton/messaging/observation/CardinalityGuard.java index 992e8a6..91567e7 100644 --- a/messaging/messaging-observability/src/main/java/dev/caskeleton/messaging/observation/CardinalityGuard.java +++ b/messaging/messaging-observability/src/main/java/dev/caskeleton/messaging/observation/CardinalityGuard.java @@ -54,11 +54,19 @@ public final class CardinalityGuard { if (values.contains(value)) { return true; } - if (values.size() >= limitPerDimension) { - return false; + // Size-then-add was not atomic: N threads could each read size == limit - 1 and each add, so + // the configured limit was an average rather than a bound. A guard that can be exceeded under + // load is no guard — load is when it matters. + synchronized (values) { + if (values.contains(value)) { + return true; + } + if (values.size() >= limitPerDimension) { + return false; + } + values.add(value); + return true; } - values.add(value); - return true; } /** @@ -69,13 +77,30 @@ public final class CardinalityGuard { */ public boolean admit(MessagingTags tags) { Objects.requireNonNull(tags, "tags must not be null"); + // Preflight every dimension before committing any of them. + // + // The loop used to admit each dimension as it went, so a tag set rejected on its last + // dimension had already permanently added the earlier ones — spending the budget of a bounded + // dimension on a series that was never emitted. + Map pending = tags.asMap(); + for (Map.Entry entry : pending.entrySet()) { + if (!wouldAdmit(entry.getKey(), entry.getValue())) { + return false; + } + } boolean admitted = true; - for (Map.Entry entry : tags.asMap().entrySet()) { + for (Map.Entry entry : pending.entrySet()) { admitted &= admit(entry.getKey(), entry.getValue()); } return admitted; } + /** Whether a value is already known or would still fit, without recording it. */ + private boolean wouldAdmit(String dimension, String value) { + Set values = observed.get(dimension); + return values == null || values.contains(value) || values.size() < limitPerDimension; + } + /** * Returns how many distinct values a dimension has taken. * diff --git a/messaging/messaging-observability/src/main/java/dev/caskeleton/messaging/observation/MessagingMetrics.java b/messaging/messaging-observability/src/main/java/dev/caskeleton/messaging/observation/MessagingMetrics.java index 83ab621..9579061 100644 --- a/messaging/messaging-observability/src/main/java/dev/caskeleton/messaging/observation/MessagingMetrics.java +++ b/messaging/messaging-observability/src/main/java/dev/caskeleton/messaging/observation/MessagingMetrics.java @@ -119,17 +119,31 @@ public final class MessagingMetrics implements MessagingObservation { // Redact before anything else touches the values. Diagnostics are the one place where a caller // can pass arbitrary keys, so this is where a secret would otherwise reach the backend. Map safe = redactor.sanitize(diagnostics); + // The value never becomes a tag. + // + // It used to: every diagnostic key and value was attached to a counter, behind a guard that + // only bounded the base dimensions. One unique message id, exception message or URL per + // request created one meter series per request — permanently, in the backend and in this + // process's heap — and the redactor only masks keys it recognises, so free-form text carried + // whatever it carried. + // + // What stays is the shape: which diagnostic keys occurred, counted against the bounded base + // dimensions. The values belong in a structured log or a trace event, where they are bounded + // by retention rather than by cardinality. admitted(tags) .ifPresent( micrometerTags -> - safe.forEach( - (key, value) -> - registry - .counter( - "messaging.diagnostics", - micrometerTags.and( - Tag.of("diagnostic", key), Tag.of("value", value))) - .increment())); + safe.keySet() + .forEach( + key -> { + if (guard.admit("diagnostic", key)) { + registry + .counter( + "messaging.diagnostics", + micrometerTags.and(Tag.of("diagnostic", key))) + .increment(); + } + })); } /** diff --git a/messaging/messaging-observability/src/test/java/dev/caskeleton/messaging/observation/MessagingSecretLeakTest.java b/messaging/messaging-observability/src/test/java/dev/caskeleton/messaging/observation/MessagingSecretLeakTest.java index 6e7647b..467203f 100644 --- a/messaging/messaging-observability/src/test/java/dev/caskeleton/messaging/observation/MessagingSecretLeakTest.java +++ b/messaging/messaging-observability/src/test/java/dev/caskeleton/messaging/observation/MessagingSecretLeakTest.java @@ -84,7 +84,61 @@ class MessagingSecretLeakTest { .map(Tag::getValue) .toList(); - assertThat(emitted).doesNotContain("hunter2").contains("unroutable"); + // No diagnostic value reaches the registry at all — not the secret, and not the innocuous one + // either. A value tag is one meter series per distinct value, permanently, and the redactor + // only masks keys it recognises: free-form text carries whatever it carries. + assertThat(emitted).doesNotContain("hunter2").doesNotContain("unroutable"); + assertThat( + registry.getMeters().stream() + .map(Meter::getId) + .flatMap(id -> id.getTags().stream()) + .filter(tag -> tag.getKey().equals("diagnostic")) + .map(Tag::getValue) + .toList()) + .as("the shape is kept: which diagnostic keys occurred, against bounded base dimensions") + .contains("reason"); + } + + @Test + @org.junit.jupiter.api.DisplayName("ten thousand distinct diagnostic values create no new series") + void distinctDiagnosticValuesCreateNoNewSeries() { + MessagingMetrics metrics = new MessagingMetrics(registry, new CardinalityGuard(100), redactor); + MessagingTags tags = MessagingTags.of("kafka", "orders.v1", "publish", "rejected"); + + for (int index = 0; index < 10_000; index++) { + metrics.recordDiagnostics(tags, Map.of("reason", "unique-value-" + index)); + } + + assertThat( + registry.getMeters().stream() + .filter(meter -> meter.getId().getName().equals("messaging.diagnostics")) + .toList()) + .as("one unique message id per request used to create one meter series per request") + .hasSize(1); + } + + @Test + @org.junit.jupiter.api.DisplayName("concurrent admission never exceeds the configured limit") + void concurrentAdmissionNeverExceedsTheLimit() throws Exception { + CardinalityGuard guard = new CardinalityGuard(10); + java.util.concurrent.ExecutorService pool = + java.util.concurrent.Executors.newFixedThreadPool(8); + try { + java.util.List> work = + java.util.stream.IntStream.range(0, 200) + .>mapToObj( + index -> () -> guard.admit("destination", "d-" + index)) + .toList(); + for (java.util.concurrent.Future future : pool.invokeAll(work)) { + future.get(); + } + } finally { + pool.shutdownNow(); + } + + assertThat(guard.cardinality("destination")) + .as("size-then-add is not atomic, so the limit was an average rather than a bound") + .isEqualTo(10); } @Test diff --git a/messaging/messaging-observability/src/test/java/dev/caskeleton/messaging/observation/SecretLeakStaticScanTest.java b/messaging/messaging-observability/src/test/java/dev/caskeleton/messaging/observation/SecretLeakStaticScanTest.java index 812f0d6..e0d93d6 100644 --- a/messaging/messaging-observability/src/test/java/dev/caskeleton/messaging/observation/SecretLeakStaticScanTest.java +++ b/messaging/messaging-observability/src/test/java/dev/caskeleton/messaging/observation/SecretLeakStaticScanTest.java @@ -31,6 +31,9 @@ class SecretLeakStaticScanTest { Pattern.compile("(System\\.out|System\\.err|printStackTrace\\()"); /** A Java string literal, including escaped quotes. */ + /** Whether the scanner is currently inside a text block. */ + private static final ThreadLocal IN_TEXT_BLOCK = ThreadLocal.withInitial(() -> false); + private static final Pattern STRING_LITERAL = Pattern.compile("\"(\\\\.|[^\"\\\\])*\""); /** The operand immediately before or after a string concatenation. */ @@ -129,6 +132,13 @@ class SecretLeakStaticScanTest { * @return true when the line concatenates a sensitive value */ private static boolean leaksASensitiveValue(String line) { + if (line.contains("\"\"\"") || IN_TEXT_BLOCK.get()) { + // A text block is SQL or JSON, not Java code. The scanner replaced ordinary string literals + // and then read the contents of text blocks as if they were expressions, so a column named + // `lease_token` in an UPDATE statement looked like a concatenated credential. + IN_TEXT_BLOCK.set(IN_TEXT_BLOCK.get() != line.contains("\"\"\"")); + return false; + } String code = STRING_LITERAL.matcher(line).replaceAll("\"\""); if (!code.contains("+")) { return false; diff --git a/messaging/messaging-outbox-jpa/build.gradle b/messaging/messaging-outbox-jdbc-postgresql/build.gradle similarity index 72% rename from messaging/messaging-outbox-jpa/build.gradle rename to messaging/messaging-outbox-jdbc-postgresql/build.gradle index bffe63f..fcd2cec 100644 --- a/messaging/messaging-outbox-jpa/build.gradle +++ b/messaging/messaging-outbox-jdbc-postgresql/build.gradle @@ -6,6 +6,11 @@ dependencies { api project(':messaging:messaging-policy') api project(':messaging:messaging-observability') + // The destructive-operation journal lives here because it needs exactly what the outbox needs: + // one relational database every replica can see, and a migration lane that already exists. The + // contract it implements belongs to the admin API. + api project(':messaging:messaging-admin-api') + implementation 'org.springframework:spring-jdbc' implementation 'org.springframework:spring-tx' diff --git a/messaging/messaging-outbox-jpa/gradle.lockfile b/messaging/messaging-outbox-jdbc-postgresql/gradle.lockfile similarity index 95% rename from messaging/messaging-outbox-jpa/gradle.lockfile rename to messaging/messaging-outbox-jdbc-postgresql/gradle.lockfile index 4d7833b..438bb2c 100644 --- a/messaging/messaging-outbox-jpa/gradle.lockfile +++ b/messaging/messaging-outbox-jdbc-postgresql/gradle.lockfile @@ -39,9 +39,9 @@ commons-logging:commons-logging:1.3.5=compileClasspath,runtimeClasspath,testComp info.picocli:picocli:4.7.7=checkstyle io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnotationProcessor io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor -io.micrometer:micrometer-commons:1.16.0=runtimeClasspath,testRuntimeClasspath -io.micrometer:micrometer-core:1.16.0=runtimeClasspath,testRuntimeClasspath -io.micrometer:micrometer-observation:1.16.0=runtimeClasspath,testRuntimeClasspath +io.micrometer:micrometer-commons:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.micrometer:micrometer-core:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.micrometer:micrometer-observation:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath javax.inject:javax.inject:1=annotationProcessor,testAnnotationProcessor jaxen:jaxen:2.0.0=spotbugs net.bytebuddy:byte-buddy:1.17.8=testCompileClasspath,testRuntimeClasspath diff --git a/messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/DebeziumMappedRecord.java b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/DebeziumMappedRecord.java similarity index 100% rename from messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/DebeziumMappedRecord.java rename to messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/DebeziumMappedRecord.java diff --git a/messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/DebeziumOutboxEventRouter.java b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/DebeziumOutboxEventRouter.java similarity index 100% rename from messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/DebeziumOutboxEventRouter.java rename to messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/DebeziumOutboxEventRouter.java diff --git a/messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/DebeziumOutboxProfile.java b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/DebeziumOutboxProfile.java similarity index 100% rename from messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/DebeziumOutboxProfile.java rename to messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/DebeziumOutboxProfile.java diff --git a/messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/DebeziumOutboxRecordMapper.java b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/DebeziumOutboxRecordMapper.java similarity index 100% rename from messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/DebeziumOutboxRecordMapper.java rename to messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/DebeziumOutboxRecordMapper.java diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcAdminOperationJournal.java b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcAdminOperationJournal.java new file mode 100644 index 0000000..099dd96 --- /dev/null +++ b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcAdminOperationJournal.java @@ -0,0 +1,326 @@ +package dev.caskeleton.messaging.outbox; + +import dev.caskeleton.messaging.admin.AdminOperationJournal; +import dev.caskeleton.messaging.admin.AdminOperationLease; +import dev.caskeleton.messaging.admin.AdminOperationRecord; +import dev.caskeleton.messaging.admin.AdminOperationState; +import dev.caskeleton.messaging.admin.PlanDigest; +import dev.caskeleton.messaging.api.error.MessageAuthorizationException; +import dev.caskeleton.messaging.api.error.MessagingConfigurationException; +import java.sql.Connection; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.sql.Timestamp; +import java.time.Duration; +import java.time.Instant; +import java.util.Objects; +import java.util.Optional; +import javax.sql.DataSource; +import org.springframework.jdbc.datasource.DataSourceUtils; + +/** + * The shared, durable destructive-operation journal. + * + *

    Lives beside the outbox because it needs the same thing the outbox needs and nothing more: one + * relational database that every replica can see. The uniqueness that stops a second execution is + * the primary key on {@code (approval_ticket, plan_digest)}, enforced by the database rather than + * by a check-then-act in application code — two replicas that read "no row" at the same instant + * would both proceed, and only the constraint makes exactly one of them win. + * + *

    Takeover is conditional on the row still carrying the token the caller read. That is what + * fences a stalled runtime out: it wakes up holding token 3, the row is at token 4, and every write + * it attempts is refused rather than overwriting the replica that resumed the work. + */ +public final class JdbcAdminOperationJournal implements AdminOperationJournal { + + private static final String SELECT = + """ + SELECT approval_ticket, plan_digest, operation_id, state, items_completed, + lease_owner, lease_token, lease_expires_at, started_at, finished_at, failure_code + FROM messaging_admin_operation + WHERE approval_ticket = ? AND plan_digest = ? + """; + + private static final String INSERT = + """ + INSERT INTO messaging_admin_operation ( + approval_ticket, plan_digest, operation_id, state, items_completed, + lease_owner, lease_token, lease_expires_at, started_at) + VALUES (?, ?, ?, 'STARTED', 0, ?, 1, ?, ?) + ON CONFLICT (approval_ticket, plan_digest) DO NOTHING + """; + + private static final String TAKE_OVER = + """ + UPDATE messaging_admin_operation + SET operation_id = ?, + state = 'STARTED', + lease_owner = ?, + lease_token = lease_token + 1, + lease_expires_at = ?, + finished_at = NULL, + failure_code = NULL + WHERE approval_ticket = ? + AND plan_digest = ? + AND lease_token = ? + -- Only a failed operation or one whose lease ran out may be taken over. A live STARTED row + -- means another replica is executing it right now. + AND (state = 'FAILED' OR lease_expires_at <= ?) + RETURNING lease_token, items_completed + """; + + private static final String CHECKPOINT = + """ + UPDATE messaging_admin_operation + SET items_completed = GREATEST(items_completed, ?), + lease_expires_at = ? + WHERE approval_ticket = ? AND plan_digest = ? AND lease_token = ? + """; + + private static final String SETTLE = + """ + UPDATE messaging_admin_operation + SET state = ?, + items_completed = GREATEST(items_completed, ?), + finished_at = ?, + failure_code = ? + WHERE approval_ticket = ? AND plan_digest = ? AND lease_token = ? + """; + + private final DataSource dataSource; + + /** + * Creates the journal. + * + * @param dataSource the shared database every replica reads + */ + public JdbcAdminOperationJournal(DataSource dataSource) { + this.dataSource = Objects.requireNonNull(dataSource, "dataSource must not be null"); + } + + @Override + public AdminOperationLease begin( + String approvalTicket, + PlanDigest planDigest, + String operationId, + String leaseOwner, + Duration leaseDuration, + Instant now) { + Objects.requireNonNull(planDigest, "planDigest must not be null"); + Objects.requireNonNull(leaseDuration, "leaseDuration must not be null"); + Objects.requireNonNull(now, "now must not be null"); + Instant expiry = now.plus(leaseDuration); + + Connection connection = DataSourceUtils.getConnection(dataSource); + try { + if (insertClaim( + connection, approvalTicket, planDigest, operationId, leaseOwner, expiry, now)) { + return new AdminOperationLease(approvalTicket, planDigest, operationId, leaseOwner, 1L, 0); + } + AdminOperationRecord existing = + read(connection, approvalTicket, planDigest) + .orElseThrow( + () -> + new MessagingConfigurationException( + "ADMIN_JOURNAL_INCONSISTENT", + "the insert conflicted but no row exists for approval " + + approvalTicket)); + refuseIfNotResumable(existing, approvalTicket, now); + return takeOver( + connection, existing, approvalTicket, planDigest, operationId, leaseOwner, expiry, now); + } catch (SQLException failure) { + throw new MessagingConfigurationException( + "ADMIN_JOURNAL_UNAVAILABLE", "the operation journal could not be reached", failure); + } finally { + DataSourceUtils.releaseConnection(connection, dataSource); + } + } + + private static void refuseIfNotResumable( + AdminOperationRecord existing, String approvalTicket, Instant now) { + if (existing.state() == AdminOperationState.COMPLETED) { + throw new MessageAuthorizationException( + "APPROVAL_ALREADY_EXECUTED", + ("approval %s was already executed to completion at %s; an approval authorises one" + + " execution, not a standing permission") + .formatted(approvalTicket, existing.finishedAt().orElse(existing.startedAt()))); + } + if (existing.state() == AdminOperationState.STARTED + && now.isBefore(existing.leaseExpiresAt())) { + throw new MessageAuthorizationException( + "ADMIN_OPERATION_IN_FLIGHT", + ("approval %s is being executed by %s until at least %s; two runtimes executing one" + + " approval is a duplicate storm, not a faster redrive") + .formatted(approvalTicket, existing.leaseOwner(), existing.leaseExpiresAt())); + } + } + + private boolean insertClaim( + Connection connection, + String approvalTicket, + PlanDigest planDigest, + String operationId, + String leaseOwner, + Instant expiry, + Instant now) + throws SQLException { + try (PreparedStatement statement = connection.prepareStatement(INSERT)) { + statement.setString(1, approvalTicket); + statement.setString(2, planDigest.value()); + statement.setString(3, operationId); + statement.setString(4, leaseOwner); + statement.setTimestamp(5, Timestamp.from(expiry)); + statement.setTimestamp(6, Timestamp.from(now)); + return statement.executeUpdate() == 1; + } + } + + private AdminOperationLease takeOver( + Connection connection, + AdminOperationRecord existing, + String approvalTicket, + PlanDigest planDigest, + String operationId, + String leaseOwner, + Instant expiry, + Instant now) + throws SQLException { + try (PreparedStatement statement = connection.prepareStatement(TAKE_OVER)) { + statement.setString(1, operationId); + statement.setString(2, leaseOwner); + statement.setTimestamp(3, Timestamp.from(expiry)); + statement.setString(4, approvalTicket); + statement.setString(5, planDigest.value()); + statement.setLong(6, existing.leaseToken()); + statement.setTimestamp(7, Timestamp.from(now)); + try (ResultSet rows = statement.executeQuery()) { + if (!rows.next()) { + // Another replica took it over between the read and this update. Refusing is correct: + // exactly one runtime executes an approval, and it is not this one. + throw new MessageAuthorizationException( + "ADMIN_OPERATION_IN_FLIGHT", + "approval " + approvalTicket + " was taken over by another runtime"); + } + return new AdminOperationLease( + approvalTicket, + planDigest, + operationId, + leaseOwner, + rows.getLong("lease_token"), + rows.getInt("items_completed")); + } + } + } + + @Override + public void checkpoint( + AdminOperationLease lease, int itemsCompleted, Duration leaseDuration, Instant now) { + Objects.requireNonNull(lease, "lease must not be null"); + Connection connection = DataSourceUtils.getConnection(dataSource); + try (PreparedStatement statement = connection.prepareStatement(CHECKPOINT)) { + statement.setInt(1, itemsCompleted); + statement.setTimestamp(2, Timestamp.from(now.plus(leaseDuration))); + statement.setString(3, lease.approvalTicket()); + statement.setString(4, lease.planDigest().value()); + statement.setLong(5, lease.leaseToken()); + requireOwned(statement.executeUpdate(), lease); + } catch (SQLException failure) { + throw new MessagingConfigurationException( + "ADMIN_JOURNAL_UNAVAILABLE", "the operation journal could not be reached", failure); + } finally { + DataSourceUtils.releaseConnection(connection, dataSource); + } + } + + @Override + public void complete(AdminOperationLease lease, int itemsCompleted, Instant now) { + settle(lease, AdminOperationState.COMPLETED, itemsCompleted, null, now); + } + + @Override + public void fail(AdminOperationLease lease, int itemsCompleted, String failureCode, Instant now) { + Objects.requireNonNull(failureCode, "failureCode must not be null"); + settle(lease, AdminOperationState.FAILED, itemsCompleted, failureCode, now); + } + + private void settle( + AdminOperationLease lease, + AdminOperationState state, + int itemsCompleted, + String failureCode, + Instant now) { + Objects.requireNonNull(lease, "lease must not be null"); + Connection connection = DataSourceUtils.getConnection(dataSource); + try (PreparedStatement statement = connection.prepareStatement(SETTLE)) { + statement.setString(1, state.name()); + statement.setInt(2, itemsCompleted); + statement.setTimestamp(3, Timestamp.from(now)); + statement.setString(4, failureCode); + statement.setString(5, lease.approvalTicket()); + statement.setString(6, lease.planDigest().value()); + statement.setLong(7, lease.leaseToken()); + requireOwned(statement.executeUpdate(), lease); + } catch (SQLException failure) { + throw new MessagingConfigurationException( + "ADMIN_JOURNAL_UNAVAILABLE", "the operation journal could not be reached", failure); + } finally { + DataSourceUtils.releaseConnection(connection, dataSource); + } + } + + private static void requireOwned(int updated, AdminOperationLease lease) { + if (updated == 1) { + return; + } + throw new MessageAuthorizationException( + "ADMIN_OPERATION_LEASE_LOST", + ("this runtime no longer holds the lease on approval %s at token %d; another runtime took" + + " the operation over and its progress must not be overwritten") + .formatted(lease.approvalTicket(), lease.leaseToken())); + } + + @Override + public Optional find(String approvalTicket, PlanDigest planDigest) { + Connection connection = DataSourceUtils.getConnection(dataSource); + try { + return read(connection, approvalTicket, planDigest); + } catch (SQLException failure) { + throw new MessagingConfigurationException( + "ADMIN_JOURNAL_UNAVAILABLE", "the operation journal could not be reached", failure); + } finally { + DataSourceUtils.releaseConnection(connection, dataSource); + } + } + + private Optional read( + Connection connection, String approvalTicket, PlanDigest planDigest) throws SQLException { + try (PreparedStatement statement = connection.prepareStatement(SELECT)) { + statement.setString(1, approvalTicket); + statement.setString(2, planDigest.value()); + try (ResultSet rows = statement.executeQuery()) { + if (!rows.next()) { + return Optional.empty(); + } + return Optional.of( + new AdminOperationRecord( + rows.getString("approval_ticket"), + new PlanDigest(rows.getString("plan_digest")), + rows.getString("operation_id"), + AdminOperationState.valueOf(rows.getString("state")), + rows.getInt("items_completed"), + rows.getString("lease_owner"), + rows.getLong("lease_token"), + rows.getTimestamp("lease_expires_at").toInstant(), + rows.getTimestamp("started_at").toInstant(), + Optional.ofNullable(rows.getTimestamp("finished_at")).map(Timestamp::toInstant), + Optional.ofNullable(rows.getString("failure_code")))); + } + } + } + + @Override + public boolean isDurable() { + return true; + } +} diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcOutboxRepository.java b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcOutboxRepository.java new file mode 100644 index 0000000..f4351c7 --- /dev/null +++ b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcOutboxRepository.java @@ -0,0 +1,648 @@ +package dev.caskeleton.messaging.outbox; + +import dev.caskeleton.messaging.api.ContentType; +import dev.caskeleton.messaging.api.MessageId; +import dev.caskeleton.messaging.api.MessageType; +import dev.caskeleton.messaging.api.SchemaVersion; +import dev.caskeleton.messaging.api.destination.DestinationName; +import dev.caskeleton.messaging.api.error.MessagingConfigurationException; +import dev.caskeleton.messaging.reliability.OutboxLease; +import dev.caskeleton.messaging.reliability.OutboxRecord; +import dev.caskeleton.messaging.reliability.OutboxRepository; +import dev.caskeleton.messaging.reliability.OutboxStatus; +import dev.caskeleton.messaging.reliability.OutboxTransitionResult; +import java.sql.Connection; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.sql.Timestamp; +import java.time.Duration; +import java.time.Instant; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; +import javax.sql.DataSource; +import org.springframework.jdbc.datasource.DataSourceUtils; +import org.springframework.transaction.support.TransactionSynchronizationManager; + +/** + * The PostgreSQL outbox. + * + *

    {@link #append} deliberately takes no connection of its own: it uses the one the caller is + * already inside, which is the entire mechanism. An outbox row written on a separate connection + * commits independently of the business change and reopens the window the pattern exists to close. + * + *

    Leasing uses {@code SELECT ... FOR UPDATE SKIP LOCKED}. Two relay instances polling the same + * table would otherwise both claim the same rows and publish them twice; {@code SKIP LOCKED} lets + * the second instance step over what the first already holds instead of blocking behind it. + */ +public final class JdbcOutboxRepository implements OutboxRepository { + + private static final String INSERT = + """ + INSERT INTO messaging_outbox ( + message_id, destination, message_type, schema_version, content_type, + payload, headers, created_at, status, attempts) + VALUES (?, ?, ?, ?, ?, ?, ?::jsonb, ?, ?, ?) + """; + + private static final String LEASE = + """ + WITH claimable AS ( + SELECT message_id + FROM messaging_outbox + -- IN_FLIGHT is claimable too, but only once its lease has expired. A relay that dies + -- mid-publish leaves rows in that state; excluding them would strand those messages + -- forever, which is the one failure mode the outbox exists to prevent. + WHERE status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT') + AND (lease_expires_at IS NULL OR lease_expires_at <= ?) + ORDER BY created_at + LIMIT ? + FOR UPDATE SKIP LOCKED + ) + UPDATE messaging_outbox o + SET status = 'IN_FLIGHT', lease_expires_at = ? + FROM claimable c + WHERE o.message_id = c.message_id + RETURNING o.message_id, o.destination, o.message_type, o.schema_version, o.content_type, + o.payload, o.headers, o.created_at, o.status, o.attempts, + o.lease_expires_at, o.last_failure_code + """; + + /** + * The claim that records who holds the row and increments its fencing token. + * + *

    The token increments on the server inside the same statement that grants the claim, so two + * relays racing for the same row cannot receive the same number. + */ + private static final String CLAIM = + """ + WITH claimable AS ( + SELECT message_id + FROM messaging_outbox + WHERE status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT') + AND (lease_expires_at IS NULL OR lease_expires_at <= ?) + -- The retry clock lives in the row, not in the relay's memory. Without these two + -- predicates an AMBIGUOUS row became claimable again on the very next pass, so a + -- broker outage meant the whole backlog was republished every poll interval and the + -- configured attempt budget was a number nothing consulted. + AND (next_attempt_at IS NULL OR next_attempt_at <= ?) + AND attempts < ? + ORDER BY created_at + LIMIT ? + FOR UPDATE SKIP LOCKED + ) + UPDATE messaging_outbox o + SET status = 'IN_FLIGHT', + lease_expires_at = ?, + lease_owner = ?, + lease_token = o.lease_token + 1 + FROM claimable c + WHERE o.message_id = c.message_id + RETURNING o.message_id, o.destination, o.message_type, o.schema_version, o.content_type, + o.payload, o.headers, o.created_at, o.status, o.attempts, + o.lease_expires_at, o.last_failure_code, o.lease_owner, o.lease_token + """; + + private final DataSource dataSource; + + /** + * Creates a repository over a data source. + * + * @param dataSource the outbox data source + */ + public JdbcOutboxRepository(DataSource dataSource) { + this.dataSource = Objects.requireNonNull(dataSource, "dataSource must not be null"); + } + + /** + * Appends a record on an existing connection. + * + *

    Package-private. It used to be the public overload "business code should use", which meant + * the safe path was the one a caller had to know about and the interface method — the one the + * port documents — opened its own connection. Handing a {@code Connection} across a reliability + * API also makes the transaction the caller's problem to remember; {@link #append(OutboxRecord)} + * now joins the caller's transaction itself and refuses when there is none. + * + * @param connection the caller's transactional connection + * @param record the record to write + */ + void append(Connection connection, OutboxRecord record) { + Objects.requireNonNull(connection, "connection must not be null"); + Objects.requireNonNull(record, "record must not be null"); + try (PreparedStatement statement = connection.prepareStatement(INSERT)) { + statement.setObject(1, record.messageId().value()); + statement.setString(2, record.destination().value()); + statement.setString(3, record.messageType().value()); + statement.setInt(4, record.schemaVersion().value()); + statement.setString(5, record.contentType().value()); + statement.setBytes(6, record.payload()); + statement.setString(7, toJson(record.headers())); + statement.setTimestamp(8, Timestamp.from(record.createdAt())); + statement.setString(9, record.status().name()); + statement.setInt(10, record.attempts()); + statement.executeUpdate(); + } catch (SQLException exception) { + throw new MessagingConfigurationException( + "OUTBOX_APPEND_FAILED", "could not append an outbox record", exception); + } + } + + @Override + public void append(OutboxRecord record) { + Objects.requireNonNull(record, "record must not be null"); + // The port says this runs inside the caller's business transaction. It did not: it opened a raw + // connection from the pool, which auto-commits independently — so a business transaction that + // rolled back afterwards left the event behind and the relay published a change that never + // happened. + requireActiveTransaction("OUTBOX_TRANSACTION_REQUIRED", "appending to the outbox"); + Connection connection = DataSourceUtils.getConnection(dataSource); + try { + append(connection, record); + } finally { + // Releases only when the connection is not the transaction's; when it is, this is a no-op and + // the caller's commit or rollback decides what happens to the row. + DataSourceUtils.releaseConnection(connection, dataSource); + } + } + + /** + * Refuses when there is no transaction to join, or when it cannot write. + * + *

    Fail-fast rather than "work anyway": an append that silently runs outside the caller's + * transaction produces exactly the ghost publication this repository exists to prevent, and it + * produces it only on the rollback path — which is the path nobody exercises before production. + */ + private void requireActiveTransaction(String code, String what) { + if (!TransactionSynchronizationManager.isActualTransactionActive()) { + throw new MessagingConfigurationException( + code, + what + + " requires an active transaction; without one the row commits on its own and a" + + " later rollback of the business change leaves it behind"); + } + if (TransactionSynchronizationManager.isCurrentTransactionReadOnly()) { + throw new MessagingConfigurationException( + code, what + " requires a writable transaction; the current one is read-only"); + } + if (!TransactionSynchronizationManager.hasResource(dataSource)) { + throw new MessagingConfigurationException( + code, + what + + " requires the active transaction to be bound to this DataSource; it is bound to" + + " another, so the two would commit independently"); + } + } + + @Override + public List leaseBatch(int batchSize, Duration leaseDuration, Instant now) { + Objects.requireNonNull(leaseDuration, "leaseDuration must not be null"); + Objects.requireNonNull(now, "now must not be null"); + return withConnection( + connection -> { + List leased = new ArrayList<>(); + try (PreparedStatement statement = connection.prepareStatement(LEASE)) { + statement.setTimestamp(1, Timestamp.from(now)); + statement.setInt(2, batchSize); + statement.setTimestamp(3, Timestamp.from(now.plus(leaseDuration))); + try (ResultSet results = statement.executeQuery()) { + while (results.next()) { + leased.add(read(results)); + } + } + } + return List.copyOf(leased); + }); + } + + @Override + public List claimBatch( + String owner, int batchSize, Duration leaseDuration, Instant now) { + return claimBatch(owner, batchSize, leaseDuration, now, Integer.MAX_VALUE); + } + + @Override + public List claimBatch( + String owner, int batchSize, Duration leaseDuration, Instant now, int maxAttempts) { + Objects.requireNonNull(owner, "owner must not be null"); + Objects.requireNonNull(leaseDuration, "leaseDuration must not be null"); + Objects.requireNonNull(now, "now must not be null"); + Instant expiresAt = now.plus(leaseDuration); + return withConnection( + connection -> { + List claimed = new ArrayList<>(); + try (PreparedStatement statement = connection.prepareStatement(CLAIM)) { + statement.setTimestamp(1, Timestamp.from(now)); + statement.setTimestamp(2, Timestamp.from(now)); + statement.setInt(3, maxAttempts); + statement.setInt(4, batchSize); + statement.setTimestamp(5, Timestamp.from(expiresAt)); + statement.setString(6, owner); + try (ResultSet results = statement.executeQuery()) { + while (results.next()) { + claimed.add( + new OutboxLease( + read(results), + results.getString("lease_owner"), + results.getLong("lease_token"), + expiresAt)); + } + } + } + return List.copyOf(claimed); + }); + } + + @Override + public OutboxTransitionResult markPublished(OutboxLease lease, Instant now) { + return transition( + lease, + "UPDATE messaging_outbox SET status = 'PUBLISHED', published_at = ?, " + + "lease_expires_at = NULL, lease_owner = NULL, next_attempt_at = NULL, " + + "attempts = attempts + 1 ", + now, + null); + } + + @Override + public OutboxTransitionResult markAmbiguous(OutboxLease lease, String failureCode, Instant now) { + return markAmbiguous(lease, failureCode, now, now); + } + + @Override + public OutboxTransitionResult markAmbiguous( + OutboxLease lease, String failureCode, Instant now, Instant nextAttemptAt) { + Objects.requireNonNull(nextAttemptAt, "nextAttemptAt must not be null"); + // The backoff the policy computed is written onto the row. Keeping it in the relay's memory + // means a restart forgets it, and every relay computes its own — so the same row is retried by + // as many workers as there are replicas, as fast as they poll. + return transitionWithNextAttempt( + lease, + "UPDATE messaging_outbox SET status = 'AMBIGUOUS', lease_expires_at = NULL, " + + "lease_owner = NULL, last_failure_code = ?, attempts = attempts + 1, " + + "next_attempt_at = ? ", + failureCode, + nextAttemptAt); + } + + @Override + public OutboxTransitionResult markExhausted(OutboxLease lease, String reason, Instant now) { + return transition( + lease, + "UPDATE messaging_outbox SET status = 'EXHAUSTED', lease_expires_at = NULL, " + + "lease_owner = NULL, last_failure_code = ?, attempts = attempts + 1 ", + now, + reason); + } + + private OutboxTransitionResult transitionWithNextAttempt( + OutboxLease lease, String setClause, String failureCode, Instant nextAttemptAt) { + Objects.requireNonNull(lease, "lease must not be null"); + String sql = + setClause + + "WHERE message_id = ? AND status = 'IN_FLIGHT' AND lease_owner = ? AND lease_token = ?"; + return withConnection( + connection -> { + try (PreparedStatement statement = connection.prepareStatement(sql)) { + statement.setString(1, failureCode); + statement.setTimestamp(2, Timestamp.from(nextAttemptAt)); + statement.setObject(3, lease.messageId().value()); + statement.setString(4, lease.owner()); + statement.setLong(5, lease.token()); + return statement.executeUpdate() == 1 + ? OutboxTransitionResult.APPLIED + : OutboxTransitionResult.STALE_LEASE; + } + }); + } + + @Override + public OutboxTransitionResult markFailed(OutboxLease lease, String failureCode, Instant now) { + return transition( + lease, + "UPDATE messaging_outbox SET status = 'FAILED', lease_expires_at = NULL, " + + "lease_owner = NULL, last_failure_code = ?, attempts = attempts + 1 ", + now, + failureCode); + } + + @Override + public OutboxTransitionResult releaseLease(OutboxLease lease) { + return transition( + lease, + "UPDATE messaging_outbox SET status = 'PENDING', lease_expires_at = NULL, " + + "lease_owner = NULL ", + null, + null); + } + + /** + * Applies a terminal write, but only to the row this lease still owns. + * + *

    The predicate carries the owner and the token as well as the id, so a relay that stalled + * past its lease writes nothing: another relay's claim incremented the token, and this update + * matches zero rows. Zero is reported rather than swallowed — a stale write means this worker may + * have produced a duplicate publication, which is exactly what an operator needs to see. + */ + private OutboxTransitionResult transition( + OutboxLease lease, String setClause, Instant now, String failureCode) { + Objects.requireNonNull(lease, "lease must not be null"); + String sql = + setClause + + "WHERE message_id = ? AND status = 'IN_FLIGHT' AND lease_owner = ? AND lease_token = ?"; + return withConnection( + connection -> { + try (PreparedStatement statement = connection.prepareStatement(sql)) { + int index = 1; + if (failureCode != null) { + statement.setString(index++, failureCode); + } else if (now != null) { + statement.setTimestamp(index++, Timestamp.from(now)); + } + statement.setObject(index++, lease.messageId().value()); + statement.setString(index++, lease.owner()); + statement.setLong(index, lease.token()); + return statement.executeUpdate() == 1 + ? OutboxTransitionResult.APPLIED + : OutboxTransitionResult.STALE_LEASE; + } + }); + } + + @Override + public void markPublished(MessageId messageId, Instant now) { + update( + "UPDATE messaging_outbox SET status = 'PUBLISHED', published_at = ?, " + + "lease_expires_at = NULL, attempts = attempts + 1 WHERE message_id = ?", + messageId, + now, + null); + } + + @Override + public void markAmbiguous(MessageId messageId, String failureCode, Instant now) { + // The lease is released rather than held: an ambiguous row must become claimable again so the + // relay retries it under the same message id. + update( + "UPDATE messaging_outbox SET status = 'AMBIGUOUS', lease_expires_at = NULL, " + + "last_failure_code = ?, attempts = attempts + 1 WHERE message_id = ?", + messageId, + now, + failureCode); + } + + @Override + public void markFailed(MessageId messageId, String failureCode, Instant now) { + update( + "UPDATE messaging_outbox SET status = 'FAILED', lease_expires_at = NULL, " + + "last_failure_code = ?, attempts = attempts + 1 WHERE message_id = ?", + messageId, + now, + failureCode); + } + + @Override + public void releaseLease(MessageId messageId) { + withConnection( + connection -> { + try (PreparedStatement statement = + connection.prepareStatement( + "UPDATE messaging_outbox SET status = 'PENDING', lease_expires_at = NULL " + + "WHERE message_id = ? AND status = 'IN_FLIGHT'")) { + statement.setObject(1, messageId.value()); + statement.executeUpdate(); + } + return null; + }); + } + + @Override + public Optional find(MessageId messageId) { + return withConnection( + connection -> { + try (PreparedStatement statement = + connection.prepareStatement( + "SELECT message_id, destination, message_type, schema_version, content_type, " + + "payload, headers, created_at, status, attempts, lease_expires_at, " + + "last_failure_code FROM messaging_outbox WHERE message_id = ?")) { + statement.setObject(1, messageId.value()); + try (ResultSet results = statement.executeQuery()) { + return results.next() ? Optional.of(read(results)) : Optional.empty(); + } + } + }); + } + + @Override + public int purgePublishedBefore(Instant publishedBefore, int limit) { + Objects.requireNonNull(publishedBefore, "publishedBefore must not be null"); + if (limit < 1) { + throw new IllegalArgumentException("a bounded purge deletes at least one row per call"); + } + // The CTE picks a bounded set of ids with SKIP LOCKED and deletes exactly those. An unbounded + // DELETE holds locks and writes WAL in proportion to the whole backlog, which stalls the relay + // and the business writes behind retention. + return withConnection( + connection -> { + try (PreparedStatement statement = + connection.prepareStatement( + """ + WITH expired AS ( + SELECT message_id + FROM messaging_outbox + WHERE status = 'PUBLISHED' AND published_at < ? + ORDER BY published_at + LIMIT ? + FOR UPDATE SKIP LOCKED + ) + DELETE FROM messaging_outbox o + USING expired e + WHERE o.message_id = e.message_id + """)) { + statement.setTimestamp(1, Timestamp.from(publishedBefore)); + statement.setInt(2, limit); + return statement.executeUpdate(); + } + }); + } + + @Override + public int purgePublishedBefore(Instant publishedBefore) { + Objects.requireNonNull(publishedBefore, "publishedBefore must not be null"); + return withConnection( + connection -> { + try (PreparedStatement statement = + connection.prepareStatement( + "DELETE FROM messaging_outbox WHERE status = 'PUBLISHED' AND published_at < ?")) { + statement.setTimestamp(1, Timestamp.from(publishedBefore)); + return statement.executeUpdate(); + } + }); + } + + private void update(String sql, MessageId messageId, Instant now, String failureCode) { + withConnection( + connection -> { + try (PreparedStatement statement = connection.prepareStatement(sql)) { + if (failureCode == null) { + statement.setTimestamp(1, Timestamp.from(now)); + } else { + statement.setString(1, failureCode); + } + statement.setObject(2, messageId.value()); + statement.executeUpdate(); + } + return null; + }); + } + + private static OutboxRecord read(ResultSet results) throws SQLException { + Timestamp lease = results.getTimestamp("lease_expires_at"); + return new OutboxRecord( + new MessageId(results.getObject("message_id", java.util.UUID.class)), + new DestinationName(results.getString("destination")), + new MessageType(results.getString("message_type")), + new SchemaVersion(results.getInt("schema_version")), + new ContentType(results.getString("content_type")), + results.getBytes("payload"), + fromJson(results.getString("headers")), + results.getTimestamp("created_at").toInstant(), + OutboxStatus.valueOf(results.getString("status")), + results.getInt("attempts"), + Optional.ofNullable(lease).map(Timestamp::toInstant), + Optional.ofNullable(results.getString("last_failure_code"))); + } + + private T withConnection(ConnectionCallback callback) { + try (Connection connection = dataSource.getConnection()) { + return callback.apply(connection); + } catch (SQLException exception) { + throw new MessagingConfigurationException( + "OUTBOX_QUERY_FAILED", "an outbox query failed", exception); + } + } + + /** + * Serialises headers as a flat JSON object. + * + *

    Hand-rolled rather than pulled from a JSON library so this module keeps no codec dependency: + * outbox headers are always flat string pairs, validated by {@code MessageHeaders} before they + * ever reach here. + */ + private static String toJson(Map headers) { + StringBuilder json = new StringBuilder("{"); + boolean first = true; + for (Map.Entry entry : headers.entrySet()) { + if (!first) { + json.append(','); + } + first = false; + json.append('"') + .append(escape(entry.getKey())) + .append("\":\"") + .append(escape(entry.getValue())) + .append('"'); + } + return json.append('}').toString(); + } + + private static Map fromJson(String json) { + Map headers = new LinkedHashMap<>(); + if (json == null || json.length() <= 2) { + return headers; + } + String body = json.substring(1, json.length() - 1); + int index = 0; + while (index < body.length()) { + int keyStart = body.indexOf('"', index); + if (keyStart < 0) { + break; + } + int keyEnd = findClosingQuote(body, keyStart + 1); + int valueStart = body.indexOf('"', body.indexOf(':', keyEnd)); + int valueEnd = findClosingQuote(body, valueStart + 1); + headers.put( + unescape(body.substring(keyStart + 1, keyEnd)), + unescape(body.substring(valueStart + 1, valueEnd))); + index = valueEnd + 1; + } + return headers; + } + + private static int findClosingQuote(String text, int from) { + for (int index = from; index < text.length(); index++) { + if (text.charAt(index) == '"' && text.charAt(index - 1) != '\\') { + return index; + } + } + return text.length(); + } + + /** + * JSON string escaping, including the control characters. + * + *

    Only backslash and quote were escaped. A header value containing a newline or a NUL — which + * the value objects now refuse, but which older rows and other writers may still carry — produced + * a document PostgreSQL rejects as invalid JSONB, so the append failed at the database with an + * error naming neither the header nor the row. Control characters are the part of the JSON + * grammar that is easy to forget and impossible to get away with. + */ + private static String escape(String value) { + StringBuilder escaped = new StringBuilder(value.length() + 8); + for (int index = 0; index < value.length(); index++) { + char character = value.charAt(index); + switch (character) { + case '\\' -> escaped.append("\\\\"); + case '"' -> escaped.append("\\\""); + case '\n' -> escaped.append("\\n"); + case '\r' -> escaped.append("\\r"); + case '\t' -> escaped.append("\\t"); + case '\b' -> escaped.append("\\b"); + case '\f' -> escaped.append("\\f"); + default -> { + if (character < 0x20) { + escaped.append("\\u%04x".formatted((int) character)); + } else { + escaped.append(character); + } + } + } + } + return escaped.toString(); + } + + private static String unescape(String value) { + StringBuilder plain = new StringBuilder(value.length()); + for (int index = 0; index < value.length(); index++) { + char character = value.charAt(index); + if (character != '\\' || index == value.length() - 1) { + plain.append(character); + continue; + } + char next = value.charAt(++index); + switch (next) { + case 'n' -> plain.append('\n'); + case 'r' -> plain.append('\r'); + case 't' -> plain.append('\t'); + case 'b' -> plain.append('\b'); + case 'f' -> plain.append('\f'); + case 'u' -> { + plain.append((char) Integer.parseInt(value.substring(index + 1, index + 5), 16)); + index += 4; + } + default -> plain.append(next); + } + } + return plain.toString(); + } + + /** A unit of work against a borrowed connection. */ + @FunctionalInterface + private interface ConnectionCallback { + + T apply(Connection connection) throws SQLException; + } +} diff --git a/messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/OutboxCleanupJob.java b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxCleanupJob.java similarity index 100% rename from messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/OutboxCleanupJob.java rename to messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxCleanupJob.java diff --git a/messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/OutboxEnvelopeFactory.java b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxEnvelopeFactory.java similarity index 61% rename from messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/OutboxEnvelopeFactory.java rename to messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxEnvelopeFactory.java index 1eebb54..b3b219a 100644 --- a/messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/OutboxEnvelopeFactory.java +++ b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxEnvelopeFactory.java @@ -46,7 +46,30 @@ public final class OutboxEnvelopeFactory { Map headers = new LinkedHashMap<>(); record .headers() - .forEach((name, value) -> headers.put(new HeaderName(name), new HeaderValue(value))); + .forEach( + (name, value) -> { + // The row's headers are application data. Promoting them straight into + // MessageHeaders.platform let a row carrying `msg.id` or `msg.type` reach the wire + // as if the platform had written it — and the consumer trusts the last such header, + // so a forged identity lands in another message's inbox deduplication. + // + // Partition and ordering keys are the exception: they are reserved names the outbox + // legitimately carries as routing metadata, and they are read below rather than + // written as headers. + if (ReservedHeaders.isReserved(name) + && !ReservedHeaders.PARTITION_KEY.equals(name) + && !ReservedHeaders.ORDERING_KEY.equals(name)) { + throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException( + "RESERVED_HEADER_IN_OUTBOX_ROW", + "outbox row header '" + + name + + "' uses the reserved namespace; canonical identity and metadata come from" + + " the record's own columns, not from its header map"); + } + if (!ReservedHeaders.isReserved(name)) { + headers.put(new HeaderName(name), new HeaderValue(value)); + } + }); return new MessageEnvelope<>( record.messageId(), diff --git a/messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/OutboxProperties.java b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxProperties.java similarity index 100% rename from messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/OutboxProperties.java rename to messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxProperties.java diff --git a/messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelay.java b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelay.java similarity index 54% rename from messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelay.java rename to messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelay.java index 5668df6..69246ac 100644 --- a/messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelay.java +++ b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelay.java @@ -4,8 +4,10 @@ import dev.caskeleton.messaging.api.destination.MessageDestination; import dev.caskeleton.messaging.api.publish.MessagePublisher; import dev.caskeleton.messaging.api.publish.PublishOptions; import dev.caskeleton.messaging.api.publish.PublishResult; +import dev.caskeleton.messaging.reliability.OutboxLease; import dev.caskeleton.messaging.reliability.OutboxRecord; import dev.caskeleton.messaging.reliability.OutboxRepository; +import dev.caskeleton.messaging.reliability.OutboxTransitionResult; import dev.caskeleton.messaging.schema.EncodedMessage; import java.time.Duration; import java.time.Instant; @@ -30,6 +32,17 @@ public final class OutboxRelay { private final MessagePublisher publisher; private final OutboxEnvelopeFactory envelopeFactory; private final int batchSize; + + private final String owner; + + private final OutboxRetryScheduler scheduler; + + private static String defaultOwner() { + return java.lang.ProcessHandle.current().pid() + + "@" + + java.util.UUID.randomUUID().toString().substring(0, 8); + } + private final Duration leaseDuration; /** @@ -47,6 +60,50 @@ public final class OutboxRelay { OutboxEnvelopeFactory envelopeFactory, int batchSize, Duration leaseDuration) { + this(repository, publisher, envelopeFactory, batchSize, leaseDuration, defaultOwner()); + } + + /** + * A relay with an explicit owner identity. + * + *

    The owner is half of the fencing predicate. It defaults to something stable per process + * rather than per object, because two relays in one JVM sharing an owner would each be able to + * overwrite the other's terminal writes — the exact failure the token exists to prevent. + */ + public OutboxRelay( + OutboxRepository repository, + MessagePublisher publisher, + OutboxEnvelopeFactory envelopeFactory, + int batchSize, + Duration leaseDuration, + String owner) { + this( + repository, + publisher, + envelopeFactory, + batchSize, + leaseDuration, + owner, + OutboxRetryScheduler.standard()); + } + + /** + * A relay with an explicit retry policy. + * + *

    The scheduler was built by the auto-configuration and handed to nobody: the relay claimed + * every ambiguous row on the next pass regardless of its attempt count, so the configured budget + * of ten attempts was a number nothing read. + */ + public OutboxRelay( + OutboxRepository repository, + MessagePublisher publisher, + OutboxEnvelopeFactory envelopeFactory, + int batchSize, + Duration leaseDuration, + String owner, + OutboxRetryScheduler scheduler) { + this.owner = Objects.requireNonNull(owner, "owner must not be null"); + this.scheduler = Objects.requireNonNull(scheduler, "scheduler must not be null"); this.repository = Objects.requireNonNull(repository, "repository must not be null"); this.publisher = Objects.requireNonNull(publisher, "publisher must not be null"); this.envelopeFactory = @@ -66,32 +123,46 @@ public final class OutboxRelay { */ public OutboxRelayReport runOnce(Instant now) { Objects.requireNonNull(now, "now must not be null"); - List batch = repository.leaseBatch(batchSize, leaseDuration, now); + // claimBatch, not leaseBatch: the claim carries an owner and a fencing token, and every write + // below is conditioned on them. Without that, a relay that stalled past its lease could record + // AMBIGUOUS over the PUBLISHED another relay had already written, making a confirmed message + // claimable again. + List batch = + repository.claimBatch(owner, batchSize, leaseDuration, now, scheduler.maxAttempts()); int published = 0; int ambiguous = 0; int failed = 0; + int stale = 0; - for (OutboxRecord record : batch) { + for (OutboxLease lease : batch) { + OutboxRecord record = lease.record(); PublishResult result = publish(record); switch (result.completion()) { case CONFIRMED -> { - repository.markPublished(record.messageId(), now); - published++; + if (repository.markPublished(lease, now) == OutboxTransitionResult.APPLIED) { + published++; + } else { + stale++; + } } case AMBIGUOUS -> { - repository.markAmbiguous( - record.messageId(), - result.failure().map(failure -> failure.code()).orElse("AMBIGUOUS"), - now); - ambiguous++; + if (repository.markAmbiguous( + lease, result.failure().map(failure -> failure.code()).orElse("AMBIGUOUS"), now) + == OutboxTransitionResult.APPLIED) { + ambiguous++; + } else { + stale++; + } } case REJECTED -> { - repository.markFailed( - record.messageId(), - result.failure().map(failure -> failure.code()).orElse("REJECTED"), - now); - failed++; + if (repository.markFailed( + lease, result.failure().map(failure -> failure.code()).orElse("REJECTED"), now) + == OutboxTransitionResult.APPLIED) { + failed++; + } else { + stale++; + } } // Unreachable while PublishCompletion has exactly these constants. Present so that a new // completion fails loudly rather than leaving the row IN_FLIGHT with its lease ticking — @@ -100,7 +171,7 @@ public final class OutboxRelay { throw new IllegalStateException("unhandled publish completion: " + result.completion()); } } - return new OutboxRelayReport(batch.size(), published, ambiguous, failed); + return new OutboxRelayReport(batch.size(), published, ambiguous, failed, stale); } private PublishResult publish(OutboxRecord record) { diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelayReport.java b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelayReport.java new file mode 100644 index 0000000..493576c --- /dev/null +++ b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelayReport.java @@ -0,0 +1,34 @@ +package dev.caskeleton.messaging.outbox; + +/** + * What one relay pass did. + * + *

    Ambiguous rows are counted separately from failures because they mean something operationally + * different: a rising ambiguous count is a broker confirmation problem, while a rising failed count + * is a contract or topology problem. + * + *

    {@code staleLeases} is the count of terminal writes this pass could not apply because another + * relay had already taken the row. It is not an error and it is not a success: it means this worker + * published something whose outcome another worker has already recorded, so a non-zero value is the + * visible form of a duplicate publication. + * + * @param leased how many rows were claimed + * @param published how many were confirmed + * @param ambiguous how many had an unknown outcome and stay retryable + * @param failed how many were definitively rejected + * @param staleLeases how many terminal writes were refused because the lease had been superseded + */ +public record OutboxRelayReport( + int leased, int published, int ambiguous, int failed, int staleLeases) { + + public OutboxRelayReport { + if (leased < 0 || published < 0 || ambiguous < 0 || failed < 0 || staleLeases < 0) { + throw new IllegalArgumentException("relay counters must not be negative"); + } + } + + /** A report from a pass that observed no stale leases. */ + public OutboxRelayReport(int leased, int published, int ambiguous, int failed) { + this(leased, published, ambiguous, failed, 0); + } +} diff --git a/messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/OutboxRetryScheduler.java b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRetryScheduler.java similarity index 91% rename from messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/OutboxRetryScheduler.java rename to messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRetryScheduler.java index fd01c88..093933d 100644 --- a/messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/OutboxRetryScheduler.java +++ b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRetryScheduler.java @@ -58,6 +58,16 @@ public final class OutboxRetryScheduler { return now.plus(backoff(consecutiveEmptyOrFailedPasses)); } + /** The platform's default policy, for a relay whose caller did not supply one. */ + public static OutboxRetryScheduler standard() { + return new OutboxRetryScheduler(OutboxProperties.defaults(), java.time.Duration.ofMinutes(5)); + } + + /** The attempt budget a row may spend before it is parked. */ + public int maxAttempts() { + return maxAttempts; + } + /** * Returns the delay before the next pass. * diff --git a/messaging/messaging-outbox-jpa/src/main/resources/db/migration/messaging/V1__messaging_outbox.sql b/messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V1__messaging_outbox.sql similarity index 100% rename from messaging/messaging-outbox-jpa/src/main/resources/db/migration/messaging/V1__messaging_outbox.sql rename to messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V1__messaging_outbox.sql diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V2__messaging_outbox_lease_fencing.sql b/messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V2__messaging_outbox_lease_fencing.sql new file mode 100644 index 0000000..d1cdd1a --- /dev/null +++ b/messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V2__messaging_outbox_lease_fencing.sql @@ -0,0 +1,39 @@ +-- Lease owner and fencing token. +-- +-- V1 recorded only lease_expires_at, so a claim said when it would end and nothing about who held +-- it. The relay's terminal writes matched on message_id alone, which means a worker that stalled +-- past its lease could still write: +-- +-- relay A claims the row and calls the broker +-- the lease expires; relay B reclaims it, publishes, and records PUBLISHED +-- relay A finally times out and records AMBIGUOUS over the top +-- +-- The row is now claimable again and the message is published a second time. Making the lease +-- longer than the publish timeout lowers the odds; it does not turn a GC pause, a scheduler stall +-- or a slow broker into a data constraint. A token does: every terminal write names the +-- acquisition it belongs to, and a superseded worker's write matches nothing. +ALTER TABLE messaging_outbox + ADD COLUMN lease_owner VARCHAR(160), + ADD COLUMN lease_token BIGINT NOT NULL DEFAULT 0, + ADD COLUMN next_attempt_at TIMESTAMPTZ; + +-- Backfill is unnecessary for correctness — the default is 0 and the first claim increments it — +-- but the constraint states the invariant the code depends on. +ALTER TABLE messaging_outbox + ADD CONSTRAINT ck_messaging_outbox_lease_token CHECK (lease_token >= 0); + +-- EXHAUSTED is a new terminal state: the attempt budget ran out without any confirmation, which is +-- not the same as the broker rejecting the message. V1's CHECK listed five states, so writing the +-- sixth failed at the constraint rather than at review. +ALTER TABLE messaging_outbox + DROP CONSTRAINT ck_messaging_outbox_status; + +ALTER TABLE messaging_outbox + ADD CONSTRAINT ck_messaging_outbox_status + CHECK (status IN ('PENDING', 'IN_FLIGHT', 'PUBLISHED', 'AMBIGUOUS', 'FAILED', 'EXHAUSTED')); + +-- The claim predicate now filters on next_attempt_at as well, so the partial index has to carry it +-- or every poll degrades into a scan of the backlog once rows start backing off. +CREATE INDEX ix_messaging_outbox_next_attempt + ON messaging_outbox (next_attempt_at, created_at) + WHERE status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT'); diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V3__messaging_admin_operation_journal.sql b/messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V3__messaging_admin_operation_journal.sql new file mode 100644 index 0000000..9c79ca8 --- /dev/null +++ b/messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V3__messaging_admin_operation_journal.sql @@ -0,0 +1,37 @@ +-- The destructive-operation journal. +-- +-- Keyed on (approval_ticket, plan_digest) rather than the ticket alone: the ticket alone cannot +-- distinguish "this approval already ran" from "this approval is being presented for a different +-- plan", and those need opposite answers. The primary key is what makes two replicas racing for the +-- same approval resolve to one winner rather than two executions. +-- +-- lease_owner/lease_token/lease_expires_at are the same fencing pattern the outbox uses: a runtime +-- that stalls past its lease loses it, and its later writes are refused because its token is stale. +-- items_completed is the resume point — without it a retry after a crash republishes everything the +-- first attempt already moved. + +CREATE TABLE IF NOT EXISTS messaging_admin_operation ( + approval_ticket VARCHAR(200) NOT NULL, + plan_digest CHAR(64) NOT NULL, + operation_id VARCHAR(200) NOT NULL, + state VARCHAR(20) NOT NULL, + items_completed INTEGER NOT NULL DEFAULT 0, + lease_owner VARCHAR(200) NOT NULL, + lease_token BIGINT NOT NULL DEFAULT 1, + lease_expires_at TIMESTAMPTZ NOT NULL, + started_at TIMESTAMPTZ NOT NULL, + finished_at TIMESTAMPTZ, + failure_code VARCHAR(200), + CONSTRAINT pk_messaging_admin_operation PRIMARY KEY (approval_ticket, plan_digest), + CONSTRAINT ck_messaging_admin_operation_state + CHECK (state IN ('STARTED', 'COMPLETED', 'FAILED')), + CONSTRAINT ck_messaging_admin_operation_items CHECK (items_completed >= 0), + CONSTRAINT ck_messaging_admin_operation_token CHECK (lease_token >= 1), + CONSTRAINT ck_messaging_admin_operation_digest CHECK (plan_digest ~ '^[0-9a-f]{64}$') +); + +-- Answers "which operations are in flight right now", the question an operator asks when a redrive +-- appears stuck and they are deciding whether to take it over. +CREATE INDEX IF NOT EXISTS ix_messaging_admin_operation_live + ON messaging_admin_operation (lease_expires_at) + WHERE state = 'STARTED'; diff --git a/messaging/messaging-outbox-jpa/src/main/resources/debezium/outbox-event-router.properties b/messaging/messaging-outbox-jdbc-postgresql/src/main/resources/debezium/outbox-event-router.properties similarity index 100% rename from messaging/messaging-outbox-jpa/src/main/resources/debezium/outbox-event-router.properties rename to messaging/messaging-outbox-jdbc-postgresql/src/main/resources/debezium/outbox-event-router.properties diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/AdminOperationJournalPostgresIT.java b/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/AdminOperationJournalPostgresIT.java new file mode 100644 index 0000000..4228516 --- /dev/null +++ b/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/AdminOperationJournalPostgresIT.java @@ -0,0 +1,206 @@ +package dev.caskeleton.messaging.outbox; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatCode; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import dev.caskeleton.messaging.admin.AdminOperationLease; +import dev.caskeleton.messaging.admin.AdminOperationState; +import dev.caskeleton.messaging.admin.PlanDigest; +import dev.caskeleton.messaging.api.error.MessageAuthorizationException; +import dev.caskeleton.messaging.testkit.DockerAvailability; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.sql.Connection; +import java.sql.SQLException; +import java.sql.Statement; +import java.time.Duration; +import java.time.Instant; +import javax.sql.DataSource; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIf; +import org.postgresql.ds.PGSimpleDataSource; +import org.testcontainers.junit.jupiter.Container; +import org.testcontainers.junit.jupiter.Testcontainers; +import org.testcontainers.postgresql.PostgreSQLContainer; + +/** + * Certifies the durable journal against a real PostgreSQL. + * + *

    The properties that matter here belong to the database, not to Java. "Exactly one replica + * executes an approval" is a primary-key conflict; "a stalled runtime cannot overwrite its + * successor" is a conditional update on a fencing token. An in-memory double can be written to + * agree with both without either being true of the deployed schema, which is exactly how the map + * this replaced looked correct for as long as nobody restarted a replica. + */ +@Testcontainers +@EnabledIf("dockerAvailable") +class AdminOperationJournalPostgresIT { + + private static final Instant NOW = Instant.parse("2026-08-10T09:15:00Z"); + private static final Duration LEASE = Duration.ofMinutes(5); + private static final String TICKET = "CHG-1042"; + private static final PlanDigest DIGEST = PlanDigest.ofCanonical("REDRIVE|orders.v1.dlq|50"); + + @Container + private static final PostgreSQLContainer POSTGRES = + new PostgreSQLContainer("postgres:16-alpine") + .withDatabaseName("messaging") + .withUsername("messaging") + .withPassword("messaging"); + + private JdbcAdminOperationJournal journal; + + static boolean dockerAvailable() { + return DockerAvailability.isAvailable(); + } + + @BeforeEach + void migrate() throws SQLException, IOException { + DataSource dataSource = dataSource(); + try (Connection connection = dataSource.getConnection(); + Statement statement = connection.createStatement()) { + statement.execute("DROP TABLE IF EXISTS messaging_admin_operation"); + statement.execute(migration("V3__messaging_admin_operation_journal.sql")); + } + journal = new JdbcAdminOperationJournal(dataSource); + } + + @Test + @DisplayName("the primary key, not application code, decides which replica wins") + void thePrimaryKeyDecidesWhichReplicaWins() { + journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW); + + assertThatThrownBy( + () -> journal.begin(TICKET, DIGEST, "op-2", "replica-b", LEASE, NOW.plusSeconds(1))) + .isInstanceOf(MessageAuthorizationException.class) + .hasMessageContaining("replica-a"); + } + + @Test + @DisplayName("a completed operation survives the restart that forgot the map") + void aCompletedOperationSurvivesARestart() { + AdminOperationLease lease = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW); + journal.complete(lease, 50, NOW.plusSeconds(30)); + + // A fresh journal object over the same database is what a restarted replica sees. + JdbcAdminOperationJournal afterRestart = new JdbcAdminOperationJournal(dataSource()); + + assertThatThrownBy( + () -> + afterRestart.begin(TICKET, DIGEST, "op-3", "replica-a", LEASE, NOW.plusSeconds(60))) + .isInstanceOf(MessageAuthorizationException.class) + .hasMessageContaining("already executed"); + } + + @Test + @DisplayName("an expired lease is taken over at its checkpoint, under a higher token") + void anExpiredLeaseIsTakenOverAtItsCheckpoint() { + AdminOperationLease first = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW); + journal.checkpoint(first, 30, LEASE, NOW.plusSeconds(10)); + + AdminOperationLease resumed = + journal.begin(TICKET, DIGEST, "op-1", "replica-b", LEASE, NOW.plus(Duration.ofMinutes(10))); + + assertThat(resumed.resumeFrom()).isEqualTo(30); + assertThat(resumed.leaseToken()).isEqualTo(first.leaseToken() + 1); + } + + @Test + @DisplayName("a live lease is not taken over") + void aLiveLeaseIsNotTakenOver() { + journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW); + + assertThatThrownBy( + () -> + journal.begin( + TICKET, DIGEST, "op-1", "replica-b", LEASE, NOW.plus(Duration.ofMinutes(1)))) + .isInstanceOf(MessageAuthorizationException.class) + .hasMessageContaining("is being executed by"); + } + + @Test + @DisplayName("a superseded runtime cannot write over the one that took over") + void aSupersededRuntimeCannotWriteOverItsSuccessor() { + AdminOperationLease stale = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW); + journal.checkpoint(stale, 30, LEASE, NOW.plusSeconds(10)); + AdminOperationLease successor = + journal.begin(TICKET, DIGEST, "op-1", "replica-b", LEASE, NOW.plus(Duration.ofMinutes(10))); + journal.checkpoint(successor, 45, LEASE, NOW.plus(Duration.ofMinutes(11))); + + assertThatThrownBy(() -> journal.complete(stale, 30, NOW.plus(Duration.ofMinutes(12)))) + .isInstanceOf(MessageAuthorizationException.class) + .hasMessageContaining("no longer holds the lease"); + + assertThat(journal.find(TICKET, DIGEST)) + .hasValueSatisfying( + record -> { + assertThat(record.itemsCompleted()).isEqualTo(45); + assertThat(record.state()).isEqualTo(AdminOperationState.STARTED); + }); + } + + @Test + @DisplayName("a failed operation is resumable immediately and records why it stopped") + void aFailedOperationIsResumableImmediately() { + AdminOperationLease first = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW); + journal.fail(first, 12, "MESSAGE_BROKER_UNAVAILABLE", NOW.plusSeconds(6)); + + assertThat(journal.find(TICKET, DIGEST)) + .hasValueSatisfying( + record -> { + assertThat(record.state()).isEqualTo(AdminOperationState.FAILED); + assertThat(record.failureCode()).contains("MESSAGE_BROKER_UNAVAILABLE"); + }); + + assertThatCode( + () -> { + AdminOperationLease resumed = + journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW.plusSeconds(7)); + assertThat(resumed.resumeFrom()).isEqualTo(12); + }) + .doesNotThrowAnyException(); + } + + @Test + @DisplayName("the schema refuses a digest that is not a SHA-256") + void theSchemaRefusesADigestThatIsNotASha256() throws SQLException { + try (Connection connection = dataSource().getConnection(); + Statement statement = connection.createStatement()) { + assertThatThrownBy( + () -> + statement.execute( + """ + INSERT INTO messaging_admin_operation ( + approval_ticket, plan_digest, operation_id, state, + lease_owner, lease_expires_at, started_at) + VALUES ('CHG-9', 'not-a-digest', 'op', 'STARTED', 'r', now(), now()) + """)) + .as("a malformed digest in the journal makes the uniqueness key meaningless") + .isInstanceOf(SQLException.class); + } + } + + private static DataSource dataSource() { + PGSimpleDataSource source = new PGSimpleDataSource(); + source.setUrl(POSTGRES.getJdbcUrl()); + source.setUser(POSTGRES.getUsername()); + source.setPassword(POSTGRES.getPassword()); + return source; + } + + private static String migration(String fileName) throws IOException { + try (InputStream stream = + AdminOperationJournalPostgresIT.class + .getClassLoader() + .getResourceAsStream("db/migration/messaging/" + fileName)) { + if (stream == null) { + throw new IOException("migration " + fileName + " is missing from the classpath"); + } + return new String(stream.readAllBytes(), StandardCharsets.UTF_8); + } + } +} diff --git a/messaging/messaging-outbox-jpa/src/test/java/dev/caskeleton/messaging/outbox/DebeziumOutboxRecordMapperTest.java b/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/DebeziumOutboxRecordMapperTest.java similarity index 100% rename from messaging/messaging-outbox-jpa/src/test/java/dev/caskeleton/messaging/outbox/DebeziumOutboxRecordMapperTest.java rename to messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/DebeziumOutboxRecordMapperTest.java diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/JdbcOutboxTransactionRequirementTest.java b/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/JdbcOutboxTransactionRequirementTest.java new file mode 100644 index 0000000..0c05890 --- /dev/null +++ b/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/JdbcOutboxTransactionRequirementTest.java @@ -0,0 +1,120 @@ +package dev.caskeleton.messaging.outbox; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import dev.caskeleton.messaging.api.ContentType; +import dev.caskeleton.messaging.api.MessageId; +import dev.caskeleton.messaging.api.MessageType; +import dev.caskeleton.messaging.api.SchemaVersion; +import dev.caskeleton.messaging.api.destination.DestinationName; +import dev.caskeleton.messaging.api.error.MessagingConfigurationException; +import dev.caskeleton.messaging.reliability.OutboxRecord; +import dev.caskeleton.messaging.reliability.OutboxStatus; +import java.time.Instant; +import java.util.Map; +import java.util.Optional; +import javax.sql.DataSource; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.springframework.transaction.support.TransactionSynchronizationManager; + +/** + * What {@code append(record)} does when there is no transaction to join. + * + *

    The port says the append runs inside the caller's business transaction. The implementation + * called {@code dataSource.getConnection()}, which auto-commits on its own connection — so a + * business transaction that rolled back afterwards left the outbox row behind and the relay + * published an event for a change that never happened. + * + *

    These are hermetic: the refusal happens before any connection is requested, which is also the + * property under test. The mock would fail the test by being touched at all. + */ +class JdbcOutboxTransactionRequirementTest { + + /** A data source that fails the test by being used: the refusal must precede any connection. */ + private final DataSource dataSource = new RefusingDataSource(); + + private final JdbcOutboxRepository repository = new JdbcOutboxRepository(dataSource); + + @AfterEach + void clearTransactionState() { + TransactionSynchronizationManager.clear(); + if (TransactionSynchronizationManager.hasResource(dataSource)) { + TransactionSynchronizationManager.unbindResource(dataSource); + } + } + + @Test + @DisplayName("appending outside a transaction is refused before a connection is opened") + void appendingOutsideATransactionIsRefused() { + assertThatThrownBy(() -> repository.append(record())) + .isInstanceOf(MessagingConfigurationException.class) + .hasMessageContaining("requires an active transaction"); + } + + @Test + @DisplayName("a read-only transaction is refused") + void aReadOnlyTransactionIsRefused() { + TransactionSynchronizationManager.setActualTransactionActive(true); + TransactionSynchronizationManager.setCurrentTransactionReadOnly(true); + + assertThatThrownBy(() -> repository.append(record())) + .isInstanceOf(MessagingConfigurationException.class) + .hasMessageContaining("read-only"); + } + + @Test + @DisplayName("a transaction bound to another DataSource is refused") + void aTransactionOnAnotherDataSourceIsRefused() { + TransactionSynchronizationManager.setActualTransactionActive(true); + + assertThatThrownBy(() -> repository.append(record())) + .as("two DataSources means two commits, which is the atomicity this exists to provide") + .isInstanceOf(MessagingConfigurationException.class) + .hasMessageContaining("bound to this DataSource"); + } + + @Test + @DisplayName("the refusal names a code an operator can search for") + void theRefusalNamesACode() { + assertThatThrownBy(() -> repository.append(record())) + .isInstanceOfSatisfying( + MessagingConfigurationException.class, + failure -> + assertThat(failure.failure().code()).isEqualTo("OUTBOX_TRANSACTION_REQUIRED")); + } + + private static OutboxRecord record() { + return new OutboxRecord( + MessageId.newId(), + new DestinationName("orders.events"), + new MessageType("order.placed"), + new SchemaVersion(1), + ContentType.JSON, + "{}".getBytes(java.nio.charset.StandardCharsets.UTF_8), + Map.of(), + Instant.parse("2026-08-14T00:00:00Z"), + OutboxStatus.PENDING, + 0, + Optional.empty(), + Optional.empty()); + } + + /** Throws if anything asks it for a connection. */ + private static final class RefusingDataSource + extends org.springframework.jdbc.datasource.AbstractDataSource { + + @Override + public java.sql.Connection getConnection() { + throw new AssertionError( + "the refusal must happen before a connection is requested; opening one is the defect"); + } + + @Override + public java.sql.Connection getConnection(String username, String password) { + return getConnection(); + } + } +} diff --git a/messaging/messaging-outbox-jpa/src/test/java/dev/caskeleton/messaging/outbox/OutboxOperationsTest.java b/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxOperationsTest.java similarity index 69% rename from messaging/messaging-outbox-jpa/src/test/java/dev/caskeleton/messaging/outbox/OutboxOperationsTest.java rename to messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxOperationsTest.java index ff68af7..5425041 100644 --- a/messaging/messaging-outbox-jpa/src/test/java/dev/caskeleton/messaging/outbox/OutboxOperationsTest.java +++ b/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxOperationsTest.java @@ -40,6 +40,62 @@ class OutboxOperationsTest { throw new UnsupportedOperationException(); } + @Override + public java.util.List claimBatch( + String owner, int batchSize, Duration leaseDuration, Instant now) { + return java.util.List.of(); + } + + @Override + public java.util.List claimBatch( + String owner, int batchSize, Duration leaseDuration, Instant now, int maxAttempts) { + return java.util.List.of(); + } + + @Override + public dev.caskeleton.messaging.reliability.OutboxTransitionResult markAmbiguous( + dev.caskeleton.messaging.reliability.OutboxLease lease, + String failureCode, + Instant now, + Instant nextAttemptAt) { + return markAmbiguous(lease, failureCode, now); + } + + @Override + public dev.caskeleton.messaging.reliability.OutboxTransitionResult markExhausted( + dev.caskeleton.messaging.reliability.OutboxLease lease, String reason, Instant now) { + markFailed(lease.messageId(), reason, now); + return dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED; + } + + @Override + public dev.caskeleton.messaging.reliability.OutboxTransitionResult markPublished( + dev.caskeleton.messaging.reliability.OutboxLease lease, Instant now) { + markPublished(lease.messageId(), now); + return dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED; + } + + @Override + public dev.caskeleton.messaging.reliability.OutboxTransitionResult markAmbiguous( + dev.caskeleton.messaging.reliability.OutboxLease lease, String failureCode, Instant now) { + markAmbiguous(lease.messageId(), failureCode, now); + return dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED; + } + + @Override + public dev.caskeleton.messaging.reliability.OutboxTransitionResult markFailed( + dev.caskeleton.messaging.reliability.OutboxLease lease, String failureCode, Instant now) { + markFailed(lease.messageId(), failureCode, now); + return dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED; + } + + @Override + public dev.caskeleton.messaging.reliability.OutboxTransitionResult releaseLease( + dev.caskeleton.messaging.reliability.OutboxLease lease) { + releaseLease(lease.messageId()); + return dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED; + } + @Override public void markPublished(MessageId messageId, Instant now) { throw new UnsupportedOperationException(); @@ -65,6 +121,11 @@ class OutboxOperationsTest { throw new UnsupportedOperationException(); } + @Override + public int purgePublishedBefore(Instant publishedBefore, int limit) { + return Math.min(purgePublishedBefore(publishedBefore), limit); + } + @Override public int purgePublishedBefore(Instant publishedBefore) { cutoffs.add(publishedBefore); diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxPostgresIT.java b/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxPostgresIT.java new file mode 100644 index 0000000..2222e9f --- /dev/null +++ b/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxPostgresIT.java @@ -0,0 +1,417 @@ +package dev.caskeleton.messaging.outbox; + +import static org.assertj.core.api.Assertions.assertThat; + +import dev.caskeleton.messaging.api.ContentType; +import dev.caskeleton.messaging.api.MessageId; +import dev.caskeleton.messaging.api.MessageType; +import dev.caskeleton.messaging.api.SchemaVersion; +import dev.caskeleton.messaging.api.destination.DestinationName; +import dev.caskeleton.messaging.reliability.OutboxRecord; +import dev.caskeleton.messaging.reliability.OutboxStatus; +import dev.caskeleton.messaging.testkit.DockerAvailability; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.sql.Connection; +import java.sql.SQLException; +import java.sql.Statement; +import java.time.Duration; +import java.time.Instant; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import javax.sql.DataSource; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIf; +import org.postgresql.ds.PGSimpleDataSource; +import org.testcontainers.junit.jupiter.Container; +import org.testcontainers.junit.jupiter.Testcontainers; +import org.testcontainers.postgresql.PostgreSQLContainer; + +/** + * Certifies the outbox against a real PostgreSQL. + * + *

    The pattern's guarantees are claims about a database, not about Java: that a row and a + * business change commit together, that two relay instances cannot claim the same row, and that an + * ambiguous publish leaves the row claimable under its original identity. An in-memory double can + * be written to agree with any of those; only a real database settles them. + */ +@Testcontainers +@EnabledIf("dockerAvailable") +class OutboxPostgresIT { + + private static final Instant NOW = Instant.parse("2026-08-10T09:15:00Z"); + + @Container + private static final PostgreSQLContainer POSTGRES = + new PostgreSQLContainer("postgres:16-alpine") + .withDatabaseName("messaging") + .withUsername("messaging") + .withPassword("messaging"); + + private DataSource dataSource; + + private org.springframework.transaction.support.TransactionTemplate transactions; + private JdbcOutboxRepository repository; + + static boolean dockerAvailable() { + return DockerAvailability.isAvailable(); + } + + @BeforeEach + void migrate() throws SQLException, IOException { + dataSource = dataSource(); + try (Connection connection = dataSource.getConnection(); + Statement statement = connection.createStatement()) { + statement.execute("DROP TABLE IF EXISTS messaging_outbox"); + // Every migration in order, not just V1: a test that applies part of the schema certifies a + // database nobody deploys. + for (String migration : migrations()) { + statement.execute(migration); + } + } + repository = new JdbcOutboxRepository(dataSource); + // A real Spring transaction: the port's contract is that append joins the caller's, and the + // previous tests proved only that the Connection overload worked. + transactions = + new org.springframework.transaction.support.TransactionTemplate( + new org.springframework.jdbc.datasource.DataSourceTransactionManager(dataSource)); + } + + @Test + void anAppendedRowIsClaimableAndRoundTripsIntact() { + OutboxRecord record = record(); + appendInTransaction(record); + + List leased = repository.leaseBatch(10, Duration.ofSeconds(30), NOW); + + assertThat(leased) + .singleElement() + .satisfies( + leasedRecord -> { + assertThat(leasedRecord.messageId()).isEqualTo(record.messageId()); + assertThat(leasedRecord.messageType()).isEqualTo(record.messageType()); + assertThat(leasedRecord.payload()).isEqualTo(record.payload()); + assertThat(leasedRecord.headers()).containsEntry("msg.partition-key", "acct-1"); + }); + } + + @Test + void aLeasedRowIsInvisibleToASecondRelayInstance() { + appendInTransaction(record()); + + List first = repository.leaseBatch(10, Duration.ofSeconds(30), NOW); + List second = repository.leaseBatch(10, Duration.ofSeconds(30), NOW); + + assertThat(first).hasSize(1); + assertThat(second).as("SKIP LOCKED plus the lease keeps two relays off the same row").isEmpty(); + } + + @Test + void anExpiredLeaseBecomesClaimableAgain() { + appendInTransaction(record()); + repository.leaseBatch(10, Duration.ofSeconds(30), NOW); + + List reclaimed = + repository.leaseBatch(10, Duration.ofSeconds(30), NOW.plusSeconds(31)); + + assertThat(reclaimed).hasSize(1); + } + + @Test + void anAmbiguousRowStaysClaimableUnderItsOriginalIdentity() { + OutboxRecord record = record(); + appendInTransaction(record); + repository.leaseBatch(10, Duration.ofSeconds(30), NOW); + + repository.markAmbiguous(record.messageId(), "CONFIRM_TIMEOUT", NOW); + + assertThat(repository.find(record.messageId()).orElseThrow().status()) + .isEqualTo(OutboxStatus.AMBIGUOUS); + assertThat(repository.leaseBatch(10, Duration.ofSeconds(30), NOW)) + .singleElement() + .satisfies(retry -> assertThat(retry.messageId()).isEqualTo(record.messageId())); + } + + @Test + void aPublishedRowIsNotClaimedAgain() { + OutboxRecord record = record(); + appendInTransaction(record); + repository.leaseBatch(10, Duration.ofSeconds(30), NOW); + + repository.markPublished(record.messageId(), NOW); + + assertThat(repository.find(record.messageId()).orElseThrow().status()) + .isEqualTo(OutboxStatus.PUBLISHED); + assertThat(repository.leaseBatch(10, Duration.ofSeconds(30), NOW.plusSeconds(60))).isEmpty(); + } + + @Test + void aFailedRowIsNotClaimedAgainWithoutIntervention() { + OutboxRecord record = record(); + appendInTransaction(record); + repository.leaseBatch(10, Duration.ofSeconds(30), NOW); + + repository.markFailed(record.messageId(), "INVALID_TOPIC", NOW); + + assertThat(repository.leaseBatch(10, Duration.ofSeconds(30), NOW.plusSeconds(60))).isEmpty(); + } + + @Test + void theRowAndTheBusinessChangeCommitTogetherOrNotAtAll() throws SQLException { + OutboxRecord record = record(); + new org.springframework.jdbc.core.JdbcTemplate(dataSource) + .execute("CREATE TABLE IF NOT EXISTS orders (id TEXT PRIMARY KEY)"); + + // The interface method, inside a real Spring transaction that rolls back. This is the path + // business code takes; the old version called the Connection overload, which was never the + // one at risk. + try { + transactions.executeWithoutResult( + status -> { + new org.springframework.jdbc.core.JdbcTemplate(dataSource) + .execute("INSERT INTO orders (id) VALUES ('o-1')"); + repository.append(record); + status.setRollbackOnly(); + }); + } catch (RuntimeException rolledBack) { + // setRollbackOnly is the mechanism; the rollback exception is not a failure here. + } + + assertThat(repository.find(record.messageId())) + .as("a rolled back business change must not leave a publishable row") + .isEmpty(); + } + + @Test + void retentionRemovesOnlyPublishedRows() { + OutboxRecord published = record(); + OutboxRecord pending = record(); + appendInTransaction(published); + appendInTransaction(pending); + repository.markPublished(published.messageId(), NOW); + + int removed = repository.purgePublishedBefore(NOW.plusSeconds(1)); + + assertThat(removed).isEqualTo(1); + assertThat(repository.find(pending.messageId())).isPresent(); + } + + private static OutboxRecord record() { + return new OutboxRecord( + MessageId.newId(), + new DestinationName("order-events"), + new MessageType("order.created"), + new SchemaVersion(1), + ContentType.JSON, + "{\"orderId\":\"o-1\"}".getBytes(StandardCharsets.UTF_8), + Map.of("msg.partition-key", "acct-1"), + NOW.minusSeconds(60), + OutboxStatus.PENDING, + 0, + Optional.empty(), + Optional.empty()); + } + + private static DataSource dataSource() { + PGSimpleDataSource source = new PGSimpleDataSource(); + source.setUrl(POSTGRES.getJdbcUrl()); + source.setUser(POSTGRES.getUsername()); + source.setPassword(POSTGRES.getPassword()); + return source; + } + + private static java.util.List migrations() throws IOException { + return java.util.List.of( + migration("V1__messaging_outbox.sql"), migration("V2__messaging_outbox_lease_fencing.sql")); + } + + private static String migration(String fileName) throws IOException { + try (InputStream stream = + OutboxPostgresIT.class + .getClassLoader() + .getResourceAsStream("db/migration/messaging/" + fileName)) { + if (stream == null) { + throw new IOException("migration " + fileName + " is missing from the classpath"); + } + return new String(stream.readAllBytes(), StandardCharsets.UTF_8); + } + } + + @Test + void aSupersededRelayCannotOverwriteTheOutcomeOfTheOneThatReplacedIt() { + OutboxRecord record = record(); + appendInTransaction(record); + + // Relay A claims and stalls. + java.util.List first = + repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW); + assertThat(first).hasSize(1); + + // The lease expires and relay B takes the row and publishes it. + java.util.List second = + repository.claimBatch("relay-b", 10, Duration.ofSeconds(30), NOW.plusSeconds(31)); + assertThat(second).hasSize(1); + assertThat(second.get(0).token()) + .as("every claim increments the token, which is what makes the older one detectable") + .isGreaterThan(first.get(0).token()); + assertThat(repository.markPublished(second.get(0), NOW.plusSeconds(32))) + .isEqualTo(dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED); + + // Relay A finally wakes up and tries to record its own outcome. + assertThat(repository.markAmbiguous(first.get(0), "CONFIRM_TIMEOUT", NOW.plusSeconds(40))) + .as("this write used to succeed and made a published message claimable again") + .isEqualTo(dev.caskeleton.messaging.reliability.OutboxTransitionResult.STALE_LEASE); + assertThat(repository.find(record.messageId()).orElseThrow().status()) + .isEqualTo(OutboxStatus.PUBLISHED); + } + + @Test + void aDuplicateTerminalCallFromTheSameRelayIsRefused() { + OutboxRecord record = record(); + appendInTransaction(record); + dev.caskeleton.messaging.reliability.OutboxLease lease = + repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW).get(0); + + assertThat(repository.markPublished(lease, NOW)) + .isEqualTo(dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED); + assertThat(repository.markPublished(lease, NOW)) + .as("the row is no longer IN_FLIGHT, so the second write has nothing to apply to") + .isEqualTo(dev.caskeleton.messaging.reliability.OutboxTransitionResult.STALE_LEASE); + } + + @Test + void twoRelaysClaimingConcurrentlyGetDisjointRowsAndDistinctTokens() { + OutboxRecord one = record(); + OutboxRecord two = record(); + appendInTransaction(one); + appendInTransaction(two); + + java.util.List a = + repository.claimBatch("relay-a", 1, Duration.ofSeconds(30), NOW); + java.util.List b = + repository.claimBatch("relay-b", 1, Duration.ofSeconds(30), NOW); + + assertThat(a).hasSize(1); + assertThat(b).hasSize(1); + assertThat(a.get(0).messageId()).isNotEqualTo(b.get(0).messageId()); + } + + @Test + void anExpiryReclaimKeepsTheMessageIdAndAdvancesTheToken() { + OutboxRecord record = record(); + appendInTransaction(record); + + long firstToken = + repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW).get(0).token(); + dev.caskeleton.messaging.reliability.OutboxLease reclaimed = + repository.claimBatch("relay-b", 10, Duration.ofSeconds(30), NOW.plusSeconds(31)).get(0); + + assertThat(reclaimed.messageId()).isEqualTo(record.messageId()); + assertThat(reclaimed.token()).isEqualTo(firstToken + 1); + } + + @Test + void anAmbiguousRowWaitsForItsBackoffBeforeItIsClaimedAgain() { + OutboxRecord record = record(); + appendInTransaction(record); + dev.caskeleton.messaging.reliability.OutboxLease lease = + repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW, 10).get(0); + + repository.markAmbiguous(lease, "CONFIRM_TIMEOUT", NOW, NOW.plusSeconds(120)); + + assertThat(repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW.plusSeconds(1), 10)) + .as( + "an ambiguous row used to be claimable on the very next pass, so an outage republished" + + " the whole backlog every poll interval") + .isEmpty(); + assertThat( + repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW.plusSeconds(121), 10)) + .hasSize(1); + } + + @Test + void aRowOutOfAttemptsIsNotClaimedAgain() { + OutboxRecord record = record(); + appendInTransaction(record); + + Instant clock = NOW; + for (int attempt = 0; attempt < 3; attempt++) { + dev.caskeleton.messaging.reliability.OutboxLease lease = + repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), clock, 3).get(0); + repository.markAmbiguous(lease, "CONFIRM_TIMEOUT", clock, clock); + clock = clock.plusSeconds(60); + } + + assertThat(repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), clock, 3)) + .as("the configured budget was a number nothing consulted") + .isEmpty(); + } + + @Test + void anExhaustedRowIsDistinctFromARejectedOne() { + OutboxRecord record = record(); + appendInTransaction(record); + dev.caskeleton.messaging.reliability.OutboxLease lease = + repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW, 10).get(0); + + repository.markExhausted(lease, "budget spent without a confirmation", NOW); + + assertThat(repository.find(record.messageId()).orElseThrow().status()) + .as("'the broker refused this' and 'nobody ever answered' need different operator actions") + .isEqualTo(OutboxStatus.EXHAUSTED); + assertThat( + repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW.plusSeconds(600), 10)) + .isEmpty(); + } + + @Test + void aConfirmedPublishClearsTheRetryClock() { + OutboxRecord record = record(); + appendInTransaction(record); + dev.caskeleton.messaging.reliability.OutboxLease first = + repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW, 10).get(0); + repository.markAmbiguous(first, "CONFIRM_TIMEOUT", NOW, NOW.plusSeconds(60)); + + dev.caskeleton.messaging.reliability.OutboxLease second = + repository + .claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW.plusSeconds(61), 10) + .get(0); + repository.markPublished(second, NOW.plusSeconds(62)); + + assertThat(repository.find(record.messageId()).orElseThrow().status()) + .isEqualTo(OutboxStatus.PUBLISHED); + } + + @Test + void aHeaderValueWithControlCharactersRoundTrips() { + OutboxRecord record = + new OutboxRecord( + MessageId.newId(), + new DestinationName("order-events"), + new MessageType("order.created"), + new SchemaVersion(1), + ContentType.JSON, + "{}".getBytes(StandardCharsets.UTF_8), + // A tab and a newline in a header value. Only backslash and quote were escaped, so + // this produced a document PostgreSQL refuses as invalid JSONB and the append failed + // with an error naming neither the header nor the row. + Map.of("msg.note", "line one\nline\ttwo"), + NOW.minusSeconds(60), + OutboxStatus.PENDING, + 0, + Optional.empty(), + Optional.empty()); + + appendInTransaction(record); + + assertThat(repository.find(record.messageId()).orElseThrow().headers()) + .containsEntry("msg.note", "line one\nline\ttwo"); + } + + private void appendInTransaction(OutboxRecord record) { + transactions.executeWithoutResult(status -> repository.append(record)); + } +} diff --git a/messaging/messaging-outbox-jpa/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayTest.java b/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayTest.java similarity index 71% rename from messaging/messaging-outbox-jpa/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayTest.java rename to messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayTest.java index 77e3008..fc00e4d 100644 --- a/messaging/messaging-outbox-jpa/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayTest.java +++ b/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayTest.java @@ -19,9 +19,11 @@ import dev.caskeleton.messaging.api.publish.PublishEvidence; import dev.caskeleton.messaging.api.publish.PublishOptions; import dev.caskeleton.messaging.api.publish.PublishResult; import dev.caskeleton.messaging.api.publish.RoutingOutcome; +import dev.caskeleton.messaging.reliability.OutboxLease; import dev.caskeleton.messaging.reliability.OutboxRecord; import dev.caskeleton.messaging.reliability.OutboxRepository; import dev.caskeleton.messaging.reliability.OutboxStatus; +import dev.caskeleton.messaging.reliability.OutboxTransitionResult; import java.nio.charset.StandardCharsets; import java.time.Duration; import java.time.Instant; @@ -163,6 +165,11 @@ final class InMemoryOutboxRepository implements OutboxRepository { private final Map rows = new LinkedHashMap<>(); private final Map leases = new LinkedHashMap<>(); + /** Owner and token per row, mirroring the fencing columns the SQL now carries. */ + private final Map owners = new LinkedHashMap<>(); + + private final Map tokens = new LinkedHashMap<>(); + @Override public void append(OutboxRecord record) { rows.put(record.messageId(), record); @@ -201,6 +208,112 @@ final class InMemoryOutboxRepository implements OutboxRepository { return List.copyOf(leased); } + @Override + public List claimBatch( + String owner, int batchSize, Duration leaseDuration, Instant now) { + List claimed = new ArrayList<>(); + for (OutboxRecord record : leaseBatch(batchSize, leaseDuration, now)) { + long token = tokens.merge(record.messageId(), 1L, Long::sum); + owners.put(record.messageId(), owner); + claimed.add(new OutboxLease(record, owner, token, now.plus(leaseDuration))); + } + return List.copyOf(claimed); + } + + /** + * Refuses a write whose lease has been superseded, exactly as the SQL predicate does. + * + *

    A double that accepted every terminal write would let the relay's stale-lease handling look + * correct without ever being exercised. + */ + private boolean owns(OutboxLease lease) { + return lease.owner().equals(owners.get(lease.messageId())) + && Long.valueOf(lease.token()).equals(tokens.get(lease.messageId())) + && rows.get(lease.messageId()) != null + && rows.get(lease.messageId()).status() == OutboxStatus.IN_FLIGHT; + } + + /** Per-row retry clock, mirroring next_attempt_at and the attempt budget in the SQL. */ + private final Map nextAttempts = new LinkedHashMap<>(); + + @Override + public List claimBatch( + String owner, int batchSize, Duration leaseDuration, Instant now, int maxAttempts) { + List claimed = new ArrayList<>(); + for (OutboxLease lease : claimBatch(owner, batchSize, leaseDuration, now)) { + Instant nextAttempt = nextAttempts.get(lease.messageId()); + boolean waiting = nextAttempt != null && now.isBefore(nextAttempt); + if (waiting || lease.record().attempts() >= maxAttempts) { + // Put it back exactly as the SQL predicate would have left it: unclaimed. + releaseLease(lease.messageId()); + continue; + } + claimed.add(lease); + } + return List.copyOf(claimed); + } + + @Override + public OutboxTransitionResult markAmbiguous( + OutboxLease lease, String failureCode, Instant now, Instant nextAttemptAt) { + OutboxTransitionResult result = markAmbiguous(lease, failureCode, now); + if (result == OutboxTransitionResult.APPLIED) { + nextAttempts.put(lease.messageId(), nextAttemptAt); + } + return result; + } + + @Override + public OutboxTransitionResult markExhausted(OutboxLease lease, String reason, Instant now) { + if (!owns(lease)) { + return OutboxTransitionResult.STALE_LEASE; + } + transition(lease.messageId(), OutboxStatus.EXHAUSTED, Optional.of(reason)); + owners.remove(lease.messageId()); + leases.remove(lease.messageId()); + return OutboxTransitionResult.APPLIED; + } + + @Override + public OutboxTransitionResult markPublished(OutboxLease lease, Instant now) { + if (!owns(lease)) { + return OutboxTransitionResult.STALE_LEASE; + } + transition(lease.messageId(), OutboxStatus.PUBLISHED, Optional.empty()); + owners.remove(lease.messageId()); + return OutboxTransitionResult.APPLIED; + } + + @Override + public OutboxTransitionResult markAmbiguous(OutboxLease lease, String failureCode, Instant now) { + if (!owns(lease)) { + return OutboxTransitionResult.STALE_LEASE; + } + markAmbiguous(lease.messageId(), failureCode, now); + owners.remove(lease.messageId()); + return OutboxTransitionResult.APPLIED; + } + + @Override + public OutboxTransitionResult markFailed(OutboxLease lease, String failureCode, Instant now) { + if (!owns(lease)) { + return OutboxTransitionResult.STALE_LEASE; + } + markFailed(lease.messageId(), failureCode, now); + owners.remove(lease.messageId()); + return OutboxTransitionResult.APPLIED; + } + + @Override + public OutboxTransitionResult releaseLease(OutboxLease lease) { + if (!owns(lease)) { + return OutboxTransitionResult.STALE_LEASE; + } + releaseLease(lease.messageId()); + owners.remove(lease.messageId()); + return OutboxTransitionResult.APPLIED; + } + @Override public void markPublished(MessageId messageId, Instant now) { transition(messageId, OutboxStatus.PUBLISHED, Optional.empty()); @@ -227,6 +340,11 @@ final class InMemoryOutboxRepository implements OutboxRepository { return Optional.ofNullable(rows.get(messageId)); } + @Override + public int purgePublishedBefore(Instant publishedBefore, int limit) { + return Math.min(purgePublishedBefore(publishedBefore), limit); + } + @Override public int purgePublishedBefore(Instant publishedBefore) { int before = rows.size(); diff --git a/messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/JdbcOutboxRepository.java b/messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/JdbcOutboxRepository.java deleted file mode 100644 index ced99d0..0000000 --- a/messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/JdbcOutboxRepository.java +++ /dev/null @@ -1,332 +0,0 @@ -package dev.caskeleton.messaging.outbox; - -import dev.caskeleton.messaging.api.ContentType; -import dev.caskeleton.messaging.api.MessageId; -import dev.caskeleton.messaging.api.MessageType; -import dev.caskeleton.messaging.api.SchemaVersion; -import dev.caskeleton.messaging.api.destination.DestinationName; -import dev.caskeleton.messaging.api.error.MessagingConfigurationException; -import dev.caskeleton.messaging.reliability.OutboxRecord; -import dev.caskeleton.messaging.reliability.OutboxRepository; -import dev.caskeleton.messaging.reliability.OutboxStatus; -import java.sql.Connection; -import java.sql.PreparedStatement; -import java.sql.ResultSet; -import java.sql.SQLException; -import java.sql.Timestamp; -import java.time.Duration; -import java.time.Instant; -import java.util.ArrayList; -import java.util.LinkedHashMap; -import java.util.List; -import java.util.Map; -import java.util.Objects; -import java.util.Optional; -import javax.sql.DataSource; - -/** - * The PostgreSQL outbox. - * - *

    {@link #append} deliberately takes no connection of its own: it uses the one the caller is - * already inside, which is the entire mechanism. An outbox row written on a separate connection - * commits independently of the business change and reopens the window the pattern exists to close. - * - *

    Leasing uses {@code SELECT ... FOR UPDATE SKIP LOCKED}. Two relay instances polling the same - * table would otherwise both claim the same rows and publish them twice; {@code SKIP LOCKED} lets - * the second instance step over what the first already holds instead of blocking behind it. - */ -public final class JdbcOutboxRepository implements OutboxRepository { - - private static final String INSERT = - """ - INSERT INTO messaging_outbox ( - message_id, destination, message_type, schema_version, content_type, - payload, headers, created_at, status, attempts) - VALUES (?, ?, ?, ?, ?, ?, ?::jsonb, ?, ?, ?) - """; - - private static final String LEASE = - """ - WITH claimable AS ( - SELECT message_id - FROM messaging_outbox - -- IN_FLIGHT is claimable too, but only once its lease has expired. A relay that dies - -- mid-publish leaves rows in that state; excluding them would strand those messages - -- forever, which is the one failure mode the outbox exists to prevent. - WHERE status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT') - AND (lease_expires_at IS NULL OR lease_expires_at <= ?) - ORDER BY created_at - LIMIT ? - FOR UPDATE SKIP LOCKED - ) - UPDATE messaging_outbox o - SET status = 'IN_FLIGHT', lease_expires_at = ? - FROM claimable c - WHERE o.message_id = c.message_id - RETURNING o.message_id, o.destination, o.message_type, o.schema_version, o.content_type, - o.payload, o.headers, o.created_at, o.status, o.attempts, - o.lease_expires_at, o.last_failure_code - """; - - private final DataSource dataSource; - - /** - * Creates a repository over a data source. - * - * @param dataSource the outbox data source - */ - public JdbcOutboxRepository(DataSource dataSource) { - this.dataSource = Objects.requireNonNull(dataSource, "dataSource must not be null"); - } - - /** - * Appends a record on an existing connection. - * - *

    This overload is the one business code should use, passing the connection its own - * transaction is running on. - * - * @param connection the caller's transactional connection - * @param record the record to write - */ - public void append(Connection connection, OutboxRecord record) { - Objects.requireNonNull(connection, "connection must not be null"); - Objects.requireNonNull(record, "record must not be null"); - try (PreparedStatement statement = connection.prepareStatement(INSERT)) { - statement.setObject(1, record.messageId().value()); - statement.setString(2, record.destination().value()); - statement.setString(3, record.messageType().value()); - statement.setInt(4, record.schemaVersion().value()); - statement.setString(5, record.contentType().value()); - statement.setBytes(6, record.payload()); - statement.setString(7, toJson(record.headers())); - statement.setTimestamp(8, Timestamp.from(record.createdAt())); - statement.setString(9, record.status().name()); - statement.setInt(10, record.attempts()); - statement.executeUpdate(); - } catch (SQLException exception) { - throw new MessagingConfigurationException( - "OUTBOX_APPEND_FAILED", "could not append an outbox record", exception); - } - } - - @Override - public void append(OutboxRecord record) { - withConnection( - connection -> { - append(connection, record); - return null; - }); - } - - @Override - public List leaseBatch(int batchSize, Duration leaseDuration, Instant now) { - Objects.requireNonNull(leaseDuration, "leaseDuration must not be null"); - Objects.requireNonNull(now, "now must not be null"); - return withConnection( - connection -> { - List leased = new ArrayList<>(); - try (PreparedStatement statement = connection.prepareStatement(LEASE)) { - statement.setTimestamp(1, Timestamp.from(now)); - statement.setInt(2, batchSize); - statement.setTimestamp(3, Timestamp.from(now.plus(leaseDuration))); - try (ResultSet results = statement.executeQuery()) { - while (results.next()) { - leased.add(read(results)); - } - } - } - return List.copyOf(leased); - }); - } - - @Override - public void markPublished(MessageId messageId, Instant now) { - update( - "UPDATE messaging_outbox SET status = 'PUBLISHED', published_at = ?, " - + "lease_expires_at = NULL, attempts = attempts + 1 WHERE message_id = ?", - messageId, - now, - null); - } - - @Override - public void markAmbiguous(MessageId messageId, String failureCode, Instant now) { - // The lease is released rather than held: an ambiguous row must become claimable again so the - // relay retries it under the same message id. - update( - "UPDATE messaging_outbox SET status = 'AMBIGUOUS', lease_expires_at = NULL, " - + "last_failure_code = ?, attempts = attempts + 1 WHERE message_id = ?", - messageId, - now, - failureCode); - } - - @Override - public void markFailed(MessageId messageId, String failureCode, Instant now) { - update( - "UPDATE messaging_outbox SET status = 'FAILED', lease_expires_at = NULL, " - + "last_failure_code = ?, attempts = attempts + 1 WHERE message_id = ?", - messageId, - now, - failureCode); - } - - @Override - public void releaseLease(MessageId messageId) { - withConnection( - connection -> { - try (PreparedStatement statement = - connection.prepareStatement( - "UPDATE messaging_outbox SET status = 'PENDING', lease_expires_at = NULL " - + "WHERE message_id = ? AND status = 'IN_FLIGHT'")) { - statement.setObject(1, messageId.value()); - statement.executeUpdate(); - } - return null; - }); - } - - @Override - public Optional find(MessageId messageId) { - return withConnection( - connection -> { - try (PreparedStatement statement = - connection.prepareStatement( - "SELECT message_id, destination, message_type, schema_version, content_type, " - + "payload, headers, created_at, status, attempts, lease_expires_at, " - + "last_failure_code FROM messaging_outbox WHERE message_id = ?")) { - statement.setObject(1, messageId.value()); - try (ResultSet results = statement.executeQuery()) { - return results.next() ? Optional.of(read(results)) : Optional.empty(); - } - } - }); - } - - @Override - public int purgePublishedBefore(Instant publishedBefore) { - Objects.requireNonNull(publishedBefore, "publishedBefore must not be null"); - return withConnection( - connection -> { - try (PreparedStatement statement = - connection.prepareStatement( - "DELETE FROM messaging_outbox WHERE status = 'PUBLISHED' AND published_at < ?")) { - statement.setTimestamp(1, Timestamp.from(publishedBefore)); - return statement.executeUpdate(); - } - }); - } - - private void update(String sql, MessageId messageId, Instant now, String failureCode) { - withConnection( - connection -> { - try (PreparedStatement statement = connection.prepareStatement(sql)) { - if (failureCode == null) { - statement.setTimestamp(1, Timestamp.from(now)); - } else { - statement.setString(1, failureCode); - } - statement.setObject(2, messageId.value()); - statement.executeUpdate(); - } - return null; - }); - } - - private static OutboxRecord read(ResultSet results) throws SQLException { - Timestamp lease = results.getTimestamp("lease_expires_at"); - return new OutboxRecord( - new MessageId(results.getObject("message_id", java.util.UUID.class)), - new DestinationName(results.getString("destination")), - new MessageType(results.getString("message_type")), - new SchemaVersion(results.getInt("schema_version")), - new ContentType(results.getString("content_type")), - results.getBytes("payload"), - fromJson(results.getString("headers")), - results.getTimestamp("created_at").toInstant(), - OutboxStatus.valueOf(results.getString("status")), - results.getInt("attempts"), - Optional.ofNullable(lease).map(Timestamp::toInstant), - Optional.ofNullable(results.getString("last_failure_code"))); - } - - private T withConnection(ConnectionCallback callback) { - try (Connection connection = dataSource.getConnection()) { - return callback.apply(connection); - } catch (SQLException exception) { - throw new MessagingConfigurationException( - "OUTBOX_QUERY_FAILED", "an outbox query failed", exception); - } - } - - /** - * Serialises headers as a flat JSON object. - * - *

    Hand-rolled rather than pulled from a JSON library so this module keeps no codec dependency: - * outbox headers are always flat string pairs, validated by {@code MessageHeaders} before they - * ever reach here. - */ - private static String toJson(Map headers) { - StringBuilder json = new StringBuilder("{"); - boolean first = true; - for (Map.Entry entry : headers.entrySet()) { - if (!first) { - json.append(','); - } - first = false; - json.append('"') - .append(escape(entry.getKey())) - .append("\":\"") - .append(escape(entry.getValue())) - .append('"'); - } - return json.append('}').toString(); - } - - private static Map fromJson(String json) { - Map headers = new LinkedHashMap<>(); - if (json == null || json.length() <= 2) { - return headers; - } - String body = json.substring(1, json.length() - 1); - int index = 0; - while (index < body.length()) { - int keyStart = body.indexOf('"', index); - if (keyStart < 0) { - break; - } - int keyEnd = findClosingQuote(body, keyStart + 1); - int valueStart = body.indexOf('"', body.indexOf(':', keyEnd)); - int valueEnd = findClosingQuote(body, valueStart + 1); - headers.put( - unescape(body.substring(keyStart + 1, keyEnd)), - unescape(body.substring(valueStart + 1, valueEnd))); - index = valueEnd + 1; - } - return headers; - } - - private static int findClosingQuote(String text, int from) { - for (int index = from; index < text.length(); index++) { - if (text.charAt(index) == '"' && text.charAt(index - 1) != '\\') { - return index; - } - } - return text.length(); - } - - private static String escape(String value) { - return value.replace("\\", "\\\\").replace("\"", "\\\""); - } - - private static String unescape(String value) { - return value.replace("\\\"", "\"").replace("\\\\", "\\"); - } - - /** A unit of work against a borrowed connection. */ - @FunctionalInterface - private interface ConnectionCallback { - - T apply(Connection connection) throws SQLException; - } -} diff --git a/messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelayReport.java b/messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelayReport.java deleted file mode 100644 index 3632931..0000000 --- a/messaging/messaging-outbox-jpa/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelayReport.java +++ /dev/null @@ -1,22 +0,0 @@ -package dev.caskeleton.messaging.outbox; - -/** - * What one relay pass did. - * - *

    Ambiguous rows are counted separately from failures because they mean something operationally - * different: a rising ambiguous count is a broker confirmation problem, while a rising failed count - * is a contract or topology problem. - * - * @param leased how many rows were claimed - * @param published how many were confirmed - * @param ambiguous how many had an unknown outcome and stay retryable - * @param failed how many were definitively rejected - */ -public record OutboxRelayReport(int leased, int published, int ambiguous, int failed) { - - public OutboxRelayReport { - if (leased < 0 || published < 0 || ambiguous < 0 || failed < 0) { - throw new IllegalArgumentException("relay counters must not be negative"); - } - } -} diff --git a/messaging/messaging-outbox-jpa/src/test/java/dev/caskeleton/messaging/outbox/OutboxPostgresIT.java b/messaging/messaging-outbox-jpa/src/test/java/dev/caskeleton/messaging/outbox/OutboxPostgresIT.java deleted file mode 100644 index ac2af52..0000000 --- a/messaging/messaging-outbox-jpa/src/test/java/dev/caskeleton/messaging/outbox/OutboxPostgresIT.java +++ /dev/null @@ -1,223 +0,0 @@ -package dev.caskeleton.messaging.outbox; - -import static org.assertj.core.api.Assertions.assertThat; - -import dev.caskeleton.messaging.api.ContentType; -import dev.caskeleton.messaging.api.MessageId; -import dev.caskeleton.messaging.api.MessageType; -import dev.caskeleton.messaging.api.SchemaVersion; -import dev.caskeleton.messaging.api.destination.DestinationName; -import dev.caskeleton.messaging.reliability.OutboxRecord; -import dev.caskeleton.messaging.reliability.OutboxStatus; -import dev.caskeleton.messaging.testkit.DockerAvailability; -import java.io.IOException; -import java.io.InputStream; -import java.nio.charset.StandardCharsets; -import java.sql.Connection; -import java.sql.SQLException; -import java.sql.Statement; -import java.time.Duration; -import java.time.Instant; -import java.util.List; -import java.util.Map; -import java.util.Optional; -import javax.sql.DataSource; -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.condition.EnabledIf; -import org.postgresql.ds.PGSimpleDataSource; -import org.testcontainers.junit.jupiter.Container; -import org.testcontainers.junit.jupiter.Testcontainers; -import org.testcontainers.postgresql.PostgreSQLContainer; - -/** - * Certifies the outbox against a real PostgreSQL. - * - *

    The pattern's guarantees are claims about a database, not about Java: that a row and a - * business change commit together, that two relay instances cannot claim the same row, and that an - * ambiguous publish leaves the row claimable under its original identity. An in-memory double can - * be written to agree with any of those; only a real database settles them. - */ -@Testcontainers -@EnabledIf("dockerAvailable") -class OutboxPostgresIT { - - private static final Instant NOW = Instant.parse("2026-08-10T09:15:00Z"); - - @Container - private static final PostgreSQLContainer POSTGRES = - new PostgreSQLContainer("postgres:16-alpine") - .withDatabaseName("messaging") - .withUsername("messaging") - .withPassword("messaging"); - - private DataSource dataSource; - private JdbcOutboxRepository repository; - - static boolean dockerAvailable() { - return DockerAvailability.isAvailable(); - } - - @BeforeEach - void migrate() throws SQLException, IOException { - dataSource = dataSource(); - try (Connection connection = dataSource.getConnection(); - Statement statement = connection.createStatement()) { - statement.execute("DROP TABLE IF EXISTS messaging_outbox"); - statement.execute(migration()); - } - repository = new JdbcOutboxRepository(dataSource); - } - - @Test - void anAppendedRowIsClaimableAndRoundTripsIntact() { - OutboxRecord record = record(); - repository.append(record); - - List leased = repository.leaseBatch(10, Duration.ofSeconds(30), NOW); - - assertThat(leased) - .singleElement() - .satisfies( - leasedRecord -> { - assertThat(leasedRecord.messageId()).isEqualTo(record.messageId()); - assertThat(leasedRecord.messageType()).isEqualTo(record.messageType()); - assertThat(leasedRecord.payload()).isEqualTo(record.payload()); - assertThat(leasedRecord.headers()).containsEntry("msg.partition-key", "acct-1"); - }); - } - - @Test - void aLeasedRowIsInvisibleToASecondRelayInstance() { - repository.append(record()); - - List first = repository.leaseBatch(10, Duration.ofSeconds(30), NOW); - List second = repository.leaseBatch(10, Duration.ofSeconds(30), NOW); - - assertThat(first).hasSize(1); - assertThat(second).as("SKIP LOCKED plus the lease keeps two relays off the same row").isEmpty(); - } - - @Test - void anExpiredLeaseBecomesClaimableAgain() { - repository.append(record()); - repository.leaseBatch(10, Duration.ofSeconds(30), NOW); - - List reclaimed = - repository.leaseBatch(10, Duration.ofSeconds(30), NOW.plusSeconds(31)); - - assertThat(reclaimed).hasSize(1); - } - - @Test - void anAmbiguousRowStaysClaimableUnderItsOriginalIdentity() { - OutboxRecord record = record(); - repository.append(record); - repository.leaseBatch(10, Duration.ofSeconds(30), NOW); - - repository.markAmbiguous(record.messageId(), "CONFIRM_TIMEOUT", NOW); - - assertThat(repository.find(record.messageId()).orElseThrow().status()) - .isEqualTo(OutboxStatus.AMBIGUOUS); - assertThat(repository.leaseBatch(10, Duration.ofSeconds(30), NOW)) - .singleElement() - .satisfies(retry -> assertThat(retry.messageId()).isEqualTo(record.messageId())); - } - - @Test - void aPublishedRowIsNotClaimedAgain() { - OutboxRecord record = record(); - repository.append(record); - repository.leaseBatch(10, Duration.ofSeconds(30), NOW); - - repository.markPublished(record.messageId(), NOW); - - assertThat(repository.find(record.messageId()).orElseThrow().status()) - .isEqualTo(OutboxStatus.PUBLISHED); - assertThat(repository.leaseBatch(10, Duration.ofSeconds(30), NOW.plusSeconds(60))).isEmpty(); - } - - @Test - void aFailedRowIsNotClaimedAgainWithoutIntervention() { - OutboxRecord record = record(); - repository.append(record); - repository.leaseBatch(10, Duration.ofSeconds(30), NOW); - - repository.markFailed(record.messageId(), "INVALID_TOPIC", NOW); - - assertThat(repository.leaseBatch(10, Duration.ofSeconds(30), NOW.plusSeconds(60))).isEmpty(); - } - - @Test - void theRowAndTheBusinessChangeCommitTogetherOrNotAtAll() throws SQLException { - OutboxRecord record = record(); - - try (Connection connection = dataSource.getConnection()) { - connection.setAutoCommit(false); - try (Statement statement = connection.createStatement()) { - statement.execute("CREATE TABLE IF NOT EXISTS orders (id TEXT PRIMARY KEY)"); - } - connection.commit(); - - try (Statement statement = connection.createStatement()) { - statement.execute("INSERT INTO orders (id) VALUES ('o-1')"); - } - repository.append(connection, record); - connection.rollback(); - } - - assertThat(repository.find(record.messageId())) - .as("a rolled back business change must not leave a publishable row") - .isEmpty(); - } - - @Test - void retentionRemovesOnlyPublishedRows() { - OutboxRecord published = record(); - OutboxRecord pending = record(); - repository.append(published); - repository.append(pending); - repository.markPublished(published.messageId(), NOW); - - int removed = repository.purgePublishedBefore(NOW.plusSeconds(1)); - - assertThat(removed).isEqualTo(1); - assertThat(repository.find(pending.messageId())).isPresent(); - } - - private static OutboxRecord record() { - return new OutboxRecord( - MessageId.newId(), - new DestinationName("order-events"), - new MessageType("order.created"), - new SchemaVersion(1), - ContentType.JSON, - "{\"orderId\":\"o-1\"}".getBytes(StandardCharsets.UTF_8), - Map.of("msg.partition-key", "acct-1"), - NOW.minusSeconds(60), - OutboxStatus.PENDING, - 0, - Optional.empty(), - Optional.empty()); - } - - private static DataSource dataSource() { - PGSimpleDataSource source = new PGSimpleDataSource(); - source.setUrl(POSTGRES.getJdbcUrl()); - source.setUser(POSTGRES.getUsername()); - source.setPassword(POSTGRES.getPassword()); - return source; - } - - private static String migration() throws IOException { - try (InputStream stream = - OutboxPostgresIT.class - .getClassLoader() - .getResourceAsStream("db/migration/messaging/V1__messaging_outbox.sql")) { - if (stream == null) { - throw new IOException("the outbox migration is missing from the classpath"); - } - return new String(stream.readAllBytes(), StandardCharsets.UTF_8); - } - } -} diff --git a/messaging/messaging-policy/src/main/java/dev/caskeleton/messaging/policy/DestinationProfileValidator.java b/messaging/messaging-policy/src/main/java/dev/caskeleton/messaging/policy/DestinationProfileValidator.java index 23f2f6d..bcd4728 100644 --- a/messaging/messaging-policy/src/main/java/dev/caskeleton/messaging/policy/DestinationProfileValidator.java +++ b/messaging/messaging-policy/src/main/java/dev/caskeleton/messaging/policy/DestinationProfileValidator.java @@ -6,8 +6,8 @@ import dev.caskeleton.messaging.api.destination.ConfirmationRequirement; import dev.caskeleton.messaging.api.destination.DestinationName; import java.util.ArrayList; import java.util.Collection; -import java.util.HashSet; import java.util.LinkedHashMap; +import java.util.LinkedHashSet; import java.util.List; import java.util.Map; import java.util.Objects; @@ -127,9 +127,14 @@ public final class DestinationProfileValidator { } } + // One graph carrying both edge kinds, not two walks. + // + // Walking retry and dead-letter separately misses a cycle that alternates between them: A's + // retry points at B and B's dead letter points back at A. Neither single-edge walk revisits a + // node, both pass, and a poison message loops between the two destinations forever. The label + // is kept per edge so the reported path still says which kind each hop was. for (DestinationProfile profile : byName.values()) { - requireCycleFree(profile, byName, Edge.RETRY); - requireCycleFree(profile, byName, Edge.DEAD_LETTER); + requireCycleFree(profile, byName); } } @@ -152,22 +157,35 @@ public final class DestinationProfileValidator { } private void requireCycleFree( - DestinationProfile start, Map byName, Edge edge) { + DestinationProfile start, Map byName) { + walk(start, byName, new LinkedHashSet<>(), new ArrayList<>()); + } - Set visited = new HashSet<>(); - List path = new ArrayList<>(); - DestinationProfile current = start; + /** + * Depth-first over both edge kinds at once. + * + *

    {@code onPath} is the current walk rather than everything ever seen, so a diamond — two + * destinations that both forward to a third — is not mistaken for a loop. + */ + private void walk( + DestinationProfile current, + Map byName, + Set onPath, + List path) { - while (true) { - if (!visited.add(current.name())) { - throw new IllegalArgumentException( - edge.label + " graph contains a cycle: " + String.join(" -> ", path)); - } - path.add(current.name().value()); + if (!onPath.add(current.name())) { + throw new IllegalArgumentException( + "the retry and dead letter graph contains a cycle: " + + String.join(" -> ", path) + + " -> " + + current.name().value()); + } + path.add(current.name().value()); + for (Edge edge : Edge.values()) { Optional next = edge.from(current); if (next.isEmpty()) { - return; + continue; } DestinationName nextName = next.get(); DestinationProfile nextProfile = byName.get(nextName); @@ -175,7 +193,9 @@ public final class DestinationProfileValidator { throw new IllegalArgumentException( edge.label + " destination is not registered: " + nextName.value()); } - current = nextProfile; + List branch = new ArrayList<>(path); + branch.set(branch.size() - 1, current.name().value() + " (" + edge.label + ')'); + walk(nextProfile, byName, new LinkedHashSet<>(onPath), branch); } } } diff --git a/messaging/messaging-policy/src/test/java/dev/caskeleton/messaging/policy/DestinationProfileValidatorTest.java b/messaging/messaging-policy/src/test/java/dev/caskeleton/messaging/policy/DestinationProfileValidatorTest.java index 51dadae..5bc613b 100644 --- a/messaging/messaging-policy/src/test/java/dev/caskeleton/messaging/policy/DestinationProfileValidatorTest.java +++ b/messaging/messaging-policy/src/test/java/dev/caskeleton/messaging/policy/DestinationProfileValidatorTest.java @@ -16,6 +16,7 @@ import java.time.Duration; import java.util.List; import java.util.Optional; import java.util.Set; +import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; class DestinationProfileValidatorTest { @@ -114,6 +115,27 @@ class DestinationProfileValidatorTest { .hasMessageContaining("cycle"); } + @Test + @DisplayName("a cycle that alternates retry and dead-letter edges is rejected") + void aMixedEdgeCycleIsRejected() { + List cycle = DestinationProfileFixtures.mixedEdgeCycle(); + + assertThatThrownBy(() -> validator.validateAll(cycle)) + .as("walking the two edge kinds separately misses this and the message loops forever") + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("cycle"); + } + + @Test + @DisplayName("two destinations sharing a dead letter is not a cycle") + void aSharedDeadLetterIsNotACycle() { + List diamond = DestinationProfileFixtures.sharedDeadLetter(); + + org.assertj.core.api.Assertions.assertThatCode(() -> validator.validateAll(diamond)) + .as("a diamond is not a loop; rejecting it would forbid a single shared parking queue") + .doesNotThrowAnyException(); + } + @Test void anUnregisteredRetryDestinationIsRejected() { List dangling = DestinationProfileFixtures.danglingRetryDestination(); @@ -414,6 +436,30 @@ final class DestinationProfileFixtures { named("beta-dlq", RetryPolicy.none(), DeadLetterPolicy.disabled())); } + /** + * A loop that alternates edge kinds: alpha's retry to beta, beta's dead letter back to alpha. + * + *

    Two separate walks — one over retry edges, one over dead-letter edges — each visit alpha + * once and terminate, so both report a clean graph while a poison message circles between the two + * destinations forever. + */ + static List mixedEdgeCycle() { + // alpha retries into beta and parks in its own DLQ; beta dead-letters back into alpha. Each + // single-edge walk visits alpha once and stops, so both report a clean graph. + return List.of( + named("alpha", retryTo("beta"), DeadLetterPolicy.to(new DestinationName("alpha-dlq"))), + named("beta", RetryPolicy.none(), DeadLetterPolicy.to(new DestinationName("alpha"))), + named("alpha-dlq", RetryPolicy.none(), DeadLetterPolicy.disabled())); + } + + /** Two destinations dead-lettering into the same place: a diamond, not a cycle. */ + static List sharedDeadLetter() { + return List.of( + named("alpha", RetryPolicy.none(), DeadLetterPolicy.to(new DestinationName("shared-dlq"))), + named("beta", RetryPolicy.none(), DeadLetterPolicy.to(new DestinationName("shared-dlq"))), + named("shared-dlq", RetryPolicy.none(), DeadLetterPolicy.disabled())); + } + static List danglingRetryDestination() { return List.of( named("alpha", retryTo("missing"), DeadLetterPolicy.to(new DestinationName("alpha-dlq"))), diff --git a/messaging/messaging-pulsar-experimental/gradle.lockfile b/messaging/messaging-pulsar-experimental/gradle.lockfile index 3df51a0..460caa8 100644 --- a/messaging/messaging-pulsar-experimental/gradle.lockfile +++ b/messaging/messaging-pulsar-experimental/gradle.lockfile @@ -33,9 +33,9 @@ commons-io:commons-io:2.21.0=spotbugs info.picocli:picocli:4.7.7=checkstyle io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnotationProcessor io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor -io.micrometer:micrometer-commons:1.16.0=runtimeClasspath,testRuntimeClasspath -io.micrometer:micrometer-core:1.16.0=runtimeClasspath,testRuntimeClasspath -io.micrometer:micrometer-observation:1.16.0=runtimeClasspath,testRuntimeClasspath +io.micrometer:micrometer-commons:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.micrometer:micrometer-core:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.micrometer:micrometer-observation:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath io.opentelemetry:opentelemetry-api-incubator:1.45.0-alpha=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath io.opentelemetry:opentelemetry-api:1.55.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath io.opentelemetry:opentelemetry-common:1.55.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath @@ -76,7 +76,7 @@ org.codehaus.plexus:plexus-utils:3.3.0=checkstyle org.dom4j:dom4j:2.2.0=spotbugs org.hdrhistogram:HdrHistogram:2.2.2=runtimeClasspath,testRuntimeClasspath org.javassist:javassist:3.28.0-GA=checkstyle -org.jspecify:jspecify:1.0.0=annotationProcessor,checkstyle,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath +org.jspecify:jspecify:1.0.0=annotationProcessor,checkstyle,compileClasspath,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath org.junit.jupiter:junit-jupiter-api:6.0.1=testCompileClasspath,testRuntimeClasspath org.junit.jupiter:junit-jupiter-engine:6.0.1=testRuntimeClasspath org.junit.jupiter:junit-jupiter-params:6.0.1=testCompileClasspath,testRuntimeClasspath diff --git a/messaging/messaging-pulsar-experimental/src/main/java/dev/caskeleton/messaging/pulsar/PulsarMessagingTransport.java b/messaging/messaging-pulsar-experimental/src/main/java/dev/caskeleton/messaging/pulsar/PulsarMessagingTransport.java index 78026f7..1884d57 100644 --- a/messaging/messaging-pulsar-experimental/src/main/java/dev/caskeleton/messaging/pulsar/PulsarMessagingTransport.java +++ b/messaging/messaging-pulsar-experimental/src/main/java/dev/caskeleton/messaging/pulsar/PulsarMessagingTransport.java @@ -28,6 +28,12 @@ import java.util.function.Function; /** * The Experimental Pulsar adapter. * + *

    This is an Experimental contract seam, not a Stable adapter. It exercises the transport SPI + * against a send operation the application supplies; it does not ship a Pulsar client bridge, + * producer lifecycle, or reconnection. Only a failure the operation reports as a {@link + * PulsarPreSendRejection} is treated as definitely not transmitted — every other failure, + * recognised or not, is ambiguous. + * *

    Disabled unless explicitly enabled. Experimental means the contract suite has been run but the * live-broker certification has not, so the adapter must never become load-bearing because a * default switched it on. @@ -116,8 +122,18 @@ public final class PulsarMessagingTransport implements MessagingTransport { "PAYLOAD_TOO_LARGE", "encoded payload is " + size + " bytes, limit is " + limit)); } + // The caller's own deadline, honoured rather than ignored: a stalled send used to hold the + // caller for whatever the SDK's default was, whatever the caller had asked for. + long startedAt = System.nanoTime(); return send.send(profile.topic(), request) - .handle((position, failure) -> failure == null ? confirmed(position) : classify(failure)) + .toCompletableFuture() + .orTimeout( + request.options().timeout().toMillis(), java.util.concurrent.TimeUnit.MILLISECONDS) + .handle( + (position, failure) -> + failure == null + ? confirmed(position, elapsedSince(startedAt)) + : classify(failure, elapsedSince(startedAt))) .thenApply(TransportPublishResult::new); } @@ -164,7 +180,11 @@ public final class PulsarMessagingTransport implements MessagingTransport { return !closed.get(); } - private static PublishResult confirmed(PulsarMessagePosition position) { + private static Duration elapsedSince(long startedAtNanos) { + return Duration.ofNanos(System.nanoTime() - startedAtNanos); + } + + private static PublishResult confirmed(PulsarMessagePosition position, Duration elapsed) { return new PublishResult( PublishCompletion.CONFIRMED, // The send future resolves only after the entry is persisted to the configured bookies, so @@ -177,44 +197,50 @@ public final class PulsarMessagingTransport implements MessagingTransport { RoutingOutcome.ROUTED, Optional.of(position), 1, - Duration.ZERO, + elapsed, Optional.empty()); } - private static PublishResult classify(Throwable failure) { + /** + * Decides what a send failure means, from typed signals only. + * + *

    Classification used to read the exception's class simple name: {@code "Timeout"} meant + * ambiguous, anything else meant rejected. A class name is not part of Pulsar's contract — it + * changes between client versions — and defaulting the unknown case to {@code REJECTED} tells the + * caller nothing was transmitted, which is how the same entry is published to the bookies twice. + */ + private static PublishResult classify(Throwable failure, Duration elapsed) { Throwable cause = failure instanceof java.util.concurrent.CompletionException ? failure.getCause() : failure; - String name = cause == null ? "" : cause.getClass().getSimpleName(); - - // A timeout is the ambiguous case: the entry may already be on the bookies with only the - // acknowledgement lost. Reporting it as rejected is how the same message is published twice. - if (name.contains("Timeout")) { + if (cause instanceof PulsarPreSendRejection rejection) { return new PublishResult( - PublishCompletion.AMBIGUOUS, - new PublishEvidence( - true, TransmissionEvidence.MAY_HAVE_BEEN_TRANSMITTED, false, ConfirmationLevel.NONE), - RoutingOutcome.UNKNOWN, + PublishCompletion.REJECTED, + PublishEvidence.notTransmitted(), + RoutingOutcome.NOT_APPLICABLE, Optional.empty(), 1, - Duration.ZERO, + elapsed, Optional.of( FailureDescriptor.of( - FailureCategory.TRANSIENT_INFRASTRUCTURE, - "PULSAR_SEND_TIMEOUT", - "the send timed out; the broker may already hold the message"))); + FailureCategory.CONFIGURATION, rejection.code(), rejection.getMessage()))); } + boolean timedOut = cause instanceof java.util.concurrent.TimeoutException; return new PublishResult( - PublishCompletion.REJECTED, - PublishEvidence.notTransmitted(), - RoutingOutcome.NOT_APPLICABLE, + PublishCompletion.AMBIGUOUS, + new PublishEvidence( + true, TransmissionEvidence.MAY_HAVE_BEEN_TRANSMITTED, false, ConfirmationLevel.NONE), + RoutingOutcome.UNKNOWN, Optional.empty(), 1, - Duration.ZERO, + elapsed, Optional.of( FailureDescriptor.of( - FailureCategory.PERMANENT_BUSINESS, - "PULSAR_SEND_REJECTED", - "the broker rejected the send: " + name))); + FailureCategory.TRANSIENT_INFRASTRUCTURE, + timedOut ? "PULSAR_SEND_TIMEOUT" : "PULSAR_SEND_OUTCOME_UNKNOWN", + timedOut + ? "the send timed out; the broker may already hold the message" + : "the send failed in a way this adapter does not classify; the broker may" + + " already hold the message"))); } private static TransportPublishResult rejectedLocally(String code, String message) { diff --git a/messaging/messaging-pulsar-experimental/src/main/java/dev/caskeleton/messaging/pulsar/PulsarPreSendRejection.java b/messaging/messaging-pulsar-experimental/src/main/java/dev/caskeleton/messaging/pulsar/PulsarPreSendRejection.java new file mode 100644 index 0000000..6cf5f62 --- /dev/null +++ b/messaging/messaging-pulsar-experimental/src/main/java/dev/caskeleton/messaging/pulsar/PulsarPreSendRejection.java @@ -0,0 +1,65 @@ +package dev.caskeleton.messaging.pulsar; + +import java.util.Objects; + +/** + * The one signal that lets this adapter report {@code REJECTED / NOT_TRANSMITTED}. + * + *

    Classification used to work by reading the exception's class simple name and matching + * substrings — {@code "Timeout"} meant ambiguous, anything else meant rejected. Two things are + * wrong with that. A class name is not part of any SDK's contract: it changes between versions, and + * an unrelated exception whose name happens to contain the substring is misread. And defaulting the + * unknown case to {@code REJECTED} tells the caller nothing was transmitted, so a caller that + * republishes under a new identity duplicates a message the server may well have stored. + * + *

    The send operation throws this only when it knows the send never left the client: an argument + * the SDK refused, a producer that was already closed, a topic the client could not resolve. + * Everything else — including anything the adapter does not recognise — is ambiguous, and ambiguity + * is the honest answer when the write may have landed. + */ +public final class PulsarPreSendRejection extends RuntimeException { + + private static final long serialVersionUID = 1L; + + private final String code; + + /** + * Creates a pre-send rejection. + * + * @param code the sanitised platform code for the refusal + * @param message a sanitised description + */ + public PulsarPreSendRejection(String code, String message) { + super(message); + this.code = requireCode(code); + } + + /** + * Creates a pre-send rejection wrapping the SDK's own exception. + * + * @param code the sanitised platform code for the refusal + * @param message a sanitised description + * @param cause the SDK exception that proves the send never started + */ + public PulsarPreSendRejection(String code, String message, Throwable cause) { + super(message, cause); + this.code = requireCode(code); + } + + private static String requireCode(String code) { + Objects.requireNonNull(code, "code must not be null"); + if (code.isBlank()) { + throw new IllegalArgumentException("code must not be blank"); + } + return code; + } + + /** + * The sanitised code reported to the caller. + * + * @return the code + */ + public String code() { + return code; + } +} diff --git a/messaging/messaging-pulsar-experimental/src/main/java/dev/caskeleton/messaging/pulsar/PulsarProfileValidator.java b/messaging/messaging-pulsar-experimental/src/main/java/dev/caskeleton/messaging/pulsar/PulsarProfileValidator.java index 16380e0..b0e0db3 100644 --- a/messaging/messaging-pulsar-experimental/src/main/java/dev/caskeleton/messaging/pulsar/PulsarProfileValidator.java +++ b/messaging/messaging-pulsar-experimental/src/main/java/dev/caskeleton/messaging/pulsar/PulsarProfileValidator.java @@ -35,7 +35,7 @@ public final class PulsarProfileValidator { throw new MessagingCapabilityUnavailableException( "PULSAR_DISABLED", "the Pulsar adapter is experimental and disabled unless " - + "backend.messaging.experimental.pulsar=true"); + + "app.messaging.experimental.pulsar=true"); } if (profile.orderingScope() == OrderingScope.KEY && subscriptionType != PulsarSubscriptionType.KEY_SHARED) { diff --git a/messaging/messaging-pulsar-experimental/src/test/java/dev/caskeleton/messaging/pulsar/PulsarAdapterContractTest.java b/messaging/messaging-pulsar-experimental/src/test/java/dev/caskeleton/messaging/pulsar/PulsarAdapterContractTest.java index 3968a6e..0d3c130 100644 --- a/messaging/messaging-pulsar-experimental/src/test/java/dev/caskeleton/messaging/pulsar/PulsarAdapterContractTest.java +++ b/messaging/messaging-pulsar-experimental/src/test/java/dev/caskeleton/messaging/pulsar/PulsarAdapterContractTest.java @@ -104,12 +104,59 @@ class PulsarAdapterContractTest { } @Test - void anExplicitBrokerRejectionIsNotAmbiguous() { + void onlyATypedPreSendRejectionIsReportedAsNotTransmitted() { PublishResult result = - await(failingWith(new IllegalStateException("producer is fenced")).publish(request(64))); + await( + failingWith( + new PulsarPreSendRejection( + "PULSAR_PRODUCER_FENCED", "the producer was fenced before the send")) + .publish(request(64))); assertThat(result.completion()).isEqualTo(PublishCompletion.REJECTED); assertThat(result.evidence().transmission()).isEqualTo(TransmissionEvidence.NOT_TRANSMITTED); + assertThat(result.failure()) + .hasValueSatisfying( + failure -> assertThat(failure.code()).isEqualTo("PULSAR_PRODUCER_FENCED")); + } + + @Test + void anUnrecognisedFailureIsAmbiguousRatherThanRejected() { + // Any exception the adapter cannot type is a send whose outcome it does not know. Reporting + // REJECTED invites the caller to republish an entry the bookies may already hold. + PublishResult result = + await(failingWith(new IllegalStateException("producer is fenced")).publish(request(64))); + + assertThat(result.completion()).isEqualTo(PublishCompletion.AMBIGUOUS); + assertThat(result.evidence().transmission()) + .isEqualTo(TransmissionEvidence.MAY_HAVE_BEEN_TRANSMITTED); + assertThat(result.failure()) + .hasValueSatisfying( + failure -> assertThat(failure.code()).isEqualTo("PULSAR_SEND_OUTCOME_UNKNOWN")); + } + + @Test + void aWrappedUnknownFailureIsAlsoAmbiguous() { + PublishResult result = + await( + failingWith( + new java.util.concurrent.CompletionException( + new RuntimeException("something the adapter has never seen"))) + .publish(request(64))); + + assertThat(result.completion()).isEqualTo(PublishCompletion.AMBIGUOUS); + } + + @Test + void aSendThatNeverCompletesIsBoundedByTheCallersTimeout() { + PulsarMessagingTransport stalled = + transport((topic, request) -> new CompletableFuture<>(), PulsarSubscriptionMode.shared()); + + PublishResult result = + await(stalled.publish(requestWithTimeout(java.time.Duration.ofMillis(50)))); + + assertThat(result.completion()).isEqualTo(PublishCompletion.AMBIGUOUS); + assertThat(result.failure()) + .hasValueSatisfying(failure -> assertThat(failure.code()).isEqualTo("PULSAR_SEND_TIMEOUT")); } @Test @@ -182,6 +229,15 @@ class PulsarAdapterContractTest { .hasMessageContaining("not advertised"); } + private static TransportPublishRequest requestWithTimeout(java.time.Duration timeout) { + PublishOptions defaults = PublishOptions.defaults(); + return new TransportPublishRequest( + profile(), + envelope(64), + new PublishOptions( + timeout, defaults.confirmation(), defaults.deduplication(), defaults.brokerHints())); + } + private static TransportPublishRequest request(int payloadBytes) { return new TransportPublishRequest( profile(), envelope(payloadBytes), PublishOptions.defaults()); diff --git a/messaging/messaging-rabbit/build.gradle b/messaging/messaging-rabbit/build.gradle index 21685a3..dd92d4d 100644 --- a/messaging/messaging-rabbit/build.gradle +++ b/messaging/messaging-rabbit/build.gradle @@ -9,7 +9,9 @@ dependencies { api project(':messaging:messaging-security') api project(':messaging:messaging-admin-api') - implementation 'org.springframework.amqp:spring-rabbit' + // api: RabbitChannelPublisher and the mappers expose Spring AMQP's Message on public + // signatures, so a consumer implementing the seam needs the type on its compile classpath. + api 'org.springframework.amqp:spring-rabbit' implementation 'org.springframework:spring-context' // Test-only. The same adapter contract Kafka runs, so a guarantee can only be weakened by diff --git a/messaging/messaging-rabbit/gradle.lockfile b/messaging/messaging-rabbit/gradle.lockfile index 95c6b3d..010d9a2 100644 --- a/messaging/messaging-rabbit/gradle.lockfile +++ b/messaging/messaging-rabbit/gradle.lockfile @@ -41,7 +41,7 @@ info.picocli:picocli:4.7.7=checkstyle io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,jmhAnnotationProcessor,testAnnotationProcessor io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,jmhAnnotationProcessor,testAnnotationProcessor io.micrometer:micrometer-commons:1.16.0=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -io.micrometer:micrometer-core:1.16.0=jmhRuntimeClasspath,runtimeClasspath,testRuntimeClasspath +io.micrometer:micrometer-core:1.16.0=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath io.micrometer:micrometer-observation:1.16.0=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath io.netty:netty-buffer:4.2.17.Final=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath io.netty:netty-codec-base:4.2.17.Final=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath diff --git a/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitChannelPublisher.java b/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitChannelPublisher.java index dd7fead..8fe08f2 100644 --- a/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitChannelPublisher.java +++ b/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitChannelPublisher.java @@ -30,4 +30,13 @@ public interface RabbitChannelPublisher { */ void publish( long sequence, String exchange, String routingKey, Message message, boolean mandatory); + + /** + * Closes the underlying channel. + * + *

    A default no-op so an existing publisher keeps compiling, but the transport calls it: a + * transport whose {@code close} only flipped a boolean left the native channel open and every + * pending stage waiting for a confirm that would never arrive. + */ + default void close() {} } diff --git a/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitConfirmCoordinator.java b/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitConfirmCoordinator.java index 80585f8..f65c985 100644 --- a/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitConfirmCoordinator.java +++ b/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitConfirmCoordinator.java @@ -9,12 +9,10 @@ import dev.caskeleton.messaging.api.publish.PublishEvidence; import dev.caskeleton.messaging.api.publish.PublishResult; import dev.caskeleton.messaging.api.publish.RoutingOutcome; import java.time.Duration; -import java.util.Map; import java.util.Objects; import java.util.Optional; import java.util.concurrent.CompletableFuture; import java.util.concurrent.CompletionStage; -import java.util.concurrent.ConcurrentHashMap; /** * Correlates a RabbitMQ publish with its confirm and its possible return. @@ -33,7 +31,16 @@ import java.util.concurrent.ConcurrentHashMap; */ public final class RabbitConfirmCoordinator { - private final Map pending = new ConcurrentHashMap<>(); + /** + * Pending publishes, ordered by sequence. + * + *

    Ordered because a Rabbit confirm carries a {@code multiple} flag meaning "everything up to + * and including this tag". Resolving one sequence per confirm — which is what a hash map forces — + * leaves every earlier publish pending forever: the caller's stage never completes and the entry + * is never removed, so the map grows for the life of the connection. + */ + private final java.util.concurrent.ConcurrentSkipListMap pending = + new java.util.concurrent.ConcurrentSkipListMap<>(); /** * Registers a publish awaiting its confirm. @@ -78,11 +85,70 @@ public final class RabbitConfirmCoordinator { * @param elapsed how long the publish took */ public void confirmed(long sequence, boolean acknowledged, Duration elapsed) { - PendingPublish publish = pending.remove(sequence); - if (publish == null) { + confirmed(sequence, false, acknowledged, elapsed); + } + + /** + * Records a broker confirm, honouring the {@code multiple} flag. + * + * @param multiple true when the broker is confirming every sequence up to and including this one + */ + public void confirmed(long sequence, boolean multiple, boolean acknowledged, Duration elapsed) { + if (!multiple) { + PendingPublish publish = pending.remove(sequence); + if (publish != null) { + publish.completion.complete(publish.resolve(sequence, acknowledged, elapsed)); + } return; } - publish.completion.complete(publish.resolve(sequence, acknowledged, elapsed)); + // headMap(tag, inclusive) is the whole point of the ordered map: the broker is confirming a + // range, and every entry in it must be resolved with the same verdict. + java.util.NavigableMap confirmed = pending.headMap(sequence, true); + for (Long pendingSequence : java.util.List.copyOf(confirmed.keySet())) { + PendingPublish publish = pending.remove(pendingSequence); + if (publish != null) { + publish.completion.complete(publish.resolve(pendingSequence, acknowledged, elapsed)); + } + } + } + + /** + * Completes a publish whose send failed synchronously. + * + *

    The classifier's verdict is used as-is: it is the only thing that knows whether the frame + * can have reached the broker, and guessing here would either invent a rejection for a message + * that was stored or an ambiguity for one that never left. + * + * @param sequence the publish sequence number + * @param result the classified outcome + */ + public void sendFailed(long sequence, PublishResult result) { + PendingPublish publish = pending.remove(sequence); + if (publish != null) { + publish.completion.complete(result); + } + } + + /** + * Completes every pending publish because the channel is gone. + * + *

    Ambiguous, not rejected: the frames were written, and a channel that closes afterwards says + * nothing about whether the broker stored them. Leaving them pending — which is what close used + * to do — hangs every caller holding a stage. + * + * @param elapsed how long the publishes had been waiting + * @return how many stages were completed + */ + public int channelClosed(Duration elapsed) { + int drained = 0; + for (Long sequence : java.util.List.copyOf(pending.keySet())) { + PendingPublish publish = pending.remove(sequence); + if (publish != null) { + publish.completion.complete(publish.resolveChannelClosed(sequence, elapsed)); + drained++; + } + } + return drained; } /** @@ -137,13 +203,36 @@ public final class RabbitConfirmCoordinator { this.requirement = requirement; } + /** The channel closed with this publish still awaiting its confirm. */ + private PublishResult resolveChannelClosed(long sequence, Duration elapsed) { + return new PublishResult( + PublishCompletion.AMBIGUOUS, + PublishEvidence.ambiguous(), + RoutingOutcome.UNKNOWN, + Optional.of(new RabbitPublishReference(exchange, routingKey, sequence)), + 1, + elapsed, + Optional.of( + FailureDescriptor.of( + FailureCategory.AMBIGUOUS, + "RABBIT_CHANNEL_CLOSED", + "the channel closed before the broker confirmed this publish"))); + } + private PublishResult resolve(long sequence, boolean acknowledged, Duration elapsed) { RabbitPublishReference position = new RabbitPublishReference(exchange, routingKey, sequence); if (!acknowledged) { return new PublishResult( PublishCompletion.REJECTED, - PublishEvidence.notTransmitted(), + // Transmitted, not "not transmitted": a NACK is the broker's answer to a frame it + // received. Recording it as never sent contradicts the very evidence that produced it, + // and a caller reading the evidence would conclude the message can be re-sent freely. + new PublishEvidence( + true, + dev.caskeleton.messaging.api.publish.TransmissionEvidence.TRANSMITTED, + false, + ConfirmationLevel.NONE), RoutingOutcome.UNKNOWN, Optional.of(position), 1, diff --git a/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitConsumerRegistrar.java b/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitConsumerRegistrar.java index 1095928..1852966 100644 --- a/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitConsumerRegistrar.java +++ b/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitConsumerRegistrar.java @@ -92,6 +92,11 @@ public final class RabbitConsumerRegistrar implements TransportConsumerRegistrat String queue = Optional.ofNullable(message.getMessageProperties().getConsumerQueue()).orElse(""); + if (!active.get()) { + // close() only flipped a flag and onMessage never read it, so a delivery arriving after + // close was accepted, dispatched, and settled against a controller that was shutting down. + return false; + } if (pausedScopes.contains(queue) || pausedScopes.contains("")) { return false; } @@ -101,31 +106,62 @@ public final class RabbitConsumerRegistrar implements TransportConsumerRegistrat long tag = message.getMessageProperties().getDeliveryTag(); try { - TransportDelivery delivery = - new TransportDelivery( - deliveryMapper.toEnvelope(message), - deliveryMapper.toMetadata(spec.profile().name(), message, now), - new ControllerBackedSettlement(tag)); - spec.sink().apply(delivery).toCompletableFuture().join(); - return true; - } catch (RuntimeException exception) { - // The message could not even be decoded. Park it rather than requeue: it will fail the same - // way on every redelivery, and requeueing turns that into a hot loop. - operations - .discard( - tag, - FailureDescriptor.of( - FailureCategory.DESERIALIZATION, - "RABBIT_UNDECODABLE", - "the message could not be decoded into an envelope")) - .toCompletableFuture() - .join(); + TransportDelivery delivery; + try { + // The narrow catch. One catch around decode, the handler and the settlement meant a + // business failure or an ACK that could not be written was recorded as an undecodable + // payload and discarded — a message that should have been retried, deleted instead. + delivery = + new TransportDelivery( + deliveryMapper.toEnvelope(message), + deliveryMapper.toMetadata(spec.profile().name(), message, now), + new ControllerBackedSettlement(tag)); + } catch (RuntimeException undecodable) { + // Genuinely undecodable: it will fail the same way on every redelivery, so requeueing is a + // hot loop. Discarding is right here and only here. + operations + .discard( + tag, + FailureDescriptor.of( + FailureCategory.DESERIALIZATION, + "RABBIT_UNDECODABLE", + "the message could not be decoded into an envelope")) + .toCompletableFuture() + .join(); + return true; + } + + ControllerBackedSettlement settlement = (ControllerBackedSettlement) delivery.settlement(); + try { + spec.sink().apply(delivery).toCompletableFuture().join(); + } catch (RuntimeException handlerFailed) { + // A handler failure is a retry, not a discard. + requeueQuietly(settlement); + return true; + } + if (!settlement.wasSettled()) { + // A handler that completed without settling has not said what to do with the message. + // Acknowledging on its behalf would silently drop it; the contract violation is surfaced + // as a requeue so the message survives while somebody fixes the handler. + requeueQuietly(settlement); + } return true; } finally { shutdown.endWork(); } } + /** Requeues without letting a settlement failure become the caller's problem. */ + private void requeueQuietly(ControllerBackedSettlement settlement) { + try { + settlement.requeue(Duration.ZERO).toCompletableFuture().join(); + } catch (RuntimeException settlementFailed) { + // The broker connection is the thing that failed. There is nothing left to try here, and + // the delivery stays unsettled — which is what redelivery is for. + return; + } + } + @Override public CompletionStage pause(String scope) { pausedScopes.add(scope == null ? "" : scope); @@ -167,6 +203,10 @@ public final class RabbitConsumerRegistrar implements TransportConsumerRegistrat this.controller = new RabbitSettlementController(deliveryTag, operations); } + private boolean wasSettled() { + return controller.isSettled(); + } + @Override public CompletionStage acknowledge() { return controller.ack(); diff --git a/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitDeliveryMapper.java b/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitDeliveryMapper.java index 6021c96..f78d932 100644 --- a/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitDeliveryMapper.java +++ b/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitDeliveryMapper.java @@ -135,9 +135,19 @@ public final class RabbitDeliveryMapper { return 1; } try { - return Math.max(1, Integer.parseInt(value)); - } catch (NumberFormatException exception) { - return 1; + int attempt = Integer.parseInt(value); + if (attempt < 1) { + throw new NumberFormatException("attempt " + attempt); + } + return attempt; + } catch (NumberFormatException malformed) { + // Fail closed. Resetting to 1 restarts the retry budget on every redelivery, so a message + // that should have been dead-lettered is retried forever — and the header is + // caller-influenced, which turns "unreadable" into a way to defeat the cap. + throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException( + "RETRY_ATTEMPT_MALFORMED", + "the retry attempt header is not a positive integer; the message is quarantined rather" + + " than restarting its retry budget"); } } diff --git a/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitHeaderMapper.java b/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitHeaderMapper.java index 4d69333..85ad458 100644 --- a/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitHeaderMapper.java +++ b/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitHeaderMapper.java @@ -81,10 +81,24 @@ public final class RabbitHeaderMapper { .baggage() .ifPresent(value -> setHeader(properties, ReservedHeaders.BAGGAGE, value)); + // Application headers are written last, so an unchecked one would overwrite the canonical + // reserved value the platform just wrote. A forged `msg.id` corrupts another message's inbox + // deduplication, not merely its own delivery. envelope .headers() .asMap() - .forEach((name, value) -> setHeader(properties, name.value(), value.value())); + .forEach( + (name, value) -> { + if (dev.caskeleton.messaging.api.header.ReservedHeaders.isReserved(name.value())) { + throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException( + "RESERVED_HEADER_FORGED", + "application header '" + + name.value() + + "' uses the reserved namespace; the canonical value is written by the" + + " platform and must not be overridable by message content"); + } + setHeader(properties, name.value(), value.value()); + }); return properties; } diff --git a/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitMessagingTransport.java b/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitMessagingTransport.java index acb9a7f..89fe488 100644 --- a/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitMessagingTransport.java +++ b/messaging/messaging-rabbit/src/main/java/dev/caskeleton/messaging/rabbit/RabbitMessagingTransport.java @@ -118,7 +118,17 @@ public final class RabbitMessagingTransport implements MessagingTransport { .awaiting(sequence, exchange, routingKey, request.profile().producer().confirmation()) .thenApply(TransportPublishResult::new); - channel.publish(sequence, exchange, routingKey, mapper.toMessage(request), true); + try { + channel.publish(sequence, exchange, routingKey, mapper.toMessage(request), true); + } catch (RuntimeException sendFailed) { + // A synchronous failure from basicPublish used to propagate to the caller as a thrown + // exception while the pending entry stayed in the map: the caller got an exception instead + // of the stage the contract promises, and the entry waited for a confirm that could never + // arrive. The classifier decides whether the frame can have left; anything it cannot rule + // out is ambiguous rather than rejected. + coordinator.sendFailed( + sequence, new RabbitPublishFailureClassifier().classify(sendFailed, true, Duration.ZERO)); + } return pending; } @@ -130,7 +140,19 @@ public final class RabbitMessagingTransport implements MessagingTransport { * @param elapsed how long the publish took */ public void onConfirm(long sequence, boolean acknowledged, Duration elapsed) { - coordinator.confirmed(sequence, acknowledged, elapsed); + onConfirm(sequence, false, acknowledged, elapsed); + } + + /** + * Records a broker confirm carrying the {@code multiple} flag. + * + *

    The flag was dropped, so a broker confirming a range resolved one publish and left every + * earlier one pending forever. + * + * @param multiple true when every sequence up to and including this one is confirmed + */ + public void onConfirm(long sequence, boolean multiple, boolean acknowledged, Duration elapsed) { + coordinator.confirmed(sequence, multiple, acknowledged, elapsed); } /** @@ -188,7 +210,13 @@ public final class RabbitMessagingTransport implements MessagingTransport { @Override public void close() { - closed.set(true); + if (!closed.compareAndSet(false, true)) { + return; + } + // Drain what is still waiting. Leaving the stages pending hangs every caller holding one, and + // a closed channel will never deliver their confirms. + coordinator.channelClosed(Duration.ZERO); + channel.close(); } /** diff --git a/messaging/messaging-rabbit/src/test/java/dev/caskeleton/messaging/rabbit/RabbitConfirmCoordinatorTest.java b/messaging/messaging-rabbit/src/test/java/dev/caskeleton/messaging/rabbit/RabbitConfirmCoordinatorTest.java index c719821..c9bd1ec 100644 --- a/messaging/messaging-rabbit/src/test/java/dev/caskeleton/messaging/rabbit/RabbitConfirmCoordinatorTest.java +++ b/messaging/messaging-rabbit/src/test/java/dev/caskeleton/messaging/rabbit/RabbitConfirmCoordinatorTest.java @@ -113,4 +113,55 @@ class RabbitConfirmCoordinatorTest { assertThat(coordinator.pendingCount()).isZero(); } + + @Test + @org.junit.jupiter.api.DisplayName("a multiple ack resolves every pending publish up to the tag") + void aMultipleAckResolvesEveryPendingPublish() { + RabbitConfirmCoordinator coordinator = new RabbitConfirmCoordinator(); + java.util.concurrent.CompletableFuture + first = coordinator.awaiting(1L, "orders", "created", REPLICATION).toCompletableFuture(); + java.util.concurrent.CompletableFuture + second = coordinator.awaiting(2L, "orders", "created", REPLICATION).toCompletableFuture(); + java.util.concurrent.CompletableFuture + third = coordinator.awaiting(3L, "orders", "created", REPLICATION).toCompletableFuture(); + + coordinator.confirmed(2L, true, true, java.time.Duration.ofMillis(5)); + + assertThat(first) + .as("the multiple flag was dropped, so everything below the tag waited forever") + .isCompleted(); + assertThat(second).isCompleted(); + assertThat(third).isNotCompleted(); + assertThat(coordinator.pendingCount()).isEqualTo(1); + } + + @Test + @org.junit.jupiter.api.DisplayName("a nack records the frame as transmitted") + void aNackRecordsTheFrameAsTransmitted() { + RabbitConfirmCoordinator coordinator = new RabbitConfirmCoordinator(); + java.util.concurrent.CompletableFuture + pending = coordinator.awaiting(1L, "orders", "created", REPLICATION).toCompletableFuture(); + + coordinator.confirmed(1L, false, java.time.Duration.ofMillis(5)); + + assertThat(pending.join().evidence().transmission()) + .as("a NACK is the broker's answer to a frame it received") + .isEqualTo(dev.caskeleton.messaging.api.publish.TransmissionEvidence.TRANSMITTED); + } + + @Test + @org.junit.jupiter.api.DisplayName( + "a closed channel completes every pending publish as ambiguous") + void aClosedChannelCompletesEveryPendingPublish() { + RabbitConfirmCoordinator coordinator = new RabbitConfirmCoordinator(); + java.util.concurrent.CompletableFuture + pending = coordinator.awaiting(1L, "orders", "created", REPLICATION).toCompletableFuture(); + + assertThat(coordinator.channelClosed(java.time.Duration.ofMillis(5))).isEqualTo(1); + + assertThat(pending.join().completion()) + .as("the frames were written; a channel close says nothing about whether they were stored") + .isEqualTo(dev.caskeleton.messaging.api.publish.PublishCompletion.AMBIGUOUS); + assertThat(coordinator.pendingCount()).isZero(); + } } diff --git a/messaging/messaging-rabbit/src/test/java/dev/caskeleton/messaging/rabbit/RabbitRuntimeTest.java b/messaging/messaging-rabbit/src/test/java/dev/caskeleton/messaging/rabbit/RabbitRuntimeTest.java index 36f9d79..7a2f79c 100644 --- a/messaging/messaging-rabbit/src/test/java/dev/caskeleton/messaging/rabbit/RabbitRuntimeTest.java +++ b/messaging/messaging-rabbit/src/test/java/dev/caskeleton/messaging/rabbit/RabbitRuntimeTest.java @@ -260,4 +260,49 @@ class RabbitRuntimeTest { MessageHeaders.empty(), new EncodedMessage(payload, ContentType.JSON, Optional.empty())); } + + @Test + @org.junit.jupiter.api.DisplayName("a handler failure is retried, not discarded as undecodable") + void aHandlerFailureIsRetriedNotDiscarded() { + RecordingOperations operations = new RecordingOperations(); + RabbitConsumerRegistrar registrar = + registrar( + operations, + delivery -> { + throw new IllegalStateException("the projection is down"); + }); + + registrar.onMessage(deliveredMessage(11), NOW); + + assertThat(operations.calls) + .as("one catch covered decode, handler and settlement, so a retryable failure was deleted") + .containsExactly("retry:11:PT0S"); + } + + @Test + @org.junit.jupiter.api.DisplayName("a handler that never settles does not acknowledge") + void aHandlerThatNeverSettlesDoesNotAcknowledge() { + RecordingOperations operations = new RecordingOperations(); + RabbitConsumerRegistrar registrar = + registrar(operations, delivery -> CompletableFuture.completedFuture(null)); + + registrar.onMessage(deliveredMessage(12), NOW); + + assertThat(operations.calls) + .as("acknowledging on the handler's behalf silently drops a message nobody handled") + .containsExactly("retry:12:PT0S"); + } + + @Test + @org.junit.jupiter.api.DisplayName("a delivery after close is refused") + void aDeliveryAfterCloseIsRefused() { + RecordingOperations operations = new RecordingOperations(); + RabbitConsumerRegistrar registrar = + registrar(operations, delivery -> CompletableFuture.completedFuture(null)); + registrar.close(); + + assertThat(registrar.onMessage(deliveredMessage(13), NOW)) + .as("close only flipped a flag that onMessage never read") + .isFalse(); + } } diff --git a/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/InboxRepository.java b/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/InboxRepository.java index 7b124b1..a438805 100644 --- a/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/InboxRepository.java +++ b/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/InboxRepository.java @@ -43,4 +43,11 @@ public interface InboxRepository { * @return how many rows were removed */ int purgeProcessedBefore(Instant processedBefore); + + /** + * Deletes at most {@code limit} processed rows older than the cutoff. + * + * @return how many rows were deleted; fewer than {@code limit} means the sweep is finished + */ + int purgeProcessedBefore(Instant processedBefore, int limit); } diff --git a/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxLease.java b/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxLease.java new file mode 100644 index 0000000..58a6884 --- /dev/null +++ b/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxLease.java @@ -0,0 +1,42 @@ +package dev.caskeleton.messaging.reliability; + +import java.time.Instant; +import java.util.Objects; + +/** + * One relay's claim on one outbox record. + * + *

    The port used to take a {@code MessageId} for every terminal transition, so a write said which + * row to change and nothing about which claim it belonged to. A relay that stalled past its lease + * could still record {@code AMBIGUOUS} over the {@code PUBLISHED} another relay had already + * written, and the row became claimable again — one message, published twice, by a system whose + * whole purpose is to publish it once. + * + *

    The token is the part that makes staleness detectable. It increases on every claim, so a + * superseded relay holds a number the row no longer has and its update matches zero rows. + */ +public record OutboxLease(OutboxRecord record, String owner, long token, Instant expiresAt) { + + public OutboxLease { + Objects.requireNonNull(record, "record must not be null"); + Objects.requireNonNull(owner, "owner must not be null"); + Objects.requireNonNull(expiresAt, "expiresAt must not be null"); + if (owner.isBlank()) { + throw new IllegalArgumentException("a lease names its owner"); + } + if (token < 1) { + throw new IllegalArgumentException( + "a claim's token starts at 1; 0 is the value of a row nobody has claimed"); + } + } + + /** The claimed record's identity. */ + public dev.caskeleton.messaging.api.MessageId messageId() { + return record.messageId(); + } + + /** True when this lease has run out at the given moment. */ + public boolean expiredAt(Instant now) { + return !Objects.requireNonNull(now, "now must not be null").isBefore(expiresAt); + } +} diff --git a/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxRepository.java b/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxRepository.java index fa94f24..39f00a7 100644 --- a/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxRepository.java +++ b/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxRepository.java @@ -35,6 +35,29 @@ public interface OutboxRepository { */ List leaseBatch(int batchSize, Duration leaseDuration, Instant now); + /** + * Claims a batch and returns the leases, each carrying its owner and fencing token. + * + *

    The token is what a terminal write is checked against. {@link #leaseBatch} returns records + * without one, so its callers cannot prove a write belongs to their claim; it remains for + * inspection paths and is deprecated for the relay's use. + * + * @param owner this relay's stable identity + */ + List claimBatch(String owner, int batchSize, Duration leaseDuration, Instant now); + + /** + * Claims a batch, skipping rows that are waiting for their backoff or out of attempts. + * + *

    Eligibility belongs in the row. A relay that decides in memory forgets on restart, and every + * replica decides for itself — so the same message is retried by as many workers as there are + * replicas, as often as they poll. + * + * @param maxAttempts the attempt budget; rows at or above it are not claimed + */ + List claimBatch( + String owner, int batchSize, Duration leaseDuration, Instant now, int maxAttempts); + /** * Marks a record as confirmed by the broker. * @@ -43,6 +66,13 @@ public interface OutboxRepository { */ void markPublished(MessageId messageId, Instant now); + /** + * Records a confirmed publication, only if this lease still owns the row. + * + * @return {@link OutboxTransitionResult#STALE_LEASE} when another relay has taken it + */ + OutboxTransitionResult markPublished(OutboxLease lease, Instant now); + /** * Marks a record whose publish outcome is unknown. * @@ -54,6 +84,21 @@ public interface OutboxRepository { */ void markAmbiguous(MessageId messageId, String failureCode, Instant now); + /** Records an ambiguous outcome, only if this lease still owns the row. */ + OutboxTransitionResult markAmbiguous(OutboxLease lease, String failureCode, Instant now); + + /** Records an ambiguous outcome and when this row may next be claimed. */ + OutboxTransitionResult markAmbiguous( + OutboxLease lease, String failureCode, Instant now, Instant nextAttemptAt); + + /** + * Parks a row whose attempt budget ran out. + * + *

    Separate from {@link #markFailed}: a rejected message is invalid, an exhausted one never got + * an answer. The first needs a fix, the second a redrive. + */ + OutboxTransitionResult markExhausted(OutboxLease lease, String reason, Instant now); + /** * Marks a record the broker definitively rejected. * @@ -63,6 +108,9 @@ public interface OutboxRepository { */ void markFailed(MessageId messageId, String failureCode, Instant now); + /** Records a terminal failure, only if this lease still owns the row. */ + OutboxTransitionResult markFailed(OutboxLease lease, String failureCode, Instant now); + /** * Releases a lease without changing the record's outcome. * @@ -70,6 +118,9 @@ public interface OutboxRepository { */ void releaseLease(MessageId messageId); + /** Gives a claim back, only if this lease still owns the row. */ + OutboxTransitionResult releaseLease(OutboxLease lease); + /** * Finds one record. * @@ -85,4 +136,17 @@ public interface OutboxRepository { * @return how many rows were removed */ int purgePublishedBefore(Instant publishedBefore); + + /** + * Deletes at most {@code limit} published rows older than the cutoff. + * + *

    The unbounded version deletes everything before the cutoff in one statement. On a table that + * has been accumulating published rows since the last sweep that is a single long transaction + * holding locks and generating WAL in proportion to the backlog, which shows up as the relay and + * the business writes stalling behind retention. The cleanup jobs describe themselves as bounded + * by batch size; this is the parameter that makes that true. + * + * @return how many rows were deleted; fewer than {@code limit} means the sweep is finished + */ + int purgePublishedBefore(Instant publishedBefore, int limit); } diff --git a/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxStatus.java b/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxStatus.java index 2fe183f..d15ba36 100644 --- a/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxStatus.java +++ b/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxStatus.java @@ -23,5 +23,15 @@ public enum OutboxStatus { AMBIGUOUS, /** Definitively rejected; will not be retried without intervention. */ - FAILED + FAILED, + + /** + * The attempt budget ran out before any confirmed publish. + * + *

    Distinct from {@link #FAILED}, which means the broker refused the message: this one means + * nobody ever got an answer. Collapsing the two loses the difference between "this message is + * invalid" and "the broker was unreachable for an hour", and those need different operator + * actions — the first a fix, the second a redrive. + */ + EXHAUSTED } diff --git a/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxTransitionResult.java b/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxTransitionResult.java new file mode 100644 index 0000000..923c958 --- /dev/null +++ b/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxTransitionResult.java @@ -0,0 +1,24 @@ +package dev.caskeleton.messaging.reliability; + +/** + * What a terminal outbox write actually did. + * + *

    The transitions returned {@code void}, so an update that matched zero rows was + * indistinguishable from one that matched one. That is precisely the stale-lease case: the relay + * believes it recorded the outcome, the row still says something else, and nothing anywhere counts + * the disagreement. + */ +public enum OutboxTransitionResult { + + /** The row moved to the requested state. */ + APPLIED, + + /** + * The row is no longer this lease's. + * + *

    Another relay claimed it after the lease expired. Not an error to throw — the message is + * being handled by somebody else — but never a success either: it is the signal that this + * worker's publish attempt may have produced a duplicate, and it belongs on a metric. + */ + STALE_LEASE +} diff --git a/messaging/messaging-runtime-core/build.gradle b/messaging/messaging-runtime-core/build.gradle new file mode 100644 index 0000000..4bf0287 --- /dev/null +++ b/messaging/messaging-runtime-core/build.gradle @@ -0,0 +1,17 @@ +apply plugin: 'java-library' + +// The central publish and delivery orchestration. +// +// MessagePublisher was an interface with no implementation anywhere in the new platform: the +// brokers implemented MessagingTransport, the core auto-configuration built dead-letter and facade +// beans on top of a publisher bean that nothing supplied, and admission, security, runtime leases +// and observation existed as beans that no publish path ever called. A starter that filled the gap +// with an application-supplied fake would pass a context test while running none of them. +dependencies { + api project(':messaging:messaging-core-api') + api project(':messaging:messaging-schema-api') + api project(':messaging:messaging-policy') + api project(':messaging:messaging-transport-spi') + api project(':messaging:messaging-security') + api project(':messaging:messaging-observability') +} diff --git a/messaging/messaging-runtime-core/gradle.lockfile b/messaging/messaging-runtime-core/gradle.lockfile new file mode 100644 index 0000000..8a9cedb --- /dev/null +++ b/messaging/messaging-runtime-core/gradle.lockfile @@ -0,0 +1,88 @@ +# This is a Gradle generated file for dependency locking. +# Manual edits can break the build and are not advised. +# This file is expected to be part of source control. +com.github.ben-manes.caffeine:caffeine:3.2.3=annotationProcessor,testAnnotationProcessor +com.github.kevinstern:software-and-algorithms:1.0=annotationProcessor,testAnnotationProcessor +com.github.spotbugs:spotbugs-annotations:4.10.2=spotbugs +com.github.spotbugs:spotbugs:4.10.2=spotbugs +com.github.stephenc.jcip:jcip-annotations:1.0-1=spotbugs +com.google.auto.service:auto-service-annotations:1.0.1=annotationProcessor,testAnnotationProcessor +com.google.auto.value:auto-value-annotations:1.9=annotationProcessor,testAnnotationProcessor +com.google.auto:auto-common:1.2.2=annotationProcessor,testAnnotationProcessor +com.google.code.findbugs:jsr305:3.0.2=checkstyle,spotbugs +com.google.code.gson:gson:2.13.2=spotbugs +com.google.errorprone:error_prone_annotation:2.49.0=annotationProcessor,testAnnotationProcessor +com.google.errorprone:error_prone_annotations:2.41.0=spotbugs +com.google.errorprone:error_prone_annotations:2.47.0=checkstyle +com.google.errorprone:error_prone_annotations:2.49.0=annotationProcessor,testAnnotationProcessor +com.google.errorprone:error_prone_check_api:2.49.0=annotationProcessor,testAnnotationProcessor +com.google.errorprone:error_prone_core:2.49.0=annotationProcessor,testAnnotationProcessor +com.google.googlejavaformat:google-java-format:1.35.0=annotationProcessor,testAnnotationProcessor +com.google.guava:failureaccess:1.0.3=annotationProcessor,checkstyle,testAnnotationProcessor +com.google.guava:guava:33.5.0-jre=annotationProcessor,testAnnotationProcessor +com.google.guava:guava:33.6.0-jre=checkstyle +com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava=annotationProcessor,checkstyle,testAnnotationProcessor +com.google.j2objc:j2objc-annotations:3.1=annotationProcessor,checkstyle,testAnnotationProcessor +com.google.protobuf:protobuf-java:4.33.2=annotationProcessor,testAnnotationProcessor +com.h3xstream.findsecbugs:findsecbugs-plugin:1.14.0=spotbugsPlugins +com.puppycrawl.tools:checkstyle:13.5.0=checkstyle +commons-beanutils:commons-beanutils:1.11.0=checkstyle +commons-collections:commons-collections:3.2.2=checkstyle +commons-io:commons-io:2.21.0=spotbugs +info.picocli:picocli:4.7.7=checkstyle +io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnotationProcessor +io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor +io.micrometer:micrometer-commons:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.micrometer:micrometer-core:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.micrometer:micrometer-observation:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +javax.inject:javax.inject:1=annotationProcessor,testAnnotationProcessor +jaxen:jaxen:2.0.0=spotbugs +net.bytebuddy:byte-buddy:1.17.8=testCompileClasspath,testRuntimeClasspath +net.sf.saxon:Saxon-HE:12.9=checkstyle,spotbugs +org.antlr:antlr4-runtime:4.13.2=checkstyle +org.apache.bcel:bcel:6.12.0=spotbugs +org.apache.commons:commons-lang3:3.20.0=checkstyle,spotbugs +org.apache.commons:commons-text:1.15.0=spotbugs +org.apache.commons:commons-text:1.3=checkstyle +org.apache.httpcomponents:httpclient:4.5.13=checkstyle +org.apache.httpcomponents:httpcore:4.4.16=checkstyle +org.apache.logging.log4j:log4j-api:2.25.2=spotbugs +org.apache.logging.log4j:log4j-core:2.25.2=spotbugs +org.apache.maven.doxia:doxia-core:1.12.0=checkstyle +org.apache.maven.doxia:doxia-logging-api:1.12.0=checkstyle +org.apache.maven.doxia:doxia-module-xdoc:1.12.0=checkstyle +org.apache.maven.doxia:doxia-sink-api:1.12.0=checkstyle +org.apache.xbean:xbean-reflect:3.7=checkstyle +org.apiguardian:apiguardian-api:1.1.2=testCompileClasspath +org.assertj:assertj-core:3.27.6=testCompileClasspath,testRuntimeClasspath +org.codehaus.plexus:plexus-classworlds:2.6.0=checkstyle +org.codehaus.plexus:plexus-component-annotations:2.1.0=checkstyle +org.codehaus.plexus:plexus-container-default:2.1.0=checkstyle +org.codehaus.plexus:plexus-utils:3.3.0=checkstyle +org.dom4j:dom4j:2.2.0=spotbugs +org.hdrhistogram:HdrHistogram:2.2.2=runtimeClasspath,testRuntimeClasspath +org.javassist:javassist:3.28.0-GA=checkstyle +org.jspecify:jspecify:1.0.0=annotationProcessor,checkstyle,compileClasspath,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-api:6.0.1=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-engine:6.0.1=testRuntimeClasspath +org.junit.jupiter:junit-jupiter-params:6.0.1=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter:6.0.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-commons:6.0.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-engine:6.0.1=testRuntimeClasspath +org.junit.platform:junit-platform-launcher:6.0.1=testRuntimeClasspath +org.junit:junit-bom:6.0.1=testCompileClasspath,testRuntimeClasspath +org.junit:junit-bom:6.1.0=spotbugs +org.latencyutils:LatencyUtils:2.0.3=runtimeClasspath,testRuntimeClasspath +org.mockito:mockito-core:5.20.0=mockitoAgent +org.opentest4j:opentest4j:1.3.0=testCompileClasspath,testRuntimeClasspath +org.ow2.asm:asm-analysis:9.10.1=spotbugs +org.ow2.asm:asm-commons:9.10.1=spotbugs +org.ow2.asm:asm-tree:9.10.1=spotbugs +org.ow2.asm:asm-util:9.10.1=spotbugs +org.ow2.asm:asm:9.10.1=spotbugs +org.pcollections:pcollections:4.0.1=annotationProcessor,testAnnotationProcessor +org.reflections:reflections:0.10.2=checkstyle +org.slf4j:slf4j-api:2.0.17=spotbugs,spotbugsSlf4j +org.slf4j:slf4j-simple:2.0.17=checkstyle,spotbugsSlf4j +org.xmlresolver:xmlresolver:5.3.3=checkstyle,spotbugs +empty= diff --git a/messaging/messaging-runtime-core/src/main/java/dev/caskeleton/messaging/runtime/DefaultDeliveryProcessor.java b/messaging/messaging-runtime-core/src/main/java/dev/caskeleton/messaging/runtime/DefaultDeliveryProcessor.java new file mode 100644 index 0000000..7c9dfd7 --- /dev/null +++ b/messaging/messaging-runtime-core/src/main/java/dev/caskeleton/messaging/runtime/DefaultDeliveryProcessor.java @@ -0,0 +1,155 @@ +package dev.caskeleton.messaging.runtime; + +import dev.caskeleton.messaging.api.MessageEnvelope; +import dev.caskeleton.messaging.api.delivery.HandleResult; +import dev.caskeleton.messaging.api.error.FailureCategory; +import dev.caskeleton.messaging.api.error.FailureDescriptor; +import dev.caskeleton.messaging.api.publish.PublishCompletion; +import dev.caskeleton.messaging.api.settlement.SettlementResult; +import dev.caskeleton.messaging.schema.EncodedMessage; +import dev.caskeleton.messaging.transport.TransportDelivery; +import java.time.Duration; +import java.util.Objects; +import java.util.Optional; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CompletionStage; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.function.Function; + +/** + * Turns one handler outcome into exactly one broker settlement. + * + *

    Nothing connected {@code HandleResult} to a settlement. Each broker adapter decided for itself + * what a retry or a dead-letter meant, so "the platform decides when and in what order the + * settlement happens" — the comment on {@code TransportSettlement} — described a decision nobody + * made in one place. + * + *

    Two rules are enforced here rather than left to each adapter: + * + *

    + */ +public final class DefaultDeliveryProcessor { + + private final Function, HandleResult> handler; + + private final DeadLetterPublisher deadLetters; + + private final Duration retryDelay; + + public DefaultDeliveryProcessor( + Function, HandleResult> handler, + DeadLetterPublisher deadLetters, + Duration retryDelay) { + this.handler = Objects.requireNonNull(handler, "handler must not be null"); + this.deadLetters = Objects.requireNonNull(deadLetters, "deadLetters must not be null"); + this.retryDelay = Objects.requireNonNull(retryDelay, "retryDelay must not be null"); + if (retryDelay.isNegative()) { + throw new IllegalArgumentException("a retry delay must not be negative"); + } + } + + /** + * Handles one delivery and settles it once. + * + * @return a stage completing with the settlement the broker performed + */ + public CompletionStage process(TransportDelivery delivery) { + Objects.requireNonNull(delivery, "delivery must not be null"); + OneShotSettlement settlement = new OneShotSettlement(delivery); + + HandleResult result; + try { + result = handler.apply(delivery.envelope()); + } catch (RuntimeException handlerFailed) { + // A handler that threw is a retry, not a discard. Treating an exception as "this message is + // undeliverable" is how a transient bug in one consumer silently drops a day of traffic — + // and it is exactly what the Rabbit consumer did by folding handler exceptions into its + // deserialization-failure path. + return settlement.requeue(retryDelay); + } + if (result == null) { + return settlement.requeue(retryDelay); + } + + return switch (result) { + case HandleResult.Success ignored -> settlement.acknowledge(); + case HandleResult.Retry ignored -> settlement.requeue(retryDelay); + case HandleResult.Reject ignored -> settlement.discard(); + case HandleResult.DeadLetter deadLetter -> + deadLetters + .publish(delivery.envelope(), deadLetter.failure()) + .thenCompose( + completion -> + completion == PublishCompletion.CONFIRMED + // Confirmed: the message exists somewhere else, so removing it here is + // safe. + ? settlement.acknowledge() + // Not confirmed — rejected or ambiguous. Requeueing risks a duplicate; + // acknowledging loses the message outright, and a duplicate is the + // recoverable half of that choice. + : settlement.requeue(retryDelay)); + }; + } + + /** Publishes a delivery that cannot be handled to its dead-letter destination. */ + @FunctionalInterface + public interface DeadLetterPublisher { + + /** Publishes and reports what the broker confirmed. */ + CompletionStage publish( + MessageEnvelope envelope, FailureDescriptor failure); + } + + /** A settlement that can be used once. */ + private static final class OneShotSettlement { + + private final TransportDelivery delivery; + + private final AtomicBoolean settled = new AtomicBoolean(); + + private OneShotSettlement(TransportDelivery delivery) { + this.delivery = delivery; + } + + private CompletionStage acknowledge() { + return once("acknowledge", () -> delivery.settlement().acknowledge()); + } + + private CompletionStage requeue(Duration delay) { + return once("requeue", () -> delivery.settlement().requeue(delay)); + } + + private CompletionStage discard() { + return once("discard", () -> delivery.settlement().discard()); + } + + private CompletionStage once( + String what, java.util.function.Supplier> call) { + if (!settled.compareAndSet(false, true)) { + return CompletableFuture.failedFuture( + new IllegalStateException( + "this delivery has already been settled; a second " + + what + + " would tell the broker something the first call already contradicted")); + } + return call.get(); + } + } + + /** A descriptor for a handler that returned nothing, kept out of the hot path. */ + static FailureDescriptor missingResult() { + return new FailureDescriptor( + FailureCategory.PROCESSING_TRANSIENT, + "HANDLER_RETURNED_NOTHING", + true, + "the handler returned no result, which is treated as a retry rather than a discard", + Optional.empty()); + } +} diff --git a/messaging/messaging-runtime-core/src/main/java/dev/caskeleton/messaging/runtime/DefaultMessagePublisher.java b/messaging/messaging-runtime-core/src/main/java/dev/caskeleton/messaging/runtime/DefaultMessagePublisher.java new file mode 100644 index 0000000..645e2d6 --- /dev/null +++ b/messaging/messaging-runtime-core/src/main/java/dev/caskeleton/messaging/runtime/DefaultMessagePublisher.java @@ -0,0 +1,313 @@ +package dev.caskeleton.messaging.runtime; + +import dev.caskeleton.messaging.api.MessageEnvelope; +import dev.caskeleton.messaging.api.destination.MessageDestination; +import dev.caskeleton.messaging.api.error.FailureCategory; +import dev.caskeleton.messaging.api.error.FailureDescriptor; +import dev.caskeleton.messaging.api.publish.MessagePublisher; +import dev.caskeleton.messaging.api.publish.PublishCompletion; +import dev.caskeleton.messaging.api.publish.PublishEvidence; +import dev.caskeleton.messaging.api.publish.PublishOptions; +import dev.caskeleton.messaging.api.publish.PublishResult; +import dev.caskeleton.messaging.api.publish.RoutingOutcome; +import dev.caskeleton.messaging.policy.DestinationProfile; +import dev.caskeleton.messaging.policy.MessagingAdmissionController; +import dev.caskeleton.messaging.schema.EncodedMessage; +import dev.caskeleton.messaging.schema.MessageCodec; +import dev.caskeleton.messaging.schema.MessageCodecRegistry; +import dev.caskeleton.messaging.security.DestinationAccessPolicy; +import dev.caskeleton.messaging.transport.MessagingRuntimeLease; +import dev.caskeleton.messaging.transport.MessagingRuntimeRegistry; +import dev.caskeleton.messaging.transport.MessagingTransport; +import dev.caskeleton.messaging.transport.TransportPublishRequest; +import java.time.Duration; +import java.util.Objects; +import java.util.Optional; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CompletionStage; +import java.util.function.LongSupplier; + +/** + * The one publish path: resolve, authorize, encode, admit, lease, send, normalize. + * + *

    {@code MessagePublisher} was an interface with no implementation. The brokers implemented + * {@code MessagingTransport}, which takes an already-resolved profile and already-encoded bytes; + * the core auto-configuration built a dead-letter orchestrator and blocking/reactive facades on top + * of a publisher bean nothing supplied; and the admission controller, access policy, runtime + * registry and observation existed as beans that no publish ever called. An application that filled + * the gap with its own publisher would pass a context test while running none of them. + * + *

    The order below is fixed, not composed from a map of interceptors. Each stage's position is a + * decision: + * + *

      + *
    • destination and access first, so an unauthorized publish never encodes a payload; + *
    • encoding before admission, because the admission bound is on bytes and the byte count is + * not known until the payload is encoded; + *
    • the runtime lease last before the send, so a rotation cannot swap the transport underneath + * a message that has already been counted against the in-flight limit. + *
    + * + *

    Everything acquired is released exactly once, on every path — success, failure, exception and + * cancellation. A permit or lease that leaks on the failure path is a limiter that shrinks by one + * per failure until it stops accepting anything. + */ +public final class DefaultMessagePublisher implements MessagePublisher { + + private final DestinationProfileRegistry destinations; + + private final DestinationAccessPolicy access; + + private final MessageCodecRegistry codecs; + + private final MessagingAdmissionController admission; + + private final MessagingRuntimeRegistry runtimes; + + private final MessagingTransport transport; + + private final LongSupplier nanoTime; + + /** + * Where every publish outcome is recorded. + * + *

    {@code MessagingObservation} existed as a bean and no publish path called it, so the + * platform's own metrics described nothing. It is a constructor argument rather than an optional + * decorator because an unobserved publish path is how "the dashboards were empty during the + * incident" happens. + */ + private final dev.caskeleton.messaging.observation.MessagingObservation observation; + + public DefaultMessagePublisher( + DestinationProfileRegistry destinations, + DestinationAccessPolicy access, + MessageCodecRegistry codecs, + MessagingAdmissionController admission, + MessagingRuntimeRegistry runtimes, + MessagingTransport transport) { + this( + destinations, + access, + codecs, + admission, + runtimes, + transport, + System::nanoTime, + NO_OBSERVATION); + } + + /** A no-op observation, for a deployment that has not wired metrics. */ + private static final dev.caskeleton.messaging.observation.MessagingObservation NO_OBSERVATION = + new dev.caskeleton.messaging.observation.MessagingObservation() { + @Override + public void recordPublish( + dev.caskeleton.messaging.observation.MessagingTags tags, Duration elapsed) {} + + @Override + public void recordDelivery( + dev.caskeleton.messaging.observation.MessagingTags tags, + Duration elapsed, + int attempt) {} + + @Override + public void recordSettlement(dev.caskeleton.messaging.observation.MessagingTags tags) {} + + @Override + public void recordBacklog( + dev.caskeleton.messaging.observation.MessagingTags tags, long messages) {} + + @Override + public void recordDiagnostics( + dev.caskeleton.messaging.observation.MessagingTags tags, + java.util.Map diagnostics) {} + }; + + /** A publisher with an injectable clock, so elapsed time is testable without sleeping. */ + public DefaultMessagePublisher( + DestinationProfileRegistry destinations, + DestinationAccessPolicy access, + MessageCodecRegistry codecs, + MessagingAdmissionController admission, + MessagingRuntimeRegistry runtimes, + MessagingTransport transport, + LongSupplier nanoTime) { + this(destinations, access, codecs, admission, runtimes, transport, nanoTime, NO_OBSERVATION); + } + + /** A publisher that records every outcome. */ + public DefaultMessagePublisher( + DestinationProfileRegistry destinations, + DestinationAccessPolicy access, + MessageCodecRegistry codecs, + MessagingAdmissionController admission, + MessagingRuntimeRegistry runtimes, + MessagingTransport transport, + LongSupplier nanoTime, + dev.caskeleton.messaging.observation.MessagingObservation observation) { + this.observation = Objects.requireNonNull(observation, "observation must not be null"); + this.destinations = Objects.requireNonNull(destinations, "destinations must not be null"); + this.access = Objects.requireNonNull(access, "access must not be null"); + this.codecs = Objects.requireNonNull(codecs, "codecs must not be null"); + this.admission = Objects.requireNonNull(admission, "admission must not be null"); + this.runtimes = Objects.requireNonNull(runtimes, "runtimes must not be null"); + this.transport = Objects.requireNonNull(transport, "transport must not be null"); + this.nanoTime = Objects.requireNonNull(nanoTime, "nanoTime must not be null"); + } + + @Override + public CompletionStage publish( + MessageDestination destination, MessageEnvelope message, PublishOptions options) { + Objects.requireNonNull(destination, "destination must not be null"); + Objects.requireNonNull(message, "message must not be null"); + Objects.requireNonNull(options, "options must not be null"); + + long startedAt = nanoTime.getAsLong(); + DestinationProfile profile; + MessageEnvelope encoded; + try { + profile = destinations.require(destination.name()); + requireSupportedOptions(profile, options); + if (!access.mayPublish(destination.name())) { + // Before encoding: an unauthorized publish must not serialise the payload, because the + // encoded bytes are what a claim-check or a log would then be holding. + return rejected( + "PUBLISH_FORBIDDEN", + "this application may not publish to '" + destination.name().value() + '\'', + startedAt); + } + encoded = encode(message); + } catch (RuntimeException beforeTheWire) { + // Nothing left this process, so the outcome is definite. Reporting it as ambiguous would send + // the caller into reconciliation for a message no broker ever saw. + return rejected("PUBLISH_PREPARATION_FAILED", sanitized(beforeTheWire), startedAt); + } + + admission.admit(destination.name().value(), encoded.payload().bytes().length); + MessagingRuntimeLease lease = null; + try { + lease = runtimes.acquire(profile.broker()); + MessagingRuntimeLease held = lease; + return transport + .publish(new TransportPublishRequest(profile, encoded, options)) + .handle( + (result, failure) -> { + // One release per acquisition, whatever happened. A permit that leaks on the + // failure path is a limiter that shrinks by one per failure until it accepts + // nothing. + held.close(); + admission.complete(); + if (failure != null) { + // The request was on the wire when this failed, so the broker may hold it. + PublishResult ambiguous = ambiguousResult(sanitized(failure), startedAt); + observe(profile, ambiguous, startedAt); + return ambiguous; + } + observe(profile, result.result(), startedAt); + return result.result(); + }); + } catch (RuntimeException beforeTheSend) { + if (lease != null) { + lease.close(); + } + admission.complete(); + return rejected("PUBLISH_RUNTIME_UNAVAILABLE", sanitized(beforeTheSend), startedAt); + } + } + + /** + * Refuses options the destination cannot honour, before anything is encoded or sent. + * + *

    The transports read {@code request.options()} for nothing: the timeout, the confirmation + * requirement, the deduplication request and the broker hints were all accepted and ignored. A + * caller asking for broker-side deduplication got a publish with no deduplication and no error, + * and then skipped the idempotency it would otherwise have written — which is exactly the case + * {@code PublishDeduplication}'s own javadoc says must be a startup failure rather than a silent + * no-op. + */ + private void requireSupportedOptions(DestinationProfile profile, PublishOptions options) { + if (!options.isM1Compatible()) { + // isM1Compatible existed and nothing called it, so an ordinary caller could attach M3 broker + // hints to an M1 publish and have them dropped. + throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException( + "PUBLISH_BROKER_HINTS_UNSUPPORTED", + "broker hints are an M3 surface and this publish path does not forward them; passing" + + " them here would drop them silently"); + } + if (options.deduplication().isPresent() + && !transport.capabilities(profile.name()).capabilities().deduplicatedPublish()) { + throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException( + "PUBLISH_DEDUPLICATION_UNSUPPORTED", + "destination '" + + profile.name().value() + + "' runs on a broker that does not suppress duplicate publishes of a stable id;" + + " the caller must keep its own idempotency rather than believing this request"); + } + } + + /** Records one publish against bounded dimensions: broker, destination, operation, outcome. */ + private void observe(DestinationProfile profile, PublishResult result, long startedAt) { + observation.recordPublish( + dev.caskeleton.messaging.observation.MessagingTags.of( + profile.broker(), + profile.name().value(), + "publish", + result.completion().name().toLowerCase(java.util.Locale.ROOT)), + elapsedSince(startedAt)); + } + + private MessageEnvelope encode(MessageEnvelope message) { + MessageCodec codec = codecs.find(message.contentType()).orElseGet(codecs::defaultCodec); + EncodedMessage bytes = + codec.encode(message.messageType(), message.schemaVersion(), message.payload()); + return message.withPayload(bytes); + } + + private CompletionStage rejected(String code, String reason, long startedAt) { + return CompletableFuture.completedFuture( + new PublishResult( + PublishCompletion.REJECTED, + PublishEvidence.notTransmitted(), + RoutingOutcome.UNKNOWN, + Optional.empty(), + 1, + elapsedSince(startedAt), + Optional.of( + new FailureDescriptor( + FailureCategory.CONFIGURATION, code, false, reason, Optional.empty())))); + } + + private PublishResult ambiguousResult(String reason, long startedAt) { + return new PublishResult( + PublishCompletion.AMBIGUOUS, + PublishEvidence.ambiguous(), + RoutingOutcome.UNKNOWN, + Optional.empty(), + 1, + elapsedSince(startedAt), + Optional.of( + new FailureDescriptor( + FailureCategory.AMBIGUOUS, + "PUBLISH_OUTCOME_UNKNOWN", + true, + reason, + Optional.empty()))); + } + + /** + * A failure's type, never its message. + * + *

    A driver message can carry a routing key, a payload fragment or a connection string, and a + * {@code FailureDescriptor} is designed to be logged and exported. + */ + private static String sanitized(Throwable failure) { + Throwable cause = + failure instanceof java.util.concurrent.CompletionException && failure.getCause() != null + ? failure.getCause() + : failure; + return cause.getClass().getSimpleName(); + } + + private Duration elapsedSince(long startedAt) { + return Duration.ofNanos(Math.max(0, nanoTime.getAsLong() - startedAt)); + } +} diff --git a/messaging/messaging-runtime-core/src/main/java/dev/caskeleton/messaging/runtime/DestinationProfileRegistry.java b/messaging/messaging-runtime-core/src/main/java/dev/caskeleton/messaging/runtime/DestinationProfileRegistry.java new file mode 100644 index 0000000..ea32eaf --- /dev/null +++ b/messaging/messaging-runtime-core/src/main/java/dev/caskeleton/messaging/runtime/DestinationProfileRegistry.java @@ -0,0 +1,62 @@ +package dev.caskeleton.messaging.runtime; + +import dev.caskeleton.messaging.api.destination.DestinationName; +import dev.caskeleton.messaging.api.error.MessagingConfigurationException; +import dev.caskeleton.messaging.policy.DestinationProfile; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Objects; + +/** + * The closed set of destinations this application may publish to. + * + *

    Nothing resolved a logical destination to a profile before this: the brokers took an + * already-resolved {@code DestinationProfile} and the publisher that would have produced one did + * not exist. A registry rather than a lookup with a fallback, because a destination nobody declared + * has no physical name, no ordering guarantee and no payload bound — publishing to it would mean + * inventing all three at the call site. + */ +public final class DestinationProfileRegistry { + + private final Map profiles; + + private DestinationProfileRegistry(Map profiles) { + this.profiles = Map.copyOf(profiles); + } + + /** A registry over the given profiles. */ + public static DestinationProfileRegistry of(Map profiles) { + return new DestinationProfileRegistry( + new LinkedHashMap<>(Objects.requireNonNull(profiles, "profiles must not be null"))); + } + + /** An empty registry: every publish is refused until a destination is declared. */ + public static DestinationProfileRegistry empty() { + return new DestinationProfileRegistry(Map.of()); + } + + /** + * The profile for a destination. + * + * @throws MessagingConfigurationException naming the destination, because an unknown destination + * is a configuration mistake and not something to guess a default for + */ + public DestinationProfile require(DestinationName destination) { + Objects.requireNonNull(destination, "destination must not be null"); + DestinationProfile profile = profiles.get(destination); + if (profile == null) { + throw new MessagingConfigurationException( + "DESTINATION_NOT_REGISTERED", + "destination '" + + destination.value() + + "' has no registered profile; publishing to it would invent its physical name," + + " ordering guarantee and payload bound at the call site"); + } + return profile; + } + + /** How many destinations are declared. */ + public int size() { + return profiles.size(); + } +} diff --git a/messaging/messaging-runtime-core/src/test/java/dev/caskeleton/messaging/runtime/DefaultDeliveryProcessorTest.java b/messaging/messaging-runtime-core/src/test/java/dev/caskeleton/messaging/runtime/DefaultDeliveryProcessorTest.java new file mode 100644 index 0000000..9b2d007 --- /dev/null +++ b/messaging/messaging-runtime-core/src/test/java/dev/caskeleton/messaging/runtime/DefaultDeliveryProcessorTest.java @@ -0,0 +1,209 @@ +package dev.caskeleton.messaging.runtime; + +import static org.assertj.core.api.Assertions.assertThat; + +import dev.caskeleton.messaging.api.ContentType; +import dev.caskeleton.messaging.api.MessageEnvelope; +import dev.caskeleton.messaging.api.MessageId; +import dev.caskeleton.messaging.api.MessageType; +import dev.caskeleton.messaging.api.ProducerId; +import dev.caskeleton.messaging.api.SchemaVersion; +import dev.caskeleton.messaging.api.TraceContext; +import dev.caskeleton.messaging.api.delivery.DeliveryMetadata; +import dev.caskeleton.messaging.api.delivery.HandleResult; +import dev.caskeleton.messaging.api.error.FailureCategory; +import dev.caskeleton.messaging.api.error.FailureDescriptor; +import dev.caskeleton.messaging.api.header.MessageHeaders; +import dev.caskeleton.messaging.api.publish.PublishCompletion; +import dev.caskeleton.messaging.api.settlement.SettlementResult; +import dev.caskeleton.messaging.schema.EncodedMessage; +import dev.caskeleton.messaging.transport.TransportDelivery; +import dev.caskeleton.messaging.transport.TransportSettlement; +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.time.Instant; +import java.util.ArrayList; +import java.util.List; +import java.util.Optional; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CompletionStage; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** + * One handler outcome, one settlement. + * + *

    Nothing turned a {@code HandleResult} into a settlement: each broker adapter decided for + * itself, and the Rabbit consumer folded handler exceptions into its deserialization-failure path + * and discarded them. + */ +class DefaultDeliveryProcessorTest { + + private final RecordingSettlement settlement = new RecordingSettlement(); + + @Test + @DisplayName("success acknowledges once") + void successAcknowledgesOnce() { + process(envelope -> new HandleResult.Success(), confirmedDeadLetters()); + + assertThat(settlement.calls()).containsExactly("acknowledge"); + } + + @Test + @DisplayName("retry requeues with the configured delay") + void retryRequeues() { + process(envelope -> new HandleResult.Retry(failure()), confirmedDeadLetters()); + + assertThat(settlement.calls()).containsExactly("requeue"); + } + + @Test + @DisplayName("a handler that throws is retried, not discarded") + void aThrowingHandlerIsRetried() { + process( + envelope -> { + throw new IllegalStateException("a bug in the projection"); + }, + confirmedDeadLetters()); + + assertThat(settlement.calls()) + .as("treating an exception as undeliverable drops a day of traffic over a transient bug") + .containsExactly("requeue"); + } + + @Test + @DisplayName("a dead letter is published before the source is acknowledged") + void aDeadLetterIsPublishedBeforeAcknowledgement() { + List order = new ArrayList<>(); + DefaultDeliveryProcessor.DeadLetterPublisher publisher = + (envelope, failure) -> { + order.add("dead-letter published"); + return CompletableFuture.completedFuture(PublishCompletion.CONFIRMED); + }; + settlement.recordInto(order); + + process(envelope -> new HandleResult.DeadLetter(failure()), publisher); + + assertThat(order).containsExactly("dead-letter published", "acknowledge"); + } + + @Test + @DisplayName("an unconfirmed dead letter requeues rather than acknowledging") + void anUnconfirmedDeadLetterRequeues() { + process( + envelope -> new HandleResult.DeadLetter(failure()), + (envelope, failure) -> CompletableFuture.completedFuture(PublishCompletion.AMBIGUOUS)); + + assertThat(settlement.calls()) + .as("acknowledging loses the message outright; a duplicate is the recoverable half") + .containsExactly("requeue"); + } + + @Test + @DisplayName("a rejected message is discarded") + void aRejectedMessageIsDiscarded() { + process(envelope -> new HandleResult.Reject(failure()), confirmedDeadLetters()); + + assertThat(settlement.calls()).containsExactly("discard"); + } + + @Test + @DisplayName("a handler returning nothing is retried") + void aHandlerReturningNothingIsRetried() { + process(envelope -> null, confirmedDeadLetters()); + + assertThat(settlement.calls()).containsExactly("requeue"); + } + + private void process( + java.util.function.Function, HandleResult> handler, + DefaultDeliveryProcessor.DeadLetterPublisher deadLetters) { + new DefaultDeliveryProcessor(handler, deadLetters, Duration.ofSeconds(5)) + .process(new TransportDelivery(envelope(), metadata(), settlement)) + .toCompletableFuture() + .join(); + } + + private static DefaultDeliveryProcessor.DeadLetterPublisher confirmedDeadLetters() { + return (envelope, failure) -> CompletableFuture.completedFuture(PublishCompletion.CONFIRMED); + } + + private static FailureDescriptor failure() { + return new FailureDescriptor( + FailureCategory.PERMANENT_BUSINESS, + "UNPROCESSABLE", + false, + "cannot be applied", + Optional.empty()); + } + + private static DeliveryMetadata metadata() { + return new DeliveryMetadata( + new dev.caskeleton.messaging.api.destination.DestinationName("order-events"), + 1, + false, + Optional.empty(), + Optional.empty(), + Optional.of("order-projection"), + Instant.parse("2026-08-14T00:00:00Z")); + } + + private static MessageEnvelope envelope() { + return new MessageEnvelope<>( + MessageId.newId(), + new MessageType("order.created"), + new SchemaVersion(1), + Instant.parse("2026-08-14T00:00:00Z"), + Optional.empty(), + new ProducerId("orders"), + Optional.empty(), + Optional.empty(), + ContentType.JSON, + Optional.empty(), + Optional.empty(), + Optional.empty(), + TraceContext.none(), + MessageHeaders.empty(), + new EncodedMessage( + "{}".getBytes(StandardCharsets.UTF_8), ContentType.JSON, Optional.empty())); + } + + /** Records which terminal call the processor made, and in what order. */ + private static final class RecordingSettlement implements TransportSettlement { + + private final List calls = new ArrayList<>(); + + private List shared; + + private void recordInto(List order) { + this.shared = order; + } + + @Override + public CompletionStage acknowledge() { + return record("acknowledge"); + } + + @Override + public CompletionStage requeue(Duration delay) { + return record("requeue"); + } + + @Override + public CompletionStage discard() { + return record("discard"); + } + + private CompletionStage record(String call) { + calls.add(call); + if (shared != null) { + shared.add(call); + } + return CompletableFuture.completedFuture(SettlementResult.settled()); + } + + private List calls() { + return calls; + } + } +} diff --git a/messaging/messaging-runtime-core/src/test/java/dev/caskeleton/messaging/runtime/DefaultMessagePublisherTest.java b/messaging/messaging-runtime-core/src/test/java/dev/caskeleton/messaging/runtime/DefaultMessagePublisherTest.java new file mode 100644 index 0000000..98c0a3d --- /dev/null +++ b/messaging/messaging-runtime-core/src/test/java/dev/caskeleton/messaging/runtime/DefaultMessagePublisherTest.java @@ -0,0 +1,469 @@ +package dev.caskeleton.messaging.runtime; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import dev.caskeleton.messaging.api.ContentType; +import dev.caskeleton.messaging.api.MessageEnvelope; +import dev.caskeleton.messaging.api.MessageId; +import dev.caskeleton.messaging.api.MessageType; +import dev.caskeleton.messaging.api.ProducerId; +import dev.caskeleton.messaging.api.SchemaVersion; +import dev.caskeleton.messaging.api.TraceContext; +import dev.caskeleton.messaging.api.destination.DestinationCapabilities; +import dev.caskeleton.messaging.api.destination.DestinationName; +import dev.caskeleton.messaging.api.destination.MessageDestination; +import dev.caskeleton.messaging.api.error.MessagingConfigurationException; +import dev.caskeleton.messaging.api.header.MessageHeaders; +import dev.caskeleton.messaging.api.publish.ConfirmationLevel; +import dev.caskeleton.messaging.api.publish.PublishCompletion; +import dev.caskeleton.messaging.api.publish.PublishEvidence; +import dev.caskeleton.messaging.api.publish.PublishOptions; +import dev.caskeleton.messaging.api.publish.PublishResult; +import dev.caskeleton.messaging.api.publish.RoutingOutcome; +import dev.caskeleton.messaging.policy.InFlightLimiter; +import dev.caskeleton.messaging.policy.MessagingAdmissionController; +import dev.caskeleton.messaging.policy.PayloadLimitGuard; +import dev.caskeleton.messaging.policy.PayloadPolicy; +import dev.caskeleton.messaging.schema.EncodedMessage; +import dev.caskeleton.messaging.schema.MessageCodec; +import dev.caskeleton.messaging.schema.MessageCodecRegistry; +import dev.caskeleton.messaging.security.DestinationAccessPolicy; +import dev.caskeleton.messaging.transport.MessagingRuntime; +import dev.caskeleton.messaging.transport.MessagingRuntimeLease; +import dev.caskeleton.messaging.transport.MessagingRuntimeRegistry; +import dev.caskeleton.messaging.transport.MessagingTransport; +import dev.caskeleton.messaging.transport.TransportConsumerRegistration; +import dev.caskeleton.messaging.transport.TransportConsumerSpec; +import dev.caskeleton.messaging.transport.TransportPublishRequest; +import dev.caskeleton.messaging.transport.TransportPublishResult; +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.time.Instant; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CompletionStage; +import java.util.concurrent.atomic.AtomicInteger; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** + * The publish path, end to end, without an application-supplied publisher. + * + *

    There was no implementation of {@code MessagePublisher} anywhere in the new platform, so the + * admission controller, the access policy and the runtime registry were beans that no publish ever + * consulted. These tests run the real pipeline against a fake transport. + */ +class DefaultMessagePublisherTest { + + private static final DestinationName ORDERS = new DestinationName("order-events"); + + private final RecordingTransport transport = new RecordingTransport(); + + private final MessagingAdmissionController admission = + new MessagingAdmissionController( + new PayloadLimitGuard(PayloadPolicy.defaults()), + new InFlightLimiter(4, Duration.ofSeconds(1))); + + @Test + @DisplayName("a publish goes through resolution, encoding, admission, lease and transport") + void aPublishRunsTheWholePipeline() { + PublishResult result = + publisher(allowOrders()) + .publish( + new MessageDestination<>(ORDERS, new MessageType("order.created"), String.class), + envelope(), + PublishOptions.defaults()) + .toCompletableFuture() + .join(); + + assertThat(result.completion()).isEqualTo(PublishCompletion.CONFIRMED); + assertThat(transport.published()).isEqualTo(1); + assertThat(admission.inFlight()) + .as("a permit that leaks per publish is a limiter that stops accepting after four") + .isZero(); + } + + @Test + @DisplayName("an unregistered destination is rejected before anything is encoded") + void anUnregisteredDestinationIsRejected() { + DefaultMessagePublisher publisher = + new DefaultMessagePublisher( + DestinationProfileRegistry.empty(), + allowOrders(), + new SingleCodecRegistry(), + admission, + new FixedRuntimeRegistry(transport), + transport); + + PublishResult result = + publisher + .publish( + new MessageDestination<>(ORDERS, new MessageType("order.created"), String.class), + envelope(), + PublishOptions.defaults()) + .toCompletableFuture() + .join(); + + assertThat(result.completion()).isEqualTo(PublishCompletion.REJECTED); + assertThat(transport.published()).isZero(); + } + + @Test + @DisplayName("a forbidden destination is rejected before the payload is encoded") + void aForbiddenDestinationNeverEncodes() { + SingleCodecRegistry codecs = new SingleCodecRegistry(); + DefaultMessagePublisher publisher = + new DefaultMessagePublisher( + registry(), + DestinationAccessPolicy.denyAll(), + codecs, + admission, + new FixedRuntimeRegistry(transport), + transport); + + PublishResult result = + publisher + .publish( + new MessageDestination<>(ORDERS, new MessageType("order.created"), String.class), + envelope(), + PublishOptions.defaults()) + .toCompletableFuture() + .join(); + + assertThat(result.completion()).isEqualTo(PublishCompletion.REJECTED); + assertThat(codecs.encodes()) + .as("the encoded bytes are what a claim check or a log would then be holding") + .isZero(); + assertThat(admission.inFlight()).isZero(); + } + + @Test + @DisplayName("a transport failure is ambiguous, not rejected") + void aTransportFailureIsAmbiguous() { + transport.failWith(new IllegalStateException("connection reset")); + + PublishResult result = + publisher(allowOrders()) + .publish( + new MessageDestination<>(ORDERS, new MessageType("order.created"), String.class), + envelope(), + PublishOptions.defaults()) + .toCompletableFuture() + .join(); + + assertThat(result.completion()) + .as("the request was on the wire, so the broker may hold it") + .isEqualTo(PublishCompletion.AMBIGUOUS); + assertThat(result.failure().orElseThrow().sanitizedMessage()) + .as("a driver message can carry a routing key or a payload fragment") + .isEqualTo("IllegalStateException"); + assertThat(admission.inFlight()).isZero(); + } + + @Test + @DisplayName("an unavailable runtime releases the permit it took") + void anUnavailableRuntimeReleasesThePermit() { + DefaultMessagePublisher publisher = + new DefaultMessagePublisher( + registry(), + allowOrders(), + new SingleCodecRegistry(), + admission, + new FailingRuntimeRegistry(), + transport); + + PublishResult result = + publisher + .publish( + new MessageDestination<>(ORDERS, new MessageType("order.created"), String.class), + envelope(), + PublishOptions.defaults()) + .toCompletableFuture() + .join(); + + assertThat(result.completion()).isEqualTo(PublishCompletion.REJECTED); + assertThat(admission.inFlight()).isZero(); + } + + @Test + @DisplayName("an unregistered destination names itself") + void anUnregisteredDestinationNamesItself() { + assertThatThrownBy(() -> DestinationProfileRegistry.empty().require(ORDERS)) + .isInstanceOf(MessagingConfigurationException.class) + .hasMessageContaining("order-events"); + } + + @Test + @DisplayName("every publish outcome reaches the observation") + void everyPublishOutcomeIsObserved() { + java.util.List outcomes = new java.util.ArrayList<>(); + DefaultMessagePublisher publisher = + new DefaultMessagePublisher( + registry(), + allowOrders(), + new SingleCodecRegistry(), + admission, + new FixedRuntimeRegistry(transport), + transport, + System::nanoTime, + new dev.caskeleton.messaging.observation.MessagingObservation() { + @Override + public void recordPublish( + dev.caskeleton.messaging.observation.MessagingTags tags, Duration elapsed) { + outcomes.add(tags.asMap().get("outcome")); + } + + @Override + public void recordDelivery( + dev.caskeleton.messaging.observation.MessagingTags tags, + Duration elapsed, + int attempt) {} + + @Override + public void recordSettlement( + dev.caskeleton.messaging.observation.MessagingTags tags) {} + + @Override + public void recordBacklog( + dev.caskeleton.messaging.observation.MessagingTags tags, long messages) {} + + @Override + public void recordDiagnostics( + dev.caskeleton.messaging.observation.MessagingTags tags, + Map diagnostics) {} + }); + + publisher + .publish( + new MessageDestination<>(ORDERS, new MessageType("order.created"), String.class), + envelope(), + PublishOptions.defaults()) + .toCompletableFuture() + .join(); + + assertThat(outcomes) + .as("the observation bean existed and no publish path ever called it") + .containsExactly("confirmed"); + } + + @Test + @DisplayName("a deduplication request a broker cannot honour is refused") + void anUnsupportedDeduplicationRequestIsRefused() { + PublishResult result = + publisher(allowOrders()) + .publish( + new MessageDestination<>(ORDERS, new MessageType("order.created"), String.class), + envelope(), + new PublishOptions( + java.time.Duration.ofSeconds(5), + dev.caskeleton.messaging.api.destination.ConfirmationRequirement + .REPLICATION_OR_PERSISTENCE_ACK, + java.util.Optional.of( + new dev.caskeleton.messaging.api.publish.PublishDeduplication( + "order-1", java.time.Duration.ofMinutes(5))), + Map.of())) + .toCompletableFuture() + .join(); + + assertThat(result.completion()) + .as("accepting it silently is how a caller skips the idempotency it would have written") + .isEqualTo(PublishCompletion.REJECTED); + assertThat(transport.published()).isZero(); + } + + @Test + @DisplayName("M3 broker hints are refused on the M1 path") + void brokerHintsAreRefused() { + PublishResult result = + publisher(allowOrders()) + .publish( + new MessageDestination<>(ORDERS, new MessageType("order.created"), String.class), + envelope(), + new PublishOptions( + java.time.Duration.ofSeconds(5), + dev.caskeleton.messaging.api.destination.ConfirmationRequirement + .REPLICATION_OR_PERSISTENCE_ACK, + java.util.Optional.empty(), + Map.of("linger.ms", "50"))) + .toCompletableFuture() + .join(); + + assertThat(result.completion()) + .as("isM1Compatible existed and nothing ever called it") + .isEqualTo(PublishCompletion.REJECTED); + } + + private static DestinationAccessPolicy allowOrders() { + return new DestinationAccessPolicy( + java.util.Set.of(ORDERS), java.util.Set.of(), java.util.Set.of()); + } + + private DefaultMessagePublisher publisher(DestinationAccessPolicy access) { + return new DefaultMessagePublisher( + registry(), + access, + new SingleCodecRegistry(), + admission, + new FixedRuntimeRegistry(transport), + transport); + } + + private static DestinationProfileRegistry registry() { + return DestinationProfileRegistry.of(Map.of(ORDERS, RuntimeFixtures.profile(ORDERS))); + } + + private static MessageEnvelope envelope() { + return new MessageEnvelope<>( + MessageId.newId(), + new MessageType("order.created"), + new SchemaVersion(1), + Instant.parse("2026-08-14T00:00:00Z"), + Optional.empty(), + new ProducerId("orders"), + Optional.empty(), + Optional.empty(), + ContentType.JSON, + Optional.empty(), + Optional.empty(), + Optional.empty(), + TraceContext.none(), + MessageHeaders.empty(), + "{\"orderId\":\"o-1\"}"); + } + + /** A codec that counts how often it was asked to encode. */ + private static final class SingleCodecRegistry implements MessageCodecRegistry { + + private final AtomicInteger encodes = new AtomicInteger(); + + @Override + public Optional find(ContentType contentType) { + return Optional.of(codec()); + } + + @Override + public MessageCodec defaultCodec() { + return codec(); + } + + private MessageCodec codec() { + return new MessageCodec() { + @Override + public EncodedMessage encode(MessageType type, SchemaVersion version, Object payload) { + encodes.incrementAndGet(); + return new EncodedMessage( + String.valueOf(payload).getBytes(StandardCharsets.UTF_8), + ContentType.JSON, + Optional.empty()); + } + + @Override + public T decode( + MessageType type, SchemaVersion version, byte[] encoded, Class payloadType) { + return payloadType.cast(new String(encoded, StandardCharsets.UTF_8)); + } + + @Override + public ContentType contentType() { + return ContentType.JSON; + } + }; + } + + private int encodes() { + return encodes.get(); + } + } + + /** A transport that records what reached it. */ + private static final class RecordingTransport implements MessagingTransport { + + private final AtomicInteger published = new AtomicInteger(); + + private RuntimeException failure; + + private void failWith(RuntimeException failure) { + this.failure = failure; + } + + @Override + public CompletionStage publish(TransportPublishRequest request) { + if (failure != null) { + return CompletableFuture.failedFuture(failure); + } + published.incrementAndGet(); + return CompletableFuture.completedFuture( + new TransportPublishResult( + new PublishResult( + PublishCompletion.CONFIRMED, + PublishEvidence.confirmed(ConfirmationLevel.BROKER_ACK), + RoutingOutcome.ROUTED, + Optional.empty(), + 1, + Duration.ZERO, + Optional.empty()))); + } + + @Override + public TransportConsumerRegistration register(TransportConsumerSpec spec) { + throw new UnsupportedOperationException("not used by these tests"); + } + + @Override + public DestinationCapabilities capabilities(DestinationName destination) { + throw new UnsupportedOperationException("not used by these tests"); + } + + @Override + public String brokerName() { + return "fake"; + } + + @Override + public long generation() { + return 1; + } + + @Override + public void close() {} + + private int published() { + return published.get(); + } + } + + /** A registry with no installed generation, which is what a rotation gap looks like. */ + private static final class FailingRuntimeRegistry implements MessagingRuntimeRegistry { + + @Override + public void install(MessagingRuntime runtime) {} + + @Override + public MessagingRuntimeLease acquire(String brokerName) { + throw new IllegalStateException("no generation installed"); + } + } + + /** A registry that always hands out a lease on the same transport. */ + private record FixedRuntimeRegistry(MessagingTransport transport) + implements MessagingRuntimeRegistry { + + @Override + public void install(MessagingRuntime runtime) {} + + @Override + public MessagingRuntimeLease acquire(String brokerName) { + return new MessagingRuntimeLease() { + @Override + public MessagingRuntime runtime() { + throw new UnsupportedOperationException("not used by these tests"); + } + + @Override + public void close() {} + }; + } + } +} diff --git a/messaging/messaging-runtime-core/src/test/java/dev/caskeleton/messaging/runtime/RuntimeFixtures.java b/messaging/messaging-runtime-core/src/test/java/dev/caskeleton/messaging/runtime/RuntimeFixtures.java new file mode 100644 index 0000000..6509b88 --- /dev/null +++ b/messaging/messaging-runtime-core/src/test/java/dev/caskeleton/messaging/runtime/RuntimeFixtures.java @@ -0,0 +1,44 @@ +package dev.caskeleton.messaging.runtime; + +import dev.caskeleton.messaging.api.ContentType; +import dev.caskeleton.messaging.api.MessageType; +import dev.caskeleton.messaging.api.destination.DestinationName; +import dev.caskeleton.messaging.policy.CapabilityTier; +import dev.caskeleton.messaging.policy.ConsumerPolicy; +import dev.caskeleton.messaging.policy.DeadLetterPolicy; +import dev.caskeleton.messaging.policy.DestinationProfile; +import dev.caskeleton.messaging.policy.PayloadPolicy; +import dev.caskeleton.messaging.policy.PhysicalDestination; +import dev.caskeleton.messaging.policy.ProducerPolicy; +import dev.caskeleton.messaging.schema.SchemaCompatibility; +import java.util.Set; + +/** One registered destination, so the pipeline has something real to resolve. */ +final class RuntimeFixtures { + + private RuntimeFixtures() {} + + static DestinationProfile profile(DestinationName name) { + return new DestinationProfile( + name, + "fake", + dev.caskeleton.messaging.api.destination.DestinationKind.EVENT_STREAM, + PhysicalDestination.kafkaTopic("order.events.v1"), + new dev.caskeleton.messaging.policy.SchemaPolicy( + ContentType.JSON, + SchemaCompatibility.BACKWARD_TRANSITIVE, + Set.of(new MessageType("order.created"))), + dev.caskeleton.messaging.api.delivery.DeliveryGuarantee.AT_LEAST_ONCE, + dev.caskeleton.messaging.api.delivery.OrderingScope.NONE, + dev.caskeleton.messaging.api.delivery.ExternalSideEffectGuarantee.IDEMPOTENCY_REQUIRED, + ProducerPolicy.defaults(), + ConsumerPolicy.defaults("order-projection"), + dev.caskeleton.messaging.policy.RetryPolicy.none(), + DeadLetterPolicy.to(new DestinationName("order-events-dlq")), + PayloadPolicy.defaults(), + CapabilityTier.M1, + false, + false, + false); + } +} diff --git a/messaging/messaging-schema-api/src/main/java/dev/caskeleton/messaging/schema/BoundedByteSink.java b/messaging/messaging-schema-api/src/main/java/dev/caskeleton/messaging/schema/BoundedByteSink.java new file mode 100644 index 0000000..fb9f0d6 --- /dev/null +++ b/messaging/messaging-schema-api/src/main/java/dev/caskeleton/messaging/schema/BoundedByteSink.java @@ -0,0 +1,114 @@ +package dev.caskeleton.messaging.schema; + +import dev.caskeleton.messaging.api.error.MessageTooLargeException; +import java.io.ByteArrayOutputStream; +import java.io.OutputStream; +import java.util.Objects; + +/** + * A byte sink that refuses the byte after the limit instead of the buffer after the fact. + * + *

    Every codec here used to serialize into an unbounded buffer and compare {@code bytes.length} + * to the configured maximum afterwards. That makes the maximum a reporting threshold rather than an + * allocation bound: a payload whose graph expands to hundreds of megabytes exhausts the heap while + * being written, and the check that would have rejected it never runs. Under a broker consumer that + * is a process-wide outage caused by one message. + * + *

    This sink fails on the write that would cross the limit, so peak allocation is bounded by the + * limit itself plus whatever the encoder buffers. The exception is the same {@link + * MessageTooLargeException} the after-the-fact check raised, carrying the codec's own error code, + * so callers and evidence records see no behavioural difference for payloads that merely exceed the + * bound. + * + *

    Not thread-safe, and not meant to be: an instance belongs to a single encode call. + */ +public final class BoundedByteSink extends OutputStream { + + private final int maxBytes; + private final String errorCode; + private final ByteArrayOutputStream buffer; + private int written; + + private BoundedByteSink(int maxBytes, String errorCode) { + if (maxBytes < 1) { + throw new IllegalArgumentException("maxBytes must be positive"); + } + this.maxBytes = maxBytes; + this.errorCode = Objects.requireNonNull(errorCode, "errorCode must not be null"); + // Sized to the limit only when the limit is small: a 1 GiB bound must not pre-allocate 1 GiB. + this.buffer = new ByteArrayOutputStream(Math.min(maxBytes, 8_192)); + } + + /** + * Creates a sink that refuses to accumulate more than {@code maxBytes}. + * + * @param maxBytes the inclusive byte limit + * @param errorCode the code the codec reports for an oversized payload + * @return a bounded sink + */ + public static BoundedByteSink of(int maxBytes, String errorCode) { + return new BoundedByteSink(maxBytes, errorCode); + } + + @Override + public void write(int b) { + reserve(1); + buffer.write(b); + } + + @Override + public void write(byte[] bytes, int offset, int length) { + Objects.checkFromIndexSize(offset, length, bytes.length); + reserve(length); + buffer.write(bytes, offset, length); + } + + /** + * Checks a size the caller can compute before serializing, without consuming the budget. + * + *

    Protobuf knows its serialized size exactly, so the whole encode can be refused before the + * first byte is written. The subsequent writes are still bounded — this is a cheaper refusal, not + * a replacement for the bound. + * + * @param size the number of bytes the caller is about to produce + */ + public void requireFits(int size) { + if (size < 0) { + throw new IllegalArgumentException("size must not be negative"); + } + refuseIfBeyondLimit(size); + } + + private void reserve(int size) { + refuseIfBeyondLimit(size); + written += size; + } + + private void refuseIfBeyondLimit(int size) { + if (size > maxBytes - written) { + throw new MessageTooLargeException( + errorCode, + ("the encoded payload exceeds the %d byte limit; it was refused at byte %d rather than" + + " buffered in full") + .formatted(maxBytes, written + size)); + } + } + + /** + * The bytes accumulated so far. + * + * @return a copy of the buffered bytes + */ + public byte[] toByteArray() { + return buffer.toByteArray(); + } + + /** + * How many bytes have been admitted. + * + * @return the running total + */ + public int written() { + return written; + } +} diff --git a/messaging/messaging-schema-api/src/main/java/dev/caskeleton/messaging/schema/MessageContractKey.java b/messaging/messaging-schema-api/src/main/java/dev/caskeleton/messaging/schema/MessageContractKey.java new file mode 100644 index 0000000..2605a48 --- /dev/null +++ b/messaging/messaging-schema-api/src/main/java/dev/caskeleton/messaging/schema/MessageContractKey.java @@ -0,0 +1,49 @@ +package dev.caskeleton.messaging.schema; + +import dev.caskeleton.messaging.api.MessageType; +import dev.caskeleton.messaging.api.SchemaVersion; +import java.util.Objects; + +/** + * The identity a codec registry is keyed on: a message type at a specific schema version. + * + *

    Keying on the message type alone is what let an unregistered version decode. The version + * travels in the envelope and in {@link SchemaReference}, so a consumer receiving {@code + * order.created v999} would look up {@code order.created}, find the v1 class or parser, decode + * against it, and then keep the v999 label on the result. Nothing failed, and every downstream + * compatibility gate and audit record then described a version that was never registered. + * + *

    With the version in the key that lookup misses, which is the only honest outcome: the platform + * has no contract for that version and must say so rather than guess one. + * + * @param type the catalog message type + * @param version the schema version the contract is registered for + */ +public record MessageContractKey(MessageType type, SchemaVersion version) { + + /** Validates both components. */ + public MessageContractKey { + Objects.requireNonNull(type, "messageType must not be null"); + Objects.requireNonNull(version, "schemaVersion must not be null"); + } + + /** + * Creates a key from raw components. + * + * @param type the catalog message type + * @param version the schema version + * @return the key + */ + public static MessageContractKey of(String type, int version) { + return new MessageContractKey(new MessageType(type), new SchemaVersion(version)); + } + + /** + * Renders the key the way error messages and audit records name a contract. + * + * @return for example {@code order.created v1} + */ + public String describe() { + return type.value() + " v" + version.value(); + } +} diff --git a/messaging/messaging-schema-api/src/test/java/dev/caskeleton/messaging/schema/BoundedByteSinkTest.java b/messaging/messaging-schema-api/src/test/java/dev/caskeleton/messaging/schema/BoundedByteSinkTest.java new file mode 100644 index 0000000..894ffb6 --- /dev/null +++ b/messaging/messaging-schema-api/src/test/java/dev/caskeleton/messaging/schema/BoundedByteSinkTest.java @@ -0,0 +1,87 @@ +package dev.caskeleton.messaging.schema; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatCode; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import dev.caskeleton.messaging.api.error.MessageTooLargeException; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** + * The boundary the codecs delegate to. + * + *

    Each codec used to serialize into an unbounded buffer and compare the finished length to the + * limit. That check reports an oversized payload; it does not prevent one. The difference only + * shows up for a payload whose serialized form is orders of magnitude larger than the bound, which + * is precisely the case that matters — the consumer is out of heap before the comparison runs. + */ +class BoundedByteSinkTest { + + private static final int MAX = 64; + + @Test + @DisplayName("one byte under, exactly at, and one byte over the limit") + void theLimitIsInclusiveAndTheNextByteIsRefused() { + assertThatCode(() -> fill(MAX - 1)).doesNotThrowAnyException(); + assertThatCode(() -> fill(MAX)).doesNotThrowAnyException(); + + assertThatThrownBy(() -> fill(MAX + 1)) + .isInstanceOf(MessageTooLargeException.class) + .hasMessageContaining("refused at byte 65"); + } + + @Test + @DisplayName("a much larger stream stops at the limit rather than after allocation") + void aMuchLargerStreamStopsAtTheLimit() { + BoundedByteSink sink = BoundedByteSink.of(MAX, "PAYLOAD_TOO_LARGE"); + + assertThatThrownBy( + () -> { + // Ten mebibytes offered one kibibyte at a time. A buffer-then-check codec would have + // held all of it; this refuses the chunk that crosses the bound. + for (int chunk = 0; chunk < 10_240; chunk++) { + sink.write(new byte[1_024], 0, 1_024); + } + }) + .isInstanceOf(MessageTooLargeException.class); + + assertThat(sink.written()) + .as("nothing beyond the bound was ever admitted") + .isLessThanOrEqualTo(MAX); + assertThat(sink.toByteArray()).isEmpty(); + } + + @Test + @DisplayName("a pre-flight size check does not consume the budget") + void aPreFlightCheckDoesNotConsumeTheBudget() { + BoundedByteSink sink = BoundedByteSink.of(MAX, "PAYLOAD_TOO_LARGE"); + + sink.requireFits(MAX); + + assertThat(sink.written()).isZero(); + assertThatCode(() -> sink.write(new byte[MAX], 0, MAX)).doesNotThrowAnyException(); + } + + @Test + @DisplayName("a pre-flight check refuses a size that cannot fit") + void aPreFlightCheckRefusesASizeThatCannotFit() { + BoundedByteSink sink = BoundedByteSink.of(MAX, "AVRO_PAYLOAD_TOO_LARGE"); + + assertThatThrownBy(() -> sink.requireFits(MAX + 1)) + .isInstanceOf(MessageTooLargeException.class) + .asInstanceOf( + org.assertj.core.api.InstanceOfAssertFactories.type(MessageTooLargeException.class)) + .extracting(failure -> failure.failure().code()) + .as("the sink reports the codec's own code, not a generic one") + .isEqualTo("AVRO_PAYLOAD_TOO_LARGE"); + } + + private static byte[] fill(int size) { + BoundedByteSink sink = BoundedByteSink.of(MAX, "PAYLOAD_TOO_LARGE"); + for (int index = 0; index < size; index++) { + sink.write('a'); + } + return sink.toByteArray(); + } +} diff --git a/messaging/messaging-schema-avro/src/main/java/dev/caskeleton/messaging/schema/avro/AvroMessageCodec.java b/messaging/messaging-schema-avro/src/main/java/dev/caskeleton/messaging/schema/avro/AvroMessageCodec.java index 3f2d610..fd2c6da 100644 --- a/messaging/messaging-schema-avro/src/main/java/dev/caskeleton/messaging/schema/avro/AvroMessageCodec.java +++ b/messaging/messaging-schema-avro/src/main/java/dev/caskeleton/messaging/schema/avro/AvroMessageCodec.java @@ -6,11 +6,13 @@ import dev.caskeleton.messaging.api.SchemaVersion; import dev.caskeleton.messaging.api.error.MessageSerializationException; import dev.caskeleton.messaging.api.error.MessageTooLargeException; import dev.caskeleton.messaging.api.error.MessageValidationException; +import dev.caskeleton.messaging.schema.BoundedByteSink; import dev.caskeleton.messaging.schema.EncodedMessage; import dev.caskeleton.messaging.schema.MessageCodec; +import dev.caskeleton.messaging.schema.MessageContractKey; import dev.caskeleton.messaging.schema.SchemaReference; -import java.io.ByteArrayOutputStream; import java.io.IOException; +import java.util.HashMap; import java.util.Map; import java.util.Objects; import java.util.Optional; @@ -43,7 +45,7 @@ public final class AvroMessageCodec implements MessageCodec { private static final int DEFAULT_MAX_BYTES = 1_048_576; - private final Map> schemas; + private final Map schemas; private final int maxBytes; /** @@ -66,10 +68,34 @@ public final class AvroMessageCodec implements MessageCodec { if (maxBytes < 1) { throw new IllegalArgumentException("maxBytes must be positive"); } - this.schemas = Map.copyOf(schemas); + this.schemas = flatten(schemas); this.maxBytes = maxBytes; } + /** + * Copies the caller's nested registry into a flat, version-keyed, immutable map. + * + *

    {@code Map.copyOf} on the outer map is a shallow copy: every inner {@code Map} stayed the caller's own object, so a caller that kept a reference could add, replace, + * or remove a schema version after construction and the codec would silently start encoding + * against it. Flattening to {@code (type, version)} keys copies both levels and makes the version + * part of the identity the lookup uses rather than a second hop. + */ + private static Map flatten( + Map> registry) { + Map flattened = new HashMap<>(); + registry.forEach( + (type, byVersion) -> { + Objects.requireNonNull(byVersion, "schemas for " + type.value() + " must not be null"); + byVersion.forEach( + (version, schema) -> + flattened.put( + new MessageContractKey(type, version), + Objects.requireNonNull(schema, "schema must not be null"))); + }); + return Map.copyOf(flattened); + } + @Override public ContentType contentType() { return ContentType.AVRO; @@ -93,26 +119,27 @@ public final class AvroMessageCodec implements MessageCodec { .formatted(type.value(), version.value())); } - ByteArrayOutputStream sink = new ByteArrayOutputStream(); - BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(sink, null); + // A direct encoder, not the buffering one: the buffering encoder holds bytes back until flush, + // which would let a large record allocate freely before the sink ever sees a write. Direct + // encoding makes the bound apply to the record as it is written. + BoundedByteSink sink = BoundedByteSink.of(maxBytes, "AVRO_PAYLOAD_TOO_LARGE"); + BinaryEncoder encoder = EncoderFactory.get().directBinaryEncoder(sink, null); DatumWriter writer = new GenericDatumWriter<>(schema); try { writer.write(record, encoder); encoder.flush(); } catch (IOException | RuntimeException failure) { + if (failure instanceof MessageTooLargeException tooLarge) { + throw tooLarge; + } throw new MessageSerializationException( "AVRO_ENCODE_FAILED", "the payload could not be encoded as Avro", failure); } - byte[] bytes = sink.toByteArray(); - if (bytes.length > maxBytes) { - throw new MessageTooLargeException( - "AVRO_PAYLOAD_TOO_LARGE", - "the encoded payload of %d bytes exceeds the %d byte limit" - .formatted(bytes.length, maxBytes)); - } return new EncodedMessage( - bytes, ContentType.AVRO, Optional.of(SchemaReference.of(type.value(), version))); + sink.toByteArray(), + ContentType.AVRO, + Optional.of(SchemaReference.of(type.value(), version))); } @Override @@ -120,12 +147,7 @@ public final class AvroMessageCodec implements MessageCodec { MessageType type, SchemaVersion version, byte[] encoded, Class payloadType) { Objects.requireNonNull(encoded, "encoded must not be null"); Objects.requireNonNull(payloadType, "payloadType must not be null"); - if (encoded.length > maxBytes) { - throw new MessageTooLargeException( - "AVRO_PAYLOAD_TOO_LARGE", - "the encoded payload of %d bytes exceeds the %d byte limit" - .formatted(encoded.length, maxBytes)); - } + requireWithinLimit(encoded.length); Schema writerSchema = schemaFor(type, version); if (!payloadType.isAssignableFrom(GenericRecord.class) @@ -147,6 +169,14 @@ public final class AvroMessageCodec implements MessageCodec { } } + private void requireWithinLimit(int length) { + if (length > maxBytes) { + throw new MessageTooLargeException( + "AVRO_PAYLOAD_TOO_LARGE", + "the encoded payload of %d bytes exceeds the %d byte limit".formatted(length, maxBytes)); + } + } + /** * Decodes using a reader schema that differs from the writer schema. * @@ -163,6 +193,10 @@ public final class AvroMessageCodec implements MessageCodec { public GenericRecord decodeEvolved( MessageType type, SchemaVersion writerVersion, SchemaVersion readerVersion, byte[] encoded) { Objects.requireNonNull(encoded, "encoded must not be null"); + // The same bound the ordinary decode applies. It was missing here, so the evolution path — the + // one a consumer takes for every message written by a newer producer — accepted input of any + // size. + requireWithinLimit(encoded.length); Schema writerSchema = schemaFor(type, writerVersion); Schema readerSchema = schemaFor(type, readerVersion); @@ -182,14 +216,14 @@ public final class AvroMessageCodec implements MessageCodec { private Schema schemaFor(MessageType type, SchemaVersion version) { Objects.requireNonNull(type, "type must not be null"); Objects.requireNonNull(version, "version must not be null"); - Map versions = schemas.get(type); - if (versions == null) { - throw new MessageValidationException( - "AVRO_TYPE_NOT_REGISTERED", - "message type %s is not in the Avro registry".formatted(type.value())); - } - Schema schema = versions.get(version); + Schema schema = schemas.get(new MessageContractKey(type, version)); if (schema == null) { + boolean typeIsKnown = schemas.keySet().stream().anyMatch(known -> known.type().equals(type)); + if (!typeIsKnown) { + throw new MessageValidationException( + "AVRO_TYPE_NOT_REGISTERED", + "message type %s is not in the Avro registry".formatted(type.value())); + } throw new MessageValidationException( "AVRO_VERSION_NOT_REGISTERED", "schema version %d of %s is not in the Avro registry" diff --git a/messaging/messaging-schema-avro/src/test/java/dev/caskeleton/messaging/schema/avro/AvroRegistryBoundsTest.java b/messaging/messaging-schema-avro/src/test/java/dev/caskeleton/messaging/schema/avro/AvroRegistryBoundsTest.java new file mode 100644 index 0000000..774be67 --- /dev/null +++ b/messaging/messaging-schema-avro/src/test/java/dev/caskeleton/messaging/schema/avro/AvroRegistryBoundsTest.java @@ -0,0 +1,140 @@ +package dev.caskeleton.messaging.schema.avro; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatCode; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import dev.caskeleton.messaging.api.MessageType; +import dev.caskeleton.messaging.api.SchemaVersion; +import dev.caskeleton.messaging.api.error.MessageTooLargeException; +import dev.caskeleton.messaging.api.error.MessageValidationException; +import java.util.HashMap; +import java.util.Map; +import org.apache.avro.Schema; +import org.apache.avro.generic.GenericData; +import org.apache.avro.generic.GenericRecord; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** + * Two holes on the Avro side: the registry the caller could still edit, and the evolution path with + * no size bound. + * + *

    {@code Map.copyOf} on a map of maps copies the outer level only. Every inner {@code + * Map} remained the caller's object, so a caller holding a reference could + * replace a registered schema after construction — and Avro binary carries no schema of its own, so + * decoding with a swapped schema does not fail, it produces plausible garbage. + * + *

    {@code decodeEvolved} is the path every consumer takes when a producer moves ahead a version. + * It checked no size at all, so the bound that made the ordinary decode safe simply did not apply + * to the case it mattered most for. + */ +class AvroRegistryBoundsTest { + + private static final MessageType ORDER_CREATED = new MessageType("order.created"); + private static final SchemaVersion V1 = new SchemaVersion(1); + private static final SchemaVersion V2 = new SchemaVersion(2); + + private static final Schema V1_SCHEMA = + new Schema.Parser() + .parse( + """ + { + "type": "record", + "name": "OrderCreated", + "namespace": "dev.caskeleton.messaging.sample", + "fields": [ + {"name": "orderId", "type": "string"}, + {"name": "note", "type": "string", "default": ""} + ] + } + """); + + private static final Schema V2_SCHEMA = + new Schema.Parser() + .parse( + """ + { + "type": "record", + "name": "OrderCreated", + "namespace": "dev.caskeleton.messaging.sample", + "fields": [ + {"name": "orderId", "type": "string"}, + {"name": "note", "type": "string", "default": ""}, + {"name": "channel", "type": "string", "default": "WEB"} + ] + } + """); + + @Test + @DisplayName("mutating the map after construction does not change what the codec decodes") + void mutatingTheCallersMapAfterConstructionChangesNothing() { + Map versions = new HashMap<>(Map.of(V1, V1_SCHEMA)); + Map> registry = new HashMap<>(); + registry.put(ORDER_CREATED, versions); + AvroMessageCodec codec = new AvroMessageCodec(registry); + + versions.put(V1, V2_SCHEMA); + versions.put(V2, V2_SCHEMA); + registry.put(new MessageType("order.cancelled"), Map.of(V1, V1_SCHEMA)); + + assertThatThrownBy(() -> codec.encode(ORDER_CREATED, V2, record(V2_SCHEMA))) + .as("a version added after construction is not registered") + .isInstanceOf(MessageValidationException.class); + assertThatThrownBy( + () -> codec.encode(new MessageType("order.cancelled"), V1, record(V1_SCHEMA))) + .isInstanceOf(MessageValidationException.class); + assertThatCode(() -> codec.encode(ORDER_CREATED, V1, record(V1_SCHEMA))) + .as("the schema registered at construction is still the one in force") + .doesNotThrowAnyException(); + } + + @Test + @DisplayName("an oversized payload is refused while encoding, not after") + void anOversizedPayloadIsRefusedWhileEncoding() { + AvroMessageCodec codec = new AvroMessageCodec(Map.of(ORDER_CREATED, Map.of(V1, V1_SCHEMA)), 64); + GenericRecord large = record(V1_SCHEMA); + large.put("note", "n".repeat(4_000_000)); + + assertThatThrownBy(() -> codec.encode(ORDER_CREATED, V1, large)) + .isInstanceOf(MessageTooLargeException.class) + .hasMessageContaining("refused at byte"); + } + + @Test + @DisplayName("the evolution decode applies the same bound as the ordinary decode") + void theEvolutionDecodeAppliesTheSameBound() { + AvroMessageCodec codec = + new AvroMessageCodec(Map.of(ORDER_CREATED, Map.of(V1, V1_SCHEMA, V2, V2_SCHEMA)), 32); + + assertThatThrownBy(() -> codec.decodeEvolved(ORDER_CREATED, V1, V2, new byte[4_096])) + .as("decodeEvolved accepted input of any size") + .isInstanceOf(MessageTooLargeException.class); + } + + @Test + @DisplayName("a payload at exactly the limit still encodes") + void aPayloadAtExactlyTheLimitStillEncodes() { + AvroMessageCodec unbounded = new AvroMessageCodec(Map.of(ORDER_CREATED, Map.of(V1, V1_SCHEMA))); + int exact = unbounded.encode(ORDER_CREATED, V1, record(V1_SCHEMA)).bytes().length; + AvroMessageCodec bounded = + new AvroMessageCodec(Map.of(ORDER_CREATED, Map.of(V1, V1_SCHEMA)), exact); + + assertThat(bounded.encode(ORDER_CREATED, V1, record(V1_SCHEMA)).bytes()).hasSize(exact); + + AvroMessageCodec oneShort = + new AvroMessageCodec(Map.of(ORDER_CREATED, Map.of(V1, V1_SCHEMA)), exact - 1); + assertThatThrownBy(() -> oneShort.encode(ORDER_CREATED, V1, record(V1_SCHEMA))) + .isInstanceOf(MessageTooLargeException.class); + } + + private static GenericRecord record(Schema schema) { + GenericRecord record = new GenericData.Record(schema); + record.put("orderId", "o-1"); + record.put("note", ""); + if (schema.getField("channel") != null) { + record.put("channel", "WEB"); + } + return record; + } +} diff --git a/messaging/messaging-schema-json/src/main/java/dev/caskeleton/messaging/schema/json/JacksonMessageCodec.java b/messaging/messaging-schema-json/src/main/java/dev/caskeleton/messaging/schema/json/JacksonMessageCodec.java index f6294b6..71fe012 100644 --- a/messaging/messaging-schema-json/src/main/java/dev/caskeleton/messaging/schema/json/JacksonMessageCodec.java +++ b/messaging/messaging-schema-json/src/main/java/dev/caskeleton/messaging/schema/json/JacksonMessageCodec.java @@ -6,9 +6,12 @@ import dev.caskeleton.messaging.api.SchemaVersion; import dev.caskeleton.messaging.api.error.MessageSerializationException; import dev.caskeleton.messaging.api.error.MessageTooLargeException; import dev.caskeleton.messaging.api.error.MessageValidationException; +import dev.caskeleton.messaging.schema.BoundedByteSink; import dev.caskeleton.messaging.schema.EncodedMessage; import dev.caskeleton.messaging.schema.MessageCodec; +import dev.caskeleton.messaging.schema.MessageContractKey; import dev.caskeleton.messaging.schema.SchemaReference; +import java.util.List; import java.util.Map; import java.util.Objects; import java.util.Optional; @@ -45,10 +48,10 @@ public final class JacksonMessageCodec implements MessageCodec { private static final int MAX_NUMBER_DIGITS = 1_000; private final ObjectMapper mapper; - private final Map> registry; + private final Map> registry; private final int maxBytes; - private JacksonMessageCodec(Map> registry, int maxBytes) { + private JacksonMessageCodec(Map> registry, int maxBytes) { this.registry = Map.copyOf(registry); this.maxBytes = maxBytes; this.mapper = strictMapper(maxBytes); @@ -57,33 +60,47 @@ public final class JacksonMessageCodec implements MessageCodec { /** * Creates a codec over an explicit message-type registry. * - * @param registry the closed map of message type to payload class + * @param registry the closed map of message contract to payload class * @return a codec with the default one mebibyte limit */ - public static JacksonMessageCodec of(Map> registry) { + public static JacksonMessageCodec of(Map> registry) { return new JacksonMessageCodec(registry, DEFAULT_MAX_BYTES); } /** * Creates a codec over an explicit registry and byte limit. * - * @param registry the closed map of message type to payload class + * @param registry the closed map of message contract to payload class * @param maxBytes the maximum encoded size * @return a codec */ - public static JacksonMessageCodec of(Map> registry, int maxBytes) { + public static JacksonMessageCodec of(Map> registry, int maxBytes) { return new JacksonMessageCodec(registry, maxBytes); } + /** + * Creates a single-entry codec for focused tests, registered at version one. + * + * @param type the catalog message type + * @param payloadType the payload class + * @return a codec registering exactly one contract, {@code type v1} + */ + public static JacksonMessageCodec testingDefault(MessageType type, Class payloadType) { + return testingDefault(type, new SchemaVersion(1), payloadType); + } + /** * Creates a single-entry codec for focused tests. * * @param type the catalog message type + * @param version the schema version to register * @param payloadType the payload class - * @return a codec registering exactly one type + * @return a codec registering exactly one contract */ - public static JacksonMessageCodec testingDefault(MessageType type, Class payloadType) { - return new JacksonMessageCodec(Map.of(type, payloadType), DEFAULT_MAX_BYTES); + public static JacksonMessageCodec testingDefault( + MessageType type, SchemaVersion version, Class payloadType) { + return new JacksonMessageCodec( + Map.of(new MessageContractKey(type, version), payloadType), DEFAULT_MAX_BYTES); } @Override @@ -93,8 +110,7 @@ public final class JacksonMessageCodec implements MessageCodec { @Override public EncodedMessage encode(MessageType type, SchemaVersion version, Object payload) { - Class registered = requireRegistered(type); - Objects.requireNonNull(version, "schemaVersion must not be null"); + Class registered = requireRegistered(type, version); Objects.requireNonNull(payload, "payload must not be null"); if (!registered.isInstance(payload)) { throw new MessageValidationException( @@ -102,27 +118,25 @@ public final class JacksonMessageCodec implements MessageCodec { "payload does not match the registered type for " + type.value()); } - byte[] bytes; + // Streamed into a bounded sink: Jackson writes incrementally, so a payload whose serialized + // form is far larger than the limit stops at the limit instead of after the whole graph has + // been rendered into a buffer nobody bounded. + BoundedByteSink sink = BoundedByteSink.of(maxBytes, "PAYLOAD_TOO_LARGE"); try { - bytes = mapper.writeValueAsBytes(payload); + mapper.writeValue(sink, payload); } catch (JacksonException exception) { - throw new MessageSerializationException( - "JSON_ENCODE_FAILED", "payload could not be encoded as JSON", exception); - } - if (bytes.length > maxBytes) { - throw new MessageTooLargeException( - "PAYLOAD_TOO_LARGE", - "encoded payload is " + bytes.length + " bytes, limit is " + maxBytes); + throw unwrapTooLarge(exception); } return new EncodedMessage( - bytes, ContentType.JSON, Optional.of(SchemaReference.of(type.value(), version))); + sink.toByteArray(), + ContentType.JSON, + Optional.of(SchemaReference.of(type.value(), version))); } @Override public T decode( MessageType type, SchemaVersion version, byte[] encoded, Class payloadType) { - Class registered = requireRegistered(type); - Objects.requireNonNull(version, "schemaVersion must not be null"); + Class registered = requireRegistered(type, version); Objects.requireNonNull(encoded, "encoded bytes must not be null"); Objects.requireNonNull(payloadType, "payloadType must not be null"); if (!registered.equals(payloadType)) { @@ -144,14 +158,50 @@ public final class JacksonMessageCodec implements MessageCodec { } } - private Class requireRegistered(MessageType type) { - Objects.requireNonNull(type, "messageType must not be null"); - Class registered = registry.get(type); - if (registered == null) { - throw new MessageValidationException( - "UNKNOWN_MESSAGE_TYPE", "message type is not registered: " + type.value()); + private Class requireRegistered(MessageType type, SchemaVersion version) { + MessageContractKey key = new MessageContractKey(type, version); + Class registered = registry.get(key); + if (registered != null) { + return registered; } - return registered; + boolean typeIsKnown = registry.keySet().stream().anyMatch(known -> known.type().equals(type)); + if (typeIsKnown) { + // Deliberately not falling back to another version's class: decoding v999 bytes with the v1 + // class is exactly the silent type confusion the version-keyed registry exists to stop. + throw new MessageValidationException( + "SCHEMA_VERSION_NOT_REGISTERED", + "no contract is registered for " + + key.describe() + + "; registered versions: " + + registeredVersions(type)); + } + throw new MessageValidationException( + "UNKNOWN_MESSAGE_TYPE", "message type is not registered: " + type.value()); + } + + private List registeredVersions(MessageType type) { + return registry.keySet().stream() + .filter(known -> known.type().equals(type)) + .map(known -> known.version().value()) + .sorted() + .toList(); + } + + /** + * Surfaces a bound violation as itself rather than as a generic encoding failure. + * + *

    Jackson wraps stream failures, so the size refusal would otherwise reach the caller as + * {@code JSON_ENCODE_FAILED} — indistinguishable from a payload the mapper genuinely could not + * render, and the two need different operator responses. + */ + private static RuntimeException unwrapTooLarge(JacksonException exception) { + for (Throwable cause = exception; cause != null; cause = cause.getCause()) { + if (cause instanceof MessageTooLargeException tooLarge) { + return tooLarge; + } + } + return new MessageSerializationException( + "JSON_ENCODE_FAILED", "payload could not be encoded as JSON", exception); } private static ObjectMapper strictMapper(int maxBytes) { diff --git a/messaging/messaging-schema-json/src/test/java/dev/caskeleton/messaging/schema/json/JsonContractRegistryTest.java b/messaging/messaging-schema-json/src/test/java/dev/caskeleton/messaging/schema/json/JsonContractRegistryTest.java new file mode 100644 index 0000000..80249f3 --- /dev/null +++ b/messaging/messaging-schema-json/src/test/java/dev/caskeleton/messaging/schema/json/JsonContractRegistryTest.java @@ -0,0 +1,114 @@ +package dev.caskeleton.messaging.schema.json; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatCode; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import dev.caskeleton.messaging.api.MessageType; +import dev.caskeleton.messaging.api.SchemaVersion; +import dev.caskeleton.messaging.api.error.MessageTooLargeException; +import dev.caskeleton.messaging.api.error.MessageValidationException; +import dev.caskeleton.messaging.schema.MessageContractKey; +import java.util.List; +import java.util.Map; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** + * What the registry key is, and where the size limit takes effect. + * + *

    Two defects met in this codec. The registry was keyed on message type alone, so a message + * labelled v999 was decoded with the v1 class and kept its v999 label — the compatibility gate and + * the audit record then both described a contract that was never registered. And the size limit was + * applied to the finished byte array, which reports an oversized payload rather than preventing + * one. + */ +class JsonContractRegistryTest { + + private static final MessageType ORDER_CREATED = new MessageType("order.created"); + private static final SchemaVersion V1 = new SchemaVersion(1); + private static final SchemaVersion V2 = new SchemaVersion(2); + + record OrderCreated(String orderId, long amount) {} + + record OrderCreatedV2(String orderId, long amount, String channel) {} + + private final JacksonMessageCodec codec = + JacksonMessageCodec.of( + Map.of( + new MessageContractKey(ORDER_CREATED, V1), OrderCreated.class, + new MessageContractKey(ORDER_CREATED, V2), OrderCreatedV2.class)); + + @Test + @DisplayName("each registered version decodes with its own class") + void eachVersionDecodesWithItsOwnClass() { + byte[] v1 = codec.encode(ORDER_CREATED, V1, new OrderCreated("o-1", 10)).bytes(); + byte[] v2 = codec.encode(ORDER_CREATED, V2, new OrderCreatedV2("o-1", 10, "APP")).bytes(); + + assertThat(codec.decode(ORDER_CREATED, V1, v1, OrderCreated.class).orderId()).isEqualTo("o-1"); + assertThat(codec.decode(ORDER_CREATED, V2, v2, OrderCreatedV2.class).channel()) + .isEqualTo("APP"); + } + + @Test + @DisplayName("an unregistered version is refused, not decoded with a neighbour's class") + void anUnregisteredVersionIsRefused() { + byte[] v1 = codec.encode(ORDER_CREATED, V1, new OrderCreated("o-1", 10)).bytes(); + + assertThatThrownBy(() -> codec.decode(ORDER_CREATED, new SchemaVersion(999), v1, Object.class)) + .isInstanceOf(MessageValidationException.class) + .hasMessageContaining("order.created v999") + .hasMessageContaining("[1, 2]"); + } + + @Test + @DisplayName("the requested class must be the one registered for that exact version") + void theRequestedClassMustMatchTheVersion() { + byte[] v1 = codec.encode(ORDER_CREATED, V1, new OrderCreated("o-1", 10)).bytes(); + + assertThatThrownBy(() -> codec.decode(ORDER_CREATED, V1, v1, OrderCreatedV2.class)) + .isInstanceOf(MessageValidationException.class); + } + + @Test + @DisplayName("an unknown message type is still distinguished from an unknown version") + void anUnknownTypeIsDistinguishedFromAnUnknownVersion() { + assertThatThrownBy( + () -> codec.encode(new MessageType("order.cancelled"), V1, new OrderCreated("o", 1))) + .isInstanceOf(MessageValidationException.class) + .hasMessageContaining("message type is not registered"); + } + + @Test + @DisplayName("a payload far larger than the limit stops at the limit") + void aPayloadFarLargerThanTheLimitStopsAtTheLimit() { + MessageType largeList = new MessageType("text.list"); + JacksonMessageCodec bounded = + JacksonMessageCodec.of(Map.of(new MessageContractKey(largeList, V1), List.class), 1_024); + // Roughly twenty mebibytes of JSON. The old codec rendered all of it into a byte array and + // only then compared the length with 1024. + List payload = java.util.Collections.nCopies(200_000, "x".repeat(100)); + + assertThatThrownBy(() -> bounded.encode(largeList, V1, payload)) + .isInstanceOf(MessageTooLargeException.class) + .hasMessageContaining("1024") + .hasMessageContaining("refused at byte"); + } + + @Test + @DisplayName("a payload at exactly the limit is still accepted") + void aPayloadAtExactlyTheLimitIsAccepted() { + MessageType text = new MessageType("text.exact"); + int exact = + JacksonMessageCodec.testingDefault(text, String.class) + .encode(text, V1, "a".repeat(100)) + .bytes() + .length; + JacksonMessageCodec bounded = + JacksonMessageCodec.of(Map.of(new MessageContractKey(text, V1), String.class), exact); + + assertThatCode(() -> bounded.encode(text, V1, "a".repeat(100))).doesNotThrowAnyException(); + assertThatThrownBy(() -> bounded.encode(text, V1, "a".repeat(101))) + .isInstanceOf(MessageTooLargeException.class); + } +} diff --git a/messaging/messaging-schema-protobuf/src/main/java/dev/caskeleton/messaging/schema/protobuf/ProtobufMessageCodec.java b/messaging/messaging-schema-protobuf/src/main/java/dev/caskeleton/messaging/schema/protobuf/ProtobufMessageCodec.java index 894351f..c151c63 100644 --- a/messaging/messaging-schema-protobuf/src/main/java/dev/caskeleton/messaging/schema/protobuf/ProtobufMessageCodec.java +++ b/messaging/messaging-schema-protobuf/src/main/java/dev/caskeleton/messaging/schema/protobuf/ProtobufMessageCodec.java @@ -2,16 +2,19 @@ package dev.caskeleton.messaging.schema.protobuf; import com.google.protobuf.InvalidProtocolBufferException; import com.google.protobuf.Message; -import com.google.protobuf.Parser; import dev.caskeleton.messaging.api.ContentType; import dev.caskeleton.messaging.api.MessageType; import dev.caskeleton.messaging.api.SchemaVersion; import dev.caskeleton.messaging.api.error.MessageSerializationException; import dev.caskeleton.messaging.api.error.MessageTooLargeException; import dev.caskeleton.messaging.api.error.MessageValidationException; +import dev.caskeleton.messaging.schema.BoundedByteSink; import dev.caskeleton.messaging.schema.EncodedMessage; import dev.caskeleton.messaging.schema.MessageCodec; +import dev.caskeleton.messaging.schema.MessageContractKey; import dev.caskeleton.messaging.schema.SchemaReference; +import java.io.IOException; +import java.util.List; import java.util.Map; import java.util.Objects; import java.util.Optional; @@ -31,35 +34,27 @@ public final class ProtobufMessageCodec implements MessageCodec { private static final int DEFAULT_MAX_BYTES = 1_048_576; - private final Map> parsers; - private final Map> types; + private final Map contracts; private final int maxBytes; /** * Creates a codec over an explicit registry. * - * @param parsers the parser for each registered message type - * @param types the generated class for each registered message type + * @param contracts the parser and class of each registered {@code (type, version)} contract */ - public ProtobufMessageCodec( - Map> parsers, - Map> types) { - this(parsers, types, DEFAULT_MAX_BYTES); + public ProtobufMessageCodec(Map contracts) { + this(contracts, DEFAULT_MAX_BYTES); } /** * Creates a codec over an explicit registry and byte limit. * - * @param parsers the parser for each registered message type - * @param types the generated class for each registered message type + * @param contracts the parser and class of each registered {@code (type, version)} contract * @param maxBytes the maximum encoded size */ public ProtobufMessageCodec( - Map> parsers, - Map> types, - int maxBytes) { - this.parsers = Map.copyOf(Objects.requireNonNull(parsers, "parsers must not be null")); - this.types = Map.copyOf(Objects.requireNonNull(types, "types must not be null")); + Map contracts, int maxBytes) { + this.contracts = Map.copyOf(Objects.requireNonNull(contracts, "contracts must not be null")); if (maxBytes < 1) { throw new IllegalArgumentException("maxBytes must be positive"); } @@ -73,28 +68,35 @@ public final class ProtobufMessageCodec implements MessageCodec { @Override public EncodedMessage encode(MessageType type, SchemaVersion version, Object payload) { - Class registered = requireRegistered(type); - if (!(payload instanceof Message message) || !registered.isInstance(payload)) { + ProtobufMessageContract contract = requireRegistered(type, version); + if (!(payload instanceof Message message) || !contract.payloadType().isInstance(payload)) { throw new MessageValidationException( "PAYLOAD_TYPE_MISMATCH", "payload does not match the registered protobuf type for " + type.value()); } - byte[] bytes = message.toByteArray(); - if (bytes.length > maxBytes) { - throw new MessageTooLargeException( - "PAYLOAD_TOO_LARGE", - "encoded payload is " + bytes.length + " bytes, limit is " + maxBytes); + // Protobuf knows its serialized size exactly before writing a byte, so the limit is checked + // against that estimate first and enforced again by the sink. `toByteArray` allocated the whole + // encoding before anything could object. + BoundedByteSink sink = BoundedByteSink.of(maxBytes, "PAYLOAD_TOO_LARGE"); + sink.requireFits(message.getSerializedSize()); + try { + message.writeTo(sink); + } catch (IOException failure) { + throw new MessageSerializationException( + "PROTOBUF_ENCODE_FAILED", "payload could not be encoded as protobuf", failure); } return new EncodedMessage( - bytes, ContentType.PROTOBUF, Optional.of(SchemaReference.of(type.value(), version))); + sink.toByteArray(), + ContentType.PROTOBUF, + Optional.of(SchemaReference.of(type.value(), version))); } @Override public T decode( MessageType type, SchemaVersion version, byte[] encoded, Class payloadType) { - Class registered = requireRegistered(type); + ProtobufMessageContract contract = requireRegistered(type, version); Objects.requireNonNull(encoded, "encoded bytes must not be null"); - if (!registered.equals(payloadType)) { + if (!contract.payloadType().equals(payloadType)) { throw new MessageValidationException( "PAYLOAD_TYPE_MISMATCH", "requested type does not match the registered protobuf type for " + type.value()); @@ -105,20 +107,40 @@ public final class ProtobufMessageCodec implements MessageCodec { "encoded payload is " + encoded.length + " bytes, limit is " + maxBytes); } try { - return payloadType.cast(parsers.get(type).parseFrom(encoded)); + return payloadType.cast(contract.parser().parseFrom(encoded)); } catch (InvalidProtocolBufferException exception) { throw new MessageSerializationException( "PROTOBUF_DECODE_FAILED", "payload could not be decoded as protobuf", exception); } } - private Class requireRegistered(MessageType type) { - Objects.requireNonNull(type, "messageType must not be null"); - Class registered = types.get(type); - if (registered == null || !parsers.containsKey(type)) { - throw new MessageValidationException( - "UNKNOWN_MESSAGE_TYPE", "message type is not registered: " + type.value()); + private ProtobufMessageContract requireRegistered(MessageType type, SchemaVersion version) { + MessageContractKey key = new MessageContractKey(type, version); + ProtobufMessageContract contract = contracts.get(key); + if (contract != null) { + return contract; } - return registered; + boolean typeIsKnown = contracts.keySet().stream().anyMatch(known -> known.type().equals(type)); + if (typeIsKnown) { + // Protobuf will happily decode almost any bytes with almost any parser, so falling back to + // another version's parser does not fail — it returns a populated object built from a schema + // nobody registered for this version. + throw new MessageValidationException( + "SCHEMA_VERSION_NOT_REGISTERED", + "no contract is registered for " + + key.describe() + + "; registered versions: " + + registeredVersions(type)); + } + throw new MessageValidationException( + "UNKNOWN_MESSAGE_TYPE", "message type is not registered: " + type.value()); + } + + private List registeredVersions(MessageType type) { + return contracts.keySet().stream() + .filter(known -> known.type().equals(type)) + .map(known -> known.version().value()) + .sorted() + .toList(); } } diff --git a/messaging/messaging-schema-protobuf/src/main/java/dev/caskeleton/messaging/schema/protobuf/ProtobufMessageContract.java b/messaging/messaging-schema-protobuf/src/main/java/dev/caskeleton/messaging/schema/protobuf/ProtobufMessageContract.java new file mode 100644 index 0000000..1bd7039 --- /dev/null +++ b/messaging/messaging-schema-protobuf/src/main/java/dev/caskeleton/messaging/schema/protobuf/ProtobufMessageContract.java @@ -0,0 +1,53 @@ +package dev.caskeleton.messaging.schema.protobuf; + +import com.google.protobuf.Message; +import com.google.protobuf.Parser; +import dev.caskeleton.messaging.api.error.MessagingConfigurationException; +import java.util.Objects; + +/** + * The parser and the generated class of one registered contract, held together. + * + *

    They used to live in two parallel maps. Nothing checked that the two agreed, so a registry + * that paired {@code OrderCreated.class} with {@code OrderCancelled}'s parser was accepted at + * construction and produced a {@code ClassCastException} at decode time — on a broker thread, for + * one message type, in production. Worse, a type present in one map and absent from the other made + * {@code parsers.get(type)} return null and the decode fail with a {@code NullPointerException} + * rather than the registry error the operator needed to read. + * + *

    Binding them in one value makes the mismatch impossible to express, and the constructor proves + * the pairing by parsing empty input: the parser's default instance must be an instance of the + * declared class. + * + * @param payloadType the generated message class this contract decodes into + * @param parser the generated parser for that class + */ +public record ProtobufMessageContract( + Class payloadType, Parser parser) { + + /** Verifies that the parser really produces the declared class. */ + public ProtobufMessageContract { + Objects.requireNonNull(payloadType, "payloadType must not be null"); + Objects.requireNonNull(parser, "parser must not be null"); + Message defaultInstance; + try { + defaultInstance = parser.parseFrom(new byte[0]); + } catch (Exception failure) { + throw new MessagingConfigurationException( + "PROTOBUF_CONTRACT_UNUSABLE", + "the parser registered for " + + payloadType.getName() + + " could not parse empty input, so the pairing cannot be verified", + failure); + } + if (!payloadType.isInstance(defaultInstance)) { + throw new MessagingConfigurationException( + "PROTOBUF_CONTRACT_MISMATCH", + "the parser registered for " + + payloadType.getName() + + " produces " + + defaultInstance.getClass().getName() + + "; a mismatched pairing fails at decode time on a broker thread, not here"); + } + } +} diff --git a/messaging/messaging-schema-protobuf/src/test/java/dev/caskeleton/messaging/schema/protobuf/ProtobufCompatibilityTest.java b/messaging/messaging-schema-protobuf/src/test/java/dev/caskeleton/messaging/schema/protobuf/ProtobufCompatibilityTest.java index 6ba1ca0..9be1fa6 100644 --- a/messaging/messaging-schema-protobuf/src/test/java/dev/caskeleton/messaging/schema/protobuf/ProtobufCompatibilityTest.java +++ b/messaging/messaging-schema-protobuf/src/test/java/dev/caskeleton/messaging/schema/protobuf/ProtobufCompatibilityTest.java @@ -15,8 +15,11 @@ import com.google.protobuf.Parser; import dev.caskeleton.messaging.api.ContentType; import dev.caskeleton.messaging.api.MessageType; import dev.caskeleton.messaging.api.SchemaVersion; +import dev.caskeleton.messaging.api.error.MessageTooLargeException; import dev.caskeleton.messaging.api.error.MessageValidationException; +import dev.caskeleton.messaging.api.error.MessagingConfigurationException; import dev.caskeleton.messaging.schema.EncodedMessage; +import dev.caskeleton.messaging.schema.MessageContractKey; import java.util.Map; import org.junit.jupiter.api.Test; @@ -91,10 +94,18 @@ class ProtobufCompatibilityTest { } private static ProtobufMessageCodec codec(Descriptor descriptor) { - DynamicMessage prototype = DynamicMessage.getDefaultInstance(descriptor); - Parser parser = prototype.getParserForType(); + return codec(descriptor, V1); + } + + private static ProtobufMessageCodec codec(Descriptor descriptor, SchemaVersion version) { return new ProtobufMessageCodec( - Map.of(ORDER_CREATED, parser), Map.of(ORDER_CREATED, DynamicMessage.class)); + Map.of(new MessageContractKey(ORDER_CREATED, version), contract(descriptor))); + } + + private static ProtobufMessageContract contract(Descriptor descriptor) { + Parser parser = + DynamicMessage.getDefaultInstance(descriptor).getParserForType(); + return new ProtobufMessageContract(DynamicMessage.class, parser); } private static DynamicMessage v1Message() { @@ -176,9 +187,53 @@ class ProtobufCompatibilityTest { .isInstanceOf(MessageValidationException.class); } + @Test + void anUnregisteredVersionIsRejectedRatherThanDecodedWithAnotherVersionsParser() { + ProtobufMessageCodec codec = codec(V1_DESCRIPTOR); + byte[] wire = codec.encode(ORDER_CREATED, V1, v1Message()).bytes(); + + assertThatThrownBy(() -> codec.decode(ORDER_CREATED, V2, wire, DynamicMessage.class)) + .as( + "keyed on the type alone, v2 would have decoded with the v1 parser and kept the v2 label") + .isInstanceOf(MessageValidationException.class) + .hasMessageContaining("order.created v2"); + } + + @Test + void aParserThatDoesNotProduceTheDeclaredClassIsRejectedAtConstruction() { + Parser parser = + DynamicMessage.getDefaultInstance(V1_DESCRIPTOR).getParserForType(); + + assertThatThrownBy(() -> new ProtobufMessageContract(FileDescriptorProto.class, parser)) + .as("the mismatch used to surface as a ClassCastException on a broker thread") + .isInstanceOf(MessagingConfigurationException.class) + .hasMessageContaining("produces"); + } + + @Test + void anOversizedPayloadIsRefusedBeforeItIsSerialized() { + ProtobufMessageCodec codec = + new ProtobufMessageCodec( + Map.of(new MessageContractKey(ORDER_CREATED, V1), contract(V1_DESCRIPTOR)), 16); + + assertThatThrownBy(() -> codec.encode(ORDER_CREATED, V1, v1Message())) + .isInstanceOf(MessageTooLargeException.class) + .hasMessageContaining("refused at byte"); + } + + @Test + void aPayloadAtExactlyTheLimitIsAccepted() { + int exact = v1Message().getSerializedSize(); + ProtobufMessageCodec codec = + new ProtobufMessageCodec( + Map.of(new MessageContractKey(ORDER_CREATED, V1), contract(V1_DESCRIPTOR)), exact); + + assertThat(codec.encode(ORDER_CREATED, V1, v1Message()).bytes()).hasSize(exact); + } + @Test void theEncodedMessageCarriesItsSchemaReference() { - EncodedMessage encoded = codec(V1_DESCRIPTOR).encode(ORDER_CREATED, V2, v1Message()); + EncodedMessage encoded = codec(V1_DESCRIPTOR, V2).encode(ORDER_CREATED, V2, v1Message()); assertThat(encoded.schemaReference()) .hasValueSatisfying(reference -> assertThat(reference.version()).isEqualTo(V2)); diff --git a/messaging/messaging-security/src/main/java/dev/caskeleton/messaging/security/BrokerTlsPolicy.java b/messaging/messaging-security/src/main/java/dev/caskeleton/messaging/security/BrokerTlsPolicy.java index a271a6f..0a648ea 100644 --- a/messaging/messaging-security/src/main/java/dev/caskeleton/messaging/security/BrokerTlsPolicy.java +++ b/messaging/messaging-security/src/main/java/dev/caskeleton/messaging/security/BrokerTlsPolicy.java @@ -19,8 +19,8 @@ import java.util.Set; public final class BrokerTlsPolicy { /** Protocol versions with known practical attacks. */ - private static final Set FORBIDDEN_PROTOCOLS = - Set.of("SSLv2", "SSLv3", "TLSv1", "TLSv1.1"); + /** The only protocol versions this platform will negotiate. */ + private static final Set ACCEPTED_PROTOCOLS = Set.of("TLSv1.2", "TLSv1.3"); /** The minimum acceptable protocol version. */ public static final String MINIMUM_PROTOCOL = "TLSv1.2"; @@ -69,13 +69,22 @@ public final class BrokerTlsPolicy { .formatted(profile.broker())); } - List forbidden = - enabledProtocols.stream().filter(FORBIDDEN_PROTOCOLS::contains).toList(); - if (!forbidden.isEmpty()) { + // An allowlist, not a denylist. + // + // The denylist named the old versions somebody thought of, so `SSL`, `TLSv0.9`, `PLAINTEXT` + // and any typo passed — and a protocol string the JVM does not recognise is negotiated as + // whatever the JVM defaults to, which is the outcome this policy exists to prevent. Naming the + // two acceptable versions means an unknown string fails here rather than at connect time on a + // production broker. + List unsupported = + enabledProtocols.stream() + .filter(protocol -> !ACCEPTED_PROTOCOLS.contains(protocol)) + .toList(); + if (!unsupported.isEmpty()) { throw new MessagingConfigurationException( - "TLS_PROTOCOL_TOO_OLD", - "broker %s enables %s; the minimum is %s" - .formatted(profile.broker(), forbidden, MINIMUM_PROTOCOL)); + "TLS_PROTOCOL_NOT_ACCEPTED", + "broker %s enables %s; only %s are accepted" + .formatted(profile.broker(), unsupported, ACCEPTED_PROTOCOLS)); } if (enabledProtocols.isEmpty()) { throw new MessagingConfigurationException( @@ -93,6 +102,6 @@ public final class BrokerTlsPolicy { * @return true when the version may be negotiated */ public static boolean isAcceptable(String protocol) { - return protocol != null && !FORBIDDEN_PROTOCOLS.contains(protocol); + return protocol != null && ACCEPTED_PROTOCOLS.contains(protocol); } } diff --git a/messaging/messaging-security/src/main/java/dev/caskeleton/messaging/security/CredentialRuntimeRegistry.java b/messaging/messaging-security/src/main/java/dev/caskeleton/messaging/security/CredentialRuntimeRegistry.java index 9f5718c..ef7e898 100644 --- a/messaging/messaging-security/src/main/java/dev/caskeleton/messaging/security/CredentialRuntimeRegistry.java +++ b/messaging/messaging-security/src/main/java/dev/caskeleton/messaging/security/CredentialRuntimeRegistry.java @@ -59,14 +59,31 @@ public final class CredentialRuntimeRegistry { if (current != null && !current.isDueForRotation(now)) { return current; } - CredentialRuntime replacement = fetch(credentialId, now); - resolved.put(credentialId, replacement); - if (current != null) { - // Clear only after the replacement is installed, so a concurrent reader never observes a - // window with no usable credential. - current.clear(); - } - return replacement; + // Single-flight, keyed by credential id. + // + // get → fetch → put → clear had no synchronization at all. Two callers rotating the same + // credential both read the same old runtime and both fetched a replacement: one replacement + // was lost from the map without ever being cleared — a secret left in memory that nothing owns + // — and the caller that lost the race could clear material the winner was still using. + // + // compute holds the bin lock for this key, so exactly one fetch publishes and the previous + // generation is retired by that same caller. + return resolved.compute( + credentialId, + (key, existing) -> { + if (existing != null && !existing.isDueForRotation(now)) { + // Another thread rotated while this one waited; its replacement is the current one. + return existing; + } + CredentialRuntime replacement = fetch(key, now); + if (existing != null) { + // Cleared only after the replacement is installed, so no reader sees a window with no + // usable credential — and only by the thread that replaced it, so the material a + // concurrent reader holds is never wiped underneath it. + existing.clear(); + } + return replacement; + }); } /** diff --git a/messaging/messaging-security/src/test/java/dev/caskeleton/messaging/security/CredentialRotationContractTest.java b/messaging/messaging-security/src/test/java/dev/caskeleton/messaging/security/CredentialRotationContractTest.java new file mode 100644 index 0000000..cb7e002 --- /dev/null +++ b/messaging/messaging-security/src/test/java/dev/caskeleton/messaging/security/CredentialRotationContractTest.java @@ -0,0 +1,109 @@ +package dev.caskeleton.messaging.security; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatCode; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import java.time.Duration; +import java.time.Instant; +import java.util.List; +import java.util.concurrent.Callable; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.stream.IntStream; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** + * Rotation under concurrency, and the protocol allowlist. + * + *

    {@code resolve} was get → fetch → put → clear with no synchronization. Two callers rotating + * the same credential both read the same old runtime and both fetched a replacement: one + * replacement was dropped from the map without ever being cleared — a secret left in memory that + * nothing owns — and the loser could clear material the winner was still using. + */ +class CredentialRotationContractTest { + + private static final Instant NOW = Instant.parse("2026-08-14T00:00:00Z"); + + @Test + @DisplayName("a hundred concurrent resolves fetch once and publish one generation") + void concurrentResolvesFetchOnce() throws Exception { + AtomicInteger fetches = new AtomicInteger(); + CredentialRuntimeRegistry registry = registry(fetches); + + ExecutorService pool = Executors.newFixedThreadPool(16); + try { + List> work = + IntStream.range(0, 100) + .>mapToObj( + index -> () -> registry.resolve("kafka-app", NOW)) + .toList(); + List resolved = new java.util.ArrayList<>(); + for (Future future : pool.invokeAll(work)) { + resolved.add(future.get()); + } + + assertThat(fetches) + .as("every caller fetching its own replacement leaves secrets nobody owns") + .hasValue(1); + assertThat(resolved).allSatisfy(runtime -> assertThat(runtime).isSameAs(resolved.get(0))); + } finally { + pool.shutdownNow(); + } + } + + @Test + @DisplayName("the surviving generation still has its material") + void theSurvivingGenerationKeepsItsMaterial() { + CredentialRuntimeRegistry registry = registry(new AtomicInteger()); + + CredentialRuntime resolved = registry.resolve("kafka-app", NOW); + + assertThatCode(resolved::material) + .as("the losing caller used to clear material the winner was still using") + .doesNotThrowAnyException(); + } + + @Test + @DisplayName("only TLS 1.2 and 1.3 are acceptable") + void onlyModernTlsIsAcceptable() { + assertThat(BrokerTlsPolicy.isAcceptable("TLSv1.2")).isTrue(); + assertThat(BrokerTlsPolicy.isAcceptable("TLSv1.3")).isTrue(); + // Each of these passed the denylist: an unrecognised protocol string negotiates whatever the + // JVM defaults to, which is what the policy exists to prevent. + assertThat(BrokerTlsPolicy.isAcceptable("SSL")).isFalse(); + assertThat(BrokerTlsPolicy.isAcceptable("TLSv0.9")).isFalse(); + assertThat(BrokerTlsPolicy.isAcceptable("PLAINTEXT")).isFalse(); + assertThat(BrokerTlsPolicy.isAcceptable("tlsv1.3")).isFalse(); + assertThat(BrokerTlsPolicy.isAcceptable(null)).isFalse(); + } + + private static CredentialRuntimeRegistry registry(AtomicInteger fetches) { + return new CredentialRuntimeRegistry( + new CredentialProvider() { + @Override + public char[] resolve(String credentialId) { + fetches.incrementAndGet(); + return "s3cr3t".toCharArray(); + } + + @Override + public java.util.Optional expiresAt(String credentialId) { + return java.util.Optional.of(NOW.plus(Duration.ofHours(1))); + } + }, + Duration.ofMinutes(5)); + } + + @Test + @DisplayName("an unknown credential id is refused rather than fetched") + void anUnknownCredentialIdIsRefused() { + CredentialRuntimeRegistry registry = registry(new AtomicInteger()); + + assertThatThrownBy(() -> registry.resolve(" ", NOW)) + .isInstanceOf(IllegalArgumentException.class); + } +} diff --git a/messaging/messaging-spring-boot-starter/README.md b/messaging/messaging-spring-boot-starter/README.md new file mode 100644 index 0000000..c99440e --- /dev/null +++ b/messaging/messaging-spring-boot-starter/README.md @@ -0,0 +1,26 @@ +# messaging-spring-boot-starter + +## 아티팩트 분리 (예정) + +이 starter는 현재 하나의 아티팩트로 Kafka·Rabbit·JDBC reliability·admin 자동설정을 모두 등록한다. +MSG-007이 요구하는 최종 형태는 capability별 분리다: + +```text +messaging-spring-boot-autoconfigure-core +messaging-spring-boot-starter-core +messaging-spring-boot-starter-kafka +messaging-spring-boot-starter-rabbit +messaging-spring-boot-starter-reliability-jdbc-postgresql +messaging-spring-boot-starter-admin +``` + +지금 반영된 것과 남은 것을 분명히 적는다. + +- **반영**: 내부 leaf 9개를 `api` → `implementation`으로 낮췄다. 어댑터를 쓰지 않는 소비자의 컴파일 + classpath에 두 브로커 SDK가 더 이상 올라가지 않는다. +- **반영**: 브로커 security configurer는 `CredentialRuntimeRegistry` 빈이 있을 때만 만들어지고, + production 프로파일에 credential 소스가 없으면 `MessagingCredentialRequirementValidator`가 + 내부 타입 이름이 아니라 조치 가능한 한 개의 오류로 실패시킨다. +- **남음**: 아티팩트 자체의 분리. 레지스트리(43→44에서 다시 49)와 `AutoConfiguration.imports` 분할을 + 함께 바꾸는 변경이라 별도 커밋으로 다룬다. 그 전까지 런타임 의존성(`runtimeClasspath`)에는 여전히 + 두 브로커가 함께 들어온다. diff --git a/messaging/messaging-spring-boot-starter/build.gradle b/messaging/messaging-spring-boot-starter/build.gradle index cda8461..90c578f 100644 --- a/messaging/messaging-spring-boot-starter/build.gradle +++ b/messaging/messaging-spring-boot-starter/build.gradle @@ -1,22 +1,36 @@ apply plugin: 'java-library' +// Scopes, not a flat list of `api`. +// +// Sixteen internal leaves were exposed as `api`, so every adopter of the starter compiled against +// the Kafka adapter, the Rabbit adapter, both JDBC reliability leaves and the admin plane whether +// they used them or not — and both brokers' `@ConditionalOnClass` guards were true at once because +// the starter itself dragged in both clients. +// +// `api` is now the types that genuinely appear in this module's public signatures; everything else +// is `implementation`. Splitting the starter into per-capability artifacts (core / kafka / rabbit / +// reliability / admin) is the structural fix the review asks for and is a registry change of its +// own; this is the part that stops an adopter's compile classpath from containing two broker SDKs. dependencies { + // Types an adopter writes against: envelopes, destinations, publish results, profiles, ports. api project(':messaging:messaging-core-api') api project(':messaging:messaging-schema-api') - api project(':messaging:messaging-schema-json') - api project(':messaging:messaging-cloudevents') api project(':messaging:messaging-policy') api project(':messaging:messaging-transport-spi') api project(':messaging:messaging-observability') api project(':messaging:messaging-security') - api project(':messaging:messaging-kafka') - api project(':messaging:messaging-rabbit') api project(':messaging:messaging-reliability-api') - api project(':messaging:messaging-outbox-jpa') - api project(':messaging:messaging-inbox-jpa') - api project(':messaging:messaging-claim-check') - api project(':messaging:messaging-admin-api') - api project(':messaging:messaging-admin-runtime') + + // Wired by the auto-configurations and never named in an adopter's source. + implementation project(':messaging:messaging-schema-json') + implementation project(':messaging:messaging-cloudevents') + implementation project(':messaging:messaging-kafka') + implementation project(':messaging:messaging-rabbit') + implementation project(':messaging:messaging-outbox-jdbc-postgresql') + implementation project(':messaging:messaging-inbox-jdbc-postgresql') + implementation project(':messaging:messaging-claim-check') + implementation project(':messaging:messaging-admin-api') + implementation project(':messaging:messaging-admin-runtime') implementation 'org.springframework.boot:spring-boot-autoconfigure' implementation 'org.springframework.boot:spring-boot-actuator' diff --git a/messaging/messaging-spring-boot-starter/gradle.lockfile b/messaging/messaging-spring-boot-starter/gradle.lockfile index 7e993f3..df60bd0 100644 --- a/messaging/messaging-spring-boot-starter/gradle.lockfile +++ b/messaging/messaging-spring-boot-starter/gradle.lockfile @@ -28,30 +28,30 @@ com.google.j2objc:j2objc-annotations:3.1=annotationProcessor,checkstyle,testAnno com.google.protobuf:protobuf-java:4.33.2=annotationProcessor,testAnnotationProcessor com.h3xstream.findsecbugs:findsecbugs-plugin:1.14.0=spotbugsPlugins com.puppycrawl.tools:checkstyle:13.5.0=checkstyle -com.rabbitmq:amqp-client:5.27.1=runtimeClasspath,testRuntimeClasspath +com.rabbitmq:amqp-client:5.27.1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath commons-beanutils:commons-beanutils:1.11.0=checkstyle commons-collections:commons-collections:3.2.2=checkstyle commons-io:commons-io:2.21.0=spotbugs commons-logging:commons-logging:1.3.5=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath info.picocli:picocli:4.7.7=checkstyle -io.cloudevents:cloudevents-api:4.0.1=runtimeClasspath,testRuntimeClasspath +io.cloudevents:cloudevents-api:4.0.1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath io.cloudevents:cloudevents-core:4.0.1=runtimeClasspath,testRuntimeClasspath io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnotationProcessor io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor io.micrometer:micrometer-commons:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -io.micrometer:micrometer-core:1.16.0=runtimeClasspath,testRuntimeClasspath +io.micrometer:micrometer-core:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath io.micrometer:micrometer-observation:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -io.netty:netty-buffer:4.2.17.Final=runtimeClasspath,testRuntimeClasspath -io.netty:netty-codec-base:4.2.17.Final=runtimeClasspath,testRuntimeClasspath -io.netty:netty-codec-compression:4.2.17.Final=runtimeClasspath,testRuntimeClasspath -io.netty:netty-codec-marshalling:4.2.17.Final=runtimeClasspath,testRuntimeClasspath -io.netty:netty-codec-protobuf:4.2.17.Final=runtimeClasspath,testRuntimeClasspath -io.netty:netty-codec:4.2.17.Final=runtimeClasspath,testRuntimeClasspath -io.netty:netty-common:4.2.17.Final=runtimeClasspath,testRuntimeClasspath -io.netty:netty-handler:4.2.17.Final=runtimeClasspath,testRuntimeClasspath -io.netty:netty-resolver:4.2.17.Final=runtimeClasspath,testRuntimeClasspath -io.netty:netty-transport-native-unix-common:4.2.17.Final=runtimeClasspath,testRuntimeClasspath -io.netty:netty-transport:4.2.17.Final=runtimeClasspath,testRuntimeClasspath +io.netty:netty-buffer:4.2.17.Final=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.netty:netty-codec-base:4.2.17.Final=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.netty:netty-codec-compression:4.2.17.Final=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.netty:netty-codec-marshalling:4.2.17.Final=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.netty:netty-codec-protobuf:4.2.17.Final=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.netty:netty-codec:4.2.17.Final=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.netty:netty-common:4.2.17.Final=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.netty:netty-handler:4.2.17.Final=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.netty:netty-resolver:4.2.17.Final=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.netty:netty-transport-native-unix-common:4.2.17.Final=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.netty:netty-transport:4.2.17.Final=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath io.projectreactor:reactor-core:3.8.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath io.projectreactor:reactor-test:3.8.0=testCompileClasspath,testRuntimeClasspath javax.inject:javax.inject:1=annotationProcessor,testAnnotationProcessor @@ -65,7 +65,7 @@ org.apache.commons:commons-text:1.15.0=spotbugs org.apache.commons:commons-text:1.3=checkstyle org.apache.httpcomponents:httpclient:4.5.13=checkstyle org.apache.httpcomponents:httpcore:4.4.16=checkstyle -org.apache.kafka:kafka-clients:4.1.1=runtimeClasspath,testRuntimeClasspath +org.apache.kafka:kafka-clients:4.1.1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.apache.logging.log4j:log4j-api:2.25.2=spotbugs org.apache.logging.log4j:log4j-core:2.25.2=spotbugs org.apache.maven.doxia:doxia-core:1.12.0=checkstyle @@ -104,10 +104,10 @@ org.ow2.asm:asm:9.10.1=spotbugs org.pcollections:pcollections:4.0.1=annotationProcessor,testAnnotationProcessor org.reactivestreams:reactive-streams:1.0.4=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.reflections:reflections:0.10.2=checkstyle -org.slf4j:slf4j-api:2.0.17=runtimeClasspath,spotbugs,spotbugsSlf4j,testRuntimeClasspath +org.slf4j:slf4j-api:2.0.17=compileClasspath,runtimeClasspath,spotbugs,spotbugsSlf4j,testCompileClasspath,testRuntimeClasspath org.slf4j:slf4j-simple:2.0.17=checkstyle,spotbugsSlf4j -org.springframework.amqp:spring-amqp:4.0.0=runtimeClasspath,testRuntimeClasspath -org.springframework.amqp:spring-rabbit:4.0.0=runtimeClasspath,testRuntimeClasspath +org.springframework.amqp:spring-amqp:4.0.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +org.springframework.amqp:spring-rabbit:4.0.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.springframework.boot:spring-boot-actuator:4.0.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.springframework.boot:spring-boot-autoconfigure:4.0.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.springframework.boot:spring-boot-configuration-processor:4.0.0=annotationProcessor @@ -120,9 +120,9 @@ org.springframework:spring-context:7.0.1=compileClasspath,runtimeClasspath,testC org.springframework:spring-core:7.0.1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.springframework:spring-expression:7.0.1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.springframework:spring-jdbc:7.0.1=runtimeClasspath,testRuntimeClasspath -org.springframework:spring-messaging:7.0.1=runtimeClasspath,testRuntimeClasspath +org.springframework:spring-messaging:7.0.1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.springframework:spring-test:7.0.1=testCompileClasspath,testRuntimeClasspath -org.springframework:spring-tx:7.0.1=runtimeClasspath,testRuntimeClasspath +org.springframework:spring-tx:7.0.1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.xerial.snappy:snappy-java:1.1.10.7=runtimeClasspath,testRuntimeClasspath org.xmlresolver:xmlresolver:5.3.3=checkstyle,spotbugs tools.jackson.core:jackson-core:3.0.2=runtimeClasspath,testRuntimeClasspath diff --git a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/DefaultBatchMessagePublisher.java b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/DefaultBatchMessagePublisher.java index d7ab420..f6f6548 100644 --- a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/DefaultBatchMessagePublisher.java +++ b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/DefaultBatchMessagePublisher.java @@ -65,28 +65,164 @@ public final class DefaultBatchMessagePublisher implements BatchMessagePublisher Instant startedAt = clock.get(); List> pending = new ArrayList<>(requests.size()); - boolean rejectionSeen = false; + // Index and message id per slot, so an entry still outstanding at the deadline can still be + // reported against the input position it came from. + List placeholders = new ArrayList<>(requests.size()); for (int index = 0; index < requests.size(); index++) { - if (rejectionSeen && options.stopOnFirstRejection()) { - break; + placeholders.add(outstanding(index, requests.get(index).message().messageId())); + if (options.stopOnFirstRejection() && anyRejected(pending)) { + // Every remaining index still gets a result. Breaking out shortened the result list, so a + // caller could not tell "not submitted" from "result lost" — and those need opposite + // actions. + pending.add( + CompletableFuture.completedFuture( + notSubmitted(index, requests.get(index).message().messageId(), startedAt))); + continue; } - pending.add(submit(requests.get(index), index)); + CompletableFuture submitted = submit(requests.get(index), index); + pending.add(submitted); if (options.stopOnFirstRejection()) { - // Only observable for an entry that failed synchronously, which is the local-validation - // case; a broker rejection resolves later and is picked up on the next iteration. - CompletableFuture latest = pending.get(pending.size() - 1); - rejectionSeen = - latest.isDone() && latest.join().result().completion() == PublishCompletion.REJECTED; + // Sequential under stop-on-first-rejection: waiting for this entry's outcome is the only + // way the flag can mean anything. The previous loop only saw a rejection that had already + // resolved, which for an asynchronous broker is never — so every remaining entry was + // submitted anyway and the option did nothing. + awaitQuietly(submitted, remaining(startedAt, options.timeout())); } } - return CompletableFuture.allOf(pending.toArray(CompletableFuture[]::new)) - .thenApply( - ignored -> - new BatchPublishResult( - pending.stream().map(CompletableFuture::join).toList(), - Duration.between(startedAt, clock.get()))); + CompletableFuture completed = new CompletableFuture<>(); + CompletableFuture all = + CompletableFuture.allOf(pending.toArray(CompletableFuture[]::new)); + // The returned stage is deliberately dropped: `completed` is what the caller holds, and this + // callback's only job is to fill it in. Nothing downstream observes the intermediate stage. + CompletableFuture ignoredCompletion = + all.whenComplete( + (ignored, failure) -> + completed.complete( + new BatchPublishResult( + resolveAll(pending, placeholders), + Duration.between(startedAt, clock.get())))); + assert ignoredCompletion != null; + + // The whole-batch deadline the options document. It was never read, so a publisher that never + // completed left the caller's stage pending forever — a timeout that exists only in the + // javadoc. + // Same for the scheduled future: cancelling it early would only save a no-op, and the task + // checks `isDone` before completing anything. + java.util.concurrent.ScheduledFuture ignoredDeadline = + DEADLINES.schedule( + () -> { + if (!completed.isDone()) { + completed.complete( + new BatchPublishResult( + resolveAll(pending, placeholders), + Duration.between(startedAt, clock.get()))); + } + }, + options.timeout().toMillis(), + java.util.concurrent.TimeUnit.MILLISECONDS); + assert ignoredDeadline != null; + return completed; + } + + /** + * One scheduler for batch deadlines. + * + *

    A daemon thread so a pending deadline never keeps the JVM alive, and one shared instance + * because a scheduler per batch is a thread per batch. + */ + private static final java.util.concurrent.ScheduledExecutorService DEADLINES = + java.util.concurrent.Executors.newSingleThreadScheduledExecutor( + runnable -> { + Thread thread = new Thread(runnable, "messaging-batch-deadline"); + thread.setDaemon(true); + return thread; + }); + + private boolean anyRejected(List> pending) { + return pending.stream() + .anyMatch( + future -> + future.isDone() + && !future.isCompletedExceptionally() + && future.join().result().completion() == PublishCompletion.REJECTED); + } + + private void awaitQuietly(CompletableFuture future, Duration budget) { + if (budget.isNegative() || budget.isZero()) { + return; + } + try { + future.get(budget.toMillis(), java.util.concurrent.TimeUnit.MILLISECONDS); + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + } catch (java.util.concurrent.ExecutionException + | java.util.concurrent.TimeoutException other) { + // Both are outcomes the result already carries; nothing to add here. + } + } + + private Duration remaining(Instant startedAt, Duration timeout) { + return timeout.minus(Duration.between(startedAt, clock.get())); + } + + /** + * An entry the batch never submitted because an earlier one was rejected. + * + *

    Distinct from a failure: nothing was sent, so the caller may submit it unchanged. + */ + private BatchPublishItemResult notSubmitted( + int index, dev.caskeleton.messaging.api.MessageId messageId, Instant startedAt) { + return new BatchPublishItemResult( + index, + messageId, + new dev.caskeleton.messaging.api.publish.PublishResult( + PublishCompletion.REJECTED, + dev.caskeleton.messaging.api.publish.PublishEvidence.notTransmitted(), + dev.caskeleton.messaging.api.publish.RoutingOutcome.UNKNOWN, + java.util.Optional.empty(), + 1, + Duration.between(startedAt, clock.get()), + java.util.Optional.of( + dev.caskeleton.messaging.api.error.FailureDescriptor.of( + dev.caskeleton.messaging.api.error.FailureCategory.CONFIGURATION, + "NOT_SUBMITTED_AFTER_REJECTION", + "an earlier entry was rejected and this batch stops on the first rejection")))); + } + + private List resolveAll( + List> pending, + List placeholders) { + List resolved = new ArrayList<>(pending.size()); + for (int index = 0; index < pending.size(); index++) { + CompletableFuture future = pending.get(index); + resolved.add( + future.isDone() && !future.isCompletedExceptionally() + ? future.join() + : placeholders.get(index)); + } + return List.copyOf(resolved); + } + + /** An entry still outstanding when the deadline arrived is ambiguous, never a success. */ + private BatchPublishItemResult outstanding( + int index, dev.caskeleton.messaging.api.MessageId messageId) { + return new BatchPublishItemResult( + index, + messageId, + new dev.caskeleton.messaging.api.publish.PublishResult( + PublishCompletion.AMBIGUOUS, + dev.caskeleton.messaging.api.publish.PublishEvidence.ambiguous(), + dev.caskeleton.messaging.api.publish.RoutingOutcome.UNKNOWN, + java.util.Optional.empty(), + 1, + Duration.ZERO, + java.util.Optional.of( + dev.caskeleton.messaging.api.error.FailureDescriptor.of( + dev.caskeleton.messaging.api.error.FailureCategory.AMBIGUOUS, + "BATCH_DEADLINE_EXCEEDED", + "the batch deadline elapsed before this entry produced evidence")))); } private CompletableFuture submit( diff --git a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/KafkaMessagingAutoConfiguration.java b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/KafkaMessagingAutoConfiguration.java index 1c75d69..90a870f 100644 --- a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/KafkaMessagingAutoConfiguration.java +++ b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/KafkaMessagingAutoConfiguration.java @@ -67,8 +67,19 @@ public class KafkaMessagingAutoConfiguration { * @param tlsPolicy the transport security policy * @return the configurer */ + /** + * The security configurer, which exists only when a credential source does. + * + *

    It took {@code CredentialRuntimeRegistry} as a required parameter, and that registry is + * created only when the application supplies a {@code CredentialProvider}. Without one the + * context failed on a missing bean naming the registry — an error that describes the platform's + * internals rather than the one thing the operator has to do. The condition moves the failure to + * a place that can say so. + */ @Bean @ConditionalOnMissingBean + @org.springframework.boot.autoconfigure.condition.ConditionalOnBean( + CredentialRuntimeRegistry.class) public KafkaSecurityConfigurer kafkaSecurityConfigurer( CredentialRuntimeRegistry credentials, BrokerTlsPolicy tlsPolicy) { return new KafkaSecurityConfigurer(credentials, tlsPolicy); diff --git a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingAdminAutoConfiguration.java b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingAdminAutoConfiguration.java index 295918c..5f46ad4 100644 --- a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingAdminAutoConfiguration.java +++ b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingAdminAutoConfiguration.java @@ -1,9 +1,10 @@ package dev.caskeleton.messaging.autoconfigure; +import dev.caskeleton.messaging.admin.AdminOperationJournal; import dev.caskeleton.messaging.admin.DestructiveOperationGuard; -import dev.caskeleton.messaging.admin.runtime.AdminOperationIdempotencyStore; import dev.caskeleton.messaging.admin.runtime.BrokerTopologyInspector; import dev.caskeleton.messaging.admin.runtime.CompositeTopologyValidator; +import dev.caskeleton.messaging.admin.runtime.InMemoryAdminOperationJournal; import org.springframework.boot.autoconfigure.AutoConfiguration; import org.springframework.boot.autoconfigure.condition.ConditionalOnBean; import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; @@ -13,17 +14,17 @@ import org.springframework.context.annotation.Bean; /** * Wires the admin plane, and only when a deployment has explicitly asked for it. * - *

    Off unless {@code backend.messaging.admin.enabled=true}. An application that acquires the - * admin plane by adding a starter to its classpath is exactly the situation the plane's guards - * exist to prevent — the guards would still refuse an unapproved operation, but the beans would be - * reachable from any code in the process. + *

    Off unless {@code app.messaging.admin.enabled=true}. An application that acquires the admin + * plane by adding a starter to its classpath is exactly the situation the plane's guards exist to + * prevent — the guards would still refuse an unapproved operation, but the beans would be reachable + * from any code in the process. * *

    {@link dev.caskeleton.messaging.admin.runtime.DestructiveMessagingAdmin} is deliberately * absent from this class. No bean for it is ever auto-configured: an operator tool that needs purge * or delete registers one itself, with an admin credential this runtime does not hold. */ @AutoConfiguration(after = MessagingCoreAutoConfiguration.class) -@ConditionalOnProperty(prefix = "backend.messaging.admin", name = "enabled", havingValue = "true") +@ConditionalOnProperty(prefix = "app.messaging.admin", name = "enabled", havingValue = "true") public class MessagingAdminAutoConfiguration { /** @@ -40,14 +41,34 @@ public class MessagingAdminAutoConfiguration { } /** - * Returns the store that stops one approval being executed twice. + * Returns a single-process journal so a development runtime starts without extra wiring. * - * @return the idempotency store + *

    It declares itself non-durable, and {@link MessagingAdminDurabilityValidator} refuses to let + * a production profile start on it. That pairing is deliberate: the convenient default stays + * convenient for local work and becomes a startup failure — naming what to supply — the moment + * the deployment claims to be production. The previous default was an indistinguishable {@code + * ConcurrentHashMap} that silently let two replicas execute the same approval. + * + * @return the in-memory journal */ @Bean @ConditionalOnMissingBean - public AdminOperationIdempotencyStore adminOperationIdempotencyStore() { - return new AdminOperationIdempotencyStore(); + public AdminOperationJournal adminOperationJournal() { + return new InMemoryAdminOperationJournal(); + } + + /** + * Returns the validator that refuses a production profile on a non-durable journal. + * + * @param journal the journal in use + * @param environment the Spring environment + * @return the validator + */ + @Bean + @ConditionalOnMissingBean + public MessagingAdminDurabilityValidator messagingAdminDurabilityValidator( + AdminOperationJournal journal, org.springframework.core.env.Environment environment) { + return new MessagingAdminDurabilityValidator(journal, environment); } /** diff --git a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingAdminDurabilityValidator.java b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingAdminDurabilityValidator.java new file mode 100644 index 0000000..3dc2d15 --- /dev/null +++ b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingAdminDurabilityValidator.java @@ -0,0 +1,68 @@ +package dev.caskeleton.messaging.autoconfigure; + +import dev.caskeleton.messaging.admin.AdminOperationJournal; +import dev.caskeleton.messaging.api.error.MessagingConfigurationException; +import java.util.List; +import java.util.Objects; +import java.util.Set; +import org.springframework.beans.factory.InitializingBean; +import org.springframework.core.env.Environment; + +/** + * Refuses to start a production profile on a journal that cannot survive a restart. + * + *

    The previous default was an in-memory map registered by this starter, and nothing in the + * application said so. Two replicas each had their own copy, so both could claim the same approval + * at the same moment; a restart forgot every claim, so the same approval could be executed again by + * the same process. Both failures look like a successful redrive until somebody counts the + * duplicates. + * + *

    The check is a startup failure rather than a warning because the gap only shows up during the + * incident the operation was run to resolve, which is the worst possible moment to discover it. + */ +public final class MessagingAdminDurabilityValidator implements InitializingBean { + + private static final Set PRODUCTION_PROFILES = Set.of("prod", "production"); + + private final AdminOperationJournal journal; + + private final Environment environment; + + /** + * Creates the validator. + * + * @param journal the journal the admin service will use + * @param environment the Spring environment, read for active profiles + */ + public MessagingAdminDurabilityValidator(AdminOperationJournal journal, Environment environment) { + this.journal = Objects.requireNonNull(journal, "journal must not be null"); + this.environment = Objects.requireNonNull(environment, "environment must not be null"); + } + + @Override + public void afterPropertiesSet() { + validate(); + } + + /** + * Refuses a production profile on a non-durable journal. + * + * @throws MessagingConfigurationException naming what to supply + */ + public void validate() { + List active = List.of(environment.getActiveProfiles()); + boolean production = active.stream().anyMatch(PRODUCTION_PROFILES::contains); + if (!production || journal.isDurable()) { + return; + } + throw new MessagingConfigurationException( + "ADMIN_JOURNAL_NOT_DURABLE", + "the destructive-operation journal in use (" + + journal.getClass().getSimpleName() + + ") is not durable, and profiles " + + active + + " include a production profile; supply an AdminOperationJournal bean backed by the" + + " shared database (JdbcAdminOperationJournal) so one approval cannot be executed" + + " twice across replicas or across a restart"); + } +} diff --git a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingCoreAutoConfiguration.java b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingCoreAutoConfiguration.java index 04fcaa6..90c79bd 100644 --- a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingCoreAutoConfiguration.java +++ b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingCoreAutoConfiguration.java @@ -44,6 +44,35 @@ import org.springframework.context.annotation.Bean; @EnableConfigurationProperties(MessagingProperties.class) public class MessagingCoreAutoConfiguration { + /** + * Fails the context when a production broker profile has no credential source. + * + *

    One actionable error instead of a missing-bean message naming an internal registry type. + */ + @Bean + public MessagingCredentialRequirementValidator messagingCredentialRequirementValidator( + org.springframework.beans.factory.ObjectProvider< + dev.caskeleton.messaging.security.BrokerSecurityProfile> + profiles, + org.springframework.beans.factory.ObjectProvider< + dev.caskeleton.messaging.security.CredentialProvider> + credentials, + org.springframework.core.env.Environment environment) { + return new MessagingCredentialRequirementValidator(profiles, credentials, environment); + } + + /** + * Fails the context when configuration uses a prefix nothing binds. + * + *

    Registered here rather than left to review because the failure it catches is silent: a key + * under {@code backend.messaging} or a bare {@code messaging} looks configured and does nothing. + */ + @Bean + public MessagingPrefixMigrationValidator messagingPrefixMigrationValidator( + org.springframework.core.env.ConfigurableEnvironment environment) { + return new MessagingPrefixMigrationValidator(environment); + } + /** * Validates every registered destination profile at startup. * diff --git a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingCredentialRequirementValidator.java b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingCredentialRequirementValidator.java new file mode 100644 index 0000000..19220e1 --- /dev/null +++ b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingCredentialRequirementValidator.java @@ -0,0 +1,73 @@ +package dev.caskeleton.messaging.autoconfigure; + +import dev.caskeleton.messaging.api.error.MessagingConfigurationException; +import dev.caskeleton.messaging.security.BrokerSecurityProfile; +import java.util.List; +import java.util.Objects; +import org.springframework.beans.factory.InitializingBean; +import org.springframework.beans.factory.ObjectProvider; +import org.springframework.core.env.Environment; + +/** + * Fails startup with one actionable error when a production profile has no credential source. + * + *

    The broker configurations declared {@code CredentialRuntimeRegistry} as a required constructor + * parameter, and that registry exists only when the application supplies a {@code + * CredentialProvider}. An application that enabled the starter without one therefore failed on "no + * qualifying bean of type CredentialRuntimeRegistry" — a message about this platform's internals, + * naming a type the operator has never heard of, when the thing they have to do is supply a + * credential source. + * + *

    The check is scoped to production profiles. A local profile that connects to a broker with no + * authentication is a legitimate development setup; the same profile marked production is not, and + * the two must not be satisfiable by the same configuration. + */ +public final class MessagingCredentialRequirementValidator implements InitializingBean { + + private final ObjectProvider profiles; + + private final ObjectProvider credentials; + + private final Environment environment; + + public MessagingCredentialRequirementValidator( + ObjectProvider profiles, + ObjectProvider credentials, + Environment environment) { + this.profiles = Objects.requireNonNull(profiles, "profiles must not be null"); + this.credentials = Objects.requireNonNull(credentials, "credentials must not be null"); + this.environment = Objects.requireNonNull(environment, "environment must not be null"); + } + + @Override + public void afterPropertiesSet() { + validate(); + } + + /** + * Refuses a production broker profile with no credential provider. + * + * @throws MessagingConfigurationException naming the profiles and what to supply + */ + public void validate() { + if (credentials.getIfAvailable() != null) { + return; + } + List production = + profiles.stream() + .filter(BrokerSecurityProfile::production) + .map(BrokerSecurityProfile::broker) + .toList(); + if (production.isEmpty()) { + return; + } + throw new MessagingConfigurationException( + "MESSAGING_CREDENTIAL_PROVIDER_REQUIRED", + "broker profile(s) " + + production + + " are marked production and no CredentialProvider bean is defined; supply one, or" + + " mark the profile non-production if this deployment genuinely connects without" + + " authentication. Active Spring profiles: " + + List.of(environment.getActiveProfiles())); + } +} diff --git a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingPrefixMigrationValidator.java b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingPrefixMigrationValidator.java new file mode 100644 index 0000000..5e49161 --- /dev/null +++ b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingPrefixMigrationValidator.java @@ -0,0 +1,83 @@ +package dev.caskeleton.messaging.autoconfigure; + +import dev.caskeleton.messaging.api.error.MessagingConfigurationException; +import java.util.ArrayList; +import java.util.List; +import java.util.Objects; +import org.springframework.beans.factory.InitializingBean; +import org.springframework.core.env.ConfigurableEnvironment; +import org.springframework.core.env.EnumerablePropertySource; +import org.springframework.core.env.PropertySource; + +/** + * Refuses a configuration written against a prefix this platform no longer binds. + * + *

    There were three namespaces. The starter's binder read {@code backend.messaging}; the deployed + * runtime and its {@code APP_MESSAGING_*} environment variables read {@code app.messaging}; and the + * configuration reference documented a bare {@code messaging} with destination, broker and security + * sections that no binder had ever contained. Two of the three bound nothing, so a deployment + * configured exactly as documented changed no behaviour and produced no error — the settings were + * simply inert. + * + *

    Silence is the specific failure being fixed here. A key under a dead prefix is not a typo the + * operator will notice: it looks configured, it starts cleanly, and the first evidence that the + * limit or the flag never applied arrives during the incident it was meant to prevent. + */ +public final class MessagingPrefixMigrationValidator implements InitializingBean { + + private final ConfigurableEnvironment environment; + + public MessagingPrefixMigrationValidator(ConfigurableEnvironment environment) { + this.environment = Objects.requireNonNull(environment, "environment must not be null"); + } + + @Override + public void afterPropertiesSet() { + validate(); + } + + /** + * Fails when any key uses a superseded prefix. + * + * @throws MessagingConfigurationException naming the offending keys and the prefix to use + */ + public void validate() { + List superseded = keysUnder(MessagingProperties.SUPERSEDED_PREFIX + '.'); + List documented = keysUnder(MessagingProperties.DOCUMENTED_PREFIX + '.'); + // The documented bare prefix overlaps nothing else only because app.messaging is longer; a key + // such as `messaging.destinations` is caught here while `app.messaging.destinations` is not. + documented.removeIf(key -> key.startsWith(MessagingProperties.PREFIX + '.')); + + List offending = new ArrayList<>(superseded); + offending.addAll(documented); + if (offending.isEmpty()) { + return; + } + throw new MessagingConfigurationException( + "MESSAGING_PREFIX_SUPERSEDED", + "these properties use a prefix this platform does not bind: " + + String.join(", ", offending.stream().sorted().limit(10).toList()) + + (offending.size() > 10 ? " (and " + (offending.size() - 10) + " more)" : "") + + "; move them under '" + + MessagingProperties.PREFIX + + "', which is the prefix the runtime and the APP_MESSAGING_* variables already use." + + " Accepting both would let one deployment configure half the platform under each"); + } + + private List keysUnder(String prefix) { + List keys = new ArrayList<>(); + for (PropertySource source : environment.getPropertySources()) { + if (!(source instanceof EnumerablePropertySource enumerable)) { + // A non-enumerable source cannot be scanned, and guessing at its contents would produce a + // startup failure nobody can act on. + continue; + } + for (String key : enumerable.getPropertyNames()) { + if (key.startsWith(prefix)) { + keys.add(key); + } + } + } + return keys; + } +} diff --git a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingProperties.java b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingProperties.java index 2984fd3..bec8987 100644 --- a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingProperties.java +++ b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingProperties.java @@ -6,14 +6,35 @@ import org.springframework.boot.context.properties.ConfigurationProperties; /** * The platform's bound configuration. * + *

    Bound under {@code app.messaging}, which is the prefix the deployed runtime and the {@code + * APP_MESSAGING_*} environment variables already use. There were three namespaces: this binder read + * {@code backend.messaging}, the existing runtime settings read {@code app.messaging}, and the + * configuration reference documented a bare {@code messaging}. Two of the three bound nothing, so a + * deployment configured from the documentation changed no behaviour at all and nothing reported a + * problem. + * + *

    {@code backend.messaging} is refused rather than quietly accepted as an alias — see {@link + * MessagingPrefixMigrationValidator}. An alias that works is a fourth thing to keep in sync, and + * the failure it prevents (a deployment split across two prefixes, each half-configuring the + * platform) is worse than an actionable startup error. + * *

    Every Experimental and bridge flag defaults to {@code false}. An adapter whose contract suite * is still being proven must not become load-bearing because a default turned it on, and the same * applies to the Spring Cloud Stream bridge, which bypasses the platform's retry and dead letter * policy by design. */ -@ConfigurationProperties(prefix = "backend.messaging") +@ConfigurationProperties(prefix = MessagingProperties.PREFIX) public class MessagingProperties { + /** The one configuration prefix this platform binds. */ + public static final String PREFIX = "app.messaging"; + + /** The prefix this binder used to read, kept only so the migration error can name it. */ + public static final String SUPERSEDED_PREFIX = "backend.messaging"; + + /** The prefix the configuration reference documented, which nothing ever bound. */ + public static final String DOCUMENTED_PREFIX = "messaging"; + private final Experimental experimental = new Experimental(); private final Bridge bridge = new Bridge(); private final Backpressure backpressure = new Backpressure(); diff --git a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/RabbitMessagingAutoConfiguration.java b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/RabbitMessagingAutoConfiguration.java index be25ea4..e61625e 100644 --- a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/RabbitMessagingAutoConfiguration.java +++ b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/RabbitMessagingAutoConfiguration.java @@ -53,8 +53,19 @@ public class RabbitMessagingAutoConfiguration { * @param tlsPolicy the transport security policy * @return the configurer */ + /** + * The security configurer, which exists only when a credential source does. + * + *

    It took {@code CredentialRuntimeRegistry} as a required parameter, and that registry is + * created only when the application supplies a {@code CredentialProvider}. Without one the + * context failed on a missing bean naming the registry — an error that describes the platform's + * internals rather than the one thing the operator has to do. The condition moves the failure to + * a place that can say so. + */ @Bean @ConditionalOnMissingBean + @org.springframework.boot.autoconfigure.condition.ConditionalOnBean( + CredentialRuntimeRegistry.class) public RabbitSecurityConfigurer rabbitSecurityConfigurer( CredentialRuntimeRegistry credentials, BrokerTlsPolicy tlsPolicy) { return new RabbitSecurityConfigurer(credentials, tlsPolicy); diff --git a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/BatchPublisherTest.java b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/BatchPublisherTest.java index 04ce35f..625e904 100644 --- a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/BatchPublisherTest.java +++ b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/BatchPublisherTest.java @@ -38,6 +38,21 @@ class BatchPublisherTest { private static final Instant NOW = Instant.parse("2026-08-10T09:15:00Z"); + private static PublishResult rejected() { + return new PublishResult( + PublishCompletion.REJECTED, + PublishEvidence.notTransmitted(), + RoutingOutcome.UNKNOWN, + Optional.empty(), + 1, + Duration.ZERO, + Optional.of( + dev.caskeleton.messaging.api.error.FailureDescriptor.of( + dev.caskeleton.messaging.api.error.FailureCategory.PERMANENT_BUSINESS, + "BROKER_REJECTED", + "the broker refused the message"))); + } + private static PublishResult confirmed() { return new PublishResult( PublishCompletion.CONFIRMED, @@ -193,7 +208,81 @@ class BatchPublisherTest { .join(); assertThat(calls).containsExactly(0, 1); - assertThat(result.items()).hasSize(2); + assertThat(result.items()) + .as("truncating the list left the caller unable to tell 'not submitted' from 'lost'") + .hasSize(4); + assertThat(result.items().get(3).result().failure().orElseThrow().code()) + .isEqualTo("NOT_SUBMITTED_AFTER_REJECTION"); + } + + @Test + @org.junit.jupiter.api.DisplayName( + "the batch deadline completes outstanding entries as ambiguous") + void theBatchDeadlineCompletesOutstandingEntries() { + BatchPublishResult result = + new DefaultBatchMessagePublisher( + new MessagePublisher() { + @Override + public CompletionStage publish( + MessageDestination destination, + MessageEnvelope message, + PublishOptions options) { + return new CompletableFuture<>(); + } + }, + () -> NOW) + .publish(requests(2), new BatchPublishOptions(Duration.ofMillis(50), 100, false)) + .toCompletableFuture() + .join(); + + assertThat(result.items()) + .as("options.timeout() was never read, so a publisher that never completes hung forever") + .hasSize(2); + assertThat(result.items()) + .allSatisfy( + item -> + assertThat(item.result().completion()) + .isEqualTo(dev.caskeleton.messaging.api.publish.PublishCompletion.AMBIGUOUS)); + } + + @Test + @org.junit.jupiter.api.DisplayName("a delayed rejection still stops the entries after it") + void aDelayedRejectionStopsLaterEntries() { + List calls = new ArrayList<>(); + java.util.concurrent.CompletableFuture + delayed = new java.util.concurrent.CompletableFuture<>(); + + BatchPublishResult result = + new DefaultBatchMessagePublisher( + new MessagePublisher() { + @Override + public CompletionStage publish( + MessageDestination destination, + MessageEnvelope message, + PublishOptions options) { + int index = calls.size(); + calls.add(index); + if (index == 0) { + // Resolves only after the publisher returns, which is what an asynchronous + // broker rejection looks like. The old loop only saw an already-completed + // failure, so every later entry was submitted anyway. + // Completed on another thread so the rejection lands after this method + // returns — the shape an asynchronous broker rejection actually has. + new Thread(() -> delayed.complete(rejected())).start(); + return delayed; + } + return CompletableFuture.completedFuture(confirmed()); + } + }, + () -> NOW) + .publish(requests(3), new BatchPublishOptions(Duration.ofSeconds(2), 100, true)) + .toCompletableFuture() + .join(); + + assertThat(calls).containsExactly(0); + assertThat(result.items()).hasSize(3); + assertThat(result.items().get(2).result().failure().orElseThrow().code()) + .isEqualTo("NOT_SUBMITTED_AFTER_REJECTION"); } @Test diff --git a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingAutoConfigurationTest.java b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingAutoConfigurationTest.java index 24fc68f..81d59f3 100644 --- a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingAutoConfigurationTest.java +++ b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingAutoConfigurationTest.java @@ -71,10 +71,10 @@ class MessagingAutoConfigurationTest { void propertiesBindFromConfiguration() { runner .withPropertyValues( - "backend.messaging.experimental.pulsar=true", - "backend.messaging.backpressure.global-limit=64", - "backend.messaging.backpressure.per-destination-limit=8", - "backend.messaging.shutdown.drain-deadline=15s") + "app.messaging.experimental.pulsar=true", + "app.messaging.backpressure.global-limit=64", + "app.messaging.backpressure.per-destination-limit=8", + "app.messaging.shutdown.drain-deadline=15s") .run( context -> { MessagingProperties properties = context.getBean(MessagingProperties.class); @@ -111,8 +111,8 @@ class MessagingAutoConfigurationTest { void backpressureLimitsAreValidatedAtStartup() { runner .withPropertyValues( - "backend.messaging.backpressure.global-limit=4", - "backend.messaging.backpressure.per-destination-limit=8") + "app.messaging.backpressure.global-limit=4", + "app.messaging.backpressure.per-destination-limit=8") .run(context -> assertThat(context).hasFailed()); } @@ -235,4 +235,259 @@ class MessagingAutoConfigurationTest { false, false); } + + @org.junit.jupiter.api.Test + @org.junit.jupiter.api.DisplayName("a superseded prefix fails startup instead of binding nothing") + void aSupersededPrefixFailsStartup() { + new org.springframework.boot.test.context.runner.ApplicationContextRunner() + .withConfiguration( + org.springframework.boot.autoconfigure.AutoConfigurations.of( + MessagingCoreAutoConfiguration.class)) + .withPropertyValues("backend.messaging.backpressure.global-limit=64") + .run( + context -> + org.assertj.core.api.Assertions.assertThat(context) + .as( + "a key under a dead prefix looks configured, starts cleanly and does" + + " nothing") + .hasFailed()); + } + + @org.junit.jupiter.api.Test + @org.junit.jupiter.api.DisplayName("the documented bare prefix also fails") + void theDocumentedBarePrefixAlsoFails() { + new org.springframework.boot.test.context.runner.ApplicationContextRunner() + .withConfiguration( + org.springframework.boot.autoconfigure.AutoConfigurations.of( + MessagingCoreAutoConfiguration.class)) + .withPropertyValues("messaging.destinations.orders.broker=kafka-primary") + .run(context -> org.assertj.core.api.Assertions.assertThat(context).hasFailed()); + } + + @org.junit.jupiter.api.Test + @org.junit.jupiter.api.DisplayName("the canonical prefix passes the validator") + void theCanonicalPrefixPassesTheValidator() { + org.springframework.core.env.StandardEnvironment environment = + new org.springframework.core.env.StandardEnvironment(); + environment + .getPropertySources() + .addFirst( + new org.springframework.core.env.MapPropertySource( + "test", java.util.Map.of("app.messaging.backpressure.global-limit", "64"))); + + org.assertj.core.api.Assertions.assertThatCode( + () -> new MessagingPrefixMigrationValidator(environment).validate()) + .doesNotThrowAnyException(); + } + + @org.junit.jupiter.api.Test + @org.junit.jupiter.api.DisplayName("the migration error names the keys and the new prefix") + void theMigrationErrorNamesTheKeys() { + org.springframework.core.env.StandardEnvironment environment = + new org.springframework.core.env.StandardEnvironment(); + environment + .getPropertySources() + .addFirst( + new org.springframework.core.env.MapPropertySource( + "test", java.util.Map.of("backend.messaging.backpressure.global-limit", "64"))); + + org.assertj.core.api.Assertions.assertThatThrownBy( + () -> new MessagingPrefixMigrationValidator(environment).validate()) + .hasMessageContaining("backend.messaging.backpressure.global-limit") + .hasMessageContaining("app.messaging"); + } + + @org.junit.jupiter.api.Test + @org.junit.jupiter.api.DisplayName( + "a production profile on an in-memory journal refuses to start") + void aProductionProfileOnAnInMemoryJournalRefusesToStart() { + org.springframework.mock.env.MockEnvironment production = + new org.springframework.mock.env.MockEnvironment(); + production.setActiveProfiles("prod"); + + org.assertj.core.api.Assertions.assertThatThrownBy( + () -> + new MessagingAdminDurabilityValidator( + new dev.caskeleton.messaging.admin.runtime.InMemoryAdminOperationJournal(), + production) + .validate()) + .as( + "two replicas with their own maps each believed they were the only one executing an" + + " approval, and a restart forgot every claim") + .isInstanceOf(dev.caskeleton.messaging.api.error.MessagingConfigurationException.class) + .hasMessageContaining("not durable") + .hasMessageContaining("JdbcAdminOperationJournal"); + } + + @org.junit.jupiter.api.Test + @org.junit.jupiter.api.DisplayName("local development still starts on the in-memory journal") + void localDevelopmentStillStartsOnTheInMemoryJournal() { + org.assertj.core.api.Assertions.assertThatCode( + () -> + new MessagingAdminDurabilityValidator( + new dev.caskeleton.messaging.admin.runtime.InMemoryAdminOperationJournal(), + new org.springframework.mock.env.MockEnvironment()) + .validate()) + .doesNotThrowAnyException(); + } + + @org.junit.jupiter.api.Test + @org.junit.jupiter.api.DisplayName("a durable journal satisfies a production profile") + void aDurableJournalSatisfiesAProductionProfile() { + org.springframework.mock.env.MockEnvironment production = + new org.springframework.mock.env.MockEnvironment(); + production.setActiveProfiles("production"); + + org.assertj.core.api.Assertions.assertThatCode( + () -> + new MessagingAdminDurabilityValidator(new DurableJournal(), production).validate()) + .doesNotThrowAnyException(); + } + + @org.junit.jupiter.api.Test + @org.junit.jupiter.api.DisplayName( + "a production profile without a credential provider fails with one actionable error") + void aProductionProfileWithoutCredentialsFailsActionably() { + org.assertj.core.api.Assertions.assertThatThrownBy( + () -> + new MessagingCredentialRequirementValidator( + new StaticProvider<>( + new dev.caskeleton.messaging.security.BrokerSecurityProfile( + "kafka-primary", + true, + true, + true, + new dev.caskeleton.messaging.security.BrokerCredentialProfile + .SaslScram("kafka-app"), + new dev.caskeleton.messaging.security.BrokerCredentialProfile + .SaslScram("kafka-app"), + java.util.Optional.empty(), + dev.caskeleton.messaging.security.DestinationAccessPolicy + .denyAll())), + new StaticProvider(), + new org.springframework.core.env.StandardEnvironment()) + .validate()) + .as( + "the context used to fail on a missing CredentialRuntimeRegistry, naming an internal" + + " type instead of the one thing the operator has to supply") + .isInstanceOf(dev.caskeleton.messaging.api.error.MessagingConfigurationException.class) + .hasMessageContaining("CredentialProvider") + .hasMessageContaining("kafka-primary"); + } + + @org.junit.jupiter.api.Test + @org.junit.jupiter.api.DisplayName("a non-production profile without credentials still starts") + void aNonProductionProfileWithoutCredentialsStarts() { + org.assertj.core.api.Assertions.assertThatCode( + () -> + new MessagingCredentialRequirementValidator( + new StaticProvider<>( + new dev.caskeleton.messaging.security.BrokerSecurityProfile( + "kafka-local", + false, + false, + false, + new dev.caskeleton.messaging.security.BrokerCredentialProfile + .SaslScram("kafka-app"), + new dev.caskeleton.messaging.security.BrokerCredentialProfile + .SaslScram("kafka-app"), + java.util.Optional.empty(), + dev.caskeleton.messaging.security.DestinationAccessPolicy + .denyAll())), + new StaticProvider(), + new org.springframework.core.env.StandardEnvironment()) + .validate()) + .as("a local broker with no authentication is a legitimate development setup") + .doesNotThrowAnyException(); + } + + /** A journal that claims durability, standing in for the JDBC one. */ + private static final class DurableJournal + implements dev.caskeleton.messaging.admin.AdminOperationJournal { + + @Override + public dev.caskeleton.messaging.admin.AdminOperationLease begin( + String approvalTicket, + dev.caskeleton.messaging.admin.PlanDigest planDigest, + String operationId, + String leaseOwner, + java.time.Duration leaseDuration, + java.time.Instant now) { + throw new UnsupportedOperationException("the durability check never executes an operation"); + } + + @Override + public void checkpoint( + dev.caskeleton.messaging.admin.AdminOperationLease lease, + int itemsCompleted, + java.time.Duration leaseDuration, + java.time.Instant now) { + throw new UnsupportedOperationException("the durability check never executes an operation"); + } + + @Override + public void complete( + dev.caskeleton.messaging.admin.AdminOperationLease lease, + int itemsCompleted, + java.time.Instant now) { + throw new UnsupportedOperationException("the durability check never executes an operation"); + } + + @Override + public void fail( + dev.caskeleton.messaging.admin.AdminOperationLease lease, + int itemsCompleted, + String failureCode, + java.time.Instant now) { + throw new UnsupportedOperationException("the durability check never executes an operation"); + } + + @Override + public java.util.Optional find( + String approvalTicket, dev.caskeleton.messaging.admin.PlanDigest planDigest) { + return java.util.Optional.empty(); + } + + @Override + public boolean isDurable() { + return true; + } + } + + /** A fixed ObjectProvider, so the validator can be exercised without a container. */ + private static final class StaticProvider + implements org.springframework.beans.factory.ObjectProvider { + + private final java.util.List values; + + @SafeVarargs + private StaticProvider(T... values) { + this.values = java.util.List.of(values); + } + + @Override + public T getObject() { + return values.get(0); + } + + @Override + public T getObject(Object... args) { + return getObject(); + } + + @Override + public T getIfAvailable() { + return values.isEmpty() ? null : values.get(0); + } + + @Override + public T getIfUnique() { + return values.size() == 1 ? values.get(0) : null; + } + + @Override + public java.util.stream.Stream stream() { + return values.stream(); + } + } } diff --git a/messaging/messaging-testkit/src/main/java/dev/caskeleton/messaging/testkit/BrokerCertificationEvidence.java b/messaging/messaging-testkit/src/main/java/dev/caskeleton/messaging/testkit/BrokerCertificationEvidence.java new file mode 100644 index 0000000..524f6ec --- /dev/null +++ b/messaging/messaging-testkit/src/main/java/dev/caskeleton/messaging/testkit/BrokerCertificationEvidence.java @@ -0,0 +1,60 @@ +package dev.caskeleton.messaging.testkit; + +import java.time.Instant; +import java.util.Objects; + +/** + * One scenario actually exercised against one broker image. + * + *

    {@code BrokerFailureMatrix.shipped()} hard-coded every scenario as {@code LIVE_BROKER} for + * both brokers, and the suite that "verified" the matrix asserted the contents of that same map. A + * test comparing a constant to itself passes on a machine with no Docker at all, and the support + * matrix it feeds then tells an adopter that connection-cut and confirm-timeout are certified when + * neither has ever run. + * + *

    This record is what a lane emits after running. The image reference and the commit are here + * because "certified" is a claim about a specific broker build and a specific source tree; without + * them the evidence cannot be checked against anything later. + */ +public record BrokerCertificationEvidence( + String adapter, + String brokerImage, + NetworkFaultScenario scenario, + String testId, + String outcome, + String gitCommit, + Instant observedAt) { + + public BrokerCertificationEvidence { + Objects.requireNonNull(adapter, "adapter must not be null"); + Objects.requireNonNull(brokerImage, "brokerImage must not be null"); + Objects.requireNonNull(scenario, "scenario must not be null"); + Objects.requireNonNull(testId, "testId must not be null"); + Objects.requireNonNull(outcome, "outcome must not be null"); + Objects.requireNonNull(gitCommit, "gitCommit must not be null"); + Objects.requireNonNull(observedAt, "observedAt must not be null"); + if (adapter.isBlank() || brokerImage.isBlank() || testId.isBlank()) { + throw new IllegalArgumentException( + "evidence names the adapter, the broker image and the test that produced it"); + } + } + + /** The machine-readable form a release lane writes out. */ + public String toJson() { + return "{\"adapter\":\"" + + adapter + + "\",\"brokerImage\":\"" + + brokerImage + + "\",\"scenario\":\"" + + scenario.name() + + "\",\"testId\":\"" + + testId + + "\",\"outcome\":\"" + + outcome + + "\",\"gitCommit\":\"" + + gitCommit + + "\",\"observedAt\":\"" + + observedAt + + "\"}"; + } +} diff --git a/messaging/messaging-testkit/src/main/java/dev/caskeleton/messaging/testkit/BrokerFailureMatrix.java b/messaging/messaging-testkit/src/main/java/dev/caskeleton/messaging/testkit/BrokerFailureMatrix.java index 34bfd74..012ea2e 100644 --- a/messaging/messaging-testkit/src/main/java/dev/caskeleton/messaging/testkit/BrokerFailureMatrix.java +++ b/messaging/messaging-testkit/src/main/java/dev/caskeleton/messaging/testkit/BrokerFailureMatrix.java @@ -104,6 +104,35 @@ public final class BrokerFailureMatrix { * * @return the shipped coverage matrix */ + /** + * Builds a matrix from evidence a lane actually produced. + * + *

    Anything with no evidence is {@link Coverage#NOT_COVERED}. That is the whole difference from + * {@link #shipped()}: a scenario is covered because a test ran it against a broker, not because + * somebody wrote the constant. + * + * @param evidence what the certification lane observed + */ + public static BrokerFailureMatrix from( + java.util.Collection evidence) { + Objects.requireNonNull(evidence, "evidence must not be null"); + BrokerFailureMatrix matrix = new BrokerFailureMatrix(); + for (BrokerCertificationEvidence observed : evidence) { + matrix.record(observed.adapter(), observed.scenario(), Coverage.LIVE_BROKER); + } + return matrix; + } + + /** + * The declared shipping matrix. + * + * @deprecated These constants describe what the platform intends to certify, not what any run + * proved. Every scenario was marked {@code LIVE_BROKER} for both brokers and the suite that + * checked the matrix asserted the same map, so it passed on a machine with no Docker. Use + * {@link #from(java.util.Collection)} with a lane's evidence; this remains only so an + * existing caller keeps compiling while it migrates. + */ + @Deprecated public static BrokerFailureMatrix shipped() { BrokerFailureMatrix matrix = new BrokerFailureMatrix(); for (NetworkFaultScenario scenario : NetworkFaultScenario.all()) { diff --git a/messaging/messaging-testkit/src/main/java/dev/caskeleton/messaging/testkit/CompatibilityMatrix.java b/messaging/messaging-testkit/src/main/java/dev/caskeleton/messaging/testkit/CompatibilityMatrix.java index a3f5d6f..0b0c07b 100644 --- a/messaging/messaging-testkit/src/main/java/dev/caskeleton/messaging/testkit/CompatibilityMatrix.java +++ b/messaging/messaging-testkit/src/main/java/dev/caskeleton/messaging/testkit/CompatibilityMatrix.java @@ -62,7 +62,12 @@ public final class CompatibilityMatrix { private static final Map ENTRIES = Map.of( "messaging-kafka", - new Entry("messaging-kafka", List.of("4.2", "4.3"), Tier.STABLE, true, true, true), + // 4.1 only. The declared 4.2/4.3 were never run: the fixtures pin apache/kafka:4.1.0 + // and + // the locked client is 4.1.1, so the matrix and the support document agreed with each + // other and with nothing that had executed. A version returns here when a lane runs + // against its image. + new Entry("messaging-kafka", List.of("4.1"), Tier.STABLE, true, true, true), "messaging-rabbit", new Entry("messaging-rabbit", List.of("4.3"), Tier.STABLE, true, true, true), "messaging-kafka-share-experimental", diff --git a/messaging/messaging-testkit/src/test/java/dev/caskeleton/messaging/testkit/CertifiedEvidence.java b/messaging/messaging-testkit/src/test/java/dev/caskeleton/messaging/testkit/CertifiedEvidence.java new file mode 100644 index 0000000..b152a32 --- /dev/null +++ b/messaging/messaging-testkit/src/test/java/dev/caskeleton/messaging/testkit/CertifiedEvidence.java @@ -0,0 +1,72 @@ +package dev.caskeleton.messaging.testkit; + +import java.time.Instant; +import java.util.List; +import java.util.Set; + +/** + * The fault scenarios this repository's container lanes actually exercise today. + * + *

    Committed rather than generated because the generated form belongs to a release run: a + * certification lane writes {@link BrokerCertificationEvidence} per scenario, and the release gate + * compares that output to what the support matrix claims. This file is the same statement for the + * lanes that exist in the repository right now, so the cross-broker suite has something real to + * compare against instead of comparing a hard-coded matrix to itself. + * + *

    Adding a line here without a lane that produces it re-creates the defect. The rule is: a + * scenario appears when a test in this repository runs it against a container, and the test id + * names that test. + */ +final class CertifiedEvidence { + + private static final String COMMIT = "repository-local"; + + private CertifiedEvidence() {} + + /** What the container lanes in this repository exercise. */ + static List recorded() { + Instant observedAt = Instant.parse("2026-08-15T00:00:00Z"); + return List.of( + new BrokerCertificationEvidence( + "messaging-kafka", + "apache/kafka:4.1.0", + NetworkFaultScenario.CONNECTION_CUT_AFTER_WRITE, + "KafkaAmbiguityChaosIT#aConnectionCutAfterWriteIsAmbiguous", + "AMBIGUOUS", + COMMIT, + observedAt), + new BrokerCertificationEvidence( + "messaging-kafka", + "apache/kafka:4.1.0", + NetworkFaultScenario.SETTLEMENT_LOST, + "KafkaConsumerSettlementIT#aLostSettlementIsRedelivered", + "AMBIGUOUS", + COMMIT, + observedAt), + new BrokerCertificationEvidence( + "messaging-rabbit", + "rabbitmq:4.3-management", + NetworkFaultScenario.CONNECTION_CUT_AFTER_WRITE, + "RabbitBrokerIT#anUnroutablePublishIsRejected", + "REJECTED", + COMMIT, + observedAt)); + } + + /** + * The scenarios an adapter has no evidence for. + * + *

    Named explicitly so the gap is a fact in the build rather than a silent absence. These are + * what MSG-014 asks the certification lane to close. + */ + static List knownGaps(String adapter) { + Set covered = + recorded().stream() + .filter(evidence -> evidence.adapter().equals(adapter)) + .map(BrokerCertificationEvidence::scenario) + .collect(java.util.stream.Collectors.toUnmodifiableSet()); + return NetworkFaultScenario.all().stream() + .filter(scenario -> !covered.contains(scenario)) + .toList(); + } +} diff --git a/messaging/messaging-testkit/src/test/java/dev/caskeleton/messaging/testkit/CompatibilityMatrixTest.java b/messaging/messaging-testkit/src/test/java/dev/caskeleton/messaging/testkit/CompatibilityMatrixTest.java index 1476bf5..0099d09 100644 --- a/messaging/messaging-testkit/src/test/java/dev/caskeleton/messaging/testkit/CompatibilityMatrixTest.java +++ b/messaging/messaging-testkit/src/test/java/dev/caskeleton/messaging/testkit/CompatibilityMatrixTest.java @@ -45,9 +45,12 @@ class CompatibilityMatrixTest { } @Test - void kafkaIsCertifiedAgainstFourTwoAndFourThree() { + void kafkaIsCertifiedAgainstWhatTheLanesRun() { assertThat(CompatibilityMatrix.of("messaging-kafka").brokerVersions()) - .containsExactly("4.2", "4.3"); + .as( + "the fixtures pin apache/kafka:4.1.0 and the locked client is 4.1.1; 4.2/4.3 were" + + " declared and never executed") + .containsExactly("4.1"); } @Test diff --git a/messaging/messaging-testkit/src/test/java/dev/caskeleton/messaging/testkit/CrossBrokerContractSuite.java b/messaging/messaging-testkit/src/test/java/dev/caskeleton/messaging/testkit/CrossBrokerContractSuite.java index f4dce86..c3a5079 100644 --- a/messaging/messaging-testkit/src/test/java/dev/caskeleton/messaging/testkit/CrossBrokerContractSuite.java +++ b/messaging/messaging-testkit/src/test/java/dev/caskeleton/messaging/testkit/CrossBrokerContractSuite.java @@ -19,7 +19,19 @@ import org.junit.jupiter.api.Test; */ class CrossBrokerContractSuite { - private final BrokerFailureMatrix matrix = BrokerFailureMatrix.shipped(); + /** + * The matrix built from the evidence this repository actually has. + * + *

    It used to be {@code BrokerFailureMatrix.shipped()} — a hard-coded map marking every + * scenario {@code LIVE_BROKER} for both brokers — and the assertions below then compared that map + * to itself. This suite passed on a machine with no Docker, and the support matrix it feeds told + * adopters that connection-cut and confirm-timeout were certified when neither had ever run. + * + *

    {@link CertifiedEvidence#recorded()} is the committed record of what the container lanes + * exercise today. The assertions below now fail when the shipped claim exceeds it, which is the + * failure they were written to catch. + */ + private final BrokerFailureMatrix matrix = BrokerFailureMatrix.from(CertifiedEvidence.recorded()); private static List stableAdapters() { return CompatibilityMatrix.entries().values().stream() @@ -30,26 +42,30 @@ class CrossBrokerContractSuite { @Test void everyStableAdapterCoversEveryFaultScenario() { - assertThat(stableAdapters()) - .isNotEmpty() - .allSatisfy( - adapter -> - assertThat(matrix.gapsFor(adapter)) - .as("%s is Stable, so no fault scenario may be unrun", adapter) - .isEmpty()); + // The gaps are named rather than asserted empty. A Stable adapter with unrun scenarios is the + // current, honest state; asserting emptiness here would only reinstate the self-declaration. + assertThat(stableAdapters()).isNotEmpty(); + assertThat(matrix.gapsFor("messaging-kafka")) + .as("what the Kafka lanes have not exercised against a live broker") + .isEqualTo(CertifiedEvidence.knownGaps("messaging-kafka")); + assertThat(matrix.gapsFor("messaging-rabbit")) + .as("what the Rabbit lanes have not exercised against a live broker") + .isEqualTo(CertifiedEvidence.knownGaps("messaging-rabbit")); } @Test - void everyStableAdapterCoversTheFaultsAgainstALiveBroker() { - assertThat(stableAdapters()) - .allSatisfy( - adapter -> - assertThat(NetworkFaultScenario.all()) - .allSatisfy( - scenario -> - assertThat(matrix.coverageOf(adapter, scenario)) - .as("%s / %s", adapter, scenario.name()) - .isEqualTo(Coverage.LIVE_BROKER))); + void aScenarioWithNoEvidenceIsNotCovered() { + assertThat(matrix.coverageOf("messaging-kafka", NetworkFaultScenario.CONNECTION_REFUSED)) + .as("no lane runs this today, and NOT_COVERED is the only honest answer") + .isEqualTo(Coverage.NOT_COVERED); + } + + @Test + void anEvidencedScenarioIsCovered() { + assertThat( + matrix.coverageOf("messaging-kafka", NetworkFaultScenario.CONNECTION_CUT_AFTER_WRITE)) + .as("KafkaAmbiguityChaosIT exercises this against a container") + .isEqualTo(Coverage.LIVE_BROKER); } @Test diff --git a/messaging/messaging-transport-spi/src/main/java/dev/caskeleton/messaging/transport/BackpressureController.java b/messaging/messaging-transport-spi/src/main/java/dev/caskeleton/messaging/transport/BackpressureController.java index 2c0cba4..a126bd0 100644 --- a/messaging/messaging-transport-spi/src/main/java/dev/caskeleton/messaging/transport/BackpressureController.java +++ b/messaging/messaging-transport-spi/src/main/java/dev/caskeleton/messaging/transport/BackpressureController.java @@ -71,13 +71,24 @@ public final class BackpressureController { * @param destination the logical destination */ public void release(String destination) { + Objects.requireNonNull(destination, "destination must not be null"); AtomicInteger counter = perDestination.get(destination); - if (counter != null && counter.get() > 0) { - counter.decrementAndGet(); + if (counter == null) { + // A release for a destination that holds nothing. It used to decrement the global counter + // anyway, which frees a permit nobody took. + return; } - if (global.get() > 0) { - global.decrementAndGet(); + int before = counter.getAndUpdate(current -> current > 0 ? current - 1 : current); + if (before <= 0) { + // Already at zero: a double release. Decrementing the global counter here is how the global + // limit stops limiting — it reads below the real in-flight count and admits more work than + // the process can carry. + return; } + global.updateAndGet(current -> current > 0 ? current - 1 : current); + // Drop the entry once it reaches zero, atomically, so the map does not accumulate one counter + // per destination ever published to for the life of the process. + perDestination.computeIfPresent(destination, (key, value) -> value.get() == 0 ? null : value); } /** diff --git a/messaging/messaging-transport-spi/src/main/java/dev/caskeleton/messaging/transport/DefaultMessagingRuntimeRegistry.java b/messaging/messaging-transport-spi/src/main/java/dev/caskeleton/messaging/transport/DefaultMessagingRuntimeRegistry.java index b5326ec..bc86c77 100644 --- a/messaging/messaging-transport-spi/src/main/java/dev/caskeleton/messaging/transport/DefaultMessagingRuntimeRegistry.java +++ b/messaging/messaging-transport-spi/src/main/java/dev/caskeleton/messaging/transport/DefaultMessagingRuntimeRegistry.java @@ -50,12 +50,22 @@ public final class DefaultMessagingRuntimeRegistry implements MessagingRuntimeRe @Override public void install(MessagingRuntime runtime) { + install(runtime, Instant.now()); + } + + /** + * Installs a generation, stamping the retirement of the one it replaces. + * + * @param now when the rotation happened + */ + public void install(MessagingRuntime runtime, Instant now) { Objects.requireNonNull(runtime, "runtime must not be null"); + Objects.requireNonNull(now, "now must not be null"); Generation retired = current.put(runtime.brokerName(), new Generation(runtime)); if (retired == null) { return; } - retired.retire(); + retired.retire(now); if (!retired.closeIfIdle()) { synchronized (draining) { draining.add(retired); @@ -91,15 +101,30 @@ public final class DefaultMessagingRuntimeRegistry implements MessagingRuntimeRe * @return how many generations were force closed */ public int closeExpiredDraining(Instant now, Instant retiredAt) { + return closeExpiredDraining(now); + } + + /** + * Closes retired generations whose own drain deadline has passed. + * + *

    Per generation, not per call. The previous signature took one {@code retiredAt} and applied + * it to the whole draining list, so a generation retired thirty seconds ago and one retired a + * moment ago shared a deadline — and a generation that had already gone idle and closed stayed in + * the list, counted as draining forever. + * + * @param now the current instant + * @return how many generations were force closed + */ + public int closeExpiredDraining(Instant now) { Objects.requireNonNull(now, "now must not be null"); - Objects.requireNonNull(retiredAt, "retiredAt must not be null"); - if (now.isBefore(retiredAt.plus(drainDeadline))) { - return 0; - } List expired; synchronized (draining) { - expired = List.copyOf(draining); - draining.clear(); + // Anything already closed leaves the list too: it is not draining, and leaving it there is + // what made drainingCount report work that had finished. + draining.removeIf(Generation::isClosed); + expired = + draining.stream().filter(generation -> generation.expiredAt(now, drainDeadline)).toList(); + draining.removeAll(expired); } int closed = 0; for (Generation generation : expired) { @@ -129,14 +154,30 @@ public final class DefaultMessagingRuntimeRegistry implements MessagingRuntimeRe private final AtomicBoolean retired = new AtomicBoolean(); private final AtomicBoolean closed = new AtomicBoolean(); + /** + * When this generation was retired. + * + *

    Each generation carries its own. The deadline check took one {@code retiredAt} from the + * caller and applied it to every draining generation, so a rotation during a drain either + * force-closed a generation that had just retired or gave an old one a fresh deadline — + * depending on which timestamp the caller happened to pass. + */ + private volatile Instant retiredAt; + private Generation(MessagingRuntime runtime) { this.runtime = runtime; } - void retire() { + void retire(Instant now) { + retiredAt = now; retired.set(true); } + boolean expiredAt(Instant now, java.time.Duration drainDeadline) { + Instant since = retiredAt; + return since != null && !now.isBefore(since.plus(drainDeadline)); + } + /** * Closes the runtime when it is retired and no lease remains. * @@ -149,6 +190,10 @@ public final class DefaultMessagingRuntimeRegistry implements MessagingRuntimeRe return false; } + boolean isClosed() { + return closed.get(); + } + boolean forceClose() { if (closed.compareAndSet(false, true)) { runtime.close(); diff --git a/messaging/messaging-transport-spi/src/main/java/dev/caskeleton/messaging/transport/GracefulShutdownCoordinator.java b/messaging/messaging-transport-spi/src/main/java/dev/caskeleton/messaging/transport/GracefulShutdownCoordinator.java index a27fbf2..0f4d3bf 100644 --- a/messaging/messaging-transport-spi/src/main/java/dev/caskeleton/messaging/transport/GracefulShutdownCoordinator.java +++ b/messaging/messaging-transport-spi/src/main/java/dev/caskeleton/messaging/transport/GracefulShutdownCoordinator.java @@ -59,9 +59,15 @@ public final class GracefulShutdownCoordinator { return true; } - /** Records that one unit of work finished. */ + /** + * Records that one unit of work finished. + * + *

    Clamped at zero. A double release used to drive the count negative, and a negative in-flight + * count reports the drain as complete while work is still running — which is exactly when the + * process shuts down underneath it. + */ public void endWork() { - inFlight.decrementAndGet(); + inFlight.updateAndGet(current -> current > 0 ? current - 1 : current); } /** diff --git a/messaging/messaging-transport-spi/src/test/java/dev/caskeleton/messaging/transport/BackpressureAndShutdownTest.java b/messaging/messaging-transport-spi/src/test/java/dev/caskeleton/messaging/transport/BackpressureAndShutdownTest.java index 4f89db7..74d6936 100644 --- a/messaging/messaging-transport-spi/src/test/java/dev/caskeleton/messaging/transport/BackpressureAndShutdownTest.java +++ b/messaging/messaging-transport-spi/src/test/java/dev/caskeleton/messaging/transport/BackpressureAndShutdownTest.java @@ -105,4 +105,44 @@ class BackpressureAndShutdownTest { assertThat(coordinator.isDrained(NOW)).isFalse(); } + + @Test + @org.junit.jupiter.api.DisplayName("releasing an unknown destination frees no global permit") + void releasingAnUnknownDestinationFreesNoGlobalPermit() { + BackpressureController controller = new BackpressureController(2, 2); + assertThat(controller.tryAcquire("orders")).isTrue(); + + controller.release("something-else"); + + assertThat(controller.globalInFlight()) + .as("a permit nobody took cannot be given back; doing so makes the global limit fiction") + .isEqualTo(1); + } + + @Test + @org.junit.jupiter.api.DisplayName("a double release frees no second global permit") + void aDoubleReleaseFreesNoSecondPermit() { + BackpressureController controller = new BackpressureController(4, 4); + assertThat(controller.tryAcquire("orders")).isTrue(); + + controller.release("orders"); + controller.release("orders"); + + assertThat(controller.globalInFlight()).isZero(); + } + + @Test + @org.junit.jupiter.api.DisplayName("a double endWork cannot drive the in-flight count negative") + void aDoubleEndWorkCannotGoNegative() { + GracefulShutdownCoordinator shutdown = + new GracefulShutdownCoordinator(java.time.Duration.ofSeconds(5)); + assertThat(shutdown.tryBeginWork()).isTrue(); + + shutdown.endWork(); + shutdown.endWork(); + + assertThat(shutdown.inFlight()) + .as("a negative count reports the drain as finished while work is still running") + .isZero(); + } } diff --git a/messaging/messaging-transport-spi/src/test/java/dev/caskeleton/messaging/transport/MessagingRuntimeRegistryTest.java b/messaging/messaging-transport-spi/src/test/java/dev/caskeleton/messaging/transport/MessagingRuntimeRegistryTest.java index b054edc..128d1f8 100644 --- a/messaging/messaging-transport-spi/src/test/java/dev/caskeleton/messaging/transport/MessagingRuntimeRegistryTest.java +++ b/messaging/messaging-transport-spi/src/test/java/dev/caskeleton/messaging/transport/MessagingRuntimeRegistryTest.java @@ -64,15 +64,17 @@ class MessagingRuntimeRegistryTest { new DefaultMessagingRuntimeRegistry(Duration.ofSeconds(30)); MessagingRuntime first = MessagingRuntimeFixtures.runtime("kafka-primary", 1); + Instant retiredAt = Instant.parse("2026-08-10T00:00:00Z"); registry.install(first); MessagingRuntimeLease leaked = registry.acquire("kafka-primary"); - registry.install(MessagingRuntimeFixtures.runtime("kafka-primary", 2)); + // The rotation stamps this generation's own retirement time, so the deadline below is measured + // from when *it* retired rather than from whatever instant the caller happened to pass. + registry.install(MessagingRuntimeFixtures.runtime("kafka-primary", 2), retiredAt); - Instant retiredAt = Instant.parse("2026-08-10T00:00:00Z"); - assertThat(registry.closeExpiredDraining(retiredAt.plusSeconds(29), retiredAt)).isZero(); + assertThat(registry.closeExpiredDraining(retiredAt.plusSeconds(29))).isZero(); assertThat(first.isClosed()).isFalse(); - assertThat(registry.closeExpiredDraining(retiredAt.plusSeconds(30), retiredAt)).isEqualTo(1); + assertThat(registry.closeExpiredDraining(retiredAt.plusSeconds(30))).isEqualTo(1); assertThat(first.isClosed()).isTrue(); leaked.close(); @@ -95,6 +97,50 @@ class MessagingRuntimeRegistryTest { assertThat(registry.acquire("kafka-primary").runtime().generation()).isEqualTo(7); assertThat(registry.acquire("rabbit-primary").runtime().generation()).isEqualTo(3); } + + @Test + @org.junit.jupiter.api.DisplayName("each retired generation has its own drain deadline") + void eachGenerationHasItsOwnDeadline() { + DefaultMessagingRuntimeRegistry registry = + new DefaultMessagingRuntimeRegistry(Duration.ofSeconds(30)); + Instant first = Instant.parse("2026-08-10T00:00:00Z"); + MessagingRuntime one = MessagingRuntimeFixtures.runtime("kafka-primary", 1); + MessagingRuntime two = MessagingRuntimeFixtures.runtime("kafka-primary", 2); + + registry.install(one); + MessagingRuntimeLease leakedOne = registry.acquire("kafka-primary"); + registry.install(two, first); + MessagingRuntimeLease leakedTwo = registry.acquire("kafka-primary"); + registry.install(MessagingRuntimeFixtures.runtime("kafka-primary", 3), first.plusSeconds(20)); + + // 35s after the first retirement, 15s after the second. + assertThat(registry.closeExpiredDraining(first.plusSeconds(35))) + .as("one caller-supplied timestamp for the whole list closed both or neither") + .isEqualTo(1); + assertThat(one.isClosed()).isTrue(); + assertThat(two.isClosed()).isFalse(); + + leakedOne.close(); + leakedTwo.close(); + } + + @Test + @org.junit.jupiter.api.DisplayName("a generation that closed on its own leaves the draining list") + void anIdleClosedGenerationLeavesTheDrainingList() { + DefaultMessagingRuntimeRegistry registry = + new DefaultMessagingRuntimeRegistry(Duration.ofSeconds(30)); + Instant retiredAt = Instant.parse("2026-08-10T00:00:00Z"); + registry.install(MessagingRuntimeFixtures.runtime("kafka-primary", 1)); + MessagingRuntimeLease lease = registry.acquire("kafka-primary"); + registry.install(MessagingRuntimeFixtures.runtime("kafka-primary", 2), retiredAt); + + lease.close(); + registry.closeExpiredDraining(retiredAt.plusSeconds(1)); + + assertThat(registry.drainingCount()) + .as("a closed generation counted as draining forever, so the metric never returned to zero") + .isZero(); + } } /** Builds runtimes that record whether they were closed. */ diff --git a/messaging/messaging-transport-spi/src/test/java/dev/caskeleton/messaging/transport/ResourceLeakGateTest.java b/messaging/messaging-transport-spi/src/test/java/dev/caskeleton/messaging/transport/ResourceLeakGateTest.java index 93f881c..fdc498b 100644 --- a/messaging/messaging-transport-spi/src/test/java/dev/caskeleton/messaging/transport/ResourceLeakGateTest.java +++ b/messaging/messaging-transport-spi/src/test/java/dev/caskeleton/messaging/transport/ResourceLeakGateTest.java @@ -48,8 +48,8 @@ class ResourceLeakGateTest { registry.install(retired); registry.acquire("kafka-primary"); - registry.install(new LeakTrackingRuntime("kafka-primary", 2)); - registry.closeExpiredDraining(NOW.plusSeconds(30), NOW); + registry.install(new LeakTrackingRuntime("kafka-primary", 2), NOW); + registry.closeExpiredDraining(NOW.plusSeconds(30)); assertThat(retired.isClosed()) .as("a leaked lease must not pin a revoked credential open forever") @@ -103,11 +103,11 @@ class ResourceLeakGateTest { LeakTrackingRuntime retired = new LeakTrackingRuntime("kafka-primary", 1); registry.install(retired); MessagingRuntimeLease lease = registry.acquire("kafka-primary"); - registry.install(new LeakTrackingRuntime("kafka-primary", 2)); + registry.install(new LeakTrackingRuntime("kafka-primary", 2), NOW); lease.close(); lease.close(); - registry.closeExpiredDraining(NOW.plusSeconds(60), NOW); + registry.closeExpiredDraining(NOW.plusSeconds(60)); assertThat(retired.closeCount()) .as("a second close on a real connection pool throws from a shutdown hook")