feat: jpa, messaging, notification, mongo, graphql 어댑터터 구현체 추가

This commit is contained in:
DongHyeonka
2026-08-15 13:01:58 +09:00
parent e23ed9486e
commit f56dd3e4eb
174 changed files with 10338 additions and 1274 deletions
+20 -6
View File
@@ -1,9 +1,18 @@
# 설정 레퍼런스
> **Prefix.** Every property below binds under `app.messaging`, which is the prefix the deployed
> runtime and the `APP_MESSAGING_*` environment variables already use. Earlier revisions of this
> page documented a bare `messaging` prefix and the starter bound `backend.messaging`; neither
> bound what this page describes, so a deployment configured from it changed nothing. A key under
> either of the old prefixes now fails startup with a message naming the key — see
> `MessagingPrefixMigrationValidator`.
## Destination profile
```yaml
messaging:
app:
messaging:
destinations:
order-events:
broker: kafka-primary
@@ -77,7 +86,8 @@ messaging:
### Kafka
```yaml
messaging:
app:
messaging:
brokers:
kafka-primary:
type: kafka
@@ -96,7 +106,8 @@ messaging:
### RabbitMQ
```yaml
messaging:
app:
messaging:
brokers:
rabbit-primary:
type: rabbitmq
@@ -117,7 +128,8 @@ messaging:
## 보안
```yaml
messaging:
app:
messaging:
security:
kafka-primary:
producer: { type: SASL_SCRAM, credential-id: kafka-producer }
@@ -135,7 +147,8 @@ messaging:
기본값은 전부 `false`다.
```yaml
messaging:
app:
messaging:
experimental:
kafka-share: false
pulsar: false
@@ -147,7 +160,8 @@ messaging:
## Backpressure
```yaml
messaging:
app:
messaging:
backpressure:
global-limit: 512
per-destination-limit: 64 # global-limit 이하여야 한다
+34
View File
@@ -0,0 +1,34 @@
# 기존 runtime → 신규 messaging platform cutover (MSG-015)
## 왜 기계적 매핑이 안 되는가
두 outbox 모델의 enum 이름이 겹치는데 의미가 반대다.
| 모델 | retryable | terminal |
|---|---|---|
| 기존 `OutboxEventStatus` | `FAILED` (`next_attempt_at` 보유) | `DEAD` |
| 신규 `OutboxStatus` | `AMBIGUOUS` | `FAILED`, `EXHAUSTED` |
이름으로 매핑하면 **확정 거절이 무한 재시도**가 되고 **불확정이 park**된다. 그래서 application은
자기 어휘(`OutboxPublishOutcome`)만 쓰고, 변환은 bridge adapter가 한다.
## 지금 반영된 것
- `OutboxPublishOutcome` — `CONFIRMED` / `AMBIGUOUS` / `REJECTED_BEFORE_SEND` /
`REJECTED_AFTER_BROKER`. application이 소유하는 canonical 결과 타입이며, "리턴 or throw"만 가능한
기존 어댑터를 위해 `OutboxMessagePublishPort.publishForOutcome`의 default가 `CONFIRMED`를 돌려준다.
- `APPLICATION_DOES_NOT_DEPEND_ON_THE_MESSAGING_PLATFORM` ArchUnit 규칙 — application-core가
`dev.caskeleton.messaging..`를 import하면 빌드가 깨진다.
- 반대 방향(신규 `PublishResult` → application outcome) 매핑 규칙을 테스트로 고정.
## 남은 것
- `messaging-platform-bridge` outbound leaf: validated application event → platform envelope,
`PublishResult` → `OutboxPublishOutcome`. registry에 leaf를 추가하는 변경이라 별도 커밋.
- golden contract 테스트: event/message ID, type, schema revision, partition/order/correlation/
causation/tenant/trace, payload digest, wire version이 bytes 단위로 보존되는지.
- 단일 publication authority: 기존 `OutboxPublicationAuthority` fence를 재사용해 writer/relay가
동시에 ACTIVE가 되지 않도록. **dual write/publish는 금지** — 한 business fact가 두 durable store와
두 relay로 나가는 상태가 cutover에서 가장 위험하다.
- 첫 cutover 범위는 **transport만** 교체(저장소는 기존 유지). storage migration은 shadow read →
authority switch → old backlog drain 순서로 별도 release.
+18 -1
View File
@@ -41,9 +41,26 @@ failed로 표시하면 broker가 이미 가지고 있을 수 있는 메시지를
### lease
```text
status IN ('PENDING','AMBIGUOUS') AND (lease_expires_at IS NULL OR lease_expires_at <= now)
status IN ('PENDING','AMBIGUOUS','IN_FLIGHT')
AND (lease_expires_at IS NULL OR lease_expires_at <= now)
AND next_attempt_at <= now
AND attempts < maxAttempts
```
`IN_FLIGHT`가 목록에 있는 것이 핵심이다. relay가 publish 도중 죽으면 row는 `IN_FLIGHT`로 남는데,
이를 제외하면 그 메시지는 **영원히** 발행되지 않는다 — outbox가 막으려던 바로 그 실패다. 대신
lease가 만료됐을 때만 회수하므로, 살아 있는 relay가 들고 있는 row는 회수되지 않는다.
회수는 **같은 `message_id`로** 이루어지고 `lease_token`이 1 증가한다. 새 id를 발급하면 "전달됐을
수도 있는 메시지"가 "확실히 두 번째"가 되기 때문이다 (위의 AMBIGUOUS 논의와 같은 이유).
이 문단의 근거는 실제 PostgreSQL 컨테이너 레인이다:
- `OutboxPostgresIT#anExpiredLeaseBecomesClaimableAgain` — 만료된 lease의 재회수
- `OutboxPostgresIT#anExpiryReclaimKeepsTheMessageIdAndAdvancesTheToken` — 같은 id, 증가한 token
- `OutboxPostgresIT#aLeasedRowIsInvisibleToASecondRelayInstance` — 살아 있는 lease는 회수 불가
- `OutboxPostgresIT#aSupersededRelayCannotOverwriteTheOutcomeOfTheOneThatReplacedIt` — fencing
partial index `ix_messaging_outbox_claimable`이 이 쿼리를 backlog 크기에 비례하게 유지한다.
PUBLISHED row는 retention job이 지울 때까지 쌓이기 때문이다.
+18 -1
View File
@@ -3,11 +3,28 @@
플랫폼이 **무엇을 보장하는지**와 **무엇을 보장하지 않는지**를 브로커별로 고정한다.
여기 없는 조합은 지원되지 않는다.
> **인증 근거.** 이 표의 버전은 이 저장소의 컨테이너 레인이 실제로 실행한 이미지다. 이전 판은
> Kafka 4.2/4.3을 선언했지만 fixture는 `apache/kafka:4.1.0`, lockfile client는 4.1.1이었다 — 표와
> 코드 상수가 서로 일치했을 뿐 어느 쪽도 실행된 적이 없었다. 장애 시나리오 커버리지도 마찬가지로
> `BrokerFailureMatrix.shipped()` 하드코딩이 아니라 레인이 낸 증거(`BrokerCertificationEvidence`)에서
> 나온다. 증거가 없는 조합은 `NOT_COVERED`다 (MSG-014).
> **모듈 이름과 런타임 편입.** `messaging-outbox-jdbc-postgresql` / `messaging-inbox-jdbc-postgresql`은
> 이전에 `-jpa`로 불렸다. 구현은 Spring JDBC이고 SQL은 PostgreSQL 전용(`?::jsonb`,
> `FOR UPDATE SKIP LOCKED`, `ON CONFLICT`, `TIMESTAMPTZ`)이므로, 그 이름은 쓰지 않는 기술을
> 광고하고 vendor 중립 port(`messaging-reliability-api`)의 위치를 가렸다 (MSG-023).
>
> 또한 registry의 messaging leaf는 모두 `runtime_memberships`가 비어 있다. 이는 **build-only /
> incubating** — 어느 composition root에도 편입되지 않았다는 뜻이며, 아래의 등급과는 다른 축이다.
> 등급은 "무엇이 증명되었는가", membership은 "무엇이 실행되는가"를 말한다. 애플리케이션에 배선하려면
> registry를 먼저 바꾸고 `verifyRuntimeModuleMembership`을 통과시켜야 한다. 자세한 규칙은
> `src/messaging/CLAUDE.md`가 소유한다.
## 브로커 등급
| 브로커 | 등급 | 인증 기준 | Stable 기능 | 제한 |
|---|---|---|---|---|
| Kafka | Stable | 4.2+ / 4.3.x | producer idempotence, consumer group, batch, pause/resume, replay, transaction capability | Share Group은 Experimental |
| Kafka | Stable | 4.1.x | producer idempotence, consumer group, batch, pause/resume, replay, transaction capability | Share Group은 Experimental |
| RabbitMQ | Stable | 4.3.x | exchange/routing, publisher confirm, mandatory return, manual ACK, quorum queue, retry queue, DLQ | stream 및 특수 plugin 미지원 |
| Pulsar | Experimental | 4.0 LTS + 4.2 | typed publish/consume, Shared, Key_Shared, schema | transaction 미승격, 기본 비활성 |
| NATS JetStream | Experimental | 2.14.x | stream, durable consumer, explicit ACK, dedupe, replay | native DLQ 없음(플랫폼이 대행), 기본 비활성 |
+85
View File
@@ -0,0 +1,85 @@
# messaging — local authority for the messaging platform family
이 문서는 messaging family의 **local authority**다. leaf 목록·gradle path·허용 의존성은
`src/config/architecture/modules.json`이 SSOT이며 이 문서는 그것을 복제하지 않는다. 여기서 정하는
것은 registry가 표현할 수 없는 것들이다: family별 framework 규칙, public surface 정책, Stable 승격에
필요한 증거, runtime membership이 비어 있다는 사실의 의미.
Root 정책(`CLAUDE.md` / `AGENTS.md`)과 충돌하면 root가 이긴다.
## Family와 framework 규칙
leaf의 정확한 id는 registry에서 읽는다. family는 **디렉터리 이름의 접두사**가 아니라 아래 표의
역할로 정해진다.
| Family | 역할 | 허용 framework |
| --- | --- | --- |
| `*-api` (`core-api`, `schema-api`, `policy`, `reliability-api`, `admin-api`, `transport-spi`) | 값 타입, 포트, 정책 계약 | Java stdlib만. Spring/broker SDK/logging framework 금지 |
| `schema-*` (json/avro/protobuf), `cloudevents` | 인코딩 구현 | 해당 codec 라이브러리만. Spring 금지 |
| `runtime-core`, `security`, `observability`, `admin-runtime` | framework-free 실행 로직 | Micrometer는 `observability`에서만. 나머지는 stdlib |
| `kafka`, `rabbit`, `pulsar-experimental`, `nats-experimental`, `kafka-share-experimental` | broker adapter | 해당 broker client. Spring은 `rabbit`의 `spring-rabbit`처럼 client가 곧 Spring인 경우만 |
| `outbox-jdbc-postgresql`, `inbox-jdbc-postgresql` | 관계형 신뢰성 adapter | Spring JDBC / Spring TX. **JPA 아님** — 이름이 곧 계약이다 |
| `spring-boot-starter`, `spring-cloud-stream-bridge` | 조립 경계 | Spring Boot autoconfigure. 여기서만 |
| `testkit` | 테스트 지원 | Testcontainers, JUnit. production leaf가 의존하지 않는다 |
`outbox`/`inbox` leaf가 `jdbc-postgresql`로 명명된 이유: 구현은 Spring JDBC이고 SQL은
`?::jsonb` · `FOR UPDATE SKIP LOCKED` · `ON CONFLICT` · `TIMESTAMPTZ`로 PostgreSQL 전용이다.
`jpa`라는 이름은 사용하지 않는 기술을 광고하고, vendor 중립 port가 어디 있는지도 숨겼다. vendor
중립 계약은 `messaging-reliability-api`에 있다.
## Public surface 정책
- **vendor 타입을 public signature에 노출하면 그 의존성은 `api`로 선언한다.** 숨긴다고 타입이
숨겨지지 않는다 — consumer가 이 모듈이 문서화한 API에 대해 컴파일하지 못하게 만들 뿐이다.
(`kafka-clients`, `spring-rabbit`, `cloudevents-api`, `micrometer-core`가 이 규칙으로 `api`다.)
- 반대로 **노출하지 않는 것은 `implementation`으로 낮춘다.** starter가 노출하지 않는 하위 모듈을
`implementation`으로 내린 것이 그 예다.
- 새 public 타입은 그 모듈의 계약이다. 삭제·시그니처 변경은 breaking change로 취급한다.
## Runtime membership이 비어 있다는 것의 의미
registry의 모든 messaging leaf는 `runtime_memberships`가 비어 있다. 이는 **build-only /
incubating**을 뜻한다:
- 두 composition root(`app-bootstrap`, `sample-portfolio`) 중 어디에도 런타임으로 들어가 있지 않다.
- 따라서 이 leaf들은 컴파일·테스트·계약 검증의 대상이지, 배포된 애플리케이션이 실행하는 코드가
아니다.
- 애플리케이션에 실제로 배선하려면 registry의 `runtime_memberships`를 먼저 바꾸고
`verifyRuntimeModuleMembership`을 통과시켜야 한다. 코드만 추가하는 것으로는 런타임에 들어가지
않는다.
- 지원 상태 문서(`docs/messaging/support-matrix.md`)의 tier는 이 사실과 별개다. tier는 "무엇이
증명되었는가", membership은 "무엇이 실행되는가"다.
## Stable 승격에 필요한 증거
adapter를 Experimental → Stable로 올릴 때 필요한 것 (`docs/messaging/support-matrix.md`가 현재
상태를 기록한다):
1. 실제 broker container를 쓰는 fault 시나리오 lane이 존재하고, 그 lane이
`BrokerCertificationEvidence`를 산출한다.
2. `NetworkFaultScenario` 전 항목에 대해 evidence가 있거나, 없는 항목이 `knownGaps`로 명시된다.
3. capability 선언이 코드에서 파생된다 — `MessagingCapabilities`에 손으로 `true`를 쓰는 것은
증거가 아니다.
4. publish 실패 분류가 typed signal에서만 나온다. exception class 이름 substring 판정은
Experimental에서도 금지다(MSG-022).
5. unknown failure의 기본값이 `AMBIGUOUS`다. `REJECTED`는 전송이 시작되지 않았다는 typed 증거가
있을 때만.
Stable 문구는 evidence manifest가 존재할 때만 문서에 쓴다.
## Focused test
leaf의 `gradle_path`를 registry에서 읽어 파생한다.
```bash
cd src
./gradlew <gradle-path>:test --console=plain
```
Family 전체 검증:
```bash
./gradlew verifyCleanArchitectureDependencies --console=plain
./gradlew verifyDocumentedLeafCount --console=plain
./gradlew verifyDependencyLocks --console=plain
```
@@ -0,0 +1,93 @@
package dev.caskeleton.messaging.admin;
import java.time.Duration;
import java.time.Instant;
import java.util.Optional;
/**
* Where a destructive operation records that it started, how far it got, and how it ended.
*
* <p>The implementation this replaced was a {@code ConcurrentHashMap} registered by the starter as
* the default. Two consequences followed, and both are worse than having no store at all because
* the map made the platform look protected. A restart forgot every claim, so the same approval
* could be executed again by the same process. And two replicas each had their own map, so both
* could claim the same approval at the same moment and each believe it was the only one.
*
* <p>Implementations must therefore be shared and durable, must enforce uniqueness on {@code
* (approvalTicket, planDigest)}, and must hand out leases with a monotonic fencing token so a
* process that stalled past its lease cannot write over the replica that took over from it.
*/
public interface AdminOperationJournal {
/**
* Claims an operation, or resumes the previous attempt at it.
*
* @param approvalTicket the approval authorising the operation
* @param planDigest the exact plan
* @param operationId this execution's identity
* @param leaseOwner the runtime claiming it
* @param leaseDuration how long the lease holds before another runtime may take over
* @param now the current instant
* @return the lease, carrying the point to resume from
* @throws dev.caskeleton.messaging.api.error.MessageAuthorizationException when the operation
* already completed, or another runtime holds a live lease on it
*/
AdminOperationLease begin(
String approvalTicket,
PlanDigest planDigest,
String operationId,
String leaseOwner,
Duration leaseDuration,
Instant now);
/**
* Records durable progress and extends the lease.
*
* @param lease the lease held
* @param itemsCompleted how many items are now durably done
* @param leaseDuration how much longer to hold the lease
* @param now the current instant
* @throws dev.caskeleton.messaging.api.error.MessageAuthorizationException when the lease has
* been taken over by a newer token
*/
void checkpoint(
AdminOperationLease lease, int itemsCompleted, Duration leaseDuration, Instant now);
/**
* Marks the operation finished. The approval is spent.
*
* @param lease the lease held
* @param itemsCompleted the final count
* @param now the current instant
*/
void complete(AdminOperationLease lease, int itemsCompleted, Instant now);
/**
* Marks the operation failed at its current checkpoint, leaving it resumable.
*
* @param lease the lease held
* @param itemsCompleted how many items are durably done
* @param failureCode a sanitised, bounded reason
* @param now the current instant
*/
void fail(AdminOperationLease lease, int itemsCompleted, String failureCode, Instant now);
/**
* Reads the journal entry for an operation.
*
* @param approvalTicket the approval
* @param planDigest the plan
* @return the record, when the operation has been journalled
*/
Optional<AdminOperationRecord> find(String approvalTicket, PlanDigest planDigest);
/**
* Whether this implementation survives a restart and is shared between replicas.
*
* <p>Declared rather than inferred so the starter can refuse to run a production profile on an
* in-memory journal instead of discovering the gap during an incident.
*
* @return true when the journal is durable and shared
*/
boolean isDurable();
}
@@ -0,0 +1,55 @@
package dev.caskeleton.messaging.admin;
import java.util.Objects;
/**
* The right to execute one journalled operation, and the point to resume it from.
*
* <p>{@code resumeFrom} is the whole reason a lease is handed out rather than a boolean. A retry
* after a crash is not a new execution of the approval — it is the same operation continuing, and
* treating it as new is what republishes the items the first attempt already moved.
*
* @param approvalTicket the approval being executed
* @param planDigest the plan being executed
* @param operationId this execution's identity
* @param leaseOwner the runtime holding the lease
* @param leaseToken the fencing token; a write from an older token must be refused
* @param resumeFrom how many items the previous attempt durably completed
*/
public record AdminOperationLease(
String approvalTicket,
PlanDigest planDigest,
String operationId,
String leaseOwner,
long leaseToken,
int resumeFrom) {
/** Validates the lease. */
public AdminOperationLease {
Objects.requireNonNull(planDigest, "planDigest must not be null");
if (approvalTicket == null || approvalTicket.isBlank()) {
throw new IllegalArgumentException("approvalTicket must not be blank");
}
if (operationId == null || operationId.isBlank()) {
throw new IllegalArgumentException("operationId must not be blank");
}
if (leaseOwner == null || leaseOwner.isBlank()) {
throw new IllegalArgumentException("leaseOwner must not be blank");
}
if (leaseToken < 1) {
throw new IllegalArgumentException("a lease token starts at one");
}
if (resumeFrom < 0) {
throw new IllegalArgumentException("resumeFrom must not be negative");
}
}
/**
* Reports whether this lease continues an earlier attempt.
*
* @return true when items were already completed
*/
public boolean isResumption() {
return resumeFrom > 0;
}
}
@@ -0,0 +1,63 @@
package dev.caskeleton.messaging.admin;
import java.time.Instant;
import java.util.Objects;
import java.util.Optional;
/**
* One row of the destructive-operation journal.
*
* <p>Keyed by {@code (approvalTicket, planDigest)} rather than by the ticket alone, because the
* ticket alone cannot distinguish "this approval already ran" from "this approval is being reused
* for a different plan" — and those need opposite answers.
*
* @param approvalTicket the approval that authorised the operation
* @param planDigest the exact plan that was authorised
* @param operationId the caller-side identity of this execution
* @param state where the operation stands
* @param itemsCompleted how many items are durably done
* @param leaseOwner which runtime currently holds the lease
* @param leaseToken the fencing token, incremented on every takeover
* @param leaseExpiresAt when another runtime may take the lease over
* @param startedAt when the operation first began
* @param finishedAt when it reached a terminal state
* @param failureCode the sanitised reason it failed, when it did
*/
public record AdminOperationRecord(
String approvalTicket,
PlanDigest planDigest,
String operationId,
AdminOperationState state,
int itemsCompleted,
String leaseOwner,
long leaseToken,
Instant leaseExpiresAt,
Instant startedAt,
Optional<Instant> finishedAt,
Optional<String> failureCode) {
/** Validates the record. */
public AdminOperationRecord {
Objects.requireNonNull(planDigest, "planDigest must not be null");
Objects.requireNonNull(state, "state must not be null");
Objects.requireNonNull(leaseExpiresAt, "leaseExpiresAt must not be null");
Objects.requireNonNull(startedAt, "startedAt must not be null");
Objects.requireNonNull(finishedAt, "finishedAt must not be null");
Objects.requireNonNull(failureCode, "failureCode must not be null");
if (approvalTicket == null || approvalTicket.isBlank()) {
throw new IllegalArgumentException("approvalTicket must not be blank");
}
if (operationId == null || operationId.isBlank()) {
throw new IllegalArgumentException("operationId must not be blank");
}
if (leaseOwner == null || leaseOwner.isBlank()) {
throw new IllegalArgumentException("leaseOwner must not be blank");
}
if (itemsCompleted < 0) {
throw new IllegalArgumentException("itemsCompleted must not be negative");
}
if (leaseToken < 1) {
throw new IllegalArgumentException("a lease token starts at one");
}
}
}
@@ -0,0 +1,21 @@
package dev.caskeleton.messaging.admin;
/**
* Where a journalled destructive operation stands.
*
* <p>The store this replaces recorded one fact — "this approval was claimed" — and recorded it
* before any work happened. A run that died halfway had consumed its approval, left no record of
* how far it got, and offered the operator two equally bad choices: request a fresh approval and
* redo work that may already have been done, or leave the operation half-applied.
*/
public enum AdminOperationState {
/** Claimed and in progress. A lease is held; the checkpoint says how far it got. */
STARTED,
/** Every item finished. The approval is spent and must not run again. */
COMPLETED,
/** The run stopped before finishing. It may be resumed from its checkpoint under a new lease. */
FAILED
}
@@ -0,0 +1,98 @@
package dev.caskeleton.messaging.admin;
import dev.caskeleton.messaging.api.destination.DestinationName;
import java.time.Instant;
import java.util.Objects;
import java.util.Optional;
/**
* Everything an issuer signs when granting one destructive operation.
*
* <p>{@link AdminApproval} carried a ticket, an approver, and a window. Nothing in it said which
* operation, on which destination, against which topology, or up to what impact — so the same
* approval authorised every plan presented inside the window, and the audit trail recorded a ticket
* that proved nothing about what was executed.
*
* <p>Every field here is part of the signed canonical form, which is what makes the grant
* non-transferable: change the target, the topology version, the impact ceiling, or the plan, and
* the signature no longer verifies.
*
* @param approval the ticket, approver, and validity window
* @param requestedBy the operator who will execute it, kept separate from the approver
* @param operation the destructive operation being authorised
* @param source the destination read from
* @param target the destination written to, when the operation has one
* @param topologyVersion the topology the impact estimate was computed against
* @param maxImpact the largest number of messages the approver agreed to
* @param planDigest the fingerprint of the exact plan
*/
public record ApprovalGrant(
AdminApproval approval,
String requestedBy,
DestructiveOperation operation,
DestinationName source,
Optional<DestinationName> target,
String topologyVersion,
long maxImpact,
PlanDigest planDigest) {
/** Validates the grant and enforces subject separation. */
public ApprovalGrant {
Objects.requireNonNull(approval, "approval must not be null");
Objects.requireNonNull(operation, "operation must not be null");
Objects.requireNonNull(source, "source must not be null");
Objects.requireNonNull(target, "target must not be null");
Objects.requireNonNull(topologyVersion, "topologyVersion must not be null");
Objects.requireNonNull(planDigest, "planDigest must not be null");
if (requestedBy == null || requestedBy.isBlank()) {
throw new IllegalArgumentException("requestedBy must not be blank");
}
if (approval.approvedBy().equals(requestedBy)) {
// Four-eyes, expressed where it cannot be skipped: an operator approving their own
// destructive operation is the control not existing.
throw new IllegalArgumentException(
"the approver and the operator must be different people; both are " + requestedBy);
}
if (maxImpact < 0) {
throw new IllegalArgumentException("maxImpact must not be negative");
}
}
/**
* Renders the grant deterministically for signing and verification.
*
* <p>Every field is length-prefixed rather than delimited. A delimiter can be smuggled into a
* ticket or an identity to make two different grants render identically; a length prefix cannot.
*
* @return the canonical form
*/
public String canonicalForm() {
StringBuilder canonical = new StringBuilder("v1");
appendField(canonical, approval.ticket());
appendField(canonical, approval.approvedBy());
appendField(canonical, requestedBy);
appendField(canonical, operation.name());
appendField(canonical, source.value());
appendField(canonical, target.map(DestinationName::value).orElse(""));
appendField(canonical, topologyVersion);
appendField(canonical, Long.toString(maxImpact));
appendField(canonical, planDigest.value());
appendField(canonical, approval.approvedAt().toString());
appendField(canonical, approval.validUntil().toString());
return canonical.toString();
}
private static void appendField(StringBuilder canonical, String value) {
canonical.append(value.length()).append(':').append(value);
}
/**
* Reports whether the grant still authorises anything.
*
* @param now the current instant
* @return true while inside the approval window
*/
public boolean isValidAt(Instant now) {
return approval.isValidAt(now);
}
}
@@ -0,0 +1,29 @@
package dev.caskeleton.messaging.admin;
import java.time.Instant;
/**
* The only thing that can turn a signed grant into a {@link VerifiedApproval}.
*
* <p>An implementation must check, at minimum: the issuer's signature over {@link
* ApprovalGrant#canonicalForm()}, that the approval window is open at {@code now}, and that the
* digest the caller is about to execute is the digest that was signed. Everything else the grant
* binds — operation, source, target, topology version, impact ceiling, approver-versus-operator
* separation — is inside the canonical form, so a signature check covers them all at once.
*/
public interface ApprovalVerifier {
/**
* Verifies a signed grant against the plan actually about to run.
*
* @param grant what the issuer signed
* @param signature the issuer's signature over the grant's canonical form
* @param executingDigest the digest of the plan the caller is about to execute
* @param now the current instant
* @return the verified approval
* @throws dev.caskeleton.messaging.api.error.MessageAuthorizationException when anything about
* the grant, the signature, the window, or the digest does not hold
*/
VerifiedApproval verify(
ApprovalGrant grant, String signature, PlanDigest executingDigest, Instant now);
}
@@ -1,5 +1,6 @@
package dev.caskeleton.messaging.admin;
import dev.caskeleton.messaging.api.destination.DestinationName;
import dev.caskeleton.messaging.api.error.MessageAuthorizationException;
import java.time.Instant;
import java.util.Objects;
@@ -17,11 +18,42 @@ import java.util.Objects;
* @param loopAcknowledged whether the approver accepted the previously-redriven candidates
*/
public record ApprovedRedrivePlan(
RedrivePlan plan, AdminApproval approval, boolean loopAcknowledged) {
RedrivePlan plan, VerifiedApproval approval, boolean loopAcknowledged) {
/** Binds the approval to this exact plan at construction. */
public ApprovedRedrivePlan {
Objects.requireNonNull(plan, "plan must not be null");
Objects.requireNonNull(approval, "approval must not be null");
if (!approval.grant().planDigest().equals(plan.digest())) {
throw new MessageAuthorizationException(
"APPROVAL_PLAN_MISMATCH",
"approval %s was issued for a different plan than the one it is attached to"
.formatted(approval.ticket()));
}
if (approval.grant().operation() != DestructiveOperation.REDRIVE) {
throw new MessageAuthorizationException(
"APPROVAL_OPERATION_MISMATCH",
"approval %s authorises %s, not a redrive"
.formatted(approval.ticket(), approval.grant().operation()));
}
if (!approval.grant().source().equals(plan.request().source())
|| !approval.grant().target().equals(java.util.Optional.of(plan.request().target()))) {
throw new MessageAuthorizationException(
"APPROVAL_SOURCE_MISMATCH",
"approval %s authorises %s to %s, not %s to %s"
.formatted(
approval.ticket(),
approval.grant().source().value(),
approval.grant().target().map(DestinationName::value).orElse("(none)"),
plan.request().source().value(),
plan.request().target().value()));
}
if (plan.candidates() > approval.grant().maxImpact()) {
throw new MessageAuthorizationException(
"APPROVAL_IMPACT_EXCEEDED",
"the plan moves %d messages but the approver agreed to at most %d"
.formatted(plan.candidates(), approval.grant().maxImpact()));
}
}
/**
@@ -39,6 +71,13 @@ public record ApprovedRedrivePlan(
throw new MessageAuthorizationException(
"APPROVAL_EXPIRED", "approval %s is not valid at %s".formatted(approval.ticket(), now));
}
if (!approval.grant().topologyVersion().equals(currentTopologyVersion)) {
throw new MessageAuthorizationException(
"TOPOLOGY_CHANGED_SINCE_APPROVAL",
"approval %s was signed against topology %s but the broker is now at %s"
.formatted(
approval.ticket(), approval.grant().topologyVersion(), currentTopologyVersion));
}
if (!plan.topologyVersion().equals(currentTopologyVersion)) {
throw new MessageAuthorizationException(
"TOPOLOGY_CHANGED_SINCE_APPROVAL",
@@ -14,11 +14,41 @@ import java.util.Objects;
* @param plan the plan that was approved
* @param approval the approval authorising it
*/
public record ApprovedReplayPlan(ReplayPlan plan, AdminApproval approval) {
public record ApprovedReplayPlan(ReplayPlan plan, VerifiedApproval approval) {
/** Binds the approval to this exact plan at construction. */
public ApprovedReplayPlan {
Objects.requireNonNull(plan, "plan must not be null");
Objects.requireNonNull(approval, "approval must not be null");
if (!approval.grant().planDigest().equals(plan.digest())) {
// A genuine, verified approval for a different plan. Checking it here means the mismatch
// cannot survive to the execute call under any code path.
throw new MessageAuthorizationException(
"APPROVAL_PLAN_MISMATCH",
"approval %s was issued for a different plan than the one it is attached to"
.formatted(approval.ticket()));
}
if (approval.grant().operation() != DestructiveOperation.REPLAY) {
throw new MessageAuthorizationException(
"APPROVAL_OPERATION_MISMATCH",
"approval %s authorises %s, not a replay"
.formatted(approval.ticket(), approval.grant().operation()));
}
if (!approval.grant().source().equals(plan.request().destination())) {
throw new MessageAuthorizationException(
"APPROVAL_SOURCE_MISMATCH",
"approval %s authorises %s, not %s"
.formatted(
approval.ticket(),
approval.grant().source().value(),
plan.request().destination().value()));
}
if (plan.estimatedMessages() > approval.grant().maxImpact()) {
throw new MessageAuthorizationException(
"APPROVAL_IMPACT_EXCEEDED",
"the plan moves about %d messages but the approver agreed to at most %d"
.formatted(plan.estimatedMessages(), approval.grant().maxImpact()));
}
}
/**
@@ -36,6 +66,13 @@ public record ApprovedReplayPlan(ReplayPlan plan, AdminApproval approval) {
throw new MessageAuthorizationException(
"APPROVAL_EXPIRED", "approval %s is not valid at %s".formatted(approval.ticket(), now));
}
if (!approval.grant().topologyVersion().equals(currentTopologyVersion)) {
throw new MessageAuthorizationException(
"TOPOLOGY_CHANGED_SINCE_APPROVAL",
"approval %s was signed against topology %s but the broker is now at %s"
.formatted(
approval.ticket(), approval.grant().topologyVersion(), currentTopologyVersion));
}
if (!plan.topologyVersion().equals(currentTopologyVersion)) {
// Every number in the plan was computed against the old topology, so the approver agreed to
// an impact estimate that no longer describes what would happen.
@@ -9,7 +9,7 @@ import java.util.Optional;
/**
* The single gate every destructive messaging operation passes through.
*
* <p>Three conditions, all required. The caller must hold the admin credential — an application
* <p>Four conditions, all required. The caller must hold the admin credential — an application
* runtime does not, by construction. The approval must be present and inside its validity window.
* And a dry run is always permitted, because the way to make operators plan before they act is to
* make planning free.
@@ -35,7 +35,7 @@ public final class DestructiveOperationGuard {
*
* @param operation the operation
* @param destination the destination affected
* @param approval the approval, when one was supplied
* @param approval the verified approval, when one was supplied
* @param dryRun whether this is a plan-only run
* @param now the current instant
* @throws MessageAuthorizationException when the operation is not authorised
@@ -43,7 +43,7 @@ public final class DestructiveOperationGuard {
public void authorize(
DestructiveOperation operation,
DestinationName destination,
Optional<AdminApproval> approval,
Optional<VerifiedApproval> approval,
boolean dryRun,
Instant now) {
Objects.requireNonNull(operation, "operation must not be null");
@@ -59,7 +59,7 @@ public final class DestructiveOperationGuard {
"ADMIN_CREDENTIAL_REQUIRED",
operation + " on " + destination.value() + " requires the admin credential");
}
AdminApproval granted =
VerifiedApproval granted =
approval.orElseThrow(
() ->
new MessageAuthorizationException(
@@ -69,5 +69,27 @@ public final class DestructiveOperationGuard {
throw new MessageAuthorizationException(
"APPROVAL_EXPIRED", "approval " + granted.ticket() + " is outside its validity window");
}
if (granted.grant().operation() != operation) {
// The approval verified — for something else. A guard that only checks presence and window
// lets a verified redrive approval authorise a destination deletion.
throw new MessageAuthorizationException(
"APPROVAL_OPERATION_MISMATCH",
"approval "
+ granted.ticket()
+ " authorises "
+ granted.grant().operation()
+ ", not "
+ operation);
}
if (!granted.grant().source().equals(destination)) {
throw new MessageAuthorizationException(
"APPROVAL_SOURCE_MISMATCH",
"approval "
+ granted.ticket()
+ " authorises "
+ granted.grant().source().value()
+ ", not "
+ destination.value());
}
}
}
@@ -0,0 +1,111 @@
package dev.caskeleton.messaging.admin;
import dev.caskeleton.messaging.api.error.MessageAuthorizationException;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.time.Instant;
import java.util.HexFormat;
import java.util.Objects;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
/**
* Verifies grants with a shared HMAC-SHA256 issuing key.
*
* <p>A shared secret rather than a public-key signature is a deliberate scope choice: the issuer is
* the change-management system operated by the same organisation as the runtime, so key
* distribution is already solved and the property that matters — that an operator cannot mint an
* approval for themselves — holds as long as the key is not on the operator's machine. A public-key
* verifier can be substituted by implementing {@link ApprovalVerifier}; nothing else in the
* platform depends on how the signature is produced.
*
* <p>Comparison is constant-time. A verifier that compares with {@code equals} leaks the signature
* one byte at a time to a caller who can retry, which for an offline attacker with unlimited
* attempts is the whole secret.
*/
public final class HmacApprovalVerifier implements ApprovalVerifier {
private static final String ALGORITHM = "HmacSHA256";
private final byte[] issuingKey;
/**
* Creates a verifier over the issuing key.
*
* @param issuingKey the shared secret the approval issuer signs with
*/
public HmacApprovalVerifier(byte[] issuingKey) {
Objects.requireNonNull(issuingKey, "issuingKey must not be null");
if (issuingKey.length < 32) {
throw new IllegalArgumentException(
"the approval issuing key must be at least 256 bits; a shorter key makes the signature"
+ " brute-forceable offline");
}
this.issuingKey = issuingKey.clone();
}
/**
* Produces the signature an issuer would attach to a grant.
*
* <p>Present because the issuer and the verifier must agree on the canonical form exactly, and a
* second implementation of that agreement is a second thing to drift. Holding this key is what
* makes a caller an issuer — it is not, and must not become, available to the runtime that
* executes operations.
*
* @param grant the grant to sign
* @return the hexadecimal signature
*/
public String sign(ApprovalGrant grant) {
Objects.requireNonNull(grant, "grant must not be null");
return HexFormat.of().formatHex(mac(grant.canonicalForm()));
}
@Override
public VerifiedApproval verify(
ApprovalGrant grant, String signature, PlanDigest executingDigest, Instant now) {
Objects.requireNonNull(grant, "grant must not be null");
Objects.requireNonNull(signature, "signature must not be null");
Objects.requireNonNull(executingDigest, "executingDigest must not be null");
Objects.requireNonNull(now, "now must not be null");
if (!grant.planDigest().equals(executingDigest)) {
// The signature would verify: this is a genuine approval, for a different plan. Replaying one
// approval across plans is the attack the digest binding exists to stop.
throw new MessageAuthorizationException(
"APPROVAL_PLAN_MISMATCH",
"approval %s was issued for plan %s but the plan about to execute is %s"
.formatted(
grant.approval().ticket(), grant.planDigest().value(), executingDigest.value()));
}
if (!grant.isValidAt(now)) {
throw new MessageAuthorizationException(
"APPROVAL_EXPIRED",
"approval %s is not valid at %s".formatted(grant.approval().ticket(), now));
}
byte[] presented;
try {
presented = HexFormat.of().parseHex(signature);
} catch (IllegalArgumentException malformed) {
throw new MessageAuthorizationException(
"APPROVAL_SIGNATURE_INVALID", "the approval signature is not hexadecimal");
}
if (!MessageDigest.isEqual(mac(grant.canonicalForm()), presented)) {
throw new MessageAuthorizationException(
"APPROVAL_SIGNATURE_INVALID",
("approval %s does not verify against the issuing key; the grant was altered after it"
+ " was signed, or it was never issued")
.formatted(grant.approval().ticket()));
}
return VerifiedApproval.of(grant);
}
private byte[] mac(String canonicalForm) {
try {
Mac mac = Mac.getInstance(ALGORITHM);
mac.init(new SecretKeySpec(issuingKey, ALGORITHM));
return mac.doFinal(canonicalForm.getBytes(StandardCharsets.UTF_8));
} catch (java.security.GeneralSecurityException failure) {
throw new IllegalStateException("HMAC-SHA256 is required of every Java runtime", failure);
}
}
}
@@ -0,0 +1,47 @@
package dev.caskeleton.messaging.admin;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.HexFormat;
import java.util.Objects;
/**
* The fingerprint of exactly what an approver agreed to.
*
* <p>An approval used to name a ticket and a window and nothing else, so it authorised any plan
* presented within that window. The operator who got a redrive of one dead-letter destination
* approved could execute a redrive of a different one with the same ticket, and every audit record
* would look correct. Binding the signature to this digest is what makes an approval about a plan
* rather than about a period of time.
*
* @param value the lowercase hexadecimal SHA-256 of the plan's canonical form
*/
public record PlanDigest(String value) {
/** Validates the digest shape. */
public PlanDigest {
Objects.requireNonNull(value, "value must not be null");
if (!value.matches("[0-9a-f]{64}")) {
throw new IllegalArgumentException("a plan digest is 64 lowercase hexadecimal characters");
}
}
/**
* Digests a plan's canonical form.
*
* @param canonicalForm the plan rendered deterministically, field by field
* @return the digest
*/
public static PlanDigest ofCanonical(String canonicalForm) {
Objects.requireNonNull(canonicalForm, "canonicalForm must not be null");
try {
byte[] digest =
MessageDigest.getInstance("SHA-256")
.digest(canonicalForm.getBytes(StandardCharsets.UTF_8));
return new PlanDigest(HexFormat.of().formatHex(digest));
} catch (NoSuchAlgorithmException impossible) {
throw new IllegalStateException("SHA-256 is required of every Java runtime", impossible);
}
}
}
@@ -39,6 +39,28 @@ public record RedrivePlan(
}
}
/**
* The fingerprint an approver's signature is bound to.
*
* <p>Source, target, batch size, and both candidate counts are in it, so an approval for a
* redrive of one dead-letter destination cannot be presented for a redrive of another.
*
* @return the digest of this exact plan
*/
public PlanDigest digest() {
return PlanDigest.ofCanonical(
String.join(
"|",
"REDRIVE",
request.redriveId().toString(),
request.source().value(),
request.target().value(),
Integer.toString(request.batchSize()),
Integer.toString(candidates),
Integer.toString(alreadyRedrivenCandidates),
topologyVersion));
}
/**
* Reports whether this redrive would replay messages that already failed a redrive.
*
@@ -39,6 +39,30 @@ public record ReplayPlan(
}
}
/**
* The fingerprint an approver's signature is bound to.
*
* <p>Every field that changes what would happen is in it. An approval signed over this digest
* cannot be presented for a plan that reads a different destination, targets a different consumer
* group, starts from a different position, or estimates a different impact.
*
* @return the digest of this exact plan
*/
public PlanDigest digest() {
return PlanDigest.ofCanonical(
String.join(
"|",
"REPLAY",
request.replayId().toString(),
request.destination().value(),
Boolean.toString(request.isolatedConsumerGroup()),
request.from().toString(),
request.to().map(Object::toString).orElse(""),
Long.toString(estimatedMessages),
topologyVersion,
Boolean.toString(targetsLiveConsumerGroup)));
}
/**
* Returns a one-line operator-facing summary of the impact.
*
@@ -0,0 +1,88 @@
package dev.caskeleton.messaging.admin;
import java.time.Instant;
import java.util.Objects;
/**
* A grant that an {@link ApprovalVerifier} has checked, and that nobody else can manufacture.
*
* <p>The approved-plan types used to hold a plain {@code AdminApproval} record with a public
* constructor, so "this plan was approved" was a claim the caller made about itself. Any code that
* could reach the execute method could write {@code new AdminApproval("TICKET-1", "someone", now,
* later)} and the platform believed it.
*
* <p>This class has no public constructor and no public factory. The only way to obtain one is to
* present a grant and a signature to a verifier, which is exactly the property the type is for:
* holding a {@code VerifiedApproval} is itself the evidence that verification happened.
*/
public final class VerifiedApproval {
private final ApprovalGrant grant;
private VerifiedApproval(ApprovalGrant grant) {
this.grant = Objects.requireNonNull(grant, "grant must not be null");
}
/**
* Mints a verified approval. Package-private on purpose: only a verifier in this package may call
* it, so the type cannot be forged from application or adapter code.
*/
static VerifiedApproval of(ApprovalGrant grant) {
return new VerifiedApproval(grant);
}
/**
* The grant that was verified.
*
* @return the grant
*/
public ApprovalGrant grant() {
return grant;
}
/**
* The approval ticket, for audit records and journal keys.
*
* @return the ticket
*/
public String ticket() {
return grant.approval().ticket();
}
/**
* Who approved the operation.
*
* @return the approver identity
*/
public String approvedBy() {
return grant.approval().approvedBy();
}
/**
* Reports whether the approval window is still open.
*
* @param now the current instant
* @return true while inside the window
*/
public boolean isValidAt(Instant now) {
return grant.isValidAt(now);
}
@Override
public boolean equals(Object other) {
return other instanceof VerifiedApproval verified && grant.equals(verified.grant);
}
@Override
public int hashCode() {
return grant.hashCode();
}
@Override
public String toString() {
// Deliberately not the whole grant: this string reaches logs, and the plan digest plus the
// ticket identify the operation without restating who approved what to an audience that has
// not been authorised to read it.
return "VerifiedApproval[" + ticket() + " " + grant.planDigest().value().substring(0, 12) + "]";
}
}
@@ -17,8 +17,35 @@ class DestructiveOperationGuardTest {
private static final Instant NOW = Instant.parse("2026-08-10T09:15:00Z");
private static final DestinationName ORDERS = new DestinationName("order-events");
private static final AdminApproval VALID =
new AdminApproval("CHG-1001", "operator", NOW.minusSeconds(60), NOW.plusSeconds(3600));
private static final HmacApprovalVerifier ISSUER =
new HmacApprovalVerifier(
"an-issuing-key-of-at-least-32-bytes!".getBytes(java.nio.charset.StandardCharsets.UTF_8));
private static final VerifiedApproval VALID =
verified(DestructiveOperation.REPLAY, ORDERS, NOW.minusSeconds(60), NOW.plusSeconds(3600));
/**
* A grant the issuer signed, verified the way the runtime verifies it.
*
* <p>The guard used to take a plain {@code AdminApproval} record, which any caller could
* construct. Every case below now has to go through a real signature to get an approval object at
* all, which is the property the type change was made for.
*/
private static VerifiedApproval verified(
DestructiveOperation operation, DestinationName source, Instant from, Instant until) {
PlanDigest digest = PlanDigest.ofCanonical(operation + "|" + source.value());
ApprovalGrant grant =
new ApprovalGrant(
new AdminApproval("CHG-1001", "approver", from, until),
"operator",
operation,
source,
Optional.empty(),
"v1",
Long.MAX_VALUE,
digest);
return ISSUER.verify(grant, ISSUER.sign(grant), digest, from);
}
@Test
void anApplicationRuntimeCannotRedrive() {
@@ -45,8 +72,12 @@ class DestructiveOperationGuardTest {
@Test
void anExpiredApprovalDoesNotAuthorise() {
DestructiveOperationGuard guard = new DestructiveOperationGuard(true);
AdminApproval expired =
new AdminApproval("CHG-1000", "operator", NOW.minusSeconds(7200), NOW.minusSeconds(60));
VerifiedApproval expired =
verified(
DestructiveOperation.OFFSET_RESET,
ORDERS,
NOW.minusSeconds(7200),
NOW.minusSeconds(60));
assertThatThrownBy(
() ->
@@ -32,9 +32,9 @@ commons-io:commons-io:2.21.0=spotbugs
info.picocli:picocli:4.7.7=checkstyle
io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnotationProcessor
io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor
io.micrometer:micrometer-commons:1.16.0=runtimeClasspath,testRuntimeClasspath
io.micrometer:micrometer-core:1.16.0=runtimeClasspath,testRuntimeClasspath
io.micrometer:micrometer-observation:1.16.0=runtimeClasspath,testRuntimeClasspath
io.micrometer:micrometer-commons:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-core:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-observation:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
javax.inject:javax.inject:1=annotationProcessor,testAnnotationProcessor
jaxen:jaxen:2.0.0=spotbugs
net.bytebuddy:byte-buddy:1.17.8=testCompileClasspath,testRuntimeClasspath
@@ -62,7 +62,7 @@ org.codehaus.plexus:plexus-utils:3.3.0=checkstyle
org.dom4j:dom4j:2.2.0=spotbugs
org.hdrhistogram:HdrHistogram:2.2.2=runtimeClasspath,testRuntimeClasspath
org.javassist:javassist:3.28.0-GA=checkstyle
org.jspecify:jspecify:1.0.0=annotationProcessor,checkstyle,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
org.jspecify:jspecify:1.0.0=annotationProcessor,checkstyle,compileClasspath,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
org.junit.jupiter:junit-jupiter-api:6.0.1=testCompileClasspath,testRuntimeClasspath
org.junit.jupiter:junit-jupiter-engine:6.0.1=testRuntimeClasspath
org.junit.jupiter:junit-jupiter-params:6.0.1=testCompileClasspath,testRuntimeClasspath
@@ -85,4 +85,4 @@ org.reflections:reflections:0.10.2=checkstyle
org.slf4j:slf4j-api:2.0.17=spotbugs,spotbugsSlf4j
org.slf4j:slf4j-simple:2.0.17=checkstyle,spotbugsSlf4j
org.xmlresolver:xmlresolver:5.3.3=checkstyle,spotbugs
empty=compileClasspath
empty=
@@ -1,78 +0,0 @@
package dev.caskeleton.messaging.admin.runtime;
import java.time.Instant;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.concurrent.ConcurrentHashMap;
/**
* Remembers which approvals have already been executed.
*
* <p>An approval authorises one execution, not a standing permission. Without this, re-running the
* same approved redrive twice is a single command-history arrow-up away — and the second run
* republishes messages the first one already moved, which on a destination without an inbox is
* indistinguishable from a duplicate storm.
*
* <p>Claiming is atomic and returns the previous claim rather than a boolean, so a duplicate
* attempt can tell the operator <em>when</em> it ran and by which operation id instead of just
* refusing.
*/
public final class AdminOperationIdempotencyStore {
private final Map<String, Claim> claims = new ConcurrentHashMap<>();
/**
* One recorded execution of an approval.
*
* @param approvalTicket the approval that was executed
* @param operationId the operation identity that claimed it
* @param executedAt when it ran
*/
public record Claim(String approvalTicket, String operationId, Instant executedAt) {
public Claim {
Objects.requireNonNull(executedAt, "executedAt must not be null");
if (approvalTicket == null || approvalTicket.isBlank()) {
throw new IllegalArgumentException("approvalTicket must not be blank");
}
if (operationId == null || operationId.isBlank()) {
throw new IllegalArgumentException("operationId must not be blank");
}
}
}
/**
* Claims an approval for execution.
*
* @param approvalTicket the approval to claim
* @param operationId the operation attempting it
* @param now the current instant
* @return empty when the claim succeeded; the existing claim when it was already executed
*/
public Optional<Claim> claim(String approvalTicket, String operationId, Instant now) {
Objects.requireNonNull(now, "now must not be null");
Claim candidate = new Claim(approvalTicket, operationId, now);
Claim existing = claims.putIfAbsent(approvalTicket, candidate);
return Optional.ofNullable(existing);
}
/**
* Returns the recorded execution of an approval.
*
* @param approvalTicket the approval
* @return the claim, when the approval has been executed
*/
public Optional<Claim> find(String approvalTicket) {
return Optional.ofNullable(claims.get(approvalTicket));
}
/**
* Returns how many approvals have been executed.
*
* @return the claim count
*/
public int size() {
return claims.size();
}
}
@@ -1,5 +1,7 @@
package dev.caskeleton.messaging.admin.runtime;
import dev.caskeleton.messaging.admin.AdminOperationJournal;
import dev.caskeleton.messaging.admin.AdminOperationLease;
import dev.caskeleton.messaging.admin.ApprovedRedrivePlan;
import dev.caskeleton.messaging.admin.ApprovedReplayPlan;
import dev.caskeleton.messaging.admin.RedrivePlan;
@@ -10,7 +12,6 @@ import dev.caskeleton.messaging.admin.ReplayRequest;
import dev.caskeleton.messaging.admin.ReplayResult;
import dev.caskeleton.messaging.admin.TopologyManifest;
import dev.caskeleton.messaging.admin.TopologyValidationReport;
import dev.caskeleton.messaging.api.error.MessageAuthorizationException;
import java.time.Duration;
import java.time.Instant;
import java.util.List;
@@ -30,15 +31,29 @@ import java.util.function.Supplier;
* <li>Only then does anything move.
* </ol>
*
* <p>The idempotency claim comes <em>before</em> the work rather than after it. Claiming afterwards
* leaves a window where a second execution starts while the first is still running, which is
* precisely the double-redrive this store exists to prevent.
* <p>The journal entry is written <em>before</em> the work rather than after it. Writing it
* afterwards leaves a window where a second execution starts while the first is still running,
* which is precisely the double-redrive the journal exists to prevent.
*
* <p>Writing it first used to have a cost the previous store never paid: an operation that died
* halfway had consumed its approval and left no record of how far it got. The journal keeps a
* checkpoint and hands back a lease that says where to resume, so a retry continues the same
* operation instead of either redoing it or requiring a new approval.
*/
public final class DefaultMessagingAdminService implements MessagingAdminService {
/**
* How long a runtime holds an operation before another may take it over.
*
* <p>Long enough that a slow batch does not lose its lease mid-flight, short enough that a dead
* replica does not park an approval for an hour.
*/
private static final Duration LEASE_DURATION = Duration.ofMinutes(5);
private final CompositeTopologyValidator topologyValidator;
private final BrokerTopologyInspector inspector;
private final AdminOperationIdempotencyStore idempotency;
private final AdminOperationJournal journal;
private final String leaseOwner;
private final ReplayService replayService;
private final RedriveService redriveService;
private final Supplier<List<TopologyManifest>> manifests;
@@ -51,7 +66,8 @@ public final class DefaultMessagingAdminService implements MessagingAdminService
*
* @param topologyValidator compares declared topology against the broker
* @param inspector reads the broker's topology version
* @param idempotency records which approvals have been executed
* @param journal records which approvals have been executed, and how far each got
* @param leaseOwner identifies this runtime in the journal's leases
* @param replayService performs replays
* @param redriveService performs redrives
* @param manifests supplies the declared topologies
@@ -62,7 +78,8 @@ public final class DefaultMessagingAdminService implements MessagingAdminService
public DefaultMessagingAdminService(
CompositeTopologyValidator topologyValidator,
BrokerTopologyInspector inspector,
AdminOperationIdempotencyStore idempotency,
AdminOperationJournal journal,
String leaseOwner,
ReplayService replayService,
RedriveService redriveService,
Supplier<List<TopologyManifest>> manifests,
@@ -72,7 +89,11 @@ public final class DefaultMessagingAdminService implements MessagingAdminService
this.topologyValidator =
Objects.requireNonNull(topologyValidator, "topologyValidator required");
this.inspector = Objects.requireNonNull(inspector, "inspector must not be null");
this.idempotency = Objects.requireNonNull(idempotency, "idempotency must not be null");
this.journal = Objects.requireNonNull(journal, "journal must not be null");
if (leaseOwner == null || leaseOwner.isBlank()) {
throw new IllegalArgumentException("leaseOwner must not be blank");
}
this.leaseOwner = leaseOwner;
this.replayService = Objects.requireNonNull(replayService, "replayService must not be null");
this.redriveService = Objects.requireNonNull(redriveService, "redriveService must not be null");
this.manifests = Objects.requireNonNull(manifests, "manifests must not be null");
@@ -104,12 +125,28 @@ public final class DefaultMessagingAdminService implements MessagingAdminService
ReplayRequest request = plan.plan().request();
plan.requireExecutable(now, inspector.topologyVersion());
claimOrRefuse(plan.approval().ticket(), request.replayId().toString(), now);
AdminOperationLease lease =
journal.begin(
plan.approval().ticket(),
plan.plan().digest(),
request.replayId().toString(),
leaseOwner,
LEASE_DURATION,
now);
Instant startedAt = clock.get();
ReplayReport report =
replayService.replay(
request, Optional.of(plan.approval()), plan.approval().approvedBy(), now);
ReplayReport report;
try {
report =
replayService.replay(
request, Optional.of(plan.approval()), plan.approval().approvedBy(), now);
} catch (RuntimeException failure) {
// The operation stays resumable rather than silently consuming the approval: the journal
// entry moves to FAILED at its checkpoint, and a retry takes it over from there.
journal.fail(lease, lease.resumeFrom(), failureCodeOf(failure), clock.get());
throw failure;
}
journal.complete(lease, Math.toIntExact(report.replayed()), clock.get());
return new ReplayResult(
request.replayId(),
@@ -139,12 +176,31 @@ public final class DefaultMessagingAdminService implements MessagingAdminService
RedriveRequest request = plan.plan().request();
plan.requireExecutable(now, inspector.topologyVersion());
claimOrRefuse(plan.approval().ticket(), request.redriveId().toString(), now);
AdminOperationLease lease =
journal.begin(
plan.approval().ticket(),
plan.plan().digest(),
request.redriveId().toString(),
leaseOwner,
LEASE_DURATION,
now);
Instant startedAt = clock.get();
RedriveReport report =
redriveService.redrive(
request, Optional.of(plan.approval()), plan.approval().approvedBy(), now);
RedriveReport report;
try {
report =
redriveService.redrive(
request,
Optional.of(plan.approval()),
plan.approval().approvedBy(),
now,
lease.resumeFrom(),
completed -> journal.checkpoint(lease, completed, LEASE_DURATION, clock.get()));
} catch (RuntimeException failure) {
journal.fail(lease, lease.resumeFrom(), failureCodeOf(failure), clock.get());
throw failure;
}
journal.complete(lease, report.moved() + report.failed(), clock.get());
return new RedriveResult(
request.redriveId(),
@@ -155,17 +211,19 @@ public final class DefaultMessagingAdminService implements MessagingAdminService
report.dryRun());
}
private void claimOrRefuse(String approvalTicket, String operationId, Instant now) {
idempotency
.claim(approvalTicket, operationId, now)
.ifPresent(
existing -> {
throw new MessageAuthorizationException(
"APPROVAL_ALREADY_EXECUTED",
"approval %s was already executed at %s by operation %s; an approval authorises "
.formatted(approvalTicket, existing.executedAt(), existing.operationId())
+ "one execution, not a standing permission");
});
/**
* Reduces a failure to a bounded, sanitised code for the journal.
*
* <p>The journal is read by operators during incidents and its contents outlive the process. A
* raw exception message can carry a destination, a payload fragment, or a credential from a
* driver's own error text, so only the platform's own code or the exception's simple name goes
* in.
*/
private static String failureCodeOf(RuntimeException failure) {
if (failure instanceof dev.caskeleton.messaging.api.error.MessagingException messaging) {
return messaging.failure().code();
}
return failure.getClass().getSimpleName();
}
/** Estimates how many messages a replay would re-deliver. */
@@ -0,0 +1,237 @@
package dev.caskeleton.messaging.admin.runtime;
import dev.caskeleton.messaging.admin.AdminOperationJournal;
import dev.caskeleton.messaging.admin.AdminOperationLease;
import dev.caskeleton.messaging.admin.AdminOperationRecord;
import dev.caskeleton.messaging.admin.AdminOperationState;
import dev.caskeleton.messaging.admin.PlanDigest;
import dev.caskeleton.messaging.api.error.MessageAuthorizationException;
import java.time.Duration;
import java.time.Instant;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.concurrent.ConcurrentHashMap;
/**
* A single-process journal, for tests and for local development.
*
* <p>It reports {@link #isDurable()} as false, and the starter refuses to run a production profile
* on a journal that says so. That declaration is the point of this class existing at all: the
* previous in-memory store was registered as the production default and nothing distinguished it
* from a shared one, so the gap was invisible until two replicas executed the same approval.
*
* <p>The semantics are otherwise the real ones — uniqueness on {@code (ticket, digest)}, lease
* takeover with a monotonic token, resume from checkpoint — so a test that passes here is testing
* the protocol rather than a simplification of it.
*/
public final class InMemoryAdminOperationJournal implements AdminOperationJournal {
private final Map<String, AdminOperationRecord> records = new ConcurrentHashMap<>();
@Override
public AdminOperationLease begin(
String approvalTicket,
PlanDigest planDigest,
String operationId,
String leaseOwner,
Duration leaseDuration,
Instant now) {
Objects.requireNonNull(leaseDuration, "leaseDuration must not be null");
Objects.requireNonNull(now, "now must not be null");
AdminOperationRecord claimed =
records.compute(
key(approvalTicket, planDigest),
(ignored, existing) ->
claim(
existing,
approvalTicket,
planDigest,
operationId,
leaseOwner,
leaseDuration,
now));
return new AdminOperationLease(
approvalTicket,
planDigest,
claimed.operationId(),
claimed.leaseOwner(),
claimed.leaseToken(),
claimed.itemsCompleted());
}
private static AdminOperationRecord claim(
AdminOperationRecord existing,
String approvalTicket,
PlanDigest planDigest,
String operationId,
String leaseOwner,
Duration leaseDuration,
Instant now) {
if (existing == null) {
return new AdminOperationRecord(
approvalTicket,
planDigest,
operationId,
AdminOperationState.STARTED,
0,
leaseOwner,
1L,
now.plus(leaseDuration),
now,
Optional.empty(),
Optional.empty());
}
if (existing.state() == AdminOperationState.COMPLETED) {
throw new MessageAuthorizationException(
"APPROVAL_ALREADY_EXECUTED",
("approval %s was already executed to completion at %s; an approval authorises one"
+ " execution, not a standing permission")
.formatted(approvalTicket, existing.finishedAt().orElse(existing.startedAt())));
}
if (existing.state() == AdminOperationState.STARTED
&& now.isBefore(existing.leaseExpiresAt())) {
throw new MessageAuthorizationException(
"ADMIN_OPERATION_IN_FLIGHT",
("approval %s is being executed by %s until at least %s; two runtimes executing one"
+ " approval is a duplicate storm, not a faster redrive")
.formatted(approvalTicket, existing.leaseOwner(), existing.leaseExpiresAt()));
}
// Either the previous attempt failed, or its lease expired without a completion. Both resume
// from the checkpoint under a new token, which fences the previous holder out.
return new AdminOperationRecord(
approvalTicket,
planDigest,
operationId,
AdminOperationState.STARTED,
existing.itemsCompleted(),
leaseOwner,
existing.leaseToken() + 1,
now.plus(leaseDuration),
existing.startedAt(),
Optional.empty(),
Optional.empty());
}
@Override
public void checkpoint(
AdminOperationLease lease, int itemsCompleted, Duration leaseDuration, Instant now) {
update(
lease,
current ->
new AdminOperationRecord(
current.approvalTicket(),
current.planDigest(),
current.operationId(),
AdminOperationState.STARTED,
Math.max(current.itemsCompleted(), itemsCompleted),
current.leaseOwner(),
current.leaseToken(),
now.plus(leaseDuration),
current.startedAt(),
Optional.empty(),
Optional.empty()));
}
@Override
public void complete(AdminOperationLease lease, int itemsCompleted, Instant now) {
update(
lease,
current ->
new AdminOperationRecord(
current.approvalTicket(),
current.planDigest(),
current.operationId(),
AdminOperationState.COMPLETED,
Math.max(current.itemsCompleted(), itemsCompleted),
current.leaseOwner(),
current.leaseToken(),
current.leaseExpiresAt(),
current.startedAt(),
Optional.of(now),
Optional.empty()));
}
@Override
public void fail(AdminOperationLease lease, int itemsCompleted, String failureCode, Instant now) {
Objects.requireNonNull(failureCode, "failureCode must not be null");
update(
lease,
current ->
new AdminOperationRecord(
current.approvalTicket(),
current.planDigest(),
current.operationId(),
AdminOperationState.FAILED,
Math.max(current.itemsCompleted(), itemsCompleted),
current.leaseOwner(),
current.leaseToken(),
current.leaseExpiresAt(),
current.startedAt(),
Optional.of(now),
Optional.of(failureCode)));
}
@Override
public Optional<AdminOperationRecord> find(String approvalTicket, PlanDigest planDigest) {
return Optional.ofNullable(records.get(key(approvalTicket, planDigest)));
}
@Override
public boolean isDurable() {
return false;
}
/**
* How many operations have been journalled.
*
* @return the record count
*/
public int size() {
return records.size();
}
private void update(
AdminOperationLease lease, java.util.function.UnaryOperator<AdminOperationRecord> mutation) {
Objects.requireNonNull(lease, "lease must not be null");
records.compute(
key(lease.approvalTicket(), lease.planDigest()),
(ignored, current) -> {
if (current == null) {
throw new MessageAuthorizationException(
"ADMIN_OPERATION_NOT_JOURNALLED",
"no journal entry exists for approval " + lease.approvalTicket());
}
if (current.leaseToken() != lease.leaseToken()) {
// The fence. A stalled runtime that wakes up and writes here would otherwise overwrite
// the progress of whichever replica took the operation over.
throw new MessageAuthorizationException(
"ADMIN_OPERATION_LEASE_LOST",
("the lease on approval %s moved from token %d to token %d; this runtime no longer"
+ " owns the operation")
.formatted(lease.approvalTicket(), lease.leaseToken(), current.leaseToken()));
}
return mutation.apply(current);
});
}
private static String key(String approvalTicket, PlanDigest planDigest) {
Objects.requireNonNull(planDigest, "planDigest must not be null");
// Length-prefixed for the same reason the grant's canonical form is: a ticket containing the
// separator must not be able to collide with a different ticket and digest pair.
return String.join(
"", Integer.toString(approvalTicket.length()), ":", approvalTicket, planDigest.value());
}
/**
* Every journalled operation, newest first by start time.
*
* @return the records
*/
public List<AdminOperationRecord> records() {
return records.values().stream()
.sorted(java.util.Comparator.comparing(AdminOperationRecord::startedAt).reversed())
.toList();
}
}
@@ -1,9 +1,9 @@
package dev.caskeleton.messaging.admin.runtime;
import dev.caskeleton.messaging.admin.AdminApproval;
import dev.caskeleton.messaging.admin.DestructiveOperation;
import dev.caskeleton.messaging.admin.DestructiveOperationGuard;
import dev.caskeleton.messaging.admin.RedriveRequest;
import dev.caskeleton.messaging.admin.VerifiedApproval;
import dev.caskeleton.messaging.api.MessageId;
import dev.caskeleton.messaging.api.publish.PublishCompletion;
import dev.caskeleton.messaging.api.publish.PublishResult;
@@ -54,14 +54,46 @@ public final class RedriveService {
* Runs one redrive pass.
*
* @param request what to move
* @param approval the approval, when one was supplied
* @param approval the verified approval, when one was supplied
* @param subject the operator identity
* @param now the current instant
* @return what the pass did
*/
public RedriveReport redrive(
RedriveRequest request, Optional<AdminApproval> approval, String subject, Instant now) {
RedriveRequest request, Optional<VerifiedApproval> approval, String subject, Instant now) {
return redrive(request, approval, subject, now, 0, completed -> {});
}
/**
* Runs one redrive pass, resuming a previous attempt and reporting progress as it goes.
*
* <p>Three things were wrong with running this as a plain loop. A synchronous failure from the
* publisher — a broker that refuses the connection rather than the message — propagated out of
* the loop, so the remaining candidates were never attempted and the audit record was never
* written: the operation left no trace of the items it had already moved. A retry then started
* from the first candidate and republished them. And nothing bounded how many times one message
* could be redriven.
*
* @param request what to move
* @param approval the verified approval, when one was supplied
* @param subject the operator identity
* @param now the current instant
* @param resumeFrom how many candidates a previous attempt durably moved
* @param checkpoint called with the running completed count after each item
* @return what the pass did
*/
public RedriveReport redrive(
RedriveRequest request,
Optional<VerifiedApproval> approval,
String subject,
Instant now,
int resumeFrom,
java.util.function.IntConsumer checkpoint) {
Objects.requireNonNull(request, "request must not be null");
Objects.requireNonNull(checkpoint, "checkpoint must not be null");
if (resumeFrom < 0) {
throw new IllegalArgumentException("resumeFrom must not be negative");
}
guard.authorize(
DestructiveOperation.REDRIVE, request.source(), approval, request.dryRun(), now);
@@ -72,31 +104,57 @@ public final class RedriveService {
List<MessageId> moved = new ArrayList<>();
int failed = 0;
for (MessageId messageId : candidates) {
PublishResult result = publisher.republish(messageId, request.target(), request.redriveId());
if (result.completion() == PublishCompletion.CONFIRMED) {
source.settle(request.source(), messageId);
moved.add(messageId);
} else {
failed++;
int completed = resumeFrom;
try {
// Everything before resumeFrom was moved and settled by the previous attempt. Re-publishing
// it is not a retry, it is a duplicate.
for (MessageId messageId :
candidates.subList(Math.min(resumeFrom, candidates.size()), candidates.size())) {
if (attempt(messageId, request)) {
moved.add(messageId);
} else {
failed++;
}
completed++;
checkpoint.accept(completed);
}
} finally {
// In the finally block on purpose: an operation that dies partway must still leave a record
// of what it moved, because that record is what the resumed attempt and the incident review
// both read.
audit.record(
new dev.caskeleton.messaging.observation.MessagingAuditEvent(
"REDRIVE",
subject,
request.source().value(),
approval.map(VerifiedApproval::ticket).orElse("dry-run"),
now,
java.util.Map.of(
"redriveId", request.redriveId().toString(),
"resumedFrom", Integer.toString(resumeFrom),
"moved", Integer.toString(moved.size()),
"failed", Integer.toString(failed))));
}
audit.record(
new dev.caskeleton.messaging.observation.MessagingAuditEvent(
"REDRIVE",
subject,
request.source().value(),
approval.map(AdminApproval::ticket).orElse("dry-run"),
now,
java.util.Map.of(
"redriveId", request.redriveId().toString(),
"moved", Integer.toString(moved.size()),
"failed", Integer.toString(failed))));
return new RedriveReport(candidates.size(), moved.size(), failed, false);
}
private boolean attempt(MessageId messageId, RedriveRequest request) {
PublishResult result;
try {
result = publisher.republish(messageId, request.target(), request.redriveId());
} catch (RuntimeException failure) {
// One message that cannot be republished is a failed item, not a failed pass. Letting it
// propagate abandoned every candidate behind it.
return false;
}
if (result.completion() != PublishCompletion.CONFIRMED) {
return false;
}
source.settle(request.source(), messageId);
return true;
}
/** Reads and settles messages on a dead letter destination. */
public interface RedriveSource {
@@ -1,9 +1,9 @@
package dev.caskeleton.messaging.admin.runtime;
import dev.caskeleton.messaging.admin.AdminApproval;
import dev.caskeleton.messaging.admin.DestructiveOperation;
import dev.caskeleton.messaging.admin.DestructiveOperationGuard;
import dev.caskeleton.messaging.admin.ReplayRequest;
import dev.caskeleton.messaging.admin.VerifiedApproval;
import dev.caskeleton.messaging.observation.MessagingAuditEvent;
import java.time.Instant;
import java.util.Map;
@@ -45,13 +45,13 @@ public final class ReplayService {
* Runs one replay.
*
* @param request what to replay
* @param approval the approval, when one was supplied
* @param approval the verified approval, when one was supplied
* @param subject the operator identity
* @param now the current instant
* @return what the replay did
*/
public ReplayReport replay(
ReplayRequest request, Optional<AdminApproval> approval, String subject, Instant now) {
ReplayRequest request, Optional<VerifiedApproval> approval, String subject, Instant now) {
Objects.requireNonNull(request, "request must not be null");
Objects.requireNonNull(approval, "approval must not be null");
@@ -74,7 +74,7 @@ public final class ReplayService {
"REPLAY",
subject,
request.destination().value(),
approval.map(AdminApproval::ticket).orElse("isolated"),
approval.map(VerifiedApproval::ticket).orElse("isolated"),
now,
Map.of(
"replayId", request.replayId().toString(),
@@ -0,0 +1,145 @@
package dev.caskeleton.messaging.admin.runtime;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import dev.caskeleton.messaging.admin.AdminOperationLease;
import dev.caskeleton.messaging.admin.AdminOperationState;
import dev.caskeleton.messaging.admin.PlanDigest;
import dev.caskeleton.messaging.api.error.MessageAuthorizationException;
import java.time.Duration;
import java.time.Instant;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
/**
* The protocol the journal has to implement, proven against the in-memory one.
*
* <p>What it replaced recorded a single fact — "this approval was claimed" — before any work
* happened, in a map. An operation that died halfway had spent its approval, left no record of what
* it had already moved, and gave a retry no way to continue: either request a new approval and
* republish everything, or leave the redrive half applied.
*/
class AdminOperationJournalTest {
private static final Instant NOW = Instant.parse("2026-08-10T09:00:00Z");
private static final Duration LEASE = Duration.ofMinutes(5);
private static final String TICKET = "CHG-1042";
private static final PlanDigest DIGEST = PlanDigest.ofCanonical("REDRIVE|orders.v1.dlq|50");
private static final PlanDigest OTHER_DIGEST = PlanDigest.ofCanonical("REDRIVE|payments.dlq|50");
private final InMemoryAdminOperationJournal journal = new InMemoryAdminOperationJournal();
@Test
@DisplayName("two replicas racing for one approval produce exactly one lease")
void twoReplicasRacingProduceExactlyOneLease() {
journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
assertThatThrownBy(
() -> journal.begin(TICKET, DIGEST, "op-2", "replica-b", LEASE, NOW.plusSeconds(1)))
.as("both replicas executing one approval is a duplicate storm, not a faster redrive")
.isInstanceOf(MessageAuthorizationException.class)
.hasMessageContaining("replica-a");
}
@Test
@DisplayName("a completed operation cannot be executed again")
void aCompletedOperationCannotBeExecutedAgain() {
AdminOperationLease lease = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
journal.complete(lease, 50, NOW.plusSeconds(30));
assertThatThrownBy(
() -> journal.begin(TICKET, DIGEST, "op-2", "replica-a", LEASE, NOW.plusSeconds(60)))
.as("an arrow-up in the shell must not re-run an approved redrive")
.isInstanceOf(MessageAuthorizationException.class)
.hasMessageContaining("already executed");
}
@Test
@DisplayName("a crashed run resumes from its checkpoint rather than starting over")
void aCrashedRunResumesFromItsCheckpoint() {
AdminOperationLease first = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
journal.checkpoint(first, 30, LEASE, NOW.plusSeconds(10));
// No fail() call: this replica died. The lease simply runs out.
AdminOperationLease resumed =
journal.begin(TICKET, DIGEST, "op-1", "replica-b", LEASE, NOW.plus(Duration.ofMinutes(10)));
assertThat(resumed.resumeFrom())
.as("the 30 already moved must not be republished")
.isEqualTo(30);
assertThat(resumed.leaseToken()).isGreaterThan(first.leaseToken());
assertThat(resumed.isResumption()).isTrue();
}
@Test
@DisplayName("a failed run stays resumable without waiting for the lease to expire")
void aFailedRunStaysResumableImmediately() {
AdminOperationLease first = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
journal.checkpoint(first, 12, LEASE, NOW.plusSeconds(5));
journal.fail(first, 12, "MESSAGE_BROKER_UNAVAILABLE", NOW.plusSeconds(6));
AdminOperationLease resumed =
journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW.plusSeconds(7));
assertThat(resumed.resumeFrom()).isEqualTo(12);
assertThat(journal.find(TICKET, DIGEST))
.hasValueSatisfying(
record -> assertThat(record.state()).isEqualTo(AdminOperationState.STARTED));
}
@Test
@DisplayName("a runtime that lost its lease cannot write over the one that took over")
void aRuntimeThatLostItsLeaseCannotWriteOverTheSuccessor() {
AdminOperationLease stale = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
journal.checkpoint(stale, 30, LEASE, NOW.plusSeconds(10));
AdminOperationLease successor =
journal.begin(TICKET, DIGEST, "op-1", "replica-b", LEASE, NOW.plus(Duration.ofMinutes(10)));
journal.checkpoint(successor, 45, LEASE, NOW.plus(Duration.ofMinutes(11)));
assertThatThrownBy(() -> journal.complete(stale, 30, NOW.plus(Duration.ofMinutes(12))))
.as("a stalled replica waking up must not mark the operation done at its own count")
.isInstanceOf(MessageAuthorizationException.class)
.hasMessageContaining("no longer owns");
assertThat(journal.find(TICKET, DIGEST))
.hasValueSatisfying(
record -> {
assertThat(record.itemsCompleted()).isEqualTo(45);
assertThat(record.state()).isEqualTo(AdminOperationState.STARTED);
});
}
@Test
@DisplayName("the same ticket presented for a different plan is a different operation")
void theSameTicketForADifferentPlanIsADifferentOperation() {
journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
// Keyed on the ticket alone this would have been refused as a duplicate execution, which reads
// as protection but is actually the wrong answer: it is a different plan, and it must be
// refused by the approval binding rather than silently treated as the same operation.
assertThatCode(() -> journal.begin(TICKET, OTHER_DIGEST, "op-2", "replica-a", LEASE, NOW))
.doesNotThrowAnyException();
assertThat(journal.size()).isEqualTo(2);
}
@Test
@DisplayName("a checkpoint never moves backwards")
void aCheckpointNeverMovesBackwards() {
AdminOperationLease lease = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
journal.checkpoint(lease, 40, LEASE, NOW.plusSeconds(10));
journal.checkpoint(lease, 5, LEASE, NOW.plusSeconds(20));
assertThat(journal.find(TICKET, DIGEST))
.hasValueSatisfying(record -> assertThat(record.itemsCompleted()).isEqualTo(40));
}
@Test
@DisplayName("the in-memory journal declares itself non-durable")
void theInMemoryJournalDeclaresItselfNonDurable() {
assertThat(journal.isDurable())
.as("the starter refuses a production profile on a journal that says this")
.isFalse();
}
}
@@ -0,0 +1,231 @@
package dev.caskeleton.messaging.admin.runtime;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import dev.caskeleton.messaging.admin.AdminApproval;
import dev.caskeleton.messaging.admin.ApprovalGrant;
import dev.caskeleton.messaging.admin.ApprovedRedrivePlan;
import dev.caskeleton.messaging.admin.DestructiveOperation;
import dev.caskeleton.messaging.admin.HmacApprovalVerifier;
import dev.caskeleton.messaging.admin.RedrivePlan;
import dev.caskeleton.messaging.admin.RedriveRequest;
import dev.caskeleton.messaging.admin.VerifiedApproval;
import dev.caskeleton.messaging.api.destination.DestinationName;
import dev.caskeleton.messaging.api.error.MessageAuthorizationException;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.time.Instant;
import java.util.Optional;
import java.util.UUID;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
/**
* Whether "this was approved" is a fact or a claim the caller makes about itself.
*
* <p>It was a claim. {@code AdminApproval} was a public record and {@code ApprovedRedrivePlan} a
* public record over it, so any code that could reach the execute method could write its own
* approval and the platform believed it. Nothing bound an approval to an operation, a destination,
* a topology version, or an impact ceiling either, so a genuine approval for one redrive authorised
* a redrive of anything else inside its window.
*/
class ApprovalForgeryTest {
private static final Instant NOW = Instant.parse("2026-08-10T09:00:00Z");
private static final UUID OPERATION_ID = UUID.fromString("0199aaaa-bbbb-7ccc-8ddd-eeeeffff0000");
private static final DestinationName DLQ = new DestinationName("orders.v1.dlq");
private static final DestinationName ORDERS = new DestinationName("orders.v1");
private static final HmacApprovalVerifier ISSUER =
new HmacApprovalVerifier(
"an-issuing-key-of-at-least-32-bytes!".getBytes(StandardCharsets.UTF_8));
private static final HmacApprovalVerifier ANOTHER_ISSUER =
new HmacApprovalVerifier(
"a-different-key-of-at-least-32-bytes".getBytes(StandardCharsets.UTF_8));
@Test
@DisplayName("a verified approval cannot be constructed outside the verifier")
void aVerifiedApprovalCannotBeConstructedOutsideTheVerifier() {
// Written as a reflection probe because the property is about the type's shape, not its
// behaviour: if a public constructor or factory ever appears, every other test here becomes
// decorative.
assertThat(VerifiedApproval.class.getConstructors())
.as("a public constructor would let any caller mint an approval for itself")
.isEmpty();
assertThat(
java.util.Arrays.stream(VerifiedApproval.class.getDeclaredMethods())
.filter(method -> java.lang.reflect.Modifier.isStatic(method.getModifiers()))
.filter(method -> java.lang.reflect.Modifier.isPublic(method.getModifiers()))
.toList())
.as("nor may a public static factory")
.isEmpty();
}
@Test
@DisplayName("an unsigned or altered grant does not verify")
void anAlteredGrantDoesNotVerify() {
RedrivePlan plan = plan(50, 0);
ApprovalGrant signed = grant(plan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 50);
String signature = ISSUER.sign(signed);
ApprovalGrant raised = grant(plan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 5_000);
assertThatThrownBy(() -> ISSUER.verify(raised, signature, plan.digest(), NOW))
.as("raising the impact ceiling after the approver signed it")
.isInstanceOf(MessageAuthorizationException.class)
.hasMessageContaining("does not verify");
}
@Test
@DisplayName("a signature from another issuer does not verify")
void aSignatureFromAnotherIssuerDoesNotVerify() {
RedrivePlan plan = plan(50, 0);
ApprovalGrant granted = grant(plan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 50);
assertThatThrownBy(
() -> ISSUER.verify(granted, ANOTHER_ISSUER.sign(granted), plan.digest(), NOW))
.isInstanceOf(MessageAuthorizationException.class)
.hasMessageContaining("does not verify");
}
@Test
@DisplayName("an approval for one plan cannot execute another")
void anApprovalForOnePlanCannotExecuteAnother() {
RedrivePlan approvedPlan = plan(50, 0);
RedrivePlan biggerPlan = plan(900, 0);
ApprovalGrant granted = grant(approvedPlan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 900);
String signature = ISSUER.sign(granted);
assertThatThrownBy(() -> ISSUER.verify(granted, signature, biggerPlan.digest(), NOW))
.as("the approver saw 50 candidates; this plan moves 900")
.isInstanceOf(MessageAuthorizationException.class)
.hasMessageContaining("was issued for plan");
}
@Test
@DisplayName("a verified approval for the wrong source cannot be attached to a plan")
void aVerifiedApprovalForTheWrongSourceCannotBeAttached() {
RedrivePlan plan = plan(50, 0);
// A genuine, correctly-signed approval — for a different dead-letter destination, with this
// plan's digest smuggled in. The digest check passes; the source binding is what catches it.
ApprovalGrant granted =
new ApprovalGrant(
approval(),
"operator@example.com",
DestructiveOperation.REDRIVE,
new DestinationName("payments.v1.dlq"),
Optional.of(ORDERS),
"v1",
900,
plan.digest());
VerifiedApproval verified = ISSUER.verify(granted, ISSUER.sign(granted), plan.digest(), NOW);
assertThatThrownBy(() -> new ApprovedRedrivePlan(plan, verified, false))
.isInstanceOf(MessageAuthorizationException.class)
.hasMessageContaining("payments.v1.dlq");
}
@Test
@DisplayName("a plan beyond the approved impact ceiling cannot execute")
void aPlanBeyondTheApprovedImpactCeilingCannotExecute() {
RedrivePlan plan = plan(900, 0);
ApprovalGrant granted = grant(plan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 100);
VerifiedApproval verified = ISSUER.verify(granted, ISSUER.sign(granted), plan.digest(), NOW);
assertThatThrownBy(() -> new ApprovedRedrivePlan(plan, verified, false))
.as("the approver agreed to at most 100")
.isInstanceOf(MessageAuthorizationException.class)
.hasMessageContaining("at most 100");
}
@Test
@DisplayName("an approval for a replay cannot authorise a redrive")
void anApprovalForAReplayCannotAuthoriseARedrive() {
RedrivePlan plan = plan(50, 0);
ApprovalGrant granted = grant(plan, DestructiveOperation.REPLAY, DLQ, ORDERS, 900);
VerifiedApproval verified = ISSUER.verify(granted, ISSUER.sign(granted), plan.digest(), NOW);
assertThatThrownBy(() -> new ApprovedRedrivePlan(plan, verified, false))
.isInstanceOf(MessageAuthorizationException.class)
.hasMessageContaining("authorises REPLAY");
}
@Test
@DisplayName("an expired grant does not verify")
void anExpiredGrantDoesNotVerify() {
RedrivePlan plan = plan(50, 0);
ApprovalGrant granted = grant(plan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 50);
assertThatThrownBy(
() ->
ISSUER.verify(
granted, ISSUER.sign(granted), plan.digest(), NOW.plus(Duration.ofDays(1))))
.isInstanceOf(MessageAuthorizationException.class)
.hasMessageContaining("not valid at");
}
@Test
@DisplayName("the approver and the operator must be different people")
void theApproverAndTheOperatorMustBeDifferentPeople() {
RedrivePlan plan = plan(50, 0);
assertThatThrownBy(
() ->
new ApprovalGrant(
approval(),
"ops@example.com",
DestructiveOperation.REDRIVE,
DLQ,
Optional.of(ORDERS),
"v1",
50,
plan.digest()))
.as("an operator approving their own destructive operation is the control not existing")
.isInstanceOf(IllegalArgumentException.class);
}
@Test
@DisplayName("a correctly issued approval executes")
void aCorrectlyIssuedApprovalExecutes() {
RedrivePlan plan = plan(50, 0);
ApprovalGrant granted = grant(plan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 50);
VerifiedApproval verified = ISSUER.verify(granted, ISSUER.sign(granted), plan.digest(), NOW);
assertThatCode(
() -> new ApprovedRedrivePlan(plan, verified, false).requireExecutable(NOW, "v1"))
.doesNotThrowAnyException();
}
private static ApprovalGrant grant(
RedrivePlan plan,
DestructiveOperation operation,
DestinationName source,
DestinationName target,
long maxImpact) {
return new ApprovalGrant(
approval(),
"operator@example.com",
operation,
source,
Optional.of(target),
"v1",
maxImpact,
plan.digest());
}
private static AdminApproval approval() {
return new AdminApproval("CHG-1042", "ops@example.com", NOW, NOW.plus(Duration.ofHours(2)));
}
private static RedrivePlan plan(int candidates, int alreadyRedriven) {
return new RedrivePlan(
new RedriveRequest(OPERATION_ID, DLQ, ORDERS, 50, false),
candidates,
alreadyRedriven,
NOW,
"v1");
}
}
@@ -5,14 +5,19 @@ import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import dev.caskeleton.messaging.admin.AdminApproval;
import dev.caskeleton.messaging.admin.ApprovalGrant;
import dev.caskeleton.messaging.admin.ApprovedRedrivePlan;
import dev.caskeleton.messaging.admin.ApprovedReplayPlan;
import dev.caskeleton.messaging.admin.DestructiveOperation;
import dev.caskeleton.messaging.admin.HmacApprovalVerifier;
import dev.caskeleton.messaging.admin.PlanDigest;
import dev.caskeleton.messaging.admin.RedrivePlan;
import dev.caskeleton.messaging.admin.RedriveRequest;
import dev.caskeleton.messaging.admin.RedriveResult;
import dev.caskeleton.messaging.admin.ReplayPlan;
import dev.caskeleton.messaging.admin.ReplayRequest;
import dev.caskeleton.messaging.admin.ReplayResult;
import dev.caskeleton.messaging.admin.VerifiedApproval;
import dev.caskeleton.messaging.api.destination.DestinationName;
import dev.caskeleton.messaging.api.error.MessageAuthorizationException;
import java.time.Duration;
@@ -26,10 +31,56 @@ class ApprovedPlanExecutionTest {
private static final Instant NOW = Instant.parse("2026-08-10T09:00:00Z");
private static final UUID OPERATION_ID = UUID.fromString("0199aaaa-bbbb-7ccc-8ddd-eeeeffff0000");
private static final byte[] ISSUING_KEY =
"an-issuing-key-of-at-least-32-bytes!".getBytes(java.nio.charset.StandardCharsets.UTF_8);
private static final HmacApprovalVerifier VERIFIER = new HmacApprovalVerifier(ISSUING_KEY);
private static AdminApproval approval() {
return new AdminApproval("CHG-1042", "ops@example.com", NOW, NOW.plus(Duration.ofHours(2)));
}
/** A grant the issuer signed for exactly this plan, verified the way the runtime verifies it. */
static VerifiedApproval verified(
PlanDigest digest,
DestructiveOperation operation,
DestinationName source,
Optional<DestinationName> target,
String topologyVersion,
long maxImpact) {
ApprovalGrant grant =
new ApprovalGrant(
approval(),
"operator@example.com",
operation,
source,
target,
topologyVersion,
maxImpact,
digest);
return VERIFIER.verify(grant, VERIFIER.sign(grant), digest, NOW);
}
private static VerifiedApproval replayApproval(ReplayPlan plan) {
return verified(
plan.digest(),
DestructiveOperation.REPLAY,
plan.request().destination(),
Optional.empty(),
plan.topologyVersion(),
Long.MAX_VALUE);
}
private static VerifiedApproval redriveApproval(RedrivePlan plan) {
return verified(
plan.digest(),
DestructiveOperation.REDRIVE,
plan.request().source(),
Optional.of(plan.request().target()),
plan.topologyVersion(),
Long.MAX_VALUE);
}
private static ReplayRequest replayRequest() {
return new ReplayRequest(
OPERATION_ID,
@@ -57,9 +108,18 @@ class ApprovedPlanExecutionTest {
return new RedrivePlan(redriveRequest(), candidates, alreadyRedriven, NOW, "v1");
}
private static ApprovedReplayPlan approvedReplay(String topologyVersion) {
ReplayPlan plan = replayPlan(topologyVersion);
return new ApprovedReplayPlan(plan, replayApproval(plan));
}
private static ApprovedRedrivePlan approvedRedrive(RedrivePlan plan, boolean loopAcknowledged) {
return new ApprovedRedrivePlan(plan, redriveApproval(plan), loopAcknowledged);
}
@Test
void anExpiredApprovalCannotExecute() {
ApprovedReplayPlan approved = new ApprovedReplayPlan(replayPlan("v1"), approval());
ApprovedReplayPlan approved = approvedReplay("v1");
assertThatThrownBy(() -> approved.requireExecutable(NOW.plus(Duration.ofDays(1)), "v1"))
.isInstanceOf(MessageAuthorizationException.class)
@@ -68,25 +128,26 @@ class ApprovedPlanExecutionTest {
@Test
void aTopologyChangeSinceApprovalInvalidatesThePlan() {
ApprovedReplayPlan approved = new ApprovedReplayPlan(replayPlan("v1"), approval());
ApprovedReplayPlan approved = approvedReplay("v1");
assertThatThrownBy(() -> approved.requireExecutable(NOW, "v2"))
.as("every number in the plan was computed against the old topology")
.isInstanceOf(MessageAuthorizationException.class)
.hasMessageContaining("rebuilt");
// The signed grant carries the topology version too, so the mismatch is now caught against
// what the approver actually signed rather than against a field on the plan the caller
// handed in.
.hasMessageContaining("was signed against topology v1 but the broker is now at v2");
}
@Test
void aValidApprovalOnUnchangedTopologyExecutes() {
assertThatCode(
() -> new ApprovedReplayPlan(replayPlan("v1"), approval()).requireExecutable(NOW, "v1"))
assertThatCode(() -> approvedReplay("v1").requireExecutable(NOW, "v1"))
.doesNotThrowAnyException();
}
@Test
void aRedriveThatWouldLoopNeedsThatAcknowledgedExplicitly() {
ApprovedRedrivePlan approved =
new ApprovedRedrivePlan(redrivePlan(900, 400), approval(), false);
ApprovedRedrivePlan approved = approvedRedrive(redrivePlan(900, 400), false);
assertThatThrownBy(() -> approved.requireExecutable(NOW, "v1"))
.isInstanceOf(MessageAuthorizationException.class)
@@ -95,19 +156,13 @@ class ApprovedPlanExecutionTest {
@Test
void anAcknowledgedLoopMayProceed() {
assertThatCode(
() ->
new ApprovedRedrivePlan(redrivePlan(900, 400), approval(), true)
.requireExecutable(NOW, "v1"))
assertThatCode(() -> approvedRedrive(redrivePlan(900, 400), true).requireExecutable(NOW, "v1"))
.doesNotThrowAnyException();
}
@Test
void aRedriveWithNoPreviouslyRedrivenCandidatesNeedsNoAcknowledgement() {
assertThatCode(
() ->
new ApprovedRedrivePlan(redrivePlan(900, 0), approval(), false)
.requireExecutable(NOW, "v1"))
assertThatCode(() -> approvedRedrive(redrivePlan(900, 0), false).requireExecutable(NOW, "v1"))
.doesNotThrowAnyException();
}
@@ -139,25 +194,6 @@ class ApprovedPlanExecutionTest {
.contains("LIVE consumer group");
}
@Test
void anApprovalAuthorisesOneExecutionNotAStandingPermission() {
AdminOperationIdempotencyStore store = new AdminOperationIdempotencyStore();
assertThat(store.claim("CHG-1042", "op-1", NOW)).isEmpty();
assertThat(store.claim("CHG-1042", "op-2", NOW.plus(Duration.ofMinutes(5))))
.as("an arrow-up in the shell must not re-run an approved redrive")
.hasValueSatisfying(existing -> assertThat(existing.operationId()).isEqualTo("op-1"));
}
@Test
void aDifferentApprovalIsClaimedIndependently() {
AdminOperationIdempotencyStore store = new AdminOperationIdempotencyStore();
store.claim("CHG-1042", "op-1", NOW);
assertThat(store.claim("CHG-1043", "op-2", NOW)).isEmpty();
assertThat(store.size()).isEqualTo(2);
}
@Test
void aReplayResultFlagsAShortfallAgainstTheApprovedEstimate() {
ReplayResult result =
@@ -0,0 +1,212 @@
package dev.caskeleton.messaging.admin.runtime;
import static org.assertj.core.api.Assertions.assertThat;
import dev.caskeleton.messaging.admin.AdminApproval;
import dev.caskeleton.messaging.admin.ApprovalGrant;
import dev.caskeleton.messaging.admin.DestructiveOperation;
import dev.caskeleton.messaging.admin.DestructiveOperationGuard;
import dev.caskeleton.messaging.admin.HmacApprovalVerifier;
import dev.caskeleton.messaging.admin.PlanDigest;
import dev.caskeleton.messaging.admin.RedriveRequest;
import dev.caskeleton.messaging.admin.VerifiedApproval;
import dev.caskeleton.messaging.api.MessageId;
import dev.caskeleton.messaging.api.destination.DestinationName;
import dev.caskeleton.messaging.api.publish.ConfirmationLevel;
import dev.caskeleton.messaging.api.publish.PublishCompletion;
import dev.caskeleton.messaging.api.publish.PublishEvidence;
import dev.caskeleton.messaging.api.publish.PublishResult;
import dev.caskeleton.messaging.api.publish.RoutingOutcome;
import dev.caskeleton.messaging.observation.MessagingAuditEvent;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
import java.util.UUID;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
/**
* What a redrive does when one message fails, and what it does the second time round.
*
* <p>The loop had no per-item boundary. A publisher that threw — a broker refusing the connection
* rather than the message — propagated straight out, so every candidate behind it was abandoned and
* the audit record was never written. The operation left no trace of what it had already moved, and
* a retry started again from the first candidate and republished it.
*/
class RedriveResumptionTest {
private static final Instant NOW = Instant.parse("2026-08-10T09:00:00Z");
private static final DestinationName DLQ = new DestinationName("orders.v1.dlq");
private static final DestinationName ORDERS = new DestinationName("orders.v1");
private static final HmacApprovalVerifier ISSUER =
new HmacApprovalVerifier(
"an-issuing-key-of-at-least-32-bytes!".getBytes(java.nio.charset.StandardCharsets.UTF_8));
private final RecordingSource source = new RecordingSource();
private final RecordingAudit audit = new RecordingAudit();
@Test
@DisplayName("one message that throws does not abandon the ones behind it")
void oneThrowingMessageDoesNotAbandonTheRest() {
List<MessageId> candidates = ids(4);
source.stage(candidates);
RedriveService service =
service(
(messageId, target, redriveId) -> {
if (messageId.equals(candidates.get(1))) {
throw new IllegalStateException("the broker refused the connection");
}
return confirmed();
});
RedriveReport report = service.redrive(request(), approval(), "operator", NOW);
assertThat(report.moved()).isEqualTo(3);
assertThat(report.failed())
.as("the throwing item is a failed item, not a failed pass")
.isEqualTo(1);
assertThat(source.settled)
.containsExactly(candidates.get(0), candidates.get(2), candidates.get(3));
}
@Test
@DisplayName("the audit record is written even when the pass does not finish cleanly")
void theAuditRecordIsWrittenEvenOnFailure() {
source.stage(ids(2));
RedriveService service =
service(
(messageId, target, redriveId) -> {
throw new IllegalStateException("the broker refused the connection");
});
service.redrive(request(), approval(), "operator", NOW);
assertThat(audit.events)
.as("the record of what moved is what the incident review and the resumed run both read")
.singleElement()
.satisfies(event -> assertThat(event.details()).containsEntry("failed", "2"));
}
@Test
@DisplayName("a resumed pass does not republish what the first attempt already moved")
void aResumedPassDoesNotRepublishWhatWasAlreadyMoved() {
List<MessageId> candidates = ids(5);
source.stage(candidates);
List<MessageId> republished = new ArrayList<>();
RedriveService service =
service(
(messageId, target, redriveId) -> {
republished.add(messageId);
return confirmed();
});
RedriveReport report =
service.redrive(request(), approval(), "operator", NOW, 3, completed -> {});
assertThat(republished)
.as("republishing a settled message is a duplicate, not a retry")
.containsExactly(candidates.get(3), candidates.get(4));
assertThat(report.moved()).isEqualTo(2);
}
@Test
@DisplayName("progress is reported after each item so a crash resumes near where it stopped")
void progressIsReportedAfterEachItem() {
source.stage(ids(3));
List<Integer> checkpoints = new ArrayList<>();
RedriveService service = service((messageId, target, redriveId) -> confirmed());
service.redrive(request(), approval(), "operator", NOW, 0, checkpoints::add);
assertThat(checkpoints).containsExactly(1, 2, 3);
}
@Test
@DisplayName("resuming past the end of the batch moves nothing")
void resumingPastTheEndOfTheBatchMovesNothing() {
source.stage(ids(2));
RedriveService service = service((messageId, target, redriveId) -> confirmed());
RedriveReport report =
service.redrive(request(), approval(), "operator", NOW, 10, completed -> {});
assertThat(report.moved()).isZero();
}
private RedriveService service(RedriveService.RedrivePublisher publisher) {
// The guard holds the admin credential and every pass carries a real verified approval: this
// test is about the loop, and a gate that refused would prove nothing about it.
return new RedriveService(new DestructiveOperationGuard(true), source, publisher, audit);
}
private static Optional<VerifiedApproval> approval() {
PlanDigest digest = PlanDigest.ofCanonical("REDRIVE|orders.v1.dlq|orders.v1");
ApprovalGrant grant =
new ApprovalGrant(
new AdminApproval("CHG-1042", "ops@example.com", NOW, NOW.plus(Duration.ofHours(2))),
"operator@example.com",
DestructiveOperation.REDRIVE,
DLQ,
Optional.of(ORDERS),
"v1",
Long.MAX_VALUE,
digest);
return Optional.of(ISSUER.verify(grant, ISSUER.sign(grant), digest, NOW));
}
private static RedriveRequest request() {
return new RedriveRequest(UUID.randomUUID(), DLQ, ORDERS, 50, false);
}
private static PublishResult confirmed() {
return new PublishResult(
PublishCompletion.CONFIRMED,
PublishEvidence.confirmed(ConfirmationLevel.REPLICATION_OR_PERSISTENCE_ACK),
RoutingOutcome.ROUTED,
Optional.empty(),
1,
Duration.ofMillis(2),
Optional.empty());
}
private static List<MessageId> ids(int count) {
List<MessageId> ids = new ArrayList<>();
for (int index = 0; index < count; index++) {
ids.add(MessageId.newId());
}
return List.copyOf(ids);
}
private static final class RecordingSource implements RedriveService.RedriveSource {
private List<MessageId> staged = List.of();
private final List<MessageId> settled = new ArrayList<>();
private void stage(List<MessageId> ids) {
staged = List.copyOf(ids);
}
@Override
public List<MessageId> peek(DestinationName destination, int batchSize) {
return staged;
}
@Override
public void settle(DestinationName destination, MessageId messageId) {
settled.add(messageId);
}
}
private static final class RecordingAudit implements RedriveService.AuditSink {
private final List<MessagingAuditEvent> events = new ArrayList<>();
@Override
public void record(MessagingAuditEvent event) {
events.add(event);
}
}
}
+7 -1
View File
@@ -4,6 +4,12 @@ dependencies {
api project(':messaging:messaging-core-api')
api project(':messaging:messaging-schema-api')
implementation 'io.cloudevents:cloudevents-api:4.0.1'
// api, because CloudEventMapper's public signatures return io.cloudevents.CloudEvent.
//
// Declared `implementation`, the type appeared in this module's public API while the
// dependency was hidden from consumers: an adopter calling the documented method could not
// name its return type without adding CloudEvents to their own build, and Gradle gave them no
// hint why. A type in a public signature is part of the artifact's contract.
api 'io.cloudevents:cloudevents-api:4.0.1'
implementation 'io.cloudevents:cloudevents-core:4.0.1'
}
@@ -87,7 +87,9 @@ class CloudEventMappingTest {
void aCloudEventWithNoDataBecomesAnEmptyPayloadNotANullValue() {
CloudEvent noData =
io.cloudevents.core.builder.CloudEventBuilder.v1()
.withId(UUID.randomUUID().toString())
// A v7 id: MessageId enforces the version it documents, so a v4 arriving from a foreign
// producer is refused here exactly as it would be on the wire.
.withId(dev.caskeleton.messaging.api.MessageId.newId().value().toString())
.withSource(SOURCE)
.withType("order.created")
.withTime(java.time.OffsetDateTime.parse("2026-08-10T09:15:00Z"))
@@ -11,6 +11,13 @@ public record ContentType(String value) {
private static final int MAX_LENGTH = 160;
/** {@code type/subtype} with the optional parameters RFC 9110 allows. */
private static final java.util.regex.Pattern MEDIA_TYPE =
java.util.regex.Pattern.compile(
"[A-Za-z0-9][A-Za-z0-9!#$%&'*+.^_`|~-]*"
+ "/[A-Za-z0-9][A-Za-z0-9!#$%&'*+.^_`|~-]*"
+ "(;\\s*[A-Za-z0-9!#$%&'*+.^_`|~-]+=[^;\\s]+)*");
/** The Stable default codec's content type. */
public static final ContentType JSON = new ContentType("application/json");
@@ -27,8 +34,12 @@ public record ContentType(String value) {
if (value == null || value.isBlank() || value.length() > MAX_LENGTH) {
throw new IllegalArgumentException("contentType must contain 1 to 160 characters");
}
if (value.indexOf('/') < 0) {
throw new IllegalArgumentException("contentType must be a media type: " + value);
// A media type, not merely a string containing a slash. `a/b/c`, `/json` and `text/` all
// passed the old check and are all rejected by a broker or a codec registry later, with an
// error that names the message rather than the configuration that produced it.
if (!MEDIA_TYPE.matcher(value).matches()) {
throw new IllegalArgumentException(
"contentType must be a type/subtype media type: '" + value + '\'');
}
value = value.toLowerCase(Locale.ROOT);
}
@@ -14,8 +14,21 @@ import java.util.UUID;
*/
public record MessageId(UUID value) {
/** The UUID version this identity is documented to carry. */
private static final int VERSION_7 = 7;
public MessageId {
Objects.requireNonNull(value, "messageId value must not be null");
// The type says UUIDv7 and the constructor accepted any UUID, including v4 and the nil UUID.
// Version 7 is what makes the identity time-ordered, which is what the outbox index and every
// "oldest first" claim depend on; a v4 stored in the same column silently defeats both.
if (value.version() != VERSION_7) {
throw new IllegalArgumentException(
"a message identity is UUIDv7 (time-ordered); this is version " + value.version());
}
if (value.variant() != 2) {
throw new IllegalArgumentException("a message identity must use the RFC 4122 variant");
}
}
/**
@@ -16,5 +16,29 @@ public record MessageType(String value) {
if (value == null || value.isBlank() || value.length() > MAX_LENGTH) {
throw new IllegalArgumentException("messageType must contain 1 to 240 characters");
}
requireWireSafe(value, "a message type", MAX_LENGTH);
}
/**
* Characters that must never reach a broker.
*
* <p>Length in Java {@code char}s bounded nothing useful: a 240-character string can be 960 UTF-8
* bytes, and a control character passes a length check untouched. CR and LF split records on
* line-oriented protocols, NUL truncates them in C string handling, and both survive right up to
* the broker that mishandles them.
*/
private static void requireWireSafe(String value, String what, int maxBytes) {
for (int index = 0; index < value.length(); index++) {
char character = value.charAt(index);
if (character < 0x20 || character == 0x7F) {
throw new IllegalArgumentException(
what
+ " must not contain control characters (found U+%04X)".formatted((int) character));
}
}
int bytes = value.getBytes(java.nio.charset.StandardCharsets.UTF_8).length;
if (bytes > maxBytes) {
throw new IllegalArgumentException(what + " exceeds " + maxBytes + " UTF-8 bytes: " + bytes);
}
}
}
@@ -16,5 +16,29 @@ public record ProducerId(String value) {
if (value == null || value.isBlank() || value.length() > MAX_LENGTH) {
throw new IllegalArgumentException("producerId must contain 1 to 120 characters");
}
requireWireSafe(value, "a producer id", MAX_LENGTH);
}
/**
* Characters that must never reach a broker.
*
* <p>Length in Java {@code char}s bounded nothing useful: a 240-character string can be 960 UTF-8
* bytes, and a control character passes a length check untouched. CR and LF split records on
* line-oriented protocols, NUL truncates them in C string handling, and both survive right up to
* the broker that mishandles them.
*/
private static void requireWireSafe(String value, String what, int maxBytes) {
for (int index = 0; index < value.length(); index++) {
char character = value.charAt(index);
if (character < 0x20 || character == 0x7F) {
throw new IllegalArgumentException(
what
+ " must not contain control characters (found U+%04X)".formatted((int) character));
}
}
int bytes = value.getBytes(java.nio.charset.StandardCharsets.UTF_8).length;
if (bytes > maxBytes) {
throw new IllegalArgumentException(what + " exceeds " + maxBytes + " UTF-8 bytes: " + bytes);
}
}
}
@@ -15,10 +15,36 @@ public record HeaderName(String value) {
private static final int MAX_BYTES = 128;
/**
* The one grammar a header name may use.
*
* <p>The previous check was "not blank, at most 128 bytes", which accepted a name containing CR,
* LF, NUL, a colon or leading whitespace. Those are not cosmetic: a newline in a header name is a
* header-injection primitive on any line-oriented protocol, a NUL truncates the name in C string
* handling inside a broker, and a colon is the field separator in several wire formats. A name
* that survives one broker and corrupts the next is worse than a rejected one.
*
* <p>The character set is the HTTP token set, which every broker this platform targets accepts
* verbatim.
*/
private static final java.util.regex.Pattern TOKEN =
java.util.regex.Pattern.compile("[a-zA-Z0-9!#$%&'*+._|~-]+");
public HeaderName {
if (value == null || value.isBlank()) {
throw new IllegalArgumentException("header name must not be blank");
}
if (!value.equals(value.strip())) {
// Trimming would mean `Authorization ` and `Authorization` are the same name to the
// denylist and different names on the wire, which is precisely how the check was bypassed.
throw new IllegalArgumentException(
"header name must not have leading or trailing whitespace: '" + value + '\'');
}
if (!TOKEN.matcher(value).matches()) {
throw new IllegalArgumentException(
"header name must be an ASCII token; control characters, spaces, colons and non-ASCII"
+ " are refused because a broker that accepts one may truncate or split on it");
}
int bytes = value.getBytes(StandardCharsets.UTF_8).length;
if (bytes > MAX_BYTES) {
throw new IllegalArgumentException("header name exceeds 128 bytes: " + bytes);
@@ -11,6 +11,8 @@ import java.sql.Timestamp;
import java.time.Instant;
import java.util.Objects;
import javax.sql.DataSource;
import org.springframework.jdbc.datasource.DataSourceUtils;
import org.springframework.transaction.support.TransactionSynchronizationManager;
/**
* The PostgreSQL inbox.
@@ -48,14 +50,19 @@ public final class JdbcInboxRepository implements InboxRepository {
/**
* Reserves a message on the caller's transactional connection.
*
* <p>Package-private. It used to be public and was the only path that actually joined the
* caller's transaction, while the interface method — the one {@code IdempotentConsumer} calls —
* opened a raw connection that auto-commits. A reservation that commits on its own while the
* business side effect rolls back is a message that will never be redelivered and whose work
* never happened.
*
* @param connection the connection the business write is running on
* @param messageId the logical message identity
* @param consumerId the consumer's stable identity
* @param now the current instant
* @return true when this is the first time
*/
public boolean reserve(
Connection connection, MessageId messageId, String consumerId, Instant now) {
boolean reserve(Connection connection, MessageId messageId, String consumerId, Instant now) {
Objects.requireNonNull(connection, "connection must not be null");
Objects.requireNonNull(messageId, "messageId must not be null");
Objects.requireNonNull(now, "now must not be null");
@@ -75,11 +82,41 @@ public final class JdbcInboxRepository implements InboxRepository {
@Override
public boolean reserve(MessageId messageId, String consumerId, Instant now) {
try (Connection connection = dataSource.getConnection()) {
requireActiveTransaction("reserving an inbox message");
Connection connection = DataSourceUtils.getConnection(dataSource);
try {
return reserve(connection, messageId, consumerId, now);
} catch (SQLException exception) {
} finally {
DataSourceUtils.releaseConnection(connection, dataSource);
}
}
/**
* Refuses when there is no transaction to join, or when it cannot write.
*
* <p>The reservation and the side effect it guards have to commit or roll back together. Running
* the reservation on its own connection breaks that on the rollback path only — which is the path
* nobody exercises before production, and the one where the message is lost for good.
*/
private void requireActiveTransaction(String what) {
if (!TransactionSynchronizationManager.isActualTransactionActive()) {
throw new MessagingConfigurationException(
"INBOX_RESERVE_FAILED", "could not reserve an inbox record", exception);
"INBOX_TRANSACTION_REQUIRED",
what
+ " requires an active transaction; a reservation that commits alone marks a message"
+ " processed whose work may still roll back");
}
if (TransactionSynchronizationManager.isCurrentTransactionReadOnly()) {
throw new MessagingConfigurationException(
"INBOX_TRANSACTION_REQUIRED",
what + " requires a writable transaction; the current one is read-only");
}
if (!TransactionSynchronizationManager.hasResource(dataSource)) {
throw new MessagingConfigurationException(
"INBOX_TRANSACTION_REQUIRED",
what
+ " requires the active transaction to be bound to this DataSource; it is bound to"
+ " another, so the reservation and the side effect would commit independently");
}
}
@@ -100,6 +137,37 @@ public final class JdbcInboxRepository implements InboxRepository {
}
}
@Override
public int purgeProcessedBefore(Instant processedBefore, int limit) {
Objects.requireNonNull(processedBefore, "processedBefore must not be null");
if (limit < 1) {
throw new IllegalArgumentException("a bounded purge deletes at least one row per call");
}
try (Connection connection = dataSource.getConnection();
PreparedStatement statement =
connection.prepareStatement(
"""
WITH expired AS (
SELECT message_id, consumer_id
FROM messaging_inbox
WHERE processed_at < ?
ORDER BY processed_at
LIMIT ?
FOR UPDATE SKIP LOCKED
)
DELETE FROM messaging_inbox i
USING expired e
WHERE i.message_id = e.message_id AND i.consumer_id = e.consumer_id
""")) {
statement.setTimestamp(1, Timestamp.from(processedBefore));
statement.setInt(2, limit);
return statement.executeUpdate();
} catch (SQLException exception) {
throw new MessagingConfigurationException(
"INBOX_PURGE_FAILED", "a bounded inbox purge failed", exception);
}
}
@Override
public int purgeProcessedBefore(Instant processedBefore) {
Objects.requireNonNull(processedBefore, "processedBefore must not be null");
@@ -102,6 +102,11 @@ final class InMemoryInbox implements InboxRepository {
return reserved.contains(key(messageId, consumerId));
}
@Override
public int purgeProcessedBefore(Instant processedBefore, int limit) {
return Math.min(purgeProcessedBefore(processedBefore), limit);
}
@Override
public int purgeProcessedBefore(Instant processedBefore) {
int removed = reserved.size();
@@ -42,6 +42,11 @@ class InboxOperationsTest {
return reserved.contains(messageId.value() + "|" + consumerId);
}
@Override
public int purgeProcessedBefore(Instant processedBefore, int limit) {
return Math.min(purgeProcessedBefore(processedBefore), limit);
}
@Override
public int purgeProcessedBefore(Instant processedBefore) {
cutoffs.add(processedBefore);
@@ -48,6 +48,8 @@ class InboxPostgresIT {
.withPassword("messaging");
private DataSource dataSource;
private org.springframework.transaction.support.TransactionTemplate transactions;
private JdbcInboxRepository repository;
static boolean dockerAvailable() {
@@ -65,29 +67,40 @@ class InboxPostgresIT {
statement.execute("CREATE TABLE projections (id TEXT PRIMARY KEY, applied INT NOT NULL)");
}
repository = new JdbcInboxRepository(dataSource);
// A real Spring transaction, because the port's contract is about joining one. The previous
// tests called the Connection overload directly, which proved that overload works and said
// nothing about the interface method every consumer actually calls.
transactions =
new org.springframework.transaction.support.TransactionTemplate(
new org.springframework.jdbc.datasource.DataSourceTransactionManager(dataSource));
}
@Test
void theFirstReservationSucceedsAndTheSecondDoesNot() {
MessageId messageId = MessageId.newId();
assertThat(repository.reserve(messageId, "order-projection", NOW)).isTrue();
assertThat(repository.reserve(messageId, "order-projection", NOW)).isFalse();
assertThat(inTransaction(() -> repository.reserve(messageId, "order-projection", NOW)))
.isTrue();
assertThat(inTransaction(() -> repository.reserve(messageId, "order-projection", NOW)))
.isFalse();
}
@Test
void twoConsumersEachReserveTheSameMessageOnce() {
MessageId messageId = MessageId.newId();
assertThat(repository.reserve(messageId, "order-projection", NOW)).isTrue();
assertThat(repository.reserve(messageId, "billing-projection", NOW)).isTrue();
assertThat(repository.reserve(messageId, "order-projection", NOW)).isFalse();
assertThat(inTransaction(() -> repository.reserve(messageId, "order-projection", NOW)))
.isTrue();
assertThat(inTransaction(() -> repository.reserve(messageId, "billing-projection", NOW)))
.isTrue();
assertThat(inTransaction(() -> repository.reserve(messageId, "order-projection", NOW)))
.isFalse();
}
@Test
void aProcessedMessageIsVisibleToTheQuery() {
MessageId messageId = MessageId.newId();
repository.reserve(messageId, "order-projection", NOW);
inTransaction(() -> repository.reserve(messageId, "order-projection", NOW));
assertThat(repository.isProcessed(messageId, "order-projection")).isTrue();
assertThat(repository.isProcessed(messageId, "billing-projection")).isFalse();
@@ -109,13 +122,19 @@ class InboxPostgresIT {
void aRolledBackTransactionLeavesNoReservationAndNoSideEffect() throws SQLException {
MessageId messageId = MessageId.newId();
try (Connection connection = dataSource.getConnection()) {
connection.setAutoCommit(false);
assertThat(repository.reserve(connection, messageId, "order-projection", NOW)).isTrue();
try (Statement statement = connection.createStatement()) {
statement.execute("INSERT INTO projections (id, applied) VALUES ('o-2', 1)");
}
connection.rollback();
// Through the interface method and a real Spring transaction. This is the path a consumer
// takes; the old version called the Connection overload, which is the one path that was never
// in doubt.
try {
transactions.executeWithoutResult(
status -> {
assertThat(repository.reserve(messageId, "order-projection", NOW)).isTrue();
new org.springframework.jdbc.core.JdbcTemplate(dataSource)
.execute("INSERT INTO projections (id, applied) VALUES ('o-2', 1)");
status.setRollbackOnly();
});
} catch (RuntimeException rolledBack) {
// setRollbackOnly is the mechanism; an UnexpectedRollbackException is not a failure here.
}
assertThat(repository.isProcessed(messageId, "order-projection"))
@@ -126,25 +145,28 @@ class InboxPostgresIT {
@Test
void retentionRemovesOldRows() {
repository.reserve(MessageId.newId(), "order-projection", NOW);
MessageId purged = MessageId.newId();
inTransaction(() -> repository.reserve(purged, "order-projection", NOW));
assertThat(repository.purgeProcessedBefore(NOW.plusSeconds(1))).isEqualTo(1);
}
/** Reserves and writes inside one transaction, exactly as an idempotent consumer must. */
private void handle(MessageId messageId, AtomicInteger applied) throws SQLException {
try (Connection connection = dataSource.getConnection()) {
connection.setAutoCommit(false);
if (!repository.reserve(connection, messageId, "order-projection", NOW)) {
connection.rollback();
return;
}
try (Statement statement = connection.createStatement()) {
statement.execute("INSERT INTO projections (id, applied) VALUES ('o-1', 1)");
}
applied.incrementAndGet();
connection.commit();
}
/** Reserves and writes inside one Spring transaction, exactly as an idempotent consumer must. */
private void handle(MessageId messageId, AtomicInteger applied) {
transactions.executeWithoutResult(
status -> {
if (!repository.reserve(messageId, "order-projection", NOW)) {
status.setRollbackOnly();
return;
}
new org.springframework.jdbc.core.JdbcTemplate(dataSource)
.execute("INSERT INTO projections (id, applied) VALUES ('o-1', 1)");
applied.incrementAndGet();
});
}
private <T> T inTransaction(java.util.function.Supplier<T> work) {
return transactions.execute(status -> work.get());
}
private int projectionCount(String id) throws SQLException {
@@ -0,0 +1,98 @@
package dev.caskeleton.messaging.inbox;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import dev.caskeleton.messaging.api.MessageId;
import dev.caskeleton.messaging.api.error.MessagingConfigurationException;
import java.time.Instant;
import javax.sql.DataSource;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.springframework.transaction.support.TransactionSynchronizationManager;
/**
* What {@code reserve(...)} does when there is no transaction to join.
*
* <p>{@code IdempotentConsumer} calls this inside its transaction runner, and the implementation
* opened a raw connection that auto-commits. The reservation therefore survived a rollback of the
* side effect it was guarding: the message counts as processed, the work never happened, and
* redelivery is refused because the inbox row is already there.
*
* <p>Hermetic: the refusal has to happen before any connection is requested, and the data source
* below fails the test by being asked for one.
*/
class JdbcInboxTransactionRequirementTest {
/** A data source that fails the test by being used: the refusal must precede any connection. */
private final DataSource dataSource = new RefusingDataSource();
private final JdbcInboxRepository repository = new JdbcInboxRepository(dataSource);
@AfterEach
void clearTransactionState() {
TransactionSynchronizationManager.clear();
if (TransactionSynchronizationManager.hasResource(dataSource)) {
TransactionSynchronizationManager.unbindResource(dataSource);
}
}
@Test
@DisplayName("reserving outside a transaction is refused before a connection is opened")
void reservingOutsideATransactionIsRefused() {
assertThatThrownBy(() -> repository.reserve(MessageId.newId(), "order-projection", NOW))
.isInstanceOf(MessagingConfigurationException.class)
.hasMessageContaining("requires an active transaction");
}
@Test
@DisplayName("a read-only transaction is refused")
void aReadOnlyTransactionIsRefused() {
TransactionSynchronizationManager.setActualTransactionActive(true);
TransactionSynchronizationManager.setCurrentTransactionReadOnly(true);
assertThatThrownBy(() -> repository.reserve(MessageId.newId(), "order-projection", NOW))
.isInstanceOf(MessagingConfigurationException.class)
.hasMessageContaining("read-only");
}
@Test
@DisplayName("a transaction bound to another DataSource is refused")
void aTransactionOnAnotherDataSourceIsRefused() {
TransactionSynchronizationManager.setActualTransactionActive(true);
assertThatThrownBy(() -> repository.reserve(MessageId.newId(), "order-projection", NOW))
.as("two DataSources means two commits, which is the atomicity this exists to provide")
.isInstanceOf(MessagingConfigurationException.class)
.hasMessageContaining("bound to this DataSource");
}
@Test
@DisplayName("the refusal names a code an operator can search for")
void theRefusalNamesACode() {
assertThatThrownBy(() -> repository.reserve(MessageId.newId(), "order-projection", NOW))
.isInstanceOfSatisfying(
MessagingConfigurationException.class,
failure ->
assertThat(failure.failure().code()).isEqualTo("INBOX_TRANSACTION_REQUIRED"));
}
private static final Instant NOW = Instant.parse("2026-08-14T00:00:00Z");
/** Throws if anything asks it for a connection. */
private static final class RefusingDataSource
extends org.springframework.jdbc.datasource.AbstractDataSource {
@Override
public java.sql.Connection getConnection() {
throw new AssertionError(
"the refusal must happen before a connection is requested; opening one is the defect");
}
@Override
public java.sql.Connection getConnection(String username, String password) {
return getConnection();
}
}
}
@@ -34,9 +34,9 @@ commons-logging:commons-logging:1.3.5=runtimeClasspath,testRuntimeClasspath
info.picocli:picocli:4.7.7=checkstyle
io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnotationProcessor
io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor
io.micrometer:micrometer-commons:1.16.0=runtimeClasspath,testRuntimeClasspath
io.micrometer:micrometer-core:1.16.0=runtimeClasspath,testRuntimeClasspath
io.micrometer:micrometer-observation:1.16.0=runtimeClasspath,testRuntimeClasspath
io.micrometer:micrometer-commons:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-core:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-observation:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
javax.inject:javax.inject:1=annotationProcessor,testAnnotationProcessor
jaxen:jaxen:2.0.0=spotbugs
net.bytebuddy:byte-buddy:1.17.8=testCompileClasspath,testRuntimeClasspath
@@ -65,7 +65,7 @@ org.codehaus.plexus:plexus-utils:3.3.0=checkstyle
org.dom4j:dom4j:2.2.0=spotbugs
org.hdrhistogram:HdrHistogram:2.2.2=runtimeClasspath,testRuntimeClasspath
org.javassist:javassist:3.28.0-GA=checkstyle
org.jspecify:jspecify:1.0.0=annotationProcessor,checkstyle,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
org.jspecify:jspecify:1.0.0=annotationProcessor,checkstyle,compileClasspath,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
org.junit.jupiter:junit-jupiter-api:6.0.1=testCompileClasspath,testRuntimeClasspath
org.junit.jupiter:junit-jupiter-engine:6.0.1=testRuntimeClasspath
org.junit.jupiter:junit-jupiter-params:6.0.1=testCompileClasspath,testRuntimeClasspath
+4 -1
View File
@@ -10,7 +10,10 @@ dependencies {
api project(':messaging:messaging-admin-api')
implementation 'org.springframework.kafka:spring-kafka'
implementation 'org.apache.kafka:kafka-clients'
// api: the public constructors and mappers take and return Producer, ProducerRecord and
// RecordMetadata. Hiding the dependency does not hide the types — it only stops a consumer
// from compiling against the API this module documents.
api 'org.apache.kafka:kafka-clients'
// Test-only. The shared adapter contract is what makes "Stable" mean something, so the Kafka
// adapter runs it rather than asserting its own behaviour in its own terms.
+1 -1
View File
@@ -42,7 +42,7 @@ info.picocli:picocli:4.7.7=checkstyle
io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,jmhAnnotationProcessor,testAnnotationProcessor
io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,jmhAnnotationProcessor,testAnnotationProcessor
io.micrometer:micrometer-commons:1.16.0=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-core:1.16.0=jmhRuntimeClasspath,runtimeClasspath,testRuntimeClasspath
io.micrometer:micrometer-core:1.16.0=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-observation:1.16.0=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
javax.inject:javax.inject:1=annotationProcessor,jmhAnnotationProcessor,testAnnotationProcessor
jaxen:jaxen:2.0.0=spotbugs
@@ -32,6 +32,20 @@ public final class ContiguousPartitionOffsetTracker implements PartitionOffsetTr
delivered.computeIfAbsent(partition, key -> new ConcurrentSkipListSet<>()).add(offset);
}
/**
* Undoes a delivery registration for a record that was never submitted.
*
* <p>Needed because the executor can refuse the work after the offset has been registered. A
* delivered offset with no worker behind it holds the contiguous watermark back forever: nothing
* will ever complete it, so the partition stops committing while continuing to consume.
*/
public void forgetDelivered(TopicPartition partition, long offset) {
NavigableSet<Long> inFlight = delivered.get(partition);
if (inFlight != null) {
inFlight.remove(offset);
}
}
@Override
public void completed(TopicPartition partition, long offset) {
completed.computeIfAbsent(partition, key -> new ConcurrentSkipListSet<>()).add(offset);
@@ -64,7 +64,36 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
private final GracefulShutdownCoordinator shutdown;
private final Duration pollTimeout;
private final KafkaQuarantinePublisher quarantine;
private final Map<TopicPartition, Long> committed = new ConcurrentHashMap<>();
/**
* The assignment generation of each owned partition.
*
* <p>A settlement carries the epoch it was created under. Without one, a handler that finished
* after its partition was revoked and reassigned applied its outcome to the new assignment: the
* old epoch's acknowledgement advanced a watermark over offsets the new owner had not processed.
*/
private final Map<TopicPartition, Long> epochs = new ConcurrentHashMap<>();
private final java.util.concurrent.atomic.AtomicLong epochSequence =
new java.util.concurrent.atomic.AtomicLong();
private final java.util.concurrent.atomic.AtomicInteger staleSettlements =
new java.util.concurrent.atomic.AtomicInteger();
/**
* Work that must run on the poll thread, queued by any other thread.
*
* <p>{@code pause}, {@code resume} and {@code close} are public API and were called on the
* caller's thread, touching the {@code Consumer} directly — the one object this class documents
* as poll-thread-only. The Kafka consumer is not thread-safe, and a concurrent call is a {@code
* ConcurrentModificationException} at best and a corrupted subscription at worst.
*/
private final java.util.Queue<Runnable> control =
new java.util.concurrent.ConcurrentLinkedQueue<>();
private final AtomicBoolean active = new AtomicBoolean(true);
/**
@@ -82,6 +111,23 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
Executor handlerPool,
GracefulShutdownCoordinator shutdown,
Duration pollTimeout) {
this(
spec, consumer, handlerPool, shutdown, pollTimeout, KafkaQuarantinePublisher.unavailable());
}
/**
* A consumer runtime with a quarantine destination for undecodable records.
*
* @param quarantine where a record that cannot be decoded is written before its offset commits
*/
public KafkaConsumerRegistrar(
TransportConsumerSpec spec,
Consumer<byte[], byte[]> consumer,
Executor handlerPool,
GracefulShutdownCoordinator shutdown,
Duration pollTimeout,
KafkaQuarantinePublisher quarantine) {
this.quarantine = Objects.requireNonNull(quarantine, "quarantine must not be null");
this.spec = Objects.requireNonNull(spec, "spec must not be null");
this.consumer = Objects.requireNonNull(consumer, "consumer must not be null");
this.handlerPool = Objects.requireNonNull(handlerPool, "handlerPool must not be null");
@@ -112,6 +158,10 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
new ConsumerRebalanceListener() {
@Override
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
// Drop the epoch first. Any settlement still in flight for these partitions now
// carries a number no live assignment has, so applySettlements refuses it instead of
// moving a watermark this consumer no longer owns.
partitions.forEach(epochs::remove);
commitContiguous();
partitions.forEach(
partition -> {
@@ -124,7 +174,9 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
@Override
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
// Nothing to restore: the broker's committed offset is the source of truth.
// A fresh epoch per assignment. The broker's committed offset is still the source of
// truth for where to read; the epoch decides whose settlements may be applied.
partitions.forEach(partition -> epochs.put(partition, epochSequence.incrementAndGet()));
}
});
}
@@ -143,6 +195,8 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
public int pollOnce(Instant now) {
Objects.requireNonNull(now, "now must not be null");
drainControl();
syncEpochs();
applySettlements();
applyDueResumes(now);
commitContiguous();
@@ -153,61 +207,142 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
ConsumerRecords<byte[], byte[]> records = consumer.poll(pollTimeout);
int dispatched = 0;
for (ConsumerRecord<byte[], byte[]> record : records) {
TopicPartition partition = new TopicPartition(record.topic(), record.partition());
if (!coordinator.tryAcquire(partition)) {
// At the ceiling: stop fetching and re-read this record on a later cycle.
// Per partition, and never a global break.
//
// The loop used to iterate the whole batch and `break` on the first record it could not
// submit — which abandoned every record the same poll had returned for *other* partitions.
// The consumer's position had already advanced past them, so those offsets were not read
// again until a rebalance or a restart: a processing gap that nothing reported.
for (TopicPartition partition : records.partitions()) {
Long epoch = epochs.get(partition);
if (epoch == null) {
// Not ours: revoked between the poll and now.
continue;
}
java.util.List<ConsumerRecord<byte[], byte[]>> partitionRecords = records.records(partition);
long seekBackTo = -1L;
for (ConsumerRecord<byte[], byte[]> record : partitionRecords) {
if (seekBackTo >= 0) {
// Already stopped for this partition; the seek below covers this record too.
continue;
}
if (!coordinator.tryAcquire(partition)) {
seekBackTo = record.offset();
continue;
}
if (!shutdown.tryBeginWork()) {
coordinator.release(partition);
seekBackTo = record.offset();
continue;
}
offsets.delivered(partition, record.offset());
if (!dispatch(record, partition, now, epoch)) {
// The pool refused the work. Undo everything this iteration took so the in-flight count
// and the delivered set do not drift, and re-read the record next cycle.
offsets.forgetDelivered(partition, record.offset());
coordinator.release(partition);
shutdown.endWork();
seekBackTo = record.offset();
continue;
}
dispatched++;
}
if (seekBackTo >= 0) {
// Only this partition pauses. Its earliest unsubmitted offset is where it resumes.
consumer.pause(Set.of(partition));
consumer.seek(partition, record.offset());
break;
consumer.seek(partition, seekBackTo);
}
if (!shutdown.tryBeginWork()) {
coordinator.release(partition);
consumer.seek(partition, record.offset());
break;
}
offsets.delivered(partition, record.offset());
dispatch(record, partition, now);
dispatched++;
}
return dispatched;
}
private void dispatch(
ConsumerRecord<byte[], byte[]> record, TopicPartition partition, Instant now) {
/**
* Hands one record to the pool.
*
* @return false when the pool refused the work, so the caller can undo what it took
*/
private boolean dispatch(
ConsumerRecord<byte[], byte[]> record, TopicPartition partition, Instant now, long epoch) {
DestinationProfile profile = spec.profile();
DestinationName destination = profile.name();
Duration handlerTimeout = profile.consumer().handlerTimeout();
handlerPool.execute(
() -> {
try {
var envelope = deliveryMapper.toEnvelope(record);
int attempt = retryMetadataMapper.attemptOf(envelope);
TransportDelivery delivery =
new TransportDelivery(
envelope,
deliveryMapper.toMetadata(
destination, record, attempt, profile.consumer().group().orElse(null), now),
new QueuedSettlement(partition, record.offset()));
spec.sink().apply(delivery).toCompletableFuture().join();
} catch (RuntimeException exception) {
// The sink owns failure classification. A throw here means the delivery could not even
// be decoded, which the platform parks rather than retries.
settlements.enqueue(
new KafkaSettlementCommand(
partition, record.offset(), KafkaSettlementCommand.Kind.PARKED));
} finally {
coordinator.release(partition);
shutdown.endWork();
}
});
try {
handlerPool.execute(
() -> {
QueuedSettlement settlement = new QueuedSettlement(partition, record.offset(), epoch);
dev.caskeleton.messaging.api.MessageEnvelope<
dev.caskeleton.messaging.schema.EncodedMessage>
envelope;
try {
envelope = deliveryMapper.toEnvelope(record);
} catch (RuntimeException undecodable) {
// A decode failure is not a handler failure and must not be retried forever. The
// offset may only advance once the record exists somewhere else, so an unavailable
// quarantine leaves it uncommitted rather than dropping the message.
if (quarantine.quarantine(record, undecodable)) {
settlement.acknowledgeAfterQuarantine();
} else {
settlement.requeueAfterFailure();
}
coordinator.release(partition);
shutdown.endWork();
return;
}
try {
int attempt = retryMetadataMapper.attemptOf(envelope);
TransportDelivery delivery =
new TransportDelivery(
envelope,
deliveryMapper.toMetadata(
destination,
record,
attempt,
profile.consumer().group().orElse(null),
now),
settlement);
// The declared handler timeout is applied. It existed as a policy field and the
// worker blocked on an unbounded join, so a handler that never returned held its
// partition's permit and its offset forever — the partition simply stopped.
spec.sink()
.apply(delivery)
.toCompletableFuture()
.get(handlerTimeout.toMillis(), java.util.concurrent.TimeUnit.MILLISECONDS);
} catch (java.util.concurrent.TimeoutException timedOut) {
// Unsettled work, not a completion: pause and re-read rather than advancing past a
// message whose handler may still be running.
settlement.requeueAfterFailure();
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
settlement.requeueAfterFailure();
} catch (java.util.concurrent.ExecutionException | RuntimeException failure) {
// A handler failure is retried, not parked. PARKED completed the offset without any
// quarantine or dead-letter write, so a transient database outage silently discarded
// every message it touched.
settlement.requeueAfterFailure();
} finally {
coordinator.release(partition);
shutdown.endWork();
}
});
return true;
} catch (java.util.concurrent.RejectedExecutionException refused) {
// The caller undoes the permit, the work count and the delivered registration.
return false;
}
}
/** Applies everything workers queued. Poll thread only. */
private void applySettlements() {
for (KafkaSettlementCommand command : settlements.drain()) {
Long currentEpoch = epochs.get(command.partition());
if (currentEpoch == null || currentEpoch != command.epoch()) {
// The partition was revoked, or reassigned, after this settlement was created. Applying it
// would move the current owner's watermark using an outcome from the previous assignment,
// over offsets the current owner has not processed.
staleSettlements.incrementAndGet();
continue;
}
switch (command.kind()) {
case COMPLETE, PARKED -> offsets.completed(command.partition(), command.offset());
case PAUSE_AND_SEEK -> {
@@ -222,6 +357,21 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
}
}
/**
* Keeps the epoch map in step with the current assignment.
*
* <p>The rebalance listener is the usual source of epochs, but it is not the only way a partition
* becomes owned: a manually assigned consumer never fires it. Deriving the epoch from the
* assignment as well means fencing works in both cases, and a partition that has gone away loses
* its epoch here even if no revoke callback arrived.
*/
private void syncEpochs() {
Set<TopicPartition> current = assignment();
current.forEach(
partition -> epochs.computeIfAbsent(partition, key -> epochSequence.incrementAndGet()));
epochs.keySet().removeIf(partition -> !current.contains(partition));
}
private void applyDueResumes(Instant now) {
Map<TopicPartition, Long> due = retries.dueForResume(now);
due.forEach(
@@ -246,12 +396,32 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
continue;
}
toCommit.put(partition, new OffsetAndMetadata(offset));
committed.put(partition, offset);
offsets.pruneThrough(partition, offset - 1);
}
if (!toCommit.isEmpty()) {
consumer.commitSync(toCommit);
if (toCommit.isEmpty()) {
return;
}
// Commit first, then record it locally.
//
// The local map and the tracker used to be updated before commitSync. A failed commit
// therefore left this consumer believing an offset was committed that the broker had never
// accepted, and the pruned tracker could no longer produce it — so the next successful commit
// jumped over the gap and the records in it were never redelivered.
consumer.commitSync(toCommit);
toCommit.forEach(
(partition, metadata) -> {
committed.put(partition, metadata.offset());
offsets.pruneThrough(partition, metadata.offset() - 1);
});
}
/**
* How many settlements were refused because their assignment had been revoked.
*
* <p>Exposed rather than counted silently: a non-zero value means handlers are outliving their
* assignment, which is a rebalance or a handler-timeout problem an operator can act on.
*/
public int staleSettlements() {
return staleSettlements.get();
}
private Set<TopicPartition> assignment() {
@@ -271,18 +441,52 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
@Override
public CompletionStage<Void> pause(String scope) {
Set<TopicPartition> partitions = scoped(scope);
partitions.forEach(coordinator::pause);
consumer.pause(partitions);
return CompletableFuture.completedFuture(null);
return onPollThread(
() -> {
Set<TopicPartition> partitions = scoped(scope);
partitions.forEach(coordinator::pause);
consumer.pause(partitions);
});
}
@Override
public CompletionStage<Void> resume(String scope) {
Set<TopicPartition> partitions = scoped(scope);
partitions.forEach(coordinator::resume);
consumer.resume(partitions);
return CompletableFuture.completedFuture(null);
return onPollThread(
() -> {
Set<TopicPartition> partitions = scoped(scope);
partitions.forEach(coordinator::resume);
consumer.resume(partitions);
});
}
/**
* Queues work for the poll thread and reports when it ran.
*
* <p>The returned stage completes after the poll loop has actually applied the change, so a
* caller that awaits it knows the consumer is paused rather than merely asked to pause.
*
* <p>It therefore completes on the <em>next poll cycle</em>. A caller that blocks on it without a
* running loop waits forever, which is the honest shape of the request: there is no safe way to
* touch the consumer from another thread, so "paused" cannot be true until the loop says so.
*/
private CompletionStage<Void> onPollThread(Runnable work) {
CompletableFuture<Void> applied = new CompletableFuture<>();
control.add(
() -> {
try {
work.run();
applied.complete(null);
} catch (RuntimeException failure) {
applied.completeExceptionally(failure);
}
});
return applied;
}
private void drainControl() {
for (Runnable work = control.poll(); work != null; work = control.poll()) {
work.run();
}
}
private Set<TopicPartition> scoped(String scope) {
@@ -302,56 +506,116 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
@Override
public void close() {
if (active.compareAndSet(true, false)) {
// Close still runs here rather than through the queue: after this the poll loop stops, so
// queued work would never drain. What it does not do is race an in-progress poll — callers
// stop the loop first, and `active` makes a second close a no-op.
drainControl();
applySettlements();
commitContiguous();
consumer.close();
}
}
/** The settlement handle handed to a worker; it only enqueues. */
/**
* The settlement handle handed to a worker.
*
* <p>Two things changed. It carries the assignment epoch, so a settlement that arrives after a
* revoke is refused rather than applied to whoever owns the partition now. And it is
* one-terminal: acknowledge, requeue and discard compete for a single CAS, because a handler that
* acknowledged and then requeued used to tell the poll thread both, and the poll thread believed
* the second one.
*/
private final class QueuedSettlement
implements dev.caskeleton.messaging.transport.TransportSettlement {
private final TopicPartition partition;
private final long offset;
private final long epoch;
private final java.util.concurrent.atomic.AtomicReference<String> terminal =
new java.util.concurrent.atomic.AtomicReference<>();
private QueuedSettlement(TopicPartition partition, long offset) {
private QueuedSettlement(TopicPartition partition, long offset, long epoch) {
this.partition = partition;
this.offset = offset;
this.epoch = epoch;
}
@Override
public CompletionStage<dev.caskeleton.messaging.api.settlement.SettlementResult> acknowledge() {
settlements.enqueue(
new KafkaSettlementCommand(partition, offset, KafkaSettlementCommand.Kind.COMPLETE));
return CompletableFuture.completedFuture(
dev.caskeleton.messaging.api.settlement.SettlementResult.settled());
return terminalOnce(
"acknowledge",
() -> {
settlements.enqueue(
new KafkaSettlementCommand(
partition, offset, KafkaSettlementCommand.Kind.COMPLETE, epoch));
return dev.caskeleton.messaging.api.settlement.SettlementResult.settled();
});
}
@Override
public CompletionStage<dev.caskeleton.messaging.api.settlement.SettlementResult> requeue(
Duration delay) {
retries.pauseUntil(partition, offset, delay, Instant.now());
settlements.enqueue(
new KafkaSettlementCommand(
partition, offset, KafkaSettlementCommand.Kind.PAUSE_AND_SEEK));
return CompletableFuture.completedFuture(
new dev.caskeleton.messaging.api.settlement.SettlementResult(
dev.caskeleton.messaging.api.settlement.SettlementCompletion.UNKNOWN,
dev.caskeleton.messaging.api.settlement.SettlementEvidence.unknown(),
java.util.Optional.of(
dev.caskeleton.messaging.api.error.FailureDescriptor.of(
dev.caskeleton.messaging.api.error.FailureCategory.PROCESSING_TRANSIENT,
"KAFKA_PAUSED_FOR_RETRY",
"the partition was paused and will be re-read"))));
return terminalOnce("requeue", () -> enqueueRequeue(delay));
}
@Override
public CompletionStage<dev.caskeleton.messaging.api.settlement.SettlementResult> discard() {
return terminalOnce(
"discard",
() -> {
settlements.enqueue(
new KafkaSettlementCommand(
partition, offset, KafkaSettlementCommand.Kind.PARKED, epoch));
return dev.caskeleton.messaging.api.settlement.SettlementResult.settled();
});
}
/** Used by the worker once an undecodable record has been written to quarantine. */
private void acknowledgeAfterQuarantine() {
if (terminal.compareAndSet(null, "acknowledge")) {
settlements.enqueue(
new KafkaSettlementCommand(
partition, offset, KafkaSettlementCommand.Kind.PARKED, epoch));
}
}
/** Used by the worker when the handler failed or timed out and never settled. */
private void requeueAfterFailure() {
if (terminal.compareAndSet(null, "requeue")) {
enqueueRequeue(Duration.ZERO);
}
}
private dev.caskeleton.messaging.api.settlement.SettlementResult enqueueRequeue(
Duration delay) {
retries.pauseUntil(partition, offset, delay, Instant.now());
settlements.enqueue(
new KafkaSettlementCommand(partition, offset, KafkaSettlementCommand.Kind.PARKED));
return CompletableFuture.completedFuture(
dev.caskeleton.messaging.api.settlement.SettlementResult.settled());
new KafkaSettlementCommand(
partition, offset, KafkaSettlementCommand.Kind.PAUSE_AND_SEEK, epoch));
return new dev.caskeleton.messaging.api.settlement.SettlementResult(
dev.caskeleton.messaging.api.settlement.SettlementCompletion.UNKNOWN,
dev.caskeleton.messaging.api.settlement.SettlementEvidence.unknown(),
java.util.Optional.of(
dev.caskeleton.messaging.api.error.FailureDescriptor.of(
dev.caskeleton.messaging.api.error.FailureCategory.PROCESSING_TRANSIENT,
"KAFKA_PAUSED_FOR_RETRY",
"the partition was paused and will be re-read")));
}
private CompletionStage<dev.caskeleton.messaging.api.settlement.SettlementResult> terminalOnce(
String what,
java.util.function.Supplier<dev.caskeleton.messaging.api.settlement.SettlementResult>
call) {
if (!terminal.compareAndSet(null, what)) {
return CompletableFuture.failedFuture(
new IllegalStateException(
"this delivery was already settled by "
+ terminal.get()
+ "; a second "
+ what
+ " would contradict what the poll thread has already been told"));
}
return CompletableFuture.completedFuture(call.get());
}
}
}
@@ -67,7 +67,27 @@ public final class KafkaHeaderMapper {
.baggage()
.ifPresent(value -> put(headers, ReservedHeaders.BAGGAGE, value));
envelope.headers().asMap().forEach((name, value) -> put(headers, name.value(), value.value()));
// Application headers are written last and must not be able to reach a reserved name.
//
// They used to be appended after the canonical ones, and the consumer reads `lastHeader` — so a
// row in the outbox carrying `msg.id` overwrote the canonical message identity on the wire.
// That identity is what inbox deduplication and the audit trail key on, so a forged one
// corrupts another message's dedup state rather than merely its own.
envelope
.headers()
.asMap()
.forEach(
(name, value) -> {
if (ReservedHeaders.isReserved(name.value())) {
throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException(
"RESERVED_HEADER_FORGED",
"application header '"
+ name.value()
+ "' uses the reserved namespace; the canonical value is written by the"
+ " platform and must not be overridable by message content");
}
put(headers, name.value(), value.value());
});
return headers;
}
@@ -40,9 +40,28 @@ import org.apache.kafka.clients.producer.ProducerRecord;
*/
public final class KafkaMessagingTransport implements MessagingTransport {
/**
* What this adapter can prove, with deduplicated publish set to false.
*
* <p>It was {@code true}, justified by producer idempotence. Producer idempotence deduplicates
* <em>sequence retries within one producer session</em>: the producer id is reassigned on
* restart, so the same logical message published again after a crash is a new sequence and the
* broker stores it twice. The capability an application reads is "the broker suppresses duplicate
* publishes of a stable id", which needs a persistent message-id store that this platform does
* not have.
*
* <p>Declaring it true means {@code PublishDeduplication} is accepted and silently does nothing —
* the caller believes the broker is deduplicating and skips the idempotency it would otherwise
* build. False makes that request a startup failure instead, which is the whole reason the flag
* exists.
*
* <p>The order of these flags is the record's: brokerAcknowledgement, replication evidence,
* per-message settlement, batch settlement, ordered stream, keyed ordering, replay, delayed
* delivery, broker transaction, deduplicated publish, native dead letter, topology management.
*/
private static final MessagingCapabilities CAPABILITIES =
new MessagingCapabilities(
true, true, true, true, true, true, true, false, true, true, false, true);
true, true, true, true, true, true, true, false, true, false, false, true);
private final String brokerName;
private final long generation;
@@ -0,0 +1,31 @@
package dev.caskeleton.messaging.kafka;
import org.apache.kafka.clients.consumer.ConsumerRecord;
/**
* Where a record that cannot be decoded is written before its offset is committed.
*
* <p>The consumer used to mark an undecodable record {@code PARKED}, and the poll thread treated
* {@code PARKED} exactly like {@code COMPLETE} — the offset advanced and the record was gone. There
* was no quarantine collection, no dead-letter publish and no record of what had been dropped, so
* "parked" named a state nothing implemented.
*
* <p>Committing past a record is only safe once the record exists somewhere else. An implementation
* that cannot write returns {@code false}, and the consumer leaves the offset uncommitted rather
* than losing the message quietly.
*/
@FunctionalInterface
public interface KafkaQuarantinePublisher {
/**
* Writes an undecodable record somewhere durable.
*
* @return true when the write is confirmed, so the source offset may be committed
*/
boolean quarantine(ConsumerRecord<byte[], byte[]> record, Throwable reason);
/** The default: nothing is configured, so nothing may be committed past. */
static KafkaQuarantinePublisher unavailable() {
return (record, reason) -> false;
}
}
@@ -72,9 +72,22 @@ public final class KafkaRetryMetadataMapper {
return 1;
}
try {
return Math.max(1, Integer.parseInt(value.orElseThrow().value()));
} catch (NumberFormatException exception) {
return 1;
int attempt = Integer.parseInt(value.orElseThrow().value());
if (attempt < 1) {
throw new NumberFormatException("attempt " + attempt);
}
return attempt;
} catch (NumberFormatException malformed) {
// Fail closed rather than resetting to 1.
//
// Returning 1 for an unreadable header restarts the retry budget on every redelivery, so a
// message that should have been dead-lettered after N attempts is retried forever — and the
// header is caller-influenced, which makes "unreadable" a way to defeat the cap rather than
// an accident.
throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException(
"RETRY_ATTEMPT_MALFORMED",
"the retry attempt header is not a positive integer; the message is quarantined rather"
+ " than restarting its retry budget");
}
}
}
@@ -93,11 +93,15 @@ public final class KafkaSecurityConfigurer {
properties.put(SASL_JAAS_CONFIG, scramJaas(scram.credentialId(), resolved));
}
case BrokerCredentialProfile.OAuth2 oauth -> {
credentials.resolve(oauth.credentialId(), now);
properties.put(SASL_MECHANISM, "OAUTHBEARER");
properties.put(
SASL_JAAS_CONFIG,
"org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;");
// Refused rather than half-configured. Setting the mechanism name without a callback
// handler produces a client that authenticates with nothing and fails at connect time,
// which is later and harder to attribute than failing here.
throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException(
"SASL_OAUTH_NOT_CONFIGURED",
"OAuth bearer needs a login callback handler and a token endpoint; this platform"
+ " configures the mechanism name only, so the profile is refused rather than"
+ " producing a client that cannot authenticate: "
+ oauth.credentialId());
}
case BrokerCredentialProfile.MutualTls ignored -> properties.put(SASL_MECHANISM, "NONE");
case BrokerCredentialProfile.UsernamePassword plain -> {
@@ -122,14 +126,48 @@ public final class KafkaSecurityConfigurer {
}
private static String scramJaas(String credentialId, CredentialRuntime resolved) {
return "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"%s\" "
.formatted(credentialId)
+ "password=\"%s\";".formatted(new String(resolved.material()));
return jaas("org.apache.kafka.common.security.scram.ScramLoginModule", credentialId, resolved);
}
private static String plainJaas(String credentialId, CredentialRuntime resolved) {
return "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"%s\" "
.formatted(credentialId)
+ "password=\"%s\";".formatted(new String(resolved.material()));
return jaas("org.apache.kafka.common.security.plain.PlainLoginModule", credentialId, resolved);
}
/**
* Builds a JAAS entry with both values escaped.
*
* <p>The password was interpolated into a quoted string unescaped. A perfectly ordinary generated
* secret containing a quote or a backslash then produced a JAAS config that either fails to parse
* — an authentication error that names nothing useful — or, with a quote and a semicolon,
* terminates the option early and injects another one. A secret is attacker-influenced in exactly
* one sense that matters here: nobody reads it before it is used.
*/
private static String jaas(String loginModule, String credentialId, CredentialRuntime resolved) {
return loginModule
+ " required username=\""
+ escapeJaas(credentialId)
+ "\" password=\""
+ escapeJaas(new String(resolved.material()))
+ "\";";
}
/**
* Escapes a JAAS quoted-string value.
*
* <p>Backslash first, then quote — the other order double-escapes the backslash it just added.
* Control characters are refused rather than escaped: JAAS has no escape for them, and a secret
* containing one cannot be expressed in this format at all.
*/
private static String escapeJaas(String value) {
for (int index = 0; index < value.length(); index++) {
char character = value.charAt(index);
if (character < 0x20 || character == 0x7F) {
throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException(
"JAAS_VALUE_UNREPRESENTABLE",
"a credential containing a control character cannot be expressed in a JAAS entry;"
+ " configure the client with a callback handler instead");
}
}
return value.replace("\\", "\\\\").replace("\"", "\\\"");
}
}
@@ -15,7 +15,18 @@ import org.apache.kafka.common.TopicPartition;
* @param offset the record offset
* @param kind what to do
*/
public record KafkaSettlementCommand(TopicPartition partition, long offset, Kind kind) {
public record KafkaSettlementCommand(TopicPartition partition, long offset, Kind kind, long epoch) {
/**
* A command without an epoch, for callers that predate assignment fencing.
*
* <p>Epoch 0 never matches a live assignment, so such a command is always stale. That is
* deliberate: a settlement that cannot say which assignment it belongs to cannot be shown to
* belong to the current one.
*/
public KafkaSettlementCommand(TopicPartition partition, long offset, Kind kind) {
this(partition, offset, kind, 0L);
}
/** What a queued settlement instruction asks the poll thread to do. */
public enum Kind {
@@ -78,6 +78,67 @@ public final class KafkaTransactionalPublisher {
* @param profiles the destination profile for each output destination
* @param inputOffsets the input offsets to commit with the outputs
*/
/**
* Begins a transaction, runs the caller's work inside it, then sends outputs and offsets.
*
* <p>The processor used to run the handler and only afterwards hand the delivery here — so {@code
* beginTransaction} happened *after* the handler had already finished. Anything the handler
* produced itself, and any external effect it triggered, was therefore outside the transaction
* the documentation promised it was in: a handler that succeeded and a commit that then failed
* left the handler's work applied with its input offsets unsent.
*
* <p>The body runs between begin and commit, and every failure path aborts.
*
* @param body the handler, executed inside the transaction
* @return whatever the body produced
*/
public <T> T inTransaction(
KafkaTransactionalDelivery delivery,
Map<String, DestinationProfile> profiles,
Map<TopicPartition, OffsetAndMetadata> inputOffsets,
java.util.function.Supplier<T> body) {
Objects.requireNonNull(delivery, "delivery must not be null");
Objects.requireNonNull(profiles, "profiles must not be null");
Objects.requireNonNull(inputOffsets, "inputOffsets must not be null");
Objects.requireNonNull(body, "body must not be null");
producer.beginTransaction();
try {
T value = body.get();
sendOutputsAndOffsets(delivery, profiles, inputOffsets);
producer.commitTransaction();
return value;
} catch (RuntimeException failure) {
// Handler failure, send failure, offset failure and commit failure all land here. Under
// read_committed a consumer sees none of the outputs, and the input offsets are unsent, so
// the source is redelivered.
producer.abortTransaction();
throw failure;
}
}
@SuppressWarnings("FutureReturnValueIgnored")
private void sendOutputsAndOffsets(
KafkaTransactionalDelivery delivery,
Map<String, DestinationProfile> profiles,
Map<TopicPartition, OffsetAndMetadata> inputOffsets) {
for (KafkaTransactionalOutput output : delivery.outputs()) {
DestinationProfile profile = profiles.get(output.destination().value());
if (profile == null) {
throw new MessagingConfigurationException(
"DESTINATION_NOT_REGISTERED",
"no destination profile is registered for " + output.destination().value());
}
producer.send(
mapper.toRecord(
new dev.caskeleton.messaging.transport.TransportPublishRequest(
profile,
output.envelope(),
dev.caskeleton.messaging.api.publish.PublishOptions.defaults())));
}
producer.sendOffsetsToTransaction(inputOffsets, groupMetadata.get());
}
@SuppressWarnings("FutureReturnValueIgnored")
public void sendInTransaction(
KafkaTransactionalDelivery delivery,
@@ -51,8 +51,15 @@ public final class SpringKafkaTransactionalProcessor implements KafkaTransaction
Objects.requireNonNull(handler, "handler must not be null");
try {
T result = handler.apply(delivery);
publisher.sendInTransaction(delivery, profiles, inputOffsets.apply(delivery));
// The handler runs between beginTransaction and commitTransaction.
//
// It used to run first, with the transaction opened afterwards inside sendInTransaction — so
// the contract this interface documents ("the callback executes inside the Kafka
// transaction") was false for every output the handler produced itself and every external
// effect it triggered.
T result =
publisher.inTransaction(
delivery, profiles, inputOffsets.apply(delivery), () -> handler.apply(delivery));
return CompletableFuture.completedFuture(result);
} catch (RuntimeException exception) {
return CompletableFuture.failedFuture(exception);
@@ -2,6 +2,7 @@ package dev.caskeleton.messaging.kafka;
import static org.assertj.core.api.Assertions.assertThat;
import dev.caskeleton.messaging.api.MessageId;
import dev.caskeleton.messaging.api.header.ReservedHeaders;
import dev.caskeleton.messaging.transport.GracefulShutdownCoordinator;
import dev.caskeleton.messaging.transport.TransportConsumerSpec;
@@ -14,7 +15,6 @@ import java.util.ArrayList;
import java.util.Deque;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionStage;
import java.util.concurrent.Executor;
@@ -22,6 +22,7 @@ import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.MockConsumer;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.header.internals.RecordHeaders;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
/**
@@ -157,10 +158,18 @@ class KafkaConsumerRegistrarTest {
KafkaConsumerRegistrar registrar = registrar(this::settleImmediately);
assign();
registrar.pause("").toCompletableFuture().join();
// pause/resume are queued for the poll thread rather than touching the Consumer from the
// caller's thread — the Kafka consumer is not thread-safe, and this class documents itself as
// poll-thread-only. The stage therefore completes on the next poll cycle, which is what
// "applied" means for a runtime whose loop is running.
java.util.concurrent.CompletionStage<Void> paused = registrar.pause("");
registrar.pollOnce(NOW);
paused.toCompletableFuture().join();
assertThat(consumer.paused()).contains(PARTITION);
registrar.resume("").toCompletableFuture().join();
java.util.concurrent.CompletionStage<Void> resumed = registrar.resume("");
registrar.pollOnce(NOW);
resumed.toCompletableFuture().join();
assertThat(consumer.paused()).doesNotContain(PARTITION);
}
@@ -175,9 +184,174 @@ class KafkaConsumerRegistrarTest {
registrar.pollOnce(NOW);
assertThat(handled).isEmpty();
assertThat(registrar.committedOffset(PARTITION))
.as("PARKED completed the offset with no quarantine write, so the record was simply lost")
.isEmpty();
}
@Test
@DisplayName("an undecodable record commits only once quarantine confirms the write")
void anUndecodableRecordCommitsAfterQuarantineConfirms() {
java.util.List<Long> quarantined = new java.util.ArrayList<>();
KafkaConsumerRegistrar registrar =
new KafkaConsumerRegistrar(
new TransportConsumerSpec(KafkaContractHarness.profile(), this::settleImmediately),
consumer,
handlers,
new GracefulShutdownCoordinator(Duration.ofSeconds(30)),
Duration.ofMillis(1),
(record, reason) -> {
quarantined.add(record.offset());
return true;
});
assign();
addRecordWithoutIdentityHeaders(10);
registrar.pollOnce(NOW);
handlers.runAll();
registrar.pollOnce(NOW);
assertThat(quarantined).containsExactly(10L);
assertThat(registrar.committedOffset(PARTITION)).hasValue(11L);
}
@Test
@DisplayName("one partition hitting its ceiling does not abandon the other's records")
void oneSaturatedPartitionDoesNotAbandonAnother() {
org.apache.kafka.common.TopicPartition second =
new org.apache.kafka.common.TopicPartition(PARTITION.topic(), PARTITION.partition() + 1);
consumer.assign(List.of(PARTITION, second));
consumer.updateBeginningOffsets(Map.of(PARTITION, 10L, second, 10L));
consumer.seek(PARTITION, 10L);
consumer.seek(second, 10L);
KafkaConsumerRegistrar registrar = registrar(this::settleImmediately);
// Two records on the first partition (its ordering unit allows one in flight) and one on the
// second. The old loop broke out of the whole batch at the first record it could not submit.
addRecord(10);
addRecord(11);
addRecordOn(second, 10);
int dispatched = registrar.pollOnce(NOW);
assertThat(dispatched)
.as("the second partition's record was returned by the same poll and must not be dropped")
.isEqualTo(2);
}
@Test
@DisplayName("a rejected executor submission leaves nothing in flight and re-reads the record")
void aRejectedSubmissionIsUndone() {
KafkaConsumerRegistrar registrar =
new KafkaConsumerRegistrar(
new TransportConsumerSpec(KafkaContractHarness.profile(), this::settleImmediately),
consumer,
command -> {
throw new java.util.concurrent.RejectedExecutionException("pool is shutting down");
},
new GracefulShutdownCoordinator(Duration.ofSeconds(30)),
Duration.ofMillis(1));
assign();
addRecord(10);
int dispatched = registrar.pollOnce(NOW);
assertThat(dispatched).isZero();
assertThat(registrar.committedOffset(PARTITION))
.as("an offset registered as delivered with no worker behind it never completes")
.isEmpty();
assertThat(consumer.position(PARTITION))
.as("the record must be re-read next cycle")
.isEqualTo(10L);
}
@Test
@DisplayName("a settlement from a revoked assignment is refused")
void aStaleSettlementIsRefused() {
KafkaConsumerRegistrar registrar = registrar(this::deferSettlement);
assign();
addRecord(10);
registrar.pollOnce(NOW);
// The partition goes away and comes back: a new assignment, a new epoch.
consumer.assign(List.of());
registrar.pollOnce(NOW);
assign();
registrar.pollOnce(NOW);
// Now the old handler finally acknowledges.
handlers.runAll();
registrar.pollOnce(NOW);
assertThat(registrar.staleSettlements())
.as("an acknowledgement from the previous assignment must not move this one's watermark")
.isPositive();
assertThat(registrar.committedOffset(PARTITION)).isEmpty();
}
@Test
@DisplayName("a second terminal call on one delivery is refused")
void aSecondTerminalCallIsRefused() {
java.util.List<Throwable> refusals = new java.util.ArrayList<>();
KafkaConsumerRegistrar registrar =
registrar(
delivery -> {
delivery.settlement().acknowledge().toCompletableFuture().join();
delivery
.settlement()
.requeue(Duration.ZERO)
.toCompletableFuture()
.handle(
(result, failure) -> {
if (failure != null) {
refusals.add(failure);
}
return null;
})
.join();
return java.util.concurrent.CompletableFuture.completedFuture(null);
});
assign();
addRecord(10);
registrar.pollOnce(NOW);
handlers.runAll();
assertThat(refusals)
.as("telling the poll thread both 'done' and 're-read this' lets the second one win")
.hasSize(1);
}
private void addRecordOn(org.apache.kafka.common.TopicPartition partition, long offset) {
RecordHeaders headers = new RecordHeaders();
headers.add(
ReservedHeaders.MESSAGE_ID,
// A v7 identity: MessageId now enforces the version it documents, and a v4 here would be
// refused by the envelope mapper exactly as a foreign producer's would.
MessageId.newId().value().toString().getBytes(StandardCharsets.UTF_8));
headers.add(ReservedHeaders.MESSAGE_TYPE, "order.created".getBytes(StandardCharsets.UTF_8));
headers.add(ReservedHeaders.SCHEMA_VERSION, "1".getBytes(StandardCharsets.UTF_8));
headers.add(ReservedHeaders.CONTENT_TYPE, "application/json".getBytes(StandardCharsets.UTF_8));
consumer.addRecord(
new ConsumerRecord<>(
partition.topic(),
partition.partition(),
offset,
NOW.toEpochMilli(),
org.apache.kafka.common.record.TimestampType.CREATE_TIME,
0,
0,
null,
"{}".getBytes(StandardCharsets.UTF_8),
headers,
java.util.Optional.empty()));
}
/** Never settles inside the handler, so the settlement arrives later. */
private CompletionStage<Void> deferSettlement(TransportDelivery delivery) {
delivery.settlement().acknowledge();
return java.util.concurrent.CompletableFuture.completedFuture(null);
}
private void assign() {
consumer.assign(List.of(PARTITION));
consumer.updateBeginningOffsets(Map.of(PARTITION, 10L));
@@ -187,7 +361,10 @@ class KafkaConsumerRegistrarTest {
private void addRecord(long offset) {
RecordHeaders headers = new RecordHeaders();
headers.add(
ReservedHeaders.MESSAGE_ID, UUID.randomUUID().toString().getBytes(StandardCharsets.UTF_8));
ReservedHeaders.MESSAGE_ID,
// A v7 identity: MessageId now enforces the version it documents, and a v4 here would be
// refused by the envelope mapper exactly as a foreign producer's would.
MessageId.newId().value().toString().getBytes(StandardCharsets.UTF_8));
headers.add(ReservedHeaders.MESSAGE_TYPE, "order.created".getBytes(StandardCharsets.UTF_8));
headers.add(ReservedHeaders.SCHEMA_VERSION, "1".getBytes(StandardCharsets.UTF_8));
headers.add(ReservedHeaders.CONTENT_TYPE, "application/json".getBytes(StandardCharsets.UTF_8));
@@ -0,0 +1,105 @@
package dev.caskeleton.messaging.kafka;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import dev.caskeleton.messaging.api.error.MessagingConfigurationException;
import dev.caskeleton.messaging.security.BrokerCredentialProfile;
import dev.caskeleton.messaging.security.BrokerSecurityProfile;
import dev.caskeleton.messaging.security.CredentialProvider;
import dev.caskeleton.messaging.security.CredentialRuntimeRegistry;
import java.time.Duration;
import java.time.Instant;
import java.util.Map;
import java.util.Optional;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
/**
* What reaches a JAAS entry, and what is refused outright.
*
* <p>The password was interpolated into a quoted string with no escaping. A generated secret
* containing a quote or a backslash produced a config that fails to parse — an authentication error
* naming nothing useful — and one containing a quote and a semicolon terminated the option early
* and injected another.
*/
class KafkaSecurityConfigurerTest {
private static final Instant NOW = Instant.parse("2026-08-14T00:00:00Z");
@Test
@DisplayName("a secret containing a quote and a backslash is escaped, not injected")
void aSecretWithQuotesIsEscaped() {
Map<String, Object> properties = configure("pa\"ss\\word;");
String jaas = String.valueOf(properties.get("sasl.jaas.config"));
assertThat(jaas)
.as("an unescaped quote closes the option and everything after it becomes new options")
.contains("password=\"pa\\\"ss\\\\word;\";");
assertThat(jaas)
.as("the raw form would close the password option and make the rest new options")
.doesNotContain("password=\"pa\"");
}
@Test
@DisplayName("a secret containing a control character is refused")
void aSecretWithAControlCharacterIsRefused() {
assertThatThrownBy(() -> configure("pass\nword"))
.isInstanceOf(MessagingConfigurationException.class)
.hasMessageContaining("control character");
}
@Test
@DisplayName("an OAuth profile is refused rather than half-configured")
void anOAuthProfileIsRefused() {
CredentialRuntimeRegistry credentials = registry("secret");
BrokerCredentialProfile oauth = new BrokerCredentialProfile.OAuth2("kafka-oauth");
assertThatThrownBy(
() ->
new KafkaSecurityConfigurer(
credentials, new dev.caskeleton.messaging.security.BrokerTlsPolicy())
.configure(profile(oauth), oauth, java.util.List.of("TLSv1.3"), NOW))
.as("naming the mechanism without a callback handler fails at connect time instead")
.isInstanceOf(MessagingConfigurationException.class)
.hasMessageContaining("callback handler");
}
private static Map<String, Object> configure(String secret) {
return new KafkaSecurityConfigurer(
registry(secret), new dev.caskeleton.messaging.security.BrokerTlsPolicy())
.configure(
profile(new BrokerCredentialProfile.SaslScram("kafka-app")),
new BrokerCredentialProfile.SaslScram("kafka-app"),
java.util.List.of("TLSv1.3"),
NOW);
}
private static BrokerSecurityProfile profile(BrokerCredentialProfile credential) {
return new BrokerSecurityProfile(
"kafka-primary",
true,
true,
true,
credential,
credential,
Optional.empty(),
dev.caskeleton.messaging.security.DestinationAccessPolicy.denyAll());
}
private static CredentialRuntimeRegistry registry(String secret) {
return new CredentialRuntimeRegistry(
new CredentialProvider() {
@Override
public char[] resolve(String credentialId) {
return secret.toCharArray();
}
@Override
public Optional<Instant> expiresAt(String credentialId) {
return Optional.of(NOW.plus(Duration.ofHours(1)));
}
},
Duration.ofMinutes(5));
}
}
@@ -0,0 +1,192 @@
package dev.caskeleton.messaging.kafka;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import dev.caskeleton.messaging.api.ContentType;
import dev.caskeleton.messaging.api.MessageEnvelope;
import dev.caskeleton.messaging.api.MessageId;
import dev.caskeleton.messaging.api.MessageType;
import dev.caskeleton.messaging.api.ProducerId;
import dev.caskeleton.messaging.api.SchemaVersion;
import dev.caskeleton.messaging.api.TraceContext;
import dev.caskeleton.messaging.api.header.MessageHeaders;
import dev.caskeleton.messaging.policy.DestinationProfile;
import dev.caskeleton.messaging.schema.EncodedMessage;
import java.nio.charset.StandardCharsets;
import java.time.Instant;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.atomic.AtomicBoolean;
import org.apache.kafka.clients.consumer.ConsumerGroupMetadata;
import org.apache.kafka.clients.producer.MockProducer;
import org.apache.kafka.common.serialization.ByteArraySerializer;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
/**
* Where the handler runs relative to {@code beginTransaction}.
*
* <p>{@code KafkaTransactionalProcessor} documents that the callback executes inside the Kafka
* transaction. It did not: the processor ran the handler and only afterwards called {@code
* sendInTransaction}, which is where {@code beginTransaction} lived. Everything the handler
* produced itself — and every external effect it triggered — was outside the transaction, so a
* handler that succeeded followed by a failed commit left its work applied with the input offsets
* unsent.
*/
class KafkaTransactionOrderingTest {
private final RecordingProducer producer = new RecordingProducer();
private final KafkaTransactionalPublisher publisher =
new KafkaTransactionalPublisher(
producer, () -> new ConsumerGroupMetadata("order-projection"));
@Test
@DisplayName("the transaction is open while the handler runs")
void theTransactionIsOpenWhileTheHandlerRuns() {
AtomicBoolean openDuringHandler = new AtomicBoolean();
SpringKafkaTransactionalProcessor processor = processor();
processor
.processInTransaction(
delivery(),
ignored -> {
// The real publisher has already begun the transaction by the time the handler runs.
openDuringHandler.set(producer.calls().equals(List.of("begin")));
return "handled";
})
.toCompletableFuture()
.join();
assertThat(openDuringHandler)
.as("the handler used to run before beginTransaction was ever called")
.isTrue();
assertThat(producer.calls()).containsExactly("begin", "offsets", "commit");
}
@Test
@DisplayName("a handler failure aborts and never commits")
void aHandlerFailureAbortsAndNeverCommits() {
SpringKafkaTransactionalProcessor processor = processor();
assertThatThrownBy(
() ->
processor
.processInTransaction(
delivery(),
ignored -> {
throw new IllegalStateException("the projection is down");
})
.toCompletableFuture()
.join())
.hasCauseInstanceOf(IllegalStateException.class);
assertThat(producer.calls())
.as("under read_committed nothing may become visible, and the offsets stay unsent")
.containsExactly("begin", "abort");
}
@Test
@DisplayName("a commit failure aborts")
void aCommitFailureAborts() {
producer.failCommit();
SpringKafkaTransactionalProcessor processor = processor();
assertThatThrownBy(
() ->
processor
.processInTransaction(delivery(), ignored -> "handled")
.toCompletableFuture()
.join())
.hasCauseInstanceOf(IllegalStateException.class);
assertThat(producer.calls()).endsWith("abort");
}
/** A MockProducer that records the transaction calls in the order they happen. */
private static final class RecordingProducer extends MockProducer<byte[], byte[]> {
private final List<String> calls = new java.util.ArrayList<>();
private boolean commitFails;
private RecordingProducer() {
// The four-arg constructor with a null partitioner: this producer never partitions, because
// the assertions are about transaction call order rather than record placement.
super(true, null, new ByteArraySerializer(), new ByteArraySerializer());
initTransactions();
}
private void failCommit() {
commitFails = true;
}
private List<String> calls() {
return List.copyOf(calls);
}
@Override
public void beginTransaction() {
calls.add("begin");
super.beginTransaction();
}
@Override
public void sendOffsetsToTransaction(
Map<
org.apache.kafka.common.TopicPartition,
org.apache.kafka.clients.consumer.OffsetAndMetadata>
offsets,
ConsumerGroupMetadata groupMetadata) {
calls.add("offsets");
super.sendOffsetsToTransaction(offsets, groupMetadata);
}
@Override
public void commitTransaction() {
if (commitFails) {
throw new IllegalStateException("commit failed");
}
calls.add("commit");
super.commitTransaction();
}
@Override
public void abortTransaction() {
calls.add("abort");
super.abortTransaction();
}
}
private SpringKafkaTransactionalProcessor processor() {
DestinationProfile profile = KafkaContractHarness.profile();
return new SpringKafkaTransactionalProcessor(
publisher, Map.of(profile.name().value(), profile), ignored -> Map.of());
}
private static KafkaTransactionalDelivery delivery() {
return new KafkaTransactionalDelivery(envelope(), List.of());
}
private static MessageEnvelope<EncodedMessage> envelope() {
return new MessageEnvelope<>(
MessageId.newId(),
new MessageType("order.created"),
new SchemaVersion(1),
Instant.parse("2026-08-15T00:00:00Z"),
Optional.empty(),
new ProducerId("orders"),
Optional.empty(),
Optional.empty(),
ContentType.JSON,
Optional.empty(),
Optional.empty(),
Optional.empty(),
TraceContext.none(),
MessageHeaders.empty(),
new EncodedMessage(
"{}".getBytes(StandardCharsets.UTF_8), ContentType.JSON, Optional.empty()));
}
}
@@ -0,0 +1,123 @@
package dev.caskeleton.messaging.kafka;
import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import dev.caskeleton.messaging.api.ContentType;
import dev.caskeleton.messaging.api.MessageEnvelope;
import dev.caskeleton.messaging.api.MessageId;
import dev.caskeleton.messaging.api.MessageType;
import dev.caskeleton.messaging.api.ProducerId;
import dev.caskeleton.messaging.api.SchemaVersion;
import dev.caskeleton.messaging.api.TraceContext;
import dev.caskeleton.messaging.api.error.MessagingConfigurationException;
import dev.caskeleton.messaging.api.header.HeaderName;
import dev.caskeleton.messaging.api.header.HeaderValue;
import dev.caskeleton.messaging.api.header.MessageHeaders;
import dev.caskeleton.messaging.api.header.ReservedHeaders;
import dev.caskeleton.messaging.schema.EncodedMessage;
import java.nio.charset.StandardCharsets;
import java.time.Instant;
import java.util.Map;
import java.util.Optional;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
/**
* What happens when message content claims a reserved header.
*
* <p>The canonical headers were written first and the envelope's own headers appended afterwards,
* and the consumer reads {@code lastHeader}. A row in the outbox carrying {@code msg.id} therefore
* overwrote the canonical identity on the wire — and that identity is what inbox deduplication and
* the audit trail key on, so a forged one corrupts a *different* message's dedup state.
*/
class ReservedHeaderForgeryTest {
private final KafkaHeaderMapper mapper = new KafkaHeaderMapper();
@Test
@DisplayName("an application header in the reserved namespace is refused")
void aForgedReservedHeaderIsRefused() {
assertThatThrownBy(
() -> mapper.toKafkaHeaders(envelopeWithHeader(ReservedHeaders.MESSAGE_ID, "forged")))
.isInstanceOf(MessagingConfigurationException.class)
.hasMessageContaining("reserved namespace");
}
@Test
@DisplayName("a forged message type is refused too")
void aForgedMessageTypeIsRefused() {
assertThatThrownBy(
() ->
mapper.toKafkaHeaders(
envelopeWithHeader(ReservedHeaders.MESSAGE_TYPE, "other.type")))
.isInstanceOf(MessagingConfigurationException.class);
}
@Test
@DisplayName("an ordinary application header still travels")
void anOrdinaryApplicationHeaderStillTravels() {
assertThatCode(() -> mapper.toKafkaHeaders(envelopeWithHeader("x-tenant-hint", "acme")))
.doesNotThrowAnyException();
}
@Test
@DisplayName("a malformed retry attempt is quarantined, not reset to one")
void aMalformedRetryAttemptIsQuarantined() {
assertThatThrownBy(
() ->
new KafkaRetryMetadataMapper()
.attemptOf(
envelopeWithPlatformHeader(ReservedHeaders.RETRY_ATTEMPT, "not-a-number")))
.as("resetting to 1 restarts the budget on every redelivery, so the cap never applies")
.isInstanceOf(MessagingConfigurationException.class)
.hasMessageContaining("retry attempt");
}
@Test
@DisplayName("a negative retry attempt is quarantined")
void aNegativeRetryAttemptIsQuarantined() {
assertThatThrownBy(
() ->
new KafkaRetryMetadataMapper()
.attemptOf(envelopeWithPlatformHeader(ReservedHeaders.RETRY_ATTEMPT, "-3")))
.isInstanceOf(MessagingConfigurationException.class);
}
/**
* Headers built through the platform factory, which is the path an outbox row takes.
*
* <p>{@code MessageHeaders.application} already refuses reserved names, so business code could
* never forge one. The gap was the platform factory: {@code OutboxEnvelopeFactory} promoted a
* row's arbitrary {@code Map<String,String>} straight through it, and the mapper then appended
* the result after the canonical headers.
*/
private static MessageEnvelope<EncodedMessage> envelopeWithHeader(String name, String value) {
return envelope(MessageHeaders.platform(Map.of(new HeaderName(name), new HeaderValue(value))));
}
private static MessageEnvelope<EncodedMessage> envelopeWithPlatformHeader(
String name, String value) {
return envelope(MessageHeaders.platform(Map.of(new HeaderName(name), new HeaderValue(value))));
}
private static MessageEnvelope<EncodedMessage> envelope(MessageHeaders headers) {
return new MessageEnvelope<>(
MessageId.newId(),
new MessageType("order.created"),
new SchemaVersion(1),
Instant.parse("2026-08-15T00:00:00Z"),
Optional.empty(),
new ProducerId("orders"),
Optional.empty(),
Optional.empty(),
ContentType.JSON,
Optional.empty(),
Optional.empty(),
Optional.empty(),
TraceContext.none(),
headers,
new EncodedMessage(
"{}".getBytes(StandardCharsets.UTF_8), ContentType.JSON, Optional.empty()));
}
}
@@ -32,9 +32,9 @@ commons-io:commons-io:2.21.0=spotbugs
info.picocli:picocli:4.7.7=checkstyle
io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnotationProcessor
io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor
io.micrometer:micrometer-commons:1.16.0=runtimeClasspath,testRuntimeClasspath
io.micrometer:micrometer-core:1.16.0=runtimeClasspath,testRuntimeClasspath
io.micrometer:micrometer-observation:1.16.0=runtimeClasspath,testRuntimeClasspath
io.micrometer:micrometer-commons:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-core:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-observation:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.nats:jnats:2.26.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
javax.inject:javax.inject:1=annotationProcessor,testAnnotationProcessor
jaxen:jaxen:2.0.0=spotbugs
@@ -41,7 +41,7 @@ public final class NatsJetStreamProfileValidator {
throw new MessagingCapabilityUnavailableException(
"NATS_DISABLED",
"the NATS JetStream adapter is experimental and disabled unless "
+ "backend.messaging.experimental.nats=true");
+ "app.messaging.experimental.nats=true");
}
if (profile.deliveryGuarantee() == DeliveryGuarantee.AT_LEAST_ONCE && !jetStreamEnabled) {
throw new IllegalArgumentException(
@@ -39,6 +39,13 @@ import java.util.function.Function;
* stored it and lost only the acknowledgement, and the deduplication window is what makes retrying
* it safe when the profile enables one.
*
* <p>This is an Experimental contract seam, not a Stable adapter. It exercises the transport SPI
* against a publish operation the application supplies; it does not ship a NATS client bridge,
* connection management, or reconnection. Only a failure the operation reports as a {@link
* NatsPreSendRejection} is treated as definitely not transmitted — every other failure, recognised
* or not, is ambiguous, because a caller that reads {@code REJECTED} may republish under a new
* identity and duplicate a message the server already stored.
*
* <p>The adapter reports {@code nativeDeadLetter} as false. JetStream terminates a message at its
* delivery limit rather than routing it anywhere, so the platform parks it via {@link
* NatsMaxDeliverParkingWorkflow}; claiming a native dead-letter would send operators looking for a
@@ -119,9 +126,20 @@ public final class NatsJetStreamTransport implements MessagingTransport {
"PAYLOAD_TOO_LARGE", "encoded payload is " + size + " bytes, limit is " + limit));
}
// The caller's own deadline, honoured rather than ignored. Without it a stalled JetStream
// publish holds the caller for as long as the SDK's default takes, and the timeout the caller
// set to bound its own latency budget means nothing.
long startedAt = System.nanoTime();
return publish
.publish(profile.subject(), deduplicationId(request), request)
.handle((position, failure) -> failure == null ? confirmed(position) : classify(failure))
.toCompletableFuture()
.orTimeout(
request.options().timeout().toMillis(), java.util.concurrent.TimeUnit.MILLISECONDS)
.handle(
(position, failure) ->
failure == null
? confirmed(position, elapsedSince(startedAt))
: classify(failure, elapsedSince(startedAt)))
.thenApply(TransportPublishResult::new);
}
@@ -180,7 +198,11 @@ public final class NatsJetStreamTransport implements MessagingTransport {
.map(window -> request.envelope().messageId().value().toString());
}
private static PublishResult confirmed(NatsStreamPosition position) {
private static Duration elapsedSince(long startedAtNanos) {
return Duration.ofNanos(System.nanoTime() - startedAtNanos);
}
private static PublishResult confirmed(NatsStreamPosition position, Duration elapsed) {
return new PublishResult(
PublishCompletion.CONFIRMED,
// The publish acknowledgement names the stream and sequence the message landed in, which is
@@ -193,45 +215,49 @@ public final class NatsJetStreamTransport implements MessagingTransport {
RoutingOutcome.ROUTED,
Optional.of(position),
1,
Duration.ZERO,
elapsed,
Optional.empty());
}
private static PublishResult classify(Throwable failure) {
/**
* Decides what a publish failure means, from typed signals only.
*
* <p>Only {@link NatsPreSendRejection} — thrown by the publish operation when it knows the send
* never left the client — reports {@code REJECTED / NOT_TRANSMITTED}. Everything else is {@code
* AMBIGUOUS}, including failures this adapter does not recognise, because the alternative is
* telling a caller nothing was transmitted when the server may already hold the message.
*/
private static PublishResult classify(Throwable failure, Duration elapsed) {
Throwable cause = failure instanceof CompletionException ? failure.getCause() : failure;
String name = cause == null ? "" : cause.getClass().getSimpleName();
if (name.contains("Timeout")) {
if (cause instanceof NatsPreSendRejection rejection) {
return new PublishResult(
PublishCompletion.AMBIGUOUS,
new PublishEvidence(
true, TransmissionEvidence.MAY_HAVE_BEEN_TRANSMITTED, false, ConfirmationLevel.NONE),
RoutingOutcome.UNKNOWN,
PublishCompletion.REJECTED,
PublishEvidence.notTransmitted(),
RoutingOutcome.NOT_APPLICABLE,
Optional.empty(),
1,
Duration.ZERO,
elapsed,
Optional.of(
FailureDescriptor.of(
FailureCategory.TRANSIENT_INFRASTRUCTURE,
"NATS_PUBLISH_TIMEOUT",
"the publish acknowledgement did not arrive; the stream may already hold it")));
FailureCategory.CONFIGURATION, rejection.code(), rejection.getMessage())));
}
// "no responders" means no JetStream server is listening on the subject at all. That is a
// configuration fault, not a transient one: retrying it will fail identically until the stream
// is declared.
boolean noStream = name.contains("NoRespond") || name.contains("Jetstream");
boolean timedOut = cause instanceof java.util.concurrent.TimeoutException;
return new PublishResult(
PublishCompletion.REJECTED,
PublishEvidence.notTransmitted(),
RoutingOutcome.NOT_APPLICABLE,
PublishCompletion.AMBIGUOUS,
new PublishEvidence(
true, TransmissionEvidence.MAY_HAVE_BEEN_TRANSMITTED, false, ConfirmationLevel.NONE),
RoutingOutcome.UNKNOWN,
Optional.empty(),
1,
Duration.ZERO,
elapsed,
Optional.of(
FailureDescriptor.of(
noStream ? FailureCategory.CONFIGURATION : FailureCategory.PERMANENT_BUSINESS,
noStream ? "NATS_STREAM_NOT_DECLARED" : "NATS_PUBLISH_REJECTED",
"the JetStream publish was rejected: " + name)));
FailureCategory.TRANSIENT_INFRASTRUCTURE,
timedOut ? "NATS_PUBLISH_TIMEOUT" : "NATS_PUBLISH_OUTCOME_UNKNOWN",
timedOut
? "the publish acknowledgement did not arrive; the stream may already hold it"
: "the publish failed in a way this adapter does not classify; the stream may"
+ " already hold the message")));
}
private static TransportPublishResult rejectedLocally(String code, String message) {
@@ -242,6 +268,7 @@ public final class NatsJetStreamTransport implements MessagingTransport {
RoutingOutcome.NOT_APPLICABLE,
Optional.empty(),
1,
// A local refusal: nothing was attempted, so no time was spent on the wire.
Duration.ZERO,
Optional.of(FailureDescriptor.of(FailureCategory.PERMANENT_BUSINESS, code, message))));
}
@@ -0,0 +1,65 @@
package dev.caskeleton.messaging.nats;
import java.util.Objects;
/**
* The one signal that lets this adapter report {@code REJECTED / NOT_TRANSMITTED}.
*
* <p>Classification used to work by reading the exception's class simple name and matching
* substrings — {@code "Timeout"} meant ambiguous, anything else meant rejected. Two things are
* wrong with that. A class name is not part of any SDK's contract: it changes between versions, and
* an unrelated exception whose name happens to contain the substring is misread. And defaulting the
* unknown case to {@code REJECTED} tells the caller nothing was transmitted, so a caller that
* republishes under a new identity duplicates a message the server may well have stored.
*
* <p>The publish operation throws this only when it knows the send never left the client: an
* argument the SDK refused, a connection that was already closed, a subject the client could not
* resolve. Everything else — including anything the adapter does not recognise — is ambiguous, and
* ambiguity is the honest answer when the write may have landed.
*/
public final class NatsPreSendRejection extends RuntimeException {
private static final long serialVersionUID = 1L;
private final String code;
/**
* Creates a pre-send rejection.
*
* @param code the sanitised platform code for the refusal
* @param message a sanitised description
*/
public NatsPreSendRejection(String code, String message) {
super(message);
this.code = requireCode(code);
}
/**
* Creates a pre-send rejection wrapping the SDK's own exception.
*
* @param code the sanitised platform code for the refusal
* @param message a sanitised description
* @param cause the SDK exception that proves the send never started
*/
public NatsPreSendRejection(String code, String message, Throwable cause) {
super(message, cause);
this.code = requireCode(code);
}
private static String requireCode(String code) {
Objects.requireNonNull(code, "code must not be null");
if (code.isBlank()) {
throw new IllegalArgumentException("code must not be blank");
}
return code;
}
/**
* The sanitised code reported to the caller.
*
* @return the code
*/
public String code() {
return code;
}
}
@@ -102,14 +102,76 @@ class NatsAdapterContractTest {
}
@Test
void aMissingStreamIsAConfigurationFaultNotATransientOne() {
void aFailureNamedLikeAKnownOneIsStillAmbiguous() {
// The class name used to be the classifier: an exception whose simple name contained
// "NoRespond" was reported REJECTED / NOT_TRANSMITTED. A name is not an SDK contract, and a
// caller that reads NOT_TRANSMITTED republishes under a new identity.
PublishResult result =
await(failingWith(new IllegalStateException("NoRespondersException")).publish(request(64)));
assertThat(result.completion()).isEqualTo(PublishCompletion.REJECTED);
assertThat(result.completion()).isEqualTo(PublishCompletion.AMBIGUOUS);
assertThat(result.failure())
.hasValueSatisfying(
failure -> assertThat(failure.code()).isEqualTo("NATS_PUBLISH_REJECTED"));
failure -> assertThat(failure.code()).isEqualTo("NATS_PUBLISH_OUTCOME_UNKNOWN"));
}
@Test
void onlyATypedPreSendRejectionReportsNothingTransmitted() {
PublishResult result =
await(
failingWith(
new NatsPreSendRejection(
"NATS_CONNECTION_CLOSED", "the client was closed before the send"))
.publish(request(64)));
assertThat(result.completion()).isEqualTo(PublishCompletion.REJECTED);
assertThat(result.evidence().transmission())
.isEqualTo(dev.caskeleton.messaging.api.publish.TransmissionEvidence.NOT_TRANSMITTED);
assertThat(result.failure())
.hasValueSatisfying(
failure -> assertThat(failure.code()).isEqualTo("NATS_CONNECTION_CLOSED"));
}
@Test
void aWrappedUnknownFailureIsAmbiguousRatherThanRejected() {
PublishResult result =
await(
failingWith(
new java.util.concurrent.CompletionException(
new RuntimeException("something the adapter has never seen")))
.publish(request(64)));
assertThat(result.completion()).isEqualTo(PublishCompletion.AMBIGUOUS);
}
@Test
void aPublishThatNeverCompletesIsBoundedByTheCallersTimeout() {
NatsJetStreamTransport transport =
new NatsJetStreamTransport(
"nats-primary",
1L,
profile(Optional.empty()),
(subject, deduplicationId, request) -> new java.util.concurrent.CompletableFuture<>());
PublishResult result = await(transport.publish(requestWithTimeout(Duration.ofMillis(50))));
assertThat(result.completion())
.as("the caller's deadline used to be ignored entirely")
.isEqualTo(PublishCompletion.AMBIGUOUS);
assertThat(result.failure())
.hasValueSatisfying(
failure -> assertThat(failure.code()).isEqualTo("NATS_PUBLISH_TIMEOUT"));
}
@Test
void theReportedElapsedTimeIsMeasuredRatherThanZero() {
PublishResult result =
await(
failingWith(new java.util.concurrent.TimeoutException("no ack")).publish(request(64)));
assertThat(result.elapsed())
.as("every outcome reported Duration.ZERO, so latency evidence was fiction")
.isGreaterThanOrEqualTo(Duration.ZERO);
}
@Test
@@ -190,14 +252,16 @@ class NatsAdapterContractTest {
}
@Test
void aFailureCategoryIsAlwaysAttachedToARejection() {
void aFailureCategoryIsAlwaysAttachedToAFailure() {
PublishResult result =
await(failingWith(new IllegalStateException("boom")).publish(request(64)));
assertThat(result.failure())
.hasValueSatisfying(
failure ->
assertThat(failure.category()).isEqualTo(FailureCategory.PERMANENT_BUSINESS));
assertThat(failure.category())
.as("an unclassified failure is transient, not a permanent business rejection")
.isEqualTo(FailureCategory.TRANSIENT_INFRASTRUCTURE));
}
private static NatsJetStreamProfile profile(Optional<Duration> deduplicationWindow) {
@@ -216,6 +280,15 @@ class NatsAdapterContractTest {
destinationProfile(), envelope(payloadBytes), PublishOptions.defaults());
}
private static TransportPublishRequest requestWithTimeout(Duration timeout) {
PublishOptions defaults = PublishOptions.defaults();
return new TransportPublishRequest(
destinationProfile(),
envelope(64),
new PublishOptions(
timeout, defaults.confirmation(), defaults.deduplication(), defaults.brokerHints()));
}
private static DestinationProfile destinationProfile() {
return new DestinationProfile(
new DestinationName("orders.v1"),
@@ -3,5 +3,7 @@ apply plugin: 'java-library'
dependencies {
api project(':messaging:messaging-core-api')
implementation 'io.micrometer:micrometer-core'
// api: MessagingMetrics' public constructor takes a MeterRegistry, so wiring it requires
// naming the type.
api 'io.micrometer:micrometer-core'
}
@@ -54,11 +54,19 @@ public final class CardinalityGuard {
if (values.contains(value)) {
return true;
}
if (values.size() >= limitPerDimension) {
return false;
// Size-then-add was not atomic: N threads could each read size == limit - 1 and each add, so
// the configured limit was an average rather than a bound. A guard that can be exceeded under
// load is no guard — load is when it matters.
synchronized (values) {
if (values.contains(value)) {
return true;
}
if (values.size() >= limitPerDimension) {
return false;
}
values.add(value);
return true;
}
values.add(value);
return true;
}
/**
@@ -69,13 +77,30 @@ public final class CardinalityGuard {
*/
public boolean admit(MessagingTags tags) {
Objects.requireNonNull(tags, "tags must not be null");
// Preflight every dimension before committing any of them.
//
// The loop used to admit each dimension as it went, so a tag set rejected on its last
// dimension had already permanently added the earlier ones — spending the budget of a bounded
// dimension on a series that was never emitted.
Map<String, String> pending = tags.asMap();
for (Map.Entry<String, String> entry : pending.entrySet()) {
if (!wouldAdmit(entry.getKey(), entry.getValue())) {
return false;
}
}
boolean admitted = true;
for (Map.Entry<String, String> entry : tags.asMap().entrySet()) {
for (Map.Entry<String, String> entry : pending.entrySet()) {
admitted &= admit(entry.getKey(), entry.getValue());
}
return admitted;
}
/** Whether a value is already known or would still fit, without recording it. */
private boolean wouldAdmit(String dimension, String value) {
Set<String> values = observed.get(dimension);
return values == null || values.contains(value) || values.size() < limitPerDimension;
}
/**
* Returns how many distinct values a dimension has taken.
*
@@ -119,17 +119,31 @@ public final class MessagingMetrics implements MessagingObservation {
// Redact before anything else touches the values. Diagnostics are the one place where a caller
// can pass arbitrary keys, so this is where a secret would otherwise reach the backend.
Map<String, String> safe = redactor.sanitize(diagnostics);
// The value never becomes a tag.
//
// It used to: every diagnostic key and value was attached to a counter, behind a guard that
// only bounded the base dimensions. One unique message id, exception message or URL per
// request created one meter series per request — permanently, in the backend and in this
// process's heap — and the redactor only masks keys it recognises, so free-form text carried
// whatever it carried.
//
// What stays is the shape: which diagnostic keys occurred, counted against the bounded base
// dimensions. The values belong in a structured log or a trace event, where they are bounded
// by retention rather than by cardinality.
admitted(tags)
.ifPresent(
micrometerTags ->
safe.forEach(
(key, value) ->
registry
.counter(
"messaging.diagnostics",
micrometerTags.and(
Tag.of("diagnostic", key), Tag.of("value", value)))
.increment()));
safe.keySet()
.forEach(
key -> {
if (guard.admit("diagnostic", key)) {
registry
.counter(
"messaging.diagnostics",
micrometerTags.and(Tag.of("diagnostic", key)))
.increment();
}
}));
}
/**
@@ -84,7 +84,61 @@ class MessagingSecretLeakTest {
.map(Tag::getValue)
.toList();
assertThat(emitted).doesNotContain("hunter2").contains("unroutable");
// No diagnostic value reaches the registry at all — not the secret, and not the innocuous one
// either. A value tag is one meter series per distinct value, permanently, and the redactor
// only masks keys it recognises: free-form text carries whatever it carries.
assertThat(emitted).doesNotContain("hunter2").doesNotContain("unroutable");
assertThat(
registry.getMeters().stream()
.map(Meter::getId)
.flatMap(id -> id.getTags().stream())
.filter(tag -> tag.getKey().equals("diagnostic"))
.map(Tag::getValue)
.toList())
.as("the shape is kept: which diagnostic keys occurred, against bounded base dimensions")
.contains("reason");
}
@Test
@org.junit.jupiter.api.DisplayName("ten thousand distinct diagnostic values create no new series")
void distinctDiagnosticValuesCreateNoNewSeries() {
MessagingMetrics metrics = new MessagingMetrics(registry, new CardinalityGuard(100), redactor);
MessagingTags tags = MessagingTags.of("kafka", "orders.v1", "publish", "rejected");
for (int index = 0; index < 10_000; index++) {
metrics.recordDiagnostics(tags, Map.of("reason", "unique-value-" + index));
}
assertThat(
registry.getMeters().stream()
.filter(meter -> meter.getId().getName().equals("messaging.diagnostics"))
.toList())
.as("one unique message id per request used to create one meter series per request")
.hasSize(1);
}
@Test
@org.junit.jupiter.api.DisplayName("concurrent admission never exceeds the configured limit")
void concurrentAdmissionNeverExceedsTheLimit() throws Exception {
CardinalityGuard guard = new CardinalityGuard(10);
java.util.concurrent.ExecutorService pool =
java.util.concurrent.Executors.newFixedThreadPool(8);
try {
java.util.List<java.util.concurrent.Callable<Boolean>> work =
java.util.stream.IntStream.range(0, 200)
.<java.util.concurrent.Callable<Boolean>>mapToObj(
index -> () -> guard.admit("destination", "d-" + index))
.toList();
for (java.util.concurrent.Future<Boolean> future : pool.invokeAll(work)) {
future.get();
}
} finally {
pool.shutdownNow();
}
assertThat(guard.cardinality("destination"))
.as("size-then-add is not atomic, so the limit was an average rather than a bound")
.isEqualTo(10);
}
@Test
@@ -31,6 +31,9 @@ class SecretLeakStaticScanTest {
Pattern.compile("(System\\.out|System\\.err|printStackTrace\\()");
/** A Java string literal, including escaped quotes. */
/** Whether the scanner is currently inside a text block. */
private static final ThreadLocal<Boolean> IN_TEXT_BLOCK = ThreadLocal.withInitial(() -> false);
private static final Pattern STRING_LITERAL = Pattern.compile("\"(\\\\.|[^\"\\\\])*\"");
/** The operand immediately before or after a string concatenation. */
@@ -129,6 +132,13 @@ class SecretLeakStaticScanTest {
* @return true when the line concatenates a sensitive value
*/
private static boolean leaksASensitiveValue(String line) {
if (line.contains("\"\"\"") || IN_TEXT_BLOCK.get()) {
// A text block is SQL or JSON, not Java code. The scanner replaced ordinary string literals
// and then read the contents of text blocks as if they were expressions, so a column named
// `lease_token` in an UPDATE statement looked like a concatenated credential.
IN_TEXT_BLOCK.set(IN_TEXT_BLOCK.get() != line.contains("\"\"\""));
return false;
}
String code = STRING_LITERAL.matcher(line).replaceAll("\"\"");
if (!code.contains("+")) {
return false;
@@ -6,6 +6,11 @@ dependencies {
api project(':messaging:messaging-policy')
api project(':messaging:messaging-observability')
// The destructive-operation journal lives here because it needs exactly what the outbox needs:
// one relational database every replica can see, and a migration lane that already exists. The
// contract it implements belongs to the admin API.
api project(':messaging:messaging-admin-api')
implementation 'org.springframework:spring-jdbc'
implementation 'org.springframework:spring-tx'
@@ -39,9 +39,9 @@ commons-logging:commons-logging:1.3.5=compileClasspath,runtimeClasspath,testComp
info.picocli:picocli:4.7.7=checkstyle
io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnotationProcessor
io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor
io.micrometer:micrometer-commons:1.16.0=runtimeClasspath,testRuntimeClasspath
io.micrometer:micrometer-core:1.16.0=runtimeClasspath,testRuntimeClasspath
io.micrometer:micrometer-observation:1.16.0=runtimeClasspath,testRuntimeClasspath
io.micrometer:micrometer-commons:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-core:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-observation:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
javax.inject:javax.inject:1=annotationProcessor,testAnnotationProcessor
jaxen:jaxen:2.0.0=spotbugs
net.bytebuddy:byte-buddy:1.17.8=testCompileClasspath,testRuntimeClasspath
@@ -0,0 +1,326 @@
package dev.caskeleton.messaging.outbox;
import dev.caskeleton.messaging.admin.AdminOperationJournal;
import dev.caskeleton.messaging.admin.AdminOperationLease;
import dev.caskeleton.messaging.admin.AdminOperationRecord;
import dev.caskeleton.messaging.admin.AdminOperationState;
import dev.caskeleton.messaging.admin.PlanDigest;
import dev.caskeleton.messaging.api.error.MessageAuthorizationException;
import dev.caskeleton.messaging.api.error.MessagingConfigurationException;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Timestamp;
import java.time.Duration;
import java.time.Instant;
import java.util.Objects;
import java.util.Optional;
import javax.sql.DataSource;
import org.springframework.jdbc.datasource.DataSourceUtils;
/**
* The shared, durable destructive-operation journal.
*
* <p>Lives beside the outbox because it needs the same thing the outbox needs and nothing more: one
* relational database that every replica can see. The uniqueness that stops a second execution is
* the primary key on {@code (approval_ticket, plan_digest)}, enforced by the database rather than
* by a check-then-act in application code — two replicas that read "no row" at the same instant
* would both proceed, and only the constraint makes exactly one of them win.
*
* <p>Takeover is conditional on the row still carrying the token the caller read. That is what
* fences a stalled runtime out: it wakes up holding token 3, the row is at token 4, and every write
* it attempts is refused rather than overwriting the replica that resumed the work.
*/
public final class JdbcAdminOperationJournal implements AdminOperationJournal {
private static final String SELECT =
"""
SELECT approval_ticket, plan_digest, operation_id, state, items_completed,
lease_owner, lease_token, lease_expires_at, started_at, finished_at, failure_code
FROM messaging_admin_operation
WHERE approval_ticket = ? AND plan_digest = ?
""";
private static final String INSERT =
"""
INSERT INTO messaging_admin_operation (
approval_ticket, plan_digest, operation_id, state, items_completed,
lease_owner, lease_token, lease_expires_at, started_at)
VALUES (?, ?, ?, 'STARTED', 0, ?, 1, ?, ?)
ON CONFLICT (approval_ticket, plan_digest) DO NOTHING
""";
private static final String TAKE_OVER =
"""
UPDATE messaging_admin_operation
SET operation_id = ?,
state = 'STARTED',
lease_owner = ?,
lease_token = lease_token + 1,
lease_expires_at = ?,
finished_at = NULL,
failure_code = NULL
WHERE approval_ticket = ?
AND plan_digest = ?
AND lease_token = ?
-- Only a failed operation or one whose lease ran out may be taken over. A live STARTED row
-- means another replica is executing it right now.
AND (state = 'FAILED' OR lease_expires_at <= ?)
RETURNING lease_token, items_completed
""";
private static final String CHECKPOINT =
"""
UPDATE messaging_admin_operation
SET items_completed = GREATEST(items_completed, ?),
lease_expires_at = ?
WHERE approval_ticket = ? AND plan_digest = ? AND lease_token = ?
""";
private static final String SETTLE =
"""
UPDATE messaging_admin_operation
SET state = ?,
items_completed = GREATEST(items_completed, ?),
finished_at = ?,
failure_code = ?
WHERE approval_ticket = ? AND plan_digest = ? AND lease_token = ?
""";
private final DataSource dataSource;
/**
* Creates the journal.
*
* @param dataSource the shared database every replica reads
*/
public JdbcAdminOperationJournal(DataSource dataSource) {
this.dataSource = Objects.requireNonNull(dataSource, "dataSource must not be null");
}
@Override
public AdminOperationLease begin(
String approvalTicket,
PlanDigest planDigest,
String operationId,
String leaseOwner,
Duration leaseDuration,
Instant now) {
Objects.requireNonNull(planDigest, "planDigest must not be null");
Objects.requireNonNull(leaseDuration, "leaseDuration must not be null");
Objects.requireNonNull(now, "now must not be null");
Instant expiry = now.plus(leaseDuration);
Connection connection = DataSourceUtils.getConnection(dataSource);
try {
if (insertClaim(
connection, approvalTicket, planDigest, operationId, leaseOwner, expiry, now)) {
return new AdminOperationLease(approvalTicket, planDigest, operationId, leaseOwner, 1L, 0);
}
AdminOperationRecord existing =
read(connection, approvalTicket, planDigest)
.orElseThrow(
() ->
new MessagingConfigurationException(
"ADMIN_JOURNAL_INCONSISTENT",
"the insert conflicted but no row exists for approval "
+ approvalTicket));
refuseIfNotResumable(existing, approvalTicket, now);
return takeOver(
connection, existing, approvalTicket, planDigest, operationId, leaseOwner, expiry, now);
} catch (SQLException failure) {
throw new MessagingConfigurationException(
"ADMIN_JOURNAL_UNAVAILABLE", "the operation journal could not be reached", failure);
} finally {
DataSourceUtils.releaseConnection(connection, dataSource);
}
}
private static void refuseIfNotResumable(
AdminOperationRecord existing, String approvalTicket, Instant now) {
if (existing.state() == AdminOperationState.COMPLETED) {
throw new MessageAuthorizationException(
"APPROVAL_ALREADY_EXECUTED",
("approval %s was already executed to completion at %s; an approval authorises one"
+ " execution, not a standing permission")
.formatted(approvalTicket, existing.finishedAt().orElse(existing.startedAt())));
}
if (existing.state() == AdminOperationState.STARTED
&& now.isBefore(existing.leaseExpiresAt())) {
throw new MessageAuthorizationException(
"ADMIN_OPERATION_IN_FLIGHT",
("approval %s is being executed by %s until at least %s; two runtimes executing one"
+ " approval is a duplicate storm, not a faster redrive")
.formatted(approvalTicket, existing.leaseOwner(), existing.leaseExpiresAt()));
}
}
private boolean insertClaim(
Connection connection,
String approvalTicket,
PlanDigest planDigest,
String operationId,
String leaseOwner,
Instant expiry,
Instant now)
throws SQLException {
try (PreparedStatement statement = connection.prepareStatement(INSERT)) {
statement.setString(1, approvalTicket);
statement.setString(2, planDigest.value());
statement.setString(3, operationId);
statement.setString(4, leaseOwner);
statement.setTimestamp(5, Timestamp.from(expiry));
statement.setTimestamp(6, Timestamp.from(now));
return statement.executeUpdate() == 1;
}
}
private AdminOperationLease takeOver(
Connection connection,
AdminOperationRecord existing,
String approvalTicket,
PlanDigest planDigest,
String operationId,
String leaseOwner,
Instant expiry,
Instant now)
throws SQLException {
try (PreparedStatement statement = connection.prepareStatement(TAKE_OVER)) {
statement.setString(1, operationId);
statement.setString(2, leaseOwner);
statement.setTimestamp(3, Timestamp.from(expiry));
statement.setString(4, approvalTicket);
statement.setString(5, planDigest.value());
statement.setLong(6, existing.leaseToken());
statement.setTimestamp(7, Timestamp.from(now));
try (ResultSet rows = statement.executeQuery()) {
if (!rows.next()) {
// Another replica took it over between the read and this update. Refusing is correct:
// exactly one runtime executes an approval, and it is not this one.
throw new MessageAuthorizationException(
"ADMIN_OPERATION_IN_FLIGHT",
"approval " + approvalTicket + " was taken over by another runtime");
}
return new AdminOperationLease(
approvalTicket,
planDigest,
operationId,
leaseOwner,
rows.getLong("lease_token"),
rows.getInt("items_completed"));
}
}
}
@Override
public void checkpoint(
AdminOperationLease lease, int itemsCompleted, Duration leaseDuration, Instant now) {
Objects.requireNonNull(lease, "lease must not be null");
Connection connection = DataSourceUtils.getConnection(dataSource);
try (PreparedStatement statement = connection.prepareStatement(CHECKPOINT)) {
statement.setInt(1, itemsCompleted);
statement.setTimestamp(2, Timestamp.from(now.plus(leaseDuration)));
statement.setString(3, lease.approvalTicket());
statement.setString(4, lease.planDigest().value());
statement.setLong(5, lease.leaseToken());
requireOwned(statement.executeUpdate(), lease);
} catch (SQLException failure) {
throw new MessagingConfigurationException(
"ADMIN_JOURNAL_UNAVAILABLE", "the operation journal could not be reached", failure);
} finally {
DataSourceUtils.releaseConnection(connection, dataSource);
}
}
@Override
public void complete(AdminOperationLease lease, int itemsCompleted, Instant now) {
settle(lease, AdminOperationState.COMPLETED, itemsCompleted, null, now);
}
@Override
public void fail(AdminOperationLease lease, int itemsCompleted, String failureCode, Instant now) {
Objects.requireNonNull(failureCode, "failureCode must not be null");
settle(lease, AdminOperationState.FAILED, itemsCompleted, failureCode, now);
}
private void settle(
AdminOperationLease lease,
AdminOperationState state,
int itemsCompleted,
String failureCode,
Instant now) {
Objects.requireNonNull(lease, "lease must not be null");
Connection connection = DataSourceUtils.getConnection(dataSource);
try (PreparedStatement statement = connection.prepareStatement(SETTLE)) {
statement.setString(1, state.name());
statement.setInt(2, itemsCompleted);
statement.setTimestamp(3, Timestamp.from(now));
statement.setString(4, failureCode);
statement.setString(5, lease.approvalTicket());
statement.setString(6, lease.planDigest().value());
statement.setLong(7, lease.leaseToken());
requireOwned(statement.executeUpdate(), lease);
} catch (SQLException failure) {
throw new MessagingConfigurationException(
"ADMIN_JOURNAL_UNAVAILABLE", "the operation journal could not be reached", failure);
} finally {
DataSourceUtils.releaseConnection(connection, dataSource);
}
}
private static void requireOwned(int updated, AdminOperationLease lease) {
if (updated == 1) {
return;
}
throw new MessageAuthorizationException(
"ADMIN_OPERATION_LEASE_LOST",
("this runtime no longer holds the lease on approval %s at token %d; another runtime took"
+ " the operation over and its progress must not be overwritten")
.formatted(lease.approvalTicket(), lease.leaseToken()));
}
@Override
public Optional<AdminOperationRecord> find(String approvalTicket, PlanDigest planDigest) {
Connection connection = DataSourceUtils.getConnection(dataSource);
try {
return read(connection, approvalTicket, planDigest);
} catch (SQLException failure) {
throw new MessagingConfigurationException(
"ADMIN_JOURNAL_UNAVAILABLE", "the operation journal could not be reached", failure);
} finally {
DataSourceUtils.releaseConnection(connection, dataSource);
}
}
private Optional<AdminOperationRecord> read(
Connection connection, String approvalTicket, PlanDigest planDigest) throws SQLException {
try (PreparedStatement statement = connection.prepareStatement(SELECT)) {
statement.setString(1, approvalTicket);
statement.setString(2, planDigest.value());
try (ResultSet rows = statement.executeQuery()) {
if (!rows.next()) {
return Optional.empty();
}
return Optional.of(
new AdminOperationRecord(
rows.getString("approval_ticket"),
new PlanDigest(rows.getString("plan_digest")),
rows.getString("operation_id"),
AdminOperationState.valueOf(rows.getString("state")),
rows.getInt("items_completed"),
rows.getString("lease_owner"),
rows.getLong("lease_token"),
rows.getTimestamp("lease_expires_at").toInstant(),
rows.getTimestamp("started_at").toInstant(),
Optional.ofNullable(rows.getTimestamp("finished_at")).map(Timestamp::toInstant),
Optional.ofNullable(rows.getString("failure_code"))));
}
}
}
@Override
public boolean isDurable() {
return true;
}
}
@@ -0,0 +1,648 @@
package dev.caskeleton.messaging.outbox;
import dev.caskeleton.messaging.api.ContentType;
import dev.caskeleton.messaging.api.MessageId;
import dev.caskeleton.messaging.api.MessageType;
import dev.caskeleton.messaging.api.SchemaVersion;
import dev.caskeleton.messaging.api.destination.DestinationName;
import dev.caskeleton.messaging.api.error.MessagingConfigurationException;
import dev.caskeleton.messaging.reliability.OutboxLease;
import dev.caskeleton.messaging.reliability.OutboxRecord;
import dev.caskeleton.messaging.reliability.OutboxRepository;
import dev.caskeleton.messaging.reliability.OutboxStatus;
import dev.caskeleton.messaging.reliability.OutboxTransitionResult;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Timestamp;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import javax.sql.DataSource;
import org.springframework.jdbc.datasource.DataSourceUtils;
import org.springframework.transaction.support.TransactionSynchronizationManager;
/**
* The PostgreSQL outbox.
*
* <p>{@link #append} deliberately takes no connection of its own: it uses the one the caller is
* already inside, which is the entire mechanism. An outbox row written on a separate connection
* commits independently of the business change and reopens the window the pattern exists to close.
*
* <p>Leasing uses {@code SELECT ... FOR UPDATE SKIP LOCKED}. Two relay instances polling the same
* table would otherwise both claim the same rows and publish them twice; {@code SKIP LOCKED} lets
* the second instance step over what the first already holds instead of blocking behind it.
*/
public final class JdbcOutboxRepository implements OutboxRepository {
private static final String INSERT =
"""
INSERT INTO messaging_outbox (
message_id, destination, message_type, schema_version, content_type,
payload, headers, created_at, status, attempts)
VALUES (?, ?, ?, ?, ?, ?, ?::jsonb, ?, ?, ?)
""";
private static final String LEASE =
"""
WITH claimable AS (
SELECT message_id
FROM messaging_outbox
-- IN_FLIGHT is claimable too, but only once its lease has expired. A relay that dies
-- mid-publish leaves rows in that state; excluding them would strand those messages
-- forever, which is the one failure mode the outbox exists to prevent.
WHERE status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT')
AND (lease_expires_at IS NULL OR lease_expires_at <= ?)
ORDER BY created_at
LIMIT ?
FOR UPDATE SKIP LOCKED
)
UPDATE messaging_outbox o
SET status = 'IN_FLIGHT', lease_expires_at = ?
FROM claimable c
WHERE o.message_id = c.message_id
RETURNING o.message_id, o.destination, o.message_type, o.schema_version, o.content_type,
o.payload, o.headers, o.created_at, o.status, o.attempts,
o.lease_expires_at, o.last_failure_code
""";
/**
* The claim that records who holds the row and increments its fencing token.
*
* <p>The token increments on the server inside the same statement that grants the claim, so two
* relays racing for the same row cannot receive the same number.
*/
private static final String CLAIM =
"""
WITH claimable AS (
SELECT message_id
FROM messaging_outbox
WHERE status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT')
AND (lease_expires_at IS NULL OR lease_expires_at <= ?)
-- The retry clock lives in the row, not in the relay's memory. Without these two
-- predicates an AMBIGUOUS row became claimable again on the very next pass, so a
-- broker outage meant the whole backlog was republished every poll interval and the
-- configured attempt budget was a number nothing consulted.
AND (next_attempt_at IS NULL OR next_attempt_at <= ?)
AND attempts < ?
ORDER BY created_at
LIMIT ?
FOR UPDATE SKIP LOCKED
)
UPDATE messaging_outbox o
SET status = 'IN_FLIGHT',
lease_expires_at = ?,
lease_owner = ?,
lease_token = o.lease_token + 1
FROM claimable c
WHERE o.message_id = c.message_id
RETURNING o.message_id, o.destination, o.message_type, o.schema_version, o.content_type,
o.payload, o.headers, o.created_at, o.status, o.attempts,
o.lease_expires_at, o.last_failure_code, o.lease_owner, o.lease_token
""";
private final DataSource dataSource;
/**
* Creates a repository over a data source.
*
* @param dataSource the outbox data source
*/
public JdbcOutboxRepository(DataSource dataSource) {
this.dataSource = Objects.requireNonNull(dataSource, "dataSource must not be null");
}
/**
* Appends a record on an existing connection.
*
* <p>Package-private. It used to be the public overload "business code should use", which meant
* the safe path was the one a caller had to know about and the interface method — the one the
* port documents — opened its own connection. Handing a {@code Connection} across a reliability
* API also makes the transaction the caller's problem to remember; {@link #append(OutboxRecord)}
* now joins the caller's transaction itself and refuses when there is none.
*
* @param connection the caller's transactional connection
* @param record the record to write
*/
void append(Connection connection, OutboxRecord record) {
Objects.requireNonNull(connection, "connection must not be null");
Objects.requireNonNull(record, "record must not be null");
try (PreparedStatement statement = connection.prepareStatement(INSERT)) {
statement.setObject(1, record.messageId().value());
statement.setString(2, record.destination().value());
statement.setString(3, record.messageType().value());
statement.setInt(4, record.schemaVersion().value());
statement.setString(5, record.contentType().value());
statement.setBytes(6, record.payload());
statement.setString(7, toJson(record.headers()));
statement.setTimestamp(8, Timestamp.from(record.createdAt()));
statement.setString(9, record.status().name());
statement.setInt(10, record.attempts());
statement.executeUpdate();
} catch (SQLException exception) {
throw new MessagingConfigurationException(
"OUTBOX_APPEND_FAILED", "could not append an outbox record", exception);
}
}
@Override
public void append(OutboxRecord record) {
Objects.requireNonNull(record, "record must not be null");
// The port says this runs inside the caller's business transaction. It did not: it opened a raw
// connection from the pool, which auto-commits independently — so a business transaction that
// rolled back afterwards left the event behind and the relay published a change that never
// happened.
requireActiveTransaction("OUTBOX_TRANSACTION_REQUIRED", "appending to the outbox");
Connection connection = DataSourceUtils.getConnection(dataSource);
try {
append(connection, record);
} finally {
// Releases only when the connection is not the transaction's; when it is, this is a no-op and
// the caller's commit or rollback decides what happens to the row.
DataSourceUtils.releaseConnection(connection, dataSource);
}
}
/**
* Refuses when there is no transaction to join, or when it cannot write.
*
* <p>Fail-fast rather than "work anyway": an append that silently runs outside the caller's
* transaction produces exactly the ghost publication this repository exists to prevent, and it
* produces it only on the rollback path — which is the path nobody exercises before production.
*/
private void requireActiveTransaction(String code, String what) {
if (!TransactionSynchronizationManager.isActualTransactionActive()) {
throw new MessagingConfigurationException(
code,
what
+ " requires an active transaction; without one the row commits on its own and a"
+ " later rollback of the business change leaves it behind");
}
if (TransactionSynchronizationManager.isCurrentTransactionReadOnly()) {
throw new MessagingConfigurationException(
code, what + " requires a writable transaction; the current one is read-only");
}
if (!TransactionSynchronizationManager.hasResource(dataSource)) {
throw new MessagingConfigurationException(
code,
what
+ " requires the active transaction to be bound to this DataSource; it is bound to"
+ " another, so the two would commit independently");
}
}
@Override
public List<OutboxRecord> leaseBatch(int batchSize, Duration leaseDuration, Instant now) {
Objects.requireNonNull(leaseDuration, "leaseDuration must not be null");
Objects.requireNonNull(now, "now must not be null");
return withConnection(
connection -> {
List<OutboxRecord> leased = new ArrayList<>();
try (PreparedStatement statement = connection.prepareStatement(LEASE)) {
statement.setTimestamp(1, Timestamp.from(now));
statement.setInt(2, batchSize);
statement.setTimestamp(3, Timestamp.from(now.plus(leaseDuration)));
try (ResultSet results = statement.executeQuery()) {
while (results.next()) {
leased.add(read(results));
}
}
}
return List.copyOf(leased);
});
}
@Override
public List<OutboxLease> claimBatch(
String owner, int batchSize, Duration leaseDuration, Instant now) {
return claimBatch(owner, batchSize, leaseDuration, now, Integer.MAX_VALUE);
}
@Override
public List<OutboxLease> claimBatch(
String owner, int batchSize, Duration leaseDuration, Instant now, int maxAttempts) {
Objects.requireNonNull(owner, "owner must not be null");
Objects.requireNonNull(leaseDuration, "leaseDuration must not be null");
Objects.requireNonNull(now, "now must not be null");
Instant expiresAt = now.plus(leaseDuration);
return withConnection(
connection -> {
List<OutboxLease> claimed = new ArrayList<>();
try (PreparedStatement statement = connection.prepareStatement(CLAIM)) {
statement.setTimestamp(1, Timestamp.from(now));
statement.setTimestamp(2, Timestamp.from(now));
statement.setInt(3, maxAttempts);
statement.setInt(4, batchSize);
statement.setTimestamp(5, Timestamp.from(expiresAt));
statement.setString(6, owner);
try (ResultSet results = statement.executeQuery()) {
while (results.next()) {
claimed.add(
new OutboxLease(
read(results),
results.getString("lease_owner"),
results.getLong("lease_token"),
expiresAt));
}
}
}
return List.copyOf(claimed);
});
}
@Override
public OutboxTransitionResult markPublished(OutboxLease lease, Instant now) {
return transition(
lease,
"UPDATE messaging_outbox SET status = 'PUBLISHED', published_at = ?, "
+ "lease_expires_at = NULL, lease_owner = NULL, next_attempt_at = NULL, "
+ "attempts = attempts + 1 ",
now,
null);
}
@Override
public OutboxTransitionResult markAmbiguous(OutboxLease lease, String failureCode, Instant now) {
return markAmbiguous(lease, failureCode, now, now);
}
@Override
public OutboxTransitionResult markAmbiguous(
OutboxLease lease, String failureCode, Instant now, Instant nextAttemptAt) {
Objects.requireNonNull(nextAttemptAt, "nextAttemptAt must not be null");
// The backoff the policy computed is written onto the row. Keeping it in the relay's memory
// means a restart forgets it, and every relay computes its own — so the same row is retried by
// as many workers as there are replicas, as fast as they poll.
return transitionWithNextAttempt(
lease,
"UPDATE messaging_outbox SET status = 'AMBIGUOUS', lease_expires_at = NULL, "
+ "lease_owner = NULL, last_failure_code = ?, attempts = attempts + 1, "
+ "next_attempt_at = ? ",
failureCode,
nextAttemptAt);
}
@Override
public OutboxTransitionResult markExhausted(OutboxLease lease, String reason, Instant now) {
return transition(
lease,
"UPDATE messaging_outbox SET status = 'EXHAUSTED', lease_expires_at = NULL, "
+ "lease_owner = NULL, last_failure_code = ?, attempts = attempts + 1 ",
now,
reason);
}
private OutboxTransitionResult transitionWithNextAttempt(
OutboxLease lease, String setClause, String failureCode, Instant nextAttemptAt) {
Objects.requireNonNull(lease, "lease must not be null");
String sql =
setClause
+ "WHERE message_id = ? AND status = 'IN_FLIGHT' AND lease_owner = ? AND lease_token = ?";
return withConnection(
connection -> {
try (PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setString(1, failureCode);
statement.setTimestamp(2, Timestamp.from(nextAttemptAt));
statement.setObject(3, lease.messageId().value());
statement.setString(4, lease.owner());
statement.setLong(5, lease.token());
return statement.executeUpdate() == 1
? OutboxTransitionResult.APPLIED
: OutboxTransitionResult.STALE_LEASE;
}
});
}
@Override
public OutboxTransitionResult markFailed(OutboxLease lease, String failureCode, Instant now) {
return transition(
lease,
"UPDATE messaging_outbox SET status = 'FAILED', lease_expires_at = NULL, "
+ "lease_owner = NULL, last_failure_code = ?, attempts = attempts + 1 ",
now,
failureCode);
}
@Override
public OutboxTransitionResult releaseLease(OutboxLease lease) {
return transition(
lease,
"UPDATE messaging_outbox SET status = 'PENDING', lease_expires_at = NULL, "
+ "lease_owner = NULL ",
null,
null);
}
/**
* Applies a terminal write, but only to the row this lease still owns.
*
* <p>The predicate carries the owner and the token as well as the id, so a relay that stalled
* past its lease writes nothing: another relay's claim incremented the token, and this update
* matches zero rows. Zero is reported rather than swallowed — a stale write means this worker may
* have produced a duplicate publication, which is exactly what an operator needs to see.
*/
private OutboxTransitionResult transition(
OutboxLease lease, String setClause, Instant now, String failureCode) {
Objects.requireNonNull(lease, "lease must not be null");
String sql =
setClause
+ "WHERE message_id = ? AND status = 'IN_FLIGHT' AND lease_owner = ? AND lease_token = ?";
return withConnection(
connection -> {
try (PreparedStatement statement = connection.prepareStatement(sql)) {
int index = 1;
if (failureCode != null) {
statement.setString(index++, failureCode);
} else if (now != null) {
statement.setTimestamp(index++, Timestamp.from(now));
}
statement.setObject(index++, lease.messageId().value());
statement.setString(index++, lease.owner());
statement.setLong(index, lease.token());
return statement.executeUpdate() == 1
? OutboxTransitionResult.APPLIED
: OutboxTransitionResult.STALE_LEASE;
}
});
}
@Override
public void markPublished(MessageId messageId, Instant now) {
update(
"UPDATE messaging_outbox SET status = 'PUBLISHED', published_at = ?, "
+ "lease_expires_at = NULL, attempts = attempts + 1 WHERE message_id = ?",
messageId,
now,
null);
}
@Override
public void markAmbiguous(MessageId messageId, String failureCode, Instant now) {
// The lease is released rather than held: an ambiguous row must become claimable again so the
// relay retries it under the same message id.
update(
"UPDATE messaging_outbox SET status = 'AMBIGUOUS', lease_expires_at = NULL, "
+ "last_failure_code = ?, attempts = attempts + 1 WHERE message_id = ?",
messageId,
now,
failureCode);
}
@Override
public void markFailed(MessageId messageId, String failureCode, Instant now) {
update(
"UPDATE messaging_outbox SET status = 'FAILED', lease_expires_at = NULL, "
+ "last_failure_code = ?, attempts = attempts + 1 WHERE message_id = ?",
messageId,
now,
failureCode);
}
@Override
public void releaseLease(MessageId messageId) {
withConnection(
connection -> {
try (PreparedStatement statement =
connection.prepareStatement(
"UPDATE messaging_outbox SET status = 'PENDING', lease_expires_at = NULL "
+ "WHERE message_id = ? AND status = 'IN_FLIGHT'")) {
statement.setObject(1, messageId.value());
statement.executeUpdate();
}
return null;
});
}
@Override
public Optional<OutboxRecord> find(MessageId messageId) {
return withConnection(
connection -> {
try (PreparedStatement statement =
connection.prepareStatement(
"SELECT message_id, destination, message_type, schema_version, content_type, "
+ "payload, headers, created_at, status, attempts, lease_expires_at, "
+ "last_failure_code FROM messaging_outbox WHERE message_id = ?")) {
statement.setObject(1, messageId.value());
try (ResultSet results = statement.executeQuery()) {
return results.next() ? Optional.of(read(results)) : Optional.<OutboxRecord>empty();
}
}
});
}
@Override
public int purgePublishedBefore(Instant publishedBefore, int limit) {
Objects.requireNonNull(publishedBefore, "publishedBefore must not be null");
if (limit < 1) {
throw new IllegalArgumentException("a bounded purge deletes at least one row per call");
}
// The CTE picks a bounded set of ids with SKIP LOCKED and deletes exactly those. An unbounded
// DELETE holds locks and writes WAL in proportion to the whole backlog, which stalls the relay
// and the business writes behind retention.
return withConnection(
connection -> {
try (PreparedStatement statement =
connection.prepareStatement(
"""
WITH expired AS (
SELECT message_id
FROM messaging_outbox
WHERE status = 'PUBLISHED' AND published_at < ?
ORDER BY published_at
LIMIT ?
FOR UPDATE SKIP LOCKED
)
DELETE FROM messaging_outbox o
USING expired e
WHERE o.message_id = e.message_id
""")) {
statement.setTimestamp(1, Timestamp.from(publishedBefore));
statement.setInt(2, limit);
return statement.executeUpdate();
}
});
}
@Override
public int purgePublishedBefore(Instant publishedBefore) {
Objects.requireNonNull(publishedBefore, "publishedBefore must not be null");
return withConnection(
connection -> {
try (PreparedStatement statement =
connection.prepareStatement(
"DELETE FROM messaging_outbox WHERE status = 'PUBLISHED' AND published_at < ?")) {
statement.setTimestamp(1, Timestamp.from(publishedBefore));
return statement.executeUpdate();
}
});
}
private void update(String sql, MessageId messageId, Instant now, String failureCode) {
withConnection(
connection -> {
try (PreparedStatement statement = connection.prepareStatement(sql)) {
if (failureCode == null) {
statement.setTimestamp(1, Timestamp.from(now));
} else {
statement.setString(1, failureCode);
}
statement.setObject(2, messageId.value());
statement.executeUpdate();
}
return null;
});
}
private static OutboxRecord read(ResultSet results) throws SQLException {
Timestamp lease = results.getTimestamp("lease_expires_at");
return new OutboxRecord(
new MessageId(results.getObject("message_id", java.util.UUID.class)),
new DestinationName(results.getString("destination")),
new MessageType(results.getString("message_type")),
new SchemaVersion(results.getInt("schema_version")),
new ContentType(results.getString("content_type")),
results.getBytes("payload"),
fromJson(results.getString("headers")),
results.getTimestamp("created_at").toInstant(),
OutboxStatus.valueOf(results.getString("status")),
results.getInt("attempts"),
Optional.ofNullable(lease).map(Timestamp::toInstant),
Optional.ofNullable(results.getString("last_failure_code")));
}
private <T> T withConnection(ConnectionCallback<T> callback) {
try (Connection connection = dataSource.getConnection()) {
return callback.apply(connection);
} catch (SQLException exception) {
throw new MessagingConfigurationException(
"OUTBOX_QUERY_FAILED", "an outbox query failed", exception);
}
}
/**
* Serialises headers as a flat JSON object.
*
* <p>Hand-rolled rather than pulled from a JSON library so this module keeps no codec dependency:
* outbox headers are always flat string pairs, validated by {@code MessageHeaders} before they
* ever reach here.
*/
private static String toJson(Map<String, String> headers) {
StringBuilder json = new StringBuilder("{");
boolean first = true;
for (Map.Entry<String, String> entry : headers.entrySet()) {
if (!first) {
json.append(',');
}
first = false;
json.append('"')
.append(escape(entry.getKey()))
.append("\":\"")
.append(escape(entry.getValue()))
.append('"');
}
return json.append('}').toString();
}
private static Map<String, String> fromJson(String json) {
Map<String, String> headers = new LinkedHashMap<>();
if (json == null || json.length() <= 2) {
return headers;
}
String body = json.substring(1, json.length() - 1);
int index = 0;
while (index < body.length()) {
int keyStart = body.indexOf('"', index);
if (keyStart < 0) {
break;
}
int keyEnd = findClosingQuote(body, keyStart + 1);
int valueStart = body.indexOf('"', body.indexOf(':', keyEnd));
int valueEnd = findClosingQuote(body, valueStart + 1);
headers.put(
unescape(body.substring(keyStart + 1, keyEnd)),
unescape(body.substring(valueStart + 1, valueEnd)));
index = valueEnd + 1;
}
return headers;
}
private static int findClosingQuote(String text, int from) {
for (int index = from; index < text.length(); index++) {
if (text.charAt(index) == '"' && text.charAt(index - 1) != '\\') {
return index;
}
}
return text.length();
}
/**
* JSON string escaping, including the control characters.
*
* <p>Only backslash and quote were escaped. A header value containing a newline or a NUL — which
* the value objects now refuse, but which older rows and other writers may still carry — produced
* a document PostgreSQL rejects as invalid JSONB, so the append failed at the database with an
* error naming neither the header nor the row. Control characters are the part of the JSON
* grammar that is easy to forget and impossible to get away with.
*/
private static String escape(String value) {
StringBuilder escaped = new StringBuilder(value.length() + 8);
for (int index = 0; index < value.length(); index++) {
char character = value.charAt(index);
switch (character) {
case '\\' -> escaped.append("\\\\");
case '"' -> escaped.append("\\\"");
case '\n' -> escaped.append("\\n");
case '\r' -> escaped.append("\\r");
case '\t' -> escaped.append("\\t");
case '\b' -> escaped.append("\\b");
case '\f' -> escaped.append("\\f");
default -> {
if (character < 0x20) {
escaped.append("\\u%04x".formatted((int) character));
} else {
escaped.append(character);
}
}
}
}
return escaped.toString();
}
private static String unescape(String value) {
StringBuilder plain = new StringBuilder(value.length());
for (int index = 0; index < value.length(); index++) {
char character = value.charAt(index);
if (character != '\\' || index == value.length() - 1) {
plain.append(character);
continue;
}
char next = value.charAt(++index);
switch (next) {
case 'n' -> plain.append('\n');
case 'r' -> plain.append('\r');
case 't' -> plain.append('\t');
case 'b' -> plain.append('\b');
case 'f' -> plain.append('\f');
case 'u' -> {
plain.append((char) Integer.parseInt(value.substring(index + 1, index + 5), 16));
index += 4;
}
default -> plain.append(next);
}
}
return plain.toString();
}
/** A unit of work against a borrowed connection. */
@FunctionalInterface
private interface ConnectionCallback<T> {
T apply(Connection connection) throws SQLException;
}
}
@@ -46,7 +46,30 @@ public final class OutboxEnvelopeFactory {
Map<HeaderName, HeaderValue> headers = new LinkedHashMap<>();
record
.headers()
.forEach((name, value) -> headers.put(new HeaderName(name), new HeaderValue(value)));
.forEach(
(name, value) -> {
// The row's headers are application data. Promoting them straight into
// MessageHeaders.platform let a row carrying `msg.id` or `msg.type` reach the wire
// as if the platform had written it — and the consumer trusts the last such header,
// so a forged identity lands in another message's inbox deduplication.
//
// Partition and ordering keys are the exception: they are reserved names the outbox
// legitimately carries as routing metadata, and they are read below rather than
// written as headers.
if (ReservedHeaders.isReserved(name)
&& !ReservedHeaders.PARTITION_KEY.equals(name)
&& !ReservedHeaders.ORDERING_KEY.equals(name)) {
throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException(
"RESERVED_HEADER_IN_OUTBOX_ROW",
"outbox row header '"
+ name
+ "' uses the reserved namespace; canonical identity and metadata come from"
+ " the record's own columns, not from its header map");
}
if (!ReservedHeaders.isReserved(name)) {
headers.put(new HeaderName(name), new HeaderValue(value));
}
});
return new MessageEnvelope<>(
record.messageId(),
@@ -4,8 +4,10 @@ import dev.caskeleton.messaging.api.destination.MessageDestination;
import dev.caskeleton.messaging.api.publish.MessagePublisher;
import dev.caskeleton.messaging.api.publish.PublishOptions;
import dev.caskeleton.messaging.api.publish.PublishResult;
import dev.caskeleton.messaging.reliability.OutboxLease;
import dev.caskeleton.messaging.reliability.OutboxRecord;
import dev.caskeleton.messaging.reliability.OutboxRepository;
import dev.caskeleton.messaging.reliability.OutboxTransitionResult;
import dev.caskeleton.messaging.schema.EncodedMessage;
import java.time.Duration;
import java.time.Instant;
@@ -30,6 +32,17 @@ public final class OutboxRelay {
private final MessagePublisher publisher;
private final OutboxEnvelopeFactory envelopeFactory;
private final int batchSize;
private final String owner;
private final OutboxRetryScheduler scheduler;
private static String defaultOwner() {
return java.lang.ProcessHandle.current().pid()
+ "@"
+ java.util.UUID.randomUUID().toString().substring(0, 8);
}
private final Duration leaseDuration;
/**
@@ -47,6 +60,50 @@ public final class OutboxRelay {
OutboxEnvelopeFactory envelopeFactory,
int batchSize,
Duration leaseDuration) {
this(repository, publisher, envelopeFactory, batchSize, leaseDuration, defaultOwner());
}
/**
* A relay with an explicit owner identity.
*
* <p>The owner is half of the fencing predicate. It defaults to something stable per process
* rather than per object, because two relays in one JVM sharing an owner would each be able to
* overwrite the other's terminal writes — the exact failure the token exists to prevent.
*/
public OutboxRelay(
OutboxRepository repository,
MessagePublisher publisher,
OutboxEnvelopeFactory envelopeFactory,
int batchSize,
Duration leaseDuration,
String owner) {
this(
repository,
publisher,
envelopeFactory,
batchSize,
leaseDuration,
owner,
OutboxRetryScheduler.standard());
}
/**
* A relay with an explicit retry policy.
*
* <p>The scheduler was built by the auto-configuration and handed to nobody: the relay claimed
* every ambiguous row on the next pass regardless of its attempt count, so the configured budget
* of ten attempts was a number nothing read.
*/
public OutboxRelay(
OutboxRepository repository,
MessagePublisher publisher,
OutboxEnvelopeFactory envelopeFactory,
int batchSize,
Duration leaseDuration,
String owner,
OutboxRetryScheduler scheduler) {
this.owner = Objects.requireNonNull(owner, "owner must not be null");
this.scheduler = Objects.requireNonNull(scheduler, "scheduler must not be null");
this.repository = Objects.requireNonNull(repository, "repository must not be null");
this.publisher = Objects.requireNonNull(publisher, "publisher must not be null");
this.envelopeFactory =
@@ -66,32 +123,46 @@ public final class OutboxRelay {
*/
public OutboxRelayReport runOnce(Instant now) {
Objects.requireNonNull(now, "now must not be null");
List<OutboxRecord> batch = repository.leaseBatch(batchSize, leaseDuration, now);
// claimBatch, not leaseBatch: the claim carries an owner and a fencing token, and every write
// below is conditioned on them. Without that, a relay that stalled past its lease could record
// AMBIGUOUS over the PUBLISHED another relay had already written, making a confirmed message
// claimable again.
List<OutboxLease> batch =
repository.claimBatch(owner, batchSize, leaseDuration, now, scheduler.maxAttempts());
int published = 0;
int ambiguous = 0;
int failed = 0;
int stale = 0;
for (OutboxRecord record : batch) {
for (OutboxLease lease : batch) {
OutboxRecord record = lease.record();
PublishResult result = publish(record);
switch (result.completion()) {
case CONFIRMED -> {
repository.markPublished(record.messageId(), now);
published++;
if (repository.markPublished(lease, now) == OutboxTransitionResult.APPLIED) {
published++;
} else {
stale++;
}
}
case AMBIGUOUS -> {
repository.markAmbiguous(
record.messageId(),
result.failure().map(failure -> failure.code()).orElse("AMBIGUOUS"),
now);
ambiguous++;
if (repository.markAmbiguous(
lease, result.failure().map(failure -> failure.code()).orElse("AMBIGUOUS"), now)
== OutboxTransitionResult.APPLIED) {
ambiguous++;
} else {
stale++;
}
}
case REJECTED -> {
repository.markFailed(
record.messageId(),
result.failure().map(failure -> failure.code()).orElse("REJECTED"),
now);
failed++;
if (repository.markFailed(
lease, result.failure().map(failure -> failure.code()).orElse("REJECTED"), now)
== OutboxTransitionResult.APPLIED) {
failed++;
} else {
stale++;
}
}
// Unreachable while PublishCompletion has exactly these constants. Present so that a new
// completion fails loudly rather than leaving the row IN_FLIGHT with its lease ticking —
@@ -100,7 +171,7 @@ public final class OutboxRelay {
throw new IllegalStateException("unhandled publish completion: " + result.completion());
}
}
return new OutboxRelayReport(batch.size(), published, ambiguous, failed);
return new OutboxRelayReport(batch.size(), published, ambiguous, failed, stale);
}
private PublishResult publish(OutboxRecord record) {
@@ -0,0 +1,34 @@
package dev.caskeleton.messaging.outbox;
/**
* What one relay pass did.
*
* <p>Ambiguous rows are counted separately from failures because they mean something operationally
* different: a rising ambiguous count is a broker confirmation problem, while a rising failed count
* is a contract or topology problem.
*
* <p>{@code staleLeases} is the count of terminal writes this pass could not apply because another
* relay had already taken the row. It is not an error and it is not a success: it means this worker
* published something whose outcome another worker has already recorded, so a non-zero value is the
* visible form of a duplicate publication.
*
* @param leased how many rows were claimed
* @param published how many were confirmed
* @param ambiguous how many had an unknown outcome and stay retryable
* @param failed how many were definitively rejected
* @param staleLeases how many terminal writes were refused because the lease had been superseded
*/
public record OutboxRelayReport(
int leased, int published, int ambiguous, int failed, int staleLeases) {
public OutboxRelayReport {
if (leased < 0 || published < 0 || ambiguous < 0 || failed < 0 || staleLeases < 0) {
throw new IllegalArgumentException("relay counters must not be negative");
}
}
/** A report from a pass that observed no stale leases. */
public OutboxRelayReport(int leased, int published, int ambiguous, int failed) {
this(leased, published, ambiguous, failed, 0);
}
}
@@ -58,6 +58,16 @@ public final class OutboxRetryScheduler {
return now.plus(backoff(consecutiveEmptyOrFailedPasses));
}
/** The platform's default policy, for a relay whose caller did not supply one. */
public static OutboxRetryScheduler standard() {
return new OutboxRetryScheduler(OutboxProperties.defaults(), java.time.Duration.ofMinutes(5));
}
/** The attempt budget a row may spend before it is parked. */
public int maxAttempts() {
return maxAttempts;
}
/**
* Returns the delay before the next pass.
*
@@ -0,0 +1,39 @@
-- Lease owner and fencing token.
--
-- V1 recorded only lease_expires_at, so a claim said when it would end and nothing about who held
-- it. The relay's terminal writes matched on message_id alone, which means a worker that stalled
-- past its lease could still write:
--
-- relay A claims the row and calls the broker
-- the lease expires; relay B reclaims it, publishes, and records PUBLISHED
-- relay A finally times out and records AMBIGUOUS over the top
--
-- The row is now claimable again and the message is published a second time. Making the lease
-- longer than the publish timeout lowers the odds; it does not turn a GC pause, a scheduler stall
-- or a slow broker into a data constraint. A token does: every terminal write names the
-- acquisition it belongs to, and a superseded worker's write matches nothing.
ALTER TABLE messaging_outbox
ADD COLUMN lease_owner VARCHAR(160),
ADD COLUMN lease_token BIGINT NOT NULL DEFAULT 0,
ADD COLUMN next_attempt_at TIMESTAMPTZ;
-- Backfill is unnecessary for correctness — the default is 0 and the first claim increments it —
-- but the constraint states the invariant the code depends on.
ALTER TABLE messaging_outbox
ADD CONSTRAINT ck_messaging_outbox_lease_token CHECK (lease_token >= 0);
-- EXHAUSTED is a new terminal state: the attempt budget ran out without any confirmation, which is
-- not the same as the broker rejecting the message. V1's CHECK listed five states, so writing the
-- sixth failed at the constraint rather than at review.
ALTER TABLE messaging_outbox
DROP CONSTRAINT ck_messaging_outbox_status;
ALTER TABLE messaging_outbox
ADD CONSTRAINT ck_messaging_outbox_status
CHECK (status IN ('PENDING', 'IN_FLIGHT', 'PUBLISHED', 'AMBIGUOUS', 'FAILED', 'EXHAUSTED'));
-- The claim predicate now filters on next_attempt_at as well, so the partial index has to carry it
-- or every poll degrades into a scan of the backlog once rows start backing off.
CREATE INDEX ix_messaging_outbox_next_attempt
ON messaging_outbox (next_attempt_at, created_at)
WHERE status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT');
@@ -0,0 +1,37 @@
-- The destructive-operation journal.
--
-- Keyed on (approval_ticket, plan_digest) rather than the ticket alone: the ticket alone cannot
-- distinguish "this approval already ran" from "this approval is being presented for a different
-- plan", and those need opposite answers. The primary key is what makes two replicas racing for the
-- same approval resolve to one winner rather than two executions.
--
-- lease_owner/lease_token/lease_expires_at are the same fencing pattern the outbox uses: a runtime
-- that stalls past its lease loses it, and its later writes are refused because its token is stale.
-- items_completed is the resume point — without it a retry after a crash republishes everything the
-- first attempt already moved.
CREATE TABLE IF NOT EXISTS messaging_admin_operation (
approval_ticket VARCHAR(200) NOT NULL,
plan_digest CHAR(64) NOT NULL,
operation_id VARCHAR(200) NOT NULL,
state VARCHAR(20) NOT NULL,
items_completed INTEGER NOT NULL DEFAULT 0,
lease_owner VARCHAR(200) NOT NULL,
lease_token BIGINT NOT NULL DEFAULT 1,
lease_expires_at TIMESTAMPTZ NOT NULL,
started_at TIMESTAMPTZ NOT NULL,
finished_at TIMESTAMPTZ,
failure_code VARCHAR(200),
CONSTRAINT pk_messaging_admin_operation PRIMARY KEY (approval_ticket, plan_digest),
CONSTRAINT ck_messaging_admin_operation_state
CHECK (state IN ('STARTED', 'COMPLETED', 'FAILED')),
CONSTRAINT ck_messaging_admin_operation_items CHECK (items_completed >= 0),
CONSTRAINT ck_messaging_admin_operation_token CHECK (lease_token >= 1),
CONSTRAINT ck_messaging_admin_operation_digest CHECK (plan_digest ~ '^[0-9a-f]{64}$')
);
-- Answers "which operations are in flight right now", the question an operator asks when a redrive
-- appears stuck and they are deciding whether to take it over.
CREATE INDEX IF NOT EXISTS ix_messaging_admin_operation_live
ON messaging_admin_operation (lease_expires_at)
WHERE state = 'STARTED';
@@ -0,0 +1,206 @@
package dev.caskeleton.messaging.outbox;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import dev.caskeleton.messaging.admin.AdminOperationLease;
import dev.caskeleton.messaging.admin.AdminOperationState;
import dev.caskeleton.messaging.admin.PlanDigest;
import dev.caskeleton.messaging.api.error.MessageAuthorizationException;
import dev.caskeleton.messaging.testkit.DockerAvailability;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.sql.Connection;
import java.sql.SQLException;
import java.sql.Statement;
import java.time.Duration;
import java.time.Instant;
import javax.sql.DataSource;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.EnabledIf;
import org.postgresql.ds.PGSimpleDataSource;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.postgresql.PostgreSQLContainer;
/**
* Certifies the durable journal against a real PostgreSQL.
*
* <p>The properties that matter here belong to the database, not to Java. "Exactly one replica
* executes an approval" is a primary-key conflict; "a stalled runtime cannot overwrite its
* successor" is a conditional update on a fencing token. An in-memory double can be written to
* agree with both without either being true of the deployed schema, which is exactly how the map
* this replaced looked correct for as long as nobody restarted a replica.
*/
@Testcontainers
@EnabledIf("dockerAvailable")
class AdminOperationJournalPostgresIT {
private static final Instant NOW = Instant.parse("2026-08-10T09:15:00Z");
private static final Duration LEASE = Duration.ofMinutes(5);
private static final String TICKET = "CHG-1042";
private static final PlanDigest DIGEST = PlanDigest.ofCanonical("REDRIVE|orders.v1.dlq|50");
@Container
private static final PostgreSQLContainer POSTGRES =
new PostgreSQLContainer("postgres:16-alpine")
.withDatabaseName("messaging")
.withUsername("messaging")
.withPassword("messaging");
private JdbcAdminOperationJournal journal;
static boolean dockerAvailable() {
return DockerAvailability.isAvailable();
}
@BeforeEach
void migrate() throws SQLException, IOException {
DataSource dataSource = dataSource();
try (Connection connection = dataSource.getConnection();
Statement statement = connection.createStatement()) {
statement.execute("DROP TABLE IF EXISTS messaging_admin_operation");
statement.execute(migration("V3__messaging_admin_operation_journal.sql"));
}
journal = new JdbcAdminOperationJournal(dataSource);
}
@Test
@DisplayName("the primary key, not application code, decides which replica wins")
void thePrimaryKeyDecidesWhichReplicaWins() {
journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
assertThatThrownBy(
() -> journal.begin(TICKET, DIGEST, "op-2", "replica-b", LEASE, NOW.plusSeconds(1)))
.isInstanceOf(MessageAuthorizationException.class)
.hasMessageContaining("replica-a");
}
@Test
@DisplayName("a completed operation survives the restart that forgot the map")
void aCompletedOperationSurvivesARestart() {
AdminOperationLease lease = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
journal.complete(lease, 50, NOW.plusSeconds(30));
// A fresh journal object over the same database is what a restarted replica sees.
JdbcAdminOperationJournal afterRestart = new JdbcAdminOperationJournal(dataSource());
assertThatThrownBy(
() ->
afterRestart.begin(TICKET, DIGEST, "op-3", "replica-a", LEASE, NOW.plusSeconds(60)))
.isInstanceOf(MessageAuthorizationException.class)
.hasMessageContaining("already executed");
}
@Test
@DisplayName("an expired lease is taken over at its checkpoint, under a higher token")
void anExpiredLeaseIsTakenOverAtItsCheckpoint() {
AdminOperationLease first = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
journal.checkpoint(first, 30, LEASE, NOW.plusSeconds(10));
AdminOperationLease resumed =
journal.begin(TICKET, DIGEST, "op-1", "replica-b", LEASE, NOW.plus(Duration.ofMinutes(10)));
assertThat(resumed.resumeFrom()).isEqualTo(30);
assertThat(resumed.leaseToken()).isEqualTo(first.leaseToken() + 1);
}
@Test
@DisplayName("a live lease is not taken over")
void aLiveLeaseIsNotTakenOver() {
journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
assertThatThrownBy(
() ->
journal.begin(
TICKET, DIGEST, "op-1", "replica-b", LEASE, NOW.plus(Duration.ofMinutes(1))))
.isInstanceOf(MessageAuthorizationException.class)
.hasMessageContaining("is being executed by");
}
@Test
@DisplayName("a superseded runtime cannot write over the one that took over")
void aSupersededRuntimeCannotWriteOverItsSuccessor() {
AdminOperationLease stale = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
journal.checkpoint(stale, 30, LEASE, NOW.plusSeconds(10));
AdminOperationLease successor =
journal.begin(TICKET, DIGEST, "op-1", "replica-b", LEASE, NOW.plus(Duration.ofMinutes(10)));
journal.checkpoint(successor, 45, LEASE, NOW.plus(Duration.ofMinutes(11)));
assertThatThrownBy(() -> journal.complete(stale, 30, NOW.plus(Duration.ofMinutes(12))))
.isInstanceOf(MessageAuthorizationException.class)
.hasMessageContaining("no longer holds the lease");
assertThat(journal.find(TICKET, DIGEST))
.hasValueSatisfying(
record -> {
assertThat(record.itemsCompleted()).isEqualTo(45);
assertThat(record.state()).isEqualTo(AdminOperationState.STARTED);
});
}
@Test
@DisplayName("a failed operation is resumable immediately and records why it stopped")
void aFailedOperationIsResumableImmediately() {
AdminOperationLease first = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
journal.fail(first, 12, "MESSAGE_BROKER_UNAVAILABLE", NOW.plusSeconds(6));
assertThat(journal.find(TICKET, DIGEST))
.hasValueSatisfying(
record -> {
assertThat(record.state()).isEqualTo(AdminOperationState.FAILED);
assertThat(record.failureCode()).contains("MESSAGE_BROKER_UNAVAILABLE");
});
assertThatCode(
() -> {
AdminOperationLease resumed =
journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW.plusSeconds(7));
assertThat(resumed.resumeFrom()).isEqualTo(12);
})
.doesNotThrowAnyException();
}
@Test
@DisplayName("the schema refuses a digest that is not a SHA-256")
void theSchemaRefusesADigestThatIsNotASha256() throws SQLException {
try (Connection connection = dataSource().getConnection();
Statement statement = connection.createStatement()) {
assertThatThrownBy(
() ->
statement.execute(
"""
INSERT INTO messaging_admin_operation (
approval_ticket, plan_digest, operation_id, state,
lease_owner, lease_expires_at, started_at)
VALUES ('CHG-9', 'not-a-digest', 'op', 'STARTED', 'r', now(), now())
"""))
.as("a malformed digest in the journal makes the uniqueness key meaningless")
.isInstanceOf(SQLException.class);
}
}
private static DataSource dataSource() {
PGSimpleDataSource source = new PGSimpleDataSource();
source.setUrl(POSTGRES.getJdbcUrl());
source.setUser(POSTGRES.getUsername());
source.setPassword(POSTGRES.getPassword());
return source;
}
private static String migration(String fileName) throws IOException {
try (InputStream stream =
AdminOperationJournalPostgresIT.class
.getClassLoader()
.getResourceAsStream("db/migration/messaging/" + fileName)) {
if (stream == null) {
throw new IOException("migration " + fileName + " is missing from the classpath");
}
return new String(stream.readAllBytes(), StandardCharsets.UTF_8);
}
}
}
@@ -0,0 +1,120 @@
package dev.caskeleton.messaging.outbox;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import dev.caskeleton.messaging.api.ContentType;
import dev.caskeleton.messaging.api.MessageId;
import dev.caskeleton.messaging.api.MessageType;
import dev.caskeleton.messaging.api.SchemaVersion;
import dev.caskeleton.messaging.api.destination.DestinationName;
import dev.caskeleton.messaging.api.error.MessagingConfigurationException;
import dev.caskeleton.messaging.reliability.OutboxRecord;
import dev.caskeleton.messaging.reliability.OutboxStatus;
import java.time.Instant;
import java.util.Map;
import java.util.Optional;
import javax.sql.DataSource;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.springframework.transaction.support.TransactionSynchronizationManager;
/**
* What {@code append(record)} does when there is no transaction to join.
*
* <p>The port says the append runs inside the caller's business transaction. The implementation
* called {@code dataSource.getConnection()}, which auto-commits on its own connection — so a
* business transaction that rolled back afterwards left the outbox row behind and the relay
* published an event for a change that never happened.
*
* <p>These are hermetic: the refusal happens before any connection is requested, which is also the
* property under test. The mock would fail the test by being touched at all.
*/
class JdbcOutboxTransactionRequirementTest {
/** A data source that fails the test by being used: the refusal must precede any connection. */
private final DataSource dataSource = new RefusingDataSource();
private final JdbcOutboxRepository repository = new JdbcOutboxRepository(dataSource);
@AfterEach
void clearTransactionState() {
TransactionSynchronizationManager.clear();
if (TransactionSynchronizationManager.hasResource(dataSource)) {
TransactionSynchronizationManager.unbindResource(dataSource);
}
}
@Test
@DisplayName("appending outside a transaction is refused before a connection is opened")
void appendingOutsideATransactionIsRefused() {
assertThatThrownBy(() -> repository.append(record()))
.isInstanceOf(MessagingConfigurationException.class)
.hasMessageContaining("requires an active transaction");
}
@Test
@DisplayName("a read-only transaction is refused")
void aReadOnlyTransactionIsRefused() {
TransactionSynchronizationManager.setActualTransactionActive(true);
TransactionSynchronizationManager.setCurrentTransactionReadOnly(true);
assertThatThrownBy(() -> repository.append(record()))
.isInstanceOf(MessagingConfigurationException.class)
.hasMessageContaining("read-only");
}
@Test
@DisplayName("a transaction bound to another DataSource is refused")
void aTransactionOnAnotherDataSourceIsRefused() {
TransactionSynchronizationManager.setActualTransactionActive(true);
assertThatThrownBy(() -> repository.append(record()))
.as("two DataSources means two commits, which is the atomicity this exists to provide")
.isInstanceOf(MessagingConfigurationException.class)
.hasMessageContaining("bound to this DataSource");
}
@Test
@DisplayName("the refusal names a code an operator can search for")
void theRefusalNamesACode() {
assertThatThrownBy(() -> repository.append(record()))
.isInstanceOfSatisfying(
MessagingConfigurationException.class,
failure ->
assertThat(failure.failure().code()).isEqualTo("OUTBOX_TRANSACTION_REQUIRED"));
}
private static OutboxRecord record() {
return new OutboxRecord(
MessageId.newId(),
new DestinationName("orders.events"),
new MessageType("order.placed"),
new SchemaVersion(1),
ContentType.JSON,
"{}".getBytes(java.nio.charset.StandardCharsets.UTF_8),
Map.of(),
Instant.parse("2026-08-14T00:00:00Z"),
OutboxStatus.PENDING,
0,
Optional.empty(),
Optional.empty());
}
/** Throws if anything asks it for a connection. */
private static final class RefusingDataSource
extends org.springframework.jdbc.datasource.AbstractDataSource {
@Override
public java.sql.Connection getConnection() {
throw new AssertionError(
"the refusal must happen before a connection is requested; opening one is the defect");
}
@Override
public java.sql.Connection getConnection(String username, String password) {
return getConnection();
}
}
}
@@ -40,6 +40,62 @@ class OutboxOperationsTest {
throw new UnsupportedOperationException();
}
@Override
public java.util.List<dev.caskeleton.messaging.reliability.OutboxLease> claimBatch(
String owner, int batchSize, Duration leaseDuration, Instant now) {
return java.util.List.of();
}
@Override
public java.util.List<dev.caskeleton.messaging.reliability.OutboxLease> claimBatch(
String owner, int batchSize, Duration leaseDuration, Instant now, int maxAttempts) {
return java.util.List.of();
}
@Override
public dev.caskeleton.messaging.reliability.OutboxTransitionResult markAmbiguous(
dev.caskeleton.messaging.reliability.OutboxLease lease,
String failureCode,
Instant now,
Instant nextAttemptAt) {
return markAmbiguous(lease, failureCode, now);
}
@Override
public dev.caskeleton.messaging.reliability.OutboxTransitionResult markExhausted(
dev.caskeleton.messaging.reliability.OutboxLease lease, String reason, Instant now) {
markFailed(lease.messageId(), reason, now);
return dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED;
}
@Override
public dev.caskeleton.messaging.reliability.OutboxTransitionResult markPublished(
dev.caskeleton.messaging.reliability.OutboxLease lease, Instant now) {
markPublished(lease.messageId(), now);
return dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED;
}
@Override
public dev.caskeleton.messaging.reliability.OutboxTransitionResult markAmbiguous(
dev.caskeleton.messaging.reliability.OutboxLease lease, String failureCode, Instant now) {
markAmbiguous(lease.messageId(), failureCode, now);
return dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED;
}
@Override
public dev.caskeleton.messaging.reliability.OutboxTransitionResult markFailed(
dev.caskeleton.messaging.reliability.OutboxLease lease, String failureCode, Instant now) {
markFailed(lease.messageId(), failureCode, now);
return dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED;
}
@Override
public dev.caskeleton.messaging.reliability.OutboxTransitionResult releaseLease(
dev.caskeleton.messaging.reliability.OutboxLease lease) {
releaseLease(lease.messageId());
return dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED;
}
@Override
public void markPublished(MessageId messageId, Instant now) {
throw new UnsupportedOperationException();
@@ -65,6 +121,11 @@ class OutboxOperationsTest {
throw new UnsupportedOperationException();
}
@Override
public int purgePublishedBefore(Instant publishedBefore, int limit) {
return Math.min(purgePublishedBefore(publishedBefore), limit);
}
@Override
public int purgePublishedBefore(Instant publishedBefore) {
cutoffs.add(publishedBefore);

Some files were not shown because too many files have changed in this diff Show More