feat: jpa, messaging, notification, mongo, graphql 어댑터터 구현체 추가
This commit is contained in:
@@ -1,9 +1,18 @@
|
||||
# 설정 레퍼런스
|
||||
|
||||
> **Prefix.** Every property below binds under `app.messaging`, which is the prefix the deployed
|
||||
> runtime and the `APP_MESSAGING_*` environment variables already use. Earlier revisions of this
|
||||
> page documented a bare `messaging` prefix and the starter bound `backend.messaging`; neither
|
||||
> bound what this page describes, so a deployment configured from it changed nothing. A key under
|
||||
> either of the old prefixes now fails startup with a message naming the key — see
|
||||
> `MessagingPrefixMigrationValidator`.
|
||||
|
||||
|
||||
## Destination profile
|
||||
|
||||
```yaml
|
||||
messaging:
|
||||
app:
|
||||
messaging:
|
||||
destinations:
|
||||
order-events:
|
||||
broker: kafka-primary
|
||||
@@ -77,7 +86,8 @@ messaging:
|
||||
### Kafka
|
||||
|
||||
```yaml
|
||||
messaging:
|
||||
app:
|
||||
messaging:
|
||||
brokers:
|
||||
kafka-primary:
|
||||
type: kafka
|
||||
@@ -96,7 +106,8 @@ messaging:
|
||||
### RabbitMQ
|
||||
|
||||
```yaml
|
||||
messaging:
|
||||
app:
|
||||
messaging:
|
||||
brokers:
|
||||
rabbit-primary:
|
||||
type: rabbitmq
|
||||
@@ -117,7 +128,8 @@ messaging:
|
||||
## 보안
|
||||
|
||||
```yaml
|
||||
messaging:
|
||||
app:
|
||||
messaging:
|
||||
security:
|
||||
kafka-primary:
|
||||
producer: { type: SASL_SCRAM, credential-id: kafka-producer }
|
||||
@@ -135,7 +147,8 @@ messaging:
|
||||
기본값은 전부 `false`다.
|
||||
|
||||
```yaml
|
||||
messaging:
|
||||
app:
|
||||
messaging:
|
||||
experimental:
|
||||
kafka-share: false
|
||||
pulsar: false
|
||||
@@ -147,7 +160,8 @@ messaging:
|
||||
## Backpressure
|
||||
|
||||
```yaml
|
||||
messaging:
|
||||
app:
|
||||
messaging:
|
||||
backpressure:
|
||||
global-limit: 512
|
||||
per-destination-limit: 64 # global-limit 이하여야 한다
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
# 기존 runtime → 신규 messaging platform cutover (MSG-015)
|
||||
|
||||
## 왜 기계적 매핑이 안 되는가
|
||||
|
||||
두 outbox 모델의 enum 이름이 겹치는데 의미가 반대다.
|
||||
|
||||
| 모델 | retryable | terminal |
|
||||
|---|---|---|
|
||||
| 기존 `OutboxEventStatus` | `FAILED` (`next_attempt_at` 보유) | `DEAD` |
|
||||
| 신규 `OutboxStatus` | `AMBIGUOUS` | `FAILED`, `EXHAUSTED` |
|
||||
|
||||
이름으로 매핑하면 **확정 거절이 무한 재시도**가 되고 **불확정이 park**된다. 그래서 application은
|
||||
자기 어휘(`OutboxPublishOutcome`)만 쓰고, 변환은 bridge adapter가 한다.
|
||||
|
||||
## 지금 반영된 것
|
||||
|
||||
- `OutboxPublishOutcome` — `CONFIRMED` / `AMBIGUOUS` / `REJECTED_BEFORE_SEND` /
|
||||
`REJECTED_AFTER_BROKER`. application이 소유하는 canonical 결과 타입이며, "리턴 or throw"만 가능한
|
||||
기존 어댑터를 위해 `OutboxMessagePublishPort.publishForOutcome`의 default가 `CONFIRMED`를 돌려준다.
|
||||
- `APPLICATION_DOES_NOT_DEPEND_ON_THE_MESSAGING_PLATFORM` ArchUnit 규칙 — application-core가
|
||||
`dev.caskeleton.messaging..`를 import하면 빌드가 깨진다.
|
||||
- 반대 방향(신규 `PublishResult` → application outcome) 매핑 규칙을 테스트로 고정.
|
||||
|
||||
## 남은 것
|
||||
|
||||
- `messaging-platform-bridge` outbound leaf: validated application event → platform envelope,
|
||||
`PublishResult` → `OutboxPublishOutcome`. registry에 leaf를 추가하는 변경이라 별도 커밋.
|
||||
- golden contract 테스트: event/message ID, type, schema revision, partition/order/correlation/
|
||||
causation/tenant/trace, payload digest, wire version이 bytes 단위로 보존되는지.
|
||||
- 단일 publication authority: 기존 `OutboxPublicationAuthority` fence를 재사용해 writer/relay가
|
||||
동시에 ACTIVE가 되지 않도록. **dual write/publish는 금지** — 한 business fact가 두 durable store와
|
||||
두 relay로 나가는 상태가 cutover에서 가장 위험하다.
|
||||
- 첫 cutover 범위는 **transport만** 교체(저장소는 기존 유지). storage migration은 shadow read →
|
||||
authority switch → old backlog drain 순서로 별도 release.
|
||||
@@ -41,9 +41,26 @@ failed로 표시하면 broker가 이미 가지고 있을 수 있는 메시지를
|
||||
### lease
|
||||
|
||||
```text
|
||||
status IN ('PENDING','AMBIGUOUS') AND (lease_expires_at IS NULL OR lease_expires_at <= now)
|
||||
status IN ('PENDING','AMBIGUOUS','IN_FLIGHT')
|
||||
AND (lease_expires_at IS NULL OR lease_expires_at <= now)
|
||||
AND next_attempt_at <= now
|
||||
AND attempts < maxAttempts
|
||||
```
|
||||
|
||||
`IN_FLIGHT`가 목록에 있는 것이 핵심이다. relay가 publish 도중 죽으면 row는 `IN_FLIGHT`로 남는데,
|
||||
이를 제외하면 그 메시지는 **영원히** 발행되지 않는다 — outbox가 막으려던 바로 그 실패다. 대신
|
||||
lease가 만료됐을 때만 회수하므로, 살아 있는 relay가 들고 있는 row는 회수되지 않는다.
|
||||
|
||||
회수는 **같은 `message_id`로** 이루어지고 `lease_token`이 1 증가한다. 새 id를 발급하면 "전달됐을
|
||||
수도 있는 메시지"가 "확실히 두 번째"가 되기 때문이다 (위의 AMBIGUOUS 논의와 같은 이유).
|
||||
|
||||
이 문단의 근거는 실제 PostgreSQL 컨테이너 레인이다:
|
||||
|
||||
- `OutboxPostgresIT#anExpiredLeaseBecomesClaimableAgain` — 만료된 lease의 재회수
|
||||
- `OutboxPostgresIT#anExpiryReclaimKeepsTheMessageIdAndAdvancesTheToken` — 같은 id, 증가한 token
|
||||
- `OutboxPostgresIT#aLeasedRowIsInvisibleToASecondRelayInstance` — 살아 있는 lease는 회수 불가
|
||||
- `OutboxPostgresIT#aSupersededRelayCannotOverwriteTheOutcomeOfTheOneThatReplacedIt` — fencing
|
||||
|
||||
partial index `ix_messaging_outbox_claimable`이 이 쿼리를 backlog 크기에 비례하게 유지한다.
|
||||
PUBLISHED row는 retention job이 지울 때까지 쌓이기 때문이다.
|
||||
|
||||
|
||||
@@ -3,11 +3,28 @@
|
||||
플랫폼이 **무엇을 보장하는지**와 **무엇을 보장하지 않는지**를 브로커별로 고정한다.
|
||||
여기 없는 조합은 지원되지 않는다.
|
||||
|
||||
> **인증 근거.** 이 표의 버전은 이 저장소의 컨테이너 레인이 실제로 실행한 이미지다. 이전 판은
|
||||
> Kafka 4.2/4.3을 선언했지만 fixture는 `apache/kafka:4.1.0`, lockfile client는 4.1.1이었다 — 표와
|
||||
> 코드 상수가 서로 일치했을 뿐 어느 쪽도 실행된 적이 없었다. 장애 시나리오 커버리지도 마찬가지로
|
||||
> `BrokerFailureMatrix.shipped()` 하드코딩이 아니라 레인이 낸 증거(`BrokerCertificationEvidence`)에서
|
||||
> 나온다. 증거가 없는 조합은 `NOT_COVERED`다 (MSG-014).
|
||||
|
||||
> **모듈 이름과 런타임 편입.** `messaging-outbox-jdbc-postgresql` / `messaging-inbox-jdbc-postgresql`은
|
||||
> 이전에 `-jpa`로 불렸다. 구현은 Spring JDBC이고 SQL은 PostgreSQL 전용(`?::jsonb`,
|
||||
> `FOR UPDATE SKIP LOCKED`, `ON CONFLICT`, `TIMESTAMPTZ`)이므로, 그 이름은 쓰지 않는 기술을
|
||||
> 광고하고 vendor 중립 port(`messaging-reliability-api`)의 위치를 가렸다 (MSG-023).
|
||||
>
|
||||
> 또한 registry의 messaging leaf는 모두 `runtime_memberships`가 비어 있다. 이는 **build-only /
|
||||
> incubating** — 어느 composition root에도 편입되지 않았다는 뜻이며, 아래의 등급과는 다른 축이다.
|
||||
> 등급은 "무엇이 증명되었는가", membership은 "무엇이 실행되는가"를 말한다. 애플리케이션에 배선하려면
|
||||
> registry를 먼저 바꾸고 `verifyRuntimeModuleMembership`을 통과시켜야 한다. 자세한 규칙은
|
||||
> `src/messaging/CLAUDE.md`가 소유한다.
|
||||
|
||||
## 브로커 등급
|
||||
|
||||
| 브로커 | 등급 | 인증 기준 | Stable 기능 | 제한 |
|
||||
|---|---|---|---|---|
|
||||
| Kafka | Stable | 4.2+ / 4.3.x | producer idempotence, consumer group, batch, pause/resume, replay, transaction capability | Share Group은 Experimental |
|
||||
| Kafka | Stable | 4.1.x | producer idempotence, consumer group, batch, pause/resume, replay, transaction capability | Share Group은 Experimental |
|
||||
| RabbitMQ | Stable | 4.3.x | exchange/routing, publisher confirm, mandatory return, manual ACK, quorum queue, retry queue, DLQ | stream 및 특수 plugin 미지원 |
|
||||
| Pulsar | Experimental | 4.0 LTS + 4.2 | typed publish/consume, Shared, Key_Shared, schema | transaction 미승격, 기본 비활성 |
|
||||
| NATS JetStream | Experimental | 2.14.x | stream, durable consumer, explicit ACK, dedupe, replay | native DLQ 없음(플랫폼이 대행), 기본 비활성 |
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
# messaging — local authority for the messaging platform family
|
||||
|
||||
이 문서는 messaging family의 **local authority**다. leaf 목록·gradle path·허용 의존성은
|
||||
`src/config/architecture/modules.json`이 SSOT이며 이 문서는 그것을 복제하지 않는다. 여기서 정하는
|
||||
것은 registry가 표현할 수 없는 것들이다: family별 framework 규칙, public surface 정책, Stable 승격에
|
||||
필요한 증거, runtime membership이 비어 있다는 사실의 의미.
|
||||
|
||||
Root 정책(`CLAUDE.md` / `AGENTS.md`)과 충돌하면 root가 이긴다.
|
||||
|
||||
## Family와 framework 규칙
|
||||
|
||||
leaf의 정확한 id는 registry에서 읽는다. family는 **디렉터리 이름의 접두사**가 아니라 아래 표의
|
||||
역할로 정해진다.
|
||||
|
||||
| Family | 역할 | 허용 framework |
|
||||
| --- | --- | --- |
|
||||
| `*-api` (`core-api`, `schema-api`, `policy`, `reliability-api`, `admin-api`, `transport-spi`) | 값 타입, 포트, 정책 계약 | Java stdlib만. Spring/broker SDK/logging framework 금지 |
|
||||
| `schema-*` (json/avro/protobuf), `cloudevents` | 인코딩 구현 | 해당 codec 라이브러리만. Spring 금지 |
|
||||
| `runtime-core`, `security`, `observability`, `admin-runtime` | framework-free 실행 로직 | Micrometer는 `observability`에서만. 나머지는 stdlib |
|
||||
| `kafka`, `rabbit`, `pulsar-experimental`, `nats-experimental`, `kafka-share-experimental` | broker adapter | 해당 broker client. Spring은 `rabbit`의 `spring-rabbit`처럼 client가 곧 Spring인 경우만 |
|
||||
| `outbox-jdbc-postgresql`, `inbox-jdbc-postgresql` | 관계형 신뢰성 adapter | Spring JDBC / Spring TX. **JPA 아님** — 이름이 곧 계약이다 |
|
||||
| `spring-boot-starter`, `spring-cloud-stream-bridge` | 조립 경계 | Spring Boot autoconfigure. 여기서만 |
|
||||
| `testkit` | 테스트 지원 | Testcontainers, JUnit. production leaf가 의존하지 않는다 |
|
||||
|
||||
`outbox`/`inbox` leaf가 `jdbc-postgresql`로 명명된 이유: 구현은 Spring JDBC이고 SQL은
|
||||
`?::jsonb` · `FOR UPDATE SKIP LOCKED` · `ON CONFLICT` · `TIMESTAMPTZ`로 PostgreSQL 전용이다.
|
||||
`jpa`라는 이름은 사용하지 않는 기술을 광고하고, vendor 중립 port가 어디 있는지도 숨겼다. vendor
|
||||
중립 계약은 `messaging-reliability-api`에 있다.
|
||||
|
||||
## Public surface 정책
|
||||
|
||||
- **vendor 타입을 public signature에 노출하면 그 의존성은 `api`로 선언한다.** 숨긴다고 타입이
|
||||
숨겨지지 않는다 — consumer가 이 모듈이 문서화한 API에 대해 컴파일하지 못하게 만들 뿐이다.
|
||||
(`kafka-clients`, `spring-rabbit`, `cloudevents-api`, `micrometer-core`가 이 규칙으로 `api`다.)
|
||||
- 반대로 **노출하지 않는 것은 `implementation`으로 낮춘다.** starter가 노출하지 않는 하위 모듈을
|
||||
`implementation`으로 내린 것이 그 예다.
|
||||
- 새 public 타입은 그 모듈의 계약이다. 삭제·시그니처 변경은 breaking change로 취급한다.
|
||||
|
||||
## Runtime membership이 비어 있다는 것의 의미
|
||||
|
||||
registry의 모든 messaging leaf는 `runtime_memberships`가 비어 있다. 이는 **build-only /
|
||||
incubating**을 뜻한다:
|
||||
|
||||
- 두 composition root(`app-bootstrap`, `sample-portfolio`) 중 어디에도 런타임으로 들어가 있지 않다.
|
||||
- 따라서 이 leaf들은 컴파일·테스트·계약 검증의 대상이지, 배포된 애플리케이션이 실행하는 코드가
|
||||
아니다.
|
||||
- 애플리케이션에 실제로 배선하려면 registry의 `runtime_memberships`를 먼저 바꾸고
|
||||
`verifyRuntimeModuleMembership`을 통과시켜야 한다. 코드만 추가하는 것으로는 런타임에 들어가지
|
||||
않는다.
|
||||
- 지원 상태 문서(`docs/messaging/support-matrix.md`)의 tier는 이 사실과 별개다. tier는 "무엇이
|
||||
증명되었는가", membership은 "무엇이 실행되는가"다.
|
||||
|
||||
## Stable 승격에 필요한 증거
|
||||
|
||||
adapter를 Experimental → Stable로 올릴 때 필요한 것 (`docs/messaging/support-matrix.md`가 현재
|
||||
상태를 기록한다):
|
||||
|
||||
1. 실제 broker container를 쓰는 fault 시나리오 lane이 존재하고, 그 lane이
|
||||
`BrokerCertificationEvidence`를 산출한다.
|
||||
2. `NetworkFaultScenario` 전 항목에 대해 evidence가 있거나, 없는 항목이 `knownGaps`로 명시된다.
|
||||
3. capability 선언이 코드에서 파생된다 — `MessagingCapabilities`에 손으로 `true`를 쓰는 것은
|
||||
증거가 아니다.
|
||||
4. publish 실패 분류가 typed signal에서만 나온다. exception class 이름 substring 판정은
|
||||
Experimental에서도 금지다(MSG-022).
|
||||
5. unknown failure의 기본값이 `AMBIGUOUS`다. `REJECTED`는 전송이 시작되지 않았다는 typed 증거가
|
||||
있을 때만.
|
||||
|
||||
Stable 문구는 evidence manifest가 존재할 때만 문서에 쓴다.
|
||||
|
||||
## Focused test
|
||||
|
||||
leaf의 `gradle_path`를 registry에서 읽어 파생한다.
|
||||
|
||||
```bash
|
||||
cd src
|
||||
./gradlew <gradle-path>:test --console=plain
|
||||
```
|
||||
|
||||
Family 전체 검증:
|
||||
|
||||
```bash
|
||||
./gradlew verifyCleanArchitectureDependencies --console=plain
|
||||
./gradlew verifyDocumentedLeafCount --console=plain
|
||||
./gradlew verifyDependencyLocks --console=plain
|
||||
```
|
||||
+93
@@ -0,0 +1,93 @@
|
||||
package dev.caskeleton.messaging.admin;
|
||||
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.Optional;
|
||||
|
||||
/**
|
||||
* Where a destructive operation records that it started, how far it got, and how it ended.
|
||||
*
|
||||
* <p>The implementation this replaced was a {@code ConcurrentHashMap} registered by the starter as
|
||||
* the default. Two consequences followed, and both are worse than having no store at all because
|
||||
* the map made the platform look protected. A restart forgot every claim, so the same approval
|
||||
* could be executed again by the same process. And two replicas each had their own map, so both
|
||||
* could claim the same approval at the same moment and each believe it was the only one.
|
||||
*
|
||||
* <p>Implementations must therefore be shared and durable, must enforce uniqueness on {@code
|
||||
* (approvalTicket, planDigest)}, and must hand out leases with a monotonic fencing token so a
|
||||
* process that stalled past its lease cannot write over the replica that took over from it.
|
||||
*/
|
||||
public interface AdminOperationJournal {
|
||||
|
||||
/**
|
||||
* Claims an operation, or resumes the previous attempt at it.
|
||||
*
|
||||
* @param approvalTicket the approval authorising the operation
|
||||
* @param planDigest the exact plan
|
||||
* @param operationId this execution's identity
|
||||
* @param leaseOwner the runtime claiming it
|
||||
* @param leaseDuration how long the lease holds before another runtime may take over
|
||||
* @param now the current instant
|
||||
* @return the lease, carrying the point to resume from
|
||||
* @throws dev.caskeleton.messaging.api.error.MessageAuthorizationException when the operation
|
||||
* already completed, or another runtime holds a live lease on it
|
||||
*/
|
||||
AdminOperationLease begin(
|
||||
String approvalTicket,
|
||||
PlanDigest planDigest,
|
||||
String operationId,
|
||||
String leaseOwner,
|
||||
Duration leaseDuration,
|
||||
Instant now);
|
||||
|
||||
/**
|
||||
* Records durable progress and extends the lease.
|
||||
*
|
||||
* @param lease the lease held
|
||||
* @param itemsCompleted how many items are now durably done
|
||||
* @param leaseDuration how much longer to hold the lease
|
||||
* @param now the current instant
|
||||
* @throws dev.caskeleton.messaging.api.error.MessageAuthorizationException when the lease has
|
||||
* been taken over by a newer token
|
||||
*/
|
||||
void checkpoint(
|
||||
AdminOperationLease lease, int itemsCompleted, Duration leaseDuration, Instant now);
|
||||
|
||||
/**
|
||||
* Marks the operation finished. The approval is spent.
|
||||
*
|
||||
* @param lease the lease held
|
||||
* @param itemsCompleted the final count
|
||||
* @param now the current instant
|
||||
*/
|
||||
void complete(AdminOperationLease lease, int itemsCompleted, Instant now);
|
||||
|
||||
/**
|
||||
* Marks the operation failed at its current checkpoint, leaving it resumable.
|
||||
*
|
||||
* @param lease the lease held
|
||||
* @param itemsCompleted how many items are durably done
|
||||
* @param failureCode a sanitised, bounded reason
|
||||
* @param now the current instant
|
||||
*/
|
||||
void fail(AdminOperationLease lease, int itemsCompleted, String failureCode, Instant now);
|
||||
|
||||
/**
|
||||
* Reads the journal entry for an operation.
|
||||
*
|
||||
* @param approvalTicket the approval
|
||||
* @param planDigest the plan
|
||||
* @return the record, when the operation has been journalled
|
||||
*/
|
||||
Optional<AdminOperationRecord> find(String approvalTicket, PlanDigest planDigest);
|
||||
|
||||
/**
|
||||
* Whether this implementation survives a restart and is shared between replicas.
|
||||
*
|
||||
* <p>Declared rather than inferred so the starter can refuse to run a production profile on an
|
||||
* in-memory journal instead of discovering the gap during an incident.
|
||||
*
|
||||
* @return true when the journal is durable and shared
|
||||
*/
|
||||
boolean isDurable();
|
||||
}
|
||||
+55
@@ -0,0 +1,55 @@
|
||||
package dev.caskeleton.messaging.admin;
|
||||
|
||||
import java.util.Objects;
|
||||
|
||||
/**
|
||||
* The right to execute one journalled operation, and the point to resume it from.
|
||||
*
|
||||
* <p>{@code resumeFrom} is the whole reason a lease is handed out rather than a boolean. A retry
|
||||
* after a crash is not a new execution of the approval — it is the same operation continuing, and
|
||||
* treating it as new is what republishes the items the first attempt already moved.
|
||||
*
|
||||
* @param approvalTicket the approval being executed
|
||||
* @param planDigest the plan being executed
|
||||
* @param operationId this execution's identity
|
||||
* @param leaseOwner the runtime holding the lease
|
||||
* @param leaseToken the fencing token; a write from an older token must be refused
|
||||
* @param resumeFrom how many items the previous attempt durably completed
|
||||
*/
|
||||
public record AdminOperationLease(
|
||||
String approvalTicket,
|
||||
PlanDigest planDigest,
|
||||
String operationId,
|
||||
String leaseOwner,
|
||||
long leaseToken,
|
||||
int resumeFrom) {
|
||||
|
||||
/** Validates the lease. */
|
||||
public AdminOperationLease {
|
||||
Objects.requireNonNull(planDigest, "planDigest must not be null");
|
||||
if (approvalTicket == null || approvalTicket.isBlank()) {
|
||||
throw new IllegalArgumentException("approvalTicket must not be blank");
|
||||
}
|
||||
if (operationId == null || operationId.isBlank()) {
|
||||
throw new IllegalArgumentException("operationId must not be blank");
|
||||
}
|
||||
if (leaseOwner == null || leaseOwner.isBlank()) {
|
||||
throw new IllegalArgumentException("leaseOwner must not be blank");
|
||||
}
|
||||
if (leaseToken < 1) {
|
||||
throw new IllegalArgumentException("a lease token starts at one");
|
||||
}
|
||||
if (resumeFrom < 0) {
|
||||
throw new IllegalArgumentException("resumeFrom must not be negative");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Reports whether this lease continues an earlier attempt.
|
||||
*
|
||||
* @return true when items were already completed
|
||||
*/
|
||||
public boolean isResumption() {
|
||||
return resumeFrom > 0;
|
||||
}
|
||||
}
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
package dev.caskeleton.messaging.admin;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
|
||||
/**
|
||||
* One row of the destructive-operation journal.
|
||||
*
|
||||
* <p>Keyed by {@code (approvalTicket, planDigest)} rather than by the ticket alone, because the
|
||||
* ticket alone cannot distinguish "this approval already ran" from "this approval is being reused
|
||||
* for a different plan" — and those need opposite answers.
|
||||
*
|
||||
* @param approvalTicket the approval that authorised the operation
|
||||
* @param planDigest the exact plan that was authorised
|
||||
* @param operationId the caller-side identity of this execution
|
||||
* @param state where the operation stands
|
||||
* @param itemsCompleted how many items are durably done
|
||||
* @param leaseOwner which runtime currently holds the lease
|
||||
* @param leaseToken the fencing token, incremented on every takeover
|
||||
* @param leaseExpiresAt when another runtime may take the lease over
|
||||
* @param startedAt when the operation first began
|
||||
* @param finishedAt when it reached a terminal state
|
||||
* @param failureCode the sanitised reason it failed, when it did
|
||||
*/
|
||||
public record AdminOperationRecord(
|
||||
String approvalTicket,
|
||||
PlanDigest planDigest,
|
||||
String operationId,
|
||||
AdminOperationState state,
|
||||
int itemsCompleted,
|
||||
String leaseOwner,
|
||||
long leaseToken,
|
||||
Instant leaseExpiresAt,
|
||||
Instant startedAt,
|
||||
Optional<Instant> finishedAt,
|
||||
Optional<String> failureCode) {
|
||||
|
||||
/** Validates the record. */
|
||||
public AdminOperationRecord {
|
||||
Objects.requireNonNull(planDigest, "planDigest must not be null");
|
||||
Objects.requireNonNull(state, "state must not be null");
|
||||
Objects.requireNonNull(leaseExpiresAt, "leaseExpiresAt must not be null");
|
||||
Objects.requireNonNull(startedAt, "startedAt must not be null");
|
||||
Objects.requireNonNull(finishedAt, "finishedAt must not be null");
|
||||
Objects.requireNonNull(failureCode, "failureCode must not be null");
|
||||
if (approvalTicket == null || approvalTicket.isBlank()) {
|
||||
throw new IllegalArgumentException("approvalTicket must not be blank");
|
||||
}
|
||||
if (operationId == null || operationId.isBlank()) {
|
||||
throw new IllegalArgumentException("operationId must not be blank");
|
||||
}
|
||||
if (leaseOwner == null || leaseOwner.isBlank()) {
|
||||
throw new IllegalArgumentException("leaseOwner must not be blank");
|
||||
}
|
||||
if (itemsCompleted < 0) {
|
||||
throw new IllegalArgumentException("itemsCompleted must not be negative");
|
||||
}
|
||||
if (leaseToken < 1) {
|
||||
throw new IllegalArgumentException("a lease token starts at one");
|
||||
}
|
||||
}
|
||||
}
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
package dev.caskeleton.messaging.admin;
|
||||
|
||||
/**
|
||||
* Where a journalled destructive operation stands.
|
||||
*
|
||||
* <p>The store this replaces recorded one fact — "this approval was claimed" — and recorded it
|
||||
* before any work happened. A run that died halfway had consumed its approval, left no record of
|
||||
* how far it got, and offered the operator two equally bad choices: request a fresh approval and
|
||||
* redo work that may already have been done, or leave the operation half-applied.
|
||||
*/
|
||||
public enum AdminOperationState {
|
||||
|
||||
/** Claimed and in progress. A lease is held; the checkpoint says how far it got. */
|
||||
STARTED,
|
||||
|
||||
/** Every item finished. The approval is spent and must not run again. */
|
||||
COMPLETED,
|
||||
|
||||
/** The run stopped before finishing. It may be resumed from its checkpoint under a new lease. */
|
||||
FAILED
|
||||
}
|
||||
+98
@@ -0,0 +1,98 @@
|
||||
package dev.caskeleton.messaging.admin;
|
||||
|
||||
import dev.caskeleton.messaging.api.destination.DestinationName;
|
||||
import java.time.Instant;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
|
||||
/**
|
||||
* Everything an issuer signs when granting one destructive operation.
|
||||
*
|
||||
* <p>{@link AdminApproval} carried a ticket, an approver, and a window. Nothing in it said which
|
||||
* operation, on which destination, against which topology, or up to what impact — so the same
|
||||
* approval authorised every plan presented inside the window, and the audit trail recorded a ticket
|
||||
* that proved nothing about what was executed.
|
||||
*
|
||||
* <p>Every field here is part of the signed canonical form, which is what makes the grant
|
||||
* non-transferable: change the target, the topology version, the impact ceiling, or the plan, and
|
||||
* the signature no longer verifies.
|
||||
*
|
||||
* @param approval the ticket, approver, and validity window
|
||||
* @param requestedBy the operator who will execute it, kept separate from the approver
|
||||
* @param operation the destructive operation being authorised
|
||||
* @param source the destination read from
|
||||
* @param target the destination written to, when the operation has one
|
||||
* @param topologyVersion the topology the impact estimate was computed against
|
||||
* @param maxImpact the largest number of messages the approver agreed to
|
||||
* @param planDigest the fingerprint of the exact plan
|
||||
*/
|
||||
public record ApprovalGrant(
|
||||
AdminApproval approval,
|
||||
String requestedBy,
|
||||
DestructiveOperation operation,
|
||||
DestinationName source,
|
||||
Optional<DestinationName> target,
|
||||
String topologyVersion,
|
||||
long maxImpact,
|
||||
PlanDigest planDigest) {
|
||||
|
||||
/** Validates the grant and enforces subject separation. */
|
||||
public ApprovalGrant {
|
||||
Objects.requireNonNull(approval, "approval must not be null");
|
||||
Objects.requireNonNull(operation, "operation must not be null");
|
||||
Objects.requireNonNull(source, "source must not be null");
|
||||
Objects.requireNonNull(target, "target must not be null");
|
||||
Objects.requireNonNull(topologyVersion, "topologyVersion must not be null");
|
||||
Objects.requireNonNull(planDigest, "planDigest must not be null");
|
||||
if (requestedBy == null || requestedBy.isBlank()) {
|
||||
throw new IllegalArgumentException("requestedBy must not be blank");
|
||||
}
|
||||
if (approval.approvedBy().equals(requestedBy)) {
|
||||
// Four-eyes, expressed where it cannot be skipped: an operator approving their own
|
||||
// destructive operation is the control not existing.
|
||||
throw new IllegalArgumentException(
|
||||
"the approver and the operator must be different people; both are " + requestedBy);
|
||||
}
|
||||
if (maxImpact < 0) {
|
||||
throw new IllegalArgumentException("maxImpact must not be negative");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Renders the grant deterministically for signing and verification.
|
||||
*
|
||||
* <p>Every field is length-prefixed rather than delimited. A delimiter can be smuggled into a
|
||||
* ticket or an identity to make two different grants render identically; a length prefix cannot.
|
||||
*
|
||||
* @return the canonical form
|
||||
*/
|
||||
public String canonicalForm() {
|
||||
StringBuilder canonical = new StringBuilder("v1");
|
||||
appendField(canonical, approval.ticket());
|
||||
appendField(canonical, approval.approvedBy());
|
||||
appendField(canonical, requestedBy);
|
||||
appendField(canonical, operation.name());
|
||||
appendField(canonical, source.value());
|
||||
appendField(canonical, target.map(DestinationName::value).orElse(""));
|
||||
appendField(canonical, topologyVersion);
|
||||
appendField(canonical, Long.toString(maxImpact));
|
||||
appendField(canonical, planDigest.value());
|
||||
appendField(canonical, approval.approvedAt().toString());
|
||||
appendField(canonical, approval.validUntil().toString());
|
||||
return canonical.toString();
|
||||
}
|
||||
|
||||
private static void appendField(StringBuilder canonical, String value) {
|
||||
canonical.append(value.length()).append(':').append(value);
|
||||
}
|
||||
|
||||
/**
|
||||
* Reports whether the grant still authorises anything.
|
||||
*
|
||||
* @param now the current instant
|
||||
* @return true while inside the approval window
|
||||
*/
|
||||
public boolean isValidAt(Instant now) {
|
||||
return approval.isValidAt(now);
|
||||
}
|
||||
}
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
package dev.caskeleton.messaging.admin;
|
||||
|
||||
import java.time.Instant;
|
||||
|
||||
/**
|
||||
* The only thing that can turn a signed grant into a {@link VerifiedApproval}.
|
||||
*
|
||||
* <p>An implementation must check, at minimum: the issuer's signature over {@link
|
||||
* ApprovalGrant#canonicalForm()}, that the approval window is open at {@code now}, and that the
|
||||
* digest the caller is about to execute is the digest that was signed. Everything else the grant
|
||||
* binds — operation, source, target, topology version, impact ceiling, approver-versus-operator
|
||||
* separation — is inside the canonical form, so a signature check covers them all at once.
|
||||
*/
|
||||
public interface ApprovalVerifier {
|
||||
|
||||
/**
|
||||
* Verifies a signed grant against the plan actually about to run.
|
||||
*
|
||||
* @param grant what the issuer signed
|
||||
* @param signature the issuer's signature over the grant's canonical form
|
||||
* @param executingDigest the digest of the plan the caller is about to execute
|
||||
* @param now the current instant
|
||||
* @return the verified approval
|
||||
* @throws dev.caskeleton.messaging.api.error.MessageAuthorizationException when anything about
|
||||
* the grant, the signature, the window, or the digest does not hold
|
||||
*/
|
||||
VerifiedApproval verify(
|
||||
ApprovalGrant grant, String signature, PlanDigest executingDigest, Instant now);
|
||||
}
|
||||
+40
-1
@@ -1,5 +1,6 @@
|
||||
package dev.caskeleton.messaging.admin;
|
||||
|
||||
import dev.caskeleton.messaging.api.destination.DestinationName;
|
||||
import dev.caskeleton.messaging.api.error.MessageAuthorizationException;
|
||||
import java.time.Instant;
|
||||
import java.util.Objects;
|
||||
@@ -17,11 +18,42 @@ import java.util.Objects;
|
||||
* @param loopAcknowledged whether the approver accepted the previously-redriven candidates
|
||||
*/
|
||||
public record ApprovedRedrivePlan(
|
||||
RedrivePlan plan, AdminApproval approval, boolean loopAcknowledged) {
|
||||
RedrivePlan plan, VerifiedApproval approval, boolean loopAcknowledged) {
|
||||
|
||||
/** Binds the approval to this exact plan at construction. */
|
||||
public ApprovedRedrivePlan {
|
||||
Objects.requireNonNull(plan, "plan must not be null");
|
||||
Objects.requireNonNull(approval, "approval must not be null");
|
||||
if (!approval.grant().planDigest().equals(plan.digest())) {
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_PLAN_MISMATCH",
|
||||
"approval %s was issued for a different plan than the one it is attached to"
|
||||
.formatted(approval.ticket()));
|
||||
}
|
||||
if (approval.grant().operation() != DestructiveOperation.REDRIVE) {
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_OPERATION_MISMATCH",
|
||||
"approval %s authorises %s, not a redrive"
|
||||
.formatted(approval.ticket(), approval.grant().operation()));
|
||||
}
|
||||
if (!approval.grant().source().equals(plan.request().source())
|
||||
|| !approval.grant().target().equals(java.util.Optional.of(plan.request().target()))) {
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_SOURCE_MISMATCH",
|
||||
"approval %s authorises %s to %s, not %s to %s"
|
||||
.formatted(
|
||||
approval.ticket(),
|
||||
approval.grant().source().value(),
|
||||
approval.grant().target().map(DestinationName::value).orElse("(none)"),
|
||||
plan.request().source().value(),
|
||||
plan.request().target().value()));
|
||||
}
|
||||
if (plan.candidates() > approval.grant().maxImpact()) {
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_IMPACT_EXCEEDED",
|
||||
"the plan moves %d messages but the approver agreed to at most %d"
|
||||
.formatted(plan.candidates(), approval.grant().maxImpact()));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -39,6 +71,13 @@ public record ApprovedRedrivePlan(
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_EXPIRED", "approval %s is not valid at %s".formatted(approval.ticket(), now));
|
||||
}
|
||||
if (!approval.grant().topologyVersion().equals(currentTopologyVersion)) {
|
||||
throw new MessageAuthorizationException(
|
||||
"TOPOLOGY_CHANGED_SINCE_APPROVAL",
|
||||
"approval %s was signed against topology %s but the broker is now at %s"
|
||||
.formatted(
|
||||
approval.ticket(), approval.grant().topologyVersion(), currentTopologyVersion));
|
||||
}
|
||||
if (!plan.topologyVersion().equals(currentTopologyVersion)) {
|
||||
throw new MessageAuthorizationException(
|
||||
"TOPOLOGY_CHANGED_SINCE_APPROVAL",
|
||||
|
||||
+38
-1
@@ -14,11 +14,41 @@ import java.util.Objects;
|
||||
* @param plan the plan that was approved
|
||||
* @param approval the approval authorising it
|
||||
*/
|
||||
public record ApprovedReplayPlan(ReplayPlan plan, AdminApproval approval) {
|
||||
public record ApprovedReplayPlan(ReplayPlan plan, VerifiedApproval approval) {
|
||||
|
||||
/** Binds the approval to this exact plan at construction. */
|
||||
public ApprovedReplayPlan {
|
||||
Objects.requireNonNull(plan, "plan must not be null");
|
||||
Objects.requireNonNull(approval, "approval must not be null");
|
||||
if (!approval.grant().planDigest().equals(plan.digest())) {
|
||||
// A genuine, verified approval for a different plan. Checking it here means the mismatch
|
||||
// cannot survive to the execute call under any code path.
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_PLAN_MISMATCH",
|
||||
"approval %s was issued for a different plan than the one it is attached to"
|
||||
.formatted(approval.ticket()));
|
||||
}
|
||||
if (approval.grant().operation() != DestructiveOperation.REPLAY) {
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_OPERATION_MISMATCH",
|
||||
"approval %s authorises %s, not a replay"
|
||||
.formatted(approval.ticket(), approval.grant().operation()));
|
||||
}
|
||||
if (!approval.grant().source().equals(plan.request().destination())) {
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_SOURCE_MISMATCH",
|
||||
"approval %s authorises %s, not %s"
|
||||
.formatted(
|
||||
approval.ticket(),
|
||||
approval.grant().source().value(),
|
||||
plan.request().destination().value()));
|
||||
}
|
||||
if (plan.estimatedMessages() > approval.grant().maxImpact()) {
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_IMPACT_EXCEEDED",
|
||||
"the plan moves about %d messages but the approver agreed to at most %d"
|
||||
.formatted(plan.estimatedMessages(), approval.grant().maxImpact()));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -36,6 +66,13 @@ public record ApprovedReplayPlan(ReplayPlan plan, AdminApproval approval) {
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_EXPIRED", "approval %s is not valid at %s".formatted(approval.ticket(), now));
|
||||
}
|
||||
if (!approval.grant().topologyVersion().equals(currentTopologyVersion)) {
|
||||
throw new MessageAuthorizationException(
|
||||
"TOPOLOGY_CHANGED_SINCE_APPROVAL",
|
||||
"approval %s was signed against topology %s but the broker is now at %s"
|
||||
.formatted(
|
||||
approval.ticket(), approval.grant().topologyVersion(), currentTopologyVersion));
|
||||
}
|
||||
if (!plan.topologyVersion().equals(currentTopologyVersion)) {
|
||||
// Every number in the plan was computed against the old topology, so the approver agreed to
|
||||
// an impact estimate that no longer describes what would happen.
|
||||
|
||||
+26
-4
@@ -9,7 +9,7 @@ import java.util.Optional;
|
||||
/**
|
||||
* The single gate every destructive messaging operation passes through.
|
||||
*
|
||||
* <p>Three conditions, all required. The caller must hold the admin credential — an application
|
||||
* <p>Four conditions, all required. The caller must hold the admin credential — an application
|
||||
* runtime does not, by construction. The approval must be present and inside its validity window.
|
||||
* And a dry run is always permitted, because the way to make operators plan before they act is to
|
||||
* make planning free.
|
||||
@@ -35,7 +35,7 @@ public final class DestructiveOperationGuard {
|
||||
*
|
||||
* @param operation the operation
|
||||
* @param destination the destination affected
|
||||
* @param approval the approval, when one was supplied
|
||||
* @param approval the verified approval, when one was supplied
|
||||
* @param dryRun whether this is a plan-only run
|
||||
* @param now the current instant
|
||||
* @throws MessageAuthorizationException when the operation is not authorised
|
||||
@@ -43,7 +43,7 @@ public final class DestructiveOperationGuard {
|
||||
public void authorize(
|
||||
DestructiveOperation operation,
|
||||
DestinationName destination,
|
||||
Optional<AdminApproval> approval,
|
||||
Optional<VerifiedApproval> approval,
|
||||
boolean dryRun,
|
||||
Instant now) {
|
||||
Objects.requireNonNull(operation, "operation must not be null");
|
||||
@@ -59,7 +59,7 @@ public final class DestructiveOperationGuard {
|
||||
"ADMIN_CREDENTIAL_REQUIRED",
|
||||
operation + " on " + destination.value() + " requires the admin credential");
|
||||
}
|
||||
AdminApproval granted =
|
||||
VerifiedApproval granted =
|
||||
approval.orElseThrow(
|
||||
() ->
|
||||
new MessageAuthorizationException(
|
||||
@@ -69,5 +69,27 @@ public final class DestructiveOperationGuard {
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_EXPIRED", "approval " + granted.ticket() + " is outside its validity window");
|
||||
}
|
||||
if (granted.grant().operation() != operation) {
|
||||
// The approval verified — for something else. A guard that only checks presence and window
|
||||
// lets a verified redrive approval authorise a destination deletion.
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_OPERATION_MISMATCH",
|
||||
"approval "
|
||||
+ granted.ticket()
|
||||
+ " authorises "
|
||||
+ granted.grant().operation()
|
||||
+ ", not "
|
||||
+ operation);
|
||||
}
|
||||
if (!granted.grant().source().equals(destination)) {
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_SOURCE_MISMATCH",
|
||||
"approval "
|
||||
+ granted.ticket()
|
||||
+ " authorises "
|
||||
+ granted.grant().source().value()
|
||||
+ ", not "
|
||||
+ destination.value());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+111
@@ -0,0 +1,111 @@
|
||||
package dev.caskeleton.messaging.admin;
|
||||
|
||||
import dev.caskeleton.messaging.api.error.MessageAuthorizationException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.MessageDigest;
|
||||
import java.time.Instant;
|
||||
import java.util.HexFormat;
|
||||
import java.util.Objects;
|
||||
import javax.crypto.Mac;
|
||||
import javax.crypto.spec.SecretKeySpec;
|
||||
|
||||
/**
|
||||
* Verifies grants with a shared HMAC-SHA256 issuing key.
|
||||
*
|
||||
* <p>A shared secret rather than a public-key signature is a deliberate scope choice: the issuer is
|
||||
* the change-management system operated by the same organisation as the runtime, so key
|
||||
* distribution is already solved and the property that matters — that an operator cannot mint an
|
||||
* approval for themselves — holds as long as the key is not on the operator's machine. A public-key
|
||||
* verifier can be substituted by implementing {@link ApprovalVerifier}; nothing else in the
|
||||
* platform depends on how the signature is produced.
|
||||
*
|
||||
* <p>Comparison is constant-time. A verifier that compares with {@code equals} leaks the signature
|
||||
* one byte at a time to a caller who can retry, which for an offline attacker with unlimited
|
||||
* attempts is the whole secret.
|
||||
*/
|
||||
public final class HmacApprovalVerifier implements ApprovalVerifier {
|
||||
|
||||
private static final String ALGORITHM = "HmacSHA256";
|
||||
|
||||
private final byte[] issuingKey;
|
||||
|
||||
/**
|
||||
* Creates a verifier over the issuing key.
|
||||
*
|
||||
* @param issuingKey the shared secret the approval issuer signs with
|
||||
*/
|
||||
public HmacApprovalVerifier(byte[] issuingKey) {
|
||||
Objects.requireNonNull(issuingKey, "issuingKey must not be null");
|
||||
if (issuingKey.length < 32) {
|
||||
throw new IllegalArgumentException(
|
||||
"the approval issuing key must be at least 256 bits; a shorter key makes the signature"
|
||||
+ " brute-forceable offline");
|
||||
}
|
||||
this.issuingKey = issuingKey.clone();
|
||||
}
|
||||
|
||||
/**
|
||||
* Produces the signature an issuer would attach to a grant.
|
||||
*
|
||||
* <p>Present because the issuer and the verifier must agree on the canonical form exactly, and a
|
||||
* second implementation of that agreement is a second thing to drift. Holding this key is what
|
||||
* makes a caller an issuer — it is not, and must not become, available to the runtime that
|
||||
* executes operations.
|
||||
*
|
||||
* @param grant the grant to sign
|
||||
* @return the hexadecimal signature
|
||||
*/
|
||||
public String sign(ApprovalGrant grant) {
|
||||
Objects.requireNonNull(grant, "grant must not be null");
|
||||
return HexFormat.of().formatHex(mac(grant.canonicalForm()));
|
||||
}
|
||||
|
||||
@Override
|
||||
public VerifiedApproval verify(
|
||||
ApprovalGrant grant, String signature, PlanDigest executingDigest, Instant now) {
|
||||
Objects.requireNonNull(grant, "grant must not be null");
|
||||
Objects.requireNonNull(signature, "signature must not be null");
|
||||
Objects.requireNonNull(executingDigest, "executingDigest must not be null");
|
||||
Objects.requireNonNull(now, "now must not be null");
|
||||
|
||||
if (!grant.planDigest().equals(executingDigest)) {
|
||||
// The signature would verify: this is a genuine approval, for a different plan. Replaying one
|
||||
// approval across plans is the attack the digest binding exists to stop.
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_PLAN_MISMATCH",
|
||||
"approval %s was issued for plan %s but the plan about to execute is %s"
|
||||
.formatted(
|
||||
grant.approval().ticket(), grant.planDigest().value(), executingDigest.value()));
|
||||
}
|
||||
if (!grant.isValidAt(now)) {
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_EXPIRED",
|
||||
"approval %s is not valid at %s".formatted(grant.approval().ticket(), now));
|
||||
}
|
||||
byte[] presented;
|
||||
try {
|
||||
presented = HexFormat.of().parseHex(signature);
|
||||
} catch (IllegalArgumentException malformed) {
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_SIGNATURE_INVALID", "the approval signature is not hexadecimal");
|
||||
}
|
||||
if (!MessageDigest.isEqual(mac(grant.canonicalForm()), presented)) {
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_SIGNATURE_INVALID",
|
||||
("approval %s does not verify against the issuing key; the grant was altered after it"
|
||||
+ " was signed, or it was never issued")
|
||||
.formatted(grant.approval().ticket()));
|
||||
}
|
||||
return VerifiedApproval.of(grant);
|
||||
}
|
||||
|
||||
private byte[] mac(String canonicalForm) {
|
||||
try {
|
||||
Mac mac = Mac.getInstance(ALGORITHM);
|
||||
mac.init(new SecretKeySpec(issuingKey, ALGORITHM));
|
||||
return mac.doFinal(canonicalForm.getBytes(StandardCharsets.UTF_8));
|
||||
} catch (java.security.GeneralSecurityException failure) {
|
||||
throw new IllegalStateException("HMAC-SHA256 is required of every Java runtime", failure);
|
||||
}
|
||||
}
|
||||
}
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
package dev.caskeleton.messaging.admin;
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.MessageDigest;
|
||||
import java.security.NoSuchAlgorithmException;
|
||||
import java.util.HexFormat;
|
||||
import java.util.Objects;
|
||||
|
||||
/**
|
||||
* The fingerprint of exactly what an approver agreed to.
|
||||
*
|
||||
* <p>An approval used to name a ticket and a window and nothing else, so it authorised any plan
|
||||
* presented within that window. The operator who got a redrive of one dead-letter destination
|
||||
* approved could execute a redrive of a different one with the same ticket, and every audit record
|
||||
* would look correct. Binding the signature to this digest is what makes an approval about a plan
|
||||
* rather than about a period of time.
|
||||
*
|
||||
* @param value the lowercase hexadecimal SHA-256 of the plan's canonical form
|
||||
*/
|
||||
public record PlanDigest(String value) {
|
||||
|
||||
/** Validates the digest shape. */
|
||||
public PlanDigest {
|
||||
Objects.requireNonNull(value, "value must not be null");
|
||||
if (!value.matches("[0-9a-f]{64}")) {
|
||||
throw new IllegalArgumentException("a plan digest is 64 lowercase hexadecimal characters");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Digests a plan's canonical form.
|
||||
*
|
||||
* @param canonicalForm the plan rendered deterministically, field by field
|
||||
* @return the digest
|
||||
*/
|
||||
public static PlanDigest ofCanonical(String canonicalForm) {
|
||||
Objects.requireNonNull(canonicalForm, "canonicalForm must not be null");
|
||||
try {
|
||||
byte[] digest =
|
||||
MessageDigest.getInstance("SHA-256")
|
||||
.digest(canonicalForm.getBytes(StandardCharsets.UTF_8));
|
||||
return new PlanDigest(HexFormat.of().formatHex(digest));
|
||||
} catch (NoSuchAlgorithmException impossible) {
|
||||
throw new IllegalStateException("SHA-256 is required of every Java runtime", impossible);
|
||||
}
|
||||
}
|
||||
}
|
||||
+22
@@ -39,6 +39,28 @@ public record RedrivePlan(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The fingerprint an approver's signature is bound to.
|
||||
*
|
||||
* <p>Source, target, batch size, and both candidate counts are in it, so an approval for a
|
||||
* redrive of one dead-letter destination cannot be presented for a redrive of another.
|
||||
*
|
||||
* @return the digest of this exact plan
|
||||
*/
|
||||
public PlanDigest digest() {
|
||||
return PlanDigest.ofCanonical(
|
||||
String.join(
|
||||
"|",
|
||||
"REDRIVE",
|
||||
request.redriveId().toString(),
|
||||
request.source().value(),
|
||||
request.target().value(),
|
||||
Integer.toString(request.batchSize()),
|
||||
Integer.toString(candidates),
|
||||
Integer.toString(alreadyRedrivenCandidates),
|
||||
topologyVersion));
|
||||
}
|
||||
|
||||
/**
|
||||
* Reports whether this redrive would replay messages that already failed a redrive.
|
||||
*
|
||||
|
||||
+24
@@ -39,6 +39,30 @@ public record ReplayPlan(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The fingerprint an approver's signature is bound to.
|
||||
*
|
||||
* <p>Every field that changes what would happen is in it. An approval signed over this digest
|
||||
* cannot be presented for a plan that reads a different destination, targets a different consumer
|
||||
* group, starts from a different position, or estimates a different impact.
|
||||
*
|
||||
* @return the digest of this exact plan
|
||||
*/
|
||||
public PlanDigest digest() {
|
||||
return PlanDigest.ofCanonical(
|
||||
String.join(
|
||||
"|",
|
||||
"REPLAY",
|
||||
request.replayId().toString(),
|
||||
request.destination().value(),
|
||||
Boolean.toString(request.isolatedConsumerGroup()),
|
||||
request.from().toString(),
|
||||
request.to().map(Object::toString).orElse(""),
|
||||
Long.toString(estimatedMessages),
|
||||
topologyVersion,
|
||||
Boolean.toString(targetsLiveConsumerGroup)));
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns a one-line operator-facing summary of the impact.
|
||||
*
|
||||
|
||||
+88
@@ -0,0 +1,88 @@
|
||||
package dev.caskeleton.messaging.admin;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.Objects;
|
||||
|
||||
/**
|
||||
* A grant that an {@link ApprovalVerifier} has checked, and that nobody else can manufacture.
|
||||
*
|
||||
* <p>The approved-plan types used to hold a plain {@code AdminApproval} record with a public
|
||||
* constructor, so "this plan was approved" was a claim the caller made about itself. Any code that
|
||||
* could reach the execute method could write {@code new AdminApproval("TICKET-1", "someone", now,
|
||||
* later)} and the platform believed it.
|
||||
*
|
||||
* <p>This class has no public constructor and no public factory. The only way to obtain one is to
|
||||
* present a grant and a signature to a verifier, which is exactly the property the type is for:
|
||||
* holding a {@code VerifiedApproval} is itself the evidence that verification happened.
|
||||
*/
|
||||
public final class VerifiedApproval {
|
||||
|
||||
private final ApprovalGrant grant;
|
||||
|
||||
private VerifiedApproval(ApprovalGrant grant) {
|
||||
this.grant = Objects.requireNonNull(grant, "grant must not be null");
|
||||
}
|
||||
|
||||
/**
|
||||
* Mints a verified approval. Package-private on purpose: only a verifier in this package may call
|
||||
* it, so the type cannot be forged from application or adapter code.
|
||||
*/
|
||||
static VerifiedApproval of(ApprovalGrant grant) {
|
||||
return new VerifiedApproval(grant);
|
||||
}
|
||||
|
||||
/**
|
||||
* The grant that was verified.
|
||||
*
|
||||
* @return the grant
|
||||
*/
|
||||
public ApprovalGrant grant() {
|
||||
return grant;
|
||||
}
|
||||
|
||||
/**
|
||||
* The approval ticket, for audit records and journal keys.
|
||||
*
|
||||
* @return the ticket
|
||||
*/
|
||||
public String ticket() {
|
||||
return grant.approval().ticket();
|
||||
}
|
||||
|
||||
/**
|
||||
* Who approved the operation.
|
||||
*
|
||||
* @return the approver identity
|
||||
*/
|
||||
public String approvedBy() {
|
||||
return grant.approval().approvedBy();
|
||||
}
|
||||
|
||||
/**
|
||||
* Reports whether the approval window is still open.
|
||||
*
|
||||
* @param now the current instant
|
||||
* @return true while inside the window
|
||||
*/
|
||||
public boolean isValidAt(Instant now) {
|
||||
return grant.isValidAt(now);
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean equals(Object other) {
|
||||
return other instanceof VerifiedApproval verified && grant.equals(verified.grant);
|
||||
}
|
||||
|
||||
@Override
|
||||
public int hashCode() {
|
||||
return grant.hashCode();
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
// Deliberately not the whole grant: this string reaches logs, and the plan digest plus the
|
||||
// ticket identify the operation without restating who approved what to an audience that has
|
||||
// not been authorised to read it.
|
||||
return "VerifiedApproval[" + ticket() + " " + grant.planDigest().value().substring(0, 12) + "]";
|
||||
}
|
||||
}
|
||||
+35
-4
@@ -17,8 +17,35 @@ class DestructiveOperationGuardTest {
|
||||
private static final Instant NOW = Instant.parse("2026-08-10T09:15:00Z");
|
||||
private static final DestinationName ORDERS = new DestinationName("order-events");
|
||||
|
||||
private static final AdminApproval VALID =
|
||||
new AdminApproval("CHG-1001", "operator", NOW.minusSeconds(60), NOW.plusSeconds(3600));
|
||||
private static final HmacApprovalVerifier ISSUER =
|
||||
new HmacApprovalVerifier(
|
||||
"an-issuing-key-of-at-least-32-bytes!".getBytes(java.nio.charset.StandardCharsets.UTF_8));
|
||||
|
||||
private static final VerifiedApproval VALID =
|
||||
verified(DestructiveOperation.REPLAY, ORDERS, NOW.minusSeconds(60), NOW.plusSeconds(3600));
|
||||
|
||||
/**
|
||||
* A grant the issuer signed, verified the way the runtime verifies it.
|
||||
*
|
||||
* <p>The guard used to take a plain {@code AdminApproval} record, which any caller could
|
||||
* construct. Every case below now has to go through a real signature to get an approval object at
|
||||
* all, which is the property the type change was made for.
|
||||
*/
|
||||
private static VerifiedApproval verified(
|
||||
DestructiveOperation operation, DestinationName source, Instant from, Instant until) {
|
||||
PlanDigest digest = PlanDigest.ofCanonical(operation + "|" + source.value());
|
||||
ApprovalGrant grant =
|
||||
new ApprovalGrant(
|
||||
new AdminApproval("CHG-1001", "approver", from, until),
|
||||
"operator",
|
||||
operation,
|
||||
source,
|
||||
Optional.empty(),
|
||||
"v1",
|
||||
Long.MAX_VALUE,
|
||||
digest);
|
||||
return ISSUER.verify(grant, ISSUER.sign(grant), digest, from);
|
||||
}
|
||||
|
||||
@Test
|
||||
void anApplicationRuntimeCannotRedrive() {
|
||||
@@ -45,8 +72,12 @@ class DestructiveOperationGuardTest {
|
||||
@Test
|
||||
void anExpiredApprovalDoesNotAuthorise() {
|
||||
DestructiveOperationGuard guard = new DestructiveOperationGuard(true);
|
||||
AdminApproval expired =
|
||||
new AdminApproval("CHG-1000", "operator", NOW.minusSeconds(7200), NOW.minusSeconds(60));
|
||||
VerifiedApproval expired =
|
||||
verified(
|
||||
DestructiveOperation.OFFSET_RESET,
|
||||
ORDERS,
|
||||
NOW.minusSeconds(7200),
|
||||
NOW.minusSeconds(60));
|
||||
|
||||
assertThatThrownBy(
|
||||
() ->
|
||||
|
||||
@@ -32,9 +32,9 @@ commons-io:commons-io:2.21.0=spotbugs
|
||||
info.picocli:picocli:4.7.7=checkstyle
|
||||
io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnotationProcessor
|
||||
io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor
|
||||
io.micrometer:micrometer-commons:1.16.0=runtimeClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-core:1.16.0=runtimeClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-observation:1.16.0=runtimeClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-commons:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-core:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-observation:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
javax.inject:javax.inject:1=annotationProcessor,testAnnotationProcessor
|
||||
jaxen:jaxen:2.0.0=spotbugs
|
||||
net.bytebuddy:byte-buddy:1.17.8=testCompileClasspath,testRuntimeClasspath
|
||||
@@ -62,7 +62,7 @@ org.codehaus.plexus:plexus-utils:3.3.0=checkstyle
|
||||
org.dom4j:dom4j:2.2.0=spotbugs
|
||||
org.hdrhistogram:HdrHistogram:2.2.2=runtimeClasspath,testRuntimeClasspath
|
||||
org.javassist:javassist:3.28.0-GA=checkstyle
|
||||
org.jspecify:jspecify:1.0.0=annotationProcessor,checkstyle,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
|
||||
org.jspecify:jspecify:1.0.0=annotationProcessor,checkstyle,compileClasspath,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.jupiter:junit-jupiter-api:6.0.1=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.jupiter:junit-jupiter-engine:6.0.1=testRuntimeClasspath
|
||||
org.junit.jupiter:junit-jupiter-params:6.0.1=testCompileClasspath,testRuntimeClasspath
|
||||
@@ -85,4 +85,4 @@ org.reflections:reflections:0.10.2=checkstyle
|
||||
org.slf4j:slf4j-api:2.0.17=spotbugs,spotbugsSlf4j
|
||||
org.slf4j:slf4j-simple:2.0.17=checkstyle,spotbugsSlf4j
|
||||
org.xmlresolver:xmlresolver:5.3.3=checkstyle,spotbugs
|
||||
empty=compileClasspath
|
||||
empty=
|
||||
|
||||
-78
@@ -1,78 +0,0 @@
|
||||
package dev.caskeleton.messaging.admin.runtime;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
|
||||
/**
|
||||
* Remembers which approvals have already been executed.
|
||||
*
|
||||
* <p>An approval authorises one execution, not a standing permission. Without this, re-running the
|
||||
* same approved redrive twice is a single command-history arrow-up away — and the second run
|
||||
* republishes messages the first one already moved, which on a destination without an inbox is
|
||||
* indistinguishable from a duplicate storm.
|
||||
*
|
||||
* <p>Claiming is atomic and returns the previous claim rather than a boolean, so a duplicate
|
||||
* attempt can tell the operator <em>when</em> it ran and by which operation id instead of just
|
||||
* refusing.
|
||||
*/
|
||||
public final class AdminOperationIdempotencyStore {
|
||||
|
||||
private final Map<String, Claim> claims = new ConcurrentHashMap<>();
|
||||
|
||||
/**
|
||||
* One recorded execution of an approval.
|
||||
*
|
||||
* @param approvalTicket the approval that was executed
|
||||
* @param operationId the operation identity that claimed it
|
||||
* @param executedAt when it ran
|
||||
*/
|
||||
public record Claim(String approvalTicket, String operationId, Instant executedAt) {
|
||||
|
||||
public Claim {
|
||||
Objects.requireNonNull(executedAt, "executedAt must not be null");
|
||||
if (approvalTicket == null || approvalTicket.isBlank()) {
|
||||
throw new IllegalArgumentException("approvalTicket must not be blank");
|
||||
}
|
||||
if (operationId == null || operationId.isBlank()) {
|
||||
throw new IllegalArgumentException("operationId must not be blank");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Claims an approval for execution.
|
||||
*
|
||||
* @param approvalTicket the approval to claim
|
||||
* @param operationId the operation attempting it
|
||||
* @param now the current instant
|
||||
* @return empty when the claim succeeded; the existing claim when it was already executed
|
||||
*/
|
||||
public Optional<Claim> claim(String approvalTicket, String operationId, Instant now) {
|
||||
Objects.requireNonNull(now, "now must not be null");
|
||||
Claim candidate = new Claim(approvalTicket, operationId, now);
|
||||
Claim existing = claims.putIfAbsent(approvalTicket, candidate);
|
||||
return Optional.ofNullable(existing);
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the recorded execution of an approval.
|
||||
*
|
||||
* @param approvalTicket the approval
|
||||
* @return the claim, when the approval has been executed
|
||||
*/
|
||||
public Optional<Claim> find(String approvalTicket) {
|
||||
return Optional.ofNullable(claims.get(approvalTicket));
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns how many approvals have been executed.
|
||||
*
|
||||
* @return the claim count
|
||||
*/
|
||||
public int size() {
|
||||
return claims.size();
|
||||
}
|
||||
}
|
||||
+85
-27
@@ -1,5 +1,7 @@
|
||||
package dev.caskeleton.messaging.admin.runtime;
|
||||
|
||||
import dev.caskeleton.messaging.admin.AdminOperationJournal;
|
||||
import dev.caskeleton.messaging.admin.AdminOperationLease;
|
||||
import dev.caskeleton.messaging.admin.ApprovedRedrivePlan;
|
||||
import dev.caskeleton.messaging.admin.ApprovedReplayPlan;
|
||||
import dev.caskeleton.messaging.admin.RedrivePlan;
|
||||
@@ -10,7 +12,6 @@ import dev.caskeleton.messaging.admin.ReplayRequest;
|
||||
import dev.caskeleton.messaging.admin.ReplayResult;
|
||||
import dev.caskeleton.messaging.admin.TopologyManifest;
|
||||
import dev.caskeleton.messaging.admin.TopologyValidationReport;
|
||||
import dev.caskeleton.messaging.api.error.MessageAuthorizationException;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
@@ -30,15 +31,29 @@ import java.util.function.Supplier;
|
||||
* <li>Only then does anything move.
|
||||
* </ol>
|
||||
*
|
||||
* <p>The idempotency claim comes <em>before</em> the work rather than after it. Claiming afterwards
|
||||
* leaves a window where a second execution starts while the first is still running, which is
|
||||
* precisely the double-redrive this store exists to prevent.
|
||||
* <p>The journal entry is written <em>before</em> the work rather than after it. Writing it
|
||||
* afterwards leaves a window where a second execution starts while the first is still running,
|
||||
* which is precisely the double-redrive the journal exists to prevent.
|
||||
*
|
||||
* <p>Writing it first used to have a cost the previous store never paid: an operation that died
|
||||
* halfway had consumed its approval and left no record of how far it got. The journal keeps a
|
||||
* checkpoint and hands back a lease that says where to resume, so a retry continues the same
|
||||
* operation instead of either redoing it or requiring a new approval.
|
||||
*/
|
||||
public final class DefaultMessagingAdminService implements MessagingAdminService {
|
||||
|
||||
/**
|
||||
* How long a runtime holds an operation before another may take it over.
|
||||
*
|
||||
* <p>Long enough that a slow batch does not lose its lease mid-flight, short enough that a dead
|
||||
* replica does not park an approval for an hour.
|
||||
*/
|
||||
private static final Duration LEASE_DURATION = Duration.ofMinutes(5);
|
||||
|
||||
private final CompositeTopologyValidator topologyValidator;
|
||||
private final BrokerTopologyInspector inspector;
|
||||
private final AdminOperationIdempotencyStore idempotency;
|
||||
private final AdminOperationJournal journal;
|
||||
private final String leaseOwner;
|
||||
private final ReplayService replayService;
|
||||
private final RedriveService redriveService;
|
||||
private final Supplier<List<TopologyManifest>> manifests;
|
||||
@@ -51,7 +66,8 @@ public final class DefaultMessagingAdminService implements MessagingAdminService
|
||||
*
|
||||
* @param topologyValidator compares declared topology against the broker
|
||||
* @param inspector reads the broker's topology version
|
||||
* @param idempotency records which approvals have been executed
|
||||
* @param journal records which approvals have been executed, and how far each got
|
||||
* @param leaseOwner identifies this runtime in the journal's leases
|
||||
* @param replayService performs replays
|
||||
* @param redriveService performs redrives
|
||||
* @param manifests supplies the declared topologies
|
||||
@@ -62,7 +78,8 @@ public final class DefaultMessagingAdminService implements MessagingAdminService
|
||||
public DefaultMessagingAdminService(
|
||||
CompositeTopologyValidator topologyValidator,
|
||||
BrokerTopologyInspector inspector,
|
||||
AdminOperationIdempotencyStore idempotency,
|
||||
AdminOperationJournal journal,
|
||||
String leaseOwner,
|
||||
ReplayService replayService,
|
||||
RedriveService redriveService,
|
||||
Supplier<List<TopologyManifest>> manifests,
|
||||
@@ -72,7 +89,11 @@ public final class DefaultMessagingAdminService implements MessagingAdminService
|
||||
this.topologyValidator =
|
||||
Objects.requireNonNull(topologyValidator, "topologyValidator required");
|
||||
this.inspector = Objects.requireNonNull(inspector, "inspector must not be null");
|
||||
this.idempotency = Objects.requireNonNull(idempotency, "idempotency must not be null");
|
||||
this.journal = Objects.requireNonNull(journal, "journal must not be null");
|
||||
if (leaseOwner == null || leaseOwner.isBlank()) {
|
||||
throw new IllegalArgumentException("leaseOwner must not be blank");
|
||||
}
|
||||
this.leaseOwner = leaseOwner;
|
||||
this.replayService = Objects.requireNonNull(replayService, "replayService must not be null");
|
||||
this.redriveService = Objects.requireNonNull(redriveService, "redriveService must not be null");
|
||||
this.manifests = Objects.requireNonNull(manifests, "manifests must not be null");
|
||||
@@ -104,12 +125,28 @@ public final class DefaultMessagingAdminService implements MessagingAdminService
|
||||
ReplayRequest request = plan.plan().request();
|
||||
|
||||
plan.requireExecutable(now, inspector.topologyVersion());
|
||||
claimOrRefuse(plan.approval().ticket(), request.replayId().toString(), now);
|
||||
AdminOperationLease lease =
|
||||
journal.begin(
|
||||
plan.approval().ticket(),
|
||||
plan.plan().digest(),
|
||||
request.replayId().toString(),
|
||||
leaseOwner,
|
||||
LEASE_DURATION,
|
||||
now);
|
||||
|
||||
Instant startedAt = clock.get();
|
||||
ReplayReport report =
|
||||
replayService.replay(
|
||||
request, Optional.of(plan.approval()), plan.approval().approvedBy(), now);
|
||||
ReplayReport report;
|
||||
try {
|
||||
report =
|
||||
replayService.replay(
|
||||
request, Optional.of(plan.approval()), plan.approval().approvedBy(), now);
|
||||
} catch (RuntimeException failure) {
|
||||
// The operation stays resumable rather than silently consuming the approval: the journal
|
||||
// entry moves to FAILED at its checkpoint, and a retry takes it over from there.
|
||||
journal.fail(lease, lease.resumeFrom(), failureCodeOf(failure), clock.get());
|
||||
throw failure;
|
||||
}
|
||||
journal.complete(lease, Math.toIntExact(report.replayed()), clock.get());
|
||||
|
||||
return new ReplayResult(
|
||||
request.replayId(),
|
||||
@@ -139,12 +176,31 @@ public final class DefaultMessagingAdminService implements MessagingAdminService
|
||||
RedriveRequest request = plan.plan().request();
|
||||
|
||||
plan.requireExecutable(now, inspector.topologyVersion());
|
||||
claimOrRefuse(plan.approval().ticket(), request.redriveId().toString(), now);
|
||||
AdminOperationLease lease =
|
||||
journal.begin(
|
||||
plan.approval().ticket(),
|
||||
plan.plan().digest(),
|
||||
request.redriveId().toString(),
|
||||
leaseOwner,
|
||||
LEASE_DURATION,
|
||||
now);
|
||||
|
||||
Instant startedAt = clock.get();
|
||||
RedriveReport report =
|
||||
redriveService.redrive(
|
||||
request, Optional.of(plan.approval()), plan.approval().approvedBy(), now);
|
||||
RedriveReport report;
|
||||
try {
|
||||
report =
|
||||
redriveService.redrive(
|
||||
request,
|
||||
Optional.of(plan.approval()),
|
||||
plan.approval().approvedBy(),
|
||||
now,
|
||||
lease.resumeFrom(),
|
||||
completed -> journal.checkpoint(lease, completed, LEASE_DURATION, clock.get()));
|
||||
} catch (RuntimeException failure) {
|
||||
journal.fail(lease, lease.resumeFrom(), failureCodeOf(failure), clock.get());
|
||||
throw failure;
|
||||
}
|
||||
journal.complete(lease, report.moved() + report.failed(), clock.get());
|
||||
|
||||
return new RedriveResult(
|
||||
request.redriveId(),
|
||||
@@ -155,17 +211,19 @@ public final class DefaultMessagingAdminService implements MessagingAdminService
|
||||
report.dryRun());
|
||||
}
|
||||
|
||||
private void claimOrRefuse(String approvalTicket, String operationId, Instant now) {
|
||||
idempotency
|
||||
.claim(approvalTicket, operationId, now)
|
||||
.ifPresent(
|
||||
existing -> {
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_ALREADY_EXECUTED",
|
||||
"approval %s was already executed at %s by operation %s; an approval authorises "
|
||||
.formatted(approvalTicket, existing.executedAt(), existing.operationId())
|
||||
+ "one execution, not a standing permission");
|
||||
});
|
||||
/**
|
||||
* Reduces a failure to a bounded, sanitised code for the journal.
|
||||
*
|
||||
* <p>The journal is read by operators during incidents and its contents outlive the process. A
|
||||
* raw exception message can carry a destination, a payload fragment, or a credential from a
|
||||
* driver's own error text, so only the platform's own code or the exception's simple name goes
|
||||
* in.
|
||||
*/
|
||||
private static String failureCodeOf(RuntimeException failure) {
|
||||
if (failure instanceof dev.caskeleton.messaging.api.error.MessagingException messaging) {
|
||||
return messaging.failure().code();
|
||||
}
|
||||
return failure.getClass().getSimpleName();
|
||||
}
|
||||
|
||||
/** Estimates how many messages a replay would re-deliver. */
|
||||
|
||||
+237
@@ -0,0 +1,237 @@
|
||||
package dev.caskeleton.messaging.admin.runtime;
|
||||
|
||||
import dev.caskeleton.messaging.admin.AdminOperationJournal;
|
||||
import dev.caskeleton.messaging.admin.AdminOperationLease;
|
||||
import dev.caskeleton.messaging.admin.AdminOperationRecord;
|
||||
import dev.caskeleton.messaging.admin.AdminOperationState;
|
||||
import dev.caskeleton.messaging.admin.PlanDigest;
|
||||
import dev.caskeleton.messaging.api.error.MessageAuthorizationException;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
|
||||
/**
|
||||
* A single-process journal, for tests and for local development.
|
||||
*
|
||||
* <p>It reports {@link #isDurable()} as false, and the starter refuses to run a production profile
|
||||
* on a journal that says so. That declaration is the point of this class existing at all: the
|
||||
* previous in-memory store was registered as the production default and nothing distinguished it
|
||||
* from a shared one, so the gap was invisible until two replicas executed the same approval.
|
||||
*
|
||||
* <p>The semantics are otherwise the real ones — uniqueness on {@code (ticket, digest)}, lease
|
||||
* takeover with a monotonic token, resume from checkpoint — so a test that passes here is testing
|
||||
* the protocol rather than a simplification of it.
|
||||
*/
|
||||
public final class InMemoryAdminOperationJournal implements AdminOperationJournal {
|
||||
|
||||
private final Map<String, AdminOperationRecord> records = new ConcurrentHashMap<>();
|
||||
|
||||
@Override
|
||||
public AdminOperationLease begin(
|
||||
String approvalTicket,
|
||||
PlanDigest planDigest,
|
||||
String operationId,
|
||||
String leaseOwner,
|
||||
Duration leaseDuration,
|
||||
Instant now) {
|
||||
Objects.requireNonNull(leaseDuration, "leaseDuration must not be null");
|
||||
Objects.requireNonNull(now, "now must not be null");
|
||||
AdminOperationRecord claimed =
|
||||
records.compute(
|
||||
key(approvalTicket, planDigest),
|
||||
(ignored, existing) ->
|
||||
claim(
|
||||
existing,
|
||||
approvalTicket,
|
||||
planDigest,
|
||||
operationId,
|
||||
leaseOwner,
|
||||
leaseDuration,
|
||||
now));
|
||||
return new AdminOperationLease(
|
||||
approvalTicket,
|
||||
planDigest,
|
||||
claimed.operationId(),
|
||||
claimed.leaseOwner(),
|
||||
claimed.leaseToken(),
|
||||
claimed.itemsCompleted());
|
||||
}
|
||||
|
||||
private static AdminOperationRecord claim(
|
||||
AdminOperationRecord existing,
|
||||
String approvalTicket,
|
||||
PlanDigest planDigest,
|
||||
String operationId,
|
||||
String leaseOwner,
|
||||
Duration leaseDuration,
|
||||
Instant now) {
|
||||
if (existing == null) {
|
||||
return new AdminOperationRecord(
|
||||
approvalTicket,
|
||||
planDigest,
|
||||
operationId,
|
||||
AdminOperationState.STARTED,
|
||||
0,
|
||||
leaseOwner,
|
||||
1L,
|
||||
now.plus(leaseDuration),
|
||||
now,
|
||||
Optional.empty(),
|
||||
Optional.empty());
|
||||
}
|
||||
if (existing.state() == AdminOperationState.COMPLETED) {
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_ALREADY_EXECUTED",
|
||||
("approval %s was already executed to completion at %s; an approval authorises one"
|
||||
+ " execution, not a standing permission")
|
||||
.formatted(approvalTicket, existing.finishedAt().orElse(existing.startedAt())));
|
||||
}
|
||||
if (existing.state() == AdminOperationState.STARTED
|
||||
&& now.isBefore(existing.leaseExpiresAt())) {
|
||||
throw new MessageAuthorizationException(
|
||||
"ADMIN_OPERATION_IN_FLIGHT",
|
||||
("approval %s is being executed by %s until at least %s; two runtimes executing one"
|
||||
+ " approval is a duplicate storm, not a faster redrive")
|
||||
.formatted(approvalTicket, existing.leaseOwner(), existing.leaseExpiresAt()));
|
||||
}
|
||||
// Either the previous attempt failed, or its lease expired without a completion. Both resume
|
||||
// from the checkpoint under a new token, which fences the previous holder out.
|
||||
return new AdminOperationRecord(
|
||||
approvalTicket,
|
||||
planDigest,
|
||||
operationId,
|
||||
AdminOperationState.STARTED,
|
||||
existing.itemsCompleted(),
|
||||
leaseOwner,
|
||||
existing.leaseToken() + 1,
|
||||
now.plus(leaseDuration),
|
||||
existing.startedAt(),
|
||||
Optional.empty(),
|
||||
Optional.empty());
|
||||
}
|
||||
|
||||
@Override
|
||||
public void checkpoint(
|
||||
AdminOperationLease lease, int itemsCompleted, Duration leaseDuration, Instant now) {
|
||||
update(
|
||||
lease,
|
||||
current ->
|
||||
new AdminOperationRecord(
|
||||
current.approvalTicket(),
|
||||
current.planDigest(),
|
||||
current.operationId(),
|
||||
AdminOperationState.STARTED,
|
||||
Math.max(current.itemsCompleted(), itemsCompleted),
|
||||
current.leaseOwner(),
|
||||
current.leaseToken(),
|
||||
now.plus(leaseDuration),
|
||||
current.startedAt(),
|
||||
Optional.empty(),
|
||||
Optional.empty()));
|
||||
}
|
||||
|
||||
@Override
|
||||
public void complete(AdminOperationLease lease, int itemsCompleted, Instant now) {
|
||||
update(
|
||||
lease,
|
||||
current ->
|
||||
new AdminOperationRecord(
|
||||
current.approvalTicket(),
|
||||
current.planDigest(),
|
||||
current.operationId(),
|
||||
AdminOperationState.COMPLETED,
|
||||
Math.max(current.itemsCompleted(), itemsCompleted),
|
||||
current.leaseOwner(),
|
||||
current.leaseToken(),
|
||||
current.leaseExpiresAt(),
|
||||
current.startedAt(),
|
||||
Optional.of(now),
|
||||
Optional.empty()));
|
||||
}
|
||||
|
||||
@Override
|
||||
public void fail(AdminOperationLease lease, int itemsCompleted, String failureCode, Instant now) {
|
||||
Objects.requireNonNull(failureCode, "failureCode must not be null");
|
||||
update(
|
||||
lease,
|
||||
current ->
|
||||
new AdminOperationRecord(
|
||||
current.approvalTicket(),
|
||||
current.planDigest(),
|
||||
current.operationId(),
|
||||
AdminOperationState.FAILED,
|
||||
Math.max(current.itemsCompleted(), itemsCompleted),
|
||||
current.leaseOwner(),
|
||||
current.leaseToken(),
|
||||
current.leaseExpiresAt(),
|
||||
current.startedAt(),
|
||||
Optional.of(now),
|
||||
Optional.of(failureCode)));
|
||||
}
|
||||
|
||||
@Override
|
||||
public Optional<AdminOperationRecord> find(String approvalTicket, PlanDigest planDigest) {
|
||||
return Optional.ofNullable(records.get(key(approvalTicket, planDigest)));
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean isDurable() {
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* How many operations have been journalled.
|
||||
*
|
||||
* @return the record count
|
||||
*/
|
||||
public int size() {
|
||||
return records.size();
|
||||
}
|
||||
|
||||
private void update(
|
||||
AdminOperationLease lease, java.util.function.UnaryOperator<AdminOperationRecord> mutation) {
|
||||
Objects.requireNonNull(lease, "lease must not be null");
|
||||
records.compute(
|
||||
key(lease.approvalTicket(), lease.planDigest()),
|
||||
(ignored, current) -> {
|
||||
if (current == null) {
|
||||
throw new MessageAuthorizationException(
|
||||
"ADMIN_OPERATION_NOT_JOURNALLED",
|
||||
"no journal entry exists for approval " + lease.approvalTicket());
|
||||
}
|
||||
if (current.leaseToken() != lease.leaseToken()) {
|
||||
// The fence. A stalled runtime that wakes up and writes here would otherwise overwrite
|
||||
// the progress of whichever replica took the operation over.
|
||||
throw new MessageAuthorizationException(
|
||||
"ADMIN_OPERATION_LEASE_LOST",
|
||||
("the lease on approval %s moved from token %d to token %d; this runtime no longer"
|
||||
+ " owns the operation")
|
||||
.formatted(lease.approvalTicket(), lease.leaseToken(), current.leaseToken()));
|
||||
}
|
||||
return mutation.apply(current);
|
||||
});
|
||||
}
|
||||
|
||||
private static String key(String approvalTicket, PlanDigest planDigest) {
|
||||
Objects.requireNonNull(planDigest, "planDigest must not be null");
|
||||
// Length-prefixed for the same reason the grant's canonical form is: a ticket containing the
|
||||
// separator must not be able to collide with a different ticket and digest pair.
|
||||
return String.join(
|
||||
"", Integer.toString(approvalTicket.length()), ":", approvalTicket, planDigest.value());
|
||||
}
|
||||
|
||||
/**
|
||||
* Every journalled operation, newest first by start time.
|
||||
*
|
||||
* @return the records
|
||||
*/
|
||||
public List<AdminOperationRecord> records() {
|
||||
return records.values().stream()
|
||||
.sorted(java.util.Comparator.comparing(AdminOperationRecord::startedAt).reversed())
|
||||
.toList();
|
||||
}
|
||||
}
|
||||
+80
-22
@@ -1,9 +1,9 @@
|
||||
package dev.caskeleton.messaging.admin.runtime;
|
||||
|
||||
import dev.caskeleton.messaging.admin.AdminApproval;
|
||||
import dev.caskeleton.messaging.admin.DestructiveOperation;
|
||||
import dev.caskeleton.messaging.admin.DestructiveOperationGuard;
|
||||
import dev.caskeleton.messaging.admin.RedriveRequest;
|
||||
import dev.caskeleton.messaging.admin.VerifiedApproval;
|
||||
import dev.caskeleton.messaging.api.MessageId;
|
||||
import dev.caskeleton.messaging.api.publish.PublishCompletion;
|
||||
import dev.caskeleton.messaging.api.publish.PublishResult;
|
||||
@@ -54,14 +54,46 @@ public final class RedriveService {
|
||||
* Runs one redrive pass.
|
||||
*
|
||||
* @param request what to move
|
||||
* @param approval the approval, when one was supplied
|
||||
* @param approval the verified approval, when one was supplied
|
||||
* @param subject the operator identity
|
||||
* @param now the current instant
|
||||
* @return what the pass did
|
||||
*/
|
||||
public RedriveReport redrive(
|
||||
RedriveRequest request, Optional<AdminApproval> approval, String subject, Instant now) {
|
||||
RedriveRequest request, Optional<VerifiedApproval> approval, String subject, Instant now) {
|
||||
return redrive(request, approval, subject, now, 0, completed -> {});
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs one redrive pass, resuming a previous attempt and reporting progress as it goes.
|
||||
*
|
||||
* <p>Three things were wrong with running this as a plain loop. A synchronous failure from the
|
||||
* publisher — a broker that refuses the connection rather than the message — propagated out of
|
||||
* the loop, so the remaining candidates were never attempted and the audit record was never
|
||||
* written: the operation left no trace of the items it had already moved. A retry then started
|
||||
* from the first candidate and republished them. And nothing bounded how many times one message
|
||||
* could be redriven.
|
||||
*
|
||||
* @param request what to move
|
||||
* @param approval the verified approval, when one was supplied
|
||||
* @param subject the operator identity
|
||||
* @param now the current instant
|
||||
* @param resumeFrom how many candidates a previous attempt durably moved
|
||||
* @param checkpoint called with the running completed count after each item
|
||||
* @return what the pass did
|
||||
*/
|
||||
public RedriveReport redrive(
|
||||
RedriveRequest request,
|
||||
Optional<VerifiedApproval> approval,
|
||||
String subject,
|
||||
Instant now,
|
||||
int resumeFrom,
|
||||
java.util.function.IntConsumer checkpoint) {
|
||||
Objects.requireNonNull(request, "request must not be null");
|
||||
Objects.requireNonNull(checkpoint, "checkpoint must not be null");
|
||||
if (resumeFrom < 0) {
|
||||
throw new IllegalArgumentException("resumeFrom must not be negative");
|
||||
}
|
||||
guard.authorize(
|
||||
DestructiveOperation.REDRIVE, request.source(), approval, request.dryRun(), now);
|
||||
|
||||
@@ -72,31 +104,57 @@ public final class RedriveService {
|
||||
|
||||
List<MessageId> moved = new ArrayList<>();
|
||||
int failed = 0;
|
||||
for (MessageId messageId : candidates) {
|
||||
PublishResult result = publisher.republish(messageId, request.target(), request.redriveId());
|
||||
if (result.completion() == PublishCompletion.CONFIRMED) {
|
||||
source.settle(request.source(), messageId);
|
||||
moved.add(messageId);
|
||||
} else {
|
||||
failed++;
|
||||
int completed = resumeFrom;
|
||||
try {
|
||||
// Everything before resumeFrom was moved and settled by the previous attempt. Re-publishing
|
||||
// it is not a retry, it is a duplicate.
|
||||
for (MessageId messageId :
|
||||
candidates.subList(Math.min(resumeFrom, candidates.size()), candidates.size())) {
|
||||
if (attempt(messageId, request)) {
|
||||
moved.add(messageId);
|
||||
} else {
|
||||
failed++;
|
||||
}
|
||||
completed++;
|
||||
checkpoint.accept(completed);
|
||||
}
|
||||
} finally {
|
||||
// In the finally block on purpose: an operation that dies partway must still leave a record
|
||||
// of what it moved, because that record is what the resumed attempt and the incident review
|
||||
// both read.
|
||||
audit.record(
|
||||
new dev.caskeleton.messaging.observation.MessagingAuditEvent(
|
||||
"REDRIVE",
|
||||
subject,
|
||||
request.source().value(),
|
||||
approval.map(VerifiedApproval::ticket).orElse("dry-run"),
|
||||
now,
|
||||
java.util.Map.of(
|
||||
"redriveId", request.redriveId().toString(),
|
||||
"resumedFrom", Integer.toString(resumeFrom),
|
||||
"moved", Integer.toString(moved.size()),
|
||||
"failed", Integer.toString(failed))));
|
||||
}
|
||||
|
||||
audit.record(
|
||||
new dev.caskeleton.messaging.observation.MessagingAuditEvent(
|
||||
"REDRIVE",
|
||||
subject,
|
||||
request.source().value(),
|
||||
approval.map(AdminApproval::ticket).orElse("dry-run"),
|
||||
now,
|
||||
java.util.Map.of(
|
||||
"redriveId", request.redriveId().toString(),
|
||||
"moved", Integer.toString(moved.size()),
|
||||
"failed", Integer.toString(failed))));
|
||||
|
||||
return new RedriveReport(candidates.size(), moved.size(), failed, false);
|
||||
}
|
||||
|
||||
private boolean attempt(MessageId messageId, RedriveRequest request) {
|
||||
PublishResult result;
|
||||
try {
|
||||
result = publisher.republish(messageId, request.target(), request.redriveId());
|
||||
} catch (RuntimeException failure) {
|
||||
// One message that cannot be republished is a failed item, not a failed pass. Letting it
|
||||
// propagate abandoned every candidate behind it.
|
||||
return false;
|
||||
}
|
||||
if (result.completion() != PublishCompletion.CONFIRMED) {
|
||||
return false;
|
||||
}
|
||||
source.settle(request.source(), messageId);
|
||||
return true;
|
||||
}
|
||||
|
||||
/** Reads and settles messages on a dead letter destination. */
|
||||
public interface RedriveSource {
|
||||
|
||||
|
||||
+4
-4
@@ -1,9 +1,9 @@
|
||||
package dev.caskeleton.messaging.admin.runtime;
|
||||
|
||||
import dev.caskeleton.messaging.admin.AdminApproval;
|
||||
import dev.caskeleton.messaging.admin.DestructiveOperation;
|
||||
import dev.caskeleton.messaging.admin.DestructiveOperationGuard;
|
||||
import dev.caskeleton.messaging.admin.ReplayRequest;
|
||||
import dev.caskeleton.messaging.admin.VerifiedApproval;
|
||||
import dev.caskeleton.messaging.observation.MessagingAuditEvent;
|
||||
import java.time.Instant;
|
||||
import java.util.Map;
|
||||
@@ -45,13 +45,13 @@ public final class ReplayService {
|
||||
* Runs one replay.
|
||||
*
|
||||
* @param request what to replay
|
||||
* @param approval the approval, when one was supplied
|
||||
* @param approval the verified approval, when one was supplied
|
||||
* @param subject the operator identity
|
||||
* @param now the current instant
|
||||
* @return what the replay did
|
||||
*/
|
||||
public ReplayReport replay(
|
||||
ReplayRequest request, Optional<AdminApproval> approval, String subject, Instant now) {
|
||||
ReplayRequest request, Optional<VerifiedApproval> approval, String subject, Instant now) {
|
||||
Objects.requireNonNull(request, "request must not be null");
|
||||
Objects.requireNonNull(approval, "approval must not be null");
|
||||
|
||||
@@ -74,7 +74,7 @@ public final class ReplayService {
|
||||
"REPLAY",
|
||||
subject,
|
||||
request.destination().value(),
|
||||
approval.map(AdminApproval::ticket).orElse("isolated"),
|
||||
approval.map(VerifiedApproval::ticket).orElse("isolated"),
|
||||
now,
|
||||
Map.of(
|
||||
"replayId", request.replayId().toString(),
|
||||
|
||||
+145
@@ -0,0 +1,145 @@
|
||||
package dev.caskeleton.messaging.admin.runtime;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatCode;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
import dev.caskeleton.messaging.admin.AdminOperationLease;
|
||||
import dev.caskeleton.messaging.admin.AdminOperationState;
|
||||
import dev.caskeleton.messaging.admin.PlanDigest;
|
||||
import dev.caskeleton.messaging.api.error.MessageAuthorizationException;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* The protocol the journal has to implement, proven against the in-memory one.
|
||||
*
|
||||
* <p>What it replaced recorded a single fact — "this approval was claimed" — before any work
|
||||
* happened, in a map. An operation that died halfway had spent its approval, left no record of what
|
||||
* it had already moved, and gave a retry no way to continue: either request a new approval and
|
||||
* republish everything, or leave the redrive half applied.
|
||||
*/
|
||||
class AdminOperationJournalTest {
|
||||
|
||||
private static final Instant NOW = Instant.parse("2026-08-10T09:00:00Z");
|
||||
private static final Duration LEASE = Duration.ofMinutes(5);
|
||||
private static final String TICKET = "CHG-1042";
|
||||
private static final PlanDigest DIGEST = PlanDigest.ofCanonical("REDRIVE|orders.v1.dlq|50");
|
||||
private static final PlanDigest OTHER_DIGEST = PlanDigest.ofCanonical("REDRIVE|payments.dlq|50");
|
||||
|
||||
private final InMemoryAdminOperationJournal journal = new InMemoryAdminOperationJournal();
|
||||
|
||||
@Test
|
||||
@DisplayName("two replicas racing for one approval produce exactly one lease")
|
||||
void twoReplicasRacingProduceExactlyOneLease() {
|
||||
journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
|
||||
|
||||
assertThatThrownBy(
|
||||
() -> journal.begin(TICKET, DIGEST, "op-2", "replica-b", LEASE, NOW.plusSeconds(1)))
|
||||
.as("both replicas executing one approval is a duplicate storm, not a faster redrive")
|
||||
.isInstanceOf(MessageAuthorizationException.class)
|
||||
.hasMessageContaining("replica-a");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a completed operation cannot be executed again")
|
||||
void aCompletedOperationCannotBeExecutedAgain() {
|
||||
AdminOperationLease lease = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
|
||||
journal.complete(lease, 50, NOW.plusSeconds(30));
|
||||
|
||||
assertThatThrownBy(
|
||||
() -> journal.begin(TICKET, DIGEST, "op-2", "replica-a", LEASE, NOW.plusSeconds(60)))
|
||||
.as("an arrow-up in the shell must not re-run an approved redrive")
|
||||
.isInstanceOf(MessageAuthorizationException.class)
|
||||
.hasMessageContaining("already executed");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a crashed run resumes from its checkpoint rather than starting over")
|
||||
void aCrashedRunResumesFromItsCheckpoint() {
|
||||
AdminOperationLease first = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
|
||||
journal.checkpoint(first, 30, LEASE, NOW.plusSeconds(10));
|
||||
// No fail() call: this replica died. The lease simply runs out.
|
||||
|
||||
AdminOperationLease resumed =
|
||||
journal.begin(TICKET, DIGEST, "op-1", "replica-b", LEASE, NOW.plus(Duration.ofMinutes(10)));
|
||||
|
||||
assertThat(resumed.resumeFrom())
|
||||
.as("the 30 already moved must not be republished")
|
||||
.isEqualTo(30);
|
||||
assertThat(resumed.leaseToken()).isGreaterThan(first.leaseToken());
|
||||
assertThat(resumed.isResumption()).isTrue();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a failed run stays resumable without waiting for the lease to expire")
|
||||
void aFailedRunStaysResumableImmediately() {
|
||||
AdminOperationLease first = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
|
||||
journal.checkpoint(first, 12, LEASE, NOW.plusSeconds(5));
|
||||
journal.fail(first, 12, "MESSAGE_BROKER_UNAVAILABLE", NOW.plusSeconds(6));
|
||||
|
||||
AdminOperationLease resumed =
|
||||
journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW.plusSeconds(7));
|
||||
|
||||
assertThat(resumed.resumeFrom()).isEqualTo(12);
|
||||
assertThat(journal.find(TICKET, DIGEST))
|
||||
.hasValueSatisfying(
|
||||
record -> assertThat(record.state()).isEqualTo(AdminOperationState.STARTED));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a runtime that lost its lease cannot write over the one that took over")
|
||||
void aRuntimeThatLostItsLeaseCannotWriteOverTheSuccessor() {
|
||||
AdminOperationLease stale = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
|
||||
journal.checkpoint(stale, 30, LEASE, NOW.plusSeconds(10));
|
||||
AdminOperationLease successor =
|
||||
journal.begin(TICKET, DIGEST, "op-1", "replica-b", LEASE, NOW.plus(Duration.ofMinutes(10)));
|
||||
journal.checkpoint(successor, 45, LEASE, NOW.plus(Duration.ofMinutes(11)));
|
||||
|
||||
assertThatThrownBy(() -> journal.complete(stale, 30, NOW.plus(Duration.ofMinutes(12))))
|
||||
.as("a stalled replica waking up must not mark the operation done at its own count")
|
||||
.isInstanceOf(MessageAuthorizationException.class)
|
||||
.hasMessageContaining("no longer owns");
|
||||
|
||||
assertThat(journal.find(TICKET, DIGEST))
|
||||
.hasValueSatisfying(
|
||||
record -> {
|
||||
assertThat(record.itemsCompleted()).isEqualTo(45);
|
||||
assertThat(record.state()).isEqualTo(AdminOperationState.STARTED);
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("the same ticket presented for a different plan is a different operation")
|
||||
void theSameTicketForADifferentPlanIsADifferentOperation() {
|
||||
journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
|
||||
|
||||
// Keyed on the ticket alone this would have been refused as a duplicate execution, which reads
|
||||
// as protection but is actually the wrong answer: it is a different plan, and it must be
|
||||
// refused by the approval binding rather than silently treated as the same operation.
|
||||
assertThatCode(() -> journal.begin(TICKET, OTHER_DIGEST, "op-2", "replica-a", LEASE, NOW))
|
||||
.doesNotThrowAnyException();
|
||||
assertThat(journal.size()).isEqualTo(2);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a checkpoint never moves backwards")
|
||||
void aCheckpointNeverMovesBackwards() {
|
||||
AdminOperationLease lease = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
|
||||
journal.checkpoint(lease, 40, LEASE, NOW.plusSeconds(10));
|
||||
journal.checkpoint(lease, 5, LEASE, NOW.plusSeconds(20));
|
||||
|
||||
assertThat(journal.find(TICKET, DIGEST))
|
||||
.hasValueSatisfying(record -> assertThat(record.itemsCompleted()).isEqualTo(40));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("the in-memory journal declares itself non-durable")
|
||||
void theInMemoryJournalDeclaresItselfNonDurable() {
|
||||
assertThat(journal.isDurable())
|
||||
.as("the starter refuses a production profile on a journal that says this")
|
||||
.isFalse();
|
||||
}
|
||||
}
|
||||
+231
@@ -0,0 +1,231 @@
|
||||
package dev.caskeleton.messaging.admin.runtime;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatCode;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
import dev.caskeleton.messaging.admin.AdminApproval;
|
||||
import dev.caskeleton.messaging.admin.ApprovalGrant;
|
||||
import dev.caskeleton.messaging.admin.ApprovedRedrivePlan;
|
||||
import dev.caskeleton.messaging.admin.DestructiveOperation;
|
||||
import dev.caskeleton.messaging.admin.HmacApprovalVerifier;
|
||||
import dev.caskeleton.messaging.admin.RedrivePlan;
|
||||
import dev.caskeleton.messaging.admin.RedriveRequest;
|
||||
import dev.caskeleton.messaging.admin.VerifiedApproval;
|
||||
import dev.caskeleton.messaging.api.destination.DestinationName;
|
||||
import dev.caskeleton.messaging.api.error.MessageAuthorizationException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.Optional;
|
||||
import java.util.UUID;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* Whether "this was approved" is a fact or a claim the caller makes about itself.
|
||||
*
|
||||
* <p>It was a claim. {@code AdminApproval} was a public record and {@code ApprovedRedrivePlan} a
|
||||
* public record over it, so any code that could reach the execute method could write its own
|
||||
* approval and the platform believed it. Nothing bound an approval to an operation, a destination,
|
||||
* a topology version, or an impact ceiling either, so a genuine approval for one redrive authorised
|
||||
* a redrive of anything else inside its window.
|
||||
*/
|
||||
class ApprovalForgeryTest {
|
||||
|
||||
private static final Instant NOW = Instant.parse("2026-08-10T09:00:00Z");
|
||||
private static final UUID OPERATION_ID = UUID.fromString("0199aaaa-bbbb-7ccc-8ddd-eeeeffff0000");
|
||||
private static final DestinationName DLQ = new DestinationName("orders.v1.dlq");
|
||||
private static final DestinationName ORDERS = new DestinationName("orders.v1");
|
||||
|
||||
private static final HmacApprovalVerifier ISSUER =
|
||||
new HmacApprovalVerifier(
|
||||
"an-issuing-key-of-at-least-32-bytes!".getBytes(StandardCharsets.UTF_8));
|
||||
|
||||
private static final HmacApprovalVerifier ANOTHER_ISSUER =
|
||||
new HmacApprovalVerifier(
|
||||
"a-different-key-of-at-least-32-bytes".getBytes(StandardCharsets.UTF_8));
|
||||
|
||||
@Test
|
||||
@DisplayName("a verified approval cannot be constructed outside the verifier")
|
||||
void aVerifiedApprovalCannotBeConstructedOutsideTheVerifier() {
|
||||
// Written as a reflection probe because the property is about the type's shape, not its
|
||||
// behaviour: if a public constructor or factory ever appears, every other test here becomes
|
||||
// decorative.
|
||||
assertThat(VerifiedApproval.class.getConstructors())
|
||||
.as("a public constructor would let any caller mint an approval for itself")
|
||||
.isEmpty();
|
||||
assertThat(
|
||||
java.util.Arrays.stream(VerifiedApproval.class.getDeclaredMethods())
|
||||
.filter(method -> java.lang.reflect.Modifier.isStatic(method.getModifiers()))
|
||||
.filter(method -> java.lang.reflect.Modifier.isPublic(method.getModifiers()))
|
||||
.toList())
|
||||
.as("nor may a public static factory")
|
||||
.isEmpty();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("an unsigned or altered grant does not verify")
|
||||
void anAlteredGrantDoesNotVerify() {
|
||||
RedrivePlan plan = plan(50, 0);
|
||||
ApprovalGrant signed = grant(plan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 50);
|
||||
String signature = ISSUER.sign(signed);
|
||||
|
||||
ApprovalGrant raised = grant(plan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 5_000);
|
||||
|
||||
assertThatThrownBy(() -> ISSUER.verify(raised, signature, plan.digest(), NOW))
|
||||
.as("raising the impact ceiling after the approver signed it")
|
||||
.isInstanceOf(MessageAuthorizationException.class)
|
||||
.hasMessageContaining("does not verify");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a signature from another issuer does not verify")
|
||||
void aSignatureFromAnotherIssuerDoesNotVerify() {
|
||||
RedrivePlan plan = plan(50, 0);
|
||||
ApprovalGrant granted = grant(plan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 50);
|
||||
|
||||
assertThatThrownBy(
|
||||
() -> ISSUER.verify(granted, ANOTHER_ISSUER.sign(granted), plan.digest(), NOW))
|
||||
.isInstanceOf(MessageAuthorizationException.class)
|
||||
.hasMessageContaining("does not verify");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("an approval for one plan cannot execute another")
|
||||
void anApprovalForOnePlanCannotExecuteAnother() {
|
||||
RedrivePlan approvedPlan = plan(50, 0);
|
||||
RedrivePlan biggerPlan = plan(900, 0);
|
||||
ApprovalGrant granted = grant(approvedPlan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 900);
|
||||
String signature = ISSUER.sign(granted);
|
||||
|
||||
assertThatThrownBy(() -> ISSUER.verify(granted, signature, biggerPlan.digest(), NOW))
|
||||
.as("the approver saw 50 candidates; this plan moves 900")
|
||||
.isInstanceOf(MessageAuthorizationException.class)
|
||||
.hasMessageContaining("was issued for plan");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a verified approval for the wrong source cannot be attached to a plan")
|
||||
void aVerifiedApprovalForTheWrongSourceCannotBeAttached() {
|
||||
RedrivePlan plan = plan(50, 0);
|
||||
// A genuine, correctly-signed approval — for a different dead-letter destination, with this
|
||||
// plan's digest smuggled in. The digest check passes; the source binding is what catches it.
|
||||
ApprovalGrant granted =
|
||||
new ApprovalGrant(
|
||||
approval(),
|
||||
"operator@example.com",
|
||||
DestructiveOperation.REDRIVE,
|
||||
new DestinationName("payments.v1.dlq"),
|
||||
Optional.of(ORDERS),
|
||||
"v1",
|
||||
900,
|
||||
plan.digest());
|
||||
VerifiedApproval verified = ISSUER.verify(granted, ISSUER.sign(granted), plan.digest(), NOW);
|
||||
|
||||
assertThatThrownBy(() -> new ApprovedRedrivePlan(plan, verified, false))
|
||||
.isInstanceOf(MessageAuthorizationException.class)
|
||||
.hasMessageContaining("payments.v1.dlq");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a plan beyond the approved impact ceiling cannot execute")
|
||||
void aPlanBeyondTheApprovedImpactCeilingCannotExecute() {
|
||||
RedrivePlan plan = plan(900, 0);
|
||||
ApprovalGrant granted = grant(plan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 100);
|
||||
VerifiedApproval verified = ISSUER.verify(granted, ISSUER.sign(granted), plan.digest(), NOW);
|
||||
|
||||
assertThatThrownBy(() -> new ApprovedRedrivePlan(plan, verified, false))
|
||||
.as("the approver agreed to at most 100")
|
||||
.isInstanceOf(MessageAuthorizationException.class)
|
||||
.hasMessageContaining("at most 100");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("an approval for a replay cannot authorise a redrive")
|
||||
void anApprovalForAReplayCannotAuthoriseARedrive() {
|
||||
RedrivePlan plan = plan(50, 0);
|
||||
ApprovalGrant granted = grant(plan, DestructiveOperation.REPLAY, DLQ, ORDERS, 900);
|
||||
VerifiedApproval verified = ISSUER.verify(granted, ISSUER.sign(granted), plan.digest(), NOW);
|
||||
|
||||
assertThatThrownBy(() -> new ApprovedRedrivePlan(plan, verified, false))
|
||||
.isInstanceOf(MessageAuthorizationException.class)
|
||||
.hasMessageContaining("authorises REPLAY");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("an expired grant does not verify")
|
||||
void anExpiredGrantDoesNotVerify() {
|
||||
RedrivePlan plan = plan(50, 0);
|
||||
ApprovalGrant granted = grant(plan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 50);
|
||||
|
||||
assertThatThrownBy(
|
||||
() ->
|
||||
ISSUER.verify(
|
||||
granted, ISSUER.sign(granted), plan.digest(), NOW.plus(Duration.ofDays(1))))
|
||||
.isInstanceOf(MessageAuthorizationException.class)
|
||||
.hasMessageContaining("not valid at");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("the approver and the operator must be different people")
|
||||
void theApproverAndTheOperatorMustBeDifferentPeople() {
|
||||
RedrivePlan plan = plan(50, 0);
|
||||
|
||||
assertThatThrownBy(
|
||||
() ->
|
||||
new ApprovalGrant(
|
||||
approval(),
|
||||
"ops@example.com",
|
||||
DestructiveOperation.REDRIVE,
|
||||
DLQ,
|
||||
Optional.of(ORDERS),
|
||||
"v1",
|
||||
50,
|
||||
plan.digest()))
|
||||
.as("an operator approving their own destructive operation is the control not existing")
|
||||
.isInstanceOf(IllegalArgumentException.class);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a correctly issued approval executes")
|
||||
void aCorrectlyIssuedApprovalExecutes() {
|
||||
RedrivePlan plan = plan(50, 0);
|
||||
ApprovalGrant granted = grant(plan, DestructiveOperation.REDRIVE, DLQ, ORDERS, 50);
|
||||
VerifiedApproval verified = ISSUER.verify(granted, ISSUER.sign(granted), plan.digest(), NOW);
|
||||
|
||||
assertThatCode(
|
||||
() -> new ApprovedRedrivePlan(plan, verified, false).requireExecutable(NOW, "v1"))
|
||||
.doesNotThrowAnyException();
|
||||
}
|
||||
|
||||
private static ApprovalGrant grant(
|
||||
RedrivePlan plan,
|
||||
DestructiveOperation operation,
|
||||
DestinationName source,
|
||||
DestinationName target,
|
||||
long maxImpact) {
|
||||
return new ApprovalGrant(
|
||||
approval(),
|
||||
"operator@example.com",
|
||||
operation,
|
||||
source,
|
||||
Optional.of(target),
|
||||
"v1",
|
||||
maxImpact,
|
||||
plan.digest());
|
||||
}
|
||||
|
||||
private static AdminApproval approval() {
|
||||
return new AdminApproval("CHG-1042", "ops@example.com", NOW, NOW.plus(Duration.ofHours(2)));
|
||||
}
|
||||
|
||||
private static RedrivePlan plan(int candidates, int alreadyRedriven) {
|
||||
return new RedrivePlan(
|
||||
new RedriveRequest(OPERATION_ID, DLQ, ORDERS, 50, false),
|
||||
candidates,
|
||||
alreadyRedriven,
|
||||
NOW,
|
||||
"v1");
|
||||
}
|
||||
}
|
||||
+70
-34
@@ -5,14 +5,19 @@ import static org.assertj.core.api.Assertions.assertThatCode;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
import dev.caskeleton.messaging.admin.AdminApproval;
|
||||
import dev.caskeleton.messaging.admin.ApprovalGrant;
|
||||
import dev.caskeleton.messaging.admin.ApprovedRedrivePlan;
|
||||
import dev.caskeleton.messaging.admin.ApprovedReplayPlan;
|
||||
import dev.caskeleton.messaging.admin.DestructiveOperation;
|
||||
import dev.caskeleton.messaging.admin.HmacApprovalVerifier;
|
||||
import dev.caskeleton.messaging.admin.PlanDigest;
|
||||
import dev.caskeleton.messaging.admin.RedrivePlan;
|
||||
import dev.caskeleton.messaging.admin.RedriveRequest;
|
||||
import dev.caskeleton.messaging.admin.RedriveResult;
|
||||
import dev.caskeleton.messaging.admin.ReplayPlan;
|
||||
import dev.caskeleton.messaging.admin.ReplayRequest;
|
||||
import dev.caskeleton.messaging.admin.ReplayResult;
|
||||
import dev.caskeleton.messaging.admin.VerifiedApproval;
|
||||
import dev.caskeleton.messaging.api.destination.DestinationName;
|
||||
import dev.caskeleton.messaging.api.error.MessageAuthorizationException;
|
||||
import java.time.Duration;
|
||||
@@ -26,10 +31,56 @@ class ApprovedPlanExecutionTest {
|
||||
private static final Instant NOW = Instant.parse("2026-08-10T09:00:00Z");
|
||||
private static final UUID OPERATION_ID = UUID.fromString("0199aaaa-bbbb-7ccc-8ddd-eeeeffff0000");
|
||||
|
||||
private static final byte[] ISSUING_KEY =
|
||||
"an-issuing-key-of-at-least-32-bytes!".getBytes(java.nio.charset.StandardCharsets.UTF_8);
|
||||
|
||||
private static final HmacApprovalVerifier VERIFIER = new HmacApprovalVerifier(ISSUING_KEY);
|
||||
|
||||
private static AdminApproval approval() {
|
||||
return new AdminApproval("CHG-1042", "ops@example.com", NOW, NOW.plus(Duration.ofHours(2)));
|
||||
}
|
||||
|
||||
/** A grant the issuer signed for exactly this plan, verified the way the runtime verifies it. */
|
||||
static VerifiedApproval verified(
|
||||
PlanDigest digest,
|
||||
DestructiveOperation operation,
|
||||
DestinationName source,
|
||||
Optional<DestinationName> target,
|
||||
String topologyVersion,
|
||||
long maxImpact) {
|
||||
ApprovalGrant grant =
|
||||
new ApprovalGrant(
|
||||
approval(),
|
||||
"operator@example.com",
|
||||
operation,
|
||||
source,
|
||||
target,
|
||||
topologyVersion,
|
||||
maxImpact,
|
||||
digest);
|
||||
return VERIFIER.verify(grant, VERIFIER.sign(grant), digest, NOW);
|
||||
}
|
||||
|
||||
private static VerifiedApproval replayApproval(ReplayPlan plan) {
|
||||
return verified(
|
||||
plan.digest(),
|
||||
DestructiveOperation.REPLAY,
|
||||
plan.request().destination(),
|
||||
Optional.empty(),
|
||||
plan.topologyVersion(),
|
||||
Long.MAX_VALUE);
|
||||
}
|
||||
|
||||
private static VerifiedApproval redriveApproval(RedrivePlan plan) {
|
||||
return verified(
|
||||
plan.digest(),
|
||||
DestructiveOperation.REDRIVE,
|
||||
plan.request().source(),
|
||||
Optional.of(plan.request().target()),
|
||||
plan.topologyVersion(),
|
||||
Long.MAX_VALUE);
|
||||
}
|
||||
|
||||
private static ReplayRequest replayRequest() {
|
||||
return new ReplayRequest(
|
||||
OPERATION_ID,
|
||||
@@ -57,9 +108,18 @@ class ApprovedPlanExecutionTest {
|
||||
return new RedrivePlan(redriveRequest(), candidates, alreadyRedriven, NOW, "v1");
|
||||
}
|
||||
|
||||
private static ApprovedReplayPlan approvedReplay(String topologyVersion) {
|
||||
ReplayPlan plan = replayPlan(topologyVersion);
|
||||
return new ApprovedReplayPlan(plan, replayApproval(plan));
|
||||
}
|
||||
|
||||
private static ApprovedRedrivePlan approvedRedrive(RedrivePlan plan, boolean loopAcknowledged) {
|
||||
return new ApprovedRedrivePlan(plan, redriveApproval(plan), loopAcknowledged);
|
||||
}
|
||||
|
||||
@Test
|
||||
void anExpiredApprovalCannotExecute() {
|
||||
ApprovedReplayPlan approved = new ApprovedReplayPlan(replayPlan("v1"), approval());
|
||||
ApprovedReplayPlan approved = approvedReplay("v1");
|
||||
|
||||
assertThatThrownBy(() -> approved.requireExecutable(NOW.plus(Duration.ofDays(1)), "v1"))
|
||||
.isInstanceOf(MessageAuthorizationException.class)
|
||||
@@ -68,25 +128,26 @@ class ApprovedPlanExecutionTest {
|
||||
|
||||
@Test
|
||||
void aTopologyChangeSinceApprovalInvalidatesThePlan() {
|
||||
ApprovedReplayPlan approved = new ApprovedReplayPlan(replayPlan("v1"), approval());
|
||||
ApprovedReplayPlan approved = approvedReplay("v1");
|
||||
|
||||
assertThatThrownBy(() -> approved.requireExecutable(NOW, "v2"))
|
||||
.as("every number in the plan was computed against the old topology")
|
||||
.isInstanceOf(MessageAuthorizationException.class)
|
||||
.hasMessageContaining("rebuilt");
|
||||
// The signed grant carries the topology version too, so the mismatch is now caught against
|
||||
// what the approver actually signed rather than against a field on the plan the caller
|
||||
// handed in.
|
||||
.hasMessageContaining("was signed against topology v1 but the broker is now at v2");
|
||||
}
|
||||
|
||||
@Test
|
||||
void aValidApprovalOnUnchangedTopologyExecutes() {
|
||||
assertThatCode(
|
||||
() -> new ApprovedReplayPlan(replayPlan("v1"), approval()).requireExecutable(NOW, "v1"))
|
||||
assertThatCode(() -> approvedReplay("v1").requireExecutable(NOW, "v1"))
|
||||
.doesNotThrowAnyException();
|
||||
}
|
||||
|
||||
@Test
|
||||
void aRedriveThatWouldLoopNeedsThatAcknowledgedExplicitly() {
|
||||
ApprovedRedrivePlan approved =
|
||||
new ApprovedRedrivePlan(redrivePlan(900, 400), approval(), false);
|
||||
ApprovedRedrivePlan approved = approvedRedrive(redrivePlan(900, 400), false);
|
||||
|
||||
assertThatThrownBy(() -> approved.requireExecutable(NOW, "v1"))
|
||||
.isInstanceOf(MessageAuthorizationException.class)
|
||||
@@ -95,19 +156,13 @@ class ApprovedPlanExecutionTest {
|
||||
|
||||
@Test
|
||||
void anAcknowledgedLoopMayProceed() {
|
||||
assertThatCode(
|
||||
() ->
|
||||
new ApprovedRedrivePlan(redrivePlan(900, 400), approval(), true)
|
||||
.requireExecutable(NOW, "v1"))
|
||||
assertThatCode(() -> approvedRedrive(redrivePlan(900, 400), true).requireExecutable(NOW, "v1"))
|
||||
.doesNotThrowAnyException();
|
||||
}
|
||||
|
||||
@Test
|
||||
void aRedriveWithNoPreviouslyRedrivenCandidatesNeedsNoAcknowledgement() {
|
||||
assertThatCode(
|
||||
() ->
|
||||
new ApprovedRedrivePlan(redrivePlan(900, 0), approval(), false)
|
||||
.requireExecutable(NOW, "v1"))
|
||||
assertThatCode(() -> approvedRedrive(redrivePlan(900, 0), false).requireExecutable(NOW, "v1"))
|
||||
.doesNotThrowAnyException();
|
||||
}
|
||||
|
||||
@@ -139,25 +194,6 @@ class ApprovedPlanExecutionTest {
|
||||
.contains("LIVE consumer group");
|
||||
}
|
||||
|
||||
@Test
|
||||
void anApprovalAuthorisesOneExecutionNotAStandingPermission() {
|
||||
AdminOperationIdempotencyStore store = new AdminOperationIdempotencyStore();
|
||||
|
||||
assertThat(store.claim("CHG-1042", "op-1", NOW)).isEmpty();
|
||||
assertThat(store.claim("CHG-1042", "op-2", NOW.plus(Duration.ofMinutes(5))))
|
||||
.as("an arrow-up in the shell must not re-run an approved redrive")
|
||||
.hasValueSatisfying(existing -> assertThat(existing.operationId()).isEqualTo("op-1"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void aDifferentApprovalIsClaimedIndependently() {
|
||||
AdminOperationIdempotencyStore store = new AdminOperationIdempotencyStore();
|
||||
store.claim("CHG-1042", "op-1", NOW);
|
||||
|
||||
assertThat(store.claim("CHG-1043", "op-2", NOW)).isEmpty();
|
||||
assertThat(store.size()).isEqualTo(2);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aReplayResultFlagsAShortfallAgainstTheApprovedEstimate() {
|
||||
ReplayResult result =
|
||||
|
||||
+212
@@ -0,0 +1,212 @@
|
||||
package dev.caskeleton.messaging.admin.runtime;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import dev.caskeleton.messaging.admin.AdminApproval;
|
||||
import dev.caskeleton.messaging.admin.ApprovalGrant;
|
||||
import dev.caskeleton.messaging.admin.DestructiveOperation;
|
||||
import dev.caskeleton.messaging.admin.DestructiveOperationGuard;
|
||||
import dev.caskeleton.messaging.admin.HmacApprovalVerifier;
|
||||
import dev.caskeleton.messaging.admin.PlanDigest;
|
||||
import dev.caskeleton.messaging.admin.RedriveRequest;
|
||||
import dev.caskeleton.messaging.admin.VerifiedApproval;
|
||||
import dev.caskeleton.messaging.api.MessageId;
|
||||
import dev.caskeleton.messaging.api.destination.DestinationName;
|
||||
import dev.caskeleton.messaging.api.publish.ConfirmationLevel;
|
||||
import dev.caskeleton.messaging.api.publish.PublishCompletion;
|
||||
import dev.caskeleton.messaging.api.publish.PublishEvidence;
|
||||
import dev.caskeleton.messaging.api.publish.PublishResult;
|
||||
import dev.caskeleton.messaging.api.publish.RoutingOutcome;
|
||||
import dev.caskeleton.messaging.observation.MessagingAuditEvent;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import java.util.UUID;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* What a redrive does when one message fails, and what it does the second time round.
|
||||
*
|
||||
* <p>The loop had no per-item boundary. A publisher that threw — a broker refusing the connection
|
||||
* rather than the message — propagated straight out, so every candidate behind it was abandoned and
|
||||
* the audit record was never written. The operation left no trace of what it had already moved, and
|
||||
* a retry started again from the first candidate and republished it.
|
||||
*/
|
||||
class RedriveResumptionTest {
|
||||
|
||||
private static final Instant NOW = Instant.parse("2026-08-10T09:00:00Z");
|
||||
private static final DestinationName DLQ = new DestinationName("orders.v1.dlq");
|
||||
private static final DestinationName ORDERS = new DestinationName("orders.v1");
|
||||
|
||||
private static final HmacApprovalVerifier ISSUER =
|
||||
new HmacApprovalVerifier(
|
||||
"an-issuing-key-of-at-least-32-bytes!".getBytes(java.nio.charset.StandardCharsets.UTF_8));
|
||||
|
||||
private final RecordingSource source = new RecordingSource();
|
||||
private final RecordingAudit audit = new RecordingAudit();
|
||||
|
||||
@Test
|
||||
@DisplayName("one message that throws does not abandon the ones behind it")
|
||||
void oneThrowingMessageDoesNotAbandonTheRest() {
|
||||
List<MessageId> candidates = ids(4);
|
||||
source.stage(candidates);
|
||||
RedriveService service =
|
||||
service(
|
||||
(messageId, target, redriveId) -> {
|
||||
if (messageId.equals(candidates.get(1))) {
|
||||
throw new IllegalStateException("the broker refused the connection");
|
||||
}
|
||||
return confirmed();
|
||||
});
|
||||
|
||||
RedriveReport report = service.redrive(request(), approval(), "operator", NOW);
|
||||
|
||||
assertThat(report.moved()).isEqualTo(3);
|
||||
assertThat(report.failed())
|
||||
.as("the throwing item is a failed item, not a failed pass")
|
||||
.isEqualTo(1);
|
||||
assertThat(source.settled)
|
||||
.containsExactly(candidates.get(0), candidates.get(2), candidates.get(3));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("the audit record is written even when the pass does not finish cleanly")
|
||||
void theAuditRecordIsWrittenEvenOnFailure() {
|
||||
source.stage(ids(2));
|
||||
RedriveService service =
|
||||
service(
|
||||
(messageId, target, redriveId) -> {
|
||||
throw new IllegalStateException("the broker refused the connection");
|
||||
});
|
||||
|
||||
service.redrive(request(), approval(), "operator", NOW);
|
||||
|
||||
assertThat(audit.events)
|
||||
.as("the record of what moved is what the incident review and the resumed run both read")
|
||||
.singleElement()
|
||||
.satisfies(event -> assertThat(event.details()).containsEntry("failed", "2"));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a resumed pass does not republish what the first attempt already moved")
|
||||
void aResumedPassDoesNotRepublishWhatWasAlreadyMoved() {
|
||||
List<MessageId> candidates = ids(5);
|
||||
source.stage(candidates);
|
||||
List<MessageId> republished = new ArrayList<>();
|
||||
RedriveService service =
|
||||
service(
|
||||
(messageId, target, redriveId) -> {
|
||||
republished.add(messageId);
|
||||
return confirmed();
|
||||
});
|
||||
|
||||
RedriveReport report =
|
||||
service.redrive(request(), approval(), "operator", NOW, 3, completed -> {});
|
||||
|
||||
assertThat(republished)
|
||||
.as("republishing a settled message is a duplicate, not a retry")
|
||||
.containsExactly(candidates.get(3), candidates.get(4));
|
||||
assertThat(report.moved()).isEqualTo(2);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("progress is reported after each item so a crash resumes near where it stopped")
|
||||
void progressIsReportedAfterEachItem() {
|
||||
source.stage(ids(3));
|
||||
List<Integer> checkpoints = new ArrayList<>();
|
||||
RedriveService service = service((messageId, target, redriveId) -> confirmed());
|
||||
|
||||
service.redrive(request(), approval(), "operator", NOW, 0, checkpoints::add);
|
||||
|
||||
assertThat(checkpoints).containsExactly(1, 2, 3);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("resuming past the end of the batch moves nothing")
|
||||
void resumingPastTheEndOfTheBatchMovesNothing() {
|
||||
source.stage(ids(2));
|
||||
RedriveService service = service((messageId, target, redriveId) -> confirmed());
|
||||
|
||||
RedriveReport report =
|
||||
service.redrive(request(), approval(), "operator", NOW, 10, completed -> {});
|
||||
|
||||
assertThat(report.moved()).isZero();
|
||||
}
|
||||
|
||||
private RedriveService service(RedriveService.RedrivePublisher publisher) {
|
||||
// The guard holds the admin credential and every pass carries a real verified approval: this
|
||||
// test is about the loop, and a gate that refused would prove nothing about it.
|
||||
return new RedriveService(new DestructiveOperationGuard(true), source, publisher, audit);
|
||||
}
|
||||
|
||||
private static Optional<VerifiedApproval> approval() {
|
||||
PlanDigest digest = PlanDigest.ofCanonical("REDRIVE|orders.v1.dlq|orders.v1");
|
||||
ApprovalGrant grant =
|
||||
new ApprovalGrant(
|
||||
new AdminApproval("CHG-1042", "ops@example.com", NOW, NOW.plus(Duration.ofHours(2))),
|
||||
"operator@example.com",
|
||||
DestructiveOperation.REDRIVE,
|
||||
DLQ,
|
||||
Optional.of(ORDERS),
|
||||
"v1",
|
||||
Long.MAX_VALUE,
|
||||
digest);
|
||||
return Optional.of(ISSUER.verify(grant, ISSUER.sign(grant), digest, NOW));
|
||||
}
|
||||
|
||||
private static RedriveRequest request() {
|
||||
return new RedriveRequest(UUID.randomUUID(), DLQ, ORDERS, 50, false);
|
||||
}
|
||||
|
||||
private static PublishResult confirmed() {
|
||||
return new PublishResult(
|
||||
PublishCompletion.CONFIRMED,
|
||||
PublishEvidence.confirmed(ConfirmationLevel.REPLICATION_OR_PERSISTENCE_ACK),
|
||||
RoutingOutcome.ROUTED,
|
||||
Optional.empty(),
|
||||
1,
|
||||
Duration.ofMillis(2),
|
||||
Optional.empty());
|
||||
}
|
||||
|
||||
private static List<MessageId> ids(int count) {
|
||||
List<MessageId> ids = new ArrayList<>();
|
||||
for (int index = 0; index < count; index++) {
|
||||
ids.add(MessageId.newId());
|
||||
}
|
||||
return List.copyOf(ids);
|
||||
}
|
||||
|
||||
private static final class RecordingSource implements RedriveService.RedriveSource {
|
||||
|
||||
private List<MessageId> staged = List.of();
|
||||
private final List<MessageId> settled = new ArrayList<>();
|
||||
|
||||
private void stage(List<MessageId> ids) {
|
||||
staged = List.copyOf(ids);
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<MessageId> peek(DestinationName destination, int batchSize) {
|
||||
return staged;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void settle(DestinationName destination, MessageId messageId) {
|
||||
settled.add(messageId);
|
||||
}
|
||||
}
|
||||
|
||||
private static final class RecordingAudit implements RedriveService.AuditSink {
|
||||
|
||||
private final List<MessagingAuditEvent> events = new ArrayList<>();
|
||||
|
||||
@Override
|
||||
public void record(MessagingAuditEvent event) {
|
||||
events.add(event);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -4,6 +4,12 @@ dependencies {
|
||||
api project(':messaging:messaging-core-api')
|
||||
api project(':messaging:messaging-schema-api')
|
||||
|
||||
implementation 'io.cloudevents:cloudevents-api:4.0.1'
|
||||
// api, because CloudEventMapper's public signatures return io.cloudevents.CloudEvent.
|
||||
//
|
||||
// Declared `implementation`, the type appeared in this module's public API while the
|
||||
// dependency was hidden from consumers: an adopter calling the documented method could not
|
||||
// name its return type without adding CloudEvents to their own build, and Gradle gave them no
|
||||
// hint why. A type in a public signature is part of the artifact's contract.
|
||||
api 'io.cloudevents:cloudevents-api:4.0.1'
|
||||
implementation 'io.cloudevents:cloudevents-core:4.0.1'
|
||||
}
|
||||
|
||||
+3
-1
@@ -87,7 +87,9 @@ class CloudEventMappingTest {
|
||||
void aCloudEventWithNoDataBecomesAnEmptyPayloadNotANullValue() {
|
||||
CloudEvent noData =
|
||||
io.cloudevents.core.builder.CloudEventBuilder.v1()
|
||||
.withId(UUID.randomUUID().toString())
|
||||
// A v7 id: MessageId enforces the version it documents, so a v4 arriving from a foreign
|
||||
// producer is refused here exactly as it would be on the wire.
|
||||
.withId(dev.caskeleton.messaging.api.MessageId.newId().value().toString())
|
||||
.withSource(SOURCE)
|
||||
.withType("order.created")
|
||||
.withTime(java.time.OffsetDateTime.parse("2026-08-10T09:15:00Z"))
|
||||
|
||||
+13
-2
@@ -11,6 +11,13 @@ public record ContentType(String value) {
|
||||
|
||||
private static final int MAX_LENGTH = 160;
|
||||
|
||||
/** {@code type/subtype} with the optional parameters RFC 9110 allows. */
|
||||
private static final java.util.regex.Pattern MEDIA_TYPE =
|
||||
java.util.regex.Pattern.compile(
|
||||
"[A-Za-z0-9][A-Za-z0-9!#$%&'*+.^_`|~-]*"
|
||||
+ "/[A-Za-z0-9][A-Za-z0-9!#$%&'*+.^_`|~-]*"
|
||||
+ "(;\\s*[A-Za-z0-9!#$%&'*+.^_`|~-]+=[^;\\s]+)*");
|
||||
|
||||
/** The Stable default codec's content type. */
|
||||
public static final ContentType JSON = new ContentType("application/json");
|
||||
|
||||
@@ -27,8 +34,12 @@ public record ContentType(String value) {
|
||||
if (value == null || value.isBlank() || value.length() > MAX_LENGTH) {
|
||||
throw new IllegalArgumentException("contentType must contain 1 to 160 characters");
|
||||
}
|
||||
if (value.indexOf('/') < 0) {
|
||||
throw new IllegalArgumentException("contentType must be a media type: " + value);
|
||||
// A media type, not merely a string containing a slash. `a/b/c`, `/json` and `text/` all
|
||||
// passed the old check and are all rejected by a broker or a codec registry later, with an
|
||||
// error that names the message rather than the configuration that produced it.
|
||||
if (!MEDIA_TYPE.matcher(value).matches()) {
|
||||
throw new IllegalArgumentException(
|
||||
"contentType must be a type/subtype media type: '" + value + '\'');
|
||||
}
|
||||
value = value.toLowerCase(Locale.ROOT);
|
||||
}
|
||||
|
||||
+13
@@ -14,8 +14,21 @@ import java.util.UUID;
|
||||
*/
|
||||
public record MessageId(UUID value) {
|
||||
|
||||
/** The UUID version this identity is documented to carry. */
|
||||
private static final int VERSION_7 = 7;
|
||||
|
||||
public MessageId {
|
||||
Objects.requireNonNull(value, "messageId value must not be null");
|
||||
// The type says UUIDv7 and the constructor accepted any UUID, including v4 and the nil UUID.
|
||||
// Version 7 is what makes the identity time-ordered, which is what the outbox index and every
|
||||
// "oldest first" claim depend on; a v4 stored in the same column silently defeats both.
|
||||
if (value.version() != VERSION_7) {
|
||||
throw new IllegalArgumentException(
|
||||
"a message identity is UUIDv7 (time-ordered); this is version " + value.version());
|
||||
}
|
||||
if (value.variant() != 2) {
|
||||
throw new IllegalArgumentException("a message identity must use the RFC 4122 variant");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+24
@@ -16,5 +16,29 @@ public record MessageType(String value) {
|
||||
if (value == null || value.isBlank() || value.length() > MAX_LENGTH) {
|
||||
throw new IllegalArgumentException("messageType must contain 1 to 240 characters");
|
||||
}
|
||||
requireWireSafe(value, "a message type", MAX_LENGTH);
|
||||
}
|
||||
|
||||
/**
|
||||
* Characters that must never reach a broker.
|
||||
*
|
||||
* <p>Length in Java {@code char}s bounded nothing useful: a 240-character string can be 960 UTF-8
|
||||
* bytes, and a control character passes a length check untouched. CR and LF split records on
|
||||
* line-oriented protocols, NUL truncates them in C string handling, and both survive right up to
|
||||
* the broker that mishandles them.
|
||||
*/
|
||||
private static void requireWireSafe(String value, String what, int maxBytes) {
|
||||
for (int index = 0; index < value.length(); index++) {
|
||||
char character = value.charAt(index);
|
||||
if (character < 0x20 || character == 0x7F) {
|
||||
throw new IllegalArgumentException(
|
||||
what
|
||||
+ " must not contain control characters (found U+%04X)".formatted((int) character));
|
||||
}
|
||||
}
|
||||
int bytes = value.getBytes(java.nio.charset.StandardCharsets.UTF_8).length;
|
||||
if (bytes > maxBytes) {
|
||||
throw new IllegalArgumentException(what + " exceeds " + maxBytes + " UTF-8 bytes: " + bytes);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+24
@@ -16,5 +16,29 @@ public record ProducerId(String value) {
|
||||
if (value == null || value.isBlank() || value.length() > MAX_LENGTH) {
|
||||
throw new IllegalArgumentException("producerId must contain 1 to 120 characters");
|
||||
}
|
||||
requireWireSafe(value, "a producer id", MAX_LENGTH);
|
||||
}
|
||||
|
||||
/**
|
||||
* Characters that must never reach a broker.
|
||||
*
|
||||
* <p>Length in Java {@code char}s bounded nothing useful: a 240-character string can be 960 UTF-8
|
||||
* bytes, and a control character passes a length check untouched. CR and LF split records on
|
||||
* line-oriented protocols, NUL truncates them in C string handling, and both survive right up to
|
||||
* the broker that mishandles them.
|
||||
*/
|
||||
private static void requireWireSafe(String value, String what, int maxBytes) {
|
||||
for (int index = 0; index < value.length(); index++) {
|
||||
char character = value.charAt(index);
|
||||
if (character < 0x20 || character == 0x7F) {
|
||||
throw new IllegalArgumentException(
|
||||
what
|
||||
+ " must not contain control characters (found U+%04X)".formatted((int) character));
|
||||
}
|
||||
}
|
||||
int bytes = value.getBytes(java.nio.charset.StandardCharsets.UTF_8).length;
|
||||
if (bytes > maxBytes) {
|
||||
throw new IllegalArgumentException(what + " exceeds " + maxBytes + " UTF-8 bytes: " + bytes);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+26
@@ -15,10 +15,36 @@ public record HeaderName(String value) {
|
||||
|
||||
private static final int MAX_BYTES = 128;
|
||||
|
||||
/**
|
||||
* The one grammar a header name may use.
|
||||
*
|
||||
* <p>The previous check was "not blank, at most 128 bytes", which accepted a name containing CR,
|
||||
* LF, NUL, a colon or leading whitespace. Those are not cosmetic: a newline in a header name is a
|
||||
* header-injection primitive on any line-oriented protocol, a NUL truncates the name in C string
|
||||
* handling inside a broker, and a colon is the field separator in several wire formats. A name
|
||||
* that survives one broker and corrupts the next is worse than a rejected one.
|
||||
*
|
||||
* <p>The character set is the HTTP token set, which every broker this platform targets accepts
|
||||
* verbatim.
|
||||
*/
|
||||
private static final java.util.regex.Pattern TOKEN =
|
||||
java.util.regex.Pattern.compile("[a-zA-Z0-9!#$%&'*+._|~-]+");
|
||||
|
||||
public HeaderName {
|
||||
if (value == null || value.isBlank()) {
|
||||
throw new IllegalArgumentException("header name must not be blank");
|
||||
}
|
||||
if (!value.equals(value.strip())) {
|
||||
// Trimming would mean `Authorization ` and `Authorization` are the same name to the
|
||||
// denylist and different names on the wire, which is precisely how the check was bypassed.
|
||||
throw new IllegalArgumentException(
|
||||
"header name must not have leading or trailing whitespace: '" + value + '\'');
|
||||
}
|
||||
if (!TOKEN.matcher(value).matches()) {
|
||||
throw new IllegalArgumentException(
|
||||
"header name must be an ASCII token; control characters, spaces, colons and non-ASCII"
|
||||
+ " are refused because a broker that accepts one may truncate or split on it");
|
||||
}
|
||||
int bytes = value.getBytes(StandardCharsets.UTF_8).length;
|
||||
if (bytes > MAX_BYTES) {
|
||||
throw new IllegalArgumentException("header name exceeds 128 bytes: " + bytes);
|
||||
|
||||
BIN
Binary file not shown.
+73
-5
@@ -11,6 +11,8 @@ import java.sql.Timestamp;
|
||||
import java.time.Instant;
|
||||
import java.util.Objects;
|
||||
import javax.sql.DataSource;
|
||||
import org.springframework.jdbc.datasource.DataSourceUtils;
|
||||
import org.springframework.transaction.support.TransactionSynchronizationManager;
|
||||
|
||||
/**
|
||||
* The PostgreSQL inbox.
|
||||
@@ -48,14 +50,19 @@ public final class JdbcInboxRepository implements InboxRepository {
|
||||
/**
|
||||
* Reserves a message on the caller's transactional connection.
|
||||
*
|
||||
* <p>Package-private. It used to be public and was the only path that actually joined the
|
||||
* caller's transaction, while the interface method — the one {@code IdempotentConsumer} calls —
|
||||
* opened a raw connection that auto-commits. A reservation that commits on its own while the
|
||||
* business side effect rolls back is a message that will never be redelivered and whose work
|
||||
* never happened.
|
||||
*
|
||||
* @param connection the connection the business write is running on
|
||||
* @param messageId the logical message identity
|
||||
* @param consumerId the consumer's stable identity
|
||||
* @param now the current instant
|
||||
* @return true when this is the first time
|
||||
*/
|
||||
public boolean reserve(
|
||||
Connection connection, MessageId messageId, String consumerId, Instant now) {
|
||||
boolean reserve(Connection connection, MessageId messageId, String consumerId, Instant now) {
|
||||
Objects.requireNonNull(connection, "connection must not be null");
|
||||
Objects.requireNonNull(messageId, "messageId must not be null");
|
||||
Objects.requireNonNull(now, "now must not be null");
|
||||
@@ -75,11 +82,41 @@ public final class JdbcInboxRepository implements InboxRepository {
|
||||
|
||||
@Override
|
||||
public boolean reserve(MessageId messageId, String consumerId, Instant now) {
|
||||
try (Connection connection = dataSource.getConnection()) {
|
||||
requireActiveTransaction("reserving an inbox message");
|
||||
Connection connection = DataSourceUtils.getConnection(dataSource);
|
||||
try {
|
||||
return reserve(connection, messageId, consumerId, now);
|
||||
} catch (SQLException exception) {
|
||||
} finally {
|
||||
DataSourceUtils.releaseConnection(connection, dataSource);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Refuses when there is no transaction to join, or when it cannot write.
|
||||
*
|
||||
* <p>The reservation and the side effect it guards have to commit or roll back together. Running
|
||||
* the reservation on its own connection breaks that on the rollback path only — which is the path
|
||||
* nobody exercises before production, and the one where the message is lost for good.
|
||||
*/
|
||||
private void requireActiveTransaction(String what) {
|
||||
if (!TransactionSynchronizationManager.isActualTransactionActive()) {
|
||||
throw new MessagingConfigurationException(
|
||||
"INBOX_RESERVE_FAILED", "could not reserve an inbox record", exception);
|
||||
"INBOX_TRANSACTION_REQUIRED",
|
||||
what
|
||||
+ " requires an active transaction; a reservation that commits alone marks a message"
|
||||
+ " processed whose work may still roll back");
|
||||
}
|
||||
if (TransactionSynchronizationManager.isCurrentTransactionReadOnly()) {
|
||||
throw new MessagingConfigurationException(
|
||||
"INBOX_TRANSACTION_REQUIRED",
|
||||
what + " requires a writable transaction; the current one is read-only");
|
||||
}
|
||||
if (!TransactionSynchronizationManager.hasResource(dataSource)) {
|
||||
throw new MessagingConfigurationException(
|
||||
"INBOX_TRANSACTION_REQUIRED",
|
||||
what
|
||||
+ " requires the active transaction to be bound to this DataSource; it is bound to"
|
||||
+ " another, so the reservation and the side effect would commit independently");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -100,6 +137,37 @@ public final class JdbcInboxRepository implements InboxRepository {
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public int purgeProcessedBefore(Instant processedBefore, int limit) {
|
||||
Objects.requireNonNull(processedBefore, "processedBefore must not be null");
|
||||
if (limit < 1) {
|
||||
throw new IllegalArgumentException("a bounded purge deletes at least one row per call");
|
||||
}
|
||||
try (Connection connection = dataSource.getConnection();
|
||||
PreparedStatement statement =
|
||||
connection.prepareStatement(
|
||||
"""
|
||||
WITH expired AS (
|
||||
SELECT message_id, consumer_id
|
||||
FROM messaging_inbox
|
||||
WHERE processed_at < ?
|
||||
ORDER BY processed_at
|
||||
LIMIT ?
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
DELETE FROM messaging_inbox i
|
||||
USING expired e
|
||||
WHERE i.message_id = e.message_id AND i.consumer_id = e.consumer_id
|
||||
""")) {
|
||||
statement.setTimestamp(1, Timestamp.from(processedBefore));
|
||||
statement.setInt(2, limit);
|
||||
return statement.executeUpdate();
|
||||
} catch (SQLException exception) {
|
||||
throw new MessagingConfigurationException(
|
||||
"INBOX_PURGE_FAILED", "a bounded inbox purge failed", exception);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public int purgeProcessedBefore(Instant processedBefore) {
|
||||
Objects.requireNonNull(processedBefore, "processedBefore must not be null");
|
||||
+5
@@ -102,6 +102,11 @@ final class InMemoryInbox implements InboxRepository {
|
||||
return reserved.contains(key(messageId, consumerId));
|
||||
}
|
||||
|
||||
@Override
|
||||
public int purgeProcessedBefore(Instant processedBefore, int limit) {
|
||||
return Math.min(purgeProcessedBefore(processedBefore), limit);
|
||||
}
|
||||
|
||||
@Override
|
||||
public int purgeProcessedBefore(Instant processedBefore) {
|
||||
int removed = reserved.size();
|
||||
+5
@@ -42,6 +42,11 @@ class InboxOperationsTest {
|
||||
return reserved.contains(messageId.value() + "|" + consumerId);
|
||||
}
|
||||
|
||||
@Override
|
||||
public int purgeProcessedBefore(Instant processedBefore, int limit) {
|
||||
return Math.min(purgeProcessedBefore(processedBefore), limit);
|
||||
}
|
||||
|
||||
@Override
|
||||
public int purgeProcessedBefore(Instant processedBefore) {
|
||||
cutoffs.add(processedBefore);
|
||||
+50
-28
@@ -48,6 +48,8 @@ class InboxPostgresIT {
|
||||
.withPassword("messaging");
|
||||
|
||||
private DataSource dataSource;
|
||||
|
||||
private org.springframework.transaction.support.TransactionTemplate transactions;
|
||||
private JdbcInboxRepository repository;
|
||||
|
||||
static boolean dockerAvailable() {
|
||||
@@ -65,29 +67,40 @@ class InboxPostgresIT {
|
||||
statement.execute("CREATE TABLE projections (id TEXT PRIMARY KEY, applied INT NOT NULL)");
|
||||
}
|
||||
repository = new JdbcInboxRepository(dataSource);
|
||||
// A real Spring transaction, because the port's contract is about joining one. The previous
|
||||
// tests called the Connection overload directly, which proved that overload works and said
|
||||
// nothing about the interface method every consumer actually calls.
|
||||
transactions =
|
||||
new org.springframework.transaction.support.TransactionTemplate(
|
||||
new org.springframework.jdbc.datasource.DataSourceTransactionManager(dataSource));
|
||||
}
|
||||
|
||||
@Test
|
||||
void theFirstReservationSucceedsAndTheSecondDoesNot() {
|
||||
MessageId messageId = MessageId.newId();
|
||||
|
||||
assertThat(repository.reserve(messageId, "order-projection", NOW)).isTrue();
|
||||
assertThat(repository.reserve(messageId, "order-projection", NOW)).isFalse();
|
||||
assertThat(inTransaction(() -> repository.reserve(messageId, "order-projection", NOW)))
|
||||
.isTrue();
|
||||
assertThat(inTransaction(() -> repository.reserve(messageId, "order-projection", NOW)))
|
||||
.isFalse();
|
||||
}
|
||||
|
||||
@Test
|
||||
void twoConsumersEachReserveTheSameMessageOnce() {
|
||||
MessageId messageId = MessageId.newId();
|
||||
|
||||
assertThat(repository.reserve(messageId, "order-projection", NOW)).isTrue();
|
||||
assertThat(repository.reserve(messageId, "billing-projection", NOW)).isTrue();
|
||||
assertThat(repository.reserve(messageId, "order-projection", NOW)).isFalse();
|
||||
assertThat(inTransaction(() -> repository.reserve(messageId, "order-projection", NOW)))
|
||||
.isTrue();
|
||||
assertThat(inTransaction(() -> repository.reserve(messageId, "billing-projection", NOW)))
|
||||
.isTrue();
|
||||
assertThat(inTransaction(() -> repository.reserve(messageId, "order-projection", NOW)))
|
||||
.isFalse();
|
||||
}
|
||||
|
||||
@Test
|
||||
void aProcessedMessageIsVisibleToTheQuery() {
|
||||
MessageId messageId = MessageId.newId();
|
||||
repository.reserve(messageId, "order-projection", NOW);
|
||||
inTransaction(() -> repository.reserve(messageId, "order-projection", NOW));
|
||||
|
||||
assertThat(repository.isProcessed(messageId, "order-projection")).isTrue();
|
||||
assertThat(repository.isProcessed(messageId, "billing-projection")).isFalse();
|
||||
@@ -109,13 +122,19 @@ class InboxPostgresIT {
|
||||
void aRolledBackTransactionLeavesNoReservationAndNoSideEffect() throws SQLException {
|
||||
MessageId messageId = MessageId.newId();
|
||||
|
||||
try (Connection connection = dataSource.getConnection()) {
|
||||
connection.setAutoCommit(false);
|
||||
assertThat(repository.reserve(connection, messageId, "order-projection", NOW)).isTrue();
|
||||
try (Statement statement = connection.createStatement()) {
|
||||
statement.execute("INSERT INTO projections (id, applied) VALUES ('o-2', 1)");
|
||||
}
|
||||
connection.rollback();
|
||||
// Through the interface method and a real Spring transaction. This is the path a consumer
|
||||
// takes; the old version called the Connection overload, which is the one path that was never
|
||||
// in doubt.
|
||||
try {
|
||||
transactions.executeWithoutResult(
|
||||
status -> {
|
||||
assertThat(repository.reserve(messageId, "order-projection", NOW)).isTrue();
|
||||
new org.springframework.jdbc.core.JdbcTemplate(dataSource)
|
||||
.execute("INSERT INTO projections (id, applied) VALUES ('o-2', 1)");
|
||||
status.setRollbackOnly();
|
||||
});
|
||||
} catch (RuntimeException rolledBack) {
|
||||
// setRollbackOnly is the mechanism; an UnexpectedRollbackException is not a failure here.
|
||||
}
|
||||
|
||||
assertThat(repository.isProcessed(messageId, "order-projection"))
|
||||
@@ -126,25 +145,28 @@ class InboxPostgresIT {
|
||||
|
||||
@Test
|
||||
void retentionRemovesOldRows() {
|
||||
repository.reserve(MessageId.newId(), "order-projection", NOW);
|
||||
MessageId purged = MessageId.newId();
|
||||
inTransaction(() -> repository.reserve(purged, "order-projection", NOW));
|
||||
|
||||
assertThat(repository.purgeProcessedBefore(NOW.plusSeconds(1))).isEqualTo(1);
|
||||
}
|
||||
|
||||
/** Reserves and writes inside one transaction, exactly as an idempotent consumer must. */
|
||||
private void handle(MessageId messageId, AtomicInteger applied) throws SQLException {
|
||||
try (Connection connection = dataSource.getConnection()) {
|
||||
connection.setAutoCommit(false);
|
||||
if (!repository.reserve(connection, messageId, "order-projection", NOW)) {
|
||||
connection.rollback();
|
||||
return;
|
||||
}
|
||||
try (Statement statement = connection.createStatement()) {
|
||||
statement.execute("INSERT INTO projections (id, applied) VALUES ('o-1', 1)");
|
||||
}
|
||||
applied.incrementAndGet();
|
||||
connection.commit();
|
||||
}
|
||||
/** Reserves and writes inside one Spring transaction, exactly as an idempotent consumer must. */
|
||||
private void handle(MessageId messageId, AtomicInteger applied) {
|
||||
transactions.executeWithoutResult(
|
||||
status -> {
|
||||
if (!repository.reserve(messageId, "order-projection", NOW)) {
|
||||
status.setRollbackOnly();
|
||||
return;
|
||||
}
|
||||
new org.springframework.jdbc.core.JdbcTemplate(dataSource)
|
||||
.execute("INSERT INTO projections (id, applied) VALUES ('o-1', 1)");
|
||||
applied.incrementAndGet();
|
||||
});
|
||||
}
|
||||
|
||||
private <T> T inTransaction(java.util.function.Supplier<T> work) {
|
||||
return transactions.execute(status -> work.get());
|
||||
}
|
||||
|
||||
private int projectionCount(String id) throws SQLException {
|
||||
+98
@@ -0,0 +1,98 @@
|
||||
package dev.caskeleton.messaging.inbox;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
import dev.caskeleton.messaging.api.MessageId;
|
||||
import dev.caskeleton.messaging.api.error.MessagingConfigurationException;
|
||||
import java.time.Instant;
|
||||
import javax.sql.DataSource;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.transaction.support.TransactionSynchronizationManager;
|
||||
|
||||
/**
|
||||
* What {@code reserve(...)} does when there is no transaction to join.
|
||||
*
|
||||
* <p>{@code IdempotentConsumer} calls this inside its transaction runner, and the implementation
|
||||
* opened a raw connection that auto-commits. The reservation therefore survived a rollback of the
|
||||
* side effect it was guarding: the message counts as processed, the work never happened, and
|
||||
* redelivery is refused because the inbox row is already there.
|
||||
*
|
||||
* <p>Hermetic: the refusal has to happen before any connection is requested, and the data source
|
||||
* below fails the test by being asked for one.
|
||||
*/
|
||||
class JdbcInboxTransactionRequirementTest {
|
||||
|
||||
/** A data source that fails the test by being used: the refusal must precede any connection. */
|
||||
private final DataSource dataSource = new RefusingDataSource();
|
||||
|
||||
private final JdbcInboxRepository repository = new JdbcInboxRepository(dataSource);
|
||||
|
||||
@AfterEach
|
||||
void clearTransactionState() {
|
||||
TransactionSynchronizationManager.clear();
|
||||
if (TransactionSynchronizationManager.hasResource(dataSource)) {
|
||||
TransactionSynchronizationManager.unbindResource(dataSource);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("reserving outside a transaction is refused before a connection is opened")
|
||||
void reservingOutsideATransactionIsRefused() {
|
||||
assertThatThrownBy(() -> repository.reserve(MessageId.newId(), "order-projection", NOW))
|
||||
.isInstanceOf(MessagingConfigurationException.class)
|
||||
.hasMessageContaining("requires an active transaction");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a read-only transaction is refused")
|
||||
void aReadOnlyTransactionIsRefused() {
|
||||
TransactionSynchronizationManager.setActualTransactionActive(true);
|
||||
TransactionSynchronizationManager.setCurrentTransactionReadOnly(true);
|
||||
|
||||
assertThatThrownBy(() -> repository.reserve(MessageId.newId(), "order-projection", NOW))
|
||||
.isInstanceOf(MessagingConfigurationException.class)
|
||||
.hasMessageContaining("read-only");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a transaction bound to another DataSource is refused")
|
||||
void aTransactionOnAnotherDataSourceIsRefused() {
|
||||
TransactionSynchronizationManager.setActualTransactionActive(true);
|
||||
|
||||
assertThatThrownBy(() -> repository.reserve(MessageId.newId(), "order-projection", NOW))
|
||||
.as("two DataSources means two commits, which is the atomicity this exists to provide")
|
||||
.isInstanceOf(MessagingConfigurationException.class)
|
||||
.hasMessageContaining("bound to this DataSource");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("the refusal names a code an operator can search for")
|
||||
void theRefusalNamesACode() {
|
||||
assertThatThrownBy(() -> repository.reserve(MessageId.newId(), "order-projection", NOW))
|
||||
.isInstanceOfSatisfying(
|
||||
MessagingConfigurationException.class,
|
||||
failure ->
|
||||
assertThat(failure.failure().code()).isEqualTo("INBOX_TRANSACTION_REQUIRED"));
|
||||
}
|
||||
|
||||
private static final Instant NOW = Instant.parse("2026-08-14T00:00:00Z");
|
||||
|
||||
/** Throws if anything asks it for a connection. */
|
||||
private static final class RefusingDataSource
|
||||
extends org.springframework.jdbc.datasource.AbstractDataSource {
|
||||
|
||||
@Override
|
||||
public java.sql.Connection getConnection() {
|
||||
throw new AssertionError(
|
||||
"the refusal must happen before a connection is requested; opening one is the defect");
|
||||
}
|
||||
|
||||
@Override
|
||||
public java.sql.Connection getConnection(String username, String password) {
|
||||
return getConnection();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -34,9 +34,9 @@ commons-logging:commons-logging:1.3.5=runtimeClasspath,testRuntimeClasspath
|
||||
info.picocli:picocli:4.7.7=checkstyle
|
||||
io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnotationProcessor
|
||||
io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor
|
||||
io.micrometer:micrometer-commons:1.16.0=runtimeClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-core:1.16.0=runtimeClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-observation:1.16.0=runtimeClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-commons:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-core:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-observation:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
javax.inject:javax.inject:1=annotationProcessor,testAnnotationProcessor
|
||||
jaxen:jaxen:2.0.0=spotbugs
|
||||
net.bytebuddy:byte-buddy:1.17.8=testCompileClasspath,testRuntimeClasspath
|
||||
@@ -65,7 +65,7 @@ org.codehaus.plexus:plexus-utils:3.3.0=checkstyle
|
||||
org.dom4j:dom4j:2.2.0=spotbugs
|
||||
org.hdrhistogram:HdrHistogram:2.2.2=runtimeClasspath,testRuntimeClasspath
|
||||
org.javassist:javassist:3.28.0-GA=checkstyle
|
||||
org.jspecify:jspecify:1.0.0=annotationProcessor,checkstyle,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
|
||||
org.jspecify:jspecify:1.0.0=annotationProcessor,checkstyle,compileClasspath,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.jupiter:junit-jupiter-api:6.0.1=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.jupiter:junit-jupiter-engine:6.0.1=testRuntimeClasspath
|
||||
org.junit.jupiter:junit-jupiter-params:6.0.1=testCompileClasspath,testRuntimeClasspath
|
||||
|
||||
@@ -10,7 +10,10 @@ dependencies {
|
||||
api project(':messaging:messaging-admin-api')
|
||||
|
||||
implementation 'org.springframework.kafka:spring-kafka'
|
||||
implementation 'org.apache.kafka:kafka-clients'
|
||||
// api: the public constructors and mappers take and return Producer, ProducerRecord and
|
||||
// RecordMetadata. Hiding the dependency does not hide the types — it only stops a consumer
|
||||
// from compiling against the API this module documents.
|
||||
api 'org.apache.kafka:kafka-clients'
|
||||
|
||||
// Test-only. The shared adapter contract is what makes "Stable" mean something, so the Kafka
|
||||
// adapter runs it rather than asserting its own behaviour in its own terms.
|
||||
|
||||
@@ -42,7 +42,7 @@ info.picocli:picocli:4.7.7=checkstyle
|
||||
io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,jmhAnnotationProcessor,testAnnotationProcessor
|
||||
io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,jmhAnnotationProcessor,testAnnotationProcessor
|
||||
io.micrometer:micrometer-commons:1.16.0=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-core:1.16.0=jmhRuntimeClasspath,runtimeClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-core:1.16.0=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-observation:1.16.0=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
javax.inject:javax.inject:1=annotationProcessor,jmhAnnotationProcessor,testAnnotationProcessor
|
||||
jaxen:jaxen:2.0.0=spotbugs
|
||||
|
||||
+14
@@ -32,6 +32,20 @@ public final class ContiguousPartitionOffsetTracker implements PartitionOffsetTr
|
||||
delivered.computeIfAbsent(partition, key -> new ConcurrentSkipListSet<>()).add(offset);
|
||||
}
|
||||
|
||||
/**
|
||||
* Undoes a delivery registration for a record that was never submitted.
|
||||
*
|
||||
* <p>Needed because the executor can refuse the work after the offset has been registered. A
|
||||
* delivered offset with no worker behind it holds the contiguous watermark back forever: nothing
|
||||
* will ever complete it, so the partition stops committing while continuing to consume.
|
||||
*/
|
||||
public void forgetDelivered(TopicPartition partition, long offset) {
|
||||
NavigableSet<Long> inFlight = delivered.get(partition);
|
||||
if (inFlight != null) {
|
||||
inFlight.remove(offset);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public void completed(TopicPartition partition, long offset) {
|
||||
completed.computeIfAbsent(partition, key -> new ConcurrentSkipListSet<>()).add(offset);
|
||||
|
||||
+340
-76
@@ -64,7 +64,36 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
|
||||
private final GracefulShutdownCoordinator shutdown;
|
||||
private final Duration pollTimeout;
|
||||
|
||||
private final KafkaQuarantinePublisher quarantine;
|
||||
|
||||
private final Map<TopicPartition, Long> committed = new ConcurrentHashMap<>();
|
||||
|
||||
/**
|
||||
* The assignment generation of each owned partition.
|
||||
*
|
||||
* <p>A settlement carries the epoch it was created under. Without one, a handler that finished
|
||||
* after its partition was revoked and reassigned applied its outcome to the new assignment: the
|
||||
* old epoch's acknowledgement advanced a watermark over offsets the new owner had not processed.
|
||||
*/
|
||||
private final Map<TopicPartition, Long> epochs = new ConcurrentHashMap<>();
|
||||
|
||||
private final java.util.concurrent.atomic.AtomicLong epochSequence =
|
||||
new java.util.concurrent.atomic.AtomicLong();
|
||||
|
||||
private final java.util.concurrent.atomic.AtomicInteger staleSettlements =
|
||||
new java.util.concurrent.atomic.AtomicInteger();
|
||||
|
||||
/**
|
||||
* Work that must run on the poll thread, queued by any other thread.
|
||||
*
|
||||
* <p>{@code pause}, {@code resume} and {@code close} are public API and were called on the
|
||||
* caller's thread, touching the {@code Consumer} directly — the one object this class documents
|
||||
* as poll-thread-only. The Kafka consumer is not thread-safe, and a concurrent call is a {@code
|
||||
* ConcurrentModificationException} at best and a corrupted subscription at worst.
|
||||
*/
|
||||
private final java.util.Queue<Runnable> control =
|
||||
new java.util.concurrent.ConcurrentLinkedQueue<>();
|
||||
|
||||
private final AtomicBoolean active = new AtomicBoolean(true);
|
||||
|
||||
/**
|
||||
@@ -82,6 +111,23 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
|
||||
Executor handlerPool,
|
||||
GracefulShutdownCoordinator shutdown,
|
||||
Duration pollTimeout) {
|
||||
this(
|
||||
spec, consumer, handlerPool, shutdown, pollTimeout, KafkaQuarantinePublisher.unavailable());
|
||||
}
|
||||
|
||||
/**
|
||||
* A consumer runtime with a quarantine destination for undecodable records.
|
||||
*
|
||||
* @param quarantine where a record that cannot be decoded is written before its offset commits
|
||||
*/
|
||||
public KafkaConsumerRegistrar(
|
||||
TransportConsumerSpec spec,
|
||||
Consumer<byte[], byte[]> consumer,
|
||||
Executor handlerPool,
|
||||
GracefulShutdownCoordinator shutdown,
|
||||
Duration pollTimeout,
|
||||
KafkaQuarantinePublisher quarantine) {
|
||||
this.quarantine = Objects.requireNonNull(quarantine, "quarantine must not be null");
|
||||
this.spec = Objects.requireNonNull(spec, "spec must not be null");
|
||||
this.consumer = Objects.requireNonNull(consumer, "consumer must not be null");
|
||||
this.handlerPool = Objects.requireNonNull(handlerPool, "handlerPool must not be null");
|
||||
@@ -112,6 +158,10 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
|
||||
new ConsumerRebalanceListener() {
|
||||
@Override
|
||||
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
|
||||
// Drop the epoch first. Any settlement still in flight for these partitions now
|
||||
// carries a number no live assignment has, so applySettlements refuses it instead of
|
||||
// moving a watermark this consumer no longer owns.
|
||||
partitions.forEach(epochs::remove);
|
||||
commitContiguous();
|
||||
partitions.forEach(
|
||||
partition -> {
|
||||
@@ -124,7 +174,9 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
|
||||
|
||||
@Override
|
||||
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
|
||||
// Nothing to restore: the broker's committed offset is the source of truth.
|
||||
// A fresh epoch per assignment. The broker's committed offset is still the source of
|
||||
// truth for where to read; the epoch decides whose settlements may be applied.
|
||||
partitions.forEach(partition -> epochs.put(partition, epochSequence.incrementAndGet()));
|
||||
}
|
||||
});
|
||||
}
|
||||
@@ -143,6 +195,8 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
|
||||
public int pollOnce(Instant now) {
|
||||
Objects.requireNonNull(now, "now must not be null");
|
||||
|
||||
drainControl();
|
||||
syncEpochs();
|
||||
applySettlements();
|
||||
applyDueResumes(now);
|
||||
commitContiguous();
|
||||
@@ -153,61 +207,142 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
|
||||
|
||||
ConsumerRecords<byte[], byte[]> records = consumer.poll(pollTimeout);
|
||||
int dispatched = 0;
|
||||
for (ConsumerRecord<byte[], byte[]> record : records) {
|
||||
TopicPartition partition = new TopicPartition(record.topic(), record.partition());
|
||||
|
||||
if (!coordinator.tryAcquire(partition)) {
|
||||
// At the ceiling: stop fetching and re-read this record on a later cycle.
|
||||
// Per partition, and never a global break.
|
||||
//
|
||||
// The loop used to iterate the whole batch and `break` on the first record it could not
|
||||
// submit — which abandoned every record the same poll had returned for *other* partitions.
|
||||
// The consumer's position had already advanced past them, so those offsets were not read
|
||||
// again until a rebalance or a restart: a processing gap that nothing reported.
|
||||
for (TopicPartition partition : records.partitions()) {
|
||||
Long epoch = epochs.get(partition);
|
||||
if (epoch == null) {
|
||||
// Not ours: revoked between the poll and now.
|
||||
continue;
|
||||
}
|
||||
java.util.List<ConsumerRecord<byte[], byte[]>> partitionRecords = records.records(partition);
|
||||
long seekBackTo = -1L;
|
||||
for (ConsumerRecord<byte[], byte[]> record : partitionRecords) {
|
||||
if (seekBackTo >= 0) {
|
||||
// Already stopped for this partition; the seek below covers this record too.
|
||||
continue;
|
||||
}
|
||||
if (!coordinator.tryAcquire(partition)) {
|
||||
seekBackTo = record.offset();
|
||||
continue;
|
||||
}
|
||||
if (!shutdown.tryBeginWork()) {
|
||||
coordinator.release(partition);
|
||||
seekBackTo = record.offset();
|
||||
continue;
|
||||
}
|
||||
offsets.delivered(partition, record.offset());
|
||||
if (!dispatch(record, partition, now, epoch)) {
|
||||
// The pool refused the work. Undo everything this iteration took so the in-flight count
|
||||
// and the delivered set do not drift, and re-read the record next cycle.
|
||||
offsets.forgetDelivered(partition, record.offset());
|
||||
coordinator.release(partition);
|
||||
shutdown.endWork();
|
||||
seekBackTo = record.offset();
|
||||
continue;
|
||||
}
|
||||
dispatched++;
|
||||
}
|
||||
if (seekBackTo >= 0) {
|
||||
// Only this partition pauses. Its earliest unsubmitted offset is where it resumes.
|
||||
consumer.pause(Set.of(partition));
|
||||
consumer.seek(partition, record.offset());
|
||||
break;
|
||||
consumer.seek(partition, seekBackTo);
|
||||
}
|
||||
if (!shutdown.tryBeginWork()) {
|
||||
coordinator.release(partition);
|
||||
consumer.seek(partition, record.offset());
|
||||
break;
|
||||
}
|
||||
|
||||
offsets.delivered(partition, record.offset());
|
||||
dispatch(record, partition, now);
|
||||
dispatched++;
|
||||
}
|
||||
return dispatched;
|
||||
}
|
||||
|
||||
private void dispatch(
|
||||
ConsumerRecord<byte[], byte[]> record, TopicPartition partition, Instant now) {
|
||||
/**
|
||||
* Hands one record to the pool.
|
||||
*
|
||||
* @return false when the pool refused the work, so the caller can undo what it took
|
||||
*/
|
||||
private boolean dispatch(
|
||||
ConsumerRecord<byte[], byte[]> record, TopicPartition partition, Instant now, long epoch) {
|
||||
DestinationProfile profile = spec.profile();
|
||||
DestinationName destination = profile.name();
|
||||
Duration handlerTimeout = profile.consumer().handlerTimeout();
|
||||
|
||||
handlerPool.execute(
|
||||
() -> {
|
||||
try {
|
||||
var envelope = deliveryMapper.toEnvelope(record);
|
||||
int attempt = retryMetadataMapper.attemptOf(envelope);
|
||||
TransportDelivery delivery =
|
||||
new TransportDelivery(
|
||||
envelope,
|
||||
deliveryMapper.toMetadata(
|
||||
destination, record, attempt, profile.consumer().group().orElse(null), now),
|
||||
new QueuedSettlement(partition, record.offset()));
|
||||
spec.sink().apply(delivery).toCompletableFuture().join();
|
||||
} catch (RuntimeException exception) {
|
||||
// The sink owns failure classification. A throw here means the delivery could not even
|
||||
// be decoded, which the platform parks rather than retries.
|
||||
settlements.enqueue(
|
||||
new KafkaSettlementCommand(
|
||||
partition, record.offset(), KafkaSettlementCommand.Kind.PARKED));
|
||||
} finally {
|
||||
coordinator.release(partition);
|
||||
shutdown.endWork();
|
||||
}
|
||||
});
|
||||
try {
|
||||
handlerPool.execute(
|
||||
() -> {
|
||||
QueuedSettlement settlement = new QueuedSettlement(partition, record.offset(), epoch);
|
||||
dev.caskeleton.messaging.api.MessageEnvelope<
|
||||
dev.caskeleton.messaging.schema.EncodedMessage>
|
||||
envelope;
|
||||
try {
|
||||
envelope = deliveryMapper.toEnvelope(record);
|
||||
} catch (RuntimeException undecodable) {
|
||||
// A decode failure is not a handler failure and must not be retried forever. The
|
||||
// offset may only advance once the record exists somewhere else, so an unavailable
|
||||
// quarantine leaves it uncommitted rather than dropping the message.
|
||||
if (quarantine.quarantine(record, undecodable)) {
|
||||
settlement.acknowledgeAfterQuarantine();
|
||||
} else {
|
||||
settlement.requeueAfterFailure();
|
||||
}
|
||||
coordinator.release(partition);
|
||||
shutdown.endWork();
|
||||
return;
|
||||
}
|
||||
try {
|
||||
int attempt = retryMetadataMapper.attemptOf(envelope);
|
||||
TransportDelivery delivery =
|
||||
new TransportDelivery(
|
||||
envelope,
|
||||
deliveryMapper.toMetadata(
|
||||
destination,
|
||||
record,
|
||||
attempt,
|
||||
profile.consumer().group().orElse(null),
|
||||
now),
|
||||
settlement);
|
||||
// The declared handler timeout is applied. It existed as a policy field and the
|
||||
// worker blocked on an unbounded join, so a handler that never returned held its
|
||||
// partition's permit and its offset forever — the partition simply stopped.
|
||||
spec.sink()
|
||||
.apply(delivery)
|
||||
.toCompletableFuture()
|
||||
.get(handlerTimeout.toMillis(), java.util.concurrent.TimeUnit.MILLISECONDS);
|
||||
} catch (java.util.concurrent.TimeoutException timedOut) {
|
||||
// Unsettled work, not a completion: pause and re-read rather than advancing past a
|
||||
// message whose handler may still be running.
|
||||
settlement.requeueAfterFailure();
|
||||
} catch (InterruptedException interrupted) {
|
||||
Thread.currentThread().interrupt();
|
||||
settlement.requeueAfterFailure();
|
||||
} catch (java.util.concurrent.ExecutionException | RuntimeException failure) {
|
||||
// A handler failure is retried, not parked. PARKED completed the offset without any
|
||||
// quarantine or dead-letter write, so a transient database outage silently discarded
|
||||
// every message it touched.
|
||||
settlement.requeueAfterFailure();
|
||||
} finally {
|
||||
coordinator.release(partition);
|
||||
shutdown.endWork();
|
||||
}
|
||||
});
|
||||
return true;
|
||||
} catch (java.util.concurrent.RejectedExecutionException refused) {
|
||||
// The caller undoes the permit, the work count and the delivered registration.
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/** Applies everything workers queued. Poll thread only. */
|
||||
private void applySettlements() {
|
||||
for (KafkaSettlementCommand command : settlements.drain()) {
|
||||
Long currentEpoch = epochs.get(command.partition());
|
||||
if (currentEpoch == null || currentEpoch != command.epoch()) {
|
||||
// The partition was revoked, or reassigned, after this settlement was created. Applying it
|
||||
// would move the current owner's watermark using an outcome from the previous assignment,
|
||||
// over offsets the current owner has not processed.
|
||||
staleSettlements.incrementAndGet();
|
||||
continue;
|
||||
}
|
||||
switch (command.kind()) {
|
||||
case COMPLETE, PARKED -> offsets.completed(command.partition(), command.offset());
|
||||
case PAUSE_AND_SEEK -> {
|
||||
@@ -222,6 +357,21 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Keeps the epoch map in step with the current assignment.
|
||||
*
|
||||
* <p>The rebalance listener is the usual source of epochs, but it is not the only way a partition
|
||||
* becomes owned: a manually assigned consumer never fires it. Deriving the epoch from the
|
||||
* assignment as well means fencing works in both cases, and a partition that has gone away loses
|
||||
* its epoch here even if no revoke callback arrived.
|
||||
*/
|
||||
private void syncEpochs() {
|
||||
Set<TopicPartition> current = assignment();
|
||||
current.forEach(
|
||||
partition -> epochs.computeIfAbsent(partition, key -> epochSequence.incrementAndGet()));
|
||||
epochs.keySet().removeIf(partition -> !current.contains(partition));
|
||||
}
|
||||
|
||||
private void applyDueResumes(Instant now) {
|
||||
Map<TopicPartition, Long> due = retries.dueForResume(now);
|
||||
due.forEach(
|
||||
@@ -246,12 +396,32 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
|
||||
continue;
|
||||
}
|
||||
toCommit.put(partition, new OffsetAndMetadata(offset));
|
||||
committed.put(partition, offset);
|
||||
offsets.pruneThrough(partition, offset - 1);
|
||||
}
|
||||
if (!toCommit.isEmpty()) {
|
||||
consumer.commitSync(toCommit);
|
||||
if (toCommit.isEmpty()) {
|
||||
return;
|
||||
}
|
||||
// Commit first, then record it locally.
|
||||
//
|
||||
// The local map and the tracker used to be updated before commitSync. A failed commit
|
||||
// therefore left this consumer believing an offset was committed that the broker had never
|
||||
// accepted, and the pruned tracker could no longer produce it — so the next successful commit
|
||||
// jumped over the gap and the records in it were never redelivered.
|
||||
consumer.commitSync(toCommit);
|
||||
toCommit.forEach(
|
||||
(partition, metadata) -> {
|
||||
committed.put(partition, metadata.offset());
|
||||
offsets.pruneThrough(partition, metadata.offset() - 1);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* How many settlements were refused because their assignment had been revoked.
|
||||
*
|
||||
* <p>Exposed rather than counted silently: a non-zero value means handlers are outliving their
|
||||
* assignment, which is a rebalance or a handler-timeout problem an operator can act on.
|
||||
*/
|
||||
public int staleSettlements() {
|
||||
return staleSettlements.get();
|
||||
}
|
||||
|
||||
private Set<TopicPartition> assignment() {
|
||||
@@ -271,18 +441,52 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
|
||||
|
||||
@Override
|
||||
public CompletionStage<Void> pause(String scope) {
|
||||
Set<TopicPartition> partitions = scoped(scope);
|
||||
partitions.forEach(coordinator::pause);
|
||||
consumer.pause(partitions);
|
||||
return CompletableFuture.completedFuture(null);
|
||||
return onPollThread(
|
||||
() -> {
|
||||
Set<TopicPartition> partitions = scoped(scope);
|
||||
partitions.forEach(coordinator::pause);
|
||||
consumer.pause(partitions);
|
||||
});
|
||||
}
|
||||
|
||||
@Override
|
||||
public CompletionStage<Void> resume(String scope) {
|
||||
Set<TopicPartition> partitions = scoped(scope);
|
||||
partitions.forEach(coordinator::resume);
|
||||
consumer.resume(partitions);
|
||||
return CompletableFuture.completedFuture(null);
|
||||
return onPollThread(
|
||||
() -> {
|
||||
Set<TopicPartition> partitions = scoped(scope);
|
||||
partitions.forEach(coordinator::resume);
|
||||
consumer.resume(partitions);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Queues work for the poll thread and reports when it ran.
|
||||
*
|
||||
* <p>The returned stage completes after the poll loop has actually applied the change, so a
|
||||
* caller that awaits it knows the consumer is paused rather than merely asked to pause.
|
||||
*
|
||||
* <p>It therefore completes on the <em>next poll cycle</em>. A caller that blocks on it without a
|
||||
* running loop waits forever, which is the honest shape of the request: there is no safe way to
|
||||
* touch the consumer from another thread, so "paused" cannot be true until the loop says so.
|
||||
*/
|
||||
private CompletionStage<Void> onPollThread(Runnable work) {
|
||||
CompletableFuture<Void> applied = new CompletableFuture<>();
|
||||
control.add(
|
||||
() -> {
|
||||
try {
|
||||
work.run();
|
||||
applied.complete(null);
|
||||
} catch (RuntimeException failure) {
|
||||
applied.completeExceptionally(failure);
|
||||
}
|
||||
});
|
||||
return applied;
|
||||
}
|
||||
|
||||
private void drainControl() {
|
||||
for (Runnable work = control.poll(); work != null; work = control.poll()) {
|
||||
work.run();
|
||||
}
|
||||
}
|
||||
|
||||
private Set<TopicPartition> scoped(String scope) {
|
||||
@@ -302,56 +506,116 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
|
||||
@Override
|
||||
public void close() {
|
||||
if (active.compareAndSet(true, false)) {
|
||||
// Close still runs here rather than through the queue: after this the poll loop stops, so
|
||||
// queued work would never drain. What it does not do is race an in-progress poll — callers
|
||||
// stop the loop first, and `active` makes a second close a no-op.
|
||||
drainControl();
|
||||
applySettlements();
|
||||
commitContiguous();
|
||||
consumer.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** The settlement handle handed to a worker; it only enqueues. */
|
||||
/**
|
||||
* The settlement handle handed to a worker.
|
||||
*
|
||||
* <p>Two things changed. It carries the assignment epoch, so a settlement that arrives after a
|
||||
* revoke is refused rather than applied to whoever owns the partition now. And it is
|
||||
* one-terminal: acknowledge, requeue and discard compete for a single CAS, because a handler that
|
||||
* acknowledged and then requeued used to tell the poll thread both, and the poll thread believed
|
||||
* the second one.
|
||||
*/
|
||||
private final class QueuedSettlement
|
||||
implements dev.caskeleton.messaging.transport.TransportSettlement {
|
||||
|
||||
private final TopicPartition partition;
|
||||
private final long offset;
|
||||
private final long epoch;
|
||||
private final java.util.concurrent.atomic.AtomicReference<String> terminal =
|
||||
new java.util.concurrent.atomic.AtomicReference<>();
|
||||
|
||||
private QueuedSettlement(TopicPartition partition, long offset) {
|
||||
private QueuedSettlement(TopicPartition partition, long offset, long epoch) {
|
||||
this.partition = partition;
|
||||
this.offset = offset;
|
||||
this.epoch = epoch;
|
||||
}
|
||||
|
||||
@Override
|
||||
public CompletionStage<dev.caskeleton.messaging.api.settlement.SettlementResult> acknowledge() {
|
||||
settlements.enqueue(
|
||||
new KafkaSettlementCommand(partition, offset, KafkaSettlementCommand.Kind.COMPLETE));
|
||||
return CompletableFuture.completedFuture(
|
||||
dev.caskeleton.messaging.api.settlement.SettlementResult.settled());
|
||||
return terminalOnce(
|
||||
"acknowledge",
|
||||
() -> {
|
||||
settlements.enqueue(
|
||||
new KafkaSettlementCommand(
|
||||
partition, offset, KafkaSettlementCommand.Kind.COMPLETE, epoch));
|
||||
return dev.caskeleton.messaging.api.settlement.SettlementResult.settled();
|
||||
});
|
||||
}
|
||||
|
||||
@Override
|
||||
public CompletionStage<dev.caskeleton.messaging.api.settlement.SettlementResult> requeue(
|
||||
Duration delay) {
|
||||
retries.pauseUntil(partition, offset, delay, Instant.now());
|
||||
settlements.enqueue(
|
||||
new KafkaSettlementCommand(
|
||||
partition, offset, KafkaSettlementCommand.Kind.PAUSE_AND_SEEK));
|
||||
return CompletableFuture.completedFuture(
|
||||
new dev.caskeleton.messaging.api.settlement.SettlementResult(
|
||||
dev.caskeleton.messaging.api.settlement.SettlementCompletion.UNKNOWN,
|
||||
dev.caskeleton.messaging.api.settlement.SettlementEvidence.unknown(),
|
||||
java.util.Optional.of(
|
||||
dev.caskeleton.messaging.api.error.FailureDescriptor.of(
|
||||
dev.caskeleton.messaging.api.error.FailureCategory.PROCESSING_TRANSIENT,
|
||||
"KAFKA_PAUSED_FOR_RETRY",
|
||||
"the partition was paused and will be re-read"))));
|
||||
return terminalOnce("requeue", () -> enqueueRequeue(delay));
|
||||
}
|
||||
|
||||
@Override
|
||||
public CompletionStage<dev.caskeleton.messaging.api.settlement.SettlementResult> discard() {
|
||||
return terminalOnce(
|
||||
"discard",
|
||||
() -> {
|
||||
settlements.enqueue(
|
||||
new KafkaSettlementCommand(
|
||||
partition, offset, KafkaSettlementCommand.Kind.PARKED, epoch));
|
||||
return dev.caskeleton.messaging.api.settlement.SettlementResult.settled();
|
||||
});
|
||||
}
|
||||
|
||||
/** Used by the worker once an undecodable record has been written to quarantine. */
|
||||
private void acknowledgeAfterQuarantine() {
|
||||
if (terminal.compareAndSet(null, "acknowledge")) {
|
||||
settlements.enqueue(
|
||||
new KafkaSettlementCommand(
|
||||
partition, offset, KafkaSettlementCommand.Kind.PARKED, epoch));
|
||||
}
|
||||
}
|
||||
|
||||
/** Used by the worker when the handler failed or timed out and never settled. */
|
||||
private void requeueAfterFailure() {
|
||||
if (terminal.compareAndSet(null, "requeue")) {
|
||||
enqueueRequeue(Duration.ZERO);
|
||||
}
|
||||
}
|
||||
|
||||
private dev.caskeleton.messaging.api.settlement.SettlementResult enqueueRequeue(
|
||||
Duration delay) {
|
||||
retries.pauseUntil(partition, offset, delay, Instant.now());
|
||||
settlements.enqueue(
|
||||
new KafkaSettlementCommand(partition, offset, KafkaSettlementCommand.Kind.PARKED));
|
||||
return CompletableFuture.completedFuture(
|
||||
dev.caskeleton.messaging.api.settlement.SettlementResult.settled());
|
||||
new KafkaSettlementCommand(
|
||||
partition, offset, KafkaSettlementCommand.Kind.PAUSE_AND_SEEK, epoch));
|
||||
return new dev.caskeleton.messaging.api.settlement.SettlementResult(
|
||||
dev.caskeleton.messaging.api.settlement.SettlementCompletion.UNKNOWN,
|
||||
dev.caskeleton.messaging.api.settlement.SettlementEvidence.unknown(),
|
||||
java.util.Optional.of(
|
||||
dev.caskeleton.messaging.api.error.FailureDescriptor.of(
|
||||
dev.caskeleton.messaging.api.error.FailureCategory.PROCESSING_TRANSIENT,
|
||||
"KAFKA_PAUSED_FOR_RETRY",
|
||||
"the partition was paused and will be re-read")));
|
||||
}
|
||||
|
||||
private CompletionStage<dev.caskeleton.messaging.api.settlement.SettlementResult> terminalOnce(
|
||||
String what,
|
||||
java.util.function.Supplier<dev.caskeleton.messaging.api.settlement.SettlementResult>
|
||||
call) {
|
||||
if (!terminal.compareAndSet(null, what)) {
|
||||
return CompletableFuture.failedFuture(
|
||||
new IllegalStateException(
|
||||
"this delivery was already settled by "
|
||||
+ terminal.get()
|
||||
+ "; a second "
|
||||
+ what
|
||||
+ " would contradict what the poll thread has already been told"));
|
||||
}
|
||||
return CompletableFuture.completedFuture(call.get());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+21
-1
@@ -67,7 +67,27 @@ public final class KafkaHeaderMapper {
|
||||
.baggage()
|
||||
.ifPresent(value -> put(headers, ReservedHeaders.BAGGAGE, value));
|
||||
|
||||
envelope.headers().asMap().forEach((name, value) -> put(headers, name.value(), value.value()));
|
||||
// Application headers are written last and must not be able to reach a reserved name.
|
||||
//
|
||||
// They used to be appended after the canonical ones, and the consumer reads `lastHeader` — so a
|
||||
// row in the outbox carrying `msg.id` overwrote the canonical message identity on the wire.
|
||||
// That identity is what inbox deduplication and the audit trail key on, so a forged one
|
||||
// corrupts another message's dedup state rather than merely its own.
|
||||
envelope
|
||||
.headers()
|
||||
.asMap()
|
||||
.forEach(
|
||||
(name, value) -> {
|
||||
if (ReservedHeaders.isReserved(name.value())) {
|
||||
throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException(
|
||||
"RESERVED_HEADER_FORGED",
|
||||
"application header '"
|
||||
+ name.value()
|
||||
+ "' uses the reserved namespace; the canonical value is written by the"
|
||||
+ " platform and must not be overridable by message content");
|
||||
}
|
||||
put(headers, name.value(), value.value());
|
||||
});
|
||||
|
||||
return headers;
|
||||
}
|
||||
|
||||
+20
-1
@@ -40,9 +40,28 @@ import org.apache.kafka.clients.producer.ProducerRecord;
|
||||
*/
|
||||
public final class KafkaMessagingTransport implements MessagingTransport {
|
||||
|
||||
/**
|
||||
* What this adapter can prove, with deduplicated publish set to false.
|
||||
*
|
||||
* <p>It was {@code true}, justified by producer idempotence. Producer idempotence deduplicates
|
||||
* <em>sequence retries within one producer session</em>: the producer id is reassigned on
|
||||
* restart, so the same logical message published again after a crash is a new sequence and the
|
||||
* broker stores it twice. The capability an application reads is "the broker suppresses duplicate
|
||||
* publishes of a stable id", which needs a persistent message-id store that this platform does
|
||||
* not have.
|
||||
*
|
||||
* <p>Declaring it true means {@code PublishDeduplication} is accepted and silently does nothing —
|
||||
* the caller believes the broker is deduplicating and skips the idempotency it would otherwise
|
||||
* build. False makes that request a startup failure instead, which is the whole reason the flag
|
||||
* exists.
|
||||
*
|
||||
* <p>The order of these flags is the record's: brokerAcknowledgement, replication evidence,
|
||||
* per-message settlement, batch settlement, ordered stream, keyed ordering, replay, delayed
|
||||
* delivery, broker transaction, deduplicated publish, native dead letter, topology management.
|
||||
*/
|
||||
private static final MessagingCapabilities CAPABILITIES =
|
||||
new MessagingCapabilities(
|
||||
true, true, true, true, true, true, true, false, true, true, false, true);
|
||||
true, true, true, true, true, true, true, false, true, false, false, true);
|
||||
|
||||
private final String brokerName;
|
||||
private final long generation;
|
||||
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
package dev.caskeleton.messaging.kafka;
|
||||
|
||||
import org.apache.kafka.clients.consumer.ConsumerRecord;
|
||||
|
||||
/**
|
||||
* Where a record that cannot be decoded is written before its offset is committed.
|
||||
*
|
||||
* <p>The consumer used to mark an undecodable record {@code PARKED}, and the poll thread treated
|
||||
* {@code PARKED} exactly like {@code COMPLETE} — the offset advanced and the record was gone. There
|
||||
* was no quarantine collection, no dead-letter publish and no record of what had been dropped, so
|
||||
* "parked" named a state nothing implemented.
|
||||
*
|
||||
* <p>Committing past a record is only safe once the record exists somewhere else. An implementation
|
||||
* that cannot write returns {@code false}, and the consumer leaves the offset uncommitted rather
|
||||
* than losing the message quietly.
|
||||
*/
|
||||
@FunctionalInterface
|
||||
public interface KafkaQuarantinePublisher {
|
||||
|
||||
/**
|
||||
* Writes an undecodable record somewhere durable.
|
||||
*
|
||||
* @return true when the write is confirmed, so the source offset may be committed
|
||||
*/
|
||||
boolean quarantine(ConsumerRecord<byte[], byte[]> record, Throwable reason);
|
||||
|
||||
/** The default: nothing is configured, so nothing may be committed past. */
|
||||
static KafkaQuarantinePublisher unavailable() {
|
||||
return (record, reason) -> false;
|
||||
}
|
||||
}
|
||||
+16
-3
@@ -72,9 +72,22 @@ public final class KafkaRetryMetadataMapper {
|
||||
return 1;
|
||||
}
|
||||
try {
|
||||
return Math.max(1, Integer.parseInt(value.orElseThrow().value()));
|
||||
} catch (NumberFormatException exception) {
|
||||
return 1;
|
||||
int attempt = Integer.parseInt(value.orElseThrow().value());
|
||||
if (attempt < 1) {
|
||||
throw new NumberFormatException("attempt " + attempt);
|
||||
}
|
||||
return attempt;
|
||||
} catch (NumberFormatException malformed) {
|
||||
// Fail closed rather than resetting to 1.
|
||||
//
|
||||
// Returning 1 for an unreadable header restarts the retry budget on every redelivery, so a
|
||||
// message that should have been dead-lettered after N attempts is retried forever — and the
|
||||
// header is caller-influenced, which makes "unreadable" a way to defeat the cap rather than
|
||||
// an accident.
|
||||
throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException(
|
||||
"RETRY_ATTEMPT_MALFORMED",
|
||||
"the retry attempt header is not a positive integer; the message is quarantined rather"
|
||||
+ " than restarting its retry budget");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+49
-11
@@ -93,11 +93,15 @@ public final class KafkaSecurityConfigurer {
|
||||
properties.put(SASL_JAAS_CONFIG, scramJaas(scram.credentialId(), resolved));
|
||||
}
|
||||
case BrokerCredentialProfile.OAuth2 oauth -> {
|
||||
credentials.resolve(oauth.credentialId(), now);
|
||||
properties.put(SASL_MECHANISM, "OAUTHBEARER");
|
||||
properties.put(
|
||||
SASL_JAAS_CONFIG,
|
||||
"org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;");
|
||||
// Refused rather than half-configured. Setting the mechanism name without a callback
|
||||
// handler produces a client that authenticates with nothing and fails at connect time,
|
||||
// which is later and harder to attribute than failing here.
|
||||
throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException(
|
||||
"SASL_OAUTH_NOT_CONFIGURED",
|
||||
"OAuth bearer needs a login callback handler and a token endpoint; this platform"
|
||||
+ " configures the mechanism name only, so the profile is refused rather than"
|
||||
+ " producing a client that cannot authenticate: "
|
||||
+ oauth.credentialId());
|
||||
}
|
||||
case BrokerCredentialProfile.MutualTls ignored -> properties.put(SASL_MECHANISM, "NONE");
|
||||
case BrokerCredentialProfile.UsernamePassword plain -> {
|
||||
@@ -122,14 +126,48 @@ public final class KafkaSecurityConfigurer {
|
||||
}
|
||||
|
||||
private static String scramJaas(String credentialId, CredentialRuntime resolved) {
|
||||
return "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"%s\" "
|
||||
.formatted(credentialId)
|
||||
+ "password=\"%s\";".formatted(new String(resolved.material()));
|
||||
return jaas("org.apache.kafka.common.security.scram.ScramLoginModule", credentialId, resolved);
|
||||
}
|
||||
|
||||
private static String plainJaas(String credentialId, CredentialRuntime resolved) {
|
||||
return "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"%s\" "
|
||||
.formatted(credentialId)
|
||||
+ "password=\"%s\";".formatted(new String(resolved.material()));
|
||||
return jaas("org.apache.kafka.common.security.plain.PlainLoginModule", credentialId, resolved);
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds a JAAS entry with both values escaped.
|
||||
*
|
||||
* <p>The password was interpolated into a quoted string unescaped. A perfectly ordinary generated
|
||||
* secret containing a quote or a backslash then produced a JAAS config that either fails to parse
|
||||
* — an authentication error that names nothing useful — or, with a quote and a semicolon,
|
||||
* terminates the option early and injects another one. A secret is attacker-influenced in exactly
|
||||
* one sense that matters here: nobody reads it before it is used.
|
||||
*/
|
||||
private static String jaas(String loginModule, String credentialId, CredentialRuntime resolved) {
|
||||
return loginModule
|
||||
+ " required username=\""
|
||||
+ escapeJaas(credentialId)
|
||||
+ "\" password=\""
|
||||
+ escapeJaas(new String(resolved.material()))
|
||||
+ "\";";
|
||||
}
|
||||
|
||||
/**
|
||||
* Escapes a JAAS quoted-string value.
|
||||
*
|
||||
* <p>Backslash first, then quote — the other order double-escapes the backslash it just added.
|
||||
* Control characters are refused rather than escaped: JAAS has no escape for them, and a secret
|
||||
* containing one cannot be expressed in this format at all.
|
||||
*/
|
||||
private static String escapeJaas(String value) {
|
||||
for (int index = 0; index < value.length(); index++) {
|
||||
char character = value.charAt(index);
|
||||
if (character < 0x20 || character == 0x7F) {
|
||||
throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException(
|
||||
"JAAS_VALUE_UNREPRESENTABLE",
|
||||
"a credential containing a control character cannot be expressed in a JAAS entry;"
|
||||
+ " configure the client with a callback handler instead");
|
||||
}
|
||||
}
|
||||
return value.replace("\\", "\\\\").replace("\"", "\\\"");
|
||||
}
|
||||
}
|
||||
|
||||
+12
-1
@@ -15,7 +15,18 @@ import org.apache.kafka.common.TopicPartition;
|
||||
* @param offset the record offset
|
||||
* @param kind what to do
|
||||
*/
|
||||
public record KafkaSettlementCommand(TopicPartition partition, long offset, Kind kind) {
|
||||
public record KafkaSettlementCommand(TopicPartition partition, long offset, Kind kind, long epoch) {
|
||||
|
||||
/**
|
||||
* A command without an epoch, for callers that predate assignment fencing.
|
||||
*
|
||||
* <p>Epoch 0 never matches a live assignment, so such a command is always stale. That is
|
||||
* deliberate: a settlement that cannot say which assignment it belongs to cannot be shown to
|
||||
* belong to the current one.
|
||||
*/
|
||||
public KafkaSettlementCommand(TopicPartition partition, long offset, Kind kind) {
|
||||
this(partition, offset, kind, 0L);
|
||||
}
|
||||
|
||||
/** What a queued settlement instruction asks the poll thread to do. */
|
||||
public enum Kind {
|
||||
|
||||
+61
@@ -78,6 +78,67 @@ public final class KafkaTransactionalPublisher {
|
||||
* @param profiles the destination profile for each output destination
|
||||
* @param inputOffsets the input offsets to commit with the outputs
|
||||
*/
|
||||
/**
|
||||
* Begins a transaction, runs the caller's work inside it, then sends outputs and offsets.
|
||||
*
|
||||
* <p>The processor used to run the handler and only afterwards hand the delivery here — so {@code
|
||||
* beginTransaction} happened *after* the handler had already finished. Anything the handler
|
||||
* produced itself, and any external effect it triggered, was therefore outside the transaction
|
||||
* the documentation promised it was in: a handler that succeeded and a commit that then failed
|
||||
* left the handler's work applied with its input offsets unsent.
|
||||
*
|
||||
* <p>The body runs between begin and commit, and every failure path aborts.
|
||||
*
|
||||
* @param body the handler, executed inside the transaction
|
||||
* @return whatever the body produced
|
||||
*/
|
||||
public <T> T inTransaction(
|
||||
KafkaTransactionalDelivery delivery,
|
||||
Map<String, DestinationProfile> profiles,
|
||||
Map<TopicPartition, OffsetAndMetadata> inputOffsets,
|
||||
java.util.function.Supplier<T> body) {
|
||||
Objects.requireNonNull(delivery, "delivery must not be null");
|
||||
Objects.requireNonNull(profiles, "profiles must not be null");
|
||||
Objects.requireNonNull(inputOffsets, "inputOffsets must not be null");
|
||||
Objects.requireNonNull(body, "body must not be null");
|
||||
|
||||
producer.beginTransaction();
|
||||
try {
|
||||
T value = body.get();
|
||||
sendOutputsAndOffsets(delivery, profiles, inputOffsets);
|
||||
producer.commitTransaction();
|
||||
return value;
|
||||
} catch (RuntimeException failure) {
|
||||
// Handler failure, send failure, offset failure and commit failure all land here. Under
|
||||
// read_committed a consumer sees none of the outputs, and the input offsets are unsent, so
|
||||
// the source is redelivered.
|
||||
producer.abortTransaction();
|
||||
throw failure;
|
||||
}
|
||||
}
|
||||
|
||||
@SuppressWarnings("FutureReturnValueIgnored")
|
||||
private void sendOutputsAndOffsets(
|
||||
KafkaTransactionalDelivery delivery,
|
||||
Map<String, DestinationProfile> profiles,
|
||||
Map<TopicPartition, OffsetAndMetadata> inputOffsets) {
|
||||
for (KafkaTransactionalOutput output : delivery.outputs()) {
|
||||
DestinationProfile profile = profiles.get(output.destination().value());
|
||||
if (profile == null) {
|
||||
throw new MessagingConfigurationException(
|
||||
"DESTINATION_NOT_REGISTERED",
|
||||
"no destination profile is registered for " + output.destination().value());
|
||||
}
|
||||
producer.send(
|
||||
mapper.toRecord(
|
||||
new dev.caskeleton.messaging.transport.TransportPublishRequest(
|
||||
profile,
|
||||
output.envelope(),
|
||||
dev.caskeleton.messaging.api.publish.PublishOptions.defaults())));
|
||||
}
|
||||
producer.sendOffsetsToTransaction(inputOffsets, groupMetadata.get());
|
||||
}
|
||||
|
||||
@SuppressWarnings("FutureReturnValueIgnored")
|
||||
public void sendInTransaction(
|
||||
KafkaTransactionalDelivery delivery,
|
||||
|
||||
+9
-2
@@ -51,8 +51,15 @@ public final class SpringKafkaTransactionalProcessor implements KafkaTransaction
|
||||
Objects.requireNonNull(handler, "handler must not be null");
|
||||
|
||||
try {
|
||||
T result = handler.apply(delivery);
|
||||
publisher.sendInTransaction(delivery, profiles, inputOffsets.apply(delivery));
|
||||
// The handler runs between beginTransaction and commitTransaction.
|
||||
//
|
||||
// It used to run first, with the transaction opened afterwards inside sendInTransaction — so
|
||||
// the contract this interface documents ("the callback executes inside the Kafka
|
||||
// transaction") was false for every output the handler produced itself and every external
|
||||
// effect it triggered.
|
||||
T result =
|
||||
publisher.inTransaction(
|
||||
delivery, profiles, inputOffsets.apply(delivery), () -> handler.apply(delivery));
|
||||
return CompletableFuture.completedFuture(result);
|
||||
} catch (RuntimeException exception) {
|
||||
return CompletableFuture.failedFuture(exception);
|
||||
|
||||
+181
-4
@@ -2,6 +2,7 @@ package dev.caskeleton.messaging.kafka;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import dev.caskeleton.messaging.api.MessageId;
|
||||
import dev.caskeleton.messaging.api.header.ReservedHeaders;
|
||||
import dev.caskeleton.messaging.transport.GracefulShutdownCoordinator;
|
||||
import dev.caskeleton.messaging.transport.TransportConsumerSpec;
|
||||
@@ -14,7 +15,6 @@ import java.util.ArrayList;
|
||||
import java.util.Deque;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.CompletionStage;
|
||||
import java.util.concurrent.Executor;
|
||||
@@ -22,6 +22,7 @@ import org.apache.kafka.clients.consumer.ConsumerRecord;
|
||||
import org.apache.kafka.clients.consumer.MockConsumer;
|
||||
import org.apache.kafka.common.TopicPartition;
|
||||
import org.apache.kafka.common.header.internals.RecordHeaders;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
@@ -157,10 +158,18 @@ class KafkaConsumerRegistrarTest {
|
||||
KafkaConsumerRegistrar registrar = registrar(this::settleImmediately);
|
||||
assign();
|
||||
|
||||
registrar.pause("").toCompletableFuture().join();
|
||||
// pause/resume are queued for the poll thread rather than touching the Consumer from the
|
||||
// caller's thread — the Kafka consumer is not thread-safe, and this class documents itself as
|
||||
// poll-thread-only. The stage therefore completes on the next poll cycle, which is what
|
||||
// "applied" means for a runtime whose loop is running.
|
||||
java.util.concurrent.CompletionStage<Void> paused = registrar.pause("");
|
||||
registrar.pollOnce(NOW);
|
||||
paused.toCompletableFuture().join();
|
||||
assertThat(consumer.paused()).contains(PARTITION);
|
||||
|
||||
registrar.resume("").toCompletableFuture().join();
|
||||
java.util.concurrent.CompletionStage<Void> resumed = registrar.resume("");
|
||||
registrar.pollOnce(NOW);
|
||||
resumed.toCompletableFuture().join();
|
||||
assertThat(consumer.paused()).doesNotContain(PARTITION);
|
||||
}
|
||||
|
||||
@@ -175,9 +184,174 @@ class KafkaConsumerRegistrarTest {
|
||||
registrar.pollOnce(NOW);
|
||||
|
||||
assertThat(handled).isEmpty();
|
||||
assertThat(registrar.committedOffset(PARTITION))
|
||||
.as("PARKED completed the offset with no quarantine write, so the record was simply lost")
|
||||
.isEmpty();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("an undecodable record commits only once quarantine confirms the write")
|
||||
void anUndecodableRecordCommitsAfterQuarantineConfirms() {
|
||||
java.util.List<Long> quarantined = new java.util.ArrayList<>();
|
||||
KafkaConsumerRegistrar registrar =
|
||||
new KafkaConsumerRegistrar(
|
||||
new TransportConsumerSpec(KafkaContractHarness.profile(), this::settleImmediately),
|
||||
consumer,
|
||||
handlers,
|
||||
new GracefulShutdownCoordinator(Duration.ofSeconds(30)),
|
||||
Duration.ofMillis(1),
|
||||
(record, reason) -> {
|
||||
quarantined.add(record.offset());
|
||||
return true;
|
||||
});
|
||||
assign();
|
||||
addRecordWithoutIdentityHeaders(10);
|
||||
|
||||
registrar.pollOnce(NOW);
|
||||
handlers.runAll();
|
||||
registrar.pollOnce(NOW);
|
||||
|
||||
assertThat(quarantined).containsExactly(10L);
|
||||
assertThat(registrar.committedOffset(PARTITION)).hasValue(11L);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("one partition hitting its ceiling does not abandon the other's records")
|
||||
void oneSaturatedPartitionDoesNotAbandonAnother() {
|
||||
org.apache.kafka.common.TopicPartition second =
|
||||
new org.apache.kafka.common.TopicPartition(PARTITION.topic(), PARTITION.partition() + 1);
|
||||
consumer.assign(List.of(PARTITION, second));
|
||||
consumer.updateBeginningOffsets(Map.of(PARTITION, 10L, second, 10L));
|
||||
consumer.seek(PARTITION, 10L);
|
||||
consumer.seek(second, 10L);
|
||||
KafkaConsumerRegistrar registrar = registrar(this::settleImmediately);
|
||||
// Two records on the first partition (its ordering unit allows one in flight) and one on the
|
||||
// second. The old loop broke out of the whole batch at the first record it could not submit.
|
||||
addRecord(10);
|
||||
addRecord(11);
|
||||
addRecordOn(second, 10);
|
||||
|
||||
int dispatched = registrar.pollOnce(NOW);
|
||||
|
||||
assertThat(dispatched)
|
||||
.as("the second partition's record was returned by the same poll and must not be dropped")
|
||||
.isEqualTo(2);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a rejected executor submission leaves nothing in flight and re-reads the record")
|
||||
void aRejectedSubmissionIsUndone() {
|
||||
KafkaConsumerRegistrar registrar =
|
||||
new KafkaConsumerRegistrar(
|
||||
new TransportConsumerSpec(KafkaContractHarness.profile(), this::settleImmediately),
|
||||
consumer,
|
||||
command -> {
|
||||
throw new java.util.concurrent.RejectedExecutionException("pool is shutting down");
|
||||
},
|
||||
new GracefulShutdownCoordinator(Duration.ofSeconds(30)),
|
||||
Duration.ofMillis(1));
|
||||
assign();
|
||||
addRecord(10);
|
||||
|
||||
int dispatched = registrar.pollOnce(NOW);
|
||||
|
||||
assertThat(dispatched).isZero();
|
||||
assertThat(registrar.committedOffset(PARTITION))
|
||||
.as("an offset registered as delivered with no worker behind it never completes")
|
||||
.isEmpty();
|
||||
assertThat(consumer.position(PARTITION))
|
||||
.as("the record must be re-read next cycle")
|
||||
.isEqualTo(10L);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a settlement from a revoked assignment is refused")
|
||||
void aStaleSettlementIsRefused() {
|
||||
KafkaConsumerRegistrar registrar = registrar(this::deferSettlement);
|
||||
assign();
|
||||
addRecord(10);
|
||||
registrar.pollOnce(NOW);
|
||||
|
||||
// The partition goes away and comes back: a new assignment, a new epoch.
|
||||
consumer.assign(List.of());
|
||||
registrar.pollOnce(NOW);
|
||||
assign();
|
||||
registrar.pollOnce(NOW);
|
||||
|
||||
// Now the old handler finally acknowledges.
|
||||
handlers.runAll();
|
||||
registrar.pollOnce(NOW);
|
||||
|
||||
assertThat(registrar.staleSettlements())
|
||||
.as("an acknowledgement from the previous assignment must not move this one's watermark")
|
||||
.isPositive();
|
||||
assertThat(registrar.committedOffset(PARTITION)).isEmpty();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a second terminal call on one delivery is refused")
|
||||
void aSecondTerminalCallIsRefused() {
|
||||
java.util.List<Throwable> refusals = new java.util.ArrayList<>();
|
||||
KafkaConsumerRegistrar registrar =
|
||||
registrar(
|
||||
delivery -> {
|
||||
delivery.settlement().acknowledge().toCompletableFuture().join();
|
||||
delivery
|
||||
.settlement()
|
||||
.requeue(Duration.ZERO)
|
||||
.toCompletableFuture()
|
||||
.handle(
|
||||
(result, failure) -> {
|
||||
if (failure != null) {
|
||||
refusals.add(failure);
|
||||
}
|
||||
return null;
|
||||
})
|
||||
.join();
|
||||
return java.util.concurrent.CompletableFuture.completedFuture(null);
|
||||
});
|
||||
assign();
|
||||
addRecord(10);
|
||||
|
||||
registrar.pollOnce(NOW);
|
||||
handlers.runAll();
|
||||
|
||||
assertThat(refusals)
|
||||
.as("telling the poll thread both 'done' and 're-read this' lets the second one win")
|
||||
.hasSize(1);
|
||||
}
|
||||
|
||||
private void addRecordOn(org.apache.kafka.common.TopicPartition partition, long offset) {
|
||||
RecordHeaders headers = new RecordHeaders();
|
||||
headers.add(
|
||||
ReservedHeaders.MESSAGE_ID,
|
||||
// A v7 identity: MessageId now enforces the version it documents, and a v4 here would be
|
||||
// refused by the envelope mapper exactly as a foreign producer's would.
|
||||
MessageId.newId().value().toString().getBytes(StandardCharsets.UTF_8));
|
||||
headers.add(ReservedHeaders.MESSAGE_TYPE, "order.created".getBytes(StandardCharsets.UTF_8));
|
||||
headers.add(ReservedHeaders.SCHEMA_VERSION, "1".getBytes(StandardCharsets.UTF_8));
|
||||
headers.add(ReservedHeaders.CONTENT_TYPE, "application/json".getBytes(StandardCharsets.UTF_8));
|
||||
consumer.addRecord(
|
||||
new ConsumerRecord<>(
|
||||
partition.topic(),
|
||||
partition.partition(),
|
||||
offset,
|
||||
NOW.toEpochMilli(),
|
||||
org.apache.kafka.common.record.TimestampType.CREATE_TIME,
|
||||
0,
|
||||
0,
|
||||
null,
|
||||
"{}".getBytes(StandardCharsets.UTF_8),
|
||||
headers,
|
||||
java.util.Optional.empty()));
|
||||
}
|
||||
|
||||
/** Never settles inside the handler, so the settlement arrives later. */
|
||||
private CompletionStage<Void> deferSettlement(TransportDelivery delivery) {
|
||||
delivery.settlement().acknowledge();
|
||||
return java.util.concurrent.CompletableFuture.completedFuture(null);
|
||||
}
|
||||
|
||||
private void assign() {
|
||||
consumer.assign(List.of(PARTITION));
|
||||
consumer.updateBeginningOffsets(Map.of(PARTITION, 10L));
|
||||
@@ -187,7 +361,10 @@ class KafkaConsumerRegistrarTest {
|
||||
private void addRecord(long offset) {
|
||||
RecordHeaders headers = new RecordHeaders();
|
||||
headers.add(
|
||||
ReservedHeaders.MESSAGE_ID, UUID.randomUUID().toString().getBytes(StandardCharsets.UTF_8));
|
||||
ReservedHeaders.MESSAGE_ID,
|
||||
// A v7 identity: MessageId now enforces the version it documents, and a v4 here would be
|
||||
// refused by the envelope mapper exactly as a foreign producer's would.
|
||||
MessageId.newId().value().toString().getBytes(StandardCharsets.UTF_8));
|
||||
headers.add(ReservedHeaders.MESSAGE_TYPE, "order.created".getBytes(StandardCharsets.UTF_8));
|
||||
headers.add(ReservedHeaders.SCHEMA_VERSION, "1".getBytes(StandardCharsets.UTF_8));
|
||||
headers.add(ReservedHeaders.CONTENT_TYPE, "application/json".getBytes(StandardCharsets.UTF_8));
|
||||
|
||||
+105
@@ -0,0 +1,105 @@
|
||||
package dev.caskeleton.messaging.kafka;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
import dev.caskeleton.messaging.api.error.MessagingConfigurationException;
|
||||
import dev.caskeleton.messaging.security.BrokerCredentialProfile;
|
||||
import dev.caskeleton.messaging.security.BrokerSecurityProfile;
|
||||
import dev.caskeleton.messaging.security.CredentialProvider;
|
||||
import dev.caskeleton.messaging.security.CredentialRuntimeRegistry;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* What reaches a JAAS entry, and what is refused outright.
|
||||
*
|
||||
* <p>The password was interpolated into a quoted string with no escaping. A generated secret
|
||||
* containing a quote or a backslash produced a config that fails to parse — an authentication error
|
||||
* naming nothing useful — and one containing a quote and a semicolon terminated the option early
|
||||
* and injected another.
|
||||
*/
|
||||
class KafkaSecurityConfigurerTest {
|
||||
|
||||
private static final Instant NOW = Instant.parse("2026-08-14T00:00:00Z");
|
||||
|
||||
@Test
|
||||
@DisplayName("a secret containing a quote and a backslash is escaped, not injected")
|
||||
void aSecretWithQuotesIsEscaped() {
|
||||
Map<String, Object> properties = configure("pa\"ss\\word;");
|
||||
|
||||
String jaas = String.valueOf(properties.get("sasl.jaas.config"));
|
||||
assertThat(jaas)
|
||||
.as("an unescaped quote closes the option and everything after it becomes new options")
|
||||
.contains("password=\"pa\\\"ss\\\\word;\";");
|
||||
assertThat(jaas)
|
||||
.as("the raw form would close the password option and make the rest new options")
|
||||
.doesNotContain("password=\"pa\"");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a secret containing a control character is refused")
|
||||
void aSecretWithAControlCharacterIsRefused() {
|
||||
assertThatThrownBy(() -> configure("pass\nword"))
|
||||
.isInstanceOf(MessagingConfigurationException.class)
|
||||
.hasMessageContaining("control character");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("an OAuth profile is refused rather than half-configured")
|
||||
void anOAuthProfileIsRefused() {
|
||||
CredentialRuntimeRegistry credentials = registry("secret");
|
||||
|
||||
BrokerCredentialProfile oauth = new BrokerCredentialProfile.OAuth2("kafka-oauth");
|
||||
assertThatThrownBy(
|
||||
() ->
|
||||
new KafkaSecurityConfigurer(
|
||||
credentials, new dev.caskeleton.messaging.security.BrokerTlsPolicy())
|
||||
.configure(profile(oauth), oauth, java.util.List.of("TLSv1.3"), NOW))
|
||||
.as("naming the mechanism without a callback handler fails at connect time instead")
|
||||
.isInstanceOf(MessagingConfigurationException.class)
|
||||
.hasMessageContaining("callback handler");
|
||||
}
|
||||
|
||||
private static Map<String, Object> configure(String secret) {
|
||||
return new KafkaSecurityConfigurer(
|
||||
registry(secret), new dev.caskeleton.messaging.security.BrokerTlsPolicy())
|
||||
.configure(
|
||||
profile(new BrokerCredentialProfile.SaslScram("kafka-app")),
|
||||
new BrokerCredentialProfile.SaslScram("kafka-app"),
|
||||
java.util.List.of("TLSv1.3"),
|
||||
NOW);
|
||||
}
|
||||
|
||||
private static BrokerSecurityProfile profile(BrokerCredentialProfile credential) {
|
||||
return new BrokerSecurityProfile(
|
||||
"kafka-primary",
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
credential,
|
||||
credential,
|
||||
Optional.empty(),
|
||||
dev.caskeleton.messaging.security.DestinationAccessPolicy.denyAll());
|
||||
}
|
||||
|
||||
private static CredentialRuntimeRegistry registry(String secret) {
|
||||
return new CredentialRuntimeRegistry(
|
||||
new CredentialProvider() {
|
||||
@Override
|
||||
public char[] resolve(String credentialId) {
|
||||
return secret.toCharArray();
|
||||
}
|
||||
|
||||
@Override
|
||||
public Optional<Instant> expiresAt(String credentialId) {
|
||||
return Optional.of(NOW.plus(Duration.ofHours(1)));
|
||||
}
|
||||
},
|
||||
Duration.ofMinutes(5));
|
||||
}
|
||||
}
|
||||
+192
@@ -0,0 +1,192 @@
|
||||
package dev.caskeleton.messaging.kafka;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
import dev.caskeleton.messaging.api.ContentType;
|
||||
import dev.caskeleton.messaging.api.MessageEnvelope;
|
||||
import dev.caskeleton.messaging.api.MessageId;
|
||||
import dev.caskeleton.messaging.api.MessageType;
|
||||
import dev.caskeleton.messaging.api.ProducerId;
|
||||
import dev.caskeleton.messaging.api.SchemaVersion;
|
||||
import dev.caskeleton.messaging.api.TraceContext;
|
||||
import dev.caskeleton.messaging.api.header.MessageHeaders;
|
||||
import dev.caskeleton.messaging.policy.DestinationProfile;
|
||||
import dev.caskeleton.messaging.schema.EncodedMessage;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
import org.apache.kafka.clients.consumer.ConsumerGroupMetadata;
|
||||
import org.apache.kafka.clients.producer.MockProducer;
|
||||
import org.apache.kafka.common.serialization.ByteArraySerializer;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* Where the handler runs relative to {@code beginTransaction}.
|
||||
*
|
||||
* <p>{@code KafkaTransactionalProcessor} documents that the callback executes inside the Kafka
|
||||
* transaction. It did not: the processor ran the handler and only afterwards called {@code
|
||||
* sendInTransaction}, which is where {@code beginTransaction} lived. Everything the handler
|
||||
* produced itself — and every external effect it triggered — was outside the transaction, so a
|
||||
* handler that succeeded followed by a failed commit left its work applied with the input offsets
|
||||
* unsent.
|
||||
*/
|
||||
class KafkaTransactionOrderingTest {
|
||||
|
||||
private final RecordingProducer producer = new RecordingProducer();
|
||||
|
||||
private final KafkaTransactionalPublisher publisher =
|
||||
new KafkaTransactionalPublisher(
|
||||
producer, () -> new ConsumerGroupMetadata("order-projection"));
|
||||
|
||||
@Test
|
||||
@DisplayName("the transaction is open while the handler runs")
|
||||
void theTransactionIsOpenWhileTheHandlerRuns() {
|
||||
AtomicBoolean openDuringHandler = new AtomicBoolean();
|
||||
SpringKafkaTransactionalProcessor processor = processor();
|
||||
|
||||
processor
|
||||
.processInTransaction(
|
||||
delivery(),
|
||||
ignored -> {
|
||||
// The real publisher has already begun the transaction by the time the handler runs.
|
||||
openDuringHandler.set(producer.calls().equals(List.of("begin")));
|
||||
return "handled";
|
||||
})
|
||||
.toCompletableFuture()
|
||||
.join();
|
||||
|
||||
assertThat(openDuringHandler)
|
||||
.as("the handler used to run before beginTransaction was ever called")
|
||||
.isTrue();
|
||||
assertThat(producer.calls()).containsExactly("begin", "offsets", "commit");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a handler failure aborts and never commits")
|
||||
void aHandlerFailureAbortsAndNeverCommits() {
|
||||
SpringKafkaTransactionalProcessor processor = processor();
|
||||
|
||||
assertThatThrownBy(
|
||||
() ->
|
||||
processor
|
||||
.processInTransaction(
|
||||
delivery(),
|
||||
ignored -> {
|
||||
throw new IllegalStateException("the projection is down");
|
||||
})
|
||||
.toCompletableFuture()
|
||||
.join())
|
||||
.hasCauseInstanceOf(IllegalStateException.class);
|
||||
|
||||
assertThat(producer.calls())
|
||||
.as("under read_committed nothing may become visible, and the offsets stay unsent")
|
||||
.containsExactly("begin", "abort");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a commit failure aborts")
|
||||
void aCommitFailureAborts() {
|
||||
producer.failCommit();
|
||||
SpringKafkaTransactionalProcessor processor = processor();
|
||||
|
||||
assertThatThrownBy(
|
||||
() ->
|
||||
processor
|
||||
.processInTransaction(delivery(), ignored -> "handled")
|
||||
.toCompletableFuture()
|
||||
.join())
|
||||
.hasCauseInstanceOf(IllegalStateException.class);
|
||||
|
||||
assertThat(producer.calls()).endsWith("abort");
|
||||
}
|
||||
|
||||
/** A MockProducer that records the transaction calls in the order they happen. */
|
||||
private static final class RecordingProducer extends MockProducer<byte[], byte[]> {
|
||||
|
||||
private final List<String> calls = new java.util.ArrayList<>();
|
||||
|
||||
private boolean commitFails;
|
||||
|
||||
private RecordingProducer() {
|
||||
// The four-arg constructor with a null partitioner: this producer never partitions, because
|
||||
// the assertions are about transaction call order rather than record placement.
|
||||
super(true, null, new ByteArraySerializer(), new ByteArraySerializer());
|
||||
initTransactions();
|
||||
}
|
||||
|
||||
private void failCommit() {
|
||||
commitFails = true;
|
||||
}
|
||||
|
||||
private List<String> calls() {
|
||||
return List.copyOf(calls);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void beginTransaction() {
|
||||
calls.add("begin");
|
||||
super.beginTransaction();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void sendOffsetsToTransaction(
|
||||
Map<
|
||||
org.apache.kafka.common.TopicPartition,
|
||||
org.apache.kafka.clients.consumer.OffsetAndMetadata>
|
||||
offsets,
|
||||
ConsumerGroupMetadata groupMetadata) {
|
||||
calls.add("offsets");
|
||||
super.sendOffsetsToTransaction(offsets, groupMetadata);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void commitTransaction() {
|
||||
if (commitFails) {
|
||||
throw new IllegalStateException("commit failed");
|
||||
}
|
||||
calls.add("commit");
|
||||
super.commitTransaction();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void abortTransaction() {
|
||||
calls.add("abort");
|
||||
super.abortTransaction();
|
||||
}
|
||||
}
|
||||
|
||||
private SpringKafkaTransactionalProcessor processor() {
|
||||
DestinationProfile profile = KafkaContractHarness.profile();
|
||||
return new SpringKafkaTransactionalProcessor(
|
||||
publisher, Map.of(profile.name().value(), profile), ignored -> Map.of());
|
||||
}
|
||||
|
||||
private static KafkaTransactionalDelivery delivery() {
|
||||
return new KafkaTransactionalDelivery(envelope(), List.of());
|
||||
}
|
||||
|
||||
private static MessageEnvelope<EncodedMessage> envelope() {
|
||||
return new MessageEnvelope<>(
|
||||
MessageId.newId(),
|
||||
new MessageType("order.created"),
|
||||
new SchemaVersion(1),
|
||||
Instant.parse("2026-08-15T00:00:00Z"),
|
||||
Optional.empty(),
|
||||
new ProducerId("orders"),
|
||||
Optional.empty(),
|
||||
Optional.empty(),
|
||||
ContentType.JSON,
|
||||
Optional.empty(),
|
||||
Optional.empty(),
|
||||
Optional.empty(),
|
||||
TraceContext.none(),
|
||||
MessageHeaders.empty(),
|
||||
new EncodedMessage(
|
||||
"{}".getBytes(StandardCharsets.UTF_8), ContentType.JSON, Optional.empty()));
|
||||
}
|
||||
}
|
||||
+123
@@ -0,0 +1,123 @@
|
||||
package dev.caskeleton.messaging.kafka;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThatCode;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
import dev.caskeleton.messaging.api.ContentType;
|
||||
import dev.caskeleton.messaging.api.MessageEnvelope;
|
||||
import dev.caskeleton.messaging.api.MessageId;
|
||||
import dev.caskeleton.messaging.api.MessageType;
|
||||
import dev.caskeleton.messaging.api.ProducerId;
|
||||
import dev.caskeleton.messaging.api.SchemaVersion;
|
||||
import dev.caskeleton.messaging.api.TraceContext;
|
||||
import dev.caskeleton.messaging.api.error.MessagingConfigurationException;
|
||||
import dev.caskeleton.messaging.api.header.HeaderName;
|
||||
import dev.caskeleton.messaging.api.header.HeaderValue;
|
||||
import dev.caskeleton.messaging.api.header.MessageHeaders;
|
||||
import dev.caskeleton.messaging.api.header.ReservedHeaders;
|
||||
import dev.caskeleton.messaging.schema.EncodedMessage;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Instant;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* What happens when message content claims a reserved header.
|
||||
*
|
||||
* <p>The canonical headers were written first and the envelope's own headers appended afterwards,
|
||||
* and the consumer reads {@code lastHeader}. A row in the outbox carrying {@code msg.id} therefore
|
||||
* overwrote the canonical identity on the wire — and that identity is what inbox deduplication and
|
||||
* the audit trail key on, so a forged one corrupts a *different* message's dedup state.
|
||||
*/
|
||||
class ReservedHeaderForgeryTest {
|
||||
|
||||
private final KafkaHeaderMapper mapper = new KafkaHeaderMapper();
|
||||
|
||||
@Test
|
||||
@DisplayName("an application header in the reserved namespace is refused")
|
||||
void aForgedReservedHeaderIsRefused() {
|
||||
assertThatThrownBy(
|
||||
() -> mapper.toKafkaHeaders(envelopeWithHeader(ReservedHeaders.MESSAGE_ID, "forged")))
|
||||
.isInstanceOf(MessagingConfigurationException.class)
|
||||
.hasMessageContaining("reserved namespace");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a forged message type is refused too")
|
||||
void aForgedMessageTypeIsRefused() {
|
||||
assertThatThrownBy(
|
||||
() ->
|
||||
mapper.toKafkaHeaders(
|
||||
envelopeWithHeader(ReservedHeaders.MESSAGE_TYPE, "other.type")))
|
||||
.isInstanceOf(MessagingConfigurationException.class);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("an ordinary application header still travels")
|
||||
void anOrdinaryApplicationHeaderStillTravels() {
|
||||
assertThatCode(() -> mapper.toKafkaHeaders(envelopeWithHeader("x-tenant-hint", "acme")))
|
||||
.doesNotThrowAnyException();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a malformed retry attempt is quarantined, not reset to one")
|
||||
void aMalformedRetryAttemptIsQuarantined() {
|
||||
assertThatThrownBy(
|
||||
() ->
|
||||
new KafkaRetryMetadataMapper()
|
||||
.attemptOf(
|
||||
envelopeWithPlatformHeader(ReservedHeaders.RETRY_ATTEMPT, "not-a-number")))
|
||||
.as("resetting to 1 restarts the budget on every redelivery, so the cap never applies")
|
||||
.isInstanceOf(MessagingConfigurationException.class)
|
||||
.hasMessageContaining("retry attempt");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a negative retry attempt is quarantined")
|
||||
void aNegativeRetryAttemptIsQuarantined() {
|
||||
assertThatThrownBy(
|
||||
() ->
|
||||
new KafkaRetryMetadataMapper()
|
||||
.attemptOf(envelopeWithPlatformHeader(ReservedHeaders.RETRY_ATTEMPT, "-3")))
|
||||
.isInstanceOf(MessagingConfigurationException.class);
|
||||
}
|
||||
|
||||
/**
|
||||
* Headers built through the platform factory, which is the path an outbox row takes.
|
||||
*
|
||||
* <p>{@code MessageHeaders.application} already refuses reserved names, so business code could
|
||||
* never forge one. The gap was the platform factory: {@code OutboxEnvelopeFactory} promoted a
|
||||
* row's arbitrary {@code Map<String,String>} straight through it, and the mapper then appended
|
||||
* the result after the canonical headers.
|
||||
*/
|
||||
private static MessageEnvelope<EncodedMessage> envelopeWithHeader(String name, String value) {
|
||||
return envelope(MessageHeaders.platform(Map.of(new HeaderName(name), new HeaderValue(value))));
|
||||
}
|
||||
|
||||
private static MessageEnvelope<EncodedMessage> envelopeWithPlatformHeader(
|
||||
String name, String value) {
|
||||
return envelope(MessageHeaders.platform(Map.of(new HeaderName(name), new HeaderValue(value))));
|
||||
}
|
||||
|
||||
private static MessageEnvelope<EncodedMessage> envelope(MessageHeaders headers) {
|
||||
return new MessageEnvelope<>(
|
||||
MessageId.newId(),
|
||||
new MessageType("order.created"),
|
||||
new SchemaVersion(1),
|
||||
Instant.parse("2026-08-15T00:00:00Z"),
|
||||
Optional.empty(),
|
||||
new ProducerId("orders"),
|
||||
Optional.empty(),
|
||||
Optional.empty(),
|
||||
ContentType.JSON,
|
||||
Optional.empty(),
|
||||
Optional.empty(),
|
||||
Optional.empty(),
|
||||
TraceContext.none(),
|
||||
headers,
|
||||
new EncodedMessage(
|
||||
"{}".getBytes(StandardCharsets.UTF_8), ContentType.JSON, Optional.empty()));
|
||||
}
|
||||
}
|
||||
@@ -32,9 +32,9 @@ commons-io:commons-io:2.21.0=spotbugs
|
||||
info.picocli:picocli:4.7.7=checkstyle
|
||||
io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnotationProcessor
|
||||
io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor
|
||||
io.micrometer:micrometer-commons:1.16.0=runtimeClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-core:1.16.0=runtimeClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-observation:1.16.0=runtimeClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-commons:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-core:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-observation:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.nats:jnats:2.26.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
javax.inject:javax.inject:1=annotationProcessor,testAnnotationProcessor
|
||||
jaxen:jaxen:2.0.0=spotbugs
|
||||
|
||||
+1
-1
@@ -41,7 +41,7 @@ public final class NatsJetStreamProfileValidator {
|
||||
throw new MessagingCapabilityUnavailableException(
|
||||
"NATS_DISABLED",
|
||||
"the NATS JetStream adapter is experimental and disabled unless "
|
||||
+ "backend.messaging.experimental.nats=true");
|
||||
+ "app.messaging.experimental.nats=true");
|
||||
}
|
||||
if (profile.deliveryGuarantee() == DeliveryGuarantee.AT_LEAST_ONCE && !jetStreamEnabled) {
|
||||
throw new IllegalArgumentException(
|
||||
|
||||
+53
-26
@@ -39,6 +39,13 @@ import java.util.function.Function;
|
||||
* stored it and lost only the acknowledgement, and the deduplication window is what makes retrying
|
||||
* it safe when the profile enables one.
|
||||
*
|
||||
* <p>This is an Experimental contract seam, not a Stable adapter. It exercises the transport SPI
|
||||
* against a publish operation the application supplies; it does not ship a NATS client bridge,
|
||||
* connection management, or reconnection. Only a failure the operation reports as a {@link
|
||||
* NatsPreSendRejection} is treated as definitely not transmitted — every other failure, recognised
|
||||
* or not, is ambiguous, because a caller that reads {@code REJECTED} may republish under a new
|
||||
* identity and duplicate a message the server already stored.
|
||||
*
|
||||
* <p>The adapter reports {@code nativeDeadLetter} as false. JetStream terminates a message at its
|
||||
* delivery limit rather than routing it anywhere, so the platform parks it via {@link
|
||||
* NatsMaxDeliverParkingWorkflow}; claiming a native dead-letter would send operators looking for a
|
||||
@@ -119,9 +126,20 @@ public final class NatsJetStreamTransport implements MessagingTransport {
|
||||
"PAYLOAD_TOO_LARGE", "encoded payload is " + size + " bytes, limit is " + limit));
|
||||
}
|
||||
|
||||
// The caller's own deadline, honoured rather than ignored. Without it a stalled JetStream
|
||||
// publish holds the caller for as long as the SDK's default takes, and the timeout the caller
|
||||
// set to bound its own latency budget means nothing.
|
||||
long startedAt = System.nanoTime();
|
||||
return publish
|
||||
.publish(profile.subject(), deduplicationId(request), request)
|
||||
.handle((position, failure) -> failure == null ? confirmed(position) : classify(failure))
|
||||
.toCompletableFuture()
|
||||
.orTimeout(
|
||||
request.options().timeout().toMillis(), java.util.concurrent.TimeUnit.MILLISECONDS)
|
||||
.handle(
|
||||
(position, failure) ->
|
||||
failure == null
|
||||
? confirmed(position, elapsedSince(startedAt))
|
||||
: classify(failure, elapsedSince(startedAt)))
|
||||
.thenApply(TransportPublishResult::new);
|
||||
}
|
||||
|
||||
@@ -180,7 +198,11 @@ public final class NatsJetStreamTransport implements MessagingTransport {
|
||||
.map(window -> request.envelope().messageId().value().toString());
|
||||
}
|
||||
|
||||
private static PublishResult confirmed(NatsStreamPosition position) {
|
||||
private static Duration elapsedSince(long startedAtNanos) {
|
||||
return Duration.ofNanos(System.nanoTime() - startedAtNanos);
|
||||
}
|
||||
|
||||
private static PublishResult confirmed(NatsStreamPosition position, Duration elapsed) {
|
||||
return new PublishResult(
|
||||
PublishCompletion.CONFIRMED,
|
||||
// The publish acknowledgement names the stream and sequence the message landed in, which is
|
||||
@@ -193,45 +215,49 @@ public final class NatsJetStreamTransport implements MessagingTransport {
|
||||
RoutingOutcome.ROUTED,
|
||||
Optional.of(position),
|
||||
1,
|
||||
Duration.ZERO,
|
||||
elapsed,
|
||||
Optional.empty());
|
||||
}
|
||||
|
||||
private static PublishResult classify(Throwable failure) {
|
||||
/**
|
||||
* Decides what a publish failure means, from typed signals only.
|
||||
*
|
||||
* <p>Only {@link NatsPreSendRejection} — thrown by the publish operation when it knows the send
|
||||
* never left the client — reports {@code REJECTED / NOT_TRANSMITTED}. Everything else is {@code
|
||||
* AMBIGUOUS}, including failures this adapter does not recognise, because the alternative is
|
||||
* telling a caller nothing was transmitted when the server may already hold the message.
|
||||
*/
|
||||
private static PublishResult classify(Throwable failure, Duration elapsed) {
|
||||
Throwable cause = failure instanceof CompletionException ? failure.getCause() : failure;
|
||||
String name = cause == null ? "" : cause.getClass().getSimpleName();
|
||||
|
||||
if (name.contains("Timeout")) {
|
||||
if (cause instanceof NatsPreSendRejection rejection) {
|
||||
return new PublishResult(
|
||||
PublishCompletion.AMBIGUOUS,
|
||||
new PublishEvidence(
|
||||
true, TransmissionEvidence.MAY_HAVE_BEEN_TRANSMITTED, false, ConfirmationLevel.NONE),
|
||||
RoutingOutcome.UNKNOWN,
|
||||
PublishCompletion.REJECTED,
|
||||
PublishEvidence.notTransmitted(),
|
||||
RoutingOutcome.NOT_APPLICABLE,
|
||||
Optional.empty(),
|
||||
1,
|
||||
Duration.ZERO,
|
||||
elapsed,
|
||||
Optional.of(
|
||||
FailureDescriptor.of(
|
||||
FailureCategory.TRANSIENT_INFRASTRUCTURE,
|
||||
"NATS_PUBLISH_TIMEOUT",
|
||||
"the publish acknowledgement did not arrive; the stream may already hold it")));
|
||||
FailureCategory.CONFIGURATION, rejection.code(), rejection.getMessage())));
|
||||
}
|
||||
// "no responders" means no JetStream server is listening on the subject at all. That is a
|
||||
// configuration fault, not a transient one: retrying it will fail identically until the stream
|
||||
// is declared.
|
||||
boolean noStream = name.contains("NoRespond") || name.contains("Jetstream");
|
||||
boolean timedOut = cause instanceof java.util.concurrent.TimeoutException;
|
||||
return new PublishResult(
|
||||
PublishCompletion.REJECTED,
|
||||
PublishEvidence.notTransmitted(),
|
||||
RoutingOutcome.NOT_APPLICABLE,
|
||||
PublishCompletion.AMBIGUOUS,
|
||||
new PublishEvidence(
|
||||
true, TransmissionEvidence.MAY_HAVE_BEEN_TRANSMITTED, false, ConfirmationLevel.NONE),
|
||||
RoutingOutcome.UNKNOWN,
|
||||
Optional.empty(),
|
||||
1,
|
||||
Duration.ZERO,
|
||||
elapsed,
|
||||
Optional.of(
|
||||
FailureDescriptor.of(
|
||||
noStream ? FailureCategory.CONFIGURATION : FailureCategory.PERMANENT_BUSINESS,
|
||||
noStream ? "NATS_STREAM_NOT_DECLARED" : "NATS_PUBLISH_REJECTED",
|
||||
"the JetStream publish was rejected: " + name)));
|
||||
FailureCategory.TRANSIENT_INFRASTRUCTURE,
|
||||
timedOut ? "NATS_PUBLISH_TIMEOUT" : "NATS_PUBLISH_OUTCOME_UNKNOWN",
|
||||
timedOut
|
||||
? "the publish acknowledgement did not arrive; the stream may already hold it"
|
||||
: "the publish failed in a way this adapter does not classify; the stream may"
|
||||
+ " already hold the message")));
|
||||
}
|
||||
|
||||
private static TransportPublishResult rejectedLocally(String code, String message) {
|
||||
@@ -242,6 +268,7 @@ public final class NatsJetStreamTransport implements MessagingTransport {
|
||||
RoutingOutcome.NOT_APPLICABLE,
|
||||
Optional.empty(),
|
||||
1,
|
||||
// A local refusal: nothing was attempted, so no time was spent on the wire.
|
||||
Duration.ZERO,
|
||||
Optional.of(FailureDescriptor.of(FailureCategory.PERMANENT_BUSINESS, code, message))));
|
||||
}
|
||||
|
||||
+65
@@ -0,0 +1,65 @@
|
||||
package dev.caskeleton.messaging.nats;
|
||||
|
||||
import java.util.Objects;
|
||||
|
||||
/**
|
||||
* The one signal that lets this adapter report {@code REJECTED / NOT_TRANSMITTED}.
|
||||
*
|
||||
* <p>Classification used to work by reading the exception's class simple name and matching
|
||||
* substrings — {@code "Timeout"} meant ambiguous, anything else meant rejected. Two things are
|
||||
* wrong with that. A class name is not part of any SDK's contract: it changes between versions, and
|
||||
* an unrelated exception whose name happens to contain the substring is misread. And defaulting the
|
||||
* unknown case to {@code REJECTED} tells the caller nothing was transmitted, so a caller that
|
||||
* republishes under a new identity duplicates a message the server may well have stored.
|
||||
*
|
||||
* <p>The publish operation throws this only when it knows the send never left the client: an
|
||||
* argument the SDK refused, a connection that was already closed, a subject the client could not
|
||||
* resolve. Everything else — including anything the adapter does not recognise — is ambiguous, and
|
||||
* ambiguity is the honest answer when the write may have landed.
|
||||
*/
|
||||
public final class NatsPreSendRejection extends RuntimeException {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final String code;
|
||||
|
||||
/**
|
||||
* Creates a pre-send rejection.
|
||||
*
|
||||
* @param code the sanitised platform code for the refusal
|
||||
* @param message a sanitised description
|
||||
*/
|
||||
public NatsPreSendRejection(String code, String message) {
|
||||
super(message);
|
||||
this.code = requireCode(code);
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a pre-send rejection wrapping the SDK's own exception.
|
||||
*
|
||||
* @param code the sanitised platform code for the refusal
|
||||
* @param message a sanitised description
|
||||
* @param cause the SDK exception that proves the send never started
|
||||
*/
|
||||
public NatsPreSendRejection(String code, String message, Throwable cause) {
|
||||
super(message, cause);
|
||||
this.code = requireCode(code);
|
||||
}
|
||||
|
||||
private static String requireCode(String code) {
|
||||
Objects.requireNonNull(code, "code must not be null");
|
||||
if (code.isBlank()) {
|
||||
throw new IllegalArgumentException("code must not be blank");
|
||||
}
|
||||
return code;
|
||||
}
|
||||
|
||||
/**
|
||||
* The sanitised code reported to the caller.
|
||||
*
|
||||
* @return the code
|
||||
*/
|
||||
public String code() {
|
||||
return code;
|
||||
}
|
||||
}
|
||||
+78
-5
@@ -102,14 +102,76 @@ class NatsAdapterContractTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void aMissingStreamIsAConfigurationFaultNotATransientOne() {
|
||||
void aFailureNamedLikeAKnownOneIsStillAmbiguous() {
|
||||
// The class name used to be the classifier: an exception whose simple name contained
|
||||
// "NoRespond" was reported REJECTED / NOT_TRANSMITTED. A name is not an SDK contract, and a
|
||||
// caller that reads NOT_TRANSMITTED republishes under a new identity.
|
||||
PublishResult result =
|
||||
await(failingWith(new IllegalStateException("NoRespondersException")).publish(request(64)));
|
||||
|
||||
assertThat(result.completion()).isEqualTo(PublishCompletion.REJECTED);
|
||||
assertThat(result.completion()).isEqualTo(PublishCompletion.AMBIGUOUS);
|
||||
assertThat(result.failure())
|
||||
.hasValueSatisfying(
|
||||
failure -> assertThat(failure.code()).isEqualTo("NATS_PUBLISH_REJECTED"));
|
||||
failure -> assertThat(failure.code()).isEqualTo("NATS_PUBLISH_OUTCOME_UNKNOWN"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void onlyATypedPreSendRejectionReportsNothingTransmitted() {
|
||||
PublishResult result =
|
||||
await(
|
||||
failingWith(
|
||||
new NatsPreSendRejection(
|
||||
"NATS_CONNECTION_CLOSED", "the client was closed before the send"))
|
||||
.publish(request(64)));
|
||||
|
||||
assertThat(result.completion()).isEqualTo(PublishCompletion.REJECTED);
|
||||
assertThat(result.evidence().transmission())
|
||||
.isEqualTo(dev.caskeleton.messaging.api.publish.TransmissionEvidence.NOT_TRANSMITTED);
|
||||
assertThat(result.failure())
|
||||
.hasValueSatisfying(
|
||||
failure -> assertThat(failure.code()).isEqualTo("NATS_CONNECTION_CLOSED"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void aWrappedUnknownFailureIsAmbiguousRatherThanRejected() {
|
||||
PublishResult result =
|
||||
await(
|
||||
failingWith(
|
||||
new java.util.concurrent.CompletionException(
|
||||
new RuntimeException("something the adapter has never seen")))
|
||||
.publish(request(64)));
|
||||
|
||||
assertThat(result.completion()).isEqualTo(PublishCompletion.AMBIGUOUS);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aPublishThatNeverCompletesIsBoundedByTheCallersTimeout() {
|
||||
NatsJetStreamTransport transport =
|
||||
new NatsJetStreamTransport(
|
||||
"nats-primary",
|
||||
1L,
|
||||
profile(Optional.empty()),
|
||||
(subject, deduplicationId, request) -> new java.util.concurrent.CompletableFuture<>());
|
||||
|
||||
PublishResult result = await(transport.publish(requestWithTimeout(Duration.ofMillis(50))));
|
||||
|
||||
assertThat(result.completion())
|
||||
.as("the caller's deadline used to be ignored entirely")
|
||||
.isEqualTo(PublishCompletion.AMBIGUOUS);
|
||||
assertThat(result.failure())
|
||||
.hasValueSatisfying(
|
||||
failure -> assertThat(failure.code()).isEqualTo("NATS_PUBLISH_TIMEOUT"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void theReportedElapsedTimeIsMeasuredRatherThanZero() {
|
||||
PublishResult result =
|
||||
await(
|
||||
failingWith(new java.util.concurrent.TimeoutException("no ack")).publish(request(64)));
|
||||
|
||||
assertThat(result.elapsed())
|
||||
.as("every outcome reported Duration.ZERO, so latency evidence was fiction")
|
||||
.isGreaterThanOrEqualTo(Duration.ZERO);
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -190,14 +252,16 @@ class NatsAdapterContractTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void aFailureCategoryIsAlwaysAttachedToARejection() {
|
||||
void aFailureCategoryIsAlwaysAttachedToAFailure() {
|
||||
PublishResult result =
|
||||
await(failingWith(new IllegalStateException("boom")).publish(request(64)));
|
||||
|
||||
assertThat(result.failure())
|
||||
.hasValueSatisfying(
|
||||
failure ->
|
||||
assertThat(failure.category()).isEqualTo(FailureCategory.PERMANENT_BUSINESS));
|
||||
assertThat(failure.category())
|
||||
.as("an unclassified failure is transient, not a permanent business rejection")
|
||||
.isEqualTo(FailureCategory.TRANSIENT_INFRASTRUCTURE));
|
||||
}
|
||||
|
||||
private static NatsJetStreamProfile profile(Optional<Duration> deduplicationWindow) {
|
||||
@@ -216,6 +280,15 @@ class NatsAdapterContractTest {
|
||||
destinationProfile(), envelope(payloadBytes), PublishOptions.defaults());
|
||||
}
|
||||
|
||||
private static TransportPublishRequest requestWithTimeout(Duration timeout) {
|
||||
PublishOptions defaults = PublishOptions.defaults();
|
||||
return new TransportPublishRequest(
|
||||
destinationProfile(),
|
||||
envelope(64),
|
||||
new PublishOptions(
|
||||
timeout, defaults.confirmation(), defaults.deduplication(), defaults.brokerHints()));
|
||||
}
|
||||
|
||||
private static DestinationProfile destinationProfile() {
|
||||
return new DestinationProfile(
|
||||
new DestinationName("orders.v1"),
|
||||
|
||||
@@ -3,5 +3,7 @@ apply plugin: 'java-library'
|
||||
dependencies {
|
||||
api project(':messaging:messaging-core-api')
|
||||
|
||||
implementation 'io.micrometer:micrometer-core'
|
||||
// api: MessagingMetrics' public constructor takes a MeterRegistry, so wiring it requires
|
||||
// naming the type.
|
||||
api 'io.micrometer:micrometer-core'
|
||||
}
|
||||
|
||||
+30
-5
@@ -54,11 +54,19 @@ public final class CardinalityGuard {
|
||||
if (values.contains(value)) {
|
||||
return true;
|
||||
}
|
||||
if (values.size() >= limitPerDimension) {
|
||||
return false;
|
||||
// Size-then-add was not atomic: N threads could each read size == limit - 1 and each add, so
|
||||
// the configured limit was an average rather than a bound. A guard that can be exceeded under
|
||||
// load is no guard — load is when it matters.
|
||||
synchronized (values) {
|
||||
if (values.contains(value)) {
|
||||
return true;
|
||||
}
|
||||
if (values.size() >= limitPerDimension) {
|
||||
return false;
|
||||
}
|
||||
values.add(value);
|
||||
return true;
|
||||
}
|
||||
values.add(value);
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -69,13 +77,30 @@ public final class CardinalityGuard {
|
||||
*/
|
||||
public boolean admit(MessagingTags tags) {
|
||||
Objects.requireNonNull(tags, "tags must not be null");
|
||||
// Preflight every dimension before committing any of them.
|
||||
//
|
||||
// The loop used to admit each dimension as it went, so a tag set rejected on its last
|
||||
// dimension had already permanently added the earlier ones — spending the budget of a bounded
|
||||
// dimension on a series that was never emitted.
|
||||
Map<String, String> pending = tags.asMap();
|
||||
for (Map.Entry<String, String> entry : pending.entrySet()) {
|
||||
if (!wouldAdmit(entry.getKey(), entry.getValue())) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
boolean admitted = true;
|
||||
for (Map.Entry<String, String> entry : tags.asMap().entrySet()) {
|
||||
for (Map.Entry<String, String> entry : pending.entrySet()) {
|
||||
admitted &= admit(entry.getKey(), entry.getValue());
|
||||
}
|
||||
return admitted;
|
||||
}
|
||||
|
||||
/** Whether a value is already known or would still fit, without recording it. */
|
||||
private boolean wouldAdmit(String dimension, String value) {
|
||||
Set<String> values = observed.get(dimension);
|
||||
return values == null || values.contains(value) || values.size() < limitPerDimension;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns how many distinct values a dimension has taken.
|
||||
*
|
||||
|
||||
+22
-8
@@ -119,17 +119,31 @@ public final class MessagingMetrics implements MessagingObservation {
|
||||
// Redact before anything else touches the values. Diagnostics are the one place where a caller
|
||||
// can pass arbitrary keys, so this is where a secret would otherwise reach the backend.
|
||||
Map<String, String> safe = redactor.sanitize(diagnostics);
|
||||
// The value never becomes a tag.
|
||||
//
|
||||
// It used to: every diagnostic key and value was attached to a counter, behind a guard that
|
||||
// only bounded the base dimensions. One unique message id, exception message or URL per
|
||||
// request created one meter series per request — permanently, in the backend and in this
|
||||
// process's heap — and the redactor only masks keys it recognises, so free-form text carried
|
||||
// whatever it carried.
|
||||
//
|
||||
// What stays is the shape: which diagnostic keys occurred, counted against the bounded base
|
||||
// dimensions. The values belong in a structured log or a trace event, where they are bounded
|
||||
// by retention rather than by cardinality.
|
||||
admitted(tags)
|
||||
.ifPresent(
|
||||
micrometerTags ->
|
||||
safe.forEach(
|
||||
(key, value) ->
|
||||
registry
|
||||
.counter(
|
||||
"messaging.diagnostics",
|
||||
micrometerTags.and(
|
||||
Tag.of("diagnostic", key), Tag.of("value", value)))
|
||||
.increment()));
|
||||
safe.keySet()
|
||||
.forEach(
|
||||
key -> {
|
||||
if (guard.admit("diagnostic", key)) {
|
||||
registry
|
||||
.counter(
|
||||
"messaging.diagnostics",
|
||||
micrometerTags.and(Tag.of("diagnostic", key)))
|
||||
.increment();
|
||||
}
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+55
-1
@@ -84,7 +84,61 @@ class MessagingSecretLeakTest {
|
||||
.map(Tag::getValue)
|
||||
.toList();
|
||||
|
||||
assertThat(emitted).doesNotContain("hunter2").contains("unroutable");
|
||||
// No diagnostic value reaches the registry at all — not the secret, and not the innocuous one
|
||||
// either. A value tag is one meter series per distinct value, permanently, and the redactor
|
||||
// only masks keys it recognises: free-form text carries whatever it carries.
|
||||
assertThat(emitted).doesNotContain("hunter2").doesNotContain("unroutable");
|
||||
assertThat(
|
||||
registry.getMeters().stream()
|
||||
.map(Meter::getId)
|
||||
.flatMap(id -> id.getTags().stream())
|
||||
.filter(tag -> tag.getKey().equals("diagnostic"))
|
||||
.map(Tag::getValue)
|
||||
.toList())
|
||||
.as("the shape is kept: which diagnostic keys occurred, against bounded base dimensions")
|
||||
.contains("reason");
|
||||
}
|
||||
|
||||
@Test
|
||||
@org.junit.jupiter.api.DisplayName("ten thousand distinct diagnostic values create no new series")
|
||||
void distinctDiagnosticValuesCreateNoNewSeries() {
|
||||
MessagingMetrics metrics = new MessagingMetrics(registry, new CardinalityGuard(100), redactor);
|
||||
MessagingTags tags = MessagingTags.of("kafka", "orders.v1", "publish", "rejected");
|
||||
|
||||
for (int index = 0; index < 10_000; index++) {
|
||||
metrics.recordDiagnostics(tags, Map.of("reason", "unique-value-" + index));
|
||||
}
|
||||
|
||||
assertThat(
|
||||
registry.getMeters().stream()
|
||||
.filter(meter -> meter.getId().getName().equals("messaging.diagnostics"))
|
||||
.toList())
|
||||
.as("one unique message id per request used to create one meter series per request")
|
||||
.hasSize(1);
|
||||
}
|
||||
|
||||
@Test
|
||||
@org.junit.jupiter.api.DisplayName("concurrent admission never exceeds the configured limit")
|
||||
void concurrentAdmissionNeverExceedsTheLimit() throws Exception {
|
||||
CardinalityGuard guard = new CardinalityGuard(10);
|
||||
java.util.concurrent.ExecutorService pool =
|
||||
java.util.concurrent.Executors.newFixedThreadPool(8);
|
||||
try {
|
||||
java.util.List<java.util.concurrent.Callable<Boolean>> work =
|
||||
java.util.stream.IntStream.range(0, 200)
|
||||
.<java.util.concurrent.Callable<Boolean>>mapToObj(
|
||||
index -> () -> guard.admit("destination", "d-" + index))
|
||||
.toList();
|
||||
for (java.util.concurrent.Future<Boolean> future : pool.invokeAll(work)) {
|
||||
future.get();
|
||||
}
|
||||
} finally {
|
||||
pool.shutdownNow();
|
||||
}
|
||||
|
||||
assertThat(guard.cardinality("destination"))
|
||||
.as("size-then-add is not atomic, so the limit was an average rather than a bound")
|
||||
.isEqualTo(10);
|
||||
}
|
||||
|
||||
@Test
|
||||
|
||||
+10
@@ -31,6 +31,9 @@ class SecretLeakStaticScanTest {
|
||||
Pattern.compile("(System\\.out|System\\.err|printStackTrace\\()");
|
||||
|
||||
/** A Java string literal, including escaped quotes. */
|
||||
/** Whether the scanner is currently inside a text block. */
|
||||
private static final ThreadLocal<Boolean> IN_TEXT_BLOCK = ThreadLocal.withInitial(() -> false);
|
||||
|
||||
private static final Pattern STRING_LITERAL = Pattern.compile("\"(\\\\.|[^\"\\\\])*\"");
|
||||
|
||||
/** The operand immediately before or after a string concatenation. */
|
||||
@@ -129,6 +132,13 @@ class SecretLeakStaticScanTest {
|
||||
* @return true when the line concatenates a sensitive value
|
||||
*/
|
||||
private static boolean leaksASensitiveValue(String line) {
|
||||
if (line.contains("\"\"\"") || IN_TEXT_BLOCK.get()) {
|
||||
// A text block is SQL or JSON, not Java code. The scanner replaced ordinary string literals
|
||||
// and then read the contents of text blocks as if they were expressions, so a column named
|
||||
// `lease_token` in an UPDATE statement looked like a concatenated credential.
|
||||
IN_TEXT_BLOCK.set(IN_TEXT_BLOCK.get() != line.contains("\"\"\""));
|
||||
return false;
|
||||
}
|
||||
String code = STRING_LITERAL.matcher(line).replaceAll("\"\"");
|
||||
if (!code.contains("+")) {
|
||||
return false;
|
||||
|
||||
+5
@@ -6,6 +6,11 @@ dependencies {
|
||||
api project(':messaging:messaging-policy')
|
||||
api project(':messaging:messaging-observability')
|
||||
|
||||
// The destructive-operation journal lives here because it needs exactly what the outbox needs:
|
||||
// one relational database every replica can see, and a migration lane that already exists. The
|
||||
// contract it implements belongs to the admin API.
|
||||
api project(':messaging:messaging-admin-api')
|
||||
|
||||
implementation 'org.springframework:spring-jdbc'
|
||||
implementation 'org.springframework:spring-tx'
|
||||
|
||||
+3
-3
@@ -39,9 +39,9 @@ commons-logging:commons-logging:1.3.5=compileClasspath,runtimeClasspath,testComp
|
||||
info.picocli:picocli:4.7.7=checkstyle
|
||||
io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnotationProcessor
|
||||
io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor
|
||||
io.micrometer:micrometer-commons:1.16.0=runtimeClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-core:1.16.0=runtimeClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-observation:1.16.0=runtimeClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-commons:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-core:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-observation:1.16.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
javax.inject:javax.inject:1=annotationProcessor,testAnnotationProcessor
|
||||
jaxen:jaxen:2.0.0=spotbugs
|
||||
net.bytebuddy:byte-buddy:1.17.8=testCompileClasspath,testRuntimeClasspath
|
||||
+326
@@ -0,0 +1,326 @@
|
||||
package dev.caskeleton.messaging.outbox;
|
||||
|
||||
import dev.caskeleton.messaging.admin.AdminOperationJournal;
|
||||
import dev.caskeleton.messaging.admin.AdminOperationLease;
|
||||
import dev.caskeleton.messaging.admin.AdminOperationRecord;
|
||||
import dev.caskeleton.messaging.admin.AdminOperationState;
|
||||
import dev.caskeleton.messaging.admin.PlanDigest;
|
||||
import dev.caskeleton.messaging.api.error.MessageAuthorizationException;
|
||||
import dev.caskeleton.messaging.api.error.MessagingConfigurationException;
|
||||
import java.sql.Connection;
|
||||
import java.sql.PreparedStatement;
|
||||
import java.sql.ResultSet;
|
||||
import java.sql.SQLException;
|
||||
import java.sql.Timestamp;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
import javax.sql.DataSource;
|
||||
import org.springframework.jdbc.datasource.DataSourceUtils;
|
||||
|
||||
/**
|
||||
* The shared, durable destructive-operation journal.
|
||||
*
|
||||
* <p>Lives beside the outbox because it needs the same thing the outbox needs and nothing more: one
|
||||
* relational database that every replica can see. The uniqueness that stops a second execution is
|
||||
* the primary key on {@code (approval_ticket, plan_digest)}, enforced by the database rather than
|
||||
* by a check-then-act in application code — two replicas that read "no row" at the same instant
|
||||
* would both proceed, and only the constraint makes exactly one of them win.
|
||||
*
|
||||
* <p>Takeover is conditional on the row still carrying the token the caller read. That is what
|
||||
* fences a stalled runtime out: it wakes up holding token 3, the row is at token 4, and every write
|
||||
* it attempts is refused rather than overwriting the replica that resumed the work.
|
||||
*/
|
||||
public final class JdbcAdminOperationJournal implements AdminOperationJournal {
|
||||
|
||||
private static final String SELECT =
|
||||
"""
|
||||
SELECT approval_ticket, plan_digest, operation_id, state, items_completed,
|
||||
lease_owner, lease_token, lease_expires_at, started_at, finished_at, failure_code
|
||||
FROM messaging_admin_operation
|
||||
WHERE approval_ticket = ? AND plan_digest = ?
|
||||
""";
|
||||
|
||||
private static final String INSERT =
|
||||
"""
|
||||
INSERT INTO messaging_admin_operation (
|
||||
approval_ticket, plan_digest, operation_id, state, items_completed,
|
||||
lease_owner, lease_token, lease_expires_at, started_at)
|
||||
VALUES (?, ?, ?, 'STARTED', 0, ?, 1, ?, ?)
|
||||
ON CONFLICT (approval_ticket, plan_digest) DO NOTHING
|
||||
""";
|
||||
|
||||
private static final String TAKE_OVER =
|
||||
"""
|
||||
UPDATE messaging_admin_operation
|
||||
SET operation_id = ?,
|
||||
state = 'STARTED',
|
||||
lease_owner = ?,
|
||||
lease_token = lease_token + 1,
|
||||
lease_expires_at = ?,
|
||||
finished_at = NULL,
|
||||
failure_code = NULL
|
||||
WHERE approval_ticket = ?
|
||||
AND plan_digest = ?
|
||||
AND lease_token = ?
|
||||
-- Only a failed operation or one whose lease ran out may be taken over. A live STARTED row
|
||||
-- means another replica is executing it right now.
|
||||
AND (state = 'FAILED' OR lease_expires_at <= ?)
|
||||
RETURNING lease_token, items_completed
|
||||
""";
|
||||
|
||||
private static final String CHECKPOINT =
|
||||
"""
|
||||
UPDATE messaging_admin_operation
|
||||
SET items_completed = GREATEST(items_completed, ?),
|
||||
lease_expires_at = ?
|
||||
WHERE approval_ticket = ? AND plan_digest = ? AND lease_token = ?
|
||||
""";
|
||||
|
||||
private static final String SETTLE =
|
||||
"""
|
||||
UPDATE messaging_admin_operation
|
||||
SET state = ?,
|
||||
items_completed = GREATEST(items_completed, ?),
|
||||
finished_at = ?,
|
||||
failure_code = ?
|
||||
WHERE approval_ticket = ? AND plan_digest = ? AND lease_token = ?
|
||||
""";
|
||||
|
||||
private final DataSource dataSource;
|
||||
|
||||
/**
|
||||
* Creates the journal.
|
||||
*
|
||||
* @param dataSource the shared database every replica reads
|
||||
*/
|
||||
public JdbcAdminOperationJournal(DataSource dataSource) {
|
||||
this.dataSource = Objects.requireNonNull(dataSource, "dataSource must not be null");
|
||||
}
|
||||
|
||||
@Override
|
||||
public AdminOperationLease begin(
|
||||
String approvalTicket,
|
||||
PlanDigest planDigest,
|
||||
String operationId,
|
||||
String leaseOwner,
|
||||
Duration leaseDuration,
|
||||
Instant now) {
|
||||
Objects.requireNonNull(planDigest, "planDigest must not be null");
|
||||
Objects.requireNonNull(leaseDuration, "leaseDuration must not be null");
|
||||
Objects.requireNonNull(now, "now must not be null");
|
||||
Instant expiry = now.plus(leaseDuration);
|
||||
|
||||
Connection connection = DataSourceUtils.getConnection(dataSource);
|
||||
try {
|
||||
if (insertClaim(
|
||||
connection, approvalTicket, planDigest, operationId, leaseOwner, expiry, now)) {
|
||||
return new AdminOperationLease(approvalTicket, planDigest, operationId, leaseOwner, 1L, 0);
|
||||
}
|
||||
AdminOperationRecord existing =
|
||||
read(connection, approvalTicket, planDigest)
|
||||
.orElseThrow(
|
||||
() ->
|
||||
new MessagingConfigurationException(
|
||||
"ADMIN_JOURNAL_INCONSISTENT",
|
||||
"the insert conflicted but no row exists for approval "
|
||||
+ approvalTicket));
|
||||
refuseIfNotResumable(existing, approvalTicket, now);
|
||||
return takeOver(
|
||||
connection, existing, approvalTicket, planDigest, operationId, leaseOwner, expiry, now);
|
||||
} catch (SQLException failure) {
|
||||
throw new MessagingConfigurationException(
|
||||
"ADMIN_JOURNAL_UNAVAILABLE", "the operation journal could not be reached", failure);
|
||||
} finally {
|
||||
DataSourceUtils.releaseConnection(connection, dataSource);
|
||||
}
|
||||
}
|
||||
|
||||
private static void refuseIfNotResumable(
|
||||
AdminOperationRecord existing, String approvalTicket, Instant now) {
|
||||
if (existing.state() == AdminOperationState.COMPLETED) {
|
||||
throw new MessageAuthorizationException(
|
||||
"APPROVAL_ALREADY_EXECUTED",
|
||||
("approval %s was already executed to completion at %s; an approval authorises one"
|
||||
+ " execution, not a standing permission")
|
||||
.formatted(approvalTicket, existing.finishedAt().orElse(existing.startedAt())));
|
||||
}
|
||||
if (existing.state() == AdminOperationState.STARTED
|
||||
&& now.isBefore(existing.leaseExpiresAt())) {
|
||||
throw new MessageAuthorizationException(
|
||||
"ADMIN_OPERATION_IN_FLIGHT",
|
||||
("approval %s is being executed by %s until at least %s; two runtimes executing one"
|
||||
+ " approval is a duplicate storm, not a faster redrive")
|
||||
.formatted(approvalTicket, existing.leaseOwner(), existing.leaseExpiresAt()));
|
||||
}
|
||||
}
|
||||
|
||||
private boolean insertClaim(
|
||||
Connection connection,
|
||||
String approvalTicket,
|
||||
PlanDigest planDigest,
|
||||
String operationId,
|
||||
String leaseOwner,
|
||||
Instant expiry,
|
||||
Instant now)
|
||||
throws SQLException {
|
||||
try (PreparedStatement statement = connection.prepareStatement(INSERT)) {
|
||||
statement.setString(1, approvalTicket);
|
||||
statement.setString(2, planDigest.value());
|
||||
statement.setString(3, operationId);
|
||||
statement.setString(4, leaseOwner);
|
||||
statement.setTimestamp(5, Timestamp.from(expiry));
|
||||
statement.setTimestamp(6, Timestamp.from(now));
|
||||
return statement.executeUpdate() == 1;
|
||||
}
|
||||
}
|
||||
|
||||
private AdminOperationLease takeOver(
|
||||
Connection connection,
|
||||
AdminOperationRecord existing,
|
||||
String approvalTicket,
|
||||
PlanDigest planDigest,
|
||||
String operationId,
|
||||
String leaseOwner,
|
||||
Instant expiry,
|
||||
Instant now)
|
||||
throws SQLException {
|
||||
try (PreparedStatement statement = connection.prepareStatement(TAKE_OVER)) {
|
||||
statement.setString(1, operationId);
|
||||
statement.setString(2, leaseOwner);
|
||||
statement.setTimestamp(3, Timestamp.from(expiry));
|
||||
statement.setString(4, approvalTicket);
|
||||
statement.setString(5, planDigest.value());
|
||||
statement.setLong(6, existing.leaseToken());
|
||||
statement.setTimestamp(7, Timestamp.from(now));
|
||||
try (ResultSet rows = statement.executeQuery()) {
|
||||
if (!rows.next()) {
|
||||
// Another replica took it over between the read and this update. Refusing is correct:
|
||||
// exactly one runtime executes an approval, and it is not this one.
|
||||
throw new MessageAuthorizationException(
|
||||
"ADMIN_OPERATION_IN_FLIGHT",
|
||||
"approval " + approvalTicket + " was taken over by another runtime");
|
||||
}
|
||||
return new AdminOperationLease(
|
||||
approvalTicket,
|
||||
planDigest,
|
||||
operationId,
|
||||
leaseOwner,
|
||||
rows.getLong("lease_token"),
|
||||
rows.getInt("items_completed"));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public void checkpoint(
|
||||
AdminOperationLease lease, int itemsCompleted, Duration leaseDuration, Instant now) {
|
||||
Objects.requireNonNull(lease, "lease must not be null");
|
||||
Connection connection = DataSourceUtils.getConnection(dataSource);
|
||||
try (PreparedStatement statement = connection.prepareStatement(CHECKPOINT)) {
|
||||
statement.setInt(1, itemsCompleted);
|
||||
statement.setTimestamp(2, Timestamp.from(now.plus(leaseDuration)));
|
||||
statement.setString(3, lease.approvalTicket());
|
||||
statement.setString(4, lease.planDigest().value());
|
||||
statement.setLong(5, lease.leaseToken());
|
||||
requireOwned(statement.executeUpdate(), lease);
|
||||
} catch (SQLException failure) {
|
||||
throw new MessagingConfigurationException(
|
||||
"ADMIN_JOURNAL_UNAVAILABLE", "the operation journal could not be reached", failure);
|
||||
} finally {
|
||||
DataSourceUtils.releaseConnection(connection, dataSource);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public void complete(AdminOperationLease lease, int itemsCompleted, Instant now) {
|
||||
settle(lease, AdminOperationState.COMPLETED, itemsCompleted, null, now);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void fail(AdminOperationLease lease, int itemsCompleted, String failureCode, Instant now) {
|
||||
Objects.requireNonNull(failureCode, "failureCode must not be null");
|
||||
settle(lease, AdminOperationState.FAILED, itemsCompleted, failureCode, now);
|
||||
}
|
||||
|
||||
private void settle(
|
||||
AdminOperationLease lease,
|
||||
AdminOperationState state,
|
||||
int itemsCompleted,
|
||||
String failureCode,
|
||||
Instant now) {
|
||||
Objects.requireNonNull(lease, "lease must not be null");
|
||||
Connection connection = DataSourceUtils.getConnection(dataSource);
|
||||
try (PreparedStatement statement = connection.prepareStatement(SETTLE)) {
|
||||
statement.setString(1, state.name());
|
||||
statement.setInt(2, itemsCompleted);
|
||||
statement.setTimestamp(3, Timestamp.from(now));
|
||||
statement.setString(4, failureCode);
|
||||
statement.setString(5, lease.approvalTicket());
|
||||
statement.setString(6, lease.planDigest().value());
|
||||
statement.setLong(7, lease.leaseToken());
|
||||
requireOwned(statement.executeUpdate(), lease);
|
||||
} catch (SQLException failure) {
|
||||
throw new MessagingConfigurationException(
|
||||
"ADMIN_JOURNAL_UNAVAILABLE", "the operation journal could not be reached", failure);
|
||||
} finally {
|
||||
DataSourceUtils.releaseConnection(connection, dataSource);
|
||||
}
|
||||
}
|
||||
|
||||
private static void requireOwned(int updated, AdminOperationLease lease) {
|
||||
if (updated == 1) {
|
||||
return;
|
||||
}
|
||||
throw new MessageAuthorizationException(
|
||||
"ADMIN_OPERATION_LEASE_LOST",
|
||||
("this runtime no longer holds the lease on approval %s at token %d; another runtime took"
|
||||
+ " the operation over and its progress must not be overwritten")
|
||||
.formatted(lease.approvalTicket(), lease.leaseToken()));
|
||||
}
|
||||
|
||||
@Override
|
||||
public Optional<AdminOperationRecord> find(String approvalTicket, PlanDigest planDigest) {
|
||||
Connection connection = DataSourceUtils.getConnection(dataSource);
|
||||
try {
|
||||
return read(connection, approvalTicket, planDigest);
|
||||
} catch (SQLException failure) {
|
||||
throw new MessagingConfigurationException(
|
||||
"ADMIN_JOURNAL_UNAVAILABLE", "the operation journal could not be reached", failure);
|
||||
} finally {
|
||||
DataSourceUtils.releaseConnection(connection, dataSource);
|
||||
}
|
||||
}
|
||||
|
||||
private Optional<AdminOperationRecord> read(
|
||||
Connection connection, String approvalTicket, PlanDigest planDigest) throws SQLException {
|
||||
try (PreparedStatement statement = connection.prepareStatement(SELECT)) {
|
||||
statement.setString(1, approvalTicket);
|
||||
statement.setString(2, planDigest.value());
|
||||
try (ResultSet rows = statement.executeQuery()) {
|
||||
if (!rows.next()) {
|
||||
return Optional.empty();
|
||||
}
|
||||
return Optional.of(
|
||||
new AdminOperationRecord(
|
||||
rows.getString("approval_ticket"),
|
||||
new PlanDigest(rows.getString("plan_digest")),
|
||||
rows.getString("operation_id"),
|
||||
AdminOperationState.valueOf(rows.getString("state")),
|
||||
rows.getInt("items_completed"),
|
||||
rows.getString("lease_owner"),
|
||||
rows.getLong("lease_token"),
|
||||
rows.getTimestamp("lease_expires_at").toInstant(),
|
||||
rows.getTimestamp("started_at").toInstant(),
|
||||
Optional.ofNullable(rows.getTimestamp("finished_at")).map(Timestamp::toInstant),
|
||||
Optional.ofNullable(rows.getString("failure_code"))));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean isDurable() {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
+648
@@ -0,0 +1,648 @@
|
||||
package dev.caskeleton.messaging.outbox;
|
||||
|
||||
import dev.caskeleton.messaging.api.ContentType;
|
||||
import dev.caskeleton.messaging.api.MessageId;
|
||||
import dev.caskeleton.messaging.api.MessageType;
|
||||
import dev.caskeleton.messaging.api.SchemaVersion;
|
||||
import dev.caskeleton.messaging.api.destination.DestinationName;
|
||||
import dev.caskeleton.messaging.api.error.MessagingConfigurationException;
|
||||
import dev.caskeleton.messaging.reliability.OutboxLease;
|
||||
import dev.caskeleton.messaging.reliability.OutboxRecord;
|
||||
import dev.caskeleton.messaging.reliability.OutboxRepository;
|
||||
import dev.caskeleton.messaging.reliability.OutboxStatus;
|
||||
import dev.caskeleton.messaging.reliability.OutboxTransitionResult;
|
||||
import java.sql.Connection;
|
||||
import java.sql.PreparedStatement;
|
||||
import java.sql.ResultSet;
|
||||
import java.sql.SQLException;
|
||||
import java.sql.Timestamp;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
import javax.sql.DataSource;
|
||||
import org.springframework.jdbc.datasource.DataSourceUtils;
|
||||
import org.springframework.transaction.support.TransactionSynchronizationManager;
|
||||
|
||||
/**
|
||||
* The PostgreSQL outbox.
|
||||
*
|
||||
* <p>{@link #append} deliberately takes no connection of its own: it uses the one the caller is
|
||||
* already inside, which is the entire mechanism. An outbox row written on a separate connection
|
||||
* commits independently of the business change and reopens the window the pattern exists to close.
|
||||
*
|
||||
* <p>Leasing uses {@code SELECT ... FOR UPDATE SKIP LOCKED}. Two relay instances polling the same
|
||||
* table would otherwise both claim the same rows and publish them twice; {@code SKIP LOCKED} lets
|
||||
* the second instance step over what the first already holds instead of blocking behind it.
|
||||
*/
|
||||
public final class JdbcOutboxRepository implements OutboxRepository {
|
||||
|
||||
private static final String INSERT =
|
||||
"""
|
||||
INSERT INTO messaging_outbox (
|
||||
message_id, destination, message_type, schema_version, content_type,
|
||||
payload, headers, created_at, status, attempts)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?::jsonb, ?, ?, ?)
|
||||
""";
|
||||
|
||||
private static final String LEASE =
|
||||
"""
|
||||
WITH claimable AS (
|
||||
SELECT message_id
|
||||
FROM messaging_outbox
|
||||
-- IN_FLIGHT is claimable too, but only once its lease has expired. A relay that dies
|
||||
-- mid-publish leaves rows in that state; excluding them would strand those messages
|
||||
-- forever, which is the one failure mode the outbox exists to prevent.
|
||||
WHERE status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT')
|
||||
AND (lease_expires_at IS NULL OR lease_expires_at <= ?)
|
||||
ORDER BY created_at
|
||||
LIMIT ?
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
UPDATE messaging_outbox o
|
||||
SET status = 'IN_FLIGHT', lease_expires_at = ?
|
||||
FROM claimable c
|
||||
WHERE o.message_id = c.message_id
|
||||
RETURNING o.message_id, o.destination, o.message_type, o.schema_version, o.content_type,
|
||||
o.payload, o.headers, o.created_at, o.status, o.attempts,
|
||||
o.lease_expires_at, o.last_failure_code
|
||||
""";
|
||||
|
||||
/**
|
||||
* The claim that records who holds the row and increments its fencing token.
|
||||
*
|
||||
* <p>The token increments on the server inside the same statement that grants the claim, so two
|
||||
* relays racing for the same row cannot receive the same number.
|
||||
*/
|
||||
private static final String CLAIM =
|
||||
"""
|
||||
WITH claimable AS (
|
||||
SELECT message_id
|
||||
FROM messaging_outbox
|
||||
WHERE status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT')
|
||||
AND (lease_expires_at IS NULL OR lease_expires_at <= ?)
|
||||
-- The retry clock lives in the row, not in the relay's memory. Without these two
|
||||
-- predicates an AMBIGUOUS row became claimable again on the very next pass, so a
|
||||
-- broker outage meant the whole backlog was republished every poll interval and the
|
||||
-- configured attempt budget was a number nothing consulted.
|
||||
AND (next_attempt_at IS NULL OR next_attempt_at <= ?)
|
||||
AND attempts < ?
|
||||
ORDER BY created_at
|
||||
LIMIT ?
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
UPDATE messaging_outbox o
|
||||
SET status = 'IN_FLIGHT',
|
||||
lease_expires_at = ?,
|
||||
lease_owner = ?,
|
||||
lease_token = o.lease_token + 1
|
||||
FROM claimable c
|
||||
WHERE o.message_id = c.message_id
|
||||
RETURNING o.message_id, o.destination, o.message_type, o.schema_version, o.content_type,
|
||||
o.payload, o.headers, o.created_at, o.status, o.attempts,
|
||||
o.lease_expires_at, o.last_failure_code, o.lease_owner, o.lease_token
|
||||
""";
|
||||
|
||||
private final DataSource dataSource;
|
||||
|
||||
/**
|
||||
* Creates a repository over a data source.
|
||||
*
|
||||
* @param dataSource the outbox data source
|
||||
*/
|
||||
public JdbcOutboxRepository(DataSource dataSource) {
|
||||
this.dataSource = Objects.requireNonNull(dataSource, "dataSource must not be null");
|
||||
}
|
||||
|
||||
/**
|
||||
* Appends a record on an existing connection.
|
||||
*
|
||||
* <p>Package-private. It used to be the public overload "business code should use", which meant
|
||||
* the safe path was the one a caller had to know about and the interface method — the one the
|
||||
* port documents — opened its own connection. Handing a {@code Connection} across a reliability
|
||||
* API also makes the transaction the caller's problem to remember; {@link #append(OutboxRecord)}
|
||||
* now joins the caller's transaction itself and refuses when there is none.
|
||||
*
|
||||
* @param connection the caller's transactional connection
|
||||
* @param record the record to write
|
||||
*/
|
||||
void append(Connection connection, OutboxRecord record) {
|
||||
Objects.requireNonNull(connection, "connection must not be null");
|
||||
Objects.requireNonNull(record, "record must not be null");
|
||||
try (PreparedStatement statement = connection.prepareStatement(INSERT)) {
|
||||
statement.setObject(1, record.messageId().value());
|
||||
statement.setString(2, record.destination().value());
|
||||
statement.setString(3, record.messageType().value());
|
||||
statement.setInt(4, record.schemaVersion().value());
|
||||
statement.setString(5, record.contentType().value());
|
||||
statement.setBytes(6, record.payload());
|
||||
statement.setString(7, toJson(record.headers()));
|
||||
statement.setTimestamp(8, Timestamp.from(record.createdAt()));
|
||||
statement.setString(9, record.status().name());
|
||||
statement.setInt(10, record.attempts());
|
||||
statement.executeUpdate();
|
||||
} catch (SQLException exception) {
|
||||
throw new MessagingConfigurationException(
|
||||
"OUTBOX_APPEND_FAILED", "could not append an outbox record", exception);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public void append(OutboxRecord record) {
|
||||
Objects.requireNonNull(record, "record must not be null");
|
||||
// The port says this runs inside the caller's business transaction. It did not: it opened a raw
|
||||
// connection from the pool, which auto-commits independently — so a business transaction that
|
||||
// rolled back afterwards left the event behind and the relay published a change that never
|
||||
// happened.
|
||||
requireActiveTransaction("OUTBOX_TRANSACTION_REQUIRED", "appending to the outbox");
|
||||
Connection connection = DataSourceUtils.getConnection(dataSource);
|
||||
try {
|
||||
append(connection, record);
|
||||
} finally {
|
||||
// Releases only when the connection is not the transaction's; when it is, this is a no-op and
|
||||
// the caller's commit or rollback decides what happens to the row.
|
||||
DataSourceUtils.releaseConnection(connection, dataSource);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Refuses when there is no transaction to join, or when it cannot write.
|
||||
*
|
||||
* <p>Fail-fast rather than "work anyway": an append that silently runs outside the caller's
|
||||
* transaction produces exactly the ghost publication this repository exists to prevent, and it
|
||||
* produces it only on the rollback path — which is the path nobody exercises before production.
|
||||
*/
|
||||
private void requireActiveTransaction(String code, String what) {
|
||||
if (!TransactionSynchronizationManager.isActualTransactionActive()) {
|
||||
throw new MessagingConfigurationException(
|
||||
code,
|
||||
what
|
||||
+ " requires an active transaction; without one the row commits on its own and a"
|
||||
+ " later rollback of the business change leaves it behind");
|
||||
}
|
||||
if (TransactionSynchronizationManager.isCurrentTransactionReadOnly()) {
|
||||
throw new MessagingConfigurationException(
|
||||
code, what + " requires a writable transaction; the current one is read-only");
|
||||
}
|
||||
if (!TransactionSynchronizationManager.hasResource(dataSource)) {
|
||||
throw new MessagingConfigurationException(
|
||||
code,
|
||||
what
|
||||
+ " requires the active transaction to be bound to this DataSource; it is bound to"
|
||||
+ " another, so the two would commit independently");
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<OutboxRecord> leaseBatch(int batchSize, Duration leaseDuration, Instant now) {
|
||||
Objects.requireNonNull(leaseDuration, "leaseDuration must not be null");
|
||||
Objects.requireNonNull(now, "now must not be null");
|
||||
return withConnection(
|
||||
connection -> {
|
||||
List<OutboxRecord> leased = new ArrayList<>();
|
||||
try (PreparedStatement statement = connection.prepareStatement(LEASE)) {
|
||||
statement.setTimestamp(1, Timestamp.from(now));
|
||||
statement.setInt(2, batchSize);
|
||||
statement.setTimestamp(3, Timestamp.from(now.plus(leaseDuration)));
|
||||
try (ResultSet results = statement.executeQuery()) {
|
||||
while (results.next()) {
|
||||
leased.add(read(results));
|
||||
}
|
||||
}
|
||||
}
|
||||
return List.copyOf(leased);
|
||||
});
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<OutboxLease> claimBatch(
|
||||
String owner, int batchSize, Duration leaseDuration, Instant now) {
|
||||
return claimBatch(owner, batchSize, leaseDuration, now, Integer.MAX_VALUE);
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<OutboxLease> claimBatch(
|
||||
String owner, int batchSize, Duration leaseDuration, Instant now, int maxAttempts) {
|
||||
Objects.requireNonNull(owner, "owner must not be null");
|
||||
Objects.requireNonNull(leaseDuration, "leaseDuration must not be null");
|
||||
Objects.requireNonNull(now, "now must not be null");
|
||||
Instant expiresAt = now.plus(leaseDuration);
|
||||
return withConnection(
|
||||
connection -> {
|
||||
List<OutboxLease> claimed = new ArrayList<>();
|
||||
try (PreparedStatement statement = connection.prepareStatement(CLAIM)) {
|
||||
statement.setTimestamp(1, Timestamp.from(now));
|
||||
statement.setTimestamp(2, Timestamp.from(now));
|
||||
statement.setInt(3, maxAttempts);
|
||||
statement.setInt(4, batchSize);
|
||||
statement.setTimestamp(5, Timestamp.from(expiresAt));
|
||||
statement.setString(6, owner);
|
||||
try (ResultSet results = statement.executeQuery()) {
|
||||
while (results.next()) {
|
||||
claimed.add(
|
||||
new OutboxLease(
|
||||
read(results),
|
||||
results.getString("lease_owner"),
|
||||
results.getLong("lease_token"),
|
||||
expiresAt));
|
||||
}
|
||||
}
|
||||
}
|
||||
return List.copyOf(claimed);
|
||||
});
|
||||
}
|
||||
|
||||
@Override
|
||||
public OutboxTransitionResult markPublished(OutboxLease lease, Instant now) {
|
||||
return transition(
|
||||
lease,
|
||||
"UPDATE messaging_outbox SET status = 'PUBLISHED', published_at = ?, "
|
||||
+ "lease_expires_at = NULL, lease_owner = NULL, next_attempt_at = NULL, "
|
||||
+ "attempts = attempts + 1 ",
|
||||
now,
|
||||
null);
|
||||
}
|
||||
|
||||
@Override
|
||||
public OutboxTransitionResult markAmbiguous(OutboxLease lease, String failureCode, Instant now) {
|
||||
return markAmbiguous(lease, failureCode, now, now);
|
||||
}
|
||||
|
||||
@Override
|
||||
public OutboxTransitionResult markAmbiguous(
|
||||
OutboxLease lease, String failureCode, Instant now, Instant nextAttemptAt) {
|
||||
Objects.requireNonNull(nextAttemptAt, "nextAttemptAt must not be null");
|
||||
// The backoff the policy computed is written onto the row. Keeping it in the relay's memory
|
||||
// means a restart forgets it, and every relay computes its own — so the same row is retried by
|
||||
// as many workers as there are replicas, as fast as they poll.
|
||||
return transitionWithNextAttempt(
|
||||
lease,
|
||||
"UPDATE messaging_outbox SET status = 'AMBIGUOUS', lease_expires_at = NULL, "
|
||||
+ "lease_owner = NULL, last_failure_code = ?, attempts = attempts + 1, "
|
||||
+ "next_attempt_at = ? ",
|
||||
failureCode,
|
||||
nextAttemptAt);
|
||||
}
|
||||
|
||||
@Override
|
||||
public OutboxTransitionResult markExhausted(OutboxLease lease, String reason, Instant now) {
|
||||
return transition(
|
||||
lease,
|
||||
"UPDATE messaging_outbox SET status = 'EXHAUSTED', lease_expires_at = NULL, "
|
||||
+ "lease_owner = NULL, last_failure_code = ?, attempts = attempts + 1 ",
|
||||
now,
|
||||
reason);
|
||||
}
|
||||
|
||||
private OutboxTransitionResult transitionWithNextAttempt(
|
||||
OutboxLease lease, String setClause, String failureCode, Instant nextAttemptAt) {
|
||||
Objects.requireNonNull(lease, "lease must not be null");
|
||||
String sql =
|
||||
setClause
|
||||
+ "WHERE message_id = ? AND status = 'IN_FLIGHT' AND lease_owner = ? AND lease_token = ?";
|
||||
return withConnection(
|
||||
connection -> {
|
||||
try (PreparedStatement statement = connection.prepareStatement(sql)) {
|
||||
statement.setString(1, failureCode);
|
||||
statement.setTimestamp(2, Timestamp.from(nextAttemptAt));
|
||||
statement.setObject(3, lease.messageId().value());
|
||||
statement.setString(4, lease.owner());
|
||||
statement.setLong(5, lease.token());
|
||||
return statement.executeUpdate() == 1
|
||||
? OutboxTransitionResult.APPLIED
|
||||
: OutboxTransitionResult.STALE_LEASE;
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@Override
|
||||
public OutboxTransitionResult markFailed(OutboxLease lease, String failureCode, Instant now) {
|
||||
return transition(
|
||||
lease,
|
||||
"UPDATE messaging_outbox SET status = 'FAILED', lease_expires_at = NULL, "
|
||||
+ "lease_owner = NULL, last_failure_code = ?, attempts = attempts + 1 ",
|
||||
now,
|
||||
failureCode);
|
||||
}
|
||||
|
||||
@Override
|
||||
public OutboxTransitionResult releaseLease(OutboxLease lease) {
|
||||
return transition(
|
||||
lease,
|
||||
"UPDATE messaging_outbox SET status = 'PENDING', lease_expires_at = NULL, "
|
||||
+ "lease_owner = NULL ",
|
||||
null,
|
||||
null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Applies a terminal write, but only to the row this lease still owns.
|
||||
*
|
||||
* <p>The predicate carries the owner and the token as well as the id, so a relay that stalled
|
||||
* past its lease writes nothing: another relay's claim incremented the token, and this update
|
||||
* matches zero rows. Zero is reported rather than swallowed — a stale write means this worker may
|
||||
* have produced a duplicate publication, which is exactly what an operator needs to see.
|
||||
*/
|
||||
private OutboxTransitionResult transition(
|
||||
OutboxLease lease, String setClause, Instant now, String failureCode) {
|
||||
Objects.requireNonNull(lease, "lease must not be null");
|
||||
String sql =
|
||||
setClause
|
||||
+ "WHERE message_id = ? AND status = 'IN_FLIGHT' AND lease_owner = ? AND lease_token = ?";
|
||||
return withConnection(
|
||||
connection -> {
|
||||
try (PreparedStatement statement = connection.prepareStatement(sql)) {
|
||||
int index = 1;
|
||||
if (failureCode != null) {
|
||||
statement.setString(index++, failureCode);
|
||||
} else if (now != null) {
|
||||
statement.setTimestamp(index++, Timestamp.from(now));
|
||||
}
|
||||
statement.setObject(index++, lease.messageId().value());
|
||||
statement.setString(index++, lease.owner());
|
||||
statement.setLong(index, lease.token());
|
||||
return statement.executeUpdate() == 1
|
||||
? OutboxTransitionResult.APPLIED
|
||||
: OutboxTransitionResult.STALE_LEASE;
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@Override
|
||||
public void markPublished(MessageId messageId, Instant now) {
|
||||
update(
|
||||
"UPDATE messaging_outbox SET status = 'PUBLISHED', published_at = ?, "
|
||||
+ "lease_expires_at = NULL, attempts = attempts + 1 WHERE message_id = ?",
|
||||
messageId,
|
||||
now,
|
||||
null);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void markAmbiguous(MessageId messageId, String failureCode, Instant now) {
|
||||
// The lease is released rather than held: an ambiguous row must become claimable again so the
|
||||
// relay retries it under the same message id.
|
||||
update(
|
||||
"UPDATE messaging_outbox SET status = 'AMBIGUOUS', lease_expires_at = NULL, "
|
||||
+ "last_failure_code = ?, attempts = attempts + 1 WHERE message_id = ?",
|
||||
messageId,
|
||||
now,
|
||||
failureCode);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void markFailed(MessageId messageId, String failureCode, Instant now) {
|
||||
update(
|
||||
"UPDATE messaging_outbox SET status = 'FAILED', lease_expires_at = NULL, "
|
||||
+ "last_failure_code = ?, attempts = attempts + 1 WHERE message_id = ?",
|
||||
messageId,
|
||||
now,
|
||||
failureCode);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void releaseLease(MessageId messageId) {
|
||||
withConnection(
|
||||
connection -> {
|
||||
try (PreparedStatement statement =
|
||||
connection.prepareStatement(
|
||||
"UPDATE messaging_outbox SET status = 'PENDING', lease_expires_at = NULL "
|
||||
+ "WHERE message_id = ? AND status = 'IN_FLIGHT'")) {
|
||||
statement.setObject(1, messageId.value());
|
||||
statement.executeUpdate();
|
||||
}
|
||||
return null;
|
||||
});
|
||||
}
|
||||
|
||||
@Override
|
||||
public Optional<OutboxRecord> find(MessageId messageId) {
|
||||
return withConnection(
|
||||
connection -> {
|
||||
try (PreparedStatement statement =
|
||||
connection.prepareStatement(
|
||||
"SELECT message_id, destination, message_type, schema_version, content_type, "
|
||||
+ "payload, headers, created_at, status, attempts, lease_expires_at, "
|
||||
+ "last_failure_code FROM messaging_outbox WHERE message_id = ?")) {
|
||||
statement.setObject(1, messageId.value());
|
||||
try (ResultSet results = statement.executeQuery()) {
|
||||
return results.next() ? Optional.of(read(results)) : Optional.<OutboxRecord>empty();
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@Override
|
||||
public int purgePublishedBefore(Instant publishedBefore, int limit) {
|
||||
Objects.requireNonNull(publishedBefore, "publishedBefore must not be null");
|
||||
if (limit < 1) {
|
||||
throw new IllegalArgumentException("a bounded purge deletes at least one row per call");
|
||||
}
|
||||
// The CTE picks a bounded set of ids with SKIP LOCKED and deletes exactly those. An unbounded
|
||||
// DELETE holds locks and writes WAL in proportion to the whole backlog, which stalls the relay
|
||||
// and the business writes behind retention.
|
||||
return withConnection(
|
||||
connection -> {
|
||||
try (PreparedStatement statement =
|
||||
connection.prepareStatement(
|
||||
"""
|
||||
WITH expired AS (
|
||||
SELECT message_id
|
||||
FROM messaging_outbox
|
||||
WHERE status = 'PUBLISHED' AND published_at < ?
|
||||
ORDER BY published_at
|
||||
LIMIT ?
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
DELETE FROM messaging_outbox o
|
||||
USING expired e
|
||||
WHERE o.message_id = e.message_id
|
||||
""")) {
|
||||
statement.setTimestamp(1, Timestamp.from(publishedBefore));
|
||||
statement.setInt(2, limit);
|
||||
return statement.executeUpdate();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@Override
|
||||
public int purgePublishedBefore(Instant publishedBefore) {
|
||||
Objects.requireNonNull(publishedBefore, "publishedBefore must not be null");
|
||||
return withConnection(
|
||||
connection -> {
|
||||
try (PreparedStatement statement =
|
||||
connection.prepareStatement(
|
||||
"DELETE FROM messaging_outbox WHERE status = 'PUBLISHED' AND published_at < ?")) {
|
||||
statement.setTimestamp(1, Timestamp.from(publishedBefore));
|
||||
return statement.executeUpdate();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
private void update(String sql, MessageId messageId, Instant now, String failureCode) {
|
||||
withConnection(
|
||||
connection -> {
|
||||
try (PreparedStatement statement = connection.prepareStatement(sql)) {
|
||||
if (failureCode == null) {
|
||||
statement.setTimestamp(1, Timestamp.from(now));
|
||||
} else {
|
||||
statement.setString(1, failureCode);
|
||||
}
|
||||
statement.setObject(2, messageId.value());
|
||||
statement.executeUpdate();
|
||||
}
|
||||
return null;
|
||||
});
|
||||
}
|
||||
|
||||
private static OutboxRecord read(ResultSet results) throws SQLException {
|
||||
Timestamp lease = results.getTimestamp("lease_expires_at");
|
||||
return new OutboxRecord(
|
||||
new MessageId(results.getObject("message_id", java.util.UUID.class)),
|
||||
new DestinationName(results.getString("destination")),
|
||||
new MessageType(results.getString("message_type")),
|
||||
new SchemaVersion(results.getInt("schema_version")),
|
||||
new ContentType(results.getString("content_type")),
|
||||
results.getBytes("payload"),
|
||||
fromJson(results.getString("headers")),
|
||||
results.getTimestamp("created_at").toInstant(),
|
||||
OutboxStatus.valueOf(results.getString("status")),
|
||||
results.getInt("attempts"),
|
||||
Optional.ofNullable(lease).map(Timestamp::toInstant),
|
||||
Optional.ofNullable(results.getString("last_failure_code")));
|
||||
}
|
||||
|
||||
private <T> T withConnection(ConnectionCallback<T> callback) {
|
||||
try (Connection connection = dataSource.getConnection()) {
|
||||
return callback.apply(connection);
|
||||
} catch (SQLException exception) {
|
||||
throw new MessagingConfigurationException(
|
||||
"OUTBOX_QUERY_FAILED", "an outbox query failed", exception);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Serialises headers as a flat JSON object.
|
||||
*
|
||||
* <p>Hand-rolled rather than pulled from a JSON library so this module keeps no codec dependency:
|
||||
* outbox headers are always flat string pairs, validated by {@code MessageHeaders} before they
|
||||
* ever reach here.
|
||||
*/
|
||||
private static String toJson(Map<String, String> headers) {
|
||||
StringBuilder json = new StringBuilder("{");
|
||||
boolean first = true;
|
||||
for (Map.Entry<String, String> entry : headers.entrySet()) {
|
||||
if (!first) {
|
||||
json.append(',');
|
||||
}
|
||||
first = false;
|
||||
json.append('"')
|
||||
.append(escape(entry.getKey()))
|
||||
.append("\":\"")
|
||||
.append(escape(entry.getValue()))
|
||||
.append('"');
|
||||
}
|
||||
return json.append('}').toString();
|
||||
}
|
||||
|
||||
private static Map<String, String> fromJson(String json) {
|
||||
Map<String, String> headers = new LinkedHashMap<>();
|
||||
if (json == null || json.length() <= 2) {
|
||||
return headers;
|
||||
}
|
||||
String body = json.substring(1, json.length() - 1);
|
||||
int index = 0;
|
||||
while (index < body.length()) {
|
||||
int keyStart = body.indexOf('"', index);
|
||||
if (keyStart < 0) {
|
||||
break;
|
||||
}
|
||||
int keyEnd = findClosingQuote(body, keyStart + 1);
|
||||
int valueStart = body.indexOf('"', body.indexOf(':', keyEnd));
|
||||
int valueEnd = findClosingQuote(body, valueStart + 1);
|
||||
headers.put(
|
||||
unescape(body.substring(keyStart + 1, keyEnd)),
|
||||
unescape(body.substring(valueStart + 1, valueEnd)));
|
||||
index = valueEnd + 1;
|
||||
}
|
||||
return headers;
|
||||
}
|
||||
|
||||
private static int findClosingQuote(String text, int from) {
|
||||
for (int index = from; index < text.length(); index++) {
|
||||
if (text.charAt(index) == '"' && text.charAt(index - 1) != '\\') {
|
||||
return index;
|
||||
}
|
||||
}
|
||||
return text.length();
|
||||
}
|
||||
|
||||
/**
|
||||
* JSON string escaping, including the control characters.
|
||||
*
|
||||
* <p>Only backslash and quote were escaped. A header value containing a newline or a NUL — which
|
||||
* the value objects now refuse, but which older rows and other writers may still carry — produced
|
||||
* a document PostgreSQL rejects as invalid JSONB, so the append failed at the database with an
|
||||
* error naming neither the header nor the row. Control characters are the part of the JSON
|
||||
* grammar that is easy to forget and impossible to get away with.
|
||||
*/
|
||||
private static String escape(String value) {
|
||||
StringBuilder escaped = new StringBuilder(value.length() + 8);
|
||||
for (int index = 0; index < value.length(); index++) {
|
||||
char character = value.charAt(index);
|
||||
switch (character) {
|
||||
case '\\' -> escaped.append("\\\\");
|
||||
case '"' -> escaped.append("\\\"");
|
||||
case '\n' -> escaped.append("\\n");
|
||||
case '\r' -> escaped.append("\\r");
|
||||
case '\t' -> escaped.append("\\t");
|
||||
case '\b' -> escaped.append("\\b");
|
||||
case '\f' -> escaped.append("\\f");
|
||||
default -> {
|
||||
if (character < 0x20) {
|
||||
escaped.append("\\u%04x".formatted((int) character));
|
||||
} else {
|
||||
escaped.append(character);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return escaped.toString();
|
||||
}
|
||||
|
||||
private static String unescape(String value) {
|
||||
StringBuilder plain = new StringBuilder(value.length());
|
||||
for (int index = 0; index < value.length(); index++) {
|
||||
char character = value.charAt(index);
|
||||
if (character != '\\' || index == value.length() - 1) {
|
||||
plain.append(character);
|
||||
continue;
|
||||
}
|
||||
char next = value.charAt(++index);
|
||||
switch (next) {
|
||||
case 'n' -> plain.append('\n');
|
||||
case 'r' -> plain.append('\r');
|
||||
case 't' -> plain.append('\t');
|
||||
case 'b' -> plain.append('\b');
|
||||
case 'f' -> plain.append('\f');
|
||||
case 'u' -> {
|
||||
plain.append((char) Integer.parseInt(value.substring(index + 1, index + 5), 16));
|
||||
index += 4;
|
||||
}
|
||||
default -> plain.append(next);
|
||||
}
|
||||
}
|
||||
return plain.toString();
|
||||
}
|
||||
|
||||
/** A unit of work against a borrowed connection. */
|
||||
@FunctionalInterface
|
||||
private interface ConnectionCallback<T> {
|
||||
|
||||
T apply(Connection connection) throws SQLException;
|
||||
}
|
||||
}
|
||||
+24
-1
@@ -46,7 +46,30 @@ public final class OutboxEnvelopeFactory {
|
||||
Map<HeaderName, HeaderValue> headers = new LinkedHashMap<>();
|
||||
record
|
||||
.headers()
|
||||
.forEach((name, value) -> headers.put(new HeaderName(name), new HeaderValue(value)));
|
||||
.forEach(
|
||||
(name, value) -> {
|
||||
// The row's headers are application data. Promoting them straight into
|
||||
// MessageHeaders.platform let a row carrying `msg.id` or `msg.type` reach the wire
|
||||
// as if the platform had written it — and the consumer trusts the last such header,
|
||||
// so a forged identity lands in another message's inbox deduplication.
|
||||
//
|
||||
// Partition and ordering keys are the exception: they are reserved names the outbox
|
||||
// legitimately carries as routing metadata, and they are read below rather than
|
||||
// written as headers.
|
||||
if (ReservedHeaders.isReserved(name)
|
||||
&& !ReservedHeaders.PARTITION_KEY.equals(name)
|
||||
&& !ReservedHeaders.ORDERING_KEY.equals(name)) {
|
||||
throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException(
|
||||
"RESERVED_HEADER_IN_OUTBOX_ROW",
|
||||
"outbox row header '"
|
||||
+ name
|
||||
+ "' uses the reserved namespace; canonical identity and metadata come from"
|
||||
+ " the record's own columns, not from its header map");
|
||||
}
|
||||
if (!ReservedHeaders.isReserved(name)) {
|
||||
headers.put(new HeaderName(name), new HeaderValue(value));
|
||||
}
|
||||
});
|
||||
|
||||
return new MessageEnvelope<>(
|
||||
record.messageId(),
|
||||
+86
-15
@@ -4,8 +4,10 @@ import dev.caskeleton.messaging.api.destination.MessageDestination;
|
||||
import dev.caskeleton.messaging.api.publish.MessagePublisher;
|
||||
import dev.caskeleton.messaging.api.publish.PublishOptions;
|
||||
import dev.caskeleton.messaging.api.publish.PublishResult;
|
||||
import dev.caskeleton.messaging.reliability.OutboxLease;
|
||||
import dev.caskeleton.messaging.reliability.OutboxRecord;
|
||||
import dev.caskeleton.messaging.reliability.OutboxRepository;
|
||||
import dev.caskeleton.messaging.reliability.OutboxTransitionResult;
|
||||
import dev.caskeleton.messaging.schema.EncodedMessage;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
@@ -30,6 +32,17 @@ public final class OutboxRelay {
|
||||
private final MessagePublisher publisher;
|
||||
private final OutboxEnvelopeFactory envelopeFactory;
|
||||
private final int batchSize;
|
||||
|
||||
private final String owner;
|
||||
|
||||
private final OutboxRetryScheduler scheduler;
|
||||
|
||||
private static String defaultOwner() {
|
||||
return java.lang.ProcessHandle.current().pid()
|
||||
+ "@"
|
||||
+ java.util.UUID.randomUUID().toString().substring(0, 8);
|
||||
}
|
||||
|
||||
private final Duration leaseDuration;
|
||||
|
||||
/**
|
||||
@@ -47,6 +60,50 @@ public final class OutboxRelay {
|
||||
OutboxEnvelopeFactory envelopeFactory,
|
||||
int batchSize,
|
||||
Duration leaseDuration) {
|
||||
this(repository, publisher, envelopeFactory, batchSize, leaseDuration, defaultOwner());
|
||||
}
|
||||
|
||||
/**
|
||||
* A relay with an explicit owner identity.
|
||||
*
|
||||
* <p>The owner is half of the fencing predicate. It defaults to something stable per process
|
||||
* rather than per object, because two relays in one JVM sharing an owner would each be able to
|
||||
* overwrite the other's terminal writes — the exact failure the token exists to prevent.
|
||||
*/
|
||||
public OutboxRelay(
|
||||
OutboxRepository repository,
|
||||
MessagePublisher publisher,
|
||||
OutboxEnvelopeFactory envelopeFactory,
|
||||
int batchSize,
|
||||
Duration leaseDuration,
|
||||
String owner) {
|
||||
this(
|
||||
repository,
|
||||
publisher,
|
||||
envelopeFactory,
|
||||
batchSize,
|
||||
leaseDuration,
|
||||
owner,
|
||||
OutboxRetryScheduler.standard());
|
||||
}
|
||||
|
||||
/**
|
||||
* A relay with an explicit retry policy.
|
||||
*
|
||||
* <p>The scheduler was built by the auto-configuration and handed to nobody: the relay claimed
|
||||
* every ambiguous row on the next pass regardless of its attempt count, so the configured budget
|
||||
* of ten attempts was a number nothing read.
|
||||
*/
|
||||
public OutboxRelay(
|
||||
OutboxRepository repository,
|
||||
MessagePublisher publisher,
|
||||
OutboxEnvelopeFactory envelopeFactory,
|
||||
int batchSize,
|
||||
Duration leaseDuration,
|
||||
String owner,
|
||||
OutboxRetryScheduler scheduler) {
|
||||
this.owner = Objects.requireNonNull(owner, "owner must not be null");
|
||||
this.scheduler = Objects.requireNonNull(scheduler, "scheduler must not be null");
|
||||
this.repository = Objects.requireNonNull(repository, "repository must not be null");
|
||||
this.publisher = Objects.requireNonNull(publisher, "publisher must not be null");
|
||||
this.envelopeFactory =
|
||||
@@ -66,32 +123,46 @@ public final class OutboxRelay {
|
||||
*/
|
||||
public OutboxRelayReport runOnce(Instant now) {
|
||||
Objects.requireNonNull(now, "now must not be null");
|
||||
List<OutboxRecord> batch = repository.leaseBatch(batchSize, leaseDuration, now);
|
||||
// claimBatch, not leaseBatch: the claim carries an owner and a fencing token, and every write
|
||||
// below is conditioned on them. Without that, a relay that stalled past its lease could record
|
||||
// AMBIGUOUS over the PUBLISHED another relay had already written, making a confirmed message
|
||||
// claimable again.
|
||||
List<OutboxLease> batch =
|
||||
repository.claimBatch(owner, batchSize, leaseDuration, now, scheduler.maxAttempts());
|
||||
|
||||
int published = 0;
|
||||
int ambiguous = 0;
|
||||
int failed = 0;
|
||||
int stale = 0;
|
||||
|
||||
for (OutboxRecord record : batch) {
|
||||
for (OutboxLease lease : batch) {
|
||||
OutboxRecord record = lease.record();
|
||||
PublishResult result = publish(record);
|
||||
switch (result.completion()) {
|
||||
case CONFIRMED -> {
|
||||
repository.markPublished(record.messageId(), now);
|
||||
published++;
|
||||
if (repository.markPublished(lease, now) == OutboxTransitionResult.APPLIED) {
|
||||
published++;
|
||||
} else {
|
||||
stale++;
|
||||
}
|
||||
}
|
||||
case AMBIGUOUS -> {
|
||||
repository.markAmbiguous(
|
||||
record.messageId(),
|
||||
result.failure().map(failure -> failure.code()).orElse("AMBIGUOUS"),
|
||||
now);
|
||||
ambiguous++;
|
||||
if (repository.markAmbiguous(
|
||||
lease, result.failure().map(failure -> failure.code()).orElse("AMBIGUOUS"), now)
|
||||
== OutboxTransitionResult.APPLIED) {
|
||||
ambiguous++;
|
||||
} else {
|
||||
stale++;
|
||||
}
|
||||
}
|
||||
case REJECTED -> {
|
||||
repository.markFailed(
|
||||
record.messageId(),
|
||||
result.failure().map(failure -> failure.code()).orElse("REJECTED"),
|
||||
now);
|
||||
failed++;
|
||||
if (repository.markFailed(
|
||||
lease, result.failure().map(failure -> failure.code()).orElse("REJECTED"), now)
|
||||
== OutboxTransitionResult.APPLIED) {
|
||||
failed++;
|
||||
} else {
|
||||
stale++;
|
||||
}
|
||||
}
|
||||
// Unreachable while PublishCompletion has exactly these constants. Present so that a new
|
||||
// completion fails loudly rather than leaving the row IN_FLIGHT with its lease ticking —
|
||||
@@ -100,7 +171,7 @@ public final class OutboxRelay {
|
||||
throw new IllegalStateException("unhandled publish completion: " + result.completion());
|
||||
}
|
||||
}
|
||||
return new OutboxRelayReport(batch.size(), published, ambiguous, failed);
|
||||
return new OutboxRelayReport(batch.size(), published, ambiguous, failed, stale);
|
||||
}
|
||||
|
||||
private PublishResult publish(OutboxRecord record) {
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
package dev.caskeleton.messaging.outbox;
|
||||
|
||||
/**
|
||||
* What one relay pass did.
|
||||
*
|
||||
* <p>Ambiguous rows are counted separately from failures because they mean something operationally
|
||||
* different: a rising ambiguous count is a broker confirmation problem, while a rising failed count
|
||||
* is a contract or topology problem.
|
||||
*
|
||||
* <p>{@code staleLeases} is the count of terminal writes this pass could not apply because another
|
||||
* relay had already taken the row. It is not an error and it is not a success: it means this worker
|
||||
* published something whose outcome another worker has already recorded, so a non-zero value is the
|
||||
* visible form of a duplicate publication.
|
||||
*
|
||||
* @param leased how many rows were claimed
|
||||
* @param published how many were confirmed
|
||||
* @param ambiguous how many had an unknown outcome and stay retryable
|
||||
* @param failed how many were definitively rejected
|
||||
* @param staleLeases how many terminal writes were refused because the lease had been superseded
|
||||
*/
|
||||
public record OutboxRelayReport(
|
||||
int leased, int published, int ambiguous, int failed, int staleLeases) {
|
||||
|
||||
public OutboxRelayReport {
|
||||
if (leased < 0 || published < 0 || ambiguous < 0 || failed < 0 || staleLeases < 0) {
|
||||
throw new IllegalArgumentException("relay counters must not be negative");
|
||||
}
|
||||
}
|
||||
|
||||
/** A report from a pass that observed no stale leases. */
|
||||
public OutboxRelayReport(int leased, int published, int ambiguous, int failed) {
|
||||
this(leased, published, ambiguous, failed, 0);
|
||||
}
|
||||
}
|
||||
+10
@@ -58,6 +58,16 @@ public final class OutboxRetryScheduler {
|
||||
return now.plus(backoff(consecutiveEmptyOrFailedPasses));
|
||||
}
|
||||
|
||||
/** The platform's default policy, for a relay whose caller did not supply one. */
|
||||
public static OutboxRetryScheduler standard() {
|
||||
return new OutboxRetryScheduler(OutboxProperties.defaults(), java.time.Duration.ofMinutes(5));
|
||||
}
|
||||
|
||||
/** The attempt budget a row may spend before it is parked. */
|
||||
public int maxAttempts() {
|
||||
return maxAttempts;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the delay before the next pass.
|
||||
*
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
-- Lease owner and fencing token.
|
||||
--
|
||||
-- V1 recorded only lease_expires_at, so a claim said when it would end and nothing about who held
|
||||
-- it. The relay's terminal writes matched on message_id alone, which means a worker that stalled
|
||||
-- past its lease could still write:
|
||||
--
|
||||
-- relay A claims the row and calls the broker
|
||||
-- the lease expires; relay B reclaims it, publishes, and records PUBLISHED
|
||||
-- relay A finally times out and records AMBIGUOUS over the top
|
||||
--
|
||||
-- The row is now claimable again and the message is published a second time. Making the lease
|
||||
-- longer than the publish timeout lowers the odds; it does not turn a GC pause, a scheduler stall
|
||||
-- or a slow broker into a data constraint. A token does: every terminal write names the
|
||||
-- acquisition it belongs to, and a superseded worker's write matches nothing.
|
||||
ALTER TABLE messaging_outbox
|
||||
ADD COLUMN lease_owner VARCHAR(160),
|
||||
ADD COLUMN lease_token BIGINT NOT NULL DEFAULT 0,
|
||||
ADD COLUMN next_attempt_at TIMESTAMPTZ;
|
||||
|
||||
-- Backfill is unnecessary for correctness — the default is 0 and the first claim increments it —
|
||||
-- but the constraint states the invariant the code depends on.
|
||||
ALTER TABLE messaging_outbox
|
||||
ADD CONSTRAINT ck_messaging_outbox_lease_token CHECK (lease_token >= 0);
|
||||
|
||||
-- EXHAUSTED is a new terminal state: the attempt budget ran out without any confirmation, which is
|
||||
-- not the same as the broker rejecting the message. V1's CHECK listed five states, so writing the
|
||||
-- sixth failed at the constraint rather than at review.
|
||||
ALTER TABLE messaging_outbox
|
||||
DROP CONSTRAINT ck_messaging_outbox_status;
|
||||
|
||||
ALTER TABLE messaging_outbox
|
||||
ADD CONSTRAINT ck_messaging_outbox_status
|
||||
CHECK (status IN ('PENDING', 'IN_FLIGHT', 'PUBLISHED', 'AMBIGUOUS', 'FAILED', 'EXHAUSTED'));
|
||||
|
||||
-- The claim predicate now filters on next_attempt_at as well, so the partial index has to carry it
|
||||
-- or every poll degrades into a scan of the backlog once rows start backing off.
|
||||
CREATE INDEX ix_messaging_outbox_next_attempt
|
||||
ON messaging_outbox (next_attempt_at, created_at)
|
||||
WHERE status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT');
|
||||
+37
@@ -0,0 +1,37 @@
|
||||
-- The destructive-operation journal.
|
||||
--
|
||||
-- Keyed on (approval_ticket, plan_digest) rather than the ticket alone: the ticket alone cannot
|
||||
-- distinguish "this approval already ran" from "this approval is being presented for a different
|
||||
-- plan", and those need opposite answers. The primary key is what makes two replicas racing for the
|
||||
-- same approval resolve to one winner rather than two executions.
|
||||
--
|
||||
-- lease_owner/lease_token/lease_expires_at are the same fencing pattern the outbox uses: a runtime
|
||||
-- that stalls past its lease loses it, and its later writes are refused because its token is stale.
|
||||
-- items_completed is the resume point — without it a retry after a crash republishes everything the
|
||||
-- first attempt already moved.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS messaging_admin_operation (
|
||||
approval_ticket VARCHAR(200) NOT NULL,
|
||||
plan_digest CHAR(64) NOT NULL,
|
||||
operation_id VARCHAR(200) NOT NULL,
|
||||
state VARCHAR(20) NOT NULL,
|
||||
items_completed INTEGER NOT NULL DEFAULT 0,
|
||||
lease_owner VARCHAR(200) NOT NULL,
|
||||
lease_token BIGINT NOT NULL DEFAULT 1,
|
||||
lease_expires_at TIMESTAMPTZ NOT NULL,
|
||||
started_at TIMESTAMPTZ NOT NULL,
|
||||
finished_at TIMESTAMPTZ,
|
||||
failure_code VARCHAR(200),
|
||||
CONSTRAINT pk_messaging_admin_operation PRIMARY KEY (approval_ticket, plan_digest),
|
||||
CONSTRAINT ck_messaging_admin_operation_state
|
||||
CHECK (state IN ('STARTED', 'COMPLETED', 'FAILED')),
|
||||
CONSTRAINT ck_messaging_admin_operation_items CHECK (items_completed >= 0),
|
||||
CONSTRAINT ck_messaging_admin_operation_token CHECK (lease_token >= 1),
|
||||
CONSTRAINT ck_messaging_admin_operation_digest CHECK (plan_digest ~ '^[0-9a-f]{64}$')
|
||||
);
|
||||
|
||||
-- Answers "which operations are in flight right now", the question an operator asks when a redrive
|
||||
-- appears stuck and they are deciding whether to take it over.
|
||||
CREATE INDEX IF NOT EXISTS ix_messaging_admin_operation_live
|
||||
ON messaging_admin_operation (lease_expires_at)
|
||||
WHERE state = 'STARTED';
|
||||
+206
@@ -0,0 +1,206 @@
|
||||
package dev.caskeleton.messaging.outbox;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatCode;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
import dev.caskeleton.messaging.admin.AdminOperationLease;
|
||||
import dev.caskeleton.messaging.admin.AdminOperationState;
|
||||
import dev.caskeleton.messaging.admin.PlanDigest;
|
||||
import dev.caskeleton.messaging.api.error.MessageAuthorizationException;
|
||||
import dev.caskeleton.messaging.testkit.DockerAvailability;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.sql.Connection;
|
||||
import java.sql.SQLException;
|
||||
import java.sql.Statement;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import javax.sql.DataSource;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.condition.EnabledIf;
|
||||
import org.postgresql.ds.PGSimpleDataSource;
|
||||
import org.testcontainers.junit.jupiter.Container;
|
||||
import org.testcontainers.junit.jupiter.Testcontainers;
|
||||
import org.testcontainers.postgresql.PostgreSQLContainer;
|
||||
|
||||
/**
|
||||
* Certifies the durable journal against a real PostgreSQL.
|
||||
*
|
||||
* <p>The properties that matter here belong to the database, not to Java. "Exactly one replica
|
||||
* executes an approval" is a primary-key conflict; "a stalled runtime cannot overwrite its
|
||||
* successor" is a conditional update on a fencing token. An in-memory double can be written to
|
||||
* agree with both without either being true of the deployed schema, which is exactly how the map
|
||||
* this replaced looked correct for as long as nobody restarted a replica.
|
||||
*/
|
||||
@Testcontainers
|
||||
@EnabledIf("dockerAvailable")
|
||||
class AdminOperationJournalPostgresIT {
|
||||
|
||||
private static final Instant NOW = Instant.parse("2026-08-10T09:15:00Z");
|
||||
private static final Duration LEASE = Duration.ofMinutes(5);
|
||||
private static final String TICKET = "CHG-1042";
|
||||
private static final PlanDigest DIGEST = PlanDigest.ofCanonical("REDRIVE|orders.v1.dlq|50");
|
||||
|
||||
@Container
|
||||
private static final PostgreSQLContainer POSTGRES =
|
||||
new PostgreSQLContainer("postgres:16-alpine")
|
||||
.withDatabaseName("messaging")
|
||||
.withUsername("messaging")
|
||||
.withPassword("messaging");
|
||||
|
||||
private JdbcAdminOperationJournal journal;
|
||||
|
||||
static boolean dockerAvailable() {
|
||||
return DockerAvailability.isAvailable();
|
||||
}
|
||||
|
||||
@BeforeEach
|
||||
void migrate() throws SQLException, IOException {
|
||||
DataSource dataSource = dataSource();
|
||||
try (Connection connection = dataSource.getConnection();
|
||||
Statement statement = connection.createStatement()) {
|
||||
statement.execute("DROP TABLE IF EXISTS messaging_admin_operation");
|
||||
statement.execute(migration("V3__messaging_admin_operation_journal.sql"));
|
||||
}
|
||||
journal = new JdbcAdminOperationJournal(dataSource);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("the primary key, not application code, decides which replica wins")
|
||||
void thePrimaryKeyDecidesWhichReplicaWins() {
|
||||
journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
|
||||
|
||||
assertThatThrownBy(
|
||||
() -> journal.begin(TICKET, DIGEST, "op-2", "replica-b", LEASE, NOW.plusSeconds(1)))
|
||||
.isInstanceOf(MessageAuthorizationException.class)
|
||||
.hasMessageContaining("replica-a");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a completed operation survives the restart that forgot the map")
|
||||
void aCompletedOperationSurvivesARestart() {
|
||||
AdminOperationLease lease = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
|
||||
journal.complete(lease, 50, NOW.plusSeconds(30));
|
||||
|
||||
// A fresh journal object over the same database is what a restarted replica sees.
|
||||
JdbcAdminOperationJournal afterRestart = new JdbcAdminOperationJournal(dataSource());
|
||||
|
||||
assertThatThrownBy(
|
||||
() ->
|
||||
afterRestart.begin(TICKET, DIGEST, "op-3", "replica-a", LEASE, NOW.plusSeconds(60)))
|
||||
.isInstanceOf(MessageAuthorizationException.class)
|
||||
.hasMessageContaining("already executed");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("an expired lease is taken over at its checkpoint, under a higher token")
|
||||
void anExpiredLeaseIsTakenOverAtItsCheckpoint() {
|
||||
AdminOperationLease first = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
|
||||
journal.checkpoint(first, 30, LEASE, NOW.plusSeconds(10));
|
||||
|
||||
AdminOperationLease resumed =
|
||||
journal.begin(TICKET, DIGEST, "op-1", "replica-b", LEASE, NOW.plus(Duration.ofMinutes(10)));
|
||||
|
||||
assertThat(resumed.resumeFrom()).isEqualTo(30);
|
||||
assertThat(resumed.leaseToken()).isEqualTo(first.leaseToken() + 1);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a live lease is not taken over")
|
||||
void aLiveLeaseIsNotTakenOver() {
|
||||
journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
|
||||
|
||||
assertThatThrownBy(
|
||||
() ->
|
||||
journal.begin(
|
||||
TICKET, DIGEST, "op-1", "replica-b", LEASE, NOW.plus(Duration.ofMinutes(1))))
|
||||
.isInstanceOf(MessageAuthorizationException.class)
|
||||
.hasMessageContaining("is being executed by");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a superseded runtime cannot write over the one that took over")
|
||||
void aSupersededRuntimeCannotWriteOverItsSuccessor() {
|
||||
AdminOperationLease stale = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
|
||||
journal.checkpoint(stale, 30, LEASE, NOW.plusSeconds(10));
|
||||
AdminOperationLease successor =
|
||||
journal.begin(TICKET, DIGEST, "op-1", "replica-b", LEASE, NOW.plus(Duration.ofMinutes(10)));
|
||||
journal.checkpoint(successor, 45, LEASE, NOW.plus(Duration.ofMinutes(11)));
|
||||
|
||||
assertThatThrownBy(() -> journal.complete(stale, 30, NOW.plus(Duration.ofMinutes(12))))
|
||||
.isInstanceOf(MessageAuthorizationException.class)
|
||||
.hasMessageContaining("no longer holds the lease");
|
||||
|
||||
assertThat(journal.find(TICKET, DIGEST))
|
||||
.hasValueSatisfying(
|
||||
record -> {
|
||||
assertThat(record.itemsCompleted()).isEqualTo(45);
|
||||
assertThat(record.state()).isEqualTo(AdminOperationState.STARTED);
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a failed operation is resumable immediately and records why it stopped")
|
||||
void aFailedOperationIsResumableImmediately() {
|
||||
AdminOperationLease first = journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW);
|
||||
journal.fail(first, 12, "MESSAGE_BROKER_UNAVAILABLE", NOW.plusSeconds(6));
|
||||
|
||||
assertThat(journal.find(TICKET, DIGEST))
|
||||
.hasValueSatisfying(
|
||||
record -> {
|
||||
assertThat(record.state()).isEqualTo(AdminOperationState.FAILED);
|
||||
assertThat(record.failureCode()).contains("MESSAGE_BROKER_UNAVAILABLE");
|
||||
});
|
||||
|
||||
assertThatCode(
|
||||
() -> {
|
||||
AdminOperationLease resumed =
|
||||
journal.begin(TICKET, DIGEST, "op-1", "replica-a", LEASE, NOW.plusSeconds(7));
|
||||
assertThat(resumed.resumeFrom()).isEqualTo(12);
|
||||
})
|
||||
.doesNotThrowAnyException();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("the schema refuses a digest that is not a SHA-256")
|
||||
void theSchemaRefusesADigestThatIsNotASha256() throws SQLException {
|
||||
try (Connection connection = dataSource().getConnection();
|
||||
Statement statement = connection.createStatement()) {
|
||||
assertThatThrownBy(
|
||||
() ->
|
||||
statement.execute(
|
||||
"""
|
||||
INSERT INTO messaging_admin_operation (
|
||||
approval_ticket, plan_digest, operation_id, state,
|
||||
lease_owner, lease_expires_at, started_at)
|
||||
VALUES ('CHG-9', 'not-a-digest', 'op', 'STARTED', 'r', now(), now())
|
||||
"""))
|
||||
.as("a malformed digest in the journal makes the uniqueness key meaningless")
|
||||
.isInstanceOf(SQLException.class);
|
||||
}
|
||||
}
|
||||
|
||||
private static DataSource dataSource() {
|
||||
PGSimpleDataSource source = new PGSimpleDataSource();
|
||||
source.setUrl(POSTGRES.getJdbcUrl());
|
||||
source.setUser(POSTGRES.getUsername());
|
||||
source.setPassword(POSTGRES.getPassword());
|
||||
return source;
|
||||
}
|
||||
|
||||
private static String migration(String fileName) throws IOException {
|
||||
try (InputStream stream =
|
||||
AdminOperationJournalPostgresIT.class
|
||||
.getClassLoader()
|
||||
.getResourceAsStream("db/migration/messaging/" + fileName)) {
|
||||
if (stream == null) {
|
||||
throw new IOException("migration " + fileName + " is missing from the classpath");
|
||||
}
|
||||
return new String(stream.readAllBytes(), StandardCharsets.UTF_8);
|
||||
}
|
||||
}
|
||||
}
|
||||
+120
@@ -0,0 +1,120 @@
|
||||
package dev.caskeleton.messaging.outbox;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
import dev.caskeleton.messaging.api.ContentType;
|
||||
import dev.caskeleton.messaging.api.MessageId;
|
||||
import dev.caskeleton.messaging.api.MessageType;
|
||||
import dev.caskeleton.messaging.api.SchemaVersion;
|
||||
import dev.caskeleton.messaging.api.destination.DestinationName;
|
||||
import dev.caskeleton.messaging.api.error.MessagingConfigurationException;
|
||||
import dev.caskeleton.messaging.reliability.OutboxRecord;
|
||||
import dev.caskeleton.messaging.reliability.OutboxStatus;
|
||||
import java.time.Instant;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import javax.sql.DataSource;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.transaction.support.TransactionSynchronizationManager;
|
||||
|
||||
/**
|
||||
* What {@code append(record)} does when there is no transaction to join.
|
||||
*
|
||||
* <p>The port says the append runs inside the caller's business transaction. The implementation
|
||||
* called {@code dataSource.getConnection()}, which auto-commits on its own connection — so a
|
||||
* business transaction that rolled back afterwards left the outbox row behind and the relay
|
||||
* published an event for a change that never happened.
|
||||
*
|
||||
* <p>These are hermetic: the refusal happens before any connection is requested, which is also the
|
||||
* property under test. The mock would fail the test by being touched at all.
|
||||
*/
|
||||
class JdbcOutboxTransactionRequirementTest {
|
||||
|
||||
/** A data source that fails the test by being used: the refusal must precede any connection. */
|
||||
private final DataSource dataSource = new RefusingDataSource();
|
||||
|
||||
private final JdbcOutboxRepository repository = new JdbcOutboxRepository(dataSource);
|
||||
|
||||
@AfterEach
|
||||
void clearTransactionState() {
|
||||
TransactionSynchronizationManager.clear();
|
||||
if (TransactionSynchronizationManager.hasResource(dataSource)) {
|
||||
TransactionSynchronizationManager.unbindResource(dataSource);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("appending outside a transaction is refused before a connection is opened")
|
||||
void appendingOutsideATransactionIsRefused() {
|
||||
assertThatThrownBy(() -> repository.append(record()))
|
||||
.isInstanceOf(MessagingConfigurationException.class)
|
||||
.hasMessageContaining("requires an active transaction");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a read-only transaction is refused")
|
||||
void aReadOnlyTransactionIsRefused() {
|
||||
TransactionSynchronizationManager.setActualTransactionActive(true);
|
||||
TransactionSynchronizationManager.setCurrentTransactionReadOnly(true);
|
||||
|
||||
assertThatThrownBy(() -> repository.append(record()))
|
||||
.isInstanceOf(MessagingConfigurationException.class)
|
||||
.hasMessageContaining("read-only");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a transaction bound to another DataSource is refused")
|
||||
void aTransactionOnAnotherDataSourceIsRefused() {
|
||||
TransactionSynchronizationManager.setActualTransactionActive(true);
|
||||
|
||||
assertThatThrownBy(() -> repository.append(record()))
|
||||
.as("two DataSources means two commits, which is the atomicity this exists to provide")
|
||||
.isInstanceOf(MessagingConfigurationException.class)
|
||||
.hasMessageContaining("bound to this DataSource");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("the refusal names a code an operator can search for")
|
||||
void theRefusalNamesACode() {
|
||||
assertThatThrownBy(() -> repository.append(record()))
|
||||
.isInstanceOfSatisfying(
|
||||
MessagingConfigurationException.class,
|
||||
failure ->
|
||||
assertThat(failure.failure().code()).isEqualTo("OUTBOX_TRANSACTION_REQUIRED"));
|
||||
}
|
||||
|
||||
private static OutboxRecord record() {
|
||||
return new OutboxRecord(
|
||||
MessageId.newId(),
|
||||
new DestinationName("orders.events"),
|
||||
new MessageType("order.placed"),
|
||||
new SchemaVersion(1),
|
||||
ContentType.JSON,
|
||||
"{}".getBytes(java.nio.charset.StandardCharsets.UTF_8),
|
||||
Map.of(),
|
||||
Instant.parse("2026-08-14T00:00:00Z"),
|
||||
OutboxStatus.PENDING,
|
||||
0,
|
||||
Optional.empty(),
|
||||
Optional.empty());
|
||||
}
|
||||
|
||||
/** Throws if anything asks it for a connection. */
|
||||
private static final class RefusingDataSource
|
||||
extends org.springframework.jdbc.datasource.AbstractDataSource {
|
||||
|
||||
@Override
|
||||
public java.sql.Connection getConnection() {
|
||||
throw new AssertionError(
|
||||
"the refusal must happen before a connection is requested; opening one is the defect");
|
||||
}
|
||||
|
||||
@Override
|
||||
public java.sql.Connection getConnection(String username, String password) {
|
||||
return getConnection();
|
||||
}
|
||||
}
|
||||
}
|
||||
+61
@@ -40,6 +40,62 @@ class OutboxOperationsTest {
|
||||
throw new UnsupportedOperationException();
|
||||
}
|
||||
|
||||
@Override
|
||||
public java.util.List<dev.caskeleton.messaging.reliability.OutboxLease> claimBatch(
|
||||
String owner, int batchSize, Duration leaseDuration, Instant now) {
|
||||
return java.util.List.of();
|
||||
}
|
||||
|
||||
@Override
|
||||
public java.util.List<dev.caskeleton.messaging.reliability.OutboxLease> claimBatch(
|
||||
String owner, int batchSize, Duration leaseDuration, Instant now, int maxAttempts) {
|
||||
return java.util.List.of();
|
||||
}
|
||||
|
||||
@Override
|
||||
public dev.caskeleton.messaging.reliability.OutboxTransitionResult markAmbiguous(
|
||||
dev.caskeleton.messaging.reliability.OutboxLease lease,
|
||||
String failureCode,
|
||||
Instant now,
|
||||
Instant nextAttemptAt) {
|
||||
return markAmbiguous(lease, failureCode, now);
|
||||
}
|
||||
|
||||
@Override
|
||||
public dev.caskeleton.messaging.reliability.OutboxTransitionResult markExhausted(
|
||||
dev.caskeleton.messaging.reliability.OutboxLease lease, String reason, Instant now) {
|
||||
markFailed(lease.messageId(), reason, now);
|
||||
return dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED;
|
||||
}
|
||||
|
||||
@Override
|
||||
public dev.caskeleton.messaging.reliability.OutboxTransitionResult markPublished(
|
||||
dev.caskeleton.messaging.reliability.OutboxLease lease, Instant now) {
|
||||
markPublished(lease.messageId(), now);
|
||||
return dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED;
|
||||
}
|
||||
|
||||
@Override
|
||||
public dev.caskeleton.messaging.reliability.OutboxTransitionResult markAmbiguous(
|
||||
dev.caskeleton.messaging.reliability.OutboxLease lease, String failureCode, Instant now) {
|
||||
markAmbiguous(lease.messageId(), failureCode, now);
|
||||
return dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED;
|
||||
}
|
||||
|
||||
@Override
|
||||
public dev.caskeleton.messaging.reliability.OutboxTransitionResult markFailed(
|
||||
dev.caskeleton.messaging.reliability.OutboxLease lease, String failureCode, Instant now) {
|
||||
markFailed(lease.messageId(), failureCode, now);
|
||||
return dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED;
|
||||
}
|
||||
|
||||
@Override
|
||||
public dev.caskeleton.messaging.reliability.OutboxTransitionResult releaseLease(
|
||||
dev.caskeleton.messaging.reliability.OutboxLease lease) {
|
||||
releaseLease(lease.messageId());
|
||||
return dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void markPublished(MessageId messageId, Instant now) {
|
||||
throw new UnsupportedOperationException();
|
||||
@@ -65,6 +121,11 @@ class OutboxOperationsTest {
|
||||
throw new UnsupportedOperationException();
|
||||
}
|
||||
|
||||
@Override
|
||||
public int purgePublishedBefore(Instant publishedBefore, int limit) {
|
||||
return Math.min(purgePublishedBefore(publishedBefore), limit);
|
||||
}
|
||||
|
||||
@Override
|
||||
public int purgePublishedBefore(Instant publishedBefore) {
|
||||
cutoffs.add(publishedBefore);
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user