review-followup Wave 2: 정합성 수정 통합과 검증, 현황표·재개 프롬프트

13개 작업 묶음(P1–P13)의 W2 커밋을 합치고 전체 게이트로 검증했다.

- 묶음별 정합성 수정(notification dispatch·recovery, object storage staged
  publication, fileserver 복구·정리, gRPC 플랫폼, Redis 분류·lease TTL, JPA
  outbox claim 순서·durable operation HTTP, web 요청 id·category 필수, graphql
  에러 mapper, Mongo profile 바인딩, build 소스 보존 게이트·compose smoke info 확인)
- 통합 때 드러난 3건 수정: outbox claim 테스트 fixture의 가짜 envelope hash,
  sample scan이 새 operation controller를 등록하던 문제, SpotBugs aux classpath
  파일이 truncate되지 않아 로컬에서 ".jar.jar"로 실패하던 문제
- 현황표: 묶음 결과 반영, 소유자 결정 21–27, 후속 발견 결함 47행, 요약 갱신
- docs/capability-status.md 「제거된 코드」에 85행 추가
- 묶음 결과 파일·에이전트 지시·스크립트를 docs로 옮기고 desktop 재개 프롬프트 추가

검증: check, ci verifyDependencyLocks, mainCheck, architecture/public path/env
keys/config contract, bootstrap·sample integration/system, build-logic·build-tools
test, optional-platforms ci, qualificationBootJar 통과. integrationCheck는 이번
작업 전에도 실패하던 lane(Redis topology 모드 인자, httpclient failure
injection, AWS 보호 lane)만 실패.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
DongHyeonka
2026-09-28 14:46:30 +09:00
co-authored by Claude Opus 5.5
parent 4d4cc35e6c
commit b5e3c03777
39 changed files with 1024 additions and 226 deletions
@@ -166,6 +166,12 @@ app:
`kafka`뿐이다. 위 예시가
`production: false`인 것은 이 페이지가 그대로 실행되는 fixture이기 때문이며, 실 배포는 셋 다 참이다.
Kafka producer는 이 항목 하나로 만든다. `bootstrap-servers`, `acks`, `enable-idempotence`,
`max-in-flight-requests-per-connection`, `delivery-timeout`이 그대로 client 설정이 된다.
`spring.kafka.bootstrap-servers`는 읽지 않는다. type이 `kafka`인 항목이 정확히 하나가 아니면 기동에
실패한다. `send()`가 metadata를 기다리며 호출 스레드를 막는 시간(`max.block.ms`)은 기본 publish
timeout(5s)이다.
## 보안
```yaml
+17 -1
View File
@@ -21,7 +21,7 @@ Flyway version `1000..1999`를 예약해서 애플리케이션의 일반적인
classpath:db/migration/messaging
```
현재 순서는 `V1001..V1004`(outbox/admin)와 `V1101`(inbox)이다. starter 테스트가 두 JDBC
현재 순서는 `V1001..V1005`(outbox/admin)와 `V1101`(inbox)이다. starter 테스트가 두 JDBC
모듈을 실제 combined classpath로 올려 version 중복과 예약 대역 이탈을 검사한다. 개별 DDL의
의미와 전환 순서는 [migration-guide.md](migration-guide.md)를 따른다.
@@ -46,6 +46,11 @@ leaseBatch(100, 30s) -- lease로 다중 relay 인스턴스 안전
→ REJECTED → markFailed
```
row 하나의 실패는 그 row에서 끝난다. envelope로 만들 수 없는 row(예약 header `msg.*`를 담은 row
등)는 `FAILED`가 되고, publisher가 예외를 던지면 결과를 모르므로 `AMBIGUOUS`가 된다. 같은 batch의
나머지 row는 그대로 publish된다. pass 전체가 실패하면(DB 연결 등) WARN 로그를 남기고
`messaging.outbox.relay.failed.passes` 카운터를 올린다.
### 핵심 규칙: ambiguous는 같은 messageId로 재시도
새 id를 발급하면 "전달됐을 수도 있는 메시지"가 "확실히 두 번째인 메시지"가 되어
@@ -61,8 +66,16 @@ status IN ('PENDING','AMBIGUOUS','IN_FLIGHT')
AND (lease_expires_at IS NULL OR lease_expires_at <= now)
AND next_attempt_at <= now
AND attempts < maxAttempts
AND (ordering_key IS NULL
OR 같은 destination·ordering_key의 더 오래된 미완료 row가 없음)
```
`ordering_key`가 있는 row는 같은 key의 더 오래된 row가 `PENDING`·`AMBIGUOUS`·`IN_FLIGHT`인 동안
claim되지 않는다. 이 조건이 없으면 두 relay가 같은 key의 두 row를 나눠 가져가거나, AMBIGUOUS로
재시도되는 row보다 뒤의 row가 먼저 나간다(`OutboxPostgresIT#aNewerRowWaitsWhileAnOlderRowWithTheSameOrderingKeyIsUnfinished`).
`V1005`의 partial index가 이 확인을 받친다. 더 오래된 row가 `FAILED`·`EXHAUSTED`가 되면 뒤의 row는
진행한다.
`IN_FLIGHT`가 목록에 있는 것이 핵심이다. relay가 publish 도중 죽으면 row는 `IN_FLIGHT`로 남는데,
이를 제외하면 그 메시지는 **영원히** 발행되지 않는다 — outbox가 막으려던 바로 그 실패다. 대신
lease가 만료됐을 때만 회수하므로, 살아 있는 relay가 들고 있는 row는 회수되지 않는다.
@@ -107,6 +120,9 @@ message_id만으로 중복 제거하면 같은 event를 소비하는 두 번째
broker의 최대 redelivery window보다 **길어야** 한다.
row를 먼저 지우면 늦게 도착한 redelivery가 두 번 처리된다.
outbox·inbox cleanup job은 한 번에 최대 1,000행씩 지우고, 지운 수가 1,000보다 적으면 멈춘다.
repository port에는 개수 제한 없는 삭제 메서드가 없다.
## CDC 대안
relay는 polling 하나뿐이다. Debezium Event Router 매핑(`DebeziumOutboxEventRouter` 등)은 조립하는
+9
View File
@@ -49,6 +49,15 @@ full jitter는 `[0, delay]` 균등 분포다. jitter가 없으면 같은 초에
pause-and-seek는 메시지가 로그의 자기 자리를 떠나지 않는다. partition을 멈추고, 기다리고,
같은 offset으로 seek해 재전달한다. 뒤의 메시지도 함께 기다리며 이것이 의도된 동작이다.
`KafkaConsumerRegistrar`가 handler 실패(예외, settle 없이 끝남, timeout)를 다루는 방식:
- 재전달은 destination `retry`의 backoff 뒤에 한다. 최소 1초. 0초 재시도는 없다.
- 같은 offset의 실패 횟수가 `retry.maxAttempts`에 닿으면 `KafkaQuarantinePublisher`에 쓰고,
쓰기가 확인되면 offset을 넘긴다. quarantine이 없으면 backoff로 계속 재시도한다(메시지를 버리지
않는다). 그래서 `production` destination은 quarantine 없이 만들 수 없다(`KAFKA_QUARANTINE_REQUIRED`).
- handler timeout이 지나도 handler가 끝날 때까지 partition slot을 잡고 있는다. `CompletableFuture`를
cancel해도 handler는 멈추지 않으므로, slot을 먼저 풀면 같은 offset이 두 번 동시에 돈다.
## RabbitMQ: delayed retry queue
core broker에 per-message delay가 없으므로 **TTL + DLX**로 구현한다.
@@ -151,7 +151,7 @@ class InboxPostgresIT {
MessageId purged = MessageId.newId();
inTransaction(() -> repository.reserve(purged, "order-projection", NOW));
assertThat(repository.purgeProcessedBefore(NOW.plusSeconds(1))).isEqualTo(1);
assertThat(repository.purgeProcessedBefore(NOW.plusSeconds(1), 100)).isEqualTo(1);
}
/** Reserves and writes inside one Spring transaction, exactly as an idempotent consumer must. */
@@ -53,10 +53,10 @@ public final class InboxCleanupJob {
int removed = 0;
for (int batch = 0; batch < maxBatches; batch++) {
int deleted = inbox.purgeProcessedBefore(cutoff);
int deleted = inbox.purgeProcessedBefore(cutoff, DEFAULT_BATCH_SIZE);
removed += deleted;
// A short batch means the backlog is drained; continuing would just re-scan an empty range.
if (deleted == 0) {
if (deleted < DEFAULT_BATCH_SIZE) {
break;
}
}
@@ -167,18 +167,4 @@ public final class JdbcInboxRepository implements InboxRepository {
"INBOX_PURGE_FAILED", "a bounded inbox purge failed", exception);
}
}
@Override
public int purgeProcessedBefore(Instant processedBefore) {
Objects.requireNonNull(processedBefore, "processedBefore must not be null");
try (Connection connection = dataSource.getConnection();
PreparedStatement statement =
connection.prepareStatement("DELETE FROM messaging_inbox WHERE processed_at < ?")) {
statement.setTimestamp(1, Timestamp.from(processedBefore));
return statement.executeUpdate();
} catch (SQLException exception) {
throw new MessagingConfigurationException(
"INBOX_PURGE_FAILED", "could not purge the inbox", exception);
}
}
}
@@ -82,7 +82,7 @@ class IdempotentConsumerTest {
void retentionPruningRemovesOldRows() {
consumer.runOnce(MessageId.newId(), "order-projection", NOW, () -> "done");
assertThat(inbox.purgeProcessedBefore(NOW.plusSeconds(1))).isEqualTo(1);
assertThat(inbox.purgeProcessedBefore(NOW.plusSeconds(1), 100)).isEqualTo(1);
assertThat(inbox.size()).isZero();
}
}
@@ -104,12 +104,7 @@ final class InMemoryInbox implements InboxRepository {
@Override
public int purgeProcessedBefore(Instant processedBefore, int limit) {
return Math.min(purgeProcessedBefore(processedBefore), limit);
}
@Override
public int purgeProcessedBefore(Instant processedBefore) {
int removed = reserved.size();
int removed = Math.min(reserved.size(), limit);
reserved.clear();
return removed;
}
@@ -42,15 +42,13 @@ class InboxOperationsTest {
return reserved.contains(messageId.value() + "|" + consumerId);
}
@Override
public int purgeProcessedBefore(Instant processedBefore, int limit) {
return Math.min(purgeProcessedBefore(processedBefore), limit);
}
private final List<Integer> limits = new ArrayList<>();
@Override
public int purgeProcessedBefore(Instant processedBefore) {
public int purgeProcessedBefore(Instant processedBefore, int limit) {
cutoffs.add(processedBefore);
return pass < deletions.size() ? deletions.get(pass++) : 0;
limits.add(limit);
return Math.min(pass < deletions.size() ? deletions.get(pass++) : 0, limit);
}
}
@@ -91,14 +89,17 @@ class InboxOperationsTest {
}
@Test
void cleanupDeletesInBoundedBatches() {
void cleanupAsksForBoundedBatchesAndStopsAtTheFirstShortOne() {
// given
InMemoryInbox inbox = new InMemoryInbox(List.of(1000, 500));
int removed =
new InboxCleanupJob(inbox, policy(Duration.ofDays(7), Duration.ofDays(1)), 10).runOnce(NOW);
// when
new InboxCleanupJob(inbox, policy(Duration.ofDays(7), Duration.ofDays(1)), 10).runOnce(NOW);
assertThat(removed).isEqualTo(1500);
assertThat(inbox.cutoffs).hasSize(3);
// then
assertThat(inbox.limits)
.as("DEFAULT_BATCH_SIZE was declared and never passed to the repository")
.containsExactly(InboxCleanupJob.DEFAULT_BATCH_SIZE, InboxCleanupJob.DEFAULT_BATCH_SIZE);
}
@Test
@@ -66,6 +66,35 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
private final KafkaQuarantinePublisher quarantine;
private final java.time.InstantSource clock;
/** The shortest wait before a failed delivery is read again, whatever the policy computes. */
private static final Duration MIN_RETRY_DELAY = Duration.ofSeconds(1);
private static final dev.caskeleton.messaging.policy.BackoffCalculator BACKOFF =
new dev.caskeleton.messaging.policy.BackoffCalculator();
/** A delivery that failed, by where it sits in the log. */
private record FailedOffset(TopicPartition partition, long offset) {}
/**
* How many times each offset has failed without being settled.
*
* <p>Pause-and-seek re-reads the same record, so the envelope's own attempt header never moves;
* the count has to live here. Without it a poison message was retried with no delay and no limit,
* and its partition never advanced again.
*/
private final Map<FailedOffset, Integer> failures = new ConcurrentHashMap<>();
/**
* Partitions paused because a record could not be dispatched — the ceiling was reached, the
* runtime was draining, or the pool refused. Poll thread only.
*
* <p>They were paused and never resumed: an ordered partition handed two records by one poll ran
* the first and then stopped reading for good.
*/
private final Set<TopicPartition> saturated = new java.util.HashSet<>();
private final Map<TopicPartition, Long> committed = new ConcurrentHashMap<>();
/**
@@ -127,8 +156,38 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
GracefulShutdownCoordinator shutdown,
Duration pollTimeout,
KafkaQuarantinePublisher quarantine) {
this(
spec,
consumer,
handlerPool,
shutdown,
pollTimeout,
quarantine,
java.time.InstantSource.system());
}
/** A consumer runtime with the clock retry delays are measured on, for tests. */
KafkaConsumerRegistrar(
TransportConsumerSpec spec,
Consumer<byte[], byte[]> consumer,
Executor handlerPool,
GracefulShutdownCoordinator shutdown,
Duration pollTimeout,
KafkaQuarantinePublisher quarantine,
java.time.InstantSource clock) {
this.clock = Objects.requireNonNull(clock, "clock must not be null");
this.quarantine = Objects.requireNonNull(quarantine, "quarantine must not be null");
this.spec = Objects.requireNonNull(spec, "spec must not be null");
if (spec.profile().production() && quarantine == KafkaQuarantinePublisher.unavailable()) {
// Without somewhere to park a message that keeps failing, its partition retries it forever
// and nothing behind it is ever read.
throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException(
"KAFKA_QUARANTINE_REQUIRED",
"production destination "
+ spec.profile().name().value()
+ " needs a KafkaQuarantinePublisher; without one a message that exhausts its"
+ " attempts stops its partition for good");
}
this.consumer = Objects.requireNonNull(consumer, "consumer must not be null");
this.handlerPool = Objects.requireNonNull(handlerPool, "handlerPool must not be null");
this.shutdown = Objects.requireNonNull(shutdown, "shutdown must not be null");
@@ -169,6 +228,8 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
coordinator.forget(partition);
retries.forget(partition);
committed.remove(partition);
saturated.remove(partition);
failures.keySet().removeIf(failed -> failed.partition().equals(partition));
});
}
@@ -199,6 +260,7 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
syncEpochs();
applySettlements();
applyDueResumes(now);
resumeSaturated();
commitContiguous();
if (!shutdown.isAcceptingWork()) {
@@ -251,6 +313,7 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
// Only this partition pauses. Its earliest unsubmitted offset is where it resumes.
consumer.pause(Set.of(partition));
consumer.seek(partition, seekBackTo);
saturated.add(partition);
}
}
return dispatched;
@@ -283,7 +346,7 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
if (quarantine.quarantine(record, undecodable)) {
settlement.acknowledgeAfterQuarantine();
} else {
settlement.requeueAfterFailure();
settlement.requeueAfterFailure(record, undecodable);
}
coordinator.release(partition);
shutdown.endWork();
@@ -308,37 +371,36 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
handler = spec.sink().apply(delivery).toCompletableFuture();
handler.get(handlerTimeout.toMillis(), java.util.concurrent.TimeUnit.MILLISECONDS);
} catch (java.util.concurrent.TimeoutException timedOut) {
// A timed-out handler may still be applying side effects. Cancel it before releasing
// the partition permit; otherwise the same offset can be dispatched again while the
// first handler is still running.
// A timed-out handler may still be applying side effects, and cancelling the
// CompletableFuture does not stop it — it only completes the stage this worker holds,
// which released the permit and let the same offset run again beside the first. So
// the slot stays taken until the handler really finishes, and only then is a delivery
// it left unsettled requeued.
java.util.concurrent.CompletableFuture<Void> timedOutHandler = handler;
settlement.requeueAfterFailure();
if (timedOutHandler != null
&& !timedOutHandler.cancel(true)
&& !timedOutHandler.isDone()) {
releaseImmediately = false;
java.util.concurrent.CompletableFuture<Void> releaseAfterCompletion =
timedOutHandler.whenComplete(
(ignored, failure) -> {
coordinator.release(partition);
shutdown.endWork();
});
// Keep the continuation strongly referenced through this worker frame. The
// completion action owns the delayed permit/work-count release; its own result is
// intentionally not joined here.
Objects.requireNonNull(releaseAfterCompletion);
}
releaseImmediately = false;
java.util.concurrent.CompletableFuture<Void> releaseAfterCompletion =
timedOutHandler.whenComplete(
(ignored, failure) -> {
settlement.requeueAfterFailure(
record, failure == null ? timedOut : failure);
coordinator.release(partition);
shutdown.endWork();
});
// Keep the continuation strongly referenced through this worker frame. The
// completion action owns the delayed permit/work-count release; its own result is
// intentionally not joined here.
Objects.requireNonNull(releaseAfterCompletion);
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
if (handler != null) {
handler.cancel(true);
}
settlement.requeueAfterFailure();
settlement.requeueAfterFailure(record, interrupted);
} catch (java.util.concurrent.ExecutionException | RuntimeException failure) {
// A handler failure is retried, not parked. PARKED completed the offset without any
// quarantine or dead-letter write, so a transient database outage silently discarded
// every message it touched.
settlement.requeueAfterFailure();
settlement.requeueAfterFailure(record, failure);
} finally {
if (releaseImmediately) {
coordinator.release(partition);
@@ -365,7 +427,10 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
continue;
}
switch (command.kind()) {
case COMPLETE, PARKED -> offsets.completed(command.partition(), command.offset());
case COMPLETE, PARKED -> {
offsets.completed(command.partition(), command.offset());
failures.remove(new FailedOffset(command.partition(), command.offset()));
}
case PAUSE_AND_SEEK -> {
coordinator.pause(command.partition());
consumer.pause(Set.of(command.partition()));
@@ -391,6 +456,22 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
current.forEach(
partition -> epochs.computeIfAbsent(partition, key -> epochSequence.incrementAndGet()));
epochs.keySet().removeIf(partition -> !current.contains(partition));
saturated.retainAll(current);
}
/** Resumes a partition paused for lack of a slot once it has one and nothing else holds it. */
private void resumeSaturated() {
int ceiling = spec.profile().consumer().maxInFlightPerOrderingUnit();
saturated.removeIf(
partition -> {
if (coordinator.isPaused(partition)
|| retries.isPaused(partition)
|| coordinator.inFlight(partition) >= ceiling) {
return false;
}
consumer.resume(Set.of(partition));
return true;
});
}
private void applyDueResumes(Instant now) {
@@ -603,16 +684,36 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati
}
}
/** Used by the worker when the handler failed or timed out and never settled. */
private void requeueAfterFailure() {
/**
* Used by the worker when the handler failed or timed out and never settled.
*
* <p>Retried after the destination's backoff, never sooner than {@code MIN_RETRY_DELAY}; once
* the attempt budget is spent the record is written to quarantine and committed past. When
* there is no quarantine it keeps being retried — the offset may not advance past a message
* that exists nowhere else — which is why a production destination must have one.
*/
private void requeueAfterFailure(ConsumerRecord<byte[], byte[]> record, Throwable reason) {
if (terminal.get() != null) {
// The handler settled before it failed or before its late completion; that answer stands.
return;
}
FailedOffset key = new FailedOffset(partition, offset);
int attempt = failures.merge(key, 1, Integer::sum);
dev.caskeleton.messaging.policy.RetryPolicy retry = spec.profile().retry();
if (attempt >= retry.maxAttempts() && quarantine.quarantine(record, reason)) {
failures.remove(key);
acknowledgeAfterQuarantine();
return;
}
if (terminal.compareAndSet(null, "requeue")) {
enqueueRequeue(Duration.ZERO);
Duration backoff = BACKOFF.delayFor(retry, attempt);
enqueueRequeue(backoff.compareTo(MIN_RETRY_DELAY) < 0 ? MIN_RETRY_DELAY : backoff);
}
}
private dev.caskeleton.messaging.api.settlement.SettlementResult enqueueRequeue(
Duration delay) {
retries.pauseUntil(partition, offset, delay, Instant.now());
retries.pauseUntil(partition, offset, delay, clock.instant());
settlements.enqueue(
new KafkaSettlementCommand(
partition, offset, KafkaSettlementCommand.Kind.PAUSE_AND_SEEK, epoch));
@@ -24,8 +24,21 @@ public interface KafkaQuarantinePublisher {
*/
boolean quarantine(ConsumerRecord<byte[], byte[]> record, Throwable reason);
/** The default: nothing is configured, so nothing may be committed past. */
/**
* The default: nothing is configured, so nothing may be committed past.
*
* <p>One instance, so a consumer can tell it apart from a configured quarantine and refuse it on
* a production destination.
*/
static KafkaQuarantinePublisher unavailable() {
return (record, reason) -> false;
return Unavailable.INSTANCE;
}
/** Holds the single "not configured" quarantine. */
final class Unavailable {
private static final KafkaQuarantinePublisher INSTANCE = (record, reason) -> false;
private Unavailable() {}
}
}
@@ -4,6 +4,9 @@ import static org.assertj.core.api.Assertions.assertThat;
import dev.caskeleton.messaging.api.MessageId;
import dev.caskeleton.messaging.api.header.ReservedHeaders;
import dev.caskeleton.messaging.policy.OrderingImpact;
import dev.caskeleton.messaging.policy.RetryMode;
import dev.caskeleton.messaging.policy.RetryPolicy;
import dev.caskeleton.messaging.transport.GracefulShutdownCoordinator;
import dev.caskeleton.messaging.transport.TransportConsumerSpec;
import dev.caskeleton.messaging.transport.TransportDelivery;
@@ -154,51 +157,165 @@ class KafkaConsumerRegistrarTest {
}
@Test
void handlerTimeoutCancelsTheOutstandingHandlerBeforeThePartitionPermitIsReleased() {
CompletableFuture<Void> handler = new CompletableFuture<>();
dev.caskeleton.messaging.policy.DestinationProfile base = KafkaContractHarness.profile();
dev.caskeleton.messaging.policy.DestinationProfile fastTimeout =
new dev.caskeleton.messaging.policy.DestinationProfile(
base.name(),
base.broker(),
base.kind(),
base.physical(),
base.schema(),
base.deliveryGuarantee(),
base.orderingScope(),
base.externalSideEffectGuarantee(),
base.producer(),
new dev.caskeleton.messaging.policy.ConsumerPolicy(
base.consumer().group(),
base.consumer().concurrency(),
base.consumer().maxInFlightPerOrderingUnit(),
base.consumer().prefetch(),
Duration.ofMillis(5),
base.consumer().manualSettlement()),
base.retry(),
base.deadLetter(),
base.payload(),
base.tier(),
base.production(),
base.keyResolverConfigured(),
base.topologyAutoCreate());
void aTimedOutHandlerStillRunningIsNotDispatchedASecondTime() {
// given
Instant start = Instant.now();
CompletableFuture<Void> stillRunning = new CompletableFuture<>();
KafkaConsumerRegistrar registrar =
new KafkaConsumerRegistrar(
new TransportConsumerSpec(fastTimeout, delivery -> handler),
consumer,
handlers,
new GracefulShutdownCoordinator(Duration.ofSeconds(30)),
Duration.ofMillis(1));
registrar(
profile(Duration.ofMillis(5), RetryPolicy.none(), false),
delivery -> {
handled.add(delivery);
return stillRunning;
},
KafkaQuarantinePublisher.unavailable(),
() -> start);
assign();
addRecord(10);
registrar.pollOnce(start);
handlers.runAll();
// What the broker serves again if the consumer seeks back to the timed-out offset.
addRecord(10);
// when
registrar.pollOnce(start.plusSeconds(60));
handlers.runAll();
// then
assertThat(handled)
.as("cancelling a CompletableFuture does not stop the handler behind it")
.hasSize(1);
}
@Test
void aTimedOutHandlerIsRedeliveredOnceItFinishesWithoutSettling() {
// given
CompletableFuture<Void> stillRunning = new CompletableFuture<>();
KafkaConsumerRegistrar registrar =
registrar(
profile(Duration.ofMillis(5), RetryPolicy.none(), false),
delivery -> {
handled.add(delivery);
return stillRunning;
},
KafkaQuarantinePublisher.unavailable(),
() -> NOW);
assign();
addRecord(10);
registrar.pollOnce(NOW);
handlers.runAll();
stillRunning.complete(null);
addRecord(10);
// when
registrar.pollOnce(NOW.plusSeconds(60));
handlers.runAll();
// then
assertThat(handled).hasSize(2);
}
@Test
void aFailedHandlerWaitsOutItsBackoffBeforeThePartitionIsReadAgain() {
// given
Instant start = Instant.now();
RetryPolicy threeAttempts =
new RetryPolicy(
RetryMode.PAUSE_PARTITION,
3,
Duration.ofSeconds(1),
Duration.ofSeconds(10),
2.0,
false,
OrderingImpact.PRESERVE,
java.util.Set.of(),
java.util.Set.of());
KafkaConsumerRegistrar registrar =
registrar(
profile(Duration.ofSeconds(1), threeAttempts, false),
this::fail,
KafkaQuarantinePublisher.unavailable(),
() -> start);
assign();
addRecord(10);
registrar.pollOnce(start);
handlers.runAll();
// when
registrar.pollOnce(start.plusMillis(500));
// then
assertThat(consumer.paused())
.as("a failure retried with no delay re-reads the same record as fast as the loop polls")
.contains(PARTITION);
}
@Test
void aHandlerThatSpendsItsAttemptBudgetIsQuarantinedAndCommittedPast() {
// given
List<Long> quarantined = new ArrayList<>();
KafkaConsumerRegistrar registrar =
registrar(
KafkaContractHarness.profile(),
this::fail,
(record, reason) -> quarantined.add(record.offset()),
() -> NOW);
assign();
addRecord(10);
registrar.pollOnce(NOW);
handlers.runAll();
assertThat(handler)
.as("a timed-out handler must be cancelled before the same partition can be redelivered")
.isCancelled();
// when
registrar.pollOnce(NOW);
// then
assertThat(registrar.committedOffset(PARTITION))
.as("a poison message retried forever stops its partition for good")
.hasValue(11L);
}
@Test
void aProductionDestinationWithoutAQuarantineIsRefused() {
// given
dev.caskeleton.messaging.policy.DestinationProfile production =
profile(Duration.ofSeconds(30), RetryPolicy.none(), true);
// when
Throwable thrown =
org.assertj.core.api.Assertions.catchThrowable(
() -> registrar(production, this::settleImmediately));
// then
assertThat(thrown)
.isInstanceOf(dev.caskeleton.messaging.api.error.MessagingConfigurationException.class)
.extracting(
failure ->
((dev.caskeleton.messaging.api.error.MessagingConfigurationException) failure)
.failure()
.code())
.isEqualTo("KAFKA_QUARANTINE_REQUIRED");
}
@Test
void aPartitionPausedAtItsCeilingResumesOnceItsDeliveryFinishes() {
// given
KafkaConsumerRegistrar registrar = registrar(this::settleImmediately);
assign();
addRecord(10);
addRecord(11);
registrar.pollOnce(NOW);
handlers.runAll();
// The second record was returned by the first poll and handed back with a seek.
addRecord(11);
// when
registrar.pollOnce(NOW);
handlers.runAll();
// then
assertThat(handled)
.as("pausing a full partition is backpressure only if something resumes it")
.hasSize(2);
}
@Test
@@ -495,6 +612,68 @@ class KafkaConsumerRegistrarTest {
Duration.ofMillis(1));
}
private KafkaConsumerRegistrar registrar(
dev.caskeleton.messaging.policy.DestinationProfile profile,
java.util.function.Function<TransportDelivery, CompletionStage<Void>> sink) {
return new KafkaConsumerRegistrar(
new TransportConsumerSpec(profile, sink),
consumer,
handlers,
new GracefulShutdownCoordinator(Duration.ofSeconds(30)),
Duration.ofMillis(1));
}
private KafkaConsumerRegistrar registrar(
dev.caskeleton.messaging.policy.DestinationProfile profile,
java.util.function.Function<TransportDelivery, CompletionStage<Void>> sink,
KafkaQuarantinePublisher quarantine,
java.time.InstantSource clock) {
return new KafkaConsumerRegistrar(
new TransportConsumerSpec(profile, sink),
consumer,
handlers,
new GracefulShutdownCoordinator(Duration.ofSeconds(30)),
Duration.ofMillis(1),
quarantine,
clock);
}
/** The harness profile with the three settings these tests vary. */
private static dev.caskeleton.messaging.policy.DestinationProfile profile(
Duration handlerTimeout, RetryPolicy retry, boolean production) {
dev.caskeleton.messaging.policy.DestinationProfile base = KafkaContractHarness.profile();
return new dev.caskeleton.messaging.policy.DestinationProfile(
base.name(),
base.broker(),
base.kind(),
base.physical(),
base.schema(),
base.deliveryGuarantee(),
base.orderingScope(),
base.externalSideEffectGuarantee(),
base.producer(),
new dev.caskeleton.messaging.policy.ConsumerPolicy(
base.consumer().group(),
base.consumer().concurrency(),
base.consumer().maxInFlightPerOrderingUnit(),
base.consumer().prefetch(),
handlerTimeout,
base.consumer().manualSettlement()),
retry,
base.deadLetter(),
base.payload(),
base.tier(),
production,
base.keyResolverConfigured(),
base.topologyAutoCreate());
}
/** A handler that fails every delivery it is given. */
private CompletionStage<Void> fail(TransportDelivery delivery) {
handled.add(delivery);
throw new IllegalStateException("the downstream this handler writes to is down");
}
/** Acknowledges inside the handler, as a successful M1 delivery does. */
private CompletionStage<Void> settleImmediately(TransportDelivery delivery) {
handled.add(delivery);
@@ -205,7 +205,7 @@ class OutboxPostgresIT {
appendInTransaction(pending);
repository.markPublishedUnfenced(published.messageId(), NOW);
int removed = repository.purgePublishedBefore(NOW.plusSeconds(1));
int removed = repository.purgePublishedBefore(NOW.plusSeconds(1), 100);
assertThat(removed).isEqualTo(1);
assertThat(repository.find(pending.messageId())).isPresent();
@@ -217,7 +217,35 @@ class OutboxPostgresIT {
OutboxCanonicalMetadata.none().withRoutingKeys("acct-1", "acct-1"));
}
@Test
void aNewerRowWaitsWhileAnOlderRowWithTheSameOrderingKeyIsUnfinished() {
// given
OutboxRecord older = record(ordered("customer-42"), NOW.minusSeconds(60));
OutboxRecord newer = record(ordered("customer-42"), NOW.minusSeconds(30));
OutboxRecord otherKey = record(ordered("customer-7"), NOW.minusSeconds(30));
appendInTransaction(older);
appendInTransaction(newer);
appendInTransaction(otherKey);
// when
List<OutboxLease> claimed = repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW);
// then
assertThat(claimed)
.extracting(OutboxLease::messageId)
.as("a second relay, or a retry of the older row, would put the newer one first")
.containsExactlyInAnyOrder(older.messageId(), otherKey.messageId());
}
private static OutboxCanonicalMetadata ordered(String key) {
return OutboxCanonicalMetadata.none().withRoutingKeys(key, key);
}
private static OutboxRecord record(OutboxCanonicalMetadata metadata) {
return record(metadata, NOW.minusSeconds(60));
}
private static OutboxRecord record(OutboxCanonicalMetadata metadata, Instant createdAt) {
return new OutboxRecord(
MessageId.newId(),
new DestinationName("order-events"),
@@ -227,7 +255,7 @@ class OutboxPostgresIT {
"{\"orderId\":\"o-1\"}".getBytes(StandardCharsets.UTF_8),
Map.of(),
metadata,
NOW.minusSeconds(60),
createdAt,
OutboxStatus.PENDING,
0,
Optional.empty(),
@@ -361,8 +389,8 @@ class OutboxPostgresIT {
@Test
void twoRelaysClaimingConcurrentlyGetDisjointRowsAndDistinctTokens() {
OutboxRecord one = record();
OutboxRecord two = record();
OutboxRecord one = record(OutboxCanonicalMetadata.none().withRoutingKeys("acct-1", "acct-1"));
OutboxRecord two = record(OutboxCanonicalMetadata.none().withRoutingKeys("acct-2", "acct-2"));
appendInTransaction(one);
appendInTransaction(two);
@@ -113,17 +113,29 @@ public final class JdbcOutboxRepository implements OutboxRepository {
private static final String CLAIM =
"""
WITH claimable AS (
SELECT message_id
FROM messaging_outbox
WHERE status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT')
AND (lease_expires_at IS NULL OR lease_expires_at <= ?)
SELECT m.message_id
FROM messaging_outbox m
WHERE m.status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT')
AND (m.lease_expires_at IS NULL OR m.lease_expires_at <= ?)
-- The retry clock lives in the row, not in the relay's memory. Without these two
-- predicates an AMBIGUOUS row became claimable again on the very next pass, so a
-- broker outage meant the whole backlog was republished every poll interval and the
-- configured attempt budget was a number nothing consulted.
AND (next_attempt_at IS NULL OR next_attempt_at <= ?)
AND attempts < ?
ORDER BY created_at
AND (m.next_attempt_at IS NULL OR m.next_attempt_at <= ?)
AND m.attempts < ?
-- A row with an ordering key waits while an older row with the same key is still
-- unfinished. SKIP LOCKED let a second relay take the newer row while the first still
-- held the older one, and an AMBIGUOUS older row was retried after newer ones had
-- already gone out. md5 matches the partial index, which cannot hold a key that long.
AND (m.ordering_key IS NULL OR NOT EXISTS (
SELECT 1
FROM messaging_outbox older
WHERE older.destination = m.destination
AND md5(older.ordering_key) = md5(m.ordering_key)
AND older.ordering_key = m.ordering_key
AND older.status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT')
AND (older.created_at, older.message_id) < (m.created_at, m.message_id)))
ORDER BY m.created_at, m.message_id
LIMIT ?
FOR UPDATE SKIP LOCKED
)
@@ -514,20 +526,6 @@ public final class JdbcOutboxRepository implements OutboxRepository {
});
}
@Override
public int purgePublishedBefore(Instant publishedBefore) {
Objects.requireNonNull(publishedBefore, "publishedBefore must not be null");
return withConnection(
connection -> {
try (PreparedStatement statement =
connection.prepareStatement(
"DELETE FROM messaging_outbox WHERE status = 'PUBLISHED' AND published_at < ?")) {
statement.setTimestamp(1, Timestamp.from(publishedBefore));
return statement.executeUpdate();
}
});
}
private void update(String sql, MessageId messageId, Instant now, String failureCode) {
withConnection(
connection -> {
@@ -15,6 +15,9 @@ import java.util.Objects;
*/
public final class OutboxCleanupJob {
/** How many rows one delete statement removes before yielding. */
public static final int DEFAULT_BATCH_SIZE = 1_000;
private final OutboxRepository outbox;
private final OutboxProperties properties;
private final int maxBatches;
@@ -47,9 +50,10 @@ public final class OutboxCleanupJob {
int removed = 0;
for (int batch = 0; batch < maxBatches; batch++) {
int deleted = outbox.purgePublishedBefore(cutoff);
int deleted = outbox.purgePublishedBefore(cutoff, DEFAULT_BATCH_SIZE);
removed += deleted;
if (deleted == 0) {
// A short batch means the backlog is drained; asking again would re-scan an empty range.
if (deleted < DEFAULT_BATCH_SIZE) {
break;
}
}
@@ -1,9 +1,16 @@
package dev.caskeleton.messaging.outbox;
import dev.caskeleton.messaging.api.MessageEnvelope;
import dev.caskeleton.messaging.api.destination.MessageDestination;
import dev.caskeleton.messaging.api.error.FailureCategory;
import dev.caskeleton.messaging.api.error.FailureDescriptor;
import dev.caskeleton.messaging.api.error.MessagingException;
import dev.caskeleton.messaging.api.publish.MessagePublisher;
import dev.caskeleton.messaging.api.publish.PublishCompletion;
import dev.caskeleton.messaging.api.publish.PublishEvidence;
import dev.caskeleton.messaging.api.publish.PublishOptions;
import dev.caskeleton.messaging.api.publish.PublishResult;
import dev.caskeleton.messaging.api.publish.RoutingOutcome;
import dev.caskeleton.messaging.reliability.OutboxLease;
import dev.caskeleton.messaging.reliability.OutboxRecord;
import dev.caskeleton.messaging.reliability.OutboxRepository;
@@ -13,6 +20,7 @@ import java.time.Duration;
import java.time.Instant;
import java.util.List;
import java.util.Objects;
import java.util.Optional;
/**
* Publishes outbox rows, treating an unknown outcome as retryable rather than final.
@@ -165,7 +173,22 @@ public final class OutboxRelay {
for (OutboxLease lease : batch) {
OutboxRecord record = lease.record();
PublishResult result = publish(record);
MessageEnvelope<EncodedMessage> envelope;
try {
envelope = envelopeFactory.toEnvelope(record);
} catch (RuntimeException unpublishable) {
// A row that cannot become an envelope never will. Thrown out of the pass, it took the
// rest of the batch with it: those rows stayed IN_FLIGHT until their lease expired, and the
// same row was claimed and thrown again on the next pass, forever.
if (repository.markFailed(lease, failureCode(unpublishable), now)
== OutboxTransitionResult.APPLIED) {
failed++;
} else {
stale++;
}
continue;
}
PublishResult result = publish(record, envelope);
switch (result.completion()) {
case CONFIRMED -> {
if (repository.markPublished(lease, now) == OutboxTransitionResult.APPLIED) {
@@ -220,12 +243,35 @@ public final class OutboxRelay {
return new OutboxRelayReport(batch.size(), published, ambiguous, failed, stale, exhausted);
}
private PublishResult publish(OutboxRecord record) {
private PublishResult publish(OutboxRecord record, MessageEnvelope<EncodedMessage> envelope) {
MessageDestination<EncodedMessage> destination =
new MessageDestination<>(record.destination(), record.messageType(), EncodedMessage.class);
return publisher
.publish(destination, envelopeFactory.toEnvelope(record), PublishOptions.defaults())
.toCompletableFuture()
.join();
try {
return publisher
.publish(destination, envelope, PublishOptions.defaults())
.toCompletableFuture()
.join();
} catch (RuntimeException unknown) {
// An exception says nothing about whether the broker took the message, so the row is retried
// under the same id like any other unknown outcome — not failed, not left IN_FLIGHT.
return new PublishResult(
PublishCompletion.AMBIGUOUS,
PublishEvidence.ambiguous(),
RoutingOutcome.UNKNOWN,
Optional.empty(),
1,
Duration.ZERO,
Optional.of(
FailureDescriptor.of(
FailureCategory.AMBIGUOUS,
"PUBLISH_OUTCOME_UNKNOWN",
"the publisher threw " + unknown.getClass().getSimpleName())));
}
}
private static String failureCode(RuntimeException failure) {
return failure instanceof MessagingException known
? known.failure().code()
: "OUTBOX_ROW_UNPUBLISHABLE";
}
}
@@ -34,6 +34,8 @@ public final class OutboxRelayWorker implements AutoCloseable {
/** How long {@link #close()} waits for a running pass when no deadline is given. */
public static final Duration DEFAULT_DRAIN_DEADLINE = Duration.ofSeconds(30);
private static final System.Logger LOG = System.getLogger(OutboxRelayWorker.class.getName());
private final OutboxRelay relay;
private final OutboxRetryScheduler scheduler;
@@ -184,9 +186,14 @@ public final class OutboxRelayWorker implements AutoCloseable {
} catch (RuntimeException passFailed) {
// A failed pass must not stop the loop: the scheduled task's own exception would cancel every
// future pass, turning one broker error into a relay that never runs again. The failure is
// counted and the next pass backs off as if nothing was published, which is true.
failedPasses.incrementAndGet();
// counted and the next pass backs off as if nothing was published, which is true. It is also
// logged: counted silently, a relay failing every pass looked exactly like an idle one.
long failures = failedPasses.incrementAndGet();
unproductive = unproductivePasses.incrementAndGet();
LOG.log(
System.Logger.Level.WARNING,
"outbox relay pass failed (" + failures + " failed so far); retrying with backoff",
passFailed);
}
completedPasses.incrementAndGet();
schedule(scheduler.backoff(unproductive));
@@ -0,0 +1,9 @@
-- The claim now holds back a row while an older, unfinished row shares its destination and
-- ordering key. That check runs once per candidate row, so it needs an index or every claim scans
-- the backlog once per row it considers.
--
-- md5 of the key rather than the key: ordering_key is VARCHAR(1024), and a long multi-byte key
-- would exceed the btree entry limit and fail the business transaction that appends the row.
CREATE INDEX ix_messaging_outbox_unfinished_ordering_key
ON messaging_outbox (destination, md5(ordering_key), created_at)
WHERE ordering_key IS NOT NULL AND status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT');
@@ -96,15 +96,13 @@ class OutboxOperationsTest {
throw new UnsupportedOperationException();
}
@Override
public int purgePublishedBefore(Instant publishedBefore, int limit) {
return Math.min(purgePublishedBefore(publishedBefore), limit);
}
private final List<Integer> limits = new ArrayList<>();
@Override
public int purgePublishedBefore(Instant publishedBefore) {
public int purgePublishedBefore(Instant publishedBefore, int limit) {
cutoffs.add(publishedBefore);
return pass < deletions.size() ? deletions.get(pass++) : 0;
limits.add(limit);
return Math.min(pass < deletions.size() ? deletions.get(pass++) : 0, limit);
}
}
@@ -130,13 +128,20 @@ class OutboxOperationsTest {
}
@Test
void cleanupDeletesInBoundedBatchesRatherThanOneLongStatement() {
void cleanupAsksForBoundedBatchesAndStopsAtTheFirstShortOne() {
// given
RecordingRepository repository = new RecordingRepository(List.of(1000, 1000, 250));
int removed = new OutboxCleanupJob(repository, OutboxProperties.defaults(), 10).runOnce(NOW);
// when
new OutboxCleanupJob(repository, OutboxProperties.defaults(), 10).runOnce(NOW);
assertThat(removed).isEqualTo(2250);
assertThat(repository.cutoffs).hasSize(4);
// then
assertThat(repository.limits)
.as("one DELETE over the whole backlog holds its locks for as long as the backlog is big")
.containsExactly(
OutboxCleanupJob.DEFAULT_BATCH_SIZE,
OutboxCleanupJob.DEFAULT_BATCH_SIZE,
OutboxCleanupJob.DEFAULT_BATCH_SIZE);
}
@Test
@@ -86,6 +86,49 @@ class OutboxRelayTest {
.isEqualTo(OutboxStatus.PUBLISHED);
}
@Test
void aRowThatCannotBecomeAnEnvelopeFailsWhileTheRestOfItsBatchPublishes() {
// given
OutboxRecord poison = OutboxFixtures.withHeaders(Map.of("msg.id", "forged"));
OutboxRecord healthy = OutboxFixtures.pending();
repository.append(poison);
repository.append(healthy);
// when
relay(RecordingOutboxPublisher.confirming()).runOnce(NOW);
// then
assertThat(List.of(poison, healthy))
.extracting(row -> repository.find(row.messageId()).orElseThrow().status())
.as("one bad row threw out of the pass and held the whole batch until its lease expired")
.containsExactly(OutboxStatus.FAILED, OutboxStatus.PUBLISHED);
}
@Test
void aPublisherThatThrowsLeavesTheRowAmbiguousRatherThanStrandingTheBatch() {
// given
OutboxRecord first = OutboxFixtures.pending();
repository.append(first);
MessagePublisher throwing =
new MessagePublisher() {
@Override
public <T> CompletionStage<PublishResult> publish(
MessageDestination<T> destination,
MessageEnvelope<T> message,
PublishOptions options) {
throw new IllegalStateException("the broker connection is gone");
}
};
// when
relay(throwing).runOnce(NOW);
// then
assertThat(repository.find(first.messageId()).orElseThrow().status())
.as("an exception says nothing about whether the broker took the message")
.isEqualTo(OutboxStatus.AMBIGUOUS);
}
@Test
void anAmbiguousPublishStaysRetryableRatherThanFailing() {
OutboxRecord record = OutboxFixtures.pending();
@@ -263,6 +306,10 @@ final class OutboxFixtures {
private OutboxFixtures() {}
static OutboxRecord pending() {
return withHeaders(Map.of());
}
static OutboxRecord withHeaders(Map<String, String> headers) {
return new OutboxRecord(
MessageId.newId(),
new DestinationName("order-events"),
@@ -270,7 +317,7 @@ final class OutboxFixtures {
new SchemaVersion(1),
ContentType.JSON,
"{\"orderId\":\"o-1\"}".getBytes(StandardCharsets.UTF_8),
Map.of(),
headers,
dev.caskeleton.messaging.reliability.OutboxCanonicalMetadata.none(),
Instant.parse("2026-08-10T09:14:00Z"),
OutboxStatus.PENDING,
@@ -455,14 +502,9 @@ final class InMemoryOutboxRepository implements OutboxRepository {
@Override
public int purgePublishedBefore(Instant publishedBefore, int limit) {
return Math.min(purgePublishedBefore(publishedBefore), limit);
}
@Override
public int purgePublishedBefore(Instant publishedBefore) {
int before = rows.size();
rows.values().removeIf(record -> record.status() == OutboxStatus.PUBLISHED);
return before - rows.size();
return Math.min(before - rows.size(), limit);
}
private void transition(MessageId messageId, OutboxStatus status, Optional<String> failureCode) {
@@ -79,8 +79,7 @@ class OutboxRelayWorkerTest {
@Test
@DisplayName("a pass that throws does not stop every pass after it")
void aFailedPassDoesNotEndTheLoop() throws InterruptedException {
repository.append(OutboxFixtures.pending());
OutboxRelayWorker worker = worker(new ThrowingPublisher());
OutboxRelayWorker worker = worker(unreachableDatabase(), new ThrowingPublisher());
worker.start();
try {
@@ -99,9 +98,25 @@ class OutboxRelayWorkerTest {
}
private OutboxRelayWorker worker(MessagePublisher publisher) {
return worker(repository, publisher);
}
/** An outbox whose every call fails, as it does while its database is down. */
private static dev.caskeleton.messaging.reliability.OutboxRepository unreachableDatabase() {
return (dev.caskeleton.messaging.reliability.OutboxRepository)
java.lang.reflect.Proxy.newProxyInstance(
OutboxRelayWorkerTest.class.getClassLoader(),
new Class<?>[] {dev.caskeleton.messaging.reliability.OutboxRepository.class},
(proxy, method, arguments) -> {
throw new IllegalStateException("the outbox database is unreachable");
});
}
private static OutboxRelayWorker worker(
dev.caskeleton.messaging.reliability.OutboxRepository outbox, MessagePublisher publisher) {
OutboxRelay relay =
new OutboxRelay(
repository,
outbox,
publisher,
new OutboxEnvelopeFactory(new ProducerId("order-api")),
100,
@@ -50,7 +50,20 @@ public final class InFlightLimiter {
* @throws InterruptedException when the waiting thread is interrupted
*/
public boolean tryAcquire() throws InterruptedException {
return permits.tryAcquire(maxWait.toNanos(), TimeUnit.NANOSECONDS);
return tryAcquire(maxWait);
}
/**
* Tries to acquire one permit, waiting no longer than the configured bound or the given budget.
*
* @param budget what the caller has left to spend on the wait
* @return true when a permit was acquired
* @throws InterruptedException when the waiting thread is interrupted
*/
public boolean tryAcquire(Duration budget) throws InterruptedException {
Objects.requireNonNull(budget, "budget must not be null");
long wait = Math.max(0, Math.min(maxWait.toNanos(), budget.toNanos()));
return permits.tryAcquire(wait, TimeUnit.NANOSECONDS);
}
/** Returns one permit to the pool. */
@@ -73,7 +73,24 @@ public final class MessagingAdmissionController {
* @throws MessageBackpressureException when shutting down or no permit became available
*/
public void admit(String destination, int payloadBytes) {
admit(destination, payloadBytes, java.time.Duration.ofNanos(Long.MAX_VALUE));
}
/**
* Admits one publish, waiting for a permit no longer than the caller has left.
*
* <p>The limiter's own bound was the only one: a publish with a 100ms deadline could block its
* caller for the limiter's full wait before the deadline was even consulted.
*
* @param destination the logical destination name
* @param payloadBytes the encoded payload size
* @param budget the most this call may wait for a permit
* @throws MessageTooLargeException when the payload exceeds the destination limit
* @throws MessageBackpressureException when shutting down or no permit became available
*/
public void admit(String destination, int payloadBytes, java.time.Duration budget) {
Objects.requireNonNull(destination, "destination must not be null");
Objects.requireNonNull(budget, "budget must not be null");
payloadGuard.checkPayload(destination, payloadBytes);
if (!acceptingNewWork) {
@@ -91,7 +108,7 @@ public final class MessagingAdmissionController {
boolean acquired;
try {
acquired = limiter.tryAcquire();
acquired = limiter.tryAcquire(budget);
} catch (InterruptedException interrupted) {
// The destination slot was taken a moment ago and no publish will use it, so it goes back
// here: a slot leaked per interruption shrinks the destination's ceiling until it is zero.
@@ -34,19 +34,11 @@ public interface InboxRepository {
boolean isProcessed(MessageId messageId, String consumerId);
/**
* Deletes records older than a cutoff.
* Deletes at most {@code limit} processed rows older than the cutoff.
*
* <p>Retention must outlive the broker's maximum redelivery window, otherwise a late redelivery
* arrives after its inbox row was pruned and is processed a second time.
*
* @param processedBefore the retention cutoff
* @return how many rows were removed
*/
int purgeProcessedBefore(Instant processedBefore);
/**
* Deletes at most {@code limit} processed rows older than the cutoff.
*
* @return how many rows were deleted; fewer than {@code limit} means the sweep is finished
*/
int purgeProcessedBefore(Instant processedBefore, int limit);
@@ -94,22 +94,12 @@ public interface OutboxRepository {
*/
Optional<OutboxRecord> find(MessageId messageId);
/**
* Deletes records published before a cutoff.
*
* @param publishedBefore the retention cutoff
* @return how many rows were removed
*/
int purgePublishedBefore(Instant publishedBefore);
/**
* Deletes at most {@code limit} published rows older than the cutoff.
*
* <p>The unbounded version deletes everything before the cutoff in one statement. On a table that
* has been accumulating published rows since the last sweep that is a single long transaction
* holding locks and generating WAL in proportion to the backlog, which shows up as the relay and
* the business writes stalling behind retention. The cleanup jobs describe themselves as bounded
* by batch size; this is the parameter that makes that true.
* <p>There is no unbounded version. Deleting everything before the cutoff in one statement is a
* single long transaction holding locks and generating WAL in proportion to the backlog, which
* shows up as the relay and the business writes stalling behind retention.
*
* @return how many rows were deleted; fewer than {@code limit} means the sweep is finished
*/
@@ -224,7 +224,16 @@ public final class DefaultMessagePublisher implements MessagePublisher, EncodedM
startedAt);
}
admission.admit(destination.name().value(), encoded.payload().bytes().length);
// Refusals are results, not exceptions. They were thrown out of publish() on the caller's
// thread, so a CompletionStage caller — the outbox relay, the reactive facade — saw neither a
// REJECTED result nor a failed stage. The permit wait is bounded by what is left of the
// caller's deadline, not by the limiter's own bound.
try {
admission.admit(destination.name().value(), encoded.payload().bytes().length, remaining);
} catch (dev.caskeleton.messaging.api.error.MessageTooLargeException
| dev.caskeleton.messaging.api.error.MessageBackpressureException refused) {
return refused(refused.failure(), startedAt);
}
MessagingRuntimeLease lease = null;
try {
lease = runtimes.acquire(profile.broker());
@@ -316,6 +325,12 @@ public final class DefaultMessagePublisher implements MessagePublisher, EncodedM
}
private CompletionStage<PublishResult> rejected(String code, String reason, long startedAt) {
return refused(
new FailureDescriptor(FailureCategory.CONFIGURATION, code, false, reason, Optional.empty()),
startedAt);
}
private CompletionStage<PublishResult> refused(FailureDescriptor failure, long startedAt) {
return CompletableFuture.completedFuture(
new PublishResult(
PublishCompletion.REJECTED,
@@ -324,9 +339,7 @@ public final class DefaultMessagePublisher implements MessagePublisher, EncodedM
Optional.empty(),
1,
elapsedSince(startedAt),
Optional.of(
new FailureDescriptor(
FailureCategory.CONFIGURATION, code, false, reason, Optional.empty()))));
Optional.of(failure)));
}
private PublishResult ambiguousResult(String code, String reason, long startedAt) {
@@ -312,6 +312,89 @@ class DefaultMessagePublisherTest {
assertThat(admission.inFlight()).isZero();
}
@Test
void anOversizedPayloadIsRejectedAsNotRetryableInsteadOfThrown() {
// given
DefaultMessagePublisher publisher =
new DefaultMessagePublisher(
registry(),
allowOrders(),
new SingleCodecRegistry(),
new MessagingAdmissionController(
new PayloadLimitGuard(new PayloadPolicy(4, 4)),
new InFlightLimiter(4, Duration.ofSeconds(1))),
new FixedRuntimeRegistry(transport),
transport);
// when
CompletableFuture<PublishResult> result =
CompletableFuture.supplyAsync(
() ->
publisher
.publish(
new MessageDestination<>(
ORDERS, new MessageType("order.created"), String.class),
envelope(),
PublishOptions.defaults())
.toCompletableFuture()
.join());
// then
assertThat(result)
.succeedsWithin(Duration.ofSeconds(5))
.extracting(
PublishResult::completion, outcome -> outcome.failure().orElseThrow().retryable())
.containsExactly(PublishCompletion.REJECTED, false);
}
@Test
void aSaturatedPermitPoolRejectsAsRetryableWithinTheCallersDeadline() {
// given
DestinationName audit = new DestinationName("audit-events");
DefaultMessagePublisher publisher =
new DefaultMessagePublisher(
DestinationProfileRegistry.of(
Map.of(
ORDERS,
RuntimeFixtures.profile(ORDERS),
audit,
RuntimeFixtures.profile(audit))),
new DestinationAccessPolicy(
java.util.Set.of(ORDERS, audit), java.util.Set.of(), java.util.Set.of()),
new SingleCodecRegistry(),
new MessagingAdmissionController(
new PayloadLimitGuard(PayloadPolicy.defaults()),
new InFlightLimiter(1, Duration.ofMinutes(1))),
new FixedRuntimeRegistry(transport),
transport);
transport.stall();
publisher.publish(
new MessageDestination<>(ORDERS, new MessageType("order.created"), String.class),
envelope(),
withTimeout(Duration.ofMinutes(1)));
// when
CompletableFuture<PublishResult> second =
CompletableFuture.supplyAsync(
() ->
publisher
.publish(
new MessageDestination<>(
audit, new MessageType("order.created"), String.class),
envelope(),
withTimeout(Duration.ofMillis(100)))
.toCompletableFuture()
.join());
// then
assertThat(second)
.as("the permit wait was the limiter's own minute, not what was left of 100ms")
.succeedsWithin(Duration.ofSeconds(5))
.extracting(
PublishResult::completion, outcome -> outcome.failure().orElseThrow().retryable())
.containsExactly(PublishCompletion.REJECTED, true);
}
@Test
@DisplayName("an unavailable runtime releases the permit it took")
void anUnavailableRuntimeReleasesThePermit() {
@@ -121,15 +121,14 @@ public class KafkaMessagingAutoConfiguration {
/**
* The producer this transport publishes through (MSG-INT-003).
*
* <p>Built from Spring Boot's own {@code spring.kafka.bootstrap-servers} rather than from a
* second description on {@code app.messaging.*}. One broker described twice is the defect this
* repository has already paid for in {@code app.jpa-platform.datasource.*}, and the
* operator-facing property for "where is Kafka" is the one they already know.
* <p>Built from the one {@code app.messaging.brokers.<name>} entry of type {@code kafka}. It used
* to read {@code spring.kafka.bootstrap-servers} while the compiler required and validated the
* entry's own {@code bootstrap-servers}, so one broker was described twice and only one of the
* two descriptions reached the client.
*
* <p>Byte-array serializers on purpose: the platform's codec registry decides the wire format,
* and a serializer configured here would be a second encoder disagreeing with it.
*
* @param environment the resolved environment
* @param compiled the broker and security profiles compiled from {@code app.messaging}
* @param declaredSecurity the security profiles the application declared as beans
* @param securityConfigurer the Kafka security configurer, present only with a credential source
@@ -138,45 +137,75 @@ public class KafkaMessagingAutoConfiguration {
@Bean(destroyMethod = "close")
@ConditionalOnMissingBean
public org.apache.kafka.clients.producer.Producer<byte[], byte[]> messagingKafkaProducer(
org.springframework.core.env.Environment environment,
CompiledMessagingConfiguration compiled,
ObjectProvider<BrokerSecurityProfile> declaredSecurity,
ObjectProvider<KafkaSecurityConfigurer> securityConfigurer) {
return new org.apache.kafka.clients.producer.KafkaProducer<>(
producerConfig(environment, compiled, declaredSecurity, securityConfigurer));
producerConfig(compiled, declaredSecurity, securityConfigurer));
}
/** The producer's client configuration; package-visible so a test can read what it holds. */
static java.util.Map<String, Object> producerConfig(
org.springframework.core.env.Environment environment,
CompiledMessagingConfiguration compiled,
ObjectProvider<BrokerSecurityProfile> declaredSecurity,
ObjectProvider<KafkaSecurityConfigurer> securityConfigurer) {
String bootstrapServers = environment.getProperty("spring.kafka.bootstrap-servers", "").trim();
if (bootstrapServers.isEmpty()) {
throw new IllegalStateException(
"app.messaging.broker=kafka needs spring.kafka.bootstrap-servers; a producer with no "
+ "broker address defaults to localhost:9092 and fails at the first publish rather "
+ "than at startup");
}
BrokerSettings broker = kafkaBroker(compiled);
java.util.Map<String, Object> config = new java.util.HashMap<>();
config.put(
org.apache.kafka.clients.producer.ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
bootstrapServers);
String.join(",", broker.bootstrapServers()));
config.put(
org.apache.kafka.clients.producer.ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
org.apache.kafka.common.serialization.ByteArraySerializer.class);
config.put(
org.apache.kafka.clients.producer.ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
org.apache.kafka.common.serialization.ByteArraySerializer.class);
// Durability is not a default worth inheriting: acks=1 loses an accepted publish to a leader
// failover, which is exactly the outcome an outbox exists to prevent.
config.put(org.apache.kafka.clients.producer.ProducerConfig.ACKS_CONFIG, "all");
config.put(org.apache.kafka.clients.producer.ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
// The entry's delivery settings, which KafkaProfileValidator checks (a stable broker must keep
// acks=all and idempotence) and which used to be replaced here by hard-coded values.
config.put(org.apache.kafka.clients.producer.ProducerConfig.ACKS_CONFIG, broker.acks());
config.put(
org.apache.kafka.clients.producer.ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG,
broker.enableIdempotence());
config.put(
org.apache.kafka.clients.producer.ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION,
broker.maxInFlightRequestsPerConnection());
// Kafka refuses delivery.timeout.ms below linger.ms + request.timeout.ms (5ms + 30s by
// default), which made the documented 30s a startup failure. The request timeout follows a
// short delivery deadline down, and records are not held back to fill a batch.
long delivery = broker.deliveryTimeout().toMillis();
config.put(org.apache.kafka.clients.producer.ProducerConfig.LINGER_MS_CONFIG, 0);
config.put(
org.apache.kafka.clients.producer.ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG,
Math.toIntExact(Math.min(30_000L, delivery)));
config.put(
org.apache.kafka.clients.producer.ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG,
Math.toIntExact(delivery));
// send() blocks the caller while it waits for metadata or buffer space, for Kafka's default
// 60s — before the publish deadline is even applied. Bounded by the default deadline instead.
config.put(
org.apache.kafka.clients.producer.ProducerConfig.MAX_BLOCK_MS_CONFIG,
dev.caskeleton.messaging.api.publish.PublishOptions.defaults().timeout().toMillis());
config.putAll(producerSecurity(compiled, declaredSecurity, securityConfigurer));
return config;
}
/** The one Kafka broker entry the producer connects to. */
private static BrokerSettings kafkaBroker(CompiledMessagingConfiguration compiled) {
java.util.List<BrokerSettings> kafka =
compiled.brokers().values().stream()
.filter(broker -> BrokerSettings.KAFKA.equalsIgnoreCase(broker.type()))
.toList();
if (kafka.size() != 1) {
throw new IllegalStateException(
"app.messaging.broker=kafka needs exactly one app.messaging.brokers.<name> entry with"
+ " type kafka and its bootstrap-servers; found "
+ kafka.size()
+ ". A producer with no broker address defaults to localhost:9092 and fails at the"
+ " first publish rather than at startup");
}
return kafka.getFirst();
}
/**
* The producer's security properties, from the one security profile of the Kafka broker.
*
@@ -77,7 +77,8 @@ class MessagingLiveRoundTripQualificationTest {
.withPropertyValues(
"app.messaging.enabled=true",
"app.messaging.broker=kafka",
"spring.kafka.bootstrap-servers=" + KAFKA.getBootstrapServers())
"app.messaging.brokers.kafka.type=kafka",
"app.messaging.brokers.kafka.bootstrap-servers[0]=" + KAFKA.getBootstrapServers())
// Both, because ValidatedDestinationRegistry refuses a profile whose dead-letter target is
// not itself registered — a good invariant, and one a fixture declaring only the happy path
// walks straight into.
@@ -159,7 +159,6 @@ class MessagingSecuredBrokerQualificationTest {
.withPropertyValues(
"app.messaging.enabled=true",
"app.messaging.broker=kafka",
"spring.kafka.bootstrap-servers=" + kafka.getBootstrapServers(),
"app.messaging.brokers.kafka.type=kafka",
"app.messaging.brokers.kafka.bootstrap-servers[0]=" + kafka.getBootstrapServers(),
"app.messaging.brokers.kafka.tls-enabled=true",
@@ -34,6 +34,10 @@ public final class DefaultReactiveMessagePublisher implements ReactiveMessagePub
Objects.requireNonNull(destination, "destination must not be null");
Objects.requireNonNull(message, "message must not be null");
Objects.requireNonNull(options, "options must not be null");
return Mono.fromCompletionStage(() -> delegate.publish(destination, message, options));
// publish() is not purely asynchronous: admission may wait for a permit and the Kafka producer
// may wait for metadata before the stage exists. Subscribing on a Netty event loop would pin
// that loop for the wait, so the call runs on the bounded elastic pool instead.
return Mono.fromCompletionStage(() -> delegate.publish(destination, message, options))
.subscribeOn(reactor.core.scheduler.Schedulers.boundedElastic());
}
}
@@ -109,6 +109,33 @@ public class MessagingReliabilityAutoConfiguration {
return new OutboxRelayWorker(relay, scheduler);
}
/**
* Publishes the relay's failed-pass count when a meter registry exists.
*
* <p>{@code failedPasses()} was counted and read by nothing, so a relay failing every pass looked
* on every dashboard exactly like one with nothing to do.
*
* @param worker the relay worker
* @param registries the meter registry, when the application has one
* @return the registration, run once the context has the registry
*/
@Bean
@ConditionalOnBean(OutboxRelayWorker.class)
public org.springframework.beans.factory.InitializingBean outboxRelayMeters(
OutboxRelayWorker worker,
org.springframework.beans.factory.ObjectProvider<io.micrometer.core.instrument.MeterRegistry>
registries) {
return () ->
registries.ifAvailable(
registry ->
io.micrometer.core.instrument.FunctionCounter.builder(
"messaging.outbox.relay.failed.passes",
worker,
OutboxRelayWorker::failedPasses)
.description("Outbox relay passes that ended in an exception")
.register(registry));
}
/**
* Binds the relay worker to the context lifecycle.
*
@@ -38,7 +38,6 @@ class BrokerAccessListTest {
.withPropertyValues(
"app.messaging.enabled=true",
"app.messaging.broker=kafka",
"spring.kafka.bootstrap-servers=localhost:9092",
"app.messaging.brokers.kafka-primary.type=kafka",
"app.messaging.brokers.kafka-primary.bootstrap-servers[0]=localhost:9092",
"app.messaging.security.kafka-primary.producer.type=SASL_SCRAM",
@@ -0,0 +1,96 @@
package dev.caskeleton.messaging.autoconfigure;
import static org.assertj.core.api.Assertions.assertThat;
import dev.caskeleton.messaging.api.publish.PublishOptions;
import dev.caskeleton.messaging.kafka.KafkaSecurityConfigurer;
import dev.caskeleton.messaging.security.BrokerSecurityProfile;
import java.util.Map;
import org.junit.jupiter.api.Test;
import org.springframework.boot.autoconfigure.AutoConfigurations;
import org.springframework.boot.test.context.assertj.AssertableApplicationContext;
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
/**
* Where the producer connects and how it delivers come from the one broker entry.
*
* <p>The producer read {@code spring.kafka.bootstrap-servers} while the compiler required and
* validated {@code app.messaging.brokers.<name>.bootstrap-servers}, so a broker was described twice
* and only one description was used. The entry's delivery settings were validated and then replaced
* by hard-coded values.
*/
class KafkaProducerConfigTest {
private final ApplicationContextRunner runner =
new ApplicationContextRunner()
.withConfiguration(AutoConfigurations.of(MessagingPlatformRootAutoConfiguration.class))
.withPropertyValues(
"app.messaging.enabled=true",
"app.messaging.broker=kafka",
"app.messaging.brokers.kafka-primary.type=kafka",
// Resolvable names: the client refuses an address it cannot look up at construction.
"app.messaging.brokers.kafka-primary.bootstrap-servers[0]=localhost:9093",
"app.messaging.brokers.kafka-primary.bootstrap-servers[1]=127.0.0.1:9094");
@Test
void theProducerConnectsToTheBrokerEntryAddressesNotToSpringKafka() {
runner
.withPropertyValues("spring.kafka.bootstrap-servers=elsewhere:9092")
.run(
context ->
assertThat(producerConfig(context).get("bootstrap.servers"))
.isEqualTo("localhost:9093,127.0.0.1:9094"));
}
@Test
void theProducerAppliesTheBrokerEntryDeliverySettings() {
runner
.withPropertyValues(
"app.messaging.brokers.kafka-primary.enable-idempotence=false",
"app.messaging.brokers.kafka-primary.acks=1",
"app.messaging.brokers.kafka-primary.max-in-flight-requests-per-connection=3",
"app.messaging.brokers.kafka-primary.delivery-timeout=45s")
.run(
context ->
assertThat(producerConfig(context))
.extractingByKeys(
"enable.idempotence",
"acks",
"max.in.flight.requests.per.connection",
"delivery.timeout.ms")
.containsExactly(false, "1", 3, 45_000));
}
@Test
void theProducerStopsWaitingForMetadataWithinTheDefaultPublishDeadline() {
runner.run(
context ->
assertThat(producerConfig(context).get("max.block.ms"))
.as("the caller's thread waited up to Kafka's own 60s before any deadline applied")
.isEqualTo(PublishOptions.defaults().timeout().toMillis()));
}
@Test
void aBrokerWithoutAnEntryFailsStartupNamingTheKey() {
new ApplicationContextRunner()
.withConfiguration(AutoConfigurations.of(MessagingPlatformRootAutoConfiguration.class))
.withPropertyValues(
"app.messaging.enabled=true",
"app.messaging.broker=kafka",
"spring.kafka.bootstrap-servers=localhost:9092")
.run(
context ->
assertThat(context)
.hasFailed()
.getFailure()
.rootCause()
.hasMessageContaining("app.messaging.brokers.<name> entry with type kafka"));
}
private static Map<String, Object> producerConfig(AssertableApplicationContext context) {
return KafkaMessagingAutoConfiguration.producerConfig(
context.getBean(CompiledMessagingConfiguration.class),
context.getBeanProvider(BrokerSecurityProfile.class),
context.getBeanProvider(KafkaSecurityConfigurer.class));
}
}
@@ -39,7 +39,6 @@ class KafkaProducerSecurityTest {
.withPropertyValues(
"app.messaging.enabled=true",
"app.messaging.broker=kafka",
"spring.kafka.bootstrap-servers=localhost:9092",
"app.messaging.brokers.kafka-primary.type=kafka",
"app.messaging.brokers.kafka-primary.bootstrap-servers[0]=localhost:9092");
@@ -228,7 +227,6 @@ class KafkaProducerSecurityTest {
private static Map<String, Object> producerConfig(AssertableApplicationContext context) {
return KafkaMessagingAutoConfiguration.producerConfig(
context.getEnvironment(),
context.getBean(CompiledMessagingConfiguration.class),
context.getBeanProvider(BrokerSecurityProfile.class),
context.getBeanProvider(KafkaSecurityConfigurer.class));
@@ -314,7 +314,9 @@ class MessagingAutoConfigurationTest {
org.springframework.boot.autoconfigure.AutoConfigurations.of(
MessagingCoreAutoConfiguration.class, KafkaMessagingAutoConfiguration.class))
.withUserConfiguration(PublisherConfiguration.class, AutoCommittingKafkaProfile.class)
.withPropertyValues("spring.kafka.bootstrap-servers=localhost:9092")
.withPropertyValues(
"app.messaging.brokers.kafka.type=kafka",
"app.messaging.brokers.kafka.bootstrap-servers[0]=localhost:9092")
.run(
context ->
org.assertj.core.api.Assertions.assertThat(context)
@@ -258,9 +258,7 @@ class MessagingConfigurationBindingTest {
runner
.withConfiguration(AutoConfigurations.of(KafkaMessagingAutoConfiguration.class))
.withPropertyValues(minimal())
.withPropertyValues(
"spring.kafka.bootstrap-servers=localhost:9092",
"app.messaging.brokers.kafka-primary.production=true")
.withPropertyValues("app.messaging.brokers.kafka-primary.production=true")
.run(
context ->
assertThat(context)
@@ -73,6 +73,55 @@ class MessagingOutboxRelayLifecycleTest {
.isFalse();
}
@Test
void aRelayPassThatFailsIsCountedOnTheMeterRegistry() {
new ApplicationContextRunner()
.withConfiguration(AutoConfigurations.of(MessagingReliabilityAutoConfiguration.class))
.withBean(
io.micrometer.core.instrument.MeterRegistry.class,
io.micrometer.core.instrument.simple.SimpleMeterRegistry::new)
.withBean(
OutboxRepository.class,
() ->
(OutboxRepository)
java.lang.reflect.Proxy.newProxyInstance(
getClass().getClassLoader(),
new Class<?>[] {OutboxRepository.class},
(proxy, method, arguments) -> {
throw new IllegalStateException("the outbox database is unreachable");
}))
.withBean(
OutboxEnvelopeFactory.class,
() -> new OutboxEnvelopeFactory(new ProducerId("order-api")))
.withBean(MessagePublisher.class, () -> new RelayDependencies().messagePublisher())
.run(
context -> {
// given
io.micrometer.core.instrument.MeterRegistry registry =
context.getBean(io.micrometer.core.instrument.MeterRegistry.class);
// when
double failed = awaitFailedPasses(registry);
// then
assertThat(failed)
.as("a pass that failed was swallowed with no log and a counter nothing read")
.isPositive();
});
}
/** Reads the failed-pass counter until the relay's first pass has had time to fail. */
private static double awaitFailedPasses(io.micrometer.core.instrument.MeterRegistry registry)
throws InterruptedException {
long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5);
double failed = 0;
while (failed == 0 && System.nanoTime() < deadline) {
Thread.sleep(10);
failed = registry.get("messaging.outbox.relay.failed.passes").functionCounter().count();
}
return failed;
}
/** The two beans only the application can supply, plus the publisher the platform needs. */
@Configuration(proxyBeanMethods = false)
static class RelayDependencies {
@@ -170,11 +219,6 @@ class MessagingOutboxRelayLifecycleTest {
throw new UnsupportedOperationException("not used by this test");
}
@Override
public int purgePublishedBefore(Instant publishedBefore) {
throw new UnsupportedOperationException("not used by this test");
}
@Override
public int purgePublishedBefore(Instant publishedBefore, int limit) {
throw new UnsupportedOperationException("not used by this test");
@@ -41,7 +41,9 @@ class MessagingStarterOffContractTest {
void presentWithoutTheSwitchIsOn() {
runner
.withPropertyValues(
"app.messaging.broker=kafka", "spring.kafka.bootstrap-servers=localhost:9092")
"app.messaging.broker=kafka",
"app.messaging.brokers.kafka.type=kafka",
"app.messaging.brokers.kafka.bootstrap-servers[0]=localhost:9092")
.run(
context -> {
assertThat(context).hasNotFailed();
@@ -123,7 +125,8 @@ class MessagingStarterOffContractTest {
.withPropertyValues(
"app.messaging.enabled=true",
"app.messaging.broker=kafka",
"spring.kafka.bootstrap-servers=localhost:9092")
"app.messaging.brokers.kafka.type=kafka",
"app.messaging.brokers.kafka.bootstrap-servers[0]=localhost:9092")
.run(
context -> {
assertThat(context).hasNotFailed();
@@ -138,7 +141,8 @@ class MessagingStarterOffContractTest {
.withPropertyValues(
"app.messaging.enabled=true",
"app.messaging.broker=kafka",
"spring.kafka.bootstrap-servers=localhost:9092")
"app.messaging.brokers.kafka.type=kafka",
"app.messaging.brokers.kafka.bootstrap-servers[0]=localhost:9092")
.run(
context -> {
assertThat(context)
@@ -165,7 +169,8 @@ class MessagingStarterOffContractTest {
.withPropertyValues(
"app.messaging.enabled=true",
"app.messaging.broker=kafka",
"spring.kafka.bootstrap-servers=localhost:9092")
"app.messaging.brokers.kafka.type=kafka",
"app.messaging.brokers.kafka.bootstrap-servers[0]=localhost:9092")
.run(
context -> {
assertThat(context)
@@ -203,7 +208,7 @@ class MessagingStarterOffContractTest {
.hasFailed()
.getFailure()
.rootCause()
.hasMessageContaining("spring.kafka.bootstrap-servers"));
.hasMessageContaining("app.messaging.brokers.<name>"));
}
/**
@@ -91,6 +91,34 @@ class ReactiveFacadeTest {
.verify();
}
@Test
void theBlockingPublishRunsOffTheSubscribingThread() {
// given
java.util.concurrent.atomic.AtomicReference<Thread> publishedOn =
new java.util.concurrent.atomic.AtomicReference<>();
MessagePublisher recording =
new MessagePublisher() {
@Override
public <T> CompletionStage<PublishResult> publish(
MessageDestination<T> destination,
MessageEnvelope<T> message,
PublishOptions options) {
publishedOn.set(Thread.currentThread());
return CompletableFuture.completedFuture(confirmed());
}
};
// when
new DefaultReactiveMessagePublisher(recording)
.publish(ORDERS, envelope(), PublishOptions.defaults())
.block();
// then
assertThat(publishedOn.get())
.as("admission and the producer's metadata wait block; an event loop must not pay for it")
.isNotSameAs(Thread.currentThread());
}
private MessagePublisher counting() {
return new MessagePublisher() {
@Override