diff --git a/docs/messaging/configuration-reference.md b/docs/messaging/configuration-reference.md index 0e3c748..2dd4be8 100644 --- a/docs/messaging/configuration-reference.md +++ b/docs/messaging/configuration-reference.md @@ -166,6 +166,12 @@ app: `kafka`뿐이다. 위 예시가 `production: false`인 것은 이 페이지가 그대로 실행되는 fixture이기 때문이며, 실 배포는 셋 다 참이다. +Kafka producer는 이 항목 하나로 만든다. `bootstrap-servers`, `acks`, `enable-idempotence`, +`max-in-flight-requests-per-connection`, `delivery-timeout`이 그대로 client 설정이 된다. +`spring.kafka.bootstrap-servers`는 읽지 않는다. type이 `kafka`인 항목이 정확히 하나가 아니면 기동에 +실패한다. `send()`가 metadata를 기다리며 호출 스레드를 막는 시간(`max.block.ms`)은 기본 publish +timeout(5s)이다. + ## 보안 ```yaml diff --git a/docs/messaging/outbox-inbox.md b/docs/messaging/outbox-inbox.md index d4b68ab..fede443 100644 --- a/docs/messaging/outbox-inbox.md +++ b/docs/messaging/outbox-inbox.md @@ -21,7 +21,7 @@ Flyway version `1000..1999`를 예약해서 애플리케이션의 일반적인 classpath:db/migration/messaging ``` -현재 순서는 `V1001..V1004`(outbox/admin)와 `V1101`(inbox)이다. starter 테스트가 두 JDBC +현재 순서는 `V1001..V1005`(outbox/admin)와 `V1101`(inbox)이다. starter 테스트가 두 JDBC 모듈을 실제 combined classpath로 올려 version 중복과 예약 대역 이탈을 검사한다. 개별 DDL의 의미와 전환 순서는 [migration-guide.md](migration-guide.md)를 따른다. @@ -46,6 +46,11 @@ leaseBatch(100, 30s) -- lease로 다중 relay 인스턴스 안전 → REJECTED → markFailed ``` +row 하나의 실패는 그 row에서 끝난다. envelope로 만들 수 없는 row(예약 header `msg.*`를 담은 row +등)는 `FAILED`가 되고, publisher가 예외를 던지면 결과를 모르므로 `AMBIGUOUS`가 된다. 같은 batch의 +나머지 row는 그대로 publish된다. pass 전체가 실패하면(DB 연결 등) WARN 로그를 남기고 +`messaging.outbox.relay.failed.passes` 카운터를 올린다. + ### 핵심 규칙: ambiguous는 같은 messageId로 재시도 새 id를 발급하면 "전달됐을 수도 있는 메시지"가 "확실히 두 번째인 메시지"가 되어 @@ -61,8 +66,16 @@ status IN ('PENDING','AMBIGUOUS','IN_FLIGHT') AND (lease_expires_at IS NULL OR lease_expires_at <= now) AND next_attempt_at <= now AND attempts < maxAttempts + AND (ordering_key IS NULL + OR 같은 destination·ordering_key의 더 오래된 미완료 row가 없음) ``` +`ordering_key`가 있는 row는 같은 key의 더 오래된 row가 `PENDING`·`AMBIGUOUS`·`IN_FLIGHT`인 동안 +claim되지 않는다. 이 조건이 없으면 두 relay가 같은 key의 두 row를 나눠 가져가거나, AMBIGUOUS로 +재시도되는 row보다 뒤의 row가 먼저 나간다(`OutboxPostgresIT#aNewerRowWaitsWhileAnOlderRowWithTheSameOrderingKeyIsUnfinished`). +`V1005`의 partial index가 이 확인을 받친다. 더 오래된 row가 `FAILED`·`EXHAUSTED`가 되면 뒤의 row는 +진행한다. + `IN_FLIGHT`가 목록에 있는 것이 핵심이다. relay가 publish 도중 죽으면 row는 `IN_FLIGHT`로 남는데, 이를 제외하면 그 메시지는 **영원히** 발행되지 않는다 — outbox가 막으려던 바로 그 실패다. 대신 lease가 만료됐을 때만 회수하므로, 살아 있는 relay가 들고 있는 row는 회수되지 않는다. @@ -107,6 +120,9 @@ message_id만으로 중복 제거하면 같은 event를 소비하는 두 번째 broker의 최대 redelivery window보다 **길어야** 한다. row를 먼저 지우면 늦게 도착한 redelivery가 두 번 처리된다. +outbox·inbox cleanup job은 한 번에 최대 1,000행씩 지우고, 지운 수가 1,000보다 적으면 멈춘다. +repository port에는 개수 제한 없는 삭제 메서드가 없다. + ## CDC 대안 relay는 polling 하나뿐이다. Debezium Event Router 매핑(`DebeziumOutboxEventRouter` 등)은 조립하는 diff --git a/docs/messaging/retry-dlq-redrive.md b/docs/messaging/retry-dlq-redrive.md index 39355e6..9d64b63 100644 --- a/docs/messaging/retry-dlq-redrive.md +++ b/docs/messaging/retry-dlq-redrive.md @@ -49,6 +49,15 @@ full jitter는 `[0, delay]` 균등 분포다. jitter가 없으면 같은 초에 pause-and-seek는 메시지가 로그의 자기 자리를 떠나지 않는다. partition을 멈추고, 기다리고, 같은 offset으로 seek해 재전달한다. 뒤의 메시지도 함께 기다리며 이것이 의도된 동작이다. +`KafkaConsumerRegistrar`가 handler 실패(예외, settle 없이 끝남, timeout)를 다루는 방식: + +- 재전달은 destination `retry`의 backoff 뒤에 한다. 최소 1초. 0초 재시도는 없다. +- 같은 offset의 실패 횟수가 `retry.maxAttempts`에 닿으면 `KafkaQuarantinePublisher`에 쓰고, + 쓰기가 확인되면 offset을 넘긴다. quarantine이 없으면 backoff로 계속 재시도한다(메시지를 버리지 + 않는다). 그래서 `production` destination은 quarantine 없이 만들 수 없다(`KAFKA_QUARANTINE_REQUIRED`). +- handler timeout이 지나도 handler가 끝날 때까지 partition slot을 잡고 있는다. `CompletableFuture`를 + cancel해도 handler는 멈추지 않으므로, slot을 먼저 풀면 같은 offset이 두 번 동시에 돈다. + ## RabbitMQ: delayed retry queue core broker에 per-message delay가 없으므로 **TTL + DLX**로 구현한다. diff --git a/messaging/messaging-inbox-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/inbox/InboxPostgresIT.java b/messaging/messaging-inbox-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/inbox/InboxPostgresIT.java index 367a6f5..899707c 100644 --- a/messaging/messaging-inbox-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/inbox/InboxPostgresIT.java +++ b/messaging/messaging-inbox-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/inbox/InboxPostgresIT.java @@ -151,7 +151,7 @@ class InboxPostgresIT { MessageId purged = MessageId.newId(); inTransaction(() -> repository.reserve(purged, "order-projection", NOW)); - assertThat(repository.purgeProcessedBefore(NOW.plusSeconds(1))).isEqualTo(1); + assertThat(repository.purgeProcessedBefore(NOW.plusSeconds(1), 100)).isEqualTo(1); } /** Reserves and writes inside one Spring transaction, exactly as an idempotent consumer must. */ diff --git a/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/InboxCleanupJob.java b/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/InboxCleanupJob.java index fbeda2e..6baceae 100644 --- a/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/InboxCleanupJob.java +++ b/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/InboxCleanupJob.java @@ -53,10 +53,10 @@ public final class InboxCleanupJob { int removed = 0; for (int batch = 0; batch < maxBatches; batch++) { - int deleted = inbox.purgeProcessedBefore(cutoff); + int deleted = inbox.purgeProcessedBefore(cutoff, DEFAULT_BATCH_SIZE); removed += deleted; // A short batch means the backlog is drained; continuing would just re-scan an empty range. - if (deleted == 0) { + if (deleted < DEFAULT_BATCH_SIZE) { break; } } diff --git a/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/JdbcInboxRepository.java b/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/JdbcInboxRepository.java index 1ac930c..d941d73 100644 --- a/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/JdbcInboxRepository.java +++ b/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/JdbcInboxRepository.java @@ -167,18 +167,4 @@ public final class JdbcInboxRepository implements InboxRepository { "INBOX_PURGE_FAILED", "a bounded inbox purge failed", exception); } } - - @Override - public int purgeProcessedBefore(Instant processedBefore) { - Objects.requireNonNull(processedBefore, "processedBefore must not be null"); - try (Connection connection = dataSource.getConnection(); - PreparedStatement statement = - connection.prepareStatement("DELETE FROM messaging_inbox WHERE processed_at < ?")) { - statement.setTimestamp(1, Timestamp.from(processedBefore)); - return statement.executeUpdate(); - } catch (SQLException exception) { - throw new MessagingConfigurationException( - "INBOX_PURGE_FAILED", "could not purge the inbox", exception); - } - } } diff --git a/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/IdempotentConsumerTest.java b/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/IdempotentConsumerTest.java index 013fe84..b4331a6 100644 --- a/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/IdempotentConsumerTest.java +++ b/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/IdempotentConsumerTest.java @@ -82,7 +82,7 @@ class IdempotentConsumerTest { void retentionPruningRemovesOldRows() { consumer.runOnce(MessageId.newId(), "order-projection", NOW, () -> "done"); - assertThat(inbox.purgeProcessedBefore(NOW.plusSeconds(1))).isEqualTo(1); + assertThat(inbox.purgeProcessedBefore(NOW.plusSeconds(1), 100)).isEqualTo(1); assertThat(inbox.size()).isZero(); } } @@ -104,12 +104,7 @@ final class InMemoryInbox implements InboxRepository { @Override public int purgeProcessedBefore(Instant processedBefore, int limit) { - return Math.min(purgeProcessedBefore(processedBefore), limit); - } - - @Override - public int purgeProcessedBefore(Instant processedBefore) { - int removed = reserved.size(); + int removed = Math.min(reserved.size(), limit); reserved.clear(); return removed; } diff --git a/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/InboxOperationsTest.java b/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/InboxOperationsTest.java index 17409b1..b822834 100644 --- a/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/InboxOperationsTest.java +++ b/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/InboxOperationsTest.java @@ -42,15 +42,13 @@ class InboxOperationsTest { return reserved.contains(messageId.value() + "|" + consumerId); } - @Override - public int purgeProcessedBefore(Instant processedBefore, int limit) { - return Math.min(purgeProcessedBefore(processedBefore), limit); - } + private final List limits = new ArrayList<>(); @Override - public int purgeProcessedBefore(Instant processedBefore) { + public int purgeProcessedBefore(Instant processedBefore, int limit) { cutoffs.add(processedBefore); - return pass < deletions.size() ? deletions.get(pass++) : 0; + limits.add(limit); + return Math.min(pass < deletions.size() ? deletions.get(pass++) : 0, limit); } } @@ -91,14 +89,17 @@ class InboxOperationsTest { } @Test - void cleanupDeletesInBoundedBatches() { + void cleanupAsksForBoundedBatchesAndStopsAtTheFirstShortOne() { + // given InMemoryInbox inbox = new InMemoryInbox(List.of(1000, 500)); - int removed = - new InboxCleanupJob(inbox, policy(Duration.ofDays(7), Duration.ofDays(1)), 10).runOnce(NOW); + // when + new InboxCleanupJob(inbox, policy(Duration.ofDays(7), Duration.ofDays(1)), 10).runOnce(NOW); - assertThat(removed).isEqualTo(1500); - assertThat(inbox.cutoffs).hasSize(3); + // then + assertThat(inbox.limits) + .as("DEFAULT_BATCH_SIZE was declared and never passed to the repository") + .containsExactly(InboxCleanupJob.DEFAULT_BATCH_SIZE, InboxCleanupJob.DEFAULT_BATCH_SIZE); } @Test diff --git a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaConsumerRegistrar.java b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaConsumerRegistrar.java index c285b5c..9bfec58 100644 --- a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaConsumerRegistrar.java +++ b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaConsumerRegistrar.java @@ -66,6 +66,35 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati private final KafkaQuarantinePublisher quarantine; + private final java.time.InstantSource clock; + + /** The shortest wait before a failed delivery is read again, whatever the policy computes. */ + private static final Duration MIN_RETRY_DELAY = Duration.ofSeconds(1); + + private static final dev.caskeleton.messaging.policy.BackoffCalculator BACKOFF = + new dev.caskeleton.messaging.policy.BackoffCalculator(); + + /** A delivery that failed, by where it sits in the log. */ + private record FailedOffset(TopicPartition partition, long offset) {} + + /** + * How many times each offset has failed without being settled. + * + *

Pause-and-seek re-reads the same record, so the envelope's own attempt header never moves; + * the count has to live here. Without it a poison message was retried with no delay and no limit, + * and its partition never advanced again. + */ + private final Map failures = new ConcurrentHashMap<>(); + + /** + * Partitions paused because a record could not be dispatched — the ceiling was reached, the + * runtime was draining, or the pool refused. Poll thread only. + * + *

They were paused and never resumed: an ordered partition handed two records by one poll ran + * the first and then stopped reading for good. + */ + private final Set saturated = new java.util.HashSet<>(); + private final Map committed = new ConcurrentHashMap<>(); /** @@ -127,8 +156,38 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati GracefulShutdownCoordinator shutdown, Duration pollTimeout, KafkaQuarantinePublisher quarantine) { + this( + spec, + consumer, + handlerPool, + shutdown, + pollTimeout, + quarantine, + java.time.InstantSource.system()); + } + + /** A consumer runtime with the clock retry delays are measured on, for tests. */ + KafkaConsumerRegistrar( + TransportConsumerSpec spec, + Consumer consumer, + Executor handlerPool, + GracefulShutdownCoordinator shutdown, + Duration pollTimeout, + KafkaQuarantinePublisher quarantine, + java.time.InstantSource clock) { + this.clock = Objects.requireNonNull(clock, "clock must not be null"); this.quarantine = Objects.requireNonNull(quarantine, "quarantine must not be null"); this.spec = Objects.requireNonNull(spec, "spec must not be null"); + if (spec.profile().production() && quarantine == KafkaQuarantinePublisher.unavailable()) { + // Without somewhere to park a message that keeps failing, its partition retries it forever + // and nothing behind it is ever read. + throw new dev.caskeleton.messaging.api.error.MessagingConfigurationException( + "KAFKA_QUARANTINE_REQUIRED", + "production destination " + + spec.profile().name().value() + + " needs a KafkaQuarantinePublisher; without one a message that exhausts its" + + " attempts stops its partition for good"); + } this.consumer = Objects.requireNonNull(consumer, "consumer must not be null"); this.handlerPool = Objects.requireNonNull(handlerPool, "handlerPool must not be null"); this.shutdown = Objects.requireNonNull(shutdown, "shutdown must not be null"); @@ -169,6 +228,8 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati coordinator.forget(partition); retries.forget(partition); committed.remove(partition); + saturated.remove(partition); + failures.keySet().removeIf(failed -> failed.partition().equals(partition)); }); } @@ -199,6 +260,7 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati syncEpochs(); applySettlements(); applyDueResumes(now); + resumeSaturated(); commitContiguous(); if (!shutdown.isAcceptingWork()) { @@ -251,6 +313,7 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati // Only this partition pauses. Its earliest unsubmitted offset is where it resumes. consumer.pause(Set.of(partition)); consumer.seek(partition, seekBackTo); + saturated.add(partition); } } return dispatched; @@ -283,7 +346,7 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati if (quarantine.quarantine(record, undecodable)) { settlement.acknowledgeAfterQuarantine(); } else { - settlement.requeueAfterFailure(); + settlement.requeueAfterFailure(record, undecodable); } coordinator.release(partition); shutdown.endWork(); @@ -308,37 +371,36 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati handler = spec.sink().apply(delivery).toCompletableFuture(); handler.get(handlerTimeout.toMillis(), java.util.concurrent.TimeUnit.MILLISECONDS); } catch (java.util.concurrent.TimeoutException timedOut) { - // A timed-out handler may still be applying side effects. Cancel it before releasing - // the partition permit; otherwise the same offset can be dispatched again while the - // first handler is still running. + // A timed-out handler may still be applying side effects, and cancelling the + // CompletableFuture does not stop it — it only completes the stage this worker holds, + // which released the permit and let the same offset run again beside the first. So + // the slot stays taken until the handler really finishes, and only then is a delivery + // it left unsettled requeued. java.util.concurrent.CompletableFuture timedOutHandler = handler; - settlement.requeueAfterFailure(); - if (timedOutHandler != null - && !timedOutHandler.cancel(true) - && !timedOutHandler.isDone()) { - releaseImmediately = false; - java.util.concurrent.CompletableFuture releaseAfterCompletion = - timedOutHandler.whenComplete( - (ignored, failure) -> { - coordinator.release(partition); - shutdown.endWork(); - }); - // Keep the continuation strongly referenced through this worker frame. The - // completion action owns the delayed permit/work-count release; its own result is - // intentionally not joined here. - Objects.requireNonNull(releaseAfterCompletion); - } + releaseImmediately = false; + java.util.concurrent.CompletableFuture releaseAfterCompletion = + timedOutHandler.whenComplete( + (ignored, failure) -> { + settlement.requeueAfterFailure( + record, failure == null ? timedOut : failure); + coordinator.release(partition); + shutdown.endWork(); + }); + // Keep the continuation strongly referenced through this worker frame. The + // completion action owns the delayed permit/work-count release; its own result is + // intentionally not joined here. + Objects.requireNonNull(releaseAfterCompletion); } catch (InterruptedException interrupted) { Thread.currentThread().interrupt(); if (handler != null) { handler.cancel(true); } - settlement.requeueAfterFailure(); + settlement.requeueAfterFailure(record, interrupted); } catch (java.util.concurrent.ExecutionException | RuntimeException failure) { // A handler failure is retried, not parked. PARKED completed the offset without any // quarantine or dead-letter write, so a transient database outage silently discarded // every message it touched. - settlement.requeueAfterFailure(); + settlement.requeueAfterFailure(record, failure); } finally { if (releaseImmediately) { coordinator.release(partition); @@ -365,7 +427,10 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati continue; } switch (command.kind()) { - case COMPLETE, PARKED -> offsets.completed(command.partition(), command.offset()); + case COMPLETE, PARKED -> { + offsets.completed(command.partition(), command.offset()); + failures.remove(new FailedOffset(command.partition(), command.offset())); + } case PAUSE_AND_SEEK -> { coordinator.pause(command.partition()); consumer.pause(Set.of(command.partition())); @@ -391,6 +456,22 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati current.forEach( partition -> epochs.computeIfAbsent(partition, key -> epochSequence.incrementAndGet())); epochs.keySet().removeIf(partition -> !current.contains(partition)); + saturated.retainAll(current); + } + + /** Resumes a partition paused for lack of a slot once it has one and nothing else holds it. */ + private void resumeSaturated() { + int ceiling = spec.profile().consumer().maxInFlightPerOrderingUnit(); + saturated.removeIf( + partition -> { + if (coordinator.isPaused(partition) + || retries.isPaused(partition) + || coordinator.inFlight(partition) >= ceiling) { + return false; + } + consumer.resume(Set.of(partition)); + return true; + }); } private void applyDueResumes(Instant now) { @@ -603,16 +684,36 @@ public final class KafkaConsumerRegistrar implements TransportConsumerRegistrati } } - /** Used by the worker when the handler failed or timed out and never settled. */ - private void requeueAfterFailure() { + /** + * Used by the worker when the handler failed or timed out and never settled. + * + *

Retried after the destination's backoff, never sooner than {@code MIN_RETRY_DELAY}; once + * the attempt budget is spent the record is written to quarantine and committed past. When + * there is no quarantine it keeps being retried — the offset may not advance past a message + * that exists nowhere else — which is why a production destination must have one. + */ + private void requeueAfterFailure(ConsumerRecord record, Throwable reason) { + if (terminal.get() != null) { + // The handler settled before it failed or before its late completion; that answer stands. + return; + } + FailedOffset key = new FailedOffset(partition, offset); + int attempt = failures.merge(key, 1, Integer::sum); + dev.caskeleton.messaging.policy.RetryPolicy retry = spec.profile().retry(); + if (attempt >= retry.maxAttempts() && quarantine.quarantine(record, reason)) { + failures.remove(key); + acknowledgeAfterQuarantine(); + return; + } if (terminal.compareAndSet(null, "requeue")) { - enqueueRequeue(Duration.ZERO); + Duration backoff = BACKOFF.delayFor(retry, attempt); + enqueueRequeue(backoff.compareTo(MIN_RETRY_DELAY) < 0 ? MIN_RETRY_DELAY : backoff); } } private dev.caskeleton.messaging.api.settlement.SettlementResult enqueueRequeue( Duration delay) { - retries.pauseUntil(partition, offset, delay, Instant.now()); + retries.pauseUntil(partition, offset, delay, clock.instant()); settlements.enqueue( new KafkaSettlementCommand( partition, offset, KafkaSettlementCommand.Kind.PAUSE_AND_SEEK, epoch)); diff --git a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaQuarantinePublisher.java b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaQuarantinePublisher.java index bdb4365..20489c9 100644 --- a/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaQuarantinePublisher.java +++ b/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaQuarantinePublisher.java @@ -24,8 +24,21 @@ public interface KafkaQuarantinePublisher { */ boolean quarantine(ConsumerRecord record, Throwable reason); - /** The default: nothing is configured, so nothing may be committed past. */ + /** + * The default: nothing is configured, so nothing may be committed past. + * + *

One instance, so a consumer can tell it apart from a configured quarantine and refuse it on + * a production destination. + */ static KafkaQuarantinePublisher unavailable() { - return (record, reason) -> false; + return Unavailable.INSTANCE; + } + + /** Holds the single "not configured" quarantine. */ + final class Unavailable { + + private static final KafkaQuarantinePublisher INSTANCE = (record, reason) -> false; + + private Unavailable() {} } } diff --git a/messaging/messaging-kafka/src/test/java/dev/caskeleton/messaging/kafka/KafkaConsumerRegistrarTest.java b/messaging/messaging-kafka/src/test/java/dev/caskeleton/messaging/kafka/KafkaConsumerRegistrarTest.java index 941abf3..db4e105 100644 --- a/messaging/messaging-kafka/src/test/java/dev/caskeleton/messaging/kafka/KafkaConsumerRegistrarTest.java +++ b/messaging/messaging-kafka/src/test/java/dev/caskeleton/messaging/kafka/KafkaConsumerRegistrarTest.java @@ -4,6 +4,9 @@ import static org.assertj.core.api.Assertions.assertThat; import dev.caskeleton.messaging.api.MessageId; import dev.caskeleton.messaging.api.header.ReservedHeaders; +import dev.caskeleton.messaging.policy.OrderingImpact; +import dev.caskeleton.messaging.policy.RetryMode; +import dev.caskeleton.messaging.policy.RetryPolicy; import dev.caskeleton.messaging.transport.GracefulShutdownCoordinator; import dev.caskeleton.messaging.transport.TransportConsumerSpec; import dev.caskeleton.messaging.transport.TransportDelivery; @@ -154,51 +157,165 @@ class KafkaConsumerRegistrarTest { } @Test - void handlerTimeoutCancelsTheOutstandingHandlerBeforeThePartitionPermitIsReleased() { - CompletableFuture handler = new CompletableFuture<>(); - dev.caskeleton.messaging.policy.DestinationProfile base = KafkaContractHarness.profile(); - dev.caskeleton.messaging.policy.DestinationProfile fastTimeout = - new dev.caskeleton.messaging.policy.DestinationProfile( - base.name(), - base.broker(), - base.kind(), - base.physical(), - base.schema(), - base.deliveryGuarantee(), - base.orderingScope(), - base.externalSideEffectGuarantee(), - base.producer(), - new dev.caskeleton.messaging.policy.ConsumerPolicy( - base.consumer().group(), - base.consumer().concurrency(), - base.consumer().maxInFlightPerOrderingUnit(), - base.consumer().prefetch(), - Duration.ofMillis(5), - base.consumer().manualSettlement()), - base.retry(), - base.deadLetter(), - base.payload(), - base.tier(), - base.production(), - base.keyResolverConfigured(), - base.topologyAutoCreate()); - + void aTimedOutHandlerStillRunningIsNotDispatchedASecondTime() { + // given + Instant start = Instant.now(); + CompletableFuture stillRunning = new CompletableFuture<>(); KafkaConsumerRegistrar registrar = - new KafkaConsumerRegistrar( - new TransportConsumerSpec(fastTimeout, delivery -> handler), - consumer, - handlers, - new GracefulShutdownCoordinator(Duration.ofSeconds(30)), - Duration.ofMillis(1)); + registrar( + profile(Duration.ofMillis(5), RetryPolicy.none(), false), + delivery -> { + handled.add(delivery); + return stillRunning; + }, + KafkaQuarantinePublisher.unavailable(), + () -> start); + assign(); + addRecord(10); + registrar.pollOnce(start); + handlers.runAll(); + // What the broker serves again if the consumer seeks back to the timed-out offset. + addRecord(10); + // when + registrar.pollOnce(start.plusSeconds(60)); + handlers.runAll(); + + // then + assertThat(handled) + .as("cancelling a CompletableFuture does not stop the handler behind it") + .hasSize(1); + } + + @Test + void aTimedOutHandlerIsRedeliveredOnceItFinishesWithoutSettling() { + // given + CompletableFuture stillRunning = new CompletableFuture<>(); + KafkaConsumerRegistrar registrar = + registrar( + profile(Duration.ofMillis(5), RetryPolicy.none(), false), + delivery -> { + handled.add(delivery); + return stillRunning; + }, + KafkaQuarantinePublisher.unavailable(), + () -> NOW); + assign(); + addRecord(10); + registrar.pollOnce(NOW); + handlers.runAll(); + stillRunning.complete(null); + addRecord(10); + + // when + registrar.pollOnce(NOW.plusSeconds(60)); + handlers.runAll(); + + // then + assertThat(handled).hasSize(2); + } + + @Test + void aFailedHandlerWaitsOutItsBackoffBeforeThePartitionIsReadAgain() { + // given + Instant start = Instant.now(); + RetryPolicy threeAttempts = + new RetryPolicy( + RetryMode.PAUSE_PARTITION, + 3, + Duration.ofSeconds(1), + Duration.ofSeconds(10), + 2.0, + false, + OrderingImpact.PRESERVE, + java.util.Set.of(), + java.util.Set.of()); + KafkaConsumerRegistrar registrar = + registrar( + profile(Duration.ofSeconds(1), threeAttempts, false), + this::fail, + KafkaQuarantinePublisher.unavailable(), + () -> start); + assign(); + addRecord(10); + registrar.pollOnce(start); + handlers.runAll(); + + // when + registrar.pollOnce(start.plusMillis(500)); + + // then + assertThat(consumer.paused()) + .as("a failure retried with no delay re-reads the same record as fast as the loop polls") + .contains(PARTITION); + } + + @Test + void aHandlerThatSpendsItsAttemptBudgetIsQuarantinedAndCommittedPast() { + // given + List quarantined = new ArrayList<>(); + KafkaConsumerRegistrar registrar = + registrar( + KafkaContractHarness.profile(), + this::fail, + (record, reason) -> quarantined.add(record.offset()), + () -> NOW); assign(); addRecord(10); registrar.pollOnce(NOW); handlers.runAll(); - assertThat(handler) - .as("a timed-out handler must be cancelled before the same partition can be redelivered") - .isCancelled(); + // when + registrar.pollOnce(NOW); + + // then + assertThat(registrar.committedOffset(PARTITION)) + .as("a poison message retried forever stops its partition for good") + .hasValue(11L); + } + + @Test + void aProductionDestinationWithoutAQuarantineIsRefused() { + // given + dev.caskeleton.messaging.policy.DestinationProfile production = + profile(Duration.ofSeconds(30), RetryPolicy.none(), true); + + // when + Throwable thrown = + org.assertj.core.api.Assertions.catchThrowable( + () -> registrar(production, this::settleImmediately)); + + // then + assertThat(thrown) + .isInstanceOf(dev.caskeleton.messaging.api.error.MessagingConfigurationException.class) + .extracting( + failure -> + ((dev.caskeleton.messaging.api.error.MessagingConfigurationException) failure) + .failure() + .code()) + .isEqualTo("KAFKA_QUARANTINE_REQUIRED"); + } + + @Test + void aPartitionPausedAtItsCeilingResumesOnceItsDeliveryFinishes() { + // given + KafkaConsumerRegistrar registrar = registrar(this::settleImmediately); + assign(); + addRecord(10); + addRecord(11); + registrar.pollOnce(NOW); + handlers.runAll(); + // The second record was returned by the first poll and handed back with a seek. + addRecord(11); + + // when + registrar.pollOnce(NOW); + handlers.runAll(); + + // then + assertThat(handled) + .as("pausing a full partition is backpressure only if something resumes it") + .hasSize(2); } @Test @@ -495,6 +612,68 @@ class KafkaConsumerRegistrarTest { Duration.ofMillis(1)); } + private KafkaConsumerRegistrar registrar( + dev.caskeleton.messaging.policy.DestinationProfile profile, + java.util.function.Function> sink) { + return new KafkaConsumerRegistrar( + new TransportConsumerSpec(profile, sink), + consumer, + handlers, + new GracefulShutdownCoordinator(Duration.ofSeconds(30)), + Duration.ofMillis(1)); + } + + private KafkaConsumerRegistrar registrar( + dev.caskeleton.messaging.policy.DestinationProfile profile, + java.util.function.Function> sink, + KafkaQuarantinePublisher quarantine, + java.time.InstantSource clock) { + return new KafkaConsumerRegistrar( + new TransportConsumerSpec(profile, sink), + consumer, + handlers, + new GracefulShutdownCoordinator(Duration.ofSeconds(30)), + Duration.ofMillis(1), + quarantine, + clock); + } + + /** The harness profile with the three settings these tests vary. */ + private static dev.caskeleton.messaging.policy.DestinationProfile profile( + Duration handlerTimeout, RetryPolicy retry, boolean production) { + dev.caskeleton.messaging.policy.DestinationProfile base = KafkaContractHarness.profile(); + return new dev.caskeleton.messaging.policy.DestinationProfile( + base.name(), + base.broker(), + base.kind(), + base.physical(), + base.schema(), + base.deliveryGuarantee(), + base.orderingScope(), + base.externalSideEffectGuarantee(), + base.producer(), + new dev.caskeleton.messaging.policy.ConsumerPolicy( + base.consumer().group(), + base.consumer().concurrency(), + base.consumer().maxInFlightPerOrderingUnit(), + base.consumer().prefetch(), + handlerTimeout, + base.consumer().manualSettlement()), + retry, + base.deadLetter(), + base.payload(), + base.tier(), + production, + base.keyResolverConfigured(), + base.topologyAutoCreate()); + } + + /** A handler that fails every delivery it is given. */ + private CompletionStage fail(TransportDelivery delivery) { + handled.add(delivery); + throw new IllegalStateException("the downstream this handler writes to is down"); + } + /** Acknowledges inside the handler, as a successful M1 delivery does. */ private CompletionStage settleImmediately(TransportDelivery delivery) { handled.add(delivery); diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/outbox/OutboxPostgresIT.java b/messaging/messaging-outbox-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/outbox/OutboxPostgresIT.java index 0e2d870..66fba44 100644 --- a/messaging/messaging-outbox-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/outbox/OutboxPostgresIT.java +++ b/messaging/messaging-outbox-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/outbox/OutboxPostgresIT.java @@ -205,7 +205,7 @@ class OutboxPostgresIT { appendInTransaction(pending); repository.markPublishedUnfenced(published.messageId(), NOW); - int removed = repository.purgePublishedBefore(NOW.plusSeconds(1)); + int removed = repository.purgePublishedBefore(NOW.plusSeconds(1), 100); assertThat(removed).isEqualTo(1); assertThat(repository.find(pending.messageId())).isPresent(); @@ -217,7 +217,35 @@ class OutboxPostgresIT { OutboxCanonicalMetadata.none().withRoutingKeys("acct-1", "acct-1")); } + @Test + void aNewerRowWaitsWhileAnOlderRowWithTheSameOrderingKeyIsUnfinished() { + // given + OutboxRecord older = record(ordered("customer-42"), NOW.minusSeconds(60)); + OutboxRecord newer = record(ordered("customer-42"), NOW.minusSeconds(30)); + OutboxRecord otherKey = record(ordered("customer-7"), NOW.minusSeconds(30)); + appendInTransaction(older); + appendInTransaction(newer); + appendInTransaction(otherKey); + + // when + List claimed = repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW); + + // then + assertThat(claimed) + .extracting(OutboxLease::messageId) + .as("a second relay, or a retry of the older row, would put the newer one first") + .containsExactlyInAnyOrder(older.messageId(), otherKey.messageId()); + } + + private static OutboxCanonicalMetadata ordered(String key) { + return OutboxCanonicalMetadata.none().withRoutingKeys(key, key); + } + private static OutboxRecord record(OutboxCanonicalMetadata metadata) { + return record(metadata, NOW.minusSeconds(60)); + } + + private static OutboxRecord record(OutboxCanonicalMetadata metadata, Instant createdAt) { return new OutboxRecord( MessageId.newId(), new DestinationName("order-events"), @@ -227,7 +255,7 @@ class OutboxPostgresIT { "{\"orderId\":\"o-1\"}".getBytes(StandardCharsets.UTF_8), Map.of(), metadata, - NOW.minusSeconds(60), + createdAt, OutboxStatus.PENDING, 0, Optional.empty(), @@ -361,8 +389,8 @@ class OutboxPostgresIT { @Test void twoRelaysClaimingConcurrentlyGetDisjointRowsAndDistinctTokens() { - OutboxRecord one = record(); - OutboxRecord two = record(); + OutboxRecord one = record(OutboxCanonicalMetadata.none().withRoutingKeys("acct-1", "acct-1")); + OutboxRecord two = record(OutboxCanonicalMetadata.none().withRoutingKeys("acct-2", "acct-2")); appendInTransaction(one); appendInTransaction(two); diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcOutboxRepository.java b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcOutboxRepository.java index 44a986a..13b87a3 100644 --- a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcOutboxRepository.java +++ b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcOutboxRepository.java @@ -113,17 +113,29 @@ public final class JdbcOutboxRepository implements OutboxRepository { private static final String CLAIM = """ WITH claimable AS ( - SELECT message_id - FROM messaging_outbox - WHERE status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT') - AND (lease_expires_at IS NULL OR lease_expires_at <= ?) + SELECT m.message_id + FROM messaging_outbox m + WHERE m.status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT') + AND (m.lease_expires_at IS NULL OR m.lease_expires_at <= ?) -- The retry clock lives in the row, not in the relay's memory. Without these two -- predicates an AMBIGUOUS row became claimable again on the very next pass, so a -- broker outage meant the whole backlog was republished every poll interval and the -- configured attempt budget was a number nothing consulted. - AND (next_attempt_at IS NULL OR next_attempt_at <= ?) - AND attempts < ? - ORDER BY created_at + AND (m.next_attempt_at IS NULL OR m.next_attempt_at <= ?) + AND m.attempts < ? + -- A row with an ordering key waits while an older row with the same key is still + -- unfinished. SKIP LOCKED let a second relay take the newer row while the first still + -- held the older one, and an AMBIGUOUS older row was retried after newer ones had + -- already gone out. md5 matches the partial index, which cannot hold a key that long. + AND (m.ordering_key IS NULL OR NOT EXISTS ( + SELECT 1 + FROM messaging_outbox older + WHERE older.destination = m.destination + AND md5(older.ordering_key) = md5(m.ordering_key) + AND older.ordering_key = m.ordering_key + AND older.status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT') + AND (older.created_at, older.message_id) < (m.created_at, m.message_id))) + ORDER BY m.created_at, m.message_id LIMIT ? FOR UPDATE SKIP LOCKED ) @@ -514,20 +526,6 @@ public final class JdbcOutboxRepository implements OutboxRepository { }); } - @Override - public int purgePublishedBefore(Instant publishedBefore) { - Objects.requireNonNull(publishedBefore, "publishedBefore must not be null"); - return withConnection( - connection -> { - try (PreparedStatement statement = - connection.prepareStatement( - "DELETE FROM messaging_outbox WHERE status = 'PUBLISHED' AND published_at < ?")) { - statement.setTimestamp(1, Timestamp.from(publishedBefore)); - return statement.executeUpdate(); - } - }); - } - private void update(String sql, MessageId messageId, Instant now, String failureCode) { withConnection( connection -> { diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxCleanupJob.java b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxCleanupJob.java index 2bed4fc..5d067bd 100644 --- a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxCleanupJob.java +++ b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxCleanupJob.java @@ -15,6 +15,9 @@ import java.util.Objects; */ public final class OutboxCleanupJob { + /** How many rows one delete statement removes before yielding. */ + public static final int DEFAULT_BATCH_SIZE = 1_000; + private final OutboxRepository outbox; private final OutboxProperties properties; private final int maxBatches; @@ -47,9 +50,10 @@ public final class OutboxCleanupJob { int removed = 0; for (int batch = 0; batch < maxBatches; batch++) { - int deleted = outbox.purgePublishedBefore(cutoff); + int deleted = outbox.purgePublishedBefore(cutoff, DEFAULT_BATCH_SIZE); removed += deleted; - if (deleted == 0) { + // A short batch means the backlog is drained; asking again would re-scan an empty range. + if (deleted < DEFAULT_BATCH_SIZE) { break; } } diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelay.java b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelay.java index 64decb8..52350c5 100644 --- a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelay.java +++ b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelay.java @@ -1,9 +1,16 @@ package dev.caskeleton.messaging.outbox; +import dev.caskeleton.messaging.api.MessageEnvelope; import dev.caskeleton.messaging.api.destination.MessageDestination; +import dev.caskeleton.messaging.api.error.FailureCategory; +import dev.caskeleton.messaging.api.error.FailureDescriptor; +import dev.caskeleton.messaging.api.error.MessagingException; import dev.caskeleton.messaging.api.publish.MessagePublisher; +import dev.caskeleton.messaging.api.publish.PublishCompletion; +import dev.caskeleton.messaging.api.publish.PublishEvidence; import dev.caskeleton.messaging.api.publish.PublishOptions; import dev.caskeleton.messaging.api.publish.PublishResult; +import dev.caskeleton.messaging.api.publish.RoutingOutcome; import dev.caskeleton.messaging.reliability.OutboxLease; import dev.caskeleton.messaging.reliability.OutboxRecord; import dev.caskeleton.messaging.reliability.OutboxRepository; @@ -13,6 +20,7 @@ import java.time.Duration; import java.time.Instant; import java.util.List; import java.util.Objects; +import java.util.Optional; /** * Publishes outbox rows, treating an unknown outcome as retryable rather than final. @@ -165,7 +173,22 @@ public final class OutboxRelay { for (OutboxLease lease : batch) { OutboxRecord record = lease.record(); - PublishResult result = publish(record); + MessageEnvelope envelope; + try { + envelope = envelopeFactory.toEnvelope(record); + } catch (RuntimeException unpublishable) { + // A row that cannot become an envelope never will. Thrown out of the pass, it took the + // rest of the batch with it: those rows stayed IN_FLIGHT until their lease expired, and the + // same row was claimed and thrown again on the next pass, forever. + if (repository.markFailed(lease, failureCode(unpublishable), now) + == OutboxTransitionResult.APPLIED) { + failed++; + } else { + stale++; + } + continue; + } + PublishResult result = publish(record, envelope); switch (result.completion()) { case CONFIRMED -> { if (repository.markPublished(lease, now) == OutboxTransitionResult.APPLIED) { @@ -220,12 +243,35 @@ public final class OutboxRelay { return new OutboxRelayReport(batch.size(), published, ambiguous, failed, stale, exhausted); } - private PublishResult publish(OutboxRecord record) { + private PublishResult publish(OutboxRecord record, MessageEnvelope envelope) { MessageDestination destination = new MessageDestination<>(record.destination(), record.messageType(), EncodedMessage.class); - return publisher - .publish(destination, envelopeFactory.toEnvelope(record), PublishOptions.defaults()) - .toCompletableFuture() - .join(); + try { + return publisher + .publish(destination, envelope, PublishOptions.defaults()) + .toCompletableFuture() + .join(); + } catch (RuntimeException unknown) { + // An exception says nothing about whether the broker took the message, so the row is retried + // under the same id like any other unknown outcome — not failed, not left IN_FLIGHT. + return new PublishResult( + PublishCompletion.AMBIGUOUS, + PublishEvidence.ambiguous(), + RoutingOutcome.UNKNOWN, + Optional.empty(), + 1, + Duration.ZERO, + Optional.of( + FailureDescriptor.of( + FailureCategory.AMBIGUOUS, + "PUBLISH_OUTCOME_UNKNOWN", + "the publisher threw " + unknown.getClass().getSimpleName()))); + } + } + + private static String failureCode(RuntimeException failure) { + return failure instanceof MessagingException known + ? known.failure().code() + : "OUTBOX_ROW_UNPUBLISHABLE"; } } diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelayWorker.java b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelayWorker.java index 5a3ac4c..5de58c6 100644 --- a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelayWorker.java +++ b/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelayWorker.java @@ -34,6 +34,8 @@ public final class OutboxRelayWorker implements AutoCloseable { /** How long {@link #close()} waits for a running pass when no deadline is given. */ public static final Duration DEFAULT_DRAIN_DEADLINE = Duration.ofSeconds(30); + private static final System.Logger LOG = System.getLogger(OutboxRelayWorker.class.getName()); + private final OutboxRelay relay; private final OutboxRetryScheduler scheduler; @@ -184,9 +186,14 @@ public final class OutboxRelayWorker implements AutoCloseable { } catch (RuntimeException passFailed) { // A failed pass must not stop the loop: the scheduled task's own exception would cancel every // future pass, turning one broker error into a relay that never runs again. The failure is - // counted and the next pass backs off as if nothing was published, which is true. - failedPasses.incrementAndGet(); + // counted and the next pass backs off as if nothing was published, which is true. It is also + // logged: counted silently, a relay failing every pass looked exactly like an idle one. + long failures = failedPasses.incrementAndGet(); unproductive = unproductivePasses.incrementAndGet(); + LOG.log( + System.Logger.Level.WARNING, + "outbox relay pass failed (" + failures + " failed so far); retrying with backoff", + passFailed); } completedPasses.incrementAndGet(); schedule(scheduler.backoff(unproductive)); diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V1005__messaging_outbox_ordering_key_index.sql b/messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V1005__messaging_outbox_ordering_key_index.sql new file mode 100644 index 0000000..f62af01 --- /dev/null +++ b/messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V1005__messaging_outbox_ordering_key_index.sql @@ -0,0 +1,9 @@ +-- The claim now holds back a row while an older, unfinished row shares its destination and +-- ordering key. That check runs once per candidate row, so it needs an index or every claim scans +-- the backlog once per row it considers. +-- +-- md5 of the key rather than the key: ordering_key is VARCHAR(1024), and a long multi-byte key +-- would exceed the btree entry limit and fail the business transaction that appends the row. +CREATE INDEX ix_messaging_outbox_unfinished_ordering_key + ON messaging_outbox (destination, md5(ordering_key), created_at) + WHERE ordering_key IS NOT NULL AND status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT'); diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxOperationsTest.java b/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxOperationsTest.java index f7df5cb..7eb3f0a 100644 --- a/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxOperationsTest.java +++ b/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxOperationsTest.java @@ -96,15 +96,13 @@ class OutboxOperationsTest { throw new UnsupportedOperationException(); } - @Override - public int purgePublishedBefore(Instant publishedBefore, int limit) { - return Math.min(purgePublishedBefore(publishedBefore), limit); - } + private final List limits = new ArrayList<>(); @Override - public int purgePublishedBefore(Instant publishedBefore) { + public int purgePublishedBefore(Instant publishedBefore, int limit) { cutoffs.add(publishedBefore); - return pass < deletions.size() ? deletions.get(pass++) : 0; + limits.add(limit); + return Math.min(pass < deletions.size() ? deletions.get(pass++) : 0, limit); } } @@ -130,13 +128,20 @@ class OutboxOperationsTest { } @Test - void cleanupDeletesInBoundedBatchesRatherThanOneLongStatement() { + void cleanupAsksForBoundedBatchesAndStopsAtTheFirstShortOne() { + // given RecordingRepository repository = new RecordingRepository(List.of(1000, 1000, 250)); - int removed = new OutboxCleanupJob(repository, OutboxProperties.defaults(), 10).runOnce(NOW); + // when + new OutboxCleanupJob(repository, OutboxProperties.defaults(), 10).runOnce(NOW); - assertThat(removed).isEqualTo(2250); - assertThat(repository.cutoffs).hasSize(4); + // then + assertThat(repository.limits) + .as("one DELETE over the whole backlog holds its locks for as long as the backlog is big") + .containsExactly( + OutboxCleanupJob.DEFAULT_BATCH_SIZE, + OutboxCleanupJob.DEFAULT_BATCH_SIZE, + OutboxCleanupJob.DEFAULT_BATCH_SIZE); } @Test diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayTest.java b/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayTest.java index 3a8256a..8749e0b 100644 --- a/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayTest.java +++ b/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayTest.java @@ -86,6 +86,49 @@ class OutboxRelayTest { .isEqualTo(OutboxStatus.PUBLISHED); } + @Test + void aRowThatCannotBecomeAnEnvelopeFailsWhileTheRestOfItsBatchPublishes() { + // given + OutboxRecord poison = OutboxFixtures.withHeaders(Map.of("msg.id", "forged")); + OutboxRecord healthy = OutboxFixtures.pending(); + repository.append(poison); + repository.append(healthy); + + // when + relay(RecordingOutboxPublisher.confirming()).runOnce(NOW); + + // then + assertThat(List.of(poison, healthy)) + .extracting(row -> repository.find(row.messageId()).orElseThrow().status()) + .as("one bad row threw out of the pass and held the whole batch until its lease expired") + .containsExactly(OutboxStatus.FAILED, OutboxStatus.PUBLISHED); + } + + @Test + void aPublisherThatThrowsLeavesTheRowAmbiguousRatherThanStrandingTheBatch() { + // given + OutboxRecord first = OutboxFixtures.pending(); + repository.append(first); + MessagePublisher throwing = + new MessagePublisher() { + @Override + public CompletionStage publish( + MessageDestination destination, + MessageEnvelope message, + PublishOptions options) { + throw new IllegalStateException("the broker connection is gone"); + } + }; + + // when + relay(throwing).runOnce(NOW); + + // then + assertThat(repository.find(first.messageId()).orElseThrow().status()) + .as("an exception says nothing about whether the broker took the message") + .isEqualTo(OutboxStatus.AMBIGUOUS); + } + @Test void anAmbiguousPublishStaysRetryableRatherThanFailing() { OutboxRecord record = OutboxFixtures.pending(); @@ -263,6 +306,10 @@ final class OutboxFixtures { private OutboxFixtures() {} static OutboxRecord pending() { + return withHeaders(Map.of()); + } + + static OutboxRecord withHeaders(Map headers) { return new OutboxRecord( MessageId.newId(), new DestinationName("order-events"), @@ -270,7 +317,7 @@ final class OutboxFixtures { new SchemaVersion(1), ContentType.JSON, "{\"orderId\":\"o-1\"}".getBytes(StandardCharsets.UTF_8), - Map.of(), + headers, dev.caskeleton.messaging.reliability.OutboxCanonicalMetadata.none(), Instant.parse("2026-08-10T09:14:00Z"), OutboxStatus.PENDING, @@ -455,14 +502,9 @@ final class InMemoryOutboxRepository implements OutboxRepository { @Override public int purgePublishedBefore(Instant publishedBefore, int limit) { - return Math.min(purgePublishedBefore(publishedBefore), limit); - } - - @Override - public int purgePublishedBefore(Instant publishedBefore) { int before = rows.size(); rows.values().removeIf(record -> record.status() == OutboxStatus.PUBLISHED); - return before - rows.size(); + return Math.min(before - rows.size(), limit); } private void transition(MessageId messageId, OutboxStatus status, Optional failureCode) { diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayWorkerTest.java b/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayWorkerTest.java index cadc679..9f71564 100644 --- a/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayWorkerTest.java +++ b/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayWorkerTest.java @@ -79,8 +79,7 @@ class OutboxRelayWorkerTest { @Test @DisplayName("a pass that throws does not stop every pass after it") void aFailedPassDoesNotEndTheLoop() throws InterruptedException { - repository.append(OutboxFixtures.pending()); - OutboxRelayWorker worker = worker(new ThrowingPublisher()); + OutboxRelayWorker worker = worker(unreachableDatabase(), new ThrowingPublisher()); worker.start(); try { @@ -99,9 +98,25 @@ class OutboxRelayWorkerTest { } private OutboxRelayWorker worker(MessagePublisher publisher) { + return worker(repository, publisher); + } + + /** An outbox whose every call fails, as it does while its database is down. */ + private static dev.caskeleton.messaging.reliability.OutboxRepository unreachableDatabase() { + return (dev.caskeleton.messaging.reliability.OutboxRepository) + java.lang.reflect.Proxy.newProxyInstance( + OutboxRelayWorkerTest.class.getClassLoader(), + new Class[] {dev.caskeleton.messaging.reliability.OutboxRepository.class}, + (proxy, method, arguments) -> { + throw new IllegalStateException("the outbox database is unreachable"); + }); + } + + private static OutboxRelayWorker worker( + dev.caskeleton.messaging.reliability.OutboxRepository outbox, MessagePublisher publisher) { OutboxRelay relay = new OutboxRelay( - repository, + outbox, publisher, new OutboxEnvelopeFactory(new ProducerId("order-api")), 100, diff --git a/messaging/messaging-policy/src/main/java/dev/caskeleton/messaging/policy/InFlightLimiter.java b/messaging/messaging-policy/src/main/java/dev/caskeleton/messaging/policy/InFlightLimiter.java index 30ef288..e15b7e6 100644 --- a/messaging/messaging-policy/src/main/java/dev/caskeleton/messaging/policy/InFlightLimiter.java +++ b/messaging/messaging-policy/src/main/java/dev/caskeleton/messaging/policy/InFlightLimiter.java @@ -50,7 +50,20 @@ public final class InFlightLimiter { * @throws InterruptedException when the waiting thread is interrupted */ public boolean tryAcquire() throws InterruptedException { - return permits.tryAcquire(maxWait.toNanos(), TimeUnit.NANOSECONDS); + return tryAcquire(maxWait); + } + + /** + * Tries to acquire one permit, waiting no longer than the configured bound or the given budget. + * + * @param budget what the caller has left to spend on the wait + * @return true when a permit was acquired + * @throws InterruptedException when the waiting thread is interrupted + */ + public boolean tryAcquire(Duration budget) throws InterruptedException { + Objects.requireNonNull(budget, "budget must not be null"); + long wait = Math.max(0, Math.min(maxWait.toNanos(), budget.toNanos())); + return permits.tryAcquire(wait, TimeUnit.NANOSECONDS); } /** Returns one permit to the pool. */ diff --git a/messaging/messaging-policy/src/main/java/dev/caskeleton/messaging/policy/MessagingAdmissionController.java b/messaging/messaging-policy/src/main/java/dev/caskeleton/messaging/policy/MessagingAdmissionController.java index 4b2cc98..118f998 100644 --- a/messaging/messaging-policy/src/main/java/dev/caskeleton/messaging/policy/MessagingAdmissionController.java +++ b/messaging/messaging-policy/src/main/java/dev/caskeleton/messaging/policy/MessagingAdmissionController.java @@ -73,7 +73,24 @@ public final class MessagingAdmissionController { * @throws MessageBackpressureException when shutting down or no permit became available */ public void admit(String destination, int payloadBytes) { + admit(destination, payloadBytes, java.time.Duration.ofNanos(Long.MAX_VALUE)); + } + + /** + * Admits one publish, waiting for a permit no longer than the caller has left. + * + *

The limiter's own bound was the only one: a publish with a 100ms deadline could block its + * caller for the limiter's full wait before the deadline was even consulted. + * + * @param destination the logical destination name + * @param payloadBytes the encoded payload size + * @param budget the most this call may wait for a permit + * @throws MessageTooLargeException when the payload exceeds the destination limit + * @throws MessageBackpressureException when shutting down or no permit became available + */ + public void admit(String destination, int payloadBytes, java.time.Duration budget) { Objects.requireNonNull(destination, "destination must not be null"); + Objects.requireNonNull(budget, "budget must not be null"); payloadGuard.checkPayload(destination, payloadBytes); if (!acceptingNewWork) { @@ -91,7 +108,7 @@ public final class MessagingAdmissionController { boolean acquired; try { - acquired = limiter.tryAcquire(); + acquired = limiter.tryAcquire(budget); } catch (InterruptedException interrupted) { // The destination slot was taken a moment ago and no publish will use it, so it goes back // here: a slot leaked per interruption shrinks the destination's ceiling until it is zero. diff --git a/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/InboxRepository.java b/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/InboxRepository.java index a438805..55846b4 100644 --- a/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/InboxRepository.java +++ b/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/InboxRepository.java @@ -34,19 +34,11 @@ public interface InboxRepository { boolean isProcessed(MessageId messageId, String consumerId); /** - * Deletes records older than a cutoff. + * Deletes at most {@code limit} processed rows older than the cutoff. * *

Retention must outlive the broker's maximum redelivery window, otherwise a late redelivery * arrives after its inbox row was pruned and is processed a second time. * - * @param processedBefore the retention cutoff - * @return how many rows were removed - */ - int purgeProcessedBefore(Instant processedBefore); - - /** - * Deletes at most {@code limit} processed rows older than the cutoff. - * * @return how many rows were deleted; fewer than {@code limit} means the sweep is finished */ int purgeProcessedBefore(Instant processedBefore, int limit); diff --git a/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxRepository.java b/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxRepository.java index 6552a3b..8197a2c 100644 --- a/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxRepository.java +++ b/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxRepository.java @@ -94,22 +94,12 @@ public interface OutboxRepository { */ Optional find(MessageId messageId); - /** - * Deletes records published before a cutoff. - * - * @param publishedBefore the retention cutoff - * @return how many rows were removed - */ - int purgePublishedBefore(Instant publishedBefore); - /** * Deletes at most {@code limit} published rows older than the cutoff. * - *

The unbounded version deletes everything before the cutoff in one statement. On a table that - * has been accumulating published rows since the last sweep that is a single long transaction - * holding locks and generating WAL in proportion to the backlog, which shows up as the relay and - * the business writes stalling behind retention. The cleanup jobs describe themselves as bounded - * by batch size; this is the parameter that makes that true. + *

There is no unbounded version. Deleting everything before the cutoff in one statement is a + * single long transaction holding locks and generating WAL in proportion to the backlog, which + * shows up as the relay and the business writes stalling behind retention. * * @return how many rows were deleted; fewer than {@code limit} means the sweep is finished */ diff --git a/messaging/messaging-runtime-core/src/main/java/dev/caskeleton/messaging/runtime/DefaultMessagePublisher.java b/messaging/messaging-runtime-core/src/main/java/dev/caskeleton/messaging/runtime/DefaultMessagePublisher.java index fe59a3e..46c7546 100644 --- a/messaging/messaging-runtime-core/src/main/java/dev/caskeleton/messaging/runtime/DefaultMessagePublisher.java +++ b/messaging/messaging-runtime-core/src/main/java/dev/caskeleton/messaging/runtime/DefaultMessagePublisher.java @@ -224,7 +224,16 @@ public final class DefaultMessagePublisher implements MessagePublisher, EncodedM startedAt); } - admission.admit(destination.name().value(), encoded.payload().bytes().length); + // Refusals are results, not exceptions. They were thrown out of publish() on the caller's + // thread, so a CompletionStage caller — the outbox relay, the reactive facade — saw neither a + // REJECTED result nor a failed stage. The permit wait is bounded by what is left of the + // caller's deadline, not by the limiter's own bound. + try { + admission.admit(destination.name().value(), encoded.payload().bytes().length, remaining); + } catch (dev.caskeleton.messaging.api.error.MessageTooLargeException + | dev.caskeleton.messaging.api.error.MessageBackpressureException refused) { + return refused(refused.failure(), startedAt); + } MessagingRuntimeLease lease = null; try { lease = runtimes.acquire(profile.broker()); @@ -316,6 +325,12 @@ public final class DefaultMessagePublisher implements MessagePublisher, EncodedM } private CompletionStage rejected(String code, String reason, long startedAt) { + return refused( + new FailureDescriptor(FailureCategory.CONFIGURATION, code, false, reason, Optional.empty()), + startedAt); + } + + private CompletionStage refused(FailureDescriptor failure, long startedAt) { return CompletableFuture.completedFuture( new PublishResult( PublishCompletion.REJECTED, @@ -324,9 +339,7 @@ public final class DefaultMessagePublisher implements MessagePublisher, EncodedM Optional.empty(), 1, elapsedSince(startedAt), - Optional.of( - new FailureDescriptor( - FailureCategory.CONFIGURATION, code, false, reason, Optional.empty())))); + Optional.of(failure))); } private PublishResult ambiguousResult(String code, String reason, long startedAt) { diff --git a/messaging/messaging-runtime-core/src/test/java/dev/caskeleton/messaging/runtime/DefaultMessagePublisherTest.java b/messaging/messaging-runtime-core/src/test/java/dev/caskeleton/messaging/runtime/DefaultMessagePublisherTest.java index 5302ed6..4ac4854 100644 --- a/messaging/messaging-runtime-core/src/test/java/dev/caskeleton/messaging/runtime/DefaultMessagePublisherTest.java +++ b/messaging/messaging-runtime-core/src/test/java/dev/caskeleton/messaging/runtime/DefaultMessagePublisherTest.java @@ -312,6 +312,89 @@ class DefaultMessagePublisherTest { assertThat(admission.inFlight()).isZero(); } + @Test + void anOversizedPayloadIsRejectedAsNotRetryableInsteadOfThrown() { + // given + DefaultMessagePublisher publisher = + new DefaultMessagePublisher( + registry(), + allowOrders(), + new SingleCodecRegistry(), + new MessagingAdmissionController( + new PayloadLimitGuard(new PayloadPolicy(4, 4)), + new InFlightLimiter(4, Duration.ofSeconds(1))), + new FixedRuntimeRegistry(transport), + transport); + + // when + CompletableFuture result = + CompletableFuture.supplyAsync( + () -> + publisher + .publish( + new MessageDestination<>( + ORDERS, new MessageType("order.created"), String.class), + envelope(), + PublishOptions.defaults()) + .toCompletableFuture() + .join()); + + // then + assertThat(result) + .succeedsWithin(Duration.ofSeconds(5)) + .extracting( + PublishResult::completion, outcome -> outcome.failure().orElseThrow().retryable()) + .containsExactly(PublishCompletion.REJECTED, false); + } + + @Test + void aSaturatedPermitPoolRejectsAsRetryableWithinTheCallersDeadline() { + // given + DestinationName audit = new DestinationName("audit-events"); + DefaultMessagePublisher publisher = + new DefaultMessagePublisher( + DestinationProfileRegistry.of( + Map.of( + ORDERS, + RuntimeFixtures.profile(ORDERS), + audit, + RuntimeFixtures.profile(audit))), + new DestinationAccessPolicy( + java.util.Set.of(ORDERS, audit), java.util.Set.of(), java.util.Set.of()), + new SingleCodecRegistry(), + new MessagingAdmissionController( + new PayloadLimitGuard(PayloadPolicy.defaults()), + new InFlightLimiter(1, Duration.ofMinutes(1))), + new FixedRuntimeRegistry(transport), + transport); + transport.stall(); + publisher.publish( + new MessageDestination<>(ORDERS, new MessageType("order.created"), String.class), + envelope(), + withTimeout(Duration.ofMinutes(1))); + + // when + CompletableFuture second = + CompletableFuture.supplyAsync( + () -> + publisher + .publish( + new MessageDestination<>( + audit, new MessageType("order.created"), String.class), + envelope(), + withTimeout(Duration.ofMillis(100))) + .toCompletableFuture() + .join()); + + // then + assertThat(second) + .as("the permit wait was the limiter's own minute, not what was left of 100ms") + .succeedsWithin(Duration.ofSeconds(5)) + .extracting( + PublishResult::completion, outcome -> outcome.failure().orElseThrow().retryable()) + .containsExactly(PublishCompletion.REJECTED, true); + } + @Test @DisplayName("an unavailable runtime releases the permit it took") void anUnavailableRuntimeReleasesThePermit() { diff --git a/messaging/messaging-spring-boot-starter-kafka/src/main/java/dev/caskeleton/messaging/autoconfigure/KafkaMessagingAutoConfiguration.java b/messaging/messaging-spring-boot-starter-kafka/src/main/java/dev/caskeleton/messaging/autoconfigure/KafkaMessagingAutoConfiguration.java index e4eea40..49a1ccb 100644 --- a/messaging/messaging-spring-boot-starter-kafka/src/main/java/dev/caskeleton/messaging/autoconfigure/KafkaMessagingAutoConfiguration.java +++ b/messaging/messaging-spring-boot-starter-kafka/src/main/java/dev/caskeleton/messaging/autoconfigure/KafkaMessagingAutoConfiguration.java @@ -121,15 +121,14 @@ public class KafkaMessagingAutoConfiguration { /** * The producer this transport publishes through (MSG-INT-003). * - *

Built from Spring Boot's own {@code spring.kafka.bootstrap-servers} rather than from a - * second description on {@code app.messaging.*}. One broker described twice is the defect this - * repository has already paid for in {@code app.jpa-platform.datasource.*}, and the - * operator-facing property for "where is Kafka" is the one they already know. + *

Built from the one {@code app.messaging.brokers.} entry of type {@code kafka}. It used + * to read {@code spring.kafka.bootstrap-servers} while the compiler required and validated the + * entry's own {@code bootstrap-servers}, so one broker was described twice and only one of the + * two descriptions reached the client. * *

Byte-array serializers on purpose: the platform's codec registry decides the wire format, * and a serializer configured here would be a second encoder disagreeing with it. * - * @param environment the resolved environment * @param compiled the broker and security profiles compiled from {@code app.messaging} * @param declaredSecurity the security profiles the application declared as beans * @param securityConfigurer the Kafka security configurer, present only with a credential source @@ -138,45 +137,75 @@ public class KafkaMessagingAutoConfiguration { @Bean(destroyMethod = "close") @ConditionalOnMissingBean public org.apache.kafka.clients.producer.Producer messagingKafkaProducer( - org.springframework.core.env.Environment environment, CompiledMessagingConfiguration compiled, ObjectProvider declaredSecurity, ObjectProvider securityConfigurer) { return new org.apache.kafka.clients.producer.KafkaProducer<>( - producerConfig(environment, compiled, declaredSecurity, securityConfigurer)); + producerConfig(compiled, declaredSecurity, securityConfigurer)); } /** The producer's client configuration; package-visible so a test can read what it holds. */ static java.util.Map producerConfig( - org.springframework.core.env.Environment environment, CompiledMessagingConfiguration compiled, ObjectProvider declaredSecurity, ObjectProvider securityConfigurer) { - String bootstrapServers = environment.getProperty("spring.kafka.bootstrap-servers", "").trim(); - if (bootstrapServers.isEmpty()) { - throw new IllegalStateException( - "app.messaging.broker=kafka needs spring.kafka.bootstrap-servers; a producer with no " - + "broker address defaults to localhost:9092 and fails at the first publish rather " - + "than at startup"); - } + BrokerSettings broker = kafkaBroker(compiled); java.util.Map config = new java.util.HashMap<>(); config.put( org.apache.kafka.clients.producer.ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, - bootstrapServers); + String.join(",", broker.bootstrapServers())); config.put( org.apache.kafka.clients.producer.ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, org.apache.kafka.common.serialization.ByteArraySerializer.class); config.put( org.apache.kafka.clients.producer.ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, org.apache.kafka.common.serialization.ByteArraySerializer.class); - // Durability is not a default worth inheriting: acks=1 loses an accepted publish to a leader - // failover, which is exactly the outcome an outbox exists to prevent. - config.put(org.apache.kafka.clients.producer.ProducerConfig.ACKS_CONFIG, "all"); - config.put(org.apache.kafka.clients.producer.ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true); + // The entry's delivery settings, which KafkaProfileValidator checks (a stable broker must keep + // acks=all and idempotence) and which used to be replaced here by hard-coded values. + config.put(org.apache.kafka.clients.producer.ProducerConfig.ACKS_CONFIG, broker.acks()); + config.put( + org.apache.kafka.clients.producer.ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, + broker.enableIdempotence()); + config.put( + org.apache.kafka.clients.producer.ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, + broker.maxInFlightRequestsPerConnection()); + // Kafka refuses delivery.timeout.ms below linger.ms + request.timeout.ms (5ms + 30s by + // default), which made the documented 30s a startup failure. The request timeout follows a + // short delivery deadline down, and records are not held back to fill a batch. + long delivery = broker.deliveryTimeout().toMillis(); + config.put(org.apache.kafka.clients.producer.ProducerConfig.LINGER_MS_CONFIG, 0); + config.put( + org.apache.kafka.clients.producer.ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, + Math.toIntExact(Math.min(30_000L, delivery))); + config.put( + org.apache.kafka.clients.producer.ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, + Math.toIntExact(delivery)); + // send() blocks the caller while it waits for metadata or buffer space, for Kafka's default + // 60s — before the publish deadline is even applied. Bounded by the default deadline instead. + config.put( + org.apache.kafka.clients.producer.ProducerConfig.MAX_BLOCK_MS_CONFIG, + dev.caskeleton.messaging.api.publish.PublishOptions.defaults().timeout().toMillis()); config.putAll(producerSecurity(compiled, declaredSecurity, securityConfigurer)); return config; } + /** The one Kafka broker entry the producer connects to. */ + private static BrokerSettings kafkaBroker(CompiledMessagingConfiguration compiled) { + java.util.List kafka = + compiled.brokers().values().stream() + .filter(broker -> BrokerSettings.KAFKA.equalsIgnoreCase(broker.type())) + .toList(); + if (kafka.size() != 1) { + throw new IllegalStateException( + "app.messaging.broker=kafka needs exactly one app.messaging.brokers. entry with" + + " type kafka and its bootstrap-servers; found " + + kafka.size() + + ". A producer with no broker address defaults to localhost:9092 and fails at the" + + " first publish rather than at startup"); + } + return kafka.getFirst(); + } + /** * The producer's security properties, from the one security profile of the Kafka broker. * diff --git a/messaging/messaging-spring-boot-starter/src/integrationTest/java/dev/caskeleton/messaging/autoconfigure/MessagingLiveRoundTripQualificationTest.java b/messaging/messaging-spring-boot-starter/src/integrationTest/java/dev/caskeleton/messaging/autoconfigure/MessagingLiveRoundTripQualificationTest.java index 4edfe2b..65c4340 100644 --- a/messaging/messaging-spring-boot-starter/src/integrationTest/java/dev/caskeleton/messaging/autoconfigure/MessagingLiveRoundTripQualificationTest.java +++ b/messaging/messaging-spring-boot-starter/src/integrationTest/java/dev/caskeleton/messaging/autoconfigure/MessagingLiveRoundTripQualificationTest.java @@ -77,7 +77,8 @@ class MessagingLiveRoundTripQualificationTest { .withPropertyValues( "app.messaging.enabled=true", "app.messaging.broker=kafka", - "spring.kafka.bootstrap-servers=" + KAFKA.getBootstrapServers()) + "app.messaging.brokers.kafka.type=kafka", + "app.messaging.brokers.kafka.bootstrap-servers[0]=" + KAFKA.getBootstrapServers()) // Both, because ValidatedDestinationRegistry refuses a profile whose dead-letter target is // not itself registered — a good invariant, and one a fixture declaring only the happy path // walks straight into. diff --git a/messaging/messaging-spring-boot-starter/src/integrationTest/java/dev/caskeleton/messaging/autoconfigure/MessagingSecuredBrokerQualificationTest.java b/messaging/messaging-spring-boot-starter/src/integrationTest/java/dev/caskeleton/messaging/autoconfigure/MessagingSecuredBrokerQualificationTest.java index 5d34659..5e823a3 100644 --- a/messaging/messaging-spring-boot-starter/src/integrationTest/java/dev/caskeleton/messaging/autoconfigure/MessagingSecuredBrokerQualificationTest.java +++ b/messaging/messaging-spring-boot-starter/src/integrationTest/java/dev/caskeleton/messaging/autoconfigure/MessagingSecuredBrokerQualificationTest.java @@ -159,7 +159,6 @@ class MessagingSecuredBrokerQualificationTest { .withPropertyValues( "app.messaging.enabled=true", "app.messaging.broker=kafka", - "spring.kafka.bootstrap-servers=" + kafka.getBootstrapServers(), "app.messaging.brokers.kafka.type=kafka", "app.messaging.brokers.kafka.bootstrap-servers[0]=" + kafka.getBootstrapServers(), "app.messaging.brokers.kafka.tls-enabled=true", diff --git a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/DefaultReactiveMessagePublisher.java b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/DefaultReactiveMessagePublisher.java index a17c0e3..5c3a25b 100644 --- a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/DefaultReactiveMessagePublisher.java +++ b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/DefaultReactiveMessagePublisher.java @@ -34,6 +34,10 @@ public final class DefaultReactiveMessagePublisher implements ReactiveMessagePub Objects.requireNonNull(destination, "destination must not be null"); Objects.requireNonNull(message, "message must not be null"); Objects.requireNonNull(options, "options must not be null"); - return Mono.fromCompletionStage(() -> delegate.publish(destination, message, options)); + // publish() is not purely asynchronous: admission may wait for a permit and the Kafka producer + // may wait for metadata before the stage exists. Subscribing on a Netty event loop would pin + // that loop for the wait, so the call runs on the bounded elastic pool instead. + return Mono.fromCompletionStage(() -> delegate.publish(destination, message, options)) + .subscribeOn(reactor.core.scheduler.Schedulers.boundedElastic()); } } diff --git a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingReliabilityAutoConfiguration.java b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingReliabilityAutoConfiguration.java index 72350cc..0fa0d2d 100644 --- a/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingReliabilityAutoConfiguration.java +++ b/messaging/messaging-spring-boot-starter/src/main/java/dev/caskeleton/messaging/autoconfigure/MessagingReliabilityAutoConfiguration.java @@ -109,6 +109,33 @@ public class MessagingReliabilityAutoConfiguration { return new OutboxRelayWorker(relay, scheduler); } + /** + * Publishes the relay's failed-pass count when a meter registry exists. + * + *

{@code failedPasses()} was counted and read by nothing, so a relay failing every pass looked + * on every dashboard exactly like one with nothing to do. + * + * @param worker the relay worker + * @param registries the meter registry, when the application has one + * @return the registration, run once the context has the registry + */ + @Bean + @ConditionalOnBean(OutboxRelayWorker.class) + public org.springframework.beans.factory.InitializingBean outboxRelayMeters( + OutboxRelayWorker worker, + org.springframework.beans.factory.ObjectProvider + registries) { + return () -> + registries.ifAvailable( + registry -> + io.micrometer.core.instrument.FunctionCounter.builder( + "messaging.outbox.relay.failed.passes", + worker, + OutboxRelayWorker::failedPasses) + .description("Outbox relay passes that ended in an exception") + .register(registry)); + } + /** * Binds the relay worker to the context lifecycle. * diff --git a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/BrokerAccessListTest.java b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/BrokerAccessListTest.java index fef555d..936c710 100644 --- a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/BrokerAccessListTest.java +++ b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/BrokerAccessListTest.java @@ -38,7 +38,6 @@ class BrokerAccessListTest { .withPropertyValues( "app.messaging.enabled=true", "app.messaging.broker=kafka", - "spring.kafka.bootstrap-servers=localhost:9092", "app.messaging.brokers.kafka-primary.type=kafka", "app.messaging.brokers.kafka-primary.bootstrap-servers[0]=localhost:9092", "app.messaging.security.kafka-primary.producer.type=SASL_SCRAM", diff --git a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/KafkaProducerConfigTest.java b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/KafkaProducerConfigTest.java new file mode 100644 index 0000000..643a088 --- /dev/null +++ b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/KafkaProducerConfigTest.java @@ -0,0 +1,96 @@ +package dev.caskeleton.messaging.autoconfigure; + +import static org.assertj.core.api.Assertions.assertThat; + +import dev.caskeleton.messaging.api.publish.PublishOptions; +import dev.caskeleton.messaging.kafka.KafkaSecurityConfigurer; +import dev.caskeleton.messaging.security.BrokerSecurityProfile; +import java.util.Map; +import org.junit.jupiter.api.Test; +import org.springframework.boot.autoconfigure.AutoConfigurations; +import org.springframework.boot.test.context.assertj.AssertableApplicationContext; +import org.springframework.boot.test.context.runner.ApplicationContextRunner; + +/** + * Where the producer connects and how it delivers come from the one broker entry. + * + *

The producer read {@code spring.kafka.bootstrap-servers} while the compiler required and + * validated {@code app.messaging.brokers..bootstrap-servers}, so a broker was described twice + * and only one description was used. The entry's delivery settings were validated and then replaced + * by hard-coded values. + */ +class KafkaProducerConfigTest { + + private final ApplicationContextRunner runner = + new ApplicationContextRunner() + .withConfiguration(AutoConfigurations.of(MessagingPlatformRootAutoConfiguration.class)) + .withPropertyValues( + "app.messaging.enabled=true", + "app.messaging.broker=kafka", + "app.messaging.brokers.kafka-primary.type=kafka", + // Resolvable names: the client refuses an address it cannot look up at construction. + "app.messaging.brokers.kafka-primary.bootstrap-servers[0]=localhost:9093", + "app.messaging.brokers.kafka-primary.bootstrap-servers[1]=127.0.0.1:9094"); + + @Test + void theProducerConnectsToTheBrokerEntryAddressesNotToSpringKafka() { + runner + .withPropertyValues("spring.kafka.bootstrap-servers=elsewhere:9092") + .run( + context -> + assertThat(producerConfig(context).get("bootstrap.servers")) + .isEqualTo("localhost:9093,127.0.0.1:9094")); + } + + @Test + void theProducerAppliesTheBrokerEntryDeliverySettings() { + runner + .withPropertyValues( + "app.messaging.brokers.kafka-primary.enable-idempotence=false", + "app.messaging.brokers.kafka-primary.acks=1", + "app.messaging.brokers.kafka-primary.max-in-flight-requests-per-connection=3", + "app.messaging.brokers.kafka-primary.delivery-timeout=45s") + .run( + context -> + assertThat(producerConfig(context)) + .extractingByKeys( + "enable.idempotence", + "acks", + "max.in.flight.requests.per.connection", + "delivery.timeout.ms") + .containsExactly(false, "1", 3, 45_000)); + } + + @Test + void theProducerStopsWaitingForMetadataWithinTheDefaultPublishDeadline() { + runner.run( + context -> + assertThat(producerConfig(context).get("max.block.ms")) + .as("the caller's thread waited up to Kafka's own 60s before any deadline applied") + .isEqualTo(PublishOptions.defaults().timeout().toMillis())); + } + + @Test + void aBrokerWithoutAnEntryFailsStartupNamingTheKey() { + new ApplicationContextRunner() + .withConfiguration(AutoConfigurations.of(MessagingPlatformRootAutoConfiguration.class)) + .withPropertyValues( + "app.messaging.enabled=true", + "app.messaging.broker=kafka", + "spring.kafka.bootstrap-servers=localhost:9092") + .run( + context -> + assertThat(context) + .hasFailed() + .getFailure() + .rootCause() + .hasMessageContaining("app.messaging.brokers. entry with type kafka")); + } + + private static Map producerConfig(AssertableApplicationContext context) { + return KafkaMessagingAutoConfiguration.producerConfig( + context.getBean(CompiledMessagingConfiguration.class), + context.getBeanProvider(BrokerSecurityProfile.class), + context.getBeanProvider(KafkaSecurityConfigurer.class)); + } +} diff --git a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/KafkaProducerSecurityTest.java b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/KafkaProducerSecurityTest.java index 2efd948..98c4bde 100644 --- a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/KafkaProducerSecurityTest.java +++ b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/KafkaProducerSecurityTest.java @@ -39,7 +39,6 @@ class KafkaProducerSecurityTest { .withPropertyValues( "app.messaging.enabled=true", "app.messaging.broker=kafka", - "spring.kafka.bootstrap-servers=localhost:9092", "app.messaging.brokers.kafka-primary.type=kafka", "app.messaging.brokers.kafka-primary.bootstrap-servers[0]=localhost:9092"); @@ -228,7 +227,6 @@ class KafkaProducerSecurityTest { private static Map producerConfig(AssertableApplicationContext context) { return KafkaMessagingAutoConfiguration.producerConfig( - context.getEnvironment(), context.getBean(CompiledMessagingConfiguration.class), context.getBeanProvider(BrokerSecurityProfile.class), context.getBeanProvider(KafkaSecurityConfigurer.class)); diff --git a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingAutoConfigurationTest.java b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingAutoConfigurationTest.java index 6c82922..0d4708c 100644 --- a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingAutoConfigurationTest.java +++ b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingAutoConfigurationTest.java @@ -314,7 +314,9 @@ class MessagingAutoConfigurationTest { org.springframework.boot.autoconfigure.AutoConfigurations.of( MessagingCoreAutoConfiguration.class, KafkaMessagingAutoConfiguration.class)) .withUserConfiguration(PublisherConfiguration.class, AutoCommittingKafkaProfile.class) - .withPropertyValues("spring.kafka.bootstrap-servers=localhost:9092") + .withPropertyValues( + "app.messaging.brokers.kafka.type=kafka", + "app.messaging.brokers.kafka.bootstrap-servers[0]=localhost:9092") .run( context -> org.assertj.core.api.Assertions.assertThat(context) diff --git a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingConfigurationBindingTest.java b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingConfigurationBindingTest.java index 73f8a08..c3239aa 100644 --- a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingConfigurationBindingTest.java +++ b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingConfigurationBindingTest.java @@ -258,9 +258,7 @@ class MessagingConfigurationBindingTest { runner .withConfiguration(AutoConfigurations.of(KafkaMessagingAutoConfiguration.class)) .withPropertyValues(minimal()) - .withPropertyValues( - "spring.kafka.bootstrap-servers=localhost:9092", - "app.messaging.brokers.kafka-primary.production=true") + .withPropertyValues("app.messaging.brokers.kafka-primary.production=true") .run( context -> assertThat(context) diff --git a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingOutboxRelayLifecycleTest.java b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingOutboxRelayLifecycleTest.java index 6527539..5c9e3b0 100644 --- a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingOutboxRelayLifecycleTest.java +++ b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingOutboxRelayLifecycleTest.java @@ -73,6 +73,55 @@ class MessagingOutboxRelayLifecycleTest { .isFalse(); } + @Test + void aRelayPassThatFailsIsCountedOnTheMeterRegistry() { + new ApplicationContextRunner() + .withConfiguration(AutoConfigurations.of(MessagingReliabilityAutoConfiguration.class)) + .withBean( + io.micrometer.core.instrument.MeterRegistry.class, + io.micrometer.core.instrument.simple.SimpleMeterRegistry::new) + .withBean( + OutboxRepository.class, + () -> + (OutboxRepository) + java.lang.reflect.Proxy.newProxyInstance( + getClass().getClassLoader(), + new Class[] {OutboxRepository.class}, + (proxy, method, arguments) -> { + throw new IllegalStateException("the outbox database is unreachable"); + })) + .withBean( + OutboxEnvelopeFactory.class, + () -> new OutboxEnvelopeFactory(new ProducerId("order-api"))) + .withBean(MessagePublisher.class, () -> new RelayDependencies().messagePublisher()) + .run( + context -> { + // given + io.micrometer.core.instrument.MeterRegistry registry = + context.getBean(io.micrometer.core.instrument.MeterRegistry.class); + + // when + double failed = awaitFailedPasses(registry); + + // then + assertThat(failed) + .as("a pass that failed was swallowed with no log and a counter nothing read") + .isPositive(); + }); + } + + /** Reads the failed-pass counter until the relay's first pass has had time to fail. */ + private static double awaitFailedPasses(io.micrometer.core.instrument.MeterRegistry registry) + throws InterruptedException { + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + double failed = 0; + while (failed == 0 && System.nanoTime() < deadline) { + Thread.sleep(10); + failed = registry.get("messaging.outbox.relay.failed.passes").functionCounter().count(); + } + return failed; + } + /** The two beans only the application can supply, plus the publisher the platform needs. */ @Configuration(proxyBeanMethods = false) static class RelayDependencies { @@ -170,11 +219,6 @@ class MessagingOutboxRelayLifecycleTest { throw new UnsupportedOperationException("not used by this test"); } - @Override - public int purgePublishedBefore(Instant publishedBefore) { - throw new UnsupportedOperationException("not used by this test"); - } - @Override public int purgePublishedBefore(Instant publishedBefore, int limit) { throw new UnsupportedOperationException("not used by this test"); diff --git a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingStarterOffContractTest.java b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingStarterOffContractTest.java index 4da59b9..e611c07 100644 --- a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingStarterOffContractTest.java +++ b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingStarterOffContractTest.java @@ -41,7 +41,9 @@ class MessagingStarterOffContractTest { void presentWithoutTheSwitchIsOn() { runner .withPropertyValues( - "app.messaging.broker=kafka", "spring.kafka.bootstrap-servers=localhost:9092") + "app.messaging.broker=kafka", + "app.messaging.brokers.kafka.type=kafka", + "app.messaging.brokers.kafka.bootstrap-servers[0]=localhost:9092") .run( context -> { assertThat(context).hasNotFailed(); @@ -123,7 +125,8 @@ class MessagingStarterOffContractTest { .withPropertyValues( "app.messaging.enabled=true", "app.messaging.broker=kafka", - "spring.kafka.bootstrap-servers=localhost:9092") + "app.messaging.brokers.kafka.type=kafka", + "app.messaging.brokers.kafka.bootstrap-servers[0]=localhost:9092") .run( context -> { assertThat(context).hasNotFailed(); @@ -138,7 +141,8 @@ class MessagingStarterOffContractTest { .withPropertyValues( "app.messaging.enabled=true", "app.messaging.broker=kafka", - "spring.kafka.bootstrap-servers=localhost:9092") + "app.messaging.brokers.kafka.type=kafka", + "app.messaging.brokers.kafka.bootstrap-servers[0]=localhost:9092") .run( context -> { assertThat(context) @@ -165,7 +169,8 @@ class MessagingStarterOffContractTest { .withPropertyValues( "app.messaging.enabled=true", "app.messaging.broker=kafka", - "spring.kafka.bootstrap-servers=localhost:9092") + "app.messaging.brokers.kafka.type=kafka", + "app.messaging.brokers.kafka.bootstrap-servers[0]=localhost:9092") .run( context -> { assertThat(context) @@ -203,7 +208,7 @@ class MessagingStarterOffContractTest { .hasFailed() .getFailure() .rootCause() - .hasMessageContaining("spring.kafka.bootstrap-servers")); + .hasMessageContaining("app.messaging.brokers.")); } /** diff --git a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/ReactiveFacadeTest.java b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/ReactiveFacadeTest.java index 21931ea..91e82fe 100644 --- a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/ReactiveFacadeTest.java +++ b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/ReactiveFacadeTest.java @@ -91,6 +91,34 @@ class ReactiveFacadeTest { .verify(); } + @Test + void theBlockingPublishRunsOffTheSubscribingThread() { + // given + java.util.concurrent.atomic.AtomicReference publishedOn = + new java.util.concurrent.atomic.AtomicReference<>(); + MessagePublisher recording = + new MessagePublisher() { + @Override + public CompletionStage publish( + MessageDestination destination, + MessageEnvelope message, + PublishOptions options) { + publishedOn.set(Thread.currentThread()); + return CompletableFuture.completedFuture(confirmed()); + } + }; + + // when + new DefaultReactiveMessagePublisher(recording) + .publish(ORDERS, envelope(), PublishOptions.defaults()) + .block(); + + // then + assertThat(publishedOn.get()) + .as("admission and the producer's metadata wait block; an event loop must not pay for it") + .isNotSameAs(Thread.currentThread()); + } + private MessagePublisher counting() { return new MessagePublisher() { @Override