diff --git a/docs/messaging/configuration-reference.md b/docs/messaging/configuration-reference.md index 2dd4be8..0f87d73 100644 --- a/docs/messaging/configuration-reference.md +++ b/docs/messaging/configuration-reference.md @@ -21,20 +21,16 @@ Application의 canonical integration event를 platform publish pipeline으로 producing-service identity다. 값이 없으면 `IntegrationEventPublishPort` bridge 자체를 만들지 않는다. `spring.application.name`이나 -현재 process 이름으로 추론하지 않는다. 기존 legacy `OutboxEvent`/realtime 경로는 별도 cutover가 -끝날 때까지 `app.messaging.broker` 경로를 유지한다. +현재 process 이름으로 추론하지 않는다. 모든 발행(outbox relay 포함)이 이 bridge를 지나므로 messaging +모듈이 켜진 상태에서 값이 없으면 기동이 거부된다. -## Outbox canonical transport-only cutover +## Outbox -`APP_OUTBOX_CANONICAL_TRANSPORT_ENABLED` / `ca-skeleton.outbox.canonical-transport-enabled`은 -기존 `outbox_event` writer/claim/status authority를 유지한 채 canonical row의 **transport만** platform -publish path로 보내는 compatibility gate다. 기본값은 `false`이며 두 번째 relay를 만들지 않는다. - -`true`일 때는 `OutboxAppendPort`가 `ValidatedIntegrationEvent`의 exact envelope bytes와 canonical -metadata를 기존 outbox row에 저장하고, claim된 canonical row는 `IntegrationEventPublishPort`로 간다. -legacy row는 계속 `MessageBroker`를 사용한다. 따라서 mixed-row compatibility 기간에는 relay가 켜져 -있다면 `app.messaging.broker`도 계속 필요하며, canonical path를 위해 `IntegrationEventPublishPort`도 -추가로 필요하다. legacy backlog가 0이라는 별도 증거 없이 broker 요구를 제거하지 않는다. +outbox는 canonical 하나다(2026-09-28). use case가 `IntegrationEventEncoderPort`로 만든 +`ValidatedIntegrationEvent`를 `OutboxAppendPort`로 append하고, relay는 claim한 row의 exact envelope +bytes를 `IntegrationEventPublishPort`로 보낸다. 예전의 `ca-skeleton.outbox.canonical-transport-enabled` +gate와 legacy `MessageBroker → KafkaSender` 경로는 삭제됐다. 발행할 contract의 logical destination마다 +아래 destination profile을 선언한다(템플릿의 durable operation event는 `durable-operations`). ## Destination profile diff --git a/docs/messaging/outbox-inbox.md b/docs/messaging/outbox-inbox.md index fede443..d71c30f 100644 --- a/docs/messaging/outbox-inbox.md +++ b/docs/messaging/outbox-inbox.md @@ -21,8 +21,8 @@ Flyway version `1000..1999`를 예약해서 애플리케이션의 일반적인 classpath:db/migration/messaging ``` -현재 순서는 `V1001..V1005`(outbox/admin)와 `V1101`(inbox)이다. starter 테스트가 두 JDBC -모듈을 실제 combined classpath로 올려 version 중복과 예약 대역 이탈을 검사한다. 개별 DDL의 +현재 순서는 `V1001..V1005`(outbox/admin)와 `V1101`(inbox)이고, 모두 +`messaging-reliability-jdbc-postgresql`에 있다. starter 테스트가 그 모듈을 실제 combined classpath로 올려 version 중복과 예약 대역 이탈을 검사한다. 개별 DDL의 의미와 전환 순서는 [migration-guide.md](migration-guide.md)를 따른다. ## Outbox @@ -39,7 +39,7 @@ COMMIT; ### relay ```text -leaseBatch(100, 30s) -- lease로 다중 relay 인스턴스 안전 +claimBatch(owner, 100, 30s, now, maxAttempts) -- owner + fencing token으로 다중 relay 인스턴스 안전 → publish (messageId 그대로) → CONFIRMED → markPublished → AMBIGUOUS → markAmbiguous (같은 messageId로 재시도 가능) diff --git a/docs/messaging/support-matrix.md b/docs/messaging/support-matrix.md index 84c9647..bd3164c 100644 --- a/docs/messaging/support-matrix.md +++ b/docs/messaging/support-matrix.md @@ -17,8 +17,10 @@ > `BrokerFailureMatrix.shipped()` 하드코딩이 아니라 레인이 낸 증거(`BrokerCertificationEvidence`)에서 > 나온다. 증거가 없는 조합은 `NOT_COVERED`다 (MSG-014). -> **모듈 이름과 런타임 편입.** `messaging-outbox-jdbc-postgresql` / `messaging-inbox-jdbc-postgresql`은 -> 이전에 `-jpa`로 불렸다. 구현은 Spring JDBC이고 SQL은 PostgreSQL 전용(`?::jsonb`, +> **모듈 이름과 런타임 편입.** 관계형 신뢰성 구현은 `messaging-reliability-jdbc-postgresql`, +> 그 port만 보는 relay·consumer 로직은 `messaging-reliability`다(2026-09-28에 +> `messaging-outbox-jdbc-postgresql` / `messaging-inbox-jdbc-postgresql`을 이렇게 나눴다). 그 전에는 +> `-jpa`로 불렸다. 구현은 Spring JDBC이고 SQL은 PostgreSQL 전용(`jsonb`, > `FOR UPDATE SKIP LOCKED`, `ON CONFLICT`, `TIMESTAMPTZ`)이므로, 그 이름은 쓰지 않는 기술을 > 광고하고 vendor 중립 port(`messaging-reliability-api`)의 위치를 가렸다 (MSG-023). > diff --git a/messaging/CLAUDE.md b/messaging/CLAUDE.md index 6b2ad64..57e6932 100644 --- a/messaging/CLAUDE.md +++ b/messaging/CLAUDE.md @@ -16,16 +16,19 @@ leaf의 정확한 id는 registry에서 읽는다. family는 **디렉터리 이 | --- | --- | --- | | `*-api` (`core-api`, `schema-api`, `policy`, `reliability-api`, `admin-api`, `transport-spi`) | 값 타입, 포트, 정책 계약 | Java stdlib만. Spring/broker SDK/logging framework 금지 | | `schema-*` (json/avro/protobuf), `cloudevents` | 인코딩 구현 | 해당 codec 라이브러리만. Spring 금지 | -| `runtime-core`, `security`, `observability`, `admin-runtime` | framework-free 실행 로직 | Micrometer는 `observability`에서만. 나머지는 stdlib | +| `runtime-core`, `security`, `observability`, `admin-runtime`, `reliability` | framework-free 실행 로직 | Micrometer는 `observability`에서만. 나머지는 stdlib | | `kafka` | broker adapter | 해당 broker client. Spring은 client가 곧 Spring인 경우만 | -| `outbox-jdbc-postgresql`, `inbox-jdbc-postgresql` | 관계형 신뢰성 adapter | Spring JDBC / Spring TX. **JPA 아님** — 이름이 곧 계약이다 | +| `reliability-jdbc-postgresql` | 관계형 신뢰성 adapter (outbox·inbox repository, admin journal) | Spring JDBC / Spring TX. **JPA 아님** — 이름이 곧 계약이다 | | `spring-boot-starter`, `spring-boot-starter-kafka` | 조립 경계 | Spring Boot autoconfigure. 여기서만 | | `testkit` | 테스트 지원 | Testcontainers, JUnit. production leaf가 의존하지 않는다 | -`outbox`/`inbox` leaf가 `jdbc-postgresql`로 명명된 이유: 구현은 Spring JDBC이고 SQL은 -`?::jsonb` · `FOR UPDATE SKIP LOCKED` · `ON CONFLICT` · `TIMESTAMPTZ`로 PostgreSQL 전용이다. -`jpa`라는 이름은 사용하지 않는 기술을 광고하고, vendor 중립 port가 어디 있는지도 숨겼다. vendor -중립 계약은 `messaging-reliability-api`에 있다. +신뢰성 기능은 세 leaf다. `messaging-reliability-api`가 vendor 중립 port(`OutboxRepository`, +`InboxRepository`)를, `messaging-reliability`가 그 port만 보는 relay·consumer 로직(stdlib)을, +`messaging-reliability-jdbc-postgresql`이 port의 관계형 구현과 migration(`V1001..V1005`, `V1101`)을 +갖는다. 예전에는 `outbox-jdbc-postgresql` 하나가 relay 로직과 PG repository와 admin journal을 함께 +가졌고 inbox도 같은 모양이었다(S7-#5b). 구현 leaf가 `jdbc-postgresql`로 명명된 이유: 구현은 Spring +JDBC이고 SQL은 `jsonb` · `FOR UPDATE SKIP LOCKED` · `ON CONFLICT` · `TIMESTAMPTZ`로 PostgreSQL +전용이다. `jpa`라는 이름은 사용하지 않는 기술을 광고하고, vendor 중립 port가 어디 있는지도 숨겼다. ## Public surface 정책 @@ -57,7 +60,7 @@ RabbitMQ·Pulsar·NATS·Kafka Share·Spring Cloud Stream bridge leaf와 Debezium 새 leaf를 application에 채택할 때는 starter 또는 composition dependency와 architecture edge policy, qualification/support 상태를 함께 변경한다. -## application이 이 family에 도달하는 경로 (MSG-015, canonical bridge 구현 / legacy cutover 잔존) +## application이 이 family에 도달하는 경로 (MSG-015, cutover 완료) 리뷰(`docs/reviews/2026-08-14-messaging-module-code-review.md` §6.1·§6.3)가 정한 `application-owned port → platform anti-corruption bridge → platform` 경로는 이제 존재한다. @@ -72,21 +75,21 @@ lease/transport/outcome normalization/observation은 platform 중앙 publish pat package는 `messaging-core-api`와 `messaging-schema-api`만 보며 concrete runtime/transport/Kafka 타입을 직접 보지 않는다. -다만 **MSG-015의 전체 cutover가 끝난 것은 아니다.** legacy `OutboxEvent`와 realtime publisher는 -schema/order/tenant 등 canonical metadata가 부족해 아직 `MessageBroker → KafkaSender` R0 경로를 -사용한다. 그 metadata를 추측해 platform envelope를 만드는 것은 금지한다. 따라서 legacy outbox -storage/relay migration을 별도 change set으로 완료한 뒤에만 R0 seam을 제거한다. +**MSG-015 cutover는 끝났다(2026-09-28, 소유자 결정 7).** application의 outbox append는 canonical +`OutboxAppendPort` 하나이고, relay는 claim한 row의 validated event를 이 bridge로만 보낸다. legacy +`OutboxEvent`·`LegacyOutboxAppendPort`와 `MessageBroker → KafkaSender` R0 경로(TLS/SASL을 적용하지 +않던 두 번째 producer)는 삭제했다. 남은 Kafka producer는 platform의 것 하나이고, broker security +profile(`KafkaSecurityConfigurer`)을 적용한다. canonical bridge의 producer identity는 `app.messaging.producer-id`로 명시한다. 값이 없으면 bridge bean을 -만들지 않으며 `spring.application.name`/host/pod 이름에서 identity를 발명하지 않는다. +만들지 않으며 `spring.application.name`/host/pod 이름에서 identity를 발명하지 않는다. bridge가 없으면 +relay가 발행할 곳이 없으므로 messaging 모듈이 켜진 상태에서는 기동이 거부된다. -`ca-skeleton.outbox.canonical-transport-enabled=true`는 bridge 존재만으로 충분하지 않다. Stable -composition은 애플리케이션 계약 catalog/schema source를 임의로 만들지 않으므로 adopter가 -`IntegrationEventEncoderPort`를 명시적으로 제공해야 한다. 템플릿은 구현체를 주지 않는다(후보였던 -R1 JSON Schema encoder는 2026-09-25에 삭제). `OutboxCanonicalTransportRequirementValidator`는 -publisher 또는 encoder 중 하나라도 -없으면 startup을 거부한다. 즉 canonical flag가 켜졌는데 정본 encoder가 없는 상태로 성공 기동하는 -경로는 없다. +`IntegrationEventEncoderPort`는 템플릿이 제공한다: `adapter:outbound:support`의 +`JsonIntegrationEventEncoder`가 envelope v1(`shared-contract`의 `contracts/messaging/envelope/v1.schema.json`) +안에 payload record를 결정적 JSON으로 싣는다. 계약 catalog는 애플리케이션이 bean으로 등록한 +`IntegrationEventContractContribution`뿐이고, 선언한 payload schema hash가 classpath의 schema 파일과 +다르면 기동이 실패한다. payload를 JSON Schema로 매번 검증하려는 서비스는 자기 encoder bean을 두면 된다. ## Stable 승격에 필요한 증거 diff --git a/messaging/messaging-inbox-jdbc-postgresql/build.gradle b/messaging/messaging-inbox-jdbc-postgresql/build.gradle deleted file mode 100644 index bf339f0..0000000 --- a/messaging/messaging-inbox-jdbc-postgresql/build.gradle +++ /dev/null @@ -1,33 +0,0 @@ -plugins { - id 'ca.auxiliary-source-set' - id 'ca.platform-module' -} - -auxiliarySourceSets { - sourceSet('integrationTest') { - compilesAgainst 'main' - } -} - -dependencies { - api project(':messaging:messaging-core-api') - api project(':messaging:messaging-reliability-api') - - implementation 'org.springframework:spring-jdbc' - implementation 'org.springframework:spring-tx' - - // Live-database certification. The reliability patterns are claims about transaction - // boundaries and uniqueness constraints, and only a real database can settle them. - testImplementation project(':messaging:messaging-testkit') - integrationTestImplementation 'org.testcontainers:testcontainers-postgresql' - integrationTestImplementation 'org.testcontainers:testcontainers-junit-jupiter' - integrationTestImplementation 'org.postgresql:postgresql' -} - -strictTestLanes { - lane('integrationTest') { - integration() - sourceSet = 'integrationTest' - description = 'Runs inbox transaction and uniqueness contracts against real PostgreSQL.' - } -} diff --git a/messaging/messaging-outbox-jdbc-postgresql/gradle.lockfile b/messaging/messaging-outbox-jdbc-postgresql/gradle.lockfile deleted file mode 100644 index 608e6f7..0000000 --- a/messaging/messaging-outbox-jdbc-postgresql/gradle.lockfile +++ /dev/null @@ -1,111 +0,0 @@ -# This is a Gradle generated file for dependency locking. -# Manual edits can break the build and are not advised. -# This file is expected to be part of source control. -com.fasterxml.jackson.core:jackson-annotations:2.21=integrationTestCompileClasspath,integrationTestRuntimeClasspath -com.github.ben-manes.caffeine:caffeine:3.2.4=annotationProcessor,integrationTestAnnotationProcessor,testAnnotationProcessor -com.github.docker-java:docker-java-api:3.7.1=integrationTestCompileClasspath,integrationTestRuntimeClasspath -com.github.docker-java:docker-java-transport-zerodep:3.7.1=integrationTestCompileClasspath,integrationTestRuntimeClasspath -com.github.docker-java:docker-java-transport:3.7.1=integrationTestCompileClasspath,integrationTestRuntimeClasspath -com.github.kevinstern:software-and-algorithms:1.0=annotationProcessor,integrationTestAnnotationProcessor,testAnnotationProcessor -com.github.spotbugs:spotbugs-annotations:4.10.2=spotbugs -com.github.spotbugs:spotbugs:4.10.2=spotbugs -com.github.stephenc.jcip:jcip-annotations:1.0-1=spotbugs -com.google.auto.service:auto-service-annotations:1.0.1=annotationProcessor,integrationTestAnnotationProcessor,testAnnotationProcessor -com.google.auto.value:auto-value-annotations:1.9=annotationProcessor,integrationTestAnnotationProcessor,testAnnotationProcessor -com.google.auto:auto-common:1.2.2=annotationProcessor,integrationTestAnnotationProcessor,testAnnotationProcessor -com.google.code.findbugs:jsr305:3.0.2=checkstyle,spotbugs -com.google.code.gson:gson:2.13.2=spotbugs -com.google.errorprone:error_prone_annotation:2.49.0=annotationProcessor,integrationTestAnnotationProcessor,testAnnotationProcessor -com.google.errorprone:error_prone_annotations:2.41.0=spotbugs -com.google.errorprone:error_prone_annotations:2.47.0=checkstyle -com.google.errorprone:error_prone_annotations:2.49.0=annotationProcessor,integrationTestAnnotationProcessor,testAnnotationProcessor -com.google.errorprone:error_prone_check_api:2.49.0=annotationProcessor,integrationTestAnnotationProcessor,testAnnotationProcessor -com.google.errorprone:error_prone_core:2.49.0=annotationProcessor,integrationTestAnnotationProcessor,testAnnotationProcessor -com.google.googlejavaformat:google-java-format:1.35.0=annotationProcessor,integrationTestAnnotationProcessor,testAnnotationProcessor -com.google.guava:failureaccess:1.0.3=annotationProcessor,checkstyle,integrationTestAnnotationProcessor,testAnnotationProcessor -com.google.guava:guava:33.5.0-jre=annotationProcessor,integrationTestAnnotationProcessor,testAnnotationProcessor -com.google.guava:guava:33.6.0-jre=checkstyle -com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava=annotationProcessor,checkstyle,integrationTestAnnotationProcessor,testAnnotationProcessor -com.google.j2objc:j2objc-annotations:3.1=annotationProcessor,checkstyle,integrationTestAnnotationProcessor,testAnnotationProcessor -com.google.protobuf:protobuf-java:4.33.2=annotationProcessor,integrationTestAnnotationProcessor,testAnnotationProcessor -com.h3xstream.findsecbugs:findsecbugs-plugin:1.14.0=spotbugsPlugins -com.puppycrawl.tools:checkstyle:13.5.0=checkstyle -commons-beanutils:commons-beanutils:1.11.0=checkstyle -commons-codec:commons-codec:1.19.0=integrationTestCompileClasspath,integrationTestRuntimeClasspath -commons-collections:commons-collections:3.2.2=checkstyle -commons-io:commons-io:2.20.0=integrationTestCompileClasspath,integrationTestRuntimeClasspath -commons-io:commons-io:2.21.0=spotbugs -commons-logging:commons-logging:1.3.6=compileClasspath,integrationTestCompileClasspath,integrationTestRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -info.picocli:picocli:4.7.7=checkstyle -io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,integrationTestAnnotationProcessor,testAnnotationProcessor -io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,integrationTestAnnotationProcessor,testAnnotationProcessor -io.micrometer:micrometer-commons:1.16.7=compileClasspath,integrationTestCompileClasspath,integrationTestRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -io.micrometer:micrometer-core:1.16.7=compileClasspath,integrationTestCompileClasspath,integrationTestRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -io.micrometer:micrometer-observation:1.16.7=compileClasspath,integrationTestCompileClasspath,integrationTestRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -javax.inject:javax.inject:1=annotationProcessor,integrationTestAnnotationProcessor,testAnnotationProcessor -jaxen:jaxen:2.0.6=spotbugs -net.bytebuddy:byte-buddy:1.17.8=integrationTestCompileClasspath,integrationTestRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -net.java.dev.jna:jna:5.18.1=integrationTestCompileClasspath,integrationTestRuntimeClasspath -net.sf.saxon:Saxon-HE:12.9=checkstyle,spotbugs -org.antlr:antlr4-runtime:4.13.2=checkstyle -org.apache.bcel:bcel:6.12.0=spotbugs -org.apache.commons:commons-compress:1.28.0=integrationTestCompileClasspath,integrationTestRuntimeClasspath -org.apache.commons:commons-lang3:3.20.0=checkstyle,integrationTestCompileClasspath,integrationTestRuntimeClasspath,spotbugs -org.apache.commons:commons-text:1.15.0=spotbugs -org.apache.commons:commons-text:1.3=checkstyle -org.apache.httpcomponents:httpclient:4.5.13=checkstyle -org.apache.httpcomponents:httpcore:4.4.16=checkstyle -org.apache.logging.log4j:log4j-api:2.25.5=spotbugs -org.apache.logging.log4j:log4j-core:2.25.5=spotbugs -org.apache.maven.doxia:doxia-core:1.12.0=checkstyle -org.apache.maven.doxia:doxia-logging-api:1.12.0=checkstyle -org.apache.maven.doxia:doxia-module-xdoc:1.12.0=checkstyle -org.apache.maven.doxia:doxia-sink-api:1.12.0=checkstyle -org.apache.xbean:xbean-reflect:3.7=checkstyle -org.apiguardian:apiguardian-api:1.1.2=integrationTestCompileClasspath,testCompileClasspath -org.assertj:assertj-core:3.27.7=integrationTestCompileClasspath,integrationTestRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.checkerframework:checker-qual:3.55.1=integrationTestRuntimeClasspath -org.codehaus.plexus:plexus-classworlds:2.6.0=checkstyle -org.codehaus.plexus:plexus-component-annotations:2.1.0=checkstyle -org.codehaus.plexus:plexus-container-default:2.1.0=checkstyle -org.codehaus.plexus:plexus-utils:3.3.0=checkstyle -org.dom4j:dom4j:2.2.0=spotbugs -org.hdrhistogram:HdrHistogram:2.2.2=integrationTestRuntimeClasspath,runtimeClasspath,testRuntimeClasspath -org.javassist:javassist:3.28.0-GA=checkstyle -org.jetbrains:annotations:17.0.0=integrationTestCompileClasspath,integrationTestRuntimeClasspath -org.jspecify:jspecify:1.0.1=annotationProcessor,checkstyle,compileClasspath,integrationTestAnnotationProcessor,integrationTestCompileClasspath,integrationTestRuntimeClasspath,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-api:6.0.3=integrationTestCompileClasspath,integrationTestRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-engine:6.0.3=integrationTestRuntimeClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-params:6.0.3=integrationTestCompileClasspath,integrationTestRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter:6.0.3=integrationTestCompileClasspath,integrationTestRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-commons:6.0.3=integrationTestCompileClasspath,integrationTestRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-engine:6.0.3=integrationTestRuntimeClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-launcher:6.0.3=integrationTestRuntimeClasspath,testRuntimeClasspath -org.junit:junit-bom:6.0.3=integrationTestCompileClasspath,integrationTestRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit:junit-bom:6.1.0=spotbugs -org.latencyutils:LatencyUtils:2.0.3=integrationTestRuntimeClasspath,runtimeClasspath,testRuntimeClasspath -org.mockito:mockito-core:5.20.0=mockitoAgent -org.opentest4j:opentest4j:1.3.0=integrationTestCompileClasspath,integrationTestRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.10.1=spotbugs -org.ow2.asm:asm-commons:9.10.1=spotbugs -org.ow2.asm:asm-tree:9.10.1=spotbugs -org.ow2.asm:asm-util:9.10.1=spotbugs -org.ow2.asm:asm:9.10.1=spotbugs -org.pcollections:pcollections:4.0.1=annotationProcessor,integrationTestAnnotationProcessor,testAnnotationProcessor -org.postgresql:postgresql:42.7.13=integrationTestCompileClasspath,integrationTestRuntimeClasspath -org.reflections:reflections:0.10.2=checkstyle -org.rnorth.duct-tape:duct-tape:1.0.8=integrationTestCompileClasspath,integrationTestRuntimeClasspath -org.slf4j:slf4j-api:2.0.18=integrationTestCompileClasspath,integrationTestRuntimeClasspath,spotbugs,spotbugsSlf4j -org.slf4j:slf4j-simple:2.0.17=spotbugsSlf4j -org.slf4j:slf4j-simple:2.0.18=checkstyle -org.springframework:spring-beans:7.0.9=compileClasspath,integrationTestCompileClasspath,integrationTestRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.springframework:spring-core:7.0.9=compileClasspath,integrationTestCompileClasspath,integrationTestRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.springframework:spring-jdbc:7.0.9=compileClasspath,integrationTestCompileClasspath,integrationTestRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.springframework:spring-tx:7.0.9=compileClasspath,integrationTestCompileClasspath,integrationTestRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.testcontainers:testcontainers-database-commons:2.0.5=integrationTestCompileClasspath,integrationTestRuntimeClasspath -org.testcontainers:testcontainers-jdbc:2.0.5=integrationTestCompileClasspath,integrationTestRuntimeClasspath -org.testcontainers:testcontainers-junit-jupiter:2.0.5=integrationTestCompileClasspath,integrationTestRuntimeClasspath -org.testcontainers:testcontainers-postgresql:2.0.5=integrationTestCompileClasspath,integrationTestRuntimeClasspath -org.testcontainers:testcontainers:2.0.5=integrationTestCompileClasspath,integrationTestRuntimeClasspath -org.xmlresolver:xmlresolver:5.3.3=checkstyle,spotbugs -empty= diff --git a/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxRepository.java b/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxRepository.java index 8197a2c..6765abf 100644 --- a/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxRepository.java +++ b/messaging/messaging-reliability-api/src/main/java/dev/caskeleton/messaging/reliability/OutboxRepository.java @@ -22,38 +22,23 @@ public interface OutboxRepository { void append(OutboxRecord record); /** - * Leases a batch of publishable records. + * Claims a batch and returns the leases, each carrying its owner and fencing token, skipping rows + * that are waiting for their backoff or out of attempts. * - *

Leasing rather than simply selecting is what makes multiple relay instances safe: a record - * claimed by one relay is invisible to the others until its lease expires, so the same message is - * not published concurrently by two processes. - * - * @param batchSize how many records to claim - * @param leaseDuration how long the claim holds - * @param now the current instant - * @return the leased records, oldest first - */ - List leaseBatch(int batchSize, Duration leaseDuration, Instant now); - - /** - * Claims a batch and returns the leases, each carrying its owner and fencing token. - * - *

The token is what a terminal write is checked against. {@link #leaseBatch} returns records - * without one, so its callers cannot prove a write belongs to their claim; it remains for - * inspection paths and is deprecated for the relay's use. - * - * @param owner this relay's stable identity - */ - List claimBatch(String owner, int batchSize, Duration leaseDuration, Instant now); - - /** - * Claims a batch, skipping rows that are waiting for their backoff or out of attempts. + *

Claiming rather than simply selecting is what makes multiple relay instances safe: a record + * claimed by one relay is invisible to the others until its lease expires, and the token is what + * a terminal write is checked against, so a relay that stalled past its lease writes nothing. * *

Eligibility belongs in the row. A relay that decides in memory forgets on restart, and every * replica decides for itself — so the same message is retried by as many workers as there are * replicas, as often as they poll. * + * @param owner this relay's stable identity + * @param batchSize how many records to claim + * @param leaseDuration how long the claim holds + * @param now the current instant * @param maxAttempts the attempt budget; rows at or above it are not claimed + * @return the leases, oldest first */ List claimBatch( String owner, int batchSize, Duration leaseDuration, Instant now, int maxAttempts); @@ -65,10 +50,10 @@ public interface OutboxRepository { */ OutboxTransitionResult markPublished(OutboxLease lease, Instant now); - /** Records an ambiguous outcome, only if this lease still owns the row. */ - OutboxTransitionResult markAmbiguous(OutboxLease lease, String failureCode, Instant now); - - /** Records an ambiguous outcome and when this row may next be claimed. */ + /** + * Records an ambiguous outcome and when this row may next be claimed, only if this lease still + * owns the row. + */ OutboxTransitionResult markAmbiguous( OutboxLease lease, String failureCode, Instant now, Instant nextAttemptAt); diff --git a/messaging/messaging-outbox-jdbc-postgresql/build.gradle b/messaging/messaging-reliability-jdbc-postgresql/build.gradle similarity index 69% rename from messaging/messaging-outbox-jdbc-postgresql/build.gradle rename to messaging/messaging-reliability-jdbc-postgresql/build.gradle index c933afe..e6c57bb 100644 --- a/messaging/messaging-outbox-jdbc-postgresql/build.gradle +++ b/messaging/messaging-reliability-jdbc-postgresql/build.gradle @@ -9,11 +9,13 @@ auxiliarySourceSets { } } +// The relational implementations of the reliability ports: the outbox repository, the inbox +// repository and the destructive-operation journal. Spring JDBC for the transaction join, and SQL +// that is PostgreSQL's (`jsonb`, `FOR UPDATE SKIP LOCKED`, `ON CONFLICT`, `TIMESTAMPTZ`) — the name +// is the contract. The relay and consumer logic that uses them is messaging-reliability. dependencies { api project(':messaging:messaging-core-api') api project(':messaging:messaging-reliability-api') - api project(':messaging:messaging-policy') - api project(':messaging:messaging-observability') // The destructive-operation journal lives here because it needs exactly what the outbox needs: // one relational database every replica can see, and a migration lane that already exists. The @@ -26,9 +28,7 @@ dependencies { // Live-database certification. The reliability patterns are claims about transaction // boundaries and uniqueness constraints, and only a real database can settle them. testImplementation project(':messaging:messaging-testkit') - // Test-only. The relay is also driven through the real DefaultMessagePublisher: a fake publisher - // accepted the pre-encoded rows the real one re-encoded and rejected. - testImplementation project(':messaging:messaging-runtime-core') + integrationTestImplementation project(':messaging:messaging-testkit') integrationTestImplementation 'org.testcontainers:testcontainers-postgresql' integrationTestImplementation 'org.testcontainers:testcontainers-junit-jupiter' integrationTestImplementation 'org.postgresql:postgresql' @@ -38,6 +38,6 @@ strictTestLanes { lane('integrationTest') { integration() sourceSet = 'integrationTest' - description = 'Runs outbox and admin-journal contracts against real PostgreSQL.' + description = 'Runs outbox, inbox and admin-journal contracts against real PostgreSQL.' } } diff --git a/messaging/messaging-inbox-jdbc-postgresql/gradle.lockfile b/messaging/messaging-reliability-jdbc-postgresql/gradle.lockfile similarity index 100% rename from messaging/messaging-inbox-jdbc-postgresql/gradle.lockfile rename to messaging/messaging-reliability-jdbc-postgresql/gradle.lockfile diff --git a/messaging/messaging-inbox-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/inbox/InboxPostgresIT.java b/messaging/messaging-reliability-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/inbox/InboxPostgresIT.java similarity index 100% rename from messaging/messaging-inbox-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/inbox/InboxPostgresIT.java rename to messaging/messaging-reliability-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/inbox/InboxPostgresIT.java diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/outbox/AdminOperationJournalPostgresIT.java b/messaging/messaging-reliability-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/outbox/AdminOperationJournalPostgresIT.java similarity index 100% rename from messaging/messaging-outbox-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/outbox/AdminOperationJournalPostgresIT.java rename to messaging/messaging-reliability-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/outbox/AdminOperationJournalPostgresIT.java diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/outbox/OutboxPostgresIT.java b/messaging/messaging-reliability-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/outbox/OutboxPostgresIT.java similarity index 86% rename from messaging/messaging-outbox-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/outbox/OutboxPostgresIT.java rename to messaging/messaging-reliability-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/outbox/OutboxPostgresIT.java index 66fba44..f3f1d2a 100644 --- a/messaging/messaging-outbox-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/outbox/OutboxPostgresIT.java +++ b/messaging/messaging-reliability-jdbc-postgresql/src/integrationTest/java/dev/caskeleton/messaging/outbox/OutboxPostgresIT.java @@ -71,6 +71,9 @@ class OutboxPostgresIT { try (Connection connection = dataSource.getConnection(); Statement statement = connection.createStatement()) { statement.execute("DROP TABLE IF EXISTS messaging_outbox"); + // The inbox migration ships in the same module now, so "every migration" creates its table + // too. + statement.execute("DROP TABLE IF EXISTS messaging_inbox"); // Every migration in order, not just V1: a test that applies part of the schema certifies a // database nobody deploys. for (String migration : migrations()) { @@ -90,7 +93,7 @@ class OutboxPostgresIT { OutboxRecord record = record(); appendInTransaction(record); - List leased = repository.leaseBatch(10, Duration.ofSeconds(30), NOW); + List leased = claim("relay-a", NOW).stream().map(OutboxLease::record).toList(); assertThat(leased) .singleElement() @@ -109,8 +112,8 @@ class OutboxPostgresIT { void aLeasedRowIsInvisibleToASecondRelayInstance() { appendInTransaction(record()); - List first = repository.leaseBatch(10, Duration.ofSeconds(30), NOW); - List second = repository.leaseBatch(10, Duration.ofSeconds(30), NOW); + List first = claim("relay-a", NOW); + List second = claim("relay-b", NOW); assertThat(first).hasSize(1); assertThat(second).as("SKIP LOCKED plus the lease keeps two relays off the same row").isEmpty(); @@ -119,10 +122,9 @@ class OutboxPostgresIT { @Test void anExpiredLeaseBecomesClaimableAgain() { appendInTransaction(record()); - repository.leaseBatch(10, Duration.ofSeconds(30), NOW); + claim("relay-a", NOW); - List reclaimed = - repository.leaseBatch(10, Duration.ofSeconds(30), NOW.plusSeconds(31)); + List reclaimed = claim("relay-b", NOW.plusSeconds(31)); assertThat(reclaimed).hasSize(1); } @@ -131,18 +133,16 @@ class OutboxPostgresIT { void anAmbiguousRowStaysClaimableUnderItsOriginalIdentity() { OutboxRecord record = record(); appendInTransaction(record); - // Claimed rather than leased: a terminal write is checked against the lease token, so - // `markAmbiguous` takes the lease that owns the row. The identity-keyed overload this test used - // to call is gone, and with it the possibility of one relay writing an outcome onto a row - // another relay currently owns. - OutboxLease lease = - repository.claimBatch("ambiguous-relay", 10, Duration.ofSeconds(30), NOW).getFirst(); + // A terminal write is checked against the lease token, so `markAmbiguous` takes the lease that + // owns the row; no identity-keyed write exists for one relay to put an outcome on a row another + // relay currently owns. + OutboxLease lease = claim("ambiguous-relay", NOW).getFirst(); - repository.markAmbiguous(lease, "CONFIRM_TIMEOUT", NOW); + repository.markAmbiguous(lease, "CONFIRM_TIMEOUT", NOW, NOW); assertThat(repository.find(record.messageId()).orElseThrow().status()) .isEqualTo(OutboxStatus.AMBIGUOUS); - assertThat(repository.leaseBatch(10, Duration.ofSeconds(30), NOW)) + assertThat(claim("relay-b", NOW)) .singleElement() .satisfies(retry -> assertThat(retry.messageId()).isEqualTo(record.messageId())); } @@ -151,24 +151,24 @@ class OutboxPostgresIT { void aPublishedRowIsNotClaimedAgain() { OutboxRecord record = record(); appendInTransaction(record); - repository.leaseBatch(10, Duration.ofSeconds(30), NOW); + OutboxLease lease = claim("relay-a", NOW).getFirst(); - repository.markPublishedUnfenced(record.messageId(), NOW); + repository.markPublished(lease, NOW); assertThat(repository.find(record.messageId()).orElseThrow().status()) .isEqualTo(OutboxStatus.PUBLISHED); - assertThat(repository.leaseBatch(10, Duration.ofSeconds(30), NOW.plusSeconds(60))).isEmpty(); + assertThat(claim("relay-b", NOW.plusSeconds(60))).isEmpty(); } @Test void aFailedRowIsNotClaimedAgainWithoutIntervention() { OutboxRecord record = record(); appendInTransaction(record); - repository.leaseBatch(10, Duration.ofSeconds(30), NOW); + OutboxLease lease = claim("relay-a", NOW).getFirst(); - repository.markFailedUnfenced(record.messageId(), "INVALID_TOPIC", NOW); + repository.markFailed(lease, "INVALID_TOPIC", NOW); - assertThat(repository.leaseBatch(10, Duration.ofSeconds(30), NOW.plusSeconds(60))).isEmpty(); + assertThat(claim("relay-b", NOW.plusSeconds(60))).isEmpty(); } @Test @@ -202,8 +202,8 @@ class OutboxPostgresIT { OutboxRecord published = record(); OutboxRecord pending = record(); appendInTransaction(published); + repository.markPublished(claim("relay-a", NOW).getFirst(), NOW); appendInTransaction(pending); - repository.markPublishedUnfenced(published.messageId(), NOW); int removed = repository.purgePublishedBefore(NOW.plusSeconds(1), 100); @@ -211,6 +211,11 @@ class OutboxPostgresIT { assertThat(repository.find(pending.messageId())).isPresent(); } + /** Claims with an unlimited attempt budget, as a relay does on a row's first attempts. */ + private List claim(String owner, Instant now) { + return repository.claimBatch(owner, 10, Duration.ofSeconds(30), now, Integer.MAX_VALUE); + } + private static OutboxRecord record() { return record( // The routing key is a column now, not a reserved header smuggled through the header map. @@ -228,7 +233,8 @@ class OutboxPostgresIT { appendInTransaction(otherKey); // when - List claimed = repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW); + List claimed = + repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW, Integer.MAX_VALUE); // then assertThat(claimed) @@ -352,12 +358,13 @@ class OutboxPostgresIT { // Relay A claims and stalls. java.util.List first = - repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW); + repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW, Integer.MAX_VALUE); assertThat(first).hasSize(1); // The lease expires and relay B takes the row and publishes it. java.util.List second = - repository.claimBatch("relay-b", 10, Duration.ofSeconds(30), NOW.plusSeconds(31)); + repository.claimBatch( + "relay-b", 10, Duration.ofSeconds(30), NOW.plusSeconds(31), Integer.MAX_VALUE); assertThat(second).hasSize(1); assertThat(second.get(0).token()) .as("every claim increments the token, which is what makes the older one detectable") @@ -366,7 +373,9 @@ class OutboxPostgresIT { .isEqualTo(dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED); // Relay A finally wakes up and tries to record its own outcome. - assertThat(repository.markAmbiguous(first.get(0), "CONFIRM_TIMEOUT", NOW.plusSeconds(40))) + assertThat( + repository.markAmbiguous( + first.get(0), "CONFIRM_TIMEOUT", NOW.plusSeconds(40), NOW.plusSeconds(40))) .as("this write used to succeed and made a published message claimable again") .isEqualTo(dev.caskeleton.messaging.reliability.OutboxTransitionResult.STALE_LEASE); assertThat(repository.find(record.messageId()).orElseThrow().status()) @@ -378,7 +387,7 @@ class OutboxPostgresIT { OutboxRecord record = record(); appendInTransaction(record); dev.caskeleton.messaging.reliability.OutboxLease lease = - repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW).get(0); + repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW, Integer.MAX_VALUE).get(0); assertThat(repository.markPublished(lease, NOW)) .isEqualTo(dev.caskeleton.messaging.reliability.OutboxTransitionResult.APPLIED); @@ -395,9 +404,9 @@ class OutboxPostgresIT { appendInTransaction(two); java.util.List a = - repository.claimBatch("relay-a", 1, Duration.ofSeconds(30), NOW); + repository.claimBatch("relay-a", 1, Duration.ofSeconds(30), NOW, Integer.MAX_VALUE); java.util.List b = - repository.claimBatch("relay-b", 1, Duration.ofSeconds(30), NOW); + repository.claimBatch("relay-b", 1, Duration.ofSeconds(30), NOW, Integer.MAX_VALUE); assertThat(a).hasSize(1); assertThat(b).hasSize(1); @@ -410,9 +419,15 @@ class OutboxPostgresIT { appendInTransaction(record); long firstToken = - repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW).get(0).token(); + repository + .claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW, Integer.MAX_VALUE) + .get(0) + .token(); dev.caskeleton.messaging.reliability.OutboxLease reclaimed = - repository.claimBatch("relay-b", 10, Duration.ofSeconds(30), NOW.plusSeconds(31)).get(0); + repository + .claimBatch( + "relay-b", 10, Duration.ofSeconds(30), NOW.plusSeconds(31), Integer.MAX_VALUE) + .get(0); assertThat(reclaimed.messageId()).isEqualTo(record.messageId()); assertThat(reclaimed.token()).isEqualTo(firstToken + 1); @@ -589,6 +604,38 @@ class OutboxPostgresIT { .containsEntry("x-note", "line one\nline\ttwo"); } + /** + * The hand-rolled reader took a quote preceded by a backslash for an escaped quote, so a value + * ending in an escaped backslash ran on into the next header and both came back wrong. + */ + @Test + void aHeaderValueEndingInABackslashRoundTripsWithoutSwallowingTheNextHeader() { + // given + OutboxRecord record = + new OutboxRecord( + MessageId.newId(), + new DestinationName("order-events"), + new MessageType("order.created"), + new SchemaVersion(1), + ContentType.JSON, + "{}".getBytes(StandardCharsets.UTF_8), + Map.of("x-path", "C:\\exports\\", "x-next", "after"), + OutboxCanonicalMetadata.none(), + NOW.minusSeconds(60), + OutboxStatus.PENDING, + 0, + Optional.empty(), + Optional.empty()); + appendInTransaction(record); + + // when + Map headers = repository.find(record.messageId()).orElseThrow().headers(); + + // then + assertThat(headers) + .containsExactlyInAnyOrderEntriesOf(Map.of("x-path", "C:\\exports\\", "x-next", "after")); + } + private void appendInTransaction(OutboxRecord record) { transactions.executeWithoutResult(status -> repository.append(record)); } diff --git a/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/JdbcInboxRepository.java b/messaging/messaging-reliability-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/JdbcInboxRepository.java similarity index 100% rename from messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/JdbcInboxRepository.java rename to messaging/messaging-reliability-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/JdbcInboxRepository.java diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcAdminOperationJournal.java b/messaging/messaging-reliability-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcAdminOperationJournal.java similarity index 100% rename from messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcAdminOperationJournal.java rename to messaging/messaging-reliability-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcAdminOperationJournal.java diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcOutboxRepository.java b/messaging/messaging-reliability-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcOutboxRepository.java similarity index 71% rename from messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcOutboxRepository.java rename to messaging/messaging-reliability-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcOutboxRepository.java index 13b87a3..49eec64 100644 --- a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcOutboxRepository.java +++ b/messaging/messaging-reliability-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/JdbcOutboxRepository.java @@ -17,6 +17,7 @@ import dev.caskeleton.messaging.reliability.OutboxRecord; import dev.caskeleton.messaging.reliability.OutboxRepository; import dev.caskeleton.messaging.reliability.OutboxStatus; import dev.caskeleton.messaging.reliability.OutboxTransitionResult; +import java.sql.Array; import java.sql.Connection; import java.sql.PreparedStatement; import java.sql.ResultSet; @@ -41,7 +42,7 @@ import org.springframework.transaction.support.TransactionSynchronizationManager * already inside, which is the entire mechanism. An outbox row written on a separate connection * commits independently of the business change and reopens the window the pattern exists to close. * - *

Leasing uses {@code SELECT ... FOR UPDATE SKIP LOCKED}. Two relay instances polling the same + *

Claiming uses {@code SELECT ... FOR UPDATE SKIP LOCKED}. Two relay instances polling the same * table would otherwise both claim the same rows and publish them twice; {@code SKIP LOCKED} lets * the second instance step over what the first already holds instead of blocking behind it. */ @@ -55,7 +56,7 @@ public final class JdbcOutboxRepository implements OutboxRepository { producer, occurred_at, correlation_id, causation_id, tenant, partition_key, ordering_key, schema_subject, schema_uri, traceparent, tracestate, baggage) - VALUES (?, ?, ?, ?, ?, ?, ?::jsonb, ?, ?, ?, + VALUES (?, ?, ?, ?, ?, ?, jsonb_object(?::text[]), ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) @@ -72,38 +73,15 @@ public final class JdbcOutboxRepository implements OutboxRepository { private static final String COLUMNS = """ message_id, destination, message_type, schema_version, content_type, - payload, headers, created_at, status, attempts, lease_expires_at, last_failure_code, + payload, + ARRAY(SELECT h.key FROM jsonb_each_text(headers) h ORDER BY h.key) AS header_keys, + ARRAY(SELECT h.value FROM jsonb_each_text(headers) h ORDER BY h.key) AS header_values, + created_at, status, attempts, lease_expires_at, last_failure_code, producer, occurred_at, correlation_id, causation_id, tenant, partition_key, ordering_key, schema_subject, schema_uri, traceparent, tracestate, baggage\ """; - private static final String LEASE = - """ - WITH claimable AS ( - SELECT message_id - FROM messaging_outbox - -- IN_FLIGHT is claimable too, but only once its lease has expired. A relay that dies - -- mid-publish leaves rows in that state; excluding them would strand those messages - -- forever, which is the one failure mode the outbox exists to prevent. - WHERE status IN ('PENDING', 'AMBIGUOUS', 'IN_FLIGHT') - AND (lease_expires_at IS NULL OR lease_expires_at <= ?) - ORDER BY created_at - LIMIT ? - FOR UPDATE SKIP LOCKED - ) - UPDATE messaging_outbox o - SET status = 'IN_FLIGHT', lease_expires_at = ? - FROM claimable c - WHERE o.message_id = c.message_id - RETURNING o.message_id, o.destination, o.message_type, o.schema_version, o.content_type, - o.payload, o.headers, o.created_at, o.status, o.attempts, - o.lease_expires_at, o.last_failure_code, - o.producer, o.occurred_at, o.correlation_id, o.causation_id, o.tenant, - o.partition_key, o.ordering_key, o.schema_subject, o.schema_uri, - o.traceparent, o.tracestate, o.baggage - """; - /** * The claim that records who holds the row and increments its fencing token. * @@ -147,7 +125,11 @@ public final class JdbcOutboxRepository implements OutboxRepository { FROM claimable c WHERE o.message_id = c.message_id RETURNING o.message_id, o.destination, o.message_type, o.schema_version, o.content_type, - o.payload, o.headers, o.created_at, o.status, o.attempts, + o.payload, + ARRAY(SELECT h.key FROM jsonb_each_text(o.headers) h ORDER BY h.key) AS header_keys, + ARRAY(SELECT h.value FROM jsonb_each_text(o.headers) h ORDER BY h.key) + AS header_values, + o.created_at, o.status, o.attempts, o.lease_expires_at, o.last_failure_code, o.lease_owner, o.lease_token, o.producer, o.occurred_at, o.correlation_id, o.causation_id, o.tenant, o.partition_key, o.ordering_key, o.schema_subject, o.schema_uri, @@ -187,7 +169,7 @@ public final class JdbcOutboxRepository implements OutboxRepository { statement.setInt(4, record.schemaVersion().value()); statement.setString(5, record.contentType().value()); statement.setBytes(6, record.payload()); - statement.setString(7, toJson(record.headers())); + statement.setArray(7, connection.createArrayOf("text", flatten(record.headers()))); statement.setTimestamp(8, Timestamp.from(record.createdAt())); statement.setString(9, record.status().name()); statement.setInt(10, record.attempts()); @@ -258,33 +240,6 @@ public final class JdbcOutboxRepository implements OutboxRepository { } } - @Override - public List leaseBatch(int batchSize, Duration leaseDuration, Instant now) { - Objects.requireNonNull(leaseDuration, "leaseDuration must not be null"); - Objects.requireNonNull(now, "now must not be null"); - return withConnection( - connection -> { - List leased = new ArrayList<>(); - try (PreparedStatement statement = connection.prepareStatement(LEASE)) { - statement.setTimestamp(1, Timestamp.from(now)); - statement.setInt(2, batchSize); - statement.setTimestamp(3, Timestamp.from(now.plus(leaseDuration))); - try (ResultSet results = statement.executeQuery()) { - while (results.next()) { - leased.add(read(results)); - } - } - } - return List.copyOf(leased); - }); - } - - @Override - public List claimBatch( - String owner, int batchSize, Duration leaseDuration, Instant now) { - return claimBatch(owner, batchSize, leaseDuration, now, Integer.MAX_VALUE); - } - @Override public List claimBatch( String owner, int batchSize, Duration leaseDuration, Instant now, int maxAttempts) { @@ -328,11 +283,6 @@ public final class JdbcOutboxRepository implements OutboxRepository { null); } - @Override - public OutboxTransitionResult markAmbiguous(OutboxLease lease, String failureCode, Instant now) { - return markAmbiguous(lease, failureCode, now, now); - } - @Override public OutboxTransitionResult markAmbiguous( OutboxLease lease, String failureCode, Instant now, Instant nextAttemptAt) { @@ -435,49 +385,6 @@ public final class JdbcOutboxRepository implements OutboxRepository { }); } - public void markPublishedUnfenced(MessageId messageId, Instant now) { - update( - "UPDATE messaging_outbox SET status = 'PUBLISHED', published_at = ?, " - + "lease_expires_at = NULL, attempts = attempts + 1 WHERE message_id = ?", - messageId, - now, - null); - } - - public void markAmbiguousUnfenced(MessageId messageId, String failureCode, Instant now) { - // The lease is released rather than held: an ambiguous row must become claimable again so the - // relay retries it under the same message id. - update( - "UPDATE messaging_outbox SET status = 'AMBIGUOUS', lease_expires_at = NULL, " - + "last_failure_code = ?, attempts = attempts + 1 WHERE message_id = ?", - messageId, - now, - failureCode); - } - - public void markFailedUnfenced(MessageId messageId, String failureCode, Instant now) { - update( - "UPDATE messaging_outbox SET status = 'FAILED', lease_expires_at = NULL, " - + "last_failure_code = ?, attempts = attempts + 1 WHERE message_id = ?", - messageId, - now, - failureCode); - } - - public void releaseLeaseUnfenced(MessageId messageId) { - withConnection( - connection -> { - try (PreparedStatement statement = - connection.prepareStatement( - "UPDATE messaging_outbox SET status = 'PENDING', lease_expires_at = NULL " - + "WHERE message_id = ? AND status = 'IN_FLIGHT'")) { - statement.setObject(1, messageId.value()); - statement.executeUpdate(); - } - return null; - }); - } - @Override public Optional find(MessageId messageId) { return withConnection( @@ -526,22 +433,6 @@ public final class JdbcOutboxRepository implements OutboxRepository { }); } - private void update(String sql, MessageId messageId, Instant now, String failureCode) { - withConnection( - connection -> { - try (PreparedStatement statement = connection.prepareStatement(sql)) { - if (failureCode == null) { - statement.setTimestamp(1, Timestamp.from(now)); - } else { - statement.setString(1, failureCode); - } - statement.setObject(2, messageId.value()); - statement.executeUpdate(); - } - return null; - }); - } - private static OutboxRecord read(ResultSet results) throws SQLException { Timestamp lease = results.getTimestamp("lease_expires_at"); return new OutboxRecord( @@ -551,7 +442,7 @@ public final class JdbcOutboxRepository implements OutboxRepository { new SchemaVersion(results.getInt("schema_version")), new ContentType(results.getString("content_type")), results.getBytes("payload"), - fromJson(results.getString("headers")), + headers(results.getArray("header_keys"), results.getArray("header_values")), readMetadata(results), results.getTimestamp("created_at").toInstant(), OutboxStatus.valueOf(results.getString("status")), @@ -597,119 +488,36 @@ public final class JdbcOutboxRepository implements OutboxRepository { } /** - * Serialises headers as a flat JSON object. + * The headers as one flat key, value, key, value array for {@code jsonb_object(text[])}. * - *

Hand-rolled rather than pulled from a JSON library so this module keeps no codec dependency: - * outbox headers are always flat string pairs, validated by {@code MessageHeaders} before they - * ever reach here. + *

PostgreSQL builds and escapes the JSON. The hand-rolled writer and reader this replaces got + * the escaping right one way and wrong the other: the reader took a quote after an escaped + * backslash for an escaped quote, so a value ending in a backslash ran on into the next header. */ - private static String toJson(Map headers) { - StringBuilder json = new StringBuilder("{"); - boolean first = true; + private static String[] flatten(Map headers) { + String[] flat = new String[headers.size() * 2]; + int index = 0; for (Map.Entry entry : headers.entrySet()) { - if (!first) { - json.append(','); - } - first = false; - json.append('"') - .append(escape(entry.getKey())) - .append("\":\"") - .append(escape(entry.getValue())) - .append('"'); + flat[index++] = entry.getKey(); + flat[index++] = entry.getValue(); } - return json.append('}').toString(); + return flat; } - private static Map fromJson(String json) { + /** Rebuilds the headers from the aligned key and value arrays the reads select. */ + private static Map headers(Array keys, Array values) throws SQLException { Map headers = new LinkedHashMap<>(); - if (json == null || json.length() <= 2) { + if (keys == null || values == null) { return headers; } - String body = json.substring(1, json.length() - 1); - int index = 0; - while (index < body.length()) { - int keyStart = body.indexOf('"', index); - if (keyStart < 0) { - break; - } - int keyEnd = findClosingQuote(body, keyStart + 1); - int valueStart = body.indexOf('"', body.indexOf(':', keyEnd)); - int valueEnd = findClosingQuote(body, valueStart + 1); - headers.put( - unescape(body.substring(keyStart + 1, keyEnd)), - unescape(body.substring(valueStart + 1, valueEnd))); - index = valueEnd + 1; + String[] names = (String[]) keys.getArray(); + String[] texts = (String[]) values.getArray(); + for (int index = 0; index < names.length; index++) { + headers.put(names[index], texts[index]); } return headers; } - private static int findClosingQuote(String text, int from) { - for (int index = from; index < text.length(); index++) { - if (text.charAt(index) == '"' && text.charAt(index - 1) != '\\') { - return index; - } - } - return text.length(); - } - - /** - * JSON string escaping, including the control characters. - * - *

Only backslash and quote were escaped. A header value containing a newline or a NUL — which - * the value objects now refuse, but which older rows and other writers may still carry — produced - * a document PostgreSQL rejects as invalid JSONB, so the append failed at the database with an - * error naming neither the header nor the row. Control characters are the part of the JSON - * grammar that is easy to forget and impossible to get away with. - */ - private static String escape(String value) { - StringBuilder escaped = new StringBuilder(value.length() + 8); - for (int index = 0; index < value.length(); index++) { - char character = value.charAt(index); - switch (character) { - case '\\' -> escaped.append("\\\\"); - case '"' -> escaped.append("\\\""); - case '\n' -> escaped.append("\\n"); - case '\r' -> escaped.append("\\r"); - case '\t' -> escaped.append("\\t"); - case '\b' -> escaped.append("\\b"); - case '\f' -> escaped.append("\\f"); - default -> { - if (character < 0x20) { - escaped.append("\\u%04x".formatted((int) character)); - } else { - escaped.append(character); - } - } - } - } - return escaped.toString(); - } - - private static String unescape(String value) { - StringBuilder plain = new StringBuilder(value.length()); - for (int index = 0; index < value.length(); index++) { - char character = value.charAt(index); - if (character != '\\' || index == value.length() - 1) { - plain.append(character); - continue; - } - char next = value.charAt(++index); - switch (next) { - case 'n' -> plain.append('\n'); - case 'r' -> plain.append('\r'); - case 't' -> plain.append('\t'); - case 'b' -> plain.append('\b'); - case 'f' -> plain.append('\f'); - case 'u' -> { - plain.append((char) Integer.parseInt(value.substring(index + 1, index + 5), 16)); - index += 4; - } - default -> plain.append(next); - } - } - return plain.toString(); - } - /** A unit of work against a borrowed connection. */ @FunctionalInterface private interface ConnectionCallback { diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V1001__messaging_outbox.sql b/messaging/messaging-reliability-jdbc-postgresql/src/main/resources/db/migration/messaging/V1001__messaging_outbox.sql similarity index 100% rename from messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V1001__messaging_outbox.sql rename to messaging/messaging-reliability-jdbc-postgresql/src/main/resources/db/migration/messaging/V1001__messaging_outbox.sql diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V1002__messaging_outbox_lease_fencing.sql b/messaging/messaging-reliability-jdbc-postgresql/src/main/resources/db/migration/messaging/V1002__messaging_outbox_lease_fencing.sql similarity index 100% rename from messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V1002__messaging_outbox_lease_fencing.sql rename to messaging/messaging-reliability-jdbc-postgresql/src/main/resources/db/migration/messaging/V1002__messaging_outbox_lease_fencing.sql diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V1003__messaging_admin_operation_journal.sql b/messaging/messaging-reliability-jdbc-postgresql/src/main/resources/db/migration/messaging/V1003__messaging_admin_operation_journal.sql similarity index 100% rename from messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V1003__messaging_admin_operation_journal.sql rename to messaging/messaging-reliability-jdbc-postgresql/src/main/resources/db/migration/messaging/V1003__messaging_admin_operation_journal.sql diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V1004__messaging_outbox_canonical_metadata.sql b/messaging/messaging-reliability-jdbc-postgresql/src/main/resources/db/migration/messaging/V1004__messaging_outbox_canonical_metadata.sql similarity index 100% rename from messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V1004__messaging_outbox_canonical_metadata.sql rename to messaging/messaging-reliability-jdbc-postgresql/src/main/resources/db/migration/messaging/V1004__messaging_outbox_canonical_metadata.sql diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V1005__messaging_outbox_ordering_key_index.sql b/messaging/messaging-reliability-jdbc-postgresql/src/main/resources/db/migration/messaging/V1005__messaging_outbox_ordering_key_index.sql similarity index 100% rename from messaging/messaging-outbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V1005__messaging_outbox_ordering_key_index.sql rename to messaging/messaging-reliability-jdbc-postgresql/src/main/resources/db/migration/messaging/V1005__messaging_outbox_ordering_key_index.sql diff --git a/messaging/messaging-inbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V1101__messaging_inbox.sql b/messaging/messaging-reliability-jdbc-postgresql/src/main/resources/db/migration/messaging/V1101__messaging_inbox.sql similarity index 100% rename from messaging/messaging-inbox-jdbc-postgresql/src/main/resources/db/migration/messaging/V1101__messaging_inbox.sql rename to messaging/messaging-reliability-jdbc-postgresql/src/main/resources/db/migration/messaging/V1101__messaging_inbox.sql diff --git a/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/JdbcInboxTransactionRequirementTest.java b/messaging/messaging-reliability-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/JdbcInboxTransactionRequirementTest.java similarity index 100% rename from messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/JdbcInboxTransactionRequirementTest.java rename to messaging/messaging-reliability-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/JdbcInboxTransactionRequirementTest.java diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/JdbcOutboxTransactionRequirementTest.java b/messaging/messaging-reliability-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/JdbcOutboxTransactionRequirementTest.java similarity index 100% rename from messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/JdbcOutboxTransactionRequirementTest.java rename to messaging/messaging-reliability-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/JdbcOutboxTransactionRequirementTest.java diff --git a/messaging/messaging-reliability/build.gradle b/messaging/messaging-reliability/build.gradle new file mode 100644 index 0000000..9c08eb5 --- /dev/null +++ b/messaging/messaging-reliability/build.gradle @@ -0,0 +1,19 @@ +plugins { + id 'ca.platform-module' +} + +// The outbox relay and the inbox consumer logic, framework-free. What they store and claim goes +// through the ports in messaging-reliability-api; the relational implementation of those ports is +// messaging-reliability-jdbc-postgresql. Split so a leaf's name says what it depends on: this one +// is Java stdlib plus the messaging APIs, the other is Spring JDBC and PostgreSQL SQL. +dependencies { + api project(':messaging:messaging-core-api') + api project(':messaging:messaging-reliability-api') + // The relay hands encoded rows to the publisher as EncodedMessage. + api project(':messaging:messaging-schema-api') + + // Test-only. The relay is also driven through the real DefaultMessagePublisher: a fake publisher + // accepted the pre-encoded rows the real one re-encoded and rejected. + testImplementation project(':messaging:messaging-runtime-core') + testImplementation project(':messaging:messaging-policy') +} diff --git a/messaging/messaging-reliability/gradle.lockfile b/messaging/messaging-reliability/gradle.lockfile new file mode 100644 index 0000000..58cc2a9 --- /dev/null +++ b/messaging/messaging-reliability/gradle.lockfile @@ -0,0 +1,89 @@ +# This is a Gradle generated file for dependency locking. +# Manual edits can break the build and are not advised. +# This file is expected to be part of source control. +com.github.ben-manes.caffeine:caffeine:3.2.4=annotationProcessor,testAnnotationProcessor +com.github.kevinstern:software-and-algorithms:1.0=annotationProcessor,testAnnotationProcessor +com.github.spotbugs:spotbugs-annotations:4.10.2=spotbugs +com.github.spotbugs:spotbugs:4.10.2=spotbugs +com.github.stephenc.jcip:jcip-annotations:1.0-1=spotbugs +com.google.auto.service:auto-service-annotations:1.0.1=annotationProcessor,testAnnotationProcessor +com.google.auto.value:auto-value-annotations:1.9=annotationProcessor,testAnnotationProcessor +com.google.auto:auto-common:1.2.2=annotationProcessor,testAnnotationProcessor +com.google.code.findbugs:jsr305:3.0.2=checkstyle,spotbugs +com.google.code.gson:gson:2.13.2=spotbugs +com.google.errorprone:error_prone_annotation:2.49.0=annotationProcessor,testAnnotationProcessor +com.google.errorprone:error_prone_annotations:2.41.0=spotbugs +com.google.errorprone:error_prone_annotations:2.47.0=checkstyle +com.google.errorprone:error_prone_annotations:2.49.0=annotationProcessor,testAnnotationProcessor +com.google.errorprone:error_prone_check_api:2.49.0=annotationProcessor,testAnnotationProcessor +com.google.errorprone:error_prone_core:2.49.0=annotationProcessor,testAnnotationProcessor +com.google.googlejavaformat:google-java-format:1.35.0=annotationProcessor,testAnnotationProcessor +com.google.guava:failureaccess:1.0.3=annotationProcessor,checkstyle,testAnnotationProcessor +com.google.guava:guava:33.5.0-jre=annotationProcessor,testAnnotationProcessor +com.google.guava:guava:33.6.0-jre=checkstyle +com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava=annotationProcessor,checkstyle,testAnnotationProcessor +com.google.j2objc:j2objc-annotations:3.1=annotationProcessor,checkstyle,testAnnotationProcessor +com.google.protobuf:protobuf-java:4.33.2=annotationProcessor,testAnnotationProcessor +com.h3xstream.findsecbugs:findsecbugs-plugin:1.14.0=spotbugsPlugins +com.puppycrawl.tools:checkstyle:13.5.0=checkstyle +commons-beanutils:commons-beanutils:1.11.0=checkstyle +commons-collections:commons-collections:3.2.2=checkstyle +commons-io:commons-io:2.21.0=spotbugs +info.picocli:picocli:4.7.7=checkstyle +io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnotationProcessor +io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor +io.micrometer:micrometer-commons:1.16.7=testCompileClasspath,testRuntimeClasspath +io.micrometer:micrometer-core:1.16.7=testCompileClasspath,testRuntimeClasspath +io.micrometer:micrometer-observation:1.16.7=testCompileClasspath,testRuntimeClasspath +javax.inject:javax.inject:1=annotationProcessor,testAnnotationProcessor +jaxen:jaxen:2.0.6=spotbugs +net.bytebuddy:byte-buddy:1.17.8=testCompileClasspath,testRuntimeClasspath +net.sf.saxon:Saxon-HE:12.9=checkstyle,spotbugs +org.antlr:antlr4-runtime:4.13.2=checkstyle +org.apache.bcel:bcel:6.12.0=spotbugs +org.apache.commons:commons-lang3:3.20.0=checkstyle,spotbugs +org.apache.commons:commons-text:1.15.0=spotbugs +org.apache.commons:commons-text:1.3=checkstyle +org.apache.httpcomponents:httpclient:4.5.13=checkstyle +org.apache.httpcomponents:httpcore:4.4.16=checkstyle +org.apache.logging.log4j:log4j-api:2.25.5=spotbugs +org.apache.logging.log4j:log4j-core:2.25.5=spotbugs +org.apache.maven.doxia:doxia-core:1.12.0=checkstyle +org.apache.maven.doxia:doxia-logging-api:1.12.0=checkstyle +org.apache.maven.doxia:doxia-module-xdoc:1.12.0=checkstyle +org.apache.maven.doxia:doxia-sink-api:1.12.0=checkstyle +org.apache.xbean:xbean-reflect:3.7=checkstyle +org.apiguardian:apiguardian-api:1.1.2=testCompileClasspath +org.assertj:assertj-core:3.27.7=testCompileClasspath,testRuntimeClasspath +org.codehaus.plexus:plexus-classworlds:2.6.0=checkstyle +org.codehaus.plexus:plexus-component-annotations:2.1.0=checkstyle +org.codehaus.plexus:plexus-container-default:2.1.0=checkstyle +org.codehaus.plexus:plexus-utils:3.3.0=checkstyle +org.dom4j:dom4j:2.2.0=spotbugs +org.hdrhistogram:HdrHistogram:2.2.2=testRuntimeClasspath +org.javassist:javassist:3.28.0-GA=checkstyle +org.jspecify:jspecify:1.0.1=annotationProcessor,checkstyle,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-api:6.0.3=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-engine:6.0.3=testRuntimeClasspath +org.junit.jupiter:junit-jupiter-params:6.0.3=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter:6.0.3=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-commons:6.0.3=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-engine:6.0.3=testRuntimeClasspath +org.junit.platform:junit-platform-launcher:6.0.3=testRuntimeClasspath +org.junit:junit-bom:6.0.3=testCompileClasspath,testRuntimeClasspath +org.junit:junit-bom:6.1.0=spotbugs +org.latencyutils:LatencyUtils:2.0.3=testRuntimeClasspath +org.mockito:mockito-core:5.20.0=mockitoAgent +org.opentest4j:opentest4j:1.3.0=testCompileClasspath,testRuntimeClasspath +org.ow2.asm:asm-analysis:9.10.1=spotbugs +org.ow2.asm:asm-commons:9.10.1=spotbugs +org.ow2.asm:asm-tree:9.10.1=spotbugs +org.ow2.asm:asm-util:9.10.1=spotbugs +org.ow2.asm:asm:9.10.1=spotbugs +org.pcollections:pcollections:4.0.1=annotationProcessor,testAnnotationProcessor +org.reflections:reflections:0.10.2=checkstyle +org.slf4j:slf4j-api:2.0.18=spotbugs,spotbugsSlf4j +org.slf4j:slf4j-simple:2.0.17=spotbugsSlf4j +org.slf4j:slf4j-simple:2.0.18=checkstyle +org.xmlresolver:xmlresolver:5.3.3=checkstyle,spotbugs +empty=compileClasspath,runtimeClasspath diff --git a/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/IdempotentConsumer.java b/messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/inbox/IdempotentConsumer.java similarity index 100% rename from messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/IdempotentConsumer.java rename to messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/inbox/IdempotentConsumer.java diff --git a/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/InboxCleanupJob.java b/messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/inbox/InboxCleanupJob.java similarity index 100% rename from messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/InboxCleanupJob.java rename to messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/inbox/InboxCleanupJob.java diff --git a/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/InboxOutcome.java b/messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/inbox/InboxOutcome.java similarity index 100% rename from messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/InboxOutcome.java rename to messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/inbox/InboxOutcome.java diff --git a/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/InboxRetentionPolicy.java b/messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/inbox/InboxRetentionPolicy.java similarity index 100% rename from messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/InboxRetentionPolicy.java rename to messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/inbox/InboxRetentionPolicy.java diff --git a/messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/TransactionalInboxHandler.java b/messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/inbox/TransactionalInboxHandler.java similarity index 100% rename from messaging/messaging-inbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/inbox/TransactionalInboxHandler.java rename to messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/inbox/TransactionalInboxHandler.java diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxCleanupJob.java b/messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/outbox/OutboxCleanupJob.java similarity index 100% rename from messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxCleanupJob.java rename to messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/outbox/OutboxCleanupJob.java diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxEnvelopeFactory.java b/messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/outbox/OutboxEnvelopeFactory.java similarity index 100% rename from messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxEnvelopeFactory.java rename to messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/outbox/OutboxEnvelopeFactory.java diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxProperties.java b/messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/outbox/OutboxProperties.java similarity index 100% rename from messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxProperties.java rename to messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/outbox/OutboxProperties.java diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelay.java b/messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelay.java similarity index 99% rename from messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelay.java rename to messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelay.java index 52350c5..566f93c 100644 --- a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelay.java +++ b/messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelay.java @@ -158,7 +158,7 @@ public final class OutboxRelay { */ public OutboxRelayReport runOnce(Instant now) { Objects.requireNonNull(now, "now must not be null"); - // claimBatch, not leaseBatch: the claim carries an owner and a fencing token, and every write + // The claim carries an owner and a fencing token, and every write // below is conditioned on them. Without that, a relay that stalled past its lease could record // AMBIGUOUS over the PUBLISHED another relay had already written, making a confirmed message // claimable again. diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelayReport.java b/messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelayReport.java similarity index 100% rename from messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelayReport.java rename to messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelayReport.java diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelayWorker.java b/messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelayWorker.java similarity index 100% rename from messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelayWorker.java rename to messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/outbox/OutboxRelayWorker.java diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRetryScheduler.java b/messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/outbox/OutboxRetryScheduler.java similarity index 100% rename from messaging/messaging-outbox-jdbc-postgresql/src/main/java/dev/caskeleton/messaging/outbox/OutboxRetryScheduler.java rename to messaging/messaging-reliability/src/main/java/dev/caskeleton/messaging/outbox/OutboxRetryScheduler.java diff --git a/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/IdempotentConsumerTest.java b/messaging/messaging-reliability/src/test/java/dev/caskeleton/messaging/inbox/IdempotentConsumerTest.java similarity index 100% rename from messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/IdempotentConsumerTest.java rename to messaging/messaging-reliability/src/test/java/dev/caskeleton/messaging/inbox/IdempotentConsumerTest.java diff --git a/messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/InboxOperationsTest.java b/messaging/messaging-reliability/src/test/java/dev/caskeleton/messaging/inbox/InboxOperationsTest.java similarity index 100% rename from messaging/messaging-inbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/inbox/InboxOperationsTest.java rename to messaging/messaging-reliability/src/test/java/dev/caskeleton/messaging/inbox/InboxOperationsTest.java diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxEnvelopeFactoryTest.java b/messaging/messaging-reliability/src/test/java/dev/caskeleton/messaging/outbox/OutboxEnvelopeFactoryTest.java similarity index 100% rename from messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxEnvelopeFactoryTest.java rename to messaging/messaging-reliability/src/test/java/dev/caskeleton/messaging/outbox/OutboxEnvelopeFactoryTest.java diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxOperationsTest.java b/messaging/messaging-reliability/src/test/java/dev/caskeleton/messaging/outbox/OutboxOperationsTest.java similarity index 91% rename from messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxOperationsTest.java rename to messaging/messaging-reliability/src/test/java/dev/caskeleton/messaging/outbox/OutboxOperationsTest.java index 7eb3f0a..2a52e0e 100644 --- a/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxOperationsTest.java +++ b/messaging/messaging-reliability/src/test/java/dev/caskeleton/messaging/outbox/OutboxOperationsTest.java @@ -35,17 +35,6 @@ class OutboxOperationsTest { throw new UnsupportedOperationException(); } - @Override - public List leaseBatch(int batchSize, Duration leaseDuration, Instant now) { - throw new UnsupportedOperationException(); - } - - @Override - public java.util.List claimBatch( - String owner, int batchSize, Duration leaseDuration, Instant now) { - return java.util.List.of(); - } - @Override public java.util.List claimBatch( String owner, int batchSize, Duration leaseDuration, Instant now, int maxAttempts) { @@ -58,7 +47,7 @@ class OutboxOperationsTest { String failureCode, Instant now, Instant nextAttemptAt) { - return markAmbiguous(lease, failureCode, now); + throw new UnsupportedOperationException(); } @Override @@ -73,12 +62,6 @@ class OutboxOperationsTest { throw new UnsupportedOperationException(); } - @Override - public dev.caskeleton.messaging.reliability.OutboxTransitionResult markAmbiguous( - dev.caskeleton.messaging.reliability.OutboxLease lease, String failureCode, Instant now) { - throw new UnsupportedOperationException(); - } - @Override public dev.caskeleton.messaging.reliability.OutboxTransitionResult markFailed( dev.caskeleton.messaging.reliability.OutboxLease lease, String failureCode, Instant now) { diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayTest.java b/messaging/messaging-reliability/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayTest.java similarity index 95% rename from messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayTest.java rename to messaging/messaging-reliability/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayTest.java index 8749e0b..fdc8591 100644 --- a/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayTest.java +++ b/messaging/messaging-reliability/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayTest.java @@ -199,15 +199,18 @@ class OutboxRelayTest { assertThat(repository.find(record.messageId()).orElseThrow().status()) .isEqualTo(OutboxStatus.FAILED); - assertThat(repository.leaseBatch(10, Duration.ofSeconds(30), NOW)).isEmpty(); + assertThat(repository.claimBatch("relay", 10, Duration.ofSeconds(30), NOW, Integer.MAX_VALUE)) + .isEmpty(); } @Test void aLeasedRowIsInvisibleToAnotherRelayInstance() { repository.append(OutboxFixtures.pending()); - List first = repository.leaseBatch(10, Duration.ofSeconds(30), NOW); - List second = repository.leaseBatch(10, Duration.ofSeconds(30), NOW); + List first = + repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW, Integer.MAX_VALUE); + List second = + repository.claimBatch("relay-b", 10, Duration.ofSeconds(30), NOW, Integer.MAX_VALUE); assertThat(first).hasSize(1); assertThat(second).isEmpty(); @@ -216,10 +219,11 @@ class OutboxRelayTest { @Test void anExpiredLeaseBecomesClaimableAgain() { repository.append(OutboxFixtures.pending()); - repository.leaseBatch(10, Duration.ofSeconds(30), NOW); + repository.claimBatch("relay-a", 10, Duration.ofSeconds(30), NOW, Integer.MAX_VALUE); - List reclaimed = - repository.leaseBatch(10, Duration.ofSeconds(30), NOW.plusSeconds(31)); + List reclaimed = + repository.claimBatch( + "relay-b", 10, Duration.ofSeconds(30), NOW.plusSeconds(31), Integer.MAX_VALUE); assertThat(reclaimed).hasSize(1); } @@ -352,8 +356,7 @@ final class InMemoryOutboxRepository implements OutboxRepository { * thing it stands in for hides exactly the defects it was written to catch, so this one follows * the same three-state rule. */ - @Override - public List leaseBatch(int batchSize, Duration leaseDuration, Instant now) { + private List lease(int batchSize, Duration leaseDuration, Instant now) { List leased = new ArrayList<>(); for (OutboxRecord record : List.copyOf(rows.values())) { if (leased.size() >= batchSize) { @@ -376,11 +379,10 @@ final class InMemoryOutboxRepository implements OutboxRepository { return List.copyOf(leased); } - @Override - public List claimBatch( + private List claimIgnoringBudget( String owner, int batchSize, Duration leaseDuration, Instant now) { List claimed = new ArrayList<>(); - for (OutboxRecord record : leaseBatch(batchSize, leaseDuration, now)) { + for (OutboxRecord record : lease(batchSize, leaseDuration, now)) { long token = tokens.merge(record.messageId(), 1L, Long::sum); owners.put(record.messageId(), owner); claimed.add(new OutboxLease(record, owner, token, now.plus(leaseDuration))); @@ -408,7 +410,7 @@ final class InMemoryOutboxRepository implements OutboxRepository { public List claimBatch( String owner, int batchSize, Duration leaseDuration, Instant now, int maxAttempts) { List claimed = new ArrayList<>(); - for (OutboxLease lease : claimBatch(owner, batchSize, leaseDuration, now)) { + for (OutboxLease lease : claimIgnoringBudget(owner, batchSize, leaseDuration, now)) { Instant nextAttempt = nextAttempts.get(lease.messageId()); boolean waiting = nextAttempt != null && now.isBefore(nextAttempt); if (waiting || lease.record().attempts() >= maxAttempts) { @@ -424,11 +426,13 @@ final class InMemoryOutboxRepository implements OutboxRepository { @Override public OutboxTransitionResult markAmbiguous( OutboxLease lease, String failureCode, Instant now, Instant nextAttemptAt) { - OutboxTransitionResult result = markAmbiguous(lease, failureCode, now); - if (result == OutboxTransitionResult.APPLIED) { - nextAttempts.put(lease.messageId(), nextAttemptAt); + if (!owns(lease)) { + return OutboxTransitionResult.STALE_LEASE; } - return result; + markAmbiguousUnfenced(lease.messageId(), failureCode); + owners.remove(lease.messageId()); + nextAttempts.put(lease.messageId(), nextAttemptAt); + return OutboxTransitionResult.APPLIED; } @Override @@ -452,16 +456,6 @@ final class InMemoryOutboxRepository implements OutboxRepository { return OutboxTransitionResult.APPLIED; } - @Override - public OutboxTransitionResult markAmbiguous(OutboxLease lease, String failureCode, Instant now) { - if (!owns(lease)) { - return OutboxTransitionResult.STALE_LEASE; - } - markAmbiguousUnfenced(lease.messageId(), failureCode); - owners.remove(lease.messageId()); - return OutboxTransitionResult.APPLIED; - } - @Override public OutboxTransitionResult markFailed(OutboxLease lease, String failureCode, Instant now) { if (!owns(lease)) { diff --git a/messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayWorkerTest.java b/messaging/messaging-reliability/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayWorkerTest.java similarity index 100% rename from messaging/messaging-outbox-jdbc-postgresql/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayWorkerTest.java rename to messaging/messaging-reliability/src/test/java/dev/caskeleton/messaging/outbox/OutboxRelayWorkerTest.java diff --git a/messaging/messaging-spring-boot-starter/build.gradle b/messaging/messaging-spring-boot-starter/build.gradle index acae04e..88f4717 100644 --- a/messaging/messaging-spring-boot-starter/build.gradle +++ b/messaging/messaging-spring-boot-starter/build.gradle @@ -31,8 +31,9 @@ dependencies { // which is why MessagingCoreAutoConfiguration consumed a MessagePublisher that nothing in // reach could produce (MSG-INT-003). implementation project(':messaging:messaging-runtime-core') - implementation project(':messaging:messaging-outbox-jdbc-postgresql') - implementation project(':messaging:messaging-inbox-jdbc-postgresql') + // The relay and consumer logic, and the PostgreSQL stores behind their ports. + implementation project(':messaging:messaging-reliability') + implementation project(':messaging:messaging-reliability-jdbc-postgresql') implementation project(':messaging:messaging-admin-api') implementation project(':messaging:messaging-admin-runtime') diff --git a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingOutboxRelayLifecycleTest.java b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingOutboxRelayLifecycleTest.java index 5c9e3b0..db68ef1 100644 --- a/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingOutboxRelayLifecycleTest.java +++ b/messaging/messaging-spring-boot-starter/src/test/java/dev/caskeleton/messaging/autoconfigure/MessagingOutboxRelayLifecycleTest.java @@ -171,28 +171,11 @@ class MessagingOutboxRelayLifecycleTest { throw new UnsupportedOperationException("not used by this test"); } - @Override - public List leaseBatch(int batchSize, Duration leaseDuration, Instant now) { - throw new UnsupportedOperationException("not used by this test"); - } - - @Override - public List claimBatch( - String owner, int batchSize, Duration leaseDuration, Instant now) { - throw new UnsupportedOperationException("not used by this test"); - } - @Override public OutboxTransitionResult markPublished(OutboxLease lease, Instant now) { throw new UnsupportedOperationException("not used by this test"); } - @Override - public OutboxTransitionResult markAmbiguous( - OutboxLease lease, String failureCode, Instant now) { - throw new UnsupportedOperationException("not used by this test"); - } - @Override public OutboxTransitionResult markAmbiguous( OutboxLease lease, String failureCode, Instant now, Instant nextAttemptAt) {