fix!: 재리뷰가 찾은 결함과 색인에서 빠졌던 발견 5건을 고친다

문제: 성숙도 리뷰 수정을 다시 읽은 재리뷰(Critical 0, 회귀 0)가
Important 27건·Minor 41건을 찾았다. 수정이 새로 만든 결함이 많았다.
@SystemInvoked가 권한 규칙의 탈출구가 됐고, 데모 계정 거부가 정확히
prod일 때만 걸렸고, outbox 순서 번호가 업그레이드 때 뒤로 갔고,
GraphQL union 아래 복잡도가 0이었고, 서블릿 오류 응답이 success:true로
나갔고, 호출자 deadline이 breaker를 열었다. 원래 리뷰 색인이 06 영역
발견 5건(06-10~14)을 빠뜨려 그동안 처리되지 않았다.

방법: 영역별 7묶음이 test-driven-development 절차로 실패 테스트를
먼저 쓰고 고쳤다. 결정은 재리뷰 README 4절에 적었다: 순서 번호는 V21이
기록한 전역 floor부터, 데모 계정은 local·test·demo 허용 목록,
@SystemInvoked bean은 기동 실패, complete 단계 실패는
IDEMPOTENCY_COMPLETION_UNKNOWN(결과 불명), 캐시 조건부 쓰기는 별도
SCRIPT lane, strict lane 정책은 build cache 입력, 끝나지 않는 재시도
행은 원인 코드·지표·PARKED 절차. 레지스트리가 선언만 하고 아무 코드도
내보내지 않던 redis.capability.* 지표 6개는 빼고 runbook을 실제
지표(redis.lane.*) 기준으로 고쳤다.

위험·한계: V21을 V22 없이 직접 고쳤다(어느 배포에도 아직 없음). 이
브랜치의 V21을 적용한 개발 DB는 다시 만들어야 한다. 서블릿 오류
디스패치를 보안 체인이 허용한다. 발행·저장소 포트 시그니처가 바뀌어
파생 서비스 코드가 깨진다. prod에서 fixture-로 시작하는 설정값은
기동을 막는다. httpclient failure-injection lane의 main 단계 중복
실행은 이전 결정대로 남겼다. 실제 CI runner 실행은 확인하지 못했다.

검증: ci, mainCheck, :integrationCheck, -p optional-platforms ci
integrationCheck performanceCheck, :performanceCheck, :releaseCheck
(-PgitRevision), verify-compose-profile-contracts(18 lane) 통과.
첫 실행의 실패 2건(FindSecBugs 1, sample 동시 생성 테스트의 v4 event
id)은 고친 뒤 ci·integrationCheck·mainCheck를 다시 돌려 통과했다.

Refs: docs/reviews/2026-11-03-rereview/README.md 3·4·6·7절
BREAKING CHANGE: outbox 발행 포트와 저장소 포트 시그니처, 순서 번호
첫 값(V21 floor 다음), 샘플 데모 계정 프로필 허용 목록, 새 오류 코드
IDEMPOTENCY_COMPLETION_UNKNOWN, prod의 fixture- 값 거부, 기동 rule은
static·의존 없음이어야 한다.
This commit is contained in:
DongHyeonka
2026-11-03 23:29:35 +09:00
parent 7ef8d0b4cb
commit bbe944bd3c
406 changed files with 13493 additions and 1656 deletions
+18 -6
View File
@@ -128,12 +128,15 @@ Focused tests drive the semantic adapters against `InMemorySemanticRedisCommands
served through `StubRedisRuntimeClient`. R1/R2 promotion requires a separate real Redis service lane; it may never
be silently skipped when selected.
Two real-server lanes. `redisContainerContractTest` (integration category, in `integrationCheck` and
this leaf's `mainCheck`) starts its own Redis with Testcontainers — the `redis` image of
`config/testing/container-images.properties`, loading `infra/redis-sdk/acl/all-accounts.acl` — and
runs `LiveRedisCacheRegionContractTest` and `LiveRedisDistributedLeaseContractTest` (the
`application-core` port contracts the in-memory fakes also pass), `LiveRedisSemanticPortsTest` and
`LiveRedisSemanticScriptAclProbeTest`; a skip fails it (review 2026-11-01 05-16). `RedisTestContainer`
Two real-server lanes. `redisContainerContractTest` (integration category, in `integrationCheck` only —
not in this leaf's `mainCheck` too, which ran it twice per push to main) starts its own Redis with Testcontainers — the `redis` image of
`config/testing/container-images.properties`, loading `infra/redis-sdk/acl/lane-fixture-accounts.acl` — and
runs `LiveRedisCacheRegionContractTest`, `LiveRedisDistributedLeaseContractTest` and
`LiveRedisIdempotencyStoreContractTest` (the `application-core` port contracts the in-memory fakes
also pass), `LiveRedisSemanticPortsTest` and `LiveRedisSemanticScriptAclProbeTest`; a skip fails it
(review 2026-11-01 05-16). The idempotency contract was on the explicit-only topology lane alone
until review 2026-11-03 RR-04-05, so the Lua state machine's stale-handle, renew-refusal and
retry cases reached a real server in no gate. `RedisTestContainer`
starts the server only when that lane names the ACL file.
`redisTopologyTest` is the external-topology lane, declared through `ca.redis-topology-lane`. This
@@ -177,6 +180,15 @@ deployment still gets exactly one client and one event loop. The `SCRIPT` lane i
the reason it exists: `SCRIPT LOAD` and `EVALSHA` belong to the advanced account, so the account
that reads a cache entry cannot execute a script.
The cache's conditional writes run on the same account but on a lane of their own,
`CacheScriptLane.CACHE_SCRIPT` (`app.capabilities.cache.maximum-script-connections`, contributed by
`RedisCacheConfig` as a `RedisLaneContribution`). Do not move them back to `SCRIPT`: that lane is the
idempotency, rate-limit and lease budget, and a cache under load must not take it (review 2026-11-03
README §4, RR-04-02). Each lane's wait, refusals and reconnects are meters of `redis-runtime`
(`RedisLaneMetrics`, `redis.lane.*`, tag `lane`), so whether the cache stays off `SCRIPT` is a query,
not an assumption. The `DEGRADED` health order ships with `redis-runtime`, which reports the
status (`config/module-defaults/redis-runtime.yml`, RR-04-10).
A Cluster transaction runs on one node, so the SDK's `RedisTransactionRunner` derives a routing key
from the watched keys (or an explicit `RedisSlotTag`) and borrows `RedisSdkLane.TRANSACTION`, which
is pinned to its key's owner, on the node that owns that slot. Routed leases are never pooled: a pooled connection is pinned to the previous caller's
+54 -7
View File
@@ -97,7 +97,10 @@ Connection lane limits and command queue limits are different contracts. In part
- `app.redis.capacity.maximum-in-flight-commands` bounds the per-connection command queue.
- `app.redis.capacity.maximum-script-connections` bounds physical SCRIPT lane connections
(default 8).
(default 8). Idempotency, rate limiting and leases share them.
- `app.capabilities.cache.maximum-script-connections` (`APP_CACHE_REDIS_MAXIMUM_SCRIPT_CONNECTIONS`)
bounds the cache's own script lane (`CacheScriptLane`, default 4), contributed only when the cache
binding is `redis`. See "Why the cache has its own script lane" below.
- `app.redis.lifecycle.acquire-timeout` (default 2s) is how long a borrow waits for a returned
connection when its lane is full.
@@ -153,7 +156,7 @@ Important contracts:
Every lookup returns a `CacheWriteCondition` naming the state it saw — `absent`, `vacant:<digest>`
(an invalidation marker, an entry of an older region generation, an expired entry) or
`present:<digest>`. A record that carries the condition applies only while the key is still in that
state (registered program, SCRIPT lane). This is what rejects a source load that began before an
state (registered program, cache script lane). This is what rejects a source load that began before an
invalidation and finished after it.
A hit also returns an opaque SHA-1 observation token over the stored bytes. SHA-1 here is **not a
@@ -181,10 +184,43 @@ example one written by `UPSERT` as `rev-1`), another schema version, a generatio
writer knows — is `NOT_RECORDED_CONDITION`.
Both conditional intents, and any record carrying a write condition, run registered scripts, so
they borrow the SCRIPT lane (the advanced account when one is configured, the application account
otherwise). The application account is not granted `EVALSHA`; on the REGULAR lane every conditional
write came back `INDETERMINATE`. Unconditioned `UPSERT` and `ONLY_IF_ABSENT` stay on the REGULAR
lane.
they run on the scripting account (the advanced account when one is configured, the application
account otherwise), on the cache's own script lane. The application account is not granted
`EVALSHA`; on the REGULAR lane every conditional write came back `INDETERMINATE`. Unconditioned
`UPSERT` and `ONLY_IF_ABSENT` stay on the REGULAR lane.
### Why the cache has its own script lane
Recording with the condition a lookup returned is the normal way to use this port, so nearly every
cache fill is a script. On the runtime's SCRIPT lane those fills shared eight connections per pod
with idempotency claims, rate-limit evaluations and lease calls, and a timed-out lease is discarded
rather than pooled, so each timeout also cost a reconnect. A slow Redis or a burst of misses after a
region invalidation filled the lane with cache work; an idempotency claim then waited out its
timeout and answered `UNAVAILABLE`, and a rate-limit call fell to its failure policy. The cache is a
speed feature that may degrade; idempotency, rate limiting and leases are correctness features that
must not (review 2026-11-03 README §4, RR-04-02).
So `CacheScriptLane.CACHE_SCRIPT` is a lane of its own — the same scripting account, its own
connections, its own ceiling (`app.capabilities.cache.maximum-script-connections`, default 4). A cache
that runs out of them degrades (`DEGRADED_UNAVAILABLE`, nothing sent) without touching the SCRIPT
lane. Alternatives not taken: a per-capability permit cap inside the shared SCRIPT lane (still one
pool of connections and one reconnect cost), and running the cache programs on the REGULAR lane
(grants the application account `EVALSHA`, which the account split exists to withhold).
Whether the split holds under load is visible per lane. With a `MeterRegistry` in the context,
`redis-runtime` registers these for every lane it pools — `REGULAR`, `SCRIPT`, `CACHE_SCRIPT` and the
typed SDK's — tagged `lane=<name>` (`RedisLaneMetrics`; rows in `docs/registries/metrics.yaml`):
| Meter | Type | Read it as |
| --- | --- | --- |
| `redis.lane.acquire.wait.seconds` (`outcome` = `admitted`/`refused`) | timer, histogram | how long a borrow queued for a connection; a growing admitted p99 comes before refusals |
| `redis.lane.acquire.refused.total` (`reason` = `saturated`/`closing`/`interrupted`/`connect_failed`) | counter | a borrow that got no connection, nothing sent; `saturated` on `SCRIPT` is the starvation this lane exists to prevent |
| `redis.lane.connections.opened.total` | counter | new connections; after warm-up every one is a reconnect |
| `redis.lane.connections.discarded.total` (`reason` = `invalidated`/`broken`) | counter | connections closed instead of pooled — `invalidated` is a timed-out or failed call (the lease is not reused), and is what the reconnects replace |
| `redis.lane.connections.active`, `redis.lane.connections.pending` | gauge | connections lent out now; borrows waiting now |
Every counter series exists from startup at zero, so an alert on "refusals above zero" sees the first
one. A lane refusal is also logged, per call, by `RedisSemanticFailureLogger`.
### Key invalidation
@@ -300,10 +336,21 @@ Useful focused checks:
`redisContainerContractTest` runs the cache-region and lease port contracts (the ones the in-memory
fakes pass) and the live semantic-port tests against a Redis it starts itself, under the standalone
topology's ACL file. It is in `integrationCheck` and `mainCheck`; the image is
topology's ACL file. It is in `integrationCheck` (integration-main.yml's `integration-check` job), and
only there: listing it in `mainCheck` as well ran it twice on every push to main. The image is
`config/testing/container-images.properties` `redis`.
Live Redis/topology qualification remains separate from fast unit tests. A release claim for a
specific Redis topology or ACL layout requires the corresponding live qualification lane; the
in-memory gateway is for deterministic contract tests, not evidence that an external Redis
deployment accepts the same network/TLS/ACL configuration.
## 아직 내보내지 않는 지표
`redis.capability.*` 여섯 개(operations.total, duration.seconds, admission.rejected.total,
inflight.total, readiness.total, lifecycle.drain.total)는 redis-production-capability 설계(Task 16)에
있었지만 어떤 코드도 내보내지 않았다. 레지스트리(`docs/registries/metrics.yaml`)와 incident runbook이 있는
것처럼 적고 있어서, 2026-11-03 재리뷰 수정에서 레지스트리에서 빼고 runbook을 지금 내보내는
`redis.lane.*`(redis-runtime, lane별 대기·거절·연결) 기준으로 고쳤다. capability 단위 지표가 필요해지면
operation 결과를 아는 semantic adapter(이 leaf)에서 내보내고, 그때 레지스트리에 다시 넣는다.
+13 -7
View File
@@ -48,7 +48,8 @@ dependencies {
// command surface the catalog exists to prevent. Zero imports.
// io.micrometer:micrometer-core — observation leaves this leaf as RedisObservation through a
// Consumer sink; binding it to a meter registry belongs to the composition root, not here.
// Zero imports.
// Zero imports. The connection lanes' meters, CACHE_SCRIPT's included, are redis-runtime's
// (RedisLaneMetrics): the lanes are pooled there, so that is where they are counted.
}
// Redis topology qualification is opt-in and owned by `ca.redis-topology-lane`: mode/property
@@ -69,8 +70,12 @@ redisTopologyLane {
// adapters' port contracts reached a real server in no stage, and the fake and the adapter could hide
// the same defect (05-02, 05-03). The image is config/testing/container-images.properties' `redis`,
// and the ACL file is infra/redis-sdk's own, so the accounts and their rights are the standalone
// topology's. Integration category: integrationCheck runs it, and so does this leaf's mainCheck, like
// the HTTP client's Toxiproxy lane. Sentinel, cluster and TLS remain the topology lane's.
// topology's. Integration category, so integrationCheck runs it — integration-main.yml's
// `integration-check` job, on every push to main and nightly. It is deliberately not in this leaf's
// mainCheck as well: `main-check` and `integration-check` are two jobs of the same workflow run, so a
// lane in both started a second Redis container and ran every live contract twice per push without
// covering anything the first run did not (found while fixing review 2026-11-03). Sentinel, cluster
// and TLS remain the topology lane's.
def redisAclDirectory = rootProject.projectDir.parentFile.toPath().resolve('infra/redis-sdk/acl').toFile()
strictTestLanes {
lane('redisContainerContractTest') {
@@ -78,15 +83,16 @@ strictTestLanes {
description = 'Runs the Redis semantic adapters\' live contracts against a Testcontainers Redis on the pinned image.'
requires 'dev.caskeleton.adapter.outbound.cache.redis.cache.LiveRedisCacheRegionContractTest',
'dev.caskeleton.adapter.outbound.cache.redis.lease.LiveRedisDistributedLeaseContractTest',
// The fake carries a Java copy of the idempotency Lua state machine; only a real
// server proves the two agree (05-02 was the fake copying the Lua's bug). It ran on
// the explicit-only topology lane alone, so no gate ever ran it (RR-04-05).
'dev.caskeleton.adapter.outbound.cache.redis.idempotency.LiveRedisIdempotencyStoreContractTest',
'dev.caskeleton.adapter.outbound.cache.redis.LiveRedisSemanticPortsTest',
'dev.caskeleton.adapter.outbound.cache.redis.capability.LiveRedisSemanticScriptAclProbeTest'
rejectSkipped = true
systemProperty 'redis.container.acl-file', new File(redisAclDirectory, 'all-accounts.acl').absolutePath
systemProperty 'redis.container.acl-file', new File(redisAclDirectory, 'lane-fixture-accounts.acl').absolutePath
inputDirectory redisAclDirectory
shouldRunAfter 'test'
}
}
tasks.named('mainCheck') {
dependsOn 'redisContainerContractTest'
}
@@ -50,7 +50,7 @@ io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnota
io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor
io.lettuce:lettuce-core:6.8.2.RELEASE=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-commons:1.16.7=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-core:1.16.7=testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-core:1.16.7=runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-jakarta9:1.16.7=testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-observation:1.16.7=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.netty:netty-buffer:4.2.17.Final=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
@@ -103,7 +103,7 @@ org.codehaus.plexus:plexus-container-default:2.1.0=checkstyle
org.codehaus.plexus:plexus-utils:3.3.0=checkstyle
org.dom4j:dom4j:2.2.0=spotbugs
org.hamcrest:hamcrest:3.0=testCompileClasspath,testRuntimeClasspath
org.hdrhistogram:HdrHistogram:2.2.2=testRuntimeClasspath
org.hdrhistogram:HdrHistogram:2.2.2=runtimeClasspath,testRuntimeClasspath
org.javassist:javassist:3.28.0-GA=checkstyle
org.jetbrains:annotations:17.0.0=testCompileClasspath,testRuntimeClasspath
org.jspecify:jspecify:1.0.1=annotationProcessor,checkstyle,compileClasspath,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
@@ -116,7 +116,7 @@ org.junit.platform:junit-platform-engine:6.0.3=testRuntimeClasspath
org.junit.platform:junit-platform-launcher:6.0.3=testRuntimeClasspath
org.junit:junit-bom:6.0.3=testCompileClasspath,testRuntimeClasspath
org.junit:junit-bom:6.1.0=spotbugs
org.latencyutils:LatencyUtils:2.0.3=testRuntimeClasspath
org.latencyutils:LatencyUtils:2.0.3=runtimeClasspath,testRuntimeClasspath
org.mockito:mockito-core:5.20.0=mockitoAgent,testCompileClasspath,testRuntimeClasspath
org.mockito:mockito-junit-jupiter:5.20.0=testCompileClasspath,testRuntimeClasspath
org.objenesis:objenesis:3.3=testRuntimeClasspath
@@ -0,0 +1,34 @@
package dev.caskeleton.adapter.outbound.cache.redis.cache;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisAccount;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisLane;
/**
* The cache's own script lane, so a cache's load cannot starve a correctness capability (review
* 2026-11-03 README §4, RR-04-02).
*
* <p>Every conditional cache write is a registered program, and a registered program needs the
* scripting account. On the runtime's SCRIPT lane those writes shared eight connections with
* idempotency claims, rate-limit evaluations and lease calls — and the normal way to use the cache
* (record with the condition the lookup returned) makes every cache fill a script. A slow Redis or
* a burst of misses after a region invalidation then held those connections with cache work while
* an idempotency claim waited out its timeout and failed. The cache is a speed feature and may
* degrade; idempotency and leases are correctness features and must not. They get separate
* connection budgets.
*
* <p>Same account as the SCRIPT lane: the grant is the same, only the connections and their ceiling
* ({@code app.capabilities.cache.maximum-script-connections}) are the cache's. Alternatives that
* were not taken: a per-capability permit cap inside the shared SCRIPT lane would still share the
* connections and their reconnect cost after a discarded lease; running cache programs on the
* REGULAR lane would grant the application account {@code EVALSHA}, which the account split exists
* to withhold.
*/
public enum CacheScriptLane implements RedisLane {
/** The cache region's conditional and revision-ordered writes. */
CACHE_SCRIPT;
@Override
public RedisAccount account() {
return RedisAccount.ADVANCED;
}
}
@@ -68,8 +68,10 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
private final SemanticRedisCall call;
/**
* The conditional writes, which are registered programs and so run on the SCRIPT lane: the
* application account the REGULAR lane authenticates as is not granted {@code EVALSHA}.
* The conditional writes, which are registered programs and so run on the scripting account: the
* application account the REGULAR lane authenticates as is not granted {@code EVALSHA}. They run
* on the cache's own lane ({@link CacheScriptLane}), not the runtime's SCRIPT lane, so a cache
* under load cannot hold the connections idempotency, rate limiting and leases need (RR-04-02).
*/
private final SemanticRedisCall scriptCall;
@@ -133,7 +135,7 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
Objects.requireNonNull(commandTimeout, "command timeout must be non-null"),
"cache");
this.scriptCall =
new SemanticRedisCall(owner, RedisConnectionKind.SCRIPT, commandTimeout, "cache");
new SemanticRedisCall(owner, CacheScriptLane.CACHE_SCRIPT, commandTimeout, "cache");
this.keys = Objects.requireNonNull(keys, "keys must be non-null");
this.keyDigest = Objects.requireNonNull(keyDigest, "key digest must be non-null");
this.encoder = Objects.requireNonNull(encoder, "encoder must be non-null");
@@ -1,6 +1,8 @@
package dev.caskeleton.adapter.outbound.cache.redis.capability;
import dev.caskeleton.adapter.outbound.cache.redis.cache.CacheScriptLane;
import dev.caskeleton.adapter.outbound.cache.redis.cache.RedisCacheRegionAdapter;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisLaneContribution;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisRuntimeSettings;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisRuntimeOwner;
import dev.caskeleton.adapter.outbound.support.CapabilitySelectors;
@@ -56,4 +58,24 @@ class RedisCacheConfig {
cache.getNegativeTtl(),
cache.getCommandTimeout());
}
/**
* The cache's own script lane and its connection ceiling.
*
* <p>The runtime pools a contributed lane like its own SCRIPT lane, on the same scripting
* account, but with connections and a ceiling of its own: the cache's conditional writes no
* longer compete with idempotency claims, rate-limit evaluations and lease calls for the SCRIPT
* lane (review 2026-11-03 README §4, RR-04-02).
*
* @param capabilities the capability settings
* @return the lane contribution
*/
@Bean
@ConditionalOnProperty(name = CapabilitySelectors.CACHE_DEFAULT_BINDING, havingValue = "redis")
RedisLaneContribution redisCacheScriptLane(RedisCapabilitySettings capabilities) {
RedisCapabilitySettings.Cache cache = capabilities.getCache();
cache.validate();
return new RedisLaneContribution(
CacheScriptLane.CACHE_SCRIPT, cache.getMaximumScriptConnections());
}
}
@@ -86,6 +86,7 @@ public class RedisCapabilitySettings {
private Duration positiveHardTtl = Duration.ofMinutes(5);
private Duration negativeTtl = Duration.ofSeconds(10);
private Duration minimumHardTtl = Duration.ofSeconds(1);
private int maximumScriptConnections = 4;
/**
* Reports whether the default binding selected Redis.
@@ -119,6 +120,10 @@ public class RedisCapabilitySettings {
if (keyVersion < 1) {
throw new CapabilityStartupValidationException("the cache key version must be positive");
}
if (maximumScriptConnections < 1) {
throw new CapabilityStartupValidationException(
"the cache script lane needs at least one connection");
}
}
public Map<String, String> getBindings() {
@@ -192,6 +197,23 @@ public class RedisCapabilitySettings {
public void setMinimumHardTtl(Duration minimumHardTtl) {
this.minimumHardTtl = minimumHardTtl;
}
/**
* The ceiling on the cache's own script-lane connections.
*
* <p>Separate from {@code app.redis.capacity.maximum-script-connections}, which idempotency,
* rate limiting and leases share: a cache is a speed feature and may run out of connections and
* degrade; it must not take them from the correctness features (review 2026-11-03 README §4).
*
* @return the ceiling, at least 1
*/
public int getMaximumScriptConnections() {
return maximumScriptConnections;
}
public void setMaximumScriptConnections(int maximumScriptConnections) {
this.maximumScriptConnections = maximumScriptConnections;
}
}
/** Edge rate-limit provider composition. */
@@ -28,11 +28,12 @@ import org.springframework.beans.factory.SmartInitializingSingleton;
* {@code {1}} only when the value it wrote reads back.
*
* <p>A cache-only deployment is probed too: its conditional writes and revision-ordered writes are
* registered programs on the same lane, and without the permission every one of them would come
* registered programs on the same account, and without the permission every one of them would come
* back degraded while plain reads looked healthy. Its probe runs only what the cache programs call
* ({@code GET}, {@code SET}), and a Redis it cannot reach at startup is logged, not fatal — a cache
* outage degrades, it never stops a pod from starting. A server that answers and refuses the
* scripts is a configuration error and fails startup as it does for the correctness roles.
* ({@code GET}, {@code SET}). Only a server that answers and refuses the scripts — an ACL refusal,
* {@code NOSCRIPT}, or a reply other than the value the probe wrote — is a configuration error and
* fails startup as it does for the correctness roles. Every other failure is a cache outage and is
* logged, not fatal: a cache outage degrades, it never stops a pod from starting.
*/
final class RedisSemanticScriptAclProbe implements SmartInitializingSingleton {
@@ -89,7 +90,7 @@ final class RedisSemanticScriptAclProbe implements SmartInitializingSingleton {
try {
probe(CACHE_PROBE_SCRIPT);
} catch (CapabilityStartupValidationException failure) {
if (!unreachable(failure.getCause())) {
if (refusedByTheServer(failure.getCause())) {
throw new CapabilityStartupValidationException(
"Redis cache is selected but its account cannot run the cache's conditional-write"
+ " scripts (SCRIPT LOAD/EVALSHA, GET, SET); every conditional record would"
@@ -98,15 +99,49 @@ final class RedisSemanticScriptAclProbe implements SmartInitializingSingleton {
failure.getCause());
}
LOG.warn(
"Redis cache is selected but Redis was unreachable at startup; the cache script"
+ " permission is unverified and the cache runs degraded until Redis answers",
"Redis cache is selected but the startup script probe could not complete; the cache"
+ " script permission is unverified and the cache runs degraded until Redis answers",
failure.getCause());
}
}
}
private static boolean unreachable(Throwable cause) {
return cause instanceof RedisLaneUnavailableException || cause instanceof TimeoutException;
/**
* Whether the cache probe failed because the server refused the cache's scripts.
*
* <p>"Refused" is a statement about the account, and only three answers make it: an ACL refusal
* ({@code NOPERM}, or a script's inner command denied), {@code NOSCRIPT} for the program the same
* connection had just loaded, and a reply other than the value the probe wrote. The check used to
* run the other way — only a lane refusal or a timeout counted as "unreachable" and everything
* else was a configuration error — so a connection that dropped mid-probe, a restarting Redis
* answering {@code LOADING}, or a busy one answering {@code BUSY} stopped a cache-only pod from
* starting, against this leaf's promise that a cache outage never does (review 2026-11-03
* RR-04-07).
*
* @param cause the probe's failure
* @return {@code true} when the account, not the server's availability, is at fault
*/
private static boolean refusedByTheServer(Throwable cause) {
for (Throwable current = cause;
current != null;
current = current.getCause() == current ? null : current.getCause()) {
if (current instanceof CapabilityStartupValidationException) {
// The probe ran and read back something other than what it wrote.
return true;
}
if (current instanceof RedisLaneUnavailableException || current instanceof TimeoutException) {
return false;
}
String message = current.getMessage();
if (message != null
&& (message.startsWith("NOPERM")
|| message.startsWith("NOSCRIPT")
|| message.contains("no permissions")
|| message.contains("ACL failure"))) {
return true;
}
}
return false;
}
private void probe(byte[] script) {
@@ -141,6 +141,12 @@ public final class IdempotencyScripts {
* it started from and the lease it asked for. A retry from that revision with the same lease is
* the same renewal; with a different lease it is a different request, because confirming it would
* leave the caller believing it holds the record for a lease the record does not carry.
*
* <p>The key's own TTL is stretched to cover the new lease. Claim sets it to the replay TTL,
* which may be shorter than a lease renewed past it; the key then expired while the holder was
* still executing, and the next request got a fresh claim and ran the side effect again (review
* 2026-11-03 RR-04-08). The TTL is only ever lengthened here — completion sets the replay TTL
* again.
*/
private static final String RENEW =
"""
@@ -177,6 +183,10 @@ public final class IdempotencyScripts {
redis.call('HSET', KEYS[1],
'leaseUntil', nowMillis + tonumber(ARGV[4]), 'rev', rev + 1,
'renewedFromRev', ARGV[2], 'renewedLease', ARGV[4])
local ttl = redis.call('PTTL', KEYS[1])
if ttl >= 0 and ttl < tonumber(ARGV[4]) then
redis.call('PEXPIRE', KEYS[1], ARGV[4])
end
return {'APPLIED', attempt, rev + 1, owner, '', ''}
""";
@@ -39,6 +39,10 @@ app:
# Floor on the hard TTL. Below it, entries expire faster than the round trip that wrote them:
# every read misses and every miss writes.
minimum-hard-ttl: ${APP_CACHE_REDIS_MINIMUM_HARD_TTL:1s}
# The cache's own script-lane connections, apart from the SCRIPT lane that idempotency, rate
# limiting and leases share (app.redis.capacity.maximum-script-connections). A cache under
# load runs out of these and degrades; it cannot take the correctness features' connections.
maximum-script-connections: ${APP_CACHE_REDIS_MAXIMUM_SCRIPT_CONNECTIONS:4}
idempotency:
# Redis store tuning only; the provider selector and lifecycle policy stay with the
# application's idempotency executor.
@@ -75,15 +79,8 @@ app:
maximum-cost: ${APP_RATE_LIMIT_MAXIMUM_COST:10}
cleanup-grace: ${APP_RATE_LIMIT_CLEANUP_GRACE:5s}
# redis-runtime 의 redisOptional 은 Redis 에 닿지 못하면 DEGRADED 를 보고한다. Boot 기본 순서
# (DOWN, OUT_OF_SERVICE, UP, UNKNOWN)에 없는 상태는 DOWN 보다 앞에 정렬돼, 다른 contributor 가 DOWN
# 이어도 전체 health 가 HTTP 200 DEGRADED 로 보였다(리뷰 05-23e). DEGRADED 를 DOWN 뒤, UP 앞에 둔다.
# 이 키를 정하는 module-defaults 는 이 파일 하나뿐이다. 서비스가 순서를 바꾸면 DEGRADED 를 넣어야 한다.
management:
endpoint:
health:
status:
order: DOWN,OUT_OF_SERVICE,DEGRADED,UP,UNKNOWN
# DEGRADED 상태 순서(management.endpoint.health.status.order)는 그 상태를 내는 redis-runtime 이
# 싣는다(config/module-defaults/redis-runtime.yml, 리뷰 2026-11-03 RR-04-10).
# /actuator/info 에 싣는 이 모듈의 상태(management.info.env). compose smoke 가 lane 이 실제로 켠
# capability 를 이 값으로 확인한다. 위 스위치를 해석한 값이고, 기본값은 RedisCapabilityConfig 의
@@ -1,5 +1,6 @@
package dev.caskeleton.adapter.outbound.cache.redis;
import dev.caskeleton.adapter.outbound.cache.redis.cache.CacheScriptLane;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.RedisTopologyEndpoint;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisCredentialResolver.RedisCredentials;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisRuntimeSettings;
@@ -71,6 +72,9 @@ public final class LiveRedisRuntime {
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
limits.put(kind, 4);
}
// The cache region's own script lane, which the composition root contributes when the cache
// binding is redis (RedisCacheConfig).
limits.put(CacheScriptLane.CACHE_SCRIPT, 4);
return new RedisRuntimeOwner(
new RedisTopologyClientFactory(
settings,
@@ -28,7 +28,7 @@ final class RedisTestContainer {
static final String ACL_FILE_PROPERTY = "redis.container.acl-file";
private static final String IMAGE_PROPERTY = "ca.test.image.redis";
private static final String ACL_IN_CONTAINER = "/etc/redis/acl/all-accounts.acl";
private static final String ACL_IN_CONTAINER = "/etc/redis/acl/lane-fixture-accounts.acl";
private static final int PORT = 6379;
private static GenericContainer<?> server;
@@ -60,8 +60,17 @@ class RedisCacheRegionAdapterTest {
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
limits.put(kind, 4);
}
return new RedisCacheRegionAdapter<>(
limits.put(CacheScriptLane.CACHE_SCRIPT, 4);
return cache(
now,
new RedisRuntimeOwner(client, limits, Duration.ofSeconds(1), Duration.ofSeconds(1)),
cacheKeys);
}
private RedisCacheRegionAdapter<String, String> cache(
Instant now, RedisRuntimeOwner owner, RedisCacheRegionAdapter.CacheKeys cacheKeys) {
return new RedisCacheRegionAdapter<>(
owner,
cacheKeys,
key -> "digest-" + key,
value -> value.getBytes(StandardCharsets.UTF_8),
@@ -428,6 +437,31 @@ class RedisCacheRegionAdapterTest {
.hasMessageContaining("ONLY_IF_SOURCE_REVISION_NEWER");
}
@Test
@DisplayName("멱등성·lease가 SCRIPT lane을 모두 잡고 있어도 캐시 조건부 기록은 캐시 전용 lane에서 기록된다")
void aConditionalCacheWriteDoesNotWaitForTheCorrectnessScriptLane() {
// given
Map<RedisLane, Integer> limits = new java.util.LinkedHashMap<>();
limits.put(RedisConnectionKind.REGULAR, 4);
limits.put(RedisConnectionKind.SCRIPT, 1);
limits.put(CacheScriptLane.CACHE_SCRIPT, 1);
try (RedisRuntimeOwner owner =
new RedisRuntimeOwner(
StubRedisRuntimeClient.serving(gateway),
limits,
Duration.ofMillis(100),
Duration.ofSeconds(1));
var heldByAnIdempotencyClaim = owner.borrow(RedisConnectionKind.SCRIPT)) {
RedisCacheRegionAdapter<String, String> cache = cache(T0, owner, keys);
// when
CacheRecordOutcome outcome = cache.record("order-1", "payload", newerThan("2"));
// then
assertThat(outcome).isEqualTo(CacheRecordOutcome.RECORDED);
}
}
@Test
@DisplayName("an unreachable Redis degrades: the cache is bypassed, nothing fails")
void anUnreachableRedisDegrades() {
@@ -23,6 +23,7 @@ class RedisCacheRegionContractTest extends CacheRegionContract {
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
limits.put(kind, 4);
}
limits.put(CacheScriptLane.CACHE_SCRIPT, 4);
return new RedisCacheRegionAdapter<>(
new RedisRuntimeOwner(
StubRedisRuntimeClient.serving(new InMemorySemanticRedisCommands()),
@@ -3,6 +3,9 @@ package dev.caskeleton.adapter.outbound.cache.redis.capability;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.Mockito.mock;
import dev.caskeleton.adapter.outbound.cache.redis.cache.CacheScriptLane;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisLaneContribution;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisLaneMetrics;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisRuntimeAutoConfiguration;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisRuntimeOwner;
import dev.caskeleton.adapter.outbound.support.startup.StartupFailureException;
@@ -11,6 +14,8 @@ import dev.caskeleton.application.idempotency.IdempotencyStorePort;
import dev.caskeleton.application.lease.DistributedLeasePort;
import dev.caskeleton.application.secret.SecretSourcePort;
import dev.caskeleton.shared.ratelimit.EdgeRateLimitPort;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.simple.SimpleMeterRegistry;
import java.time.Clock;
import java.time.Duration;
import java.util.Arrays;
@@ -94,6 +99,66 @@ class RedisCapabilityCompositionTest {
});
}
@Test
@DisplayName("Redis 캐시 binding은 설정한 상한으로 캐시 전용 script lane을 SCRIPT lane과 따로 기여한다")
void theCacheBindingContributesItsOwnScriptLane() {
runner
.withPropertyValues(
"app.capabilities.cache.bindings.default=redis",
"app.capabilities.cache.maximum-script-connections=3")
.run(
context -> {
// when
List<RedisLaneContribution> contributions =
List.copyOf(context.getBeansOfType(RedisLaneContribution.class).values());
// then
assertThat(context).hasNotFailed();
assertThat(contributions)
.containsExactly(new RedisLaneContribution(CacheScriptLane.CACHE_SCRIPT, 3));
});
}
@Test
@DisplayName("Redis 캐시 binding의 캐시 전용 lane은 SCRIPT lane과 따로 대기·거절·재연결 지표를 낸다")
void theCacheScriptLaneHasMetersOfItsOwn() {
runner
.withPropertyValues("app.capabilities.cache.bindings.default=redis")
.withBean(MeterRegistry.class, SimpleMeterRegistry::new)
.run(
context -> {
// when
MeterRegistry registry = context.getBean(MeterRegistry.class);
// then
assertThat(context).hasNotFailed();
for (String lane : List.of(CacheScriptLane.CACHE_SCRIPT.name(), "SCRIPT")) {
assertThat(
registry
.find(RedisLaneMetrics.CONNECTIONS_PENDING)
.tag("lane", lane)
.gauge())
.as("waiting borrows of lane %s", lane)
.isNotNull();
assertThat(
registry
.find(RedisLaneMetrics.ACQUIRE_REFUSED)
.tag("lane", lane)
.tag("reason", "saturated")
.counter())
.as("refusals of lane %s", lane)
.isNotNull();
assertThat(
registry
.find(RedisLaneMetrics.CONNECTIONS_OPENED)
.tag("lane", lane)
.counter())
.as("connections opened on lane %s", lane)
.isNotNull();
}
});
}
@Test
@DisplayName("a rate limiter with no configured policy is refused at startup")
void aRateLimiterWithoutPoliciesIsRefused() {
@@ -191,6 +191,85 @@ class RedisSemanticScriptAclProbeTest {
assertThat(failure).isNull();
}
@ParameterizedTest
@ValueSource(
strings = {
"LOADING Redis is loading the dataset in memory",
"Connection closed",
"BUSY Redis is busy running a script"
})
@DisplayName("cache만 고른 배포는 probe 도중 연결이 끊기거나 Redis가 아직 준비 중이어도 권한 오류로 보지 않고 기동한다")
void aCacheOnlyDeploymentStartsWhenTheProbeIsInterruptedByATransientFailure(String reply) {
// given
RedisCapabilitySettings capabilities = new RedisCapabilitySettings();
capabilities.getCache().getBindings().put("default", "redis");
RedisSemanticScriptAclProbe probe =
new RedisSemanticScriptAclProbe(
owner(gatewayFailingLoadWith(reply), new AtomicInteger()),
capabilities,
"prod:svc:shared:probe".getBytes(StandardCharsets.UTF_8),
Duration.ofSeconds(1));
// when
Throwable failure = catchThrowable(probe::afterSingletonsInstantiated);
// then
assertThat(failure).isNull();
}
@Test
@DisplayName("cache만 고른 배포도 probe가 기대와 다른 답을 받으면 기동에서 실패한다")
void aCacheOnlyDeploymentWhoseProbeReadsBackTheWrongValueFailsStartup() {
// given
RedisCapabilitySettings capabilities = new RedisCapabilitySettings();
capabilities.getCache().getBindings().put("default", "redis");
RedisSemanticScriptAclProbe probe =
new RedisSemanticScriptAclProbe(
owner(gatewayAnswering(List.of(0L)), new AtomicInteger()),
capabilities,
"prod:svc:shared:probe".getBytes(StandardCharsets.UTF_8),
Duration.ofSeconds(1));
// when
Throwable failure = catchThrowable(probe::afterSingletonsInstantiated);
// then
assertThat(failure)
.isInstanceOf(CapabilityStartupValidationException.class)
.hasMessageContaining("cache");
}
private static SemanticRedisCommands gatewayFailingLoadWith(String reply) {
return (SemanticRedisCommands)
Proxy.newProxyInstance(
SemanticRedisCommands.class.getClassLoader(),
new Class<?>[] {SemanticRedisCommands.class},
(proxy, method, arguments) ->
switch (method.getName()) {
case "loadScript" -> CompletableFuture.failedFuture(new RuntimeException(reply));
case "toString" -> "RedisSemanticScriptAclProbeTestGateway";
default ->
throw new UnsupportedOperationException(
"unexpected gateway method: " + method.getName());
});
}
private static SemanticRedisCommands gatewayAnswering(List<Object> reply) {
return (SemanticRedisCommands)
Proxy.newProxyInstance(
SemanticRedisCommands.class.getClassLoader(),
new Class<?>[] {SemanticRedisCommands.class},
(proxy, method, arguments) ->
switch (method.getName()) {
case "loadScript" -> CompletableFuture.completedFuture("probe-digest");
case "evaluateRegisteredForList" -> CompletableFuture.completedFuture(reply);
case "toString" -> "RedisSemanticScriptAclProbeTestGateway";
default ->
throw new UnsupportedOperationException(
"unexpected gateway method: " + method.getName());
});
}
private static RedisRuntimeOwner owner(SemanticRedisCommands gateway, AtomicInteger opened) {
RedisRuntimeClient client =
new RedisRuntimeClient() {
@@ -71,26 +71,15 @@ public final class RoleBasedFileAccessPolicy implements FileAccessPolicyPort {
if (permitted.stream().noneMatch(subject.roles()::contains)) {
throw denied(FileserverErrorCode.ACCESS_DENIED, "subject does not hold a permitted role");
}
if (descriptor.isPresent() && !isOwnedBy(descriptor.get(), subject)) {
// Ownership is FileAccessSubject.owns, the rule the port owns (owner decision 14): a file with
// no recorded owner predates ownership and belongs to nobody on the data plane. An admin role
// does not bypass it — acting on somebody else's file is what the audited management plane is
// for, and its operations carry no descriptor.
if (descriptor.isPresent() && !subject.owns(descriptor.get())) {
throw denied(FileserverErrorCode.ACCESS_DENIED, "subject does not own the file");
}
}
/**
* Whether the file was created by this caller, in this caller's tenant.
*
* <p>A file without a recorded owner predates ownership and belongs to nobody on the data plane.
* An admin role does not bypass this: acting on somebody else's file is what the audited
* management plane is for, and its operations carry no descriptor.
*/
private static boolean isOwnedBy(FileDescriptor descriptor, FileAccessSubject subject) {
return descriptor
.owner()
.filter(owner -> owner.principalId().equals(subject.principalId()))
.filter(owner -> owner.tenantId().equals(subject.tenant()))
.isPresent();
}
private static boolean isAnonymous(FileAccessSubject subject) {
return subject.principalId().equals(FileAccessSubject.anonymous().principalId());
}
@@ -0,0 +1,31 @@
package dev.caskeleton.adapter.outbound.fileserver.platform.security;
import dev.caskeleton.application.fileserver.api.security.FileAccessPolicyPort;
import dev.caskeleton.application.fileserver.api.security.FileAccessSubject;
import dev.caskeleton.application.fileserver.testkit.FileAccessPolicyPortContract;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
/** The production policy held to the ownership contract its fake is held to (RR-07-06). */
class RoleBasedFileAccessPolicyContractTest extends FileAccessPolicyPortContract {
private static final String READ = "ROLE_FILE_READ";
private static final String WRITE = "ROLE_FILE_WRITE";
private final RoleBasedFileAccessPolicy policy =
new RoleBasedFileAccessPolicy(Set.of(READ), Set.of(WRITE), Set.of("ROLE_FILE_ADMIN"), false);
@Override
protected FileAccessPolicyPort policy() {
return policy;
}
@Override
protected FileAccessSubject permittedSubject(String principalId, Optional<String> tenant) {
return new FileAccessSubject(
principalId,
Set.of(READ, WRITE),
tenant.map(value -> Map.of(FileAccessSubject.TENANT_ATTRIBUTE, value)).orElse(Map.of()));
}
}
+8 -2
View File
@@ -41,7 +41,11 @@ no reactive (`WebClient`/Reactor Netty) surface; it was removed on 2026-09-25 (d
cancels the request at the deadline, the JDK transport bounds its exchange timeout by the time
left, the experimental Jetty HTTP/3 transport aborts the request, and the caller gets
`HttpDeadlineExceededException`. A streaming download's setup (up to the response headers) runs
inside the same deadline; after hand-over each body gap is bounded by `read-idle`.
inside the same deadline; after hand-over each body gap is bounded by `read-idle` and nothing else,
on both transports — the JDK transport arms its own per-read timer for this, because the JDK client
has no socket read timeout. A body that fails after hand-over throws the same stable exceptions as
a buffered call (`HttpResponseTimeoutException` for a stalled body, `HttpResponseTruncatedException`
for a cut one).
- **Evidence.** Each attempt reports `NOT_SENT`, `SENT_NO_RESPONSE`, `RESPONSE_RECEIVED`, or
`PARTIAL_RESPONSE`. `NOT_SENT` is only claimed when a stage failure proves it.
- **Retry.** Never from the HTTP method alone. Idempotency, body replayability, evidence, deadline,
@@ -53,7 +57,9 @@ no reactive (`WebClient`/Reactor Netty) surface; it was removed on 2026-09-25 (d
whether the call throws them or returns them as a result (`RETURN_RESULT`). Any other 4xx (404,
409, 422 …) is a healthy answer, even when it reaches the caller as `HttpRemoteErrorException`.
Window, threshold and open-state wait are per profile under `clients[N].circuit-breaker.*`
(defaults are Resilience4j's; see `docs/httpclient/configuration-reference.md`).
(defaults are Resilience4j's; see `docs/httpclient/configuration-reference.md`). An attempt that
the caller's own deadline (`HttpOperation.deadline()`) ended is not counted at all: the caller ran
out of time, the upstream did not fail. An attempt the profile's `total-call` ended still counts.
- **Resources.** Connections and buffers are reclaimed after success, error, decode failure, size
rejection, and cancellation alike.
- **Secrets.** No URL, query value, path variable, token, cookie, or idempotency key reaches a log,
@@ -21,22 +21,35 @@ import org.springframework.http.client.JdkClientHttpRequestFactory;
* timeout is fixed per factory, so a factory is made per request with {@code min(response-header
* timeout, remaining deadline)} from {@link AttemptDeadline}. The factory only holds references;
* the client and the write executor are shared.
*
* <p>A streaming download cannot use that request. Its body is handed to the caller and may run for
* minutes, but Spring's timer only stops when the body stream is closed, so every JDK download
* longer than the response-header timeout was cut mid-body (review 2026-11-03 RR-04-03). Inside a
* streaming setup scope the factory makes a {@link JdkStreamingClientHttpRequest} instead: the
* JDK's own request timeout bounds the wait for headers, the attempt deadline cancels the exchange
* up to hand-over, and the body is bounded per gap by {@code read-idle}.
*/
final class DeadlineAwareJdkRequestFactory implements ClientHttpRequestFactory {
private final HttpClient client;
private final Executor writeExecutor;
private final Duration exchangeTimeout;
private final Duration readIdle;
DeadlineAwareJdkRequestFactory(HttpClient client, Duration exchangeTimeout) {
DeadlineAwareJdkRequestFactory(HttpClient client, Duration exchangeTimeout, Duration readIdle) {
this.client = Objects.requireNonNull(client, "jdk http client");
this.exchangeTimeout = Objects.requireNonNull(exchangeTimeout, "exchange timeout");
this.readIdle = Objects.requireNonNull(readIdle, "read idle timeout");
// The executor Spring would pick itself, chosen once instead of once per request.
this.writeExecutor = client.executor().orElseGet(SimpleAsyncTaskExecutor::new);
}
@Override
public ClientHttpRequest createRequest(URI uri, HttpMethod httpMethod) throws IOException {
if (AttemptDeadline.boundsSetupOnly()) {
return new JdkStreamingClientHttpRequest(
client, uri, httpMethod, AttemptDeadline.bound(exchangeTimeout), readIdle);
}
JdkClientHttpRequestFactory delegate = new JdkClientHttpRequestFactory(client, writeExecutor);
delegate.setReadTimeout(AttemptDeadline.bound(exchangeTimeout));
return delegate.createRequest(uri, httpMethod);
@@ -62,7 +62,8 @@ public final class JdkBlockingTransportProvider implements BlockingTransportProv
HttpClient client = clientFactory.create(profile, tlsMaterialResolver.apply(profile));
clients.put(new TransportResourceKey(profile.name(), generation), client);
ClientHttpRequestFactory factory =
new DeadlineAwareJdkRequestFactory(client, profile.timeout().responseHeader());
new DeadlineAwareJdkRequestFactory(
client, profile.timeout().responseHeader(), profile.timeout().readIdle());
listener.onRuntimeCreated(profile.name(), ID);
return factory;
}
@@ -0,0 +1,128 @@
package dev.caskeleton.adapter.outbound.httpclient.jdk;
import dev.caskeleton.adapter.outbound.httpclient.transport.AttemptDeadline;
import java.io.IOException;
import java.io.InputStream;
import java.io.InterruptedIOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.Objects;
import java.util.Set;
import java.util.TreeSet;
import java.util.concurrent.CancellationException;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.client.AbstractBufferingClientHttpRequest;
import org.springframework.http.client.ClientHttpResponse;
/**
* A JDK request for a streaming download, whose timeouts end at the response headers (review
* 2026-11-03 RR-04-03).
*
* <p>Three limits, each where the JDK can enforce it:
*
* <ul>
* <li>the JDK's own request timeout bounds the wait for the response headers and stops when they
* arrive;
* <li>the attempt deadline cancels the pending exchange, so connect, send and the header wait
* stay inside the call's total budget;
* <li>after hand-over, {@link ReadIdleInputStream} bounds each gap of the body by {@code
* read-idle} — the JDK client has no socket read timeout of its own.
* </ul>
*
* <p>The request body is buffered: a streaming download sends none or a small one, and the JDK
* needs a body publisher before the exchange starts.
*/
final class JdkStreamingClientHttpRequest extends AbstractBufferingClientHttpRequest {
/** Headers the JDK client refuses to set, as Spring's own JDK request lists them. */
private static final Set<String> RESTRICTED_HEADERS = restrictedHeaders();
private final HttpClient client;
private final URI uri;
private final HttpMethod method;
private final Duration responseHeaderTimeout;
private final Duration readIdle;
JdkStreamingClientHttpRequest(
HttpClient client,
URI uri,
HttpMethod method,
Duration responseHeaderTimeout,
Duration readIdle) {
this.client = Objects.requireNonNull(client, "jdk http client");
this.uri = Objects.requireNonNull(uri, "uri");
this.method = Objects.requireNonNull(method, "method");
this.responseHeaderTimeout =
Objects.requireNonNull(responseHeaderTimeout, "response header timeout");
this.readIdle = Objects.requireNonNull(readIdle, "read idle timeout");
}
@Override
public HttpMethod getMethod() {
return method;
}
@Override
public URI getURI() {
return uri;
}
@Override
protected ClientHttpResponse executeInternal(HttpHeaders headers, byte[] body)
throws IOException {
HttpRequest.Builder request =
HttpRequest.newBuilder(uri)
.timeout(responseHeaderTimeout)
.method(
method.name(),
body.length == 0
? HttpRequest.BodyPublishers.noBody()
: HttpRequest.BodyPublishers.ofByteArray(body));
headers.forEach(
(name, values) -> {
if (!RESTRICTED_HEADERS.contains(name)) {
values.forEach(value -> request.header(name, value));
}
});
CompletableFuture<HttpResponse<InputStream>> pending =
client.sendAsync(request.build(), HttpResponse.BodyHandlers.ofInputStream());
AttemptDeadline.abortOnExpiry(() -> pending.cancel(true));
HttpResponse<InputStream> response = await(pending);
return new JdkStreamingClientHttpResponse(
response, new ReadIdleInputStream(response.body(), readIdle));
}
private static HttpResponse<InputStream> await(
CompletableFuture<HttpResponse<InputStream>> pending) throws IOException {
try {
return pending.get();
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
pending.cancel(true);
InterruptedIOException failure = new InterruptedIOException("streaming setup interrupted");
failure.initCause(interrupted);
throw failure;
} catch (CancellationException cancelled) {
// Only the attempt deadline cancels this future; the gateway names the deadline.
throw new IOException("streaming setup cancelled at the attempt deadline", cancelled);
} catch (ExecutionException failed) {
Throwable cause = failed.getCause();
if (cause instanceof IOException io) {
throw io;
}
throw new IOException("streaming setup failed", cause);
}
}
private static Set<String> restrictedHeaders() {
Set<String> restricted = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
restricted.addAll(Set.of("connection", "content-length", "expect", "host", "upgrade"));
return restricted;
}
}
@@ -0,0 +1,63 @@
package dev.caskeleton.adapter.outbound.httpclient.jdk;
import java.io.IOException;
import java.io.InputStream;
import java.net.http.HttpResponse;
import java.util.Objects;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.HttpStatusCode;
import org.springframework.http.client.ClientHttpResponse;
/** The response of a {@link JdkStreamingClientHttpRequest}, body bounded per gap. */
final class JdkStreamingClientHttpResponse implements ClientHttpResponse {
private final HttpResponse<InputStream> response;
private final InputStream body;
private final HttpHeaders headers = new HttpHeaders();
JdkStreamingClientHttpResponse(HttpResponse<InputStream> response, InputStream body) {
this.response = Objects.requireNonNull(response, "jdk response");
this.body = Objects.requireNonNull(body, "body stream");
// HTTP/2 pseudo-headers (":status") are protocol framing, not response headers.
response
.headers()
.map()
.forEach(
(name, values) -> {
if (!name.startsWith(":")) {
headers.addAll(name, values);
}
});
}
@Override
public HttpStatusCode getStatusCode() {
return HttpStatusCode.valueOf(response.statusCode());
}
@Override
public String getStatusText() {
HttpStatus known = HttpStatus.resolve(response.statusCode());
return known == null ? "" : known.getReasonPhrase();
}
@Override
public HttpHeaders getHeaders() {
return headers;
}
@Override
public InputStream getBody() {
return body;
}
@Override
public void close() {
try {
body.close();
} catch (IOException ignored) {
// Closing releases the exchange; a failure to close has nothing left to release.
}
}
}
@@ -0,0 +1,90 @@
package dev.caskeleton.adapter.outbound.httpclient.jdk;
import java.io.FilterInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.SocketTimeoutException;
import java.time.Duration;
import java.util.Objects;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
/**
* A JDK response body whose every read is bounded by {@code read-idle} (review 2026-11-03
* RR-04-03).
*
* <p>The JDK client has no socket read timeout, so a streamed body that stopped arriving held the
* caller's thread and the connection forever. Each read arms a timer on the JDK's shared delay
* scheduler; when it fires the stream is closed, which wakes the blocked read, and the read reports
* a {@link SocketTimeoutException} — the same exception Apache's socket timeout raises, so one
* classifier reading names both. A read that returns first disarms its timer.
*/
final class ReadIdleInputStream extends FilterInputStream {
@FunctionalInterface
private interface Read {
long run() throws IOException;
}
private final Duration readIdle;
private final AtomicBoolean idled = new AtomicBoolean();
ReadIdleInputStream(InputStream delegate, Duration readIdle) {
super(Objects.requireNonNull(delegate, "body stream"));
this.readIdle = Objects.requireNonNull(readIdle, "read idle timeout");
}
@Override
public int read() throws IOException {
return (int) guarded(super::read);
}
@Override
public int read(byte[] buffer, int offset, int length) throws IOException {
return (int) guarded(() -> super.read(buffer, offset, length));
}
@Override
public long skip(long count) throws IOException {
return guarded(() -> super.skip(count));
}
private long guarded(Read read) throws IOException {
requireNotIdled();
CompletableFuture<Void> timer = new CompletableFuture<>();
CompletableFuture<Void> expiry =
timer.completeOnTimeout(null, readIdle.toNanos(), TimeUnit.NANOSECONDS).thenRun(this::idle);
try {
long result = read.run();
requireNotIdled();
return result;
} catch (IOException failure) {
requireNotIdled();
throw failure;
} finally {
// Completing first skips the idle action and lets the scheduler drop the pending task.
timer.cancel(false);
expiry.cancel(false);
}
}
private void idle() {
idled.set(true);
try {
in.close();
} catch (IOException ignored) {
// The read this wakes reports the timeout; a failed close has nothing more to say.
}
}
/**
* After the timer fired, whatever the read returned — data, end of stream or "closed" — is not
* the rest of the body.
*/
private void requireNotIdled() throws SocketTimeoutException {
if (idled.get()) {
throw new SocketTimeoutException("no response body bytes within the read-idle timeout");
}
}
}
@@ -17,6 +17,13 @@ public interface AttemptCircuitBreaker {
void onError(long durationNanos, Throwable failure);
/**
* Gives back a permission without recording an outcome, for an attempt that says nothing about
* the upstream (an attempt its caller's own deadline ended). Half-open trial slots are returned;
* the window is left as it was.
*/
void releasePermission();
String state();
static AttemptCircuitBreaker alwaysClosed() {
@@ -36,6 +43,11 @@ public interface AttemptCircuitBreaker {
// Nothing to record.
}
@Override
public void releasePermission() {
// Nothing was taken.
}
@Override
public String state() {
return "CLOSED";
@@ -60,6 +72,11 @@ public interface AttemptCircuitBreaker {
delegate.onError(durationNanos, TimeUnit.NANOSECONDS, failure);
}
@Override
public void releasePermission() {
delegate.releasePermission();
}
@Override
public String state() {
return delegate.getState().name();
@@ -7,6 +7,7 @@ import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpFailureMetadata;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRateLimitRejectedException;
import java.util.Objects;
import java.util.Optional;
import java.util.function.BooleanSupplier;
import java.util.function.Function;
import java.util.function.Supplier;
@@ -21,6 +22,11 @@ import java.util.function.Supplier;
* <p>A local rejection (rate limiter or bulkhead) is deliberately <em>not</em> recorded as a
* circuit error: the upstream never saw the request, and counting our own back-pressure as upstream
* failure would open the breaker on a healthy dependency.
*
* <p>An attempt its caller abandoned is not recorded either, for the same reason: an attempt that a
* caller's own deadline ended while the upstream was still answering within the profile's limits is
* the caller running out of time, not the upstream failing (review 2026-11-03 RR-04-01). Its
* permission is given back instead.
*/
public final class AttemptResiliencePipeline {
@@ -79,7 +85,24 @@ public final class AttemptResiliencePipeline {
* @return the attempt's value, whether or not it represents a remote failure
*/
public <T> T execute(AttemptCall<T> call, Function<T, Optional<Throwable>> remoteFailure) {
return execute(call, remoteFailure, () -> false);
}
/**
* Runs one attempt, recording nothing against the upstream when its caller abandoned it.
*
* @param call the attempt, including any redirect hops it follows
* @param remoteFailure returns the failure to record when the value represents an upstream error
* @param abandonedByCaller read after the attempt failed: whether the caller's own deadline ended
* it, in which case the failure is not the upstream's and the breaker records nothing
* @return the attempt's value, whether or not it represents a remote failure
*/
public <T> T execute(
AttemptCall<T> call,
Function<T, Optional<Throwable>> remoteFailure,
BooleanSupplier abandonedByCaller) {
Objects.requireNonNull(remoteFailure, "remote failure classifier");
Objects.requireNonNull(abandonedByCaller, "abandoned-by-caller signal");
if (!circuitBreaker.tryAcquirePermission()) {
rejections.circuitOpen();
throw new HttpCircuitOpenException(
@@ -110,7 +133,11 @@ public final class AttemptResiliencePipeline {
return result;
} catch (Throwable failure) {
releaseAttemptPermits();
circuitBreaker.onError(System.nanoTime() - started, failure);
if (abandonedByCaller.getAsBoolean()) {
circuitBreaker.releasePermission();
} else {
circuitBreaker.onError(System.nanoTime() - started, failure);
}
throw translate(failure);
}
}
@@ -10,11 +10,31 @@ import java.util.Objects;
*
* <p>Everything — pool acquire, DNS, connect, TLS, request write, response read, and every retry
* backoff — is spent inside this single budget.
*
* @param at the instant the call ends
* @param source who set that instant, which decides what an attempt it ends says about the upstream
*/
public record Deadline(Instant at) {
public record Deadline(Instant at, Source source) {
/**
* Who set the instant a call ends at.
*
* <p>The two say different things when they end an attempt. The profile's total-call timeout is
* the operator's statement of how long a healthy upstream may take, so an attempt it ends is the
* upstream's failure. A caller's own deadline is a statement about the caller — it had only that
* much time left — so an attempt it ends says nothing about the upstream (review 2026-11-03
* RR-04-01).
*/
public enum Source {
/** The profile's {@code timeout.total-call}. */
PROFILE_TOTAL_CALL,
/** The caller's {@code HttpOperation.deadline()}, when it is earlier than the profile's. */
CALLER
}
public Deadline {
Objects.requireNonNull(at, "deadline instant");
Objects.requireNonNull(source, "deadline source");
}
public Duration remaining(Clock clock) {
@@ -26,6 +46,11 @@ public record Deadline(Instant at) {
return !clock.instant().isBefore(at);
}
/** Whether the instant is the caller's own budget rather than the profile's total-call. */
public boolean setByCaller() {
return source == Source.CALLER;
}
/** The earlier of two deadlines; a parent budget can only ever shrink a child's. */
public Deadline earliest(Deadline other) {
return at.isBefore(other.at) ? this : other;
@@ -9,6 +9,9 @@ import java.util.Optional;
/**
* Derives the effective deadline from the caller's parent budget and the profile total-call timeout
* (design §15.2): {@code effectiveDeadline = min(parentDeadline, now + totalCallTimeout)}.
*
* <p>The result also records which of the two won. A tie goes to the profile: the profile's limit
* would have ended the call at that instant anyway.
*/
public final class DeadlineCalculator {
@@ -17,6 +20,9 @@ public final class DeadlineCalculator {
Objects.requireNonNull(totalCall, "total call timeout");
Objects.requireNonNull(clock, "clock");
Instant local = clock.instant().plus(totalCall);
return new Deadline(parent.map(value -> value.isBefore(local) ? value : local).orElse(local));
return parent
.filter(value -> value.isBefore(local))
.map(value -> new Deadline(value, Deadline.Source.CALLER))
.orElseGet(() -> new Deadline(local, Deadline.Source.PROFILE_TOTAL_CALL));
}
}
@@ -63,6 +63,7 @@ public final class BlockingAttemptExecutor {
try (AttemptDeadline.Scope scope = AttemptDeadline.open(deadline.remaining(support.clock()))) {
return executeWithin(
scope,
deadline,
runtime,
prepared,
responseType,
@@ -76,6 +77,7 @@ public final class BlockingAttemptExecutor {
private <T> AttemptOutcome<T> executeWithin(
AttemptDeadline.Scope scope,
Deadline deadline,
BlockingClientRuntime runtime,
PreparedOperation prepared,
ResponseType<T> responseType,
@@ -146,7 +148,12 @@ public final class BlockingAttemptExecutor {
tracker.firstByteDelivered());
}
},
outcome -> upstreamFailure(outcome, baseMetadata));
outcome -> upstreamFailure(outcome, baseMetadata),
// The caller's own deadline ending the attempt is not the upstream failing. The
// profile's total-call ending it is: that limit is the operator's statement of
// how long a healthy upstream may take, and a slow-drip upstream that never trips
// a per-gap timeout is seen by the breaker only through it (RR-04-01).
() -> deadline.setByCaller() && scope.expired());
return finish(observation, scope, attemptOutcome);
} catch (HttpClientException stable) {
return finish(
@@ -5,6 +5,7 @@ import dev.caskeleton.adapter.outbound.httpclient.api.HttpStatus;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpDeadlineExceededException;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpFailureMetadata;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRemoteErrorException;
import dev.caskeleton.adapter.outbound.httpclient.api.operation.AttemptStage;
import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence;
import dev.caskeleton.adapter.outbound.httpclient.api.operation.HttpOperation;
import dev.caskeleton.adapter.outbound.httpclient.api.result.BlockingStreamingResponse;
@@ -78,9 +79,11 @@ public final class BlockingStreamingGateway {
// which deregisters the transport's abort, so the body is then bounded by read-idle per gap
// and a long download is not cut at an arbitrary total. Without the scope a setup was bounded
// only by per-gap stage timeouts, and an upstream that held or dripped its headers kept the
// caller past its deadline (finding 05-06).
// caller past its deadline (finding 05-06). It is a setup scope, not an attempt scope: the
// JDK transport arms an exchange-wide timeout for an attempt, which cut every download
// longer than the response-header timeout (RR-04-03).
AttemptDeadline.Scope setupDeadline =
AttemptDeadline.open(setup.deadline().remaining(runtime.support().clock()));
AttemptDeadline.openStreamSetup(setup.deadline().remaining(runtime.support().clock()));
try (setupDeadline) {
response =
spec.exchange(
@@ -102,9 +105,24 @@ public final class BlockingStreamingGateway {
.withStatus(new HttpStatus(status))
.withEvidence(ExecutionEvidence.RESPONSE_RECEIVED));
}
// A body that fails after hand-over reaches the caller as the same stable
// exceptions a buffered call throws, classified by the transport's own
// classifier at the body stage — not as the engine's bare IOException, which
// carried no category, no evidence and nothing to log (RR-04-03).
InputStream bounded =
new CountingBoundedInputStream(
rawResponse.getBody(), limiter::recordWireBytes);
new ClassifiedBodyInputStream(
new CountingBoundedInputStream(
rawResponse.getBody(), limiter::recordWireBytes),
bodyFailure ->
runtime
.support()
.exceptionMapper()
.map(
runtime
.failureClassifier()
.classify(bodyFailure, AttemptStage.RESPONSE_BODY),
metadata,
bodyFailure));
return new DefaultBlockingStreamingResponse(
new HttpStatus(status), responseHeaders, bounded, rawResponse::close);
},
@@ -0,0 +1,64 @@
package dev.caskeleton.adapter.outbound.httpclient.restclient;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpClientException;
import java.io.FilterInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.Objects;
import java.util.function.Function;
/**
* A handed-over response body whose read failures reach the caller as stable exceptions (review
* 2026-11-03 RR-04-03).
*
* <p>A streamed body is read after the gateway has returned, so the gateway's own failure mapping
* never sees it. Without this a read-idle timeout or a cut connection reached the caller as the
* engine's bare {@code IOException} — no category, no evidence, nothing a retry policy or a log
* could branch on. Closing is the caller's own act and is passed through unclassified.
*/
final class ClassifiedBodyInputStream extends FilterInputStream {
private final Function<IOException, HttpClientException> classifier;
ClassifiedBodyInputStream(
InputStream delegate, Function<IOException, HttpClientException> classifier) {
super(Objects.requireNonNull(delegate, "body stream"));
this.classifier = Objects.requireNonNull(classifier, "body failure classifier");
}
@Override
public int read() throws IOException {
try {
return super.read();
} catch (IOException failure) {
throw classifier.apply(failure);
}
}
@Override
public int read(byte[] buffer, int offset, int length) throws IOException {
try {
return super.read(buffer, offset, length);
} catch (IOException failure) {
throw classifier.apply(failure);
}
}
@Override
public long skip(long count) throws IOException {
try {
return super.skip(count);
} catch (IOException failure) {
throw classifier.apply(failure);
}
}
@Override
public int available() throws IOException {
try {
return super.available();
} catch (IOException failure) {
throw classifier.apply(failure);
}
}
}
@@ -28,6 +28,11 @@ import java.util.concurrent.TimeUnit;
* <p>The scope is thread-confined because a blocking attempt runs on the caller's thread from
* request creation to the last body byte. When no scope is open — a transport used outside the
* gateway — both methods leave the transport's own behaviour unchanged.
*
* <p>A streaming download opens a different scope ({@link #openStreamSetup(Duration)}): it bounds
* the exchange only up to the response headers and is closed when the body is handed over. A
* transport whose engine timeout would otherwise keep running over the body reads {@link
* #boundsSetupOnly()} and keeps that timeout off it (review 2026-11-03 RR-04-03).
*/
public final class AttemptDeadline {
@@ -41,8 +46,34 @@ public final class AttemptDeadline {
* @param remaining the time left before the logical call's effective deadline
*/
public static Scope open(Duration remaining) {
return open(remaining, false);
}
/**
* Opens the deadline scope for a streaming download's setup on the current thread.
*
* <p>The caller closes it when the response is handed over, so the deadline never reaches the
* body: a download may legitimately outlast any fixed total (design §15.3).
*
* @param remaining the time left before the logical call's effective deadline
*/
public static Scope openStreamSetup(Duration remaining) {
return open(remaining, true);
}
/**
* Whether the current scope bounds only a streaming setup, so a timeout the transport arms must
* not outlive the response headers.
*/
public static boolean boundsSetupOnly() {
Scope scope = CURRENT.get();
return scope != null && scope.setupOnly;
}
private static Scope open(Duration remaining, boolean setupOnly) {
Objects.requireNonNull(remaining, "remaining deadline");
Scope scope = new Scope(remaining.isNegative() ? Duration.ZERO : remaining, CURRENT.get());
Scope scope =
new Scope(remaining.isNegative() ? Duration.ZERO : remaining, setupOnly, CURRENT.get());
CURRENT.set(scope);
return scope;
}
@@ -78,6 +109,7 @@ public final class AttemptDeadline {
public static final class Scope implements AutoCloseable {
private final long endNanos;
private final boolean setupOnly;
private final Scope outer;
private final CompletableFuture<Void> timer = new CompletableFuture<>();
private final CompletableFuture<Void> expiry;
@@ -85,8 +117,9 @@ public final class AttemptDeadline {
private boolean fired;
private boolean closed;
private Scope(Duration remaining, Scope outer) {
private Scope(Duration remaining, boolean setupOnly, Scope outer) {
this.endNanos = System.nanoTime() + remaining.toNanos();
this.setupOnly = setupOnly;
this.outer = outer;
// The JDK's shared delay scheduler, the same one Spring's JDK request uses for its timeout.
// Completing the future on close() also cancels the scheduled task, so a finished attempt
@@ -34,7 +34,8 @@ class DeadlineCalculatorTest {
@Test
void refusesAttemptWhenBackoffConsumesRemainingBudget() {
Deadline deadline = new Deadline(Instant.parse("2026-08-08T00:00:01Z"));
Deadline deadline =
new Deadline(Instant.parse("2026-08-08T00:00:01Z"), Deadline.Source.PROFILE_TOTAL_CALL);
Optional<AttemptBudget> result =
new AttemptBudgetCalculator(clock)
.nextAttempt(
@@ -44,7 +45,8 @@ class DeadlineCalculatorTest {
@Test
void grantsAttemptWhenBudgetCoversBackoffMinimumAndCleanup() {
Deadline deadline = new Deadline(Instant.parse("2026-08-08T00:00:02Z"));
Deadline deadline =
new Deadline(Instant.parse("2026-08-08T00:00:02Z"), Deadline.Source.PROFILE_TOTAL_CALL);
Optional<AttemptBudget> result =
new AttemptBudgetCalculator(clock)
.nextAttempt(
@@ -60,7 +62,8 @@ class DeadlineCalculatorTest {
assertThatThrownBy(
() ->
guard.requireTimeRemaining(
new Deadline(Instant.parse("2026-08-07T23:59:59Z")),
new Deadline(
Instant.parse("2026-08-07T23:59:59Z"), Deadline.Source.PROFILE_TOTAL_CALL),
CoreFixtures.notSentMetadata()))
.isInstanceOf(HttpDeadlineExceededException.class);
assertThatThrownBy(
@@ -147,7 +147,7 @@ class RetryCoordinatorTest {
@Override
public Deadline deadline() {
return new Deadline(clock.instant().plusSeconds(5));
return new Deadline(clock.instant().plusSeconds(5), Deadline.Source.PROFILE_TOTAL_CALL);
}
@Override
@@ -202,7 +202,7 @@ class RetryCoordinatorTest {
@Override
public Deadline deadline() {
return new Deadline(clock.instant().plusSeconds(5));
return new Deadline(clock.instant().plusSeconds(5), Deadline.Source.PROFILE_TOTAL_CALL);
}
@Override
@@ -5,20 +5,28 @@ import static org.assertj.core.api.Assertions.catchThrowable;
import dev.caskeleton.adapter.outbound.httpclient.api.OperationName;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpCircuitOpenException;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpDeadlineExceededException;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRemoteErrorException;
import dev.caskeleton.adapter.outbound.httpclient.api.operation.HttpOperation;
import dev.caskeleton.adapter.outbound.httpclient.api.result.HttpCallResult;
import dev.caskeleton.adapter.outbound.httpclient.api.result.ResponseType;
import dev.caskeleton.adapter.outbound.httpclient.profile.CircuitBreakerSettings;
import dev.caskeleton.adapter.outbound.httpclient.profile.TimeoutSettings;
import dev.caskeleton.adapter.outbound.httpclient.profile.TransportType;
import dev.caskeleton.adapter.outbound.httpclient.testkit.ClientProfiles;
import dev.caskeleton.adapter.outbound.httpclient.testkit.MockHttpServer;
import dev.caskeleton.adapter.outbound.httpclient.testkit.TestGateways;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.EnumSource;
/**
* What the per-profile circuit breaker counts as an upstream failure, end to end (finding 05-05).
@@ -31,6 +39,10 @@ class CircuitBreakerStatusAccountingTest {
private static final int CALLS = 200;
private static final int LIBRARY_DEFAULT_WINDOW = 100;
private static final int SMALL_WINDOW = 5;
private static final int TINY_WINDOW = 2;
private static final Duration HEALTHY_ANSWER_TIME = Duration.ofMillis(600);
private static final Duration CALLER_BUDGET = Duration.ofMillis(200);
private static final Duration PROFILE_TOTAL_CALL = Duration.ofSeconds(1);
@Test
@DisplayName("정상 응답인 404가 200번 이어져도 circuit이 열리지 않고 모든 호출이 상대 서버에 닿는다")
@@ -158,6 +170,113 @@ class CircuitBreakerStatusAccountingTest {
}
}
@ParameterizedTest(name = "{0}")
@EnumSource(
value = TransportType.class,
names = {"APACHE", "JDK"})
@DisplayName("호출자 deadline으로 끊은 시도는 상대 장애로 세지 않아 창을 채워도 다음 호출이 상대 서버에 닿는다")
void attemptsCutShortByTheCallersOwnDeadlineDoNotOpenTheCircuit(TransportType transport)
throws Exception {
try (MockHttpServer server = MockHttpServer.start();
TestGateways.Harness harness =
TestGateways.forProfile(
ClientProfiles.builder("caller-deadline")
.baseUrl(server.uri("/"))
.transport(transport)
.circuitBreaker(
new CircuitBreakerSettings(
50, SMALL_WINDOW, SMALL_WINDOW, Duration.ofMinutes(1), 1))
.build())) {
// given
for (int i = 0; i <= SMALL_WINDOW; i++) {
server.enqueueDelayedHeaders(
200,
"application/json",
"\"ok\"".getBytes(StandardCharsets.UTF_8),
HEALTHY_ANSWER_TIME);
}
List<Throwable> windowFailures = new ArrayList<>();
for (int i = 0; i < SMALL_WINDOW; i++) {
windowFailures.add(
catchThrowable(
() -> getUserWithin(harness, Optional.of(Instant.now().plus(CALLER_BUDGET)))));
}
assertThat(windowFailures)
.allSatisfy(f -> assertThat(f).isInstanceOf(HttpDeadlineExceededException.class));
// when
Throwable next = catchThrowable(() -> getUserWithin(harness, Optional.empty()));
// then
assertThat(next).isNull();
assertThat(server.requestCount()).isEqualTo(SMALL_WINDOW + 1);
}
}
@ParameterizedTest(name = "{0}")
@EnumSource(
value = TransportType.class,
names = {"APACHE", "JDK"})
@DisplayName("프로필 total-call 시간으로 끊긴 시도는 상대 장애로 세어 창을 채우면 circuit이 열린다")
void attemptsCutShortByTheProfileTotalCallStillOpenTheCircuit(TransportType transport)
throws Exception {
TimeoutSettings base = ClientProfiles.timeouts();
try (MockHttpServer server = MockHttpServer.start();
TestGateways.Harness harness =
TestGateways.forProfile(
ClientProfiles.builder("profile-deadline")
.baseUrl(server.uri("/"))
.transport(transport)
.timeout(
new TimeoutSettings(
base.dns(),
base.connect(),
base.tlsHandshake(),
base.responseHeader(),
base.readIdle(),
PROFILE_TOTAL_CALL))
.circuitBreaker(
new CircuitBreakerSettings(
50, TINY_WINDOW, TINY_WINDOW, Duration.ofMinutes(1), 1))
.build())) {
// given
List<Throwable> windowFailures = new ArrayList<>();
for (int i = 0; i < TINY_WINDOW; i++) {
server.enqueueSlowDripBody(200, 100, Duration.ofMillis(50));
windowFailures.add(catchThrowable(() -> getUserWithin(harness, Optional.empty())));
}
assertThat(windowFailures)
.allSatisfy(f -> assertThat(f).isInstanceOf(HttpDeadlineExceededException.class));
// when
Throwable next = catchThrowable(() -> getUserWithin(harness, Optional.empty()));
// then
assertThat(next).isInstanceOf(HttpCircuitOpenException.class);
assertThat(server.requestCount()).isEqualTo(TINY_WINDOW);
}
}
private static HttpCallResult<String> getUserWithin(
TestGateways.Harness harness, Optional<Instant> deadline) {
HttpOperation get = HttpOperation.get(new OperationName("get-user"), "/users/1", Map.of());
return harness
.gateway()
.exchange(
harness.profile().name(),
new HttpOperation(
get.operationName(),
get.method(),
get.uriTemplate(),
get.uriVariables(),
get.headers(),
get.body(),
get.idempotency(),
get.idempotencyKey(),
deadline),
ResponseType.of(String.class));
}
private static HttpCallResult<String> getUserAsResult(TestGateways.Harness harness) {
return harness
.gateway()
@@ -0,0 +1,124 @@
package dev.caskeleton.adapter.outbound.httpclient.restclient;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.catchThrowable;
import dev.caskeleton.adapter.outbound.httpclient.api.OperationName;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpResponseTimeoutException;
import dev.caskeleton.adapter.outbound.httpclient.api.operation.AttemptStage;
import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence;
import dev.caskeleton.adapter.outbound.httpclient.api.operation.HttpOperation;
import dev.caskeleton.adapter.outbound.httpclient.api.result.BlockingStreamingResponse;
import dev.caskeleton.adapter.outbound.httpclient.profile.ClientProfile;
import dev.caskeleton.adapter.outbound.httpclient.profile.ResponseLimits;
import dev.caskeleton.adapter.outbound.httpclient.profile.TimeoutSettings;
import dev.caskeleton.adapter.outbound.httpclient.profile.TransportType;
import dev.caskeleton.adapter.outbound.httpclient.testkit.ClientProfiles;
import dev.caskeleton.adapter.outbound.httpclient.testkit.MockHttpServer;
import dev.caskeleton.adapter.outbound.httpclient.testkit.TestGateways;
import java.time.Duration;
import java.util.Map;
import java.util.Set;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.EnumSource;
/**
* What bounds a streaming download's body once it has been handed over (review 2026-11-03 RR-04-03,
* design §15.3).
*
* <p>The documented contract is transport-independent: the total budget and the response-header
* timeout bound the setup, and after hand-over each gap of the body is bounded by {@code read-idle}
* alone, so a long download is not cut at an arbitrary total. The JDK transport used to break it:
* Spring's JDK request enforces its read timeout over the whole exchange and closes the body stream
* when it fires, so every JDK download longer than the response-header timeout was cut, and the
* caller got an unclassified {@code IOException: closed}.
*
* <p>The profile's response-header timeout (500 ms) and total budget (1 s) are both shorter than
* the body (about 2 s), and the read-idle timeout (1 s) is longer than every gap of it.
*/
class StreamingBodyLifetimeTest {
private static final int BODY_BYTES = 20;
private static final Duration DRIP_GAP = Duration.ofMillis(100);
private static final Duration STALL_GAP = Duration.ofMillis(2500);
@ParameterizedTest(name = "{0}")
@EnumSource(
value = TransportType.class,
names = {"APACHE", "JDK"})
@DisplayName("넘겨받은 스트리밍 본문은 응답 헤더 시간과 전체 예산을 넘겨도 끝까지 읽힌다")
void aStreamedBodyOutlivesTheResponseHeaderTimeoutAndTheTotalBudget(TransportType transport)
throws Exception {
try (MockHttpServer server = MockHttpServer.start();
TestGateways.Harness harness = TestGateways.forProfile(profile(server, transport))) {
// given
server.enqueueSlowDripBody(200, BODY_BYTES, DRIP_GAP);
BlockingStreamingGateway gateway = new BlockingStreamingGateway(harness.registry());
// when
byte[] body;
try (BlockingStreamingResponse response = gateway.download(harness.profile().name(), get())) {
body = response.body().readAllBytes();
}
// then
assertThat(body).hasSize(BODY_BYTES);
}
}
@ParameterizedTest(name = "{0}")
@EnumSource(
value = TransportType.class,
names = {"APACHE", "JDK"})
@DisplayName("본문 사이 공백이 read-idle을 넘으면 분류된 응답 시간 초과로 끝난다")
void aBodyGapLongerThanReadIdleEndsWithAClassifiedTimeout(TransportType transport)
throws Exception {
try (MockHttpServer server = MockHttpServer.start();
TestGateways.Harness harness = TestGateways.forProfile(profile(server, transport))) {
// given
server.enqueueSlowDripBody(200, BODY_BYTES, STALL_GAP);
BlockingStreamingGateway gateway = new BlockingStreamingGateway(harness.registry());
long started = System.nanoTime();
// when
Throwable failure =
catchThrowable(
() -> {
try (BlockingStreamingResponse response =
gateway.download(harness.profile().name(), get())) {
response.body().readAllBytes();
}
});
// then
Duration elapsed = Duration.ofNanos(System.nanoTime() - started);
assertThat(failure).isInstanceOf(HttpResponseTimeoutException.class);
assertThat(elapsed).isLessThan(STALL_GAP);
HttpResponseTimeoutException timeout = (HttpResponseTimeoutException) failure;
assertThat(timeout.metadata().stage()).isEqualTo(AttemptStage.RESPONSE_BODY);
assertThat(timeout.metadata().evidence()).isEqualTo(ExecutionEvidence.PARTIAL_RESPONSE);
}
}
private static HttpOperation get() {
return HttpOperation.get(new OperationName("download"), "/download", Map.of());
}
private static ClientProfile profile(MockHttpServer server, TransportType transport) {
TimeoutSettings base = ClientProfiles.timeouts();
return ClientProfiles.builder("streaming-body")
.baseUrl(server.uri("/"))
.transport(transport)
.timeout(
new TimeoutSettings(
base.dns(),
base.connect(),
base.tlsHandshake(),
Duration.ofMillis(500),
Duration.ofSeconds(1),
Duration.ofSeconds(1)))
.response(new ResponseLimits(1024, 1024, Set.of("text/plain")))
.build();
}
}
@@ -68,6 +68,11 @@ public final class RecordingResilienceComponents {
events.add("circuit-error");
}
@Override
public void releasePermission() {
events.add("circuit-release");
}
@Override
public String state() {
return circuitOpen ? "OPEN" : "CLOSED";
+11 -4
View File
@@ -20,16 +20,22 @@ build choices) live in [README.md](README.md).
- `UuidV7DurableOperationIdFactory` — `DurableOperationIdFactory` implementation (UUIDv7 via
`uuid-creator`). The operation id is also the submission event's id, and the messaging platform's
message id accepts only UUIDv7 (review 2026-11-01 06-06). Wired by `composition:durable-operation`.
- The leaf name says "identifier"; general identifier generation, clock and random sources are not here.
- The leaf name says "identifier"; general-purpose clock and random sources are not here.
The name is kept because Gradle paths and consumers use it (README, review 05-21).
- Wiring is `composition/service-runtime` (`PseudonymizationConfig`), not this leaf.
## Allowed
- `:application-core` (declared in [build.gradle](build.gradle)); the registry entry is the SSOT.
Declared in [build.gradle](build.gradle); the registry entry is the SSOT.
- `:application-core` — the ports both classes implement.
- `:shared-contract`, for `HmacSha256` only (review 2026-11-01 01-11). The pseudonym is the hex HMAC
of the principal; `HmacUserPrincipalPseudonymizerVectorTest` pins it to a vector computed outside
the JVM, because logs written before and after a release are joined on it.
- `:domain-core` — a feature's server-assigned id adapter implements its domain `*IdFactory` port
here (registry edge `adapter:outbound:identifier -> domain-core`; `docs/manual/02-new-feature.md`).
- `com.github.f4b6a3:uuid-creator` (`libs.uuid.creator`) — UUIDv7 for
`UuidV7DurableOperationIdFactory`, the generator the sample's id adapters use.
## Forbidden
@@ -42,8 +48,9 @@ build choices) live in [README.md](README.md).
## Test
Pure unit tests, no Spring context (`HmacUserPrincipalPseudonymizerTest`, including the RFC 4231
known-answer vector).
Pure unit tests, no Spring context: `HmacUserPrincipalPseudonymizerTest` (including the RFC 4231
known-answer vector), `HmacUserPrincipalPseudonymizerVectorTest` and
`UuidV7DurableOperationIdFactoryTest`.
```bash
cd src
+10 -5
View File
@@ -9,7 +9,8 @@
## 이름과 내용이 다른 이유 (review 05-21)
leaf 이름은 `identifier`지만 범용 식별자 생성기·clock·random 소스는 여기 없다. 식별자 생성은
durable operation id(UUIDv7) 하나뿐이다(아래). 예전 문서가 범용 생성기 계획을 현재 내용처럼 적었다. 이름은 바꾸지 않았다: Gradle 경로·registry id·소비자
durable operation id(UUIDv7) 하나뿐이고(아래), 기능이 서버에서 id를 정하면 그 기능의 domain
`*IdFactory` 구현도 여기 둔다(`:domain-core` 의존이 그 때문이다). 예전 문서가 범용 생성기 계획을 현재 내용처럼 적었다. 이름은 바꾸지 않았다: Gradle 경로·registry id·소비자
(`composition:service-runtime`, `app-bootstrap`, `sample-portfolio`)의 의존 선언이 모두 이 이름을
쓰고, 바꿔서 얻는 것은 이름 하나뿐이다. 외부 연동이 없는 crypto 능력을 새로 둘 곳이 필요하면 이
leaf에 둔다. 그때 이 절을 고친다.
@@ -25,10 +26,10 @@ leaf에 둔다. 그때 이 절을 고친다.
대소문자를 바꾸면 테스트가 실패한다.
- Salt: `APP_PRIVACY_PSEUDONYMIZATION_SALT`(secret, 90일 회전)에서 온다. 이 클래스는 salt를 스스로
읽지 않고 생성자로 받는다. 받은 배열은 복사해 둔다.
- Thread safety: `Mac`은 thread-safe하지 않아서 호출마다 새로 만든다. 그래서 싱글톤 bean으로
공유해도 된다. `HmacSHA256`은 JDK 필수 알고리즘이라 `NoSuchAlgorithmException`·
`InvalidKeyException`은 사실상 나지 않는다. checked exception을 호출부에 남기지 않으려고
`IllegalStateException`으로 감싼다.
- 서명: 이 클래스는 `Mac`을 직접 다루지 않는다. shared-contract의 `HmacSha256.sign`에 맡긴다(리뷰
2026-11-01 01-11). `HmacSha256`이 호출마다 새 `Mac`을 만들고(`Mac`은 thread-safe하지 않다),
JDK 필수 알고리즘이라 사실상 나지 않는 checked exception을 `IllegalStateException`으로 감싼다.
그래서 이 클래스는 싱글톤 bean으로 공유해도 된다.
## UuidV7DurableOperationIdFactory
@@ -46,6 +47,10 @@ Spring 어노테이션이 없다. bean은
## 빌드 결정
의존은 넷이다: `:application-core`(두 클래스가 구현하는 port), `:shared-contract`(`HmacSha256`),
`:domain-core`(기능의 `*IdFactory` 구현 자리), `uuid-creator`(UUIDv7). 목록과 이유는
[CLAUDE.md](CLAUDE.md) Allowed 절이 기준이다.
`:application-core`를 `implementation`으로 받는다. 이 leaf의 public 타입 시그니처에
application-core 타입이 드러나지만, 조립하는 쪽이 application-core를 이미 classpath에 갖고 있다.
허용 edge는 `src/config/architecture/modules.json`이 기준이다.
@@ -1,5 +1,7 @@
/**
* The HMAC-SHA-256 user-principal pseudonymizer. Despite the leaf name, no identifier generation,
* clock or random source lives here; see README for why the name was kept.
* Two identifier adapters with no external IO: the HMAC-SHA-256 user-principal pseudonymizer and
* the UUIDv7 durable operation id factory. A feature's server-assigned id adapter for a domain
* {@code *IdFactory} port belongs here too. General clock and random sources do not live here; see
* README for why the leaf name was kept.
*/
package dev.caskeleton.adapter.outbound.identifier;
@@ -0,0 +1,14 @@
# adapter:outbound:messaging-kafka
- Module ID `adapter-outbound-messaging-kafka`, Gradle path `:adapter:outbound:messaging-kafka`.
Registry SSOT: `src/config/architecture/modules.json`.
- Responsibility: the Kafka publishing capability as one `runtimeOnly` line. It has no production
code: it bundles `adapter:outbound:messaging` (the application's bridge) with
`messaging:messaging-spring-boot-starter-kafka` (the vendored platform and its Kafka transport) and
ships the module defaults in `config/module-defaults/messaging.yml`.
- Boundaries: add no code here. A publish rule belongs to the bridge (`adapter:outbound:messaging`),
a Kafka rule to the vendored product (`src/messaging/CLAUDE.md`); a class here would be a third
place. Allowed edges come only from the registry entry.
- Tests: activation tests (`./gradlew :adapter:outbound:messaging-kafka:test`) and the real-Kafka
lane `messagingKafkaRelayIntegrationTest` (relay → bridge → broker record bytes; `:integrationCheck`).
- Design: `README.md`.
@@ -8,6 +8,7 @@ import dev.caskeleton.application.outbox.OutboxAppendPort;
import dev.caskeleton.application.outbox.OutboxCompletion;
import dev.caskeleton.application.outbox.OutboxEventStatus;
import dev.caskeleton.application.outbox.OutboxLease;
import dev.caskeleton.application.outbox.OutboxLongRetries;
import dev.caskeleton.application.outbox.OutboxStorePort;
import dev.caskeleton.application.outbox.testkit.InMemoryOutboxStore;
import dev.caskeleton.application.outbox.testkit.OutboxEvents;
@@ -282,13 +283,15 @@ class ApplicationRelayKafkaIntegrationTest {
}
@Override
public synchronized OutboxCompletion markFailed(OutboxLease lease, Instant nextAttemptAt) {
return delegate.markFailed(lease, nextAttemptAt);
public synchronized OutboxCompletion markFailed(
OutboxLease lease, Instant nextAttemptAt, String failureCode) {
return delegate.markFailed(lease, nextAttemptAt, failureCode);
}
@Override
public synchronized OutboxCompletion markDeferred(OutboxLease lease, Instant nextAttemptAt) {
return delegate.markDeferred(lease, nextAttemptAt);
public synchronized OutboxCompletion markDeferred(
OutboxLease lease, Instant nextAttemptAt, String failureCode) {
return delegate.markDeferred(lease, nextAttemptAt, failureCode);
}
@Override
@@ -297,8 +300,13 @@ class ApplicationRelayKafkaIntegrationTest {
}
@Override
public synchronized OutboxCompletion markDead(OutboxLease lease) {
return delegate.markDead(lease);
public synchronized OutboxCompletion markDead(OutboxLease lease, String failureCode) {
return delegate.markDead(lease, failureCode);
}
@Override
public synchronized OutboxLongRetries longRetries(int minDeferredClaims, Instant now) {
return delegate.longRetries(minDeferredClaims, now);
}
@Override
+1 -1
View File
@@ -5,7 +5,7 @@
- Module ID: `adapter-outbound-messaging`
- Gradle path: `:adapter:outbound:messaging`
- Focused test (derived from Gradle path): `./gradlew :adapter:outbound:messaging:test --console=plain`
- Runtime baseline: Java 21; repository framework baseline: Spring Boot 4.0.0.
- Runtime baseline: Java 21; repository framework baseline: Spring Boot 4.0.8.
- Registry SSOT: `src/config/architecture/modules.json`.
Package root: `dev.caskeleton.adapter.outbound.messaging`.
+27 -8
View File
@@ -42,13 +42,33 @@ platform `REJECTED` 중 `FailureDescriptor.retryable()`이 true인 것(종료
멈춘다(2026-11-01 리뷰 06-01). `AMBIGUOUS`로 번역하지 않은 이유: 동작은 같지만 "전송됐을 수
있음"이라는 거짓 정보가 남아 중복 경고를 오염시킨다.
### 실패 원인 코드
결과(outcome)는 relay가 무엇을 할지만 정한다. 종료 중 거부, 닫힌 transport, 거부된 credential은 모두
`REJECTED_RETRYABLE` 하나다. 그런데 이 셋은 끝이 없는 재시도다(시도 예산을 쓰지 않는다). 그래서 bridge는
platform `FailureDescriptor.code()`를 버리지 않고 `OutboxPublishAttempt.failureCode()`로 함께 넘긴다
(재리뷰 2026-11-03 RR-05-03). relay는 그 코드를 행의 `last_failure_code`(V22)와 실패 로그의
`error.reason`에 남긴다. 운영자는 이 값으로 "credential이 진짜 틀렸다"와 "broker가 이 레코드만 받지
않는다"를 가른다(`docs/runbooks/outbox-publish-failed.md` §4.1).
- platform code는 이미 정제된 이름이다(`KAFKA_AUTHENTICATIONEXCEPTION`, `PUBLISH_DEADLINE_EXCEEDED` 등).
driver 메시지가 아니므로 그대로 옮긴다. application 코드 문법(`[A-Za-z0-9][A-Za-z0-9._:-]*`, 120자)에
맞지 않는 code는 그 실패의 `FailureCategory` 이름으로 옮긴다.
- platform으로 옮길 수 없는 event(아래 id 규칙)는 bridge가 `EVENT_NOT_REPRESENTABLE`로 거부한다.
- 버린 대안: relay가 platform 타입을 직접 보는 것(경계가 깨진다), outcome 값을 원인별로 늘리는 것(relay가
할 일은 같은데 분기만 는다).
### event id → message id
application `EventId`는 문자열이고 platform `MessageId`는 **UUIDv7**만 받는다. bridge는 event id가
정규 소문자 UUIDv7 문자열일 때만 같은 값으로 옮기고, 그 밖(대문자·축약 표기·v4 등 다른 버전)은 다른
문자열의 message id로 바꿔 보내지 않고 `REJECTED_BEFORE_SEND`로 거부한다(relay는 DEAD). 소비자 중복
제거 키가 event id라서 값을 바꿔 보내면 중복 제거가 깨지기 때문이다(리뷰 2026-11-01 06-06). 이 거부는
append 시점(`EventId` 문법)에서 먼저 일어나야 한다 — application 쪽 문법 일원화는 별도 작업이다.
제거 키가 event id라서 값을 바꿔 보내면 중복 제거가 깨지기 때문이다(리뷰 2026-11-01 06-06).
같은 규칙이 append 쪽에 먼저 있다. `IntegrationEventDraft`가 정규 소문자 UUIDv7이 아닌 event id와
causation id를 거부하므로(`IntegrationIdentity.isCanonicalUuidV7`), 그런 id는 업무 트랜잭션 안에서 실패한다
(재리뷰 2026-11-03 RR-05-04). bridge의 거부는 이 규칙 전에 쓰인 행(저장 문법 `EventId`는 그대로 넓다)을 위한
마지막 방어선이다.
## outbox 발행 포트에는 대역이 없다
@@ -70,9 +90,10 @@ event가 다시 publish될 수 있다. 같은 event id로 다시 나가므로 co
있으면 `messaging-platform`이다. 이 adapter는 confirmed FAILED/DEAD report 하나를 SLF4J 2 fluent
ERROR 하나로 렌더링한다.
공통 field는 `error.code`, `error.category`, `dependency_name`,
공통 field는 `error.code`, `error.category`, `error.cause_type`, `error.reason`, `dependency_name`,
`dependency_type=messaging`, `outcome`, `event_id`, `event_type`, `aggregate_id`,
`correlation_id`, `attempt_count`, `runbook_link`다. retry report만 `next_attempt_at`을 추가한다.
`error.reason`은 위 실패 원인 코드다.
payload/envelope/exception-derived field는 받거나 렌더링하지 않는다. logging 내부
`RuntimeException`은 adapter와 use case 양쪽에서 방어하므로 persisted FAILED/DEAD outcome을 바꾸지
않는다.
@@ -85,10 +106,8 @@ payload/envelope/exception-derived field는 받거나 렌더링하지 않는다.
`envelope`·`realtime`)은 2026-09-25에 삭제됐고 복구 경로는 `docs/capability-status.md` 제거 표에
있다.
공통 build evidence manifest 검증은 build-tools로 옮겼다. 루트 `verifyMessagingContracts`
(`VerifyMessagingEvidenceTask`)가 application-core·shared-contract 두 qualification 레인 결과로
`build/messaging-evidence/contracts-schema/manifest.json`을 쓰고, 쓴 바이트를 바로 common schema로
검증한다. 이 leaf는 그 검증에 관여하지 않는다.
integration event 계약 검증은 루트 `verifyMessagingContracts`(`src/build.gradle`)가 한다. application-core·
shared-contract·sample-portfolio의 qualification lane 셋을 묶은 task다. 이 leaf는 그 검증에 관여하지 않는다.
canonical path는 platform `PublishResult`의 confirmation/transmission evidence를 application
`OutboxPublishOutcome`으로 명시적으로 변환한다.
`OutboxPublishOutcome`으로 명시적으로 변환하고, 실패 code를 `OutboxPublishAttempt`에 함께 싣는다.
@@ -4,17 +4,21 @@ import dev.caskeleton.application.messaging.event.IntegrationEventPublishPort;
import dev.caskeleton.application.outbox.ClaimedOutboxEvent;
import dev.caskeleton.application.outbox.OutboxMessagePublishPort;
import dev.caskeleton.application.outbox.OutboxPublishAmbiguousException;
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
import dev.caskeleton.application.outbox.OutboxPublishRefusedException;
import java.time.Duration;
import java.util.Objects;
import java.util.Optional;
/**
* The relay's publish port: a claimed row's validated event, delegated unchanged to the canonical
* {@link IntegrationEventPublishPort}.
*
* <p>The event is the one the append validated, restored from its exact envelope bytes; nothing
* here re-renders it. The platform's four-valued outcome is passed through, so the relay can tell a
* possibly-stored message (ambiguous) from a refused one.
* here re-renders it. The canonical publisher's attempt — its outcome and its failure code — is
* passed through unchanged, so the relay can tell a possibly-stored message (ambiguous) from a
* refused one, and a refused credential from a closed transport.
*/
public final class OutboxMessagePublishAdapter implements OutboxMessagePublishPort {
@@ -27,7 +31,7 @@ public final class OutboxMessagePublishAdapter implements OutboxMessagePublishPo
@Override
public void publish(ClaimedOutboxEvent event) {
OutboxPublishOutcome outcome = publishForOutcome(event);
OutboxPublishOutcome outcome = publishForOutcome(event).outcome();
if (outcome == OutboxPublishOutcome.CONFIRMED) {
return;
}
@@ -37,8 +41,14 @@ public final class OutboxMessagePublishAdapter implements OutboxMessagePublishPo
throw new OutboxPublishRefusedException(outcome);
}
/** The canonical publisher's, unchanged: this adapter adds no wait of its own. */
@Override
public OutboxPublishOutcome publishForOutcome(ClaimedOutboxEvent event) {
public Optional<Duration> longestAttempt() {
return canonicalPublisher.longestAttempt();
}
@Override
public OutboxPublishAttempt publishForOutcome(ClaimedOutboxEvent event) {
Objects.requireNonNull(event, "event must not be null");
return canonicalPublisher.publish(event.event()).toCompletableFuture().join();
}
@@ -48,6 +48,10 @@ public final class Slf4jOutboxRelayFailureReportAdapter implements OutboxRelayFa
// occasionally credentials. The class name is a type; the code is bounded.
.addKeyValue("error.code", report.code().code())
.addKeyValue("error.cause_type", report.causeType())
// Why, in the publisher's words: the cause type of a returned outcome names only
// which
// outcome it was, and a refused credential and a closed transport are one outcome.
.addKeyValue("error.reason", report.failureCode())
.addKeyValue("error.category", report.code().category().name())
.addKeyValue("dependency_name", dependencyName)
.addKeyValue("dependency_type", DEPENDENCY_TYPE)
@@ -2,6 +2,7 @@ package dev.caskeleton.adapter.outbound.messaging.platformbridge;
import dev.caskeleton.application.messaging.event.IntegrationEventPublishPort;
import dev.caskeleton.application.messaging.event.ValidatedIntegrationEvent;
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
import dev.caskeleton.messaging.api.CausationId;
import dev.caskeleton.messaging.api.ContentType;
@@ -26,6 +27,7 @@ import dev.caskeleton.messaging.api.publish.TransmissionEvidence;
import dev.caskeleton.messaging.schema.EncodedMessage;
import dev.caskeleton.messaging.schema.EncodedMessagePublisher;
import dev.caskeleton.messaging.schema.SchemaReference;
import java.time.Duration;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.Objects;
@@ -45,6 +47,12 @@ import java.util.concurrent.CompletionStage;
*/
public final class PlatformIntegrationEventPublishAdapter implements IntegrationEventPublishPort {
/**
* The failure code of an event this bridge refuses before the platform sees it: a value — an id
* above all — the platform's types cannot carry.
*/
public static final String NOT_REPRESENTABLE = "EVENT_NOT_REPRESENTABLE";
private final EncodedMessagePublisher publisher;
private final ProducerId producer;
@@ -55,7 +63,7 @@ public final class PlatformIntegrationEventPublishAdapter implements Integration
}
@Override
public CompletionStage<OutboxPublishOutcome> publish(ValidatedIntegrationEvent event) {
public CompletionStage<OutboxPublishAttempt> publish(ValidatedIntegrationEvent event) {
Objects.requireNonNull(event, "event must not be null");
final PreparedPublish prepared;
try {
@@ -64,12 +72,27 @@ public final class PlatformIntegrationEventPublishAdapter implements Integration
// Only a value the platform cannot represent is a definite refusal: the same payload is
// refused again, so the relay dead-letters it at once. Any other failure is not evidence of
// that and propagates, so the relay treats it as a retryable failure with its real cause.
return CompletableFuture.completedFuture(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
return CompletableFuture.completedFuture(
OutboxPublishAttempt.failed(
OutboxPublishOutcome.REJECTED_BEFORE_SEND, NOT_REPRESENTABLE));
}
return publisher
.publishEncoded(prepared.destination(), prepared.envelope(), PublishOptions.defaults())
.thenApply(PlatformIntegrationEventPublishAdapter::mapOutcome);
.thenApply(PlatformIntegrationEventPublishAdapter::translate);
}
/**
* Twice the publish deadline this bridge asks for ({@link PublishOptions#defaults()}).
*
* <p>The platform measures the deadline from the call, but a transport may block inside its send
* before the deadline is applied to the stage it returns — Kafka's {@code max.block.ms} is set to
* the same default for exactly that wait — so an attempt can take the deadline twice over. The
* relay sizes its batch against this (review 2026-11-01 06-12).
*/
@Override
public Optional<Duration> longestAttempt() {
return Optional.of(PublishOptions.defaults().timeout().multipliedBy(2));
}
private PreparedPublish prepare(ValidatedIntegrationEvent event) {
@@ -146,8 +169,10 @@ public final class PlatformIntegrationEventPublishAdapter implements Integration
* </ul>
*
* <p>Either refusal is {@code REJECTED_BEFORE_SEND}: the same id is refused again, so the relay
* dead-letters it. The append side's {@code EventId} grammar is where such an id should fail
* first, in the writer's transaction; until it does, this is where the mismatch is caught.
* dead-letters it. A new event fails earlier, in the writer's transaction: {@code
* IntegrationEventDraft} refuses an id that is not a canonical lower-case UUIDv7 (re-review
* 2026-11-03 RR-05-04). This check stays for rows appended before that rule — the stored {@code
* EventId} grammar is wider, so they can still be read back.
*
* @throws IllegalArgumentException when the value is not a canonical lower-case UUIDv7
*/
@@ -177,8 +202,37 @@ public final class PlatformIntegrationEventPublishAdapter implements Integration
* committed event, and its aggregate behind it, for a cause that clears on its own. It gets its
* own value instead.
*/
private static OutboxPublishOutcome mapOutcome(PublishResult result) {
private static OutboxPublishAttempt translate(PublishResult result) {
Objects.requireNonNull(result, "publish result must not be null");
OutboxPublishOutcome outcome = mapOutcome(result);
if (outcome == OutboxPublishOutcome.CONFIRMED) {
return OutboxPublishAttempt.confirmed();
}
return result
.failure()
.map(PlatformIntegrationEventPublishAdapter::failureCode)
.map(code -> OutboxPublishAttempt.failed(outcome, code))
.orElseGet(() -> OutboxPublishAttempt.of(outcome));
}
/**
* The platform's failure code, carried as the application's failure code (re-review 2026-11-03
* RR-05-03).
*
* <p>The outcome is all the relay acts on, and it loses the reason on purpose: a shutdown, a
* closed transport and a refused credential are one {@code REJECTED_RETRYABLE}. The code keeps it
* for the operator, on the row and in the failure log. The platform's code is already sanitized —
* a stable name, never a driver message — and passes as it is; one outside the application's code
* grammar (the platform only bounds its length) is carried as the failure's category instead, a
* name the platform owns too.
*/
private static String failureCode(FailureDescriptor failure) {
return OutboxPublishAttempt.isFailureCode(failure.code())
? failure.code()
: failure.category().name();
}
private static OutboxPublishOutcome mapOutcome(PublishResult result) {
if (result.completion() == PublishCompletion.CONFIRMED) {
return OutboxPublishOutcome.CONFIRMED;
}
@@ -5,7 +5,7 @@ import static org.assertj.core.api.Assertions.assertThat;
import dev.caskeleton.adapter.outbound.messaging.outbox.OutboxMessagePublishAdapter;
import dev.caskeleton.application.messaging.event.IntegrationEventPublishPort;
import dev.caskeleton.application.outbox.OutboxMessagePublishPort;
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
import java.util.concurrent.CompletableFuture;
import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
@@ -32,7 +32,7 @@ class MessagingConfigTest {
runner
.withBean(
IntegrationEventPublishPort.class,
() -> event -> CompletableFuture.completedFuture(OutboxPublishOutcome.CONFIRMED))
() -> event -> CompletableFuture.completedFuture(OutboxPublishAttempt.confirmed()))
.run(
context ->
assertThat(context.getBean(OutboxMessagePublishPort.class))
@@ -51,7 +51,14 @@ class Slf4jOutboxRelayFailureReportAdapterTest {
adapter.report(
OutboxRelayFailureReport.retryableFailure(
"evt-1", "WorkLogReserved", "agg-1", "corr-1", 2, nextAttemptAt, cause));
"evt-1",
"WorkLogReserved",
"agg-1",
"corr-1",
2,
nextAttemptAt,
cause,
"KAFKA_AUTHENTICATIONEXCEPTION"));
assertThat(appender.list).hasSize(1);
ILoggingEvent event = appender.list.getFirst();
@@ -71,6 +78,7 @@ class Slf4jOutboxRelayFailureReportAdapterTest {
Map.entry("correlation_id", "corr-1"),
Map.entry("attempt_count", 2),
Map.entry("error.cause_type", "java.lang.RuntimeException"),
Map.entry("error.reason", "KAFKA_AUTHENTICATIONEXCEPTION"),
Map.entry("runbook_link", "runbook://outbox/publish-failed"),
Map.entry("next_attempt_at", "2026-07-25T01:02:03Z")));
assertThat(event.getThrowableProxy())
@@ -89,7 +97,7 @@ class Slf4jOutboxRelayFailureReportAdapterTest {
adapter.report(
OutboxRelayFailureReport.deadLetter(
"evt-2", "WorkLogReserved", "agg-2", "corr-2", 3, cause));
"evt-2", "WorkLogReserved", "agg-2", "corr-2", 3, cause, "REJECTED_BEFORE_SEND"));
assertThat(appender.list).hasSize(1);
ILoggingEvent event = appender.list.getFirst();
@@ -142,7 +150,8 @@ class Slf4jOutboxRelayFailureReportAdapterTest {
"agg-3",
"corr-3",
3,
new RuntimeException("broker down"))))
new RuntimeException("broker down"),
"RuntimeException")))
.doesNotThrowAnyException();
}
@@ -12,6 +12,7 @@ import dev.caskeleton.application.messaging.event.IntegrationEventPublishPort;
import dev.caskeleton.application.messaging.event.ValidatedIntegrationEvent;
import dev.caskeleton.application.messaging.event.testkit.BrokerAnswer;
import dev.caskeleton.application.messaging.event.testkit.IntegrationEventPublishContract;
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
import dev.caskeleton.messaging.api.ContentType;
import dev.caskeleton.messaging.api.MessageEnvelope;
@@ -59,7 +60,7 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
ValidatedIntegrationEvent event = event(EVENT_ID, Optional.of(CAUSATION_ID));
assertThat(bridge.publish(event).toCompletableFuture().join())
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
.isEqualTo(OutboxPublishOutcome.CONFIRMED);
MessageDestination<EncodedMessage> destination = publisher.destination;
@@ -99,19 +100,19 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
ValidatedIntegrationEvent event = event(EVENT_ID, Optional.empty());
assertThat(bridge.publish(event).toCompletableFuture().join())
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
.isEqualTo(OutboxPublishOutcome.CONFIRMED);
publisher.result = ambiguous();
assertThat(bridge.publish(event).toCompletableFuture().join())
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
.isEqualTo(OutboxPublishOutcome.AMBIGUOUS);
publisher.result = rejected(PublishEvidence.notTransmitted());
assertThat(bridge.publish(event).toCompletableFuture().join())
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
.isEqualTo(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
publisher.result =
rejected(
new PublishEvidence(
true, TransmissionEvidence.TRANSMITTED, false, ConfirmationLevel.NONE));
assertThat(bridge.publish(event).toCompletableFuture().join())
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
.isEqualTo(OutboxPublishOutcome.REJECTED_AFTER_BROKER);
}
@@ -132,12 +133,101 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
// when
OutboxPublishOutcome outcome =
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join();
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join().outcome();
// then
assertThat(outcome).isEqualTo(OutboxPublishOutcome.REJECTED_RETRYABLE);
}
@Test
@DisplayName("platform이 붙인 실패 코드는 원인 코드로 그대로 옮긴다")
void thePlatformsFailureCodeIsCarriedAsTheFailureCode() {
// given: the broker refused this producer's credentials
RecordingPublisher publisher =
new RecordingPublisher(
PublishResult.rejected(
Duration.ZERO,
new FailureDescriptor(
FailureCategory.AUTHENTICATION,
"KAFKA_AUTHENTICATIONEXCEPTION",
true,
"kafka publish failed with AuthenticationException",
Optional.of("AuthenticationException"))));
PlatformIntegrationEventPublishAdapter bridge =
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
// when
OutboxPublishAttempt attempt =
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join();
// then
assertThat(attempt)
.isEqualTo(
OutboxPublishAttempt.failed(
OutboxPublishOutcome.REJECTED_RETRYABLE, "KAFKA_AUTHENTICATIONEXCEPTION"));
}
@Test
@DisplayName("코드 문법에 맞지 않는 platform 실패 코드는 그 실패의 분류 이름으로 옮긴다")
void aPlatformCodeOutsideTheCodeGrammarIsCarriedAsItsCategory() {
// given
RecordingPublisher publisher =
new RecordingPublisher(
PublishResult.ambiguous(
Duration.ZERO,
FailureDescriptor.of(
FailureCategory.AMBIGUOUS, "timed out after 5000 ms", "no answer")));
PlatformIntegrationEventPublishAdapter bridge =
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
// when
OutboxPublishAttempt attempt =
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join();
// then
assertThat(attempt.failureCode()).contains("AMBIGUOUS");
}
@Test
@DisplayName("platform으로 옮길 수 없는 event는 그 이유를 원인 코드로 알린다")
void anEventThePlatformCannotRepresentIsRefusedWithItsOwnFailureCode() {
// given
RecordingPublisher publisher = new RecordingPublisher(confirmed());
PlatformIntegrationEventPublishAdapter bridge =
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
// when
OutboxPublishAttempt attempt =
bridge.publish(event("event-1", Optional.empty())).toCompletableFuture().join();
// then
assertThat(attempt)
.isEqualTo(
OutboxPublishAttempt.failed(
OutboxPublishOutcome.REJECTED_BEFORE_SEND,
PlatformIntegrationEventPublishAdapter.NOT_REPRESENTABLE));
}
/**
* The relay sizes its batch against this (review 2026-11-01 06-12). The platform's deadline is
* measured from the call, but a transport may block in its send before the deadline is applied —
* Kafka's {@code max.block.ms} is set to the same default — so an attempt can take twice it.
*/
@Test
@DisplayName("발행 하나의 최악 시간은 platform 기본 발행 기한의 두 배다")
void theLongestAttemptIsTwiceThePlatformsDefaultPublishDeadline() {
// given
PlatformIntegrationEventPublishAdapter bridge =
new PlatformIntegrationEventPublishAdapter(
new RecordingPublisher(confirmed()), "portfolio-service");
// when
Optional<Duration> longest = bridge.longestAttempt();
// then
assertThat(longest).contains(PublishOptions.defaults().timeout().multipliedBy(2));
}
@Test
@DisplayName("다시 시도해도 같은 결과인 거부는 지금처럼 확정 거부로 번역한다")
void aRefusalThePlatformMarksFinalStaysADefiniteRejection() {
@@ -153,7 +243,7 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
// when
OutboxPublishOutcome outcome =
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join();
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join().outcome();
// then
assertThat(outcome).isEqualTo(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
@@ -165,7 +255,12 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
PlatformIntegrationEventPublishAdapter bridge =
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
assertThat(bridge.publish(event("event-1", Optional.empty())).toCompletableFuture().join())
assertThat(
bridge
.publish(event("event-1", Optional.empty()))
.toCompletableFuture()
.join()
.outcome())
.isEqualTo(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
assertThat(publisher.calls).isZero();
}
@@ -192,7 +287,7 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
// when
OutboxPublishOutcome outcome =
bridge.publish(event(eventId, Optional.empty())).toCompletableFuture().join();
bridge.publish(event(eventId, Optional.empty())).toCompletableFuture().join().outcome();
// then
assertThat(outcome).isEqualTo(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
@@ -64,17 +64,39 @@ class NotificationCallbackProviderConfig {
return cachingCertificates(uri -> fetchCertificate(client, uri));
}
private static byte[] fetchCertificate(HttpClient client, URI uri) {
/**
* Fetches the certificate document at a host-checked SNS URL.
*
* <p>The status decides what a failed fetch says. The URL already passed the SNS host check, so
* the host answering is SNS. A 4xx is SNS saying it has no such certificate: a real SNS message
* names one that exists, so the envelope is forged — {@link IllegalArgumentException}, which the
* verifier turns into an invalid signature that is refused and audited. It used to be "cannot
* verify now" like every other non-2xx, so a forged envelope pointing at a certificate SNS never
* issued got a 503, was never audited, and was logged as a provider hiccup (review 2026-11-03
* RR-07-03). A 5xx, a 408 or 429 (slow or throttling host), or a network failure says nothing
* about the certificate and stays "cannot verify now".
*
* @param client the no-redirect client
* @param uri the host-checked certificate URL
* @return the certificate document, at most {@link #MAX_SNS_CERTIFICATE_BYTES}
* @throws IllegalArgumentException when SNS answers that the certificate does not exist
* @throws CallbackVerificationUnavailableException when the certificate could not be fetched now
*/
static byte[] fetchCertificate(HttpClient client, URI uri) {
HttpRequest request =
HttpRequest.newBuilder(uri).timeout(SNS_CERTIFICATE_TIMEOUT).GET().build();
try {
HttpResponse<InputStream> response =
client.send(request, HttpResponse.BodyHandlers.ofInputStream());
try (InputStream body = response.body()) {
if (response.statusCode() / 100 != 2) {
int status = response.statusCode();
if (status / 100 == 4 && status != 408 && status != 429) {
throw new IllegalArgumentException(
"SNS has no signing certificate at the URL the message names (HTTP " + status + ")");
}
if (status / 100 != 2) {
throw certificateUnavailable(
new IOException(
"SNS signing certificate fetch failed with HTTP " + response.statusCode()));
new IOException("SNS signing certificate fetch failed with HTTP " + status));
}
return body.readNBytes(MAX_SNS_CERTIFICATE_BYTES);
}
@@ -16,6 +16,7 @@ import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import tools.jackson.databind.JsonNode;
@@ -36,8 +37,12 @@ import tools.jackson.databind.JsonNode;
* signed string), for up to {@link #SNS_HTTP_RETRY_HORIZON}. A window of five minutes threw away
* every bounce and complaint that arrived after an outage longer than that, and with them the
* suppressions they should have created (review 05-20). Accepting a replay inside the window costs
* nothing: the provider event ledger is unique on {@code (provider_profile_id, provider_event_id)},
* so a second delivery of the same event is a duplicate, not a second event.
* nothing: the provider event ledger is unique on {@code (provider_profile_id, provider_event_id)}
* whenever an event id is present, and this adapter gives every SES event the signed SNS {@code
* MessageId} as that id, so a second delivery of the same message is a duplicate, not a second
* event. It used to give none, and the only identity left was the {@code (provider_profile_id,
* event_fingerprint)} index — a fingerprint keyed by the active HMAC key, so a redelivery after a
* key rotation inside the window was recorded twice (review 2026-11-03 RR-07-07).
*/
public final class SesCallbackAdapter implements ProviderCallbackAdapter {
@@ -118,7 +123,10 @@ public final class SesCallbackAdapter implements ProviderCallbackAdapter {
return List.of();
}
String message = envelope.getOrDefault("Message", "{}");
return normalizer.normalize(message);
// The SNS MessageId is in the signed string and survives every redelivery, so it — not a
// fingerprint under the current HMAC key — is what the ledger recognises a replay by.
return normalizer.normalize(
message, Optional.ofNullable(envelope.get("MessageId")).filter(id -> !id.isBlank()));
}
private boolean withinReplayWindow(String timestamp, Instant receivedAt) {
@@ -7,6 +7,7 @@ import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import tools.jackson.databind.JsonNode;
@@ -19,8 +20,16 @@ import tools.jackson.databind.JsonNode;
*/
public final class SesEventNormalizer {
/** Normalize one SES notification payload. */
public List<NormalizedProviderEvent> normalize(String payload) {
/**
* Normalize one SES notification payload.
*
* @param payload the SES event JSON (the SNS envelope's {@code Message})
* @param providerEventId the identity of the delivery that carried it — the signed SNS {@code
* MessageId}, which SNS keeps across redeliveries — or empty when the envelope had none
* @return the provider events, each carrying {@code providerEventId}
*/
public List<NormalizedProviderEvent> normalize(String payload, Optional<String> providerEventId) {
Objects.requireNonNull(providerEventId, "providerEventId");
JsonNode root = NotificationJsonMapper.mapper().readTree(payload);
String type = text(root, "eventType").orElse(text(root, "notificationType").orElse("Unknown"));
Optional<String> messageId =
@@ -30,24 +39,26 @@ public final class SesEventNormalizer {
List<NormalizedProviderEvent> events = new ArrayList<>(1);
events.add(
switch (type) {
case "Send" ->
event(NormalizedEventType.PROVIDER_ACCEPTED, type, messageId, occurredAt, Map.of());
case "Send" -> event(NormalizedEventType.PROVIDER_ACCEPTED, type, messageId, occurredAt);
case "Delivery" ->
event(NormalizedEventType.DELIVERY_CONFIRMED, type, messageId, occurredAt, Map.of());
event(NormalizedEventType.DELIVERY_CONFIRMED, type, messageId, occurredAt);
case "DeliveryDelay" ->
event(NormalizedEventType.DELIVERY_DELAYED, type, messageId, occurredAt, Map.of());
event(NormalizedEventType.DELIVERY_DELAYED, type, messageId, occurredAt);
case "Bounce" -> bounce(root, type, messageId, occurredAt);
case "Complaint" ->
event(NormalizedEventType.COMPLAINT, type, messageId, occurredAt, Map.of());
case "Complaint" -> event(NormalizedEventType.COMPLAINT, type, messageId, occurredAt);
case "Reject" ->
event(NormalizedEventType.PROVIDER_REJECTED, type, messageId, occurredAt, Map.of());
event(NormalizedEventType.PROVIDER_REJECTED, type, messageId, occurredAt);
case "RenderingFailure" ->
event(NormalizedEventType.TEMPLATE_FAILURE, type, messageId, occurredAt, Map.of());
case "Open" -> event(NormalizedEventType.OPENED, type, messageId, occurredAt, Map.of());
case "Click" -> event(NormalizedEventType.CLICKED, type, messageId, occurredAt, Map.of());
default -> event(NormalizedEventType.UNKNOWN, type, messageId, occurredAt, Map.of());
event(NormalizedEventType.TEMPLATE_FAILURE, type, messageId, occurredAt);
case "Open" -> event(NormalizedEventType.OPENED, type, messageId, occurredAt);
case "Click" -> event(NormalizedEventType.CLICKED, type, messageId, occurredAt);
default -> event(NormalizedEventType.UNKNOWN, type, messageId, occurredAt);
});
return List.copyOf(events);
// One SNS delivery carries one SES event, so the delivery's id is the event's. It is what makes
// a redelivery a duplicate in the ledger: the other identity, the event fingerprint, is keyed
// by
// the active HMAC key and changes when the key rotates (review 2026-11-03 RR-07-07).
return events.stream().map(event -> withEventId(event, providerEventId)).toList();
}
private static NormalizedProviderEvent bounce(
@@ -68,6 +79,14 @@ public final class SesEventNormalizer {
Map.of("bounceType", bounceType));
}
private static NormalizedProviderEvent event(
NormalizedEventType type,
String nativeType,
Optional<String> messageId,
Optional<Instant> occurredAt) {
return event(type, nativeType, messageId, occurredAt, Map.of());
}
private static NormalizedProviderEvent event(
NormalizedEventType type,
String nativeType,
@@ -78,6 +97,17 @@ public final class SesEventNormalizer {
type, nativeType, Optional.empty(), messageId, occurredAt, attributes);
}
private static NormalizedProviderEvent withEventId(
NormalizedProviderEvent event, Optional<String> providerEventId) {
return new NormalizedProviderEvent(
event.type(),
event.providerNativeType(),
providerEventId,
event.providerRequestId(),
event.providerOccurredAt(),
event.attributes());
}
private static Optional<String> text(JsonNode node, String field) {
JsonNode value = node.get(field);
return value == null || value.isNull() ? Optional.empty() : Optional.of(value.asString());
@@ -112,7 +112,12 @@ public final class JavaMailSenderSmtpDispatch implements SmtpDispatch {
} else if (cause instanceof SMTPSendFailedException send) {
code = send.getReturnCode();
}
return code >= 200 && code < 600 ? Optional.of(code) : Optional.empty();
// Only a refusal is a reply code here: 4xx (transient) and 5xx (permanent). Angus's sendpartial
// path throws SMTPSendFailedException carrying the last reply, which can be a success (250);
// read as a reply code it reached the classifier as a permanent refusal of a message the relay
// had partly accepted. Anything else takes the conservative "may have been delivered" path
// below (review 2026-11-03 RR-04-11).
return code >= 400 && code < 600 ? Optional.of(code) : Optional.empty();
}
/**
@@ -0,0 +1,104 @@
package dev.caskeleton.adapter.outbound.notification.platform.autoconfigure;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.catchThrowable;
import com.sun.net.httpserver.HttpServer;
import dev.caskeleton.adapter.outbound.notification.platform.callback.CallbackVerificationUnavailableException;
import java.io.IOException;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.ServerSocket;
import java.net.URI;
import java.net.http.HttpClient;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
/**
* What a failed SNS signing-certificate fetch says about the callback (review 2026-11-03 RR-07-03).
*
* <p>The URL has passed the SNS host check before it is fetched, so the host is SNS's. A real SNS
* message names a certificate that exists; when SNS itself answers 4xx for it, the envelope named a
* certificate SNS never issued — a forged callback, to be rejected and audited. Only a failure that
* says nothing about the certificate — a 5xx, a 408 or 429 (the host is slow or throttling), or a
* network failure — is "cannot verify now" (503, the provider retries).
*/
class SnsCertificateFetchTest {
private final HttpClient client = HttpClient.newHttpClient();
@ParameterizedTest
@ValueSource(ints = {403, 404, 410})
@DisplayName("인증서 URL이 4xx로 답하면 없는 인증서를 가리킨 위조 콜백이므로 검증 실패로 거절된다")
void aCertificateTheHostDoesNotHaveFailsVerification(int status) throws IOException {
// given
HttpServer server = answering(status);
try {
URI url = URI.create("http://127.0.0.1:" + server.getAddress().getPort() + "/cert.pem");
// when
Throwable failure =
catchThrowable(() -> NotificationCallbackProviderConfig.fetchCertificate(client, url));
// then
assertThat(failure)
.isInstanceOf(IllegalArgumentException.class)
.isNotInstanceOf(CallbackVerificationUnavailableException.class);
} finally {
server.stop(0);
}
}
@ParameterizedTest
@ValueSource(ints = {408, 429, 500, 503})
@DisplayName("인증서 URL이 5xx나 시간 초과·요청 제한(408·429)으로 답하면 아무것도 모르므로 지금 검증할 수 없다고 답한다")
void aFailingCertificateHostLeavesTheCallbackUnverified(int status) throws IOException {
// given
HttpServer server = answering(status);
try {
URI url = URI.create("http://127.0.0.1:" + server.getAddress().getPort() + "/cert.pem");
// when
Throwable failure =
catchThrowable(() -> NotificationCallbackProviderConfig.fetchCertificate(client, url));
// then
assertThat(failure).isInstanceOf(CallbackVerificationUnavailableException.class);
} finally {
server.stop(0);
}
}
@Test
@DisplayName("인증서 호스트에 닿지 못하면 지금 검증할 수 없다고 답한다")
void anUnreachableCertificateHostLeavesTheCallbackUnverified() throws IOException {
// given
int closedPort;
try (ServerSocket probe = new ServerSocket(0)) {
closedPort = probe.getLocalPort();
}
URI url = URI.create("http://127.0.0.1:" + closedPort + "/cert.pem");
// when
Throwable failure =
catchThrowable(() -> NotificationCallbackProviderConfig.fetchCertificate(client, url));
// then
assertThat(failure).isInstanceOf(CallbackVerificationUnavailableException.class);
}
private static HttpServer answering(int status) throws IOException {
HttpServer server =
HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress(), 0), 0);
server.createContext(
"/",
exchange -> {
exchange.sendResponseHeaders(status, -1);
exchange.close();
});
server.start();
return server;
}
}
@@ -8,6 +8,7 @@ import dev.caskeleton.adapter.outbound.notification.platform.callback.CallbackVe
import dev.caskeleton.adapter.outbound.notification.platform.callback.CallbackVerificationUnavailableException;
import dev.caskeleton.adapter.outbound.notification.platform.callback.ProviderCallbackVerificationRequest;
import dev.caskeleton.adapter.outbound.notification.platform.template.NotificationJsonMapper;
import dev.caskeleton.application.notification.platform.api.NormalizedProviderEvent;
import dev.caskeleton.application.notification.platform.api.ProviderId;
import dev.caskeleton.application.notification.platform.api.ProviderProfileId;
import java.nio.charset.StandardCharsets;
@@ -19,6 +20,7 @@ import java.time.Duration;
import java.time.Instant;
import java.util.Base64;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
@@ -135,6 +137,28 @@ class SesCallbackAdapterTest {
assertThat(unavailable.reasonCode()).isEqualTo("SNS_CERTIFICATE_UNAVAILABLE"));
}
@Test
@DisplayName(
"SES 이벤트의 provider event id는 서명된 SNS MessageId라 fingerprint 키를 회전해도 같은 콜백이 같은 이벤트로 남는다")
void theSignedSnsMessageIdIsTheProviderEventId() {
// given
CallbackVerificationResult verified =
adapter.verify(request(signed(ALLOWED_TOPIC, RECEIVED_AT)));
// when
List<NormalizedProviderEvent> events =
adapter.normalize(verified.verifiedCallback().orElseThrow());
// then
// The ledger is unique on (profile, provider_event_id) whenever an id is present. Without one
// the only identity was a fingerprint keyed by the active HMAC key, so a redelivery after a
// key rotation inside the one-hour replay window became a second row (RR-07-07).
assertThat(events)
.singleElement()
.extracting(NormalizedProviderEvent::providerEventId)
.isEqualTo(Optional.of("sns-message-1"));
}
/** The adapter with the window a deployment gets when it sets nothing. */
private static SesCallbackAdapter withDefaultWindow() {
return new SesCallbackAdapter(
@@ -16,9 +16,11 @@ import java.nio.charset.StandardCharsets;
import java.util.List;
import java.util.Optional;
import java.util.Properties;
import org.eclipse.angus.mail.smtp.SMTPSendFailedException;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.springframework.mail.MailAuthenticationException;
import org.springframework.mail.MailSendException;
import org.springframework.mail.javamail.JavaMailSenderImpl;
/**
@@ -136,6 +138,38 @@ class JavaMailSenderSmtpDispatchTest {
.containsExactly(Optional.empty(), false);
}
@Test
@DisplayName("성공 코드(250)를 실은 부분 전송 실패는 거절 reply code가 아니라 전달됐을 수 있는 실패로 보고된다")
void aPartialSendCarryingASuccessCodeIsNotReportedAsARefusal() {
// given
// Angus's sendpartial path reports a partly delivered message as SMTPSendFailedException
// carrying the last reply, which is a success (250), not a refusal.
JavaMailSenderImpl partlyDelivered =
new JavaMailSenderImpl() {
@Override
public void send(MimeMessage... messages) {
throw new MailSendException(
"partly sent",
new SMTPSendFailedException(
"DATA",
250,
"250 2.0.0 queued",
null,
new jakarta.mail.Address[0],
new jakarta.mail.Address[0],
new jakarta.mail.Address[0]));
}
};
JavaMailSenderSmtpDispatch dispatch = new JavaMailSenderSmtpDispatch(partlyDelivered);
// when
SmtpDispatchException failure = sendExpectingFailure(dispatch);
// then
assertThat(List.of(failure.replyCode(), failure.dataCommitted()))
.containsExactly(Optional.empty(), true);
}
private static SmtpDispatchException sendExpectingFailure(JavaMailSenderSmtpDispatch dispatch) {
return catchThrowableOfType(SmtpDispatchException.class, () -> dispatch.send(message()));
}
@@ -155,9 +155,9 @@ strictTestLanes {
}
}
// The MinIO lanes start a container, so they are the main-branch stage of this leaf, not its
// pull-request `check` (review BUILD-계획-B3); object-storage-qualification.yml runs them on pull
// requests that touch object storage, and integrationCheck runs them with every integration lane.
tasks.named('mainCheck') {
dependsOn 'objectStorageMinioContractTest', 'objectStorageMinioFaultTest'
}
// The MinIO lanes start a container, so they are not this leaf's pull-request `check` (review
// BUILD-계획-B3). They are integration lanes, and the root integrationCheck owns every container lane
// on main and nightly; object-storage-qualification.yml runs them on pull requests that touch object
// storage. This leaf's `mainCheck` used to run them too, so each main push ran them twice in two
// parallel jobs (review 2026-11-03 RR-06-09, original 08-04 (d)).
@@ -46,12 +46,18 @@ in `platform`), so the two capability leaves read the same way (review 04-16).
write; Spring Data repositories are mocks).
- `FileserverPersistenceActivationTest` — the scan covers this package only, behind
`app.fileserver-platform.enabled`, and runs before the JPA root auto-configuration.
- `repository.CleanupClaimSqlTest` — the claim keeps its one-statement shape (the Claims rule).
- `repository.CleanupClaimSqlTest` — the claim's text keeps its one-statement shape (the Claims
rule); the locking itself is checked on PostgreSQL by
`PostgreSqlFileserverReclamationIntegrationTest`.
- `entity.AssignedIdEntityTest` — newness from `Persistable`, on Hibernate over H2.
A new store write adds its case to the guard test; a new auto-configuration adds its line to the
activation test.
- Port contracts live in `application-fileserver` testFixtures (`*PortContract`). The in-memory
fakes and the PostgreSQL tests both extend them.
- Port contracts live in `application-fileserver` testFixtures (`*PortContract`, and
`FileQuotaLedgerContract` for the three quota ports, which share one table). The in-memory fakes
and the PostgreSQL tests both extend them; every port this leaf implements has one
(`docs/testing/TESTING_STRATEGY.md` 9.3). The PostgreSQL side of the seven added in review
2026-11-03 RR-01-04 is tagged `jpa-contract` and runs in persistence-jpa's
`jpaPlatformContractTest` lane on `FileserverPostgreSqlStack`, whose clock a case moves.
- The real-PostgreSQL tests are still in `persistence-jpa`'s `postgresqlIntegrationTest` source
set, which owns the container and Flyway wiring, and the JPA readiness card
(`config/jpa/readiness-cards.yaml`, checked by `JpaReadinessRegistryTest`) names them as that
@@ -18,8 +18,11 @@ import java.util.UUID;
/**
* JPA-backed {@link FileQuotaPort}.
*
* <p>Reservation, extension, commit, and release are conditional statements, so a reservation that
* already expired or was released can never be extended or committed.
* <p>Reservation, extension, commit, and release are conditional statements: a reservation that
* expired or was released can never be extended, and a released or already committed one can never
* be committed. An expired reservation that was not released still commits — bytes that were
* written are durable usage whether or not the reservation outlived its TTL. {@code
* FileQuotaLedgerContract} holds this adapter and the in-memory ledger to the same cases.
*/
public class JpaFileQuota implements FileQuotaPort {
@@ -15,8 +15,8 @@ package dev.caskeleton.adapter.outbound.persistence.fileserver.repository;
* every claimed item is fenced on its own.
*
* <p>Package-private: the repository is its one reader, and the leaf's public surface is a ceiling.
* {@link #positional(String)} rewrites the named parameters to JDBC placeholders for a test in this
* package that runs the identical text on a bare {@code Connection}.
* The PostgreSQL lane runs it through the repository ({@code
* PostgreSqlFileserverReclamationIntegrationTest}); nothing runs the text on its own.
*/
final class CleanupClaimSql {
@@ -43,14 +43,4 @@ final class CleanupClaimSql {
+ " c.file_id, c.upload_id, c.content_key";
private CleanupClaimSql() {}
/**
* Rewrites the named parameters to JDBC placeholders, left to right.
*
* @param namedSql the statement above
* @return the same statement with {@code ?} in place of every {@code :name}
*/
static String positional(String namedSql) {
return namedSql.replaceAll("(?<!:):[A-Za-z][A-Za-z0-9]*", "?");
}
}
@@ -7,16 +7,17 @@ import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
/**
* The claim statement keeps the shape the leaf's claim rule names: one statement that locks the
* rows it picks, skips rows another claim holds, gives every row a fresh token and a higher fence,
* and returns them in the same round trip. The PostgreSQL lane runs it; this pins the shape without
* a database, so a rewrite into a select-then-update fails here first.
* The claim statement's text keeps the shape the leaf's claim rule names: one statement that picks
* rows with {@code FOR UPDATE SKIP LOCKED}, gives every row a fresh token and a higher fence, and
* returns them. This pins the text without a database, so a rewrite into a select-then-update fails
* here first. Whether the statement actually locks, skips and fences is a behaviour of PostgreSQL,
* and {@code PostgreSqlFileserverReclamationIntegrationTest} runs it there.
*/
class CleanupClaimSqlTest {
@Test
@DisplayName("claim은 고른 행을 잠그고, 다른 claim이 잡은 행은 기다리지 않고 건너뛴다")
void theClaimLocksWhatItPicksAndSkipsWhatAnotherClaimHolds() {
@DisplayName("claim 문장은 한 문장의 CTE 안에서 FOR UPDATE SKIP LOCKED로 행을 고른다")
void theClaimTextPicksRowsWithForUpdateSkipLockedInOneStatement() {
// when
String claim = CleanupClaimSql.CLAIM_DUE;
@@ -25,8 +26,8 @@ class CleanupClaimSqlTest {
}
@Test
@DisplayName("claim한 행마다 새 토큰과 하나 높은 fence를 받고 같은 왕복에 돌아온다")
void everyClaimedRowGetsAFreshTokenAndAHigherFenceInTheSameRoundTrip() {
@DisplayName("claim 문장은 행마다 새 토큰을 쓰고 fence를 하나 올리고 RETURNING으로 돌려준다")
void theClaimTextWritesAFreshTokenRaisesTheFenceAndReturnsTheRows() {
// when
String claim = CleanupClaimSql.CLAIM_DUE;
@@ -38,14 +39,4 @@ class CleanupClaimSqlTest {
claim.contains("RETURNING c.cleanup_id, c.claim_token")))
.containsOnly(true);
}
@Test
@DisplayName("이름 있는 매개변수를 JDBC 자리표시자로 바꿀 때 PostgreSQL 형 변환은 건드리지 않는다")
void positionalParametersDoNotRewritePostgreSqlCasts() {
// when
String positional = CleanupClaimSql.positional("payload::jsonb @> :expected::jsonb");
// then
assertThat(positional).isEqualTo("payload::jsonb @> ?::jsonb");
}
}
@@ -235,7 +235,11 @@ made Spring call `getTransactionIsolation()` at every begin, which pgjdbc answer
Every mode is time-bounded (review 04-10). Spring hands the remaining time to each statement, so a
slow query or lock wait is cancelled by the server (`57014` → `DB_QUERY_CANCELED`), and a deadline
that passed before the next statement (`TransactionTimedOutException`) gets the same code.
that passed before the next statement gets the same code: Spring's `TransactionTimedOutException`
on the JDBC path, and on the JPA path (`JpaTransactionManager`, production) Hibernate's own
`TransactionException`, which has no SQLState and is classified because `SpringTransaction` knows
its deadline passed (`translateAfterDeadline`, review RR-03-03). A business exception thrown after
the deadline is passed on unchanged.
Pre-built templates are immutable after construction so concurrent callers cannot
observe each other's reconfiguration. `inRootWrite` reuses the pre-built write template,
@@ -302,7 +306,9 @@ the two paths disagree on any state.
**No SQLState, classified by type:** a lost `@Version` race (`OptimisticLockingFailureException`,
JPA `OptimisticLockException`) → `DB_OPTIMISTIC_CONFLICT` (409, review 04-05); an expired Spring
transaction deadline (`TransactionTimedOutException`) → `DB_QUERY_CANCELED`.
transaction deadline (`TransactionTimedOutException`) → `DB_QUERY_CANCELED`; any other persistence
failure (JPA, Spring data-access or Spring transaction exception) of a transaction
`SpringTransaction` began, once its deadline has passed → `DB_QUERY_CANCELED`.
**Commit phase (`translateCommitFailure`, ADR-JPA-003):** once the action has returned and
`SpringTransaction` is committing a write transaction it began, a failure with `08*`, `40003`, a
+15 -3
View File
@@ -296,14 +296,26 @@ A가 FAILED로 바꾸면 다시 발행되고(중복 전달), A의 늦은 PUBLISH
`WHERE event_id = ? AND status = 'IN_FLIGHT' AND lease_owner = ? AND lease_token = ?`.
owner는 비우고 token은 그대로 둔다(token은 증가만 한다). 1 row면 `APPLIED`, 0 row면 `STALE`.
`findById` + setter 경로와 `@Version`은 쓰지 않는다.
- **실패 원인 코드(`V22`)**: `markFailed`·`markDeferred`·`markDead`는 같은 조건부 UPDATE에서
`last_failure_code`를 쓴다(publisher가 붙인 코드, 없으면 결과·예외 타입 이름). entity 필드는 읽기
전용이라 append INSERT와 claim flush는 이 열을 건드리지 않는다. `longRetries(n, now)`는 미룬 claim이
n번 이상인 FAILED 행 수와 가장 오래된 행의 나이를 센다(`outbox.retry.long.*` 게이지). 상태 `PARKED`는
DDL이 없다 — claim이 PENDING·FAILED·IN_FLIGHT만 고르고, FIFO 게이트는 PUBLISHED 아닌 모든 행을 막는 앞
행으로 본다. claim이 시도 횟수가 어긋난 행(`deferred_attempt_count > attempt_count`)을 만나면 DEAD로
옮기고 WARN `error.reason=ATTEMPT_COUNTERS_INCONSISTENT`로 원인을 말한다(재리뷰 2026-11-03 RR-05-01·03).
- **미룬 실패와 claim 해제(`V20`)**: `markDeferred(lease, next)`는 `markFailed`와 같은 전이에
`deferred_attempt_count + 1`을 더한다(일시적 실패는 시도 예산 밖). `releaseLease(lease, at)`는
손대지 않은 claim을 `PENDING`, `next_attempt_at = at`, `attempt_count - 1`로 되돌린다(종료 중인
relay가 쓴다). 둘 다 같은 lease 조건부 UPDATE다.
- **aggregate 순서 번호(`V21`)**: `JpaOutboxStore`는 `OutboxAggregateSequencePort`도 구현한다.
`outbox_aggregate_sequence` counter를 `UPDATE ... + 1`(row lock)로 올리고, 없으면 그 aggregate의 기존
outbox 최대 `aggregate_sequence` + 1로 만든다. V21은 `outbox_event`의 순서 키 유일 인덱스도 만든다 —
기존 중복 row가 있으면 migration이 실패하므로 먼저 정리해야 한다.
`outbox_aggregate_sequence` counter를 `UPDATE`(row lock)로 올린다. 올린 값은 그 aggregate의 가장 큰
outbox 번호보다 작아지지 않는다(rolling 배포 중 옛 인스턴스가 쓴 벽시계 번호가 먼저 온다). counter가
없으면 `INSERT ... ON CONFLICT DO NOTHING`으로 만든다. 시작값은 남은 outbox 최대 번호와 V21이
기록한 floor(`outbox_aggregate_sequence_floor`, V21 적용 시각의 epoch 마이크로초) 중 큰 값 + 1이다.
첫 쓰기 두 개가 겹치면 두 번째는 첫 번째 커밋을 기다렸다가 아무것도 넣지 않고 증가만 한다(리뷰
RR-03-01). V21은 남은 row가 있는 aggregate의 counter를 그 최대 번호로 미리 만들고, `outbox_event`의
순서 키 유일 인덱스도 만든다 — 기존 중복 row가 있으면 migration이 실패하므로 먼저 정리해야 한다
(`docs/runbooks/outbox-dead-letter.md` §8).
- **STALE은 예외가 아니다**: row의 결과는 지금 lease를 가진 relay가 쓴다. use case는 결과를
`OutboxRelayResult.Outcome.STALE`로 남기고 `OutboxRelayFailureReportPort.reportStaleCompletion`
으로 알린 뒤 다음 이벤트로 넘어간다. FAILED/DEAD report는 보내지 않는다(아무것도 저장되지
@@ -187,6 +187,7 @@ def readinessLanes = [
postgresqlOutboxRowLifecycleIntegrationTest : ['PostgreSqlOutboxRowLifecycleIntegrationTest'],
postgresqlOutboxAggregateSequenceIntegrationTest : ['PostgreSqlOutboxAggregateSequenceIntegrationTest'],
postgresqlOutboxOrderKeyRunbookIntegrationTest : ['PostgreSqlOutboxOrderKeyRunbookIntegrationTest'],
postgresqlOutboxDispositionRunbookIntegrationTest : ['PostgreSqlOutboxDispositionRunbookIntegrationTest'],
postgresqlFileserverMigrationIntegrationTest : ['PostgreSqlFileserverMigrationIntegrationTest'],
postgresqlFileserverMetadataIntegrationTest : ['PostgreSqlFileserverMetadataStoreIntegrationTest'],
postgresqlFileserverReclamationIntegrationTest : ['PostgreSqlFileserverReclamationIntegrationTest'],
@@ -243,13 +244,20 @@ strictTestLanes {
}
}
// The V21 pre-deploy check and cleanup are SQL an operator runs from the outbox runbook (review
// 2026-11-01, README 15 decision). The lane reads them out of the runbook, so the document is one of
// its inputs: editing either block reruns the lane against real PostgreSQL.
tasks.named('postgresqlOutboxOrderKeyRunbookIntegrationTest', Test) {
def outboxRunbook = new File(rootProject.projectDir.parentFile, 'docs/runbooks/outbox-dead-letter.md')
inputs.file(outboxRunbook).withPathSensitivity(PathSensitivity.RELATIVE).withPropertyName('outboxRunbook')
systemProperty 'outbox.runbook', outboxRunbook.absolutePath
// The outbox runbooks give an operator SQL to run: the V21 pre-deploy check and cleanup (review
// 2026-11-01, README 15 decision), the manual dispositions of a DEAD row (re-review 2026-11-03
// RR-05-01), and finding, parking, resuming and skipping a row that is retried without end
// (RR-05-03). Each lane reads its blocks out of the runbooks, so the documents are its inputs:
// editing a block reruns the lane against real PostgreSQL.
['postgresqlOutboxOrderKeyRunbookIntegrationTest', 'postgresqlOutboxDispositionRunbookIntegrationTest'].each { laneName ->
tasks.named(laneName, Test) {
def outboxRunbook = new File(rootProject.projectDir.parentFile, 'docs/runbooks/outbox-dead-letter.md')
inputs.file(outboxRunbook).withPathSensitivity(PathSensitivity.RELATIVE).withPropertyName('outboxRunbook')
systemProperty 'outbox.runbook', outboxRunbook.absolutePath
def publishFailedRunbook = new File(rootProject.projectDir.parentFile, 'docs/runbooks/outbox-publish-failed.md')
inputs.file(publishFailedRunbook).withPathSensitivity(PathSensitivity.RELATIVE).withPropertyName('publishFailedRunbook')
systemProperty 'outbox.publish-failed.runbook', publishFailedRunbook.absolutePath
}
}
tasks.named('postgresqlSecurityBaselineIntegrationTest') {
@@ -46,8 +46,15 @@ public enum FailureCategory {
QUERY_TIMEOUT,
/**
* {@code 25P03} — the server ended a session that sat idle inside an open transaction. Nothing
* was committed, which is why its code is retryable where {@link #QUERY_TIMEOUT}'s is not.
* Not a transaction timeout: {@code 25P03}, the server ended a session that sat idle inside an
* open transaction ({@code idle_in_transaction_session_timeout}). Nothing was committed, which is
* why its code ({@code DB_IDLE_IN_TX_TIMEOUT}) is retryable.
*
* <p>A transaction that ran past its own deadline ({@code SpringTransaction.Timeouts}) is {@link
* #QUERY_TIMEOUT} ({@code DB_QUERY_CANCELED}), not this. The name says otherwise and is kept only
* because the enum is public API ({@code docs/architecture/jpa-api-surface.txt}); it becomes
* {@code IDLE_IN_TRANSACTION_TIMEOUT}, with {@link TransactionTimeoutException}, at the next
* release that breaks this API anyway (review RR-03-10).
*/
TRANSACTION_TIMEOUT,
@@ -1,10 +1,13 @@
package dev.caskeleton.adapter.outbound.persistence.api.error;
/**
* A transaction exceeded its configured transaction timeout (design §15.4).
* The server ended a session that sat idle inside an open transaction ({@code 25P03}, {@link
* FailureCategory#TRANSACTION_TIMEOUT}). Not a transaction that ran past its own deadline: that is
* a cancelled statement ({@link FailureCategory#QUERY_TIMEOUT}).
*
* <p>The transaction is rolled back; whether the use case may be re-run is a business decision, so
* the platform does not retry it automatically.
* <p>The transaction is rolled back and nothing was committed, so its code is retryable; whether
* the use case is re-run is still the caller's decision, and the platform does not retry it
* automatically. The name is kept until the next API-breaking release (review RR-03-10).
*/
public final class TransactionTimeoutException extends JpaPersistenceException {
@@ -8,6 +8,7 @@ import dev.caskeleton.shared.error.ApiErrorCode;
import dev.caskeleton.shared.error.OperationalError;
import dev.caskeleton.shared.error.PersistenceFailureException;
import jakarta.persistence.OptimisticLockException;
import jakarta.persistence.PersistenceException;
import java.io.IOException;
import java.sql.SQLException;
import java.sql.SQLNonTransientConnectionException;
@@ -23,6 +24,7 @@ import java.util.Optional;
import java.util.Set;
import org.springframework.dao.DataAccessException;
import org.springframework.dao.OptimisticLockingFailureException;
import org.springframework.transaction.TransactionException;
import org.springframework.transaction.TransactionTimedOutException;
/**
@@ -140,6 +142,42 @@ public final class PersistenceExceptionTranslator {
return Optional.empty();
}
/**
* Classify a failure raised once the transaction's deadline had passed, when neither a SQLState
* nor a type above identifies it.
*
* <p>Past the deadline Hibernate refuses the next statement itself, before the driver is asked:
* an {@code org.hibernate.TransactionException} with no SQLState that is not Spring's {@link
* TransactionTimedOutException} — in the action as it is, at commit wrapped in a {@code
* JpaSystemException} (review RR-03-03). That is the production path ({@code
* JpaTransactionManager}); the JDBC path gets {@code 57014} from the server instead. Both are
* {@code DB_QUERY_CANCELED}. The caller decides that the deadline has passed; this method only
* makes sure the failure came from the persistence stack — a JPA, Spring data-access or Spring
* transaction exception anywhere in the cause chain — so a business exception thrown late is
* passed on as it is, and one already classified is not classified again.
*/
public Optional<PersistenceFailureException> translateAfterDeadline(Throwable ex) {
Objects.requireNonNull(ex, "ex");
if (ex instanceof PersistenceFailureException) {
return Optional.empty();
}
for (Throwable t = ex; t != null; t = t.getCause()) {
if (t instanceof PersistenceException
|| t instanceof DataAccessException
|| t instanceof TransactionException) {
return Optional.of(
new PersistenceFailureException(
OperationalError.DB_QUERY_CANCELED,
"persistence failure after the transaction deadline passed",
ex));
}
if (t.getCause() == t) {
break;
}
}
return Optional.empty();
}
/**
* Classify a failure raised while committing a write transaction this call began.
*
@@ -14,6 +14,7 @@ import dev.caskeleton.application.outbox.OutboxAppendPort;
import dev.caskeleton.application.outbox.OutboxCompletion;
import dev.caskeleton.application.outbox.OutboxEventStatus;
import dev.caskeleton.application.outbox.OutboxLease;
import dev.caskeleton.application.outbox.OutboxLongRetries;
import dev.caskeleton.application.outbox.OutboxStorePort;
import dev.caskeleton.shared.digest.Sha256;
import java.nio.charset.StandardCharsets;
@@ -29,6 +30,7 @@ import java.util.Objects;
import java.util.Optional;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.spi.LoggingEventBuilder;
/**
* JPA-backed {@link OutboxAppendPort} + {@link OutboxStorePort}. {@link #append} and the claim/mark
@@ -103,12 +105,21 @@ public class JpaOutboxStore
*
* <p>The increment is an {@code UPDATE}, which holds the counter row's lock until the caller
* commits: a second writer of the same aggregate waits, then increments the committed value, so
* the numbers follow commit order. An aggregate's first number is inserted instead, seeded after
* the highest {@code aggregate_sequence} its outbox rows already carry — rows numbered before the
* counter existed (the sample used epoch microseconds) must still come first. Two writers
* creating the same counter at once collide on its primary key and one transaction fails: a
* rollback the caller retries, never two events with one number. The counter outlives the
* reaper's deletes, so a number is never handed out twice.
* the numbers follow commit order. It never goes below the aggregate's highest outbox row, so a
* row an instance still on the wall-clock scheme writes during a rolling deploy comes first. The
* counter outlives the reaper's deletes, so a number is never handed out twice.
*
* <p>An aggregate's first number is inserted instead, after the larger of its highest remaining
* row and the floor V21 recorded — the moment it ran, above every clock number already sent for
* rows the reaper has deleted or that were written under an identity that has since changed
* (review RR-03-02). Two writers creating the same counter at once both succeed: the insert is
* {@code ON CONFLICT DO NOTHING}, so the second waits for the first to commit, writes nothing,
* and increments the committed counter instead (review RR-03-01).
*
* <p>Statements rather than one {@code INSERT … ON CONFLICT DO UPDATE … RETURNING} like {@code
* JpaLiveEventReplay}'s position allocator: the sample appends events on the H2 developer run
* ({@code bootRunH2}), and H2 accepts {@code ON CONFLICT DO NOTHING} but not {@code DO UPDATE}.
* The common case — the counter exists — is still one {@code UPDATE} and one read.
*/
@Override
public long nextSequence(AggregateIdentity aggregate) {
@@ -118,8 +129,11 @@ public class JpaOutboxStore
String tenant = aggregate.tenantScope();
String type = aggregate.aggregateType();
String id = aggregate.aggregateId();
if (repository.incrementAggregateSequence(tenant, type, id) == 0) {
repository.createAggregateSequence(tenant, type, id);
if (repository.incrementAggregateSequence(tenant, type, id) == 0
&& repository.createAggregateSequence(tenant, type, id) == 0) {
// Another writer created the counter after our UPDATE found none; its commit ended our wait
// in the insert, so the counter is there to increment now.
repository.incrementAggregateSequence(tenant, type, id);
}
return repository.aggregateSequence(tenant, type, id);
}
@@ -132,6 +146,11 @@ public class JpaOutboxStore
* instead rolled the whole claim back, and the next claim selected the same row first again, so
* one bad row stopped the relay for good. DEAD is fail-closed: the row is never published, and it
* waits for the DEAD-row runbook like any other.
*
* <p>The WARN names the cause. A row whose deferred claims outnumber its claims is not damaged:
* it is what a manual re-drive that reset {@code attempt_count} and left {@code
* deferred_attempt_count} produces (re-review 2026-11-03 RR-05-01), and "cannot read" sent the
* operator looking for corruption instead of re-running the corrected runbook SQL.
*/
@Override
public List<OutboxLease> claimBatch(
@@ -145,13 +164,26 @@ public class JpaOutboxStore
entity.setNextAttemptAt(now.plus(inFlightTimeout));
entity.setLeaseOwner(owner);
entity.setLeaseToken(entity.getLeaseToken() + 1);
if (entity.getDeferredAttemptCount() > entity.getAttemptCount()) {
deadLetterOnClaim(entity)
.addKeyValue("error.reason", "ATTEMPT_COUNTERS_INCONSISTENT")
.log(
"outbox claim moved a row to DEAD: deferred_attempt_count ({}) exceeds"
+ " attempt_count ({}); a manual re-drive that did not reset"
+ " deferred_attempt_count leaves this behind — re-drive it with the runbook"
+ " SQL",
entity.getDeferredAttemptCount(),
entity.getAttemptCount());
continue;
}
ClaimedOutboxEvent event;
try {
event = toClaimedOutboxEvent(entity);
} catch (RuntimeException unreadable) {
entity.setStatus(OutboxEventStatus.DEAD.name());
entity.setLeaseOwner(null);
log.warn("outbox claim moved a row it cannot read to DEAD", unreadable);
deadLetterOnClaim(entity)
.addKeyValue("error.reason", "UNREADABLE_ROW")
.setCause(unreadable)
.log("outbox claim moved a row it cannot read as an event to DEAD");
continue;
}
leases.add(new OutboxLease(event, owner, entity.getLeaseToken()));
@@ -159,6 +191,15 @@ public class JpaOutboxStore
return List.copyOf(leases);
}
/** Moves a claimed row it will not hand out to DEAD and starts the WARN that says why. */
private static LoggingEventBuilder deadLetterOnClaim(OutboxEventEntity entity) {
entity.setStatus(OutboxEventStatus.DEAD.name());
entity.setLeaseOwner(null);
return log.atWarn()
.addKeyValue("event_id", entity.getEventId())
.addKeyValue("runbook_link", "runbook://outbox/dead-letter");
}
/**
* One conditional UPDATE, no load-and-set: it applies only while the lease still holds the row
* (see README "outbox"). Zero rows is {@link OutboxCompletion#STALE}, not an error — the relay
@@ -174,18 +215,21 @@ public class JpaOutboxStore
/** Marks the leased row FAILED with its next attempt time; see {@link #markPublished}. */
@Override
public OutboxCompletion markFailed(OutboxLease lease, Instant nextAttemptAt) {
public OutboxCompletion markFailed(OutboxLease lease, Instant nextAttemptAt, String failureCode) {
requireTransaction("markFailed");
return completion(
repository.failLease(lease.eventId(), lease.owner(), lease.token(), nextAttemptAt));
repository.failLease(
lease.eventId(), lease.owner(), lease.token(), nextAttemptAt, failureCode));
}
/** Marks the leased row FAILED as a deferred claim; see {@link #markPublished}. */
@Override
public OutboxCompletion markDeferred(OutboxLease lease, Instant nextAttemptAt) {
public OutboxCompletion markDeferred(
OutboxLease lease, Instant nextAttemptAt, String failureCode) {
requireTransaction("markDeferred");
return completion(
repository.deferLease(lease.eventId(), lease.owner(), lease.token(), nextAttemptAt));
repository.deferLease(
lease.eventId(), lease.owner(), lease.token(), nextAttemptAt, failureCode));
}
/** Returns the leased row to PENDING without spending an attempt; see {@link #markPublished}. */
@@ -198,11 +242,10 @@ public class JpaOutboxStore
/** Marks the leased row DEAD; see {@link #markPublished}. */
@Override
public OutboxCompletion markDead(OutboxLease lease) {
public OutboxCompletion markDead(OutboxLease lease, String failureCode) {
requireTransaction("markDead");
return completion(
repository.completeLease(
lease.eventId(), lease.owner(), lease.token(), OutboxEventStatus.DEAD.name()));
repository.deadLease(lease.eventId(), lease.owner(), lease.token(), failureCode));
}
private static OutboxCompletion completion(int updatedRows) {
@@ -237,6 +280,21 @@ public class JpaOutboxStore
return result;
}
@Override
public OutboxLongRetries longRetries(int minDeferredClaims, Instant now) {
if (minDeferredClaims < 1) {
throw new IllegalArgumentException(
"minDeferredClaims must be >= 1, was " + minDeferredClaims);
}
Object[] row = repository.countLongRetries(minDeferredClaims).getFirst();
long count = (Long) row[0];
if (count == 0) {
return OutboxLongRetries.NONE;
}
Instant oldest = (Instant) row[1];
return new OutboxLongRetries(count, Math.max(0, Duration.between(oldest, now).toSeconds()));
}
/**
* Restores the appended event. A row without its envelope — written by the pre-canonical
* generation, or damaged — is unreadable, and so is one whose stored hash is not the hash of its
@@ -62,18 +62,22 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
@Param("token") long token,
@Param("status") String status);
/** {@link #completeLease} to FAILED, scheduling the retry in the same statement. */
/**
* {@link #completeLease} to FAILED, scheduling the retry and recording why (V22) in the same
* statement.
*/
@Modifying(flushAutomatically = true)
@Query(
"update OutboxEventEntity e set e.status = 'FAILED', e.nextAttemptAt = :nextAttemptAt, "
+ "e.leaseOwner = null "
+ "e.lastFailureCode = :failureCode, e.leaseOwner = null "
+ "where e.eventId = :eventId and e.status = 'IN_FLIGHT' "
+ "and e.leaseOwner = :owner and e.leaseToken = :token")
int failLease(
@Param("eventId") String eventId,
@Param("owner") String owner,
@Param("token") long token,
@Param("nextAttemptAt") Instant nextAttemptAt);
@Param("nextAttemptAt") Instant nextAttemptAt,
@Param("failureCode") String failureCode);
/**
* {@link #failLease} for a claim that ended deferred: same transition, and the row counts the
@@ -82,14 +86,29 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
@Modifying(flushAutomatically = true)
@Query(
"update OutboxEventEntity e set e.status = 'FAILED', e.nextAttemptAt = :nextAttemptAt, "
+ "e.deferredAttemptCount = e.deferredAttemptCount + 1, e.leaseOwner = null "
+ "e.deferredAttemptCount = e.deferredAttemptCount + 1, "
+ "e.lastFailureCode = :failureCode, e.leaseOwner = null "
+ "where e.eventId = :eventId and e.status = 'IN_FLIGHT' "
+ "and e.leaseOwner = :owner and e.leaseToken = :token")
int deferLease(
@Param("eventId") String eventId,
@Param("owner") String owner,
@Param("token") long token,
@Param("nextAttemptAt") Instant nextAttemptAt);
@Param("nextAttemptAt") Instant nextAttemptAt,
@Param("failureCode") String failureCode);
/** {@link #completeLease} to DEAD, recording why (V22) in the same statement. */
@Modifying(flushAutomatically = true)
@Query(
"update OutboxEventEntity e set e.status = 'DEAD', e.lastFailureCode = :failureCode, "
+ "e.leaseOwner = null "
+ "where e.eventId = :eventId and e.status = 'IN_FLIGHT' "
+ "and e.leaseOwner = :owner and e.leaseToken = :token")
int deadLease(
@Param("eventId") String eventId,
@Param("owner") String owner,
@Param("token") long token,
@Param("failureCode") String failureCode);
/**
* Gives an unused claim back: PENDING, claimable from {@code claimableAt}, and the claim's
@@ -110,12 +129,18 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
/**
* Increments an aggregate's order counter (V21), taking its row lock until the caller commits.
* Returns 0 when the aggregate has no counter yet.
* Never below the aggregate's highest outbox row: an instance still on the wall-clock scheme may
* have written a larger number after the counter existed (rolling deploy, review RR-03-02).
* Returns 0 when the aggregate has no counter yet, or when another transaction's new counter is
* not committed yet.
*/
@Modifying(flushAutomatically = true)
@Query(
value =
"UPDATE outbox_aggregate_sequence SET last_sequence = last_sequence + 1"
"UPDATE outbox_aggregate_sequence SET last_sequence = GREATEST(last_sequence,"
+ " (SELECT COALESCE(MAX(e.aggregate_sequence), 0) FROM outbox_event e"
+ " WHERE e.tenant_scope = :tenantScope AND e.aggregate_type = :aggregateType"
+ " AND e.aggregate_id = :aggregateId)) + 1"
+ " WHERE tenant_scope = :tenantScope AND aggregate_type = :aggregateType"
+ " AND aggregate_id = :aggregateId",
nativeQuery = true)
@@ -125,8 +150,10 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
@Param("aggregateId") String aggregateId);
/**
* Creates an aggregate's counter, seeded after the highest order its outbox rows already carry.
* Two creators of the same counter collide on its primary key.
* Creates an aggregate's counter at its first number: one after the larger of its highest outbox
* row and the V21 floor (0 where no migration wrote one). Returns 0, and writes nothing, when
* another transaction created it first; {@code ON CONFLICT DO NOTHING} waits for that transaction
* to end rather than failing on the primary key (review RR-03-01).
*/
@Modifying(flushAutomatically = true)
@Query(
@@ -134,9 +161,12 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
"INSERT INTO outbox_aggregate_sequence"
+ " (tenant_scope, aggregate_type, aggregate_id, last_sequence)"
+ " SELECT :tenantScope, :aggregateType, :aggregateId,"
+ " COALESCE(MAX(e.aggregate_sequence), 0) + 1 FROM outbox_event e"
+ " GREATEST(COALESCE(MAX(e.aggregate_sequence), 0),"
+ " COALESCE((SELECT f.floor_sequence FROM outbox_aggregate_sequence_floor f), 0))"
+ " + 1 FROM outbox_event e"
+ " WHERE e.tenant_scope = :tenantScope AND e.aggregate_type = :aggregateType"
+ " AND e.aggregate_id = :aggregateId",
+ " AND e.aggregate_id = :aggregateId"
+ " ON CONFLICT DO NOTHING",
nativeQuery = true)
int createAggregateSequence(
@Param("tenantScope") String tenantScope,
@@ -174,4 +204,14 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
"select e.eventType, min(e.occurredAt) from OutboxEventEntity e "
+ "where e.status <> 'PUBLISHED' group by e.eventType")
List<Object[]> findOldestUnpublishedOccurredAtByEventType();
/**
* {@code [count (Long), oldestOccurredAt (Instant)]} of the FAILED rows with at least {@code
* minDeferredClaims} deferred claims, for the long-retry gauges. One row, whose instant is null
* when the count is 0.
*/
@Query(
"select count(e), min(e.occurredAt) from OutboxEventEntity e "
+ "where e.status = 'FAILED' and e.deferredAttemptCount >= :minDeferredClaims")
List<Object[]> countLongRetries(@Param("minDeferredClaims") int minDeferredClaims);
}
@@ -0,0 +1,32 @@
package dev.caskeleton.adapter.outbound.persistence.outbox.jpa.entity;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
/**
* The one-row floor a new aggregate counter starts after ({@code outbox_aggregate_sequence_floor},
* V21): the moment V21 ran, in epoch microseconds, above every number the replaced wall-clock
* scheme wrote. Read by {@code JpaOutboxStore#nextSequence} with a native statement; mapped, like
* {@link OutboxAggregateSequenceEntity}, so the table exists where Hibernate builds the schema (H2,
* where no migration writes the row and the floor is 0) and is validated where Flyway does.
* Package-private: no code reads it as an entity.
*/
@Entity
@Table(name = "outbox_aggregate_sequence_floor")
class OutboxAggregateSequenceFloorEntity {
@Id
@Column(name = "singleton", nullable = false)
private boolean singleton;
@Column(name = "floor_sequence", nullable = false)
private long floorSequence;
protected OutboxAggregateSequenceFloorEntity() {}
long floorSequence() {
return floorSequence;
}
}
@@ -58,6 +58,18 @@ public class OutboxEventEntity implements Persistable<String> {
private int deferredAttemptCount;
/** Dual-purpose per status (PENDING / IN_FLIGHT / FAILED) — see README "outbox". */
/**
* The code of the row's last failed completion (V22): the publisher's own name for the failure,
* or the outcome or exception type when it gave none.
*
* <p>Read-only to the entity: only the relay's failed completions write it, each in its one
* fenced UPDATE ({@code OutboxEventJpaRepository#failLease} and its siblings). The append's
* INSERT and the claim's flush leave it out, so they neither clear a code nor need the column on
* a database that has not run V22 yet.
*/
@Column(name = "last_failure_code", length = 120, insertable = false, updatable = false)
private String lastFailureCode;
@Column(name = "next_attempt_at", nullable = false)
private Instant nextAttemptAt;
@@ -379,4 +391,8 @@ public class OutboxEventEntity implements Persistable<String> {
public void setCausationId(String causationId) {
this.causationId = causationId;
}
public String getLastFailureCode() {
return lastFailureCode;
}
}
@@ -123,7 +123,9 @@ public final class SpringTransaction implements TransactionPort {
* dry and every other request fails at {@code connection-timeout}. With one, Spring hands the
* remaining time to every statement (Hibernate and {@code JdbcTemplate} both set the JDBC query
* timeout), so the server cancels the statement ({@code 57014}, {@code DB_QUERY_CANCELED}) and
* the connection comes back.
* the connection comes back. A statement Hibernate is asked to run after the deadline has passed
* is refused by Hibernate itself, with no SQLState; this class reports that as {@code
* DB_QUERY_CANCELED} too (review RR-03-03).
*
* <p>The default is 30 seconds for both: the budget {@code inNew} already has, and the Hikari
* {@code leak-detection-threshold} at which the pool already reports a held connection as a leak.
@@ -194,6 +196,9 @@ public final class SpringTransaction implements TransactionPort {
private <T> T execute(ModeTemplate mode, Supplier<T> action) {
long startedAt = System.nanoTime();
// Whether this call began the transaction, and so owns its deadline. A call that joined an
// outer one leaves the deadline to the call that began it.
AtomicBoolean beganTransaction = new AtomicBoolean();
// Set only once the action has returned in a write transaction this call began: from then on
// the only thing TransactionTemplate.execute still does is commit it.
AtomicBoolean committingOwnWrite = new AtomicBoolean();
@@ -201,20 +206,32 @@ public final class SpringTransaction implements TransactionPort {
return mode.template()
.execute(
status -> {
beganTransaction.set(status.isNewTransaction());
T result = action.get();
committingOwnWrite.set(status.isNewTransaction() && !mode.template().isReadOnly());
return result;
});
} catch (RuntimeException failure) {
Duration elapsed = Duration.ofNanos(System.nanoTime() - startedAt);
Optional<PersistenceFailureException> translated =
committingOwnWrite.get()
? exceptionTranslator.translateCommitFailure(
failure, mode.commitOperation(), Duration.ofNanos(System.nanoTime() - startedAt))
? exceptionTranslator.translateCommitFailure(failure, mode.commitOperation(), elapsed)
: exceptionTranslator.translate(failure);
if (translated.isEmpty() && beganTransaction.get() && pastDeadline(mode, elapsed)) {
// Hibernate's own deadline refusal carries no SQLState (review RR-03-03).
translated = exceptionTranslator.translateAfterDeadline(failure);
}
throw translated.map(RuntimeException.class::cast).orElse(failure);
}
}
/** Whether {@code elapsed} reached the timeout the mode's transactions run under. */
private static boolean pastDeadline(ModeTemplate mode, Duration elapsed) {
int timeoutSeconds = mode.template().getTimeout();
return timeoutSeconds != TransactionDefinition.TIMEOUT_DEFAULT
&& elapsed.compareTo(Duration.ofSeconds(timeoutSeconds)) >= 0;
}
/**
* The vendor-neutral default translator.
*
@@ -7,8 +7,32 @@
--
-- outbox_aggregate_sequence holds one counter per aggregate. Incrementing it locks the row until
-- the writer commits, so the numbers follow commit order; it outlives the reaper's deletes, so a
-- number is never reused. JpaOutboxAggregateSequences seeds a new counter from the aggregate's
-- highest existing aggregate_sequence, so rows numbered by the old clock scheme still come first.
-- number is never reused. JpaOutboxStore#nextSequence reads and writes it.
--
-- Consumers have already seen clock numbers (about 1.7e15) for aggregates whose rows this
-- database no longer holds: the reaper deleted them, or they were written under an identity that
-- has since changed (the sample's tenant scope). A counter that started after the rows still
-- present would start at 1 for those aggregates, behind numbers already sent (review RR-03-02,
-- RR-05-02). So this migration records one floor, the moment it runs in epoch microseconds, which
-- is above every clock number written before it; a counter created later starts after the floor
-- or after the aggregate's highest remaining row, whichever is larger. The aggregates that still
-- have rows get their counter here, at their highest number. A clock number an instance still on
-- the old version writes during the rollout is above the floor; nextSequence also never goes below
-- the highest row it can see, so such a row still comes first.
--
-- This file was edited after it was first written, before any deployment applied it (the branch
-- that added it was not merged): splitting the floor into a V22 would have left V21 alone handing
-- out numbers from 1 for one release, and its comment pointed at a class that does not exist.
-- applied-migrations.sha256 carries the new checksum. A development database that applied the
-- earlier text fails Flyway validation; recreate it.
-- Bound the wait for the table lock the unique index below takes (SHARE on outbox_event, held until
-- this migration commits). Without a bound, the migration queues behind any open outbox write and
-- every outbox write after it queues behind the migration. Five seconds, then this migration fails,
-- rolls back whole and the database stays at V20: deploy again, or build the index first
-- (runbook §8, CONCURRENTLY).
SET LOCAL lock_timeout = '5s';
CREATE TABLE IF NOT EXISTS outbox_aggregate_sequence (
tenant_scope VARCHAR(96) NOT NULL,
aggregate_type VARCHAR(64) NOT NULL,
@@ -18,10 +42,55 @@ CREATE TABLE IF NOT EXISTS outbox_aggregate_sequence (
CONSTRAINT ck_outbox_aggregate_sequence_positive CHECK (last_sequence >= 1)
);
-- One row, written once: the lowest number a counter created from now on may start after.
CREATE TABLE IF NOT EXISTS outbox_aggregate_sequence_floor (
singleton BOOLEAN NOT NULL DEFAULT TRUE,
floor_sequence BIGINT NOT NULL,
CONSTRAINT pk_outbox_aggregate_sequence_floor PRIMARY KEY (singleton),
CONSTRAINT ck_outbox_aggregate_sequence_floor_singleton CHECK (singleton),
CONSTRAINT ck_outbox_aggregate_sequence_floor_non_negative CHECK (floor_sequence >= 0)
);
INSERT INTO outbox_aggregate_sequence_floor (singleton, floor_sequence)
VALUES (TRUE, (extract(epoch FROM clock_timestamp()) * 1000000)::BIGINT)
ON CONFLICT (singleton) DO NOTHING;
INSERT INTO outbox_aggregate_sequence (tenant_scope, aggregate_type, aggregate_id, last_sequence)
SELECT tenant_scope, aggregate_type, aggregate_id, MAX(aggregate_sequence)
FROM outbox_event
WHERE tenant_scope IS NOT NULL
AND aggregate_type IS NOT NULL
AND aggregate_sequence IS NOT NULL
GROUP BY tenant_scope, aggregate_type, aggregate_id
ON CONFLICT (tenant_scope, aggregate_type, aggregate_id) DO NOTHING;
-- One number, one event: a second event appended under an order its aggregate already used is
-- refused at append time, in the writer's transaction, rather than released by the gate in no
-- defined order. Rows without canonical order columns (pre-canonical, all NULL) are not compared.
-- This fails if duplicates already exist; resolve them (they are exactly the rows the gate could
-- not order) before migrating.
-- not order) before migrating (runbook §8).
--
-- Built here, inside this migration's transaction, not CONCURRENTLY: a duplicate then rolls the
-- whole of V21 back and leaves the database at V20 with nothing to clean up, which is what the
-- runbook's retry relies on. A failed CONCURRENTLY build leaves an INVALID index that IF NOT EXISTS
-- would then skip, so the uniqueness would silently not hold; and CONCURRENTLY cannot run in the
-- transaction that writes the counters and the floor. The reaper bounds outbox_event to the
-- retention window, so the build is normally short. An outbox too large to lock for that long is
-- indexed first with the runbook's CONCURRENTLY statement (same name, same columns); the check
-- below refuses an index such a build left invalid, and IF NOT EXISTS keeps the valid one.
DO $$
BEGIN
IF EXISTS (SELECT 1
FROM pg_index
WHERE indexrelid = to_regclass('ux_outbox_event_aggregate_order')
AND NOT (indisvalid AND indisunique)) THEN
RAISE EXCEPTION 'ux_outbox_event_aggregate_order exists but is not a valid unique index'
USING HINT = 'A CREATE INDEX CONCURRENTLY failed or is still running. '
|| 'DROP INDEX CONCURRENTLY ux_outbox_event_aggregate_order, then rebuild it '
|| 'or let V21 build it (runbook outbox-dead-letter §8).';
END IF;
END
$$;
CREATE UNIQUE INDEX IF NOT EXISTS ux_outbox_event_aggregate_order
ON outbox_event (tenant_scope, aggregate_type, aggregate_id, aggregate_sequence, event_index);
@@ -0,0 +1,21 @@
-- Why a row keeps failing, kept on the row.
--
-- A publish refused for now (a closed transport, a credential refused) or left unanswered is
-- retried without spending the attempt budget (V20), so it has no end: the relay retries it five
-- minutes apart for as long as nobody looks. The outcome the relay records cannot say why — a
-- shutdown, a closed transport and a refused credential are one outcome — and the relay's log line
-- said it in one sentence for all four. The publisher's failure code can (a messaging platform code
-- such as KAFKA_AUTHENTICATIONEXCEPTION); every failed completion writes it here, so the runbook's
-- query for rows that have been retrying long shows what each one is waiting on (re-review
-- 2026-11-03 RR-05-03).
--
-- A code, never a message: a driver message can carry an endpoint, a statement or a credential.
-- Nullable with no default, so adding it rewrites no row; rows written before it stay NULL until
-- their next failure.
--
-- The new status PARKED needs no DDL: status has no CHECK, the claim selects PENDING, FAILED and
-- IN_FLIGHT only (so the partial index ix_outbox_event_eligible leaves PARKED out), and the order
-- gate and its index ix_outbox_event_unpublished_aggregate_order treat every status but PUBLISHED as
-- blocking — which is what holding a row in its aggregate's place means.
ALTER TABLE outbox_event
ADD COLUMN last_failure_code VARCHAR(120);
@@ -23,7 +23,8 @@ import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
/**
* What a worker's death leaves behind, at each phase of a dispatch.
* What a worker's death leaves behind, at each phase of a dispatch — the rows, not what recovery
* does with them.
*
* <p>This is the {@code recoverable} claim in {@code docs/notification/evidence-manifest.json}, and
* until now nothing backed it. Lease recovery and event replay had unit coverage against fakes; the
@@ -36,9 +37,13 @@ import org.junit.jupiter.api.Test;
* worker is from the database's point of view — a lease nobody will renew and a phase nobody will
* advance.
*
* <p>The matrix walks the phases a dispatch passes through. For each, the question is the same and
* has only two acceptable answers: the delivery becomes claimable again, or it becomes a question
* for the provider. Never both, and never neither.
* <p>The matrix walks the phases a dispatch passes through and pins, for each, the state recovery
* starts from: a lease that has passed (the claim takes the row again) and, from the attempt insert
* on, an unfinished attempt row carrying the evidence of that phase (what reconciliation asks the
* provider about). These rows are written by this test, so the tests check the shape of the input
* and the predicates recovery reads, not recovery itself. {@code
* PostgreSqlNotificationBackgroundPassIntegrationTest} runs recovery and reconciliation over rows
* the dispatcher's own factory wrote (review 05-01, RR-03-07).
*/
@Tag("jpa-contract")
class WorkerCrashRecoveryContractTest {
@@ -73,8 +78,8 @@ class WorkerCrashRecoveryContractTest {
}
@Test
@DisplayName("a crash after claiming and before any attempt leaves the delivery claimable")
void aCrashBeforeAnyAttemptLeavesItClaimable() throws SQLException {
@DisplayName("claim 뒤 attempt 전에 죽은 delivery는 attempt 행이 없고 lease가 지나 있다")
void aDeliveryDeadBeforeAnyAttemptHasNoAttemptRowAndAPassedLease() throws SQLException {
UUID delivery = deliveryHeldByADeadWorker();
assertThat(isClaimableAgain(delivery))
@@ -88,12 +93,12 @@ class WorkerCrashRecoveryContractTest {
/**
* The attempt row is written before the call and keeps the factory's "unknown" evidence for the
* whole call (review 05-01), so a crash anywhere between the insert and the provider's answer
* leaves the same row. Recovery cannot tell "never called" from "called and died", so the row
* goes to reconciliation rather than back to the queue.
* leaves the same row. This test writes that row itself; that recovery hands it to reconciliation
* is {@code PostgreSqlNotificationBackgroundPassIntegrationTest}'s.
*/
@Test
@DisplayName("attempt 행을 쓴 뒤 호출이 끝나기 전 죽으면 시작 여부 불명인 미완료 attempt 가 남아 reconcile 대상이 된다")
void aCrashAfterTheAttemptRowIsWrittenLeavesAnUnfinishedAttemptOfUnknownStart()
@DisplayName("attempt 행을 쓰고 호출 전에 죽은 delivery는 시작 여부 불명인 미완료 attempt와 지난 lease를 남긴다")
void aDeliveryDeadAfterTheAttemptInsertLeavesAnUnfinishedAttemptOfUnknownStart()
throws SQLException {
// given
UUID delivery = deliveryHeldByADeadWorker();
@@ -103,17 +108,18 @@ class WorkerCrashRecoveryContractTest {
// then
assertThat(evidenceOf(attempt))
.as("the row the factory writes records that whether the call started is not known")
.as("the row stands for the factory's: whether the call started is not known")
.containsExactly("false", "UNKNOWN", "false", "UNKNOWN");
assertThat(isUnfinished(attempt))
.as("an unfinished attempt is what recovery hands to reconciliation")
.as("completed_at is empty, the predicate recovery selects unfinished attempts by")
.isTrue();
assertThat(isClaimableAgain(delivery)).isTrue();
}
@Test
@DisplayName("a crash after the request started is a question for the provider, not a requeue")
void aCrashAfterTheRequestStartedIsAQuestion() throws SQLException {
@DisplayName("요청 시작 뒤 죽은 delivery는 시작 증거가 있는 미완료 attempt와 지난 lease를 남긴다")
void aDeliveryDeadAfterTheRequestStartedLeavesAnUnfinishedAttemptWithStartEvidence()
throws SQLException {
UUID delivery = deliveryHeldByADeadWorker();
UUID attempt = attemptOn(delivery, true, "PROVEN", false, "UNKNOWN");
@@ -125,13 +131,14 @@ class WorkerCrashRecoveryContractTest {
"the lease still expires — the row must not be stranded — but the attempt is unfinished")
.isTrue();
assertThat(isUnfinished(attempt))
.as("an unfinished attempt is what recovery hands to reconciliation")
.as("completed_at is empty, the predicate recovery selects unfinished attempts by")
.isTrue();
}
@Test
@DisplayName("a crash after the body was committed is never silently retried")
void aCrashAfterTheBodyWasCommittedIsNeverSilentlyRetried() throws SQLException {
@DisplayName("본문 전송 뒤 죽은 delivery는 전송 증거가 있는 미완료 attempt를 남긴다")
void aDeliveryDeadAfterTheBodyWasSentLeavesAnUnfinishedAttemptWithSendEvidence()
throws SQLException {
UUID delivery = deliveryHeldByADeadWorker();
UUID attempt = attemptOn(delivery, true, "PROVEN", true, "PROVEN");
@@ -142,8 +149,8 @@ class WorkerCrashRecoveryContractTest {
}
@Test
@DisplayName("a completed attempt is not recovered at all")
void aCompletedAttemptIsNotRecovered() throws SQLException {
@DisplayName("완료 시각이 적힌 attempt는 미완료로 보이지 않는다")
void aCompletedAttemptIsNotUnfinished() throws SQLException {
UUID delivery = deliveryHeldByADeadWorker();
UUID attempt = attemptOn(delivery, true, "PROVEN", true, "PROVEN");
complete(attempt);
@@ -154,8 +161,9 @@ class WorkerCrashRecoveryContractTest {
}
@Test
@DisplayName("across every phase, exactly one delivery row survives with one attempt each")
void acrossEveryPhaseNothingIsLostOrDuplicated() throws SQLException {
@DisplayName("네 단계에서 죽은 워커들은 delivery 4개와 attempt 3개를 남기고 모든 lease가 지나 있다")
void deadWorkersAcrossEveryPhaseLeaveFourDeliveriesThreeAttemptsAndPassedLeases()
throws SQLException {
List<UUID> deliveries = new ArrayList<>();
// The four crash phases, run together, so the recovery sweep sees a mixed table rather than one
// hand-picked row.
@@ -59,7 +59,14 @@ import org.springframework.transaction.PlatformTransactionManager;
class DurableOperationSubmissionOutboxIntegrationTest {
private static final Instant NOW = Instant.parse("2026-08-25T09:00:00Z");
private static final DurableOperationId ID = new DurableOperationId("op-1");
/**
* A canonical lower-case UUIDv7, as the HTTP adapter mints: the operation id is then the event id
* and the aggregate id itself (an id of any other spelling becomes a UUIDv7 derived from it).
*/
private static final String OPERATION_UUID = "01994e11-4d88-7000-8000-0000000000a1";
private static final DurableOperationId ID = new DurableOperationId(OPERATION_UUID);
private static JpaPlatformContractSupport support;
private static AnnotationConfigApplicationContext context;
@@ -103,7 +110,7 @@ class DurableOperationSubmissionOutboxIntegrationTest {
// then
assertThat(queryStrings("SELECT contract_id || '|' || aggregate_id FROM outbox_event"))
.containsExactly("durable-operation.submitted|op-1");
.containsExactly("durable-operation.submitted|" + OPERATION_UUID);
}
@Test
@@ -0,0 +1,159 @@
package dev.caskeleton.adapter.outbound.persistence.readiness;
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaFileMetadata;
import dev.caskeleton.application.fileserver.api.FileId;
import dev.caskeleton.application.fileserver.api.StorageNamespace;
import dev.caskeleton.application.fileserver.api.metadata.FileOwner;
import dev.caskeleton.application.fileserver.api.metadata.FileRecordDraft;
import java.time.Clock;
import java.time.Duration;
import java.time.Instant;
import java.time.ZoneId;
import java.time.ZoneOffset;
import java.time.temporal.ChronoUnit;
import java.util.Optional;
import java.util.OptionalLong;
import java.util.UUID;
import java.util.function.Supplier;
import org.flywaydb.core.Flyway;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.transaction.support.TransactionTemplate;
/**
* One real PostgreSQL with the Fileserver schema and stores, for the port contract tests that hold
* the JPA stores to the cases their in-memory fakes pass (review 2026-11-03 RR-01-04).
*
* <p>The stores read a clock this stack owns and moves, so a contract case about expiry passes time
* the way it does against a fake instead of sleeping. One server per test class; {@link #empty()}
* makes each case independent of the one before it.
*/
final class FileserverPostgreSqlStack implements AutoCloseable {
private final PostgreSqlReadinessSupport postgres;
private final MovableClock clock;
private final FileserverJpaTestContext context;
private final TransactionTemplate transactions;
private final JdbcTemplate jdbc;
private FileserverPostgreSqlStack(PostgreSqlReadinessSupport postgres) {
this.postgres = postgres;
migrate("classpath:db/migration/postgresql", "flyway_schema_history", false, null);
migrate(
"classpath:db/migration/jpa/core",
"flyway_jpa_core_history",
true,
"explicit-jpa-core-adoption");
migrate(
"classpath:db/migration/jpa/fileserver",
"flyway_jpa_fileserver_history",
true,
"explicit-jpa-fileserver-adoption");
this.clock = new MovableClock(Instant.now().truncatedTo(ChronoUnit.MICROS));
this.context = new FileserverJpaTestContext(postgres.dataSource(), clock);
this.transactions = context.transactions();
this.jdbc = new JdbcTemplate(postgres.dataSource());
}
/** Starts a server, applies the three schema streams and builds the Fileserver stores. */
static FileserverPostgreSqlStack start() {
PostgreSqlReadinessSupport.assertDockerAvailable();
return new FileserverPostgreSqlStack(PostgreSqlReadinessSupport.start(8, 2_000));
}
<T> T bean(Class<T> type) {
return context.bean(type);
}
/** Runs {@code work} in one committed transaction, as the application's transaction port does. */
<T> T inTransaction(Supplier<T> work) {
return transactions.execute(status -> work.get());
}
/** The instant the stores read as now. */
Instant now() {
return clock.instant();
}
/** Moves the stores' clock forward. */
void elapse(Duration duration) {
clock.advance(duration);
}
/** Records a file in {@code namespace}, so rows that reference a file may name it. */
FileId recordFileIn(StorageNamespace namespace) {
FileId fileId = FileId.of(UUID.randomUUID());
inTransaction(
() ->
bean(JpaFileMetadata.class)
.insert(
new FileRecordDraft(
fileId,
namespace,
"report.bin",
Optional.of("application/octet-stream"),
OptionalLong.of(1_000),
new FileOwner("user-1", Optional.empty()))));
return fileId;
}
/** Deletes every Fileserver row, children first. */
void empty() {
jdbc.execute(
"TRUNCATE fs_recovery_item, fs_cleanup_item, fs_quota_reservation,"
+ " fs_verification_result, fs_upload_session, fs_file CASCADE");
}
@Override
public void close() {
context.close();
postgres.close();
}
private void migrate(
String location, String historyTable, boolean independent, String baselineDescription) {
var configuration =
Flyway.configure()
.dataSource(postgres.dataSource())
.locations(location)
.table(historyTable)
.baselineOnMigrate(false)
.outOfOrder(false);
if (independent) {
configuration.baselineVersion("0").baselineDescription(baselineDescription);
}
Flyway flyway = configuration.load();
if (independent) {
flyway.baseline();
}
flyway.migrate();
}
/** A clock a test moves forward; microsecond precision, as PostgreSQL stores timestamps. */
private static final class MovableClock extends Clock {
private Instant now;
private MovableClock(Instant start) {
this.now = start;
}
private synchronized void advance(Duration duration) {
now = now.plus(duration);
}
@Override
public synchronized Instant instant() {
return now;
}
@Override
public ZoneId getZone() {
return ZoneOffset.UTC;
}
@Override
public Clock withZone(ZoneId zone) {
throw new UnsupportedOperationException("the stack's clock is UTC");
}
}
}
@@ -0,0 +1,80 @@
package dev.caskeleton.adapter.outbound.persistence.readiness;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Map;
/**
* The SQL an outbox runbook gives an operator, read out of the runbook itself.
*
* <p>A runbook block is marked {@code <!-- runbook-sql: id -->} and followed by a {@code sql}
* fence. The lanes that run these blocks against real PostgreSQL read them from the document rather
* than keep a copy, so editing a block in the runbook is what the lane tests: a copy would pass
* while the document an operator follows is wrong.
*/
final class OutboxRunbookSql {
private final String runbook;
private OutboxRunbookSql(String runbook) {
this.runbook = runbook;
}
/**
* Reads the runbook the lane was given in the system property {@code property}.
*
* @param property the system property holding the runbook's absolute path
* @return the runbook's SQL blocks
* @throws IOException when the file cannot be read
*/
static OutboxRunbookSql fromSystemProperty(String property) throws IOException {
String path = System.getProperty(property);
if (path == null || path.isBlank()) {
throw new IllegalStateException(
"system property " + property + " must name the runbook this lane runs");
}
return new OutboxRunbookSql(Files.readString(Path.of(path), StandardCharsets.UTF_8));
}
/**
* The SQL block that follows {@code <!-- runbook-sql: id -->}.
*
* @param id the block's marker id
* @return the SQL text between the fences
*/
String block(String id) {
String marker = "<!-- runbook-sql: " + id + " -->";
int at = runbook.indexOf(marker);
if (at < 0) {
throw new IllegalStateException("the outbox runbook has no SQL block marked " + id);
}
int open = runbook.indexOf("```sql\n", at);
int close = runbook.indexOf("\n```", open + 1);
if (open < 0 || close < 0) {
throw new IllegalStateException("the block marked " + id + " is not a closed sql fence");
}
return runbook.substring(open + "```sql\n".length(), close);
}
/**
* The block with its {@code <placeholder>}s filled in, as an operator would before running it.
*
* @param id the block's marker id
* @param values each placeholder as written in the runbook, without the angle brackets, and its
* value
* @return the runnable SQL
*/
String block(String id, Map<String, String> values) {
String sql = block(id);
for (Map.Entry<String, String> value : values.entrySet()) {
String placeholder = "<" + value.getKey() + ">";
if (!sql.contains(placeholder)) {
throw new IllegalStateException("the block marked " + id + " has no " + placeholder);
}
sql = sql.replace(placeholder, value.getValue());
}
return sql;
}
}
@@ -0,0 +1,54 @@
package dev.caskeleton.adapter.outbound.persistence.readiness;
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaContentReferenceLedger;
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaFileMetadata;
import dev.caskeleton.application.fileserver.admin.ContentReferenceLedgerPort;
import dev.caskeleton.application.fileserver.api.metadata.FileMetadataPort;
import dev.caskeleton.application.fileserver.testkit.ContentReferenceLedgerPortContract;
import java.util.function.Supplier;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Tag;
/**
* {@link JpaContentReferenceLedger} held to the contract the in-memory metadata store passes, on
* real PostgreSQL (review 2026-11-03 RR-01-04).
*/
@Tag("jpa-contract")
class PostgreSqlContentReferenceLedgerContractTest extends ContentReferenceLedgerPortContract {
private static FileserverPostgreSqlStack stack;
@BeforeAll
static void start() {
stack = FileserverPostgreSqlStack.start();
}
@AfterAll
static void stop() {
if (stack != null) {
stack.close();
}
}
@BeforeEach
void empty() {
stack.empty();
}
@Override
protected ContentReferenceLedgerPort ledger() {
return stack.bean(JpaContentReferenceLedger.class);
}
@Override
protected FileMetadataPort metadata() {
return stack.bean(JpaFileMetadata.class);
}
@Override
protected <T> T inTransaction(Supplier<T> work) {
return stack.inTransaction(work);
}
}
@@ -0,0 +1,92 @@
package dev.caskeleton.adapter.outbound.persistence.readiness;
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaFileQuota;
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaQuotaCommit;
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaQuotaReclaim;
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaUploadSession;
import dev.caskeleton.application.fileserver.api.FileId;
import dev.caskeleton.application.fileserver.api.StorageNamespace;
import dev.caskeleton.application.fileserver.api.UploadId;
import dev.caskeleton.application.fileserver.api.metadata.FileQuotaPort;
import dev.caskeleton.application.fileserver.api.metadata.UploadSession;
import dev.caskeleton.application.fileserver.api.metadata.UploadSessionDraft;
import dev.caskeleton.application.fileserver.api.transfer.UploadMode;
import dev.caskeleton.application.fileserver.cleanup.QuotaReclaimPort;
import dev.caskeleton.application.fileserver.testkit.FileQuotaLedgerContract;
import dev.caskeleton.application.fileserver.upload.QuotaCommitPort;
import java.time.Duration;
import java.util.OptionalLong;
import java.util.UUID;
import java.util.function.Supplier;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Tag;
/**
* {@link JpaFileQuota}, {@link JpaQuotaCommit} and {@link JpaQuotaReclaim} held to the ledger
* contract the in-memory ledger passes, on real PostgreSQL (review 2026-11-03 RR-01-04).
*/
@Tag("jpa-contract")
class PostgreSqlFileQuotaLedgerContractTest extends FileQuotaLedgerContract {
private static FileserverPostgreSqlStack stack;
@BeforeAll
static void start() {
stack = FileserverPostgreSqlStack.start();
}
@AfterAll
static void stop() {
if (stack != null) {
stack.close();
}
}
@BeforeEach
void empty() {
stack.empty();
}
@Override
protected FileQuotaPort quota() {
return stack.bean(JpaFileQuota.class);
}
@Override
protected QuotaCommitPort commits() {
return stack.bean(JpaQuotaCommit.class);
}
@Override
protected QuotaReclaimPort reclaims() {
return stack.bean(JpaQuotaReclaim.class);
}
@Override
protected UploadSession uploadIn(StorageNamespace namespace) {
FileId fileId = stack.recordFileIn(namespace);
return stack.inTransaction(
() ->
stack
.bean(JpaUploadSession.class)
.create(
new UploadSessionDraft(
UploadId.of(UUID.randomUUID()),
fileId,
UploadMode.SINGLE_SHOT,
OptionalLong.of(1_000),
stack.now().plus(Duration.ofHours(1)))));
}
@Override
protected void elapse(Duration duration) {
stack.elapse(duration);
}
@Override
protected <T> T inTransaction(Supplier<T> work) {
return stack.inTransaction(work);
}
}
@@ -50,7 +50,7 @@ class PostgreSqlMigrationIntegrationTest {
assertThat(appliedVersions(postgres, "flyway_schema_history"))
.containsExactly(
"1", "3", "4", "5", "6", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18",
"19", "20", "21");
"19", "20", "21", "22");
assertThat(
singleValue(
postgres,
@@ -19,11 +19,18 @@ import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.jdbc.core.JdbcTemplate;
/**
* The aggregate order counter and the order uniqueness rule against real PostgreSQL (V21, review
* 06-07): the shared {@link OutboxAggregateSequenceContract}, the seed from rows numbered before
* the counter existed, and a second append under a used order refused at append time.
* 06-07): the shared {@link OutboxAggregateSequenceContract}, where a counter starts when the
* aggregate already has history, and a second append under a used order refused at append time.
*
* <p>"History" is what the old wall-clock scheme left (epoch microseconds, about 1.7e15): rows that
* are still in the outbox, rows the reaper deleted, rows under an identity that has since changed,
* and rows an instance still on the old version writes during a rolling deploy. A consumer that
* reads the number as "which change came last" must never see a later change with a smaller number
* (review RR-03-02, RR-05-02).
*/
class PostgreSqlOutboxAggregateSequenceIntegrationTest extends OutboxAggregateSequenceContract {
@@ -32,6 +39,7 @@ class PostgreSqlOutboxAggregateSequenceIntegrationTest extends OutboxAggregateSe
private static PostgreSqlReadinessSupport postgres;
private static AnnotationConfigApplicationContext context;
private static OutboxAggregateSequencePort sequences;
private static JdbcTemplate jdbc;
@BeforeAll
static void setup() {
@@ -40,6 +48,7 @@ class PostgreSqlOutboxAggregateSequenceIntegrationTest extends OutboxAggregateSe
OutboxIntegrationSupport.migrate(dataSource);
context = OutboxIntegrationSupport.buildContext(dataSource, e -> {}, Clock.systemUTC());
sequences = context.getBean(JpaOutboxStore.class);
jdbc = new JdbcTemplate(dataSource);
}
@AfterAll
@@ -63,25 +72,71 @@ class PostgreSqlOutboxAggregateSequenceIntegrationTest extends OutboxAggregateSe
}
@Test
@DisplayName("카운터가 없던 aggregate의 첫 번호는 이미 있는 outbox 행의 가장 큰 번호 다음이다")
@DisplayName("카운터가 없던 aggregate의 첫 번호는 floor보다 큰 남은 행의 가장 큰 번호 다음이다")
void aNewCounterStartsAfterTheHighestOrderTheAggregatesRowsAlreadyCarry() {
// given: a row numbered by the old wall-clock scheme
// given: a row an instance still on the clock scheme wrote five seconds after V21 ran
String aggregateId = "clock-numbered-" + UUID.randomUUID();
long clockNumber = 1_761_955_200_000_000L;
long clockNumber = floor() + 5_000_000L;
append(aggregateId, clockNumber);
// when
long next =
transactions()
.inWrite(
() ->
sequences.nextSequence(
new AggregateIdentity("default", "testkit", aggregateId)));
long next = next(aggregateId);
// then
assertThat(next).isEqualTo(clockNumber + 1);
}
@Test
@DisplayName("옛 행이 reaper로 지워진 aggregate의 첫 번호는 지워진 번호가 아니라 floor 다음이다")
void anAggregateWhoseRowsWereReapedStartsAboveTheFloor() {
// given: a week-old clock-numbered row, published and deleted by the reaper since
String aggregateId = "reaped-" + UUID.randomUUID();
long clockNumber = 1_761_955_200_000_000L;
append(aggregateId, clockNumber);
jdbc.update("DELETE FROM outbox_event WHERE aggregate_id = ?", aggregateId);
// when
long next = next(aggregateId);
// then
assertThat(next).as("a number the consumer has already seen").isGreaterThan(clockNumber);
assertThat(next).isEqualTo(floor() + 1);
}
@Test
@DisplayName("tenant scope가 바뀐 aggregate의 첫 번호는 옛 scope 행의 번호보다 큰 floor 다음이다")
void anAggregateWhoseIdentityChangedStartsAboveTheFloor() {
// given: the sample's rows were written under scope "portfolio" before it moved to one scope
String aggregateId = "rescoped-" + UUID.randomUUID();
long clockNumber = 1_761_955_200_000_000L;
append(aggregateId, clockNumber);
jdbc.update(
"UPDATE outbox_event SET tenant_scope = 'portfolio' WHERE aggregate_id = ?", aggregateId);
// when
long next = next(aggregateId);
// then
assertThat(next).as("a number the consumer has already seen").isGreaterThan(clockNumber);
assertThat(next).isEqualTo(floor() + 1);
}
@Test
@DisplayName("카운터가 생긴 뒤 옛 버전 인스턴스가 더 큰 번호로 쓴 행이 있으면 다음 번호는 그 행 다음이다")
void aRowAnOldInstanceWroteAfterTheCounterExistedStillComesFirst() {
// given: a new instance numbered the aggregate, then an old one wrote its clock number
String aggregateId = "rolling-" + UUID.randomUUID();
long counted = next(aggregateId);
long oldInstancesNumber = counted + 10_000_000L;
append(aggregateId, oldInstancesNumber);
// when
long next = next(aggregateId);
// then
assertThat(next).isEqualTo(oldInstancesNumber + 1);
}
@Test
@DisplayName("같은 aggregate에 같은 순서 번호로 두 번 append하면 두 번째는 실패한다")
void aSecondEventUnderAnOrderItsAggregateAlreadyUsedIsRefused() {
@@ -96,6 +151,19 @@ class PostgreSqlOutboxAggregateSequenceIntegrationTest extends OutboxAggregateSe
assertThat(thrown).isNotNull();
}
private static long next(String aggregateId) {
return context
.getBean(SpringTransaction.class)
.inWrite(
() -> sequences.nextSequence(new AggregateIdentity("default", "testkit", aggregateId)));
}
/** The number V21 recorded once: the moment it ran, in epoch microseconds. */
private static long floor() {
return jdbc.queryForObject(
"SELECT floor_sequence FROM outbox_aggregate_sequence_floor", Long.class);
}
private static void append(String aggregateId, long sequence) {
JpaOutboxStore store = context.getBean(JpaOutboxStore.class);
context
@@ -179,6 +179,13 @@ class PostgreSqlOutboxConcurrentRelayIntegrationTest extends OutboxLeaseFencingC
"select status from outbox_event where event_id = ?", String.class, eventId));
}
@Override
protected String lastFailureCodeOf(String eventId) {
return new JdbcTemplate(sharedDataSource)
.queryForObject(
"select last_failure_code from outbox_event where event_id = ?", String.class, eventId);
}
/** Runs the relay in a loop until claimedCount == 0 (store drained). */
private int drainRelay(PublishPendingOutboxEventsUseCase relay) {
int total = 0;
@@ -0,0 +1,279 @@
package dev.caskeleton.adapter.outbound.persistence.readiness;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.tuple;
import com.zaxxer.hikari.HikariDataSource;
import dev.caskeleton.adapter.outbound.persistence.outbox.jpa.JpaOutboxStore;
import dev.caskeleton.adapter.outbound.persistence.transaction.SpringTransaction;
import dev.caskeleton.application.outbox.OutboxLease;
import dev.caskeleton.application.outbox.testkit.OutboxEvents;
import java.io.IOException;
import java.sql.SQLException;
import java.time.Clock;
import java.time.Duration;
import java.time.Instant;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.UUID;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.jdbc.core.JdbcTemplate;
/**
* The outbox runbooks' manual dispositions, run as written against real PostgreSQL.
*
* <p>An operator resolves a DEAD row by re-driving or skipping it ({@code
* docs/runbooks/outbox-dead-letter.md} §4), and a row that keeps failing for ever by finding it,
* parking it, and resuming or skipping it ({@code docs/runbooks/outbox-publish-failed.md}, "끝나지 않는
* 재시도"). The SQL for each is read out of the runbook, not copied, and its result is checked the
* only way that matters: whether the relay's next claim takes the row. A re-drive that leaves a
* counter behind is accepted by the database and then refused by the claim, which moves the row
* back to DEAD — the failure the first re-drive test pins (re-review 2026-11-03 RR-05-01).
*/
class PostgreSqlOutboxDispositionRunbookIntegrationTest {
private static final Instant OCCURRED = Instant.parse("2026-11-01T00:00:00Z");
private static final Duration IN_FLIGHT_TIMEOUT = Duration.ofMinutes(5);
private static PostgreSqlReadinessSupport postgres;
private static OutboxRunbookSql deadLetterRunbook;
private static OutboxRunbookSql publishFailedRunbook;
@BeforeAll
static void setup() throws IOException {
deadLetterRunbook = OutboxRunbookSql.fromSystemProperty("outbox.runbook");
publishFailedRunbook = OutboxRunbookSql.fromSystemProperty("outbox.publish-failed.runbook");
postgres = PostgreSqlReadinessSupport.start();
}
@AfterAll
static void cleanup() {
if (postgres != null) {
postgres.close();
}
}
@Test
@DisplayName("미뤄진 시도가 쌓인 DEAD 행을 runbook 재발행 SQL로 되돌리면 다음 claim이 그 행을 가져간다")
void theRedriveMakesADeadRowWithDeferredClaimsClaimableAgain() throws SQLException {
try (Outbox outbox = Outbox.create()) {
// given: dead-lettered after two claims an outage deferred and three that spent the budget
String dead = outbox.append("order-" + UUID.randomUUID(), 1, OCCURRED);
outbox.setCounters(dead, "DEAD", 5, 2);
String redrive = deadLetterRunbook.block("dead-redrive-one", Map.of("id", dead));
// when
outbox.jdbc().execute(redrive);
// then
assertThat(outbox.claimedEventIds()).containsExactly(dead);
}
}
@Test
@DisplayName("runbook 구간 재발행 SQL은 그 구간의 DEAD 행만 claim 가능하게 되돌린다")
void theWindowRedriveReturnsOnlyTheDeadRowsOfTheWindow() throws SQLException {
try (Outbox outbox = Outbox.create()) {
// given
String early = outbox.append("order-" + UUID.randomUUID(), 1, OCCURRED);
String late = outbox.append("order-" + UUID.randomUUID(), 1, OCCURRED.plusSeconds(60));
String outside = outbox.append("order-" + UUID.randomUUID(), 1, OCCURRED.plusSeconds(3600));
for (String eventId : List.of(early, late, outside)) {
outbox.setCounters(eventId, "DEAD", 3, 2);
}
String redrive =
deadLetterRunbook.block(
"dead-redrive-window",
Map.of(
"장애 시작", OCCURRED.minusSeconds(1).toString(),
"장애 끝", OCCURRED.plusSeconds(61).toString()));
// when
outbox.jdbc().execute(redrive);
// then
assertThat(outbox.claimedEventIds()).containsExactlyInAnyOrder(early, late);
}
}
@Test
@DisplayName("runbook skip SQL은 DEAD 행을 발행된 것으로 두어 그 aggregate의 다음 행을 풀어 준다")
void theSkipReleasesTheNextRowOfTheDeadRowsAggregate() throws SQLException {
try (Outbox outbox = Outbox.create()) {
// given
String aggregate = "order-" + UUID.randomUUID();
String dead = outbox.append(aggregate, 1, OCCURRED);
String next = outbox.append(aggregate, 2, OCCURRED.plusSeconds(1));
outbox.setCounters(dead, "DEAD", 1, 0);
String skip = deadLetterRunbook.block("dead-skip-one", Map.of("id", dead));
// when
outbox.jdbc().execute(skip);
// then
assertThat(List.of(outbox.claimedEventIds(), outbox.status(dead)))
.containsExactly(List.of(next), "PUBLISHED");
}
}
@Test
@DisplayName("오래 재시도 중인 행 조회는 미룬 claim이 기준 수 이상인 FAILED 행과 그 원인 코드를 보여 준다")
void theLongRetryQueryListsFailedRowsDeferredLongEnoughWithTheirFailureCode()
throws SQLException {
try (Outbox outbox = Outbox.create()) {
// given: one row deferred long enough, one that has only just started failing
String stuck = outbox.append("order-" + UUID.randomUUID(), 1, OCCURRED);
String fresh = outbox.append("order-" + UUID.randomUUID(), 1, OCCURRED);
outbox.setCounters(stuck, "FAILED", 12, 12);
outbox.setCounters(fresh, "FAILED", 1, 1);
outbox.setFailureCode(stuck, "KAFKA_INVALIDTIMESTAMPEXCEPTION");
// when
List<Map<String, Object>> found =
outbox.jdbc().queryForList(publishFailedRunbook.block("long-retries"));
// then
assertThat(found)
.extracting(row -> row.get("event_id"), row -> row.get("last_failure_code"))
.containsExactly(tuple(stuck, "KAFKA_INVALIDTIMESTAMPEXCEPTION"));
}
}
@Test
@DisplayName("runbook park SQL로 보류한 행은 claim되지 않고 그 aggregate의 다음 행도 막는다")
void aParkedRowIsNotClaimedAndStillHoldsItsAggregatesNextRow() throws SQLException {
try (Outbox outbox = Outbox.create()) {
// given
String aggregate = "order-" + UUID.randomUUID();
String failing = outbox.append(aggregate, 1, OCCURRED);
outbox.append(aggregate, 2, OCCURRED.plusSeconds(1));
outbox.setCounters(failing, "FAILED", 12, 12);
String park = publishFailedRunbook.block("park-one", Map.of("id", failing));
// when
outbox.jdbc().execute(park);
// then
assertThat(List.of(outbox.claimedEventIds(), outbox.status(failing)))
.containsExactly(List.of(), "PARKED");
}
}
@Test
@DisplayName("runbook resume SQL로 되돌린 보류 행은 다음 claim이 가져간다")
void aResumedRowIsClaimedAgain() throws SQLException {
try (Outbox outbox = Outbox.create()) {
// given
String parked = outbox.append("order-" + UUID.randomUUID(), 1, OCCURRED);
outbox.setCounters(parked, "FAILED", 12, 12);
outbox.jdbc().execute(publishFailedRunbook.block("park-one", Map.of("id", parked)));
String resume = publishFailedRunbook.block("resume-one", Map.of("id", parked));
// when
outbox.jdbc().execute(resume);
// then
assertThat(outbox.claimedEventIds()).containsExactly(parked);
}
}
@Test
@DisplayName("runbook skip SQL로 버린 보류 행은 발행된 것으로 남고 그 aggregate의 다음 행을 풀어 준다")
void aSkippedParkedRowReleasesItsAggregatesNextRow() throws SQLException {
try (Outbox outbox = Outbox.create()) {
// given
String aggregate = "order-" + UUID.randomUUID();
String parked = outbox.append(aggregate, 1, OCCURRED);
String next = outbox.append(aggregate, 2, OCCURRED.plusSeconds(1));
outbox.setCounters(parked, "FAILED", 12, 12);
outbox.jdbc().execute(publishFailedRunbook.block("park-one", Map.of("id", parked)));
String skip = publishFailedRunbook.block("parked-skip-one", Map.of("id", parked));
// when
outbox.jdbc().execute(skip);
// then
assertThat(List.of(outbox.claimedEventIds(), outbox.status(parked)))
.containsExactly(List.of(next), "PUBLISHED");
}
}
/** A fresh database at the latest migration with the JPA outbox store over it. */
private static final class Outbox implements AutoCloseable {
private final HikariDataSource dataSource;
private final AnnotationConfigApplicationContext context;
private final JdbcTemplate jdbc;
private Outbox(HikariDataSource dataSource) {
this.dataSource = dataSource;
OutboxIntegrationSupport.migrate(dataSource);
this.context = OutboxIntegrationSupport.buildContext(dataSource, e -> {}, Clock.systemUTC());
this.jdbc = new JdbcTemplate(dataSource);
}
static Outbox create() throws SQLException {
String name =
"runbook_" + UUID.randomUUID().toString().replace("-", "").toLowerCase(Locale.ROOT);
return new Outbox(postgres.createDatabase(name));
}
JdbcTemplate jdbc() {
return jdbc;
}
String append(String aggregateId, long sequence, Instant occurredAt) {
String eventId = UUID.randomUUID().toString();
JpaOutboxStore store = context.getBean(JpaOutboxStore.class);
context
.getBean(SpringTransaction.class)
.inWrite(
() -> store.append(OutboxEvents.event(eventId, aggregateId, sequence, occurredAt)));
return eventId;
}
/** The counters a relay leaves on a row; written directly because only their values matter. */
void setCounters(String eventId, String status, int attempts, int deferred) {
jdbc.update(
"UPDATE outbox_event SET status = ?, attempt_count = ?, deferred_attempt_count = ?"
+ " WHERE event_id = ?",
status,
attempts,
deferred,
eventId);
}
void setFailureCode(String eventId, String failureCode) {
jdbc.update(
"UPDATE outbox_event SET last_failure_code = ? WHERE event_id = ?", failureCode, eventId);
}
String status(String eventId) {
return jdbc.queryForObject(
"SELECT status FROM outbox_event WHERE event_id = ?", String.class, eventId);
}
/** What the relay's next claim takes, a minute from now so every re-driven row is due. */
List<String> claimedEventIds() {
JpaOutboxStore store = context.getBean(JpaOutboxStore.class);
Instant now = Instant.now().plusSeconds(60);
return context
.getBean(SpringTransaction.class)
.inWrite(() -> store.claimBatch("runbook-relay", 50, now, IN_FLIGHT_TIMEOUT))
.stream()
.map(OutboxLease::eventId)
.toList();
}
@Override
public void close() {
context.close();
dataSource.close();
}
}
}
@@ -9,16 +9,20 @@ import dev.caskeleton.adapter.outbound.persistence.outbox.jpa.JpaOutboxStore;
import dev.caskeleton.adapter.outbound.persistence.transaction.SpringTransaction;
import dev.caskeleton.application.outbox.testkit.OutboxEvents;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.sql.Connection;
import java.sql.SQLException;
import java.sql.Statement;
import java.time.Clock;
import java.time.Instant;
import java.util.Arrays;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;
import org.flywaydb.core.Flyway;
import org.flywaydb.core.api.FlywayException;
import org.junit.jupiter.api.AfterAll;
@@ -43,15 +47,15 @@ class PostgreSqlOutboxOrderKeyRunbookIntegrationTest {
private static final Instant OCCURRED = Instant.parse("2026-11-01T00:00:00Z");
private static PostgreSqlReadinessSupport postgres;
private static OutboxRunbookSql runbook;
private static String duplicateCheck;
private static String removePublishedDuplicates;
@BeforeAll
static void setup() throws IOException {
String runbook =
Files.readString(Path.of(System.getProperty("outbox.runbook")), StandardCharsets.UTF_8);
duplicateCheck = runbookSql(runbook, "v21-duplicate-order-keys");
removePublishedDuplicates = runbookSql(runbook, "v21-remove-published-duplicates");
runbook = OutboxRunbookSql.fromSystemProperty("outbox.runbook");
duplicateCheck = runbook.block("v21-duplicate-order-keys");
removePublishedDuplicates = runbook.block("v21-remove-published-duplicates");
postgres = PostgreSqlReadinessSupport.start();
}
@@ -176,19 +180,143 @@ class PostgreSqlOutboxOrderKeyRunbookIntegrationTest {
}
}
/** The SQL block that follows {@code <!-- runbook-sql: id -->} in the runbook. */
private static String runbookSql(String runbook, String id) {
String marker = "<!-- runbook-sql: " + id + " -->";
int at = runbook.indexOf(marker);
if (at < 0) {
throw new IllegalStateException("the outbox runbook has no SQL block marked " + id);
@Test
@DisplayName("V21은 남은 행이 있는 aggregate의 카운터를 그 aggregate의 가장 큰 번호로 미리 만든다")
void v21SeedsTheCounterOfEveryAggregateWithRowsAtItsHighestNumber() throws SQLException {
try (BeforeV21 database = BeforeV21.create()) {
// given
String twice = "twice-" + UUID.randomUUID();
database.append(twice, 1_761_955_200_000_003L);
database.append(twice, 1_761_955_200_000_009L);
String once = "once-" + UUID.randomUUID();
database.append(once, 1_761_955_200_000_004L);
// when
database.migrateToLatest();
// then
assertThat(List.of(database.counterOf(twice), database.counterOf(once)))
.containsExactly(1_761_955_200_000_009L, 1_761_955_200_000_004L);
}
int open = runbook.indexOf("```sql\n", at);
int close = runbook.indexOf("\n```", open + 1);
if (open < 0 || close < 0) {
throw new IllegalStateException("the block marked " + id + " is not a closed sql fence");
}
@Test
@DisplayName("V21은 적용한 시각을 마이크로초 floor로 한 번 기록한다")
void v21RecordsTheMomentItRanAsTheFloorInMicroseconds() throws SQLException {
try (BeforeV21 database = BeforeV21.create()) {
// given
long before = microsecondsNow() - 1_000_000L;
// when
database.migrateToLatest();
// then
long after = microsecondsNow() + 1_000_000L;
assertThat(
database
.jdbc()
.queryForList(
"SELECT floor_sequence FROM outbox_aggregate_sequence_floor", Long.class))
.singleElement()
.satisfies(floor -> assertThat(floor).isBetween(before, after));
}
return runbook.substring(open + "```sql\n".length(), close);
}
@Test
@DisplayName("outbox 쓰기 트랜잭션이 잠금을 잡고 있으면 V21은 기다리다 실패하고 DB는 V20 그대로 남는다")
void v21GivesUpOnALockAWriterHoldsAndLeavesTheDatabaseAtV20() throws Exception {
try (BeforeV21 database = BeforeV21.create();
Connection writer = database.connection()) {
// given: a writer transaction holding the lock every outbox insert takes, released after 15 s
writer.setAutoCommit(false);
try (Statement statement = writer.createStatement()) {
statement.execute("LOCK TABLE outbox_event IN ROW EXCLUSIVE MODE");
}
ScheduledExecutorService release = Executors.newSingleThreadScheduledExecutor();
ScheduledFuture<Void> released =
release.schedule(
() -> {
writer.rollback();
return null;
},
15,
TimeUnit.SECONDS);
try {
// when
Throwable thrown = catchThrowable(database::migrateToLatest);
// then
assertThat(
List.of(
thrown instanceof FlywayException,
database.flyway(null).info().current().getVersion().getVersion()))
.containsExactly(true, "20");
} finally {
released.cancel(true);
release.shutdownNow();
}
}
}
@Test
@DisplayName("실패한 CONCURRENTLY 빌드가 남긴 무효 인덱스가 있으면 V21은 그것을 건너뛰지 않고 실패한다")
void anInvalidIndexLeftByAFailedConcurrentBuildFailsV21() throws SQLException {
try (BeforeV21 database = BeforeV21.create()) {
// given: a concurrent build that met a duplicate leaves an invalid index behind
String shared = "shared-" + UUID.randomUUID();
String kept = database.append(shared, 1);
database.append(shared, 1);
Throwable failedBuild =
catchThrowable(
() ->
database
.jdbc()
.execute(
"CREATE UNIQUE INDEX CONCURRENTLY ux_outbox_event_aggregate_order ON"
+ " outbox_event (tenant_scope, aggregate_type, aggregate_id,"
+ " aggregate_sequence, event_index)"));
assertThat(failedBuild).as("the concurrent build met the duplicate").isNotNull();
database
.jdbc()
.update(
"DELETE FROM outbox_event WHERE aggregate_id = ? AND event_id <> ?", shared, kept);
// when
Throwable thrown = catchThrowable(database::migrateToLatest);
// then
assertThat(
List.of(
thrown instanceof FlywayException,
database.flyway(null).info().current().getVersion().getVersion()))
.containsExactly(true, "20");
}
}
@Test
@DisplayName("runbook대로 순서 키 인덱스를 CONCURRENTLY로 먼저 만들면 V21은 그 인덱스를 쓰고 적용된다")
void v21AppliesOverTheIndexTheRunbookBuiltConcurrently() throws SQLException {
try (BeforeV21 database = BeforeV21.create()) {
// given
database.append("ordered-" + UUID.randomUUID(), 1);
database.jdbc().execute(runbook.block("v21-prebuild-order-index"));
// when
database.migrateToLatest();
// then
assertThat(
List.of(
Arrays.stream(database.flyway(null).info().applied())
.anyMatch(migration -> "21".equals(migration.getVersion().getVersion())),
database.validIndexExists("ux_outbox_event_aggregate_order")))
.containsExactly(true, true);
}
}
private static long microsecondsNow() {
Instant now = Instant.now();
return now.getEpochSecond() * 1_000_000L + now.getNano() / 1_000L;
}
/**
@@ -259,6 +387,26 @@ class PostgreSqlOutboxOrderKeyRunbookIntegrationTest {
jdbc.queryForObject("SELECT to_regclass(?) IS NOT NULL", Boolean.class, table));
}
Connection connection() throws SQLException {
return dataSource.getConnection();
}
long counterOf(String aggregateId) {
return jdbc.queryForObject(
"SELECT last_sequence FROM outbox_aggregate_sequence WHERE aggregate_id = ?",
Long.class,
aggregateId);
}
boolean validIndexExists(String index) {
return Boolean.TRUE.equals(
jdbc.queryForObject(
"SELECT EXISTS (SELECT 1 FROM pg_index WHERE indexrelid = to_regclass(?)"
+ " AND indisvalid)",
Boolean.class,
index));
}
boolean indexExists(String index) {
return Boolean.TRUE.equals(
jdbc.queryForObject(
@@ -0,0 +1,60 @@
package dev.caskeleton.adapter.outbound.persistence.readiness;
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaRecoveryQueue;
import dev.caskeleton.application.fileserver.api.FileId;
import dev.caskeleton.application.fileserver.api.StorageNamespace;
import dev.caskeleton.application.fileserver.recovery.RecoveryQueuePort;
import dev.caskeleton.application.fileserver.testkit.RecoveryQueuePortContract;
import java.time.Duration;
import java.util.function.Supplier;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Tag;
/**
* {@link JpaRecoveryQueue} held to the contract the in-memory queue passes, on real PostgreSQL
* (review 2026-11-03 RR-01-04).
*/
@Tag("jpa-contract")
class PostgreSqlRecoveryQueueContractTest extends RecoveryQueuePortContract {
private static FileserverPostgreSqlStack stack;
@BeforeAll
static void start() {
stack = FileserverPostgreSqlStack.start();
}
@AfterAll
static void stop() {
if (stack != null) {
stack.close();
}
}
@BeforeEach
void empty() {
stack.empty();
}
@Override
protected RecoveryQueuePort queue() {
return stack.bean(JpaRecoveryQueue.class);
}
@Override
protected FileId recordedFile() {
return stack.recordFileIn(StorageNamespace.of("tenant-a"));
}
@Override
protected void elapse(Duration duration) {
stack.elapse(duration);
}
@Override
protected <T> T inTransaction(Supplier<T> work) {
return stack.inTransaction(work);
}
}
@@ -0,0 +1,73 @@
package dev.caskeleton.adapter.outbound.persistence.readiness;
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaStagingUploadLocator;
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaUploadSession;
import dev.caskeleton.application.fileserver.api.FileId;
import dev.caskeleton.application.fileserver.api.StorageNamespace;
import dev.caskeleton.application.fileserver.api.metadata.UploadSessionPort;
import dev.caskeleton.application.fileserver.recovery.StagingUploadLocatorPort;
import dev.caskeleton.application.fileserver.testkit.StagingUploadLocatorPortContract;
import java.time.Duration;
import java.time.Instant;
import java.util.function.Supplier;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Tag;
/**
* {@link JpaStagingUploadLocator} held to the contract the in-memory session store passes, on real
* PostgreSQL (review 2026-11-03 RR-01-04).
*/
@Tag("jpa-contract")
class PostgreSqlStagingUploadLocatorContractTest extends StagingUploadLocatorPortContract {
private static FileserverPostgreSqlStack stack;
@BeforeAll
static void start() {
stack = FileserverPostgreSqlStack.start();
}
@AfterAll
static void stop() {
if (stack != null) {
stack.close();
}
}
@BeforeEach
void empty() {
stack.empty();
}
@Override
protected StagingUploadLocatorPort locator() {
return stack.bean(JpaStagingUploadLocator.class);
}
@Override
protected UploadSessionPort sessions() {
return stack.bean(JpaUploadSession.class);
}
@Override
protected FileId recordedFile() {
return stack.recordFileIn(StorageNamespace.of("tenant-a"));
}
@Override
protected Instant now() {
return stack.now();
}
@Override
protected void elapse(Duration duration) {
stack.elapse(duration);
}
@Override
protected <T> T inTransaction(Supplier<T> work) {
return stack.inTransaction(work);
}
}
@@ -293,6 +293,49 @@ class PostgreSqlTransactionIntegrationTest {
assertThat(elapsed).isLessThan(Duration.ofSeconds(5));
}
/**
* The production transaction manager is {@code JpaTransactionManager}, and past the deadline
* Hibernate refuses the next statement itself, before the driver is asked: no {@code 57014}, no
* SQLState at all (review RR-03-03). The test above covers the JDBC path only.
*/
@Test
@Tag("jpa-failure")
@Timeout(20)
@DisplayName("JPA 트랜잭션이 제한 시간을 넘긴 뒤 다음 문장을 Hibernate가 거부해도 DB_QUERY_CANCELED로 끝난다")
void aJpaWriteTransactionPastItsBoundIsAQueryCancellation() {
// given
Configuration configuration = new Configuration();
configuration.getProperties().put("hibernate.connection.datasource", postgres.dataSource());
configuration.setProperty("hibernate.hbm2ddl.auto", "create");
configuration.addAnnotatedClass(VersionedRow.class);
try (SessionFactory sessionFactory = configuration.buildSessionFactory()) {
EntityManager entityManager =
SharedEntityManagerCreator.createSharedEntityManager(sessionFactory);
SpringTransaction port =
new SpringTransaction(
new JpaTransactionManager(sessionFactory),
translator(),
new SpringTransaction.Timeouts(Duration.ofSeconds(1), Duration.ofSeconds(1)));
// when
Throwable thrown =
catchThrowable(
() ->
port.inWrite(
() -> {
sleepPastOneSecond();
entityManager.persist(new VersionedRow(7L, "late"));
entityManager.flush();
return null;
}));
// then
assertThat(thrown).isInstanceOf(PersistenceFailureException.class);
assertThat(((PersistenceFailureException) thrown).errorCode())
.isEqualTo(OperationalError.DB_QUERY_CANCELED);
}
}
// ---- review 09-05: READ COMMITTED comes from the pool, with no per-transaction round trip ----
@Test
@@ -323,6 +366,15 @@ class PostgreSqlTransactionIntegrationTest {
}
}
private static void sleepPastOneSecond() {
try {
Thread.sleep(1_200);
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
throw new IllegalStateException(interrupted);
}
}
/** A data source whose connections count {@code getTransactionIsolation()} calls. */
private static DataSource countingIsolationQueries(DataSource target, AtomicInteger calls) {
return (DataSource)
@@ -0,0 +1,66 @@
package dev.caskeleton.adapter.outbound.persistence.readiness;
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaUploadSession;
import dev.caskeleton.application.fileserver.api.FileId;
import dev.caskeleton.application.fileserver.api.StorageNamespace;
import dev.caskeleton.application.fileserver.api.metadata.UploadSessionPort;
import dev.caskeleton.application.fileserver.testkit.UploadSessionPortContract;
import java.time.Duration;
import java.time.Instant;
import java.util.function.Supplier;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Tag;
/**
* {@link JpaUploadSession} held to the writer-lease contract the in-memory store passes, on real
* PostgreSQL (review 2026-11-03 RR-01-04).
*/
@Tag("jpa-contract")
class PostgreSqlUploadSessionContractTest extends UploadSessionPortContract {
private static FileserverPostgreSqlStack stack;
@BeforeAll
static void start() {
stack = FileserverPostgreSqlStack.start();
}
@AfterAll
static void stop() {
if (stack != null) {
stack.close();
}
}
@BeforeEach
void empty() {
stack.empty();
}
@Override
protected UploadSessionPort sessions() {
return stack.bean(JpaUploadSession.class);
}
@Override
protected FileId recordedFile() {
return stack.recordFileIn(StorageNamespace.of("tenant-a"));
}
@Override
protected Instant now() {
return stack.now();
}
@Override
protected void elapse(Duration duration) {
stack.elapse(duration);
}
@Override
protected <T> T inTransaction(Supplier<T> work) {
return stack.inTransaction(work);
}
}
@@ -13,6 +13,9 @@ import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyNoInteractions;
import static org.mockito.Mockito.when;
import ch.qos.logback.classic.Logger;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.core.read.ListAppender;
import dev.caskeleton.adapter.outbound.persistence.outbox.jpa.entity.OutboxEventEntity;
import dev.caskeleton.application.messaging.contract.ContractId;
import dev.caskeleton.application.messaging.contract.LogicalDestinationId;
@@ -29,13 +32,16 @@ import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.time.Instant;
import java.util.HexFormat;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.mockito.ArgumentCaptor;
import org.slf4j.LoggerFactory;
import org.springframework.transaction.support.TransactionSynchronizationManager;
/**
@@ -181,10 +187,10 @@ class JpaOutboxStoreTest {
assertThatThrownBy(() -> adapter.markPublished(lease("evt-guard")))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("markPublished");
assertThatThrownBy(() -> adapter.markFailed(lease("evt-guard"), NOW.plusSeconds(1)))
assertThatThrownBy(() -> adapter.markFailed(lease("evt-guard"), NOW.plusSeconds(1), "CODE"))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("markFailed");
assertThatThrownBy(() -> adapter.markDead(lease("evt-guard")))
assertThatThrownBy(() -> adapter.markDead(lease("evt-guard"), "CODE"))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("markDead");
}
@@ -281,6 +287,56 @@ class JpaOutboxStoreTest {
assertThat(List.of(claimed, entity.getStatus())).containsExactly(List.of(), "DEAD");
}
/**
* A row whose deferred claims outnumber its claims was not damaged: a manual re-drive reset
* {@code attempt_count} and left {@code deferred_attempt_count} (re-review 2026-11-03 RR-05-01).
* The WARN says that, so the operator re-runs the corrected runbook SQL instead of hunting for
* corruption.
*/
@Test
@DisplayName("claim이 시도 횟수가 어긋난 행을 DEAD로 옮기면 WARN이 어긋난 두 값과 runbook을 말한다")
void claimBatchNamesInconsistentAttemptCountersWhenItMovesSuchARowToDead() {
// given
OutboxEventEntity redriven = pendingEntity("evt-redriven", "worklog-42");
redriven.setDeferredAttemptCount(2);
when(claimRepo.claimEligible(eq(NOW), eq(1))).thenReturn(List.of(redriven));
Logger storeLog = (Logger) LoggerFactory.getLogger(JpaOutboxStore.class);
ListAppender<ILoggingEvent> captured = new ListAppender<>();
captured.start();
storeLog.addAppender(captured);
// when
try {
adapter.claimBatch(OWNER, 1, NOW, IN_FLIGHT_TIMEOUT);
} finally {
storeLog.detachAppender(captured);
}
// then
assertThat(captured.list)
.singleElement()
.satisfies(
warn ->
assertThat(List.of(warn.getFormattedMessage(), keyValues(warn)))
.containsExactly(
"outbox claim moved a row to DEAD: deferred_attempt_count (2) exceeds"
+ " attempt_count (1); a manual re-drive that did not reset"
+ " deferred_attempt_count leaves this behind — re-drive it with the"
+ " runbook SQL",
Map.of(
"event_id", "evt-redriven",
"error.reason", "ATTEMPT_COUNTERS_INCONSISTENT",
"runbook_link", "runbook://outbox/dead-letter")));
}
private static Map<String, Object> keyValues(ILoggingEvent event) {
Map<String, Object> values = new LinkedHashMap<>();
if (event.getKeyValuePairs() != null) {
event.getKeyValuePairs().forEach(pair -> values.put(pair.key, pair.value));
}
return values;
}
@Test
void claimBatchReturnsEmptyWhenRepoReturnsNothing() {
when(claimRepo.claimEligible(any(), anyInt())).thenReturn(List.of());
@@ -319,16 +375,18 @@ class JpaOutboxStoreTest {
@Test
void markFailedCompletesOnlyUnderTheLeaseAndSchedulesTheRetry() {
Instant retryAt = NOW.plus(Duration.ofSeconds(30));
when(repo.failLease("evt-004", OWNER, 1, retryAt)).thenReturn(1);
when(repo.failLease("evt-004", OWNER, 1, retryAt, "IllegalStateException")).thenReturn(1);
assertThat(adapter.markFailed(lease("evt-004"), retryAt)).isEqualTo(OutboxCompletion.APPLIED);
assertThat(adapter.markFailed(lease("evt-004"), retryAt, "IllegalStateException"))
.isEqualTo(OutboxCompletion.APPLIED);
}
@Test
void markDeadCompletesOnlyUnderTheLease() {
when(repo.completeLease("evt-005", OWNER, 1, "DEAD")).thenReturn(1);
when(repo.deadLease("evt-005", OWNER, 1, "REJECTED_BEFORE_SEND")).thenReturn(1);
assertThat(adapter.markDead(lease("evt-005"))).isEqualTo(OutboxCompletion.APPLIED);
assertThat(adapter.markDead(lease("evt-005"), "REJECTED_BEFORE_SEND"))
.isEqualTo(OutboxCompletion.APPLIED);
}
/**
@@ -111,7 +111,7 @@ class FlywayProdSafetyValidatorTest {
@Configuration
static class ValidatorConfig {
@Bean
FlywayProdSafetyValidator flywayProdSafetyValidator(Environment environment) {
static FlywayProdSafetyValidator flywayProdSafetyValidator(Environment environment) {
return new FlywayProdSafetyValidator();
}
}
@@ -300,7 +300,7 @@ class HikariPoolConstraintValidatorTest {
@Configuration
static class ValidatorConfig {
@Bean
HikariPoolConstraintValidator hikariPoolConstraintValidator(Environment environment) {
static HikariPoolConstraintValidator hikariPoolConstraintValidator(Environment environment) {
return new HikariPoolConstraintValidator();
}
}
@@ -72,7 +72,7 @@ class OpenInViewSafetyValidatorTest {
@Configuration
static class ValidatorConfig {
@Bean
OpenInViewSafetyValidator openInViewSafetyValidator(Environment environment) {
static OpenInViewSafetyValidator openInViewSafetyValidator(Environment environment) {
return new OpenInViewSafetyValidator();
}
}
@@ -101,7 +101,7 @@ class PersistenceVendorProdSafetyValidatorTest {
static class ValidatorConfig {
@Bean
PersistenceVendorProdSafetyValidator validator(Environment environment) {
static PersistenceVendorProdSafetyValidator validator(Environment environment) {
return new PersistenceVendorProdSafetyValidator();
}
}
@@ -102,7 +102,7 @@ class PostgreSqlTransportSecurityValidatorTest {
static class ValidatorConfig {
@Bean
PostgreSqlTransportSecurityValidator postgreSqlTransportSecurityValidator(
static PostgreSqlTransportSecurityValidator postgreSqlTransportSecurityValidator(
Environment environment) {
return new PostgreSqlTransportSecurityValidator();
}
@@ -81,7 +81,7 @@ class RequiredEnvironmentValidatorTest {
@Configuration
static class ValidatorConfig {
@Bean
RequiredEnvironmentValidator requiredEnvironmentValidator(Environment environment) {
static RequiredEnvironmentValidator requiredEnvironmentValidator(Environment environment) {
return new RequiredEnvironmentValidator();
}
}
@@ -10,6 +10,8 @@ import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.FinalEnti
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.InverseCollectionParentEntity;
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.InverseManyToManyEntity;
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.InverseOneToOneEntity;
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.JoinColumnChildEntity;
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.JoinColumnParentEntity;
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.NoDefaultConstructorEntity;
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.OrderEntity;
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.OwningChildEntity;
@@ -19,6 +21,7 @@ import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.Unidirect
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.web.BadOrderController;
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.web.GoodOrderController;
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.web.ListReturningController;
import dev.caskeleton.adapter.outbound.persistence.testkit.outsider.OrderEntityReader;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
@@ -181,6 +184,27 @@ class JpaArchitectureRulesTest {
.hasMessageNotContaining("OwningChildEntity.parent");
}
@Test
@DisplayName("mappedBy 없이 두 엔티티가 서로를 연관으로 가리키면 양쪽 모두 거부된다")
void aPairThatPointsBothWaysWithoutMappedByIsRefusedOnBothSides() {
// given
var classes =
new ClassFileImporter()
.importClasses(JoinColumnParentEntity.class, JoinColumnChildEntity.class);
// when
Throwable failure =
catchThrowable(
() -> JpaArchitectureRules.associationsAreMappedFromOneSideOnly().check(classes));
// then
assertThat(failure)
.isInstanceOf(AssertionError.class)
.hasMessageContaining("JoinColumnParentEntity.children")
.hasMessageContaining("JoinColumnChildEntity.parent")
.hasMessageContaining("one side only");
}
@Test
@DisplayName("단방향 연관관계는 통과한다")
void unidirectionalAssociationsPass() {
@@ -195,4 +219,44 @@ class JpaArchitectureRulesTest {
assertThatCode(() -> JpaArchitectureRules.associationsAreMappedFromOneSideOnly().check(classes))
.doesNotThrowAnyException();
}
@Test
@DisplayName("leaf 밖의 클래스가 그 leaf의 엔티티를 쓰면 거부된다")
void aClassOutsideTheLeafThatUsesItsEntityIsRefused() {
// given
var classes = new ClassFileImporter().importClasses(OrderEntity.class, OrderEntityReader.class);
// when
Throwable failure =
catchThrowable(
() ->
JpaArchitectureRules.entitiesAreUsedOnlyInsideTheirLeaf(
"dev.caskeleton.adapter.outbound.persistence.testkit.arch")
.check(classes));
// then
assertThat(failure)
.isInstanceOf(AssertionError.class)
.hasMessageContaining("OrderEntityReader")
.hasMessageContaining("OrderEntity");
}
@Test
@DisplayName("같은 leaf 안의 클래스가 그 leaf의 엔티티를 쓰는 것은 통과한다")
void aClassInsideTheLeafMayUseItsEntity() {
// given
var classes =
new ClassFileImporter().importClasses(OrderEntity.class, BadOrderController.class);
// when
Throwable failure =
catchThrowable(
() ->
JpaArchitectureRules.entitiesAreUsedOnlyInsideTheirLeaf(
"dev.caskeleton.adapter.outbound.persistence.testkit.arch")
.check(classes));
// then
assertThat(failure).isNull();
}
}
@@ -0,0 +1,36 @@
package dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity;
import jakarta.persistence.Entity;
import jakarta.persistence.FetchType;
import jakarta.persistence.Id;
import jakarta.persistence.JoinColumn;
import jakarta.persistence.ManyToOne;
/**
* Refused by the association rule: the read-only half of {@link JoinColumnParentEntity}'s pair.
*
* <p>{@code insertable = false, updatable = false} makes this field a second, read-only copy of the
* link the parent's collection writes. Within one persistence context it does not follow a change
* made through the collection.
*/
@Entity
public class JoinColumnChildEntity {
@Id private Long id;
@ManyToOne(fetch = FetchType.LAZY)
@JoinColumn(name = "parent_id", insertable = false, updatable = false)
private JoinColumnParentEntity parent;
protected JoinColumnChildEntity() {}
/** The identifier. */
public Long id() {
return id;
}
/** The read-only reference back to the parent. */
public JoinColumnParentEntity parent() {
return parent;
}
}

Some files were not shown because too many files have changed in this diff Show More