fix!: 재리뷰가 찾은 결함과 색인에서 빠졌던 발견 5건을 고친다
문제: 성숙도 리뷰 수정을 다시 읽은 재리뷰(Critical 0, 회귀 0)가 Important 27건·Minor 41건을 찾았다. 수정이 새로 만든 결함이 많았다. @SystemInvoked가 권한 규칙의 탈출구가 됐고, 데모 계정 거부가 정확히 prod일 때만 걸렸고, outbox 순서 번호가 업그레이드 때 뒤로 갔고, GraphQL union 아래 복잡도가 0이었고, 서블릿 오류 응답이 success:true로 나갔고, 호출자 deadline이 breaker를 열었다. 원래 리뷰 색인이 06 영역 발견 5건(06-10~14)을 빠뜨려 그동안 처리되지 않았다. 방법: 영역별 7묶음이 test-driven-development 절차로 실패 테스트를 먼저 쓰고 고쳤다. 결정은 재리뷰 README 4절에 적었다: 순서 번호는 V21이 기록한 전역 floor부터, 데모 계정은 local·test·demo 허용 목록, @SystemInvoked bean은 기동 실패, complete 단계 실패는 IDEMPOTENCY_COMPLETION_UNKNOWN(결과 불명), 캐시 조건부 쓰기는 별도 SCRIPT lane, strict lane 정책은 build cache 입력, 끝나지 않는 재시도 행은 원인 코드·지표·PARKED 절차. 레지스트리가 선언만 하고 아무 코드도 내보내지 않던 redis.capability.* 지표 6개는 빼고 runbook을 실제 지표(redis.lane.*) 기준으로 고쳤다. 위험·한계: V21을 V22 없이 직접 고쳤다(어느 배포에도 아직 없음). 이 브랜치의 V21을 적용한 개발 DB는 다시 만들어야 한다. 서블릿 오류 디스패치를 보안 체인이 허용한다. 발행·저장소 포트 시그니처가 바뀌어 파생 서비스 코드가 깨진다. prod에서 fixture-로 시작하는 설정값은 기동을 막는다. httpclient failure-injection lane의 main 단계 중복 실행은 이전 결정대로 남겼다. 실제 CI runner 실행은 확인하지 못했다. 검증: ci, mainCheck, :integrationCheck, -p optional-platforms ci integrationCheck performanceCheck, :performanceCheck, :releaseCheck (-PgitRevision), verify-compose-profile-contracts(18 lane) 통과. 첫 실행의 실패 2건(FindSecBugs 1, sample 동시 생성 테스트의 v4 event id)은 고친 뒤 ci·integrationCheck·mainCheck를 다시 돌려 통과했다. Refs: docs/reviews/2026-11-03-rereview/README.md 3·4·6·7절 BREAKING CHANGE: outbox 발행 포트와 저장소 포트 시그니처, 순서 번호 첫 값(V21 floor 다음), 샘플 데모 계정 프로필 허용 목록, 새 오류 코드 IDEMPOTENCY_COMPLETION_UNKNOWN, prod의 fixture- 값 거부, 기동 rule은 static·의존 없음이어야 한다.
This commit is contained in:
@@ -128,12 +128,15 @@ Focused tests drive the semantic adapters against `InMemorySemanticRedisCommands
|
||||
served through `StubRedisRuntimeClient`. R1/R2 promotion requires a separate real Redis service lane; it may never
|
||||
be silently skipped when selected.
|
||||
|
||||
Two real-server lanes. `redisContainerContractTest` (integration category, in `integrationCheck` and
|
||||
this leaf's `mainCheck`) starts its own Redis with Testcontainers — the `redis` image of
|
||||
`config/testing/container-images.properties`, loading `infra/redis-sdk/acl/all-accounts.acl` — and
|
||||
runs `LiveRedisCacheRegionContractTest` and `LiveRedisDistributedLeaseContractTest` (the
|
||||
`application-core` port contracts the in-memory fakes also pass), `LiveRedisSemanticPortsTest` and
|
||||
`LiveRedisSemanticScriptAclProbeTest`; a skip fails it (review 2026-11-01 05-16). `RedisTestContainer`
|
||||
Two real-server lanes. `redisContainerContractTest` (integration category, in `integrationCheck` only —
|
||||
not in this leaf's `mainCheck` too, which ran it twice per push to main) starts its own Redis with Testcontainers — the `redis` image of
|
||||
`config/testing/container-images.properties`, loading `infra/redis-sdk/acl/lane-fixture-accounts.acl` — and
|
||||
runs `LiveRedisCacheRegionContractTest`, `LiveRedisDistributedLeaseContractTest` and
|
||||
`LiveRedisIdempotencyStoreContractTest` (the `application-core` port contracts the in-memory fakes
|
||||
also pass), `LiveRedisSemanticPortsTest` and `LiveRedisSemanticScriptAclProbeTest`; a skip fails it
|
||||
(review 2026-11-01 05-16). The idempotency contract was on the explicit-only topology lane alone
|
||||
until review 2026-11-03 RR-04-05, so the Lua state machine's stale-handle, renew-refusal and
|
||||
retry cases reached a real server in no gate. `RedisTestContainer`
|
||||
starts the server only when that lane names the ACL file.
|
||||
|
||||
`redisTopologyTest` is the external-topology lane, declared through `ca.redis-topology-lane`. This
|
||||
@@ -177,6 +180,15 @@ deployment still gets exactly one client and one event loop. The `SCRIPT` lane i
|
||||
the reason it exists: `SCRIPT LOAD` and `EVALSHA` belong to the advanced account, so the account
|
||||
that reads a cache entry cannot execute a script.
|
||||
|
||||
The cache's conditional writes run on the same account but on a lane of their own,
|
||||
`CacheScriptLane.CACHE_SCRIPT` (`app.capabilities.cache.maximum-script-connections`, contributed by
|
||||
`RedisCacheConfig` as a `RedisLaneContribution`). Do not move them back to `SCRIPT`: that lane is the
|
||||
idempotency, rate-limit and lease budget, and a cache under load must not take it (review 2026-11-03
|
||||
README §4, RR-04-02). Each lane's wait, refusals and reconnects are meters of `redis-runtime`
|
||||
(`RedisLaneMetrics`, `redis.lane.*`, tag `lane`), so whether the cache stays off `SCRIPT` is a query,
|
||||
not an assumption. The `DEGRADED` health order ships with `redis-runtime`, which reports the
|
||||
status (`config/module-defaults/redis-runtime.yml`, RR-04-10).
|
||||
|
||||
A Cluster transaction runs on one node, so the SDK's `RedisTransactionRunner` derives a routing key
|
||||
from the watched keys (or an explicit `RedisSlotTag`) and borrows `RedisSdkLane.TRANSACTION`, which
|
||||
is pinned to its key's owner, on the node that owns that slot. Routed leases are never pooled: a pooled connection is pinned to the previous caller's
|
||||
|
||||
@@ -97,7 +97,10 @@ Connection lane limits and command queue limits are different contracts. In part
|
||||
|
||||
- `app.redis.capacity.maximum-in-flight-commands` bounds the per-connection command queue.
|
||||
- `app.redis.capacity.maximum-script-connections` bounds physical SCRIPT lane connections
|
||||
(default 8).
|
||||
(default 8). Idempotency, rate limiting and leases share them.
|
||||
- `app.capabilities.cache.maximum-script-connections` (`APP_CACHE_REDIS_MAXIMUM_SCRIPT_CONNECTIONS`)
|
||||
bounds the cache's own script lane (`CacheScriptLane`, default 4), contributed only when the cache
|
||||
binding is `redis`. See "Why the cache has its own script lane" below.
|
||||
- `app.redis.lifecycle.acquire-timeout` (default 2s) is how long a borrow waits for a returned
|
||||
connection when its lane is full.
|
||||
|
||||
@@ -153,7 +156,7 @@ Important contracts:
|
||||
Every lookup returns a `CacheWriteCondition` naming the state it saw — `absent`, `vacant:<digest>`
|
||||
(an invalidation marker, an entry of an older region generation, an expired entry) or
|
||||
`present:<digest>`. A record that carries the condition applies only while the key is still in that
|
||||
state (registered program, SCRIPT lane). This is what rejects a source load that began before an
|
||||
state (registered program, cache script lane). This is what rejects a source load that began before an
|
||||
invalidation and finished after it.
|
||||
|
||||
A hit also returns an opaque SHA-1 observation token over the stored bytes. SHA-1 here is **not a
|
||||
@@ -181,10 +184,43 @@ example one written by `UPSERT` as `rev-1`), another schema version, a generatio
|
||||
writer knows — is `NOT_RECORDED_CONDITION`.
|
||||
|
||||
Both conditional intents, and any record carrying a write condition, run registered scripts, so
|
||||
they borrow the SCRIPT lane (the advanced account when one is configured, the application account
|
||||
otherwise). The application account is not granted `EVALSHA`; on the REGULAR lane every conditional
|
||||
write came back `INDETERMINATE`. Unconditioned `UPSERT` and `ONLY_IF_ABSENT` stay on the REGULAR
|
||||
lane.
|
||||
they run on the scripting account (the advanced account when one is configured, the application
|
||||
account otherwise), on the cache's own script lane. The application account is not granted
|
||||
`EVALSHA`; on the REGULAR lane every conditional write came back `INDETERMINATE`. Unconditioned
|
||||
`UPSERT` and `ONLY_IF_ABSENT` stay on the REGULAR lane.
|
||||
|
||||
### Why the cache has its own script lane
|
||||
|
||||
Recording with the condition a lookup returned is the normal way to use this port, so nearly every
|
||||
cache fill is a script. On the runtime's SCRIPT lane those fills shared eight connections per pod
|
||||
with idempotency claims, rate-limit evaluations and lease calls, and a timed-out lease is discarded
|
||||
rather than pooled, so each timeout also cost a reconnect. A slow Redis or a burst of misses after a
|
||||
region invalidation filled the lane with cache work; an idempotency claim then waited out its
|
||||
timeout and answered `UNAVAILABLE`, and a rate-limit call fell to its failure policy. The cache is a
|
||||
speed feature that may degrade; idempotency, rate limiting and leases are correctness features that
|
||||
must not (review 2026-11-03 README §4, RR-04-02).
|
||||
|
||||
So `CacheScriptLane.CACHE_SCRIPT` is a lane of its own — the same scripting account, its own
|
||||
connections, its own ceiling (`app.capabilities.cache.maximum-script-connections`, default 4). A cache
|
||||
that runs out of them degrades (`DEGRADED_UNAVAILABLE`, nothing sent) without touching the SCRIPT
|
||||
lane. Alternatives not taken: a per-capability permit cap inside the shared SCRIPT lane (still one
|
||||
pool of connections and one reconnect cost), and running the cache programs on the REGULAR lane
|
||||
(grants the application account `EVALSHA`, which the account split exists to withhold).
|
||||
|
||||
Whether the split holds under load is visible per lane. With a `MeterRegistry` in the context,
|
||||
`redis-runtime` registers these for every lane it pools — `REGULAR`, `SCRIPT`, `CACHE_SCRIPT` and the
|
||||
typed SDK's — tagged `lane=<name>` (`RedisLaneMetrics`; rows in `docs/registries/metrics.yaml`):
|
||||
|
||||
| Meter | Type | Read it as |
|
||||
| --- | --- | --- |
|
||||
| `redis.lane.acquire.wait.seconds` (`outcome` = `admitted`/`refused`) | timer, histogram | how long a borrow queued for a connection; a growing admitted p99 comes before refusals |
|
||||
| `redis.lane.acquire.refused.total` (`reason` = `saturated`/`closing`/`interrupted`/`connect_failed`) | counter | a borrow that got no connection, nothing sent; `saturated` on `SCRIPT` is the starvation this lane exists to prevent |
|
||||
| `redis.lane.connections.opened.total` | counter | new connections; after warm-up every one is a reconnect |
|
||||
| `redis.lane.connections.discarded.total` (`reason` = `invalidated`/`broken`) | counter | connections closed instead of pooled — `invalidated` is a timed-out or failed call (the lease is not reused), and is what the reconnects replace |
|
||||
| `redis.lane.connections.active`, `redis.lane.connections.pending` | gauge | connections lent out now; borrows waiting now |
|
||||
|
||||
Every counter series exists from startup at zero, so an alert on "refusals above zero" sees the first
|
||||
one. A lane refusal is also logged, per call, by `RedisSemanticFailureLogger`.
|
||||
|
||||
### Key invalidation
|
||||
|
||||
@@ -300,10 +336,21 @@ Useful focused checks:
|
||||
|
||||
`redisContainerContractTest` runs the cache-region and lease port contracts (the ones the in-memory
|
||||
fakes pass) and the live semantic-port tests against a Redis it starts itself, under the standalone
|
||||
topology's ACL file. It is in `integrationCheck` and `mainCheck`; the image is
|
||||
topology's ACL file. It is in `integrationCheck` (integration-main.yml's `integration-check` job), and
|
||||
only there: listing it in `mainCheck` as well ran it twice on every push to main. The image is
|
||||
`config/testing/container-images.properties` `redis`.
|
||||
|
||||
Live Redis/topology qualification remains separate from fast unit tests. A release claim for a
|
||||
specific Redis topology or ACL layout requires the corresponding live qualification lane; the
|
||||
in-memory gateway is for deterministic contract tests, not evidence that an external Redis
|
||||
deployment accepts the same network/TLS/ACL configuration.
|
||||
|
||||
## 아직 내보내지 않는 지표
|
||||
|
||||
`redis.capability.*` 여섯 개(operations.total, duration.seconds, admission.rejected.total,
|
||||
inflight.total, readiness.total, lifecycle.drain.total)는 redis-production-capability 설계(Task 16)에
|
||||
있었지만 어떤 코드도 내보내지 않았다. 레지스트리(`docs/registries/metrics.yaml`)와 incident runbook이 있는
|
||||
것처럼 적고 있어서, 2026-11-03 재리뷰 수정에서 레지스트리에서 빼고 runbook을 지금 내보내는
|
||||
`redis.lane.*`(redis-runtime, lane별 대기·거절·연결) 기준으로 고쳤다. capability 단위 지표가 필요해지면
|
||||
operation 결과를 아는 semantic adapter(이 leaf)에서 내보내고, 그때 레지스트리에 다시 넣는다.
|
||||
|
||||
|
||||
@@ -48,7 +48,8 @@ dependencies {
|
||||
// command surface the catalog exists to prevent. Zero imports.
|
||||
// io.micrometer:micrometer-core — observation leaves this leaf as RedisObservation through a
|
||||
// Consumer sink; binding it to a meter registry belongs to the composition root, not here.
|
||||
// Zero imports.
|
||||
// Zero imports. The connection lanes' meters, CACHE_SCRIPT's included, are redis-runtime's
|
||||
// (RedisLaneMetrics): the lanes are pooled there, so that is where they are counted.
|
||||
}
|
||||
|
||||
// Redis topology qualification is opt-in and owned by `ca.redis-topology-lane`: mode/property
|
||||
@@ -69,8 +70,12 @@ redisTopologyLane {
|
||||
// adapters' port contracts reached a real server in no stage, and the fake and the adapter could hide
|
||||
// the same defect (05-02, 05-03). The image is config/testing/container-images.properties' `redis`,
|
||||
// and the ACL file is infra/redis-sdk's own, so the accounts and their rights are the standalone
|
||||
// topology's. Integration category: integrationCheck runs it, and so does this leaf's mainCheck, like
|
||||
// the HTTP client's Toxiproxy lane. Sentinel, cluster and TLS remain the topology lane's.
|
||||
// topology's. Integration category, so integrationCheck runs it — integration-main.yml's
|
||||
// `integration-check` job, on every push to main and nightly. It is deliberately not in this leaf's
|
||||
// mainCheck as well: `main-check` and `integration-check` are two jobs of the same workflow run, so a
|
||||
// lane in both started a second Redis container and ran every live contract twice per push without
|
||||
// covering anything the first run did not (found while fixing review 2026-11-03). Sentinel, cluster
|
||||
// and TLS remain the topology lane's.
|
||||
def redisAclDirectory = rootProject.projectDir.parentFile.toPath().resolve('infra/redis-sdk/acl').toFile()
|
||||
strictTestLanes {
|
||||
lane('redisContainerContractTest') {
|
||||
@@ -78,15 +83,16 @@ strictTestLanes {
|
||||
description = 'Runs the Redis semantic adapters\' live contracts against a Testcontainers Redis on the pinned image.'
|
||||
requires 'dev.caskeleton.adapter.outbound.cache.redis.cache.LiveRedisCacheRegionContractTest',
|
||||
'dev.caskeleton.adapter.outbound.cache.redis.lease.LiveRedisDistributedLeaseContractTest',
|
||||
// The fake carries a Java copy of the idempotency Lua state machine; only a real
|
||||
// server proves the two agree (05-02 was the fake copying the Lua's bug). It ran on
|
||||
// the explicit-only topology lane alone, so no gate ever ran it (RR-04-05).
|
||||
'dev.caskeleton.adapter.outbound.cache.redis.idempotency.LiveRedisIdempotencyStoreContractTest',
|
||||
'dev.caskeleton.adapter.outbound.cache.redis.LiveRedisSemanticPortsTest',
|
||||
'dev.caskeleton.adapter.outbound.cache.redis.capability.LiveRedisSemanticScriptAclProbeTest'
|
||||
rejectSkipped = true
|
||||
systemProperty 'redis.container.acl-file', new File(redisAclDirectory, 'all-accounts.acl').absolutePath
|
||||
systemProperty 'redis.container.acl-file', new File(redisAclDirectory, 'lane-fixture-accounts.acl').absolutePath
|
||||
inputDirectory redisAclDirectory
|
||||
shouldRunAfter 'test'
|
||||
}
|
||||
}
|
||||
|
||||
tasks.named('mainCheck') {
|
||||
dependsOn 'redisContainerContractTest'
|
||||
}
|
||||
|
||||
@@ -50,7 +50,7 @@ io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnota
|
||||
io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor
|
||||
io.lettuce:lettuce-core:6.8.2.RELEASE=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-commons:1.16.7=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-core:1.16.7=testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-core:1.16.7=runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-jakarta9:1.16.7=testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-observation:1.16.7=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.netty:netty-buffer:4.2.17.Final=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
@@ -103,7 +103,7 @@ org.codehaus.plexus:plexus-container-default:2.1.0=checkstyle
|
||||
org.codehaus.plexus:plexus-utils:3.3.0=checkstyle
|
||||
org.dom4j:dom4j:2.2.0=spotbugs
|
||||
org.hamcrest:hamcrest:3.0=testCompileClasspath,testRuntimeClasspath
|
||||
org.hdrhistogram:HdrHistogram:2.2.2=testRuntimeClasspath
|
||||
org.hdrhistogram:HdrHistogram:2.2.2=runtimeClasspath,testRuntimeClasspath
|
||||
org.javassist:javassist:3.28.0-GA=checkstyle
|
||||
org.jetbrains:annotations:17.0.0=testCompileClasspath,testRuntimeClasspath
|
||||
org.jspecify:jspecify:1.0.1=annotationProcessor,checkstyle,compileClasspath,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
|
||||
@@ -116,7 +116,7 @@ org.junit.platform:junit-platform-engine:6.0.3=testRuntimeClasspath
|
||||
org.junit.platform:junit-platform-launcher:6.0.3=testRuntimeClasspath
|
||||
org.junit:junit-bom:6.0.3=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit:junit-bom:6.1.0=spotbugs
|
||||
org.latencyutils:LatencyUtils:2.0.3=testRuntimeClasspath
|
||||
org.latencyutils:LatencyUtils:2.0.3=runtimeClasspath,testRuntimeClasspath
|
||||
org.mockito:mockito-core:5.20.0=mockitoAgent,testCompileClasspath,testRuntimeClasspath
|
||||
org.mockito:mockito-junit-jupiter:5.20.0=testCompileClasspath,testRuntimeClasspath
|
||||
org.objenesis:objenesis:3.3=testRuntimeClasspath
|
||||
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
package dev.caskeleton.adapter.outbound.cache.redis.cache;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisAccount;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisLane;
|
||||
|
||||
/**
|
||||
* The cache's own script lane, so a cache's load cannot starve a correctness capability (review
|
||||
* 2026-11-03 README §4, RR-04-02).
|
||||
*
|
||||
* <p>Every conditional cache write is a registered program, and a registered program needs the
|
||||
* scripting account. On the runtime's SCRIPT lane those writes shared eight connections with
|
||||
* idempotency claims, rate-limit evaluations and lease calls — and the normal way to use the cache
|
||||
* (record with the condition the lookup returned) makes every cache fill a script. A slow Redis or
|
||||
* a burst of misses after a region invalidation then held those connections with cache work while
|
||||
* an idempotency claim waited out its timeout and failed. The cache is a speed feature and may
|
||||
* degrade; idempotency and leases are correctness features and must not. They get separate
|
||||
* connection budgets.
|
||||
*
|
||||
* <p>Same account as the SCRIPT lane: the grant is the same, only the connections and their ceiling
|
||||
* ({@code app.capabilities.cache.maximum-script-connections}) are the cache's. Alternatives that
|
||||
* were not taken: a per-capability permit cap inside the shared SCRIPT lane would still share the
|
||||
* connections and their reconnect cost after a discarded lease; running cache programs on the
|
||||
* REGULAR lane would grant the application account {@code EVALSHA}, which the account split exists
|
||||
* to withhold.
|
||||
*/
|
||||
public enum CacheScriptLane implements RedisLane {
|
||||
/** The cache region's conditional and revision-ordered writes. */
|
||||
CACHE_SCRIPT;
|
||||
|
||||
@Override
|
||||
public RedisAccount account() {
|
||||
return RedisAccount.ADVANCED;
|
||||
}
|
||||
}
|
||||
+5
-3
@@ -68,8 +68,10 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
|
||||
private final SemanticRedisCall call;
|
||||
|
||||
/**
|
||||
* The conditional writes, which are registered programs and so run on the SCRIPT lane: the
|
||||
* application account the REGULAR lane authenticates as is not granted {@code EVALSHA}.
|
||||
* The conditional writes, which are registered programs and so run on the scripting account: the
|
||||
* application account the REGULAR lane authenticates as is not granted {@code EVALSHA}. They run
|
||||
* on the cache's own lane ({@link CacheScriptLane}), not the runtime's SCRIPT lane, so a cache
|
||||
* under load cannot hold the connections idempotency, rate limiting and leases need (RR-04-02).
|
||||
*/
|
||||
private final SemanticRedisCall scriptCall;
|
||||
|
||||
@@ -133,7 +135,7 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
|
||||
Objects.requireNonNull(commandTimeout, "command timeout must be non-null"),
|
||||
"cache");
|
||||
this.scriptCall =
|
||||
new SemanticRedisCall(owner, RedisConnectionKind.SCRIPT, commandTimeout, "cache");
|
||||
new SemanticRedisCall(owner, CacheScriptLane.CACHE_SCRIPT, commandTimeout, "cache");
|
||||
this.keys = Objects.requireNonNull(keys, "keys must be non-null");
|
||||
this.keyDigest = Objects.requireNonNull(keyDigest, "key digest must be non-null");
|
||||
this.encoder = Objects.requireNonNull(encoder, "encoder must be non-null");
|
||||
|
||||
+22
@@ -1,6 +1,8 @@
|
||||
package dev.caskeleton.adapter.outbound.cache.redis.capability;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.cache.CacheScriptLane;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.cache.RedisCacheRegionAdapter;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisLaneContribution;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisRuntimeSettings;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisRuntimeOwner;
|
||||
import dev.caskeleton.adapter.outbound.support.CapabilitySelectors;
|
||||
@@ -56,4 +58,24 @@ class RedisCacheConfig {
|
||||
cache.getNegativeTtl(),
|
||||
cache.getCommandTimeout());
|
||||
}
|
||||
|
||||
/**
|
||||
* The cache's own script lane and its connection ceiling.
|
||||
*
|
||||
* <p>The runtime pools a contributed lane like its own SCRIPT lane, on the same scripting
|
||||
* account, but with connections and a ceiling of its own: the cache's conditional writes no
|
||||
* longer compete with idempotency claims, rate-limit evaluations and lease calls for the SCRIPT
|
||||
* lane (review 2026-11-03 README §4, RR-04-02).
|
||||
*
|
||||
* @param capabilities the capability settings
|
||||
* @return the lane contribution
|
||||
*/
|
||||
@Bean
|
||||
@ConditionalOnProperty(name = CapabilitySelectors.CACHE_DEFAULT_BINDING, havingValue = "redis")
|
||||
RedisLaneContribution redisCacheScriptLane(RedisCapabilitySettings capabilities) {
|
||||
RedisCapabilitySettings.Cache cache = capabilities.getCache();
|
||||
cache.validate();
|
||||
return new RedisLaneContribution(
|
||||
CacheScriptLane.CACHE_SCRIPT, cache.getMaximumScriptConnections());
|
||||
}
|
||||
}
|
||||
|
||||
+22
@@ -86,6 +86,7 @@ public class RedisCapabilitySettings {
|
||||
private Duration positiveHardTtl = Duration.ofMinutes(5);
|
||||
private Duration negativeTtl = Duration.ofSeconds(10);
|
||||
private Duration minimumHardTtl = Duration.ofSeconds(1);
|
||||
private int maximumScriptConnections = 4;
|
||||
|
||||
/**
|
||||
* Reports whether the default binding selected Redis.
|
||||
@@ -119,6 +120,10 @@ public class RedisCapabilitySettings {
|
||||
if (keyVersion < 1) {
|
||||
throw new CapabilityStartupValidationException("the cache key version must be positive");
|
||||
}
|
||||
if (maximumScriptConnections < 1) {
|
||||
throw new CapabilityStartupValidationException(
|
||||
"the cache script lane needs at least one connection");
|
||||
}
|
||||
}
|
||||
|
||||
public Map<String, String> getBindings() {
|
||||
@@ -192,6 +197,23 @@ public class RedisCapabilitySettings {
|
||||
public void setMinimumHardTtl(Duration minimumHardTtl) {
|
||||
this.minimumHardTtl = minimumHardTtl;
|
||||
}
|
||||
|
||||
/**
|
||||
* The ceiling on the cache's own script-lane connections.
|
||||
*
|
||||
* <p>Separate from {@code app.redis.capacity.maximum-script-connections}, which idempotency,
|
||||
* rate limiting and leases share: a cache is a speed feature and may run out of connections and
|
||||
* degrade; it must not take them from the correctness features (review 2026-11-03 README §4).
|
||||
*
|
||||
* @return the ceiling, at least 1
|
||||
*/
|
||||
public int getMaximumScriptConnections() {
|
||||
return maximumScriptConnections;
|
||||
}
|
||||
|
||||
public void setMaximumScriptConnections(int maximumScriptConnections) {
|
||||
this.maximumScriptConnections = maximumScriptConnections;
|
||||
}
|
||||
}
|
||||
|
||||
/** Edge rate-limit provider composition. */
|
||||
|
||||
+44
-9
@@ -28,11 +28,12 @@ import org.springframework.beans.factory.SmartInitializingSingleton;
|
||||
* {@code {1}} only when the value it wrote reads back.
|
||||
*
|
||||
* <p>A cache-only deployment is probed too: its conditional writes and revision-ordered writes are
|
||||
* registered programs on the same lane, and without the permission every one of them would come
|
||||
* registered programs on the same account, and without the permission every one of them would come
|
||||
* back degraded while plain reads looked healthy. Its probe runs only what the cache programs call
|
||||
* ({@code GET}, {@code SET}), and a Redis it cannot reach at startup is logged, not fatal — a cache
|
||||
* outage degrades, it never stops a pod from starting. A server that answers and refuses the
|
||||
* scripts is a configuration error and fails startup as it does for the correctness roles.
|
||||
* ({@code GET}, {@code SET}). Only a server that answers and refuses the scripts — an ACL refusal,
|
||||
* {@code NOSCRIPT}, or a reply other than the value the probe wrote — is a configuration error and
|
||||
* fails startup as it does for the correctness roles. Every other failure is a cache outage and is
|
||||
* logged, not fatal: a cache outage degrades, it never stops a pod from starting.
|
||||
*/
|
||||
final class RedisSemanticScriptAclProbe implements SmartInitializingSingleton {
|
||||
|
||||
@@ -89,7 +90,7 @@ final class RedisSemanticScriptAclProbe implements SmartInitializingSingleton {
|
||||
try {
|
||||
probe(CACHE_PROBE_SCRIPT);
|
||||
} catch (CapabilityStartupValidationException failure) {
|
||||
if (!unreachable(failure.getCause())) {
|
||||
if (refusedByTheServer(failure.getCause())) {
|
||||
throw new CapabilityStartupValidationException(
|
||||
"Redis cache is selected but its account cannot run the cache's conditional-write"
|
||||
+ " scripts (SCRIPT LOAD/EVALSHA, GET, SET); every conditional record would"
|
||||
@@ -98,15 +99,49 @@ final class RedisSemanticScriptAclProbe implements SmartInitializingSingleton {
|
||||
failure.getCause());
|
||||
}
|
||||
LOG.warn(
|
||||
"Redis cache is selected but Redis was unreachable at startup; the cache script"
|
||||
+ " permission is unverified and the cache runs degraded until Redis answers",
|
||||
"Redis cache is selected but the startup script probe could not complete; the cache"
|
||||
+ " script permission is unverified and the cache runs degraded until Redis answers",
|
||||
failure.getCause());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static boolean unreachable(Throwable cause) {
|
||||
return cause instanceof RedisLaneUnavailableException || cause instanceof TimeoutException;
|
||||
/**
|
||||
* Whether the cache probe failed because the server refused the cache's scripts.
|
||||
*
|
||||
* <p>"Refused" is a statement about the account, and only three answers make it: an ACL refusal
|
||||
* ({@code NOPERM}, or a script's inner command denied), {@code NOSCRIPT} for the program the same
|
||||
* connection had just loaded, and a reply other than the value the probe wrote. The check used to
|
||||
* run the other way — only a lane refusal or a timeout counted as "unreachable" and everything
|
||||
* else was a configuration error — so a connection that dropped mid-probe, a restarting Redis
|
||||
* answering {@code LOADING}, or a busy one answering {@code BUSY} stopped a cache-only pod from
|
||||
* starting, against this leaf's promise that a cache outage never does (review 2026-11-03
|
||||
* RR-04-07).
|
||||
*
|
||||
* @param cause the probe's failure
|
||||
* @return {@code true} when the account, not the server's availability, is at fault
|
||||
*/
|
||||
private static boolean refusedByTheServer(Throwable cause) {
|
||||
for (Throwable current = cause;
|
||||
current != null;
|
||||
current = current.getCause() == current ? null : current.getCause()) {
|
||||
if (current instanceof CapabilityStartupValidationException) {
|
||||
// The probe ran and read back something other than what it wrote.
|
||||
return true;
|
||||
}
|
||||
if (current instanceof RedisLaneUnavailableException || current instanceof TimeoutException) {
|
||||
return false;
|
||||
}
|
||||
String message = current.getMessage();
|
||||
if (message != null
|
||||
&& (message.startsWith("NOPERM")
|
||||
|| message.startsWith("NOSCRIPT")
|
||||
|| message.contains("no permissions")
|
||||
|| message.contains("ACL failure"))) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
private void probe(byte[] script) {
|
||||
|
||||
+10
@@ -141,6 +141,12 @@ public final class IdempotencyScripts {
|
||||
* it started from and the lease it asked for. A retry from that revision with the same lease is
|
||||
* the same renewal; with a different lease it is a different request, because confirming it would
|
||||
* leave the caller believing it holds the record for a lease the record does not carry.
|
||||
*
|
||||
* <p>The key's own TTL is stretched to cover the new lease. Claim sets it to the replay TTL,
|
||||
* which may be shorter than a lease renewed past it; the key then expired while the holder was
|
||||
* still executing, and the next request got a fresh claim and ran the side effect again (review
|
||||
* 2026-11-03 RR-04-08). The TTL is only ever lengthened here — completion sets the replay TTL
|
||||
* again.
|
||||
*/
|
||||
private static final String RENEW =
|
||||
"""
|
||||
@@ -177,6 +183,10 @@ public final class IdempotencyScripts {
|
||||
redis.call('HSET', KEYS[1],
|
||||
'leaseUntil', nowMillis + tonumber(ARGV[4]), 'rev', rev + 1,
|
||||
'renewedFromRev', ARGV[2], 'renewedLease', ARGV[4])
|
||||
local ttl = redis.call('PTTL', KEYS[1])
|
||||
if ttl >= 0 and ttl < tonumber(ARGV[4]) then
|
||||
redis.call('PEXPIRE', KEYS[1], ARGV[4])
|
||||
end
|
||||
return {'APPLIED', attempt, rev + 1, owner, '', ''}
|
||||
""";
|
||||
|
||||
|
||||
+6
-9
@@ -39,6 +39,10 @@ app:
|
||||
# Floor on the hard TTL. Below it, entries expire faster than the round trip that wrote them:
|
||||
# every read misses and every miss writes.
|
||||
minimum-hard-ttl: ${APP_CACHE_REDIS_MINIMUM_HARD_TTL:1s}
|
||||
# The cache's own script-lane connections, apart from the SCRIPT lane that idempotency, rate
|
||||
# limiting and leases share (app.redis.capacity.maximum-script-connections). A cache under
|
||||
# load runs out of these and degrades; it cannot take the correctness features' connections.
|
||||
maximum-script-connections: ${APP_CACHE_REDIS_MAXIMUM_SCRIPT_CONNECTIONS:4}
|
||||
idempotency:
|
||||
# Redis store tuning only; the provider selector and lifecycle policy stay with the
|
||||
# application's idempotency executor.
|
||||
@@ -75,15 +79,8 @@ app:
|
||||
maximum-cost: ${APP_RATE_LIMIT_MAXIMUM_COST:10}
|
||||
cleanup-grace: ${APP_RATE_LIMIT_CLEANUP_GRACE:5s}
|
||||
|
||||
# redis-runtime 의 redisOptional 은 Redis 에 닿지 못하면 DEGRADED 를 보고한다. Boot 기본 순서
|
||||
# (DOWN, OUT_OF_SERVICE, UP, UNKNOWN)에 없는 상태는 DOWN 보다 앞에 정렬돼, 다른 contributor 가 DOWN
|
||||
# 이어도 전체 health 가 HTTP 200 DEGRADED 로 보였다(리뷰 05-23e). DEGRADED 를 DOWN 뒤, UP 앞에 둔다.
|
||||
# 이 키를 정하는 module-defaults 는 이 파일 하나뿐이다. 서비스가 순서를 바꾸면 DEGRADED 를 넣어야 한다.
|
||||
management:
|
||||
endpoint:
|
||||
health:
|
||||
status:
|
||||
order: DOWN,OUT_OF_SERVICE,DEGRADED,UP,UNKNOWN
|
||||
# DEGRADED 상태 순서(management.endpoint.health.status.order)는 그 상태를 내는 redis-runtime 이
|
||||
# 싣는다(config/module-defaults/redis-runtime.yml, 리뷰 2026-11-03 RR-04-10).
|
||||
|
||||
# /actuator/info 에 싣는 이 모듈의 상태(management.info.env). compose smoke 가 lane 이 실제로 켠
|
||||
# capability 를 이 값으로 확인한다. 위 스위치를 해석한 값이고, 기본값은 RedisCapabilityConfig 의
|
||||
|
||||
+4
@@ -1,5 +1,6 @@
|
||||
package dev.caskeleton.adapter.outbound.cache.redis;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.cache.CacheScriptLane;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.RedisTopologyEndpoint;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisCredentialResolver.RedisCredentials;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisRuntimeSettings;
|
||||
@@ -71,6 +72,9 @@ public final class LiveRedisRuntime {
|
||||
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
|
||||
limits.put(kind, 4);
|
||||
}
|
||||
// The cache region's own script lane, which the composition root contributes when the cache
|
||||
// binding is redis (RedisCacheConfig).
|
||||
limits.put(CacheScriptLane.CACHE_SCRIPT, 4);
|
||||
return new RedisRuntimeOwner(
|
||||
new RedisTopologyClientFactory(
|
||||
settings,
|
||||
|
||||
+1
-1
@@ -28,7 +28,7 @@ final class RedisTestContainer {
|
||||
static final String ACL_FILE_PROPERTY = "redis.container.acl-file";
|
||||
|
||||
private static final String IMAGE_PROPERTY = "ca.test.image.redis";
|
||||
private static final String ACL_IN_CONTAINER = "/etc/redis/acl/all-accounts.acl";
|
||||
private static final String ACL_IN_CONTAINER = "/etc/redis/acl/lane-fixture-accounts.acl";
|
||||
private static final int PORT = 6379;
|
||||
|
||||
private static GenericContainer<?> server;
|
||||
|
||||
+35
-1
@@ -60,8 +60,17 @@ class RedisCacheRegionAdapterTest {
|
||||
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
|
||||
limits.put(kind, 4);
|
||||
}
|
||||
return new RedisCacheRegionAdapter<>(
|
||||
limits.put(CacheScriptLane.CACHE_SCRIPT, 4);
|
||||
return cache(
|
||||
now,
|
||||
new RedisRuntimeOwner(client, limits, Duration.ofSeconds(1), Duration.ofSeconds(1)),
|
||||
cacheKeys);
|
||||
}
|
||||
|
||||
private RedisCacheRegionAdapter<String, String> cache(
|
||||
Instant now, RedisRuntimeOwner owner, RedisCacheRegionAdapter.CacheKeys cacheKeys) {
|
||||
return new RedisCacheRegionAdapter<>(
|
||||
owner,
|
||||
cacheKeys,
|
||||
key -> "digest-" + key,
|
||||
value -> value.getBytes(StandardCharsets.UTF_8),
|
||||
@@ -428,6 +437,31 @@ class RedisCacheRegionAdapterTest {
|
||||
.hasMessageContaining("ONLY_IF_SOURCE_REVISION_NEWER");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("멱등성·lease가 SCRIPT lane을 모두 잡고 있어도 캐시 조건부 기록은 캐시 전용 lane에서 기록된다")
|
||||
void aConditionalCacheWriteDoesNotWaitForTheCorrectnessScriptLane() {
|
||||
// given
|
||||
Map<RedisLane, Integer> limits = new java.util.LinkedHashMap<>();
|
||||
limits.put(RedisConnectionKind.REGULAR, 4);
|
||||
limits.put(RedisConnectionKind.SCRIPT, 1);
|
||||
limits.put(CacheScriptLane.CACHE_SCRIPT, 1);
|
||||
try (RedisRuntimeOwner owner =
|
||||
new RedisRuntimeOwner(
|
||||
StubRedisRuntimeClient.serving(gateway),
|
||||
limits,
|
||||
Duration.ofMillis(100),
|
||||
Duration.ofSeconds(1));
|
||||
var heldByAnIdempotencyClaim = owner.borrow(RedisConnectionKind.SCRIPT)) {
|
||||
RedisCacheRegionAdapter<String, String> cache = cache(T0, owner, keys);
|
||||
|
||||
// when
|
||||
CacheRecordOutcome outcome = cache.record("order-1", "payload", newerThan("2"));
|
||||
|
||||
// then
|
||||
assertThat(outcome).isEqualTo(CacheRecordOutcome.RECORDED);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("an unreachable Redis degrades: the cache is bypassed, nothing fails")
|
||||
void anUnreachableRedisDegrades() {
|
||||
|
||||
+1
@@ -23,6 +23,7 @@ class RedisCacheRegionContractTest extends CacheRegionContract {
|
||||
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
|
||||
limits.put(kind, 4);
|
||||
}
|
||||
limits.put(CacheScriptLane.CACHE_SCRIPT, 4);
|
||||
return new RedisCacheRegionAdapter<>(
|
||||
new RedisRuntimeOwner(
|
||||
StubRedisRuntimeClient.serving(new InMemorySemanticRedisCommands()),
|
||||
|
||||
+65
@@ -3,6 +3,9 @@ package dev.caskeleton.adapter.outbound.cache.redis.capability;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockito.Mockito.mock;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.cache.CacheScriptLane;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisLaneContribution;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisLaneMetrics;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisRuntimeAutoConfiguration;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisRuntimeOwner;
|
||||
import dev.caskeleton.adapter.outbound.support.startup.StartupFailureException;
|
||||
@@ -11,6 +14,8 @@ import dev.caskeleton.application.idempotency.IdempotencyStorePort;
|
||||
import dev.caskeleton.application.lease.DistributedLeasePort;
|
||||
import dev.caskeleton.application.secret.SecretSourcePort;
|
||||
import dev.caskeleton.shared.ratelimit.EdgeRateLimitPort;
|
||||
import io.micrometer.core.instrument.MeterRegistry;
|
||||
import io.micrometer.core.instrument.simple.SimpleMeterRegistry;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.util.Arrays;
|
||||
@@ -94,6 +99,66 @@ class RedisCapabilityCompositionTest {
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("Redis 캐시 binding은 설정한 상한으로 캐시 전용 script lane을 SCRIPT lane과 따로 기여한다")
|
||||
void theCacheBindingContributesItsOwnScriptLane() {
|
||||
runner
|
||||
.withPropertyValues(
|
||||
"app.capabilities.cache.bindings.default=redis",
|
||||
"app.capabilities.cache.maximum-script-connections=3")
|
||||
.run(
|
||||
context -> {
|
||||
// when
|
||||
List<RedisLaneContribution> contributions =
|
||||
List.copyOf(context.getBeansOfType(RedisLaneContribution.class).values());
|
||||
|
||||
// then
|
||||
assertThat(context).hasNotFailed();
|
||||
assertThat(contributions)
|
||||
.containsExactly(new RedisLaneContribution(CacheScriptLane.CACHE_SCRIPT, 3));
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("Redis 캐시 binding의 캐시 전용 lane은 SCRIPT lane과 따로 대기·거절·재연결 지표를 낸다")
|
||||
void theCacheScriptLaneHasMetersOfItsOwn() {
|
||||
runner
|
||||
.withPropertyValues("app.capabilities.cache.bindings.default=redis")
|
||||
.withBean(MeterRegistry.class, SimpleMeterRegistry::new)
|
||||
.run(
|
||||
context -> {
|
||||
// when
|
||||
MeterRegistry registry = context.getBean(MeterRegistry.class);
|
||||
|
||||
// then
|
||||
assertThat(context).hasNotFailed();
|
||||
for (String lane : List.of(CacheScriptLane.CACHE_SCRIPT.name(), "SCRIPT")) {
|
||||
assertThat(
|
||||
registry
|
||||
.find(RedisLaneMetrics.CONNECTIONS_PENDING)
|
||||
.tag("lane", lane)
|
||||
.gauge())
|
||||
.as("waiting borrows of lane %s", lane)
|
||||
.isNotNull();
|
||||
assertThat(
|
||||
registry
|
||||
.find(RedisLaneMetrics.ACQUIRE_REFUSED)
|
||||
.tag("lane", lane)
|
||||
.tag("reason", "saturated")
|
||||
.counter())
|
||||
.as("refusals of lane %s", lane)
|
||||
.isNotNull();
|
||||
assertThat(
|
||||
registry
|
||||
.find(RedisLaneMetrics.CONNECTIONS_OPENED)
|
||||
.tag("lane", lane)
|
||||
.counter())
|
||||
.as("connections opened on lane %s", lane)
|
||||
.isNotNull();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a rate limiter with no configured policy is refused at startup")
|
||||
void aRateLimiterWithoutPoliciesIsRefused() {
|
||||
|
||||
+79
@@ -191,6 +191,85 @@ class RedisSemanticScriptAclProbeTest {
|
||||
assertThat(failure).isNull();
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(
|
||||
strings = {
|
||||
"LOADING Redis is loading the dataset in memory",
|
||||
"Connection closed",
|
||||
"BUSY Redis is busy running a script"
|
||||
})
|
||||
@DisplayName("cache만 고른 배포는 probe 도중 연결이 끊기거나 Redis가 아직 준비 중이어도 권한 오류로 보지 않고 기동한다")
|
||||
void aCacheOnlyDeploymentStartsWhenTheProbeIsInterruptedByATransientFailure(String reply) {
|
||||
// given
|
||||
RedisCapabilitySettings capabilities = new RedisCapabilitySettings();
|
||||
capabilities.getCache().getBindings().put("default", "redis");
|
||||
RedisSemanticScriptAclProbe probe =
|
||||
new RedisSemanticScriptAclProbe(
|
||||
owner(gatewayFailingLoadWith(reply), new AtomicInteger()),
|
||||
capabilities,
|
||||
"prod:svc:shared:probe".getBytes(StandardCharsets.UTF_8),
|
||||
Duration.ofSeconds(1));
|
||||
|
||||
// when
|
||||
Throwable failure = catchThrowable(probe::afterSingletonsInstantiated);
|
||||
|
||||
// then
|
||||
assertThat(failure).isNull();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("cache만 고른 배포도 probe가 기대와 다른 답을 받으면 기동에서 실패한다")
|
||||
void aCacheOnlyDeploymentWhoseProbeReadsBackTheWrongValueFailsStartup() {
|
||||
// given
|
||||
RedisCapabilitySettings capabilities = new RedisCapabilitySettings();
|
||||
capabilities.getCache().getBindings().put("default", "redis");
|
||||
RedisSemanticScriptAclProbe probe =
|
||||
new RedisSemanticScriptAclProbe(
|
||||
owner(gatewayAnswering(List.of(0L)), new AtomicInteger()),
|
||||
capabilities,
|
||||
"prod:svc:shared:probe".getBytes(StandardCharsets.UTF_8),
|
||||
Duration.ofSeconds(1));
|
||||
|
||||
// when
|
||||
Throwable failure = catchThrowable(probe::afterSingletonsInstantiated);
|
||||
|
||||
// then
|
||||
assertThat(failure)
|
||||
.isInstanceOf(CapabilityStartupValidationException.class)
|
||||
.hasMessageContaining("cache");
|
||||
}
|
||||
|
||||
private static SemanticRedisCommands gatewayFailingLoadWith(String reply) {
|
||||
return (SemanticRedisCommands)
|
||||
Proxy.newProxyInstance(
|
||||
SemanticRedisCommands.class.getClassLoader(),
|
||||
new Class<?>[] {SemanticRedisCommands.class},
|
||||
(proxy, method, arguments) ->
|
||||
switch (method.getName()) {
|
||||
case "loadScript" -> CompletableFuture.failedFuture(new RuntimeException(reply));
|
||||
case "toString" -> "RedisSemanticScriptAclProbeTestGateway";
|
||||
default ->
|
||||
throw new UnsupportedOperationException(
|
||||
"unexpected gateway method: " + method.getName());
|
||||
});
|
||||
}
|
||||
|
||||
private static SemanticRedisCommands gatewayAnswering(List<Object> reply) {
|
||||
return (SemanticRedisCommands)
|
||||
Proxy.newProxyInstance(
|
||||
SemanticRedisCommands.class.getClassLoader(),
|
||||
new Class<?>[] {SemanticRedisCommands.class},
|
||||
(proxy, method, arguments) ->
|
||||
switch (method.getName()) {
|
||||
case "loadScript" -> CompletableFuture.completedFuture("probe-digest");
|
||||
case "evaluateRegisteredForList" -> CompletableFuture.completedFuture(reply);
|
||||
case "toString" -> "RedisSemanticScriptAclProbeTestGateway";
|
||||
default ->
|
||||
throw new UnsupportedOperationException(
|
||||
"unexpected gateway method: " + method.getName());
|
||||
});
|
||||
}
|
||||
|
||||
private static RedisRuntimeOwner owner(SemanticRedisCommands gateway, AtomicInteger opened) {
|
||||
RedisRuntimeClient client =
|
||||
new RedisRuntimeClient() {
|
||||
|
||||
+5
-16
@@ -71,26 +71,15 @@ public final class RoleBasedFileAccessPolicy implements FileAccessPolicyPort {
|
||||
if (permitted.stream().noneMatch(subject.roles()::contains)) {
|
||||
throw denied(FileserverErrorCode.ACCESS_DENIED, "subject does not hold a permitted role");
|
||||
}
|
||||
if (descriptor.isPresent() && !isOwnedBy(descriptor.get(), subject)) {
|
||||
// Ownership is FileAccessSubject.owns, the rule the port owns (owner decision 14): a file with
|
||||
// no recorded owner predates ownership and belongs to nobody on the data plane. An admin role
|
||||
// does not bypass it — acting on somebody else's file is what the audited management plane is
|
||||
// for, and its operations carry no descriptor.
|
||||
if (descriptor.isPresent() && !subject.owns(descriptor.get())) {
|
||||
throw denied(FileserverErrorCode.ACCESS_DENIED, "subject does not own the file");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the file was created by this caller, in this caller's tenant.
|
||||
*
|
||||
* <p>A file without a recorded owner predates ownership and belongs to nobody on the data plane.
|
||||
* An admin role does not bypass this: acting on somebody else's file is what the audited
|
||||
* management plane is for, and its operations carry no descriptor.
|
||||
*/
|
||||
private static boolean isOwnedBy(FileDescriptor descriptor, FileAccessSubject subject) {
|
||||
return descriptor
|
||||
.owner()
|
||||
.filter(owner -> owner.principalId().equals(subject.principalId()))
|
||||
.filter(owner -> owner.tenantId().equals(subject.tenant()))
|
||||
.isPresent();
|
||||
}
|
||||
|
||||
private static boolean isAnonymous(FileAccessSubject subject) {
|
||||
return subject.principalId().equals(FileAccessSubject.anonymous().principalId());
|
||||
}
|
||||
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
package dev.caskeleton.adapter.outbound.fileserver.platform.security;
|
||||
|
||||
import dev.caskeleton.application.fileserver.api.security.FileAccessPolicyPort;
|
||||
import dev.caskeleton.application.fileserver.api.security.FileAccessSubject;
|
||||
import dev.caskeleton.application.fileserver.testkit.FileAccessPolicyPortContract;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.Set;
|
||||
|
||||
/** The production policy held to the ownership contract its fake is held to (RR-07-06). */
|
||||
class RoleBasedFileAccessPolicyContractTest extends FileAccessPolicyPortContract {
|
||||
|
||||
private static final String READ = "ROLE_FILE_READ";
|
||||
private static final String WRITE = "ROLE_FILE_WRITE";
|
||||
|
||||
private final RoleBasedFileAccessPolicy policy =
|
||||
new RoleBasedFileAccessPolicy(Set.of(READ), Set.of(WRITE), Set.of("ROLE_FILE_ADMIN"), false);
|
||||
|
||||
@Override
|
||||
protected FileAccessPolicyPort policy() {
|
||||
return policy;
|
||||
}
|
||||
|
||||
@Override
|
||||
protected FileAccessSubject permittedSubject(String principalId, Optional<String> tenant) {
|
||||
return new FileAccessSubject(
|
||||
principalId,
|
||||
Set.of(READ, WRITE),
|
||||
tenant.map(value -> Map.of(FileAccessSubject.TENANT_ATTRIBUTE, value)).orElse(Map.of()));
|
||||
}
|
||||
}
|
||||
@@ -41,7 +41,11 @@ no reactive (`WebClient`/Reactor Netty) surface; it was removed on 2026-09-25 (d
|
||||
cancels the request at the deadline, the JDK transport bounds its exchange timeout by the time
|
||||
left, the experimental Jetty HTTP/3 transport aborts the request, and the caller gets
|
||||
`HttpDeadlineExceededException`. A streaming download's setup (up to the response headers) runs
|
||||
inside the same deadline; after hand-over each body gap is bounded by `read-idle`.
|
||||
inside the same deadline; after hand-over each body gap is bounded by `read-idle` and nothing else,
|
||||
on both transports — the JDK transport arms its own per-read timer for this, because the JDK client
|
||||
has no socket read timeout. A body that fails after hand-over throws the same stable exceptions as
|
||||
a buffered call (`HttpResponseTimeoutException` for a stalled body, `HttpResponseTruncatedException`
|
||||
for a cut one).
|
||||
- **Evidence.** Each attempt reports `NOT_SENT`, `SENT_NO_RESPONSE`, `RESPONSE_RECEIVED`, or
|
||||
`PARTIAL_RESPONSE`. `NOT_SENT` is only claimed when a stage failure proves it.
|
||||
- **Retry.** Never from the HTTP method alone. Idempotency, body replayability, evidence, deadline,
|
||||
@@ -53,7 +57,9 @@ no reactive (`WebClient`/Reactor Netty) surface; it was removed on 2026-09-25 (d
|
||||
whether the call throws them or returns them as a result (`RETURN_RESULT`). Any other 4xx (404,
|
||||
409, 422 …) is a healthy answer, even when it reaches the caller as `HttpRemoteErrorException`.
|
||||
Window, threshold and open-state wait are per profile under `clients[N].circuit-breaker.*`
|
||||
(defaults are Resilience4j's; see `docs/httpclient/configuration-reference.md`).
|
||||
(defaults are Resilience4j's; see `docs/httpclient/configuration-reference.md`). An attempt that
|
||||
the caller's own deadline (`HttpOperation.deadline()`) ended is not counted at all: the caller ran
|
||||
out of time, the upstream did not fail. An attempt the profile's `total-call` ended still counts.
|
||||
- **Resources.** Connections and buffers are reclaimed after success, error, decode failure, size
|
||||
rejection, and cancellation alike.
|
||||
- **Secrets.** No URL, query value, path variable, token, cookie, or idempotency key reaches a log,
|
||||
|
||||
+14
-1
@@ -21,22 +21,35 @@ import org.springframework.http.client.JdkClientHttpRequestFactory;
|
||||
* timeout is fixed per factory, so a factory is made per request with {@code min(response-header
|
||||
* timeout, remaining deadline)} from {@link AttemptDeadline}. The factory only holds references;
|
||||
* the client and the write executor are shared.
|
||||
*
|
||||
* <p>A streaming download cannot use that request. Its body is handed to the caller and may run for
|
||||
* minutes, but Spring's timer only stops when the body stream is closed, so every JDK download
|
||||
* longer than the response-header timeout was cut mid-body (review 2026-11-03 RR-04-03). Inside a
|
||||
* streaming setup scope the factory makes a {@link JdkStreamingClientHttpRequest} instead: the
|
||||
* JDK's own request timeout bounds the wait for headers, the attempt deadline cancels the exchange
|
||||
* up to hand-over, and the body is bounded per gap by {@code read-idle}.
|
||||
*/
|
||||
final class DeadlineAwareJdkRequestFactory implements ClientHttpRequestFactory {
|
||||
|
||||
private final HttpClient client;
|
||||
private final Executor writeExecutor;
|
||||
private final Duration exchangeTimeout;
|
||||
private final Duration readIdle;
|
||||
|
||||
DeadlineAwareJdkRequestFactory(HttpClient client, Duration exchangeTimeout) {
|
||||
DeadlineAwareJdkRequestFactory(HttpClient client, Duration exchangeTimeout, Duration readIdle) {
|
||||
this.client = Objects.requireNonNull(client, "jdk http client");
|
||||
this.exchangeTimeout = Objects.requireNonNull(exchangeTimeout, "exchange timeout");
|
||||
this.readIdle = Objects.requireNonNull(readIdle, "read idle timeout");
|
||||
// The executor Spring would pick itself, chosen once instead of once per request.
|
||||
this.writeExecutor = client.executor().orElseGet(SimpleAsyncTaskExecutor::new);
|
||||
}
|
||||
|
||||
@Override
|
||||
public ClientHttpRequest createRequest(URI uri, HttpMethod httpMethod) throws IOException {
|
||||
if (AttemptDeadline.boundsSetupOnly()) {
|
||||
return new JdkStreamingClientHttpRequest(
|
||||
client, uri, httpMethod, AttemptDeadline.bound(exchangeTimeout), readIdle);
|
||||
}
|
||||
JdkClientHttpRequestFactory delegate = new JdkClientHttpRequestFactory(client, writeExecutor);
|
||||
delegate.setReadTimeout(AttemptDeadline.bound(exchangeTimeout));
|
||||
return delegate.createRequest(uri, httpMethod);
|
||||
|
||||
+2
-1
@@ -62,7 +62,8 @@ public final class JdkBlockingTransportProvider implements BlockingTransportProv
|
||||
HttpClient client = clientFactory.create(profile, tlsMaterialResolver.apply(profile));
|
||||
clients.put(new TransportResourceKey(profile.name(), generation), client);
|
||||
ClientHttpRequestFactory factory =
|
||||
new DeadlineAwareJdkRequestFactory(client, profile.timeout().responseHeader());
|
||||
new DeadlineAwareJdkRequestFactory(
|
||||
client, profile.timeout().responseHeader(), profile.timeout().readIdle());
|
||||
listener.onRuntimeCreated(profile.name(), ID);
|
||||
return factory;
|
||||
}
|
||||
|
||||
+128
@@ -0,0 +1,128 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.jdk;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.transport.AttemptDeadline;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.InterruptedIOException;
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.time.Duration;
|
||||
import java.util.Objects;
|
||||
import java.util.Set;
|
||||
import java.util.TreeSet;
|
||||
import java.util.concurrent.CancellationException;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.ExecutionException;
|
||||
import org.springframework.http.HttpHeaders;
|
||||
import org.springframework.http.HttpMethod;
|
||||
import org.springframework.http.client.AbstractBufferingClientHttpRequest;
|
||||
import org.springframework.http.client.ClientHttpResponse;
|
||||
|
||||
/**
|
||||
* A JDK request for a streaming download, whose timeouts end at the response headers (review
|
||||
* 2026-11-03 RR-04-03).
|
||||
*
|
||||
* <p>Three limits, each where the JDK can enforce it:
|
||||
*
|
||||
* <ul>
|
||||
* <li>the JDK's own request timeout bounds the wait for the response headers and stops when they
|
||||
* arrive;
|
||||
* <li>the attempt deadline cancels the pending exchange, so connect, send and the header wait
|
||||
* stay inside the call's total budget;
|
||||
* <li>after hand-over, {@link ReadIdleInputStream} bounds each gap of the body by {@code
|
||||
* read-idle} — the JDK client has no socket read timeout of its own.
|
||||
* </ul>
|
||||
*
|
||||
* <p>The request body is buffered: a streaming download sends none or a small one, and the JDK
|
||||
* needs a body publisher before the exchange starts.
|
||||
*/
|
||||
final class JdkStreamingClientHttpRequest extends AbstractBufferingClientHttpRequest {
|
||||
|
||||
/** Headers the JDK client refuses to set, as Spring's own JDK request lists them. */
|
||||
private static final Set<String> RESTRICTED_HEADERS = restrictedHeaders();
|
||||
|
||||
private final HttpClient client;
|
||||
private final URI uri;
|
||||
private final HttpMethod method;
|
||||
private final Duration responseHeaderTimeout;
|
||||
private final Duration readIdle;
|
||||
|
||||
JdkStreamingClientHttpRequest(
|
||||
HttpClient client,
|
||||
URI uri,
|
||||
HttpMethod method,
|
||||
Duration responseHeaderTimeout,
|
||||
Duration readIdle) {
|
||||
this.client = Objects.requireNonNull(client, "jdk http client");
|
||||
this.uri = Objects.requireNonNull(uri, "uri");
|
||||
this.method = Objects.requireNonNull(method, "method");
|
||||
this.responseHeaderTimeout =
|
||||
Objects.requireNonNull(responseHeaderTimeout, "response header timeout");
|
||||
this.readIdle = Objects.requireNonNull(readIdle, "read idle timeout");
|
||||
}
|
||||
|
||||
@Override
|
||||
public HttpMethod getMethod() {
|
||||
return method;
|
||||
}
|
||||
|
||||
@Override
|
||||
public URI getURI() {
|
||||
return uri;
|
||||
}
|
||||
|
||||
@Override
|
||||
protected ClientHttpResponse executeInternal(HttpHeaders headers, byte[] body)
|
||||
throws IOException {
|
||||
HttpRequest.Builder request =
|
||||
HttpRequest.newBuilder(uri)
|
||||
.timeout(responseHeaderTimeout)
|
||||
.method(
|
||||
method.name(),
|
||||
body.length == 0
|
||||
? HttpRequest.BodyPublishers.noBody()
|
||||
: HttpRequest.BodyPublishers.ofByteArray(body));
|
||||
headers.forEach(
|
||||
(name, values) -> {
|
||||
if (!RESTRICTED_HEADERS.contains(name)) {
|
||||
values.forEach(value -> request.header(name, value));
|
||||
}
|
||||
});
|
||||
CompletableFuture<HttpResponse<InputStream>> pending =
|
||||
client.sendAsync(request.build(), HttpResponse.BodyHandlers.ofInputStream());
|
||||
AttemptDeadline.abortOnExpiry(() -> pending.cancel(true));
|
||||
HttpResponse<InputStream> response = await(pending);
|
||||
return new JdkStreamingClientHttpResponse(
|
||||
response, new ReadIdleInputStream(response.body(), readIdle));
|
||||
}
|
||||
|
||||
private static HttpResponse<InputStream> await(
|
||||
CompletableFuture<HttpResponse<InputStream>> pending) throws IOException {
|
||||
try {
|
||||
return pending.get();
|
||||
} catch (InterruptedException interrupted) {
|
||||
Thread.currentThread().interrupt();
|
||||
pending.cancel(true);
|
||||
InterruptedIOException failure = new InterruptedIOException("streaming setup interrupted");
|
||||
failure.initCause(interrupted);
|
||||
throw failure;
|
||||
} catch (CancellationException cancelled) {
|
||||
// Only the attempt deadline cancels this future; the gateway names the deadline.
|
||||
throw new IOException("streaming setup cancelled at the attempt deadline", cancelled);
|
||||
} catch (ExecutionException failed) {
|
||||
Throwable cause = failed.getCause();
|
||||
if (cause instanceof IOException io) {
|
||||
throw io;
|
||||
}
|
||||
throw new IOException("streaming setup failed", cause);
|
||||
}
|
||||
}
|
||||
|
||||
private static Set<String> restrictedHeaders() {
|
||||
Set<String> restricted = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
|
||||
restricted.addAll(Set.of("connection", "content-length", "expect", "host", "upgrade"));
|
||||
return restricted;
|
||||
}
|
||||
}
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.jdk;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.util.Objects;
|
||||
import org.springframework.http.HttpHeaders;
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.http.HttpStatusCode;
|
||||
import org.springframework.http.client.ClientHttpResponse;
|
||||
|
||||
/** The response of a {@link JdkStreamingClientHttpRequest}, body bounded per gap. */
|
||||
final class JdkStreamingClientHttpResponse implements ClientHttpResponse {
|
||||
|
||||
private final HttpResponse<InputStream> response;
|
||||
private final InputStream body;
|
||||
private final HttpHeaders headers = new HttpHeaders();
|
||||
|
||||
JdkStreamingClientHttpResponse(HttpResponse<InputStream> response, InputStream body) {
|
||||
this.response = Objects.requireNonNull(response, "jdk response");
|
||||
this.body = Objects.requireNonNull(body, "body stream");
|
||||
// HTTP/2 pseudo-headers (":status") are protocol framing, not response headers.
|
||||
response
|
||||
.headers()
|
||||
.map()
|
||||
.forEach(
|
||||
(name, values) -> {
|
||||
if (!name.startsWith(":")) {
|
||||
headers.addAll(name, values);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@Override
|
||||
public HttpStatusCode getStatusCode() {
|
||||
return HttpStatusCode.valueOf(response.statusCode());
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getStatusText() {
|
||||
HttpStatus known = HttpStatus.resolve(response.statusCode());
|
||||
return known == null ? "" : known.getReasonPhrase();
|
||||
}
|
||||
|
||||
@Override
|
||||
public HttpHeaders getHeaders() {
|
||||
return headers;
|
||||
}
|
||||
|
||||
@Override
|
||||
public InputStream getBody() {
|
||||
return body;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
try {
|
||||
body.close();
|
||||
} catch (IOException ignored) {
|
||||
// Closing releases the exchange; a failure to close has nothing left to release.
|
||||
}
|
||||
}
|
||||
}
|
||||
+90
@@ -0,0 +1,90 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.jdk;
|
||||
|
||||
import java.io.FilterInputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.net.SocketTimeoutException;
|
||||
import java.time.Duration;
|
||||
import java.util.Objects;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
|
||||
/**
|
||||
* A JDK response body whose every read is bounded by {@code read-idle} (review 2026-11-03
|
||||
* RR-04-03).
|
||||
*
|
||||
* <p>The JDK client has no socket read timeout, so a streamed body that stopped arriving held the
|
||||
* caller's thread and the connection forever. Each read arms a timer on the JDK's shared delay
|
||||
* scheduler; when it fires the stream is closed, which wakes the blocked read, and the read reports
|
||||
* a {@link SocketTimeoutException} — the same exception Apache's socket timeout raises, so one
|
||||
* classifier reading names both. A read that returns first disarms its timer.
|
||||
*/
|
||||
final class ReadIdleInputStream extends FilterInputStream {
|
||||
|
||||
@FunctionalInterface
|
||||
private interface Read {
|
||||
long run() throws IOException;
|
||||
}
|
||||
|
||||
private final Duration readIdle;
|
||||
private final AtomicBoolean idled = new AtomicBoolean();
|
||||
|
||||
ReadIdleInputStream(InputStream delegate, Duration readIdle) {
|
||||
super(Objects.requireNonNull(delegate, "body stream"));
|
||||
this.readIdle = Objects.requireNonNull(readIdle, "read idle timeout");
|
||||
}
|
||||
|
||||
@Override
|
||||
public int read() throws IOException {
|
||||
return (int) guarded(super::read);
|
||||
}
|
||||
|
||||
@Override
|
||||
public int read(byte[] buffer, int offset, int length) throws IOException {
|
||||
return (int) guarded(() -> super.read(buffer, offset, length));
|
||||
}
|
||||
|
||||
@Override
|
||||
public long skip(long count) throws IOException {
|
||||
return guarded(() -> super.skip(count));
|
||||
}
|
||||
|
||||
private long guarded(Read read) throws IOException {
|
||||
requireNotIdled();
|
||||
CompletableFuture<Void> timer = new CompletableFuture<>();
|
||||
CompletableFuture<Void> expiry =
|
||||
timer.completeOnTimeout(null, readIdle.toNanos(), TimeUnit.NANOSECONDS).thenRun(this::idle);
|
||||
try {
|
||||
long result = read.run();
|
||||
requireNotIdled();
|
||||
return result;
|
||||
} catch (IOException failure) {
|
||||
requireNotIdled();
|
||||
throw failure;
|
||||
} finally {
|
||||
// Completing first skips the idle action and lets the scheduler drop the pending task.
|
||||
timer.cancel(false);
|
||||
expiry.cancel(false);
|
||||
}
|
||||
}
|
||||
|
||||
private void idle() {
|
||||
idled.set(true);
|
||||
try {
|
||||
in.close();
|
||||
} catch (IOException ignored) {
|
||||
// The read this wakes reports the timeout; a failed close has nothing more to say.
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* After the timer fired, whatever the read returned — data, end of stream or "closed" — is not
|
||||
* the rest of the body.
|
||||
*/
|
||||
private void requireNotIdled() throws SocketTimeoutException {
|
||||
if (idled.get()) {
|
||||
throw new SocketTimeoutException("no response body bytes within the read-idle timeout");
|
||||
}
|
||||
}
|
||||
}
|
||||
+17
@@ -17,6 +17,13 @@ public interface AttemptCircuitBreaker {
|
||||
|
||||
void onError(long durationNanos, Throwable failure);
|
||||
|
||||
/**
|
||||
* Gives back a permission without recording an outcome, for an attempt that says nothing about
|
||||
* the upstream (an attempt its caller's own deadline ended). Half-open trial slots are returned;
|
||||
* the window is left as it was.
|
||||
*/
|
||||
void releasePermission();
|
||||
|
||||
String state();
|
||||
|
||||
static AttemptCircuitBreaker alwaysClosed() {
|
||||
@@ -36,6 +43,11 @@ public interface AttemptCircuitBreaker {
|
||||
// Nothing to record.
|
||||
}
|
||||
|
||||
@Override
|
||||
public void releasePermission() {
|
||||
// Nothing was taken.
|
||||
}
|
||||
|
||||
@Override
|
||||
public String state() {
|
||||
return "CLOSED";
|
||||
@@ -60,6 +72,11 @@ public interface AttemptCircuitBreaker {
|
||||
delegate.onError(durationNanos, TimeUnit.NANOSECONDS, failure);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void releasePermission() {
|
||||
delegate.releasePermission();
|
||||
}
|
||||
|
||||
@Override
|
||||
public String state() {
|
||||
return delegate.getState().name();
|
||||
|
||||
+28
-1
@@ -7,6 +7,7 @@ import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpFailureMetadata;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRateLimitRejectedException;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
import java.util.function.BooleanSupplier;
|
||||
import java.util.function.Function;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
@@ -21,6 +22,11 @@ import java.util.function.Supplier;
|
||||
* <p>A local rejection (rate limiter or bulkhead) is deliberately <em>not</em> recorded as a
|
||||
* circuit error: the upstream never saw the request, and counting our own back-pressure as upstream
|
||||
* failure would open the breaker on a healthy dependency.
|
||||
*
|
||||
* <p>An attempt its caller abandoned is not recorded either, for the same reason: an attempt that a
|
||||
* caller's own deadline ended while the upstream was still answering within the profile's limits is
|
||||
* the caller running out of time, not the upstream failing (review 2026-11-03 RR-04-01). Its
|
||||
* permission is given back instead.
|
||||
*/
|
||||
public final class AttemptResiliencePipeline {
|
||||
|
||||
@@ -79,7 +85,24 @@ public final class AttemptResiliencePipeline {
|
||||
* @return the attempt's value, whether or not it represents a remote failure
|
||||
*/
|
||||
public <T> T execute(AttemptCall<T> call, Function<T, Optional<Throwable>> remoteFailure) {
|
||||
return execute(call, remoteFailure, () -> false);
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs one attempt, recording nothing against the upstream when its caller abandoned it.
|
||||
*
|
||||
* @param call the attempt, including any redirect hops it follows
|
||||
* @param remoteFailure returns the failure to record when the value represents an upstream error
|
||||
* @param abandonedByCaller read after the attempt failed: whether the caller's own deadline ended
|
||||
* it, in which case the failure is not the upstream's and the breaker records nothing
|
||||
* @return the attempt's value, whether or not it represents a remote failure
|
||||
*/
|
||||
public <T> T execute(
|
||||
AttemptCall<T> call,
|
||||
Function<T, Optional<Throwable>> remoteFailure,
|
||||
BooleanSupplier abandonedByCaller) {
|
||||
Objects.requireNonNull(remoteFailure, "remote failure classifier");
|
||||
Objects.requireNonNull(abandonedByCaller, "abandoned-by-caller signal");
|
||||
if (!circuitBreaker.tryAcquirePermission()) {
|
||||
rejections.circuitOpen();
|
||||
throw new HttpCircuitOpenException(
|
||||
@@ -110,7 +133,11 @@ public final class AttemptResiliencePipeline {
|
||||
return result;
|
||||
} catch (Throwable failure) {
|
||||
releaseAttemptPermits();
|
||||
circuitBreaker.onError(System.nanoTime() - started, failure);
|
||||
if (abandonedByCaller.getAsBoolean()) {
|
||||
circuitBreaker.releasePermission();
|
||||
} else {
|
||||
circuitBreaker.onError(System.nanoTime() - started, failure);
|
||||
}
|
||||
throw translate(failure);
|
||||
}
|
||||
}
|
||||
|
||||
+26
-1
@@ -10,11 +10,31 @@ import java.util.Objects;
|
||||
*
|
||||
* <p>Everything — pool acquire, DNS, connect, TLS, request write, response read, and every retry
|
||||
* backoff — is spent inside this single budget.
|
||||
*
|
||||
* @param at the instant the call ends
|
||||
* @param source who set that instant, which decides what an attempt it ends says about the upstream
|
||||
*/
|
||||
public record Deadline(Instant at) {
|
||||
public record Deadline(Instant at, Source source) {
|
||||
|
||||
/**
|
||||
* Who set the instant a call ends at.
|
||||
*
|
||||
* <p>The two say different things when they end an attempt. The profile's total-call timeout is
|
||||
* the operator's statement of how long a healthy upstream may take, so an attempt it ends is the
|
||||
* upstream's failure. A caller's own deadline is a statement about the caller — it had only that
|
||||
* much time left — so an attempt it ends says nothing about the upstream (review 2026-11-03
|
||||
* RR-04-01).
|
||||
*/
|
||||
public enum Source {
|
||||
/** The profile's {@code timeout.total-call}. */
|
||||
PROFILE_TOTAL_CALL,
|
||||
/** The caller's {@code HttpOperation.deadline()}, when it is earlier than the profile's. */
|
||||
CALLER
|
||||
}
|
||||
|
||||
public Deadline {
|
||||
Objects.requireNonNull(at, "deadline instant");
|
||||
Objects.requireNonNull(source, "deadline source");
|
||||
}
|
||||
|
||||
public Duration remaining(Clock clock) {
|
||||
@@ -26,6 +46,11 @@ public record Deadline(Instant at) {
|
||||
return !clock.instant().isBefore(at);
|
||||
}
|
||||
|
||||
/** Whether the instant is the caller's own budget rather than the profile's total-call. */
|
||||
public boolean setByCaller() {
|
||||
return source == Source.CALLER;
|
||||
}
|
||||
|
||||
/** The earlier of two deadlines; a parent budget can only ever shrink a child's. */
|
||||
public Deadline earliest(Deadline other) {
|
||||
return at.isBefore(other.at) ? this : other;
|
||||
|
||||
+7
-1
@@ -9,6 +9,9 @@ import java.util.Optional;
|
||||
/**
|
||||
* Derives the effective deadline from the caller's parent budget and the profile total-call timeout
|
||||
* (design §15.2): {@code effectiveDeadline = min(parentDeadline, now + totalCallTimeout)}.
|
||||
*
|
||||
* <p>The result also records which of the two won. A tie goes to the profile: the profile's limit
|
||||
* would have ended the call at that instant anyway.
|
||||
*/
|
||||
public final class DeadlineCalculator {
|
||||
|
||||
@@ -17,6 +20,9 @@ public final class DeadlineCalculator {
|
||||
Objects.requireNonNull(totalCall, "total call timeout");
|
||||
Objects.requireNonNull(clock, "clock");
|
||||
Instant local = clock.instant().plus(totalCall);
|
||||
return new Deadline(parent.map(value -> value.isBefore(local) ? value : local).orElse(local));
|
||||
return parent
|
||||
.filter(value -> value.isBefore(local))
|
||||
.map(value -> new Deadline(value, Deadline.Source.CALLER))
|
||||
.orElseGet(() -> new Deadline(local, Deadline.Source.PROFILE_TOTAL_CALL));
|
||||
}
|
||||
}
|
||||
|
||||
+8
-1
@@ -63,6 +63,7 @@ public final class BlockingAttemptExecutor {
|
||||
try (AttemptDeadline.Scope scope = AttemptDeadline.open(deadline.remaining(support.clock()))) {
|
||||
return executeWithin(
|
||||
scope,
|
||||
deadline,
|
||||
runtime,
|
||||
prepared,
|
||||
responseType,
|
||||
@@ -76,6 +77,7 @@ public final class BlockingAttemptExecutor {
|
||||
|
||||
private <T> AttemptOutcome<T> executeWithin(
|
||||
AttemptDeadline.Scope scope,
|
||||
Deadline deadline,
|
||||
BlockingClientRuntime runtime,
|
||||
PreparedOperation prepared,
|
||||
ResponseType<T> responseType,
|
||||
@@ -146,7 +148,12 @@ public final class BlockingAttemptExecutor {
|
||||
tracker.firstByteDelivered());
|
||||
}
|
||||
},
|
||||
outcome -> upstreamFailure(outcome, baseMetadata));
|
||||
outcome -> upstreamFailure(outcome, baseMetadata),
|
||||
// The caller's own deadline ending the attempt is not the upstream failing. The
|
||||
// profile's total-call ending it is: that limit is the operator's statement of
|
||||
// how long a healthy upstream may take, and a slow-drip upstream that never trips
|
||||
// a per-gap timeout is seen by the breaker only through it (RR-04-01).
|
||||
() -> deadline.setByCaller() && scope.expired());
|
||||
return finish(observation, scope, attemptOutcome);
|
||||
} catch (HttpClientException stable) {
|
||||
return finish(
|
||||
|
||||
+22
-4
@@ -5,6 +5,7 @@ import dev.caskeleton.adapter.outbound.httpclient.api.HttpStatus;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpDeadlineExceededException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpFailureMetadata;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRemoteErrorException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.AttemptStage;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.HttpOperation;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.result.BlockingStreamingResponse;
|
||||
@@ -78,9 +79,11 @@ public final class BlockingStreamingGateway {
|
||||
// which deregisters the transport's abort, so the body is then bounded by read-idle per gap
|
||||
// and a long download is not cut at an arbitrary total. Without the scope a setup was bounded
|
||||
// only by per-gap stage timeouts, and an upstream that held or dripped its headers kept the
|
||||
// caller past its deadline (finding 05-06).
|
||||
// caller past its deadline (finding 05-06). It is a setup scope, not an attempt scope: the
|
||||
// JDK transport arms an exchange-wide timeout for an attempt, which cut every download
|
||||
// longer than the response-header timeout (RR-04-03).
|
||||
AttemptDeadline.Scope setupDeadline =
|
||||
AttemptDeadline.open(setup.deadline().remaining(runtime.support().clock()));
|
||||
AttemptDeadline.openStreamSetup(setup.deadline().remaining(runtime.support().clock()));
|
||||
try (setupDeadline) {
|
||||
response =
|
||||
spec.exchange(
|
||||
@@ -102,9 +105,24 @@ public final class BlockingStreamingGateway {
|
||||
.withStatus(new HttpStatus(status))
|
||||
.withEvidence(ExecutionEvidence.RESPONSE_RECEIVED));
|
||||
}
|
||||
// A body that fails after hand-over reaches the caller as the same stable
|
||||
// exceptions a buffered call throws, classified by the transport's own
|
||||
// classifier at the body stage — not as the engine's bare IOException, which
|
||||
// carried no category, no evidence and nothing to log (RR-04-03).
|
||||
InputStream bounded =
|
||||
new CountingBoundedInputStream(
|
||||
rawResponse.getBody(), limiter::recordWireBytes);
|
||||
new ClassifiedBodyInputStream(
|
||||
new CountingBoundedInputStream(
|
||||
rawResponse.getBody(), limiter::recordWireBytes),
|
||||
bodyFailure ->
|
||||
runtime
|
||||
.support()
|
||||
.exceptionMapper()
|
||||
.map(
|
||||
runtime
|
||||
.failureClassifier()
|
||||
.classify(bodyFailure, AttemptStage.RESPONSE_BODY),
|
||||
metadata,
|
||||
bodyFailure));
|
||||
return new DefaultBlockingStreamingResponse(
|
||||
new HttpStatus(status), responseHeaders, bounded, rawResponse::close);
|
||||
},
|
||||
|
||||
+64
@@ -0,0 +1,64 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.restclient;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpClientException;
|
||||
import java.io.FilterInputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.util.Objects;
|
||||
import java.util.function.Function;
|
||||
|
||||
/**
|
||||
* A handed-over response body whose read failures reach the caller as stable exceptions (review
|
||||
* 2026-11-03 RR-04-03).
|
||||
*
|
||||
* <p>A streamed body is read after the gateway has returned, so the gateway's own failure mapping
|
||||
* never sees it. Without this a read-idle timeout or a cut connection reached the caller as the
|
||||
* engine's bare {@code IOException} — no category, no evidence, nothing a retry policy or a log
|
||||
* could branch on. Closing is the caller's own act and is passed through unclassified.
|
||||
*/
|
||||
final class ClassifiedBodyInputStream extends FilterInputStream {
|
||||
|
||||
private final Function<IOException, HttpClientException> classifier;
|
||||
|
||||
ClassifiedBodyInputStream(
|
||||
InputStream delegate, Function<IOException, HttpClientException> classifier) {
|
||||
super(Objects.requireNonNull(delegate, "body stream"));
|
||||
this.classifier = Objects.requireNonNull(classifier, "body failure classifier");
|
||||
}
|
||||
|
||||
@Override
|
||||
public int read() throws IOException {
|
||||
try {
|
||||
return super.read();
|
||||
} catch (IOException failure) {
|
||||
throw classifier.apply(failure);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public int read(byte[] buffer, int offset, int length) throws IOException {
|
||||
try {
|
||||
return super.read(buffer, offset, length);
|
||||
} catch (IOException failure) {
|
||||
throw classifier.apply(failure);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public long skip(long count) throws IOException {
|
||||
try {
|
||||
return super.skip(count);
|
||||
} catch (IOException failure) {
|
||||
throw classifier.apply(failure);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public int available() throws IOException {
|
||||
try {
|
||||
return super.available();
|
||||
} catch (IOException failure) {
|
||||
throw classifier.apply(failure);
|
||||
}
|
||||
}
|
||||
}
|
||||
+35
-2
@@ -28,6 +28,11 @@ import java.util.concurrent.TimeUnit;
|
||||
* <p>The scope is thread-confined because a blocking attempt runs on the caller's thread from
|
||||
* request creation to the last body byte. When no scope is open — a transport used outside the
|
||||
* gateway — both methods leave the transport's own behaviour unchanged.
|
||||
*
|
||||
* <p>A streaming download opens a different scope ({@link #openStreamSetup(Duration)}): it bounds
|
||||
* the exchange only up to the response headers and is closed when the body is handed over. A
|
||||
* transport whose engine timeout would otherwise keep running over the body reads {@link
|
||||
* #boundsSetupOnly()} and keeps that timeout off it (review 2026-11-03 RR-04-03).
|
||||
*/
|
||||
public final class AttemptDeadline {
|
||||
|
||||
@@ -41,8 +46,34 @@ public final class AttemptDeadline {
|
||||
* @param remaining the time left before the logical call's effective deadline
|
||||
*/
|
||||
public static Scope open(Duration remaining) {
|
||||
return open(remaining, false);
|
||||
}
|
||||
|
||||
/**
|
||||
* Opens the deadline scope for a streaming download's setup on the current thread.
|
||||
*
|
||||
* <p>The caller closes it when the response is handed over, so the deadline never reaches the
|
||||
* body: a download may legitimately outlast any fixed total (design §15.3).
|
||||
*
|
||||
* @param remaining the time left before the logical call's effective deadline
|
||||
*/
|
||||
public static Scope openStreamSetup(Duration remaining) {
|
||||
return open(remaining, true);
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the current scope bounds only a streaming setup, so a timeout the transport arms must
|
||||
* not outlive the response headers.
|
||||
*/
|
||||
public static boolean boundsSetupOnly() {
|
||||
Scope scope = CURRENT.get();
|
||||
return scope != null && scope.setupOnly;
|
||||
}
|
||||
|
||||
private static Scope open(Duration remaining, boolean setupOnly) {
|
||||
Objects.requireNonNull(remaining, "remaining deadline");
|
||||
Scope scope = new Scope(remaining.isNegative() ? Duration.ZERO : remaining, CURRENT.get());
|
||||
Scope scope =
|
||||
new Scope(remaining.isNegative() ? Duration.ZERO : remaining, setupOnly, CURRENT.get());
|
||||
CURRENT.set(scope);
|
||||
return scope;
|
||||
}
|
||||
@@ -78,6 +109,7 @@ public final class AttemptDeadline {
|
||||
public static final class Scope implements AutoCloseable {
|
||||
|
||||
private final long endNanos;
|
||||
private final boolean setupOnly;
|
||||
private final Scope outer;
|
||||
private final CompletableFuture<Void> timer = new CompletableFuture<>();
|
||||
private final CompletableFuture<Void> expiry;
|
||||
@@ -85,8 +117,9 @@ public final class AttemptDeadline {
|
||||
private boolean fired;
|
||||
private boolean closed;
|
||||
|
||||
private Scope(Duration remaining, Scope outer) {
|
||||
private Scope(Duration remaining, boolean setupOnly, Scope outer) {
|
||||
this.endNanos = System.nanoTime() + remaining.toNanos();
|
||||
this.setupOnly = setupOnly;
|
||||
this.outer = outer;
|
||||
// The JDK's shared delay scheduler, the same one Spring's JDK request uses for its timeout.
|
||||
// Completing the future on close() also cancels the scheduled task, so a finished attempt
|
||||
|
||||
+6
-3
@@ -34,7 +34,8 @@ class DeadlineCalculatorTest {
|
||||
|
||||
@Test
|
||||
void refusesAttemptWhenBackoffConsumesRemainingBudget() {
|
||||
Deadline deadline = new Deadline(Instant.parse("2026-08-08T00:00:01Z"));
|
||||
Deadline deadline =
|
||||
new Deadline(Instant.parse("2026-08-08T00:00:01Z"), Deadline.Source.PROFILE_TOTAL_CALL);
|
||||
Optional<AttemptBudget> result =
|
||||
new AttemptBudgetCalculator(clock)
|
||||
.nextAttempt(
|
||||
@@ -44,7 +45,8 @@ class DeadlineCalculatorTest {
|
||||
|
||||
@Test
|
||||
void grantsAttemptWhenBudgetCoversBackoffMinimumAndCleanup() {
|
||||
Deadline deadline = new Deadline(Instant.parse("2026-08-08T00:00:02Z"));
|
||||
Deadline deadline =
|
||||
new Deadline(Instant.parse("2026-08-08T00:00:02Z"), Deadline.Source.PROFILE_TOTAL_CALL);
|
||||
Optional<AttemptBudget> result =
|
||||
new AttemptBudgetCalculator(clock)
|
||||
.nextAttempt(
|
||||
@@ -60,7 +62,8 @@ class DeadlineCalculatorTest {
|
||||
assertThatThrownBy(
|
||||
() ->
|
||||
guard.requireTimeRemaining(
|
||||
new Deadline(Instant.parse("2026-08-07T23:59:59Z")),
|
||||
new Deadline(
|
||||
Instant.parse("2026-08-07T23:59:59Z"), Deadline.Source.PROFILE_TOTAL_CALL),
|
||||
CoreFixtures.notSentMetadata()))
|
||||
.isInstanceOf(HttpDeadlineExceededException.class);
|
||||
assertThatThrownBy(
|
||||
|
||||
+2
-2
@@ -147,7 +147,7 @@ class RetryCoordinatorTest {
|
||||
|
||||
@Override
|
||||
public Deadline deadline() {
|
||||
return new Deadline(clock.instant().plusSeconds(5));
|
||||
return new Deadline(clock.instant().plusSeconds(5), Deadline.Source.PROFILE_TOTAL_CALL);
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -202,7 +202,7 @@ class RetryCoordinatorTest {
|
||||
|
||||
@Override
|
||||
public Deadline deadline() {
|
||||
return new Deadline(clock.instant().plusSeconds(5));
|
||||
return new Deadline(clock.instant().plusSeconds(5), Deadline.Source.PROFILE_TOTAL_CALL);
|
||||
}
|
||||
|
||||
@Override
|
||||
|
||||
+119
@@ -5,20 +5,28 @@ import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.OperationName;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpCircuitOpenException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpDeadlineExceededException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRemoteErrorException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.HttpOperation;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.result.HttpCallResult;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.result.ResponseType;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.CircuitBreakerSettings;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.TimeoutSettings;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.TransportType;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.ClientProfiles;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.MockHttpServer;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.TestGateways;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.EnumSource;
|
||||
|
||||
/**
|
||||
* What the per-profile circuit breaker counts as an upstream failure, end to end (finding 05-05).
|
||||
@@ -31,6 +39,10 @@ class CircuitBreakerStatusAccountingTest {
|
||||
private static final int CALLS = 200;
|
||||
private static final int LIBRARY_DEFAULT_WINDOW = 100;
|
||||
private static final int SMALL_WINDOW = 5;
|
||||
private static final int TINY_WINDOW = 2;
|
||||
private static final Duration HEALTHY_ANSWER_TIME = Duration.ofMillis(600);
|
||||
private static final Duration CALLER_BUDGET = Duration.ofMillis(200);
|
||||
private static final Duration PROFILE_TOTAL_CALL = Duration.ofSeconds(1);
|
||||
|
||||
@Test
|
||||
@DisplayName("정상 응답인 404가 200번 이어져도 circuit이 열리지 않고 모든 호출이 상대 서버에 닿는다")
|
||||
@@ -158,6 +170,113 @@ class CircuitBreakerStatusAccountingTest {
|
||||
}
|
||||
}
|
||||
|
||||
@ParameterizedTest(name = "{0}")
|
||||
@EnumSource(
|
||||
value = TransportType.class,
|
||||
names = {"APACHE", "JDK"})
|
||||
@DisplayName("호출자 deadline으로 끊은 시도는 상대 장애로 세지 않아 창을 채워도 다음 호출이 상대 서버에 닿는다")
|
||||
void attemptsCutShortByTheCallersOwnDeadlineDoNotOpenTheCircuit(TransportType transport)
|
||||
throws Exception {
|
||||
try (MockHttpServer server = MockHttpServer.start();
|
||||
TestGateways.Harness harness =
|
||||
TestGateways.forProfile(
|
||||
ClientProfiles.builder("caller-deadline")
|
||||
.baseUrl(server.uri("/"))
|
||||
.transport(transport)
|
||||
.circuitBreaker(
|
||||
new CircuitBreakerSettings(
|
||||
50, SMALL_WINDOW, SMALL_WINDOW, Duration.ofMinutes(1), 1))
|
||||
.build())) {
|
||||
// given
|
||||
for (int i = 0; i <= SMALL_WINDOW; i++) {
|
||||
server.enqueueDelayedHeaders(
|
||||
200,
|
||||
"application/json",
|
||||
"\"ok\"".getBytes(StandardCharsets.UTF_8),
|
||||
HEALTHY_ANSWER_TIME);
|
||||
}
|
||||
List<Throwable> windowFailures = new ArrayList<>();
|
||||
for (int i = 0; i < SMALL_WINDOW; i++) {
|
||||
windowFailures.add(
|
||||
catchThrowable(
|
||||
() -> getUserWithin(harness, Optional.of(Instant.now().plus(CALLER_BUDGET)))));
|
||||
}
|
||||
assertThat(windowFailures)
|
||||
.allSatisfy(f -> assertThat(f).isInstanceOf(HttpDeadlineExceededException.class));
|
||||
|
||||
// when
|
||||
Throwable next = catchThrowable(() -> getUserWithin(harness, Optional.empty()));
|
||||
|
||||
// then
|
||||
assertThat(next).isNull();
|
||||
assertThat(server.requestCount()).isEqualTo(SMALL_WINDOW + 1);
|
||||
}
|
||||
}
|
||||
|
||||
@ParameterizedTest(name = "{0}")
|
||||
@EnumSource(
|
||||
value = TransportType.class,
|
||||
names = {"APACHE", "JDK"})
|
||||
@DisplayName("프로필 total-call 시간으로 끊긴 시도는 상대 장애로 세어 창을 채우면 circuit이 열린다")
|
||||
void attemptsCutShortByTheProfileTotalCallStillOpenTheCircuit(TransportType transport)
|
||||
throws Exception {
|
||||
TimeoutSettings base = ClientProfiles.timeouts();
|
||||
try (MockHttpServer server = MockHttpServer.start();
|
||||
TestGateways.Harness harness =
|
||||
TestGateways.forProfile(
|
||||
ClientProfiles.builder("profile-deadline")
|
||||
.baseUrl(server.uri("/"))
|
||||
.transport(transport)
|
||||
.timeout(
|
||||
new TimeoutSettings(
|
||||
base.dns(),
|
||||
base.connect(),
|
||||
base.tlsHandshake(),
|
||||
base.responseHeader(),
|
||||
base.readIdle(),
|
||||
PROFILE_TOTAL_CALL))
|
||||
.circuitBreaker(
|
||||
new CircuitBreakerSettings(
|
||||
50, TINY_WINDOW, TINY_WINDOW, Duration.ofMinutes(1), 1))
|
||||
.build())) {
|
||||
// given
|
||||
List<Throwable> windowFailures = new ArrayList<>();
|
||||
for (int i = 0; i < TINY_WINDOW; i++) {
|
||||
server.enqueueSlowDripBody(200, 100, Duration.ofMillis(50));
|
||||
windowFailures.add(catchThrowable(() -> getUserWithin(harness, Optional.empty())));
|
||||
}
|
||||
assertThat(windowFailures)
|
||||
.allSatisfy(f -> assertThat(f).isInstanceOf(HttpDeadlineExceededException.class));
|
||||
|
||||
// when
|
||||
Throwable next = catchThrowable(() -> getUserWithin(harness, Optional.empty()));
|
||||
|
||||
// then
|
||||
assertThat(next).isInstanceOf(HttpCircuitOpenException.class);
|
||||
assertThat(server.requestCount()).isEqualTo(TINY_WINDOW);
|
||||
}
|
||||
}
|
||||
|
||||
private static HttpCallResult<String> getUserWithin(
|
||||
TestGateways.Harness harness, Optional<Instant> deadline) {
|
||||
HttpOperation get = HttpOperation.get(new OperationName("get-user"), "/users/1", Map.of());
|
||||
return harness
|
||||
.gateway()
|
||||
.exchange(
|
||||
harness.profile().name(),
|
||||
new HttpOperation(
|
||||
get.operationName(),
|
||||
get.method(),
|
||||
get.uriTemplate(),
|
||||
get.uriVariables(),
|
||||
get.headers(),
|
||||
get.body(),
|
||||
get.idempotency(),
|
||||
get.idempotencyKey(),
|
||||
deadline),
|
||||
ResponseType.of(String.class));
|
||||
}
|
||||
|
||||
private static HttpCallResult<String> getUserAsResult(TestGateways.Harness harness) {
|
||||
return harness
|
||||
.gateway()
|
||||
|
||||
+124
@@ -0,0 +1,124 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.restclient;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.OperationName;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpResponseTimeoutException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.AttemptStage;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.HttpOperation;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.result.BlockingStreamingResponse;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ClientProfile;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ResponseLimits;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.TimeoutSettings;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.TransportType;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.ClientProfiles;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.MockHttpServer;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.TestGateways;
|
||||
import java.time.Duration;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.EnumSource;
|
||||
|
||||
/**
|
||||
* What bounds a streaming download's body once it has been handed over (review 2026-11-03 RR-04-03,
|
||||
* design §15.3).
|
||||
*
|
||||
* <p>The documented contract is transport-independent: the total budget and the response-header
|
||||
* timeout bound the setup, and after hand-over each gap of the body is bounded by {@code read-idle}
|
||||
* alone, so a long download is not cut at an arbitrary total. The JDK transport used to break it:
|
||||
* Spring's JDK request enforces its read timeout over the whole exchange and closes the body stream
|
||||
* when it fires, so every JDK download longer than the response-header timeout was cut, and the
|
||||
* caller got an unclassified {@code IOException: closed}.
|
||||
*
|
||||
* <p>The profile's response-header timeout (500 ms) and total budget (1 s) are both shorter than
|
||||
* the body (about 2 s), and the read-idle timeout (1 s) is longer than every gap of it.
|
||||
*/
|
||||
class StreamingBodyLifetimeTest {
|
||||
|
||||
private static final int BODY_BYTES = 20;
|
||||
private static final Duration DRIP_GAP = Duration.ofMillis(100);
|
||||
private static final Duration STALL_GAP = Duration.ofMillis(2500);
|
||||
|
||||
@ParameterizedTest(name = "{0}")
|
||||
@EnumSource(
|
||||
value = TransportType.class,
|
||||
names = {"APACHE", "JDK"})
|
||||
@DisplayName("넘겨받은 스트리밍 본문은 응답 헤더 시간과 전체 예산을 넘겨도 끝까지 읽힌다")
|
||||
void aStreamedBodyOutlivesTheResponseHeaderTimeoutAndTheTotalBudget(TransportType transport)
|
||||
throws Exception {
|
||||
try (MockHttpServer server = MockHttpServer.start();
|
||||
TestGateways.Harness harness = TestGateways.forProfile(profile(server, transport))) {
|
||||
// given
|
||||
server.enqueueSlowDripBody(200, BODY_BYTES, DRIP_GAP);
|
||||
BlockingStreamingGateway gateway = new BlockingStreamingGateway(harness.registry());
|
||||
|
||||
// when
|
||||
byte[] body;
|
||||
try (BlockingStreamingResponse response = gateway.download(harness.profile().name(), get())) {
|
||||
body = response.body().readAllBytes();
|
||||
}
|
||||
|
||||
// then
|
||||
assertThat(body).hasSize(BODY_BYTES);
|
||||
}
|
||||
}
|
||||
|
||||
@ParameterizedTest(name = "{0}")
|
||||
@EnumSource(
|
||||
value = TransportType.class,
|
||||
names = {"APACHE", "JDK"})
|
||||
@DisplayName("본문 사이 공백이 read-idle을 넘으면 분류된 응답 시간 초과로 끝난다")
|
||||
void aBodyGapLongerThanReadIdleEndsWithAClassifiedTimeout(TransportType transport)
|
||||
throws Exception {
|
||||
try (MockHttpServer server = MockHttpServer.start();
|
||||
TestGateways.Harness harness = TestGateways.forProfile(profile(server, transport))) {
|
||||
// given
|
||||
server.enqueueSlowDripBody(200, BODY_BYTES, STALL_GAP);
|
||||
BlockingStreamingGateway gateway = new BlockingStreamingGateway(harness.registry());
|
||||
long started = System.nanoTime();
|
||||
|
||||
// when
|
||||
Throwable failure =
|
||||
catchThrowable(
|
||||
() -> {
|
||||
try (BlockingStreamingResponse response =
|
||||
gateway.download(harness.profile().name(), get())) {
|
||||
response.body().readAllBytes();
|
||||
}
|
||||
});
|
||||
|
||||
// then
|
||||
Duration elapsed = Duration.ofNanos(System.nanoTime() - started);
|
||||
assertThat(failure).isInstanceOf(HttpResponseTimeoutException.class);
|
||||
assertThat(elapsed).isLessThan(STALL_GAP);
|
||||
HttpResponseTimeoutException timeout = (HttpResponseTimeoutException) failure;
|
||||
assertThat(timeout.metadata().stage()).isEqualTo(AttemptStage.RESPONSE_BODY);
|
||||
assertThat(timeout.metadata().evidence()).isEqualTo(ExecutionEvidence.PARTIAL_RESPONSE);
|
||||
}
|
||||
}
|
||||
|
||||
private static HttpOperation get() {
|
||||
return HttpOperation.get(new OperationName("download"), "/download", Map.of());
|
||||
}
|
||||
|
||||
private static ClientProfile profile(MockHttpServer server, TransportType transport) {
|
||||
TimeoutSettings base = ClientProfiles.timeouts();
|
||||
return ClientProfiles.builder("streaming-body")
|
||||
.baseUrl(server.uri("/"))
|
||||
.transport(transport)
|
||||
.timeout(
|
||||
new TimeoutSettings(
|
||||
base.dns(),
|
||||
base.connect(),
|
||||
base.tlsHandshake(),
|
||||
Duration.ofMillis(500),
|
||||
Duration.ofSeconds(1),
|
||||
Duration.ofSeconds(1)))
|
||||
.response(new ResponseLimits(1024, 1024, Set.of("text/plain")))
|
||||
.build();
|
||||
}
|
||||
}
|
||||
+5
@@ -68,6 +68,11 @@ public final class RecordingResilienceComponents {
|
||||
events.add("circuit-error");
|
||||
}
|
||||
|
||||
@Override
|
||||
public void releasePermission() {
|
||||
events.add("circuit-release");
|
||||
}
|
||||
|
||||
@Override
|
||||
public String state() {
|
||||
return circuitOpen ? "OPEN" : "CLOSED";
|
||||
|
||||
@@ -20,16 +20,22 @@ build choices) live in [README.md](README.md).
|
||||
- `UuidV7DurableOperationIdFactory` — `DurableOperationIdFactory` implementation (UUIDv7 via
|
||||
`uuid-creator`). The operation id is also the submission event's id, and the messaging platform's
|
||||
message id accepts only UUIDv7 (review 2026-11-01 06-06). Wired by `composition:durable-operation`.
|
||||
- The leaf name says "identifier"; general identifier generation, clock and random sources are not here.
|
||||
- The leaf name says "identifier"; general-purpose clock and random sources are not here.
|
||||
The name is kept because Gradle paths and consumers use it (README, review 05-21).
|
||||
- Wiring is `composition/service-runtime` (`PseudonymizationConfig`), not this leaf.
|
||||
|
||||
## Allowed
|
||||
|
||||
- `:application-core` (declared in [build.gradle](build.gradle)); the registry entry is the SSOT.
|
||||
Declared in [build.gradle](build.gradle); the registry entry is the SSOT.
|
||||
|
||||
- `:application-core` — the ports both classes implement.
|
||||
- `:shared-contract`, for `HmacSha256` only (review 2026-11-01 01-11). The pseudonym is the hex HMAC
|
||||
of the principal; `HmacUserPrincipalPseudonymizerVectorTest` pins it to a vector computed outside
|
||||
the JVM, because logs written before and after a release are joined on it.
|
||||
- `:domain-core` — a feature's server-assigned id adapter implements its domain `*IdFactory` port
|
||||
here (registry edge `adapter:outbound:identifier -> domain-core`; `docs/manual/02-new-feature.md`).
|
||||
- `com.github.f4b6a3:uuid-creator` (`libs.uuid.creator`) — UUIDv7 for
|
||||
`UuidV7DurableOperationIdFactory`, the generator the sample's id adapters use.
|
||||
|
||||
## Forbidden
|
||||
|
||||
@@ -42,8 +48,9 @@ build choices) live in [README.md](README.md).
|
||||
|
||||
## Test
|
||||
|
||||
Pure unit tests, no Spring context (`HmacUserPrincipalPseudonymizerTest`, including the RFC 4231
|
||||
known-answer vector).
|
||||
Pure unit tests, no Spring context: `HmacUserPrincipalPseudonymizerTest` (including the RFC 4231
|
||||
known-answer vector), `HmacUserPrincipalPseudonymizerVectorTest` and
|
||||
`UuidV7DurableOperationIdFactoryTest`.
|
||||
|
||||
```bash
|
||||
cd src
|
||||
|
||||
@@ -9,7 +9,8 @@
|
||||
## 이름과 내용이 다른 이유 (review 05-21)
|
||||
|
||||
leaf 이름은 `identifier`지만 범용 식별자 생성기·clock·random 소스는 여기 없다. 식별자 생성은
|
||||
durable operation id(UUIDv7) 하나뿐이다(아래). 예전 문서가 범용 생성기 계획을 현재 내용처럼 적었다. 이름은 바꾸지 않았다: Gradle 경로·registry id·소비자
|
||||
durable operation id(UUIDv7) 하나뿐이고(아래), 기능이 서버에서 id를 정하면 그 기능의 domain
|
||||
`*IdFactory` 구현도 여기 둔다(`:domain-core` 의존이 그 때문이다). 예전 문서가 범용 생성기 계획을 현재 내용처럼 적었다. 이름은 바꾸지 않았다: Gradle 경로·registry id·소비자
|
||||
(`composition:service-runtime`, `app-bootstrap`, `sample-portfolio`)의 의존 선언이 모두 이 이름을
|
||||
쓰고, 바꿔서 얻는 것은 이름 하나뿐이다. 외부 연동이 없는 crypto 능력을 새로 둘 곳이 필요하면 이
|
||||
leaf에 둔다. 그때 이 절을 고친다.
|
||||
@@ -25,10 +26,10 @@ leaf에 둔다. 그때 이 절을 고친다.
|
||||
대소문자를 바꾸면 테스트가 실패한다.
|
||||
- Salt: `APP_PRIVACY_PSEUDONYMIZATION_SALT`(secret, 90일 회전)에서 온다. 이 클래스는 salt를 스스로
|
||||
읽지 않고 생성자로 받는다. 받은 배열은 복사해 둔다.
|
||||
- Thread safety: `Mac`은 thread-safe하지 않아서 호출마다 새로 만든다. 그래서 싱글톤 bean으로
|
||||
공유해도 된다. `HmacSHA256`은 JDK 필수 알고리즘이라 `NoSuchAlgorithmException`·
|
||||
`InvalidKeyException`은 사실상 나지 않는다. checked exception을 호출부에 남기지 않으려고
|
||||
`IllegalStateException`으로 감싼다.
|
||||
- 서명: 이 클래스는 `Mac`을 직접 다루지 않는다. shared-contract의 `HmacSha256.sign`에 맡긴다(리뷰
|
||||
2026-11-01 01-11). `HmacSha256`이 호출마다 새 `Mac`을 만들고(`Mac`은 thread-safe하지 않다),
|
||||
JDK 필수 알고리즘이라 사실상 나지 않는 checked exception을 `IllegalStateException`으로 감싼다.
|
||||
그래서 이 클래스는 싱글톤 bean으로 공유해도 된다.
|
||||
|
||||
## UuidV7DurableOperationIdFactory
|
||||
|
||||
@@ -46,6 +47,10 @@ Spring 어노테이션이 없다. bean은
|
||||
|
||||
## 빌드 결정
|
||||
|
||||
의존은 넷이다: `:application-core`(두 클래스가 구현하는 port), `:shared-contract`(`HmacSha256`),
|
||||
`:domain-core`(기능의 `*IdFactory` 구현 자리), `uuid-creator`(UUIDv7). 목록과 이유는
|
||||
[CLAUDE.md](CLAUDE.md) Allowed 절이 기준이다.
|
||||
|
||||
`:application-core`를 `implementation`으로 받는다. 이 leaf의 public 타입 시그니처에
|
||||
application-core 타입이 드러나지만, 조립하는 쪽이 application-core를 이미 classpath에 갖고 있다.
|
||||
허용 edge는 `src/config/architecture/modules.json`이 기준이다.
|
||||
|
||||
+4
-2
@@ -1,5 +1,7 @@
|
||||
/**
|
||||
* The HMAC-SHA-256 user-principal pseudonymizer. Despite the leaf name, no identifier generation,
|
||||
* clock or random source lives here; see README for why the name was kept.
|
||||
* Two identifier adapters with no external IO: the HMAC-SHA-256 user-principal pseudonymizer and
|
||||
* the UUIDv7 durable operation id factory. A feature's server-assigned id adapter for a domain
|
||||
* {@code *IdFactory} port belongs here too. General clock and random sources do not live here; see
|
||||
* README for why the leaf name was kept.
|
||||
*/
|
||||
package dev.caskeleton.adapter.outbound.identifier;
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
# adapter:outbound:messaging-kafka
|
||||
|
||||
- Module ID `adapter-outbound-messaging-kafka`, Gradle path `:adapter:outbound:messaging-kafka`.
|
||||
Registry SSOT: `src/config/architecture/modules.json`.
|
||||
- Responsibility: the Kafka publishing capability as one `runtimeOnly` line. It has no production
|
||||
code: it bundles `adapter:outbound:messaging` (the application's bridge) with
|
||||
`messaging:messaging-spring-boot-starter-kafka` (the vendored platform and its Kafka transport) and
|
||||
ships the module defaults in `config/module-defaults/messaging.yml`.
|
||||
- Boundaries: add no code here. A publish rule belongs to the bridge (`adapter:outbound:messaging`),
|
||||
a Kafka rule to the vendored product (`src/messaging/CLAUDE.md`); a class here would be a third
|
||||
place. Allowed edges come only from the registry entry.
|
||||
- Tests: activation tests (`./gradlew :adapter:outbound:messaging-kafka:test`) and the real-Kafka
|
||||
lane `messagingKafkaRelayIntegrationTest` (relay → bridge → broker record bytes; `:integrationCheck`).
|
||||
- Design: `README.md`.
|
||||
+14
-6
@@ -8,6 +8,7 @@ import dev.caskeleton.application.outbox.OutboxAppendPort;
|
||||
import dev.caskeleton.application.outbox.OutboxCompletion;
|
||||
import dev.caskeleton.application.outbox.OutboxEventStatus;
|
||||
import dev.caskeleton.application.outbox.OutboxLease;
|
||||
import dev.caskeleton.application.outbox.OutboxLongRetries;
|
||||
import dev.caskeleton.application.outbox.OutboxStorePort;
|
||||
import dev.caskeleton.application.outbox.testkit.InMemoryOutboxStore;
|
||||
import dev.caskeleton.application.outbox.testkit.OutboxEvents;
|
||||
@@ -282,13 +283,15 @@ class ApplicationRelayKafkaIntegrationTest {
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized OutboxCompletion markFailed(OutboxLease lease, Instant nextAttemptAt) {
|
||||
return delegate.markFailed(lease, nextAttemptAt);
|
||||
public synchronized OutboxCompletion markFailed(
|
||||
OutboxLease lease, Instant nextAttemptAt, String failureCode) {
|
||||
return delegate.markFailed(lease, nextAttemptAt, failureCode);
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized OutboxCompletion markDeferred(OutboxLease lease, Instant nextAttemptAt) {
|
||||
return delegate.markDeferred(lease, nextAttemptAt);
|
||||
public synchronized OutboxCompletion markDeferred(
|
||||
OutboxLease lease, Instant nextAttemptAt, String failureCode) {
|
||||
return delegate.markDeferred(lease, nextAttemptAt, failureCode);
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -297,8 +300,13 @@ class ApplicationRelayKafkaIntegrationTest {
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized OutboxCompletion markDead(OutboxLease lease) {
|
||||
return delegate.markDead(lease);
|
||||
public synchronized OutboxCompletion markDead(OutboxLease lease, String failureCode) {
|
||||
return delegate.markDead(lease, failureCode);
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized OutboxLongRetries longRetries(int minDeferredClaims, Instant now) {
|
||||
return delegate.longRetries(minDeferredClaims, now);
|
||||
}
|
||||
|
||||
@Override
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
- Module ID: `adapter-outbound-messaging`
|
||||
- Gradle path: `:adapter:outbound:messaging`
|
||||
- Focused test (derived from Gradle path): `./gradlew :adapter:outbound:messaging:test --console=plain`
|
||||
- Runtime baseline: Java 21; repository framework baseline: Spring Boot 4.0.0.
|
||||
- Runtime baseline: Java 21; repository framework baseline: Spring Boot 4.0.8.
|
||||
- Registry SSOT: `src/config/architecture/modules.json`.
|
||||
|
||||
Package root: `dev.caskeleton.adapter.outbound.messaging`.
|
||||
|
||||
@@ -42,13 +42,33 @@ platform `REJECTED` 중 `FailureDescriptor.retryable()`이 true인 것(종료
|
||||
멈춘다(2026-11-01 리뷰 06-01). `AMBIGUOUS`로 번역하지 않은 이유: 동작은 같지만 "전송됐을 수
|
||||
있음"이라는 거짓 정보가 남아 중복 경고를 오염시킨다.
|
||||
|
||||
### 실패 원인 코드
|
||||
|
||||
결과(outcome)는 relay가 무엇을 할지만 정한다. 종료 중 거부, 닫힌 transport, 거부된 credential은 모두
|
||||
`REJECTED_RETRYABLE` 하나다. 그런데 이 셋은 끝이 없는 재시도다(시도 예산을 쓰지 않는다). 그래서 bridge는
|
||||
platform `FailureDescriptor.code()`를 버리지 않고 `OutboxPublishAttempt.failureCode()`로 함께 넘긴다
|
||||
(재리뷰 2026-11-03 RR-05-03). relay는 그 코드를 행의 `last_failure_code`(V22)와 실패 로그의
|
||||
`error.reason`에 남긴다. 운영자는 이 값으로 "credential이 진짜 틀렸다"와 "broker가 이 레코드만 받지
|
||||
않는다"를 가른다(`docs/runbooks/outbox-publish-failed.md` §4.1).
|
||||
|
||||
- platform code는 이미 정제된 이름이다(`KAFKA_AUTHENTICATIONEXCEPTION`, `PUBLISH_DEADLINE_EXCEEDED` 등).
|
||||
driver 메시지가 아니므로 그대로 옮긴다. application 코드 문법(`[A-Za-z0-9][A-Za-z0-9._:-]*`, 120자)에
|
||||
맞지 않는 code는 그 실패의 `FailureCategory` 이름으로 옮긴다.
|
||||
- platform으로 옮길 수 없는 event(아래 id 규칙)는 bridge가 `EVENT_NOT_REPRESENTABLE`로 거부한다.
|
||||
- 버린 대안: relay가 platform 타입을 직접 보는 것(경계가 깨진다), outcome 값을 원인별로 늘리는 것(relay가
|
||||
할 일은 같은데 분기만 는다).
|
||||
|
||||
### event id → message id
|
||||
|
||||
application `EventId`는 문자열이고 platform `MessageId`는 **UUIDv7**만 받는다. bridge는 event id가
|
||||
정규 소문자 UUIDv7 문자열일 때만 같은 값으로 옮기고, 그 밖(대문자·축약 표기·v4 등 다른 버전)은 다른
|
||||
문자열의 message id로 바꿔 보내지 않고 `REJECTED_BEFORE_SEND`로 거부한다(relay는 DEAD). 소비자 중복
|
||||
제거 키가 event id라서 값을 바꿔 보내면 중복 제거가 깨지기 때문이다(리뷰 2026-11-01 06-06). 이 거부는
|
||||
append 시점(`EventId` 문법)에서 먼저 일어나야 한다 — application 쪽 문법 일원화는 별도 작업이다.
|
||||
제거 키가 event id라서 값을 바꿔 보내면 중복 제거가 깨지기 때문이다(리뷰 2026-11-01 06-06).
|
||||
|
||||
같은 규칙이 append 쪽에 먼저 있다. `IntegrationEventDraft`가 정규 소문자 UUIDv7이 아닌 event id와
|
||||
causation id를 거부하므로(`IntegrationIdentity.isCanonicalUuidV7`), 그런 id는 업무 트랜잭션 안에서 실패한다
|
||||
(재리뷰 2026-11-03 RR-05-04). bridge의 거부는 이 규칙 전에 쓰인 행(저장 문법 `EventId`는 그대로 넓다)을 위한
|
||||
마지막 방어선이다.
|
||||
|
||||
## outbox 발행 포트에는 대역이 없다
|
||||
|
||||
@@ -70,9 +90,10 @@ event가 다시 publish될 수 있다. 같은 event id로 다시 나가므로 co
|
||||
있으면 `messaging-platform`이다. 이 adapter는 confirmed FAILED/DEAD report 하나를 SLF4J 2 fluent
|
||||
ERROR 하나로 렌더링한다.
|
||||
|
||||
공통 field는 `error.code`, `error.category`, `dependency_name`,
|
||||
공통 field는 `error.code`, `error.category`, `error.cause_type`, `error.reason`, `dependency_name`,
|
||||
`dependency_type=messaging`, `outcome`, `event_id`, `event_type`, `aggregate_id`,
|
||||
`correlation_id`, `attempt_count`, `runbook_link`다. retry report만 `next_attempt_at`을 추가한다.
|
||||
`error.reason`은 위 실패 원인 코드다.
|
||||
payload/envelope/exception-derived field는 받거나 렌더링하지 않는다. logging 내부
|
||||
`RuntimeException`은 adapter와 use case 양쪽에서 방어하므로 persisted FAILED/DEAD outcome을 바꾸지
|
||||
않는다.
|
||||
@@ -85,10 +106,8 @@ payload/envelope/exception-derived field는 받거나 렌더링하지 않는다.
|
||||
`envelope`·`realtime`)은 2026-09-25에 삭제됐고 복구 경로는 `docs/capability-status.md` 제거 표에
|
||||
있다.
|
||||
|
||||
공통 build evidence manifest 검증은 build-tools로 옮겼다. 루트 `verifyMessagingContracts`
|
||||
(`VerifyMessagingEvidenceTask`)가 application-core·shared-contract 두 qualification 레인 결과로
|
||||
`build/messaging-evidence/contracts-schema/manifest.json`을 쓰고, 쓴 바이트를 바로 common schema로
|
||||
검증한다. 이 leaf는 그 검증에 관여하지 않는다.
|
||||
integration event 계약 검증은 루트 `verifyMessagingContracts`(`src/build.gradle`)가 한다. application-core·
|
||||
shared-contract·sample-portfolio의 qualification lane 셋을 묶은 task다. 이 leaf는 그 검증에 관여하지 않는다.
|
||||
|
||||
canonical path는 platform `PublishResult`의 confirmation/transmission evidence를 application
|
||||
`OutboxPublishOutcome`으로 명시적으로 변환한다.
|
||||
`OutboxPublishOutcome`으로 명시적으로 변환하고, 실패 code를 `OutboxPublishAttempt`에 함께 싣는다.
|
||||
|
||||
+14
-4
@@ -4,17 +4,21 @@ import dev.caskeleton.application.messaging.event.IntegrationEventPublishPort;
|
||||
import dev.caskeleton.application.outbox.ClaimedOutboxEvent;
|
||||
import dev.caskeleton.application.outbox.OutboxMessagePublishPort;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishAmbiguousException;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishRefusedException;
|
||||
import java.time.Duration;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
|
||||
/**
|
||||
* The relay's publish port: a claimed row's validated event, delegated unchanged to the canonical
|
||||
* {@link IntegrationEventPublishPort}.
|
||||
*
|
||||
* <p>The event is the one the append validated, restored from its exact envelope bytes; nothing
|
||||
* here re-renders it. The platform's four-valued outcome is passed through, so the relay can tell a
|
||||
* possibly-stored message (ambiguous) from a refused one.
|
||||
* here re-renders it. The canonical publisher's attempt — its outcome and its failure code — is
|
||||
* passed through unchanged, so the relay can tell a possibly-stored message (ambiguous) from a
|
||||
* refused one, and a refused credential from a closed transport.
|
||||
*/
|
||||
public final class OutboxMessagePublishAdapter implements OutboxMessagePublishPort {
|
||||
|
||||
@@ -27,7 +31,7 @@ public final class OutboxMessagePublishAdapter implements OutboxMessagePublishPo
|
||||
|
||||
@Override
|
||||
public void publish(ClaimedOutboxEvent event) {
|
||||
OutboxPublishOutcome outcome = publishForOutcome(event);
|
||||
OutboxPublishOutcome outcome = publishForOutcome(event).outcome();
|
||||
if (outcome == OutboxPublishOutcome.CONFIRMED) {
|
||||
return;
|
||||
}
|
||||
@@ -37,8 +41,14 @@ public final class OutboxMessagePublishAdapter implements OutboxMessagePublishPo
|
||||
throw new OutboxPublishRefusedException(outcome);
|
||||
}
|
||||
|
||||
/** The canonical publisher's, unchanged: this adapter adds no wait of its own. */
|
||||
@Override
|
||||
public OutboxPublishOutcome publishForOutcome(ClaimedOutboxEvent event) {
|
||||
public Optional<Duration> longestAttempt() {
|
||||
return canonicalPublisher.longestAttempt();
|
||||
}
|
||||
|
||||
@Override
|
||||
public OutboxPublishAttempt publishForOutcome(ClaimedOutboxEvent event) {
|
||||
Objects.requireNonNull(event, "event must not be null");
|
||||
return canonicalPublisher.publish(event.event()).toCompletableFuture().join();
|
||||
}
|
||||
|
||||
+4
@@ -48,6 +48,10 @@ public final class Slf4jOutboxRelayFailureReportAdapter implements OutboxRelayFa
|
||||
// occasionally credentials. The class name is a type; the code is bounded.
|
||||
.addKeyValue("error.code", report.code().code())
|
||||
.addKeyValue("error.cause_type", report.causeType())
|
||||
// Why, in the publisher's words: the cause type of a returned outcome names only
|
||||
// which
|
||||
// outcome it was, and a refused credential and a closed transport are one outcome.
|
||||
.addKeyValue("error.reason", report.failureCode())
|
||||
.addKeyValue("error.category", report.code().category().name())
|
||||
.addKeyValue("dependency_name", dependencyName)
|
||||
.addKeyValue("dependency_type", DEPENDENCY_TYPE)
|
||||
|
||||
+60
-6
@@ -2,6 +2,7 @@ package dev.caskeleton.adapter.outbound.messaging.platformbridge;
|
||||
|
||||
import dev.caskeleton.application.messaging.event.IntegrationEventPublishPort;
|
||||
import dev.caskeleton.application.messaging.event.ValidatedIntegrationEvent;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
|
||||
import dev.caskeleton.messaging.api.CausationId;
|
||||
import dev.caskeleton.messaging.api.ContentType;
|
||||
@@ -26,6 +27,7 @@ import dev.caskeleton.messaging.api.publish.TransmissionEvidence;
|
||||
import dev.caskeleton.messaging.schema.EncodedMessage;
|
||||
import dev.caskeleton.messaging.schema.EncodedMessagePublisher;
|
||||
import dev.caskeleton.messaging.schema.SchemaReference;
|
||||
import java.time.Duration;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
@@ -45,6 +47,12 @@ import java.util.concurrent.CompletionStage;
|
||||
*/
|
||||
public final class PlatformIntegrationEventPublishAdapter implements IntegrationEventPublishPort {
|
||||
|
||||
/**
|
||||
* The failure code of an event this bridge refuses before the platform sees it: a value — an id
|
||||
* above all — the platform's types cannot carry.
|
||||
*/
|
||||
public static final String NOT_REPRESENTABLE = "EVENT_NOT_REPRESENTABLE";
|
||||
|
||||
private final EncodedMessagePublisher publisher;
|
||||
private final ProducerId producer;
|
||||
|
||||
@@ -55,7 +63,7 @@ public final class PlatformIntegrationEventPublishAdapter implements Integration
|
||||
}
|
||||
|
||||
@Override
|
||||
public CompletionStage<OutboxPublishOutcome> publish(ValidatedIntegrationEvent event) {
|
||||
public CompletionStage<OutboxPublishAttempt> publish(ValidatedIntegrationEvent event) {
|
||||
Objects.requireNonNull(event, "event must not be null");
|
||||
final PreparedPublish prepared;
|
||||
try {
|
||||
@@ -64,12 +72,27 @@ public final class PlatformIntegrationEventPublishAdapter implements Integration
|
||||
// Only a value the platform cannot represent is a definite refusal: the same payload is
|
||||
// refused again, so the relay dead-letters it at once. Any other failure is not evidence of
|
||||
// that and propagates, so the relay treats it as a retryable failure with its real cause.
|
||||
return CompletableFuture.completedFuture(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
|
||||
return CompletableFuture.completedFuture(
|
||||
OutboxPublishAttempt.failed(
|
||||
OutboxPublishOutcome.REJECTED_BEFORE_SEND, NOT_REPRESENTABLE));
|
||||
}
|
||||
|
||||
return publisher
|
||||
.publishEncoded(prepared.destination(), prepared.envelope(), PublishOptions.defaults())
|
||||
.thenApply(PlatformIntegrationEventPublishAdapter::mapOutcome);
|
||||
.thenApply(PlatformIntegrationEventPublishAdapter::translate);
|
||||
}
|
||||
|
||||
/**
|
||||
* Twice the publish deadline this bridge asks for ({@link PublishOptions#defaults()}).
|
||||
*
|
||||
* <p>The platform measures the deadline from the call, but a transport may block inside its send
|
||||
* before the deadline is applied to the stage it returns — Kafka's {@code max.block.ms} is set to
|
||||
* the same default for exactly that wait — so an attempt can take the deadline twice over. The
|
||||
* relay sizes its batch against this (review 2026-11-01 06-12).
|
||||
*/
|
||||
@Override
|
||||
public Optional<Duration> longestAttempt() {
|
||||
return Optional.of(PublishOptions.defaults().timeout().multipliedBy(2));
|
||||
}
|
||||
|
||||
private PreparedPublish prepare(ValidatedIntegrationEvent event) {
|
||||
@@ -146,8 +169,10 @@ public final class PlatformIntegrationEventPublishAdapter implements Integration
|
||||
* </ul>
|
||||
*
|
||||
* <p>Either refusal is {@code REJECTED_BEFORE_SEND}: the same id is refused again, so the relay
|
||||
* dead-letters it. The append side's {@code EventId} grammar is where such an id should fail
|
||||
* first, in the writer's transaction; until it does, this is where the mismatch is caught.
|
||||
* dead-letters it. A new event fails earlier, in the writer's transaction: {@code
|
||||
* IntegrationEventDraft} refuses an id that is not a canonical lower-case UUIDv7 (re-review
|
||||
* 2026-11-03 RR-05-04). This check stays for rows appended before that rule — the stored {@code
|
||||
* EventId} grammar is wider, so they can still be read back.
|
||||
*
|
||||
* @throws IllegalArgumentException when the value is not a canonical lower-case UUIDv7
|
||||
*/
|
||||
@@ -177,8 +202,37 @@ public final class PlatformIntegrationEventPublishAdapter implements Integration
|
||||
* committed event, and its aggregate behind it, for a cause that clears on its own. It gets its
|
||||
* own value instead.
|
||||
*/
|
||||
private static OutboxPublishOutcome mapOutcome(PublishResult result) {
|
||||
private static OutboxPublishAttempt translate(PublishResult result) {
|
||||
Objects.requireNonNull(result, "publish result must not be null");
|
||||
OutboxPublishOutcome outcome = mapOutcome(result);
|
||||
if (outcome == OutboxPublishOutcome.CONFIRMED) {
|
||||
return OutboxPublishAttempt.confirmed();
|
||||
}
|
||||
return result
|
||||
.failure()
|
||||
.map(PlatformIntegrationEventPublishAdapter::failureCode)
|
||||
.map(code -> OutboxPublishAttempt.failed(outcome, code))
|
||||
.orElseGet(() -> OutboxPublishAttempt.of(outcome));
|
||||
}
|
||||
|
||||
/**
|
||||
* The platform's failure code, carried as the application's failure code (re-review 2026-11-03
|
||||
* RR-05-03).
|
||||
*
|
||||
* <p>The outcome is all the relay acts on, and it loses the reason on purpose: a shutdown, a
|
||||
* closed transport and a refused credential are one {@code REJECTED_RETRYABLE}. The code keeps it
|
||||
* for the operator, on the row and in the failure log. The platform's code is already sanitized —
|
||||
* a stable name, never a driver message — and passes as it is; one outside the application's code
|
||||
* grammar (the platform only bounds its length) is carried as the failure's category instead, a
|
||||
* name the platform owns too.
|
||||
*/
|
||||
private static String failureCode(FailureDescriptor failure) {
|
||||
return OutboxPublishAttempt.isFailureCode(failure.code())
|
||||
? failure.code()
|
||||
: failure.category().name();
|
||||
}
|
||||
|
||||
private static OutboxPublishOutcome mapOutcome(PublishResult result) {
|
||||
if (result.completion() == PublishCompletion.CONFIRMED) {
|
||||
return OutboxPublishOutcome.CONFIRMED;
|
||||
}
|
||||
|
||||
+2
-2
@@ -5,7 +5,7 @@ import static org.assertj.core.api.Assertions.assertThat;
|
||||
import dev.caskeleton.adapter.outbound.messaging.outbox.OutboxMessagePublishAdapter;
|
||||
import dev.caskeleton.application.messaging.event.IntegrationEventPublishPort;
|
||||
import dev.caskeleton.application.outbox.OutboxMessagePublishPort;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
|
||||
@@ -32,7 +32,7 @@ class MessagingConfigTest {
|
||||
runner
|
||||
.withBean(
|
||||
IntegrationEventPublishPort.class,
|
||||
() -> event -> CompletableFuture.completedFuture(OutboxPublishOutcome.CONFIRMED))
|
||||
() -> event -> CompletableFuture.completedFuture(OutboxPublishAttempt.confirmed()))
|
||||
.run(
|
||||
context ->
|
||||
assertThat(context.getBean(OutboxMessagePublishPort.class))
|
||||
|
||||
+12
-3
@@ -51,7 +51,14 @@ class Slf4jOutboxRelayFailureReportAdapterTest {
|
||||
|
||||
adapter.report(
|
||||
OutboxRelayFailureReport.retryableFailure(
|
||||
"evt-1", "WorkLogReserved", "agg-1", "corr-1", 2, nextAttemptAt, cause));
|
||||
"evt-1",
|
||||
"WorkLogReserved",
|
||||
"agg-1",
|
||||
"corr-1",
|
||||
2,
|
||||
nextAttemptAt,
|
||||
cause,
|
||||
"KAFKA_AUTHENTICATIONEXCEPTION"));
|
||||
|
||||
assertThat(appender.list).hasSize(1);
|
||||
ILoggingEvent event = appender.list.getFirst();
|
||||
@@ -71,6 +78,7 @@ class Slf4jOutboxRelayFailureReportAdapterTest {
|
||||
Map.entry("correlation_id", "corr-1"),
|
||||
Map.entry("attempt_count", 2),
|
||||
Map.entry("error.cause_type", "java.lang.RuntimeException"),
|
||||
Map.entry("error.reason", "KAFKA_AUTHENTICATIONEXCEPTION"),
|
||||
Map.entry("runbook_link", "runbook://outbox/publish-failed"),
|
||||
Map.entry("next_attempt_at", "2026-07-25T01:02:03Z")));
|
||||
assertThat(event.getThrowableProxy())
|
||||
@@ -89,7 +97,7 @@ class Slf4jOutboxRelayFailureReportAdapterTest {
|
||||
|
||||
adapter.report(
|
||||
OutboxRelayFailureReport.deadLetter(
|
||||
"evt-2", "WorkLogReserved", "agg-2", "corr-2", 3, cause));
|
||||
"evt-2", "WorkLogReserved", "agg-2", "corr-2", 3, cause, "REJECTED_BEFORE_SEND"));
|
||||
|
||||
assertThat(appender.list).hasSize(1);
|
||||
ILoggingEvent event = appender.list.getFirst();
|
||||
@@ -142,7 +150,8 @@ class Slf4jOutboxRelayFailureReportAdapterTest {
|
||||
"agg-3",
|
||||
"corr-3",
|
||||
3,
|
||||
new RuntimeException("broker down"))))
|
||||
new RuntimeException("broker down"),
|
||||
"RuntimeException")))
|
||||
.doesNotThrowAnyException();
|
||||
}
|
||||
|
||||
|
||||
+104
-9
@@ -12,6 +12,7 @@ import dev.caskeleton.application.messaging.event.IntegrationEventPublishPort;
|
||||
import dev.caskeleton.application.messaging.event.ValidatedIntegrationEvent;
|
||||
import dev.caskeleton.application.messaging.event.testkit.BrokerAnswer;
|
||||
import dev.caskeleton.application.messaging.event.testkit.IntegrationEventPublishContract;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
|
||||
import dev.caskeleton.messaging.api.ContentType;
|
||||
import dev.caskeleton.messaging.api.MessageEnvelope;
|
||||
@@ -59,7 +60,7 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
|
||||
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
|
||||
ValidatedIntegrationEvent event = event(EVENT_ID, Optional.of(CAUSATION_ID));
|
||||
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join())
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
|
||||
.isEqualTo(OutboxPublishOutcome.CONFIRMED);
|
||||
|
||||
MessageDestination<EncodedMessage> destination = publisher.destination;
|
||||
@@ -99,19 +100,19 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
|
||||
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
|
||||
ValidatedIntegrationEvent event = event(EVENT_ID, Optional.empty());
|
||||
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join())
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
|
||||
.isEqualTo(OutboxPublishOutcome.CONFIRMED);
|
||||
publisher.result = ambiguous();
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join())
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
|
||||
.isEqualTo(OutboxPublishOutcome.AMBIGUOUS);
|
||||
publisher.result = rejected(PublishEvidence.notTransmitted());
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join())
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
|
||||
.isEqualTo(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
|
||||
publisher.result =
|
||||
rejected(
|
||||
new PublishEvidence(
|
||||
true, TransmissionEvidence.TRANSMITTED, false, ConfirmationLevel.NONE));
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join())
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
|
||||
.isEqualTo(OutboxPublishOutcome.REJECTED_AFTER_BROKER);
|
||||
}
|
||||
|
||||
@@ -132,12 +133,101 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
|
||||
|
||||
// when
|
||||
OutboxPublishOutcome outcome =
|
||||
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join();
|
||||
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join().outcome();
|
||||
|
||||
// then
|
||||
assertThat(outcome).isEqualTo(OutboxPublishOutcome.REJECTED_RETRYABLE);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("platform이 붙인 실패 코드는 원인 코드로 그대로 옮긴다")
|
||||
void thePlatformsFailureCodeIsCarriedAsTheFailureCode() {
|
||||
// given: the broker refused this producer's credentials
|
||||
RecordingPublisher publisher =
|
||||
new RecordingPublisher(
|
||||
PublishResult.rejected(
|
||||
Duration.ZERO,
|
||||
new FailureDescriptor(
|
||||
FailureCategory.AUTHENTICATION,
|
||||
"KAFKA_AUTHENTICATIONEXCEPTION",
|
||||
true,
|
||||
"kafka publish failed with AuthenticationException",
|
||||
Optional.of("AuthenticationException"))));
|
||||
PlatformIntegrationEventPublishAdapter bridge =
|
||||
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
|
||||
|
||||
// when
|
||||
OutboxPublishAttempt attempt =
|
||||
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join();
|
||||
|
||||
// then
|
||||
assertThat(attempt)
|
||||
.isEqualTo(
|
||||
OutboxPublishAttempt.failed(
|
||||
OutboxPublishOutcome.REJECTED_RETRYABLE, "KAFKA_AUTHENTICATIONEXCEPTION"));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("코드 문법에 맞지 않는 platform 실패 코드는 그 실패의 분류 이름으로 옮긴다")
|
||||
void aPlatformCodeOutsideTheCodeGrammarIsCarriedAsItsCategory() {
|
||||
// given
|
||||
RecordingPublisher publisher =
|
||||
new RecordingPublisher(
|
||||
PublishResult.ambiguous(
|
||||
Duration.ZERO,
|
||||
FailureDescriptor.of(
|
||||
FailureCategory.AMBIGUOUS, "timed out after 5000 ms", "no answer")));
|
||||
PlatformIntegrationEventPublishAdapter bridge =
|
||||
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
|
||||
|
||||
// when
|
||||
OutboxPublishAttempt attempt =
|
||||
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join();
|
||||
|
||||
// then
|
||||
assertThat(attempt.failureCode()).contains("AMBIGUOUS");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("platform으로 옮길 수 없는 event는 그 이유를 원인 코드로 알린다")
|
||||
void anEventThePlatformCannotRepresentIsRefusedWithItsOwnFailureCode() {
|
||||
// given
|
||||
RecordingPublisher publisher = new RecordingPublisher(confirmed());
|
||||
PlatformIntegrationEventPublishAdapter bridge =
|
||||
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
|
||||
|
||||
// when
|
||||
OutboxPublishAttempt attempt =
|
||||
bridge.publish(event("event-1", Optional.empty())).toCompletableFuture().join();
|
||||
|
||||
// then
|
||||
assertThat(attempt)
|
||||
.isEqualTo(
|
||||
OutboxPublishAttempt.failed(
|
||||
OutboxPublishOutcome.REJECTED_BEFORE_SEND,
|
||||
PlatformIntegrationEventPublishAdapter.NOT_REPRESENTABLE));
|
||||
}
|
||||
|
||||
/**
|
||||
* The relay sizes its batch against this (review 2026-11-01 06-12). The platform's deadline is
|
||||
* measured from the call, but a transport may block in its send before the deadline is applied —
|
||||
* Kafka's {@code max.block.ms} is set to the same default — so an attempt can take twice it.
|
||||
*/
|
||||
@Test
|
||||
@DisplayName("발행 하나의 최악 시간은 platform 기본 발행 기한의 두 배다")
|
||||
void theLongestAttemptIsTwiceThePlatformsDefaultPublishDeadline() {
|
||||
// given
|
||||
PlatformIntegrationEventPublishAdapter bridge =
|
||||
new PlatformIntegrationEventPublishAdapter(
|
||||
new RecordingPublisher(confirmed()), "portfolio-service");
|
||||
|
||||
// when
|
||||
Optional<Duration> longest = bridge.longestAttempt();
|
||||
|
||||
// then
|
||||
assertThat(longest).contains(PublishOptions.defaults().timeout().multipliedBy(2));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("다시 시도해도 같은 결과인 거부는 지금처럼 확정 거부로 번역한다")
|
||||
void aRefusalThePlatformMarksFinalStaysADefiniteRejection() {
|
||||
@@ -153,7 +243,7 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
|
||||
|
||||
// when
|
||||
OutboxPublishOutcome outcome =
|
||||
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join();
|
||||
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join().outcome();
|
||||
|
||||
// then
|
||||
assertThat(outcome).isEqualTo(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
|
||||
@@ -165,7 +255,12 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
|
||||
PlatformIntegrationEventPublishAdapter bridge =
|
||||
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
|
||||
|
||||
assertThat(bridge.publish(event("event-1", Optional.empty())).toCompletableFuture().join())
|
||||
assertThat(
|
||||
bridge
|
||||
.publish(event("event-1", Optional.empty()))
|
||||
.toCompletableFuture()
|
||||
.join()
|
||||
.outcome())
|
||||
.isEqualTo(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
|
||||
assertThat(publisher.calls).isZero();
|
||||
}
|
||||
@@ -192,7 +287,7 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
|
||||
|
||||
// when
|
||||
OutboxPublishOutcome outcome =
|
||||
bridge.publish(event(eventId, Optional.empty())).toCompletableFuture().join();
|
||||
bridge.publish(event(eventId, Optional.empty())).toCompletableFuture().join().outcome();
|
||||
|
||||
// then
|
||||
assertThat(outcome).isEqualTo(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
|
||||
|
||||
+26
-4
@@ -64,17 +64,39 @@ class NotificationCallbackProviderConfig {
|
||||
return cachingCertificates(uri -> fetchCertificate(client, uri));
|
||||
}
|
||||
|
||||
private static byte[] fetchCertificate(HttpClient client, URI uri) {
|
||||
/**
|
||||
* Fetches the certificate document at a host-checked SNS URL.
|
||||
*
|
||||
* <p>The status decides what a failed fetch says. The URL already passed the SNS host check, so
|
||||
* the host answering is SNS. A 4xx is SNS saying it has no such certificate: a real SNS message
|
||||
* names one that exists, so the envelope is forged — {@link IllegalArgumentException}, which the
|
||||
* verifier turns into an invalid signature that is refused and audited. It used to be "cannot
|
||||
* verify now" like every other non-2xx, so a forged envelope pointing at a certificate SNS never
|
||||
* issued got a 503, was never audited, and was logged as a provider hiccup (review 2026-11-03
|
||||
* RR-07-03). A 5xx, a 408 or 429 (slow or throttling host), or a network failure says nothing
|
||||
* about the certificate and stays "cannot verify now".
|
||||
*
|
||||
* @param client the no-redirect client
|
||||
* @param uri the host-checked certificate URL
|
||||
* @return the certificate document, at most {@link #MAX_SNS_CERTIFICATE_BYTES}
|
||||
* @throws IllegalArgumentException when SNS answers that the certificate does not exist
|
||||
* @throws CallbackVerificationUnavailableException when the certificate could not be fetched now
|
||||
*/
|
||||
static byte[] fetchCertificate(HttpClient client, URI uri) {
|
||||
HttpRequest request =
|
||||
HttpRequest.newBuilder(uri).timeout(SNS_CERTIFICATE_TIMEOUT).GET().build();
|
||||
try {
|
||||
HttpResponse<InputStream> response =
|
||||
client.send(request, HttpResponse.BodyHandlers.ofInputStream());
|
||||
try (InputStream body = response.body()) {
|
||||
if (response.statusCode() / 100 != 2) {
|
||||
int status = response.statusCode();
|
||||
if (status / 100 == 4 && status != 408 && status != 429) {
|
||||
throw new IllegalArgumentException(
|
||||
"SNS has no signing certificate at the URL the message names (HTTP " + status + ")");
|
||||
}
|
||||
if (status / 100 != 2) {
|
||||
throw certificateUnavailable(
|
||||
new IOException(
|
||||
"SNS signing certificate fetch failed with HTTP " + response.statusCode()));
|
||||
new IOException("SNS signing certificate fetch failed with HTTP " + status));
|
||||
}
|
||||
return body.readNBytes(MAX_SNS_CERTIFICATE_BYTES);
|
||||
}
|
||||
|
||||
+11
-3
@@ -16,6 +16,7 @@ import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
import java.util.Set;
|
||||
import tools.jackson.databind.JsonNode;
|
||||
|
||||
@@ -36,8 +37,12 @@ import tools.jackson.databind.JsonNode;
|
||||
* signed string), for up to {@link #SNS_HTTP_RETRY_HORIZON}. A window of five minutes threw away
|
||||
* every bounce and complaint that arrived after an outage longer than that, and with them the
|
||||
* suppressions they should have created (review 05-20). Accepting a replay inside the window costs
|
||||
* nothing: the provider event ledger is unique on {@code (provider_profile_id, provider_event_id)},
|
||||
* so a second delivery of the same event is a duplicate, not a second event.
|
||||
* nothing: the provider event ledger is unique on {@code (provider_profile_id, provider_event_id)}
|
||||
* whenever an event id is present, and this adapter gives every SES event the signed SNS {@code
|
||||
* MessageId} as that id, so a second delivery of the same message is a duplicate, not a second
|
||||
* event. It used to give none, and the only identity left was the {@code (provider_profile_id,
|
||||
* event_fingerprint)} index — a fingerprint keyed by the active HMAC key, so a redelivery after a
|
||||
* key rotation inside the window was recorded twice (review 2026-11-03 RR-07-07).
|
||||
*/
|
||||
public final class SesCallbackAdapter implements ProviderCallbackAdapter {
|
||||
|
||||
@@ -118,7 +123,10 @@ public final class SesCallbackAdapter implements ProviderCallbackAdapter {
|
||||
return List.of();
|
||||
}
|
||||
String message = envelope.getOrDefault("Message", "{}");
|
||||
return normalizer.normalize(message);
|
||||
// The SNS MessageId is in the signed string and survives every redelivery, so it — not a
|
||||
// fingerprint under the current HMAC key — is what the ledger recognises a replay by.
|
||||
return normalizer.normalize(
|
||||
message, Optional.ofNullable(envelope.get("MessageId")).filter(id -> !id.isBlank()));
|
||||
}
|
||||
|
||||
private boolean withinReplayWindow(String timestamp, Instant receivedAt) {
|
||||
|
||||
+44
-14
@@ -7,6 +7,7 @@ import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
import tools.jackson.databind.JsonNode;
|
||||
|
||||
@@ -19,8 +20,16 @@ import tools.jackson.databind.JsonNode;
|
||||
*/
|
||||
public final class SesEventNormalizer {
|
||||
|
||||
/** Normalize one SES notification payload. */
|
||||
public List<NormalizedProviderEvent> normalize(String payload) {
|
||||
/**
|
||||
* Normalize one SES notification payload.
|
||||
*
|
||||
* @param payload the SES event JSON (the SNS envelope's {@code Message})
|
||||
* @param providerEventId the identity of the delivery that carried it — the signed SNS {@code
|
||||
* MessageId}, which SNS keeps across redeliveries — or empty when the envelope had none
|
||||
* @return the provider events, each carrying {@code providerEventId}
|
||||
*/
|
||||
public List<NormalizedProviderEvent> normalize(String payload, Optional<String> providerEventId) {
|
||||
Objects.requireNonNull(providerEventId, "providerEventId");
|
||||
JsonNode root = NotificationJsonMapper.mapper().readTree(payload);
|
||||
String type = text(root, "eventType").orElse(text(root, "notificationType").orElse("Unknown"));
|
||||
Optional<String> messageId =
|
||||
@@ -30,24 +39,26 @@ public final class SesEventNormalizer {
|
||||
List<NormalizedProviderEvent> events = new ArrayList<>(1);
|
||||
events.add(
|
||||
switch (type) {
|
||||
case "Send" ->
|
||||
event(NormalizedEventType.PROVIDER_ACCEPTED, type, messageId, occurredAt, Map.of());
|
||||
case "Send" -> event(NormalizedEventType.PROVIDER_ACCEPTED, type, messageId, occurredAt);
|
||||
case "Delivery" ->
|
||||
event(NormalizedEventType.DELIVERY_CONFIRMED, type, messageId, occurredAt, Map.of());
|
||||
event(NormalizedEventType.DELIVERY_CONFIRMED, type, messageId, occurredAt);
|
||||
case "DeliveryDelay" ->
|
||||
event(NormalizedEventType.DELIVERY_DELAYED, type, messageId, occurredAt, Map.of());
|
||||
event(NormalizedEventType.DELIVERY_DELAYED, type, messageId, occurredAt);
|
||||
case "Bounce" -> bounce(root, type, messageId, occurredAt);
|
||||
case "Complaint" ->
|
||||
event(NormalizedEventType.COMPLAINT, type, messageId, occurredAt, Map.of());
|
||||
case "Complaint" -> event(NormalizedEventType.COMPLAINT, type, messageId, occurredAt);
|
||||
case "Reject" ->
|
||||
event(NormalizedEventType.PROVIDER_REJECTED, type, messageId, occurredAt, Map.of());
|
||||
event(NormalizedEventType.PROVIDER_REJECTED, type, messageId, occurredAt);
|
||||
case "RenderingFailure" ->
|
||||
event(NormalizedEventType.TEMPLATE_FAILURE, type, messageId, occurredAt, Map.of());
|
||||
case "Open" -> event(NormalizedEventType.OPENED, type, messageId, occurredAt, Map.of());
|
||||
case "Click" -> event(NormalizedEventType.CLICKED, type, messageId, occurredAt, Map.of());
|
||||
default -> event(NormalizedEventType.UNKNOWN, type, messageId, occurredAt, Map.of());
|
||||
event(NormalizedEventType.TEMPLATE_FAILURE, type, messageId, occurredAt);
|
||||
case "Open" -> event(NormalizedEventType.OPENED, type, messageId, occurredAt);
|
||||
case "Click" -> event(NormalizedEventType.CLICKED, type, messageId, occurredAt);
|
||||
default -> event(NormalizedEventType.UNKNOWN, type, messageId, occurredAt);
|
||||
});
|
||||
return List.copyOf(events);
|
||||
// One SNS delivery carries one SES event, so the delivery's id is the event's. It is what makes
|
||||
// a redelivery a duplicate in the ledger: the other identity, the event fingerprint, is keyed
|
||||
// by
|
||||
// the active HMAC key and changes when the key rotates (review 2026-11-03 RR-07-07).
|
||||
return events.stream().map(event -> withEventId(event, providerEventId)).toList();
|
||||
}
|
||||
|
||||
private static NormalizedProviderEvent bounce(
|
||||
@@ -68,6 +79,14 @@ public final class SesEventNormalizer {
|
||||
Map.of("bounceType", bounceType));
|
||||
}
|
||||
|
||||
private static NormalizedProviderEvent event(
|
||||
NormalizedEventType type,
|
||||
String nativeType,
|
||||
Optional<String> messageId,
|
||||
Optional<Instant> occurredAt) {
|
||||
return event(type, nativeType, messageId, occurredAt, Map.of());
|
||||
}
|
||||
|
||||
private static NormalizedProviderEvent event(
|
||||
NormalizedEventType type,
|
||||
String nativeType,
|
||||
@@ -78,6 +97,17 @@ public final class SesEventNormalizer {
|
||||
type, nativeType, Optional.empty(), messageId, occurredAt, attributes);
|
||||
}
|
||||
|
||||
private static NormalizedProviderEvent withEventId(
|
||||
NormalizedProviderEvent event, Optional<String> providerEventId) {
|
||||
return new NormalizedProviderEvent(
|
||||
event.type(),
|
||||
event.providerNativeType(),
|
||||
providerEventId,
|
||||
event.providerRequestId(),
|
||||
event.providerOccurredAt(),
|
||||
event.attributes());
|
||||
}
|
||||
|
||||
private static Optional<String> text(JsonNode node, String field) {
|
||||
JsonNode value = node.get(field);
|
||||
return value == null || value.isNull() ? Optional.empty() : Optional.of(value.asString());
|
||||
|
||||
+6
-1
@@ -112,7 +112,12 @@ public final class JavaMailSenderSmtpDispatch implements SmtpDispatch {
|
||||
} else if (cause instanceof SMTPSendFailedException send) {
|
||||
code = send.getReturnCode();
|
||||
}
|
||||
return code >= 200 && code < 600 ? Optional.of(code) : Optional.empty();
|
||||
// Only a refusal is a reply code here: 4xx (transient) and 5xx (permanent). Angus's sendpartial
|
||||
// path throws SMTPSendFailedException carrying the last reply, which can be a success (250);
|
||||
// read as a reply code it reached the classifier as a permanent refusal of a message the relay
|
||||
// had partly accepted. Anything else takes the conservative "may have been delivered" path
|
||||
// below (review 2026-11-03 RR-04-11).
|
||||
return code >= 400 && code < 600 ? Optional.of(code) : Optional.empty();
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+104
@@ -0,0 +1,104 @@
|
||||
package dev.caskeleton.adapter.outbound.notification.platform.autoconfigure;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import com.sun.net.httpserver.HttpServer;
|
||||
import dev.caskeleton.adapter.outbound.notification.platform.callback.CallbackVerificationUnavailableException;
|
||||
import java.io.IOException;
|
||||
import java.net.InetAddress;
|
||||
import java.net.InetSocketAddress;
|
||||
import java.net.ServerSocket;
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpClient;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
|
||||
/**
|
||||
* What a failed SNS signing-certificate fetch says about the callback (review 2026-11-03 RR-07-03).
|
||||
*
|
||||
* <p>The URL has passed the SNS host check before it is fetched, so the host is SNS's. A real SNS
|
||||
* message names a certificate that exists; when SNS itself answers 4xx for it, the envelope named a
|
||||
* certificate SNS never issued — a forged callback, to be rejected and audited. Only a failure that
|
||||
* says nothing about the certificate — a 5xx, a 408 or 429 (the host is slow or throttling), or a
|
||||
* network failure — is "cannot verify now" (503, the provider retries).
|
||||
*/
|
||||
class SnsCertificateFetchTest {
|
||||
|
||||
private final HttpClient client = HttpClient.newHttpClient();
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(ints = {403, 404, 410})
|
||||
@DisplayName("인증서 URL이 4xx로 답하면 없는 인증서를 가리킨 위조 콜백이므로 검증 실패로 거절된다")
|
||||
void aCertificateTheHostDoesNotHaveFailsVerification(int status) throws IOException {
|
||||
// given
|
||||
HttpServer server = answering(status);
|
||||
try {
|
||||
URI url = URI.create("http://127.0.0.1:" + server.getAddress().getPort() + "/cert.pem");
|
||||
|
||||
// when
|
||||
Throwable failure =
|
||||
catchThrowable(() -> NotificationCallbackProviderConfig.fetchCertificate(client, url));
|
||||
|
||||
// then
|
||||
assertThat(failure)
|
||||
.isInstanceOf(IllegalArgumentException.class)
|
||||
.isNotInstanceOf(CallbackVerificationUnavailableException.class);
|
||||
} finally {
|
||||
server.stop(0);
|
||||
}
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(ints = {408, 429, 500, 503})
|
||||
@DisplayName("인증서 URL이 5xx나 시간 초과·요청 제한(408·429)으로 답하면 아무것도 모르므로 지금 검증할 수 없다고 답한다")
|
||||
void aFailingCertificateHostLeavesTheCallbackUnverified(int status) throws IOException {
|
||||
// given
|
||||
HttpServer server = answering(status);
|
||||
try {
|
||||
URI url = URI.create("http://127.0.0.1:" + server.getAddress().getPort() + "/cert.pem");
|
||||
|
||||
// when
|
||||
Throwable failure =
|
||||
catchThrowable(() -> NotificationCallbackProviderConfig.fetchCertificate(client, url));
|
||||
|
||||
// then
|
||||
assertThat(failure).isInstanceOf(CallbackVerificationUnavailableException.class);
|
||||
} finally {
|
||||
server.stop(0);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("인증서 호스트에 닿지 못하면 지금 검증할 수 없다고 답한다")
|
||||
void anUnreachableCertificateHostLeavesTheCallbackUnverified() throws IOException {
|
||||
// given
|
||||
int closedPort;
|
||||
try (ServerSocket probe = new ServerSocket(0)) {
|
||||
closedPort = probe.getLocalPort();
|
||||
}
|
||||
URI url = URI.create("http://127.0.0.1:" + closedPort + "/cert.pem");
|
||||
|
||||
// when
|
||||
Throwable failure =
|
||||
catchThrowable(() -> NotificationCallbackProviderConfig.fetchCertificate(client, url));
|
||||
|
||||
// then
|
||||
assertThat(failure).isInstanceOf(CallbackVerificationUnavailableException.class);
|
||||
}
|
||||
|
||||
private static HttpServer answering(int status) throws IOException {
|
||||
HttpServer server =
|
||||
HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress(), 0), 0);
|
||||
server.createContext(
|
||||
"/",
|
||||
exchange -> {
|
||||
exchange.sendResponseHeaders(status, -1);
|
||||
exchange.close();
|
||||
});
|
||||
server.start();
|
||||
return server;
|
||||
}
|
||||
}
|
||||
+24
@@ -8,6 +8,7 @@ import dev.caskeleton.adapter.outbound.notification.platform.callback.CallbackVe
|
||||
import dev.caskeleton.adapter.outbound.notification.platform.callback.CallbackVerificationUnavailableException;
|
||||
import dev.caskeleton.adapter.outbound.notification.platform.callback.ProviderCallbackVerificationRequest;
|
||||
import dev.caskeleton.adapter.outbound.notification.platform.template.NotificationJsonMapper;
|
||||
import dev.caskeleton.application.notification.platform.api.NormalizedProviderEvent;
|
||||
import dev.caskeleton.application.notification.platform.api.ProviderId;
|
||||
import dev.caskeleton.application.notification.platform.api.ProviderProfileId;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
@@ -19,6 +20,7 @@ import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.Base64;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.Set;
|
||||
@@ -135,6 +137,28 @@ class SesCallbackAdapterTest {
|
||||
assertThat(unavailable.reasonCode()).isEqualTo("SNS_CERTIFICATE_UNAVAILABLE"));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName(
|
||||
"SES 이벤트의 provider event id는 서명된 SNS MessageId라 fingerprint 키를 회전해도 같은 콜백이 같은 이벤트로 남는다")
|
||||
void theSignedSnsMessageIdIsTheProviderEventId() {
|
||||
// given
|
||||
CallbackVerificationResult verified =
|
||||
adapter.verify(request(signed(ALLOWED_TOPIC, RECEIVED_AT)));
|
||||
|
||||
// when
|
||||
List<NormalizedProviderEvent> events =
|
||||
adapter.normalize(verified.verifiedCallback().orElseThrow());
|
||||
|
||||
// then
|
||||
// The ledger is unique on (profile, provider_event_id) whenever an id is present. Without one
|
||||
// the only identity was a fingerprint keyed by the active HMAC key, so a redelivery after a
|
||||
// key rotation inside the one-hour replay window became a second row (RR-07-07).
|
||||
assertThat(events)
|
||||
.singleElement()
|
||||
.extracting(NormalizedProviderEvent::providerEventId)
|
||||
.isEqualTo(Optional.of("sns-message-1"));
|
||||
}
|
||||
|
||||
/** The adapter with the window a deployment gets when it sets nothing. */
|
||||
private static SesCallbackAdapter withDefaultWindow() {
|
||||
return new SesCallbackAdapter(
|
||||
|
||||
+34
@@ -16,9 +16,11 @@ import java.nio.charset.StandardCharsets;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import java.util.Properties;
|
||||
import org.eclipse.angus.mail.smtp.SMTPSendFailedException;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.mail.MailAuthenticationException;
|
||||
import org.springframework.mail.MailSendException;
|
||||
import org.springframework.mail.javamail.JavaMailSenderImpl;
|
||||
|
||||
/**
|
||||
@@ -136,6 +138,38 @@ class JavaMailSenderSmtpDispatchTest {
|
||||
.containsExactly(Optional.empty(), false);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("성공 코드(250)를 실은 부분 전송 실패는 거절 reply code가 아니라 전달됐을 수 있는 실패로 보고된다")
|
||||
void aPartialSendCarryingASuccessCodeIsNotReportedAsARefusal() {
|
||||
// given
|
||||
// Angus's sendpartial path reports a partly delivered message as SMTPSendFailedException
|
||||
// carrying the last reply, which is a success (250), not a refusal.
|
||||
JavaMailSenderImpl partlyDelivered =
|
||||
new JavaMailSenderImpl() {
|
||||
@Override
|
||||
public void send(MimeMessage... messages) {
|
||||
throw new MailSendException(
|
||||
"partly sent",
|
||||
new SMTPSendFailedException(
|
||||
"DATA",
|
||||
250,
|
||||
"250 2.0.0 queued",
|
||||
null,
|
||||
new jakarta.mail.Address[0],
|
||||
new jakarta.mail.Address[0],
|
||||
new jakarta.mail.Address[0]));
|
||||
}
|
||||
};
|
||||
JavaMailSenderSmtpDispatch dispatch = new JavaMailSenderSmtpDispatch(partlyDelivered);
|
||||
|
||||
// when
|
||||
SmtpDispatchException failure = sendExpectingFailure(dispatch);
|
||||
|
||||
// then
|
||||
assertThat(List.of(failure.replyCode(), failure.dataCommitted()))
|
||||
.containsExactly(Optional.empty(), true);
|
||||
}
|
||||
|
||||
private static SmtpDispatchException sendExpectingFailure(JavaMailSenderSmtpDispatch dispatch) {
|
||||
return catchThrowableOfType(SmtpDispatchException.class, () -> dispatch.send(message()));
|
||||
}
|
||||
|
||||
@@ -155,9 +155,9 @@ strictTestLanes {
|
||||
}
|
||||
}
|
||||
|
||||
// The MinIO lanes start a container, so they are the main-branch stage of this leaf, not its
|
||||
// pull-request `check` (review BUILD-계획-B3); object-storage-qualification.yml runs them on pull
|
||||
// requests that touch object storage, and integrationCheck runs them with every integration lane.
|
||||
tasks.named('mainCheck') {
|
||||
dependsOn 'objectStorageMinioContractTest', 'objectStorageMinioFaultTest'
|
||||
}
|
||||
// The MinIO lanes start a container, so they are not this leaf's pull-request `check` (review
|
||||
// BUILD-계획-B3). They are integration lanes, and the root integrationCheck owns every container lane
|
||||
// on main and nightly; object-storage-qualification.yml runs them on pull requests that touch object
|
||||
// storage. This leaf's `mainCheck` used to run them too, so each main push ran them twice in two
|
||||
// parallel jobs (review 2026-11-03 RR-06-09, original 08-04 (d)).
|
||||
|
||||
|
||||
@@ -46,12 +46,18 @@ in `platform`), so the two capability leaves read the same way (review 04-16).
|
||||
write; Spring Data repositories are mocks).
|
||||
- `FileserverPersistenceActivationTest` — the scan covers this package only, behind
|
||||
`app.fileserver-platform.enabled`, and runs before the JPA root auto-configuration.
|
||||
- `repository.CleanupClaimSqlTest` — the claim keeps its one-statement shape (the Claims rule).
|
||||
- `repository.CleanupClaimSqlTest` — the claim's text keeps its one-statement shape (the Claims
|
||||
rule); the locking itself is checked on PostgreSQL by
|
||||
`PostgreSqlFileserverReclamationIntegrationTest`.
|
||||
- `entity.AssignedIdEntityTest` — newness from `Persistable`, on Hibernate over H2.
|
||||
A new store write adds its case to the guard test; a new auto-configuration adds its line to the
|
||||
activation test.
|
||||
- Port contracts live in `application-fileserver` testFixtures (`*PortContract`). The in-memory
|
||||
fakes and the PostgreSQL tests both extend them.
|
||||
- Port contracts live in `application-fileserver` testFixtures (`*PortContract`, and
|
||||
`FileQuotaLedgerContract` for the three quota ports, which share one table). The in-memory fakes
|
||||
and the PostgreSQL tests both extend them; every port this leaf implements has one
|
||||
(`docs/testing/TESTING_STRATEGY.md` 9.3). The PostgreSQL side of the seven added in review
|
||||
2026-11-03 RR-01-04 is tagged `jpa-contract` and runs in persistence-jpa's
|
||||
`jpaPlatformContractTest` lane on `FileserverPostgreSqlStack`, whose clock a case moves.
|
||||
- The real-PostgreSQL tests are still in `persistence-jpa`'s `postgresqlIntegrationTest` source
|
||||
set, which owns the container and Flyway wiring, and the JPA readiness card
|
||||
(`config/jpa/readiness-cards.yaml`, checked by `JpaReadinessRegistryTest`) names them as that
|
||||
|
||||
+5
-2
@@ -18,8 +18,11 @@ import java.util.UUID;
|
||||
/**
|
||||
* JPA-backed {@link FileQuotaPort}.
|
||||
*
|
||||
* <p>Reservation, extension, commit, and release are conditional statements, so a reservation that
|
||||
* already expired or was released can never be extended or committed.
|
||||
* <p>Reservation, extension, commit, and release are conditional statements: a reservation that
|
||||
* expired or was released can never be extended, and a released or already committed one can never
|
||||
* be committed. An expired reservation that was not released still commits — bytes that were
|
||||
* written are durable usage whether or not the reservation outlived its TTL. {@code
|
||||
* FileQuotaLedgerContract} holds this adapter and the in-memory ledger to the same cases.
|
||||
*/
|
||||
public class JpaFileQuota implements FileQuotaPort {
|
||||
|
||||
|
||||
+2
-12
@@ -15,8 +15,8 @@ package dev.caskeleton.adapter.outbound.persistence.fileserver.repository;
|
||||
* every claimed item is fenced on its own.
|
||||
*
|
||||
* <p>Package-private: the repository is its one reader, and the leaf's public surface is a ceiling.
|
||||
* {@link #positional(String)} rewrites the named parameters to JDBC placeholders for a test in this
|
||||
* package that runs the identical text on a bare {@code Connection}.
|
||||
* The PostgreSQL lane runs it through the repository ({@code
|
||||
* PostgreSqlFileserverReclamationIntegrationTest}); nothing runs the text on its own.
|
||||
*/
|
||||
final class CleanupClaimSql {
|
||||
|
||||
@@ -43,14 +43,4 @@ final class CleanupClaimSql {
|
||||
+ " c.file_id, c.upload_id, c.content_key";
|
||||
|
||||
private CleanupClaimSql() {}
|
||||
|
||||
/**
|
||||
* Rewrites the named parameters to JDBC placeholders, left to right.
|
||||
*
|
||||
* @param namedSql the statement above
|
||||
* @return the same statement with {@code ?} in place of every {@code :name}
|
||||
*/
|
||||
static String positional(String namedSql) {
|
||||
return namedSql.replaceAll("(?<!:):[A-Za-z][A-Za-z0-9]*", "?");
|
||||
}
|
||||
}
|
||||
|
||||
+9
-18
@@ -7,16 +7,17 @@ import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* The claim statement keeps the shape the leaf's claim rule names: one statement that locks the
|
||||
* rows it picks, skips rows another claim holds, gives every row a fresh token and a higher fence,
|
||||
* and returns them in the same round trip. The PostgreSQL lane runs it; this pins the shape without
|
||||
* a database, so a rewrite into a select-then-update fails here first.
|
||||
* The claim statement's text keeps the shape the leaf's claim rule names: one statement that picks
|
||||
* rows with {@code FOR UPDATE SKIP LOCKED}, gives every row a fresh token and a higher fence, and
|
||||
* returns them. This pins the text without a database, so a rewrite into a select-then-update fails
|
||||
* here first. Whether the statement actually locks, skips and fences is a behaviour of PostgreSQL,
|
||||
* and {@code PostgreSqlFileserverReclamationIntegrationTest} runs it there.
|
||||
*/
|
||||
class CleanupClaimSqlTest {
|
||||
|
||||
@Test
|
||||
@DisplayName("claim은 고른 행을 잠그고, 다른 claim이 잡은 행은 기다리지 않고 건너뛴다")
|
||||
void theClaimLocksWhatItPicksAndSkipsWhatAnotherClaimHolds() {
|
||||
@DisplayName("claim 문장은 한 문장의 CTE 안에서 FOR UPDATE SKIP LOCKED로 행을 고른다")
|
||||
void theClaimTextPicksRowsWithForUpdateSkipLockedInOneStatement() {
|
||||
// when
|
||||
String claim = CleanupClaimSql.CLAIM_DUE;
|
||||
|
||||
@@ -25,8 +26,8 @@ class CleanupClaimSqlTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("claim한 행마다 새 토큰과 하나 높은 fence를 받고 같은 왕복에 돌아온다")
|
||||
void everyClaimedRowGetsAFreshTokenAndAHigherFenceInTheSameRoundTrip() {
|
||||
@DisplayName("claim 문장은 행마다 새 토큰을 쓰고 fence를 하나 올리고 RETURNING으로 돌려준다")
|
||||
void theClaimTextWritesAFreshTokenRaisesTheFenceAndReturnsTheRows() {
|
||||
// when
|
||||
String claim = CleanupClaimSql.CLAIM_DUE;
|
||||
|
||||
@@ -38,14 +39,4 @@ class CleanupClaimSqlTest {
|
||||
claim.contains("RETURNING c.cleanup_id, c.claim_token")))
|
||||
.containsOnly(true);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("이름 있는 매개변수를 JDBC 자리표시자로 바꿀 때 PostgreSQL 형 변환은 건드리지 않는다")
|
||||
void positionalParametersDoNotRewritePostgreSqlCasts() {
|
||||
// when
|
||||
String positional = CleanupClaimSql.positional("payload::jsonb @> :expected::jsonb");
|
||||
|
||||
// then
|
||||
assertThat(positional).isEqualTo("payload::jsonb @> ?::jsonb");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -235,7 +235,11 @@ made Spring call `getTransactionIsolation()` at every begin, which pgjdbc answer
|
||||
|
||||
Every mode is time-bounded (review 04-10). Spring hands the remaining time to each statement, so a
|
||||
slow query or lock wait is cancelled by the server (`57014` → `DB_QUERY_CANCELED`), and a deadline
|
||||
that passed before the next statement (`TransactionTimedOutException`) gets the same code.
|
||||
that passed before the next statement gets the same code: Spring's `TransactionTimedOutException`
|
||||
on the JDBC path, and on the JPA path (`JpaTransactionManager`, production) Hibernate's own
|
||||
`TransactionException`, which has no SQLState and is classified because `SpringTransaction` knows
|
||||
its deadline passed (`translateAfterDeadline`, review RR-03-03). A business exception thrown after
|
||||
the deadline is passed on unchanged.
|
||||
|
||||
Pre-built templates are immutable after construction so concurrent callers cannot
|
||||
observe each other's reconfiguration. `inRootWrite` reuses the pre-built write template,
|
||||
@@ -302,7 +306,9 @@ the two paths disagree on any state.
|
||||
|
||||
**No SQLState, classified by type:** a lost `@Version` race (`OptimisticLockingFailureException`,
|
||||
JPA `OptimisticLockException`) → `DB_OPTIMISTIC_CONFLICT` (409, review 04-05); an expired Spring
|
||||
transaction deadline (`TransactionTimedOutException`) → `DB_QUERY_CANCELED`.
|
||||
transaction deadline (`TransactionTimedOutException`) → `DB_QUERY_CANCELED`; any other persistence
|
||||
failure (JPA, Spring data-access or Spring transaction exception) of a transaction
|
||||
`SpringTransaction` began, once its deadline has passed → `DB_QUERY_CANCELED`.
|
||||
|
||||
**Commit phase (`translateCommitFailure`, ADR-JPA-003):** once the action has returned and
|
||||
`SpringTransaction` is committing a write transaction it began, a failure with `08*`, `40003`, a
|
||||
|
||||
@@ -296,14 +296,26 @@ A가 FAILED로 바꾸면 다시 발행되고(중복 전달), A의 늦은 PUBLISH
|
||||
`WHERE event_id = ? AND status = 'IN_FLIGHT' AND lease_owner = ? AND lease_token = ?`.
|
||||
owner는 비우고 token은 그대로 둔다(token은 증가만 한다). 1 row면 `APPLIED`, 0 row면 `STALE`.
|
||||
`findById` + setter 경로와 `@Version`은 쓰지 않는다.
|
||||
- **실패 원인 코드(`V22`)**: `markFailed`·`markDeferred`·`markDead`는 같은 조건부 UPDATE에서
|
||||
`last_failure_code`를 쓴다(publisher가 붙인 코드, 없으면 결과·예외 타입 이름). entity 필드는 읽기
|
||||
전용이라 append INSERT와 claim flush는 이 열을 건드리지 않는다. `longRetries(n, now)`는 미룬 claim이
|
||||
n번 이상인 FAILED 행 수와 가장 오래된 행의 나이를 센다(`outbox.retry.long.*` 게이지). 상태 `PARKED`는
|
||||
DDL이 없다 — claim이 PENDING·FAILED·IN_FLIGHT만 고르고, FIFO 게이트는 PUBLISHED 아닌 모든 행을 막는 앞
|
||||
행으로 본다. claim이 시도 횟수가 어긋난 행(`deferred_attempt_count > attempt_count`)을 만나면 DEAD로
|
||||
옮기고 WARN `error.reason=ATTEMPT_COUNTERS_INCONSISTENT`로 원인을 말한다(재리뷰 2026-11-03 RR-05-01·03).
|
||||
- **미룬 실패와 claim 해제(`V20`)**: `markDeferred(lease, next)`는 `markFailed`와 같은 전이에
|
||||
`deferred_attempt_count + 1`을 더한다(일시적 실패는 시도 예산 밖). `releaseLease(lease, at)`는
|
||||
손대지 않은 claim을 `PENDING`, `next_attempt_at = at`, `attempt_count - 1`로 되돌린다(종료 중인
|
||||
relay가 쓴다). 둘 다 같은 lease 조건부 UPDATE다.
|
||||
- **aggregate 순서 번호(`V21`)**: `JpaOutboxStore`는 `OutboxAggregateSequencePort`도 구현한다.
|
||||
`outbox_aggregate_sequence` counter를 `UPDATE ... + 1`(row lock)로 올리고, 없으면 그 aggregate의 기존
|
||||
outbox 최대 `aggregate_sequence` + 1로 만든다. V21은 `outbox_event`의 순서 키 유일 인덱스도 만든다 —
|
||||
기존 중복 row가 있으면 migration이 실패하므로 먼저 정리해야 한다.
|
||||
`outbox_aggregate_sequence` counter를 `UPDATE`(row lock)로 올린다. 올린 값은 그 aggregate의 가장 큰
|
||||
outbox 번호보다 작아지지 않는다(rolling 배포 중 옛 인스턴스가 쓴 벽시계 번호가 먼저 온다). counter가
|
||||
없으면 `INSERT ... ON CONFLICT DO NOTHING`으로 만든다. 시작값은 남은 outbox 최대 번호와 V21이
|
||||
기록한 floor(`outbox_aggregate_sequence_floor`, V21 적용 시각의 epoch 마이크로초) 중 큰 값 + 1이다.
|
||||
첫 쓰기 두 개가 겹치면 두 번째는 첫 번째 커밋을 기다렸다가 아무것도 넣지 않고 증가만 한다(리뷰
|
||||
RR-03-01). V21은 남은 row가 있는 aggregate의 counter를 그 최대 번호로 미리 만들고, `outbox_event`의
|
||||
순서 키 유일 인덱스도 만든다 — 기존 중복 row가 있으면 migration이 실패하므로 먼저 정리해야 한다
|
||||
(`docs/runbooks/outbox-dead-letter.md` §8).
|
||||
- **STALE은 예외가 아니다**: row의 결과는 지금 lease를 가진 relay가 쓴다. use case는 결과를
|
||||
`OutboxRelayResult.Outcome.STALE`로 남기고 `OutboxRelayFailureReportPort.reportStaleCompletion`
|
||||
으로 알린 뒤 다음 이벤트로 넘어간다. FAILED/DEAD report는 보내지 않는다(아무것도 저장되지
|
||||
|
||||
@@ -187,6 +187,7 @@ def readinessLanes = [
|
||||
postgresqlOutboxRowLifecycleIntegrationTest : ['PostgreSqlOutboxRowLifecycleIntegrationTest'],
|
||||
postgresqlOutboxAggregateSequenceIntegrationTest : ['PostgreSqlOutboxAggregateSequenceIntegrationTest'],
|
||||
postgresqlOutboxOrderKeyRunbookIntegrationTest : ['PostgreSqlOutboxOrderKeyRunbookIntegrationTest'],
|
||||
postgresqlOutboxDispositionRunbookIntegrationTest : ['PostgreSqlOutboxDispositionRunbookIntegrationTest'],
|
||||
postgresqlFileserverMigrationIntegrationTest : ['PostgreSqlFileserverMigrationIntegrationTest'],
|
||||
postgresqlFileserverMetadataIntegrationTest : ['PostgreSqlFileserverMetadataStoreIntegrationTest'],
|
||||
postgresqlFileserverReclamationIntegrationTest : ['PostgreSqlFileserverReclamationIntegrationTest'],
|
||||
@@ -243,13 +244,20 @@ strictTestLanes {
|
||||
}
|
||||
}
|
||||
|
||||
// The V21 pre-deploy check and cleanup are SQL an operator runs from the outbox runbook (review
|
||||
// 2026-11-01, README 15 decision). The lane reads them out of the runbook, so the document is one of
|
||||
// its inputs: editing either block reruns the lane against real PostgreSQL.
|
||||
tasks.named('postgresqlOutboxOrderKeyRunbookIntegrationTest', Test) {
|
||||
def outboxRunbook = new File(rootProject.projectDir.parentFile, 'docs/runbooks/outbox-dead-letter.md')
|
||||
inputs.file(outboxRunbook).withPathSensitivity(PathSensitivity.RELATIVE).withPropertyName('outboxRunbook')
|
||||
systemProperty 'outbox.runbook', outboxRunbook.absolutePath
|
||||
// The outbox runbooks give an operator SQL to run: the V21 pre-deploy check and cleanup (review
|
||||
// 2026-11-01, README 15 decision), the manual dispositions of a DEAD row (re-review 2026-11-03
|
||||
// RR-05-01), and finding, parking, resuming and skipping a row that is retried without end
|
||||
// (RR-05-03). Each lane reads its blocks out of the runbooks, so the documents are its inputs:
|
||||
// editing a block reruns the lane against real PostgreSQL.
|
||||
['postgresqlOutboxOrderKeyRunbookIntegrationTest', 'postgresqlOutboxDispositionRunbookIntegrationTest'].each { laneName ->
|
||||
tasks.named(laneName, Test) {
|
||||
def outboxRunbook = new File(rootProject.projectDir.parentFile, 'docs/runbooks/outbox-dead-letter.md')
|
||||
inputs.file(outboxRunbook).withPathSensitivity(PathSensitivity.RELATIVE).withPropertyName('outboxRunbook')
|
||||
systemProperty 'outbox.runbook', outboxRunbook.absolutePath
|
||||
def publishFailedRunbook = new File(rootProject.projectDir.parentFile, 'docs/runbooks/outbox-publish-failed.md')
|
||||
inputs.file(publishFailedRunbook).withPathSensitivity(PathSensitivity.RELATIVE).withPropertyName('publishFailedRunbook')
|
||||
systemProperty 'outbox.publish-failed.runbook', publishFailedRunbook.absolutePath
|
||||
}
|
||||
}
|
||||
|
||||
tasks.named('postgresqlSecurityBaselineIntegrationTest') {
|
||||
|
||||
+9
-2
@@ -46,8 +46,15 @@ public enum FailureCategory {
|
||||
QUERY_TIMEOUT,
|
||||
|
||||
/**
|
||||
* {@code 25P03} — the server ended a session that sat idle inside an open transaction. Nothing
|
||||
* was committed, which is why its code is retryable where {@link #QUERY_TIMEOUT}'s is not.
|
||||
* Not a transaction timeout: {@code 25P03}, the server ended a session that sat idle inside an
|
||||
* open transaction ({@code idle_in_transaction_session_timeout}). Nothing was committed, which is
|
||||
* why its code ({@code DB_IDLE_IN_TX_TIMEOUT}) is retryable.
|
||||
*
|
||||
* <p>A transaction that ran past its own deadline ({@code SpringTransaction.Timeouts}) is {@link
|
||||
* #QUERY_TIMEOUT} ({@code DB_QUERY_CANCELED}), not this. The name says otherwise and is kept only
|
||||
* because the enum is public API ({@code docs/architecture/jpa-api-surface.txt}); it becomes
|
||||
* {@code IDLE_IN_TRANSACTION_TIMEOUT}, with {@link TransactionTimeoutException}, at the next
|
||||
* release that breaks this API anyway (review RR-03-10).
|
||||
*/
|
||||
TRANSACTION_TIMEOUT,
|
||||
|
||||
|
||||
+6
-3
@@ -1,10 +1,13 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.api.error;
|
||||
|
||||
/**
|
||||
* A transaction exceeded its configured transaction timeout (design §15.4).
|
||||
* The server ended a session that sat idle inside an open transaction ({@code 25P03}, {@link
|
||||
* FailureCategory#TRANSACTION_TIMEOUT}). Not a transaction that ran past its own deadline: that is
|
||||
* a cancelled statement ({@link FailureCategory#QUERY_TIMEOUT}).
|
||||
*
|
||||
* <p>The transaction is rolled back; whether the use case may be re-run is a business decision, so
|
||||
* the platform does not retry it automatically.
|
||||
* <p>The transaction is rolled back and nothing was committed, so its code is retryable; whether
|
||||
* the use case is re-run is still the caller's decision, and the platform does not retry it
|
||||
* automatically. The name is kept until the next API-breaking release (review RR-03-10).
|
||||
*/
|
||||
public final class TransactionTimeoutException extends JpaPersistenceException {
|
||||
|
||||
|
||||
+38
@@ -8,6 +8,7 @@ import dev.caskeleton.shared.error.ApiErrorCode;
|
||||
import dev.caskeleton.shared.error.OperationalError;
|
||||
import dev.caskeleton.shared.error.PersistenceFailureException;
|
||||
import jakarta.persistence.OptimisticLockException;
|
||||
import jakarta.persistence.PersistenceException;
|
||||
import java.io.IOException;
|
||||
import java.sql.SQLException;
|
||||
import java.sql.SQLNonTransientConnectionException;
|
||||
@@ -23,6 +24,7 @@ import java.util.Optional;
|
||||
import java.util.Set;
|
||||
import org.springframework.dao.DataAccessException;
|
||||
import org.springframework.dao.OptimisticLockingFailureException;
|
||||
import org.springframework.transaction.TransactionException;
|
||||
import org.springframework.transaction.TransactionTimedOutException;
|
||||
|
||||
/**
|
||||
@@ -140,6 +142,42 @@ public final class PersistenceExceptionTranslator {
|
||||
return Optional.empty();
|
||||
}
|
||||
|
||||
/**
|
||||
* Classify a failure raised once the transaction's deadline had passed, when neither a SQLState
|
||||
* nor a type above identifies it.
|
||||
*
|
||||
* <p>Past the deadline Hibernate refuses the next statement itself, before the driver is asked:
|
||||
* an {@code org.hibernate.TransactionException} with no SQLState that is not Spring's {@link
|
||||
* TransactionTimedOutException} — in the action as it is, at commit wrapped in a {@code
|
||||
* JpaSystemException} (review RR-03-03). That is the production path ({@code
|
||||
* JpaTransactionManager}); the JDBC path gets {@code 57014} from the server instead. Both are
|
||||
* {@code DB_QUERY_CANCELED}. The caller decides that the deadline has passed; this method only
|
||||
* makes sure the failure came from the persistence stack — a JPA, Spring data-access or Spring
|
||||
* transaction exception anywhere in the cause chain — so a business exception thrown late is
|
||||
* passed on as it is, and one already classified is not classified again.
|
||||
*/
|
||||
public Optional<PersistenceFailureException> translateAfterDeadline(Throwable ex) {
|
||||
Objects.requireNonNull(ex, "ex");
|
||||
if (ex instanceof PersistenceFailureException) {
|
||||
return Optional.empty();
|
||||
}
|
||||
for (Throwable t = ex; t != null; t = t.getCause()) {
|
||||
if (t instanceof PersistenceException
|
||||
|| t instanceof DataAccessException
|
||||
|| t instanceof TransactionException) {
|
||||
return Optional.of(
|
||||
new PersistenceFailureException(
|
||||
OperationalError.DB_QUERY_CANCELED,
|
||||
"persistence failure after the transaction deadline passed",
|
||||
ex));
|
||||
}
|
||||
if (t.getCause() == t) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
return Optional.empty();
|
||||
}
|
||||
|
||||
/**
|
||||
* Classify a failure raised while committing a write transaction this call began.
|
||||
*
|
||||
|
||||
+76
-18
@@ -14,6 +14,7 @@ import dev.caskeleton.application.outbox.OutboxAppendPort;
|
||||
import dev.caskeleton.application.outbox.OutboxCompletion;
|
||||
import dev.caskeleton.application.outbox.OutboxEventStatus;
|
||||
import dev.caskeleton.application.outbox.OutboxLease;
|
||||
import dev.caskeleton.application.outbox.OutboxLongRetries;
|
||||
import dev.caskeleton.application.outbox.OutboxStorePort;
|
||||
import dev.caskeleton.shared.digest.Sha256;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
@@ -29,6 +30,7 @@ import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.slf4j.spi.LoggingEventBuilder;
|
||||
|
||||
/**
|
||||
* JPA-backed {@link OutboxAppendPort} + {@link OutboxStorePort}. {@link #append} and the claim/mark
|
||||
@@ -103,12 +105,21 @@ public class JpaOutboxStore
|
||||
*
|
||||
* <p>The increment is an {@code UPDATE}, which holds the counter row's lock until the caller
|
||||
* commits: a second writer of the same aggregate waits, then increments the committed value, so
|
||||
* the numbers follow commit order. An aggregate's first number is inserted instead, seeded after
|
||||
* the highest {@code aggregate_sequence} its outbox rows already carry — rows numbered before the
|
||||
* counter existed (the sample used epoch microseconds) must still come first. Two writers
|
||||
* creating the same counter at once collide on its primary key and one transaction fails: a
|
||||
* rollback the caller retries, never two events with one number. The counter outlives the
|
||||
* reaper's deletes, so a number is never handed out twice.
|
||||
* the numbers follow commit order. It never goes below the aggregate's highest outbox row, so a
|
||||
* row an instance still on the wall-clock scheme writes during a rolling deploy comes first. The
|
||||
* counter outlives the reaper's deletes, so a number is never handed out twice.
|
||||
*
|
||||
* <p>An aggregate's first number is inserted instead, after the larger of its highest remaining
|
||||
* row and the floor V21 recorded — the moment it ran, above every clock number already sent for
|
||||
* rows the reaper has deleted or that were written under an identity that has since changed
|
||||
* (review RR-03-02). Two writers creating the same counter at once both succeed: the insert is
|
||||
* {@code ON CONFLICT DO NOTHING}, so the second waits for the first to commit, writes nothing,
|
||||
* and increments the committed counter instead (review RR-03-01).
|
||||
*
|
||||
* <p>Statements rather than one {@code INSERT … ON CONFLICT DO UPDATE … RETURNING} like {@code
|
||||
* JpaLiveEventReplay}'s position allocator: the sample appends events on the H2 developer run
|
||||
* ({@code bootRunH2}), and H2 accepts {@code ON CONFLICT DO NOTHING} but not {@code DO UPDATE}.
|
||||
* The common case — the counter exists — is still one {@code UPDATE} and one read.
|
||||
*/
|
||||
@Override
|
||||
public long nextSequence(AggregateIdentity aggregate) {
|
||||
@@ -118,8 +129,11 @@ public class JpaOutboxStore
|
||||
String tenant = aggregate.tenantScope();
|
||||
String type = aggregate.aggregateType();
|
||||
String id = aggregate.aggregateId();
|
||||
if (repository.incrementAggregateSequence(tenant, type, id) == 0) {
|
||||
repository.createAggregateSequence(tenant, type, id);
|
||||
if (repository.incrementAggregateSequence(tenant, type, id) == 0
|
||||
&& repository.createAggregateSequence(tenant, type, id) == 0) {
|
||||
// Another writer created the counter after our UPDATE found none; its commit ended our wait
|
||||
// in the insert, so the counter is there to increment now.
|
||||
repository.incrementAggregateSequence(tenant, type, id);
|
||||
}
|
||||
return repository.aggregateSequence(tenant, type, id);
|
||||
}
|
||||
@@ -132,6 +146,11 @@ public class JpaOutboxStore
|
||||
* instead rolled the whole claim back, and the next claim selected the same row first again, so
|
||||
* one bad row stopped the relay for good. DEAD is fail-closed: the row is never published, and it
|
||||
* waits for the DEAD-row runbook like any other.
|
||||
*
|
||||
* <p>The WARN names the cause. A row whose deferred claims outnumber its claims is not damaged:
|
||||
* it is what a manual re-drive that reset {@code attempt_count} and left {@code
|
||||
* deferred_attempt_count} produces (re-review 2026-11-03 RR-05-01), and "cannot read" sent the
|
||||
* operator looking for corruption instead of re-running the corrected runbook SQL.
|
||||
*/
|
||||
@Override
|
||||
public List<OutboxLease> claimBatch(
|
||||
@@ -145,13 +164,26 @@ public class JpaOutboxStore
|
||||
entity.setNextAttemptAt(now.plus(inFlightTimeout));
|
||||
entity.setLeaseOwner(owner);
|
||||
entity.setLeaseToken(entity.getLeaseToken() + 1);
|
||||
if (entity.getDeferredAttemptCount() > entity.getAttemptCount()) {
|
||||
deadLetterOnClaim(entity)
|
||||
.addKeyValue("error.reason", "ATTEMPT_COUNTERS_INCONSISTENT")
|
||||
.log(
|
||||
"outbox claim moved a row to DEAD: deferred_attempt_count ({}) exceeds"
|
||||
+ " attempt_count ({}); a manual re-drive that did not reset"
|
||||
+ " deferred_attempt_count leaves this behind — re-drive it with the runbook"
|
||||
+ " SQL",
|
||||
entity.getDeferredAttemptCount(),
|
||||
entity.getAttemptCount());
|
||||
continue;
|
||||
}
|
||||
ClaimedOutboxEvent event;
|
||||
try {
|
||||
event = toClaimedOutboxEvent(entity);
|
||||
} catch (RuntimeException unreadable) {
|
||||
entity.setStatus(OutboxEventStatus.DEAD.name());
|
||||
entity.setLeaseOwner(null);
|
||||
log.warn("outbox claim moved a row it cannot read to DEAD", unreadable);
|
||||
deadLetterOnClaim(entity)
|
||||
.addKeyValue("error.reason", "UNREADABLE_ROW")
|
||||
.setCause(unreadable)
|
||||
.log("outbox claim moved a row it cannot read as an event to DEAD");
|
||||
continue;
|
||||
}
|
||||
leases.add(new OutboxLease(event, owner, entity.getLeaseToken()));
|
||||
@@ -159,6 +191,15 @@ public class JpaOutboxStore
|
||||
return List.copyOf(leases);
|
||||
}
|
||||
|
||||
/** Moves a claimed row it will not hand out to DEAD and starts the WARN that says why. */
|
||||
private static LoggingEventBuilder deadLetterOnClaim(OutboxEventEntity entity) {
|
||||
entity.setStatus(OutboxEventStatus.DEAD.name());
|
||||
entity.setLeaseOwner(null);
|
||||
return log.atWarn()
|
||||
.addKeyValue("event_id", entity.getEventId())
|
||||
.addKeyValue("runbook_link", "runbook://outbox/dead-letter");
|
||||
}
|
||||
|
||||
/**
|
||||
* One conditional UPDATE, no load-and-set: it applies only while the lease still holds the row
|
||||
* (see README "outbox"). Zero rows is {@link OutboxCompletion#STALE}, not an error — the relay
|
||||
@@ -174,18 +215,21 @@ public class JpaOutboxStore
|
||||
|
||||
/** Marks the leased row FAILED with its next attempt time; see {@link #markPublished}. */
|
||||
@Override
|
||||
public OutboxCompletion markFailed(OutboxLease lease, Instant nextAttemptAt) {
|
||||
public OutboxCompletion markFailed(OutboxLease lease, Instant nextAttemptAt, String failureCode) {
|
||||
requireTransaction("markFailed");
|
||||
return completion(
|
||||
repository.failLease(lease.eventId(), lease.owner(), lease.token(), nextAttemptAt));
|
||||
repository.failLease(
|
||||
lease.eventId(), lease.owner(), lease.token(), nextAttemptAt, failureCode));
|
||||
}
|
||||
|
||||
/** Marks the leased row FAILED as a deferred claim; see {@link #markPublished}. */
|
||||
@Override
|
||||
public OutboxCompletion markDeferred(OutboxLease lease, Instant nextAttemptAt) {
|
||||
public OutboxCompletion markDeferred(
|
||||
OutboxLease lease, Instant nextAttemptAt, String failureCode) {
|
||||
requireTransaction("markDeferred");
|
||||
return completion(
|
||||
repository.deferLease(lease.eventId(), lease.owner(), lease.token(), nextAttemptAt));
|
||||
repository.deferLease(
|
||||
lease.eventId(), lease.owner(), lease.token(), nextAttemptAt, failureCode));
|
||||
}
|
||||
|
||||
/** Returns the leased row to PENDING without spending an attempt; see {@link #markPublished}. */
|
||||
@@ -198,11 +242,10 @@ public class JpaOutboxStore
|
||||
|
||||
/** Marks the leased row DEAD; see {@link #markPublished}. */
|
||||
@Override
|
||||
public OutboxCompletion markDead(OutboxLease lease) {
|
||||
public OutboxCompletion markDead(OutboxLease lease, String failureCode) {
|
||||
requireTransaction("markDead");
|
||||
return completion(
|
||||
repository.completeLease(
|
||||
lease.eventId(), lease.owner(), lease.token(), OutboxEventStatus.DEAD.name()));
|
||||
repository.deadLease(lease.eventId(), lease.owner(), lease.token(), failureCode));
|
||||
}
|
||||
|
||||
private static OutboxCompletion completion(int updatedRows) {
|
||||
@@ -237,6 +280,21 @@ public class JpaOutboxStore
|
||||
return result;
|
||||
}
|
||||
|
||||
@Override
|
||||
public OutboxLongRetries longRetries(int minDeferredClaims, Instant now) {
|
||||
if (minDeferredClaims < 1) {
|
||||
throw new IllegalArgumentException(
|
||||
"minDeferredClaims must be >= 1, was " + minDeferredClaims);
|
||||
}
|
||||
Object[] row = repository.countLongRetries(minDeferredClaims).getFirst();
|
||||
long count = (Long) row[0];
|
||||
if (count == 0) {
|
||||
return OutboxLongRetries.NONE;
|
||||
}
|
||||
Instant oldest = (Instant) row[1];
|
||||
return new OutboxLongRetries(count, Math.max(0, Duration.between(oldest, now).toSeconds()));
|
||||
}
|
||||
|
||||
/**
|
||||
* Restores the appended event. A row without its envelope — written by the pre-canonical
|
||||
* generation, or damaged — is unreadable, and so is one whose stored hash is not the hash of its
|
||||
|
||||
+51
-11
@@ -62,18 +62,22 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
|
||||
@Param("token") long token,
|
||||
@Param("status") String status);
|
||||
|
||||
/** {@link #completeLease} to FAILED, scheduling the retry in the same statement. */
|
||||
/**
|
||||
* {@link #completeLease} to FAILED, scheduling the retry and recording why (V22) in the same
|
||||
* statement.
|
||||
*/
|
||||
@Modifying(flushAutomatically = true)
|
||||
@Query(
|
||||
"update OutboxEventEntity e set e.status = 'FAILED', e.nextAttemptAt = :nextAttemptAt, "
|
||||
+ "e.leaseOwner = null "
|
||||
+ "e.lastFailureCode = :failureCode, e.leaseOwner = null "
|
||||
+ "where e.eventId = :eventId and e.status = 'IN_FLIGHT' "
|
||||
+ "and e.leaseOwner = :owner and e.leaseToken = :token")
|
||||
int failLease(
|
||||
@Param("eventId") String eventId,
|
||||
@Param("owner") String owner,
|
||||
@Param("token") long token,
|
||||
@Param("nextAttemptAt") Instant nextAttemptAt);
|
||||
@Param("nextAttemptAt") Instant nextAttemptAt,
|
||||
@Param("failureCode") String failureCode);
|
||||
|
||||
/**
|
||||
* {@link #failLease} for a claim that ended deferred: same transition, and the row counts the
|
||||
@@ -82,14 +86,29 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
|
||||
@Modifying(flushAutomatically = true)
|
||||
@Query(
|
||||
"update OutboxEventEntity e set e.status = 'FAILED', e.nextAttemptAt = :nextAttemptAt, "
|
||||
+ "e.deferredAttemptCount = e.deferredAttemptCount + 1, e.leaseOwner = null "
|
||||
+ "e.deferredAttemptCount = e.deferredAttemptCount + 1, "
|
||||
+ "e.lastFailureCode = :failureCode, e.leaseOwner = null "
|
||||
+ "where e.eventId = :eventId and e.status = 'IN_FLIGHT' "
|
||||
+ "and e.leaseOwner = :owner and e.leaseToken = :token")
|
||||
int deferLease(
|
||||
@Param("eventId") String eventId,
|
||||
@Param("owner") String owner,
|
||||
@Param("token") long token,
|
||||
@Param("nextAttemptAt") Instant nextAttemptAt);
|
||||
@Param("nextAttemptAt") Instant nextAttemptAt,
|
||||
@Param("failureCode") String failureCode);
|
||||
|
||||
/** {@link #completeLease} to DEAD, recording why (V22) in the same statement. */
|
||||
@Modifying(flushAutomatically = true)
|
||||
@Query(
|
||||
"update OutboxEventEntity e set e.status = 'DEAD', e.lastFailureCode = :failureCode, "
|
||||
+ "e.leaseOwner = null "
|
||||
+ "where e.eventId = :eventId and e.status = 'IN_FLIGHT' "
|
||||
+ "and e.leaseOwner = :owner and e.leaseToken = :token")
|
||||
int deadLease(
|
||||
@Param("eventId") String eventId,
|
||||
@Param("owner") String owner,
|
||||
@Param("token") long token,
|
||||
@Param("failureCode") String failureCode);
|
||||
|
||||
/**
|
||||
* Gives an unused claim back: PENDING, claimable from {@code claimableAt}, and the claim's
|
||||
@@ -110,12 +129,18 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
|
||||
|
||||
/**
|
||||
* Increments an aggregate's order counter (V21), taking its row lock until the caller commits.
|
||||
* Returns 0 when the aggregate has no counter yet.
|
||||
* Never below the aggregate's highest outbox row: an instance still on the wall-clock scheme may
|
||||
* have written a larger number after the counter existed (rolling deploy, review RR-03-02).
|
||||
* Returns 0 when the aggregate has no counter yet, or when another transaction's new counter is
|
||||
* not committed yet.
|
||||
*/
|
||||
@Modifying(flushAutomatically = true)
|
||||
@Query(
|
||||
value =
|
||||
"UPDATE outbox_aggregate_sequence SET last_sequence = last_sequence + 1"
|
||||
"UPDATE outbox_aggregate_sequence SET last_sequence = GREATEST(last_sequence,"
|
||||
+ " (SELECT COALESCE(MAX(e.aggregate_sequence), 0) FROM outbox_event e"
|
||||
+ " WHERE e.tenant_scope = :tenantScope AND e.aggregate_type = :aggregateType"
|
||||
+ " AND e.aggregate_id = :aggregateId)) + 1"
|
||||
+ " WHERE tenant_scope = :tenantScope AND aggregate_type = :aggregateType"
|
||||
+ " AND aggregate_id = :aggregateId",
|
||||
nativeQuery = true)
|
||||
@@ -125,8 +150,10 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
|
||||
@Param("aggregateId") String aggregateId);
|
||||
|
||||
/**
|
||||
* Creates an aggregate's counter, seeded after the highest order its outbox rows already carry.
|
||||
* Two creators of the same counter collide on its primary key.
|
||||
* Creates an aggregate's counter at its first number: one after the larger of its highest outbox
|
||||
* row and the V21 floor (0 where no migration wrote one). Returns 0, and writes nothing, when
|
||||
* another transaction created it first; {@code ON CONFLICT DO NOTHING} waits for that transaction
|
||||
* to end rather than failing on the primary key (review RR-03-01).
|
||||
*/
|
||||
@Modifying(flushAutomatically = true)
|
||||
@Query(
|
||||
@@ -134,9 +161,12 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
|
||||
"INSERT INTO outbox_aggregate_sequence"
|
||||
+ " (tenant_scope, aggregate_type, aggregate_id, last_sequence)"
|
||||
+ " SELECT :tenantScope, :aggregateType, :aggregateId,"
|
||||
+ " COALESCE(MAX(e.aggregate_sequence), 0) + 1 FROM outbox_event e"
|
||||
+ " GREATEST(COALESCE(MAX(e.aggregate_sequence), 0),"
|
||||
+ " COALESCE((SELECT f.floor_sequence FROM outbox_aggregate_sequence_floor f), 0))"
|
||||
+ " + 1 FROM outbox_event e"
|
||||
+ " WHERE e.tenant_scope = :tenantScope AND e.aggregate_type = :aggregateType"
|
||||
+ " AND e.aggregate_id = :aggregateId",
|
||||
+ " AND e.aggregate_id = :aggregateId"
|
||||
+ " ON CONFLICT DO NOTHING",
|
||||
nativeQuery = true)
|
||||
int createAggregateSequence(
|
||||
@Param("tenantScope") String tenantScope,
|
||||
@@ -174,4 +204,14 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
|
||||
"select e.eventType, min(e.occurredAt) from OutboxEventEntity e "
|
||||
+ "where e.status <> 'PUBLISHED' group by e.eventType")
|
||||
List<Object[]> findOldestUnpublishedOccurredAtByEventType();
|
||||
|
||||
/**
|
||||
* {@code [count (Long), oldestOccurredAt (Instant)]} of the FAILED rows with at least {@code
|
||||
* minDeferredClaims} deferred claims, for the long-retry gauges. One row, whose instant is null
|
||||
* when the count is 0.
|
||||
*/
|
||||
@Query(
|
||||
"select count(e), min(e.occurredAt) from OutboxEventEntity e "
|
||||
+ "where e.status = 'FAILED' and e.deferredAttemptCount >= :minDeferredClaims")
|
||||
List<Object[]> countLongRetries(@Param("minDeferredClaims") int minDeferredClaims);
|
||||
}
|
||||
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.outbox.jpa.entity;
|
||||
|
||||
import jakarta.persistence.Column;
|
||||
import jakarta.persistence.Entity;
|
||||
import jakarta.persistence.Id;
|
||||
import jakarta.persistence.Table;
|
||||
|
||||
/**
|
||||
* The one-row floor a new aggregate counter starts after ({@code outbox_aggregate_sequence_floor},
|
||||
* V21): the moment V21 ran, in epoch microseconds, above every number the replaced wall-clock
|
||||
* scheme wrote. Read by {@code JpaOutboxStore#nextSequence} with a native statement; mapped, like
|
||||
* {@link OutboxAggregateSequenceEntity}, so the table exists where Hibernate builds the schema (H2,
|
||||
* where no migration writes the row and the floor is 0) and is validated where Flyway does.
|
||||
* Package-private: no code reads it as an entity.
|
||||
*/
|
||||
@Entity
|
||||
@Table(name = "outbox_aggregate_sequence_floor")
|
||||
class OutboxAggregateSequenceFloorEntity {
|
||||
|
||||
@Id
|
||||
@Column(name = "singleton", nullable = false)
|
||||
private boolean singleton;
|
||||
|
||||
@Column(name = "floor_sequence", nullable = false)
|
||||
private long floorSequence;
|
||||
|
||||
protected OutboxAggregateSequenceFloorEntity() {}
|
||||
|
||||
long floorSequence() {
|
||||
return floorSequence;
|
||||
}
|
||||
}
|
||||
+16
@@ -58,6 +58,18 @@ public class OutboxEventEntity implements Persistable<String> {
|
||||
private int deferredAttemptCount;
|
||||
|
||||
/** Dual-purpose per status (PENDING / IN_FLIGHT / FAILED) — see README "outbox". */
|
||||
/**
|
||||
* The code of the row's last failed completion (V22): the publisher's own name for the failure,
|
||||
* or the outcome or exception type when it gave none.
|
||||
*
|
||||
* <p>Read-only to the entity: only the relay's failed completions write it, each in its one
|
||||
* fenced UPDATE ({@code OutboxEventJpaRepository#failLease} and its siblings). The append's
|
||||
* INSERT and the claim's flush leave it out, so they neither clear a code nor need the column on
|
||||
* a database that has not run V22 yet.
|
||||
*/
|
||||
@Column(name = "last_failure_code", length = 120, insertable = false, updatable = false)
|
||||
private String lastFailureCode;
|
||||
|
||||
@Column(name = "next_attempt_at", nullable = false)
|
||||
private Instant nextAttemptAt;
|
||||
|
||||
@@ -379,4 +391,8 @@ public class OutboxEventEntity implements Persistable<String> {
|
||||
public void setCausationId(String causationId) {
|
||||
this.causationId = causationId;
|
||||
}
|
||||
|
||||
public String getLastFailureCode() {
|
||||
return lastFailureCode;
|
||||
}
|
||||
}
|
||||
|
||||
+20
-3
@@ -123,7 +123,9 @@ public final class SpringTransaction implements TransactionPort {
|
||||
* dry and every other request fails at {@code connection-timeout}. With one, Spring hands the
|
||||
* remaining time to every statement (Hibernate and {@code JdbcTemplate} both set the JDBC query
|
||||
* timeout), so the server cancels the statement ({@code 57014}, {@code DB_QUERY_CANCELED}) and
|
||||
* the connection comes back.
|
||||
* the connection comes back. A statement Hibernate is asked to run after the deadline has passed
|
||||
* is refused by Hibernate itself, with no SQLState; this class reports that as {@code
|
||||
* DB_QUERY_CANCELED} too (review RR-03-03).
|
||||
*
|
||||
* <p>The default is 30 seconds for both: the budget {@code inNew} already has, and the Hikari
|
||||
* {@code leak-detection-threshold} at which the pool already reports a held connection as a leak.
|
||||
@@ -194,6 +196,9 @@ public final class SpringTransaction implements TransactionPort {
|
||||
|
||||
private <T> T execute(ModeTemplate mode, Supplier<T> action) {
|
||||
long startedAt = System.nanoTime();
|
||||
// Whether this call began the transaction, and so owns its deadline. A call that joined an
|
||||
// outer one leaves the deadline to the call that began it.
|
||||
AtomicBoolean beganTransaction = new AtomicBoolean();
|
||||
// Set only once the action has returned in a write transaction this call began: from then on
|
||||
// the only thing TransactionTemplate.execute still does is commit it.
|
||||
AtomicBoolean committingOwnWrite = new AtomicBoolean();
|
||||
@@ -201,20 +206,32 @@ public final class SpringTransaction implements TransactionPort {
|
||||
return mode.template()
|
||||
.execute(
|
||||
status -> {
|
||||
beganTransaction.set(status.isNewTransaction());
|
||||
T result = action.get();
|
||||
committingOwnWrite.set(status.isNewTransaction() && !mode.template().isReadOnly());
|
||||
return result;
|
||||
});
|
||||
} catch (RuntimeException failure) {
|
||||
Duration elapsed = Duration.ofNanos(System.nanoTime() - startedAt);
|
||||
Optional<PersistenceFailureException> translated =
|
||||
committingOwnWrite.get()
|
||||
? exceptionTranslator.translateCommitFailure(
|
||||
failure, mode.commitOperation(), Duration.ofNanos(System.nanoTime() - startedAt))
|
||||
? exceptionTranslator.translateCommitFailure(failure, mode.commitOperation(), elapsed)
|
||||
: exceptionTranslator.translate(failure);
|
||||
if (translated.isEmpty() && beganTransaction.get() && pastDeadline(mode, elapsed)) {
|
||||
// Hibernate's own deadline refusal carries no SQLState (review RR-03-03).
|
||||
translated = exceptionTranslator.translateAfterDeadline(failure);
|
||||
}
|
||||
throw translated.map(RuntimeException.class::cast).orElse(failure);
|
||||
}
|
||||
}
|
||||
|
||||
/** Whether {@code elapsed} reached the timeout the mode's transactions run under. */
|
||||
private static boolean pastDeadline(ModeTemplate mode, Duration elapsed) {
|
||||
int timeoutSeconds = mode.template().getTimeout();
|
||||
return timeoutSeconds != TransactionDefinition.TIMEOUT_DEFAULT
|
||||
&& elapsed.compareTo(Duration.ofSeconds(timeoutSeconds)) >= 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* The vendor-neutral default translator.
|
||||
*
|
||||
|
||||
+72
-3
@@ -7,8 +7,32 @@
|
||||
--
|
||||
-- outbox_aggregate_sequence holds one counter per aggregate. Incrementing it locks the row until
|
||||
-- the writer commits, so the numbers follow commit order; it outlives the reaper's deletes, so a
|
||||
-- number is never reused. JpaOutboxAggregateSequences seeds a new counter from the aggregate's
|
||||
-- highest existing aggregate_sequence, so rows numbered by the old clock scheme still come first.
|
||||
-- number is never reused. JpaOutboxStore#nextSequence reads and writes it.
|
||||
--
|
||||
-- Consumers have already seen clock numbers (about 1.7e15) for aggregates whose rows this
|
||||
-- database no longer holds: the reaper deleted them, or they were written under an identity that
|
||||
-- has since changed (the sample's tenant scope). A counter that started after the rows still
|
||||
-- present would start at 1 for those aggregates, behind numbers already sent (review RR-03-02,
|
||||
-- RR-05-02). So this migration records one floor, the moment it runs in epoch microseconds, which
|
||||
-- is above every clock number written before it; a counter created later starts after the floor
|
||||
-- or after the aggregate's highest remaining row, whichever is larger. The aggregates that still
|
||||
-- have rows get their counter here, at their highest number. A clock number an instance still on
|
||||
-- the old version writes during the rollout is above the floor; nextSequence also never goes below
|
||||
-- the highest row it can see, so such a row still comes first.
|
||||
--
|
||||
-- This file was edited after it was first written, before any deployment applied it (the branch
|
||||
-- that added it was not merged): splitting the floor into a V22 would have left V21 alone handing
|
||||
-- out numbers from 1 for one release, and its comment pointed at a class that does not exist.
|
||||
-- applied-migrations.sha256 carries the new checksum. A development database that applied the
|
||||
-- earlier text fails Flyway validation; recreate it.
|
||||
|
||||
-- Bound the wait for the table lock the unique index below takes (SHARE on outbox_event, held until
|
||||
-- this migration commits). Without a bound, the migration queues behind any open outbox write and
|
||||
-- every outbox write after it queues behind the migration. Five seconds, then this migration fails,
|
||||
-- rolls back whole and the database stays at V20: deploy again, or build the index first
|
||||
-- (runbook §8, CONCURRENTLY).
|
||||
SET LOCAL lock_timeout = '5s';
|
||||
|
||||
CREATE TABLE IF NOT EXISTS outbox_aggregate_sequence (
|
||||
tenant_scope VARCHAR(96) NOT NULL,
|
||||
aggregate_type VARCHAR(64) NOT NULL,
|
||||
@@ -18,10 +42,55 @@ CREATE TABLE IF NOT EXISTS outbox_aggregate_sequence (
|
||||
CONSTRAINT ck_outbox_aggregate_sequence_positive CHECK (last_sequence >= 1)
|
||||
);
|
||||
|
||||
-- One row, written once: the lowest number a counter created from now on may start after.
|
||||
CREATE TABLE IF NOT EXISTS outbox_aggregate_sequence_floor (
|
||||
singleton BOOLEAN NOT NULL DEFAULT TRUE,
|
||||
floor_sequence BIGINT NOT NULL,
|
||||
CONSTRAINT pk_outbox_aggregate_sequence_floor PRIMARY KEY (singleton),
|
||||
CONSTRAINT ck_outbox_aggregate_sequence_floor_singleton CHECK (singleton),
|
||||
CONSTRAINT ck_outbox_aggregate_sequence_floor_non_negative CHECK (floor_sequence >= 0)
|
||||
);
|
||||
|
||||
INSERT INTO outbox_aggregate_sequence_floor (singleton, floor_sequence)
|
||||
VALUES (TRUE, (extract(epoch FROM clock_timestamp()) * 1000000)::BIGINT)
|
||||
ON CONFLICT (singleton) DO NOTHING;
|
||||
|
||||
INSERT INTO outbox_aggregate_sequence (tenant_scope, aggregate_type, aggregate_id, last_sequence)
|
||||
SELECT tenant_scope, aggregate_type, aggregate_id, MAX(aggregate_sequence)
|
||||
FROM outbox_event
|
||||
WHERE tenant_scope IS NOT NULL
|
||||
AND aggregate_type IS NOT NULL
|
||||
AND aggregate_sequence IS NOT NULL
|
||||
GROUP BY tenant_scope, aggregate_type, aggregate_id
|
||||
ON CONFLICT (tenant_scope, aggregate_type, aggregate_id) DO NOTHING;
|
||||
|
||||
-- One number, one event: a second event appended under an order its aggregate already used is
|
||||
-- refused at append time, in the writer's transaction, rather than released by the gate in no
|
||||
-- defined order. Rows without canonical order columns (pre-canonical, all NULL) are not compared.
|
||||
-- This fails if duplicates already exist; resolve them (they are exactly the rows the gate could
|
||||
-- not order) before migrating.
|
||||
-- not order) before migrating (runbook §8).
|
||||
--
|
||||
-- Built here, inside this migration's transaction, not CONCURRENTLY: a duplicate then rolls the
|
||||
-- whole of V21 back and leaves the database at V20 with nothing to clean up, which is what the
|
||||
-- runbook's retry relies on. A failed CONCURRENTLY build leaves an INVALID index that IF NOT EXISTS
|
||||
-- would then skip, so the uniqueness would silently not hold; and CONCURRENTLY cannot run in the
|
||||
-- transaction that writes the counters and the floor. The reaper bounds outbox_event to the
|
||||
-- retention window, so the build is normally short. An outbox too large to lock for that long is
|
||||
-- indexed first with the runbook's CONCURRENTLY statement (same name, same columns); the check
|
||||
-- below refuses an index such a build left invalid, and IF NOT EXISTS keeps the valid one.
|
||||
DO $$
|
||||
BEGIN
|
||||
IF EXISTS (SELECT 1
|
||||
FROM pg_index
|
||||
WHERE indexrelid = to_regclass('ux_outbox_event_aggregate_order')
|
||||
AND NOT (indisvalid AND indisunique)) THEN
|
||||
RAISE EXCEPTION 'ux_outbox_event_aggregate_order exists but is not a valid unique index'
|
||||
USING HINT = 'A CREATE INDEX CONCURRENTLY failed or is still running. '
|
||||
|| 'DROP INDEX CONCURRENTLY ux_outbox_event_aggregate_order, then rebuild it '
|
||||
|| 'or let V21 build it (runbook outbox-dead-letter §8).';
|
||||
END IF;
|
||||
END
|
||||
$$;
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS ux_outbox_event_aggregate_order
|
||||
ON outbox_event (tenant_scope, aggregate_type, aggregate_id, aggregate_sequence, event_index);
|
||||
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
-- Why a row keeps failing, kept on the row.
|
||||
--
|
||||
-- A publish refused for now (a closed transport, a credential refused) or left unanswered is
|
||||
-- retried without spending the attempt budget (V20), so it has no end: the relay retries it five
|
||||
-- minutes apart for as long as nobody looks. The outcome the relay records cannot say why — a
|
||||
-- shutdown, a closed transport and a refused credential are one outcome — and the relay's log line
|
||||
-- said it in one sentence for all four. The publisher's failure code can (a messaging platform code
|
||||
-- such as KAFKA_AUTHENTICATIONEXCEPTION); every failed completion writes it here, so the runbook's
|
||||
-- query for rows that have been retrying long shows what each one is waiting on (re-review
|
||||
-- 2026-11-03 RR-05-03).
|
||||
--
|
||||
-- A code, never a message: a driver message can carry an endpoint, a statement or a credential.
|
||||
-- Nullable with no default, so adding it rewrites no row; rows written before it stay NULL until
|
||||
-- their next failure.
|
||||
--
|
||||
-- The new status PARKED needs no DDL: status has no CHECK, the claim selects PENDING, FAILED and
|
||||
-- IN_FLIGHT only (so the partial index ix_outbox_event_eligible leaves PARKED out), and the order
|
||||
-- gate and its index ix_outbox_event_unpublished_aggregate_order treat every status but PUBLISHED as
|
||||
-- blocking — which is what holding a row in its aggregate's place means.
|
||||
ALTER TABLE outbox_event
|
||||
ADD COLUMN last_failure_code VARCHAR(120);
|
||||
+29
-21
@@ -23,7 +23,8 @@ import org.junit.jupiter.api.Tag;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* What a worker's death leaves behind, at each phase of a dispatch.
|
||||
* What a worker's death leaves behind, at each phase of a dispatch — the rows, not what recovery
|
||||
* does with them.
|
||||
*
|
||||
* <p>This is the {@code recoverable} claim in {@code docs/notification/evidence-manifest.json}, and
|
||||
* until now nothing backed it. Lease recovery and event replay had unit coverage against fakes; the
|
||||
@@ -36,9 +37,13 @@ import org.junit.jupiter.api.Test;
|
||||
* worker is from the database's point of view — a lease nobody will renew and a phase nobody will
|
||||
* advance.
|
||||
*
|
||||
* <p>The matrix walks the phases a dispatch passes through. For each, the question is the same and
|
||||
* has only two acceptable answers: the delivery becomes claimable again, or it becomes a question
|
||||
* for the provider. Never both, and never neither.
|
||||
* <p>The matrix walks the phases a dispatch passes through and pins, for each, the state recovery
|
||||
* starts from: a lease that has passed (the claim takes the row again) and, from the attempt insert
|
||||
* on, an unfinished attempt row carrying the evidence of that phase (what reconciliation asks the
|
||||
* provider about). These rows are written by this test, so the tests check the shape of the input
|
||||
* and the predicates recovery reads, not recovery itself. {@code
|
||||
* PostgreSqlNotificationBackgroundPassIntegrationTest} runs recovery and reconciliation over rows
|
||||
* the dispatcher's own factory wrote (review 05-01, RR-03-07).
|
||||
*/
|
||||
@Tag("jpa-contract")
|
||||
class WorkerCrashRecoveryContractTest {
|
||||
@@ -73,8 +78,8 @@ class WorkerCrashRecoveryContractTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a crash after claiming and before any attempt leaves the delivery claimable")
|
||||
void aCrashBeforeAnyAttemptLeavesItClaimable() throws SQLException {
|
||||
@DisplayName("claim 뒤 attempt 전에 죽은 delivery는 attempt 행이 없고 lease가 지나 있다")
|
||||
void aDeliveryDeadBeforeAnyAttemptHasNoAttemptRowAndAPassedLease() throws SQLException {
|
||||
UUID delivery = deliveryHeldByADeadWorker();
|
||||
|
||||
assertThat(isClaimableAgain(delivery))
|
||||
@@ -88,12 +93,12 @@ class WorkerCrashRecoveryContractTest {
|
||||
/**
|
||||
* The attempt row is written before the call and keeps the factory's "unknown" evidence for the
|
||||
* whole call (review 05-01), so a crash anywhere between the insert and the provider's answer
|
||||
* leaves the same row. Recovery cannot tell "never called" from "called and died", so the row
|
||||
* goes to reconciliation rather than back to the queue.
|
||||
* leaves the same row. This test writes that row itself; that recovery hands it to reconciliation
|
||||
* is {@code PostgreSqlNotificationBackgroundPassIntegrationTest}'s.
|
||||
*/
|
||||
@Test
|
||||
@DisplayName("attempt 행을 쓴 뒤 호출이 끝나기 전 죽으면 시작 여부 불명인 미완료 attempt 가 남아 reconcile 대상이 된다")
|
||||
void aCrashAfterTheAttemptRowIsWrittenLeavesAnUnfinishedAttemptOfUnknownStart()
|
||||
@DisplayName("attempt 행을 쓰고 호출 전에 죽은 delivery는 시작 여부 불명인 미완료 attempt와 지난 lease를 남긴다")
|
||||
void aDeliveryDeadAfterTheAttemptInsertLeavesAnUnfinishedAttemptOfUnknownStart()
|
||||
throws SQLException {
|
||||
// given
|
||||
UUID delivery = deliveryHeldByADeadWorker();
|
||||
@@ -103,17 +108,18 @@ class WorkerCrashRecoveryContractTest {
|
||||
|
||||
// then
|
||||
assertThat(evidenceOf(attempt))
|
||||
.as("the row the factory writes records that whether the call started is not known")
|
||||
.as("the row stands for the factory's: whether the call started is not known")
|
||||
.containsExactly("false", "UNKNOWN", "false", "UNKNOWN");
|
||||
assertThat(isUnfinished(attempt))
|
||||
.as("an unfinished attempt is what recovery hands to reconciliation")
|
||||
.as("completed_at is empty, the predicate recovery selects unfinished attempts by")
|
||||
.isTrue();
|
||||
assertThat(isClaimableAgain(delivery)).isTrue();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a crash after the request started is a question for the provider, not a requeue")
|
||||
void aCrashAfterTheRequestStartedIsAQuestion() throws SQLException {
|
||||
@DisplayName("요청 시작 뒤 죽은 delivery는 시작 증거가 있는 미완료 attempt와 지난 lease를 남긴다")
|
||||
void aDeliveryDeadAfterTheRequestStartedLeavesAnUnfinishedAttemptWithStartEvidence()
|
||||
throws SQLException {
|
||||
UUID delivery = deliveryHeldByADeadWorker();
|
||||
UUID attempt = attemptOn(delivery, true, "PROVEN", false, "UNKNOWN");
|
||||
|
||||
@@ -125,13 +131,14 @@ class WorkerCrashRecoveryContractTest {
|
||||
"the lease still expires — the row must not be stranded — but the attempt is unfinished")
|
||||
.isTrue();
|
||||
assertThat(isUnfinished(attempt))
|
||||
.as("an unfinished attempt is what recovery hands to reconciliation")
|
||||
.as("completed_at is empty, the predicate recovery selects unfinished attempts by")
|
||||
.isTrue();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a crash after the body was committed is never silently retried")
|
||||
void aCrashAfterTheBodyWasCommittedIsNeverSilentlyRetried() throws SQLException {
|
||||
@DisplayName("본문 전송 뒤 죽은 delivery는 전송 증거가 있는 미완료 attempt를 남긴다")
|
||||
void aDeliveryDeadAfterTheBodyWasSentLeavesAnUnfinishedAttemptWithSendEvidence()
|
||||
throws SQLException {
|
||||
UUID delivery = deliveryHeldByADeadWorker();
|
||||
UUID attempt = attemptOn(delivery, true, "PROVEN", true, "PROVEN");
|
||||
|
||||
@@ -142,8 +149,8 @@ class WorkerCrashRecoveryContractTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a completed attempt is not recovered at all")
|
||||
void aCompletedAttemptIsNotRecovered() throws SQLException {
|
||||
@DisplayName("완료 시각이 적힌 attempt는 미완료로 보이지 않는다")
|
||||
void aCompletedAttemptIsNotUnfinished() throws SQLException {
|
||||
UUID delivery = deliveryHeldByADeadWorker();
|
||||
UUID attempt = attemptOn(delivery, true, "PROVEN", true, "PROVEN");
|
||||
complete(attempt);
|
||||
@@ -154,8 +161,9 @@ class WorkerCrashRecoveryContractTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("across every phase, exactly one delivery row survives with one attempt each")
|
||||
void acrossEveryPhaseNothingIsLostOrDuplicated() throws SQLException {
|
||||
@DisplayName("네 단계에서 죽은 워커들은 delivery 4개와 attempt 3개를 남기고 모든 lease가 지나 있다")
|
||||
void deadWorkersAcrossEveryPhaseLeaveFourDeliveriesThreeAttemptsAndPassedLeases()
|
||||
throws SQLException {
|
||||
List<UUID> deliveries = new ArrayList<>();
|
||||
// The four crash phases, run together, so the recovery sweep sees a mixed table rather than one
|
||||
// hand-picked row.
|
||||
|
||||
+9
-2
@@ -59,7 +59,14 @@ import org.springframework.transaction.PlatformTransactionManager;
|
||||
class DurableOperationSubmissionOutboxIntegrationTest {
|
||||
|
||||
private static final Instant NOW = Instant.parse("2026-08-25T09:00:00Z");
|
||||
private static final DurableOperationId ID = new DurableOperationId("op-1");
|
||||
|
||||
/**
|
||||
* A canonical lower-case UUIDv7, as the HTTP adapter mints: the operation id is then the event id
|
||||
* and the aggregate id itself (an id of any other spelling becomes a UUIDv7 derived from it).
|
||||
*/
|
||||
private static final String OPERATION_UUID = "01994e11-4d88-7000-8000-0000000000a1";
|
||||
|
||||
private static final DurableOperationId ID = new DurableOperationId(OPERATION_UUID);
|
||||
|
||||
private static JpaPlatformContractSupport support;
|
||||
private static AnnotationConfigApplicationContext context;
|
||||
@@ -103,7 +110,7 @@ class DurableOperationSubmissionOutboxIntegrationTest {
|
||||
|
||||
// then
|
||||
assertThat(queryStrings("SELECT contract_id || '|' || aggregate_id FROM outbox_event"))
|
||||
.containsExactly("durable-operation.submitted|op-1");
|
||||
.containsExactly("durable-operation.submitted|" + OPERATION_UUID);
|
||||
}
|
||||
|
||||
@Test
|
||||
|
||||
+159
@@ -0,0 +1,159 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.readiness;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaFileMetadata;
|
||||
import dev.caskeleton.application.fileserver.api.FileId;
|
||||
import dev.caskeleton.application.fileserver.api.StorageNamespace;
|
||||
import dev.caskeleton.application.fileserver.api.metadata.FileOwner;
|
||||
import dev.caskeleton.application.fileserver.api.metadata.FileRecordDraft;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneId;
|
||||
import java.time.ZoneOffset;
|
||||
import java.time.temporal.ChronoUnit;
|
||||
import java.util.Optional;
|
||||
import java.util.OptionalLong;
|
||||
import java.util.UUID;
|
||||
import java.util.function.Supplier;
|
||||
import org.flywaydb.core.Flyway;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.transaction.support.TransactionTemplate;
|
||||
|
||||
/**
|
||||
* One real PostgreSQL with the Fileserver schema and stores, for the port contract tests that hold
|
||||
* the JPA stores to the cases their in-memory fakes pass (review 2026-11-03 RR-01-04).
|
||||
*
|
||||
* <p>The stores read a clock this stack owns and moves, so a contract case about expiry passes time
|
||||
* the way it does against a fake instead of sleeping. One server per test class; {@link #empty()}
|
||||
* makes each case independent of the one before it.
|
||||
*/
|
||||
final class FileserverPostgreSqlStack implements AutoCloseable {
|
||||
|
||||
private final PostgreSqlReadinessSupport postgres;
|
||||
private final MovableClock clock;
|
||||
private final FileserverJpaTestContext context;
|
||||
private final TransactionTemplate transactions;
|
||||
private final JdbcTemplate jdbc;
|
||||
|
||||
private FileserverPostgreSqlStack(PostgreSqlReadinessSupport postgres) {
|
||||
this.postgres = postgres;
|
||||
migrate("classpath:db/migration/postgresql", "flyway_schema_history", false, null);
|
||||
migrate(
|
||||
"classpath:db/migration/jpa/core",
|
||||
"flyway_jpa_core_history",
|
||||
true,
|
||||
"explicit-jpa-core-adoption");
|
||||
migrate(
|
||||
"classpath:db/migration/jpa/fileserver",
|
||||
"flyway_jpa_fileserver_history",
|
||||
true,
|
||||
"explicit-jpa-fileserver-adoption");
|
||||
this.clock = new MovableClock(Instant.now().truncatedTo(ChronoUnit.MICROS));
|
||||
this.context = new FileserverJpaTestContext(postgres.dataSource(), clock);
|
||||
this.transactions = context.transactions();
|
||||
this.jdbc = new JdbcTemplate(postgres.dataSource());
|
||||
}
|
||||
|
||||
/** Starts a server, applies the three schema streams and builds the Fileserver stores. */
|
||||
static FileserverPostgreSqlStack start() {
|
||||
PostgreSqlReadinessSupport.assertDockerAvailable();
|
||||
return new FileserverPostgreSqlStack(PostgreSqlReadinessSupport.start(8, 2_000));
|
||||
}
|
||||
|
||||
<T> T bean(Class<T> type) {
|
||||
return context.bean(type);
|
||||
}
|
||||
|
||||
/** Runs {@code work} in one committed transaction, as the application's transaction port does. */
|
||||
<T> T inTransaction(Supplier<T> work) {
|
||||
return transactions.execute(status -> work.get());
|
||||
}
|
||||
|
||||
/** The instant the stores read as now. */
|
||||
Instant now() {
|
||||
return clock.instant();
|
||||
}
|
||||
|
||||
/** Moves the stores' clock forward. */
|
||||
void elapse(Duration duration) {
|
||||
clock.advance(duration);
|
||||
}
|
||||
|
||||
/** Records a file in {@code namespace}, so rows that reference a file may name it. */
|
||||
FileId recordFileIn(StorageNamespace namespace) {
|
||||
FileId fileId = FileId.of(UUID.randomUUID());
|
||||
inTransaction(
|
||||
() ->
|
||||
bean(JpaFileMetadata.class)
|
||||
.insert(
|
||||
new FileRecordDraft(
|
||||
fileId,
|
||||
namespace,
|
||||
"report.bin",
|
||||
Optional.of("application/octet-stream"),
|
||||
OptionalLong.of(1_000),
|
||||
new FileOwner("user-1", Optional.empty()))));
|
||||
return fileId;
|
||||
}
|
||||
|
||||
/** Deletes every Fileserver row, children first. */
|
||||
void empty() {
|
||||
jdbc.execute(
|
||||
"TRUNCATE fs_recovery_item, fs_cleanup_item, fs_quota_reservation,"
|
||||
+ " fs_verification_result, fs_upload_session, fs_file CASCADE");
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
context.close();
|
||||
postgres.close();
|
||||
}
|
||||
|
||||
private void migrate(
|
||||
String location, String historyTable, boolean independent, String baselineDescription) {
|
||||
var configuration =
|
||||
Flyway.configure()
|
||||
.dataSource(postgres.dataSource())
|
||||
.locations(location)
|
||||
.table(historyTable)
|
||||
.baselineOnMigrate(false)
|
||||
.outOfOrder(false);
|
||||
if (independent) {
|
||||
configuration.baselineVersion("0").baselineDescription(baselineDescription);
|
||||
}
|
||||
Flyway flyway = configuration.load();
|
||||
if (independent) {
|
||||
flyway.baseline();
|
||||
}
|
||||
flyway.migrate();
|
||||
}
|
||||
|
||||
/** A clock a test moves forward; microsecond precision, as PostgreSQL stores timestamps. */
|
||||
private static final class MovableClock extends Clock {
|
||||
|
||||
private Instant now;
|
||||
|
||||
private MovableClock(Instant start) {
|
||||
this.now = start;
|
||||
}
|
||||
|
||||
private synchronized void advance(Duration duration) {
|
||||
now = now.plus(duration);
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized Instant instant() {
|
||||
return now;
|
||||
}
|
||||
|
||||
@Override
|
||||
public ZoneId getZone() {
|
||||
return ZoneOffset.UTC;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Clock withZone(ZoneId zone) {
|
||||
throw new UnsupportedOperationException("the stack's clock is UTC");
|
||||
}
|
||||
}
|
||||
}
|
||||
+80
@@ -0,0 +1,80 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.readiness;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* The SQL an outbox runbook gives an operator, read out of the runbook itself.
|
||||
*
|
||||
* <p>A runbook block is marked {@code <!-- runbook-sql: id -->} and followed by a {@code sql}
|
||||
* fence. The lanes that run these blocks against real PostgreSQL read them from the document rather
|
||||
* than keep a copy, so editing a block in the runbook is what the lane tests: a copy would pass
|
||||
* while the document an operator follows is wrong.
|
||||
*/
|
||||
final class OutboxRunbookSql {
|
||||
|
||||
private final String runbook;
|
||||
|
||||
private OutboxRunbookSql(String runbook) {
|
||||
this.runbook = runbook;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads the runbook the lane was given in the system property {@code property}.
|
||||
*
|
||||
* @param property the system property holding the runbook's absolute path
|
||||
* @return the runbook's SQL blocks
|
||||
* @throws IOException when the file cannot be read
|
||||
*/
|
||||
static OutboxRunbookSql fromSystemProperty(String property) throws IOException {
|
||||
String path = System.getProperty(property);
|
||||
if (path == null || path.isBlank()) {
|
||||
throw new IllegalStateException(
|
||||
"system property " + property + " must name the runbook this lane runs");
|
||||
}
|
||||
return new OutboxRunbookSql(Files.readString(Path.of(path), StandardCharsets.UTF_8));
|
||||
}
|
||||
|
||||
/**
|
||||
* The SQL block that follows {@code <!-- runbook-sql: id -->}.
|
||||
*
|
||||
* @param id the block's marker id
|
||||
* @return the SQL text between the fences
|
||||
*/
|
||||
String block(String id) {
|
||||
String marker = "<!-- runbook-sql: " + id + " -->";
|
||||
int at = runbook.indexOf(marker);
|
||||
if (at < 0) {
|
||||
throw new IllegalStateException("the outbox runbook has no SQL block marked " + id);
|
||||
}
|
||||
int open = runbook.indexOf("```sql\n", at);
|
||||
int close = runbook.indexOf("\n```", open + 1);
|
||||
if (open < 0 || close < 0) {
|
||||
throw new IllegalStateException("the block marked " + id + " is not a closed sql fence");
|
||||
}
|
||||
return runbook.substring(open + "```sql\n".length(), close);
|
||||
}
|
||||
|
||||
/**
|
||||
* The block with its {@code <placeholder>}s filled in, as an operator would before running it.
|
||||
*
|
||||
* @param id the block's marker id
|
||||
* @param values each placeholder as written in the runbook, without the angle brackets, and its
|
||||
* value
|
||||
* @return the runnable SQL
|
||||
*/
|
||||
String block(String id, Map<String, String> values) {
|
||||
String sql = block(id);
|
||||
for (Map.Entry<String, String> value : values.entrySet()) {
|
||||
String placeholder = "<" + value.getKey() + ">";
|
||||
if (!sql.contains(placeholder)) {
|
||||
throw new IllegalStateException("the block marked " + id + " has no " + placeholder);
|
||||
}
|
||||
sql = sql.replace(placeholder, value.getValue());
|
||||
}
|
||||
return sql;
|
||||
}
|
||||
}
|
||||
+54
@@ -0,0 +1,54 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.readiness;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaContentReferenceLedger;
|
||||
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaFileMetadata;
|
||||
import dev.caskeleton.application.fileserver.admin.ContentReferenceLedgerPort;
|
||||
import dev.caskeleton.application.fileserver.api.metadata.FileMetadataPort;
|
||||
import dev.caskeleton.application.fileserver.testkit.ContentReferenceLedgerPortContract;
|
||||
import java.util.function.Supplier;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Tag;
|
||||
|
||||
/**
|
||||
* {@link JpaContentReferenceLedger} held to the contract the in-memory metadata store passes, on
|
||||
* real PostgreSQL (review 2026-11-03 RR-01-04).
|
||||
*/
|
||||
@Tag("jpa-contract")
|
||||
class PostgreSqlContentReferenceLedgerContractTest extends ContentReferenceLedgerPortContract {
|
||||
|
||||
private static FileserverPostgreSqlStack stack;
|
||||
|
||||
@BeforeAll
|
||||
static void start() {
|
||||
stack = FileserverPostgreSqlStack.start();
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
static void stop() {
|
||||
if (stack != null) {
|
||||
stack.close();
|
||||
}
|
||||
}
|
||||
|
||||
@BeforeEach
|
||||
void empty() {
|
||||
stack.empty();
|
||||
}
|
||||
|
||||
@Override
|
||||
protected ContentReferenceLedgerPort ledger() {
|
||||
return stack.bean(JpaContentReferenceLedger.class);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected FileMetadataPort metadata() {
|
||||
return stack.bean(JpaFileMetadata.class);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected <T> T inTransaction(Supplier<T> work) {
|
||||
return stack.inTransaction(work);
|
||||
}
|
||||
}
|
||||
+92
@@ -0,0 +1,92 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.readiness;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaFileQuota;
|
||||
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaQuotaCommit;
|
||||
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaQuotaReclaim;
|
||||
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaUploadSession;
|
||||
import dev.caskeleton.application.fileserver.api.FileId;
|
||||
import dev.caskeleton.application.fileserver.api.StorageNamespace;
|
||||
import dev.caskeleton.application.fileserver.api.UploadId;
|
||||
import dev.caskeleton.application.fileserver.api.metadata.FileQuotaPort;
|
||||
import dev.caskeleton.application.fileserver.api.metadata.UploadSession;
|
||||
import dev.caskeleton.application.fileserver.api.metadata.UploadSessionDraft;
|
||||
import dev.caskeleton.application.fileserver.api.transfer.UploadMode;
|
||||
import dev.caskeleton.application.fileserver.cleanup.QuotaReclaimPort;
|
||||
import dev.caskeleton.application.fileserver.testkit.FileQuotaLedgerContract;
|
||||
import dev.caskeleton.application.fileserver.upload.QuotaCommitPort;
|
||||
import java.time.Duration;
|
||||
import java.util.OptionalLong;
|
||||
import java.util.UUID;
|
||||
import java.util.function.Supplier;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Tag;
|
||||
|
||||
/**
|
||||
* {@link JpaFileQuota}, {@link JpaQuotaCommit} and {@link JpaQuotaReclaim} held to the ledger
|
||||
* contract the in-memory ledger passes, on real PostgreSQL (review 2026-11-03 RR-01-04).
|
||||
*/
|
||||
@Tag("jpa-contract")
|
||||
class PostgreSqlFileQuotaLedgerContractTest extends FileQuotaLedgerContract {
|
||||
|
||||
private static FileserverPostgreSqlStack stack;
|
||||
|
||||
@BeforeAll
|
||||
static void start() {
|
||||
stack = FileserverPostgreSqlStack.start();
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
static void stop() {
|
||||
if (stack != null) {
|
||||
stack.close();
|
||||
}
|
||||
}
|
||||
|
||||
@BeforeEach
|
||||
void empty() {
|
||||
stack.empty();
|
||||
}
|
||||
|
||||
@Override
|
||||
protected FileQuotaPort quota() {
|
||||
return stack.bean(JpaFileQuota.class);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected QuotaCommitPort commits() {
|
||||
return stack.bean(JpaQuotaCommit.class);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected QuotaReclaimPort reclaims() {
|
||||
return stack.bean(JpaQuotaReclaim.class);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected UploadSession uploadIn(StorageNamespace namespace) {
|
||||
FileId fileId = stack.recordFileIn(namespace);
|
||||
return stack.inTransaction(
|
||||
() ->
|
||||
stack
|
||||
.bean(JpaUploadSession.class)
|
||||
.create(
|
||||
new UploadSessionDraft(
|
||||
UploadId.of(UUID.randomUUID()),
|
||||
fileId,
|
||||
UploadMode.SINGLE_SHOT,
|
||||
OptionalLong.of(1_000),
|
||||
stack.now().plus(Duration.ofHours(1)))));
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void elapse(Duration duration) {
|
||||
stack.elapse(duration);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected <T> T inTransaction(Supplier<T> work) {
|
||||
return stack.inTransaction(work);
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -50,7 +50,7 @@ class PostgreSqlMigrationIntegrationTest {
|
||||
assertThat(appliedVersions(postgres, "flyway_schema_history"))
|
||||
.containsExactly(
|
||||
"1", "3", "4", "5", "6", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18",
|
||||
"19", "20", "21");
|
||||
"19", "20", "21", "22");
|
||||
assertThat(
|
||||
singleValue(
|
||||
postgres,
|
||||
|
||||
+79
-11
@@ -19,11 +19,18 @@ import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
|
||||
/**
|
||||
* The aggregate order counter and the order uniqueness rule against real PostgreSQL (V21, review
|
||||
* 06-07): the shared {@link OutboxAggregateSequenceContract}, the seed from rows numbered before
|
||||
* the counter existed, and a second append under a used order refused at append time.
|
||||
* 06-07): the shared {@link OutboxAggregateSequenceContract}, where a counter starts when the
|
||||
* aggregate already has history, and a second append under a used order refused at append time.
|
||||
*
|
||||
* <p>"History" is what the old wall-clock scheme left (epoch microseconds, about 1.7e15): rows that
|
||||
* are still in the outbox, rows the reaper deleted, rows under an identity that has since changed,
|
||||
* and rows an instance still on the old version writes during a rolling deploy. A consumer that
|
||||
* reads the number as "which change came last" must never see a later change with a smaller number
|
||||
* (review RR-03-02, RR-05-02).
|
||||
*/
|
||||
class PostgreSqlOutboxAggregateSequenceIntegrationTest extends OutboxAggregateSequenceContract {
|
||||
|
||||
@@ -32,6 +39,7 @@ class PostgreSqlOutboxAggregateSequenceIntegrationTest extends OutboxAggregateSe
|
||||
private static PostgreSqlReadinessSupport postgres;
|
||||
private static AnnotationConfigApplicationContext context;
|
||||
private static OutboxAggregateSequencePort sequences;
|
||||
private static JdbcTemplate jdbc;
|
||||
|
||||
@BeforeAll
|
||||
static void setup() {
|
||||
@@ -40,6 +48,7 @@ class PostgreSqlOutboxAggregateSequenceIntegrationTest extends OutboxAggregateSe
|
||||
OutboxIntegrationSupport.migrate(dataSource);
|
||||
context = OutboxIntegrationSupport.buildContext(dataSource, e -> {}, Clock.systemUTC());
|
||||
sequences = context.getBean(JpaOutboxStore.class);
|
||||
jdbc = new JdbcTemplate(dataSource);
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
@@ -63,25 +72,71 @@ class PostgreSqlOutboxAggregateSequenceIntegrationTest extends OutboxAggregateSe
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("카운터가 없던 aggregate의 첫 번호는 이미 있는 outbox 행의 가장 큰 번호 다음이다")
|
||||
@DisplayName("카운터가 없던 aggregate의 첫 번호는 floor보다 큰 남은 행의 가장 큰 번호 다음이다")
|
||||
void aNewCounterStartsAfterTheHighestOrderTheAggregatesRowsAlreadyCarry() {
|
||||
// given: a row numbered by the old wall-clock scheme
|
||||
// given: a row an instance still on the clock scheme wrote five seconds after V21 ran
|
||||
String aggregateId = "clock-numbered-" + UUID.randomUUID();
|
||||
long clockNumber = 1_761_955_200_000_000L;
|
||||
long clockNumber = floor() + 5_000_000L;
|
||||
append(aggregateId, clockNumber);
|
||||
|
||||
// when
|
||||
long next =
|
||||
transactions()
|
||||
.inWrite(
|
||||
() ->
|
||||
sequences.nextSequence(
|
||||
new AggregateIdentity("default", "testkit", aggregateId)));
|
||||
long next = next(aggregateId);
|
||||
|
||||
// then
|
||||
assertThat(next).isEqualTo(clockNumber + 1);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("옛 행이 reaper로 지워진 aggregate의 첫 번호는 지워진 번호가 아니라 floor 다음이다")
|
||||
void anAggregateWhoseRowsWereReapedStartsAboveTheFloor() {
|
||||
// given: a week-old clock-numbered row, published and deleted by the reaper since
|
||||
String aggregateId = "reaped-" + UUID.randomUUID();
|
||||
long clockNumber = 1_761_955_200_000_000L;
|
||||
append(aggregateId, clockNumber);
|
||||
jdbc.update("DELETE FROM outbox_event WHERE aggregate_id = ?", aggregateId);
|
||||
|
||||
// when
|
||||
long next = next(aggregateId);
|
||||
|
||||
// then
|
||||
assertThat(next).as("a number the consumer has already seen").isGreaterThan(clockNumber);
|
||||
assertThat(next).isEqualTo(floor() + 1);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("tenant scope가 바뀐 aggregate의 첫 번호는 옛 scope 행의 번호보다 큰 floor 다음이다")
|
||||
void anAggregateWhoseIdentityChangedStartsAboveTheFloor() {
|
||||
// given: the sample's rows were written under scope "portfolio" before it moved to one scope
|
||||
String aggregateId = "rescoped-" + UUID.randomUUID();
|
||||
long clockNumber = 1_761_955_200_000_000L;
|
||||
append(aggregateId, clockNumber);
|
||||
jdbc.update(
|
||||
"UPDATE outbox_event SET tenant_scope = 'portfolio' WHERE aggregate_id = ?", aggregateId);
|
||||
|
||||
// when
|
||||
long next = next(aggregateId);
|
||||
|
||||
// then
|
||||
assertThat(next).as("a number the consumer has already seen").isGreaterThan(clockNumber);
|
||||
assertThat(next).isEqualTo(floor() + 1);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("카운터가 생긴 뒤 옛 버전 인스턴스가 더 큰 번호로 쓴 행이 있으면 다음 번호는 그 행 다음이다")
|
||||
void aRowAnOldInstanceWroteAfterTheCounterExistedStillComesFirst() {
|
||||
// given: a new instance numbered the aggregate, then an old one wrote its clock number
|
||||
String aggregateId = "rolling-" + UUID.randomUUID();
|
||||
long counted = next(aggregateId);
|
||||
long oldInstancesNumber = counted + 10_000_000L;
|
||||
append(aggregateId, oldInstancesNumber);
|
||||
|
||||
// when
|
||||
long next = next(aggregateId);
|
||||
|
||||
// then
|
||||
assertThat(next).isEqualTo(oldInstancesNumber + 1);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("같은 aggregate에 같은 순서 번호로 두 번 append하면 두 번째는 실패한다")
|
||||
void aSecondEventUnderAnOrderItsAggregateAlreadyUsedIsRefused() {
|
||||
@@ -96,6 +151,19 @@ class PostgreSqlOutboxAggregateSequenceIntegrationTest extends OutboxAggregateSe
|
||||
assertThat(thrown).isNotNull();
|
||||
}
|
||||
|
||||
private static long next(String aggregateId) {
|
||||
return context
|
||||
.getBean(SpringTransaction.class)
|
||||
.inWrite(
|
||||
() -> sequences.nextSequence(new AggregateIdentity("default", "testkit", aggregateId)));
|
||||
}
|
||||
|
||||
/** The number V21 recorded once: the moment it ran, in epoch microseconds. */
|
||||
private static long floor() {
|
||||
return jdbc.queryForObject(
|
||||
"SELECT floor_sequence FROM outbox_aggregate_sequence_floor", Long.class);
|
||||
}
|
||||
|
||||
private static void append(String aggregateId, long sequence) {
|
||||
JpaOutboxStore store = context.getBean(JpaOutboxStore.class);
|
||||
context
|
||||
|
||||
+7
@@ -179,6 +179,13 @@ class PostgreSqlOutboxConcurrentRelayIntegrationTest extends OutboxLeaseFencingC
|
||||
"select status from outbox_event where event_id = ?", String.class, eventId));
|
||||
}
|
||||
|
||||
@Override
|
||||
protected String lastFailureCodeOf(String eventId) {
|
||||
return new JdbcTemplate(sharedDataSource)
|
||||
.queryForObject(
|
||||
"select last_failure_code from outbox_event where event_id = ?", String.class, eventId);
|
||||
}
|
||||
|
||||
/** Runs the relay in a loop until claimedCount == 0 (store drained). */
|
||||
private int drainRelay(PublishPendingOutboxEventsUseCase relay) {
|
||||
int total = 0;
|
||||
|
||||
+279
@@ -0,0 +1,279 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.readiness;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.tuple;
|
||||
|
||||
import com.zaxxer.hikari.HikariDataSource;
|
||||
import dev.caskeleton.adapter.outbound.persistence.outbox.jpa.JpaOutboxStore;
|
||||
import dev.caskeleton.adapter.outbound.persistence.transaction.SpringTransaction;
|
||||
import dev.caskeleton.application.outbox.OutboxLease;
|
||||
import dev.caskeleton.application.outbox.testkit.OutboxEvents;
|
||||
import java.io.IOException;
|
||||
import java.sql.SQLException;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
|
||||
/**
|
||||
* The outbox runbooks' manual dispositions, run as written against real PostgreSQL.
|
||||
*
|
||||
* <p>An operator resolves a DEAD row by re-driving or skipping it ({@code
|
||||
* docs/runbooks/outbox-dead-letter.md} §4), and a row that keeps failing for ever by finding it,
|
||||
* parking it, and resuming or skipping it ({@code docs/runbooks/outbox-publish-failed.md}, "끝나지 않는
|
||||
* 재시도"). The SQL for each is read out of the runbook, not copied, and its result is checked the
|
||||
* only way that matters: whether the relay's next claim takes the row. A re-drive that leaves a
|
||||
* counter behind is accepted by the database and then refused by the claim, which moves the row
|
||||
* back to DEAD — the failure the first re-drive test pins (re-review 2026-11-03 RR-05-01).
|
||||
*/
|
||||
class PostgreSqlOutboxDispositionRunbookIntegrationTest {
|
||||
|
||||
private static final Instant OCCURRED = Instant.parse("2026-11-01T00:00:00Z");
|
||||
private static final Duration IN_FLIGHT_TIMEOUT = Duration.ofMinutes(5);
|
||||
|
||||
private static PostgreSqlReadinessSupport postgres;
|
||||
private static OutboxRunbookSql deadLetterRunbook;
|
||||
private static OutboxRunbookSql publishFailedRunbook;
|
||||
|
||||
@BeforeAll
|
||||
static void setup() throws IOException {
|
||||
deadLetterRunbook = OutboxRunbookSql.fromSystemProperty("outbox.runbook");
|
||||
publishFailedRunbook = OutboxRunbookSql.fromSystemProperty("outbox.publish-failed.runbook");
|
||||
postgres = PostgreSqlReadinessSupport.start();
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
static void cleanup() {
|
||||
if (postgres != null) {
|
||||
postgres.close();
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("미뤄진 시도가 쌓인 DEAD 행을 runbook 재발행 SQL로 되돌리면 다음 claim이 그 행을 가져간다")
|
||||
void theRedriveMakesADeadRowWithDeferredClaimsClaimableAgain() throws SQLException {
|
||||
try (Outbox outbox = Outbox.create()) {
|
||||
// given: dead-lettered after two claims an outage deferred and three that spent the budget
|
||||
String dead = outbox.append("order-" + UUID.randomUUID(), 1, OCCURRED);
|
||||
outbox.setCounters(dead, "DEAD", 5, 2);
|
||||
String redrive = deadLetterRunbook.block("dead-redrive-one", Map.of("id", dead));
|
||||
|
||||
// when
|
||||
outbox.jdbc().execute(redrive);
|
||||
|
||||
// then
|
||||
assertThat(outbox.claimedEventIds()).containsExactly(dead);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("runbook 구간 재발행 SQL은 그 구간의 DEAD 행만 claim 가능하게 되돌린다")
|
||||
void theWindowRedriveReturnsOnlyTheDeadRowsOfTheWindow() throws SQLException {
|
||||
try (Outbox outbox = Outbox.create()) {
|
||||
// given
|
||||
String early = outbox.append("order-" + UUID.randomUUID(), 1, OCCURRED);
|
||||
String late = outbox.append("order-" + UUID.randomUUID(), 1, OCCURRED.plusSeconds(60));
|
||||
String outside = outbox.append("order-" + UUID.randomUUID(), 1, OCCURRED.plusSeconds(3600));
|
||||
for (String eventId : List.of(early, late, outside)) {
|
||||
outbox.setCounters(eventId, "DEAD", 3, 2);
|
||||
}
|
||||
String redrive =
|
||||
deadLetterRunbook.block(
|
||||
"dead-redrive-window",
|
||||
Map.of(
|
||||
"장애 시작", OCCURRED.minusSeconds(1).toString(),
|
||||
"장애 끝", OCCURRED.plusSeconds(61).toString()));
|
||||
|
||||
// when
|
||||
outbox.jdbc().execute(redrive);
|
||||
|
||||
// then
|
||||
assertThat(outbox.claimedEventIds()).containsExactlyInAnyOrder(early, late);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("runbook skip SQL은 DEAD 행을 발행된 것으로 두어 그 aggregate의 다음 행을 풀어 준다")
|
||||
void theSkipReleasesTheNextRowOfTheDeadRowsAggregate() throws SQLException {
|
||||
try (Outbox outbox = Outbox.create()) {
|
||||
// given
|
||||
String aggregate = "order-" + UUID.randomUUID();
|
||||
String dead = outbox.append(aggregate, 1, OCCURRED);
|
||||
String next = outbox.append(aggregate, 2, OCCURRED.plusSeconds(1));
|
||||
outbox.setCounters(dead, "DEAD", 1, 0);
|
||||
String skip = deadLetterRunbook.block("dead-skip-one", Map.of("id", dead));
|
||||
|
||||
// when
|
||||
outbox.jdbc().execute(skip);
|
||||
|
||||
// then
|
||||
assertThat(List.of(outbox.claimedEventIds(), outbox.status(dead)))
|
||||
.containsExactly(List.of(next), "PUBLISHED");
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("오래 재시도 중인 행 조회는 미룬 claim이 기준 수 이상인 FAILED 행과 그 원인 코드를 보여 준다")
|
||||
void theLongRetryQueryListsFailedRowsDeferredLongEnoughWithTheirFailureCode()
|
||||
throws SQLException {
|
||||
try (Outbox outbox = Outbox.create()) {
|
||||
// given: one row deferred long enough, one that has only just started failing
|
||||
String stuck = outbox.append("order-" + UUID.randomUUID(), 1, OCCURRED);
|
||||
String fresh = outbox.append("order-" + UUID.randomUUID(), 1, OCCURRED);
|
||||
outbox.setCounters(stuck, "FAILED", 12, 12);
|
||||
outbox.setCounters(fresh, "FAILED", 1, 1);
|
||||
outbox.setFailureCode(stuck, "KAFKA_INVALIDTIMESTAMPEXCEPTION");
|
||||
|
||||
// when
|
||||
List<Map<String, Object>> found =
|
||||
outbox.jdbc().queryForList(publishFailedRunbook.block("long-retries"));
|
||||
|
||||
// then
|
||||
assertThat(found)
|
||||
.extracting(row -> row.get("event_id"), row -> row.get("last_failure_code"))
|
||||
.containsExactly(tuple(stuck, "KAFKA_INVALIDTIMESTAMPEXCEPTION"));
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("runbook park SQL로 보류한 행은 claim되지 않고 그 aggregate의 다음 행도 막는다")
|
||||
void aParkedRowIsNotClaimedAndStillHoldsItsAggregatesNextRow() throws SQLException {
|
||||
try (Outbox outbox = Outbox.create()) {
|
||||
// given
|
||||
String aggregate = "order-" + UUID.randomUUID();
|
||||
String failing = outbox.append(aggregate, 1, OCCURRED);
|
||||
outbox.append(aggregate, 2, OCCURRED.plusSeconds(1));
|
||||
outbox.setCounters(failing, "FAILED", 12, 12);
|
||||
String park = publishFailedRunbook.block("park-one", Map.of("id", failing));
|
||||
|
||||
// when
|
||||
outbox.jdbc().execute(park);
|
||||
|
||||
// then
|
||||
assertThat(List.of(outbox.claimedEventIds(), outbox.status(failing)))
|
||||
.containsExactly(List.of(), "PARKED");
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("runbook resume SQL로 되돌린 보류 행은 다음 claim이 가져간다")
|
||||
void aResumedRowIsClaimedAgain() throws SQLException {
|
||||
try (Outbox outbox = Outbox.create()) {
|
||||
// given
|
||||
String parked = outbox.append("order-" + UUID.randomUUID(), 1, OCCURRED);
|
||||
outbox.setCounters(parked, "FAILED", 12, 12);
|
||||
outbox.jdbc().execute(publishFailedRunbook.block("park-one", Map.of("id", parked)));
|
||||
String resume = publishFailedRunbook.block("resume-one", Map.of("id", parked));
|
||||
|
||||
// when
|
||||
outbox.jdbc().execute(resume);
|
||||
|
||||
// then
|
||||
assertThat(outbox.claimedEventIds()).containsExactly(parked);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("runbook skip SQL로 버린 보류 행은 발행된 것으로 남고 그 aggregate의 다음 행을 풀어 준다")
|
||||
void aSkippedParkedRowReleasesItsAggregatesNextRow() throws SQLException {
|
||||
try (Outbox outbox = Outbox.create()) {
|
||||
// given
|
||||
String aggregate = "order-" + UUID.randomUUID();
|
||||
String parked = outbox.append(aggregate, 1, OCCURRED);
|
||||
String next = outbox.append(aggregate, 2, OCCURRED.plusSeconds(1));
|
||||
outbox.setCounters(parked, "FAILED", 12, 12);
|
||||
outbox.jdbc().execute(publishFailedRunbook.block("park-one", Map.of("id", parked)));
|
||||
String skip = publishFailedRunbook.block("parked-skip-one", Map.of("id", parked));
|
||||
|
||||
// when
|
||||
outbox.jdbc().execute(skip);
|
||||
|
||||
// then
|
||||
assertThat(List.of(outbox.claimedEventIds(), outbox.status(parked)))
|
||||
.containsExactly(List.of(next), "PUBLISHED");
|
||||
}
|
||||
}
|
||||
|
||||
/** A fresh database at the latest migration with the JPA outbox store over it. */
|
||||
private static final class Outbox implements AutoCloseable {
|
||||
|
||||
private final HikariDataSource dataSource;
|
||||
private final AnnotationConfigApplicationContext context;
|
||||
private final JdbcTemplate jdbc;
|
||||
|
||||
private Outbox(HikariDataSource dataSource) {
|
||||
this.dataSource = dataSource;
|
||||
OutboxIntegrationSupport.migrate(dataSource);
|
||||
this.context = OutboxIntegrationSupport.buildContext(dataSource, e -> {}, Clock.systemUTC());
|
||||
this.jdbc = new JdbcTemplate(dataSource);
|
||||
}
|
||||
|
||||
static Outbox create() throws SQLException {
|
||||
String name =
|
||||
"runbook_" + UUID.randomUUID().toString().replace("-", "").toLowerCase(Locale.ROOT);
|
||||
return new Outbox(postgres.createDatabase(name));
|
||||
}
|
||||
|
||||
JdbcTemplate jdbc() {
|
||||
return jdbc;
|
||||
}
|
||||
|
||||
String append(String aggregateId, long sequence, Instant occurredAt) {
|
||||
String eventId = UUID.randomUUID().toString();
|
||||
JpaOutboxStore store = context.getBean(JpaOutboxStore.class);
|
||||
context
|
||||
.getBean(SpringTransaction.class)
|
||||
.inWrite(
|
||||
() -> store.append(OutboxEvents.event(eventId, aggregateId, sequence, occurredAt)));
|
||||
return eventId;
|
||||
}
|
||||
|
||||
/** The counters a relay leaves on a row; written directly because only their values matter. */
|
||||
void setCounters(String eventId, String status, int attempts, int deferred) {
|
||||
jdbc.update(
|
||||
"UPDATE outbox_event SET status = ?, attempt_count = ?, deferred_attempt_count = ?"
|
||||
+ " WHERE event_id = ?",
|
||||
status,
|
||||
attempts,
|
||||
deferred,
|
||||
eventId);
|
||||
}
|
||||
|
||||
void setFailureCode(String eventId, String failureCode) {
|
||||
jdbc.update(
|
||||
"UPDATE outbox_event SET last_failure_code = ? WHERE event_id = ?", failureCode, eventId);
|
||||
}
|
||||
|
||||
String status(String eventId) {
|
||||
return jdbc.queryForObject(
|
||||
"SELECT status FROM outbox_event WHERE event_id = ?", String.class, eventId);
|
||||
}
|
||||
|
||||
/** What the relay's next claim takes, a minute from now so every re-driven row is due. */
|
||||
List<String> claimedEventIds() {
|
||||
JpaOutboxStore store = context.getBean(JpaOutboxStore.class);
|
||||
Instant now = Instant.now().plusSeconds(60);
|
||||
return context
|
||||
.getBean(SpringTransaction.class)
|
||||
.inWrite(() -> store.claimBatch("runbook-relay", 50, now, IN_FLIGHT_TIMEOUT))
|
||||
.stream()
|
||||
.map(OutboxLease::eventId)
|
||||
.toList();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
context.close();
|
||||
dataSource.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
+166
-18
@@ -9,16 +9,20 @@ import dev.caskeleton.adapter.outbound.persistence.outbox.jpa.JpaOutboxStore;
|
||||
import dev.caskeleton.adapter.outbound.persistence.transaction.SpringTransaction;
|
||||
import dev.caskeleton.application.outbox.testkit.OutboxEvents;
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.sql.Connection;
|
||||
import java.sql.SQLException;
|
||||
import java.sql.Statement;
|
||||
import java.time.Clock;
|
||||
import java.time.Instant;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.ScheduledExecutorService;
|
||||
import java.util.concurrent.ScheduledFuture;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import org.flywaydb.core.Flyway;
|
||||
import org.flywaydb.core.api.FlywayException;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
@@ -43,15 +47,15 @@ class PostgreSqlOutboxOrderKeyRunbookIntegrationTest {
|
||||
private static final Instant OCCURRED = Instant.parse("2026-11-01T00:00:00Z");
|
||||
|
||||
private static PostgreSqlReadinessSupport postgres;
|
||||
private static OutboxRunbookSql runbook;
|
||||
private static String duplicateCheck;
|
||||
private static String removePublishedDuplicates;
|
||||
|
||||
@BeforeAll
|
||||
static void setup() throws IOException {
|
||||
String runbook =
|
||||
Files.readString(Path.of(System.getProperty("outbox.runbook")), StandardCharsets.UTF_8);
|
||||
duplicateCheck = runbookSql(runbook, "v21-duplicate-order-keys");
|
||||
removePublishedDuplicates = runbookSql(runbook, "v21-remove-published-duplicates");
|
||||
runbook = OutboxRunbookSql.fromSystemProperty("outbox.runbook");
|
||||
duplicateCheck = runbook.block("v21-duplicate-order-keys");
|
||||
removePublishedDuplicates = runbook.block("v21-remove-published-duplicates");
|
||||
postgres = PostgreSqlReadinessSupport.start();
|
||||
}
|
||||
|
||||
@@ -176,19 +180,143 @@ class PostgreSqlOutboxOrderKeyRunbookIntegrationTest {
|
||||
}
|
||||
}
|
||||
|
||||
/** The SQL block that follows {@code <!-- runbook-sql: id -->} in the runbook. */
|
||||
private static String runbookSql(String runbook, String id) {
|
||||
String marker = "<!-- runbook-sql: " + id + " -->";
|
||||
int at = runbook.indexOf(marker);
|
||||
if (at < 0) {
|
||||
throw new IllegalStateException("the outbox runbook has no SQL block marked " + id);
|
||||
@Test
|
||||
@DisplayName("V21은 남은 행이 있는 aggregate의 카운터를 그 aggregate의 가장 큰 번호로 미리 만든다")
|
||||
void v21SeedsTheCounterOfEveryAggregateWithRowsAtItsHighestNumber() throws SQLException {
|
||||
try (BeforeV21 database = BeforeV21.create()) {
|
||||
// given
|
||||
String twice = "twice-" + UUID.randomUUID();
|
||||
database.append(twice, 1_761_955_200_000_003L);
|
||||
database.append(twice, 1_761_955_200_000_009L);
|
||||
String once = "once-" + UUID.randomUUID();
|
||||
database.append(once, 1_761_955_200_000_004L);
|
||||
|
||||
// when
|
||||
database.migrateToLatest();
|
||||
|
||||
// then
|
||||
assertThat(List.of(database.counterOf(twice), database.counterOf(once)))
|
||||
.containsExactly(1_761_955_200_000_009L, 1_761_955_200_000_004L);
|
||||
}
|
||||
int open = runbook.indexOf("```sql\n", at);
|
||||
int close = runbook.indexOf("\n```", open + 1);
|
||||
if (open < 0 || close < 0) {
|
||||
throw new IllegalStateException("the block marked " + id + " is not a closed sql fence");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("V21은 적용한 시각을 마이크로초 floor로 한 번 기록한다")
|
||||
void v21RecordsTheMomentItRanAsTheFloorInMicroseconds() throws SQLException {
|
||||
try (BeforeV21 database = BeforeV21.create()) {
|
||||
// given
|
||||
long before = microsecondsNow() - 1_000_000L;
|
||||
|
||||
// when
|
||||
database.migrateToLatest();
|
||||
|
||||
// then
|
||||
long after = microsecondsNow() + 1_000_000L;
|
||||
assertThat(
|
||||
database
|
||||
.jdbc()
|
||||
.queryForList(
|
||||
"SELECT floor_sequence FROM outbox_aggregate_sequence_floor", Long.class))
|
||||
.singleElement()
|
||||
.satisfies(floor -> assertThat(floor).isBetween(before, after));
|
||||
}
|
||||
return runbook.substring(open + "```sql\n".length(), close);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("outbox 쓰기 트랜잭션이 잠금을 잡고 있으면 V21은 기다리다 실패하고 DB는 V20 그대로 남는다")
|
||||
void v21GivesUpOnALockAWriterHoldsAndLeavesTheDatabaseAtV20() throws Exception {
|
||||
try (BeforeV21 database = BeforeV21.create();
|
||||
Connection writer = database.connection()) {
|
||||
// given: a writer transaction holding the lock every outbox insert takes, released after 15 s
|
||||
writer.setAutoCommit(false);
|
||||
try (Statement statement = writer.createStatement()) {
|
||||
statement.execute("LOCK TABLE outbox_event IN ROW EXCLUSIVE MODE");
|
||||
}
|
||||
ScheduledExecutorService release = Executors.newSingleThreadScheduledExecutor();
|
||||
ScheduledFuture<Void> released =
|
||||
release.schedule(
|
||||
() -> {
|
||||
writer.rollback();
|
||||
return null;
|
||||
},
|
||||
15,
|
||||
TimeUnit.SECONDS);
|
||||
try {
|
||||
// when
|
||||
Throwable thrown = catchThrowable(database::migrateToLatest);
|
||||
|
||||
// then
|
||||
assertThat(
|
||||
List.of(
|
||||
thrown instanceof FlywayException,
|
||||
database.flyway(null).info().current().getVersion().getVersion()))
|
||||
.containsExactly(true, "20");
|
||||
} finally {
|
||||
released.cancel(true);
|
||||
release.shutdownNow();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("실패한 CONCURRENTLY 빌드가 남긴 무효 인덱스가 있으면 V21은 그것을 건너뛰지 않고 실패한다")
|
||||
void anInvalidIndexLeftByAFailedConcurrentBuildFailsV21() throws SQLException {
|
||||
try (BeforeV21 database = BeforeV21.create()) {
|
||||
// given: a concurrent build that met a duplicate leaves an invalid index behind
|
||||
String shared = "shared-" + UUID.randomUUID();
|
||||
String kept = database.append(shared, 1);
|
||||
database.append(shared, 1);
|
||||
Throwable failedBuild =
|
||||
catchThrowable(
|
||||
() ->
|
||||
database
|
||||
.jdbc()
|
||||
.execute(
|
||||
"CREATE UNIQUE INDEX CONCURRENTLY ux_outbox_event_aggregate_order ON"
|
||||
+ " outbox_event (tenant_scope, aggregate_type, aggregate_id,"
|
||||
+ " aggregate_sequence, event_index)"));
|
||||
assertThat(failedBuild).as("the concurrent build met the duplicate").isNotNull();
|
||||
database
|
||||
.jdbc()
|
||||
.update(
|
||||
"DELETE FROM outbox_event WHERE aggregate_id = ? AND event_id <> ?", shared, kept);
|
||||
|
||||
// when
|
||||
Throwable thrown = catchThrowable(database::migrateToLatest);
|
||||
|
||||
// then
|
||||
assertThat(
|
||||
List.of(
|
||||
thrown instanceof FlywayException,
|
||||
database.flyway(null).info().current().getVersion().getVersion()))
|
||||
.containsExactly(true, "20");
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("runbook대로 순서 키 인덱스를 CONCURRENTLY로 먼저 만들면 V21은 그 인덱스를 쓰고 적용된다")
|
||||
void v21AppliesOverTheIndexTheRunbookBuiltConcurrently() throws SQLException {
|
||||
try (BeforeV21 database = BeforeV21.create()) {
|
||||
// given
|
||||
database.append("ordered-" + UUID.randomUUID(), 1);
|
||||
database.jdbc().execute(runbook.block("v21-prebuild-order-index"));
|
||||
|
||||
// when
|
||||
database.migrateToLatest();
|
||||
|
||||
// then
|
||||
assertThat(
|
||||
List.of(
|
||||
Arrays.stream(database.flyway(null).info().applied())
|
||||
.anyMatch(migration -> "21".equals(migration.getVersion().getVersion())),
|
||||
database.validIndexExists("ux_outbox_event_aggregate_order")))
|
||||
.containsExactly(true, true);
|
||||
}
|
||||
}
|
||||
|
||||
private static long microsecondsNow() {
|
||||
Instant now = Instant.now();
|
||||
return now.getEpochSecond() * 1_000_000L + now.getNano() / 1_000L;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -259,6 +387,26 @@ class PostgreSqlOutboxOrderKeyRunbookIntegrationTest {
|
||||
jdbc.queryForObject("SELECT to_regclass(?) IS NOT NULL", Boolean.class, table));
|
||||
}
|
||||
|
||||
Connection connection() throws SQLException {
|
||||
return dataSource.getConnection();
|
||||
}
|
||||
|
||||
long counterOf(String aggregateId) {
|
||||
return jdbc.queryForObject(
|
||||
"SELECT last_sequence FROM outbox_aggregate_sequence WHERE aggregate_id = ?",
|
||||
Long.class,
|
||||
aggregateId);
|
||||
}
|
||||
|
||||
boolean validIndexExists(String index) {
|
||||
return Boolean.TRUE.equals(
|
||||
jdbc.queryForObject(
|
||||
"SELECT EXISTS (SELECT 1 FROM pg_index WHERE indexrelid = to_regclass(?)"
|
||||
+ " AND indisvalid)",
|
||||
Boolean.class,
|
||||
index));
|
||||
}
|
||||
|
||||
boolean indexExists(String index) {
|
||||
return Boolean.TRUE.equals(
|
||||
jdbc.queryForObject(
|
||||
|
||||
+60
@@ -0,0 +1,60 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.readiness;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaRecoveryQueue;
|
||||
import dev.caskeleton.application.fileserver.api.FileId;
|
||||
import dev.caskeleton.application.fileserver.api.StorageNamespace;
|
||||
import dev.caskeleton.application.fileserver.recovery.RecoveryQueuePort;
|
||||
import dev.caskeleton.application.fileserver.testkit.RecoveryQueuePortContract;
|
||||
import java.time.Duration;
|
||||
import java.util.function.Supplier;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Tag;
|
||||
|
||||
/**
|
||||
* {@link JpaRecoveryQueue} held to the contract the in-memory queue passes, on real PostgreSQL
|
||||
* (review 2026-11-03 RR-01-04).
|
||||
*/
|
||||
@Tag("jpa-contract")
|
||||
class PostgreSqlRecoveryQueueContractTest extends RecoveryQueuePortContract {
|
||||
|
||||
private static FileserverPostgreSqlStack stack;
|
||||
|
||||
@BeforeAll
|
||||
static void start() {
|
||||
stack = FileserverPostgreSqlStack.start();
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
static void stop() {
|
||||
if (stack != null) {
|
||||
stack.close();
|
||||
}
|
||||
}
|
||||
|
||||
@BeforeEach
|
||||
void empty() {
|
||||
stack.empty();
|
||||
}
|
||||
|
||||
@Override
|
||||
protected RecoveryQueuePort queue() {
|
||||
return stack.bean(JpaRecoveryQueue.class);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected FileId recordedFile() {
|
||||
return stack.recordFileIn(StorageNamespace.of("tenant-a"));
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void elapse(Duration duration) {
|
||||
stack.elapse(duration);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected <T> T inTransaction(Supplier<T> work) {
|
||||
return stack.inTransaction(work);
|
||||
}
|
||||
}
|
||||
+73
@@ -0,0 +1,73 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.readiness;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaStagingUploadLocator;
|
||||
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaUploadSession;
|
||||
import dev.caskeleton.application.fileserver.api.FileId;
|
||||
import dev.caskeleton.application.fileserver.api.StorageNamespace;
|
||||
import dev.caskeleton.application.fileserver.api.metadata.UploadSessionPort;
|
||||
import dev.caskeleton.application.fileserver.recovery.StagingUploadLocatorPort;
|
||||
import dev.caskeleton.application.fileserver.testkit.StagingUploadLocatorPortContract;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.function.Supplier;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Tag;
|
||||
|
||||
/**
|
||||
* {@link JpaStagingUploadLocator} held to the contract the in-memory session store passes, on real
|
||||
* PostgreSQL (review 2026-11-03 RR-01-04).
|
||||
*/
|
||||
@Tag("jpa-contract")
|
||||
class PostgreSqlStagingUploadLocatorContractTest extends StagingUploadLocatorPortContract {
|
||||
|
||||
private static FileserverPostgreSqlStack stack;
|
||||
|
||||
@BeforeAll
|
||||
static void start() {
|
||||
stack = FileserverPostgreSqlStack.start();
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
static void stop() {
|
||||
if (stack != null) {
|
||||
stack.close();
|
||||
}
|
||||
}
|
||||
|
||||
@BeforeEach
|
||||
void empty() {
|
||||
stack.empty();
|
||||
}
|
||||
|
||||
@Override
|
||||
protected StagingUploadLocatorPort locator() {
|
||||
return stack.bean(JpaStagingUploadLocator.class);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected UploadSessionPort sessions() {
|
||||
return stack.bean(JpaUploadSession.class);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected FileId recordedFile() {
|
||||
return stack.recordFileIn(StorageNamespace.of("tenant-a"));
|
||||
}
|
||||
|
||||
@Override
|
||||
protected Instant now() {
|
||||
return stack.now();
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void elapse(Duration duration) {
|
||||
stack.elapse(duration);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected <T> T inTransaction(Supplier<T> work) {
|
||||
return stack.inTransaction(work);
|
||||
}
|
||||
}
|
||||
+52
@@ -293,6 +293,49 @@ class PostgreSqlTransactionIntegrationTest {
|
||||
assertThat(elapsed).isLessThan(Duration.ofSeconds(5));
|
||||
}
|
||||
|
||||
/**
|
||||
* The production transaction manager is {@code JpaTransactionManager}, and past the deadline
|
||||
* Hibernate refuses the next statement itself, before the driver is asked: no {@code 57014}, no
|
||||
* SQLState at all (review RR-03-03). The test above covers the JDBC path only.
|
||||
*/
|
||||
@Test
|
||||
@Tag("jpa-failure")
|
||||
@Timeout(20)
|
||||
@DisplayName("JPA 트랜잭션이 제한 시간을 넘긴 뒤 다음 문장을 Hibernate가 거부해도 DB_QUERY_CANCELED로 끝난다")
|
||||
void aJpaWriteTransactionPastItsBoundIsAQueryCancellation() {
|
||||
// given
|
||||
Configuration configuration = new Configuration();
|
||||
configuration.getProperties().put("hibernate.connection.datasource", postgres.dataSource());
|
||||
configuration.setProperty("hibernate.hbm2ddl.auto", "create");
|
||||
configuration.addAnnotatedClass(VersionedRow.class);
|
||||
try (SessionFactory sessionFactory = configuration.buildSessionFactory()) {
|
||||
EntityManager entityManager =
|
||||
SharedEntityManagerCreator.createSharedEntityManager(sessionFactory);
|
||||
SpringTransaction port =
|
||||
new SpringTransaction(
|
||||
new JpaTransactionManager(sessionFactory),
|
||||
translator(),
|
||||
new SpringTransaction.Timeouts(Duration.ofSeconds(1), Duration.ofSeconds(1)));
|
||||
|
||||
// when
|
||||
Throwable thrown =
|
||||
catchThrowable(
|
||||
() ->
|
||||
port.inWrite(
|
||||
() -> {
|
||||
sleepPastOneSecond();
|
||||
entityManager.persist(new VersionedRow(7L, "late"));
|
||||
entityManager.flush();
|
||||
return null;
|
||||
}));
|
||||
|
||||
// then
|
||||
assertThat(thrown).isInstanceOf(PersistenceFailureException.class);
|
||||
assertThat(((PersistenceFailureException) thrown).errorCode())
|
||||
.isEqualTo(OperationalError.DB_QUERY_CANCELED);
|
||||
}
|
||||
}
|
||||
|
||||
// ---- review 09-05: READ COMMITTED comes from the pool, with no per-transaction round trip ----
|
||||
|
||||
@Test
|
||||
@@ -323,6 +366,15 @@ class PostgreSqlTransactionIntegrationTest {
|
||||
}
|
||||
}
|
||||
|
||||
private static void sleepPastOneSecond() {
|
||||
try {
|
||||
Thread.sleep(1_200);
|
||||
} catch (InterruptedException interrupted) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new IllegalStateException(interrupted);
|
||||
}
|
||||
}
|
||||
|
||||
/** A data source whose connections count {@code getTransactionIsolation()} calls. */
|
||||
private static DataSource countingIsolationQueries(DataSource target, AtomicInteger calls) {
|
||||
return (DataSource)
|
||||
|
||||
+66
@@ -0,0 +1,66 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.readiness;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaUploadSession;
|
||||
import dev.caskeleton.application.fileserver.api.FileId;
|
||||
import dev.caskeleton.application.fileserver.api.StorageNamespace;
|
||||
import dev.caskeleton.application.fileserver.api.metadata.UploadSessionPort;
|
||||
import dev.caskeleton.application.fileserver.testkit.UploadSessionPortContract;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.function.Supplier;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Tag;
|
||||
|
||||
/**
|
||||
* {@link JpaUploadSession} held to the writer-lease contract the in-memory store passes, on real
|
||||
* PostgreSQL (review 2026-11-03 RR-01-04).
|
||||
*/
|
||||
@Tag("jpa-contract")
|
||||
class PostgreSqlUploadSessionContractTest extends UploadSessionPortContract {
|
||||
|
||||
private static FileserverPostgreSqlStack stack;
|
||||
|
||||
@BeforeAll
|
||||
static void start() {
|
||||
stack = FileserverPostgreSqlStack.start();
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
static void stop() {
|
||||
if (stack != null) {
|
||||
stack.close();
|
||||
}
|
||||
}
|
||||
|
||||
@BeforeEach
|
||||
void empty() {
|
||||
stack.empty();
|
||||
}
|
||||
|
||||
@Override
|
||||
protected UploadSessionPort sessions() {
|
||||
return stack.bean(JpaUploadSession.class);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected FileId recordedFile() {
|
||||
return stack.recordFileIn(StorageNamespace.of("tenant-a"));
|
||||
}
|
||||
|
||||
@Override
|
||||
protected Instant now() {
|
||||
return stack.now();
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void elapse(Duration duration) {
|
||||
stack.elapse(duration);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected <T> T inTransaction(Supplier<T> work) {
|
||||
return stack.inTransaction(work);
|
||||
}
|
||||
}
|
||||
+64
-6
@@ -13,6 +13,9 @@ import static org.mockito.Mockito.verify;
|
||||
import static org.mockito.Mockito.verifyNoInteractions;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
import ch.qos.logback.classic.Logger;
|
||||
import ch.qos.logback.classic.spi.ILoggingEvent;
|
||||
import ch.qos.logback.core.read.ListAppender;
|
||||
import dev.caskeleton.adapter.outbound.persistence.outbox.jpa.entity.OutboxEventEntity;
|
||||
import dev.caskeleton.application.messaging.contract.ContractId;
|
||||
import dev.caskeleton.application.messaging.contract.LogicalDestinationId;
|
||||
@@ -29,13 +32,16 @@ import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.HexFormat;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.mockito.ArgumentCaptor;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.transaction.support.TransactionSynchronizationManager;
|
||||
|
||||
/**
|
||||
@@ -181,10 +187,10 @@ class JpaOutboxStoreTest {
|
||||
assertThatThrownBy(() -> adapter.markPublished(lease("evt-guard")))
|
||||
.isInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining("markPublished");
|
||||
assertThatThrownBy(() -> adapter.markFailed(lease("evt-guard"), NOW.plusSeconds(1)))
|
||||
assertThatThrownBy(() -> adapter.markFailed(lease("evt-guard"), NOW.plusSeconds(1), "CODE"))
|
||||
.isInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining("markFailed");
|
||||
assertThatThrownBy(() -> adapter.markDead(lease("evt-guard")))
|
||||
assertThatThrownBy(() -> adapter.markDead(lease("evt-guard"), "CODE"))
|
||||
.isInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining("markDead");
|
||||
}
|
||||
@@ -281,6 +287,56 @@ class JpaOutboxStoreTest {
|
||||
assertThat(List.of(claimed, entity.getStatus())).containsExactly(List.of(), "DEAD");
|
||||
}
|
||||
|
||||
/**
|
||||
* A row whose deferred claims outnumber its claims was not damaged: a manual re-drive reset
|
||||
* {@code attempt_count} and left {@code deferred_attempt_count} (re-review 2026-11-03 RR-05-01).
|
||||
* The WARN says that, so the operator re-runs the corrected runbook SQL instead of hunting for
|
||||
* corruption.
|
||||
*/
|
||||
@Test
|
||||
@DisplayName("claim이 시도 횟수가 어긋난 행을 DEAD로 옮기면 WARN이 어긋난 두 값과 runbook을 말한다")
|
||||
void claimBatchNamesInconsistentAttemptCountersWhenItMovesSuchARowToDead() {
|
||||
// given
|
||||
OutboxEventEntity redriven = pendingEntity("evt-redriven", "worklog-42");
|
||||
redriven.setDeferredAttemptCount(2);
|
||||
when(claimRepo.claimEligible(eq(NOW), eq(1))).thenReturn(List.of(redriven));
|
||||
Logger storeLog = (Logger) LoggerFactory.getLogger(JpaOutboxStore.class);
|
||||
ListAppender<ILoggingEvent> captured = new ListAppender<>();
|
||||
captured.start();
|
||||
storeLog.addAppender(captured);
|
||||
|
||||
// when
|
||||
try {
|
||||
adapter.claimBatch(OWNER, 1, NOW, IN_FLIGHT_TIMEOUT);
|
||||
} finally {
|
||||
storeLog.detachAppender(captured);
|
||||
}
|
||||
|
||||
// then
|
||||
assertThat(captured.list)
|
||||
.singleElement()
|
||||
.satisfies(
|
||||
warn ->
|
||||
assertThat(List.of(warn.getFormattedMessage(), keyValues(warn)))
|
||||
.containsExactly(
|
||||
"outbox claim moved a row to DEAD: deferred_attempt_count (2) exceeds"
|
||||
+ " attempt_count (1); a manual re-drive that did not reset"
|
||||
+ " deferred_attempt_count leaves this behind — re-drive it with the"
|
||||
+ " runbook SQL",
|
||||
Map.of(
|
||||
"event_id", "evt-redriven",
|
||||
"error.reason", "ATTEMPT_COUNTERS_INCONSISTENT",
|
||||
"runbook_link", "runbook://outbox/dead-letter")));
|
||||
}
|
||||
|
||||
private static Map<String, Object> keyValues(ILoggingEvent event) {
|
||||
Map<String, Object> values = new LinkedHashMap<>();
|
||||
if (event.getKeyValuePairs() != null) {
|
||||
event.getKeyValuePairs().forEach(pair -> values.put(pair.key, pair.value));
|
||||
}
|
||||
return values;
|
||||
}
|
||||
|
||||
@Test
|
||||
void claimBatchReturnsEmptyWhenRepoReturnsNothing() {
|
||||
when(claimRepo.claimEligible(any(), anyInt())).thenReturn(List.of());
|
||||
@@ -319,16 +375,18 @@ class JpaOutboxStoreTest {
|
||||
@Test
|
||||
void markFailedCompletesOnlyUnderTheLeaseAndSchedulesTheRetry() {
|
||||
Instant retryAt = NOW.plus(Duration.ofSeconds(30));
|
||||
when(repo.failLease("evt-004", OWNER, 1, retryAt)).thenReturn(1);
|
||||
when(repo.failLease("evt-004", OWNER, 1, retryAt, "IllegalStateException")).thenReturn(1);
|
||||
|
||||
assertThat(adapter.markFailed(lease("evt-004"), retryAt)).isEqualTo(OutboxCompletion.APPLIED);
|
||||
assertThat(adapter.markFailed(lease("evt-004"), retryAt, "IllegalStateException"))
|
||||
.isEqualTo(OutboxCompletion.APPLIED);
|
||||
}
|
||||
|
||||
@Test
|
||||
void markDeadCompletesOnlyUnderTheLease() {
|
||||
when(repo.completeLease("evt-005", OWNER, 1, "DEAD")).thenReturn(1);
|
||||
when(repo.deadLease("evt-005", OWNER, 1, "REJECTED_BEFORE_SEND")).thenReturn(1);
|
||||
|
||||
assertThat(adapter.markDead(lease("evt-005"))).isEqualTo(OutboxCompletion.APPLIED);
|
||||
assertThat(adapter.markDead(lease("evt-005"), "REJECTED_BEFORE_SEND"))
|
||||
.isEqualTo(OutboxCompletion.APPLIED);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+1
-1
@@ -111,7 +111,7 @@ class FlywayProdSafetyValidatorTest {
|
||||
@Configuration
|
||||
static class ValidatorConfig {
|
||||
@Bean
|
||||
FlywayProdSafetyValidator flywayProdSafetyValidator(Environment environment) {
|
||||
static FlywayProdSafetyValidator flywayProdSafetyValidator(Environment environment) {
|
||||
return new FlywayProdSafetyValidator();
|
||||
}
|
||||
}
|
||||
|
||||
+1
-1
@@ -300,7 +300,7 @@ class HikariPoolConstraintValidatorTest {
|
||||
@Configuration
|
||||
static class ValidatorConfig {
|
||||
@Bean
|
||||
HikariPoolConstraintValidator hikariPoolConstraintValidator(Environment environment) {
|
||||
static HikariPoolConstraintValidator hikariPoolConstraintValidator(Environment environment) {
|
||||
return new HikariPoolConstraintValidator();
|
||||
}
|
||||
}
|
||||
|
||||
+1
-1
@@ -72,7 +72,7 @@ class OpenInViewSafetyValidatorTest {
|
||||
@Configuration
|
||||
static class ValidatorConfig {
|
||||
@Bean
|
||||
OpenInViewSafetyValidator openInViewSafetyValidator(Environment environment) {
|
||||
static OpenInViewSafetyValidator openInViewSafetyValidator(Environment environment) {
|
||||
return new OpenInViewSafetyValidator();
|
||||
}
|
||||
}
|
||||
|
||||
+1
-1
@@ -101,7 +101,7 @@ class PersistenceVendorProdSafetyValidatorTest {
|
||||
static class ValidatorConfig {
|
||||
|
||||
@Bean
|
||||
PersistenceVendorProdSafetyValidator validator(Environment environment) {
|
||||
static PersistenceVendorProdSafetyValidator validator(Environment environment) {
|
||||
return new PersistenceVendorProdSafetyValidator();
|
||||
}
|
||||
}
|
||||
|
||||
+1
-1
@@ -102,7 +102,7 @@ class PostgreSqlTransportSecurityValidatorTest {
|
||||
static class ValidatorConfig {
|
||||
|
||||
@Bean
|
||||
PostgreSqlTransportSecurityValidator postgreSqlTransportSecurityValidator(
|
||||
static PostgreSqlTransportSecurityValidator postgreSqlTransportSecurityValidator(
|
||||
Environment environment) {
|
||||
return new PostgreSqlTransportSecurityValidator();
|
||||
}
|
||||
|
||||
+1
-1
@@ -81,7 +81,7 @@ class RequiredEnvironmentValidatorTest {
|
||||
@Configuration
|
||||
static class ValidatorConfig {
|
||||
@Bean
|
||||
RequiredEnvironmentValidator requiredEnvironmentValidator(Environment environment) {
|
||||
static RequiredEnvironmentValidator requiredEnvironmentValidator(Environment environment) {
|
||||
return new RequiredEnvironmentValidator();
|
||||
}
|
||||
}
|
||||
|
||||
+64
@@ -10,6 +10,8 @@ import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.FinalEnti
|
||||
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.InverseCollectionParentEntity;
|
||||
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.InverseManyToManyEntity;
|
||||
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.InverseOneToOneEntity;
|
||||
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.JoinColumnChildEntity;
|
||||
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.JoinColumnParentEntity;
|
||||
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.NoDefaultConstructorEntity;
|
||||
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.OrderEntity;
|
||||
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.OwningChildEntity;
|
||||
@@ -19,6 +21,7 @@ import dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity.Unidirect
|
||||
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.web.BadOrderController;
|
||||
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.web.GoodOrderController;
|
||||
import dev.caskeleton.adapter.outbound.persistence.testkit.arch.web.ListReturningController;
|
||||
import dev.caskeleton.adapter.outbound.persistence.testkit.outsider.OrderEntityReader;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
@@ -181,6 +184,27 @@ class JpaArchitectureRulesTest {
|
||||
.hasMessageNotContaining("OwningChildEntity.parent");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("mappedBy 없이 두 엔티티가 서로를 연관으로 가리키면 양쪽 모두 거부된다")
|
||||
void aPairThatPointsBothWaysWithoutMappedByIsRefusedOnBothSides() {
|
||||
// given
|
||||
var classes =
|
||||
new ClassFileImporter()
|
||||
.importClasses(JoinColumnParentEntity.class, JoinColumnChildEntity.class);
|
||||
|
||||
// when
|
||||
Throwable failure =
|
||||
catchThrowable(
|
||||
() -> JpaArchitectureRules.associationsAreMappedFromOneSideOnly().check(classes));
|
||||
|
||||
// then
|
||||
assertThat(failure)
|
||||
.isInstanceOf(AssertionError.class)
|
||||
.hasMessageContaining("JoinColumnParentEntity.children")
|
||||
.hasMessageContaining("JoinColumnChildEntity.parent")
|
||||
.hasMessageContaining("one side only");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("단방향 연관관계는 통과한다")
|
||||
void unidirectionalAssociationsPass() {
|
||||
@@ -195,4 +219,44 @@ class JpaArchitectureRulesTest {
|
||||
assertThatCode(() -> JpaArchitectureRules.associationsAreMappedFromOneSideOnly().check(classes))
|
||||
.doesNotThrowAnyException();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("leaf 밖의 클래스가 그 leaf의 엔티티를 쓰면 거부된다")
|
||||
void aClassOutsideTheLeafThatUsesItsEntityIsRefused() {
|
||||
// given
|
||||
var classes = new ClassFileImporter().importClasses(OrderEntity.class, OrderEntityReader.class);
|
||||
|
||||
// when
|
||||
Throwable failure =
|
||||
catchThrowable(
|
||||
() ->
|
||||
JpaArchitectureRules.entitiesAreUsedOnlyInsideTheirLeaf(
|
||||
"dev.caskeleton.adapter.outbound.persistence.testkit.arch")
|
||||
.check(classes));
|
||||
|
||||
// then
|
||||
assertThat(failure)
|
||||
.isInstanceOf(AssertionError.class)
|
||||
.hasMessageContaining("OrderEntityReader")
|
||||
.hasMessageContaining("OrderEntity");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("같은 leaf 안의 클래스가 그 leaf의 엔티티를 쓰는 것은 통과한다")
|
||||
void aClassInsideTheLeafMayUseItsEntity() {
|
||||
// given
|
||||
var classes =
|
||||
new ClassFileImporter().importClasses(OrderEntity.class, BadOrderController.class);
|
||||
|
||||
// when
|
||||
Throwable failure =
|
||||
catchThrowable(
|
||||
() ->
|
||||
JpaArchitectureRules.entitiesAreUsedOnlyInsideTheirLeaf(
|
||||
"dev.caskeleton.adapter.outbound.persistence.testkit.arch")
|
||||
.check(classes));
|
||||
|
||||
// then
|
||||
assertThat(failure).isNull();
|
||||
}
|
||||
}
|
||||
|
||||
+36
@@ -0,0 +1,36 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.testkit.arch.entity;
|
||||
|
||||
import jakarta.persistence.Entity;
|
||||
import jakarta.persistence.FetchType;
|
||||
import jakarta.persistence.Id;
|
||||
import jakarta.persistence.JoinColumn;
|
||||
import jakarta.persistence.ManyToOne;
|
||||
|
||||
/**
|
||||
* Refused by the association rule: the read-only half of {@link JoinColumnParentEntity}'s pair.
|
||||
*
|
||||
* <p>{@code insertable = false, updatable = false} makes this field a second, read-only copy of the
|
||||
* link the parent's collection writes. Within one persistence context it does not follow a change
|
||||
* made through the collection.
|
||||
*/
|
||||
@Entity
|
||||
public class JoinColumnChildEntity {
|
||||
|
||||
@Id private Long id;
|
||||
|
||||
@ManyToOne(fetch = FetchType.LAZY)
|
||||
@JoinColumn(name = "parent_id", insertable = false, updatable = false)
|
||||
private JoinColumnParentEntity parent;
|
||||
|
||||
protected JoinColumnChildEntity() {}
|
||||
|
||||
/** The identifier. */
|
||||
public Long id() {
|
||||
return id;
|
||||
}
|
||||
|
||||
/** The read-only reference back to the parent. */
|
||||
public JoinColumnParentEntity parent() {
|
||||
return parent;
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user