From 45d2a478b800586f1d84151723629a560265d58e Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Wed, 4 Nov 2026 22:46:05 +0900 Subject: [PATCH] =?UTF-8?q?fix!:=20=EB=A6=AC=EB=B7=B0=EA=B0=80=20=EA=B8=B0?= =?UTF-8?q?=EB=A1=9D=EB=A7=8C=20=ED=95=98=EA=B3=A0=20=EB=82=A8=EA=B2=A8=20?= =?UTF-8?q?=EB=91=94=20=EA=B2=B0=ED=95=A8=EC=9D=84=20=EB=AA=A8=EB=91=90=20?= =?UTF-8?q?=EA=B3=A0=EC=B9=9C=EB=8B=A4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 문제: 확인 리뷰들이 찾은 Minor와 "남음" 항목을 고치지 않고 기록만 했다. 그중 동작에 닿는 것이 여럿이었다. verifyStrictLaneStages가 `./gradlew ci || true`, `| tee`, `if ./gradlew …` 같은 실패를 막지 못하는 명령을 실행으로 세는 거짓 통과, Redis 멱등 저장소가 lease가 지난 CLAIMED의 시작을 받아 주던 것(PostgreSQL은 거절), gRPC가 INTERNAL_ERROR 실패를 아무 데도 남기지 않고 STOMP가 SQL 원인까지 로그에 쓰던 것, 잠금 대기 검사기가 Flyway가 트랜잭션 밖에서 돌리는 VACUUM FULL을 몰라 통과시키던 것, 템플릿 패키지 목록과 버전 판정 정규식이 두 곳에 갈라져 있던 것. 방법: 영역별 4묶음이 실패 테스트를 먼저 확인하고 고쳤다(재리뷰 README 16절). - workflow run 스크립트를 bash 문법으로 읽고, 실패를 막지 못하는 명령과 셸이 실행하지 않는 줄의 cd는 이유를 대며 세지 않는다. - 두 멱등 저장소가 lease가 지난 시작을 LEASE_EXPIRED로 거절하고, executor는 아무것도 실행하지 않았으므로 503으로 답한다. - web·GraphQL·gRPC·STOMP가 shared-contract ServerFailureLog로 같은 실패 로그 줄을 쓴다(persistence 실패는 분류만). - @SystemCaller 규칙이 인터페이스로 주입한 경로도 보고, 조립 설정은 사용자 전송으로 세지 않는다. - 갈라진 복사본은 하나로: 템플릿 패키지 표가 정본이고 클래스를 대조하며, 버전 판정 코드는 grpc 빌드가 Sync로 가져와 컴파일한다. - httpclient 장애 주입 lane이 main 단계에서 두 번 돌던 것을 한 번으로. - 게이트에서 드러난 httpclient drain 스레드 경쟁: close()가 스레드를 join하고, 런타임이 닫기를 거부해도 drain 스레드를 먼저 멈춘다. 위험·한계: 목록에 넣지 않은 새 템플릿 패키지는 기능 패키지와 구별할 수 없다(문서화). 서브셸·eval·source 안의 cd 뒤 명령과 `|| exit 1`은 세지 않는다 (거짓 실패 쪽). Gradle 옵션 판독은 Gradle 9.0 동작을 TestKit으로 고정했다. 검증: 첫 ci가 httpclient ClientRuntimeRegistryTest 1건으로 실패(기존 코드의 drain 스레드 경쟁, README 16.1절) — 고친 뒤 ci 재실행 통과. 두 빌드 verifyStrictLaneStages, mainCheck, :integrationCheck, -p optional-platforms ci integrationCheck performanceCheck, :performanceCheck, :releaseCheck, compose 18 lane 통과. httpclient check·testQualityCheck·failure injection·performance lane을 수정 뒤 다시 돌려 통과. Refs: docs/reviews/2026-11-03-rereview/README.md 16절 BREAKING CHANGE: 멱등 저장소 시작·renew 결과에 LEASE_EXPIRED가 생겼고, lease가 지난 시작은 503 IDEMPOTENCY_UNAVAILABLE이다. verifyStrictLaneStages가 실패를 막지 못하는 workflow 명령을 세지 않는다. --- AGENTS.md | 8 +- CLAUDE.md | 4 +- docs/httpclient/configuration-reference.md | 5 +- docs/jpa/migration-guide.md | 41 +- docs/manual/04-persistence.md | 5 +- docs/manual/05-idempotency.md | 34 +- docs/manual/07-authorization.md | 34 +- docs/registries/env-keys.yaml | 10 +- docs/reviews/2026-11-03-rereview/README.md | 37 + docs/runbooks/idempotency-unavailable.md | 61 +- docs/study/postgresql-jpa-walkthrough.ko.md | 6 +- docs/upgrade/2026-11-maturity-review.md | 19 +- docs/web/http-contract.md | 37 +- src/.env.example | 3 +- src/README.md | 5 +- .../graphql/error/GraphQlWireErrorMapper.java | 15 + .../error/GraphQlWireErrorMapperTest.java | 90 ++ src/adapter/inbound/grpc/README.md | 6 + .../GrpcExceptionHandlingInterceptor.java | 53 +- .../GrpcExceptionHandlingInterceptorTest.java | 167 ++- src/adapter/inbound/messaging/CLAUDE.md | 4 +- src/adapter/inbound/messaging/README.md | 4 +- .../web/error/GlobalExceptionHandler.java | 47 +- src/adapter/inbound/websocket/README.md | 11 +- .../SafeStompSubProtocolErrorHandler.java | 9 +- .../stomp/StompHandlerFailureLog.java | 39 + .../websocket/stomp/WebSocketConfig.java | 9 + .../SafeStompSubProtocolErrorHandlerTest.java | 2 +- .../stomp/StompHandlerFailureLogTest.java | 185 +++ .../stomp/WebSocketAutoConfigurationTest.java | 5 + src/adapter/outbound/cache-redis/README.md | 8 +- src/adapter/outbound/cache-redis/build.gradle | 12 +- .../redis/idempotency/IdempotencyScripts.java | 30 +- .../RedisIdempotencyStoreAdapter.java | 2 + .../LiveRedisIdempotencyRunbookTest.java | 203 +++ .../RedisIdempotencyStoreAdapterTest.java | 80 +- src/adapter/outbound/httpclient/README.md | 6 +- src/adapter/outbound/httpclient/build.gradle | 18 +- .../httpclient/jdk/ReadIdleInputStream.java | 69 +- .../profile/ClientRuntimeRegistry.java | 57 +- .../jdk/ReadIdleInputStreamTest.java | 125 +- .../profile/ClientRuntimeRegistryTest.java | 39 +- .../outbound/persistence-jpa/README.md | 12 +- .../IdempotencyTransitionGateway.java | 5 +- .../PostgreSqlIdempotencyStore.java | 11 +- .../PostgreSqlIdempotencyIntegrationTest.java | 127 ++ ...reSqlMigrationLockWaitIntegrationTest.java | 140 ++ .../LockWaitMigrationInspectorTest.java | 132 +- .../VersionedMigrationFilesTest.java | 50 +- .../migration/LockWaitMigrationInspector.java | 263 +++- .../migration/VersionedMigrationFiles.java | 52 + .../InMemorySemanticRedisCommands.java | 9 +- src/app-bootstrap/README.md | 3 + src/app-bootstrap/build.gradle | 4 + src/app-bootstrap/gradle.lockfile | 6 +- .../ApplicationBoundaryArchitectureTest.java | 138 +- .../ArchRuleDiscoveryContractTest.java | 1 + .../ArchitectureViolationFixtureTest.java | 158 +++ .../TemplatePackageOwnershipRuleTest.java | 4 +- .../inbound/web/PingControllerFixture.java | 14 + .../adapter/inbound/web/package-info.java | 6 + ...erBesideControllerRegistrationFixture.java | 21 + ...CallerBesideWebMvcRegistrationFixture.java | 21 + .../allowed/configuration/package-info.java | 9 + .../GraphQlJavaSystemCallerFixture.java | 23 + .../IndirectlyRunnableSystemJobFixture.java | 19 + .../InterfaceExposedSystemJobFixture.java | 19 + .../inbound/scheduler/RunNowJobFixture.java | 11 + .../scheduler/RunnableSystemJobFixture.java | 21 + .../scheduler/ScheduledWorkFixture.java | 7 + ...JobInterfaceInvokingControllerFixture.java | 26 + .../TemplatePackageOwnership.java | 99 +- .../TemplatePackageOwnershipDocumentTest.java | 133 ++ src/application-core/CLAUDE.md | 26 +- src/application-core/README.md | 8 +- .../idempotency/IdempotencyExecutor.java | 11 +- .../idempotency/IdempotencyRenewOutcome.java | 27 +- .../idempotency/IdempotencyStartOutcome.java | 20 + .../idempotency/IdempotencyStorePort.java | 14 +- .../IdempotencyUnavailableException.java | 9 +- .../application/security/SystemCaller.java | 16 +- .../application/security/SystemInvoked.java | 9 +- .../idempotency/IdempotencyExecutorTest.java | 37 + .../testkit/IdempotencyStoreContract.java | 166 ++- src/build-logic/README.md | 2 +- .../buildlogic/strictlane/ShellScript.java | 1213 +++++++++++++++++ .../strictlane/StrictLaneStagesPlugin.java | 33 +- .../strictlane/WorkflowGradleInvocations.java | 522 +++---- .../StrictLaneStagesFunctionalTest.java | 111 ++ .../WorkflowGradleInvocationsTest.java | 376 +++++ src/composition/idempotency/README.md | 4 +- .../config/module-defaults/idempotency.yml | 6 +- .../PermissionEnforcementGuard.java | 3 +- .../grpc-operation-ledger-jpa/build.gradle | 16 + ...rpcOperationLedgerMigrationLockWaitIT.java | 9 +- .../ledger/GrpcOperationLedgerMigrations.java | 27 +- src/shared-contract/README.md | 11 + .../shared/error/ServerFailureLog.java | 102 ++ .../shared/error/ServerFailureLogTest.java | 129 ++ 99 files changed, 5419 insertions(+), 636 deletions(-) create mode 100644 src/adapter/inbound/websocket/src/main/java/dev/caskeleton/adapter/inbound/websocket/stomp/StompHandlerFailureLog.java create mode 100644 src/adapter/inbound/websocket/src/test/java/dev/caskeleton/adapter/inbound/websocket/stomp/StompHandlerFailureLogTest.java create mode 100644 src/adapter/outbound/cache-redis/src/test/java/dev/caskeleton/adapter/outbound/cache/redis/idempotency/LiveRedisIdempotencyRunbookTest.java create mode 100644 src/app-bootstrap/src/architectureTest/java/dev/caskeleton/bootstrap/architecture/allowed/adapter/inbound/web/PingControllerFixture.java create mode 100644 src/app-bootstrap/src/architectureTest/java/dev/caskeleton/bootstrap/architecture/allowed/adapter/inbound/web/package-info.java create mode 100644 src/app-bootstrap/src/architectureTest/java/dev/caskeleton/bootstrap/architecture/allowed/configuration/SystemCallerBesideControllerRegistrationFixture.java create mode 100644 src/app-bootstrap/src/architectureTest/java/dev/caskeleton/bootstrap/architecture/allowed/configuration/SystemCallerBesideWebMvcRegistrationFixture.java create mode 100644 src/app-bootstrap/src/architectureTest/java/dev/caskeleton/bootstrap/architecture/allowed/configuration/package-info.java create mode 100644 src/app-bootstrap/src/architectureTest/java/dev/caskeleton/bootstrap/architecture/violations/adapter/inbound/messaging/graphqljava/GraphQlJavaSystemCallerFixture.java create mode 100644 src/app-bootstrap/src/architectureTest/java/dev/caskeleton/bootstrap/architecture/violations/adapter/inbound/scheduler/IndirectlyRunnableSystemJobFixture.java create mode 100644 src/app-bootstrap/src/architectureTest/java/dev/caskeleton/bootstrap/architecture/violations/adapter/inbound/scheduler/InterfaceExposedSystemJobFixture.java create mode 100644 src/app-bootstrap/src/architectureTest/java/dev/caskeleton/bootstrap/architecture/violations/adapter/inbound/scheduler/RunNowJobFixture.java create mode 100644 src/app-bootstrap/src/architectureTest/java/dev/caskeleton/bootstrap/architecture/violations/adapter/inbound/scheduler/RunnableSystemJobFixture.java create mode 100644 src/app-bootstrap/src/architectureTest/java/dev/caskeleton/bootstrap/architecture/violations/adapter/inbound/scheduler/ScheduledWorkFixture.java create mode 100644 src/app-bootstrap/src/architectureTest/java/dev/caskeleton/bootstrap/architecture/violations/adapter/inbound/web/controller/SystemJobInterfaceInvokingControllerFixture.java rename src/app-bootstrap/src/{architectureTest => test}/java/dev/caskeleton/bootstrap/architecture/TemplatePackageOwnership.java (51%) create mode 100644 src/app-bootstrap/src/test/java/dev/caskeleton/bootstrap/architecture/TemplatePackageOwnershipDocumentTest.java create mode 100644 src/build-logic/src/main/java/dev/caskeleton/buildlogic/strictlane/ShellScript.java create mode 100644 src/shared-contract/src/main/java/dev/caskeleton/shared/error/ServerFailureLog.java create mode 100644 src/shared-contract/src/test/java/dev/caskeleton/shared/error/ServerFailureLogTest.java diff --git a/AGENTS.md b/AGENTS.md index 85ddbe81..c4091257 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -84,9 +84,11 @@ root `CLAUDE.md`는 이 목록의 동기화된 요약이다. 두 문서가 어 줄 때문에 옵션을 별개 명령으로 떼어 내지 않는지 검사한다. 두 빌드가 각자 검사를 돌린다. `-x`로 뺀 lane(Gradle처럼 축약 이름도 맞춰 본다), `continue-on-error` 아래 명령, 상태 함수(`always()`·`success()`· `!cancelled()`)가 아닌 `if:` 아래 명령, `workflow_dispatch`로만 시작하는 workflow의 명령은 실행으로 세지 - 않는다. 명령이 어느 빌드를 실행하는지는 `working-directory`에서 시작해 같은 셸이 그 앞에서 확실히 - 실행한 `cd <상대 경로>`를 따라간 뒤 `-p`를 합친 디렉터리로 정한다. 따라갈 수 없는 `cd`(`cd "$VAR"`· - `cd -`·`pushd`·조건부·서브셸 등) 뒤의 명령은 추측하지 않고 세지 않는다. + 않는다. 실패해도 단계를 실패시키지 못하는 명령(`|| true`, pipefail 없는 파이프 왼쪽, `!`, `&`, + `if`·`while` 조건과 복합 명령·함수·명령 치환 안, `set +e` 뒤, 스크립트 마지막이 아닌 `&&` 목록)과 + Gradle이 거부하는 명령줄(`-mx`)도 세지 않는다. 명령이 어느 빌드를 실행하는지는 `working-directory`에서 + 시작해 같은 셸이 그 앞에서 확실히 실행한 `cd <상대 경로>`를 따라간 뒤 `-p`를 합친 디렉터리로 정한다. + 따라갈 수 없는 `cd`(`cd "$VAR"`·`cd -`·`pushd`·조건부·서브셸 등) 뒤의 명령은 추측하지 않고 세지 않는다. 작업 파일의 소유 leaf는 registry의 `source_path`로 판단하고 가장 가까운 `src/**/CLAUDE.md`를 함께 읽는다. focused test는 registry의 `gradle_path`에서 diff --git a/CLAUDE.md b/CLAUDE.md index 86008ebb..ae4a2ca0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -41,7 +41,9 @@ If this summary drifts from `AGENTS.md`, `AGENTS.md` wins and this summary must request; the main build and, as `-p optional-platforms`, the optional build each run their own) fails when a non-`explicitOnly` strict lane is in no stage a workflow executes — a lane the command removes with `-x` (abbreviations matched as Gradle does), a command under - `continue-on-error` or a non-status `if:`, and a `workflow_dispatch`-only workflow do not count; + `continue-on-error` or a non-status `if:`, a command whose failure cannot fail its step + (`|| true`, a pipe without pipefail, an `if` condition, …), a command line Gradle rejects, and a + `workflow_dispatch`-only workflow do not count; the build a command runs is its `working-directory`, then the `cd ` the same shell surely ran before it, joined with `-p` (a `cd` it cannot follow is not guessed: the command does not count) — or a folded workflow `run:` block splits an option off its command. diff --git a/docs/httpclient/configuration-reference.md b/docs/httpclient/configuration-reference.md index c694fa9b..d23ff7c0 100644 --- a/docs/httpclient/configuration-reference.md +++ b/docs/httpclient/configuration-reference.md @@ -107,7 +107,10 @@ transports. The JDK transport keeps its exchange-wide timeout off a streamed bod watchdog bounds each read's wait by `read-idle`, because the JDK client has no socket read timeout. The watchdog wakes on one daemon thread of its own (`httpclient-jdk-read-idle`), not on the common `ForkJoinPool`, so a service that fills that pool with blocking work does not delay it (re-review -2026-11-04 RR3-03-01). Before review 2026-11-03 RR-04-03 every JDK download longer than +2026-11-04 RR3-03-01). That thread ends 60 s after the last wake it had queued and the next stream +starts it again; it inherits no thread-locals from the request thread that starts it and uses the +module's class loader as its context loader, so an idle service keeps no thread and a redeployed +application's loader is not pinned (RR4-03-04). Before review 2026-11-03 RR-04-03 every JDK download longer than `response-header` was cut mid-body. A body failure after hand-over is thrown from `read` as `HttpBodyReadException` — an `IOException`, as `InputStream` promises, so `catch (IOException)` still sees it — whose `failure()` is `HttpResponseTimeoutException` (a wait longer than `read-idle`) or `HttpResponseTruncatedException` diff --git a/docs/jpa/migration-guide.md b/docs/jpa/migration-guide.md index 04de288c..77374b6f 100644 --- a/docs/jpa/migration-guide.md +++ b/docs/jpa/migration-guide.md @@ -52,9 +52,26 @@ goes into a migration that runs in one. A statement on a table the same migratio earlier waits for nothing and needs no bound; `CREATE TABLE IF NOT EXISTS` does not count, because the table may already be there. +Flyway also takes a migration out of its transaction on its own, without a `.sql.conf`: on +PostgreSQL 12 and later, any statement that starts with `VACUUM`, `CREATE [UNIQUE] INDEX +CONCURRENTLY`, `DROP INDEX CONCURRENTLY`, `REINDEX [VERBOSE] SCHEMA|DATABASE|SYSTEM`, `CREATE` or +`DROP DATABASE|TABLESPACE|SUBSCRIPTION`, `ALTER SYSTEM` or `DISCARD ALL` (Flyway 11, +`PostgreSQLParser`; words inside parentheses are skipped, so `VACUUM (FULL)` and `REINDEX (VERBOSE) +SCHEMA` count). One such statement puts the whole script outside a transaction. Next to any other +statement — a `SET LOCAL lock_timeout` included — Flyway refuses the migration at deploy under its +default `mixed=false`; a service that sets `mixed=true` gets it run, with the bound silently gone. +Put such a statement in a migration of its own. `REINDEX TABLE|INDEX CONCURRENTLY` is not on +Flyway's list: Flyway runs it in the transaction, PostgreSQL refuses it there, and it needs +`executeInTransaction=false` (below). A locking statement that cannot run in a transaction at all — +`VACUUM FULL`, a non-concurrent `REINDEX SCHEMA|DATABASE|SYSTEM` — can never be bounded this way and +does not belong in a migration: run it by hand as an operations step, outside a deploy, in a psql +session that sets `SET lock_timeout = '5s';` first (the session ends with the step, so the plain +`SET` is the bound there). + Enforced, not remembered: `LockWaitMigrationInspector` (persistence-jpa testkit) reads a script — -and the `.sql.conf` beside it — and names the statements from its list that lock an existing table -with no bound in force: `ALTER TABLE`, `LOCK TABLE`, `TRUNCATE`, `DROP TABLE`; `CREATE INDEX`, +and the `.sql.conf` beside it, as Flyway reads it (a properties file, `false` in any case), and +otherwise the statements Flyway takes out of the transaction on its own — and names the statements +from its list that lock an existing table with no bound in force: `ALTER TABLE`, `LOCK TABLE`, `TRUNCATE`, `DROP TABLE`; `CREATE INDEX`, `DROP INDEX`, `REINDEX` and `REFRESH MATERIALIZED VIEW` without `CONCURRENTLY`, `CLUSTER`, `VACUUM FULL`; `REFERENCES` on the table it references, in a new table as much as in an `ALTER TABLE` (a foreign key locks its parent against writes while it is added); `CREATE TRIGGER` and `DROP TRIGGER` @@ -67,16 +84,24 @@ stay that way (they are applied, and applied files are never edited); the test l refuses any new one. `PostgreSqlMigrationLockWaitIntegrationTest` (`postgresqlMigrationLockWaitIntegrationTest`) shows on real PostgreSQL that a bounded migration gives up within the bound while another transaction holds the table, and leaves the version below -it. The gRPC ledger's tree lives in the optional build, which cannot reach the testkit: its -`GrpcOperationLedgerMigrationLockWaitIT` runs every ledger migration after V001, read from the tree, -against a held ledger instead, so its next migration is checked on real PostgreSQL. The rule came -from V21 having the bound and V22, in the same commit, not having it (re-review 2026-11-03 round 2, -RR-02-09; round 3, RR3-02-01, RR3-02-03, RR3-02-04). +it; it also holds the inspector's copy of Flyway's list to Flyway itself (the statements it names +are exactly those Flyway refuses next to a `SET LOCAL`), and shows a `VACUUM FULL` run under +`mixed=true` waiting out a reader despite its `SET LOCAL`. The gRPC ledger's tree lives in the +optional build, which cannot depend on the testkit: its `GrpcOperationLedgerMigrationLockWaitIT` +runs every ledger migration after V001, read from the tree, against a held ledger instead, so its +next migration is checked on real PostgreSQL. It finds and orders that tree with the testkit's own +`VersionedMigrationFiles`, which the ledger's `build.gradle` copies into its integration-test sources +at build time (`sharedMigrationFileSources`), so both builds read one definition. The rule came from +V21 having the bound and V22, in the same commit, not having it (re-review 2026-11-03 round 2, +RR-02-09; round 3, RR3-02-01, RR3-02-03, RR3-02-04; round 4, RR4-02-01). ## Concurrent index builds `CREATE INDEX CONCURRENTLY` cannot run inside a transaction block, and Flyway wraps migrations in -one by default. The migration therefore needs a companion configuration: +one by default. Flyway recognizes `CREATE`/`DROP INDEX CONCURRENTLY` and runs such a script outside +the transaction on its own (above), but not `REINDEX … CONCURRENTLY`; write the companion +configuration for every concurrent statement anyway, so the file says what it needs and no reader +has to know Flyway's list: ```conf # V42__order_index.sql.conf diff --git a/docs/manual/04-persistence.md b/docs/manual/04-persistence.md index 663319ce..fbc35f6e 100644 --- a/docs/manual/04-persistence.md +++ b/docs/manual/04-persistence.md @@ -131,7 +131,10 @@ capability 스트림 db/migration/jpa/ 각자 histor - 운영 중인 테이블을 잠그는 문장(`ALTER TABLE`, `CONCURRENTLY` 없는 `CREATE INDEX`, `UPDATE`·`DELETE`, 기존 테이블을 가리키는 `REFERENCES`, trigger 만들기·지우기 등 — 전체 목록은 `docs/jpa/migration-guide.md`)이 있으면 그 앞에 `SET LOCAL lock_timeout = '5s';`를 둔다(`V8_1`, `V22`). 값은 0이 아니어야 하고, - `executeInTransaction=false` 마이그레이션에서는 `SET LOCAL`이 효과가 없으니 잠그는 문장을 거기 두지 않는다. 상한이 없으면 그 테이블을 건드린 열린 + `executeInTransaction=false` 마이그레이션에서는 `SET LOCAL`이 효과가 없으니 잠그는 문장을 거기 두지 않는다. Flyway가 + 스스로 트랜잭션 밖에서 돌리는 문장(`VACUUM`, `CREATE INDEX CONCURRENTLY` 등 — 목록은 같은 가이드)이 하나라도 있는 파일도 + 트랜잭션 밖에서 돌므로, 그런 문장은 따로 한 파일에 둔다. `VACUUM FULL`처럼 트랜잭션 안에서 돌 수 없는 잠금 문장은 + 상한을 걸 수 없으니 마이그레이션에 두지 않고 운영 절차로 돌린다. 상한이 없으면 그 테이블을 건드린 열린 트랜잭션 하나(`ALTER TABLE`이면 읽기만 한 것도) 뒤에서 마이그레이션이 끝없이 기다리고, 그동안 뒤에 온 그 테이블의 쓰기(`ALTER TABLE`이면 읽기까지)가 모두 마이그레이션 뒤에 줄을 선다. 상한이 있으면 5초 뒤 실패·롤백되고 새 버전이 뜨지 않을 뿐이다. 자기 스트림에도 `SampleMigrationLockWaitTest`처럼 `LockWaitMigrationInspector`(persistence-jpa testkit) 검사를 둔다 diff --git a/docs/manual/05-idempotency.md b/docs/manual/05-idempotency.md index 66e4c6e4..2442a1d1 100644 --- a/docs/manual/05-idempotency.md +++ b/docs/manual/05-idempotency.md @@ -49,21 +49,35 @@ DB 커밋 중 연결 끊김 ───────────────── | --- | --- | --- | | 409 `IDEMPOTENT_IN_FLIGHT`, `retryable=true`, `Retry-After: 2` | 같은 키가 아직 실행 중. 결과 불명 | **같은 키로** 기다렸다 재전송. 새 키 금지 | | 422 `IDEMPOTENT_REQUEST_MISMATCH` | 같은 키에 다른 본문 | 클라이언트 버그. 재전송 안 함 | -| 503 `IDEMPOTENCY_UNAVAILABLE` | 실행 전 거부 | 같은 키로 나중에 | +| 503 `IDEMPOTENCY_UNAVAILABLE` | 실행 전 거부(저장소 응답 없음, 또는 claim의 lease가 시작 전에 끝남) | 같은 키로 나중에 | | 503 `IDEMPOTENCY_COMPLETION_UNKNOWN`, `retryable=true` | 실행 뒤 결과 기록 실패. 적용됐을 수 있음 | "적용 안 됨"이라 하지 않는다. **같은 키로** 나중에 재전송, 새 키 금지 | | 500 `IDEMPOTENCY_RECOVERY_REQUIRED` | 이전 시도 효과를 증명 못함 | 자동 재시도 안 함. 같은 키 재전송은 안전(재생·실행·다시 500을 서버가 판정, 다시 돌리지 않음). 업무 데이터로 확인하고, 효과가 없다고 확인된 뒤에만 운영 절차에 따라 새 키 | | 500 `DB_COMPLETION_UNKNOWN`, `retryable=false` | DB 커밋 중 연결 끊김. 커밋됐을 수도 있음 | 키 없이 재시도 금지. KEYED면 같은 키로 재전송하면 서버가 판정 | -같은 키 재전송은 작업을 다시 돌리지 않는다. 첫 시도가 실행을 시작한 뒤 processing lease(기본 30s)가 -지나면 두 저장소 모두 그 기록을 넘겨주지 않고 500 `IDEMPOTENCY_RECOVERY_REQUIRED`로 답한다. 넘겨받는 것은 -실행 전 claim과 재시도 가능한 실패뿐이다. 이 답은 기록을 바꾸지 않는다. executor는 lease를 늘리지 않아 -30s를 넘는 정상 요청도 lease가 지나는데, 그 첫 시도가 나중에 끝나면 결과가 기록되고 첫 요청은 정상 응답을 -받으며 그 뒤 같은 키 요청은 그 결과를 재생한다(2026-11-04 소유자 결정). 기록은 lease + `replay-ttl` 동안 -남고, `replay-ttl`이 lease보다 짧으면 기동이 실패한다(`IdempotencyStoreContract`, 재리뷰 2026-11-04 -RR-01-11·RR-03-11). 작업이 실패로 끝나 executor가 포기나 재시도 가능으로 표시한 기록은 표시한 때부터 -`failure-retention`(기본 24h) 동안 남는다. `failure-retention`이 `replay-ttl`보다 짧으면 기동이 실패한다. +같은 키 재전송은 작업을 다시 돌리지 않는다. 다만 첫 시도가 processing lease(기본 30s)보다 오래 걸릴 때 +보이는 모습은 저장소마다 다르다(재리뷰 2026-11-04 RR5-01-05, 정본 `docs/web/http-contract.md` 2절). + +- **Redis**: 첫 시도가 실행을 시작한 뒤 lease가 지나면 그 기록을 넘겨주지 않고 500 + `IDEMPOTENCY_RECOVERY_REQUIRED`로 답한다. 넘겨받는 것은 실행 전 claim과 재시도 가능한 실패뿐이다. 이 답은 + 기록을 바꾸지 않는다. executor는 lease를 늘리지 않아 30s를 넘는 정상 요청도 lease가 지나는데, 그 첫 + 시도가 나중에 끝나면 결과가 기록되고 첫 요청은 정상 응답을 받으며 그 뒤 같은 키 요청은 그 결과를 + 재생한다(2026-11-04 소유자 결정). +- **PostgreSQL**: executor를 유스케이스의 쓰기 트랜잭션 하나 안에서 부르면(템플릿의 모양) claim 기록은 그 + 트랜잭션과 함께 커밋되거나 롤백된다. 같은 키 요청은 409·500 대신 첫 요청이 끝날 때까지 기다렸다가 + 재생(커밋)이나 새 실행(롤백)을 받는다. 작업이 예외로 끝나면 실패 기록도 함께 롤백되어 다음 같은 키 + 요청이 다시 실행한다. 쓰기 트랜잭션 제한(기본 30s)을 넘긴 요청도 기록과 함께 롤백된다 + (`DB_QUERY_CANCELED`). 그래서 이 저장소가 막는 중복은 같은 트랜잭션 안의 DB 효과다. 트랜잭션 밖 효과는 + 다음 요청에서 다시 일어날 수 있다. + +claim의 lease가 작업 시작 전에 끝나면 두 저장소 모두 시작을 거절하고(`LEASE_EXPIRED`), executor는 작업을 +돌리지 않은 채 503 `IDEMPOTENCY_UNAVAILABLE`로 답한다. 다음 같은 키 요청이 그 claim을 넘겨받아 실행한다. +기록은 lease + `replay-ttl` 동안 남고, `replay-ttl`이 lease보다 짧으면 기동이 실패한다 +(`IdempotencyStoreContract`, 재리뷰 2026-11-04 RR-01-11·RR-03-11). 작업이 실패로 끝나 executor가 포기나 +재시도 가능으로 표시한 기록은 표시한 때부터 `failure-retention`(기본 24h) 동안 남는다(PostgreSQL을 트랜잭션 +하나로 쓰면 위처럼 롤백과 함께 사라진다). `failure-retention`이 `replay-ttl`보다 짧으면 기동이 실패한다. 짧으면 효과 불명으로 포기한 기록이 먼저 사라져 같은 키 재전송이 작업을 다시 돌리기 때문이다(RR3-03-02, -2026-11-04 소유자 결정). +2026-11-04 소유자 결정). 거꾸로 말하면 `replay-ttl`(`APP_IDEMPOTENCY_TTL`)은 lease 이상, `failure-retention` +이하로 둔다. 409를 `retryable=true`로 바꾼 이유(리뷰 02-01 결정): 같은 키 재전송은 서버가 안전하게 판정하므로 다시 보내는 것이 맞고, 새 키로 보내면 서버가 막지 못한다. 짝 프론트 분류기도 이 표를 따른다. diff --git a/docs/manual/07-authorization.md b/docs/manual/07-authorization.md index 0c1104fd..4d945532 100644 --- a/docs/manual/07-authorization.md +++ b/docs/manual/07-authorization.md @@ -72,7 +72,8 @@ JWT의 `realm_access.roles`와 **자기 client**(`app.security.audience`)의 `re ```text 사용자 요청 ─▶ web·GraphQL·gRPC·WebSocket ──✗──▶ @SystemInvoked 유스케이스 (빌드 실패) -사용자 요청 ─▶ web·GraphQL·gRPC·WebSocket ──✗──▶ @SystemCaller (빌드 실패, 중간 클래스를 거쳐도) +사용자 요청 ─▶ web·GraphQL·gRPC·WebSocket ──✗──▶ @SystemCaller (빌드 실패, 중간 클래스·작업의 인터페이스를 거쳐도. + 타입을 거치지 않는 길은 아래 「규칙이 못 보는 길」) 시스템 ─▶ 스케줄러(조립 모듈이 new로 넘김) ─────▶ @SystemInvoked 유스케이스 시스템 ─▶ 메시지 handler(@SystemCaller, 직접 new) ─▶ @SystemInvoked 유스케이스 시스템 ─▶ 예약 작업(@SystemCaller, 직접 new) ──────▶ @SystemInvoked 유스케이스 @@ -89,7 +90,7 @@ JWT의 `realm_access.roles`와 **자기 client**(`app.security.audience`)의 `re | --- | --- | --- | | 조립 모듈의 스케줄러 | `composition:*` leaf. 예: outbox relay(`src/composition/outbox-relay`) | 조립 모듈의 `@Bean` 메서드가 `new`로 만들어 스케줄러에 넘긴다 | | 메시지 handler | `adapter:inbound:messaging` leaf, 또는 서비스 leaf의 `..adapter.inbound.messaging..` 패키지 | handler 생성자가 port를 받아 `new`로 만든다. handler에 `@SystemCaller`. 작업 순서는 09장 「소비」 4단계 | -| 예약 작업 | 서비스 자기 leaf의 `..adapter.inbound.scheduler..` 패키지. 등록된 scheduler leaf는 없다 | 작업 클래스 생성자가 port를 받아 `new`로 만든다. 작업 클래스에 `@SystemCaller`. 모양은 메시지 handler와 같고, 깨우는 것이 `@Scheduled` 메서드다 | +| 예약 작업 | 서비스 자기 leaf의 `..adapter.inbound.scheduler..` 패키지. 등록된 scheduler leaf는 없다 | 작업 클래스 생성자가 port를 받아 `new`로 만든다. 작업 클래스에 `@SystemCaller`. 모양은 메시지 handler와 같고, 깨우는 것이 `@Scheduled` 메서드다. `Runnable`·`ApplicationRunner` 같은 프로젝트 밖 타입을 구현하지 않는다. task scheduler에 직접 등록하면 그 자리에서 `job::run`을 넘긴다 | 예약 작업은 controller와 같은 leaf에 놓이기 쉽다. 그때 "지금 실행" endpoint가 작업 bean을 주입받아 부르면 권한 검사 없는 쓰기를 사용자가 부르게 된다. 아래 `USER_TRANSPORTS_DO_NOT_REACH_SYSTEM_CALLERS`가 이것을 @@ -101,7 +102,8 @@ JWT의 `realm_access.roles`와 **자기 client**(`app.security.audience`)의 `re | bean 금지 | `@SystemInvoked` 유스케이스가 bean이면 기동 실패. 호출자가 `new`로 만들어 혼자 들고 있는 것만 허용한다 | `src/composition/service-runtime/src/main/java/dev/caskeleton/composition/runtime/authorization/PermissionEnforcementGuard.java` | | 시스템 호출자만 | inbound adapter가 그 클래스에 의존하면 빌드 실패. `@SystemCaller(reason = "...")`를 단 클래스와 그 안에 중첩된 클래스만 예외 | `ONLY_SYSTEM_CALLERS_REACH_SYSTEM_INVOKED_USE_CASES`(`ApplicationBoundaryArchitectureTest`) | | 표시 검사 | `@SystemCaller` 클래스와 그 안에 중첩된 클래스는 `..adapter.inbound.messaging..`·`..adapter.inbound.scheduler..`에만 있고, 사용자 전송 타입(controller·JAX-RS·GraphQL·STOMP mapping·gRPC·servlet)에 의존하지 않는다 | `SYSTEM_CALLERS_ARE_SYSTEM_DRIVEN_INBOUND_ADAPTERS`(같은 파일) | -| 전송 → 시스템 호출자 금지 | 사용자 전송 클래스가 `@SystemCaller`에 의존하면 빌드 실패. 중간 클래스를 거쳐도 잡는다. 사용자 전송 클래스는 사용자 전송 패키지에 있거나 사용자 전송 타입에 의존하는 클래스다. 그래서 handler를 bean으로 등록하는 `@Configuration`은 걸리지 않는다 | `USER_TRANSPORTS_DO_NOT_REACH_SYSTEM_CALLERS`(같은 파일) | +| 전송 → 시스템 호출자 금지 | 사용자 전송 클래스가 `@SystemCaller`나, 그 클래스가 구현·상속하는 프로젝트 타입(작업의 인터페이스 등)에 의존하면 빌드 실패. 중간 클래스를 거쳐도 잡는다. 사용자 전송 클래스는 사용자 전송 패키지에 있는 클래스, 또는 사용자 전송 타입에 의존하는 클래스 가운데 설정(`@Configuration`·`@AutoConfiguration`)이 아닌 것이다. 그래서 handler를 bean으로 등록하는 설정은 같은 클래스에서 `WebMvcConfigurer`·interceptor·controller bean을 함께 만들어도 걸리지 않는다. 설정이 만드는 controller는 controller 자신이 검사받는다 | `USER_TRANSPORTS_DO_NOT_REACH_SYSTEM_CALLERS`(같은 파일) | +| 프로젝트 밖 상위 타입 금지 | `@SystemCaller` 클래스와 그 안에 중첩된 클래스(익명·지역 클래스 제외)는 프로젝트(`dev.caskeleton..`) 밖의 클래스·인터페이스를 상속·구현하지 않는다. `Runnable`은 누구나 의존하는 타입이라, endpoint가 작업을 `Runnable`로 주입받아도 위 규칙이 다른 사용과 구별하지 못하기 때문이다. 메시지 handler의 `MessageHandler`는 프로젝트 타입이라 된다 | `SYSTEM_CALLERS_HAVE_ONLY_PROJECT_SUPERTYPES`(같은 파일) | 규칙은 "누가 닿으면 안 되나"가 아니라 "누가 닿아도 되나"를 적는다. 이 성질은 **패키지** 쪽에서만 온다. 패키지는 허용 목록이라, 나중에 새 전송 패키지가 생겨도 따로 고치지 않아도 막힌다. 사용자 전송 **타입** @@ -109,13 +111,30 @@ JWT의 `realm_access.roles`와 **자기 client**(`app.security.audience`)의 `re 목록이라, 목록에 없는 전송 라이브러리로 만든 endpoint는 전송으로 보이지 않는다. 새 전송 라이브러리를 들이면 그 패키지를 이 목록에 더한다. -이 규칙들은 클래스 의존만 본다. 실행 중에 bean을 찾는 코드(`getBean`, `ObjectProvider`)는 의존이 아니라서 -잡지 못한다. - D4 실패(`MUTATING_USE_CASES_DECLARE_REQUIRED_PERMISSION`)를 끄려고 이 표시를 쓰지 않는다. 사용자가 부르는 쓰기라면 권한을 선언하고 역할에 매핑한다. 같은 쓰기를 사용자도 시스템도 부르면 유스케이스를 둘로 나눈다 (사용자용 `@RequiresPermission`, 시스템용 `@SystemInvoked`). 업무 규칙은 domain에 두고 둘이 같이 쓴다. +### 규칙이 못 보는 길 + +이 규칙들은 클래스 의존만 본다. 그래서 타입 이름이 코드에 남는 길은 모두 잡는다. `ObjectProvider<작업 클래스>`, +`getBean(작업 클래스.class)`, 작업의 인터페이스 주입이 여기에 든다. 타입 이름이 남지 않는 길은 못 본다. + +- 이름이나 `Object`로 bean을 찾는 코드: `getBean("nightlyJob")`, `Map` 주입 뒤 reflection 호출. +- 설정이 사용자 전송에 작업을 감싼 함수를 넘기는 코드: `new RunNowController(job::run)`처럼 controller는 + `Runnable`만 받는다. 의존은 설정 → 작업 쪽에만 있고, 설정은 사용자 전송이 아니다. + +이 길을 막는 것은 ArchUnit 규칙이 아니라 다음 셋이다. + +- `@SystemInvoked` 유스케이스는 bean이 아니고(`PermissionEnforcementGuard`), 시스템 호출자가 `new`로 만들어 + 혼자 든다. 그래서 사용자 전송은 시스템 호출자 객체를 거치지 않고는 그 유스케이스를 부를 수 없다. +- 의존 방향 게이트(`src/config/architecture/modules.json`): 등록된 leaf 가운데 `adapter:inbound:messaging`에 + 의존할 수 있는 것은 조립 루트(`app-bootstrap`)뿐이다. 다른 leaf의 사용자 전송은 handler 타입을 컴파일조차 + 못 한다. 예약 작업은 서비스 자기 leaf에 있어서 같은 leaf의 controller와 사이에는 이 게이트가 없다. 그래서 + 위 두 길은 사실상 controller와 작업이 같은 leaf에 있을 때, 또는 둘을 모두 보는 설정에서 생긴다. +- 리뷰: 설정에서 시스템 호출자를 함수로 감싸 다른 bean에 넘기거나, 사용자 전송이 bean을 이름으로 찾는 코드는 + 리뷰에서 거절한다. 사용자도 같은 일을 시켜야 하면 위 문단대로 사용자용 유스케이스를 따로 만든다. + ## web 없는 조립 web이 빠진 조립(gRPC 전용·배치 전용)에는 `@RequiresPermission`을 검사할 advisor가 없다. 그런 조립에 @@ -127,6 +146,7 @@ web이 빠진 조립(gRPC 전용·배치 전용)에는 `@RequiresPermission`을 - 쓰기 = `@RequiresPermission` + non-final + bean + 역할 매핑. - 시스템만 부르는 쓰기 = `@SystemInvoked` + bean 아님 + 사용자 전송에서 안 닿음. 부르는 쪽은 셋뿐이다: 조립 모듈의 스케줄러, `@SystemCaller` 메시지 handler, `@SystemCaller` 예약 작업. 사용자 전송은 - `@SystemCaller`에도 닿지 못한다. + `@SystemCaller`에도, 그 작업의 인터페이스에도 닿지 못한다. 타입 이름이 남지 않는 길(이름으로 찾기, + 설정이 넘기는 함수)은 빌드가 못 보니 리뷰가 막는다. - 객체 판정은 유스케이스 안, 남의 것은 404. - 역할은 자기 client 것만 읽는다. diff --git a/docs/registries/env-keys.yaml b/docs/registries/env-keys.yaml index e1f99361..954205da 100644 --- a/docs/registries/env-keys.yaml +++ b/docs/registries/env-keys.yaml @@ -1601,9 +1601,11 @@ env_keys: - name: APP_IDEMPOTENCY_TTL # app.capabilities.idempotency.replay-ttl: how long a completed response stays - # replayable. One value for every provider; IdempotencySettings refuses more than 72h, and less - # than APP_IDEMPOTENCY_PROCESSING_LEASE, at startup (re-review 2026-11-04 RR-03-11). Each record - # lives for the processing lease plus this value in both stores. + # replayable. One value for every provider; IdempotencySettings refuses more than 72h, less than + # APP_IDEMPOTENCY_PROCESSING_LEASE (re-review 2026-11-04 RR-03-11), and more than + # APP_IDEMPOTENCY_FAILURE_RETENTION (RR3-03-02, owner decision 2026-11-04), at startup: raising + # this key alone past the failure retention (default 24h) fails startup, so raise both + # (RR5-03-01). Each record lives for the processing lease plus this value in both stores. type: duration default: 24h allowed_values: null @@ -1611,7 +1613,7 @@ env_keys: required: false reload_policy: restart-only owner_branch: feature-rate-limit-idempotency-contract - validation: spring_duration_shorthand_le_72h_ge_processing_lease + validation: spring_duration_shorthand_le_72h_ge_processing_lease_le_failure_retention compatibility_impact: behavior-change required_test: idempotency-contract:ttl-applied diff --git a/docs/reviews/2026-11-03-rereview/README.md b/docs/reviews/2026-11-03-rereview/README.md index 52e6fb91..f5246a85 100644 --- a/docs/reviews/2026-11-03-rereview/README.md +++ b/docs/reviews/2026-11-03-rereview/README.md @@ -324,3 +324,40 @@ PR #1을 만든 뒤 사용자가 묶음이 기본값으로 정한 세 가지를 리뷰어 1명([round5/review.md](round5/review.md)): Critical 0, **Important 0**, 회귀 0, Minor 9(7건은 이 커밋, 2건은 이전부터). 결정 ①②③이 코드·공유 계약·문서에 결정대로 들어갔고, lease가 지난 `EXECUTING`을 다시 실행하는 경로는 남지 않았으며, PostgreSQL과 Redis가 같은 판정을 한다. Minor 중 RR5-01-01(`IDEMPOTENCY_RECOVERY_REQUIRED`의 클라이언트 안내가 `http-contract.md` 표·매뉴얼 05 표·runbook에서 서로 다름 — 표는 "새 키 금지", runbook은 "효과가 없으면 새 키")은 오케스트레이터가 이번 커밋에서 만든 모순이라 바로 고쳤다: 세 문서와 프론트 인계 문서를 "자동 재시도 안 함, 같은 키 재전송은 안전(서버가 판정하고 다시 돌리지 않음), 효과가 없다고 확인된 뒤에만 운영 절차에 따라 새 키"로 맞췄다. 나머지 Minor 8건은 11.1절 결정대로 기록만 한다(round5/review.md) — 그중 RR5-02-01은 heredoc·호출하지 않은 함수·여러 줄 문자열 안의 `cd`도 따라간다는 것으로, 이 repo에서는 그 step이 실패해 조용한 거짓 통과는 아니다. + +## 16. 남은 결함 전부 수정 r9 (2026-11-05) + +소유자 지시: "보고할 때 남은 문제가 없어야 한다." 11.1절의 "Minor는 기록만" 결정을 거두고, 리뷰가 찾고 기록만 했던 것(13절 Minor 7건, 15.2절 Minor 8건, 10·12·15절 "남음", 원래 리뷰 16.3절)을 영역 묶음 4개가 `c508b1ba`에서 고쳤다. 고치다 새로 찾은 영역 안의 결함도 고쳤다. + +| 묶음 | 항목 | 처리 | 검증 (에이전트) | 결정 | +|---|---|---|---|---| +| r9-a core·inbound | RR4-01-01, RR4-01-02, `graphql..` fixture, 원래 리뷰 16.3, RR4-03-03 | 규칙 대상을 `@SystemCaller`와 그 프로젝트 상위 타입으로 넓히고 `SYSTEM_CALLERS_HAVE_ONLY_PROJECT_SUPERTYPES`(프로젝트 밖 타입 구현 금지)를 더했다(RED). `@Configuration`·`@AutoConfiguration`은 사용자 전송이 아니다(RED 2). graphql-java를 architectureTest 컴파일 전용으로 넣어 `graphql..` 항목을 fixture로 증명. 템플릿 패키지 목록: `application-core/CLAUDE.md` 표를 정본으로, `TemplatePackageOwnership`과 소유 leaf를 대조하는 문서 대조 테스트(RED 3). shared-contract `ServerFailureLog`로 web·GraphQL·gRPC·STOMP가 같은 실패 로그 줄을 쓴다(RED gRPC 4·GraphQL 2). 새로 찾아 고친 것: gRPC가 `INTERNAL_ERROR` 실패를 아무 데도 남기지 않음, STOMP가 persistence 실패의 SQL 원인까지 로그에 씀 | 9 leaf `check`·`qualityCheck`·`testQualityCheck`, architectureTest 216, documentContractTest 65, architectureCheck, verifyDependencyLocks, `-p optional-platforms ci`, 전송 qualification lane, systemTest | 규칙으로 막을 수 없는 길(bean 이름 조회, 설정이 함수로 감싸 넘김)과 그 길을 막는 다른 방어를 매뉴얼 07에 적었다. 목록에 넣지 않은 새 템플릿 패키지는 기능 패키지와 구별할 수 없다(한계로 문서화) | +| r9-b 멱등성·outbound | README 15 남음(CLAIMED → 시작의 lease), RR5-01-02~05, RR5-03-01, RR4-03-01·02·04 | 두 저장소 모두 lease가 지난 CLAIMED의 시작을 `LEASE_EXPIRED`로 거절하고 executor는 503 `IDEMPOTENCY_UNAVAILABLE`(실행 안 됨)로 답한다(RED 실 Redis·실 PG). renew·start 결과 값마다 Javadoc. 포트 Javadoc "먼저 온 보고 하나". 늦은 실패 뒤 다음 claim까지 보는 계약 4건. PostgreSQL 기본 사용 모양(쓰기 트랜잭션 안의 executor)에서는 lease 지난 EXECUTING이 남에게 보이지 않는다는 사실을 문서에 저장소별로. TTL 관계(lease ≤ replay-ttl ≤ failure-retention) 안내. watchdog RED 테스트를 공용 풀 병렬도와 무관하게, runbook의 Redis 기록 삭제를 `EVALSHA` 한 번으로(실 Redis·실 ACL에서 runbook 블록 그대로 실행), watchdog 스레드는 60초 유휴 뒤 끝나고 요청 스레드의 thread-local·class loader를 물려받지 않는다(RED) | 6 leaf `check`·`testQualityCheck`, `redisContainerContractTest` 72, PG 계약 31, `postgresqlIdempotencyIntegrationTest` 10, `httpClientPerformanceTest` 5, architectureCheck, verifyEnvKeys, documentContractTest | 애플리케이션 Redis 계정은 ACL이 `EVAL`을 막아 runbook은 `SCRIPT LOAD`+`EVALSHA`. 시작 거절은 아무것도 실행되지 않았으므로 503 | +| r9-c persistence | RR4-02-01, gRPC 버전 정규식 복사본, V21 주석 | 잠금 대기 검사기가 Flyway 11.14.1이 스스로 트랜잭션 밖에서 돌리는 문장 목록을 쓰고(실 PG 21개 판정 일치), `VACUUM FULL`은 마이그레이션에서 거부(RED 25). 버전 판정 정본 `VersionedMigrationFiles` 하나를 grpc 빌드가 Sync 태스크로 가져와 컴파일하고 복사본을 지웠다(RED 2). V21 주석은 `bbe944bd`에서 이미 고쳐져 있었다. 새로 찾아 고친 것: 가이드의 "CONCURRENTLY는 Flyway가 트랜잭션으로 감싼다" 오류 | persistence-jpa `check` 647, sample `check` 339, `postgresqlMigrationLockWaitIntegrationTest` 28, optional grpc ledger `check`·`integrationTest` 46, architectureCheck 두 빌드, documentContractTest | 버린 대안: 원본 디렉터리를 소스 루트로 직접 추가(Spotless가 프로젝트 밖 파일 거부), 주 빌드 의존, 대조 테스트(정의가 둘로 남음) | +| r9-d build-logic | README 15 남음 + RR5-02-02, RR5-02-01, RR5-02-03, `-mx`, httpclient 장애 주입 lane 두 번 실행 | `ShellScript`가 run 스크립트를 bash 문법으로 읽는다. 실패를 막지 못하는 명령(`|| true`, pipefail 없는 `| tee`, `!`, `&`, `echo`, `if`·`while` 조건, 함수 본문, `set +e`, `shell: pwsh`)은 이유를 대며 세지 않는다(RED). heredoc·여러 줄 문자열 안의 `cd`는 데이터(RED). 실패 메시지가 다른 빌드로 읽힌 명령·dry-run·안 센 이유를 보여 준다(RED). 짧은 플래그 묶음은 Gradle 9.0 파서대로(TestKit으로 Gradle 동작 고정). 장애 주입 lane을 `mainCheck`에서 뺐다(`integrationCheck`가 main에서 돌아 두 번 돌릴 근거가 사라졌다). 새로 찾아 고친 것: `$(( ))` 파싱, `help --task ci`, `${{ }}` 안 공백, `-x=b` | `-p build-logic check javadoc` 146, 두 빌드 verifyStrictLaneStages, 실제 workflow 판독 전후 같음, 변형 실험 8, run 스크립트 253개 파싱 실패 0 | 서브셸·`eval`·`source` 안의 `cd` 뒤 명령과 `|| exit 1`은 세지 않는다(거짓 실패 쪽) | + +합칠 때 오케스트레이터: 업그레이드 안내를 r9-a·r9-b와 손으로 합쳤다(5절에 세 묶음이 각자 더한 행을 모두 살림, B-41은 r9-b). + +### 16.1 합친 뒤 게이트에서 드러난 것 + +첫 `ci`가 httpclient `ClientRuntimeRegistryTest.closingTheRegistryReleasesEveryGenerationAndLeavesNoThread` 하나로 실패했다(부하 16, r9 변경과 무관한 기존 코드). 원인 둘: +- `ClientRuntimeRegistry.close()`는 drain scheduler의 `awaitTermination`만 기다렸다. 실행기는 worker 스레드 안에서 종료를 알린 뒤 그 스레드가 끝나므로, 부하가 높으면 `close()`가 돌아왔을 때 스레드가 아직 살아 있다. "스레드를 남기지 않는다"는 약속이 타이밍에 기댔다. +- 런타임 하나가 닫기를 거부하면 `close()`가 scheduler를 멈추기 전에 예외를 던져 drain 스레드가 계속 살았다. + +고친 것: scheduler가 만든 스레드를 기억해 `close()`가 종료를 기다린 뒤 그 스레드를 join한다(마감 안에 안 끝나면 실패). 런타임이 닫기를 거부해도 scheduler를 먼저 멈추고 예외는 그 뒤에 던진다. RED: 새 테스트 `aRuntimeThatRefusesToCloseStillLeavesNoDrainThread`가 옛 코드에서 실패(기존 테스트도 함께 실패). GREEN: httpclient `check`·`testQualityCheck`·`httpClientFailureInjectionTest`·`httpClientPerformanceTest`, 그 테스트 클래스 5회 반복 통과. + +### 16.2 게이트 (2026-11-05, 공유 잠금) + +| 게이트 | 결과 | 시간 | 실행 직전 부하(1분) | +|---|---|---|---| +| `ci` 첫 실행 | 실패 — httpclient `ClientRuntimeRegistryTest` 1건(16.1절) | 8분 48초 | 16.28 | +| `ci` 재실행(16.1절 수정 뒤) | rc=0 | 1분 22초(나머지 캐시) | 17.54 | +| `verifyStrictLaneStages --dry-run` (주 빌드 / optional) | rc=0 / rc=0 | 19초 / 4초 | 20.30 / 18.13 | +| `mainCheck` | rc=0 | 10분 59초 | 17.48 | +| `:integrationCheck` | rc=0 | 20분 11초 | 14.92 | +| `-p optional-platforms ci integrationCheck performanceCheck` | rc=0 | 2분 16초 | 15.57 | +| `:performanceCheck` | rc=0 | 36초 | 24.43 | +| `:releaseCheck -PgitRevision=` | rc=0 | 5초 | 19.00 | +| `scripts/verify-compose-profile-contracts.sh` | rc=0, 18 lane | — | 18.44 | + +`mainCheck`·`integrationCheck`는 16.1절 수정 전 트리에서 돌았다. 그 수정은 httpclient만 바꿔서 httpclient `check`·`testQualityCheck`·`httpClientFailureInjectionTest`·`httpClientPerformanceTest`를 따로 다시 돌렸다(통과). diff --git a/docs/runbooks/idempotency-unavailable.md b/docs/runbooks/idempotency-unavailable.md index 457a53af..6f0f25fa 100644 --- a/docs/runbooks/idempotency-unavailable.md +++ b/docs/runbooks/idempotency-unavailable.md @@ -4,7 +4,7 @@ category: TRANSIENT_DEPENDENCY error_codes: [IDEMPOTENCY_UNAVAILABLE, IDEMPOTENCY_COMPLETION_UNKNOWN, IDEMPOTENCY_RECOVERY_REQUIRED] severity: P2 owner: oncall -last_updated: 2026-11-04 +last_updated: 2026-11-05 status: active --- @@ -17,6 +17,11 @@ status: active - `IdempotencyExecutor` refused to run the action because the replay store (JPA or Redis, whichever `app.idempotency` selects) did not answer. Nothing was executed: the refusal happens before the action, so a client retry is safe. +- The same 503 when the store refused to start a claim whose processing lease had already run out + — the request spent longer than the lease between its claim and the start of its action (a + stalled process, a slow store call). The ERROR line's message then reads `the claim's processing + lease ran out before the action started; nothing ran: `, and the store may be healthy. + Nothing ran here either, and the next request with the key takes the claim over and runs it. - HTTP 503 `IDEMPOTENCY_COMPLETION_UNKNOWN` (`retryable=true`) is the same outage hitting after the action ran: the store did not answer while the result was being recorded. When the use case called the executor inside its write transaction, that transaction's database writes rolled back with the @@ -32,6 +37,10 @@ status: active record. If the attempt is still running and finishes, its result is recorded, the first request gets its normal answer, and later requests with the key get that result replayed; a failure it reports is recorded the same way. So this 500 can stop on its own (owner decision 2026-11-04). + This is the Redis store's case. The PostgreSQL store, called inside the use case's write + transaction as the template calls it, never shows another request an unfinished record: that + request waits for the first transaction to end, and gets the replay if it committed or a fresh + run if it rolled back (`docs/web/http-contract.md` §2). ## Diagnosis @@ -39,16 +48,27 @@ status: active JPA datasource or Redis, depending on the selected provider). - Look for connection-pool saturation or timeouts on that store in the same time window; the executor reports saturation as unavailable rather than guessing. -- For `IDEMPOTENCY_RECOVERY_REQUIRED`, find the scope in the server's ERROR log of the failed request - (the web adapter logs `failure classified as IDEMPOTENCY_RECOVERY_REQUIRED` with the exception, - under the request's `X-Request-Id`). The exception message ends with +- For `IDEMPOTENCY_RECOVERY_REQUIRED`, find the scope in the server's ERROR log of the failed request. + Every transport writes the same line, `failure classified as IDEMPOTENCY_RECOVERY_REQUIRED + (category=DATA_INTEGRITY, retryable=false)`, with the exception attached + (shared-contract `ServerFailureLog`), and adds its own correlation: + - web: no suffix; the line carries the request's `X-Request-Id` (MDC `request_id`). + - gRPC: ` (method=/)`; MDC `request_id` too when the service closed the call on + its callback thread. + - GraphQL: ` (executionId=)`, the `executionId` the client got in the error's `extensions`. + - STOMP (`@MessageMapping`): no suffix; MDC `request_id` is the frame's. + + The exception message ends with `::v::`, for example `claim requires reconciliation: CHARGE_CARD::v1::3f9a…` (64 hex characters). Then look at the record in the store the service uses (`APP_IDEMPOTENCY_PROVIDER`). There is no tool for this; these are the commands. - - Redis. The key is - `:::idem:v1:d::`, - for example `prod:order:shared:idem:v1:d1:CHARGE_CARD:3f9a…`. + - Redis. The key, for example `prod:order:shared:idem:v1:d1:CHARGE_CARD:3f9a…`, is + + + ``` + :::idem:v1:d:: + ``` ``` HMGET state attempt leaseUntil @@ -77,7 +97,10 @@ status: active ## Action -- `IDEMPOTENCY_UNAVAILABLE`: restore the store; clients retry with the same key and backoff. +- `IDEMPOTENCY_UNAVAILABLE`: restore the store; clients retry with the same key and backoff. When + the message says the claim's processing lease ran out before the action started, the store is + not the cause: find what held those requests between claim and start (thread dumps, GC pauses, + the store's latency in that window). Nothing to repair; the next same-key request runs. - `IDEMPOTENCY_COMPLETION_UNKNOWN`: restore the store, then check the effects outside the transaction for the failed requests (the ERROR log line names the code). A same-key resend replays a result that was in fact recorded, gets `IDEMPOTENT_IN_FLIGHT` while the attempt's lease lasts, @@ -100,8 +123,26 @@ status: active `EXECUTING` record, even past its lease: if its attempt is still alive, its late result finds no record (`ABSENT`) — the result is lost, the first request gets this 500 after its work took effect, and the next request with the key runs the work a second time. Leave it to expire. - - Redis: check `HMGET state` is `ABANDONED` and `PTTL ` has minutes left, then - `DEL `. A record that is about to expire needs nothing; once it is gone the key is new. + - Redis: check and delete in one step, as the account the service's registered scripts run as + (`APP_REDIS_AUTHENTICATION_ADVANCED_CREDENTIAL_REFERENCE`, or the application account when it + is unset). Checking with `HMGET` and then sending `DEL` is two commands: a record that expires + between them and is claimed again by a new request is deleted while that request runs, and + the next request with the key runs the work a second time (re-review 2026-11-04 RR4-03-02). + The program below deletes the key only while its `state` is `ABANDONED`. The application + accounts are denied `EVAL` (`infra/redis-sdk/acl/README.md`), so load it and run it by the + digest `SCRIPT LOAD` prints, as the service runs its own scripts: + + + ``` + SCRIPT LOAD "if redis.call('HGET', KEYS[1], 'state') == 'ABANDONED' then return redis.call('DEL', KEYS[1]) end return 0" + EVALSHA 1 + ``` + + `1` deleted the record; the key is new. `0` deleted nothing: the record is no longer + `ABANDONED` (or is gone) — look again before doing anything else. `NOSCRIPT` means the node + that holds the key (a cluster) has not loaded the program: run `SCRIPT LOAD` on that node and + `EVALSHA` again; nothing was deleted. A record that is about to expire needs nothing; once it + is gone the key is new. - PostgreSQL: ```sql diff --git a/docs/study/postgresql-jpa-walkthrough.ko.md b/docs/study/postgresql-jpa-walkthrough.ko.md index 8e95475e..5133a1d4 100644 --- a/docs/study/postgresql-jpa-walkthrough.ko.md +++ b/docs/study/postgresql-jpa-walkthrough.ko.md @@ -1748,6 +1748,9 @@ update idempotency_record > 무언가가 record를 움직였으니 caller는 자기 view를 현재로 취급하면 안 된다. row를 읽고 나서 > scope만으로 update하면 lease가 만료된 worker가 takeover한 worker의 상태를 덮어쓸 수 있다. +START/RENEW만은 store가 잠근 row를 owner 튜플과 먼저 비교한 뒤에 돌리므로, 그 둘의 0은 lease가 끝난 +것이다. 그래서 `NOT_OWNER`가 아니라 `LEASE_EXPIRED`로 답한다(재리뷰 2026-11-04 RR5-01-02). + `IdempotencyDigestPolicy`의 기록이 특히 좋다. > `transition|operationId|ownerToken|attempt|stateRevision`은 **누가, 언제**를 덮고 **무엇**은 @@ -2832,7 +2835,8 @@ Flyway는 세 옵션이 **pin**되어 있다(env-driven 아님): `baseline-on-mi `reaper-interval: PT10M`, `published-retention: P7D`. `ca-skeleton.capabilities.idempotency.*`: `provider: disabled`, `processing-lease: 30s`, -`replay-ttl: 24h`(`APP_IDEMPOTENCY_TTL`, 최대 72h), `failure-retention: 24h`, +`replay-ttl: 24h`(`APP_IDEMPOTENCY_TTL`, 최대 72h, `processing-lease` 이상·`failure-retention` 이하), +`failure-retention: 24h`, `reaper-interval: 10m`. 한 record(`IdempotencySettings`)가 전부 바인딩한다. --- diff --git a/docs/upgrade/2026-11-maturity-review.md b/docs/upgrade/2026-11-maturity-review.md index 6ea0595b..a9697fe6 100644 --- a/docs/upgrade/2026-11-maturity-review.md +++ b/docs/upgrade/2026-11-maturity-review.md @@ -25,7 +25,7 @@ | 2 | dev·prod 환경 변수의 `APP_FLYWAY_ENABLED` 값을 확인한다 | `false`가 남아 있으면 이제 정말로 마이그레이션이 안 돈다 | persistence-jpa 선택 | | 3 | prod 비밀값(레지스트리 `secret` 행, 모듈이 기여한 비밀, `secret://…/`이 가리키는 변수)이 `fixture-`로 시작하지 않게 한다 | prod 기동 실패(exit 78) | prod 배포 전부 | | 4 | 직접 만든 `StartupRule`·`RequiredProductionSecrets` bean을 `static @Bean`, 의존성 없이 바꾸고, 그 클래스에서 `@Value`·`@Autowired` 멤버와 `@ConfigurationProperties`를 뺀다 | 기동 실패 `startup rule or secret list '<이름>' created [...]` 또는 `'<이름>' (...) is a StartupRule or RequiredProductionSecrets ... but it needs one`(exit 78) | 자기 rule이 있는 서비스 | -| 5 | 직접 구현한 outbox·발행·알림·파일 서버 포트를 새 시그니처로 고친다(3.1절) | 컴파일 오류 | 해당 포트를 구현·호출하는 서비스 | +| 5 | 직접 구현한 outbox·발행·알림·파일 서버 포트를 새 시그니처로, 직접 구현한 멱등성 store를 새 결과 값(B-41)으로 고친다(3.1절) | 컴파일 오류, 멱등성 공유 계약 실패 | 해당 포트를 구현·호출하는 서비스 | | 6 | `dev.caskeleton.application.outbound.*` import를 `application.secret.*`·`application.objectstorage.content.CallBudget`으로 바꾼다 | 컴파일 오류 | 해당 타입을 쓰는 서비스 | | 7 | 이벤트 id를 소문자 UUIDv7로 만들고 `IntegrationEventDraft`는 `IntegrationEventDrafts.draft(...)`로만 만든다 | 쓰기 요청이 `IllegalArgumentException`으로 실패, architectureTest 실패 | 통합 이벤트를 내는 서비스 | | 8 | V21 배포 전에 `docs/runbooks/outbox-dead-letter.md` §8 Step 1 쿼리를 운영 DB에 돌린다 | 중복 순서 키가 있으면 기동 실패(exit 70, V20에 남음) | persistence-jpa + outbox | @@ -62,6 +62,7 @@ | B-10 | adapter 공개 타입 삭제·이동: graphql `execution.GraphQlTimeoutPolicy`·`dataloader.GraphQlDataLoaderRequestRegistry` 삭제(`runtime.GraphQlDeadlineInstrumentation`·`GraphQlPerParentFetchInstrumentation` 추가). jpa `experimental.tenant.TenantId` 삭제 → `dev.caskeleton.application.tenant.TenantId`. persistence-notification의 엔티티는 `...platform.entity`, 저장소는 `...platform.repository`로 이동, `inbox` 하위 클래스는 `platform`으로 올라옴, `PreferenceJpaRepository`·`SuppressionJpaRepository` 삭제, `JpaNotificationSideEffect` → application-notification `dispatch.SuppressionSideEffect`. graphql `GraphQlClientPolicy.effectivePageSize(Integer, Integer)` 삭제 → `pagination.GraphQlConnectionPolicy.effectivePageSize`(상한 검사 포함, 재리뷰 RR-01-14). grpc(optional-platforms) `GrpcOperationLedger.release(identity)` → `boolean release(identity, GrpcClaimToken)`: 진행 중이고 그 token이 가진 claim만 지운다(RR-01-15) | import 실패, 호출부·직접 구현한 ledger 컴파일 오류. leaf `check`의 API surface 비교가 스냅샷과 달라 실패(자기 공개 타입을 더했다면 merge 충돌) | import를 바꾼다. 스냅샷 파일은 upstream 것을 받고 자기 추가분을 다시 승인한다 | `1400dc15`·`5786126c` · `docs/architecture/graphql-api-surface.txt`, `docs/architecture/jpa-api-surface.txt`(166→165), `docs/notification/api-surface-snapshot.txt`(362→364) | | B-11 | 조립·지원 타입: `bootstrap.autoconfigure.operation.DurableOperationConfig` → `composition:durable-operation`의 `composition.operation.DurableOperationAutoConfiguration`. `bootstrap.settings.LoggingSettings` 삭제. `composition.outbox.OutboxLeaderElectionToken` 삭제. `TracingSamplingEnvironmentPostProcessor` → `TracingEnvironmentPostProcessor`, `StartupOrder.TRACING_SAMPLING` → `TRACING_PROPERTIES`. `StartupRuleRunner(Environment, List)` → `StartupRuleRunner(Environment)`. `SecretSourceValidator(ConfigurableEnvironment, SecretSourcePort, List)` → `SecretSourceValidator(Supplier>)` | 이 타입을 참조하는 코드·테스트 컴파일 오류 | 새 이름으로 바꾼다 | `1400dc15`, `34565130`, `bbe944bd` · `src/composition/durable-operation/`, `src/adapter/outbound/support/.../startup/` | | B-39 | vendored messaging SPI `KafkaQuarantinePublisher`(`messaging-kafka`): `quarantine(ConsumerRecord, Throwable)` → `quarantine(ConsumerRecord, FailureCategory category, Throwable reason)`. 호출자가 격리 이유(읽을 수 없는 레코드 `DESERIALIZATION`, 시도 예산 소진은 실패의 category 또는 `POISON_MESSAGE`)를 넘기고, 구현은 그것을 `msg.failure-category` header로 쓴다 | 이 SPI를 직접 구현한 서비스는 `does not override abstract method` 컴파일 오류. 람다로 만든 것도 인자 수 오류 | 세 인자 메서드로 고친다. category를 header나 저장소에 그대로 남겨야 redrive가 두 경우를 가른다(템플릿 `KafkaDeadLetterQuarantine`, 테스트 `KafkaDeadLetterQuarantineTest`) | `bbe944bd`(리뷰 06-14), 3차 수정에서 목록에 추가(재리뷰 2차 RR-02-16) · `src/messaging/messaging-kafka/src/main/java/dev/caskeleton/messaging/kafka/KafkaQuarantinePublisher.java` | +| B-41 | enum 값 추가: `IdempotencyStartOutcome.LEASE_EXPIRED`, `IdempotencyRenewOutcome.LEASE_EXPIRED`(둘 다 `carriesOwner()`=false). 주인이 아직 기록을 가졌지만 processing lease가 끝나 시작·renew를 거절한다는 뜻이다. 전에는 PostgreSQL이 둘 다 `NOT_OWNER`, Redis가 renew는 `NOT_OWNER`·시작은 `STARTED`였다 | `default` 없는 switch 식이 "does not cover all possible input values" 컴파일 오류. 직접 구현한 store는 `IdempotencyStoreContract`의 새 시간 케이스(lease 지난 시작·renew)에서 실패 | 새 값 분기를 더한다. `LEASE_EXPIRED`를 `NOT_OWNER`처럼 "기록을 잃었다"로 읽지 않는다 — 실행 중이면 주인은 여전히 `complete`·`markFailed` 한다. store는 lease가 끝난 시작·renew에 이 값을 돌려준다 | r9(재리뷰 README 15 남음, RR5-01-02) · `src/application-core/.../idempotency/IdempotencyStartOutcome.java`, `IdempotencyRenewOutcome.java`, `src/application-core/src/testFixtures/.../IdempotencyStoreContract.java` | ### 3.2 설정·기동 — 기동이 실패하거나 런타임 예외가 난다 @@ -71,7 +72,7 @@ | B-13 | `app.security.*` strict 바인딩: 접두어 아래 오타 키, 또는 설정·placeholder 어디에도 쓰이지 않는 `APP_SECURITY_*` 환경 변수가 기동을 막는다 | 기동 실패(메시지는 `StrictSettingsBinder`가 키 이름을 담아 냄, 정확한 문구 확인 못 함) | 쓰지 않는 `APP_SECURITY_*` 변수를 지운다 | `5786126c` · `SecuritySettings.bind`, `src/adapter/outbound/support/.../StrictSettingsBinder.java` | | B-14 | prod 프로필에서 `fixture-`로 시작하는 비밀값을 거부(레지스트리 `secret` 행, 모듈이 기여한 비밀, `secret://…/` 참조 대상. `HOSTNAME`·앱 이름 같은 비밀 아닌 값은 보지 않는다 — RR2-04-01). Redis ACL fixture 파일 이름 변경: `infra/redis-sdk/acl/all-accounts.acl` → `lane-fixture-accounts.acl` | prod 기동 실패, 요약 `prod secret/config-source contract is invalid`(exit 78). 옛 ACL 경로를 mount하던 compose는 파일 없음 | prod 값을 진짜 비밀로 바꾼다. compose의 ACL 경로를 새 이름으로. Redis SDK admin 계정(`RedisAccountContribution`)을 조립했다면 그 변수를 `static` `RequiredProductionSecrets` bean으로도 낸다(코드에 든 참조는 검사가 보지 않는다, RR3-04-03, `src/adapter/outbound/redis-sdk/README.md`) | `bbe944bd` · `src/app-bootstrap/.../runtime/SecretSourceValidator.java`, `docs/registries/secrets-classification.yaml` | | B-15 | 기동 rule이 모든 singleton보다 먼저 돈다(`StartupRuleRunner`가 `BeanFactoryPostProcessor`). `StartupRule`과 `RequiredProductionSecrets` bean은 `static @Bean`이고 `Environment`, bean factory, static 무인자 `RequiredProductionSecrets`의 `ObjectProvider`만 받아야 한다. 두 타입의 클래스는 `@Value`·`@Autowired`·`@Inject`·`@Resource` 멤버, `@PostConstruct`·`@PreDestroy`, `@ConfigurationProperties`를 가질 수 없다(post-processor보다 먼저 만들어져 비어 있게 된다, RR2-04-03) | non-static `@Bean`이거나 다른 bean을 받으면 기동 실패 `startup rule or secret list '<이름>' created [] while creating it ... Declare the rule with a static @Bean method ...`, 위 멤버가 있으면 `'<이름>' (...) is a StartupRule or RequiredProductionSecrets ... but it needs one: [...]`(둘 다 exit 78, 둘 다 모든 프로필. runner가 rule과 비밀 목록을 프로필과 무관하게 먼저 만들어, 인터페이스 타입 `@Bean`이 돌려준 실제 클래스까지 본다, RR3-04-01) | rule 선언을 `static`으로, 인자를 위 셋으로 줄인다. 값은 `violations(Environment)`가 받는 `Environment`에서 읽는다. 다른 bean이 필요한 검사는 rule이 아니라 일반 bean으로 둔다 | `34565130`, `bbe944bd` · `src/adapter/outbound/support/.../startup/StartupRule.java`, `StartupRuleRunner.java` | -| B-16 | 권한 가드(새 `PermissionEnforcementGuard`): `@SystemInvoked` 유스케이스가 bean이면 기동 실패. `@RequiresPermission`을 단 bean에 적용되는 advisor가 없어도 기동 실패. ArchUnit `SYSTEM_INVOKED_USE_CASES_DECLARE_NO_PERMISSION`, `ONLY_SYSTEM_CALLERS_REACH_SYSTEM_INVOKED_USE_CASES`, `SYSTEM_CALLERS_ARE_SYSTEM_DRIVEN_INBOUND_ADAPTERS`, `USER_TRANSPORTS_DO_NOT_REACH_SYSTEM_CALLERS` 추가, 새 표시 `@SystemCaller` | 기동 실패 `@SystemInvoked use cases are registered as beans ...` 또는 `beans declare @RequiresPermission but no advisor enforces it ...`(exit 72). `@SystemCaller`가 아닌 inbound adapter가 그 유스케이스를 부르거나, 사용자 전송(controller 등)이 `@SystemCaller` bean을 직접·중간 클래스를 거쳐 부르면 architectureTest 실패 | 시스템만 부르는 쓰기 유스케이스는 composition에서 `new`로 만들어 스케줄러에 넘기거나, 메시지 handler·예약 작업이면 그 클래스에 `@SystemCaller`를 달고 생성자에서 `new`로 만든다. 예약 작업은 서비스 leaf의 `..adapter.inbound.scheduler..` 패키지에 둔다. 세 경로와 두는 곳은 매뉴얼 07장 「시스템이 부르는 쓰기」(handler 작업 순서는 09장 「소비」 4단계). "지금 실행" endpoint가 필요하면 `@RequiresPermission` 유스케이스를 따로 만든다. `@RequiresPermission`을 쓰면 web adapter(`MethodSecurityConfig`)가 조립돼 있어야 한다 | `1400dc15`, `bbe944bd` · `src/composition/service-runtime/.../authorization/PermissionEnforcementGuard.java`, `src/application-core/.../security/SystemInvoked.java`, `SystemCaller.java` | +| B-16 | 권한 가드(새 `PermissionEnforcementGuard`): `@SystemInvoked` 유스케이스가 bean이면 기동 실패. `@RequiresPermission`을 단 bean에 적용되는 advisor가 없어도 기동 실패. ArchUnit `SYSTEM_INVOKED_USE_CASES_DECLARE_NO_PERMISSION`, `ONLY_SYSTEM_CALLERS_REACH_SYSTEM_INVOKED_USE_CASES`, `SYSTEM_CALLERS_ARE_SYSTEM_DRIVEN_INBOUND_ADAPTERS`, `USER_TRANSPORTS_DO_NOT_REACH_SYSTEM_CALLERS`, `SYSTEM_CALLERS_HAVE_ONLY_PROJECT_SUPERTYPES` 추가, 새 표시 `@SystemCaller` | 기동 실패 `@SystemInvoked use cases are registered as beans ...` 또는 `beans declare @RequiresPermission but no advisor enforces it ...`(exit 72). `@SystemCaller`가 아닌 inbound adapter가 그 유스케이스를 부르거나, 사용자 전송(controller 등)이 `@SystemCaller` bean을 직접·중간 클래스를 거쳐·그 작업이 구현하는 프로젝트 인터페이스로 부르면 architectureTest 실패. `@SystemCaller` 클래스가 `Runnable`·`ApplicationRunner` 같은 프로젝트 밖 타입을 구현·상속해도 실패. 설정(`@Configuration`)은 웹 bean과 handler bean을 함께 등록해도 사용자 전송으로 세지 않는다 | 시스템만 부르는 쓰기 유스케이스는 composition에서 `new`로 만들어 스케줄러에 넘기거나, 메시지 handler·예약 작업이면 그 클래스에 `@SystemCaller`를 달고 생성자에서 `new`로 만든다. 예약 작업은 서비스 leaf의 `..adapter.inbound.scheduler..` 패키지에 둔다. 세 경로와 두는 곳은 매뉴얼 07장 「시스템이 부르는 쓰기」(handler 작업 순서는 09장 「소비」 4단계). "지금 실행" endpoint가 필요하면 `@RequiresPermission` 유스케이스를 따로 만든다. 작업을 task scheduler에 직접 등록하던 코드는 작업이 `Runnable`을 구현하는 대신 등록하는 곳에서 `job::run`을 넘긴다. `@RequiresPermission`을 쓰면 web adapter(`MethodSecurityConfig`)가 조립돼 있어야 한다 | `1400dc15`, `bbe944bd`, r9 · `src/composition/service-runtime/.../authorization/PermissionEnforcementGuard.java`, `src/application-core/.../security/SystemInvoked.java`, `SystemCaller.java` | | B-17 | `IntegrationEventDraft`: `eventId`·`causationId`는 소문자 UUIDv7만. ArchUnit `INTEGRATION_EVENT_DRAFTS_ARE_BUILT_BY_THE_SHARED_HELPER`: `new IntegrationEventDraft<>(...)` 금지, `IntegrationEventDrafts.draft(...)`만 | 이벤트를 만드는 쓰기 요청이 `IllegalArgumentException: eventId must be a canonical lower-case UUIDv7...`로 실패. architectureTest 실패 | event id를 UUIDv7로 만든다(예: sample `UuidOutboxEventIdFactory`가 `UuidCreator.getTimeOrderedEpochPlus1()`). draft는 헬퍼로 만든다. tenant가 없는 이벤트는 `IntegrationEventDrafts.SINGLE_TENANT_SCOPE`(`global`) | `1400dc15`, `bbe944bd` · `src/application-core/.../messaging/event/IntegrationEventDraft.java`, `IntegrationEventDrafts.java` | | B-18 | `MessageHandler.schemaVersions()`(vendored `messaging-core-api`, default `{v1}`): 소비자 decode 목록을 handler 선언으로 만든다. 같은 (메시지 타입, 버전)에 다른 payload class를 선언하면 실패 | 컴파일은 된다. v1이 아닌 메시지는 `SCHEMA_VERSION_NOT_REGISTERED`로 거부. 발행 등록(`MessageContracts`)과 class가 다르면 기동 실패 `MESSAGING_CONTRACT_CONFLICT` | v2 이상을 읽는 handler는 `schemaVersions()`를 override한다. 소비 타입을 `MessageContracts`에 따로 등록하던 코드는 지운다 | `5786126c` · `src/messaging/messaging-core-api/.../delivery/MessageHandler.java`, `messaging-spring-boot-starter/.../MessageContracts.java` | | B-19 | `APP_FLYWAY_ENABLED`가 모든 프로필에서 읽힌다. dev·prod 프로필 파일의 `spring.flyway.enabled: true`·`app.persistence.vendor` 고정값을 지우고 persistence-jpa `config/module-defaults`가 `${APP_FLYWAY_ENABLED:true}`를 싣는다 | dev·prod 환경에 `APP_FLYWAY_ENABLED=false`가 있으면 전에는 무시됐지만 이제 기동 때 마이그레이션이 안 돈다 | dev·prod 환경 값을 확인한다. 자기 `application-dev.yml`·`application-prod.yml`에 같은 고정값을 다시 넣지 않는다 | `5786126c` · `src/app-bootstrap/src/main/resources/application-{dev,prod}.yml`, `src/adapter/outbound/persistence-jpa/.../module-defaults/persistence-jpa.yml` | @@ -101,9 +102,9 @@ | # | 무엇이 바뀌었나 | 파생 서비스 증상 | 할 일 | 근거 | |---|---|---|---|---| -| B-34 | 템플릿 패키지 규칙(ArchUnit): `FEATURE_PACKAGES_DO_NOT_REUSE_TEMPLATE_PACKAGE_NAMES`, `TEMPLATE_PACKAGES_DO_NOT_DEPEND_ON_FEATURE_PACKAGES`. 템플릿 이름: application `usecase command query capability transaction security tenant observability secret messaging outbox idempotency operation lease lock cache realtime notification fileserver objectstorage`, domain `identifier stereotype` | 기능 패키지 이름이 이 목록과 겹치거나(예: `domain.lock`), 업무 코드를 템플릿 패키지(예: `application.usecase.*`)에 두고 기능 패키지를 import하면 `:app-bootstrap:architectureTest` 실패 | 기능 패키지 이름을 바꾸고, 업무 코드를 `dev.caskeleton.application.`로 옮긴다. 규칙 정본은 `src/application-core/CLAUDE.md` "Package ownership" | `5786126c` · `src/app-bootstrap/src/architectureTest/.../TemplatePackageOwnership.java` | +| B-34 | 템플릿 패키지 규칙(ArchUnit): `FEATURE_PACKAGES_DO_NOT_REUSE_TEMPLATE_PACKAGE_NAMES`, `TEMPLATE_PACKAGES_DO_NOT_DEPEND_ON_FEATURE_PACKAGES`. 템플릿 이름: application `usecase command query capability transaction security tenant observability secret messaging outbox idempotency operation lease lock cache realtime notification fileserver objectstorage`, domain `identifier stereotype` | 기능 패키지 이름이 이 목록과 겹치거나(예: `domain.lock`), 업무 코드를 템플릿 패키지(예: `application.usecase.*`)에 두고 기능 패키지를 import하면 `:app-bootstrap:architectureTest` 실패 | 기능 패키지 이름을 바꾸고, 업무 코드를 `dev.caskeleton.application.`로 옮긴다. 규칙 정본은 `src/application-core/CLAUDE.md` "Package ownership" | `5786126c`, r9 · `src/app-bootstrap/src/test/.../TemplatePackageOwnership.java`(표와 대조: `TemplatePackageOwnershipDocumentTest`) | | B-35 | JPA 규칙: 양방향 연관은 한쪽에서만 매핑(`mappedBy`), notification·fileserver 저장소 엔티티는 그 leaf 안에서만, `@Version` 엔티티의 bulk update는 version을 올려야 함 | 위반 엔티티·쿼리가 있으면 architectureTest 실패 | 매핑·쿼리를 고친다 | `34565130`, `1400dc15`, `bbe944bd` · `src/app-bootstrap/src/architectureTest/.../JpaProductionArchitectureTest.java` | -| B-36 | 빌드 검사 강화: `modules.json`에 없는 `build.gradle`이 있으면 `verifyCleanArchitectureDependencies` 실패. `verifyStrictLaneStages --dry-run`은 모든 strict lane이 workflow가 실행하는 단계(`:ci` `:sample-portfolio:check` `:mainCheck` `:integrationCheck` `:performanceCheck` `:releaseCheck`)에 속해야 하고 `.github/workflows`를 읽는다. 명령의 빌드는 `working-directory`(단계·job·workflow `defaults`)에서 시작해 같은 `run:` 셸이 그 앞에서 실행한 `cd <상대 경로>`(`cd src && ./gradlew …`, `cd src; ./gradlew …`, 앞 줄의 `cd src`)를 따라간 뒤 `-p`를 붙인 디렉터리로 정하고(`70fb80bf`(2026-11-04 소유자 결정)), `workflow_dispatch`로만 시작하는 workflow의 명령은 세지 않으며, 축약한 `-x`도 Gradle처럼 맞춰 본다(RR2-04-04). 붙여 쓴 `-x`·`-p`도 Gradle처럼 읽는다(RR3-04-02). `verifyEnvKeys`: lane 계약 json의 `APP_` 밖 키는 `SPRING_`·`MANAGEMENT_`·`SERVER_`이거나 `secret://…/`가 가리키는 변수여야 하고, 레지스트리 `property:` 행은 그 property가 설정 metadata에 있어야 읽히는 것으로 친다(`@ConditionalOnProperty`로만 읽는 스위치는 `META-INF/additional-spring-configuration-metadata.json`에 선언, RR2-04-05). SpotBugs 신뢰도 HIGH → MEDIUM | 등록 안 한 leaf, 단계에 안 걸린 자기 lane, 자기 코드의 MEDIUM 발견이 `ci`·`qualityCheck`를 막는다. `working-directory: src`도 `cd src`도 없이 `./gradlew`를 부르는 자기 workflow 단계는 실행으로 세지 않는다. 확실히 따라갈 수 없는 디렉터리 이동(`cd "$VAR"`, `cd -`, `cd ~`, 절대 경로, `pushd`, `if`/`for`/`case`·`{ }`·서브셸 안의 `cd`, `||`·`|`로 이은 `cd`, 다른 명령과 `&&`로 묶인 `cd` 다음 줄) 뒤의 명령도 세지 않고, 실패 메시지에 그 명령과 이유를 적는다 | leaf를 등록하거나 지운다. 자기 lane은 단계에 건다. 단계를 부르는 workflow 명령에 `working-directory`를 두거나 같은 셸에서 `cd <상대 경로>`로 옮긴 뒤 부른다. SpotBugs 발견은 고치거나 이유를 단 제외로 | `1400dc15`, `34565130`, `bbe944bd`, `5786126c`, `70fb80bf`(2026-11-04 소유자 결정) · `src/build-logic/.../ModuleRegistry.java`, `.../strictlane/StrictLaneStagesPlugin.java`, `.../strictlane/WorkflowGradleInvocations.java`, `.../quality/QualityConventionsPlugin.java`, `src/build.gradle` | +| B-36 | 빌드 검사 강화: `modules.json`에 없는 `build.gradle`이 있으면 `verifyCleanArchitectureDependencies` 실패. `verifyStrictLaneStages --dry-run`은 모든 strict lane이 workflow가 실행하는 단계(`:ci` `:sample-portfolio:check` `:mainCheck` `:integrationCheck` `:performanceCheck` `:releaseCheck`)에 속해야 하고 `.github/workflows`를 읽는다. 명령의 빌드는 `working-directory`(단계·job·workflow `defaults`)에서 시작해 같은 `run:` 셸이 그 앞에서 실행한 `cd <상대 경로>`(`cd src && ./gradlew …`, `cd src; ./gradlew …`, 앞 줄의 `cd src`)를 따라간 뒤 `-p`를 붙인 디렉터리로 정하고(`70fb80bf`(2026-11-04 소유자 결정)), `workflow_dispatch`로만 시작하는 workflow의 명령은 세지 않으며, 축약한 `-x`도 Gradle처럼 맞춰 본다(RR2-04-04). 붙여 쓴 `-x`·`-p`도 Gradle처럼 읽는다(RR3-04-02). `run:` 스크립트를 bash 문법으로 읽어(heredoc·여러 줄 문자열·함수 본문 안의 `cd`는 따라가지 않음), 실패해도 단계를 실패시키지 못하는 명령(`\|\| true`·`\|\| exit 1`, pipefail 없는 파이프 왼쪽, `!`, `&`, `if`·`while` 조건과 `if`·`for`·`case`·함수·`$( )`·`bash -c` 안, `set +e`·최상위 `exit` 뒤, 스크립트 마지막이 아닌 `&&` 목록), Gradle이 거부하는 명령줄(`-mx`)과 묶음 dry-run(`-qm`)은 세지 않고, `shell:`이 기본·`bash`·`sh`가 아닌 단계도 세지 않는다. 단계를 부르지만 세지 않은 명령은 다른 빌드로 읽힌 것까지 실패 메시지에 이유와 함께 적는다(r9). `verifyEnvKeys`: lane 계약 json의 `APP_` 밖 키는 `SPRING_`·`MANAGEMENT_`·`SERVER_`이거나 `secret://…/`가 가리키는 변수여야 하고, 레지스트리 `property:` 행은 그 property가 설정 metadata에 있어야 읽히는 것으로 친다(`@ConditionalOnProperty`로만 읽는 스위치는 `META-INF/additional-spring-configuration-metadata.json`에 선언, RR2-04-05). SpotBugs 신뢰도 HIGH → MEDIUM | 등록 안 한 leaf, 단계에 안 걸린 자기 lane, 자기 코드의 MEDIUM 발견이 `ci`·`qualityCheck`를 막는다. `working-directory: src`도 `cd src`도 없이 `./gradlew`를 부르는 자기 workflow 단계는 실행으로 세지 않는다. 확실히 따라갈 수 없는 디렉터리 이동(`cd "$VAR"`, `cd -`, `cd ~`, 절대 경로, `pushd`, `if`/`for`/`case`·`{ }`·서브셸 안의 `cd`, `||`·`|`로 이은 `cd`, 다른 명령과 `&&`로 묶인 `cd` 다음 줄) 뒤의 명령도 세지 않고, 실패 메시지에 그 명령과 이유를 적는다. 단계를 `./gradlew ci \| tee log`(기본 셸), `./gradlew ci \|\| true`, `if ./gradlew ci; then` 같은 모양으로만 부르는 자기 workflow도 실패한다 | leaf를 등록하거나 지운다. 자기 lane은 단계에 건다. 단계를 부르는 workflow 명령에 `working-directory`를 두거나 같은 셸에서 `cd <상대 경로>`로 옮긴 뒤 부른다. 단계 명령은 혼자 한 줄로 두고, 로그를 파이프로 남기려면 `shell: bash`나 앞 줄 `set -o pipefail`을 둔다. SpotBugs 발견은 고치거나 이유를 단 제외로 | `1400dc15`, `34565130`, `bbe944bd`, `5786126c`, `70fb80bf`(2026-11-04 소유자 결정), r9 · `src/build-logic/.../ModuleRegistry.java`, `.../strictlane/StrictLaneStagesPlugin.java`, `.../strictlane/WorkflowGradleInvocations.java`, `.../strictlane/ShellScript.java`, `.../quality/QualityConventionsPlugin.java`, `src/build.gradle` | | B-37 | app-bootstrap 테스트 lane 분리: 저장소 파일을 읽는 테스트는 태그 `document-contract`(main 단계) 또는 `repository-contract`(`check`)여야 하고, 일반 `test` lane에는 `ca.repository.root`가 없다 | app-bootstrap에 둔 자기 테스트가 저장소 파일을 읽으면 `test`에서 실패 | 태그를 단다 | `1400dc15` · `src/app-bootstrap/build.gradle` | | B-40 | 테스트 코드 정적 분석 `testQualityCheck`: SpotBugs·FindSecBugs를 main 밖 모든 source set(test, testFixtures, integrationTest, systemTest 등)에 돌린다. 테스트용 제외 목록은 `config/spotbugs/exclude-test.xml`(보안 범주·내부 상태 노출·생성자 예외·반환값 null 역참조·아키텍처 위반 fixture). 분석 classpath가 그 source set의 runtime classpath로 넓어졌고, 분석이 불완전하면(빠진 클래스) 실패한다. main 빌드는 `mainCheck`(`qualificationCheck`), optional 빌드는 `ci` | 자기 테스트 코드의 SpotBugs 발견(예: `CountDownLatch.await` 반환값 무시, 공유 변수 비원자 쓰기, 닫지 않은 `Statement`)이 main 단계를 막는다. 테스트 classpath에 없는 타입을 참조하면 `SpotBugs analysis incomplete` | 고치거나, 클래스·메서드로 좁힌 Match를 이유와 함께 `exclude-test.xml`에 더한다. 빠진 타입은 그 테스트 suite 의존성에 넣는다 | `2678955c` · `src/build-logic/.../quality/QualityConventionsPlugin.java`, `src/config/spotbugs/exclude-test.xml`, `src/build.gradle` | @@ -141,6 +142,13 @@ 파일의 `SET LOCAL`을 상한으로 세지 않는다. 자기 스트림의 이미 배포된 파일이 새로 걸리면 그 파일은 고치지 않고 규칙 이전 목록에 더한다(템플릿은 idempotency V1, notification V3, messaging V1003, postgresql V11·V12를 더했다). 파일 경로를 넘기는 `unboundedLockingStatements(Path)`가 옆의 `.sql.conf`까지 읽으니, 복사한 검사는 문자열 대신 경로를 넘긴다. + r9부터 검사기는 Flyway가 `.sql.conf` 없이 스스로 트랜잭션 밖에서 돌리는 문장(`VACUUM`, `CREATE`·`DROP INDEX CONCURRENTLY`, + `REINDEX SCHEMA` 등)이 있는 파일도 트랜잭션 밖으로 보고 그 파일의 `SET LOCAL`을 상한으로 세지 않으며(그런 파일은 다른 문장과 + 섞이면 Flyway가 기본 `mixed=false`로 거부한다), `VACUUM FULL`은 "마이그레이션에 두지 말고 운영 절차로" 거부한다. + `.sql.conf`는 Flyway처럼 properties로 읽어 `executeInTransaction=FALSE`·`executeInTransaction: false`도 트랜잭션 밖으로 본다. + `GRANT … REFERENCES ON`과 `REINDEX (CONCURRENTLY) …`는 더 이상 거부하지 않는다. 자기 스트림 검사가 새로 실패하면 그 파일은 + Flyway가 거부하거나(`mixed=false`) 상한 없이 도는(`mixed=true`) 파일이다. 아직 배포 전이면 문장을 나누고, 이미 배포했으면 + 규칙 이전 목록에 더한다(r9, 재리뷰 RR4-02-01). - **번호 충돌**: 템플릿이 이제 postgresql V20~V22, notification V11~V13, sample V8_1, grpc V002·V003을 쓴다. 파생 서비스가 같은 위치에 같은 번호를 두었다면 Flyway가 같은 버전 두 개로 실패한다. 자기 마이그레이션은 템플릿과 떨어진 번호대로 옮긴다 (이미 적용된 파일 이름을 바꾸면 checksum·history가 어긋나므로 운영 DB에 적용된 것이면 DB 담당과 정한다). @@ -224,6 +232,9 @@ V21 실패의 분류는 `docs/runbooks/migration-failed.md` "Classify the failur | HTTP client(JDK transport) | 스트리밍 응답의 read-idle 감시가 전용 데몬 스레드 하나(`httpclient-jdk-read-idle`)에서 돈다. 스트림마다 감시 작업 하나, 스트림을 닫으면 작업을 바로 취소 | 스레드 덤프에 그 스레드가 하나 보인다. 공용 `ForkJoinPool`이 바빠도 timeout이 설정값에 난다(전에는 늦게 났다). Apache transport(기본)는 영향 없음 | 3차 확인 리뷰 RR3-03-01 · `src/adapter/outbound/httpclient/.../jdk/ReadIdleInputStream.java` | | 멱등성 | `IDEMPOTENCY_RECOVERY_REQUIRED`를 내는 예외 메시지 끝에 scope digest가 붙는다. Redis 기록 조회(INSPECT 스크립트)가 `leaseUntil`을 돌려줘 `processingLeaseUntil`이 실제 lease 끝이다(전에는 key TTL 끝, 즉 lease + replay TTL) | 예외 메시지 전체에 맞추던 로그 매처는 고친다. 회복 절차는 `docs/runbooks/idempotency-unavailable.md`(digest로 기록을 찾는다) | 3차 확인 리뷰 RR3-03-03·05 · `src/application-core/.../idempotency/IdempotencyRecoveryRequiredException.java`, `src/adapter/outbound/cache-redis/.../idempotency/IdempotencyScripts.java` | | 멱등성(lease가 지난 실행) | lease가 지난 `EXECUTING` 기록을 claim이 만나면 기록을 바꾸지 않고 500 `IDEMPOTENCY_RECOVERY_REQUIRED`로 답한다. 그 실행의 주인이 늦게 부른 `complete`·`markFailed`는 기록된다. 전에는 두 저장소 모두 기록을 `ABANDONED`로 바꿔(PostgreSQL은 `last_transition_kind` `EXPIRED_EXECUTION`) 늦은 결과를 거절했고 첫 요청도 500이었다. 이미 끝난 lease의 `renew`는 두 저장소 모두 거절한다(Redis는 전에 받아들였다). 남의 lease 지난 실행을 `inspect`하면 `ABANDONED`(전에는 `IN_PROGRESS_OTHER`) | 30s를 넘는 정상 요청이 이제 결과를 남기고 정상 응답을 받는다. 같은 키 재전송은 500을 받은 뒤 재생을 받을 수 있다. 직접 구현한 store는 `IdempotencyStoreContract`의 바뀐 시간 케이스로 확인한다. 운영자는 `EXECUTING` 기록을 지우지 않는다 — 살아 있는 실행의 늦은 결과가 기록을 못 찾아(`ABSENT`) 사라진다(`docs/runbooks/idempotency-unavailable.md`) | `70fb80bf`(2026-11-04 소유자 결정) · `src/application-core/src/testFixtures/.../IdempotencyStoreContract.java`, `src/adapter/outbound/cache-redis/.../idempotency/IdempotencyScripts.java`, `src/adapter/outbound/persistence-jpa/.../postgresql/PostgreSqlIdempotencyStore.java` | +| 멱등성(lease가 끝난 시작, r9) | lease가 지난 `CLAIMED`는 주인도 시작할 수 없다(`LEASE_EXPIRED`). executor는 작업을 돌리지 않고 503 `IDEMPOTENCY_UNAVAILABLE`(실행 전 거부, 메시지 `the claim's processing lease ran out before the action started`)로 답하고, 다음 같은 키 요청이 그 claim을 넘겨받아 실행한다. 전에는 Redis가 시작해 작업을 돌렸고 그동안 다른 요청은 lease가 지난 실행으로 500을 받았다. PostgreSQL은 `NOT_OWNER`로 거절해 500 `IDEMPOTENCY_RECOVERY_REQUIRED`였다. | claim과 시작 사이가 lease(기본 30s)보다 길었던 드문 요청이 500 대신 503(같은 키 재전송)을 받는다. 클라이언트는 `error.code`로 이미 가르므로 바꿀 것이 없다. 운영자는 이 503에서 저장소 장애를 찾지 않는다(runbook) | r9(재리뷰 README 15 남음, RR5-01-02·03) · `IdempotencyExecutor.java`, `src/adapter/outbound/cache-redis/.../idempotency/IdempotencyScripts.java`, `src/adapter/outbound/persistence-jpa/.../postgresql/PostgreSqlIdempotencyStore.java` | +| 빌드 단계(HTTP client) | `httpClientFailureInjectionTest`(Toxiproxy 장애 주입)가 `mainCheck`에서 빠졌다. `integrationCheck`에만 있고, main push·nightly의 `integration-main.yml` `integration-check` job이 돌린다 | main push마다 같은 lane이 `main-check`·`integration-check` 두 job에서 두 번 돌던 것이 한 번이 된다. 로컬에서 `mainCheck`만 돌리면 이 lane은 돌지 않는다(`integrationCheck`로 돌린다) | r9 · `src/adapter/outbound/httpclient/build.gradle` | +| 서버 로그(GraphQL·gRPC·STOMP) | carrier로 답한 실패를 web과 같은 줄(shared-contract `ServerFailureLog`)로 ERROR에 남긴다: persistence 실패는 분류만, 5xx carrier는 예외와 함께, 4xx carrier는 남기지 않는다. gRPC는 ` (method=…)`, GraphQL은 ` (executionId=…)`를 붙인다. gRPC는 인식 못 한 실패(`INTERNAL_ERROR`)도 `unhandled failure on …`로 남긴다(클라이언트 취소 제외). STOMP `@MessageMapping` 실패는 Spring의 `Unhandled exception from message handler method`(예외 전체) 대신 같은 줄로 남는다 | ERROR 로그가 늘어난다(전에는 gRPC가 아무것도 남기지 않았다). STOMP에서 Spring 문구로 경보를 걸었다면 고친다. `IDEMPOTENCY_RECOVERY_REQUIRED`의 scope digest를 모든 전송에서 찾을 수 있다(`docs/runbooks/idempotency-unavailable.md`) | r9 · `src/shared-contract/.../error/ServerFailureLog.java`, `GrpcExceptionHandlingInterceptor.java`, `GraphQlWireErrorMapper.java`, `StompHandlerFailureLog.java` | | sample-portfolio | 익명 쓰기 401, 포스터 제목 대소문자 무시 유일(409 `POSTER_ALREADY_EXISTS`), 관리자도 남의 work log 수정 불가, 샘플 이벤트 tenant `global` | sample을 남긴 서비스만 | `34565130`, `1400dc15` | ## 6. 따라오는 순서 diff --git a/docs/web/http-contract.md b/docs/web/http-contract.md index 2fdd0868..67cb9f91 100644 --- a/docs/web/http-contract.md +++ b/docs/web/http-contract.md @@ -86,7 +86,7 @@ status는 코드 표(`HttpStatusErrorCodes`)에 있는 것만 그대로 나간 | 경우 | status | `error.code` | `retryable` | `Retry-After` | 클라이언트 행동 | | --- | --- | --- | --- | --- | --- | | 재생(replay): 같은 키·같은 본문, 첫 요청 완료 | 첫 요청과 같은 성공 status | — (성공 Envelope) | — | 없음 | 첫 요청의 결과로 받아들인다. `meta`의 id는 이번 요청의 것이다 | -| 처리 중(in-flight): 같은 키, 첫 요청이 아직 실행 중 | 409 | `IDEMPOTENT_IN_FLIGHT` | `true` | `2` | 결과 불명. **같은 `Idempotency-Key`로** `Retry-After` 뒤 재전송한다. **새 키 금지** | +| 처리 중(in-flight): 같은 키, 첫 요청이 아직 실행 중(Redis 저장소. PostgreSQL 저장소는 409 대신 첫 요청이 끝날 때까지 기다린다 — 아래 "첫 시도가 lease보다 오래 걸릴 때") | 409 | `IDEMPOTENT_IN_FLIGHT` | `true` | `2` | 결과 불명. **같은 `Idempotency-Key`로** `Retry-After` 뒤 재전송한다. **새 키 금지** | | 본문 불일치(mismatch): 같은 키·다른 본문 | 422 | `IDEMPOTENT_REQUEST_MISMATCH` | `false` | 없음 | 클라이언트 버그. 다시 보내지 않는다. 새 작업이면 새 키를 만든다 | | 키 누락·형식 오류 | 400 | `IDEMPOTENCY_KEY_INVALID` (`details.reason` = `MISSING`/`MALFORMED`) | `false` | 없음 | 문법에 맞는 키를 붙여 다시 보낸다 | @@ -108,7 +108,7 @@ status는 코드 표(`HttpStatusErrorCodes`)에 있는 것만 그대로 나간 | 경우 | status | `error.code` | `retryable` | `Retry-After` | 클라이언트 행동 | | --- | --- | --- | --- | --- | --- | -| 작업 실행 전(claim·시작 단계) 멱등성 저장소 응답 없음 | 503 | `IDEMPOTENCY_UNAVAILABLE` | `true` | 없음 | 작업은 실행되지 않았다. 같은 키로 나중에 다시 보낸다 | +| 작업 실행 전(claim·시작 단계) 멱등성 저장소 응답 없음. 또는 claim의 processing lease가 작업 시작 전에 끝나 저장소가 시작을 거절함 | 503 | `IDEMPOTENCY_UNAVAILABLE` | `true` | 없음 | 작업은 실행되지 않았다. 같은 키로 나중에 다시 보낸다(다음 요청이 그 claim을 넘겨받아 실행한다) | | 작업 실행 뒤 결과 기록(complete 단계) 중 저장소 응답 없음 — **결과 불명** | 503 | `IDEMPOTENCY_COMPLETION_UNKNOWN` | `true` | 없음 | "적용되지 않았다"고 알리지 않는다. **같은 키로** 나중에 다시 보낸다. 새 키 금지 | | 이전 시도의 효과를 증명할 수 없음 | 500 | `IDEMPOTENCY_RECOVERY_REQUIRED` | `false` | 없음 | 자동 재시도하지 않는다. 같은 키 재전송은 안전하다 — 서버는 효과가 있었을 수 있는 작업을 다시 돌리지 않고, 첫 시도가 늦게 끝났으면 그 결과를 재생하며, 효과 없이 끝났으면 실행하고, 아니면 다시 500이다. 결과는 업무 데이터로 확인하고, 효과가 없다고 확인된 뒤에만 운영 절차(`docs/runbooks/idempotency-unavailable.md`)에 따라 새 키로 다시 보낸다 | @@ -118,15 +118,27 @@ status는 코드 표(`HttpStatusErrorCodes`)에 있는 것만 그대로 나간 503이어도 code가 다르다. 클라이언트는 status가 아니라 `error.code`로 가른다. - **같은 키로 다시 보내면 서버가 판정하고, 작업을 다시 돌리지 않는다.** 결과가 실제로는 기록됐으면 재생(첫 요청의 성공 응답), 첫 시도의 processing lease(`processing-lease`, 기본 30s)가 아직 남았으면 409 - `IDEMPOTENT_IN_FLIGHT`, lease가 지났으면 500 `IDEMPOTENCY_RECOVERY_REQUIRED`다. 새 키로 보내면 서버는 다른 - 요청으로 보고 막지 못한다(재리뷰 2026-11-03 README 4절 결정 RR-02-03, 2026-11-04 9.1절 결정 RR-01-11). -- **첫 시도가 lease보다 오래 걸려도 그 결과는 기록된다.** executor는 lease를 늘리지 않으므로 30s를 넘는 - 정상 요청도 lease가 지난다. lease가 지난 실행 기록을 다음 요청이 만나면 두 저장소(PostgreSQL·Redis) 모두 - 500으로 답하고 기록은 건드리지 않는다. 첫 시도가 그 뒤에 끝나면 결과가 그대로 기록되고, 첫 요청은 자기 - 응답을 정상으로 받는다. 그 뒤 같은 키 요청은 그 결과를 재생한다. 첫 시도가 실패를 보고하면 그 실패가 - 기록된다(효과 불명이면 계속 500, 효과 없는 재시도 가능 실패면 다음 같은 키 요청이 다시 실행한다). 기록이 - 이미 사라진 뒤에 끝나면 결과를 기록하지 못해 첫 요청도 500 `IDEMPOTENCY_RECOVERY_REQUIRED`를 받는다 - (2026-11-04 소유자 결정). + `IDEMPOTENT_IN_FLIGHT`, lease가 지났으면 500 `IDEMPOTENCY_RECOVERY_REQUIRED`다. 이것은 Redis 저장소의 + 판정이다. PostgreSQL 저장소는 아래처럼 첫 요청이 끝날 때까지 기다린 뒤 판정한다. 새 키로 보내면 서버는 + 다른 요청으로 보고 막지 못한다(재리뷰 2026-11-03 README 4절 결정 RR-02-03, 2026-11-04 9.1절 결정 + RR-01-11). +- **첫 시도가 lease보다 오래 걸릴 때 — 저장소마다 다르다.** executor는 lease를 늘리지 않으므로 30s를 넘는 + 정상 요청도 lease가 지난다(재리뷰 2026-11-04 RR5-01-05). + - Redis: lease가 지난 실행 기록을 다음 요청이 만나면 500으로 답하고 기록은 건드리지 않는다. 첫 시도가 + 그 뒤에 끝나면 결과가 그대로 기록되고, 첫 요청은 자기 응답을 정상으로 받는다. 그 뒤 같은 키 요청은 그 + 결과를 재생한다. 첫 시도가 실패를 보고하면 먼저 온 보고 하나가 기록된다(효과 불명이면 계속 500, 효과 + 없는 재시도 가능 실패면 다음 같은 키 요청이 다시 실행한다). 기록이 이미 사라진 뒤에 끝나면 결과를 + 기록하지 못해 첫 요청도 500 `IDEMPOTENCY_RECOVERY_REQUIRED`를 받는다(2026-11-04 소유자 결정). + - PostgreSQL: 템플릿처럼 executor를 유스케이스의 쓰기 트랜잭션 하나 안에서 부르면 claim 기록은 그 + 트랜잭션이 끝날 때 함께 커밋되거나 롤백된다. 그래서 같은 키 요청은 409나 500을 받지 않고 그 기록에서 + 기다린다. 첫 트랜잭션이 커밋되면 재생을, 롤백되면 새 실행을 받는다. lease가 지나도 같다. 작업이 실패해 + 예외로 끝난 요청은 executor가 남긴 실패 기록도 함께 롤백되므로, 효과 불명 실패도 다음 같은 키 요청이 + 다시 실행한다. 또 쓰기 트랜잭션 제한(`app.persistence-jpa.transaction.write-timeout`, 기본 30s — 기본 + lease와 같다, 업그레이드 안내 B-21)이 두 요청에 모두 걸린다. 첫 요청이 그보다 오래 걸리면 취소되고 claim + 기록도 함께 롤백된다(503 `DB_QUERY_CANCELED`). 기다리던 요청도 자기 제한을 넘기면 같은 코드로 끝난다. + 어느 경우든 롤백된 DB 변경은 남지 않지만, 트랜잭션 밖 효과(다른 시스템 호출, 메시지)는 다음 같은 키 + 요청에서 두 번 일어날 수 있다. 이 저장소가 막는 중복은 같은 트랜잭션 안의 DB 효과다 + (`PostgreSqlIdempotencyIntegrationTest`). - **서버가 기억하는 기간.** 위 판정은 기록이 남아 있는 동안만 선다. 기록이 사라지면 같은 키는 새 요청으로 실행된다. 기록의 수명은 시도가 어떻게 끝났는지로 갈린다. - 완료된 응답: 완료부터 `replay-ttl`(`APP_IDEMPOTENCY_TTL`, 기본 24h). @@ -134,7 +146,8 @@ status는 코드 표(`HttpStatusErrorCodes`)에 있는 것만 그대로 나간 쓴다. 그 사이 첫 시도가 끝나면 그때부터 위(완료)나 아래(실패)의 수명으로 바뀐다. - 실패로 표시된 시도: 작업이 예외를 던졌거나 효과 불명으로 끝나 포기(abandoned)된 시도, 효과 없이 재시도 가능으로 끝난 시도. 표시한 때부터 `failure-retention`(`APP_IDEMPOTENCY_FAILURE_RETENTION`, - 기본 24h). + 기본 24h). PostgreSQL 저장소를 쓰기 트랜잭션 하나 안에서 쓰면 실패 기록은 요청의 롤백과 함께 사라져 + 남지 않는다(위 PostgreSQL 항목). `replay-ttl`이 `processing-lease`보다 짧으면 기동이 실패한다(재리뷰 2026-11-04 RR-03-11). `failure-retention`이 `replay-ttl`보다 짧아도 기동이 실패한다. 더 짧으면 효과 불명으로 포기한 기록이 그만큼 diff --git a/src/.env.example b/src/.env.example index 25b09cca..9ba52278 100644 --- a/src/.env.example +++ b/src/.env.example @@ -121,7 +121,8 @@ APP_RATE_LIMIT_REFILL_TOKENS=100 APP_RATE_LIMIT_REFILL_PERIOD=1s APP_RATE_LIMIT_MAXIMUM_COST=10 APP_RATE_LIMIT_CLEANUP_GRACE=5s -# Replay TTL: at most 72h and not shorter than APP_IDEMPOTENCY_PROCESSING_LEASE (startup fails). +# Replay TTL: at most 72h, not shorter than APP_IDEMPOTENCY_PROCESSING_LEASE and not longer than +# APP_IDEMPOTENCY_FAILURE_RETENTION (startup fails) — raising it past 24h means raising that too. APP_IDEMPOTENCY_TTL=24h # disabled | redis | postgresql. postgresql refuses a non-PostgreSQL data source, so bootRunH2 # runs with disabled or redis. diff --git a/src/README.md b/src/README.md index a1ffcf78..c49fcb98 100644 --- a/src/README.md +++ b/src/README.md @@ -28,7 +28,7 @@ Checkstyle은 **포매터가 표현할 수 없는 것만** — 네이밍과 몇 | `qualificationCheck` | 전체 기동(system) 레인과 보조 source set Checkstyle을 실행 | | `integrationCheck` | 선언된 strict integration lane(sample 포함)을 실행. 대부분 container runtime이 필요. 입력이 필요한 lane(Redis topology: `-Predis.topology.mode`, AWS: 보호 자격 증명 env)은 입력이 없으면 빠지고 집계가 그 이유를 출력 | | `performanceCheck` | 선언된 strict performance lane(부하·종료·pool·자원 상한)을 실행. 실서버·container가 필요. `integration-main.yml` `performance-check`가 main·nightly에 돌림 | -| `verifyStrictLaneStages` | `--dry-run` 전용. `explicitOnly`가 아닌 strict lane이 workflow가 실제 실행하는 단계(`ci`·`:sample-portfolio:check`·`mainCheck`·`integrationCheck`·`performanceCheck`·`releaseCheck`) 중 하나의 계획에 있는지, 각 단계를 workflow가 `--dry-run` 없이 실행하는지 검사. PR 게이트가 돌림 | +| `verifyStrictLaneStages` | `--dry-run` 전용. `explicitOnly`가 아닌 strict lane이 workflow가 실제 실행하는 단계(`ci`·`:sample-portfolio:check`·`mainCheck`·`integrationCheck`·`performanceCheck`·`releaseCheck`) 중 하나의 계획에 있는지, 각 단계를 workflow가 `--dry-run` 없이 실행하는지 검사. 실패해도 단계를 실패시키지 못하는 명령(`\|\| true`, pipefail 없는 파이프, `if` 조건 등)은 실행으로 세지 않는다. PR 게이트가 돌림 | | `ci` | PR 게이트: 모든 leaf `check`(API surface 검사 포함, 소유자 결정 20) + `architectureCheck` + `qualityCheck` | | `mainCheck` | main 단계: `configContractCheck` + `qualificationCheck` + sample web 계약 + sample 전체 기동(`:sample-portfolio:systemTest`) + `verifyPublicPathSnapshot` + 전송·메시징 qualification 레인 + 각 leaf의 `mainCheck`(문서 대조 레인 `*DocumentContractTest`, objectstorage MinIO 레인) | | `releaseCheck` | tag 전용 검사: `verifyReleaseProvenance` + `verifyPublicPathSnapshot`(`ci`를 다시 돌리지 않음) | @@ -237,7 +237,8 @@ drift하기 시작하면 그때 자동화합니다. 진짜 항목을 상대로, `disabled` 나 `redis` 를 씁니다. - **`APP_IDEMPOTENCY_TTL`** — 완료 응답을 재생할 수 있는 기간(`replay-ttl`). duration(예: `24h`). 최대 72h 이고 넘으면 기동을 거부합니다. (D6) `APP_IDEMPOTENCY_PROCESSING_LEASE`(기본 30s)보다 - 짧아도 기동을 거부합니다. + 짧아도, `APP_IDEMPOTENCY_FAILURE_RETENTION`(기본 24h)보다 길어도 기동을 거부합니다. 24h보다 + 올리려면 `APP_IDEMPOTENCY_FAILURE_RETENTION`도 함께 올립니다. ### Async executor diff --git a/src/adapter/inbound/graphql/src/main/java/dev/caskeleton/adapter/inbound/graphql/error/GraphQlWireErrorMapper.java b/src/adapter/inbound/graphql/src/main/java/dev/caskeleton/adapter/inbound/graphql/error/GraphQlWireErrorMapper.java index d0740657..8497dcff 100644 --- a/src/adapter/inbound/graphql/src/main/java/dev/caskeleton/adapter/inbound/graphql/error/GraphQlWireErrorMapper.java +++ b/src/adapter/inbound/graphql/src/main/java/dev/caskeleton/adapter/inbound/graphql/error/GraphQlWireErrorMapper.java @@ -5,11 +5,14 @@ import dev.caskeleton.shared.error.ApiErrorCode; import dev.caskeleton.shared.error.Category; import dev.caskeleton.shared.error.ClientErrorMessages; import dev.caskeleton.shared.error.OperationalError; +import dev.caskeleton.shared.error.ServerFailureLog; import graphql.ErrorClassification; import graphql.GraphQLError; import java.util.EnumMap; import java.util.Map; import java.util.Objects; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import org.springframework.graphql.execution.ErrorType; /** @@ -33,9 +36,16 @@ import org.springframework.graphql.execution.ErrorType; *

An unrecognised failure never contributes its own message. That is the fail-closed direction: * a new exception type nobody mapped becomes an opaque internal error with a correlation id, and * the detail goes to the log under that id ({@link GraphQlExceptionResolver}). + * + *

A carrier is logged here, with {@link ServerFailureLog}'s line — the one the web and gRPC + * adapters write — plus the execution id the client received. A server-side code therefore leaves + * its detail in the log, such as the scope digest of an {@code IDEMPOTENCY_RECOVERY_REQUIRED} + * (review 2026-11-04 RR4-03-03); a persistence failure leaves its classification only. */ public final class GraphQlWireErrorMapper { + private static final Logger LOG = LoggerFactory.getLogger(GraphQlWireErrorMapper.class); + /** How one shared category is rendered on the GraphQL wire. */ private record Rendering(GraphQlErrorCategory wireCategory, ErrorClassification classification) {} @@ -104,6 +114,11 @@ public final class GraphQlWireErrorMapper { } ApiErrorCode shared = sharedCodeOf(failure); if (shared != null) { + ServerFailureLog.of(failure, shared) + .ifPresent( + line -> + LOG.error( + line.format(" (executionId={})"), line.arguments(context.executionId()))); return fromApiError(shared, context); } return applicationMappings.resolve(failure, context); diff --git a/src/adapter/inbound/graphql/src/test/java/dev/caskeleton/adapter/inbound/graphql/error/GraphQlWireErrorMapperTest.java b/src/adapter/inbound/graphql/src/test/java/dev/caskeleton/adapter/inbound/graphql/error/GraphQlWireErrorMapperTest.java index c27ccb56..aa186249 100644 --- a/src/adapter/inbound/graphql/src/test/java/dev/caskeleton/adapter/inbound/graphql/error/GraphQlWireErrorMapperTest.java +++ b/src/adapter/inbound/graphql/src/test/java/dev/caskeleton/adapter/inbound/graphql/error/GraphQlWireErrorMapperTest.java @@ -2,14 +2,24 @@ package dev.caskeleton.adapter.inbound.graphql.error; import static org.assertj.core.api.Assertions.assertThat; +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.Logger; +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; import dev.caskeleton.adapter.inbound.graphql.dataloader.GraphQlBatchTimeoutException; import dev.caskeleton.adapter.inbound.graphql.dataloader.GraphQlMissingKeyException; import dev.caskeleton.adapter.inbound.graphql.execution.GraphQlRequestCancelledException; +import dev.caskeleton.application.idempotency.IdempotencyRecoveryRequiredException; +import dev.caskeleton.shared.error.OperationalError; +import dev.caskeleton.shared.error.PersistenceFailureException; +import java.sql.SQLException; +import java.util.List; import java.util.stream.Stream; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.Arguments; import org.junit.jupiter.params.provider.MethodSource; +import org.slf4j.LoggerFactory; class GraphQlWireErrorMapperTest { @@ -128,4 +138,84 @@ class GraphQlWireErrorMapperTest { extensions -> extensions.get("retryable")) .containsExactly("DEPENDENCY_TIMEOUT", "DEPENDENCY", true); } + + @org.junit.jupiter.api.Test + void aServerSideCarrierIsLoggedAtErrorWithTheExecutionIdAndTheFailureThatNamesTheScopeDigest() { + // given + GraphQlWireErrorMapper mapper = GraphQlWireErrorMapper.masking(); + RuntimeException failure = + new IdempotencyRecoveryRequiredException( + "claim requires reconciliation: CHARGE_CARD::v1::3f9a"); + + // when + List logged = + loggedWhile(() -> mapper.toGraphQlError(failure, GraphQlErrorContext.of("exec-9"))); + + // then + assertThat(logged) + .singleElement() + .extracting( + ILoggingEvent::getLevel, + ILoggingEvent::getFormattedMessage, + event -> event.getThrowableProxy().getMessage()) + .containsExactly( + Level.ERROR, + "failure classified as IDEMPOTENCY_RECOVERY_REQUIRED (category=DATA_INTEGRITY," + + " retryable=false) (executionId=exec-9)", + "claim requires reconciliation: CHARGE_CARD::v1::3f9a"); + } + + @org.junit.jupiter.api.Test + void aPersistenceFailureIsLoggedByItsClassificationWithoutItsCause() { + // given + GraphQlWireErrorMapper mapper = GraphQlWireErrorMapper.masking(); + RuntimeException failure = + new PersistenceFailureException( + OperationalError.DB_UNAVAILABLE, + "08006", + new SQLException("jdbc:postgresql://db.internal/customer?password=secret", "08006")); + + // when + List logged = + loggedWhile(() -> mapper.map(failure, GraphQlErrorContext.of("exec-9"))); + + // then + assertThat(logged) + .singleElement() + .extracting(ILoggingEvent::getFormattedMessage, ILoggingEvent::getThrowableProxy) + .containsExactly( + "persistence failure classified as DB_UNAVAILABLE (category=TRANSIENT_DEPENDENCY," + + " retryable=true) (executionId=exec-9)", + null); + } + + @org.junit.jupiter.api.Test + void aClientSideCarrierIsNotLogged() { + // given + GraphQlWireErrorMapper mapper = GraphQlWireErrorMapper.masking(); + + // when + List logged = + loggedWhile( + () -> + mapper.map( + new IllegalArgumentException("bad cursor"), GraphQlErrorContext.of("exec-9"))); + + // then + assertThat(logged).isEmpty(); + } + + private static List loggedWhile(Runnable action) { + Logger logger = (Logger) LoggerFactory.getLogger(GraphQlWireErrorMapper.class); + ListAppender appender = new ListAppender<>(); + appender.start(); + logger.addAppender(appender); + try { + action.run(); + } finally { + logger.detachAppender(appender); + appender.stop(); + } + return appender.list; + } } diff --git a/src/adapter/inbound/grpc/README.md b/src/adapter/inbound/grpc/README.md index 80201aa2..50c7e389 100644 --- a/src/adapter/inbound/grpc/README.md +++ b/src/adapter/inbound/grpc/README.md @@ -151,6 +151,12 @@ handler throw, listener callback throw, `responseObserver.onError(...)`, raw 예외 메시지(SQLState/업스트림 세부를 담을 수 있음)는 절대 클라이언트에 내보내지 않는다. 인식되지 않은 예외와 raw gRPC status/description은 원래 status를 신뢰하지 않고 `Status.INTERNAL` + `INTERNAL_ERROR`로 폴백한다. 입력 trailers도 폐기한다. +- **서버 로그**: carrier는 web과 같은 줄(shared-contract `ServerFailureLog`)에 ` (method=<서비스/메서드>)`를 + 붙여 ERROR로 남긴다. persistence 실패는 분류만, 5xx carrier는 예외와 함께, 4xx carrier는 남기지 않는다. + 인식되지 않아 `INTERNAL_ERROR`로 답한 실패는 `unhandled failure on answered as INTERNAL_ERROR + (status=)`에 원인 예외를 붙인다. raw status만 있으면 description은 남기지 않는다(입력이 들 수 + 있다). 클라이언트가 취소한 호출(`CANCELLED`)은 남기지 않는다. 그래서 `IDEMPOTENCY_RECOVERY_REQUIRED`도 + web처럼 scope digest를 로그에 남긴다(재리뷰 RR4-03-03, `docs/runbooks/idempotency-unavailable.md`). `Category → Status` 표(설계 스펙 Error Mapping SSOT): diff --git a/src/adapter/inbound/grpc/src/main/java/dev/caskeleton/adapter/inbound/grpc/GrpcExceptionHandlingInterceptor.java b/src/adapter/inbound/grpc/src/main/java/dev/caskeleton/adapter/inbound/grpc/GrpcExceptionHandlingInterceptor.java index d133fbe3..fcfd8ebd 100644 --- a/src/adapter/inbound/grpc/src/main/java/dev/caskeleton/adapter/inbound/grpc/GrpcExceptionHandlingInterceptor.java +++ b/src/adapter/inbound/grpc/src/main/java/dev/caskeleton/adapter/inbound/grpc/GrpcExceptionHandlingInterceptor.java @@ -3,6 +3,7 @@ package dev.caskeleton.adapter.inbound.grpc; import dev.caskeleton.shared.error.ApiErrorCarrier; import dev.caskeleton.shared.error.ApiErrorCode; import dev.caskeleton.shared.error.OperationalError; +import dev.caskeleton.shared.error.ServerFailureLog; import io.grpc.ForwardingServerCall.SimpleForwardingServerCall; import io.grpc.ForwardingServerCallListener.SimpleForwardingServerCallListener; import io.grpc.Metadata; @@ -11,6 +12,8 @@ import io.grpc.ServerCallHandler; import io.grpc.ServerInterceptor; import io.grpc.Status; import java.util.concurrent.atomic.AtomicBoolean; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; /** * Centralises the gRPC error contract: a feature service just throws or calls {@code onError}, and @@ -29,9 +32,20 @@ import java.util.concurrent.atomic.AtomicBoolean; * client sees the stable code, category and retryability in the trailers and the shared category * message as the status description — raw descriptions and input trailers, which may carry a * SQLState or upstream detail, are never surfaced. + * + *

What the client cannot see goes to the server log, as on the web: a carrier is logged with + * {@link ServerFailureLog}'s line, the one the web and GraphQL adapters write, plus the method — so + * an {@code IDEMPOTENCY_RECOVERY_REQUIRED} answer leaves the scope digest the runbook looks the + * record up by ({@code docs/runbooks/idempotency-unavailable.md}, review 2026-11-04 RR4-03-03). An + * unrecognised failure answered as {@code INTERNAL_ERROR} is logged with its exception, or by its + * status code alone when the service closed with a raw status — its description may echo input. A + * call the client cancelled is not logged: nobody is waiting for the answer, and nothing failed on + * the server. */ public class GrpcExceptionHandlingInterceptor implements ServerInterceptor { + private static final Logger log = LoggerFactory.getLogger(GrpcExceptionHandlingInterceptor.class); + private final GrpcStatusMapper statusMapper; public GrpcExceptionHandlingInterceptor(GrpcStatusMapper statusMapper) { @@ -107,12 +121,18 @@ public class GrpcExceptionHandlingInterceptor implements ServerInterceptor { return; } - ApiErrorCode code = errorCodeOf(status.getCause()); - if (code == null) { - code = - status.getCause() instanceof IllegalArgumentException - ? OperationalError.BAD_PARAMETER - : OperationalError.INTERNAL_ERROR; + String method = delegate.getMethodDescriptor().getFullMethodName(); + Throwable carrier = carrierOf(status.getCause()); + ApiErrorCode code; + if (carrier != null) { + code = ((ApiErrorCarrier) carrier).errorCode(); + ServerFailureLog.of(carrier, code) + .ifPresent(line -> log.error(line.format(" (method={})"), line.arguments(method))); + } else if (status.getCause() instanceof IllegalArgumentException) { + code = OperationalError.BAD_PARAMETER; + } else { + code = OperationalError.INTERNAL_ERROR; + logUnrecognised(method, status); } super.close(statusMapper.statusFor(code), statusMapper.trailersFor(code)); } @@ -123,11 +143,26 @@ public class GrpcExceptionHandlingInterceptor implements ServerInterceptor { call.close(Status.fromThrowable(exception).withCause(exception), new Metadata()); } - private static ApiErrorCode errorCodeOf(Throwable throwable) { + private static void logUnrecognised(String method, Status status) { + if (status.getCode() == Status.Code.CANCELLED) { + return; + } + // The cause, when there is one, is the last argument and so the logged exception; the raw + // description is never logged. + log.error( + "unhandled failure on {} answered as {} (status={})", + method, + OperationalError.INTERNAL_ERROR.code(), + status.getCode(), + status.getCause()); + } + + /** The first {@link ApiErrorCarrier} in the cause chain, or {@code null}. */ + private static Throwable carrierOf(Throwable throwable) { Throwable current = throwable; while (current != null) { - if (current instanceof ApiErrorCarrier carrier) { - return carrier.errorCode(); + if (current instanceof ApiErrorCarrier) { + return current; } if (current.getCause() == current) { break; diff --git a/src/adapter/inbound/grpc/src/test/java/dev/caskeleton/adapter/inbound/grpc/GrpcExceptionHandlingInterceptorTest.java b/src/adapter/inbound/grpc/src/test/java/dev/caskeleton/adapter/inbound/grpc/GrpcExceptionHandlingInterceptorTest.java index 4ffdbed0..918d48f9 100644 --- a/src/adapter/inbound/grpc/src/test/java/dev/caskeleton/adapter/inbound/grpc/GrpcExceptionHandlingInterceptorTest.java +++ b/src/adapter/inbound/grpc/src/test/java/dev/caskeleton/adapter/inbound/grpc/GrpcExceptionHandlingInterceptorTest.java @@ -2,6 +2,10 @@ package dev.caskeleton.adapter.inbound.grpc; import static org.assertj.core.api.Assertions.assertThat; +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.Logger; +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; import dev.caskeleton.shared.error.ApiErrorCarrier; import dev.caskeleton.shared.error.ApiErrorCode; import dev.caskeleton.shared.error.ClientErrorMessages; @@ -12,8 +16,11 @@ import io.grpc.MethodDescriptor; import io.grpc.ServerCall; import io.grpc.ServerCallHandler; import io.grpc.Status; +import java.sql.SQLException; +import java.util.List; import java.util.concurrent.atomic.AtomicBoolean; import org.junit.jupiter.api.Test; +import org.slf4j.LoggerFactory; /** * Verifies the interceptor translates a synchronous handler exception into a mapped {@code @@ -116,6 +123,138 @@ class GrpcExceptionHandlingInterceptorTest { assertThat(cancelled).isTrue(); } + @Test + void aServerSideCarrierIsLoggedAtErrorWithTheMethodAndTheFailureThatNamesTheScopeDigest() { + // given + CarrierException recoveryRequired = + new CarrierException( + OperationalError.IDEMPOTENCY_RECOVERY_REQUIRED, + "claim requires reconciliation: CHARGE_CARD::v1::3f9a"); + + // when + List logged = loggedWhile(() -> closeAfterThrowing(recoveryRequired)); + + // then + assertThat(logged) + .singleElement() + .extracting( + ILoggingEvent::getLevel, + ILoggingEvent::getFormattedMessage, + event -> event.getThrowableProxy().getMessage()) + .containsExactly( + Level.ERROR, + "failure classified as IDEMPOTENCY_RECOVERY_REQUIRED (category=DATA_INTEGRITY," + + " retryable=false) (method=test.Fixture/Call)", + "claim requires reconciliation: CHARGE_CARD::v1::3f9a"); + } + + @Test + void aPersistenceFailureIsLoggedByItsClassificationWithoutItsCause() { + // given + PersistenceFailureException failure = + new PersistenceFailureException( + OperationalError.DB_UNAVAILABLE, + "08006", + new SQLException("jdbc:postgresql://db.internal/customer?password=secret", "08006")); + + // when + List logged = loggedWhile(() -> closeAfterThrowing(failure)); + + // then + assertThat(logged) + .singleElement() + .extracting(ILoggingEvent::getFormattedMessage, ILoggingEvent::getThrowableProxy) + .containsExactly( + "persistence failure classified as DB_UNAVAILABLE (category=TRANSIENT_DEPENDENCY," + + " retryable=true) (method=test.Fixture/Call)", + null); + } + + @Test + void aClientSideCarrierIsNotLogged() { + // when + List logged = + loggedWhile(() -> closeAfterThrowing(new CarrierException(OperationalError.BAD_PARAMETER))); + + // then + assertThat(logged).isEmpty(); + } + + @Test + void anUnrecognisedFailureIsLoggedAtErrorWithTheFailure() { + // when + List logged = + loggedWhile(() -> closeAfterThrowing(new IllegalStateException("boom"))); + + // then + assertThat(logged) + .singleElement() + .extracting( + ILoggingEvent::getLevel, + ILoggingEvent::getFormattedMessage, + event -> event.getThrowableProxy().getClassName()) + .containsExactly( + Level.ERROR, + "unhandled failure on test.Fixture/Call answered as INTERNAL_ERROR (status=UNKNOWN)", + IllegalStateException.class.getName()); + } + + @Test + void aRawStatusWithoutACauseIsLoggedByItsCodeAlone() { + // given + CapturingServerCall call = new CapturingServerCall(); + ServerCallHandler handler = + (serverCall, headers) -> { + serverCall.close( + Status.NOT_FOUND.withDescription("row 42 of customer alice"), new Metadata()); + return new ServerCall.Listener<>() {}; + }; + + // when + List logged = + loggedWhile(() -> interceptor.interceptCall(call, new Metadata(), handler)); + + // then + assertThat(logged) + .singleElement() + .extracting(ILoggingEvent::getFormattedMessage, ILoggingEvent::getThrowableProxy) + .containsExactly( + "unhandled failure on test.Fixture/Call answered as INTERNAL_ERROR (status=NOT_FOUND)", + null); + } + + @Test + void aCallTheClientCancelledIsNotLogged() { + // given + CapturingServerCall call = new CapturingServerCall(); + ServerCallHandler handler = + (serverCall, headers) -> { + serverCall.close(Status.CANCELLED.withDescription("client cancelled"), new Metadata()); + return new ServerCall.Listener<>() {}; + }; + + // when + List logged = + loggedWhile(() -> interceptor.interceptCall(call, new Metadata(), handler)); + + // then + assertThat(logged).isEmpty(); + } + + private static List loggedWhile(Runnable action) { + Logger logger = (Logger) LoggerFactory.getLogger(GrpcExceptionHandlingInterceptor.class); + ListAppender appender = new ListAppender<>(); + appender.start(); + logger.addAppender(appender); + try { + action.run(); + } finally { + logger.detachAppender(appender); + appender.stop(); + } + return appender.list; + } + private CapturingServerCall closeAfterThrowing(RuntimeException thrown) { CapturingServerCall call = new CapturingServerCall(); ServerCallHandler handler = @@ -139,7 +278,11 @@ class GrpcExceptionHandlingInterceptorTest { private final ApiErrorCode errorCode; CarrierException(ApiErrorCode errorCode) { - super(errorCode.code()); + this(errorCode, errorCode.code()); + } + + CarrierException(ApiErrorCode errorCode, String message) { + super(message); this.errorCode = errorCode; } @@ -151,6 +294,26 @@ class GrpcExceptionHandlingInterceptorTest { /** Minimal {@link ServerCall} that records the {@code close(status, trailers)} arguments. */ private static final class CapturingServerCall extends ServerCall { + private static final MethodDescriptor.Marshaller NO_WIRE = + new MethodDescriptor.Marshaller<>() { + @Override + public java.io.InputStream stream(String value) { + throw new UnsupportedOperationException("no wire in this test"); + } + + @Override + public String parse(java.io.InputStream stream) { + throw new UnsupportedOperationException("no wire in this test"); + } + }; + private static final MethodDescriptor METHOD = + MethodDescriptor.newBuilder() + .setType(MethodDescriptor.MethodType.UNARY) + .setFullMethodName("test.Fixture/Call") + .setRequestMarshaller(NO_WIRE) + .setResponseMarshaller(NO_WIRE) + .build(); + private Status status; private Metadata trailers; @@ -176,7 +339,7 @@ class GrpcExceptionHandlingInterceptorTest { @Override public MethodDescriptor getMethodDescriptor() { - return null; + return METHOD; } } } diff --git a/src/adapter/inbound/messaging/CLAUDE.md b/src/adapter/inbound/messaging/CLAUDE.md index 2eaacaa0..11d9c1a2 100644 --- a/src/adapter/inbound/messaging/CLAUDE.md +++ b/src/adapter/inbound/messaging/CLAUDE.md @@ -14,7 +14,9 @@ `@SystemInvoked` use case itself; the use case is never a bean, and no user transport calls the handler (manual 09 「소비」 step 4, manual 07 「시스템이 부르는 쓰기」, `ONLY_SYSTEM_CALLERS_REACH_SYSTEM_INVOKED_USE_CASES`, `SYSTEM_CALLERS_ARE_SYSTEM_DRIVEN_INBOUND_ADAPTERS`, - `USER_TRANSPORTS_DO_NOT_REACH_SYSTEM_CALLERS`). + `USER_TRANSPORTS_DO_NOT_REACH_SYSTEM_CALLERS`). The handler implements only project types + (`MessageHandler`), never `Runnable` or another type outside the project + (`SYSTEM_CALLERS_HAVE_ONLY_PROJECT_SUPERTYPES`). - Tests: focused auto-configuration and inbox behavior tests; `./gradlew :adapter:inbound:messaging:check`. The real Kafka + PostgreSQL redelivery lane is `messagingInboxRedeliveryIntegrationTest` (`:integrationCheck`). diff --git a/src/adapter/inbound/messaging/README.md b/src/adapter/inbound/messaging/README.md index a457a018..4cd0586c 100644 --- a/src/adapter/inbound/messaging/README.md +++ b/src/adapter/inbound/messaging/README.md @@ -105,7 +105,9 @@ PostgreSQL에서 같은 모양은 `InboxRedeliveryIntegrationTest`의 `Projectio 요청이 들어오는 전송(web·GraphQL·gRPC·WebSocket)만 그 use case에 닿지 못하고, 표시는 ArchUnit이 검사한다 (`SYSTEM_CALLERS_ARE_SYSTEM_DRIVEN_INBOUND_ADAPTERS`: 표시한 클래스와 그 안에 중첩된 클래스는 messaging· scheduler inbound 패키지에만, 사용자 전송 타입에 의존하지 않음). 사용자 전송이 handler bean을 주입받아 - 부르는 길은 `USER_TRANSPORTS_DO_NOT_REACH_SYSTEM_CALLERS`가 막는다(RR3-01-01). 기동 가드는 바꾸지 않는다: + 부르는 길은 `USER_TRANSPORTS_DO_NOT_REACH_SYSTEM_CALLERS`가 막는다(RR3-01-01). handler가 구현하는 + `MessageHandler` 같은 프로젝트 인터페이스로 주입받아도 같다. handler는 프로젝트 밖 타입(`Runnable` 등)을 + 구현하지 않는다(`SYSTEM_CALLERS_HAVE_ONLY_PROJECT_SUPERTYPES`, RR4-01-01). 기동 가드는 바꾸지 않는다: `@SystemInvoked` bean은 예외 없이 실패다. 규칙 전체는 매뉴얼 07장 「시스템이 부르는 쓰기」. - 버림: use case를 bean으로 두고 가드에 "handler만 주입받으면 허용" 예외 — 누가 주입받는지는 기동 때 확정되지 않는다(`ObjectProvider`, `getBean`, 나중에 추가된 bean). 예외 조건이 곧 우회 경로가 된다. diff --git a/src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/error/GlobalExceptionHandler.java b/src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/error/GlobalExceptionHandler.java index f1e29196..1832fd4d 100644 --- a/src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/error/GlobalExceptionHandler.java +++ b/src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/error/GlobalExceptionHandler.java @@ -7,9 +7,9 @@ import dev.caskeleton.adapter.inbound.web.observability.RetryAfterAdvisor; import dev.caskeleton.adapter.inbound.web.pagination.PageValidationException; import dev.caskeleton.shared.error.ApiErrorCarrier; import dev.caskeleton.shared.error.ApiErrorCode; -import dev.caskeleton.shared.error.DependencyFailureException; import dev.caskeleton.shared.error.OperationalError; import dev.caskeleton.shared.error.PersistenceFailureException; +import dev.caskeleton.shared.error.ServerFailureLog; import dev.caskeleton.shared.response.Envelope; import dev.caskeleton.shared.tracing.SpanErrorRecorder; import jakarta.servlet.http.HttpServletResponse; @@ -159,40 +159,27 @@ public class GlobalExceptionHandler extends ResponseEntityExceptionHandler { * through its own {@link ApiErrorCode}. One branch, so the web adapter answers a carrier exactly * as the GraphQL, gRPC and STOMP adapters do. * - *

A persistence carrier's cause can hold SQL values, constraints, credentials and endpoints, - * so its log line and span receive only the stable classification. A server-side (5xx) carrier is - * logged with its cause; a client-side one is not logged, because it is an answer, not an - * incident. A retryable code with a registered delay also carries {@code Retry-After}. + *

What reaches the log is {@link ServerFailureLog}'s line, the one the GraphQL and gRPC + * adapters write too: a persistence carrier by its classification only (its cause can hold SQL + * values, constraints, credentials and endpoints), a dependency carrier and any other server-side + * (5xx) carrier with the exception, a client-side carrier not at all — it is an answer, not an + * incident. The span gets the same failure, a persistence one again reduced to its + * classification. A retryable code with a registered delay also carries {@code Retry-After}. * * @param failure a throwable that implements {@link ApiErrorCarrier} */ public ResponseEntity> handleCarrier(Exception failure) { ApiErrorCode code = ((ApiErrorCarrier) failure).errorCode(); - if (failure instanceof PersistenceFailureException) { - log.error( - "persistence failure classified as {} (category={}, retryable={})", - code.code(), - code.category(), - code.retryable()); - spanErrorRecorder.recordException(sanitizedPersistenceFailure(code), code.code()); - } else if (failure instanceof DependencyFailureException dependency) { - log.error( - "dependency failure classified as {} (category={}, retryable={}, dependency={})", - code.code(), - code.category(), - code.retryable(), - dependency.dependencyName(), - failure); - spanErrorRecorder.recordException(failure, code.code()); - } else if (code.httpStatus() >= 500) { - log.error( - "failure classified as {} (category={}, retryable={})", - code.code(), - code.category(), - code.retryable(), - failure); - spanErrorRecorder.recordException(failure, code.code()); - } + ServerFailureLog.of(failure, code) + .ifPresent( + line -> { + log.error(line.format(""), line.arguments()); + spanErrorRecorder.recordException( + failure instanceof PersistenceFailureException + ? sanitizedPersistenceFailure(code) + : failure, + code.code()); + }); ResponseEntity.BodyBuilder builder = ResponseEntity.status(code.httpStatus()); RetryAfterAdvisor.retryAfterSeconds(code) .ifPresent(seconds -> builder.header(ApiHeaders.RETRY_AFTER, String.valueOf(seconds))); diff --git a/src/adapter/inbound/websocket/README.md b/src/adapter/inbound/websocket/README.md index 79bef041..dcdd19c4 100644 --- a/src/adapter/inbound/websocket/README.md +++ b/src/adapter/inbound/websocket/README.md @@ -30,9 +30,14 @@ 세션 종료 콜백뿐이고, 통과한 SockJS handshake가 세션이 되지 않을 수도 있어 두 곳에서 세면 자리가 샌다. 1008이 아니라 1013인 이유: 거부 사유가 클라이언트가 아니라 지금 이 노드의 부하라서, 일반 클라이언트가 물러났다가(가능하면 다른 노드로) 다시 붙게 한다. -- 처리 실패는 모두 빈 본문의 ERROR 프레임이 되고 `message` 헤더는 고정값 - `WEBSOCKET_REQUEST_REJECTED`다. `error-code` 헤더는 원인 사슬에 `ApiErrorCarrier`가 있으면 그 - 코드, `IllegalArgumentException`이면 `BAD_PARAMETER`, 나머지는 `WEBSOCKET_REQUEST_REJECTED`다. +- handler 메서드가 돌기 전의 처리 실패(inbound interceptor의 거부 등)는 빈 본문의 ERROR 프레임이 되고 + `message` 헤더는 고정값 `WEBSOCKET_REQUEST_REJECTED`다. `error-code` 헤더는 원인 사슬에 + `ApiErrorCarrier`가 있으면 그 코드, `IllegalArgumentException`이면 `BAD_PARAMETER`, 나머지는 + `WEBSOCKET_REQUEST_REJECTED`다. +- `@MessageMapping` 메서드의 실패는 inbound 채널 executor에서 끝나서 프레임이 나가지 않는다. 서버 로그는 + `StompHandlerFailureLog`가 web·GraphQL·gRPC와 같은 줄로 남긴다(`ServerFailureLog`): persistence 실패는 + 분류만, 5xx carrier는 예외와 함께, 4xx carrier는 남기지 않는다. 그 밖의 예외는 예외와 함께 ERROR. + 전에는 Spring이 예외 전체(persistence 실패의 SQL 원인 포함)를 ERROR로 남겼다. - `LiveEventProjector` bean이 정확히 하나 있는 `@DomainEvent` 타입만 broadcast한다. 도메인 객체를 그대로 직렬화하지 않고, projector가 만든 bounded `Map`만 보낸다. diff --git a/src/adapter/inbound/websocket/src/main/java/dev/caskeleton/adapter/inbound/websocket/stomp/SafeStompSubProtocolErrorHandler.java b/src/adapter/inbound/websocket/src/main/java/dev/caskeleton/adapter/inbound/websocket/stomp/SafeStompSubProtocolErrorHandler.java index f464e0d7..7b957598 100644 --- a/src/adapter/inbound/websocket/src/main/java/dev/caskeleton/adapter/inbound/websocket/stomp/SafeStompSubProtocolErrorHandler.java +++ b/src/adapter/inbound/websocket/src/main/java/dev/caskeleton/adapter/inbound/websocket/stomp/SafeStompSubProtocolErrorHandler.java @@ -13,10 +13,15 @@ import org.springframework.web.socket.messaging.StompSubProtocolErrorHandler; * *

The frame's {@code message} header is fixed, so no exception text can reach the client. Its * {@code error-code} header is the stable code every other transport would send for the same - * failure: an {@link ApiErrorCarrier} anywhere in the cause chain (the broker wraps use-case - * failures in a {@code MessagingException}) reports its own code, an {@link + * failure: an {@link ApiErrorCarrier} anywhere in the cause chain (Spring wraps a failure raised + * while the frame is processed in a {@code MessagingException}) reports its own code, an {@link * IllegalArgumentException} reports {@code BAD_PARAMETER}, and everything else stays {@link * #SAFE_ERROR_CODE}. + * + *

What reaches it is what fails before a handler method runs — an inbound interceptor's refusal, + * on the thread that received the frame. A {@code @MessageMapping} method runs on the inbound + * channel's executor, and its failure stays there: {@link StompHandlerFailureLog} logs it and no + * frame is sent. */ final class SafeStompSubProtocolErrorHandler extends StompSubProtocolErrorHandler { diff --git a/src/adapter/inbound/websocket/src/main/java/dev/caskeleton/adapter/inbound/websocket/stomp/StompHandlerFailureLog.java b/src/adapter/inbound/websocket/src/main/java/dev/caskeleton/adapter/inbound/websocket/stomp/StompHandlerFailureLog.java new file mode 100644 index 00000000..cc1201c2 --- /dev/null +++ b/src/adapter/inbound/websocket/src/main/java/dev/caskeleton/adapter/inbound/websocket/stomp/StompHandlerFailureLog.java @@ -0,0 +1,39 @@ +package dev.caskeleton.adapter.inbound.websocket.stomp; + +import dev.caskeleton.shared.error.ApiErrorCarrier; +import dev.caskeleton.shared.error.ServerFailureLog; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.messaging.handler.annotation.MessageExceptionHandler; +import org.springframework.web.bind.annotation.ControllerAdvice; + +/** + * Writes the server-log line for a {@code @MessageMapping} method that throws, the way the web, + * GraphQL and gRPC adapters do. + * + *

Such a failure is handled on the inbound channel's executor, so it never reaches {@link + * SafeStompSubProtocolErrorHandler}, which only sees what fails before a handler runs (an + * interceptor's refusal). Without a handler for it, Spring's method handler logs the whole + * exception and stops: a persistence failure's SQL cause — values, constraint names, credentials — + * went to the log, and a client-side refusal was logged as an error. With this advice a carrier + * gets {@link ServerFailureLog}'s line (a persistence failure its classification only, a + * client-side code nothing), and anything else is logged with its exception, as before. The request + * id is on MDC ({@link WebSocketCorrelationInterceptor}). The client is answered as before: a + * handler method that throws sends no frame. A feature's own {@code @MessageExceptionHandler} still + * takes precedence over this one. + */ +@ControllerAdvice +class StompHandlerFailureLog { + + private static final Logger log = LoggerFactory.getLogger(StompHandlerFailureLog.class); + + @MessageExceptionHandler + void logFailure(Exception failure) { + if (failure instanceof ApiErrorCarrier carrier) { + ServerFailureLog.of(failure, carrier.errorCode()) + .ifPresent(line -> log.error(line.format(""), line.arguments())); + return; + } + log.error("unhandled failure in a STOMP message handler", failure); + } +} diff --git a/src/adapter/inbound/websocket/src/main/java/dev/caskeleton/adapter/inbound/websocket/stomp/WebSocketConfig.java b/src/adapter/inbound/websocket/src/main/java/dev/caskeleton/adapter/inbound/websocket/stomp/WebSocketConfig.java index 3243f0a1..15abcbd5 100644 --- a/src/adapter/inbound/websocket/src/main/java/dev/caskeleton/adapter/inbound/websocket/stomp/WebSocketConfig.java +++ b/src/adapter/inbound/websocket/src/main/java/dev/caskeleton/adapter/inbound/websocket/stomp/WebSocketConfig.java @@ -134,6 +134,15 @@ public class WebSocketConfig implements WebSocketMessageBrokerConfigurer { return connectionLimit; } + /** + * Logs a {@code @MessageMapping} method's failure with the line the other transports write: a + * persistence failure by its classification only, a client-side code not at all. + */ + @Bean + StompHandlerFailureLog stompHandlerFailureLog() { + return new StompHandlerFailureLog(); + } + @Bean LiveEventStompBroadcaster liveEventStompBroadcaster( SimpMessagingTemplate messaging, diff --git a/src/adapter/inbound/websocket/src/test/java/dev/caskeleton/adapter/inbound/websocket/stomp/SafeStompSubProtocolErrorHandlerTest.java b/src/adapter/inbound/websocket/src/test/java/dev/caskeleton/adapter/inbound/websocket/stomp/SafeStompSubProtocolErrorHandlerTest.java index dba3b314..702f7bad 100644 --- a/src/adapter/inbound/websocket/src/test/java/dev/caskeleton/adapter/inbound/websocket/stomp/SafeStompSubProtocolErrorHandlerTest.java +++ b/src/adapter/inbound/websocket/src/test/java/dev/caskeleton/adapter/inbound/websocket/stomp/SafeStompSubProtocolErrorHandlerTest.java @@ -32,7 +32,7 @@ class SafeStompSubProtocolErrorHandlerTest { @Test void aWrappedCarrierFailureReportsItsOwnCode() { - // given: the broker wraps a use-case failure before it reaches the error handler + // given: Spring wraps a failure raised while the frame was processed (an interceptor's) Throwable failure = new MessagingException( "SECRET_SENTINEL", new MappingException("SECRET_SENTINEL mapping detail")); diff --git a/src/adapter/inbound/websocket/src/test/java/dev/caskeleton/adapter/inbound/websocket/stomp/StompHandlerFailureLogTest.java b/src/adapter/inbound/websocket/src/test/java/dev/caskeleton/adapter/inbound/websocket/stomp/StompHandlerFailureLogTest.java new file mode 100644 index 00000000..d03eccac --- /dev/null +++ b/src/adapter/inbound/websocket/src/test/java/dev/caskeleton/adapter/inbound/websocket/stomp/StompHandlerFailureLogTest.java @@ -0,0 +1,185 @@ +package dev.caskeleton.adapter.inbound.websocket.stomp; + +import static org.assertj.core.api.Assertions.assertThat; + +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.Logger; +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; +import dev.caskeleton.shared.error.ApiErrorCarrier; +import dev.caskeleton.shared.error.ApiErrorCode; +import dev.caskeleton.shared.error.OperationalError; +import dev.caskeleton.shared.error.PersistenceFailureException; +import java.sql.SQLException; +import java.util.HashMap; +import java.util.List; +import org.junit.jupiter.api.Test; +import org.slf4j.LoggerFactory; +import org.springframework.context.support.StaticApplicationContext; +import org.springframework.messaging.Message; +import org.springframework.messaging.handler.annotation.MessageMapping; +import org.springframework.messaging.simp.SimpMessageHeaderAccessor; +import org.springframework.messaging.simp.SimpMessageType; +import org.springframework.messaging.simp.SimpMessagingTemplate; +import org.springframework.messaging.support.ExecutorSubscribableChannel; +import org.springframework.messaging.support.MessageBuilder; +import org.springframework.stereotype.Controller; +import org.springframework.web.socket.messaging.WebSocketAnnotationMethodMessageHandler; + +/** + * A {@code @MessageMapping} method that throws is handled on the inbound channel's executor, so the + * failure never reaches {@link SafeStompSubProtocolErrorHandler}: Spring's method handler logs it + * and stops. Without an exception handler that line is the whole exception, a persistence failure's + * SQL cause included. Driven through Spring's real STOMP method handler, so the advice is proven to + * be the one the handler picks. + */ +class StompHandlerFailureLogTest { + + @Test + void aServerSideCarrierFromAMessageMappingIsLoggedAtErrorWithTheFailure() { + // when + List logged = loggedWhileHandling("/app/recovery-required"); + + // then + assertThat(logged) + .singleElement() + .extracting( + ILoggingEvent::getLevel, + ILoggingEvent::getFormattedMessage, + event -> event.getThrowableProxy().getMessage()) + .containsExactly( + Level.ERROR, + "failure classified as IDEMPOTENCY_RECOVERY_REQUIRED (category=DATA_INTEGRITY," + + " retryable=false)", + "claim requires reconciliation: CHARGE_CARD::v1::3f9a"); + } + + @Test + void aPersistenceFailureFromAMessageMappingIsLoggedByItsClassificationWithoutItsCause() { + // when + List logged = loggedWhileHandling("/app/persistence"); + + // then + assertThat(logged) + .singleElement() + .extracting(ILoggingEvent::getFormattedMessage, ILoggingEvent::getThrowableProxy) + .containsExactly( + "persistence failure classified as DB_UNAVAILABLE (category=TRANSIENT_DEPENDENCY," + + " retryable=true)", + null); + } + + @Test + void aClientSideCarrierFromAMessageMappingIsNotLogged() { + // when + List logged = loggedWhileHandling("/app/bad-parameter"); + + // then + assertThat(logged).isEmpty(); + } + + @Test + void anUnrecognisedFailureFromAMessageMappingIsLoggedAtErrorWithTheFailure() { + // when + List logged = loggedWhileHandling("/app/unrecognised"); + + // then + assertThat(logged) + .singleElement() + .extracting( + ILoggingEvent::getLevel, + ILoggingEvent::getFormattedMessage, + event -> event.getThrowableProxy().getClassName()) + .containsExactly( + Level.ERROR, + "unhandled failure in a STOMP message handler", + IllegalStateException.class.getName()); + } + + /** + * Every ERROR or WARN line written anywhere while one message to {@code destination} is handled. + */ + private static List loggedWhileHandling(String destination) { + StaticApplicationContext context = new StaticApplicationContext(); + context.registerSingleton("failingController", FailingController.class); + context.registerSingleton("stompHandlerFailureLog", StompHandlerFailureLog.class); + context.refresh(); + // The handler @EnableWebSocketMessageBroker wires; it is the one that reads @ControllerAdvice. + WebSocketAnnotationMethodMessageHandler handler = + new WebSocketAnnotationMethodMessageHandler( + new ExecutorSubscribableChannel(), + new ExecutorSubscribableChannel(), + new SimpMessagingTemplate(new ExecutorSubscribableChannel())); + handler.setDestinationPrefixes(List.of("/app")); + handler.setApplicationContext(context); + handler.afterPropertiesSet(); + Message message = + MessageBuilder.withPayload(new byte[0]) + .setHeader(SimpMessageHeaderAccessor.MESSAGE_TYPE_HEADER, SimpMessageType.MESSAGE) + .setHeader(SimpMessageHeaderAccessor.DESTINATION_HEADER, destination) + .setHeader(SimpMessageHeaderAccessor.SESSION_ID_HEADER, "session-1") + .setHeader(SimpMessageHeaderAccessor.SESSION_ATTRIBUTES, new HashMap()) + .build(); + + Logger root = (Logger) LoggerFactory.getLogger(org.slf4j.Logger.ROOT_LOGGER_NAME); + ListAppender appender = new ListAppender<>(); + appender.start(); + root.addAppender(appender); + try { + handler.handleMessage(message); + } finally { + root.detachAppender(appender); + appender.stop(); + context.close(); + } + return appender.list.stream() + .filter(event -> event.getLevel().isGreaterOrEqual(Level.WARN)) + .toList(); + } + + /** A feature's STOMP controller whose use case fails in each of the ways that matter. */ + @Controller + static class FailingController { + + @MessageMapping("/recovery-required") + void recoveryRequired() { + throw new CarrierException( + OperationalError.IDEMPOTENCY_RECOVERY_REQUIRED, + "claim requires reconciliation: CHARGE_CARD::v1::3f9a"); + } + + @MessageMapping("/persistence") + void persistence() { + throw new PersistenceFailureException( + OperationalError.DB_UNAVAILABLE, + "08006", + new SQLException("jdbc:postgresql://db.internal/customer?password=secret", "08006")); + } + + @MessageMapping("/bad-parameter") + void badParameter() { + throw new CarrierException(OperationalError.BAD_PARAMETER, "bad input"); + } + + @MessageMapping("/unrecognised") + void unrecognised() { + throw new IllegalStateException("boom"); + } + } + + /** A use-case failure carrying a shared code. */ + private static final class CarrierException extends RuntimeException implements ApiErrorCarrier { + private static final long serialVersionUID = 1L; + private final transient ApiErrorCode code; + + CarrierException(ApiErrorCode code, String message) { + super(message); + this.code = code; + } + + @Override + public ApiErrorCode errorCode() { + return code; + } + } +} diff --git a/src/adapter/inbound/websocket/src/test/java/dev/caskeleton/adapter/inbound/websocket/stomp/WebSocketAutoConfigurationTest.java b/src/adapter/inbound/websocket/src/test/java/dev/caskeleton/adapter/inbound/websocket/stomp/WebSocketAutoConfigurationTest.java index c8cdfeea..d8f78019 100644 --- a/src/adapter/inbound/websocket/src/test/java/dev/caskeleton/adapter/inbound/websocket/stomp/WebSocketAutoConfigurationTest.java +++ b/src/adapter/inbound/websocket/src/test/java/dev/caskeleton/adapter/inbound/websocket/stomp/WebSocketAutoConfigurationTest.java @@ -117,6 +117,11 @@ class WebSocketAutoConfigurationTest { }); } + @Test + void theStompHandlerFailureLogIsRegisteredWhenTheChannelIsOn() { + runner.run(context -> assertThat(context).hasSingleBean(StompHandlerFailureLog.class)); + } + @Test void switchedOffHoldsNothing() { runner diff --git a/src/adapter/outbound/cache-redis/README.md b/src/adapter/outbound/cache-redis/README.md index 623c297b..89cd197e 100644 --- a/src/adapter/outbound/cache-redis/README.md +++ b/src/adapter/outbound/cache-redis/README.md @@ -288,8 +288,12 @@ retryable. A record still `EXECUTING` when its lease ran out has an unknown effe program answers `RECOVERY_REQUIRED` and leaves the record as it is, because the holder may still be running. The holder's late completion or failure goes through the transition program as it would have inside the lease, and the next claim replays or retries it; only the lease is final — `RENEW` -refuses one that has run out, and `INSPECT` reports somebody else's expired execution as -`RECOVERY_REQUIRED`. The PostgreSQL store answers the same; the shared `IdempotencyStoreContract` +refuses one that has run out with `LEASE_EXPIRED` (not `NOT_OWNER`: the holder still completes the +record, re-review 2026-11-04 RR5-01-02), and `INSPECT` reports somebody else's expired execution as +`RECOVERY_REQUIRED`. The transition program reads the lease for a start only: a claim whose lease +ran out is answered `LEASE_EXPIRED` and not started, so nothing runs and the next claim takes it +over (it used to start, and every other request was then told "recovery required" for work that +had just begun; README 15). The PostgreSQL store answers the same; the shared `IdempotencyStoreContract` holds both to it on a real server (re-review 2026-11-04 RR-01-11 — this store used to take the record over and run the work a second time; owner decision 2026-11-04 — both stores then abandoned the record, and a holder that outlived its lease lost its result). diff --git a/src/adapter/outbound/cache-redis/build.gradle b/src/adapter/outbound/cache-redis/build.gradle index 1719cf67..f9909060 100644 --- a/src/adapter/outbound/cache-redis/build.gradle +++ b/src/adapter/outbound/cache-redis/build.gradle @@ -88,7 +88,10 @@ strictTestLanes { // the explicit-only topology lane alone, so no gate ever ran it (RR-04-05). 'dev.caskeleton.adapter.outbound.cache.redis.idempotency.LiveRedisIdempotencyStoreContractTest', 'dev.caskeleton.adapter.outbound.cache.redis.LiveRedisSemanticPortsTest', - 'dev.caskeleton.adapter.outbound.cache.redis.capability.LiveRedisSemanticScriptAclProbeTest' + 'dev.caskeleton.adapter.outbound.cache.redis.capability.LiveRedisSemanticScriptAclProbeTest', + // The runbook's Redis repair, run as written under the lane's ACL (re-review + // 2026-11-04 RR4-03-02). + 'dev.caskeleton.adapter.outbound.cache.redis.idempotency.LiveRedisIdempotencyRunbookTest' rejectSkipped = true systemProperty 'redis.container.acl-file', new File(redisAclDirectory, 'lane-fixture-accounts.acl').absolutePath inputDirectory redisAclDirectory @@ -96,3 +99,10 @@ strictTestLanes { } } +// LiveRedisIdempotencyRunbookTest reads its blocks out of the runbook an operator follows rather than +// keeping a copy, so the runbook is an input of the lane: editing a block reruns it against Redis. +tasks.named('redisContainerContractTest', Test) { + def idempotencyRunbook = new File(rootProject.projectDir.parentFile, 'docs/runbooks/idempotency-unavailable.md') + inputs.file(idempotencyRunbook).withPathSensitivity(PathSensitivity.RELATIVE).withPropertyName('idempotencyRunbook') + systemProperty 'idempotency.runbook', idempotencyRunbook.absolutePath +} diff --git a/src/adapter/outbound/cache-redis/src/main/java/dev/caskeleton/adapter/outbound/cache/redis/idempotency/IdempotencyScripts.java b/src/adapter/outbound/cache-redis/src/main/java/dev/caskeleton/adapter/outbound/cache/redis/idempotency/IdempotencyScripts.java index bc4c6f4b..1ffa2941 100644 --- a/src/adapter/outbound/cache-redis/src/main/java/dev/caskeleton/adapter/outbound/cache/redis/idempotency/IdempotencyScripts.java +++ b/src/adapter/outbound/cache-redis/src/main/java/dev/caskeleton/adapter/outbound/cache/redis/idempotency/IdempotencyScripts.java @@ -108,7 +108,17 @@ public final class IdempotencyScripts { return {'IN_PROGRESS', attempt, rev, owner, '', tostring(leaseUntil - nowMillis)} """; - /** A generic owner+revision compare-and-set transition. */ + /** + * A generic owner+revision compare-and-set transition. + * + *

A start ({@code CLAIMED} to {@code EXECUTING}) also needs the claim's lease to be running, + * judged by the server clock as {@link #CLAIM} judges it: a claim whose lease ran out is answered + * {@code LEASE_EXPIRED} and not started, so nothing runs and the next claim takes it over. This + * program used to start it, and every other request was then told "recovery required" for an + * execution that had just begun; the PostgreSQL store refused it (re-review 2026-11-04 README 15, + * {@code IdempotencyStoreContract}). Completion and failure read no lease: the holder's late + * result is recorded (owner decision 2026-11-04). + */ private static final String TRANSITION = """ local state = redis.call('HGET', KEYS[1], 'state') @@ -142,6 +152,14 @@ public final class IdempotencyScripts { if state ~= ARGV[4] then return {'WRONG_STATE', attempt, rev, owner, state, ''} end + if state == 'CLAIMED' then + local t = redis.call('TIME') + local nowMillis = (tonumber(t[1]) * 1000) + math.floor(tonumber(t[2]) / 1000) + local leaseUntil = tonumber(redis.call('HGET', KEYS[1], 'leaseUntil')) + if leaseUntil ~= nil and leaseUntil <= nowMillis then + return {'LEASE_EXPIRED', attempt, rev, owner, '', ''} + end + end redis.call('HSET', KEYS[1], 'state', ARGV[5], 'rev', rev + 1) if ARGV[6] ~= '' then redis.call('HSET', KEYS[1], 'resp', ARGV[6]) @@ -179,8 +197,10 @@ public final class IdempotencyScripts { * *

A lease that has already run out is not renewed: a resend may have been told "recovery * required" since, and a lease brought back would answer the next one "in progress" for the same - * unknown effect. The holder can still complete or fail the record. The PostgreSQL store refuses - * the same renewal ({@code IdempotencyStoreContract}, owner decision 2026-11-04). + * unknown effect. The answer is {@code LEASE_EXPIRED}, not {@code NOT_OWNER}: the holder still + * owns the record and can still complete or fail it (re-review 2026-11-04 RR5-01-02). The + * PostgreSQL store refuses the same renewal ({@code IdempotencyStoreContract}, owner decision + * 2026-11-04). */ private static final String RENEW = """ @@ -216,7 +236,7 @@ public final class IdempotencyScripts { local nowMillis = (tonumber(t[1]) * 1000) + math.floor(tonumber(t[2]) / 1000) local leaseUntil = tonumber(redis.call('HGET', KEYS[1], 'leaseUntil')) if leaseUntil ~= nil and leaseUntil <= nowMillis then - return {'NOT_OWNER', attempt, rev, owner, '', ''} + return {'LEASE_EXPIRED', attempt, rev, owner, '', ''} end redis.call('HSET', KEYS[1], 'leaseUntil', nowMillis + tonumber(ARGV[4]), 'rev', rev + 1, @@ -366,6 +386,8 @@ public final class IdempotencyScripts { ALREADY, ABSENT, NOT_OWNER, + /** The caller still holds the record, but its processing lease has run out. */ + LEASE_EXPIRED, OPERATION_CONFLICT, WRONG_STATE, CLAIMED, diff --git a/src/adapter/outbound/cache-redis/src/main/java/dev/caskeleton/adapter/outbound/cache/redis/idempotency/RedisIdempotencyStoreAdapter.java b/src/adapter/outbound/cache-redis/src/main/java/dev/caskeleton/adapter/outbound/cache/redis/idempotency/RedisIdempotencyStoreAdapter.java index 606f4f26..7329ae62 100644 --- a/src/adapter/outbound/cache-redis/src/main/java/dev/caskeleton/adapter/outbound/cache/redis/idempotency/RedisIdempotencyStoreAdapter.java +++ b/src/adapter/outbound/cache-redis/src/main/java/dev/caskeleton/adapter/outbound/cache/redis/idempotency/RedisIdempotencyStoreAdapter.java @@ -175,6 +175,7 @@ public final class RedisIdempotencyStoreAdapter implements IdempotencyStorePort ScriptStatus.ALREADY, IdempotencyStartOutcome.ALREADY_STARTED_SAME_OPERATION, ScriptStatus.ABSENT, IdempotencyStartOutcome.ABSENT, ScriptStatus.NOT_OWNER, IdempotencyStartOutcome.NOT_OWNER, + ScriptStatus.LEASE_EXPIRED, IdempotencyStartOutcome.LEASE_EXPIRED, ScriptStatus.WRONG_STATE, IdempotencyStartOutcome.NOT_CLAIMED, ScriptStatus.OPERATION_CONFLICT, IdempotencyStartOutcome.OPERATION_CONFLICT), IdempotencyStartOutcome.INDETERMINATE, @@ -204,6 +205,7 @@ public final class RedisIdempotencyStoreAdapter implements IdempotencyStorePort ScriptStatus.ALREADY, IdempotencyRenewOutcome.ALREADY_RENEWED_SAME_OPERATION, ScriptStatus.ABSENT, IdempotencyRenewOutcome.ABSENT, ScriptStatus.NOT_OWNER, IdempotencyRenewOutcome.NOT_OWNER, + ScriptStatus.LEASE_EXPIRED, IdempotencyRenewOutcome.LEASE_EXPIRED, ScriptStatus.WRONG_STATE, IdempotencyRenewOutcome.NOT_IN_PROGRESS, ScriptStatus.OPERATION_CONFLICT, IdempotencyRenewOutcome.OPERATION_CONFLICT), IdempotencyRenewOutcome.INDETERMINATE, diff --git a/src/adapter/outbound/cache-redis/src/test/java/dev/caskeleton/adapter/outbound/cache/redis/idempotency/LiveRedisIdempotencyRunbookTest.java b/src/adapter/outbound/cache-redis/src/test/java/dev/caskeleton/adapter/outbound/cache/redis/idempotency/LiveRedisIdempotencyRunbookTest.java new file mode 100644 index 00000000..5b151076 --- /dev/null +++ b/src/adapter/outbound/cache-redis/src/test/java/dev/caskeleton/adapter/outbound/cache/redis/idempotency/LiveRedisIdempotencyRunbookTest.java @@ -0,0 +1,203 @@ +package dev.caskeleton.adapter.outbound.cache.redis.idempotency; + +import static org.assertj.core.api.Assertions.assertThat; + +import dev.caskeleton.adapter.outbound.cache.redis.LiveRedisRuntime; +import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisConnectionKind; +import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisLease; +import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisRuntimeOwner; +import dev.caskeleton.adapter.outbound.cache.redis.runtime.key.RedisNamespace; +import dev.caskeleton.application.idempotency.IdempotencyClaimAttempt; +import dev.caskeleton.application.idempotency.IdempotencyClaimOutcome; +import dev.caskeleton.application.idempotency.IdempotencyClaimRequest; +import dev.caskeleton.application.idempotency.IdempotencyFailureDisposition; +import dev.caskeleton.application.idempotency.IdempotencyInspectionOutcome; +import dev.caskeleton.application.idempotency.IdempotencyInspectionRequest; +import dev.caskeleton.application.idempotency.IdempotencyOwner; +import dev.caskeleton.application.idempotency.IdempotencyScopeDigest; +import dev.caskeleton.application.idempotency.RequestFingerprint; +import dev.caskeleton.application.transaction.OperationId; +import io.lettuce.core.ScriptOutputType; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Clock; +import java.time.Duration; +import java.util.UUID; +import java.util.concurrent.TimeUnit; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; + +/** + * The Redis repair in {@code docs/runbooks/idempotency-unavailable.md}, run as written against a + * real server under the lane's ACL. + * + *

The runbook used to have the operator check the record's state and then delete it with a + * second command; a record that expired between the two and was claimed again by a new request was + * deleted while that request ran, and the next request with the key ran the work a second time + * (re-review 2026-11-04 RR4-03-02). The repair is now one Lua program that checks and deletes + * atomically. The application accounts are denied {@code EVAL}, so the runbook loads it and runs it + * by digest, as the service's own scripts run — and this test runs it the same way, as the advanced + * account, with the record key rendered from the runbook's key template rather than from the + * adapter: a wrong template finds no record and fails here. + * + *

The blocks are read out of the runbook, not copied, as the PostgreSQL runbook lanes read + * theirs: a copy would pass while the document an operator follows is wrong. + */ +@Tag("redis-topology") +class LiveRedisIdempotencyRunbookTest { + + private static final Pattern SCRIPT_LOAD = Pattern.compile("^SCRIPT LOAD \"(.+)\"$"); + + private static final Pattern EVALSHA = Pattern.compile("^EVALSHA <[^>]+> 1 $"); + + private static final IdempotencyScopeDigest SCOPE = + new IdempotencyScopeDigest("d".repeat(64), 1, "CHARGE_CARD"); + + private static final RequestFingerprint FINGERPRINT = new RequestFingerprint("e".repeat(64)); + + private static String runbook; + + private RedisNamespace namespace; + + private RedisRuntimeOwner owner; + + private RedisIdempotencyStoreAdapter store; + + @BeforeAll + static void readRunbook() throws IOException { + String path = System.getProperty("idempotency.runbook"); + if (path == null || path.isBlank()) { + throw new IllegalStateException( + "system property idempotency.runbook must name the runbook this lane runs"); + } + runbook = Files.readString(Path.of(path), StandardCharsets.UTF_8); + } + + @BeforeEach + void openStore() { + namespace = + new RedisNamespace("prod", "order", "r" + UUID.randomUUID().toString().substring(0, 8)); + owner = LiveRedisRuntime.open(namespace, true); + store = + new RedisIdempotencyStoreAdapter( + owner, + new RedisIdempotencyStoreAdapter.IdempotencyKeys(namespace, 1), + new IdempotencyScripts(), + Clock.systemUTC(), + Duration.ofSeconds(2)); + } + + @AfterEach + void closeOwner() { + owner.close(); + } + + @Test + void theRunbookDeleteRemovesAnAbandonedRecordSoTheKeyRunsAgain() throws Exception { + // given + IdempotencyOwner executing = executing(store.newClaimAttempt(new OperationId("op-abandon"))); + store.markFailed( + executing, + IdempotencyFailureDisposition.EFFECT_UNKNOWN_ABANDONED, + Duration.ofHours(1), + new OperationId("op-abandon")); + + // when + long deleted = runTheRunbookDelete(); + + // then + assertThat(deleted).isEqualTo(1); + assertThat(store.claim(claim(store.newClaimAttempt(new OperationId("op-again"))))) + .isInstanceOf(IdempotencyClaimOutcome.Acquired.class); + } + + @Test + void theRunbookDeleteLeavesARecordThatIsNoLongerAbandoned() throws Exception { + // given + // What the operator's key holds when the abandoned record expired after they looked at it and + // a new request claimed the key and started: the delete must leave that request's record. + IdempotencyClaimAttempt running = store.newClaimAttempt(new OperationId("op-running")); + IdempotencyOwner executing = executing(running); + + // when + long deleted = runTheRunbookDelete(); + + // then + assertThat(deleted).isZero(); + var holder = store.inspect(new IdempotencyInspectionRequest(SCOPE, FINGERPRINT, running)); + assertThat(holder.outcome()).isEqualTo(IdempotencyInspectionOutcome.EXECUTING_SAME_OPERATION); + assertThat(holder.owner()).contains(executing); + } + + private IdempotencyOwner executing(IdempotencyClaimAttempt attempt) { + IdempotencyOwner claimed = + ((IdempotencyClaimOutcome.Acquired) store.claim(claim(attempt))).owner(); + return store.markExecutionStarted(claimed, attempt.operationId()).owner().orElseThrow(); + } + + private static IdempotencyClaimRequest claim(IdempotencyClaimAttempt attempt) { + return new IdempotencyClaimRequest( + SCOPE, FINGERPRINT, attempt, Duration.ofSeconds(30), Duration.ofHours(1), "json-v2", 1); + } + + /** + * Runs the runbook's two lines as an operator would: load the program, then run it by the digest + * the load printed, on the key the runbook's template names. + */ + private long runTheRunbookDelete() throws Exception { + String[] lines = block("idempotency-delete-abandoned").strip().split("\n"); + assertThat(lines).as("the repair is a load and one run").hasSize(2); + Matcher load = SCRIPT_LOAD.matcher(lines[0].strip()); + assertThat(load.matches()).as("first line loads the program: %s", lines[0]).isTrue(); + assertThat(EVALSHA.matcher(lines[1].strip()).matches()) + .as("second line runs it by digest on the one key: %s", lines[1]) + .isTrue(); + byte[] key = recordKeyFromTheRunbook().getBytes(StandardCharsets.UTF_8); + try (RedisLease lease = owner.borrow(RedisConnectionKind.SCRIPT)) { + var commands = lease.driver().commands(); + String digest = + commands + .scriptLoad(load.group(1).getBytes(StandardCharsets.UTF_8)) + .toCompletableFuture() + .get(2, TimeUnit.SECONDS); + return commands + .evalsha(digest, ScriptOutputType.INTEGER, key) + .toCompletableFuture() + .get(2, TimeUnit.SECONDS); + } + } + + /** The record key, from the runbook's template filled in as an operator would. */ + private String recordKeyFromTheRunbook() { + return block("idempotency-record-key") + .strip() + .replace("", namespace.environment()) + .replace("", namespace.service()) + .replace("", namespace.domain()) + .replace("", Integer.toString(SCOPE.keyDigestVersion())) + .replace("", SCOPE.operationCode()) + .replace("", SCOPE.digest()); + } + + /** The fenced block that follows {@code }. */ + private static String block(String id) { + String marker = ""; + int at = runbook.indexOf(marker); + if (at < 0) { + throw new IllegalStateException("the idempotency runbook has no Redis block marked " + id); + } + int open = runbook.indexOf("```\n", at); + int close = runbook.indexOf("```", open + 4); + if (open < 0 || close < 0) { + throw new IllegalStateException("the block marked " + id + " is not a closed fence"); + } + return runbook.substring(open + 4, close); + } +} diff --git a/src/adapter/outbound/cache-redis/src/test/java/dev/caskeleton/adapter/outbound/cache/redis/idempotency/RedisIdempotencyStoreAdapterTest.java b/src/adapter/outbound/cache-redis/src/test/java/dev/caskeleton/adapter/outbound/cache/redis/idempotency/RedisIdempotencyStoreAdapterTest.java index 13c584e0..b8188d9b 100644 --- a/src/adapter/outbound/cache-redis/src/test/java/dev/caskeleton/adapter/outbound/cache/redis/idempotency/RedisIdempotencyStoreAdapterTest.java +++ b/src/adapter/outbound/cache-redis/src/test/java/dev/caskeleton/adapter/outbound/cache/redis/idempotency/RedisIdempotencyStoreAdapterTest.java @@ -364,7 +364,85 @@ class RedisIdempotencyStoreAdapterTest { var renewed = store.renew(executing, Duration.ofSeconds(30), OPERATION); // then - assertThat(renewed.outcome()).isEqualTo(IdempotencyRenewOutcome.NOT_OWNER); + // The holder still owns the record — it still completes it — so not NOT_OWNER (RR5-01-02). + assertThat(renewed.outcome()).isEqualTo(IdempotencyRenewOutcome.LEASE_EXPIRED); + } + + @Test + @DisplayName("lease가 지난 실행 전 claim은 주인도 시작할 수 없다") + void aClaimWhoseLeaseRanOutBeforeItStartedIsNotStarted() { + // given + // The live contract proves the program; this proves the fixture's copy of TRANSITION refuses + // the late start as the program does (re-review 2026-11-04 README 15). + IdempotencyOwner claimed = + ((IdempotencyClaimOutcome.Acquired) store.claim(claim(store.newClaimAttempt(OPERATION)))) + .owner(); + gateway.advance(Duration.ofSeconds(60)); + + // when + var started = store.markExecutionStarted(claimed, OPERATION); + + // then + assertThat(started.outcome()).isEqualTo(IdempotencyStartOutcome.LEASE_EXPIRED); + } + + @Test + @DisplayName("lease가 지나 시작을 거절당한 claim은 다음 claim이 attempt 2로 넘겨받는다") + void aClaimWhoseLateStartWasRefusedIsTakenOver() { + // given + IdempotencyOwner claimed = + ((IdempotencyClaimOutcome.Acquired) store.claim(claim(store.newClaimAttempt(OPERATION)))) + .owner(); + gateway.advance(Duration.ofSeconds(60)); + store.markExecutionStarted(claimed, OPERATION); + + // when + IdempotencyClaimOutcome next = store.claim(claim(store.newClaimAttempt(OTHER_OPERATION))); + + // then + assertThat(next) + .isInstanceOfSatisfying( + IdempotencyClaimOutcome.TakenOverClaimed.class, + takenOver -> assertThat(takenOver.owner().attempt()).isEqualTo(2)); + } + + @Test + @DisplayName("lease가 지난 실행의 주인이 늦게 보고한 재시도 가능 실패 뒤에는 다음 claim이 넘겨받는다") + void aLateRetryableFailureOfAnExpiredExecutionLetsTheNextClaimTakeOver() { + // given + // The late markFailed through the fixture's copy of TRANSITION, which no test reached + // (re-review 2026-11-04 RR5-01-04). + IdempotencyOwner late = executing(); + gateway.advance(Duration.ofSeconds(60)); + store.claim(claim(store.newClaimAttempt(OTHER_OPERATION))); + store.markFailed( + late, IdempotencyFailureDisposition.NO_EFFECT_RETRYABLE, Duration.ofHours(1), OPERATION); + + // when + IdempotencyClaimOutcome next = store.claim(claim(store.newClaimAttempt(OTHER_OPERATION))); + + // then + assertThat(next).isInstanceOf(IdempotencyClaimOutcome.TakenOverClaimed.class); + } + + @Test + @DisplayName("lease가 지난 실행의 주인이 늦게 보고한 효과 불명 실패 뒤에도 다음 claim은 회복 필요를 받는다") + void aLateUnknownEffectOfAnExpiredExecutionKeepsTheNextClaimInRecovery() { + // given + IdempotencyOwner late = executing(); + gateway.advance(Duration.ofSeconds(60)); + store.claim(claim(store.newClaimAttempt(OTHER_OPERATION))); + store.markFailed( + late, + IdempotencyFailureDisposition.EFFECT_UNKNOWN_ABANDONED, + Duration.ofHours(1), + OPERATION); + + // when + IdempotencyClaimOutcome next = store.claim(claim(store.newClaimAttempt(OTHER_OPERATION))); + + // then + assertThat(next).isInstanceOf(IdempotencyClaimOutcome.RecoveryRequired.class); } @Test diff --git a/src/adapter/outbound/httpclient/README.md b/src/adapter/outbound/httpclient/README.md index bb5e72f4..9ae38727 100644 --- a/src/adapter/outbound/httpclient/README.md +++ b/src/adapter/outbound/httpclient/README.md @@ -43,7 +43,11 @@ no reactive (`WebClient`/Reactor Netty) surface; it was removed on 2026-09-25 (d `HttpDeadlineExceededException`. A streaming download's setup (up to the response headers) runs inside the same deadline; after hand-over each body gap is bounded by `read-idle` and nothing else, on both transports — on the JDK transport, which has no socket read timeout, a per-stream watchdog - on its own daemon thread (`httpclient-jdk-read-idle`) bounds each read's wait. A body that fails + bounds each read's wait. The watchdogs share one daemon thread (`httpclient-jdk-read-idle`): the + first stream that needs it starts it, it ends 60 s after the last wake it had queued, and the next + stream starts it again. It inherits no thread-locals from the request thread that starts it and + uses this module's class loader as its context loader, so it pins neither a request's state nor a + redeployed application's loader. A body that fails after hand-over throws `HttpBodyReadException`, an `IOException` as `InputStream.read` promises, whose `failure()` is the same stable exception a buffered call throws (`HttpResponseTimeoutException` for a stalled body, `HttpResponseTruncatedException` for a cut diff --git a/src/adapter/outbound/httpclient/build.gradle b/src/adapter/outbound/httpclient/build.gradle index 220ab3a3..40108ed1 100644 --- a/src/adapter/outbound/httpclient/build.gradle +++ b/src/adapter/outbound/httpclient/build.gradle @@ -227,14 +227,18 @@ strictTestLanes { // name's fail-closed selection is proven before a release tag, not first on one (review 2026-11-01 // 08-01). // -// httpClientFailureInjectionTest joins the main stage too (review 2026-11-01 05-06, 05-16). It is -// the only lane that proves the deadline and evidence contracts against real TCP faults — a slow -// drip, a cut stream, a refused connect — and it was reachable only through integrationCheck, so a -// main-branch build could go green with those contracts broken. The main stage already runs Docker -// lanes (MinIO), so the lane's fail-closed Docker requirement costs that stage nothing new; it stays -// out of `check` for the reason above. +// httpClientFailureInjectionTest is deliberately not in mainCheck. It is the only lane that proves +// the deadline and evidence contracts against real TCP faults — a slow drip, a cut stream, a refused +// connect — and review 2026-11-01 05-06/05-16 put it in the main stage because it was reachable only +// through integrationCheck, which then ran on no main-branch build. That premise no longer holds: +// integration-main.yml's `integration-check` job runs integrationCheck (an integration lane, so this +// one) on every push to main and nightly, in the same workflow run as the `main-check` job. In both +// stages, every main push started a second Toxiproxy and httpbin and ran every fault twice without +// covering anything the first run did not (review 2026-11-03 README §8, closed in round 9; the +// cache-redis container lane left mainCheck for the same reason). verifyStrictLaneStages still fails +// if no executed stage reaches it. It stays out of `check` for the reason above. tasks.named('mainCheck') { - dependsOn 'spring70CompatibilityTest', 'httpClientFailureInjectionTest' + dependsOn 'spring70CompatibilityTest' } tasks.named('check') { diff --git a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/jdk/ReadIdleInputStream.java b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/jdk/ReadIdleInputStream.java index 36ffbdc2..276ace6b 100644 --- a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/jdk/ReadIdleInputStream.java +++ b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/jdk/ReadIdleInputStream.java @@ -9,6 +9,7 @@ import java.util.Objects; import java.util.concurrent.ScheduledExecutorService; import java.util.concurrent.ScheduledFuture; import java.util.concurrent.ScheduledThreadPoolExecutor; +import java.util.concurrent.ThreadFactory; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; @@ -33,19 +34,29 @@ import java.util.concurrent.atomic.AtomicBoolean; * it moves the queue off the shared scheduler but keeps a queue insertion and removal on every * read. * - *

The watchdog wakes on a thread of its own: one daemon thread, {@code - * httpclient-jdk-read-idle}, shared by every JDK stream and started with the first (re-review - * 2026-11-04 RR3-03-01). It used to wake through {@code CompletableFuture.delayedExecutor}, whose - * task runs on the common {@code ForkJoinPool}; a service that filled that pool with blocking work - * — {@code supplyAsync(blockingCall)} without an executor — held every timeout until a pool thread - * came free, and a 300 ms {@code read-idle} ended after 2.7 s. The wake is O(1), and the close it - * may do waits on no I/O — the JDK body marks itself closed, cancels its subscription and wakes the - * read — so both run on this thread, as the per-read timer before RR-03-13 closed on the JDK's - * delay thread. Rejected: {@code delayedExecutor(delay, unit, Runnable::run)}, which judges on the - * JDK's own delay thread — that thread fires every {@code CompletableFuture} timeout in the - * process, so a slow close there would delay them all, and it returns no handle to take a closed - * stream's wake off the queue. Closing the stream cancels its pending wake, so a closed stream is - * not kept reachable until its {@code read-idle} passes. + *

The watchdog wakes on a thread of its own: at most one daemon thread, {@code + * httpclient-jdk-read-idle}, shared by every JDK stream (re-review 2026-11-04 RR3-03-01). It used + * to wake through {@code CompletableFuture.delayedExecutor}, whose task runs on the common {@code + * ForkJoinPool}; a service that filled that pool with blocking work — {@code + * supplyAsync(blockingCall)} without an executor — held every timeout until a pool thread came + * free, and a 300 ms {@code read-idle} ended after 2.7 s. The wake is O(1), and the close it may do + * waits on no I/O — the JDK body marks itself closed, cancels its subscription and wakes the read — + * so both run on this thread, as the per-read timer before RR-03-13 closed on the JDK's delay + * thread. Rejected: {@code delayedExecutor(delay, unit, Runnable::run)}, which judges on the JDK's + * own delay thread — that thread fires every {@code CompletableFuture} timeout in the process, so a + * slow close there would delay them all, and it returns no handle to take a closed stream's wake + * off the queue. Closing the stream cancels its pending wake, so a closed stream is not kept + * reachable until its {@code read-idle} passes. + * + *

The thread's lifetime (RR4-03-04). The first stream that schedules a wake starts it; once no + * wake has been queued for {@link #IDLE_THREAD_KEEP_ALIVE} it ends, and the next stream that needs + * it starts a new one. While reads keep coming a wake is always queued, so a busy service keeps one + * thread; an idle one keeps none. It used to live as long as the class did: the JVM's life for a + * runnable jar, but a WAR redeploy or a restart class loader left it holding the old loader. It + * takes nothing from the request thread that starts it — no inheritable thread-locals, and this + * class's own loader as its context loader — so it never keeps a request's state or a foreign + * loader alive while it waits. Restarting costs one thread start per idle spell, against a + * scheduled task per {@code read-idle} on every live stream. * *

{@link #skip} reads through this stream rather than delegating: the JDK body does not override * {@code skip}, so {@code InputStream.skip} ran many reads inside one guarded call and a long skip @@ -56,6 +67,9 @@ final class ReadIdleInputStream extends FilterInputStream { /** The largest scratch buffer {@link #skip} reads into, as {@code InputStream.skip} uses. */ private static final int SKIP_BUFFER_BYTES = 2048; + /** How long the watchdog thread outlives the last wake it had queued. */ + static final Duration IDLE_THREAD_KEEP_ALIVE = Duration.ofSeconds(60); + /** {@link #readStartedAt} while no read is waiting. */ private static final long NOT_READING = Long.MIN_VALUE; @@ -84,19 +98,38 @@ final class ReadIdleInputStream extends FilterInputStream { } /** - * A watchdog scheduler as every stream gets it: one daemon thread, and a cancelled wake leaves - * the queue at once instead of when it would have fired. + * A watchdog scheduler as every stream gets it: at most one daemon thread, which ends after + * {@link #IDLE_THREAD_KEEP_ALIVE} with nothing queued, and a cancelled wake leaves the queue at + * once instead of when it would have fired. */ static ScheduledThreadPoolExecutor newWatchdogs() { + return newWatchdogs(IDLE_THREAD_KEEP_ALIVE); + } + + /** As {@link #newWatchdogs()}, with the thread ending after {@code idleThreadKeepAlive}. */ + static ScheduledThreadPoolExecutor newWatchdogs(Duration idleThreadKeepAlive) { + ClassLoader ownLoader = ReadIdleInputStream.class.getClassLoader(); + // factory(), not the builder: the builder is not thread-safe, and any request thread may be + // the one whose schedule() starts the worker. + ThreadFactory threads = + Thread.ofPlatform() + .name("httpclient-jdk-read-idle") + .daemon(true) + .inheritInheritableThreadLocals(false) + .factory(); ScheduledThreadPoolExecutor scheduler = new ScheduledThreadPoolExecutor( 1, runnable -> { - Thread thread = new Thread(runnable, "httpclient-jdk-read-idle"); - thread.setDaemon(true); + Thread thread = threads.newThread(runnable); + thread.setContextClassLoader(ownLoader); return thread; }); scheduler.setRemoveOnCancelPolicy(true); + // The worker exits only when the queue is empty as well as idle (ThreadPoolExecutor keeps the + // last worker while a task is queued), and the next schedule() starts one again. + scheduler.setKeepAliveTime(idleThreadKeepAlive.toNanos(), TimeUnit.NANOSECONDS); + scheduler.allowCoreThreadTimeOut(true); return scheduler; } @@ -213,7 +246,7 @@ final class ReadIdleInputStream extends FilterInputStream { } } - /** The scheduler every stream shares, started with the first stream that is read. */ + /** The scheduler every stream shares; its thread comes and goes as described above. */ private static final class SharedWatchdogs { static final ScheduledExecutorService INSTANCE = newWatchdogs(); diff --git a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/profile/ClientRuntimeRegistry.java b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/profile/ClientRuntimeRegistry.java index 09bb0b65..8078fc62 100644 --- a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/profile/ClientRuntimeRegistry.java +++ b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/profile/ClientRuntimeRegistry.java @@ -7,8 +7,10 @@ import java.util.List; import java.util.Map; import java.util.NoSuchElementException; import java.util.Objects; +import java.util.Queue; import java.util.Set; import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentLinkedQueue; import java.util.concurrent.ConcurrentMap; import java.util.concurrent.Executors; import java.util.concurrent.ScheduledExecutorService; @@ -31,6 +33,9 @@ public final class ClientRuntimeRegistry implements AutoCloseable { new ConcurrentHashMap<>(); private final AtomicReference drainScheduler = new AtomicReference<>(); + /** Threads the drain scheduler created, so close can wait for them to exit (stopDrainThread). */ + private final Queue drainThreads = new ConcurrentLinkedQueue<>(); + /** * Generations that have been replaced but are not yet closed. * @@ -140,21 +145,52 @@ public final class ClientRuntimeRegistry implements AutoCloseable { } retired.clear(); runtimes.clear(); + // The drain thread is stopped whether or not a runtime refused to close. Throwing first left + // the scheduler, and its thread, running after a failed shutdown. + try { + stopDrainThread(); + } catch (RuntimeException failure) { + if (firstFailure == null) { + firstFailure = failure; + } else { + firstFailure.addSuppressed(failure); + } + } if (firstFailure != null) { throw firstFailure; } + } + + /** + * Stops the drain scheduler and waits for its thread to be gone. + * + *

A registry that returns while its drain thread is still alive leaks a thread per rotation + * cycle, which the resource-bound suite exists to catch. {@code awaitTermination} is not enough: + * the executor reports termination from inside the worker before that thread exits, so under load + * the thread could still be listed when close returned. Joining the thread is what makes "no + * thread left" true when this returns. + */ + private void stopDrainThread() { ScheduledExecutorService scheduler = drainScheduler.getAndSet(null); - if (scheduler != null) { - // Await termination: a registry that returns while its drain thread is still alive would - // leak a thread per rotation cycle, which the resource-bound suite exists to catch. - scheduler.shutdownNow(); - try { - if (!scheduler.awaitTermination(SHUTDOWN_AWAIT_MILLIS, TimeUnit.MILLISECONDS)) { - throw new IllegalStateException("http client drain scheduler did not terminate"); - } - } catch (InterruptedException interrupted) { - Thread.currentThread().interrupt(); + if (scheduler == null) { + return; + } + scheduler.shutdownNow(); + long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(SHUTDOWN_AWAIT_MILLIS); + try { + if (!scheduler.awaitTermination(SHUTDOWN_AWAIT_MILLIS, TimeUnit.MILLISECONDS)) { + throw new IllegalStateException("http client drain scheduler did not terminate"); } + for (Thread thread : drainThreads) { + long remaining = TimeUnit.NANOSECONDS.toMillis(deadline - System.nanoTime()); + thread.join(Math.max(1L, remaining)); + if (thread.isAlive()) { + throw new IllegalStateException("http client drain thread did not exit"); + } + } + drainThreads.clear(); + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); } } @@ -176,6 +212,7 @@ public final class ClientRuntimeRegistry implements AutoCloseable { runnable -> { Thread thread = new Thread(runnable, "httpclient-runtime-drain"); thread.setDaemon(true); + drainThreads.add(thread); return thread; }); if (drainScheduler.compareAndSet(null, created)) { diff --git a/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/jdk/ReadIdleInputStreamTest.java b/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/jdk/ReadIdleInputStreamTest.java index 3f9ab53f..fc45ca49 100644 --- a/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/jdk/ReadIdleInputStreamTest.java +++ b/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/jdk/ReadIdleInputStreamTest.java @@ -6,7 +6,11 @@ import static org.assertj.core.api.Assertions.catchThrowable; import java.io.IOException; import java.io.InputStream; import java.net.SocketTimeoutException; +import java.net.URL; +import java.net.URLClassLoader; import java.time.Duration; +import java.util.Arrays; +import java.util.List; import java.util.concurrent.CountDownLatch; import java.util.concurrent.ForkJoinPool; import java.util.concurrent.ScheduledThreadPoolExecutor; @@ -18,13 +22,15 @@ import org.junit.jupiter.api.Test; * What the JDK transport's read-idle bound promises: each read that waits longer than {@code * read-idle} for the server ends the body, and nothing else does (re-review 2026-11-04 RR-03-12, * RR-03-13); the watchdog keeps that time whatever the common pool is doing, and leaves nothing - * scheduled for a stream that is closed or timed out (RR3-03-01). + * scheduled for a stream that is closed or timed out (RR3-03-01); and its thread ends when no + * stream needs it and holds nothing of the request thread that started it (RR4-03-04). */ class ReadIdleInputStreamTest { private static final Duration READ_IDLE = Duration.ofMillis(300); private static final Duration GAP = Duration.ofMillis(20); private static final Duration COMMON_POOL_HOLD = Duration.ofSeconds(3); + private static final Duration KEEP_ALIVE = Duration.ofMillis(50); @Test @DisplayName("데이터가 read-idle보다 짧은 간격으로 오면 read-idle보다 오래 걸리는 skip도 끝까지 건너뛴다") @@ -72,13 +78,16 @@ class ReadIdleInputStreamTest { } @Test - @DisplayName("공용 ForkJoinPool이 블로킹 작업으로 차 있어도 멈춘 read는 read-idle 무렵에 끝난다") + @DisplayName("공용 ForkJoinPool이 블로킹 작업으로 차 있어도 멈춘 read는 read-idle 무렵에 watchdog 전용 스레드에서 끝난다") void aStalledReadTimesOutOnTimeWhileTheCommonPoolIsBlocked() throws Exception { // given // Every common-pool worker parks on a latch, which does not make the pool add a spare thread — // what a service's supplyAsync(blockingCall) without an executor does. The hold is bounded so // a watchdog that needs a pool worker still ends the read, late (re-review 2026-11-04 - // RR3-03-01: 300 ms read-idle timed out after 2,699 ms). + // RR3-03-01: 300 ms read-idle timed out after 2,699 ms). The timing alone does not catch that + // where the common pool's parallelism is 1 (two CPUs): the JDK then runs a delayed task on a + // new thread each time, on time. The thread that closed the stream does, on any machine + // (RR4-03-01). int workers = Math.max(1, ForkJoinPool.getCommonPoolParallelism()); CountDownLatch occupied = new CountDownLatch(workers); CountDownLatch release = new CountDownLatch(1); @@ -91,7 +100,8 @@ class ReadIdleInputStreamTest { }); } assertThat(occupied.await(5, TimeUnit.SECONDS)).as("every common-pool worker is held").isTrue(); - InputStream body = new ReadIdleInputStream(new StalledStream(), READ_IDLE); + StalledStream stalled = new StalledStream(); + InputStream body = new ReadIdleInputStream(stalled, READ_IDLE); long started = System.nanoTime(); try { // when @@ -101,6 +111,7 @@ class ReadIdleInputStreamTest { Duration waited = Duration.ofNanos(System.nanoTime() - started); assertThat(failure).isInstanceOf(SocketTimeoutException.class); assertThat(waited).isGreaterThanOrEqualTo(READ_IDLE).isLessThan(READ_IDLE.multipliedBy(3)); + assertThat(stalled.closedBy()).isEqualTo("httpclient-jdk-read-idle"); } finally { release.countDown(); } @@ -144,6 +155,93 @@ class ReadIdleInputStreamTest { } } + @Test + @DisplayName("기다리는 스트림이 없으면 watchdog 스레드는 유지 시간 뒤 끝난다") + void theWatchdogThreadEndsWhenNoStreamNeedsIt() throws Exception { + // given + // A thread that never ended kept this class's loader alive for the life of the JVM — a WAR + // redeploy or a restart class loader leaked the old one (re-review 2026-11-04 RR4-03-04). + ScheduledThreadPoolExecutor watchdogs = ReadIdleInputStream.newWatchdogs(KEEP_ALIVE); + try { + try (InputStream body = + new ReadIdleInputStream(new DrippingStream(1, GAP), READ_IDLE, watchdogs)) { + body.read(); + } + + // when + int threadsLeft = poolSizeOnceSettled(watchdogs, KEEP_ALIVE.multipliedBy(40)); + + // then + assertThat(threadsLeft).isZero(); + } finally { + watchdogs.shutdownNow(); + } + } + + @Test + @DisplayName("watchdog 스레드가 끝난 뒤의 스트림도 멈춘 read를 read-idle로 끝낸다") + void aStreamAfterTheWatchdogThreadEndedStillTimesOut() throws Exception { + // given + ScheduledThreadPoolExecutor watchdogs = ReadIdleInputStream.newWatchdogs(KEEP_ALIVE); + try { + try (InputStream first = + new ReadIdleInputStream(new DrippingStream(1, GAP), READ_IDLE, watchdogs)) { + first.read(); + } + assertThat(poolSizeOnceSettled(watchdogs, KEEP_ALIVE.multipliedBy(40))) + .as("the first thread has ended") + .isZero(); + InputStream next = new ReadIdleInputStream(new StalledStream(), READ_IDLE, watchdogs); + + // when + Throwable failure = catchThrowable(next::read); + + // then + assertThat(failure) + .isInstanceOf(SocketTimeoutException.class) + .hasMessageContaining("read-idle"); + } finally { + watchdogs.shutdownNow(); + } + } + + @Test + @DisplayName( + "watchdog 스레드는 자기를 만든 요청 스레드의 inheritable thread-local과 context class loader를 물려받지 않는다") + void theWatchdogThreadHoldsNothingOfTheThreadThatStartedIt() throws Exception { + // given + // The thread is started by whichever request thread schedules the first wake, and now again + // after every idle spell. Inheriting from it kept that request's inheritable thread-locals + // and its context class loader for the thread's whole life (RR4-03-04). + InheritableThreadLocal requestScoped = new InheritableThreadLocal<>(); + requestScoped.set("request-42"); + Thread current = Thread.currentThread(); + ClassLoader requestLoader = current.getContextClassLoader(); + ScheduledThreadPoolExecutor watchdogs = ReadIdleInputStream.newWatchdogs(KEEP_ALIVE); + try (URLClassLoader foreign = new URLClassLoader(new URL[0], null)) { + current.setContextClassLoader(foreign); + List seen; + try { + // when + seen = + watchdogs + .submit( + () -> + Arrays.asList( + requestScoped.get(), Thread.currentThread().getContextClassLoader())) + .get(5, TimeUnit.SECONDS); + } finally { + current.setContextClassLoader(requestLoader); + } + + // then + assertThat(seen).containsExactly(null, ReadIdleInputStream.class.getClassLoader()); + } finally { + requestScoped.remove(); + watchdogs.shutdownNow(); + } + } + @Test @DisplayName("호출자가 read 사이에 오래 머문 시간은 read-idle로 세지 않는다") void timeTheCallerSpendsBetweenReadsIsNotIdle() throws Exception { @@ -160,6 +258,16 @@ class ReadIdleInputStreamTest { } } + /** The scheduler's thread count once its keep-alive has had {@code atMost} to end them. */ + private static int poolSizeOnceSettled(ScheduledThreadPoolExecutor watchdogs, Duration atMost) + throws InterruptedException { + long deadline = System.nanoTime() + atMost.toNanos(); + while (watchdogs.getPoolSize() > 0 && System.nanoTime() < deadline) { + TimeUnit.MILLISECONDS.sleep(10); + } + return watchdogs.getPoolSize(); + } + private static void awaitQuietly(CountDownLatch latch, Duration atMost) { try { latch.await(atMost.toNanos(), TimeUnit.NANOSECONDS); @@ -212,6 +320,12 @@ class ReadIdleInputStreamTest { private static final class StalledStream extends InputStream { private final CountDownLatch closed = new CountDownLatch(1); + private volatile String closedBy; + + /** The name of the thread that closed this stream first, or {@code null} while open. */ + String closedBy() { + return closedBy; + } @Override public int read() throws IOException { @@ -231,6 +345,9 @@ class ReadIdleInputStreamTest { @Override public void close() { + if (closedBy == null) { + closedBy = Thread.currentThread().getName(); + } closed.countDown(); } } diff --git a/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/profile/ClientRuntimeRegistryTest.java b/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/profile/ClientRuntimeRegistryTest.java index 6a992e40..8b3a2501 100644 --- a/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/profile/ClientRuntimeRegistryTest.java +++ b/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/profile/ClientRuntimeRegistryTest.java @@ -1,6 +1,7 @@ package dev.caskeleton.adapter.outbound.httpclient.profile; import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.catchThrowable; import dev.caskeleton.adapter.outbound.httpclient.testkit.ClientProfiles; import java.time.Duration; @@ -55,10 +56,38 @@ class ClientRuntimeRegistryTest { assertThat(first.state()).isEqualTo(ClientRuntimeState.CLOSED); assertThat(second.state()).isEqualTo(ClientRuntimeState.CLOSED); - Set threadNames = - Thread.getAllStackTraces().keySet().stream() - .map(Thread::getName) - .collect(Collectors.toSet()); - assertThat(threadNames).noneMatch(name -> name.startsWith("httpclient-runtime-drain")); + assertThat(liveDrainThreads()).isEmpty(); + } + + @Test + void aRuntimeThatRefusesToCloseStillLeavesNoDrainThread() { + // given + ClientRuntime first = running(1); + ClientRuntime refusing = + new ClientRuntime( + ClientProfiles.builder("payment").build(), + new RuntimeGeneration(2), + () -> { + throw new IllegalStateException("pool refused to close"); + }); + ClientRuntimeRegistry registry = new ClientRuntimeRegistry(Map.of(first.name(), first)); + ClientRuntimeLease lease = registry.acquire(first.name()); + registry.swap(first.name(), refusing, Duration.ofSeconds(30)); + + // when + Throwable failure = catchThrowable(registry::close); + + // then + lease.close(); + assertThat(failure).hasMessage("pool refused to close"); + assertThat(liveDrainThreads()).isEmpty(); + } + + private static Set liveDrainThreads() { + return Thread.getAllStackTraces().keySet().stream() + .filter(Thread::isAlive) + .map(Thread::getName) + .filter(name -> name.startsWith("httpclient-runtime-drain")) + .collect(Collectors.toSet()); } } diff --git a/src/adapter/outbound/persistence-jpa/README.md b/src/adapter/outbound/persistence-jpa/README.md index 01755f23..62dae4eb 100644 --- a/src/adapter/outbound/persistence-jpa/README.md +++ b/src/adapter/outbound/persistence-jpa/README.md @@ -235,9 +235,16 @@ failure translation contract 다. 여기서는 SPI 구조와 fallback 근거만 실제로 돌려 이 경우를 본다. reaper 는 provider 가 `postgresql` 일 때만 돈다. - **lease 가 지난 실행**: lease 가 지난 `EXECUTING` 행은 넘겨받지 않고 `RecoveryRequired` 로 답하며, 행은 바꾸지 않는다. 주인이 늦게 complete·markFailed 하면 그대로 기록되고, 다음 claim 은 그 결과를 - 재생하거나 재시도한다. renew 는 이미 끝난 lease 를 늘리지 않는다. inspect 는 남의 lease 지난 실행을 + 재생하거나 재시도한다. renew 는 이미 끝난 lease 를 늘리지 않고, lease 가 지난 `CLAIMED` 는 주인도 시작할 + 수 없다 — 둘 다 `LEASE_EXPIRED`(전에는 `NOT_OWNER`, 재리뷰 RR5-01-02). inspect 는 남의 lease 지난 실행을 `ABANDONED` 로 보고한다. Redis store 도 같다(`IdempotencyStoreContract`, RR-01-11, 2026-11-04 소유자 결정 — 전에는 행을 `ABANDONED`·`EXPIRED_EXECUTION` 으로 바꿔 늦은 결과를 버렸다). +- **한 트랜잭션 안의 모양**: executor 를 유스케이스의 쓰기 트랜잭션 하나 안에서 부르면(템플릿의 모양) + claim 행은 그 트랜잭션이 끝날 때까지 커밋되지 않는다. 그래서 위의 "lease 가 지난 실행"을 다른 요청이 + 보는 일이 없다: 같은 키 요청은 `on conflict` 에서 기다렸다가 커밋이면 재생, 롤백이면 새 실행을 받는다. + 롤백(작업 예외, 쓰기 트랜잭션 제한 30s)은 executor 가 남긴 실패 기록도 지운다. 이 store 가 막는 중복은 + 같은 트랜잭션 안의 DB 효과이고, 트랜잭션 밖 효과는 다음 요청에서 다시 일어날 수 있다 + (`PostgreSqlIdempotencyIntegrationTest`, 재리뷰 RR5-01-05, `docs/web/http-contract.md` §2). ## outbox — `JpaOutboxStore` 외 @@ -415,7 +422,8 @@ machine-readable SSOT다(개수는 registry가 소유한다). `selected` base ca `PostgreSqlIdempotencyStore`는 row lock을 얻은 뒤 `clock_timestamp()`를 평가한다. claim takeover와 start/renew/complete/fail/release는 scope/state/owner/attempt/claim operation/ state revision을 SQL predicate로 다시 검증한다. expired `CLAIMED`만 takeover하며 expired -`EXECUTING`은 `ABANDONED`로 닫고 reconciliation을 요구한다. raw client key는 저장하지 않고 +`EXECUTING`은 행을 바꾸지 않고 reconciliation(`RecoveryRequired`)을 요구한다(2026-11-04 소유자 결정 — +이 문단은 그 전의 `ABANDONED`로 닫던 동작을 적고 있었다). raw client key는 저장하지 않고 versioned HMAC scope digest만 쓴다. Redis store·in-memory fake와 같은 계약(`IdempotencyStoreContract`)을 실 PostgreSQL에서 통과한다 diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/idempotency/postgresql/IdempotencyTransitionGateway.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/idempotency/postgresql/IdempotencyTransitionGateway.java index 33282cb7..cd59e4de 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/idempotency/postgresql/IdempotencyTransitionGateway.java +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/idempotency/postgresql/IdempotencyTransitionGateway.java @@ -12,7 +12,10 @@ import org.springframework.jdbc.core.JdbcOperations; * revision — in its {@code where} clause, so the update count is the answer: one row means * this owner was still the owner at this revision, zero means something else moved the record and * the caller must not treat its own view as current. Reading the row and then updating on the scope - * alone would let a worker whose lease expired overwrite the state of the one that took over. + * alone would let a worker whose lease expired overwrite the state of the one that took over. Start + * and renew also require the lease to be running; the store has already compared the locked row + * with the owner tuple before it runs them, so for those two a zero is the lease that ran out + * ({@code LEASE_EXPIRED}). * *

Collected here rather than in the store because they are one family: same guard, same * interpretation of the count, same reason a caller may not skip the guard. The store decides which diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/idempotency/postgresql/PostgreSqlIdempotencyStore.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/idempotency/postgresql/PostgreSqlIdempotencyStore.java index 1119663f..cbc2f035 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/idempotency/postgresql/PostgreSqlIdempotencyStore.java +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/idempotency/postgresql/PostgreSqlIdempotencyStore.java @@ -194,9 +194,13 @@ public final class PostgreSqlIdempotencyStore implements IdempotencyStorePort { if (mismatch != null) { return startResult(mismatch, null); } + // The row is locked and the owner, revision and state already match it, so the one condition + // left in START_SQL is the lease: a claim whose lease ran out is not started — nothing runs, + // and the next claim takes it over. It used to be answered NOT_OWNER, which says the record + // went to somebody else (re-review 2026-11-04 README 15, RR5-01-02). int updated = transitions.start(owner, operationId.value(), startDigest); if (updated != 1) { - return startResult(IdempotencyStartOutcome.NOT_OWNER, null); + return startResult(IdempotencyStartOutcome.LEASE_EXPIRED, null); } return startResult( IdempotencyStartOutcome.STARTED, owner.withStateRevision(owner.stateRevision() + 1)); @@ -233,9 +237,12 @@ public final class PostgreSqlIdempotencyStore implements IdempotencyStorePort { if (mismatch != null) { return renewResult(mismatch, null); } + // As for the start: with the row locked and matched, a renewal that updates nothing found the + // lease already ended. The holder still owns the row and still completes or fails it, so this + // is LEASE_EXPIRED, not NOT_OWNER (RR5-01-02). int updated = transitions.renew(owner, processingLeaseTtl, operationId.value(), renewDigest); if (updated != 1) { - return renewResult(IdempotencyRenewOutcome.NOT_OWNER, null); + return renewResult(IdempotencyRenewOutcome.LEASE_EXPIRED, null); } return renewResult( IdempotencyRenewOutcome.RENEWED, owner.withStateRevision(owner.stateRevision() + 1)); diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlIdempotencyIntegrationTest.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlIdempotencyIntegrationTest.java index e0e861c7..c231797e 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlIdempotencyIntegrationTest.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlIdempotencyIntegrationTest.java @@ -1,6 +1,7 @@ package dev.caskeleton.adapter.outbound.persistence.readiness; import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.catchThrowable; import dev.caskeleton.adapter.outbound.persistence.idempotency.postgresql.PostgreSqlIdempotencyStore; import dev.caskeleton.application.idempotency.IdempotencyClaimAttempt; @@ -23,6 +24,7 @@ import java.time.Duration; import java.util.concurrent.CountDownLatch; import java.util.concurrent.Executors; import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; import org.flywaydb.core.Flyway; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; @@ -254,6 +256,131 @@ class PostgreSqlIdempotencyIntegrationTest { .isEqualTo(1); } + /** + * The shape the template calls the store in — the executor inside the use case's one write + * transaction — never shows another request an execution past its lease: the claim row is not + * committed until the first request ends, so a same-key request waits on it instead of reading + * it. What decision 2026-11-04 describes (500 now, the late result replayed later) is the Redis + * store's; here the waiting request gets the replay directly (re-review 2026-11-04 RR5-01-05). + */ + @Test + void aSameKeyClaimWaitsForTheFirstTransactionPastItsLeaseAndReplaysTheResult() throws Exception { + // given + Duration lease = Duration.ofMillis(200); + CountDownLatch firstStarted = new CountDownLatch(1); + CountDownLatch allowFirstToFinish = new CountDownLatch(1); + IdempotencyClaimRequest firstRequest = + request(store.newClaimAttempt(new OperationId("claim-slow")), lease); + IdempotencyClaimRequest secondRequest = + request(store.newClaimAttempt(new OperationId("claim-resent")), lease); + + try (var executor = Executors.newVirtualThreadPerTaskExecutor()) { + Future first = + executor.submit( + () -> { + transactions.executeWithoutResult( + ignored -> { + IdempotencyOwner owner = + ((IdempotencyClaimOutcome.Acquired) store.claim(firstRequest)).owner(); + IdempotencyOwner executing = + store + .markExecutionStarted(owner, new OperationId("start-slow")) + .owner() + .orElseThrow(); + firstStarted.countDown(); + await(allowFirstToFinish); + store.complete( + executing, + new StoredResponse("slow-but-done"), + Duration.ofHours(1), + new OperationId("complete-slow")); + }); + return null; + }); + firstStarted.await(); + Future second = + executor.submit(() -> transactions.execute(ignored -> store.claim(secondRequest))); + // Long past the 200 ms lease, and the resend is still waiting on the uncommitted claim row. + TimeUnit.MILLISECONDS.sleep(600); + assertThat(second.isDone()).as("the resend waits instead of answering").isFalse(); + allowFirstToFinish.countDown(); + first.get(); + + // when + IdempotencyClaimOutcome resent = second.get(); + + // then + assertThat(resent) + .isInstanceOfSatisfying( + IdempotencyClaimOutcome.CompletedReplay.class, + replay -> assertThat(replay.response().payload()).isEqualTo("slow-but-done")); + } + } + + /** + * The other side of the same shape: a first request whose transaction rolls back — the action + * threw, or the write transaction outlived its limit (30s, the default lease too) — takes its + * claim row with it, and with it the failure the executor recorded before rethrowing, even an + * unknown effect. The next same-key request runs the work afresh. Database effects rolled back + * with it; anything the action did outside the transaction may then happen twice (re-review + * 2026-11-04 RR5-01-05). + */ + @Test + void aSameKeyClaimAfterTheFirstTransactionRolledBackItsAbandonedRecordRunsAfresh() + throws Exception { + // given + CountDownLatch firstStarted = new CountDownLatch(1); + CountDownLatch allowFirstToFail = new CountDownLatch(1); + IdempotencyClaimRequest firstRequest = + request(store.newClaimAttempt(new OperationId("claim-fails")), Duration.ofSeconds(5)); + IdempotencyClaimRequest secondRequest = + request(store.newClaimAttempt(new OperationId("claim-after")), Duration.ofSeconds(5)); + + try (var executor = Executors.newVirtualThreadPerTaskExecutor()) { + Future first = + executor.submit( + () -> { + transactions.executeWithoutResult( + ignored -> { + IdempotencyOwner owner = + ((IdempotencyClaimOutcome.Acquired) store.claim(firstRequest)).owner(); + IdempotencyOwner executing = + store + .markExecutionStarted(owner, new OperationId("start-fails")) + .owner() + .orElseThrow(); + firstStarted.countDown(); + await(allowFirstToFail); + // What the executor does with an action that threw: record the unknown + // effect, then rethrow — which rolls the record back too. + store.markFailed( + executing, + IdempotencyFailureDisposition.EFFECT_UNKNOWN_ABANDONED, + Duration.ofHours(1), + new OperationId("fail-unknown")); + throw new IllegalStateException("the action failed"); + }); + return null; + }); + firstStarted.await(); + Future second = + executor.submit(() -> transactions.execute(ignored -> store.claim(secondRequest))); + allowFirstToFail.countDown(); + assertThat(catchThrowable(first::get)) + .as("the first request's transaction rolled back") + .hasRootCauseInstanceOf(IllegalStateException.class); + + // when + IdempotencyClaimOutcome resent = second.get(); + + // then + assertThat(resent) + .isInstanceOfSatisfying( + IdempotencyClaimOutcome.Acquired.class, + acquired -> assertThat(acquired.owner().attempt()).isEqualTo(1)); + } + } + @Test void aRetriedFailureWithTheSameArgumentsIsConfirmedRatherThanAppliedTwice() { IdempotencyOwner executing = claimAndStart("claim-fail-replay", "start-fail-replay"); diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlMigrationLockWaitIntegrationTest.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlMigrationLockWaitIntegrationTest.java index 73ccad04..d396a512 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlMigrationLockWaitIntegrationTest.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlMigrationLockWaitIntegrationTest.java @@ -4,6 +4,9 @@ import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.catchThrowable; import com.zaxxer.hikari.HikariDataSource; +import dev.caskeleton.adapter.outbound.persistence.testkit.migration.LockWaitMigrationInspector; +import java.nio.file.Files; +import java.nio.file.Path; import java.sql.Connection; import java.sql.SQLException; import java.sql.Statement; @@ -19,8 +22,11 @@ import org.flywaydb.core.Flyway; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.CsvSource; +import org.junit.jupiter.params.provider.ValueSource; import org.springframework.jdbc.core.JdbcTemplate; /** @@ -35,12 +41,22 @@ import org.springframework.jdbc.core.JdbcTemplate; * migration ends up applied. Bounded, it fails with {@code lock_not_available} long before that, * Flyway rolls it back whole, and the history still ends one version below. V21 has the same case * against a writer in {@code PostgreSqlOutboxOrderKeyRunbookIntegrationTest}. + * + *

The text check ({@link LockWaitMigrationInspector}) repeats which statements Flyway runs + * outside a transaction, where {@code SET LOCAL} bounds nothing. Two cases here hold that copy to + * Flyway itself on this server: the statements it names are exactly those Flyway refuses next to a + * {@code SET LOCAL} (default {@code mixed=false}), and a {@code VACUUM FULL} that Flyway runs + * anyway ({@code mixed=true}) waits out the reader despite its {@code SET LOCAL} (re-review + * 2026-11-03 round 4, RR4-02-01). */ class PostgreSqlMigrationLockWaitIntegrationTest { private static final String LOCK_NOT_AVAILABLE = "55P03"; + private static final String ACTIVE_SQL_TRANSACTION = "25001"; private static final Duration READER_STAYS = Duration.ofSeconds(15); private static final Duration GIVES_UP_WITHIN = Duration.ofSeconds(10); + private static final String FLYWAY_REFUSES_A_MIXED_MIGRATION = + "Detected both transactional and non-transactional statements"; private static PostgreSqlReadinessSupport postgres; @@ -110,6 +126,130 @@ class PostgreSqlMigrationLockWaitIntegrationTest { } } + @ParameterizedTest + @ValueSource( + strings = { + "VACUUM FULL probe", + "vacuum (full) probe", + "VACUUM ANALYZE probe", + "CREATE INDEX CONCURRENTLY ix_probe ON probe (id)", + "CREATE UNIQUE INDEX CONCURRENTLY ux_probe ON probe (id)", + "DROP INDEX CONCURRENTLY IF EXISTS ix_probe", + "REINDEX SCHEMA public", + "REINDEX (VERBOSE) DATABASE probe", + "CREATE DATABASE probe_other", + "DROP DATABASE IF EXISTS probe_other", + "ALTER SYSTEM SET work_mem = '8MB'", + "DISCARD ALL", + "REINDEX TABLE CONCURRENTLY probe", + "REINDEX (CONCURRENTLY) TABLE probe", + "REFRESH MATERIALIZED VIEW CONCURRENTLY probe_view", + "ALTER TYPE probe_status ADD VALUE 'b'", + "ANALYZE probe", + "ALTER TABLE probe ADD COLUMN c INT", + "CREATE INDEX ix_probe_plain ON probe (id)", + "COMMENT ON TABLE probe IS 'vacuum full'", + "DO $$ BEGIN PERFORM 1; END $$" + }) + @DisplayName("검사기가 트랜잭션 밖이라고 보는 문장은 Flyway가 SET LOCAL 옆에서 거부하는 문장과 같다") + void theInspectorNamesExactlyTheStatementsFlywayRunsOutsideATransaction( + String statement, @TempDir Path stream) throws Exception { + try (HikariDataSource database = postgres.createDatabase(databaseName())) { + // given + new JdbcTemplate(database) + .execute( + "CREATE TABLE probe (id INT PRIMARY KEY);" + + " CREATE MATERIALIZED VIEW probe_view AS SELECT id FROM probe;" + + " CREATE UNIQUE INDEX ux_probe_view ON probe_view (id);" + + " CREATE TYPE probe_status AS ENUM ('a')"); + String sql = "SET LOCAL lock_timeout = '5s';\n" + statement + ";\n"; + Files.writeString(stream.resolve("V1__probe.sql"), sql); + Flyway flyway = + Flyway.configure() + .dataSource(database) + .locations("filesystem:" + stream.toAbsolutePath()) + .baselineOnMigrate(true) + .baselineVersion("0") + .load(); + + // when + Throwable thrown = catchThrowable(flyway::migrate); + + // then + assertThat( + new LockWaitMigrationInspector() + .statementsFlywayRunsOutsideATransaction(sql) + .isEmpty()) + .as( + "Flyway %s %s outside a transaction (%s)", + refusedAsMixed(thrown) ? "runs" : "does not run", statement, thrown) + .isEqualTo(!refusedAsMixed(thrown)); + } + } + + @Test + @DisplayName( + "mixed=true로 Flyway가 트랜잭션 밖에서 돌린 VACUUM FULL은 SET LOCAL lock_timeout이 있어도 읽는 트랜잭션이 끝날 때까지 기다린다") + void aVacuumFullFlywayRunsOutsideATransactionWaitsOutTheReaderDespiteSetLocal( + @TempDir Path stream) throws Exception { + try (HikariDataSource database = postgres.createDatabase(databaseName())) { + // given + new JdbcTemplate(database).execute("CREATE TABLE probe (id INT PRIMARY KEY)"); + Files.writeString( + stream.resolve("V1__vacuum.sql"), "SET LOCAL lock_timeout = '1s';\nVACUUM FULL probe;\n"); + Flyway mixed = + Flyway.configure() + .dataSource(database) + .locations("filesystem:" + stream.toAbsolutePath()) + .baselineOnMigrate(true) + .baselineVersion("0") + .mixed(true) + .load(); + + // when + Attempt attempt = whileAReaderHolds(database, "probe", mixed::migrate); + + // then + assertThat(List.of(String.valueOf(attempt.thrown()), attempt.tookLessThan(GIVES_UP_WITHIN))) + .containsExactly("null", false); + } + } + + @Test + @DisplayName( + "Flyway는 REINDEX TABLE CONCURRENTLY를 트랜잭션 안에서 돌려 PostgreSQL이 거부하므로 executeInTransaction=false가 필요하다") + void flywayRunsReindexConcurrentlyInATransactionWherePostgresqlRefusesIt(@TempDir Path stream) + throws Exception { + try (HikariDataSource database = postgres.createDatabase(databaseName())) { + // given + new JdbcTemplate(database).execute("CREATE TABLE probe (id INT PRIMARY KEY)"); + Files.writeString(stream.resolve("V1__reindex.sql"), "REINDEX TABLE CONCURRENTLY probe;\n"); + Flyway flyway = + Flyway.configure() + .dataSource(database) + .locations("filesystem:" + stream.toAbsolutePath()) + .baselineOnMigrate(true) + .baselineVersion("0") + .load(); + + // when + Throwable thrown = catchThrowable(flyway::migrate); + + // then + assertThat(new Attempt(thrown, Duration.ZERO).sqlState()).isEqualTo(ACTIVE_SQL_TRANSACTION); + } + } + + /** Whether Flyway refused the migration for mixing statements it runs in and out of one. */ + private static boolean refusedAsMixed(Throwable thrown) { + for (Throwable cause = thrown; cause != null; cause = cause.getCause()) { + if (String.valueOf(cause.getMessage()).contains(FLYWAY_REFUSES_A_MIXED_MIGRATION)) { + return true; + } + } + return false; + } + /** * Runs the migration while another connection holds an open transaction that has read the table, * and that transaction ends on its own after {@link #READER_STAYS}. diff --git a/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/LockWaitMigrationInspectorTest.java b/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/LockWaitMigrationInspectorTest.java index 5cfb63d1..4d74ebbd 100644 --- a/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/LockWaitMigrationInspectorTest.java +++ b/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/LockWaitMigrationInspectorTest.java @@ -12,6 +12,7 @@ import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; import org.junit.jupiter.params.provider.ValueSource; /** @@ -48,7 +49,9 @@ class LockWaitMigrationInspectorTest { "REFRESH MATERIALIZED VIEW outbox_summary;", "CLUSTER outbox_event USING ix_outbox_event_status;", "VACUUM FULL outbox_event;", - "CREATE TABLE IF NOT EXISTS ledger (id BIGINT);\nALTER TABLE ledger ADD c INT;" + "CREATE TABLE IF NOT EXISTS ledger (id BIGINT);\nALTER TABLE ledger ADD c INT;", + "REINDEX (CONCURRENTLY false) TABLE outbox_event;", + "REINDEX (VERBOSE) TABLE outbox_event;" }) @DisplayName("이미 있는 테이블에 잠금을 거는 문장 앞에 SET LOCAL lock_timeout이 없으면 거부된다") void aLockOnAnExistingTableWithoutABoundBeforeItIsRefused(String sql) { @@ -80,7 +83,11 @@ class LockWaitMigrationInspectorTest { "SET LOCAL lock_timeout TO 5000;\nDROP TRIGGER trg_outbox ON outbox_event;", "-- ALTER TABLE outbox_event ADD COLUMN c INT;\n/* DELETE FROM outbox_event; */\n" + "COMMENT ON TABLE outbox_event IS 'update outbox_event set status';", - "CREATE TABLE poster_copy AS SELECT * FROM poster;" + "CREATE TABLE poster_copy AS SELECT * FROM poster;", + "REINDEX (CONCURRENTLY) TABLE outbox_event;", + "REINDEX (VERBOSE, CONCURRENTLY) INDEX ux_outbox_event_aggregate_order;", + "GRANT SELECT, REFERENCES ON TABLE outbox_event TO reader;", + "REVOKE REFERENCES ON outbox_event FROM reader;" }) @DisplayName("잠금 대기에 상한을 두었거나 잠글 기존 테이블이 없으면 통과한다") void aBoundedOrLockFreeMigrationPasses(String sql) { @@ -168,4 +175,125 @@ class LockWaitMigrationInspectorTest { .hasMessageContaining("delete from notification_suppression") .hasMessageContaining("alter table notification_suppression"); } + + @ParameterizedTest + @ValueSource( + strings = { + "SET LOCAL lock_timeout = '5s';\nALTER TABLE outbox_event ADD COLUMN c INT;\n" + + "CREATE INDEX CONCURRENTLY ix_outbox_c ON outbox_event (c);", + "SET LOCAL lock_timeout = '5s';\nALTER TABLE outbox_event ADD COLUMN c INT;\n" + + "VACUUM ANALYZE outbox_event;", + "SET LOCAL lock_timeout = '5s';\nDROP INDEX CONCURRENTLY IF EXISTS ix_old;\n" + + "ALTER TABLE outbox_event ADD COLUMN c INT;" + }) + @DisplayName("Flyway가 스스로 트랜잭션 밖에서 돌리는 문장이 있는 마이그레이션의 SET LOCAL lock_timeout은 상한으로 세지 않는다") + void setLocalInAMigrationFlywayRunsOutsideATransactionIsNotABound(String sql) { + // given + MigrationResource migration = new MigrationResource("V99__change.sql", sql); + + // when + Throwable refused = catchThrowable(() -> inspector.requireBoundedLockWait(migration)); + + // then + assertThat(refused) + .isInstanceOf(IllegalStateException.class) + .hasMessageContaining("alter table outbox_event") + .hasMessageContaining("Flyway runs this migration outside a transaction"); + } + + @ParameterizedTest + @ValueSource( + strings = { + "SET LOCAL lock_timeout = '5s';\nVACUUM FULL outbox_event;", + "VACUUM (FULL, ANALYZE) outbox_event;", + "SET LOCAL lock_timeout = '5s';\nREINDEX SCHEMA public;" + }) + @DisplayName("트랜잭션 안에서 돌 수 없는 잠금 문장은 마이그레이션에 두지 말라는 이유로 거부된다") + void aLockingStatementThatCannotRunInATransactionIsRefusedAsAnOperationsStep(String sql) { + // given + MigrationResource migration = new MigrationResource("V99__change.sql", sql); + + // when + Throwable refused = catchThrowable(() -> inspector.requireBoundedLockWait(migration)); + + // then + assertThat(refused) + .isInstanceOf(IllegalStateException.class) + .hasMessageContaining("cannot run in a transaction") + .hasMessageContaining("keep it out of migrations"); + } + + @ParameterizedTest + @CsvSource( + delimiter = '|', + quoteCharacter = '"', + value = { + "VACUUM FULL outbox_event | vacuum", + "vacuum (full) outbox_event | vacuum", + "CREATE INDEX CONCURRENTLY ix ON outbox_event (status) | create index concurrently", + "CREATE UNIQUE INDEX CONCURRENTLY ux ON outbox_event (id) | create unique index concurrently", + "DROP INDEX CONCURRENTLY IF EXISTS ix | drop index concurrently", + "REINDEX SCHEMA public | reindex schema", + "REINDEX (VERBOSE) DATABASE app | reindex database", + "CREATE DATABASE other | create database", + "ALTER SYSTEM SET work_mem = '8MB' | alter system", + "DISCARD ALL | discard all", + "REINDEX TABLE CONCURRENTLY outbox_event | \"\"", + "REINDEX (CONCURRENTLY) TABLE outbox_event | \"\"", + "REFRESH MATERIALIZED VIEW CONCURRENTLY outbox_summary | \"\"", + "ALTER TYPE outbox_status ADD VALUE 'PARKED' | \"\"", + "ANALYZE outbox_event | \"\"", + "COMMENT ON TABLE outbox_event IS 'vacuum full' | \"\"", + "DO $$ BEGIN PERFORM 1; END $$ | \"\"" + }) + @DisplayName("Flyway가 PostgreSQL 12 이상에서 트랜잭션 밖으로 빼는 문장만 그렇게 판정한다") + void onlyTheStatementsFlywayRunsOutsideATransactionAreNamed(String statement, String expected) { + // given + String sql = "SET LOCAL lock_timeout = '5s';\n" + statement + ";"; + + // when + List outside = inspector.statementsFlywayRunsOutsideATransaction(sql); + + // then + assertThat(outside).isEqualTo(expected.isEmpty() ? List.of() : List.of(expected)); + } + + @ParameterizedTest + @ValueSource( + strings = { + "executeInTransaction=FALSE\n", + "executeInTransaction = false\n", + "executeInTransaction: false\n", + " executeInTransaction false \n" + }) + @DisplayName("Flyway가 읽는 모든 모양의 executeInTransaction=false 설정에서 SET LOCAL lock_timeout은 상한이 아니다") + void everySpellingFlywayReadsAsOutsideATransactionMakesSetLocalNoBound( + String configuration, @TempDir Path stream) throws IOException { + // given + Path script = Files.writeString(stream.resolve("V99__change.sql"), BOUNDED_ALTER); + Files.writeString(stream.resolve("V99__change.sql.conf"), configuration); + + // when + List unbounded = inspector.unboundedLockingStatements(script); + + // then + assertThat(unbounded).singleElement(STRING).contains("alter table outbox_event"); + } + + @Test + @DisplayName("executeInTransaction=true 설정은 Flyway의 판정을 덮어 SET LOCAL lock_timeout을 상한으로 만든다") + void executeInTransactionTrueOverridesFlywaysDetection(@TempDir Path stream) throws IOException { + // given + Path script = + Files.writeString( + stream.resolve("V99__change.sql"), + BOUNDED_ALTER + "\nDROP INDEX CONCURRENTLY IF EXISTS ix_old;"); + Files.writeString(stream.resolve("V99__change.sql.conf"), "executeInTransaction=true\n"); + + // when + List unbounded = inspector.unboundedLockingStatements(script); + + // then + assertThat(unbounded).isEmpty(); + } } diff --git a/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/VersionedMigrationFilesTest.java b/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/VersionedMigrationFilesTest.java index e38d3faa..87c09f27 100644 --- a/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/VersionedMigrationFilesTest.java +++ b/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/VersionedMigrationFilesTest.java @@ -1,10 +1,12 @@ package dev.caskeleton.adapter.outbound.persistence.testkit.migration; import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.catchThrowable; import java.io.IOException; import java.nio.file.Files; import java.nio.file.Path; +import java.util.List; import java.util.Set; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; @@ -13,7 +15,8 @@ import org.junit.jupiter.api.io.TempDir; /** * One definition of a versioned migration for every check that walks a tree (re-review 2026-11-03 * round 3, RR3-02-03): the operating trees' checks once skipped {@code V22_1__} while the sample's - * counted it. + * counted it. The gRPC ledger's list in the optional build is compiled from this same source + * (README 12, r6-b; r9). */ class VersionedMigrationFilesTest { @@ -45,4 +48,49 @@ class VersionedMigrationFilesTest { "postgresql/V22_1__outbox_fix.sql", "postgresql/V22.2__outbox_index.sql"); } + + @Test + @DisplayName("버전 마이그레이션을 Flyway 순서대로, 버전의 각 부분을 수로 비교해 돌려준다") + void theMigrationsComeInFlywaysOrderEachVersionPartComparedAsANumber(@TempDir Path tree) + throws IOException { + // given + for (String file : + Set.of( + "V10__e.sql", + "V2.10__d.sql", + "V2_1__c.sql", + "V002__b.sql", + "V1__a.sql", + "V2__b.sql.conf", + "R__view.sql")) { + Files.writeString(tree.resolve(file), "SELECT 1;"); + } + + // when + List ordered = + VersionedMigrationFiles.inVersionOrder(tree).stream() + .map(file -> file.getFileName().toString()) + .toList(); + + // then + assertThat(ordered) + .containsExactly("V1__a.sql", "V002__b.sql", "V2_1__c.sql", "V2.10__d.sql", "V10__e.sql"); + } + + @Test + @DisplayName("Flyway가 같은 버전으로 보는 두 마이그레이션은 둘 다 이름을 대며 거부한다") + void twoMigrationsFlywayReadsAsOneVersionAreRefusedByName(@TempDir Path tree) throws IOException { + // given + Files.writeString(tree.resolve("V1__a.sql"), "SELECT 1;"); + Files.writeString(tree.resolve("V1_0__b.sql"), "SELECT 1;"); + + // when + Throwable refused = catchThrowable(() -> VersionedMigrationFiles.inVersionOrder(tree)); + + // then + assertThat(refused) + .isInstanceOf(IllegalStateException.class) + .hasMessageContaining("V1__a.sql") + .hasMessageContaining("V1_0__b.sql"); + } } diff --git a/src/adapter/outbound/persistence-jpa/src/testFixtures/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/LockWaitMigrationInspector.java b/src/adapter/outbound/persistence-jpa/src/testFixtures/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/LockWaitMigrationInspector.java index 60ca7711..034f068a 100644 --- a/src/adapter/outbound/persistence-jpa/src/testFixtures/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/LockWaitMigrationInspector.java +++ b/src/adapter/outbound/persistence-jpa/src/testFixtures/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/LockWaitMigrationInspector.java @@ -1,6 +1,7 @@ package dev.caskeleton.adapter.outbound.persistence.testkit.migration; import java.io.IOException; +import java.io.Reader; import java.nio.file.Files; import java.nio.file.Path; import java.util.ArrayList; @@ -10,6 +11,7 @@ import java.util.Locale; import java.util.Map; import java.util.NavigableMap; import java.util.Objects; +import java.util.Properties; import java.util.TreeMap; import java.util.regex.Matcher; import java.util.regex.Pattern; @@ -44,12 +46,23 @@ import java.util.regex.Pattern; * {@code CREATE TABLE IF NOT EXISTS} does not count as creating it, because the table may already * be there. The bound in force at a statement is the last {@code lock_timeout} setting before it, * and it counts only as {@code SET LOCAL} with a value that is not zero: {@code 0}, {@code DEFAULT} - * and {@code RESET} wait without end, a plain {@code SET} outlives the migration. A migration - * Flyway runs outside a transaction ({@code executeInTransaction=false} in its {@code .sql.conf}, - * read by {@link #unboundedLockingStatements(Path)}) has no bound at all: {@code SET LOCAL} there - * only warns. Comments and quoted literals are not read as statements. The check reads text, not a - * parse tree: it is a guard against the forgotten line for the statements above, not a proof that - * no other statement waits (re-review 2026-11-03 round 3, RR3-02-01). + * and {@code RESET} wait without end, a plain {@code SET} outlives the migration. Comments and + * quoted literals are not read as statements. The check reads text, not a parse tree: it is a guard + * against the forgotten line for the statements above, not a proof that no other statement waits + * (re-review 2026-11-03 round 3, RR3-02-01). + * + *

A migration Flyway runs outside a transaction has no bound at all: {@code SET LOCAL} there + * only warns. Flyway decides that the way {@link #statementsFlywayRunsOutsideATransaction(String)} + * repeats: {@code executeInTransaction} in the script's {@code .sql.conf} when there is one (read + * by {@link #unboundedLockingStatements(Path)}), and otherwise the script's own statements — one + * that PostgreSQL refuses inside a transaction ({@code VACUUM}, {@code CREATE INDEX CONCURRENTLY}, + * ...) puts the whole script outside one. With Flyway's default {@code mixed=false} such a script + * that holds any other statement is refused at deploy; with {@code mixed=true} it runs, unbounded. + * A locking statement that cannot run in a transaction itself ({@code VACUUM FULL}, a + * non-concurrent {@code REINDEX SCHEMA}) can never be bounded and is refused with the advice to + * keep it out of migrations (re-review 2026-11-03 round 4, RR4-02-01). The check assumes Flyway's + * defaults, as the template runs it: {@code executeInTransaction=true} and {@code mixed=false} for + * the whole run. */ public final class LockWaitMigrationInspector { @@ -61,6 +74,14 @@ public final class LockWaitMigrationInspector { /** A possibly qualified name, captured as group 1: the table the statement locks. */ private static final String NAME = "(" + OTHER_NAME + ")"; + /** + * A parenthesised option list ({@code REINDEX (VERBOSE) TABLE t}) that does not turn {@code + * CONCURRENTLY} on; {@code (CONCURRENTLY)} and {@code (CONCURRENTLY true)} do, {@code + * (CONCURRENTLY false)} does not. + */ + private static final String NOT_CONCURRENT_OPTIONS = + "\\((?![^)]*\\bconcurrently\\b(?!\\s*(?:false|off|0)\\b))[^)]*\\)"; + private static final Pattern CREATED_TABLE = Pattern.compile( "\\bcreate\\s+(?:(?:temp|temporary|unlogged)\\s+)?(?:table|materialized\\s+view)\\s+" @@ -77,7 +98,9 @@ public final class LockWaitMigrationInspector { + NAME), Pattern.compile("\\bdrop\\s+index\\s+(?!concurrently\\b)"), Pattern.compile( - "\\breindex\\s+(?:\\([^)]*\\)\\s+)?(?:(?:index|table)\\s+(?!concurrently\\b)" + "\\breindex\\s+(?:" + + NOT_CONCURRENT_OPTIONS + + "\\s+)?(?:(?:index|table)\\s+(?!concurrently\\b)" + NAME + "|(?:schema|database|system)\\b(?!\\s+concurrently\\b))"), Pattern.compile("\\brefresh\\s+materialized\\s+view\\s+(?!concurrently\\b)" + NAME), @@ -90,7 +113,8 @@ public final class LockWaitMigrationInspector { + "(?:\\s+(?:freeze|verbose|analyze)\\b)*(?:\\s+" + NAME + ")?"), - Pattern.compile("\\breferences\\s+" + NAME), + // Not GRANT/REVOKE ... REFERENCES ON t: a privilege, not a foreign key. + Pattern.compile("\\breferences\\s+(?!on\\b)" + NAME), Pattern.compile( "\\bcreate\\s+(?:or\\s+replace\\s+)?(?:constraint\\s+)?trigger\\s+" + OTHER_NAME @@ -118,9 +142,33 @@ public final class LockWaitMigrationInspector { private static final Pattern DURATION = Pattern.compile("[0-9.]+\\s*[a-z]*", Pattern.CASE_INSENSITIVE); - /** The line in a migration's {@code .sql.conf} that makes Flyway run it outside a transaction. */ - private static final Pattern OUTSIDE_A_TRANSACTION = - Pattern.compile("(?m)^\\s*executeInTransaction\\s*=\\s*false\\s*$"); + /** + * The statements Flyway runs outside a transaction on PostgreSQL 12 and later, as Flyway 11's + * {@code PostgreSQLParser#detectCanExecuteInTransaction} lists them: the statement's leading + * words outside parentheses, lower case and one space apart, match one of these in full. {@code + * ALTER TYPE ... ADD VALUE} is on Flyway's list only below PostgreSQL 12. {@code REINDEX TABLE + * CONCURRENTLY} is not on it: Flyway runs it in the transaction and PostgreSQL refuses it there. + * {@code postgresqlMigrationLockWaitIntegrationTest} holds this list to Flyway itself on real + * PostgreSQL, so a Flyway upgrade that changes it fails there rather than here. + */ + private static final List FLYWAY_OUTSIDE_A_TRANSACTION = + List.of( + Pattern.compile("(?:create|drop) (?:database|tablespace|subscription)"), + Pattern.compile("alter system"), + Pattern.compile("(?:create|drop)(?: unique)? index concurrently"), + Pattern.compile("reindex(?: verbose)? (?:schema|database|system)"), + Pattern.compile("vacuum"), + Pattern.compile("discard all")); + + /** Flyway stops looking after this many words of a statement and runs it in the transaction. */ + private static final int FLYWAY_DETECTION_WORDS = 10; + + /** A word Flyway reads as a keyword: unquoted, starting with a letter or underscore. */ + private static final Pattern WORD = Pattern.compile("[a-z_][a-z0-9_$]*"); + + /** A dollar-quoted body ({@code $$ ... $$}, {@code $fn$ ... $fn$}), which holds no statement. */ + private static final Pattern DOLLAR_QUOTED = + Pattern.compile("\\$([a-z_][a-z0-9_]*)?\\$.*?\\$\\1\\$", Pattern.DOTALL); /** * Fails when the migration locks a table it did not create before it bounds the wait. @@ -137,42 +185,66 @@ public final class LockWaitMigrationInspector { migration.name() + " waits without a bound for a lock on a table already in use (" + String.join(", ", unbounded) - + "). Put SET LOCAL lock_timeout = '5s'; before the first of them: while it waits, every" - + " read and write of the table queues behind it. LOCAL, so the setting ends with the" - + " migration's transaction (docs/jpa/migration-guide.md, \"Bound the lock wait\")"); + + "). In a migration Flyway runs in a transaction, put SET LOCAL lock_timeout = '5s';" + + " before the first of them: while it waits, every read and write of the table queues" + + " behind it. LOCAL, so the setting ends with the migration's transaction" + + " (docs/jpa/migration-guide.md, \"Bound the lock wait\")"); } /** * The statements that wait for a lock on an existing table without a {@code SET LOCAL * lock_timeout} in force, each as its leading words and table; empty when there are none. The - * script is taken to run in a transaction, as Flyway runs a migration by default. + * script is taken to have no {@code .sql.conf}: Flyway runs it in a transaction unless it holds a + * statement Flyway runs outside one ({@link #statementsFlywayRunsOutsideATransaction(String)}). */ public List unboundedLockingStatements(String sql) { - return unboundedLockingStatements(sql, true); + return unboundedLockingStatements(sql, Configured.NOTHING); } /** - * The same, for a migration file on disk: a {@code