fix!: 재리뷰가 찾은 결함과 색인에서 빠졌던 발견 5건을 고친다

문제: 성숙도 리뷰 수정을 다시 읽은 재리뷰(Critical 0, 회귀 0)가
Important 27건·Minor 41건을 찾았다. 수정이 새로 만든 결함이 많았다.
@SystemInvoked가 권한 규칙의 탈출구가 됐고, 데모 계정 거부가 정확히
prod일 때만 걸렸고, outbox 순서 번호가 업그레이드 때 뒤로 갔고,
GraphQL union 아래 복잡도가 0이었고, 서블릿 오류 응답이 success:true로
나갔고, 호출자 deadline이 breaker를 열었다. 원래 리뷰 색인이 06 영역
발견 5건(06-10~14)을 빠뜨려 그동안 처리되지 않았다.

방법: 영역별 7묶음이 test-driven-development 절차로 실패 테스트를
먼저 쓰고 고쳤다. 결정은 재리뷰 README 4절에 적었다: 순서 번호는 V21이
기록한 전역 floor부터, 데모 계정은 local·test·demo 허용 목록,
@SystemInvoked bean은 기동 실패, complete 단계 실패는
IDEMPOTENCY_COMPLETION_UNKNOWN(결과 불명), 캐시 조건부 쓰기는 별도
SCRIPT lane, strict lane 정책은 build cache 입력, 끝나지 않는 재시도
행은 원인 코드·지표·PARKED 절차. 레지스트리가 선언만 하고 아무 코드도
내보내지 않던 redis.capability.* 지표 6개는 빼고 runbook을 실제
지표(redis.lane.*) 기준으로 고쳤다.

위험·한계: V21을 V22 없이 직접 고쳤다(어느 배포에도 아직 없음). 이
브랜치의 V21을 적용한 개발 DB는 다시 만들어야 한다. 서블릿 오류
디스패치를 보안 체인이 허용한다. 발행·저장소 포트 시그니처가 바뀌어
파생 서비스 코드가 깨진다. prod에서 fixture-로 시작하는 설정값은
기동을 막는다. httpclient failure-injection lane의 main 단계 중복
실행은 이전 결정대로 남겼다. 실제 CI runner 실행은 확인하지 못했다.

검증: ci, mainCheck, :integrationCheck, -p optional-platforms ci
integrationCheck performanceCheck, :performanceCheck, :releaseCheck
(-PgitRevision), verify-compose-profile-contracts(18 lane) 통과.
첫 실행의 실패 2건(FindSecBugs 1, sample 동시 생성 테스트의 v4 event
id)은 고친 뒤 ci·integrationCheck·mainCheck를 다시 돌려 통과했다.

Refs: docs/reviews/2026-11-03-rereview/README.md 3·4·6·7절
BREAKING CHANGE: outbox 발행 포트와 저장소 포트 시그니처, 순서 번호
첫 값(V21 floor 다음), 샘플 데모 계정 프로필 허용 목록, 새 오류 코드
IDEMPOTENCY_COMPLETION_UNKNOWN, prod의 fixture- 값 거부, 기동 rule은
static·의존 없음이어야 한다.
This commit is contained in:
DongHyeonka
2026-11-03 23:29:35 +09:00
parent 7ef8d0b4cb
commit bbe944bd3c
406 changed files with 13493 additions and 1656 deletions
+9 -1
View File
@@ -65,9 +65,12 @@ jobs:
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # actions/checkout@v4.2.2
- uses: ./.github/actions/setup-gradle-java
# `qualityCheck` too: the root one covers the production leaves only, so the sample — the code
# adopters copy first — was analysed by no stage, and its SpotBugs exclusions matched nothing
# (review 2026-11-03 RR-06-05).
- name: Verify the standalone sample application
working-directory: src
run: ./gradlew :sample-portfolio:check --stacktrace
run: ./gradlew :sample-portfolio:check :sample-portfolio:qualityCheck --stacktrace
optional-platforms:
runs-on: ubuntu-latest
@@ -77,6 +80,11 @@ jobs:
- name: Verify the optional gRPC platform build
working-directory: src
run: ./gradlew -p optional-platforms ci --stacktrace
# The optional build's own lane-reachability check: the main build's cannot see these lanes
# (review 2026-11-03 RR-06-03).
- name: Require every optional-platform strict lane to be in an executed stage
working-directory: src
run: ./gradlew -p optional-platforms :verifyStrictLaneStages --dry-run --stacktrace
# gRPC schema policy (owner decision 18). protoc compiles src/grpc/grpc-contract inside the
# optional-platforms build above; lint and breaking-change policy belong to Buf, configured by
+6
View File
@@ -203,6 +203,12 @@ jobs:
- name: Run every performance-category strict lane
working-directory: src
run: ./gradlew :performanceCheck --continue --stacktrace
# The optional gRPC build's performance lanes (grpcPerformanceTest). A separate build, so a
# separate command; its own verifyStrictLaneStages counts this line (review 2026-11-03 RR-06-03).
- name: Run every optional-platform performance-category strict lane
if: always()
working-directory: src
run: ./gradlew -p optional-platforms performanceCheck --continue --stacktrace
- name: Publish the performance lane reports
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # actions/upload-artifact@v4.6.2
+5 -9
View File
@@ -120,9 +120,10 @@ jobs:
# container and no fixed cadence — grpcNettyContractTest opens an ephemeral socket, not a broker —
# so the guard is worth asserting once per tag and is not worth a runner every night.
#
# grpcPerformanceTest is deliberately absent. The leaf excludes it from `test` and says why: "a
# measurement in the release gate is a flaky test on a shared CI runner; it runs when somebody asks
# for it, by name." It is recorded as a manual entrypoint in the repository README instead.
# The optional build's `releaseCheck` is those hermetic lanes, so a lane added there is here
# without a workflow edit. grpcPerformanceTest is deliberately absent: a measurement is not release
# evidence on a shared runner. It runs in integration-main.yml's performance-check job (the
# optional build's `performanceCheck`).
grpc-stable-release-gate:
runs-on: ubuntu-latest
timeout-minutes: 45
@@ -131,12 +132,7 @@ jobs:
- uses: ./.github/actions/setup-gradle-java
- name: Run the gRPC Stable certification lanes
working-directory: src
run: >-
./gradlew -p optional-platforms
:grpc:grpc-testkit:grpcInProcessContractTest
:grpc:grpc-testkit:grpcNettyContractTest
:grpc:grpc-testkit:grpcFaultTest
--stacktrace
run: ./gradlew -p optional-platforms releaseCheck --stacktrace
# The operation ledger's real-PostgreSQL lane ran only on main and nightly
# (integration-main.yml), so a tag could be cut from a main whose ledger lane was red. It needs
# Docker, which a release runner has (review 2026-11-01 03-12).
+9 -6
View File
@@ -77,10 +77,13 @@ root `CLAUDE.md`는 이 목록의 동기화된 요약이다. 두 문서가 어
`ca.config-contract`다. `src/build-tools/` included build는 비워서 삭제했다.
- `src/build.gradle`: 루트 라이프사이클(`ci`, `mainCheck`, `integrationCheck`, `performanceCheck`,
`releaseCheck`, `qualityCheck`, `configContractCheck`)과 버전/리비전. `verifyStrictLaneStages
--dry-run`(PR 게이트)은 `explicitOnly`가 아닌 모든 strict lane이 workflow가 실제로 실행하는 단계
(`ci`·`sample-portfolio:check`·`mainCheck`·`integrationCheck`·`performanceCheck`·`releaseCheck`)
중 하나에 들어 있는지, 그리고 workflow의 접힌(`>`) run 블록이 빈 줄 때문에 옵션을 별개 명령으로
떼어 내지 않는지 검사한다.
--dry-run`(PR 게이트)은 그 빌드에서 `explicitOnly`가 아닌 strict lane이 workflow가 실제로 실행하는
단계(main 빌드: `ci`·`sample-portfolio:check`·`mainCheck`·`integrationCheck`·`performanceCheck`·
`releaseCheck`, optional-platforms 빌드: `-p optional-platforms`로 부르는 `ci`·`integrationCheck`·
`performanceCheck`·`releaseCheck`) 중 하나에 들어 있는지, 그리고 workflow의 접힌(`>`) run 블록이 빈
줄 때문에 옵션을 별개 명령으로 떼어 내지 않는지 검사한다. 두 빌드가 각자 검사를 돌린다. `-x`로 뺀
lane, `continue-on-error` 아래 명령, 상태 함수(`always()`·`success()`·`!cancelled()`)가 아닌 `if:`
아래 명령은 실행으로 세지 않는다.
작업 파일의 소유 leaf는 registry의 `source_path`로 판단하고 가장 가까운 `src/**/CLAUDE.md`를
함께 읽는다. focused test는 registry의 `gradle_path`에서
@@ -202,10 +205,10 @@ cd src
./gradlew architectureCheck # 의존 방향·domain-core/shared-contract/application-* 순수성·ArchUnit
./gradlew qualityCheck # SpotBugs + FindSecBugs (leaf check에는 없다)
./gradlew ci # PR 게이트 = leaf check + architecture + quality
./gradlew mainCheck # main 단계 = env 계약·전체 기동·public path·전송/메시징 레인·leaf 문서 대조 레인
./gradlew mainCheck # main 단계 = env 계약·전체 기동(샘플 포함)·public path·전송/메시징 레인·leaf 문서 대조 레인·기능 생성 왕복(verifyFeatureScaffold)
./gradlew integrationCheck # 실제 컨테이너 integration 레인 전체(main·nightly, Docker 필요)
./gradlew performanceCheck # 부하·종료·자원 한계 레인(main·nightly)
./gradlew verifyStrictLaneStages --dry-run # 모든 strict lane이 workflow가 실행하는 단계에 속하는지
./gradlew verifyStrictLaneStages --dry-run # 이 빌드의 strict lane이 workflow가 실행하는 단계에 속하는지(optional 빌드는 -p optional-platforms로 따로)
./gradlew verifyPublicPathSnapshot
./gradlew :app-bootstrap:verifyEnvKeys
```
+9 -5
View File
@@ -38,8 +38,10 @@ If this summary drifts from `AGENTS.md`, `AGENTS.md` wins and this summary must
`ca.config-contract`. The `src/build-tools/` included build was emptied and removed.
- `src/build.gradle`: the root lifecycle (`ci`, `mainCheck`, `integrationCheck`, `performanceCheck`,
`releaseCheck` and the named repository-wide gates). `verifyStrictLaneStages --dry-run` (pull
request) fails when a non-`explicitOnly` strict lane is in no stage a workflow executes, or a
folded workflow `run:` block splits an option off its command.
request; the main build and, as `-p optional-platforms`, the optional build each run their own)
fails when a non-`explicitOnly` strict lane is in no stage a workflow executes — a lane the
command removes with `-x` and a command under `continue-on-error` or a non-status `if:` do not
count — or a folded workflow `run:` block splits an option off its command.
Commit policy is `human-only`: agents do not stage, commit, amend, or push implementation changes.
@@ -132,9 +134,11 @@ declared strict integration lanes; a lane that needs an input the build was not
and named), `performanceCheck` (the declared performance lanes; main and nightly). `ci` (pull
request) is leaf checks (API surfaces included, owner decision 20) + architectureCheck +
qualityCheck, and the workflow also runs `verifyStrictLaneStages --dry-run`; `mainCheck` (main branch) is configContractCheck +
qualificationCheck + the sample web contract + the public-path snapshot + the transport and
messaging qualification lanes + every leaf's `mainCheck` (document-contract lanes, MinIO lanes);
`releaseCheck` is the tag-only checks: release provenance and the public-path snapshot.
qualificationCheck + the sample web contract + the sample full boot (`:sample-portfolio:systemTest`)
+ the public-path snapshot + the transport and messaging qualification lanes + the feature
generator round trip (`verifyFeatureScaffold`) + every leaf's `mainCheck` (document-contract lanes
and any lane the leaf adds). `releaseCheck` is the tag-only checks: release provenance and the
public-path snapshot.
Test-writing rules (behavior-sentence names that assert what they promise, `@DisplayName` in Korean
only when the name cannot say it (owner decision 5), given/when/then comments, one `// when`, one
+7 -5
View File
@@ -137,10 +137,12 @@ services:
start_period: 30s
# ---- Redis — local compose contract for the shipped Redis runtime ----------------
# Authenticated, with the accounts a deployment loads: infra/redis-sdk/acl/all-accounts.acl, the
# same file the Redis topology lanes use, mounted as it is rather than copied. Its first line turns
# the passwordless `default` user off, so the application connects as the named accounts the lane
# contract gives it (compose-profile-contracts.json, $redisComment).
# Authenticated the way a deployment authenticates, with the lanes' test-fixture accounts:
# infra/redis-sdk/acl/lane-fixture-accounts.acl, the same file the Redis topology lanes use, mounted
# as it is rather than copied. It is not a deployment file — its passwords are public and it carries
# a lane-only `+@all` account (infra/redis-sdk/acl/README.md). Its first line turns the passwordless
# `default` user off, so the application connects as the named accounts the lane contract gives it
# (compose-profile-contracts.json, $redisComment).
#
# This Redis was open and the lanes declared app.redis.authentication.anonymous-access-accepted.
# The application then logged that trade, and the missing advanced account, as two startup
@@ -162,7 +164,7 @@ services:
- --save
- ""
- --aclfile
- /etc/redis/acl/all-accounts.acl
- /etc/redis/acl/lane-fixture-accounts.acl
volumes:
- type: bind
source: ./infra/redis-sdk/acl
@@ -69,6 +69,36 @@ method can run longer than any fixed TTL; revisit when one exists. A sweeper job
rows — it needs a scheduler this family does not have, and deleting a claim makes the next retry a
fresh claim with no record that an attempt was abandoned.
**A taken-over owner is fenced by a claim token** (re-review 2026-11-03, RR-02-04). The owner a
takeover presumes dead may only be slow. Without knowing which attempt held the claim, its late
`markFailed` turned the new attempt's claim `FAILED_TERMINAL`, the next retry re-claimed it, and two
attempts ran the mutation at once; its late `markCommitted` committed beside the new attempt. Every
claim now carries a `GrpcClaimToken` (`claim_attempt`, migration V003): 1 for the first claim, one more
for each `reclaimFailed` or `reclaimStale`. `decide` returns it on `PROCEED`, and `recordCommit` /
`recordFailure` hand it back. A commit or a failure is one conditional
`UPDATE … WHERE state = 'IN_PROGRESS' AND claim_attempt = :attempt`: a commit by an attempt that lost
the claim is refused with an exception, so its business transaction rolls back; a failure by one is
ignored (`markFailed` returns false). The same conditional update closes the JPA commit race —
load-check-save let two commits that both read `IN_PROGRESS` both write; now exactly one matches.
A claim whose conflicting row is purged between insert and read retries the insert once instead of
failing the call.
Chosen over the claim moment as the token (`WHERE claimed_at = :claimedAt`), which needs no column:
PostgreSQL stores microseconds while `Instant` carries nanoseconds, so the token a caller holds and
the one stored can differ by rounding, and two processes' clocks can give two attempts the same
moment. The counter has neither problem. The heartbeat lease above stays rejected for the same reason
as before; the token fences without one.
**What `operation-ledger-enabled=true` promises (03-11 (b), recorded 2026-11-03, RR-02-07).** The
2026-11-01 review offered (a) delete the flag and judge by whether a ledger bean exists, or (b) keep the
flag and have the starter assemble the decision function from it. (b) is kept: the assembly and the
TTL validation live in one place instead of in every adopter. (a) was rejected because it removes the
false promise only by removing the assembly with it. The limit stays and is stated where an operator
meets it: the starter builds no server, so the flag assembles the decision function only, and keys are
honoured once the adopter's `IDEMPOTENCY` stage interceptor or service adapter calls `decide` and
`recordCommit`. `GrpcPlatformStartupValidator`'s refusal for a keyed method says exactly that; a running
process cannot see whether the stage exists, so the validator does not claim to check it.
**Finished rows are purged after a retention; in-progress rows never are** (03-10 follow-up).
`GrpcOperationLedger.purgeCompletedBefore(cutoff, limit)` deletes at most `limit` `COMMITTED` or
`FAILED_TERMINAL` rows that completed before the cutoff, oldest first — on PostgreSQL one bounded
+3 -3
View File
@@ -30,14 +30,14 @@ lanes and CI; it is not a deployment artifact.
| Relational persistence (JPA/PostgreSQL) | `ca-skeleton.persistence-jpa.enabled` / `APP_PERSISTENCE_JPA_ENABLED`, `true` | yes | yes (one line: `runtimeOnly project(':adapter:outbound:persistence-jpa')`) | yes | Not in the default composition. The one line brings everything — its own `AutoConfiguration.imports` (`PersistenceJpaAutoConfiguration`, `PersistenceJpaRuntimeConfig`, `JpaStartupAutoConfiguration`), its environment handling (`JpaActivationEnvironmentPostProcessor`, `spring.factories`: readiness `db` while on; with `false` it also excludes Boot's JDBC, Hibernate, Spring Data JPA and Flyway auto-configuration, so no DataSource is bound), the startup guards (Flyway strategy, datasource/OSIV/DDL/pool/TLS/vendor checks, exit codes unchanged) and its settings (`config/module-defaults/persistence-jpa.yml`). `persistence-jpa:PersistenceJpaCapabilityAssemblyTest` proves the module assembles on its own; the `postgresql*IntegrationTest` readiness lanes, run by `integration-main.yml` → `integration-check` (`:integrationCheck`), qualify it against PostgreSQL. `sample-portfolio` selects it by its build dependency and the module's auto-configuration (no broad scan of the persistence package). `bootRunH2` adds the module to its own classpath only. |
| Document persistence (MongoDB) | `ca-skeleton.persistence-mongo.enabled` / `APP_PERSISTENCE_MONGO_ENABLED`, `true` | yes | yes (one line: `runtimeOnly project(':adapter:outbound:persistence-mongo')`) | yes | On when present; `false` also excludes Boot's Mongo auto-configuration (`MongoCapabilityActivationTest`). The connection-source rule (`SPRING_MONGODB_URI` or the selected profile's `uri-secret`) is checked by the module's own `MongoDefaultsEnvironmentPostProcessor`. Six container lanes (`mongoReplicaSetTest`, `mongoFailoverTest`, …), run by `integration-main.yml` → `integration-check` (`:integrationCheck`) |
| Messaging (Kafka) | `app.messaging.enabled` / `APP_MESSAGING_ENABLED`, `true` | yes | yes (one line: `runtimeOnly project(':adapter:outbound:messaging-kafka')`, which brings the application bridge, the legacy `KafkaSender` seam and the vendored platform with its Kafka transport) | Kafka only | On when present (`KafkaCapabilityActivationTest`). `messaging-kafka:messagingCertificationTest` behind `verifyMessagingCertificationEvidence`, run by `messaging-certification.yml`; the starter's `integrationTest` is the live Kafka round trip. Kafka is the only registered broker; the RabbitMQ broker keys were removed (no transport). |
| Redis capabilities (cache / rate-limit / lease / idempotency) | `app.redis.enabled` / `APP_REDIS_ENABLED`, `true` | yes | yes (one line: `runtimeOnly project(':adapter:outbound:cache-redis')`, which brings `redis-runtime`) | yes | `cache-redis:redisContainerContractTest` (Testcontainers Redis, `integrationCheck` and `mainCheck`); `cache-redis:redisTopologyTest`, run by `redis-sdk-topology.yml`. Not in the default composition. On when present; `false` contributes no Redis bean — no settings, client, connection, thread or health contributor. The module owns its readiness contribution (`RedisReadinessGroupPostProcessor`, `spring.factories`) and its settings (`config/module-defaults/cache-redis.yml`: `app.redis.enabled`, `app.rate-limit.*`, the Redis `ca-skeleton.capabilities.*`). No `app-bootstrap` main source names a Redis type. |
| Request idempotency | `APP_IDEMPOTENCY_PROVIDER` (`disabled` \| `redis` \| `postgresql`; a selector, not an on/off key) | yes | yes | Redis only | One generation: the application `IdempotencyExecutor`, built once by `app-bootstrap` `IdempotencyProviderSelectionConfig` over the one store the selected provider contributes (`cache-redis` or `persistence-jpa`); startup is refused unless exactly one store exists. `sample-portfolio` cannot depend on `app-bootstrap`, so its own `SampleIdempotencyConfig` builds its one executor from the same keys. Redis is covered by the row above. PostgreSQL has `persistence-jpa:postgresqlIdempotencyIntegrationTest`, but no workflow runs it. The `postgresql` provider refuses a non-PostgreSQL data source, so `bootRunH2` uses `disabled` or `redis`.. The executor stays in the composition root on purpose — it needs only application-core and its settings — while each store comes with its module: `postgresql` needs the persistence-jpa line, `redis` the cache-redis one. |
| Redis capabilities (cache / rate-limit / lease / idempotency) | `app.redis.enabled` / `APP_REDIS_ENABLED`, `true` | yes | yes (one line: `runtimeOnly project(':adapter:outbound:cache-redis')`, which brings `redis-runtime`) | yes | `cache-redis:redisContainerContractTest` (Testcontainers Redis, `integrationCheck`); `cache-redis:redisTopologyTest`, run by `redis-sdk-topology.yml`. Not in the default composition. On when present; `false` contributes no Redis bean — no settings, client, connection, thread or health contributor. The module owns its readiness contribution (`RedisReadinessGroupPostProcessor`, `spring.factories`) and its settings (`config/module-defaults/cache-redis.yml`: `app.redis.enabled`, `app.rate-limit.*`, the Redis `ca-skeleton.capabilities.*`). No `app-bootstrap` main source names a Redis type. |
| Request idempotency | `APP_IDEMPOTENCY_PROVIDER` (`disabled` \| `redis` \| `postgresql`; a selector, not an on/off key) | yes | yes | Redis and PostgreSQL | One generation: the application `IdempotencyExecutor`, built once by `composition:idempotency` (`IdempotencyAutoConfiguration`) over the one store the selected provider contributes (`cache-redis` or `persistence-jpa`); startup is refused unless exactly one store exists. `app-bootstrap` and `sample-portfolio` both take that module with one `runtimeOnly` line, so there is no second assembly. Redis is covered by the row above. PostgreSQL has `persistence-jpa:postgresqlIdempotencyIntegrationTest`, run by `integration-main.yml` → `integration-check` (`:integrationCheck`). The `postgresql` provider refuses a non-PostgreSQL data source, so `bootRunH2` uses `disabled` or `redis`. Each store comes with its module: `postgresql` needs the persistence-jpa line, `redis` the cache-redis one. |
| Transactional outbox relay | `ca-skeleton.outbox.enabled` / `APP_OUTBOX_ENABLED`, `false` (+ `APP_OUTBOX_RELAY_ENABLED`, `false`; persistence-jpa settings) | yes | yes (with the persistence-jpa line; the scheduler also needs the messaging-kafka line) | yes | The store and its settings are persistence-jpa's; the relay (`OutboxConfig`) is application composition in `app-bootstrap` because a driven adapter must not build or call a use case. It turns on only when an `OutboxStorePort` bean exists, and the scheduler only when an `OutboxMessagePublishPort` does (the messaging bridge), each ordered after its contributor by `afterName` — so an environment with no publisher keeps its PENDING rows. The bridge registers no stand-in publisher: a present-and-enabled bridge with a blank broker and no canonical transport is refused at startup. `postgresqlLegacyOutboxConcurrentRelayIntegrationTest` qualifies claim/fencing against PostgreSQL. |
| Notification platform | `ca-skeleton.notification.platform.enabled` / `APP_NOTIFICATION_PLATFORM_ENABLED`, `true` | yes | yes (one line: `runtimeOnly project(':composition:notification-platform')`, plus the persistence-jpa line) | claim-scoped | Not in the default composition. The one line brings the use-case assembly, `adapter:outbound:notification`, the HTTP surface `adapter:inbound:web-notification` and its own settings (`config/module-defaults/notification.yml`), each registered through the module's own `AutoConfiguration.imports`; `adapter:inbound:web` no longer depends on `application-notification`. `verifyNotificationEvidence` refuses a grade in `docs/notification/support-matrix.md` that no proven claim backs; run by `notification-platform.yml`. There is no real-SMTP lane, and the matrix says so rather than implying one. |
| File server platform | `app.fileserver-platform.enabled` / `APP_FILESERVER_PLATFORM_ENABLED`, `true` | yes | yes (one line: `runtimeOnly project(':composition:fileserver-platform')`, plus the persistence-jpa line) | yes | On when present; module-owned `config/module-defaults/fileserver.yml` also carries the servlet multipart ceiling. `app-bootstrap:integrationTest` → `FileserverCompositionIntegrationTest` (required by the strict lane), run by `integration-main.yml` → `integration-check` (`:integrationCheck`) |
| Outbound HTTP client | `app.httpclient.enabled` / `APP_HTTPCLIENT_ENABLED`, `true` | yes | yes (one line: `runtimeOnly project(':adapter:outbound:httpclient')`) | yes | On when present (`HttpClientPlatformActivationTest`). `httpclient:httpClientFailureInjectionTest` against a faulted upstream, run by `integration-main.yml` → `integration-check` (`:integrationCheck`) |
| GraphQL transport | `backend.graphql.enabled` / `APP_GRAPHQL_ENABLED`, `true` | yes | yes (one line: `runtimeOnly project(':adapter:inbound:graphql')`) | leaf-level | On when present; `false` also excludes Spring GraphQL's own auto-configuration, so no `/graphql` (`GraphQlCapabilityActivationTest`). `graphqlTransportQualificationTest` runs the HTTP boundary. The deployment-mode/runtime-profile rule is the module's own `GraphQlPlatformStartupValidator`. The default jar does not ship GraphQL, so a compose lane that exercises it runs the qualification jar. |
| Object storage | `app.object-storage.enabled` (no `APP_*` row), `true` | yes | yes (one line: `runtimeOnly project(':adapter:outbound:objectstorage')`) | MinIO only | Own `ObjectStorageAutoConfiguration` (was bound through bootstrap's properties scan); on when present, `false` leaves only the explicit legacy seam (`ObjectStorageAutoConfigurationTest`). `app-bootstrap:integrationTest` assembles it against digest-pinned MinIO. |
| Object storage | `app.object-storage.enabled` / `APP_OBJECTSTORAGE_ENABLED`, `true` | yes | yes (one line: `runtimeOnly project(':adapter:outbound:objectstorage')`) | MinIO only | Own `ObjectStorageAutoConfiguration` (was bound through bootstrap's properties scan); on when present, `false` leaves only the explicit legacy seam (`ObjectStorageAutoConfigurationTest`). `app-bootstrap:integrationTest` assembles it against digest-pinned MinIO. |
| gRPC platform | — (build-level) | yes | **no** | yes | Absent from the main build entirely: `optional-platforms` is a separate Gradle build, so the switch is the `include`, not an environment variable. `-p optional-platforms ci` runs on every PR and `integrationCheck` certifies the operation ledger against real PostgreSQL. |
| gRPC transport (`adapter:inbound:grpc`) | `ca-skeleton.grpc.server.enabled` (no `APP_*` row), `true` | yes | yes (one line: `runtimeOnly project(':adapter:inbound:grpc')`) | leaf-level | Own `AutoConfiguration.imports` (`GrpcServerConfig`); on when present, and the loopback plaintext listener still needs `ca-skeleton.grpc.server.allow-insecure-local=true` (`GrpcSafeActivationTest`, `grpcTransportQualificationTest`). |
| WebSocket transport | `app.websocket.enabled` / `APP_WEBSOCKET_ENABLED`, `true` | yes | yes (one line: `runtimeOnly project(':adapter:inbound:websocket')`) | — | Own `AutoConfiguration.imports` (`WebSocketConfig`), on in a servlet app when present (`WebSocketAutoConfigurationTest`); `ConditionalTransportBuildContractTest` asserts it has no default production edge. |
+5 -1
View File
@@ -128,13 +128,17 @@ Set `inline-safe-profile: true` only when downloads are never served inline from
| `direct-download-permits` | `64` | concurrent non-delegated downloads |
| `soft-high-water` | `0.70` | storage fraction at which pressure is reported |
| `hard-high-water` | `0.85` | storage fraction at which uploads and copies are refused |
| `namespace-limit` | unset (unlimited) | most bytes one namespace may hold, live reservations plus committed usage; a create or copy that would pass it is refused with `QUOTA_EXCEEDED` (413) before any record or byte exists |
| `namespace-limit` | unset (unlimited) | most bytes one namespace may hold, live reservations plus committed usage; a create, copy or move that would pass it is refused with `QUOTA_EXCEEDED` (413) before any record, byte or namespace changes |
`namespace-limit` is checked against the usage the quota ledger reports and then reserved, in one
transaction but without a per-namespace lock: reservations admitted at the same moment can overshoot
the limit by at most their own size, and the next one is refused. A service that needs an exact cap
adds that lock in its persistence adapter.
A move to another namespace carries the file's bytes in the ledger: in the transaction that relocates
the record it checks the target's limit, books the bytes as committed usage there and reclaims them
from the source. A refused move leaves the file, and both namespaces' usage, as they were.
When the storage fraction cannot be read, admission treats it as unknown and does not apply the
high-water rule — a synthetic `0` would silently disable the guard, and a synthetic `1` would take
the capability down over a failed syscall.
+1 -1
View File
@@ -15,7 +15,7 @@ and fails if one is missing, so this file cannot silently fall behind the code.
| HEAD | `/v1/files/{fileId}/content` | `200`, `304` | identical headers, no body |
| DELETE | `/v1/files/{fileId}` | `202`, `204` | logical delete first |
| POST | `/v1/files/{fileId}:copy` | `202` | create-only target |
| POST | `/v1/files/{fileId}:move` | `200` | logical namespace change only |
| POST | `/v1/files/{fileId}:move` | `200` | namespace change only, bytes stay; the file's quota moves with it, and a target namespace past its `namespace-limit` is `413` |
| OPTIONS | `/v1/uploads` | `204` | tus capability discovery |
| POST | `/v1/uploads` | `201` | tus creation |
| HEAD | `/v1/uploads/{uploadId}` | `204` | tus offset |
+15 -1
View File
@@ -102,7 +102,13 @@ greater than zero (`TIMEOUT_NOT_POSITIVE`): both engines read zero as "wait fore
`total-call` also bounds the setup of a streaming download — connect, send, and the wait for
response headers. Once the stream is handed over, each gap of the body is bounded by `read-idle`
instead, so a long download is not cut at an arbitrary total (design §15.3).
instead, so a long download is not cut at an arbitrary total (design §15.3). This holds on both
transports. The JDK transport keeps its exchange-wide timeout off a streamed body and bounds each
read with its own `read-idle` timer, because the JDK client has no socket read timeout; before
review 2026-11-03 RR-04-03 every JDK download longer than `response-header` was cut mid-body. A body
failure after hand-over is thrown as `HttpResponseTimeoutException` (a gap longer than `read-idle`)
or `HttpResponseTruncatedException` (the connection ended early), stage `RESPONSE_BODY`, evidence
`PARTIAL_RESPONSE`.
### Removed settings
@@ -126,6 +132,14 @@ it as a result — and every transport failure. Every other 4xx is a healthy ans
only say how many failures, out of how many calls, open the circuit. Defaults are Resilience4j's own,
which every profile ran with before the keys existed.
Two deadlines can end an attempt in flight, and they are counted differently (review 2026-11-03
RR-04-01):
| Ended by | Breaker | Why |
|---|---|---|
| The caller's own deadline (`HttpOperation.deadline()`, when earlier than `total-call`) | Not recorded; the permission is given back | The caller ran out of time. Counting it opened the circuit on a healthy upstream whenever callers passed short budgets |
| The profile's `timeout.total-call` | Recorded as a failure | `total-call` is the operator's statement of how long a healthy upstream may take; a slow-drip upstream that never trips a per-gap timeout is seen only through it |
| Property | Default | Meaning |
|---|---|---|
| `failure-rate-threshold` | `50` | Percent of failed calls in the window that opens the circuit, 1–100 |
+11 -5
View File
@@ -72,17 +72,23 @@ What to use, in order of preference:
A unidirectional `@OneToMany` with its own `@JoinColumn` is the exception, not the default. Use it
only for child entities the root genuinely owns and changes only through itself (the testkit's
`LifecycleParent` and `PagedParent` are this shape). It still costs an `UPDATE` per child when the
join column is nullable, and the collection still loads as a unit.
join column is nullable, and the collection still loads as a unit. The child then has no
association back to the root — not even a read-only `@ManyToOne` on the same join column
(`insertable = false, updatable = false`): that is the bidirectional pair again, without the
`mappedBy` that names it. If the child needs the root, it holds the root's ID.
`CascadeType.ALL` with `orphanRemoval` is correct only for a child the aggregate genuinely owns.
Between independent aggregates it deletes rows another part of the system still owns — another
reason those are ID references.
`JpaArchitectureRules.associationsAreMappedFromOneSideOnly()` (testkit) fails the build on any
`@OneToMany`, `@OneToOne` or `@ManyToMany` that declares `mappedBy`, on `@Entity`,
`@MappedSuperclass` and `@Embeddable` types. `JpaProductionArchitectureTest` (app-bootstrap
`JpaArchitectureRules.associationsAreMappedFromOneSideOnly()` (testkit) fails the build on every
bidirectional pair, on `@Entity`, `@MappedSuperclass` and `@Embeddable` types: an association that
declares `mappedBy` (only that inverse half is reported), and two entities whose associations point
at each other without `mappedBy` (both halves are reported — the `@JoinColumn` form above, review
RR-03-04). A class that links to itself is a pair only when a second member points back the other
way (`@ManyToOne parent` with `@OneToMany children`). `JpaProductionArchitectureTest` (app-bootstrap
`architectureTest`) runs it over the production graph and `SampleArchitectureTest` over the sample.
The failure message names the field and these alternatives.
The failure message names the fields and these alternatives.
Chosen and rejected:
+10
View File
@@ -28,6 +28,9 @@ use case ── tx.inWrite ──▶ XRepository (domain 포트)
3. 엔티티·Spring Data repository·mapper·adapter를 쓴다. adapter는 aggregate만 주고받는다
(`OUTBOUND_ADAPTER_METHOD_RETURNS_ONLY_DOMAIN_OR_PRIMITIVES`).
4. `JpaXRepositoryContractTest extends XRepositoryContract`를 실 PostgreSQL lane에 둔다(08장 lane).
adapter는 호출자 트랜잭션 밖의 호출을 거부하므로(아래 「트랜잭션 안에서만 쓴다」), 계약의
`inTransaction(Supplier)`을 재정의해 거기서 트랜잭션을 열고 커밋한다. 계약은 모든 저장소 호출을 그
hook으로 부른다. fake는 기본값(그냥 실행)을 쓴다.
예: `src/sample-portfolio/src/integrationTest/java/dev/caskeleton/sample/portfolio/adapter/outbound/persistence/repository/WorkLogRepositoryAdapterContractTest.java`.
5. adapter를 bean으로 등록하는 자동 설정을 쓰고, 기능의 `XConfig`에 `afterName`으로 그 순서를 적는다.
@@ -80,6 +83,13 @@ store는 자기 트랜잭션을 열지 않는다. 유스케이스가 연 트랜
예: `src/sample-portfolio/src/main/java/dev/caskeleton/sample/portfolio/adapter/outbound/persistence/repository/PosterRepositoryAdapter.java`
(`DataIntegrityViolationException` → 제약 이름 확인 → `PosterTitleAlreadyExistsException` 409),
제약은 `src/sample-portfolio/src/main/resources/db/sample-migration/V8_1__poster_title_unique.sql`.
- **충돌을 바꾸는 곳은 하나다.** 업무 규칙 충돌(같은 제목)은 포트 옆 domain 패키지의 예외 타입이고,
포트가 `@throws`로 선언하고 adapter가 던진다. 유스케이스의 사전 조회도 같은 타입을 던져 두 경로가 같은
409가 된다. application 예외를 domain 포트가 Javadoc으로만 가리키면 fake·mock은 그 약속을 지킬 수 없다.
기술 충돌(버전 불일치·교착)은 adapter가 프레임워크 예외 그대로 두고 아래 `PersistenceExceptionTranslator`가
`DB_*` 코드로 바꾼다(WorkLog의 `ObjectOptimisticLockingFailureException` → `DB_OPTIMISTIC_CONFLICT`).
- 그 약속은 포트 소유자 testFixtures의 공유 계약 테스트로 고정한다. fake와 adapter가 같은 계약을 상속한다.
예: `PosterRepositoryContract`(`InMemoryPosterRepository`, `PosterRepositoryAdapterContractTest`).
- 여러 인스턴스가 집는 작업 → `FOR UPDATE SKIP LOCKED`는 **트랜잭션 안에서**, 그리고 lease(만료 시각)와
fence 값을 함께 저장한다. 예: `ReconciliationJobStorePort.claimDue`(application-notification),
gRPC operation ledger의 claim.
+1
View File
@@ -50,6 +50,7 @@ DB 커밋 중 연결 끊김 ─────────────────
| 409 `IDEMPOTENT_IN_FLIGHT`, `retryable=true`, `Retry-After: 2` | 같은 키가 아직 실행 중. 결과 불명 | **같은 키로** 기다렸다 재전송. 새 키 금지 |
| 422 `IDEMPOTENT_REQUEST_MISMATCH` | 같은 키에 다른 본문 | 클라이언트 버그. 재전송 안 함 |
| 503 `IDEMPOTENCY_UNAVAILABLE` | 실행 전 거부 | 같은 키로 나중에 |
| 503 `IDEMPOTENCY_COMPLETION_UNKNOWN`, `retryable=true` | 실행 뒤 결과 기록 실패. 적용됐을 수 있음 | "적용 안 됨"이라 하지 않는다. **같은 키로** 나중에 재전송, 새 키 금지 |
| 500 `IDEMPOTENCY_RECOVERY_REQUIRED` | 이전 시도 효과를 증명 못함 | 재전송·재실행 모두 위험. 업무 데이터로 확인 |
| 500 `DB_COMPLETION_UNKNOWN`, `retryable=false` | DB 커밋 중 연결 끊김. 커밋됐을 수도 있음 | 키 없이 재시도 금지. KEYED면 같은 키로 재전송하면 서버가 판정 |
+3 -2
View File
@@ -41,8 +41,9 @@ gRPC/GraphQL/WebSocket: 같은 ApiErrorCarrier를 자기 status 모양으로
| 서버 조립·설정 오류, 있을 수 없는 상태 | `IllegalStateException` | 500, 로그 남음 |
| DB·외부 의존 실패 | adapter가 shared-contract carrier(`PersistenceFailureException`, `DependencyFailureException`)로 | 분류표대로(04·05장) |
- `IllegalArgumentException`을 "서버 조립이 틀렸다"에 쓰지 않는다. web은 그것을 로그 없이 400으로
바꿔서 서버 결함이 클라이언트 오류로 숨는다(리뷰 01-04).
- `IllegalArgumentException`을 "서버 조립이 틀렸다"에 쓰지 않는다. web은 그것을 400으로 바꾸고 WARN
한 줄만 남긴다(`GlobalExceptionHandler`). 서버 결함이 클라이언트 오류 통계에 섞여 경보가 울리지 않는다
(리뷰 01-04). 서버 쪽 잘못은 `IllegalStateException`(500, 로그)이다.
- 메시지는 로그용이다. 클라이언트에는 코드와 category별 고정 문구(`ClientErrorMessages`)만 간다.
메시지에 SQLState·내부 경로·토큰을 넣어도 클라이언트에 새지 않지만, 로그 정책은 지킨다.
+20 -3
View File
@@ -62,13 +62,30 @@ JWT의 `realm_access.roles`와 **자기 client**(`app.security.audience`)의 `re
batch 메서드로 N+1 회피)를 구현한다. 존재를 숨길 필요가 없는 관리 화면은 403도 된다.
- tenant: 인증 정보에서만 만든 `TenantId`를 command에 싣는다. 요청 본문의 tenant 값을 믿지 않는다.
## 지금 확인되지 않는 것
## 시스템이 부르는 쓰기: `@SystemInvoked`
- web이 빠진 조립(gRPC 전용·배치 전용)에서는 advisor가 없어 선언이 검사되지 않는다. 이때 기동을 막는
장치는 아직 없다(리뷰 01-06). web 없이 조립하는 서비스는 그 전에 이 문제를 먼저 푼다.
스케줄러·relay처럼 사람이 아니라 시스템이 부르는 쓰기 유스케이스는 `@RequiresPermission` 대신
`@SystemInvoked(reason = "...")`를 단다(`src/application-core/src/main/java/dev/caskeleton/application/security/SystemInvoked.java`).
스케줄러 스레드에는 인증이 없어 권한 선언은 장식이 되기 때문이다. 이 표시가 붙은 유스케이스는 **권한
검사가 없는 쓰기**이므로 전송 계층이 닿으면 안 된다. 그래서 두 가지가 막는다.
| 막는 것 | 어디 |
| --- | --- |
| bean으로 등록하면 기동 실패. 조립 모듈이 `new`로 만들어 스케줄러에만 준다(예: `composition:outbox-relay`) | `src/composition/service-runtime/src/main/java/dev/caskeleton/composition/runtime/authorization/PermissionEnforcementGuard.java` |
| inbound adapter가 그 클래스를 직접 쓰면 빌드 실패 | `INBOUND_ADAPTERS_DO_NOT_REACH_SYSTEM_INVOKED_USE_CASES`(`ApplicationBoundaryArchitectureTest`) |
D4 실패(`MUTATING_USE_CASES_DECLARE_REQUIRED_PERMISSION`)를 끄려고 이 표시를 쓰지 않는다. 사용자가 부르는
쓰기라면 권한을 선언하고 역할에 매핑한다.
## web 없는 조립
web이 빠진 조립(gRPC 전용·배치 전용)에는 `@RequiresPermission`을 검사할 advisor가 없다. 그런 조립에
권한을 선언한 bean이 있으면 `PermissionEnforcementGuard`가 기동을 실패시킨다(리뷰 01-06). web 없이
조립하는 서비스는 자기 전송에 맞는 검사 advisor를 함께 넣어야 기동한다.
## 지금 이해하면 되는 것
- 쓰기 = `@RequiresPermission` + non-final + bean + 역할 매핑.
- 시스템만 부르는 쓰기 = `@SystemInvoked` + bean 아님 + 전송에서 안 닿음.
- 객체 판정은 유스케이스 안, 남의 것은 404.
- 역할은 자기 client 것만 읽는다.
+31 -11
View File
@@ -32,6 +32,10 @@ DB 저장과 broker 발행을 따로 하면(dual write) 둘 중 하나만 성공
등록을 빠뜨리면 append 때 실패한다. 예: `DurableOperationAutoConfiguration.durableOperationSubmittedContract()`.
5. 테스트: 계약 contribution 테스트(해시 고정), 유스케이스가 append했는지(`InMemoryOutboxStore`),
소비자 중복 제거 계약(sample `WorkLogReservedConsumerDedupeContractTest`).
6. schema를 출시한다: `src/test/resources/contracts/released-payload-schemas.sha256`에 새 version 파일의
해시를 한 줄 더한다. 출시한 version 파일은 고치지 않는다 — 바뀌면 `ReleasedPayloadSchemaContract`
테스트가 실패한다. 필드를 바꾸는 변경은 새 version으로 낸다. 절차: `src/application-core/README.md`
"payload version 올리기"(리뷰 06-10).
## 판단 지점
@@ -49,22 +53,38 @@ DB 저장과 broker 발행을 따로 하면(dual write) 둘 중 하나만 성공
### event id·correlation·순서
- event id: UUIDv7, 소비자 중복 제거 키. 같은 작업의 재시도가 같은 event id를 내야 하면(durable
operation) 작업 id에서 파생한다.
- correlation: 요청의 correlation id가 문법에 맞으면 그것, 아니면 event id. 문법 밖 헤더가 업무 쓰기를
실패시키면 안 된다.
draft는 `IntegrationEventDrafts.draft(contract, eventId, aggregate, order, occurredAt, requestCorrelationId, payload)`로
만든다(`src/application-core/src/main/java/dev/caskeleton/application/messaging/event/IntegrationEventDrafts.java`).
계약 id·payload 버전·목적지는 계약에서 오고, 기능이 정하는 것은 aggregate·순서·payload뿐이다.
`new IntegrationEventDraft<>(…)`를 직접 쓰면 빌드가 막는다(`INTEGRATION_EVENT_DRAFTS_ARE_BUILT_BY_THE_SHARED_HELPER`).
예: sample `PortfolioEventOutbox`, `SubmitDurableOperationUseCase`.
- event id: 정규 소문자 UUIDv7, 소비자 중복 제거 키. draft(`IntegrationEventDraft`)가 다른 값(대문자,
v4, `event-1` 같은 문자열)을 거부하므로 업무 쓰기에서 바로 실패한다 — 커밋된 뒤 relay에서 DEAD가 되지
않는다(리뷰 06-06). 같은 작업의 재시도가 같은 event id를 내야 하면(durable operation) 작업 id를 쓰고,
작업 id가 UUIDv7이 아니면 작업 id와 접수 시각에서 UUIDv7을 파생한다.
- correlation: 헬퍼가 정한다. 요청의 correlation id가 `IntegrationIdentity` 문법·길이에 맞으면 그것,
아니면 event id. 문법 밖 헤더가 업무 쓰기를 실패시키지 않는다. 값을 미리 검사해야 하면 문법을 베끼지
말고 `IntegrationIdentity.isCanonical(value, IntegrationIdentity.CORRELATION_MAX_LENGTH)`를 부른다
(aggregate id는 `AGGREGATE_ID_MAX_LENGTH`).
- tenant 범위: tenant가 없는 기능은 `IntegrationEventDrafts.SINGLE_TENANT_SCOPE`(`global`).
- aggregate 순서: `OutboxAggregateSequencePort.nextSequence(aggregate)`로 같은 `tx.inWrite` 안에서
받는다(V21 카운터, 커밋 순서를 따른다). 이벤트가 하나뿐인 aggregate는 `AggregateOrder.first()`. 발생
시각(벽시계)은 쓰지 않는다 — 서버 시계가 어긋나면 순서가 뒤집힌다(리뷰 06-07).
- 이 규칙은 아직 기능마다 다시 쓴다(sample `PortfolioEventOutbox`). application-core 헬퍼로 올리는 일은
리뷰 01-09 후속이다. 그 전까지는 위 세 규칙을 지킨다.
시각(벽시계)은 쓰지 않는다 — 서버 시계가 어긋나면 순서가 뒤집힌다(리뷰 06-07). 벽시계 번호를 쓰던
서비스가 포트로 옮기면 V21 floor 덕분에 새 번호는 옛 번호 뒤에서 시작한다. 그래서 첫 번호가 1이
아니라 약 1.7×10¹⁵일 수 있다. 소비자는 번호를 "마지막으로 본 값보다 큰가"로만 비교하고 연속성을
가정하지 않는다.
## relay와 실패
- relay는 `composition:outbox-relay`가 조립한다. 유스케이스로 만든 relay를 bean으로 등록하지 않는다
(스케줄러 스레드에는 인증이 없어 권한 프록시가 매번 거부한다).
- broker의 **재시도 가능한** 거부는 DEAD가 아니라 backoff 재시도다. broker가 멈춰도 대량 DEAD가 되지
않는다(1라운드 06-01·06-03). 진짜 인증 오류도 재시도로 남으니 경보로 잡는다.
- relay는 `composition:outbox-relay`가 `new`로 조립해 스케줄러에만 준다. relay는 `@SystemInvoked`라 권한
선언이 없어서, bean이 되면 전송 계층이 **권한 검사 없이** 닿는다. 그래서 bean으로 등록하면
`PermissionEnforcementGuard`가 기동을 실패시킨다(07장 「시스템이 부르는 쓰기」).
- broker의 **재시도 가능한** 거부와 응답 없음은 DEAD가 아니라 backoff 재시도다. broker가 멈춰도 대량
DEAD가 되지 않는다(1라운드 06-01·06-03). 대신 끝이 없다: 진짜 인증 오류, Kafka adapter가 이름을 모르는
broker 거부도 영원히 재시도된다. 행의 `last_failure_code`와 로그의 `error.reason`이 원인 코드를 남기고,
`outbox.retry.long.size`가 오래 재시도 중인 행을 센다. 처분(보류·재개·skip)은 publish-failed runbook
§4.1이다(재리뷰 RR-05-03).
- 발행 응답을 못 받으면 결과 불명(`OutboxPublishAmbiguousException`) — 05장 규칙과 같다.
- 운영: `docs/runbooks/outbox-dead-letter.md`, `docs/runbooks/outbox-publish-failed.md`.
+3 -1
View File
@@ -26,7 +26,9 @@ producing-service identity다.
## Outbox
outbox는 canonical 하나다(2026-09-28). use case가 `IntegrationEventEncoderPort`로 만든
application의 발행 경로는 하나다(2026-09-28): `outbox_event` 표와 `composition:outbox-relay`. vendored
platform의 `messaging_outbox`(`outbox-inbox.md`)는 이 경로에 쓰이지 않는다 — 두 모델의 상태 대응은
`outbox-inbox.md` 머리의 표에 있다. use case가 `IntegrationEventEncoderPort`로 만든
`ValidatedIntegrationEvent`를 `OutboxAppendPort`로 append하고, relay는 claim한 row의 exact envelope
bytes를 `IntegrationEventPublishPort`로 보낸다. 예전의 `app.outbox.canonical-transport-enabled`
gate와 legacy `MessageBroker → KafkaSender` 경로는 삭제됐다. 발행할 contract의 logical destination마다
+20
View File
@@ -1,5 +1,25 @@
# Outbox · Inbox
> **이 문서의 Outbox는 vendored messaging platform의 outbox다**(`messaging_outbox` 표,
> `messaging-reliability`의 `OutboxRelay`). 이 템플릿 application이 쓰는 outbox는 그것이 아니다.
> application은 `outbox_event` 표(`adapter:outbound:persistence-jpa`)에 append하고, relay는
> `composition:outbox-relay`가 돌린다. 그 동작과 운영은 `docs/manual/09-outbox-events.md`,
> `src/composition/outbox-relay/README.md`, `docs/runbooks/outbox-dead-letter.md`,
> `docs/runbooks/outbox-publish-failed.md`에 있다. platform outbox는 소유자 결정으로 제품 안에 남아
> 있지만, platform outbox repository bean이 없으면 그 relay는 만들어지지 않는다(리뷰 2026-11-01 06-11).
>
> 두 모델은 같은 이름의 상태가 반대 뜻이다. 이름으로 옮기면 확정 거부는 영원히 재시도되고 응답 없는
> 메시지는 버려진다. 대응은 아래 표 하나로만 읽는다(`OutboxPublishOutcome` Javadoc과 같은 내용).
>
> | 일어난 일 | application `outbox_event` | platform `messaging_outbox` |
> |---|---|---|
> | broker가 확인했다 | `PUBLISHED` | `PUBLISHED` |
> | 응답이 없었다(저장됐을 수 있음) | `FAILED`(재시도, 예산 안 씀, 끝 없음) | `AMBIGUOUS`(재시도, 예산을 다 쓰면 `EXHAUSTED`) |
> | 지금만 거부됐다(종료·포화·credential) | `FAILED`(재시도, 예산 안 씀, 끝 없음) | `FAILED`(종단) |
> | 확정 거부됐다(같은 payload는 또 거부) | `DEAD`(종단, 수동 처분) | `FAILED`(종단) |
> | publisher가 예외를 던졌고 시도 예산을 다 썼다 | `DEAD`(종단, 수동 처분) | `EXHAUSTED`(종단) |
> | 운영자가 재시도를 멈췄다 | `PARKED`(수동 재개·skip) | 없음 |
>
> **이 템플릿 application의 소비 경로**: application의 `MessageHandler`와 inbox 부품(애플리케이션
> DataSource 위 `JdbcInboxRepository`, `PlatformTransactionManager` 위 `TransactionRunner`)은
> `adapter:inbound:messaging`(`src/adapter/inbound/messaging/README.md`)이 둔다. platform starter는 두
+1 -1
View File
@@ -48,7 +48,7 @@ still waiting on gets claimed by a second worker, and the recipient receives the
| `callbacks.replay-skew` | `APP_NOTIFICATION_PLATFORM_CALLBACK_REPLAY_SKEW` | `1h` | positive; SNS messages whose `Timestamp` is further than this from receipt, either way, are rejected. The default is SNS's longest HTTP/S retry period (3,600 s), so a bounce or complaint SNS redelivers after an outage is still accepted; a replay inside the window is a duplicate in the provider event ledger, not a second event (review 05-20) |
| `callbacks.trusted-proxies` | `APP_NOTIFICATION_PLATFORM_CALLBACK_TRUSTED_PROXIES` | *(empty)* | CSV of peer addresses |
콜백 응답: 기록(중복 포함) `204`, 거절 `400`(서명·topic·시각·형식), 본문 초과 `413`. SNS 서명 인증서를 가져오지 못하면 거절하지 않고 `503`+`Retry-After`로 답한다 — SNS는 4xx를 처리 완료로 보고 재전송을 멈추지만 5xx는 다시 보낸다(review 05-20).
콜백 응답: 기록(중복 포함) `204`, 거절 `400`(서명·topic·시각·형식), 본문 초과 `413`. SNS 서명 인증서를 가져오지 못하면(5xx·408·429·네트워크 실패) 거절하지 않고 `503`+`Retry-After`로 답한다 — SNS는 4xx를 처리 완료로 보고 재전송을 멈추지만 5xx는 다시 보낸다(review 05-20). 인증서 URL이 그 밖의 4xx(403·404 등)로 답하면 SNS가 발급하지 않은 인증서를 가리킨 위조 콜백이므로 서명 검증 실패로 `400` 거절하고 보안 감사(`callbackSignatureRejected`)에 남긴다(review 2026-11-03 RR-07-03). 호스트 검사를 통과한 URL이라 답하는 쪽은 SNS다.
여러 provider가 요청 URL에 서명하므로, 그 URL을 잘못 재구성하면 정상 webhook이 전부 서명 실패가 된다.
`trusted-proxies`가 비어 있으면 forwarded 헤더를 **믿지 않고** 컨테이너가 관측한 값을 쓴다. 무조건 믿으면
+2 -1
View File
@@ -54,7 +54,8 @@
"status": "satisfied",
"lane": ":adapter:outbound:persistence-jpa:jpaPlatformContractTest",
"evidence": [
"adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/WorkerCrashRecoveryContractTest.java"
"adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/WorkerCrashRecoveryContractTest.java",
"adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlNotificationBackgroundPassIntegrationTest.java"
]
},
"provider-wire-qualified": {
+7 -4
View File
@@ -49,10 +49,13 @@ assembler is absent because the relay was not configured.
| Extension | nothing; the receiving contract owns its own guarantees |
| Experimental | nothing; the grade is the warning |
`recoverable` was met on 2026-08-15 by `WorkerCrashRecoveryContractTest`, which walks the four
phases a dispatch passes through — claimed, attempt written, request started, body committed — and
asserts for each that the delivery becomes claimable again or becomes a question for the provider,
never both and never neither.
`recoverable` is backed by two tests. `WorkerCrashRecoveryContractTest` walks the four phases a
dispatch passes through — claimed, attempt written, request started, body committed — and pins the
rows a dead worker leaves at each: a passed lease, and from the attempt insert on an unfinished
attempt carrying that phase's evidence. It writes those rows itself, so it checks recovery's input,
not recovery. `PostgreSqlNotificationBackgroundPassIntegrationTest` runs lease recovery and
reconciliation over rows the dispatcher's own factory wrote and asserts the outcome: requeued before
any attempt, reconciled after one (review 05-01, RR-03-07).
The remaining unmet claim, and what would meet it:
+54 -3
View File
@@ -1670,7 +1670,7 @@ env_keys:
required: false
reload_policy: restart-only
owner_branch: redis-production-capability-completion
validation: positive_duration_le_1h
validation: duration_gt_in_flight_retry_after_2s_le_1h
compatibility_impact: additive
required_test: redis-idempotency-contract:processing-lease-bounded
@@ -1806,6 +1806,24 @@ env_keys:
compatibility_impact: additive
required_test: redis-capability:cache-region-composition
- name: APP_CACHE_REDIS_MAXIMUM_SCRIPT_CONNECTIONS
# The cache's own script-lane connections. Separate from APP_REDIS_CAPACITY_MAXIMUM_SCRIPT_CONNECTIONS,
# which idempotency, rate limiting and leases share, so cache fills cannot starve them (review
# 2026-11-03 README section 4, RR-04-02).
property: app.capabilities.cache.maximum-script-connections
owner_module: adapter-outbound-cache-redis
type: integer
default: 4
allowed_values: null
classification: public-config
required: false
required_when: app.capabilities.cache.bindings.default=redis
reload_policy: restart-only
owner_branch: refactor/maturity-review-2026-11
validation: positive_integer
compatibility_impact: additive
required_test: redis-capability:cache-script-lane-separate-from-correctness
- name: APP_IDEMPOTENCY_REDIS_COMMAND_TIMEOUT
# The ceiling on one owner-safe transition.
type: duration
@@ -2000,7 +2018,7 @@ env_keys:
# secret://<acl-account>@<source>/<name> form as the application reference. Unset, scripts run
# as the application account and startup logs a warning saying so. The Compose Redis lanes use
# secret://ca-skeleton-application-advanced@environment/REDIS_LANE_ADVANCED_PASSWORD, a fixture
# account of infra/redis-sdk/acl/all-accounts.acl.
# account of infra/redis-sdk/acl/lane-fixture-accounts.acl.
property: app.redis.authentication.advanced-credential-reference
owner_module: adapter-outbound-cache-redis
type: string
@@ -2043,7 +2061,7 @@ env_keys:
# <name>. Without the account part the client authenticates as `default`, which a hardened
# Redis turns off. The Compose Redis lanes use
# secret://ca-skeleton-application@environment/REDIS_LANE_APPLICATION_PASSWORD, a fixture
# account of infra/redis-sdk/acl/all-accounts.acl.
# account of infra/redis-sdk/acl/lane-fixture-accounts.acl.
property: app.redis.authentication.credential-reference
owner_module: adapter-outbound-cache-redis
type: string
@@ -4043,6 +4061,39 @@ env_keys:
compatibility_impact: behavior-change
required_test: websocket-contract:canonical-property-namespace
- name: APP_OBJECTSTORAGE_ENABLED
# 2026-11-03 RR-06-07: the explicit off the Compose smoke wrapper sets for every lane that is
# not about object storage. It bound by relaxed binding and was in no registry, so the env
# contract could not tell it from an unread name. The module is on when present; absent from
# the build is the ordinary off state.
property: app.object-storage.enabled
owner_module: adapter-outbound-objectstorage
type: boolean
default: true
allowed_values: [true, false]
classification: public-config
required: false
reload_policy: restart-only
owner_branch: refactor/maturity-review-2026-11
validation: boolean
compatibility_impact: none
required_test: off-state-zero-side-effects-and-env-round-trip
- name: APP_GRPC_SERVER_ENABLED
# 2026-11-03 RR-06-07: the explicit off for the gRPC server the Compose lanes set (it replaced
# CASKELETON_GRPC_ENABLED, which nothing read). The module is on when present; absent from the
# build is the ordinary off state.
property: app.grpc.server.enabled
owner_module: adapter-inbound-grpc
type: boolean
default: true
allowed_values: [true, false]
classification: public-config
required: false
reload_policy: restart-only
owner_branch: refactor/maturity-review-2026-11
validation: boolean
compatibility_impact: none
required_test: off-state-zero-side-effects-and-env-round-trip
- name: APP_GRAPHQL_ENABLED
# 2026-09-25 capability packaging: the module is on when present (its own config yml maps
# this key with default true); false is the explicit off. Absent from the build is the
+20 -1
View File
@@ -401,6 +401,9 @@ errors:
# source: docs/web/http-contract.md §2 (2026-11-01 소유자 결정, 리뷰 02-01 대안 A) — 결과 불명. 같은 Idempotency-Key로 Retry-After 뒤 재전송. 새 키 금지.
# 2초 = 기본 processing lease(30s)보다 짧은 고정값. 이전 "client는 polling"(retryable=false)은 조회 엔드포인트가 없어 폐기.
# 값 하나: IdempotencyExecutor.IN_FLIGHT_RETRY_AFTER. processing-lease가 이 값 이하면 기동 실패(재리뷰 RR-02-05).
# behavior-change: retryable false→true는 web Envelope뿐 아니라 gRPC trailer error-retryable(status ABORTED)과
# GraphQL extension retryable에도 나간다(재리뷰 RR-02-06, GrpcStatusMapperTest·GraphQlWireErrorMapperTest가 고정).
- code: IDEMPOTENT_IN_FLIGHT
category: CONFLICT
http_status: 409
@@ -411,7 +414,7 @@ errors:
client_safe_message: "현재 상태에서는 요청을 처리할 수 없습니다."
log_level: INFO
runbook_link: null
compatibility_impact: none
compatibility_impact: behavior-change
required_test: contract-verification:idempotency
# source: feature-rate-limit-idempotency-contract L72 — "fingerprint mismatch (same key + different body) = 422 IDEMPOTENT_REQUEST_MISMATCH"
@@ -442,6 +445,22 @@ errors:
compatibility_impact: additive
required_test: contract-verification:idempotency
# source: 재리뷰 2026-11-03 README 4절 결정, RR-02-03 — action이 돈 뒤 complete 단계에서 저장소가 응답하지 않음.
# 실행 전 거부(IDEMPOTENCY_UNAVAILABLE)와 code로 구분한다. 결과 불명: 같은 키로 재전송하면 서버가 replay·in-flight·
# recovery-required로 판정한다. 새 키 금지. docs/web/http-contract.md §2.
- code: IDEMPOTENCY_COMPLETION_UNKNOWN
category: TRANSIENT_DEPENDENCY
http_status: 503
retryable: true
retry_after_seconds: null
owner_branch: feature-rate-limit-idempotency-contract
owner_layer: application
client_safe_message: "일시적인 오류입니다. 잠시 후 다시 시도해 주세요."
log_level: ERROR
runbook_link: "runbook://idempotency/unavailable"
compatibility_impact: additive
required_test: contract-verification:idempotency
# source: review-followup P15 (X2-P2-2a) — replay cannot decide whether an external effect happened
- code: IDEMPOTENCY_RECOVERY_REQUIRED
category: DATA_INTEGRITY
+116 -202
View File
@@ -349,8 +349,10 @@ metrics:
cardinality_limit: 50
- name: outcome
cardinality_limit: 5
# STALE: the relay's claim was taken over before its result could be written (V17 lease fence).
allowed_values: [PUBLISHED, FAILED, DEAD, IN_FLIGHT, STALE]
# The relay cycle outcomes (OutboxRelayResult.Outcome), not row statuses: IN_FLIGHT is a status
# and is never counted. STALE: the relay's claim was taken over before its result could be
# written (V17 lease fence). RELEASED: a stopping relay gave back a claim it had not used.
allowed_values: [PUBLISHED, FAILED, DEAD, STALE, RELEASED]
percentiles: null
histogram_buckets: null
alert_severity_thresholds:
@@ -378,13 +380,14 @@ metrics:
required_test: contract-verification:metrics-cardinality
# source: feature-domain-event-outbox-contract — row status enum + Outbox Defaults attemptCount
# PARKED: an operator held a row that was retried without end (re-review 2026-11-03 RR-05-03).
- name: outbox.pending.size
type: gauge
unit: total
tags:
- name: status
cardinality_limit: 5
allowed_values: [PENDING, IN_FLIGHT, PUBLISHED, FAILED, DEAD]
cardinality_limit: 6
allowed_values: [PENDING, IN_FLIGHT, PUBLISHED, FAILED, DEAD, PARKED]
percentiles: null
histogram_buckets: null
alert_severity_thresholds:
@@ -394,6 +397,35 @@ metrics:
compatibility_impact: additive
required_test: contract-verification:metrics-cardinality
# source: re-review 2026-11-03 RR-05-03 — a refusal for now and an unanswered publish are retried
# without end; these count the FAILED rows deferred at least 10 times (OutboxMetrics
# LONG_RETRY_DEFERRED_CLAIMS, the publish-failed runbook's §4.1 query) and age the oldest of them.
- name: outbox.retry.long.size
type: gauge
unit: total
tags: []
percentiles: null
histogram_buckets: null
alert_severity_thresholds:
p2: "> 0 for 10m (runbook://outbox/publish-failed §4.1)"
owner_branch: feature-domain-event-outbox-contract
log_field_mapping: []
compatibility_impact: additive
required_test: contract-verification:metrics-cardinality
- name: outbox.retry.long.age
type: gauge
unit: seconds
tags: []
percentiles: null
histogram_buckets: null
alert_severity_thresholds:
p2: "> 3600 (runbook://outbox/publish-failed §4.1)"
owner_branch: feature-domain-event-outbox-contract
log_field_mapping: []
compatibility_impact: additive
required_test: contract-verification:metrics-cardinality
# === Cache ===
# source: feature-cache-consistency-contract — Decisionized Work Items "cache pattern | cache-aside default"
- name: cache.gets.total
@@ -501,240 +533,122 @@ metrics:
compatibility_impact: additive
required_test: contract-verification:metrics-cardinality
# source: redis-production-capability Task 16 — closed semantic operation outcomes
- name: redis.capability.operations.total
type: counter
unit: total
tags:
- name: capability
cardinality_limit: 6
allowed_values: [cache, rate_limit, idempotency, efficiency_lease, session, runtime]
- name: role
cardinality_limit: 3
allowed_values: [cache, coordination, session]
- name: operation
cardinality_limit: 24
allowed_values:
- lookup
- record
- invalidate
- refresh_claim
- refresh_release
- rate_evaluate
- idempotency_claim
- idempotency_start
- idempotency_renew
- idempotency_complete
- idempotency_fail
- idempotency_release
- idempotency_inspect
- lease_acquire
- lease_inspect
- lease_renew
- lease_release
- session_create
- session_inspect
- session_save
- session_touch
- session_revoke
- session_rotate
- route_command
- name: redis_outcome
cardinality_limit: 15
allowed_values:
- success
- hit
- miss
- denied
- contended
- conflict
- incompatible
- unavailable
- overloaded
- closed
- indeterminate
- stale
- skipped
- tombstoned
- absolute_expired
- name: certainty
cardinality_limit: 3
allowed_values: [definite, not_applied, indeterminate]
percentiles: null
histogram_buckets: null
alert_severity_thresholds:
p1: "required coordination/session unavailable or indeterminate mutation sustained for 2m"
p2: "optional cache unavailable or overloaded above baseline for 5m"
owner_branch: redis-production-capability
log_field_mapping: [capability, role, operation, redis_outcome, certainty]
compatibility_impact: additive
required_test: contract-verification:metrics-cardinality
# source: redis-production-capability Task 16 — monotonic semantic operation duration
- name: redis.capability.duration.seconds
# === Redis connection lanes ===
# source: review 2026-11-03 RR-04-02 — the cache's conditional writes got their own lane
# (CACHE_SCRIPT) so that cache fills cannot take the SCRIPT lane's connections from idempotency,
# rate limiting and leases. These meters show, per lane, whether that holds. Emitted by
# redis-runtime RedisLaneMetrics for every lane RedisRuntimeOwner pools, only when the context has
# a MeterRegistry. Every counter series is registered at zero at startup.
- name: redis.lane.acquire.wait.seconds
type: timer
unit: seconds
tags:
- name: capability
cardinality_limit: 6
allowed_values: [cache, rate_limit, idempotency, efficiency_lease, session, runtime]
- name: role
cardinality_limit: 3
allowed_values: [cache, coordination, session]
- name: operation
cardinality_limit: 24
allowed_values:
- lookup
- record
- invalidate
- refresh_claim
- refresh_release
- rate_evaluate
- idempotency_claim
- idempotency_start
- idempotency_renew
- idempotency_complete
- idempotency_fail
- idempotency_release
- idempotency_inspect
- lease_acquire
- lease_inspect
- lease_renew
- lease_release
- session_create
- session_inspect
- session_save
- session_touch
- session_revoke
- session_rotate
- route_command
- name: redis_outcome
cardinality_limit: 15
allowed_values:
- success
- hit
- miss
- denied
- contended
- conflict
- incompatible
- unavailable
- overloaded
- closed
- indeterminate
- stale
- skipped
- tombstoned
- absolute_expired
- name: lane
# The lane's name: REGULAR, SCRIPT (redis-runtime), CACHE_SCRIPT (cache-redis), BLOCKING,
# TRANSACTION, ADMIN (redis-sdk), plus any lane a service composes as a RedisLaneContribution.
cardinality_limit: 12
- name: outcome
cardinality_limit: 2
allowed_values: [admitted, refused]
percentiles: [0.5, 0.9, 0.95, 0.99]
histogram_buckets: slo_driven
alert_severity_thresholds:
p2: "p99 approaches the configured command or caller deadline for 10m"
p2: "SCRIPT admitted p99 > 100ms for 10m (the correctness lane is queueing)"
p3: "CACHE_SCRIPT admitted p99 > 100ms for 30m"
owner_branch: redis-production-capability
log_field_mapping: [capability, role, operation, redis_outcome]
log_field_mapping: [lane, outcome]
compatibility_impact: additive
required_test: contract-verification:metrics-cardinality
# source: redis-production-capability Task 16 — admission rejected before command ownership
- name: redis.capability.admission.rejected.total
- name: redis.lane.acquire.refused.total
type: counter
unit: total
tags:
- name: role
cardinality_limit: 3
allowed_values: [cache, coordination, session]
- name: admission
cardinality_limit: 2
allowed_values: [rejected_saturated, rejected_closed]
- name: lane
# The lane's name: REGULAR, SCRIPT (redis-runtime), CACHE_SCRIPT (cache-redis), BLOCKING,
# TRANSACTION, ADMIN (redis-sdk), plus any lane a service composes as a RedisLaneContribution.
cardinality_limit: 12
- name: reason
cardinality_limit: 4
allowed_values: [saturated, closing, interrupted, connect_failed]
percentiles: null
histogram_buckets: null
alert_severity_thresholds:
p1: "required role rejection sustained above zero for 2m"
p2: "optional cache saturation sustained for 5m"
p1: "SCRIPT or REGULAR saturated or connect_failed above zero for 2m while a correctness role is selected"
p2: "CACHE_SCRIPT saturated above baseline for 10m"
owner_branch: redis-production-capability
log_field_mapping: [role, admission]
log_field_mapping: [lane, reason]
compatibility_impact: additive
required_test: contract-verification:metrics-cardinality
# source: redis-production-capability Task 16 — bounded admitted command count observation
- name: redis.capability.inflight.total
- name: redis.lane.connections.opened.total
type: counter
unit: total
tags:
- name: lane
# The lane's name: REGULAR, SCRIPT (redis-runtime), CACHE_SCRIPT (cache-redis), BLOCKING,
# TRANSACTION, ADMIN (redis-sdk), plus any lane a service composes as a RedisLaneContribution.
cardinality_limit: 12
percentiles: null
histogram_buckets: null
alert_severity_thresholds:
p3: "opened rate after warm-up above baseline for 15m (reconnect churn against a slow Redis)"
owner_branch: redis-production-capability
log_field_mapping: [lane]
compatibility_impact: additive
required_test: contract-verification:metrics-cardinality
- name: redis.lane.connections.discarded.total
type: counter
unit: total
tags:
- name: lane
# The lane's name: REGULAR, SCRIPT (redis-runtime), CACHE_SCRIPT (cache-redis), BLOCKING,
# TRANSACTION, ADMIN (redis-sdk), plus any lane a service composes as a RedisLaneContribution.
cardinality_limit: 12
- name: reason
cardinality_limit: 2
allowed_values: [invalidated, broken]
percentiles: null
histogram_buckets: null
alert_severity_thresholds:
p3: "invalidated rate above baseline for 15m (timed-out or failed calls discarding connections)"
owner_branch: redis-production-capability
log_field_mapping: [lane, reason]
compatibility_impact: additive
required_test: contract-verification:metrics-cardinality
- name: redis.lane.connections.active
type: gauge
unit: total
tags:
- name: role
cardinality_limit: 3
allowed_values: [cache, coordination, session]
- name: state
cardinality_limit: 3
allowed_values: [idle, active, saturated]
- name: lane
# The lane's name: REGULAR, SCRIPT (redis-runtime), CACHE_SCRIPT (cache-redis), BLOCKING,
# TRANSACTION, ADMIN (redis-sdk), plus any lane a service composes as a RedisLaneContribution.
cardinality_limit: 12
percentiles: null
histogram_buckets: null
alert_severity_thresholds:
p2: "saturated series remains nonzero for 5m"
p2: "SCRIPT active == configured maximum-script-connections for 5m"
owner_branch: redis-production-capability
log_field_mapping: [role, state]
log_field_mapping: [lane]
compatibility_impact: additive
required_test: contract-verification:metrics-cardinality
# source: redis-production-capability Task 16 — observations of exact sanitized RoleHealth
- name: redis.capability.readiness.total
type: counter
- name: redis.lane.connections.pending
type: gauge
unit: total
tags:
- name: capability
cardinality_limit: 5
allowed_values: [cache, rate_limit, idempotency, efficiency_lease, session]
- name: role
cardinality_limit: 3
allowed_values: [cache, coordination, session]
- name: state
cardinality_limit: 3
allowed_values: [available, unavailable, overloaded]
- name: reason
cardinality_limit: 11
allowed_values:
- command_unavailable
- route_closed
- semantic_probe_succeeded
- semantic_read_write_failed
- semantic_program_acl_denied
- semantic_program_failed
- server_version_unsupported
- semantic_probe_in_progress
- semantic_observation_stale
- command_saturated
- recent_command_failure
- name: requirement
cardinality_limit: 2
allowed_values: [optional, required]
- name: lane
# The lane's name: REGULAR, SCRIPT (redis-runtime), CACHE_SCRIPT (cache-redis), BLOCKING,
# TRANSACTION, ADMIN (redis-sdk), plus any lane a service composes as a RedisLaneContribution.
cardinality_limit: 12
percentiles: null
histogram_buckets: null
alert_severity_thresholds:
p1: "required coordination/session unavailable for 2m"
p2: "optional cache unavailable or overloaded for 5m"
p2: "SCRIPT pending > 0 for 5m"
owner_branch: redis-production-capability
log_field_mapping: [capability, role, state, reason, requirement]
compatibility_impact: additive
required_test: contract-verification:metrics-cardinality
# source: redis-production-capability Task 16 — bounded router shutdown drain result
- name: redis.capability.lifecycle.drain.total
type: counter
unit: total
tags:
- name: role
cardinality_limit: 3
allowed_values: [cache, coordination, session]
- name: drain_outcome
cardinality_limit: 3
allowed_values: [drained, forced_after_timeout, interrupted]
percentiles: null
histogram_buckets: null
alert_severity_thresholds:
p1: "required role forced_after_timeout or interrupted during shutdown"
p2: "optional cache forced close during shutdown"
owner_branch: redis-production-capability
log_field_mapping: [role, drain_outcome]
log_field_mapping: [lane]
compatibility_impact: additive
required_test: contract-verification:metrics-cardinality
@@ -10,6 +10,9 @@
# (feature-secrets-config-source-contract 2026-05-22: "dev/local sentinel value prefix = __LOCAL_DEV_").
# - prod profile에서 `__LOCAL_DEV_` prefix 발견 시 startup fail
# (feature-secrets-config-source-contract 2026-05-22).
# - prod profile에서 `fixture-` prefix 값(이 저장소에 적힌 lane fixture 비밀번호 — infra/redis-sdk/acl,
# compose lane의 REDIS_LANE_*)도 같은 이유로 startup fail. 어떤 변수 이름이든 검사한다
# (review 2026-11-03 RR-07-04).
# - Masking 기본 = `full_except_last_4` (feature-secrets-config-source-contract 2026-05-22:
# "full mask except last 4 chars for non-secret tokens"). 진짜 secret(password/private key)은 `full`.
# - Naming suffix는 보조 신호 (feature-secrets-config-source-contract: "_TOKEN, _KEY, _PASSWORD").
@@ -39,8 +39,8 @@
| --- | --- | --- | --- |
| techlead H10 | 자동조립 publisher가 관측을 기록하지 않음 | 해결 | `messaging-spring-boot-starter/.../MessagingPublishConfiguration.java:224-231` observation 주입 |
| techlead H3 | consumer가 배선되지 않음 | 바뀜 | starter-kafka가 `MessageHandler` bean마다 consumer를 연다(`MessagingConsumeConfiguration.java:60-68`). 다만 application leaf 쪽 경로가 없다 → 06-08 |
| techlead M4 | 실행 경로에 스키마 호환성 검사 없음 | 바뀜 | 문서가 "선언만 하고 강제하지 않음"으로 표시(`docs/messaging/configuration-reference.md:123`). 앱 payload 버전에는 여전히 없음 → NN-10 |
| techlead M5 | outbox 두 개 공존 | 남음(소유자 결정으로 유지) | vendored `MessagingReliabilityAutoConfiguration`·`JdbcOutboxRepository`가 runtime classpath에 있음. 문서·종료 순서 비용 → 06-02, NN-11 |
| techlead M4 | 실행 경로에 스키마 호환성 검사 없음 | 바뀜 | 문서가 "선언만 하고 강제하지 않음"으로 표시(`docs/messaging/configuration-reference.md:123`). 앱 payload 버전에는 여전히 없음 → 06-10 |
| techlead M5 | outbox 두 개 공존 | 남음(소유자 결정으로 유지) | vendored `MessagingReliabilityAutoConfiguration`·`JdbcOutboxRepository`가 runtime classpath에 있음. 문서·종료 순서 비용 → 06-02, 06-11 |
| techlead M7 / H-1b | producer-id 키 부재, 빈 값 통과 | 해결 | `messaging-kafka/src/main/resources/config/module-defaults/messaging.yml` `producer-id`, `MessagingBridgeRootAutoConfiguration.java:77-87` |
| H-1a | bridge 조건이 platform보다 먼저 평가 | 해결 | `MessagingBridgeRootAutoConfiguration.java:32-33` `afterName` |
| H-8 | bridge `prepare`가 모든 예외를 REJECTED로 | 해결 | `PlatformIntegrationEventPublishAdapter.java:59-67`, 테스트 `aFailureOtherThanAnIncompatibleValueIsNotTurnedIntoARefusal` |
@@ -248,7 +248,7 @@
- 완료 기준: 앱 relay가 실제 Kafka로 발행한 레코드를 테스트가 읽어 envelope 바이트를 비교한다. 이 lane이 이름 있는 게이트에 들어 있다.
- 검증 방법: 그 lane의 실행 로그.
### NN-10 — 이벤트 payload 버전을 올리는 방법과 호환성 검사가 없다
### 06-10 — 이벤트 payload 버전을 올리는 방법과 호환성 검사가 없다
- 성격: 구조적 위험
- 관련 관점: 17, 11, 20
@@ -267,7 +267,7 @@
- 완료 기준: 필드 삭제 같은 깨지는 변경이 테스트에서 실패한다.
- 검증 방법: 샘플 contract에 깨지는 v2를 넣는 변이 실험.
### NN-11 — outbox가 둘인데 문서와 주석은 하나처럼 말한다
### 06-11 — outbox가 둘인데 문서와 주석은 하나처럼 말한다
- 성격: 개선 제안(이해 비용)
- 관련 관점: 4, 5, 23, 24
@@ -285,7 +285,7 @@
- 완료 기준: `docs/messaging`에서 앱 outbox 문서로 가는 링크가 있고, 상태 대응표가 하나 있다.
- 검증 방법: 문서 리뷰.
### NN-12 — 배치 크기와 lease 시간의 관계를 검사하지 않는다
### 06-12 — 배치 크기와 lease 시간의 관계를 검사하지 않는다
- 성격: 구조적 위험
- 관련 관점: 12, 19, 21
@@ -300,7 +300,7 @@
- 완료 기준: lease를 넘을 수 있는 설정 조합이 기동에서 거부된다.
- 검증 방법: `OutboxSettings` 바인딩 테스트에 경계 조합 추가.
### NN-13 — 이 영역 문서·주석의 낡은 설명
### 06-13 — 이 영역 문서·주석의 낡은 설명
- 성격: 개선 제안
- 관련 관점: 24, 9, 26
@@ -317,7 +317,7 @@
- 완료 기준: 위 위치의 설명이 코드와 맞는다.
- 검증 방법: 문서 리뷰.
### NN-14 — DLQ 격리 레코드가 모든 원인을 `POISON_MESSAGE`로 적는다
### 06-14 — DLQ 격리 레코드가 모든 원인을 `POISON_MESSAGE`로 적는다
- 성격: 개선 제안
- 관련 관점: 8, 28
@@ -355,7 +355,7 @@
- claim은 `FOR UPDATE SKIP LOCKED`, 인스턴스마다 UUID owner, row마다 token 증가(`JpaOutboxStore.java:105-127`).
- 완료는 owner·token이 맞을 때만 적용된다. 아니면 STALE(`:136-162`). `PostgreSqlOutboxConcurrentRelayIntegrationTest`가 있다.
- 같은 aggregate의 두 row는 두 인스턴스가 동시에 가져갈 수 없다. 앞 row가 PUBLISHED가 아니면 뒤 row는 gate에 걸린다.
- 남은 위험: 배치 시간이 lease를 넘는 설정(NN-12).
- 남은 위험: 배치 시간이 lease를 넘는 설정(06-12).
### 4.4 외부 응답 지연·유실 (Kafka 느림·정지)
@@ -373,7 +373,7 @@
- outbox row는 append 때의 바이트를 그대로 내보낸다. 배포가 바뀌어도 옛 row의 봉투는 변하지 않는다.
- destination 선언이 새 배포에서 빠지면 옛 row는 `PUBLISH_PREPARATION_FAILED` REJECTED → DEAD가 된다. 이것은 설정 오류라 확정 거부로 보는 것이 맞다.
- payload 버전 호환성 검사와 절차가 없다(NN-10).
- payload 버전 호환성 검사와 절차가 없다(06-10).
### 4.7 선택 어댑터 제거
@@ -416,7 +416,7 @@
1. 필요한 것: application은 검증된 봉투를 같은 트랜잭션에 기록하는 outbox가 필요하다. platform은 제품으로서 자기 outbox를 가진다.
2. 지킬 성질: 개발자가 어느 outbox가 동작 중인지 헷갈리지 않는다. platform의 종료·지표 설계가 앱 경로에도 맞는다.
3. 현재 보장: bean 조건으로 vendored relay가 앱에서 생기지 않는다. 문서·종료 순서는 vendored relay 기준이다(06-02, NN-11).
3. 현재 보장: bean 조건으로 vendored relay가 앱에서 생기지 않는다. 문서·종료 순서는 vendored relay 기준이다(06-02, 06-11).
4. 대안: (a) 결정 유지 + 경계 표시 보강, (b) vendored outbox를 starter `implementation`에서 빼서 opt-in 의존으로, (c) 하나로 합치기.
5. 비용: (a) 문서·주석 수정. (b) starter 의존 정리와 vendored 테스트 classpath 조정. (c) 결정 번복.
6. 선택: (a)를 먼저, (b)는 다음 단계 후보. 결정은 유지한다.
@@ -445,7 +445,7 @@
- 이벤트 소비(handler 위치, inbox 조립) 예제와 leaf(06-08).
- aggregate 순서 번호를 만드는 올바른 예(06-07).
- payload 버전 올리기 절차와 호환성 테스트(NN-10).
- payload 버전 올리기 절차와 호환성 테스트(06-10).
- 앱 relay를 실제 Kafka로 돌리는 테스트 틀(06-09).
- `MessagePublisher` fake와 본체의 공통 계약(TEST-FAKE-MessagePublisher).
@@ -457,7 +457,7 @@
4. payload를 바꾸는가? 바꾸면 새 `payloadVersion`을 추가하고 소비자가 먼저 두 버전을 읽게 한다.
5. 이 destination의 소비자는 부작용이 있는가? 있으면 INBOX_TRANSACTIONAL을 쓴다.
6. 한 aggregate가 초당 몇 건을 만드는가? 0.2건/초를 넘으면 현재 relay 기본값으로는 지연이 쌓인다(06-04).
7. relay를 몇 개 인스턴스에서 켤 것인가? batch-size를 올리면 lease 시간과 함께 계산한다(NN-12).
7. relay를 몇 개 인스턴스에서 켤 것인가? batch-size를 올리면 lease 시간과 함께 계산한다(06-12).
## 8. 28개 관점 판정표
@@ -467,27 +467,27 @@
| --- | --- | --- | --- |
| 1 | 코드 성숙도 | P | 실패 경로를 의식한 코드가 많다. platform과 application 사이 실패 의미 번역에 결함(06-01, 06-06) |
| 2 | 구조적 성숙도 | P | 발행 경로는 port → bridge → platform으로 깨끗하다. 소비 경로에 application leaf가 없다(06-08) |
| 3 | 변경 용이성 | P | 새 이벤트 추가 경로는 짧다. payload 버전 변경 경로가 없다(NN-10) |
| 4 | 인지 부하 | P | 두 outbox 모델과 반대 뜻의 상태 이름(NN-11) |
| 3 | 변경 용이성 | P | 새 이벤트 추가 경로는 짧다. payload 버전 변경 경로가 없다(06-10) |
| 4 | 인지 부하 | P | 두 outbox 모델과 반대 뜻의 상태 이름(06-11) |
| 5 | 위험 가시성 | P | 중복 가능 구간은 README에 있다. "REJECTED = 영구"라는 가정이 코드에서 보이지 않는다(06-01) |
| 6 | 실패 성숙도 | X | 일시 거부·종료·짧은 장애가 영구 DEAD로 이어진다(06-01, 06-02, 06-03) |
| 7 | 운영 성숙도 | P | runbook·경보·구조화 ERROR가 있다. relay 꺼짐·실패 중 지표 정지(06-05), runbook 조회가 옛 키 |
| 8 | 디버깅 가능성 | P | failure report에 event·aggregate·correlation id가 있다. DLQ 사유가 한 값(NN-14) |
| 9 | 유지보수 성숙도 | P | 주석·README에 낡은 위치 설명(NN-13) |
| 8 | 디버깅 가능성 | P | failure report에 event·aggregate·correlation id가 있다. DLQ 사유가 한 값(06-14) |
| 9 | 유지보수 성숙도 | P | 주석·README에 낡은 위치 설명(06-13) |
| 10 | 테스트 성숙도 | P | 단위 테스트 촘촘하고 빠름(합계 수 초). 앱 경로 실연 테스트 없음(06-09), 지표 변이 생존(06-05) |
| 11 | 데이터 성숙도 | P | 봉투 바이트·hash 보존이 좋다. 순서 키 유일 제약 없음(06-07) |
| 12 | 동시성 성숙도 | P | SKIP LOCKED + fencing은 좋다. 배치·lease 관계 미검사(NN-12), 종료 순서(06-02) |
| 12 | 동시성 성숙도 | P | SKIP LOCKED + fencing은 좋다. 배치·lease 관계 미검사(06-12), 종료 순서(06-02) |
| 13 | 확장성 | P | 인스턴스 수평 확장은 된다. aggregate당 처리량 상한(06-04) |
| 14 | 성능 성숙도 | ? | 계산만 했다. 실측 없음. 회차당 약 43 DB 문장, 인스턴스당 최대 약 4건/초 |
| 15 | 보안 성숙도 | O | Kafka TLS/SASL·ACL 적용 확인(M-2 해결). 실패 report에 payload를 싣지 못하게 타입으로 막음 |
| 16 | 의존성 성숙도 | P | broker SDK는 선택 줄을 지우면 빠진다. starter가 쓰지 않는 vendored outbox 구현을 `implementation`으로 싣는다 |
| 17 | API / Contract 성숙도 | P | 봉투 v1 schema와 hash 검사가 있다. EventId/MessageId 불일치(06-06), payload 호환성 없음(NN-10) |
| 17 | API / Contract 성숙도 | P | 봉투 v1 schema와 hash 검사가 있다. EventId/MessageId 불일치(06-06), payload 호환성 없음(06-10) |
| 18 | 배포 성숙도 | X | 배포 종료 중 row가 DEAD가 될 수 있다(06-02). broker 실연 lane이 기본 게이트 밖(06-09) |
| 19 | 설정 성숙도 | P | strict 바인딩, producer-id 필수. relay 기본 꺼짐의 결과가 안 보임(06-05), 값 사이 관계 미검사(NN-12) |
| 20 | 개발자 경험 | P | 발행 한 줄 켜기가 쉽다. 소비·버전·순서 번호 안내가 없다(06-07, 06-08, NN-10) |
| 19 | 설정 성숙도 | P | strict 바인딩, producer-id 필수. relay 기본 꺼짐의 결과가 안 보임(06-05), 값 사이 관계 미검사(06-12) |
| 20 | 개발자 경험 | P | 발행 한 줄 켜기가 쉽다. 소비·버전·순서 번호 안내가 없다(06-07, 06-08, 06-10) |
| 21 | Guardrail 성숙도 | P | caller 트랜잭션 강제, 대역 publisher 금지, producer-id 필수가 좋다. 이벤트 id·순서 번호·설정 조합 가드 없음 |
| 22 | 추상화 성숙도 | P | bridge는 필요한 만큼만 안다. outcome 어휘가 platform 의미 하나를 담지 못한다(06-01) |
| 23 | 일관성 | P | platform 종료 설계와 앱 relay 방식이 다르다(06-02, NN-11) |
| 23 | 일관성 | P | platform 종료 설계와 앱 relay 방식이 다르다(06-02, 06-11) |
| 24 | 네이밍 / 의미 성숙도 | P | `OutboxLeaderElectionToken`이 하지 않는 일을 이름으로 말한다. 두 outbox의 FAILED 의미 반대 |
| 25 | 조직 확장성 | P | 발행은 팀별로 독립 추가 가능. 소비는 registry 간선을 각자 만들게 된다(06-08) |
| 26 | Ownership 성숙도 | P | leaf README가 근거를 잘 남긴다. composition·messaging-kafka에 `CLAUDE.md` 없음 |
@@ -196,7 +196,9 @@
## 12. 발견 색인
영역 문서에 36.3 양식으로 적힌 발견 119건: High 21 · Medium 64 · Low 34, P1 25 · P2 59 · P3 35. Critical 0. 두 영역이 같은 결함을 적은 묶음(07-03+09-03, 02-03+07-05, 08-01+04-04+03-12, 05-09+08-04, 04-09+09-01)은 색인에 각각 남기고 2절에서 합쳤다. 06·08 영역 문서는 원래 `NN-` 접두사로 적혀 있어 `06-`·`08-`로 바꿨다.
영역 문서에 36.3 양식으로 적힌 발견 **124건**: High 21 · Medium 67 · Low 36, P1 25 · P2 60 · P3 39.
> **정정 (2026-11-03 재리뷰)**: 처음 이 색인은 119건이었다. 06 영역 문서가 `NN-` 접두사로 쓰여 있었는데 오케스트레이터가 `NN-0x`만 `06-0x`로 바꿔 `NN-10`~`NN-14` 5건이 색인과 모든 수정 묶음에서 빠졌다. 13~16절의 "모든 발견 처리"는 이 5건에 대해 틀렸다. 5건은 `06-10`~`06-14`로 다시 매겨 아래 표에 넣었고, 재리뷰 수정(`../2026-11-03-rereview/README.md` 6절)에서 고쳤다. Critical 0. 두 영역이 같은 결함을 적은 묶음(07-03+09-03, 02-03+07-05, 08-01+04-04+03-12, 05-09+08-04, 04-09+09-01)은 색인에 각각 남기고 2절에서 합쳤다. 06·08 영역 문서는 원래 `NN-` 접두사로 적혀 있어 `06-`·`08-`로 바꿨다.
| ID | 심각도 | 우선순위 | 내용 |
|---|---|---|---|
@@ -319,6 +321,11 @@
| [09-09](09-performance-measurement.md) | Low | P3 | persistence-jpa가 H2 드라이버를 운영 아티팩트에 실어 보냄 |
| [09-10](09-performance-measurement.md) | Low | P3 | `local` 프로필이 `server.port`를 8080으로 고정 |
| [09-11](09-performance-measurement.md) | Low | P3 | 기본 조합 jar에 Jackson 2·3과 kotlin-stdlib가 함께 실림 |
| [06-10](06-messaging.md) | Medium | P2 | 이벤트 payload 버전을 올리는 방법과 호환성 검사가 없다 (색인 누락 정정) |
| [06-11](06-messaging.md) | Medium | P3 | outbox가 둘인데 문서와 주석은 하나처럼 말한다 (색인 누락 정정) |
| [06-12](06-messaging.md) | Medium | P3 | 배치 크기와 lease 시간의 관계를 검사하지 않는다 (색인 누락 정정) |
| [06-13](06-messaging.md) | Low | P3 | 이 영역 문서·주석의 낡은 설명 (색인 누락 정정) |
| [06-14](06-messaging.md) | Low | P3 | DLQ 격리 레코드가 모든 원인을 `POISON_MESSAGE`로 적는다 (색인 누락 정정) |
## 13. 조치 현황 (개선 순서 1~3)
@@ -67,3 +67,46 @@ Minor 41건은 각 문서에 있다.
## 5. 다음
위 Important 전부, Minor 41건, 06-10~14를 영역별 수정 묶음으로 고친다. 각 묶음은 `5786126c`에서 만든 worktree에서 RED→GREEN, 오케스트레이터가 합쳐 전체 게이트 후 커밋·push한다. 결과는 이 문서 6절.
## 6. 수정 결과 (2026-11-03)
영역별 수정 묶음 7개가 `5786126c`(r4-G는 A·B·C·D·F를 합친 임시 기준)에서 `test-driven-development` 스킬 절차로 RED→GREEN 했고, 오케스트레이터가 합쳤다. 하위 에이전트 없음.
| 묶음 | 상태 | 검증 (에이전트) | 처리 내용 | 결정·합칠 때 조치·남은 것 |
|---|---|---|---|---|
| r4-a | 완료 | leaf check 9, cache-redis test, architectureCheck, jpaPlatformContractTest fileserver 47, outbox sequence PG, verifyFeatureScaffold, 소유자 규칙 변이 잡힘 | RR-01-02=RR-07-01 @SystemInvoked bean 기동 실패+ArchUnit, RR-01-01·07·10 생성 명령, RR-01-03·09 매뉴얼·IntegrationEventDrafts ArchUnit, RR-01-04+RR-07-06 fileserver 계약 6개·fake 정합·소유자 정본 FileAccessSubject.owns(RED 17), RR-01-05=RR-04-04 move 쿼터(RED 2), RR-01-06 순서 계약 보강, RR-01-08 | 합칠 때 매뉴얼 09장·OutboxAggregateSequenceContract를 r4-c와 수동 병합(r4-a 새 케이스는 첫 번호 기준 상대값이라 floor와 충돌 없음). 결정: 만료·미해제 예약 확정은 받음, move는 복사 후 삭제 안 씀 |
| r4-b | 완료(sample 데모 체인 한 줄은 r4-f로) | leaf check 6, graphql·web systemTest, webJettyCompat, -p optional-platforms ci integrationCheck(ledger PG 41), architectureCheck, verifyEnvKeys, documentContractTest, verifyPublicPathSnapshot, 영향 모듈 test | RR-02-01 union 복잡도, RR-02-02 EnvelopeErrorController+ERROR 디스패치만 허용(JWT 체인 401→400), RR-02-03 IDEMPOTENCY_COMPLETION_UNKNOWN(503 재시도 가능), RR-02-04 GrpcClaimToken·claim_attempt·V003(옛 owner commit 거부·실패 무시, 조건부 commit), RR-02-05 Retry-After 상수·lease 이하면 기동 실패, RR-02-06~08 | 결정: 새 코드 503(code로 구분), 시도 번호는 카운터, ERROR 디스패치 허용(보안 정책 변경) |
| r4-c | 완료 | leaf check 5, notification test, sample·app-bootstrap architectureTest, architectureCheck, 실 PG jpaPlatformContract 334·Migration 45·Failure 6·V21·runbook·트랜잭션 lane 등 skip 0 | RR-03-01 counter ON CONFLICT(RED 실 PG), RR-03-02=RR-05-02 V21 floor 테이블+counter 미리 생성+행 최대값 하한(RED 3), RR-03-05 lock_timeout 5s·무효 인덱스 거부·CONCURRENTLY 절차 runbook, RR-03-03 마감 초과 분류(H2+실 PG), RR-03-04 양쪽 매핑 양방향 금지, RR-03-06~10 | 결정: V21 직접 수정(아직 어디에도 미적용 — 적용한 개발 DB는 재생성), H2 호환 위해 다문장 counter, 계약 '1부터'→'첫 번호부터 1씩'(JPA 첫 번호 ≈1.7e15) |
| r4-d | 완료(RR-04-02 지표·RR-07-04 운영 거부 남음) | leaf check 6, Redis container 52·topology standalone, Toxiproxy 4, spring70 compat, perf, notification 문서·환경 계약, PG 멱등성 계약 14, verifyEnvKeys, architectureCheck, verifyStrictLaneStages, compose 정적 18 | RR-04-01 호출자 deadline은 breaker 미집계(RED), RR-04-03 JDK 스트리밍 read-idle(RED), RR-04-02 CacheScriptLane(기본 4), RR-04-05 lane 연결, RR-04-06~08·10·11, RR-07-03 SNS 4xx 거절+감사(RED), RR-07-04 ACL 파일 이름 lane-fixture-accounts.acl, RR-07-07 SNS MessageId 기준 | 결정: total-call이 끊은 시도는 실패로 셈, 캐시 lane 4. 남음: lane별 지표(meter registry 없음), 운영 SecretSourceValidator가 fixture- 값 거부 |
| r4-e | 완료 | 건드린 leaf 13 check, Kafka relay·inbox, outbox PG 6·migration·jpaPlatformContract, architectureCheck, verifyDependencyLocks, verifyStrictLaneStages, documentContractTest | RR-05-01 재발행 SQL deferred 초기화+runbook SQL lane(RED 실 PG), RR-05-03 last_failure_code(V22)·오래 재시도 게이지·PARKED·runbook SQL(RED 4), RR-05-04 draft가 비UUIDv7 거부, RR-05-05~09, 06-10 schema 버전 해시 고정, 06-11~14 | 합칠 때 4곳 수동 병합: OrderKeyRunbook IT를 공용 OutboxRunbookSql로 + V22 추가로 latest가 22가 되어 'V21 적용' 단언을 applied 목록 기준으로, sha256에 V22, application-core README, 매뉴얼 09. 결정: PARKED는 DDL 없이, 발행 포트·저장소 포트 시그니처 변경(파생 서비스 깨짐), 오래 재시도 기준 10회, DB CHECK 제약 안 넣음. 남음: 2026-11-01 README 색인 정정(오케스트레이터), V21 주석의 없는 클래스명 |
| r4-f | 완료 | build-logic 107, ci, app-bootstrap documentContract·systemTest, sample check·webSystemTest·integrationTest·posterImageMigration, -p optional-platforms ci, 두 빌드 verifyStrictLaneStages, verifyEnvKeys, compose 정적 | RR-06-01=RR-04-09 규칙 생성 중 다른 singleton 생기면 기동 실패(RED), RR-06-02 lane 정책을 Test 입력으로(RED), RR-06-03·04 optional 빌드 단계 검사·-x/continue-on-error/if 해석(RED), RR-07-02 데모 계정 허용 목록 local·test·demo(RED 6), RR-06-05+07-05 CSRF 제외 축소·sample qualityCheck, RR-06-06~12, RR-02-02 샘플 체인 ERROR 디스패치 허용 | 남음: cache-redis 컨테이너 lane main 단계 중복 실행. 결정: release job --rerun-tasks 안 넣음 |
| r4-g | 완료 | redis-runtime 64·redis-sdk 310·cache-redis 192·app-bootstrap test 222·systemTest 11·documentContract 60, redisContainerContract 53, topology standalone, verifyEnvKeys, architectureCheck, verifyDependencyLocks, verifyStrictLaneStages | RR-04-02 redis.lane.* 지표 6개(RED 10), RR-07-04 prod에서 fixture- 값 거부(RED), cache-redis 컨테이너 lane main 중복 제거 | 합친 뒤 오케스트레이터: r4-g가 찾은 '레지스트리의 redis.capability.* 6개를 아무 코드도 내보내지 않음' → 레지스트리에서 빼고 incident runbook을 redis.lane.* 기준으로, cache-redis README에 미구현 지표로 기록. 남음(의도): httpclient failure injection lane main 2회(05-06/05-16 결정) |
### 6.1 합칠 때 오케스트레이터가 고친 것
- 수동 병합: 매뉴얼 09장(r4-A·r4-C·r4-E), `OutboxAggregateSequenceContract`(r4-A·r4-C), `PostgreSqlOutboxOrderKeyRunbookIntegrationTest`(r4-C·r4-E — 공용 `OutboxRunbookSql`로, V22가 생겨 "V21 적용" 단언을 적용 목록 기준으로), `applied-migrations.sha256`(V22), application-core README.
- r4-G가 찾은 거짓 선언: `docs/registries/metrics.yaml`의 `redis.capability.*` 6개를 어떤 코드도 내보내지 않았고 incident runbook이 그것을 보라고 했다. 레지스트리에서 빼고 runbook을 실제로 나오는 `redis.lane.*` 기준으로 고쳤다. 원래 설계는 cache-redis README 「아직 내보내지 않는 지표」.
- 최종 게이트에서 드러난 2건: (1) sample `CreatePosterConcurrencyIntegrationTest`가 v4 event id를 써서 RR-05-04 이후 16건 모두 거부 → 운영과 같은 `UuidOutboxEventIdFactory`로. (2) `EnvelopeErrorController`의 `@RequestMapping`이 FindSecBugs `SPRING_CSRF_UNRESTRICTED_REQUEST_MAPPING` → 오류 디스패치 전용(보안 체인이 ERROR dispatch만 허용)이라 그 메서드에만 이유를 단 제외.
- 원래 리뷰 README 12절 색인에 06-10~06-14를 넣고 발견 수를 124건으로 정정했다.
## 7. 게이트 (2026-11-03, 합친 메인 작업 트리, 공유 잠금)
첫 실행(22:27)은 `ci`(FindSecBugs 1건)와 `integrationCheck`(sample 동시 생성 테스트)가 실패했고 6.1절대로 고쳤다. 재실행 결과:
| 게이트 | 결과 | 시간 | 실행 직전 부하 |
|---|---|---|---|
| `ci` | rc=0 | 215초 | load 8.27 |
| `:integrationCheck` | rc=0 | 1,647초(부하 높음) | load 29.15 |
| `mainCheck` | rc=0 | 260초 | load 35.16 |
| `-p optional-platforms ci integrationCheck performanceCheck` | rc=0 | 62초 | load 16.51 (첫 실행, 두 수정과 무관) |
| `:performanceCheck` | rc=0 | 25초 | load 20.55 (첫 실행) |
| `:releaseCheck -PgitRevision=<HEAD>` | rc=0 | 4초 | 첫 실행 |
| `scripts/verify-compose-profile-contracts.sh` | rc=0, 18 lane | — | 첫 실행 |
## 8. 남은 것
- **의도적으로 둔 것**: httpclient failure-injection lane이 main 단계에서 두 번 돈다(원래 리뷰 05-06·05-16 결정). capability 단위 Redis 지표(`redis.capability.*`)는 설계만 있고 구현하지 않았다(새 기능 범위, cache-redis README).
- **소유자 확인이 필요한 결정**: 4절 표, 그리고 묶음이 기본값으로 정한 것(6절 "결정" 칸) — 특히 서블릿 오류 디스패치 허용(보안 정책), 데모 계정 허용 목록, V21을 V22 없이 직접 수정(이 브랜치의 V21을 적용한 개발 DB는 재생성), 발행·저장소 포트 시그니처 변경, `IDEMPOTENCY_COMPLETION_UNKNOWN` 신설, prod에서 `fixture-` 값 거부.
- **짝 프론트**: `IDEMPOTENCY_COMPLETION_UNKNOWN`(503, 결과 불명)을 인계 문서에 더했다.
- **이 환경에서 확인할 수 없는 것**: 실제 CI runner 실행.
+3 -1
View File
@@ -44,7 +44,9 @@ the time, and which half is not knowable from the status.
and a later reconciler pass sees that attempt's `COMMITTED` or `FAILED_TERMINAL`. Until a retry
arrives, the case stays pending. The re-run is safe only when the ledger commit is recorded in the
mutation's own transaction; if this service commits them separately, check the business resource
before letting the retry through.
before letting the retry through. An owner that was only slow cannot spoil the takeover: its claim
attempt (`claim_attempt`) is behind, so its late commit is refused and its late failure ignored.
A log line saying an operation "is no longer held by attempt N" is that refusal, not an outage.
**Escalate when** the reconciler's pending list grows across passes. That means the ledger is
unreachable rather than slow. A case pending as `IN_PROGRESS` for longer than the claim TTL with no
+10 -2
View File
@@ -1,10 +1,10 @@
---
title: Runbook — IDEMPOTENCY_UNAVAILABLE (멱등성 조정 저장소 사용 불가)
category: TRANSIENT_DEPENDENCY
error_codes: [IDEMPOTENCY_UNAVAILABLE, IDEMPOTENCY_RECOVERY_REQUIRED]
error_codes: [IDEMPOTENCY_UNAVAILABLE, IDEMPOTENCY_COMPLETION_UNKNOWN, IDEMPOTENCY_RECOVERY_REQUIRED]
severity: P2
owner: oncall
last_updated: 2026-09-28
last_updated: 2026-11-03
status: active
---
@@ -17,6 +17,10 @@ status: active
- `IdempotencyExecutor` refused to run the action because the replay store (JPA or Redis, whichever
`app.idempotency` selects) did not answer. Nothing was executed: the refusal happens before the
action, so a client retry is safe.
- HTTP 503 `IDEMPOTENCY_COMPLETION_UNKNOWN` (`retryable=true`) is the same outage hitting after the
action ran: the store did not answer while the result was being recorded. Effects in the action's
own transaction rolled back; anything else it did (an outbound call, a message) may have happened.
This is not "nothing was executed". Clients resend with the same key.
- HTTP 500 `IDEMPOTENCY_RECOVERY_REQUIRED` (`retryable=false`) means the opposite case: the store
cannot tell whether an earlier attempt's external effect already happened.
@@ -32,6 +36,10 @@ status: active
## Action
- `IDEMPOTENCY_UNAVAILABLE`: restore the store; clients retry with the same key and backoff.
- `IDEMPOTENCY_COMPLETION_UNKNOWN`: restore the store, then check the effects outside the
transaction for the failed requests (the ERROR log line names the code). A same-key resend replays
a result that was in fact recorded, gets `IDEMPOTENT_IN_FLIGHT` while the attempt's lease lasts,
and `IDEMPOTENCY_RECOVERY_REQUIRED` after it — handle that one as below.
- `IDEMPOTENCY_RECOVERY_REQUIRED`: do not ask the client to retry blindly. Confirm whether the
downstream effect of the earlier attempt exists, then complete or release the record so the next
request with that key replays or re-executes deterministically.
+5 -2
View File
@@ -4,7 +4,7 @@ category: TRANSIENT_DEPENDENCY
error_codes: [JOB_TIMEOUT]
severity: P2
owner: oncall
last_updated: 2026-06-13
last_updated: 2026-11-03
status: stub
---
@@ -48,7 +48,10 @@ status: stub
## 5. Escalation
- 의존성 지연이 근본 원인이면 해당 의존성 오너에게 escalate
- shutdown 예산(20s) vs k8s `terminationGracePeriodSeconds`(30s) 정합 이슈면 플랫폼/런타임 오너에게 escalate
- 종료 예산은 차례로 더한 값이다: 웹 drain(`APP_SERVER_SHUTDOWN_TIMEOUT`, 기본 30s)이 끝난 뒤 async 대기 19s,
합계 49s에 preStop(5s)과 여유를 더한다. k8s `terminationGracePeriodSeconds`가 이보다 작으면(예: 기본 30s)
async 대기 중 SIGKILL이다. compose는 60s(`docker-compose.yml` `stop_grace_period`, 합계는
`DeveloperExperienceContractTest`가 검사). 정합 이슈면 플랫폼/런타임 오너에게 escalate
## 6. Recovery / Verification
+94 -14
View File
@@ -4,7 +4,7 @@ category: INTERNAL
error_codes: [OUTBOX_DEAD_LETTER]
severity: P1
owner: oncall
last_updated: 2026-06-11
last_updated: 2026-11-03
status: stub
---
@@ -21,7 +21,12 @@ status: stub
## 2. First Response (5분 이내)
### Step 1 — 확인
1. ERROR log에서 `OUTBOX_DEAD_LETTER` 라인 확인: `event_id`, `event_type`, `correlation_id`, 마지막 실패 원인 예외 추출
1. ERROR log에서 `OUTBOX_DEAD_LETTER` 라인 확인: `event_id`, `event_type`, `correlation_id`, `error.reason`(원인
코드 — publisher가 붙인 코드, 없으면 거부 결과 이름, 분류되지 않은 예외면 그 타입 이름) 추출. 같은 값이 행의
`last_failure_code`에 남는다. ERROR가 없고 WARN `outbox claim moved a row ... to DEAD`만 있으면 relay가 아니라
claim이 행을 DEAD로 옮긴 것이다 — `error.reason`이 `ATTEMPT_COUNTERS_INCONSISTENT`면 `deferred_attempt_count`를
되돌리지 않은 수동 재발행이 원인이니 §4의 SQL로 다시 재발행한다. `UNREADABLE_ROW`면 행 자체가 event로 읽히지
않는다(canonical 봉투 없음, 저장된 해시와 바이트 불일치)
2. DB에서 DEAD row 확인: `SELECT * FROM outbox_event WHERE status = 'DEAD' ORDER BY occurred_at;`
3. **차단 영향 파악 (중요)**: strict per-aggregate FIFO 정책상 DEAD row는 같은 aggregate(`tenant_scope`, `aggregate_type`, `aggregate_id`)의 후행 이벤트를 계속 차단함. 순서는 `(aggregate_sequence, event_index)`로 비교한다(claim SQL `PostgreSqlOutboxClaimRepository`와 같은 키) —
`SELECT count(*) FROM outbox_event b WHERE b.status <> 'PUBLISHED' AND EXISTS (SELECT 1 FROM outbox_event d WHERE d.status='DEAD' AND d.tenant_scope=b.tenant_scope AND d.aggregate_type=b.aggregate_type AND d.aggregate_id=b.aggregate_id AND (d.aggregate_sequence < b.aggregate_sequence OR (d.aggregate_sequence = b.aggregate_sequence AND d.event_index < b.event_index)));`
@@ -42,14 +47,35 @@ status: stub
## 4. Mitigation (수동 처분 — 둘 중 하나)
- **재발행 (기본)**: 원인 해소 후 해당 row를 다시 claim 가능 상태로 되돌림 —
`UPDATE outbox_event SET status = 'PENDING', attempt_count = 0, next_attempt_at = now() WHERE event_id = '<id>' AND status = 'DEAD';`
장애 구간 전체를 한 번에 되돌리려면(원인이 해소된 것을 확인한 뒤):
`UPDATE outbox_event SET status = 'PENDING', attempt_count = 0, next_attempt_at = now() WHERE status = 'DEAD' AND occurred_at BETWEEN '<장애 시작>' AND '<장애 끝>';` (outbox_event에는 DEAD가 된 시각 컬럼이 없어 이벤트 발생 시각으로 고른다 — 먼저 같은 조건의 SELECT로 대상을 확인한다)
(consumer는 at-least-once + idempotencyKey dedupe 의무가 있으므로 중복 발행은 안전)
- **skip (영구 폐기)**: 이벤트가 더 이상 유효하지 않으면 PUBLISHED로 마킹해 FIFO 차단을 해제 —
`UPDATE outbox_event SET status = 'PUBLISHED' WHERE event_id = '<id>' AND status = 'DEAD';`
⚠️ skip은 다운스트림에 영구 이벤트 갭을 만든다 — 비즈니스 오너 승인 후에만 수행하고 incident 기록에 남김
**재발행 (기본)**: 원인 해소 후 해당 row를 다시 claim 가능 상태로 되돌린다. 시도 횟수 두 개
(`attempt_count`, `deferred_attempt_count`)를 함께 0으로 되돌린다. `attempt_count`만 되돌리면 다음 claim에서
미룬 claim 수가 전체 claim 수보다 많아지고, claim은 그 행을 읽지 않고 다시 DEAD로 옮긴다(재리뷰 2026-11-03
RR-05-01).
<!-- runbook-sql: dead-redrive-one -->
```sql
UPDATE outbox_event SET status = 'PENDING', attempt_count = 0, deferred_attempt_count = 0, next_attempt_at = now() WHERE event_id = '<id>' AND status = 'DEAD';
```
장애 구간 전체를 한 번에 되돌리려면(원인이 해소된 것을 확인한 뒤). outbox_event에는 DEAD가 된 시각
컬럼이 없어 이벤트 발생 시각으로 고른다 — 먼저 같은 조건의 SELECT로 대상을 확인한다.
<!-- runbook-sql: dead-redrive-window -->
```sql
UPDATE outbox_event SET status = 'PENDING', attempt_count = 0, deferred_attempt_count = 0, next_attempt_at = now() WHERE status = 'DEAD' AND occurred_at BETWEEN '<장애 시작>' AND '<장애 끝>';
```
consumer는 at-least-once + idempotencyKey dedupe 의무가 있으므로 중복 발행은 안전하다.
**skip (영구 폐기)**: 이벤트가 더 이상 유효하지 않으면 PUBLISHED로 마킹해 FIFO 차단을 해제한다.
⚠️ skip은 다운스트림에 영구 이벤트 갭을 만든다 — 비즈니스 오너 승인 후에만 수행하고 incident 기록에
남긴다.
<!-- runbook-sql: dead-skip-one -->
```sql
UPDATE outbox_event SET status = 'PUBLISHED' WHERE event_id = '<id>' AND status = 'DEAD';
```
- 장기: poison event 재발 방지(payload 계약 테스트 보강), DEAD 빈발 event_type의 producer 검증 강화
## 5. Escalation
@@ -67,7 +93,10 @@ status: stub
- error-codes.yaml rows: `OUTBOX_DEAD_LETTER` (INTERNAL, retryable=false)
- metrics.yaml: `outbox.publisher.published.total{outcome=DEAD}`, `outbox.pending.size{status=DEAD}`, `outbox.publisher.lag`
- 코드: `application-core` `PublishPendingOutboxEventsUseCase`(FAILED→DEAD 전이), `adapter-persistence` `outbox/OutboxEventJpaRepository`(FIFO 게이트 — DEAD가 후행 차단)
- 코드: `application-core` `PublishPendingOutboxEventsUseCase`(FAILED→DEAD 전이), `adapter:outbound:persistence-jpa` `outbox/postgresql/PostgreSqlOutboxClaimRepository`(FIFO 게이트 — DEAD가 후행 차단), `outbox/jpa/JpaOutboxStore#claimBatch`(읽을 수 없는 행을 claim 때 DEAD로)
- §4의 SQL 블록은 문서에서 그대로 읽어 실제 PostgreSQL에서 실행하는 테스트가 있다
(`PostgreSqlOutboxDispositionRunbookIntegrationTest`, lane `postgresqlOutboxDispositionRunbookIntegrationTest`).
블록을 고치면 그 lane을 다시 돌린다
- 관련 runbook: [[outbox-publish-failed]]
- 관련 branch: [[feature-domain-event-outbox-contract]], [[feature-background-job-async-contract]] (max attempts/DLQ vocabulary SSOT)
@@ -80,8 +109,58 @@ V21(`V21__outbox_aggregate_sequence.sql`)은 `outbox_event`의 순서 키
README 15절 결정). 같은 키의 행들은 claim gate가 순서를 정하지 못한 행이다. 그냥 넘기면 순서 위반이
숨는다. V21은 transaction 하나로 적용되므로, 실패하면 V21 전체가 롤백되고 V20 상태로 남는다.
V21이 하는 일은 세 가지 더 있다.
- **floor 기록.** `outbox_aggregate_sequence_floor`에 V21 적용 시각(epoch 마이크로초)을 한 번 쓴다. 새로
생기는 aggregate counter는 이 값과 남은 행의 최대 번호 중 큰 값 다음에서 시작한다. 그래서 reaper가
행을 지운 aggregate, tenant scope가 바뀐 aggregate도 소비자가 이미 받은 벽시계 번호(약 1.7×10¹⁵)보다
작은 번호를 받지 않는다(리뷰 RR-03-02). 벽시계 번호를 쓴 적 없는 서비스도 첫 번호가 floor + 1이다.
- **counter 미리 만들기.** 남은 행이 있는 aggregate의 counter를 그 최대 번호로 만든다.
- **잠금 대기 5초.** 유일 인덱스는 만드는 동안 `outbox_event`에 SHARE 잠금을 건다. 그동안 outbox를 쓰는
업무 쓰기가 모두 멈춘다. V21은 `lock_timeout = 5s`로 잠금을 기다린다. 열린 outbox 쓰기 트랜잭션 때문에
5초 안에 못 잡으면 V21이 실패하고 V20에 남는다(`MIGRATION_FAILED`). 덜 바쁜 시각에 다시 배포하거나
Step 0으로 인덱스를 먼저 만든다.
rolling 배포 중 아직 옛 버전인 인스턴스가 벽시계 번호로 쓴 행은 floor보다 크다. 새 버전의 counter는
그 aggregate의 가장 큰 행 번호보다 작아지지 않으므로 그 행 뒤에 번호를 준다. 서로 커밋되지 않은 채
겹친 옛·새 쓰기 사이의 순서는 보장하지 않는다 — 옛 벽시계 방식도 보장하지 않던 것이다.
tenant scope를 바꾼 서비스(샘플은 `portfolio` → 단일 scope)는 배포 전에 옛 scope의 미발행 행이
없는지 확인한다. 옛 scope 행과 새 scope 행은 서로 다른 aggregate로 보여 claim gate가 서로를 막지 않고,
partition key도 달라 다른 partition으로 간다.
```sql
SELECT tenant_scope, count(*) AS unpublished
FROM outbox_event
WHERE status <> 'PUBLISHED'
GROUP BY tenant_scope;
```
V20 이하 DB를 V21 이상으로 올리는 배포 전에 아래 순서로 확인한다.
**Step 0 — (outbox가 큰 경우만) 순서 키 인덱스를 먼저 만든다**
V21이 인덱스를 만드는 동안 outbox 쓰기가 멈춘다. reaper가 `outbox_event`를 보존 기간 안으로 줄이므로
보통은 짧다. 그 시간도 멈출 수 없을 만큼 크면, Step 1~3으로 중복을 없앤 뒤 같은 이름·같은 열의 인덱스를
`CONCURRENTLY`로 먼저 만든다. 쓰기를 막지 않는다. transaction 밖(autocommit)에서 실행한다. V21의
`IF NOT EXISTS`가 이 인덱스를 그대로 쓴다.
<!-- runbook-sql: v21-prebuild-order-index -->
```sql
CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS ux_outbox_event_aggregate_order
ON outbox_event (tenant_scope, aggregate_type, aggregate_id, aggregate_sequence, event_index);
```
`CONCURRENTLY` 빌드가 중복이나 중단으로 실패하면 INVALID 인덱스가 남는다. 이 인덱스는 유일성을 지키지
않는다. V21은 이 이름의 인덱스가 유효한 유일 인덱스가 아니면 건너뛰지 않고 실패한다. 그때는
`DROP INDEX CONCURRENTLY ux_outbox_event_aggregate_order;` 뒤 Step 1부터 다시 한다. 빌드가 아직 도는
중에도 INVALID로 보이므로, 지우기 전에 `pg_stat_progress_create_index`에 진행 중인 빌드가 없는지 본다.
V21 안에서 `CONCURRENTLY`를 쓰지 않은 이유: `CONCURRENTLY`는 transaction 안에서 돌 수 없다. V21을
transaction 밖으로 빼면 중복을 만났을 때 "V21 전체 롤백, V20 그대로"가 깨지고, 반쯤 적용된 counter·floor와
INVALID 인덱스가 남는다. 인덱스만 V22로 떼어 내도 실패 시 INVALID 인덱스와 Flyway 실패 이력이 남아 수동
복구가 필요하다. 그래서 기본은 transaction 안 빌드 + 잠금 대기 5초, 큰 outbox는 이 Step 0이다.
**Step 1 — 중복 확인 (읽기 전용)**
<!-- runbook-sql: v21-duplicate-order-keys -->
@@ -149,9 +228,10 @@ Step 1을 다시 돌려 0행이면 배포한다. 정리와 배포 사이에 옛
순서 번호를 만든 샘플 코드가 그 원인이었다). 그래도 V21은 실패하고 롤백될 뿐이니 Step 1부터 다시 한다.
`outbox_event_v21_duplicates`는 확인이 끝나면 지운다.
확인 쿼리와 정리 SQL은 이 문서에서 그대로 읽어 실제 PostgreSQL에서 실행하는 테스트가 있다
(`PostgreSqlOutboxOrderKeyRunbookIntegrationTest`, lane `postgresqlOutboxOrderKeyRunbookIntegrationTest`).
두 SQL 블록을 고치면 그 lane을 다시 돌린다.
확인 쿼리, 정리 SQL, Step 0의 인덱스 SQL은 이 문서에서 그대로 읽어 실제 PostgreSQL에서 실행하는
테스트가 있다(`PostgreSqlOutboxOrderKeyRunbookIntegrationTest`, lane
`postgresqlOutboxOrderKeyRunbookIntegrationTest`). 같은 lane이 V21의 counter·floor, 잠금 대기 실패,
INVALID 인덱스 거부도 본다. 이 절의 `runbook-sql` 블록을 고치면 그 lane을 다시 돌린다.
---
+76 -8
View File
@@ -4,7 +4,7 @@ category: TRANSIENT_DEPENDENCY
error_codes: [OUTBOX_PUBLISH_FAILED]
severity: P2
owner: oncall
last_updated: 2026-06-11
last_updated: 2026-11-03
status: stub
---
@@ -14,17 +14,23 @@ status: stub
이 runbook은 다음 alert에서 발동됩니다.
- alert name: `outbox_publish_failed_rate` 또는 `outbox_publisher_lag`
- alert payload 필수 field: `error.code=OUTBOX_PUBLISH_FAILED`, `event_type`, `correlation_id`, `runbook_link`
- alert name: `outbox_publish_failed_rate`, `outbox_publisher_lag` 또는 `outbox_retry_long`
- alert payload 필수 field: `error.code=OUTBOX_PUBLISH_FAILED`, `error.reason`, `event_type`, `correlation_id`, `runbook_link`
- 임계 (metrics.yaml verbatim):
- P2: `outbox.publisher.published.total{outcome=FAILED}` rate > 1% for 10m
- P2: `outbox.publisher.lag` > 60s for 10m / P1: > 300s for 5m
- P2: `outbox.pending.size{status=PENDING}` growing for 10m
- P2: `outbox.retry.long.size` > 0 for 10m — §4.1 "끝나지 않는 재시도"로 간다
## 2. First Response (5분 이내)
### Step 1 — 확인
1. ERROR log에서 `OUTBOX_PUBLISH_FAILED` 라인 확인: `event_type`, `event_id`, `correlation_id`, `attempt_count` 추출
1. ERROR log에서 `OUTBOX_PUBLISH_FAILED` 라인 확인: `event_type`, `event_id`, `correlation_id`, `attempt_count`,
`error.reason` 추출. `error.reason`이 원인 코드다 — publisher가 붙인 코드(예: `KAFKA_AUTHENTICATIONEXCEPTION`,
`KAFKA_TRANSPORT_CLOSED`, `PUBLISH_RUNTIME_UNAVAILABLE`, `PUBLISH_DEADLINE_EXCEEDED`), 코드가 없으면 결과
이름(`AMBIGUOUS`, `REJECTED_RETRYABLE`), 분류되지 않고 던져진 실패면 예외 타입 이름이다. 같은 값이 행의
`last_failure_code`에도 남는다. `error.cause_type`은 어느 결과였는지만 말한다(`OutboxPublishRefusedException`이면
거부, `OutboxPublishAmbiguousException`이면 응답 없음)
2. broker 상태 확인: `APP_MESSAGING_BROKER` 값과 broker endpoint 가용성. 이 키는 활성화 스위치가
아니라 **선택자**다 — messaging을 끄는 것은 `APP_MESSAGING_ENABLED=false`이고, 이 값을 비운다고
messaging이 꺼지지는 않는다.
@@ -40,8 +46,12 @@ status: stub
3. `outbox.pending.size` status 분포 확인 (FAILED 누적 vs PENDING 누적)
### Step 2 — 임시 격리
- 일시 실패는 자동 backoff 재시도(30s × 2^(attempt-1) + jitter, 상한 5분)가 동작 — 즉시 수동 개입 불필요
- 일시 실패는 자동 backoff 재시도(30s × 2^(attempt-1) + jitter, 상한 5분)가 동작 — broker 장애처럼 원인이 저절로
풀리는 경우는 수동 개입이 필요 없다
- broker 정지·종료 중 거부·Kafka 인증/인가 실패(`AMBIGUOUS`, `REJECTED_RETRYABLE`)는 시도 예산을 쓰지 않는다 — 장애가 길어도 DEAD가 되지 않고, broker가 돌아오면 늦어도 약 5분 30초 안에 자동 재발행된다
- **그 대신 끝이 없다.** 원인이 저절로 풀리지 않으면(진짜로 틀린 credential·ACL, Kafka adapter가 이름을 모르는
broker 거부 — topic timestamp 정책, 없는 topic의 metadata 대기 초과는 기본값으로 `AMBIGUOUS`가 된다) 그 행은 5분
간격으로 영원히 FAILED가 되고 그 aggregate의 뒤 이벤트를 막는다. `outbox.retry.long.size` > 0이면 §4.1로 간다
- max attempts 3은 adapter가 분류하지 못하고 던진 발행 예외에만 적용된다 (소진 시 `runbook://outbox/dead-letter`로 이관)
- relay 자체를 멈춰야 하면 `app.outbox.relay-enabled=false`로 스케줄러 비활성 (이벤트는 outbox 테이블에 안전하게 보존됨 — 유실 없음)
@@ -57,7 +67,10 @@ status: stub
- broker outage/네트워크 → broker 측 회복 대기
- Kafka adapter 미구성(enabled인데 brokers 누락은 기동 시 차단됨) / disabled 상태에서 producer 활성화
- poison event (직렬화 불가/payload 계약 위반) → 확정 거부(`REJECTED_*`)면 바로, 분류되지 않은 예외면 attempts 소진 후 DEAD로 흘러감 (의도된 동작)
- 같은 행의 `OUTBOX_PUBLISH_FAILED`가 5분 간격으로 끝없이 반복되고 원인이 Kafka 인증/인가면 credential·ACL이 실제로 틀린 것이다 — 고치면 다음 재시도에서 저절로 빠진다
- 같은 행의 `OUTBOX_PUBLISH_FAILED`가 5분 간격으로 끝없이 반복되면 §4.1의 조회로 그 행의 `last_failure_code`를
본다. `KAFKA_AUTHENTICATIONEXCEPTION`·`KAFKA_*AUTHORIZATIONEXCEPTION`이면 credential·ACL이 실제로 틀린
것이다 — 고치면 다음 재시도에서 저절로 빠진다. 그 밖의 코드가 같은 행에서만 반복되면 broker가 그 레코드를
받지 않는 것이다 — §4.1로 보류한다
- 동일 aggregate head 실패로 후행 이벤트가 FIFO 게이트에 차단되어 lag 증가 (strict per-aggregate FIFO — 설계 의도)
## 4. Mitigation
@@ -66,6 +79,61 @@ status: stub
- IN_FLIGHT orphan(claim 후 crash)은 in-flight-timeout(기본 PT5M) 경과 후 자동 재claim — at-least-once이므로 중복 발행 가능, consumer dedupe(idempotencyKey)가 흡수
- 장기: `app.outbox.poll-interval`/`batch-size` 조정, broker 가용성 SLA 점검, 빈발 event_type의 payload 계약 검토
### 4.1 끝나지 않는 재시도 — 찾기, 보류(park), 재개(resume), 버리기(skip)
시도 예산을 쓰지 않는 실패(§2 Step 2)는 끝 상태가 없다. 리뷰 2026-11-01 README 15절 결정(재시도 유지)은 그대로 두고,
운영자가 그런 행을 보고 멈출 수 있게 한다(재리뷰 2026-11-03 RR-05-03 결정).
**Step 1 — 찾기 (읽기 전용).** 미룬 claim(`deferred_attempt_count`)이 10번 이상인 FAILED 행이다. backoff 상한이
5분이라 10번은 적어도 약 35분 동안 계속 실패했다는 뜻이다. `outbox.retry.long.size`·`outbox.retry.long.age`가
같은 기준(`OutboxMetrics.LONG_RETRY_DEFERRED_CLAIMS`)으로 센다.
<!-- runbook-sql: long-retries -->
```sql
SELECT event_id, event_type, aggregate_type, aggregate_id, attempt_count, deferred_attempt_count,
last_failure_code, occurred_at, next_attempt_at
FROM outbox_event
WHERE status = 'FAILED'
AND deferred_attempt_count >= 10
ORDER BY occurred_at, event_id;
```
`last_failure_code`로 원인을 가른다(§3). 원인이 고쳐지는 것(credential·ACL·topic 설정)이면 고치고 기다린다 —
다음 재시도에서 저절로 빠진다. 고칠 때까지 재시도 소음과 실패 로그를 멈추고 싶거나, 그 레코드만 broker가
받지 않는 것이면 Step 2로 보류한다.
**Step 2 — 보류(park).** FAILED 행 하나를 `PARKED`로 둔다. relay는 PARKED 행을 claim하지 않는다. 그 행은
여전히 미발행이라 같은 aggregate의 뒤 이벤트를 계속 막는다 — 순서는 지킨다. `outbox.publisher.lag`도 계속
오른다. 보류는 원인을 찾는 동안 재시도만 멈추는 것이지 incident를 닫는 것이 아니다.
<!-- runbook-sql: park-one -->
```sql
UPDATE outbox_event SET status = 'PARKED' WHERE event_id = '<id>' AND status = 'FAILED';
```
`UPDATE 0`이면 그 사이 relay가 행을 claim해 IN_FLIGHT다. `next_attempt_at`이 지난 뒤 다시 실행한다.
보류된 행 수는 `outbox.pending.size{status=PARKED}`로 보인다.
**Step 3 — 재개(resume) 또는 버리기(skip).** 원인을 고쳤으면 재개한다. 행은 FAILED로 돌아가 바로 claim
대상이 된다. 시도 횟수는 그대로 둔다(미룬 claim은 예산을 쓰지 않으므로 DEAD로 바로 가지 않는다).
<!-- runbook-sql: resume-one -->
```sql
UPDATE outbox_event SET status = 'FAILED', next_attempt_at = now() WHERE event_id = '<id>' AND status = 'PARKED';
```
그 이벤트를 영원히 보내지 않기로 했으면 버린다. DEAD skip과 같은 승인 조건이다 — 다운스트림에 영구 이벤트
갭이 생기므로 비즈니스 오너 승인 후에만 하고 incident 기록에 남긴다.
<!-- runbook-sql: parked-skip-one -->
```sql
UPDATE outbox_event SET status = 'PUBLISHED' WHERE event_id = '<id>' AND status = 'PARKED';
```
이 절의 SQL 블록은 문서에서 그대로 읽어 실제 PostgreSQL에서 실행하는 테스트가 있다
(`PostgreSqlOutboxDispositionRunbookIntegrationTest`, lane `postgresqlOutboxDispositionRunbookIntegrationTest`).
블록을 고치면 그 lane을 다시 돌린다.
## 5. Escalation
- P1 lag(>300s 5m) 지속 + broker 회복 불가면 broker/infra 팀에 page
@@ -79,8 +147,8 @@ status: stub
## 7. Related
- error-codes.yaml rows: `OUTBOX_PUBLISH_FAILED` (TRANSIENT_DEPENDENCY, retryable=true, retry_after 30s)
- metrics.yaml: `outbox.publisher.published.total`, `outbox.publisher.lag`, `outbox.pending.size`
- 코드: `application-core` `PublishPendingOutboxEventsUseCase`(상태머신), `adapter-persistence` `outbox/OutboxEventJpaRepository`(SKIP LOCKED claim + FIFO 게이트), `adapter-outbound` `messaging/outbox/KafkaOutboxMessagePublishAdapter`(fail-closed)
- metrics.yaml: `outbox.publisher.published.total`, `outbox.publisher.lag`, `outbox.pending.size`, `outbox.retry.long.size`, `outbox.retry.long.age`
- 코드: `application-core` `PublishPendingOutboxEventsUseCase`(상태머신), `adapter:outbound:persistence-jpa` `outbox/postgresql/PostgreSqlOutboxClaimRepository`(SKIP LOCKED claim + FIFO 게이트), `adapter:outbound:messaging` `outbox/OutboxMessagePublishAdapter`·`platformbridge/PlatformIntegrationEventPublishAdapter`(fail-closed, 원인 코드 번역)
- 관련 runbook: [[outbox-dead-letter]]
- 관련 branch: [[feature-domain-event-outbox-contract]], [[feature-background-job-async-contract]] (retry/DLQ vocabulary SSOT)
+15 -17
View File
@@ -34,20 +34,19 @@ Redis liveness 실패만으로 pod를 재시작하지 않는다. 재시작 폭
- `evictionValidation=CONFIGURED_EXPECTATION_ONLY`와
`externalEvictionAttestation=INCOMPLETE`는 effective server policy가 증명되지 않았다는
뜻이다. 이를 정상 attestation으로 해석하지 않는다.
- `redis.capability.operations.total`과 `redis.capability.duration.seconds`에서 affected
capability/role/operation의 실제 반환 outcome을 확인한다. mutation의
`certainty=indeterminate`는 timeout이나 연결 끊김을 미실행 증거로 바꾸지 않는다.
- `redis.capability.admission.rejected.total`에서 `rejected_saturated`와
`rejected_closed`를 구분하고, `redis.capability.inflight.total`의 같은 role에 대해 현재 0이
아닌 state를 확인한다. in-flight gauge는 bounded command count이며 byte 수나 queue depth가
아니다.
- `redis.capability.readiness.total`은 현재 상태 gauge가 아니라 exact sanitized
`RoleHealth` 관측 횟수다. 최신 health detail의 state/reason/requirement와 함께 해석한다.
optional cache의 degraded serving과 required coordination/session의 fail-closed 결정을
같은 availability 의미로 합치지 않는다.
- 종료 시 `redis.capability.lifecycle.drain.total`에서 `drained`,
`forced_after_timeout`, `interrupted`를 구분한다. repeated close는 새 drain을 시작하거나
중복 관측을 만들지 않는다.
- **지금 내보내는 Redis 지표는 connection lane 단위 `redis.lane.*`다**(`docs/registries/metrics.yaml`,
`lane` 태그). capability·operation 단위 지표(`redis.capability.*`)는 설계만 있고 아직 어떤 코드도
내보내지 않는다 — 대시보드에서 찾지 않는다(재리뷰 2026-11-03, `src/adapter/outbound/cache-redis/README.md`
「아직 내보내지 않는 지표」).
- lane 포화: `redis.lane.acquire.refused.total`(거절)과 `redis.lane.acquire.wait.seconds`(대기)를 lane별로
본다. `CACHE_SCRIPT`만 차 있으면 캐시 조건부 쓰기의 문제이고, 멱등성·lease가 쓰는 `SCRIPT` lane은
따로 본다(재리뷰 RR-04-02).
- 연결 불안정: `redis.lane.connections.opened.total`·`redis.lane.connections.discarded.total`의 급증,
`redis.lane.connections.active`·`pending`.
- operation 결과(hit/miss/denied/conflict, `certainty=indeterminate`)는 지표가 아니라 각 capability의
로그와 health detail에서 확인한다. timeout이나 연결 끊김은 미실행 증거가 아니다.
- readiness는 최신 health detail의 state/reason/requirement로 본다. optional cache의 degraded serving과
required coordination/session의 fail-closed 결정을 같은 availability 의미로 합치지 않는다.
- reconnect, cache source-load, session repository error 지표의 변화를 함께 본다.
- Redis server 측에서는 memory/eviction, rejected clients, replication link/lag,
persistence error, Cluster coverage를 operator dashboard에서 확인한다.
@@ -56,9 +55,8 @@ Redis liveness 실패만으로 pod를 재시작하지 않는다. 재시작 폭
### Observability and lifecycle boundaries
- 여섯 `redis.capability.*` meter의 tag는 닫힌 enum에서만 생성된다. key, subject, session id,
token, endpoint, exception text, script/SHA, value 같은 identity/wire material을 metric이나
ticket에 복사하지 않는다.
- `redis.lane.*` meter의 tag는 닫힌 lane 이름뿐이다. key, subject, session id, token, endpoint,
exception text, script/SHA, value 같은 identity/wire material을 metric이나 ticket에 복사하지 않는다.
- semantic operation 계측은 logical provider가 실제로 반환한 hit/miss/denied/conflict/
unavailable/indeterminate 결과를 기록한다. cache의 `stale`/`skipped`, session의
`tombstoned`/`absolute_expired`도 정상 hit/miss와 분리한다. meter registry, classifier,
+41 -4
View File
@@ -189,8 +189,8 @@ adapter leaf의 실 DB·Redis lane에서 돈다. fake가 실제와 다르게 동
`public interface *Port`)와, `application-notification`이 소유하고 `persistence-notification`이 구현하는
저장소 포트 전부다(review 04-14, 표에 `(notification)`으로 표시). notification 포트의 계약은
`application-notification` testFixtures에 있고, 실제 구현 쪽은 persistence-jpa의 `postgresqlIntegrationTest`
source set에서 태그 `jpa-contract`로 `jpaPlatformContractTest` lane이 돌린다. fileserver 포트는
`application-fileserver` testFixtures의 `*PortContract`가 맡고 `persistence-fileserver/CLAUDE.md`에 적혀 있다.
source set에서 태그 `jpa-contract`로 `jpaPlatformContractTest` lane이 돌린다. fileserver 저장 포트는
9.3에 있다.
### 9.1 계약이 있는 포트
@@ -248,8 +248,15 @@ source set에서 태그 `jpa-contract`로 `jpaPlatformContractTest` lane이 돌
고쳤다. in-memory store는 `DirectTransactionPort`를 받으면 같은 규칙(트랜잭션 밖·읽기 전용 거부,
롤백 시 취소)으로 동작한다. 인자 없이 만들면 지금처럼 바로 쓴다(relay 테스트가 행을 미리 심는 용도).
- `IdempotencyStoreContract`의 시간 케이스(renew가 lease를 실제로 늘린다, review 05-02)는 시계를
가진 store(실 Redis, PostgreSQL)에서만 의미가 있다. in-memory fake는 만료가 없어 `elapse`가
no-op이다. PostgreSQL에서 renew가 lease를 늘리지 않게 바꾼 변이로 이 케이스가 실패함을 확인했다.
가진 store(실 Redis, PostgreSQL)에서만 의미가 있다. 시계를 움직일 수 없는 store(in-memory fake,
command fake 위 Redis)는 `elapse`를 재정의하지 않고, 기본 `elapse`가 그 케이스를 abort해 보고서에
skipped로 남는다. 예전에는 no-op이라 시간이 흐르지 않은 채 통과했다(review 2026-11-03 RR-01-06).
PostgreSQL에서 renew가 lease를 늘리지 않게 바꾼 변이로 이 케이스가 실패함을 확인했다.
- `OutboxAggregateSequenceContract`는 포트 Javadoc의 세 약속 중 둘을 보지 않았다(RR-01-06). "쓰기
트랜잭션 밖 거부"와 "롤백된 번호는 다시 준다"를 넣었다. fake는 `DirectTransactionPort`를 받으면 같은
규칙으로 동작한다(인자 없이 만들면 예전처럼 트랜잭션을 보지 않는다 — sample 테스트가 쓰는 방식).
`CacheRegionContract`에는 2^53 위 revision 한 자리 차이 경우를 넣었다. fake가 revision을 double로
비교하게 바꾼 변이가 이제 실패한다(전에는 통과).
notification 계약 다섯 개(deduplication·request·reconciliation job·admin operation·template, 2026-11-03)는
PostgreSQL 쪽에만 있던 테스트를 계약으로 옮긴 것이다. `ReconciliationJobClaimContractTest`(8건)와
@@ -301,6 +308,36 @@ r3-J: queue가 fence를 보지 않음(같은 worker 이름 재claim 1건), queue
`SuppressionSideEffect`가 complaint를 INVALID로 기록함(1건). relay
use case가 결과 조회 대신 예외만 보는 `publish`를 부르게 바꾸면 relay 테스트 7건이 실패한다.
### 9.3 application-fileserver 저장 포트 (review 2026-11-03 RR-01-04·RR-07-06)
`persistence-fileserver`가 구현하는 포트 9개와 소유자 규칙 포트. 계약은 `application-fileserver`
testFixtures(`dev.caskeleton.application.fileserver.testkit`)에 있고, 실제 구현 쪽은 persistence-jpa의
`postgresqlIntegrationTest` source set(`readiness` 패키지, 공용 기동 `FileserverPostgreSqlStack`)에서 태그
`jpa-contract`로 `jpaPlatformContractTest` lane이 돌린다. 시간이 걸린 경우는 stack이 움직이는 시계로 본다.
| 포트 | 계약 | fake | 상속하는 실제 구현 테스트 |
| --- | --- | --- | --- |
| `FileMetadataPort` | `FileMetadataPortContract` | `InMemoryFileMetadataStore` | `PostgreSqlFileserverMetadataStoreIntegrationTest` (`postgresqlFileserverMetadataIntegrationTest`) |
| `CleanupQueuePort` | `CleanupQueuePortContract` | `InMemoryCleanupQueue` | `PostgreSqlFileserverReclamationIntegrationTest` (`postgresqlFileserverReclamationIntegrationTest`) |
| `FileQuotaPort`·`QuotaCommitPort`·`QuotaReclaimPort` | `FileQuotaLedgerContract` — 세 포트가 한 장부라 한 계약 | `FakeFileQuotaService`(장부) + `FakeQuotaCommitGateway`·`RecordingQuotaReclaimGateway`(같은 장부에 쓴다) | `PostgreSqlFileQuotaLedgerContractTest` |
| `UploadSessionPort` | `UploadSessionPortContract` | `InMemoryUploadSessionStore` | `PostgreSqlUploadSessionContractTest` |
| `StagingUploadLocatorPort` | `StagingUploadLocatorPortContract` | `InMemoryUploadSessionStore`(같은 세션) | `PostgreSqlStagingUploadLocatorContractTest` |
| `RecoveryQueuePort` | `RecoveryQueuePortContract` | `InMemoryRecoveryQueue` | `PostgreSqlRecoveryQueueContractTest` |
| `ContentReferenceLedgerPort` | `ContentReferenceLedgerPortContract` | `InMemoryFileMetadataStore`(같은 레코드) | `PostgreSqlContentReferenceLedgerContractTest` |
| `FileAccessPolicyPort` | `FileAccessPolicyPortContract` — 규칙 정본은 `FileAccessSubject.owns` | `FakeFileAccessPolicy` | `RoleBasedFileAccessPolicyContractTest` — fileserver adapter (`test`) |
계약을 처음 적용할 때 fake가 실제와 다르게 답한 경우와 처리(모두 fake를 실제에 맞췄다):
- 쿼터: fake에 시계가 없어 만료된 예약도 사용량에 셌고, 해제한 예약의 확정·연장을 조용히 받았다. 확정·
회수 fake는 기록만 하고 장부를 바꾸지 않았다. 이제 한 장부에 시계가 있고 JPA와 같은 조건으로 거절한다.
만료됐지만 해제되지 않은 예약의 확정은 JPA SQL이 받으므로(쓴 바이트는 사용량이다) 계약도 받는다 —
`JpaFileQuota` Javadoc의 "만료된 예약은 확정할 수 없다"는 SQL과 달라 고쳤다.
- 업로드 세션: fake가 기한이 지난 업로드에 lease를 줬고, lease가 만료된 writer의 offset 확정을 받았고,
`findExpired`를 기한 순이 아니라 넣은 순서로 줬다.
- 복구 큐: fake가 한도 0을 거절하지 않았다.
- 소유자 규칙: fake와 실제 정책이 같은 규칙을 따로 썼다(principal·tenant 비교 두 벌). 정본
`FileAccessSubject.owns` 하나를 둘 다 부른다. fake 규칙에서 tenant 비교를 빼는 변이가 계약 2건으로 잡힌다.
### 9.2 계약이 없는 포트 (의도)
| 포트 | 실제 구현 | fake | 계약이 없는 이유 | 다음 단계 |
+30 -9
View File
@@ -7,8 +7,11 @@
코드가 이 문서를 지키는지는 다음 테스트가 확인한다.
- 응답 원문의 키 집합: `src/adapter/inbound/web/src/test/java/dev/caskeleton/adapter/inbound/web/envelope/EnvelopeWireShapeTest.java`
- 핸들러 밖 오류(`/error` 디스패치)의 실패 Envelope: 실제 컨테이너에서 `WebContractAssertions`
(`TomcatWebContractIT`·Jetty lane), 기본 JWT 체인에서 `JwtChainErrorDispatchIT`
- 멱등성 응답의 status·code·retryable·`Retry-After`: `IdempotencyExceptionMappingTest`,
`RetryAfterAdvisorTest`, `OperationalErrorTest`, 키 형식은 `IdempotencyKeyHeaderTest`
`RetryAfterAdvisorTest`, `OperationalErrorTest`, 키 형식은 `IdempotencyKeyHeaderTest`,
실행 전 거부와 결과 불명의 구분은 `IdempotencyExecutorTest`, lease 하한은 `IdempotencySettingsTest`
- 헤더 규칙(3절): `WebMvcRequestIdFilterTest`, `CorsSecurityFilterIntegrationTest`
- web 전용 코드(4절): `TransportErrorHandlingTest`, `IdempotencyKeyHeaderTest`, web-advanced
`WebAdvancedErrorMappingTest`
@@ -51,8 +54,14 @@ JSON 응답은 모두 `Envelope` 하나의 모양이다(소유자 결정 2). **
| `correlationId` | 위와 같음. 값이 없으면 키가 없다 |
| `page` | 페이지 목록 응답에만. `number`·`size`·`total`·`sort`(미정렬이면 `sort: null`) |
필터 앞에서 끊긴 응답(필터 체인 밖의 오류)은 id 키가 빠질 수 있다. 클라이언트는 `meta`의 모르는
키를 무시한다.
컨트롤러에 닿기 전에 거부된 요청(보안 방화벽이 막은 경로, 필터의 `sendError`, 필터가 던진 예외)도 같은
실패 Envelope다. 컨테이너가 `/error`로 넘긴 요청을 `EnvelopeErrorController`가 status에 맞는 코드로 답한다.
본문에 요청 경로나 시각을 싣지 않는다. 원래 status가 그대로 나가고, 기본 JWT 체인에서도 익명 400이 401로
바뀌지 않는다(재리뷰 RR-02-02). 다만 요청 id 필터를 지나기 전에 끊긴 응답은 `meta`의 id 키가 빠질 수 있다.
클라이언트는 `meta`의 모르는 키를 무시한다.
4xx·5xx status로 나가는 본문은 `success:true`로 감싸지 않는다. 실패 Envelope가 아닌 본문을 오류 status로
돌려주는 핸들러는 그 본문이 그대로 나간다.
### 왜 이렇게 나가는가
@@ -79,21 +88,31 @@ JSON 응답은 모두 `Envelope` 하나의 모양이다(소유자 결정 2). **
첫 요청이 끝나면 같은 작업이 두 번 적용된다.
- **`Retry-After: 2`의 근거.** 기본 processing lease(`app.capabilities.idempotency.processing-lease`,
30s)보다 짧아, 첫 시도가 아직 끝날 수 있는 동안 다시 묻는다. 너무 이르면 409를 한 번 더 받을 뿐이고
결과가 바뀌지는 않는다. lease를 2초보다 짧게 설정한 서비스는 이 값이 lease를 넘으므로 값을 다시
정한다(`RetryAfterAdvisor`, `docs/registries/error-codes.yaml`).
결과가 바뀌지는 않는다. 값은 `IdempotencyExecutor.IN_FLIGHT_RETRY_AFTER` 하나이고 `RetryAfterAdvisor`가
그 값을 쓴다. `processing-lease`를 이 값(2초) 이하로 설정하면 기동이 실패한다(`IdempotencySettings`,
재리뷰 RR-02-05).
- executor는 기다리지 않는다. 처리 중이면 바로 409를 돌려준다. 키로 결과를 조회하는 엔드포인트는 없다.
- **키 문법**: 보이는 ASCII(`0x21`–`0x7E`) 1~256자. 서버는 trim하거나 고치지 않는다. 짝 프론트의
`isValidIdempotencyKey`와 같은 규칙이다. KEYED 엔드포인트는 모두 `IdempotencyKeyHeader.parse`로 헤더를
읽어 같은 코드로 거부한다(리뷰 02-12). 키 원문은 응답에 돌려 쓰지 않는다. 키를 요구하지 않는
엔드포인트는 키 없이 보호 없이 실행된다.
같은 executor가 내는 나머지 두 실패:
같은 executor가 내는 나머지 세 실패. 저장소가 응답하지 않은 경우는 **언제** 멈췄는지로 코드가 갈린다.
| 경우 | status | `error.code` | `retryable` | `Retry-After` | 클라이언트 행동 |
| --- | --- | --- | --- | --- | --- |
| 멱등성 저장소 응답 없음 | 503 | `IDEMPOTENCY_UNAVAILABLE` | `true` | 없음 | 작업 실행 전에 거부됐다. 같은 키로 나중에 다시 보낸다 |
| 작업 실행 전(claim·시작 단계) 멱등성 저장소 응답 없음 | 503 | `IDEMPOTENCY_UNAVAILABLE` | `true` | 없음 | 작업은 실행되지 않았다. 같은 키로 나중에 다시 보낸다 |
| 작업 실행 뒤 결과 기록(complete 단계) 중 저장소 응답 없음 — **결과 불명** | 503 | `IDEMPOTENCY_COMPLETION_UNKNOWN` | `true` | 없음 | "적용되지 않았다"고 알리지 않는다. **같은 키로** 나중에 다시 보낸다. 새 키 금지 |
| 이전 시도의 효과를 증명할 수 없음 | 500 | `IDEMPOTENCY_RECOVERY_REQUIRED` | `false` | 없음 | 재전송도 재실행도 안전하지 않다. 업무 데이터로 결과를 확인한다 |
- **결과 불명이 무엇을 뜻하나.** 작업은 이미 돌았다. 그 작업의 트랜잭션 안 DB 변경은 예외와 함께
롤백되지만, 트랜잭션 밖 효과(다른 시스템 호출, 메시지)는 일어났을 수 있다. 그래서 실행 전 거부와 같은
503이어도 code가 다르다. 클라이언트는 status가 아니라 `error.code`로 가른다.
- **같은 키로 다시 보내면 서버가 판정한다.** 결과가 실제로는 기록됐으면 재생(첫 요청의 성공 응답), 첫 시도의
processing lease가 아직 남았으면 409 `IDEMPOTENT_IN_FLIGHT`, lease가 지났으면 500
`IDEMPOTENCY_RECOVERY_REQUIRED`다. 새 키로 보내면 서버는 다른 요청으로 보고 막지 못한다
(재리뷰 2026-11-03 README 4절 결정, RR-02-03).
## 3. 헤더
### 요청 id·correlation id
@@ -127,7 +146,7 @@ CORS가 켜져 있으면(`app.cors.enabled=true`) cross-origin 실제 응답의
| 코드 | status | category | `retryable` | `Retry-After` | 언제 |
| --- | --- | --- | --- | --- | --- |
| `REQUEST_TIMEOUT` | 503 | `TRANSIENT_DEPENDENCY` | `true` | 없음 | 비동기 요청(`Callable`·`DeferredResult`·스트리밍)이 시간 안에 끝나지 않았다 |
| `REQUEST_TIMEOUT` | 503 | `TRANSIENT_DEPENDENCY` | `true` | 없음 | 비동기 요청(`Callable`·`DeferredResult`·스트리밍)이 시간 안에 끝나지 않았다. GraphQL도 같은 이름을 쓰지만 그쪽 `category`는 GraphQL 전용 어휘 `TIMEOUT`이다(`src/adapter/inbound/graphql/README.md`) |
| `IDEMPOTENCY_KEY_INVALID` | 400 | `VALIDATION` | `false` | 없음 | 2절 |
| `RATE_LIMIT_UNAVAILABLE` | 503 | `TRANSIENT_DEPENDENCY` | `true` | 있음 | rate-limit 저장소 응답 없음(`docs/web/runbook.md`) |
| `RATE_LIMIT_INCOMPATIBLE` | 503 | `INTERNAL` | `false` | 없음 | rate-limit 제공자가 계약과 맞지 않음 |
@@ -143,6 +162,8 @@ CORS가 켜져 있으면(`app.cors.enabled=true`) cross-origin 실제 응답의
- 응답 키를 더하거나 빼면 `EnvelopeWireShapeTest`와 이 문서 1절을 같이 고친다.
- 응답 헤더를 새로 쓰면 `CorsExposedHeaders`에 넣고 3절을 고친다.
- 멱등성 코드의 status·retryable·`Retry-After`를 바꾸면 `OperationalError`, `RetryAfterAdvisor`,
`docs/registries/error-codes.yaml`, 이 문서 2절을 같은 변경에서 고친다.
`docs/registries/error-codes.yaml`, 이 문서 2절을 같은 변경에서 고친다. 공유 코드의 `retryable`은
gRPC trailer(`error-retryable`)와 GraphQL extension(`retryable`)에도 그대로 나가므로 그 둘의 README와
매핑 테스트(`GrpcStatusMapperTest`, `GraphQlWireErrorMapperTest`)도 같이 본다.
- 짝 프론트의 백엔드 계약 문서(`clean-architecture-frontend-template`
`docs/architecture/backend-api-and-server-state-contract.md` §4.1·§6)는 이 문서를 가리켜야 한다.
+1 -1
View File
@@ -38,7 +38,7 @@ qualifies nothing, so "no CA configured" is an error rather than a client with v
## The ACL fixture
`acl/all-accounts.acl` provisions the accounts every lane uses. Two things about it matter, and
`acl/lane-fixture-accounts.acl` provisions the accounts every lane uses. Two things about it matter, and
neither can be written in the file itself — **Redis refuses to start if an `aclfile` contains a
comment line**, so the whole file is directives and the explanation lives here.
+20 -10
View File
@@ -1,4 +1,16 @@
# ACL accounts
# ACL accounts — test fixtures, not a deployment file
**Nothing in this directory is loaded by a deployment.** Every file here is a fixture for the local
and test lanes: its passwords are public (`fixture-*`, written in this repository), and
`lane-fixture-accounts.acl` also provisions three accounts that only a lane topology needs — among
them `ca-skeleton-cluster-bootstrap`, which has `~* &* +@all`, every command on every key. Loaded into
a real Redis, that line alone gives anyone who can reach the server and read this repository full
control of it (`CONFIG`, `MODULE`, every key). A deployment writes its own ACL file of the same
shape: the application accounts it actually uses, passwords from its secret store, and none of the
lane-only accounts (review 2026-11-03 RR-07-04). The application enforces its half of that: under
the `prod` runtime profile it refuses to start while any setting or environment variable holds a
value beginning with `fixture-` (`SecretSourceValidator`), so a deployment that copied a lane's
`REDIS_LANE_*` values stops before it connects.
One file per `CommandAccess` level. Each account is deliberately narrower than the SDK's own rules:
the account is the last enforcement boundary and a permit issued inside the process never widens it,
@@ -7,23 +19,21 @@ so a mistake in the SDK is still refused by the server.
A Redis ACL file accepts nothing but complete `user` lines — no comments and no line continuations —
which is why the rationale lives here instead of inline.
Every account here carries a **fixture** password (`fixture-*`). None of them is a deployment
credential and none is read by anything but the local lanes; a deployment replaces the whole file
with one whose passwords come from its secret store. They are written down rather than left as
`nopass` because an account that accepts any password makes every assertion about authentication
pass for the same reason a typo would have — and because these files are what a fork copies. The
four per-role files carried `nopass` long after `all-accounts.acl`, the file a deployment actually
loads, had been fixed; a fork that copied one of them got an open account and a green lane.
Every account here carries a **fixture** password (`fixture-*`). They are written down rather than
left as `nopass` because an account that accepts any password makes every assertion about
authentication pass for the same reason a typo would have — and because these files are what a fork
copies. The four per-role files carried `nopass` long after the combined lane file had been fixed;
a fork that copied one of them got an open account and a green lane.
| File | Account | Grants |
| --- | --- | --- |
| `application.acl` | `CommandAccess.APPLICATION` | the typed data-structure commands over `prod:*`, with the dangerous and deprecated names denied |
| `application-advanced.acl` | `APPLICATION_ADVANCED` | the above plus pub/sub, transactions, and the registered-script path. `EVAL` is absent: only `EVALSHA` of an already loaded digest is reachable. `+time` is granted because the rate-limit and idempotency scripts read the server clock with `redis.call('TIME')`, and `TIME` is in no data category (`@read`/`@write`): without it every such script fails with an ACL error. The startup SCRIPT probe runs `TIME` so a template missing it fails at boot |
| `raw-gateway.acl` | `RAW_GATEWAY` | exactly the commands the catalog classifies `RAW_ONLY`, and nothing else |
| `all-accounts.acl` | – | the four accounts concatenated; Redis takes one `aclfile`, so this is what a deployment loads |
| `lane-fixture-accounts.acl` | – | what the lanes load (Redis takes one `aclfile`): `user default off`, the four per-role accounts of this table, and three lane-only accounts — `ca-skeleton-replication` (replica sync) and `ca-skeleton-sentinel` for the replicated and sentinel topologies, `ca-skeleton-cluster-bootstrap` (`+@all`) to create the cluster lane's slots. Never a deployment file |
| `admin-readonly.acl` | `ADMIN_READONLY` | read-only diagnostics. Every destructive counterpart is denied here and blocked in the command policy catalog — two independent controls for the same rule |
## Who loads `all-accounts.acl`
## Who loads `lane-fixture-accounts.acl`
The Redis topology lanes (`../standalone`, `../sentinel`, `../cluster`, `../tls`) and the
application's Compose lanes that switch Redis on (`local-redis`, `all-adapters` in
+1 -1
View File
@@ -42,7 +42,7 @@ x-node: &node
--min-replicas-max-lag 1 \
--masteruser ca-skeleton-replication \
--masterauth fixture-replication \
--aclfile /etc/redis/acl/all-accounts.acl
--aclfile /etc/redis/acl/lane-fixture-accounts.acl
healthcheck:
test: ["CMD-SHELL", "[ \"$$(redis-cli -p $$NODE_PORT --user ca-skeleton-application --pass fixture-application --no-auth-warning ping)\" = PONG ]"]
interval: 2s
+1 -1
View File
@@ -51,7 +51,7 @@ x-data-node: &data-node
--min-replicas-max-lag 1 \
--masteruser ca-skeleton-replication \
--masterauth fixture-replication \
--aclfile /etc/redis/acl/all-accounts.acl
--aclfile /etc/redis/acl/lane-fixture-accounts.acl
healthcheck:
test: ["CMD-SHELL", "[ \"$$(redis-cli -p $$NODE_PORT --user ca-skeleton-application --pass fixture-application --no-auth-warning ping)\" = PONG ]"]
interval: 2s
+1 -1
View File
@@ -14,7 +14,7 @@ services:
- --save
- ""
- --aclfile
- /etc/redis/acl/all-accounts.acl
- /etc/redis/acl/lane-fixture-accounts.acl
volumes:
- ../acl:/etc/redis/acl:ro
ports:
+1 -1
View File
@@ -61,7 +61,7 @@ services:
- --save
- ""
- --aclfile
- /etc/redis/acl/all-accounts.acl
- /etc/redis/acl/lane-fixture-accounts.acl
ports:
- "${REDIS_PORT:-6390}:6379"
healthcheck:
@@ -0,0 +1,51 @@
package dev.caskeleton.domain.__feature__.testkit;
import dev.caskeleton.domain.__feature__.__Feature__;
import dev.caskeleton.domain.__feature__.__Feature__Id;
import dev.caskeleton.domain.__feature__.__Feature__Repository;
import java.util.Optional;
import java.util.function.Supplier;
/**
* Round-trip probe, never generated into a feature: a repository that refuses work outside its
* caller's transaction, as docs/manual/04-persistence.md teaches a real adapter to, inherits the
* generated contract and passes it (review 2026-11-03 RR-01-01).
*/
class CallerTransaction__Feature__RepositoryContractTest extends __Feature__RepositoryContract {
private final InMemory__Feature__Repository rows = new InMemory__Feature__Repository();
private boolean transactionActive;
@Override
protected __Feature__Repository repository() {
return new __Feature__Repository() {
@Override
public __Feature__ save(__Feature__ aggregate) {
requireTransaction();
return rows.save(aggregate);
}
@Override
public Optional<__Feature__> findById(__Feature__Id id) {
requireTransaction();
return rows.findById(id);
}
};
}
@Override
protected <T> T inTransaction(Supplier<T> work) {
transactionActive = true;
try {
return work.get();
} finally {
transactionActive = false;
}
}
private void requireTransaction() {
if (!transactionActive) {
throw new IllegalStateException("requires an active caller transaction");
}
}
}
@@ -19,15 +19,22 @@ import org.springframework.context.annotation.Import;
*
* <p>Turns itself on from what is on the classpath: a {@link __Feature__Repository} adapter and a
* {@link TransactionPort}. Until a repository adapter exists the feature is absent rather than half
* present (docs/manual/02-new-feature.md step 5). When the adapter comes from another
* auto-configuration, name it in {@code afterName} so this condition sees its bean.
* present (docs/manual/02-new-feature.md step 5).
*
* <p>{@code @ConditionalOnBean} only sees beans of auto-configurations processed before this one,
* and nothing orders them by itself: without {@code afterName} the feature would be on or off
* depending on the class names' alphabetical order. {@code TransactionPort} comes from
* persistence-jpa's {@code PersistenceJpaAutoConfiguration}, named here. When the repository adapter
* comes from another auto-configuration, add its name to the list.
*
* <p>The use cases are beans, not objects the controller builds, because the method security
* advisor proxies beans: {@code @RequiresPermission} on create is enforced by that proxy. Grant
* {@code __feature_kebab__:create} to a role under {@code app.authz.role-permissions}; until then
* every create is refused (docs/manual/07-authorization.md).
*/
@AutoConfiguration
@AutoConfiguration(
afterName =
"dev.caskeleton.adapter.outbound.persistence.autoconfigure.PersistenceJpaAutoConfiguration")
@ConditionalOnWebApplication(type = ConditionalOnWebApplication.Type.SERVLET)
@ConditionalOnBean({__Feature__Repository.class, TransactionPort.class})
@Import(__Feature__Controller.class)
@@ -6,12 +6,17 @@ import dev.caskeleton.domain.__feature__.__Feature__;
import dev.caskeleton.domain.__feature__.__Feature__Id;
import dev.caskeleton.domain.__feature__.__Feature__Repository;
import java.util.Optional;
import java.util.function.Supplier;
import org.junit.jupiter.api.Test;
/**
* What every {@link __Feature__Repository} promises. {@link InMemory__Feature__Repository} extends
* it in domain-core's tests; a real adapter extends it in a lane that runs against its real store
* (docs/manual/04-persistence.md). Add a test here before adding a repository method.
*
* <p>Every repository call goes through {@link #inTransaction(Supplier)}. A real adapter refuses
* work outside its caller's transaction ({@code CallerTransaction.require}), so its contract test
* opens one there; the fake needs none and keeps the default.
*/
public abstract class __Feature__RepositoryContract {
@@ -25,13 +30,25 @@ public abstract class __Feature__RepositoryContract {
*/
protected abstract __Feature__Repository repository();
/**
* Runs one repository call the way a use case does, inside its transaction. The default runs it
* directly; an adapter that requires a caller transaction opens and commits one here.
*
* @param work one call on the repository
* @param <T> what the call returns
* @return what the call returned
*/
protected <T> T inTransaction(Supplier<T> work) {
return work.get();
}
@Test
void aSaved__Feature__IsFoundByItsId() {
// given
repository().save(__Feature__.create(FIRST, "first"));
inTransaction(() -> repository().save(__Feature__.create(FIRST, "first")));
// when
Optional<__Feature__> found = repository().findById(FIRST);
Optional<__Feature__> found = inTransaction(() -> repository().findById(FIRST));
// then
assertThat(found.map(__Feature__::name)).contains("first");
@@ -39,18 +56,19 @@ public abstract class __Feature__RepositoryContract {
@Test
void anIdNothingWasSavedUnderIsNotFound() {
assertThat(repository().findById(UNKNOWN)).isEmpty();
assertThat(inTransaction(() -> repository().findById(UNKNOWN))).isEmpty();
}
@Test
void savingTheSameIdAgainReplacesTheStoredState() {
// given
repository().save(__Feature__.create(FIRST, "first"));
inTransaction(() -> repository().save(__Feature__.create(FIRST, "first")));
// when
repository().save(__Feature__.restore(FIRST, "renamed"));
inTransaction(() -> repository().save(__Feature__.restore(FIRST, "renamed")));
// then
assertThat(repository().findById(FIRST).map(__Feature__::name)).contains("renamed");
assertThat(inTransaction(() -> repository().findById(FIRST)).map(__Feature__::name))
.contains("renamed");
}
}
+64 -11
View File
@@ -12,6 +12,9 @@
# 그 전에 템플릿 패키지 이름(Operation, Notification, Common)을 넣어 거절되는지와 아무것도 쓰지
# 않는지 확인한다.
#
# 생성물의 테스트가 모두 실제로 돌았는지 템플릿 목록으로 확인하고, 생성된 저장소 계약을 호출자
# 트랜잭션을 요구하는 저장소로 한 번 더 돌린다(scripts/feature-scaffold/probes, 기능에는 생성되지 않는다).
#
# 루트 Gradle 태스크 `verifyFeatureScaffold`가 이 스크립트를 부르고 mainCheck에 들어 있다.
set -euo pipefail
@@ -92,18 +95,68 @@ cd "$copy/src"
:app-bootstrap:check \
architectureCheck
# 생성된 테스트가 실제로 실행되었는지: 각 leaf의 테스트 결과에 기능 테스트 클래스가 있어야 한다.
# 생성된 테스트가 실제로 실행되었는지: 템플릿의 테스트 클래스 하나하나가 자기 leaf의 결과 XML을
# 남겨야 한다. 목록을 손으로 적지 않고 템플릿에서 만든다 — 손으로 적은 목록은 8개 중 4개만 보았고,
# 필터·태그가 바뀌어 나머지가 실행되지 않아도 녹색이었다(review 2026-11-03 RR-01-10).
scaffold_templates="$copy/scripts/feature-scaffold/templates/src"
mapfile -t test_templates < <(cd "$scaffold_templates" && find . -path '*/src/test/java/*' -name '*Test.java.tmpl' | sort)
[ "${#test_templates[@]}" -gt 0 ] || { echo "템플릿에 테스트가 없다: $scaffold_templates" >&2; exit 1; }
# 결과 XML 하나를 확인한다: 있고, 한 건 이상 돌았고, 실패·오류가 없다.
require_test_result() {
local result="$1" what="$2"
[ -f "$result" ] || { echo "$what 실행되지 않았다: $result" >&2; exit 1; }
if grep -q 'tests="0"' "$result"; then
echo "$what 한 건도 돌지 않았다: $result" >&2
exit 1
fi
if grep -q '<failure\|<error' "$result"; then
echo "$what 실패했다: $result" >&2
exit 1
fi
}
# 템플릿 경로(<leaf>/src/test/java/<패키지 경로>/<클래스>.java.tmpl)를 그 leaf의 결과 XML 경로로.
result_of() {
local template="${1#./}" name="$2" package_name="$3"
local leaf="${template%%/src/test/java/*}"
local class_path="${template#*/src/test/java/}"
class_path="${class_path%.java.tmpl}"
local fqcn
fqcn="$(printf '%s' "$class_path" | sed -e "s/__Feature__/$name/g" -e "s/__feature__/$package_name/g" | tr '/' '.')"
printf '%s/build/test-results/test/TEST-%s.xml' "$leaf" "$fqcn"
}
for name in "${names[@]}"; do
for result in \
"domain-core/build/test-results/test/TEST-dev.caskeleton.domain.$(printf '%s' "$name" | tr '[:upper:]' '[:lower:]').${name}Test.xml" \
"application-core/build/test-results/test/TEST-dev.caskeleton.application.$(printf '%s' "$name" | tr '[:upper:]' '[:lower:]').Create${name}UseCaseTest.xml" \
"adapter/inbound/web/build/test-results/test/TEST-dev.caskeleton.adapter.inbound.web.$(printf '%s' "$name" | tr '[:upper:]' '[:lower:]').${name}ControllerTest.xml" \
"app-bootstrap/build/test-results/test/TEST-dev.caskeleton.bootstrap.autoconfigure.$(printf '%s' "$name" | tr '[:upper:]' '[:lower:]').${name}ConfigTest.xml"; do
[ -f "$result" ] || { echo "생성된 테스트가 실행되지 않았다: $result" >&2; exit 1; }
if grep -q '<failure\|<error' "$result"; then
echo "생성된 테스트가 실패했다: $result" >&2
exit 1
fi
package_name="$(printf '%s' "$name" | tr '[:upper:]' '[:lower:]')"
for template in "${test_templates[@]}"; do
require_test_result "$(result_of "$template" "$name" "$package_name")" "생성된 테스트가"
done
done
# 생성된 저장소 계약은 실제 adapter가 상속할 수 있어야 한다. 매뉴얼 04장의 adapter는 호출자
# 트랜잭션 밖의 호출을 거부하므로, 그렇게 하는 저장소로 계약을 돌려 본다(review 2026-11-03 RR-01-01).
# 탐침은 기능에 생성되지 않는 테스트라 PR 게이트(check)와 따로 test 태스크만 돌린다.
probe_templates="$copy/scripts/feature-scaffold/probes/src"
mapfile -t probes < <(cd "$probe_templates" && find . -type f -name '*.java.tmpl' | sort)
probe_filters=()
for name in "${names[@]}"; do
package_name="$(printf '%s' "$name" | tr '[:upper:]' '[:lower:]')"
for probe in "${probes[@]}"; do
target="$(printf '%s' "${probe#./}" | sed -e "s/__Feature__/$name/g" -e "s/__feature__/$package_name/g")"
target="${target%.tmpl}"
mkdir -p "$(dirname "$copy/src/$target")"
sed -e "s/__Feature__/$name/g" -e "s/__feature__/$package_name/g" \
"$probe_templates/${probe#./}" > "$copy/src/$target"
class_path="${target#*/src/test/java/}"
probe_filters+=(--tests "$(printf '%s' "${class_path%.java}" | tr '/' '.')")
done
done
./gradlew --console=plain :domain-core:test "${probe_filters[@]}"
for name in "${names[@]}"; do
package_name="$(printf '%s' "$name" | tr '[:upper:]' '[:lower:]')"
for probe in "${probes[@]}"; do
require_test_result "$(result_of "$probe" "$name" "$package_name")" "저장소 계약 탐침이"
done
done
echo "feature scaffold 왕복 통과: ${names[*]}"
+5 -1
View File
@@ -135,6 +135,7 @@ APP_CACHE_CANONICAL_DEFAULT_PROVIDER=disabled
APP_REDIS_ENABLED=true
APP_CACHE_REDIS_POSITIVE_HARD_TTL=5m
APP_CACHE_REDIS_NEGATIVE_TTL=10s
APP_CACHE_REDIS_MAXIMUM_SCRIPT_CONNECTIONS=4
APP_IDEMPOTENCY_REDIS_COMMAND_TIMEOUT=200ms
APP_LEASE_REDIS_COMMAND_TIMEOUT=200ms
APP_LEASE_REDIS_CONTENTION_RETRY_AFTER=50ms
@@ -151,7 +152,7 @@ APP_REDIS_TIMEOUT_SCRIPT=1s
# reads the password from <VARIABLE> and authenticates as <acl-account>. With the application
# reference empty and anonymous access not accepted, the application refuses to start while Redis
# is on; with the advanced one empty, scripts run as the application account and startup warns. The
# Compose Redis lanes connect as the fixture accounts of infra/redis-sdk/acl/all-accounts.acl (the
# Compose Redis lanes connect as the fixture accounts of infra/redis-sdk/acl/lane-fixture-accounts.acl (the
# Redis comment in src/config/runtime/compose-profile-contracts.json).
APP_REDIS_AUTHENTICATION_ADVANCED_CREDENTIAL_REFERENCE=
APP_REDIS_AUTHENTICATION_ANONYMOUS_ACCESS_ACCEPTED=false
@@ -293,6 +294,9 @@ APP_GRAPHQL_PER_PARENT_FETCH_WARNING_THRESHOLD=10
# Optional build-composition: these bind only when :adapter:inbound:websocket is present.
# The default bootJar does not contain that leaf.
APP_WEBSOCKET_ENABLED=true
# Explicit offs for two more capabilities that are on when present (the Compose lanes set them).
APP_OBJECTSTORAGE_ENABLED=true
APP_GRPC_SERVER_ENABLED=true
APP_WEBSOCKET_ENDPOINT=/ws
APP_WEBSOCKET_ALLOWED_ORIGINS=http://localhost:3000
APP_WEBSOCKET_BROADCAST_DESTINATION=/topic/events
+6 -1
View File
@@ -77,7 +77,12 @@ instrumentation, data fetcher resolver 세 곳이 같은 bean 을 받는다).
- **client 문구**: shared-contract `ClientErrorMessages`(Category 별 한국어 문구 표 하나, 소유자
결정 11). 예외 메시지는 절대 나가지 않는다.
- **retryable**: 코드의 `retryable()` 이다(category 에서 계산하지 않는다). 플랫폼 자체 코드만 wire
category 기본값을 쓴다.
category 기본값을 쓴다. 그래서 공유 코드의 값을 바꾸면 이 wire 도 바뀐다: 처리 중 중복
`IDEMPOTENT_IN_FLIGHT` 는 2026-11-01 리뷰(02-01)부터 `retryable: true` 다 — 같은 키로 다시 보내면 서버가
판정한다(`docs/web/http-contract.md` §2). `GraphQlWireErrorMapperTest` 가 고정한다(재리뷰 RR-02-06).
- **`REQUEST_TIMEOUT` 은 web 과 이름만 같다.** 여기의 `category` 는 GraphQL wire 전용 6값 어휘(`TIMEOUT`)이고,
web 의 같은 이름 코드는 HTTP 503·operational category `TRANSIENT_DEPENDENCY` 다
(`docs/web/http-contract.md` §4). 둘 다 `retryable: true` 다.
- **body 크기 초과**: `PAYLOAD_TOO_LARGE`(web 과 같은 코드). 문서 모양 거부(`GRAPHQL_DOCUMENT_SHAPE_REJECTED`)는
별도 코드.
@@ -14,6 +14,9 @@ import java.time.Instant;
* <li>{@code GraphQlPlatformWebInterceptor} ends the request with {@code REQUEST_TIMEOUT} when it
* passes, and Spring cancels the execution so no further field is dispatched. A resolver
* already running is not interrupted.
* <li>{@code GraphQlDeadlineInstrumentation} refuses to start a non-trivial resolver once it has
* passed, failing that field with {@code REQUEST_TIMEOUT}; this is what still holds when the
* whole execution runs on the request thread and the interceptor's timer starts too late.
* <li>{@code GraphQlCostBudgetHandler} refuses to start an execution whose deadline has passed.
* <li>{@code GraphQlBatchExecutor} clamps each platform-registered DataLoader batch to what is
* left, and stops between chunks once it is spent.
@@ -121,11 +121,6 @@ public final class GraphQlComplexityCalculator {
}
}
/** The effective page size for a connection request. */
public int effectivePageSize(Integer first, Integer last) {
return first != null ? first : last != null ? last : defaultPageSize;
}
private void requireAllowedPage(Integer requested) {
if (requested == null) {
return;
@@ -15,6 +15,7 @@ import graphql.language.Value;
import graphql.language.VariableDefinition;
import graphql.language.VariableReference;
import graphql.schema.GraphQLArgument;
import graphql.schema.GraphQLCompositeType;
import graphql.schema.GraphQLFieldDefinition;
import graphql.schema.GraphQLFieldsContainer;
import graphql.schema.GraphQLList;
@@ -59,6 +60,13 @@ import java.util.Objects;
* <p>Pricing an omitted argument as one row was the bypass: {@code items(first: 20)} scored 410
* while {@code items} scored 40, and four nested lists added up instead of multiplying.
*
* <p>A selection set is walked whatever kind of composite type it selects from. A union has no
* fields of its own — only {@code __typename} — so its selections are inline fragments and fragment
* spreads, each priced against its type condition. Treating "not a fields container" as "nothing
* below" was the second bypass (re-review 2026-11-03, RR-02-01): {@code search { ... on Post {
* comments(first: 100000) } }} scored 20, and neither the budget nor the page maximum applied under
* any union-typed field.
*
* <p>Traversal is bounded, and fragment cycles are cut by tracking the expansion path. This runs on
* documents that have passed validation, but the bound stays because the scorer is also used from
* the pre-execution path where a hostile document is exactly what it is meant to price.
@@ -143,7 +151,7 @@ public final class GraphQlDocumentComplexityScorer {
private long selectionSetCost(
GraphQLSchema schema,
GraphQLFieldsContainer parent,
GraphQLCompositeType parent,
SelectionSet selectionSet,
Map<String, FragmentDefinition> fragments,
PageVariables variables,
@@ -173,10 +181,10 @@ public final class GraphQlDocumentComplexityScorer {
expansionPath,
listsAreThePage));
} else if (selection instanceof InlineFragment inlineFragment) {
GraphQLFieldsContainer target =
GraphQLCompositeType target =
inlineFragment.getTypeCondition() == null
? parent
: fieldsContainer(schema, inlineFragment.getTypeCondition().getName());
: compositeType(schema, inlineFragment.getTypeCondition().getName());
total =
GraphQlComplexityCalculator.saturatedAdd(
total,
@@ -196,10 +204,10 @@ public final class GraphQlDocumentComplexityScorer {
if (fragment == null || expansionPath.contains(spread.getName())) {
continue;
}
GraphQLFieldsContainer target =
GraphQLCompositeType target =
fragment.getTypeCondition() == null
? parent
: fieldsContainer(schema, fragment.getTypeCondition().getName());
: compositeType(schema, fragment.getTypeCondition().getName());
expansionPath.push(spread.getName());
total =
GraphQlComplexityCalculator.saturatedAdd(
@@ -221,7 +229,7 @@ public final class GraphQlDocumentComplexityScorer {
private long fieldCost(
GraphQLSchema schema,
GraphQLFieldsContainer parent,
GraphQLCompositeType parent,
Field field,
Map<String, FragmentDefinition> fragments,
PageVariables variables,
@@ -235,8 +243,12 @@ public final class GraphQlDocumentComplexityScorer {
return 0;
}
GraphQLFieldDefinition definition = parent.getFieldDefinition(field.getName());
String coordinate = parent.getName() + "." + field.getName();
// A union declares no fields; validation rejects anything but __typename directly under one.
GraphQLFieldDefinition definition =
parent instanceof GraphQLFieldsContainer container
? container.getFieldDefinition(field.getName())
: null;
if (definition == null) {
// Validation rejects an unknown field; price what is there rather than invent a type.
return calculator.fieldCost(coordinate, 0).total();
@@ -250,7 +262,7 @@ public final class GraphQlDocumentComplexityScorer {
long childCost =
selectionSetCost(
schema,
fieldsContainer(definition.getType()),
compositeType(definition.getType()),
field.getSelectionSet(),
fragments,
variables,
@@ -353,14 +365,14 @@ public final class GraphQlDocumentComplexityScorer {
};
}
private static GraphQLFieldsContainer fieldsContainer(GraphQLSchema schema, String typeName) {
GraphQLType type = schema.getType(typeName);
return fieldsContainer(type);
private static GraphQLCompositeType compositeType(GraphQLSchema schema, String typeName) {
return compositeType(schema.getType(typeName));
}
private static GraphQLFieldsContainer fieldsContainer(GraphQLType type) {
/** The object, interface or union a selection set selects from; null for a leaf type. */
private static GraphQLCompositeType compositeType(GraphQLType type) {
GraphQLType unwrapped = unwrap(type);
return unwrapped instanceof GraphQLFieldsContainer container ? container : null;
return unwrapped instanceof GraphQLCompositeType composite ? composite : null;
}
private static GraphQLType unwrap(GraphQLType type) {
@@ -34,11 +34,19 @@ class GraphQlDocumentComplexityScorerTest {
pagedWithDefault(first: Int = 50): [Item!]!
itemConnection(first: Int): ItemConnection
node: Item
allItems: [Item!]!
search: [SearchResult!]!
nodes: [Named!]!
}
type Item {
union SearchResult = Item
interface Named {
name: String!
}
type Item implements Named {
name: String!
children: [Item!]!
grid: [[Item!]!]!
paged(first: Int): [Item!]!
}
type ItemConnection {
edges: [ItemEdge!]!
@@ -48,10 +56,15 @@ class GraphQlDocumentComplexityScorerTest {
}
""";
// Pricing never resolves a runtime type; the resolvers only satisfy the schema generator.
private static final GraphQLSchema SCHEMA =
new SchemaGenerator()
.makeExecutableSchema(
new SchemaParser().parse(SDL), RuntimeWiring.newRuntimeWiring().build());
new SchemaParser().parse(SDL),
RuntimeWiring.newRuntimeWiring()
.type("SearchResult", type -> type.typeResolver(environment -> null))
.type("Named", type -> type.typeResolver(environment -> null))
.build());
private static final long BUDGET = 10_000;
@@ -246,6 +259,87 @@ class GraphQlDocumentComplexityScorerTest {
assertThat(score).isEqualTo(20L + 20L);
}
@Test
@DisplayName("union 필드 아래 inline fragment 의 중첩 list 도 일반 list 와 같이 곱한다")
void nestedListsUnderAUnionFieldMultiplyLikeAPlainList() {
// given
String plain = "query Q { allItems { children { children { children { name } } } } }";
// when
long score =
score(
"query Q { search { ... on Item { children { children { children { name } } } } } }",
Map.of());
// then
assertThat(score).isEqualTo(score(plain, Map.of()));
assertThatThrownBy(() -> calculator.verify(new GraphQlComplexityResult(score), BUDGET))
.isInstanceOf(GraphQlComplexityRejectedException.class);
}
@Test
@DisplayName("union 필드 아래 fragment spread 도 그 타입으로 내려가 곱한다")
void aFragmentSpreadUnderAUnionFieldIsPricedAgainstItsTypeCondition() {
// given
String plain = "query Q { allItems { children { children { children { name } } } } }";
// when
long score =
score(
"query Q { search { ...F } } "
+ "fragment F on Item { children { children { children { name } } } }",
Map.of());
// then
assertThat(score).isEqualTo(score(plain, Map.of()));
}
@Test
@DisplayName("union 필드 아래 page 인자도 상한을 넘으면 page size exceeds maximum 으로 거부된다")
void aPageSizeUnderAUnionFieldIsCheckedAgainstTheMaximum() {
// given
String document = "query Q { search { ... on Item { paged(first: 100000) { name } } } }";
// when
Throwable thrown = catchThrowable(() -> score(document, Map.of()));
// then
assertThat(thrown)
.isInstanceOf(GraphQlComplexityRejectedException.class)
.hasMessageContaining("page size exceeds maximum");
}
@Test
@DisplayName("interface 필드 아래 inline fragment 의 중첩 list 도 일반 list 와 같이 곱한다")
void nestedListsUnderAnInterfaceFieldMultiplyLikeAPlainList() {
// given
String plain = "query Q { allItems { children { children { children { name } } } } }";
// when
long score =
score(
"query Q { nodes { ... on Item { children { children { children { name } } } } } }",
Map.of());
// then
assertThat(score).isEqualTo(score(plain, Map.of()));
}
@Test
@DisplayName("interface 필드 아래 page 인자도 상한을 넘으면 page size exceeds maximum 으로 거부된다")
void aPageSizeUnderAnInterfaceFieldIsCheckedAgainstTheMaximum() {
// given
String document = "query Q { nodes { ... on Item { paged(first: 100000) { name } } } }";
// when
Throwable thrown = catchThrowable(() -> score(document, Map.of()));
// then
assertThat(thrown)
.isInstanceOf(GraphQlComplexityRejectedException.class)
.hasMessageContaining("page size exceeds maximum");
}
private long score(String source, Map<String, Object> variables) {
return score(calculator, source, variables);
}
@@ -55,6 +55,29 @@ class GraphQlWireErrorMapperTest {
.containsExactly("DB_QUERY_CANCELED", false);
}
/**
* The in-flight duplicate's {@code retryable} became {@code true} in the 2026-11-01 review
* (02-01): a resend with the same key is judged by the server, so it is safe. GraphQL carries the
* code's own value, so the change reached this wire too; this pins it (re-review RR-02-06).
*/
@org.junit.jupiter.api.Test
@DisplayName("처리 중 중복(IDEMPOTENT_IN_FLIGHT)은 GraphQL extension 에서도 retryable=true 로 나간다")
void anInFlightDuplicateIsRetryableOnTheGraphQlWire() {
// given
GraphQlWireErrorMapper mapper = GraphQlWireErrorMapper.masking();
RuntimeException failure =
dev.caskeleton.application.idempotency.IdempotencyInFlightException.forScopeDigest(
"digest");
// when
GraphQlWireError error = mapper.map(failure, GraphQlErrorContext.test());
// then
assertThat(error.extensions())
.extracting(extensions -> extensions.get("code"), extensions -> extensions.get("retryable"))
.containsExactly("IDEMPOTENT_IN_FLIGHT", true);
}
@org.junit.jupiter.api.Test
void anIllegalArgumentIsABadParameterRequestError() {
// given
+2 -1
View File
@@ -140,7 +140,8 @@ handler throw, listener callback throw, `responseObserver.onError(...)`, raw
- **와이어 status(coarse)** 는 `GrpcStatusMapper.toStatus(Category)` 가 결정한다(HTTP status 가
coarse 인 것과 동형). 정확한 `code`/`category`/`retryable` 은 `Status` trailer `Metadata`
(`error-code` / `error-category` / `error-retryable`)에 실어 클라이언트가 switch 하게 한다.
`retryable` 은 코드의 값이다. status description 은 shared-contract `ClientErrorMessages` 의
`retryable` 은 코드의 값이다(처리 중 중복 `IDEMPOTENT_IN_FLIGHT` 는 2026-11-01 리뷰 02-01부터 `ABORTED` +
`error-retryable: true` — 같은 키로 다시 보내면 서버가 판정한다. `GrpcStatusMapperTest` 가 고정, 재리뷰 RR-02-06). status description 은 shared-contract `ClientErrorMessages` 의
Category 별 한국어 문구(web·GraphQL 과 같은 표)다.
- **`IllegalArgumentException`** 은 `BAD_PARAMETER`(`INVALID_ARGUMENT`) — 모든 전송이 같다.
- **`ApiErrorCode` 추출**: feature 예외는 `ApiErrorCarrier`(이 모듈이 제공하는 전송-중립 hook)를
@@ -7,6 +7,7 @@ import dev.caskeleton.shared.error.OperationalError;
import io.grpc.Metadata;
import io.grpc.Status;
import java.util.EnumSet;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
@@ -64,4 +65,22 @@ class GrpcStatusMapperTest {
// then
assertThat(retryable).isEqualTo("false");
}
/**
* The in-flight duplicate's {@code retryable} became {@code true} in the 2026-11-01 review
* (02-01). The trailer carries the code's own value, so the change reached the gRPC wire as well:
* status {@code ABORTED} (category {@code CONFLICT}) with {@code error-retryable: true}
* (re-review RR-02-06).
*/
@Test
@DisplayName("처리 중 중복(IDEMPOTENT_IN_FLIGHT)은 ABORTED 이고 trailer 의 retryable 은 true 다")
void anInFlightDuplicateIsAbortedAndRetryableOnTheGrpcWire() {
// when
Metadata trailers = mapper.trailersFor(OperationalError.IDEMPOTENT_IN_FLIGHT);
// then
assertThat(mapper.statusFor(OperationalError.IDEMPOTENT_IN_FLIGHT).getCode())
.isEqualTo(Status.Code.ABORTED);
assertThat(trailers.get(GrpcStatusMapper.RETRYABLE_KEY)).isEqualTo("true");
}
}
+18
View File
@@ -241,6 +241,7 @@ commit-aware response wrapper 계약을 구현한다. 또한 이 모듈은 HTML
| `PreconditionFailedException` | `PRECONDITION_FAILED` | 412 | D15: `If-Match` 불일치 쓰기 = 낙관적 동시성 충돌 → 412 (raw 409/500 금지) |
| `PageValidationException` | `VALIDATION_FAILED` | 400 | D18/D20/D21: 계약 범위 밖 페이지/정렬/필터 파라미터. field + reasonCode 를 details 로 |
| `IdempotencyUnavailableException` | `IDEMPOTENCY_UNAVAILABLE` | 503 (retryable=true) | 조정 저장소가 응답하지 않아 action 전에 거부. carrier 분기 |
| `IdempotencyCompletionUnknownException` | `IDEMPOTENCY_COMPLETION_UNKNOWN` | 503 (retryable=true) | action 이 돈 뒤 결과 기록(complete) 중 저장소가 응답하지 않음. 결과 불명 — 같은 키로 재전송, 새 키 금지(`docs/web/http-contract.md` §2, 재리뷰 RR-02-03). carrier 분기 |
| `IdempotencyRecoveryRequiredException` | `IDEMPOTENCY_RECOVERY_REQUIRED` | 500 (retryable=false) | 이전 시도의 외부 효과 여부를 판단할 수 없음. carrier 분기 |
| `IdempotencyInFlightException` | `IDEMPOTENT_IN_FLIGHT` | 409 (retryable=true, `Retry-After: 2`) | 같은 scope 의 다른 시도가 아직 처리 중(executor 는 기다리지 않는다). 클라이언트 행동: 결과 불명. 같은 `Idempotency-Key`로 `Retry-After` 뒤 재전송. 새 키 금지(`docs/web/http-contract.md` §2). 진단정보(scope digest)는 클라이언트 미도달 |
| `IdempotencyRequestMismatchException` | `IDEMPOTENT_REQUEST_MISMATCH` | 422 (retryable=false) | D8: `Idempotency-Key` 를 다른 본문으로 재사용. 다시 보내지 않는다(`docs/web/http-contract.md` §2). fingerprint/scope 노출 금지 |
@@ -284,9 +285,26 @@ commit-aware response wrapper 계약을 구현한다. 또한 이 모듈은 HTML
advice 가 같은 mapper 로 Envelope 를 문자열로 직렬화해 넘긴다. 텍스트를 요청한 클라이언트는 그대로
문자열을 받는다. ArchUnit 으로 `String` 반환을 금지하는 안은 버렸다: `text/plain` 엔드포인트의 정당한
용도를 막고, 규칙을 돌리지 않는 서비스에는 함정이 그대로 남는다.
- **4xx·5xx 본문은 감싸지 않는다.** `success` 는 클라이언트가 가르는 판별자라, 오류 status 본문을
`success:true` 로 감싸면 성공으로 읽힌다. Boot `/error` 의 map 이 그렇게 나갔다(재리뷰 RR-02-02).
- 위치: sample 모듈이 아니라 adapter-web. 실행 앱은 adapter-web 에 의존하지만 sample-portfolio 에는 의존하지
않으므로, 응답 래핑이 실제로 동작하려면 여기 있어야 한다.
### EnvelopeErrorController
- 컨트롤러에 닿기 전에 생긴 오류(보안 방화벽의 `RequestRejectedException`, 필터의 `sendError`, 필터가
던진 예외)는 `GlobalExceptionHandler` 를 지나지 않고 컨테이너의 `/error` 디스패치로 간다. Boot 기본
`BasicErrorController` 는 거기서 `{timestamp, status, error, path}` 를 썼고, advice 가 그것을
`success:true` 로 감쌌으며 `path` 는 요청 경로 원문이었다(재리뷰 RR-02-02). 이 컨트롤러가 그 자리를
대신해 status 를 `GlobalExceptionHandler` 와 같은 표(`HttpStatusErrorCodes`)로 코드에 매핑하고 실패
Envelope 를 `EnvelopeResponseWriter` 로 쓴다. 요청에서 가져온 값은 싣지 않는다.
- `WebMvcPlatformAutoConfiguration` 이 `ErrorMvcAutoConfiguration` 보다 먼저 `ErrorController` 가 없을 때만
등록한다. 어댑터가 자기 `ErrorController` 를 두면 물러난다.
- `SecurityConfig` 는 오류 **디스패치**만 허용한다(`DispatcherType.ERROR`). 예전에는 `/error` 가 공개 경로가
아니라서 익명 요청의 400 이 401 로 바뀌었다. `/error` 를 직접 부른 요청은 여전히 인증이 필요하다.
단언은 실제 Tomcat 의 `TomcatWebContractIT`(공유 `WebContractAssertions`)와 `JwtChainErrorDispatchIT`.
- 앱이 보안 체인을 직접 정의하면(예: sample 의 데모 체인) 같은 한 줄
(`dispatcherTypeMatchers(DispatcherType.ERROR).permitAll()`)을 그 체인에도 둬야 원래 status 가 나간다.
### CacheControlFilter
- 스켈레톤 기본 HTTP 캐시 정책.
- `Cache-Control: no-store` 는 인증된 API 의 안전한 기본값. `Vary: Accept, Accept-Encoding, Authorization` 로
@@ -2,6 +2,7 @@ package dev.caskeleton.adapter.inbound.web.auth;
import dev.caskeleton.adapter.inbound.web.settings.CorsSettings;
import dev.caskeleton.adapter.inbound.web.settings.SecuritySettings;
import jakarta.servlet.DispatcherType;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@@ -83,6 +84,11 @@ public class SecurityConfig {
.requestCache(cache -> cache.disable())
.authorizeHttpRequests(
auth -> {
// The container's error dispatch for a request that already failed. Authorising it
// again turned an anonymous 400 into a 401 that blamed authentication. Only the
// dispatch type is opened: a request that names /error itself is still a request.
// The error controller writes nothing from the request (re-review RR-02-02).
auth.dispatcherTypeMatchers(DispatcherType.ERROR).permitAll();
if (publicPaths.length > 0) {
auth.requestMatchers(publicPaths).permitAll();
}
@@ -12,6 +12,7 @@ import org.springframework.http.converter.HttpMessageConverter;
import org.springframework.http.converter.StringHttpMessageConverter;
import org.springframework.http.server.ServerHttpRequest;
import org.springframework.http.server.ServerHttpResponse;
import org.springframework.http.server.ServletServerHttpResponse;
import org.springframework.web.bind.annotation.RestControllerAdvice;
import org.springframework.web.servlet.mvc.method.annotation.ResponseBodyAdvice;
import tools.jackson.databind.ObjectMapper;
@@ -34,6 +35,13 @@ import tools.jackson.databind.json.JsonMapper;
* still sees {@code 200 text/plain}. So when the String converter is about to write JSON, the
* envelope is serialised here, with the same mapper and wire rules, and the converter writes that
* text (review 2026-11-01 02-06). A text client still gets the plain string.
*
* <p>A body sent with a 4xx or 5xx status is never wrapped. {@code success} is the discriminator a
* client branches on, so a {@code success:true} envelope around an error-status body tells it the
* request worked. Boot's {@code /error} map reached clients exactly that way until the error
* dispatch got its own envelope writer ({@code EnvelopeErrorController}, re-review 2026-11-03
* RR-02-02). Failure envelopes come from the error handlers; a handler that returns some other body
* with an error status gets that body on the wire as it wrote it, not a false success.
*/
@RestControllerAdvice
public final class EnvelopeBodyAdvice implements ResponseBodyAdvice<Object> {
@@ -74,10 +82,18 @@ public final class EnvelopeBodyAdvice implements ResponseBodyAdvice<Object> {
if (selectedContentType != null && !MediaType.APPLICATION_JSON.includes(selectedContentType)) {
return body;
}
if (isErrorStatus(response)) {
return body;
}
Envelope<Object> envelope = Envelope.ok(body, ResponseMetaFactory.fromMdc());
if (StringHttpMessageConverter.class.isAssignableFrom(selectedConverterType)) {
return objectMapper.writeValueAsString(envelope);
}
return envelope;
}
private static boolean isErrorStatus(ServerHttpResponse response) {
return response instanceof ServletServerHttpResponse servlet
&& servlet.getServletResponse().getStatus() >= 400;
}
}
@@ -29,7 +29,7 @@ public class EnvelopeWireFormatConfig {
}
/** The module itself, for a writer that has no Boot mapper to take it from. */
static SimpleModule module() {
public static SimpleModule module() {
SimpleModule module = new SimpleModule("EnvelopeWireFormatModule");
module.setMixInAnnotation(Envelope.class, OmitNullFields.class);
module.setMixInAnnotation(BulkEnvelope.class, OmitNullFields.class);
@@ -0,0 +1,72 @@
package dev.caskeleton.adapter.inbound.web.error;
import dev.caskeleton.adapter.inbound.web.core.OwnsAbsoluteWebRoute;
import dev.caskeleton.adapter.inbound.web.http.ApiHeaders;
import dev.caskeleton.adapter.inbound.web.observability.RetryAfterAdvisor;
import dev.caskeleton.shared.error.ApiErrorCode;
import dev.caskeleton.shared.error.OperationalError;
import jakarta.servlet.RequestDispatcher;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.util.Objects;
import org.springframework.boot.webmvc.error.ErrorController;
import org.springframework.http.HttpStatusCode;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import tools.jackson.databind.ObjectMapper;
/**
* Answers the container's error dispatch with the failure {@code Envelope}.
*
* <p>A failure raised outside the MVC handler — the security firewall's {@code
* RequestRejectedException}, a filter's {@code sendError}, an exception thrown by a filter — never
* reaches {@link GlobalExceptionHandler}. The container forwards it to {@code /error} instead,
* where Boot's {@code BasicErrorController} wrote {@code {timestamp, status, error, path}}; the
* success advice then wrapped that map as {@code success:true}, and {@code path} echoed the raw
* request URI back (re-review 2026-11-03, RR-02-02). This replaces that controller: the error
* status is classified by the same table the exception handler uses, and the body is the failure
* envelope, with nothing from the request in it.
*
* <p>Written straight to the response by {@link EnvelopeResponseWriter}, like the other refusals
* made outside a converter. An error dispatch has no handler of its own to negotiate for, and a
* browser's {@code Accept: text/html} would otherwise turn the error into a 406 about the error.
*
* <p>A request that names {@code /error} itself carries no error status; it is answered as an
* unknown route. The security chain permits only the error dispatch, so such a request still needs
* a token under the JWT chain.
*
* <p>An absolute route ({@link OwnsAbsoluteWebRoute}): the container forwards to {@code /error},
* not to {@code /error} under the presentation prefix.
*/
@Controller
@OwnsAbsoluteWebRoute
@RequestMapping("${server.error.path:${error.path:/error}}")
public final class EnvelopeErrorController implements ErrorController {
private final ObjectMapper objectMapper;
/**
* Creates the controller.
*
* @param objectMapper Boot's mapper, which carries the envelope wire rules
*/
public EnvelopeErrorController(ObjectMapper objectMapper) {
this.objectMapper = Objects.requireNonNull(objectMapper, "objectMapper");
}
/** Writes the failure envelope for the status the error dispatch carries. */
@RequestMapping
public void error(HttpServletRequest request, HttpServletResponse response) throws IOException {
if (response.isCommitted()) {
return;
}
ApiErrorCode code =
request.getAttribute(RequestDispatcher.ERROR_STATUS_CODE) instanceof Integer status
? HttpStatusErrorCodes.forStatus(HttpStatusCode.valueOf(status))
: OperationalError.ROUTE_NOT_FOUND;
RetryAfterAdvisor.retryAfterSeconds(code)
.ifPresent(seconds -> response.setHeader(ApiHeaders.RETRY_AFTER, String.valueOf(seconds)));
EnvelopeResponseWriter.write(response, code, objectMapper);
}
}
@@ -407,7 +407,7 @@ public class GlobalExceptionHandler extends ResponseEntityExceptionHandler {
return null;
}
}
ApiErrorCode code = errorCodeFor(statusCode);
ApiErrorCode code = HttpStatusErrorCodes.forStatus(statusCode);
if (code.httpStatus() >= 500) {
log.error("spring mvc failure classified as {}", code.code(), ex);
spanErrorRecorder.recordException(ex, code.code());
@@ -415,25 +415,4 @@ public class GlobalExceptionHandler extends ResponseEntityExceptionHandler {
return new ResponseEntity<>(
ErrorResponseFactory.body(code, null), headers, HttpStatusCode.valueOf(code.httpStatus()));
}
private static ApiErrorCode errorCodeFor(HttpStatusCode status) {
return switch (status.value()) {
case 401 -> OperationalError.UNAUTHENTICATED;
case 403 -> OperationalError.FORBIDDEN;
case 404 -> OperationalError.ROUTE_NOT_FOUND;
case 405 -> OperationalError.METHOD_NOT_ALLOWED;
case 406 -> OperationalError.NOT_ACCEPTABLE;
case 412 -> OperationalError.PRECONDITION_FAILED;
case 413 -> OperationalError.PAYLOAD_TOO_LARGE;
case 414 -> OperationalError.URI_TOO_LONG;
case 415 -> OperationalError.UNSUPPORTED_MEDIA_TYPE;
case 429 -> OperationalError.RATE_LIMIT_EXCEEDED;
// Spring answers its async request timeout with 503: transient, not an internal fault.
case 503 -> WebTransportError.REQUEST_TIMEOUT;
default ->
status.is4xxClientError()
? OperationalError.BAD_PARAMETER
: OperationalError.INTERNAL_ERROR;
};
}
}
@@ -0,0 +1,38 @@
package dev.caskeleton.adapter.inbound.web.error;
import dev.caskeleton.shared.error.ApiErrorCode;
import dev.caskeleton.shared.error.OperationalError;
import org.springframework.http.HttpStatusCode;
/**
* The operational code a bare HTTP status stands for, when a status is all that is known.
*
* <p>Two places only have a status to classify: a Spring MVC exception this adapter has no
* dedicated handler for, and the container's error dispatch for a failure raised outside the
* handler. They share this one table, so the same status never reaches a client as two codes.
*/
final class HttpStatusErrorCodes {
private HttpStatusErrorCodes() {}
static ApiErrorCode forStatus(HttpStatusCode status) {
return switch (status.value()) {
case 401 -> OperationalError.UNAUTHENTICATED;
case 403 -> OperationalError.FORBIDDEN;
case 404 -> OperationalError.ROUTE_NOT_FOUND;
case 405 -> OperationalError.METHOD_NOT_ALLOWED;
case 406 -> OperationalError.NOT_ACCEPTABLE;
case 412 -> OperationalError.PRECONDITION_FAILED;
case 413 -> OperationalError.PAYLOAD_TOO_LARGE;
case 414 -> OperationalError.URI_TOO_LONG;
case 415 -> OperationalError.UNSUPPORTED_MEDIA_TYPE;
case 429 -> OperationalError.RATE_LIMIT_EXCEEDED;
// Spring answers its async request timeout with 503: transient, not an internal fault.
case 503 -> WebTransportError.REQUEST_TIMEOUT;
default ->
status.is4xxClientError()
? OperationalError.BAD_PARAMETER
: OperationalError.INTERNAL_ERROR;
};
}
}
@@ -1,5 +1,7 @@
package dev.caskeleton.adapter.inbound.web.mvc.autoconfigure;
import dev.caskeleton.adapter.inbound.web.envelope.EnvelopeWireFormatConfig;
import dev.caskeleton.adapter.inbound.web.error.EnvelopeErrorController;
import dev.caskeleton.adapter.inbound.web.error.GlobalExceptionHandler;
import dev.caskeleton.adapter.inbound.web.mvc.filter.JsonBodyLimitFilter;
import dev.caskeleton.adapter.inbound.web.mvc.filter.WebMvcRequestIdFilter;
@@ -12,8 +14,12 @@ import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.autoconfigure.condition.ConditionalOnWebApplication;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.boot.jackson.autoconfigure.JsonMapperBuilderCustomizer;
import org.springframework.boot.webmvc.autoconfigure.error.ErrorMvcAutoConfiguration;
import org.springframework.boot.webmvc.error.ErrorController;
import org.springframework.context.annotation.Bean;
import tools.jackson.core.StreamReadFeature;
import tools.jackson.databind.ObjectMapper;
import tools.jackson.databind.json.JsonMapper;
/**
* Wires the web platform into a servlet application.
@@ -30,8 +36,12 @@ import tools.jackson.core.StreamReadFeature;
*
* <p>Every bean is {@link ConditionalOnMissingBean}: the platform supplies the one that would
* otherwise be missing rather than the one that must be used.
*
* <p>Before {@link ErrorMvcAutoConfiguration}, so the envelope error controller is the {@link
* ErrorController} that configuration finds and Boot's {@code BasicErrorController} backs off. Its
* error-page registration with the container stays: that is what forwards to {@code /error}.
*/
@AutoConfiguration
@AutoConfiguration(before = ErrorMvcAutoConfiguration.class)
@ConditionalOnWebApplication(type = ConditionalOnWebApplication.Type.SERVLET)
@ConditionalOnProperty(
prefix = "app.web.mvc",
@@ -54,6 +64,22 @@ public class WebMvcPlatformAutoConfiguration {
return new GlobalExceptionHandler(spanErrorRecorderProvider);
}
/**
* Answers the container's error dispatch — failures raised outside the MVC handler — with the
* failure envelope instead of Boot's {@code {timestamp, status, error, path}} map.
*
* <p>Boot's mapper when there is one, which already carries the envelope wire rules; otherwise a
* mapper with only those rules, as {@code EnvelopeBodyAdvice} does without Boot.
*/
@Bean
@ConditionalOnMissingBean(ErrorController.class)
public EnvelopeErrorController envelopeErrorController(
ObjectProvider<ObjectMapper> objectMapper) {
return new EnvelopeErrorController(
objectMapper.getIfAvailable(
() -> JsonMapper.builder().addModule(EnvelopeWireFormatConfig.module()).build()));
}
/**
* Assigns the correlation identifiers, validating anything a caller proposed. With a tracer in
* the composition the trace id is the tracer's: the filter runs inside Boot's server observation
@@ -1,5 +1,6 @@
package dev.caskeleton.adapter.inbound.web.observability;
import dev.caskeleton.application.idempotency.IdempotencyExecutor;
import dev.caskeleton.shared.error.ApiErrorCode;
import java.util.Map;
import java.util.OptionalInt;
@@ -13,17 +14,26 @@ public final class RetryAfterAdvisor {
/** Per-code {@code Retry-After} values (seconds). */
private static final Map<String, Integer> RETRY_AFTER_SECONDS =
Map.of(
"RATE_LIMIT_EXCEEDED", 1,
// Below the default processing lease (30s): the resend arrives while the first attempt
// can still finish, and an early resend costs one more 409. docs/web/http-contract.md.
"IDEMPOTENT_IN_FLIGHT", 2,
"AUTH_KID_UNKNOWN", 5,
"AUTH_JWKS_UNAVAILABLE", 30,
"DEPENDENCY_TIMEOUT", 2,
"DEPENDENCY_CONNECT_FAILED", 2,
"DEPENDENCY_DNS_FAILED", 5,
"DEPENDENCY_5XX_SERVER", 2,
"DEPENDENCY_CIRCUIT_OPEN", 10);
"RATE_LIMIT_EXCEEDED",
1,
// The application's value, which the processing-lease check also reads: the resend
// arrives while the first attempt can still finish. docs/web/http-contract.md §2.
"IDEMPOTENT_IN_FLIGHT",
Math.toIntExact(IdempotencyExecutor.IN_FLIGHT_RETRY_AFTER.toSeconds()),
"AUTH_KID_UNKNOWN",
5,
"AUTH_JWKS_UNAVAILABLE",
30,
"DEPENDENCY_TIMEOUT",
2,
"DEPENDENCY_CONNECT_FAILED",
2,
"DEPENDENCY_DNS_FAILED",
5,
"DEPENDENCY_5XX_SERVER",
2,
"DEPENDENCY_CIRCUIT_OPEN",
10);
private RetryAfterAdvisor() {}
@@ -0,0 +1,91 @@
package dev.caskeleton.adapter.inbound.web.testkit.mvc;
import static org.assertj.core.api.Assertions.assertThat;
import dev.caskeleton.adapter.inbound.web.envelope.EnvelopeWireFormatConfig;
import dev.caskeleton.adapter.inbound.web.mvc.autoconfigure.WebAdapterAutoConfiguration;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.web.server.LocalServerPort;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Import;
/**
* The error dispatch under the production JWT security chain, over a real Tomcat.
*
* <p>The chain authorises every dispatch, the container's error dispatch to {@code /error}
* included. {@code /error} is not a public path, so an anonymous request whose original failure was
* a 400 used to come back as 401 — the client was told to authenticate when its request was
* malformed (re-review 2026-11-03, RR-02-02). The error dispatch is now permitted; a request that
* names {@code /error} directly is still a request and still needs a token.
*
* <p>The chain is {@code WebSecurityAutoConfiguration} itself. The JWT decoder is lazy, so the
* unreachable issuer below is never contacted: no request here carries a token.
*/
@SpringBootTest(
classes = JwtChainErrorDispatchIT.JwtChainApplication.class,
webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
properties = {
"spring.main.banner-mode=off",
"spring.mvc.problemdetails.enabled=false",
"app.web.mvc.enabled=true",
"app.security.auth-mode=jwt",
"app.security.issuer-uri=https://issuer.invalid/realms/ca",
"app.security.audience=ca-skeleton-api",
"app.security.public-paths=/api/v1/fixtures/**",
"app.cors.enabled=false"
})
class JwtChainErrorDispatchIT {
private final HttpClient client =
HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
@LocalServerPort private int port;
@Test
@DisplayName("JWT 체인에서 익명 요청이 핸들러 밖에서 400 으로 거부되면 401 이 아니라 400 실패 Envelope 로 나간다")
void anAnonymousRequestRejectedOutsideTheHandlerKeepsItsOwnStatus() throws Exception {
// when
HttpResponse<String> rejected = get(WebContractAssertions.FIREWALL_REJECTED_PATH);
// then
assertThat(rejected.statusCode()).isEqualTo(400);
WebContractAssertions.assertFailureEnvelope(rejected, "BAD_PARAMETER");
}
@Test
@DisplayName("오류 디스패치만 허용한다 — /error 를 직접 부른 익명 요청은 여전히 401 이다")
void aDirectRequestForTheErrorPathStillNeedsAToken() throws Exception {
// when
HttpResponse<String> direct = get("/error");
// then
assertThat(direct.statusCode()).isEqualTo(401);
}
private HttpResponse<String> get(String path) throws Exception {
return client.send(
HttpRequest.newBuilder(URI.create("http://localhost:" + port + path))
.header("Accept", "application/json")
.GET()
.build(),
HttpResponse.BodyHandlers.ofString());
}
/**
* The production web stack minus {@link WebAdapterAutoConfiguration}, whose request-logging
* filter needs the composition root's pseudonymizer, plus that root's envelope wire format; the
* security chain is the shipped one.
*/
@Configuration(proxyBeanMethods = false)
@EnableAutoConfiguration(exclude = WebAdapterAutoConfiguration.class)
@Import({ContractFixtureController.class, EnvelopeWireFormatConfig.class})
static class JwtChainApplication {}
}
@@ -23,6 +23,7 @@ import org.springframework.boot.webmvc.test.autoconfigure.AutoConfigureMockMvc;
import org.springframework.boot.webmvc.test.autoconfigure.WebMvcTest;
import org.springframework.context.annotation.Import;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.test.json.JsonCompareMode;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.web.bind.annotation.GetMapping;
@@ -64,6 +65,12 @@ class EnvelopeBodyAdviceTest {
return "{\"openapi\":\"3.1.0\"}".getBytes(StandardCharsets.UTF_8);
}
/** A failure status with a body that is not an envelope. */
@GetMapping("/probe/conflict")
public ResponseEntity<Map<String, String>> conflict() {
return ResponseEntity.status(409).body(Map.of("k", "v"));
}
@GetMapping("/probe/bulk")
public BulkEnvelope<Map<String, String>> bulk() {
return BulkEnvelope.allOk(List.of(Map.of("k", "v")), new ResponseMeta("r", "t", "c"));
@@ -149,6 +156,25 @@ class EnvelopeBodyAdviceTest {
.andExpect(content().json("{\"openapi\":\"3.1.0\"}", JsonCompareMode.STRICT));
}
/**
* A 4xx/5xx body is never a success envelope. {@code success} is the discriminator clients branch
* on (docs/web/http-contract.md §1), so wrapping an error-status body as {@code success:true}
* tells a contract-following client the request worked — Boot's {@code /error} map arrived that
* way (re-review 2026-11-03, RR-02-02). The body is left as the handler wrote it.
*/
@Test
@DisplayName("4xx·5xx 응답 본문은 success:true Envelope 로 감싸지 않는다")
void anErrorStatusBodyIsNotWrappedAsASuccess() throws Exception {
// when
var result = mvc.perform(get("/probe/conflict").accept(MediaType.APPLICATION_JSON));
// then
result
.andExpect(status().isConflict())
.andExpect(jsonPath("$.success").doesNotExist())
.andExpect(content().json("{\"k\":\"v\"}", JsonCompareMode.STRICT));
}
@SpringBootConfiguration
@EnableAutoConfiguration(exclude = SecurityAutoConfiguration.class)
static class TestBootstrap {}
@@ -3,6 +3,7 @@ package dev.caskeleton.adapter.inbound.web.error;
import static org.assertj.core.api.Assertions.assertThat;
import dev.caskeleton.adapter.inbound.web.http.ApiHeaders;
import dev.caskeleton.application.idempotency.IdempotencyCompletionUnknownException;
import dev.caskeleton.application.idempotency.IdempotencyInFlightException;
import dev.caskeleton.application.idempotency.IdempotencyRequestMismatchException;
import dev.caskeleton.shared.error.Category;
@@ -74,4 +75,24 @@ class IdempotencyExceptionMappingTest {
false,
ClientErrorMessages.of(Category.VALIDATION));
}
@Test
@DisplayName("complete 단계 결과 불명은 실행 전 거부(IDEMPOTENCY_UNAVAILABLE)와 다른 code 의 retryable 503 이다")
void completionUnknownMapsToItsOwnRetryable503SoTheClientDoesNotReadItAsNotApplied() {
// given
IdempotencyCompletionUnknownException unknown = new IdempotencyCompletionUnknownException();
// when
ResponseEntity<Envelope<Void>> response = handler.handleCarrier(unknown);
// then
assertThat(response)
.extracting(
r -> r.getStatusCode().value(),
r -> r.getHeaders().getFirst(ApiHeaders.RETRY_AFTER),
r -> r.getBody().error().code(),
r -> r.getBody().error().category(),
r -> r.getBody().error().retryable())
.containsExactly(503, null, "IDEMPOTENCY_COMPLETION_UNKNOWN", "TRANSIENT_DEPENDENCY", true);
}
}
@@ -8,6 +8,8 @@ import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import tools.jackson.databind.JsonNode;
import tools.jackson.databind.json.JsonMapper;
/**
* The status contract as socket-level assertions, written once and run against every container.
@@ -22,6 +24,11 @@ import java.time.Duration;
*/
public final class WebContractAssertions {
/** A path the servlet firewall refuses before any handler runs ({@code ;} in the URI). */
public static final String FIREWALL_REJECTED_PATH = "/api/v1/fixtures/f1;x=1";
private static final JsonMapper JSON = JsonMapper.builder().build();
private final HttpClient client =
HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
@@ -122,6 +129,46 @@ public final class WebContractAssertions {
assertThat(post("/api/v1/fixtures", "{\"name\":").statusCode()).isEqualTo(400);
}
/**
* An error raised outside the MVC handler is still a failure envelope.
*
* <p>The firewall rejects the request before any controller runs, and the container answers it
* with an error dispatch to {@code /error}. Boot's default error controller wrote a map there,
* which the success advice then wrapped as {@code success:true} with the raw request path inside
* (re-review 2026-11-03, RR-02-02). MockMvc performs no error dispatch, so only a container can
* show this.
*/
public void aRequestRejectedOutsideTheHandlerIsAFailureEnvelope()
throws IOException, InterruptedException {
HttpResponse<String> rejected = get(FIREWALL_REJECTED_PATH);
assertThat(rejected.statusCode()).isEqualTo(400);
assertFailureEnvelope(rejected, "BAD_PARAMETER");
}
/**
* Asserts the failure envelope of {@code docs/web/http-contract.md} §1 on an error response.
*
* @param response a response the container wrote through its error dispatch
* @param expectedCode the {@code error.code} the status maps to
*/
public static void assertFailureEnvelope(HttpResponse<String> response, String expectedCode) {
JsonNode body = JSON.readTree(response.body());
assertThat(body.path("success").isBoolean() && !body.path("success").booleanValue())
.as(
"a 4xx/5xx body that says success:true is read as a success by a client that follows"
+ " the contract: %s",
response.body())
.isTrue();
assertThat(body.path("error").path("code").asString()).isEqualTo(expectedCode);
assertThat(body.has("data")).as("a failure envelope has no data key").isFalse();
assertThat(response.body())
.as("the error body must not echo the request path back")
.doesNotContain(";x=1")
.doesNotContain("\"path\"")
.doesNotContain("timestamp");
}
/** Runs every assertion, so a lane cannot silently cover a subset. */
public void assertWholeContract() throws IOException, InterruptedException {
createdCarriesALocationHeader();
@@ -131,6 +178,7 @@ public final class WebContractAssertions {
anUnknownMethodOnAKnownRouteIsRefused();
anUnsupportedMediaTypeIsRefused();
malformedJsonIsRefused();
aRequestRejectedOutsideTheHandlerIsAFailureEnvelope();
}
private HttpResponse<String> post(String path, String body)
@@ -143,6 +191,12 @@ public final class WebContractAssertions {
HttpResponse.BodyHandlers.ofString());
}
private HttpResponse<String> get(String path) throws IOException, InterruptedException {
return client.send(
HttpRequest.newBuilder(uri(path)).header("Accept", "application/json").GET().build(),
HttpResponse.BodyHandlers.ofString());
}
private URI uri(String path) {
return URI.create("http://localhost:" + port + path);
}
@@ -1,5 +1,6 @@
package dev.caskeleton.webtestkit;
import dev.caskeleton.adapter.inbound.web.envelope.EnvelopeWireFormatConfig;
import dev.caskeleton.adapter.inbound.web.mvc.autoconfigure.WebAdapterAutoConfiguration;
import dev.caskeleton.adapter.inbound.web.mvc.autoconfigure.WebSecurityAutoConfiguration;
import dev.caskeleton.adapter.inbound.web.testkit.fault.FaultFixtureController;
@@ -31,6 +32,10 @@ import org.springframework.security.web.SecurityFilterChain;
* are the "sum of the leaf" the paragraph above keeps out. Excluded by class, so a rename breaks
* compilation rather than the gate. The third root, {@code WebMethodSecurityAutoConfiguration}, is
* left in: no fixture route runs a permissioned use case, so it decides nothing here.
*
* <p>{@link EnvelopeWireFormatConfig} is imported by name from the excluded web root: it is the
* wire shape of every envelope this gate reads (a failure body has no {@code data} key), not a
* capability, and without it the gate would certify {@code "data": null} that no deployment sends.
*/
// Outside `dev.caskeleton.adapter.inbound.web`, which the composition root component-scans.
// A fixture application under that root is loaded into the real deployment: its permit-all
@@ -40,7 +45,11 @@ import org.springframework.security.web.SecurityFilterChain;
@Configuration(proxyBeanMethods = false)
@EnableAutoConfiguration(
exclude = {WebAdapterAutoConfiguration.class, WebSecurityAutoConfiguration.class})
@Import({ContractFixtureController.class, FaultFixtureController.class})
@Import({
ContractFixtureController.class,
FaultFixtureController.class,
EnvelopeWireFormatConfig.class
})
public class ContractFixtureApplication {
/**
+18 -6
View File
@@ -128,12 +128,15 @@ Focused tests drive the semantic adapters against `InMemorySemanticRedisCommands
served through `StubRedisRuntimeClient`. R1/R2 promotion requires a separate real Redis service lane; it may never
be silently skipped when selected.
Two real-server lanes. `redisContainerContractTest` (integration category, in `integrationCheck` and
this leaf's `mainCheck`) starts its own Redis with Testcontainers — the `redis` image of
`config/testing/container-images.properties`, loading `infra/redis-sdk/acl/all-accounts.acl` — and
runs `LiveRedisCacheRegionContractTest` and `LiveRedisDistributedLeaseContractTest` (the
`application-core` port contracts the in-memory fakes also pass), `LiveRedisSemanticPortsTest` and
`LiveRedisSemanticScriptAclProbeTest`; a skip fails it (review 2026-11-01 05-16). `RedisTestContainer`
Two real-server lanes. `redisContainerContractTest` (integration category, in `integrationCheck` only —
not in this leaf's `mainCheck` too, which ran it twice per push to main) starts its own Redis with Testcontainers — the `redis` image of
`config/testing/container-images.properties`, loading `infra/redis-sdk/acl/lane-fixture-accounts.acl` — and
runs `LiveRedisCacheRegionContractTest`, `LiveRedisDistributedLeaseContractTest` and
`LiveRedisIdempotencyStoreContractTest` (the `application-core` port contracts the in-memory fakes
also pass), `LiveRedisSemanticPortsTest` and `LiveRedisSemanticScriptAclProbeTest`; a skip fails it
(review 2026-11-01 05-16). The idempotency contract was on the explicit-only topology lane alone
until review 2026-11-03 RR-04-05, so the Lua state machine's stale-handle, renew-refusal and
retry cases reached a real server in no gate. `RedisTestContainer`
starts the server only when that lane names the ACL file.
`redisTopologyTest` is the external-topology lane, declared through `ca.redis-topology-lane`. This
@@ -177,6 +180,15 @@ deployment still gets exactly one client and one event loop. The `SCRIPT` lane i
the reason it exists: `SCRIPT LOAD` and `EVALSHA` belong to the advanced account, so the account
that reads a cache entry cannot execute a script.
The cache's conditional writes run on the same account but on a lane of their own,
`CacheScriptLane.CACHE_SCRIPT` (`app.capabilities.cache.maximum-script-connections`, contributed by
`RedisCacheConfig` as a `RedisLaneContribution`). Do not move them back to `SCRIPT`: that lane is the
idempotency, rate-limit and lease budget, and a cache under load must not take it (review 2026-11-03
README §4, RR-04-02). Each lane's wait, refusals and reconnects are meters of `redis-runtime`
(`RedisLaneMetrics`, `redis.lane.*`, tag `lane`), so whether the cache stays off `SCRIPT` is a query,
not an assumption. The `DEGRADED` health order ships with `redis-runtime`, which reports the
status (`config/module-defaults/redis-runtime.yml`, RR-04-10).
A Cluster transaction runs on one node, so the SDK's `RedisTransactionRunner` derives a routing key
from the watched keys (or an explicit `RedisSlotTag`) and borrows `RedisSdkLane.TRANSACTION`, which
is pinned to its key's owner, on the node that owns that slot. Routed leases are never pooled: a pooled connection is pinned to the previous caller's
+54 -7
View File
@@ -97,7 +97,10 @@ Connection lane limits and command queue limits are different contracts. In part
- `app.redis.capacity.maximum-in-flight-commands` bounds the per-connection command queue.
- `app.redis.capacity.maximum-script-connections` bounds physical SCRIPT lane connections
(default 8).
(default 8). Idempotency, rate limiting and leases share them.
- `app.capabilities.cache.maximum-script-connections` (`APP_CACHE_REDIS_MAXIMUM_SCRIPT_CONNECTIONS`)
bounds the cache's own script lane (`CacheScriptLane`, default 4), contributed only when the cache
binding is `redis`. See "Why the cache has its own script lane" below.
- `app.redis.lifecycle.acquire-timeout` (default 2s) is how long a borrow waits for a returned
connection when its lane is full.
@@ -153,7 +156,7 @@ Important contracts:
Every lookup returns a `CacheWriteCondition` naming the state it saw — `absent`, `vacant:<digest>`
(an invalidation marker, an entry of an older region generation, an expired entry) or
`present:<digest>`. A record that carries the condition applies only while the key is still in that
state (registered program, SCRIPT lane). This is what rejects a source load that began before an
state (registered program, cache script lane). This is what rejects a source load that began before an
invalidation and finished after it.
A hit also returns an opaque SHA-1 observation token over the stored bytes. SHA-1 here is **not a
@@ -181,10 +184,43 @@ example one written by `UPSERT` as `rev-1`), another schema version, a generatio
writer knows — is `NOT_RECORDED_CONDITION`.
Both conditional intents, and any record carrying a write condition, run registered scripts, so
they borrow the SCRIPT lane (the advanced account when one is configured, the application account
otherwise). The application account is not granted `EVALSHA`; on the REGULAR lane every conditional
write came back `INDETERMINATE`. Unconditioned `UPSERT` and `ONLY_IF_ABSENT` stay on the REGULAR
lane.
they run on the scripting account (the advanced account when one is configured, the application
account otherwise), on the cache's own script lane. The application account is not granted
`EVALSHA`; on the REGULAR lane every conditional write came back `INDETERMINATE`. Unconditioned
`UPSERT` and `ONLY_IF_ABSENT` stay on the REGULAR lane.
### Why the cache has its own script lane
Recording with the condition a lookup returned is the normal way to use this port, so nearly every
cache fill is a script. On the runtime's SCRIPT lane those fills shared eight connections per pod
with idempotency claims, rate-limit evaluations and lease calls, and a timed-out lease is discarded
rather than pooled, so each timeout also cost a reconnect. A slow Redis or a burst of misses after a
region invalidation filled the lane with cache work; an idempotency claim then waited out its
timeout and answered `UNAVAILABLE`, and a rate-limit call fell to its failure policy. The cache is a
speed feature that may degrade; idempotency, rate limiting and leases are correctness features that
must not (review 2026-11-03 README §4, RR-04-02).
So `CacheScriptLane.CACHE_SCRIPT` is a lane of its own — the same scripting account, its own
connections, its own ceiling (`app.capabilities.cache.maximum-script-connections`, default 4). A cache
that runs out of them degrades (`DEGRADED_UNAVAILABLE`, nothing sent) without touching the SCRIPT
lane. Alternatives not taken: a per-capability permit cap inside the shared SCRIPT lane (still one
pool of connections and one reconnect cost), and running the cache programs on the REGULAR lane
(grants the application account `EVALSHA`, which the account split exists to withhold).
Whether the split holds under load is visible per lane. With a `MeterRegistry` in the context,
`redis-runtime` registers these for every lane it pools — `REGULAR`, `SCRIPT`, `CACHE_SCRIPT` and the
typed SDK's — tagged `lane=<name>` (`RedisLaneMetrics`; rows in `docs/registries/metrics.yaml`):
| Meter | Type | Read it as |
| --- | --- | --- |
| `redis.lane.acquire.wait.seconds` (`outcome` = `admitted`/`refused`) | timer, histogram | how long a borrow queued for a connection; a growing admitted p99 comes before refusals |
| `redis.lane.acquire.refused.total` (`reason` = `saturated`/`closing`/`interrupted`/`connect_failed`) | counter | a borrow that got no connection, nothing sent; `saturated` on `SCRIPT` is the starvation this lane exists to prevent |
| `redis.lane.connections.opened.total` | counter | new connections; after warm-up every one is a reconnect |
| `redis.lane.connections.discarded.total` (`reason` = `invalidated`/`broken`) | counter | connections closed instead of pooled — `invalidated` is a timed-out or failed call (the lease is not reused), and is what the reconnects replace |
| `redis.lane.connections.active`, `redis.lane.connections.pending` | gauge | connections lent out now; borrows waiting now |
Every counter series exists from startup at zero, so an alert on "refusals above zero" sees the first
one. A lane refusal is also logged, per call, by `RedisSemanticFailureLogger`.
### Key invalidation
@@ -300,10 +336,21 @@ Useful focused checks:
`redisContainerContractTest` runs the cache-region and lease port contracts (the ones the in-memory
fakes pass) and the live semantic-port tests against a Redis it starts itself, under the standalone
topology's ACL file. It is in `integrationCheck` and `mainCheck`; the image is
topology's ACL file. It is in `integrationCheck` (integration-main.yml's `integration-check` job), and
only there: listing it in `mainCheck` as well ran it twice on every push to main. The image is
`config/testing/container-images.properties` `redis`.
Live Redis/topology qualification remains separate from fast unit tests. A release claim for a
specific Redis topology or ACL layout requires the corresponding live qualification lane; the
in-memory gateway is for deterministic contract tests, not evidence that an external Redis
deployment accepts the same network/TLS/ACL configuration.
## 아직 내보내지 않는 지표
`redis.capability.*` 여섯 개(operations.total, duration.seconds, admission.rejected.total,
inflight.total, readiness.total, lifecycle.drain.total)는 redis-production-capability 설계(Task 16)에
있었지만 어떤 코드도 내보내지 않았다. 레지스트리(`docs/registries/metrics.yaml`)와 incident runbook이 있는
것처럼 적고 있어서, 2026-11-03 재리뷰 수정에서 레지스트리에서 빼고 runbook을 지금 내보내는
`redis.lane.*`(redis-runtime, lane별 대기·거절·연결) 기준으로 고쳤다. capability 단위 지표가 필요해지면
operation 결과를 아는 semantic adapter(이 leaf)에서 내보내고, 그때 레지스트리에 다시 넣는다.
+13 -7
View File
@@ -48,7 +48,8 @@ dependencies {
// command surface the catalog exists to prevent. Zero imports.
// io.micrometer:micrometer-core — observation leaves this leaf as RedisObservation through a
// Consumer sink; binding it to a meter registry belongs to the composition root, not here.
// Zero imports.
// Zero imports. The connection lanes' meters, CACHE_SCRIPT's included, are redis-runtime's
// (RedisLaneMetrics): the lanes are pooled there, so that is where they are counted.
}
// Redis topology qualification is opt-in and owned by `ca.redis-topology-lane`: mode/property
@@ -69,8 +70,12 @@ redisTopologyLane {
// adapters' port contracts reached a real server in no stage, and the fake and the adapter could hide
// the same defect (05-02, 05-03). The image is config/testing/container-images.properties' `redis`,
// and the ACL file is infra/redis-sdk's own, so the accounts and their rights are the standalone
// topology's. Integration category: integrationCheck runs it, and so does this leaf's mainCheck, like
// the HTTP client's Toxiproxy lane. Sentinel, cluster and TLS remain the topology lane's.
// topology's. Integration category, so integrationCheck runs it — integration-main.yml's
// `integration-check` job, on every push to main and nightly. It is deliberately not in this leaf's
// mainCheck as well: `main-check` and `integration-check` are two jobs of the same workflow run, so a
// lane in both started a second Redis container and ran every live contract twice per push without
// covering anything the first run did not (found while fixing review 2026-11-03). Sentinel, cluster
// and TLS remain the topology lane's.
def redisAclDirectory = rootProject.projectDir.parentFile.toPath().resolve('infra/redis-sdk/acl').toFile()
strictTestLanes {
lane('redisContainerContractTest') {
@@ -78,15 +83,16 @@ strictTestLanes {
description = 'Runs the Redis semantic adapters\' live contracts against a Testcontainers Redis on the pinned image.'
requires 'dev.caskeleton.adapter.outbound.cache.redis.cache.LiveRedisCacheRegionContractTest',
'dev.caskeleton.adapter.outbound.cache.redis.lease.LiveRedisDistributedLeaseContractTest',
// The fake carries a Java copy of the idempotency Lua state machine; only a real
// server proves the two agree (05-02 was the fake copying the Lua's bug). It ran on
// the explicit-only topology lane alone, so no gate ever ran it (RR-04-05).
'dev.caskeleton.adapter.outbound.cache.redis.idempotency.LiveRedisIdempotencyStoreContractTest',
'dev.caskeleton.adapter.outbound.cache.redis.LiveRedisSemanticPortsTest',
'dev.caskeleton.adapter.outbound.cache.redis.capability.LiveRedisSemanticScriptAclProbeTest'
rejectSkipped = true
systemProperty 'redis.container.acl-file', new File(redisAclDirectory, 'all-accounts.acl').absolutePath
systemProperty 'redis.container.acl-file', new File(redisAclDirectory, 'lane-fixture-accounts.acl').absolutePath
inputDirectory redisAclDirectory
shouldRunAfter 'test'
}
}
tasks.named('mainCheck') {
dependsOn 'redisContainerContractTest'
}
@@ -50,7 +50,7 @@ io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnota
io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor
io.lettuce:lettuce-core:6.8.2.RELEASE=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-commons:1.16.7=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-core:1.16.7=testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-core:1.16.7=runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-jakarta9:1.16.7=testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-observation:1.16.7=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.netty:netty-buffer:4.2.17.Final=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
@@ -103,7 +103,7 @@ org.codehaus.plexus:plexus-container-default:2.1.0=checkstyle
org.codehaus.plexus:plexus-utils:3.3.0=checkstyle
org.dom4j:dom4j:2.2.0=spotbugs
org.hamcrest:hamcrest:3.0=testCompileClasspath,testRuntimeClasspath
org.hdrhistogram:HdrHistogram:2.2.2=testRuntimeClasspath
org.hdrhistogram:HdrHistogram:2.2.2=runtimeClasspath,testRuntimeClasspath
org.javassist:javassist:3.28.0-GA=checkstyle
org.jetbrains:annotations:17.0.0=testCompileClasspath,testRuntimeClasspath
org.jspecify:jspecify:1.0.1=annotationProcessor,checkstyle,compileClasspath,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
@@ -116,7 +116,7 @@ org.junit.platform:junit-platform-engine:6.0.3=testRuntimeClasspath
org.junit.platform:junit-platform-launcher:6.0.3=testRuntimeClasspath
org.junit:junit-bom:6.0.3=testCompileClasspath,testRuntimeClasspath
org.junit:junit-bom:6.1.0=spotbugs
org.latencyutils:LatencyUtils:2.0.3=testRuntimeClasspath
org.latencyutils:LatencyUtils:2.0.3=runtimeClasspath,testRuntimeClasspath
org.mockito:mockito-core:5.20.0=mockitoAgent,testCompileClasspath,testRuntimeClasspath
org.mockito:mockito-junit-jupiter:5.20.0=testCompileClasspath,testRuntimeClasspath
org.objenesis:objenesis:3.3=testRuntimeClasspath
@@ -0,0 +1,34 @@
package dev.caskeleton.adapter.outbound.cache.redis.cache;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisAccount;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisLane;
/**
* The cache's own script lane, so a cache's load cannot starve a correctness capability (review
* 2026-11-03 README §4, RR-04-02).
*
* <p>Every conditional cache write is a registered program, and a registered program needs the
* scripting account. On the runtime's SCRIPT lane those writes shared eight connections with
* idempotency claims, rate-limit evaluations and lease calls — and the normal way to use the cache
* (record with the condition the lookup returned) makes every cache fill a script. A slow Redis or
* a burst of misses after a region invalidation then held those connections with cache work while
* an idempotency claim waited out its timeout and failed. The cache is a speed feature and may
* degrade; idempotency and leases are correctness features and must not. They get separate
* connection budgets.
*
* <p>Same account as the SCRIPT lane: the grant is the same, only the connections and their ceiling
* ({@code app.capabilities.cache.maximum-script-connections}) are the cache's. Alternatives that
* were not taken: a per-capability permit cap inside the shared SCRIPT lane would still share the
* connections and their reconnect cost after a discarded lease; running cache programs on the
* REGULAR lane would grant the application account {@code EVALSHA}, which the account split exists
* to withhold.
*/
public enum CacheScriptLane implements RedisLane {
/** The cache region's conditional and revision-ordered writes. */
CACHE_SCRIPT;
@Override
public RedisAccount account() {
return RedisAccount.ADVANCED;
}
}
@@ -68,8 +68,10 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
private final SemanticRedisCall call;
/**
* The conditional writes, which are registered programs and so run on the SCRIPT lane: the
* application account the REGULAR lane authenticates as is not granted {@code EVALSHA}.
* The conditional writes, which are registered programs and so run on the scripting account: the
* application account the REGULAR lane authenticates as is not granted {@code EVALSHA}. They run
* on the cache's own lane ({@link CacheScriptLane}), not the runtime's SCRIPT lane, so a cache
* under load cannot hold the connections idempotency, rate limiting and leases need (RR-04-02).
*/
private final SemanticRedisCall scriptCall;
@@ -133,7 +135,7 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
Objects.requireNonNull(commandTimeout, "command timeout must be non-null"),
"cache");
this.scriptCall =
new SemanticRedisCall(owner, RedisConnectionKind.SCRIPT, commandTimeout, "cache");
new SemanticRedisCall(owner, CacheScriptLane.CACHE_SCRIPT, commandTimeout, "cache");
this.keys = Objects.requireNonNull(keys, "keys must be non-null");
this.keyDigest = Objects.requireNonNull(keyDigest, "key digest must be non-null");
this.encoder = Objects.requireNonNull(encoder, "encoder must be non-null");
@@ -1,6 +1,8 @@
package dev.caskeleton.adapter.outbound.cache.redis.capability;
import dev.caskeleton.adapter.outbound.cache.redis.cache.CacheScriptLane;
import dev.caskeleton.adapter.outbound.cache.redis.cache.RedisCacheRegionAdapter;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisLaneContribution;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisRuntimeSettings;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisRuntimeOwner;
import dev.caskeleton.adapter.outbound.support.CapabilitySelectors;
@@ -56,4 +58,24 @@ class RedisCacheConfig {
cache.getNegativeTtl(),
cache.getCommandTimeout());
}
/**
* The cache's own script lane and its connection ceiling.
*
* <p>The runtime pools a contributed lane like its own SCRIPT lane, on the same scripting
* account, but with connections and a ceiling of its own: the cache's conditional writes no
* longer compete with idempotency claims, rate-limit evaluations and lease calls for the SCRIPT
* lane (review 2026-11-03 README §4, RR-04-02).
*
* @param capabilities the capability settings
* @return the lane contribution
*/
@Bean
@ConditionalOnProperty(name = CapabilitySelectors.CACHE_DEFAULT_BINDING, havingValue = "redis")
RedisLaneContribution redisCacheScriptLane(RedisCapabilitySettings capabilities) {
RedisCapabilitySettings.Cache cache = capabilities.getCache();
cache.validate();
return new RedisLaneContribution(
CacheScriptLane.CACHE_SCRIPT, cache.getMaximumScriptConnections());
}
}
@@ -86,6 +86,7 @@ public class RedisCapabilitySettings {
private Duration positiveHardTtl = Duration.ofMinutes(5);
private Duration negativeTtl = Duration.ofSeconds(10);
private Duration minimumHardTtl = Duration.ofSeconds(1);
private int maximumScriptConnections = 4;
/**
* Reports whether the default binding selected Redis.
@@ -119,6 +120,10 @@ public class RedisCapabilitySettings {
if (keyVersion < 1) {
throw new CapabilityStartupValidationException("the cache key version must be positive");
}
if (maximumScriptConnections < 1) {
throw new CapabilityStartupValidationException(
"the cache script lane needs at least one connection");
}
}
public Map<String, String> getBindings() {
@@ -192,6 +197,23 @@ public class RedisCapabilitySettings {
public void setMinimumHardTtl(Duration minimumHardTtl) {
this.minimumHardTtl = minimumHardTtl;
}
/**
* The ceiling on the cache's own script-lane connections.
*
* <p>Separate from {@code app.redis.capacity.maximum-script-connections}, which idempotency,
* rate limiting and leases share: a cache is a speed feature and may run out of connections and
* degrade; it must not take them from the correctness features (review 2026-11-03 README §4).
*
* @return the ceiling, at least 1
*/
public int getMaximumScriptConnections() {
return maximumScriptConnections;
}
public void setMaximumScriptConnections(int maximumScriptConnections) {
this.maximumScriptConnections = maximumScriptConnections;
}
}
/** Edge rate-limit provider composition. */
@@ -28,11 +28,12 @@ import org.springframework.beans.factory.SmartInitializingSingleton;
* {@code {1}} only when the value it wrote reads back.
*
* <p>A cache-only deployment is probed too: its conditional writes and revision-ordered writes are
* registered programs on the same lane, and without the permission every one of them would come
* registered programs on the same account, and without the permission every one of them would come
* back degraded while plain reads looked healthy. Its probe runs only what the cache programs call
* ({@code GET}, {@code SET}), and a Redis it cannot reach at startup is logged, not fatal — a cache
* outage degrades, it never stops a pod from starting. A server that answers and refuses the
* scripts is a configuration error and fails startup as it does for the correctness roles.
* ({@code GET}, {@code SET}). Only a server that answers and refuses the scripts — an ACL refusal,
* {@code NOSCRIPT}, or a reply other than the value the probe wrote — is a configuration error and
* fails startup as it does for the correctness roles. Every other failure is a cache outage and is
* logged, not fatal: a cache outage degrades, it never stops a pod from starting.
*/
final class RedisSemanticScriptAclProbe implements SmartInitializingSingleton {
@@ -89,7 +90,7 @@ final class RedisSemanticScriptAclProbe implements SmartInitializingSingleton {
try {
probe(CACHE_PROBE_SCRIPT);
} catch (CapabilityStartupValidationException failure) {
if (!unreachable(failure.getCause())) {
if (refusedByTheServer(failure.getCause())) {
throw new CapabilityStartupValidationException(
"Redis cache is selected but its account cannot run the cache's conditional-write"
+ " scripts (SCRIPT LOAD/EVALSHA, GET, SET); every conditional record would"
@@ -98,15 +99,49 @@ final class RedisSemanticScriptAclProbe implements SmartInitializingSingleton {
failure.getCause());
}
LOG.warn(
"Redis cache is selected but Redis was unreachable at startup; the cache script"
+ " permission is unverified and the cache runs degraded until Redis answers",
"Redis cache is selected but the startup script probe could not complete; the cache"
+ " script permission is unverified and the cache runs degraded until Redis answers",
failure.getCause());
}
}
}
private static boolean unreachable(Throwable cause) {
return cause instanceof RedisLaneUnavailableException || cause instanceof TimeoutException;
/**
* Whether the cache probe failed because the server refused the cache's scripts.
*
* <p>"Refused" is a statement about the account, and only three answers make it: an ACL refusal
* ({@code NOPERM}, or a script's inner command denied), {@code NOSCRIPT} for the program the same
* connection had just loaded, and a reply other than the value the probe wrote. The check used to
* run the other way — only a lane refusal or a timeout counted as "unreachable" and everything
* else was a configuration error — so a connection that dropped mid-probe, a restarting Redis
* answering {@code LOADING}, or a busy one answering {@code BUSY} stopped a cache-only pod from
* starting, against this leaf's promise that a cache outage never does (review 2026-11-03
* RR-04-07).
*
* @param cause the probe's failure
* @return {@code true} when the account, not the server's availability, is at fault
*/
private static boolean refusedByTheServer(Throwable cause) {
for (Throwable current = cause;
current != null;
current = current.getCause() == current ? null : current.getCause()) {
if (current instanceof CapabilityStartupValidationException) {
// The probe ran and read back something other than what it wrote.
return true;
}
if (current instanceof RedisLaneUnavailableException || current instanceof TimeoutException) {
return false;
}
String message = current.getMessage();
if (message != null
&& (message.startsWith("NOPERM")
|| message.startsWith("NOSCRIPT")
|| message.contains("no permissions")
|| message.contains("ACL failure"))) {
return true;
}
}
return false;
}
private void probe(byte[] script) {
@@ -141,6 +141,12 @@ public final class IdempotencyScripts {
* it started from and the lease it asked for. A retry from that revision with the same lease is
* the same renewal; with a different lease it is a different request, because confirming it would
* leave the caller believing it holds the record for a lease the record does not carry.
*
* <p>The key's own TTL is stretched to cover the new lease. Claim sets it to the replay TTL,
* which may be shorter than a lease renewed past it; the key then expired while the holder was
* still executing, and the next request got a fresh claim and ran the side effect again (review
* 2026-11-03 RR-04-08). The TTL is only ever lengthened here — completion sets the replay TTL
* again.
*/
private static final String RENEW =
"""
@@ -177,6 +183,10 @@ public final class IdempotencyScripts {
redis.call('HSET', KEYS[1],
'leaseUntil', nowMillis + tonumber(ARGV[4]), 'rev', rev + 1,
'renewedFromRev', ARGV[2], 'renewedLease', ARGV[4])
local ttl = redis.call('PTTL', KEYS[1])
if ttl >= 0 and ttl < tonumber(ARGV[4]) then
redis.call('PEXPIRE', KEYS[1], ARGV[4])
end
return {'APPLIED', attempt, rev + 1, owner, '', ''}
""";
@@ -39,6 +39,10 @@ app:
# Floor on the hard TTL. Below it, entries expire faster than the round trip that wrote them:
# every read misses and every miss writes.
minimum-hard-ttl: ${APP_CACHE_REDIS_MINIMUM_HARD_TTL:1s}
# The cache's own script-lane connections, apart from the SCRIPT lane that idempotency, rate
# limiting and leases share (app.redis.capacity.maximum-script-connections). A cache under
# load runs out of these and degrades; it cannot take the correctness features' connections.
maximum-script-connections: ${APP_CACHE_REDIS_MAXIMUM_SCRIPT_CONNECTIONS:4}
idempotency:
# Redis store tuning only; the provider selector and lifecycle policy stay with the
# application's idempotency executor.
@@ -75,15 +79,8 @@ app:
maximum-cost: ${APP_RATE_LIMIT_MAXIMUM_COST:10}
cleanup-grace: ${APP_RATE_LIMIT_CLEANUP_GRACE:5s}
# redis-runtime 의 redisOptional 은 Redis 에 닿지 못하면 DEGRADED 를 보고한다. Boot 기본 순서
# (DOWN, OUT_OF_SERVICE, UP, UNKNOWN)에 없는 상태는 DOWN 보다 앞에 정렬돼, 다른 contributor 가 DOWN
# 이어도 전체 health 가 HTTP 200 DEGRADED 로 보였다(리뷰 05-23e). DEGRADED 를 DOWN 뒤, UP 앞에 둔다.
# 이 키를 정하는 module-defaults 는 이 파일 하나뿐이다. 서비스가 순서를 바꾸면 DEGRADED 를 넣어야 한다.
management:
endpoint:
health:
status:
order: DOWN,OUT_OF_SERVICE,DEGRADED,UP,UNKNOWN
# DEGRADED 상태 순서(management.endpoint.health.status.order)는 그 상태를 내는 redis-runtime 이
# 싣는다(config/module-defaults/redis-runtime.yml, 리뷰 2026-11-03 RR-04-10).
# /actuator/info 에 싣는 이 모듈의 상태(management.info.env). compose smoke 가 lane 이 실제로 켠
# capability 를 이 값으로 확인한다. 위 스위치를 해석한 값이고, 기본값은 RedisCapabilityConfig 의
@@ -1,5 +1,6 @@
package dev.caskeleton.adapter.outbound.cache.redis;
import dev.caskeleton.adapter.outbound.cache.redis.cache.CacheScriptLane;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.RedisTopologyEndpoint;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisCredentialResolver.RedisCredentials;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisRuntimeSettings;
@@ -71,6 +72,9 @@ public final class LiveRedisRuntime {
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
limits.put(kind, 4);
}
// The cache region's own script lane, which the composition root contributes when the cache
// binding is redis (RedisCacheConfig).
limits.put(CacheScriptLane.CACHE_SCRIPT, 4);
return new RedisRuntimeOwner(
new RedisTopologyClientFactory(
settings,
@@ -28,7 +28,7 @@ final class RedisTestContainer {
static final String ACL_FILE_PROPERTY = "redis.container.acl-file";
private static final String IMAGE_PROPERTY = "ca.test.image.redis";
private static final String ACL_IN_CONTAINER = "/etc/redis/acl/all-accounts.acl";
private static final String ACL_IN_CONTAINER = "/etc/redis/acl/lane-fixture-accounts.acl";
private static final int PORT = 6379;
private static GenericContainer<?> server;
@@ -60,8 +60,17 @@ class RedisCacheRegionAdapterTest {
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
limits.put(kind, 4);
}
return new RedisCacheRegionAdapter<>(
limits.put(CacheScriptLane.CACHE_SCRIPT, 4);
return cache(
now,
new RedisRuntimeOwner(client, limits, Duration.ofSeconds(1), Duration.ofSeconds(1)),
cacheKeys);
}
private RedisCacheRegionAdapter<String, String> cache(
Instant now, RedisRuntimeOwner owner, RedisCacheRegionAdapter.CacheKeys cacheKeys) {
return new RedisCacheRegionAdapter<>(
owner,
cacheKeys,
key -> "digest-" + key,
value -> value.getBytes(StandardCharsets.UTF_8),
@@ -428,6 +437,31 @@ class RedisCacheRegionAdapterTest {
.hasMessageContaining("ONLY_IF_SOURCE_REVISION_NEWER");
}
@Test
@DisplayName("멱등성·lease가 SCRIPT lane을 모두 잡고 있어도 캐시 조건부 기록은 캐시 전용 lane에서 기록된다")
void aConditionalCacheWriteDoesNotWaitForTheCorrectnessScriptLane() {
// given
Map<RedisLane, Integer> limits = new java.util.LinkedHashMap<>();
limits.put(RedisConnectionKind.REGULAR, 4);
limits.put(RedisConnectionKind.SCRIPT, 1);
limits.put(CacheScriptLane.CACHE_SCRIPT, 1);
try (RedisRuntimeOwner owner =
new RedisRuntimeOwner(
StubRedisRuntimeClient.serving(gateway),
limits,
Duration.ofMillis(100),
Duration.ofSeconds(1));
var heldByAnIdempotencyClaim = owner.borrow(RedisConnectionKind.SCRIPT)) {
RedisCacheRegionAdapter<String, String> cache = cache(T0, owner, keys);
// when
CacheRecordOutcome outcome = cache.record("order-1", "payload", newerThan("2"));
// then
assertThat(outcome).isEqualTo(CacheRecordOutcome.RECORDED);
}
}
@Test
@DisplayName("an unreachable Redis degrades: the cache is bypassed, nothing fails")
void anUnreachableRedisDegrades() {
@@ -23,6 +23,7 @@ class RedisCacheRegionContractTest extends CacheRegionContract {
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
limits.put(kind, 4);
}
limits.put(CacheScriptLane.CACHE_SCRIPT, 4);
return new RedisCacheRegionAdapter<>(
new RedisRuntimeOwner(
StubRedisRuntimeClient.serving(new InMemorySemanticRedisCommands()),
@@ -3,6 +3,9 @@ package dev.caskeleton.adapter.outbound.cache.redis.capability;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.Mockito.mock;
import dev.caskeleton.adapter.outbound.cache.redis.cache.CacheScriptLane;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisLaneContribution;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisLaneMetrics;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisRuntimeAutoConfiguration;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisRuntimeOwner;
import dev.caskeleton.adapter.outbound.support.startup.StartupFailureException;
@@ -11,6 +14,8 @@ import dev.caskeleton.application.idempotency.IdempotencyStorePort;
import dev.caskeleton.application.lease.DistributedLeasePort;
import dev.caskeleton.application.secret.SecretSourcePort;
import dev.caskeleton.shared.ratelimit.EdgeRateLimitPort;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.simple.SimpleMeterRegistry;
import java.time.Clock;
import java.time.Duration;
import java.util.Arrays;
@@ -94,6 +99,66 @@ class RedisCapabilityCompositionTest {
});
}
@Test
@DisplayName("Redis 캐시 binding은 설정한 상한으로 캐시 전용 script lane을 SCRIPT lane과 따로 기여한다")
void theCacheBindingContributesItsOwnScriptLane() {
runner
.withPropertyValues(
"app.capabilities.cache.bindings.default=redis",
"app.capabilities.cache.maximum-script-connections=3")
.run(
context -> {
// when
List<RedisLaneContribution> contributions =
List.copyOf(context.getBeansOfType(RedisLaneContribution.class).values());
// then
assertThat(context).hasNotFailed();
assertThat(contributions)
.containsExactly(new RedisLaneContribution(CacheScriptLane.CACHE_SCRIPT, 3));
});
}
@Test
@DisplayName("Redis 캐시 binding의 캐시 전용 lane은 SCRIPT lane과 따로 대기·거절·재연결 지표를 낸다")
void theCacheScriptLaneHasMetersOfItsOwn() {
runner
.withPropertyValues("app.capabilities.cache.bindings.default=redis")
.withBean(MeterRegistry.class, SimpleMeterRegistry::new)
.run(
context -> {
// when
MeterRegistry registry = context.getBean(MeterRegistry.class);
// then
assertThat(context).hasNotFailed();
for (String lane : List.of(CacheScriptLane.CACHE_SCRIPT.name(), "SCRIPT")) {
assertThat(
registry
.find(RedisLaneMetrics.CONNECTIONS_PENDING)
.tag("lane", lane)
.gauge())
.as("waiting borrows of lane %s", lane)
.isNotNull();
assertThat(
registry
.find(RedisLaneMetrics.ACQUIRE_REFUSED)
.tag("lane", lane)
.tag("reason", "saturated")
.counter())
.as("refusals of lane %s", lane)
.isNotNull();
assertThat(
registry
.find(RedisLaneMetrics.CONNECTIONS_OPENED)
.tag("lane", lane)
.counter())
.as("connections opened on lane %s", lane)
.isNotNull();
}
});
}
@Test
@DisplayName("a rate limiter with no configured policy is refused at startup")
void aRateLimiterWithoutPoliciesIsRefused() {
@@ -191,6 +191,85 @@ class RedisSemanticScriptAclProbeTest {
assertThat(failure).isNull();
}
@ParameterizedTest
@ValueSource(
strings = {
"LOADING Redis is loading the dataset in memory",
"Connection closed",
"BUSY Redis is busy running a script"
})
@DisplayName("cache만 고른 배포는 probe 도중 연결이 끊기거나 Redis가 아직 준비 중이어도 권한 오류로 보지 않고 기동한다")
void aCacheOnlyDeploymentStartsWhenTheProbeIsInterruptedByATransientFailure(String reply) {
// given
RedisCapabilitySettings capabilities = new RedisCapabilitySettings();
capabilities.getCache().getBindings().put("default", "redis");
RedisSemanticScriptAclProbe probe =
new RedisSemanticScriptAclProbe(
owner(gatewayFailingLoadWith(reply), new AtomicInteger()),
capabilities,
"prod:svc:shared:probe".getBytes(StandardCharsets.UTF_8),
Duration.ofSeconds(1));
// when
Throwable failure = catchThrowable(probe::afterSingletonsInstantiated);
// then
assertThat(failure).isNull();
}
@Test
@DisplayName("cache만 고른 배포도 probe가 기대와 다른 답을 받으면 기동에서 실패한다")
void aCacheOnlyDeploymentWhoseProbeReadsBackTheWrongValueFailsStartup() {
// given
RedisCapabilitySettings capabilities = new RedisCapabilitySettings();
capabilities.getCache().getBindings().put("default", "redis");
RedisSemanticScriptAclProbe probe =
new RedisSemanticScriptAclProbe(
owner(gatewayAnswering(List.of(0L)), new AtomicInteger()),
capabilities,
"prod:svc:shared:probe".getBytes(StandardCharsets.UTF_8),
Duration.ofSeconds(1));
// when
Throwable failure = catchThrowable(probe::afterSingletonsInstantiated);
// then
assertThat(failure)
.isInstanceOf(CapabilityStartupValidationException.class)
.hasMessageContaining("cache");
}
private static SemanticRedisCommands gatewayFailingLoadWith(String reply) {
return (SemanticRedisCommands)
Proxy.newProxyInstance(
SemanticRedisCommands.class.getClassLoader(),
new Class<?>[] {SemanticRedisCommands.class},
(proxy, method, arguments) ->
switch (method.getName()) {
case "loadScript" -> CompletableFuture.failedFuture(new RuntimeException(reply));
case "toString" -> "RedisSemanticScriptAclProbeTestGateway";
default ->
throw new UnsupportedOperationException(
"unexpected gateway method: " + method.getName());
});
}
private static SemanticRedisCommands gatewayAnswering(List<Object> reply) {
return (SemanticRedisCommands)
Proxy.newProxyInstance(
SemanticRedisCommands.class.getClassLoader(),
new Class<?>[] {SemanticRedisCommands.class},
(proxy, method, arguments) ->
switch (method.getName()) {
case "loadScript" -> CompletableFuture.completedFuture("probe-digest");
case "evaluateRegisteredForList" -> CompletableFuture.completedFuture(reply);
case "toString" -> "RedisSemanticScriptAclProbeTestGateway";
default ->
throw new UnsupportedOperationException(
"unexpected gateway method: " + method.getName());
});
}
private static RedisRuntimeOwner owner(SemanticRedisCommands gateway, AtomicInteger opened) {
RedisRuntimeClient client =
new RedisRuntimeClient() {
@@ -71,26 +71,15 @@ public final class RoleBasedFileAccessPolicy implements FileAccessPolicyPort {
if (permitted.stream().noneMatch(subject.roles()::contains)) {
throw denied(FileserverErrorCode.ACCESS_DENIED, "subject does not hold a permitted role");
}
if (descriptor.isPresent() && !isOwnedBy(descriptor.get(), subject)) {
// Ownership is FileAccessSubject.owns, the rule the port owns (owner decision 14): a file with
// no recorded owner predates ownership and belongs to nobody on the data plane. An admin role
// does not bypass it — acting on somebody else's file is what the audited management plane is
// for, and its operations carry no descriptor.
if (descriptor.isPresent() && !subject.owns(descriptor.get())) {
throw denied(FileserverErrorCode.ACCESS_DENIED, "subject does not own the file");
}
}
/**
* Whether the file was created by this caller, in this caller's tenant.
*
* <p>A file without a recorded owner predates ownership and belongs to nobody on the data plane.
* An admin role does not bypass this: acting on somebody else's file is what the audited
* management plane is for, and its operations carry no descriptor.
*/
private static boolean isOwnedBy(FileDescriptor descriptor, FileAccessSubject subject) {
return descriptor
.owner()
.filter(owner -> owner.principalId().equals(subject.principalId()))
.filter(owner -> owner.tenantId().equals(subject.tenant()))
.isPresent();
}
private static boolean isAnonymous(FileAccessSubject subject) {
return subject.principalId().equals(FileAccessSubject.anonymous().principalId());
}
@@ -0,0 +1,31 @@
package dev.caskeleton.adapter.outbound.fileserver.platform.security;
import dev.caskeleton.application.fileserver.api.security.FileAccessPolicyPort;
import dev.caskeleton.application.fileserver.api.security.FileAccessSubject;
import dev.caskeleton.application.fileserver.testkit.FileAccessPolicyPortContract;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
/** The production policy held to the ownership contract its fake is held to (RR-07-06). */
class RoleBasedFileAccessPolicyContractTest extends FileAccessPolicyPortContract {
private static final String READ = "ROLE_FILE_READ";
private static final String WRITE = "ROLE_FILE_WRITE";
private final RoleBasedFileAccessPolicy policy =
new RoleBasedFileAccessPolicy(Set.of(READ), Set.of(WRITE), Set.of("ROLE_FILE_ADMIN"), false);
@Override
protected FileAccessPolicyPort policy() {
return policy;
}
@Override
protected FileAccessSubject permittedSubject(String principalId, Optional<String> tenant) {
return new FileAccessSubject(
principalId,
Set.of(READ, WRITE),
tenant.map(value -> Map.of(FileAccessSubject.TENANT_ATTRIBUTE, value)).orElse(Map.of()));
}
}
+8 -2
View File
@@ -41,7 +41,11 @@ no reactive (`WebClient`/Reactor Netty) surface; it was removed on 2026-09-25 (d
cancels the request at the deadline, the JDK transport bounds its exchange timeout by the time
left, the experimental Jetty HTTP/3 transport aborts the request, and the caller gets
`HttpDeadlineExceededException`. A streaming download's setup (up to the response headers) runs
inside the same deadline; after hand-over each body gap is bounded by `read-idle`.
inside the same deadline; after hand-over each body gap is bounded by `read-idle` and nothing else,
on both transports — the JDK transport arms its own per-read timer for this, because the JDK client
has no socket read timeout. A body that fails after hand-over throws the same stable exceptions as
a buffered call (`HttpResponseTimeoutException` for a stalled body, `HttpResponseTruncatedException`
for a cut one).
- **Evidence.** Each attempt reports `NOT_SENT`, `SENT_NO_RESPONSE`, `RESPONSE_RECEIVED`, or
`PARTIAL_RESPONSE`. `NOT_SENT` is only claimed when a stage failure proves it.
- **Retry.** Never from the HTTP method alone. Idempotency, body replayability, evidence, deadline,
@@ -53,7 +57,9 @@ no reactive (`WebClient`/Reactor Netty) surface; it was removed on 2026-09-25 (d
whether the call throws them or returns them as a result (`RETURN_RESULT`). Any other 4xx (404,
409, 422 …) is a healthy answer, even when it reaches the caller as `HttpRemoteErrorException`.
Window, threshold and open-state wait are per profile under `clients[N].circuit-breaker.*`
(defaults are Resilience4j's; see `docs/httpclient/configuration-reference.md`).
(defaults are Resilience4j's; see `docs/httpclient/configuration-reference.md`). An attempt that
the caller's own deadline (`HttpOperation.deadline()`) ended is not counted at all: the caller ran
out of time, the upstream did not fail. An attempt the profile's `total-call` ended still counts.
- **Resources.** Connections and buffers are reclaimed after success, error, decode failure, size
rejection, and cancellation alike.
- **Secrets.** No URL, query value, path variable, token, cookie, or idempotency key reaches a log,
@@ -21,22 +21,35 @@ import org.springframework.http.client.JdkClientHttpRequestFactory;
* timeout is fixed per factory, so a factory is made per request with {@code min(response-header
* timeout, remaining deadline)} from {@link AttemptDeadline}. The factory only holds references;
* the client and the write executor are shared.
*
* <p>A streaming download cannot use that request. Its body is handed to the caller and may run for
* minutes, but Spring's timer only stops when the body stream is closed, so every JDK download
* longer than the response-header timeout was cut mid-body (review 2026-11-03 RR-04-03). Inside a
* streaming setup scope the factory makes a {@link JdkStreamingClientHttpRequest} instead: the
* JDK's own request timeout bounds the wait for headers, the attempt deadline cancels the exchange
* up to hand-over, and the body is bounded per gap by {@code read-idle}.
*/
final class DeadlineAwareJdkRequestFactory implements ClientHttpRequestFactory {
private final HttpClient client;
private final Executor writeExecutor;
private final Duration exchangeTimeout;
private final Duration readIdle;
DeadlineAwareJdkRequestFactory(HttpClient client, Duration exchangeTimeout) {
DeadlineAwareJdkRequestFactory(HttpClient client, Duration exchangeTimeout, Duration readIdle) {
this.client = Objects.requireNonNull(client, "jdk http client");
this.exchangeTimeout = Objects.requireNonNull(exchangeTimeout, "exchange timeout");
this.readIdle = Objects.requireNonNull(readIdle, "read idle timeout");
// The executor Spring would pick itself, chosen once instead of once per request.
this.writeExecutor = client.executor().orElseGet(SimpleAsyncTaskExecutor::new);
}
@Override
public ClientHttpRequest createRequest(URI uri, HttpMethod httpMethod) throws IOException {
if (AttemptDeadline.boundsSetupOnly()) {
return new JdkStreamingClientHttpRequest(
client, uri, httpMethod, AttemptDeadline.bound(exchangeTimeout), readIdle);
}
JdkClientHttpRequestFactory delegate = new JdkClientHttpRequestFactory(client, writeExecutor);
delegate.setReadTimeout(AttemptDeadline.bound(exchangeTimeout));
return delegate.createRequest(uri, httpMethod);
@@ -62,7 +62,8 @@ public final class JdkBlockingTransportProvider implements BlockingTransportProv
HttpClient client = clientFactory.create(profile, tlsMaterialResolver.apply(profile));
clients.put(new TransportResourceKey(profile.name(), generation), client);
ClientHttpRequestFactory factory =
new DeadlineAwareJdkRequestFactory(client, profile.timeout().responseHeader());
new DeadlineAwareJdkRequestFactory(
client, profile.timeout().responseHeader(), profile.timeout().readIdle());
listener.onRuntimeCreated(profile.name(), ID);
return factory;
}
@@ -0,0 +1,128 @@
package dev.caskeleton.adapter.outbound.httpclient.jdk;
import dev.caskeleton.adapter.outbound.httpclient.transport.AttemptDeadline;
import java.io.IOException;
import java.io.InputStream;
import java.io.InterruptedIOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.Objects;
import java.util.Set;
import java.util.TreeSet;
import java.util.concurrent.CancellationException;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.client.AbstractBufferingClientHttpRequest;
import org.springframework.http.client.ClientHttpResponse;
/**
* A JDK request for a streaming download, whose timeouts end at the response headers (review
* 2026-11-03 RR-04-03).
*
* <p>Three limits, each where the JDK can enforce it:
*
* <ul>
* <li>the JDK's own request timeout bounds the wait for the response headers and stops when they
* arrive;
* <li>the attempt deadline cancels the pending exchange, so connect, send and the header wait
* stay inside the call's total budget;
* <li>after hand-over, {@link ReadIdleInputStream} bounds each gap of the body by {@code
* read-idle} — the JDK client has no socket read timeout of its own.
* </ul>
*
* <p>The request body is buffered: a streaming download sends none or a small one, and the JDK
* needs a body publisher before the exchange starts.
*/
final class JdkStreamingClientHttpRequest extends AbstractBufferingClientHttpRequest {
/** Headers the JDK client refuses to set, as Spring's own JDK request lists them. */
private static final Set<String> RESTRICTED_HEADERS = restrictedHeaders();
private final HttpClient client;
private final URI uri;
private final HttpMethod method;
private final Duration responseHeaderTimeout;
private final Duration readIdle;
JdkStreamingClientHttpRequest(
HttpClient client,
URI uri,
HttpMethod method,
Duration responseHeaderTimeout,
Duration readIdle) {
this.client = Objects.requireNonNull(client, "jdk http client");
this.uri = Objects.requireNonNull(uri, "uri");
this.method = Objects.requireNonNull(method, "method");
this.responseHeaderTimeout =
Objects.requireNonNull(responseHeaderTimeout, "response header timeout");
this.readIdle = Objects.requireNonNull(readIdle, "read idle timeout");
}
@Override
public HttpMethod getMethod() {
return method;
}
@Override
public URI getURI() {
return uri;
}
@Override
protected ClientHttpResponse executeInternal(HttpHeaders headers, byte[] body)
throws IOException {
HttpRequest.Builder request =
HttpRequest.newBuilder(uri)
.timeout(responseHeaderTimeout)
.method(
method.name(),
body.length == 0
? HttpRequest.BodyPublishers.noBody()
: HttpRequest.BodyPublishers.ofByteArray(body));
headers.forEach(
(name, values) -> {
if (!RESTRICTED_HEADERS.contains(name)) {
values.forEach(value -> request.header(name, value));
}
});
CompletableFuture<HttpResponse<InputStream>> pending =
client.sendAsync(request.build(), HttpResponse.BodyHandlers.ofInputStream());
AttemptDeadline.abortOnExpiry(() -> pending.cancel(true));
HttpResponse<InputStream> response = await(pending);
return new JdkStreamingClientHttpResponse(
response, new ReadIdleInputStream(response.body(), readIdle));
}
private static HttpResponse<InputStream> await(
CompletableFuture<HttpResponse<InputStream>> pending) throws IOException {
try {
return pending.get();
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
pending.cancel(true);
InterruptedIOException failure = new InterruptedIOException("streaming setup interrupted");
failure.initCause(interrupted);
throw failure;
} catch (CancellationException cancelled) {
// Only the attempt deadline cancels this future; the gateway names the deadline.
throw new IOException("streaming setup cancelled at the attempt deadline", cancelled);
} catch (ExecutionException failed) {
Throwable cause = failed.getCause();
if (cause instanceof IOException io) {
throw io;
}
throw new IOException("streaming setup failed", cause);
}
}
private static Set<String> restrictedHeaders() {
Set<String> restricted = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
restricted.addAll(Set.of("connection", "content-length", "expect", "host", "upgrade"));
return restricted;
}
}
@@ -0,0 +1,63 @@
package dev.caskeleton.adapter.outbound.httpclient.jdk;
import java.io.IOException;
import java.io.InputStream;
import java.net.http.HttpResponse;
import java.util.Objects;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.HttpStatusCode;
import org.springframework.http.client.ClientHttpResponse;
/** The response of a {@link JdkStreamingClientHttpRequest}, body bounded per gap. */
final class JdkStreamingClientHttpResponse implements ClientHttpResponse {
private final HttpResponse<InputStream> response;
private final InputStream body;
private final HttpHeaders headers = new HttpHeaders();
JdkStreamingClientHttpResponse(HttpResponse<InputStream> response, InputStream body) {
this.response = Objects.requireNonNull(response, "jdk response");
this.body = Objects.requireNonNull(body, "body stream");
// HTTP/2 pseudo-headers (":status") are protocol framing, not response headers.
response
.headers()
.map()
.forEach(
(name, values) -> {
if (!name.startsWith(":")) {
headers.addAll(name, values);
}
});
}
@Override
public HttpStatusCode getStatusCode() {
return HttpStatusCode.valueOf(response.statusCode());
}
@Override
public String getStatusText() {
HttpStatus known = HttpStatus.resolve(response.statusCode());
return known == null ? "" : known.getReasonPhrase();
}
@Override
public HttpHeaders getHeaders() {
return headers;
}
@Override
public InputStream getBody() {
return body;
}
@Override
public void close() {
try {
body.close();
} catch (IOException ignored) {
// Closing releases the exchange; a failure to close has nothing left to release.
}
}
}
@@ -0,0 +1,90 @@
package dev.caskeleton.adapter.outbound.httpclient.jdk;
import java.io.FilterInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.SocketTimeoutException;
import java.time.Duration;
import java.util.Objects;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
/**
* A JDK response body whose every read is bounded by {@code read-idle} (review 2026-11-03
* RR-04-03).
*
* <p>The JDK client has no socket read timeout, so a streamed body that stopped arriving held the
* caller's thread and the connection forever. Each read arms a timer on the JDK's shared delay
* scheduler; when it fires the stream is closed, which wakes the blocked read, and the read reports
* a {@link SocketTimeoutException} — the same exception Apache's socket timeout raises, so one
* classifier reading names both. A read that returns first disarms its timer.
*/
final class ReadIdleInputStream extends FilterInputStream {
@FunctionalInterface
private interface Read {
long run() throws IOException;
}
private final Duration readIdle;
private final AtomicBoolean idled = new AtomicBoolean();
ReadIdleInputStream(InputStream delegate, Duration readIdle) {
super(Objects.requireNonNull(delegate, "body stream"));
this.readIdle = Objects.requireNonNull(readIdle, "read idle timeout");
}
@Override
public int read() throws IOException {
return (int) guarded(super::read);
}
@Override
public int read(byte[] buffer, int offset, int length) throws IOException {
return (int) guarded(() -> super.read(buffer, offset, length));
}
@Override
public long skip(long count) throws IOException {
return guarded(() -> super.skip(count));
}
private long guarded(Read read) throws IOException {
requireNotIdled();
CompletableFuture<Void> timer = new CompletableFuture<>();
CompletableFuture<Void> expiry =
timer.completeOnTimeout(null, readIdle.toNanos(), TimeUnit.NANOSECONDS).thenRun(this::idle);
try {
long result = read.run();
requireNotIdled();
return result;
} catch (IOException failure) {
requireNotIdled();
throw failure;
} finally {
// Completing first skips the idle action and lets the scheduler drop the pending task.
timer.cancel(false);
expiry.cancel(false);
}
}
private void idle() {
idled.set(true);
try {
in.close();
} catch (IOException ignored) {
// The read this wakes reports the timeout; a failed close has nothing more to say.
}
}
/**
* After the timer fired, whatever the read returned — data, end of stream or "closed" — is not
* the rest of the body.
*/
private void requireNotIdled() throws SocketTimeoutException {
if (idled.get()) {
throw new SocketTimeoutException("no response body bytes within the read-idle timeout");
}
}
}
@@ -17,6 +17,13 @@ public interface AttemptCircuitBreaker {
void onError(long durationNanos, Throwable failure);
/**
* Gives back a permission without recording an outcome, for an attempt that says nothing about
* the upstream (an attempt its caller's own deadline ended). Half-open trial slots are returned;
* the window is left as it was.
*/
void releasePermission();
String state();
static AttemptCircuitBreaker alwaysClosed() {
@@ -36,6 +43,11 @@ public interface AttemptCircuitBreaker {
// Nothing to record.
}
@Override
public void releasePermission() {
// Nothing was taken.
}
@Override
public String state() {
return "CLOSED";
@@ -60,6 +72,11 @@ public interface AttemptCircuitBreaker {
delegate.onError(durationNanos, TimeUnit.NANOSECONDS, failure);
}
@Override
public void releasePermission() {
delegate.releasePermission();
}
@Override
public String state() {
return delegate.getState().name();
@@ -7,6 +7,7 @@ import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpFailureMetadata;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRateLimitRejectedException;
import java.util.Objects;
import java.util.Optional;
import java.util.function.BooleanSupplier;
import java.util.function.Function;
import java.util.function.Supplier;
@@ -21,6 +22,11 @@ import java.util.function.Supplier;
* <p>A local rejection (rate limiter or bulkhead) is deliberately <em>not</em> recorded as a
* circuit error: the upstream never saw the request, and counting our own back-pressure as upstream
* failure would open the breaker on a healthy dependency.
*
* <p>An attempt its caller abandoned is not recorded either, for the same reason: an attempt that a
* caller's own deadline ended while the upstream was still answering within the profile's limits is
* the caller running out of time, not the upstream failing (review 2026-11-03 RR-04-01). Its
* permission is given back instead.
*/
public final class AttemptResiliencePipeline {
@@ -79,7 +85,24 @@ public final class AttemptResiliencePipeline {
* @return the attempt's value, whether or not it represents a remote failure
*/
public <T> T execute(AttemptCall<T> call, Function<T, Optional<Throwable>> remoteFailure) {
return execute(call, remoteFailure, () -> false);
}
/**
* Runs one attempt, recording nothing against the upstream when its caller abandoned it.
*
* @param call the attempt, including any redirect hops it follows
* @param remoteFailure returns the failure to record when the value represents an upstream error
* @param abandonedByCaller read after the attempt failed: whether the caller's own deadline ended
* it, in which case the failure is not the upstream's and the breaker records nothing
* @return the attempt's value, whether or not it represents a remote failure
*/
public <T> T execute(
AttemptCall<T> call,
Function<T, Optional<Throwable>> remoteFailure,
BooleanSupplier abandonedByCaller) {
Objects.requireNonNull(remoteFailure, "remote failure classifier");
Objects.requireNonNull(abandonedByCaller, "abandoned-by-caller signal");
if (!circuitBreaker.tryAcquirePermission()) {
rejections.circuitOpen();
throw new HttpCircuitOpenException(
@@ -110,7 +133,11 @@ public final class AttemptResiliencePipeline {
return result;
} catch (Throwable failure) {
releaseAttemptPermits();
circuitBreaker.onError(System.nanoTime() - started, failure);
if (abandonedByCaller.getAsBoolean()) {
circuitBreaker.releasePermission();
} else {
circuitBreaker.onError(System.nanoTime() - started, failure);
}
throw translate(failure);
}
}
@@ -10,11 +10,31 @@ import java.util.Objects;
*
* <p>Everything — pool acquire, DNS, connect, TLS, request write, response read, and every retry
* backoff — is spent inside this single budget.
*
* @param at the instant the call ends
* @param source who set that instant, which decides what an attempt it ends says about the upstream
*/
public record Deadline(Instant at) {
public record Deadline(Instant at, Source source) {
/**
* Who set the instant a call ends at.
*
* <p>The two say different things when they end an attempt. The profile's total-call timeout is
* the operator's statement of how long a healthy upstream may take, so an attempt it ends is the
* upstream's failure. A caller's own deadline is a statement about the caller — it had only that
* much time left — so an attempt it ends says nothing about the upstream (review 2026-11-03
* RR-04-01).
*/
public enum Source {
/** The profile's {@code timeout.total-call}. */
PROFILE_TOTAL_CALL,
/** The caller's {@code HttpOperation.deadline()}, when it is earlier than the profile's. */
CALLER
}
public Deadline {
Objects.requireNonNull(at, "deadline instant");
Objects.requireNonNull(source, "deadline source");
}
public Duration remaining(Clock clock) {
@@ -26,6 +46,11 @@ public record Deadline(Instant at) {
return !clock.instant().isBefore(at);
}
/** Whether the instant is the caller's own budget rather than the profile's total-call. */
public boolean setByCaller() {
return source == Source.CALLER;
}
/** The earlier of two deadlines; a parent budget can only ever shrink a child's. */
public Deadline earliest(Deadline other) {
return at.isBefore(other.at) ? this : other;
@@ -9,6 +9,9 @@ import java.util.Optional;
/**
* Derives the effective deadline from the caller's parent budget and the profile total-call timeout
* (design §15.2): {@code effectiveDeadline = min(parentDeadline, now + totalCallTimeout)}.
*
* <p>The result also records which of the two won. A tie goes to the profile: the profile's limit
* would have ended the call at that instant anyway.
*/
public final class DeadlineCalculator {
@@ -17,6 +20,9 @@ public final class DeadlineCalculator {
Objects.requireNonNull(totalCall, "total call timeout");
Objects.requireNonNull(clock, "clock");
Instant local = clock.instant().plus(totalCall);
return new Deadline(parent.map(value -> value.isBefore(local) ? value : local).orElse(local));
return parent
.filter(value -> value.isBefore(local))
.map(value -> new Deadline(value, Deadline.Source.CALLER))
.orElseGet(() -> new Deadline(local, Deadline.Source.PROFILE_TOTAL_CALL));
}
}

Some files were not shown because too many files have changed in this diff Show More