fix!: 재리뷰가 찾은 결함과 색인에서 빠졌던 발견 5건을 고친다

문제: 성숙도 리뷰 수정을 다시 읽은 재리뷰(Critical 0, 회귀 0)가
Important 27건·Minor 41건을 찾았다. 수정이 새로 만든 결함이 많았다.
@SystemInvoked가 권한 규칙의 탈출구가 됐고, 데모 계정 거부가 정확히
prod일 때만 걸렸고, outbox 순서 번호가 업그레이드 때 뒤로 갔고,
GraphQL union 아래 복잡도가 0이었고, 서블릿 오류 응답이 success:true로
나갔고, 호출자 deadline이 breaker를 열었다. 원래 리뷰 색인이 06 영역
발견 5건(06-10~14)을 빠뜨려 그동안 처리되지 않았다.

방법: 영역별 7묶음이 test-driven-development 절차로 실패 테스트를
먼저 쓰고 고쳤다. 결정은 재리뷰 README 4절에 적었다: 순서 번호는 V21이
기록한 전역 floor부터, 데모 계정은 local·test·demo 허용 목록,
@SystemInvoked bean은 기동 실패, complete 단계 실패는
IDEMPOTENCY_COMPLETION_UNKNOWN(결과 불명), 캐시 조건부 쓰기는 별도
SCRIPT lane, strict lane 정책은 build cache 입력, 끝나지 않는 재시도
행은 원인 코드·지표·PARKED 절차. 레지스트리가 선언만 하고 아무 코드도
내보내지 않던 redis.capability.* 지표 6개는 빼고 runbook을 실제
지표(redis.lane.*) 기준으로 고쳤다.

위험·한계: V21을 V22 없이 직접 고쳤다(어느 배포에도 아직 없음). 이
브랜치의 V21을 적용한 개발 DB는 다시 만들어야 한다. 서블릿 오류
디스패치를 보안 체인이 허용한다. 발행·저장소 포트 시그니처가 바뀌어
파생 서비스 코드가 깨진다. prod에서 fixture-로 시작하는 설정값은
기동을 막는다. httpclient failure-injection lane의 main 단계 중복
실행은 이전 결정대로 남겼다. 실제 CI runner 실행은 확인하지 못했다.

검증: ci, mainCheck, :integrationCheck, -p optional-platforms ci
integrationCheck performanceCheck, :performanceCheck, :releaseCheck
(-PgitRevision), verify-compose-profile-contracts(18 lane) 통과.
첫 실행의 실패 2건(FindSecBugs 1, sample 동시 생성 테스트의 v4 event
id)은 고친 뒤 ci·integrationCheck·mainCheck를 다시 돌려 통과했다.

Refs: docs/reviews/2026-11-03-rereview/README.md 3·4·6·7절
BREAKING CHANGE: outbox 발행 포트와 저장소 포트 시그니처, 순서 번호
첫 값(V21 floor 다음), 샘플 데모 계정 프로필 허용 목록, 새 오류 코드
IDEMPOTENCY_COMPLETION_UNKNOWN, prod의 fixture- 값 거부, 기동 rule은
static·의존 없음이어야 한다.
This commit is contained in:
DongHyeonka
2026-11-03 23:29:35 +09:00
parent 7ef8d0b4cb
commit bbe944bd3c
406 changed files with 13493 additions and 1656 deletions
+6 -1
View File
@@ -77,7 +77,12 @@ instrumentation, data fetcher resolver 세 곳이 같은 bean 을 받는다).
- **client 문구**: shared-contract `ClientErrorMessages`(Category 별 한국어 문구 표 하나, 소유자
결정 11). 예외 메시지는 절대 나가지 않는다.
- **retryable**: 코드의 `retryable()` 이다(category 에서 계산하지 않는다). 플랫폼 자체 코드만 wire
category 기본값을 쓴다.
category 기본값을 쓴다. 그래서 공유 코드의 값을 바꾸면 이 wire 도 바뀐다: 처리 중 중복
`IDEMPOTENT_IN_FLIGHT` 는 2026-11-01 리뷰(02-01)부터 `retryable: true` 다 — 같은 키로 다시 보내면 서버가
판정한다(`docs/web/http-contract.md` §2). `GraphQlWireErrorMapperTest` 가 고정한다(재리뷰 RR-02-06).
- **`REQUEST_TIMEOUT` 은 web 과 이름만 같다.** 여기의 `category` 는 GraphQL wire 전용 6값 어휘(`TIMEOUT`)이고,
web 의 같은 이름 코드는 HTTP 503·operational category `TRANSIENT_DEPENDENCY` 다
(`docs/web/http-contract.md` §4). 둘 다 `retryable: true` 다.
- **body 크기 초과**: `PAYLOAD_TOO_LARGE`(web 과 같은 코드). 문서 모양 거부(`GRAPHQL_DOCUMENT_SHAPE_REJECTED`)는
별도 코드.
@@ -14,6 +14,9 @@ import java.time.Instant;
* <li>{@code GraphQlPlatformWebInterceptor} ends the request with {@code REQUEST_TIMEOUT} when it
* passes, and Spring cancels the execution so no further field is dispatched. A resolver
* already running is not interrupted.
* <li>{@code GraphQlDeadlineInstrumentation} refuses to start a non-trivial resolver once it has
* passed, failing that field with {@code REQUEST_TIMEOUT}; this is what still holds when the
* whole execution runs on the request thread and the interceptor's timer starts too late.
* <li>{@code GraphQlCostBudgetHandler} refuses to start an execution whose deadline has passed.
* <li>{@code GraphQlBatchExecutor} clamps each platform-registered DataLoader batch to what is
* left, and stops between chunks once it is spent.
@@ -121,11 +121,6 @@ public final class GraphQlComplexityCalculator {
}
}
/** The effective page size for a connection request. */
public int effectivePageSize(Integer first, Integer last) {
return first != null ? first : last != null ? last : defaultPageSize;
}
private void requireAllowedPage(Integer requested) {
if (requested == null) {
return;
@@ -15,6 +15,7 @@ import graphql.language.Value;
import graphql.language.VariableDefinition;
import graphql.language.VariableReference;
import graphql.schema.GraphQLArgument;
import graphql.schema.GraphQLCompositeType;
import graphql.schema.GraphQLFieldDefinition;
import graphql.schema.GraphQLFieldsContainer;
import graphql.schema.GraphQLList;
@@ -59,6 +60,13 @@ import java.util.Objects;
* <p>Pricing an omitted argument as one row was the bypass: {@code items(first: 20)} scored 410
* while {@code items} scored 40, and four nested lists added up instead of multiplying.
*
* <p>A selection set is walked whatever kind of composite type it selects from. A union has no
* fields of its own — only {@code __typename} — so its selections are inline fragments and fragment
* spreads, each priced against its type condition. Treating "not a fields container" as "nothing
* below" was the second bypass (re-review 2026-11-03, RR-02-01): {@code search { ... on Post {
* comments(first: 100000) } }} scored 20, and neither the budget nor the page maximum applied under
* any union-typed field.
*
* <p>Traversal is bounded, and fragment cycles are cut by tracking the expansion path. This runs on
* documents that have passed validation, but the bound stays because the scorer is also used from
* the pre-execution path where a hostile document is exactly what it is meant to price.
@@ -143,7 +151,7 @@ public final class GraphQlDocumentComplexityScorer {
private long selectionSetCost(
GraphQLSchema schema,
GraphQLFieldsContainer parent,
GraphQLCompositeType parent,
SelectionSet selectionSet,
Map<String, FragmentDefinition> fragments,
PageVariables variables,
@@ -173,10 +181,10 @@ public final class GraphQlDocumentComplexityScorer {
expansionPath,
listsAreThePage));
} else if (selection instanceof InlineFragment inlineFragment) {
GraphQLFieldsContainer target =
GraphQLCompositeType target =
inlineFragment.getTypeCondition() == null
? parent
: fieldsContainer(schema, inlineFragment.getTypeCondition().getName());
: compositeType(schema, inlineFragment.getTypeCondition().getName());
total =
GraphQlComplexityCalculator.saturatedAdd(
total,
@@ -196,10 +204,10 @@ public final class GraphQlDocumentComplexityScorer {
if (fragment == null || expansionPath.contains(spread.getName())) {
continue;
}
GraphQLFieldsContainer target =
GraphQLCompositeType target =
fragment.getTypeCondition() == null
? parent
: fieldsContainer(schema, fragment.getTypeCondition().getName());
: compositeType(schema, fragment.getTypeCondition().getName());
expansionPath.push(spread.getName());
total =
GraphQlComplexityCalculator.saturatedAdd(
@@ -221,7 +229,7 @@ public final class GraphQlDocumentComplexityScorer {
private long fieldCost(
GraphQLSchema schema,
GraphQLFieldsContainer parent,
GraphQLCompositeType parent,
Field field,
Map<String, FragmentDefinition> fragments,
PageVariables variables,
@@ -235,8 +243,12 @@ public final class GraphQlDocumentComplexityScorer {
return 0;
}
GraphQLFieldDefinition definition = parent.getFieldDefinition(field.getName());
String coordinate = parent.getName() + "." + field.getName();
// A union declares no fields; validation rejects anything but __typename directly under one.
GraphQLFieldDefinition definition =
parent instanceof GraphQLFieldsContainer container
? container.getFieldDefinition(field.getName())
: null;
if (definition == null) {
// Validation rejects an unknown field; price what is there rather than invent a type.
return calculator.fieldCost(coordinate, 0).total();
@@ -250,7 +262,7 @@ public final class GraphQlDocumentComplexityScorer {
long childCost =
selectionSetCost(
schema,
fieldsContainer(definition.getType()),
compositeType(definition.getType()),
field.getSelectionSet(),
fragments,
variables,
@@ -353,14 +365,14 @@ public final class GraphQlDocumentComplexityScorer {
};
}
private static GraphQLFieldsContainer fieldsContainer(GraphQLSchema schema, String typeName) {
GraphQLType type = schema.getType(typeName);
return fieldsContainer(type);
private static GraphQLCompositeType compositeType(GraphQLSchema schema, String typeName) {
return compositeType(schema.getType(typeName));
}
private static GraphQLFieldsContainer fieldsContainer(GraphQLType type) {
/** The object, interface or union a selection set selects from; null for a leaf type. */
private static GraphQLCompositeType compositeType(GraphQLType type) {
GraphQLType unwrapped = unwrap(type);
return unwrapped instanceof GraphQLFieldsContainer container ? container : null;
return unwrapped instanceof GraphQLCompositeType composite ? composite : null;
}
private static GraphQLType unwrap(GraphQLType type) {
@@ -34,11 +34,19 @@ class GraphQlDocumentComplexityScorerTest {
pagedWithDefault(first: Int = 50): [Item!]!
itemConnection(first: Int): ItemConnection
node: Item
allItems: [Item!]!
search: [SearchResult!]!
nodes: [Named!]!
}
type Item {
union SearchResult = Item
interface Named {
name: String!
}
type Item implements Named {
name: String!
children: [Item!]!
grid: [[Item!]!]!
paged(first: Int): [Item!]!
}
type ItemConnection {
edges: [ItemEdge!]!
@@ -48,10 +56,15 @@ class GraphQlDocumentComplexityScorerTest {
}
""";
// Pricing never resolves a runtime type; the resolvers only satisfy the schema generator.
private static final GraphQLSchema SCHEMA =
new SchemaGenerator()
.makeExecutableSchema(
new SchemaParser().parse(SDL), RuntimeWiring.newRuntimeWiring().build());
new SchemaParser().parse(SDL),
RuntimeWiring.newRuntimeWiring()
.type("SearchResult", type -> type.typeResolver(environment -> null))
.type("Named", type -> type.typeResolver(environment -> null))
.build());
private static final long BUDGET = 10_000;
@@ -246,6 +259,87 @@ class GraphQlDocumentComplexityScorerTest {
assertThat(score).isEqualTo(20L + 20L);
}
@Test
@DisplayName("union 필드 아래 inline fragment 의 중첩 list 도 일반 list 와 같이 곱한다")
void nestedListsUnderAUnionFieldMultiplyLikeAPlainList() {
// given
String plain = "query Q { allItems { children { children { children { name } } } } }";
// when
long score =
score(
"query Q { search { ... on Item { children { children { children { name } } } } } }",
Map.of());
// then
assertThat(score).isEqualTo(score(plain, Map.of()));
assertThatThrownBy(() -> calculator.verify(new GraphQlComplexityResult(score), BUDGET))
.isInstanceOf(GraphQlComplexityRejectedException.class);
}
@Test
@DisplayName("union 필드 아래 fragment spread 도 그 타입으로 내려가 곱한다")
void aFragmentSpreadUnderAUnionFieldIsPricedAgainstItsTypeCondition() {
// given
String plain = "query Q { allItems { children { children { children { name } } } } }";
// when
long score =
score(
"query Q { search { ...F } } "
+ "fragment F on Item { children { children { children { name } } } }",
Map.of());
// then
assertThat(score).isEqualTo(score(plain, Map.of()));
}
@Test
@DisplayName("union 필드 아래 page 인자도 상한을 넘으면 page size exceeds maximum 으로 거부된다")
void aPageSizeUnderAUnionFieldIsCheckedAgainstTheMaximum() {
// given
String document = "query Q { search { ... on Item { paged(first: 100000) { name } } } }";
// when
Throwable thrown = catchThrowable(() -> score(document, Map.of()));
// then
assertThat(thrown)
.isInstanceOf(GraphQlComplexityRejectedException.class)
.hasMessageContaining("page size exceeds maximum");
}
@Test
@DisplayName("interface 필드 아래 inline fragment 의 중첩 list 도 일반 list 와 같이 곱한다")
void nestedListsUnderAnInterfaceFieldMultiplyLikeAPlainList() {
// given
String plain = "query Q { allItems { children { children { children { name } } } } }";
// when
long score =
score(
"query Q { nodes { ... on Item { children { children { children { name } } } } } }",
Map.of());
// then
assertThat(score).isEqualTo(score(plain, Map.of()));
}
@Test
@DisplayName("interface 필드 아래 page 인자도 상한을 넘으면 page size exceeds maximum 으로 거부된다")
void aPageSizeUnderAnInterfaceFieldIsCheckedAgainstTheMaximum() {
// given
String document = "query Q { nodes { ... on Item { paged(first: 100000) { name } } } }";
// when
Throwable thrown = catchThrowable(() -> score(document, Map.of()));
// then
assertThat(thrown)
.isInstanceOf(GraphQlComplexityRejectedException.class)
.hasMessageContaining("page size exceeds maximum");
}
private long score(String source, Map<String, Object> variables) {
return score(calculator, source, variables);
}
@@ -55,6 +55,29 @@ class GraphQlWireErrorMapperTest {
.containsExactly("DB_QUERY_CANCELED", false);
}
/**
* The in-flight duplicate's {@code retryable} became {@code true} in the 2026-11-01 review
* (02-01): a resend with the same key is judged by the server, so it is safe. GraphQL carries the
* code's own value, so the change reached this wire too; this pins it (re-review RR-02-06).
*/
@org.junit.jupiter.api.Test
@DisplayName("처리 중 중복(IDEMPOTENT_IN_FLIGHT)은 GraphQL extension 에서도 retryable=true 로 나간다")
void anInFlightDuplicateIsRetryableOnTheGraphQlWire() {
// given
GraphQlWireErrorMapper mapper = GraphQlWireErrorMapper.masking();
RuntimeException failure =
dev.caskeleton.application.idempotency.IdempotencyInFlightException.forScopeDigest(
"digest");
// when
GraphQlWireError error = mapper.map(failure, GraphQlErrorContext.test());
// then
assertThat(error.extensions())
.extracting(extensions -> extensions.get("code"), extensions -> extensions.get("retryable"))
.containsExactly("IDEMPOTENT_IN_FLIGHT", true);
}
@org.junit.jupiter.api.Test
void anIllegalArgumentIsABadParameterRequestError() {
// given
+2 -1
View File
@@ -140,7 +140,8 @@ handler throw, listener callback throw, `responseObserver.onError(...)`, raw
- **와이어 status(coarse)** 는 `GrpcStatusMapper.toStatus(Category)` 가 결정한다(HTTP status 가
coarse 인 것과 동형). 정확한 `code`/`category`/`retryable` 은 `Status` trailer `Metadata`
(`error-code` / `error-category` / `error-retryable`)에 실어 클라이언트가 switch 하게 한다.
`retryable` 은 코드의 값이다. status description 은 shared-contract `ClientErrorMessages` 의
`retryable` 은 코드의 값이다(처리 중 중복 `IDEMPOTENT_IN_FLIGHT` 는 2026-11-01 리뷰 02-01부터 `ABORTED` +
`error-retryable: true` — 같은 키로 다시 보내면 서버가 판정한다. `GrpcStatusMapperTest` 가 고정, 재리뷰 RR-02-06). status description 은 shared-contract `ClientErrorMessages` 의
Category 별 한국어 문구(web·GraphQL 과 같은 표)다.
- **`IllegalArgumentException`** 은 `BAD_PARAMETER`(`INVALID_ARGUMENT`) — 모든 전송이 같다.
- **`ApiErrorCode` 추출**: feature 예외는 `ApiErrorCarrier`(이 모듈이 제공하는 전송-중립 hook)를
@@ -7,6 +7,7 @@ import dev.caskeleton.shared.error.OperationalError;
import io.grpc.Metadata;
import io.grpc.Status;
import java.util.EnumSet;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
@@ -64,4 +65,22 @@ class GrpcStatusMapperTest {
// then
assertThat(retryable).isEqualTo("false");
}
/**
* The in-flight duplicate's {@code retryable} became {@code true} in the 2026-11-01 review
* (02-01). The trailer carries the code's own value, so the change reached the gRPC wire as well:
* status {@code ABORTED} (category {@code CONFLICT}) with {@code error-retryable: true}
* (re-review RR-02-06).
*/
@Test
@DisplayName("처리 중 중복(IDEMPOTENT_IN_FLIGHT)은 ABORTED 이고 trailer 의 retryable 은 true 다")
void anInFlightDuplicateIsAbortedAndRetryableOnTheGrpcWire() {
// when
Metadata trailers = mapper.trailersFor(OperationalError.IDEMPOTENT_IN_FLIGHT);
// then
assertThat(mapper.statusFor(OperationalError.IDEMPOTENT_IN_FLIGHT).getCode())
.isEqualTo(Status.Code.ABORTED);
assertThat(trailers.get(GrpcStatusMapper.RETRYABLE_KEY)).isEqualTo("true");
}
}
+18
View File
@@ -241,6 +241,7 @@ commit-aware response wrapper 계약을 구현한다. 또한 이 모듈은 HTML
| `PreconditionFailedException` | `PRECONDITION_FAILED` | 412 | D15: `If-Match` 불일치 쓰기 = 낙관적 동시성 충돌 → 412 (raw 409/500 금지) |
| `PageValidationException` | `VALIDATION_FAILED` | 400 | D18/D20/D21: 계약 범위 밖 페이지/정렬/필터 파라미터. field + reasonCode 를 details 로 |
| `IdempotencyUnavailableException` | `IDEMPOTENCY_UNAVAILABLE` | 503 (retryable=true) | 조정 저장소가 응답하지 않아 action 전에 거부. carrier 분기 |
| `IdempotencyCompletionUnknownException` | `IDEMPOTENCY_COMPLETION_UNKNOWN` | 503 (retryable=true) | action 이 돈 뒤 결과 기록(complete) 중 저장소가 응답하지 않음. 결과 불명 — 같은 키로 재전송, 새 키 금지(`docs/web/http-contract.md` §2, 재리뷰 RR-02-03). carrier 분기 |
| `IdempotencyRecoveryRequiredException` | `IDEMPOTENCY_RECOVERY_REQUIRED` | 500 (retryable=false) | 이전 시도의 외부 효과 여부를 판단할 수 없음. carrier 분기 |
| `IdempotencyInFlightException` | `IDEMPOTENT_IN_FLIGHT` | 409 (retryable=true, `Retry-After: 2`) | 같은 scope 의 다른 시도가 아직 처리 중(executor 는 기다리지 않는다). 클라이언트 행동: 결과 불명. 같은 `Idempotency-Key`로 `Retry-After` 뒤 재전송. 새 키 금지(`docs/web/http-contract.md` §2). 진단정보(scope digest)는 클라이언트 미도달 |
| `IdempotencyRequestMismatchException` | `IDEMPOTENT_REQUEST_MISMATCH` | 422 (retryable=false) | D8: `Idempotency-Key` 를 다른 본문으로 재사용. 다시 보내지 않는다(`docs/web/http-contract.md` §2). fingerprint/scope 노출 금지 |
@@ -284,9 +285,26 @@ commit-aware response wrapper 계약을 구현한다. 또한 이 모듈은 HTML
advice 가 같은 mapper 로 Envelope 를 문자열로 직렬화해 넘긴다. 텍스트를 요청한 클라이언트는 그대로
문자열을 받는다. ArchUnit 으로 `String` 반환을 금지하는 안은 버렸다: `text/plain` 엔드포인트의 정당한
용도를 막고, 규칙을 돌리지 않는 서비스에는 함정이 그대로 남는다.
- **4xx·5xx 본문은 감싸지 않는다.** `success` 는 클라이언트가 가르는 판별자라, 오류 status 본문을
`success:true` 로 감싸면 성공으로 읽힌다. Boot `/error` 의 map 이 그렇게 나갔다(재리뷰 RR-02-02).
- 위치: sample 모듈이 아니라 adapter-web. 실행 앱은 adapter-web 에 의존하지만 sample-portfolio 에는 의존하지
않으므로, 응답 래핑이 실제로 동작하려면 여기 있어야 한다.
### EnvelopeErrorController
- 컨트롤러에 닿기 전에 생긴 오류(보안 방화벽의 `RequestRejectedException`, 필터의 `sendError`, 필터가
던진 예외)는 `GlobalExceptionHandler` 를 지나지 않고 컨테이너의 `/error` 디스패치로 간다. Boot 기본
`BasicErrorController` 는 거기서 `{timestamp, status, error, path}` 를 썼고, advice 가 그것을
`success:true` 로 감쌌으며 `path` 는 요청 경로 원문이었다(재리뷰 RR-02-02). 이 컨트롤러가 그 자리를
대신해 status 를 `GlobalExceptionHandler` 와 같은 표(`HttpStatusErrorCodes`)로 코드에 매핑하고 실패
Envelope 를 `EnvelopeResponseWriter` 로 쓴다. 요청에서 가져온 값은 싣지 않는다.
- `WebMvcPlatformAutoConfiguration` 이 `ErrorMvcAutoConfiguration` 보다 먼저 `ErrorController` 가 없을 때만
등록한다. 어댑터가 자기 `ErrorController` 를 두면 물러난다.
- `SecurityConfig` 는 오류 **디스패치**만 허용한다(`DispatcherType.ERROR`). 예전에는 `/error` 가 공개 경로가
아니라서 익명 요청의 400 이 401 로 바뀌었다. `/error` 를 직접 부른 요청은 여전히 인증이 필요하다.
단언은 실제 Tomcat 의 `TomcatWebContractIT`(공유 `WebContractAssertions`)와 `JwtChainErrorDispatchIT`.
- 앱이 보안 체인을 직접 정의하면(예: sample 의 데모 체인) 같은 한 줄
(`dispatcherTypeMatchers(DispatcherType.ERROR).permitAll()`)을 그 체인에도 둬야 원래 status 가 나간다.
### CacheControlFilter
- 스켈레톤 기본 HTTP 캐시 정책.
- `Cache-Control: no-store` 는 인증된 API 의 안전한 기본값. `Vary: Accept, Accept-Encoding, Authorization` 로
@@ -2,6 +2,7 @@ package dev.caskeleton.adapter.inbound.web.auth;
import dev.caskeleton.adapter.inbound.web.settings.CorsSettings;
import dev.caskeleton.adapter.inbound.web.settings.SecuritySettings;
import jakarta.servlet.DispatcherType;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@@ -83,6 +84,11 @@ public class SecurityConfig {
.requestCache(cache -> cache.disable())
.authorizeHttpRequests(
auth -> {
// The container's error dispatch for a request that already failed. Authorising it
// again turned an anonymous 400 into a 401 that blamed authentication. Only the
// dispatch type is opened: a request that names /error itself is still a request.
// The error controller writes nothing from the request (re-review RR-02-02).
auth.dispatcherTypeMatchers(DispatcherType.ERROR).permitAll();
if (publicPaths.length > 0) {
auth.requestMatchers(publicPaths).permitAll();
}
@@ -12,6 +12,7 @@ import org.springframework.http.converter.HttpMessageConverter;
import org.springframework.http.converter.StringHttpMessageConverter;
import org.springframework.http.server.ServerHttpRequest;
import org.springframework.http.server.ServerHttpResponse;
import org.springframework.http.server.ServletServerHttpResponse;
import org.springframework.web.bind.annotation.RestControllerAdvice;
import org.springframework.web.servlet.mvc.method.annotation.ResponseBodyAdvice;
import tools.jackson.databind.ObjectMapper;
@@ -34,6 +35,13 @@ import tools.jackson.databind.json.JsonMapper;
* still sees {@code 200 text/plain}. So when the String converter is about to write JSON, the
* envelope is serialised here, with the same mapper and wire rules, and the converter writes that
* text (review 2026-11-01 02-06). A text client still gets the plain string.
*
* <p>A body sent with a 4xx or 5xx status is never wrapped. {@code success} is the discriminator a
* client branches on, so a {@code success:true} envelope around an error-status body tells it the
* request worked. Boot's {@code /error} map reached clients exactly that way until the error
* dispatch got its own envelope writer ({@code EnvelopeErrorController}, re-review 2026-11-03
* RR-02-02). Failure envelopes come from the error handlers; a handler that returns some other body
* with an error status gets that body on the wire as it wrote it, not a false success.
*/
@RestControllerAdvice
public final class EnvelopeBodyAdvice implements ResponseBodyAdvice<Object> {
@@ -74,10 +82,18 @@ public final class EnvelopeBodyAdvice implements ResponseBodyAdvice<Object> {
if (selectedContentType != null && !MediaType.APPLICATION_JSON.includes(selectedContentType)) {
return body;
}
if (isErrorStatus(response)) {
return body;
}
Envelope<Object> envelope = Envelope.ok(body, ResponseMetaFactory.fromMdc());
if (StringHttpMessageConverter.class.isAssignableFrom(selectedConverterType)) {
return objectMapper.writeValueAsString(envelope);
}
return envelope;
}
private static boolean isErrorStatus(ServerHttpResponse response) {
return response instanceof ServletServerHttpResponse servlet
&& servlet.getServletResponse().getStatus() >= 400;
}
}
@@ -29,7 +29,7 @@ public class EnvelopeWireFormatConfig {
}
/** The module itself, for a writer that has no Boot mapper to take it from. */
static SimpleModule module() {
public static SimpleModule module() {
SimpleModule module = new SimpleModule("EnvelopeWireFormatModule");
module.setMixInAnnotation(Envelope.class, OmitNullFields.class);
module.setMixInAnnotation(BulkEnvelope.class, OmitNullFields.class);
@@ -0,0 +1,72 @@
package dev.caskeleton.adapter.inbound.web.error;
import dev.caskeleton.adapter.inbound.web.core.OwnsAbsoluteWebRoute;
import dev.caskeleton.adapter.inbound.web.http.ApiHeaders;
import dev.caskeleton.adapter.inbound.web.observability.RetryAfterAdvisor;
import dev.caskeleton.shared.error.ApiErrorCode;
import dev.caskeleton.shared.error.OperationalError;
import jakarta.servlet.RequestDispatcher;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.util.Objects;
import org.springframework.boot.webmvc.error.ErrorController;
import org.springframework.http.HttpStatusCode;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import tools.jackson.databind.ObjectMapper;
/**
* Answers the container's error dispatch with the failure {@code Envelope}.
*
* <p>A failure raised outside the MVC handler — the security firewall's {@code
* RequestRejectedException}, a filter's {@code sendError}, an exception thrown by a filter — never
* reaches {@link GlobalExceptionHandler}. The container forwards it to {@code /error} instead,
* where Boot's {@code BasicErrorController} wrote {@code {timestamp, status, error, path}}; the
* success advice then wrapped that map as {@code success:true}, and {@code path} echoed the raw
* request URI back (re-review 2026-11-03, RR-02-02). This replaces that controller: the error
* status is classified by the same table the exception handler uses, and the body is the failure
* envelope, with nothing from the request in it.
*
* <p>Written straight to the response by {@link EnvelopeResponseWriter}, like the other refusals
* made outside a converter. An error dispatch has no handler of its own to negotiate for, and a
* browser's {@code Accept: text/html} would otherwise turn the error into a 406 about the error.
*
* <p>A request that names {@code /error} itself carries no error status; it is answered as an
* unknown route. The security chain permits only the error dispatch, so such a request still needs
* a token under the JWT chain.
*
* <p>An absolute route ({@link OwnsAbsoluteWebRoute}): the container forwards to {@code /error},
* not to {@code /error} under the presentation prefix.
*/
@Controller
@OwnsAbsoluteWebRoute
@RequestMapping("${server.error.path:${error.path:/error}}")
public final class EnvelopeErrorController implements ErrorController {
private final ObjectMapper objectMapper;
/**
* Creates the controller.
*
* @param objectMapper Boot's mapper, which carries the envelope wire rules
*/
public EnvelopeErrorController(ObjectMapper objectMapper) {
this.objectMapper = Objects.requireNonNull(objectMapper, "objectMapper");
}
/** Writes the failure envelope for the status the error dispatch carries. */
@RequestMapping
public void error(HttpServletRequest request, HttpServletResponse response) throws IOException {
if (response.isCommitted()) {
return;
}
ApiErrorCode code =
request.getAttribute(RequestDispatcher.ERROR_STATUS_CODE) instanceof Integer status
? HttpStatusErrorCodes.forStatus(HttpStatusCode.valueOf(status))
: OperationalError.ROUTE_NOT_FOUND;
RetryAfterAdvisor.retryAfterSeconds(code)
.ifPresent(seconds -> response.setHeader(ApiHeaders.RETRY_AFTER, String.valueOf(seconds)));
EnvelopeResponseWriter.write(response, code, objectMapper);
}
}
@@ -407,7 +407,7 @@ public class GlobalExceptionHandler extends ResponseEntityExceptionHandler {
return null;
}
}
ApiErrorCode code = errorCodeFor(statusCode);
ApiErrorCode code = HttpStatusErrorCodes.forStatus(statusCode);
if (code.httpStatus() >= 500) {
log.error("spring mvc failure classified as {}", code.code(), ex);
spanErrorRecorder.recordException(ex, code.code());
@@ -415,25 +415,4 @@ public class GlobalExceptionHandler extends ResponseEntityExceptionHandler {
return new ResponseEntity<>(
ErrorResponseFactory.body(code, null), headers, HttpStatusCode.valueOf(code.httpStatus()));
}
private static ApiErrorCode errorCodeFor(HttpStatusCode status) {
return switch (status.value()) {
case 401 -> OperationalError.UNAUTHENTICATED;
case 403 -> OperationalError.FORBIDDEN;
case 404 -> OperationalError.ROUTE_NOT_FOUND;
case 405 -> OperationalError.METHOD_NOT_ALLOWED;
case 406 -> OperationalError.NOT_ACCEPTABLE;
case 412 -> OperationalError.PRECONDITION_FAILED;
case 413 -> OperationalError.PAYLOAD_TOO_LARGE;
case 414 -> OperationalError.URI_TOO_LONG;
case 415 -> OperationalError.UNSUPPORTED_MEDIA_TYPE;
case 429 -> OperationalError.RATE_LIMIT_EXCEEDED;
// Spring answers its async request timeout with 503: transient, not an internal fault.
case 503 -> WebTransportError.REQUEST_TIMEOUT;
default ->
status.is4xxClientError()
? OperationalError.BAD_PARAMETER
: OperationalError.INTERNAL_ERROR;
};
}
}
@@ -0,0 +1,38 @@
package dev.caskeleton.adapter.inbound.web.error;
import dev.caskeleton.shared.error.ApiErrorCode;
import dev.caskeleton.shared.error.OperationalError;
import org.springframework.http.HttpStatusCode;
/**
* The operational code a bare HTTP status stands for, when a status is all that is known.
*
* <p>Two places only have a status to classify: a Spring MVC exception this adapter has no
* dedicated handler for, and the container's error dispatch for a failure raised outside the
* handler. They share this one table, so the same status never reaches a client as two codes.
*/
final class HttpStatusErrorCodes {
private HttpStatusErrorCodes() {}
static ApiErrorCode forStatus(HttpStatusCode status) {
return switch (status.value()) {
case 401 -> OperationalError.UNAUTHENTICATED;
case 403 -> OperationalError.FORBIDDEN;
case 404 -> OperationalError.ROUTE_NOT_FOUND;
case 405 -> OperationalError.METHOD_NOT_ALLOWED;
case 406 -> OperationalError.NOT_ACCEPTABLE;
case 412 -> OperationalError.PRECONDITION_FAILED;
case 413 -> OperationalError.PAYLOAD_TOO_LARGE;
case 414 -> OperationalError.URI_TOO_LONG;
case 415 -> OperationalError.UNSUPPORTED_MEDIA_TYPE;
case 429 -> OperationalError.RATE_LIMIT_EXCEEDED;
// Spring answers its async request timeout with 503: transient, not an internal fault.
case 503 -> WebTransportError.REQUEST_TIMEOUT;
default ->
status.is4xxClientError()
? OperationalError.BAD_PARAMETER
: OperationalError.INTERNAL_ERROR;
};
}
}
@@ -1,5 +1,7 @@
package dev.caskeleton.adapter.inbound.web.mvc.autoconfigure;
import dev.caskeleton.adapter.inbound.web.envelope.EnvelopeWireFormatConfig;
import dev.caskeleton.adapter.inbound.web.error.EnvelopeErrorController;
import dev.caskeleton.adapter.inbound.web.error.GlobalExceptionHandler;
import dev.caskeleton.adapter.inbound.web.mvc.filter.JsonBodyLimitFilter;
import dev.caskeleton.adapter.inbound.web.mvc.filter.WebMvcRequestIdFilter;
@@ -12,8 +14,12 @@ import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.autoconfigure.condition.ConditionalOnWebApplication;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.boot.jackson.autoconfigure.JsonMapperBuilderCustomizer;
import org.springframework.boot.webmvc.autoconfigure.error.ErrorMvcAutoConfiguration;
import org.springframework.boot.webmvc.error.ErrorController;
import org.springframework.context.annotation.Bean;
import tools.jackson.core.StreamReadFeature;
import tools.jackson.databind.ObjectMapper;
import tools.jackson.databind.json.JsonMapper;
/**
* Wires the web platform into a servlet application.
@@ -30,8 +36,12 @@ import tools.jackson.core.StreamReadFeature;
*
* <p>Every bean is {@link ConditionalOnMissingBean}: the platform supplies the one that would
* otherwise be missing rather than the one that must be used.
*
* <p>Before {@link ErrorMvcAutoConfiguration}, so the envelope error controller is the {@link
* ErrorController} that configuration finds and Boot's {@code BasicErrorController} backs off. Its
* error-page registration with the container stays: that is what forwards to {@code /error}.
*/
@AutoConfiguration
@AutoConfiguration(before = ErrorMvcAutoConfiguration.class)
@ConditionalOnWebApplication(type = ConditionalOnWebApplication.Type.SERVLET)
@ConditionalOnProperty(
prefix = "app.web.mvc",
@@ -54,6 +64,22 @@ public class WebMvcPlatformAutoConfiguration {
return new GlobalExceptionHandler(spanErrorRecorderProvider);
}
/**
* Answers the container's error dispatch — failures raised outside the MVC handler — with the
* failure envelope instead of Boot's {@code {timestamp, status, error, path}} map.
*
* <p>Boot's mapper when there is one, which already carries the envelope wire rules; otherwise a
* mapper with only those rules, as {@code EnvelopeBodyAdvice} does without Boot.
*/
@Bean
@ConditionalOnMissingBean(ErrorController.class)
public EnvelopeErrorController envelopeErrorController(
ObjectProvider<ObjectMapper> objectMapper) {
return new EnvelopeErrorController(
objectMapper.getIfAvailable(
() -> JsonMapper.builder().addModule(EnvelopeWireFormatConfig.module()).build()));
}
/**
* Assigns the correlation identifiers, validating anything a caller proposed. With a tracer in
* the composition the trace id is the tracer's: the filter runs inside Boot's server observation
@@ -1,5 +1,6 @@
package dev.caskeleton.adapter.inbound.web.observability;
import dev.caskeleton.application.idempotency.IdempotencyExecutor;
import dev.caskeleton.shared.error.ApiErrorCode;
import java.util.Map;
import java.util.OptionalInt;
@@ -13,17 +14,26 @@ public final class RetryAfterAdvisor {
/** Per-code {@code Retry-After} values (seconds). */
private static final Map<String, Integer> RETRY_AFTER_SECONDS =
Map.of(
"RATE_LIMIT_EXCEEDED", 1,
// Below the default processing lease (30s): the resend arrives while the first attempt
// can still finish, and an early resend costs one more 409. docs/web/http-contract.md.
"IDEMPOTENT_IN_FLIGHT", 2,
"AUTH_KID_UNKNOWN", 5,
"AUTH_JWKS_UNAVAILABLE", 30,
"DEPENDENCY_TIMEOUT", 2,
"DEPENDENCY_CONNECT_FAILED", 2,
"DEPENDENCY_DNS_FAILED", 5,
"DEPENDENCY_5XX_SERVER", 2,
"DEPENDENCY_CIRCUIT_OPEN", 10);
"RATE_LIMIT_EXCEEDED",
1,
// The application's value, which the processing-lease check also reads: the resend
// arrives while the first attempt can still finish. docs/web/http-contract.md §2.
"IDEMPOTENT_IN_FLIGHT",
Math.toIntExact(IdempotencyExecutor.IN_FLIGHT_RETRY_AFTER.toSeconds()),
"AUTH_KID_UNKNOWN",
5,
"AUTH_JWKS_UNAVAILABLE",
30,
"DEPENDENCY_TIMEOUT",
2,
"DEPENDENCY_CONNECT_FAILED",
2,
"DEPENDENCY_DNS_FAILED",
5,
"DEPENDENCY_5XX_SERVER",
2,
"DEPENDENCY_CIRCUIT_OPEN",
10);
private RetryAfterAdvisor() {}
@@ -0,0 +1,91 @@
package dev.caskeleton.adapter.inbound.web.testkit.mvc;
import static org.assertj.core.api.Assertions.assertThat;
import dev.caskeleton.adapter.inbound.web.envelope.EnvelopeWireFormatConfig;
import dev.caskeleton.adapter.inbound.web.mvc.autoconfigure.WebAdapterAutoConfiguration;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.web.server.LocalServerPort;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Import;
/**
* The error dispatch under the production JWT security chain, over a real Tomcat.
*
* <p>The chain authorises every dispatch, the container's error dispatch to {@code /error}
* included. {@code /error} is not a public path, so an anonymous request whose original failure was
* a 400 used to come back as 401 — the client was told to authenticate when its request was
* malformed (re-review 2026-11-03, RR-02-02). The error dispatch is now permitted; a request that
* names {@code /error} directly is still a request and still needs a token.
*
* <p>The chain is {@code WebSecurityAutoConfiguration} itself. The JWT decoder is lazy, so the
* unreachable issuer below is never contacted: no request here carries a token.
*/
@SpringBootTest(
classes = JwtChainErrorDispatchIT.JwtChainApplication.class,
webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
properties = {
"spring.main.banner-mode=off",
"spring.mvc.problemdetails.enabled=false",
"app.web.mvc.enabled=true",
"app.security.auth-mode=jwt",
"app.security.issuer-uri=https://issuer.invalid/realms/ca",
"app.security.audience=ca-skeleton-api",
"app.security.public-paths=/api/v1/fixtures/**",
"app.cors.enabled=false"
})
class JwtChainErrorDispatchIT {
private final HttpClient client =
HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
@LocalServerPort private int port;
@Test
@DisplayName("JWT 체인에서 익명 요청이 핸들러 밖에서 400 으로 거부되면 401 이 아니라 400 실패 Envelope 로 나간다")
void anAnonymousRequestRejectedOutsideTheHandlerKeepsItsOwnStatus() throws Exception {
// when
HttpResponse<String> rejected = get(WebContractAssertions.FIREWALL_REJECTED_PATH);
// then
assertThat(rejected.statusCode()).isEqualTo(400);
WebContractAssertions.assertFailureEnvelope(rejected, "BAD_PARAMETER");
}
@Test
@DisplayName("오류 디스패치만 허용한다 — /error 를 직접 부른 익명 요청은 여전히 401 이다")
void aDirectRequestForTheErrorPathStillNeedsAToken() throws Exception {
// when
HttpResponse<String> direct = get("/error");
// then
assertThat(direct.statusCode()).isEqualTo(401);
}
private HttpResponse<String> get(String path) throws Exception {
return client.send(
HttpRequest.newBuilder(URI.create("http://localhost:" + port + path))
.header("Accept", "application/json")
.GET()
.build(),
HttpResponse.BodyHandlers.ofString());
}
/**
* The production web stack minus {@link WebAdapterAutoConfiguration}, whose request-logging
* filter needs the composition root's pseudonymizer, plus that root's envelope wire format; the
* security chain is the shipped one.
*/
@Configuration(proxyBeanMethods = false)
@EnableAutoConfiguration(exclude = WebAdapterAutoConfiguration.class)
@Import({ContractFixtureController.class, EnvelopeWireFormatConfig.class})
static class JwtChainApplication {}
}
@@ -23,6 +23,7 @@ import org.springframework.boot.webmvc.test.autoconfigure.AutoConfigureMockMvc;
import org.springframework.boot.webmvc.test.autoconfigure.WebMvcTest;
import org.springframework.context.annotation.Import;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.test.json.JsonCompareMode;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.web.bind.annotation.GetMapping;
@@ -64,6 +65,12 @@ class EnvelopeBodyAdviceTest {
return "{\"openapi\":\"3.1.0\"}".getBytes(StandardCharsets.UTF_8);
}
/** A failure status with a body that is not an envelope. */
@GetMapping("/probe/conflict")
public ResponseEntity<Map<String, String>> conflict() {
return ResponseEntity.status(409).body(Map.of("k", "v"));
}
@GetMapping("/probe/bulk")
public BulkEnvelope<Map<String, String>> bulk() {
return BulkEnvelope.allOk(List.of(Map.of("k", "v")), new ResponseMeta("r", "t", "c"));
@@ -149,6 +156,25 @@ class EnvelopeBodyAdviceTest {
.andExpect(content().json("{\"openapi\":\"3.1.0\"}", JsonCompareMode.STRICT));
}
/**
* A 4xx/5xx body is never a success envelope. {@code success} is the discriminator clients branch
* on (docs/web/http-contract.md §1), so wrapping an error-status body as {@code success:true}
* tells a contract-following client the request worked — Boot's {@code /error} map arrived that
* way (re-review 2026-11-03, RR-02-02). The body is left as the handler wrote it.
*/
@Test
@DisplayName("4xx·5xx 응답 본문은 success:true Envelope 로 감싸지 않는다")
void anErrorStatusBodyIsNotWrappedAsASuccess() throws Exception {
// when
var result = mvc.perform(get("/probe/conflict").accept(MediaType.APPLICATION_JSON));
// then
result
.andExpect(status().isConflict())
.andExpect(jsonPath("$.success").doesNotExist())
.andExpect(content().json("{\"k\":\"v\"}", JsonCompareMode.STRICT));
}
@SpringBootConfiguration
@EnableAutoConfiguration(exclude = SecurityAutoConfiguration.class)
static class TestBootstrap {}
@@ -3,6 +3,7 @@ package dev.caskeleton.adapter.inbound.web.error;
import static org.assertj.core.api.Assertions.assertThat;
import dev.caskeleton.adapter.inbound.web.http.ApiHeaders;
import dev.caskeleton.application.idempotency.IdempotencyCompletionUnknownException;
import dev.caskeleton.application.idempotency.IdempotencyInFlightException;
import dev.caskeleton.application.idempotency.IdempotencyRequestMismatchException;
import dev.caskeleton.shared.error.Category;
@@ -74,4 +75,24 @@ class IdempotencyExceptionMappingTest {
false,
ClientErrorMessages.of(Category.VALIDATION));
}
@Test
@DisplayName("complete 단계 결과 불명은 실행 전 거부(IDEMPOTENCY_UNAVAILABLE)와 다른 code 의 retryable 503 이다")
void completionUnknownMapsToItsOwnRetryable503SoTheClientDoesNotReadItAsNotApplied() {
// given
IdempotencyCompletionUnknownException unknown = new IdempotencyCompletionUnknownException();
// when
ResponseEntity<Envelope<Void>> response = handler.handleCarrier(unknown);
// then
assertThat(response)
.extracting(
r -> r.getStatusCode().value(),
r -> r.getHeaders().getFirst(ApiHeaders.RETRY_AFTER),
r -> r.getBody().error().code(),
r -> r.getBody().error().category(),
r -> r.getBody().error().retryable())
.containsExactly(503, null, "IDEMPOTENCY_COMPLETION_UNKNOWN", "TRANSIENT_DEPENDENCY", true);
}
}
@@ -8,6 +8,8 @@ import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import tools.jackson.databind.JsonNode;
import tools.jackson.databind.json.JsonMapper;
/**
* The status contract as socket-level assertions, written once and run against every container.
@@ -22,6 +24,11 @@ import java.time.Duration;
*/
public final class WebContractAssertions {
/** A path the servlet firewall refuses before any handler runs ({@code ;} in the URI). */
public static final String FIREWALL_REJECTED_PATH = "/api/v1/fixtures/f1;x=1";
private static final JsonMapper JSON = JsonMapper.builder().build();
private final HttpClient client =
HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
@@ -122,6 +129,46 @@ public final class WebContractAssertions {
assertThat(post("/api/v1/fixtures", "{\"name\":").statusCode()).isEqualTo(400);
}
/**
* An error raised outside the MVC handler is still a failure envelope.
*
* <p>The firewall rejects the request before any controller runs, and the container answers it
* with an error dispatch to {@code /error}. Boot's default error controller wrote a map there,
* which the success advice then wrapped as {@code success:true} with the raw request path inside
* (re-review 2026-11-03, RR-02-02). MockMvc performs no error dispatch, so only a container can
* show this.
*/
public void aRequestRejectedOutsideTheHandlerIsAFailureEnvelope()
throws IOException, InterruptedException {
HttpResponse<String> rejected = get(FIREWALL_REJECTED_PATH);
assertThat(rejected.statusCode()).isEqualTo(400);
assertFailureEnvelope(rejected, "BAD_PARAMETER");
}
/**
* Asserts the failure envelope of {@code docs/web/http-contract.md} §1 on an error response.
*
* @param response a response the container wrote through its error dispatch
* @param expectedCode the {@code error.code} the status maps to
*/
public static void assertFailureEnvelope(HttpResponse<String> response, String expectedCode) {
JsonNode body = JSON.readTree(response.body());
assertThat(body.path("success").isBoolean() && !body.path("success").booleanValue())
.as(
"a 4xx/5xx body that says success:true is read as a success by a client that follows"
+ " the contract: %s",
response.body())
.isTrue();
assertThat(body.path("error").path("code").asString()).isEqualTo(expectedCode);
assertThat(body.has("data")).as("a failure envelope has no data key").isFalse();
assertThat(response.body())
.as("the error body must not echo the request path back")
.doesNotContain(";x=1")
.doesNotContain("\"path\"")
.doesNotContain("timestamp");
}
/** Runs every assertion, so a lane cannot silently cover a subset. */
public void assertWholeContract() throws IOException, InterruptedException {
createdCarriesALocationHeader();
@@ -131,6 +178,7 @@ public final class WebContractAssertions {
anUnknownMethodOnAKnownRouteIsRefused();
anUnsupportedMediaTypeIsRefused();
malformedJsonIsRefused();
aRequestRejectedOutsideTheHandlerIsAFailureEnvelope();
}
private HttpResponse<String> post(String path, String body)
@@ -143,6 +191,12 @@ public final class WebContractAssertions {
HttpResponse.BodyHandlers.ofString());
}
private HttpResponse<String> get(String path) throws IOException, InterruptedException {
return client.send(
HttpRequest.newBuilder(uri(path)).header("Accept", "application/json").GET().build(),
HttpResponse.BodyHandlers.ofString());
}
private URI uri(String path) {
return URI.create("http://localhost:" + port + path);
}
@@ -1,5 +1,6 @@
package dev.caskeleton.webtestkit;
import dev.caskeleton.adapter.inbound.web.envelope.EnvelopeWireFormatConfig;
import dev.caskeleton.adapter.inbound.web.mvc.autoconfigure.WebAdapterAutoConfiguration;
import dev.caskeleton.adapter.inbound.web.mvc.autoconfigure.WebSecurityAutoConfiguration;
import dev.caskeleton.adapter.inbound.web.testkit.fault.FaultFixtureController;
@@ -31,6 +32,10 @@ import org.springframework.security.web.SecurityFilterChain;
* are the "sum of the leaf" the paragraph above keeps out. Excluded by class, so a rename breaks
* compilation rather than the gate. The third root, {@code WebMethodSecurityAutoConfiguration}, is
* left in: no fixture route runs a permissioned use case, so it decides nothing here.
*
* <p>{@link EnvelopeWireFormatConfig} is imported by name from the excluded web root: it is the
* wire shape of every envelope this gate reads (a failure body has no {@code data} key), not a
* capability, and without it the gate would certify {@code "data": null} that no deployment sends.
*/
// Outside `dev.caskeleton.adapter.inbound.web`, which the composition root component-scans.
// A fixture application under that root is loaded into the real deployment: its permit-all
@@ -40,7 +45,11 @@ import org.springframework.security.web.SecurityFilterChain;
@Configuration(proxyBeanMethods = false)
@EnableAutoConfiguration(
exclude = {WebAdapterAutoConfiguration.class, WebSecurityAutoConfiguration.class})
@Import({ContractFixtureController.class, FaultFixtureController.class})
@Import({
ContractFixtureController.class,
FaultFixtureController.class,
EnvelopeWireFormatConfig.class
})
public class ContractFixtureApplication {
/**
+18 -6
View File
@@ -128,12 +128,15 @@ Focused tests drive the semantic adapters against `InMemorySemanticRedisCommands
served through `StubRedisRuntimeClient`. R1/R2 promotion requires a separate real Redis service lane; it may never
be silently skipped when selected.
Two real-server lanes. `redisContainerContractTest` (integration category, in `integrationCheck` and
this leaf's `mainCheck`) starts its own Redis with Testcontainers — the `redis` image of
`config/testing/container-images.properties`, loading `infra/redis-sdk/acl/all-accounts.acl` — and
runs `LiveRedisCacheRegionContractTest` and `LiveRedisDistributedLeaseContractTest` (the
`application-core` port contracts the in-memory fakes also pass), `LiveRedisSemanticPortsTest` and
`LiveRedisSemanticScriptAclProbeTest`; a skip fails it (review 2026-11-01 05-16). `RedisTestContainer`
Two real-server lanes. `redisContainerContractTest` (integration category, in `integrationCheck` only —
not in this leaf's `mainCheck` too, which ran it twice per push to main) starts its own Redis with Testcontainers — the `redis` image of
`config/testing/container-images.properties`, loading `infra/redis-sdk/acl/lane-fixture-accounts.acl` — and
runs `LiveRedisCacheRegionContractTest`, `LiveRedisDistributedLeaseContractTest` and
`LiveRedisIdempotencyStoreContractTest` (the `application-core` port contracts the in-memory fakes
also pass), `LiveRedisSemanticPortsTest` and `LiveRedisSemanticScriptAclProbeTest`; a skip fails it
(review 2026-11-01 05-16). The idempotency contract was on the explicit-only topology lane alone
until review 2026-11-03 RR-04-05, so the Lua state machine's stale-handle, renew-refusal and
retry cases reached a real server in no gate. `RedisTestContainer`
starts the server only when that lane names the ACL file.
`redisTopologyTest` is the external-topology lane, declared through `ca.redis-topology-lane`. This
@@ -177,6 +180,15 @@ deployment still gets exactly one client and one event loop. The `SCRIPT` lane i
the reason it exists: `SCRIPT LOAD` and `EVALSHA` belong to the advanced account, so the account
that reads a cache entry cannot execute a script.
The cache's conditional writes run on the same account but on a lane of their own,
`CacheScriptLane.CACHE_SCRIPT` (`app.capabilities.cache.maximum-script-connections`, contributed by
`RedisCacheConfig` as a `RedisLaneContribution`). Do not move them back to `SCRIPT`: that lane is the
idempotency, rate-limit and lease budget, and a cache under load must not take it (review 2026-11-03
README §4, RR-04-02). Each lane's wait, refusals and reconnects are meters of `redis-runtime`
(`RedisLaneMetrics`, `redis.lane.*`, tag `lane`), so whether the cache stays off `SCRIPT` is a query,
not an assumption. The `DEGRADED` health order ships with `redis-runtime`, which reports the
status (`config/module-defaults/redis-runtime.yml`, RR-04-10).
A Cluster transaction runs on one node, so the SDK's `RedisTransactionRunner` derives a routing key
from the watched keys (or an explicit `RedisSlotTag`) and borrows `RedisSdkLane.TRANSACTION`, which
is pinned to its key's owner, on the node that owns that slot. Routed leases are never pooled: a pooled connection is pinned to the previous caller's
+54 -7
View File
@@ -97,7 +97,10 @@ Connection lane limits and command queue limits are different contracts. In part
- `app.redis.capacity.maximum-in-flight-commands` bounds the per-connection command queue.
- `app.redis.capacity.maximum-script-connections` bounds physical SCRIPT lane connections
(default 8).
(default 8). Idempotency, rate limiting and leases share them.
- `app.capabilities.cache.maximum-script-connections` (`APP_CACHE_REDIS_MAXIMUM_SCRIPT_CONNECTIONS`)
bounds the cache's own script lane (`CacheScriptLane`, default 4), contributed only when the cache
binding is `redis`. See "Why the cache has its own script lane" below.
- `app.redis.lifecycle.acquire-timeout` (default 2s) is how long a borrow waits for a returned
connection when its lane is full.
@@ -153,7 +156,7 @@ Important contracts:
Every lookup returns a `CacheWriteCondition` naming the state it saw — `absent`, `vacant:<digest>`
(an invalidation marker, an entry of an older region generation, an expired entry) or
`present:<digest>`. A record that carries the condition applies only while the key is still in that
state (registered program, SCRIPT lane). This is what rejects a source load that began before an
state (registered program, cache script lane). This is what rejects a source load that began before an
invalidation and finished after it.
A hit also returns an opaque SHA-1 observation token over the stored bytes. SHA-1 here is **not a
@@ -181,10 +184,43 @@ example one written by `UPSERT` as `rev-1`), another schema version, a generatio
writer knows — is `NOT_RECORDED_CONDITION`.
Both conditional intents, and any record carrying a write condition, run registered scripts, so
they borrow the SCRIPT lane (the advanced account when one is configured, the application account
otherwise). The application account is not granted `EVALSHA`; on the REGULAR lane every conditional
write came back `INDETERMINATE`. Unconditioned `UPSERT` and `ONLY_IF_ABSENT` stay on the REGULAR
lane.
they run on the scripting account (the advanced account when one is configured, the application
account otherwise), on the cache's own script lane. The application account is not granted
`EVALSHA`; on the REGULAR lane every conditional write came back `INDETERMINATE`. Unconditioned
`UPSERT` and `ONLY_IF_ABSENT` stay on the REGULAR lane.
### Why the cache has its own script lane
Recording with the condition a lookup returned is the normal way to use this port, so nearly every
cache fill is a script. On the runtime's SCRIPT lane those fills shared eight connections per pod
with idempotency claims, rate-limit evaluations and lease calls, and a timed-out lease is discarded
rather than pooled, so each timeout also cost a reconnect. A slow Redis or a burst of misses after a
region invalidation filled the lane with cache work; an idempotency claim then waited out its
timeout and answered `UNAVAILABLE`, and a rate-limit call fell to its failure policy. The cache is a
speed feature that may degrade; idempotency, rate limiting and leases are correctness features that
must not (review 2026-11-03 README §4, RR-04-02).
So `CacheScriptLane.CACHE_SCRIPT` is a lane of its own — the same scripting account, its own
connections, its own ceiling (`app.capabilities.cache.maximum-script-connections`, default 4). A cache
that runs out of them degrades (`DEGRADED_UNAVAILABLE`, nothing sent) without touching the SCRIPT
lane. Alternatives not taken: a per-capability permit cap inside the shared SCRIPT lane (still one
pool of connections and one reconnect cost), and running the cache programs on the REGULAR lane
(grants the application account `EVALSHA`, which the account split exists to withhold).
Whether the split holds under load is visible per lane. With a `MeterRegistry` in the context,
`redis-runtime` registers these for every lane it pools — `REGULAR`, `SCRIPT`, `CACHE_SCRIPT` and the
typed SDK's — tagged `lane=<name>` (`RedisLaneMetrics`; rows in `docs/registries/metrics.yaml`):
| Meter | Type | Read it as |
| --- | --- | --- |
| `redis.lane.acquire.wait.seconds` (`outcome` = `admitted`/`refused`) | timer, histogram | how long a borrow queued for a connection; a growing admitted p99 comes before refusals |
| `redis.lane.acquire.refused.total` (`reason` = `saturated`/`closing`/`interrupted`/`connect_failed`) | counter | a borrow that got no connection, nothing sent; `saturated` on `SCRIPT` is the starvation this lane exists to prevent |
| `redis.lane.connections.opened.total` | counter | new connections; after warm-up every one is a reconnect |
| `redis.lane.connections.discarded.total` (`reason` = `invalidated`/`broken`) | counter | connections closed instead of pooled — `invalidated` is a timed-out or failed call (the lease is not reused), and is what the reconnects replace |
| `redis.lane.connections.active`, `redis.lane.connections.pending` | gauge | connections lent out now; borrows waiting now |
Every counter series exists from startup at zero, so an alert on "refusals above zero" sees the first
one. A lane refusal is also logged, per call, by `RedisSemanticFailureLogger`.
### Key invalidation
@@ -300,10 +336,21 @@ Useful focused checks:
`redisContainerContractTest` runs the cache-region and lease port contracts (the ones the in-memory
fakes pass) and the live semantic-port tests against a Redis it starts itself, under the standalone
topology's ACL file. It is in `integrationCheck` and `mainCheck`; the image is
topology's ACL file. It is in `integrationCheck` (integration-main.yml's `integration-check` job), and
only there: listing it in `mainCheck` as well ran it twice on every push to main. The image is
`config/testing/container-images.properties` `redis`.
Live Redis/topology qualification remains separate from fast unit tests. A release claim for a
specific Redis topology or ACL layout requires the corresponding live qualification lane; the
in-memory gateway is for deterministic contract tests, not evidence that an external Redis
deployment accepts the same network/TLS/ACL configuration.
## 아직 내보내지 않는 지표
`redis.capability.*` 여섯 개(operations.total, duration.seconds, admission.rejected.total,
inflight.total, readiness.total, lifecycle.drain.total)는 redis-production-capability 설계(Task 16)에
있었지만 어떤 코드도 내보내지 않았다. 레지스트리(`docs/registries/metrics.yaml`)와 incident runbook이 있는
것처럼 적고 있어서, 2026-11-03 재리뷰 수정에서 레지스트리에서 빼고 runbook을 지금 내보내는
`redis.lane.*`(redis-runtime, lane별 대기·거절·연결) 기준으로 고쳤다. capability 단위 지표가 필요해지면
operation 결과를 아는 semantic adapter(이 leaf)에서 내보내고, 그때 레지스트리에 다시 넣는다.
+13 -7
View File
@@ -48,7 +48,8 @@ dependencies {
// command surface the catalog exists to prevent. Zero imports.
// io.micrometer:micrometer-core — observation leaves this leaf as RedisObservation through a
// Consumer sink; binding it to a meter registry belongs to the composition root, not here.
// Zero imports.
// Zero imports. The connection lanes' meters, CACHE_SCRIPT's included, are redis-runtime's
// (RedisLaneMetrics): the lanes are pooled there, so that is where they are counted.
}
// Redis topology qualification is opt-in and owned by `ca.redis-topology-lane`: mode/property
@@ -69,8 +70,12 @@ redisTopologyLane {
// adapters' port contracts reached a real server in no stage, and the fake and the adapter could hide
// the same defect (05-02, 05-03). The image is config/testing/container-images.properties' `redis`,
// and the ACL file is infra/redis-sdk's own, so the accounts and their rights are the standalone
// topology's. Integration category: integrationCheck runs it, and so does this leaf's mainCheck, like
// the HTTP client's Toxiproxy lane. Sentinel, cluster and TLS remain the topology lane's.
// topology's. Integration category, so integrationCheck runs it — integration-main.yml's
// `integration-check` job, on every push to main and nightly. It is deliberately not in this leaf's
// mainCheck as well: `main-check` and `integration-check` are two jobs of the same workflow run, so a
// lane in both started a second Redis container and ran every live contract twice per push without
// covering anything the first run did not (found while fixing review 2026-11-03). Sentinel, cluster
// and TLS remain the topology lane's.
def redisAclDirectory = rootProject.projectDir.parentFile.toPath().resolve('infra/redis-sdk/acl').toFile()
strictTestLanes {
lane('redisContainerContractTest') {
@@ -78,15 +83,16 @@ strictTestLanes {
description = 'Runs the Redis semantic adapters\' live contracts against a Testcontainers Redis on the pinned image.'
requires 'dev.caskeleton.adapter.outbound.cache.redis.cache.LiveRedisCacheRegionContractTest',
'dev.caskeleton.adapter.outbound.cache.redis.lease.LiveRedisDistributedLeaseContractTest',
// The fake carries a Java copy of the idempotency Lua state machine; only a real
// server proves the two agree (05-02 was the fake copying the Lua's bug). It ran on
// the explicit-only topology lane alone, so no gate ever ran it (RR-04-05).
'dev.caskeleton.adapter.outbound.cache.redis.idempotency.LiveRedisIdempotencyStoreContractTest',
'dev.caskeleton.adapter.outbound.cache.redis.LiveRedisSemanticPortsTest',
'dev.caskeleton.adapter.outbound.cache.redis.capability.LiveRedisSemanticScriptAclProbeTest'
rejectSkipped = true
systemProperty 'redis.container.acl-file', new File(redisAclDirectory, 'all-accounts.acl').absolutePath
systemProperty 'redis.container.acl-file', new File(redisAclDirectory, 'lane-fixture-accounts.acl').absolutePath
inputDirectory redisAclDirectory
shouldRunAfter 'test'
}
}
tasks.named('mainCheck') {
dependsOn 'redisContainerContractTest'
}
@@ -50,7 +50,7 @@ io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnota
io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor
io.lettuce:lettuce-core:6.8.2.RELEASE=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-commons:1.16.7=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-core:1.16.7=testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-core:1.16.7=runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-jakarta9:1.16.7=testCompileClasspath,testRuntimeClasspath
io.micrometer:micrometer-observation:1.16.7=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
io.netty:netty-buffer:4.2.17.Final=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
@@ -103,7 +103,7 @@ org.codehaus.plexus:plexus-container-default:2.1.0=checkstyle
org.codehaus.plexus:plexus-utils:3.3.0=checkstyle
org.dom4j:dom4j:2.2.0=spotbugs
org.hamcrest:hamcrest:3.0=testCompileClasspath,testRuntimeClasspath
org.hdrhistogram:HdrHistogram:2.2.2=testRuntimeClasspath
org.hdrhistogram:HdrHistogram:2.2.2=runtimeClasspath,testRuntimeClasspath
org.javassist:javassist:3.28.0-GA=checkstyle
org.jetbrains:annotations:17.0.0=testCompileClasspath,testRuntimeClasspath
org.jspecify:jspecify:1.0.1=annotationProcessor,checkstyle,compileClasspath,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
@@ -116,7 +116,7 @@ org.junit.platform:junit-platform-engine:6.0.3=testRuntimeClasspath
org.junit.platform:junit-platform-launcher:6.0.3=testRuntimeClasspath
org.junit:junit-bom:6.0.3=testCompileClasspath,testRuntimeClasspath
org.junit:junit-bom:6.1.0=spotbugs
org.latencyutils:LatencyUtils:2.0.3=testRuntimeClasspath
org.latencyutils:LatencyUtils:2.0.3=runtimeClasspath,testRuntimeClasspath
org.mockito:mockito-core:5.20.0=mockitoAgent,testCompileClasspath,testRuntimeClasspath
org.mockito:mockito-junit-jupiter:5.20.0=testCompileClasspath,testRuntimeClasspath
org.objenesis:objenesis:3.3=testRuntimeClasspath
@@ -0,0 +1,34 @@
package dev.caskeleton.adapter.outbound.cache.redis.cache;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisAccount;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisLane;
/**
* The cache's own script lane, so a cache's load cannot starve a correctness capability (review
* 2026-11-03 README §4, RR-04-02).
*
* <p>Every conditional cache write is a registered program, and a registered program needs the
* scripting account. On the runtime's SCRIPT lane those writes shared eight connections with
* idempotency claims, rate-limit evaluations and lease calls — and the normal way to use the cache
* (record with the condition the lookup returned) makes every cache fill a script. A slow Redis or
* a burst of misses after a region invalidation then held those connections with cache work while
* an idempotency claim waited out its timeout and failed. The cache is a speed feature and may
* degrade; idempotency and leases are correctness features and must not. They get separate
* connection budgets.
*
* <p>Same account as the SCRIPT lane: the grant is the same, only the connections and their ceiling
* ({@code app.capabilities.cache.maximum-script-connections}) are the cache's. Alternatives that
* were not taken: a per-capability permit cap inside the shared SCRIPT lane would still share the
* connections and their reconnect cost after a discarded lease; running cache programs on the
* REGULAR lane would grant the application account {@code EVALSHA}, which the account split exists
* to withhold.
*/
public enum CacheScriptLane implements RedisLane {
/** The cache region's conditional and revision-ordered writes. */
CACHE_SCRIPT;
@Override
public RedisAccount account() {
return RedisAccount.ADVANCED;
}
}
@@ -68,8 +68,10 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
private final SemanticRedisCall call;
/**
* The conditional writes, which are registered programs and so run on the SCRIPT lane: the
* application account the REGULAR lane authenticates as is not granted {@code EVALSHA}.
* The conditional writes, which are registered programs and so run on the scripting account: the
* application account the REGULAR lane authenticates as is not granted {@code EVALSHA}. They run
* on the cache's own lane ({@link CacheScriptLane}), not the runtime's SCRIPT lane, so a cache
* under load cannot hold the connections idempotency, rate limiting and leases need (RR-04-02).
*/
private final SemanticRedisCall scriptCall;
@@ -133,7 +135,7 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
Objects.requireNonNull(commandTimeout, "command timeout must be non-null"),
"cache");
this.scriptCall =
new SemanticRedisCall(owner, RedisConnectionKind.SCRIPT, commandTimeout, "cache");
new SemanticRedisCall(owner, CacheScriptLane.CACHE_SCRIPT, commandTimeout, "cache");
this.keys = Objects.requireNonNull(keys, "keys must be non-null");
this.keyDigest = Objects.requireNonNull(keyDigest, "key digest must be non-null");
this.encoder = Objects.requireNonNull(encoder, "encoder must be non-null");
@@ -1,6 +1,8 @@
package dev.caskeleton.adapter.outbound.cache.redis.capability;
import dev.caskeleton.adapter.outbound.cache.redis.cache.CacheScriptLane;
import dev.caskeleton.adapter.outbound.cache.redis.cache.RedisCacheRegionAdapter;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisLaneContribution;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisRuntimeSettings;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisRuntimeOwner;
import dev.caskeleton.adapter.outbound.support.CapabilitySelectors;
@@ -56,4 +58,24 @@ class RedisCacheConfig {
cache.getNegativeTtl(),
cache.getCommandTimeout());
}
/**
* The cache's own script lane and its connection ceiling.
*
* <p>The runtime pools a contributed lane like its own SCRIPT lane, on the same scripting
* account, but with connections and a ceiling of its own: the cache's conditional writes no
* longer compete with idempotency claims, rate-limit evaluations and lease calls for the SCRIPT
* lane (review 2026-11-03 README §4, RR-04-02).
*
* @param capabilities the capability settings
* @return the lane contribution
*/
@Bean
@ConditionalOnProperty(name = CapabilitySelectors.CACHE_DEFAULT_BINDING, havingValue = "redis")
RedisLaneContribution redisCacheScriptLane(RedisCapabilitySettings capabilities) {
RedisCapabilitySettings.Cache cache = capabilities.getCache();
cache.validate();
return new RedisLaneContribution(
CacheScriptLane.CACHE_SCRIPT, cache.getMaximumScriptConnections());
}
}
@@ -86,6 +86,7 @@ public class RedisCapabilitySettings {
private Duration positiveHardTtl = Duration.ofMinutes(5);
private Duration negativeTtl = Duration.ofSeconds(10);
private Duration minimumHardTtl = Duration.ofSeconds(1);
private int maximumScriptConnections = 4;
/**
* Reports whether the default binding selected Redis.
@@ -119,6 +120,10 @@ public class RedisCapabilitySettings {
if (keyVersion < 1) {
throw new CapabilityStartupValidationException("the cache key version must be positive");
}
if (maximumScriptConnections < 1) {
throw new CapabilityStartupValidationException(
"the cache script lane needs at least one connection");
}
}
public Map<String, String> getBindings() {
@@ -192,6 +197,23 @@ public class RedisCapabilitySettings {
public void setMinimumHardTtl(Duration minimumHardTtl) {
this.minimumHardTtl = minimumHardTtl;
}
/**
* The ceiling on the cache's own script-lane connections.
*
* <p>Separate from {@code app.redis.capacity.maximum-script-connections}, which idempotency,
* rate limiting and leases share: a cache is a speed feature and may run out of connections and
* degrade; it must not take them from the correctness features (review 2026-11-03 README §4).
*
* @return the ceiling, at least 1
*/
public int getMaximumScriptConnections() {
return maximumScriptConnections;
}
public void setMaximumScriptConnections(int maximumScriptConnections) {
this.maximumScriptConnections = maximumScriptConnections;
}
}
/** Edge rate-limit provider composition. */
@@ -28,11 +28,12 @@ import org.springframework.beans.factory.SmartInitializingSingleton;
* {@code {1}} only when the value it wrote reads back.
*
* <p>A cache-only deployment is probed too: its conditional writes and revision-ordered writes are
* registered programs on the same lane, and without the permission every one of them would come
* registered programs on the same account, and without the permission every one of them would come
* back degraded while plain reads looked healthy. Its probe runs only what the cache programs call
* ({@code GET}, {@code SET}), and a Redis it cannot reach at startup is logged, not fatal — a cache
* outage degrades, it never stops a pod from starting. A server that answers and refuses the
* scripts is a configuration error and fails startup as it does for the correctness roles.
* ({@code GET}, {@code SET}). Only a server that answers and refuses the scripts — an ACL refusal,
* {@code NOSCRIPT}, or a reply other than the value the probe wrote — is a configuration error and
* fails startup as it does for the correctness roles. Every other failure is a cache outage and is
* logged, not fatal: a cache outage degrades, it never stops a pod from starting.
*/
final class RedisSemanticScriptAclProbe implements SmartInitializingSingleton {
@@ -89,7 +90,7 @@ final class RedisSemanticScriptAclProbe implements SmartInitializingSingleton {
try {
probe(CACHE_PROBE_SCRIPT);
} catch (CapabilityStartupValidationException failure) {
if (!unreachable(failure.getCause())) {
if (refusedByTheServer(failure.getCause())) {
throw new CapabilityStartupValidationException(
"Redis cache is selected but its account cannot run the cache's conditional-write"
+ " scripts (SCRIPT LOAD/EVALSHA, GET, SET); every conditional record would"
@@ -98,15 +99,49 @@ final class RedisSemanticScriptAclProbe implements SmartInitializingSingleton {
failure.getCause());
}
LOG.warn(
"Redis cache is selected but Redis was unreachable at startup; the cache script"
+ " permission is unverified and the cache runs degraded until Redis answers",
"Redis cache is selected but the startup script probe could not complete; the cache"
+ " script permission is unverified and the cache runs degraded until Redis answers",
failure.getCause());
}
}
}
private static boolean unreachable(Throwable cause) {
return cause instanceof RedisLaneUnavailableException || cause instanceof TimeoutException;
/**
* Whether the cache probe failed because the server refused the cache's scripts.
*
* <p>"Refused" is a statement about the account, and only three answers make it: an ACL refusal
* ({@code NOPERM}, or a script's inner command denied), {@code NOSCRIPT} for the program the same
* connection had just loaded, and a reply other than the value the probe wrote. The check used to
* run the other way — only a lane refusal or a timeout counted as "unreachable" and everything
* else was a configuration error — so a connection that dropped mid-probe, a restarting Redis
* answering {@code LOADING}, or a busy one answering {@code BUSY} stopped a cache-only pod from
* starting, against this leaf's promise that a cache outage never does (review 2026-11-03
* RR-04-07).
*
* @param cause the probe's failure
* @return {@code true} when the account, not the server's availability, is at fault
*/
private static boolean refusedByTheServer(Throwable cause) {
for (Throwable current = cause;
current != null;
current = current.getCause() == current ? null : current.getCause()) {
if (current instanceof CapabilityStartupValidationException) {
// The probe ran and read back something other than what it wrote.
return true;
}
if (current instanceof RedisLaneUnavailableException || current instanceof TimeoutException) {
return false;
}
String message = current.getMessage();
if (message != null
&& (message.startsWith("NOPERM")
|| message.startsWith("NOSCRIPT")
|| message.contains("no permissions")
|| message.contains("ACL failure"))) {
return true;
}
}
return false;
}
private void probe(byte[] script) {
@@ -141,6 +141,12 @@ public final class IdempotencyScripts {
* it started from and the lease it asked for. A retry from that revision with the same lease is
* the same renewal; with a different lease it is a different request, because confirming it would
* leave the caller believing it holds the record for a lease the record does not carry.
*
* <p>The key's own TTL is stretched to cover the new lease. Claim sets it to the replay TTL,
* which may be shorter than a lease renewed past it; the key then expired while the holder was
* still executing, and the next request got a fresh claim and ran the side effect again (review
* 2026-11-03 RR-04-08). The TTL is only ever lengthened here — completion sets the replay TTL
* again.
*/
private static final String RENEW =
"""
@@ -177,6 +183,10 @@ public final class IdempotencyScripts {
redis.call('HSET', KEYS[1],
'leaseUntil', nowMillis + tonumber(ARGV[4]), 'rev', rev + 1,
'renewedFromRev', ARGV[2], 'renewedLease', ARGV[4])
local ttl = redis.call('PTTL', KEYS[1])
if ttl >= 0 and ttl < tonumber(ARGV[4]) then
redis.call('PEXPIRE', KEYS[1], ARGV[4])
end
return {'APPLIED', attempt, rev + 1, owner, '', ''}
""";
@@ -39,6 +39,10 @@ app:
# Floor on the hard TTL. Below it, entries expire faster than the round trip that wrote them:
# every read misses and every miss writes.
minimum-hard-ttl: ${APP_CACHE_REDIS_MINIMUM_HARD_TTL:1s}
# The cache's own script-lane connections, apart from the SCRIPT lane that idempotency, rate
# limiting and leases share (app.redis.capacity.maximum-script-connections). A cache under
# load runs out of these and degrades; it cannot take the correctness features' connections.
maximum-script-connections: ${APP_CACHE_REDIS_MAXIMUM_SCRIPT_CONNECTIONS:4}
idempotency:
# Redis store tuning only; the provider selector and lifecycle policy stay with the
# application's idempotency executor.
@@ -75,15 +79,8 @@ app:
maximum-cost: ${APP_RATE_LIMIT_MAXIMUM_COST:10}
cleanup-grace: ${APP_RATE_LIMIT_CLEANUP_GRACE:5s}
# redis-runtime 의 redisOptional 은 Redis 에 닿지 못하면 DEGRADED 를 보고한다. Boot 기본 순서
# (DOWN, OUT_OF_SERVICE, UP, UNKNOWN)에 없는 상태는 DOWN 보다 앞에 정렬돼, 다른 contributor 가 DOWN
# 이어도 전체 health 가 HTTP 200 DEGRADED 로 보였다(리뷰 05-23e). DEGRADED 를 DOWN 뒤, UP 앞에 둔다.
# 이 키를 정하는 module-defaults 는 이 파일 하나뿐이다. 서비스가 순서를 바꾸면 DEGRADED 를 넣어야 한다.
management:
endpoint:
health:
status:
order: DOWN,OUT_OF_SERVICE,DEGRADED,UP,UNKNOWN
# DEGRADED 상태 순서(management.endpoint.health.status.order)는 그 상태를 내는 redis-runtime 이
# 싣는다(config/module-defaults/redis-runtime.yml, 리뷰 2026-11-03 RR-04-10).
# /actuator/info 에 싣는 이 모듈의 상태(management.info.env). compose smoke 가 lane 이 실제로 켠
# capability 를 이 값으로 확인한다. 위 스위치를 해석한 값이고, 기본값은 RedisCapabilityConfig 의
@@ -1,5 +1,6 @@
package dev.caskeleton.adapter.outbound.cache.redis;
import dev.caskeleton.adapter.outbound.cache.redis.cache.CacheScriptLane;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.RedisTopologyEndpoint;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisCredentialResolver.RedisCredentials;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisRuntimeSettings;
@@ -71,6 +72,9 @@ public final class LiveRedisRuntime {
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
limits.put(kind, 4);
}
// The cache region's own script lane, which the composition root contributes when the cache
// binding is redis (RedisCacheConfig).
limits.put(CacheScriptLane.CACHE_SCRIPT, 4);
return new RedisRuntimeOwner(
new RedisTopologyClientFactory(
settings,
@@ -28,7 +28,7 @@ final class RedisTestContainer {
static final String ACL_FILE_PROPERTY = "redis.container.acl-file";
private static final String IMAGE_PROPERTY = "ca.test.image.redis";
private static final String ACL_IN_CONTAINER = "/etc/redis/acl/all-accounts.acl";
private static final String ACL_IN_CONTAINER = "/etc/redis/acl/lane-fixture-accounts.acl";
private static final int PORT = 6379;
private static GenericContainer<?> server;
@@ -60,8 +60,17 @@ class RedisCacheRegionAdapterTest {
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
limits.put(kind, 4);
}
return new RedisCacheRegionAdapter<>(
limits.put(CacheScriptLane.CACHE_SCRIPT, 4);
return cache(
now,
new RedisRuntimeOwner(client, limits, Duration.ofSeconds(1), Duration.ofSeconds(1)),
cacheKeys);
}
private RedisCacheRegionAdapter<String, String> cache(
Instant now, RedisRuntimeOwner owner, RedisCacheRegionAdapter.CacheKeys cacheKeys) {
return new RedisCacheRegionAdapter<>(
owner,
cacheKeys,
key -> "digest-" + key,
value -> value.getBytes(StandardCharsets.UTF_8),
@@ -428,6 +437,31 @@ class RedisCacheRegionAdapterTest {
.hasMessageContaining("ONLY_IF_SOURCE_REVISION_NEWER");
}
@Test
@DisplayName("멱등성·lease가 SCRIPT lane을 모두 잡고 있어도 캐시 조건부 기록은 캐시 전용 lane에서 기록된다")
void aConditionalCacheWriteDoesNotWaitForTheCorrectnessScriptLane() {
// given
Map<RedisLane, Integer> limits = new java.util.LinkedHashMap<>();
limits.put(RedisConnectionKind.REGULAR, 4);
limits.put(RedisConnectionKind.SCRIPT, 1);
limits.put(CacheScriptLane.CACHE_SCRIPT, 1);
try (RedisRuntimeOwner owner =
new RedisRuntimeOwner(
StubRedisRuntimeClient.serving(gateway),
limits,
Duration.ofMillis(100),
Duration.ofSeconds(1));
var heldByAnIdempotencyClaim = owner.borrow(RedisConnectionKind.SCRIPT)) {
RedisCacheRegionAdapter<String, String> cache = cache(T0, owner, keys);
// when
CacheRecordOutcome outcome = cache.record("order-1", "payload", newerThan("2"));
// then
assertThat(outcome).isEqualTo(CacheRecordOutcome.RECORDED);
}
}
@Test
@DisplayName("an unreachable Redis degrades: the cache is bypassed, nothing fails")
void anUnreachableRedisDegrades() {
@@ -23,6 +23,7 @@ class RedisCacheRegionContractTest extends CacheRegionContract {
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
limits.put(kind, 4);
}
limits.put(CacheScriptLane.CACHE_SCRIPT, 4);
return new RedisCacheRegionAdapter<>(
new RedisRuntimeOwner(
StubRedisRuntimeClient.serving(new InMemorySemanticRedisCommands()),
@@ -3,6 +3,9 @@ package dev.caskeleton.adapter.outbound.cache.redis.capability;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.Mockito.mock;
import dev.caskeleton.adapter.outbound.cache.redis.cache.CacheScriptLane;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisLaneContribution;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisLaneMetrics;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisRuntimeAutoConfiguration;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisRuntimeOwner;
import dev.caskeleton.adapter.outbound.support.startup.StartupFailureException;
@@ -11,6 +14,8 @@ import dev.caskeleton.application.idempotency.IdempotencyStorePort;
import dev.caskeleton.application.lease.DistributedLeasePort;
import dev.caskeleton.application.secret.SecretSourcePort;
import dev.caskeleton.shared.ratelimit.EdgeRateLimitPort;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.simple.SimpleMeterRegistry;
import java.time.Clock;
import java.time.Duration;
import java.util.Arrays;
@@ -94,6 +99,66 @@ class RedisCapabilityCompositionTest {
});
}
@Test
@DisplayName("Redis 캐시 binding은 설정한 상한으로 캐시 전용 script lane을 SCRIPT lane과 따로 기여한다")
void theCacheBindingContributesItsOwnScriptLane() {
runner
.withPropertyValues(
"app.capabilities.cache.bindings.default=redis",
"app.capabilities.cache.maximum-script-connections=3")
.run(
context -> {
// when
List<RedisLaneContribution> contributions =
List.copyOf(context.getBeansOfType(RedisLaneContribution.class).values());
// then
assertThat(context).hasNotFailed();
assertThat(contributions)
.containsExactly(new RedisLaneContribution(CacheScriptLane.CACHE_SCRIPT, 3));
});
}
@Test
@DisplayName("Redis 캐시 binding의 캐시 전용 lane은 SCRIPT lane과 따로 대기·거절·재연결 지표를 낸다")
void theCacheScriptLaneHasMetersOfItsOwn() {
runner
.withPropertyValues("app.capabilities.cache.bindings.default=redis")
.withBean(MeterRegistry.class, SimpleMeterRegistry::new)
.run(
context -> {
// when
MeterRegistry registry = context.getBean(MeterRegistry.class);
// then
assertThat(context).hasNotFailed();
for (String lane : List.of(CacheScriptLane.CACHE_SCRIPT.name(), "SCRIPT")) {
assertThat(
registry
.find(RedisLaneMetrics.CONNECTIONS_PENDING)
.tag("lane", lane)
.gauge())
.as("waiting borrows of lane %s", lane)
.isNotNull();
assertThat(
registry
.find(RedisLaneMetrics.ACQUIRE_REFUSED)
.tag("lane", lane)
.tag("reason", "saturated")
.counter())
.as("refusals of lane %s", lane)
.isNotNull();
assertThat(
registry
.find(RedisLaneMetrics.CONNECTIONS_OPENED)
.tag("lane", lane)
.counter())
.as("connections opened on lane %s", lane)
.isNotNull();
}
});
}
@Test
@DisplayName("a rate limiter with no configured policy is refused at startup")
void aRateLimiterWithoutPoliciesIsRefused() {
@@ -191,6 +191,85 @@ class RedisSemanticScriptAclProbeTest {
assertThat(failure).isNull();
}
@ParameterizedTest
@ValueSource(
strings = {
"LOADING Redis is loading the dataset in memory",
"Connection closed",
"BUSY Redis is busy running a script"
})
@DisplayName("cache만 고른 배포는 probe 도중 연결이 끊기거나 Redis가 아직 준비 중이어도 권한 오류로 보지 않고 기동한다")
void aCacheOnlyDeploymentStartsWhenTheProbeIsInterruptedByATransientFailure(String reply) {
// given
RedisCapabilitySettings capabilities = new RedisCapabilitySettings();
capabilities.getCache().getBindings().put("default", "redis");
RedisSemanticScriptAclProbe probe =
new RedisSemanticScriptAclProbe(
owner(gatewayFailingLoadWith(reply), new AtomicInteger()),
capabilities,
"prod:svc:shared:probe".getBytes(StandardCharsets.UTF_8),
Duration.ofSeconds(1));
// when
Throwable failure = catchThrowable(probe::afterSingletonsInstantiated);
// then
assertThat(failure).isNull();
}
@Test
@DisplayName("cache만 고른 배포도 probe가 기대와 다른 답을 받으면 기동에서 실패한다")
void aCacheOnlyDeploymentWhoseProbeReadsBackTheWrongValueFailsStartup() {
// given
RedisCapabilitySettings capabilities = new RedisCapabilitySettings();
capabilities.getCache().getBindings().put("default", "redis");
RedisSemanticScriptAclProbe probe =
new RedisSemanticScriptAclProbe(
owner(gatewayAnswering(List.of(0L)), new AtomicInteger()),
capabilities,
"prod:svc:shared:probe".getBytes(StandardCharsets.UTF_8),
Duration.ofSeconds(1));
// when
Throwable failure = catchThrowable(probe::afterSingletonsInstantiated);
// then
assertThat(failure)
.isInstanceOf(CapabilityStartupValidationException.class)
.hasMessageContaining("cache");
}
private static SemanticRedisCommands gatewayFailingLoadWith(String reply) {
return (SemanticRedisCommands)
Proxy.newProxyInstance(
SemanticRedisCommands.class.getClassLoader(),
new Class<?>[] {SemanticRedisCommands.class},
(proxy, method, arguments) ->
switch (method.getName()) {
case "loadScript" -> CompletableFuture.failedFuture(new RuntimeException(reply));
case "toString" -> "RedisSemanticScriptAclProbeTestGateway";
default ->
throw new UnsupportedOperationException(
"unexpected gateway method: " + method.getName());
});
}
private static SemanticRedisCommands gatewayAnswering(List<Object> reply) {
return (SemanticRedisCommands)
Proxy.newProxyInstance(
SemanticRedisCommands.class.getClassLoader(),
new Class<?>[] {SemanticRedisCommands.class},
(proxy, method, arguments) ->
switch (method.getName()) {
case "loadScript" -> CompletableFuture.completedFuture("probe-digest");
case "evaluateRegisteredForList" -> CompletableFuture.completedFuture(reply);
case "toString" -> "RedisSemanticScriptAclProbeTestGateway";
default ->
throw new UnsupportedOperationException(
"unexpected gateway method: " + method.getName());
});
}
private static RedisRuntimeOwner owner(SemanticRedisCommands gateway, AtomicInteger opened) {
RedisRuntimeClient client =
new RedisRuntimeClient() {
@@ -71,26 +71,15 @@ public final class RoleBasedFileAccessPolicy implements FileAccessPolicyPort {
if (permitted.stream().noneMatch(subject.roles()::contains)) {
throw denied(FileserverErrorCode.ACCESS_DENIED, "subject does not hold a permitted role");
}
if (descriptor.isPresent() && !isOwnedBy(descriptor.get(), subject)) {
// Ownership is FileAccessSubject.owns, the rule the port owns (owner decision 14): a file with
// no recorded owner predates ownership and belongs to nobody on the data plane. An admin role
// does not bypass it — acting on somebody else's file is what the audited management plane is
// for, and its operations carry no descriptor.
if (descriptor.isPresent() && !subject.owns(descriptor.get())) {
throw denied(FileserverErrorCode.ACCESS_DENIED, "subject does not own the file");
}
}
/**
* Whether the file was created by this caller, in this caller's tenant.
*
* <p>A file without a recorded owner predates ownership and belongs to nobody on the data plane.
* An admin role does not bypass this: acting on somebody else's file is what the audited
* management plane is for, and its operations carry no descriptor.
*/
private static boolean isOwnedBy(FileDescriptor descriptor, FileAccessSubject subject) {
return descriptor
.owner()
.filter(owner -> owner.principalId().equals(subject.principalId()))
.filter(owner -> owner.tenantId().equals(subject.tenant()))
.isPresent();
}
private static boolean isAnonymous(FileAccessSubject subject) {
return subject.principalId().equals(FileAccessSubject.anonymous().principalId());
}
@@ -0,0 +1,31 @@
package dev.caskeleton.adapter.outbound.fileserver.platform.security;
import dev.caskeleton.application.fileserver.api.security.FileAccessPolicyPort;
import dev.caskeleton.application.fileserver.api.security.FileAccessSubject;
import dev.caskeleton.application.fileserver.testkit.FileAccessPolicyPortContract;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
/** The production policy held to the ownership contract its fake is held to (RR-07-06). */
class RoleBasedFileAccessPolicyContractTest extends FileAccessPolicyPortContract {
private static final String READ = "ROLE_FILE_READ";
private static final String WRITE = "ROLE_FILE_WRITE";
private final RoleBasedFileAccessPolicy policy =
new RoleBasedFileAccessPolicy(Set.of(READ), Set.of(WRITE), Set.of("ROLE_FILE_ADMIN"), false);
@Override
protected FileAccessPolicyPort policy() {
return policy;
}
@Override
protected FileAccessSubject permittedSubject(String principalId, Optional<String> tenant) {
return new FileAccessSubject(
principalId,
Set.of(READ, WRITE),
tenant.map(value -> Map.of(FileAccessSubject.TENANT_ATTRIBUTE, value)).orElse(Map.of()));
}
}
+8 -2
View File
@@ -41,7 +41,11 @@ no reactive (`WebClient`/Reactor Netty) surface; it was removed on 2026-09-25 (d
cancels the request at the deadline, the JDK transport bounds its exchange timeout by the time
left, the experimental Jetty HTTP/3 transport aborts the request, and the caller gets
`HttpDeadlineExceededException`. A streaming download's setup (up to the response headers) runs
inside the same deadline; after hand-over each body gap is bounded by `read-idle`.
inside the same deadline; after hand-over each body gap is bounded by `read-idle` and nothing else,
on both transports — the JDK transport arms its own per-read timer for this, because the JDK client
has no socket read timeout. A body that fails after hand-over throws the same stable exceptions as
a buffered call (`HttpResponseTimeoutException` for a stalled body, `HttpResponseTruncatedException`
for a cut one).
- **Evidence.** Each attempt reports `NOT_SENT`, `SENT_NO_RESPONSE`, `RESPONSE_RECEIVED`, or
`PARTIAL_RESPONSE`. `NOT_SENT` is only claimed when a stage failure proves it.
- **Retry.** Never from the HTTP method alone. Idempotency, body replayability, evidence, deadline,
@@ -53,7 +57,9 @@ no reactive (`WebClient`/Reactor Netty) surface; it was removed on 2026-09-25 (d
whether the call throws them or returns them as a result (`RETURN_RESULT`). Any other 4xx (404,
409, 422 …) is a healthy answer, even when it reaches the caller as `HttpRemoteErrorException`.
Window, threshold and open-state wait are per profile under `clients[N].circuit-breaker.*`
(defaults are Resilience4j's; see `docs/httpclient/configuration-reference.md`).
(defaults are Resilience4j's; see `docs/httpclient/configuration-reference.md`). An attempt that
the caller's own deadline (`HttpOperation.deadline()`) ended is not counted at all: the caller ran
out of time, the upstream did not fail. An attempt the profile's `total-call` ended still counts.
- **Resources.** Connections and buffers are reclaimed after success, error, decode failure, size
rejection, and cancellation alike.
- **Secrets.** No URL, query value, path variable, token, cookie, or idempotency key reaches a log,
@@ -21,22 +21,35 @@ import org.springframework.http.client.JdkClientHttpRequestFactory;
* timeout is fixed per factory, so a factory is made per request with {@code min(response-header
* timeout, remaining deadline)} from {@link AttemptDeadline}. The factory only holds references;
* the client and the write executor are shared.
*
* <p>A streaming download cannot use that request. Its body is handed to the caller and may run for
* minutes, but Spring's timer only stops when the body stream is closed, so every JDK download
* longer than the response-header timeout was cut mid-body (review 2026-11-03 RR-04-03). Inside a
* streaming setup scope the factory makes a {@link JdkStreamingClientHttpRequest} instead: the
* JDK's own request timeout bounds the wait for headers, the attempt deadline cancels the exchange
* up to hand-over, and the body is bounded per gap by {@code read-idle}.
*/
final class DeadlineAwareJdkRequestFactory implements ClientHttpRequestFactory {
private final HttpClient client;
private final Executor writeExecutor;
private final Duration exchangeTimeout;
private final Duration readIdle;
DeadlineAwareJdkRequestFactory(HttpClient client, Duration exchangeTimeout) {
DeadlineAwareJdkRequestFactory(HttpClient client, Duration exchangeTimeout, Duration readIdle) {
this.client = Objects.requireNonNull(client, "jdk http client");
this.exchangeTimeout = Objects.requireNonNull(exchangeTimeout, "exchange timeout");
this.readIdle = Objects.requireNonNull(readIdle, "read idle timeout");
// The executor Spring would pick itself, chosen once instead of once per request.
this.writeExecutor = client.executor().orElseGet(SimpleAsyncTaskExecutor::new);
}
@Override
public ClientHttpRequest createRequest(URI uri, HttpMethod httpMethod) throws IOException {
if (AttemptDeadline.boundsSetupOnly()) {
return new JdkStreamingClientHttpRequest(
client, uri, httpMethod, AttemptDeadline.bound(exchangeTimeout), readIdle);
}
JdkClientHttpRequestFactory delegate = new JdkClientHttpRequestFactory(client, writeExecutor);
delegate.setReadTimeout(AttemptDeadline.bound(exchangeTimeout));
return delegate.createRequest(uri, httpMethod);
@@ -62,7 +62,8 @@ public final class JdkBlockingTransportProvider implements BlockingTransportProv
HttpClient client = clientFactory.create(profile, tlsMaterialResolver.apply(profile));
clients.put(new TransportResourceKey(profile.name(), generation), client);
ClientHttpRequestFactory factory =
new DeadlineAwareJdkRequestFactory(client, profile.timeout().responseHeader());
new DeadlineAwareJdkRequestFactory(
client, profile.timeout().responseHeader(), profile.timeout().readIdle());
listener.onRuntimeCreated(profile.name(), ID);
return factory;
}
@@ -0,0 +1,128 @@
package dev.caskeleton.adapter.outbound.httpclient.jdk;
import dev.caskeleton.adapter.outbound.httpclient.transport.AttemptDeadline;
import java.io.IOException;
import java.io.InputStream;
import java.io.InterruptedIOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.Objects;
import java.util.Set;
import java.util.TreeSet;
import java.util.concurrent.CancellationException;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.client.AbstractBufferingClientHttpRequest;
import org.springframework.http.client.ClientHttpResponse;
/**
* A JDK request for a streaming download, whose timeouts end at the response headers (review
* 2026-11-03 RR-04-03).
*
* <p>Three limits, each where the JDK can enforce it:
*
* <ul>
* <li>the JDK's own request timeout bounds the wait for the response headers and stops when they
* arrive;
* <li>the attempt deadline cancels the pending exchange, so connect, send and the header wait
* stay inside the call's total budget;
* <li>after hand-over, {@link ReadIdleInputStream} bounds each gap of the body by {@code
* read-idle} — the JDK client has no socket read timeout of its own.
* </ul>
*
* <p>The request body is buffered: a streaming download sends none or a small one, and the JDK
* needs a body publisher before the exchange starts.
*/
final class JdkStreamingClientHttpRequest extends AbstractBufferingClientHttpRequest {
/** Headers the JDK client refuses to set, as Spring's own JDK request lists them. */
private static final Set<String> RESTRICTED_HEADERS = restrictedHeaders();
private final HttpClient client;
private final URI uri;
private final HttpMethod method;
private final Duration responseHeaderTimeout;
private final Duration readIdle;
JdkStreamingClientHttpRequest(
HttpClient client,
URI uri,
HttpMethod method,
Duration responseHeaderTimeout,
Duration readIdle) {
this.client = Objects.requireNonNull(client, "jdk http client");
this.uri = Objects.requireNonNull(uri, "uri");
this.method = Objects.requireNonNull(method, "method");
this.responseHeaderTimeout =
Objects.requireNonNull(responseHeaderTimeout, "response header timeout");
this.readIdle = Objects.requireNonNull(readIdle, "read idle timeout");
}
@Override
public HttpMethod getMethod() {
return method;
}
@Override
public URI getURI() {
return uri;
}
@Override
protected ClientHttpResponse executeInternal(HttpHeaders headers, byte[] body)
throws IOException {
HttpRequest.Builder request =
HttpRequest.newBuilder(uri)
.timeout(responseHeaderTimeout)
.method(
method.name(),
body.length == 0
? HttpRequest.BodyPublishers.noBody()
: HttpRequest.BodyPublishers.ofByteArray(body));
headers.forEach(
(name, values) -> {
if (!RESTRICTED_HEADERS.contains(name)) {
values.forEach(value -> request.header(name, value));
}
});
CompletableFuture<HttpResponse<InputStream>> pending =
client.sendAsync(request.build(), HttpResponse.BodyHandlers.ofInputStream());
AttemptDeadline.abortOnExpiry(() -> pending.cancel(true));
HttpResponse<InputStream> response = await(pending);
return new JdkStreamingClientHttpResponse(
response, new ReadIdleInputStream(response.body(), readIdle));
}
private static HttpResponse<InputStream> await(
CompletableFuture<HttpResponse<InputStream>> pending) throws IOException {
try {
return pending.get();
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
pending.cancel(true);
InterruptedIOException failure = new InterruptedIOException("streaming setup interrupted");
failure.initCause(interrupted);
throw failure;
} catch (CancellationException cancelled) {
// Only the attempt deadline cancels this future; the gateway names the deadline.
throw new IOException("streaming setup cancelled at the attempt deadline", cancelled);
} catch (ExecutionException failed) {
Throwable cause = failed.getCause();
if (cause instanceof IOException io) {
throw io;
}
throw new IOException("streaming setup failed", cause);
}
}
private static Set<String> restrictedHeaders() {
Set<String> restricted = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
restricted.addAll(Set.of("connection", "content-length", "expect", "host", "upgrade"));
return restricted;
}
}
@@ -0,0 +1,63 @@
package dev.caskeleton.adapter.outbound.httpclient.jdk;
import java.io.IOException;
import java.io.InputStream;
import java.net.http.HttpResponse;
import java.util.Objects;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.HttpStatusCode;
import org.springframework.http.client.ClientHttpResponse;
/** The response of a {@link JdkStreamingClientHttpRequest}, body bounded per gap. */
final class JdkStreamingClientHttpResponse implements ClientHttpResponse {
private final HttpResponse<InputStream> response;
private final InputStream body;
private final HttpHeaders headers = new HttpHeaders();
JdkStreamingClientHttpResponse(HttpResponse<InputStream> response, InputStream body) {
this.response = Objects.requireNonNull(response, "jdk response");
this.body = Objects.requireNonNull(body, "body stream");
// HTTP/2 pseudo-headers (":status") are protocol framing, not response headers.
response
.headers()
.map()
.forEach(
(name, values) -> {
if (!name.startsWith(":")) {
headers.addAll(name, values);
}
});
}
@Override
public HttpStatusCode getStatusCode() {
return HttpStatusCode.valueOf(response.statusCode());
}
@Override
public String getStatusText() {
HttpStatus known = HttpStatus.resolve(response.statusCode());
return known == null ? "" : known.getReasonPhrase();
}
@Override
public HttpHeaders getHeaders() {
return headers;
}
@Override
public InputStream getBody() {
return body;
}
@Override
public void close() {
try {
body.close();
} catch (IOException ignored) {
// Closing releases the exchange; a failure to close has nothing left to release.
}
}
}
@@ -0,0 +1,90 @@
package dev.caskeleton.adapter.outbound.httpclient.jdk;
import java.io.FilterInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.SocketTimeoutException;
import java.time.Duration;
import java.util.Objects;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
/**
* A JDK response body whose every read is bounded by {@code read-idle} (review 2026-11-03
* RR-04-03).
*
* <p>The JDK client has no socket read timeout, so a streamed body that stopped arriving held the
* caller's thread and the connection forever. Each read arms a timer on the JDK's shared delay
* scheduler; when it fires the stream is closed, which wakes the blocked read, and the read reports
* a {@link SocketTimeoutException} — the same exception Apache's socket timeout raises, so one
* classifier reading names both. A read that returns first disarms its timer.
*/
final class ReadIdleInputStream extends FilterInputStream {
@FunctionalInterface
private interface Read {
long run() throws IOException;
}
private final Duration readIdle;
private final AtomicBoolean idled = new AtomicBoolean();
ReadIdleInputStream(InputStream delegate, Duration readIdle) {
super(Objects.requireNonNull(delegate, "body stream"));
this.readIdle = Objects.requireNonNull(readIdle, "read idle timeout");
}
@Override
public int read() throws IOException {
return (int) guarded(super::read);
}
@Override
public int read(byte[] buffer, int offset, int length) throws IOException {
return (int) guarded(() -> super.read(buffer, offset, length));
}
@Override
public long skip(long count) throws IOException {
return guarded(() -> super.skip(count));
}
private long guarded(Read read) throws IOException {
requireNotIdled();
CompletableFuture<Void> timer = new CompletableFuture<>();
CompletableFuture<Void> expiry =
timer.completeOnTimeout(null, readIdle.toNanos(), TimeUnit.NANOSECONDS).thenRun(this::idle);
try {
long result = read.run();
requireNotIdled();
return result;
} catch (IOException failure) {
requireNotIdled();
throw failure;
} finally {
// Completing first skips the idle action and lets the scheduler drop the pending task.
timer.cancel(false);
expiry.cancel(false);
}
}
private void idle() {
idled.set(true);
try {
in.close();
} catch (IOException ignored) {
// The read this wakes reports the timeout; a failed close has nothing more to say.
}
}
/**
* After the timer fired, whatever the read returned — data, end of stream or "closed" — is not
* the rest of the body.
*/
private void requireNotIdled() throws SocketTimeoutException {
if (idled.get()) {
throw new SocketTimeoutException("no response body bytes within the read-idle timeout");
}
}
}
@@ -17,6 +17,13 @@ public interface AttemptCircuitBreaker {
void onError(long durationNanos, Throwable failure);
/**
* Gives back a permission without recording an outcome, for an attempt that says nothing about
* the upstream (an attempt its caller's own deadline ended). Half-open trial slots are returned;
* the window is left as it was.
*/
void releasePermission();
String state();
static AttemptCircuitBreaker alwaysClosed() {
@@ -36,6 +43,11 @@ public interface AttemptCircuitBreaker {
// Nothing to record.
}
@Override
public void releasePermission() {
// Nothing was taken.
}
@Override
public String state() {
return "CLOSED";
@@ -60,6 +72,11 @@ public interface AttemptCircuitBreaker {
delegate.onError(durationNanos, TimeUnit.NANOSECONDS, failure);
}
@Override
public void releasePermission() {
delegate.releasePermission();
}
@Override
public String state() {
return delegate.getState().name();
@@ -7,6 +7,7 @@ import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpFailureMetadata;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRateLimitRejectedException;
import java.util.Objects;
import java.util.Optional;
import java.util.function.BooleanSupplier;
import java.util.function.Function;
import java.util.function.Supplier;
@@ -21,6 +22,11 @@ import java.util.function.Supplier;
* <p>A local rejection (rate limiter or bulkhead) is deliberately <em>not</em> recorded as a
* circuit error: the upstream never saw the request, and counting our own back-pressure as upstream
* failure would open the breaker on a healthy dependency.
*
* <p>An attempt its caller abandoned is not recorded either, for the same reason: an attempt that a
* caller's own deadline ended while the upstream was still answering within the profile's limits is
* the caller running out of time, not the upstream failing (review 2026-11-03 RR-04-01). Its
* permission is given back instead.
*/
public final class AttemptResiliencePipeline {
@@ -79,7 +85,24 @@ public final class AttemptResiliencePipeline {
* @return the attempt's value, whether or not it represents a remote failure
*/
public <T> T execute(AttemptCall<T> call, Function<T, Optional<Throwable>> remoteFailure) {
return execute(call, remoteFailure, () -> false);
}
/**
* Runs one attempt, recording nothing against the upstream when its caller abandoned it.
*
* @param call the attempt, including any redirect hops it follows
* @param remoteFailure returns the failure to record when the value represents an upstream error
* @param abandonedByCaller read after the attempt failed: whether the caller's own deadline ended
* it, in which case the failure is not the upstream's and the breaker records nothing
* @return the attempt's value, whether or not it represents a remote failure
*/
public <T> T execute(
AttemptCall<T> call,
Function<T, Optional<Throwable>> remoteFailure,
BooleanSupplier abandonedByCaller) {
Objects.requireNonNull(remoteFailure, "remote failure classifier");
Objects.requireNonNull(abandonedByCaller, "abandoned-by-caller signal");
if (!circuitBreaker.tryAcquirePermission()) {
rejections.circuitOpen();
throw new HttpCircuitOpenException(
@@ -110,7 +133,11 @@ public final class AttemptResiliencePipeline {
return result;
} catch (Throwable failure) {
releaseAttemptPermits();
circuitBreaker.onError(System.nanoTime() - started, failure);
if (abandonedByCaller.getAsBoolean()) {
circuitBreaker.releasePermission();
} else {
circuitBreaker.onError(System.nanoTime() - started, failure);
}
throw translate(failure);
}
}
@@ -10,11 +10,31 @@ import java.util.Objects;
*
* <p>Everything — pool acquire, DNS, connect, TLS, request write, response read, and every retry
* backoff — is spent inside this single budget.
*
* @param at the instant the call ends
* @param source who set that instant, which decides what an attempt it ends says about the upstream
*/
public record Deadline(Instant at) {
public record Deadline(Instant at, Source source) {
/**
* Who set the instant a call ends at.
*
* <p>The two say different things when they end an attempt. The profile's total-call timeout is
* the operator's statement of how long a healthy upstream may take, so an attempt it ends is the
* upstream's failure. A caller's own deadline is a statement about the caller — it had only that
* much time left — so an attempt it ends says nothing about the upstream (review 2026-11-03
* RR-04-01).
*/
public enum Source {
/** The profile's {@code timeout.total-call}. */
PROFILE_TOTAL_CALL,
/** The caller's {@code HttpOperation.deadline()}, when it is earlier than the profile's. */
CALLER
}
public Deadline {
Objects.requireNonNull(at, "deadline instant");
Objects.requireNonNull(source, "deadline source");
}
public Duration remaining(Clock clock) {
@@ -26,6 +46,11 @@ public record Deadline(Instant at) {
return !clock.instant().isBefore(at);
}
/** Whether the instant is the caller's own budget rather than the profile's total-call. */
public boolean setByCaller() {
return source == Source.CALLER;
}
/** The earlier of two deadlines; a parent budget can only ever shrink a child's. */
public Deadline earliest(Deadline other) {
return at.isBefore(other.at) ? this : other;
@@ -9,6 +9,9 @@ import java.util.Optional;
/**
* Derives the effective deadline from the caller's parent budget and the profile total-call timeout
* (design §15.2): {@code effectiveDeadline = min(parentDeadline, now + totalCallTimeout)}.
*
* <p>The result also records which of the two won. A tie goes to the profile: the profile's limit
* would have ended the call at that instant anyway.
*/
public final class DeadlineCalculator {
@@ -17,6 +20,9 @@ public final class DeadlineCalculator {
Objects.requireNonNull(totalCall, "total call timeout");
Objects.requireNonNull(clock, "clock");
Instant local = clock.instant().plus(totalCall);
return new Deadline(parent.map(value -> value.isBefore(local) ? value : local).orElse(local));
return parent
.filter(value -> value.isBefore(local))
.map(value -> new Deadline(value, Deadline.Source.CALLER))
.orElseGet(() -> new Deadline(local, Deadline.Source.PROFILE_TOTAL_CALL));
}
}
@@ -63,6 +63,7 @@ public final class BlockingAttemptExecutor {
try (AttemptDeadline.Scope scope = AttemptDeadline.open(deadline.remaining(support.clock()))) {
return executeWithin(
scope,
deadline,
runtime,
prepared,
responseType,
@@ -76,6 +77,7 @@ public final class BlockingAttemptExecutor {
private <T> AttemptOutcome<T> executeWithin(
AttemptDeadline.Scope scope,
Deadline deadline,
BlockingClientRuntime runtime,
PreparedOperation prepared,
ResponseType<T> responseType,
@@ -146,7 +148,12 @@ public final class BlockingAttemptExecutor {
tracker.firstByteDelivered());
}
},
outcome -> upstreamFailure(outcome, baseMetadata));
outcome -> upstreamFailure(outcome, baseMetadata),
// The caller's own deadline ending the attempt is not the upstream failing. The
// profile's total-call ending it is: that limit is the operator's statement of
// how long a healthy upstream may take, and a slow-drip upstream that never trips
// a per-gap timeout is seen by the breaker only through it (RR-04-01).
() -> deadline.setByCaller() && scope.expired());
return finish(observation, scope, attemptOutcome);
} catch (HttpClientException stable) {
return finish(
@@ -5,6 +5,7 @@ import dev.caskeleton.adapter.outbound.httpclient.api.HttpStatus;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpDeadlineExceededException;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpFailureMetadata;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRemoteErrorException;
import dev.caskeleton.adapter.outbound.httpclient.api.operation.AttemptStage;
import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence;
import dev.caskeleton.adapter.outbound.httpclient.api.operation.HttpOperation;
import dev.caskeleton.adapter.outbound.httpclient.api.result.BlockingStreamingResponse;
@@ -78,9 +79,11 @@ public final class BlockingStreamingGateway {
// which deregisters the transport's abort, so the body is then bounded by read-idle per gap
// and a long download is not cut at an arbitrary total. Without the scope a setup was bounded
// only by per-gap stage timeouts, and an upstream that held or dripped its headers kept the
// caller past its deadline (finding 05-06).
// caller past its deadline (finding 05-06). It is a setup scope, not an attempt scope: the
// JDK transport arms an exchange-wide timeout for an attempt, which cut every download
// longer than the response-header timeout (RR-04-03).
AttemptDeadline.Scope setupDeadline =
AttemptDeadline.open(setup.deadline().remaining(runtime.support().clock()));
AttemptDeadline.openStreamSetup(setup.deadline().remaining(runtime.support().clock()));
try (setupDeadline) {
response =
spec.exchange(
@@ -102,9 +105,24 @@ public final class BlockingStreamingGateway {
.withStatus(new HttpStatus(status))
.withEvidence(ExecutionEvidence.RESPONSE_RECEIVED));
}
// A body that fails after hand-over reaches the caller as the same stable
// exceptions a buffered call throws, classified by the transport's own
// classifier at the body stage — not as the engine's bare IOException, which
// carried no category, no evidence and nothing to log (RR-04-03).
InputStream bounded =
new CountingBoundedInputStream(
rawResponse.getBody(), limiter::recordWireBytes);
new ClassifiedBodyInputStream(
new CountingBoundedInputStream(
rawResponse.getBody(), limiter::recordWireBytes),
bodyFailure ->
runtime
.support()
.exceptionMapper()
.map(
runtime
.failureClassifier()
.classify(bodyFailure, AttemptStage.RESPONSE_BODY),
metadata,
bodyFailure));
return new DefaultBlockingStreamingResponse(
new HttpStatus(status), responseHeaders, bounded, rawResponse::close);
},
@@ -0,0 +1,64 @@
package dev.caskeleton.adapter.outbound.httpclient.restclient;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpClientException;
import java.io.FilterInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.Objects;
import java.util.function.Function;
/**
* A handed-over response body whose read failures reach the caller as stable exceptions (review
* 2026-11-03 RR-04-03).
*
* <p>A streamed body is read after the gateway has returned, so the gateway's own failure mapping
* never sees it. Without this a read-idle timeout or a cut connection reached the caller as the
* engine's bare {@code IOException} — no category, no evidence, nothing a retry policy or a log
* could branch on. Closing is the caller's own act and is passed through unclassified.
*/
final class ClassifiedBodyInputStream extends FilterInputStream {
private final Function<IOException, HttpClientException> classifier;
ClassifiedBodyInputStream(
InputStream delegate, Function<IOException, HttpClientException> classifier) {
super(Objects.requireNonNull(delegate, "body stream"));
this.classifier = Objects.requireNonNull(classifier, "body failure classifier");
}
@Override
public int read() throws IOException {
try {
return super.read();
} catch (IOException failure) {
throw classifier.apply(failure);
}
}
@Override
public int read(byte[] buffer, int offset, int length) throws IOException {
try {
return super.read(buffer, offset, length);
} catch (IOException failure) {
throw classifier.apply(failure);
}
}
@Override
public long skip(long count) throws IOException {
try {
return super.skip(count);
} catch (IOException failure) {
throw classifier.apply(failure);
}
}
@Override
public int available() throws IOException {
try {
return super.available();
} catch (IOException failure) {
throw classifier.apply(failure);
}
}
}
@@ -28,6 +28,11 @@ import java.util.concurrent.TimeUnit;
* <p>The scope is thread-confined because a blocking attempt runs on the caller's thread from
* request creation to the last body byte. When no scope is open — a transport used outside the
* gateway — both methods leave the transport's own behaviour unchanged.
*
* <p>A streaming download opens a different scope ({@link #openStreamSetup(Duration)}): it bounds
* the exchange only up to the response headers and is closed when the body is handed over. A
* transport whose engine timeout would otherwise keep running over the body reads {@link
* #boundsSetupOnly()} and keeps that timeout off it (review 2026-11-03 RR-04-03).
*/
public final class AttemptDeadline {
@@ -41,8 +46,34 @@ public final class AttemptDeadline {
* @param remaining the time left before the logical call's effective deadline
*/
public static Scope open(Duration remaining) {
return open(remaining, false);
}
/**
* Opens the deadline scope for a streaming download's setup on the current thread.
*
* <p>The caller closes it when the response is handed over, so the deadline never reaches the
* body: a download may legitimately outlast any fixed total (design §15.3).
*
* @param remaining the time left before the logical call's effective deadline
*/
public static Scope openStreamSetup(Duration remaining) {
return open(remaining, true);
}
/**
* Whether the current scope bounds only a streaming setup, so a timeout the transport arms must
* not outlive the response headers.
*/
public static boolean boundsSetupOnly() {
Scope scope = CURRENT.get();
return scope != null && scope.setupOnly;
}
private static Scope open(Duration remaining, boolean setupOnly) {
Objects.requireNonNull(remaining, "remaining deadline");
Scope scope = new Scope(remaining.isNegative() ? Duration.ZERO : remaining, CURRENT.get());
Scope scope =
new Scope(remaining.isNegative() ? Duration.ZERO : remaining, setupOnly, CURRENT.get());
CURRENT.set(scope);
return scope;
}
@@ -78,6 +109,7 @@ public final class AttemptDeadline {
public static final class Scope implements AutoCloseable {
private final long endNanos;
private final boolean setupOnly;
private final Scope outer;
private final CompletableFuture<Void> timer = new CompletableFuture<>();
private final CompletableFuture<Void> expiry;
@@ -85,8 +117,9 @@ public final class AttemptDeadline {
private boolean fired;
private boolean closed;
private Scope(Duration remaining, Scope outer) {
private Scope(Duration remaining, boolean setupOnly, Scope outer) {
this.endNanos = System.nanoTime() + remaining.toNanos();
this.setupOnly = setupOnly;
this.outer = outer;
// The JDK's shared delay scheduler, the same one Spring's JDK request uses for its timeout.
// Completing the future on close() also cancels the scheduled task, so a finished attempt
@@ -34,7 +34,8 @@ class DeadlineCalculatorTest {
@Test
void refusesAttemptWhenBackoffConsumesRemainingBudget() {
Deadline deadline = new Deadline(Instant.parse("2026-08-08T00:00:01Z"));
Deadline deadline =
new Deadline(Instant.parse("2026-08-08T00:00:01Z"), Deadline.Source.PROFILE_TOTAL_CALL);
Optional<AttemptBudget> result =
new AttemptBudgetCalculator(clock)
.nextAttempt(
@@ -44,7 +45,8 @@ class DeadlineCalculatorTest {
@Test
void grantsAttemptWhenBudgetCoversBackoffMinimumAndCleanup() {
Deadline deadline = new Deadline(Instant.parse("2026-08-08T00:00:02Z"));
Deadline deadline =
new Deadline(Instant.parse("2026-08-08T00:00:02Z"), Deadline.Source.PROFILE_TOTAL_CALL);
Optional<AttemptBudget> result =
new AttemptBudgetCalculator(clock)
.nextAttempt(
@@ -60,7 +62,8 @@ class DeadlineCalculatorTest {
assertThatThrownBy(
() ->
guard.requireTimeRemaining(
new Deadline(Instant.parse("2026-08-07T23:59:59Z")),
new Deadline(
Instant.parse("2026-08-07T23:59:59Z"), Deadline.Source.PROFILE_TOTAL_CALL),
CoreFixtures.notSentMetadata()))
.isInstanceOf(HttpDeadlineExceededException.class);
assertThatThrownBy(
@@ -147,7 +147,7 @@ class RetryCoordinatorTest {
@Override
public Deadline deadline() {
return new Deadline(clock.instant().plusSeconds(5));
return new Deadline(clock.instant().plusSeconds(5), Deadline.Source.PROFILE_TOTAL_CALL);
}
@Override
@@ -202,7 +202,7 @@ class RetryCoordinatorTest {
@Override
public Deadline deadline() {
return new Deadline(clock.instant().plusSeconds(5));
return new Deadline(clock.instant().plusSeconds(5), Deadline.Source.PROFILE_TOTAL_CALL);
}
@Override
@@ -5,20 +5,28 @@ import static org.assertj.core.api.Assertions.catchThrowable;
import dev.caskeleton.adapter.outbound.httpclient.api.OperationName;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpCircuitOpenException;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpDeadlineExceededException;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRemoteErrorException;
import dev.caskeleton.adapter.outbound.httpclient.api.operation.HttpOperation;
import dev.caskeleton.adapter.outbound.httpclient.api.result.HttpCallResult;
import dev.caskeleton.adapter.outbound.httpclient.api.result.ResponseType;
import dev.caskeleton.adapter.outbound.httpclient.profile.CircuitBreakerSettings;
import dev.caskeleton.adapter.outbound.httpclient.profile.TimeoutSettings;
import dev.caskeleton.adapter.outbound.httpclient.profile.TransportType;
import dev.caskeleton.adapter.outbound.httpclient.testkit.ClientProfiles;
import dev.caskeleton.adapter.outbound.httpclient.testkit.MockHttpServer;
import dev.caskeleton.adapter.outbound.httpclient.testkit.TestGateways;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.EnumSource;
/**
* What the per-profile circuit breaker counts as an upstream failure, end to end (finding 05-05).
@@ -31,6 +39,10 @@ class CircuitBreakerStatusAccountingTest {
private static final int CALLS = 200;
private static final int LIBRARY_DEFAULT_WINDOW = 100;
private static final int SMALL_WINDOW = 5;
private static final int TINY_WINDOW = 2;
private static final Duration HEALTHY_ANSWER_TIME = Duration.ofMillis(600);
private static final Duration CALLER_BUDGET = Duration.ofMillis(200);
private static final Duration PROFILE_TOTAL_CALL = Duration.ofSeconds(1);
@Test
@DisplayName("정상 응답인 404가 200번 이어져도 circuit이 열리지 않고 모든 호출이 상대 서버에 닿는다")
@@ -158,6 +170,113 @@ class CircuitBreakerStatusAccountingTest {
}
}
@ParameterizedTest(name = "{0}")
@EnumSource(
value = TransportType.class,
names = {"APACHE", "JDK"})
@DisplayName("호출자 deadline으로 끊은 시도는 상대 장애로 세지 않아 창을 채워도 다음 호출이 상대 서버에 닿는다")
void attemptsCutShortByTheCallersOwnDeadlineDoNotOpenTheCircuit(TransportType transport)
throws Exception {
try (MockHttpServer server = MockHttpServer.start();
TestGateways.Harness harness =
TestGateways.forProfile(
ClientProfiles.builder("caller-deadline")
.baseUrl(server.uri("/"))
.transport(transport)
.circuitBreaker(
new CircuitBreakerSettings(
50, SMALL_WINDOW, SMALL_WINDOW, Duration.ofMinutes(1), 1))
.build())) {
// given
for (int i = 0; i <= SMALL_WINDOW; i++) {
server.enqueueDelayedHeaders(
200,
"application/json",
"\"ok\"".getBytes(StandardCharsets.UTF_8),
HEALTHY_ANSWER_TIME);
}
List<Throwable> windowFailures = new ArrayList<>();
for (int i = 0; i < SMALL_WINDOW; i++) {
windowFailures.add(
catchThrowable(
() -> getUserWithin(harness, Optional.of(Instant.now().plus(CALLER_BUDGET)))));
}
assertThat(windowFailures)
.allSatisfy(f -> assertThat(f).isInstanceOf(HttpDeadlineExceededException.class));
// when
Throwable next = catchThrowable(() -> getUserWithin(harness, Optional.empty()));
// then
assertThat(next).isNull();
assertThat(server.requestCount()).isEqualTo(SMALL_WINDOW + 1);
}
}
@ParameterizedTest(name = "{0}")
@EnumSource(
value = TransportType.class,
names = {"APACHE", "JDK"})
@DisplayName("프로필 total-call 시간으로 끊긴 시도는 상대 장애로 세어 창을 채우면 circuit이 열린다")
void attemptsCutShortByTheProfileTotalCallStillOpenTheCircuit(TransportType transport)
throws Exception {
TimeoutSettings base = ClientProfiles.timeouts();
try (MockHttpServer server = MockHttpServer.start();
TestGateways.Harness harness =
TestGateways.forProfile(
ClientProfiles.builder("profile-deadline")
.baseUrl(server.uri("/"))
.transport(transport)
.timeout(
new TimeoutSettings(
base.dns(),
base.connect(),
base.tlsHandshake(),
base.responseHeader(),
base.readIdle(),
PROFILE_TOTAL_CALL))
.circuitBreaker(
new CircuitBreakerSettings(
50, TINY_WINDOW, TINY_WINDOW, Duration.ofMinutes(1), 1))
.build())) {
// given
List<Throwable> windowFailures = new ArrayList<>();
for (int i = 0; i < TINY_WINDOW; i++) {
server.enqueueSlowDripBody(200, 100, Duration.ofMillis(50));
windowFailures.add(catchThrowable(() -> getUserWithin(harness, Optional.empty())));
}
assertThat(windowFailures)
.allSatisfy(f -> assertThat(f).isInstanceOf(HttpDeadlineExceededException.class));
// when
Throwable next = catchThrowable(() -> getUserWithin(harness, Optional.empty()));
// then
assertThat(next).isInstanceOf(HttpCircuitOpenException.class);
assertThat(server.requestCount()).isEqualTo(TINY_WINDOW);
}
}
private static HttpCallResult<String> getUserWithin(
TestGateways.Harness harness, Optional<Instant> deadline) {
HttpOperation get = HttpOperation.get(new OperationName("get-user"), "/users/1", Map.of());
return harness
.gateway()
.exchange(
harness.profile().name(),
new HttpOperation(
get.operationName(),
get.method(),
get.uriTemplate(),
get.uriVariables(),
get.headers(),
get.body(),
get.idempotency(),
get.idempotencyKey(),
deadline),
ResponseType.of(String.class));
}
private static HttpCallResult<String> getUserAsResult(TestGateways.Harness harness) {
return harness
.gateway()
@@ -0,0 +1,124 @@
package dev.caskeleton.adapter.outbound.httpclient.restclient;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.catchThrowable;
import dev.caskeleton.adapter.outbound.httpclient.api.OperationName;
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpResponseTimeoutException;
import dev.caskeleton.adapter.outbound.httpclient.api.operation.AttemptStage;
import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence;
import dev.caskeleton.adapter.outbound.httpclient.api.operation.HttpOperation;
import dev.caskeleton.adapter.outbound.httpclient.api.result.BlockingStreamingResponse;
import dev.caskeleton.adapter.outbound.httpclient.profile.ClientProfile;
import dev.caskeleton.adapter.outbound.httpclient.profile.ResponseLimits;
import dev.caskeleton.adapter.outbound.httpclient.profile.TimeoutSettings;
import dev.caskeleton.adapter.outbound.httpclient.profile.TransportType;
import dev.caskeleton.adapter.outbound.httpclient.testkit.ClientProfiles;
import dev.caskeleton.adapter.outbound.httpclient.testkit.MockHttpServer;
import dev.caskeleton.adapter.outbound.httpclient.testkit.TestGateways;
import java.time.Duration;
import java.util.Map;
import java.util.Set;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.EnumSource;
/**
* What bounds a streaming download's body once it has been handed over (review 2026-11-03 RR-04-03,
* design §15.3).
*
* <p>The documented contract is transport-independent: the total budget and the response-header
* timeout bound the setup, and after hand-over each gap of the body is bounded by {@code read-idle}
* alone, so a long download is not cut at an arbitrary total. The JDK transport used to break it:
* Spring's JDK request enforces its read timeout over the whole exchange and closes the body stream
* when it fires, so every JDK download longer than the response-header timeout was cut, and the
* caller got an unclassified {@code IOException: closed}.
*
* <p>The profile's response-header timeout (500 ms) and total budget (1 s) are both shorter than
* the body (about 2 s), and the read-idle timeout (1 s) is longer than every gap of it.
*/
class StreamingBodyLifetimeTest {
private static final int BODY_BYTES = 20;
private static final Duration DRIP_GAP = Duration.ofMillis(100);
private static final Duration STALL_GAP = Duration.ofMillis(2500);
@ParameterizedTest(name = "{0}")
@EnumSource(
value = TransportType.class,
names = {"APACHE", "JDK"})
@DisplayName("넘겨받은 스트리밍 본문은 응답 헤더 시간과 전체 예산을 넘겨도 끝까지 읽힌다")
void aStreamedBodyOutlivesTheResponseHeaderTimeoutAndTheTotalBudget(TransportType transport)
throws Exception {
try (MockHttpServer server = MockHttpServer.start();
TestGateways.Harness harness = TestGateways.forProfile(profile(server, transport))) {
// given
server.enqueueSlowDripBody(200, BODY_BYTES, DRIP_GAP);
BlockingStreamingGateway gateway = new BlockingStreamingGateway(harness.registry());
// when
byte[] body;
try (BlockingStreamingResponse response = gateway.download(harness.profile().name(), get())) {
body = response.body().readAllBytes();
}
// then
assertThat(body).hasSize(BODY_BYTES);
}
}
@ParameterizedTest(name = "{0}")
@EnumSource(
value = TransportType.class,
names = {"APACHE", "JDK"})
@DisplayName("본문 사이 공백이 read-idle을 넘으면 분류된 응답 시간 초과로 끝난다")
void aBodyGapLongerThanReadIdleEndsWithAClassifiedTimeout(TransportType transport)
throws Exception {
try (MockHttpServer server = MockHttpServer.start();
TestGateways.Harness harness = TestGateways.forProfile(profile(server, transport))) {
// given
server.enqueueSlowDripBody(200, BODY_BYTES, STALL_GAP);
BlockingStreamingGateway gateway = new BlockingStreamingGateway(harness.registry());
long started = System.nanoTime();
// when
Throwable failure =
catchThrowable(
() -> {
try (BlockingStreamingResponse response =
gateway.download(harness.profile().name(), get())) {
response.body().readAllBytes();
}
});
// then
Duration elapsed = Duration.ofNanos(System.nanoTime() - started);
assertThat(failure).isInstanceOf(HttpResponseTimeoutException.class);
assertThat(elapsed).isLessThan(STALL_GAP);
HttpResponseTimeoutException timeout = (HttpResponseTimeoutException) failure;
assertThat(timeout.metadata().stage()).isEqualTo(AttemptStage.RESPONSE_BODY);
assertThat(timeout.metadata().evidence()).isEqualTo(ExecutionEvidence.PARTIAL_RESPONSE);
}
}
private static HttpOperation get() {
return HttpOperation.get(new OperationName("download"), "/download", Map.of());
}
private static ClientProfile profile(MockHttpServer server, TransportType transport) {
TimeoutSettings base = ClientProfiles.timeouts();
return ClientProfiles.builder("streaming-body")
.baseUrl(server.uri("/"))
.transport(transport)
.timeout(
new TimeoutSettings(
base.dns(),
base.connect(),
base.tlsHandshake(),
Duration.ofMillis(500),
Duration.ofSeconds(1),
Duration.ofSeconds(1)))
.response(new ResponseLimits(1024, 1024, Set.of("text/plain")))
.build();
}
}
@@ -68,6 +68,11 @@ public final class RecordingResilienceComponents {
events.add("circuit-error");
}
@Override
public void releasePermission() {
events.add("circuit-release");
}
@Override
public String state() {
return circuitOpen ? "OPEN" : "CLOSED";
+11 -4
View File
@@ -20,16 +20,22 @@ build choices) live in [README.md](README.md).
- `UuidV7DurableOperationIdFactory` — `DurableOperationIdFactory` implementation (UUIDv7 via
`uuid-creator`). The operation id is also the submission event's id, and the messaging platform's
message id accepts only UUIDv7 (review 2026-11-01 06-06). Wired by `composition:durable-operation`.
- The leaf name says "identifier"; general identifier generation, clock and random sources are not here.
- The leaf name says "identifier"; general-purpose clock and random sources are not here.
The name is kept because Gradle paths and consumers use it (README, review 05-21).
- Wiring is `composition/service-runtime` (`PseudonymizationConfig`), not this leaf.
## Allowed
- `:application-core` (declared in [build.gradle](build.gradle)); the registry entry is the SSOT.
Declared in [build.gradle](build.gradle); the registry entry is the SSOT.
- `:application-core` — the ports both classes implement.
- `:shared-contract`, for `HmacSha256` only (review 2026-11-01 01-11). The pseudonym is the hex HMAC
of the principal; `HmacUserPrincipalPseudonymizerVectorTest` pins it to a vector computed outside
the JVM, because logs written before and after a release are joined on it.
- `:domain-core` — a feature's server-assigned id adapter implements its domain `*IdFactory` port
here (registry edge `adapter:outbound:identifier -> domain-core`; `docs/manual/02-new-feature.md`).
- `com.github.f4b6a3:uuid-creator` (`libs.uuid.creator`) — UUIDv7 for
`UuidV7DurableOperationIdFactory`, the generator the sample's id adapters use.
## Forbidden
@@ -42,8 +48,9 @@ build choices) live in [README.md](README.md).
## Test
Pure unit tests, no Spring context (`HmacUserPrincipalPseudonymizerTest`, including the RFC 4231
known-answer vector).
Pure unit tests, no Spring context: `HmacUserPrincipalPseudonymizerTest` (including the RFC 4231
known-answer vector), `HmacUserPrincipalPseudonymizerVectorTest` and
`UuidV7DurableOperationIdFactoryTest`.
```bash
cd src
+10 -5
View File
@@ -9,7 +9,8 @@
## 이름과 내용이 다른 이유 (review 05-21)
leaf 이름은 `identifier`지만 범용 식별자 생성기·clock·random 소스는 여기 없다. 식별자 생성은
durable operation id(UUIDv7) 하나뿐이다(아래). 예전 문서가 범용 생성기 계획을 현재 내용처럼 적었다. 이름은 바꾸지 않았다: Gradle 경로·registry id·소비자
durable operation id(UUIDv7) 하나뿐이고(아래), 기능이 서버에서 id를 정하면 그 기능의 domain
`*IdFactory` 구현도 여기 둔다(`:domain-core` 의존이 그 때문이다). 예전 문서가 범용 생성기 계획을 현재 내용처럼 적었다. 이름은 바꾸지 않았다: Gradle 경로·registry id·소비자
(`composition:service-runtime`, `app-bootstrap`, `sample-portfolio`)의 의존 선언이 모두 이 이름을
쓰고, 바꿔서 얻는 것은 이름 하나뿐이다. 외부 연동이 없는 crypto 능력을 새로 둘 곳이 필요하면 이
leaf에 둔다. 그때 이 절을 고친다.
@@ -25,10 +26,10 @@ leaf에 둔다. 그때 이 절을 고친다.
대소문자를 바꾸면 테스트가 실패한다.
- Salt: `APP_PRIVACY_PSEUDONYMIZATION_SALT`(secret, 90일 회전)에서 온다. 이 클래스는 salt를 스스로
읽지 않고 생성자로 받는다. 받은 배열은 복사해 둔다.
- Thread safety: `Mac`은 thread-safe하지 않아서 호출마다 새로 만든다. 그래서 싱글톤 bean으로
공유해도 된다. `HmacSHA256`은 JDK 필수 알고리즘이라 `NoSuchAlgorithmException`·
`InvalidKeyException`은 사실상 나지 않는다. checked exception을 호출부에 남기지 않으려고
`IllegalStateException`으로 감싼다.
- 서명: 이 클래스는 `Mac`을 직접 다루지 않는다. shared-contract의 `HmacSha256.sign`에 맡긴다(리뷰
2026-11-01 01-11). `HmacSha256`이 호출마다 새 `Mac`을 만들고(`Mac`은 thread-safe하지 않다),
JDK 필수 알고리즘이라 사실상 나지 않는 checked exception을 `IllegalStateException`으로 감싼다.
그래서 이 클래스는 싱글톤 bean으로 공유해도 된다.
## UuidV7DurableOperationIdFactory
@@ -46,6 +47,10 @@ Spring 어노테이션이 없다. bean은
## 빌드 결정
의존은 넷이다: `:application-core`(두 클래스가 구현하는 port), `:shared-contract`(`HmacSha256`),
`:domain-core`(기능의 `*IdFactory` 구현 자리), `uuid-creator`(UUIDv7). 목록과 이유는
[CLAUDE.md](CLAUDE.md) Allowed 절이 기준이다.
`:application-core`를 `implementation`으로 받는다. 이 leaf의 public 타입 시그니처에
application-core 타입이 드러나지만, 조립하는 쪽이 application-core를 이미 classpath에 갖고 있다.
허용 edge는 `src/config/architecture/modules.json`이 기준이다.
@@ -1,5 +1,7 @@
/**
* The HMAC-SHA-256 user-principal pseudonymizer. Despite the leaf name, no identifier generation,
* clock or random source lives here; see README for why the name was kept.
* Two identifier adapters with no external IO: the HMAC-SHA-256 user-principal pseudonymizer and
* the UUIDv7 durable operation id factory. A feature's server-assigned id adapter for a domain
* {@code *IdFactory} port belongs here too. General clock and random sources do not live here; see
* README for why the leaf name was kept.
*/
package dev.caskeleton.adapter.outbound.identifier;
@@ -0,0 +1,14 @@
# adapter:outbound:messaging-kafka
- Module ID `adapter-outbound-messaging-kafka`, Gradle path `:adapter:outbound:messaging-kafka`.
Registry SSOT: `src/config/architecture/modules.json`.
- Responsibility: the Kafka publishing capability as one `runtimeOnly` line. It has no production
code: it bundles `adapter:outbound:messaging` (the application's bridge) with
`messaging:messaging-spring-boot-starter-kafka` (the vendored platform and its Kafka transport) and
ships the module defaults in `config/module-defaults/messaging.yml`.
- Boundaries: add no code here. A publish rule belongs to the bridge (`adapter:outbound:messaging`),
a Kafka rule to the vendored product (`src/messaging/CLAUDE.md`); a class here would be a third
place. Allowed edges come only from the registry entry.
- Tests: activation tests (`./gradlew :adapter:outbound:messaging-kafka:test`) and the real-Kafka
lane `messagingKafkaRelayIntegrationTest` (relay → bridge → broker record bytes; `:integrationCheck`).
- Design: `README.md`.
@@ -8,6 +8,7 @@ import dev.caskeleton.application.outbox.OutboxAppendPort;
import dev.caskeleton.application.outbox.OutboxCompletion;
import dev.caskeleton.application.outbox.OutboxEventStatus;
import dev.caskeleton.application.outbox.OutboxLease;
import dev.caskeleton.application.outbox.OutboxLongRetries;
import dev.caskeleton.application.outbox.OutboxStorePort;
import dev.caskeleton.application.outbox.testkit.InMemoryOutboxStore;
import dev.caskeleton.application.outbox.testkit.OutboxEvents;
@@ -282,13 +283,15 @@ class ApplicationRelayKafkaIntegrationTest {
}
@Override
public synchronized OutboxCompletion markFailed(OutboxLease lease, Instant nextAttemptAt) {
return delegate.markFailed(lease, nextAttemptAt);
public synchronized OutboxCompletion markFailed(
OutboxLease lease, Instant nextAttemptAt, String failureCode) {
return delegate.markFailed(lease, nextAttemptAt, failureCode);
}
@Override
public synchronized OutboxCompletion markDeferred(OutboxLease lease, Instant nextAttemptAt) {
return delegate.markDeferred(lease, nextAttemptAt);
public synchronized OutboxCompletion markDeferred(
OutboxLease lease, Instant nextAttemptAt, String failureCode) {
return delegate.markDeferred(lease, nextAttemptAt, failureCode);
}
@Override
@@ -297,8 +300,13 @@ class ApplicationRelayKafkaIntegrationTest {
}
@Override
public synchronized OutboxCompletion markDead(OutboxLease lease) {
return delegate.markDead(lease);
public synchronized OutboxCompletion markDead(OutboxLease lease, String failureCode) {
return delegate.markDead(lease, failureCode);
}
@Override
public synchronized OutboxLongRetries longRetries(int minDeferredClaims, Instant now) {
return delegate.longRetries(minDeferredClaims, now);
}
@Override
+1 -1
View File
@@ -5,7 +5,7 @@
- Module ID: `adapter-outbound-messaging`
- Gradle path: `:adapter:outbound:messaging`
- Focused test (derived from Gradle path): `./gradlew :adapter:outbound:messaging:test --console=plain`
- Runtime baseline: Java 21; repository framework baseline: Spring Boot 4.0.0.
- Runtime baseline: Java 21; repository framework baseline: Spring Boot 4.0.8.
- Registry SSOT: `src/config/architecture/modules.json`.
Package root: `dev.caskeleton.adapter.outbound.messaging`.
+27 -8
View File
@@ -42,13 +42,33 @@ platform `REJECTED` 중 `FailureDescriptor.retryable()`이 true인 것(종료
멈춘다(2026-11-01 리뷰 06-01). `AMBIGUOUS`로 번역하지 않은 이유: 동작은 같지만 "전송됐을 수
있음"이라는 거짓 정보가 남아 중복 경고를 오염시킨다.
### 실패 원인 코드
결과(outcome)는 relay가 무엇을 할지만 정한다. 종료 중 거부, 닫힌 transport, 거부된 credential은 모두
`REJECTED_RETRYABLE` 하나다. 그런데 이 셋은 끝이 없는 재시도다(시도 예산을 쓰지 않는다). 그래서 bridge는
platform `FailureDescriptor.code()`를 버리지 않고 `OutboxPublishAttempt.failureCode()`로 함께 넘긴다
(재리뷰 2026-11-03 RR-05-03). relay는 그 코드를 행의 `last_failure_code`(V22)와 실패 로그의
`error.reason`에 남긴다. 운영자는 이 값으로 "credential이 진짜 틀렸다"와 "broker가 이 레코드만 받지
않는다"를 가른다(`docs/runbooks/outbox-publish-failed.md` §4.1).
- platform code는 이미 정제된 이름이다(`KAFKA_AUTHENTICATIONEXCEPTION`, `PUBLISH_DEADLINE_EXCEEDED` 등).
driver 메시지가 아니므로 그대로 옮긴다. application 코드 문법(`[A-Za-z0-9][A-Za-z0-9._:-]*`, 120자)에
맞지 않는 code는 그 실패의 `FailureCategory` 이름으로 옮긴다.
- platform으로 옮길 수 없는 event(아래 id 규칙)는 bridge가 `EVENT_NOT_REPRESENTABLE`로 거부한다.
- 버린 대안: relay가 platform 타입을 직접 보는 것(경계가 깨진다), outcome 값을 원인별로 늘리는 것(relay가
할 일은 같은데 분기만 는다).
### event id → message id
application `EventId`는 문자열이고 platform `MessageId`는 **UUIDv7**만 받는다. bridge는 event id가
정규 소문자 UUIDv7 문자열일 때만 같은 값으로 옮기고, 그 밖(대문자·축약 표기·v4 등 다른 버전)은 다른
문자열의 message id로 바꿔 보내지 않고 `REJECTED_BEFORE_SEND`로 거부한다(relay는 DEAD). 소비자 중복
제거 키가 event id라서 값을 바꿔 보내면 중복 제거가 깨지기 때문이다(리뷰 2026-11-01 06-06). 이 거부는
append 시점(`EventId` 문법)에서 먼저 일어나야 한다 — application 쪽 문법 일원화는 별도 작업이다.
제거 키가 event id라서 값을 바꿔 보내면 중복 제거가 깨지기 때문이다(리뷰 2026-11-01 06-06).
같은 규칙이 append 쪽에 먼저 있다. `IntegrationEventDraft`가 정규 소문자 UUIDv7이 아닌 event id와
causation id를 거부하므로(`IntegrationIdentity.isCanonicalUuidV7`), 그런 id는 업무 트랜잭션 안에서 실패한다
(재리뷰 2026-11-03 RR-05-04). bridge의 거부는 이 규칙 전에 쓰인 행(저장 문법 `EventId`는 그대로 넓다)을 위한
마지막 방어선이다.
## outbox 발행 포트에는 대역이 없다
@@ -70,9 +90,10 @@ event가 다시 publish될 수 있다. 같은 event id로 다시 나가므로 co
있으면 `messaging-platform`이다. 이 adapter는 confirmed FAILED/DEAD report 하나를 SLF4J 2 fluent
ERROR 하나로 렌더링한다.
공통 field는 `error.code`, `error.category`, `dependency_name`,
공통 field는 `error.code`, `error.category`, `error.cause_type`, `error.reason`, `dependency_name`,
`dependency_type=messaging`, `outcome`, `event_id`, `event_type`, `aggregate_id`,
`correlation_id`, `attempt_count`, `runbook_link`다. retry report만 `next_attempt_at`을 추가한다.
`error.reason`은 위 실패 원인 코드다.
payload/envelope/exception-derived field는 받거나 렌더링하지 않는다. logging 내부
`RuntimeException`은 adapter와 use case 양쪽에서 방어하므로 persisted FAILED/DEAD outcome을 바꾸지
않는다.
@@ -85,10 +106,8 @@ payload/envelope/exception-derived field는 받거나 렌더링하지 않는다.
`envelope`·`realtime`)은 2026-09-25에 삭제됐고 복구 경로는 `docs/capability-status.md` 제거 표에
있다.
공통 build evidence manifest 검증은 build-tools로 옮겼다. 루트 `verifyMessagingContracts`
(`VerifyMessagingEvidenceTask`)가 application-core·shared-contract 두 qualification 레인 결과로
`build/messaging-evidence/contracts-schema/manifest.json`을 쓰고, 쓴 바이트를 바로 common schema로
검증한다. 이 leaf는 그 검증에 관여하지 않는다.
integration event 계약 검증은 루트 `verifyMessagingContracts`(`src/build.gradle`)가 한다. application-core·
shared-contract·sample-portfolio의 qualification lane 셋을 묶은 task다. 이 leaf는 그 검증에 관여하지 않는다.
canonical path는 platform `PublishResult`의 confirmation/transmission evidence를 application
`OutboxPublishOutcome`으로 명시적으로 변환한다.
`OutboxPublishOutcome`으로 명시적으로 변환하고, 실패 code를 `OutboxPublishAttempt`에 함께 싣는다.
@@ -4,17 +4,21 @@ import dev.caskeleton.application.messaging.event.IntegrationEventPublishPort;
import dev.caskeleton.application.outbox.ClaimedOutboxEvent;
import dev.caskeleton.application.outbox.OutboxMessagePublishPort;
import dev.caskeleton.application.outbox.OutboxPublishAmbiguousException;
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
import dev.caskeleton.application.outbox.OutboxPublishRefusedException;
import java.time.Duration;
import java.util.Objects;
import java.util.Optional;
/**
* The relay's publish port: a claimed row's validated event, delegated unchanged to the canonical
* {@link IntegrationEventPublishPort}.
*
* <p>The event is the one the append validated, restored from its exact envelope bytes; nothing
* here re-renders it. The platform's four-valued outcome is passed through, so the relay can tell a
* possibly-stored message (ambiguous) from a refused one.
* here re-renders it. The canonical publisher's attempt — its outcome and its failure code — is
* passed through unchanged, so the relay can tell a possibly-stored message (ambiguous) from a
* refused one, and a refused credential from a closed transport.
*/
public final class OutboxMessagePublishAdapter implements OutboxMessagePublishPort {
@@ -27,7 +31,7 @@ public final class OutboxMessagePublishAdapter implements OutboxMessagePublishPo
@Override
public void publish(ClaimedOutboxEvent event) {
OutboxPublishOutcome outcome = publishForOutcome(event);
OutboxPublishOutcome outcome = publishForOutcome(event).outcome();
if (outcome == OutboxPublishOutcome.CONFIRMED) {
return;
}
@@ -37,8 +41,14 @@ public final class OutboxMessagePublishAdapter implements OutboxMessagePublishPo
throw new OutboxPublishRefusedException(outcome);
}
/** The canonical publisher's, unchanged: this adapter adds no wait of its own. */
@Override
public OutboxPublishOutcome publishForOutcome(ClaimedOutboxEvent event) {
public Optional<Duration> longestAttempt() {
return canonicalPublisher.longestAttempt();
}
@Override
public OutboxPublishAttempt publishForOutcome(ClaimedOutboxEvent event) {
Objects.requireNonNull(event, "event must not be null");
return canonicalPublisher.publish(event.event()).toCompletableFuture().join();
}
@@ -48,6 +48,10 @@ public final class Slf4jOutboxRelayFailureReportAdapter implements OutboxRelayFa
// occasionally credentials. The class name is a type; the code is bounded.
.addKeyValue("error.code", report.code().code())
.addKeyValue("error.cause_type", report.causeType())
// Why, in the publisher's words: the cause type of a returned outcome names only
// which
// outcome it was, and a refused credential and a closed transport are one outcome.
.addKeyValue("error.reason", report.failureCode())
.addKeyValue("error.category", report.code().category().name())
.addKeyValue("dependency_name", dependencyName)
.addKeyValue("dependency_type", DEPENDENCY_TYPE)
@@ -2,6 +2,7 @@ package dev.caskeleton.adapter.outbound.messaging.platformbridge;
import dev.caskeleton.application.messaging.event.IntegrationEventPublishPort;
import dev.caskeleton.application.messaging.event.ValidatedIntegrationEvent;
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
import dev.caskeleton.messaging.api.CausationId;
import dev.caskeleton.messaging.api.ContentType;
@@ -26,6 +27,7 @@ import dev.caskeleton.messaging.api.publish.TransmissionEvidence;
import dev.caskeleton.messaging.schema.EncodedMessage;
import dev.caskeleton.messaging.schema.EncodedMessagePublisher;
import dev.caskeleton.messaging.schema.SchemaReference;
import java.time.Duration;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.Objects;
@@ -45,6 +47,12 @@ import java.util.concurrent.CompletionStage;
*/
public final class PlatformIntegrationEventPublishAdapter implements IntegrationEventPublishPort {
/**
* The failure code of an event this bridge refuses before the platform sees it: a value — an id
* above all — the platform's types cannot carry.
*/
public static final String NOT_REPRESENTABLE = "EVENT_NOT_REPRESENTABLE";
private final EncodedMessagePublisher publisher;
private final ProducerId producer;
@@ -55,7 +63,7 @@ public final class PlatformIntegrationEventPublishAdapter implements Integration
}
@Override
public CompletionStage<OutboxPublishOutcome> publish(ValidatedIntegrationEvent event) {
public CompletionStage<OutboxPublishAttempt> publish(ValidatedIntegrationEvent event) {
Objects.requireNonNull(event, "event must not be null");
final PreparedPublish prepared;
try {
@@ -64,12 +72,27 @@ public final class PlatformIntegrationEventPublishAdapter implements Integration
// Only a value the platform cannot represent is a definite refusal: the same payload is
// refused again, so the relay dead-letters it at once. Any other failure is not evidence of
// that and propagates, so the relay treats it as a retryable failure with its real cause.
return CompletableFuture.completedFuture(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
return CompletableFuture.completedFuture(
OutboxPublishAttempt.failed(
OutboxPublishOutcome.REJECTED_BEFORE_SEND, NOT_REPRESENTABLE));
}
return publisher
.publishEncoded(prepared.destination(), prepared.envelope(), PublishOptions.defaults())
.thenApply(PlatformIntegrationEventPublishAdapter::mapOutcome);
.thenApply(PlatformIntegrationEventPublishAdapter::translate);
}
/**
* Twice the publish deadline this bridge asks for ({@link PublishOptions#defaults()}).
*
* <p>The platform measures the deadline from the call, but a transport may block inside its send
* before the deadline is applied to the stage it returns — Kafka's {@code max.block.ms} is set to
* the same default for exactly that wait — so an attempt can take the deadline twice over. The
* relay sizes its batch against this (review 2026-11-01 06-12).
*/
@Override
public Optional<Duration> longestAttempt() {
return Optional.of(PublishOptions.defaults().timeout().multipliedBy(2));
}
private PreparedPublish prepare(ValidatedIntegrationEvent event) {
@@ -146,8 +169,10 @@ public final class PlatformIntegrationEventPublishAdapter implements Integration
* </ul>
*
* <p>Either refusal is {@code REJECTED_BEFORE_SEND}: the same id is refused again, so the relay
* dead-letters it. The append side's {@code EventId} grammar is where such an id should fail
* first, in the writer's transaction; until it does, this is where the mismatch is caught.
* dead-letters it. A new event fails earlier, in the writer's transaction: {@code
* IntegrationEventDraft} refuses an id that is not a canonical lower-case UUIDv7 (re-review
* 2026-11-03 RR-05-04). This check stays for rows appended before that rule — the stored {@code
* EventId} grammar is wider, so they can still be read back.
*
* @throws IllegalArgumentException when the value is not a canonical lower-case UUIDv7
*/
@@ -177,8 +202,37 @@ public final class PlatformIntegrationEventPublishAdapter implements Integration
* committed event, and its aggregate behind it, for a cause that clears on its own. It gets its
* own value instead.
*/
private static OutboxPublishOutcome mapOutcome(PublishResult result) {
private static OutboxPublishAttempt translate(PublishResult result) {
Objects.requireNonNull(result, "publish result must not be null");
OutboxPublishOutcome outcome = mapOutcome(result);
if (outcome == OutboxPublishOutcome.CONFIRMED) {
return OutboxPublishAttempt.confirmed();
}
return result
.failure()
.map(PlatformIntegrationEventPublishAdapter::failureCode)
.map(code -> OutboxPublishAttempt.failed(outcome, code))
.orElseGet(() -> OutboxPublishAttempt.of(outcome));
}
/**
* The platform's failure code, carried as the application's failure code (re-review 2026-11-03
* RR-05-03).
*
* <p>The outcome is all the relay acts on, and it loses the reason on purpose: a shutdown, a
* closed transport and a refused credential are one {@code REJECTED_RETRYABLE}. The code keeps it
* for the operator, on the row and in the failure log. The platform's code is already sanitized —
* a stable name, never a driver message — and passes as it is; one outside the application's code
* grammar (the platform only bounds its length) is carried as the failure's category instead, a
* name the platform owns too.
*/
private static String failureCode(FailureDescriptor failure) {
return OutboxPublishAttempt.isFailureCode(failure.code())
? failure.code()
: failure.category().name();
}
private static OutboxPublishOutcome mapOutcome(PublishResult result) {
if (result.completion() == PublishCompletion.CONFIRMED) {
return OutboxPublishOutcome.CONFIRMED;
}
@@ -5,7 +5,7 @@ import static org.assertj.core.api.Assertions.assertThat;
import dev.caskeleton.adapter.outbound.messaging.outbox.OutboxMessagePublishAdapter;
import dev.caskeleton.application.messaging.event.IntegrationEventPublishPort;
import dev.caskeleton.application.outbox.OutboxMessagePublishPort;
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
import java.util.concurrent.CompletableFuture;
import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
@@ -32,7 +32,7 @@ class MessagingConfigTest {
runner
.withBean(
IntegrationEventPublishPort.class,
() -> event -> CompletableFuture.completedFuture(OutboxPublishOutcome.CONFIRMED))
() -> event -> CompletableFuture.completedFuture(OutboxPublishAttempt.confirmed()))
.run(
context ->
assertThat(context.getBean(OutboxMessagePublishPort.class))
@@ -51,7 +51,14 @@ class Slf4jOutboxRelayFailureReportAdapterTest {
adapter.report(
OutboxRelayFailureReport.retryableFailure(
"evt-1", "WorkLogReserved", "agg-1", "corr-1", 2, nextAttemptAt, cause));
"evt-1",
"WorkLogReserved",
"agg-1",
"corr-1",
2,
nextAttemptAt,
cause,
"KAFKA_AUTHENTICATIONEXCEPTION"));
assertThat(appender.list).hasSize(1);
ILoggingEvent event = appender.list.getFirst();
@@ -71,6 +78,7 @@ class Slf4jOutboxRelayFailureReportAdapterTest {
Map.entry("correlation_id", "corr-1"),
Map.entry("attempt_count", 2),
Map.entry("error.cause_type", "java.lang.RuntimeException"),
Map.entry("error.reason", "KAFKA_AUTHENTICATIONEXCEPTION"),
Map.entry("runbook_link", "runbook://outbox/publish-failed"),
Map.entry("next_attempt_at", "2026-07-25T01:02:03Z")));
assertThat(event.getThrowableProxy())
@@ -89,7 +97,7 @@ class Slf4jOutboxRelayFailureReportAdapterTest {
adapter.report(
OutboxRelayFailureReport.deadLetter(
"evt-2", "WorkLogReserved", "agg-2", "corr-2", 3, cause));
"evt-2", "WorkLogReserved", "agg-2", "corr-2", 3, cause, "REJECTED_BEFORE_SEND"));
assertThat(appender.list).hasSize(1);
ILoggingEvent event = appender.list.getFirst();
@@ -142,7 +150,8 @@ class Slf4jOutboxRelayFailureReportAdapterTest {
"agg-3",
"corr-3",
3,
new RuntimeException("broker down"))))
new RuntimeException("broker down"),
"RuntimeException")))
.doesNotThrowAnyException();
}
@@ -12,6 +12,7 @@ import dev.caskeleton.application.messaging.event.IntegrationEventPublishPort;
import dev.caskeleton.application.messaging.event.ValidatedIntegrationEvent;
import dev.caskeleton.application.messaging.event.testkit.BrokerAnswer;
import dev.caskeleton.application.messaging.event.testkit.IntegrationEventPublishContract;
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
import dev.caskeleton.messaging.api.ContentType;
import dev.caskeleton.messaging.api.MessageEnvelope;
@@ -59,7 +60,7 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
ValidatedIntegrationEvent event = event(EVENT_ID, Optional.of(CAUSATION_ID));
assertThat(bridge.publish(event).toCompletableFuture().join())
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
.isEqualTo(OutboxPublishOutcome.CONFIRMED);
MessageDestination<EncodedMessage> destination = publisher.destination;
@@ -99,19 +100,19 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
ValidatedIntegrationEvent event = event(EVENT_ID, Optional.empty());
assertThat(bridge.publish(event).toCompletableFuture().join())
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
.isEqualTo(OutboxPublishOutcome.CONFIRMED);
publisher.result = ambiguous();
assertThat(bridge.publish(event).toCompletableFuture().join())
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
.isEqualTo(OutboxPublishOutcome.AMBIGUOUS);
publisher.result = rejected(PublishEvidence.notTransmitted());
assertThat(bridge.publish(event).toCompletableFuture().join())
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
.isEqualTo(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
publisher.result =
rejected(
new PublishEvidence(
true, TransmissionEvidence.TRANSMITTED, false, ConfirmationLevel.NONE));
assertThat(bridge.publish(event).toCompletableFuture().join())
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
.isEqualTo(OutboxPublishOutcome.REJECTED_AFTER_BROKER);
}
@@ -132,12 +133,101 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
// when
OutboxPublishOutcome outcome =
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join();
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join().outcome();
// then
assertThat(outcome).isEqualTo(OutboxPublishOutcome.REJECTED_RETRYABLE);
}
@Test
@DisplayName("platform이 붙인 실패 코드는 원인 코드로 그대로 옮긴다")
void thePlatformsFailureCodeIsCarriedAsTheFailureCode() {
// given: the broker refused this producer's credentials
RecordingPublisher publisher =
new RecordingPublisher(
PublishResult.rejected(
Duration.ZERO,
new FailureDescriptor(
FailureCategory.AUTHENTICATION,
"KAFKA_AUTHENTICATIONEXCEPTION",
true,
"kafka publish failed with AuthenticationException",
Optional.of("AuthenticationException"))));
PlatformIntegrationEventPublishAdapter bridge =
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
// when
OutboxPublishAttempt attempt =
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join();
// then
assertThat(attempt)
.isEqualTo(
OutboxPublishAttempt.failed(
OutboxPublishOutcome.REJECTED_RETRYABLE, "KAFKA_AUTHENTICATIONEXCEPTION"));
}
@Test
@DisplayName("코드 문법에 맞지 않는 platform 실패 코드는 그 실패의 분류 이름으로 옮긴다")
void aPlatformCodeOutsideTheCodeGrammarIsCarriedAsItsCategory() {
// given
RecordingPublisher publisher =
new RecordingPublisher(
PublishResult.ambiguous(
Duration.ZERO,
FailureDescriptor.of(
FailureCategory.AMBIGUOUS, "timed out after 5000 ms", "no answer")));
PlatformIntegrationEventPublishAdapter bridge =
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
// when
OutboxPublishAttempt attempt =
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join();
// then
assertThat(attempt.failureCode()).contains("AMBIGUOUS");
}
@Test
@DisplayName("platform으로 옮길 수 없는 event는 그 이유를 원인 코드로 알린다")
void anEventThePlatformCannotRepresentIsRefusedWithItsOwnFailureCode() {
// given
RecordingPublisher publisher = new RecordingPublisher(confirmed());
PlatformIntegrationEventPublishAdapter bridge =
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
// when
OutboxPublishAttempt attempt =
bridge.publish(event("event-1", Optional.empty())).toCompletableFuture().join();
// then
assertThat(attempt)
.isEqualTo(
OutboxPublishAttempt.failed(
OutboxPublishOutcome.REJECTED_BEFORE_SEND,
PlatformIntegrationEventPublishAdapter.NOT_REPRESENTABLE));
}
/**
* The relay sizes its batch against this (review 2026-11-01 06-12). The platform's deadline is
* measured from the call, but a transport may block in its send before the deadline is applied —
* Kafka's {@code max.block.ms} is set to the same default — so an attempt can take twice it.
*/
@Test
@DisplayName("발행 하나의 최악 시간은 platform 기본 발행 기한의 두 배다")
void theLongestAttemptIsTwiceThePlatformsDefaultPublishDeadline() {
// given
PlatformIntegrationEventPublishAdapter bridge =
new PlatformIntegrationEventPublishAdapter(
new RecordingPublisher(confirmed()), "portfolio-service");
// when
Optional<Duration> longest = bridge.longestAttempt();
// then
assertThat(longest).contains(PublishOptions.defaults().timeout().multipliedBy(2));
}
@Test
@DisplayName("다시 시도해도 같은 결과인 거부는 지금처럼 확정 거부로 번역한다")
void aRefusalThePlatformMarksFinalStaysADefiniteRejection() {
@@ -153,7 +243,7 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
// when
OutboxPublishOutcome outcome =
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join();
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join().outcome();
// then
assertThat(outcome).isEqualTo(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
@@ -165,7 +255,12 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
PlatformIntegrationEventPublishAdapter bridge =
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
assertThat(bridge.publish(event("event-1", Optional.empty())).toCompletableFuture().join())
assertThat(
bridge
.publish(event("event-1", Optional.empty()))
.toCompletableFuture()
.join()
.outcome())
.isEqualTo(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
assertThat(publisher.calls).isZero();
}
@@ -192,7 +287,7 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
// when
OutboxPublishOutcome outcome =
bridge.publish(event(eventId, Optional.empty())).toCompletableFuture().join();
bridge.publish(event(eventId, Optional.empty())).toCompletableFuture().join().outcome();
// then
assertThat(outcome).isEqualTo(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
@@ -64,17 +64,39 @@ class NotificationCallbackProviderConfig {
return cachingCertificates(uri -> fetchCertificate(client, uri));
}
private static byte[] fetchCertificate(HttpClient client, URI uri) {
/**
* Fetches the certificate document at a host-checked SNS URL.
*
* <p>The status decides what a failed fetch says. The URL already passed the SNS host check, so
* the host answering is SNS. A 4xx is SNS saying it has no such certificate: a real SNS message
* names one that exists, so the envelope is forged — {@link IllegalArgumentException}, which the
* verifier turns into an invalid signature that is refused and audited. It used to be "cannot
* verify now" like every other non-2xx, so a forged envelope pointing at a certificate SNS never
* issued got a 503, was never audited, and was logged as a provider hiccup (review 2026-11-03
* RR-07-03). A 5xx, a 408 or 429 (slow or throttling host), or a network failure says nothing
* about the certificate and stays "cannot verify now".
*
* @param client the no-redirect client
* @param uri the host-checked certificate URL
* @return the certificate document, at most {@link #MAX_SNS_CERTIFICATE_BYTES}
* @throws IllegalArgumentException when SNS answers that the certificate does not exist
* @throws CallbackVerificationUnavailableException when the certificate could not be fetched now
*/
static byte[] fetchCertificate(HttpClient client, URI uri) {
HttpRequest request =
HttpRequest.newBuilder(uri).timeout(SNS_CERTIFICATE_TIMEOUT).GET().build();
try {
HttpResponse<InputStream> response =
client.send(request, HttpResponse.BodyHandlers.ofInputStream());
try (InputStream body = response.body()) {
if (response.statusCode() / 100 != 2) {
int status = response.statusCode();
if (status / 100 == 4 && status != 408 && status != 429) {
throw new IllegalArgumentException(
"SNS has no signing certificate at the URL the message names (HTTP " + status + ")");
}
if (status / 100 != 2) {
throw certificateUnavailable(
new IOException(
"SNS signing certificate fetch failed with HTTP " + response.statusCode()));
new IOException("SNS signing certificate fetch failed with HTTP " + status));
}
return body.readNBytes(MAX_SNS_CERTIFICATE_BYTES);
}
@@ -16,6 +16,7 @@ import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import tools.jackson.databind.JsonNode;
@@ -36,8 +37,12 @@ import tools.jackson.databind.JsonNode;
* signed string), for up to {@link #SNS_HTTP_RETRY_HORIZON}. A window of five minutes threw away
* every bounce and complaint that arrived after an outage longer than that, and with them the
* suppressions they should have created (review 05-20). Accepting a replay inside the window costs
* nothing: the provider event ledger is unique on {@code (provider_profile_id, provider_event_id)},
* so a second delivery of the same event is a duplicate, not a second event.
* nothing: the provider event ledger is unique on {@code (provider_profile_id, provider_event_id)}
* whenever an event id is present, and this adapter gives every SES event the signed SNS {@code
* MessageId} as that id, so a second delivery of the same message is a duplicate, not a second
* event. It used to give none, and the only identity left was the {@code (provider_profile_id,
* event_fingerprint)} index — a fingerprint keyed by the active HMAC key, so a redelivery after a
* key rotation inside the window was recorded twice (review 2026-11-03 RR-07-07).
*/
public final class SesCallbackAdapter implements ProviderCallbackAdapter {
@@ -118,7 +123,10 @@ public final class SesCallbackAdapter implements ProviderCallbackAdapter {
return List.of();
}
String message = envelope.getOrDefault("Message", "{}");
return normalizer.normalize(message);
// The SNS MessageId is in the signed string and survives every redelivery, so it — not a
// fingerprint under the current HMAC key — is what the ledger recognises a replay by.
return normalizer.normalize(
message, Optional.ofNullable(envelope.get("MessageId")).filter(id -> !id.isBlank()));
}
private boolean withinReplayWindow(String timestamp, Instant receivedAt) {
@@ -7,6 +7,7 @@ import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import tools.jackson.databind.JsonNode;
@@ -19,8 +20,16 @@ import tools.jackson.databind.JsonNode;
*/
public final class SesEventNormalizer {
/** Normalize one SES notification payload. */
public List<NormalizedProviderEvent> normalize(String payload) {
/**
* Normalize one SES notification payload.
*
* @param payload the SES event JSON (the SNS envelope's {@code Message})
* @param providerEventId the identity of the delivery that carried it — the signed SNS {@code
* MessageId}, which SNS keeps across redeliveries — or empty when the envelope had none
* @return the provider events, each carrying {@code providerEventId}
*/
public List<NormalizedProviderEvent> normalize(String payload, Optional<String> providerEventId) {
Objects.requireNonNull(providerEventId, "providerEventId");
JsonNode root = NotificationJsonMapper.mapper().readTree(payload);
String type = text(root, "eventType").orElse(text(root, "notificationType").orElse("Unknown"));
Optional<String> messageId =
@@ -30,24 +39,26 @@ public final class SesEventNormalizer {
List<NormalizedProviderEvent> events = new ArrayList<>(1);
events.add(
switch (type) {
case "Send" ->
event(NormalizedEventType.PROVIDER_ACCEPTED, type, messageId, occurredAt, Map.of());
case "Send" -> event(NormalizedEventType.PROVIDER_ACCEPTED, type, messageId, occurredAt);
case "Delivery" ->
event(NormalizedEventType.DELIVERY_CONFIRMED, type, messageId, occurredAt, Map.of());
event(NormalizedEventType.DELIVERY_CONFIRMED, type, messageId, occurredAt);
case "DeliveryDelay" ->
event(NormalizedEventType.DELIVERY_DELAYED, type, messageId, occurredAt, Map.of());
event(NormalizedEventType.DELIVERY_DELAYED, type, messageId, occurredAt);
case "Bounce" -> bounce(root, type, messageId, occurredAt);
case "Complaint" ->
event(NormalizedEventType.COMPLAINT, type, messageId, occurredAt, Map.of());
case "Complaint" -> event(NormalizedEventType.COMPLAINT, type, messageId, occurredAt);
case "Reject" ->
event(NormalizedEventType.PROVIDER_REJECTED, type, messageId, occurredAt, Map.of());
event(NormalizedEventType.PROVIDER_REJECTED, type, messageId, occurredAt);
case "RenderingFailure" ->
event(NormalizedEventType.TEMPLATE_FAILURE, type, messageId, occurredAt, Map.of());
case "Open" -> event(NormalizedEventType.OPENED, type, messageId, occurredAt, Map.of());
case "Click" -> event(NormalizedEventType.CLICKED, type, messageId, occurredAt, Map.of());
default -> event(NormalizedEventType.UNKNOWN, type, messageId, occurredAt, Map.of());
event(NormalizedEventType.TEMPLATE_FAILURE, type, messageId, occurredAt);
case "Open" -> event(NormalizedEventType.OPENED, type, messageId, occurredAt);
case "Click" -> event(NormalizedEventType.CLICKED, type, messageId, occurredAt);
default -> event(NormalizedEventType.UNKNOWN, type, messageId, occurredAt);
});
return List.copyOf(events);
// One SNS delivery carries one SES event, so the delivery's id is the event's. It is what makes
// a redelivery a duplicate in the ledger: the other identity, the event fingerprint, is keyed
// by
// the active HMAC key and changes when the key rotates (review 2026-11-03 RR-07-07).
return events.stream().map(event -> withEventId(event, providerEventId)).toList();
}
private static NormalizedProviderEvent bounce(
@@ -68,6 +79,14 @@ public final class SesEventNormalizer {
Map.of("bounceType", bounceType));
}
private static NormalizedProviderEvent event(
NormalizedEventType type,
String nativeType,
Optional<String> messageId,
Optional<Instant> occurredAt) {
return event(type, nativeType, messageId, occurredAt, Map.of());
}
private static NormalizedProviderEvent event(
NormalizedEventType type,
String nativeType,
@@ -78,6 +97,17 @@ public final class SesEventNormalizer {
type, nativeType, Optional.empty(), messageId, occurredAt, attributes);
}
private static NormalizedProviderEvent withEventId(
NormalizedProviderEvent event, Optional<String> providerEventId) {
return new NormalizedProviderEvent(
event.type(),
event.providerNativeType(),
providerEventId,
event.providerRequestId(),
event.providerOccurredAt(),
event.attributes());
}
private static Optional<String> text(JsonNode node, String field) {
JsonNode value = node.get(field);
return value == null || value.isNull() ? Optional.empty() : Optional.of(value.asString());
@@ -112,7 +112,12 @@ public final class JavaMailSenderSmtpDispatch implements SmtpDispatch {
} else if (cause instanceof SMTPSendFailedException send) {
code = send.getReturnCode();
}
return code >= 200 && code < 600 ? Optional.of(code) : Optional.empty();
// Only a refusal is a reply code here: 4xx (transient) and 5xx (permanent). Angus's sendpartial
// path throws SMTPSendFailedException carrying the last reply, which can be a success (250);
// read as a reply code it reached the classifier as a permanent refusal of a message the relay
// had partly accepted. Anything else takes the conservative "may have been delivered" path
// below (review 2026-11-03 RR-04-11).
return code >= 400 && code < 600 ? Optional.of(code) : Optional.empty();
}
/**
@@ -0,0 +1,104 @@
package dev.caskeleton.adapter.outbound.notification.platform.autoconfigure;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.catchThrowable;
import com.sun.net.httpserver.HttpServer;
import dev.caskeleton.adapter.outbound.notification.platform.callback.CallbackVerificationUnavailableException;
import java.io.IOException;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.ServerSocket;
import java.net.URI;
import java.net.http.HttpClient;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
/**
* What a failed SNS signing-certificate fetch says about the callback (review 2026-11-03 RR-07-03).
*
* <p>The URL has passed the SNS host check before it is fetched, so the host is SNS's. A real SNS
* message names a certificate that exists; when SNS itself answers 4xx for it, the envelope named a
* certificate SNS never issued — a forged callback, to be rejected and audited. Only a failure that
* says nothing about the certificate — a 5xx, a 408 or 429 (the host is slow or throttling), or a
* network failure — is "cannot verify now" (503, the provider retries).
*/
class SnsCertificateFetchTest {
private final HttpClient client = HttpClient.newHttpClient();
@ParameterizedTest
@ValueSource(ints = {403, 404, 410})
@DisplayName("인증서 URL이 4xx로 답하면 없는 인증서를 가리킨 위조 콜백이므로 검증 실패로 거절된다")
void aCertificateTheHostDoesNotHaveFailsVerification(int status) throws IOException {
// given
HttpServer server = answering(status);
try {
URI url = URI.create("http://127.0.0.1:" + server.getAddress().getPort() + "/cert.pem");
// when
Throwable failure =
catchThrowable(() -> NotificationCallbackProviderConfig.fetchCertificate(client, url));
// then
assertThat(failure)
.isInstanceOf(IllegalArgumentException.class)
.isNotInstanceOf(CallbackVerificationUnavailableException.class);
} finally {
server.stop(0);
}
}
@ParameterizedTest
@ValueSource(ints = {408, 429, 500, 503})
@DisplayName("인증서 URL이 5xx나 시간 초과·요청 제한(408·429)으로 답하면 아무것도 모르므로 지금 검증할 수 없다고 답한다")
void aFailingCertificateHostLeavesTheCallbackUnverified(int status) throws IOException {
// given
HttpServer server = answering(status);
try {
URI url = URI.create("http://127.0.0.1:" + server.getAddress().getPort() + "/cert.pem");
// when
Throwable failure =
catchThrowable(() -> NotificationCallbackProviderConfig.fetchCertificate(client, url));
// then
assertThat(failure).isInstanceOf(CallbackVerificationUnavailableException.class);
} finally {
server.stop(0);
}
}
@Test
@DisplayName("인증서 호스트에 닿지 못하면 지금 검증할 수 없다고 답한다")
void anUnreachableCertificateHostLeavesTheCallbackUnverified() throws IOException {
// given
int closedPort;
try (ServerSocket probe = new ServerSocket(0)) {
closedPort = probe.getLocalPort();
}
URI url = URI.create("http://127.0.0.1:" + closedPort + "/cert.pem");
// when
Throwable failure =
catchThrowable(() -> NotificationCallbackProviderConfig.fetchCertificate(client, url));
// then
assertThat(failure).isInstanceOf(CallbackVerificationUnavailableException.class);
}
private static HttpServer answering(int status) throws IOException {
HttpServer server =
HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress(), 0), 0);
server.createContext(
"/",
exchange -> {
exchange.sendResponseHeaders(status, -1);
exchange.close();
});
server.start();
return server;
}
}
@@ -8,6 +8,7 @@ import dev.caskeleton.adapter.outbound.notification.platform.callback.CallbackVe
import dev.caskeleton.adapter.outbound.notification.platform.callback.CallbackVerificationUnavailableException;
import dev.caskeleton.adapter.outbound.notification.platform.callback.ProviderCallbackVerificationRequest;
import dev.caskeleton.adapter.outbound.notification.platform.template.NotificationJsonMapper;
import dev.caskeleton.application.notification.platform.api.NormalizedProviderEvent;
import dev.caskeleton.application.notification.platform.api.ProviderId;
import dev.caskeleton.application.notification.platform.api.ProviderProfileId;
import java.nio.charset.StandardCharsets;
@@ -19,6 +20,7 @@ import java.time.Duration;
import java.time.Instant;
import java.util.Base64;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
@@ -135,6 +137,28 @@ class SesCallbackAdapterTest {
assertThat(unavailable.reasonCode()).isEqualTo("SNS_CERTIFICATE_UNAVAILABLE"));
}
@Test
@DisplayName(
"SES 이벤트의 provider event id는 서명된 SNS MessageId라 fingerprint 키를 회전해도 같은 콜백이 같은 이벤트로 남는다")
void theSignedSnsMessageIdIsTheProviderEventId() {
// given
CallbackVerificationResult verified =
adapter.verify(request(signed(ALLOWED_TOPIC, RECEIVED_AT)));
// when
List<NormalizedProviderEvent> events =
adapter.normalize(verified.verifiedCallback().orElseThrow());
// then
// The ledger is unique on (profile, provider_event_id) whenever an id is present. Without one
// the only identity was a fingerprint keyed by the active HMAC key, so a redelivery after a
// key rotation inside the one-hour replay window became a second row (RR-07-07).
assertThat(events)
.singleElement()
.extracting(NormalizedProviderEvent::providerEventId)
.isEqualTo(Optional.of("sns-message-1"));
}
/** The adapter with the window a deployment gets when it sets nothing. */
private static SesCallbackAdapter withDefaultWindow() {
return new SesCallbackAdapter(
@@ -16,9 +16,11 @@ import java.nio.charset.StandardCharsets;
import java.util.List;
import java.util.Optional;
import java.util.Properties;
import org.eclipse.angus.mail.smtp.SMTPSendFailedException;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.springframework.mail.MailAuthenticationException;
import org.springframework.mail.MailSendException;
import org.springframework.mail.javamail.JavaMailSenderImpl;
/**
@@ -136,6 +138,38 @@ class JavaMailSenderSmtpDispatchTest {
.containsExactly(Optional.empty(), false);
}
@Test
@DisplayName("성공 코드(250)를 실은 부분 전송 실패는 거절 reply code가 아니라 전달됐을 수 있는 실패로 보고된다")
void aPartialSendCarryingASuccessCodeIsNotReportedAsARefusal() {
// given
// Angus's sendpartial path reports a partly delivered message as SMTPSendFailedException
// carrying the last reply, which is a success (250), not a refusal.
JavaMailSenderImpl partlyDelivered =
new JavaMailSenderImpl() {
@Override
public void send(MimeMessage... messages) {
throw new MailSendException(
"partly sent",
new SMTPSendFailedException(
"DATA",
250,
"250 2.0.0 queued",
null,
new jakarta.mail.Address[0],
new jakarta.mail.Address[0],
new jakarta.mail.Address[0]));
}
};
JavaMailSenderSmtpDispatch dispatch = new JavaMailSenderSmtpDispatch(partlyDelivered);
// when
SmtpDispatchException failure = sendExpectingFailure(dispatch);
// then
assertThat(List.of(failure.replyCode(), failure.dataCommitted()))
.containsExactly(Optional.empty(), true);
}
private static SmtpDispatchException sendExpectingFailure(JavaMailSenderSmtpDispatch dispatch) {
return catchThrowableOfType(SmtpDispatchException.class, () -> dispatch.send(message()));
}
@@ -155,9 +155,9 @@ strictTestLanes {
}
}
// The MinIO lanes start a container, so they are the main-branch stage of this leaf, not its
// pull-request `check` (review BUILD-계획-B3); object-storage-qualification.yml runs them on pull
// requests that touch object storage, and integrationCheck runs them with every integration lane.
tasks.named('mainCheck') {
dependsOn 'objectStorageMinioContractTest', 'objectStorageMinioFaultTest'
}
// The MinIO lanes start a container, so they are not this leaf's pull-request `check` (review
// BUILD-계획-B3). They are integration lanes, and the root integrationCheck owns every container lane
// on main and nightly; object-storage-qualification.yml runs them on pull requests that touch object
// storage. This leaf's `mainCheck` used to run them too, so each main push ran them twice in two
// parallel jobs (review 2026-11-03 RR-06-09, original 08-04 (d)).
@@ -46,12 +46,18 @@ in `platform`), so the two capability leaves read the same way (review 04-16).
write; Spring Data repositories are mocks).
- `FileserverPersistenceActivationTest` — the scan covers this package only, behind
`app.fileserver-platform.enabled`, and runs before the JPA root auto-configuration.
- `repository.CleanupClaimSqlTest` — the claim keeps its one-statement shape (the Claims rule).
- `repository.CleanupClaimSqlTest` — the claim's text keeps its one-statement shape (the Claims
rule); the locking itself is checked on PostgreSQL by
`PostgreSqlFileserverReclamationIntegrationTest`.
- `entity.AssignedIdEntityTest` — newness from `Persistable`, on Hibernate over H2.
A new store write adds its case to the guard test; a new auto-configuration adds its line to the
activation test.
- Port contracts live in `application-fileserver` testFixtures (`*PortContract`). The in-memory
fakes and the PostgreSQL tests both extend them.
- Port contracts live in `application-fileserver` testFixtures (`*PortContract`, and
`FileQuotaLedgerContract` for the three quota ports, which share one table). The in-memory fakes
and the PostgreSQL tests both extend them; every port this leaf implements has one
(`docs/testing/TESTING_STRATEGY.md` 9.3). The PostgreSQL side of the seven added in review
2026-11-03 RR-01-04 is tagged `jpa-contract` and runs in persistence-jpa's
`jpaPlatformContractTest` lane on `FileserverPostgreSqlStack`, whose clock a case moves.
- The real-PostgreSQL tests are still in `persistence-jpa`'s `postgresqlIntegrationTest` source
set, which owns the container and Flyway wiring, and the JPA readiness card
(`config/jpa/readiness-cards.yaml`, checked by `JpaReadinessRegistryTest`) names them as that
@@ -18,8 +18,11 @@ import java.util.UUID;
/**
* JPA-backed {@link FileQuotaPort}.
*
* <p>Reservation, extension, commit, and release are conditional statements, so a reservation that
* already expired or was released can never be extended or committed.
* <p>Reservation, extension, commit, and release are conditional statements: a reservation that
* expired or was released can never be extended, and a released or already committed one can never
* be committed. An expired reservation that was not released still commits — bytes that were
* written are durable usage whether or not the reservation outlived its TTL. {@code
* FileQuotaLedgerContract} holds this adapter and the in-memory ledger to the same cases.
*/
public class JpaFileQuota implements FileQuotaPort {
@@ -15,8 +15,8 @@ package dev.caskeleton.adapter.outbound.persistence.fileserver.repository;
* every claimed item is fenced on its own.
*
* <p>Package-private: the repository is its one reader, and the leaf's public surface is a ceiling.
* {@link #positional(String)} rewrites the named parameters to JDBC placeholders for a test in this
* package that runs the identical text on a bare {@code Connection}.
* The PostgreSQL lane runs it through the repository ({@code
* PostgreSqlFileserverReclamationIntegrationTest}); nothing runs the text on its own.
*/
final class CleanupClaimSql {
@@ -43,14 +43,4 @@ final class CleanupClaimSql {
+ " c.file_id, c.upload_id, c.content_key";
private CleanupClaimSql() {}
/**
* Rewrites the named parameters to JDBC placeholders, left to right.
*
* @param namedSql the statement above
* @return the same statement with {@code ?} in place of every {@code :name}
*/
static String positional(String namedSql) {
return namedSql.replaceAll("(?<!:):[A-Za-z][A-Za-z0-9]*", "?");
}
}
@@ -7,16 +7,17 @@ import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
/**
* The claim statement keeps the shape the leaf's claim rule names: one statement that locks the
* rows it picks, skips rows another claim holds, gives every row a fresh token and a higher fence,
* and returns them in the same round trip. The PostgreSQL lane runs it; this pins the shape without
* a database, so a rewrite into a select-then-update fails here first.
* The claim statement's text keeps the shape the leaf's claim rule names: one statement that picks
* rows with {@code FOR UPDATE SKIP LOCKED}, gives every row a fresh token and a higher fence, and
* returns them. This pins the text without a database, so a rewrite into a select-then-update fails
* here first. Whether the statement actually locks, skips and fences is a behaviour of PostgreSQL,
* and {@code PostgreSqlFileserverReclamationIntegrationTest} runs it there.
*/
class CleanupClaimSqlTest {
@Test
@DisplayName("claim은 고른 행을 잠그고, 다른 claim이 잡은 행은 기다리지 않고 건너뛴다")
void theClaimLocksWhatItPicksAndSkipsWhatAnotherClaimHolds() {
@DisplayName("claim 문장은 한 문장의 CTE 안에서 FOR UPDATE SKIP LOCKED로 행을 고른다")
void theClaimTextPicksRowsWithForUpdateSkipLockedInOneStatement() {
// when
String claim = CleanupClaimSql.CLAIM_DUE;
@@ -25,8 +26,8 @@ class CleanupClaimSqlTest {
}
@Test
@DisplayName("claim한 행마다 새 토큰과 하나 높은 fence를 받고 같은 왕복에 돌아온다")
void everyClaimedRowGetsAFreshTokenAndAHigherFenceInTheSameRoundTrip() {
@DisplayName("claim 문장은 행마다 새 토큰을 쓰고 fence를 하나 올리고 RETURNING으로 돌려준다")
void theClaimTextWritesAFreshTokenRaisesTheFenceAndReturnsTheRows() {
// when
String claim = CleanupClaimSql.CLAIM_DUE;
@@ -38,14 +39,4 @@ class CleanupClaimSqlTest {
claim.contains("RETURNING c.cleanup_id, c.claim_token")))
.containsOnly(true);
}
@Test
@DisplayName("이름 있는 매개변수를 JDBC 자리표시자로 바꿀 때 PostgreSQL 형 변환은 건드리지 않는다")
void positionalParametersDoNotRewritePostgreSqlCasts() {
// when
String positional = CleanupClaimSql.positional("payload::jsonb @> :expected::jsonb");
// then
assertThat(positional).isEqualTo("payload::jsonb @> ?::jsonb");
}
}
@@ -235,7 +235,11 @@ made Spring call `getTransactionIsolation()` at every begin, which pgjdbc answer
Every mode is time-bounded (review 04-10). Spring hands the remaining time to each statement, so a
slow query or lock wait is cancelled by the server (`57014` → `DB_QUERY_CANCELED`), and a deadline
that passed before the next statement (`TransactionTimedOutException`) gets the same code.
that passed before the next statement gets the same code: Spring's `TransactionTimedOutException`
on the JDBC path, and on the JPA path (`JpaTransactionManager`, production) Hibernate's own
`TransactionException`, which has no SQLState and is classified because `SpringTransaction` knows
its deadline passed (`translateAfterDeadline`, review RR-03-03). A business exception thrown after
the deadline is passed on unchanged.
Pre-built templates are immutable after construction so concurrent callers cannot
observe each other's reconfiguration. `inRootWrite` reuses the pre-built write template,
@@ -302,7 +306,9 @@ the two paths disagree on any state.
**No SQLState, classified by type:** a lost `@Version` race (`OptimisticLockingFailureException`,
JPA `OptimisticLockException`) → `DB_OPTIMISTIC_CONFLICT` (409, review 04-05); an expired Spring
transaction deadline (`TransactionTimedOutException`) → `DB_QUERY_CANCELED`.
transaction deadline (`TransactionTimedOutException`) → `DB_QUERY_CANCELED`; any other persistence
failure (JPA, Spring data-access or Spring transaction exception) of a transaction
`SpringTransaction` began, once its deadline has passed → `DB_QUERY_CANCELED`.
**Commit phase (`translateCommitFailure`, ADR-JPA-003):** once the action has returned and
`SpringTransaction` is committing a write transaction it began, a failure with `08*`, `40003`, a
+15 -3
View File
@@ -296,14 +296,26 @@ A가 FAILED로 바꾸면 다시 발행되고(중복 전달), A의 늦은 PUBLISH
`WHERE event_id = ? AND status = 'IN_FLIGHT' AND lease_owner = ? AND lease_token = ?`.
owner는 비우고 token은 그대로 둔다(token은 증가만 한다). 1 row면 `APPLIED`, 0 row면 `STALE`.
`findById` + setter 경로와 `@Version`은 쓰지 않는다.
- **실패 원인 코드(`V22`)**: `markFailed`·`markDeferred`·`markDead`는 같은 조건부 UPDATE에서
`last_failure_code`를 쓴다(publisher가 붙인 코드, 없으면 결과·예외 타입 이름). entity 필드는 읽기
전용이라 append INSERT와 claim flush는 이 열을 건드리지 않는다. `longRetries(n, now)`는 미룬 claim이
n번 이상인 FAILED 행 수와 가장 오래된 행의 나이를 센다(`outbox.retry.long.*` 게이지). 상태 `PARKED`는
DDL이 없다 — claim이 PENDING·FAILED·IN_FLIGHT만 고르고, FIFO 게이트는 PUBLISHED 아닌 모든 행을 막는 앞
행으로 본다. claim이 시도 횟수가 어긋난 행(`deferred_attempt_count > attempt_count`)을 만나면 DEAD로
옮기고 WARN `error.reason=ATTEMPT_COUNTERS_INCONSISTENT`로 원인을 말한다(재리뷰 2026-11-03 RR-05-01·03).
- **미룬 실패와 claim 해제(`V20`)**: `markDeferred(lease, next)`는 `markFailed`와 같은 전이에
`deferred_attempt_count + 1`을 더한다(일시적 실패는 시도 예산 밖). `releaseLease(lease, at)`는
손대지 않은 claim을 `PENDING`, `next_attempt_at = at`, `attempt_count - 1`로 되돌린다(종료 중인
relay가 쓴다). 둘 다 같은 lease 조건부 UPDATE다.
- **aggregate 순서 번호(`V21`)**: `JpaOutboxStore`는 `OutboxAggregateSequencePort`도 구현한다.
`outbox_aggregate_sequence` counter를 `UPDATE ... + 1`(row lock)로 올리고, 없으면 그 aggregate의 기존
outbox 최대 `aggregate_sequence` + 1로 만든다. V21은 `outbox_event`의 순서 키 유일 인덱스도 만든다 —
기존 중복 row가 있으면 migration이 실패하므로 먼저 정리해야 한다.
`outbox_aggregate_sequence` counter를 `UPDATE`(row lock)로 올린다. 올린 값은 그 aggregate의 가장 큰
outbox 번호보다 작아지지 않는다(rolling 배포 중 옛 인스턴스가 쓴 벽시계 번호가 먼저 온다). counter가
없으면 `INSERT ... ON CONFLICT DO NOTHING`으로 만든다. 시작값은 남은 outbox 최대 번호와 V21이
기록한 floor(`outbox_aggregate_sequence_floor`, V21 적용 시각의 epoch 마이크로초) 중 큰 값 + 1이다.
첫 쓰기 두 개가 겹치면 두 번째는 첫 번째 커밋을 기다렸다가 아무것도 넣지 않고 증가만 한다(리뷰
RR-03-01). V21은 남은 row가 있는 aggregate의 counter를 그 최대 번호로 미리 만들고, `outbox_event`의
순서 키 유일 인덱스도 만든다 — 기존 중복 row가 있으면 migration이 실패하므로 먼저 정리해야 한다
(`docs/runbooks/outbox-dead-letter.md` §8).
- **STALE은 예외가 아니다**: row의 결과는 지금 lease를 가진 relay가 쓴다. use case는 결과를
`OutboxRelayResult.Outcome.STALE`로 남기고 `OutboxRelayFailureReportPort.reportStaleCompletion`
으로 알린 뒤 다음 이벤트로 넘어간다. FAILED/DEAD report는 보내지 않는다(아무것도 저장되지
@@ -187,6 +187,7 @@ def readinessLanes = [
postgresqlOutboxRowLifecycleIntegrationTest : ['PostgreSqlOutboxRowLifecycleIntegrationTest'],
postgresqlOutboxAggregateSequenceIntegrationTest : ['PostgreSqlOutboxAggregateSequenceIntegrationTest'],
postgresqlOutboxOrderKeyRunbookIntegrationTest : ['PostgreSqlOutboxOrderKeyRunbookIntegrationTest'],
postgresqlOutboxDispositionRunbookIntegrationTest : ['PostgreSqlOutboxDispositionRunbookIntegrationTest'],
postgresqlFileserverMigrationIntegrationTest : ['PostgreSqlFileserverMigrationIntegrationTest'],
postgresqlFileserverMetadataIntegrationTest : ['PostgreSqlFileserverMetadataStoreIntegrationTest'],
postgresqlFileserverReclamationIntegrationTest : ['PostgreSqlFileserverReclamationIntegrationTest'],
@@ -243,13 +244,20 @@ strictTestLanes {
}
}
// The V21 pre-deploy check and cleanup are SQL an operator runs from the outbox runbook (review
// 2026-11-01, README 15 decision). The lane reads them out of the runbook, so the document is one of
// its inputs: editing either block reruns the lane against real PostgreSQL.
tasks.named('postgresqlOutboxOrderKeyRunbookIntegrationTest', Test) {
def outboxRunbook = new File(rootProject.projectDir.parentFile, 'docs/runbooks/outbox-dead-letter.md')
inputs.file(outboxRunbook).withPathSensitivity(PathSensitivity.RELATIVE).withPropertyName('outboxRunbook')
systemProperty 'outbox.runbook', outboxRunbook.absolutePath
// The outbox runbooks give an operator SQL to run: the V21 pre-deploy check and cleanup (review
// 2026-11-01, README 15 decision), the manual dispositions of a DEAD row (re-review 2026-11-03
// RR-05-01), and finding, parking, resuming and skipping a row that is retried without end
// (RR-05-03). Each lane reads its blocks out of the runbooks, so the documents are its inputs:
// editing a block reruns the lane against real PostgreSQL.
['postgresqlOutboxOrderKeyRunbookIntegrationTest', 'postgresqlOutboxDispositionRunbookIntegrationTest'].each { laneName ->
tasks.named(laneName, Test) {
def outboxRunbook = new File(rootProject.projectDir.parentFile, 'docs/runbooks/outbox-dead-letter.md')
inputs.file(outboxRunbook).withPathSensitivity(PathSensitivity.RELATIVE).withPropertyName('outboxRunbook')
systemProperty 'outbox.runbook', outboxRunbook.absolutePath
def publishFailedRunbook = new File(rootProject.projectDir.parentFile, 'docs/runbooks/outbox-publish-failed.md')
inputs.file(publishFailedRunbook).withPathSensitivity(PathSensitivity.RELATIVE).withPropertyName('publishFailedRunbook')
systemProperty 'outbox.publish-failed.runbook', publishFailedRunbook.absolutePath
}
}
tasks.named('postgresqlSecurityBaselineIntegrationTest') {
@@ -46,8 +46,15 @@ public enum FailureCategory {
QUERY_TIMEOUT,
/**
* {@code 25P03} — the server ended a session that sat idle inside an open transaction. Nothing
* was committed, which is why its code is retryable where {@link #QUERY_TIMEOUT}'s is not.
* Not a transaction timeout: {@code 25P03}, the server ended a session that sat idle inside an
* open transaction ({@code idle_in_transaction_session_timeout}). Nothing was committed, which is
* why its code ({@code DB_IDLE_IN_TX_TIMEOUT}) is retryable.
*
* <p>A transaction that ran past its own deadline ({@code SpringTransaction.Timeouts}) is {@link
* #QUERY_TIMEOUT} ({@code DB_QUERY_CANCELED}), not this. The name says otherwise and is kept only
* because the enum is public API ({@code docs/architecture/jpa-api-surface.txt}); it becomes
* {@code IDLE_IN_TRANSACTION_TIMEOUT}, with {@link TransactionTimeoutException}, at the next
* release that breaks this API anyway (review RR-03-10).
*/
TRANSACTION_TIMEOUT,
@@ -1,10 +1,13 @@
package dev.caskeleton.adapter.outbound.persistence.api.error;
/**
* A transaction exceeded its configured transaction timeout (design §15.4).
* The server ended a session that sat idle inside an open transaction ({@code 25P03}, {@link
* FailureCategory#TRANSACTION_TIMEOUT}). Not a transaction that ran past its own deadline: that is
* a cancelled statement ({@link FailureCategory#QUERY_TIMEOUT}).
*
* <p>The transaction is rolled back; whether the use case may be re-run is a business decision, so
* the platform does not retry it automatically.
* <p>The transaction is rolled back and nothing was committed, so its code is retryable; whether
* the use case is re-run is still the caller's decision, and the platform does not retry it
* automatically. The name is kept until the next API-breaking release (review RR-03-10).
*/
public final class TransactionTimeoutException extends JpaPersistenceException {
@@ -8,6 +8,7 @@ import dev.caskeleton.shared.error.ApiErrorCode;
import dev.caskeleton.shared.error.OperationalError;
import dev.caskeleton.shared.error.PersistenceFailureException;
import jakarta.persistence.OptimisticLockException;
import jakarta.persistence.PersistenceException;
import java.io.IOException;
import java.sql.SQLException;
import java.sql.SQLNonTransientConnectionException;
@@ -23,6 +24,7 @@ import java.util.Optional;
import java.util.Set;
import org.springframework.dao.DataAccessException;
import org.springframework.dao.OptimisticLockingFailureException;
import org.springframework.transaction.TransactionException;
import org.springframework.transaction.TransactionTimedOutException;
/**
@@ -140,6 +142,42 @@ public final class PersistenceExceptionTranslator {
return Optional.empty();
}
/**
* Classify a failure raised once the transaction's deadline had passed, when neither a SQLState
* nor a type above identifies it.
*
* <p>Past the deadline Hibernate refuses the next statement itself, before the driver is asked:
* an {@code org.hibernate.TransactionException} with no SQLState that is not Spring's {@link
* TransactionTimedOutException} — in the action as it is, at commit wrapped in a {@code
* JpaSystemException} (review RR-03-03). That is the production path ({@code
* JpaTransactionManager}); the JDBC path gets {@code 57014} from the server instead. Both are
* {@code DB_QUERY_CANCELED}. The caller decides that the deadline has passed; this method only
* makes sure the failure came from the persistence stack — a JPA, Spring data-access or Spring
* transaction exception anywhere in the cause chain — so a business exception thrown late is
* passed on as it is, and one already classified is not classified again.
*/
public Optional<PersistenceFailureException> translateAfterDeadline(Throwable ex) {
Objects.requireNonNull(ex, "ex");
if (ex instanceof PersistenceFailureException) {
return Optional.empty();
}
for (Throwable t = ex; t != null; t = t.getCause()) {
if (t instanceof PersistenceException
|| t instanceof DataAccessException
|| t instanceof TransactionException) {
return Optional.of(
new PersistenceFailureException(
OperationalError.DB_QUERY_CANCELED,
"persistence failure after the transaction deadline passed",
ex));
}
if (t.getCause() == t) {
break;
}
}
return Optional.empty();
}
/**
* Classify a failure raised while committing a write transaction this call began.
*
@@ -14,6 +14,7 @@ import dev.caskeleton.application.outbox.OutboxAppendPort;
import dev.caskeleton.application.outbox.OutboxCompletion;
import dev.caskeleton.application.outbox.OutboxEventStatus;
import dev.caskeleton.application.outbox.OutboxLease;
import dev.caskeleton.application.outbox.OutboxLongRetries;
import dev.caskeleton.application.outbox.OutboxStorePort;
import dev.caskeleton.shared.digest.Sha256;
import java.nio.charset.StandardCharsets;
@@ -29,6 +30,7 @@ import java.util.Objects;
import java.util.Optional;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.spi.LoggingEventBuilder;
/**
* JPA-backed {@link OutboxAppendPort} + {@link OutboxStorePort}. {@link #append} and the claim/mark
@@ -103,12 +105,21 @@ public class JpaOutboxStore
*
* <p>The increment is an {@code UPDATE}, which holds the counter row's lock until the caller
* commits: a second writer of the same aggregate waits, then increments the committed value, so
* the numbers follow commit order. An aggregate's first number is inserted instead, seeded after
* the highest {@code aggregate_sequence} its outbox rows already carry — rows numbered before the
* counter existed (the sample used epoch microseconds) must still come first. Two writers
* creating the same counter at once collide on its primary key and one transaction fails: a
* rollback the caller retries, never two events with one number. The counter outlives the
* reaper's deletes, so a number is never handed out twice.
* the numbers follow commit order. It never goes below the aggregate's highest outbox row, so a
* row an instance still on the wall-clock scheme writes during a rolling deploy comes first. The
* counter outlives the reaper's deletes, so a number is never handed out twice.
*
* <p>An aggregate's first number is inserted instead, after the larger of its highest remaining
* row and the floor V21 recorded — the moment it ran, above every clock number already sent for
* rows the reaper has deleted or that were written under an identity that has since changed
* (review RR-03-02). Two writers creating the same counter at once both succeed: the insert is
* {@code ON CONFLICT DO NOTHING}, so the second waits for the first to commit, writes nothing,
* and increments the committed counter instead (review RR-03-01).
*
* <p>Statements rather than one {@code INSERT … ON CONFLICT DO UPDATE … RETURNING} like {@code
* JpaLiveEventReplay}'s position allocator: the sample appends events on the H2 developer run
* ({@code bootRunH2}), and H2 accepts {@code ON CONFLICT DO NOTHING} but not {@code DO UPDATE}.
* The common case — the counter exists — is still one {@code UPDATE} and one read.
*/
@Override
public long nextSequence(AggregateIdentity aggregate) {
@@ -118,8 +129,11 @@ public class JpaOutboxStore
String tenant = aggregate.tenantScope();
String type = aggregate.aggregateType();
String id = aggregate.aggregateId();
if (repository.incrementAggregateSequence(tenant, type, id) == 0) {
repository.createAggregateSequence(tenant, type, id);
if (repository.incrementAggregateSequence(tenant, type, id) == 0
&& repository.createAggregateSequence(tenant, type, id) == 0) {
// Another writer created the counter after our UPDATE found none; its commit ended our wait
// in the insert, so the counter is there to increment now.
repository.incrementAggregateSequence(tenant, type, id);
}
return repository.aggregateSequence(tenant, type, id);
}
@@ -132,6 +146,11 @@ public class JpaOutboxStore
* instead rolled the whole claim back, and the next claim selected the same row first again, so
* one bad row stopped the relay for good. DEAD is fail-closed: the row is never published, and it
* waits for the DEAD-row runbook like any other.
*
* <p>The WARN names the cause. A row whose deferred claims outnumber its claims is not damaged:
* it is what a manual re-drive that reset {@code attempt_count} and left {@code
* deferred_attempt_count} produces (re-review 2026-11-03 RR-05-01), and "cannot read" sent the
* operator looking for corruption instead of re-running the corrected runbook SQL.
*/
@Override
public List<OutboxLease> claimBatch(
@@ -145,13 +164,26 @@ public class JpaOutboxStore
entity.setNextAttemptAt(now.plus(inFlightTimeout));
entity.setLeaseOwner(owner);
entity.setLeaseToken(entity.getLeaseToken() + 1);
if (entity.getDeferredAttemptCount() > entity.getAttemptCount()) {
deadLetterOnClaim(entity)
.addKeyValue("error.reason", "ATTEMPT_COUNTERS_INCONSISTENT")
.log(
"outbox claim moved a row to DEAD: deferred_attempt_count ({}) exceeds"
+ " attempt_count ({}); a manual re-drive that did not reset"
+ " deferred_attempt_count leaves this behind — re-drive it with the runbook"
+ " SQL",
entity.getDeferredAttemptCount(),
entity.getAttemptCount());
continue;
}
ClaimedOutboxEvent event;
try {
event = toClaimedOutboxEvent(entity);
} catch (RuntimeException unreadable) {
entity.setStatus(OutboxEventStatus.DEAD.name());
entity.setLeaseOwner(null);
log.warn("outbox claim moved a row it cannot read to DEAD", unreadable);
deadLetterOnClaim(entity)
.addKeyValue("error.reason", "UNREADABLE_ROW")
.setCause(unreadable)
.log("outbox claim moved a row it cannot read as an event to DEAD");
continue;
}
leases.add(new OutboxLease(event, owner, entity.getLeaseToken()));
@@ -159,6 +191,15 @@ public class JpaOutboxStore
return List.copyOf(leases);
}
/** Moves a claimed row it will not hand out to DEAD and starts the WARN that says why. */
private static LoggingEventBuilder deadLetterOnClaim(OutboxEventEntity entity) {
entity.setStatus(OutboxEventStatus.DEAD.name());
entity.setLeaseOwner(null);
return log.atWarn()
.addKeyValue("event_id", entity.getEventId())
.addKeyValue("runbook_link", "runbook://outbox/dead-letter");
}
/**
* One conditional UPDATE, no load-and-set: it applies only while the lease still holds the row
* (see README "outbox"). Zero rows is {@link OutboxCompletion#STALE}, not an error — the relay
@@ -174,18 +215,21 @@ public class JpaOutboxStore
/** Marks the leased row FAILED with its next attempt time; see {@link #markPublished}. */
@Override
public OutboxCompletion markFailed(OutboxLease lease, Instant nextAttemptAt) {
public OutboxCompletion markFailed(OutboxLease lease, Instant nextAttemptAt, String failureCode) {
requireTransaction("markFailed");
return completion(
repository.failLease(lease.eventId(), lease.owner(), lease.token(), nextAttemptAt));
repository.failLease(
lease.eventId(), lease.owner(), lease.token(), nextAttemptAt, failureCode));
}
/** Marks the leased row FAILED as a deferred claim; see {@link #markPublished}. */
@Override
public OutboxCompletion markDeferred(OutboxLease lease, Instant nextAttemptAt) {
public OutboxCompletion markDeferred(
OutboxLease lease, Instant nextAttemptAt, String failureCode) {
requireTransaction("markDeferred");
return completion(
repository.deferLease(lease.eventId(), lease.owner(), lease.token(), nextAttemptAt));
repository.deferLease(
lease.eventId(), lease.owner(), lease.token(), nextAttemptAt, failureCode));
}
/** Returns the leased row to PENDING without spending an attempt; see {@link #markPublished}. */
@@ -198,11 +242,10 @@ public class JpaOutboxStore
/** Marks the leased row DEAD; see {@link #markPublished}. */
@Override
public OutboxCompletion markDead(OutboxLease lease) {
public OutboxCompletion markDead(OutboxLease lease, String failureCode) {
requireTransaction("markDead");
return completion(
repository.completeLease(
lease.eventId(), lease.owner(), lease.token(), OutboxEventStatus.DEAD.name()));
repository.deadLease(lease.eventId(), lease.owner(), lease.token(), failureCode));
}
private static OutboxCompletion completion(int updatedRows) {
@@ -237,6 +280,21 @@ public class JpaOutboxStore
return result;
}
@Override
public OutboxLongRetries longRetries(int minDeferredClaims, Instant now) {
if (minDeferredClaims < 1) {
throw new IllegalArgumentException(
"minDeferredClaims must be >= 1, was " + minDeferredClaims);
}
Object[] row = repository.countLongRetries(minDeferredClaims).getFirst();
long count = (Long) row[0];
if (count == 0) {
return OutboxLongRetries.NONE;
}
Instant oldest = (Instant) row[1];
return new OutboxLongRetries(count, Math.max(0, Duration.between(oldest, now).toSeconds()));
}
/**
* Restores the appended event. A row without its envelope — written by the pre-canonical
* generation, or damaged — is unreadable, and so is one whose stored hash is not the hash of its
@@ -62,18 +62,22 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
@Param("token") long token,
@Param("status") String status);
/** {@link #completeLease} to FAILED, scheduling the retry in the same statement. */
/**
* {@link #completeLease} to FAILED, scheduling the retry and recording why (V22) in the same
* statement.
*/
@Modifying(flushAutomatically = true)
@Query(
"update OutboxEventEntity e set e.status = 'FAILED', e.nextAttemptAt = :nextAttemptAt, "
+ "e.leaseOwner = null "
+ "e.lastFailureCode = :failureCode, e.leaseOwner = null "
+ "where e.eventId = :eventId and e.status = 'IN_FLIGHT' "
+ "and e.leaseOwner = :owner and e.leaseToken = :token")
int failLease(
@Param("eventId") String eventId,
@Param("owner") String owner,
@Param("token") long token,
@Param("nextAttemptAt") Instant nextAttemptAt);
@Param("nextAttemptAt") Instant nextAttemptAt,
@Param("failureCode") String failureCode);
/**
* {@link #failLease} for a claim that ended deferred: same transition, and the row counts the
@@ -82,14 +86,29 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
@Modifying(flushAutomatically = true)
@Query(
"update OutboxEventEntity e set e.status = 'FAILED', e.nextAttemptAt = :nextAttemptAt, "
+ "e.deferredAttemptCount = e.deferredAttemptCount + 1, e.leaseOwner = null "
+ "e.deferredAttemptCount = e.deferredAttemptCount + 1, "
+ "e.lastFailureCode = :failureCode, e.leaseOwner = null "
+ "where e.eventId = :eventId and e.status = 'IN_FLIGHT' "
+ "and e.leaseOwner = :owner and e.leaseToken = :token")
int deferLease(
@Param("eventId") String eventId,
@Param("owner") String owner,
@Param("token") long token,
@Param("nextAttemptAt") Instant nextAttemptAt);
@Param("nextAttemptAt") Instant nextAttemptAt,
@Param("failureCode") String failureCode);
/** {@link #completeLease} to DEAD, recording why (V22) in the same statement. */
@Modifying(flushAutomatically = true)
@Query(
"update OutboxEventEntity e set e.status = 'DEAD', e.lastFailureCode = :failureCode, "
+ "e.leaseOwner = null "
+ "where e.eventId = :eventId and e.status = 'IN_FLIGHT' "
+ "and e.leaseOwner = :owner and e.leaseToken = :token")
int deadLease(
@Param("eventId") String eventId,
@Param("owner") String owner,
@Param("token") long token,
@Param("failureCode") String failureCode);
/**
* Gives an unused claim back: PENDING, claimable from {@code claimableAt}, and the claim's
@@ -110,12 +129,18 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
/**
* Increments an aggregate's order counter (V21), taking its row lock until the caller commits.
* Returns 0 when the aggregate has no counter yet.
* Never below the aggregate's highest outbox row: an instance still on the wall-clock scheme may
* have written a larger number after the counter existed (rolling deploy, review RR-03-02).
* Returns 0 when the aggregate has no counter yet, or when another transaction's new counter is
* not committed yet.
*/
@Modifying(flushAutomatically = true)
@Query(
value =
"UPDATE outbox_aggregate_sequence SET last_sequence = last_sequence + 1"
"UPDATE outbox_aggregate_sequence SET last_sequence = GREATEST(last_sequence,"
+ " (SELECT COALESCE(MAX(e.aggregate_sequence), 0) FROM outbox_event e"
+ " WHERE e.tenant_scope = :tenantScope AND e.aggregate_type = :aggregateType"
+ " AND e.aggregate_id = :aggregateId)) + 1"
+ " WHERE tenant_scope = :tenantScope AND aggregate_type = :aggregateType"
+ " AND aggregate_id = :aggregateId",
nativeQuery = true)
@@ -125,8 +150,10 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
@Param("aggregateId") String aggregateId);
/**
* Creates an aggregate's counter, seeded after the highest order its outbox rows already carry.
* Two creators of the same counter collide on its primary key.
* Creates an aggregate's counter at its first number: one after the larger of its highest outbox
* row and the V21 floor (0 where no migration wrote one). Returns 0, and writes nothing, when
* another transaction created it first; {@code ON CONFLICT DO NOTHING} waits for that transaction
* to end rather than failing on the primary key (review RR-03-01).
*/
@Modifying(flushAutomatically = true)
@Query(
@@ -134,9 +161,12 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
"INSERT INTO outbox_aggregate_sequence"
+ " (tenant_scope, aggregate_type, aggregate_id, last_sequence)"
+ " SELECT :tenantScope, :aggregateType, :aggregateId,"
+ " COALESCE(MAX(e.aggregate_sequence), 0) + 1 FROM outbox_event e"
+ " GREATEST(COALESCE(MAX(e.aggregate_sequence), 0),"
+ " COALESCE((SELECT f.floor_sequence FROM outbox_aggregate_sequence_floor f), 0))"
+ " + 1 FROM outbox_event e"
+ " WHERE e.tenant_scope = :tenantScope AND e.aggregate_type = :aggregateType"
+ " AND e.aggregate_id = :aggregateId",
+ " AND e.aggregate_id = :aggregateId"
+ " ON CONFLICT DO NOTHING",
nativeQuery = true)
int createAggregateSequence(
@Param("tenantScope") String tenantScope,
@@ -174,4 +204,14 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
"select e.eventType, min(e.occurredAt) from OutboxEventEntity e "
+ "where e.status <> 'PUBLISHED' group by e.eventType")
List<Object[]> findOldestUnpublishedOccurredAtByEventType();
/**
* {@code [count (Long), oldestOccurredAt (Instant)]} of the FAILED rows with at least {@code
* minDeferredClaims} deferred claims, for the long-retry gauges. One row, whose instant is null
* when the count is 0.
*/
@Query(
"select count(e), min(e.occurredAt) from OutboxEventEntity e "
+ "where e.status = 'FAILED' and e.deferredAttemptCount >= :minDeferredClaims")
List<Object[]> countLongRetries(@Param("minDeferredClaims") int minDeferredClaims);
}
@@ -0,0 +1,32 @@
package dev.caskeleton.adapter.outbound.persistence.outbox.jpa.entity;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
/**
* The one-row floor a new aggregate counter starts after ({@code outbox_aggregate_sequence_floor},
* V21): the moment V21 ran, in epoch microseconds, above every number the replaced wall-clock
* scheme wrote. Read by {@code JpaOutboxStore#nextSequence} with a native statement; mapped, like
* {@link OutboxAggregateSequenceEntity}, so the table exists where Hibernate builds the schema (H2,
* where no migration writes the row and the floor is 0) and is validated where Flyway does.
* Package-private: no code reads it as an entity.
*/
@Entity
@Table(name = "outbox_aggregate_sequence_floor")
class OutboxAggregateSequenceFloorEntity {
@Id
@Column(name = "singleton", nullable = false)
private boolean singleton;
@Column(name = "floor_sequence", nullable = false)
private long floorSequence;
protected OutboxAggregateSequenceFloorEntity() {}
long floorSequence() {
return floorSequence;
}
}
@@ -58,6 +58,18 @@ public class OutboxEventEntity implements Persistable<String> {
private int deferredAttemptCount;
/** Dual-purpose per status (PENDING / IN_FLIGHT / FAILED) — see README "outbox". */
/**
* The code of the row's last failed completion (V22): the publisher's own name for the failure,
* or the outcome or exception type when it gave none.
*
* <p>Read-only to the entity: only the relay's failed completions write it, each in its one
* fenced UPDATE ({@code OutboxEventJpaRepository#failLease} and its siblings). The append's
* INSERT and the claim's flush leave it out, so they neither clear a code nor need the column on
* a database that has not run V22 yet.
*/
@Column(name = "last_failure_code", length = 120, insertable = false, updatable = false)
private String lastFailureCode;
@Column(name = "next_attempt_at", nullable = false)
private Instant nextAttemptAt;
@@ -379,4 +391,8 @@ public class OutboxEventEntity implements Persistable<String> {
public void setCausationId(String causationId) {
this.causationId = causationId;
}
public String getLastFailureCode() {
return lastFailureCode;
}
}
@@ -123,7 +123,9 @@ public final class SpringTransaction implements TransactionPort {
* dry and every other request fails at {@code connection-timeout}. With one, Spring hands the
* remaining time to every statement (Hibernate and {@code JdbcTemplate} both set the JDBC query
* timeout), so the server cancels the statement ({@code 57014}, {@code DB_QUERY_CANCELED}) and
* the connection comes back.
* the connection comes back. A statement Hibernate is asked to run after the deadline has passed
* is refused by Hibernate itself, with no SQLState; this class reports that as {@code
* DB_QUERY_CANCELED} too (review RR-03-03).
*
* <p>The default is 30 seconds for both: the budget {@code inNew} already has, and the Hikari
* {@code leak-detection-threshold} at which the pool already reports a held connection as a leak.
@@ -194,6 +196,9 @@ public final class SpringTransaction implements TransactionPort {
private <T> T execute(ModeTemplate mode, Supplier<T> action) {
long startedAt = System.nanoTime();
// Whether this call began the transaction, and so owns its deadline. A call that joined an
// outer one leaves the deadline to the call that began it.
AtomicBoolean beganTransaction = new AtomicBoolean();
// Set only once the action has returned in a write transaction this call began: from then on
// the only thing TransactionTemplate.execute still does is commit it.
AtomicBoolean committingOwnWrite = new AtomicBoolean();
@@ -201,20 +206,32 @@ public final class SpringTransaction implements TransactionPort {
return mode.template()
.execute(
status -> {
beganTransaction.set(status.isNewTransaction());
T result = action.get();
committingOwnWrite.set(status.isNewTransaction() && !mode.template().isReadOnly());
return result;
});
} catch (RuntimeException failure) {
Duration elapsed = Duration.ofNanos(System.nanoTime() - startedAt);
Optional<PersistenceFailureException> translated =
committingOwnWrite.get()
? exceptionTranslator.translateCommitFailure(
failure, mode.commitOperation(), Duration.ofNanos(System.nanoTime() - startedAt))
? exceptionTranslator.translateCommitFailure(failure, mode.commitOperation(), elapsed)
: exceptionTranslator.translate(failure);
if (translated.isEmpty() && beganTransaction.get() && pastDeadline(mode, elapsed)) {
// Hibernate's own deadline refusal carries no SQLState (review RR-03-03).
translated = exceptionTranslator.translateAfterDeadline(failure);
}
throw translated.map(RuntimeException.class::cast).orElse(failure);
}
}
/** Whether {@code elapsed} reached the timeout the mode's transactions run under. */
private static boolean pastDeadline(ModeTemplate mode, Duration elapsed) {
int timeoutSeconds = mode.template().getTimeout();
return timeoutSeconds != TransactionDefinition.TIMEOUT_DEFAULT
&& elapsed.compareTo(Duration.ofSeconds(timeoutSeconds)) >= 0;
}
/**
* The vendor-neutral default translator.
*
@@ -7,8 +7,32 @@
--
-- outbox_aggregate_sequence holds one counter per aggregate. Incrementing it locks the row until
-- the writer commits, so the numbers follow commit order; it outlives the reaper's deletes, so a
-- number is never reused. JpaOutboxAggregateSequences seeds a new counter from the aggregate's
-- highest existing aggregate_sequence, so rows numbered by the old clock scheme still come first.
-- number is never reused. JpaOutboxStore#nextSequence reads and writes it.
--
-- Consumers have already seen clock numbers (about 1.7e15) for aggregates whose rows this
-- database no longer holds: the reaper deleted them, or they were written under an identity that
-- has since changed (the sample's tenant scope). A counter that started after the rows still
-- present would start at 1 for those aggregates, behind numbers already sent (review RR-03-02,
-- RR-05-02). So this migration records one floor, the moment it runs in epoch microseconds, which
-- is above every clock number written before it; a counter created later starts after the floor
-- or after the aggregate's highest remaining row, whichever is larger. The aggregates that still
-- have rows get their counter here, at their highest number. A clock number an instance still on
-- the old version writes during the rollout is above the floor; nextSequence also never goes below
-- the highest row it can see, so such a row still comes first.
--
-- This file was edited after it was first written, before any deployment applied it (the branch
-- that added it was not merged): splitting the floor into a V22 would have left V21 alone handing
-- out numbers from 1 for one release, and its comment pointed at a class that does not exist.
-- applied-migrations.sha256 carries the new checksum. A development database that applied the
-- earlier text fails Flyway validation; recreate it.
-- Bound the wait for the table lock the unique index below takes (SHARE on outbox_event, held until
-- this migration commits). Without a bound, the migration queues behind any open outbox write and
-- every outbox write after it queues behind the migration. Five seconds, then this migration fails,
-- rolls back whole and the database stays at V20: deploy again, or build the index first
-- (runbook §8, CONCURRENTLY).
SET LOCAL lock_timeout = '5s';
CREATE TABLE IF NOT EXISTS outbox_aggregate_sequence (
tenant_scope VARCHAR(96) NOT NULL,
aggregate_type VARCHAR(64) NOT NULL,
@@ -18,10 +42,55 @@ CREATE TABLE IF NOT EXISTS outbox_aggregate_sequence (
CONSTRAINT ck_outbox_aggregate_sequence_positive CHECK (last_sequence >= 1)
);
-- One row, written once: the lowest number a counter created from now on may start after.
CREATE TABLE IF NOT EXISTS outbox_aggregate_sequence_floor (
singleton BOOLEAN NOT NULL DEFAULT TRUE,
floor_sequence BIGINT NOT NULL,
CONSTRAINT pk_outbox_aggregate_sequence_floor PRIMARY KEY (singleton),
CONSTRAINT ck_outbox_aggregate_sequence_floor_singleton CHECK (singleton),
CONSTRAINT ck_outbox_aggregate_sequence_floor_non_negative CHECK (floor_sequence >= 0)
);
INSERT INTO outbox_aggregate_sequence_floor (singleton, floor_sequence)
VALUES (TRUE, (extract(epoch FROM clock_timestamp()) * 1000000)::BIGINT)
ON CONFLICT (singleton) DO NOTHING;
INSERT INTO outbox_aggregate_sequence (tenant_scope, aggregate_type, aggregate_id, last_sequence)
SELECT tenant_scope, aggregate_type, aggregate_id, MAX(aggregate_sequence)
FROM outbox_event
WHERE tenant_scope IS NOT NULL
AND aggregate_type IS NOT NULL
AND aggregate_sequence IS NOT NULL
GROUP BY tenant_scope, aggregate_type, aggregate_id
ON CONFLICT (tenant_scope, aggregate_type, aggregate_id) DO NOTHING;
-- One number, one event: a second event appended under an order its aggregate already used is
-- refused at append time, in the writer's transaction, rather than released by the gate in no
-- defined order. Rows without canonical order columns (pre-canonical, all NULL) are not compared.
-- This fails if duplicates already exist; resolve them (they are exactly the rows the gate could
-- not order) before migrating.
-- not order) before migrating (runbook §8).
--
-- Built here, inside this migration's transaction, not CONCURRENTLY: a duplicate then rolls the
-- whole of V21 back and leaves the database at V20 with nothing to clean up, which is what the
-- runbook's retry relies on. A failed CONCURRENTLY build leaves an INVALID index that IF NOT EXISTS
-- would then skip, so the uniqueness would silently not hold; and CONCURRENTLY cannot run in the
-- transaction that writes the counters and the floor. The reaper bounds outbox_event to the
-- retention window, so the build is normally short. An outbox too large to lock for that long is
-- indexed first with the runbook's CONCURRENTLY statement (same name, same columns); the check
-- below refuses an index such a build left invalid, and IF NOT EXISTS keeps the valid one.
DO $$
BEGIN
IF EXISTS (SELECT 1
FROM pg_index
WHERE indexrelid = to_regclass('ux_outbox_event_aggregate_order')
AND NOT (indisvalid AND indisunique)) THEN
RAISE EXCEPTION 'ux_outbox_event_aggregate_order exists but is not a valid unique index'
USING HINT = 'A CREATE INDEX CONCURRENTLY failed or is still running. '
|| 'DROP INDEX CONCURRENTLY ux_outbox_event_aggregate_order, then rebuild it '
|| 'or let V21 build it (runbook outbox-dead-letter §8).';
END IF;
END
$$;
CREATE UNIQUE INDEX IF NOT EXISTS ux_outbox_event_aggregate_order
ON outbox_event (tenant_scope, aggregate_type, aggregate_id, aggregate_sequence, event_index);
@@ -0,0 +1,21 @@
-- Why a row keeps failing, kept on the row.
--
-- A publish refused for now (a closed transport, a credential refused) or left unanswered is
-- retried without spending the attempt budget (V20), so it has no end: the relay retries it five
-- minutes apart for as long as nobody looks. The outcome the relay records cannot say why — a
-- shutdown, a closed transport and a refused credential are one outcome — and the relay's log line
-- said it in one sentence for all four. The publisher's failure code can (a messaging platform code
-- such as KAFKA_AUTHENTICATIONEXCEPTION); every failed completion writes it here, so the runbook's
-- query for rows that have been retrying long shows what each one is waiting on (re-review
-- 2026-11-03 RR-05-03).
--
-- A code, never a message: a driver message can carry an endpoint, a statement or a credential.
-- Nullable with no default, so adding it rewrites no row; rows written before it stay NULL until
-- their next failure.
--
-- The new status PARKED needs no DDL: status has no CHECK, the claim selects PENDING, FAILED and
-- IN_FLIGHT only (so the partial index ix_outbox_event_eligible leaves PARKED out), and the order
-- gate and its index ix_outbox_event_unpublished_aggregate_order treat every status but PUBLISHED as
-- blocking — which is what holding a row in its aggregate's place means.
ALTER TABLE outbox_event
ADD COLUMN last_failure_code VARCHAR(120);
@@ -23,7 +23,8 @@ import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
/**
* What a worker's death leaves behind, at each phase of a dispatch.
* What a worker's death leaves behind, at each phase of a dispatch — the rows, not what recovery
* does with them.
*
* <p>This is the {@code recoverable} claim in {@code docs/notification/evidence-manifest.json}, and
* until now nothing backed it. Lease recovery and event replay had unit coverage against fakes; the
@@ -36,9 +37,13 @@ import org.junit.jupiter.api.Test;
* worker is from the database's point of view — a lease nobody will renew and a phase nobody will
* advance.
*
* <p>The matrix walks the phases a dispatch passes through. For each, the question is the same and
* has only two acceptable answers: the delivery becomes claimable again, or it becomes a question
* for the provider. Never both, and never neither.
* <p>The matrix walks the phases a dispatch passes through and pins, for each, the state recovery
* starts from: a lease that has passed (the claim takes the row again) and, from the attempt insert
* on, an unfinished attempt row carrying the evidence of that phase (what reconciliation asks the
* provider about). These rows are written by this test, so the tests check the shape of the input
* and the predicates recovery reads, not recovery itself. {@code
* PostgreSqlNotificationBackgroundPassIntegrationTest} runs recovery and reconciliation over rows
* the dispatcher's own factory wrote (review 05-01, RR-03-07).
*/
@Tag("jpa-contract")
class WorkerCrashRecoveryContractTest {
@@ -73,8 +78,8 @@ class WorkerCrashRecoveryContractTest {
}
@Test
@DisplayName("a crash after claiming and before any attempt leaves the delivery claimable")
void aCrashBeforeAnyAttemptLeavesItClaimable() throws SQLException {
@DisplayName("claim 뒤 attempt 전에 죽은 delivery는 attempt 행이 없고 lease가 지나 있다")
void aDeliveryDeadBeforeAnyAttemptHasNoAttemptRowAndAPassedLease() throws SQLException {
UUID delivery = deliveryHeldByADeadWorker();
assertThat(isClaimableAgain(delivery))
@@ -88,12 +93,12 @@ class WorkerCrashRecoveryContractTest {
/**
* The attempt row is written before the call and keeps the factory's "unknown" evidence for the
* whole call (review 05-01), so a crash anywhere between the insert and the provider's answer
* leaves the same row. Recovery cannot tell "never called" from "called and died", so the row
* goes to reconciliation rather than back to the queue.
* leaves the same row. This test writes that row itself; that recovery hands it to reconciliation
* is {@code PostgreSqlNotificationBackgroundPassIntegrationTest}'s.
*/
@Test
@DisplayName("attempt 행을 쓴 뒤 호출이 끝나기 전 죽으면 시작 여부 불명인 미완료 attempt 가 남아 reconcile 대상이 된다")
void aCrashAfterTheAttemptRowIsWrittenLeavesAnUnfinishedAttemptOfUnknownStart()
@DisplayName("attempt 행을 쓰고 호출 전에 죽은 delivery는 시작 여부 불명인 미완료 attempt와 지난 lease를 남긴다")
void aDeliveryDeadAfterTheAttemptInsertLeavesAnUnfinishedAttemptOfUnknownStart()
throws SQLException {
// given
UUID delivery = deliveryHeldByADeadWorker();
@@ -103,17 +108,18 @@ class WorkerCrashRecoveryContractTest {
// then
assertThat(evidenceOf(attempt))
.as("the row the factory writes records that whether the call started is not known")
.as("the row stands for the factory's: whether the call started is not known")
.containsExactly("false", "UNKNOWN", "false", "UNKNOWN");
assertThat(isUnfinished(attempt))
.as("an unfinished attempt is what recovery hands to reconciliation")
.as("completed_at is empty, the predicate recovery selects unfinished attempts by")
.isTrue();
assertThat(isClaimableAgain(delivery)).isTrue();
}
@Test
@DisplayName("a crash after the request started is a question for the provider, not a requeue")
void aCrashAfterTheRequestStartedIsAQuestion() throws SQLException {
@DisplayName("요청 시작 뒤 죽은 delivery는 시작 증거가 있는 미완료 attempt와 지난 lease를 남긴다")
void aDeliveryDeadAfterTheRequestStartedLeavesAnUnfinishedAttemptWithStartEvidence()
throws SQLException {
UUID delivery = deliveryHeldByADeadWorker();
UUID attempt = attemptOn(delivery, true, "PROVEN", false, "UNKNOWN");
@@ -125,13 +131,14 @@ class WorkerCrashRecoveryContractTest {
"the lease still expires — the row must not be stranded — but the attempt is unfinished")
.isTrue();
assertThat(isUnfinished(attempt))
.as("an unfinished attempt is what recovery hands to reconciliation")
.as("completed_at is empty, the predicate recovery selects unfinished attempts by")
.isTrue();
}
@Test
@DisplayName("a crash after the body was committed is never silently retried")
void aCrashAfterTheBodyWasCommittedIsNeverSilentlyRetried() throws SQLException {
@DisplayName("본문 전송 뒤 죽은 delivery는 전송 증거가 있는 미완료 attempt를 남긴다")
void aDeliveryDeadAfterTheBodyWasSentLeavesAnUnfinishedAttemptWithSendEvidence()
throws SQLException {
UUID delivery = deliveryHeldByADeadWorker();
UUID attempt = attemptOn(delivery, true, "PROVEN", true, "PROVEN");
@@ -142,8 +149,8 @@ class WorkerCrashRecoveryContractTest {
}
@Test
@DisplayName("a completed attempt is not recovered at all")
void aCompletedAttemptIsNotRecovered() throws SQLException {
@DisplayName("완료 시각이 적힌 attempt는 미완료로 보이지 않는다")
void aCompletedAttemptIsNotUnfinished() throws SQLException {
UUID delivery = deliveryHeldByADeadWorker();
UUID attempt = attemptOn(delivery, true, "PROVEN", true, "PROVEN");
complete(attempt);
@@ -154,8 +161,9 @@ class WorkerCrashRecoveryContractTest {
}
@Test
@DisplayName("across every phase, exactly one delivery row survives with one attempt each")
void acrossEveryPhaseNothingIsLostOrDuplicated() throws SQLException {
@DisplayName("네 단계에서 죽은 워커들은 delivery 4개와 attempt 3개를 남기고 모든 lease가 지나 있다")
void deadWorkersAcrossEveryPhaseLeaveFourDeliveriesThreeAttemptsAndPassedLeases()
throws SQLException {
List<UUID> deliveries = new ArrayList<>();
// The four crash phases, run together, so the recovery sweep sees a mixed table rather than one
// hand-picked row.
@@ -59,7 +59,14 @@ import org.springframework.transaction.PlatformTransactionManager;
class DurableOperationSubmissionOutboxIntegrationTest {
private static final Instant NOW = Instant.parse("2026-08-25T09:00:00Z");
private static final DurableOperationId ID = new DurableOperationId("op-1");
/**
* A canonical lower-case UUIDv7, as the HTTP adapter mints: the operation id is then the event id
* and the aggregate id itself (an id of any other spelling becomes a UUIDv7 derived from it).
*/
private static final String OPERATION_UUID = "01994e11-4d88-7000-8000-0000000000a1";
private static final DurableOperationId ID = new DurableOperationId(OPERATION_UUID);
private static JpaPlatformContractSupport support;
private static AnnotationConfigApplicationContext context;
@@ -103,7 +110,7 @@ class DurableOperationSubmissionOutboxIntegrationTest {
// then
assertThat(queryStrings("SELECT contract_id || '|' || aggregate_id FROM outbox_event"))
.containsExactly("durable-operation.submitted|op-1");
.containsExactly("durable-operation.submitted|" + OPERATION_UUID);
}
@Test

Some files were not shown because too many files have changed in this diff Show More