fix!: 재리뷰가 찾은 결함과 색인에서 빠졌던 발견 5건을 고친다
문제: 성숙도 리뷰 수정을 다시 읽은 재리뷰(Critical 0, 회귀 0)가 Important 27건·Minor 41건을 찾았다. 수정이 새로 만든 결함이 많았다. @SystemInvoked가 권한 규칙의 탈출구가 됐고, 데모 계정 거부가 정확히 prod일 때만 걸렸고, outbox 순서 번호가 업그레이드 때 뒤로 갔고, GraphQL union 아래 복잡도가 0이었고, 서블릿 오류 응답이 success:true로 나갔고, 호출자 deadline이 breaker를 열었다. 원래 리뷰 색인이 06 영역 발견 5건(06-10~14)을 빠뜨려 그동안 처리되지 않았다. 방법: 영역별 7묶음이 test-driven-development 절차로 실패 테스트를 먼저 쓰고 고쳤다. 결정은 재리뷰 README 4절에 적었다: 순서 번호는 V21이 기록한 전역 floor부터, 데모 계정은 local·test·demo 허용 목록, @SystemInvoked bean은 기동 실패, complete 단계 실패는 IDEMPOTENCY_COMPLETION_UNKNOWN(결과 불명), 캐시 조건부 쓰기는 별도 SCRIPT lane, strict lane 정책은 build cache 입력, 끝나지 않는 재시도 행은 원인 코드·지표·PARKED 절차. 레지스트리가 선언만 하고 아무 코드도 내보내지 않던 redis.capability.* 지표 6개는 빼고 runbook을 실제 지표(redis.lane.*) 기준으로 고쳤다. 위험·한계: V21을 V22 없이 직접 고쳤다(어느 배포에도 아직 없음). 이 브랜치의 V21을 적용한 개발 DB는 다시 만들어야 한다. 서블릿 오류 디스패치를 보안 체인이 허용한다. 발행·저장소 포트 시그니처가 바뀌어 파생 서비스 코드가 깨진다. prod에서 fixture-로 시작하는 설정값은 기동을 막는다. httpclient failure-injection lane의 main 단계 중복 실행은 이전 결정대로 남겼다. 실제 CI runner 실행은 확인하지 못했다. 검증: ci, mainCheck, :integrationCheck, -p optional-platforms ci integrationCheck performanceCheck, :performanceCheck, :releaseCheck (-PgitRevision), verify-compose-profile-contracts(18 lane) 통과. 첫 실행의 실패 2건(FindSecBugs 1, sample 동시 생성 테스트의 v4 event id)은 고친 뒤 ci·integrationCheck·mainCheck를 다시 돌려 통과했다. Refs: docs/reviews/2026-11-03-rereview/README.md 3·4·6·7절 BREAKING CHANGE: outbox 발행 포트와 저장소 포트 시그니처, 순서 번호 첫 값(V21 floor 다음), 샘플 데모 계정 프로필 허용 목록, 새 오류 코드 IDEMPOTENCY_COMPLETION_UNKNOWN, prod의 fixture- 값 거부, 기동 rule은 static·의존 없음이어야 한다.
This commit is contained in:
@@ -77,7 +77,12 @@ instrumentation, data fetcher resolver 세 곳이 같은 bean 을 받는다).
|
||||
- **client 문구**: shared-contract `ClientErrorMessages`(Category 별 한국어 문구 표 하나, 소유자
|
||||
결정 11). 예외 메시지는 절대 나가지 않는다.
|
||||
- **retryable**: 코드의 `retryable()` 이다(category 에서 계산하지 않는다). 플랫폼 자체 코드만 wire
|
||||
category 기본값을 쓴다.
|
||||
category 기본값을 쓴다. 그래서 공유 코드의 값을 바꾸면 이 wire 도 바뀐다: 처리 중 중복
|
||||
`IDEMPOTENT_IN_FLIGHT` 는 2026-11-01 리뷰(02-01)부터 `retryable: true` 다 — 같은 키로 다시 보내면 서버가
|
||||
판정한다(`docs/web/http-contract.md` §2). `GraphQlWireErrorMapperTest` 가 고정한다(재리뷰 RR-02-06).
|
||||
- **`REQUEST_TIMEOUT` 은 web 과 이름만 같다.** 여기의 `category` 는 GraphQL wire 전용 6값 어휘(`TIMEOUT`)이고,
|
||||
web 의 같은 이름 코드는 HTTP 503·operational category `TRANSIENT_DEPENDENCY` 다
|
||||
(`docs/web/http-contract.md` §4). 둘 다 `retryable: true` 다.
|
||||
- **body 크기 초과**: `PAYLOAD_TOO_LARGE`(web 과 같은 코드). 문서 모양 거부(`GRAPHQL_DOCUMENT_SHAPE_REJECTED`)는
|
||||
별도 코드.
|
||||
|
||||
|
||||
+3
@@ -14,6 +14,9 @@ import java.time.Instant;
|
||||
* <li>{@code GraphQlPlatformWebInterceptor} ends the request with {@code REQUEST_TIMEOUT} when it
|
||||
* passes, and Spring cancels the execution so no further field is dispatched. A resolver
|
||||
* already running is not interrupted.
|
||||
* <li>{@code GraphQlDeadlineInstrumentation} refuses to start a non-trivial resolver once it has
|
||||
* passed, failing that field with {@code REQUEST_TIMEOUT}; this is what still holds when the
|
||||
* whole execution runs on the request thread and the interceptor's timer starts too late.
|
||||
* <li>{@code GraphQlCostBudgetHandler} refuses to start an execution whose deadline has passed.
|
||||
* <li>{@code GraphQlBatchExecutor} clamps each platform-registered DataLoader batch to what is
|
||||
* left, and stops between chunks once it is spent.
|
||||
|
||||
-5
@@ -121,11 +121,6 @@ public final class GraphQlComplexityCalculator {
|
||||
}
|
||||
}
|
||||
|
||||
/** The effective page size for a connection request. */
|
||||
public int effectivePageSize(Integer first, Integer last) {
|
||||
return first != null ? first : last != null ? last : defaultPageSize;
|
||||
}
|
||||
|
||||
private void requireAllowedPage(Integer requested) {
|
||||
if (requested == null) {
|
||||
return;
|
||||
|
||||
+25
-13
@@ -15,6 +15,7 @@ import graphql.language.Value;
|
||||
import graphql.language.VariableDefinition;
|
||||
import graphql.language.VariableReference;
|
||||
import graphql.schema.GraphQLArgument;
|
||||
import graphql.schema.GraphQLCompositeType;
|
||||
import graphql.schema.GraphQLFieldDefinition;
|
||||
import graphql.schema.GraphQLFieldsContainer;
|
||||
import graphql.schema.GraphQLList;
|
||||
@@ -59,6 +60,13 @@ import java.util.Objects;
|
||||
* <p>Pricing an omitted argument as one row was the bypass: {@code items(first: 20)} scored 410
|
||||
* while {@code items} scored 40, and four nested lists added up instead of multiplying.
|
||||
*
|
||||
* <p>A selection set is walked whatever kind of composite type it selects from. A union has no
|
||||
* fields of its own — only {@code __typename} — so its selections are inline fragments and fragment
|
||||
* spreads, each priced against its type condition. Treating "not a fields container" as "nothing
|
||||
* below" was the second bypass (re-review 2026-11-03, RR-02-01): {@code search { ... on Post {
|
||||
* comments(first: 100000) } }} scored 20, and neither the budget nor the page maximum applied under
|
||||
* any union-typed field.
|
||||
*
|
||||
* <p>Traversal is bounded, and fragment cycles are cut by tracking the expansion path. This runs on
|
||||
* documents that have passed validation, but the bound stays because the scorer is also used from
|
||||
* the pre-execution path where a hostile document is exactly what it is meant to price.
|
||||
@@ -143,7 +151,7 @@ public final class GraphQlDocumentComplexityScorer {
|
||||
|
||||
private long selectionSetCost(
|
||||
GraphQLSchema schema,
|
||||
GraphQLFieldsContainer parent,
|
||||
GraphQLCompositeType parent,
|
||||
SelectionSet selectionSet,
|
||||
Map<String, FragmentDefinition> fragments,
|
||||
PageVariables variables,
|
||||
@@ -173,10 +181,10 @@ public final class GraphQlDocumentComplexityScorer {
|
||||
expansionPath,
|
||||
listsAreThePage));
|
||||
} else if (selection instanceof InlineFragment inlineFragment) {
|
||||
GraphQLFieldsContainer target =
|
||||
GraphQLCompositeType target =
|
||||
inlineFragment.getTypeCondition() == null
|
||||
? parent
|
||||
: fieldsContainer(schema, inlineFragment.getTypeCondition().getName());
|
||||
: compositeType(schema, inlineFragment.getTypeCondition().getName());
|
||||
total =
|
||||
GraphQlComplexityCalculator.saturatedAdd(
|
||||
total,
|
||||
@@ -196,10 +204,10 @@ public final class GraphQlDocumentComplexityScorer {
|
||||
if (fragment == null || expansionPath.contains(spread.getName())) {
|
||||
continue;
|
||||
}
|
||||
GraphQLFieldsContainer target =
|
||||
GraphQLCompositeType target =
|
||||
fragment.getTypeCondition() == null
|
||||
? parent
|
||||
: fieldsContainer(schema, fragment.getTypeCondition().getName());
|
||||
: compositeType(schema, fragment.getTypeCondition().getName());
|
||||
expansionPath.push(spread.getName());
|
||||
total =
|
||||
GraphQlComplexityCalculator.saturatedAdd(
|
||||
@@ -221,7 +229,7 @@ public final class GraphQlDocumentComplexityScorer {
|
||||
|
||||
private long fieldCost(
|
||||
GraphQLSchema schema,
|
||||
GraphQLFieldsContainer parent,
|
||||
GraphQLCompositeType parent,
|
||||
Field field,
|
||||
Map<String, FragmentDefinition> fragments,
|
||||
PageVariables variables,
|
||||
@@ -235,8 +243,12 @@ public final class GraphQlDocumentComplexityScorer {
|
||||
return 0;
|
||||
}
|
||||
|
||||
GraphQLFieldDefinition definition = parent.getFieldDefinition(field.getName());
|
||||
String coordinate = parent.getName() + "." + field.getName();
|
||||
// A union declares no fields; validation rejects anything but __typename directly under one.
|
||||
GraphQLFieldDefinition definition =
|
||||
parent instanceof GraphQLFieldsContainer container
|
||||
? container.getFieldDefinition(field.getName())
|
||||
: null;
|
||||
if (definition == null) {
|
||||
// Validation rejects an unknown field; price what is there rather than invent a type.
|
||||
return calculator.fieldCost(coordinate, 0).total();
|
||||
@@ -250,7 +262,7 @@ public final class GraphQlDocumentComplexityScorer {
|
||||
long childCost =
|
||||
selectionSetCost(
|
||||
schema,
|
||||
fieldsContainer(definition.getType()),
|
||||
compositeType(definition.getType()),
|
||||
field.getSelectionSet(),
|
||||
fragments,
|
||||
variables,
|
||||
@@ -353,14 +365,14 @@ public final class GraphQlDocumentComplexityScorer {
|
||||
};
|
||||
}
|
||||
|
||||
private static GraphQLFieldsContainer fieldsContainer(GraphQLSchema schema, String typeName) {
|
||||
GraphQLType type = schema.getType(typeName);
|
||||
return fieldsContainer(type);
|
||||
private static GraphQLCompositeType compositeType(GraphQLSchema schema, String typeName) {
|
||||
return compositeType(schema.getType(typeName));
|
||||
}
|
||||
|
||||
private static GraphQLFieldsContainer fieldsContainer(GraphQLType type) {
|
||||
/** The object, interface or union a selection set selects from; null for a leaf type. */
|
||||
private static GraphQLCompositeType compositeType(GraphQLType type) {
|
||||
GraphQLType unwrapped = unwrap(type);
|
||||
return unwrapped instanceof GraphQLFieldsContainer container ? container : null;
|
||||
return unwrapped instanceof GraphQLCompositeType composite ? composite : null;
|
||||
}
|
||||
|
||||
private static GraphQLType unwrap(GraphQLType type) {
|
||||
|
||||
+96
-2
@@ -34,11 +34,19 @@ class GraphQlDocumentComplexityScorerTest {
|
||||
pagedWithDefault(first: Int = 50): [Item!]!
|
||||
itemConnection(first: Int): ItemConnection
|
||||
node: Item
|
||||
allItems: [Item!]!
|
||||
search: [SearchResult!]!
|
||||
nodes: [Named!]!
|
||||
}
|
||||
type Item {
|
||||
union SearchResult = Item
|
||||
interface Named {
|
||||
name: String!
|
||||
}
|
||||
type Item implements Named {
|
||||
name: String!
|
||||
children: [Item!]!
|
||||
grid: [[Item!]!]!
|
||||
paged(first: Int): [Item!]!
|
||||
}
|
||||
type ItemConnection {
|
||||
edges: [ItemEdge!]!
|
||||
@@ -48,10 +56,15 @@ class GraphQlDocumentComplexityScorerTest {
|
||||
}
|
||||
""";
|
||||
|
||||
// Pricing never resolves a runtime type; the resolvers only satisfy the schema generator.
|
||||
private static final GraphQLSchema SCHEMA =
|
||||
new SchemaGenerator()
|
||||
.makeExecutableSchema(
|
||||
new SchemaParser().parse(SDL), RuntimeWiring.newRuntimeWiring().build());
|
||||
new SchemaParser().parse(SDL),
|
||||
RuntimeWiring.newRuntimeWiring()
|
||||
.type("SearchResult", type -> type.typeResolver(environment -> null))
|
||||
.type("Named", type -> type.typeResolver(environment -> null))
|
||||
.build());
|
||||
|
||||
private static final long BUDGET = 10_000;
|
||||
|
||||
@@ -246,6 +259,87 @@ class GraphQlDocumentComplexityScorerTest {
|
||||
assertThat(score).isEqualTo(20L + 20L);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("union 필드 아래 inline fragment 의 중첩 list 도 일반 list 와 같이 곱한다")
|
||||
void nestedListsUnderAUnionFieldMultiplyLikeAPlainList() {
|
||||
// given
|
||||
String plain = "query Q { allItems { children { children { children { name } } } } }";
|
||||
|
||||
// when
|
||||
long score =
|
||||
score(
|
||||
"query Q { search { ... on Item { children { children { children { name } } } } } }",
|
||||
Map.of());
|
||||
|
||||
// then
|
||||
assertThat(score).isEqualTo(score(plain, Map.of()));
|
||||
assertThatThrownBy(() -> calculator.verify(new GraphQlComplexityResult(score), BUDGET))
|
||||
.isInstanceOf(GraphQlComplexityRejectedException.class);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("union 필드 아래 fragment spread 도 그 타입으로 내려가 곱한다")
|
||||
void aFragmentSpreadUnderAUnionFieldIsPricedAgainstItsTypeCondition() {
|
||||
// given
|
||||
String plain = "query Q { allItems { children { children { children { name } } } } }";
|
||||
|
||||
// when
|
||||
long score =
|
||||
score(
|
||||
"query Q { search { ...F } } "
|
||||
+ "fragment F on Item { children { children { children { name } } } }",
|
||||
Map.of());
|
||||
|
||||
// then
|
||||
assertThat(score).isEqualTo(score(plain, Map.of()));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("union 필드 아래 page 인자도 상한을 넘으면 page size exceeds maximum 으로 거부된다")
|
||||
void aPageSizeUnderAUnionFieldIsCheckedAgainstTheMaximum() {
|
||||
// given
|
||||
String document = "query Q { search { ... on Item { paged(first: 100000) { name } } } }";
|
||||
|
||||
// when
|
||||
Throwable thrown = catchThrowable(() -> score(document, Map.of()));
|
||||
|
||||
// then
|
||||
assertThat(thrown)
|
||||
.isInstanceOf(GraphQlComplexityRejectedException.class)
|
||||
.hasMessageContaining("page size exceeds maximum");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("interface 필드 아래 inline fragment 의 중첩 list 도 일반 list 와 같이 곱한다")
|
||||
void nestedListsUnderAnInterfaceFieldMultiplyLikeAPlainList() {
|
||||
// given
|
||||
String plain = "query Q { allItems { children { children { children { name } } } } }";
|
||||
|
||||
// when
|
||||
long score =
|
||||
score(
|
||||
"query Q { nodes { ... on Item { children { children { children { name } } } } } }",
|
||||
Map.of());
|
||||
|
||||
// then
|
||||
assertThat(score).isEqualTo(score(plain, Map.of()));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("interface 필드 아래 page 인자도 상한을 넘으면 page size exceeds maximum 으로 거부된다")
|
||||
void aPageSizeUnderAnInterfaceFieldIsCheckedAgainstTheMaximum() {
|
||||
// given
|
||||
String document = "query Q { nodes { ... on Item { paged(first: 100000) { name } } } }";
|
||||
|
||||
// when
|
||||
Throwable thrown = catchThrowable(() -> score(document, Map.of()));
|
||||
|
||||
// then
|
||||
assertThat(thrown)
|
||||
.isInstanceOf(GraphQlComplexityRejectedException.class)
|
||||
.hasMessageContaining("page size exceeds maximum");
|
||||
}
|
||||
|
||||
private long score(String source, Map<String, Object> variables) {
|
||||
return score(calculator, source, variables);
|
||||
}
|
||||
|
||||
+23
@@ -55,6 +55,29 @@ class GraphQlWireErrorMapperTest {
|
||||
.containsExactly("DB_QUERY_CANCELED", false);
|
||||
}
|
||||
|
||||
/**
|
||||
* The in-flight duplicate's {@code retryable} became {@code true} in the 2026-11-01 review
|
||||
* (02-01): a resend with the same key is judged by the server, so it is safe. GraphQL carries the
|
||||
* code's own value, so the change reached this wire too; this pins it (re-review RR-02-06).
|
||||
*/
|
||||
@org.junit.jupiter.api.Test
|
||||
@DisplayName("처리 중 중복(IDEMPOTENT_IN_FLIGHT)은 GraphQL extension 에서도 retryable=true 로 나간다")
|
||||
void anInFlightDuplicateIsRetryableOnTheGraphQlWire() {
|
||||
// given
|
||||
GraphQlWireErrorMapper mapper = GraphQlWireErrorMapper.masking();
|
||||
RuntimeException failure =
|
||||
dev.caskeleton.application.idempotency.IdempotencyInFlightException.forScopeDigest(
|
||||
"digest");
|
||||
|
||||
// when
|
||||
GraphQlWireError error = mapper.map(failure, GraphQlErrorContext.test());
|
||||
|
||||
// then
|
||||
assertThat(error.extensions())
|
||||
.extracting(extensions -> extensions.get("code"), extensions -> extensions.get("retryable"))
|
||||
.containsExactly("IDEMPOTENT_IN_FLIGHT", true);
|
||||
}
|
||||
|
||||
@org.junit.jupiter.api.Test
|
||||
void anIllegalArgumentIsABadParameterRequestError() {
|
||||
// given
|
||||
|
||||
@@ -140,7 +140,8 @@ handler throw, listener callback throw, `responseObserver.onError(...)`, raw
|
||||
- **와이어 status(coarse)** 는 `GrpcStatusMapper.toStatus(Category)` 가 결정한다(HTTP status 가
|
||||
coarse 인 것과 동형). 정확한 `code`/`category`/`retryable` 은 `Status` trailer `Metadata`
|
||||
(`error-code` / `error-category` / `error-retryable`)에 실어 클라이언트가 switch 하게 한다.
|
||||
`retryable` 은 코드의 값이다. status description 은 shared-contract `ClientErrorMessages` 의
|
||||
`retryable` 은 코드의 값이다(처리 중 중복 `IDEMPOTENT_IN_FLIGHT` 는 2026-11-01 리뷰 02-01부터 `ABORTED` +
|
||||
`error-retryable: true` — 같은 키로 다시 보내면 서버가 판정한다. `GrpcStatusMapperTest` 가 고정, 재리뷰 RR-02-06). status description 은 shared-contract `ClientErrorMessages` 의
|
||||
Category 별 한국어 문구(web·GraphQL 과 같은 표)다.
|
||||
- **`IllegalArgumentException`** 은 `BAD_PARAMETER`(`INVALID_ARGUMENT`) — 모든 전송이 같다.
|
||||
- **`ApiErrorCode` 추출**: feature 예외는 `ApiErrorCarrier`(이 모듈이 제공하는 전송-중립 hook)를
|
||||
|
||||
+19
@@ -7,6 +7,7 @@ import dev.caskeleton.shared.error.OperationalError;
|
||||
import io.grpc.Metadata;
|
||||
import io.grpc.Status;
|
||||
import java.util.EnumSet;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.CsvSource;
|
||||
@@ -64,4 +65,22 @@ class GrpcStatusMapperTest {
|
||||
// then
|
||||
assertThat(retryable).isEqualTo("false");
|
||||
}
|
||||
|
||||
/**
|
||||
* The in-flight duplicate's {@code retryable} became {@code true} in the 2026-11-01 review
|
||||
* (02-01). The trailer carries the code's own value, so the change reached the gRPC wire as well:
|
||||
* status {@code ABORTED} (category {@code CONFLICT}) with {@code error-retryable: true}
|
||||
* (re-review RR-02-06).
|
||||
*/
|
||||
@Test
|
||||
@DisplayName("처리 중 중복(IDEMPOTENT_IN_FLIGHT)은 ABORTED 이고 trailer 의 retryable 은 true 다")
|
||||
void anInFlightDuplicateIsAbortedAndRetryableOnTheGrpcWire() {
|
||||
// when
|
||||
Metadata trailers = mapper.trailersFor(OperationalError.IDEMPOTENT_IN_FLIGHT);
|
||||
|
||||
// then
|
||||
assertThat(mapper.statusFor(OperationalError.IDEMPOTENT_IN_FLIGHT).getCode())
|
||||
.isEqualTo(Status.Code.ABORTED);
|
||||
assertThat(trailers.get(GrpcStatusMapper.RETRYABLE_KEY)).isEqualTo("true");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -241,6 +241,7 @@ commit-aware response wrapper 계약을 구현한다. 또한 이 모듈은 HTML
|
||||
| `PreconditionFailedException` | `PRECONDITION_FAILED` | 412 | D15: `If-Match` 불일치 쓰기 = 낙관적 동시성 충돌 → 412 (raw 409/500 금지) |
|
||||
| `PageValidationException` | `VALIDATION_FAILED` | 400 | D18/D20/D21: 계약 범위 밖 페이지/정렬/필터 파라미터. field + reasonCode 를 details 로 |
|
||||
| `IdempotencyUnavailableException` | `IDEMPOTENCY_UNAVAILABLE` | 503 (retryable=true) | 조정 저장소가 응답하지 않아 action 전에 거부. carrier 분기 |
|
||||
| `IdempotencyCompletionUnknownException` | `IDEMPOTENCY_COMPLETION_UNKNOWN` | 503 (retryable=true) | action 이 돈 뒤 결과 기록(complete) 중 저장소가 응답하지 않음. 결과 불명 — 같은 키로 재전송, 새 키 금지(`docs/web/http-contract.md` §2, 재리뷰 RR-02-03). carrier 분기 |
|
||||
| `IdempotencyRecoveryRequiredException` | `IDEMPOTENCY_RECOVERY_REQUIRED` | 500 (retryable=false) | 이전 시도의 외부 효과 여부를 판단할 수 없음. carrier 분기 |
|
||||
| `IdempotencyInFlightException` | `IDEMPOTENT_IN_FLIGHT` | 409 (retryable=true, `Retry-After: 2`) | 같은 scope 의 다른 시도가 아직 처리 중(executor 는 기다리지 않는다). 클라이언트 행동: 결과 불명. 같은 `Idempotency-Key`로 `Retry-After` 뒤 재전송. 새 키 금지(`docs/web/http-contract.md` §2). 진단정보(scope digest)는 클라이언트 미도달 |
|
||||
| `IdempotencyRequestMismatchException` | `IDEMPOTENT_REQUEST_MISMATCH` | 422 (retryable=false) | D8: `Idempotency-Key` 를 다른 본문으로 재사용. 다시 보내지 않는다(`docs/web/http-contract.md` §2). fingerprint/scope 노출 금지 |
|
||||
@@ -284,9 +285,26 @@ commit-aware response wrapper 계약을 구현한다. 또한 이 모듈은 HTML
|
||||
advice 가 같은 mapper 로 Envelope 를 문자열로 직렬화해 넘긴다. 텍스트를 요청한 클라이언트는 그대로
|
||||
문자열을 받는다. ArchUnit 으로 `String` 반환을 금지하는 안은 버렸다: `text/plain` 엔드포인트의 정당한
|
||||
용도를 막고, 규칙을 돌리지 않는 서비스에는 함정이 그대로 남는다.
|
||||
- **4xx·5xx 본문은 감싸지 않는다.** `success` 는 클라이언트가 가르는 판별자라, 오류 status 본문을
|
||||
`success:true` 로 감싸면 성공으로 읽힌다. Boot `/error` 의 map 이 그렇게 나갔다(재리뷰 RR-02-02).
|
||||
- 위치: sample 모듈이 아니라 adapter-web. 실행 앱은 adapter-web 에 의존하지만 sample-portfolio 에는 의존하지
|
||||
않으므로, 응답 래핑이 실제로 동작하려면 여기 있어야 한다.
|
||||
|
||||
### EnvelopeErrorController
|
||||
- 컨트롤러에 닿기 전에 생긴 오류(보안 방화벽의 `RequestRejectedException`, 필터의 `sendError`, 필터가
|
||||
던진 예외)는 `GlobalExceptionHandler` 를 지나지 않고 컨테이너의 `/error` 디스패치로 간다. Boot 기본
|
||||
`BasicErrorController` 는 거기서 `{timestamp, status, error, path}` 를 썼고, advice 가 그것을
|
||||
`success:true` 로 감쌌으며 `path` 는 요청 경로 원문이었다(재리뷰 RR-02-02). 이 컨트롤러가 그 자리를
|
||||
대신해 status 를 `GlobalExceptionHandler` 와 같은 표(`HttpStatusErrorCodes`)로 코드에 매핑하고 실패
|
||||
Envelope 를 `EnvelopeResponseWriter` 로 쓴다. 요청에서 가져온 값은 싣지 않는다.
|
||||
- `WebMvcPlatformAutoConfiguration` 이 `ErrorMvcAutoConfiguration` 보다 먼저 `ErrorController` 가 없을 때만
|
||||
등록한다. 어댑터가 자기 `ErrorController` 를 두면 물러난다.
|
||||
- `SecurityConfig` 는 오류 **디스패치**만 허용한다(`DispatcherType.ERROR`). 예전에는 `/error` 가 공개 경로가
|
||||
아니라서 익명 요청의 400 이 401 로 바뀌었다. `/error` 를 직접 부른 요청은 여전히 인증이 필요하다.
|
||||
단언은 실제 Tomcat 의 `TomcatWebContractIT`(공유 `WebContractAssertions`)와 `JwtChainErrorDispatchIT`.
|
||||
- 앱이 보안 체인을 직접 정의하면(예: sample 의 데모 체인) 같은 한 줄
|
||||
(`dispatcherTypeMatchers(DispatcherType.ERROR).permitAll()`)을 그 체인에도 둬야 원래 status 가 나간다.
|
||||
|
||||
### CacheControlFilter
|
||||
- 스켈레톤 기본 HTTP 캐시 정책.
|
||||
- `Cache-Control: no-store` 는 인증된 API 의 안전한 기본값. `Vary: Accept, Accept-Encoding, Authorization` 로
|
||||
|
||||
+6
@@ -2,6 +2,7 @@ package dev.caskeleton.adapter.inbound.web.auth;
|
||||
|
||||
import dev.caskeleton.adapter.inbound.web.settings.CorsSettings;
|
||||
import dev.caskeleton.adapter.inbound.web.settings.SecuritySettings;
|
||||
import jakarta.servlet.DispatcherType;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
@@ -83,6 +84,11 @@ public class SecurityConfig {
|
||||
.requestCache(cache -> cache.disable())
|
||||
.authorizeHttpRequests(
|
||||
auth -> {
|
||||
// The container's error dispatch for a request that already failed. Authorising it
|
||||
// again turned an anonymous 400 into a 401 that blamed authentication. Only the
|
||||
// dispatch type is opened: a request that names /error itself is still a request.
|
||||
// The error controller writes nothing from the request (re-review RR-02-02).
|
||||
auth.dispatcherTypeMatchers(DispatcherType.ERROR).permitAll();
|
||||
if (publicPaths.length > 0) {
|
||||
auth.requestMatchers(publicPaths).permitAll();
|
||||
}
|
||||
|
||||
+16
@@ -12,6 +12,7 @@ import org.springframework.http.converter.HttpMessageConverter;
|
||||
import org.springframework.http.converter.StringHttpMessageConverter;
|
||||
import org.springframework.http.server.ServerHttpRequest;
|
||||
import org.springframework.http.server.ServerHttpResponse;
|
||||
import org.springframework.http.server.ServletServerHttpResponse;
|
||||
import org.springframework.web.bind.annotation.RestControllerAdvice;
|
||||
import org.springframework.web.servlet.mvc.method.annotation.ResponseBodyAdvice;
|
||||
import tools.jackson.databind.ObjectMapper;
|
||||
@@ -34,6 +35,13 @@ import tools.jackson.databind.json.JsonMapper;
|
||||
* still sees {@code 200 text/plain}. So when the String converter is about to write JSON, the
|
||||
* envelope is serialised here, with the same mapper and wire rules, and the converter writes that
|
||||
* text (review 2026-11-01 02-06). A text client still gets the plain string.
|
||||
*
|
||||
* <p>A body sent with a 4xx or 5xx status is never wrapped. {@code success} is the discriminator a
|
||||
* client branches on, so a {@code success:true} envelope around an error-status body tells it the
|
||||
* request worked. Boot's {@code /error} map reached clients exactly that way until the error
|
||||
* dispatch got its own envelope writer ({@code EnvelopeErrorController}, re-review 2026-11-03
|
||||
* RR-02-02). Failure envelopes come from the error handlers; a handler that returns some other body
|
||||
* with an error status gets that body on the wire as it wrote it, not a false success.
|
||||
*/
|
||||
@RestControllerAdvice
|
||||
public final class EnvelopeBodyAdvice implements ResponseBodyAdvice<Object> {
|
||||
@@ -74,10 +82,18 @@ public final class EnvelopeBodyAdvice implements ResponseBodyAdvice<Object> {
|
||||
if (selectedContentType != null && !MediaType.APPLICATION_JSON.includes(selectedContentType)) {
|
||||
return body;
|
||||
}
|
||||
if (isErrorStatus(response)) {
|
||||
return body;
|
||||
}
|
||||
Envelope<Object> envelope = Envelope.ok(body, ResponseMetaFactory.fromMdc());
|
||||
if (StringHttpMessageConverter.class.isAssignableFrom(selectedConverterType)) {
|
||||
return objectMapper.writeValueAsString(envelope);
|
||||
}
|
||||
return envelope;
|
||||
}
|
||||
|
||||
private static boolean isErrorStatus(ServerHttpResponse response) {
|
||||
return response instanceof ServletServerHttpResponse servlet
|
||||
&& servlet.getServletResponse().getStatus() >= 400;
|
||||
}
|
||||
}
|
||||
|
||||
+1
-1
@@ -29,7 +29,7 @@ public class EnvelopeWireFormatConfig {
|
||||
}
|
||||
|
||||
/** The module itself, for a writer that has no Boot mapper to take it from. */
|
||||
static SimpleModule module() {
|
||||
public static SimpleModule module() {
|
||||
SimpleModule module = new SimpleModule("EnvelopeWireFormatModule");
|
||||
module.setMixInAnnotation(Envelope.class, OmitNullFields.class);
|
||||
module.setMixInAnnotation(BulkEnvelope.class, OmitNullFields.class);
|
||||
|
||||
+72
@@ -0,0 +1,72 @@
|
||||
package dev.caskeleton.adapter.inbound.web.error;
|
||||
|
||||
import dev.caskeleton.adapter.inbound.web.core.OwnsAbsoluteWebRoute;
|
||||
import dev.caskeleton.adapter.inbound.web.http.ApiHeaders;
|
||||
import dev.caskeleton.adapter.inbound.web.observability.RetryAfterAdvisor;
|
||||
import dev.caskeleton.shared.error.ApiErrorCode;
|
||||
import dev.caskeleton.shared.error.OperationalError;
|
||||
import jakarta.servlet.RequestDispatcher;
|
||||
import jakarta.servlet.http.HttpServletRequest;
|
||||
import jakarta.servlet.http.HttpServletResponse;
|
||||
import java.io.IOException;
|
||||
import java.util.Objects;
|
||||
import org.springframework.boot.webmvc.error.ErrorController;
|
||||
import org.springframework.http.HttpStatusCode;
|
||||
import org.springframework.stereotype.Controller;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import tools.jackson.databind.ObjectMapper;
|
||||
|
||||
/**
|
||||
* Answers the container's error dispatch with the failure {@code Envelope}.
|
||||
*
|
||||
* <p>A failure raised outside the MVC handler — the security firewall's {@code
|
||||
* RequestRejectedException}, a filter's {@code sendError}, an exception thrown by a filter — never
|
||||
* reaches {@link GlobalExceptionHandler}. The container forwards it to {@code /error} instead,
|
||||
* where Boot's {@code BasicErrorController} wrote {@code {timestamp, status, error, path}}; the
|
||||
* success advice then wrapped that map as {@code success:true}, and {@code path} echoed the raw
|
||||
* request URI back (re-review 2026-11-03, RR-02-02). This replaces that controller: the error
|
||||
* status is classified by the same table the exception handler uses, and the body is the failure
|
||||
* envelope, with nothing from the request in it.
|
||||
*
|
||||
* <p>Written straight to the response by {@link EnvelopeResponseWriter}, like the other refusals
|
||||
* made outside a converter. An error dispatch has no handler of its own to negotiate for, and a
|
||||
* browser's {@code Accept: text/html} would otherwise turn the error into a 406 about the error.
|
||||
*
|
||||
* <p>A request that names {@code /error} itself carries no error status; it is answered as an
|
||||
* unknown route. The security chain permits only the error dispatch, so such a request still needs
|
||||
* a token under the JWT chain.
|
||||
*
|
||||
* <p>An absolute route ({@link OwnsAbsoluteWebRoute}): the container forwards to {@code /error},
|
||||
* not to {@code /error} under the presentation prefix.
|
||||
*/
|
||||
@Controller
|
||||
@OwnsAbsoluteWebRoute
|
||||
@RequestMapping("${server.error.path:${error.path:/error}}")
|
||||
public final class EnvelopeErrorController implements ErrorController {
|
||||
|
||||
private final ObjectMapper objectMapper;
|
||||
|
||||
/**
|
||||
* Creates the controller.
|
||||
*
|
||||
* @param objectMapper Boot's mapper, which carries the envelope wire rules
|
||||
*/
|
||||
public EnvelopeErrorController(ObjectMapper objectMapper) {
|
||||
this.objectMapper = Objects.requireNonNull(objectMapper, "objectMapper");
|
||||
}
|
||||
|
||||
/** Writes the failure envelope for the status the error dispatch carries. */
|
||||
@RequestMapping
|
||||
public void error(HttpServletRequest request, HttpServletResponse response) throws IOException {
|
||||
if (response.isCommitted()) {
|
||||
return;
|
||||
}
|
||||
ApiErrorCode code =
|
||||
request.getAttribute(RequestDispatcher.ERROR_STATUS_CODE) instanceof Integer status
|
||||
? HttpStatusErrorCodes.forStatus(HttpStatusCode.valueOf(status))
|
||||
: OperationalError.ROUTE_NOT_FOUND;
|
||||
RetryAfterAdvisor.retryAfterSeconds(code)
|
||||
.ifPresent(seconds -> response.setHeader(ApiHeaders.RETRY_AFTER, String.valueOf(seconds)));
|
||||
EnvelopeResponseWriter.write(response, code, objectMapper);
|
||||
}
|
||||
}
|
||||
+1
-22
@@ -407,7 +407,7 @@ public class GlobalExceptionHandler extends ResponseEntityExceptionHandler {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
ApiErrorCode code = errorCodeFor(statusCode);
|
||||
ApiErrorCode code = HttpStatusErrorCodes.forStatus(statusCode);
|
||||
if (code.httpStatus() >= 500) {
|
||||
log.error("spring mvc failure classified as {}", code.code(), ex);
|
||||
spanErrorRecorder.recordException(ex, code.code());
|
||||
@@ -415,25 +415,4 @@ public class GlobalExceptionHandler extends ResponseEntityExceptionHandler {
|
||||
return new ResponseEntity<>(
|
||||
ErrorResponseFactory.body(code, null), headers, HttpStatusCode.valueOf(code.httpStatus()));
|
||||
}
|
||||
|
||||
private static ApiErrorCode errorCodeFor(HttpStatusCode status) {
|
||||
return switch (status.value()) {
|
||||
case 401 -> OperationalError.UNAUTHENTICATED;
|
||||
case 403 -> OperationalError.FORBIDDEN;
|
||||
case 404 -> OperationalError.ROUTE_NOT_FOUND;
|
||||
case 405 -> OperationalError.METHOD_NOT_ALLOWED;
|
||||
case 406 -> OperationalError.NOT_ACCEPTABLE;
|
||||
case 412 -> OperationalError.PRECONDITION_FAILED;
|
||||
case 413 -> OperationalError.PAYLOAD_TOO_LARGE;
|
||||
case 414 -> OperationalError.URI_TOO_LONG;
|
||||
case 415 -> OperationalError.UNSUPPORTED_MEDIA_TYPE;
|
||||
case 429 -> OperationalError.RATE_LIMIT_EXCEEDED;
|
||||
// Spring answers its async request timeout with 503: transient, not an internal fault.
|
||||
case 503 -> WebTransportError.REQUEST_TIMEOUT;
|
||||
default ->
|
||||
status.is4xxClientError()
|
||||
? OperationalError.BAD_PARAMETER
|
||||
: OperationalError.INTERNAL_ERROR;
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
package dev.caskeleton.adapter.inbound.web.error;
|
||||
|
||||
import dev.caskeleton.shared.error.ApiErrorCode;
|
||||
import dev.caskeleton.shared.error.OperationalError;
|
||||
import org.springframework.http.HttpStatusCode;
|
||||
|
||||
/**
|
||||
* The operational code a bare HTTP status stands for, when a status is all that is known.
|
||||
*
|
||||
* <p>Two places only have a status to classify: a Spring MVC exception this adapter has no
|
||||
* dedicated handler for, and the container's error dispatch for a failure raised outside the
|
||||
* handler. They share this one table, so the same status never reaches a client as two codes.
|
||||
*/
|
||||
final class HttpStatusErrorCodes {
|
||||
|
||||
private HttpStatusErrorCodes() {}
|
||||
|
||||
static ApiErrorCode forStatus(HttpStatusCode status) {
|
||||
return switch (status.value()) {
|
||||
case 401 -> OperationalError.UNAUTHENTICATED;
|
||||
case 403 -> OperationalError.FORBIDDEN;
|
||||
case 404 -> OperationalError.ROUTE_NOT_FOUND;
|
||||
case 405 -> OperationalError.METHOD_NOT_ALLOWED;
|
||||
case 406 -> OperationalError.NOT_ACCEPTABLE;
|
||||
case 412 -> OperationalError.PRECONDITION_FAILED;
|
||||
case 413 -> OperationalError.PAYLOAD_TOO_LARGE;
|
||||
case 414 -> OperationalError.URI_TOO_LONG;
|
||||
case 415 -> OperationalError.UNSUPPORTED_MEDIA_TYPE;
|
||||
case 429 -> OperationalError.RATE_LIMIT_EXCEEDED;
|
||||
// Spring answers its async request timeout with 503: transient, not an internal fault.
|
||||
case 503 -> WebTransportError.REQUEST_TIMEOUT;
|
||||
default ->
|
||||
status.is4xxClientError()
|
||||
? OperationalError.BAD_PARAMETER
|
||||
: OperationalError.INTERNAL_ERROR;
|
||||
};
|
||||
}
|
||||
}
|
||||
+27
-1
@@ -1,5 +1,7 @@
|
||||
package dev.caskeleton.adapter.inbound.web.mvc.autoconfigure;
|
||||
|
||||
import dev.caskeleton.adapter.inbound.web.envelope.EnvelopeWireFormatConfig;
|
||||
import dev.caskeleton.adapter.inbound.web.error.EnvelopeErrorController;
|
||||
import dev.caskeleton.adapter.inbound.web.error.GlobalExceptionHandler;
|
||||
import dev.caskeleton.adapter.inbound.web.mvc.filter.JsonBodyLimitFilter;
|
||||
import dev.caskeleton.adapter.inbound.web.mvc.filter.WebMvcRequestIdFilter;
|
||||
@@ -12,8 +14,12 @@ import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnWebApplication;
|
||||
import org.springframework.boot.context.properties.EnableConfigurationProperties;
|
||||
import org.springframework.boot.jackson.autoconfigure.JsonMapperBuilderCustomizer;
|
||||
import org.springframework.boot.webmvc.autoconfigure.error.ErrorMvcAutoConfiguration;
|
||||
import org.springframework.boot.webmvc.error.ErrorController;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import tools.jackson.core.StreamReadFeature;
|
||||
import tools.jackson.databind.ObjectMapper;
|
||||
import tools.jackson.databind.json.JsonMapper;
|
||||
|
||||
/**
|
||||
* Wires the web platform into a servlet application.
|
||||
@@ -30,8 +36,12 @@ import tools.jackson.core.StreamReadFeature;
|
||||
*
|
||||
* <p>Every bean is {@link ConditionalOnMissingBean}: the platform supplies the one that would
|
||||
* otherwise be missing rather than the one that must be used.
|
||||
*
|
||||
* <p>Before {@link ErrorMvcAutoConfiguration}, so the envelope error controller is the {@link
|
||||
* ErrorController} that configuration finds and Boot's {@code BasicErrorController} backs off. Its
|
||||
* error-page registration with the container stays: that is what forwards to {@code /error}.
|
||||
*/
|
||||
@AutoConfiguration
|
||||
@AutoConfiguration(before = ErrorMvcAutoConfiguration.class)
|
||||
@ConditionalOnWebApplication(type = ConditionalOnWebApplication.Type.SERVLET)
|
||||
@ConditionalOnProperty(
|
||||
prefix = "app.web.mvc",
|
||||
@@ -54,6 +64,22 @@ public class WebMvcPlatformAutoConfiguration {
|
||||
return new GlobalExceptionHandler(spanErrorRecorderProvider);
|
||||
}
|
||||
|
||||
/**
|
||||
* Answers the container's error dispatch — failures raised outside the MVC handler — with the
|
||||
* failure envelope instead of Boot's {@code {timestamp, status, error, path}} map.
|
||||
*
|
||||
* <p>Boot's mapper when there is one, which already carries the envelope wire rules; otherwise a
|
||||
* mapper with only those rules, as {@code EnvelopeBodyAdvice} does without Boot.
|
||||
*/
|
||||
@Bean
|
||||
@ConditionalOnMissingBean(ErrorController.class)
|
||||
public EnvelopeErrorController envelopeErrorController(
|
||||
ObjectProvider<ObjectMapper> objectMapper) {
|
||||
return new EnvelopeErrorController(
|
||||
objectMapper.getIfAvailable(
|
||||
() -> JsonMapper.builder().addModule(EnvelopeWireFormatConfig.module()).build()));
|
||||
}
|
||||
|
||||
/**
|
||||
* Assigns the correlation identifiers, validating anything a caller proposed. With a tracer in
|
||||
* the composition the trace id is the tracer's: the filter runs inside Boot's server observation
|
||||
|
||||
+21
-11
@@ -1,5 +1,6 @@
|
||||
package dev.caskeleton.adapter.inbound.web.observability;
|
||||
|
||||
import dev.caskeleton.application.idempotency.IdempotencyExecutor;
|
||||
import dev.caskeleton.shared.error.ApiErrorCode;
|
||||
import java.util.Map;
|
||||
import java.util.OptionalInt;
|
||||
@@ -13,17 +14,26 @@ public final class RetryAfterAdvisor {
|
||||
/** Per-code {@code Retry-After} values (seconds). */
|
||||
private static final Map<String, Integer> RETRY_AFTER_SECONDS =
|
||||
Map.of(
|
||||
"RATE_LIMIT_EXCEEDED", 1,
|
||||
// Below the default processing lease (30s): the resend arrives while the first attempt
|
||||
// can still finish, and an early resend costs one more 409. docs/web/http-contract.md.
|
||||
"IDEMPOTENT_IN_FLIGHT", 2,
|
||||
"AUTH_KID_UNKNOWN", 5,
|
||||
"AUTH_JWKS_UNAVAILABLE", 30,
|
||||
"DEPENDENCY_TIMEOUT", 2,
|
||||
"DEPENDENCY_CONNECT_FAILED", 2,
|
||||
"DEPENDENCY_DNS_FAILED", 5,
|
||||
"DEPENDENCY_5XX_SERVER", 2,
|
||||
"DEPENDENCY_CIRCUIT_OPEN", 10);
|
||||
"RATE_LIMIT_EXCEEDED",
|
||||
1,
|
||||
// The application's value, which the processing-lease check also reads: the resend
|
||||
// arrives while the first attempt can still finish. docs/web/http-contract.md §2.
|
||||
"IDEMPOTENT_IN_FLIGHT",
|
||||
Math.toIntExact(IdempotencyExecutor.IN_FLIGHT_RETRY_AFTER.toSeconds()),
|
||||
"AUTH_KID_UNKNOWN",
|
||||
5,
|
||||
"AUTH_JWKS_UNAVAILABLE",
|
||||
30,
|
||||
"DEPENDENCY_TIMEOUT",
|
||||
2,
|
||||
"DEPENDENCY_CONNECT_FAILED",
|
||||
2,
|
||||
"DEPENDENCY_DNS_FAILED",
|
||||
5,
|
||||
"DEPENDENCY_5XX_SERVER",
|
||||
2,
|
||||
"DEPENDENCY_CIRCUIT_OPEN",
|
||||
10);
|
||||
|
||||
private RetryAfterAdvisor() {}
|
||||
|
||||
|
||||
+91
@@ -0,0 +1,91 @@
|
||||
package dev.caskeleton.adapter.inbound.web.testkit.mvc;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import dev.caskeleton.adapter.inbound.web.envelope.EnvelopeWireFormatConfig;
|
||||
import dev.caskeleton.adapter.inbound.web.mvc.autoconfigure.WebAdapterAutoConfiguration;
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.time.Duration;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.boot.test.web.server.LocalServerPort;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.context.annotation.Import;
|
||||
|
||||
/**
|
||||
* The error dispatch under the production JWT security chain, over a real Tomcat.
|
||||
*
|
||||
* <p>The chain authorises every dispatch, the container's error dispatch to {@code /error}
|
||||
* included. {@code /error} is not a public path, so an anonymous request whose original failure was
|
||||
* a 400 used to come back as 401 — the client was told to authenticate when its request was
|
||||
* malformed (re-review 2026-11-03, RR-02-02). The error dispatch is now permitted; a request that
|
||||
* names {@code /error} directly is still a request and still needs a token.
|
||||
*
|
||||
* <p>The chain is {@code WebSecurityAutoConfiguration} itself. The JWT decoder is lazy, so the
|
||||
* unreachable issuer below is never contacted: no request here carries a token.
|
||||
*/
|
||||
@SpringBootTest(
|
||||
classes = JwtChainErrorDispatchIT.JwtChainApplication.class,
|
||||
webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
|
||||
properties = {
|
||||
"spring.main.banner-mode=off",
|
||||
"spring.mvc.problemdetails.enabled=false",
|
||||
"app.web.mvc.enabled=true",
|
||||
"app.security.auth-mode=jwt",
|
||||
"app.security.issuer-uri=https://issuer.invalid/realms/ca",
|
||||
"app.security.audience=ca-skeleton-api",
|
||||
"app.security.public-paths=/api/v1/fixtures/**",
|
||||
"app.cors.enabled=false"
|
||||
})
|
||||
class JwtChainErrorDispatchIT {
|
||||
|
||||
private final HttpClient client =
|
||||
HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
|
||||
|
||||
@LocalServerPort private int port;
|
||||
|
||||
@Test
|
||||
@DisplayName("JWT 체인에서 익명 요청이 핸들러 밖에서 400 으로 거부되면 401 이 아니라 400 실패 Envelope 로 나간다")
|
||||
void anAnonymousRequestRejectedOutsideTheHandlerKeepsItsOwnStatus() throws Exception {
|
||||
// when
|
||||
HttpResponse<String> rejected = get(WebContractAssertions.FIREWALL_REJECTED_PATH);
|
||||
|
||||
// then
|
||||
assertThat(rejected.statusCode()).isEqualTo(400);
|
||||
WebContractAssertions.assertFailureEnvelope(rejected, "BAD_PARAMETER");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("오류 디스패치만 허용한다 — /error 를 직접 부른 익명 요청은 여전히 401 이다")
|
||||
void aDirectRequestForTheErrorPathStillNeedsAToken() throws Exception {
|
||||
// when
|
||||
HttpResponse<String> direct = get("/error");
|
||||
|
||||
// then
|
||||
assertThat(direct.statusCode()).isEqualTo(401);
|
||||
}
|
||||
|
||||
private HttpResponse<String> get(String path) throws Exception {
|
||||
return client.send(
|
||||
HttpRequest.newBuilder(URI.create("http://localhost:" + port + path))
|
||||
.header("Accept", "application/json")
|
||||
.GET()
|
||||
.build(),
|
||||
HttpResponse.BodyHandlers.ofString());
|
||||
}
|
||||
|
||||
/**
|
||||
* The production web stack minus {@link WebAdapterAutoConfiguration}, whose request-logging
|
||||
* filter needs the composition root's pseudonymizer, plus that root's envelope wire format; the
|
||||
* security chain is the shipped one.
|
||||
*/
|
||||
@Configuration(proxyBeanMethods = false)
|
||||
@EnableAutoConfiguration(exclude = WebAdapterAutoConfiguration.class)
|
||||
@Import({ContractFixtureController.class, EnvelopeWireFormatConfig.class})
|
||||
static class JwtChainApplication {}
|
||||
}
|
||||
+26
@@ -23,6 +23,7 @@ import org.springframework.boot.webmvc.test.autoconfigure.AutoConfigureMockMvc;
|
||||
import org.springframework.boot.webmvc.test.autoconfigure.WebMvcTest;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.test.json.JsonCompareMode;
|
||||
import org.springframework.test.web.servlet.MockMvc;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
@@ -64,6 +65,12 @@ class EnvelopeBodyAdviceTest {
|
||||
return "{\"openapi\":\"3.1.0\"}".getBytes(StandardCharsets.UTF_8);
|
||||
}
|
||||
|
||||
/** A failure status with a body that is not an envelope. */
|
||||
@GetMapping("/probe/conflict")
|
||||
public ResponseEntity<Map<String, String>> conflict() {
|
||||
return ResponseEntity.status(409).body(Map.of("k", "v"));
|
||||
}
|
||||
|
||||
@GetMapping("/probe/bulk")
|
||||
public BulkEnvelope<Map<String, String>> bulk() {
|
||||
return BulkEnvelope.allOk(List.of(Map.of("k", "v")), new ResponseMeta("r", "t", "c"));
|
||||
@@ -149,6 +156,25 @@ class EnvelopeBodyAdviceTest {
|
||||
.andExpect(content().json("{\"openapi\":\"3.1.0\"}", JsonCompareMode.STRICT));
|
||||
}
|
||||
|
||||
/**
|
||||
* A 4xx/5xx body is never a success envelope. {@code success} is the discriminator clients branch
|
||||
* on (docs/web/http-contract.md §1), so wrapping an error-status body as {@code success:true}
|
||||
* tells a contract-following client the request worked — Boot's {@code /error} map arrived that
|
||||
* way (re-review 2026-11-03, RR-02-02). The body is left as the handler wrote it.
|
||||
*/
|
||||
@Test
|
||||
@DisplayName("4xx·5xx 응답 본문은 success:true Envelope 로 감싸지 않는다")
|
||||
void anErrorStatusBodyIsNotWrappedAsASuccess() throws Exception {
|
||||
// when
|
||||
var result = mvc.perform(get("/probe/conflict").accept(MediaType.APPLICATION_JSON));
|
||||
|
||||
// then
|
||||
result
|
||||
.andExpect(status().isConflict())
|
||||
.andExpect(jsonPath("$.success").doesNotExist())
|
||||
.andExpect(content().json("{\"k\":\"v\"}", JsonCompareMode.STRICT));
|
||||
}
|
||||
|
||||
@SpringBootConfiguration
|
||||
@EnableAutoConfiguration(exclude = SecurityAutoConfiguration.class)
|
||||
static class TestBootstrap {}
|
||||
|
||||
+21
@@ -3,6 +3,7 @@ package dev.caskeleton.adapter.inbound.web.error;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import dev.caskeleton.adapter.inbound.web.http.ApiHeaders;
|
||||
import dev.caskeleton.application.idempotency.IdempotencyCompletionUnknownException;
|
||||
import dev.caskeleton.application.idempotency.IdempotencyInFlightException;
|
||||
import dev.caskeleton.application.idempotency.IdempotencyRequestMismatchException;
|
||||
import dev.caskeleton.shared.error.Category;
|
||||
@@ -74,4 +75,24 @@ class IdempotencyExceptionMappingTest {
|
||||
false,
|
||||
ClientErrorMessages.of(Category.VALIDATION));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("complete 단계 결과 불명은 실행 전 거부(IDEMPOTENCY_UNAVAILABLE)와 다른 code 의 retryable 503 이다")
|
||||
void completionUnknownMapsToItsOwnRetryable503SoTheClientDoesNotReadItAsNotApplied() {
|
||||
// given
|
||||
IdempotencyCompletionUnknownException unknown = new IdempotencyCompletionUnknownException();
|
||||
|
||||
// when
|
||||
ResponseEntity<Envelope<Void>> response = handler.handleCarrier(unknown);
|
||||
|
||||
// then
|
||||
assertThat(response)
|
||||
.extracting(
|
||||
r -> r.getStatusCode().value(),
|
||||
r -> r.getHeaders().getFirst(ApiHeaders.RETRY_AFTER),
|
||||
r -> r.getBody().error().code(),
|
||||
r -> r.getBody().error().category(),
|
||||
r -> r.getBody().error().retryable())
|
||||
.containsExactly(503, null, "IDEMPOTENCY_COMPLETION_UNKNOWN", "TRANSIENT_DEPENDENCY", true);
|
||||
}
|
||||
}
|
||||
|
||||
+54
@@ -8,6 +8,8 @@ import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.time.Duration;
|
||||
import tools.jackson.databind.JsonNode;
|
||||
import tools.jackson.databind.json.JsonMapper;
|
||||
|
||||
/**
|
||||
* The status contract as socket-level assertions, written once and run against every container.
|
||||
@@ -22,6 +24,11 @@ import java.time.Duration;
|
||||
*/
|
||||
public final class WebContractAssertions {
|
||||
|
||||
/** A path the servlet firewall refuses before any handler runs ({@code ;} in the URI). */
|
||||
public static final String FIREWALL_REJECTED_PATH = "/api/v1/fixtures/f1;x=1";
|
||||
|
||||
private static final JsonMapper JSON = JsonMapper.builder().build();
|
||||
|
||||
private final HttpClient client =
|
||||
HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
|
||||
|
||||
@@ -122,6 +129,46 @@ public final class WebContractAssertions {
|
||||
assertThat(post("/api/v1/fixtures", "{\"name\":").statusCode()).isEqualTo(400);
|
||||
}
|
||||
|
||||
/**
|
||||
* An error raised outside the MVC handler is still a failure envelope.
|
||||
*
|
||||
* <p>The firewall rejects the request before any controller runs, and the container answers it
|
||||
* with an error dispatch to {@code /error}. Boot's default error controller wrote a map there,
|
||||
* which the success advice then wrapped as {@code success:true} with the raw request path inside
|
||||
* (re-review 2026-11-03, RR-02-02). MockMvc performs no error dispatch, so only a container can
|
||||
* show this.
|
||||
*/
|
||||
public void aRequestRejectedOutsideTheHandlerIsAFailureEnvelope()
|
||||
throws IOException, InterruptedException {
|
||||
HttpResponse<String> rejected = get(FIREWALL_REJECTED_PATH);
|
||||
|
||||
assertThat(rejected.statusCode()).isEqualTo(400);
|
||||
assertFailureEnvelope(rejected, "BAD_PARAMETER");
|
||||
}
|
||||
|
||||
/**
|
||||
* Asserts the failure envelope of {@code docs/web/http-contract.md} §1 on an error response.
|
||||
*
|
||||
* @param response a response the container wrote through its error dispatch
|
||||
* @param expectedCode the {@code error.code} the status maps to
|
||||
*/
|
||||
public static void assertFailureEnvelope(HttpResponse<String> response, String expectedCode) {
|
||||
JsonNode body = JSON.readTree(response.body());
|
||||
assertThat(body.path("success").isBoolean() && !body.path("success").booleanValue())
|
||||
.as(
|
||||
"a 4xx/5xx body that says success:true is read as a success by a client that follows"
|
||||
+ " the contract: %s",
|
||||
response.body())
|
||||
.isTrue();
|
||||
assertThat(body.path("error").path("code").asString()).isEqualTo(expectedCode);
|
||||
assertThat(body.has("data")).as("a failure envelope has no data key").isFalse();
|
||||
assertThat(response.body())
|
||||
.as("the error body must not echo the request path back")
|
||||
.doesNotContain(";x=1")
|
||||
.doesNotContain("\"path\"")
|
||||
.doesNotContain("timestamp");
|
||||
}
|
||||
|
||||
/** Runs every assertion, so a lane cannot silently cover a subset. */
|
||||
public void assertWholeContract() throws IOException, InterruptedException {
|
||||
createdCarriesALocationHeader();
|
||||
@@ -131,6 +178,7 @@ public final class WebContractAssertions {
|
||||
anUnknownMethodOnAKnownRouteIsRefused();
|
||||
anUnsupportedMediaTypeIsRefused();
|
||||
malformedJsonIsRefused();
|
||||
aRequestRejectedOutsideTheHandlerIsAFailureEnvelope();
|
||||
}
|
||||
|
||||
private HttpResponse<String> post(String path, String body)
|
||||
@@ -143,6 +191,12 @@ public final class WebContractAssertions {
|
||||
HttpResponse.BodyHandlers.ofString());
|
||||
}
|
||||
|
||||
private HttpResponse<String> get(String path) throws IOException, InterruptedException {
|
||||
return client.send(
|
||||
HttpRequest.newBuilder(uri(path)).header("Accept", "application/json").GET().build(),
|
||||
HttpResponse.BodyHandlers.ofString());
|
||||
}
|
||||
|
||||
private URI uri(String path) {
|
||||
return URI.create("http://localhost:" + port + path);
|
||||
}
|
||||
|
||||
+10
-1
@@ -1,5 +1,6 @@
|
||||
package dev.caskeleton.webtestkit;
|
||||
|
||||
import dev.caskeleton.adapter.inbound.web.envelope.EnvelopeWireFormatConfig;
|
||||
import dev.caskeleton.adapter.inbound.web.mvc.autoconfigure.WebAdapterAutoConfiguration;
|
||||
import dev.caskeleton.adapter.inbound.web.mvc.autoconfigure.WebSecurityAutoConfiguration;
|
||||
import dev.caskeleton.adapter.inbound.web.testkit.fault.FaultFixtureController;
|
||||
@@ -31,6 +32,10 @@ import org.springframework.security.web.SecurityFilterChain;
|
||||
* are the "sum of the leaf" the paragraph above keeps out. Excluded by class, so a rename breaks
|
||||
* compilation rather than the gate. The third root, {@code WebMethodSecurityAutoConfiguration}, is
|
||||
* left in: no fixture route runs a permissioned use case, so it decides nothing here.
|
||||
*
|
||||
* <p>{@link EnvelopeWireFormatConfig} is imported by name from the excluded web root: it is the
|
||||
* wire shape of every envelope this gate reads (a failure body has no {@code data} key), not a
|
||||
* capability, and without it the gate would certify {@code "data": null} that no deployment sends.
|
||||
*/
|
||||
// Outside `dev.caskeleton.adapter.inbound.web`, which the composition root component-scans.
|
||||
// A fixture application under that root is loaded into the real deployment: its permit-all
|
||||
@@ -40,7 +45,11 @@ import org.springframework.security.web.SecurityFilterChain;
|
||||
@Configuration(proxyBeanMethods = false)
|
||||
@EnableAutoConfiguration(
|
||||
exclude = {WebAdapterAutoConfiguration.class, WebSecurityAutoConfiguration.class})
|
||||
@Import({ContractFixtureController.class, FaultFixtureController.class})
|
||||
@Import({
|
||||
ContractFixtureController.class,
|
||||
FaultFixtureController.class,
|
||||
EnvelopeWireFormatConfig.class
|
||||
})
|
||||
public class ContractFixtureApplication {
|
||||
|
||||
/**
|
||||
|
||||
@@ -128,12 +128,15 @@ Focused tests drive the semantic adapters against `InMemorySemanticRedisCommands
|
||||
served through `StubRedisRuntimeClient`. R1/R2 promotion requires a separate real Redis service lane; it may never
|
||||
be silently skipped when selected.
|
||||
|
||||
Two real-server lanes. `redisContainerContractTest` (integration category, in `integrationCheck` and
|
||||
this leaf's `mainCheck`) starts its own Redis with Testcontainers — the `redis` image of
|
||||
`config/testing/container-images.properties`, loading `infra/redis-sdk/acl/all-accounts.acl` — and
|
||||
runs `LiveRedisCacheRegionContractTest` and `LiveRedisDistributedLeaseContractTest` (the
|
||||
`application-core` port contracts the in-memory fakes also pass), `LiveRedisSemanticPortsTest` and
|
||||
`LiveRedisSemanticScriptAclProbeTest`; a skip fails it (review 2026-11-01 05-16). `RedisTestContainer`
|
||||
Two real-server lanes. `redisContainerContractTest` (integration category, in `integrationCheck` only —
|
||||
not in this leaf's `mainCheck` too, which ran it twice per push to main) starts its own Redis with Testcontainers — the `redis` image of
|
||||
`config/testing/container-images.properties`, loading `infra/redis-sdk/acl/lane-fixture-accounts.acl` — and
|
||||
runs `LiveRedisCacheRegionContractTest`, `LiveRedisDistributedLeaseContractTest` and
|
||||
`LiveRedisIdempotencyStoreContractTest` (the `application-core` port contracts the in-memory fakes
|
||||
also pass), `LiveRedisSemanticPortsTest` and `LiveRedisSemanticScriptAclProbeTest`; a skip fails it
|
||||
(review 2026-11-01 05-16). The idempotency contract was on the explicit-only topology lane alone
|
||||
until review 2026-11-03 RR-04-05, so the Lua state machine's stale-handle, renew-refusal and
|
||||
retry cases reached a real server in no gate. `RedisTestContainer`
|
||||
starts the server only when that lane names the ACL file.
|
||||
|
||||
`redisTopologyTest` is the external-topology lane, declared through `ca.redis-topology-lane`. This
|
||||
@@ -177,6 +180,15 @@ deployment still gets exactly one client and one event loop. The `SCRIPT` lane i
|
||||
the reason it exists: `SCRIPT LOAD` and `EVALSHA` belong to the advanced account, so the account
|
||||
that reads a cache entry cannot execute a script.
|
||||
|
||||
The cache's conditional writes run on the same account but on a lane of their own,
|
||||
`CacheScriptLane.CACHE_SCRIPT` (`app.capabilities.cache.maximum-script-connections`, contributed by
|
||||
`RedisCacheConfig` as a `RedisLaneContribution`). Do not move them back to `SCRIPT`: that lane is the
|
||||
idempotency, rate-limit and lease budget, and a cache under load must not take it (review 2026-11-03
|
||||
README §4, RR-04-02). Each lane's wait, refusals and reconnects are meters of `redis-runtime`
|
||||
(`RedisLaneMetrics`, `redis.lane.*`, tag `lane`), so whether the cache stays off `SCRIPT` is a query,
|
||||
not an assumption. The `DEGRADED` health order ships with `redis-runtime`, which reports the
|
||||
status (`config/module-defaults/redis-runtime.yml`, RR-04-10).
|
||||
|
||||
A Cluster transaction runs on one node, so the SDK's `RedisTransactionRunner` derives a routing key
|
||||
from the watched keys (or an explicit `RedisSlotTag`) and borrows `RedisSdkLane.TRANSACTION`, which
|
||||
is pinned to its key's owner, on the node that owns that slot. Routed leases are never pooled: a pooled connection is pinned to the previous caller's
|
||||
|
||||
@@ -97,7 +97,10 @@ Connection lane limits and command queue limits are different contracts. In part
|
||||
|
||||
- `app.redis.capacity.maximum-in-flight-commands` bounds the per-connection command queue.
|
||||
- `app.redis.capacity.maximum-script-connections` bounds physical SCRIPT lane connections
|
||||
(default 8).
|
||||
(default 8). Idempotency, rate limiting and leases share them.
|
||||
- `app.capabilities.cache.maximum-script-connections` (`APP_CACHE_REDIS_MAXIMUM_SCRIPT_CONNECTIONS`)
|
||||
bounds the cache's own script lane (`CacheScriptLane`, default 4), contributed only when the cache
|
||||
binding is `redis`. See "Why the cache has its own script lane" below.
|
||||
- `app.redis.lifecycle.acquire-timeout` (default 2s) is how long a borrow waits for a returned
|
||||
connection when its lane is full.
|
||||
|
||||
@@ -153,7 +156,7 @@ Important contracts:
|
||||
Every lookup returns a `CacheWriteCondition` naming the state it saw — `absent`, `vacant:<digest>`
|
||||
(an invalidation marker, an entry of an older region generation, an expired entry) or
|
||||
`present:<digest>`. A record that carries the condition applies only while the key is still in that
|
||||
state (registered program, SCRIPT lane). This is what rejects a source load that began before an
|
||||
state (registered program, cache script lane). This is what rejects a source load that began before an
|
||||
invalidation and finished after it.
|
||||
|
||||
A hit also returns an opaque SHA-1 observation token over the stored bytes. SHA-1 here is **not a
|
||||
@@ -181,10 +184,43 @@ example one written by `UPSERT` as `rev-1`), another schema version, a generatio
|
||||
writer knows — is `NOT_RECORDED_CONDITION`.
|
||||
|
||||
Both conditional intents, and any record carrying a write condition, run registered scripts, so
|
||||
they borrow the SCRIPT lane (the advanced account when one is configured, the application account
|
||||
otherwise). The application account is not granted `EVALSHA`; on the REGULAR lane every conditional
|
||||
write came back `INDETERMINATE`. Unconditioned `UPSERT` and `ONLY_IF_ABSENT` stay on the REGULAR
|
||||
lane.
|
||||
they run on the scripting account (the advanced account when one is configured, the application
|
||||
account otherwise), on the cache's own script lane. The application account is not granted
|
||||
`EVALSHA`; on the REGULAR lane every conditional write came back `INDETERMINATE`. Unconditioned
|
||||
`UPSERT` and `ONLY_IF_ABSENT` stay on the REGULAR lane.
|
||||
|
||||
### Why the cache has its own script lane
|
||||
|
||||
Recording with the condition a lookup returned is the normal way to use this port, so nearly every
|
||||
cache fill is a script. On the runtime's SCRIPT lane those fills shared eight connections per pod
|
||||
with idempotency claims, rate-limit evaluations and lease calls, and a timed-out lease is discarded
|
||||
rather than pooled, so each timeout also cost a reconnect. A slow Redis or a burst of misses after a
|
||||
region invalidation filled the lane with cache work; an idempotency claim then waited out its
|
||||
timeout and answered `UNAVAILABLE`, and a rate-limit call fell to its failure policy. The cache is a
|
||||
speed feature that may degrade; idempotency, rate limiting and leases are correctness features that
|
||||
must not (review 2026-11-03 README §4, RR-04-02).
|
||||
|
||||
So `CacheScriptLane.CACHE_SCRIPT` is a lane of its own — the same scripting account, its own
|
||||
connections, its own ceiling (`app.capabilities.cache.maximum-script-connections`, default 4). A cache
|
||||
that runs out of them degrades (`DEGRADED_UNAVAILABLE`, nothing sent) without touching the SCRIPT
|
||||
lane. Alternatives not taken: a per-capability permit cap inside the shared SCRIPT lane (still one
|
||||
pool of connections and one reconnect cost), and running the cache programs on the REGULAR lane
|
||||
(grants the application account `EVALSHA`, which the account split exists to withhold).
|
||||
|
||||
Whether the split holds under load is visible per lane. With a `MeterRegistry` in the context,
|
||||
`redis-runtime` registers these for every lane it pools — `REGULAR`, `SCRIPT`, `CACHE_SCRIPT` and the
|
||||
typed SDK's — tagged `lane=<name>` (`RedisLaneMetrics`; rows in `docs/registries/metrics.yaml`):
|
||||
|
||||
| Meter | Type | Read it as |
|
||||
| --- | --- | --- |
|
||||
| `redis.lane.acquire.wait.seconds` (`outcome` = `admitted`/`refused`) | timer, histogram | how long a borrow queued for a connection; a growing admitted p99 comes before refusals |
|
||||
| `redis.lane.acquire.refused.total` (`reason` = `saturated`/`closing`/`interrupted`/`connect_failed`) | counter | a borrow that got no connection, nothing sent; `saturated` on `SCRIPT` is the starvation this lane exists to prevent |
|
||||
| `redis.lane.connections.opened.total` | counter | new connections; after warm-up every one is a reconnect |
|
||||
| `redis.lane.connections.discarded.total` (`reason` = `invalidated`/`broken`) | counter | connections closed instead of pooled — `invalidated` is a timed-out or failed call (the lease is not reused), and is what the reconnects replace |
|
||||
| `redis.lane.connections.active`, `redis.lane.connections.pending` | gauge | connections lent out now; borrows waiting now |
|
||||
|
||||
Every counter series exists from startup at zero, so an alert on "refusals above zero" sees the first
|
||||
one. A lane refusal is also logged, per call, by `RedisSemanticFailureLogger`.
|
||||
|
||||
### Key invalidation
|
||||
|
||||
@@ -300,10 +336,21 @@ Useful focused checks:
|
||||
|
||||
`redisContainerContractTest` runs the cache-region and lease port contracts (the ones the in-memory
|
||||
fakes pass) and the live semantic-port tests against a Redis it starts itself, under the standalone
|
||||
topology's ACL file. It is in `integrationCheck` and `mainCheck`; the image is
|
||||
topology's ACL file. It is in `integrationCheck` (integration-main.yml's `integration-check` job), and
|
||||
only there: listing it in `mainCheck` as well ran it twice on every push to main. The image is
|
||||
`config/testing/container-images.properties` `redis`.
|
||||
|
||||
Live Redis/topology qualification remains separate from fast unit tests. A release claim for a
|
||||
specific Redis topology or ACL layout requires the corresponding live qualification lane; the
|
||||
in-memory gateway is for deterministic contract tests, not evidence that an external Redis
|
||||
deployment accepts the same network/TLS/ACL configuration.
|
||||
|
||||
## 아직 내보내지 않는 지표
|
||||
|
||||
`redis.capability.*` 여섯 개(operations.total, duration.seconds, admission.rejected.total,
|
||||
inflight.total, readiness.total, lifecycle.drain.total)는 redis-production-capability 설계(Task 16)에
|
||||
있었지만 어떤 코드도 내보내지 않았다. 레지스트리(`docs/registries/metrics.yaml`)와 incident runbook이 있는
|
||||
것처럼 적고 있어서, 2026-11-03 재리뷰 수정에서 레지스트리에서 빼고 runbook을 지금 내보내는
|
||||
`redis.lane.*`(redis-runtime, lane별 대기·거절·연결) 기준으로 고쳤다. capability 단위 지표가 필요해지면
|
||||
operation 결과를 아는 semantic adapter(이 leaf)에서 내보내고, 그때 레지스트리에 다시 넣는다.
|
||||
|
||||
|
||||
@@ -48,7 +48,8 @@ dependencies {
|
||||
// command surface the catalog exists to prevent. Zero imports.
|
||||
// io.micrometer:micrometer-core — observation leaves this leaf as RedisObservation through a
|
||||
// Consumer sink; binding it to a meter registry belongs to the composition root, not here.
|
||||
// Zero imports.
|
||||
// Zero imports. The connection lanes' meters, CACHE_SCRIPT's included, are redis-runtime's
|
||||
// (RedisLaneMetrics): the lanes are pooled there, so that is where they are counted.
|
||||
}
|
||||
|
||||
// Redis topology qualification is opt-in and owned by `ca.redis-topology-lane`: mode/property
|
||||
@@ -69,8 +70,12 @@ redisTopologyLane {
|
||||
// adapters' port contracts reached a real server in no stage, and the fake and the adapter could hide
|
||||
// the same defect (05-02, 05-03). The image is config/testing/container-images.properties' `redis`,
|
||||
// and the ACL file is infra/redis-sdk's own, so the accounts and their rights are the standalone
|
||||
// topology's. Integration category: integrationCheck runs it, and so does this leaf's mainCheck, like
|
||||
// the HTTP client's Toxiproxy lane. Sentinel, cluster and TLS remain the topology lane's.
|
||||
// topology's. Integration category, so integrationCheck runs it — integration-main.yml's
|
||||
// `integration-check` job, on every push to main and nightly. It is deliberately not in this leaf's
|
||||
// mainCheck as well: `main-check` and `integration-check` are two jobs of the same workflow run, so a
|
||||
// lane in both started a second Redis container and ran every live contract twice per push without
|
||||
// covering anything the first run did not (found while fixing review 2026-11-03). Sentinel, cluster
|
||||
// and TLS remain the topology lane's.
|
||||
def redisAclDirectory = rootProject.projectDir.parentFile.toPath().resolve('infra/redis-sdk/acl').toFile()
|
||||
strictTestLanes {
|
||||
lane('redisContainerContractTest') {
|
||||
@@ -78,15 +83,16 @@ strictTestLanes {
|
||||
description = 'Runs the Redis semantic adapters\' live contracts against a Testcontainers Redis on the pinned image.'
|
||||
requires 'dev.caskeleton.adapter.outbound.cache.redis.cache.LiveRedisCacheRegionContractTest',
|
||||
'dev.caskeleton.adapter.outbound.cache.redis.lease.LiveRedisDistributedLeaseContractTest',
|
||||
// The fake carries a Java copy of the idempotency Lua state machine; only a real
|
||||
// server proves the two agree (05-02 was the fake copying the Lua's bug). It ran on
|
||||
// the explicit-only topology lane alone, so no gate ever ran it (RR-04-05).
|
||||
'dev.caskeleton.adapter.outbound.cache.redis.idempotency.LiveRedisIdempotencyStoreContractTest',
|
||||
'dev.caskeleton.adapter.outbound.cache.redis.LiveRedisSemanticPortsTest',
|
||||
'dev.caskeleton.adapter.outbound.cache.redis.capability.LiveRedisSemanticScriptAclProbeTest'
|
||||
rejectSkipped = true
|
||||
systemProperty 'redis.container.acl-file', new File(redisAclDirectory, 'all-accounts.acl').absolutePath
|
||||
systemProperty 'redis.container.acl-file', new File(redisAclDirectory, 'lane-fixture-accounts.acl').absolutePath
|
||||
inputDirectory redisAclDirectory
|
||||
shouldRunAfter 'test'
|
||||
}
|
||||
}
|
||||
|
||||
tasks.named('mainCheck') {
|
||||
dependsOn 'redisContainerContractTest'
|
||||
}
|
||||
|
||||
@@ -50,7 +50,7 @@ io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,testAnnota
|
||||
io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,testAnnotationProcessor
|
||||
io.lettuce:lettuce-core:6.8.2.RELEASE=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-commons:1.16.7=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-core:1.16.7=testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-core:1.16.7=runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-jakarta9:1.16.7=testCompileClasspath,testRuntimeClasspath
|
||||
io.micrometer:micrometer-observation:1.16.7=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
io.netty:netty-buffer:4.2.17.Final=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
@@ -103,7 +103,7 @@ org.codehaus.plexus:plexus-container-default:2.1.0=checkstyle
|
||||
org.codehaus.plexus:plexus-utils:3.3.0=checkstyle
|
||||
org.dom4j:dom4j:2.2.0=spotbugs
|
||||
org.hamcrest:hamcrest:3.0=testCompileClasspath,testRuntimeClasspath
|
||||
org.hdrhistogram:HdrHistogram:2.2.2=testRuntimeClasspath
|
||||
org.hdrhistogram:HdrHistogram:2.2.2=runtimeClasspath,testRuntimeClasspath
|
||||
org.javassist:javassist:3.28.0-GA=checkstyle
|
||||
org.jetbrains:annotations:17.0.0=testCompileClasspath,testRuntimeClasspath
|
||||
org.jspecify:jspecify:1.0.1=annotationProcessor,checkstyle,compileClasspath,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
|
||||
@@ -116,7 +116,7 @@ org.junit.platform:junit-platform-engine:6.0.3=testRuntimeClasspath
|
||||
org.junit.platform:junit-platform-launcher:6.0.3=testRuntimeClasspath
|
||||
org.junit:junit-bom:6.0.3=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit:junit-bom:6.1.0=spotbugs
|
||||
org.latencyutils:LatencyUtils:2.0.3=testRuntimeClasspath
|
||||
org.latencyutils:LatencyUtils:2.0.3=runtimeClasspath,testRuntimeClasspath
|
||||
org.mockito:mockito-core:5.20.0=mockitoAgent,testCompileClasspath,testRuntimeClasspath
|
||||
org.mockito:mockito-junit-jupiter:5.20.0=testCompileClasspath,testRuntimeClasspath
|
||||
org.objenesis:objenesis:3.3=testRuntimeClasspath
|
||||
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
package dev.caskeleton.adapter.outbound.cache.redis.cache;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisAccount;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisLane;
|
||||
|
||||
/**
|
||||
* The cache's own script lane, so a cache's load cannot starve a correctness capability (review
|
||||
* 2026-11-03 README §4, RR-04-02).
|
||||
*
|
||||
* <p>Every conditional cache write is a registered program, and a registered program needs the
|
||||
* scripting account. On the runtime's SCRIPT lane those writes shared eight connections with
|
||||
* idempotency claims, rate-limit evaluations and lease calls — and the normal way to use the cache
|
||||
* (record with the condition the lookup returned) makes every cache fill a script. A slow Redis or
|
||||
* a burst of misses after a region invalidation then held those connections with cache work while
|
||||
* an idempotency claim waited out its timeout and failed. The cache is a speed feature and may
|
||||
* degrade; idempotency and leases are correctness features and must not. They get separate
|
||||
* connection budgets.
|
||||
*
|
||||
* <p>Same account as the SCRIPT lane: the grant is the same, only the connections and their ceiling
|
||||
* ({@code app.capabilities.cache.maximum-script-connections}) are the cache's. Alternatives that
|
||||
* were not taken: a per-capability permit cap inside the shared SCRIPT lane would still share the
|
||||
* connections and their reconnect cost after a discarded lease; running cache programs on the
|
||||
* REGULAR lane would grant the application account {@code EVALSHA}, which the account split exists
|
||||
* to withhold.
|
||||
*/
|
||||
public enum CacheScriptLane implements RedisLane {
|
||||
/** The cache region's conditional and revision-ordered writes. */
|
||||
CACHE_SCRIPT;
|
||||
|
||||
@Override
|
||||
public RedisAccount account() {
|
||||
return RedisAccount.ADVANCED;
|
||||
}
|
||||
}
|
||||
+5
-3
@@ -68,8 +68,10 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
|
||||
private final SemanticRedisCall call;
|
||||
|
||||
/**
|
||||
* The conditional writes, which are registered programs and so run on the SCRIPT lane: the
|
||||
* application account the REGULAR lane authenticates as is not granted {@code EVALSHA}.
|
||||
* The conditional writes, which are registered programs and so run on the scripting account: the
|
||||
* application account the REGULAR lane authenticates as is not granted {@code EVALSHA}. They run
|
||||
* on the cache's own lane ({@link CacheScriptLane}), not the runtime's SCRIPT lane, so a cache
|
||||
* under load cannot hold the connections idempotency, rate limiting and leases need (RR-04-02).
|
||||
*/
|
||||
private final SemanticRedisCall scriptCall;
|
||||
|
||||
@@ -133,7 +135,7 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
|
||||
Objects.requireNonNull(commandTimeout, "command timeout must be non-null"),
|
||||
"cache");
|
||||
this.scriptCall =
|
||||
new SemanticRedisCall(owner, RedisConnectionKind.SCRIPT, commandTimeout, "cache");
|
||||
new SemanticRedisCall(owner, CacheScriptLane.CACHE_SCRIPT, commandTimeout, "cache");
|
||||
this.keys = Objects.requireNonNull(keys, "keys must be non-null");
|
||||
this.keyDigest = Objects.requireNonNull(keyDigest, "key digest must be non-null");
|
||||
this.encoder = Objects.requireNonNull(encoder, "encoder must be non-null");
|
||||
|
||||
+22
@@ -1,6 +1,8 @@
|
||||
package dev.caskeleton.adapter.outbound.cache.redis.capability;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.cache.CacheScriptLane;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.cache.RedisCacheRegionAdapter;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisLaneContribution;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisRuntimeSettings;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisRuntimeOwner;
|
||||
import dev.caskeleton.adapter.outbound.support.CapabilitySelectors;
|
||||
@@ -56,4 +58,24 @@ class RedisCacheConfig {
|
||||
cache.getNegativeTtl(),
|
||||
cache.getCommandTimeout());
|
||||
}
|
||||
|
||||
/**
|
||||
* The cache's own script lane and its connection ceiling.
|
||||
*
|
||||
* <p>The runtime pools a contributed lane like its own SCRIPT lane, on the same scripting
|
||||
* account, but with connections and a ceiling of its own: the cache's conditional writes no
|
||||
* longer compete with idempotency claims, rate-limit evaluations and lease calls for the SCRIPT
|
||||
* lane (review 2026-11-03 README §4, RR-04-02).
|
||||
*
|
||||
* @param capabilities the capability settings
|
||||
* @return the lane contribution
|
||||
*/
|
||||
@Bean
|
||||
@ConditionalOnProperty(name = CapabilitySelectors.CACHE_DEFAULT_BINDING, havingValue = "redis")
|
||||
RedisLaneContribution redisCacheScriptLane(RedisCapabilitySettings capabilities) {
|
||||
RedisCapabilitySettings.Cache cache = capabilities.getCache();
|
||||
cache.validate();
|
||||
return new RedisLaneContribution(
|
||||
CacheScriptLane.CACHE_SCRIPT, cache.getMaximumScriptConnections());
|
||||
}
|
||||
}
|
||||
|
||||
+22
@@ -86,6 +86,7 @@ public class RedisCapabilitySettings {
|
||||
private Duration positiveHardTtl = Duration.ofMinutes(5);
|
||||
private Duration negativeTtl = Duration.ofSeconds(10);
|
||||
private Duration minimumHardTtl = Duration.ofSeconds(1);
|
||||
private int maximumScriptConnections = 4;
|
||||
|
||||
/**
|
||||
* Reports whether the default binding selected Redis.
|
||||
@@ -119,6 +120,10 @@ public class RedisCapabilitySettings {
|
||||
if (keyVersion < 1) {
|
||||
throw new CapabilityStartupValidationException("the cache key version must be positive");
|
||||
}
|
||||
if (maximumScriptConnections < 1) {
|
||||
throw new CapabilityStartupValidationException(
|
||||
"the cache script lane needs at least one connection");
|
||||
}
|
||||
}
|
||||
|
||||
public Map<String, String> getBindings() {
|
||||
@@ -192,6 +197,23 @@ public class RedisCapabilitySettings {
|
||||
public void setMinimumHardTtl(Duration minimumHardTtl) {
|
||||
this.minimumHardTtl = minimumHardTtl;
|
||||
}
|
||||
|
||||
/**
|
||||
* The ceiling on the cache's own script-lane connections.
|
||||
*
|
||||
* <p>Separate from {@code app.redis.capacity.maximum-script-connections}, which idempotency,
|
||||
* rate limiting and leases share: a cache is a speed feature and may run out of connections and
|
||||
* degrade; it must not take them from the correctness features (review 2026-11-03 README §4).
|
||||
*
|
||||
* @return the ceiling, at least 1
|
||||
*/
|
||||
public int getMaximumScriptConnections() {
|
||||
return maximumScriptConnections;
|
||||
}
|
||||
|
||||
public void setMaximumScriptConnections(int maximumScriptConnections) {
|
||||
this.maximumScriptConnections = maximumScriptConnections;
|
||||
}
|
||||
}
|
||||
|
||||
/** Edge rate-limit provider composition. */
|
||||
|
||||
+44
-9
@@ -28,11 +28,12 @@ import org.springframework.beans.factory.SmartInitializingSingleton;
|
||||
* {@code {1}} only when the value it wrote reads back.
|
||||
*
|
||||
* <p>A cache-only deployment is probed too: its conditional writes and revision-ordered writes are
|
||||
* registered programs on the same lane, and without the permission every one of them would come
|
||||
* registered programs on the same account, and without the permission every one of them would come
|
||||
* back degraded while plain reads looked healthy. Its probe runs only what the cache programs call
|
||||
* ({@code GET}, {@code SET}), and a Redis it cannot reach at startup is logged, not fatal — a cache
|
||||
* outage degrades, it never stops a pod from starting. A server that answers and refuses the
|
||||
* scripts is a configuration error and fails startup as it does for the correctness roles.
|
||||
* ({@code GET}, {@code SET}). Only a server that answers and refuses the scripts — an ACL refusal,
|
||||
* {@code NOSCRIPT}, or a reply other than the value the probe wrote — is a configuration error and
|
||||
* fails startup as it does for the correctness roles. Every other failure is a cache outage and is
|
||||
* logged, not fatal: a cache outage degrades, it never stops a pod from starting.
|
||||
*/
|
||||
final class RedisSemanticScriptAclProbe implements SmartInitializingSingleton {
|
||||
|
||||
@@ -89,7 +90,7 @@ final class RedisSemanticScriptAclProbe implements SmartInitializingSingleton {
|
||||
try {
|
||||
probe(CACHE_PROBE_SCRIPT);
|
||||
} catch (CapabilityStartupValidationException failure) {
|
||||
if (!unreachable(failure.getCause())) {
|
||||
if (refusedByTheServer(failure.getCause())) {
|
||||
throw new CapabilityStartupValidationException(
|
||||
"Redis cache is selected but its account cannot run the cache's conditional-write"
|
||||
+ " scripts (SCRIPT LOAD/EVALSHA, GET, SET); every conditional record would"
|
||||
@@ -98,15 +99,49 @@ final class RedisSemanticScriptAclProbe implements SmartInitializingSingleton {
|
||||
failure.getCause());
|
||||
}
|
||||
LOG.warn(
|
||||
"Redis cache is selected but Redis was unreachable at startup; the cache script"
|
||||
+ " permission is unverified and the cache runs degraded until Redis answers",
|
||||
"Redis cache is selected but the startup script probe could not complete; the cache"
|
||||
+ " script permission is unverified and the cache runs degraded until Redis answers",
|
||||
failure.getCause());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static boolean unreachable(Throwable cause) {
|
||||
return cause instanceof RedisLaneUnavailableException || cause instanceof TimeoutException;
|
||||
/**
|
||||
* Whether the cache probe failed because the server refused the cache's scripts.
|
||||
*
|
||||
* <p>"Refused" is a statement about the account, and only three answers make it: an ACL refusal
|
||||
* ({@code NOPERM}, or a script's inner command denied), {@code NOSCRIPT} for the program the same
|
||||
* connection had just loaded, and a reply other than the value the probe wrote. The check used to
|
||||
* run the other way — only a lane refusal or a timeout counted as "unreachable" and everything
|
||||
* else was a configuration error — so a connection that dropped mid-probe, a restarting Redis
|
||||
* answering {@code LOADING}, or a busy one answering {@code BUSY} stopped a cache-only pod from
|
||||
* starting, against this leaf's promise that a cache outage never does (review 2026-11-03
|
||||
* RR-04-07).
|
||||
*
|
||||
* @param cause the probe's failure
|
||||
* @return {@code true} when the account, not the server's availability, is at fault
|
||||
*/
|
||||
private static boolean refusedByTheServer(Throwable cause) {
|
||||
for (Throwable current = cause;
|
||||
current != null;
|
||||
current = current.getCause() == current ? null : current.getCause()) {
|
||||
if (current instanceof CapabilityStartupValidationException) {
|
||||
// The probe ran and read back something other than what it wrote.
|
||||
return true;
|
||||
}
|
||||
if (current instanceof RedisLaneUnavailableException || current instanceof TimeoutException) {
|
||||
return false;
|
||||
}
|
||||
String message = current.getMessage();
|
||||
if (message != null
|
||||
&& (message.startsWith("NOPERM")
|
||||
|| message.startsWith("NOSCRIPT")
|
||||
|| message.contains("no permissions")
|
||||
|| message.contains("ACL failure"))) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
private void probe(byte[] script) {
|
||||
|
||||
+10
@@ -141,6 +141,12 @@ public final class IdempotencyScripts {
|
||||
* it started from and the lease it asked for. A retry from that revision with the same lease is
|
||||
* the same renewal; with a different lease it is a different request, because confirming it would
|
||||
* leave the caller believing it holds the record for a lease the record does not carry.
|
||||
*
|
||||
* <p>The key's own TTL is stretched to cover the new lease. Claim sets it to the replay TTL,
|
||||
* which may be shorter than a lease renewed past it; the key then expired while the holder was
|
||||
* still executing, and the next request got a fresh claim and ran the side effect again (review
|
||||
* 2026-11-03 RR-04-08). The TTL is only ever lengthened here — completion sets the replay TTL
|
||||
* again.
|
||||
*/
|
||||
private static final String RENEW =
|
||||
"""
|
||||
@@ -177,6 +183,10 @@ public final class IdempotencyScripts {
|
||||
redis.call('HSET', KEYS[1],
|
||||
'leaseUntil', nowMillis + tonumber(ARGV[4]), 'rev', rev + 1,
|
||||
'renewedFromRev', ARGV[2], 'renewedLease', ARGV[4])
|
||||
local ttl = redis.call('PTTL', KEYS[1])
|
||||
if ttl >= 0 and ttl < tonumber(ARGV[4]) then
|
||||
redis.call('PEXPIRE', KEYS[1], ARGV[4])
|
||||
end
|
||||
return {'APPLIED', attempt, rev + 1, owner, '', ''}
|
||||
""";
|
||||
|
||||
|
||||
+6
-9
@@ -39,6 +39,10 @@ app:
|
||||
# Floor on the hard TTL. Below it, entries expire faster than the round trip that wrote them:
|
||||
# every read misses and every miss writes.
|
||||
minimum-hard-ttl: ${APP_CACHE_REDIS_MINIMUM_HARD_TTL:1s}
|
||||
# The cache's own script-lane connections, apart from the SCRIPT lane that idempotency, rate
|
||||
# limiting and leases share (app.redis.capacity.maximum-script-connections). A cache under
|
||||
# load runs out of these and degrades; it cannot take the correctness features' connections.
|
||||
maximum-script-connections: ${APP_CACHE_REDIS_MAXIMUM_SCRIPT_CONNECTIONS:4}
|
||||
idempotency:
|
||||
# Redis store tuning only; the provider selector and lifecycle policy stay with the
|
||||
# application's idempotency executor.
|
||||
@@ -75,15 +79,8 @@ app:
|
||||
maximum-cost: ${APP_RATE_LIMIT_MAXIMUM_COST:10}
|
||||
cleanup-grace: ${APP_RATE_LIMIT_CLEANUP_GRACE:5s}
|
||||
|
||||
# redis-runtime 의 redisOptional 은 Redis 에 닿지 못하면 DEGRADED 를 보고한다. Boot 기본 순서
|
||||
# (DOWN, OUT_OF_SERVICE, UP, UNKNOWN)에 없는 상태는 DOWN 보다 앞에 정렬돼, 다른 contributor 가 DOWN
|
||||
# 이어도 전체 health 가 HTTP 200 DEGRADED 로 보였다(리뷰 05-23e). DEGRADED 를 DOWN 뒤, UP 앞에 둔다.
|
||||
# 이 키를 정하는 module-defaults 는 이 파일 하나뿐이다. 서비스가 순서를 바꾸면 DEGRADED 를 넣어야 한다.
|
||||
management:
|
||||
endpoint:
|
||||
health:
|
||||
status:
|
||||
order: DOWN,OUT_OF_SERVICE,DEGRADED,UP,UNKNOWN
|
||||
# DEGRADED 상태 순서(management.endpoint.health.status.order)는 그 상태를 내는 redis-runtime 이
|
||||
# 싣는다(config/module-defaults/redis-runtime.yml, 리뷰 2026-11-03 RR-04-10).
|
||||
|
||||
# /actuator/info 에 싣는 이 모듈의 상태(management.info.env). compose smoke 가 lane 이 실제로 켠
|
||||
# capability 를 이 값으로 확인한다. 위 스위치를 해석한 값이고, 기본값은 RedisCapabilityConfig 의
|
||||
|
||||
+4
@@ -1,5 +1,6 @@
|
||||
package dev.caskeleton.adapter.outbound.cache.redis;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.cache.CacheScriptLane;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.RedisTopologyEndpoint;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisCredentialResolver.RedisCredentials;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisRuntimeSettings;
|
||||
@@ -71,6 +72,9 @@ public final class LiveRedisRuntime {
|
||||
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
|
||||
limits.put(kind, 4);
|
||||
}
|
||||
// The cache region's own script lane, which the composition root contributes when the cache
|
||||
// binding is redis (RedisCacheConfig).
|
||||
limits.put(CacheScriptLane.CACHE_SCRIPT, 4);
|
||||
return new RedisRuntimeOwner(
|
||||
new RedisTopologyClientFactory(
|
||||
settings,
|
||||
|
||||
+1
-1
@@ -28,7 +28,7 @@ final class RedisTestContainer {
|
||||
static final String ACL_FILE_PROPERTY = "redis.container.acl-file";
|
||||
|
||||
private static final String IMAGE_PROPERTY = "ca.test.image.redis";
|
||||
private static final String ACL_IN_CONTAINER = "/etc/redis/acl/all-accounts.acl";
|
||||
private static final String ACL_IN_CONTAINER = "/etc/redis/acl/lane-fixture-accounts.acl";
|
||||
private static final int PORT = 6379;
|
||||
|
||||
private static GenericContainer<?> server;
|
||||
|
||||
+35
-1
@@ -60,8 +60,17 @@ class RedisCacheRegionAdapterTest {
|
||||
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
|
||||
limits.put(kind, 4);
|
||||
}
|
||||
return new RedisCacheRegionAdapter<>(
|
||||
limits.put(CacheScriptLane.CACHE_SCRIPT, 4);
|
||||
return cache(
|
||||
now,
|
||||
new RedisRuntimeOwner(client, limits, Duration.ofSeconds(1), Duration.ofSeconds(1)),
|
||||
cacheKeys);
|
||||
}
|
||||
|
||||
private RedisCacheRegionAdapter<String, String> cache(
|
||||
Instant now, RedisRuntimeOwner owner, RedisCacheRegionAdapter.CacheKeys cacheKeys) {
|
||||
return new RedisCacheRegionAdapter<>(
|
||||
owner,
|
||||
cacheKeys,
|
||||
key -> "digest-" + key,
|
||||
value -> value.getBytes(StandardCharsets.UTF_8),
|
||||
@@ -428,6 +437,31 @@ class RedisCacheRegionAdapterTest {
|
||||
.hasMessageContaining("ONLY_IF_SOURCE_REVISION_NEWER");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("멱등성·lease가 SCRIPT lane을 모두 잡고 있어도 캐시 조건부 기록은 캐시 전용 lane에서 기록된다")
|
||||
void aConditionalCacheWriteDoesNotWaitForTheCorrectnessScriptLane() {
|
||||
// given
|
||||
Map<RedisLane, Integer> limits = new java.util.LinkedHashMap<>();
|
||||
limits.put(RedisConnectionKind.REGULAR, 4);
|
||||
limits.put(RedisConnectionKind.SCRIPT, 1);
|
||||
limits.put(CacheScriptLane.CACHE_SCRIPT, 1);
|
||||
try (RedisRuntimeOwner owner =
|
||||
new RedisRuntimeOwner(
|
||||
StubRedisRuntimeClient.serving(gateway),
|
||||
limits,
|
||||
Duration.ofMillis(100),
|
||||
Duration.ofSeconds(1));
|
||||
var heldByAnIdempotencyClaim = owner.borrow(RedisConnectionKind.SCRIPT)) {
|
||||
RedisCacheRegionAdapter<String, String> cache = cache(T0, owner, keys);
|
||||
|
||||
// when
|
||||
CacheRecordOutcome outcome = cache.record("order-1", "payload", newerThan("2"));
|
||||
|
||||
// then
|
||||
assertThat(outcome).isEqualTo(CacheRecordOutcome.RECORDED);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("an unreachable Redis degrades: the cache is bypassed, nothing fails")
|
||||
void anUnreachableRedisDegrades() {
|
||||
|
||||
+1
@@ -23,6 +23,7 @@ class RedisCacheRegionContractTest extends CacheRegionContract {
|
||||
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
|
||||
limits.put(kind, 4);
|
||||
}
|
||||
limits.put(CacheScriptLane.CACHE_SCRIPT, 4);
|
||||
return new RedisCacheRegionAdapter<>(
|
||||
new RedisRuntimeOwner(
|
||||
StubRedisRuntimeClient.serving(new InMemorySemanticRedisCommands()),
|
||||
|
||||
+65
@@ -3,6 +3,9 @@ package dev.caskeleton.adapter.outbound.cache.redis.capability;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockito.Mockito.mock;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.cache.CacheScriptLane;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisLaneContribution;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisLaneMetrics;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.config.RedisRuntimeAutoConfiguration;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisRuntimeOwner;
|
||||
import dev.caskeleton.adapter.outbound.support.startup.StartupFailureException;
|
||||
@@ -11,6 +14,8 @@ import dev.caskeleton.application.idempotency.IdempotencyStorePort;
|
||||
import dev.caskeleton.application.lease.DistributedLeasePort;
|
||||
import dev.caskeleton.application.secret.SecretSourcePort;
|
||||
import dev.caskeleton.shared.ratelimit.EdgeRateLimitPort;
|
||||
import io.micrometer.core.instrument.MeterRegistry;
|
||||
import io.micrometer.core.instrument.simple.SimpleMeterRegistry;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.util.Arrays;
|
||||
@@ -94,6 +99,66 @@ class RedisCapabilityCompositionTest {
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("Redis 캐시 binding은 설정한 상한으로 캐시 전용 script lane을 SCRIPT lane과 따로 기여한다")
|
||||
void theCacheBindingContributesItsOwnScriptLane() {
|
||||
runner
|
||||
.withPropertyValues(
|
||||
"app.capabilities.cache.bindings.default=redis",
|
||||
"app.capabilities.cache.maximum-script-connections=3")
|
||||
.run(
|
||||
context -> {
|
||||
// when
|
||||
List<RedisLaneContribution> contributions =
|
||||
List.copyOf(context.getBeansOfType(RedisLaneContribution.class).values());
|
||||
|
||||
// then
|
||||
assertThat(context).hasNotFailed();
|
||||
assertThat(contributions)
|
||||
.containsExactly(new RedisLaneContribution(CacheScriptLane.CACHE_SCRIPT, 3));
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("Redis 캐시 binding의 캐시 전용 lane은 SCRIPT lane과 따로 대기·거절·재연결 지표를 낸다")
|
||||
void theCacheScriptLaneHasMetersOfItsOwn() {
|
||||
runner
|
||||
.withPropertyValues("app.capabilities.cache.bindings.default=redis")
|
||||
.withBean(MeterRegistry.class, SimpleMeterRegistry::new)
|
||||
.run(
|
||||
context -> {
|
||||
// when
|
||||
MeterRegistry registry = context.getBean(MeterRegistry.class);
|
||||
|
||||
// then
|
||||
assertThat(context).hasNotFailed();
|
||||
for (String lane : List.of(CacheScriptLane.CACHE_SCRIPT.name(), "SCRIPT")) {
|
||||
assertThat(
|
||||
registry
|
||||
.find(RedisLaneMetrics.CONNECTIONS_PENDING)
|
||||
.tag("lane", lane)
|
||||
.gauge())
|
||||
.as("waiting borrows of lane %s", lane)
|
||||
.isNotNull();
|
||||
assertThat(
|
||||
registry
|
||||
.find(RedisLaneMetrics.ACQUIRE_REFUSED)
|
||||
.tag("lane", lane)
|
||||
.tag("reason", "saturated")
|
||||
.counter())
|
||||
.as("refusals of lane %s", lane)
|
||||
.isNotNull();
|
||||
assertThat(
|
||||
registry
|
||||
.find(RedisLaneMetrics.CONNECTIONS_OPENED)
|
||||
.tag("lane", lane)
|
||||
.counter())
|
||||
.as("connections opened on lane %s", lane)
|
||||
.isNotNull();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a rate limiter with no configured policy is refused at startup")
|
||||
void aRateLimiterWithoutPoliciesIsRefused() {
|
||||
|
||||
+79
@@ -191,6 +191,85 @@ class RedisSemanticScriptAclProbeTest {
|
||||
assertThat(failure).isNull();
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(
|
||||
strings = {
|
||||
"LOADING Redis is loading the dataset in memory",
|
||||
"Connection closed",
|
||||
"BUSY Redis is busy running a script"
|
||||
})
|
||||
@DisplayName("cache만 고른 배포는 probe 도중 연결이 끊기거나 Redis가 아직 준비 중이어도 권한 오류로 보지 않고 기동한다")
|
||||
void aCacheOnlyDeploymentStartsWhenTheProbeIsInterruptedByATransientFailure(String reply) {
|
||||
// given
|
||||
RedisCapabilitySettings capabilities = new RedisCapabilitySettings();
|
||||
capabilities.getCache().getBindings().put("default", "redis");
|
||||
RedisSemanticScriptAclProbe probe =
|
||||
new RedisSemanticScriptAclProbe(
|
||||
owner(gatewayFailingLoadWith(reply), new AtomicInteger()),
|
||||
capabilities,
|
||||
"prod:svc:shared:probe".getBytes(StandardCharsets.UTF_8),
|
||||
Duration.ofSeconds(1));
|
||||
|
||||
// when
|
||||
Throwable failure = catchThrowable(probe::afterSingletonsInstantiated);
|
||||
|
||||
// then
|
||||
assertThat(failure).isNull();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("cache만 고른 배포도 probe가 기대와 다른 답을 받으면 기동에서 실패한다")
|
||||
void aCacheOnlyDeploymentWhoseProbeReadsBackTheWrongValueFailsStartup() {
|
||||
// given
|
||||
RedisCapabilitySettings capabilities = new RedisCapabilitySettings();
|
||||
capabilities.getCache().getBindings().put("default", "redis");
|
||||
RedisSemanticScriptAclProbe probe =
|
||||
new RedisSemanticScriptAclProbe(
|
||||
owner(gatewayAnswering(List.of(0L)), new AtomicInteger()),
|
||||
capabilities,
|
||||
"prod:svc:shared:probe".getBytes(StandardCharsets.UTF_8),
|
||||
Duration.ofSeconds(1));
|
||||
|
||||
// when
|
||||
Throwable failure = catchThrowable(probe::afterSingletonsInstantiated);
|
||||
|
||||
// then
|
||||
assertThat(failure)
|
||||
.isInstanceOf(CapabilityStartupValidationException.class)
|
||||
.hasMessageContaining("cache");
|
||||
}
|
||||
|
||||
private static SemanticRedisCommands gatewayFailingLoadWith(String reply) {
|
||||
return (SemanticRedisCommands)
|
||||
Proxy.newProxyInstance(
|
||||
SemanticRedisCommands.class.getClassLoader(),
|
||||
new Class<?>[] {SemanticRedisCommands.class},
|
||||
(proxy, method, arguments) ->
|
||||
switch (method.getName()) {
|
||||
case "loadScript" -> CompletableFuture.failedFuture(new RuntimeException(reply));
|
||||
case "toString" -> "RedisSemanticScriptAclProbeTestGateway";
|
||||
default ->
|
||||
throw new UnsupportedOperationException(
|
||||
"unexpected gateway method: " + method.getName());
|
||||
});
|
||||
}
|
||||
|
||||
private static SemanticRedisCommands gatewayAnswering(List<Object> reply) {
|
||||
return (SemanticRedisCommands)
|
||||
Proxy.newProxyInstance(
|
||||
SemanticRedisCommands.class.getClassLoader(),
|
||||
new Class<?>[] {SemanticRedisCommands.class},
|
||||
(proxy, method, arguments) ->
|
||||
switch (method.getName()) {
|
||||
case "loadScript" -> CompletableFuture.completedFuture("probe-digest");
|
||||
case "evaluateRegisteredForList" -> CompletableFuture.completedFuture(reply);
|
||||
case "toString" -> "RedisSemanticScriptAclProbeTestGateway";
|
||||
default ->
|
||||
throw new UnsupportedOperationException(
|
||||
"unexpected gateway method: " + method.getName());
|
||||
});
|
||||
}
|
||||
|
||||
private static RedisRuntimeOwner owner(SemanticRedisCommands gateway, AtomicInteger opened) {
|
||||
RedisRuntimeClient client =
|
||||
new RedisRuntimeClient() {
|
||||
|
||||
+5
-16
@@ -71,26 +71,15 @@ public final class RoleBasedFileAccessPolicy implements FileAccessPolicyPort {
|
||||
if (permitted.stream().noneMatch(subject.roles()::contains)) {
|
||||
throw denied(FileserverErrorCode.ACCESS_DENIED, "subject does not hold a permitted role");
|
||||
}
|
||||
if (descriptor.isPresent() && !isOwnedBy(descriptor.get(), subject)) {
|
||||
// Ownership is FileAccessSubject.owns, the rule the port owns (owner decision 14): a file with
|
||||
// no recorded owner predates ownership and belongs to nobody on the data plane. An admin role
|
||||
// does not bypass it — acting on somebody else's file is what the audited management plane is
|
||||
// for, and its operations carry no descriptor.
|
||||
if (descriptor.isPresent() && !subject.owns(descriptor.get())) {
|
||||
throw denied(FileserverErrorCode.ACCESS_DENIED, "subject does not own the file");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the file was created by this caller, in this caller's tenant.
|
||||
*
|
||||
* <p>A file without a recorded owner predates ownership and belongs to nobody on the data plane.
|
||||
* An admin role does not bypass this: acting on somebody else's file is what the audited
|
||||
* management plane is for, and its operations carry no descriptor.
|
||||
*/
|
||||
private static boolean isOwnedBy(FileDescriptor descriptor, FileAccessSubject subject) {
|
||||
return descriptor
|
||||
.owner()
|
||||
.filter(owner -> owner.principalId().equals(subject.principalId()))
|
||||
.filter(owner -> owner.tenantId().equals(subject.tenant()))
|
||||
.isPresent();
|
||||
}
|
||||
|
||||
private static boolean isAnonymous(FileAccessSubject subject) {
|
||||
return subject.principalId().equals(FileAccessSubject.anonymous().principalId());
|
||||
}
|
||||
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
package dev.caskeleton.adapter.outbound.fileserver.platform.security;
|
||||
|
||||
import dev.caskeleton.application.fileserver.api.security.FileAccessPolicyPort;
|
||||
import dev.caskeleton.application.fileserver.api.security.FileAccessSubject;
|
||||
import dev.caskeleton.application.fileserver.testkit.FileAccessPolicyPortContract;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.Set;
|
||||
|
||||
/** The production policy held to the ownership contract its fake is held to (RR-07-06). */
|
||||
class RoleBasedFileAccessPolicyContractTest extends FileAccessPolicyPortContract {
|
||||
|
||||
private static final String READ = "ROLE_FILE_READ";
|
||||
private static final String WRITE = "ROLE_FILE_WRITE";
|
||||
|
||||
private final RoleBasedFileAccessPolicy policy =
|
||||
new RoleBasedFileAccessPolicy(Set.of(READ), Set.of(WRITE), Set.of("ROLE_FILE_ADMIN"), false);
|
||||
|
||||
@Override
|
||||
protected FileAccessPolicyPort policy() {
|
||||
return policy;
|
||||
}
|
||||
|
||||
@Override
|
||||
protected FileAccessSubject permittedSubject(String principalId, Optional<String> tenant) {
|
||||
return new FileAccessSubject(
|
||||
principalId,
|
||||
Set.of(READ, WRITE),
|
||||
tenant.map(value -> Map.of(FileAccessSubject.TENANT_ATTRIBUTE, value)).orElse(Map.of()));
|
||||
}
|
||||
}
|
||||
@@ -41,7 +41,11 @@ no reactive (`WebClient`/Reactor Netty) surface; it was removed on 2026-09-25 (d
|
||||
cancels the request at the deadline, the JDK transport bounds its exchange timeout by the time
|
||||
left, the experimental Jetty HTTP/3 transport aborts the request, and the caller gets
|
||||
`HttpDeadlineExceededException`. A streaming download's setup (up to the response headers) runs
|
||||
inside the same deadline; after hand-over each body gap is bounded by `read-idle`.
|
||||
inside the same deadline; after hand-over each body gap is bounded by `read-idle` and nothing else,
|
||||
on both transports — the JDK transport arms its own per-read timer for this, because the JDK client
|
||||
has no socket read timeout. A body that fails after hand-over throws the same stable exceptions as
|
||||
a buffered call (`HttpResponseTimeoutException` for a stalled body, `HttpResponseTruncatedException`
|
||||
for a cut one).
|
||||
- **Evidence.** Each attempt reports `NOT_SENT`, `SENT_NO_RESPONSE`, `RESPONSE_RECEIVED`, or
|
||||
`PARTIAL_RESPONSE`. `NOT_SENT` is only claimed when a stage failure proves it.
|
||||
- **Retry.** Never from the HTTP method alone. Idempotency, body replayability, evidence, deadline,
|
||||
@@ -53,7 +57,9 @@ no reactive (`WebClient`/Reactor Netty) surface; it was removed on 2026-09-25 (d
|
||||
whether the call throws them or returns them as a result (`RETURN_RESULT`). Any other 4xx (404,
|
||||
409, 422 …) is a healthy answer, even when it reaches the caller as `HttpRemoteErrorException`.
|
||||
Window, threshold and open-state wait are per profile under `clients[N].circuit-breaker.*`
|
||||
(defaults are Resilience4j's; see `docs/httpclient/configuration-reference.md`).
|
||||
(defaults are Resilience4j's; see `docs/httpclient/configuration-reference.md`). An attempt that
|
||||
the caller's own deadline (`HttpOperation.deadline()`) ended is not counted at all: the caller ran
|
||||
out of time, the upstream did not fail. An attempt the profile's `total-call` ended still counts.
|
||||
- **Resources.** Connections and buffers are reclaimed after success, error, decode failure, size
|
||||
rejection, and cancellation alike.
|
||||
- **Secrets.** No URL, query value, path variable, token, cookie, or idempotency key reaches a log,
|
||||
|
||||
+14
-1
@@ -21,22 +21,35 @@ import org.springframework.http.client.JdkClientHttpRequestFactory;
|
||||
* timeout is fixed per factory, so a factory is made per request with {@code min(response-header
|
||||
* timeout, remaining deadline)} from {@link AttemptDeadline}. The factory only holds references;
|
||||
* the client and the write executor are shared.
|
||||
*
|
||||
* <p>A streaming download cannot use that request. Its body is handed to the caller and may run for
|
||||
* minutes, but Spring's timer only stops when the body stream is closed, so every JDK download
|
||||
* longer than the response-header timeout was cut mid-body (review 2026-11-03 RR-04-03). Inside a
|
||||
* streaming setup scope the factory makes a {@link JdkStreamingClientHttpRequest} instead: the
|
||||
* JDK's own request timeout bounds the wait for headers, the attempt deadline cancels the exchange
|
||||
* up to hand-over, and the body is bounded per gap by {@code read-idle}.
|
||||
*/
|
||||
final class DeadlineAwareJdkRequestFactory implements ClientHttpRequestFactory {
|
||||
|
||||
private final HttpClient client;
|
||||
private final Executor writeExecutor;
|
||||
private final Duration exchangeTimeout;
|
||||
private final Duration readIdle;
|
||||
|
||||
DeadlineAwareJdkRequestFactory(HttpClient client, Duration exchangeTimeout) {
|
||||
DeadlineAwareJdkRequestFactory(HttpClient client, Duration exchangeTimeout, Duration readIdle) {
|
||||
this.client = Objects.requireNonNull(client, "jdk http client");
|
||||
this.exchangeTimeout = Objects.requireNonNull(exchangeTimeout, "exchange timeout");
|
||||
this.readIdle = Objects.requireNonNull(readIdle, "read idle timeout");
|
||||
// The executor Spring would pick itself, chosen once instead of once per request.
|
||||
this.writeExecutor = client.executor().orElseGet(SimpleAsyncTaskExecutor::new);
|
||||
}
|
||||
|
||||
@Override
|
||||
public ClientHttpRequest createRequest(URI uri, HttpMethod httpMethod) throws IOException {
|
||||
if (AttemptDeadline.boundsSetupOnly()) {
|
||||
return new JdkStreamingClientHttpRequest(
|
||||
client, uri, httpMethod, AttemptDeadline.bound(exchangeTimeout), readIdle);
|
||||
}
|
||||
JdkClientHttpRequestFactory delegate = new JdkClientHttpRequestFactory(client, writeExecutor);
|
||||
delegate.setReadTimeout(AttemptDeadline.bound(exchangeTimeout));
|
||||
return delegate.createRequest(uri, httpMethod);
|
||||
|
||||
+2
-1
@@ -62,7 +62,8 @@ public final class JdkBlockingTransportProvider implements BlockingTransportProv
|
||||
HttpClient client = clientFactory.create(profile, tlsMaterialResolver.apply(profile));
|
||||
clients.put(new TransportResourceKey(profile.name(), generation), client);
|
||||
ClientHttpRequestFactory factory =
|
||||
new DeadlineAwareJdkRequestFactory(client, profile.timeout().responseHeader());
|
||||
new DeadlineAwareJdkRequestFactory(
|
||||
client, profile.timeout().responseHeader(), profile.timeout().readIdle());
|
||||
listener.onRuntimeCreated(profile.name(), ID);
|
||||
return factory;
|
||||
}
|
||||
|
||||
+128
@@ -0,0 +1,128 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.jdk;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.transport.AttemptDeadline;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.InterruptedIOException;
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.time.Duration;
|
||||
import java.util.Objects;
|
||||
import java.util.Set;
|
||||
import java.util.TreeSet;
|
||||
import java.util.concurrent.CancellationException;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.ExecutionException;
|
||||
import org.springframework.http.HttpHeaders;
|
||||
import org.springframework.http.HttpMethod;
|
||||
import org.springframework.http.client.AbstractBufferingClientHttpRequest;
|
||||
import org.springframework.http.client.ClientHttpResponse;
|
||||
|
||||
/**
|
||||
* A JDK request for a streaming download, whose timeouts end at the response headers (review
|
||||
* 2026-11-03 RR-04-03).
|
||||
*
|
||||
* <p>Three limits, each where the JDK can enforce it:
|
||||
*
|
||||
* <ul>
|
||||
* <li>the JDK's own request timeout bounds the wait for the response headers and stops when they
|
||||
* arrive;
|
||||
* <li>the attempt deadline cancels the pending exchange, so connect, send and the header wait
|
||||
* stay inside the call's total budget;
|
||||
* <li>after hand-over, {@link ReadIdleInputStream} bounds each gap of the body by {@code
|
||||
* read-idle} — the JDK client has no socket read timeout of its own.
|
||||
* </ul>
|
||||
*
|
||||
* <p>The request body is buffered: a streaming download sends none or a small one, and the JDK
|
||||
* needs a body publisher before the exchange starts.
|
||||
*/
|
||||
final class JdkStreamingClientHttpRequest extends AbstractBufferingClientHttpRequest {
|
||||
|
||||
/** Headers the JDK client refuses to set, as Spring's own JDK request lists them. */
|
||||
private static final Set<String> RESTRICTED_HEADERS = restrictedHeaders();
|
||||
|
||||
private final HttpClient client;
|
||||
private final URI uri;
|
||||
private final HttpMethod method;
|
||||
private final Duration responseHeaderTimeout;
|
||||
private final Duration readIdle;
|
||||
|
||||
JdkStreamingClientHttpRequest(
|
||||
HttpClient client,
|
||||
URI uri,
|
||||
HttpMethod method,
|
||||
Duration responseHeaderTimeout,
|
||||
Duration readIdle) {
|
||||
this.client = Objects.requireNonNull(client, "jdk http client");
|
||||
this.uri = Objects.requireNonNull(uri, "uri");
|
||||
this.method = Objects.requireNonNull(method, "method");
|
||||
this.responseHeaderTimeout =
|
||||
Objects.requireNonNull(responseHeaderTimeout, "response header timeout");
|
||||
this.readIdle = Objects.requireNonNull(readIdle, "read idle timeout");
|
||||
}
|
||||
|
||||
@Override
|
||||
public HttpMethod getMethod() {
|
||||
return method;
|
||||
}
|
||||
|
||||
@Override
|
||||
public URI getURI() {
|
||||
return uri;
|
||||
}
|
||||
|
||||
@Override
|
||||
protected ClientHttpResponse executeInternal(HttpHeaders headers, byte[] body)
|
||||
throws IOException {
|
||||
HttpRequest.Builder request =
|
||||
HttpRequest.newBuilder(uri)
|
||||
.timeout(responseHeaderTimeout)
|
||||
.method(
|
||||
method.name(),
|
||||
body.length == 0
|
||||
? HttpRequest.BodyPublishers.noBody()
|
||||
: HttpRequest.BodyPublishers.ofByteArray(body));
|
||||
headers.forEach(
|
||||
(name, values) -> {
|
||||
if (!RESTRICTED_HEADERS.contains(name)) {
|
||||
values.forEach(value -> request.header(name, value));
|
||||
}
|
||||
});
|
||||
CompletableFuture<HttpResponse<InputStream>> pending =
|
||||
client.sendAsync(request.build(), HttpResponse.BodyHandlers.ofInputStream());
|
||||
AttemptDeadline.abortOnExpiry(() -> pending.cancel(true));
|
||||
HttpResponse<InputStream> response = await(pending);
|
||||
return new JdkStreamingClientHttpResponse(
|
||||
response, new ReadIdleInputStream(response.body(), readIdle));
|
||||
}
|
||||
|
||||
private static HttpResponse<InputStream> await(
|
||||
CompletableFuture<HttpResponse<InputStream>> pending) throws IOException {
|
||||
try {
|
||||
return pending.get();
|
||||
} catch (InterruptedException interrupted) {
|
||||
Thread.currentThread().interrupt();
|
||||
pending.cancel(true);
|
||||
InterruptedIOException failure = new InterruptedIOException("streaming setup interrupted");
|
||||
failure.initCause(interrupted);
|
||||
throw failure;
|
||||
} catch (CancellationException cancelled) {
|
||||
// Only the attempt deadline cancels this future; the gateway names the deadline.
|
||||
throw new IOException("streaming setup cancelled at the attempt deadline", cancelled);
|
||||
} catch (ExecutionException failed) {
|
||||
Throwable cause = failed.getCause();
|
||||
if (cause instanceof IOException io) {
|
||||
throw io;
|
||||
}
|
||||
throw new IOException("streaming setup failed", cause);
|
||||
}
|
||||
}
|
||||
|
||||
private static Set<String> restrictedHeaders() {
|
||||
Set<String> restricted = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
|
||||
restricted.addAll(Set.of("connection", "content-length", "expect", "host", "upgrade"));
|
||||
return restricted;
|
||||
}
|
||||
}
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.jdk;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.util.Objects;
|
||||
import org.springframework.http.HttpHeaders;
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.http.HttpStatusCode;
|
||||
import org.springframework.http.client.ClientHttpResponse;
|
||||
|
||||
/** The response of a {@link JdkStreamingClientHttpRequest}, body bounded per gap. */
|
||||
final class JdkStreamingClientHttpResponse implements ClientHttpResponse {
|
||||
|
||||
private final HttpResponse<InputStream> response;
|
||||
private final InputStream body;
|
||||
private final HttpHeaders headers = new HttpHeaders();
|
||||
|
||||
JdkStreamingClientHttpResponse(HttpResponse<InputStream> response, InputStream body) {
|
||||
this.response = Objects.requireNonNull(response, "jdk response");
|
||||
this.body = Objects.requireNonNull(body, "body stream");
|
||||
// HTTP/2 pseudo-headers (":status") are protocol framing, not response headers.
|
||||
response
|
||||
.headers()
|
||||
.map()
|
||||
.forEach(
|
||||
(name, values) -> {
|
||||
if (!name.startsWith(":")) {
|
||||
headers.addAll(name, values);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@Override
|
||||
public HttpStatusCode getStatusCode() {
|
||||
return HttpStatusCode.valueOf(response.statusCode());
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getStatusText() {
|
||||
HttpStatus known = HttpStatus.resolve(response.statusCode());
|
||||
return known == null ? "" : known.getReasonPhrase();
|
||||
}
|
||||
|
||||
@Override
|
||||
public HttpHeaders getHeaders() {
|
||||
return headers;
|
||||
}
|
||||
|
||||
@Override
|
||||
public InputStream getBody() {
|
||||
return body;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
try {
|
||||
body.close();
|
||||
} catch (IOException ignored) {
|
||||
// Closing releases the exchange; a failure to close has nothing left to release.
|
||||
}
|
||||
}
|
||||
}
|
||||
+90
@@ -0,0 +1,90 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.jdk;
|
||||
|
||||
import java.io.FilterInputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.net.SocketTimeoutException;
|
||||
import java.time.Duration;
|
||||
import java.util.Objects;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
|
||||
/**
|
||||
* A JDK response body whose every read is bounded by {@code read-idle} (review 2026-11-03
|
||||
* RR-04-03).
|
||||
*
|
||||
* <p>The JDK client has no socket read timeout, so a streamed body that stopped arriving held the
|
||||
* caller's thread and the connection forever. Each read arms a timer on the JDK's shared delay
|
||||
* scheduler; when it fires the stream is closed, which wakes the blocked read, and the read reports
|
||||
* a {@link SocketTimeoutException} — the same exception Apache's socket timeout raises, so one
|
||||
* classifier reading names both. A read that returns first disarms its timer.
|
||||
*/
|
||||
final class ReadIdleInputStream extends FilterInputStream {
|
||||
|
||||
@FunctionalInterface
|
||||
private interface Read {
|
||||
long run() throws IOException;
|
||||
}
|
||||
|
||||
private final Duration readIdle;
|
||||
private final AtomicBoolean idled = new AtomicBoolean();
|
||||
|
||||
ReadIdleInputStream(InputStream delegate, Duration readIdle) {
|
||||
super(Objects.requireNonNull(delegate, "body stream"));
|
||||
this.readIdle = Objects.requireNonNull(readIdle, "read idle timeout");
|
||||
}
|
||||
|
||||
@Override
|
||||
public int read() throws IOException {
|
||||
return (int) guarded(super::read);
|
||||
}
|
||||
|
||||
@Override
|
||||
public int read(byte[] buffer, int offset, int length) throws IOException {
|
||||
return (int) guarded(() -> super.read(buffer, offset, length));
|
||||
}
|
||||
|
||||
@Override
|
||||
public long skip(long count) throws IOException {
|
||||
return guarded(() -> super.skip(count));
|
||||
}
|
||||
|
||||
private long guarded(Read read) throws IOException {
|
||||
requireNotIdled();
|
||||
CompletableFuture<Void> timer = new CompletableFuture<>();
|
||||
CompletableFuture<Void> expiry =
|
||||
timer.completeOnTimeout(null, readIdle.toNanos(), TimeUnit.NANOSECONDS).thenRun(this::idle);
|
||||
try {
|
||||
long result = read.run();
|
||||
requireNotIdled();
|
||||
return result;
|
||||
} catch (IOException failure) {
|
||||
requireNotIdled();
|
||||
throw failure;
|
||||
} finally {
|
||||
// Completing first skips the idle action and lets the scheduler drop the pending task.
|
||||
timer.cancel(false);
|
||||
expiry.cancel(false);
|
||||
}
|
||||
}
|
||||
|
||||
private void idle() {
|
||||
idled.set(true);
|
||||
try {
|
||||
in.close();
|
||||
} catch (IOException ignored) {
|
||||
// The read this wakes reports the timeout; a failed close has nothing more to say.
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* After the timer fired, whatever the read returned — data, end of stream or "closed" — is not
|
||||
* the rest of the body.
|
||||
*/
|
||||
private void requireNotIdled() throws SocketTimeoutException {
|
||||
if (idled.get()) {
|
||||
throw new SocketTimeoutException("no response body bytes within the read-idle timeout");
|
||||
}
|
||||
}
|
||||
}
|
||||
+17
@@ -17,6 +17,13 @@ public interface AttemptCircuitBreaker {
|
||||
|
||||
void onError(long durationNanos, Throwable failure);
|
||||
|
||||
/**
|
||||
* Gives back a permission without recording an outcome, for an attempt that says nothing about
|
||||
* the upstream (an attempt its caller's own deadline ended). Half-open trial slots are returned;
|
||||
* the window is left as it was.
|
||||
*/
|
||||
void releasePermission();
|
||||
|
||||
String state();
|
||||
|
||||
static AttemptCircuitBreaker alwaysClosed() {
|
||||
@@ -36,6 +43,11 @@ public interface AttemptCircuitBreaker {
|
||||
// Nothing to record.
|
||||
}
|
||||
|
||||
@Override
|
||||
public void releasePermission() {
|
||||
// Nothing was taken.
|
||||
}
|
||||
|
||||
@Override
|
||||
public String state() {
|
||||
return "CLOSED";
|
||||
@@ -60,6 +72,11 @@ public interface AttemptCircuitBreaker {
|
||||
delegate.onError(durationNanos, TimeUnit.NANOSECONDS, failure);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void releasePermission() {
|
||||
delegate.releasePermission();
|
||||
}
|
||||
|
||||
@Override
|
||||
public String state() {
|
||||
return delegate.getState().name();
|
||||
|
||||
+28
-1
@@ -7,6 +7,7 @@ import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpFailureMetadata;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRateLimitRejectedException;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
import java.util.function.BooleanSupplier;
|
||||
import java.util.function.Function;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
@@ -21,6 +22,11 @@ import java.util.function.Supplier;
|
||||
* <p>A local rejection (rate limiter or bulkhead) is deliberately <em>not</em> recorded as a
|
||||
* circuit error: the upstream never saw the request, and counting our own back-pressure as upstream
|
||||
* failure would open the breaker on a healthy dependency.
|
||||
*
|
||||
* <p>An attempt its caller abandoned is not recorded either, for the same reason: an attempt that a
|
||||
* caller's own deadline ended while the upstream was still answering within the profile's limits is
|
||||
* the caller running out of time, not the upstream failing (review 2026-11-03 RR-04-01). Its
|
||||
* permission is given back instead.
|
||||
*/
|
||||
public final class AttemptResiliencePipeline {
|
||||
|
||||
@@ -79,7 +85,24 @@ public final class AttemptResiliencePipeline {
|
||||
* @return the attempt's value, whether or not it represents a remote failure
|
||||
*/
|
||||
public <T> T execute(AttemptCall<T> call, Function<T, Optional<Throwable>> remoteFailure) {
|
||||
return execute(call, remoteFailure, () -> false);
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs one attempt, recording nothing against the upstream when its caller abandoned it.
|
||||
*
|
||||
* @param call the attempt, including any redirect hops it follows
|
||||
* @param remoteFailure returns the failure to record when the value represents an upstream error
|
||||
* @param abandonedByCaller read after the attempt failed: whether the caller's own deadline ended
|
||||
* it, in which case the failure is not the upstream's and the breaker records nothing
|
||||
* @return the attempt's value, whether or not it represents a remote failure
|
||||
*/
|
||||
public <T> T execute(
|
||||
AttemptCall<T> call,
|
||||
Function<T, Optional<Throwable>> remoteFailure,
|
||||
BooleanSupplier abandonedByCaller) {
|
||||
Objects.requireNonNull(remoteFailure, "remote failure classifier");
|
||||
Objects.requireNonNull(abandonedByCaller, "abandoned-by-caller signal");
|
||||
if (!circuitBreaker.tryAcquirePermission()) {
|
||||
rejections.circuitOpen();
|
||||
throw new HttpCircuitOpenException(
|
||||
@@ -110,7 +133,11 @@ public final class AttemptResiliencePipeline {
|
||||
return result;
|
||||
} catch (Throwable failure) {
|
||||
releaseAttemptPermits();
|
||||
circuitBreaker.onError(System.nanoTime() - started, failure);
|
||||
if (abandonedByCaller.getAsBoolean()) {
|
||||
circuitBreaker.releasePermission();
|
||||
} else {
|
||||
circuitBreaker.onError(System.nanoTime() - started, failure);
|
||||
}
|
||||
throw translate(failure);
|
||||
}
|
||||
}
|
||||
|
||||
+26
-1
@@ -10,11 +10,31 @@ import java.util.Objects;
|
||||
*
|
||||
* <p>Everything — pool acquire, DNS, connect, TLS, request write, response read, and every retry
|
||||
* backoff — is spent inside this single budget.
|
||||
*
|
||||
* @param at the instant the call ends
|
||||
* @param source who set that instant, which decides what an attempt it ends says about the upstream
|
||||
*/
|
||||
public record Deadline(Instant at) {
|
||||
public record Deadline(Instant at, Source source) {
|
||||
|
||||
/**
|
||||
* Who set the instant a call ends at.
|
||||
*
|
||||
* <p>The two say different things when they end an attempt. The profile's total-call timeout is
|
||||
* the operator's statement of how long a healthy upstream may take, so an attempt it ends is the
|
||||
* upstream's failure. A caller's own deadline is a statement about the caller — it had only that
|
||||
* much time left — so an attempt it ends says nothing about the upstream (review 2026-11-03
|
||||
* RR-04-01).
|
||||
*/
|
||||
public enum Source {
|
||||
/** The profile's {@code timeout.total-call}. */
|
||||
PROFILE_TOTAL_CALL,
|
||||
/** The caller's {@code HttpOperation.deadline()}, when it is earlier than the profile's. */
|
||||
CALLER
|
||||
}
|
||||
|
||||
public Deadline {
|
||||
Objects.requireNonNull(at, "deadline instant");
|
||||
Objects.requireNonNull(source, "deadline source");
|
||||
}
|
||||
|
||||
public Duration remaining(Clock clock) {
|
||||
@@ -26,6 +46,11 @@ public record Deadline(Instant at) {
|
||||
return !clock.instant().isBefore(at);
|
||||
}
|
||||
|
||||
/** Whether the instant is the caller's own budget rather than the profile's total-call. */
|
||||
public boolean setByCaller() {
|
||||
return source == Source.CALLER;
|
||||
}
|
||||
|
||||
/** The earlier of two deadlines; a parent budget can only ever shrink a child's. */
|
||||
public Deadline earliest(Deadline other) {
|
||||
return at.isBefore(other.at) ? this : other;
|
||||
|
||||
+7
-1
@@ -9,6 +9,9 @@ import java.util.Optional;
|
||||
/**
|
||||
* Derives the effective deadline from the caller's parent budget and the profile total-call timeout
|
||||
* (design §15.2): {@code effectiveDeadline = min(parentDeadline, now + totalCallTimeout)}.
|
||||
*
|
||||
* <p>The result also records which of the two won. A tie goes to the profile: the profile's limit
|
||||
* would have ended the call at that instant anyway.
|
||||
*/
|
||||
public final class DeadlineCalculator {
|
||||
|
||||
@@ -17,6 +20,9 @@ public final class DeadlineCalculator {
|
||||
Objects.requireNonNull(totalCall, "total call timeout");
|
||||
Objects.requireNonNull(clock, "clock");
|
||||
Instant local = clock.instant().plus(totalCall);
|
||||
return new Deadline(parent.map(value -> value.isBefore(local) ? value : local).orElse(local));
|
||||
return parent
|
||||
.filter(value -> value.isBefore(local))
|
||||
.map(value -> new Deadline(value, Deadline.Source.CALLER))
|
||||
.orElseGet(() -> new Deadline(local, Deadline.Source.PROFILE_TOTAL_CALL));
|
||||
}
|
||||
}
|
||||
|
||||
+8
-1
@@ -63,6 +63,7 @@ public final class BlockingAttemptExecutor {
|
||||
try (AttemptDeadline.Scope scope = AttemptDeadline.open(deadline.remaining(support.clock()))) {
|
||||
return executeWithin(
|
||||
scope,
|
||||
deadline,
|
||||
runtime,
|
||||
prepared,
|
||||
responseType,
|
||||
@@ -76,6 +77,7 @@ public final class BlockingAttemptExecutor {
|
||||
|
||||
private <T> AttemptOutcome<T> executeWithin(
|
||||
AttemptDeadline.Scope scope,
|
||||
Deadline deadline,
|
||||
BlockingClientRuntime runtime,
|
||||
PreparedOperation prepared,
|
||||
ResponseType<T> responseType,
|
||||
@@ -146,7 +148,12 @@ public final class BlockingAttemptExecutor {
|
||||
tracker.firstByteDelivered());
|
||||
}
|
||||
},
|
||||
outcome -> upstreamFailure(outcome, baseMetadata));
|
||||
outcome -> upstreamFailure(outcome, baseMetadata),
|
||||
// The caller's own deadline ending the attempt is not the upstream failing. The
|
||||
// profile's total-call ending it is: that limit is the operator's statement of
|
||||
// how long a healthy upstream may take, and a slow-drip upstream that never trips
|
||||
// a per-gap timeout is seen by the breaker only through it (RR-04-01).
|
||||
() -> deadline.setByCaller() && scope.expired());
|
||||
return finish(observation, scope, attemptOutcome);
|
||||
} catch (HttpClientException stable) {
|
||||
return finish(
|
||||
|
||||
+22
-4
@@ -5,6 +5,7 @@ import dev.caskeleton.adapter.outbound.httpclient.api.HttpStatus;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpDeadlineExceededException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpFailureMetadata;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRemoteErrorException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.AttemptStage;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.HttpOperation;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.result.BlockingStreamingResponse;
|
||||
@@ -78,9 +79,11 @@ public final class BlockingStreamingGateway {
|
||||
// which deregisters the transport's abort, so the body is then bounded by read-idle per gap
|
||||
// and a long download is not cut at an arbitrary total. Without the scope a setup was bounded
|
||||
// only by per-gap stage timeouts, and an upstream that held or dripped its headers kept the
|
||||
// caller past its deadline (finding 05-06).
|
||||
// caller past its deadline (finding 05-06). It is a setup scope, not an attempt scope: the
|
||||
// JDK transport arms an exchange-wide timeout for an attempt, which cut every download
|
||||
// longer than the response-header timeout (RR-04-03).
|
||||
AttemptDeadline.Scope setupDeadline =
|
||||
AttemptDeadline.open(setup.deadline().remaining(runtime.support().clock()));
|
||||
AttemptDeadline.openStreamSetup(setup.deadline().remaining(runtime.support().clock()));
|
||||
try (setupDeadline) {
|
||||
response =
|
||||
spec.exchange(
|
||||
@@ -102,9 +105,24 @@ public final class BlockingStreamingGateway {
|
||||
.withStatus(new HttpStatus(status))
|
||||
.withEvidence(ExecutionEvidence.RESPONSE_RECEIVED));
|
||||
}
|
||||
// A body that fails after hand-over reaches the caller as the same stable
|
||||
// exceptions a buffered call throws, classified by the transport's own
|
||||
// classifier at the body stage — not as the engine's bare IOException, which
|
||||
// carried no category, no evidence and nothing to log (RR-04-03).
|
||||
InputStream bounded =
|
||||
new CountingBoundedInputStream(
|
||||
rawResponse.getBody(), limiter::recordWireBytes);
|
||||
new ClassifiedBodyInputStream(
|
||||
new CountingBoundedInputStream(
|
||||
rawResponse.getBody(), limiter::recordWireBytes),
|
||||
bodyFailure ->
|
||||
runtime
|
||||
.support()
|
||||
.exceptionMapper()
|
||||
.map(
|
||||
runtime
|
||||
.failureClassifier()
|
||||
.classify(bodyFailure, AttemptStage.RESPONSE_BODY),
|
||||
metadata,
|
||||
bodyFailure));
|
||||
return new DefaultBlockingStreamingResponse(
|
||||
new HttpStatus(status), responseHeaders, bounded, rawResponse::close);
|
||||
},
|
||||
|
||||
+64
@@ -0,0 +1,64 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.restclient;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpClientException;
|
||||
import java.io.FilterInputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.util.Objects;
|
||||
import java.util.function.Function;
|
||||
|
||||
/**
|
||||
* A handed-over response body whose read failures reach the caller as stable exceptions (review
|
||||
* 2026-11-03 RR-04-03).
|
||||
*
|
||||
* <p>A streamed body is read after the gateway has returned, so the gateway's own failure mapping
|
||||
* never sees it. Without this a read-idle timeout or a cut connection reached the caller as the
|
||||
* engine's bare {@code IOException} — no category, no evidence, nothing a retry policy or a log
|
||||
* could branch on. Closing is the caller's own act and is passed through unclassified.
|
||||
*/
|
||||
final class ClassifiedBodyInputStream extends FilterInputStream {
|
||||
|
||||
private final Function<IOException, HttpClientException> classifier;
|
||||
|
||||
ClassifiedBodyInputStream(
|
||||
InputStream delegate, Function<IOException, HttpClientException> classifier) {
|
||||
super(Objects.requireNonNull(delegate, "body stream"));
|
||||
this.classifier = Objects.requireNonNull(classifier, "body failure classifier");
|
||||
}
|
||||
|
||||
@Override
|
||||
public int read() throws IOException {
|
||||
try {
|
||||
return super.read();
|
||||
} catch (IOException failure) {
|
||||
throw classifier.apply(failure);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public int read(byte[] buffer, int offset, int length) throws IOException {
|
||||
try {
|
||||
return super.read(buffer, offset, length);
|
||||
} catch (IOException failure) {
|
||||
throw classifier.apply(failure);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public long skip(long count) throws IOException {
|
||||
try {
|
||||
return super.skip(count);
|
||||
} catch (IOException failure) {
|
||||
throw classifier.apply(failure);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public int available() throws IOException {
|
||||
try {
|
||||
return super.available();
|
||||
} catch (IOException failure) {
|
||||
throw classifier.apply(failure);
|
||||
}
|
||||
}
|
||||
}
|
||||
+35
-2
@@ -28,6 +28,11 @@ import java.util.concurrent.TimeUnit;
|
||||
* <p>The scope is thread-confined because a blocking attempt runs on the caller's thread from
|
||||
* request creation to the last body byte. When no scope is open — a transport used outside the
|
||||
* gateway — both methods leave the transport's own behaviour unchanged.
|
||||
*
|
||||
* <p>A streaming download opens a different scope ({@link #openStreamSetup(Duration)}): it bounds
|
||||
* the exchange only up to the response headers and is closed when the body is handed over. A
|
||||
* transport whose engine timeout would otherwise keep running over the body reads {@link
|
||||
* #boundsSetupOnly()} and keeps that timeout off it (review 2026-11-03 RR-04-03).
|
||||
*/
|
||||
public final class AttemptDeadline {
|
||||
|
||||
@@ -41,8 +46,34 @@ public final class AttemptDeadline {
|
||||
* @param remaining the time left before the logical call's effective deadline
|
||||
*/
|
||||
public static Scope open(Duration remaining) {
|
||||
return open(remaining, false);
|
||||
}
|
||||
|
||||
/**
|
||||
* Opens the deadline scope for a streaming download's setup on the current thread.
|
||||
*
|
||||
* <p>The caller closes it when the response is handed over, so the deadline never reaches the
|
||||
* body: a download may legitimately outlast any fixed total (design §15.3).
|
||||
*
|
||||
* @param remaining the time left before the logical call's effective deadline
|
||||
*/
|
||||
public static Scope openStreamSetup(Duration remaining) {
|
||||
return open(remaining, true);
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the current scope bounds only a streaming setup, so a timeout the transport arms must
|
||||
* not outlive the response headers.
|
||||
*/
|
||||
public static boolean boundsSetupOnly() {
|
||||
Scope scope = CURRENT.get();
|
||||
return scope != null && scope.setupOnly;
|
||||
}
|
||||
|
||||
private static Scope open(Duration remaining, boolean setupOnly) {
|
||||
Objects.requireNonNull(remaining, "remaining deadline");
|
||||
Scope scope = new Scope(remaining.isNegative() ? Duration.ZERO : remaining, CURRENT.get());
|
||||
Scope scope =
|
||||
new Scope(remaining.isNegative() ? Duration.ZERO : remaining, setupOnly, CURRENT.get());
|
||||
CURRENT.set(scope);
|
||||
return scope;
|
||||
}
|
||||
@@ -78,6 +109,7 @@ public final class AttemptDeadline {
|
||||
public static final class Scope implements AutoCloseable {
|
||||
|
||||
private final long endNanos;
|
||||
private final boolean setupOnly;
|
||||
private final Scope outer;
|
||||
private final CompletableFuture<Void> timer = new CompletableFuture<>();
|
||||
private final CompletableFuture<Void> expiry;
|
||||
@@ -85,8 +117,9 @@ public final class AttemptDeadline {
|
||||
private boolean fired;
|
||||
private boolean closed;
|
||||
|
||||
private Scope(Duration remaining, Scope outer) {
|
||||
private Scope(Duration remaining, boolean setupOnly, Scope outer) {
|
||||
this.endNanos = System.nanoTime() + remaining.toNanos();
|
||||
this.setupOnly = setupOnly;
|
||||
this.outer = outer;
|
||||
// The JDK's shared delay scheduler, the same one Spring's JDK request uses for its timeout.
|
||||
// Completing the future on close() also cancels the scheduled task, so a finished attempt
|
||||
|
||||
+6
-3
@@ -34,7 +34,8 @@ class DeadlineCalculatorTest {
|
||||
|
||||
@Test
|
||||
void refusesAttemptWhenBackoffConsumesRemainingBudget() {
|
||||
Deadline deadline = new Deadline(Instant.parse("2026-08-08T00:00:01Z"));
|
||||
Deadline deadline =
|
||||
new Deadline(Instant.parse("2026-08-08T00:00:01Z"), Deadline.Source.PROFILE_TOTAL_CALL);
|
||||
Optional<AttemptBudget> result =
|
||||
new AttemptBudgetCalculator(clock)
|
||||
.nextAttempt(
|
||||
@@ -44,7 +45,8 @@ class DeadlineCalculatorTest {
|
||||
|
||||
@Test
|
||||
void grantsAttemptWhenBudgetCoversBackoffMinimumAndCleanup() {
|
||||
Deadline deadline = new Deadline(Instant.parse("2026-08-08T00:00:02Z"));
|
||||
Deadline deadline =
|
||||
new Deadline(Instant.parse("2026-08-08T00:00:02Z"), Deadline.Source.PROFILE_TOTAL_CALL);
|
||||
Optional<AttemptBudget> result =
|
||||
new AttemptBudgetCalculator(clock)
|
||||
.nextAttempt(
|
||||
@@ -60,7 +62,8 @@ class DeadlineCalculatorTest {
|
||||
assertThatThrownBy(
|
||||
() ->
|
||||
guard.requireTimeRemaining(
|
||||
new Deadline(Instant.parse("2026-08-07T23:59:59Z")),
|
||||
new Deadline(
|
||||
Instant.parse("2026-08-07T23:59:59Z"), Deadline.Source.PROFILE_TOTAL_CALL),
|
||||
CoreFixtures.notSentMetadata()))
|
||||
.isInstanceOf(HttpDeadlineExceededException.class);
|
||||
assertThatThrownBy(
|
||||
|
||||
+2
-2
@@ -147,7 +147,7 @@ class RetryCoordinatorTest {
|
||||
|
||||
@Override
|
||||
public Deadline deadline() {
|
||||
return new Deadline(clock.instant().plusSeconds(5));
|
||||
return new Deadline(clock.instant().plusSeconds(5), Deadline.Source.PROFILE_TOTAL_CALL);
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -202,7 +202,7 @@ class RetryCoordinatorTest {
|
||||
|
||||
@Override
|
||||
public Deadline deadline() {
|
||||
return new Deadline(clock.instant().plusSeconds(5));
|
||||
return new Deadline(clock.instant().plusSeconds(5), Deadline.Source.PROFILE_TOTAL_CALL);
|
||||
}
|
||||
|
||||
@Override
|
||||
|
||||
+119
@@ -5,20 +5,28 @@ import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.OperationName;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpCircuitOpenException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpDeadlineExceededException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRemoteErrorException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.HttpOperation;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.result.HttpCallResult;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.result.ResponseType;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.CircuitBreakerSettings;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.TimeoutSettings;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.TransportType;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.ClientProfiles;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.MockHttpServer;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.TestGateways;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.EnumSource;
|
||||
|
||||
/**
|
||||
* What the per-profile circuit breaker counts as an upstream failure, end to end (finding 05-05).
|
||||
@@ -31,6 +39,10 @@ class CircuitBreakerStatusAccountingTest {
|
||||
private static final int CALLS = 200;
|
||||
private static final int LIBRARY_DEFAULT_WINDOW = 100;
|
||||
private static final int SMALL_WINDOW = 5;
|
||||
private static final int TINY_WINDOW = 2;
|
||||
private static final Duration HEALTHY_ANSWER_TIME = Duration.ofMillis(600);
|
||||
private static final Duration CALLER_BUDGET = Duration.ofMillis(200);
|
||||
private static final Duration PROFILE_TOTAL_CALL = Duration.ofSeconds(1);
|
||||
|
||||
@Test
|
||||
@DisplayName("정상 응답인 404가 200번 이어져도 circuit이 열리지 않고 모든 호출이 상대 서버에 닿는다")
|
||||
@@ -158,6 +170,113 @@ class CircuitBreakerStatusAccountingTest {
|
||||
}
|
||||
}
|
||||
|
||||
@ParameterizedTest(name = "{0}")
|
||||
@EnumSource(
|
||||
value = TransportType.class,
|
||||
names = {"APACHE", "JDK"})
|
||||
@DisplayName("호출자 deadline으로 끊은 시도는 상대 장애로 세지 않아 창을 채워도 다음 호출이 상대 서버에 닿는다")
|
||||
void attemptsCutShortByTheCallersOwnDeadlineDoNotOpenTheCircuit(TransportType transport)
|
||||
throws Exception {
|
||||
try (MockHttpServer server = MockHttpServer.start();
|
||||
TestGateways.Harness harness =
|
||||
TestGateways.forProfile(
|
||||
ClientProfiles.builder("caller-deadline")
|
||||
.baseUrl(server.uri("/"))
|
||||
.transport(transport)
|
||||
.circuitBreaker(
|
||||
new CircuitBreakerSettings(
|
||||
50, SMALL_WINDOW, SMALL_WINDOW, Duration.ofMinutes(1), 1))
|
||||
.build())) {
|
||||
// given
|
||||
for (int i = 0; i <= SMALL_WINDOW; i++) {
|
||||
server.enqueueDelayedHeaders(
|
||||
200,
|
||||
"application/json",
|
||||
"\"ok\"".getBytes(StandardCharsets.UTF_8),
|
||||
HEALTHY_ANSWER_TIME);
|
||||
}
|
||||
List<Throwable> windowFailures = new ArrayList<>();
|
||||
for (int i = 0; i < SMALL_WINDOW; i++) {
|
||||
windowFailures.add(
|
||||
catchThrowable(
|
||||
() -> getUserWithin(harness, Optional.of(Instant.now().plus(CALLER_BUDGET)))));
|
||||
}
|
||||
assertThat(windowFailures)
|
||||
.allSatisfy(f -> assertThat(f).isInstanceOf(HttpDeadlineExceededException.class));
|
||||
|
||||
// when
|
||||
Throwable next = catchThrowable(() -> getUserWithin(harness, Optional.empty()));
|
||||
|
||||
// then
|
||||
assertThat(next).isNull();
|
||||
assertThat(server.requestCount()).isEqualTo(SMALL_WINDOW + 1);
|
||||
}
|
||||
}
|
||||
|
||||
@ParameterizedTest(name = "{0}")
|
||||
@EnumSource(
|
||||
value = TransportType.class,
|
||||
names = {"APACHE", "JDK"})
|
||||
@DisplayName("프로필 total-call 시간으로 끊긴 시도는 상대 장애로 세어 창을 채우면 circuit이 열린다")
|
||||
void attemptsCutShortByTheProfileTotalCallStillOpenTheCircuit(TransportType transport)
|
||||
throws Exception {
|
||||
TimeoutSettings base = ClientProfiles.timeouts();
|
||||
try (MockHttpServer server = MockHttpServer.start();
|
||||
TestGateways.Harness harness =
|
||||
TestGateways.forProfile(
|
||||
ClientProfiles.builder("profile-deadline")
|
||||
.baseUrl(server.uri("/"))
|
||||
.transport(transport)
|
||||
.timeout(
|
||||
new TimeoutSettings(
|
||||
base.dns(),
|
||||
base.connect(),
|
||||
base.tlsHandshake(),
|
||||
base.responseHeader(),
|
||||
base.readIdle(),
|
||||
PROFILE_TOTAL_CALL))
|
||||
.circuitBreaker(
|
||||
new CircuitBreakerSettings(
|
||||
50, TINY_WINDOW, TINY_WINDOW, Duration.ofMinutes(1), 1))
|
||||
.build())) {
|
||||
// given
|
||||
List<Throwable> windowFailures = new ArrayList<>();
|
||||
for (int i = 0; i < TINY_WINDOW; i++) {
|
||||
server.enqueueSlowDripBody(200, 100, Duration.ofMillis(50));
|
||||
windowFailures.add(catchThrowable(() -> getUserWithin(harness, Optional.empty())));
|
||||
}
|
||||
assertThat(windowFailures)
|
||||
.allSatisfy(f -> assertThat(f).isInstanceOf(HttpDeadlineExceededException.class));
|
||||
|
||||
// when
|
||||
Throwable next = catchThrowable(() -> getUserWithin(harness, Optional.empty()));
|
||||
|
||||
// then
|
||||
assertThat(next).isInstanceOf(HttpCircuitOpenException.class);
|
||||
assertThat(server.requestCount()).isEqualTo(TINY_WINDOW);
|
||||
}
|
||||
}
|
||||
|
||||
private static HttpCallResult<String> getUserWithin(
|
||||
TestGateways.Harness harness, Optional<Instant> deadline) {
|
||||
HttpOperation get = HttpOperation.get(new OperationName("get-user"), "/users/1", Map.of());
|
||||
return harness
|
||||
.gateway()
|
||||
.exchange(
|
||||
harness.profile().name(),
|
||||
new HttpOperation(
|
||||
get.operationName(),
|
||||
get.method(),
|
||||
get.uriTemplate(),
|
||||
get.uriVariables(),
|
||||
get.headers(),
|
||||
get.body(),
|
||||
get.idempotency(),
|
||||
get.idempotencyKey(),
|
||||
deadline),
|
||||
ResponseType.of(String.class));
|
||||
}
|
||||
|
||||
private static HttpCallResult<String> getUserAsResult(TestGateways.Harness harness) {
|
||||
return harness
|
||||
.gateway()
|
||||
|
||||
+124
@@ -0,0 +1,124 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.restclient;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.OperationName;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpResponseTimeoutException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.AttemptStage;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.HttpOperation;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.result.BlockingStreamingResponse;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ClientProfile;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ResponseLimits;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.TimeoutSettings;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.TransportType;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.ClientProfiles;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.MockHttpServer;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.TestGateways;
|
||||
import java.time.Duration;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.EnumSource;
|
||||
|
||||
/**
|
||||
* What bounds a streaming download's body once it has been handed over (review 2026-11-03 RR-04-03,
|
||||
* design §15.3).
|
||||
*
|
||||
* <p>The documented contract is transport-independent: the total budget and the response-header
|
||||
* timeout bound the setup, and after hand-over each gap of the body is bounded by {@code read-idle}
|
||||
* alone, so a long download is not cut at an arbitrary total. The JDK transport used to break it:
|
||||
* Spring's JDK request enforces its read timeout over the whole exchange and closes the body stream
|
||||
* when it fires, so every JDK download longer than the response-header timeout was cut, and the
|
||||
* caller got an unclassified {@code IOException: closed}.
|
||||
*
|
||||
* <p>The profile's response-header timeout (500 ms) and total budget (1 s) are both shorter than
|
||||
* the body (about 2 s), and the read-idle timeout (1 s) is longer than every gap of it.
|
||||
*/
|
||||
class StreamingBodyLifetimeTest {
|
||||
|
||||
private static final int BODY_BYTES = 20;
|
||||
private static final Duration DRIP_GAP = Duration.ofMillis(100);
|
||||
private static final Duration STALL_GAP = Duration.ofMillis(2500);
|
||||
|
||||
@ParameterizedTest(name = "{0}")
|
||||
@EnumSource(
|
||||
value = TransportType.class,
|
||||
names = {"APACHE", "JDK"})
|
||||
@DisplayName("넘겨받은 스트리밍 본문은 응답 헤더 시간과 전체 예산을 넘겨도 끝까지 읽힌다")
|
||||
void aStreamedBodyOutlivesTheResponseHeaderTimeoutAndTheTotalBudget(TransportType transport)
|
||||
throws Exception {
|
||||
try (MockHttpServer server = MockHttpServer.start();
|
||||
TestGateways.Harness harness = TestGateways.forProfile(profile(server, transport))) {
|
||||
// given
|
||||
server.enqueueSlowDripBody(200, BODY_BYTES, DRIP_GAP);
|
||||
BlockingStreamingGateway gateway = new BlockingStreamingGateway(harness.registry());
|
||||
|
||||
// when
|
||||
byte[] body;
|
||||
try (BlockingStreamingResponse response = gateway.download(harness.profile().name(), get())) {
|
||||
body = response.body().readAllBytes();
|
||||
}
|
||||
|
||||
// then
|
||||
assertThat(body).hasSize(BODY_BYTES);
|
||||
}
|
||||
}
|
||||
|
||||
@ParameterizedTest(name = "{0}")
|
||||
@EnumSource(
|
||||
value = TransportType.class,
|
||||
names = {"APACHE", "JDK"})
|
||||
@DisplayName("본문 사이 공백이 read-idle을 넘으면 분류된 응답 시간 초과로 끝난다")
|
||||
void aBodyGapLongerThanReadIdleEndsWithAClassifiedTimeout(TransportType transport)
|
||||
throws Exception {
|
||||
try (MockHttpServer server = MockHttpServer.start();
|
||||
TestGateways.Harness harness = TestGateways.forProfile(profile(server, transport))) {
|
||||
// given
|
||||
server.enqueueSlowDripBody(200, BODY_BYTES, STALL_GAP);
|
||||
BlockingStreamingGateway gateway = new BlockingStreamingGateway(harness.registry());
|
||||
long started = System.nanoTime();
|
||||
|
||||
// when
|
||||
Throwable failure =
|
||||
catchThrowable(
|
||||
() -> {
|
||||
try (BlockingStreamingResponse response =
|
||||
gateway.download(harness.profile().name(), get())) {
|
||||
response.body().readAllBytes();
|
||||
}
|
||||
});
|
||||
|
||||
// then
|
||||
Duration elapsed = Duration.ofNanos(System.nanoTime() - started);
|
||||
assertThat(failure).isInstanceOf(HttpResponseTimeoutException.class);
|
||||
assertThat(elapsed).isLessThan(STALL_GAP);
|
||||
HttpResponseTimeoutException timeout = (HttpResponseTimeoutException) failure;
|
||||
assertThat(timeout.metadata().stage()).isEqualTo(AttemptStage.RESPONSE_BODY);
|
||||
assertThat(timeout.metadata().evidence()).isEqualTo(ExecutionEvidence.PARTIAL_RESPONSE);
|
||||
}
|
||||
}
|
||||
|
||||
private static HttpOperation get() {
|
||||
return HttpOperation.get(new OperationName("download"), "/download", Map.of());
|
||||
}
|
||||
|
||||
private static ClientProfile profile(MockHttpServer server, TransportType transport) {
|
||||
TimeoutSettings base = ClientProfiles.timeouts();
|
||||
return ClientProfiles.builder("streaming-body")
|
||||
.baseUrl(server.uri("/"))
|
||||
.transport(transport)
|
||||
.timeout(
|
||||
new TimeoutSettings(
|
||||
base.dns(),
|
||||
base.connect(),
|
||||
base.tlsHandshake(),
|
||||
Duration.ofMillis(500),
|
||||
Duration.ofSeconds(1),
|
||||
Duration.ofSeconds(1)))
|
||||
.response(new ResponseLimits(1024, 1024, Set.of("text/plain")))
|
||||
.build();
|
||||
}
|
||||
}
|
||||
+5
@@ -68,6 +68,11 @@ public final class RecordingResilienceComponents {
|
||||
events.add("circuit-error");
|
||||
}
|
||||
|
||||
@Override
|
||||
public void releasePermission() {
|
||||
events.add("circuit-release");
|
||||
}
|
||||
|
||||
@Override
|
||||
public String state() {
|
||||
return circuitOpen ? "OPEN" : "CLOSED";
|
||||
|
||||
@@ -20,16 +20,22 @@ build choices) live in [README.md](README.md).
|
||||
- `UuidV7DurableOperationIdFactory` — `DurableOperationIdFactory` implementation (UUIDv7 via
|
||||
`uuid-creator`). The operation id is also the submission event's id, and the messaging platform's
|
||||
message id accepts only UUIDv7 (review 2026-11-01 06-06). Wired by `composition:durable-operation`.
|
||||
- The leaf name says "identifier"; general identifier generation, clock and random sources are not here.
|
||||
- The leaf name says "identifier"; general-purpose clock and random sources are not here.
|
||||
The name is kept because Gradle paths and consumers use it (README, review 05-21).
|
||||
- Wiring is `composition/service-runtime` (`PseudonymizationConfig`), not this leaf.
|
||||
|
||||
## Allowed
|
||||
|
||||
- `:application-core` (declared in [build.gradle](build.gradle)); the registry entry is the SSOT.
|
||||
Declared in [build.gradle](build.gradle); the registry entry is the SSOT.
|
||||
|
||||
- `:application-core` — the ports both classes implement.
|
||||
- `:shared-contract`, for `HmacSha256` only (review 2026-11-01 01-11). The pseudonym is the hex HMAC
|
||||
of the principal; `HmacUserPrincipalPseudonymizerVectorTest` pins it to a vector computed outside
|
||||
the JVM, because logs written before and after a release are joined on it.
|
||||
- `:domain-core` — a feature's server-assigned id adapter implements its domain `*IdFactory` port
|
||||
here (registry edge `adapter:outbound:identifier -> domain-core`; `docs/manual/02-new-feature.md`).
|
||||
- `com.github.f4b6a3:uuid-creator` (`libs.uuid.creator`) — UUIDv7 for
|
||||
`UuidV7DurableOperationIdFactory`, the generator the sample's id adapters use.
|
||||
|
||||
## Forbidden
|
||||
|
||||
@@ -42,8 +48,9 @@ build choices) live in [README.md](README.md).
|
||||
|
||||
## Test
|
||||
|
||||
Pure unit tests, no Spring context (`HmacUserPrincipalPseudonymizerTest`, including the RFC 4231
|
||||
known-answer vector).
|
||||
Pure unit tests, no Spring context: `HmacUserPrincipalPseudonymizerTest` (including the RFC 4231
|
||||
known-answer vector), `HmacUserPrincipalPseudonymizerVectorTest` and
|
||||
`UuidV7DurableOperationIdFactoryTest`.
|
||||
|
||||
```bash
|
||||
cd src
|
||||
|
||||
@@ -9,7 +9,8 @@
|
||||
## 이름과 내용이 다른 이유 (review 05-21)
|
||||
|
||||
leaf 이름은 `identifier`지만 범용 식별자 생성기·clock·random 소스는 여기 없다. 식별자 생성은
|
||||
durable operation id(UUIDv7) 하나뿐이다(아래). 예전 문서가 범용 생성기 계획을 현재 내용처럼 적었다. 이름은 바꾸지 않았다: Gradle 경로·registry id·소비자
|
||||
durable operation id(UUIDv7) 하나뿐이고(아래), 기능이 서버에서 id를 정하면 그 기능의 domain
|
||||
`*IdFactory` 구현도 여기 둔다(`:domain-core` 의존이 그 때문이다). 예전 문서가 범용 생성기 계획을 현재 내용처럼 적었다. 이름은 바꾸지 않았다: Gradle 경로·registry id·소비자
|
||||
(`composition:service-runtime`, `app-bootstrap`, `sample-portfolio`)의 의존 선언이 모두 이 이름을
|
||||
쓰고, 바꿔서 얻는 것은 이름 하나뿐이다. 외부 연동이 없는 crypto 능력을 새로 둘 곳이 필요하면 이
|
||||
leaf에 둔다. 그때 이 절을 고친다.
|
||||
@@ -25,10 +26,10 @@ leaf에 둔다. 그때 이 절을 고친다.
|
||||
대소문자를 바꾸면 테스트가 실패한다.
|
||||
- Salt: `APP_PRIVACY_PSEUDONYMIZATION_SALT`(secret, 90일 회전)에서 온다. 이 클래스는 salt를 스스로
|
||||
읽지 않고 생성자로 받는다. 받은 배열은 복사해 둔다.
|
||||
- Thread safety: `Mac`은 thread-safe하지 않아서 호출마다 새로 만든다. 그래서 싱글톤 bean으로
|
||||
공유해도 된다. `HmacSHA256`은 JDK 필수 알고리즘이라 `NoSuchAlgorithmException`·
|
||||
`InvalidKeyException`은 사실상 나지 않는다. checked exception을 호출부에 남기지 않으려고
|
||||
`IllegalStateException`으로 감싼다.
|
||||
- 서명: 이 클래스는 `Mac`을 직접 다루지 않는다. shared-contract의 `HmacSha256.sign`에 맡긴다(리뷰
|
||||
2026-11-01 01-11). `HmacSha256`이 호출마다 새 `Mac`을 만들고(`Mac`은 thread-safe하지 않다),
|
||||
JDK 필수 알고리즘이라 사실상 나지 않는 checked exception을 `IllegalStateException`으로 감싼다.
|
||||
그래서 이 클래스는 싱글톤 bean으로 공유해도 된다.
|
||||
|
||||
## UuidV7DurableOperationIdFactory
|
||||
|
||||
@@ -46,6 +47,10 @@ Spring 어노테이션이 없다. bean은
|
||||
|
||||
## 빌드 결정
|
||||
|
||||
의존은 넷이다: `:application-core`(두 클래스가 구현하는 port), `:shared-contract`(`HmacSha256`),
|
||||
`:domain-core`(기능의 `*IdFactory` 구현 자리), `uuid-creator`(UUIDv7). 목록과 이유는
|
||||
[CLAUDE.md](CLAUDE.md) Allowed 절이 기준이다.
|
||||
|
||||
`:application-core`를 `implementation`으로 받는다. 이 leaf의 public 타입 시그니처에
|
||||
application-core 타입이 드러나지만, 조립하는 쪽이 application-core를 이미 classpath에 갖고 있다.
|
||||
허용 edge는 `src/config/architecture/modules.json`이 기준이다.
|
||||
|
||||
+4
-2
@@ -1,5 +1,7 @@
|
||||
/**
|
||||
* The HMAC-SHA-256 user-principal pseudonymizer. Despite the leaf name, no identifier generation,
|
||||
* clock or random source lives here; see README for why the name was kept.
|
||||
* Two identifier adapters with no external IO: the HMAC-SHA-256 user-principal pseudonymizer and
|
||||
* the UUIDv7 durable operation id factory. A feature's server-assigned id adapter for a domain
|
||||
* {@code *IdFactory} port belongs here too. General clock and random sources do not live here; see
|
||||
* README for why the leaf name was kept.
|
||||
*/
|
||||
package dev.caskeleton.adapter.outbound.identifier;
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
# adapter:outbound:messaging-kafka
|
||||
|
||||
- Module ID `adapter-outbound-messaging-kafka`, Gradle path `:adapter:outbound:messaging-kafka`.
|
||||
Registry SSOT: `src/config/architecture/modules.json`.
|
||||
- Responsibility: the Kafka publishing capability as one `runtimeOnly` line. It has no production
|
||||
code: it bundles `adapter:outbound:messaging` (the application's bridge) with
|
||||
`messaging:messaging-spring-boot-starter-kafka` (the vendored platform and its Kafka transport) and
|
||||
ships the module defaults in `config/module-defaults/messaging.yml`.
|
||||
- Boundaries: add no code here. A publish rule belongs to the bridge (`adapter:outbound:messaging`),
|
||||
a Kafka rule to the vendored product (`src/messaging/CLAUDE.md`); a class here would be a third
|
||||
place. Allowed edges come only from the registry entry.
|
||||
- Tests: activation tests (`./gradlew :adapter:outbound:messaging-kafka:test`) and the real-Kafka
|
||||
lane `messagingKafkaRelayIntegrationTest` (relay → bridge → broker record bytes; `:integrationCheck`).
|
||||
- Design: `README.md`.
|
||||
+14
-6
@@ -8,6 +8,7 @@ import dev.caskeleton.application.outbox.OutboxAppendPort;
|
||||
import dev.caskeleton.application.outbox.OutboxCompletion;
|
||||
import dev.caskeleton.application.outbox.OutboxEventStatus;
|
||||
import dev.caskeleton.application.outbox.OutboxLease;
|
||||
import dev.caskeleton.application.outbox.OutboxLongRetries;
|
||||
import dev.caskeleton.application.outbox.OutboxStorePort;
|
||||
import dev.caskeleton.application.outbox.testkit.InMemoryOutboxStore;
|
||||
import dev.caskeleton.application.outbox.testkit.OutboxEvents;
|
||||
@@ -282,13 +283,15 @@ class ApplicationRelayKafkaIntegrationTest {
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized OutboxCompletion markFailed(OutboxLease lease, Instant nextAttemptAt) {
|
||||
return delegate.markFailed(lease, nextAttemptAt);
|
||||
public synchronized OutboxCompletion markFailed(
|
||||
OutboxLease lease, Instant nextAttemptAt, String failureCode) {
|
||||
return delegate.markFailed(lease, nextAttemptAt, failureCode);
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized OutboxCompletion markDeferred(OutboxLease lease, Instant nextAttemptAt) {
|
||||
return delegate.markDeferred(lease, nextAttemptAt);
|
||||
public synchronized OutboxCompletion markDeferred(
|
||||
OutboxLease lease, Instant nextAttemptAt, String failureCode) {
|
||||
return delegate.markDeferred(lease, nextAttemptAt, failureCode);
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -297,8 +300,13 @@ class ApplicationRelayKafkaIntegrationTest {
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized OutboxCompletion markDead(OutboxLease lease) {
|
||||
return delegate.markDead(lease);
|
||||
public synchronized OutboxCompletion markDead(OutboxLease lease, String failureCode) {
|
||||
return delegate.markDead(lease, failureCode);
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized OutboxLongRetries longRetries(int minDeferredClaims, Instant now) {
|
||||
return delegate.longRetries(minDeferredClaims, now);
|
||||
}
|
||||
|
||||
@Override
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
- Module ID: `adapter-outbound-messaging`
|
||||
- Gradle path: `:adapter:outbound:messaging`
|
||||
- Focused test (derived from Gradle path): `./gradlew :adapter:outbound:messaging:test --console=plain`
|
||||
- Runtime baseline: Java 21; repository framework baseline: Spring Boot 4.0.0.
|
||||
- Runtime baseline: Java 21; repository framework baseline: Spring Boot 4.0.8.
|
||||
- Registry SSOT: `src/config/architecture/modules.json`.
|
||||
|
||||
Package root: `dev.caskeleton.adapter.outbound.messaging`.
|
||||
|
||||
@@ -42,13 +42,33 @@ platform `REJECTED` 중 `FailureDescriptor.retryable()`이 true인 것(종료
|
||||
멈춘다(2026-11-01 리뷰 06-01). `AMBIGUOUS`로 번역하지 않은 이유: 동작은 같지만 "전송됐을 수
|
||||
있음"이라는 거짓 정보가 남아 중복 경고를 오염시킨다.
|
||||
|
||||
### 실패 원인 코드
|
||||
|
||||
결과(outcome)는 relay가 무엇을 할지만 정한다. 종료 중 거부, 닫힌 transport, 거부된 credential은 모두
|
||||
`REJECTED_RETRYABLE` 하나다. 그런데 이 셋은 끝이 없는 재시도다(시도 예산을 쓰지 않는다). 그래서 bridge는
|
||||
platform `FailureDescriptor.code()`를 버리지 않고 `OutboxPublishAttempt.failureCode()`로 함께 넘긴다
|
||||
(재리뷰 2026-11-03 RR-05-03). relay는 그 코드를 행의 `last_failure_code`(V22)와 실패 로그의
|
||||
`error.reason`에 남긴다. 운영자는 이 값으로 "credential이 진짜 틀렸다"와 "broker가 이 레코드만 받지
|
||||
않는다"를 가른다(`docs/runbooks/outbox-publish-failed.md` §4.1).
|
||||
|
||||
- platform code는 이미 정제된 이름이다(`KAFKA_AUTHENTICATIONEXCEPTION`, `PUBLISH_DEADLINE_EXCEEDED` 등).
|
||||
driver 메시지가 아니므로 그대로 옮긴다. application 코드 문법(`[A-Za-z0-9][A-Za-z0-9._:-]*`, 120자)에
|
||||
맞지 않는 code는 그 실패의 `FailureCategory` 이름으로 옮긴다.
|
||||
- platform으로 옮길 수 없는 event(아래 id 규칙)는 bridge가 `EVENT_NOT_REPRESENTABLE`로 거부한다.
|
||||
- 버린 대안: relay가 platform 타입을 직접 보는 것(경계가 깨진다), outcome 값을 원인별로 늘리는 것(relay가
|
||||
할 일은 같은데 분기만 는다).
|
||||
|
||||
### event id → message id
|
||||
|
||||
application `EventId`는 문자열이고 platform `MessageId`는 **UUIDv7**만 받는다. bridge는 event id가
|
||||
정규 소문자 UUIDv7 문자열일 때만 같은 값으로 옮기고, 그 밖(대문자·축약 표기·v4 등 다른 버전)은 다른
|
||||
문자열의 message id로 바꿔 보내지 않고 `REJECTED_BEFORE_SEND`로 거부한다(relay는 DEAD). 소비자 중복
|
||||
제거 키가 event id라서 값을 바꿔 보내면 중복 제거가 깨지기 때문이다(리뷰 2026-11-01 06-06). 이 거부는
|
||||
append 시점(`EventId` 문법)에서 먼저 일어나야 한다 — application 쪽 문법 일원화는 별도 작업이다.
|
||||
제거 키가 event id라서 값을 바꿔 보내면 중복 제거가 깨지기 때문이다(리뷰 2026-11-01 06-06).
|
||||
|
||||
같은 규칙이 append 쪽에 먼저 있다. `IntegrationEventDraft`가 정규 소문자 UUIDv7이 아닌 event id와
|
||||
causation id를 거부하므로(`IntegrationIdentity.isCanonicalUuidV7`), 그런 id는 업무 트랜잭션 안에서 실패한다
|
||||
(재리뷰 2026-11-03 RR-05-04). bridge의 거부는 이 규칙 전에 쓰인 행(저장 문법 `EventId`는 그대로 넓다)을 위한
|
||||
마지막 방어선이다.
|
||||
|
||||
## outbox 발행 포트에는 대역이 없다
|
||||
|
||||
@@ -70,9 +90,10 @@ event가 다시 publish될 수 있다. 같은 event id로 다시 나가므로 co
|
||||
있으면 `messaging-platform`이다. 이 adapter는 confirmed FAILED/DEAD report 하나를 SLF4J 2 fluent
|
||||
ERROR 하나로 렌더링한다.
|
||||
|
||||
공통 field는 `error.code`, `error.category`, `dependency_name`,
|
||||
공통 field는 `error.code`, `error.category`, `error.cause_type`, `error.reason`, `dependency_name`,
|
||||
`dependency_type=messaging`, `outcome`, `event_id`, `event_type`, `aggregate_id`,
|
||||
`correlation_id`, `attempt_count`, `runbook_link`다. retry report만 `next_attempt_at`을 추가한다.
|
||||
`error.reason`은 위 실패 원인 코드다.
|
||||
payload/envelope/exception-derived field는 받거나 렌더링하지 않는다. logging 내부
|
||||
`RuntimeException`은 adapter와 use case 양쪽에서 방어하므로 persisted FAILED/DEAD outcome을 바꾸지
|
||||
않는다.
|
||||
@@ -85,10 +106,8 @@ payload/envelope/exception-derived field는 받거나 렌더링하지 않는다.
|
||||
`envelope`·`realtime`)은 2026-09-25에 삭제됐고 복구 경로는 `docs/capability-status.md` 제거 표에
|
||||
있다.
|
||||
|
||||
공통 build evidence manifest 검증은 build-tools로 옮겼다. 루트 `verifyMessagingContracts`
|
||||
(`VerifyMessagingEvidenceTask`)가 application-core·shared-contract 두 qualification 레인 결과로
|
||||
`build/messaging-evidence/contracts-schema/manifest.json`을 쓰고, 쓴 바이트를 바로 common schema로
|
||||
검증한다. 이 leaf는 그 검증에 관여하지 않는다.
|
||||
integration event 계약 검증은 루트 `verifyMessagingContracts`(`src/build.gradle`)가 한다. application-core·
|
||||
shared-contract·sample-portfolio의 qualification lane 셋을 묶은 task다. 이 leaf는 그 검증에 관여하지 않는다.
|
||||
|
||||
canonical path는 platform `PublishResult`의 confirmation/transmission evidence를 application
|
||||
`OutboxPublishOutcome`으로 명시적으로 변환한다.
|
||||
`OutboxPublishOutcome`으로 명시적으로 변환하고, 실패 code를 `OutboxPublishAttempt`에 함께 싣는다.
|
||||
|
||||
+14
-4
@@ -4,17 +4,21 @@ import dev.caskeleton.application.messaging.event.IntegrationEventPublishPort;
|
||||
import dev.caskeleton.application.outbox.ClaimedOutboxEvent;
|
||||
import dev.caskeleton.application.outbox.OutboxMessagePublishPort;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishAmbiguousException;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishRefusedException;
|
||||
import java.time.Duration;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
|
||||
/**
|
||||
* The relay's publish port: a claimed row's validated event, delegated unchanged to the canonical
|
||||
* {@link IntegrationEventPublishPort}.
|
||||
*
|
||||
* <p>The event is the one the append validated, restored from its exact envelope bytes; nothing
|
||||
* here re-renders it. The platform's four-valued outcome is passed through, so the relay can tell a
|
||||
* possibly-stored message (ambiguous) from a refused one.
|
||||
* here re-renders it. The canonical publisher's attempt — its outcome and its failure code — is
|
||||
* passed through unchanged, so the relay can tell a possibly-stored message (ambiguous) from a
|
||||
* refused one, and a refused credential from a closed transport.
|
||||
*/
|
||||
public final class OutboxMessagePublishAdapter implements OutboxMessagePublishPort {
|
||||
|
||||
@@ -27,7 +31,7 @@ public final class OutboxMessagePublishAdapter implements OutboxMessagePublishPo
|
||||
|
||||
@Override
|
||||
public void publish(ClaimedOutboxEvent event) {
|
||||
OutboxPublishOutcome outcome = publishForOutcome(event);
|
||||
OutboxPublishOutcome outcome = publishForOutcome(event).outcome();
|
||||
if (outcome == OutboxPublishOutcome.CONFIRMED) {
|
||||
return;
|
||||
}
|
||||
@@ -37,8 +41,14 @@ public final class OutboxMessagePublishAdapter implements OutboxMessagePublishPo
|
||||
throw new OutboxPublishRefusedException(outcome);
|
||||
}
|
||||
|
||||
/** The canonical publisher's, unchanged: this adapter adds no wait of its own. */
|
||||
@Override
|
||||
public OutboxPublishOutcome publishForOutcome(ClaimedOutboxEvent event) {
|
||||
public Optional<Duration> longestAttempt() {
|
||||
return canonicalPublisher.longestAttempt();
|
||||
}
|
||||
|
||||
@Override
|
||||
public OutboxPublishAttempt publishForOutcome(ClaimedOutboxEvent event) {
|
||||
Objects.requireNonNull(event, "event must not be null");
|
||||
return canonicalPublisher.publish(event.event()).toCompletableFuture().join();
|
||||
}
|
||||
|
||||
+4
@@ -48,6 +48,10 @@ public final class Slf4jOutboxRelayFailureReportAdapter implements OutboxRelayFa
|
||||
// occasionally credentials. The class name is a type; the code is bounded.
|
||||
.addKeyValue("error.code", report.code().code())
|
||||
.addKeyValue("error.cause_type", report.causeType())
|
||||
// Why, in the publisher's words: the cause type of a returned outcome names only
|
||||
// which
|
||||
// outcome it was, and a refused credential and a closed transport are one outcome.
|
||||
.addKeyValue("error.reason", report.failureCode())
|
||||
.addKeyValue("error.category", report.code().category().name())
|
||||
.addKeyValue("dependency_name", dependencyName)
|
||||
.addKeyValue("dependency_type", DEPENDENCY_TYPE)
|
||||
|
||||
+60
-6
@@ -2,6 +2,7 @@ package dev.caskeleton.adapter.outbound.messaging.platformbridge;
|
||||
|
||||
import dev.caskeleton.application.messaging.event.IntegrationEventPublishPort;
|
||||
import dev.caskeleton.application.messaging.event.ValidatedIntegrationEvent;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
|
||||
import dev.caskeleton.messaging.api.CausationId;
|
||||
import dev.caskeleton.messaging.api.ContentType;
|
||||
@@ -26,6 +27,7 @@ import dev.caskeleton.messaging.api.publish.TransmissionEvidence;
|
||||
import dev.caskeleton.messaging.schema.EncodedMessage;
|
||||
import dev.caskeleton.messaging.schema.EncodedMessagePublisher;
|
||||
import dev.caskeleton.messaging.schema.SchemaReference;
|
||||
import java.time.Duration;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
@@ -45,6 +47,12 @@ import java.util.concurrent.CompletionStage;
|
||||
*/
|
||||
public final class PlatformIntegrationEventPublishAdapter implements IntegrationEventPublishPort {
|
||||
|
||||
/**
|
||||
* The failure code of an event this bridge refuses before the platform sees it: a value — an id
|
||||
* above all — the platform's types cannot carry.
|
||||
*/
|
||||
public static final String NOT_REPRESENTABLE = "EVENT_NOT_REPRESENTABLE";
|
||||
|
||||
private final EncodedMessagePublisher publisher;
|
||||
private final ProducerId producer;
|
||||
|
||||
@@ -55,7 +63,7 @@ public final class PlatformIntegrationEventPublishAdapter implements Integration
|
||||
}
|
||||
|
||||
@Override
|
||||
public CompletionStage<OutboxPublishOutcome> publish(ValidatedIntegrationEvent event) {
|
||||
public CompletionStage<OutboxPublishAttempt> publish(ValidatedIntegrationEvent event) {
|
||||
Objects.requireNonNull(event, "event must not be null");
|
||||
final PreparedPublish prepared;
|
||||
try {
|
||||
@@ -64,12 +72,27 @@ public final class PlatformIntegrationEventPublishAdapter implements Integration
|
||||
// Only a value the platform cannot represent is a definite refusal: the same payload is
|
||||
// refused again, so the relay dead-letters it at once. Any other failure is not evidence of
|
||||
// that and propagates, so the relay treats it as a retryable failure with its real cause.
|
||||
return CompletableFuture.completedFuture(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
|
||||
return CompletableFuture.completedFuture(
|
||||
OutboxPublishAttempt.failed(
|
||||
OutboxPublishOutcome.REJECTED_BEFORE_SEND, NOT_REPRESENTABLE));
|
||||
}
|
||||
|
||||
return publisher
|
||||
.publishEncoded(prepared.destination(), prepared.envelope(), PublishOptions.defaults())
|
||||
.thenApply(PlatformIntegrationEventPublishAdapter::mapOutcome);
|
||||
.thenApply(PlatformIntegrationEventPublishAdapter::translate);
|
||||
}
|
||||
|
||||
/**
|
||||
* Twice the publish deadline this bridge asks for ({@link PublishOptions#defaults()}).
|
||||
*
|
||||
* <p>The platform measures the deadline from the call, but a transport may block inside its send
|
||||
* before the deadline is applied to the stage it returns — Kafka's {@code max.block.ms} is set to
|
||||
* the same default for exactly that wait — so an attempt can take the deadline twice over. The
|
||||
* relay sizes its batch against this (review 2026-11-01 06-12).
|
||||
*/
|
||||
@Override
|
||||
public Optional<Duration> longestAttempt() {
|
||||
return Optional.of(PublishOptions.defaults().timeout().multipliedBy(2));
|
||||
}
|
||||
|
||||
private PreparedPublish prepare(ValidatedIntegrationEvent event) {
|
||||
@@ -146,8 +169,10 @@ public final class PlatformIntegrationEventPublishAdapter implements Integration
|
||||
* </ul>
|
||||
*
|
||||
* <p>Either refusal is {@code REJECTED_BEFORE_SEND}: the same id is refused again, so the relay
|
||||
* dead-letters it. The append side's {@code EventId} grammar is where such an id should fail
|
||||
* first, in the writer's transaction; until it does, this is where the mismatch is caught.
|
||||
* dead-letters it. A new event fails earlier, in the writer's transaction: {@code
|
||||
* IntegrationEventDraft} refuses an id that is not a canonical lower-case UUIDv7 (re-review
|
||||
* 2026-11-03 RR-05-04). This check stays for rows appended before that rule — the stored {@code
|
||||
* EventId} grammar is wider, so they can still be read back.
|
||||
*
|
||||
* @throws IllegalArgumentException when the value is not a canonical lower-case UUIDv7
|
||||
*/
|
||||
@@ -177,8 +202,37 @@ public final class PlatformIntegrationEventPublishAdapter implements Integration
|
||||
* committed event, and its aggregate behind it, for a cause that clears on its own. It gets its
|
||||
* own value instead.
|
||||
*/
|
||||
private static OutboxPublishOutcome mapOutcome(PublishResult result) {
|
||||
private static OutboxPublishAttempt translate(PublishResult result) {
|
||||
Objects.requireNonNull(result, "publish result must not be null");
|
||||
OutboxPublishOutcome outcome = mapOutcome(result);
|
||||
if (outcome == OutboxPublishOutcome.CONFIRMED) {
|
||||
return OutboxPublishAttempt.confirmed();
|
||||
}
|
||||
return result
|
||||
.failure()
|
||||
.map(PlatformIntegrationEventPublishAdapter::failureCode)
|
||||
.map(code -> OutboxPublishAttempt.failed(outcome, code))
|
||||
.orElseGet(() -> OutboxPublishAttempt.of(outcome));
|
||||
}
|
||||
|
||||
/**
|
||||
* The platform's failure code, carried as the application's failure code (re-review 2026-11-03
|
||||
* RR-05-03).
|
||||
*
|
||||
* <p>The outcome is all the relay acts on, and it loses the reason on purpose: a shutdown, a
|
||||
* closed transport and a refused credential are one {@code REJECTED_RETRYABLE}. The code keeps it
|
||||
* for the operator, on the row and in the failure log. The platform's code is already sanitized —
|
||||
* a stable name, never a driver message — and passes as it is; one outside the application's code
|
||||
* grammar (the platform only bounds its length) is carried as the failure's category instead, a
|
||||
* name the platform owns too.
|
||||
*/
|
||||
private static String failureCode(FailureDescriptor failure) {
|
||||
return OutboxPublishAttempt.isFailureCode(failure.code())
|
||||
? failure.code()
|
||||
: failure.category().name();
|
||||
}
|
||||
|
||||
private static OutboxPublishOutcome mapOutcome(PublishResult result) {
|
||||
if (result.completion() == PublishCompletion.CONFIRMED) {
|
||||
return OutboxPublishOutcome.CONFIRMED;
|
||||
}
|
||||
|
||||
+2
-2
@@ -5,7 +5,7 @@ import static org.assertj.core.api.Assertions.assertThat;
|
||||
import dev.caskeleton.adapter.outbound.messaging.outbox.OutboxMessagePublishAdapter;
|
||||
import dev.caskeleton.application.messaging.event.IntegrationEventPublishPort;
|
||||
import dev.caskeleton.application.outbox.OutboxMessagePublishPort;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
|
||||
@@ -32,7 +32,7 @@ class MessagingConfigTest {
|
||||
runner
|
||||
.withBean(
|
||||
IntegrationEventPublishPort.class,
|
||||
() -> event -> CompletableFuture.completedFuture(OutboxPublishOutcome.CONFIRMED))
|
||||
() -> event -> CompletableFuture.completedFuture(OutboxPublishAttempt.confirmed()))
|
||||
.run(
|
||||
context ->
|
||||
assertThat(context.getBean(OutboxMessagePublishPort.class))
|
||||
|
||||
+12
-3
@@ -51,7 +51,14 @@ class Slf4jOutboxRelayFailureReportAdapterTest {
|
||||
|
||||
adapter.report(
|
||||
OutboxRelayFailureReport.retryableFailure(
|
||||
"evt-1", "WorkLogReserved", "agg-1", "corr-1", 2, nextAttemptAt, cause));
|
||||
"evt-1",
|
||||
"WorkLogReserved",
|
||||
"agg-1",
|
||||
"corr-1",
|
||||
2,
|
||||
nextAttemptAt,
|
||||
cause,
|
||||
"KAFKA_AUTHENTICATIONEXCEPTION"));
|
||||
|
||||
assertThat(appender.list).hasSize(1);
|
||||
ILoggingEvent event = appender.list.getFirst();
|
||||
@@ -71,6 +78,7 @@ class Slf4jOutboxRelayFailureReportAdapterTest {
|
||||
Map.entry("correlation_id", "corr-1"),
|
||||
Map.entry("attempt_count", 2),
|
||||
Map.entry("error.cause_type", "java.lang.RuntimeException"),
|
||||
Map.entry("error.reason", "KAFKA_AUTHENTICATIONEXCEPTION"),
|
||||
Map.entry("runbook_link", "runbook://outbox/publish-failed"),
|
||||
Map.entry("next_attempt_at", "2026-07-25T01:02:03Z")));
|
||||
assertThat(event.getThrowableProxy())
|
||||
@@ -89,7 +97,7 @@ class Slf4jOutboxRelayFailureReportAdapterTest {
|
||||
|
||||
adapter.report(
|
||||
OutboxRelayFailureReport.deadLetter(
|
||||
"evt-2", "WorkLogReserved", "agg-2", "corr-2", 3, cause));
|
||||
"evt-2", "WorkLogReserved", "agg-2", "corr-2", 3, cause, "REJECTED_BEFORE_SEND"));
|
||||
|
||||
assertThat(appender.list).hasSize(1);
|
||||
ILoggingEvent event = appender.list.getFirst();
|
||||
@@ -142,7 +150,8 @@ class Slf4jOutboxRelayFailureReportAdapterTest {
|
||||
"agg-3",
|
||||
"corr-3",
|
||||
3,
|
||||
new RuntimeException("broker down"))))
|
||||
new RuntimeException("broker down"),
|
||||
"RuntimeException")))
|
||||
.doesNotThrowAnyException();
|
||||
}
|
||||
|
||||
|
||||
+104
-9
@@ -12,6 +12,7 @@ import dev.caskeleton.application.messaging.event.IntegrationEventPublishPort;
|
||||
import dev.caskeleton.application.messaging.event.ValidatedIntegrationEvent;
|
||||
import dev.caskeleton.application.messaging.event.testkit.BrokerAnswer;
|
||||
import dev.caskeleton.application.messaging.event.testkit.IntegrationEventPublishContract;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
|
||||
import dev.caskeleton.messaging.api.ContentType;
|
||||
import dev.caskeleton.messaging.api.MessageEnvelope;
|
||||
@@ -59,7 +60,7 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
|
||||
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
|
||||
ValidatedIntegrationEvent event = event(EVENT_ID, Optional.of(CAUSATION_ID));
|
||||
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join())
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
|
||||
.isEqualTo(OutboxPublishOutcome.CONFIRMED);
|
||||
|
||||
MessageDestination<EncodedMessage> destination = publisher.destination;
|
||||
@@ -99,19 +100,19 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
|
||||
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
|
||||
ValidatedIntegrationEvent event = event(EVENT_ID, Optional.empty());
|
||||
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join())
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
|
||||
.isEqualTo(OutboxPublishOutcome.CONFIRMED);
|
||||
publisher.result = ambiguous();
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join())
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
|
||||
.isEqualTo(OutboxPublishOutcome.AMBIGUOUS);
|
||||
publisher.result = rejected(PublishEvidence.notTransmitted());
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join())
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
|
||||
.isEqualTo(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
|
||||
publisher.result =
|
||||
rejected(
|
||||
new PublishEvidence(
|
||||
true, TransmissionEvidence.TRANSMITTED, false, ConfirmationLevel.NONE));
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join())
|
||||
assertThat(bridge.publish(event).toCompletableFuture().join().outcome())
|
||||
.isEqualTo(OutboxPublishOutcome.REJECTED_AFTER_BROKER);
|
||||
}
|
||||
|
||||
@@ -132,12 +133,101 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
|
||||
|
||||
// when
|
||||
OutboxPublishOutcome outcome =
|
||||
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join();
|
||||
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join().outcome();
|
||||
|
||||
// then
|
||||
assertThat(outcome).isEqualTo(OutboxPublishOutcome.REJECTED_RETRYABLE);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("platform이 붙인 실패 코드는 원인 코드로 그대로 옮긴다")
|
||||
void thePlatformsFailureCodeIsCarriedAsTheFailureCode() {
|
||||
// given: the broker refused this producer's credentials
|
||||
RecordingPublisher publisher =
|
||||
new RecordingPublisher(
|
||||
PublishResult.rejected(
|
||||
Duration.ZERO,
|
||||
new FailureDescriptor(
|
||||
FailureCategory.AUTHENTICATION,
|
||||
"KAFKA_AUTHENTICATIONEXCEPTION",
|
||||
true,
|
||||
"kafka publish failed with AuthenticationException",
|
||||
Optional.of("AuthenticationException"))));
|
||||
PlatformIntegrationEventPublishAdapter bridge =
|
||||
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
|
||||
|
||||
// when
|
||||
OutboxPublishAttempt attempt =
|
||||
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join();
|
||||
|
||||
// then
|
||||
assertThat(attempt)
|
||||
.isEqualTo(
|
||||
OutboxPublishAttempt.failed(
|
||||
OutboxPublishOutcome.REJECTED_RETRYABLE, "KAFKA_AUTHENTICATIONEXCEPTION"));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("코드 문법에 맞지 않는 platform 실패 코드는 그 실패의 분류 이름으로 옮긴다")
|
||||
void aPlatformCodeOutsideTheCodeGrammarIsCarriedAsItsCategory() {
|
||||
// given
|
||||
RecordingPublisher publisher =
|
||||
new RecordingPublisher(
|
||||
PublishResult.ambiguous(
|
||||
Duration.ZERO,
|
||||
FailureDescriptor.of(
|
||||
FailureCategory.AMBIGUOUS, "timed out after 5000 ms", "no answer")));
|
||||
PlatformIntegrationEventPublishAdapter bridge =
|
||||
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
|
||||
|
||||
// when
|
||||
OutboxPublishAttempt attempt =
|
||||
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join();
|
||||
|
||||
// then
|
||||
assertThat(attempt.failureCode()).contains("AMBIGUOUS");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("platform으로 옮길 수 없는 event는 그 이유를 원인 코드로 알린다")
|
||||
void anEventThePlatformCannotRepresentIsRefusedWithItsOwnFailureCode() {
|
||||
// given
|
||||
RecordingPublisher publisher = new RecordingPublisher(confirmed());
|
||||
PlatformIntegrationEventPublishAdapter bridge =
|
||||
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
|
||||
|
||||
// when
|
||||
OutboxPublishAttempt attempt =
|
||||
bridge.publish(event("event-1", Optional.empty())).toCompletableFuture().join();
|
||||
|
||||
// then
|
||||
assertThat(attempt)
|
||||
.isEqualTo(
|
||||
OutboxPublishAttempt.failed(
|
||||
OutboxPublishOutcome.REJECTED_BEFORE_SEND,
|
||||
PlatformIntegrationEventPublishAdapter.NOT_REPRESENTABLE));
|
||||
}
|
||||
|
||||
/**
|
||||
* The relay sizes its batch against this (review 2026-11-01 06-12). The platform's deadline is
|
||||
* measured from the call, but a transport may block in its send before the deadline is applied —
|
||||
* Kafka's {@code max.block.ms} is set to the same default — so an attempt can take twice it.
|
||||
*/
|
||||
@Test
|
||||
@DisplayName("발행 하나의 최악 시간은 platform 기본 발행 기한의 두 배다")
|
||||
void theLongestAttemptIsTwiceThePlatformsDefaultPublishDeadline() {
|
||||
// given
|
||||
PlatformIntegrationEventPublishAdapter bridge =
|
||||
new PlatformIntegrationEventPublishAdapter(
|
||||
new RecordingPublisher(confirmed()), "portfolio-service");
|
||||
|
||||
// when
|
||||
Optional<Duration> longest = bridge.longestAttempt();
|
||||
|
||||
// then
|
||||
assertThat(longest).contains(PublishOptions.defaults().timeout().multipliedBy(2));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("다시 시도해도 같은 결과인 거부는 지금처럼 확정 거부로 번역한다")
|
||||
void aRefusalThePlatformMarksFinalStaysADefiniteRejection() {
|
||||
@@ -153,7 +243,7 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
|
||||
|
||||
// when
|
||||
OutboxPublishOutcome outcome =
|
||||
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join();
|
||||
bridge.publish(event(EVENT_ID, Optional.empty())).toCompletableFuture().join().outcome();
|
||||
|
||||
// then
|
||||
assertThat(outcome).isEqualTo(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
|
||||
@@ -165,7 +255,12 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
|
||||
PlatformIntegrationEventPublishAdapter bridge =
|
||||
new PlatformIntegrationEventPublishAdapter(publisher, "portfolio-service");
|
||||
|
||||
assertThat(bridge.publish(event("event-1", Optional.empty())).toCompletableFuture().join())
|
||||
assertThat(
|
||||
bridge
|
||||
.publish(event("event-1", Optional.empty()))
|
||||
.toCompletableFuture()
|
||||
.join()
|
||||
.outcome())
|
||||
.isEqualTo(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
|
||||
assertThat(publisher.calls).isZero();
|
||||
}
|
||||
@@ -192,7 +287,7 @@ class PlatformIntegrationEventPublishAdapterTest extends IntegrationEventPublish
|
||||
|
||||
// when
|
||||
OutboxPublishOutcome outcome =
|
||||
bridge.publish(event(eventId, Optional.empty())).toCompletableFuture().join();
|
||||
bridge.publish(event(eventId, Optional.empty())).toCompletableFuture().join().outcome();
|
||||
|
||||
// then
|
||||
assertThat(outcome).isEqualTo(OutboxPublishOutcome.REJECTED_BEFORE_SEND);
|
||||
|
||||
+26
-4
@@ -64,17 +64,39 @@ class NotificationCallbackProviderConfig {
|
||||
return cachingCertificates(uri -> fetchCertificate(client, uri));
|
||||
}
|
||||
|
||||
private static byte[] fetchCertificate(HttpClient client, URI uri) {
|
||||
/**
|
||||
* Fetches the certificate document at a host-checked SNS URL.
|
||||
*
|
||||
* <p>The status decides what a failed fetch says. The URL already passed the SNS host check, so
|
||||
* the host answering is SNS. A 4xx is SNS saying it has no such certificate: a real SNS message
|
||||
* names one that exists, so the envelope is forged — {@link IllegalArgumentException}, which the
|
||||
* verifier turns into an invalid signature that is refused and audited. It used to be "cannot
|
||||
* verify now" like every other non-2xx, so a forged envelope pointing at a certificate SNS never
|
||||
* issued got a 503, was never audited, and was logged as a provider hiccup (review 2026-11-03
|
||||
* RR-07-03). A 5xx, a 408 or 429 (slow or throttling host), or a network failure says nothing
|
||||
* about the certificate and stays "cannot verify now".
|
||||
*
|
||||
* @param client the no-redirect client
|
||||
* @param uri the host-checked certificate URL
|
||||
* @return the certificate document, at most {@link #MAX_SNS_CERTIFICATE_BYTES}
|
||||
* @throws IllegalArgumentException when SNS answers that the certificate does not exist
|
||||
* @throws CallbackVerificationUnavailableException when the certificate could not be fetched now
|
||||
*/
|
||||
static byte[] fetchCertificate(HttpClient client, URI uri) {
|
||||
HttpRequest request =
|
||||
HttpRequest.newBuilder(uri).timeout(SNS_CERTIFICATE_TIMEOUT).GET().build();
|
||||
try {
|
||||
HttpResponse<InputStream> response =
|
||||
client.send(request, HttpResponse.BodyHandlers.ofInputStream());
|
||||
try (InputStream body = response.body()) {
|
||||
if (response.statusCode() / 100 != 2) {
|
||||
int status = response.statusCode();
|
||||
if (status / 100 == 4 && status != 408 && status != 429) {
|
||||
throw new IllegalArgumentException(
|
||||
"SNS has no signing certificate at the URL the message names (HTTP " + status + ")");
|
||||
}
|
||||
if (status / 100 != 2) {
|
||||
throw certificateUnavailable(
|
||||
new IOException(
|
||||
"SNS signing certificate fetch failed with HTTP " + response.statusCode()));
|
||||
new IOException("SNS signing certificate fetch failed with HTTP " + status));
|
||||
}
|
||||
return body.readNBytes(MAX_SNS_CERTIFICATE_BYTES);
|
||||
}
|
||||
|
||||
+11
-3
@@ -16,6 +16,7 @@ import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
import java.util.Set;
|
||||
import tools.jackson.databind.JsonNode;
|
||||
|
||||
@@ -36,8 +37,12 @@ import tools.jackson.databind.JsonNode;
|
||||
* signed string), for up to {@link #SNS_HTTP_RETRY_HORIZON}. A window of five minutes threw away
|
||||
* every bounce and complaint that arrived after an outage longer than that, and with them the
|
||||
* suppressions they should have created (review 05-20). Accepting a replay inside the window costs
|
||||
* nothing: the provider event ledger is unique on {@code (provider_profile_id, provider_event_id)},
|
||||
* so a second delivery of the same event is a duplicate, not a second event.
|
||||
* nothing: the provider event ledger is unique on {@code (provider_profile_id, provider_event_id)}
|
||||
* whenever an event id is present, and this adapter gives every SES event the signed SNS {@code
|
||||
* MessageId} as that id, so a second delivery of the same message is a duplicate, not a second
|
||||
* event. It used to give none, and the only identity left was the {@code (provider_profile_id,
|
||||
* event_fingerprint)} index — a fingerprint keyed by the active HMAC key, so a redelivery after a
|
||||
* key rotation inside the window was recorded twice (review 2026-11-03 RR-07-07).
|
||||
*/
|
||||
public final class SesCallbackAdapter implements ProviderCallbackAdapter {
|
||||
|
||||
@@ -118,7 +123,10 @@ public final class SesCallbackAdapter implements ProviderCallbackAdapter {
|
||||
return List.of();
|
||||
}
|
||||
String message = envelope.getOrDefault("Message", "{}");
|
||||
return normalizer.normalize(message);
|
||||
// The SNS MessageId is in the signed string and survives every redelivery, so it — not a
|
||||
// fingerprint under the current HMAC key — is what the ledger recognises a replay by.
|
||||
return normalizer.normalize(
|
||||
message, Optional.ofNullable(envelope.get("MessageId")).filter(id -> !id.isBlank()));
|
||||
}
|
||||
|
||||
private boolean withinReplayWindow(String timestamp, Instant receivedAt) {
|
||||
|
||||
+44
-14
@@ -7,6 +7,7 @@ import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
import tools.jackson.databind.JsonNode;
|
||||
|
||||
@@ -19,8 +20,16 @@ import tools.jackson.databind.JsonNode;
|
||||
*/
|
||||
public final class SesEventNormalizer {
|
||||
|
||||
/** Normalize one SES notification payload. */
|
||||
public List<NormalizedProviderEvent> normalize(String payload) {
|
||||
/**
|
||||
* Normalize one SES notification payload.
|
||||
*
|
||||
* @param payload the SES event JSON (the SNS envelope's {@code Message})
|
||||
* @param providerEventId the identity of the delivery that carried it — the signed SNS {@code
|
||||
* MessageId}, which SNS keeps across redeliveries — or empty when the envelope had none
|
||||
* @return the provider events, each carrying {@code providerEventId}
|
||||
*/
|
||||
public List<NormalizedProviderEvent> normalize(String payload, Optional<String> providerEventId) {
|
||||
Objects.requireNonNull(providerEventId, "providerEventId");
|
||||
JsonNode root = NotificationJsonMapper.mapper().readTree(payload);
|
||||
String type = text(root, "eventType").orElse(text(root, "notificationType").orElse("Unknown"));
|
||||
Optional<String> messageId =
|
||||
@@ -30,24 +39,26 @@ public final class SesEventNormalizer {
|
||||
List<NormalizedProviderEvent> events = new ArrayList<>(1);
|
||||
events.add(
|
||||
switch (type) {
|
||||
case "Send" ->
|
||||
event(NormalizedEventType.PROVIDER_ACCEPTED, type, messageId, occurredAt, Map.of());
|
||||
case "Send" -> event(NormalizedEventType.PROVIDER_ACCEPTED, type, messageId, occurredAt);
|
||||
case "Delivery" ->
|
||||
event(NormalizedEventType.DELIVERY_CONFIRMED, type, messageId, occurredAt, Map.of());
|
||||
event(NormalizedEventType.DELIVERY_CONFIRMED, type, messageId, occurredAt);
|
||||
case "DeliveryDelay" ->
|
||||
event(NormalizedEventType.DELIVERY_DELAYED, type, messageId, occurredAt, Map.of());
|
||||
event(NormalizedEventType.DELIVERY_DELAYED, type, messageId, occurredAt);
|
||||
case "Bounce" -> bounce(root, type, messageId, occurredAt);
|
||||
case "Complaint" ->
|
||||
event(NormalizedEventType.COMPLAINT, type, messageId, occurredAt, Map.of());
|
||||
case "Complaint" -> event(NormalizedEventType.COMPLAINT, type, messageId, occurredAt);
|
||||
case "Reject" ->
|
||||
event(NormalizedEventType.PROVIDER_REJECTED, type, messageId, occurredAt, Map.of());
|
||||
event(NormalizedEventType.PROVIDER_REJECTED, type, messageId, occurredAt);
|
||||
case "RenderingFailure" ->
|
||||
event(NormalizedEventType.TEMPLATE_FAILURE, type, messageId, occurredAt, Map.of());
|
||||
case "Open" -> event(NormalizedEventType.OPENED, type, messageId, occurredAt, Map.of());
|
||||
case "Click" -> event(NormalizedEventType.CLICKED, type, messageId, occurredAt, Map.of());
|
||||
default -> event(NormalizedEventType.UNKNOWN, type, messageId, occurredAt, Map.of());
|
||||
event(NormalizedEventType.TEMPLATE_FAILURE, type, messageId, occurredAt);
|
||||
case "Open" -> event(NormalizedEventType.OPENED, type, messageId, occurredAt);
|
||||
case "Click" -> event(NormalizedEventType.CLICKED, type, messageId, occurredAt);
|
||||
default -> event(NormalizedEventType.UNKNOWN, type, messageId, occurredAt);
|
||||
});
|
||||
return List.copyOf(events);
|
||||
// One SNS delivery carries one SES event, so the delivery's id is the event's. It is what makes
|
||||
// a redelivery a duplicate in the ledger: the other identity, the event fingerprint, is keyed
|
||||
// by
|
||||
// the active HMAC key and changes when the key rotates (review 2026-11-03 RR-07-07).
|
||||
return events.stream().map(event -> withEventId(event, providerEventId)).toList();
|
||||
}
|
||||
|
||||
private static NormalizedProviderEvent bounce(
|
||||
@@ -68,6 +79,14 @@ public final class SesEventNormalizer {
|
||||
Map.of("bounceType", bounceType));
|
||||
}
|
||||
|
||||
private static NormalizedProviderEvent event(
|
||||
NormalizedEventType type,
|
||||
String nativeType,
|
||||
Optional<String> messageId,
|
||||
Optional<Instant> occurredAt) {
|
||||
return event(type, nativeType, messageId, occurredAt, Map.of());
|
||||
}
|
||||
|
||||
private static NormalizedProviderEvent event(
|
||||
NormalizedEventType type,
|
||||
String nativeType,
|
||||
@@ -78,6 +97,17 @@ public final class SesEventNormalizer {
|
||||
type, nativeType, Optional.empty(), messageId, occurredAt, attributes);
|
||||
}
|
||||
|
||||
private static NormalizedProviderEvent withEventId(
|
||||
NormalizedProviderEvent event, Optional<String> providerEventId) {
|
||||
return new NormalizedProviderEvent(
|
||||
event.type(),
|
||||
event.providerNativeType(),
|
||||
providerEventId,
|
||||
event.providerRequestId(),
|
||||
event.providerOccurredAt(),
|
||||
event.attributes());
|
||||
}
|
||||
|
||||
private static Optional<String> text(JsonNode node, String field) {
|
||||
JsonNode value = node.get(field);
|
||||
return value == null || value.isNull() ? Optional.empty() : Optional.of(value.asString());
|
||||
|
||||
+6
-1
@@ -112,7 +112,12 @@ public final class JavaMailSenderSmtpDispatch implements SmtpDispatch {
|
||||
} else if (cause instanceof SMTPSendFailedException send) {
|
||||
code = send.getReturnCode();
|
||||
}
|
||||
return code >= 200 && code < 600 ? Optional.of(code) : Optional.empty();
|
||||
// Only a refusal is a reply code here: 4xx (transient) and 5xx (permanent). Angus's sendpartial
|
||||
// path throws SMTPSendFailedException carrying the last reply, which can be a success (250);
|
||||
// read as a reply code it reached the classifier as a permanent refusal of a message the relay
|
||||
// had partly accepted. Anything else takes the conservative "may have been delivered" path
|
||||
// below (review 2026-11-03 RR-04-11).
|
||||
return code >= 400 && code < 600 ? Optional.of(code) : Optional.empty();
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+104
@@ -0,0 +1,104 @@
|
||||
package dev.caskeleton.adapter.outbound.notification.platform.autoconfigure;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import com.sun.net.httpserver.HttpServer;
|
||||
import dev.caskeleton.adapter.outbound.notification.platform.callback.CallbackVerificationUnavailableException;
|
||||
import java.io.IOException;
|
||||
import java.net.InetAddress;
|
||||
import java.net.InetSocketAddress;
|
||||
import java.net.ServerSocket;
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpClient;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
|
||||
/**
|
||||
* What a failed SNS signing-certificate fetch says about the callback (review 2026-11-03 RR-07-03).
|
||||
*
|
||||
* <p>The URL has passed the SNS host check before it is fetched, so the host is SNS's. A real SNS
|
||||
* message names a certificate that exists; when SNS itself answers 4xx for it, the envelope named a
|
||||
* certificate SNS never issued — a forged callback, to be rejected and audited. Only a failure that
|
||||
* says nothing about the certificate — a 5xx, a 408 or 429 (the host is slow or throttling), or a
|
||||
* network failure — is "cannot verify now" (503, the provider retries).
|
||||
*/
|
||||
class SnsCertificateFetchTest {
|
||||
|
||||
private final HttpClient client = HttpClient.newHttpClient();
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(ints = {403, 404, 410})
|
||||
@DisplayName("인증서 URL이 4xx로 답하면 없는 인증서를 가리킨 위조 콜백이므로 검증 실패로 거절된다")
|
||||
void aCertificateTheHostDoesNotHaveFailsVerification(int status) throws IOException {
|
||||
// given
|
||||
HttpServer server = answering(status);
|
||||
try {
|
||||
URI url = URI.create("http://127.0.0.1:" + server.getAddress().getPort() + "/cert.pem");
|
||||
|
||||
// when
|
||||
Throwable failure =
|
||||
catchThrowable(() -> NotificationCallbackProviderConfig.fetchCertificate(client, url));
|
||||
|
||||
// then
|
||||
assertThat(failure)
|
||||
.isInstanceOf(IllegalArgumentException.class)
|
||||
.isNotInstanceOf(CallbackVerificationUnavailableException.class);
|
||||
} finally {
|
||||
server.stop(0);
|
||||
}
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(ints = {408, 429, 500, 503})
|
||||
@DisplayName("인증서 URL이 5xx나 시간 초과·요청 제한(408·429)으로 답하면 아무것도 모르므로 지금 검증할 수 없다고 답한다")
|
||||
void aFailingCertificateHostLeavesTheCallbackUnverified(int status) throws IOException {
|
||||
// given
|
||||
HttpServer server = answering(status);
|
||||
try {
|
||||
URI url = URI.create("http://127.0.0.1:" + server.getAddress().getPort() + "/cert.pem");
|
||||
|
||||
// when
|
||||
Throwable failure =
|
||||
catchThrowable(() -> NotificationCallbackProviderConfig.fetchCertificate(client, url));
|
||||
|
||||
// then
|
||||
assertThat(failure).isInstanceOf(CallbackVerificationUnavailableException.class);
|
||||
} finally {
|
||||
server.stop(0);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("인증서 호스트에 닿지 못하면 지금 검증할 수 없다고 답한다")
|
||||
void anUnreachableCertificateHostLeavesTheCallbackUnverified() throws IOException {
|
||||
// given
|
||||
int closedPort;
|
||||
try (ServerSocket probe = new ServerSocket(0)) {
|
||||
closedPort = probe.getLocalPort();
|
||||
}
|
||||
URI url = URI.create("http://127.0.0.1:" + closedPort + "/cert.pem");
|
||||
|
||||
// when
|
||||
Throwable failure =
|
||||
catchThrowable(() -> NotificationCallbackProviderConfig.fetchCertificate(client, url));
|
||||
|
||||
// then
|
||||
assertThat(failure).isInstanceOf(CallbackVerificationUnavailableException.class);
|
||||
}
|
||||
|
||||
private static HttpServer answering(int status) throws IOException {
|
||||
HttpServer server =
|
||||
HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress(), 0), 0);
|
||||
server.createContext(
|
||||
"/",
|
||||
exchange -> {
|
||||
exchange.sendResponseHeaders(status, -1);
|
||||
exchange.close();
|
||||
});
|
||||
server.start();
|
||||
return server;
|
||||
}
|
||||
}
|
||||
+24
@@ -8,6 +8,7 @@ import dev.caskeleton.adapter.outbound.notification.platform.callback.CallbackVe
|
||||
import dev.caskeleton.adapter.outbound.notification.platform.callback.CallbackVerificationUnavailableException;
|
||||
import dev.caskeleton.adapter.outbound.notification.platform.callback.ProviderCallbackVerificationRequest;
|
||||
import dev.caskeleton.adapter.outbound.notification.platform.template.NotificationJsonMapper;
|
||||
import dev.caskeleton.application.notification.platform.api.NormalizedProviderEvent;
|
||||
import dev.caskeleton.application.notification.platform.api.ProviderId;
|
||||
import dev.caskeleton.application.notification.platform.api.ProviderProfileId;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
@@ -19,6 +20,7 @@ import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.Base64;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.Set;
|
||||
@@ -135,6 +137,28 @@ class SesCallbackAdapterTest {
|
||||
assertThat(unavailable.reasonCode()).isEqualTo("SNS_CERTIFICATE_UNAVAILABLE"));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName(
|
||||
"SES 이벤트의 provider event id는 서명된 SNS MessageId라 fingerprint 키를 회전해도 같은 콜백이 같은 이벤트로 남는다")
|
||||
void theSignedSnsMessageIdIsTheProviderEventId() {
|
||||
// given
|
||||
CallbackVerificationResult verified =
|
||||
adapter.verify(request(signed(ALLOWED_TOPIC, RECEIVED_AT)));
|
||||
|
||||
// when
|
||||
List<NormalizedProviderEvent> events =
|
||||
adapter.normalize(verified.verifiedCallback().orElseThrow());
|
||||
|
||||
// then
|
||||
// The ledger is unique on (profile, provider_event_id) whenever an id is present. Without one
|
||||
// the only identity was a fingerprint keyed by the active HMAC key, so a redelivery after a
|
||||
// key rotation inside the one-hour replay window became a second row (RR-07-07).
|
||||
assertThat(events)
|
||||
.singleElement()
|
||||
.extracting(NormalizedProviderEvent::providerEventId)
|
||||
.isEqualTo(Optional.of("sns-message-1"));
|
||||
}
|
||||
|
||||
/** The adapter with the window a deployment gets when it sets nothing. */
|
||||
private static SesCallbackAdapter withDefaultWindow() {
|
||||
return new SesCallbackAdapter(
|
||||
|
||||
+34
@@ -16,9 +16,11 @@ import java.nio.charset.StandardCharsets;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import java.util.Properties;
|
||||
import org.eclipse.angus.mail.smtp.SMTPSendFailedException;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.mail.MailAuthenticationException;
|
||||
import org.springframework.mail.MailSendException;
|
||||
import org.springframework.mail.javamail.JavaMailSenderImpl;
|
||||
|
||||
/**
|
||||
@@ -136,6 +138,38 @@ class JavaMailSenderSmtpDispatchTest {
|
||||
.containsExactly(Optional.empty(), false);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("성공 코드(250)를 실은 부분 전송 실패는 거절 reply code가 아니라 전달됐을 수 있는 실패로 보고된다")
|
||||
void aPartialSendCarryingASuccessCodeIsNotReportedAsARefusal() {
|
||||
// given
|
||||
// Angus's sendpartial path reports a partly delivered message as SMTPSendFailedException
|
||||
// carrying the last reply, which is a success (250), not a refusal.
|
||||
JavaMailSenderImpl partlyDelivered =
|
||||
new JavaMailSenderImpl() {
|
||||
@Override
|
||||
public void send(MimeMessage... messages) {
|
||||
throw new MailSendException(
|
||||
"partly sent",
|
||||
new SMTPSendFailedException(
|
||||
"DATA",
|
||||
250,
|
||||
"250 2.0.0 queued",
|
||||
null,
|
||||
new jakarta.mail.Address[0],
|
||||
new jakarta.mail.Address[0],
|
||||
new jakarta.mail.Address[0]));
|
||||
}
|
||||
};
|
||||
JavaMailSenderSmtpDispatch dispatch = new JavaMailSenderSmtpDispatch(partlyDelivered);
|
||||
|
||||
// when
|
||||
SmtpDispatchException failure = sendExpectingFailure(dispatch);
|
||||
|
||||
// then
|
||||
assertThat(List.of(failure.replyCode(), failure.dataCommitted()))
|
||||
.containsExactly(Optional.empty(), true);
|
||||
}
|
||||
|
||||
private static SmtpDispatchException sendExpectingFailure(JavaMailSenderSmtpDispatch dispatch) {
|
||||
return catchThrowableOfType(SmtpDispatchException.class, () -> dispatch.send(message()));
|
||||
}
|
||||
|
||||
@@ -155,9 +155,9 @@ strictTestLanes {
|
||||
}
|
||||
}
|
||||
|
||||
// The MinIO lanes start a container, so they are the main-branch stage of this leaf, not its
|
||||
// pull-request `check` (review BUILD-계획-B3); object-storage-qualification.yml runs them on pull
|
||||
// requests that touch object storage, and integrationCheck runs them with every integration lane.
|
||||
tasks.named('mainCheck') {
|
||||
dependsOn 'objectStorageMinioContractTest', 'objectStorageMinioFaultTest'
|
||||
}
|
||||
// The MinIO lanes start a container, so they are not this leaf's pull-request `check` (review
|
||||
// BUILD-계획-B3). They are integration lanes, and the root integrationCheck owns every container lane
|
||||
// on main and nightly; object-storage-qualification.yml runs them on pull requests that touch object
|
||||
// storage. This leaf's `mainCheck` used to run them too, so each main push ran them twice in two
|
||||
// parallel jobs (review 2026-11-03 RR-06-09, original 08-04 (d)).
|
||||
|
||||
|
||||
@@ -46,12 +46,18 @@ in `platform`), so the two capability leaves read the same way (review 04-16).
|
||||
write; Spring Data repositories are mocks).
|
||||
- `FileserverPersistenceActivationTest` — the scan covers this package only, behind
|
||||
`app.fileserver-platform.enabled`, and runs before the JPA root auto-configuration.
|
||||
- `repository.CleanupClaimSqlTest` — the claim keeps its one-statement shape (the Claims rule).
|
||||
- `repository.CleanupClaimSqlTest` — the claim's text keeps its one-statement shape (the Claims
|
||||
rule); the locking itself is checked on PostgreSQL by
|
||||
`PostgreSqlFileserverReclamationIntegrationTest`.
|
||||
- `entity.AssignedIdEntityTest` — newness from `Persistable`, on Hibernate over H2.
|
||||
A new store write adds its case to the guard test; a new auto-configuration adds its line to the
|
||||
activation test.
|
||||
- Port contracts live in `application-fileserver` testFixtures (`*PortContract`). The in-memory
|
||||
fakes and the PostgreSQL tests both extend them.
|
||||
- Port contracts live in `application-fileserver` testFixtures (`*PortContract`, and
|
||||
`FileQuotaLedgerContract` for the three quota ports, which share one table). The in-memory fakes
|
||||
and the PostgreSQL tests both extend them; every port this leaf implements has one
|
||||
(`docs/testing/TESTING_STRATEGY.md` 9.3). The PostgreSQL side of the seven added in review
|
||||
2026-11-03 RR-01-04 is tagged `jpa-contract` and runs in persistence-jpa's
|
||||
`jpaPlatformContractTest` lane on `FileserverPostgreSqlStack`, whose clock a case moves.
|
||||
- The real-PostgreSQL tests are still in `persistence-jpa`'s `postgresqlIntegrationTest` source
|
||||
set, which owns the container and Flyway wiring, and the JPA readiness card
|
||||
(`config/jpa/readiness-cards.yaml`, checked by `JpaReadinessRegistryTest`) names them as that
|
||||
|
||||
+5
-2
@@ -18,8 +18,11 @@ import java.util.UUID;
|
||||
/**
|
||||
* JPA-backed {@link FileQuotaPort}.
|
||||
*
|
||||
* <p>Reservation, extension, commit, and release are conditional statements, so a reservation that
|
||||
* already expired or was released can never be extended or committed.
|
||||
* <p>Reservation, extension, commit, and release are conditional statements: a reservation that
|
||||
* expired or was released can never be extended, and a released or already committed one can never
|
||||
* be committed. An expired reservation that was not released still commits — bytes that were
|
||||
* written are durable usage whether or not the reservation outlived its TTL. {@code
|
||||
* FileQuotaLedgerContract} holds this adapter and the in-memory ledger to the same cases.
|
||||
*/
|
||||
public class JpaFileQuota implements FileQuotaPort {
|
||||
|
||||
|
||||
+2
-12
@@ -15,8 +15,8 @@ package dev.caskeleton.adapter.outbound.persistence.fileserver.repository;
|
||||
* every claimed item is fenced on its own.
|
||||
*
|
||||
* <p>Package-private: the repository is its one reader, and the leaf's public surface is a ceiling.
|
||||
* {@link #positional(String)} rewrites the named parameters to JDBC placeholders for a test in this
|
||||
* package that runs the identical text on a bare {@code Connection}.
|
||||
* The PostgreSQL lane runs it through the repository ({@code
|
||||
* PostgreSqlFileserverReclamationIntegrationTest}); nothing runs the text on its own.
|
||||
*/
|
||||
final class CleanupClaimSql {
|
||||
|
||||
@@ -43,14 +43,4 @@ final class CleanupClaimSql {
|
||||
+ " c.file_id, c.upload_id, c.content_key";
|
||||
|
||||
private CleanupClaimSql() {}
|
||||
|
||||
/**
|
||||
* Rewrites the named parameters to JDBC placeholders, left to right.
|
||||
*
|
||||
* @param namedSql the statement above
|
||||
* @return the same statement with {@code ?} in place of every {@code :name}
|
||||
*/
|
||||
static String positional(String namedSql) {
|
||||
return namedSql.replaceAll("(?<!:):[A-Za-z][A-Za-z0-9]*", "?");
|
||||
}
|
||||
}
|
||||
|
||||
+9
-18
@@ -7,16 +7,17 @@ import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* The claim statement keeps the shape the leaf's claim rule names: one statement that locks the
|
||||
* rows it picks, skips rows another claim holds, gives every row a fresh token and a higher fence,
|
||||
* and returns them in the same round trip. The PostgreSQL lane runs it; this pins the shape without
|
||||
* a database, so a rewrite into a select-then-update fails here first.
|
||||
* The claim statement's text keeps the shape the leaf's claim rule names: one statement that picks
|
||||
* rows with {@code FOR UPDATE SKIP LOCKED}, gives every row a fresh token and a higher fence, and
|
||||
* returns them. This pins the text without a database, so a rewrite into a select-then-update fails
|
||||
* here first. Whether the statement actually locks, skips and fences is a behaviour of PostgreSQL,
|
||||
* and {@code PostgreSqlFileserverReclamationIntegrationTest} runs it there.
|
||||
*/
|
||||
class CleanupClaimSqlTest {
|
||||
|
||||
@Test
|
||||
@DisplayName("claim은 고른 행을 잠그고, 다른 claim이 잡은 행은 기다리지 않고 건너뛴다")
|
||||
void theClaimLocksWhatItPicksAndSkipsWhatAnotherClaimHolds() {
|
||||
@DisplayName("claim 문장은 한 문장의 CTE 안에서 FOR UPDATE SKIP LOCKED로 행을 고른다")
|
||||
void theClaimTextPicksRowsWithForUpdateSkipLockedInOneStatement() {
|
||||
// when
|
||||
String claim = CleanupClaimSql.CLAIM_DUE;
|
||||
|
||||
@@ -25,8 +26,8 @@ class CleanupClaimSqlTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("claim한 행마다 새 토큰과 하나 높은 fence를 받고 같은 왕복에 돌아온다")
|
||||
void everyClaimedRowGetsAFreshTokenAndAHigherFenceInTheSameRoundTrip() {
|
||||
@DisplayName("claim 문장은 행마다 새 토큰을 쓰고 fence를 하나 올리고 RETURNING으로 돌려준다")
|
||||
void theClaimTextWritesAFreshTokenRaisesTheFenceAndReturnsTheRows() {
|
||||
// when
|
||||
String claim = CleanupClaimSql.CLAIM_DUE;
|
||||
|
||||
@@ -38,14 +39,4 @@ class CleanupClaimSqlTest {
|
||||
claim.contains("RETURNING c.cleanup_id, c.claim_token")))
|
||||
.containsOnly(true);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("이름 있는 매개변수를 JDBC 자리표시자로 바꿀 때 PostgreSQL 형 변환은 건드리지 않는다")
|
||||
void positionalParametersDoNotRewritePostgreSqlCasts() {
|
||||
// when
|
||||
String positional = CleanupClaimSql.positional("payload::jsonb @> :expected::jsonb");
|
||||
|
||||
// then
|
||||
assertThat(positional).isEqualTo("payload::jsonb @> ?::jsonb");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -235,7 +235,11 @@ made Spring call `getTransactionIsolation()` at every begin, which pgjdbc answer
|
||||
|
||||
Every mode is time-bounded (review 04-10). Spring hands the remaining time to each statement, so a
|
||||
slow query or lock wait is cancelled by the server (`57014` → `DB_QUERY_CANCELED`), and a deadline
|
||||
that passed before the next statement (`TransactionTimedOutException`) gets the same code.
|
||||
that passed before the next statement gets the same code: Spring's `TransactionTimedOutException`
|
||||
on the JDBC path, and on the JPA path (`JpaTransactionManager`, production) Hibernate's own
|
||||
`TransactionException`, which has no SQLState and is classified because `SpringTransaction` knows
|
||||
its deadline passed (`translateAfterDeadline`, review RR-03-03). A business exception thrown after
|
||||
the deadline is passed on unchanged.
|
||||
|
||||
Pre-built templates are immutable after construction so concurrent callers cannot
|
||||
observe each other's reconfiguration. `inRootWrite` reuses the pre-built write template,
|
||||
@@ -302,7 +306,9 @@ the two paths disagree on any state.
|
||||
|
||||
**No SQLState, classified by type:** a lost `@Version` race (`OptimisticLockingFailureException`,
|
||||
JPA `OptimisticLockException`) → `DB_OPTIMISTIC_CONFLICT` (409, review 04-05); an expired Spring
|
||||
transaction deadline (`TransactionTimedOutException`) → `DB_QUERY_CANCELED`.
|
||||
transaction deadline (`TransactionTimedOutException`) → `DB_QUERY_CANCELED`; any other persistence
|
||||
failure (JPA, Spring data-access or Spring transaction exception) of a transaction
|
||||
`SpringTransaction` began, once its deadline has passed → `DB_QUERY_CANCELED`.
|
||||
|
||||
**Commit phase (`translateCommitFailure`, ADR-JPA-003):** once the action has returned and
|
||||
`SpringTransaction` is committing a write transaction it began, a failure with `08*`, `40003`, a
|
||||
|
||||
@@ -296,14 +296,26 @@ A가 FAILED로 바꾸면 다시 발행되고(중복 전달), A의 늦은 PUBLISH
|
||||
`WHERE event_id = ? AND status = 'IN_FLIGHT' AND lease_owner = ? AND lease_token = ?`.
|
||||
owner는 비우고 token은 그대로 둔다(token은 증가만 한다). 1 row면 `APPLIED`, 0 row면 `STALE`.
|
||||
`findById` + setter 경로와 `@Version`은 쓰지 않는다.
|
||||
- **실패 원인 코드(`V22`)**: `markFailed`·`markDeferred`·`markDead`는 같은 조건부 UPDATE에서
|
||||
`last_failure_code`를 쓴다(publisher가 붙인 코드, 없으면 결과·예외 타입 이름). entity 필드는 읽기
|
||||
전용이라 append INSERT와 claim flush는 이 열을 건드리지 않는다. `longRetries(n, now)`는 미룬 claim이
|
||||
n번 이상인 FAILED 행 수와 가장 오래된 행의 나이를 센다(`outbox.retry.long.*` 게이지). 상태 `PARKED`는
|
||||
DDL이 없다 — claim이 PENDING·FAILED·IN_FLIGHT만 고르고, FIFO 게이트는 PUBLISHED 아닌 모든 행을 막는 앞
|
||||
행으로 본다. claim이 시도 횟수가 어긋난 행(`deferred_attempt_count > attempt_count`)을 만나면 DEAD로
|
||||
옮기고 WARN `error.reason=ATTEMPT_COUNTERS_INCONSISTENT`로 원인을 말한다(재리뷰 2026-11-03 RR-05-01·03).
|
||||
- **미룬 실패와 claim 해제(`V20`)**: `markDeferred(lease, next)`는 `markFailed`와 같은 전이에
|
||||
`deferred_attempt_count + 1`을 더한다(일시적 실패는 시도 예산 밖). `releaseLease(lease, at)`는
|
||||
손대지 않은 claim을 `PENDING`, `next_attempt_at = at`, `attempt_count - 1`로 되돌린다(종료 중인
|
||||
relay가 쓴다). 둘 다 같은 lease 조건부 UPDATE다.
|
||||
- **aggregate 순서 번호(`V21`)**: `JpaOutboxStore`는 `OutboxAggregateSequencePort`도 구현한다.
|
||||
`outbox_aggregate_sequence` counter를 `UPDATE ... + 1`(row lock)로 올리고, 없으면 그 aggregate의 기존
|
||||
outbox 최대 `aggregate_sequence` + 1로 만든다. V21은 `outbox_event`의 순서 키 유일 인덱스도 만든다 —
|
||||
기존 중복 row가 있으면 migration이 실패하므로 먼저 정리해야 한다.
|
||||
`outbox_aggregate_sequence` counter를 `UPDATE`(row lock)로 올린다. 올린 값은 그 aggregate의 가장 큰
|
||||
outbox 번호보다 작아지지 않는다(rolling 배포 중 옛 인스턴스가 쓴 벽시계 번호가 먼저 온다). counter가
|
||||
없으면 `INSERT ... ON CONFLICT DO NOTHING`으로 만든다. 시작값은 남은 outbox 최대 번호와 V21이
|
||||
기록한 floor(`outbox_aggregate_sequence_floor`, V21 적용 시각의 epoch 마이크로초) 중 큰 값 + 1이다.
|
||||
첫 쓰기 두 개가 겹치면 두 번째는 첫 번째 커밋을 기다렸다가 아무것도 넣지 않고 증가만 한다(리뷰
|
||||
RR-03-01). V21은 남은 row가 있는 aggregate의 counter를 그 최대 번호로 미리 만들고, `outbox_event`의
|
||||
순서 키 유일 인덱스도 만든다 — 기존 중복 row가 있으면 migration이 실패하므로 먼저 정리해야 한다
|
||||
(`docs/runbooks/outbox-dead-letter.md` §8).
|
||||
- **STALE은 예외가 아니다**: row의 결과는 지금 lease를 가진 relay가 쓴다. use case는 결과를
|
||||
`OutboxRelayResult.Outcome.STALE`로 남기고 `OutboxRelayFailureReportPort.reportStaleCompletion`
|
||||
으로 알린 뒤 다음 이벤트로 넘어간다. FAILED/DEAD report는 보내지 않는다(아무것도 저장되지
|
||||
|
||||
@@ -187,6 +187,7 @@ def readinessLanes = [
|
||||
postgresqlOutboxRowLifecycleIntegrationTest : ['PostgreSqlOutboxRowLifecycleIntegrationTest'],
|
||||
postgresqlOutboxAggregateSequenceIntegrationTest : ['PostgreSqlOutboxAggregateSequenceIntegrationTest'],
|
||||
postgresqlOutboxOrderKeyRunbookIntegrationTest : ['PostgreSqlOutboxOrderKeyRunbookIntegrationTest'],
|
||||
postgresqlOutboxDispositionRunbookIntegrationTest : ['PostgreSqlOutboxDispositionRunbookIntegrationTest'],
|
||||
postgresqlFileserverMigrationIntegrationTest : ['PostgreSqlFileserverMigrationIntegrationTest'],
|
||||
postgresqlFileserverMetadataIntegrationTest : ['PostgreSqlFileserverMetadataStoreIntegrationTest'],
|
||||
postgresqlFileserverReclamationIntegrationTest : ['PostgreSqlFileserverReclamationIntegrationTest'],
|
||||
@@ -243,13 +244,20 @@ strictTestLanes {
|
||||
}
|
||||
}
|
||||
|
||||
// The V21 pre-deploy check and cleanup are SQL an operator runs from the outbox runbook (review
|
||||
// 2026-11-01, README 15 decision). The lane reads them out of the runbook, so the document is one of
|
||||
// its inputs: editing either block reruns the lane against real PostgreSQL.
|
||||
tasks.named('postgresqlOutboxOrderKeyRunbookIntegrationTest', Test) {
|
||||
def outboxRunbook = new File(rootProject.projectDir.parentFile, 'docs/runbooks/outbox-dead-letter.md')
|
||||
inputs.file(outboxRunbook).withPathSensitivity(PathSensitivity.RELATIVE).withPropertyName('outboxRunbook')
|
||||
systemProperty 'outbox.runbook', outboxRunbook.absolutePath
|
||||
// The outbox runbooks give an operator SQL to run: the V21 pre-deploy check and cleanup (review
|
||||
// 2026-11-01, README 15 decision), the manual dispositions of a DEAD row (re-review 2026-11-03
|
||||
// RR-05-01), and finding, parking, resuming and skipping a row that is retried without end
|
||||
// (RR-05-03). Each lane reads its blocks out of the runbooks, so the documents are its inputs:
|
||||
// editing a block reruns the lane against real PostgreSQL.
|
||||
['postgresqlOutboxOrderKeyRunbookIntegrationTest', 'postgresqlOutboxDispositionRunbookIntegrationTest'].each { laneName ->
|
||||
tasks.named(laneName, Test) {
|
||||
def outboxRunbook = new File(rootProject.projectDir.parentFile, 'docs/runbooks/outbox-dead-letter.md')
|
||||
inputs.file(outboxRunbook).withPathSensitivity(PathSensitivity.RELATIVE).withPropertyName('outboxRunbook')
|
||||
systemProperty 'outbox.runbook', outboxRunbook.absolutePath
|
||||
def publishFailedRunbook = new File(rootProject.projectDir.parentFile, 'docs/runbooks/outbox-publish-failed.md')
|
||||
inputs.file(publishFailedRunbook).withPathSensitivity(PathSensitivity.RELATIVE).withPropertyName('publishFailedRunbook')
|
||||
systemProperty 'outbox.publish-failed.runbook', publishFailedRunbook.absolutePath
|
||||
}
|
||||
}
|
||||
|
||||
tasks.named('postgresqlSecurityBaselineIntegrationTest') {
|
||||
|
||||
+9
-2
@@ -46,8 +46,15 @@ public enum FailureCategory {
|
||||
QUERY_TIMEOUT,
|
||||
|
||||
/**
|
||||
* {@code 25P03} — the server ended a session that sat idle inside an open transaction. Nothing
|
||||
* was committed, which is why its code is retryable where {@link #QUERY_TIMEOUT}'s is not.
|
||||
* Not a transaction timeout: {@code 25P03}, the server ended a session that sat idle inside an
|
||||
* open transaction ({@code idle_in_transaction_session_timeout}). Nothing was committed, which is
|
||||
* why its code ({@code DB_IDLE_IN_TX_TIMEOUT}) is retryable.
|
||||
*
|
||||
* <p>A transaction that ran past its own deadline ({@code SpringTransaction.Timeouts}) is {@link
|
||||
* #QUERY_TIMEOUT} ({@code DB_QUERY_CANCELED}), not this. The name says otherwise and is kept only
|
||||
* because the enum is public API ({@code docs/architecture/jpa-api-surface.txt}); it becomes
|
||||
* {@code IDLE_IN_TRANSACTION_TIMEOUT}, with {@link TransactionTimeoutException}, at the next
|
||||
* release that breaks this API anyway (review RR-03-10).
|
||||
*/
|
||||
TRANSACTION_TIMEOUT,
|
||||
|
||||
|
||||
+6
-3
@@ -1,10 +1,13 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.api.error;
|
||||
|
||||
/**
|
||||
* A transaction exceeded its configured transaction timeout (design §15.4).
|
||||
* The server ended a session that sat idle inside an open transaction ({@code 25P03}, {@link
|
||||
* FailureCategory#TRANSACTION_TIMEOUT}). Not a transaction that ran past its own deadline: that is
|
||||
* a cancelled statement ({@link FailureCategory#QUERY_TIMEOUT}).
|
||||
*
|
||||
* <p>The transaction is rolled back; whether the use case may be re-run is a business decision, so
|
||||
* the platform does not retry it automatically.
|
||||
* <p>The transaction is rolled back and nothing was committed, so its code is retryable; whether
|
||||
* the use case is re-run is still the caller's decision, and the platform does not retry it
|
||||
* automatically. The name is kept until the next API-breaking release (review RR-03-10).
|
||||
*/
|
||||
public final class TransactionTimeoutException extends JpaPersistenceException {
|
||||
|
||||
|
||||
+38
@@ -8,6 +8,7 @@ import dev.caskeleton.shared.error.ApiErrorCode;
|
||||
import dev.caskeleton.shared.error.OperationalError;
|
||||
import dev.caskeleton.shared.error.PersistenceFailureException;
|
||||
import jakarta.persistence.OptimisticLockException;
|
||||
import jakarta.persistence.PersistenceException;
|
||||
import java.io.IOException;
|
||||
import java.sql.SQLException;
|
||||
import java.sql.SQLNonTransientConnectionException;
|
||||
@@ -23,6 +24,7 @@ import java.util.Optional;
|
||||
import java.util.Set;
|
||||
import org.springframework.dao.DataAccessException;
|
||||
import org.springframework.dao.OptimisticLockingFailureException;
|
||||
import org.springframework.transaction.TransactionException;
|
||||
import org.springframework.transaction.TransactionTimedOutException;
|
||||
|
||||
/**
|
||||
@@ -140,6 +142,42 @@ public final class PersistenceExceptionTranslator {
|
||||
return Optional.empty();
|
||||
}
|
||||
|
||||
/**
|
||||
* Classify a failure raised once the transaction's deadline had passed, when neither a SQLState
|
||||
* nor a type above identifies it.
|
||||
*
|
||||
* <p>Past the deadline Hibernate refuses the next statement itself, before the driver is asked:
|
||||
* an {@code org.hibernate.TransactionException} with no SQLState that is not Spring's {@link
|
||||
* TransactionTimedOutException} — in the action as it is, at commit wrapped in a {@code
|
||||
* JpaSystemException} (review RR-03-03). That is the production path ({@code
|
||||
* JpaTransactionManager}); the JDBC path gets {@code 57014} from the server instead. Both are
|
||||
* {@code DB_QUERY_CANCELED}. The caller decides that the deadline has passed; this method only
|
||||
* makes sure the failure came from the persistence stack — a JPA, Spring data-access or Spring
|
||||
* transaction exception anywhere in the cause chain — so a business exception thrown late is
|
||||
* passed on as it is, and one already classified is not classified again.
|
||||
*/
|
||||
public Optional<PersistenceFailureException> translateAfterDeadline(Throwable ex) {
|
||||
Objects.requireNonNull(ex, "ex");
|
||||
if (ex instanceof PersistenceFailureException) {
|
||||
return Optional.empty();
|
||||
}
|
||||
for (Throwable t = ex; t != null; t = t.getCause()) {
|
||||
if (t instanceof PersistenceException
|
||||
|| t instanceof DataAccessException
|
||||
|| t instanceof TransactionException) {
|
||||
return Optional.of(
|
||||
new PersistenceFailureException(
|
||||
OperationalError.DB_QUERY_CANCELED,
|
||||
"persistence failure after the transaction deadline passed",
|
||||
ex));
|
||||
}
|
||||
if (t.getCause() == t) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
return Optional.empty();
|
||||
}
|
||||
|
||||
/**
|
||||
* Classify a failure raised while committing a write transaction this call began.
|
||||
*
|
||||
|
||||
+76
-18
@@ -14,6 +14,7 @@ import dev.caskeleton.application.outbox.OutboxAppendPort;
|
||||
import dev.caskeleton.application.outbox.OutboxCompletion;
|
||||
import dev.caskeleton.application.outbox.OutboxEventStatus;
|
||||
import dev.caskeleton.application.outbox.OutboxLease;
|
||||
import dev.caskeleton.application.outbox.OutboxLongRetries;
|
||||
import dev.caskeleton.application.outbox.OutboxStorePort;
|
||||
import dev.caskeleton.shared.digest.Sha256;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
@@ -29,6 +30,7 @@ import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.slf4j.spi.LoggingEventBuilder;
|
||||
|
||||
/**
|
||||
* JPA-backed {@link OutboxAppendPort} + {@link OutboxStorePort}. {@link #append} and the claim/mark
|
||||
@@ -103,12 +105,21 @@ public class JpaOutboxStore
|
||||
*
|
||||
* <p>The increment is an {@code UPDATE}, which holds the counter row's lock until the caller
|
||||
* commits: a second writer of the same aggregate waits, then increments the committed value, so
|
||||
* the numbers follow commit order. An aggregate's first number is inserted instead, seeded after
|
||||
* the highest {@code aggregate_sequence} its outbox rows already carry — rows numbered before the
|
||||
* counter existed (the sample used epoch microseconds) must still come first. Two writers
|
||||
* creating the same counter at once collide on its primary key and one transaction fails: a
|
||||
* rollback the caller retries, never two events with one number. The counter outlives the
|
||||
* reaper's deletes, so a number is never handed out twice.
|
||||
* the numbers follow commit order. It never goes below the aggregate's highest outbox row, so a
|
||||
* row an instance still on the wall-clock scheme writes during a rolling deploy comes first. The
|
||||
* counter outlives the reaper's deletes, so a number is never handed out twice.
|
||||
*
|
||||
* <p>An aggregate's first number is inserted instead, after the larger of its highest remaining
|
||||
* row and the floor V21 recorded — the moment it ran, above every clock number already sent for
|
||||
* rows the reaper has deleted or that were written under an identity that has since changed
|
||||
* (review RR-03-02). Two writers creating the same counter at once both succeed: the insert is
|
||||
* {@code ON CONFLICT DO NOTHING}, so the second waits for the first to commit, writes nothing,
|
||||
* and increments the committed counter instead (review RR-03-01).
|
||||
*
|
||||
* <p>Statements rather than one {@code INSERT … ON CONFLICT DO UPDATE … RETURNING} like {@code
|
||||
* JpaLiveEventReplay}'s position allocator: the sample appends events on the H2 developer run
|
||||
* ({@code bootRunH2}), and H2 accepts {@code ON CONFLICT DO NOTHING} but not {@code DO UPDATE}.
|
||||
* The common case — the counter exists — is still one {@code UPDATE} and one read.
|
||||
*/
|
||||
@Override
|
||||
public long nextSequence(AggregateIdentity aggregate) {
|
||||
@@ -118,8 +129,11 @@ public class JpaOutboxStore
|
||||
String tenant = aggregate.tenantScope();
|
||||
String type = aggregate.aggregateType();
|
||||
String id = aggregate.aggregateId();
|
||||
if (repository.incrementAggregateSequence(tenant, type, id) == 0) {
|
||||
repository.createAggregateSequence(tenant, type, id);
|
||||
if (repository.incrementAggregateSequence(tenant, type, id) == 0
|
||||
&& repository.createAggregateSequence(tenant, type, id) == 0) {
|
||||
// Another writer created the counter after our UPDATE found none; its commit ended our wait
|
||||
// in the insert, so the counter is there to increment now.
|
||||
repository.incrementAggregateSequence(tenant, type, id);
|
||||
}
|
||||
return repository.aggregateSequence(tenant, type, id);
|
||||
}
|
||||
@@ -132,6 +146,11 @@ public class JpaOutboxStore
|
||||
* instead rolled the whole claim back, and the next claim selected the same row first again, so
|
||||
* one bad row stopped the relay for good. DEAD is fail-closed: the row is never published, and it
|
||||
* waits for the DEAD-row runbook like any other.
|
||||
*
|
||||
* <p>The WARN names the cause. A row whose deferred claims outnumber its claims is not damaged:
|
||||
* it is what a manual re-drive that reset {@code attempt_count} and left {@code
|
||||
* deferred_attempt_count} produces (re-review 2026-11-03 RR-05-01), and "cannot read" sent the
|
||||
* operator looking for corruption instead of re-running the corrected runbook SQL.
|
||||
*/
|
||||
@Override
|
||||
public List<OutboxLease> claimBatch(
|
||||
@@ -145,13 +164,26 @@ public class JpaOutboxStore
|
||||
entity.setNextAttemptAt(now.plus(inFlightTimeout));
|
||||
entity.setLeaseOwner(owner);
|
||||
entity.setLeaseToken(entity.getLeaseToken() + 1);
|
||||
if (entity.getDeferredAttemptCount() > entity.getAttemptCount()) {
|
||||
deadLetterOnClaim(entity)
|
||||
.addKeyValue("error.reason", "ATTEMPT_COUNTERS_INCONSISTENT")
|
||||
.log(
|
||||
"outbox claim moved a row to DEAD: deferred_attempt_count ({}) exceeds"
|
||||
+ " attempt_count ({}); a manual re-drive that did not reset"
|
||||
+ " deferred_attempt_count leaves this behind — re-drive it with the runbook"
|
||||
+ " SQL",
|
||||
entity.getDeferredAttemptCount(),
|
||||
entity.getAttemptCount());
|
||||
continue;
|
||||
}
|
||||
ClaimedOutboxEvent event;
|
||||
try {
|
||||
event = toClaimedOutboxEvent(entity);
|
||||
} catch (RuntimeException unreadable) {
|
||||
entity.setStatus(OutboxEventStatus.DEAD.name());
|
||||
entity.setLeaseOwner(null);
|
||||
log.warn("outbox claim moved a row it cannot read to DEAD", unreadable);
|
||||
deadLetterOnClaim(entity)
|
||||
.addKeyValue("error.reason", "UNREADABLE_ROW")
|
||||
.setCause(unreadable)
|
||||
.log("outbox claim moved a row it cannot read as an event to DEAD");
|
||||
continue;
|
||||
}
|
||||
leases.add(new OutboxLease(event, owner, entity.getLeaseToken()));
|
||||
@@ -159,6 +191,15 @@ public class JpaOutboxStore
|
||||
return List.copyOf(leases);
|
||||
}
|
||||
|
||||
/** Moves a claimed row it will not hand out to DEAD and starts the WARN that says why. */
|
||||
private static LoggingEventBuilder deadLetterOnClaim(OutboxEventEntity entity) {
|
||||
entity.setStatus(OutboxEventStatus.DEAD.name());
|
||||
entity.setLeaseOwner(null);
|
||||
return log.atWarn()
|
||||
.addKeyValue("event_id", entity.getEventId())
|
||||
.addKeyValue("runbook_link", "runbook://outbox/dead-letter");
|
||||
}
|
||||
|
||||
/**
|
||||
* One conditional UPDATE, no load-and-set: it applies only while the lease still holds the row
|
||||
* (see README "outbox"). Zero rows is {@link OutboxCompletion#STALE}, not an error — the relay
|
||||
@@ -174,18 +215,21 @@ public class JpaOutboxStore
|
||||
|
||||
/** Marks the leased row FAILED with its next attempt time; see {@link #markPublished}. */
|
||||
@Override
|
||||
public OutboxCompletion markFailed(OutboxLease lease, Instant nextAttemptAt) {
|
||||
public OutboxCompletion markFailed(OutboxLease lease, Instant nextAttemptAt, String failureCode) {
|
||||
requireTransaction("markFailed");
|
||||
return completion(
|
||||
repository.failLease(lease.eventId(), lease.owner(), lease.token(), nextAttemptAt));
|
||||
repository.failLease(
|
||||
lease.eventId(), lease.owner(), lease.token(), nextAttemptAt, failureCode));
|
||||
}
|
||||
|
||||
/** Marks the leased row FAILED as a deferred claim; see {@link #markPublished}. */
|
||||
@Override
|
||||
public OutboxCompletion markDeferred(OutboxLease lease, Instant nextAttemptAt) {
|
||||
public OutboxCompletion markDeferred(
|
||||
OutboxLease lease, Instant nextAttemptAt, String failureCode) {
|
||||
requireTransaction("markDeferred");
|
||||
return completion(
|
||||
repository.deferLease(lease.eventId(), lease.owner(), lease.token(), nextAttemptAt));
|
||||
repository.deferLease(
|
||||
lease.eventId(), lease.owner(), lease.token(), nextAttemptAt, failureCode));
|
||||
}
|
||||
|
||||
/** Returns the leased row to PENDING without spending an attempt; see {@link #markPublished}. */
|
||||
@@ -198,11 +242,10 @@ public class JpaOutboxStore
|
||||
|
||||
/** Marks the leased row DEAD; see {@link #markPublished}. */
|
||||
@Override
|
||||
public OutboxCompletion markDead(OutboxLease lease) {
|
||||
public OutboxCompletion markDead(OutboxLease lease, String failureCode) {
|
||||
requireTransaction("markDead");
|
||||
return completion(
|
||||
repository.completeLease(
|
||||
lease.eventId(), lease.owner(), lease.token(), OutboxEventStatus.DEAD.name()));
|
||||
repository.deadLease(lease.eventId(), lease.owner(), lease.token(), failureCode));
|
||||
}
|
||||
|
||||
private static OutboxCompletion completion(int updatedRows) {
|
||||
@@ -237,6 +280,21 @@ public class JpaOutboxStore
|
||||
return result;
|
||||
}
|
||||
|
||||
@Override
|
||||
public OutboxLongRetries longRetries(int minDeferredClaims, Instant now) {
|
||||
if (minDeferredClaims < 1) {
|
||||
throw new IllegalArgumentException(
|
||||
"minDeferredClaims must be >= 1, was " + minDeferredClaims);
|
||||
}
|
||||
Object[] row = repository.countLongRetries(minDeferredClaims).getFirst();
|
||||
long count = (Long) row[0];
|
||||
if (count == 0) {
|
||||
return OutboxLongRetries.NONE;
|
||||
}
|
||||
Instant oldest = (Instant) row[1];
|
||||
return new OutboxLongRetries(count, Math.max(0, Duration.between(oldest, now).toSeconds()));
|
||||
}
|
||||
|
||||
/**
|
||||
* Restores the appended event. A row without its envelope — written by the pre-canonical
|
||||
* generation, or damaged — is unreadable, and so is one whose stored hash is not the hash of its
|
||||
|
||||
+51
-11
@@ -62,18 +62,22 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
|
||||
@Param("token") long token,
|
||||
@Param("status") String status);
|
||||
|
||||
/** {@link #completeLease} to FAILED, scheduling the retry in the same statement. */
|
||||
/**
|
||||
* {@link #completeLease} to FAILED, scheduling the retry and recording why (V22) in the same
|
||||
* statement.
|
||||
*/
|
||||
@Modifying(flushAutomatically = true)
|
||||
@Query(
|
||||
"update OutboxEventEntity e set e.status = 'FAILED', e.nextAttemptAt = :nextAttemptAt, "
|
||||
+ "e.leaseOwner = null "
|
||||
+ "e.lastFailureCode = :failureCode, e.leaseOwner = null "
|
||||
+ "where e.eventId = :eventId and e.status = 'IN_FLIGHT' "
|
||||
+ "and e.leaseOwner = :owner and e.leaseToken = :token")
|
||||
int failLease(
|
||||
@Param("eventId") String eventId,
|
||||
@Param("owner") String owner,
|
||||
@Param("token") long token,
|
||||
@Param("nextAttemptAt") Instant nextAttemptAt);
|
||||
@Param("nextAttemptAt") Instant nextAttemptAt,
|
||||
@Param("failureCode") String failureCode);
|
||||
|
||||
/**
|
||||
* {@link #failLease} for a claim that ended deferred: same transition, and the row counts the
|
||||
@@ -82,14 +86,29 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
|
||||
@Modifying(flushAutomatically = true)
|
||||
@Query(
|
||||
"update OutboxEventEntity e set e.status = 'FAILED', e.nextAttemptAt = :nextAttemptAt, "
|
||||
+ "e.deferredAttemptCount = e.deferredAttemptCount + 1, e.leaseOwner = null "
|
||||
+ "e.deferredAttemptCount = e.deferredAttemptCount + 1, "
|
||||
+ "e.lastFailureCode = :failureCode, e.leaseOwner = null "
|
||||
+ "where e.eventId = :eventId and e.status = 'IN_FLIGHT' "
|
||||
+ "and e.leaseOwner = :owner and e.leaseToken = :token")
|
||||
int deferLease(
|
||||
@Param("eventId") String eventId,
|
||||
@Param("owner") String owner,
|
||||
@Param("token") long token,
|
||||
@Param("nextAttemptAt") Instant nextAttemptAt);
|
||||
@Param("nextAttemptAt") Instant nextAttemptAt,
|
||||
@Param("failureCode") String failureCode);
|
||||
|
||||
/** {@link #completeLease} to DEAD, recording why (V22) in the same statement. */
|
||||
@Modifying(flushAutomatically = true)
|
||||
@Query(
|
||||
"update OutboxEventEntity e set e.status = 'DEAD', e.lastFailureCode = :failureCode, "
|
||||
+ "e.leaseOwner = null "
|
||||
+ "where e.eventId = :eventId and e.status = 'IN_FLIGHT' "
|
||||
+ "and e.leaseOwner = :owner and e.leaseToken = :token")
|
||||
int deadLease(
|
||||
@Param("eventId") String eventId,
|
||||
@Param("owner") String owner,
|
||||
@Param("token") long token,
|
||||
@Param("failureCode") String failureCode);
|
||||
|
||||
/**
|
||||
* Gives an unused claim back: PENDING, claimable from {@code claimableAt}, and the claim's
|
||||
@@ -110,12 +129,18 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
|
||||
|
||||
/**
|
||||
* Increments an aggregate's order counter (V21), taking its row lock until the caller commits.
|
||||
* Returns 0 when the aggregate has no counter yet.
|
||||
* Never below the aggregate's highest outbox row: an instance still on the wall-clock scheme may
|
||||
* have written a larger number after the counter existed (rolling deploy, review RR-03-02).
|
||||
* Returns 0 when the aggregate has no counter yet, or when another transaction's new counter is
|
||||
* not committed yet.
|
||||
*/
|
||||
@Modifying(flushAutomatically = true)
|
||||
@Query(
|
||||
value =
|
||||
"UPDATE outbox_aggregate_sequence SET last_sequence = last_sequence + 1"
|
||||
"UPDATE outbox_aggregate_sequence SET last_sequence = GREATEST(last_sequence,"
|
||||
+ " (SELECT COALESCE(MAX(e.aggregate_sequence), 0) FROM outbox_event e"
|
||||
+ " WHERE e.tenant_scope = :tenantScope AND e.aggregate_type = :aggregateType"
|
||||
+ " AND e.aggregate_id = :aggregateId)) + 1"
|
||||
+ " WHERE tenant_scope = :tenantScope AND aggregate_type = :aggregateType"
|
||||
+ " AND aggregate_id = :aggregateId",
|
||||
nativeQuery = true)
|
||||
@@ -125,8 +150,10 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
|
||||
@Param("aggregateId") String aggregateId);
|
||||
|
||||
/**
|
||||
* Creates an aggregate's counter, seeded after the highest order its outbox rows already carry.
|
||||
* Two creators of the same counter collide on its primary key.
|
||||
* Creates an aggregate's counter at its first number: one after the larger of its highest outbox
|
||||
* row and the V21 floor (0 where no migration wrote one). Returns 0, and writes nothing, when
|
||||
* another transaction created it first; {@code ON CONFLICT DO NOTHING} waits for that transaction
|
||||
* to end rather than failing on the primary key (review RR-03-01).
|
||||
*/
|
||||
@Modifying(flushAutomatically = true)
|
||||
@Query(
|
||||
@@ -134,9 +161,12 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
|
||||
"INSERT INTO outbox_aggregate_sequence"
|
||||
+ " (tenant_scope, aggregate_type, aggregate_id, last_sequence)"
|
||||
+ " SELECT :tenantScope, :aggregateType, :aggregateId,"
|
||||
+ " COALESCE(MAX(e.aggregate_sequence), 0) + 1 FROM outbox_event e"
|
||||
+ " GREATEST(COALESCE(MAX(e.aggregate_sequence), 0),"
|
||||
+ " COALESCE((SELECT f.floor_sequence FROM outbox_aggregate_sequence_floor f), 0))"
|
||||
+ " + 1 FROM outbox_event e"
|
||||
+ " WHERE e.tenant_scope = :tenantScope AND e.aggregate_type = :aggregateType"
|
||||
+ " AND e.aggregate_id = :aggregateId",
|
||||
+ " AND e.aggregate_id = :aggregateId"
|
||||
+ " ON CONFLICT DO NOTHING",
|
||||
nativeQuery = true)
|
||||
int createAggregateSequence(
|
||||
@Param("tenantScope") String tenantScope,
|
||||
@@ -174,4 +204,14 @@ public interface OutboxEventJpaRepository extends Repository<OutboxEventEntity,
|
||||
"select e.eventType, min(e.occurredAt) from OutboxEventEntity e "
|
||||
+ "where e.status <> 'PUBLISHED' group by e.eventType")
|
||||
List<Object[]> findOldestUnpublishedOccurredAtByEventType();
|
||||
|
||||
/**
|
||||
* {@code [count (Long), oldestOccurredAt (Instant)]} of the FAILED rows with at least {@code
|
||||
* minDeferredClaims} deferred claims, for the long-retry gauges. One row, whose instant is null
|
||||
* when the count is 0.
|
||||
*/
|
||||
@Query(
|
||||
"select count(e), min(e.occurredAt) from OutboxEventEntity e "
|
||||
+ "where e.status = 'FAILED' and e.deferredAttemptCount >= :minDeferredClaims")
|
||||
List<Object[]> countLongRetries(@Param("minDeferredClaims") int minDeferredClaims);
|
||||
}
|
||||
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.outbox.jpa.entity;
|
||||
|
||||
import jakarta.persistence.Column;
|
||||
import jakarta.persistence.Entity;
|
||||
import jakarta.persistence.Id;
|
||||
import jakarta.persistence.Table;
|
||||
|
||||
/**
|
||||
* The one-row floor a new aggregate counter starts after ({@code outbox_aggregate_sequence_floor},
|
||||
* V21): the moment V21 ran, in epoch microseconds, above every number the replaced wall-clock
|
||||
* scheme wrote. Read by {@code JpaOutboxStore#nextSequence} with a native statement; mapped, like
|
||||
* {@link OutboxAggregateSequenceEntity}, so the table exists where Hibernate builds the schema (H2,
|
||||
* where no migration writes the row and the floor is 0) and is validated where Flyway does.
|
||||
* Package-private: no code reads it as an entity.
|
||||
*/
|
||||
@Entity
|
||||
@Table(name = "outbox_aggregate_sequence_floor")
|
||||
class OutboxAggregateSequenceFloorEntity {
|
||||
|
||||
@Id
|
||||
@Column(name = "singleton", nullable = false)
|
||||
private boolean singleton;
|
||||
|
||||
@Column(name = "floor_sequence", nullable = false)
|
||||
private long floorSequence;
|
||||
|
||||
protected OutboxAggregateSequenceFloorEntity() {}
|
||||
|
||||
long floorSequence() {
|
||||
return floorSequence;
|
||||
}
|
||||
}
|
||||
+16
@@ -58,6 +58,18 @@ public class OutboxEventEntity implements Persistable<String> {
|
||||
private int deferredAttemptCount;
|
||||
|
||||
/** Dual-purpose per status (PENDING / IN_FLIGHT / FAILED) — see README "outbox". */
|
||||
/**
|
||||
* The code of the row's last failed completion (V22): the publisher's own name for the failure,
|
||||
* or the outcome or exception type when it gave none.
|
||||
*
|
||||
* <p>Read-only to the entity: only the relay's failed completions write it, each in its one
|
||||
* fenced UPDATE ({@code OutboxEventJpaRepository#failLease} and its siblings). The append's
|
||||
* INSERT and the claim's flush leave it out, so they neither clear a code nor need the column on
|
||||
* a database that has not run V22 yet.
|
||||
*/
|
||||
@Column(name = "last_failure_code", length = 120, insertable = false, updatable = false)
|
||||
private String lastFailureCode;
|
||||
|
||||
@Column(name = "next_attempt_at", nullable = false)
|
||||
private Instant nextAttemptAt;
|
||||
|
||||
@@ -379,4 +391,8 @@ public class OutboxEventEntity implements Persistable<String> {
|
||||
public void setCausationId(String causationId) {
|
||||
this.causationId = causationId;
|
||||
}
|
||||
|
||||
public String getLastFailureCode() {
|
||||
return lastFailureCode;
|
||||
}
|
||||
}
|
||||
|
||||
+20
-3
@@ -123,7 +123,9 @@ public final class SpringTransaction implements TransactionPort {
|
||||
* dry and every other request fails at {@code connection-timeout}. With one, Spring hands the
|
||||
* remaining time to every statement (Hibernate and {@code JdbcTemplate} both set the JDBC query
|
||||
* timeout), so the server cancels the statement ({@code 57014}, {@code DB_QUERY_CANCELED}) and
|
||||
* the connection comes back.
|
||||
* the connection comes back. A statement Hibernate is asked to run after the deadline has passed
|
||||
* is refused by Hibernate itself, with no SQLState; this class reports that as {@code
|
||||
* DB_QUERY_CANCELED} too (review RR-03-03).
|
||||
*
|
||||
* <p>The default is 30 seconds for both: the budget {@code inNew} already has, and the Hikari
|
||||
* {@code leak-detection-threshold} at which the pool already reports a held connection as a leak.
|
||||
@@ -194,6 +196,9 @@ public final class SpringTransaction implements TransactionPort {
|
||||
|
||||
private <T> T execute(ModeTemplate mode, Supplier<T> action) {
|
||||
long startedAt = System.nanoTime();
|
||||
// Whether this call began the transaction, and so owns its deadline. A call that joined an
|
||||
// outer one leaves the deadline to the call that began it.
|
||||
AtomicBoolean beganTransaction = new AtomicBoolean();
|
||||
// Set only once the action has returned in a write transaction this call began: from then on
|
||||
// the only thing TransactionTemplate.execute still does is commit it.
|
||||
AtomicBoolean committingOwnWrite = new AtomicBoolean();
|
||||
@@ -201,20 +206,32 @@ public final class SpringTransaction implements TransactionPort {
|
||||
return mode.template()
|
||||
.execute(
|
||||
status -> {
|
||||
beganTransaction.set(status.isNewTransaction());
|
||||
T result = action.get();
|
||||
committingOwnWrite.set(status.isNewTransaction() && !mode.template().isReadOnly());
|
||||
return result;
|
||||
});
|
||||
} catch (RuntimeException failure) {
|
||||
Duration elapsed = Duration.ofNanos(System.nanoTime() - startedAt);
|
||||
Optional<PersistenceFailureException> translated =
|
||||
committingOwnWrite.get()
|
||||
? exceptionTranslator.translateCommitFailure(
|
||||
failure, mode.commitOperation(), Duration.ofNanos(System.nanoTime() - startedAt))
|
||||
? exceptionTranslator.translateCommitFailure(failure, mode.commitOperation(), elapsed)
|
||||
: exceptionTranslator.translate(failure);
|
||||
if (translated.isEmpty() && beganTransaction.get() && pastDeadline(mode, elapsed)) {
|
||||
// Hibernate's own deadline refusal carries no SQLState (review RR-03-03).
|
||||
translated = exceptionTranslator.translateAfterDeadline(failure);
|
||||
}
|
||||
throw translated.map(RuntimeException.class::cast).orElse(failure);
|
||||
}
|
||||
}
|
||||
|
||||
/** Whether {@code elapsed} reached the timeout the mode's transactions run under. */
|
||||
private static boolean pastDeadline(ModeTemplate mode, Duration elapsed) {
|
||||
int timeoutSeconds = mode.template().getTimeout();
|
||||
return timeoutSeconds != TransactionDefinition.TIMEOUT_DEFAULT
|
||||
&& elapsed.compareTo(Duration.ofSeconds(timeoutSeconds)) >= 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* The vendor-neutral default translator.
|
||||
*
|
||||
|
||||
+72
-3
@@ -7,8 +7,32 @@
|
||||
--
|
||||
-- outbox_aggregate_sequence holds one counter per aggregate. Incrementing it locks the row until
|
||||
-- the writer commits, so the numbers follow commit order; it outlives the reaper's deletes, so a
|
||||
-- number is never reused. JpaOutboxAggregateSequences seeds a new counter from the aggregate's
|
||||
-- highest existing aggregate_sequence, so rows numbered by the old clock scheme still come first.
|
||||
-- number is never reused. JpaOutboxStore#nextSequence reads and writes it.
|
||||
--
|
||||
-- Consumers have already seen clock numbers (about 1.7e15) for aggregates whose rows this
|
||||
-- database no longer holds: the reaper deleted them, or they were written under an identity that
|
||||
-- has since changed (the sample's tenant scope). A counter that started after the rows still
|
||||
-- present would start at 1 for those aggregates, behind numbers already sent (review RR-03-02,
|
||||
-- RR-05-02). So this migration records one floor, the moment it runs in epoch microseconds, which
|
||||
-- is above every clock number written before it; a counter created later starts after the floor
|
||||
-- or after the aggregate's highest remaining row, whichever is larger. The aggregates that still
|
||||
-- have rows get their counter here, at their highest number. A clock number an instance still on
|
||||
-- the old version writes during the rollout is above the floor; nextSequence also never goes below
|
||||
-- the highest row it can see, so such a row still comes first.
|
||||
--
|
||||
-- This file was edited after it was first written, before any deployment applied it (the branch
|
||||
-- that added it was not merged): splitting the floor into a V22 would have left V21 alone handing
|
||||
-- out numbers from 1 for one release, and its comment pointed at a class that does not exist.
|
||||
-- applied-migrations.sha256 carries the new checksum. A development database that applied the
|
||||
-- earlier text fails Flyway validation; recreate it.
|
||||
|
||||
-- Bound the wait for the table lock the unique index below takes (SHARE on outbox_event, held until
|
||||
-- this migration commits). Without a bound, the migration queues behind any open outbox write and
|
||||
-- every outbox write after it queues behind the migration. Five seconds, then this migration fails,
|
||||
-- rolls back whole and the database stays at V20: deploy again, or build the index first
|
||||
-- (runbook §8, CONCURRENTLY).
|
||||
SET LOCAL lock_timeout = '5s';
|
||||
|
||||
CREATE TABLE IF NOT EXISTS outbox_aggregate_sequence (
|
||||
tenant_scope VARCHAR(96) NOT NULL,
|
||||
aggregate_type VARCHAR(64) NOT NULL,
|
||||
@@ -18,10 +42,55 @@ CREATE TABLE IF NOT EXISTS outbox_aggregate_sequence (
|
||||
CONSTRAINT ck_outbox_aggregate_sequence_positive CHECK (last_sequence >= 1)
|
||||
);
|
||||
|
||||
-- One row, written once: the lowest number a counter created from now on may start after.
|
||||
CREATE TABLE IF NOT EXISTS outbox_aggregate_sequence_floor (
|
||||
singleton BOOLEAN NOT NULL DEFAULT TRUE,
|
||||
floor_sequence BIGINT NOT NULL,
|
||||
CONSTRAINT pk_outbox_aggregate_sequence_floor PRIMARY KEY (singleton),
|
||||
CONSTRAINT ck_outbox_aggregate_sequence_floor_singleton CHECK (singleton),
|
||||
CONSTRAINT ck_outbox_aggregate_sequence_floor_non_negative CHECK (floor_sequence >= 0)
|
||||
);
|
||||
|
||||
INSERT INTO outbox_aggregate_sequence_floor (singleton, floor_sequence)
|
||||
VALUES (TRUE, (extract(epoch FROM clock_timestamp()) * 1000000)::BIGINT)
|
||||
ON CONFLICT (singleton) DO NOTHING;
|
||||
|
||||
INSERT INTO outbox_aggregate_sequence (tenant_scope, aggregate_type, aggregate_id, last_sequence)
|
||||
SELECT tenant_scope, aggregate_type, aggregate_id, MAX(aggregate_sequence)
|
||||
FROM outbox_event
|
||||
WHERE tenant_scope IS NOT NULL
|
||||
AND aggregate_type IS NOT NULL
|
||||
AND aggregate_sequence IS NOT NULL
|
||||
GROUP BY tenant_scope, aggregate_type, aggregate_id
|
||||
ON CONFLICT (tenant_scope, aggregate_type, aggregate_id) DO NOTHING;
|
||||
|
||||
-- One number, one event: a second event appended under an order its aggregate already used is
|
||||
-- refused at append time, in the writer's transaction, rather than released by the gate in no
|
||||
-- defined order. Rows without canonical order columns (pre-canonical, all NULL) are not compared.
|
||||
-- This fails if duplicates already exist; resolve them (they are exactly the rows the gate could
|
||||
-- not order) before migrating.
|
||||
-- not order) before migrating (runbook §8).
|
||||
--
|
||||
-- Built here, inside this migration's transaction, not CONCURRENTLY: a duplicate then rolls the
|
||||
-- whole of V21 back and leaves the database at V20 with nothing to clean up, which is what the
|
||||
-- runbook's retry relies on. A failed CONCURRENTLY build leaves an INVALID index that IF NOT EXISTS
|
||||
-- would then skip, so the uniqueness would silently not hold; and CONCURRENTLY cannot run in the
|
||||
-- transaction that writes the counters and the floor. The reaper bounds outbox_event to the
|
||||
-- retention window, so the build is normally short. An outbox too large to lock for that long is
|
||||
-- indexed first with the runbook's CONCURRENTLY statement (same name, same columns); the check
|
||||
-- below refuses an index such a build left invalid, and IF NOT EXISTS keeps the valid one.
|
||||
DO $$
|
||||
BEGIN
|
||||
IF EXISTS (SELECT 1
|
||||
FROM pg_index
|
||||
WHERE indexrelid = to_regclass('ux_outbox_event_aggregate_order')
|
||||
AND NOT (indisvalid AND indisunique)) THEN
|
||||
RAISE EXCEPTION 'ux_outbox_event_aggregate_order exists but is not a valid unique index'
|
||||
USING HINT = 'A CREATE INDEX CONCURRENTLY failed or is still running. '
|
||||
|| 'DROP INDEX CONCURRENTLY ux_outbox_event_aggregate_order, then rebuild it '
|
||||
|| 'or let V21 build it (runbook outbox-dead-letter §8).';
|
||||
END IF;
|
||||
END
|
||||
$$;
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS ux_outbox_event_aggregate_order
|
||||
ON outbox_event (tenant_scope, aggregate_type, aggregate_id, aggregate_sequence, event_index);
|
||||
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
-- Why a row keeps failing, kept on the row.
|
||||
--
|
||||
-- A publish refused for now (a closed transport, a credential refused) or left unanswered is
|
||||
-- retried without spending the attempt budget (V20), so it has no end: the relay retries it five
|
||||
-- minutes apart for as long as nobody looks. The outcome the relay records cannot say why — a
|
||||
-- shutdown, a closed transport and a refused credential are one outcome — and the relay's log line
|
||||
-- said it in one sentence for all four. The publisher's failure code can (a messaging platform code
|
||||
-- such as KAFKA_AUTHENTICATIONEXCEPTION); every failed completion writes it here, so the runbook's
|
||||
-- query for rows that have been retrying long shows what each one is waiting on (re-review
|
||||
-- 2026-11-03 RR-05-03).
|
||||
--
|
||||
-- A code, never a message: a driver message can carry an endpoint, a statement or a credential.
|
||||
-- Nullable with no default, so adding it rewrites no row; rows written before it stay NULL until
|
||||
-- their next failure.
|
||||
--
|
||||
-- The new status PARKED needs no DDL: status has no CHECK, the claim selects PENDING, FAILED and
|
||||
-- IN_FLIGHT only (so the partial index ix_outbox_event_eligible leaves PARKED out), and the order
|
||||
-- gate and its index ix_outbox_event_unpublished_aggregate_order treat every status but PUBLISHED as
|
||||
-- blocking — which is what holding a row in its aggregate's place means.
|
||||
ALTER TABLE outbox_event
|
||||
ADD COLUMN last_failure_code VARCHAR(120);
|
||||
+29
-21
@@ -23,7 +23,8 @@ import org.junit.jupiter.api.Tag;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* What a worker's death leaves behind, at each phase of a dispatch.
|
||||
* What a worker's death leaves behind, at each phase of a dispatch — the rows, not what recovery
|
||||
* does with them.
|
||||
*
|
||||
* <p>This is the {@code recoverable} claim in {@code docs/notification/evidence-manifest.json}, and
|
||||
* until now nothing backed it. Lease recovery and event replay had unit coverage against fakes; the
|
||||
@@ -36,9 +37,13 @@ import org.junit.jupiter.api.Test;
|
||||
* worker is from the database's point of view — a lease nobody will renew and a phase nobody will
|
||||
* advance.
|
||||
*
|
||||
* <p>The matrix walks the phases a dispatch passes through. For each, the question is the same and
|
||||
* has only two acceptable answers: the delivery becomes claimable again, or it becomes a question
|
||||
* for the provider. Never both, and never neither.
|
||||
* <p>The matrix walks the phases a dispatch passes through and pins, for each, the state recovery
|
||||
* starts from: a lease that has passed (the claim takes the row again) and, from the attempt insert
|
||||
* on, an unfinished attempt row carrying the evidence of that phase (what reconciliation asks the
|
||||
* provider about). These rows are written by this test, so the tests check the shape of the input
|
||||
* and the predicates recovery reads, not recovery itself. {@code
|
||||
* PostgreSqlNotificationBackgroundPassIntegrationTest} runs recovery and reconciliation over rows
|
||||
* the dispatcher's own factory wrote (review 05-01, RR-03-07).
|
||||
*/
|
||||
@Tag("jpa-contract")
|
||||
class WorkerCrashRecoveryContractTest {
|
||||
@@ -73,8 +78,8 @@ class WorkerCrashRecoveryContractTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a crash after claiming and before any attempt leaves the delivery claimable")
|
||||
void aCrashBeforeAnyAttemptLeavesItClaimable() throws SQLException {
|
||||
@DisplayName("claim 뒤 attempt 전에 죽은 delivery는 attempt 행이 없고 lease가 지나 있다")
|
||||
void aDeliveryDeadBeforeAnyAttemptHasNoAttemptRowAndAPassedLease() throws SQLException {
|
||||
UUID delivery = deliveryHeldByADeadWorker();
|
||||
|
||||
assertThat(isClaimableAgain(delivery))
|
||||
@@ -88,12 +93,12 @@ class WorkerCrashRecoveryContractTest {
|
||||
/**
|
||||
* The attempt row is written before the call and keeps the factory's "unknown" evidence for the
|
||||
* whole call (review 05-01), so a crash anywhere between the insert and the provider's answer
|
||||
* leaves the same row. Recovery cannot tell "never called" from "called and died", so the row
|
||||
* goes to reconciliation rather than back to the queue.
|
||||
* leaves the same row. This test writes that row itself; that recovery hands it to reconciliation
|
||||
* is {@code PostgreSqlNotificationBackgroundPassIntegrationTest}'s.
|
||||
*/
|
||||
@Test
|
||||
@DisplayName("attempt 행을 쓴 뒤 호출이 끝나기 전 죽으면 시작 여부 불명인 미완료 attempt 가 남아 reconcile 대상이 된다")
|
||||
void aCrashAfterTheAttemptRowIsWrittenLeavesAnUnfinishedAttemptOfUnknownStart()
|
||||
@DisplayName("attempt 행을 쓰고 호출 전에 죽은 delivery는 시작 여부 불명인 미완료 attempt와 지난 lease를 남긴다")
|
||||
void aDeliveryDeadAfterTheAttemptInsertLeavesAnUnfinishedAttemptOfUnknownStart()
|
||||
throws SQLException {
|
||||
// given
|
||||
UUID delivery = deliveryHeldByADeadWorker();
|
||||
@@ -103,17 +108,18 @@ class WorkerCrashRecoveryContractTest {
|
||||
|
||||
// then
|
||||
assertThat(evidenceOf(attempt))
|
||||
.as("the row the factory writes records that whether the call started is not known")
|
||||
.as("the row stands for the factory's: whether the call started is not known")
|
||||
.containsExactly("false", "UNKNOWN", "false", "UNKNOWN");
|
||||
assertThat(isUnfinished(attempt))
|
||||
.as("an unfinished attempt is what recovery hands to reconciliation")
|
||||
.as("completed_at is empty, the predicate recovery selects unfinished attempts by")
|
||||
.isTrue();
|
||||
assertThat(isClaimableAgain(delivery)).isTrue();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a crash after the request started is a question for the provider, not a requeue")
|
||||
void aCrashAfterTheRequestStartedIsAQuestion() throws SQLException {
|
||||
@DisplayName("요청 시작 뒤 죽은 delivery는 시작 증거가 있는 미완료 attempt와 지난 lease를 남긴다")
|
||||
void aDeliveryDeadAfterTheRequestStartedLeavesAnUnfinishedAttemptWithStartEvidence()
|
||||
throws SQLException {
|
||||
UUID delivery = deliveryHeldByADeadWorker();
|
||||
UUID attempt = attemptOn(delivery, true, "PROVEN", false, "UNKNOWN");
|
||||
|
||||
@@ -125,13 +131,14 @@ class WorkerCrashRecoveryContractTest {
|
||||
"the lease still expires — the row must not be stranded — but the attempt is unfinished")
|
||||
.isTrue();
|
||||
assertThat(isUnfinished(attempt))
|
||||
.as("an unfinished attempt is what recovery hands to reconciliation")
|
||||
.as("completed_at is empty, the predicate recovery selects unfinished attempts by")
|
||||
.isTrue();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a crash after the body was committed is never silently retried")
|
||||
void aCrashAfterTheBodyWasCommittedIsNeverSilentlyRetried() throws SQLException {
|
||||
@DisplayName("본문 전송 뒤 죽은 delivery는 전송 증거가 있는 미완료 attempt를 남긴다")
|
||||
void aDeliveryDeadAfterTheBodyWasSentLeavesAnUnfinishedAttemptWithSendEvidence()
|
||||
throws SQLException {
|
||||
UUID delivery = deliveryHeldByADeadWorker();
|
||||
UUID attempt = attemptOn(delivery, true, "PROVEN", true, "PROVEN");
|
||||
|
||||
@@ -142,8 +149,8 @@ class WorkerCrashRecoveryContractTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a completed attempt is not recovered at all")
|
||||
void aCompletedAttemptIsNotRecovered() throws SQLException {
|
||||
@DisplayName("완료 시각이 적힌 attempt는 미완료로 보이지 않는다")
|
||||
void aCompletedAttemptIsNotUnfinished() throws SQLException {
|
||||
UUID delivery = deliveryHeldByADeadWorker();
|
||||
UUID attempt = attemptOn(delivery, true, "PROVEN", true, "PROVEN");
|
||||
complete(attempt);
|
||||
@@ -154,8 +161,9 @@ class WorkerCrashRecoveryContractTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("across every phase, exactly one delivery row survives with one attempt each")
|
||||
void acrossEveryPhaseNothingIsLostOrDuplicated() throws SQLException {
|
||||
@DisplayName("네 단계에서 죽은 워커들은 delivery 4개와 attempt 3개를 남기고 모든 lease가 지나 있다")
|
||||
void deadWorkersAcrossEveryPhaseLeaveFourDeliveriesThreeAttemptsAndPassedLeases()
|
||||
throws SQLException {
|
||||
List<UUID> deliveries = new ArrayList<>();
|
||||
// The four crash phases, run together, so the recovery sweep sees a mixed table rather than one
|
||||
// hand-picked row.
|
||||
|
||||
+9
-2
@@ -59,7 +59,14 @@ import org.springframework.transaction.PlatformTransactionManager;
|
||||
class DurableOperationSubmissionOutboxIntegrationTest {
|
||||
|
||||
private static final Instant NOW = Instant.parse("2026-08-25T09:00:00Z");
|
||||
private static final DurableOperationId ID = new DurableOperationId("op-1");
|
||||
|
||||
/**
|
||||
* A canonical lower-case UUIDv7, as the HTTP adapter mints: the operation id is then the event id
|
||||
* and the aggregate id itself (an id of any other spelling becomes a UUIDv7 derived from it).
|
||||
*/
|
||||
private static final String OPERATION_UUID = "01994e11-4d88-7000-8000-0000000000a1";
|
||||
|
||||
private static final DurableOperationId ID = new DurableOperationId(OPERATION_UUID);
|
||||
|
||||
private static JpaPlatformContractSupport support;
|
||||
private static AnnotationConfigApplicationContext context;
|
||||
@@ -103,7 +110,7 @@ class DurableOperationSubmissionOutboxIntegrationTest {
|
||||
|
||||
// then
|
||||
assertThat(queryStrings("SELECT contract_id || '|' || aggregate_id FROM outbox_event"))
|
||||
.containsExactly("durable-operation.submitted|op-1");
|
||||
.containsExactly("durable-operation.submitted|" + OPERATION_UUID);
|
||||
}
|
||||
|
||||
@Test
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user