fix!: 성숙도 리뷰의 High·P1 결함 19건을 고친다
문제: 2026-11-01 성숙도 리뷰에서 High 21건·P1 25건이 나왔다. 결과를 모르는 일을 "확실함"으로 기록했다(발송 중 죽은 알림 재발송, 커밋 중 끊긴 쓰기를 재시도 가능으로 응답, 재시도 가능한 거부를 DEAD로). 중복을 DB가 아니라 코드 순서로 막았다(정산 job, 포스터 제목). 짝 프론트와 wire 계약이 달랐고(처리 중 409, Envelope null), JWT audience 검증이 기본으로 꺼져 있었고, 트레이스가 나가지 않았고, CI가 strict lane 14개를 한 번도 돌리지 않았다. 방법: 발견마다 실패하는 테스트를 먼저 쓰고 고쳤다. 주요 선택: - 알림 attempt는 처음부터 "시작 여부 모름"으로 쓴다(호출 직전 갱신은 쓰기가 하나 늘어 버림). - 정산 job claim은 lease·fence를 갱신하는 UPDATE 한 문장(처리 전체를 트랜잭션으로 감싸는 안은 외부 호출 동안 잠금을 잡아 버림). - 커밋 단계 연결 끊김은 DB_COMPLETION_UNKNOWN(재시도 불가). - 처리 중 409는 retryable=true + Retry-After: 2, Envelope null 필드는 web Jackson mixin으로 생략(shared-contract는 stdlib only). - 트레이싱은 자체 Tracer를 지우고 Boot 4 tracing·OTLP로. - strict lane을 단계에 연결하고 verifyStrictLaneStages가 고아를 막는다. 발견별 RED·GREEN·버린 대안은 리뷰 README 13절 표에 있다. 위험·한계: JWT 모드는 APP_SECURITY_JWT_AUDIENCE가 없으면 기동하지 않는다. 짝 프론트는 409 retryable과 Envelope 키 규칙을 맞춰야 한다. 이 커밋 시점에는 MinIO 공식 이미지가 사라져 MinIO 2 lane이 실패한다 (다음 커밋에서 대체 이미지로 해결). 검증: ci 통과. mainCheck·integrationCheck는 MinIO 2 lane만 실패(이미지 pull 거부). -p optional-platforms ci integrationCheck, performanceCheck, releaseCheck(-PgitRevision) 통과. Refs: docs/reviews/2026-11-01-maturity-review/README.md 2·13절 BREAKING CHANGE: JWT 모드에서 APP_SECURITY_JWT_AUDIENCE가 필수가 되고 role은 audience client와 realm 것만 읽는다. IDEMPOTENT_IN_FLIGHT가 retryable=true, Retry-After: 2로 바뀌고 Envelope는 null 필드를 보내지 않는다.
This commit is contained in:
@@ -292,8 +292,8 @@ application 계층이 claim·replay·복구 **정책** 을 소유하고, 저장
|
||||
- 모두 **프레임워크-free, application 소유**(`AuthorizationDeniedException` 선례). web adapter 가
|
||||
경계에서 HTTP 코드로 매핑한다. 진단 메시지는 scope digest 만 **로그용으로** 담고, 클라이언트에는
|
||||
핸들러가 고정 client-safe 메시지로 대체한다(fingerprint 자체는 노출 안 함).
|
||||
- retryable: mismatch / in-flight 모두 `false`. 클라이언트는 body 를 고치거나 결과를 polling 해야지
|
||||
단순 재시도를 하면 안 된다. recovery-required 는 재생도 재실행도 안전하지 않다 — 비즈니스
|
||||
- retryable: mismatch 는 `false`(body 를 고쳐야 한다). in-flight 는 `true`: 결과 불명. 같은
|
||||
`Idempotency-Key`로 `Retry-After` 뒤 재전송. 새 키 금지(`docs/web/http-contract.md` §2). recovery-required 는 재생도 재실행도 안전하지 않다 — 비즈니스
|
||||
데이터로만 판단할 수 있다.
|
||||
|
||||
---
|
||||
@@ -364,9 +364,20 @@ claim → 트랜잭션 밖에서 발행 → at-least-once 보장.
|
||||
2. `occurred_at` 오름차순으로 방어적 정렬(claim 쿼리도 정렬하지만, adapter 가 안 해도 relay 가
|
||||
FIFO 를 강제한다).
|
||||
3. 각 이벤트를 **트랜잭션 밖에서** 발행하고 결과로 상태 머신을 구동한다:
|
||||
- 발행 성공 → `inWrite { markPublished }` → `PUBLISHED`.
|
||||
- 발행 실패(`RuntimeException`): `attemptCount >= maxAttempts` 면 `markDead`, 아니면
|
||||
`markFailed(nextAttemptAt)`를 먼저 성공시킨 뒤 해당 typed failure report를 보낸다.
|
||||
- 발행 성공(`CONFIRMED`) → `inWrite { markPublished }` → `PUBLISHED`.
|
||||
- 확정 거부(`REJECTED_BEFORE_SEND`·`REJECTED_AFTER_BROKER`) → 시도 예산과 무관하게 바로 `markDead`.
|
||||
- 일시적 실패(`AMBIGUOUS`·`REJECTED_RETRYABLE`, 즉 `OutboxPublishOutcome.retryable()`) →
|
||||
시도 예산을 쓰지 않고 항상 `markFailed(nextAttemptAt)`. broker 정지·종료 중·permit 부족·
|
||||
credential 교체는 모든 행을 같은 방식으로 실패시키고 저절로 끝나므로, 예산을 쓰게 하면 몇 분
|
||||
장애가 aggregate 머리마다 DEAD 행을 남긴다(2026-11-01 리뷰 06-01·06-03).
|
||||
- 발행 예외(`RuntimeException`, adapter가 분류하지 못한 실패): `attemptCount >= maxAttempts` 면
|
||||
`markDead`, 아니면 `markFailed(nextAttemptAt)`. 예산은 이 경우에만 쓴다 — payload 자체 결함일
|
||||
수 있는 실패가 aggregate를 영원히 막지 않게 하는 장치다. `attemptCount`는 일시적 실패의 claim도
|
||||
센다. 그래서 장애를 오래 기다린 행은 그 뒤 첫 발행 예외에서 바로 DEAD가 된다(알려진 한계).
|
||||
- 전이를 먼저 성공시킨 뒤 해당 typed failure report를 보낸다.
|
||||
- `nextAttemptAt` 의 기준 시각은 **그 행이 실패한 시각**(`clock.instant()`)이다. 회차 시작
|
||||
시각을 쓰면 broker가 멈춘 동안 publish 하나가 deadline(약 5초)을 다 쓰므로 배치 뒤쪽 행은
|
||||
표시되는 순간 이미 재시도 시각이 지나 있어 지연이 사라진다.
|
||||
- **발행 실패는 절대 삼키지 않는다**: relay 는 각 발행 예외를 잡아 FAILED/DEAD 상태 머신을
|
||||
구동하고, 성공한 전이만 `OutboxRelayFailureReportPort`로 보고한 뒤 rethrow 하지 않는다
|
||||
(스케줄러 루프가 다음 이벤트로 계속 가야 하므로). 상태 전이가 실패하면 예외가 전파되고 report는
|
||||
@@ -399,10 +410,17 @@ claim → 트랜잭션 밖에서 발행 → at-least-once 보장.
|
||||
|
||||
- relay 재시도 스케줄링 정책(I10). 이름과 달리 "full jitter" 가 아니다 — full jitter 는 0 을 고를
|
||||
수 있고, relay 재시도가 실패한 그 순간에 다시 due 가 되면 안 된다.
|
||||
- 공식: `base = 30s`, `maxAttempts = 3`, `delay = ceiling(attemptCount) + U[0, base]`,
|
||||
`nextAttemptAt = now + delay`. `ceiling` 은 shared-contract `Backoff` 의
|
||||
`base × 2^(attemptCount-1)` 이고 마지막 시도 값(120s)에서 멈춘다. `attemptCount` 는 방금 실패한
|
||||
시도의 1-based 번호다.
|
||||
- 공식: `base = 30s`, `maxDelay = 5m`, `maxAttempts = 3`, `delay = ceiling(attemptCount) + U[0, base]`,
|
||||
`nextAttemptAt = failedAt + delay`. `ceiling` 은 shared-contract `Backoff` 의
|
||||
`base × 2^(attemptCount-1)` 이고 `maxDelay`(5분)에서 멈춘다. `attemptCount` 는 방금 실패한
|
||||
시도의 1-based 번호다. 예산 안(1·2회차)의 값은 30s·60s 그대로라 registry·runbook 숫자가 유지된다.
|
||||
- `maxAttempts` 는 발행 예외(분류되지 않은 실패)에만 적용된다. 일시적 결과(`retryable()`)는 예산 없이
|
||||
`maxDelay` 간격까지 늘어나며 계속 재시도된다. 그래서 broker가 돌아오면 늦어도 약 5분 30초 안에
|
||||
적체가 저절로 빠진다.
|
||||
- 버린 대안: `MAX_ATTEMPTS` 만 늘리기(장애 길이에 비례해 계속 키워야 하고 진짜 poison 행이 늦게
|
||||
격리된다), 예산을 설정 키로 외부화(I10 — registry·runbook과 어긋난다; 일시적 실패가 예산을 쓰지
|
||||
않으므로 장애 길이에 맞춰 예산을 조정할 이유도 사라졌다), 일시적 실패 시 `attempt_count` 되돌리기
|
||||
(store 포트·SQL·계약 테스트를 넓히고, 지수 간격을 키울 근거가 사라진다).
|
||||
- 더하는 spread 는 한 base 짜리 full jitter(`Backoff#fullJitter`, `[0, base]` 균등)이며 주입된
|
||||
`RandomGenerator` 로 뽑는다. 0 을 반환하는 generator 를 주면 테스트가 결정적이 된다(I10).
|
||||
- **상수는 고정이다(I10)**: `BASE_DELAY`·`MAX_ATTEMPTS`·jitter 범위는 설정 프로퍼티가 아니다.
|
||||
|
||||
+3
-2
@@ -6,8 +6,9 @@ import dev.caskeleton.shared.error.OperationalError;
|
||||
|
||||
/**
|
||||
* Raised when another attempt still holds the same {@link IdempotencyScopeDigest}. Framework-free,
|
||||
* application-owned; every transport maps it to a {@code IDEMPOTENT_IN_FLIGHT} 409 (retryable=false
|
||||
* — the client polls). The message carries only the opaque scope digest, for logs.
|
||||
* application-owned; every transport maps it to a {@code IDEMPOTENT_IN_FLIGHT} 409 (retryable=true
|
||||
* — the result is unknown and the client resends with the same key, never a new one; {@code
|
||||
* docs/web/http-contract.md}). The message carries only the opaque scope digest, for logs.
|
||||
*/
|
||||
public class IdempotencyInFlightException extends RuntimeException implements ApiErrorCarrier {
|
||||
|
||||
|
||||
+25
-8
@@ -9,8 +9,15 @@ import java.util.random.RandomGenerator;
|
||||
/**
|
||||
* Retry scheduling for the outbox relay: {@code delay = ceiling(attemptCount) + U[0, base]}, where
|
||||
* the ceiling is the shared {@link Backoff} exponential {@code base × 2^(attemptCount-1)} capped at
|
||||
* the last attempt's value, {@code base = 30s} and {@code maxAttempts = 3}; {@code attemptCount} is
|
||||
* the 1-based just-failed attempt.
|
||||
* {@link #MAX_DELAY}, {@code base = 30s}; {@code attemptCount} is the 1-based just-failed attempt.
|
||||
*
|
||||
* <p>The attempt budget {@link #MAX_ATTEMPTS} applies only to a failure that may be the message's
|
||||
* own fault — one the publish adapter could not classify and threw. A failure that says the broker
|
||||
* or the publisher is unavailable for now ({@link OutboxPublishOutcome#retryable()}) does not spend
|
||||
* it: it is retried for as long as it lasts, at most {@link #MAX_DELAY} apart, so an outage ends
|
||||
* with the backlog draining on its own instead of with a DEAD row at the head of every aggregate
|
||||
* that produced an event during it. Within the budget the schedule is what it always was (30s,
|
||||
* 60s), so the registry and runbook numbers still hold.
|
||||
*
|
||||
* <p>This is not "full jitter", although it used to be called that: full jitter ({@link
|
||||
* Backoff#fullJitter}) may choose zero, and a relay retry must never be due at the instant it
|
||||
@@ -23,12 +30,20 @@ public final class OutboxBackoffPolicy {
|
||||
/** Base retry delay (registry SSOT: retry_after_seconds=30). */
|
||||
public static final Duration BASE_DELAY = Duration.ofSeconds(30);
|
||||
|
||||
/** Maximum delivery attempts before a row is DEAD-lettered. */
|
||||
/**
|
||||
* The longest wait between two attempts: how late, at most, a backlog starts draining after the
|
||||
* broker comes back.
|
||||
*/
|
||||
public static final Duration MAX_DELAY = Duration.ofMinutes(5);
|
||||
|
||||
/**
|
||||
* Maximum delivery attempts before a row whose publish threw is DEAD-lettered. A transient
|
||||
* outcome ({@link OutboxPublishOutcome#retryable()}) is never dead-lettered by this budget.
|
||||
*/
|
||||
public static final int MAX_ATTEMPTS = 3;
|
||||
|
||||
/** The exponential part, capped where the last attempt already is. */
|
||||
private static final Backoff SCHEDULE =
|
||||
new Backoff(BASE_DELAY, BASE_DELAY.multipliedBy(1L << (MAX_ATTEMPTS - 1)));
|
||||
/** The exponential part, capped at {@link #MAX_DELAY}. */
|
||||
private static final Backoff SCHEDULE = new Backoff(BASE_DELAY, MAX_DELAY);
|
||||
|
||||
/** The spread: a full jitter of one base. */
|
||||
private static final Backoff SPREAD = new Backoff(BASE_DELAY, BASE_DELAY);
|
||||
@@ -46,7 +61,8 @@ public final class OutboxBackoffPolicy {
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the maximum number of delivery attempts before an event is dead-lettered.
|
||||
* Returns the maximum number of delivery attempts before an event whose publish threw is
|
||||
* dead-lettered.
|
||||
*
|
||||
* @return {@value #MAX_ATTEMPTS}
|
||||
*/
|
||||
@@ -59,7 +75,8 @@ public final class OutboxBackoffPolicy {
|
||||
*
|
||||
* @param attemptCount the 1-based attempt number that just failed (e.g. {@code 1} for the first
|
||||
* failure)
|
||||
* @param now the current wall-clock instant
|
||||
* @param now the instant the attempt failed — not the instant its relay cycle started, which on a
|
||||
* stalled broker is long enough ago that the retry would already be due
|
||||
* @return the earliest instant at which the event may be re-claimed
|
||||
* @throws IllegalArgumentException if {@code attemptCount < 1}
|
||||
*/
|
||||
|
||||
+22
-4
@@ -8,8 +8,8 @@ package dev.caskeleton.application.outbox;
|
||||
* retrying it may lose. Collapsing that into an exception forces the relay to choose one of those
|
||||
* two wrong answers, and it chooses the same one every time.
|
||||
*
|
||||
* <p>Four values, because two of them are the ones that matter operationally and the naming trap
|
||||
* between the two outbox models is real:
|
||||
* <p>Two of the values are the ones that matter operationally, because the naming trap between the
|
||||
* two outbox models is real:
|
||||
*
|
||||
* <ul>
|
||||
* <li>the legacy {@code OutboxEventStatus.FAILED} is <em>retryable</em> and {@code DEAD} is
|
||||
@@ -21,6 +21,10 @@ package dev.caskeleton.application.outbox;
|
||||
* <p>Mapping those by name inverts both meanings: a definitely-rejected message would be retried
|
||||
* forever and an unknown one would be parked. This type exists so no bridge is ever tempted to map
|
||||
* enum names.
|
||||
*
|
||||
* <p>A refusal is definite about the broker not holding the message, but not always about the
|
||||
* message being at fault: {@link #REJECTED_RETRYABLE} is the refusal whose cause clears on its own,
|
||||
* and it is retried like {@link #AMBIGUOUS} instead of being parked like the two definite ones.
|
||||
*/
|
||||
public enum OutboxPublishOutcome {
|
||||
|
||||
@@ -50,11 +54,25 @@ public enum OutboxPublishOutcome {
|
||||
* broker, so an operator investigating looks at broker-side policy — quota, ACL, message size —
|
||||
* rather than at this application's routing.
|
||||
*/
|
||||
REJECTED_AFTER_BROKER;
|
||||
REJECTED_AFTER_BROKER,
|
||||
|
||||
/**
|
||||
* Refused for a reason that belongs to the moment, not to the message.
|
||||
*
|
||||
* <p>The publisher is shutting down, its permits are exhausted, its transport is closed, or the
|
||||
* broker refused this producer's credentials while they are being rotated. The broker does not
|
||||
* hold the message, and the same payload may well be accepted a little later. Dead-lettering it
|
||||
* would park a committed business event, and every later event of its aggregate behind it, for a
|
||||
* cause that clears on its own.
|
||||
*
|
||||
* <p>Whether bytes left the process is not asserted: the refusal says only that the broker did
|
||||
* not store the message.
|
||||
*/
|
||||
REJECTED_RETRYABLE;
|
||||
|
||||
/** True when the relay may claim this row again. */
|
||||
public boolean retryable() {
|
||||
return this == AMBIGUOUS;
|
||||
return this == AMBIGUOUS || this == REJECTED_RETRYABLE;
|
||||
}
|
||||
|
||||
/** True when nothing was transmitted, so no duplicate can exist. */
|
||||
|
||||
+23
-11
@@ -3,11 +3,12 @@ package dev.caskeleton.application.outbox;
|
||||
/**
|
||||
* What the relay reports when a publish was refused without throwing.
|
||||
*
|
||||
* <p>An adapter that returns {@link OutboxPublishOutcome#REJECTED_BEFORE_SEND} or {@link
|
||||
* OutboxPublishOutcome#REJECTED_AFTER_BROKER} has already decided the attempt is over and has no
|
||||
* exception to hand up. The failure report needs a cause, and inventing a generic one would put a
|
||||
* stack trace from this class in an operator's hands with nothing in it about the refusal — so the
|
||||
* refusal itself is the cause, and it says which of the two it was.
|
||||
* <p>An adapter that returns {@link OutboxPublishOutcome#REJECTED_BEFORE_SEND}, {@link
|
||||
* OutboxPublishOutcome#REJECTED_AFTER_BROKER} or {@link OutboxPublishOutcome#REJECTED_RETRYABLE}
|
||||
* has already decided the attempt is over and has no exception to hand up. The failure report needs
|
||||
* a cause, and inventing a generic one would put a stack trace from this class in an operator's
|
||||
* hands with nothing in it about the refusal — so the refusal itself is the cause, and it says
|
||||
* which refusal it was.
|
||||
*
|
||||
* <p>Never thrown across the port. It exists only to carry a returned outcome into a report.
|
||||
*/
|
||||
@@ -23,15 +24,26 @@ public final class OutboxPublishRefusedException extends RuntimeException {
|
||||
* @param outcome the refusal the adapter reported
|
||||
*/
|
||||
public OutboxPublishRefusedException(OutboxPublishOutcome outcome) {
|
||||
super(
|
||||
outcome == OutboxPublishOutcome.REJECTED_BEFORE_SEND
|
||||
? "the broker adapter refused this event before transmitting it, so nothing was stored"
|
||||
+ " anywhere and the same payload will be refused again"
|
||||
: "the broker received this event and refused it; the cause is broker-side policy"
|
||||
+ " rather than this application's routing");
|
||||
super(describe(outcome));
|
||||
this.outcome = outcome;
|
||||
}
|
||||
|
||||
private static String describe(OutboxPublishOutcome outcome) {
|
||||
return switch (outcome) {
|
||||
case REJECTED_BEFORE_SEND ->
|
||||
"the broker adapter refused this event before transmitting it, so nothing was stored"
|
||||
+ " anywhere and the same payload will be refused again";
|
||||
case REJECTED_AFTER_BROKER ->
|
||||
"the broker received this event and refused it; the cause is broker-side policy"
|
||||
+ " rather than this application's routing";
|
||||
case REJECTED_RETRYABLE ->
|
||||
"the publisher refused this event for now (shutting down, saturated, transport closed or"
|
||||
+ " credentials refused); the broker does not hold it and it will be retried";
|
||||
case CONFIRMED, AMBIGUOUS ->
|
||||
throw new IllegalArgumentException(outcome + " is not a refusal");
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The refusal this cause carries.
|
||||
*
|
||||
|
||||
+56
-29
@@ -114,7 +114,7 @@ public class PublishPendingOutboxEventsUseCase
|
||||
// Step 3: Publish each event outside any transaction; drive status machine per result.
|
||||
List<OutboxRelayResult.EventOutcome> outcomes = new ArrayList<>(sorted.size());
|
||||
for (OutboxLease lease : sorted) {
|
||||
OutboxRelayResult.Outcome outcome = publishOne(lease, now);
|
||||
OutboxRelayResult.Outcome outcome = publishOne(lease);
|
||||
outcomes.add(
|
||||
new OutboxRelayResult.EventOutcome(lease.eventId(), lease.event().eventType(), outcome));
|
||||
}
|
||||
@@ -127,24 +127,38 @@ public class PublishPendingOutboxEventsUseCase
|
||||
* publish failure is caught here and never rethrown; a {@code markPublished} failure after a
|
||||
* successful publish is NOT caught — it propagates so the row is recovered via the in-flight
|
||||
* timeout. See README for both failure modes.
|
||||
*
|
||||
* <p>A retry is scheduled from the instant the publish failed, read from the clock after it
|
||||
* returned. The cycle's start is the wrong origin: against a stalled broker each publish spends
|
||||
* its whole deadline, so the last rows of a batch fail long after the cycle began and a delay
|
||||
* measured from its start is already over when they are marked — they are retried at once, and
|
||||
* the delay the policy promised never happens.
|
||||
*/
|
||||
private OutboxRelayResult.Outcome publishOne(OutboxLease lease, Instant now) {
|
||||
private OutboxRelayResult.Outcome publishOne(OutboxLease lease) {
|
||||
ClaimedOutboxEvent event = lease.event();
|
||||
OutboxPublishOutcome achieved;
|
||||
try {
|
||||
// The four-valued call, not the throwing one. An adapter that cannot tell an ambiguous
|
||||
// The outcome-valued call, not the throwing one. An adapter that cannot tell an ambiguous
|
||||
// publish apart still throws and is handled below exactly as before; one that can no longer
|
||||
// has to pretend a message it may have stored was definitely not stored.
|
||||
achieved = publishPort.publishForOutcome(event);
|
||||
} catch (RuntimeException publishEx) {
|
||||
// Publish failure: drive FAILED/DEAD state machine + typed report; do NOT rethrow.
|
||||
return handlePublishFailure(lease, now, publishEx);
|
||||
return handlePublishFailure(lease, clock.instant(), publishEx);
|
||||
}
|
||||
if (achieved == OutboxPublishOutcome.AMBIGUOUS) {
|
||||
// The message left this process and no acknowledgement arrived, so it may be stored. Treated
|
||||
// as retryable and reported as such: the duplicate risk is the caller's to bound with an
|
||||
// idempotency key, and the alternative — parking it — loses a message that was delivered.
|
||||
return handlePublishFailure(lease, now, new OutboxPublishAmbiguousException());
|
||||
// It does not spend the attempt budget: a broker that stopped answering fails every row this
|
||||
// way, and parking them after three tries is what turned a short outage into DEAD rows.
|
||||
return retryLater(lease, clock.instant(), new OutboxPublishAmbiguousException());
|
||||
}
|
||||
if (achieved == OutboxPublishOutcome.REJECTED_RETRYABLE) {
|
||||
// Refused for a reason that is about the moment — shutdown, saturation, a closed transport,
|
||||
// a credential being rotated — not about the message. Retried like an ambiguous outcome and
|
||||
// for the same reason; the cause says it was a refusal, so no duplicate is suspected.
|
||||
return retryLater(lease, clock.instant(), new OutboxPublishRefusedException(achieved));
|
||||
}
|
||||
if (achieved != OutboxPublishOutcome.CONFIRMED) {
|
||||
// Definite refusal. Retrying the same payload fails the same way, so the attempt budget is
|
||||
@@ -160,35 +174,48 @@ public class PublishPendingOutboxEventsUseCase
|
||||
}
|
||||
|
||||
/**
|
||||
* Drives the FAILED/DEAD state transition and reports it only after persistence succeeds. A
|
||||
* transition failure remains authoritative and propagates without a report. A diagnostic adapter
|
||||
* failure is contained and cannot change the persisted outcome. See README.
|
||||
* Drives the FAILED/DEAD state transition for a publish that threw, and reports it only after
|
||||
* persistence succeeds. A transition failure remains authoritative and propagates without a
|
||||
* report. A diagnostic adapter failure is contained and cannot change the persisted outcome. See
|
||||
* README.
|
||||
*
|
||||
* <p>The attempt budget is spent here and only here: a thrown failure is one the adapter could
|
||||
* not classify, so it may be the message's own fault, and the budget is what keeps such a row
|
||||
* from blocking its aggregate forever. {@code attemptCount} counts every claim, transient ones
|
||||
* included, so a row that has already waited out an outage is dead-lettered by the first thrown
|
||||
* failure after it.
|
||||
*/
|
||||
private OutboxRelayResult.Outcome handlePublishFailure(
|
||||
OutboxLease lease, Instant now, RuntimeException cause) {
|
||||
ClaimedOutboxEvent event = lease.event();
|
||||
|
||||
if (event.attemptCount() >= backoffPolicy.maxAttempts()) {
|
||||
OutboxLease lease, Instant failedAt, RuntimeException cause) {
|
||||
if (lease.event().attemptCount() >= backoffPolicy.maxAttempts()) {
|
||||
// All attempts exhausted — DEAD-letter the event.
|
||||
return deadLetter(lease, cause);
|
||||
} else {
|
||||
// Transient failure — schedule retry with exponential backoff.
|
||||
Instant nextAttemptAt = backoffPolicy.nextAttemptAt(event.attemptCount(), now);
|
||||
if (tx.inWrite(() -> store.markFailed(lease, nextAttemptAt)) == OutboxCompletion.STALE) {
|
||||
return stale(lease, OutboxRelayResult.Outcome.FAILED);
|
||||
}
|
||||
reportFailure(
|
||||
() ->
|
||||
OutboxRelayFailureReport.retryableFailure(
|
||||
event.eventId(),
|
||||
event.eventType(),
|
||||
event.aggregateId(),
|
||||
event.correlationId(),
|
||||
event.attemptCount(),
|
||||
nextAttemptAt,
|
||||
cause));
|
||||
return OutboxRelayResult.Outcome.FAILED;
|
||||
}
|
||||
return retryLater(lease, failedAt, cause);
|
||||
}
|
||||
|
||||
/**
|
||||
* Marks the row FAILED with its next attempt time and reports it. Never dead-letters: the caller
|
||||
* has decided this failure is worth another attempt.
|
||||
*/
|
||||
private OutboxRelayResult.Outcome retryLater(
|
||||
OutboxLease lease, Instant failedAt, RuntimeException cause) {
|
||||
ClaimedOutboxEvent event = lease.event();
|
||||
Instant nextAttemptAt = backoffPolicy.nextAttemptAt(event.attemptCount(), failedAt);
|
||||
if (tx.inWrite(() -> store.markFailed(lease, nextAttemptAt)) == OutboxCompletion.STALE) {
|
||||
return stale(lease, OutboxRelayResult.Outcome.FAILED);
|
||||
}
|
||||
reportFailure(
|
||||
() ->
|
||||
OutboxRelayFailureReport.retryableFailure(
|
||||
event.eventId(),
|
||||
event.eventType(),
|
||||
event.aggregateId(),
|
||||
event.correlationId(),
|
||||
event.attemptCount(),
|
||||
nextAttemptAt,
|
||||
cause));
|
||||
return OutboxRelayResult.Outcome.FAILED;
|
||||
}
|
||||
|
||||
private OutboxRelayResult.Outcome deadLetter(OutboxLease lease, RuntimeException cause) {
|
||||
|
||||
+17
-8
@@ -2,6 +2,7 @@ package dev.caskeleton.application.outbox;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import java.util.List;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
@@ -17,14 +18,19 @@ import org.junit.jupiter.api.Test;
|
||||
class OutboxPublishOutcomeTest {
|
||||
|
||||
@Test
|
||||
@DisplayName("only an ambiguous outcome is retryable")
|
||||
void onlyAmbiguousIsRetryable() {
|
||||
assertThat(OutboxPublishOutcome.AMBIGUOUS.retryable()).isTrue();
|
||||
assertThat(OutboxPublishOutcome.CONFIRMED.retryable()).isFalse();
|
||||
assertThat(OutboxPublishOutcome.REJECTED_BEFORE_SEND.retryable())
|
||||
.as("a definite rejection retried forever is a queue that never drains")
|
||||
.isFalse();
|
||||
assertThat(OutboxPublishOutcome.REJECTED_AFTER_BROKER.retryable()).isFalse();
|
||||
@DisplayName("모를 결과와 지금만 거부된 결과만 다시 시도할 수 있다")
|
||||
void onlyAnAmbiguousOrARetryableRefusalIsRetryable() {
|
||||
// given
|
||||
List<OutboxPublishOutcome> outcomes = List.of(OutboxPublishOutcome.values());
|
||||
|
||||
// when
|
||||
List<OutboxPublishOutcome> retryable =
|
||||
outcomes.stream().filter(OutboxPublishOutcome::retryable).toList();
|
||||
|
||||
// then: a definite rejection retried forever is a queue that never drains
|
||||
assertThat(retryable)
|
||||
.containsExactlyInAnyOrder(
|
||||
OutboxPublishOutcome.AMBIGUOUS, OutboxPublishOutcome.REJECTED_RETRYABLE);
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -37,6 +43,9 @@ class OutboxPublishOutcomeTest {
|
||||
assertThat(OutboxPublishOutcome.AMBIGUOUS.nothingTransmitted())
|
||||
.as("the whole point of AMBIGUOUS is that this cannot be asserted")
|
||||
.isFalse();
|
||||
assertThat(OutboxPublishOutcome.REJECTED_RETRYABLE.nothingTransmitted())
|
||||
.as("a retryable refusal says the broker did not store it, not where the bytes went")
|
||||
.isFalse();
|
||||
}
|
||||
|
||||
@Test
|
||||
|
||||
+207
@@ -0,0 +1,207 @@
|
||||
package dev.caskeleton.application.outbox;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import dev.caskeleton.application.outbox.PublishPendingOutboxEventsUseCaseTest.MutableClock;
|
||||
import dev.caskeleton.application.outbox.PublishPendingOutboxEventsUseCaseTest.ZeroRandom;
|
||||
import dev.caskeleton.application.outbox.testkit.InMemoryOutboxStore;
|
||||
import dev.caskeleton.application.outbox.testkit.OutboxEvents;
|
||||
import dev.caskeleton.application.transaction.testkit.DirectTransactionPort;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* What the relay does with a failure that says nothing about the message.
|
||||
*
|
||||
* <p>A broker that is down, a publisher that is shutting down, a credential being rotated: each one
|
||||
* fails every row the same way for as long as it lasts and then stops. Spending the attempt budget
|
||||
* on them turned a five-minute broker outage into a DEAD row at the head of every aggregate that
|
||||
* produced an event during it, and nothing restarted them once the broker came back.
|
||||
*/
|
||||
class TransientPublishFailureRelayTest {
|
||||
|
||||
private static final Instant START = Instant.parse("2026-11-01T09:00:00Z");
|
||||
|
||||
private static final Duration IN_FLIGHT_TIMEOUT = Duration.ofMinutes(15);
|
||||
|
||||
/** What one publish against a stopped broker costs before it gives up. */
|
||||
private static final Duration PUBLISH_DEADLINE = Duration.ofSeconds(5);
|
||||
|
||||
private InMemoryOutboxStore store;
|
||||
|
||||
private MutableClock clock;
|
||||
|
||||
private ScriptedPort port;
|
||||
|
||||
private List<OutboxRelayFailureReport> reports;
|
||||
|
||||
private PublishPendingOutboxEventsUseCase relay;
|
||||
|
||||
@BeforeEach
|
||||
void setUp() {
|
||||
store = new InMemoryOutboxStore();
|
||||
clock = new MutableClock(START, ZoneOffset.UTC);
|
||||
port = new ScriptedPort();
|
||||
reports = new ArrayList<>();
|
||||
relay =
|
||||
new PublishPendingOutboxEventsUseCase(
|
||||
store,
|
||||
port,
|
||||
reports::add,
|
||||
new DirectTransactionPort(),
|
||||
new OutboxBackoffPolicy(new ZeroRandom()),
|
||||
clock,
|
||||
20,
|
||||
IN_FLIGHT_TIMEOUT,
|
||||
"relay-under-test");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("다시 시도하면 될 거부는 DEAD가 아니라 FAILED로 남아 재시도 시각을 받는다")
|
||||
void aRetryableRefusalIsScheduledForRetryInsteadOfDeadLettered() {
|
||||
// given: the platform refused the publish because it is shutting down
|
||||
store.addClaimable(OutboxEvents.claimed("evt-shutdown", START.minusSeconds(1), 1));
|
||||
port.answer = OutboxPublishOutcome.REJECTED_RETRYABLE;
|
||||
|
||||
// when
|
||||
OutboxRelayResult result = relay.handle(PublishPendingOutboxEventsCommand.INSTANCE);
|
||||
|
||||
// then: the row waits for the next attempt and is not parked
|
||||
assertThat(result.outcomes())
|
||||
.extracting(OutboxRelayResult.EventOutcome::outcome)
|
||||
.containsExactly(OutboxRelayResult.Outcome.FAILED);
|
||||
assertThat(store.dead()).isEmpty();
|
||||
assertThat(store.status("evt-shutdown")).isEqualTo(OutboxEventStatus.FAILED);
|
||||
assertThat(reports)
|
||||
.singleElement()
|
||||
.extracting(OutboxRelayFailureReport::nextAttemptAt)
|
||||
.isNotNull();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("일시적 실패는 시도 예산을 다 쓴 행도 DEAD로 보내지 않는다")
|
||||
void aTransientFailureNeverDeadLettersEvenPastTheAttemptBudget() {
|
||||
// given: two rows already at the last budgeted attempt, one refused for now, one ambiguous
|
||||
int lastBudgetedAttempt = OutboxBackoffPolicy.MAX_ATTEMPTS;
|
||||
store.addClaimable(
|
||||
OutboxEvents.claimed("evt-refused", START.minusSeconds(2), lastBudgetedAttempt));
|
||||
store.addClaimable(
|
||||
OutboxEvents.claimed("evt-ambiguous", START.minusSeconds(1), lastBudgetedAttempt));
|
||||
port.script("evt-refused", OutboxPublishOutcome.REJECTED_RETRYABLE);
|
||||
port.script("evt-ambiguous", OutboxPublishOutcome.AMBIGUOUS);
|
||||
|
||||
// when
|
||||
relay.handle(PublishPendingOutboxEventsCommand.INSTANCE);
|
||||
|
||||
// then: the budget isolates a payload the adapter could not classify, not a broker outage
|
||||
assertThat(store.dead()).isEmpty();
|
||||
assertThat(store.failed()).containsOnlyKeys("evt-refused", "evt-ambiguous");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("재시도 시각은 회차 시작이 아니라 그 행이 실패한 시각부터 잰다")
|
||||
void theRetryIsScheduledFromWhenThePublishFailedNotFromWhenTheCycleStarted() {
|
||||
// given: each publish against the stopped broker takes the whole deadline
|
||||
store.addClaimable(OutboxEvents.claimed("evt-first", START.minusSeconds(3), 1));
|
||||
store.addClaimable(OutboxEvents.claimed("evt-last", START.minusSeconds(2), 1));
|
||||
port.answer = OutboxPublishOutcome.AMBIGUOUS;
|
||||
port.costPerPublish = PUBLISH_DEADLINE;
|
||||
|
||||
// when
|
||||
relay.handle(PublishPendingOutboxEventsCommand.INSTANCE);
|
||||
|
||||
// then: the second row failed ten seconds into the cycle and waits a full delay from there
|
||||
Duration firstDelay = OutboxBackoffPolicy.BASE_DELAY;
|
||||
assertThat(store.nextAttemptAt("evt-first"))
|
||||
.isEqualTo(START.plus(PUBLISH_DEADLINE).plus(firstDelay));
|
||||
assertThat(store.nextAttemptAt("evt-last"))
|
||||
.isEqualTo(START.plus(PUBLISH_DEADLINE.multipliedBy(2)).plus(firstDelay));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("broker가 10분 멈췄다가 돌아오면 운영자 개입 없이 모든 행이 PUBLISHED가 된다")
|
||||
void aTenMinuteBrokerOutageDrainsOnItsOwnOnceTheBrokerReturns() {
|
||||
// given: twenty events committed while the broker is down for ten minutes
|
||||
for (int i = 0; i < 20; i++) {
|
||||
store.addClaimable(OutboxEvents.claimed("evt-" + i, START.minusSeconds(60 - i), 1));
|
||||
}
|
||||
Instant brokerBack = START.plus(Duration.ofMinutes(10));
|
||||
port.answerUntil(brokerBack, OutboxPublishOutcome.AMBIGUOUS);
|
||||
port.costPerPublish = PUBLISH_DEADLINE;
|
||||
|
||||
// when: the scheduler keeps polling every five seconds for half an hour
|
||||
runRelayEvery(Duration.ofSeconds(5), START.plus(Duration.ofMinutes(30)));
|
||||
|
||||
// then
|
||||
assertThat(store.dead()).as("nothing waits for the dead-letter runbook").isEmpty();
|
||||
assertThat(store.published()).hasSize(20);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("일시적 실패가 이어지면 재시도 간격이 늘어나되 상한을 넘지 않는다")
|
||||
void repeatedTransientFailuresBackOffUpToTheCeiling() {
|
||||
// given: a row that has already been tried many times during a long outage
|
||||
store.addClaimable(OutboxEvents.claimed("evt-long-outage", START.minusSeconds(1), 12));
|
||||
port.answer = OutboxPublishOutcome.AMBIGUOUS;
|
||||
|
||||
// when
|
||||
relay.handle(PublishPendingOutboxEventsCommand.INSTANCE);
|
||||
|
||||
// then
|
||||
assertThat(store.status("evt-long-outage")).isEqualTo(OutboxEventStatus.FAILED);
|
||||
assertThat(store.nextAttemptAt("evt-long-outage"))
|
||||
.isEqualTo(START.plus(OutboxBackoffPolicy.MAX_DELAY));
|
||||
}
|
||||
|
||||
private void runRelayEvery(Duration pollInterval, Instant until) {
|
||||
while (clock.instant().isBefore(until)) {
|
||||
relay.handle(PublishPendingOutboxEventsCommand.INSTANCE);
|
||||
clock.advance(pollInterval);
|
||||
}
|
||||
}
|
||||
|
||||
/** Answers per event, or until a moment on the relay's clock, and spends time doing it. */
|
||||
private final class ScriptedPort implements OutboxMessagePublishPort {
|
||||
|
||||
private final Map<String, OutboxPublishOutcome> scripted = new HashMap<>();
|
||||
|
||||
private OutboxPublishOutcome answer = OutboxPublishOutcome.CONFIRMED;
|
||||
|
||||
private Instant answerUntil = Instant.MAX;
|
||||
|
||||
private Duration costPerPublish = Duration.ZERO;
|
||||
|
||||
void script(String eventId, OutboxPublishOutcome outcome) {
|
||||
scripted.put(eventId, outcome);
|
||||
}
|
||||
|
||||
void answerUntil(Instant until, OutboxPublishOutcome outcome) {
|
||||
this.answer = outcome;
|
||||
this.answerUntil = until;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void publish(ClaimedOutboxEvent event) {
|
||||
throw new UnsupportedOperationException("the relay asks for an outcome");
|
||||
}
|
||||
|
||||
@Override
|
||||
public OutboxPublishOutcome publishForOutcome(ClaimedOutboxEvent event) {
|
||||
Instant calledAt = clock.instant();
|
||||
clock.advance(costPerPublish);
|
||||
OutboxPublishOutcome outcome = scripted.get(event.eventId());
|
||||
if (outcome != null) {
|
||||
return outcome;
|
||||
}
|
||||
return calledAt.isBefore(answerUntil) ? answer : OutboxPublishOutcome.CONFIRMED;
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user