fix!: 성숙도 리뷰의 High·P1 결함 19건을 고친다

문제: 2026-11-01 성숙도 리뷰에서 High 21건·P1 25건이 나왔다. 결과를
모르는 일을 "확실함"으로 기록했다(발송 중 죽은 알림 재발송, 커밋 중
끊긴 쓰기를 재시도 가능으로 응답, 재시도 가능한 거부를 DEAD로). 중복을
DB가 아니라 코드 순서로 막았다(정산 job, 포스터 제목). 짝 프론트와
wire 계약이 달랐고(처리 중 409, Envelope null), JWT audience 검증이
기본으로 꺼져 있었고, 트레이스가 나가지 않았고, CI가 strict lane 14개를
한 번도 돌리지 않았다.

방법: 발견마다 실패하는 테스트를 먼저 쓰고 고쳤다. 주요 선택:
- 알림 attempt는 처음부터 "시작 여부 모름"으로 쓴다(호출 직전 갱신은
  쓰기가 하나 늘어 버림).
- 정산 job claim은 lease·fence를 갱신하는 UPDATE 한 문장(처리 전체를
  트랜잭션으로 감싸는 안은 외부 호출 동안 잠금을 잡아 버림).
- 커밋 단계 연결 끊김은 DB_COMPLETION_UNKNOWN(재시도 불가).
- 처리 중 409는 retryable=true + Retry-After: 2, Envelope null 필드는
  web Jackson mixin으로 생략(shared-contract는 stdlib only).
- 트레이싱은 자체 Tracer를 지우고 Boot 4 tracing·OTLP로.
- strict lane을 단계에 연결하고 verifyStrictLaneStages가 고아를 막는다.
발견별 RED·GREEN·버린 대안은 리뷰 README 13절 표에 있다.

위험·한계: JWT 모드는 APP_SECURITY_JWT_AUDIENCE가 없으면 기동하지
않는다. 짝 프론트는 409 retryable과 Envelope 키 규칙을 맞춰야 한다.
이 커밋 시점에는 MinIO 공식 이미지가 사라져 MinIO 2 lane이 실패한다
(다음 커밋에서 대체 이미지로 해결).

검증: ci 통과. mainCheck·integrationCheck는 MinIO 2 lane만 실패(이미지
pull 거부). -p optional-platforms ci integrationCheck, performanceCheck,
releaseCheck(-PgitRevision) 통과.

Refs: docs/reviews/2026-11-01-maturity-review/README.md 2·13절
BREAKING CHANGE: JWT 모드에서 APP_SECURITY_JWT_AUDIENCE가 필수가 되고 role은
audience client와 realm 것만 읽는다. IDEMPOTENT_IN_FLIGHT가 retryable=true,
Retry-After: 2로 바뀌고 Envelope는 null 필드를 보내지 않는다.
This commit is contained in:
DongHyeonka
2026-11-03 19:52:06 +09:00
parent 01a8a2a70f
commit 34565130e0
253 changed files with 13849 additions and 1140 deletions
@@ -17,8 +17,9 @@ import java.util.Optional;
* Builds the attempt row that is committed <em>before</em> the provider call.
*
* <p>That ordering is the whole recovery story: after a crash the row is either absent, in which
* case nothing was sent, or present in {@code DISPATCHING}, in which case reconciliation has
* something to ask about.
* case nothing was sent, or present in {@code DISPATCHING} and unfinished, in which case
* reconciliation has something to ask about. The row therefore records that whether the call
* started is unknown: it stays the stored state for the whole call.
*/
public final class DeliveryAttemptFactory {
@@ -51,9 +52,12 @@ public final class DeliveryAttemptFactory {
profile.providerId(),
profile.profileId(),
Optional.empty(),
// A row that exists before any call: nothing started, and that is a proven fact rather than
// an unknown one.
ProviderExecutionEvidence.notStarted(),
// Not notStarted(): that is true when the row is inserted and false the moment the call
// begins, yet nothing rewrites the row until the outcome is recorded. A worker killed
// during the call would leave "proven never started", and recovery would requeue — a
// second send. Proven not-started is written only by closedBeforeSubmission and by the
// outcome of an adapter that raised ProviderCallNotStartedException; both complete the row.
ProviderExecutionEvidence.undetermined(),
SubmissionOutcome.NOT_SUBMITTED,
DeliveryOutcome.UNKNOWN,
AttemptConfirmation.AMBIGUOUS,
@@ -138,7 +138,9 @@ public final class LeaseRecoveryService {
// is nothing for a provider to tell us, so asking costs a round trip to learn what is
// already written down. This branch only became possible once the certainty survived
// persistence: before, "proven not started" and "unknown whether started" read back
// identically, so every abandoned attempt had to be reconciled.
// identically, so every abandoned attempt had to be reconciled. The pre-call row the
// dispatcher writes never lands here: it records the start as unknown, because it is
// still the stored state while the provider call runs.
requeue(recipientDeliveryId);
} else {
// Outside any transaction: reconciliation asks the provider over the network and opens
@@ -20,6 +20,10 @@ import java.util.UUID;
* duplicate questions: re-scheduling one that already exists moves its due time rather than adding
* a second row.
*
* <p>{@code claimFence} is which claim this copy was handed out under. Every claim advances it, and
* the store's writes after the claim match on it, so an instance that held the job past its lease
* cannot reschedule or remove it once another instance has claimed it since.
*
* @param id the job identity
* @param attemptId the attempt whose outcome is unknown
* @param providerProfileId the profile that must answer, since capability is per profile
@@ -28,6 +32,7 @@ import java.util.UUID;
* @param lastResult the previous answer, for operators reading the table directly
* @param createdAt when the job was first scheduled
* @param updatedAt when it last changed
* @param claimFence the claim this copy belongs to; {@code 0} for a job nobody has claimed yet
*/
public record ReconciliationJob(
UUID id,
@@ -37,7 +42,8 @@ public record ReconciliationJob(
int attempts,
Optional<String> lastResult,
Instant createdAt,
Instant updatedAt) {
Instant updatedAt,
long claimFence) {
/** Validates the job. */
public ReconciliationJob {
@@ -51,5 +57,8 @@ public record ReconciliationJob(
if (attempts < 0) {
throw new IllegalArgumentException("attempts must not be negative");
}
if (claimFence < 0) {
throw new IllegalArgumentException("claimFence must not be negative");
}
}
}
@@ -2,6 +2,7 @@ package dev.caskeleton.application.notification.platform.dispatch;
import dev.caskeleton.application.notification.platform.api.DeliveryAttemptId;
import dev.caskeleton.application.notification.platform.api.ProviderProfileId;
import java.time.Duration;
import java.time.Instant;
import java.util.List;
@@ -29,18 +30,28 @@ public interface ReconciliationJobStorePort {
DeliveryAttemptId attemptId, ProviderProfileId providerProfileId, Instant dueAt);
/**
* Claims due jobs for one worker pass.
* Claims due jobs for one worker pass, for {@code lease}.
*
* <p>The claim is ownership that outlives the statement that took it: until {@code now + lease}
* no other caller is handed the same job, whether or not a transaction is open. That is what lets
* the caller ask a provider about each job with no database transaction held across the call. A
* job whose lease has passed is claimable again, so the lease must cover the whole pass.
*
* @param limit how many to take
* @param now the current instant
* @return the due jobs, oldest first
* @param lease how long the caller owns the claimed jobs
* @return the due jobs, oldest first, each carrying the fence of this claim
*/
List<ReconciliationJob> claimDue(int limit, Instant now);
List<ReconciliationJob> claimDue(int limit, Instant now, Duration lease);
/**
* Records that a job was asked and answered inconclusively, and when to ask again.
*
* @param job the job
* <p>Fenced: when another caller has claimed the job since {@code job} was handed out, nothing
* changes, because the new holder's answer is the one that counts. The claim is released so the
* job becomes due again at {@code nextCheckAt} rather than at the end of the lease.
*
* @param job the job, as claimed
* @param result a bounded description of the answer
* @param nextCheckAt when to ask next
*/
@@ -49,7 +60,9 @@ public interface ReconciliationJobStorePort {
/**
* Removes a job whose attempt now has a final outcome.
*
* @param job the job
* <p>Fenced like {@link #reschedule}: a job claimed by another caller since is left alone.
*
* @param job the job, as claimed
*/
void complete(ReconciliationJob job);
}
@@ -25,6 +25,17 @@ public record ProviderExecutionEvidence(
EvidenceFact.provenAbsent());
}
/**
* Nothing is known yet, including whether a request began.
*
* <p>The state of a row written before a call and still stored while the call runs: at any later
* moment the call may have started, so the row must not claim it did not.
*/
public static ProviderExecutionEvidence undetermined() {
return new ProviderExecutionEvidence(
EvidenceFact.UNKNOWN, EvidenceFact.UNKNOWN, EvidenceFact.UNKNOWN, EvidenceFact.UNKNOWN);
}
/** A complete response was read and understood. */
public static ProviderExecutionEvidence completed(boolean accepted) {
return new ProviderExecutionEvidence(
@@ -1,115 +1,55 @@
package dev.caskeleton.application.notification.platform.dispatch;
import static dev.caskeleton.application.notification.platform.dispatch.DeliveryAttemptFixtures.NOW;
import static dev.caskeleton.application.notification.platform.dispatch.DeliveryAttemptFixtures.profile;
import static dev.caskeleton.application.notification.platform.dispatch.DeliveryAttemptFixtures.recipient;
import static dev.caskeleton.application.notification.platform.dispatch.DeliveryAttemptFixtures.rendered;
import static dev.caskeleton.application.notification.platform.dispatch.DeliveryAttemptFixtures.route;
import static org.assertj.core.api.Assertions.assertThat;
import dev.caskeleton.application.notification.platform.api.ContactPointId;
import dev.caskeleton.application.notification.platform.api.NotificationId;
import dev.caskeleton.application.notification.platform.api.ProviderId;
import dev.caskeleton.application.notification.platform.api.ProviderProfileId;
import dev.caskeleton.application.notification.platform.api.RecipientDeliveryId;
import dev.caskeleton.application.notification.platform.api.TemplateSelection;
import dev.caskeleton.application.notification.platform.api.TenantId;
import dev.caskeleton.application.notification.platform.api.content.EmailContent;
import dev.caskeleton.application.notification.platform.api.content.EmailOptions;
import dev.caskeleton.application.notification.platform.api.delivery.AttemptConfirmation;
import dev.caskeleton.application.notification.platform.api.delivery.DeliveryOutcome;
import dev.caskeleton.application.notification.platform.api.delivery.EvidenceLevel;
import dev.caskeleton.application.notification.platform.api.delivery.RecipientDeliveryState;
import dev.caskeleton.application.notification.platform.api.delivery.SubmissionOutcome;
import dev.caskeleton.application.notification.platform.api.routing.Channel;
import dev.caskeleton.application.notification.platform.policy.RouteCandidate;
import dev.caskeleton.application.notification.platform.provider.ProviderCapabilities;
import dev.caskeleton.application.notification.platform.provider.EvidenceCertainty;
import dev.caskeleton.application.notification.platform.provider.ProviderExecutionEvidence;
import dev.caskeleton.application.notification.platform.provider.ProviderProfileSnapshot;
import dev.caskeleton.application.notification.platform.template.RenderedNotificationContent;
import java.time.Duration;
import java.time.Instant;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Optional;
import java.util.UUID;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
class DeliveryAttemptFactoryTest {
private static final Instant NOW = Instant.parse("2026-09-20T00:00:00Z");
private static final UUID ATTEMPT_ID = UUID.fromString("00000000-0000-7000-8000-000000000010");
@Test
void createsAProvenNotStartedRowBeforeTheProviderCall() {
@DisplayName("provider 호출 전에 만드는 attempt 행은 요청 시작 여부를 불명으로 적는다")
void createsARowThatDoesNotClaimTheCallNeverStarted() {
// given
RouteCandidate route = route();
ProviderProfileSnapshot profile = profile();
DeliveryAttemptFactory factory = new DeliveryAttemptFactory(() -> ATTEMPT_ID);
// when
DeliveryAttemptRecord attempt =
factory.create(recipient(route), route, profile, rendered(), 2, NOW);
// then
assertThat(attempt.id().value()).isEqualTo(ATTEMPT_ID);
assertThat(attempt.attemptNo()).isEqualTo(2);
assertThat(attempt.submissionOutcome()).isEqualTo(SubmissionOutcome.NOT_SUBMITTED);
assertThat(attempt.deliveryOutcome()).isEqualTo(DeliveryOutcome.UNKNOWN);
assertThat(attempt.confirmation()).isEqualTo(AttemptConfirmation.AMBIGUOUS);
assertThat(attempt.evidenceLevel()).isEqualTo(EvidenceLevel.PLATFORM_QUEUED);
assertThat(attempt.executionEvidence()).isEqualTo(ProviderExecutionEvidence.notStarted());
assertThat(attempt.executionEvidence().requestStarted().certainty())
.as(
"this row outlives the moment it was written: it is still the stored state while the"
+ " provider call runs, so it must not say the call provably never started")
.isEqualTo(EvidenceCertainty.UNKNOWN);
assertThat(attempt.executionEvidence()).isEqualTo(ProviderExecutionEvidence.undetermined());
assertThat(attempt.providerRequestId()).isEmpty();
assertThat(attempt.credentialGeneration()).isEqualTo(7L);
assertThat(attempt.renderedContentDigest()).isEqualTo("d".repeat(64));
}
private static RouteCandidate route() {
return new RouteCandidate(
0,
Channel.EMAIL,
new ContactPointId(UUID.fromString("00000000-0000-7000-8000-000000000011")),
new ProviderProfileId("ses-primary"),
true,
true);
}
private static ProviderProfileSnapshot profile() {
return new ProviderProfileSnapshot(
new ProviderProfileId("ses-primary"),
new ProviderId("ses"),
Channel.EMAIL,
"prod",
7L,
new ProviderCapabilities(
false, true, true, true, true, false, false, false, 1, 10_000L, Duration.ofHours(1)),
Map.of());
}
private static RenderedNotificationContent rendered() {
return new RenderedNotificationContent(
new EmailContent("subject", "body", Optional.empty(), List.of(), EmailOptions.DEFAULT),
"d".repeat(64),
new TemplateSelection("receipt", 1L, Locale.KOREAN),
Locale.KOREAN);
}
private static RecipientDeliveryRecord recipient(RouteCandidate route) {
return new RecipientDeliveryRecord(
new RecipientDeliveryId(UUID.fromString("00000000-0000-7000-8000-000000000012")),
new NotificationId(UUID.fromString("00000000-0000-7000-8000-000000000013")),
new TenantId("tenant-a"),
"user-1",
Optional.of(Locale.KOREAN),
Optional.empty(),
List.of(route),
0,
RecipientDeliveryState.READY_TO_DISPATCH,
SubmissionOutcome.NOT_SUBMITTED,
DeliveryOutcome.UNKNOWN,
EvidenceLevel.NONE,
false,
false,
Optional.empty(),
Optional.empty(),
Optional.empty(),
Optional.empty(),
0,
Optional.empty(),
NOW,
NOW);
}
}
@@ -0,0 +1,101 @@
package dev.caskeleton.application.notification.platform.dispatch;
import dev.caskeleton.application.notification.platform.api.ContactPointId;
import dev.caskeleton.application.notification.platform.api.NotificationId;
import dev.caskeleton.application.notification.platform.api.ProviderId;
import dev.caskeleton.application.notification.platform.api.ProviderProfileId;
import dev.caskeleton.application.notification.platform.api.RecipientDeliveryId;
import dev.caskeleton.application.notification.platform.api.TemplateSelection;
import dev.caskeleton.application.notification.platform.api.TenantId;
import dev.caskeleton.application.notification.platform.api.content.EmailContent;
import dev.caskeleton.application.notification.platform.api.content.EmailOptions;
import dev.caskeleton.application.notification.platform.api.delivery.DeliveryOutcome;
import dev.caskeleton.application.notification.platform.api.delivery.EvidenceLevel;
import dev.caskeleton.application.notification.platform.api.delivery.RecipientDeliveryState;
import dev.caskeleton.application.notification.platform.api.delivery.SubmissionOutcome;
import dev.caskeleton.application.notification.platform.api.routing.Channel;
import dev.caskeleton.application.notification.platform.policy.RouteCandidate;
import dev.caskeleton.application.notification.platform.provider.ProviderCapabilities;
import dev.caskeleton.application.notification.platform.provider.ProviderProfileSnapshot;
import dev.caskeleton.application.notification.platform.template.RenderedNotificationContent;
import java.time.Duration;
import java.time.Instant;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Optional;
import java.util.UUID;
/**
* The inputs the dispatcher hands {@link DeliveryAttemptFactory}, shared by every test that needs a
* row exactly as the factory writes it rather than one assembled by hand.
*/
final class DeliveryAttemptFixtures {
static final Instant NOW = Instant.parse("2026-09-20T00:00:00Z");
private DeliveryAttemptFixtures() {}
static RouteCandidate route() {
return new RouteCandidate(
0,
Channel.EMAIL,
new ContactPointId(UUID.fromString("00000000-0000-7000-8000-000000000011")),
new ProviderProfileId("ses-primary"),
true,
true);
}
static ProviderProfileSnapshot profile() {
return new ProviderProfileSnapshot(
new ProviderProfileId("ses-primary"),
new ProviderId("ses"),
Channel.EMAIL,
"prod",
7L,
new ProviderCapabilities(
false, true, true, true, true, false, false, false, 1, 10_000L, Duration.ofHours(1)),
Map.of());
}
static RenderedNotificationContent rendered() {
return new RenderedNotificationContent(
new EmailContent("subject", "body", Optional.empty(), List.of(), EmailOptions.DEFAULT),
"d".repeat(64),
new TemplateSelection("receipt", 1L, Locale.KOREAN),
Locale.KOREAN);
}
static RecipientDeliveryRecord recipient(RouteCandidate route) {
return new RecipientDeliveryRecord(
new RecipientDeliveryId(UUID.fromString("00000000-0000-7000-8000-000000000012")),
new NotificationId(UUID.fromString("00000000-0000-7000-8000-000000000013")),
new TenantId("tenant-a"),
"user-1",
Optional.of(Locale.KOREAN),
Optional.empty(),
List.of(route),
0,
RecipientDeliveryState.READY_TO_DISPATCH,
SubmissionOutcome.NOT_SUBMITTED,
DeliveryOutcome.UNKNOWN,
EvidenceLevel.NONE,
false,
false,
Optional.empty(),
Optional.empty(),
Optional.empty(),
Optional.empty(),
0,
Optional.empty(),
NOW,
NOW);
}
/** The attempt row the dispatcher commits immediately before the provider call. */
static DeliveryAttemptRecord attemptAsTheDispatcherWritesIt(UUID attemptId) {
RouteCandidate route = route();
return new DeliveryAttemptFactory(() -> attemptId)
.create(recipient(route), route, profile(), rendered(), 1, NOW);
}
}
@@ -496,11 +496,12 @@ class DispatchOutcomeRecorderTest {
0,
Optional.empty(),
COMPLETED,
COMPLETED);
COMPLETED,
0);
}
@Override
public List<ReconciliationJob> claimDue(int limit, Instant now) {
public List<ReconciliationJob> claimDue(int limit, Instant now, Duration lease) {
return List.of();
}
@@ -109,6 +109,31 @@ class LeaseRecoveryServiceTest {
assertThat(deliveries.transitions).isEmpty();
}
@Test
@DisplayName("provider 호출 중에 워커가 죽어 factory가 쓴 attempt 행만 남으면 다시 큐에 넣지 않고 정산한다")
void anAttemptLeftAsTheFactoryWroteItIsReconciledNotRequeued() {
// given
// The row the dispatcher commits before the call is, unchanged, the row a worker killed during
// the SES/SMTP call leaves behind: nothing writes it again until the outcome is recorded.
RecipientDeliveryId id = new RecipientDeliveryId(UUID.randomUUID());
DeliveryAttemptId attemptId = new DeliveryAttemptId(UUID.randomUUID());
leases.abandoned.add(id);
attempts.byDelivery.put(
id, List.of(DeliveryAttemptFixtures.attemptAsTheDispatcherWritesIt(attemptId.value())));
// when
int handled = recovery.recoverOnce();
// then
assertThat(handled).isEqualTo(1);
assertThat(reconciled)
.as("the provider may already have the message; only the provider can say whether it does")
.containsExactly(attemptId);
assertThat(deliveries.transitions)
.as("a requeue here hands the same notification to another worker, which sends it again")
.isEmpty();
}
@Test
@DisplayName("attempt가 모두 끝난 DISPATCHING 수신자는 다시 꺼낼 수 있게 되돌린다")
void aDispatchingDeliveryWhoseAttemptsAllCompletedIsRequeued() {
@@ -82,7 +82,8 @@ public final class NotificationPlatformNoops {
}
@Override
public List<ReconciliationJob> claimDue(int limit, java.time.Instant now) {
public List<ReconciliationJob> claimDue(
int limit, java.time.Instant now, java.time.Duration lease) {
return List.of();
}