fix!: 성숙도 리뷰의 High·P1 결함 19건을 고친다
문제: 2026-11-01 성숙도 리뷰에서 High 21건·P1 25건이 나왔다. 결과를 모르는 일을 "확실함"으로 기록했다(발송 중 죽은 알림 재발송, 커밋 중 끊긴 쓰기를 재시도 가능으로 응답, 재시도 가능한 거부를 DEAD로). 중복을 DB가 아니라 코드 순서로 막았다(정산 job, 포스터 제목). 짝 프론트와 wire 계약이 달랐고(처리 중 409, Envelope null), JWT audience 검증이 기본으로 꺼져 있었고, 트레이스가 나가지 않았고, CI가 strict lane 14개를 한 번도 돌리지 않았다. 방법: 발견마다 실패하는 테스트를 먼저 쓰고 고쳤다. 주요 선택: - 알림 attempt는 처음부터 "시작 여부 모름"으로 쓴다(호출 직전 갱신은 쓰기가 하나 늘어 버림). - 정산 job claim은 lease·fence를 갱신하는 UPDATE 한 문장(처리 전체를 트랜잭션으로 감싸는 안은 외부 호출 동안 잠금을 잡아 버림). - 커밋 단계 연결 끊김은 DB_COMPLETION_UNKNOWN(재시도 불가). - 처리 중 409는 retryable=true + Retry-After: 2, Envelope null 필드는 web Jackson mixin으로 생략(shared-contract는 stdlib only). - 트레이싱은 자체 Tracer를 지우고 Boot 4 tracing·OTLP로. - strict lane을 단계에 연결하고 verifyStrictLaneStages가 고아를 막는다. 발견별 RED·GREEN·버린 대안은 리뷰 README 13절 표에 있다. 위험·한계: JWT 모드는 APP_SECURITY_JWT_AUDIENCE가 없으면 기동하지 않는다. 짝 프론트는 409 retryable과 Envelope 키 규칙을 맞춰야 한다. 이 커밋 시점에는 MinIO 공식 이미지가 사라져 MinIO 2 lane이 실패한다 (다음 커밋에서 대체 이미지로 해결). 검증: ci 통과. mainCheck·integrationCheck는 MinIO 2 lane만 실패(이미지 pull 거부). -p optional-platforms ci integrationCheck, performanceCheck, releaseCheck(-PgitRevision) 통과. Refs: docs/reviews/2026-11-01-maturity-review/README.md 2·13절 BREAKING CHANGE: JWT 모드에서 APP_SECURITY_JWT_AUDIENCE가 필수가 되고 role은 audience client와 realm 것만 읽는다. IDEMPOTENT_IN_FLIGHT가 retryable=true, Retry-After: 2로 바뀌고 Envelope는 null 필드를 보내지 않는다.
This commit is contained in:
@@ -47,6 +47,26 @@ retries when everything is failing, which is the behaviour that lets a dependenc
|
||||
transaction annotations, deliberately: a `REQUIRES_NEW` would put the claim in its own transaction and
|
||||
reintroduce the window where the write is durable and the claim is not.
|
||||
|
||||
**A claim expires instead of blocking its key forever** (review 2026-11-01, 03-10). A claim still
|
||||
`IN_PROGRESS` after the claim TTL is presumed abandoned by a dead owner, and the next retry of the same
|
||||
request takes it over with `GrpcOperationLedger.reclaimStale` — one conditional
|
||||
`UPDATE … WHERE state = 'IN_PROGRESS' AND claimed_at < :cutoff`, so of two concurrent takeovers exactly
|
||||
one wins, the same shape as `reclaimFailed`. The TTL is the `claimTtl` argument of
|
||||
`GrpcIdempotencyInterceptor`, refused at construction when it is not longer than every keyed method's
|
||||
deadline. It is not a configuration key: the starter does not assemble the interceptor yet (03-11), so
|
||||
there is nothing for a key to configure.
|
||||
|
||||
The takeover is safe only because the commit is recorded in the mutation's own transaction: then a
|
||||
claim still `IN_PROGRESS` means the mutation never committed. An adopter that commits them separately
|
||||
turns an abandoned claim into an unknown outcome, and a takeover can run the mutation twice. A takeover
|
||||
of an owner that is merely slow does the same, which is why the TTL must exceed the slowest keyed
|
||||
method and not just its deadline.
|
||||
|
||||
Rejected: an owner token with a heartbeat lease (the outbox lease model) — needed only when a keyed
|
||||
method can run longer than any fixed TTL; revisit when one exists. A sweeper job that deletes stale
|
||||
rows — it needs a scheduler this family does not have, and deleting a claim makes the next retry a
|
||||
fresh claim with no record that an attempt was abandoned. Retention of terminal rows is still open.
|
||||
|
||||
**A key reused for a different request is a caller error, not a duplicate.** The stored request
|
||||
fingerprint turns that into `FAILED_PRECONDITION` rather than silently returning the first request's
|
||||
answer.
|
||||
|
||||
@@ -38,8 +38,17 @@ the time, and which half is not knowable from the status.
|
||||
6. If the method is not keyed, there is no ledger row. Resolve it against the business resource, or
|
||||
escalate to the service owner. This is the case the keyed profile exists to avoid.
|
||||
|
||||
7. `IN_PROGRESS` means a claim exists and has not finished. If its owner died, the claim is not
|
||||
stuck forever: once it is older than the claim TTL (the `claimTtl` given to
|
||||
`GrpcIdempotencyInterceptor`), the next retry of the same request takes it over and runs again,
|
||||
and a later reconciler pass sees that attempt's `COMMITTED` or `FAILED_TERMINAL`. Until a retry
|
||||
arrives, the case stays pending. The re-run is safe only when the ledger commit is recorded in the
|
||||
mutation's own transaction; if this service commits them separately, check the business resource
|
||||
before letting the retry through.
|
||||
|
||||
**Escalate when** the reconciler's pending list grows across passes. That means the ledger is
|
||||
unreachable rather than slow.
|
||||
unreachable rather than slow. A case pending as `IN_PROGRESS` for longer than the claim TTL with no
|
||||
retry arriving is a caller that gave up; it needs the business resource checked, not a ledger fix.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -134,3 +134,9 @@ the repository's `.seconds` / `.total` unit naming and the corresponding validat
|
||||
durable operation idempotency adds `:grpc:grpc-operation-ledger-jpa` explicitly, enables the ledger
|
||||
configuration, and includes that adapter's Flyway location `db/migration/grpc`. A deployment that
|
||||
does not adopt the ledger therefore pays no Spring Data JPA classpath cost.
|
||||
|
||||
An `IN_PROGRESS` claim expires after the claim TTL (`GrpcIdempotencyInterceptor`'s `claimTtl`
|
||||
argument; no configuration key while the starter does not assemble the interceptor). A retry of the
|
||||
same request then takes it over through `GrpcOperationLedger.reclaimStale`. The adopter must record
|
||||
the commit in the mutation's transaction and choose a TTL longer than the slowest keyed method; both
|
||||
reasons are in ADR-GRPC-004.
|
||||
|
||||
+30
@@ -52,6 +52,36 @@ public interface GrpcOperationLedger {
|
||||
*/
|
||||
boolean reclaimFailed(GrpcOperationIdentity identity, String requestFingerprint, Instant now);
|
||||
|
||||
/**
|
||||
* Takes over an {@code IN_PROGRESS} claim whose owner is presumed dead, for a new attempt of the
|
||||
* same request.
|
||||
*
|
||||
* <p>A claim made before {@code claimedBefore} has outlived the time its owner could still be
|
||||
* working (the claim TTL), so its owner is treated as gone. Without this, a process that dies
|
||||
* between claim and commit leaves its key blocked forever: every retry waits on, or is refused
|
||||
* by, a claim nobody will finish.
|
||||
*
|
||||
* <p>A compare-and-set like {@link #reclaimFailed}: the record becomes a fresh {@code
|
||||
* IN_PROGRESS} claim at {@code now}, and of two concurrent takeovers exactly one returns true —
|
||||
* the winner's new claim moment is no longer before the cutoff, so the loser's condition fails.
|
||||
* That is why {@code claimedBefore} after {@code now} is refused: the new claim would itself be
|
||||
* stale and a second takeover would win too. A record in any other state, claimed at or after the
|
||||
* cutoff, or made for a different request, is left untouched.
|
||||
*
|
||||
* <p>Safe only because the commit is recorded in the same transaction as the mutation (see {@link
|
||||
* #markCommitted}): then a claim still {@code IN_PROGRESS} means the mutation never committed.
|
||||
* When an adopter commits them separately, the abandoned attempt's outcome is unknown and a
|
||||
* takeover can run the mutation twice.
|
||||
*
|
||||
* @return true when this call now owns the record as a fresh attempt claimed at {@code now}
|
||||
* @throws IllegalArgumentException when {@code claimedBefore} is after {@code now}
|
||||
*/
|
||||
boolean reclaimStale(
|
||||
GrpcOperationIdentity identity,
|
||||
String requestFingerprint,
|
||||
Instant claimedBefore,
|
||||
Instant now);
|
||||
|
||||
/**
|
||||
* Releases a claim that will never complete, so the caller may try again.
|
||||
*
|
||||
|
||||
+160
@@ -4,6 +4,7 @@ import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.grpc.core.GrpcMethodName;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
@@ -12,6 +13,7 @@ import java.util.concurrent.CountDownLatch;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.Future;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
@@ -34,6 +36,10 @@ public abstract class GrpcOperationLedgerContract {
|
||||
"sha256:1122334455667788990011223344556677889900112233445566778899001122";
|
||||
private static final String OUTCOME = "outcome://document/42";
|
||||
private static final Instant NOW = Instant.parse("2026-09-25T10:00:00Z");
|
||||
private static final Duration CLAIM_TTL = Duration.ofMinutes(5);
|
||||
// A retry arriving after the TTL: its cutoff (LATER - TTL) is past the original claim at NOW.
|
||||
private static final Instant LATER = NOW.plus(CLAIM_TTL).plusSeconds(1);
|
||||
private static final Instant CUTOFF = LATER.minus(CLAIM_TTL);
|
||||
|
||||
/** The ledger under test, holding no records when a test starts. */
|
||||
protected abstract GrpcOperationLedger ledger();
|
||||
@@ -237,6 +243,160 @@ public abstract class GrpcOperationLedgerContract {
|
||||
assertThat(reclaimed).containsExactlyInAnyOrder(true, false);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("TTL이 지난 IN_PROGRESS claim은 같은 요청의 재시도가 넘겨받는다")
|
||||
void aStaleInProgressClaimIsTakenOverByARetryOfTheSameRequest() {
|
||||
// given
|
||||
ledger().claim(ID, FINGERPRINT, NOW);
|
||||
|
||||
// when
|
||||
boolean reclaimed = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER);
|
||||
|
||||
// then
|
||||
assertThat(reclaimed).isTrue();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("넘겨받은 stale claim은 넘겨받은 시각부터 진행 중인 새 시도다")
|
||||
void aTakenOverStaleClaimIsANewAttemptInProgressSinceTheTakeover() {
|
||||
// given
|
||||
ledger().claim(ID, FINGERPRINT, NOW);
|
||||
|
||||
// when
|
||||
ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER);
|
||||
|
||||
// then
|
||||
assertThat(ledger().find(ID))
|
||||
.get()
|
||||
.extracting(
|
||||
GrpcOperationLedgerRecord::state,
|
||||
GrpcOperationLedgerRecord::claimedAt,
|
||||
GrpcOperationLedgerRecord::completedAt)
|
||||
.containsExactly(GrpcOperationLedgerState.IN_PROGRESS, LATER, Optional.empty());
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("넘겨받은 stale claim은 commit할 수 있다")
|
||||
void aTakenOverStaleClaimCanBeCommitted() {
|
||||
// given
|
||||
ledger().claim(ID, FINGERPRINT, NOW);
|
||||
ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER);
|
||||
|
||||
// when
|
||||
ledger().markCommitted(ID, OUTCOME, LATER.plusSeconds(1));
|
||||
|
||||
// then
|
||||
assertThat(ledger().find(ID))
|
||||
.get()
|
||||
.extracting(GrpcOperationLedgerRecord::state, GrpcOperationLedgerRecord::outcomeReference)
|
||||
.containsExactly(GrpcOperationLedgerState.COMMITTED, Optional.of(OUTCOME));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("TTL 안의 IN_PROGRESS claim은 넘겨받지 않는다")
|
||||
void anInProgressClaimWithinTheTtlIsNotTakenOver() {
|
||||
// given
|
||||
ledger().claim(ID, FINGERPRINT, NOW);
|
||||
|
||||
// when
|
||||
boolean reclaimed = ledger().reclaimStale(ID, FINGERPRINT, NOW, NOW.plus(CLAIM_TTL));
|
||||
|
||||
// then
|
||||
assertThat(reclaimed).isFalse();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("다른 요청은 TTL이 지난 claim도 넘겨받지 못한다")
|
||||
void aStaleClaimIsNotTakenOverForADifferentRequest() {
|
||||
// given
|
||||
ledger().claim(ID, FINGERPRINT, NOW);
|
||||
|
||||
// when
|
||||
boolean reclaimed = ledger().reclaimStale(ID, OTHER_FINGERPRINT, CUTOFF, LATER);
|
||||
|
||||
// then
|
||||
assertThat(reclaimed).isFalse();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("commit된 claim은 오래되어도 stale로 넘겨받지 않는다")
|
||||
void aCommittedClaimIsNotTakenOverAsStale() {
|
||||
// given
|
||||
ledger().claim(ID, FINGERPRINT, NOW);
|
||||
ledger().markCommitted(ID, OUTCOME, NOW.plusSeconds(1));
|
||||
|
||||
// when
|
||||
boolean reclaimed = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER);
|
||||
|
||||
// then
|
||||
assertThat(reclaimed).isFalse();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("실패한 claim은 stale 회수가 아니라 실패 재claim으로만 다시 연다")
|
||||
void aFailedClaimIsNotTakenOverAsStale() {
|
||||
// given
|
||||
failedClaim();
|
||||
|
||||
// when
|
||||
boolean reclaimed = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER);
|
||||
|
||||
// then
|
||||
assertThat(reclaimed).isFalse();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("claim이 없으면 stale 회수는 아무것도 만들지 않는다")
|
||||
void takingOverAMissingClaimCreatesNothing() {
|
||||
// when
|
||||
boolean reclaimed = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER);
|
||||
|
||||
// then
|
||||
assertThat(reclaimed).isFalse();
|
||||
assertThat(ledger().find(ID)).isEmpty();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("한 번 넘겨받은 claim은 같은 cutoff로 다시 넘겨받지 못한다")
|
||||
void aTakenOverClaimIsNotTakenOverAgainWithTheSameCutoff() {
|
||||
// given
|
||||
ledger().claim(ID, FINGERPRINT, NOW);
|
||||
ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER);
|
||||
|
||||
// when
|
||||
boolean again = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER);
|
||||
|
||||
// then
|
||||
assertThat(again).isFalse();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("TTL이 지난 claim을 동시에 넘겨받으려 하면 승자는 하나다")
|
||||
void concurrentTakeoversOfAStaleClaimHaveExactlyOneWinner() throws Exception {
|
||||
// given
|
||||
ledger().claim(ID, FINGERPRINT, NOW);
|
||||
|
||||
// when
|
||||
List<Boolean> reclaimed =
|
||||
concurrently(() -> ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER));
|
||||
|
||||
// then
|
||||
assertThat(reclaimed).containsExactlyInAnyOrder(true, false);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("cutoff가 현재 시각보다 뒤면 거부한다 — 새 claim이 곧바로 stale이 되기 때문이다")
|
||||
void aCutoffAfterNowIsRefused() {
|
||||
// given
|
||||
ledger().claim(ID, FINGERPRINT, NOW);
|
||||
|
||||
// when
|
||||
Throwable refused = catchThrowable(() -> ledger().reclaimStale(ID, FINGERPRINT, LATER, CUTOFF));
|
||||
|
||||
// then
|
||||
assertThat(refused).isInstanceOf(IllegalArgumentException.class).hasMessageContaining("cutoff");
|
||||
}
|
||||
|
||||
private void failedClaim() {
|
||||
ledger().claim(ID, FINGERPRINT, NOW);
|
||||
ledger().markFailed(ID, NOW.plusSeconds(1));
|
||||
|
||||
+24
-3
@@ -10,9 +10,9 @@ import java.util.concurrent.ConcurrentMap;
|
||||
*
|
||||
* <p>{@code putIfAbsent} on a concurrent map is the in-memory equivalent of the unique constraint
|
||||
* the real adapter relies on, and {@code replace(key, expected, next)} of its conditional update,
|
||||
* so claim and re-claim are atomic here for the same reason they are there. The transitions match
|
||||
* the JPA adapter's because {@link GrpcOperationLedgerContract} runs against both; a fake that
|
||||
* allowed more would let the policy tests pass for code the database refuses.
|
||||
* so claim, re-claim and stale takeover are atomic here for the same reason they are there. The
|
||||
* transitions match the JPA adapter's because {@link GrpcOperationLedgerContract} runs against
|
||||
* both; a fake that allowed more would let the policy tests pass for code the database refuses.
|
||||
*/
|
||||
public final class InMemoryOperationLedger implements GrpcOperationLedger {
|
||||
|
||||
@@ -81,6 +81,27 @@ public final class InMemoryOperationLedger implements GrpcOperationLedger {
|
||||
GrpcOperationLedgerRecord.claim(identity, requestFingerprint, now));
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean reclaimStale(
|
||||
GrpcOperationIdentity identity,
|
||||
String requestFingerprint,
|
||||
Instant claimedBefore,
|
||||
Instant now) {
|
||||
if (claimedBefore.isAfter(now)) {
|
||||
throw new IllegalArgumentException(
|
||||
"a stale-claim cutoff after now would leave the new claim stale as well");
|
||||
}
|
||||
GrpcOperationLedgerRecord current = records.get(identity.storageKey());
|
||||
return current != null
|
||||
&& current.state() == GrpcOperationLedgerState.IN_PROGRESS
|
||||
&& current.claimedAt().isBefore(claimedBefore)
|
||||
&& current.sameRequestAs(requestFingerprint)
|
||||
&& records.replace(
|
||||
identity.storageKey(),
|
||||
current,
|
||||
GrpcOperationLedgerRecord.claim(identity, requestFingerprint, now));
|
||||
}
|
||||
|
||||
@Override
|
||||
public void release(GrpcOperationIdentity identity) {
|
||||
records.remove(identity.storageKey());
|
||||
|
||||
+50
@@ -22,6 +22,7 @@ import java.util.Optional;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.data.jpa.repository.support.JpaRepositoryFactory;
|
||||
import org.springframework.orm.jpa.JpaTransactionManager;
|
||||
@@ -120,6 +121,44 @@ class JpaGrpcOperationLedgerContractIT extends GrpcOperationLedgerContract {
|
||||
.containsExactly(GrpcOperationLedgerState.COMMITTED, Optional.of("outcome://document/42"));
|
||||
}
|
||||
|
||||
/**
|
||||
* The takeover's conditional update clears the persistence context for the same reason the failed
|
||||
* re-claim does: the claim's read-on-conflict loaded the stale row, and a commit reading that
|
||||
* copy would keep the abandoned claim moment.
|
||||
*/
|
||||
@Test
|
||||
@DisplayName("stale claim을 넘겨받아 같은 트랜잭션에서 commit하면 commit된다")
|
||||
void aRetryThatTakesOverAStaleClaimAndCommitsInOneTransactionCommits() {
|
||||
// given
|
||||
GrpcOperationIdentity id =
|
||||
new GrpcOperationIdentity(
|
||||
"tenant-1.actor-1",
|
||||
GrpcMethodName.parse("caskeleton.document.v1.DocumentService/CreateDocument"),
|
||||
"sha256:aabbccddeeff00112233445566778899aabbccddeeff00112233445566778899");
|
||||
String fingerprint = "sha256:0f1e2d3c4b5a69788796a5b4c3d2e1f00f1e2d3c4b5a69788796a5b4c3d2e1f0";
|
||||
Instant abandonedAt = Instant.parse("2026-09-25T10:00:00Z");
|
||||
Instant retryAt = abandonedAt.plusSeconds(301);
|
||||
ledger.claim(id, fingerprint, abandonedAt);
|
||||
|
||||
// when
|
||||
transactions.executeWithoutResult(
|
||||
status -> {
|
||||
adapter.claim(id, fingerprint, retryAt);
|
||||
adapter.reclaimStale(id, fingerprint, retryAt.minusSeconds(300), retryAt);
|
||||
adapter.markCommitted(id, "outcome://document/42", retryAt.plusSeconds(1));
|
||||
});
|
||||
|
||||
// then
|
||||
assertThat(ledger.find(id))
|
||||
.get()
|
||||
.extracting(
|
||||
GrpcOperationLedgerRecord::state,
|
||||
GrpcOperationLedgerRecord::claimedAt,
|
||||
GrpcOperationLedgerRecord::outcomeReference)
|
||||
.containsExactly(
|
||||
GrpcOperationLedgerState.COMMITTED, retryAt, Optional.of("outcome://document/42"));
|
||||
}
|
||||
|
||||
private static void execute(String sql) throws SQLException {
|
||||
try (Connection connection =
|
||||
DriverManager.getConnection(
|
||||
@@ -177,6 +216,17 @@ class JpaGrpcOperationLedgerContractIT extends GrpcOperationLedgerContract {
|
||||
status -> delegate.reclaimFailed(identity, requestFingerprint, now)));
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean reclaimStale(
|
||||
GrpcOperationIdentity identity,
|
||||
String requestFingerprint,
|
||||
Instant claimedBefore,
|
||||
Instant now) {
|
||||
return Boolean.TRUE.equals(
|
||||
transactions.execute(
|
||||
status -> delegate.reclaimStale(identity, requestFingerprint, claimedBefore, now)));
|
||||
}
|
||||
|
||||
@Override
|
||||
public void release(GrpcOperationIdentity identity) {
|
||||
transactions.executeWithoutResult(status -> delegate.release(identity));
|
||||
|
||||
+30
@@ -92,6 +92,36 @@ public interface GrpcOperationLedgerRepository
|
||||
@Param("requestFingerprint") String requestFingerprint,
|
||||
@Param("claimedAt") Instant claimedAt);
|
||||
|
||||
/**
|
||||
* Takes over an {@code IN_PROGRESS} row claimed before {@code claimedBefore}, for the same
|
||||
* request.
|
||||
*
|
||||
* <p>Same shape as {@link #tryReclaimFailed}: one conditional update decided by PostgreSQL's row
|
||||
* lock. A concurrent second takeover waits for the winner, re-evaluates the condition against the
|
||||
* winner's row — whose {@code claimed_at} is now {@code :claimedAt}, not before the cutoff — and
|
||||
* updates nothing. It clears the persistence context for the same reason.
|
||||
*
|
||||
* @return one when this caller took the row over, zero otherwise
|
||||
*/
|
||||
@Modifying(flushAutomatically = true, clearAutomatically = true)
|
||||
@Transactional
|
||||
@Query(
|
||||
value =
|
||||
"""
|
||||
UPDATE grpc_operation_ledger
|
||||
SET claimed_at = :claimedAt
|
||||
WHERE storage_key = :storageKey
|
||||
AND state = 'IN_PROGRESS'
|
||||
AND request_fingerprint = :requestFingerprint
|
||||
AND claimed_at < :claimedBefore
|
||||
""",
|
||||
nativeQuery = true)
|
||||
int tryReclaimStale(
|
||||
@Param("storageKey") String storageKey,
|
||||
@Param("requestFingerprint") String requestFingerprint,
|
||||
@Param("claimedBefore") Instant claimedBefore,
|
||||
@Param("claimedAt") Instant claimedAt);
|
||||
|
||||
/** Updates an existing claim after it has been loaded. */
|
||||
GrpcOperationLedgerEntity save(GrpcOperationLedgerEntity entity);
|
||||
|
||||
|
||||
+15
@@ -107,6 +107,21 @@ public final class JpaGrpcOperationLedger implements GrpcOperationLedger {
|
||||
return repository.tryReclaimFailed(identity.storageKey(), requestFingerprint, now) == 1;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean reclaimStale(
|
||||
GrpcOperationIdentity identity,
|
||||
String requestFingerprint,
|
||||
Instant claimedBefore,
|
||||
Instant now) {
|
||||
requireIdentity(identity);
|
||||
if (claimedBefore.isAfter(now)) {
|
||||
throw new IllegalArgumentException(
|
||||
"a stale-claim cutoff after now would leave the new claim stale as well");
|
||||
}
|
||||
return repository.tryReclaimStale(identity.storageKey(), requestFingerprint, claimedBefore, now)
|
||||
== 1;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void release(GrpcOperationIdentity identity) {
|
||||
requireIdentity(identity);
|
||||
|
||||
+7
@@ -66,6 +66,13 @@ class GrpcOperationLedgerRepositoryTest {
|
||||
"the re-claim is certified against PostgreSQL by JpaGrpcOperationLedgerContractIT");
|
||||
}
|
||||
|
||||
@Override
|
||||
public int tryReclaimStale(
|
||||
String storageKey, String requestFingerprint, Instant claimedBefore, Instant claimedAt) {
|
||||
throw new UnsupportedOperationException(
|
||||
"the stale takeover is certified against PostgreSQL by JpaGrpcOperationLedgerContractIT");
|
||||
}
|
||||
|
||||
@Override
|
||||
public GrpcOperationLedgerEntity save(GrpcOperationLedgerEntity entity) {
|
||||
rows.put(entity.getStorageKey(), entity);
|
||||
|
||||
+53
-2
@@ -19,6 +19,10 @@ import java.util.Optional;
|
||||
* client takes the same one before deciding whether a retry is safe. An interceptor that owns the
|
||||
* logic leaves the client re-deriving it.
|
||||
*
|
||||
* <p>A claim does not block its key forever. One still {@code IN_PROGRESS} after the claim TTL is
|
||||
* treated as abandoned by a dead owner and taken over by the next retry of the same request — one
|
||||
* winner, like the re-claim of a terminal failure.
|
||||
*
|
||||
* <p>The ordering matters and is easy to get subtly wrong. The claim is attempted first,
|
||||
* atomically; only then is the existing record examined. Reading first and claiming second leaves a
|
||||
* window in which two attempts both read nothing.
|
||||
@@ -29,18 +33,24 @@ public final class GrpcIdempotencyInterceptor {
|
||||
private final GrpcMethodPolicyCatalog catalog;
|
||||
private final Duration inProgressPollInterval;
|
||||
private final boolean waitForInProgress;
|
||||
private final Duration claimTtl;
|
||||
|
||||
/**
|
||||
* @param inProgressPollInterval how long a duplicate is told to wait before asking again
|
||||
* @param waitForInProgress whether a duplicate of a running operation waits or is refused. A
|
||||
* method whose caller is a user-facing request usually wants to be refused quickly; a worker
|
||||
* usually wants to wait.
|
||||
* @param claimTtl how long an {@code IN_PROGRESS} claim is trusted to have a live owner. After
|
||||
* it, a retry of the same request takes the claim over. It must be longer than every keyed
|
||||
* method's deadline, and longer than the slowest such method can actually run: a claim taken
|
||||
* over from an owner that is merely slow runs the mutation twice.
|
||||
*/
|
||||
public GrpcIdempotencyInterceptor(
|
||||
GrpcOperationLedger ledger,
|
||||
GrpcMethodPolicyCatalog catalog,
|
||||
Duration inProgressPollInterval,
|
||||
boolean waitForInProgress) {
|
||||
boolean waitForInProgress,
|
||||
Duration claimTtl) {
|
||||
if (ledger == null || catalog == null) {
|
||||
throw new IllegalArgumentException("an idempotency interceptor needs a ledger and a catalog");
|
||||
}
|
||||
@@ -54,6 +64,32 @@ public final class GrpcIdempotencyInterceptor {
|
||||
this.catalog = catalog;
|
||||
this.inProgressPollInterval = inProgressPollInterval;
|
||||
this.waitForInProgress = waitForInProgress;
|
||||
this.claimTtl = requireClaimTtlOutlivesKeyedDeadlines(claimTtl, catalog);
|
||||
}
|
||||
|
||||
// A TTL inside a keyed method's deadline lets a retry take over a claim whose owner is still
|
||||
// within its budget. The deadline is the only per-method bound on owner lifetime the catalog
|
||||
// knows; it does not bound a server that keeps working after the deadline fires.
|
||||
private static Duration requireClaimTtlOutlivesKeyedDeadlines(
|
||||
Duration claimTtl, GrpcMethodPolicyCatalog catalog) {
|
||||
if (claimTtl == null || claimTtl.isZero() || claimTtl.isNegative()) {
|
||||
throw new IllegalArgumentException("a claim TTL must be present and positive");
|
||||
}
|
||||
for (GrpcMethodName method : catalog.methods()) {
|
||||
GrpcMethodPolicy policy = catalog.require(method);
|
||||
if (policy.idempotency().idempotencyKeyRequired()
|
||||
&& claimTtl.compareTo(policy.deadline().total()) <= 0) {
|
||||
throw new IllegalArgumentException(
|
||||
"claim TTL "
|
||||
+ claimTtl
|
||||
+ " is not longer than the "
|
||||
+ policy.deadline().total()
|
||||
+ " deadline of keyed method '"
|
||||
+ method.canonical()
|
||||
+ "'; a retry would take over a claim whose owner is still running");
|
||||
}
|
||||
}
|
||||
return claimTtl;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -115,7 +151,22 @@ public final class GrpcIdempotencyInterceptor {
|
||||
return GrpcIdempotencyDecision.proceed(
|
||||
"the previous attempt failed terminally; this caller re-claimed it for a new attempt");
|
||||
}
|
||||
// Running, or a failed attempt another retry re-claimed first.
|
||||
if (record.state() == GrpcOperationLedgerState.IN_PROGRESS) {
|
||||
Instant cutoff = now.minus(claimTtl);
|
||||
if (record.claimedAt().isBefore(cutoff)
|
||||
&& ledger.reclaimStale(identity, requestFingerprint, cutoff, now)) {
|
||||
// The owner outlived the TTL without committing or failing, so it is presumed dead. Its
|
||||
// mutation did not commit if the commit was recorded in the mutation's own transaction,
|
||||
// which the ledger contract requires; when an adopter commits them separately, the
|
||||
// abandoned attempt's outcome is unknown and this re-run can duplicate it.
|
||||
return GrpcIdempotencyDecision.proceed(
|
||||
"the previous claim was abandoned (older than the "
|
||||
+ claimTtl
|
||||
+ " claim TTL); this caller took it over for a new attempt");
|
||||
}
|
||||
}
|
||||
// Running, a failed attempt another retry re-claimed first, or an abandoned claim another
|
||||
// retry took over first.
|
||||
return waitForInProgress
|
||||
? GrpcIdempotencyDecision.waitAndPoll(inProgressPollInterval)
|
||||
: GrpcIdempotencyDecision.rejectInProgress();
|
||||
|
||||
+24
-1
@@ -2,8 +2,10 @@ package dev.caskeleton.grpc.idempotency;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity;
|
||||
import dev.caskeleton.grpc.api.ledger.InMemoryOperationLedger;
|
||||
import dev.caskeleton.grpc.core.GrpcMethodName;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.Optional;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
@@ -29,6 +31,28 @@ class GrpcCompletionReconcilerTest {
|
||||
NOW);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("버려진 claim으로 대기열에 오른 건은 넘겨받은 재시도가 commit하면 다음 회차에 COMMITTED로 풀린다")
|
||||
void anAbandonedClaimStaysPendingUntilATakeoverCommits() {
|
||||
// given
|
||||
GrpcOperationIdentity identity =
|
||||
new GrpcOperationIdentity(CALLER, CREATE, GrpcRequestFingerprint.hashIdempotencyKey(KEY));
|
||||
String fingerprint = GrpcRequestFingerprint.of("body");
|
||||
Instant afterTtl = NOW.plus(Duration.ofMinutes(5)).plusSeconds(1);
|
||||
GrpcCompletionReconciler reconciler = new GrpcCompletionReconciler(query);
|
||||
claim();
|
||||
reconciler.reconcile(CREATE, CALLER, KEY, null);
|
||||
ledger.reclaimStale(identity, fingerprint, afterTtl.minus(Duration.ofMinutes(5)), afterTtl);
|
||||
ledger.markCommitted(identity, "outcome://document/42", afterTtl.plusSeconds(1));
|
||||
|
||||
// when
|
||||
GrpcCompletionResolution nextPass = reconciler.reconcile(CREATE, CALLER, KEY, null);
|
||||
|
||||
// then
|
||||
assertThat(nextPass.status()).isEqualTo(GrpcOperationStatus.COMMITTED);
|
||||
assertThat(nextPass.outcomeReference()).contains("outcome://document/42");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a status query returns the five documented answers")
|
||||
void theQueryReturnsTheFiveDocumentedAnswers() {
|
||||
@@ -141,6 +165,5 @@ class GrpcCompletionReconcilerTest {
|
||||
GrpcCompletionResolution resolution = reconciler.reconcile(CREATE, CALLER, KEY, null);
|
||||
|
||||
assertThat(resolution.status()).isEqualTo(GrpcOperationStatus.IN_PROGRESS);
|
||||
assertThat(reconciler.pendingCases()).hasSize(1);
|
||||
}
|
||||
}
|
||||
|
||||
+105
-1
@@ -2,6 +2,7 @@ package dev.caskeleton.grpc.idempotency;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.grpc.api.deadline.GrpcDeadlineProfile;
|
||||
import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity;
|
||||
@@ -36,6 +37,8 @@ class GrpcIdempotencyInterceptorTest {
|
||||
private static final Instant NOW = Instant.parse("2026-08-30T10:00:00Z");
|
||||
private static final String CALLER = "tenant-1|actor-1";
|
||||
private static final String KEY = "order-2026-0001";
|
||||
private static final Duration CLAIM_TTL = Duration.ofMinutes(5);
|
||||
private static final Instant AFTER_TTL = NOW.plus(CLAIM_TTL).plusSeconds(1);
|
||||
|
||||
private final InMemoryOperationLedger ledger = new InMemoryOperationLedger();
|
||||
|
||||
@@ -57,7 +60,7 @@ class GrpcIdempotencyInterceptorTest {
|
||||
|
||||
private GrpcIdempotencyInterceptor interceptor(boolean waitForInProgress) {
|
||||
return new GrpcIdempotencyInterceptor(
|
||||
ledger, catalog(), Duration.ofMillis(250), waitForInProgress);
|
||||
ledger, catalog(), Duration.ofMillis(250), waitForInProgress, CLAIM_TTL);
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -194,6 +197,107 @@ class GrpcIdempotencyInterceptorTest {
|
||||
GrpcIdempotencyDecision.Outcome.REJECT_IN_PROGRESS);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("소유자가 죽어 TTL이 지난 claim은 같은 요청의 재시도가 넘겨받아 진행한다")
|
||||
void aRetryAfterTheClaimTtlTakesOverAnAbandonedClaimAndProceeds() {
|
||||
// given
|
||||
GrpcIdempotencyInterceptor interceptor = interceptor(false);
|
||||
String fingerprint = GrpcRequestFingerprint.of("body");
|
||||
interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW);
|
||||
|
||||
// when
|
||||
GrpcIdempotencyDecision decision =
|
||||
interceptor.decide(CREATE, CALLER, KEY, fingerprint, AFTER_TTL);
|
||||
|
||||
// then
|
||||
assertThat(decision.outcome()).isEqualTo(GrpcIdempotencyDecision.Outcome.PROCEED);
|
||||
assertThat(decision.reason()).contains("abandoned");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("넘겨받은 claim은 재시도 시각에 다시 잡힌 진행 중 claim이다")
|
||||
void aTakenOverClaimIsHeldInProgressFromTheRetry() {
|
||||
// given
|
||||
GrpcIdempotencyInterceptor interceptor = interceptor(false);
|
||||
String fingerprint = GrpcRequestFingerprint.of("body");
|
||||
interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW);
|
||||
|
||||
// when
|
||||
interceptor.decide(CREATE, CALLER, KEY, fingerprint, AFTER_TTL);
|
||||
|
||||
// then
|
||||
assertThat(
|
||||
ledger
|
||||
.find(
|
||||
new GrpcOperationIdentity(
|
||||
CALLER, CREATE, GrpcRequestFingerprint.hashIdempotencyKey(KEY)))
|
||||
.map(GrpcOperationLedgerRecord::claimedAt))
|
||||
.contains(AFTER_TTL);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("TTL이 지나기 직전의 중복 요청은 여전히 대기하거나 거부된다")
|
||||
void aDuplicateAtTheTtlBoundaryIsStillTreatedAsRunning() {
|
||||
// given
|
||||
GrpcIdempotencyInterceptor interceptor = interceptor(false);
|
||||
String fingerprint = GrpcRequestFingerprint.of("body");
|
||||
interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW);
|
||||
|
||||
// when
|
||||
GrpcIdempotencyDecision decision =
|
||||
interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW.plus(CLAIM_TTL));
|
||||
|
||||
// then
|
||||
assertThat(decision.outcome()).isEqualTo(GrpcIdempotencyDecision.Outcome.REJECT_IN_PROGRESS);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("버려진 claim에 동시에 온 재시도 중 하나만 진행한다")
|
||||
void concurrentRetriesOfAnAbandonedClaimProceedOnlyOnce() throws Exception {
|
||||
// given
|
||||
GrpcIdempotencyInterceptor interceptor = interceptor(false);
|
||||
String fingerprint = GrpcRequestFingerprint.of("body");
|
||||
interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW);
|
||||
CountDownLatch start = new CountDownLatch(1);
|
||||
Callable<GrpcIdempotencyDecision.Outcome> retry =
|
||||
() -> {
|
||||
start.await();
|
||||
return interceptor.decide(CREATE, CALLER, KEY, fingerprint, AFTER_TTL).outcome();
|
||||
};
|
||||
|
||||
// when
|
||||
List<GrpcIdempotencyDecision.Outcome> outcomes;
|
||||
try (ExecutorService executor = Executors.newFixedThreadPool(2)) {
|
||||
Future<GrpcIdempotencyDecision.Outcome> first = executor.submit(retry);
|
||||
Future<GrpcIdempotencyDecision.Outcome> second = executor.submit(retry);
|
||||
start.countDown();
|
||||
outcomes = List.of(first.get(), second.get());
|
||||
}
|
||||
|
||||
// then
|
||||
assertThat(outcomes)
|
||||
.containsExactlyInAnyOrder(
|
||||
GrpcIdempotencyDecision.Outcome.PROCEED,
|
||||
GrpcIdempotencyDecision.Outcome.REJECT_IN_PROGRESS);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("claim TTL이 key 필수 method의 deadline보다 짧으면 생성 시 거부한다")
|
||||
void aClaimTtlNoLongerThanAKeyedMethodsDeadlineIsRefused() {
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(
|
||||
() ->
|
||||
new GrpcIdempotencyInterceptor(
|
||||
ledger, catalog(), Duration.ofMillis(250), false, Duration.ofSeconds(10)));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.isInstanceOf(IllegalArgumentException.class)
|
||||
.hasMessageContaining(CREATE.canonical())
|
||||
.hasMessageContaining("claim TTL");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("one caller's key does not suppress another caller's write")
|
||||
void oneCallersKeyDoesNotSuppressAnother() {
|
||||
|
||||
Reference in New Issue
Block a user