From 8e816f01c1e1ffeaecd0524f37bd39735b7e734a Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Tue, 3 Nov 2026 19:52:06 +0900 Subject: [PATCH] =?UTF-8?q?fix!:=20=EC=84=B1=EC=88=99=EB=8F=84=20=EB=A6=AC?= =?UTF-8?q?=EB=B7=B0=EC=9D=98=20High=C2=B7P1=20=EA=B2=B0=ED=95=A8=2019?= =?UTF-8?q?=EA=B1=B4=EC=9D=84=20=EA=B3=A0=EC=B9=9C=EB=8B=A4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 문제: 2026-11-01 성숙도 리뷰에서 High 21건·P1 25건이 나왔다. 결과를 모르는 일을 "확실함"으로 기록했다(발송 중 죽은 알림 재발송, 커밋 중 끊긴 쓰기를 재시도 가능으로 응답, 재시도 가능한 거부를 DEAD로). 중복을 DB가 아니라 코드 순서로 막았다(정산 job, 포스터 제목). 짝 프론트와 wire 계약이 달랐고(처리 중 409, Envelope null), JWT audience 검증이 기본으로 꺼져 있었고, 트레이스가 나가지 않았고, CI가 strict lane 14개를 한 번도 돌리지 않았다. 방법: 발견마다 실패하는 테스트를 먼저 쓰고 고쳤다. 주요 선택: - 알림 attempt는 처음부터 "시작 여부 모름"으로 쓴다(호출 직전 갱신은 쓰기가 하나 늘어 버림). - 정산 job claim은 lease·fence를 갱신하는 UPDATE 한 문장(처리 전체를 트랜잭션으로 감싸는 안은 외부 호출 동안 잠금을 잡아 버림). - 커밋 단계 연결 끊김은 DB_COMPLETION_UNKNOWN(재시도 불가). - 처리 중 409는 retryable=true + Retry-After: 2, Envelope null 필드는 web Jackson mixin으로 생략(shared-contract는 stdlib only). - 트레이싱은 자체 Tracer를 지우고 Boot 4 tracing·OTLP로. - strict lane을 단계에 연결하고 verifyStrictLaneStages가 고아를 막는다. 발견별 RED·GREEN·버린 대안은 리뷰 README 13절 표에 있다. 위험·한계: JWT 모드는 APP_SECURITY_JWT_AUDIENCE가 없으면 기동하지 않는다. 짝 프론트는 409 retryable과 Envelope 키 규칙을 맞춰야 한다. 이 커밋 시점에는 MinIO 공식 이미지가 사라져 MinIO 2 lane이 실패한다 (다음 커밋에서 대체 이미지로 해결). 검증: ci 통과. mainCheck·integrationCheck는 MinIO 2 lane만 실패(이미지 pull 거부). -p optional-platforms ci integrationCheck, performanceCheck, releaseCheck(-PgitRevision) 통과. Refs: docs/reviews/2026-11-01-maturity-review/README.md 2·13절 BREAKING CHANGE: JWT 모드에서 APP_SECURITY_JWT_AUDIENCE가 필수가 되고 role은 audience client와 realm 것만 읽는다. IDEMPOTENT_IN_FLIGHT가 retryable=true, Retry-After: 2로 바뀌고 Envelope는 null 필드를 보내지 않는다. --- ...retry-ownership-and-durable-idempotency.md | 20 +++ docs/runbooks/grpc-platform-operations.md | 11 +- grpc/CLAUDE.md | 6 + .../grpc/api/ledger/GrpcOperationLedger.java | 30 ++++ .../ledger/GrpcOperationLedgerContract.java | 160 ++++++++++++++++++ .../api/ledger/InMemoryOperationLedger.java | 27 ++- .../JpaGrpcOperationLedgerContractIT.java | 50 ++++++ .../ledger/GrpcOperationLedgerRepository.java | 30 ++++ .../grpc/ledger/JpaGrpcOperationLedger.java | 15 ++ .../GrpcOperationLedgerRepositoryTest.java | 7 + .../GrpcIdempotencyInterceptor.java | 55 +++++- .../GrpcCompletionReconcilerTest.java | 25 ++- .../GrpcIdempotencyInterceptorTest.java | 106 +++++++++++- 13 files changed, 534 insertions(+), 8 deletions(-) diff --git a/docs/adr/ADR-GRPC-004-retry-ownership-and-durable-idempotency.md b/docs/adr/ADR-GRPC-004-retry-ownership-and-durable-idempotency.md index a7e2ebf..deb788b 100644 --- a/docs/adr/ADR-GRPC-004-retry-ownership-and-durable-idempotency.md +++ b/docs/adr/ADR-GRPC-004-retry-ownership-and-durable-idempotency.md @@ -47,6 +47,26 @@ retries when everything is failing, which is the behaviour that lets a dependenc transaction annotations, deliberately: a `REQUIRES_NEW` would put the claim in its own transaction and reintroduce the window where the write is durable and the claim is not. +**A claim expires instead of blocking its key forever** (review 2026-11-01, 03-10). A claim still +`IN_PROGRESS` after the claim TTL is presumed abandoned by a dead owner, and the next retry of the same +request takes it over with `GrpcOperationLedger.reclaimStale` — one conditional +`UPDATE … WHERE state = 'IN_PROGRESS' AND claimed_at < :cutoff`, so of two concurrent takeovers exactly +one wins, the same shape as `reclaimFailed`. The TTL is the `claimTtl` argument of +`GrpcIdempotencyInterceptor`, refused at construction when it is not longer than every keyed method's +deadline. It is not a configuration key: the starter does not assemble the interceptor yet (03-11), so +there is nothing for a key to configure. + +The takeover is safe only because the commit is recorded in the mutation's own transaction: then a +claim still `IN_PROGRESS` means the mutation never committed. An adopter that commits them separately +turns an abandoned claim into an unknown outcome, and a takeover can run the mutation twice. A takeover +of an owner that is merely slow does the same, which is why the TTL must exceed the slowest keyed +method and not just its deadline. + +Rejected: an owner token with a heartbeat lease (the outbox lease model) — needed only when a keyed +method can run longer than any fixed TTL; revisit when one exists. A sweeper job that deletes stale +rows — it needs a scheduler this family does not have, and deleting a claim makes the next retry a +fresh claim with no record that an attempt was abandoned. Retention of terminal rows is still open. + **A key reused for a different request is a caller error, not a duplicate.** The stored request fingerprint turns that into `FAILED_PRECONDITION` rather than silently returning the first request's answer. diff --git a/docs/runbooks/grpc-platform-operations.md b/docs/runbooks/grpc-platform-operations.md index 07d0639..d218f26 100644 --- a/docs/runbooks/grpc-platform-operations.md +++ b/docs/runbooks/grpc-platform-operations.md @@ -38,8 +38,17 @@ the time, and which half is not knowable from the status. 6. If the method is not keyed, there is no ledger row. Resolve it against the business resource, or escalate to the service owner. This is the case the keyed profile exists to avoid. +7. `IN_PROGRESS` means a claim exists and has not finished. If its owner died, the claim is not + stuck forever: once it is older than the claim TTL (the `claimTtl` given to + `GrpcIdempotencyInterceptor`), the next retry of the same request takes it over and runs again, + and a later reconciler pass sees that attempt's `COMMITTED` or `FAILED_TERMINAL`. Until a retry + arrives, the case stays pending. The re-run is safe only when the ledger commit is recorded in the + mutation's own transaction; if this service commits them separately, check the business resource + before letting the retry through. + **Escalate when** the reconciler's pending list grows across passes. That means the ledger is -unreachable rather than slow. +unreachable rather than slow. A case pending as `IN_PROGRESS` for longer than the claim TTL with no +retry arriving is a caller that gave up; it needs the business resource checked, not a ledger fix. --- diff --git a/grpc/CLAUDE.md b/grpc/CLAUDE.md index 7bb05da..f056c0a 100644 --- a/grpc/CLAUDE.md +++ b/grpc/CLAUDE.md @@ -134,3 +134,9 @@ the repository's `.seconds` / `.total` unit naming and the corresponding validat durable operation idempotency adds `:grpc:grpc-operation-ledger-jpa` explicitly, enables the ledger configuration, and includes that adapter's Flyway location `db/migration/grpc`. A deployment that does not adopt the ledger therefore pays no Spring Data JPA classpath cost. + +An `IN_PROGRESS` claim expires after the claim TTL (`GrpcIdempotencyInterceptor`'s `claimTtl` +argument; no configuration key while the starter does not assemble the interceptor). A retry of the +same request then takes it over through `GrpcOperationLedger.reclaimStale`. The adopter must record +the commit in the mutation's transaction and choose a TTL longer than the slowest keyed method; both +reasons are in ADR-GRPC-004. diff --git a/grpc/grpc-core-api/src/main/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedger.java b/grpc/grpc-core-api/src/main/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedger.java index 4342bde..7a4ee67 100644 --- a/grpc/grpc-core-api/src/main/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedger.java +++ b/grpc/grpc-core-api/src/main/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedger.java @@ -52,6 +52,36 @@ public interface GrpcOperationLedger { */ boolean reclaimFailed(GrpcOperationIdentity identity, String requestFingerprint, Instant now); + /** + * Takes over an {@code IN_PROGRESS} claim whose owner is presumed dead, for a new attempt of the + * same request. + * + *

A claim made before {@code claimedBefore} has outlived the time its owner could still be + * working (the claim TTL), so its owner is treated as gone. Without this, a process that dies + * between claim and commit leaves its key blocked forever: every retry waits on, or is refused + * by, a claim nobody will finish. + * + *

A compare-and-set like {@link #reclaimFailed}: the record becomes a fresh {@code + * IN_PROGRESS} claim at {@code now}, and of two concurrent takeovers exactly one returns true — + * the winner's new claim moment is no longer before the cutoff, so the loser's condition fails. + * That is why {@code claimedBefore} after {@code now} is refused: the new claim would itself be + * stale and a second takeover would win too. A record in any other state, claimed at or after the + * cutoff, or made for a different request, is left untouched. + * + *

Safe only because the commit is recorded in the same transaction as the mutation (see {@link + * #markCommitted}): then a claim still {@code IN_PROGRESS} means the mutation never committed. + * When an adopter commits them separately, the abandoned attempt's outcome is unknown and a + * takeover can run the mutation twice. + * + * @return true when this call now owns the record as a fresh attempt claimed at {@code now} + * @throws IllegalArgumentException when {@code claimedBefore} is after {@code now} + */ + boolean reclaimStale( + GrpcOperationIdentity identity, + String requestFingerprint, + Instant claimedBefore, + Instant now); + /** * Releases a claim that will never complete, so the caller may try again. * diff --git a/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedgerContract.java b/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedgerContract.java index 1ff769c..b2839d6 100644 --- a/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedgerContract.java +++ b/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedgerContract.java @@ -4,6 +4,7 @@ import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.catchThrowable; import dev.caskeleton.grpc.core.GrpcMethodName; +import java.time.Duration; import java.time.Instant; import java.util.List; import java.util.Optional; @@ -12,6 +13,7 @@ import java.util.concurrent.CountDownLatch; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.Future; +import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; /** @@ -34,6 +36,10 @@ public abstract class GrpcOperationLedgerContract { "sha256:1122334455667788990011223344556677889900112233445566778899001122"; private static final String OUTCOME = "outcome://document/42"; private static final Instant NOW = Instant.parse("2026-09-25T10:00:00Z"); + private static final Duration CLAIM_TTL = Duration.ofMinutes(5); + // A retry arriving after the TTL: its cutoff (LATER - TTL) is past the original claim at NOW. + private static final Instant LATER = NOW.plus(CLAIM_TTL).plusSeconds(1); + private static final Instant CUTOFF = LATER.minus(CLAIM_TTL); /** The ledger under test, holding no records when a test starts. */ protected abstract GrpcOperationLedger ledger(); @@ -237,6 +243,160 @@ public abstract class GrpcOperationLedgerContract { assertThat(reclaimed).containsExactlyInAnyOrder(true, false); } + @Test + @DisplayName("TTL이 지난 IN_PROGRESS claim은 같은 요청의 재시도가 넘겨받는다") + void aStaleInProgressClaimIsTakenOverByARetryOfTheSameRequest() { + // given + ledger().claim(ID, FINGERPRINT, NOW); + + // when + boolean reclaimed = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER); + + // then + assertThat(reclaimed).isTrue(); + } + + @Test + @DisplayName("넘겨받은 stale claim은 넘겨받은 시각부터 진행 중인 새 시도다") + void aTakenOverStaleClaimIsANewAttemptInProgressSinceTheTakeover() { + // given + ledger().claim(ID, FINGERPRINT, NOW); + + // when + ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER); + + // then + assertThat(ledger().find(ID)) + .get() + .extracting( + GrpcOperationLedgerRecord::state, + GrpcOperationLedgerRecord::claimedAt, + GrpcOperationLedgerRecord::completedAt) + .containsExactly(GrpcOperationLedgerState.IN_PROGRESS, LATER, Optional.empty()); + } + + @Test + @DisplayName("넘겨받은 stale claim은 commit할 수 있다") + void aTakenOverStaleClaimCanBeCommitted() { + // given + ledger().claim(ID, FINGERPRINT, NOW); + ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER); + + // when + ledger().markCommitted(ID, OUTCOME, LATER.plusSeconds(1)); + + // then + assertThat(ledger().find(ID)) + .get() + .extracting(GrpcOperationLedgerRecord::state, GrpcOperationLedgerRecord::outcomeReference) + .containsExactly(GrpcOperationLedgerState.COMMITTED, Optional.of(OUTCOME)); + } + + @Test + @DisplayName("TTL 안의 IN_PROGRESS claim은 넘겨받지 않는다") + void anInProgressClaimWithinTheTtlIsNotTakenOver() { + // given + ledger().claim(ID, FINGERPRINT, NOW); + + // when + boolean reclaimed = ledger().reclaimStale(ID, FINGERPRINT, NOW, NOW.plus(CLAIM_TTL)); + + // then + assertThat(reclaimed).isFalse(); + } + + @Test + @DisplayName("다른 요청은 TTL이 지난 claim도 넘겨받지 못한다") + void aStaleClaimIsNotTakenOverForADifferentRequest() { + // given + ledger().claim(ID, FINGERPRINT, NOW); + + // when + boolean reclaimed = ledger().reclaimStale(ID, OTHER_FINGERPRINT, CUTOFF, LATER); + + // then + assertThat(reclaimed).isFalse(); + } + + @Test + @DisplayName("commit된 claim은 오래되어도 stale로 넘겨받지 않는다") + void aCommittedClaimIsNotTakenOverAsStale() { + // given + ledger().claim(ID, FINGERPRINT, NOW); + ledger().markCommitted(ID, OUTCOME, NOW.plusSeconds(1)); + + // when + boolean reclaimed = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER); + + // then + assertThat(reclaimed).isFalse(); + } + + @Test + @DisplayName("실패한 claim은 stale 회수가 아니라 실패 재claim으로만 다시 연다") + void aFailedClaimIsNotTakenOverAsStale() { + // given + failedClaim(); + + // when + boolean reclaimed = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER); + + // then + assertThat(reclaimed).isFalse(); + } + + @Test + @DisplayName("claim이 없으면 stale 회수는 아무것도 만들지 않는다") + void takingOverAMissingClaimCreatesNothing() { + // when + boolean reclaimed = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER); + + // then + assertThat(reclaimed).isFalse(); + assertThat(ledger().find(ID)).isEmpty(); + } + + @Test + @DisplayName("한 번 넘겨받은 claim은 같은 cutoff로 다시 넘겨받지 못한다") + void aTakenOverClaimIsNotTakenOverAgainWithTheSameCutoff() { + // given + ledger().claim(ID, FINGERPRINT, NOW); + ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER); + + // when + boolean again = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER); + + // then + assertThat(again).isFalse(); + } + + @Test + @DisplayName("TTL이 지난 claim을 동시에 넘겨받으려 하면 승자는 하나다") + void concurrentTakeoversOfAStaleClaimHaveExactlyOneWinner() throws Exception { + // given + ledger().claim(ID, FINGERPRINT, NOW); + + // when + List reclaimed = + concurrently(() -> ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER)); + + // then + assertThat(reclaimed).containsExactlyInAnyOrder(true, false); + } + + @Test + @DisplayName("cutoff가 현재 시각보다 뒤면 거부한다 — 새 claim이 곧바로 stale이 되기 때문이다") + void aCutoffAfterNowIsRefused() { + // given + ledger().claim(ID, FINGERPRINT, NOW); + + // when + Throwable refused = catchThrowable(() -> ledger().reclaimStale(ID, FINGERPRINT, LATER, CUTOFF)); + + // then + assertThat(refused).isInstanceOf(IllegalArgumentException.class).hasMessageContaining("cutoff"); + } + private void failedClaim() { ledger().claim(ID, FINGERPRINT, NOW); ledger().markFailed(ID, NOW.plusSeconds(1)); diff --git a/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/InMemoryOperationLedger.java b/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/InMemoryOperationLedger.java index f08ddac..207971b 100644 --- a/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/InMemoryOperationLedger.java +++ b/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/InMemoryOperationLedger.java @@ -10,9 +10,9 @@ import java.util.concurrent.ConcurrentMap; * *

{@code putIfAbsent} on a concurrent map is the in-memory equivalent of the unique constraint * the real adapter relies on, and {@code replace(key, expected, next)} of its conditional update, - * so claim and re-claim are atomic here for the same reason they are there. The transitions match - * the JPA adapter's because {@link GrpcOperationLedgerContract} runs against both; a fake that - * allowed more would let the policy tests pass for code the database refuses. + * so claim, re-claim and stale takeover are atomic here for the same reason they are there. The + * transitions match the JPA adapter's because {@link GrpcOperationLedgerContract} runs against + * both; a fake that allowed more would let the policy tests pass for code the database refuses. */ public final class InMemoryOperationLedger implements GrpcOperationLedger { @@ -81,6 +81,27 @@ public final class InMemoryOperationLedger implements GrpcOperationLedger { GrpcOperationLedgerRecord.claim(identity, requestFingerprint, now)); } + @Override + public boolean reclaimStale( + GrpcOperationIdentity identity, + String requestFingerprint, + Instant claimedBefore, + Instant now) { + if (claimedBefore.isAfter(now)) { + throw new IllegalArgumentException( + "a stale-claim cutoff after now would leave the new claim stale as well"); + } + GrpcOperationLedgerRecord current = records.get(identity.storageKey()); + return current != null + && current.state() == GrpcOperationLedgerState.IN_PROGRESS + && current.claimedAt().isBefore(claimedBefore) + && current.sameRequestAs(requestFingerprint) + && records.replace( + identity.storageKey(), + current, + GrpcOperationLedgerRecord.claim(identity, requestFingerprint, now)); + } + @Override public void release(GrpcOperationIdentity identity) { records.remove(identity.storageKey()); diff --git a/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedgerContractIT.java b/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedgerContractIT.java index 2d1e6b2..701b31b 100644 --- a/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedgerContractIT.java +++ b/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedgerContractIT.java @@ -22,6 +22,7 @@ import java.util.Optional; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; import org.springframework.data.jpa.repository.support.JpaRepositoryFactory; import org.springframework.orm.jpa.JpaTransactionManager; @@ -120,6 +121,44 @@ class JpaGrpcOperationLedgerContractIT extends GrpcOperationLedgerContract { .containsExactly(GrpcOperationLedgerState.COMMITTED, Optional.of("outcome://document/42")); } + /** + * The takeover's conditional update clears the persistence context for the same reason the failed + * re-claim does: the claim's read-on-conflict loaded the stale row, and a commit reading that + * copy would keep the abandoned claim moment. + */ + @Test + @DisplayName("stale claim을 넘겨받아 같은 트랜잭션에서 commit하면 commit된다") + void aRetryThatTakesOverAStaleClaimAndCommitsInOneTransactionCommits() { + // given + GrpcOperationIdentity id = + new GrpcOperationIdentity( + "tenant-1.actor-1", + GrpcMethodName.parse("caskeleton.document.v1.DocumentService/CreateDocument"), + "sha256:aabbccddeeff00112233445566778899aabbccddeeff00112233445566778899"); + String fingerprint = "sha256:0f1e2d3c4b5a69788796a5b4c3d2e1f00f1e2d3c4b5a69788796a5b4c3d2e1f0"; + Instant abandonedAt = Instant.parse("2026-09-25T10:00:00Z"); + Instant retryAt = abandonedAt.plusSeconds(301); + ledger.claim(id, fingerprint, abandonedAt); + + // when + transactions.executeWithoutResult( + status -> { + adapter.claim(id, fingerprint, retryAt); + adapter.reclaimStale(id, fingerprint, retryAt.minusSeconds(300), retryAt); + adapter.markCommitted(id, "outcome://document/42", retryAt.plusSeconds(1)); + }); + + // then + assertThat(ledger.find(id)) + .get() + .extracting( + GrpcOperationLedgerRecord::state, + GrpcOperationLedgerRecord::claimedAt, + GrpcOperationLedgerRecord::outcomeReference) + .containsExactly( + GrpcOperationLedgerState.COMMITTED, retryAt, Optional.of("outcome://document/42")); + } + private static void execute(String sql) throws SQLException { try (Connection connection = DriverManager.getConnection( @@ -177,6 +216,17 @@ class JpaGrpcOperationLedgerContractIT extends GrpcOperationLedgerContract { status -> delegate.reclaimFailed(identity, requestFingerprint, now))); } + @Override + public boolean reclaimStale( + GrpcOperationIdentity identity, + String requestFingerprint, + Instant claimedBefore, + Instant now) { + return Boolean.TRUE.equals( + transactions.execute( + status -> delegate.reclaimStale(identity, requestFingerprint, claimedBefore, now))); + } + @Override public void release(GrpcOperationIdentity identity) { transactions.executeWithoutResult(status -> delegate.release(identity)); diff --git a/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepository.java b/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepository.java index f1823b8..32d14c0 100644 --- a/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepository.java +++ b/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepository.java @@ -92,6 +92,36 @@ public interface GrpcOperationLedgerRepository @Param("requestFingerprint") String requestFingerprint, @Param("claimedAt") Instant claimedAt); + /** + * Takes over an {@code IN_PROGRESS} row claimed before {@code claimedBefore}, for the same + * request. + * + *

Same shape as {@link #tryReclaimFailed}: one conditional update decided by PostgreSQL's row + * lock. A concurrent second takeover waits for the winner, re-evaluates the condition against the + * winner's row — whose {@code claimed_at} is now {@code :claimedAt}, not before the cutoff — and + * updates nothing. It clears the persistence context for the same reason. + * + * @return one when this caller took the row over, zero otherwise + */ + @Modifying(flushAutomatically = true, clearAutomatically = true) + @Transactional + @Query( + value = + """ + UPDATE grpc_operation_ledger + SET claimed_at = :claimedAt + WHERE storage_key = :storageKey + AND state = 'IN_PROGRESS' + AND request_fingerprint = :requestFingerprint + AND claimed_at < :claimedBefore + """, + nativeQuery = true) + int tryReclaimStale( + @Param("storageKey") String storageKey, + @Param("requestFingerprint") String requestFingerprint, + @Param("claimedBefore") Instant claimedBefore, + @Param("claimedAt") Instant claimedAt); + /** Updates an existing claim after it has been loaded. */ GrpcOperationLedgerEntity save(GrpcOperationLedgerEntity entity); diff --git a/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedger.java b/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedger.java index 7cd1afc..d81fd66 100644 --- a/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedger.java +++ b/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedger.java @@ -107,6 +107,21 @@ public final class JpaGrpcOperationLedger implements GrpcOperationLedger { return repository.tryReclaimFailed(identity.storageKey(), requestFingerprint, now) == 1; } + @Override + public boolean reclaimStale( + GrpcOperationIdentity identity, + String requestFingerprint, + Instant claimedBefore, + Instant now) { + requireIdentity(identity); + if (claimedBefore.isAfter(now)) { + throw new IllegalArgumentException( + "a stale-claim cutoff after now would leave the new claim stale as well"); + } + return repository.tryReclaimStale(identity.storageKey(), requestFingerprint, claimedBefore, now) + == 1; + } + @Override public void release(GrpcOperationIdentity identity) { requireIdentity(identity); diff --git a/grpc/grpc-operation-ledger-jpa/src/test/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepositoryTest.java b/grpc/grpc-operation-ledger-jpa/src/test/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepositoryTest.java index 8da7c64..9b7fcb4 100644 --- a/grpc/grpc-operation-ledger-jpa/src/test/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepositoryTest.java +++ b/grpc/grpc-operation-ledger-jpa/src/test/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepositoryTest.java @@ -66,6 +66,13 @@ class GrpcOperationLedgerRepositoryTest { "the re-claim is certified against PostgreSQL by JpaGrpcOperationLedgerContractIT"); } + @Override + public int tryReclaimStale( + String storageKey, String requestFingerprint, Instant claimedBefore, Instant claimedAt) { + throw new UnsupportedOperationException( + "the stale takeover is certified against PostgreSQL by JpaGrpcOperationLedgerContractIT"); + } + @Override public GrpcOperationLedgerEntity save(GrpcOperationLedgerEntity entity) { rows.put(entity.getStorageKey(), entity); diff --git a/grpc/grpc-runtime/src/main/java/dev/caskeleton/grpc/idempotency/GrpcIdempotencyInterceptor.java b/grpc/grpc-runtime/src/main/java/dev/caskeleton/grpc/idempotency/GrpcIdempotencyInterceptor.java index e232669..b48d9a6 100644 --- a/grpc/grpc-runtime/src/main/java/dev/caskeleton/grpc/idempotency/GrpcIdempotencyInterceptor.java +++ b/grpc/grpc-runtime/src/main/java/dev/caskeleton/grpc/idempotency/GrpcIdempotencyInterceptor.java @@ -19,6 +19,10 @@ import java.util.Optional; * client takes the same one before deciding whether a retry is safe. An interceptor that owns the * logic leaves the client re-deriving it. * + *

A claim does not block its key forever. One still {@code IN_PROGRESS} after the claim TTL is + * treated as abandoned by a dead owner and taken over by the next retry of the same request — one + * winner, like the re-claim of a terminal failure. + * *

The ordering matters and is easy to get subtly wrong. The claim is attempted first, * atomically; only then is the existing record examined. Reading first and claiming second leaves a * window in which two attempts both read nothing. @@ -29,18 +33,24 @@ public final class GrpcIdempotencyInterceptor { private final GrpcMethodPolicyCatalog catalog; private final Duration inProgressPollInterval; private final boolean waitForInProgress; + private final Duration claimTtl; /** * @param inProgressPollInterval how long a duplicate is told to wait before asking again * @param waitForInProgress whether a duplicate of a running operation waits or is refused. A * method whose caller is a user-facing request usually wants to be refused quickly; a worker * usually wants to wait. + * @param claimTtl how long an {@code IN_PROGRESS} claim is trusted to have a live owner. After + * it, a retry of the same request takes the claim over. It must be longer than every keyed + * method's deadline, and longer than the slowest such method can actually run: a claim taken + * over from an owner that is merely slow runs the mutation twice. */ public GrpcIdempotencyInterceptor( GrpcOperationLedger ledger, GrpcMethodPolicyCatalog catalog, Duration inProgressPollInterval, - boolean waitForInProgress) { + boolean waitForInProgress, + Duration claimTtl) { if (ledger == null || catalog == null) { throw new IllegalArgumentException("an idempotency interceptor needs a ledger and a catalog"); } @@ -54,6 +64,32 @@ public final class GrpcIdempotencyInterceptor { this.catalog = catalog; this.inProgressPollInterval = inProgressPollInterval; this.waitForInProgress = waitForInProgress; + this.claimTtl = requireClaimTtlOutlivesKeyedDeadlines(claimTtl, catalog); + } + + // A TTL inside a keyed method's deadline lets a retry take over a claim whose owner is still + // within its budget. The deadline is the only per-method bound on owner lifetime the catalog + // knows; it does not bound a server that keeps working after the deadline fires. + private static Duration requireClaimTtlOutlivesKeyedDeadlines( + Duration claimTtl, GrpcMethodPolicyCatalog catalog) { + if (claimTtl == null || claimTtl.isZero() || claimTtl.isNegative()) { + throw new IllegalArgumentException("a claim TTL must be present and positive"); + } + for (GrpcMethodName method : catalog.methods()) { + GrpcMethodPolicy policy = catalog.require(method); + if (policy.idempotency().idempotencyKeyRequired() + && claimTtl.compareTo(policy.deadline().total()) <= 0) { + throw new IllegalArgumentException( + "claim TTL " + + claimTtl + + " is not longer than the " + + policy.deadline().total() + + " deadline of keyed method '" + + method.canonical() + + "'; a retry would take over a claim whose owner is still running"); + } + } + return claimTtl; } /** @@ -115,7 +151,22 @@ public final class GrpcIdempotencyInterceptor { return GrpcIdempotencyDecision.proceed( "the previous attempt failed terminally; this caller re-claimed it for a new attempt"); } - // Running, or a failed attempt another retry re-claimed first. + if (record.state() == GrpcOperationLedgerState.IN_PROGRESS) { + Instant cutoff = now.minus(claimTtl); + if (record.claimedAt().isBefore(cutoff) + && ledger.reclaimStale(identity, requestFingerprint, cutoff, now)) { + // The owner outlived the TTL without committing or failing, so it is presumed dead. Its + // mutation did not commit if the commit was recorded in the mutation's own transaction, + // which the ledger contract requires; when an adopter commits them separately, the + // abandoned attempt's outcome is unknown and this re-run can duplicate it. + return GrpcIdempotencyDecision.proceed( + "the previous claim was abandoned (older than the " + + claimTtl + + " claim TTL); this caller took it over for a new attempt"); + } + } + // Running, a failed attempt another retry re-claimed first, or an abandoned claim another + // retry took over first. return waitForInProgress ? GrpcIdempotencyDecision.waitAndPoll(inProgressPollInterval) : GrpcIdempotencyDecision.rejectInProgress(); diff --git a/grpc/grpc-runtime/src/test/java/dev/caskeleton/grpc/idempotency/GrpcCompletionReconcilerTest.java b/grpc/grpc-runtime/src/test/java/dev/caskeleton/grpc/idempotency/GrpcCompletionReconcilerTest.java index 972630e..90c3421 100644 --- a/grpc/grpc-runtime/src/test/java/dev/caskeleton/grpc/idempotency/GrpcCompletionReconcilerTest.java +++ b/grpc/grpc-runtime/src/test/java/dev/caskeleton/grpc/idempotency/GrpcCompletionReconcilerTest.java @@ -2,8 +2,10 @@ package dev.caskeleton.grpc.idempotency; import static org.assertj.core.api.Assertions.assertThat; +import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity; import dev.caskeleton.grpc.api.ledger.InMemoryOperationLedger; import dev.caskeleton.grpc.core.GrpcMethodName; +import java.time.Duration; import java.time.Instant; import java.util.Optional; import org.junit.jupiter.api.DisplayName; @@ -29,6 +31,28 @@ class GrpcCompletionReconcilerTest { NOW); } + @Test + @DisplayName("버려진 claim으로 대기열에 오른 건은 넘겨받은 재시도가 commit하면 다음 회차에 COMMITTED로 풀린다") + void anAbandonedClaimStaysPendingUntilATakeoverCommits() { + // given + GrpcOperationIdentity identity = + new GrpcOperationIdentity(CALLER, CREATE, GrpcRequestFingerprint.hashIdempotencyKey(KEY)); + String fingerprint = GrpcRequestFingerprint.of("body"); + Instant afterTtl = NOW.plus(Duration.ofMinutes(5)).plusSeconds(1); + GrpcCompletionReconciler reconciler = new GrpcCompletionReconciler(query); + claim(); + reconciler.reconcile(CREATE, CALLER, KEY, null); + ledger.reclaimStale(identity, fingerprint, afterTtl.minus(Duration.ofMinutes(5)), afterTtl); + ledger.markCommitted(identity, "outcome://document/42", afterTtl.plusSeconds(1)); + + // when + GrpcCompletionResolution nextPass = reconciler.reconcile(CREATE, CALLER, KEY, null); + + // then + assertThat(nextPass.status()).isEqualTo(GrpcOperationStatus.COMMITTED); + assertThat(nextPass.outcomeReference()).contains("outcome://document/42"); + } + @Test @DisplayName("a status query returns the five documented answers") void theQueryReturnsTheFiveDocumentedAnswers() { @@ -141,6 +165,5 @@ class GrpcCompletionReconcilerTest { GrpcCompletionResolution resolution = reconciler.reconcile(CREATE, CALLER, KEY, null); assertThat(resolution.status()).isEqualTo(GrpcOperationStatus.IN_PROGRESS); - assertThat(reconciler.pendingCases()).hasSize(1); } } diff --git a/grpc/grpc-runtime/src/test/java/dev/caskeleton/grpc/idempotency/GrpcIdempotencyInterceptorTest.java b/grpc/grpc-runtime/src/test/java/dev/caskeleton/grpc/idempotency/GrpcIdempotencyInterceptorTest.java index 6ed3d16..df26ea2 100644 --- a/grpc/grpc-runtime/src/test/java/dev/caskeleton/grpc/idempotency/GrpcIdempotencyInterceptorTest.java +++ b/grpc/grpc-runtime/src/test/java/dev/caskeleton/grpc/idempotency/GrpcIdempotencyInterceptorTest.java @@ -2,6 +2,7 @@ package dev.caskeleton.grpc.idempotency; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.assertj.core.api.Assertions.catchThrowable; import dev.caskeleton.grpc.api.deadline.GrpcDeadlineProfile; import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity; @@ -36,6 +37,8 @@ class GrpcIdempotencyInterceptorTest { private static final Instant NOW = Instant.parse("2026-08-30T10:00:00Z"); private static final String CALLER = "tenant-1|actor-1"; private static final String KEY = "order-2026-0001"; + private static final Duration CLAIM_TTL = Duration.ofMinutes(5); + private static final Instant AFTER_TTL = NOW.plus(CLAIM_TTL).plusSeconds(1); private final InMemoryOperationLedger ledger = new InMemoryOperationLedger(); @@ -57,7 +60,7 @@ class GrpcIdempotencyInterceptorTest { private GrpcIdempotencyInterceptor interceptor(boolean waitForInProgress) { return new GrpcIdempotencyInterceptor( - ledger, catalog(), Duration.ofMillis(250), waitForInProgress); + ledger, catalog(), Duration.ofMillis(250), waitForInProgress, CLAIM_TTL); } @Test @@ -194,6 +197,107 @@ class GrpcIdempotencyInterceptorTest { GrpcIdempotencyDecision.Outcome.REJECT_IN_PROGRESS); } + @Test + @DisplayName("소유자가 죽어 TTL이 지난 claim은 같은 요청의 재시도가 넘겨받아 진행한다") + void aRetryAfterTheClaimTtlTakesOverAnAbandonedClaimAndProceeds() { + // given + GrpcIdempotencyInterceptor interceptor = interceptor(false); + String fingerprint = GrpcRequestFingerprint.of("body"); + interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW); + + // when + GrpcIdempotencyDecision decision = + interceptor.decide(CREATE, CALLER, KEY, fingerprint, AFTER_TTL); + + // then + assertThat(decision.outcome()).isEqualTo(GrpcIdempotencyDecision.Outcome.PROCEED); + assertThat(decision.reason()).contains("abandoned"); + } + + @Test + @DisplayName("넘겨받은 claim은 재시도 시각에 다시 잡힌 진행 중 claim이다") + void aTakenOverClaimIsHeldInProgressFromTheRetry() { + // given + GrpcIdempotencyInterceptor interceptor = interceptor(false); + String fingerprint = GrpcRequestFingerprint.of("body"); + interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW); + + // when + interceptor.decide(CREATE, CALLER, KEY, fingerprint, AFTER_TTL); + + // then + assertThat( + ledger + .find( + new GrpcOperationIdentity( + CALLER, CREATE, GrpcRequestFingerprint.hashIdempotencyKey(KEY))) + .map(GrpcOperationLedgerRecord::claimedAt)) + .contains(AFTER_TTL); + } + + @Test + @DisplayName("TTL이 지나기 직전의 중복 요청은 여전히 대기하거나 거부된다") + void aDuplicateAtTheTtlBoundaryIsStillTreatedAsRunning() { + // given + GrpcIdempotencyInterceptor interceptor = interceptor(false); + String fingerprint = GrpcRequestFingerprint.of("body"); + interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW); + + // when + GrpcIdempotencyDecision decision = + interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW.plus(CLAIM_TTL)); + + // then + assertThat(decision.outcome()).isEqualTo(GrpcIdempotencyDecision.Outcome.REJECT_IN_PROGRESS); + } + + @Test + @DisplayName("버려진 claim에 동시에 온 재시도 중 하나만 진행한다") + void concurrentRetriesOfAnAbandonedClaimProceedOnlyOnce() throws Exception { + // given + GrpcIdempotencyInterceptor interceptor = interceptor(false); + String fingerprint = GrpcRequestFingerprint.of("body"); + interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW); + CountDownLatch start = new CountDownLatch(1); + Callable retry = + () -> { + start.await(); + return interceptor.decide(CREATE, CALLER, KEY, fingerprint, AFTER_TTL).outcome(); + }; + + // when + List outcomes; + try (ExecutorService executor = Executors.newFixedThreadPool(2)) { + Future first = executor.submit(retry); + Future second = executor.submit(retry); + start.countDown(); + outcomes = List.of(first.get(), second.get()); + } + + // then + assertThat(outcomes) + .containsExactlyInAnyOrder( + GrpcIdempotencyDecision.Outcome.PROCEED, + GrpcIdempotencyDecision.Outcome.REJECT_IN_PROGRESS); + } + + @Test + @DisplayName("claim TTL이 key 필수 method의 deadline보다 짧으면 생성 시 거부한다") + void aClaimTtlNoLongerThanAKeyedMethodsDeadlineIsRefused() { + // when + Throwable refused = + catchThrowable( + () -> + new GrpcIdempotencyInterceptor( + ledger, catalog(), Duration.ofMillis(250), false, Duration.ofSeconds(10))); + + // then + assertThat(refused) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining(CREATE.canonical()) + .hasMessageContaining("claim TTL"); + } + @Test @DisplayName("one caller's key does not suppress another caller's write") void oneCallersKeyDoesNotSuppressAnother() {