fix!: 재리뷰가 찾은 결함과 색인에서 빠졌던 발견 5건을 고친다

문제: 성숙도 리뷰 수정을 다시 읽은 재리뷰(Critical 0, 회귀 0)가
Important 27건·Minor 41건을 찾았다. 수정이 새로 만든 결함이 많았다.
@SystemInvoked가 권한 규칙의 탈출구가 됐고, 데모 계정 거부가 정확히
prod일 때만 걸렸고, outbox 순서 번호가 업그레이드 때 뒤로 갔고,
GraphQL union 아래 복잡도가 0이었고, 서블릿 오류 응답이 success:true로
나갔고, 호출자 deadline이 breaker를 열었다. 원래 리뷰 색인이 06 영역
발견 5건(06-10~14)을 빠뜨려 그동안 처리되지 않았다.

방법: 영역별 7묶음이 test-driven-development 절차로 실패 테스트를
먼저 쓰고 고쳤다. 결정은 재리뷰 README 4절에 적었다: 순서 번호는 V21이
기록한 전역 floor부터, 데모 계정은 local·test·demo 허용 목록,
@SystemInvoked bean은 기동 실패, complete 단계 실패는
IDEMPOTENCY_COMPLETION_UNKNOWN(결과 불명), 캐시 조건부 쓰기는 별도
SCRIPT lane, strict lane 정책은 build cache 입력, 끝나지 않는 재시도
행은 원인 코드·지표·PARKED 절차. 레지스트리가 선언만 하고 아무 코드도
내보내지 않던 redis.capability.* 지표 6개는 빼고 runbook을 실제
지표(redis.lane.*) 기준으로 고쳤다.

위험·한계: V21을 V22 없이 직접 고쳤다(어느 배포에도 아직 없음). 이
브랜치의 V21을 적용한 개발 DB는 다시 만들어야 한다. 서블릿 오류
디스패치를 보안 체인이 허용한다. 발행·저장소 포트 시그니처가 바뀌어
파생 서비스 코드가 깨진다. prod에서 fixture-로 시작하는 설정값은
기동을 막는다. httpclient failure-injection lane의 main 단계 중복
실행은 이전 결정대로 남겼다. 실제 CI runner 실행은 확인하지 못했다.

검증: ci, mainCheck, :integrationCheck, -p optional-platforms ci
integrationCheck performanceCheck, :performanceCheck, :releaseCheck
(-PgitRevision), verify-compose-profile-contracts(18 lane) 통과.
첫 실행의 실패 2건(FindSecBugs 1, sample 동시 생성 테스트의 v4 event
id)은 고친 뒤 ci·integrationCheck·mainCheck를 다시 돌려 통과했다.

Refs: docs/reviews/2026-11-03-rereview/README.md 3·4·6·7절
BREAKING CHANGE: outbox 발행 포트와 저장소 포트 시그니처, 순서 번호
첫 값(V21 floor 다음), 샘플 데모 계정 프로필 허용 목록, 새 오류 코드
IDEMPOTENCY_COMPLETION_UNKNOWN, prod의 fixture- 값 거부, 기동 rule은
static·의존 없음이어야 한다.
This commit is contained in:
DongHyeonka
2026-11-03 23:29:35 +09:00
parent 96afc8647f
commit e6db162932
25 changed files with 1032 additions and 218 deletions
+34
View File
@@ -1,5 +1,9 @@
plugins { plugins {
id 'ca.architecture' id 'ca.architecture'
// Every strict lane of this build belongs to a stage a workflow executes, checked here exactly
// as in the main build: the main build's check cannot see this build's lanes, and while only it
// applied the plugin, grpcPerformanceTest ran nowhere (review 2026-11-03 RR-06-03).
id 'ca.strict-lane-stages'
} }
// No `ca.release-provenance` any more: each gRPC leaf's `ca.java-conventions` derives its version // No `ca.release-provenance` any more: each gRPC leaf's `ca.java-conventions` derives its version
@@ -47,3 +51,33 @@ tasks.register('integrationCheck') {
description = 'Runs every optional-platform leaf integration test lane. Needs a container runtime.' description = 'Runs every optional-platform leaf integration test lane. Needs a container runtime.'
dependsOn subprojects.findAll { it.childProjects.isEmpty() }.collect { "${it.path}:integrationTestLaneCheck" } dependsOn subprojects.findAll { it.childProjects.isEmpty() }.collect { "${it.path}:integrationTestLaneCheck" }
} }
// The performance stage of this build: grpcPerformanceTest. It asserts the shape of a measurement
// (ordered percentiles, a gate that reads them) with a generous budget, not a number, so it can run
// on a shared runner. integration-main.yml's performance-check job runs it next to the main build's
// `:performanceCheck`; before this aggregate it ran when somebody asked for it by name, which was
// nobody (review 2026-11-03 RR-06-03).
tasks.register('performanceCheck') {
group = 'verification'
description = 'Runs every optional-platform leaf performance test lane.'
dependsOn subprojects.findAll { it.childProjects.isEmpty() }.collect { "${it.path}:performanceTestLaneCheck" }
}
// The tag-only stage of this build: the hermetic strict lanes (the three gRPC certification lanes).
// Their tests already run inside each leaf's `test`; the lanes add the one thing `test` cannot say —
// that a tag still selects something — which is worth asserting once per release tag. release.yml
// runs this aggregate instead of naming the lanes, so a lane added here is in the release gate
// without a workflow edit.
tasks.register('releaseCheck') {
group = 'verification'
description = 'Runs every optional-platform leaf hermetic strict test lane (the release gate).'
dependsOn subprojects.findAll { it.childProjects.isEmpty() }.collect { "${it.path}:testLaneCheck" }
}
// The stages this build's strict lanes may belong to, executed by workflows as
// `./gradlew -p optional-platforms <stage>` from `src`.
strictLaneStages {
stage ':ci', ':integrationCheck', ':performanceCheck', ':releaseCheck'
workflowDirectory = layout.projectDirectory.dir('../../.github/workflows')
workflowProjectDirectory = 'optional-platforms'
}
@@ -69,6 +69,36 @@ method can run longer than any fixed TTL; revisit when one exists. A sweeper job
rows — it needs a scheduler this family does not have, and deleting a claim makes the next retry a rows — it needs a scheduler this family does not have, and deleting a claim makes the next retry a
fresh claim with no record that an attempt was abandoned. fresh claim with no record that an attempt was abandoned.
**A taken-over owner is fenced by a claim token** (re-review 2026-11-03, RR-02-04). The owner a
takeover presumes dead may only be slow. Without knowing which attempt held the claim, its late
`markFailed` turned the new attempt's claim `FAILED_TERMINAL`, the next retry re-claimed it, and two
attempts ran the mutation at once; its late `markCommitted` committed beside the new attempt. Every
claim now carries a `GrpcClaimToken` (`claim_attempt`, migration V003): 1 for the first claim, one more
for each `reclaimFailed` or `reclaimStale`. `decide` returns it on `PROCEED`, and `recordCommit` /
`recordFailure` hand it back. A commit or a failure is one conditional
`UPDATE … WHERE state = 'IN_PROGRESS' AND claim_attempt = :attempt`: a commit by an attempt that lost
the claim is refused with an exception, so its business transaction rolls back; a failure by one is
ignored (`markFailed` returns false). The same conditional update closes the JPA commit race —
load-check-save let two commits that both read `IN_PROGRESS` both write; now exactly one matches.
A claim whose conflicting row is purged between insert and read retries the insert once instead of
failing the call.
Chosen over the claim moment as the token (`WHERE claimed_at = :claimedAt`), which needs no column:
PostgreSQL stores microseconds while `Instant` carries nanoseconds, so the token a caller holds and
the one stored can differ by rounding, and two processes' clocks can give two attempts the same
moment. The counter has neither problem. The heartbeat lease above stays rejected for the same reason
as before; the token fences without one.
**What `operation-ledger-enabled=true` promises (03-11 (b), recorded 2026-11-03, RR-02-07).** The
2026-11-01 review offered (a) delete the flag and judge by whether a ledger bean exists, or (b) keep the
flag and have the starter assemble the decision function from it. (b) is kept: the assembly and the
TTL validation live in one place instead of in every adopter. (a) was rejected because it removes the
false promise only by removing the assembly with it. The limit stays and is stated where an operator
meets it: the starter builds no server, so the flag assembles the decision function only, and keys are
honoured once the adopter's `IDEMPOTENCY` stage interceptor or service adapter calls `decide` and
`recordCommit`. `GrpcPlatformStartupValidator`'s refusal for a keyed method says exactly that; a running
process cannot see whether the stage exists, so the validator does not claim to check it.
**Finished rows are purged after a retention; in-progress rows never are** (03-10 follow-up). **Finished rows are purged after a retention; in-progress rows never are** (03-10 follow-up).
`GrpcOperationLedger.purgeCompletedBefore(cutoff, limit)` deletes at most `limit` `COMMITTED` or `GrpcOperationLedger.purgeCompletedBefore(cutoff, limit)` deletes at most `limit` `COMMITTED` or
`FAILED_TERMINAL` rows that completed before the cutoff, oldest first — on PostgreSQL one bounded `FAILED_TERMINAL` rows that completed before the cutoff, oldest first — on PostgreSQL one bounded
+3 -1
View File
@@ -44,7 +44,9 @@ the time, and which half is not knowable from the status.
and a later reconciler pass sees that attempt's `COMMITTED` or `FAILED_TERMINAL`. Until a retry and a later reconciler pass sees that attempt's `COMMITTED` or `FAILED_TERMINAL`. Until a retry
arrives, the case stays pending. The re-run is safe only when the ledger commit is recorded in the arrives, the case stays pending. The re-run is safe only when the ledger commit is recorded in the
mutation's own transaction; if this service commits them separately, check the business resource mutation's own transaction; if this service commits them separately, check the business resource
before letting the retry through. before letting the retry through. An owner that was only slow cannot spoil the takeover: its claim
attempt (`claim_attempt`) is behind, so its late commit is refused and its late failure ignored.
A log line saying an operation "is no longer held by attempt N" is that refusal, not an outage.
**Escalate when** the reconciler's pending list grows across passes. That means the ledger is **Escalate when** the reconciler's pending list grows across passes. That means the ledger is
unreachable rather than slow. A case pending as `IN_PROGRESS` for longer than the claim TTL with no unreachable rather than slow. A case pending as `IN_PROGRESS` for longer than the claim TTL with no
+8 -5
View File
@@ -145,7 +145,7 @@ the repository's `.seconds` / `.total` unit naming and the corresponding validat
`grpc-spring-boot-starter` does not pull the JPA ledger adapter transitively (11-M9). A deployment that `grpc-spring-boot-starter` does not pull the JPA ledger adapter transitively (11-M9). A deployment that
needs durable operation idempotency adds `:grpc:grpc-operation-ledger-jpa` explicitly, defines a needs durable operation idempotency adds `:grpc:grpc-operation-ledger-jpa` explicitly, defines a
`GrpcOperationLedger` bean (`new JpaGrpcOperationLedger(repository)`), includes that adapter's Flyway `GrpcOperationLedger` bean (`new JpaGrpcOperationLedger(repository)`), includes that adapter's Flyway
location `db/migration/grpc` (V001 table, V002 retention index), and sets the flag below. A deployment location `db/migration/grpc` (V001 table, V002 retention index, V003 claim attempt), and sets the flag below. A deployment
that does not adopt the ledger pays no Spring Data JPA classpath cost. that does not adopt the ledger pays no Spring Data JPA classpath cost.
| Key (`app.grpc.platform.*`) | Default | What it does | | Key (`app.grpc.platform.*`) | Default | What it does |
@@ -165,9 +165,9 @@ optional build is not part of the shipped runtime. They join the registry when a
adopts the platform. adopts the platform.
**Why the starter assembles the interceptor (03-11 (b)), and not (a).** The review proposed (a): **Why the starter assembles the interceptor (03-11 (b)), and not (a).** The review proposed (a):
delete the flag and judge by whether a ledger bean exists. The owner chose (b): the flag stays and delete the flag and judge by whether a ledger bean exists. (b) was chosen and is recorded in
means what it says — when true, the decision function is assembled and its TTL comes from ADR-GRPC-004 (re-review 2026-11-03, RR-02-07): the flag stays and means what it says — when true, the
configuration. (a) would have removed the false promise but left every adopter to rebuild the same decision function is assembled and its TTL comes from configuration. (a) would have removed the false promise but left every adopter to rebuild the same
assembly and TTL validation by hand; (b) keeps that in one place. The interceptor bean has no assembly and TTL validation by hand; (b) keeps that in one place. The interceptor bean has no
`@ConditionalOnMissingBean` on purpose: a replacement would silently ignore `operation-ledger-claim-ttl`. `@ConditionalOnMissingBean` on purpose: a replacement would silently ignore `operation-ledger-claim-ttl`.
Duplicates of a running operation are refused (`REJECT_IN_PROGRESS`), not told to wait; a Duplicates of a running operation are refused (`REJECT_IN_PROGRESS`), not told to wait; a
@@ -179,7 +179,10 @@ server, so assembling the decision does not wire the platform into `adapter:inbo
The adopter injects the bean by type and calls it from its own `IDEMPOTENCY` stage interceptor The adopter injects the bean by type and calls it from its own `IDEMPOTENCY` stage interceptor
(`GrpcServerInterceptorChain.builder().stage(GrpcServerInterceptorStage.IDEMPOTENCY, ...)`) or from (`GrpcServerInterceptorChain.builder().stage(GrpcServerInterceptorStage.IDEMPOTENCY, ...)`) or from
its typed service adapter: `decide(...)` before the mutation, `recordCommit(...)` inside the mutation's its typed service adapter: `decide(...)` before the mutation, `recordCommit(...)` inside the mutation's
transaction, `recordFailure(...)` on a terminal failure. transaction, `recordFailure(...)` on a terminal failure. Both record calls take the `GrpcClaimToken`
the `PROCEED` decision carried: a commit by an attempt whose claim was taken over is refused (its
transaction rolls back) and its failure is ignored, so a slow owner cannot write over the attempt that
took its claim (re-review RR-02-04, ADR-GRPC-004).
**Purge scheduling is the adopter's.** `GrpcOperationLedgerRetention.purgeExpired(now)` deletes **Purge scheduling is the adopter's.** `GrpcOperationLedgerRetention.purgeExpired(now)` deletes
expired finished rows in bounded batches (500 per statement) until a batch comes back short. The expired finished rows in bounded batches (500 per statement) until a batch comes back short. The
@@ -0,0 +1,35 @@
package dev.caskeleton.grpc.api.ledger;
/**
* Which attempt holds a claim: the fencing token a commit or a failure record must present.
*
* <p>A claim can change hands. A stale one is taken over after the claim TTL, and a failed one is
* re-claimed by the next retry; both make a new attempt of the same operation. The owner that lost
* the claim may still be running, and when it finishes it records its outcome. Without a token that
* record lands on the new attempt's claim: an old owner's failure turned the new attempt's claim
* {@code FAILED_TERMINAL}, a third retry re-claimed it, and two attempts ran the mutation at once
* (re-review 2026-11-03, RR-02-04). With it, the ledger records an outcome only for the attempt
* that still holds the claim.
*
* <p>A counter rather than the claim moment: it increases by one with every hand-over, so two
* attempts never share a value, whatever the clocks of the processes that claimed it.
*
* @param attempt one for the first claim of an operation, one more for each hand-over
*/
public record GrpcClaimToken(long attempt) {
/** The token of the claim that created the record. */
public static final GrpcClaimToken FIRST = new GrpcClaimToken(1);
/** Requires a positive attempt number. */
public GrpcClaimToken {
if (attempt < 1) {
throw new IllegalArgumentException("a claim attempt is numbered from one, was " + attempt);
}
}
/** The token of the attempt that takes this claim over. */
public GrpcClaimToken next() {
return new GrpcClaimToken(Math.addExact(attempt, 1));
}
}
@@ -20,7 +20,8 @@ public interface GrpcOperationLedger {
* implementation admits both concurrent attempts, which is the exact failure the ledger exists to * implementation admits both concurrent attempts, which is the exact failure the ledger exists to
* prevent, and it does so only under the load where it matters. * prevent, and it does so only under the load where it matters.
* *
* @return the existing record when one was already claimed, empty when this call created it * @return the existing record when one was already claimed, empty when this call created it — the
* caller then holds the claim as {@link GrpcClaimToken#FIRST}
*/ */
Optional<GrpcOperationLedgerRecord> claim( Optional<GrpcOperationLedgerRecord> claim(
GrpcOperationIdentity identity, String requestFingerprint, Instant now); GrpcOperationIdentity identity, String requestFingerprint, Instant now);
@@ -34,11 +35,34 @@ public interface GrpcOperationLedger {
* <p>Called inside the business transaction wherever the datastore allows it. When the ledger and * <p>Called inside the business transaction wherever the datastore allows it. When the ledger and
* the mutation commit separately, there is a window in which the mutation is durable and the * the mutation commit separately, there is a window in which the mutation is durable and the
* claim is not, and a retry in that window runs the mutation twice. * claim is not, and a retry in that window runs the mutation twice.
*
* <p>Only the attempt that holds the claim commits it. An attempt whose claim was taken over or
* re-claimed since is refused with an exception, so its business transaction rolls back rather
* than committing beside the attempt that now owns the operation. Of two concurrent commits of
* one claim, exactly one succeeds.
*
* @param claimToken the token of the attempt committing, from the claim it won
* @throws IllegalStateException when there is no claim, it already finished, or it is no longer
* held by {@code claimToken}
*/ */
void markCommitted(GrpcOperationIdentity identity, String outcomeReference, Instant now); void markCommitted(
GrpcOperationIdentity identity,
GrpcClaimToken claimToken,
String outcomeReference,
Instant now);
/** Marks a claim terminally failed. */ /**
void markFailed(GrpcOperationIdentity identity, Instant now); * Marks a claim terminally failed, if {@code claimToken} still holds it.
*
* <p>An attempt that lost its claim reports a failure the operation no longer has: another
* attempt is running it. That record is ignored rather than written over the running attempt's
* claim.
*
* @param claimToken the token of the attempt that failed, from the claim it won
* @return true when the failure was recorded; false when the claim is missing, already finished,
* or held by another attempt, in which case nothing changed
*/
boolean markFailed(GrpcOperationIdentity identity, GrpcClaimToken claimToken, Instant now);
/** /**
* Re-claims a {@code FAILED_TERMINAL} record for a new attempt of the same request. * Re-claims a {@code FAILED_TERMINAL} record for a new attempt of the same request.
@@ -48,9 +72,11 @@ public interface GrpcOperationLedger {
* request, is left untouched. A retry that proceeds without winning this leaves its commit with * request, is left untouched. A retry that proceeds without winning this leaves its commit with
* no owner and runs concurrently with every other retry of the same key. * no owner and runs concurrently with every other retry of the same key.
* *
* @return true when this call now owns the record as a fresh attempt claimed at {@code now} * @return the new attempt's token when this call now owns the record as a fresh attempt claimed
* at {@code now}; empty otherwise
*/ */
boolean reclaimFailed(GrpcOperationIdentity identity, String requestFingerprint, Instant now); Optional<GrpcClaimToken> reclaimFailed(
GrpcOperationIdentity identity, String requestFingerprint, Instant now);
/** /**
* Takes over an {@code IN_PROGRESS} claim whose owner is presumed dead, for a new attempt of the * Takes over an {@code IN_PROGRESS} claim whose owner is presumed dead, for a new attempt of the
@@ -73,10 +99,15 @@ public interface GrpcOperationLedger {
* When an adopter commits them separately, the abandoned attempt's outcome is unknown and a * When an adopter commits them separately, the abandoned attempt's outcome is unknown and a
* takeover can run the mutation twice. * takeover can run the mutation twice.
* *
* @return true when this call now owns the record as a fresh attempt claimed at {@code now} * <p>The owner taken over may be slow rather than dead. Its later commit is refused and its later
* failure ignored, because it no longer holds the claim token; that is what keeps the takeover
* from becoming a second concurrent run.
*
* @return the new attempt's token when this call now owns the record as a fresh attempt claimed
* at {@code now}; empty otherwise
* @throws IllegalArgumentException when {@code claimedBefore} is after {@code now} * @throws IllegalArgumentException when {@code claimedBefore} is after {@code now}
*/ */
boolean reclaimStale( Optional<GrpcClaimToken> reclaimStale(
GrpcOperationIdentity identity, GrpcOperationIdentity identity,
String requestFingerprint, String requestFingerprint,
Instant claimedBefore, Instant claimedBefore,
@@ -10,6 +10,9 @@ import java.util.Optional;
* A caller that reuses a key for a different request has made a mistake, and returning the first * A caller that reuses a key for a different request has made a mistake, and returning the first
* request's answer would hide it; comparing fingerprints turns that into a {@code * request's answer would hide it; comparing fingerprints turns that into a {@code
* FAILED_PRECONDITION} the caller can act on. * FAILED_PRECONDITION} the caller can act on.
*
* <p>{@code claimToken} names the attempt that holds (or last held) the claim; see {@link
* GrpcClaimToken}.
*/ */
public record GrpcOperationLedgerRecord( public record GrpcOperationLedgerRecord(
GrpcOperationIdentity identity, GrpcOperationIdentity identity,
@@ -17,9 +20,10 @@ public record GrpcOperationLedgerRecord(
String requestFingerprint, String requestFingerprint,
Optional<String> outcomeReference, Optional<String> outcomeReference,
Instant claimedAt, Instant claimedAt,
Optional<Instant> completedAt) { Optional<Instant> completedAt,
GrpcClaimToken claimToken) {
/** Requires an identity, a state, a fingerprint and a claim moment. */ /** Requires an identity, a state, a fingerprint, a claim moment and a claim token. */
public GrpcOperationLedgerRecord { public GrpcOperationLedgerRecord {
if (identity == null || state == null) { if (identity == null || state == null) {
throw new IllegalArgumentException("a ledger record needs an identity and a state"); throw new IllegalArgumentException("a ledger record needs an identity and a state");
@@ -30,6 +34,9 @@ public record GrpcOperationLedgerRecord(
if (claimedAt == null || outcomeReference == null || completedAt == null) { if (claimedAt == null || outcomeReference == null || completedAt == null) {
throw new IllegalArgumentException("a ledger record needs its timestamps and Optionals"); throw new IllegalArgumentException("a ledger record needs its timestamps and Optionals");
} }
if (claimToken == null) {
throw new IllegalArgumentException("a ledger record names the attempt that holds it");
}
if (state.terminal() && completedAt.isEmpty()) { if (state.terminal() && completedAt.isEmpty()) {
throw new IllegalArgumentException("a terminal ledger record records when it finished"); throw new IllegalArgumentException("a terminal ledger record records when it finished");
} }
@@ -40,16 +47,26 @@ public record GrpcOperationLedgerRecord(
} }
} }
/** A fresh claim. */ /** A fresh claim, held by the first attempt. */
public static GrpcOperationLedgerRecord claim( public static GrpcOperationLedgerRecord claim(
GrpcOperationIdentity identity, String requestFingerprint, Instant claimedAt) { GrpcOperationIdentity identity, String requestFingerprint, Instant claimedAt) {
return claim(identity, requestFingerprint, claimedAt, GrpcClaimToken.FIRST);
}
/** A claim in progress, held by the attempt {@code claimToken} names. */
public static GrpcOperationLedgerRecord claim(
GrpcOperationIdentity identity,
String requestFingerprint,
Instant claimedAt,
GrpcClaimToken claimToken) {
return new GrpcOperationLedgerRecord( return new GrpcOperationLedgerRecord(
identity, identity,
GrpcOperationLedgerState.IN_PROGRESS, GrpcOperationLedgerState.IN_PROGRESS,
requestFingerprint, requestFingerprint,
Optional.empty(), Optional.empty(),
claimedAt, claimedAt,
Optional.empty()); Optional.empty(),
claimToken);
} }
/** Whether {@code candidateFingerprint} is the same request this claim was made for. */ /** Whether {@code candidateFingerprint} is the same request this claim was made for. */
@@ -88,7 +88,7 @@ public abstract class GrpcOperationLedgerContract {
ledger().claim(ID, FINGERPRINT, NOW); ledger().claim(ID, FINGERPRINT, NOW);
// when // when
ledger().markCommitted(ID, OUTCOME, NOW.plusSeconds(1)); ledger().markCommitted(ID, GrpcClaimToken.FIRST, OUTCOME, NOW.plusSeconds(1));
// then // then
assertThat(ledger().find(ID)) assertThat(ledger().find(ID))
@@ -106,7 +106,8 @@ public abstract class GrpcOperationLedgerContract {
@Test @Test
void committingWithoutAClaimIsRefused() { void committingWithoutAClaimIsRefused() {
// when // when
Throwable refused = catchThrowable(() -> ledger().markCommitted(ID, OUTCOME, NOW)); Throwable refused =
catchThrowable(() -> ledger().markCommitted(ID, GrpcClaimToken.FIRST, OUTCOME, NOW));
// then // then
assertThat(refused) assertThat(refused)
@@ -118,12 +119,15 @@ public abstract class GrpcOperationLedgerContract {
void aCommittedClaimCannotBeCommittedAgain() { void aCommittedClaimCannotBeCommittedAgain() {
// given // given
ledger().claim(ID, FINGERPRINT, NOW); ledger().claim(ID, FINGERPRINT, NOW);
ledger().markCommitted(ID, OUTCOME, NOW.plusSeconds(1)); ledger().markCommitted(ID, GrpcClaimToken.FIRST, OUTCOME, NOW.plusSeconds(1));
// when // when
Throwable refused = Throwable refused =
catchThrowable( catchThrowable(
() -> ledger().markCommitted(ID, "outcome://document/99", NOW.plusSeconds(2))); () ->
ledger()
.markCommitted(
ID, GrpcClaimToken.FIRST, "outcome://document/99", NOW.plusSeconds(2)));
// then // then
assertThat(refused) assertThat(refused)
@@ -137,7 +141,7 @@ public abstract class GrpcOperationLedgerContract {
ledger().claim(ID, FINGERPRINT, NOW); ledger().claim(ID, FINGERPRINT, NOW);
// when // when
ledger().markFailed(ID, NOW.plusSeconds(1)); ledger().markFailed(ID, GrpcClaimToken.FIRST, NOW.plusSeconds(1));
// then // then
assertThat(ledger().find(ID)) assertThat(ledger().find(ID))
@@ -153,7 +157,8 @@ public abstract class GrpcOperationLedgerContract {
// when // when
Throwable refused = Throwable refused =
catchThrowable(() -> ledger().markCommitted(ID, OUTCOME, NOW.plusSeconds(2))); catchThrowable(
() -> ledger().markCommitted(ID, GrpcClaimToken.FIRST, OUTCOME, NOW.plusSeconds(2)));
// then // then
assertThat(refused) assertThat(refused)
@@ -167,10 +172,11 @@ public abstract class GrpcOperationLedgerContract {
failedClaim(); failedClaim();
// when // when
boolean reclaimed = ledger().reclaimFailed(ID, FINGERPRINT, NOW.plusSeconds(2)); Optional<GrpcClaimToken> reclaimed =
ledger().reclaimFailed(ID, FINGERPRINT, NOW.plusSeconds(2));
// then // then
assertThat(reclaimed).isTrue(); assertThat(reclaimed).isPresent();
} }
@Test @Test
@@ -196,10 +202,11 @@ public abstract class GrpcOperationLedgerContract {
void aReclaimedClaimCanBeCommitted() { void aReclaimedClaimCanBeCommitted() {
// given // given
failedClaim(); failedClaim();
ledger().reclaimFailed(ID, FINGERPRINT, NOW.plusSeconds(2)); GrpcClaimToken reclaimed =
ledger().reclaimFailed(ID, FINGERPRINT, NOW.plusSeconds(2)).orElseThrow();
// when // when
ledger().markCommitted(ID, OUTCOME, NOW.plusSeconds(3)); ledger().markCommitted(ID, reclaimed, OUTCOME, NOW.plusSeconds(3));
// then // then
assertThat(ledger().find(ID)) assertThat(ledger().find(ID))
@@ -212,27 +219,28 @@ public abstract class GrpcOperationLedgerContract {
void aFailedClaimIsNotReclaimedForADifferentRequest() { void aFailedClaimIsNotReclaimedForADifferentRequest() {
failedClaim(); failedClaim();
assertThat(ledger().reclaimFailed(ID, OTHER_FINGERPRINT, NOW.plusSeconds(2))).isFalse(); assertThat(ledger().reclaimFailed(ID, OTHER_FINGERPRINT, NOW.plusSeconds(2))).isEmpty();
} }
@Test @Test
void anInProgressClaimIsNotReclaimed() { void anInProgressClaimIsNotReclaimed() {
ledger().claim(ID, FINGERPRINT, NOW); ledger().claim(ID, FINGERPRINT, NOW);
assertThat(ledger().reclaimFailed(ID, FINGERPRINT, NOW.plusSeconds(1))).isFalse(); assertThat(ledger().reclaimFailed(ID, FINGERPRINT, NOW.plusSeconds(1))).isEmpty();
} }
@Test @Test
void aCommittedClaimIsNotReclaimed() { void aCommittedClaimIsNotReclaimed() {
// given // given
ledger().claim(ID, FINGERPRINT, NOW); ledger().claim(ID, FINGERPRINT, NOW);
ledger().markCommitted(ID, OUTCOME, NOW.plusSeconds(1)); ledger().markCommitted(ID, GrpcClaimToken.FIRST, OUTCOME, NOW.plusSeconds(1));
// when // when
boolean reclaimed = ledger().reclaimFailed(ID, FINGERPRINT, NOW.plusSeconds(2)); Optional<GrpcClaimToken> reclaimed =
ledger().reclaimFailed(ID, FINGERPRINT, NOW.plusSeconds(2));
// then // then
assertThat(reclaimed).isFalse(); assertThat(reclaimed).isEmpty();
} }
@Test @Test
@@ -242,7 +250,7 @@ public abstract class GrpcOperationLedgerContract {
// when // when
List<Boolean> reclaimed = List<Boolean> reclaimed =
concurrently(() -> ledger().reclaimFailed(ID, FINGERPRINT, NOW.plusSeconds(2))); concurrently(() -> ledger().reclaimFailed(ID, FINGERPRINT, NOW.plusSeconds(2)).isPresent());
// then // then
assertThat(reclaimed).containsExactlyInAnyOrder(true, false); assertThat(reclaimed).containsExactlyInAnyOrder(true, false);
@@ -255,10 +263,10 @@ public abstract class GrpcOperationLedgerContract {
ledger().claim(ID, FINGERPRINT, NOW); ledger().claim(ID, FINGERPRINT, NOW);
// when // when
boolean reclaimed = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER); Optional<GrpcClaimToken> reclaimed = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER);
// then // then
assertThat(reclaimed).isTrue(); assertThat(reclaimed).isPresent();
} }
@Test @Test
@@ -285,10 +293,10 @@ public abstract class GrpcOperationLedgerContract {
void aTakenOverStaleClaimCanBeCommitted() { void aTakenOverStaleClaimCanBeCommitted() {
// given // given
ledger().claim(ID, FINGERPRINT, NOW); ledger().claim(ID, FINGERPRINT, NOW);
ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER); GrpcClaimToken takenOver = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER).orElseThrow();
// when // when
ledger().markCommitted(ID, OUTCOME, LATER.plusSeconds(1)); ledger().markCommitted(ID, takenOver, OUTCOME, LATER.plusSeconds(1));
// then // then
assertThat(ledger().find(ID)) assertThat(ledger().find(ID))
@@ -304,10 +312,11 @@ public abstract class GrpcOperationLedgerContract {
ledger().claim(ID, FINGERPRINT, NOW); ledger().claim(ID, FINGERPRINT, NOW);
// when // when
boolean reclaimed = ledger().reclaimStale(ID, FINGERPRINT, NOW, NOW.plus(CLAIM_TTL)); Optional<GrpcClaimToken> reclaimed =
ledger().reclaimStale(ID, FINGERPRINT, NOW, NOW.plus(CLAIM_TTL));
// then // then
assertThat(reclaimed).isFalse(); assertThat(reclaimed).isEmpty();
} }
@Test @Test
@@ -317,10 +326,11 @@ public abstract class GrpcOperationLedgerContract {
ledger().claim(ID, FINGERPRINT, NOW); ledger().claim(ID, FINGERPRINT, NOW);
// when // when
boolean reclaimed = ledger().reclaimStale(ID, OTHER_FINGERPRINT, CUTOFF, LATER); Optional<GrpcClaimToken> reclaimed =
ledger().reclaimStale(ID, OTHER_FINGERPRINT, CUTOFF, LATER);
// then // then
assertThat(reclaimed).isFalse(); assertThat(reclaimed).isEmpty();
} }
@Test @Test
@@ -328,13 +338,13 @@ public abstract class GrpcOperationLedgerContract {
void aCommittedClaimIsNotTakenOverAsStale() { void aCommittedClaimIsNotTakenOverAsStale() {
// given // given
ledger().claim(ID, FINGERPRINT, NOW); ledger().claim(ID, FINGERPRINT, NOW);
ledger().markCommitted(ID, OUTCOME, NOW.plusSeconds(1)); ledger().markCommitted(ID, GrpcClaimToken.FIRST, OUTCOME, NOW.plusSeconds(1));
// when // when
boolean reclaimed = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER); Optional<GrpcClaimToken> reclaimed = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER);
// then // then
assertThat(reclaimed).isFalse(); assertThat(reclaimed).isEmpty();
} }
@Test @Test
@@ -344,20 +354,20 @@ public abstract class GrpcOperationLedgerContract {
failedClaim(); failedClaim();
// when // when
boolean reclaimed = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER); Optional<GrpcClaimToken> reclaimed = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER);
// then // then
assertThat(reclaimed).isFalse(); assertThat(reclaimed).isEmpty();
} }
@Test @Test
@DisplayName("claim이 없으면 stale 회수는 아무것도 만들지 않는다") @DisplayName("claim이 없으면 stale 회수는 아무것도 만들지 않는다")
void takingOverAMissingClaimCreatesNothing() { void takingOverAMissingClaimCreatesNothing() {
// when // when
boolean reclaimed = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER); Optional<GrpcClaimToken> reclaimed = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER);
// then // then
assertThat(reclaimed).isFalse(); assertThat(reclaimed).isEmpty();
assertThat(ledger().find(ID)).isEmpty(); assertThat(ledger().find(ID)).isEmpty();
} }
@@ -369,10 +379,10 @@ public abstract class GrpcOperationLedgerContract {
ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER); ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER);
// when // when
boolean again = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER); Optional<GrpcClaimToken> again = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER);
// then // then
assertThat(again).isFalse(); assertThat(again).isEmpty();
} }
@Test @Test
@@ -383,7 +393,7 @@ public abstract class GrpcOperationLedgerContract {
// when // when
List<Boolean> reclaimed = List<Boolean> reclaimed =
concurrently(() -> ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER)); concurrently(() -> ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER).isPresent());
// then // then
assertThat(reclaimed).containsExactlyInAnyOrder(true, false); assertThat(reclaimed).containsExactlyInAnyOrder(true, false);
@@ -402,6 +412,126 @@ public abstract class GrpcOperationLedgerContract {
assertThat(refused).isInstanceOf(IllegalArgumentException.class).hasMessageContaining("cutoff"); assertThat(refused).isInstanceOf(IllegalArgumentException.class).hasMessageContaining("cutoff");
} }
@Test
@DisplayName("claim 을 넘겨받을 때마다 claim token 은 다음 시도 번호가 된다")
void eachHandOverIssuesTheNextClaimToken() {
// given
ledger().claim(ID, FINGERPRINT, NOW);
// when
GrpcClaimToken takenOver = ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER).orElseThrow();
// then
assertThat(takenOver).isEqualTo(GrpcClaimToken.FIRST.next());
assertThat(ledger().find(ID))
.get()
.extracting(GrpcOperationLedgerRecord::claimToken)
.isEqualTo(takenOver);
}
/**
* The re-review probe's sequence (2026-11-03, RR-02-04): A claims and is slow, B takes the claim
* over after the TTL, then A reports a failure. Recording it would mark B's running attempt
* failed, and the next retry would re-claim it and run beside B.
*/
@Test
@DisplayName("TTL 뒤 넘겨받은 claim 에 옛 owner 가 실패를 기록하면 무시되고 새 시도는 진행 중으로 남는다")
void aFailureReportedByTheOwnerThatLostTheClaimIsIgnored() {
// given
ledger().claim(ID, FINGERPRINT, NOW);
ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER);
// when
boolean recorded = ledger().markFailed(ID, GrpcClaimToken.FIRST, LATER.plusSeconds(1));
// then
assertThat(recorded).isFalse();
assertThat(ledger().find(ID))
.get()
.extracting(
GrpcOperationLedgerRecord::state,
GrpcOperationLedgerRecord::claimedAt,
GrpcOperationLedgerRecord::completedAt)
.containsExactly(GrpcOperationLedgerState.IN_PROGRESS, LATER, Optional.empty());
}
@Test
@DisplayName("옛 owner 의 실패가 무시되므로 다음 재시도는 진행 중인 새 시도를 다시 claim 하지 못한다")
void aRetryCannotReclaimTheRunningAttemptAfterTheLostOwnerFailed() {
// given
ledger().claim(ID, FINGERPRINT, NOW);
ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER);
ledger().markFailed(ID, GrpcClaimToken.FIRST, LATER.plusSeconds(1));
// when
Optional<GrpcClaimToken> third = ledger().reclaimFailed(ID, FINGERPRINT, LATER.plusSeconds(2));
// then
assertThat(third).isEmpty();
}
@Test
@DisplayName("TTL 뒤 넘겨받은 claim 에 옛 owner 가 commit 하면 거부되고 새 시도는 진행 중으로 남는다")
void aCommitByTheOwnerThatLostTheClaimIsRefused() {
// given
ledger().claim(ID, FINGERPRINT, NOW);
ledger().reclaimStale(ID, FINGERPRINT, CUTOFF, LATER);
// when
Throwable refused =
catchThrowable(
() -> ledger().markCommitted(ID, GrpcClaimToken.FIRST, OUTCOME, LATER.plusSeconds(1)));
// then
assertThat(refused)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("no longer held by attempt 1");
assertThat(ledger().find(ID))
.get()
.extracting(GrpcOperationLedgerRecord::state)
.isEqualTo(GrpcOperationLedgerState.IN_PROGRESS);
}
@Test
@DisplayName("실패 뒤 다시 claim 된 기록에 이전 시도가 commit 하면 거부된다")
void aCommitByTheAttemptBeforeAFailedReclaimIsRefused() {
// given
failedClaim();
ledger().reclaimFailed(ID, FINGERPRINT, NOW.plusSeconds(2));
// when
Throwable refused =
catchThrowable(
() -> ledger().markCommitted(ID, GrpcClaimToken.FIRST, OUTCOME, NOW.plusSeconds(3)));
// then
assertThat(refused)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("no longer held by attempt 1");
}
@Test
@DisplayName("한 claim 을 동시에 commit 하면 하나만 성공한다")
void concurrentCommitsOfOneClaimHaveExactlyOneWinner() throws Exception {
// given
ledger().claim(ID, FINGERPRINT, NOW);
// when
List<Boolean> committed =
concurrently(
() -> {
try {
ledger().markCommitted(ID, GrpcClaimToken.FIRST, OUTCOME, NOW.plusSeconds(1));
return true;
} catch (IllegalStateException refused) {
return false;
}
});
// then
assertThat(committed).containsExactlyInAnyOrder(true, false);
}
@Test @Test
@DisplayName("보존 기간이 지난 COMMITTED·FAILED_TERMINAL 행은 정리된다") @DisplayName("보존 기간이 지난 COMMITTED·FAILED_TERMINAL 행은 정리된다")
void finishedRecordsCompletedBeforeTheCutoffArePurged() { void finishedRecordsCompletedBeforeTheCutoffArePurged() {
@@ -489,12 +619,12 @@ public abstract class GrpcOperationLedgerContract {
// cutoff is compared with. // cutoff is compared with.
private void failedClaim(GrpcOperationIdentity identity, Instant claimedAt) { private void failedClaim(GrpcOperationIdentity identity, Instant claimedAt) {
ledger().claim(identity, FINGERPRINT, claimedAt); ledger().claim(identity, FINGERPRINT, claimedAt);
ledger().markFailed(identity, claimedAt.plusSeconds(1)); ledger().markFailed(identity, GrpcClaimToken.FIRST, claimedAt.plusSeconds(1));
} }
private void committedClaim(GrpcOperationIdentity identity, Instant claimedAt) { private void committedClaim(GrpcOperationIdentity identity, Instant claimedAt) {
ledger().claim(identity, FINGERPRINT, claimedAt); ledger().claim(identity, FINGERPRINT, claimedAt);
ledger().markCommitted(identity, OUTCOME, claimedAt.plusSeconds(1)); ledger().markCommitted(identity, GrpcClaimToken.FIRST, OUTCOME, claimedAt.plusSeconds(1));
} }
private static GrpcOperationIdentity identityWithKeyHash(char hexDigit) { private static GrpcOperationIdentity identityWithKeyHash(char hexDigit) {
@@ -6,15 +6,18 @@ import java.util.List;
import java.util.Optional; import java.util.Optional;
import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap; import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.atomic.AtomicBoolean;
/** /**
* A hand-rolled ledger for tests of code that depends on {@link GrpcOperationLedger}. * A hand-rolled ledger for tests of code that depends on {@link GrpcOperationLedger}.
* *
* <p>{@code putIfAbsent} on a concurrent map is the in-memory equivalent of the unique constraint * <p>{@code putIfAbsent} on a concurrent map is the in-memory equivalent of the unique constraint
* the real adapter relies on, and {@code replace(key, expected, next)} of its conditional update, * the real adapter relies on, and {@code replace(key, expected, next)} and {@code computeIfPresent}
* so claim, re-claim, stale takeover and purge are atomic here for the same reason they are there. * of its conditional updates, so claim, re-claim, stale takeover, commit, failure and purge are
* The transitions match the JPA adapter's because {@link GrpcOperationLedgerContract} runs against * atomic here for the same reason they are there. A commit or failure is recorded only for the
* both; a fake that allowed more would let the policy tests pass for code the database refuses. * attempt whose {@link GrpcClaimToken} still holds the claim. The transitions match the JPA
* adapter's because {@link GrpcOperationLedgerContract} runs against both; a fake that allowed more
* would let the policy tests pass for code the database refuses.
*/ */
public final class InMemoryOperationLedger implements GrpcOperationLedger { public final class InMemoryOperationLedger implements GrpcOperationLedger {
@@ -46,16 +49,19 @@ public final class InMemoryOperationLedger implements GrpcOperationLedger {
} }
@Override @Override
public void markCommitted(GrpcOperationIdentity identity, String outcomeReference, Instant now) { public void markCommitted(
GrpcOperationIdentity identity,
GrpcClaimToken claimToken,
String outcomeReference,
Instant now) {
GrpcOperationLedgerRecord committed = GrpcOperationLedgerRecord committed =
records.computeIfPresent( records.computeIfPresent(
identity.storageKey(), identity.storageKey(),
(key, existing) -> (key, existing) -> {
finish( requireHeldBy(existing, claimToken);
existing, return finish(
GrpcOperationLedgerState.COMMITTED, existing, GrpcOperationLedgerState.COMMITTED, Optional.of(outcomeReference), now);
Optional.of(outcomeReference), });
now));
if (committed == null) { if (committed == null) {
throw new IllegalStateException( throw new IllegalStateException(
"cannot commit operation '" + identity.storageKey() + "': no claim exists"); "cannot commit operation '" + identity.storageKey() + "': no claim exists");
@@ -63,28 +69,36 @@ public final class InMemoryOperationLedger implements GrpcOperationLedger {
} }
@Override @Override
public void markFailed(GrpcOperationIdentity identity, Instant now) { public boolean markFailed(
GrpcOperationIdentity identity, GrpcClaimToken claimToken, Instant now) {
AtomicBoolean recorded = new AtomicBoolean();
records.computeIfPresent( records.computeIfPresent(
identity.storageKey(), identity.storageKey(),
(key, existing) -> (key, existing) -> {
finish(existing, GrpcOperationLedgerState.FAILED_TERMINAL, Optional.empty(), now)); if (existing.state() != GrpcOperationLedgerState.IN_PROGRESS
|| !existing.claimToken().equals(claimToken)) {
return existing;
}
recorded.set(true);
return finish(existing, GrpcOperationLedgerState.FAILED_TERMINAL, Optional.empty(), now);
});
return recorded.get();
} }
@Override @Override
public boolean reclaimFailed( public Optional<GrpcClaimToken> reclaimFailed(
GrpcOperationIdentity identity, String requestFingerprint, Instant now) { GrpcOperationIdentity identity, String requestFingerprint, Instant now) {
GrpcOperationLedgerRecord current = records.get(identity.storageKey()); GrpcOperationLedgerRecord current = records.get(identity.storageKey());
return current != null if (current == null
&& current.state() == GrpcOperationLedgerState.FAILED_TERMINAL || current.state() != GrpcOperationLedgerState.FAILED_TERMINAL
&& current.sameRequestAs(requestFingerprint) || !current.sameRequestAs(requestFingerprint)) {
&& records.replace( return Optional.empty();
identity.storageKey(), }
current, return handOver(identity, requestFingerprint, current, now);
GrpcOperationLedgerRecord.claim(identity, requestFingerprint, now));
} }
@Override @Override
public boolean reclaimStale( public Optional<GrpcClaimToken> reclaimStale(
GrpcOperationIdentity identity, GrpcOperationIdentity identity,
String requestFingerprint, String requestFingerprint,
Instant claimedBefore, Instant claimedBefore,
@@ -94,14 +108,28 @@ public final class InMemoryOperationLedger implements GrpcOperationLedger {
"a stale-claim cutoff after now would leave the new claim stale as well"); "a stale-claim cutoff after now would leave the new claim stale as well");
} }
GrpcOperationLedgerRecord current = records.get(identity.storageKey()); GrpcOperationLedgerRecord current = records.get(identity.storageKey());
return current != null if (current == null
&& current.state() == GrpcOperationLedgerState.IN_PROGRESS || current.state() != GrpcOperationLedgerState.IN_PROGRESS
&& current.claimedAt().isBefore(claimedBefore) || !current.claimedAt().isBefore(claimedBefore)
&& current.sameRequestAs(requestFingerprint) || !current.sameRequestAs(requestFingerprint)) {
&& records.replace( return Optional.empty();
}
return handOver(identity, requestFingerprint, current, now);
}
// The conditional replace is the compare-and-set: of two hand-overs of the same record, one wins.
private Optional<GrpcClaimToken> handOver(
GrpcOperationIdentity identity,
String requestFingerprint,
GrpcOperationLedgerRecord current,
Instant now) {
GrpcClaimToken next = current.claimToken().next();
return records.replace(
identity.storageKey(), identity.storageKey(),
current, current,
GrpcOperationLedgerRecord.claim(identity, requestFingerprint, now)); GrpcOperationLedgerRecord.claim(identity, requestFingerprint, now, next))
? Optional.of(next)
: Optional.empty();
} }
@Override @Override
@@ -132,6 +160,21 @@ public final class InMemoryOperationLedger implements GrpcOperationLedger {
records.remove(identity.storageKey()); records.remove(identity.storageKey());
} }
// An attempt that lost the claim must not commit beside the attempt that holds it; checked
// before the state, so a lost owner hears that it lost rather than that the row finished.
private static void requireHeldBy(GrpcOperationLedgerRecord existing, GrpcClaimToken claimToken) {
if (!existing.claimToken().equals(claimToken)) {
throw new IllegalStateException(
"operation '"
+ existing.identity().storageKey()
+ "' is no longer held by attempt "
+ claimToken.attempt()
+ "; attempt "
+ existing.claimToken().attempt()
+ " holds it");
}
}
// Only an IN_PROGRESS claim finishes; a terminal row is re-opened by reclaimFailed alone. // Only an IN_PROGRESS claim finishes; a terminal row is re-opened by reclaimFailed alone.
private static GrpcOperationLedgerRecord finish( private static GrpcOperationLedgerRecord finish(
GrpcOperationLedgerRecord existing, GrpcOperationLedgerRecord existing,
@@ -152,6 +195,7 @@ public final class InMemoryOperationLedger implements GrpcOperationLedger {
existing.requestFingerprint(), existing.requestFingerprint(),
outcomeReference, outcomeReference,
existing.claimedAt(), existing.claimedAt(),
Optional.of(now)); Optional.of(now),
existing.claimToken());
} }
} }
@@ -16,7 +16,8 @@ final class GrpcOperationLedgerMigrations {
private static final List<String> IN_ORDER = private static final List<String> IN_ORDER =
List.of( List.of(
"db/migration/grpc/V001__create_grpc_operation_ledger.sql", "db/migration/grpc/V001__create_grpc_operation_ledger.sql",
"db/migration/grpc/V002__index_grpc_operation_ledger_finished_rows.sql"); "db/migration/grpc/V002__index_grpc_operation_ledger_finished_rows.sql",
"db/migration/grpc/V003__add_grpc_operation_ledger_claim_attempt.sql");
private GrpcOperationLedgerMigrations() {} private GrpcOperationLedgerMigrations() {}
@@ -2,6 +2,7 @@ package dev.caskeleton.grpc.ledger;
import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThat;
import dev.caskeleton.grpc.api.ledger.GrpcClaimToken;
import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity; import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity;
import dev.caskeleton.grpc.api.ledger.GrpcOperationLedger; import dev.caskeleton.grpc.api.ledger.GrpcOperationLedger;
import dev.caskeleton.grpc.api.ledger.GrpcOperationLedgerContract; import dev.caskeleton.grpc.api.ledger.GrpcOperationLedgerContract;
@@ -102,14 +103,15 @@ class JpaGrpcOperationLedgerContractIT extends GrpcOperationLedgerContract {
String fingerprint = "sha256:0f1e2d3c4b5a69788796a5b4c3d2e1f00f1e2d3c4b5a69788796a5b4c3d2e1f0"; String fingerprint = "sha256:0f1e2d3c4b5a69788796a5b4c3d2e1f00f1e2d3c4b5a69788796a5b4c3d2e1f0";
Instant now = Instant.parse("2026-09-25T10:00:00Z"); Instant now = Instant.parse("2026-09-25T10:00:00Z");
ledger.claim(id, fingerprint, now); ledger.claim(id, fingerprint, now);
ledger.markFailed(id, now.plusSeconds(1)); ledger.markFailed(id, GrpcClaimToken.FIRST, now.plusSeconds(1));
// when // when
transactions.executeWithoutResult( transactions.executeWithoutResult(
status -> { status -> {
adapter.claim(id, fingerprint, now.plusSeconds(2)); adapter.claim(id, fingerprint, now.plusSeconds(2));
adapter.reclaimFailed(id, fingerprint, now.plusSeconds(2)); GrpcClaimToken reclaimed =
adapter.markCommitted(id, "outcome://document/42", now.plusSeconds(3)); adapter.reclaimFailed(id, fingerprint, now.plusSeconds(2)).orElseThrow();
adapter.markCommitted(id, reclaimed, "outcome://document/42", now.plusSeconds(3));
}); });
// then // then
@@ -142,8 +144,11 @@ class JpaGrpcOperationLedgerContractIT extends GrpcOperationLedgerContract {
transactions.executeWithoutResult( transactions.executeWithoutResult(
status -> { status -> {
adapter.claim(id, fingerprint, retryAt); adapter.claim(id, fingerprint, retryAt);
adapter.reclaimStale(id, fingerprint, retryAt.minusSeconds(300), retryAt); GrpcClaimToken takenOver =
adapter.markCommitted(id, "outcome://document/42", retryAt.plusSeconds(1)); adapter
.reclaimStale(id, fingerprint, retryAt.minusSeconds(300), retryAt)
.orElseThrow();
adapter.markCommitted(id, takenOver, "outcome://document/42", retryAt.plusSeconds(1));
}); });
// then // then
@@ -188,33 +193,36 @@ class JpaGrpcOperationLedgerContractIT extends GrpcOperationLedgerContract {
@Override @Override
public void markCommitted( public void markCommitted(
GrpcOperationIdentity identity, String outcomeReference, Instant now) { GrpcOperationIdentity identity,
GrpcClaimToken claimToken,
String outcomeReference,
Instant now) {
transactions.executeWithoutResult( transactions.executeWithoutResult(
status -> delegate.markCommitted(identity, outcomeReference, now)); status -> delegate.markCommitted(identity, claimToken, outcomeReference, now));
} }
@Override @Override
public void markFailed(GrpcOperationIdentity identity, Instant now) { public boolean markFailed(
transactions.executeWithoutResult(status -> delegate.markFailed(identity, now)); GrpcOperationIdentity identity, GrpcClaimToken claimToken, Instant now) {
}
@Override
public boolean reclaimFailed(
GrpcOperationIdentity identity, String requestFingerprint, Instant now) {
return Boolean.TRUE.equals( return Boolean.TRUE.equals(
transactions.execute( transactions.execute(status -> delegate.markFailed(identity, claimToken, now)));
status -> delegate.reclaimFailed(identity, requestFingerprint, now)));
} }
@Override @Override
public boolean reclaimStale( public Optional<GrpcClaimToken> reclaimFailed(
GrpcOperationIdentity identity, String requestFingerprint, Instant now) {
return transactions.execute(
status -> delegate.reclaimFailed(identity, requestFingerprint, now));
}
@Override
public Optional<GrpcClaimToken> reclaimStale(
GrpcOperationIdentity identity, GrpcOperationIdentity identity,
String requestFingerprint, String requestFingerprint,
Instant claimedBefore, Instant claimedBefore,
Instant now) { Instant now) {
return Boolean.TRUE.equals( return transactions.execute(
transactions.execute( status -> delegate.reclaimStale(identity, requestFingerprint, claimedBefore, now));
status -> delegate.reclaimStale(identity, requestFingerprint, claimedBefore, now)));
} }
@Override @Override
@@ -1,5 +1,6 @@
package dev.caskeleton.grpc.ledger; package dev.caskeleton.grpc.ledger;
import dev.caskeleton.grpc.api.ledger.GrpcClaimToken;
import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity; import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity;
import dev.caskeleton.grpc.api.ledger.GrpcOperationLedgerRecord; import dev.caskeleton.grpc.api.ledger.GrpcOperationLedgerRecord;
import dev.caskeleton.grpc.api.ledger.GrpcOperationLedgerState; import dev.caskeleton.grpc.api.ledger.GrpcOperationLedgerState;
@@ -56,6 +57,9 @@ public class GrpcOperationLedgerEntity {
@Column(name = "completed_at") @Column(name = "completed_at")
private Instant completedAt; private Instant completedAt;
@Column(name = "claim_attempt", nullable = false)
private long claimAttempt;
/** JPA requires a no-argument constructor. */ /** JPA requires a no-argument constructor. */
protected GrpcOperationLedgerEntity() { protected GrpcOperationLedgerEntity() {
// for JPA // for JPA
@@ -70,6 +74,7 @@ public class GrpcOperationLedgerEntity {
this.requestFingerprint = requestFingerprint; this.requestFingerprint = requestFingerprint;
this.state = GrpcOperationLedgerState.IN_PROGRESS; this.state = GrpcOperationLedgerState.IN_PROGRESS;
this.claimedAt = claimedAt; this.claimedAt = claimedAt;
this.claimAttempt = GrpcClaimToken.FIRST.attempt();
} }
/** A new claim, in progress. */ /** A new claim, in progress. */
@@ -114,13 +119,26 @@ public class GrpcOperationLedgerEntity {
return Optional.ofNullable(completedAt); return Optional.ofNullable(completedAt);
} }
/** The attempt that holds, or last held, the claim. */
public GrpcClaimToken getClaimToken() {
return new GrpcClaimToken(claimAttempt);
}
/** /**
* Records a commit with its replayable outcome. * Records a commit with its replayable outcome, on a row this code already holds.
* *
* @throws IllegalStateException when the claim already finished; a second terminal transition * <p>The row-level statement of the rule. {@link JpaGrpcOperationLedger} applies the same rule as
* would overwrite the outcome a retry is entitled to receive * one conditional {@code UPDATE} ({@link GrpcOperationLedgerRepository#tryMarkCommitted}), so
* that the database, not a value read earlier, decides between concurrent transitions.
*
* @param claimToken the attempt committing
* @throws IllegalStateException when another attempt holds the claim, or the claim already
* finished; a second terminal transition would overwrite the outcome a retry is entitled to
* receive
*/ */
public void markCommitted(String outcomeReference, Instant completedAt) { public void markCommitted(
GrpcClaimToken claimToken, String outcomeReference, Instant completedAt) {
requireHeldBy(claimToken);
requireInProgress(); requireInProgress();
if (outcomeReference == null || outcomeReference.isBlank()) { if (outcomeReference == null || outcomeReference.isBlank()) {
throw new IllegalArgumentException( throw new IllegalArgumentException(
@@ -131,8 +149,16 @@ public class GrpcOperationLedgerEntity {
this.completedAt = completedAt; this.completedAt = completedAt;
} }
/** Records a terminal failure. */ /**
public void markFailed(Instant completedAt) { * Records a terminal failure, on a row this code already holds; the row-level statement of {@link
* GrpcOperationLedgerRepository#tryMarkFailed}.
*
* @param claimToken the attempt that failed
* @throws IllegalStateException when another attempt holds the claim, or the claim already
* finished
*/
public void markFailed(GrpcClaimToken claimToken, Instant completedAt) {
requireHeldBy(claimToken);
requireInProgress(); requireInProgress();
this.state = GrpcOperationLedgerState.FAILED_TERMINAL; this.state = GrpcOperationLedgerState.FAILED_TERMINAL;
this.completedAt = completedAt; this.completedAt = completedAt;
@@ -146,7 +172,21 @@ public class GrpcOperationLedgerEntity {
requestFingerprint, requestFingerprint,
Optional.ofNullable(outcomeReference), Optional.ofNullable(outcomeReference),
claimedAt, claimedAt,
Optional.ofNullable(completedAt)); Optional.ofNullable(completedAt),
getClaimToken());
}
private void requireHeldBy(GrpcClaimToken claimToken) {
if (claimAttempt != claimToken.attempt()) {
throw new IllegalStateException(
"operation '"
+ storageKey
+ "' is no longer held by attempt "
+ claimToken.attempt()
+ "; attempt "
+ claimAttempt
+ " holds it");
}
} }
private void requireInProgress() { private void requireInProgress() {
@@ -11,10 +11,12 @@ import org.springframework.transaction.annotation.Transactional;
/** /**
* The narrow persistence interface the ledger adapter needs. * The narrow persistence interface the ledger adapter needs.
* *
* <p>The claim path is deliberately not {@code save}. A claim entity already has a non-null string * <p>No path here is {@code save}. A claim entity already has a non-null string id, so Spring Data
* id, so Spring Data JPA treats it as detached and routes {@code save} through {@code merge}; merge * JPA treats it as detached and routes {@code save} through {@code merge}; merge can overwrite a
* can overwrite a committed row back to {@code IN_PROGRESS}. Claiming therefore uses one native * committed row back to {@code IN_PROGRESS}, and a load-check-save commit lets two attempts that
* insert with PostgreSQL conflict handling and decides ownership from the affected-row count. * both read {@code IN_PROGRESS} both write. Claiming is therefore one native insert with PostgreSQL
* conflict handling, and every transition is one conditional update; ownership and every race are
* decided by the affected-row count.
*/ */
public interface GrpcOperationLedgerRepository public interface GrpcOperationLedgerRepository
extends Repository<GrpcOperationLedgerEntity, String> { extends Repository<GrpcOperationLedgerEntity, String> {
@@ -31,7 +33,8 @@ public interface GrpcOperationLedgerRepository
state, state,
outcome_reference, outcome_reference,
claimed_at, claimed_at,
completed_at completed_at,
claim_attempt
) VALUES ( ) VALUES (
:storageKey, :storageKey,
:callerFingerprint, :callerFingerprint,
@@ -41,7 +44,8 @@ public interface GrpcOperationLedgerRepository
'IN_PROGRESS', 'IN_PROGRESS',
NULL, NULL,
:claimedAt, :claimedAt,
NULL NULL,
1
) )
ON CONFLICT DO NOTHING ON CONFLICT DO NOTHING
"""; """;
@@ -71,7 +75,11 @@ public interface GrpcOperationLedgerRepository
* claim's read-on-conflict does) would otherwise still read {@code FAILED_TERMINAL} and refuse * claim's read-on-conflict does) would otherwise still read {@code FAILED_TERMINAL} and refuse
* the commit. * the commit.
* *
* @return one when this caller re-claimed the row, zero otherwise * <p>Conditional on the attempt the caller read as well, and the new attempt is that one plus
* one: the caller then knows its own fencing token without reading the row again.
*
* @return one when this caller re-claimed the row as attempt {@code readAttempt + 1}, zero
* otherwise
*/ */
@Modifying(flushAutomatically = true, clearAutomatically = true) @Modifying(flushAutomatically = true, clearAutomatically = true)
@Transactional @Transactional
@@ -81,15 +89,18 @@ public interface GrpcOperationLedgerRepository
UPDATE grpc_operation_ledger UPDATE grpc_operation_ledger
SET state = 'IN_PROGRESS', SET state = 'IN_PROGRESS',
claimed_at = :claimedAt, claimed_at = :claimedAt,
completed_at = NULL completed_at = NULL,
claim_attempt = claim_attempt + 1
WHERE storage_key = :storageKey WHERE storage_key = :storageKey
AND state = 'FAILED_TERMINAL' AND state = 'FAILED_TERMINAL'
AND request_fingerprint = :requestFingerprint AND request_fingerprint = :requestFingerprint
AND claim_attempt = :readAttempt
""", """,
nativeQuery = true) nativeQuery = true)
int tryReclaimFailed( int tryReclaimFailed(
@Param("storageKey") String storageKey, @Param("storageKey") String storageKey,
@Param("requestFingerprint") String requestFingerprint, @Param("requestFingerprint") String requestFingerprint,
@Param("readAttempt") long readAttempt,
@Param("claimedAt") Instant claimedAt); @Param("claimedAt") Instant claimedAt);
/** /**
@@ -99,9 +110,11 @@ public interface GrpcOperationLedgerRepository
* <p>Same shape as {@link #tryReclaimFailed}: one conditional update decided by PostgreSQL's row * <p>Same shape as {@link #tryReclaimFailed}: one conditional update decided by PostgreSQL's row
* lock. A concurrent second takeover waits for the winner, re-evaluates the condition against the * lock. A concurrent second takeover waits for the winner, re-evaluates the condition against the
* winner's row — whose {@code claimed_at} is now {@code :claimedAt}, not before the cutoff — and * winner's row — whose {@code claimed_at} is now {@code :claimedAt}, not before the cutoff — and
* updates nothing. It clears the persistence context for the same reason. * updates nothing. It clears the persistence context for the same reason. The attempt grows by
* one, as in {@link #tryReclaimFailed}, and that is what fences the owner taken over.
* *
* @return one when this caller took the row over, zero otherwise * @return one when this caller took the row over as attempt {@code readAttempt + 1}, zero
* otherwise
*/ */
@Modifying(flushAutomatically = true, clearAutomatically = true) @Modifying(flushAutomatically = true, clearAutomatically = true)
@Transactional @Transactional
@@ -109,19 +122,76 @@ public interface GrpcOperationLedgerRepository
value = value =
""" """
UPDATE grpc_operation_ledger UPDATE grpc_operation_ledger
SET claimed_at = :claimedAt SET claimed_at = :claimedAt,
claim_attempt = claim_attempt + 1
WHERE storage_key = :storageKey WHERE storage_key = :storageKey
AND state = 'IN_PROGRESS' AND state = 'IN_PROGRESS'
AND request_fingerprint = :requestFingerprint AND request_fingerprint = :requestFingerprint
AND claimed_at < :claimedBefore AND claimed_at < :claimedBefore
AND claim_attempt = :readAttempt
""", """,
nativeQuery = true) nativeQuery = true)
int tryReclaimStale( int tryReclaimStale(
@Param("storageKey") String storageKey, @Param("storageKey") String storageKey,
@Param("requestFingerprint") String requestFingerprint, @Param("requestFingerprint") String requestFingerprint,
@Param("readAttempt") long readAttempt,
@Param("claimedBefore") Instant claimedBefore, @Param("claimedBefore") Instant claimedBefore,
@Param("claimedAt") Instant claimedAt); @Param("claimedAt") Instant claimedAt);
/**
* Commits the claim if attempt {@code claimAttempt} still holds it.
*
* <p>One conditional update rather than load-check-save: two commits of the same claim both read
* {@code IN_PROGRESS} under load-check-save and both wrote; here PostgreSQL's row lock makes the
* second re-evaluate the condition against the first's {@code COMMITTED} row and update nothing.
* An attempt whose claim was taken over or re-claimed matches nothing for the same reason.
*
* @return one when the claim was committed, zero otherwise
*/
@Modifying(flushAutomatically = true, clearAutomatically = true)
@Transactional
@Query(
value =
"""
UPDATE grpc_operation_ledger
SET state = 'COMMITTED',
outcome_reference = :outcomeReference,
completed_at = :completedAt
WHERE storage_key = :storageKey
AND state = 'IN_PROGRESS'
AND claim_attempt = :claimAttempt
""",
nativeQuery = true)
int tryMarkCommitted(
@Param("storageKey") String storageKey,
@Param("claimAttempt") long claimAttempt,
@Param("outcomeReference") String outcomeReference,
@Param("completedAt") Instant completedAt);
/**
* Marks the claim terminally failed if attempt {@code claimAttempt} still holds it; the same
* conditional shape as {@link #tryMarkCommitted}.
*
* @return one when the failure was recorded, zero otherwise
*/
@Modifying(flushAutomatically = true, clearAutomatically = true)
@Transactional
@Query(
value =
"""
UPDATE grpc_operation_ledger
SET state = 'FAILED_TERMINAL',
completed_at = :completedAt
WHERE storage_key = :storageKey
AND state = 'IN_PROGRESS'
AND claim_attempt = :claimAttempt
""",
nativeQuery = true)
int tryMarkFailed(
@Param("storageKey") String storageKey,
@Param("claimAttempt") long claimAttempt,
@Param("completedAt") Instant completedAt);
/** /**
* Deletes up to {@code limit} finished rows that completed before {@code completedBefore}, oldest * Deletes up to {@code limit} finished rows that completed before {@code completedBefore}, oldest
* first. Never an {@code IN_PROGRESS} row. * first. Never an {@code IN_PROGRESS} row.
@@ -155,9 +225,6 @@ public interface GrpcOperationLedgerRepository
int deleteFinishedBefore( int deleteFinishedBefore(
@Param("completedBefore") Instant completedBefore, @Param("limit") int limit); @Param("completedBefore") Instant completedBefore, @Param("limit") int limit);
/** Updates an existing claim after it has been loaded. */
GrpcOperationLedgerEntity save(GrpcOperationLedgerEntity entity);
/** Reads a claim by its derived storage key. */ /** Reads a claim by its derived storage key. */
Optional<GrpcOperationLedgerEntity> findById(String storageKey); Optional<GrpcOperationLedgerEntity> findById(String storageKey);
@@ -1,5 +1,6 @@
package dev.caskeleton.grpc.ledger; package dev.caskeleton.grpc.ledger;
import dev.caskeleton.grpc.api.ledger.GrpcClaimToken;
import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity; import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity;
import dev.caskeleton.grpc.api.ledger.GrpcOperationLedger; import dev.caskeleton.grpc.api.ledger.GrpcOperationLedger;
import dev.caskeleton.grpc.api.ledger.GrpcOperationLedgerRecord; import dev.caskeleton.grpc.api.ledger.GrpcOperationLedgerRecord;
@@ -16,6 +17,11 @@ import java.util.Optional;
* {@code save}: this entity has a non-null application-assigned id, so save routes through merge * {@code save}: this entity has a non-null application-assigned id, so save routes through merge
* and can overwrite an existing terminal row. * and can overwrite an existing terminal row.
* *
* <p>Every transition — re-claim, takeover, commit, failure — is one conditional {@code UPDATE}
* that names the claim attempt ({@link GrpcClaimToken}). An attempt whose claim was handed over
* matches nothing: its commit is refused and its failure ignored, which is what keeps a takeover of
* a slow owner from becoming two concurrent runs (re-review 2026-11-03, RR-02-04).
*
* <p>No transaction annotations here. The commit is meant to happen inside the caller's business * <p>No transaction annotations here. The commit is meant to happen inside the caller's business
* transaction, and a {@code REQUIRES_NEW} on this adapter would put the ledger in its own * transaction, and a {@code REQUIRES_NEW} on this adapter would put the ledger in its own
* transaction — reintroducing the window in which the mutation is durable and the claim is not. * transaction — reintroducing the window in which the mutation is durable and the claim is not.
@@ -36,6 +42,28 @@ public final class JpaGrpcOperationLedger implements GrpcOperationLedger {
public Optional<GrpcOperationLedgerRecord> claim( public Optional<GrpcOperationLedgerRecord> claim(
GrpcOperationIdentity identity, String requestFingerprint, Instant now) { GrpcOperationIdentity identity, String requestFingerprint, Instant now) {
requireIdentity(identity); requireIdentity(identity);
// Twice at most. Zero rows means another claim holds the key; an empty read after that means
// the row went in between (a purge of an expired record, or a release). Treating that as
// "ours" would run the mutation without a claim row to guard it, so the insert is tried once
// more and decides again.
for (int round = 0; round < 2; round++) {
if (insertClaim(identity, requestFingerprint, now)) {
return Optional.empty();
}
Optional<GrpcOperationLedgerEntity> existing = repository.findById(identity.storageKey());
if (existing.isPresent()) {
return Optional.of(existing.get().toRecord(identity));
}
}
throw new IllegalStateException(
"claim for '"
+ identity.storageKey()
+ "' conflicted twice, but there was no row to read either time; another owner keeps"
+ " releasing it. Retry the claim.");
}
private boolean insertClaim(
GrpcOperationIdentity identity, String requestFingerprint, Instant now) {
int inserted = int inserted =
repository.tryInsertClaim( repository.tryInsertClaim(
identity.storageKey(), identity.storageKey(),
@@ -44,25 +72,10 @@ public final class JpaGrpcOperationLedger implements GrpcOperationLedger {
identity.idempotencyKeyHash(), identity.idempotencyKeyHash(),
requestFingerprint, requestFingerprint,
now); now);
if (inserted == 1) { if (inserted != 0 && inserted != 1) {
return Optional.empty();
}
if (inserted != 0) {
throw new IllegalStateException("claim insert affected an unexpected row count: " + inserted); throw new IllegalStateException("claim insert affected an unexpected row count: " + inserted);
} }
// Zero rows means another claim holds the key. An empty read now means that owner released it return inserted == 1;
// in between; treating that as "ours" would run the mutation without a claim row to guard it.
return Optional.of(
repository
.findById(identity.storageKey())
.orElseThrow(
() ->
new IllegalStateException(
"claim for '"
+ identity.storageKey()
+ "' conflicted, but there was no row to read; the other owner released"
+ " it in between. Retry the claim."))
.toRecord(identity));
} }
@Override @Override
@@ -71,10 +84,30 @@ public final class JpaGrpcOperationLedger implements GrpcOperationLedger {
return repository.findById(identity.storageKey()).map(entity -> entity.toRecord(identity)); return repository.findById(identity.storageKey()).map(entity -> entity.toRecord(identity));
} }
/**
* Commits the claim if {@code claimToken} still holds it, as one conditional update.
*
* <p>When nothing matched, the row is read only to say why; the refusal itself was the
* database's, and the exception is what rolls back the caller's business transaction.
*/
@Override @Override
public void markCommitted(GrpcOperationIdentity identity, String outcomeReference, Instant now) { public void markCommitted(
GrpcOperationIdentity identity,
GrpcClaimToken claimToken,
String outcomeReference,
Instant now) {
requireIdentity(identity); requireIdentity(identity);
GrpcOperationLedgerEntity entity = requireToken(claimToken);
if (outcomeReference == null || outcomeReference.isBlank()) {
throw new IllegalArgumentException(
"a commit records the outcome to replay; without it the row cannot answer a retry");
}
if (repository.tryMarkCommitted(
identity.storageKey(), claimToken.attempt(), outcomeReference, now)
== 1) {
return;
}
GrpcOperationLedgerEntity current =
repository repository
.findById(identity.storageKey()) .findById(identity.storageKey())
.orElseThrow( .orElseThrow(
@@ -84,31 +117,54 @@ public final class JpaGrpcOperationLedger implements GrpcOperationLedger {
+ identity.storageKey() + identity.storageKey()
+ "': no claim exists. The claim is what a retry reads; committing " + "': no claim exists. The claim is what a retry reads; committing "
+ "without one leaves the mutation durable and unguarded.")); + "without one leaves the mutation durable and unguarded."));
entity.markCommitted(outcomeReference, now); if (!current.getClaimToken().equals(claimToken)) {
repository.save(entity); throw new IllegalStateException(
"operation '"
+ identity.storageKey()
+ "' is no longer held by attempt "
+ claimToken.attempt()
+ "; attempt "
+ current.getClaimToken().attempt()
+ " holds it, so this attempt's transaction must roll back");
}
throw new IllegalStateException(
"operation '"
+ identity.storageKey()
+ "' is already "
+ current.getState()
+ " and cannot transition again");
} }
@Override @Override
public void markFailed(GrpcOperationIdentity identity, Instant now) { public boolean markFailed(
GrpcOperationIdentity identity, GrpcClaimToken claimToken, Instant now) {
requireIdentity(identity); requireIdentity(identity);
repository requireToken(claimToken);
.findById(identity.storageKey()) return repository.tryMarkFailed(identity.storageKey(), claimToken.attempt(), now) == 1;
.ifPresent(
entity -> {
entity.markFailed(now);
repository.save(entity);
});
} }
/**
* Re-claims a failed record. Conditional on the attempt read just before, so the new token is
* known exactly: that attempt plus one. A record that changed in between matches nothing, and
* this caller has lost the race it would have lost anyway.
*/
@Override @Override
public boolean reclaimFailed( public Optional<GrpcClaimToken> reclaimFailed(
GrpcOperationIdentity identity, String requestFingerprint, Instant now) { GrpcOperationIdentity identity, String requestFingerprint, Instant now) {
requireIdentity(identity); requireIdentity(identity);
return repository.tryReclaimFailed(identity.storageKey(), requestFingerprint, now) == 1; Optional<GrpcClaimToken> read = currentToken(identity);
if (read.isEmpty()) {
return Optional.empty();
}
return repository.tryReclaimFailed(
identity.storageKey(), requestFingerprint, read.get().attempt(), now)
== 1
? Optional.of(read.get().next())
: Optional.empty();
} }
@Override @Override
public boolean reclaimStale( public Optional<GrpcClaimToken> reclaimStale(
GrpcOperationIdentity identity, GrpcOperationIdentity identity,
String requestFingerprint, String requestFingerprint,
Instant claimedBefore, Instant claimedBefore,
@@ -118,8 +174,19 @@ public final class JpaGrpcOperationLedger implements GrpcOperationLedger {
throw new IllegalArgumentException( throw new IllegalArgumentException(
"a stale-claim cutoff after now would leave the new claim stale as well"); "a stale-claim cutoff after now would leave the new claim stale as well");
} }
return repository.tryReclaimStale(identity.storageKey(), requestFingerprint, claimedBefore, now) Optional<GrpcClaimToken> read = currentToken(identity);
== 1; if (read.isEmpty()) {
return Optional.empty();
}
return repository.tryReclaimStale(
identity.storageKey(), requestFingerprint, read.get().attempt(), claimedBefore, now)
== 1
? Optional.of(read.get().next())
: Optional.empty();
}
private Optional<GrpcClaimToken> currentToken(GrpcOperationIdentity identity) {
return repository.findById(identity.storageKey()).map(GrpcOperationLedgerEntity::getClaimToken);
} }
@Override @Override
@@ -139,6 +206,12 @@ public final class JpaGrpcOperationLedger implements GrpcOperationLedger {
repository.deleteById(identity.storageKey()); repository.deleteById(identity.storageKey());
} }
private static void requireToken(GrpcClaimToken claimToken) {
if (claimToken == null) {
throw new IllegalArgumentException("a transition names the attempt that holds the claim");
}
}
private static void requireIdentity(GrpcOperationIdentity identity) { private static void requireIdentity(GrpcOperationIdentity identity) {
if (identity == null) { if (identity == null) {
throw new IllegalArgumentException("an operation identity is required"); throw new IllegalArgumentException("an operation identity is required");
@@ -0,0 +1,20 @@
-- Claim attempts: the fencing token of the operation ledger (re-review 2026-11-03, RR-02-04).
--
-- A claim changes hands when a stale one is taken over after the claim TTL or a failed one is
-- re-claimed. The owner that lost it may still be running, and when it finishes it records an
-- outcome. Without knowing which attempt holds the claim, that record landed on the new attempt's
-- claim: an old owner's failure turned a running attempt FAILED_TERMINAL, the next retry re-claimed
-- it, and two attempts ran the mutation at once.
--
-- claim_attempt is 1 for the first claim and grows by one with every hand-over. A commit or a
-- failure is one conditional UPDATE that names the attempt (WHERE state = 'IN_PROGRESS' AND
-- claim_attempt = :attempt), so a lost owner's record changes nothing and two commits of the same
-- attempt have one winner.
--
-- Existing rows are their first attempt: no row has been handed over under a token yet. The table is
-- small and the platform is not adopted anywhere, so a plain ALTER with a default is enough.
ALTER TABLE grpc_operation_ledger
ADD COLUMN claim_attempt BIGINT NOT NULL DEFAULT 1;
ALTER TABLE grpc_operation_ledger
ADD CONSTRAINT ck_grpc_operation_ledger_claim_attempt_positive CHECK (claim_attempt > 0);
@@ -4,6 +4,7 @@ import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy; import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.assertj.core.api.Assertions.catchThrowable; import static org.assertj.core.api.Assertions.catchThrowable;
import dev.caskeleton.grpc.api.ledger.GrpcClaimToken;
import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity; import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity;
import dev.caskeleton.grpc.api.ledger.GrpcOperationLedger; import dev.caskeleton.grpc.api.ledger.GrpcOperationLedger;
import dev.caskeleton.grpc.api.ledger.GrpcOperationLedgerRecord; import dev.caskeleton.grpc.api.ledger.GrpcOperationLedgerRecord;
@@ -34,12 +35,12 @@ class GrpcOperationLedgerRepositoryTest {
"sha256:aabbccddeeff00112233445566778899aabbccddeeff00112233445566778899"; "sha256:aabbccddeeff00112233445566778899aabbccddeeff00112233445566778899";
/** /**
* An in-memory stand-in for Spring Data's non-null-id save semantics. * An in-memory stand-in for the repository's native statements.
* *
* <p>A newly constructed ledger entity already has its string id, so a real {@code * <p>The insert is {@code putIfAbsent}, the stand-in for the unique constraint. The commit and
* SimpleJpaRepository.save} treats it as detached and merges it. The old fake used {@code * failure are the entity's own transition rules, which state the conditions the native {@code
* putIfAbsent} and therefore tested the database constraint we wished save() would hit rather * UPDATE}s carry (state {@code IN_PROGRESS} and the same claim attempt); a refused transition
* than the merge behavior it actually uses. * updates zero rows. The re-claims and the purge are certified against PostgreSQL only.
*/ */
private static class InMemoryRepository implements GrpcOperationLedgerRepository { private static class InMemoryRepository implements GrpcOperationLedgerRepository {
private final ConcurrentMap<String, GrpcOperationLedgerEntity> rows = new ConcurrentHashMap<>(); private final ConcurrentMap<String, GrpcOperationLedgerEntity> rows = new ConcurrentHashMap<>();
@@ -61,14 +62,19 @@ class GrpcOperationLedgerRepositoryTest {
} }
@Override @Override
public int tryReclaimFailed(String storageKey, String requestFingerprint, Instant claimedAt) { public int tryReclaimFailed(
String storageKey, String requestFingerprint, long readAttempt, Instant claimedAt) {
throw new UnsupportedOperationException( throw new UnsupportedOperationException(
"the re-claim is certified against PostgreSQL by JpaGrpcOperationLedgerContractIT"); "the re-claim is certified against PostgreSQL by JpaGrpcOperationLedgerContractIT");
} }
@Override @Override
public int tryReclaimStale( public int tryReclaimStale(
String storageKey, String requestFingerprint, Instant claimedBefore, Instant claimedAt) { String storageKey,
String requestFingerprint,
long readAttempt,
Instant claimedBefore,
Instant claimedAt) {
throw new UnsupportedOperationException( throw new UnsupportedOperationException(
"the stale takeover is certified against PostgreSQL by JpaGrpcOperationLedgerContractIT"); "the stale takeover is certified against PostgreSQL by JpaGrpcOperationLedgerContractIT");
} }
@@ -80,9 +86,32 @@ class GrpcOperationLedgerRepositoryTest {
} }
@Override @Override
public GrpcOperationLedgerEntity save(GrpcOperationLedgerEntity entity) { public int tryMarkCommitted(
rows.put(entity.getStorageKey(), entity); String storageKey, long claimAttempt, String outcomeReference, Instant completedAt) {
return entity; GrpcOperationLedgerEntity row = rows.get(storageKey);
if (row == null) {
return 0;
}
try {
row.markCommitted(new GrpcClaimToken(claimAttempt), outcomeReference, completedAt);
return 1;
} catch (IllegalStateException refused) {
return 0;
}
}
@Override
public int tryMarkFailed(String storageKey, long claimAttempt, Instant completedAt) {
GrpcOperationLedgerEntity row = rows.get(storageKey);
if (row == null) {
return 0;
}
try {
row.markFailed(new GrpcClaimToken(claimAttempt), completedAt);
return 1;
} catch (IllegalStateException refused) {
return 0;
}
} }
@Override @Override
@@ -139,6 +168,7 @@ class GrpcOperationLedgerRepositoryTest {
} }
@Test @Test
@DisplayName("충돌한 행이 다시 시도해도 계속 사라지면 소유로 보지 않고 거부한다")
void aConflictWhoseRowVanishedBeforeTheReadIsNotTakenAsOwnership() { void aConflictWhoseRowVanishedBeforeTheReadIsNotTakenAsOwnership() {
// given // given
InMemoryRepository conflictThenReleased = InMemoryRepository conflictThenReleased =
@@ -166,6 +196,54 @@ class GrpcOperationLedgerRepositoryTest {
.hasMessageContaining("no row to read"); .hasMessageContaining("no row to read");
} }
/**
* A purge (or release) can delete the conflicting row between the insert and the read. The read
* then finds nothing, and the claim tries the insert once more instead of failing the call
* (re-review 2026-11-03, RR-02-04).
*/
@Test
@DisplayName("충돌한 행이 읽기 전에 사라지면 claim 은 insert 를 한 번 더 시도해 소유한다")
void aConflictWhoseRowVanishedOnceIsClaimedByTheRetriedInsert() {
// given
InMemoryRepository conflictOnceThenFree =
new InMemoryRepository() {
private boolean conflicted;
@Override
public int tryInsertClaim(
String storageKey,
String callerFingerprint,
String fullMethodName,
String idempotencyKeyHash,
String requestFingerprint,
Instant claimedAt) {
if (!conflicted) {
conflicted = true;
return 0; // a row held the key and was purged before this read
}
return super.tryInsertClaim(
storageKey,
callerFingerprint,
fullMethodName,
idempotencyKeyHash,
requestFingerprint,
claimedAt);
}
};
GrpcOperationLedger racing = new JpaGrpcOperationLedger(conflictOnceThenFree);
// when
Optional<GrpcOperationLedgerRecord> existing =
racing.claim(identity("tenant-1.actor-1"), FINGERPRINT, NOW);
// then
assertThat(existing).isEmpty();
assertThat(racing.find(identity("tenant-1.actor-1")))
.get()
.extracting(GrpcOperationLedgerRecord::state, GrpcOperationLedgerRecord::claimToken)
.containsExactly(GrpcOperationLedgerState.IN_PROGRESS, GrpcClaimToken.FIRST);
}
@Test @Test
@DisplayName("one caller's key does not collide with another's") @DisplayName("one caller's key does not collide with another's")
void callersDoNotCollide() { void callersDoNotCollide() {
@@ -179,7 +257,7 @@ class GrpcOperationLedgerRepositoryTest {
GrpcOperationIdentity id = identity("tenant-1.actor-1"); GrpcOperationIdentity id = identity("tenant-1.actor-1");
ledger.claim(id, FINGERPRINT, NOW); ledger.claim(id, FINGERPRINT, NOW);
ledger.markCommitted(id, "outcome://document/42", NOW.plusSeconds(1)); ledger.markCommitted(id, GrpcClaimToken.FIRST, "outcome://document/42", NOW.plusSeconds(1));
GrpcOperationLedgerRecord record = ledger.find(id).orElseThrow(); GrpcOperationLedgerRecord record = ledger.find(id).orElseThrow();
assertThat(record.state()).isEqualTo(GrpcOperationLedgerState.COMMITTED); assertThat(record.state()).isEqualTo(GrpcOperationLedgerState.COMMITTED);
@@ -192,7 +270,7 @@ class GrpcOperationLedgerRepositoryTest {
void retryingACommittedClaimPreservesTheWinner() { void retryingACommittedClaimPreservesTheWinner() {
GrpcOperationIdentity id = identity("tenant-1.actor-1"); GrpcOperationIdentity id = identity("tenant-1.actor-1");
assertThat(ledger.claim(id, FINGERPRINT, NOW)).isEmpty(); assertThat(ledger.claim(id, FINGERPRINT, NOW)).isEmpty();
ledger.markCommitted(id, "outcome://document/42", NOW.plusSeconds(1)); ledger.markCommitted(id, GrpcClaimToken.FIRST, "outcome://document/42", NOW.plusSeconds(1));
GrpcOperationLedgerRecord retry = GrpcOperationLedgerRecord retry =
ledger.claim(id, OTHER_FINGERPRINT, NOW.plusSeconds(2)).orElseThrow(); ledger.claim(id, OTHER_FINGERPRINT, NOW.plusSeconds(2)).orElseThrow();
@@ -206,7 +284,12 @@ class GrpcOperationLedgerRepositoryTest {
@DisplayName("a commit without a claim is refused rather than silently inserted") @DisplayName("a commit without a claim is refused rather than silently inserted")
void committingWithoutAClaimIsRefused() { void committingWithoutAClaimIsRefused() {
assertThatThrownBy( assertThatThrownBy(
() -> ledger.markCommitted(identity("tenant-1.actor-1"), "outcome://document/42", NOW)) () ->
ledger.markCommitted(
identity("tenant-1.actor-1"),
GrpcClaimToken.FIRST,
"outcome://document/42",
NOW))
.isInstanceOf(IllegalStateException.class) .isInstanceOf(IllegalStateException.class)
.hasMessageContaining("durable and unguarded"); .hasMessageContaining("durable and unguarded");
} }
@@ -216,9 +299,12 @@ class GrpcOperationLedgerRepositoryTest {
void aTerminalRowDoesNotTransitionAgain() { void aTerminalRowDoesNotTransitionAgain() {
GrpcOperationIdentity id = identity("tenant-1.actor-1"); GrpcOperationIdentity id = identity("tenant-1.actor-1");
ledger.claim(id, FINGERPRINT, NOW); ledger.claim(id, FINGERPRINT, NOW);
ledger.markCommitted(id, "outcome://document/42", NOW.plusSeconds(1)); ledger.markCommitted(id, GrpcClaimToken.FIRST, "outcome://document/42", NOW.plusSeconds(1));
assertThatThrownBy(() -> ledger.markCommitted(id, "outcome://document/99", NOW.plusSeconds(2))) assertThatThrownBy(
() ->
ledger.markCommitted(
id, GrpcClaimToken.FIRST, "outcome://document/99", NOW.plusSeconds(2)))
.isInstanceOf(IllegalStateException.class) .isInstanceOf(IllegalStateException.class)
.hasMessageContaining("cannot transition again"); .hasMessageContaining("cannot transition again");
} }
@@ -240,7 +326,7 @@ class GrpcOperationLedgerRepositoryTest {
GrpcOperationIdentity id = identity("tenant-1.actor-1"); GrpcOperationIdentity id = identity("tenant-1.actor-1");
ledger.claim(id, FINGERPRINT, NOW); ledger.claim(id, FINGERPRINT, NOW);
ledger.markFailed(id, NOW.plusSeconds(1)); ledger.markFailed(id, GrpcClaimToken.FIRST, NOW.plusSeconds(1));
assertThat(ledger.find(id).orElseThrow().state()) assertThat(ledger.find(id).orElseThrow().state())
.isEqualTo(GrpcOperationLedgerState.FAILED_TERMINAL); .isEqualTo(GrpcOperationLedgerState.FAILED_TERMINAL);
@@ -259,7 +345,8 @@ class GrpcOperationLedgerRepositoryTest {
FINGERPRINT, FINGERPRINT,
Optional.empty(), Optional.empty(),
NOW, NOW,
Optional.of(NOW))) Optional.of(NOW),
GrpcClaimToken.FIRST))
.isInstanceOf(IllegalArgumentException.class) .isInstanceOf(IllegalArgumentException.class)
.hasMessageContaining("cannot be replayed"); .hasMessageContaining("cannot be replayed");
} }
@@ -1,5 +1,6 @@
package dev.caskeleton.grpc.idempotency; package dev.caskeleton.grpc.idempotency;
import dev.caskeleton.grpc.api.ledger.GrpcClaimToken;
import java.time.Duration; import java.time.Duration;
import java.util.Optional; import java.util.Optional;
@@ -10,9 +11,18 @@ import java.util.Optional;
* duplicate". A duplicate of a committed operation should be answered with the original result; a * duplicate". A duplicate of a committed operation should be answered with the original result; a
* duplicate of a running one is a race that has to wait or be refused; and a key reused for a * duplicate of a running one is a race that has to wait or be refused; and a key reused for a
* different request is a caller bug that must be reported rather than absorbed. * different request is a caller bug that must be reported rather than absorbed.
*
* <p>A {@code PROCEED} that won a claim carries the claim's {@link GrpcClaimToken}. The caller
* hands it back to {@code recordCommit} or {@code recordFailure}: the ledger records an outcome
* only for the attempt that still holds the claim, so a caller whose claim was taken over cannot
* write over the attempt that took it (re-review 2026-11-03, RR-02-04).
*/ */
public record GrpcIdempotencyDecision( public record GrpcIdempotencyDecision(
Outcome outcome, Optional<String> outcomeReference, Duration retryAfter, String reason) { Outcome outcome,
Optional<String> outcomeReference,
Duration retryAfter,
String reason,
Optional<GrpcClaimToken> claimToken) {
/** The five ways an idempotency check can end. */ /** The five ways an idempotency check can end. */
public enum Outcome { public enum Outcome {
@@ -48,11 +58,31 @@ public record GrpcIdempotencyDecision(
if (outcome == Outcome.WAIT_AND_POLL && retryAfter.isZero()) { if (outcome == Outcome.WAIT_AND_POLL && retryAfter.isZero()) {
throw new IllegalArgumentException("a WAIT_AND_POLL decision says how long to wait"); throw new IllegalArgumentException("a WAIT_AND_POLL decision says how long to wait");
} }
if (claimToken == null) {
throw new IllegalArgumentException("the claim token Optional must be present");
}
if (outcome != Outcome.PROCEED && claimToken.isPresent()) {
throw new IllegalArgumentException("only a PROCEED decision can hold a claim");
}
} }
/** Proceed with the operation. */ /** Proceed with an operation that is not claimed: the method takes no idempotency key. */
public static GrpcIdempotencyDecision proceed(String reason) { public static GrpcIdempotencyDecision proceed(String reason) {
return new GrpcIdempotencyDecision(Outcome.PROCEED, Optional.empty(), Duration.ZERO, reason); return new GrpcIdempotencyDecision(
Outcome.PROCEED, Optional.empty(), Duration.ZERO, reason, Optional.empty());
}
/**
* Proceed as the attempt that holds the claim.
*
* @param claimToken the token to hand back when recording the commit or the failure
*/
public static GrpcIdempotencyDecision proceedHolding(GrpcClaimToken claimToken, String reason) {
if (claimToken == null) {
throw new IllegalArgumentException("a claimed PROCEED carries its claim token");
}
return new GrpcIdempotencyDecision(
Outcome.PROCEED, Optional.empty(), Duration.ZERO, reason, Optional.of(claimToken));
} }
/** Replay a stored outcome. */ /** Replay a stored outcome. */
@@ -61,13 +91,18 @@ public record GrpcIdempotencyDecision(
Outcome.REPLAY, Outcome.REPLAY,
Optional.of(outcomeReference), Optional.of(outcomeReference),
Duration.ZERO, Duration.ZERO,
"an identical request already committed; returning the stored outcome"); "an identical request already committed; returning the stored outcome",
Optional.empty());
} }
/** Wait and poll for an in-progress claim. */ /** Wait and poll for an in-progress claim. */
public static GrpcIdempotencyDecision waitAndPoll(Duration retryAfter) { public static GrpcIdempotencyDecision waitAndPoll(Duration retryAfter) {
return new GrpcIdempotencyDecision( return new GrpcIdempotencyDecision(
Outcome.WAIT_AND_POLL, Optional.empty(), retryAfter, "an identical request is in progress"); Outcome.WAIT_AND_POLL,
Optional.empty(),
retryAfter,
"an identical request is in progress",
Optional.empty());
} }
/** Refuse a duplicate of an in-progress claim. */ /** Refuse a duplicate of an in-progress claim. */
@@ -76,7 +111,8 @@ public record GrpcIdempotencyDecision(
Outcome.REJECT_IN_PROGRESS, Outcome.REJECT_IN_PROGRESS,
Optional.empty(), Optional.empty(),
Duration.ZERO, Duration.ZERO,
"an identical request is in progress and this method does not wait for it"); "an identical request is in progress and this method does not wait for it",
Optional.empty());
} }
/** Report a key reused for a different request. */ /** Report a key reused for a different request. */
@@ -85,7 +121,8 @@ public record GrpcIdempotencyDecision(
Outcome.FINGERPRINT_MISMATCH, Outcome.FINGERPRINT_MISMATCH,
Optional.empty(), Optional.empty(),
Duration.ZERO, Duration.ZERO,
"the idempotency key was already used for a different request"); "the idempotency key was already used for a different request",
Optional.empty());
} }
/** Whether the use case should run. */ /** Whether the use case should run. */
@@ -1,5 +1,6 @@
package dev.caskeleton.grpc.idempotency; package dev.caskeleton.grpc.idempotency;
import dev.caskeleton.grpc.api.ledger.GrpcClaimToken;
import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity; import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity;
import dev.caskeleton.grpc.api.ledger.GrpcOperationLedger; import dev.caskeleton.grpc.api.ledger.GrpcOperationLedger;
import dev.caskeleton.grpc.api.ledger.GrpcOperationLedgerRecord; import dev.caskeleton.grpc.api.ledger.GrpcOperationLedgerRecord;
@@ -21,7 +22,9 @@ import java.util.Optional;
* *
* <p>A claim does not block its key forever. One still {@code IN_PROGRESS} after the claim TTL is * <p>A claim does not block its key forever. One still {@code IN_PROGRESS} after the claim TTL is
* treated as abandoned by a dead owner and taken over by the next retry of the same request — one * treated as abandoned by a dead owner and taken over by the next retry of the same request — one
* winner, like the re-claim of a terminal failure. * winner, like the re-claim of a terminal failure. The owner taken over may only have been slow; it
* still holds the old {@link GrpcClaimToken}, so its later commit is refused and its later failure
* ignored instead of landing on the attempt that took the claim.
* *
* <p>The ordering matters and is easy to get subtly wrong. The claim is attempted first, * <p>The ordering matters and is easy to get subtly wrong. The claim is attempted first,
* atomically; only then is the existing record examined. Reading first and claiming second leaves a * atomically; only then is the existing record examined. Reading first and claiming second leaves a
@@ -124,7 +127,8 @@ public final class GrpcIdempotencyInterceptor {
callerFingerprint, method, GrpcRequestFingerprint.hashIdempotencyKey(idempotencyKey)); callerFingerprint, method, GrpcRequestFingerprint.hashIdempotencyKey(idempotencyKey));
Optional<GrpcOperationLedgerRecord> existing = ledger.claim(identity, requestFingerprint, now); Optional<GrpcOperationLedgerRecord> existing = ledger.claim(identity, requestFingerprint, now);
if (existing.isEmpty()) { if (existing.isEmpty()) {
return GrpcIdempotencyDecision.proceed("claimed; this caller owns the operation"); return GrpcIdempotencyDecision.proceedHolding(
GrpcClaimToken.FIRST, "claimed; this caller owns the operation");
} }
GrpcOperationLedgerRecord record = existing.get(); GrpcOperationLedgerRecord record = existing.get();
@@ -142,24 +146,32 @@ public final class GrpcIdempotencyInterceptor {
+ identity.storageKey() + identity.storageKey()
+ "' has no outcome reference to replay"))); + "' has no outcome reference to replay")));
} }
if (record.state() == GrpcOperationLedgerState.FAILED_TERMINAL if (record.state() == GrpcOperationLedgerState.FAILED_TERMINAL) {
&& ledger.reclaimFailed(identity, requestFingerprint, now)) {
// A terminal failure is not replayed: the caller asked for the operation, it did not happen, // A terminal failure is not replayed: the caller asked for the operation, it did not happen,
// and repeating it is exactly what the key makes safe — but only for the one retry that // and repeating it is exactly what the key makes safe — but only for the one retry that
// re-claims the row. Proceeding unclaimed runs concurrent retries twice and leaves the // re-claims the row. Proceeding unclaimed runs concurrent retries twice and leaves the
// commit without an owner. // commit without an owner.
return GrpcIdempotencyDecision.proceed( Optional<GrpcClaimToken> reclaimed = ledger.reclaimFailed(identity, requestFingerprint, now);
"the previous attempt failed terminally; this caller re-claimed it for a new attempt"); if (reclaimed.isPresent()) {
return GrpcIdempotencyDecision.proceedHolding(
reclaimed.get(),
"the previous attempt failed terminally; this caller re-claimed it for a new attempt");
}
} }
if (record.state() == GrpcOperationLedgerState.IN_PROGRESS) { if (record.state() == GrpcOperationLedgerState.IN_PROGRESS) {
Instant cutoff = now.minus(claimTtl); Instant cutoff = now.minus(claimTtl);
if (record.claimedAt().isBefore(cutoff) // The owner outlived the TTL without committing or failing, so it is presumed dead. Its
&& ledger.reclaimStale(identity, requestFingerprint, cutoff, now)) { // mutation did not commit if the commit was recorded in the mutation's own transaction,
// The owner outlived the TTL without committing or failing, so it is presumed dead. Its // which the ledger contract requires; when an adopter commits them separately, the
// mutation did not commit if the commit was recorded in the mutation's own transaction, // abandoned attempt's outcome is unknown and this re-run can duplicate it. An owner that was
// which the ledger contract requires; when an adopter commits them separately, the // only slow is fenced by the new claim token: its commit is refused, its failure ignored.
// abandoned attempt's outcome is unknown and this re-run can duplicate it. Optional<GrpcClaimToken> takenOver =
return GrpcIdempotencyDecision.proceed( record.claimedAt().isBefore(cutoff)
? ledger.reclaimStale(identity, requestFingerprint, cutoff, now)
: Optional.empty();
if (takenOver.isPresent()) {
return GrpcIdempotencyDecision.proceedHolding(
takenOver.get(),
"the previous claim was abandoned (older than the " "the previous claim was abandoned (older than the "
+ claimTtl + claimTtl
+ " claim TTL); this caller took it over for a new attempt"); + " claim TTL); this caller took it over for a new attempt");
@@ -178,26 +190,42 @@ public final class GrpcIdempotencyInterceptor {
* <p>Called from inside the business transaction wherever the datastore permits it. A ledger * <p>Called from inside the business transaction wherever the datastore permits it. A ledger
* committed after the mutation leaves a window in which the write is durable and the claim is * committed after the mutation leaves a window in which the write is durable and the claim is
* not. * not.
*
* @param claimToken the token of the {@code PROCEED} decision this attempt ran under
* @throws IllegalStateException when this attempt no longer holds the claim — another attempt
* took it over — so the business transaction rolls back instead of committing a second run
*/ */
public void recordCommit( public void recordCommit(
GrpcMethodName method, GrpcMethodName method,
String callerFingerprint, String callerFingerprint,
String idempotencyKey, String idempotencyKey,
GrpcClaimToken claimToken,
String outcomeReference, String outcomeReference,
Instant now) { Instant now) {
ledger.markCommitted( ledger.markCommitted(
new GrpcOperationIdentity( new GrpcOperationIdentity(
callerFingerprint, method, GrpcRequestFingerprint.hashIdempotencyKey(idempotencyKey)), callerFingerprint, method, GrpcRequestFingerprint.hashIdempotencyKey(idempotencyKey)),
claimToken,
outcomeReference, outcomeReference,
now); now);
} }
/** Records a terminal failure against the claim. */ /**
public void recordFailure( * Records a terminal failure against the claim, if this attempt still holds it.
GrpcMethodName method, String callerFingerprint, String idempotencyKey, Instant now) { *
ledger.markFailed( * @param claimToken the token of the {@code PROCEED} decision this attempt ran under
* @return false when another attempt holds the claim now; its run is not failed by this one
*/
public boolean recordFailure(
GrpcMethodName method,
String callerFingerprint,
String idempotencyKey,
GrpcClaimToken claimToken,
Instant now) {
return ledger.markFailed(
new GrpcOperationIdentity( new GrpcOperationIdentity(
callerFingerprint, method, GrpcRequestFingerprint.hashIdempotencyKey(idempotencyKey)), callerFingerprint, method, GrpcRequestFingerprint.hashIdempotencyKey(idempotencyKey)),
claimToken,
now); now);
} }
} }
@@ -2,6 +2,7 @@ package dev.caskeleton.grpc.idempotency;
import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThat;
import dev.caskeleton.grpc.api.ledger.GrpcClaimToken;
import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity; import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity;
import dev.caskeleton.grpc.api.ledger.InMemoryOperationLedger; import dev.caskeleton.grpc.api.ledger.InMemoryOperationLedger;
import dev.caskeleton.grpc.core.GrpcMethodName; import dev.caskeleton.grpc.core.GrpcMethodName;
@@ -42,8 +43,11 @@ class GrpcCompletionReconcilerTest {
GrpcCompletionReconciler reconciler = new GrpcCompletionReconciler(query); GrpcCompletionReconciler reconciler = new GrpcCompletionReconciler(query);
claim(); claim();
reconciler.reconcile(CREATE, CALLER, KEY, null); reconciler.reconcile(CREATE, CALLER, KEY, null);
ledger.reclaimStale(identity, fingerprint, afterTtl.minus(Duration.ofMinutes(5)), afterTtl); GrpcClaimToken takenOver =
ledger.markCommitted(identity, "outcome://document/42", afterTtl.plusSeconds(1)); ledger
.reclaimStale(identity, fingerprint, afterTtl.minus(Duration.ofMinutes(5)), afterTtl)
.orElseThrow();
ledger.markCommitted(identity, takenOver, "outcome://document/42", afterTtl.plusSeconds(1));
// when // when
GrpcCompletionResolution nextPass = reconciler.reconcile(CREATE, CALLER, KEY, null); GrpcCompletionResolution nextPass = reconciler.reconcile(CREATE, CALLER, KEY, null);
@@ -93,6 +97,7 @@ class GrpcCompletionReconcilerTest {
ledger.markCommitted( ledger.markCommitted(
new dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity( new dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity(
CALLER, CREATE, GrpcRequestFingerprint.hashIdempotencyKey(KEY)), CALLER, CREATE, GrpcRequestFingerprint.hashIdempotencyKey(KEY)),
GrpcClaimToken.FIRST,
"outcome://document/42", "outcome://document/42",
NOW.plusSeconds(1)); NOW.plusSeconds(1));
@@ -116,6 +121,7 @@ class GrpcCompletionReconcilerTest {
ledger.markCommitted( ledger.markCommitted(
new dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity( new dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity(
CALLER, CREATE, GrpcRequestFingerprint.hashIdempotencyKey(KEY)), CALLER, CREATE, GrpcRequestFingerprint.hashIdempotencyKey(KEY)),
GrpcClaimToken.FIRST,
"outcome://from-ledger", "outcome://from-ledger",
NOW.plusSeconds(1)); NOW.plusSeconds(1));
GrpcCompletionReconciler reconciler = new GrpcCompletionReconciler(query); GrpcCompletionReconciler reconciler = new GrpcCompletionReconciler(query);
@@ -5,6 +5,7 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.assertj.core.api.Assertions.catchThrowable; import static org.assertj.core.api.Assertions.catchThrowable;
import dev.caskeleton.grpc.api.deadline.GrpcDeadlineProfile; import dev.caskeleton.grpc.api.deadline.GrpcDeadlineProfile;
import dev.caskeleton.grpc.api.ledger.GrpcClaimToken;
import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity; import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity;
import dev.caskeleton.grpc.api.ledger.GrpcOperationLedgerRecord; import dev.caskeleton.grpc.api.ledger.GrpcOperationLedgerRecord;
import dev.caskeleton.grpc.api.ledger.GrpcOperationLedgerState; import dev.caskeleton.grpc.api.ledger.GrpcOperationLedgerState;
@@ -98,8 +99,10 @@ class GrpcIdempotencyInterceptorTest {
void aDuplicateOfACommittedOperationReplays() { void aDuplicateOfACommittedOperationReplays() {
GrpcIdempotencyInterceptor interceptor = interceptor(false); GrpcIdempotencyInterceptor interceptor = interceptor(false);
String fingerprint = GrpcRequestFingerprint.of("body"); String fingerprint = GrpcRequestFingerprint.of("body");
interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW); GrpcClaimToken claim =
interceptor.recordCommit(CREATE, CALLER, KEY, "outcome://document/42", NOW.plusSeconds(1)); interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW).claimToken().orElseThrow();
interceptor.recordCommit(
CREATE, CALLER, KEY, claim, "outcome://document/42", NOW.plusSeconds(1));
GrpcIdempotencyDecision replayed = GrpcIdempotencyDecision replayed =
interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW.plusSeconds(2)); interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW.plusSeconds(2));
@@ -139,8 +142,9 @@ class GrpcIdempotencyInterceptorTest {
void aTerminallyFailedOperationMayBeRetried() { void aTerminallyFailedOperationMayBeRetried() {
GrpcIdempotencyInterceptor interceptor = interceptor(false); GrpcIdempotencyInterceptor interceptor = interceptor(false);
String fingerprint = GrpcRequestFingerprint.of("body"); String fingerprint = GrpcRequestFingerprint.of("body");
interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW); GrpcClaimToken claim =
interceptor.recordFailure(CREATE, CALLER, KEY, NOW.plusSeconds(1)); interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW).claimToken().orElseThrow();
interceptor.recordFailure(CREATE, CALLER, KEY, claim, NOW.plusSeconds(1));
assertThat(interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW.plusSeconds(2)).outcome()) assertThat(interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW.plusSeconds(2)).outcome())
.isEqualTo(GrpcIdempotencyDecision.Outcome.PROCEED); .isEqualTo(GrpcIdempotencyDecision.Outcome.PROCEED);
@@ -151,8 +155,9 @@ class GrpcIdempotencyInterceptorTest {
// given // given
GrpcIdempotencyInterceptor interceptor = interceptor(false); GrpcIdempotencyInterceptor interceptor = interceptor(false);
String fingerprint = GrpcRequestFingerprint.of("body"); String fingerprint = GrpcRequestFingerprint.of("body");
interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW); GrpcClaimToken claim =
interceptor.recordFailure(CREATE, CALLER, KEY, NOW.plusSeconds(1)); interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW).claimToken().orElseThrow();
interceptor.recordFailure(CREATE, CALLER, KEY, claim, NOW.plusSeconds(1));
// when // when
interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW.plusSeconds(2)); interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW.plusSeconds(2));
@@ -172,8 +177,9 @@ class GrpcIdempotencyInterceptorTest {
// given // given
GrpcIdempotencyInterceptor interceptor = interceptor(false); GrpcIdempotencyInterceptor interceptor = interceptor(false);
String fingerprint = GrpcRequestFingerprint.of("body"); String fingerprint = GrpcRequestFingerprint.of("body");
interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW); GrpcClaimToken claim =
interceptor.recordFailure(CREATE, CALLER, KEY, NOW.plusSeconds(1)); interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW).claimToken().orElseThrow();
interceptor.recordFailure(CREATE, CALLER, KEY, claim, NOW.plusSeconds(1));
CountDownLatch start = new CountDownLatch(1); CountDownLatch start = new CountDownLatch(1);
Callable<GrpcIdempotencyDecision.Outcome> retry = Callable<GrpcIdempotencyDecision.Outcome> retry =
() -> { () -> {
@@ -235,6 +241,70 @@ class GrpcIdempotencyInterceptorTest {
.contains(AFTER_TTL); .contains(AFTER_TTL);
} }
@Test
@DisplayName("claim 을 이긴 PROCEED 는 그 시도의 claim token 을 들고 있다")
void aProceedThatWonAClaimHoldsItsClaimToken() {
// given
GrpcIdempotencyInterceptor interceptor = interceptor(false);
String fingerprint = GrpcRequestFingerprint.of("body");
interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW);
// when
GrpcIdempotencyDecision takenOver =
interceptor.decide(CREATE, CALLER, KEY, fingerprint, AFTER_TTL);
// then
assertThat(takenOver.claimToken()).contains(GrpcClaimToken.FIRST.next());
}
/**
* The re-review probe's sequence through the interceptor (2026-11-03, RR-02-04): A is slow, B
* takes over after the TTL, A then fails. A's failure must not free the key for C while B runs.
*/
@Test
@DisplayName("TTL 뒤 넘겨받힌 옛 owner 의 실패 기록은 무시되어 다음 재시도는 진행 중으로 거부된다")
void aFailureRecordedByTheOwnerThatLostTheClaimDoesNotLetTheNextRetryRun() {
// given
GrpcIdempotencyInterceptor interceptor = interceptor(false);
String fingerprint = GrpcRequestFingerprint.of("body");
GrpcClaimToken first =
interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW).claimToken().orElseThrow();
interceptor.decide(CREATE, CALLER, KEY, fingerprint, AFTER_TTL);
boolean recorded =
interceptor.recordFailure(CREATE, CALLER, KEY, first, AFTER_TTL.plusSeconds(1));
// when
GrpcIdempotencyDecision third =
interceptor.decide(CREATE, CALLER, KEY, fingerprint, AFTER_TTL.plusSeconds(2));
// then
assertThat(recorded).isFalse();
assertThat(third.outcome()).isEqualTo(GrpcIdempotencyDecision.Outcome.REJECT_IN_PROGRESS);
}
@Test
@DisplayName("TTL 뒤 넘겨받힌 옛 owner 의 commit 은 거부되어 그 트랜잭션이 롤백된다")
void aCommitRecordedByTheOwnerThatLostTheClaimIsRefused() {
// given
GrpcIdempotencyInterceptor interceptor = interceptor(false);
String fingerprint = GrpcRequestFingerprint.of("body");
GrpcClaimToken first =
interceptor.decide(CREATE, CALLER, KEY, fingerprint, NOW).claimToken().orElseThrow();
interceptor.decide(CREATE, CALLER, KEY, fingerprint, AFTER_TTL);
// when
Throwable refused =
org.assertj.core.api.Assertions.catchThrowable(
() ->
interceptor.recordCommit(
CREATE, CALLER, KEY, first, "outcome://document/42", AFTER_TTL.plusSeconds(1)));
// then
assertThat(refused)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("no longer held");
}
@Test @Test
@DisplayName("TTL이 지나기 직전의 중복 요청은 여전히 대기하거나 거부된다") @DisplayName("TTL이 지나기 직전의 중복 요청은 여전히 대기하거나 거부된다")
void aDuplicateAtTheTtlBoundaryIsStillTreatedAsRunning() { void aDuplicateAtTheTtlBoundaryIsStillTreatedAsRunning() {
@@ -3,6 +3,7 @@ package dev.caskeleton.grpc.idempotency;
import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.catchThrowable; import static org.assertj.core.api.Assertions.catchThrowable;
import dev.caskeleton.grpc.api.ledger.GrpcClaimToken;
import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity; import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity;
import dev.caskeleton.grpc.api.ledger.InMemoryOperationLedger; import dev.caskeleton.grpc.api.ledger.InMemoryOperationLedger;
import dev.caskeleton.grpc.core.GrpcMethodName; import dev.caskeleton.grpc.core.GrpcMethodName;
@@ -78,6 +79,7 @@ class GrpcOperationLedgerRetentionTest {
// Finished at the claim moment; the completion moment is what the retention measures. // Finished at the claim moment; the completion moment is what the retention measures.
private void committed(GrpcOperationIdentity identity, Instant claimedAt) { private void committed(GrpcOperationIdentity identity, Instant claimedAt) {
ledger.claim(identity, BODY, claimedAt); ledger.claim(identity, BODY, claimedAt);
ledger.markCommitted(identity, "outcome://document/" + identity.storageKey(), claimedAt); ledger.markCommitted(
identity, GrpcClaimToken.FIRST, "outcome://document/" + identity.storageKey(), claimedAt);
} }
} }
@@ -24,6 +24,11 @@ import java.util.Set;
* reflection in production publishes the schema, and a keyed method without a ledger accepts * reflection in production publishes the schema, and a keyed method without a ledger accepts
* idempotency keys it cannot honour. None of them fails a smoke test. * idempotency keys it cannot honour. None of them fails a smoke test.
* *
* <p>The keyed-method rule has a limit it states in its own message: with the ledger flag on, the
* starter assembles the decision function, and the server stage that calls it is the adopter's. A
* running process cannot see whether that stage exists, so passing this check with the flag on does
* not mean keys are honoured (re-review 2026-11-03, RR-02-07; ADR-GRPC-004).
*
* <p>Fails once with every violation, so a deployment learns the whole list in one restart. * <p>Fails once with every violation, so a deployment learns the whole list in one restart.
*/ */
public final class GrpcPlatformStartupValidator { public final class GrpcPlatformStartupValidator {
@@ -152,7 +157,10 @@ public final class GrpcPlatformStartupValidator {
"method '" "method '"
+ method.canonical() + method.canonical()
+ "' requires an idempotency key but app.grpc.platform.operation-ledger-enabled is" + "' requires an idempotency key but app.grpc.platform.operation-ledger-enabled is"
+ " false; the key would be accepted and never honoured"); + " false; the key would be accepted and never honoured. Turning the flag on"
+ " assembles the decision function only: the key is honoured once the server's"
+ " IDEMPOTENCY stage interceptor or the service adapter calls decide/recordCommit,"
+ " which this starter does not wire");
} }
if (!policy.rpcType().stable()) { if (!policy.rpcType().stable()) {
violations.add( violations.add(
@@ -5,6 +5,7 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.assertj.core.api.Assertions.catchThrowable; import static org.assertj.core.api.Assertions.catchThrowable;
import dev.caskeleton.grpc.api.deadline.GrpcDeadlineProfile; import dev.caskeleton.grpc.api.deadline.GrpcDeadlineProfile;
import dev.caskeleton.grpc.api.ledger.GrpcClaimToken;
import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity; import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity;
import dev.caskeleton.grpc.api.ledger.GrpcOperationLedger; import dev.caskeleton.grpc.api.ledger.GrpcOperationLedger;
import dev.caskeleton.grpc.api.ledger.InMemoryOperationLedger; import dev.caskeleton.grpc.api.ledger.InMemoryOperationLedger;
@@ -196,7 +197,8 @@ class GrpcPlatformAutoConfigurationContextTest {
GrpcOperationIdentity identity = GrpcOperationIdentity identity =
new GrpcOperationIdentity(CALLER, CREATE, GrpcRequestFingerprint.hashIdempotencyKey(KEY)); new GrpcOperationIdentity(CALLER, CREATE, GrpcRequestFingerprint.hashIdempotencyKey(KEY));
ledger.claim(identity, GrpcRequestFingerprint.of("body"), NOW); ledger.claim(identity, GrpcRequestFingerprint.of("body"), NOW);
ledger.markCommitted(identity, "outcome://document/42", NOW.plusSeconds(1)); ledger.markCommitted(
identity, GrpcClaimToken.FIRST, "outcome://document/42", NOW.plusSeconds(1));
// when // when
int purged = int purged =
@@ -200,6 +200,44 @@ class GrpcPlatformStartupValidatorTest {
.anySatisfy(violation -> assertThat(violation).contains("never honoured")); .anySatisfy(violation -> assertThat(violation).contains("never honoured"));
} }
/**
* The flag assembles the decision function only; the starter builds no server, so nothing calls
* it until the adopter's IDEMPOTENCY stage or service adapter does. The refusal must not read as
* "turn the flag on and keys are honoured" (re-review 2026-11-03, RR-02-07).
*/
@Test
@DisplayName("키 필수 메서드 거부 문구는 플래그가 결정 함수만 조립하고 서버 단계 연결은 따로라고 말한다")
void theKeyedMethodRefusalSaysTheFlagAssemblesTheDecisionOnly() {
// given
GrpcPlatformProperties properties = productionProperties();
properties.setOperationLedgerEnabled(false);
GrpcMethodPolicyCatalog catalog =
GrpcMethodPolicyCatalog.builder()
.register(
new GrpcMethodPolicy(
CREATE,
RpcType.UNARY,
RpcIdempotencyProfile.IDEMPOTENCY_KEY_REQUIRED,
GrpcDeadlineProfile.of(Duration.ofSeconds(2)),
WaitForReadyPolicy.DISABLED,
false,
1024,
1024))
.build();
// when
var violations =
GrpcPlatformStartupValidator.violations(properties, catalog, oneChannel(), stableServer());
// then
assertThat(violations)
.anySatisfy(
violation ->
assertThat(violation)
.contains("assembles the decision function only")
.contains("IDEMPOTENCY stage"));
}
@Test @Test
@DisplayName("a client-streaming method is refused as an Advanced capability") @DisplayName("a client-streaming method is refused as an Advanced capability")
void clientStreamingIsRefusedAsAdvanced() { void clientStreamingIsRefusedAsAdvanced() {
+5 -4
View File
@@ -5,8 +5,9 @@ plugins {
// Certification. The Stable plan splits this across four modules (core / in-process / netty / // Certification. The Stable plan splits this across four modules (core / in-process / netty /
// fault); this repository already expresses "these two runs are not the same kind of evidence" with // fault); this repository already expresses "these two runs are not the same kind of evidence" with
// strict test lanes rather than with module boundaries, so the four become four lanes over one // strict test lanes rather than with module boundaries, so the four become four lanes over one
// leaf (adaptation design §2). A lane that discovers nothing fails, and none of them can serve an // leaf (adaptation design §2). A lane that discovers nothing fails, and the three hermetic lanes
// up-to-date result — which is the property the split was protecting. // reuse a result only when their classes and their lane policy are unchanged — the performance lane
// never does. That is the property the split was protecting.
strictTestLanes { strictTestLanes {
lane('grpcInProcessContractTest') { lane('grpcInProcessContractTest') {
@@ -33,8 +34,8 @@ strictTestLanes {
} }
// The performance lane is excluded from the default `test` task. It measures a running server // The performance lane is excluded from the default `test` task. It measures a running server
// under load, and a measurement in the release gate is a flaky test on a shared CI runner; it runs // under load, so it is not in the pull-request or release gate; it runs in the optional build's
// when somebody asks for it, by name. // `performanceCheck`, which integration-main.yml's performance-check job runs on main and nightly.
tasks.named('test') { tasks.named('test') {
useJUnitPlatform { useJUnitPlatform {
excludeTags 'grpc-performance' excludeTags 'grpc-performance'