diff --git a/docs/adr/ADR-GRPC-004-retry-ownership-and-durable-idempotency.md b/docs/adr/ADR-GRPC-004-retry-ownership-and-durable-idempotency.md index deb788b..495d348 100644 --- a/docs/adr/ADR-GRPC-004-retry-ownership-and-durable-idempotency.md +++ b/docs/adr/ADR-GRPC-004-retry-ownership-and-durable-idempotency.md @@ -53,8 +53,10 @@ request takes it over with `GrpcOperationLedger.reclaimStale` — one conditiona `UPDATE … WHERE state = 'IN_PROGRESS' AND claimed_at < :cutoff`, so of two concurrent takeovers exactly one wins, the same shape as `reclaimFailed`. The TTL is the `claimTtl` argument of `GrpcIdempotencyInterceptor`, refused at construction when it is not longer than every keyed method's -deadline. It is not a configuration key: the starter does not assemble the interceptor yet (03-11), so -there is nothing for a key to configure. +deadline. Since the 03-11 follow-up the starter assembles the interceptor when +`app.grpc.platform.operation-ledger-enabled=true` and takes the TTL from +`app.grpc.platform.operation-ledger-claim-ttl` (default 5m); a TTL the constructor refuses fails startup +naming that key, and an enabled flag without a `GrpcOperationLedger` bean fails startup too. The takeover is safe only because the commit is recorded in the mutation's own transaction: then a claim still `IN_PROGRESS` means the mutation never committed. An adopter that commits them separately @@ -65,7 +67,18 @@ method and not just its deadline. Rejected: an owner token with a heartbeat lease (the outbox lease model) — needed only when a keyed method can run longer than any fixed TTL; revisit when one exists. A sweeper job that deletes stale rows — it needs a scheduler this family does not have, and deleting a claim makes the next retry a -fresh claim with no record that an attempt was abandoned. Retention of terminal rows is still open. +fresh claim with no record that an attempt was abandoned. + +**Finished rows are purged after a retention; in-progress rows never are** (03-10 follow-up). +`GrpcOperationLedger.purgeCompletedBefore(cutoff, limit)` deletes at most `limit` `COMMITTED` or +`FAILED_TERMINAL` rows that completed before the cutoff, oldest first — on PostgreSQL one bounded +`DELETE … WHERE storage_key IN (SELECT … LIMIT … FOR UPDATE SKIP LOCKED)` served by the partial index +of migration V002. `GrpcOperationLedgerRetention.purgeExpired(now)` applies +`app.grpc.platform.operation-ledger-retention` (default 24h, the HTTP idempotency replay TTL) in +batches. A retry after its record was purged is a fresh claim and runs again, so the retention must +exceed the callers' retry window. The adopter schedules the purge; the starter assembles the policy +only, because a scheduled bean would force a scheduler onto every deployment and the starter cannot +know which replica should run it. **A key reused for a different request is a caller error, not a duplicate.** The stored request fingerprint turns that into `FAILED_PRECONDITION` rather than silently returning the first request's diff --git a/docs/runbooks/grpc-platform-operations.md b/docs/runbooks/grpc-platform-operations.md index d218f26..0b63212 100644 --- a/docs/runbooks/grpc-platform-operations.md +++ b/docs/runbooks/grpc-platform-operations.md @@ -39,8 +39,8 @@ the time, and which half is not knowable from the status. escalate to the service owner. This is the case the keyed profile exists to avoid. 7. `IN_PROGRESS` means a claim exists and has not finished. If its owner died, the claim is not - stuck forever: once it is older than the claim TTL (the `claimTtl` given to - `GrpcIdempotencyInterceptor`), the next retry of the same request takes it over and runs again, + stuck forever: once it is older than the claim TTL (`app.grpc.platform.operation-ledger-claim-ttl`, + default 5m), the next retry of the same request takes it over and runs again, and a later reconciler pass sees that attempt's `COMMITTED` or `FAILED_TERMINAL`. Until a retry arrives, the case stays pending. The re-run is safe only when the ledger commit is recorded in the mutation's own transaction; if this service commits them separately, check the business resource @@ -52,6 +52,32 @@ retry arriving is a caller that gave up; it needs the business resource checked, --- +## The operation ledger table keeps growing + +**What you are seeing.** `grpc_operation_ledger` grows without bound, or a retry of an old key runs +the mutation again instead of replaying. + +**What it means.** Finished rows (`COMMITTED`, `FAILED_TERMINAL`) are deleted only when something +calls `GrpcOperationLedgerRetention.purgeExpired(now)`; the starter assembles that bean +(`app.grpc.platform.operation-ledger-enabled=true`) but does not schedule it. A growing table means +the service's scheduler is not calling it. A retry that re-ran after the retention +(`app.grpc.platform.operation-ledger-retention`, default 24h) is the trade the retention makes: its +record was purged, so it was a new operation. + +**What to do.** + +1. Check that the service schedules `purgeExpired`. Each call deletes in batches of 500 until a batch + comes back short, never touching `IN_PROGRESS` rows. +2. For a large backlog, call it repeatedly; each batch is a short transaction that skips rows another + transaction holds. +3. If retried keys re-run after the retention, raise the retention above the callers' retry window + rather than skipping the purge. + +**Escalate when** old `IN_PROGRESS` rows accumulate. Those are not purged by design; each is an +abandoned claim no retry has taken over (see `IN_PROGRESS` above). + +--- + ## A stream ended with FULL_RESYNC_REQUIRED **What you are seeing.** A client's resume was refused and it was told to resynchronise. diff --git a/gradle.properties b/gradle.properties new file mode 100644 index 0000000..0d1f586 --- /dev/null +++ b/gradle.properties @@ -0,0 +1,9 @@ +# optional-platforms is a build of its own (`./gradlew -p optional-platforms ...`), and Gradle reads +# gradle.properties from the build's root directory only, so src/gradle.properties never reached it: +# the optional build ran serially and without the build cache (review 2026-11-01 08-06). +# The values and their reasons are src/gradle.properties'. settings.gradle fails the build when the +# two files disagree, so this is a copy that cannot drift, not a second source. +org.gradle.workers.max=4 +org.gradle.jvmargs=-Xmx1g -XX:MaxMetaspaceSize=512m -Dfile.encoding=UTF-8 +org.gradle.caching=true +org.gradle.parallel=true diff --git a/grpc/CLAUDE.md b/grpc/CLAUDE.md index f056c0a..ada4282 100644 --- a/grpc/CLAUDE.md +++ b/grpc/CLAUDE.md @@ -96,6 +96,9 @@ application-owned port → adapter:inbound:grpc (typed service adapter) → :grp 직접 닿지 않는 규칙은 app-bootstrap ArchUnit(`INBOUND_ADAPTERS_DO_NOT_DEPEND_ON_OUTBOUND_ADAPTERS`)이 검사한다. 현재 `adapter:inbound:grpc`는 이 family에 의존하지 않는다 — registry의 `allowed_dependencies`를 보라. 배선은 별도 결정이며, 그때 registry edge와 브리지가 함께 들어간다. +starter가 `operation-ledger-enabled=true`일 때 멱등성 판단 bean을 조립하는 것(03-11)은 이 배선이 아니다 — +starter는 서버를 만들지 않고, 그 bean을 서버에 붙이는 일은 채택하는 쪽의 몫이다(아래 「Durable operation +ledger adoption」). ## 테스트 명령 @@ -130,13 +133,51 @@ the repository's `.seconds` / `.total` unit naming and the corresponding validat ### Durable operation ledger adoption -`grpc-spring-boot-starter` does not pull the JPA ledger adapter transitively. A deployment that needs -durable operation idempotency adds `:grpc:grpc-operation-ledger-jpa` explicitly, enables the ledger -configuration, and includes that adapter's Flyway location `db/migration/grpc`. A deployment that -does not adopt the ledger therefore pays no Spring Data JPA classpath cost. +`grpc-spring-boot-starter` does not pull the JPA ledger adapter transitively (11-M9). A deployment that +needs durable operation idempotency adds `:grpc:grpc-operation-ledger-jpa` explicitly, defines a +`GrpcOperationLedger` bean (`new JpaGrpcOperationLedger(repository)`), includes that adapter's Flyway +location `db/migration/grpc` (V001 table, V002 retention index), and sets the flag below. A deployment +that does not adopt the ledger pays no Spring Data JPA classpath cost. -An `IN_PROGRESS` claim expires after the claim TTL (`GrpcIdempotencyInterceptor`'s `claimTtl` -argument; no configuration key while the starter does not assemble the interceptor). A retry of the -same request then takes it over through `GrpcOperationLedger.reclaimStale`. The adopter must record -the commit in the mutation's transaction and choose a TTL longer than the slowest keyed method; both -reasons are in ADR-GRPC-004. +| Key (`app.grpc.platform.*`) | Default | What it does | +| --- | --- | --- | +| `operation-ledger-enabled` | `false` | `true`: the starter assembles `GrpcIdempotencyInterceptor` (bean `grpcIdempotencyInterceptor`) and `GrpcOperationLedgerRetention` (bean `grpcOperationLedgerRetention`) over the deployment's `GrpcOperationLedger` bean, and refuses startup when there is none. `false`: neither bean exists, and a method that is `IDEMPOTENCY_KEY_REQUIRED` refuses startup | +| `operation-ledger-claim-ttl` | `5m` | How long an `IN_PROGRESS` claim is trusted to have a live owner. After it, a retry of the same request takes the claim over (`GrpcOperationLedger.reclaimStale`). Refused at startup unless longer than every keyed method's deadline | +| `operation-ledger-retention` | `24h` | How long a finished (`COMMITTED`/`FAILED_TERMINAL`) record is kept before `GrpcOperationLedgerRetention.purgeExpired` may delete it. `IN_PROGRESS` rows are never purged | + +Defaults: the 5-minute TTL is the value the ledger contract and interceptor tests certify, far beyond +a unary deadline, and short enough that a key whose owner died unblocks within a client's retry +patience. The 24-hour retention matches the HTTP idempotency replay TTL +(`app.capabilities.idempotency.replay-ttl`). A retry of a key after its record was purged is a new +operation, so the retention must exceed the callers' retry window. + +These keys are not in `docs/registries/env-keys.yaml`: no `app.grpc.platform.*` key is, because the +optional build is not part of the shipped runtime. They join the registry when a composition root +adopts the platform. + +**Why the starter assembles the interceptor (03-11 (b)), and not (a).** The review proposed (a): +delete the flag and judge by whether a ledger bean exists. The owner chose (b): the flag stays and +means what it says — when true, the decision function is assembled and its TTL comes from +configuration. (a) would have removed the false promise but left every adopter to rebuild the same +assembly and TTL validation by hand; (b) keeps that in one place. The interceptor bean has no +`@ConditionalOnMissingBean` on purpose: a replacement would silently ignore `operation-ledger-claim-ttl`. +Duplicates of a running operation are refused (`REJECT_IN_PROGRESS`), not told to wait; a +wait-and-poll option becomes a key when a keyed method needs it. + +**Server wiring is still the adopter's, and stays a separate decision.** The starter builds no +server, so assembling the decision does not wire the platform into `adapter:inbound:grpc` (see "배선은 +별도 결정" above). `GrpcIdempotencyInterceptor` is a decision function, not an `io.grpc.ServerInterceptor`. +The adopter injects the bean by type and calls it from its own `IDEMPOTENCY` stage interceptor +(`GrpcServerInterceptorChain.builder().stage(GrpcServerInterceptorStage.IDEMPOTENCY, ...)`) or from +its typed service adapter: `decide(...)` before the mutation, `recordCommit(...)` inside the mutation's +transaction, `recordFailure(...)` on a terminal failure. + +**Purge scheduling is the adopter's.** `GrpcOperationLedgerRetention.purgeExpired(now)` deletes +expired finished rows in bounded batches (500 per statement) until a batch comes back short. The +starter does not schedule it: it owns no scheduler and cannot know which replica should run it, and a +scheduled bean would force a task scheduler onto every deployment. The adopter calls it from its own +scheduler (for example hourly); concurrent runs on several replicas are safe because the delete skips +locked rows. Running it never is safe but lets the table grow without bound. + +The commit must be recorded in the mutation's transaction and the TTL must be longer than the slowest +keyed method; both reasons are in ADR-GRPC-004. diff --git a/grpc/grpc-core-api/src/main/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedger.java b/grpc/grpc-core-api/src/main/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedger.java index 7a4ee67..2c27515 100644 --- a/grpc/grpc-core-api/src/main/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedger.java +++ b/grpc/grpc-core-api/src/main/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedger.java @@ -82,6 +82,27 @@ public interface GrpcOperationLedger { Instant claimedBefore, Instant now); + /** + * Deletes up to {@code limit} finished records ({@code COMMITTED} or {@code FAILED_TERMINAL}) + * that completed before {@code completedBefore}, oldest first. + * + *
Retention, not recovery. A finished row only answers retries, so once it is older than any + * retry a caller will still send, keeping it only grows the table. Deleting it is a trade the + * deployment makes knowingly: a retry of the same key arriving after the purge is a fresh claim + * and runs the mutation again. The retention must therefore exceed the callers' retry window. + * + *
An {@code IN_PROGRESS} record is never deleted, however old. It is either running or + * abandoned, and an abandoned one is taken over by {@link #reclaimStale}, which keeps the record + * that an attempt started; deleting it would erase that. + * + *
Bounded so a backlog is drained in short transactions rather than one long delete that holds + * locks on the hottest table of the idempotency path. + * + * @return how many records were deleted, at most {@code limit} + * @throws IllegalArgumentException when {@code limit} is not positive + */ + int purgeCompletedBefore(Instant completedBefore, int limit); + /** * Releases a claim that will never complete, so the caller may try again. * diff --git a/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedgerContract.java b/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedgerContract.java index b2839d6..0c625fc 100644 --- a/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedgerContract.java +++ b/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedgerContract.java @@ -13,6 +13,7 @@ import java.util.concurrent.CountDownLatch; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.Future; +import java.util.stream.Stream; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; @@ -40,6 +41,10 @@ public abstract class GrpcOperationLedgerContract { // A retry arriving after the TTL: its cutoff (LATER - TTL) is past the original claim at NOW. private static final Instant LATER = NOW.plus(CLAIM_TTL).plusSeconds(1); private static final Instant CUTOFF = LATER.minus(CLAIM_TTL); + // A purge run a day after NOW: rows finished around NOW are past retention. + private static final Instant RETENTION_CUTOFF = NOW.plus(Duration.ofDays(1)); + private static final GrpcOperationIdentity SECOND_ID = identityWithKeyHash('b'); + private static final GrpcOperationIdentity THIRD_ID = identityWithKeyHash('c'); /** The ledger under test, holding no records when a test starts. */ protected abstract GrpcOperationLedger ledger(); @@ -397,9 +402,104 @@ public abstract class GrpcOperationLedgerContract { assertThat(refused).isInstanceOf(IllegalArgumentException.class).hasMessageContaining("cutoff"); } - private void failedClaim() { + @Test + @DisplayName("보존 기간이 지난 COMMITTED·FAILED_TERMINAL 행은 정리된다") + void finishedRecordsCompletedBeforeTheCutoffArePurged() { + // given + committedClaim(ID, NOW); + failedClaim(SECOND_ID, NOW); + + // when + int purged = ledger().purgeCompletedBefore(RETENTION_CUTOFF, 10); + + // then + assertThat(purged).isEqualTo(2); + assertThat(ledger().find(ID)).isEmpty(); + assertThat(ledger().find(SECOND_ID)).isEmpty(); + } + + @Test + @DisplayName("IN_PROGRESS 행은 아무리 오래되어도 정리하지 않는다") + void anInProgressClaimIsNeverPurgedHoweverOld() { + // given ledger().claim(ID, FINGERPRINT, NOW); - ledger().markFailed(ID, NOW.plusSeconds(1)); + + // when + int purged = ledger().purgeCompletedBefore(RETENTION_CUTOFF, 10); + + // then + assertThat(purged).isZero(); + assertThat(ledger().find(ID)) + .get() + .extracting(GrpcOperationLedgerRecord::state) + .isEqualTo(GrpcOperationLedgerState.IN_PROGRESS); + } + + @Test + @DisplayName("cutoff 이후에 끝난 행은 남긴다") + void finishedRecordsCompletedAtOrAfterTheCutoffAreKept() { + // given + committedClaim(ID, RETENTION_CUTOFF.minusSeconds(1)); + failedClaim(SECOND_ID, RETENTION_CUTOFF.minusSeconds(1)); + + // when + int purged = ledger().purgeCompletedBefore(RETENTION_CUTOFF, 10); + + // then + assertThat(purged).isZero(); + assertThat(ledger().find(ID)).isPresent(); + assertThat(ledger().find(SECOND_ID)).isPresent(); + } + + @Test + @DisplayName("한 번의 정리는 limit 개수까지만 지운다") + void onePurgeDeletesAtMostTheLimit() { + // given + committedClaim(ID, NOW); + committedClaim(SECOND_ID, NOW.plusSeconds(1)); + committedClaim(THIRD_ID, NOW.plusSeconds(2)); + + // when + int purged = ledger().purgeCompletedBefore(RETENTION_CUTOFF, 2); + + // then + assertThat(purged).isEqualTo(2); + assertThat( + Stream.of(ID, SECOND_ID, THIRD_ID) + .filter(identity -> ledger().find(identity).isPresent()) + .count()) + .isEqualTo(1); + } + + @Test + @DisplayName("limit이 양수가 아니면 정리를 거부한다") + void aNonPositivePurgeLimitIsRefused() { + // when + Throwable refused = catchThrowable(() -> ledger().purgeCompletedBefore(RETENTION_CUTOFF, 0)); + + // then + assertThat(refused).isInstanceOf(IllegalArgumentException.class).hasMessageContaining("limit"); + } + + private void failedClaim() { + failedClaim(ID, NOW); + } + + // Claimed at `claimedAt`, finished one second later, so the completion moment is what a purge + // cutoff is compared with. + private void failedClaim(GrpcOperationIdentity identity, Instant claimedAt) { + ledger().claim(identity, FINGERPRINT, claimedAt); + ledger().markFailed(identity, claimedAt.plusSeconds(1)); + } + + private void committedClaim(GrpcOperationIdentity identity, Instant claimedAt) { + ledger().claim(identity, FINGERPRINT, claimedAt); + ledger().markCommitted(identity, OUTCOME, claimedAt.plusSeconds(1)); + } + + private static GrpcOperationIdentity identityWithKeyHash(char hexDigit) { + return new GrpcOperationIdentity( + ID.callerFingerprint(), ID.method(), "sha256:" + String.valueOf(hexDigit).repeat(64)); } /** Runs {@code attempt} on two threads released together and returns both results. */ diff --git a/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/InMemoryOperationLedger.java b/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/InMemoryOperationLedger.java index 207971b..5547914 100644 --- a/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/InMemoryOperationLedger.java +++ b/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/InMemoryOperationLedger.java @@ -1,6 +1,8 @@ package dev.caskeleton.grpc.api.ledger; import java.time.Instant; +import java.util.Comparator; +import java.util.List; import java.util.Optional; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentMap; @@ -10,8 +12,8 @@ import java.util.concurrent.ConcurrentMap; * *
{@code putIfAbsent} on a concurrent map is the in-memory equivalent of the unique constraint
* the real adapter relies on, and {@code replace(key, expected, next)} of its conditional update,
- * so claim, re-claim and stale takeover are atomic here for the same reason they are there. The
- * transitions match the JPA adapter's because {@link GrpcOperationLedgerContract} runs against
+ * so claim, re-claim, stale takeover and purge are atomic here for the same reason they are there.
+ * The transitions match the JPA adapter's because {@link GrpcOperationLedgerContract} runs against
* both; a fake that allowed more would let the policy tests pass for code the database refuses.
*/
public final class InMemoryOperationLedger implements GrpcOperationLedger {
@@ -102,6 +104,29 @@ public final class InMemoryOperationLedger implements GrpcOperationLedger {
GrpcOperationLedgerRecord.claim(identity, requestFingerprint, now));
}
+ @Override
+ public int purgeCompletedBefore(Instant completedBefore, int limit) {
+ if (limit < 1) {
+ throw new IllegalArgumentException("a purge limit must be positive");
+ }
+ List One list for every PostgreSQL lane, so a new migration is applied by all of them or by none;
+ * two copies of the list drift the first time someone adds a V003 to one of them.
+ */
+final class GrpcOperationLedgerMigrations {
+
+ private static final List {@code FOR UPDATE SKIP LOCKED} in the bounded subquery keeps the purge out of the way of a
+ * concurrent {@link #tryReclaimFailed}: a row that re-claim holds is skipped, and one it already
+ * turned back into {@code IN_PROGRESS} no longer matches. The outer state condition says the same
+ * thing again so the statement cannot delete a live claim even if the subquery is edited. The
+ * partial index {@code ix_grpc_operation_ledger_finished_completed_at} (V002) serves the
+ * subquery's filter and order.
+ *
+ * @return how many rows were deleted
+ */
+ @Modifying(flushAutomatically = true, clearAutomatically = true)
+ @Transactional
+ @Query(
+ value =
+ """
+ DELETE FROM grpc_operation_ledger
+ WHERE storage_key IN (
+ SELECT storage_key
+ FROM grpc_operation_ledger
+ WHERE state IN ('COMMITTED', 'FAILED_TERMINAL')
+ AND completed_at < :completedBefore
+ ORDER BY completed_at
+ LIMIT :limit
+ FOR UPDATE SKIP LOCKED)
+ AND state <> 'IN_PROGRESS'
+ """,
+ nativeQuery = true)
+ int deleteFinishedBefore(
+ @Param("completedBefore") Instant completedBefore, @Param("limit") int limit);
+
/** Updates an existing claim after it has been loaded. */
GrpcOperationLedgerEntity save(GrpcOperationLedgerEntity entity);
diff --git a/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedger.java b/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedger.java
index d81fd66..55d80bc 100644
--- a/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedger.java
+++ b/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedger.java
@@ -122,6 +122,17 @@ public final class JpaGrpcOperationLedger implements GrpcOperationLedger {
== 1;
}
+ @Override
+ public int purgeCompletedBefore(Instant completedBefore, int limit) {
+ if (completedBefore == null) {
+ throw new IllegalArgumentException("a purge needs a cutoff");
+ }
+ if (limit < 1) {
+ throw new IllegalArgumentException("a purge limit must be positive");
+ }
+ return repository.deleteFinishedBefore(completedBefore, limit);
+ }
+
@Override
public void release(GrpcOperationIdentity identity) {
requireIdentity(identity);
diff --git a/grpc/grpc-operation-ledger-jpa/src/main/resources/db/migration/grpc/V002__index_grpc_operation_ledger_finished_rows.sql b/grpc/grpc-operation-ledger-jpa/src/main/resources/db/migration/grpc/V002__index_grpc_operation_ledger_finished_rows.sql
new file mode 100644
index 0000000..a0c6d7d
--- /dev/null
+++ b/grpc/grpc-operation-ledger-jpa/src/main/resources/db/migration/grpc/V002__index_grpc_operation_ledger_finished_rows.sql
@@ -0,0 +1,16 @@
+-- Retention of finished ledger rows (review 2026-11-01, 03-10 follow-up).
+--
+-- GrpcOperationLedgerRepository.deleteFinishedBefore deletes COMMITTED and FAILED_TERMINAL rows
+-- whose completed_at is past the retention, oldest first and a bounded batch at a time. V001's
+-- (state, claimed_at) index orders by claim time, not completion time, so without this index every
+-- purge batch scans and sorts all finished rows.
+--
+-- Partial: IN_PROGRESS rows have no completed_at and are never purged, so they stay out of the
+-- index. The predicate repeats the query's state list literally so the planner can match it.
+--
+-- A plain CREATE INDEX, not CONCURRENTLY: Flyway runs each migration in a transaction, and the
+-- platform is not adopted anywhere yet, so the table is empty when this runs. An adopter that
+-- applies it to a large existing ledger should run the CONCURRENTLY form by hand first.
+CREATE INDEX ix_grpc_operation_ledger_finished_completed_at
+ ON grpc_operation_ledger (completed_at)
+ WHERE state IN ('COMMITTED', 'FAILED_TERMINAL');
diff --git a/grpc/grpc-operation-ledger-jpa/src/test/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepositoryTest.java b/grpc/grpc-operation-ledger-jpa/src/test/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepositoryTest.java
index 9b7fcb4..e3aada6 100644
--- a/grpc/grpc-operation-ledger-jpa/src/test/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepositoryTest.java
+++ b/grpc/grpc-operation-ledger-jpa/src/test/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepositoryTest.java
@@ -73,6 +73,12 @@ class GrpcOperationLedgerRepositoryTest {
"the stale takeover is certified against PostgreSQL by JpaGrpcOperationLedgerContractIT");
}
+ @Override
+ public int deleteFinishedBefore(Instant completedBefore, int limit) {
+ throw new UnsupportedOperationException(
+ "the bounded purge is certified against PostgreSQL by JpaGrpcOperationLedgerContractIT");
+ }
+
@Override
public GrpcOperationLedgerEntity save(GrpcOperationLedgerEntity entity) {
rows.put(entity.getStorageKey(), entity);
diff --git a/grpc/grpc-runtime/src/main/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetention.java b/grpc/grpc-runtime/src/main/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetention.java
new file mode 100644
index 0000000..ac4e22c
--- /dev/null
+++ b/grpc/grpc-runtime/src/main/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetention.java
@@ -0,0 +1,80 @@
+package dev.caskeleton.grpc.idempotency;
+
+import dev.caskeleton.grpc.api.ledger.GrpcOperationLedger;
+import java.time.Duration;
+import java.time.Instant;
+
+/**
+ * Deletes finished operation-ledger records once they are older than the retention.
+ *
+ * Without it the ledger keeps every {@code COMMITTED} and {@code FAILED_TERMINAL} row forever. A
+ * finished row exists only to answer a retry, so once it is older than any retry a caller still
+ * sends, it is dead weight on the table every keyed call writes to.
+ *
+ * This class decides what to delete; it does not decide when. Nothing here
+ * schedules itself: the platform starter does not own a scheduler, does not start the server, and
+ * cannot know which replica should run the purge, so the adopter calls {@link #purgeExpired} from
+ * its own scheduler (one replica, or every replica — the bounded delete is safe either way).
+ */
+public final class GrpcOperationLedgerRetention {
+
+ /**
+ * Rows deleted per statement. Small enough that one delete holds its row locks briefly on the
+ * hottest table of the idempotency path, large enough that a day's backlog drains in few calls.
+ */
+ public static final int DEFAULT_BATCH_SIZE = 500;
+
+ private final GrpcOperationLedger ledger;
+ private final Duration retention;
+ private final int batchSize;
+
+ /**
+ * @param retention how long a finished record is kept after it completed. It must exceed the
+ * longest window in which a caller still retries the same key: a retry after the purge finds
+ * no record, claims afresh and runs the mutation again.
+ * @param batchSize the most rows one delete statement removes
+ */
+ public GrpcOperationLedgerRetention(
+ GrpcOperationLedger ledger, Duration retention, int batchSize) {
+ if (ledger == null) {
+ throw new IllegalArgumentException("a ledger retention needs a ledger");
+ }
+ if (retention == null || retention.isZero() || retention.isNegative()) {
+ throw new IllegalArgumentException("a ledger retention must be present and positive");
+ }
+ if (batchSize < 1) {
+ throw new IllegalArgumentException("a ledger purge batch size must be positive");
+ }
+ this.ledger = ledger;
+ this.retention = retention;
+ this.batchSize = batchSize;
+ }
+
+ /** How long a finished record is kept. */
+ public Duration retention() {
+ return retention;
+ }
+
+ /**
+ * Deletes every finished record that completed more than the retention before {@code now}, in
+ * bounded batches, and never an {@code IN_PROGRESS} one.
+ *
+ * The cutoff is fixed at the start, so the loop ends: records finishing while it runs are
+ * newer than the cutoff and are left for the next call.
+ *
+ * @return how many records were deleted
+ */
+ public int purgeExpired(Instant now) {
+ if (now == null) {
+ throw new IllegalArgumentException("a purge needs the current moment");
+ }
+ Instant cutoff = now.minus(retention);
+ int total = 0;
+ int purged;
+ do {
+ purged = ledger.purgeCompletedBefore(cutoff, batchSize);
+ total += purged;
+ } while (purged == batchSize);
+ return total;
+ }
+}
diff --git a/grpc/grpc-runtime/src/test/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetentionTest.java b/grpc/grpc-runtime/src/test/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetentionTest.java
new file mode 100644
index 0000000..fca364a
--- /dev/null
+++ b/grpc/grpc-runtime/src/test/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetentionTest.java
@@ -0,0 +1,83 @@
+package dev.caskeleton.grpc.idempotency;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.catchThrowable;
+
+import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity;
+import dev.caskeleton.grpc.api.ledger.InMemoryOperationLedger;
+import dev.caskeleton.grpc.core.GrpcMethodName;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.List;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Test;
+
+class GrpcOperationLedgerRetentionTest {
+
+ private static final GrpcMethodName CREATE =
+ GrpcMethodName.parse("caskeleton.document.v1.DocumentService/CreateDocument");
+ private static final Instant NOW = Instant.parse("2026-11-01T10:00:00Z");
+ private static final Duration RETENTION = Duration.ofHours(24);
+ private static final String BODY = GrpcRequestFingerprint.of("body");
+
+ private final InMemoryOperationLedger ledger = new InMemoryOperationLedger();
+
+ @Test
+ @DisplayName("보존 기간이 지난 종결 행을 batch 크기와 무관하게 모두 지운다")
+ void everyFinishedRecordPastTheRetentionIsPurgedAcrossBatches() {
+ // given
+ List Built here rather than left to the adopter (review 2026-11-01, 03-11, option (b)): a flag
+ * that only satisfied the startup check while nothing honoured the key was a promise the platform
+ * did not keep. The ledger itself stays the deployment's bean — the starter does not depend on
+ * {@code grpc-operation-ledger-jpa} (11-M9) — and a missing one refuses startup.
+ *
+ * A duplicate of a running operation is refused rather than told to wait: keyed callers are
+ * usually user-facing requests that already retry with backoff. There is deliberately no
+ * {@code @ConditionalOnMissingBean}: a replacement bean would silently ignore {@code
+ * operation-ledger-claim-ttl}, which is the same false promise this bean exists to remove.
+ *
+ * The starter does not build a server, so nothing calls this yet: the adopter's {@code
+ * IDEMPOTENCY} stage interceptor ({@code GrpcServerInterceptorChain}) or service adapter calls
+ * {@code decide} before the mutation and {@code recordCommit} inside the mutation's transaction.
+ */
+ @Bean
+ @ConditionalOnProperty(
+ prefix = "app.grpc.platform",
+ name = "operation-ledger-enabled",
+ havingValue = "true")
+ public GrpcIdempotencyInterceptor grpcIdempotencyInterceptor(
+ GrpcPlatformProperties properties,
+ ObjectProvider Only the policy: nothing here schedules it. The starter owns no scheduler and does not know
+ * which replica should run the purge, so the adopter calls {@code purgeExpired(now)} from its own
+ * scheduler. A scheduled bean here was rejected because it would force a task scheduler onto
+ * every deployment and run on every replica whether or not that deployment wants it.
+ */
+ @Bean
+ @ConditionalOnProperty(
+ prefix = "app.grpc.platform",
+ name = "operation-ledger-enabled",
+ havingValue = "true")
+ public GrpcOperationLedgerRetention grpcOperationLedgerRetention(
+ GrpcPlatformProperties properties, ObjectProvider