diff --git a/docs/adr/ADR-GRPC-004-retry-ownership-and-durable-idempotency.md b/docs/adr/ADR-GRPC-004-retry-ownership-and-durable-idempotency.md index deb788b..495d348 100644 --- a/docs/adr/ADR-GRPC-004-retry-ownership-and-durable-idempotency.md +++ b/docs/adr/ADR-GRPC-004-retry-ownership-and-durable-idempotency.md @@ -53,8 +53,10 @@ request takes it over with `GrpcOperationLedger.reclaimStale` — one conditiona `UPDATE … WHERE state = 'IN_PROGRESS' AND claimed_at < :cutoff`, so of two concurrent takeovers exactly one wins, the same shape as `reclaimFailed`. The TTL is the `claimTtl` argument of `GrpcIdempotencyInterceptor`, refused at construction when it is not longer than every keyed method's -deadline. It is not a configuration key: the starter does not assemble the interceptor yet (03-11), so -there is nothing for a key to configure. +deadline. Since the 03-11 follow-up the starter assembles the interceptor when +`app.grpc.platform.operation-ledger-enabled=true` and takes the TTL from +`app.grpc.platform.operation-ledger-claim-ttl` (default 5m); a TTL the constructor refuses fails startup +naming that key, and an enabled flag without a `GrpcOperationLedger` bean fails startup too. The takeover is safe only because the commit is recorded in the mutation's own transaction: then a claim still `IN_PROGRESS` means the mutation never committed. An adopter that commits them separately @@ -65,7 +67,18 @@ method and not just its deadline. Rejected: an owner token with a heartbeat lease (the outbox lease model) — needed only when a keyed method can run longer than any fixed TTL; revisit when one exists. A sweeper job that deletes stale rows — it needs a scheduler this family does not have, and deleting a claim makes the next retry a -fresh claim with no record that an attempt was abandoned. Retention of terminal rows is still open. +fresh claim with no record that an attempt was abandoned. + +**Finished rows are purged after a retention; in-progress rows never are** (03-10 follow-up). +`GrpcOperationLedger.purgeCompletedBefore(cutoff, limit)` deletes at most `limit` `COMMITTED` or +`FAILED_TERMINAL` rows that completed before the cutoff, oldest first — on PostgreSQL one bounded +`DELETE … WHERE storage_key IN (SELECT … LIMIT … FOR UPDATE SKIP LOCKED)` served by the partial index +of migration V002. `GrpcOperationLedgerRetention.purgeExpired(now)` applies +`app.grpc.platform.operation-ledger-retention` (default 24h, the HTTP idempotency replay TTL) in +batches. A retry after its record was purged is a fresh claim and runs again, so the retention must +exceed the callers' retry window. The adopter schedules the purge; the starter assembles the policy +only, because a scheduled bean would force a scheduler onto every deployment and the starter cannot +know which replica should run it. **A key reused for a different request is a caller error, not a duplicate.** The stored request fingerprint turns that into `FAILED_PRECONDITION` rather than silently returning the first request's diff --git a/docs/runbooks/grpc-platform-operations.md b/docs/runbooks/grpc-platform-operations.md index d218f26..0b63212 100644 --- a/docs/runbooks/grpc-platform-operations.md +++ b/docs/runbooks/grpc-platform-operations.md @@ -39,8 +39,8 @@ the time, and which half is not knowable from the status. escalate to the service owner. This is the case the keyed profile exists to avoid. 7. `IN_PROGRESS` means a claim exists and has not finished. If its owner died, the claim is not - stuck forever: once it is older than the claim TTL (the `claimTtl` given to - `GrpcIdempotencyInterceptor`), the next retry of the same request takes it over and runs again, + stuck forever: once it is older than the claim TTL (`app.grpc.platform.operation-ledger-claim-ttl`, + default 5m), the next retry of the same request takes it over and runs again, and a later reconciler pass sees that attempt's `COMMITTED` or `FAILED_TERMINAL`. Until a retry arrives, the case stays pending. The re-run is safe only when the ledger commit is recorded in the mutation's own transaction; if this service commits them separately, check the business resource @@ -52,6 +52,32 @@ retry arriving is a caller that gave up; it needs the business resource checked, --- +## The operation ledger table keeps growing + +**What you are seeing.** `grpc_operation_ledger` grows without bound, or a retry of an old key runs +the mutation again instead of replaying. + +**What it means.** Finished rows (`COMMITTED`, `FAILED_TERMINAL`) are deleted only when something +calls `GrpcOperationLedgerRetention.purgeExpired(now)`; the starter assembles that bean +(`app.grpc.platform.operation-ledger-enabled=true`) but does not schedule it. A growing table means +the service's scheduler is not calling it. A retry that re-ran after the retention +(`app.grpc.platform.operation-ledger-retention`, default 24h) is the trade the retention makes: its +record was purged, so it was a new operation. + +**What to do.** + +1. Check that the service schedules `purgeExpired`. Each call deletes in batches of 500 until a batch + comes back short, never touching `IN_PROGRESS` rows. +2. For a large backlog, call it repeatedly; each batch is a short transaction that skips rows another + transaction holds. +3. If retried keys re-run after the retention, raise the retention above the callers' retry window + rather than skipping the purge. + +**Escalate when** old `IN_PROGRESS` rows accumulate. Those are not purged by design; each is an +abandoned claim no retry has taken over (see `IN_PROGRESS` above). + +--- + ## A stream ended with FULL_RESYNC_REQUIRED **What you are seeing.** A client's resume was refused and it was told to resynchronise. diff --git a/gradle.properties b/gradle.properties new file mode 100644 index 0000000..0d1f586 --- /dev/null +++ b/gradle.properties @@ -0,0 +1,9 @@ +# optional-platforms is a build of its own (`./gradlew -p optional-platforms ...`), and Gradle reads +# gradle.properties from the build's root directory only, so src/gradle.properties never reached it: +# the optional build ran serially and without the build cache (review 2026-11-01 08-06). +# The values and their reasons are src/gradle.properties'. settings.gradle fails the build when the +# two files disagree, so this is a copy that cannot drift, not a second source. +org.gradle.workers.max=4 +org.gradle.jvmargs=-Xmx1g -XX:MaxMetaspaceSize=512m -Dfile.encoding=UTF-8 +org.gradle.caching=true +org.gradle.parallel=true diff --git a/grpc/CLAUDE.md b/grpc/CLAUDE.md index f056c0a..ada4282 100644 --- a/grpc/CLAUDE.md +++ b/grpc/CLAUDE.md @@ -96,6 +96,9 @@ application-owned port → adapter:inbound:grpc (typed service adapter) → :grp 직접 닿지 않는 규칙은 app-bootstrap ArchUnit(`INBOUND_ADAPTERS_DO_NOT_DEPEND_ON_OUTBOUND_ADAPTERS`)이 검사한다. 현재 `adapter:inbound:grpc`는 이 family에 의존하지 않는다 — registry의 `allowed_dependencies`를 보라. 배선은 별도 결정이며, 그때 registry edge와 브리지가 함께 들어간다. +starter가 `operation-ledger-enabled=true`일 때 멱등성 판단 bean을 조립하는 것(03-11)은 이 배선이 아니다 — +starter는 서버를 만들지 않고, 그 bean을 서버에 붙이는 일은 채택하는 쪽의 몫이다(아래 「Durable operation +ledger adoption」). ## 테스트 명령 @@ -130,13 +133,51 @@ the repository's `.seconds` / `.total` unit naming and the corresponding validat ### Durable operation ledger adoption -`grpc-spring-boot-starter` does not pull the JPA ledger adapter transitively. A deployment that needs -durable operation idempotency adds `:grpc:grpc-operation-ledger-jpa` explicitly, enables the ledger -configuration, and includes that adapter's Flyway location `db/migration/grpc`. A deployment that -does not adopt the ledger therefore pays no Spring Data JPA classpath cost. +`grpc-spring-boot-starter` does not pull the JPA ledger adapter transitively (11-M9). A deployment that +needs durable operation idempotency adds `:grpc:grpc-operation-ledger-jpa` explicitly, defines a +`GrpcOperationLedger` bean (`new JpaGrpcOperationLedger(repository)`), includes that adapter's Flyway +location `db/migration/grpc` (V001 table, V002 retention index), and sets the flag below. A deployment +that does not adopt the ledger pays no Spring Data JPA classpath cost. -An `IN_PROGRESS` claim expires after the claim TTL (`GrpcIdempotencyInterceptor`'s `claimTtl` -argument; no configuration key while the starter does not assemble the interceptor). A retry of the -same request then takes it over through `GrpcOperationLedger.reclaimStale`. The adopter must record -the commit in the mutation's transaction and choose a TTL longer than the slowest keyed method; both -reasons are in ADR-GRPC-004. +| Key (`app.grpc.platform.*`) | Default | What it does | +| --- | --- | --- | +| `operation-ledger-enabled` | `false` | `true`: the starter assembles `GrpcIdempotencyInterceptor` (bean `grpcIdempotencyInterceptor`) and `GrpcOperationLedgerRetention` (bean `grpcOperationLedgerRetention`) over the deployment's `GrpcOperationLedger` bean, and refuses startup when there is none. `false`: neither bean exists, and a method that is `IDEMPOTENCY_KEY_REQUIRED` refuses startup | +| `operation-ledger-claim-ttl` | `5m` | How long an `IN_PROGRESS` claim is trusted to have a live owner. After it, a retry of the same request takes the claim over (`GrpcOperationLedger.reclaimStale`). Refused at startup unless longer than every keyed method's deadline | +| `operation-ledger-retention` | `24h` | How long a finished (`COMMITTED`/`FAILED_TERMINAL`) record is kept before `GrpcOperationLedgerRetention.purgeExpired` may delete it. `IN_PROGRESS` rows are never purged | + +Defaults: the 5-minute TTL is the value the ledger contract and interceptor tests certify, far beyond +a unary deadline, and short enough that a key whose owner died unblocks within a client's retry +patience. The 24-hour retention matches the HTTP idempotency replay TTL +(`app.capabilities.idempotency.replay-ttl`). A retry of a key after its record was purged is a new +operation, so the retention must exceed the callers' retry window. + +These keys are not in `docs/registries/env-keys.yaml`: no `app.grpc.platform.*` key is, because the +optional build is not part of the shipped runtime. They join the registry when a composition root +adopts the platform. + +**Why the starter assembles the interceptor (03-11 (b)), and not (a).** The review proposed (a): +delete the flag and judge by whether a ledger bean exists. The owner chose (b): the flag stays and +means what it says — when true, the decision function is assembled and its TTL comes from +configuration. (a) would have removed the false promise but left every adopter to rebuild the same +assembly and TTL validation by hand; (b) keeps that in one place. The interceptor bean has no +`@ConditionalOnMissingBean` on purpose: a replacement would silently ignore `operation-ledger-claim-ttl`. +Duplicates of a running operation are refused (`REJECT_IN_PROGRESS`), not told to wait; a +wait-and-poll option becomes a key when a keyed method needs it. + +**Server wiring is still the adopter's, and stays a separate decision.** The starter builds no +server, so assembling the decision does not wire the platform into `adapter:inbound:grpc` (see "배선은 +별도 결정" above). `GrpcIdempotencyInterceptor` is a decision function, not an `io.grpc.ServerInterceptor`. +The adopter injects the bean by type and calls it from its own `IDEMPOTENCY` stage interceptor +(`GrpcServerInterceptorChain.builder().stage(GrpcServerInterceptorStage.IDEMPOTENCY, ...)`) or from +its typed service adapter: `decide(...)` before the mutation, `recordCommit(...)` inside the mutation's +transaction, `recordFailure(...)` on a terminal failure. + +**Purge scheduling is the adopter's.** `GrpcOperationLedgerRetention.purgeExpired(now)` deletes +expired finished rows in bounded batches (500 per statement) until a batch comes back short. The +starter does not schedule it: it owns no scheduler and cannot know which replica should run it, and a +scheduled bean would force a task scheduler onto every deployment. The adopter calls it from its own +scheduler (for example hourly); concurrent runs on several replicas are safe because the delete skips +locked rows. Running it never is safe but lets the table grow without bound. + +The commit must be recorded in the mutation's transaction and the TTL must be longer than the slowest +keyed method; both reasons are in ADR-GRPC-004. diff --git a/grpc/grpc-core-api/src/main/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedger.java b/grpc/grpc-core-api/src/main/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedger.java index 7a4ee67..2c27515 100644 --- a/grpc/grpc-core-api/src/main/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedger.java +++ b/grpc/grpc-core-api/src/main/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedger.java @@ -82,6 +82,27 @@ public interface GrpcOperationLedger { Instant claimedBefore, Instant now); + /** + * Deletes up to {@code limit} finished records ({@code COMMITTED} or {@code FAILED_TERMINAL}) + * that completed before {@code completedBefore}, oldest first. + * + *

Retention, not recovery. A finished row only answers retries, so once it is older than any + * retry a caller will still send, keeping it only grows the table. Deleting it is a trade the + * deployment makes knowingly: a retry of the same key arriving after the purge is a fresh claim + * and runs the mutation again. The retention must therefore exceed the callers' retry window. + * + *

An {@code IN_PROGRESS} record is never deleted, however old. It is either running or + * abandoned, and an abandoned one is taken over by {@link #reclaimStale}, which keeps the record + * that an attempt started; deleting it would erase that. + * + *

Bounded so a backlog is drained in short transactions rather than one long delete that holds + * locks on the hottest table of the idempotency path. + * + * @return how many records were deleted, at most {@code limit} + * @throws IllegalArgumentException when {@code limit} is not positive + */ + int purgeCompletedBefore(Instant completedBefore, int limit); + /** * Releases a claim that will never complete, so the caller may try again. * diff --git a/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedgerContract.java b/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedgerContract.java index b2839d6..0c625fc 100644 --- a/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedgerContract.java +++ b/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedgerContract.java @@ -13,6 +13,7 @@ import java.util.concurrent.CountDownLatch; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.Future; +import java.util.stream.Stream; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; @@ -40,6 +41,10 @@ public abstract class GrpcOperationLedgerContract { // A retry arriving after the TTL: its cutoff (LATER - TTL) is past the original claim at NOW. private static final Instant LATER = NOW.plus(CLAIM_TTL).plusSeconds(1); private static final Instant CUTOFF = LATER.minus(CLAIM_TTL); + // A purge run a day after NOW: rows finished around NOW are past retention. + private static final Instant RETENTION_CUTOFF = NOW.plus(Duration.ofDays(1)); + private static final GrpcOperationIdentity SECOND_ID = identityWithKeyHash('b'); + private static final GrpcOperationIdentity THIRD_ID = identityWithKeyHash('c'); /** The ledger under test, holding no records when a test starts. */ protected abstract GrpcOperationLedger ledger(); @@ -397,9 +402,104 @@ public abstract class GrpcOperationLedgerContract { assertThat(refused).isInstanceOf(IllegalArgumentException.class).hasMessageContaining("cutoff"); } - private void failedClaim() { + @Test + @DisplayName("보존 기간이 지난 COMMITTED·FAILED_TERMINAL 행은 정리된다") + void finishedRecordsCompletedBeforeTheCutoffArePurged() { + // given + committedClaim(ID, NOW); + failedClaim(SECOND_ID, NOW); + + // when + int purged = ledger().purgeCompletedBefore(RETENTION_CUTOFF, 10); + + // then + assertThat(purged).isEqualTo(2); + assertThat(ledger().find(ID)).isEmpty(); + assertThat(ledger().find(SECOND_ID)).isEmpty(); + } + + @Test + @DisplayName("IN_PROGRESS 행은 아무리 오래되어도 정리하지 않는다") + void anInProgressClaimIsNeverPurgedHoweverOld() { + // given ledger().claim(ID, FINGERPRINT, NOW); - ledger().markFailed(ID, NOW.plusSeconds(1)); + + // when + int purged = ledger().purgeCompletedBefore(RETENTION_CUTOFF, 10); + + // then + assertThat(purged).isZero(); + assertThat(ledger().find(ID)) + .get() + .extracting(GrpcOperationLedgerRecord::state) + .isEqualTo(GrpcOperationLedgerState.IN_PROGRESS); + } + + @Test + @DisplayName("cutoff 이후에 끝난 행은 남긴다") + void finishedRecordsCompletedAtOrAfterTheCutoffAreKept() { + // given + committedClaim(ID, RETENTION_CUTOFF.minusSeconds(1)); + failedClaim(SECOND_ID, RETENTION_CUTOFF.minusSeconds(1)); + + // when + int purged = ledger().purgeCompletedBefore(RETENTION_CUTOFF, 10); + + // then + assertThat(purged).isZero(); + assertThat(ledger().find(ID)).isPresent(); + assertThat(ledger().find(SECOND_ID)).isPresent(); + } + + @Test + @DisplayName("한 번의 정리는 limit 개수까지만 지운다") + void onePurgeDeletesAtMostTheLimit() { + // given + committedClaim(ID, NOW); + committedClaim(SECOND_ID, NOW.plusSeconds(1)); + committedClaim(THIRD_ID, NOW.plusSeconds(2)); + + // when + int purged = ledger().purgeCompletedBefore(RETENTION_CUTOFF, 2); + + // then + assertThat(purged).isEqualTo(2); + assertThat( + Stream.of(ID, SECOND_ID, THIRD_ID) + .filter(identity -> ledger().find(identity).isPresent()) + .count()) + .isEqualTo(1); + } + + @Test + @DisplayName("limit이 양수가 아니면 정리를 거부한다") + void aNonPositivePurgeLimitIsRefused() { + // when + Throwable refused = catchThrowable(() -> ledger().purgeCompletedBefore(RETENTION_CUTOFF, 0)); + + // then + assertThat(refused).isInstanceOf(IllegalArgumentException.class).hasMessageContaining("limit"); + } + + private void failedClaim() { + failedClaim(ID, NOW); + } + + // Claimed at `claimedAt`, finished one second later, so the completion moment is what a purge + // cutoff is compared with. + private void failedClaim(GrpcOperationIdentity identity, Instant claimedAt) { + ledger().claim(identity, FINGERPRINT, claimedAt); + ledger().markFailed(identity, claimedAt.plusSeconds(1)); + } + + private void committedClaim(GrpcOperationIdentity identity, Instant claimedAt) { + ledger().claim(identity, FINGERPRINT, claimedAt); + ledger().markCommitted(identity, OUTCOME, claimedAt.plusSeconds(1)); + } + + private static GrpcOperationIdentity identityWithKeyHash(char hexDigit) { + return new GrpcOperationIdentity( + ID.callerFingerprint(), ID.method(), "sha256:" + String.valueOf(hexDigit).repeat(64)); } /** Runs {@code attempt} on two threads released together and returns both results. */ diff --git a/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/InMemoryOperationLedger.java b/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/InMemoryOperationLedger.java index 207971b..5547914 100644 --- a/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/InMemoryOperationLedger.java +++ b/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/InMemoryOperationLedger.java @@ -1,6 +1,8 @@ package dev.caskeleton.grpc.api.ledger; import java.time.Instant; +import java.util.Comparator; +import java.util.List; import java.util.Optional; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentMap; @@ -10,8 +12,8 @@ import java.util.concurrent.ConcurrentMap; * *

{@code putIfAbsent} on a concurrent map is the in-memory equivalent of the unique constraint * the real adapter relies on, and {@code replace(key, expected, next)} of its conditional update, - * so claim, re-claim and stale takeover are atomic here for the same reason they are there. The - * transitions match the JPA adapter's because {@link GrpcOperationLedgerContract} runs against + * so claim, re-claim, stale takeover and purge are atomic here for the same reason they are there. + * The transitions match the JPA adapter's because {@link GrpcOperationLedgerContract} runs against * both; a fake that allowed more would let the policy tests pass for code the database refuses. */ public final class InMemoryOperationLedger implements GrpcOperationLedger { @@ -102,6 +104,29 @@ public final class InMemoryOperationLedger implements GrpcOperationLedger { GrpcOperationLedgerRecord.claim(identity, requestFingerprint, now)); } + @Override + public int purgeCompletedBefore(Instant completedBefore, int limit) { + if (limit < 1) { + throw new IllegalArgumentException("a purge limit must be positive"); + } + List expired = + records.values().stream() + .filter(record -> record.state().terminal()) + .filter(record -> record.completedAt().orElseThrow().isBefore(completedBefore)) + .sorted(Comparator.comparing(record -> record.completedAt().orElseThrow())) + .limit(limit) + .toList(); + int purged = 0; + for (GrpcOperationLedgerRecord record : expired) { + // Conditional, like the database's re-checked DELETE: a row re-claimed since it was read is + // IN_PROGRESS again and stays. + if (records.remove(record.identity().storageKey(), record)) { + purged++; + } + } + return purged; + } + @Override public void release(GrpcOperationIdentity identity) { records.remove(identity.storageKey()); diff --git a/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerMigrations.java b/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerMigrations.java new file mode 100644 index 0000000..6e28dfa --- /dev/null +++ b/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerMigrations.java @@ -0,0 +1,37 @@ +package dev.caskeleton.grpc.ledger; + +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.util.List; + +/** + * The ledger's Flyway scripts in version order, as one SQL text for a test database. + * + *

One list for every PostgreSQL lane, so a new migration is applied by all of them or by none; + * two copies of the list drift the first time someone adds a V003 to one of them. + */ +final class GrpcOperationLedgerMigrations { + + private static final List IN_ORDER = + List.of( + "db/migration/grpc/V001__create_grpc_operation_ledger.sql", + "db/migration/grpc/V002__index_grpc_operation_ledger_finished_rows.sql"); + + private GrpcOperationLedgerMigrations() {} + + /** Every migration's SQL, concatenated in version order. */ + static String all() throws IOException { + StringBuilder sql = new StringBuilder(); + for (String path : IN_ORDER) { + try (InputStream stream = + GrpcOperationLedgerMigrations.class.getClassLoader().getResourceAsStream(path)) { + if (stream == null) { + throw new IOException("gRPC operation-ledger migration is missing: " + path); + } + sql.append(new String(stream.readAllBytes(), StandardCharsets.UTF_8)).append('\n'); + } + } + return sql.toString(); + } +} diff --git a/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerPostgresIT.java b/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerPostgresIT.java index 98fd2f2..df1b00b 100644 --- a/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerPostgresIT.java +++ b/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerPostgresIT.java @@ -3,8 +3,6 @@ package dev.caskeleton.grpc.ledger; import static org.assertj.core.api.Assertions.assertThat; import java.io.IOException; -import java.io.InputStream; -import java.nio.charset.StandardCharsets; import java.sql.Connection; import java.sql.PreparedStatement; import java.sql.ResultSet; @@ -41,7 +39,7 @@ class GrpcOperationLedgerPostgresIT { @SuppressWarnings("resource") @Container private static final PostgreSQLContainer POSTGRES = - new PostgreSQLContainer("postgres:16-alpine") + new PostgreSQLContainer(System.getProperty("ca.test.image.postgres-16")) .withDatabaseName("grpc") .withUsername("grpc") .withPassword("grpc"); @@ -146,14 +144,6 @@ class GrpcOperationLedgerPostgresIT { } private static String migration() throws IOException { - try (InputStream stream = - GrpcOperationLedgerPostgresIT.class - .getClassLoader() - .getResourceAsStream("db/migration/grpc/V001__create_grpc_operation_ledger.sql")) { - if (stream == null) { - throw new IOException("gRPC operation-ledger migration is missing"); - } - return new String(stream.readAllBytes(), StandardCharsets.UTF_8); - } + return GrpcOperationLedgerMigrations.all(); } } diff --git a/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedgerContractIT.java b/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedgerContractIT.java index 701b31b..7888f7e 100644 --- a/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedgerContractIT.java +++ b/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedgerContractIT.java @@ -11,8 +11,6 @@ import dev.caskeleton.grpc.core.GrpcMethodName; import jakarta.persistence.EntityManagerFactory; import jakarta.persistence.PersistenceConfiguration; import java.io.IOException; -import java.io.InputStream; -import java.nio.charset.StandardCharsets; import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; @@ -44,7 +42,7 @@ class JpaGrpcOperationLedgerContractIT extends GrpcOperationLedgerContract { @SuppressWarnings("resource") @Container private static final PostgreSQLContainer POSTGRES = - new PostgreSQLContainer("postgres:16-alpine") + new PostgreSQLContainer(System.getProperty("ca.test.image.postgres-16")) .withDatabaseName("grpc") .withUsername("grpc") .withPassword("grpc"); @@ -169,15 +167,7 @@ class JpaGrpcOperationLedgerContractIT extends GrpcOperationLedgerContract { } private static String migration() throws IOException { - try (InputStream stream = - JpaGrpcOperationLedgerContractIT.class - .getClassLoader() - .getResourceAsStream("db/migration/grpc/V001__create_grpc_operation_ledger.sql")) { - if (stream == null) { - throw new IOException("gRPC operation-ledger migration is missing"); - } - return new String(stream.readAllBytes(), StandardCharsets.UTF_8); - } + return GrpcOperationLedgerMigrations.all(); } /** Runs each call in its own transaction, as a caller's business transaction would. */ @@ -227,6 +217,13 @@ class JpaGrpcOperationLedgerContractIT extends GrpcOperationLedgerContract { status -> delegate.reclaimStale(identity, requestFingerprint, claimedBefore, now))); } + @Override + public int purgeCompletedBefore(Instant completedBefore, int limit) { + Integer purged = + transactions.execute(status -> delegate.purgeCompletedBefore(completedBefore, limit)); + return purged == null ? 0 : purged; + } + @Override public void release(GrpcOperationIdentity identity) { transactions.executeWithoutResult(status -> delegate.release(identity)); diff --git a/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepository.java b/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepository.java index 32d14c0..d48c61c 100644 --- a/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepository.java +++ b/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepository.java @@ -122,6 +122,39 @@ public interface GrpcOperationLedgerRepository @Param("claimedBefore") Instant claimedBefore, @Param("claimedAt") Instant claimedAt); + /** + * Deletes up to {@code limit} finished rows that completed before {@code completedBefore}, oldest + * first. Never an {@code IN_PROGRESS} row. + * + *

{@code FOR UPDATE SKIP LOCKED} in the bounded subquery keeps the purge out of the way of a + * concurrent {@link #tryReclaimFailed}: a row that re-claim holds is skipped, and one it already + * turned back into {@code IN_PROGRESS} no longer matches. The outer state condition says the same + * thing again so the statement cannot delete a live claim even if the subquery is edited. The + * partial index {@code ix_grpc_operation_ledger_finished_completed_at} (V002) serves the + * subquery's filter and order. + * + * @return how many rows were deleted + */ + @Modifying(flushAutomatically = true, clearAutomatically = true) + @Transactional + @Query( + value = + """ + DELETE FROM grpc_operation_ledger + WHERE storage_key IN ( + SELECT storage_key + FROM grpc_operation_ledger + WHERE state IN ('COMMITTED', 'FAILED_TERMINAL') + AND completed_at < :completedBefore + ORDER BY completed_at + LIMIT :limit + FOR UPDATE SKIP LOCKED) + AND state <> 'IN_PROGRESS' + """, + nativeQuery = true) + int deleteFinishedBefore( + @Param("completedBefore") Instant completedBefore, @Param("limit") int limit); + /** Updates an existing claim after it has been loaded. */ GrpcOperationLedgerEntity save(GrpcOperationLedgerEntity entity); diff --git a/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedger.java b/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedger.java index d81fd66..55d80bc 100644 --- a/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedger.java +++ b/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedger.java @@ -122,6 +122,17 @@ public final class JpaGrpcOperationLedger implements GrpcOperationLedger { == 1; } + @Override + public int purgeCompletedBefore(Instant completedBefore, int limit) { + if (completedBefore == null) { + throw new IllegalArgumentException("a purge needs a cutoff"); + } + if (limit < 1) { + throw new IllegalArgumentException("a purge limit must be positive"); + } + return repository.deleteFinishedBefore(completedBefore, limit); + } + @Override public void release(GrpcOperationIdentity identity) { requireIdentity(identity); diff --git a/grpc/grpc-operation-ledger-jpa/src/main/resources/db/migration/grpc/V002__index_grpc_operation_ledger_finished_rows.sql b/grpc/grpc-operation-ledger-jpa/src/main/resources/db/migration/grpc/V002__index_grpc_operation_ledger_finished_rows.sql new file mode 100644 index 0000000..a0c6d7d --- /dev/null +++ b/grpc/grpc-operation-ledger-jpa/src/main/resources/db/migration/grpc/V002__index_grpc_operation_ledger_finished_rows.sql @@ -0,0 +1,16 @@ +-- Retention of finished ledger rows (review 2026-11-01, 03-10 follow-up). +-- +-- GrpcOperationLedgerRepository.deleteFinishedBefore deletes COMMITTED and FAILED_TERMINAL rows +-- whose completed_at is past the retention, oldest first and a bounded batch at a time. V001's +-- (state, claimed_at) index orders by claim time, not completion time, so without this index every +-- purge batch scans and sorts all finished rows. +-- +-- Partial: IN_PROGRESS rows have no completed_at and are never purged, so they stay out of the +-- index. The predicate repeats the query's state list literally so the planner can match it. +-- +-- A plain CREATE INDEX, not CONCURRENTLY: Flyway runs each migration in a transaction, and the +-- platform is not adopted anywhere yet, so the table is empty when this runs. An adopter that +-- applies it to a large existing ledger should run the CONCURRENTLY form by hand first. +CREATE INDEX ix_grpc_operation_ledger_finished_completed_at + ON grpc_operation_ledger (completed_at) + WHERE state IN ('COMMITTED', 'FAILED_TERMINAL'); diff --git a/grpc/grpc-operation-ledger-jpa/src/test/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepositoryTest.java b/grpc/grpc-operation-ledger-jpa/src/test/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepositoryTest.java index 9b7fcb4..e3aada6 100644 --- a/grpc/grpc-operation-ledger-jpa/src/test/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepositoryTest.java +++ b/grpc/grpc-operation-ledger-jpa/src/test/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepositoryTest.java @@ -73,6 +73,12 @@ class GrpcOperationLedgerRepositoryTest { "the stale takeover is certified against PostgreSQL by JpaGrpcOperationLedgerContractIT"); } + @Override + public int deleteFinishedBefore(Instant completedBefore, int limit) { + throw new UnsupportedOperationException( + "the bounded purge is certified against PostgreSQL by JpaGrpcOperationLedgerContractIT"); + } + @Override public GrpcOperationLedgerEntity save(GrpcOperationLedgerEntity entity) { rows.put(entity.getStorageKey(), entity); diff --git a/grpc/grpc-runtime/src/main/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetention.java b/grpc/grpc-runtime/src/main/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetention.java new file mode 100644 index 0000000..ac4e22c --- /dev/null +++ b/grpc/grpc-runtime/src/main/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetention.java @@ -0,0 +1,80 @@ +package dev.caskeleton.grpc.idempotency; + +import dev.caskeleton.grpc.api.ledger.GrpcOperationLedger; +import java.time.Duration; +import java.time.Instant; + +/** + * Deletes finished operation-ledger records once they are older than the retention. + * + *

Without it the ledger keeps every {@code COMMITTED} and {@code FAILED_TERMINAL} row forever. A + * finished row exists only to answer a retry, so once it is older than any retry a caller still + * sends, it is dead weight on the table every keyed call writes to. + * + *

This class decides what to delete; it does not decide when. Nothing here + * schedules itself: the platform starter does not own a scheduler, does not start the server, and + * cannot know which replica should run the purge, so the adopter calls {@link #purgeExpired} from + * its own scheduler (one replica, or every replica — the bounded delete is safe either way). + */ +public final class GrpcOperationLedgerRetention { + + /** + * Rows deleted per statement. Small enough that one delete holds its row locks briefly on the + * hottest table of the idempotency path, large enough that a day's backlog drains in few calls. + */ + public static final int DEFAULT_BATCH_SIZE = 500; + + private final GrpcOperationLedger ledger; + private final Duration retention; + private final int batchSize; + + /** + * @param retention how long a finished record is kept after it completed. It must exceed the + * longest window in which a caller still retries the same key: a retry after the purge finds + * no record, claims afresh and runs the mutation again. + * @param batchSize the most rows one delete statement removes + */ + public GrpcOperationLedgerRetention( + GrpcOperationLedger ledger, Duration retention, int batchSize) { + if (ledger == null) { + throw new IllegalArgumentException("a ledger retention needs a ledger"); + } + if (retention == null || retention.isZero() || retention.isNegative()) { + throw new IllegalArgumentException("a ledger retention must be present and positive"); + } + if (batchSize < 1) { + throw new IllegalArgumentException("a ledger purge batch size must be positive"); + } + this.ledger = ledger; + this.retention = retention; + this.batchSize = batchSize; + } + + /** How long a finished record is kept. */ + public Duration retention() { + return retention; + } + + /** + * Deletes every finished record that completed more than the retention before {@code now}, in + * bounded batches, and never an {@code IN_PROGRESS} one. + * + *

The cutoff is fixed at the start, so the loop ends: records finishing while it runs are + * newer than the cutoff and are left for the next call. + * + * @return how many records were deleted + */ + public int purgeExpired(Instant now) { + if (now == null) { + throw new IllegalArgumentException("a purge needs the current moment"); + } + Instant cutoff = now.minus(retention); + int total = 0; + int purged; + do { + purged = ledger.purgeCompletedBefore(cutoff, batchSize); + total += purged; + } while (purged == batchSize); + return total; + } +} diff --git a/grpc/grpc-runtime/src/test/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetentionTest.java b/grpc/grpc-runtime/src/test/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetentionTest.java new file mode 100644 index 0000000..fca364a --- /dev/null +++ b/grpc/grpc-runtime/src/test/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetentionTest.java @@ -0,0 +1,83 @@ +package dev.caskeleton.grpc.idempotency; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.catchThrowable; + +import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity; +import dev.caskeleton.grpc.api.ledger.InMemoryOperationLedger; +import dev.caskeleton.grpc.core.GrpcMethodName; +import java.time.Duration; +import java.time.Instant; +import java.util.List; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +class GrpcOperationLedgerRetentionTest { + + private static final GrpcMethodName CREATE = + GrpcMethodName.parse("caskeleton.document.v1.DocumentService/CreateDocument"); + private static final Instant NOW = Instant.parse("2026-11-01T10:00:00Z"); + private static final Duration RETENTION = Duration.ofHours(24); + private static final String BODY = GrpcRequestFingerprint.of("body"); + + private final InMemoryOperationLedger ledger = new InMemoryOperationLedger(); + + @Test + @DisplayName("보존 기간이 지난 종결 행을 batch 크기와 무관하게 모두 지운다") + void everyFinishedRecordPastTheRetentionIsPurgedAcrossBatches() { + // given + List old = List.of(identity("k1"), identity("k2"), identity("k3")); + old.forEach(identity -> committed(identity, NOW.minus(RETENTION).minusSeconds(60))); + GrpcOperationLedgerRetention retention = new GrpcOperationLedgerRetention(ledger, RETENTION, 2); + + // when + int purged = retention.purgeExpired(NOW); + + // then + assertThat(purged).isEqualTo(3); + assertThat(old).allSatisfy(identity -> assertThat(ledger.find(identity)).isEmpty()); + } + + @Test + @DisplayName("보존 기간 안에 끝난 행과 진행 중인 행은 남긴다") + void recentFinishedRecordsAndInProgressClaimsAreKept() { + // given + GrpcOperationIdentity recent = identity("recent"); + GrpcOperationIdentity running = identity("running"); + committed(recent, NOW.minus(RETENTION).plusSeconds(60)); + ledger.claim(running, BODY, NOW.minus(Duration.ofDays(30))); + GrpcOperationLedgerRetention retention = new GrpcOperationLedgerRetention(ledger, RETENTION, 2); + + // when + int purged = retention.purgeExpired(NOW); + + // then + assertThat(purged).isZero(); + assertThat(ledger.find(recent)).isPresent(); + assertThat(ledger.find(running)).isPresent(); + } + + @Test + @DisplayName("보존 기간이 양수가 아니면 만들 수 없다") + void aNonPositiveRetentionIsRefused() { + // when + Throwable refused = + catchThrowable(() -> new GrpcOperationLedgerRetention(ledger, Duration.ZERO, 2)); + + // then + assertThat(refused) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("retention must be present and positive"); + } + + private static GrpcOperationIdentity identity(String key) { + return new GrpcOperationIdentity( + "tenant-1|actor-1", CREATE, GrpcRequestFingerprint.hashIdempotencyKey(key)); + } + + // Finished at the claim moment; the completion moment is what the retention measures. + private void committed(GrpcOperationIdentity identity, Instant claimedAt) { + ledger.claim(identity, BODY, claimedAt); + ledger.markCommitted(identity, "outcome://document/" + identity.storageKey(), claimedAt); + } +} diff --git a/grpc/grpc-spring-boot-starter/build.gradle b/grpc/grpc-spring-boot-starter/build.gradle index ad114e6..2a8fbfa 100644 --- a/grpc/grpc-spring-boot-starter/build.gradle +++ b/grpc/grpc-spring-boot-starter/build.gradle @@ -18,4 +18,6 @@ dependencies { implementation 'org.springframework.boot:spring-boot-autoconfigure' + // The shared operation-ledger fake stands in for the adopter's ledger bean in context tests. + testImplementation testFixtures(project(':grpc:grpc-core-api')) } diff --git a/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformAutoConfiguration.java b/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformAutoConfiguration.java index 8516939..02eddca 100644 --- a/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformAutoConfiguration.java +++ b/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformAutoConfiguration.java @@ -5,15 +5,20 @@ import dev.caskeleton.grpc.admin.GrpcDrainPolicy; import dev.caskeleton.grpc.admin.GrpcHealthPolicy; import dev.caskeleton.grpc.admin.GrpcReflectionPolicy; import dev.caskeleton.grpc.admin.GrpcServiceHealthRegistry; +import dev.caskeleton.grpc.api.ledger.GrpcOperationLedger; import dev.caskeleton.grpc.api.policy.GrpcMethodPolicyCatalog; import dev.caskeleton.grpc.client.GrpcNamedChannelProfile; import dev.caskeleton.grpc.context.GrpcContextBinder; import dev.caskeleton.grpc.context.GrpcContextPropagationPolicy; import dev.caskeleton.grpc.error.GrpcErrorMapper; +import dev.caskeleton.grpc.idempotency.GrpcIdempotencyInterceptor; +import dev.caskeleton.grpc.idempotency.GrpcOperationLedgerRetention; import dev.caskeleton.grpc.security.GrpcTlsProfile; import dev.caskeleton.grpc.server.GrpcAdmissionController; import dev.caskeleton.grpc.server.GrpcExecutorProfile; import dev.caskeleton.grpc.server.GrpcServerProfile; +import java.time.Duration; +import java.util.List; import java.util.UUID; import org.springframework.beans.factory.ObjectProvider; import org.springframework.beans.factory.SmartInitializingSingleton; @@ -131,6 +136,89 @@ public class GrpcPlatformAutoConfiguration { serverProfile); } + /** + * The keyed-idempotency decision, assembled when {@code operation-ledger-enabled=true}. + * + *

Built here rather than left to the adopter (review 2026-11-01, 03-11, option (b)): a flag + * that only satisfied the startup check while nothing honoured the key was a promise the platform + * did not keep. The ledger itself stays the deployment's bean — the starter does not depend on + * {@code grpc-operation-ledger-jpa} (11-M9) — and a missing one refuses startup. + * + *

A duplicate of a running operation is refused rather than told to wait: keyed callers are + * usually user-facing requests that already retry with backoff. There is deliberately no + * {@code @ConditionalOnMissingBean}: a replacement bean would silently ignore {@code + * operation-ledger-claim-ttl}, which is the same false promise this bean exists to remove. + * + *

The starter does not build a server, so nothing calls this yet: the adopter's {@code + * IDEMPOTENCY} stage interceptor ({@code GrpcServerInterceptorChain}) or service adapter calls + * {@code decide} before the mutation and {@code recordCommit} inside the mutation's transaction. + */ + @Bean + @ConditionalOnProperty( + prefix = "app.grpc.platform", + name = "operation-ledger-enabled", + havingValue = "true") + public GrpcIdempotencyInterceptor grpcIdempotencyInterceptor( + GrpcPlatformProperties properties, + ObjectProvider ledgers, + ObjectProvider catalogs) { + GrpcOperationLedger ledger = requireLedger(ledgers); + try { + return new GrpcIdempotencyInterceptor( + ledger, + catalogs.getIfAvailable(() -> GrpcMethodPolicyCatalog.builder().build()), + Duration.ZERO, + false, + properties.getOperationLedgerClaimTtl()); + } catch (IllegalArgumentException invalidTtl) { + // The ledger and catalog are present and the poll interval is fixed, so the claim TTL is the + // only input left for the constructor to refuse. + throw new GrpcPlatformConfigurationException( + List.of("app.grpc.platform.operation-ledger-claim-ttl: " + invalidTtl.getMessage())); + } + } + + /** + * Retention of finished ledger records, assembled with the interceptor. + * + *

Only the policy: nothing here schedules it. The starter owns no scheduler and does not know + * which replica should run the purge, so the adopter calls {@code purgeExpired(now)} from its own + * scheduler. A scheduled bean here was rejected because it would force a task scheduler onto + * every deployment and run on every replica whether or not that deployment wants it. + */ + @Bean + @ConditionalOnProperty( + prefix = "app.grpc.platform", + name = "operation-ledger-enabled", + havingValue = "true") + public GrpcOperationLedgerRetention grpcOperationLedgerRetention( + GrpcPlatformProperties properties, ObjectProvider ledgers) { + GrpcOperationLedger ledger = requireLedger(ledgers); + try { + return new GrpcOperationLedgerRetention( + ledger, + properties.getOperationLedgerRetention(), + GrpcOperationLedgerRetention.DEFAULT_BATCH_SIZE); + } catch (IllegalArgumentException invalidRetention) { + throw new GrpcPlatformConfigurationException( + List.of( + "app.grpc.platform.operation-ledger-retention: " + invalidRetention.getMessage())); + } + } + + private static GrpcOperationLedger requireLedger(ObjectProvider ledgers) { + GrpcOperationLedger ledger = ledgers.getIfAvailable(); + if (ledger == null) { + throw new GrpcPlatformConfigurationException( + List.of( + "app.grpc.platform.operation-ledger-enabled=true but there is no GrpcOperationLedger" + + " bean; the starter does not bring one. Add :grpc:grpc-operation-ledger-jpa and" + + " define a JpaGrpcOperationLedger bean (with the db/migration/grpc Flyway" + + " location), or turn the flag off")); + } + return ledger; + } + /** The shared error mapper. Its execution ids are random and carry no derived content. */ @Bean @ConditionalOnMissingBean diff --git a/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformProperties.java b/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformProperties.java index c480573..3d3a73b 100644 --- a/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformProperties.java +++ b/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformProperties.java @@ -3,6 +3,7 @@ package dev.caskeleton.grpc.boot; import dev.caskeleton.grpc.admin.GrpcReflectionMode; import dev.caskeleton.grpc.security.GrpcTlsProfile; import dev.caskeleton.grpc.server.GrpcServerTransport; +import java.time.Duration; import org.springframework.boot.context.properties.ConfigurationProperties; /** @@ -49,9 +50,32 @@ public class GrpcPlatformProperties { /** The bounded executor's maximum size. */ private int executorMaxPoolSize = 32; - /** Whether the operation ledger is available, which gates keyed idempotency. */ + /** + * Whether keyed idempotency runs. When true the starter assembles {@code + * GrpcIdempotencyInterceptor} and {@code GrpcOperationLedgerRetention} over the deployment's + * {@code GrpcOperationLedger} bean, and refuses to start without one; the starter never brings a + * ledger implementation itself. When false, a method that requires an idempotency key is refused + * at startup, because the key would be accepted and never honoured. + */ private boolean operationLedgerEnabled; + /** + * How long an {@code IN_PROGRESS} claim is trusted to have a live owner before a retry of the + * same request takes it over. Five minutes: far beyond a unary deadline (the interceptor refuses + * a TTL not longer than every keyed method's deadline), so a slow but live owner is not + * overtaken, and short enough that a key whose owner died unblocks within a client's normal retry + * patience. It is also the value the ledger contract and interceptor tests certify. + */ + private Duration operationLedgerClaimTtl = Duration.ofMinutes(5); + + /** + * How long a finished ({@code COMMITTED} or {@code FAILED_TERMINAL}) ledger record is kept before + * {@code GrpcOperationLedgerRetention} may delete it. A retry of the same key after that is a new + * operation, so this must exceed the callers' retry window. 24 hours, matching the HTTP + * idempotency replay TTL ({@code app.capabilities.idempotency.replay-ttl}). + */ + private Duration operationLedgerRetention = Duration.ofHours(24); + public boolean isEnabled() { return enabled; } @@ -124,6 +148,22 @@ public class GrpcPlatformProperties { this.operationLedgerEnabled = operationLedgerEnabled; } + public Duration getOperationLedgerClaimTtl() { + return operationLedgerClaimTtl; + } + + public void setOperationLedgerClaimTtl(Duration operationLedgerClaimTtl) { + this.operationLedgerClaimTtl = operationLedgerClaimTtl; + } + + public Duration getOperationLedgerRetention() { + return operationLedgerRetention; + } + + public void setOperationLedgerRetention(Duration operationLedgerRetention) { + this.operationLedgerRetention = operationLedgerRetention; + } + /** * The reflection mode this deployment runs, defaulting to the environment's floor, and to * disabled while the environment is unset (the startup check refuses that state). diff --git a/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformStartupValidator.java b/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformStartupValidator.java index 1f1a4de..a289278 100644 --- a/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformStartupValidator.java +++ b/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformStartupValidator.java @@ -151,8 +151,8 @@ public final class GrpcPlatformStartupValidator { violations.add( "method '" + method.canonical() - + "' requires an idempotency key but no operation ledger is enabled; the key would " - + "be accepted and never honoured"); + + "' requires an idempotency key but app.grpc.platform.operation-ledger-enabled is" + + " false; the key would be accepted and never honoured"); } if (!policy.rpcType().stable()) { violations.add( diff --git a/grpc/grpc-spring-boot-starter/src/test/java/dev/caskeleton/grpc/boot/GrpcPlatformAutoConfigurationContextTest.java b/grpc/grpc-spring-boot-starter/src/test/java/dev/caskeleton/grpc/boot/GrpcPlatformAutoConfigurationContextTest.java index 6957096..a07b65b 100644 --- a/grpc/grpc-spring-boot-starter/src/test/java/dev/caskeleton/grpc/boot/GrpcPlatformAutoConfigurationContextTest.java +++ b/grpc/grpc-spring-boot-starter/src/test/java/dev/caskeleton/grpc/boot/GrpcPlatformAutoConfigurationContextTest.java @@ -4,13 +4,41 @@ import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; import static org.assertj.core.api.Assertions.catchThrowable; +import dev.caskeleton.grpc.api.deadline.GrpcDeadlineProfile; +import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity; +import dev.caskeleton.grpc.api.ledger.GrpcOperationLedger; +import dev.caskeleton.grpc.api.ledger.InMemoryOperationLedger; +import dev.caskeleton.grpc.api.policy.GrpcMethodPolicy; +import dev.caskeleton.grpc.api.policy.GrpcMethodPolicyCatalog; +import dev.caskeleton.grpc.api.policy.RpcIdempotencyProfile; +import dev.caskeleton.grpc.api.policy.WaitForReadyPolicy; +import dev.caskeleton.grpc.core.GrpcMethodName; +import dev.caskeleton.grpc.core.RpcType; +import dev.caskeleton.grpc.idempotency.GrpcIdempotencyDecision; +import dev.caskeleton.grpc.idempotency.GrpcIdempotencyInterceptor; +import dev.caskeleton.grpc.idempotency.GrpcOperationLedgerRetention; +import dev.caskeleton.grpc.idempotency.GrpcRequestFingerprint; +import java.time.Duration; +import java.time.Instant; +import java.util.HashMap; import java.util.Map; +import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; import org.springframework.context.annotation.AnnotationConfigApplicationContext; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.context.annotation.Import; import org.springframework.core.env.MapPropertySource; class GrpcPlatformAutoConfigurationContextTest { + private static final GrpcMethodName CREATE = + GrpcMethodName.parse("caskeleton.document.v1.DocumentService/CreateDocument"); + private static final Duration KEYED_DEADLINE = Duration.ofSeconds(10); + private static final String CALLER = "tenant-1|actor-1"; + private static final String KEY = "order-2026-0001"; + private static final Instant NOW = Instant.parse("2026-11-01T10:00:00Z"); + @Test void invalidProductionSecurityFailsDuringActualAutoConfigurationStartup() { assertThatThrownBy( @@ -77,12 +105,169 @@ class GrpcPlatformAutoConfigurationContextTest { } } - private static AnnotationConfigApplicationContext start(Map properties) { + @Test + @DisplayName("ledger 플래그가 켜져 있고 ledger bean이 있으면 멱등성 interceptor가 조립된다") + void anEnabledLedgerWithALedgerBeanAssemblesTheIdempotencyInterceptor() { + // when + try (AnnotationConfigApplicationContext context = + start(ledgerEnabled(Map.of()), KeyedMethodWithLedger.class)) { + + // then + assertThat(context.getBeansOfType(GrpcIdempotencyInterceptor.class)).hasSize(1); + } + } + + @Test + @DisplayName("ledger 플래그가 켜져 있는데 ledger bean이 없으면 기동을 거부한다") + void anEnabledLedgerWithoutALedgerBeanRefusesToStart() { + // when + Throwable failure = catchThrowable(() -> start(ledgerEnabled(Map.of()), KeyedMethod.class)); + + // then + assertThat(failure) + .rootCause() + .isInstanceOf(GrpcPlatformConfigurationException.class) + .hasMessageContaining("app.grpc.platform.operation-ledger-enabled=true") + .hasMessageContaining("no GrpcOperationLedger bean"); + } + + @Test + @DisplayName("ledger 플래그가 꺼져 있으면 멱등성 interceptor와 보존 정리를 만들지 않는다") + void aDisabledLedgerAssemblesNoIdempotencyInterceptorOrRetention() { + // when + try (AnnotationConfigApplicationContext context = + start(localPlatform(Map.of()), LedgerOnly.class)) { + + // then + assertThat(context.getBeansOfType(GrpcIdempotencyInterceptor.class)).isEmpty(); + assertThat(context.getBeansOfType(GrpcOperationLedgerRetention.class)).isEmpty(); + } + } + + @Test + @DisplayName("claim-ttl 설정이 interceptor의 stale claim 판단에 쓰인다") + void theClaimTtlKeyDecidesWhenAnInProgressClaimIsTakenOver() { + // given + try (AnnotationConfigApplicationContext context = + start( + ledgerEnabled(Map.of("app.grpc.platform.operation-ledger-claim-ttl", "1m")), + KeyedMethodWithLedger.class)) { + GrpcIdempotencyInterceptor interceptor = context.getBean(GrpcIdempotencyInterceptor.class); + String body = GrpcRequestFingerprint.of("body"); + interceptor.decide(CREATE, CALLER, KEY, body, NOW); + + // when + GrpcIdempotencyDecision retry = + interceptor.decide(CREATE, CALLER, KEY, body, NOW.plus(Duration.ofSeconds(61))); + + // then + assertThat(retry.outcome()).isEqualTo(GrpcIdempotencyDecision.Outcome.PROCEED); + } + } + + @Test + @DisplayName("claim-ttl이 keyed method의 deadline보다 길지 않으면 기동을 거부한다") + void aClaimTtlWithinAKeyedDeadlineRefusesToStart() { + // when + Throwable failure = + catchThrowable( + () -> + start( + ledgerEnabled(Map.of("app.grpc.platform.operation-ledger-claim-ttl", "10s")), + KeyedMethodWithLedger.class)); + + // then + assertThat(failure) + .rootCause() + .isInstanceOf(GrpcPlatformConfigurationException.class) + .hasMessageContaining("app.grpc.platform.operation-ledger-claim-ttl") + .hasMessageContaining("CreateDocument"); + } + + @Test + @DisplayName("retention 설정이 지난 종결 행을 보존 정리가 지운다") + void theRetentionKeyDecidesWhichFinishedRecordsArePurged() { + // given + try (AnnotationConfigApplicationContext context = + start( + ledgerEnabled(Map.of("app.grpc.platform.operation-ledger-retention", "1h")), + KeyedMethodWithLedger.class)) { + GrpcOperationLedger ledger = context.getBean(GrpcOperationLedger.class); + GrpcOperationIdentity identity = + new GrpcOperationIdentity(CALLER, CREATE, GrpcRequestFingerprint.hashIdempotencyKey(KEY)); + ledger.claim(identity, GrpcRequestFingerprint.of("body"), NOW); + ledger.markCommitted(identity, "outcome://document/42", NOW.plusSeconds(1)); + + // when + int purged = + context + .getBean(GrpcOperationLedgerRetention.class) + .purgeExpired(NOW.plus(Duration.ofHours(2))); + + // then + assertThat(purged).isEqualTo(1); + assertThat(ledger.find(identity)).isEmpty(); + } + } + + private static Map localPlatform(Map extra) { + Map properties = new HashMap<>(); + properties.put("app.grpc.platform.enabled", "true"); + properties.put("app.grpc.platform.environment", "LOCAL"); + properties.put("app.grpc.platform.transport", "NETTY_SHADED"); + properties.putAll(extra); + return properties; + } + + private static Map ledgerEnabled(Map extra) { + Map properties = new HashMap<>(extra); + properties.put("app.grpc.platform.operation-ledger-enabled", "true"); + return localPlatform(properties); + } + + /** One keyed method with a 10-second deadline, the shape the claim TTL is measured against. */ + @Configuration(proxyBeanMethods = false) + static class KeyedMethod { + @Bean + GrpcMethodPolicyCatalog keyedCatalog() { + return GrpcMethodPolicyCatalog.builder() + .register( + new GrpcMethodPolicy( + CREATE, + RpcType.UNARY, + RpcIdempotencyProfile.IDEMPOTENCY_KEY_REQUIRED, + GrpcDeadlineProfile.of(KEYED_DEADLINE), + WaitForReadyPolicy.DISABLED, + true, + 1024, + 1024)) + .build(); + } + } + + /** The adopter's ledger, here the shared in-memory fake. */ + @Configuration(proxyBeanMethods = false) + static class LedgerOnly { + @Bean + GrpcOperationLedger operationLedger() { + return new InMemoryOperationLedger(); + } + } + + @Configuration(proxyBeanMethods = false) + @Import({KeyedMethod.class, LedgerOnly.class}) + static class KeyedMethodWithLedger {} + + private static AnnotationConfigApplicationContext start( + Map properties, Class... adopterConfigurations) { AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(); context .getEnvironment() .getPropertySources() .addFirst(new MapPropertySource("grpc-platform-test", properties)); + if (adopterConfigurations.length > 0) { + context.register(adopterConfigurations); + } context.register(GrpcPlatformAutoConfiguration.class); try { context.refresh(); diff --git a/settings.gradle b/settings.gradle index cf08c10..a34cfb8 100644 --- a/settings.gradle +++ b/settings.gradle @@ -19,3 +19,18 @@ dependencyResolutionManagement { } rootProject.name = 'optional-platforms' + +// gradle.properties here must say what src/gradle.properties says (review 2026-11-01 08-06): Gradle +// reads only this build's own file, and a copy that drifted would make the optional build quietly +// slower or differently configured than the main one. +def readBuildProperties = { File file -> + def properties = new Properties() + file.withInputStream { properties.load(it) } + properties +} +def mainBuildProperties = readBuildProperties(new File(settingsDir, '../gradle.properties')) +def ownBuildProperties = readBuildProperties(new File(settingsDir, 'gradle.properties')) +if (mainBuildProperties != ownBuildProperties) { + throw new GradleException("optional-platforms/gradle.properties differs from src/gradle.properties: " + + "${ownBuildProperties} vs ${mainBuildProperties}. Copy the main build's values.") +}