From cd4e11724737ad7e69357e530717bbd1cbb0fae7 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Tue, 3 Nov 2026 19:52:06 +0900 Subject: [PATCH] =?UTF-8?q?feat!:=20=EB=82=A8=EC=9D=80=20=EB=A6=AC?= =?UTF-8?q?=EB=B7=B0=20=EB=B0=9C=EA=B2=AC=EC=9D=84=20=EA=B3=A0=EC=B9=98?= =?UTF-8?q?=EA=B3=A0=20=EA=B0=9C=EB=B0=9C=20=EB=A7=A4=EB=89=B4=EC=96=BC?= =?UTF-8?q?=EA=B3=BC=20=EA=B8=B0=EB=8A=A5=20=EC=83=9D=EC=84=B1=20=EB=AA=85?= =?UTF-8?q?=EB=A0=B9=EC=9D=84=20=EB=8D=94=ED=95=9C=EB=8B=A4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 문제: High·P1을 고친 뒤에도 P2·P3 약 90건이 남았다. 연결·구독·메시지 크기 상한이 없는 전송(WebSocket·gRPC·GraphQL deadline), 낙관적 충돌의 500, upsert 경쟁, Mongo 무제한 timeout, 샘플 목록 N+1, 갈라진 복사본 (TenantId 3벌, 이벤트 ID 문법 5곳, SQLState 표 2벌), 문서를 읽는 PR 테스트와 캐시를 막는 lane 설정 등이다. 새 기능을 어디에 어떤 순서로 만드는지 알려 주는 문서와 도구가 없었다. MinIO 공식 이미지가 사라져 객체 저장소 lane이 돌지 않았다. 방법: 영역별 묶음으로 실패 테스트를 먼저 쓰고 고쳤다. 고치는 중에 찾은 결함도 함께 고쳤다(durable operation id가 UUIDv4라 이벤트가 전부 DEAD, S3 어댑터 생성·작은 chunk 읽기 실패, Redis 조건부 기록의 EVALSHA 권한). MinIO는 pgsty/minio를 digest로 고정했다(Chainguard는 태그·실행 사용자· mc 이미지 문제로 버림). docs/manual/에 작업 순서와 판단 지점을 쓰고, scripts/new-feature.sh가 5개 leaf에 뼈대를 만든다. 생성→빌드→통과 왕복을 verifyFeatureScaffold가 mainCheck에서 본다. 묶음별 결과와 결정은 리뷰 README 14절에 있다. 위험·한계: 파생 서비스 동작이 바뀐다. JDBC batch 50 기본, 운영 jar에서 H2 제거, 트랜잭션 30초 제한, 격리 수준을 Hikari에서 고정, gRPC· WebSocket·GraphQL 새 한도, correlation ID 규칙 밖이면 서버가 교체, 샘플 키 오류 코드 IDEMPOTENCY_KEY_INVALID. 새 leaf 두 개 (composition:durable-operation, adapter:inbound:messaging)와 새 마이그레이션(postgresql V20·V21, notification V11·V12, sample V8_1, grpc-ledger V002)이 생긴다. V21 유일 인덱스는 기존 중복이 있으면 실패한다(runbook에 확인 쿼리). 검증: ci, mainCheck(생성 명령 왕복 포함), integrationCheck(MinIO 포함 전 컨테이너 lane), -p optional-platforms ci integrationCheck 통과. Refs: docs/reviews/2026-11-01-maturity-review/README.md 14절 BREAKING CHANGE: JDBC batch 기본 켜짐, H2 운영 의존 제거, 트랜잭션 기본 제한 30초, gRPC·WebSocket·GraphQL 서버 한도 기본값 추가, 샘플 Idempotency-Key 오류 코드 변경. --- ...retry-ownership-and-durable-idempotency.md | 19 +- docs/runbooks/grpc-platform-operations.md | 30 ++- gradle.properties | 9 + grpc/CLAUDE.md | 59 +++++- .../grpc/api/ledger/GrpcOperationLedger.java | 21 ++ .../ledger/GrpcOperationLedgerContract.java | 104 +++++++++- .../api/ledger/InMemoryOperationLedger.java | 29 ++- .../ledger/GrpcOperationLedgerMigrations.java | 37 ++++ .../ledger/GrpcOperationLedgerPostgresIT.java | 14 +- .../JpaGrpcOperationLedgerContractIT.java | 21 +- .../ledger/GrpcOperationLedgerRepository.java | 33 ++++ .../grpc/ledger/JpaGrpcOperationLedger.java | 11 ++ ...ex_grpc_operation_ledger_finished_rows.sql | 16 ++ .../GrpcOperationLedgerRepositoryTest.java | 6 + .../GrpcOperationLedgerRetention.java | 80 ++++++++ .../GrpcOperationLedgerRetentionTest.java | 83 ++++++++ grpc/grpc-spring-boot-starter/build.gradle | 2 + .../boot/GrpcPlatformAutoConfiguration.java | 88 +++++++++ .../grpc/boot/GrpcPlatformProperties.java | 42 +++- .../boot/GrpcPlatformStartupValidator.java | 4 +- ...cPlatformAutoConfigurationContextTest.java | 187 +++++++++++++++++- settings.gradle | 15 ++ 22 files changed, 864 insertions(+), 46 deletions(-) create mode 100644 gradle.properties create mode 100644 grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerMigrations.java create mode 100644 grpc/grpc-operation-ledger-jpa/src/main/resources/db/migration/grpc/V002__index_grpc_operation_ledger_finished_rows.sql create mode 100644 grpc/grpc-runtime/src/main/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetention.java create mode 100644 grpc/grpc-runtime/src/test/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetentionTest.java diff --git a/docs/adr/ADR-GRPC-004-retry-ownership-and-durable-idempotency.md b/docs/adr/ADR-GRPC-004-retry-ownership-and-durable-idempotency.md index deb788b..495d348 100644 --- a/docs/adr/ADR-GRPC-004-retry-ownership-and-durable-idempotency.md +++ b/docs/adr/ADR-GRPC-004-retry-ownership-and-durable-idempotency.md @@ -53,8 +53,10 @@ request takes it over with `GrpcOperationLedger.reclaimStale` — one conditiona `UPDATE … WHERE state = 'IN_PROGRESS' AND claimed_at < :cutoff`, so of two concurrent takeovers exactly one wins, the same shape as `reclaimFailed`. The TTL is the `claimTtl` argument of `GrpcIdempotencyInterceptor`, refused at construction when it is not longer than every keyed method's -deadline. It is not a configuration key: the starter does not assemble the interceptor yet (03-11), so -there is nothing for a key to configure. +deadline. Since the 03-11 follow-up the starter assembles the interceptor when +`app.grpc.platform.operation-ledger-enabled=true` and takes the TTL from +`app.grpc.platform.operation-ledger-claim-ttl` (default 5m); a TTL the constructor refuses fails startup +naming that key, and an enabled flag without a `GrpcOperationLedger` bean fails startup too. The takeover is safe only because the commit is recorded in the mutation's own transaction: then a claim still `IN_PROGRESS` means the mutation never committed. An adopter that commits them separately @@ -65,7 +67,18 @@ method and not just its deadline. Rejected: an owner token with a heartbeat lease (the outbox lease model) — needed only when a keyed method can run longer than any fixed TTL; revisit when one exists. A sweeper job that deletes stale rows — it needs a scheduler this family does not have, and deleting a claim makes the next retry a -fresh claim with no record that an attempt was abandoned. Retention of terminal rows is still open. +fresh claim with no record that an attempt was abandoned. + +**Finished rows are purged after a retention; in-progress rows never are** (03-10 follow-up). +`GrpcOperationLedger.purgeCompletedBefore(cutoff, limit)` deletes at most `limit` `COMMITTED` or +`FAILED_TERMINAL` rows that completed before the cutoff, oldest first — on PostgreSQL one bounded +`DELETE … WHERE storage_key IN (SELECT … LIMIT … FOR UPDATE SKIP LOCKED)` served by the partial index +of migration V002. `GrpcOperationLedgerRetention.purgeExpired(now)` applies +`app.grpc.platform.operation-ledger-retention` (default 24h, the HTTP idempotency replay TTL) in +batches. A retry after its record was purged is a fresh claim and runs again, so the retention must +exceed the callers' retry window. The adopter schedules the purge; the starter assembles the policy +only, because a scheduled bean would force a scheduler onto every deployment and the starter cannot +know which replica should run it. **A key reused for a different request is a caller error, not a duplicate.** The stored request fingerprint turns that into `FAILED_PRECONDITION` rather than silently returning the first request's diff --git a/docs/runbooks/grpc-platform-operations.md b/docs/runbooks/grpc-platform-operations.md index d218f26..0b63212 100644 --- a/docs/runbooks/grpc-platform-operations.md +++ b/docs/runbooks/grpc-platform-operations.md @@ -39,8 +39,8 @@ the time, and which half is not knowable from the status. escalate to the service owner. This is the case the keyed profile exists to avoid. 7. `IN_PROGRESS` means a claim exists and has not finished. If its owner died, the claim is not - stuck forever: once it is older than the claim TTL (the `claimTtl` given to - `GrpcIdempotencyInterceptor`), the next retry of the same request takes it over and runs again, + stuck forever: once it is older than the claim TTL (`app.grpc.platform.operation-ledger-claim-ttl`, + default 5m), the next retry of the same request takes it over and runs again, and a later reconciler pass sees that attempt's `COMMITTED` or `FAILED_TERMINAL`. Until a retry arrives, the case stays pending. The re-run is safe only when the ledger commit is recorded in the mutation's own transaction; if this service commits them separately, check the business resource @@ -52,6 +52,32 @@ retry arriving is a caller that gave up; it needs the business resource checked, --- +## The operation ledger table keeps growing + +**What you are seeing.** `grpc_operation_ledger` grows without bound, or a retry of an old key runs +the mutation again instead of replaying. + +**What it means.** Finished rows (`COMMITTED`, `FAILED_TERMINAL`) are deleted only when something +calls `GrpcOperationLedgerRetention.purgeExpired(now)`; the starter assembles that bean +(`app.grpc.platform.operation-ledger-enabled=true`) but does not schedule it. A growing table means +the service's scheduler is not calling it. A retry that re-ran after the retention +(`app.grpc.platform.operation-ledger-retention`, default 24h) is the trade the retention makes: its +record was purged, so it was a new operation. + +**What to do.** + +1. Check that the service schedules `purgeExpired`. Each call deletes in batches of 500 until a batch + comes back short, never touching `IN_PROGRESS` rows. +2. For a large backlog, call it repeatedly; each batch is a short transaction that skips rows another + transaction holds. +3. If retried keys re-run after the retention, raise the retention above the callers' retry window + rather than skipping the purge. + +**Escalate when** old `IN_PROGRESS` rows accumulate. Those are not purged by design; each is an +abandoned claim no retry has taken over (see `IN_PROGRESS` above). + +--- + ## A stream ended with FULL_RESYNC_REQUIRED **What you are seeing.** A client's resume was refused and it was told to resynchronise. diff --git a/gradle.properties b/gradle.properties new file mode 100644 index 0000000..0d1f586 --- /dev/null +++ b/gradle.properties @@ -0,0 +1,9 @@ +# optional-platforms is a build of its own (`./gradlew -p optional-platforms ...`), and Gradle reads +# gradle.properties from the build's root directory only, so src/gradle.properties never reached it: +# the optional build ran serially and without the build cache (review 2026-11-01 08-06). +# The values and their reasons are src/gradle.properties'. settings.gradle fails the build when the +# two files disagree, so this is a copy that cannot drift, not a second source. +org.gradle.workers.max=4 +org.gradle.jvmargs=-Xmx1g -XX:MaxMetaspaceSize=512m -Dfile.encoding=UTF-8 +org.gradle.caching=true +org.gradle.parallel=true diff --git a/grpc/CLAUDE.md b/grpc/CLAUDE.md index f056c0a..ada4282 100644 --- a/grpc/CLAUDE.md +++ b/grpc/CLAUDE.md @@ -96,6 +96,9 @@ application-owned port → adapter:inbound:grpc (typed service adapter) → :grp 직접 닿지 않는 규칙은 app-bootstrap ArchUnit(`INBOUND_ADAPTERS_DO_NOT_DEPEND_ON_OUTBOUND_ADAPTERS`)이 검사한다. 현재 `adapter:inbound:grpc`는 이 family에 의존하지 않는다 — registry의 `allowed_dependencies`를 보라. 배선은 별도 결정이며, 그때 registry edge와 브리지가 함께 들어간다. +starter가 `operation-ledger-enabled=true`일 때 멱등성 판단 bean을 조립하는 것(03-11)은 이 배선이 아니다 — +starter는 서버를 만들지 않고, 그 bean을 서버에 붙이는 일은 채택하는 쪽의 몫이다(아래 「Durable operation +ledger adoption」). ## 테스트 명령 @@ -130,13 +133,51 @@ the repository's `.seconds` / `.total` unit naming and the corresponding validat ### Durable operation ledger adoption -`grpc-spring-boot-starter` does not pull the JPA ledger adapter transitively. A deployment that needs -durable operation idempotency adds `:grpc:grpc-operation-ledger-jpa` explicitly, enables the ledger -configuration, and includes that adapter's Flyway location `db/migration/grpc`. A deployment that -does not adopt the ledger therefore pays no Spring Data JPA classpath cost. +`grpc-spring-boot-starter` does not pull the JPA ledger adapter transitively (11-M9). A deployment that +needs durable operation idempotency adds `:grpc:grpc-operation-ledger-jpa` explicitly, defines a +`GrpcOperationLedger` bean (`new JpaGrpcOperationLedger(repository)`), includes that adapter's Flyway +location `db/migration/grpc` (V001 table, V002 retention index), and sets the flag below. A deployment +that does not adopt the ledger pays no Spring Data JPA classpath cost. -An `IN_PROGRESS` claim expires after the claim TTL (`GrpcIdempotencyInterceptor`'s `claimTtl` -argument; no configuration key while the starter does not assemble the interceptor). A retry of the -same request then takes it over through `GrpcOperationLedger.reclaimStale`. The adopter must record -the commit in the mutation's transaction and choose a TTL longer than the slowest keyed method; both -reasons are in ADR-GRPC-004. +| Key (`app.grpc.platform.*`) | Default | What it does | +| --- | --- | --- | +| `operation-ledger-enabled` | `false` | `true`: the starter assembles `GrpcIdempotencyInterceptor` (bean `grpcIdempotencyInterceptor`) and `GrpcOperationLedgerRetention` (bean `grpcOperationLedgerRetention`) over the deployment's `GrpcOperationLedger` bean, and refuses startup when there is none. `false`: neither bean exists, and a method that is `IDEMPOTENCY_KEY_REQUIRED` refuses startup | +| `operation-ledger-claim-ttl` | `5m` | How long an `IN_PROGRESS` claim is trusted to have a live owner. After it, a retry of the same request takes the claim over (`GrpcOperationLedger.reclaimStale`). Refused at startup unless longer than every keyed method's deadline | +| `operation-ledger-retention` | `24h` | How long a finished (`COMMITTED`/`FAILED_TERMINAL`) record is kept before `GrpcOperationLedgerRetention.purgeExpired` may delete it. `IN_PROGRESS` rows are never purged | + +Defaults: the 5-minute TTL is the value the ledger contract and interceptor tests certify, far beyond +a unary deadline, and short enough that a key whose owner died unblocks within a client's retry +patience. The 24-hour retention matches the HTTP idempotency replay TTL +(`app.capabilities.idempotency.replay-ttl`). A retry of a key after its record was purged is a new +operation, so the retention must exceed the callers' retry window. + +These keys are not in `docs/registries/env-keys.yaml`: no `app.grpc.platform.*` key is, because the +optional build is not part of the shipped runtime. They join the registry when a composition root +adopts the platform. + +**Why the starter assembles the interceptor (03-11 (b)), and not (a).** The review proposed (a): +delete the flag and judge by whether a ledger bean exists. The owner chose (b): the flag stays and +means what it says — when true, the decision function is assembled and its TTL comes from +configuration. (a) would have removed the false promise but left every adopter to rebuild the same +assembly and TTL validation by hand; (b) keeps that in one place. The interceptor bean has no +`@ConditionalOnMissingBean` on purpose: a replacement would silently ignore `operation-ledger-claim-ttl`. +Duplicates of a running operation are refused (`REJECT_IN_PROGRESS`), not told to wait; a +wait-and-poll option becomes a key when a keyed method needs it. + +**Server wiring is still the adopter's, and stays a separate decision.** The starter builds no +server, so assembling the decision does not wire the platform into `adapter:inbound:grpc` (see "배선은 +별도 결정" above). `GrpcIdempotencyInterceptor` is a decision function, not an `io.grpc.ServerInterceptor`. +The adopter injects the bean by type and calls it from its own `IDEMPOTENCY` stage interceptor +(`GrpcServerInterceptorChain.builder().stage(GrpcServerInterceptorStage.IDEMPOTENCY, ...)`) or from +its typed service adapter: `decide(...)` before the mutation, `recordCommit(...)` inside the mutation's +transaction, `recordFailure(...)` on a terminal failure. + +**Purge scheduling is the adopter's.** `GrpcOperationLedgerRetention.purgeExpired(now)` deletes +expired finished rows in bounded batches (500 per statement) until a batch comes back short. The +starter does not schedule it: it owns no scheduler and cannot know which replica should run it, and a +scheduled bean would force a task scheduler onto every deployment. The adopter calls it from its own +scheduler (for example hourly); concurrent runs on several replicas are safe because the delete skips +locked rows. Running it never is safe but lets the table grow without bound. + +The commit must be recorded in the mutation's transaction and the TTL must be longer than the slowest +keyed method; both reasons are in ADR-GRPC-004. diff --git a/grpc/grpc-core-api/src/main/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedger.java b/grpc/grpc-core-api/src/main/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedger.java index 7a4ee67..2c27515 100644 --- a/grpc/grpc-core-api/src/main/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedger.java +++ b/grpc/grpc-core-api/src/main/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedger.java @@ -82,6 +82,27 @@ public interface GrpcOperationLedger { Instant claimedBefore, Instant now); + /** + * Deletes up to {@code limit} finished records ({@code COMMITTED} or {@code FAILED_TERMINAL}) + * that completed before {@code completedBefore}, oldest first. + * + *

Retention, not recovery. A finished row only answers retries, so once it is older than any + * retry a caller will still send, keeping it only grows the table. Deleting it is a trade the + * deployment makes knowingly: a retry of the same key arriving after the purge is a fresh claim + * and runs the mutation again. The retention must therefore exceed the callers' retry window. + * + *

An {@code IN_PROGRESS} record is never deleted, however old. It is either running or + * abandoned, and an abandoned one is taken over by {@link #reclaimStale}, which keeps the record + * that an attempt started; deleting it would erase that. + * + *

Bounded so a backlog is drained in short transactions rather than one long delete that holds + * locks on the hottest table of the idempotency path. + * + * @return how many records were deleted, at most {@code limit} + * @throws IllegalArgumentException when {@code limit} is not positive + */ + int purgeCompletedBefore(Instant completedBefore, int limit); + /** * Releases a claim that will never complete, so the caller may try again. * diff --git a/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedgerContract.java b/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedgerContract.java index b2839d6..0c625fc 100644 --- a/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedgerContract.java +++ b/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/GrpcOperationLedgerContract.java @@ -13,6 +13,7 @@ import java.util.concurrent.CountDownLatch; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.Future; +import java.util.stream.Stream; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; @@ -40,6 +41,10 @@ public abstract class GrpcOperationLedgerContract { // A retry arriving after the TTL: its cutoff (LATER - TTL) is past the original claim at NOW. private static final Instant LATER = NOW.plus(CLAIM_TTL).plusSeconds(1); private static final Instant CUTOFF = LATER.minus(CLAIM_TTL); + // A purge run a day after NOW: rows finished around NOW are past retention. + private static final Instant RETENTION_CUTOFF = NOW.plus(Duration.ofDays(1)); + private static final GrpcOperationIdentity SECOND_ID = identityWithKeyHash('b'); + private static final GrpcOperationIdentity THIRD_ID = identityWithKeyHash('c'); /** The ledger under test, holding no records when a test starts. */ protected abstract GrpcOperationLedger ledger(); @@ -397,9 +402,104 @@ public abstract class GrpcOperationLedgerContract { assertThat(refused).isInstanceOf(IllegalArgumentException.class).hasMessageContaining("cutoff"); } - private void failedClaim() { + @Test + @DisplayName("보존 기간이 지난 COMMITTED·FAILED_TERMINAL 행은 정리된다") + void finishedRecordsCompletedBeforeTheCutoffArePurged() { + // given + committedClaim(ID, NOW); + failedClaim(SECOND_ID, NOW); + + // when + int purged = ledger().purgeCompletedBefore(RETENTION_CUTOFF, 10); + + // then + assertThat(purged).isEqualTo(2); + assertThat(ledger().find(ID)).isEmpty(); + assertThat(ledger().find(SECOND_ID)).isEmpty(); + } + + @Test + @DisplayName("IN_PROGRESS 행은 아무리 오래되어도 정리하지 않는다") + void anInProgressClaimIsNeverPurgedHoweverOld() { + // given ledger().claim(ID, FINGERPRINT, NOW); - ledger().markFailed(ID, NOW.plusSeconds(1)); + + // when + int purged = ledger().purgeCompletedBefore(RETENTION_CUTOFF, 10); + + // then + assertThat(purged).isZero(); + assertThat(ledger().find(ID)) + .get() + .extracting(GrpcOperationLedgerRecord::state) + .isEqualTo(GrpcOperationLedgerState.IN_PROGRESS); + } + + @Test + @DisplayName("cutoff 이후에 끝난 행은 남긴다") + void finishedRecordsCompletedAtOrAfterTheCutoffAreKept() { + // given + committedClaim(ID, RETENTION_CUTOFF.minusSeconds(1)); + failedClaim(SECOND_ID, RETENTION_CUTOFF.minusSeconds(1)); + + // when + int purged = ledger().purgeCompletedBefore(RETENTION_CUTOFF, 10); + + // then + assertThat(purged).isZero(); + assertThat(ledger().find(ID)).isPresent(); + assertThat(ledger().find(SECOND_ID)).isPresent(); + } + + @Test + @DisplayName("한 번의 정리는 limit 개수까지만 지운다") + void onePurgeDeletesAtMostTheLimit() { + // given + committedClaim(ID, NOW); + committedClaim(SECOND_ID, NOW.plusSeconds(1)); + committedClaim(THIRD_ID, NOW.plusSeconds(2)); + + // when + int purged = ledger().purgeCompletedBefore(RETENTION_CUTOFF, 2); + + // then + assertThat(purged).isEqualTo(2); + assertThat( + Stream.of(ID, SECOND_ID, THIRD_ID) + .filter(identity -> ledger().find(identity).isPresent()) + .count()) + .isEqualTo(1); + } + + @Test + @DisplayName("limit이 양수가 아니면 정리를 거부한다") + void aNonPositivePurgeLimitIsRefused() { + // when + Throwable refused = catchThrowable(() -> ledger().purgeCompletedBefore(RETENTION_CUTOFF, 0)); + + // then + assertThat(refused).isInstanceOf(IllegalArgumentException.class).hasMessageContaining("limit"); + } + + private void failedClaim() { + failedClaim(ID, NOW); + } + + // Claimed at `claimedAt`, finished one second later, so the completion moment is what a purge + // cutoff is compared with. + private void failedClaim(GrpcOperationIdentity identity, Instant claimedAt) { + ledger().claim(identity, FINGERPRINT, claimedAt); + ledger().markFailed(identity, claimedAt.plusSeconds(1)); + } + + private void committedClaim(GrpcOperationIdentity identity, Instant claimedAt) { + ledger().claim(identity, FINGERPRINT, claimedAt); + ledger().markCommitted(identity, OUTCOME, claimedAt.plusSeconds(1)); + } + + private static GrpcOperationIdentity identityWithKeyHash(char hexDigit) { + return new GrpcOperationIdentity( + ID.callerFingerprint(), ID.method(), "sha256:" + String.valueOf(hexDigit).repeat(64)); } /** Runs {@code attempt} on two threads released together and returns both results. */ diff --git a/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/InMemoryOperationLedger.java b/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/InMemoryOperationLedger.java index 207971b..5547914 100644 --- a/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/InMemoryOperationLedger.java +++ b/grpc/grpc-core-api/src/testFixtures/java/dev/caskeleton/grpc/api/ledger/InMemoryOperationLedger.java @@ -1,6 +1,8 @@ package dev.caskeleton.grpc.api.ledger; import java.time.Instant; +import java.util.Comparator; +import java.util.List; import java.util.Optional; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentMap; @@ -10,8 +12,8 @@ import java.util.concurrent.ConcurrentMap; * *

{@code putIfAbsent} on a concurrent map is the in-memory equivalent of the unique constraint * the real adapter relies on, and {@code replace(key, expected, next)} of its conditional update, - * so claim, re-claim and stale takeover are atomic here for the same reason they are there. The - * transitions match the JPA adapter's because {@link GrpcOperationLedgerContract} runs against + * so claim, re-claim, stale takeover and purge are atomic here for the same reason they are there. + * The transitions match the JPA adapter's because {@link GrpcOperationLedgerContract} runs against * both; a fake that allowed more would let the policy tests pass for code the database refuses. */ public final class InMemoryOperationLedger implements GrpcOperationLedger { @@ -102,6 +104,29 @@ public final class InMemoryOperationLedger implements GrpcOperationLedger { GrpcOperationLedgerRecord.claim(identity, requestFingerprint, now)); } + @Override + public int purgeCompletedBefore(Instant completedBefore, int limit) { + if (limit < 1) { + throw new IllegalArgumentException("a purge limit must be positive"); + } + List expired = + records.values().stream() + .filter(record -> record.state().terminal()) + .filter(record -> record.completedAt().orElseThrow().isBefore(completedBefore)) + .sorted(Comparator.comparing(record -> record.completedAt().orElseThrow())) + .limit(limit) + .toList(); + int purged = 0; + for (GrpcOperationLedgerRecord record : expired) { + // Conditional, like the database's re-checked DELETE: a row re-claimed since it was read is + // IN_PROGRESS again and stays. + if (records.remove(record.identity().storageKey(), record)) { + purged++; + } + } + return purged; + } + @Override public void release(GrpcOperationIdentity identity) { records.remove(identity.storageKey()); diff --git a/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerMigrations.java b/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerMigrations.java new file mode 100644 index 0000000..6e28dfa --- /dev/null +++ b/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerMigrations.java @@ -0,0 +1,37 @@ +package dev.caskeleton.grpc.ledger; + +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.util.List; + +/** + * The ledger's Flyway scripts in version order, as one SQL text for a test database. + * + *

One list for every PostgreSQL lane, so a new migration is applied by all of them or by none; + * two copies of the list drift the first time someone adds a V003 to one of them. + */ +final class GrpcOperationLedgerMigrations { + + private static final List IN_ORDER = + List.of( + "db/migration/grpc/V001__create_grpc_operation_ledger.sql", + "db/migration/grpc/V002__index_grpc_operation_ledger_finished_rows.sql"); + + private GrpcOperationLedgerMigrations() {} + + /** Every migration's SQL, concatenated in version order. */ + static String all() throws IOException { + StringBuilder sql = new StringBuilder(); + for (String path : IN_ORDER) { + try (InputStream stream = + GrpcOperationLedgerMigrations.class.getClassLoader().getResourceAsStream(path)) { + if (stream == null) { + throw new IOException("gRPC operation-ledger migration is missing: " + path); + } + sql.append(new String(stream.readAllBytes(), StandardCharsets.UTF_8)).append('\n'); + } + } + return sql.toString(); + } +} diff --git a/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerPostgresIT.java b/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerPostgresIT.java index 98fd2f2..df1b00b 100644 --- a/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerPostgresIT.java +++ b/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerPostgresIT.java @@ -3,8 +3,6 @@ package dev.caskeleton.grpc.ledger; import static org.assertj.core.api.Assertions.assertThat; import java.io.IOException; -import java.io.InputStream; -import java.nio.charset.StandardCharsets; import java.sql.Connection; import java.sql.PreparedStatement; import java.sql.ResultSet; @@ -41,7 +39,7 @@ class GrpcOperationLedgerPostgresIT { @SuppressWarnings("resource") @Container private static final PostgreSQLContainer POSTGRES = - new PostgreSQLContainer("postgres:16-alpine") + new PostgreSQLContainer(System.getProperty("ca.test.image.postgres-16")) .withDatabaseName("grpc") .withUsername("grpc") .withPassword("grpc"); @@ -146,14 +144,6 @@ class GrpcOperationLedgerPostgresIT { } private static String migration() throws IOException { - try (InputStream stream = - GrpcOperationLedgerPostgresIT.class - .getClassLoader() - .getResourceAsStream("db/migration/grpc/V001__create_grpc_operation_ledger.sql")) { - if (stream == null) { - throw new IOException("gRPC operation-ledger migration is missing"); - } - return new String(stream.readAllBytes(), StandardCharsets.UTF_8); - } + return GrpcOperationLedgerMigrations.all(); } } diff --git a/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedgerContractIT.java b/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedgerContractIT.java index 701b31b..7888f7e 100644 --- a/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedgerContractIT.java +++ b/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedgerContractIT.java @@ -11,8 +11,6 @@ import dev.caskeleton.grpc.core.GrpcMethodName; import jakarta.persistence.EntityManagerFactory; import jakarta.persistence.PersistenceConfiguration; import java.io.IOException; -import java.io.InputStream; -import java.nio.charset.StandardCharsets; import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; @@ -44,7 +42,7 @@ class JpaGrpcOperationLedgerContractIT extends GrpcOperationLedgerContract { @SuppressWarnings("resource") @Container private static final PostgreSQLContainer POSTGRES = - new PostgreSQLContainer("postgres:16-alpine") + new PostgreSQLContainer(System.getProperty("ca.test.image.postgres-16")) .withDatabaseName("grpc") .withUsername("grpc") .withPassword("grpc"); @@ -169,15 +167,7 @@ class JpaGrpcOperationLedgerContractIT extends GrpcOperationLedgerContract { } private static String migration() throws IOException { - try (InputStream stream = - JpaGrpcOperationLedgerContractIT.class - .getClassLoader() - .getResourceAsStream("db/migration/grpc/V001__create_grpc_operation_ledger.sql")) { - if (stream == null) { - throw new IOException("gRPC operation-ledger migration is missing"); - } - return new String(stream.readAllBytes(), StandardCharsets.UTF_8); - } + return GrpcOperationLedgerMigrations.all(); } /** Runs each call in its own transaction, as a caller's business transaction would. */ @@ -227,6 +217,13 @@ class JpaGrpcOperationLedgerContractIT extends GrpcOperationLedgerContract { status -> delegate.reclaimStale(identity, requestFingerprint, claimedBefore, now))); } + @Override + public int purgeCompletedBefore(Instant completedBefore, int limit) { + Integer purged = + transactions.execute(status -> delegate.purgeCompletedBefore(completedBefore, limit)); + return purged == null ? 0 : purged; + } + @Override public void release(GrpcOperationIdentity identity) { transactions.executeWithoutResult(status -> delegate.release(identity)); diff --git a/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepository.java b/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepository.java index 32d14c0..d48c61c 100644 --- a/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepository.java +++ b/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepository.java @@ -122,6 +122,39 @@ public interface GrpcOperationLedgerRepository @Param("claimedBefore") Instant claimedBefore, @Param("claimedAt") Instant claimedAt); + /** + * Deletes up to {@code limit} finished rows that completed before {@code completedBefore}, oldest + * first. Never an {@code IN_PROGRESS} row. + * + *

{@code FOR UPDATE SKIP LOCKED} in the bounded subquery keeps the purge out of the way of a + * concurrent {@link #tryReclaimFailed}: a row that re-claim holds is skipped, and one it already + * turned back into {@code IN_PROGRESS} no longer matches. The outer state condition says the same + * thing again so the statement cannot delete a live claim even if the subquery is edited. The + * partial index {@code ix_grpc_operation_ledger_finished_completed_at} (V002) serves the + * subquery's filter and order. + * + * @return how many rows were deleted + */ + @Modifying(flushAutomatically = true, clearAutomatically = true) + @Transactional + @Query( + value = + """ + DELETE FROM grpc_operation_ledger + WHERE storage_key IN ( + SELECT storage_key + FROM grpc_operation_ledger + WHERE state IN ('COMMITTED', 'FAILED_TERMINAL') + AND completed_at < :completedBefore + ORDER BY completed_at + LIMIT :limit + FOR UPDATE SKIP LOCKED) + AND state <> 'IN_PROGRESS' + """, + nativeQuery = true) + int deleteFinishedBefore( + @Param("completedBefore") Instant completedBefore, @Param("limit") int limit); + /** Updates an existing claim after it has been loaded. */ GrpcOperationLedgerEntity save(GrpcOperationLedgerEntity entity); diff --git a/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedger.java b/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedger.java index d81fd66..55d80bc 100644 --- a/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedger.java +++ b/grpc/grpc-operation-ledger-jpa/src/main/java/dev/caskeleton/grpc/ledger/JpaGrpcOperationLedger.java @@ -122,6 +122,17 @@ public final class JpaGrpcOperationLedger implements GrpcOperationLedger { == 1; } + @Override + public int purgeCompletedBefore(Instant completedBefore, int limit) { + if (completedBefore == null) { + throw new IllegalArgumentException("a purge needs a cutoff"); + } + if (limit < 1) { + throw new IllegalArgumentException("a purge limit must be positive"); + } + return repository.deleteFinishedBefore(completedBefore, limit); + } + @Override public void release(GrpcOperationIdentity identity) { requireIdentity(identity); diff --git a/grpc/grpc-operation-ledger-jpa/src/main/resources/db/migration/grpc/V002__index_grpc_operation_ledger_finished_rows.sql b/grpc/grpc-operation-ledger-jpa/src/main/resources/db/migration/grpc/V002__index_grpc_operation_ledger_finished_rows.sql new file mode 100644 index 0000000..a0c6d7d --- /dev/null +++ b/grpc/grpc-operation-ledger-jpa/src/main/resources/db/migration/grpc/V002__index_grpc_operation_ledger_finished_rows.sql @@ -0,0 +1,16 @@ +-- Retention of finished ledger rows (review 2026-11-01, 03-10 follow-up). +-- +-- GrpcOperationLedgerRepository.deleteFinishedBefore deletes COMMITTED and FAILED_TERMINAL rows +-- whose completed_at is past the retention, oldest first and a bounded batch at a time. V001's +-- (state, claimed_at) index orders by claim time, not completion time, so without this index every +-- purge batch scans and sorts all finished rows. +-- +-- Partial: IN_PROGRESS rows have no completed_at and are never purged, so they stay out of the +-- index. The predicate repeats the query's state list literally so the planner can match it. +-- +-- A plain CREATE INDEX, not CONCURRENTLY: Flyway runs each migration in a transaction, and the +-- platform is not adopted anywhere yet, so the table is empty when this runs. An adopter that +-- applies it to a large existing ledger should run the CONCURRENTLY form by hand first. +CREATE INDEX ix_grpc_operation_ledger_finished_completed_at + ON grpc_operation_ledger (completed_at) + WHERE state IN ('COMMITTED', 'FAILED_TERMINAL'); diff --git a/grpc/grpc-operation-ledger-jpa/src/test/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepositoryTest.java b/grpc/grpc-operation-ledger-jpa/src/test/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepositoryTest.java index 9b7fcb4..e3aada6 100644 --- a/grpc/grpc-operation-ledger-jpa/src/test/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepositoryTest.java +++ b/grpc/grpc-operation-ledger-jpa/src/test/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerRepositoryTest.java @@ -73,6 +73,12 @@ class GrpcOperationLedgerRepositoryTest { "the stale takeover is certified against PostgreSQL by JpaGrpcOperationLedgerContractIT"); } + @Override + public int deleteFinishedBefore(Instant completedBefore, int limit) { + throw new UnsupportedOperationException( + "the bounded purge is certified against PostgreSQL by JpaGrpcOperationLedgerContractIT"); + } + @Override public GrpcOperationLedgerEntity save(GrpcOperationLedgerEntity entity) { rows.put(entity.getStorageKey(), entity); diff --git a/grpc/grpc-runtime/src/main/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetention.java b/grpc/grpc-runtime/src/main/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetention.java new file mode 100644 index 0000000..ac4e22c --- /dev/null +++ b/grpc/grpc-runtime/src/main/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetention.java @@ -0,0 +1,80 @@ +package dev.caskeleton.grpc.idempotency; + +import dev.caskeleton.grpc.api.ledger.GrpcOperationLedger; +import java.time.Duration; +import java.time.Instant; + +/** + * Deletes finished operation-ledger records once they are older than the retention. + * + *

Without it the ledger keeps every {@code COMMITTED} and {@code FAILED_TERMINAL} row forever. A + * finished row exists only to answer a retry, so once it is older than any retry a caller still + * sends, it is dead weight on the table every keyed call writes to. + * + *

This class decides what to delete; it does not decide when. Nothing here + * schedules itself: the platform starter does not own a scheduler, does not start the server, and + * cannot know which replica should run the purge, so the adopter calls {@link #purgeExpired} from + * its own scheduler (one replica, or every replica — the bounded delete is safe either way). + */ +public final class GrpcOperationLedgerRetention { + + /** + * Rows deleted per statement. Small enough that one delete holds its row locks briefly on the + * hottest table of the idempotency path, large enough that a day's backlog drains in few calls. + */ + public static final int DEFAULT_BATCH_SIZE = 500; + + private final GrpcOperationLedger ledger; + private final Duration retention; + private final int batchSize; + + /** + * @param retention how long a finished record is kept after it completed. It must exceed the + * longest window in which a caller still retries the same key: a retry after the purge finds + * no record, claims afresh and runs the mutation again. + * @param batchSize the most rows one delete statement removes + */ + public GrpcOperationLedgerRetention( + GrpcOperationLedger ledger, Duration retention, int batchSize) { + if (ledger == null) { + throw new IllegalArgumentException("a ledger retention needs a ledger"); + } + if (retention == null || retention.isZero() || retention.isNegative()) { + throw new IllegalArgumentException("a ledger retention must be present and positive"); + } + if (batchSize < 1) { + throw new IllegalArgumentException("a ledger purge batch size must be positive"); + } + this.ledger = ledger; + this.retention = retention; + this.batchSize = batchSize; + } + + /** How long a finished record is kept. */ + public Duration retention() { + return retention; + } + + /** + * Deletes every finished record that completed more than the retention before {@code now}, in + * bounded batches, and never an {@code IN_PROGRESS} one. + * + *

The cutoff is fixed at the start, so the loop ends: records finishing while it runs are + * newer than the cutoff and are left for the next call. + * + * @return how many records were deleted + */ + public int purgeExpired(Instant now) { + if (now == null) { + throw new IllegalArgumentException("a purge needs the current moment"); + } + Instant cutoff = now.minus(retention); + int total = 0; + int purged; + do { + purged = ledger.purgeCompletedBefore(cutoff, batchSize); + total += purged; + } while (purged == batchSize); + return total; + } +} diff --git a/grpc/grpc-runtime/src/test/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetentionTest.java b/grpc/grpc-runtime/src/test/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetentionTest.java new file mode 100644 index 0000000..fca364a --- /dev/null +++ b/grpc/grpc-runtime/src/test/java/dev/caskeleton/grpc/idempotency/GrpcOperationLedgerRetentionTest.java @@ -0,0 +1,83 @@ +package dev.caskeleton.grpc.idempotency; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.catchThrowable; + +import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity; +import dev.caskeleton.grpc.api.ledger.InMemoryOperationLedger; +import dev.caskeleton.grpc.core.GrpcMethodName; +import java.time.Duration; +import java.time.Instant; +import java.util.List; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +class GrpcOperationLedgerRetentionTest { + + private static final GrpcMethodName CREATE = + GrpcMethodName.parse("caskeleton.document.v1.DocumentService/CreateDocument"); + private static final Instant NOW = Instant.parse("2026-11-01T10:00:00Z"); + private static final Duration RETENTION = Duration.ofHours(24); + private static final String BODY = GrpcRequestFingerprint.of("body"); + + private final InMemoryOperationLedger ledger = new InMemoryOperationLedger(); + + @Test + @DisplayName("보존 기간이 지난 종결 행을 batch 크기와 무관하게 모두 지운다") + void everyFinishedRecordPastTheRetentionIsPurgedAcrossBatches() { + // given + List old = List.of(identity("k1"), identity("k2"), identity("k3")); + old.forEach(identity -> committed(identity, NOW.minus(RETENTION).minusSeconds(60))); + GrpcOperationLedgerRetention retention = new GrpcOperationLedgerRetention(ledger, RETENTION, 2); + + // when + int purged = retention.purgeExpired(NOW); + + // then + assertThat(purged).isEqualTo(3); + assertThat(old).allSatisfy(identity -> assertThat(ledger.find(identity)).isEmpty()); + } + + @Test + @DisplayName("보존 기간 안에 끝난 행과 진행 중인 행은 남긴다") + void recentFinishedRecordsAndInProgressClaimsAreKept() { + // given + GrpcOperationIdentity recent = identity("recent"); + GrpcOperationIdentity running = identity("running"); + committed(recent, NOW.minus(RETENTION).plusSeconds(60)); + ledger.claim(running, BODY, NOW.minus(Duration.ofDays(30))); + GrpcOperationLedgerRetention retention = new GrpcOperationLedgerRetention(ledger, RETENTION, 2); + + // when + int purged = retention.purgeExpired(NOW); + + // then + assertThat(purged).isZero(); + assertThat(ledger.find(recent)).isPresent(); + assertThat(ledger.find(running)).isPresent(); + } + + @Test + @DisplayName("보존 기간이 양수가 아니면 만들 수 없다") + void aNonPositiveRetentionIsRefused() { + // when + Throwable refused = + catchThrowable(() -> new GrpcOperationLedgerRetention(ledger, Duration.ZERO, 2)); + + // then + assertThat(refused) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("retention must be present and positive"); + } + + private static GrpcOperationIdentity identity(String key) { + return new GrpcOperationIdentity( + "tenant-1|actor-1", CREATE, GrpcRequestFingerprint.hashIdempotencyKey(key)); + } + + // Finished at the claim moment; the completion moment is what the retention measures. + private void committed(GrpcOperationIdentity identity, Instant claimedAt) { + ledger.claim(identity, BODY, claimedAt); + ledger.markCommitted(identity, "outcome://document/" + identity.storageKey(), claimedAt); + } +} diff --git a/grpc/grpc-spring-boot-starter/build.gradle b/grpc/grpc-spring-boot-starter/build.gradle index ad114e6..2a8fbfa 100644 --- a/grpc/grpc-spring-boot-starter/build.gradle +++ b/grpc/grpc-spring-boot-starter/build.gradle @@ -18,4 +18,6 @@ dependencies { implementation 'org.springframework.boot:spring-boot-autoconfigure' + // The shared operation-ledger fake stands in for the adopter's ledger bean in context tests. + testImplementation testFixtures(project(':grpc:grpc-core-api')) } diff --git a/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformAutoConfiguration.java b/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformAutoConfiguration.java index 8516939..02eddca 100644 --- a/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformAutoConfiguration.java +++ b/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformAutoConfiguration.java @@ -5,15 +5,20 @@ import dev.caskeleton.grpc.admin.GrpcDrainPolicy; import dev.caskeleton.grpc.admin.GrpcHealthPolicy; import dev.caskeleton.grpc.admin.GrpcReflectionPolicy; import dev.caskeleton.grpc.admin.GrpcServiceHealthRegistry; +import dev.caskeleton.grpc.api.ledger.GrpcOperationLedger; import dev.caskeleton.grpc.api.policy.GrpcMethodPolicyCatalog; import dev.caskeleton.grpc.client.GrpcNamedChannelProfile; import dev.caskeleton.grpc.context.GrpcContextBinder; import dev.caskeleton.grpc.context.GrpcContextPropagationPolicy; import dev.caskeleton.grpc.error.GrpcErrorMapper; +import dev.caskeleton.grpc.idempotency.GrpcIdempotencyInterceptor; +import dev.caskeleton.grpc.idempotency.GrpcOperationLedgerRetention; import dev.caskeleton.grpc.security.GrpcTlsProfile; import dev.caskeleton.grpc.server.GrpcAdmissionController; import dev.caskeleton.grpc.server.GrpcExecutorProfile; import dev.caskeleton.grpc.server.GrpcServerProfile; +import java.time.Duration; +import java.util.List; import java.util.UUID; import org.springframework.beans.factory.ObjectProvider; import org.springframework.beans.factory.SmartInitializingSingleton; @@ -131,6 +136,89 @@ public class GrpcPlatformAutoConfiguration { serverProfile); } + /** + * The keyed-idempotency decision, assembled when {@code operation-ledger-enabled=true}. + * + *

Built here rather than left to the adopter (review 2026-11-01, 03-11, option (b)): a flag + * that only satisfied the startup check while nothing honoured the key was a promise the platform + * did not keep. The ledger itself stays the deployment's bean — the starter does not depend on + * {@code grpc-operation-ledger-jpa} (11-M9) — and a missing one refuses startup. + * + *

A duplicate of a running operation is refused rather than told to wait: keyed callers are + * usually user-facing requests that already retry with backoff. There is deliberately no + * {@code @ConditionalOnMissingBean}: a replacement bean would silently ignore {@code + * operation-ledger-claim-ttl}, which is the same false promise this bean exists to remove. + * + *

The starter does not build a server, so nothing calls this yet: the adopter's {@code + * IDEMPOTENCY} stage interceptor ({@code GrpcServerInterceptorChain}) or service adapter calls + * {@code decide} before the mutation and {@code recordCommit} inside the mutation's transaction. + */ + @Bean + @ConditionalOnProperty( + prefix = "app.grpc.platform", + name = "operation-ledger-enabled", + havingValue = "true") + public GrpcIdempotencyInterceptor grpcIdempotencyInterceptor( + GrpcPlatformProperties properties, + ObjectProvider ledgers, + ObjectProvider catalogs) { + GrpcOperationLedger ledger = requireLedger(ledgers); + try { + return new GrpcIdempotencyInterceptor( + ledger, + catalogs.getIfAvailable(() -> GrpcMethodPolicyCatalog.builder().build()), + Duration.ZERO, + false, + properties.getOperationLedgerClaimTtl()); + } catch (IllegalArgumentException invalidTtl) { + // The ledger and catalog are present and the poll interval is fixed, so the claim TTL is the + // only input left for the constructor to refuse. + throw new GrpcPlatformConfigurationException( + List.of("app.grpc.platform.operation-ledger-claim-ttl: " + invalidTtl.getMessage())); + } + } + + /** + * Retention of finished ledger records, assembled with the interceptor. + * + *

Only the policy: nothing here schedules it. The starter owns no scheduler and does not know + * which replica should run the purge, so the adopter calls {@code purgeExpired(now)} from its own + * scheduler. A scheduled bean here was rejected because it would force a task scheduler onto + * every deployment and run on every replica whether or not that deployment wants it. + */ + @Bean + @ConditionalOnProperty( + prefix = "app.grpc.platform", + name = "operation-ledger-enabled", + havingValue = "true") + public GrpcOperationLedgerRetention grpcOperationLedgerRetention( + GrpcPlatformProperties properties, ObjectProvider ledgers) { + GrpcOperationLedger ledger = requireLedger(ledgers); + try { + return new GrpcOperationLedgerRetention( + ledger, + properties.getOperationLedgerRetention(), + GrpcOperationLedgerRetention.DEFAULT_BATCH_SIZE); + } catch (IllegalArgumentException invalidRetention) { + throw new GrpcPlatformConfigurationException( + List.of( + "app.grpc.platform.operation-ledger-retention: " + invalidRetention.getMessage())); + } + } + + private static GrpcOperationLedger requireLedger(ObjectProvider ledgers) { + GrpcOperationLedger ledger = ledgers.getIfAvailable(); + if (ledger == null) { + throw new GrpcPlatformConfigurationException( + List.of( + "app.grpc.platform.operation-ledger-enabled=true but there is no GrpcOperationLedger" + + " bean; the starter does not bring one. Add :grpc:grpc-operation-ledger-jpa and" + + " define a JpaGrpcOperationLedger bean (with the db/migration/grpc Flyway" + + " location), or turn the flag off")); + } + return ledger; + } + /** The shared error mapper. Its execution ids are random and carry no derived content. */ @Bean @ConditionalOnMissingBean diff --git a/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformProperties.java b/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformProperties.java index c480573..3d3a73b 100644 --- a/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformProperties.java +++ b/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformProperties.java @@ -3,6 +3,7 @@ package dev.caskeleton.grpc.boot; import dev.caskeleton.grpc.admin.GrpcReflectionMode; import dev.caskeleton.grpc.security.GrpcTlsProfile; import dev.caskeleton.grpc.server.GrpcServerTransport; +import java.time.Duration; import org.springframework.boot.context.properties.ConfigurationProperties; /** @@ -49,9 +50,32 @@ public class GrpcPlatformProperties { /** The bounded executor's maximum size. */ private int executorMaxPoolSize = 32; - /** Whether the operation ledger is available, which gates keyed idempotency. */ + /** + * Whether keyed idempotency runs. When true the starter assembles {@code + * GrpcIdempotencyInterceptor} and {@code GrpcOperationLedgerRetention} over the deployment's + * {@code GrpcOperationLedger} bean, and refuses to start without one; the starter never brings a + * ledger implementation itself. When false, a method that requires an idempotency key is refused + * at startup, because the key would be accepted and never honoured. + */ private boolean operationLedgerEnabled; + /** + * How long an {@code IN_PROGRESS} claim is trusted to have a live owner before a retry of the + * same request takes it over. Five minutes: far beyond a unary deadline (the interceptor refuses + * a TTL not longer than every keyed method's deadline), so a slow but live owner is not + * overtaken, and short enough that a key whose owner died unblocks within a client's normal retry + * patience. It is also the value the ledger contract and interceptor tests certify. + */ + private Duration operationLedgerClaimTtl = Duration.ofMinutes(5); + + /** + * How long a finished ({@code COMMITTED} or {@code FAILED_TERMINAL}) ledger record is kept before + * {@code GrpcOperationLedgerRetention} may delete it. A retry of the same key after that is a new + * operation, so this must exceed the callers' retry window. 24 hours, matching the HTTP + * idempotency replay TTL ({@code app.capabilities.idempotency.replay-ttl}). + */ + private Duration operationLedgerRetention = Duration.ofHours(24); + public boolean isEnabled() { return enabled; } @@ -124,6 +148,22 @@ public class GrpcPlatformProperties { this.operationLedgerEnabled = operationLedgerEnabled; } + public Duration getOperationLedgerClaimTtl() { + return operationLedgerClaimTtl; + } + + public void setOperationLedgerClaimTtl(Duration operationLedgerClaimTtl) { + this.operationLedgerClaimTtl = operationLedgerClaimTtl; + } + + public Duration getOperationLedgerRetention() { + return operationLedgerRetention; + } + + public void setOperationLedgerRetention(Duration operationLedgerRetention) { + this.operationLedgerRetention = operationLedgerRetention; + } + /** * The reflection mode this deployment runs, defaulting to the environment's floor, and to * disabled while the environment is unset (the startup check refuses that state). diff --git a/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformStartupValidator.java b/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformStartupValidator.java index 1f1a4de..a289278 100644 --- a/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformStartupValidator.java +++ b/grpc/grpc-spring-boot-starter/src/main/java/dev/caskeleton/grpc/boot/GrpcPlatformStartupValidator.java @@ -151,8 +151,8 @@ public final class GrpcPlatformStartupValidator { violations.add( "method '" + method.canonical() - + "' requires an idempotency key but no operation ledger is enabled; the key would " - + "be accepted and never honoured"); + + "' requires an idempotency key but app.grpc.platform.operation-ledger-enabled is" + + " false; the key would be accepted and never honoured"); } if (!policy.rpcType().stable()) { violations.add( diff --git a/grpc/grpc-spring-boot-starter/src/test/java/dev/caskeleton/grpc/boot/GrpcPlatformAutoConfigurationContextTest.java b/grpc/grpc-spring-boot-starter/src/test/java/dev/caskeleton/grpc/boot/GrpcPlatformAutoConfigurationContextTest.java index 6957096..a07b65b 100644 --- a/grpc/grpc-spring-boot-starter/src/test/java/dev/caskeleton/grpc/boot/GrpcPlatformAutoConfigurationContextTest.java +++ b/grpc/grpc-spring-boot-starter/src/test/java/dev/caskeleton/grpc/boot/GrpcPlatformAutoConfigurationContextTest.java @@ -4,13 +4,41 @@ import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; import static org.assertj.core.api.Assertions.catchThrowable; +import dev.caskeleton.grpc.api.deadline.GrpcDeadlineProfile; +import dev.caskeleton.grpc.api.ledger.GrpcOperationIdentity; +import dev.caskeleton.grpc.api.ledger.GrpcOperationLedger; +import dev.caskeleton.grpc.api.ledger.InMemoryOperationLedger; +import dev.caskeleton.grpc.api.policy.GrpcMethodPolicy; +import dev.caskeleton.grpc.api.policy.GrpcMethodPolicyCatalog; +import dev.caskeleton.grpc.api.policy.RpcIdempotencyProfile; +import dev.caskeleton.grpc.api.policy.WaitForReadyPolicy; +import dev.caskeleton.grpc.core.GrpcMethodName; +import dev.caskeleton.grpc.core.RpcType; +import dev.caskeleton.grpc.idempotency.GrpcIdempotencyDecision; +import dev.caskeleton.grpc.idempotency.GrpcIdempotencyInterceptor; +import dev.caskeleton.grpc.idempotency.GrpcOperationLedgerRetention; +import dev.caskeleton.grpc.idempotency.GrpcRequestFingerprint; +import java.time.Duration; +import java.time.Instant; +import java.util.HashMap; import java.util.Map; +import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; import org.springframework.context.annotation.AnnotationConfigApplicationContext; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.context.annotation.Import; import org.springframework.core.env.MapPropertySource; class GrpcPlatformAutoConfigurationContextTest { + private static final GrpcMethodName CREATE = + GrpcMethodName.parse("caskeleton.document.v1.DocumentService/CreateDocument"); + private static final Duration KEYED_DEADLINE = Duration.ofSeconds(10); + private static final String CALLER = "tenant-1|actor-1"; + private static final String KEY = "order-2026-0001"; + private static final Instant NOW = Instant.parse("2026-11-01T10:00:00Z"); + @Test void invalidProductionSecurityFailsDuringActualAutoConfigurationStartup() { assertThatThrownBy( @@ -77,12 +105,169 @@ class GrpcPlatformAutoConfigurationContextTest { } } - private static AnnotationConfigApplicationContext start(Map properties) { + @Test + @DisplayName("ledger 플래그가 켜져 있고 ledger bean이 있으면 멱등성 interceptor가 조립된다") + void anEnabledLedgerWithALedgerBeanAssemblesTheIdempotencyInterceptor() { + // when + try (AnnotationConfigApplicationContext context = + start(ledgerEnabled(Map.of()), KeyedMethodWithLedger.class)) { + + // then + assertThat(context.getBeansOfType(GrpcIdempotencyInterceptor.class)).hasSize(1); + } + } + + @Test + @DisplayName("ledger 플래그가 켜져 있는데 ledger bean이 없으면 기동을 거부한다") + void anEnabledLedgerWithoutALedgerBeanRefusesToStart() { + // when + Throwable failure = catchThrowable(() -> start(ledgerEnabled(Map.of()), KeyedMethod.class)); + + // then + assertThat(failure) + .rootCause() + .isInstanceOf(GrpcPlatformConfigurationException.class) + .hasMessageContaining("app.grpc.platform.operation-ledger-enabled=true") + .hasMessageContaining("no GrpcOperationLedger bean"); + } + + @Test + @DisplayName("ledger 플래그가 꺼져 있으면 멱등성 interceptor와 보존 정리를 만들지 않는다") + void aDisabledLedgerAssemblesNoIdempotencyInterceptorOrRetention() { + // when + try (AnnotationConfigApplicationContext context = + start(localPlatform(Map.of()), LedgerOnly.class)) { + + // then + assertThat(context.getBeansOfType(GrpcIdempotencyInterceptor.class)).isEmpty(); + assertThat(context.getBeansOfType(GrpcOperationLedgerRetention.class)).isEmpty(); + } + } + + @Test + @DisplayName("claim-ttl 설정이 interceptor의 stale claim 판단에 쓰인다") + void theClaimTtlKeyDecidesWhenAnInProgressClaimIsTakenOver() { + // given + try (AnnotationConfigApplicationContext context = + start( + ledgerEnabled(Map.of("app.grpc.platform.operation-ledger-claim-ttl", "1m")), + KeyedMethodWithLedger.class)) { + GrpcIdempotencyInterceptor interceptor = context.getBean(GrpcIdempotencyInterceptor.class); + String body = GrpcRequestFingerprint.of("body"); + interceptor.decide(CREATE, CALLER, KEY, body, NOW); + + // when + GrpcIdempotencyDecision retry = + interceptor.decide(CREATE, CALLER, KEY, body, NOW.plus(Duration.ofSeconds(61))); + + // then + assertThat(retry.outcome()).isEqualTo(GrpcIdempotencyDecision.Outcome.PROCEED); + } + } + + @Test + @DisplayName("claim-ttl이 keyed method의 deadline보다 길지 않으면 기동을 거부한다") + void aClaimTtlWithinAKeyedDeadlineRefusesToStart() { + // when + Throwable failure = + catchThrowable( + () -> + start( + ledgerEnabled(Map.of("app.grpc.platform.operation-ledger-claim-ttl", "10s")), + KeyedMethodWithLedger.class)); + + // then + assertThat(failure) + .rootCause() + .isInstanceOf(GrpcPlatformConfigurationException.class) + .hasMessageContaining("app.grpc.platform.operation-ledger-claim-ttl") + .hasMessageContaining("CreateDocument"); + } + + @Test + @DisplayName("retention 설정이 지난 종결 행을 보존 정리가 지운다") + void theRetentionKeyDecidesWhichFinishedRecordsArePurged() { + // given + try (AnnotationConfigApplicationContext context = + start( + ledgerEnabled(Map.of("app.grpc.platform.operation-ledger-retention", "1h")), + KeyedMethodWithLedger.class)) { + GrpcOperationLedger ledger = context.getBean(GrpcOperationLedger.class); + GrpcOperationIdentity identity = + new GrpcOperationIdentity(CALLER, CREATE, GrpcRequestFingerprint.hashIdempotencyKey(KEY)); + ledger.claim(identity, GrpcRequestFingerprint.of("body"), NOW); + ledger.markCommitted(identity, "outcome://document/42", NOW.plusSeconds(1)); + + // when + int purged = + context + .getBean(GrpcOperationLedgerRetention.class) + .purgeExpired(NOW.plus(Duration.ofHours(2))); + + // then + assertThat(purged).isEqualTo(1); + assertThat(ledger.find(identity)).isEmpty(); + } + } + + private static Map localPlatform(Map extra) { + Map properties = new HashMap<>(); + properties.put("app.grpc.platform.enabled", "true"); + properties.put("app.grpc.platform.environment", "LOCAL"); + properties.put("app.grpc.platform.transport", "NETTY_SHADED"); + properties.putAll(extra); + return properties; + } + + private static Map ledgerEnabled(Map extra) { + Map properties = new HashMap<>(extra); + properties.put("app.grpc.platform.operation-ledger-enabled", "true"); + return localPlatform(properties); + } + + /** One keyed method with a 10-second deadline, the shape the claim TTL is measured against. */ + @Configuration(proxyBeanMethods = false) + static class KeyedMethod { + @Bean + GrpcMethodPolicyCatalog keyedCatalog() { + return GrpcMethodPolicyCatalog.builder() + .register( + new GrpcMethodPolicy( + CREATE, + RpcType.UNARY, + RpcIdempotencyProfile.IDEMPOTENCY_KEY_REQUIRED, + GrpcDeadlineProfile.of(KEYED_DEADLINE), + WaitForReadyPolicy.DISABLED, + true, + 1024, + 1024)) + .build(); + } + } + + /** The adopter's ledger, here the shared in-memory fake. */ + @Configuration(proxyBeanMethods = false) + static class LedgerOnly { + @Bean + GrpcOperationLedger operationLedger() { + return new InMemoryOperationLedger(); + } + } + + @Configuration(proxyBeanMethods = false) + @Import({KeyedMethod.class, LedgerOnly.class}) + static class KeyedMethodWithLedger {} + + private static AnnotationConfigApplicationContext start( + Map properties, Class... adopterConfigurations) { AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(); context .getEnvironment() .getPropertySources() .addFirst(new MapPropertySource("grpc-platform-test", properties)); + if (adopterConfigurations.length > 0) { + context.register(adopterConfigurations); + } context.register(GrpcPlatformAutoConfiguration.class); try { context.refresh(); diff --git a/settings.gradle b/settings.gradle index cf08c10..a34cfb8 100644 --- a/settings.gradle +++ b/settings.gradle @@ -19,3 +19,18 @@ dependencyResolutionManagement { } rootProject.name = 'optional-platforms' + +// gradle.properties here must say what src/gradle.properties says (review 2026-11-01 08-06): Gradle +// reads only this build's own file, and a copy that drifted would make the optional build quietly +// slower or differently configured than the main one. +def readBuildProperties = { File file -> + def properties = new Properties() + file.withInputStream { properties.load(it) } + properties +} +def mainBuildProperties = readBuildProperties(new File(settingsDir, '../gradle.properties')) +def ownBuildProperties = readBuildProperties(new File(settingsDir, 'gradle.properties')) +if (mainBuildProperties != ownBuildProperties) { + throw new GradleException("optional-platforms/gradle.properties differs from src/gradle.properties: " + + "${ownBuildProperties} vs ${mainBuildProperties}. Copy the main build's values.") +}