feat!: 남은 리뷰 발견을 고치고 개발 매뉴얼과 기능 생성 명령을 더한다
문제: High·P1을 고친 뒤에도 P2·P3 약 90건이 남았다. 연결·구독·메시지 크기 상한이 없는 전송(WebSocket·gRPC·GraphQL deadline), 낙관적 충돌의 500, upsert 경쟁, Mongo 무제한 timeout, 샘플 목록 N+1, 갈라진 복사본 (TenantId 3벌, 이벤트 ID 문법 5곳, SQLState 표 2벌), 문서를 읽는 PR 테스트와 캐시를 막는 lane 설정 등이다. 새 기능을 어디에 어떤 순서로 만드는지 알려 주는 문서와 도구가 없었다. MinIO 공식 이미지가 사라져 객체 저장소 lane이 돌지 않았다. 방법: 영역별 묶음으로 실패 테스트를 먼저 쓰고 고쳤다. 고치는 중에 찾은 결함도 함께 고쳤다(durable operation id가 UUIDv4라 이벤트가 전부 DEAD, S3 어댑터 생성·작은 chunk 읽기 실패, Redis 조건부 기록의 EVALSHA 권한). MinIO는 pgsty/minio를 digest로 고정했다(Chainguard는 태그·실행 사용자· mc 이미지 문제로 버림). docs/manual/에 작업 순서와 판단 지점을 쓰고, scripts/new-feature.sh가 5개 leaf에 뼈대를 만든다. 생성→빌드→통과 왕복을 verifyFeatureScaffold가 mainCheck에서 본다. 묶음별 결과와 결정은 리뷰 README 14절에 있다. 위험·한계: 파생 서비스 동작이 바뀐다. JDBC batch 50 기본, 운영 jar에서 H2 제거, 트랜잭션 30초 제한, 격리 수준을 Hikari에서 고정, gRPC· WebSocket·GraphQL 새 한도, correlation ID 규칙 밖이면 서버가 교체, 샘플 키 오류 코드 IDEMPOTENCY_KEY_INVALID. 새 leaf 두 개 (composition:durable-operation, adapter:inbound:messaging)와 새 마이그레이션(postgresql V20·V21, notification V11·V12, sample V8_1, grpc-ledger V002)이 생긴다. V21 유일 인덱스는 기존 중복이 있으면 실패한다(runbook에 확인 쿼리). 검증: ci, mainCheck(생성 명령 왕복 포함), integrationCheck(MinIO 포함 전 컨테이너 lane), -p optional-platforms ci integrationCheck 통과. Refs: docs/reviews/2026-11-01-maturity-review/README.md 14절 BREAKING CHANGE: JDBC batch 기본 켜짐, H2 운영 의존 제거, 트랜잭션 기본 제한 30초, gRPC·WebSocket·GraphQL 서버 한도 기본값 추가, 샘플 Idempotency-Key 오류 코드 변경.
This commit is contained in:
@@ -53,8 +53,10 @@ request takes it over with `GrpcOperationLedger.reclaimStale` — one conditiona
|
||||
`UPDATE … WHERE state = 'IN_PROGRESS' AND claimed_at < :cutoff`, so of two concurrent takeovers exactly
|
||||
one wins, the same shape as `reclaimFailed`. The TTL is the `claimTtl` argument of
|
||||
`GrpcIdempotencyInterceptor`, refused at construction when it is not longer than every keyed method's
|
||||
deadline. It is not a configuration key: the starter does not assemble the interceptor yet (03-11), so
|
||||
there is nothing for a key to configure.
|
||||
deadline. Since the 03-11 follow-up the starter assembles the interceptor when
|
||||
`app.grpc.platform.operation-ledger-enabled=true` and takes the TTL from
|
||||
`app.grpc.platform.operation-ledger-claim-ttl` (default 5m); a TTL the constructor refuses fails startup
|
||||
naming that key, and an enabled flag without a `GrpcOperationLedger` bean fails startup too.
|
||||
|
||||
The takeover is safe only because the commit is recorded in the mutation's own transaction: then a
|
||||
claim still `IN_PROGRESS` means the mutation never committed. An adopter that commits them separately
|
||||
@@ -65,7 +67,18 @@ method and not just its deadline.
|
||||
Rejected: an owner token with a heartbeat lease (the outbox lease model) — needed only when a keyed
|
||||
method can run longer than any fixed TTL; revisit when one exists. A sweeper job that deletes stale
|
||||
rows — it needs a scheduler this family does not have, and deleting a claim makes the next retry a
|
||||
fresh claim with no record that an attempt was abandoned. Retention of terminal rows is still open.
|
||||
fresh claim with no record that an attempt was abandoned.
|
||||
|
||||
**Finished rows are purged after a retention; in-progress rows never are** (03-10 follow-up).
|
||||
`GrpcOperationLedger.purgeCompletedBefore(cutoff, limit)` deletes at most `limit` `COMMITTED` or
|
||||
`FAILED_TERMINAL` rows that completed before the cutoff, oldest first — on PostgreSQL one bounded
|
||||
`DELETE … WHERE storage_key IN (SELECT … LIMIT … FOR UPDATE SKIP LOCKED)` served by the partial index
|
||||
of migration V002. `GrpcOperationLedgerRetention.purgeExpired(now)` applies
|
||||
`app.grpc.platform.operation-ledger-retention` (default 24h, the HTTP idempotency replay TTL) in
|
||||
batches. A retry after its record was purged is a fresh claim and runs again, so the retention must
|
||||
exceed the callers' retry window. The adopter schedules the purge; the starter assembles the policy
|
||||
only, because a scheduled bean would force a scheduler onto every deployment and the starter cannot
|
||||
know which replica should run it.
|
||||
|
||||
**A key reused for a different request is a caller error, not a duplicate.** The stored request
|
||||
fingerprint turns that into `FAILED_PRECONDITION` rather than silently returning the first request's
|
||||
|
||||
@@ -39,8 +39,8 @@ the time, and which half is not knowable from the status.
|
||||
escalate to the service owner. This is the case the keyed profile exists to avoid.
|
||||
|
||||
7. `IN_PROGRESS` means a claim exists and has not finished. If its owner died, the claim is not
|
||||
stuck forever: once it is older than the claim TTL (the `claimTtl` given to
|
||||
`GrpcIdempotencyInterceptor`), the next retry of the same request takes it over and runs again,
|
||||
stuck forever: once it is older than the claim TTL (`app.grpc.platform.operation-ledger-claim-ttl`,
|
||||
default 5m), the next retry of the same request takes it over and runs again,
|
||||
and a later reconciler pass sees that attempt's `COMMITTED` or `FAILED_TERMINAL`. Until a retry
|
||||
arrives, the case stays pending. The re-run is safe only when the ledger commit is recorded in the
|
||||
mutation's own transaction; if this service commits them separately, check the business resource
|
||||
@@ -52,6 +52,32 @@ retry arriving is a caller that gave up; it needs the business resource checked,
|
||||
|
||||
---
|
||||
|
||||
## The operation ledger table keeps growing
|
||||
|
||||
**What you are seeing.** `grpc_operation_ledger` grows without bound, or a retry of an old key runs
|
||||
the mutation again instead of replaying.
|
||||
|
||||
**What it means.** Finished rows (`COMMITTED`, `FAILED_TERMINAL`) are deleted only when something
|
||||
calls `GrpcOperationLedgerRetention.purgeExpired(now)`; the starter assembles that bean
|
||||
(`app.grpc.platform.operation-ledger-enabled=true`) but does not schedule it. A growing table means
|
||||
the service's scheduler is not calling it. A retry that re-ran after the retention
|
||||
(`app.grpc.platform.operation-ledger-retention`, default 24h) is the trade the retention makes: its
|
||||
record was purged, so it was a new operation.
|
||||
|
||||
**What to do.**
|
||||
|
||||
1. Check that the service schedules `purgeExpired`. Each call deletes in batches of 500 until a batch
|
||||
comes back short, never touching `IN_PROGRESS` rows.
|
||||
2. For a large backlog, call it repeatedly; each batch is a short transaction that skips rows another
|
||||
transaction holds.
|
||||
3. If retried keys re-run after the retention, raise the retention above the callers' retry window
|
||||
rather than skipping the purge.
|
||||
|
||||
**Escalate when** old `IN_PROGRESS` rows accumulate. Those are not purged by design; each is an
|
||||
abandoned claim no retry has taken over (see `IN_PROGRESS` above).
|
||||
|
||||
---
|
||||
|
||||
## A stream ended with FULL_RESYNC_REQUIRED
|
||||
|
||||
**What you are seeing.** A client's resume was refused and it was told to resynchronise.
|
||||
|
||||
Reference in New Issue
Block a user