문제: 성숙도 리뷰 수정을 다시 읽은 재리뷰(Critical 0, 회귀 0)가 Important 27건·Minor 41건을 찾았다. 수정이 새로 만든 결함이 많았다. @SystemInvoked가 권한 규칙의 탈출구가 됐고, 데모 계정 거부가 정확히 prod일 때만 걸렸고, outbox 순서 번호가 업그레이드 때 뒤로 갔고, GraphQL union 아래 복잡도가 0이었고, 서블릿 오류 응답이 success:true로 나갔고, 호출자 deadline이 breaker를 열었다. 원래 리뷰 색인이 06 영역 발견 5건(06-10~14)을 빠뜨려 그동안 처리되지 않았다. 방법: 영역별 7묶음이 test-driven-development 절차로 실패 테스트를 먼저 쓰고 고쳤다. 결정은 재리뷰 README 4절에 적었다: 순서 번호는 V21이 기록한 전역 floor부터, 데모 계정은 local·test·demo 허용 목록, @SystemInvoked bean은 기동 실패, complete 단계 실패는 IDEMPOTENCY_COMPLETION_UNKNOWN(결과 불명), 캐시 조건부 쓰기는 별도 SCRIPT lane, strict lane 정책은 build cache 입력, 끝나지 않는 재시도 행은 원인 코드·지표·PARKED 절차. 레지스트리가 선언만 하고 아무 코드도 내보내지 않던 redis.capability.* 지표 6개는 빼고 runbook을 실제 지표(redis.lane.*) 기준으로 고쳤다. 위험·한계: V21을 V22 없이 직접 고쳤다(어느 배포에도 아직 없음). 이 브랜치의 V21을 적용한 개발 DB는 다시 만들어야 한다. 서블릿 오류 디스패치를 보안 체인이 허용한다. 발행·저장소 포트 시그니처가 바뀌어 파생 서비스 코드가 깨진다. prod에서 fixture-로 시작하는 설정값은 기동을 막는다. httpclient failure-injection lane의 main 단계 중복 실행은 이전 결정대로 남겼다. 실제 CI runner 실행은 확인하지 못했다. 검증: ci, mainCheck, :integrationCheck, -p optional-platforms ci integrationCheck performanceCheck, :performanceCheck, :releaseCheck (-PgitRevision), verify-compose-profile-contracts(18 lane) 통과. 첫 실행의 실패 2건(FindSecBugs 1, sample 동시 생성 테스트의 v4 event id)은 고친 뒤 ci·integrationCheck·mainCheck를 다시 돌려 통과했다. Refs: docs/reviews/2026-11-03-rereview/README.md 3·4·6·7절 BREAKING CHANGE: outbox 발행 포트와 저장소 포트 시그니처, 순서 번호 첫 값(V21 floor 다음), 샘플 데모 계정 프로필 허용 목록, 새 오류 코드 IDEMPOTENCY_COMPLETION_UNKNOWN, prod의 fixture- 값 거부, 기동 rule은 static·의존 없음이어야 한다.
9.1 KiB
Runbook: gRPC platform operations
Scope: the :grpc:* family. All of it is build-only today — every leaf's runtime_memberships is
empty — so nothing here fires in production yet. It is written now because the states it covers are
the ones an on-call cannot work out from first principles at three in the morning, and shipping the
behaviour before the runbook means the first person to meet one is doing that.
Configuration lives under app.grpc.platform.* and is bound by GrpcPlatformProperties.
The platform does not start unless app.grpc.platform.enabled=true.
COMPLETION_UNKNOWN on a mutation
What you are seeing. A client received DEADLINE_EXCEEDED, UNAVAILABLE or INTERNAL on a
state-changing call, and the response trailer completion-outcome reads COMPLETION_UNKNOWN.
What it means. The server may have committed. This is not a failure and not a success; the status code cannot distinguish them, which is why the outcome is carried separately.
What not to do. Do not re-issue the call. Do not tell the caller it failed. Both are wrong half the time, and which half is not knowable from the status.
What to do.
-
Take the
error-execution-idfrom the trailers. It is the only link between what the client saw and what the server did. -
If the method is
IDEMPOTENCY_KEY_REQUIRED, query the operation ledger with the caller fingerprint, the full method name and the caller's key.GrpcOperationStatusQueryreturns one ofIN_PROGRESS,COMMITTED,FAILED_TERMINAL,NOT_FOUNDorUNKNOWN. -
COMMITTEDmeans return the stored outcome reference, not a freshly computed answer — the resource may have changed since, and a new answer would describe the state at reconciliation time rather than the state the caller's own call produced. -
NOT_FOUNDmeans the operation never started and is safe to re-issue.FAILED_TERMINALmeans the same. -
UNKNOWNmeans the ledger could not be consulted. Nothing may be concluded. The case is queued byGrpcCompletionReconcilerand retried later. -
If the method is not keyed, there is no ledger row. Resolve it against the business resource, or escalate to the service owner. This is the case the keyed profile exists to avoid.
-
IN_PROGRESSmeans a claim exists and has not finished. If its owner died, the claim is not stuck forever: once it is older than the claim TTL (app.grpc.platform.operation-ledger-claim-ttl, default 5m), the next retry of the same request takes it over and runs again, and a later reconciler pass sees that attempt'sCOMMITTEDorFAILED_TERMINAL. Until a retry arrives, the case stays pending. The re-run is safe only when the ledger commit is recorded in the mutation's own transaction; if this service commits them separately, check the business resource before letting the retry through. An owner that was only slow cannot spoil the takeover: its claim attempt (claim_attempt) is behind, so its late commit is refused and its late failure ignored. A log line saying an operation "is no longer held by attempt N" is that refusal, not an outage.
Escalate when the reconciler's pending list grows across passes. That means the ledger is
unreachable rather than slow. A case pending as IN_PROGRESS for longer than the claim TTL with no
retry arriving is a caller that gave up; it needs the business resource checked, not a ledger fix.
The operation ledger table keeps growing
What you are seeing. grpc_operation_ledger grows without bound, or a retry of an old key runs
the mutation again instead of replaying.
What it means. Finished rows (COMMITTED, FAILED_TERMINAL) are deleted only when something
calls GrpcOperationLedgerRetention.purgeExpired(now); the starter assembles that bean
(app.grpc.platform.operation-ledger-enabled=true) but does not schedule it. A growing table means
the service's scheduler is not calling it. A retry that re-ran after the retention
(app.grpc.platform.operation-ledger-retention, default 24h) is the trade the retention makes: its
record was purged, so it was a new operation.
What to do.
- Check that the service schedules
purgeExpired. Each call deletes in batches of 500 until a batch comes back short, never touchingIN_PROGRESSrows. - For a large backlog, call it repeatedly; each batch is a short transaction that skips rows another transaction holds.
- If retried keys re-run after the retention, raise the retention above the callers' retry window rather than skipping the purge.
Escalate when old IN_PROGRESS rows accumulate. Those are not purged by design; each is an
abandoned claim no retry has taken over (see IN_PROGRESS above).
A stream ended with FULL_RESYNC_REQUIRED
What you are seeing. A client's resume was refused and it was told to resynchronise.
What it means. The server can no longer replay from the client's cursor. Either the snapshot version moved, or the cursor predates retained history.
What to do. Nothing on the server. The client is expected to discard its position and start a new stream from a fresh snapshot. A client that instead retries the same token will keep receiving the same answer.
Escalate when it is happening to many clients at once. That usually means history retention was reduced, or snapshots are rotating faster than clients reconnect.
A stream ended with SLOW_CONSUMER
What you are seeing. Streams terminating with SLOW_CONSUMER, and
grpc.stream.flow_control_stalls rising.
What it means. The consumer could not keep up with the bounded queue. The stream was terminated rather than silently dropping messages, because a client cannot detect drops — the sequence numbers it sees are the ones it was sent.
What to do.
- Check whether the consumer is slow or the producer is fast.
grpc.stream.messagesagainstgrpc.stream.lifetimetells you the rate. - If the consumer is slow, the fix is on the consumer. Raising the queue bound moves the failure later and makes it larger.
- A resume is not available after this ending: the messages that overflowed the queue are gone, so continuing from the last delivered sequence would silently skip them. The client resynchronises.
RESOURCE_EXHAUSTED under load
What you are seeing. Calls refused with RESOURCE_EXHAUSTED and GrpcAdmissionController
reporting rejections.
What it means. The server is at its concurrency and queue bounds and is shedding rather than queueing. This is the designed behaviour: accepting work whose callers have already given up spends capacity on nothing.
What to do.
- Read
grpc.rpc.durationandgrpc.rpc.queue_waitseparately. Queue time rising with duration flat means the bottleneck is admission, not the work. - Check which saturation counter is moving — executor, channel or flow control. They look identical in a latency graph and have different fixes.
- Raising
app.grpc.platform.executor-queue-capacitydefers the problem; it does not remove it.GrpcExecutorProfilerefuses a queue above ten thousand for that reason.
A rollout is producing errors at every deploy
What you are seeing. A burst of UNAVAILABLE or CANCELLED each time an instance goes away.
What it means. The drain sequence is not completing, or is running out of order.
What to do.
GrpcDrainResultrecords what each drain achieved: completed and cancelled unary calls, signalled and cancelled streams, and which phases ran. A drain that routinely force-cancels is the cause.- The order matters. Readiness flips first and nothing is refused during that window, because there is a gap between an instance reporting unready and routing acting on it. Refusing during that gap turns a clean rollout into a burst of errors at every deploy.
- For long streams, check the Kubernetes profile's
streamReconnectBudgetandreadinessDrainGrace. A stream is pinned to one pod for its whole life, so every rollout ends it; a profile with long streams and no reconnect budget has not decided what clients do next.
Verifying a deployment's configuration
GrpcPlatformSnapshotService produces a secret-free snapshot for a caller on the admin network
holding an admin role. Both gates are required.
GrpcPlatformSnapshotService.driftAgainstRelease compares a running snapshot with the release
manifest and reports schema version, method policy hash and per-channel profile differences. An
instance running a configuration the release did not ship is behind a whole class of incidents that
are otherwise diagnosed by reading logs.
The snapshot carries hashes and names only. A field whose name looks like a credential is refused at construction rather than redacted.
Things that are deliberately off
- Reflection in production.
GrpcReflectionMode.defaultForreturnsDISABLEDforSTAGEandPROD. Reflection publishes the whole schema to anyone who can open a connection. - Every advanced capability. See
docs/runbooks/grpc-advanced-capabilities.md. - The platform itself.
app.grpc.platform.enableddefaults to false, and every:grpc:*leaf is build-only in the registry.