Files
grpc-library/docs/runbooks/grpc-platform-operations.md
T
DongHyeonka cd4e117247 feat!: 남은 리뷰 발견을 고치고 개발 매뉴얼과 기능 생성 명령을 더한다
문제: High·P1을 고친 뒤에도 P2·P3 약 90건이 남았다. 연결·구독·메시지
크기 상한이 없는 전송(WebSocket·gRPC·GraphQL deadline), 낙관적 충돌의
500, upsert 경쟁, Mongo 무제한 timeout, 샘플 목록 N+1, 갈라진 복사본
(TenantId 3벌, 이벤트 ID 문법 5곳, SQLState 표 2벌), 문서를 읽는 PR
테스트와 캐시를 막는 lane 설정 등이다. 새 기능을 어디에 어떤 순서로
만드는지 알려 주는 문서와 도구가 없었다. MinIO 공식 이미지가 사라져
객체 저장소 lane이 돌지 않았다.

방법: 영역별 묶음으로 실패 테스트를 먼저 쓰고 고쳤다. 고치는 중에 찾은
결함도 함께 고쳤다(durable operation id가 UUIDv4라 이벤트가 전부 DEAD,
S3 어댑터 생성·작은 chunk 읽기 실패, Redis 조건부 기록의 EVALSHA 권한).
MinIO는 pgsty/minio를 digest로 고정했다(Chainguard는 태그·실행 사용자·
mc 이미지 문제로 버림). docs/manual/에 작업 순서와 판단 지점을 쓰고,
scripts/new-feature.sh가 5개 leaf에 뼈대를 만든다. 생성→빌드→통과
왕복을 verifyFeatureScaffold가 mainCheck에서 본다. 묶음별 결과와
결정은 리뷰 README 14절에 있다.

위험·한계: 파생 서비스 동작이 바뀐다. JDBC batch 50 기본, 운영 jar에서
H2 제거, 트랜잭션 30초 제한, 격리 수준을 Hikari에서 고정, gRPC·
WebSocket·GraphQL 새 한도, correlation ID 규칙 밖이면 서버가 교체,
샘플 키 오류 코드 IDEMPOTENCY_KEY_INVALID. 새 leaf 두 개
(composition:durable-operation, adapter:inbound:messaging)와 새
마이그레이션(postgresql V20·V21, notification V11·V12, sample V8_1,
grpc-ledger V002)이 생긴다. V21 유일 인덱스는 기존 중복이 있으면
실패한다(runbook에 확인 쿼리).

검증: ci, mainCheck(생성 명령 왕복 포함), integrationCheck(MinIO 포함
전 컨테이너 lane), -p optional-platforms ci integrationCheck 통과.

Refs: docs/reviews/2026-11-01-maturity-review/README.md 14절
BREAKING CHANGE: JDBC batch 기본 켜짐, H2 운영 의존 제거, 트랜잭션
기본 제한 30초, gRPC·WebSocket·GraphQL 서버 한도 기본값 추가,
샘플 Idempotency-Key 오류 코드 변경.
2026-11-03 19:52:06 +09:00

8.8 KiB

Runbook: gRPC platform operations

Scope: the :grpc:* family. All of it is build-only today — every leaf's runtime_memberships is empty — so nothing here fires in production yet. It is written now because the states it covers are the ones an on-call cannot work out from first principles at three in the morning, and shipping the behaviour before the runbook means the first person to meet one is doing that.

Configuration lives under app.grpc.platform.* and is bound by GrpcPlatformProperties. The platform does not start unless app.grpc.platform.enabled=true.


COMPLETION_UNKNOWN on a mutation

What you are seeing. A client received DEADLINE_EXCEEDED, UNAVAILABLE or INTERNAL on a state-changing call, and the response trailer completion-outcome reads COMPLETION_UNKNOWN.

What it means. The server may have committed. This is not a failure and not a success; the status code cannot distinguish them, which is why the outcome is carried separately.

What not to do. Do not re-issue the call. Do not tell the caller it failed. Both are wrong half the time, and which half is not knowable from the status.

What to do.

  1. Take the error-execution-id from the trailers. It is the only link between what the client saw and what the server did.

  2. If the method is IDEMPOTENCY_KEY_REQUIRED, query the operation ledger with the caller fingerprint, the full method name and the caller's key. GrpcOperationStatusQuery returns one of IN_PROGRESS, COMMITTED, FAILED_TERMINAL, NOT_FOUND or UNKNOWN.

  3. COMMITTED means return the stored outcome reference, not a freshly computed answer — the resource may have changed since, and a new answer would describe the state at reconciliation time rather than the state the caller's own call produced.

  4. NOT_FOUND means the operation never started and is safe to re-issue. FAILED_TERMINAL means the same.

  5. UNKNOWN means the ledger could not be consulted. Nothing may be concluded. The case is queued by GrpcCompletionReconciler and retried later.

  6. If the method is not keyed, there is no ledger row. Resolve it against the business resource, or escalate to the service owner. This is the case the keyed profile exists to avoid.

  7. IN_PROGRESS means a claim exists and has not finished. If its owner died, the claim is not stuck forever: once it is older than the claim TTL (app.grpc.platform.operation-ledger-claim-ttl, default 5m), the next retry of the same request takes it over and runs again, and a later reconciler pass sees that attempt's COMMITTED or FAILED_TERMINAL. Until a retry arrives, the case stays pending. The re-run is safe only when the ledger commit is recorded in the mutation's own transaction; if this service commits them separately, check the business resource before letting the retry through.

Escalate when the reconciler's pending list grows across passes. That means the ledger is unreachable rather than slow. A case pending as IN_PROGRESS for longer than the claim TTL with no retry arriving is a caller that gave up; it needs the business resource checked, not a ledger fix.


The operation ledger table keeps growing

What you are seeing. grpc_operation_ledger grows without bound, or a retry of an old key runs the mutation again instead of replaying.

What it means. Finished rows (COMMITTED, FAILED_TERMINAL) are deleted only when something calls GrpcOperationLedgerRetention.purgeExpired(now); the starter assembles that bean (app.grpc.platform.operation-ledger-enabled=true) but does not schedule it. A growing table means the service's scheduler is not calling it. A retry that re-ran after the retention (app.grpc.platform.operation-ledger-retention, default 24h) is the trade the retention makes: its record was purged, so it was a new operation.

What to do.

  1. Check that the service schedules purgeExpired. Each call deletes in batches of 500 until a batch comes back short, never touching IN_PROGRESS rows.
  2. For a large backlog, call it repeatedly; each batch is a short transaction that skips rows another transaction holds.
  3. If retried keys re-run after the retention, raise the retention above the callers' retry window rather than skipping the purge.

Escalate when old IN_PROGRESS rows accumulate. Those are not purged by design; each is an abandoned claim no retry has taken over (see IN_PROGRESS above).


A stream ended with FULL_RESYNC_REQUIRED

What you are seeing. A client's resume was refused and it was told to resynchronise.

What it means. The server can no longer replay from the client's cursor. Either the snapshot version moved, or the cursor predates retained history.

What to do. Nothing on the server. The client is expected to discard its position and start a new stream from a fresh snapshot. A client that instead retries the same token will keep receiving the same answer.

Escalate when it is happening to many clients at once. That usually means history retention was reduced, or snapshots are rotating faster than clients reconnect.


A stream ended with SLOW_CONSUMER

What you are seeing. Streams terminating with SLOW_CONSUMER, and grpc.stream.flow_control_stalls rising.

What it means. The consumer could not keep up with the bounded queue. The stream was terminated rather than silently dropping messages, because a client cannot detect drops — the sequence numbers it sees are the ones it was sent.

What to do.

  1. Check whether the consumer is slow or the producer is fast. grpc.stream.messages against grpc.stream.lifetime tells you the rate.
  2. If the consumer is slow, the fix is on the consumer. Raising the queue bound moves the failure later and makes it larger.
  3. A resume is not available after this ending: the messages that overflowed the queue are gone, so continuing from the last delivered sequence would silently skip them. The client resynchronises.

RESOURCE_EXHAUSTED under load

What you are seeing. Calls refused with RESOURCE_EXHAUSTED and GrpcAdmissionController reporting rejections.

What it means. The server is at its concurrency and queue bounds and is shedding rather than queueing. This is the designed behaviour: accepting work whose callers have already given up spends capacity on nothing.

What to do.

  1. Read grpc.rpc.duration and grpc.rpc.queue_wait separately. Queue time rising with duration flat means the bottleneck is admission, not the work.
  2. Check which saturation counter is moving — executor, channel or flow control. They look identical in a latency graph and have different fixes.
  3. Raising app.grpc.platform.executor-queue-capacity defers the problem; it does not remove it. GrpcExecutorProfile refuses a queue above ten thousand for that reason.

A rollout is producing errors at every deploy

What you are seeing. A burst of UNAVAILABLE or CANCELLED each time an instance goes away.

What it means. The drain sequence is not completing, or is running out of order.

What to do.

  1. GrpcDrainResult records what each drain achieved: completed and cancelled unary calls, signalled and cancelled streams, and which phases ran. A drain that routinely force-cancels is the cause.
  2. The order matters. Readiness flips first and nothing is refused during that window, because there is a gap between an instance reporting unready and routing acting on it. Refusing during that gap turns a clean rollout into a burst of errors at every deploy.
  3. For long streams, check the Kubernetes profile's streamReconnectBudget and readinessDrainGrace. A stream is pinned to one pod for its whole life, so every rollout ends it; a profile with long streams and no reconnect budget has not decided what clients do next.

Verifying a deployment's configuration

GrpcPlatformSnapshotService produces a secret-free snapshot for a caller on the admin network holding an admin role. Both gates are required.

GrpcPlatformSnapshotService.driftAgainstRelease compares a running snapshot with the release manifest and reports schema version, method policy hash and per-channel profile differences. An instance running a configuration the release did not ship is behind a whole class of incidents that are otherwise diagnosed by reading logs.

The snapshot carries hashes and names only. A field whose name looks like a credential is refused at construction rather than redacted.


Things that are deliberately off

  • Reflection in production. GrpcReflectionMode.defaultFor returns DISABLED for STAGE and PROD. Reflection publishes the whole schema to anyone who can open a connection.
  • Every advanced capability. See docs/runbooks/grpc-advanced-capabilities.md.
  • The platform itself. app.grpc.platform.enabled defaults to false, and every :grpc:* leaf is build-only in the registry.