문제: 성숙도 리뷰 수정을 다시 읽은 재리뷰(Critical 0, 회귀 0)가 Important 27건·Minor 41건을 찾았다. 수정이 새로 만든 결함이 많았다. @SystemInvoked가 권한 규칙의 탈출구가 됐고, 데모 계정 거부가 정확히 prod일 때만 걸렸고, outbox 순서 번호가 업그레이드 때 뒤로 갔고, GraphQL union 아래 복잡도가 0이었고, 서블릿 오류 응답이 success:true로 나갔고, 호출자 deadline이 breaker를 열었다. 원래 리뷰 색인이 06 영역 발견 5건(06-10~14)을 빠뜨려 그동안 처리되지 않았다. 방법: 영역별 7묶음이 test-driven-development 절차로 실패 테스트를 먼저 쓰고 고쳤다. 결정은 재리뷰 README 4절에 적었다: 순서 번호는 V21이 기록한 전역 floor부터, 데모 계정은 local·test·demo 허용 목록, @SystemInvoked bean은 기동 실패, complete 단계 실패는 IDEMPOTENCY_COMPLETION_UNKNOWN(결과 불명), 캐시 조건부 쓰기는 별도 SCRIPT lane, strict lane 정책은 build cache 입력, 끝나지 않는 재시도 행은 원인 코드·지표·PARKED 절차. 레지스트리가 선언만 하고 아무 코드도 내보내지 않던 redis.capability.* 지표 6개는 빼고 runbook을 실제 지표(redis.lane.*) 기준으로 고쳤다. 위험·한계: V21을 V22 없이 직접 고쳤다(어느 배포에도 아직 없음). 이 브랜치의 V21을 적용한 개발 DB는 다시 만들어야 한다. 서블릿 오류 디스패치를 보안 체인이 허용한다. 발행·저장소 포트 시그니처가 바뀌어 파생 서비스 코드가 깨진다. prod에서 fixture-로 시작하는 설정값은 기동을 막는다. httpclient failure-injection lane의 main 단계 중복 실행은 이전 결정대로 남겼다. 실제 CI runner 실행은 확인하지 못했다. 검증: ci, mainCheck, :integrationCheck, -p optional-platforms ci integrationCheck performanceCheck, :performanceCheck, :releaseCheck (-PgitRevision), verify-compose-profile-contracts(18 lane) 통과. 첫 실행의 실패 2건(FindSecBugs 1, sample 동시 생성 테스트의 v4 event id)은 고친 뒤 ci·integrationCheck·mainCheck를 다시 돌려 통과했다. Refs: docs/reviews/2026-11-03-rereview/README.md 3·4·6·7절 BREAKING CHANGE: outbox 발행 포트와 저장소 포트 시그니처, 순서 번호 첫 값(V21 floor 다음), 샘플 데모 계정 프로필 허용 목록, 새 오류 코드 IDEMPOTENCY_COMPLETION_UNKNOWN, prod의 fixture- 값 거부, 기동 rule은 static·의존 없음이어야 한다.
180 lines
9.1 KiB
Markdown
180 lines
9.1 KiB
Markdown
# Runbook: gRPC platform operations
|
|
|
|
Scope: the `:grpc:*` family. All of it is build-only today — every leaf's `runtime_memberships` is
|
|
empty — so nothing here fires in production yet. It is written now because the states it covers are
|
|
the ones an on-call cannot work out from first principles at three in the morning, and shipping the
|
|
behaviour before the runbook means the first person to meet one is doing that.
|
|
|
|
Configuration lives under `app.grpc.platform.*` and is bound by `GrpcPlatformProperties`.
|
|
The platform does not start unless `app.grpc.platform.enabled=true`.
|
|
|
|
---
|
|
|
|
## COMPLETION_UNKNOWN on a mutation
|
|
|
|
**What you are seeing.** A client received `DEADLINE_EXCEEDED`, `UNAVAILABLE` or `INTERNAL` on a
|
|
state-changing call, and the response trailer `completion-outcome` reads `COMPLETION_UNKNOWN`.
|
|
|
|
**What it means.** The server may have committed. This is not a failure and not a success; the status
|
|
code cannot distinguish them, which is why the outcome is carried separately.
|
|
|
|
**What not to do.** Do not re-issue the call. Do not tell the caller it failed. Both are wrong half
|
|
the time, and which half is not knowable from the status.
|
|
|
|
**What to do.**
|
|
|
|
1. Take the `error-execution-id` from the trailers. It is the only link between what the client saw
|
|
and what the server did.
|
|
2. If the method is `IDEMPOTENCY_KEY_REQUIRED`, query the operation ledger with the caller
|
|
fingerprint, the full method name and the caller's key. `GrpcOperationStatusQuery` returns one of
|
|
`IN_PROGRESS`, `COMMITTED`, `FAILED_TERMINAL`, `NOT_FOUND` or `UNKNOWN`.
|
|
3. `COMMITTED` means return the stored outcome reference, not a freshly computed answer — the resource
|
|
may have changed since, and a new answer would describe the state at reconciliation time rather
|
|
than the state the caller's own call produced.
|
|
4. `NOT_FOUND` means the operation never started and is safe to re-issue. `FAILED_TERMINAL` means the
|
|
same.
|
|
5. `UNKNOWN` means the ledger could not be consulted. Nothing may be concluded. The case is queued by
|
|
`GrpcCompletionReconciler` and retried later.
|
|
6. If the method is not keyed, there is no ledger row. Resolve it against the business resource, or
|
|
escalate to the service owner. This is the case the keyed profile exists to avoid.
|
|
|
|
7. `IN_PROGRESS` means a claim exists and has not finished. If its owner died, the claim is not
|
|
stuck forever: once it is older than the claim TTL (`app.grpc.platform.operation-ledger-claim-ttl`,
|
|
default 5m), the next retry of the same request takes it over and runs again,
|
|
and a later reconciler pass sees that attempt's `COMMITTED` or `FAILED_TERMINAL`. Until a retry
|
|
arrives, the case stays pending. The re-run is safe only when the ledger commit is recorded in the
|
|
mutation's own transaction; if this service commits them separately, check the business resource
|
|
before letting the retry through. An owner that was only slow cannot spoil the takeover: its claim
|
|
attempt (`claim_attempt`) is behind, so its late commit is refused and its late failure ignored.
|
|
A log line saying an operation "is no longer held by attempt N" is that refusal, not an outage.
|
|
|
|
**Escalate when** the reconciler's pending list grows across passes. That means the ledger is
|
|
unreachable rather than slow. A case pending as `IN_PROGRESS` for longer than the claim TTL with no
|
|
retry arriving is a caller that gave up; it needs the business resource checked, not a ledger fix.
|
|
|
|
---
|
|
|
|
## The operation ledger table keeps growing
|
|
|
|
**What you are seeing.** `grpc_operation_ledger` grows without bound, or a retry of an old key runs
|
|
the mutation again instead of replaying.
|
|
|
|
**What it means.** Finished rows (`COMMITTED`, `FAILED_TERMINAL`) are deleted only when something
|
|
calls `GrpcOperationLedgerRetention.purgeExpired(now)`; the starter assembles that bean
|
|
(`app.grpc.platform.operation-ledger-enabled=true`) but does not schedule it. A growing table means
|
|
the service's scheduler is not calling it. A retry that re-ran after the retention
|
|
(`app.grpc.platform.operation-ledger-retention`, default 24h) is the trade the retention makes: its
|
|
record was purged, so it was a new operation.
|
|
|
|
**What to do.**
|
|
|
|
1. Check that the service schedules `purgeExpired`. Each call deletes in batches of 500 until a batch
|
|
comes back short, never touching `IN_PROGRESS` rows.
|
|
2. For a large backlog, call it repeatedly; each batch is a short transaction that skips rows another
|
|
transaction holds.
|
|
3. If retried keys re-run after the retention, raise the retention above the callers' retry window
|
|
rather than skipping the purge.
|
|
|
|
**Escalate when** old `IN_PROGRESS` rows accumulate. Those are not purged by design; each is an
|
|
abandoned claim no retry has taken over (see `IN_PROGRESS` above).
|
|
|
|
---
|
|
|
|
## A stream ended with FULL_RESYNC_REQUIRED
|
|
|
|
**What you are seeing.** A client's resume was refused and it was told to resynchronise.
|
|
|
|
**What it means.** The server can no longer replay from the client's cursor. Either the snapshot
|
|
version moved, or the cursor predates retained history.
|
|
|
|
**What to do.** Nothing on the server. The client is expected to discard its position and start a new
|
|
stream from a fresh snapshot. A client that instead retries the same token will keep receiving the
|
|
same answer.
|
|
|
|
**Escalate when** it is happening to many clients at once. That usually means history retention was
|
|
reduced, or snapshots are rotating faster than clients reconnect.
|
|
|
|
---
|
|
|
|
## A stream ended with SLOW_CONSUMER
|
|
|
|
**What you are seeing.** Streams terminating with `SLOW_CONSUMER`, and
|
|
`grpc.stream.flow_control_stalls` rising.
|
|
|
|
**What it means.** The consumer could not keep up with the bounded queue. The stream was terminated
|
|
rather than silently dropping messages, because a client cannot detect drops — the sequence numbers
|
|
it sees are the ones it was sent.
|
|
|
|
**What to do.**
|
|
|
|
1. Check whether the consumer is slow or the producer is fast. `grpc.stream.messages` against
|
|
`grpc.stream.lifetime` tells you the rate.
|
|
2. If the consumer is slow, the fix is on the consumer. Raising the queue bound moves the failure
|
|
later and makes it larger.
|
|
3. A resume is not available after this ending: the messages that overflowed the queue are gone, so
|
|
continuing from the last delivered sequence would silently skip them. The client resynchronises.
|
|
|
|
---
|
|
|
|
## RESOURCE_EXHAUSTED under load
|
|
|
|
**What you are seeing.** Calls refused with `RESOURCE_EXHAUSTED` and `GrpcAdmissionController`
|
|
reporting rejections.
|
|
|
|
**What it means.** The server is at its concurrency and queue bounds and is shedding rather than
|
|
queueing. This is the designed behaviour: accepting work whose callers have already given up spends
|
|
capacity on nothing.
|
|
|
|
**What to do.**
|
|
|
|
1. Read `grpc.rpc.duration` and `grpc.rpc.queue_wait` separately. Queue time rising with duration flat
|
|
means the bottleneck is admission, not the work.
|
|
2. Check which saturation counter is moving — executor, channel or flow control. They look identical
|
|
in a latency graph and have different fixes.
|
|
3. Raising `app.grpc.platform.executor-queue-capacity` defers the problem; it does not remove
|
|
it. `GrpcExecutorProfile` refuses a queue above ten thousand for that reason.
|
|
|
|
---
|
|
|
|
## A rollout is producing errors at every deploy
|
|
|
|
**What you are seeing.** A burst of `UNAVAILABLE` or `CANCELLED` each time an instance goes away.
|
|
|
|
**What it means.** The drain sequence is not completing, or is running out of order.
|
|
|
|
**What to do.**
|
|
|
|
1. `GrpcDrainResult` records what each drain achieved: completed and cancelled unary calls, signalled
|
|
and cancelled streams, and which phases ran. A drain that routinely force-cancels is the cause.
|
|
2. The order matters. Readiness flips first and nothing is refused during that window, because there
|
|
is a gap between an instance reporting unready and routing acting on it. Refusing during that gap
|
|
turns a clean rollout into a burst of errors at every deploy.
|
|
3. For long streams, check the Kubernetes profile's `streamReconnectBudget` and
|
|
`readinessDrainGrace`. A stream is pinned to one pod for its whole life, so every rollout ends it;
|
|
a profile with long streams and no reconnect budget has not decided what clients do next.
|
|
|
|
---
|
|
|
|
## Verifying a deployment's configuration
|
|
|
|
`GrpcPlatformSnapshotService` produces a secret-free snapshot for a caller on the admin network
|
|
holding an admin role. Both gates are required.
|
|
|
|
`GrpcPlatformSnapshotService.driftAgainstRelease` compares a running snapshot with the release
|
|
manifest and reports schema version, method policy hash and per-channel profile differences. An
|
|
instance running a configuration the release did not ship is behind a whole class of incidents that
|
|
are otherwise diagnosed by reading logs.
|
|
|
|
The snapshot carries hashes and names only. A field whose name looks like a credential is refused at
|
|
construction rather than redacted.
|
|
|
|
---
|
|
|
|
## Things that are deliberately off
|
|
|
|
- **Reflection in production.** `GrpcReflectionMode.defaultFor` returns `DISABLED` for `STAGE` and
|
|
`PROD`. Reflection publishes the whole schema to anyone who can open a connection.
|
|
- **Every advanced capability.** See `docs/runbooks/grpc-advanced-capabilities.md`.
|
|
- **The platform itself.** `app.grpc.platform.enabled` defaults to false, and every `:grpc:*`
|
|
leaf is build-only in the registry.
|