# Runbook: gRPC platform operations Scope: the `:grpc:*` family. All of it is build-only today — every leaf's `runtime_memberships` is empty — so nothing here fires in production yet. It is written now because the states it covers are the ones an on-call cannot work out from first principles at three in the morning, and shipping the behaviour before the runbook means the first person to meet one is doing that. Configuration lives under `ca-skeleton.grpc.platform.*` and is bound by `GrpcPlatformProperties`. The platform does not start unless `ca-skeleton.grpc.platform.enabled=true`. --- ## COMPLETION_UNKNOWN on a mutation **What you are seeing.** A client received `DEADLINE_EXCEEDED`, `UNAVAILABLE` or `INTERNAL` on a state-changing call, and the response trailer `completion-outcome` reads `COMPLETION_UNKNOWN`. **What it means.** The server may have committed. This is not a failure and not a success; the status code cannot distinguish them, which is why the outcome is carried separately. **What not to do.** Do not re-issue the call. Do not tell the caller it failed. Both are wrong half the time, and which half is not knowable from the status. **What to do.** 1. Take the `error-execution-id` from the trailers. It is the only link between what the client saw and what the server did. 2. If the method is `IDEMPOTENCY_KEY_REQUIRED`, query the operation ledger with the caller fingerprint, the full method name and the caller's key. `GrpcOperationStatusQuery` returns one of `IN_PROGRESS`, `COMMITTED`, `FAILED_TERMINAL`, `NOT_FOUND` or `UNKNOWN`. 3. `COMMITTED` means return the stored outcome reference, not a freshly computed answer — the resource may have changed since, and a new answer would describe the state at reconciliation time rather than the state the caller's own call produced. 4. `NOT_FOUND` means the operation never started and is safe to re-issue. `FAILED_TERMINAL` means the same. 5. `UNKNOWN` means the ledger could not be consulted. Nothing may be concluded. The case is queued by `GrpcCompletionReconciler` and retried later. 6. If the method is not keyed, there is no ledger row. Resolve it against the business resource, or escalate to the service owner. This is the case the keyed profile exists to avoid. **Escalate when** the reconciler's pending list grows across passes. That means the ledger is unreachable rather than slow. --- ## A stream ended with FULL_RESYNC_REQUIRED **What you are seeing.** A client's resume was refused and it was told to resynchronise. **What it means.** The server can no longer replay from the client's cursor. Either the snapshot version moved, or the cursor predates retained history. **What to do.** Nothing on the server. The client is expected to discard its position and start a new stream from a fresh snapshot. A client that instead retries the same token will keep receiving the same answer. **Escalate when** it is happening to many clients at once. That usually means history retention was reduced, or snapshots are rotating faster than clients reconnect. --- ## A stream ended with SLOW_CONSUMER **What you are seeing.** Streams terminating with `SLOW_CONSUMER`, and `grpc.stream.flow_control_stalls` rising. **What it means.** The consumer could not keep up with the bounded queue. The stream was terminated rather than silently dropping messages, because a client cannot detect drops — the sequence numbers it sees are the ones it was sent. **What to do.** 1. Check whether the consumer is slow or the producer is fast. `grpc.stream.messages` against `grpc.stream.lifetime` tells you the rate. 2. If the consumer is slow, the fix is on the consumer. Raising the queue bound moves the failure later and makes it larger. 3. A resume is not available after this ending: the messages that overflowed the queue are gone, so continuing from the last delivered sequence would silently skip them. The client resynchronises. --- ## RESOURCE_EXHAUSTED under load **What you are seeing.** Calls refused with `RESOURCE_EXHAUSTED` and `GrpcAdmissionController` reporting rejections. **What it means.** The server is at its concurrency and queue bounds and is shedding rather than queueing. This is the designed behaviour: accepting work whose callers have already given up spends capacity on nothing. **What to do.** 1. Read `grpc.rpc.duration` and `grpc.rpc.queue_wait` separately. Queue time rising with duration flat means the bottleneck is admission, not the work. 2. Check which saturation counter is moving — executor, channel or flow control. They look identical in a latency graph and have different fixes. 3. Raising `ca-skeleton.grpc.platform.executor-queue-capacity` defers the problem; it does not remove it. `GrpcExecutorProfile` refuses a queue above ten thousand for that reason. --- ## A rollout is producing errors at every deploy **What you are seeing.** A burst of `UNAVAILABLE` or `CANCELLED` each time an instance goes away. **What it means.** The drain sequence is not completing, or is running out of order. **What to do.** 1. `GrpcDrainResult` records what each drain achieved: completed and cancelled unary calls, signalled and cancelled streams, and which phases ran. A drain that routinely force-cancels is the cause. 2. The order matters. Readiness flips first and nothing is refused during that window, because there is a gap between an instance reporting unready and routing acting on it. Refusing during that gap turns a clean rollout into a burst of errors at every deploy. 3. For long streams, check the Kubernetes profile's `streamReconnectBudget` and `readinessDrainGrace`. A stream is pinned to one pod for its whole life, so every rollout ends it; a profile with long streams and no reconnect budget has not decided what clients do next. --- ## Verifying a deployment's configuration `GrpcPlatformSnapshotService` produces a secret-free snapshot for a caller on the admin network holding an admin role. Both gates are required. `GrpcPlatformSnapshotService.driftAgainstRelease` compares a running snapshot with the release manifest and reports schema version, method policy hash and per-channel profile differences. An instance running a configuration the release did not ship is behind a whole class of incidents that are otherwise diagnosed by reading logs. The snapshot carries hashes and names only. A field whose name looks like a credential is refused at construction rather than redacted. --- ## Things that are deliberately off - **Reflection in production.** `GrpcReflectionMode.defaultFor` returns `DISABLED` for `STAGE` and `PROD`. Reflection publishes the whole schema to anyone who can open a connection. - **Every advanced capability.** See `docs/runbooks/grpc-advanced-capabilities.md`. - **The platform itself.** `ca-skeleton.grpc.platform.enabled` defaults to false, and every `:grpc:*` leaf is build-only in the registry.