143 lines
6.8 KiB
Markdown
143 lines
6.8 KiB
Markdown
# Runbook: gRPC platform operations
|
|
|
|
Scope: the `:grpc:*` family. All of it is build-only today — every leaf's `runtime_memberships` is
|
|
empty — so nothing here fires in production yet. It is written now because the states it covers are
|
|
the ones an on-call cannot work out from first principles at three in the morning, and shipping the
|
|
behaviour before the runbook means the first person to meet one is doing that.
|
|
|
|
Configuration lives under `ca-skeleton.grpc.platform.*` and is bound by `GrpcPlatformProperties`.
|
|
The platform does not start unless `ca-skeleton.grpc.platform.enabled=true`.
|
|
|
|
---
|
|
|
|
## COMPLETION_UNKNOWN on a mutation
|
|
|
|
**What you are seeing.** A client received `DEADLINE_EXCEEDED`, `UNAVAILABLE` or `INTERNAL` on a
|
|
state-changing call, and the response trailer `completion-outcome` reads `COMPLETION_UNKNOWN`.
|
|
|
|
**What it means.** The server may have committed. This is not a failure and not a success; the status
|
|
code cannot distinguish them, which is why the outcome is carried separately.
|
|
|
|
**What not to do.** Do not re-issue the call. Do not tell the caller it failed. Both are wrong half
|
|
the time, and which half is not knowable from the status.
|
|
|
|
**What to do.**
|
|
|
|
1. Take the `error-execution-id` from the trailers. It is the only link between what the client saw
|
|
and what the server did.
|
|
2. If the method is `IDEMPOTENCY_KEY_REQUIRED`, query the operation ledger with the caller
|
|
fingerprint, the full method name and the caller's key. `GrpcOperationStatusQuery` returns one of
|
|
`IN_PROGRESS`, `COMMITTED`, `FAILED_TERMINAL`, `NOT_FOUND` or `UNKNOWN`.
|
|
3. `COMMITTED` means return the stored outcome reference, not a freshly computed answer — the resource
|
|
may have changed since, and a new answer would describe the state at reconciliation time rather
|
|
than the state the caller's own call produced.
|
|
4. `NOT_FOUND` means the operation never started and is safe to re-issue. `FAILED_TERMINAL` means the
|
|
same.
|
|
5. `UNKNOWN` means the ledger could not be consulted. Nothing may be concluded. The case is queued by
|
|
`GrpcCompletionReconciler` and retried later.
|
|
6. If the method is not keyed, there is no ledger row. Resolve it against the business resource, or
|
|
escalate to the service owner. This is the case the keyed profile exists to avoid.
|
|
|
|
**Escalate when** the reconciler's pending list grows across passes. That means the ledger is
|
|
unreachable rather than slow.
|
|
|
|
---
|
|
|
|
## A stream ended with FULL_RESYNC_REQUIRED
|
|
|
|
**What you are seeing.** A client's resume was refused and it was told to resynchronise.
|
|
|
|
**What it means.** The server can no longer replay from the client's cursor. Either the snapshot
|
|
version moved, or the cursor predates retained history.
|
|
|
|
**What to do.** Nothing on the server. The client is expected to discard its position and start a new
|
|
stream from a fresh snapshot. A client that instead retries the same token will keep receiving the
|
|
same answer.
|
|
|
|
**Escalate when** it is happening to many clients at once. That usually means history retention was
|
|
reduced, or snapshots are rotating faster than clients reconnect.
|
|
|
|
---
|
|
|
|
## A stream ended with SLOW_CONSUMER
|
|
|
|
**What you are seeing.** Streams terminating with `SLOW_CONSUMER`, and
|
|
`grpc.stream.flow_control_stalls` rising.
|
|
|
|
**What it means.** The consumer could not keep up with the bounded queue. The stream was terminated
|
|
rather than silently dropping messages, because a client cannot detect drops — the sequence numbers
|
|
it sees are the ones it was sent.
|
|
|
|
**What to do.**
|
|
|
|
1. Check whether the consumer is slow or the producer is fast. `grpc.stream.messages` against
|
|
`grpc.stream.lifetime` tells you the rate.
|
|
2. If the consumer is slow, the fix is on the consumer. Raising the queue bound moves the failure
|
|
later and makes it larger.
|
|
3. A resume is not available after this ending: the messages that overflowed the queue are gone, so
|
|
continuing from the last delivered sequence would silently skip them. The client resynchronises.
|
|
|
|
---
|
|
|
|
## RESOURCE_EXHAUSTED under load
|
|
|
|
**What you are seeing.** Calls refused with `RESOURCE_EXHAUSTED` and `GrpcAdmissionController`
|
|
reporting rejections.
|
|
|
|
**What it means.** The server is at its concurrency and queue bounds and is shedding rather than
|
|
queueing. This is the designed behaviour: accepting work whose callers have already given up spends
|
|
capacity on nothing.
|
|
|
|
**What to do.**
|
|
|
|
1. Read `grpc.rpc.duration` and `grpc.rpc.queue_wait` separately. Queue time rising with duration flat
|
|
means the bottleneck is admission, not the work.
|
|
2. Check which saturation counter is moving — executor, channel or flow control. They look identical
|
|
in a latency graph and have different fixes.
|
|
3. Raising `ca-skeleton.grpc.platform.executor-queue-capacity` defers the problem; it does not remove
|
|
it. `GrpcExecutorProfile` refuses a queue above ten thousand for that reason.
|
|
|
|
---
|
|
|
|
## A rollout is producing errors at every deploy
|
|
|
|
**What you are seeing.** A burst of `UNAVAILABLE` or `CANCELLED` each time an instance goes away.
|
|
|
|
**What it means.** The drain sequence is not completing, or is running out of order.
|
|
|
|
**What to do.**
|
|
|
|
1. `GrpcDrainResult` records what each drain achieved: completed and cancelled unary calls, signalled
|
|
and cancelled streams, and which phases ran. A drain that routinely force-cancels is the cause.
|
|
2. The order matters. Readiness flips first and nothing is refused during that window, because there
|
|
is a gap between an instance reporting unready and routing acting on it. Refusing during that gap
|
|
turns a clean rollout into a burst of errors at every deploy.
|
|
3. For long streams, check the Kubernetes profile's `streamReconnectBudget` and
|
|
`readinessDrainGrace`. A stream is pinned to one pod for its whole life, so every rollout ends it;
|
|
a profile with long streams and no reconnect budget has not decided what clients do next.
|
|
|
|
---
|
|
|
|
## Verifying a deployment's configuration
|
|
|
|
`GrpcPlatformSnapshotService` produces a secret-free snapshot for a caller on the admin network
|
|
holding an admin role. Both gates are required.
|
|
|
|
`GrpcPlatformSnapshotService.driftAgainstRelease` compares a running snapshot with the release
|
|
manifest and reports schema version, method policy hash and per-channel profile differences. An
|
|
instance running a configuration the release did not ship is behind a whole class of incidents that
|
|
are otherwise diagnosed by reading logs.
|
|
|
|
The snapshot carries hashes and names only. A field whose name looks like a credential is refused at
|
|
construction rather than redacted.
|
|
|
|
---
|
|
|
|
## Things that are deliberately off
|
|
|
|
- **Reflection in production.** `GrpcReflectionMode.defaultFor` returns `DISABLED` for `STAGE` and
|
|
`PROD`. Reflection publishes the whole schema to anyone who can open a connection.
|
|
- **Every advanced capability.** See `docs/runbooks/grpc-advanced-capabilities.md`.
|
|
- **The platform itself.** `ca-skeleton.grpc.platform.enabled` defaults to false, and every `:grpc:*`
|
|
leaf is build-only in the registry.
|