99 lines
4.4 KiB
Markdown
99 lines
4.4 KiB
Markdown
# Runbook: gRPC advanced capabilities
|
|
|
|
Scope: the `:grpc-advanced:*` family. Everything here is off by default and stays off until a
|
|
deployment names it. Nothing in this family ships in a runtime composition today.
|
|
|
|
Flags are `ca-skeleton.grpc.advanced.<capability>.enabled`. `GrpcAdvancedCapability` owns the list of
|
|
capability names; `GrpcAdvancedSupportMatrix` owns their current grades.
|
|
|
|
---
|
|
|
|
## A capability refuses to start
|
|
|
|
`GrpcAdvancedModuleGuard` gives three different refusals, and the remedy differs:
|
|
|
|
| Message contains | Meaning | Remedy |
|
|
| --- | --- | --- |
|
|
| "its feature flag is not set" | Nobody enabled it | Set the property named in the message |
|
|
| "tracked rather than implemented" | Grade is `WATCH` | Nothing to do here; the capability is not implemented |
|
|
| "uncharacterised failure modes" | Grade is `EXPERIMENTAL` and this is production | Record a production approval, or run it outside production |
|
|
|
|
The refusal message always names the property key, so the first case is a configuration line rather
|
|
than a support question.
|
|
|
|
---
|
|
|
|
## xDS: the control plane went away
|
|
|
|
**What you are seeing.** The control plane is unreachable and clients are still routing.
|
|
|
|
**What it means.** `GrpcXdsFailurePolicy` serves the last-known-good snapshot, up to its staleness
|
|
bound.
|
|
|
|
**What to do.**
|
|
|
|
1. Check the snapshot's age. `SERVE_LAST_KNOWN_GOOD` is the healthy degraded state.
|
|
2. `STALE_BEYOND_BOUND` means the snapshot is older than the policy allows and is no longer trusted.
|
|
Beyond that bound, a decommissioned backend would otherwise keep receiving traffic indefinitely.
|
|
3. `NO_SNAPSHOT_YET` on a starting instance means it never reached the control plane. It fails after
|
|
the initial fetch timeout rather than starting with no routing.
|
|
|
|
**Do not** add application-level retry policy while xDS is enabled. `GrpcXdsStartupGuard` refuses it,
|
|
because retry defined in two places has a winner that depends on resolution order rather than on a
|
|
decision.
|
|
|
|
---
|
|
|
|
## gRPC-Web: a browser call hangs and then fails with no status
|
|
|
|
**Almost always the proxy.** A gRPC status arrives as a trailer, and a browser cannot read a trailer
|
|
the proxy did not expose. Check that the proxy's CORS `expose_headers` includes `grpc-status` and
|
|
`grpc-message`; `GrpcWebProxyContract.violations` reports exactly this, and the reference
|
|
configuration in `envoy/envoy.yaml` shows it in place.
|
|
|
|
**If the method is client- or bidirectional-streaming**, it cannot work over gRPC-Web at all — a
|
|
browser has no way to send a stream of messages. `GrpcWebCompatibilityGate` reports such a method
|
|
before it is exposed.
|
|
|
|
---
|
|
|
|
## Servlet: a transport setting appears to be ignored
|
|
|
|
It is ignored. The container owns the socket, so keepalive tuning, maximum connection age and
|
|
flow-control window tuning belong to it. `GrpcServletStartupValidator` refuses those settings at
|
|
startup rather than accepting and dropping them, because a setting that is silently ignored sends the
|
|
investigation somewhere else.
|
|
|
|
A Servlet run never substitutes for Netty certification.
|
|
|
|
---
|
|
|
|
## Hedging: backend load doubled
|
|
|
|
**Expected, within a bound.** Hedging trades duplicate load for tail latency.
|
|
`GrpcHedgingResult` records both `duplicateBackendCalls` and `cancelledLoserAttempts`; a dashboard
|
|
showing only the latency improvement makes the trade look free.
|
|
|
|
**What to check.** `GrpcHedgingBudget` caps hedges as a fraction of completed calls. If duplicate
|
|
load is above that fraction, the budget is not being consumed — which means something is issuing
|
|
hedges outside the coordinator.
|
|
|
|
**Hedging is refused** for anything but a read-only unary method. A hedged mutation runs twice by
|
|
design, and an idempotency key does not help: the second attempt duplicates a success in progress
|
|
rather than retrying a failure.
|
|
|
|
---
|
|
|
|
## Promoting a capability
|
|
|
|
`GrpcAdvancedPromotionGate.evaluate` names every missing item. Promotion to `ADVANCED_STABLE` needs
|
|
compatibility evidence, a security review, fault evidence, performance evidence, an ADR, a runbook, a
|
|
real-environment test and seven days of soak. A Stable default needs thirty.
|
|
|
|
Promotions are independent: promoting one capability changes no other's grade, and
|
|
`GrpcAdvancedSupportMatrix.apply` refuses a decision made against a different matrix state.
|
|
|
|
Before citing a suite as evidence, check `GrpcAdvancedInfrastructureTestkit.missingInfrastructure`.
|
|
A suite that ran without the proxy, the container, the control plane or the toolchain it needs passed
|
|
and established nothing.
|