4.4 KiB
Runbook: gRPC advanced capabilities
Scope: the :grpc-advanced:* family. Everything here is off by default and stays off until a
deployment names it. Nothing in this family ships in a runtime composition today.
Flags are ca-skeleton.grpc.advanced.<capability>.enabled. GrpcAdvancedCapability owns the list of
capability names; GrpcAdvancedSupportMatrix owns their current grades.
A capability refuses to start
GrpcAdvancedModuleGuard gives three different refusals, and the remedy differs:
| Message contains | Meaning | Remedy |
|---|---|---|
| "its feature flag is not set" | Nobody enabled it | Set the property named in the message |
| "tracked rather than implemented" | Grade is WATCH |
Nothing to do here; the capability is not implemented |
| "uncharacterised failure modes" | Grade is EXPERIMENTAL and this is production |
Record a production approval, or run it outside production |
The refusal message always names the property key, so the first case is a configuration line rather than a support question.
xDS: the control plane went away
What you are seeing. The control plane is unreachable and clients are still routing.
What it means. GrpcXdsFailurePolicy serves the last-known-good snapshot, up to its staleness
bound.
What to do.
- Check the snapshot's age.
SERVE_LAST_KNOWN_GOODis the healthy degraded state. STALE_BEYOND_BOUNDmeans the snapshot is older than the policy allows and is no longer trusted. Beyond that bound, a decommissioned backend would otherwise keep receiving traffic indefinitely.NO_SNAPSHOT_YETon a starting instance means it never reached the control plane. It fails after the initial fetch timeout rather than starting with no routing.
Do not add application-level retry policy while xDS is enabled. GrpcXdsStartupGuard refuses it,
because retry defined in two places has a winner that depends on resolution order rather than on a
decision.
gRPC-Web: a browser call hangs and then fails with no status
Almost always the proxy. A gRPC status arrives as a trailer, and a browser cannot read a trailer
the proxy did not expose. Check that the proxy's CORS expose_headers includes grpc-status and
grpc-message; GrpcWebProxyContract.violations reports exactly this, and the reference
configuration in envoy/envoy.yaml shows it in place.
If the method is client- or bidirectional-streaming, it cannot work over gRPC-Web at all — a
browser has no way to send a stream of messages. GrpcWebCompatibilityGate reports such a method
before it is exposed.
Servlet: a transport setting appears to be ignored
It is ignored. The container owns the socket, so keepalive tuning, maximum connection age and
flow-control window tuning belong to it. GrpcServletStartupValidator refuses those settings at
startup rather than accepting and dropping them, because a setting that is silently ignored sends the
investigation somewhere else.
A Servlet run never substitutes for Netty certification.
Hedging: backend load doubled
Expected, within a bound. Hedging trades duplicate load for tail latency.
GrpcHedgingResult records both duplicateBackendCalls and cancelledLoserAttempts; a dashboard
showing only the latency improvement makes the trade look free.
What to check. GrpcHedgingBudget caps hedges as a fraction of completed calls. If duplicate
load is above that fraction, the budget is not being consumed — which means something is issuing
hedges outside the coordinator.
Hedging is refused for anything but a read-only unary method. A hedged mutation runs twice by design, and an idempotency key does not help: the second attempt duplicates a success in progress rather than retrying a failure.
Promoting a capability
GrpcAdvancedPromotionGate.evaluate names every missing item. Promotion to ADVANCED_STABLE needs
compatibility evidence, a security review, fault evidence, performance evidence, an ADR, a runbook, a
real-environment test and seven days of soak. A Stable default needs thirty.
Promotions are independent: promoting one capability changes no other's grade, and
GrpcAdvancedSupportMatrix.apply refuses a decision made against a different matrix state.
Before citing a suite as evidence, check GrpcAdvancedInfrastructureTestkit.missingInfrastructure.
A suite that ran without the proxy, the container, the control plane or the toolchain it needs passed
and established nothing.