# Runbook: gRPC advanced capabilities

Scope: the `:grpc-advanced:*` family. Everything here is off by default and stays off until a
deployment names it. Nothing in this family ships in a runtime composition today.

Flags are `ca-skeleton.grpc.advanced.<capability>.enabled`. `GrpcAdvancedCapability` owns the list of
capability names; `GrpcAdvancedSupportMatrix` owns their current grades.

---

## A capability refuses to start

`GrpcAdvancedModuleGuard` gives three different refusals, and the remedy differs:

| Message contains | Meaning | Remedy |
| --- | --- | --- |
| "its feature flag is not set" | Nobody enabled it | Set the property named in the message |
| "tracked rather than implemented" | Grade is `WATCH` | Nothing to do here; the capability is not implemented |
| "uncharacterised failure modes" | Grade is `EXPERIMENTAL` and this is production | Record a production approval, or run it outside production |

The refusal message always names the property key, so the first case is a configuration line rather
than a support question.

---

## xDS: the control plane went away

**What you are seeing.** The control plane is unreachable and clients are still routing.

**What it means.** `GrpcXdsFailurePolicy` serves the last-known-good snapshot, up to its staleness
bound.

**What to do.**

1. Check the snapshot's age. `SERVE_LAST_KNOWN_GOOD` is the healthy degraded state.
2. `STALE_BEYOND_BOUND` means the snapshot is older than the policy allows and is no longer trusted.
   Beyond that bound, a decommissioned backend would otherwise keep receiving traffic indefinitely.
3. `NO_SNAPSHOT_YET` on a starting instance means it never reached the control plane. It fails after
   the initial fetch timeout rather than starting with no routing.

**Do not** add application-level retry policy while xDS is enabled. `GrpcXdsStartupGuard` refuses it,
because retry defined in two places has a winner that depends on resolution order rather than on a
decision.

---

## gRPC-Web: a browser call hangs and then fails with no status

**Almost always the proxy.** A gRPC status arrives as a trailer, and a browser cannot read a trailer
the proxy did not expose. Check that the proxy's CORS `expose_headers` includes `grpc-status` and
`grpc-message`; `GrpcWebProxyContract.violations` reports exactly this, and the reference
configuration in `envoy/envoy.yaml` shows it in place.

**If the method is client- or bidirectional-streaming**, it cannot work over gRPC-Web at all — a
browser has no way to send a stream of messages. `GrpcWebCompatibilityGate` reports such a method
before it is exposed.

---

## Servlet: a transport setting appears to be ignored

It is ignored. The container owns the socket, so keepalive tuning, maximum connection age and
flow-control window tuning belong to it. `GrpcServletStartupValidator` refuses those settings at
startup rather than accepting and dropping them, because a setting that is silently ignored sends the
investigation somewhere else.

A Servlet run never substitutes for Netty certification.

---

## Hedging: backend load doubled

**Expected, within a bound.** Hedging trades duplicate load for tail latency.
`GrpcHedgingResult` records both `duplicateBackendCalls` and `cancelledLoserAttempts`; a dashboard
showing only the latency improvement makes the trade look free.

**What to check.** `GrpcHedgingBudget` caps hedges as a fraction of completed calls. If duplicate
load is above that fraction, the budget is not being consumed — which means something is issuing
hedges outside the coordinator.

**Hedging is refused** for anything but a read-only unary method. A hedged mutation runs twice by
design, and an idempotency key does not help: the second attempt duplicates a success in progress
rather than retrying a failure.

---

## Promoting a capability

`GrpcAdvancedPromotionGate.evaluate` names every missing item. Promotion to `ADVANCED_STABLE` needs
compatibility evidence, a security review, fault evidence, performance evidence, an ADR, a runbook, a
real-environment test and seven days of soak. A Stable default needs thirty.

Promotions are independent: promoting one capability changes no other's grade, and
`GrpcAdvancedSupportMatrix.apply` refuses a decision made against a different matrix state.

Before citing a suite as evidence, check `GrpcAdvancedInfrastructureTestkit.missingInfrastructure`.
A suite that ran without the proxy, the container, the control plane or the toolchain it needs passed
and established nothing.