4.1 KiB
ADR-GRPC-006: Stable discovery is DNS and static, and a Kubernetes profile names who balances
- Status: accepted
- Date: 2026-08-31
- Scope:
:grpc:grpc-discovery,:grpc:grpc-client
Context
A gRPC channel's discovery configuration has a failure mode with no runtime symptom: it works, and it does not do what the dashboard says it does.
The specific case is round_robin over a Kubernetes Service ClusterIP. The Service is one virtual
address, so the resolver returns one endpoint and the client-side balancer has nothing to rotate
across; kube-proxy picks a pod at connect time, and an HTTP/2 connection is long-lived, so every
request from that client goes to the same pod for the life of the connection. Nothing fails. The
configuration says round_robin, the metrics show requests spread across clients rather than pods,
and the conclusion "we have client-side load balancing" is wrong in a way nobody is prompted to
check.
The mirror-image mistake is pick_first over a headless record, which pins a client to one pod out
of many.
Separately, a service mesh changes who owns retries, and a deployment that adds mesh routing without removing its own retry policy has two retriers whose effects multiply.
Decision
Stable resolvers are Static, DNS and Unix domain socket; Stable load balancing is pick_first and
round_robin. GrpcDiscoveryPolicyValidator.requireStableScheme refuses xds, consul, etcd
and eureka by name, with a message saying they are Advanced capabilities with their own control
plane and promotion gate rather than unknown schemes.
The pairing is checked against the resolved address count, not against intent.
GrpcResolverProfile carries expectedAddressCount, and GrpcStableLoadBalancer.effective answers
whether the policy distributes anything over that many endpoints. A round_robin profile over one
address is a reported violation whose message says it describes spreading that is not happening.
A Kubernetes deployment names its routing mode, and the mode implies both the balancer and the
retry owner. GrpcKubernetesRoutingMode has three values — K8S_VIP, K8S_HEADLESS, MESH — and
GrpcKubernetesProfile refuses a mesh profile whose retry owner retries in-process.
A profile that carries long-lived streams must state a reconnect budget and a readiness drain
grace. A stream pins a client to one pod for its whole life, so every rollout, eviction and
scale-down ends it. GrpcKubernetesProfileValidator additionally reports a VIP profile carrying
long streams, and a drain grace shorter than the reconnect budget — the second means the pod stops
serving before its clients have finished reconnecting elsewhere.
A DNS profile must refresh. GrpcResolverProfile refuses a zero refresh interval on DNS,
because a channel that resolved once at startup keeps sending to addresses that stopped existing an
hour ago, and the resulting UNAVAILABLE looks like an unhealthy deployment long after the rollout
finished.
Consequences
Two validators, not one. GrpcDiscoveryPolicyValidator asks whether a balancer does anything
over the addresses it will see; GrpcKubernetesProfileValidator asks whether the deployment shape,
the retry owner and the stream obligations agree. A deployment can have a coherent resolver profile
and still have put retries in two places, so merging them would let one answer hide the other.
expectedAddressCount has to come from somewhere. It is a declared number, and a declaration can
be wrong. It is still better than the alternative, which is not comparing anything: a wrong
declaration is a wrong statement somebody wrote down, and a missing one is a question nobody asked.
GrpcChannelProfileValidator takes resolved counts where they are known at startup and skips the
check where they are not, rather than guessing and failing on a name that cannot be resolved yet.
xDS is reachable, and not by this route. It lives in grpc-advanced-resilience behind its
capability flag and its production approval, and GrpcXdsStartupGuard.advertisableAsStableSupport()
returns false so the Stable support statement cannot widen quietly. See ADR-GRPC-ADV-001.