69 lines
4.1 KiB
Markdown
69 lines
4.1 KiB
Markdown
# ADR-GRPC-006: Stable discovery is DNS and static, and a Kubernetes profile names who balances
|
|
|
|
- Status: accepted
|
|
- Date: 2026-08-31
|
|
- Scope: `:grpc:grpc-discovery`, `:grpc:grpc-client`
|
|
|
|
## Context
|
|
|
|
A gRPC channel's discovery configuration has a failure mode with no runtime symptom: it works, and
|
|
it does not do what the dashboard says it does.
|
|
|
|
The specific case is `round_robin` over a Kubernetes Service ClusterIP. The Service is one virtual
|
|
address, so the resolver returns one endpoint and the client-side balancer has nothing to rotate
|
|
across; kube-proxy picks a pod at connect time, and an HTTP/2 connection is long-lived, so every
|
|
request from that client goes to the same pod for the life of the connection. Nothing fails. The
|
|
configuration says `round_robin`, the metrics show requests spread across clients rather than pods,
|
|
and the conclusion "we have client-side load balancing" is wrong in a way nobody is prompted to
|
|
check.
|
|
|
|
The mirror-image mistake is `pick_first` over a headless record, which pins a client to one pod out
|
|
of many.
|
|
|
|
Separately, a service mesh changes who owns retries, and a deployment that adds mesh routing without
|
|
removing its own retry policy has two retriers whose effects multiply.
|
|
|
|
## Decision
|
|
|
|
**Stable resolvers are Static, DNS and Unix domain socket; Stable load balancing is `pick_first` and
|
|
`round_robin`.** `GrpcDiscoveryPolicyValidator.requireStableScheme` refuses `xds`, `consul`, `etcd`
|
|
and `eureka` by name, with a message saying they are Advanced capabilities with their own control
|
|
plane and promotion gate rather than unknown schemes.
|
|
|
|
**The pairing is checked against the resolved address count, not against intent.**
|
|
`GrpcResolverProfile` carries `expectedAddressCount`, and `GrpcStableLoadBalancer.effective` answers
|
|
whether the policy distributes anything over that many endpoints. A `round_robin` profile over one
|
|
address is a reported violation whose message says it describes spreading that is not happening.
|
|
|
|
**A Kubernetes deployment names its routing mode**, and the mode implies both the balancer and the
|
|
retry owner. `GrpcKubernetesRoutingMode` has three values — `K8S_VIP`, `K8S_HEADLESS`, `MESH` — and
|
|
`GrpcKubernetesProfile` refuses a mesh profile whose retry owner retries in-process.
|
|
|
|
**A profile that carries long-lived streams must state a reconnect budget and a readiness drain
|
|
grace.** A stream pins a client to one pod for its whole life, so every rollout, eviction and
|
|
scale-down ends it. `GrpcKubernetesProfileValidator` additionally reports a VIP profile carrying
|
|
long streams, and a drain grace shorter than the reconnect budget — the second means the pod stops
|
|
serving before its clients have finished reconnecting elsewhere.
|
|
|
|
**A DNS profile must refresh.** `GrpcResolverProfile` refuses a zero refresh interval on DNS,
|
|
because a channel that resolved once at startup keeps sending to addresses that stopped existing an
|
|
hour ago, and the resulting `UNAVAILABLE` looks like an unhealthy deployment long after the rollout
|
|
finished.
|
|
|
|
## Consequences
|
|
|
|
**Two validators, not one.** `GrpcDiscoveryPolicyValidator` asks whether a balancer does anything
|
|
over the addresses it will see; `GrpcKubernetesProfileValidator` asks whether the deployment shape,
|
|
the retry owner and the stream obligations agree. A deployment can have a coherent resolver profile
|
|
and still have put retries in two places, so merging them would let one answer hide the other.
|
|
|
|
**`expectedAddressCount` has to come from somewhere.** It is a declared number, and a declaration can
|
|
be wrong. It is still better than the alternative, which is not comparing anything: a wrong
|
|
declaration is a wrong statement somebody wrote down, and a missing one is a question nobody asked.
|
|
`GrpcChannelProfileValidator` takes resolved counts where they are known at startup and skips the
|
|
check where they are not, rather than guessing and failing on a name that cannot be resolved yet.
|
|
|
|
**xDS is reachable, and not by this route.** It lives in `grpc-advanced-resilience` behind its
|
|
capability flag and its production approval, and `GrpcXdsStartupGuard.advertisableAsStableSupport()`
|
|
returns false so the Stable support statement cannot widen quietly. See ADR-GRPC-ADV-001.
|