53 lines
2.9 KiB
Markdown
53 lines
2.9 KiB
Markdown
# ADR-GRPC-004: One retry owner, and keyed mutations need a durable ledger
|
|
|
|
- Status: accepted
|
|
- Date: 2026-08-30
|
|
- Scope: `:grpc:grpc-policy`, `:grpc:grpc-operation-ledger-jpa`, `:grpc:grpc-core-api`
|
|
|
|
## Context
|
|
|
|
Three layers can retry a gRPC call: the application, the channel's service config, and a service
|
|
mesh. Their effects multiply. Three attempts at each layer is twenty-seven requests for one call, and
|
|
the load arrives exactly when the dependency is already failing.
|
|
|
|
Separately, a mutation that is safe to repeat needs somewhere to record that it ran. Without one, a
|
|
retry after a lost response either duplicates the effect or drops it, and nothing distinguishes the
|
|
two afterwards.
|
|
|
|
## Decision
|
|
|
|
**Exactly one retry owner per channel.** `GrpcRetryOwner` has four values including `NONE`, which is a
|
|
decision rather than an omission. `GrpcServiceConfigPolicy` refuses an in-process retry entry when the
|
|
owner is the mesh or nobody, and `GrpcRetryOwnershipValidator` compares the service config's method
|
|
names against the policy catalog — a renamed method leaves its retry entry matching nothing, silently,
|
|
and the method then runs with channel defaults.
|
|
|
|
**Retry eligibility reads the method, the evidence and the status together.**
|
|
`GrpcRetryEligibility` refuses a non-idempotent method outright, refuses any call whose stream
|
|
delivered a prefix, and turns a `DEADLINE_EXCEEDED` or post-send `UNAVAILABLE` mutation into
|
|
"resolve the completion first" rather than a retry.
|
|
|
|
**A keyed mutation is retryable only with both a caller key and a durable ledger.**
|
|
`GrpcOperationLedger` is a port in `grpc-core-api`, so the policy layer can require durable
|
|
idempotency without depending on a database. Its `claim` contract is a single atomic insert-or-read
|
|
against a unique constraint: `JpaGrpcOperationLedger` inserts first and reads on constraint violation,
|
|
because a read-then-insert implementation has a window exactly as wide as the race it closes and
|
|
passes every test that does not run two attempts concurrently.
|
|
|
|
The identity is caller fingerprint plus full method plus hashed key. All three are load-bearing:
|
|
without the caller, one tenant's key suppresses another's write; without the method, a key reused
|
|
across operations makes the second a replay of the first.
|
|
|
|
## Consequences
|
|
|
|
**A budget bounds retries as a fraction of traffic.** `GrpcRetryBudget` degrades to roughly no
|
|
retries when everything is failing, which is the behaviour that lets a dependency recover.
|
|
|
|
**The ledger and the mutation should commit together.** `JpaGrpcOperationLedger` carries no
|
|
transaction annotations, deliberately: a `REQUIRES_NEW` would put the claim in its own transaction and
|
|
reintroduce the window where the write is durable and the claim is not.
|
|
|
|
**A key reused for a different request is a caller error, not a duplicate.** The stored request
|
|
fingerprint turns that into `FAILED_PRECONDITION` rather than silently returning the first request's
|
|
answer.
|