clean-architecture-backend-template/docs/adr/ADR-GRPC-004-retry-ownership-and-durable-idempotency.md

ADR-GRPC-004: One retry owner, and keyed mutations need a durable ledger

• Status: accepted
• Date: 2026-08-30
• Scope: :grpc:grpc-policy, :grpc:grpc-operation-ledger-jpa, :grpc:grpc-core-api

Context

Three layers can retry a gRPC call: the application, the channel's service config, and a service mesh. Their effects multiply. Three attempts at each layer is twenty-seven requests for one call, and the load arrives exactly when the dependency is already failing.

Separately, a mutation that is safe to repeat needs somewhere to record that it ran. Without one, a retry after a lost response either duplicates the effect or drops it, and nothing distinguishes the two afterwards.

Decision

Exactly one retry owner per channel. GrpcRetryOwner has four values including NONE, which is a decision rather than an omission. GrpcServiceConfigPolicy refuses an in-process retry entry when the owner is the mesh or nobody, and GrpcRetryOwnershipValidator compares the service config's method names against the policy catalog — a renamed method leaves its retry entry matching nothing, silently, and the method then runs with channel defaults.

Retry eligibility reads the method, the evidence and the status together. GrpcRetryEligibility refuses a non-idempotent method outright, refuses any call whose stream delivered a prefix, and turns a DEADLINE_EXCEEDED or post-send UNAVAILABLE mutation into "resolve the completion first" rather than a retry.

A keyed mutation is retryable only with both a caller key and a durable ledger. GrpcOperationLedger is a port in grpc-core-api, so the policy layer can require durable idempotency without depending on a database. Its claim contract is a single atomic insert-or-read against a unique constraint: JpaGrpcOperationLedger inserts first and reads on constraint violation, because a read-then-insert implementation has a window exactly as wide as the race it closes and passes every test that does not run two attempts concurrently.

The identity is caller fingerprint plus full method plus hashed key. All three are load-bearing: without the caller, one tenant's key suppresses another's write; without the method, a key reused across operations makes the second a replay of the first.

Consequences

A budget bounds retries as a fraction of traffic. GrpcRetryBudget degrades to roughly no retries when everything is failing, which is the behaviour that lets a dependency recover.

The ledger and the mutation should commit together. JpaGrpcOperationLedger carries no transaction annotations, deliberately: a REQUIRES_NEW would put the claim in its own transaction and reintroduce the window where the write is durable and the claim is not.

A key reused for a different request is a caller error, not a duplicate. The stored request fingerprint turns that into FAILED_PRECONDITION rather than silently returning the first request's answer.