tech-log-backend/docs/httpclient/operations.md

HTTP Client Platform — Operations Runbook

Metrics

Metric	Meaning
http.client.requests	Physical attempt timer (Spring standard name, kept deliberately)
http.client.logical.calls	User-visible logical call timer
http.client.attempts	Attempt counter
http.client.retry.count	Retries by reason
http.client.retry.exhausted	Retry budget exhausted
http.client.ambiguous	Ambiguous outcomes
http.client.timeout	Timeouts by stage
http.client.request.bytes	Request wire bytes
http.client.response.bytes	Response bytes
http.client.active	In-flight attempts
http.client.pool.connections	Leased and available connections
http.client.pool.pending	Pool waiters
http.client.pool.acquire.duration	Pool wait time
http.client.dns.duration	DNS time
http.client.connect.duration	Connect time
http.client.tls.duration	TLS time
http.client.circuit.state	Circuit state
http.client.bulkhead.rejected	Bulkhead rejections
http.client.rate_limit.rejected	Local rate-limit rejections
http.client.oauth.refresh	Token refresh outcomes
http.client.ssrf.rejected	Dynamic target rejections

http.client.requests counts attempts and http.client.logical.calls counts user calls. When they diverge, retries are absorbing failures — which is the first thing to look at during an incident.

Reading an incident

Symptom	Likely cause	Where to look
logical calls fine, attempts spiking	upstream degraded, retries absorbing it	http.client.retry.count by reason
http.client.ambiguous non-zero	non-idempotent writes reaching SENT_NO_RESPONSE	reconcile with the upstream; consider an idempotency key
pool pending climbing	pool too small or upstream slow	http.client.pool.acquire.duration, pool.connections
circuit open	sustained upstream failure	http.client.circuit.state; local rejections do not open it
http.client.ssrf.rejected non-zero	a caller is submitting internal URLs	Dynamic Target policy and audit trail

Actuator

GET /actuator/httpclients reports profile name, runtime generation, state, transport, API, protocols, active leases, pool ceiling, credential type, TLS profile id, redirect flag, retry policy, and capability warnings. Base URL, credentials, trust store paths, and resolved IPs are deliberately absent: an actuator endpoint is reachable by more people than a secret store is.

Rotation

Certificates and secrets rotate by building a new runtime generation and swapping the registry pointer, never by mutating a live client. A connection pool holds sockets established under the previous identity, so replacing material without replacing the pool leaves live connections authenticated by a certificate that is meant to be gone.

build new generation → validate → atomic swap → new calls use it
old generation → DRAINING → in-flight calls finish → no new retries → forced close at the drain deadline

Shutdown

RUNNING → DRAINING
new logical calls refused or routed to the new generation
in-flight attempts complete
new retries refused
shutdown timeout
remaining calls cancelled
pool closed

Retry ownership

Exactly one of the application client, an external SDK, or the service mesh may own retries. Two owners multiply traffic during an incident. Record the owner per upstream and check it whenever a mesh retry policy changes.

Error model

Every outbound failure is one of these stable types. The type is derived from the classified failure category, not from whatever the engine happened to throw, so it means the same thing on Apache, JDK, and Reactor Netty. Each carries HttpFailureMetadata: client, operation, method, URI template, evidence, replayability, stage, retryability, attempt, elapsed, remaining deadline, status, trace id — and nothing else.

Exception	Raised when	Retryable
HttpConfigurationException	profile, operation, or capability configuration is invalid	never
HttpTargetRejectedException	target URI, host, port, header, or address policy refused the request	never
HttpDnsException	hostname resolution failed or timed out	yes, inside budget
HttpPoolAcquireTimeoutException	no connection or stream within the pending-acquire budget	yes, inside budget
HttpConnectException	socket connect failed	yes, inside budget
HttpProxyException	proxy connect, CONNECT tunnel, or proxy auth failed	yes, inside budget
HttpTlsException	TLS handshake failed	only a transient handshake timeout
HttpRequestWriteException	request headers or body could not be fully written	only when safely idempotent
HttpResponseTimeoutException	final headers or a body chunk did not arrive in time	only when safely idempotent
HttpResponseTruncatedException	the response ended before the body was complete	only when safely idempotent and undelivered
HttpRemoteErrorException	non-success status without a problem document	per the status rules
HttpProblemDetailException	non-success status with a bounded RFC 9457 document	per the status rules
HttpRedirectRejectedException	a hop violated hop count, origin, method, or replay policy	never
HttpAuthenticationException	credential materialization or refresh failed	never
HttpSerializationException	request encoding or response decoding failed	never
HttpResponseTooLargeException	wire or decoded bytes exceeded the profile limit	never
HttpDeadlineExceededException	the effective deadline was reached	never
HttpCircuitOpenException	the upstream circuit is open	never
HttpBulkheadRejectedException	no attempt or logical admission permit was available	never
HttpRateLimitRejectedException	the local attempt rate limit or retry budget rejected the attempt	never
HttpAmbiguousExecutionException	a non-idempotent request was sent and the outcome is unknown	never — reconcile instead

Traces

http.client.operation        logical internal span
└─ http.client.request       attempt 1 CLIENT span
└─ http.client.request       attempt 2 CLIENT span

W3C Trace Context is propagated with a Baggage allowlist. Dynamic Targets do not propagate trace context by default. Retry reason and evidence are recorded as span events; credentials and remote error bodies are never recorded as attributes.