# HTTP Client Platform — Operations Runbook

## Metrics

| Metric | Meaning |
|---|---|
| `http.client.requests` | Physical attempt timer (Spring standard name, kept deliberately) |
| `http.client.logical.calls` | User-visible logical call timer |
| `http.client.attempts` | Attempt counter |
| `http.client.retry.count` | Retries by reason |
| `http.client.retry.exhausted` | Retry budget exhausted |
| `http.client.ambiguous` | Ambiguous outcomes |
| `http.client.timeout` | Timeouts by stage |
| `http.client.request.bytes` | Request wire bytes |
| `http.client.response.bytes` | Response bytes |
| `http.client.active` | In-flight attempts |
| `http.client.pool.connections` | Leased and available connections |
| `http.client.pool.pending` | Pool waiters |
| `http.client.pool.acquire.duration` | Pool wait time |
| `http.client.dns.duration` | DNS time |
| `http.client.connect.duration` | Connect time |
| `http.client.tls.duration` | TLS time |
| `http.client.circuit.state` | Circuit state |
| `http.client.bulkhead.rejected` | Bulkhead rejections |
| `http.client.rate_limit.rejected` | Local rate-limit rejections |
| `http.client.oauth.refresh` | Token refresh outcomes |
| `http.client.ssrf.rejected` | Dynamic target rejections |

`http.client.requests` counts attempts and `http.client.logical.calls` counts user calls. When they
diverge, retries are absorbing failures — which is the first thing to look at during an incident.

## Reading an incident

| Symptom | Likely cause | Where to look |
|---|---|---|
| logical calls fine, attempts spiking | upstream degraded, retries absorbing it | `http.client.retry.count` by reason |
| `http.client.ambiguous` non-zero | non-idempotent writes reaching `SENT_NO_RESPONSE` | reconcile with the upstream; consider an idempotency key |
| pool pending climbing | pool too small or upstream slow | `http.client.pool.acquire.duration`, `pool.connections` |
| circuit open | sustained upstream failure | `http.client.circuit.state`; local rejections do not open it |
| `http.client.ssrf.rejected` non-zero | a caller is submitting internal URLs | Dynamic Target policy and audit trail |

## Actuator

`GET /actuator/httpclients` reports profile name, runtime generation, state, transport, API,
protocols, active leases, pool ceiling, credential type, TLS profile id, redirect flag, retry policy,
and capability warnings. Base URL, credentials, trust store paths, and resolved IPs are deliberately
absent: an actuator endpoint is reachable by more people than a secret store is.

## Rotation

Certificates and secrets rotate by building a new runtime generation and swapping the registry
pointer, never by mutating a live client. A connection pool holds sockets established under the
previous identity, so replacing material without replacing the pool leaves live connections
authenticated by a certificate that is meant to be gone.

```text
build new generation → validate → atomic swap → new calls use it
old generation → DRAINING → in-flight calls finish → no new retries → forced close at the drain deadline
```

## Shutdown

```text
RUNNING → DRAINING
new logical calls refused or routed to the new generation
in-flight attempts complete
new retries refused
shutdown timeout
remaining calls cancelled
pool closed
```

## Retry ownership

Exactly one of the application client, an external SDK, or the service mesh may own retries.
Two owners multiply traffic during an incident. Record the owner per upstream and check it whenever
a mesh retry policy changes.

## Error model

Every outbound failure is one of these stable types. The type is derived from the classified failure
category, not from whatever the engine happened to throw, so it means the same thing on Apache, JDK,
and Reactor Netty. Each carries `HttpFailureMetadata`: client, operation, method, URI **template**,
evidence, replayability, stage, retryability, attempt, elapsed, remaining deadline, status, trace id
— and nothing else.

| Exception | Raised when | Retryable |
|---|---|---|
| `HttpConfigurationException` | profile, operation, or capability configuration is invalid | never |
| `HttpTargetRejectedException` | target URI, host, port, header, or address policy refused the request | never |
| `HttpDnsException` | hostname resolution failed or timed out | yes, inside budget |
| `HttpPoolAcquireTimeoutException` | no connection or stream within the pending-acquire budget | yes, inside budget |
| `HttpConnectException` | socket connect failed | yes, inside budget |
| `HttpProxyException` | proxy connect, CONNECT tunnel, or proxy auth failed | yes, inside budget |
| `HttpTlsException` | TLS handshake failed | only a transient handshake timeout |
| `HttpRequestWriteException` | request headers or body could not be fully written | only when safely idempotent |
| `HttpResponseTimeoutException` | final headers or a body chunk did not arrive in time | only when safely idempotent |
| `HttpResponseTruncatedException` | the response ended before the body was complete | only when safely idempotent and undelivered |
| `HttpRemoteErrorException` | non-success status without a problem document | per the status rules |
| `HttpProblemDetailException` | non-success status with a bounded RFC 9457 document | per the status rules |
| `HttpRedirectRejectedException` | a hop violated hop count, origin, method, or replay policy | never |
| `HttpAuthenticationException` | credential materialization or refresh failed | never |
| `HttpSerializationException` | request encoding or response decoding failed | never |
| `HttpResponseTooLargeException` | wire or decoded bytes exceeded the profile limit | never |
| `HttpDeadlineExceededException` | the effective deadline was reached | never |
| `HttpCircuitOpenException` | the upstream circuit is open | never |
| `HttpBulkheadRejectedException` | no attempt or logical admission permit was available | never |
| `HttpRateLimitRejectedException` | the local attempt rate limit or retry budget rejected the attempt | never |
| `HttpAmbiguousExecutionException` | a non-idempotent request was sent and the outcome is unknown | never — reconcile instead |

## Traces

```text
http.client.operation        logical internal span
└─ http.client.request       attempt 1 CLIENT span
└─ http.client.request       attempt 2 CLIENT span
```

W3C Trace Context is propagated with a Baggage allowlist. Dynamic Targets do not propagate trace
context by default. Retry reason and evidence are recorded as span events; credentials and remote
error bodies are never recorded as attributes.