DongHyeonkaandClaude Opus 5
5f10b791d3
Snapshot of the in-flight state that already existed, identically, in both this worktree and the main checkout before this session began: the initial HTTP Client platform implementation (previously untracked), the redis-lab removal, and the JPA / object-storage / notification integration work. Kept separate from this session's HTTP Client review response, which lands in the following commit, so the two bodies of work stay reviewable apart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
122 lines
6.4 KiB
Markdown
122 lines
6.4 KiB
Markdown
# HTTP Client Platform — Operations Runbook
|
|
|
|
## Metrics
|
|
|
|
| Metric | Meaning |
|
|
|---|---|
|
|
| `http.client.requests` | Physical attempt timer (Spring standard name, kept deliberately) |
|
|
| `http.client.logical.calls` | User-visible logical call timer |
|
|
| `http.client.attempts` | Attempt counter |
|
|
| `http.client.retry.count` | Retries by reason |
|
|
| `http.client.retry.exhausted` | Retry budget exhausted |
|
|
| `http.client.ambiguous` | Ambiguous outcomes |
|
|
| `http.client.timeout` | Timeouts by stage |
|
|
| `http.client.request.bytes` | Request wire bytes |
|
|
| `http.client.response.bytes` | Response bytes |
|
|
| `http.client.active` | In-flight attempts |
|
|
| `http.client.pool.connections` | Leased and available connections |
|
|
| `http.client.pool.pending` | Pool waiters |
|
|
| `http.client.pool.acquire.duration` | Pool wait time |
|
|
| `http.client.dns.duration` | DNS time |
|
|
| `http.client.connect.duration` | Connect time |
|
|
| `http.client.tls.duration` | TLS time |
|
|
| `http.client.circuit.state` | Circuit state |
|
|
| `http.client.bulkhead.rejected` | Bulkhead rejections |
|
|
| `http.client.rate_limit.rejected` | Local rate-limit rejections |
|
|
| `http.client.oauth.refresh` | Token refresh outcomes |
|
|
| `http.client.ssrf.rejected` | Dynamic target rejections |
|
|
|
|
`http.client.requests` counts attempts and `http.client.logical.calls` counts user calls. When they
|
|
diverge, retries are absorbing failures — which is the first thing to look at during an incident.
|
|
|
|
## Reading an incident
|
|
|
|
| Symptom | Likely cause | Where to look |
|
|
|---|---|---|
|
|
| logical calls fine, attempts spiking | upstream degraded, retries absorbing it | `http.client.retry.count` by reason |
|
|
| `http.client.ambiguous` non-zero | non-idempotent writes reaching `SENT_NO_RESPONSE` | reconcile with the upstream; consider an idempotency key |
|
|
| pool pending climbing | pool too small or upstream slow | `http.client.pool.acquire.duration`, `pool.connections` |
|
|
| circuit open | sustained upstream failure | `http.client.circuit.state`; local rejections do not open it |
|
|
| `http.client.ssrf.rejected` non-zero | a caller is submitting internal URLs | Dynamic Target policy and audit trail |
|
|
|
|
## Actuator
|
|
|
|
`GET /actuator/httpclients` reports profile name, runtime generation, state, transport, API,
|
|
protocols, active leases, pool ceiling, credential type, TLS profile id, redirect flag, retry policy,
|
|
and capability warnings. Base URL, credentials, trust store paths, and resolved IPs are deliberately
|
|
absent: an actuator endpoint is reachable by more people than a secret store is.
|
|
|
|
## Rotation
|
|
|
|
Certificates and secrets rotate by building a new runtime generation and swapping the registry
|
|
pointer, never by mutating a live client. A connection pool holds sockets established under the
|
|
previous identity, so replacing material without replacing the pool leaves live connections
|
|
authenticated by a certificate that is meant to be gone.
|
|
|
|
```text
|
|
build new generation → validate → atomic swap → new calls use it
|
|
old generation → DRAINING → in-flight calls finish → no new retries → forced close at the drain deadline
|
|
```
|
|
|
|
## Shutdown
|
|
|
|
```text
|
|
RUNNING → DRAINING
|
|
new logical calls refused or routed to the new generation
|
|
in-flight attempts complete
|
|
new retries refused
|
|
shutdown timeout
|
|
remaining calls cancelled
|
|
pool closed
|
|
```
|
|
|
|
## Retry ownership
|
|
|
|
Exactly one of the application client, an external SDK, or the service mesh may own retries.
|
|
Two owners multiply traffic during an incident. Record the owner per upstream and check it whenever
|
|
a mesh retry policy changes.
|
|
|
|
## Error model
|
|
|
|
Every outbound failure is one of these stable types. The type is derived from the classified failure
|
|
category, not from whatever the engine happened to throw, so it means the same thing on Apache, JDK,
|
|
and Reactor Netty. Each carries `HttpFailureMetadata`: client, operation, method, URI **template**,
|
|
evidence, replayability, stage, retryability, attempt, elapsed, remaining deadline, status, trace id
|
|
— and nothing else.
|
|
|
|
| Exception | Raised when | Retryable |
|
|
|---|---|---|
|
|
| `HttpConfigurationException` | profile, operation, or capability configuration is invalid | never |
|
|
| `HttpTargetRejectedException` | target URI, host, port, header, or address policy refused the request | never |
|
|
| `HttpDnsException` | hostname resolution failed or timed out | yes, inside budget |
|
|
| `HttpPoolAcquireTimeoutException` | no connection or stream within the pending-acquire budget | yes, inside budget |
|
|
| `HttpConnectException` | socket connect failed | yes, inside budget |
|
|
| `HttpProxyException` | proxy connect, CONNECT tunnel, or proxy auth failed | yes, inside budget |
|
|
| `HttpTlsException` | TLS handshake failed | only a transient handshake timeout |
|
|
| `HttpRequestWriteException` | request headers or body could not be fully written | only when safely idempotent |
|
|
| `HttpResponseTimeoutException` | final headers or a body chunk did not arrive in time | only when safely idempotent |
|
|
| `HttpResponseTruncatedException` | the response ended before the body was complete | only when safely idempotent and undelivered |
|
|
| `HttpRemoteErrorException` | non-success status without a problem document | per the status rules |
|
|
| `HttpProblemDetailException` | non-success status with a bounded RFC 9457 document | per the status rules |
|
|
| `HttpRedirectRejectedException` | a hop violated hop count, origin, method, or replay policy | never |
|
|
| `HttpAuthenticationException` | credential materialization or refresh failed | never |
|
|
| `HttpSerializationException` | request encoding or response decoding failed | never |
|
|
| `HttpResponseTooLargeException` | wire or decoded bytes exceeded the profile limit | never |
|
|
| `HttpDeadlineExceededException` | the effective deadline was reached | never |
|
|
| `HttpCircuitOpenException` | the upstream circuit is open | never |
|
|
| `HttpBulkheadRejectedException` | no attempt or logical admission permit was available | never |
|
|
| `HttpRateLimitRejectedException` | the local attempt rate limit or retry budget rejected the attempt | never |
|
|
| `HttpAmbiguousExecutionException` | a non-idempotent request was sent and the outcome is unknown | never — reconcile instead |
|
|
|
|
## Traces
|
|
|
|
```text
|
|
http.client.operation logical internal span
|
|
└─ http.client.request attempt 1 CLIENT span
|
|
└─ http.client.request attempt 2 CLIENT span
|
|
```
|
|
|
|
W3C Trace Context is propagated with a Baggage allowlist. Dynamic Targets do not propagate trace
|
|
context by default. Retry reason and evidence are recorded as span events; credentials and remote
|
|
error bodies are never recorded as attributes.
|