chore: initialize from backend template 0a6dd0e

2026-08-13 20:31:02 +09:00
commit e64e701fe5
3223 changed files with 388401 additions and 0 deletions
@@ -0,0 +1,121 @@
+# HTTP Client Platform — Operations Runbook
+
+## Metrics
+
+| Metric | Meaning |
+|---|---|
+| `http.client.requests` | Physical attempt timer (Spring standard name, kept deliberately) |
+| `http.client.logical.calls` | User-visible logical call timer |
+| `http.client.attempts` | Attempt counter |
+| `http.client.retry.count` | Retries by reason |
+| `http.client.retry.exhausted` | Retry budget exhausted |
+| `http.client.ambiguous` | Ambiguous outcomes |
+| `http.client.timeout` | Timeouts by stage |
+| `http.client.request.bytes` | Request wire bytes |
+| `http.client.response.bytes` | Response bytes |
+| `http.client.active` | In-flight attempts |
+| `http.client.pool.connections` | Leased and available connections |
+| `http.client.pool.pending` | Pool waiters |
+| `http.client.pool.acquire.duration` | Pool wait time |
+| `http.client.dns.duration` | DNS time |
+| `http.client.connect.duration` | Connect time |
+| `http.client.tls.duration` | TLS time |
+| `http.client.circuit.state` | Circuit state |
+| `http.client.bulkhead.rejected` | Bulkhead rejections |
+| `http.client.rate_limit.rejected` | Local rate-limit rejections |
+| `http.client.oauth.refresh` | Token refresh outcomes |
+| `http.client.ssrf.rejected` | Dynamic target rejections |
+
+`http.client.requests` counts attempts and `http.client.logical.calls` counts user calls. When they
+diverge, retries are absorbing failures — which is the first thing to look at during an incident.
+
+## Reading an incident
+
+| Symptom | Likely cause | Where to look |
+|---|---|---|
+| logical calls fine, attempts spiking | upstream degraded, retries absorbing it | `http.client.retry.count` by reason |
+| `http.client.ambiguous` non-zero | non-idempotent writes reaching `SENT_NO_RESPONSE` | reconcile with the upstream; consider an idempotency key |
+| pool pending climbing | pool too small or upstream slow | `http.client.pool.acquire.duration`, `pool.connections` |
+| circuit open | sustained upstream failure | `http.client.circuit.state`; local rejections do not open it |
+| `http.client.ssrf.rejected` non-zero | a caller is submitting internal URLs | Dynamic Target policy and audit trail |
+
+## Actuator
+
+`GET /actuator/httpclients` reports profile name, runtime generation, state, transport, API,
+protocols, active leases, pool ceiling, credential type, TLS profile id, redirect flag, retry policy,
+and capability warnings. Base URL, credentials, trust store paths, and resolved IPs are deliberately
+absent: an actuator endpoint is reachable by more people than a secret store is.
+
+## Rotation
+
+Certificates and secrets rotate by building a new runtime generation and swapping the registry
+pointer, never by mutating a live client. A connection pool holds sockets established under the
+previous identity, so replacing material without replacing the pool leaves live connections
+authenticated by a certificate that is meant to be gone.
+
+```text
+build new generation → validate → atomic swap → new calls use it
+old generation → DRAINING → in-flight calls finish → no new retries → forced close at the drain deadline
+```
+
+## Shutdown
+
+```text
+RUNNING → DRAINING
+new logical calls refused or routed to the new generation
+in-flight attempts complete
+new retries refused
+shutdown timeout
+remaining calls cancelled
+pool closed
+```
+
+## Retry ownership
+
+Exactly one of the application client, an external SDK, or the service mesh may own retries.
+Two owners multiply traffic during an incident. Record the owner per upstream and check it whenever
+a mesh retry policy changes.
+
+## Error model
+
+Every outbound failure is one of these stable types. The type is derived from the classified failure
+category, not from whatever the engine happened to throw, so it means the same thing on Apache, JDK,
+and Reactor Netty. Each carries `HttpFailureMetadata`: client, operation, method, URI **template**,
+evidence, replayability, stage, retryability, attempt, elapsed, remaining deadline, status, trace id
+— and nothing else.
+
+| Exception | Raised when | Retryable |
+|---|---|---|
+| `HttpConfigurationException` | profile, operation, or capability configuration is invalid | never |
+| `HttpTargetRejectedException` | target URI, host, port, header, or address policy refused the request | never |
+| `HttpDnsException` | hostname resolution failed or timed out | yes, inside budget |
+| `HttpPoolAcquireTimeoutException` | no connection or stream within the pending-acquire budget | yes, inside budget |
+| `HttpConnectException` | socket connect failed | yes, inside budget |
+| `HttpProxyException` | proxy connect, CONNECT tunnel, or proxy auth failed | yes, inside budget |
+| `HttpTlsException` | TLS handshake failed | only a transient handshake timeout |
+| `HttpRequestWriteException` | request headers or body could not be fully written | only when safely idempotent |
+| `HttpResponseTimeoutException` | final headers or a body chunk did not arrive in time | only when safely idempotent |
+| `HttpResponseTruncatedException` | the response ended before the body was complete | only when safely idempotent and undelivered |
+| `HttpRemoteErrorException` | non-success status without a problem document | per the status rules |
+| `HttpProblemDetailException` | non-success status with a bounded RFC 9457 document | per the status rules |
+| `HttpRedirectRejectedException` | a hop violated hop count, origin, method, or replay policy | never |
+| `HttpAuthenticationException` | credential materialization or refresh failed | never |
+| `HttpSerializationException` | request encoding or response decoding failed | never |
+| `HttpResponseTooLargeException` | wire or decoded bytes exceeded the profile limit | never |
+| `HttpDeadlineExceededException` | the effective deadline was reached | never |
+| `HttpCircuitOpenException` | the upstream circuit is open | never |
+| `HttpBulkheadRejectedException` | no attempt or logical admission permit was available | never |
+| `HttpRateLimitRejectedException` | the local attempt rate limit or retry budget rejected the attempt | never |
+| `HttpAmbiguousExecutionException` | a non-idempotent request was sent and the outcome is unknown | never — reconcile instead |
+
+## Traces
+
+```text
+http.client.operation        logical internal span
+└─ http.client.request       attempt 1 CLIENT span
+└─ http.client.request       attempt 2 CLIENT span
+```
+
+W3C Trace Context is propagated with a Baggage allowlist. Dynamic Targets do not propagate trace
+context by default. Retry reason and evidence are recorded as span events; credentials and remote
+error bodies are never recorded as attributes.