13개 작업 묶음(P1–P13)의 W2 커밋을 합치고 전체 게이트로 검증했다. - 묶음별 정합성 수정(notification dispatch·recovery, object storage staged publication, fileserver 복구·정리, gRPC 플랫폼, Redis 분류·lease TTL, JPA outbox claim 순서·durable operation HTTP, web 요청 id·category 필수, graphql 에러 mapper, Mongo profile 바인딩, build 소스 보존 게이트·compose smoke info 확인) - 통합 때 드러난 3건 수정: outbox claim 테스트 fixture의 가짜 envelope hash, sample scan이 새 operation controller를 등록하던 문제, SpotBugs aux classpath 파일이 truncate되지 않아 로컬에서 ".jar.jar"로 실패하던 문제 - 현황표: 묶음 결과 반영, 소유자 결정 21–27, 후속 발견 결함 47행, 요약 갱신 - docs/capability-status.md 「제거된 코드」에 85행 추가 - 묶음 결과 파일·에이전트 지시·스크립트를 docs로 옮기고 desktop 재개 프롬프트 추가 검증: check, ci verifyDependencyLocks, mainCheck, architecture/public path/env keys/config contract, bootstrap·sample integration/system, build-logic·build-tools test, optional-platforms ci, qualificationBootJar 통과. integrationCheck는 이번 작업 전에도 실패하던 lane(Redis topology 모드 인자, httpclient failure injection, AWS 보호 lane)만 실패. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
6.8 KiB
HTTP Client Platform — Operations Runbook
Metrics
| Metric | Meaning |
|---|---|
http.client.requests |
Physical attempt timer (Spring standard name, kept deliberately) |
http.client.logical.calls |
User-visible logical call timer |
http.client.attempts |
Attempt counter |
http.client.retry.count |
Retries by reason |
http.client.retry.exhausted |
Retry budget exhausted |
http.client.ambiguous |
Ambiguous outcomes |
http.client.timeout |
Timeouts by stage |
http.client.request.bytes |
Request wire bytes |
http.client.response.bytes |
Response bytes |
http.client.active |
In-flight attempts |
http.client.pool.connections |
Leased and available connections |
http.client.pool.pending |
Pool waiters |
http.client.pool.acquire.duration |
Pool wait time |
http.client.dns.duration |
DNS time |
http.client.connect.duration |
Connect time |
http.client.tls.duration |
TLS time |
http.client.circuit.state |
Circuit state |
http.client.bulkhead.rejected |
Bulkhead rejections |
http.client.rate_limit.rejected |
Local rate-limit rejections |
http.client.oauth.refresh |
Token refresh outcomes |
http.client.ssrf.rejected |
Dynamic target rejections |
http.client.requests counts attempts and http.client.logical.calls counts user calls. When they
diverge, retries are absorbing failures — which is the first thing to look at during an incident.
Reading an incident
| Symptom | Likely cause | Where to look |
|---|---|---|
| logical calls fine, attempts spiking | upstream degraded, retries absorbing it | http.client.retry.count by reason |
http.client.ambiguous non-zero |
non-idempotent writes reaching SENT_NO_RESPONSE |
reconcile with the upstream; consider an idempotency key |
| pool pending climbing | pool too small or upstream slow | http.client.pool.acquire.duration, pool.connections |
| circuit open | sustained upstream failure | http.client.circuit.state; local rejections do not open it |
http.client.ssrf.rejected non-zero |
a caller is submitting internal URLs | Dynamic Target policy and audit trail |
Actuator
GET /actuator/httpclients reports profile name, runtime generation, state, transport, API,
protocols, active leases, pool ceiling, credential type, TLS profile id, redirect flag, retry policy,
and capability warnings. Base URL, credentials, trust store paths, and resolved IPs are deliberately
absent: an actuator endpoint is reachable by more people than a secret store is.
Rotation
Certificates and secrets rotate by building a new runtime generation and swapping the registry pointer, never by mutating a live client. A connection pool holds sockets established under the previous identity, so replacing material without replacing the pool leaves live connections authenticated by a certificate that is meant to be gone.
build new generation → validate → atomic swap → new calls use it
old generation → DRAINING → in-flight calls finish → no new retries → forced close at the drain deadline
Shutdown
RUNNING → DRAINING
new logical calls refused or routed to the new generation
in-flight attempts complete
new retries refused
shutdown timeout
remaining calls cancelled
pool closed
Retry ownership
Exactly one of the application client, an external SDK, or the service mesh may own retries. Two owners multiply traffic during an incident. Record the owner per upstream and check it whenever a mesh retry policy changes.
Error model
Every outbound failure is one of these stable types. The type is derived from the classified failure
category, not from whatever the engine happened to throw, so it means the same thing on Apache and
JDK. Each carries HttpFailureMetadata: client, operation, method, URI template,
evidence, replayability, stage, retryability, attempt, elapsed, remaining deadline, status, trace id
— and nothing else.
| Exception | Raised when | Retryable |
|---|---|---|
HttpConfigurationException |
profile, operation, or capability configuration is invalid | never |
HttpTargetRejectedException |
target URI, host, port, header, or address policy refused the request | never |
HttpDnsException |
hostname resolution failed or timed out | yes, inside budget |
HttpPoolAcquireTimeoutException |
no connection or stream within the pending-acquire budget | yes, inside budget |
HttpConnectException |
socket connect failed | yes, inside budget |
HttpProxyException |
proxy connect, CONNECT tunnel, or proxy auth failed | yes, inside budget |
HttpTlsException |
TLS handshake failed | only a transient handshake timeout |
HttpRequestWriteException |
request headers or body could not be fully written | only when safely idempotent |
HttpResponseTimeoutException |
final headers or a body chunk did not arrive in time | only when safely idempotent |
HttpResponseTruncatedException |
the response ended before the body was complete | only when safely idempotent and undelivered |
HttpRemoteErrorException |
non-success status without a problem document | per the status rules |
HttpProblemDetailException |
non-success status with a bounded RFC 9457 document | per the status rules |
HttpRedirectRejectedException |
a hop violated hop count, origin, method, or replay policy | never |
HttpAuthenticationException |
credential materialization or refresh failed | never |
HttpSerializationException |
request encoding or response decoding failed | never |
HttpResponseTooLargeException |
wire or decoded bytes exceeded the profile limit | never |
HttpDeadlineExceededException |
the effective deadline was reached | never |
HttpCircuitOpenException |
the upstream circuit is open | never |
HttpBulkheadRejectedException |
no attempt or logical admission permit was available | never |
HttpRateLimitRejectedException |
the local attempt rate limiter rejected the attempt | never |
HttpRetryBudgetExhaustedException |
the per-upstream retry budget refused a further attempt | never |
HttpRuntimeDrainingException |
the profile's runtime was retired (certificate rotation) before a granted retry started | never — a new call runs on the replacement |
HttpUnknownTransportException |
the engine failed in a way no category above describes; the metadata evidence, not the type, says whether it was sent | never |
HttpAmbiguousExecutionException |
a non-idempotent request was sent and the outcome is unknown | never — reconcile instead |
Traces
http.client.operation logical internal span
└─ http.client.request attempt 1 CLIENT span
└─ http.client.request attempt 2 CLIENT span
W3C Trace Context is propagated with a Baggage allowlist. Dynamic Targets do not propagate trace context by default. Retry reason and evidence are recorded as span events; credentials and remote error bodies are never recorded as attributes.