Files
clean-architecture-backend-…/docs/httpclient/operations.md
T
DongHyeonka 34565130e0 fix!: 성숙도 리뷰의 High·P1 결함 19건을 고친다
문제: 2026-11-01 성숙도 리뷰에서 High 21건·P1 25건이 나왔다. 결과를
모르는 일을 "확실함"으로 기록했다(발송 중 죽은 알림 재발송, 커밋 중
끊긴 쓰기를 재시도 가능으로 응답, 재시도 가능한 거부를 DEAD로). 중복을
DB가 아니라 코드 순서로 막았다(정산 job, 포스터 제목). 짝 프론트와
wire 계약이 달랐고(처리 중 409, Envelope null), JWT audience 검증이
기본으로 꺼져 있었고, 트레이스가 나가지 않았고, CI가 strict lane 14개를
한 번도 돌리지 않았다.

방법: 발견마다 실패하는 테스트를 먼저 쓰고 고쳤다. 주요 선택:
- 알림 attempt는 처음부터 "시작 여부 모름"으로 쓴다(호출 직전 갱신은
  쓰기가 하나 늘어 버림).
- 정산 job claim은 lease·fence를 갱신하는 UPDATE 한 문장(처리 전체를
  트랜잭션으로 감싸는 안은 외부 호출 동안 잠금을 잡아 버림).
- 커밋 단계 연결 끊김은 DB_COMPLETION_UNKNOWN(재시도 불가).
- 처리 중 409는 retryable=true + Retry-After: 2, Envelope null 필드는
  web Jackson mixin으로 생략(shared-contract는 stdlib only).
- 트레이싱은 자체 Tracer를 지우고 Boot 4 tracing·OTLP로.
- strict lane을 단계에 연결하고 verifyStrictLaneStages가 고아를 막는다.
발견별 RED·GREEN·버린 대안은 리뷰 README 13절 표에 있다.

위험·한계: JWT 모드는 APP_SECURITY_JWT_AUDIENCE가 없으면 기동하지
않는다. 짝 프론트는 409 retryable과 Envelope 키 규칙을 맞춰야 한다.
이 커밋 시점에는 MinIO 공식 이미지가 사라져 MinIO 2 lane이 실패한다
(다음 커밋에서 대체 이미지로 해결).

검증: ci 통과. mainCheck·integrationCheck는 MinIO 2 lane만 실패(이미지
pull 거부). -p optional-platforms ci integrationCheck, performanceCheck,
releaseCheck(-PgitRevision) 통과.

Refs: docs/reviews/2026-11-01-maturity-review/README.md 2·13절
BREAKING CHANGE: JWT 모드에서 APP_SECURITY_JWT_AUDIENCE가 필수가 되고 role은
audience client와 realm 것만 읽는다. IDEMPOTENT_IN_FLIGHT가 retryable=true,
Retry-After: 2로 바뀌고 Envelope는 null 필드를 보내지 않는다.
2026-11-03 19:52:06 +09:00

6.9 KiB

HTTP Client Platform — Operations Runbook

Metrics

Metric Meaning
http.client.requests Physical attempt timer (Spring standard name, kept deliberately)
http.client.logical.calls User-visible logical call timer
http.client.attempts Attempt counter
http.client.retry.count Retries by reason
http.client.retry.exhausted Retry budget exhausted
http.client.ambiguous Ambiguous outcomes
http.client.timeout Timeouts by stage
http.client.request.bytes Request wire bytes
http.client.response.bytes Response bytes
http.client.active In-flight attempts
http.client.pool.connections Leased and available connections
http.client.pool.pending Pool waiters
http.client.pool.acquire.duration Pool wait time
http.client.dns.duration DNS time
http.client.connect.duration Connect time
http.client.tls.duration TLS time
http.client.circuit.state Circuit state
http.client.bulkhead.rejected Bulkhead rejections
http.client.rate_limit.rejected Local rate-limit rejections
http.client.oauth.refresh Token refresh outcomes
http.client.ssrf.rejected Dynamic target rejections

http.client.requests counts attempts and http.client.logical.calls counts user calls. When they diverge, retries are absorbing failures — which is the first thing to look at during an incident.

Reading an incident

Symptom Likely cause Where to look
logical calls fine, attempts spiking upstream degraded, retries absorbing it http.client.retry.count by reason
http.client.ambiguous non-zero non-idempotent writes reaching SENT_NO_RESPONSE reconcile with the upstream; consider an idempotency key
pool pending climbing pool too small or upstream slow http.client.pool.acquire.duration, pool.connections
circuit open sustained upstream failure: transport errors, 5xx, 408, 429 http.client.circuit.state; local rejections and other 4xx do not open it
http.client.ssrf.rejected non-zero a caller is submitting internal URLs Dynamic Target policy and audit trail

Actuator

GET /actuator/httpclients reports profile name, runtime generation, state, transport, API, protocols, active leases, pool ceiling, credential type, TLS profile id, redirect flag, retry policy, and capability warnings. Base URL, credentials, trust store paths, and resolved IPs are deliberately absent: an actuator endpoint is reachable by more people than a secret store is.

Rotation

Certificates and secrets rotate by building a new runtime generation and swapping the registry pointer, never by mutating a live client. A connection pool holds sockets established under the previous identity, so replacing material without replacing the pool leaves live connections authenticated by a certificate that is meant to be gone.

build new generation → validate → atomic swap → new calls use it
old generation → DRAINING → in-flight calls finish → no new retries → forced close at the drain deadline

Shutdown

RUNNING → DRAINING
new logical calls refused or routed to the new generation
in-flight attempts complete
new retries refused
shutdown timeout
remaining calls cancelled
pool closed

Retry ownership

Exactly one of the application client, an external SDK, or the service mesh may own retries. Two owners multiply traffic during an incident. Record the owner per upstream and check it whenever a mesh retry policy changes.

Error model

Every outbound failure is one of these stable types. The type is derived from the classified failure category, not from whatever the engine happened to throw, so it means the same thing on Apache and JDK. Each carries HttpFailureMetadata: client, operation, method, URI template, evidence, replayability, stage, retryability, attempt, elapsed, remaining deadline, status, trace id — and nothing else.

Exception Raised when Retryable
HttpConfigurationException profile, operation, or capability configuration is invalid never
HttpTargetRejectedException target URI, host, port, header, or address policy refused the request never
HttpDnsException hostname resolution failed or timed out yes, inside budget
HttpPoolAcquireTimeoutException no connection or stream within the pending-acquire budget yes, inside budget
HttpConnectException socket connect failed yes, inside budget
HttpProxyException proxy connect, CONNECT tunnel, or proxy auth failed yes, inside budget
HttpTlsException TLS handshake failed only a transient handshake timeout
HttpRequestWriteException request headers or body could not be fully written only when safely idempotent
HttpResponseTimeoutException final headers or a body chunk did not arrive in time only when safely idempotent
HttpResponseTruncatedException the response ended before the body was complete only when safely idempotent and undelivered
HttpRemoteErrorException non-success status without a problem document per the status rules
HttpProblemDetailException non-success status with a bounded RFC 9457 document per the status rules
HttpRedirectRejectedException a hop violated hop count, origin, method, or replay policy never
HttpAuthenticationException credential materialization or refresh failed never
HttpSerializationException request encoding or response decoding failed never
HttpResponseTooLargeException wire or decoded bytes exceeded the profile limit never
HttpDeadlineExceededException the effective deadline was reached never
HttpCircuitOpenException the upstream circuit is open never
HttpBulkheadRejectedException no attempt or logical admission permit was available never
HttpRateLimitRejectedException the local attempt rate limiter rejected the attempt never
HttpRetryBudgetExhaustedException the per-upstream retry budget refused a further attempt never
HttpRuntimeDrainingException the profile's runtime was retired (certificate rotation) before a granted retry started never — a new call runs on the replacement
HttpUnknownTransportException the engine failed in a way no category above describes; the metadata evidence, not the type, says whether it was sent never
HttpAmbiguousExecutionException a non-idempotent request was sent and the outcome is unknown never — reconcile instead

Traces

http.client.operation        logical internal span
└─ http.client.request       attempt 1 CLIENT span
└─ http.client.request       attempt 2 CLIENT span

W3C Trace Context is propagated with a Baggage allowlist. Dynamic Targets do not propagate trace context by default. Retry reason and evidence are recorded as span events; credentials and remote error bodies are never recorded as attributes.