Files
clean-architecture-backend-…/docs/httpclient/operations.md
T
DongHyeonka 1885bf8da7 fix!: 2차 재리뷰가 찾은 결함 29건을 고친다
문제: 재리뷰 수정 커밋(bbe944bd)만 다시 본 2차 재리뷰가 Important 7건,
Minor 22건을 찾았다. 결과 불명 503 뒤 같은 키를 다시 보내면 Redis
store가 lease 지난 실행을 넘겨받아 작업을 다시 돌렸다. replay TTL이
lease보다 짧으면 lease 도중 key가 사라져 같은 키가 두 번 실행됐다.
@SystemInvoked 규칙이 매뉴얼 09장의 이벤트 소비 경로까지 막았다. V22가
outbox_event에 가장 강한 잠금을 lock_timeout 없이 걸었다. prod의
fixture- 거부가 HOSTNAME 같은 비밀 아닌 값까지 막았다. 데모 허용 목록의
test·demo 프로필로 sample이 뜨지 않았다. 기동 rule bean의 @Value 필드가
조용히 null이 됐다.

방법: 영역별 4묶음이 test-driven-development 절차로 실패 테스트를
먼저 쓰고 고쳤다. 결정은 재리뷰 README 9.1절이다.
- 두 멱등성 store가 같은 계약을 따른다. lease가 지난 실행 기록은 다시
  돌리지 않고 포기로 표시해 500 RECOVERY_REQUIRED로 답한다. 기록 수명은
  lease + replay TTL이고, replay-ttl < processing-lease면 기동이 실패한다.
- 사용자 transport만 @SystemInvoked use case에 닿지 못한다. 메시지
  handler·scheduler는 새 표시 @SystemCaller(reason)를 달고 use case를
  생성자에서 new로 만든다. 매뉴얼 경로를 테스트로 고정했다.
- 아직 어디에도 적용되지 않은 마이그레이션(V20·V22·알림 V11~V13·sample
  V8_1·grpc V002·V003)에 lock_timeout 5s를 넣고, 텍스트 검사기와 실 PG
  lane을 더했다. 이미 배포된 24개는 동결 목록으로 둔다.
- fixture- 검사는 비밀로 분류된 값만 본다. sample은 local·test·demo에서
  object storage를 끈다. 기동 rule bean에 주입 멤버가 있으면 모든
  프로필에서 기동이 실패한다. 리뷰어가 제안한 "static @Bean 선언 강제"는
  문제없는 등록을 막고 @Value 필드는 통과시켜서 버렸다.
- 이 수정 전에 move된 파일의 쿼터 차이를 찾는 쿼리와 재계산 SQL을
  docs/fileserver/operations.md에 적었다.

위험·한계: lease 뒤 늦게 끝난 첫 시도의 결과는 기록되지 않는다(그 요청도
500). @SystemCaller가 없는 메시지 handler, 주입 멤버가 있는 기동 rule,
working-directory 없이 ./gradlew를 부르는 workflow 명령은 파생 서비스에서
새로 실패하거나 세지지 않는다(업그레이드 안내 B-14~16·B-36·B-39).
spotbugsTestFixtures는 게이트 밖이고 기준 커밋에서도 실패하며, 새 검사기의
정규식 경고 8건이 더해졌다. 실제 CI runner 실행은 확인하지 못했다(Gitea
repo의 Actions가 꺼져 있다).

검증: ci, 두 빌드의 verifyStrictLaneStages --dry-run, mainCheck,
:integrationCheck, -p optional-platforms ci integrationCheck
performanceCheck, :performanceCheck, :releaseCheck(-PgitRevision),
verify-compose-profile-contracts(18 lane) 첫 실행 통과. 쿼터 재계산
SQL은 실 PostgreSQL 16에서 차이 2행 → 0행을 확인했다.

Refs: docs/reviews/2026-11-03-rereview/README.md 9·9.1·10절
BREAKING CHANGE: replay-ttl < processing-lease 설정은 기동 실패, ledger
release가 claim token을 받는다, HttpBodyReadException(IOException 하위)이
본문 실패를 싣는다, 기동 rule·비밀 목록 bean의 주입 멤버 금지,
verifyStrictLaneStages가 workflow 명령의 working-directory로 빌드를 정한다.
2026-11-04 01:47:50 +09:00

7.2 KiB

HTTP Client Platform — Operations Runbook

Metrics

Metric Meaning
http.client.requests Physical attempt timer (Spring standard name, kept deliberately)
http.client.logical.calls User-visible logical call timer
http.client.attempts Attempt counter
http.client.retry.count Retries by reason
http.client.retry.exhausted Retry budget exhausted
http.client.ambiguous Ambiguous outcomes
http.client.timeout Timeouts by stage
http.client.request.bytes Request wire bytes
http.client.response.bytes Response bytes
http.client.active In-flight attempts
http.client.pool.connections Leased and available connections
http.client.pool.pending Pool waiters
http.client.pool.acquire.duration Pool wait time
http.client.dns.duration DNS time
http.client.connect.duration Connect time
http.client.tls.duration TLS time
http.client.circuit.state Circuit state
http.client.bulkhead.rejected Bulkhead rejections
http.client.rate_limit.rejected Local rate-limit rejections
http.client.oauth.refresh Token refresh outcomes
http.client.ssrf.rejected Dynamic target rejections

http.client.requests counts attempts and http.client.logical.calls counts user calls. When they diverge, retries are absorbing failures — which is the first thing to look at during an incident.

Reading an incident

Symptom Likely cause Where to look
logical calls fine, attempts spiking upstream degraded, retries absorbing it http.client.retry.count by reason
http.client.ambiguous non-zero non-idempotent writes reaching SENT_NO_RESPONSE reconcile with the upstream; consider an idempotency key
pool pending climbing pool too small or upstream slow http.client.pool.acquire.duration, pool.connections
circuit open sustained upstream failure: transport errors, 5xx, 408, 429 http.client.circuit.state; local rejections and other 4xx do not open it
http.client.ssrf.rejected non-zero a caller is submitting internal URLs Dynamic Target policy and audit trail

Actuator

GET /actuator/httpclients reports profile name, runtime generation, state, transport, API, protocols, active leases, pool ceiling, credential type, TLS profile id, redirect flag, retry policy, and capability warnings. Base URL, credentials, trust store paths, and resolved IPs are deliberately absent: an actuator endpoint is reachable by more people than a secret store is.

Rotation

Certificates and secrets rotate by building a new runtime generation and swapping the registry pointer, never by mutating a live client. A connection pool holds sockets established under the previous identity, so replacing material without replacing the pool leaves live connections authenticated by a certificate that is meant to be gone.

build new generation → validate → atomic swap → new calls use it
old generation → DRAINING → in-flight calls finish → no new retries → forced close at the drain deadline

Shutdown

RUNNING → DRAINING
new logical calls refused or routed to the new generation
in-flight attempts complete
new retries refused
shutdown timeout
remaining calls cancelled
pool closed

Retry ownership

Exactly one of the application client, an external SDK, or the service mesh may own retries. Two owners multiply traffic during an incident. Record the owner per upstream and check it whenever a mesh retry policy changes.

Error model

Every outbound failure is one of these stable types. The type is derived from the classified failure category, not from whatever the engine happened to throw, so it means the same thing on Apache and JDK. Each carries HttpFailureMetadata: client, operation, method, URI template, evidence, replayability, stage, retryability, attempt, elapsed, remaining deadline, status, trace id — and nothing else.

Exception Raised when Retryable
HttpConfigurationException profile, operation, or capability configuration is invalid never
HttpTargetRejectedException target URI, host, port, header, or address policy refused the request never
HttpDnsException hostname resolution failed or timed out yes, inside budget
HttpPoolAcquireTimeoutException no connection or stream within the pending-acquire budget yes, inside budget
HttpConnectException socket connect failed yes, inside budget
HttpProxyException proxy connect, CONNECT tunnel, or proxy auth failed yes, inside budget
HttpTlsException TLS handshake failed only a transient handshake timeout
HttpRequestWriteException request headers or body could not be fully written only when safely idempotent
HttpResponseTimeoutException final headers or a body chunk did not arrive in time only when safely idempotent
HttpResponseTruncatedException the response ended before the body was complete only when safely idempotent and undelivered
HttpRemoteErrorException non-success status without a problem document per the status rules
HttpProblemDetailException non-success status with a bounded RFC 9457 document per the status rules
HttpRedirectRejectedException a hop violated hop count, origin, method, or replay policy never
HttpAuthenticationException credential materialization or refresh failed never
HttpSerializationException request encoding or response decoding failed never
HttpResponseTooLargeException wire or decoded bytes exceeded the profile limit never
HttpDeadlineExceededException the effective deadline was reached never
HttpCircuitOpenException the upstream circuit is open never
HttpBulkheadRejectedException no attempt or logical admission permit was available never
HttpRateLimitRejectedException the local attempt rate limiter rejected the attempt never
HttpRetryBudgetExhaustedException the per-upstream retry budget refused a further attempt never
HttpRuntimeDrainingException the profile's runtime was retired (certificate rotation) before a granted retry started never — a new call runs on the replacement
HttpUnknownTransportException the engine failed in a way no category above describes; the metadata evidence, not the type, says whether it was sent never
HttpAmbiguousExecutionException a non-idempotent request was sent and the outcome is unknown never — reconcile instead

A streamed body that fails after hand-over is the one place these arrive wrapped: read throws HttpBodyReadException, an IOException as InputStream promises, and its failure() is the HttpResponseTimeoutException or HttpResponseTruncatedException above (re-review 2026-11-04 RR-03-14).

Traces

http.client.operation        logical internal span
└─ http.client.request       attempt 1 CLIENT span
└─ http.client.request       attempt 2 CLIENT span

W3C Trace Context is propagated with a Baggage allowlist. Dynamic Targets do not propagate trace context by default. Retry reason and evidence are recorded as span events; credentials and remote error bodies are never recorded as attributes.