Files
clean-architecture-backend-…/docs/httpclient/operations.md
T
DongHyeonka 34565130e0 fix!: 성숙도 리뷰의 High·P1 결함 19건을 고친다
문제: 2026-11-01 성숙도 리뷰에서 High 21건·P1 25건이 나왔다. 결과를
모르는 일을 "확실함"으로 기록했다(발송 중 죽은 알림 재발송, 커밋 중
끊긴 쓰기를 재시도 가능으로 응답, 재시도 가능한 거부를 DEAD로). 중복을
DB가 아니라 코드 순서로 막았다(정산 job, 포스터 제목). 짝 프론트와
wire 계약이 달랐고(처리 중 409, Envelope null), JWT audience 검증이
기본으로 꺼져 있었고, 트레이스가 나가지 않았고, CI가 strict lane 14개를
한 번도 돌리지 않았다.

방법: 발견마다 실패하는 테스트를 먼저 쓰고 고쳤다. 주요 선택:
- 알림 attempt는 처음부터 "시작 여부 모름"으로 쓴다(호출 직전 갱신은
  쓰기가 하나 늘어 버림).
- 정산 job claim은 lease·fence를 갱신하는 UPDATE 한 문장(처리 전체를
  트랜잭션으로 감싸는 안은 외부 호출 동안 잠금을 잡아 버림).
- 커밋 단계 연결 끊김은 DB_COMPLETION_UNKNOWN(재시도 불가).
- 처리 중 409는 retryable=true + Retry-After: 2, Envelope null 필드는
  web Jackson mixin으로 생략(shared-contract는 stdlib only).
- 트레이싱은 자체 Tracer를 지우고 Boot 4 tracing·OTLP로.
- strict lane을 단계에 연결하고 verifyStrictLaneStages가 고아를 막는다.
발견별 RED·GREEN·버린 대안은 리뷰 README 13절 표에 있다.

위험·한계: JWT 모드는 APP_SECURITY_JWT_AUDIENCE가 없으면 기동하지
않는다. 짝 프론트는 409 retryable과 Envelope 키 규칙을 맞춰야 한다.
이 커밋 시점에는 MinIO 공식 이미지가 사라져 MinIO 2 lane이 실패한다
(다음 커밋에서 대체 이미지로 해결).

검증: ci 통과. mainCheck·integrationCheck는 MinIO 2 lane만 실패(이미지
pull 거부). -p optional-platforms ci integrationCheck, performanceCheck,
releaseCheck(-PgitRevision) 통과.

Refs: docs/reviews/2026-11-01-maturity-review/README.md 2·13절
BREAKING CHANGE: JWT 모드에서 APP_SECURITY_JWT_AUDIENCE가 필수가 되고 role은
audience client와 realm 것만 읽는다. IDEMPOTENT_IN_FLIGHT가 retryable=true,
Retry-After: 2로 바뀌고 Envelope는 null 필드를 보내지 않는다.
2026-11-03 19:52:06 +09:00

125 lines
6.9 KiB
Markdown

# HTTP Client Platform — Operations Runbook
## Metrics
| Metric | Meaning |
|---|---|
| `http.client.requests` | Physical attempt timer (Spring standard name, kept deliberately) |
| `http.client.logical.calls` | User-visible logical call timer |
| `http.client.attempts` | Attempt counter |
| `http.client.retry.count` | Retries by reason |
| `http.client.retry.exhausted` | Retry budget exhausted |
| `http.client.ambiguous` | Ambiguous outcomes |
| `http.client.timeout` | Timeouts by stage |
| `http.client.request.bytes` | Request wire bytes |
| `http.client.response.bytes` | Response bytes |
| `http.client.active` | In-flight attempts |
| `http.client.pool.connections` | Leased and available connections |
| `http.client.pool.pending` | Pool waiters |
| `http.client.pool.acquire.duration` | Pool wait time |
| `http.client.dns.duration` | DNS time |
| `http.client.connect.duration` | Connect time |
| `http.client.tls.duration` | TLS time |
| `http.client.circuit.state` | Circuit state |
| `http.client.bulkhead.rejected` | Bulkhead rejections |
| `http.client.rate_limit.rejected` | Local rate-limit rejections |
| `http.client.oauth.refresh` | Token refresh outcomes |
| `http.client.ssrf.rejected` | Dynamic target rejections |
`http.client.requests` counts attempts and `http.client.logical.calls` counts user calls. When they
diverge, retries are absorbing failures — which is the first thing to look at during an incident.
## Reading an incident
| Symptom | Likely cause | Where to look |
|---|---|---|
| logical calls fine, attempts spiking | upstream degraded, retries absorbing it | `http.client.retry.count` by reason |
| `http.client.ambiguous` non-zero | non-idempotent writes reaching `SENT_NO_RESPONSE` | reconcile with the upstream; consider an idempotency key |
| pool pending climbing | pool too small or upstream slow | `http.client.pool.acquire.duration`, `pool.connections` |
| circuit open | sustained upstream failure: transport errors, 5xx, 408, 429 | `http.client.circuit.state`; local rejections and other 4xx do not open it |
| `http.client.ssrf.rejected` non-zero | a caller is submitting internal URLs | Dynamic Target policy and audit trail |
## Actuator
`GET /actuator/httpclients` reports profile name, runtime generation, state, transport, API,
protocols, active leases, pool ceiling, credential type, TLS profile id, redirect flag, retry policy,
and capability warnings. Base URL, credentials, trust store paths, and resolved IPs are deliberately
absent: an actuator endpoint is reachable by more people than a secret store is.
## Rotation
Certificates and secrets rotate by building a new runtime generation and swapping the registry
pointer, never by mutating a live client. A connection pool holds sockets established under the
previous identity, so replacing material without replacing the pool leaves live connections
authenticated by a certificate that is meant to be gone.
```text
build new generation → validate → atomic swap → new calls use it
old generation → DRAINING → in-flight calls finish → no new retries → forced close at the drain deadline
```
## Shutdown
```text
RUNNING → DRAINING
new logical calls refused or routed to the new generation
in-flight attempts complete
new retries refused
shutdown timeout
remaining calls cancelled
pool closed
```
## Retry ownership
Exactly one of the application client, an external SDK, or the service mesh may own retries.
Two owners multiply traffic during an incident. Record the owner per upstream and check it whenever
a mesh retry policy changes.
## Error model
Every outbound failure is one of these stable types. The type is derived from the classified failure
category, not from whatever the engine happened to throw, so it means the same thing on Apache and
JDK. Each carries `HttpFailureMetadata`: client, operation, method, URI **template**,
evidence, replayability, stage, retryability, attempt, elapsed, remaining deadline, status, trace id
— and nothing else.
| Exception | Raised when | Retryable |
|---|---|---|
| `HttpConfigurationException` | profile, operation, or capability configuration is invalid | never |
| `HttpTargetRejectedException` | target URI, host, port, header, or address policy refused the request | never |
| `HttpDnsException` | hostname resolution failed or timed out | yes, inside budget |
| `HttpPoolAcquireTimeoutException` | no connection or stream within the pending-acquire budget | yes, inside budget |
| `HttpConnectException` | socket connect failed | yes, inside budget |
| `HttpProxyException` | proxy connect, CONNECT tunnel, or proxy auth failed | yes, inside budget |
| `HttpTlsException` | TLS handshake failed | only a transient handshake timeout |
| `HttpRequestWriteException` | request headers or body could not be fully written | only when safely idempotent |
| `HttpResponseTimeoutException` | final headers or a body chunk did not arrive in time | only when safely idempotent |
| `HttpResponseTruncatedException` | the response ended before the body was complete | only when safely idempotent and undelivered |
| `HttpRemoteErrorException` | non-success status without a problem document | per the status rules |
| `HttpProblemDetailException` | non-success status with a bounded RFC 9457 document | per the status rules |
| `HttpRedirectRejectedException` | a hop violated hop count, origin, method, or replay policy | never |
| `HttpAuthenticationException` | credential materialization or refresh failed | never |
| `HttpSerializationException` | request encoding or response decoding failed | never |
| `HttpResponseTooLargeException` | wire or decoded bytes exceeded the profile limit | never |
| `HttpDeadlineExceededException` | the effective deadline was reached | never |
| `HttpCircuitOpenException` | the upstream circuit is open | never |
| `HttpBulkheadRejectedException` | no attempt or logical admission permit was available | never |
| `HttpRateLimitRejectedException` | the local attempt rate limiter rejected the attempt | never |
| `HttpRetryBudgetExhaustedException` | the per-upstream retry budget refused a further attempt | never |
| `HttpRuntimeDrainingException` | the profile's runtime was retired (certificate rotation) before a granted retry started | never — a new call runs on the replacement |
| `HttpUnknownTransportException` | the engine failed in a way no category above describes; the metadata evidence, not the type, says whether it was sent | never |
| `HttpAmbiguousExecutionException` | a non-idempotent request was sent and the outcome is unknown | never — reconcile instead |
## Traces
```text
http.client.operation logical internal span
└─ http.client.request attempt 1 CLIENT span
└─ http.client.request attempt 2 CLIENT span
```
W3C Trace Context is propagated with a Baggage allowlist. Dynamic Targets do not propagate trace
context by default. Retry reason and evidence are recorded as span events; credentials and remote
error bodies are never recorded as attributes.