Files
clean-architecture-backend-…/docs/httpclient/operations.md
T
DongHyeonka 1885bf8da7 fix!: 2차 재리뷰가 찾은 결함 29건을 고친다
문제: 재리뷰 수정 커밋(bbe944bd)만 다시 본 2차 재리뷰가 Important 7건,
Minor 22건을 찾았다. 결과 불명 503 뒤 같은 키를 다시 보내면 Redis
store가 lease 지난 실행을 넘겨받아 작업을 다시 돌렸다. replay TTL이
lease보다 짧으면 lease 도중 key가 사라져 같은 키가 두 번 실행됐다.
@SystemInvoked 규칙이 매뉴얼 09장의 이벤트 소비 경로까지 막았다. V22가
outbox_event에 가장 강한 잠금을 lock_timeout 없이 걸었다. prod의
fixture- 거부가 HOSTNAME 같은 비밀 아닌 값까지 막았다. 데모 허용 목록의
test·demo 프로필로 sample이 뜨지 않았다. 기동 rule bean의 @Value 필드가
조용히 null이 됐다.

방법: 영역별 4묶음이 test-driven-development 절차로 실패 테스트를
먼저 쓰고 고쳤다. 결정은 재리뷰 README 9.1절이다.
- 두 멱등성 store가 같은 계약을 따른다. lease가 지난 실행 기록은 다시
  돌리지 않고 포기로 표시해 500 RECOVERY_REQUIRED로 답한다. 기록 수명은
  lease + replay TTL이고, replay-ttl < processing-lease면 기동이 실패한다.
- 사용자 transport만 @SystemInvoked use case에 닿지 못한다. 메시지
  handler·scheduler는 새 표시 @SystemCaller(reason)를 달고 use case를
  생성자에서 new로 만든다. 매뉴얼 경로를 테스트로 고정했다.
- 아직 어디에도 적용되지 않은 마이그레이션(V20·V22·알림 V11~V13·sample
  V8_1·grpc V002·V003)에 lock_timeout 5s를 넣고, 텍스트 검사기와 실 PG
  lane을 더했다. 이미 배포된 24개는 동결 목록으로 둔다.
- fixture- 검사는 비밀로 분류된 값만 본다. sample은 local·test·demo에서
  object storage를 끈다. 기동 rule bean에 주입 멤버가 있으면 모든
  프로필에서 기동이 실패한다. 리뷰어가 제안한 "static @Bean 선언 강제"는
  문제없는 등록을 막고 @Value 필드는 통과시켜서 버렸다.
- 이 수정 전에 move된 파일의 쿼터 차이를 찾는 쿼리와 재계산 SQL을
  docs/fileserver/operations.md에 적었다.

위험·한계: lease 뒤 늦게 끝난 첫 시도의 결과는 기록되지 않는다(그 요청도
500). @SystemCaller가 없는 메시지 handler, 주입 멤버가 있는 기동 rule,
working-directory 없이 ./gradlew를 부르는 workflow 명령은 파생 서비스에서
새로 실패하거나 세지지 않는다(업그레이드 안내 B-14~16·B-36·B-39).
spotbugsTestFixtures는 게이트 밖이고 기준 커밋에서도 실패하며, 새 검사기의
정규식 경고 8건이 더해졌다. 실제 CI runner 실행은 확인하지 못했다(Gitea
repo의 Actions가 꺼져 있다).

검증: ci, 두 빌드의 verifyStrictLaneStages --dry-run, mainCheck,
:integrationCheck, -p optional-platforms ci integrationCheck
performanceCheck, :performanceCheck, :releaseCheck(-PgitRevision),
verify-compose-profile-contracts(18 lane) 첫 실행 통과. 쿼터 재계산
SQL은 실 PostgreSQL 16에서 차이 2행 → 0행을 확인했다.

Refs: docs/reviews/2026-11-03-rereview/README.md 9·9.1·10절
BREAKING CHANGE: replay-ttl < processing-lease 설정은 기동 실패, ledger
release가 claim token을 받는다, HttpBodyReadException(IOException 하위)이
본문 실패를 싣는다, 기동 rule·비밀 목록 bean의 주입 멤버 금지,
verifyStrictLaneStages가 workflow 명령의 working-directory로 빌드를 정한다.
2026-11-04 01:47:50 +09:00

130 lines
7.2 KiB
Markdown

# HTTP Client Platform — Operations Runbook
## Metrics
| Metric | Meaning |
|---|---|
| `http.client.requests` | Physical attempt timer (Spring standard name, kept deliberately) |
| `http.client.logical.calls` | User-visible logical call timer |
| `http.client.attempts` | Attempt counter |
| `http.client.retry.count` | Retries by reason |
| `http.client.retry.exhausted` | Retry budget exhausted |
| `http.client.ambiguous` | Ambiguous outcomes |
| `http.client.timeout` | Timeouts by stage |
| `http.client.request.bytes` | Request wire bytes |
| `http.client.response.bytes` | Response bytes |
| `http.client.active` | In-flight attempts |
| `http.client.pool.connections` | Leased and available connections |
| `http.client.pool.pending` | Pool waiters |
| `http.client.pool.acquire.duration` | Pool wait time |
| `http.client.dns.duration` | DNS time |
| `http.client.connect.duration` | Connect time |
| `http.client.tls.duration` | TLS time |
| `http.client.circuit.state` | Circuit state |
| `http.client.bulkhead.rejected` | Bulkhead rejections |
| `http.client.rate_limit.rejected` | Local rate-limit rejections |
| `http.client.oauth.refresh` | Token refresh outcomes |
| `http.client.ssrf.rejected` | Dynamic target rejections |
`http.client.requests` counts attempts and `http.client.logical.calls` counts user calls. When they
diverge, retries are absorbing failures — which is the first thing to look at during an incident.
## Reading an incident
| Symptom | Likely cause | Where to look |
|---|---|---|
| logical calls fine, attempts spiking | upstream degraded, retries absorbing it | `http.client.retry.count` by reason |
| `http.client.ambiguous` non-zero | non-idempotent writes reaching `SENT_NO_RESPONSE` | reconcile with the upstream; consider an idempotency key |
| pool pending climbing | pool too small or upstream slow | `http.client.pool.acquire.duration`, `pool.connections` |
| circuit open | sustained upstream failure: transport errors, 5xx, 408, 429 | `http.client.circuit.state`; local rejections and other 4xx do not open it |
| `http.client.ssrf.rejected` non-zero | a caller is submitting internal URLs | Dynamic Target policy and audit trail |
## Actuator
`GET /actuator/httpclients` reports profile name, runtime generation, state, transport, API,
protocols, active leases, pool ceiling, credential type, TLS profile id, redirect flag, retry policy,
and capability warnings. Base URL, credentials, trust store paths, and resolved IPs are deliberately
absent: an actuator endpoint is reachable by more people than a secret store is.
## Rotation
Certificates and secrets rotate by building a new runtime generation and swapping the registry
pointer, never by mutating a live client. A connection pool holds sockets established under the
previous identity, so replacing material without replacing the pool leaves live connections
authenticated by a certificate that is meant to be gone.
```text
build new generation → validate → atomic swap → new calls use it
old generation → DRAINING → in-flight calls finish → no new retries → forced close at the drain deadline
```
## Shutdown
```text
RUNNING → DRAINING
new logical calls refused or routed to the new generation
in-flight attempts complete
new retries refused
shutdown timeout
remaining calls cancelled
pool closed
```
## Retry ownership
Exactly one of the application client, an external SDK, or the service mesh may own retries.
Two owners multiply traffic during an incident. Record the owner per upstream and check it whenever
a mesh retry policy changes.
## Error model
Every outbound failure is one of these stable types. The type is derived from the classified failure
category, not from whatever the engine happened to throw, so it means the same thing on Apache and
JDK. Each carries `HttpFailureMetadata`: client, operation, method, URI **template**,
evidence, replayability, stage, retryability, attempt, elapsed, remaining deadline, status, trace id
— and nothing else.
| Exception | Raised when | Retryable |
|---|---|---|
| `HttpConfigurationException` | profile, operation, or capability configuration is invalid | never |
| `HttpTargetRejectedException` | target URI, host, port, header, or address policy refused the request | never |
| `HttpDnsException` | hostname resolution failed or timed out | yes, inside budget |
| `HttpPoolAcquireTimeoutException` | no connection or stream within the pending-acquire budget | yes, inside budget |
| `HttpConnectException` | socket connect failed | yes, inside budget |
| `HttpProxyException` | proxy connect, CONNECT tunnel, or proxy auth failed | yes, inside budget |
| `HttpTlsException` | TLS handshake failed | only a transient handshake timeout |
| `HttpRequestWriteException` | request headers or body could not be fully written | only when safely idempotent |
| `HttpResponseTimeoutException` | final headers or a body chunk did not arrive in time | only when safely idempotent |
| `HttpResponseTruncatedException` | the response ended before the body was complete | only when safely idempotent and undelivered |
| `HttpRemoteErrorException` | non-success status without a problem document | per the status rules |
| `HttpProblemDetailException` | non-success status with a bounded RFC 9457 document | per the status rules |
| `HttpRedirectRejectedException` | a hop violated hop count, origin, method, or replay policy | never |
| `HttpAuthenticationException` | credential materialization or refresh failed | never |
| `HttpSerializationException` | request encoding or response decoding failed | never |
| `HttpResponseTooLargeException` | wire or decoded bytes exceeded the profile limit | never |
| `HttpDeadlineExceededException` | the effective deadline was reached | never |
| `HttpCircuitOpenException` | the upstream circuit is open | never |
| `HttpBulkheadRejectedException` | no attempt or logical admission permit was available | never |
| `HttpRateLimitRejectedException` | the local attempt rate limiter rejected the attempt | never |
| `HttpRetryBudgetExhaustedException` | the per-upstream retry budget refused a further attempt | never |
| `HttpRuntimeDrainingException` | the profile's runtime was retired (certificate rotation) before a granted retry started | never — a new call runs on the replacement |
| `HttpUnknownTransportException` | the engine failed in a way no category above describes; the metadata evidence, not the type, says whether it was sent | never |
| `HttpAmbiguousExecutionException` | a non-idempotent request was sent and the outcome is unknown | never — reconcile instead |
A streamed body that fails after hand-over is the one place these arrive wrapped: `read` throws
`HttpBodyReadException`, an `IOException` as `InputStream` promises, and its `failure()` is the
`HttpResponseTimeoutException` or `HttpResponseTruncatedException` above (re-review 2026-11-04
RR-03-14).
## Traces
```text
http.client.operation logical internal span
└─ http.client.request attempt 1 CLIENT span
└─ http.client.request attempt 2 CLIENT span
```
W3C Trace Context is propagated with a Baggage allowlist. Dynamic Targets do not propagate trace
context by default. Retry reason and evidence are recorded as span events; credentials and remote
error bodies are never recorded as attributes.