문제: 2026-11-01 성숙도 리뷰에서 High 21건·P1 25건이 나왔다. 결과를 모르는 일을 "확실함"으로 기록했다(발송 중 죽은 알림 재발송, 커밋 중 끊긴 쓰기를 재시도 가능으로 응답, 재시도 가능한 거부를 DEAD로). 중복을 DB가 아니라 코드 순서로 막았다(정산 job, 포스터 제목). 짝 프론트와 wire 계약이 달랐고(처리 중 409, Envelope null), JWT audience 검증이 기본으로 꺼져 있었고, 트레이스가 나가지 않았고, CI가 strict lane 14개를 한 번도 돌리지 않았다. 방법: 발견마다 실패하는 테스트를 먼저 쓰고 고쳤다. 주요 선택: - 알림 attempt는 처음부터 "시작 여부 모름"으로 쓴다(호출 직전 갱신은 쓰기가 하나 늘어 버림). - 정산 job claim은 lease·fence를 갱신하는 UPDATE 한 문장(처리 전체를 트랜잭션으로 감싸는 안은 외부 호출 동안 잠금을 잡아 버림). - 커밋 단계 연결 끊김은 DB_COMPLETION_UNKNOWN(재시도 불가). - 처리 중 409는 retryable=true + Retry-After: 2, Envelope null 필드는 web Jackson mixin으로 생략(shared-contract는 stdlib only). - 트레이싱은 자체 Tracer를 지우고 Boot 4 tracing·OTLP로. - strict lane을 단계에 연결하고 verifyStrictLaneStages가 고아를 막는다. 발견별 RED·GREEN·버린 대안은 리뷰 README 13절 표에 있다. 위험·한계: JWT 모드는 APP_SECURITY_JWT_AUDIENCE가 없으면 기동하지 않는다. 짝 프론트는 409 retryable과 Envelope 키 규칙을 맞춰야 한다. 이 커밋 시점에는 MinIO 공식 이미지가 사라져 MinIO 2 lane이 실패한다 (다음 커밋에서 대체 이미지로 해결). 검증: ci 통과. mainCheck·integrationCheck는 MinIO 2 lane만 실패(이미지 pull 거부). -p optional-platforms ci integrationCheck, performanceCheck, releaseCheck(-PgitRevision) 통과. Refs: docs/reviews/2026-11-01-maturity-review/README.md 2·13절 BREAKING CHANGE: JWT 모드에서 APP_SECURITY_JWT_AUDIENCE가 필수가 되고 role은 audience client와 realm 것만 읽는다. IDEMPOTENT_IN_FLIGHT가 retryable=true, Retry-After: 2로 바뀌고 Envelope는 null 필드를 보내지 않는다.
125 lines
6.9 KiB
Markdown
125 lines
6.9 KiB
Markdown
# HTTP Client Platform — Operations Runbook
|
|
|
|
## Metrics
|
|
|
|
| Metric | Meaning |
|
|
|---|---|
|
|
| `http.client.requests` | Physical attempt timer (Spring standard name, kept deliberately) |
|
|
| `http.client.logical.calls` | User-visible logical call timer |
|
|
| `http.client.attempts` | Attempt counter |
|
|
| `http.client.retry.count` | Retries by reason |
|
|
| `http.client.retry.exhausted` | Retry budget exhausted |
|
|
| `http.client.ambiguous` | Ambiguous outcomes |
|
|
| `http.client.timeout` | Timeouts by stage |
|
|
| `http.client.request.bytes` | Request wire bytes |
|
|
| `http.client.response.bytes` | Response bytes |
|
|
| `http.client.active` | In-flight attempts |
|
|
| `http.client.pool.connections` | Leased and available connections |
|
|
| `http.client.pool.pending` | Pool waiters |
|
|
| `http.client.pool.acquire.duration` | Pool wait time |
|
|
| `http.client.dns.duration` | DNS time |
|
|
| `http.client.connect.duration` | Connect time |
|
|
| `http.client.tls.duration` | TLS time |
|
|
| `http.client.circuit.state` | Circuit state |
|
|
| `http.client.bulkhead.rejected` | Bulkhead rejections |
|
|
| `http.client.rate_limit.rejected` | Local rate-limit rejections |
|
|
| `http.client.oauth.refresh` | Token refresh outcomes |
|
|
| `http.client.ssrf.rejected` | Dynamic target rejections |
|
|
|
|
`http.client.requests` counts attempts and `http.client.logical.calls` counts user calls. When they
|
|
diverge, retries are absorbing failures — which is the first thing to look at during an incident.
|
|
|
|
## Reading an incident
|
|
|
|
| Symptom | Likely cause | Where to look |
|
|
|---|---|---|
|
|
| logical calls fine, attempts spiking | upstream degraded, retries absorbing it | `http.client.retry.count` by reason |
|
|
| `http.client.ambiguous` non-zero | non-idempotent writes reaching `SENT_NO_RESPONSE` | reconcile with the upstream; consider an idempotency key |
|
|
| pool pending climbing | pool too small or upstream slow | `http.client.pool.acquire.duration`, `pool.connections` |
|
|
| circuit open | sustained upstream failure: transport errors, 5xx, 408, 429 | `http.client.circuit.state`; local rejections and other 4xx do not open it |
|
|
| `http.client.ssrf.rejected` non-zero | a caller is submitting internal URLs | Dynamic Target policy and audit trail |
|
|
|
|
## Actuator
|
|
|
|
`GET /actuator/httpclients` reports profile name, runtime generation, state, transport, API,
|
|
protocols, active leases, pool ceiling, credential type, TLS profile id, redirect flag, retry policy,
|
|
and capability warnings. Base URL, credentials, trust store paths, and resolved IPs are deliberately
|
|
absent: an actuator endpoint is reachable by more people than a secret store is.
|
|
|
|
## Rotation
|
|
|
|
Certificates and secrets rotate by building a new runtime generation and swapping the registry
|
|
pointer, never by mutating a live client. A connection pool holds sockets established under the
|
|
previous identity, so replacing material without replacing the pool leaves live connections
|
|
authenticated by a certificate that is meant to be gone.
|
|
|
|
```text
|
|
build new generation → validate → atomic swap → new calls use it
|
|
old generation → DRAINING → in-flight calls finish → no new retries → forced close at the drain deadline
|
|
```
|
|
|
|
## Shutdown
|
|
|
|
```text
|
|
RUNNING → DRAINING
|
|
new logical calls refused or routed to the new generation
|
|
in-flight attempts complete
|
|
new retries refused
|
|
shutdown timeout
|
|
remaining calls cancelled
|
|
pool closed
|
|
```
|
|
|
|
## Retry ownership
|
|
|
|
Exactly one of the application client, an external SDK, or the service mesh may own retries.
|
|
Two owners multiply traffic during an incident. Record the owner per upstream and check it whenever
|
|
a mesh retry policy changes.
|
|
|
|
## Error model
|
|
|
|
Every outbound failure is one of these stable types. The type is derived from the classified failure
|
|
category, not from whatever the engine happened to throw, so it means the same thing on Apache and
|
|
JDK. Each carries `HttpFailureMetadata`: client, operation, method, URI **template**,
|
|
evidence, replayability, stage, retryability, attempt, elapsed, remaining deadline, status, trace id
|
|
— and nothing else.
|
|
|
|
| Exception | Raised when | Retryable |
|
|
|---|---|---|
|
|
| `HttpConfigurationException` | profile, operation, or capability configuration is invalid | never |
|
|
| `HttpTargetRejectedException` | target URI, host, port, header, or address policy refused the request | never |
|
|
| `HttpDnsException` | hostname resolution failed or timed out | yes, inside budget |
|
|
| `HttpPoolAcquireTimeoutException` | no connection or stream within the pending-acquire budget | yes, inside budget |
|
|
| `HttpConnectException` | socket connect failed | yes, inside budget |
|
|
| `HttpProxyException` | proxy connect, CONNECT tunnel, or proxy auth failed | yes, inside budget |
|
|
| `HttpTlsException` | TLS handshake failed | only a transient handshake timeout |
|
|
| `HttpRequestWriteException` | request headers or body could not be fully written | only when safely idempotent |
|
|
| `HttpResponseTimeoutException` | final headers or a body chunk did not arrive in time | only when safely idempotent |
|
|
| `HttpResponseTruncatedException` | the response ended before the body was complete | only when safely idempotent and undelivered |
|
|
| `HttpRemoteErrorException` | non-success status without a problem document | per the status rules |
|
|
| `HttpProblemDetailException` | non-success status with a bounded RFC 9457 document | per the status rules |
|
|
| `HttpRedirectRejectedException` | a hop violated hop count, origin, method, or replay policy | never |
|
|
| `HttpAuthenticationException` | credential materialization or refresh failed | never |
|
|
| `HttpSerializationException` | request encoding or response decoding failed | never |
|
|
| `HttpResponseTooLargeException` | wire or decoded bytes exceeded the profile limit | never |
|
|
| `HttpDeadlineExceededException` | the effective deadline was reached | never |
|
|
| `HttpCircuitOpenException` | the upstream circuit is open | never |
|
|
| `HttpBulkheadRejectedException` | no attempt or logical admission permit was available | never |
|
|
| `HttpRateLimitRejectedException` | the local attempt rate limiter rejected the attempt | never |
|
|
| `HttpRetryBudgetExhaustedException` | the per-upstream retry budget refused a further attempt | never |
|
|
| `HttpRuntimeDrainingException` | the profile's runtime was retired (certificate rotation) before a granted retry started | never — a new call runs on the replacement |
|
|
| `HttpUnknownTransportException` | the engine failed in a way no category above describes; the metadata evidence, not the type, says whether it was sent | never |
|
|
| `HttpAmbiguousExecutionException` | a non-idempotent request was sent and the outcome is unknown | never — reconcile instead |
|
|
|
|
## Traces
|
|
|
|
```text
|
|
http.client.operation logical internal span
|
|
└─ http.client.request attempt 1 CLIENT span
|
|
└─ http.client.request attempt 2 CLIENT span
|
|
```
|
|
|
|
W3C Trace Context is propagated with a Baggage allowlist. Dynamic Targets do not propagate trace
|
|
context by default. Retry reason and evidence are recorded as span events; credentials and remote
|
|
error bodies are never recorded as attributes.
|