문제: 재리뷰 수정 커밋(bbe944bd)만 다시 본 2차 재리뷰가 Important 7건,
Minor 22건을 찾았다. 결과 불명 503 뒤 같은 키를 다시 보내면 Redis
store가 lease 지난 실행을 넘겨받아 작업을 다시 돌렸다. replay TTL이
lease보다 짧으면 lease 도중 key가 사라져 같은 키가 두 번 실행됐다.
@SystemInvoked 규칙이 매뉴얼 09장의 이벤트 소비 경로까지 막았다. V22가
outbox_event에 가장 강한 잠금을 lock_timeout 없이 걸었다. prod의
fixture- 거부가 HOSTNAME 같은 비밀 아닌 값까지 막았다. 데모 허용 목록의
test·demo 프로필로 sample이 뜨지 않았다. 기동 rule bean의 @Value 필드가
조용히 null이 됐다.
방법: 영역별 4묶음이 test-driven-development 절차로 실패 테스트를
먼저 쓰고 고쳤다. 결정은 재리뷰 README 9.1절이다.
- 두 멱등성 store가 같은 계약을 따른다. lease가 지난 실행 기록은 다시
돌리지 않고 포기로 표시해 500 RECOVERY_REQUIRED로 답한다. 기록 수명은
lease + replay TTL이고, replay-ttl < processing-lease면 기동이 실패한다.
- 사용자 transport만 @SystemInvoked use case에 닿지 못한다. 메시지
handler·scheduler는 새 표시 @SystemCaller(reason)를 달고 use case를
생성자에서 new로 만든다. 매뉴얼 경로를 테스트로 고정했다.
- 아직 어디에도 적용되지 않은 마이그레이션(V20·V22·알림 V11~V13·sample
V8_1·grpc V002·V003)에 lock_timeout 5s를 넣고, 텍스트 검사기와 실 PG
lane을 더했다. 이미 배포된 24개는 동결 목록으로 둔다.
- fixture- 검사는 비밀로 분류된 값만 본다. sample은 local·test·demo에서
object storage를 끈다. 기동 rule bean에 주입 멤버가 있으면 모든
프로필에서 기동이 실패한다. 리뷰어가 제안한 "static @Bean 선언 강제"는
문제없는 등록을 막고 @Value 필드는 통과시켜서 버렸다.
- 이 수정 전에 move된 파일의 쿼터 차이를 찾는 쿼리와 재계산 SQL을
docs/fileserver/operations.md에 적었다.
위험·한계: lease 뒤 늦게 끝난 첫 시도의 결과는 기록되지 않는다(그 요청도
500). @SystemCaller가 없는 메시지 handler, 주입 멤버가 있는 기동 rule,
working-directory 없이 ./gradlew를 부르는 workflow 명령은 파생 서비스에서
새로 실패하거나 세지지 않는다(업그레이드 안내 B-14~16·B-36·B-39).
spotbugsTestFixtures는 게이트 밖이고 기준 커밋에서도 실패하며, 새 검사기의
정규식 경고 8건이 더해졌다. 실제 CI runner 실행은 확인하지 못했다(Gitea
repo의 Actions가 꺼져 있다).
검증: ci, 두 빌드의 verifyStrictLaneStages --dry-run, mainCheck,
:integrationCheck, -p optional-platforms ci integrationCheck
performanceCheck, :performanceCheck, :releaseCheck(-PgitRevision),
verify-compose-profile-contracts(18 lane) 첫 실행 통과. 쿼터 재계산
SQL은 실 PostgreSQL 16에서 차이 2행 → 0행을 확인했다.
Refs: docs/reviews/2026-11-03-rereview/README.md 9·9.1·10절
BREAKING CHANGE: replay-ttl < processing-lease 설정은 기동 실패, ledger
release가 claim token을 받는다, HttpBodyReadException(IOException 하위)이
본문 실패를 싣는다, 기동 rule·비밀 목록 bean의 주입 멤버 금지,
verifyStrictLaneStages가 workflow 명령의 working-directory로 빌드를 정한다.
130 lines
7.2 KiB
Markdown
130 lines
7.2 KiB
Markdown
# HTTP Client Platform — Operations Runbook
|
|
|
|
## Metrics
|
|
|
|
| Metric | Meaning |
|
|
|---|---|
|
|
| `http.client.requests` | Physical attempt timer (Spring standard name, kept deliberately) |
|
|
| `http.client.logical.calls` | User-visible logical call timer |
|
|
| `http.client.attempts` | Attempt counter |
|
|
| `http.client.retry.count` | Retries by reason |
|
|
| `http.client.retry.exhausted` | Retry budget exhausted |
|
|
| `http.client.ambiguous` | Ambiguous outcomes |
|
|
| `http.client.timeout` | Timeouts by stage |
|
|
| `http.client.request.bytes` | Request wire bytes |
|
|
| `http.client.response.bytes` | Response bytes |
|
|
| `http.client.active` | In-flight attempts |
|
|
| `http.client.pool.connections` | Leased and available connections |
|
|
| `http.client.pool.pending` | Pool waiters |
|
|
| `http.client.pool.acquire.duration` | Pool wait time |
|
|
| `http.client.dns.duration` | DNS time |
|
|
| `http.client.connect.duration` | Connect time |
|
|
| `http.client.tls.duration` | TLS time |
|
|
| `http.client.circuit.state` | Circuit state |
|
|
| `http.client.bulkhead.rejected` | Bulkhead rejections |
|
|
| `http.client.rate_limit.rejected` | Local rate-limit rejections |
|
|
| `http.client.oauth.refresh` | Token refresh outcomes |
|
|
| `http.client.ssrf.rejected` | Dynamic target rejections |
|
|
|
|
`http.client.requests` counts attempts and `http.client.logical.calls` counts user calls. When they
|
|
diverge, retries are absorbing failures — which is the first thing to look at during an incident.
|
|
|
|
## Reading an incident
|
|
|
|
| Symptom | Likely cause | Where to look |
|
|
|---|---|---|
|
|
| logical calls fine, attempts spiking | upstream degraded, retries absorbing it | `http.client.retry.count` by reason |
|
|
| `http.client.ambiguous` non-zero | non-idempotent writes reaching `SENT_NO_RESPONSE` | reconcile with the upstream; consider an idempotency key |
|
|
| pool pending climbing | pool too small or upstream slow | `http.client.pool.acquire.duration`, `pool.connections` |
|
|
| circuit open | sustained upstream failure: transport errors, 5xx, 408, 429 | `http.client.circuit.state`; local rejections and other 4xx do not open it |
|
|
| `http.client.ssrf.rejected` non-zero | a caller is submitting internal URLs | Dynamic Target policy and audit trail |
|
|
|
|
## Actuator
|
|
|
|
`GET /actuator/httpclients` reports profile name, runtime generation, state, transport, API,
|
|
protocols, active leases, pool ceiling, credential type, TLS profile id, redirect flag, retry policy,
|
|
and capability warnings. Base URL, credentials, trust store paths, and resolved IPs are deliberately
|
|
absent: an actuator endpoint is reachable by more people than a secret store is.
|
|
|
|
## Rotation
|
|
|
|
Certificates and secrets rotate by building a new runtime generation and swapping the registry
|
|
pointer, never by mutating a live client. A connection pool holds sockets established under the
|
|
previous identity, so replacing material without replacing the pool leaves live connections
|
|
authenticated by a certificate that is meant to be gone.
|
|
|
|
```text
|
|
build new generation → validate → atomic swap → new calls use it
|
|
old generation → DRAINING → in-flight calls finish → no new retries → forced close at the drain deadline
|
|
```
|
|
|
|
## Shutdown
|
|
|
|
```text
|
|
RUNNING → DRAINING
|
|
new logical calls refused or routed to the new generation
|
|
in-flight attempts complete
|
|
new retries refused
|
|
shutdown timeout
|
|
remaining calls cancelled
|
|
pool closed
|
|
```
|
|
|
|
## Retry ownership
|
|
|
|
Exactly one of the application client, an external SDK, or the service mesh may own retries.
|
|
Two owners multiply traffic during an incident. Record the owner per upstream and check it whenever
|
|
a mesh retry policy changes.
|
|
|
|
## Error model
|
|
|
|
Every outbound failure is one of these stable types. The type is derived from the classified failure
|
|
category, not from whatever the engine happened to throw, so it means the same thing on Apache and
|
|
JDK. Each carries `HttpFailureMetadata`: client, operation, method, URI **template**,
|
|
evidence, replayability, stage, retryability, attempt, elapsed, remaining deadline, status, trace id
|
|
— and nothing else.
|
|
|
|
| Exception | Raised when | Retryable |
|
|
|---|---|---|
|
|
| `HttpConfigurationException` | profile, operation, or capability configuration is invalid | never |
|
|
| `HttpTargetRejectedException` | target URI, host, port, header, or address policy refused the request | never |
|
|
| `HttpDnsException` | hostname resolution failed or timed out | yes, inside budget |
|
|
| `HttpPoolAcquireTimeoutException` | no connection or stream within the pending-acquire budget | yes, inside budget |
|
|
| `HttpConnectException` | socket connect failed | yes, inside budget |
|
|
| `HttpProxyException` | proxy connect, CONNECT tunnel, or proxy auth failed | yes, inside budget |
|
|
| `HttpTlsException` | TLS handshake failed | only a transient handshake timeout |
|
|
| `HttpRequestWriteException` | request headers or body could not be fully written | only when safely idempotent |
|
|
| `HttpResponseTimeoutException` | final headers or a body chunk did not arrive in time | only when safely idempotent |
|
|
| `HttpResponseTruncatedException` | the response ended before the body was complete | only when safely idempotent and undelivered |
|
|
| `HttpRemoteErrorException` | non-success status without a problem document | per the status rules |
|
|
| `HttpProblemDetailException` | non-success status with a bounded RFC 9457 document | per the status rules |
|
|
| `HttpRedirectRejectedException` | a hop violated hop count, origin, method, or replay policy | never |
|
|
| `HttpAuthenticationException` | credential materialization or refresh failed | never |
|
|
| `HttpSerializationException` | request encoding or response decoding failed | never |
|
|
| `HttpResponseTooLargeException` | wire or decoded bytes exceeded the profile limit | never |
|
|
| `HttpDeadlineExceededException` | the effective deadline was reached | never |
|
|
| `HttpCircuitOpenException` | the upstream circuit is open | never |
|
|
| `HttpBulkheadRejectedException` | no attempt or logical admission permit was available | never |
|
|
| `HttpRateLimitRejectedException` | the local attempt rate limiter rejected the attempt | never |
|
|
| `HttpRetryBudgetExhaustedException` | the per-upstream retry budget refused a further attempt | never |
|
|
| `HttpRuntimeDrainingException` | the profile's runtime was retired (certificate rotation) before a granted retry started | never — a new call runs on the replacement |
|
|
| `HttpUnknownTransportException` | the engine failed in a way no category above describes; the metadata evidence, not the type, says whether it was sent | never |
|
|
| `HttpAmbiguousExecutionException` | a non-idempotent request was sent and the outcome is unknown | never — reconcile instead |
|
|
|
|
A streamed body that fails after hand-over is the one place these arrive wrapped: `read` throws
|
|
`HttpBodyReadException`, an `IOException` as `InputStream` promises, and its `failure()` is the
|
|
`HttpResponseTimeoutException` or `HttpResponseTruncatedException` above (re-review 2026-11-04
|
|
RR-03-14).
|
|
|
|
## Traces
|
|
|
|
```text
|
|
http.client.operation logical internal span
|
|
└─ http.client.request attempt 1 CLIENT span
|
|
└─ http.client.request attempt 2 CLIENT span
|
|
```
|
|
|
|
W3C Trace Context is propagated with a Baggage allowlist. Dynamic Targets do not propagate trace
|
|
context by default. Retry reason and evidence are recorded as span events; credentials and remote
|
|
error bodies are never recorded as attributes.
|