Files
clean-architecture-backend-…/docs/web/runbook.md
T
DongHyeonka 34565130e0 fix!: 성숙도 리뷰의 High·P1 결함 19건을 고친다
문제: 2026-11-01 성숙도 리뷰에서 High 21건·P1 25건이 나왔다. 결과를
모르는 일을 "확실함"으로 기록했다(발송 중 죽은 알림 재발송, 커밋 중
끊긴 쓰기를 재시도 가능으로 응답, 재시도 가능한 거부를 DEAD로). 중복을
DB가 아니라 코드 순서로 막았다(정산 job, 포스터 제목). 짝 프론트와
wire 계약이 달랐고(처리 중 409, Envelope null), JWT audience 검증이
기본으로 꺼져 있었고, 트레이스가 나가지 않았고, CI가 strict lane 14개를
한 번도 돌리지 않았다.

방법: 발견마다 실패하는 테스트를 먼저 쓰고 고쳤다. 주요 선택:
- 알림 attempt는 처음부터 "시작 여부 모름"으로 쓴다(호출 직전 갱신은
  쓰기가 하나 늘어 버림).
- 정산 job claim은 lease·fence를 갱신하는 UPDATE 한 문장(처리 전체를
  트랜잭션으로 감싸는 안은 외부 호출 동안 잠금을 잡아 버림).
- 커밋 단계 연결 끊김은 DB_COMPLETION_UNKNOWN(재시도 불가).
- 처리 중 409는 retryable=true + Retry-After: 2, Envelope null 필드는
  web Jackson mixin으로 생략(shared-contract는 stdlib only).
- 트레이싱은 자체 Tracer를 지우고 Boot 4 tracing·OTLP로.
- strict lane을 단계에 연결하고 verifyStrictLaneStages가 고아를 막는다.
발견별 RED·GREEN·버린 대안은 리뷰 README 13절 표에 있다.

위험·한계: JWT 모드는 APP_SECURITY_JWT_AUDIENCE가 없으면 기동하지
않는다. 짝 프론트는 409 retryable과 Envelope 키 규칙을 맞춰야 한다.
이 커밋 시점에는 MinIO 공식 이미지가 사라져 MinIO 2 lane이 실패한다
(다음 커밋에서 대체 이미지로 해결).

검증: ci 통과. mainCheck·integrationCheck는 MinIO 2 lane만 실패(이미지
pull 거부). -p optional-platforms ci integrationCheck, performanceCheck,
releaseCheck(-PgitRevision) 통과.

Refs: docs/reviews/2026-11-01-maturity-review/README.md 2·13절
BREAKING CHANGE: JWT 모드에서 APP_SECURITY_JWT_AUDIENCE가 필수가 되고 role은
audience client와 realm 것만 읽는다. IDEMPOTENT_IN_FLIGHT가 retryable=true,
Retry-After: 2로 바뀌고 Envelope는 null 필드를 보내지 않는다.
2026-11-03 19:52:06 +09:00

4.9 KiB

Web platform runbook

What an operator needs when the HTTP boundary misbehaves. Organised by what you observe, because that is what you have at 3am — not by which module owns it.

Reading a 429

RateLimitInterceptor answers 429 RATE_LIMIT_EXCEEDED with Retry-After when the edge rate limit refuses a caller. RATE_LIMIT_UNAVAILABLE means the rate-limit store could not be reached, not that the caller exceeded anything — look at the store, not the caller.

Errors that are not the Envelope

Every HTTP error the application writes is the Envelope from GlobalExceptionHandler (spring.mvc.problemdetails.enabled=false), with an error.code a client branches on.

  • How to confirm quickly: send a request that must fail validation and check the body has error.code. A body without it (Spring's ProblemDetail, or the container's error page) means the handler is not installed or the response came from the edge.

A response is truncated or the connection dies mid-document

Two different causes, distinguished by the status the client did receive.

  • Client got a status, then nothing — the response failed after commit. The status cannot be retracted, so the connection ends mid-document.
  • Client got nothing from the application — the proxy refused it. client_max_body_size in nginx answers with its own HTML rather than the Envelope. See repository-adaptation.md §8.

A retried write happened twice

The idempotency key is the control, and there are exactly three ways it fails to apply.

  1. The client did not send one. The endpoint does not require the key, so it ran unguarded. Require it if a duplicate is unacceptable.
  2. The client sent a different body. Answered 422 IDEMPOTENT_REQUEST_MISMATCH, never a replay — replaying would hand back a receipt for a request the caller never made.
  3. The record expired. TTL is per-operation with a 72h cap. A retry after expiry is a new request by definition.

A 409 IDEMPOTENT_IN_FLIGHT (retryable=true, Retry-After: 2) is not a failure: an earlier attempt is still running and its result is unknown. The caller resends with the same Idempotency-Key after Retry-After; a new key is a new request and is how a write ends up applied twice. The four idempotency answers are in http-contract.md §2.

Rolling deploy stalls with instances half-drained

shutDownGracefully waits for in-flight work, and an open-but-idle keep-alive connection can count as in-flight. Calling stop() while that wait is in progress hangs.

  • Symptom: an instance neither serving nor exiting, no error in its log.
  • Cause: an unbounded graceful wait. GracefulShutdownProbe bounds it at a stated grace period and stops the server regardless once it elapses; a deployment must do the same.
  • Setting: spring.lifecycle.timeout-per-shutdown-phase. Without a bound the deploy waits for a connection that may never go idle.

Metrics stopped arriving, or the bill jumped

Almost always a high-cardinality tag — typically a routeTemplate carrying a resolved path, which is one series per resource.

  • Check: the tag values in the metrics backend for routeTemplate. Braces mean templates; identifiers mean the resolved path leaked through.
  • Never tags: the URL, the query string, any identifier, the tenant, a key, a token, a cookie, a body. A tag value reaches the metrics backend unredacted and usually a third-party SaaS with it.

Cross-origin requests fail only in the browser

The API answers correctly and the browser refuses the response. CORS is CorsSettings (app.cors.*) applied by SecurityConfig. Startup refuses two settings:

  • APP_SECURITY_CORS_ENABLED=true with no origin.
  • * in APP_SECURITY_CORS_ORIGINS with APP_SECURITY_CORS_ALLOW_CREDENTIALS=true — no browser honours it.

Two more start fine and still fail in the browser:

  • https://*.example.com in the origin list — it is compared as a literal and matches nothing.
  • An origin with a path (https://app.example.com/ui) — an origin is scheme, host and port only.

A preflight answered 401 means CORS ran after authentication. Preflights carry no credentials by design; the order is asserted by WebPipelineOrderContract.

Two access-log lines for one request

An async request passes through the servlet filter chain twice — once for the initial request and again on the ASYNC redispatch. A completion recorded without checking isAsyncStarted() is written both times, and every latency percentile computed from that data is wrong while looking plausible. WebPipelineOrderContract asserts exactly one observation per logical request on every container.

Verification commands

cd src
./gradlew :adapter:inbound:web:test                          # unit, boundary, architecture
./gradlew :adapter:inbound:web:systemTest                    # Tomcat, real socket
./gradlew :adapter:inbound:web:webJettyCompatTest            # second servlet container