문제: High·P1을 고친 뒤에도 P2·P3 약 90건이 남았다. 연결·구독·메시지 크기 상한이 없는 전송(WebSocket·gRPC·GraphQL deadline), 낙관적 충돌의 500, upsert 경쟁, Mongo 무제한 timeout, 샘플 목록 N+1, 갈라진 복사본 (TenantId 3벌, 이벤트 ID 문법 5곳, SQLState 표 2벌), 문서를 읽는 PR 테스트와 캐시를 막는 lane 설정 등이다. 새 기능을 어디에 어떤 순서로 만드는지 알려 주는 문서와 도구가 없었다. MinIO 공식 이미지가 사라져 객체 저장소 lane이 돌지 않았다. 방법: 영역별 묶음으로 실패 테스트를 먼저 쓰고 고쳤다. 고치는 중에 찾은 결함도 함께 고쳤다(durable operation id가 UUIDv4라 이벤트가 전부 DEAD, S3 어댑터 생성·작은 chunk 읽기 실패, Redis 조건부 기록의 EVALSHA 권한). MinIO는 pgsty/minio를 digest로 고정했다(Chainguard는 태그·실행 사용자· mc 이미지 문제로 버림). docs/manual/에 작업 순서와 판단 지점을 쓰고, scripts/new-feature.sh가 5개 leaf에 뼈대를 만든다. 생성→빌드→통과 왕복을 verifyFeatureScaffold가 mainCheck에서 본다. 묶음별 결과와 결정은 리뷰 README 14절에 있다. 위험·한계: 파생 서비스 동작이 바뀐다. JDBC batch 50 기본, 운영 jar에서 H2 제거, 트랜잭션 30초 제한, 격리 수준을 Hikari에서 고정, gRPC· WebSocket·GraphQL 새 한도, correlation ID 규칙 밖이면 서버가 교체, 샘플 키 오류 코드 IDEMPOTENCY_KEY_INVALID. 새 leaf 두 개 (composition:durable-operation, adapter:inbound:messaging)와 새 마이그레이션(postgresql V20·V21, notification V11·V12, sample V8_1, grpc-ledger V002)이 생긴다. V21 유일 인덱스는 기존 중복이 있으면 실패한다(runbook에 확인 쿼리). 검증: ci, mainCheck(생성 명령 왕복 포함), integrationCheck(MinIO 포함 전 컨테이너 lane), -p optional-platforms ci integrationCheck 통과. Refs: docs/reviews/2026-11-01-maturity-review/README.md 14절 BREAKING CHANGE: JDBC batch 기본 켜짐, H2 운영 의존 제거, 트랜잭션 기본 제한 30초, gRPC·WebSocket·GraphQL 서버 한도 기본값 추가, 샘플 Idempotency-Key 오류 코드 변경.
5.4 KiB
Web platform runbook
What an operator needs when the HTTP boundary misbehaves. Organised by what you observe, because that is what you have at 3am — not by which module owns it.
Reading a 429
RateLimitInterceptor answers 429 RATE_LIMIT_EXCEEDED with Retry-After when the edge rate
limit refuses a caller. RATE_LIMIT_UNAVAILABLE means the rate-limit store could not be reached,
not that the caller exceeded anything — look at the store, not the caller.
Errors that are not the Envelope
Every HTTP error the application writes is the Envelope from GlobalExceptionHandler
(spring.mvc.problemdetails.enabled=false), with an error.code a client branches on.
- How to confirm quickly: send a request that must fail validation and check the body has
error.code. A body without it (Spring'sProblemDetail, or the container's error page) means the handler is not installed or the response came from the edge.
A response is truncated or the connection dies mid-document
Two different causes, distinguished by the status the client did receive.
- Client got a status, then nothing — the response failed after commit. The status cannot be retracted, so the connection ends mid-document.
- Client got nothing from the application — the proxy refused it.
client_max_body_sizein nginx answers with its own HTML rather than theEnvelope. Seerepository-adaptation.md§8.
A retried write happened twice
The idempotency key is the control, and there are exactly three ways it fails to apply.
- The client did not send one. The endpoint does not require the key, so it ran unguarded. Require it if a duplicate is unacceptable.
- The client sent a different body. Answered
422 IDEMPOTENT_REQUEST_MISMATCH, never a replay — replaying would hand back a receipt for a request the caller never made. - The record expired. TTL is per-operation with a 72h cap. A retry after expiry is a new request by definition.
A 409 IDEMPOTENT_IN_FLIGHT (retryable=true, Retry-After: 2) is not a failure: an earlier
attempt is still running and its result is unknown. The caller resends with the same
Idempotency-Key after Retry-After; a new key is a new request and is how a write ends up
applied twice. The four idempotency answers are in http-contract.md §2.
Rolling deploy stalls with instances half-drained
shutDownGracefully waits for in-flight work, and an open-but-idle keep-alive connection can count
as in-flight. Calling stop() while that wait is in progress hangs.
- Symptom: an instance neither serving nor exiting, no error in its log.
- Cause: an unbounded graceful wait.
GracefulShutdownProbebounds it at a stated grace period and stops the server regardless once it elapses; a deployment must do the same. - Setting:
spring.lifecycle.timeout-per-shutdown-phase. Without a bound the deploy waits for a connection that may never go idle.
Metrics stopped arriving, or the bill jumped
Almost always a high-cardinality tag — typically a routeTemplate carrying a resolved path, which
is one series per resource.
- Check: the tag values in the metrics backend for
routeTemplate. Braces mean templates; identifiers mean the resolved path leaked through. - Never tags: the URL, the query string, any identifier, the tenant, a key, a token, a cookie, a body. A tag value reaches the metrics backend unredacted and usually a third-party SaaS with it.
Cross-origin requests fail only in the browser
The API answers correctly and the browser refuses the response. CORS is CorsSettings
(app.cors.*) applied by SecurityConfig. Startup refuses two settings:
APP_SECURITY_CORS_ENABLED=truewith no origin.*inAPP_SECURITY_CORS_ORIGINSwithAPP_SECURITY_CORS_ALLOW_CREDENTIALS=true— no browser honours it.
Two more start fine and still fail in the browser:
https://*.example.comin the origin list — it is compared as a literal and matches nothing.- An origin with a path (
https://app.example.com/ui) — an origin is scheme, host and port only.
The response arrives but a header reads as null in browser code (Retry-After, X-Request-Id,
Location, a tus Upload-Offset): the header is not in Access-Control-Expose-Headers. The exposed
list is CorsExposedHeaders.PLATFORM plus each module's CorsExposedHeaders bean
(docs/web/http-contract.md §3); a header a service adds needs its own bean. A proxy that strips or
rewrites Access-Control-Expose-Headers has the same symptom.
A preflight answered 401 means CORS ran after authentication. Preflights carry no credentials by
design; the order is asserted by WebPipelineOrderContract.
Two access-log lines for one request
An async request passes through the servlet filter chain twice — once for the initial request and
again on the ASYNC redispatch. A completion recorded without checking isAsyncStarted() is written
both times, and every latency percentile computed from that data is wrong while looking plausible.
WebPipelineOrderContract asserts exactly one observation per logical request on every container.
Verification commands
cd src
./gradlew :adapter:inbound:web:test # unit, boundary, architecture
./gradlew :adapter:inbound:web:systemTest # Tomcat, real socket
./gradlew :adapter:inbound:web:webJettyCompatTest # second servlet container