문제: 2026-11-01 성숙도 리뷰에서 High 21건·P1 25건이 나왔다. 결과를 모르는 일을 "확실함"으로 기록했다(발송 중 죽은 알림 재발송, 커밋 중 끊긴 쓰기를 재시도 가능으로 응답, 재시도 가능한 거부를 DEAD로). 중복을 DB가 아니라 코드 순서로 막았다(정산 job, 포스터 제목). 짝 프론트와 wire 계약이 달랐고(처리 중 409, Envelope null), JWT audience 검증이 기본으로 꺼져 있었고, 트레이스가 나가지 않았고, CI가 strict lane 14개를 한 번도 돌리지 않았다. 방법: 발견마다 실패하는 테스트를 먼저 쓰고 고쳤다. 주요 선택: - 알림 attempt는 처음부터 "시작 여부 모름"으로 쓴다(호출 직전 갱신은 쓰기가 하나 늘어 버림). - 정산 job claim은 lease·fence를 갱신하는 UPDATE 한 문장(처리 전체를 트랜잭션으로 감싸는 안은 외부 호출 동안 잠금을 잡아 버림). - 커밋 단계 연결 끊김은 DB_COMPLETION_UNKNOWN(재시도 불가). - 처리 중 409는 retryable=true + Retry-After: 2, Envelope null 필드는 web Jackson mixin으로 생략(shared-contract는 stdlib only). - 트레이싱은 자체 Tracer를 지우고 Boot 4 tracing·OTLP로. - strict lane을 단계에 연결하고 verifyStrictLaneStages가 고아를 막는다. 발견별 RED·GREEN·버린 대안은 리뷰 README 13절 표에 있다. 위험·한계: JWT 모드는 APP_SECURITY_JWT_AUDIENCE가 없으면 기동하지 않는다. 짝 프론트는 409 retryable과 Envelope 키 규칙을 맞춰야 한다. 이 커밋 시점에는 MinIO 공식 이미지가 사라져 MinIO 2 lane이 실패한다 (다음 커밋에서 대체 이미지로 해결). 검증: ci 통과. mainCheck·integrationCheck는 MinIO 2 lane만 실패(이미지 pull 거부). -p optional-platforms ci integrationCheck, performanceCheck, releaseCheck(-PgitRevision) 통과. Refs: docs/reviews/2026-11-01-maturity-review/README.md 2·13절 BREAKING CHANGE: JWT 모드에서 APP_SECURITY_JWT_AUDIENCE가 필수가 되고 role은 audience client와 realm 것만 읽는다. IDEMPOTENT_IN_FLIGHT가 retryable=true, Retry-After: 2로 바뀌고 Envelope는 null 필드를 보내지 않는다.
99 lines
4.9 KiB
Markdown
99 lines
4.9 KiB
Markdown
# Web platform runbook
|
|
|
|
What an operator needs when the HTTP boundary misbehaves. Organised by what you observe, because
|
|
that is what you have at 3am — not by which module owns it.
|
|
|
|
## Reading a `429`
|
|
|
|
`RateLimitInterceptor` answers `429 RATE_LIMIT_EXCEEDED` with `Retry-After` when the edge rate
|
|
limit refuses a caller. `RATE_LIMIT_UNAVAILABLE` means the rate-limit store could not be reached,
|
|
not that the caller exceeded anything — look at the store, not the caller.
|
|
|
|
## Errors that are not the `Envelope`
|
|
|
|
Every HTTP error the application writes is the `Envelope` from `GlobalExceptionHandler`
|
|
(`spring.mvc.problemdetails.enabled=false`), with an `error.code` a client branches on.
|
|
|
|
- **How to confirm quickly:** send a request that must fail validation and check the body has
|
|
`error.code`. A body without it (Spring's `ProblemDetail`, or the container's error page) means
|
|
the handler is not installed or the response came from the edge.
|
|
|
|
## A response is truncated or the connection dies mid-document
|
|
|
|
Two different causes, distinguished by the status the client did receive.
|
|
|
|
- **Client got a status, then nothing** — the response failed after commit. The status cannot be
|
|
retracted, so the connection ends mid-document.
|
|
- **Client got nothing from the application** — the proxy refused it. `client_max_body_size` in
|
|
nginx answers with its own HTML rather than the `Envelope`. See `repository-adaptation.md` §8.
|
|
|
|
## A retried write happened twice
|
|
|
|
The idempotency key is the control, and there are exactly three ways it fails to apply.
|
|
|
|
1. **The client did not send one.** The endpoint does not require the key, so it ran unguarded.
|
|
Require it if a duplicate is unacceptable.
|
|
2. **The client sent a different body.** Answered `422 IDEMPOTENT_REQUEST_MISMATCH`, never a
|
|
replay — replaying would hand back a receipt for a request the caller never made.
|
|
3. **The record expired.** TTL is per-operation with a 72h cap. A retry after expiry is a new
|
|
request by definition.
|
|
|
|
A `409 IDEMPOTENT_IN_FLIGHT` (retryable=true, `Retry-After: 2`) is not a failure: an earlier
|
|
attempt is still running and its result is unknown. The caller resends with the **same**
|
|
`Idempotency-Key` after `Retry-After`; a new key is a new request and is how a write ends up
|
|
applied twice. The four idempotency answers are in `http-contract.md` §2.
|
|
|
|
## Rolling deploy stalls with instances half-drained
|
|
|
|
`shutDownGracefully` waits for in-flight work, and an open-but-idle keep-alive connection can count
|
|
as in-flight. Calling `stop()` while that wait is in progress hangs.
|
|
|
|
- **Symptom:** an instance neither serving nor exiting, no error in its log.
|
|
- **Cause:** an unbounded graceful wait. `GracefulShutdownProbe` bounds it at a stated grace period
|
|
and stops the server regardless once it elapses; a deployment must do the same.
|
|
- **Setting:** `spring.lifecycle.timeout-per-shutdown-phase`. Without a bound the deploy waits for
|
|
a connection that may never go idle.
|
|
|
|
## Metrics stopped arriving, or the bill jumped
|
|
|
|
Almost always a high-cardinality tag — typically a `routeTemplate` carrying a *resolved* path, which
|
|
is one series per resource.
|
|
|
|
- **Check:** the tag values in the metrics backend for `routeTemplate`. Braces mean templates;
|
|
identifiers mean the resolved path leaked through.
|
|
- **Never tags:** the URL, the query string, any identifier, the tenant, a key, a token, a cookie,
|
|
a body. A tag value reaches the metrics backend unredacted and usually a third-party SaaS with it.
|
|
|
|
## Cross-origin requests fail only in the browser
|
|
|
|
The API answers correctly and the browser refuses the response. CORS is `CorsSettings`
|
|
(`app.cors.*`) applied by `SecurityConfig`. Startup refuses two settings:
|
|
|
|
- `APP_SECURITY_CORS_ENABLED=true` with no origin.
|
|
- `*` in `APP_SECURITY_CORS_ORIGINS` with `APP_SECURITY_CORS_ALLOW_CREDENTIALS=true` — no browser
|
|
honours it.
|
|
|
|
Two more start fine and still fail in the browser:
|
|
|
|
- `https://*.example.com` in the origin list — it is compared as a literal and matches nothing.
|
|
- An origin with a path (`https://app.example.com/ui`) — an origin is scheme, host and port only.
|
|
|
|
A preflight answered `401` means CORS ran after authentication. Preflights carry no credentials by
|
|
design; the order is asserted by `WebPipelineOrderContract`.
|
|
|
|
## Two access-log lines for one request
|
|
|
|
An async request passes through the servlet filter chain twice — once for the initial request and
|
|
again on the ASYNC redispatch. A completion recorded without checking `isAsyncStarted()` is written
|
|
both times, and every latency percentile computed from that data is wrong while looking plausible.
|
|
`WebPipelineOrderContract` asserts exactly one observation per logical request on every container.
|
|
|
|
## Verification commands
|
|
|
|
```bash
|
|
cd src
|
|
./gradlew :adapter:inbound:web:test # unit, boundary, architecture
|
|
./gradlew :adapter:inbound:web:systemTest # Tomcat, real socket
|
|
./gradlew :adapter:inbound:web:webJettyCompatTest # second servlet container
|
|
```
|