# Web platform runbook What an operator needs when the HTTP boundary misbehaves. Organised by what you observe, because that is what you have at 3am — not by which module owns it. ## Reading a `429` `RateLimitInterceptor` answers `429 RATE_LIMIT_EXCEEDED` with `Retry-After` when the edge rate limit refuses a caller. `RATE_LIMIT_UNAVAILABLE` means the rate-limit store could not be reached, not that the caller exceeded anything — look at the store, not the caller. ## Errors that are not the `Envelope` Every HTTP error the application writes is the `Envelope` from `GlobalExceptionHandler` (`spring.mvc.problemdetails.enabled=false`), with an `error.code` a client branches on. - **How to confirm quickly:** send a request that must fail validation and check the body has `error.code`. A body without it (Spring's `ProblemDetail`, or the container's error page) means the handler is not installed or the response came from the edge. ## A response is truncated or the connection dies mid-document Two different causes, distinguished by the status the client did receive. - **Client got a status, then nothing** — the response failed after commit. The status cannot be retracted, so the connection ends mid-document. - **Client got nothing from the application** — the proxy refused it. `client_max_body_size` in nginx answers with its own HTML rather than the `Envelope`. See `repository-adaptation.md` §8. ## A retried write happened twice The idempotency key is the control, and there are exactly three ways it fails to apply. 1. **The client did not send one.** The endpoint does not require the key, so it ran unguarded. Require it if a duplicate is unacceptable. 2. **The client sent a different body.** Answered `422 IDEMPOTENT_REQUEST_MISMATCH`, never a replay — replaying would hand back a receipt for a request the caller never made. 3. **The record expired.** TTL is per-operation with a 72h cap. A retry after expiry is a new request by definition. A `409 IDEMPOTENT_IN_FLIGHT` is not a failure: an earlier attempt is still running and the caller should retry after `Retry-After`. ## Rolling deploy stalls with instances half-drained `shutDownGracefully` waits for in-flight work, and an open-but-idle keep-alive connection can count as in-flight. Calling `stop()` while that wait is in progress hangs. - **Symptom:** an instance neither serving nor exiting, no error in its log. - **Cause:** an unbounded graceful wait. `GracefulShutdownProbe` bounds it at a stated grace period and stops the server regardless once it elapses; a deployment must do the same. - **Setting:** `spring.lifecycle.timeout-per-shutdown-phase`. Without a bound the deploy waits for a connection that may never go idle. ## Metrics stopped arriving, or the bill jumped Almost always a high-cardinality tag — typically a `routeTemplate` carrying a *resolved* path, which is one series per resource. - **Check:** the tag values in the metrics backend for `routeTemplate`. Braces mean templates; identifiers mean the resolved path leaked through. - **Never tags:** the URL, the query string, any identifier, the tenant, a key, a token, a cookie, a body. A tag value reaches the metrics backend unredacted and usually a third-party SaaS with it. ## Cross-origin requests fail only in the browser The API answers correctly and the browser refuses the response. CORS is `CorsSettings` (`ca-skeleton.cors.*`) applied by `SecurityConfig`. Startup refuses two settings: - `APP_SECURITY_CORS_ENABLED=true` with no origin. - `*` in `APP_SECURITY_CORS_ORIGINS` with `APP_SECURITY_CORS_ALLOW_CREDENTIALS=true` — no browser honours it. Two more start fine and still fail in the browser: - `https://*.example.com` in the origin list — it is compared as a literal and matches nothing. - An origin with a path (`https://app.example.com/ui`) — an origin is scheme, host and port only. A preflight answered `401` means CORS ran after authentication. Preflights carry no credentials by design; the order is asserted by `WebPipelineOrderContract`. ## Two access-log lines for one request An async request passes through the servlet filter chain twice — once for the initial request and again on the ASYNC redispatch. A completion recorded without checking `isAsyncStarted()` is written both times, and every latency percentile computed from that data is wrong while looking plausible. `WebPipelineOrderContract` asserts exactly one observation per logical request on every container. ## Verification commands ```bash cd src ./gradlew :adapter:inbound:web:test # unit, boundary, architecture ./gradlew :adapter:inbound:web:systemTest # Tomcat, real socket ./gradlew :adapter:inbound:web:webJettyCompatTest # second servlet container ```