Web platform runbook

What an operator needs when the HTTP boundary misbehaves. Organised by what you observe, because that is what you have at 3am — not by which module owns it.

Reading the two "too much traffic" answers

The platform never answers these two interchangeably, and the difference tells you where to look.

You see	It means	Where to look
`429 RATE_LIMITED` climbing for one caller	that caller exceeded its quota	the caller. Nothing is wrong with the service.
`429` climbing across all callers	quota is being charged for requests the service then shed	look at 503 first; see below
`503 ADMISSION_REJECTED`	the service has no capacity	saturation: check the admission profile's concurrency and what is holding slots

The second row is a real consequence of the design and worth knowing before it confuses you. Quota is charged before capacity is requested, deliberately — the reverse order lets a caller who is about to be rate-limited occupy a slot on the way to being told so. Under sustained overload every caller's quota therefore drains on requests that never ran. Rising 429 during an incident is a symptom of shedding, not of callers misbehaving. Read the 503 rate first.

A control that appears configured and does nothing

This has happened here, and the failure is silent by construction. A configured-but-unwired control behaves exactly like a working one until it is needed.

Check first: the platform snapshot's uninstalledControls(). WebPlatformStartupValidator fails startup on a missing required control, so a running instance with one missing means it was not in the required list.
The instance found the hard way: the problem catalog was complete, fully unit-tested, and reached by nothing on the framework's error path. Spring answered failures with its own ProblemDetail — RFC 9457-shaped, so it looked correct — carrying no code field. It was found by a cross-stack parity recording, not by any test of the catalog.
How to confirm quickly: send a request that must fail validation and check the body has a code. No code means the platform's handler is not installed.

A response is truncated or the connection dies mid-document

Two different causes, distinguished by the status the client did receive.

Client got a status, then nothing — the response budget was crossed after commit. The platform cannot retract a status, so it ends the connection: a truncated response the client rejects is better than a short one it accepts as complete. Raise maxResponseBytes for that operation's profile, or make the endpoint paginate.
Client got nothing at all — either the request budget was crossed before headers, in which case there is a problem document, or the proxy refused it. client_max_body_size in nginx is below the application's body budget by default, and nginx answers with its own HTML rather than a problem document. See repository-adaptation.md §8 for which bound fires where.

A retried write happened twice

The idempotency key is the control, and there are exactly three ways it fails to apply.

The client did not send one. The operation profile says OPTIONAL, so it ran unguarded. Change the profile to REQUIRED if a duplicate is unacceptable.
The client sent a different body. Answered 422 IDEMPOTENCY_KEY_REUSED, never a replay — replaying would hand back a receipt for a request the caller never made.
The record expired. TTL is per-operation with a 72h cap. A retry after expiry is a new request by definition.

A 409 IDEMPOTENCY_REQUEST_IN_PROGRESS is not a failure: an earlier attempt is still running and the caller should retry after Retry-After.

Rolling deploy stalls with instances half-drained

shutDownGracefully waits for in-flight work, and on Reactor Netty an open-but-idle keep-alive connection counts as in-flight. Calling stop() while that wait is in progress hangs.

Symptom: an instance neither serving nor exiting, no error in its log.
Cause: an unbounded graceful wait. GracefulShutdownProbe bounds it at a stated grace period and stops the server regardless once it elapses; a deployment must do the same.
Setting: spring.lifecycle.timeout-per-shutdown-phase. Without a bound the deploy waits for a connection that may never go idle.

Metrics stopped arriving, or the bill jumped

Almost always a high-cardinality tag. WebMetricCardinalityPolicy refuses anything off an eight-name allowlist at the point of recording, so a new tag cannot appear by accident — but a routeTemplate carrying a resolved path can, and that is one series per resource.

Check: the tag values in the metrics backend for routeTemplate. Braces mean templates; identifiers mean the resolved path leaked through.
Never tags: the URL, the query string, any identifier, the tenant, a key, a token, a cookie, a body. A tag value reaches the metrics backend unredacted and usually a third-party SaaS with it.

Cross-origin requests fail only in the browser

The API answers correctly and the browser refuses the response. Everything below is refused at startup by WebCorsPolicyValidator, so a running instance with one of these means the profile was built somewhere that does not validate.

* with credentials — no browser honours it.
An origin with a path, a trailing slash, or uppercase — never matches what the browser sends.
https://*.example.com — matches nothing; CORS compares origins exactly.
Wildcard allowedHeaders on a credentialed profile — not honoured with credentials.

A preflight answered 401 means CORS ran after authentication. Preflights carry no credentials by design; the order is asserted by WebPipelineOrderContract.

Two access-log lines for one request

An async request passes through the servlet filter chain twice — once for the initial request and again on the ASYNC redispatch. A completion recorded without checking isAsyncStarted() is written both times, and every latency percentile computed from that data is wrong while looking plausible. WebPipelineOrderContract asserts exactly one observation per logical request on every container.

Verification commands

cd src
./gradlew :adapter:inbound:web:test                          # unit, boundary, architecture
./gradlew :adapter:inbound:web:webCrossStackParityTest       # Tomcat vs Jetty vs Reactor Netty
./gradlew :adapter:inbound:web:webNginxProxyTest             # real proxy; needs Docker, fails without it
./gradlew :adapter:inbound:web:webJettyCompatTest            # second servlet container
./gradlew :adapter:inbound:web:webFluxContractTest           # reactive stack

6.7 KiB Raw Blame History