refactor: 부가 기능을 capability 모듈로 포장하고 기본 조립을 web-only로 전환

설계: docs/superpowers/specs/2026-09-25-capability-module-packaging-design.md
계획/진행 기록: docs/superpowers/plans/2026-09-25-capability-module-packaging.md

- Wave 0: 규칙 고정(AGENTS.md capability 완료 기준·테스트 규칙) + 차단 결함 7건
  (compose overlay 검사, redis-sdk CI job, 적용된 V9/V13 원복 + V16, @EnableScheduling,
  messaging bridge 순서, prod 에러 상세 validator, sample loggers read-only)
- Wave 1: 데이터·보안 Critical 10건 (durable operation 25P02, operation 유스케이스 이동,
  outbox 재인코딩, Kafka TLS/SASL, 알림 트랜잭션, SNS 서명, tus checksum, GraphQL fragment,
  Redis lane, gRPC FAILED_TERMINAL)
- Wave 2: 리팩터링 잔재·실험용 자리 삭제, outbox v1 fencing(V17), 멱등성 V2 정본화
- Wave 3: 라이브러리형 코드 정리(동작하지 않는 설정·인터페이스·gate 제거, web-advanced 분리)
- Wave 4: capability 11개를 "있으면 켜짐" 모듈로 포장, 끄는 장치 제거, 기본 조립 web-only,
  qualificationBootJar 분리
- Wave 5: 빌드 로직 축소(17.2k -> 10.8k줄), sample 테스트 컨벤션 적용

삭제한 코드는 docs/capability-status.md "제거된 코드" 표에 복구 커밋(208077fd)과 함께 기록.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
DongHyeonka
2026-09-25 12:49:07 +09:00
co-authored by Claude Opus 5.5
parent 208077fddf
commit 6d27356e2b
2822 changed files with 27992 additions and 166879 deletions
+37 -66
View File
@@ -1,27 +1,26 @@
# WebSocket platform runbook
# WebSocket handler runbook
> **Scope:** the default shipped compositions do not assemble the WebSocket platform runtime.
> This runbook applies only to a deployment that deliberately adopted the websocket leaf and
> explicitly wired `PlatformWebSocketHandler` and its lifecycle into that composition. If that bean
> is absent, the platform sections below describe library behavior, not a running endpoint.
> **Scope:** the shipped compositions do not assemble a WebSocket runtime. This runbook applies to
> a service that registered `PlatformWebSocketHandler` itself (see
> `src/adapter/inbound/websocket/README.md`, "라이브러리 API"). The STOMP channel's rules are in the
> leaf `CLAUDE.md`.
Organised by what you observe, because that is what you have during an incident.
## Clients report 1006 — "closed abnormally, no reason"
The most important symptom here, because 1006 is indistinguishable from a network failure and
drives every client into its most aggressive reconnect path.
1006 is indistinguishable from a network failure and drives every client into its most aggressive
reconnect path.
- **During a deploy** — the platform did not close its connections. `PlatformWebSocketHandler`
implements `SmartLifecycle` and closes with **1001 going away**; a container's own graceful
shutdown does **not** do this, because it waits for in-flight *requests* and an established
WebSocket is not a request. Check the handler bean is registered and that its phase runs before
the web server stops.
- **During a deploy** — the handler did not close its connections. It closes them with
**1001 going away** from `SmartLifecycle.stop()`; a container's own graceful shutdown does not,
because it waits for in-flight *requests* and an established WebSocket is not a request. Check
the handler is a Spring bean (otherwise Spring never calls `stop()`). `TomcatWebSocketShutdownIT`
shows the order: with the handler's phase below the web server's, the client sees 1006.
- **Behind a proxy** — the upgrade is mishandled, so the close frame never becomes a close frame.
See the next section.
- **Neither** — something threw while closing. A close reason longer than 123 bytes makes the
container throw mid-close and turns a clean refusal into a 1006; the platform truncates for that
reason.
container throw mid-close; the handler truncates for that reason.
## Connections work locally and not behind the load balancer
@@ -33,80 +32,52 @@ Three nginx directives, each producing a different failure:
| `Upgrade` / `Connection` forwarded | Upstream sees an ordinary GET and answers 404 or 200 for a route that works when tested directly. They are hop-by-hop headers, so a proxy is *required* to drop them. |
| `proxy_read_timeout` raised | Every connection quieter than 60s is killed with no close frame. A WebSocket is idle by nature. |
`websocketNginxTest` asserts all three against a real Nginx, and carries a deliberately broken
configuration so it proves its own assertions can fail.
`websocketNginxTest` asserts these against a real Nginx, and carries a deliberately broken
configuration (`nginx-no-upgrade.conf`) so it proves its own assertions can fail.
## Connections vanish with no close frame and no error
Almost always an intermediary's idle timeout, not the application. TCP does not report a departed
peer — a closed laptop lid, a phone switching to cellular and a NAT forgetting its mapping all
produce no FIN.
produce no FIN. The handler has no heartbeat or idle timer; set the container idle timeout
(`ServletServerContainerFactoryBean.setMaxSessionIdleTimeout`) and the proxy read timeout, and have
clients ping below the smallest of them.
- Check `HeartbeatPolicy`: the ping interval must be under ~20s, because intermediaries commonly
drop idle connections at 30–60s and say nothing.
- The idle timeout must be at least two ping intervals. Below that, one dropped ping on a congested
network closes a healthy connection, and the reconnect storm makes the congestion worse.
## Clients are closed with 1009
The message exceeded a bound. Two places enforce one: the container's text buffer (Tomcat default
8 KiB) closes before the handler sees the message, and the handler closes when the message exceeds
`WebSocketConnectionBudget.maxMessageBytes` (UTF-8 bytes). Set the container buffer to the same
value so the answer does not depend on which one fired first.
## Clients are closed with 1013
A reply could not be queued: the connection's outbound buffer (`maxBufferedOutboundBytes`) or the
node-wide `GlobalBufferBudget` is full, which means the peer reads more slowly than the service
answers. Closing is deliberate — a client that reconnects has lost nothing, while one that silently
missed a reply is wrong and does not know it.
## Memory grows and nothing is failing
A slow consumer. It does not error — it reads more slowly than the server writes and the difference
accumulates. Arranging it requires no tooling: reading slowly is enough.
- **Check first:** `WebSocketNodeSnapshot.droppedOutboundMessages`. Connection counts and buffered
bytes both look healthy while data is being lost; the drop count is the only number that says so.
- **Bounds:** per connection (`maxBufferedOutboundBytes`) *and* node-wide (`GlobalBufferBudget`).
The second is not redundant — a megabyte each is fine at a hundred connections and is the whole
heap at fifty thousand.
- **Drops:** `OutboundQueue.snapshot().droppedCount()` counts droppable and conflatable messages
the queue shed. Connection counts and buffered bytes look healthy while data is being shed; the
drop count is the only number that says so.
- **If drops are zero and memory still grows:** check that the queue lock and the socket lock are
separate. When they were one lock, a stalled peer blocked every producer, the queue never filled,
and the shedding bound never fired. That is fixed, and it is the shape to look for if it recurs.
## A command ran twice
The reconnect is what makes this a WebSocket problem: a client replays everything it never saw an
answer for, on a new connection, all at once.
1. **Key scoped to the connection?** It must not be. `WebSocketCommandKey` is endpoint + actor +
the client's own message id, so it survives the reconnect.
2. **Ledger not transactional?** The only implementation that works writes in the application's own
transaction. Anything else has a window between the business commit and the ledger write.
3. **Outcome `UNKNOWN`?** That is the window. Neither replaying nor re-running is safe — replaying
invents a result, re-running duplicates a committed write. Only `CommandReconciliation` reading
the business data can settle it, and `INDETERMINATE` is a real answer to act on.
## A client cannot connect and the endpoint is definitely there
- **403** — origin. The same-origin policy does **not** protect a WebSocket handshake and there is
no preflight, so the server's `Origin` check is the entire defence and it is strict: exact match,
lowercase, no path, no wildcard.
- **401** — the ticket was spent, expired (30s cap), or minted for another endpoint. All three are
deliberate; a longer-lived or unbound ticket is a bearer token in a query string again.
- **400** — no offered subprotocol is supported. Production refuses a client that offers none.
- **503** — the endpoint is at its connection cap.
Order matters and is asserted: route → origin → capacity → credential → subprotocol. Origin is
checked before authentication because a cookie handshake from an attacker's page authenticates
perfectly; the credential is valid and the page is not allowed to use it.
## Endpoints never answer and the app reports healthy
Both runtimes on one classpath. Spring Boot deduces one application type and picks the servlet one,
so declared reactive endpoints are simply never served — no error, no warning.
`WebSocketStackExclusivity` fails startup on this; if a running instance has it, the check is not
wired.
## Metrics stopped arriving
A high-cardinality tag, and the risk is worse than for HTTP: a request produces one observation,
a connection produces them for hours, so the series outlive the connections and only accumulate.
`WebSocketMetricTags` allows eight names; `connectionId`, `sessionId` and `actor` are not among
them. `nodeId` is, because a fleet has a knowable number of nodes.
and the shedding bound never fired.
## Verification
```bash
cd src
./gradlew :adapter:inbound:websocket:test # unit, boundary, architecture, Tomcat runtime
./gradlew :adapter:inbound:websocket:test # queue, writer, budget, handler
./gradlew :adapter:inbound:websocket:systemTest # real Tomcat, incl. context close → 1001
./gradlew :adapter:inbound:websocket:websocketJettyTest # second servlet container
./gradlew :adapter:inbound:websocket:websocketNginxTest # real Nginx; needs Docker, fails without it
```