From 1196b86426bdde255c9d97f0c1ea90c41b033e1f Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Tue, 3 Nov 2026 00:37:16 +0900 Subject: [PATCH] =?UTF-8?q?fix(realtime):=20stream=20=EB=A7=88=EA=B0=90=20?= =?UTF-8?q?=EC=8B=A4=ED=8C=A8=EB=A5=BC=2013.1=EC=9D=98=20kind=EB=A1=9C=20?= =?UTF-8?q?=EB=B3=B4=EA=B3=A0=ED=95=9C=EB=8B=A4=20(04-02)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 문제: stream coordinator는 effect 마감 초과를 IDLE_TIMEOUT/APPLY로, recovery 마감 초과를 IDLE_TIMEOUT/RECOVER로, close drain 초과를 IDLE_TIMEOUT/CLOSE로 보고한다. 설계 13.1에서 IDLE_TIMEOUT은 전송 idle(heartbeat/byte deadline -> close + bounded reconnect)이다. reconnect 관측(RUN_ENDED failureKind)에는 operation이 없어서 운영에서 "서버가 조용해짐"과 "우리 effect가 멈춤"을 failureKind로 나눌 수 없다. 5b3f5b0은 13.1에 두 뜻을 적기만 했다(리뷰의 (b)). 고른 방법: 리뷰의 (a). kind만 바꾸고 retryable:false, DRAINING 유지, resume token 폐기는 그대로 둔다. - effect 마감: APPLY_FAILED/APPLY. 13.1의 "feature effect 실패"이고, WebSocket 연결도 자기 apply 마감을 APPLY_FAILED로 보고한다. - recovery 마감: PROVIDER_UNAVAILABLE/RECOVER. recovery source가 제때 답하지 않은 것이다. recovery가 throw한 경우와 같은 kind다. - close drain 마감: PROVIDER_UNAVAILABLE/CLOSE. reconnect coordinator가 drain 상한 초과를 PROVIDER_UNAVAILABLE로 보고하는 것과 맞춘다. - 관측 reason도 같은 kind로 바꾼다. - 13.1 표는 IDLE_TIMEOUT 행에서 stream coordinator 설명을 빼고, APPLY_FAILED·PROVIDER_UNAVAILABLE 행에 마감 경우를 적는다. VD-28의 같은 설명도 고친다. 이 템플릿으로 운영 중인 제품이 아직 없어서 관측 값을 바꾸는 비용이 지금 가장 작다. 버린 대안: (b) 문서만 고친다. 5b3f5b0이 한 것이고, 관측에서는 여전히 구분되지 않는다. close 마감에 CLOSED를 쓰는 것도 버렸다. 아직 DRAINING인데 "닫힘"으로 읽힌다. 위험·한계: - 공개로 관측되는 값이 바뀐다. stream coordinator 결과의 kind와 RealtimeObservation.reason에서 IDLE_TIMEOUT을 보던 조합 코드는 APPLY_FAILED/PROVIDER_UNAVAILABLE을 봐야 한다. repo 안에 그런 소비자는 없다. - live-poll handoff coordinator의 quiescence·checkpoint·close 마감은 여전히 IDLE_TIMEOUT이다. 결정 범위(stream coordinator)를 지켰다. 13.1 IDLE_TIMEOUT 행에 handoff 마감만 남겨 적었다. - realtime 레시피 번들 gzip이 40,897 B에서 40,902 B로 늘어 예산 40,900 B를 2 B 넘는다. "PROVIDER_UNAVAILABLE"이 "IDLE_TIMEOUT"보다 8자 길어 3곳에서 원본이 24 B 늘었다. 예산은 올리지 않았다. check-optional-recipes가 REFERENCE_RUNTIME_BUNDLE_BUDGET_EXCEEDED로 실패한다. 검증: - RED: 기대값을 먼저 바꾸고 stream-coordinator.test.ts 실행 -> 7 failed | 31 passed. 예: expected kind APPLY_FAILED, received IDLE_TIMEOUT. - GREEN: tests/unit/realtime 14 files, 230 passed. - tsc (tsconfig.app.json, tsconfig.test.json) 통과. 바꾼 두 파일 eslint 통과. verify-documentation-readiness PASS_SCOPED. - check-optional-recipes: realtime 40902 > 40900 실패(위험 참고). Refs: docs/reviews/2026-11-01-maturity-review/04-adapters-realtime.md 04-02 --- ...ime-events-web-push-and-bounded-polling.md | 10 ++-- ...ime-events-web-push-and-bounded-polling.md | 6 +-- src/adapters/realtime/stream-coordinator.ts | 21 +++++---- .../unit/realtime/stream-coordinator.test.ts | 47 +++++++++++++++---- 4 files changed, 60 insertions(+), 24 deletions(-) diff --git a/docs/architecture/decisions/VD-28-realtime-events-web-push-and-bounded-polling.md b/docs/architecture/decisions/VD-28-realtime-events-web-push-and-bounded-polling.md index 24f76268..117a1d83 100644 --- a/docs/architecture/decisions/VD-28-realtime-events-web-push-and-bounded-polling.md +++ b/docs/architecture/decisions/VD-28-realtime-events-web-push-and-bounded-polling.md @@ -18,14 +18,16 @@ (`UNKNOWN/CURRENT/STALE/RESYNCING`)와 별개다. - effect/recovery deadline에 도달하면 commit capability를 즉시 영구 무효화하고 - abort한다. caller에는 bounded `IDLE_TIMEOUT`(non-retryable, operation - `APPLY`/`RECOVER`)을 반환하되 **실제 task는 버리지 않고 retain**한다. + abort한다. caller에는 bounded non-retryable failure를 반환하되 **실제 task는 + 버리지 않고 retain**한다. effect 마감은 `APPLY_FAILED/APPLY`, recovery 마감은 + `PROVIDER_UNAVAILABLE/RECOVER`다(13.1). transport idle(`IDLE_TIMEOUT`)과 + 관측에서 구분하기 위해서다. - retain된 task가 하나라도 있으면 stream은 `DRAINING`이고 새 event/recovery admission을 거절한다. 실제 settlement가 일어나야 `STALE`로 돌아가 authoritative recovery를 요구하거나, close 요청이면 `CLOSED`가 된다. - `close()`는 `Promise>`다. 모든 retain task가 실제로 - settle해야 success이고, drain bound를 넘기면 `IDLE_TIMEOUT/CLOSE`를 반환하며 - stream은 계속 `DRAINING`이다. teardown success가 곧 quiescence다. + settle해야 success이고, drain bound를 넘기면 `PROVIDER_UNAVAILABLE/CLOSE`를 + 반환하며 stream은 계속 `DRAINING`이다. teardown success가 곧 quiescence다. - LIVE↔POLL overflow fail-close는 active/probe/quiescing/transition lease를 모두 abort한 뒤 **retired writer set**으로 옮기고 나서 reference를 지운다. `close()`는 current와 retired를 dedupe해 함께 기다리므로, 버려진 diff --git a/docs/architecture/realtime-events-web-push-and-bounded-polling.md b/docs/architecture/realtime-events-web-push-and-bounded-polling.md index 842c6db5..dd6ed831 100644 --- a/docs/architecture/realtime-events-web-push-and-bounded-polling.md +++ b/docs/architecture/realtime-events-web-push-and-bounded-polling.md @@ -1895,11 +1895,11 @@ credential을 보호할 수 있다고 주장하지 않는다. | `UNSUPPORTED` | browser capability 없음 | declared fallback | | `OFFLINE` | network hint/attempt failure | online hint 뒤 bounded check | | `CONNECT_TIMEOUT` | open deadline 초과 | bounded reconnect | -| `IDLE_TIMEOUT` | transport: heartbeat/byte/unsubscribe deadline 초과(operation `RECEIVE`·`SUBSCRIBE`). 내부 task: stream·handoff coordinator의 effect·recovery·quiescence·close deadline 초과(operation `APPLY`·`RECOVER`·`CLOSE`) | transport: close + bounded reconnect. 내부 task는 재연결로 풀리지 않는다: stream coordinator는 `retryable: false`로 보고하고 task가 settle할 때까지 `DRAINING`을 유지하며, handoff는 그 전환을 fail-close한다(결과는 kind 기본값대로 retryable). 두 뜻은 kind가 아니라 `operation`으로 구분한다. reconnect 관측(`RUN_ENDED failureKind`)에는 `operation`이 없어 구분되지 않는다 | +| `IDLE_TIMEOUT` | transport: heartbeat/byte/unsubscribe deadline 초과(operation `RECEIVE`·`SUBSCRIBE`). handoff coordinator의 quiescence·checkpoint·close deadline 초과(operation `RECOVER`·`CLOSE`) | transport: close + bounded reconnect. handoff 마감은 재연결로 풀리지 않는다: 그 전환을 fail-close한다(결과는 kind 기본값대로 retryable). 두 뜻은 `operation`으로 구분한다 | | `AUTH_REQUIRED` | session expired/recovery 실패 | sign-in UX | | `FORBIDDEN` | stream/resource 거절 | retry 없음 | | `RATE_LIMITED` | connection/request/provider limit | bounded server hint | -| `PROVIDER_UNAVAILABLE` | backend/push/hosting unavailable | bounded retry 또는 degraded | +| `PROVIDER_UNAVAILABLE` | backend/push/hosting unavailable. stream coordinator의 recovery deadline 초과(operation `RECOVER`)와 close drain deadline 초과(operation `CLOSE`)도 이 kind다(`retryable: false`) | bounded retry 또는 degraded. stream coordinator 마감은 재연결로 풀리지 않는다: task가 settle할 때까지 `DRAINING`을 유지한다. settle 뒤 recovery 마감은 다음 event가 recovery부터 하고, close 마감은 `CLOSED`가 된다 | | `PROTOCOL_MISMATCH` | version/subprotocol/content type | terminal/rollout rollback | | `MALFORMED_EVENT` | state-bearing syntax/schema/exact-key 실패 | cursor 미진행, freshness `UNKNOWN`, close + registered recovery | | `MAPPING_CONTRACT_VIOLATION` | validated DTO의 registered mapper 실패 | effect/cache/cursor write 0, close + registered recovery | @@ -1910,7 +1910,7 @@ credential을 보호할 수 있다고 주장하지 않는다. | `SEQUENCE_GAP` | missing event | snapshot resync | | `CURSOR_EXPIRED` | replay retention 밖 | snapshot reset | | `QUEUE_OVERFLOW` | backpressure ceiling | close + registered snapshot/session rebuild | -| `APPLY_FAILED` | feature effect 실패 | cursor 미commit, bounded registered recovery | +| `APPLY_FAILED` | feature effect 실패. stream coordinator의 effect deadline 초과(operation `APPLY`, `retryable: false`)도 이 kind다 | cursor 미commit, bounded registered recovery. deadline 초과는 recovery를 바로 하지 않는다: task가 settle할 때까지 `DRAINING`을 유지하고, settle 뒤 다음 event가 recovery부터 한다 | | `POLL_BUDGET_EXHAUSTED` | lease 한도 도달 | stale/manual retry | | `PUSH_PERMISSION_DENIED` | 사용자 거절 | foreground fallback | | `PUSH_SUBSCRIPTION_STALE` | rotation/expiry/provider gone | bounded re-registration | diff --git a/src/adapters/realtime/stream-coordinator.ts b/src/adapters/realtime/stream-coordinator.ts index 3653b734..1d23053a 100644 --- a/src/adapters/realtime/stream-coordinator.ts +++ b/src/adapters/realtime/stream-coordinator.ts @@ -102,8 +102,9 @@ export type RealtimeStreamCoordinator = Readonly<{ lifecycle(streamId: StreamRegistrationId): RealtimeStreamLifecycle; /** * R-02. Bounded quiescence. Success means every tracked task actually - * settled; `IDLE_TIMEOUT` means the coordinator is still `DRAINING` and the - * caller must not assume a clean teardown. + * settled; `PROVIDER_UNAVAILABLE` with operation `CLOSE` means the + * coordinator is still `DRAINING` and the caller must not assume a clean + * teardown. */ close(): Promise>; }>; @@ -556,14 +557,16 @@ export function createRealtimeStreamCoordinator( } if (effectTimedOut) { // R-02. Bounded for the caller; the underlying task stays tracked. + // Design 13.1: a stalled effect is APPLY_FAILED, never the transport's + // IDLE_TIMEOUT, so observation can tell it from a silent server. observe({ operation: "APPLY", outcome: "FAILED", streamId: state.registration.id, eventType: event.envelope.eventType, - reason: "IDLE_TIMEOUT", + reason: "APPLY_FAILED", }); - return realtimeFailure("IDLE_TIMEOUT", "APPLY", false); + return realtimeFailure("APPLY_FAILED", "APPLY", false); } if (closed || state.closed) { @@ -748,14 +751,15 @@ export function createRealtimeStreamCoordinator( } if (recoveryTimedOut) { // R-02. A late checkpoint from this attempt can never commit. + // Design 13.1: the recovery source did not answer in time. state.freshness = "UNKNOWN"; observe({ operation: "RECOVER", outcome: "FAILED", streamId: state.registration.id, - reason: "IDLE_TIMEOUT", + reason: "PROVIDER_UNAVAILABLE", }); - return realtimeFailure("IDLE_TIMEOUT", "RECOVER", false); + return realtimeFailure("PROVIDER_UNAVAILABLE", "RECOVER", false); } if (closed || state.closed) { state.freshness = "UNKNOWN"; @@ -1027,8 +1031,9 @@ export function createRealtimeStreamCoordinator( deadline(limits.drainTimeoutMs), ); if (drained.kind !== "VALUE") { - // Still DRAINING: the caller must not treat this as quiescence. - return realtimeFailure("IDLE_TIMEOUT", "CLOSE", false); + // Still DRAINING: the caller must not treat this as quiescence. Same + // kind as the reconnect coordinator's drain ceiling. + return realtimeFailure("PROVIDER_UNAVAILABLE", "CLOSE", false); } for (const state of states.values()) state.lifecycle = "CLOSED"; return realtimeSuccess(undefined); diff --git a/tests/unit/realtime/stream-coordinator.test.ts b/tests/unit/realtime/stream-coordinator.test.ts index 671a8f96..6bd1012d 100644 --- a/tests/unit/realtime/stream-coordinator.test.ts +++ b/tests/unit/realtime/stream-coordinator.test.ts @@ -31,9 +31,11 @@ import { describe("transport-independent realtime stream coordinator", () => { it("keeps the stream DRAINING until a non-cooperative effect settles", async () => { const wedged = deferred>(); + const observations: RealtimeObservation[] = []; const harness = createHarness({ apply: async () => wedged.promise, taskLimits: { effectTimeoutMs: 5, drainTimeoutMs: 5 }, + observe: (observation) => observations.push(observation), }); await harness.initialize(); @@ -44,11 +46,19 @@ describe("transport-independent realtime stream coordinator", () => { resumeCursor: "cursor-00000001", }), ); - // Bounded for the caller, and explicitly non-retryable. + // Bounded for the caller, and explicitly non-retryable. A stalled effect + // is an apply failure (design 13.1), not a transport IDLE_TIMEOUT. expect(applied).toMatchObject({ ok: false, - error: { kind: "IDLE_TIMEOUT", operation: "APPLY", retryable: false }, + error: { kind: "APPLY_FAILED", operation: "APPLY", retryable: false }, }); + expect(observations).toContainEqual( + expect.objectContaining({ + operation: "APPLY", + outcome: "FAILED", + reason: "APPLY_FAILED", + }), + ); expect(harness.coordinator.lifecycle(STREAM_ID)).toBe("DRAINING"); // DRAINING refuses new admission rather than queueing behind the wedge. @@ -65,7 +75,11 @@ describe("transport-independent realtime stream coordinator", () => { // close() cannot claim quiescence while the task is still running. await expect(harness.coordinator.close()).resolves.toMatchObject({ ok: false, - error: { kind: "IDLE_TIMEOUT", operation: "CLOSE" }, + error: { + kind: "PROVIDER_UNAVAILABLE", + operation: "CLOSE", + retryable: false, + }, }); // Only actual settlement ends DRAINING. @@ -79,6 +93,7 @@ describe("transport-independent realtime stream coordinator", () => { it("bounds non-cooperative recovery and rejects its late checkpoint", async () => { const wedged = deferred>(); let recoveries = 0; + const observations: RealtimeObservation[] = []; const harness = createHarness({ recover: async () => { recoveries += 1; @@ -87,6 +102,7 @@ describe("transport-independent realtime stream coordinator", () => { : wedged.promise; }, taskLimits: { recoveryTimeoutMs: 5, drainTimeoutMs: 5 }, + observe: (observation) => observations.push(observation), }); await harness.initialize(); @@ -94,10 +110,23 @@ describe("transport-independent realtime stream coordinator", () => { STREAM_ID, "SEQUENCE_GAP", ); + // A recovery source that does not answer in time is unavailable + // (design 13.1), not a transport IDLE_TIMEOUT. expect(recovered).toMatchObject({ ok: false, - error: { kind: "IDLE_TIMEOUT", operation: "RECOVER", retryable: false }, + error: { + kind: "PROVIDER_UNAVAILABLE", + operation: "RECOVER", + retryable: false, + }, }); + expect(observations).toContainEqual( + expect.objectContaining({ + operation: "RECOVER", + outcome: "FAILED", + reason: "PROVIDER_UNAVAILABLE", + }), + ); expect(harness.coordinator.lifecycle(STREAM_ID)).toBe("DRAINING"); const beforeLateCommit = harness.coordinator.getResumeState(STREAM_ID); @@ -1000,7 +1029,7 @@ describe("realtime physical task ownership", () => { // close() arrives long before the effect deadline. const closed = await harness.coordinator.close(); expect(closed.ok).toBe(false); - expect(closed.ok ? null : closed.error.kind).toBe("IDLE_TIMEOUT"); + expect(closed.ok ? null : closed.error.kind).toBe("PROVIDER_UNAVAILABLE"); expect(harness.coordinator.lifecycle(STREAM_ID)).toBe("DRAINING"); applyGate.resolve(realtimeSuccess(undefined)); @@ -1121,7 +1150,7 @@ describe("realtime physical task ownership", () => { await expect(applying).resolves.toMatchObject({ ok: false, - error: { kind: "IDLE_TIMEOUT", operation: "APPLY" }, + error: { kind: "APPLY_FAILED", operation: "APPLY" }, }); await expect(recovering).resolves.toMatchObject({ ok: false, @@ -1300,7 +1329,7 @@ describe("RT-01 reentrant close cannot pass the registration window", () => { expect(closeResult?.ok).toBe(false); expect(closeResult?.ok ? null : closeResult?.error.kind).toBe( - "IDLE_TIMEOUT", + "PROVIDER_UNAVAILABLE", ); applyGate.resolve(realtimeSuccess(undefined)); @@ -1336,7 +1365,7 @@ describe("RT-01 reentrant close cannot pass the registration window", () => { expect(closeResult?.ok).toBe(false); expect(closeResult?.ok ? null : closeResult?.error.kind).toBe( - "IDLE_TIMEOUT", + "PROVIDER_UNAVAILABLE", ); recoveryGate.resolve(realtimeSuccess(snapshotCommit("0"))); @@ -1414,7 +1443,7 @@ describe("RT-02 a throwing scheduler stays inside the result contract", () => { // A drain that could not be bounded is not proof of quiescence. expect(closed.ok).toBe(false); - expect(closed.ok ? null : closed.error.kind).toBe("IDLE_TIMEOUT"); + expect(closed.ok ? null : closed.error.kind).toBe("PROVIDER_UNAVAILABLE"); applyGate.resolve(realtimeSuccess(undefined)); await applying;