diff --git a/docs/architecture/presigned-transfer-and-image-cdn.md b/docs/architecture/presigned-transfer-and-image-cdn.md index 461f53da..6dc811ba 100644 --- a/docs/architecture/presigned-transfer-and-image-cdn.md +++ b/docs/architecture/presigned-transfer-and-image-cdn.md @@ -327,6 +327,42 @@ signed URL이나 receipt를 포함하거나 저장하지 않으며 server state reconcile에서 server 상태를 확인한다. backend는 만료된 orphan multipart를 정리하는 TTL job을 가져야 한다. +complete 결과를 모를 때 클라이언트는 다음처럼 동작한다. + +- complete 응답을 잃으면(시간 초과, 연결 끊김) `upload()`는 같은 COMPLETE key로 + `maxRetries`번 다시 보낸 뒤 `UNAVAILABLE/UPLOAD_COMPLETE, retryable:true`를 + 돌려준다. 모든 part가 수락된 checkpoint는 남는다 + (`resumable-upload-runtime.ts` `completeUpload`). +- 같은 파일로 `upload()`를 다시 부르면 `GET_STATUS`가 `QUARANTINED`일 때 + resourceId와 `replayed:true`를 돌려준다. `ACTIVE`이면 같은 key로 complete를 + 다시 보낸다. +- `abort()`에 서버가 `ALREADY_COMPLETED`로 답하면 checkpoint를 지우기 전에 + `GET_STATUS`로 resourceId를 읽어 `{ state: "ALREADY_COMPLETED", resourceId }`로 + 돌려준다. status 조회가 일시적으로 실패하면 `ABORT_PENDING` checkpoint를 남기고 + 실패를 돌려준다. `abort()`를 다시 부르면 된다. 서버가 session을 모르면(404/410) + resourceId 없이 닫는다(`resumable-upload-runtime.ts` `readCompletedResourceId`). +- 서버가 complete를 반영한 뒤 로컬 checkpoint 삭제가 실패해도(IndexedDB 오류, + complete 직후 caller abort) `upload()`는 성공을 돌려준다. 남은 행은 다음 + `upload()`나 `abort()`가 같은 자원으로 정리한다. + +서버 계약 — complete 진행 중(`ResumableUploadControlPlane.complete` 주석): + +- 같은 COMPLETE key의 complete가 아직 끝나지 않았으면 서버는 `429`와 + `Retry-After`로 답한다. 값은 전송과 runtime의 `maxRetryAfterMs`(기본 30초) + 이하여야 한다. 클라이언트는 이것을 `UNAVAILABLE, retryable:true`로 읽고 그만큼 + 기다린 뒤 같은 key로 다시 보낸다(`fetch-json-transport.ts` `statusFailure`). + `Retry-After`가 없거나 상한을 넘으면 재시도하지 않는다. `503`도 재시도되지만 + `Retry-After`는 읽지 않는다. +- `409`는 쓰지 않는다. 클라이언트는 `409`를 `CONFLICT, retryable:false`로 읽고 + 멈춘다. +- 그동안 `GET_STATUS`는 `ACTIVE`(모든 part 수락)를 유지하고, 자원이 확정된 뒤에만 + `QUARANTINED`와 resourceId를 준다. 클라이언트 status 계약(`UploadSessionStatus`)에는 + "완료 중" 값이 없고, 모르는 state는 `INTEGRITY_FAILED`로 거절한다. +- 서버 문서의 `OPERATION_IN_PROGRESS`(`server-file-capability-infrastructure.md` + 8.2, 11.1)는 이 wire 표현으로 내보낸다. +- 재시도가 모두 끝나도 진행 중이면 `upload()`는 `UNAVAILABLE/RESUME`을 돌려주고 + checkpoint가 남는다. 다음 `upload()`나 `abort()`가 위 순서로 조정한다. + application-facing upload 성공값은 `state`, opaque `resourceId`, `byteLength`, `replayed`만 반환한다. control-plane 검증에 사용한 `sessionId`, `requestBindingSha256`와 file fingerprint는 public outcome에 노출하지 않는다. @@ -633,6 +669,7 @@ analytics는 제품 presentation이 소유한다. | image capability verification concurrency ceiling | `LIMIT_EXCEEDED` | 진행 작업 종료 대기 또는 runtime 부하 조사 | | closed Image CDN runtime 사용 | `UNAVAILABLE` | 새 runtime composition | | network/429/모든 5xx | `UNAVAILABLE` | bounded retry/backoff | +| complete 응답 유실·진행 중 | `UNAVAILABLE`(`UPLOAD_COMPLETE`, retryable) | 같은 파일로 `upload()` 재호출, 또는 `abort()`가 resourceId와 함께 `ALREADY_COMPLETED` 반환(4.3) | 관측성에는 operation, safe outcome, byte/part/candidate bucket, retry bucket과 failure code만 기록한다. capability ID, URL, query, asset/resource/session ID, diff --git a/src/adapters/browser-transfer/resumable-upload/resumable-upload-runtime.ts b/src/adapters/browser-transfer/resumable-upload/resumable-upload-runtime.ts index 7f60ab0b..3167b0b8 100644 --- a/src/adapters/browser-transfer/resumable-upload/resumable-upload-runtime.ts +++ b/src/adapters/browser-transfer/resumable-upload/resumable-upload-runtime.ts @@ -1107,12 +1107,11 @@ async function completeUpload( { recovery: "RECONCILE" }, ); } - const removed = await removeCheckpoint( - dependencies, - checkpoint, - request.signal, - ); - if (!removed.ok) return removed; + // 01-02. The server has committed the resource. A failed local cleanup (an + // IndexedDB error, or the caller aborting right after complete) must not + // turn that into a failure and drop the resourceId. The leftover row is + // reconciled to this same resource by the next upload() or abort(). + await removeCheckpoint(dependencies, checkpoint, request.signal); return browserDataSuccess( quarantinedOutcome(value.resourceId, sessionFromCheckpoint(checkpoint), false), ); @@ -1213,6 +1212,16 @@ async function executeAbort( recovery: "RECONCILE", }); } + let resourceId: string | undefined; + if (aborted.value.state === "ALREADY_COMPLETED") { + // 01-02. The checkpoint is the last local link to a completed resource + // (typically a complete whose response was lost), so its id is read before + // the row goes. A transient status failure keeps the ABORT_PENDING row for + // the next abort(); a definitive one has nothing left to read. + const completed = await readCompletedResourceId(dependencies, checkpoint, signal); + if (!completed.ok) return completed; + resourceId = completed.value ?? undefined; + } const removed = await dependencies.checkpoints.remove({ uploadKey, expectedRevision: checkpoint.revision, @@ -1225,7 +1234,46 @@ async function executeAbort( : aborted.value.state === "ALREADY_COMPLETED" ? "ALREADY_COMPLETED" : "ORPHANED"; - return browserDataSuccess(Object.freeze({ state })); + return browserDataSuccess( + Object.freeze(resourceId ? { state, resourceId } : { state }), + ); +} + +async function readCompletedResourceId( + dependencies: RuntimeDependencies, + checkpoint: ResumableUploadCheckpoint, + signal: AbortSignal, +): Promise> { + const status = await callWithRetry( + dependencies, + "UPLOAD_RECONCILE", + signal, + async (attemptSignal) => + await dependencies.controlPlane.getStatus({ + protocol: RESUMABLE_UPLOAD_PROTOCOL, + sessionId: checkpoint.sessionId, + requestBindingSha256: checkpoint.requestBindingSha256, + fingerprint: checkpoint.fingerprint, + signal: attemptSignal, + }), + ); + if (!status.ok) { + return status.error.code === "NOT_FOUND" || + status.error.code === "EXPIRED_RESOURCE" + ? browserDataSuccess(null) + : remapResult(stripProviderFailure(status), "UPLOAD_ABORT"); + } + const value = status.value; + return browserDataSuccess( + value?.state === "QUARANTINED" && + value.session?.sessionId === checkpoint.sessionId && + value.session.requestBindingSha256 === + checkpoint.requestBindingSha256 && + typeof value.resourceId === "string" && + SAFE_OPAQUE_ID.test(value.resourceId) + ? value.resourceId + : null, + ); } /** Drops the durable checkpoint a finished or dead session no longer needs. */ diff --git a/src/application/ports/browser-transfer/resumable-upload.ts b/src/application/ports/browser-transfer/resumable-upload.ts index 34762ad5..9e9bf207 100644 --- a/src/application/ports/browser-transfer/resumable-upload.ts +++ b/src/application/ports/browser-transfer/resumable-upload.ts @@ -92,6 +92,12 @@ export type QuarantinedUpload = Readonly<{ export type UploadAbortOutcome = Readonly<{ state: "ABORTED" | "ORPHANED" | "ALREADY_COMPLETED" | "NOT_FOUND"; + /** + * 01-02. `ALREADY_COMPLETED` only, when the server status still names the + * quarantined resource the session completed (a complete whose response was + * lost). Absent when the server no longer knows the session. + */ + resourceId?: string; }>; export interface ResumableUploadPort { @@ -194,6 +200,15 @@ export interface ResumableUploadControlPlane { signal: AbortSignal; }>): Promise>>; + /** + * 01-02. In-progress contract. While a complete with the same idempotency + * key is still running, the server answers `429` with a `Retry-After` within + * the client's `maxRetryAfterMs`, which the client retries with the same key + * (`503` is retried too, without reading `Retry-After`). It never answers + * `409`: the client reads that as a final `CONFLICT`. Until the resource is + * committed `getStatus` stays `ACTIVE` with every part accepted; there is no + * "completing" status value. + */ complete(input: Readonly<{ protocol: ResumableUploadProtocol; sessionId: string; diff --git a/tests/unit/resumable-upload-runtime-recovery.test.ts b/tests/unit/resumable-upload-runtime-recovery.test.ts index 59f8b10a..0461914a 100644 --- a/tests/unit/resumable-upload-runtime-recovery.test.ts +++ b/tests/unit/resumable-upload-runtime-recovery.test.ts @@ -398,3 +398,204 @@ describe("resumable upload idempotency keys (01-04)", () => { ); }); }); + +describe("resumable upload completed-result reporting (01-02)", () => { + /** A server whose complete commits the resource, reported by getStatus. */ + function committingHarness( + completeResult: "SUCCEEDS" | "LOST", + ) { + const harness = createControlHarness(); + const original = harness.control.complete.bind(harness.control); + let committed = false; + harness.control.complete = async (input) => { + committed = true; + return completeResult === "SUCCEEDS" + ? await original(input) + : browserDataFailure("UNAVAILABLE", "UPLOAD_COMPLETE", { + retryable: true, + recovery: "RESUME", + }); + }; + harness.control.getStatus = async () => { + const session = harness.getSession(); + if (!session) return browserDataFailure("NOT_FOUND", "UPLOAD_RECONCILE"); + return committed + ? browserDataSuccess({ + state: "QUARANTINED", + session, + resourceId: "resource_01", + }) + : browserDataSuccess({ + state: "ACTIVE", + session, + acceptedParts: [...harness.accepted.values()], + }); + }; + harness.control.abort = async () => + browserDataSuccess({ + state: committed ? "ALREADY_COMPLETED" : "ABORTED", + }); + return harness; + } + + it("reports a completed upload even when the local checkpoint cleanup fails", async () => { + const harness = committingHarness("SUCCEEDS"); + const checkpoints = new MemoryCheckpointStore(); + const remove = checkpoints.remove.bind(checkpoints); + let failRemove = true; + checkpoints.remove = async (input) => + failRemove + ? browserDataFailure("UNAVAILABLE", "UPLOAD_RECONCILE", { + retryable: true, + recovery: "RESUME", + }) + : await remove(input); + const runtime = createResumableUploadRuntime({ + controlPlane: harness.control, + partExecutor: executorFor(harness), + checkpoints, + mutationLock: noContentionLock, + crypto, + policy: runtimePolicy({ maxRetries: 0 }), + now: () => 1_000, + sleep: async () => {}, + }); + const key = "upload_key_cleanup_fails"; + + const completed = await runtime.upload(request(key)); + + expect(completed).toEqual({ + ok: true, + value: { + state: "QUARANTINED", + resourceId: "resource_01", + byteLength: 4, + replayed: false, + }, + }); + // The stale row is the next call's to reconcile, not a reason to fail. + expect(checkpoints.rows.has(key)).toBe(true); + failRemove = false; + const replayed = await runtime.upload(request(key)); + expect(replayed).toMatchObject({ + ok: true, + value: { resourceId: "resource_01", replayed: true }, + }); + expect(checkpoints.rows.has(key)).toBe(false); + }); + + it("returns the resourceId when abort() finds the lost complete already applied", async () => { + const harness = committingHarness("LOST"); + const checkpoints = new MemoryCheckpointStore(); + const runtime = createResumableUploadRuntime({ + controlPlane: harness.control, + partExecutor: executorFor(harness), + checkpoints, + mutationLock: noContentionLock, + crypto, + policy: runtimePolicy({ maxRetries: 3 }), + now: () => 1_000, + sleep: async () => {}, + }); + const key = "upload_key_x2_lost_complete"; + + const lost = await runtime.upload(request(key)); + expect(lost).toMatchObject({ + ok: false, + error: { code: "UNAVAILABLE", operation: "UPLOAD_COMPLETE" }, + }); + const aborted = await runtime.abort({ uploadKey: key, signal: activeSignal }); + + expect(aborted).toEqual({ + ok: true, + value: { state: "ALREADY_COMPLETED", resourceId: "resource_01" }, + }); + expect(checkpoints.rows.size).toBe(0); + }); + + it("waits out an in-progress complete answered with 429 Retry-After", async () => { + // The port's in-progress contract, as the fetch transport decodes it. + const harness = createControlHarness(); + const original = harness.control.complete.bind(harness.control); + const complete = vi.fn( + async (input: Parameters[0]): ReturnType => + complete.mock.calls.length <= 2 + ? Object.freeze({ + ok: false as const, + error: Object.freeze({ + code: "UNAVAILABLE" as const, + operation: "UPLOAD_COMPLETE" as const, + retryable: true, + recovery: "RESUME" as const, + retryAfterMs: 40, + }), + }) + : await original(input), + ); + harness.control.complete = complete; + const delays: number[] = []; + const runtime = createResumableUploadRuntime({ + controlPlane: harness.control, + partExecutor: executorFor(harness), + checkpoints: new MemoryCheckpointStore(), + mutationLock: noContentionLock, + crypto, + policy: runtimePolicy({ maxRetries: 3 }), + now: () => 1_000, + random: () => 0, + sleep: async (delay) => { + delays.push(delay); + }, + }); + + const result = await runtime.upload(request("upload_key_complete_in_progress")); + + expect(result).toMatchObject({ ok: true, value: { replayed: false } }); + expect(delays).toEqual([40, 40]); + expect( + new Set(complete.mock.calls.map(([input]) => input.idempotencyKey)).size, + ).toBe(1); + }); + + it("keeps the checkpoint when the completed resource cannot be read yet", async () => { + const harness = committingHarness("LOST"); + const status = harness.control.getStatus.bind(harness.control); + let statusDown = false; + harness.control.getStatus = async (input) => + statusDown + ? browserDataFailure("UNAVAILABLE", "UPLOAD_RECONCILE", { + retryable: true, + recovery: "RESUME", + }) + : await status(input); + const checkpoints = new MemoryCheckpointStore(); + const runtime = createResumableUploadRuntime({ + controlPlane: harness.control, + partExecutor: executorFor(harness), + checkpoints, + mutationLock: noContentionLock, + crypto, + policy: runtimePolicy({ maxRetries: 0 }), + now: () => 1_000, + sleep: async () => {}, + }); + const key = "upload_key_status_unavailable"; + await runtime.upload(request(key)); + statusDown = true; + + const failed = await runtime.abort({ uploadKey: key, signal: activeSignal }); + expect(failed).toMatchObject({ + ok: false, + error: { code: "UNAVAILABLE", operation: "UPLOAD_ABORT", retryable: true }, + }); + expect(checkpoints.rows.get(key)?.state).toBe("ABORT_PENDING"); + + statusDown = false; + const retried = await runtime.abort({ uploadKey: key, signal: activeSignal }); + expect(retried).toEqual({ + ok: true, + value: { state: "ALREADY_COMPLETED", resourceId: "resource_01" }, + }); + expect(checkpoints.rows.size).toBe(0); + }); +});