fix(browser-transfer): keep the resourceId of a completed or lost-complete upload (01-02)
A failed checkpoint cleanup after a committed complete returned a failure and dropped the resourceId, and abort() on ALREADY_COMPLETED returned no id. The upload now succeeds, abort() reads the id from status first, and the port and design doc state the in-progress complete contract (429 + Retry-After). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
5cc1d002d2
commit
7b2cefbd32
@@ -327,6 +327,42 @@ signed URL이나 receipt를 포함하거나 저장하지 않으며 server state
|
||||
reconcile에서 server 상태를 확인한다. backend는 만료된 orphan multipart를
|
||||
정리하는 TTL job을 가져야 한다.
|
||||
|
||||
complete 결과를 모를 때 클라이언트는 다음처럼 동작한다.
|
||||
|
||||
- complete 응답을 잃으면(시간 초과, 연결 끊김) `upload()`는 같은 COMPLETE key로
|
||||
`maxRetries`번 다시 보낸 뒤 `UNAVAILABLE/UPLOAD_COMPLETE, retryable:true`를
|
||||
돌려준다. 모든 part가 수락된 checkpoint는 남는다
|
||||
(`resumable-upload-runtime.ts` `completeUpload`).
|
||||
- 같은 파일로 `upload()`를 다시 부르면 `GET_STATUS`가 `QUARANTINED`일 때
|
||||
resourceId와 `replayed:true`를 돌려준다. `ACTIVE`이면 같은 key로 complete를
|
||||
다시 보낸다.
|
||||
- `abort()`에 서버가 `ALREADY_COMPLETED`로 답하면 checkpoint를 지우기 전에
|
||||
`GET_STATUS`로 resourceId를 읽어 `{ state: "ALREADY_COMPLETED", resourceId }`로
|
||||
돌려준다. status 조회가 일시적으로 실패하면 `ABORT_PENDING` checkpoint를 남기고
|
||||
실패를 돌려준다. `abort()`를 다시 부르면 된다. 서버가 session을 모르면(404/410)
|
||||
resourceId 없이 닫는다(`resumable-upload-runtime.ts` `readCompletedResourceId`).
|
||||
- 서버가 complete를 반영한 뒤 로컬 checkpoint 삭제가 실패해도(IndexedDB 오류,
|
||||
complete 직후 caller abort) `upload()`는 성공을 돌려준다. 남은 행은 다음
|
||||
`upload()`나 `abort()`가 같은 자원으로 정리한다.
|
||||
|
||||
서버 계약 — complete 진행 중(`ResumableUploadControlPlane.complete` 주석):
|
||||
|
||||
- 같은 COMPLETE key의 complete가 아직 끝나지 않았으면 서버는 `429`와
|
||||
`Retry-After`로 답한다. 값은 전송과 runtime의 `maxRetryAfterMs`(기본 30초)
|
||||
이하여야 한다. 클라이언트는 이것을 `UNAVAILABLE, retryable:true`로 읽고 그만큼
|
||||
기다린 뒤 같은 key로 다시 보낸다(`fetch-json-transport.ts` `statusFailure`).
|
||||
`Retry-After`가 없거나 상한을 넘으면 재시도하지 않는다. `503`도 재시도되지만
|
||||
`Retry-After`는 읽지 않는다.
|
||||
- `409`는 쓰지 않는다. 클라이언트는 `409`를 `CONFLICT, retryable:false`로 읽고
|
||||
멈춘다.
|
||||
- 그동안 `GET_STATUS`는 `ACTIVE`(모든 part 수락)를 유지하고, 자원이 확정된 뒤에만
|
||||
`QUARANTINED`와 resourceId를 준다. 클라이언트 status 계약(`UploadSessionStatus`)에는
|
||||
"완료 중" 값이 없고, 모르는 state는 `INTEGRITY_FAILED`로 거절한다.
|
||||
- 서버 문서의 `OPERATION_IN_PROGRESS`(`server-file-capability-infrastructure.md`
|
||||
8.2, 11.1)는 이 wire 표현으로 내보낸다.
|
||||
- 재시도가 모두 끝나도 진행 중이면 `upload()`는 `UNAVAILABLE/RESUME`을 돌려주고
|
||||
checkpoint가 남는다. 다음 `upload()`나 `abort()`가 위 순서로 조정한다.
|
||||
|
||||
application-facing upload 성공값은 `state`, opaque `resourceId`, `byteLength`,
|
||||
`replayed`만 반환한다. control-plane 검증에 사용한 `sessionId`,
|
||||
`requestBindingSha256`와 file fingerprint는 public outcome에 노출하지 않는다.
|
||||
@@ -633,6 +669,7 @@ analytics는 제품 presentation이 소유한다.
|
||||
| image capability verification concurrency ceiling | `LIMIT_EXCEEDED` | 진행 작업 종료 대기 또는 runtime 부하 조사 |
|
||||
| closed Image CDN runtime 사용 | `UNAVAILABLE` | 새 runtime composition |
|
||||
| network/429/모든 5xx | `UNAVAILABLE` | bounded retry/backoff |
|
||||
| complete 응답 유실·진행 중 | `UNAVAILABLE`(`UPLOAD_COMPLETE`, retryable) | 같은 파일로 `upload()` 재호출, 또는 `abort()`가 resourceId와 함께 `ALREADY_COMPLETED` 반환(4.3) |
|
||||
|
||||
관측성에는 operation, safe outcome, byte/part/candidate bucket, retry bucket과
|
||||
failure code만 기록한다. capability ID, URL, query, asset/resource/session ID,
|
||||
|
||||
@@ -1107,12 +1107,11 @@ async function completeUpload<Capability>(
|
||||
{ recovery: "RECONCILE" },
|
||||
);
|
||||
}
|
||||
const removed = await removeCheckpoint(
|
||||
dependencies,
|
||||
checkpoint,
|
||||
request.signal,
|
||||
);
|
||||
if (!removed.ok) return removed;
|
||||
// 01-02. The server has committed the resource. A failed local cleanup (an
|
||||
// IndexedDB error, or the caller aborting right after complete) must not
|
||||
// turn that into a failure and drop the resourceId. The leftover row is
|
||||
// reconciled to this same resource by the next upload() or abort().
|
||||
await removeCheckpoint(dependencies, checkpoint, request.signal);
|
||||
return browserDataSuccess(
|
||||
quarantinedOutcome(value.resourceId, sessionFromCheckpoint(checkpoint), false),
|
||||
);
|
||||
@@ -1213,6 +1212,16 @@ async function executeAbort<Capability>(
|
||||
recovery: "RECONCILE",
|
||||
});
|
||||
}
|
||||
let resourceId: string | undefined;
|
||||
if (aborted.value.state === "ALREADY_COMPLETED") {
|
||||
// 01-02. The checkpoint is the last local link to a completed resource
|
||||
// (typically a complete whose response was lost), so its id is read before
|
||||
// the row goes. A transient status failure keeps the ABORT_PENDING row for
|
||||
// the next abort(); a definitive one has nothing left to read.
|
||||
const completed = await readCompletedResourceId(dependencies, checkpoint, signal);
|
||||
if (!completed.ok) return completed;
|
||||
resourceId = completed.value ?? undefined;
|
||||
}
|
||||
const removed = await dependencies.checkpoints.remove({
|
||||
uploadKey,
|
||||
expectedRevision: checkpoint.revision,
|
||||
@@ -1225,7 +1234,46 @@ async function executeAbort<Capability>(
|
||||
: aborted.value.state === "ALREADY_COMPLETED"
|
||||
? "ALREADY_COMPLETED"
|
||||
: "ORPHANED";
|
||||
return browserDataSuccess(Object.freeze({ state }));
|
||||
return browserDataSuccess(
|
||||
Object.freeze(resourceId ? { state, resourceId } : { state }),
|
||||
);
|
||||
}
|
||||
|
||||
async function readCompletedResourceId<Capability>(
|
||||
dependencies: RuntimeDependencies<Capability>,
|
||||
checkpoint: ResumableUploadCheckpoint,
|
||||
signal: AbortSignal,
|
||||
): Promise<BrowserDataResult<string | null>> {
|
||||
const status = await callWithRetry(
|
||||
dependencies,
|
||||
"UPLOAD_RECONCILE",
|
||||
signal,
|
||||
async (attemptSignal) =>
|
||||
await dependencies.controlPlane.getStatus({
|
||||
protocol: RESUMABLE_UPLOAD_PROTOCOL,
|
||||
sessionId: checkpoint.sessionId,
|
||||
requestBindingSha256: checkpoint.requestBindingSha256,
|
||||
fingerprint: checkpoint.fingerprint,
|
||||
signal: attemptSignal,
|
||||
}),
|
||||
);
|
||||
if (!status.ok) {
|
||||
return status.error.code === "NOT_FOUND" ||
|
||||
status.error.code === "EXPIRED_RESOURCE"
|
||||
? browserDataSuccess(null)
|
||||
: remapResult(stripProviderFailure(status), "UPLOAD_ABORT");
|
||||
}
|
||||
const value = status.value;
|
||||
return browserDataSuccess(
|
||||
value?.state === "QUARANTINED" &&
|
||||
value.session?.sessionId === checkpoint.sessionId &&
|
||||
value.session.requestBindingSha256 ===
|
||||
checkpoint.requestBindingSha256 &&
|
||||
typeof value.resourceId === "string" &&
|
||||
SAFE_OPAQUE_ID.test(value.resourceId)
|
||||
? value.resourceId
|
||||
: null,
|
||||
);
|
||||
}
|
||||
|
||||
/** Drops the durable checkpoint a finished or dead session no longer needs. */
|
||||
|
||||
@@ -92,6 +92,12 @@ export type QuarantinedUpload = Readonly<{
|
||||
|
||||
export type UploadAbortOutcome = Readonly<{
|
||||
state: "ABORTED" | "ORPHANED" | "ALREADY_COMPLETED" | "NOT_FOUND";
|
||||
/**
|
||||
* 01-02. `ALREADY_COMPLETED` only, when the server status still names the
|
||||
* quarantined resource the session completed (a complete whose response was
|
||||
* lost). Absent when the server no longer knows the session.
|
||||
*/
|
||||
resourceId?: string;
|
||||
}>;
|
||||
|
||||
export interface ResumableUploadPort {
|
||||
@@ -194,6 +200,15 @@ export interface ResumableUploadControlPlane<Capability> {
|
||||
signal: AbortSignal;
|
||||
}>): Promise<UploadProviderResult<UploadPartCapability<Capability>>>;
|
||||
|
||||
/**
|
||||
* 01-02. In-progress contract. While a complete with the same idempotency
|
||||
* key is still running, the server answers `429` with a `Retry-After` within
|
||||
* the client's `maxRetryAfterMs`, which the client retries with the same key
|
||||
* (`503` is retried too, without reading `Retry-After`). It never answers
|
||||
* `409`: the client reads that as a final `CONFLICT`. Until the resource is
|
||||
* committed `getStatus` stays `ACTIVE` with every part accepted; there is no
|
||||
* "completing" status value.
|
||||
*/
|
||||
complete(input: Readonly<{
|
||||
protocol: ResumableUploadProtocol;
|
||||
sessionId: string;
|
||||
|
||||
@@ -398,3 +398,204 @@ describe("resumable upload idempotency keys (01-04)", () => {
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("resumable upload completed-result reporting (01-02)", () => {
|
||||
/** A server whose complete commits the resource, reported by getStatus. */
|
||||
function committingHarness(
|
||||
completeResult: "SUCCEEDS" | "LOST",
|
||||
) {
|
||||
const harness = createControlHarness();
|
||||
const original = harness.control.complete.bind(harness.control);
|
||||
let committed = false;
|
||||
harness.control.complete = async (input) => {
|
||||
committed = true;
|
||||
return completeResult === "SUCCEEDS"
|
||||
? await original(input)
|
||||
: browserDataFailure("UNAVAILABLE", "UPLOAD_COMPLETE", {
|
||||
retryable: true,
|
||||
recovery: "RESUME",
|
||||
});
|
||||
};
|
||||
harness.control.getStatus = async () => {
|
||||
const session = harness.getSession();
|
||||
if (!session) return browserDataFailure("NOT_FOUND", "UPLOAD_RECONCILE");
|
||||
return committed
|
||||
? browserDataSuccess({
|
||||
state: "QUARANTINED",
|
||||
session,
|
||||
resourceId: "resource_01",
|
||||
})
|
||||
: browserDataSuccess({
|
||||
state: "ACTIVE",
|
||||
session,
|
||||
acceptedParts: [...harness.accepted.values()],
|
||||
});
|
||||
};
|
||||
harness.control.abort = async () =>
|
||||
browserDataSuccess({
|
||||
state: committed ? "ALREADY_COMPLETED" : "ABORTED",
|
||||
});
|
||||
return harness;
|
||||
}
|
||||
|
||||
it("reports a completed upload even when the local checkpoint cleanup fails", async () => {
|
||||
const harness = committingHarness("SUCCEEDS");
|
||||
const checkpoints = new MemoryCheckpointStore();
|
||||
const remove = checkpoints.remove.bind(checkpoints);
|
||||
let failRemove = true;
|
||||
checkpoints.remove = async (input) =>
|
||||
failRemove
|
||||
? browserDataFailure("UNAVAILABLE", "UPLOAD_RECONCILE", {
|
||||
retryable: true,
|
||||
recovery: "RESUME",
|
||||
})
|
||||
: await remove(input);
|
||||
const runtime = createResumableUploadRuntime({
|
||||
controlPlane: harness.control,
|
||||
partExecutor: executorFor(harness),
|
||||
checkpoints,
|
||||
mutationLock: noContentionLock,
|
||||
crypto,
|
||||
policy: runtimePolicy({ maxRetries: 0 }),
|
||||
now: () => 1_000,
|
||||
sleep: async () => {},
|
||||
});
|
||||
const key = "upload_key_cleanup_fails";
|
||||
|
||||
const completed = await runtime.upload(request(key));
|
||||
|
||||
expect(completed).toEqual({
|
||||
ok: true,
|
||||
value: {
|
||||
state: "QUARANTINED",
|
||||
resourceId: "resource_01",
|
||||
byteLength: 4,
|
||||
replayed: false,
|
||||
},
|
||||
});
|
||||
// The stale row is the next call's to reconcile, not a reason to fail.
|
||||
expect(checkpoints.rows.has(key)).toBe(true);
|
||||
failRemove = false;
|
||||
const replayed = await runtime.upload(request(key));
|
||||
expect(replayed).toMatchObject({
|
||||
ok: true,
|
||||
value: { resourceId: "resource_01", replayed: true },
|
||||
});
|
||||
expect(checkpoints.rows.has(key)).toBe(false);
|
||||
});
|
||||
|
||||
it("returns the resourceId when abort() finds the lost complete already applied", async () => {
|
||||
const harness = committingHarness("LOST");
|
||||
const checkpoints = new MemoryCheckpointStore();
|
||||
const runtime = createResumableUploadRuntime({
|
||||
controlPlane: harness.control,
|
||||
partExecutor: executorFor(harness),
|
||||
checkpoints,
|
||||
mutationLock: noContentionLock,
|
||||
crypto,
|
||||
policy: runtimePolicy({ maxRetries: 3 }),
|
||||
now: () => 1_000,
|
||||
sleep: async () => {},
|
||||
});
|
||||
const key = "upload_key_x2_lost_complete";
|
||||
|
||||
const lost = await runtime.upload(request(key));
|
||||
expect(lost).toMatchObject({
|
||||
ok: false,
|
||||
error: { code: "UNAVAILABLE", operation: "UPLOAD_COMPLETE" },
|
||||
});
|
||||
const aborted = await runtime.abort({ uploadKey: key, signal: activeSignal });
|
||||
|
||||
expect(aborted).toEqual({
|
||||
ok: true,
|
||||
value: { state: "ALREADY_COMPLETED", resourceId: "resource_01" },
|
||||
});
|
||||
expect(checkpoints.rows.size).toBe(0);
|
||||
});
|
||||
|
||||
it("waits out an in-progress complete answered with 429 Retry-After", async () => {
|
||||
// The port's in-progress contract, as the fetch transport decodes it.
|
||||
const harness = createControlHarness();
|
||||
const original = harness.control.complete.bind(harness.control);
|
||||
const complete = vi.fn(
|
||||
async (input: Parameters<typeof original>[0]): ReturnType<typeof original> =>
|
||||
complete.mock.calls.length <= 2
|
||||
? Object.freeze({
|
||||
ok: false as const,
|
||||
error: Object.freeze({
|
||||
code: "UNAVAILABLE" as const,
|
||||
operation: "UPLOAD_COMPLETE" as const,
|
||||
retryable: true,
|
||||
recovery: "RESUME" as const,
|
||||
retryAfterMs: 40,
|
||||
}),
|
||||
})
|
||||
: await original(input),
|
||||
);
|
||||
harness.control.complete = complete;
|
||||
const delays: number[] = [];
|
||||
const runtime = createResumableUploadRuntime({
|
||||
controlPlane: harness.control,
|
||||
partExecutor: executorFor(harness),
|
||||
checkpoints: new MemoryCheckpointStore(),
|
||||
mutationLock: noContentionLock,
|
||||
crypto,
|
||||
policy: runtimePolicy({ maxRetries: 3 }),
|
||||
now: () => 1_000,
|
||||
random: () => 0,
|
||||
sleep: async (delay) => {
|
||||
delays.push(delay);
|
||||
},
|
||||
});
|
||||
|
||||
const result = await runtime.upload(request("upload_key_complete_in_progress"));
|
||||
|
||||
expect(result).toMatchObject({ ok: true, value: { replayed: false } });
|
||||
expect(delays).toEqual([40, 40]);
|
||||
expect(
|
||||
new Set(complete.mock.calls.map(([input]) => input.idempotencyKey)).size,
|
||||
).toBe(1);
|
||||
});
|
||||
|
||||
it("keeps the checkpoint when the completed resource cannot be read yet", async () => {
|
||||
const harness = committingHarness("LOST");
|
||||
const status = harness.control.getStatus.bind(harness.control);
|
||||
let statusDown = false;
|
||||
harness.control.getStatus = async (input) =>
|
||||
statusDown
|
||||
? browserDataFailure("UNAVAILABLE", "UPLOAD_RECONCILE", {
|
||||
retryable: true,
|
||||
recovery: "RESUME",
|
||||
})
|
||||
: await status(input);
|
||||
const checkpoints = new MemoryCheckpointStore();
|
||||
const runtime = createResumableUploadRuntime({
|
||||
controlPlane: harness.control,
|
||||
partExecutor: executorFor(harness),
|
||||
checkpoints,
|
||||
mutationLock: noContentionLock,
|
||||
crypto,
|
||||
policy: runtimePolicy({ maxRetries: 0 }),
|
||||
now: () => 1_000,
|
||||
sleep: async () => {},
|
||||
});
|
||||
const key = "upload_key_status_unavailable";
|
||||
await runtime.upload(request(key));
|
||||
statusDown = true;
|
||||
|
||||
const failed = await runtime.abort({ uploadKey: key, signal: activeSignal });
|
||||
expect(failed).toMatchObject({
|
||||
ok: false,
|
||||
error: { code: "UNAVAILABLE", operation: "UPLOAD_ABORT", retryable: true },
|
||||
});
|
||||
expect(checkpoints.rows.get(key)?.state).toBe("ABORT_PENDING");
|
||||
|
||||
statusDown = false;
|
||||
const retried = await runtime.abort({ uploadKey: key, signal: activeSignal });
|
||||
expect(retried).toEqual({
|
||||
ok: true,
|
||||
value: { state: "ALREADY_COMPLETED", resourceId: "resource_01" },
|
||||
});
|
||||
expect(checkpoints.rows.size).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user