From c8620042ea73103196b39fea26708bc1385a181c Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Sun, 1 Nov 2026 21:22:20 +0900 Subject: [PATCH] =?UTF-8?q?test(guardian):=20=EA=B0=80=EB=93=9C=EB=A5=BC?= =?UTF-8?q?=20guardian=EC=9D=B4=20=EC=9D=BD=EC=9D=84=20=EC=A4=80=EB=B9=84?= =?UTF-8?q?=EA=B0=80=20=EB=90=9C=20=EB=92=A4=EC=97=90=20=EB=B3=B4=EB=82=B8?= =?UTF-8?q?=EB=8B=A4=20(0929/10-01)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 문제 guardian 테스트는 자식을 띄우자마자 절대 마감(Date.now() + 350)이 든 가드를 보낸다. 그래서 lease가 프로세스 기동 시간까지 낸다. 1코어에서 기동이 350ms를 넘으면 guardian이 가드를 마감 지남으로 거부하고, 테스트는 5초 뒤 "deadline guardian READY timed out"으로 실패한다(수정 전 1코어 재현 9회 중 4회). 같은 구조가 2~3초 lease를 쓰는 테스트 5곳에 있다. near-timeout 테스트는 READY 뒤 1초를 고정으로 자서, 기동이 길면 publish·commit에 남는 시간이 줄어든다. 자식 기동을 기다리는 poll 세 개(waitForFile, waitForNewDirectChild, waitForStoppedProcess)는 1초 마감을 쓴다. 파일 머리의 watchdog 설명("1초 요구를 넣지 않는다")과 어긋난다. 고른 방법 - spawnGuardian이 실제 guardian을 import하는 래퍼로 띄운다. import가 끝나면(bootstrap과 stdin reader 등록까지 끝난 시점) 래퍼가 여분 pipe (fd 7)에 1바이트를 쓰고 닫는다. 새 sendGuard는 이 신호를 받은 뒤에 마감을 계산해 가드를 보낸다. 가드를 보내는 6곳이 모두 sendGuard를 쓴다. guardian 코드와 프로토콜, 350ms·2s·3s lease 값은 그대로다. - near-timeout 테스트는 startProviderGuardian 호출 직전 시각에서 1초 뒤에 깬다. 클라이언트가 spawn 뒤에 마감을 찍으므로 마감은 그 시각 + 2초 이후다. 기동이 길어도 publish·commit에 1초가 남는다. - 1초 poll 세 개는 같은 파일의 watchdog(5초)을 쓴다. 자식 기동에는 기다릴 이벤트가 없어 poll이 남는다. - 같은 종류 guardian 두 개 테스트는 패자가 클라이언트의 고정 5초 READY 제한(RESPONSE_TIMEOUT_MS)으로만 거부된다. 테스트가 줄일 수 없는 고정 대기라 파일의 PROCESS_HEAVY_TIMEOUT_MS를 붙이고 이유를 주석에 적었다. 버린 대안 - lease를 늘리기: 기동 시간에 상한이 없어 같은 문제가 더 드물게 남는다. 350ms 테스트는 만료를 기다리는 시간도 길어진다. - marker 파일 + poll로 준비 신호: 파일 시스템 poll에 다시 마감이 필요하다. pipe는 이벤트로 끝나고 provider-evidence 목록 단언에 끼어들지 않는다. - guardian 프로세스에 가짜 시계 주입: 자식 프로세스라 테스트용 훅을 운영 스크립트에 넣어야 한다. 위험·한계 - spawnGuardian 경로의 guardian은 main 모듈이 아니라 래퍼가 import한 모듈로 돈다. guardian은 argv·import.meta를 읽지 않는다. 래퍼는 신호를 보낸 뒤 fd 7을 닫으므로 가드를 받는 시점의 상속 fd(0~6)는 운영과 같다. 클라이언트 경로 테스트는 그대로 실제 스크립트를 직접 띄운다. - 가드를 보내기 전 기동 대기는 watchdog 5초가 상한이다. 남은 실제 시계 구간은 가드 처리(decode, link, fsync, READY)뿐이다. - 따로 남는 결함(이 커밋 범위 밖): guardian은 가드를 거부하거나 link가 실패하면 failClosed가 진단을 쓰지만, stdin 읽기가 걸려 있는 동안 종료하지 않는다. 탐침: 만료된 가드를 받고 1,008ms에 진단을 쓴 뒤 stdin을 닫은 4,014ms에야 126으로 끝났다. 그래서 클라이언트에서는 거부가 즉시 오지 않고 5초 뒤 "READY timed out"이 된다(leaseMs 1과 동시 guardian 탐침 5,018~5,053ms). 위 PROCESS_HEAVY_TIMEOUT_MS는 이 결함이 고쳐지면 뺄 수 있다. 검증 - RED(수정 전, taskset -c 0, --maxWorkers=4): component·features· integration·platform/cardinality 3회 중 2회, 같은 묶음 + 바쁜 이웃 1개 3회 중 2회 "expires an uncommitted lease … deadline guardian READY timed out"(5,035~5,052ms). - GREEN(세 커밋을 합친 상태, taskset -c 0 + 같은 코어의 바쁜 이웃, --maxWorkers=4): 대상 6개 파일 6회, 위 넓은 묶음 5회(이웃 1개) 모두 통과. 이웃 2개로 3회 돌렸을 때 이 파일은 모두 통과했다(실패 1건은 http-execution-v3-live-authority, 0929/10-03). - 기본 코어: tests/unit·component·integration 212파일 2,734개 통과 (pnpm을 띄우는 security-followup-archive, fixture-node-modules, lab-performance 제외). tsc -p tsconfig.test.json, 바꾼 파일 eslint 통과. Refs: docs/reviews/2026-11-01-maturity-review/10-tests-fast-layers.md --- .../provider-guardian-transaction.test.ts | 133 ++++++++++-------- 1 file changed, 77 insertions(+), 56 deletions(-) diff --git a/tests/integration/provider-guardian-transaction.test.ts b/tests/integration/provider-guardian-transaction.test.ts index 50543deb..bc29241e 100644 --- a/tests/integration/provider-guardian-transaction.test.ts +++ b/tests/integration/provider-guardian-transaction.test.ts @@ -1,6 +1,6 @@ import { spawn } from "node:child_process"; import { createHash } from "node:crypto"; -import { closeSync, constants, fchmodSync, openSync, watch } from "node:fs"; +import { closeSync, constants, fchmodSync, openSync, watch, writeFileSync } from "node:fs"; import { lstat, mkdir, @@ -29,9 +29,11 @@ const PROCESS_HEAVY_TIMEOUT_MS = 30_000; * Test watchdog only. This is not a provider protocol deadline. Child-process * scheduling can be delayed when the repository suite is under load, so the * watchdog must not encode a one-second protocol requirement that does not - * exist in production. + * exist in production. It also bounds the polls that wait for a child to start. */ const PROCESS_HANDSHAKE_WATCHDOG_MS = 5_000; +/** The extra pipe on which spawnGuardian's wrapper reports the guardian loaded. */ +const GUARDIAN_LOADED_FD = 7; const roots: string[] = []; @@ -203,10 +205,9 @@ describe("provider guardian transaction protocol", () => { }); it("creates and returns owned raw and sealed-temp identities before READY", async () => { - const { - decodeProviderGuardianReady, - encodeProviderGuardianGuard, - } = await import("../../scripts/lib/provider-guardian-protocol.ts"); + const { decodeProviderGuardianReady } = await import( + "../../scripts/lib/provider-guardian-protocol.ts" + ); const workspace = await createWorkspace("provider-guardian-ready-"); const rawPath = path.join( workspace, @@ -216,11 +217,7 @@ describe("provider guardian transaction protocol", () => { const child = spawnGuardian(workspace, nonce); const completion = waitForChild(child); const readyPayload = readFrame(child.stdout!); - child.stdin!.write(encodeProviderGuardianGuard({ - kind: "vulnerability", - nonce, - deadlineEpochMs: Date.now() + 2_000, - })); + await sendGuard(child, nonce, 2_000); const ready = decodeProviderGuardianReady( await within(readyPayload, PROCESS_HANDSHAKE_WATCHDOG_MS, "guardian READY"), @@ -314,18 +311,11 @@ describe("provider guardian transaction protocol", () => { ); it("cleans guardian-created objects when the parent closes immediately after guard", async () => { - const { encodeProviderGuardianGuard } = await import( - "../../scripts/lib/provider-guardian-protocol.ts" - ); const workspace = await createWorkspace("provider-guardian-parent-startup-death-"); const nonce = Buffer.alloc(32, 0x2b); const child = spawnGuardian(workspace, nonce); const completion = waitForChild(child); - child.stdin!.end(encodeProviderGuardianGuard({ - kind: "vulnerability", - nonce, - deadlineEpochMs: Date.now() + 2_000, - })); + await sendGuard(child, nonce, 2_000, { closeInput: true }); await expect(within(completion, PROCESS_HANDSHAKE_WATCHDOG_MS, "guardian parent startup death")) .resolves.toEqual({ code: 125, signal: null }); @@ -338,7 +328,6 @@ describe("provider guardian transaction protocol", () => { decodeProviderGuardianPublished, decodeProviderGuardianReady, encodeProviderGuardianCommit, - encodeProviderGuardianGuard, encodeProviderGuardianPublish, } = await import("../../scripts/lib/provider-guardian-protocol.ts"); const workspace = await createWorkspace("provider-guardian-publish-"); @@ -349,11 +338,7 @@ describe("provider guardian transaction protocol", () => { const child = spawnGuardian(workspace, nonce); const completion = waitForChild(child); const readyPayload = readFrame(child.stdout!); - child.stdin!.write(encodeProviderGuardianGuard({ - kind: "vulnerability", - nonce, - deadlineEpochMs: Date.now() + 3_000, - })); + await sendGuard(child, nonce, 3_000); const ready = decodeProviderGuardianReady( await within(readyPayload, PROCESS_HANDSHAKE_WATCHDOG_MS, "guardian READY"), nonce, @@ -616,6 +601,10 @@ describe("provider guardian transaction protocol", () => { .resolves.toEqual(["untrusted"]); }); + // The loser is rejected only by the client's fixed 5s READY limit + // (RESPONSE_TIMEOUT_MS): its guardian fails the link but does not exit while + // its stdin read is still pending. That fixed wait would leave half of the + // 10s default for three guardian start-ups. it("allows exactly one same-kind guardian without deleting the winner", async () => { const { startProviderGuardian } = await import( "../../scripts/lib/provider-guardian-client.ts" @@ -651,13 +640,12 @@ describe("provider guardian transaction protocol", () => { .resolves.toEqual([]); await expect(readdir(path.join(workspace, "provider-evidence"))) .resolves.toEqual(["untrusted"]); - }); + }, PROCESS_HEAVY_TIMEOUT_MS); it("exits nonzero after cleanup when the diagnostic stderr pipe is closed", async () => { - const { - decodeProviderGuardianReady, - encodeProviderGuardianGuard, - } = await import("../../scripts/lib/provider-guardian-protocol.ts"); + const { decodeProviderGuardianReady } = await import( + "../../scripts/lib/provider-guardian-protocol.ts" + ); const workspace = await createWorkspace("provider-guardian-closed-stderr-"); const evidenceRoot = path.join(workspace, "provider-evidence"); const rawDirectory = path.join(evidenceRoot, "untrusted"); @@ -665,11 +653,7 @@ describe("provider guardian transaction protocol", () => { const child = spawnGuardian(workspace, nonce); const completion = waitForChild(child); const readyPayload = readFrame(child.stdout!); - child.stdin!.write(encodeProviderGuardianGuard({ - kind: "vulnerability", - nonce, - deadlineEpochMs: Date.now() + 3_000, - })); + await sendGuard(child, nonce, 3_000); const ready = decodeProviderGuardianReady( await within(readyPayload, PROCESS_HANDSHAKE_WATCHDOG_MS, "closed-stderr guardian READY"), nonce, @@ -687,10 +671,9 @@ describe("provider guardian transaction protocol", () => { }); it("expires an uncommitted lease only after cleaning every owned object", async () => { - const { - decodeProviderGuardianReady, - encodeProviderGuardianGuard, - } = await import("../../scripts/lib/provider-guardian-protocol.ts"); + const { decodeProviderGuardianReady } = await import( + "../../scripts/lib/provider-guardian-protocol.ts" + ); const workspace = await createWorkspace("provider-guardian-deadline-v2-"); const evidenceRoot = path.join(workspace, "provider-evidence"); const rawPath = path.join(evidenceRoot, "untrusted/vulnerability-report.json"); @@ -698,11 +681,7 @@ describe("provider guardian transaction protocol", () => { const child = spawnGuardian(workspace, nonce); const completion = waitForChild(child); const readyPayload = readFrame(child.stdout!); - child.stdin!.write(encodeProviderGuardianGuard({ - kind: "vulnerability", - nonce, - deadlineEpochMs: Date.now() + 350, - })); + await sendGuard(child, nonce, 350); const ready = decodeProviderGuardianReady( await within(readyPayload, PROCESS_HANDSHAKE_WATCHDOG_MS, "deadline guardian READY"), nonce, @@ -721,13 +700,18 @@ describe("provider guardian transaction protocol", () => { "../../scripts/lib/provider-guardian-client.ts" ); const workspace = await createWorkspace("provider-guardian-near-timeout-"); + // The client stamps the deadline after it spawns the guardian, so the + // deadline is at least 2s from here. Waking 1s from here, not 1s after + // start-up returns, keeps publish and commit 1s on a slow host too. + const startedAt = Date.now(); const lease = await startProviderGuardian({ kind: "vulnerability", workspaceRoot: workspace, leaseMs: 2_000, guardianScript: path.resolve("scripts/lib/provider-raw-guardian.ts"), }); - await new Promise((resolve) => setTimeout(resolve, 1_000)); + await new Promise((resolve) => + setTimeout(resolve, Math.max(0, startedAt + 1_000 - Date.now()))); const bytes = Buffer.from('{"validated":"near-timeout"}\n'); await lease.publish(bytes); @@ -836,7 +820,7 @@ async function createPausedGuardianFixture(workspace: string): Promise { } async function waitForFile(target: string): Promise { - const deadline = Date.now() + 1_000; + const deadline = Date.now() + PROCESS_HANDSHAKE_WATCHDOG_MS; while (Date.now() <= deadline) { try { return await readFile(target); @@ -866,7 +850,6 @@ async function establishPublishedGuardian( const { decodeProviderGuardianPublished, decodeProviderGuardianReady, - encodeProviderGuardianGuard, encodeProviderGuardianPublish, } = await import("../../scripts/lib/provider-guardian-protocol.ts"); const evidenceRoot = path.join(workspace, "provider-evidence"); @@ -875,11 +858,7 @@ async function establishPublishedGuardian( const child = spawnGuardian(workspace, nonce); const completion = waitForChild(child); const readyPayload = readFrame(child.stdout!); - child.stdin!.write(encodeProviderGuardianGuard({ - kind: "vulnerability", - nonce, - deadlineEpochMs: Date.now() + 3_000, - })); + await sendGuard(child, nonce, 3_000); const ready = decodeProviderGuardianReady( await within(readyPayload, PROCESS_HANDSHAKE_WATCHDOG_MS, "guardian READY"), nonce, @@ -986,16 +965,58 @@ function spawnGuardian( constants.O_RDWR | constants.O_CREAT | constants.O_EXCL | constants.O_NOFOLLOW, 0o600, )); - return spawn(process.execPath, [path.resolve("scripts/lib/provider-raw-guardian.ts")], { + // The wrapper imports the unchanged guardian, then writes one byte to an + // extra pipe and closes it: by then bootstrap has run and the stdin reader + // is attached (sendGuard). + const loader = path.join(workspace, "loaded-signal-guardian.mjs"); + writeFileSync(loader, [ + 'import { closeSync, writeSync } from "node:fs";', + `await import(${JSON.stringify( + pathToFileURL(path.resolve("scripts/lib/provider-raw-guardian.ts")).href, + )});`, + `writeSync(${GUARDIAN_LOADED_FD}, "1");`, + `closeSync(${GUARDIAN_LOADED_FD});`, + "", + ].join("\n")); + return spawn(process.execPath, [loader], { cwd: workspace, env: {}, - stdio: ["pipe", "pipe", "pipe", ...openedFds], + stdio: ["pipe", "pipe", "pipe", ...openedFds, "pipe"], }); } finally { for (const fd of openedFds) closeSync(fd); } } +/** + * A guard carries an absolute deadline, so it goes out only once the guardian + * reads stdin. Sent at spawn, the lease also pays for process start-up, which + * has no bound on a loaded host: a 350ms lease then expires before the + * guardian decodes the guard. + */ +async function sendGuard( + child: ReturnType, + nonce: Buffer, + leaseMs: number, + options: Readonly<{ closeInput?: boolean }> = {}, +): Promise { + const { encodeProviderGuardianGuard } = await import( + "../../scripts/lib/provider-guardian-protocol.ts" + ); + const loaded = (child.stdio as readonly unknown[])[GUARDIAN_LOADED_FD] as NodeJS.ReadableStream; + await within(new Promise((resolve, reject) => { + loaded.once("data", () => resolve()); + loaded.once("end", () => reject(new Error("guardian exited before it loaded"))); + }), PROCESS_HANDSHAKE_WATCHDOG_MS, "guardian start-up"); + const guard = encodeProviderGuardianGuard({ + kind: "vulnerability", + nonce, + deadlineEpochMs: Date.now() + leaseMs, + }); + if (options.closeInput) child.stdin!.end(guard); + else child.stdin!.write(guard); +} + function spawnGuardianWithPrivateModes( workspace: string, nonce: Buffer, @@ -1095,7 +1116,7 @@ async function directChildPids(parentPid: number): Promise> { } async function waitForNewDirectChild(previous: ReadonlySet): Promise { - const deadline = Date.now() + 1_000; + const deadline = Date.now() + PROCESS_HANDSHAKE_WATCHDOG_MS; while (Date.now() <= deadline) { for (const pid of await directChildPids(process.pid)) { if (!previous.has(pid)) return pid; @@ -1106,7 +1127,7 @@ async function waitForNewDirectChild(previous: ReadonlySet): Promise { - const deadline = Date.now() + 1_000; + const deadline = Date.now() + PROCESS_HANDSHAKE_WATCHDOG_MS; while (Date.now() <= deadline) { const status = await readFile(`/proc/${pid}/status`, "utf8"); if (/^State:\s+T/mu.test(status)) return;