test(guardian): lease가 시험 대상이 아니면 시험 timeout만큼 준다
tests/integration/provider-guardian-transaction.test.ts가 부하 때 lease
만료로 실패한다. guardian이 스스로 SIGKILL한 것이다. desktop 기록 둘:
- FE-GATE-007: 「still publishes near the lease deadline」
- FE-GATE-020 drill: 「owns raw and sealed publication through the
authenticated client lease」
FE-B 실행에서도 load 28에 같은 실패가 2건 났다. 다시 돌리면 통과한다.
원인은 두 가지다.
- startProviderGuardian은 guardian을 띄우자마자 절대 deadline
(Date.now() + leaseMs)을 찍는다. 그래서 client 경로 시험의 2~3초
lease가 node 시작 시간까지 낸다. 부하 때 시작에는 상한이 없다.
- 직접 프로토콜 시험은 guardian이 load를 마친 뒤 가드를 보낸다
(c8620042). 그래도 가드 → READY 처리는 실제 시계로 돈다. desktop에서
CPU 25%로 제한하면 이 구간이 17ms에서 334ms로 늘어나, 350ms lease로
만료를 보는 시험의 여유가 거의 없다.
lease가 시험 대상인지로 나눠 고쳤다.
- lease가 대상이 아닌 15곳(client 경로 10, 직접 경로 5)은
LEASE_NOT_UNDER_TEST_MS다. 값은 시험 timeout과 같아서, lease가 시험보다
먼저 끝날 수 없다. guardian이 멈추면 시험 timeout이 잡는다.
- lease 만료를 보는 시험은 350ms → 2초다. 가드 → READY의 CPU 25% 최대
334ms의 여섯 배다.
- near-deadline 시험은 client 경로에서 직접 프로토콜로 옮겼다.
- deadline이 guardian load 뒤에 시작해서 시작 시간이 빠진다.
- sendGuard가 보낸 deadline을 돌려준다.
- 시험은 lease 4초의 마지막 2초 안에 publish·commit을 보내고, 그
구간에 있었는지 단언한다.
- 2초는 publish·commit의 CPU 25% 최대 104ms의 약 스무 배다.
- client의 publish·commit은 「owns raw and sealed publication」이 그대로
본다.
버린 것:
- guardian에 가짜 시계 주입: 운영 스크립트에 시험용 훅을 넣어야 한다
(c8620042와 같은 이유).
- client가 guardian load 뒤에 deadline을 찍게 바꾸기: 운영 프로토콜에
준비 신호를 더해야 한다. 이 PR 범위를 넘는다.
- 모든 lease를 늘리기: 만료와 near-deadline 시험은 lease가 대상이라
늘리면 보는 것이 흐려진다.
위험:
- 시험이 실패하면 guardian이 lease 끝까지(최대 30초) 남을 수 있다.
전에는 2~3초였다.
- near-deadline 시험에는 실제 시계 구간이 남는다. publish·commit이 2초를
넘으면 실패한다. 잰 최대의 약 스무 배다.
- 운영 client가 lease에 node 시작 시간을 포함하는 것은 그대로다. 운영
lease는 분 단위라 시험만큼 빠듯하지 않다.
검증(desktop):
- 이 파일을 10회 돌려 22개 모두 통과했다(load 3~4).
- CPU 20% 제한에서 main d2239c67은 5회 모두 1~6개 실패했다. 원인은 가드
처리 중 deadline 지남(126), READY·PUBLISHED 대기 시간 초과, EPIPE였다.
같은 조건에서 이 커밋은 5회 모두 통과했다.
- 시험 timeout은 잰 최대로 고쳤다. near-deadline 3.2초, 만료 2.1초.
- 첫 최종 실행에서 FE-GATE-007이 near-deadline 시험의 「마지막 2초 안」
단언으로 실패했다(남은 2,001ms). 타이머가 Date.now()보다 1ms 일찍 깼다.
목표 시각에 닿을 때까지 다시 기다리게 고쳤다. 고친 뒤 이
파일을 30회(load 3.6~21), CPU 20% 제한으로 5회 돌려 모두 통과했다.
This commit is contained in:
@@ -35,6 +35,26 @@ const PROCESS_HEAVY_TIMEOUT_MS = 30_000;
|
||||
const PROCESS_HANDSHAKE_WATCHDOG_MS = 5_000;
|
||||
/** The extra pipe on which spawnGuardian's wrapper reports the guardian loaded. */
|
||||
const GUARDIAN_LOADED_FD = 7;
|
||||
/**
|
||||
* The lease of a test whose subject is not the lease: as long as the test's
|
||||
* timeout, so the guardian's deadline cannot end a test first. A guard carries
|
||||
* an absolute deadline. Under load a 2 to 3 s lease ran out before commit and
|
||||
* the guardian killed itself (FE-GATE-007 and FE-GATE-020 drills on desktop,
|
||||
* 2026-11-12, load average 28): startProviderGuardian stamps the deadline when
|
||||
* it spawns the guardian, so its lease also paid for node start-up.
|
||||
*/
|
||||
const LEASE_NOT_UNDER_TEST_MS = PROCESS_HEAVY_TIMEOUT_MS;
|
||||
/*
|
||||
* The leases a test measures come from desktop runs of the direct-protocol
|
||||
* publish test (2026-11-12, eight runs each, with and without a 25% CPU
|
||||
* quota): guard to READY took at most 17 ms, 334 ms under the quota; publish
|
||||
* and commit at most 34 ms, 104 ms under the quota.
|
||||
*/
|
||||
/** Has to outlast guard to READY before it expires: six times the 334 ms. */
|
||||
const EXPIRING_LEASE_MS = 2_000;
|
||||
/** Publish and commit go out in the lease's last NEAR_DEADLINE_MARGIN_MS, twenty times the 104 ms. */
|
||||
const NEAR_DEADLINE_LEASE_MS = 4_000;
|
||||
const NEAR_DEADLINE_MARGIN_MS = 2_000;
|
||||
|
||||
const roots: string[] = [];
|
||||
|
||||
@@ -218,7 +238,7 @@ describe("provider guardian transaction protocol", () => {
|
||||
const child = spawnGuardian(workspace, nonce);
|
||||
const completion = waitForChild(child);
|
||||
const readyPayload = readFrame(child.stdout!);
|
||||
await sendGuard(child, nonce, 2_000);
|
||||
await sendGuard(child, nonce, LEASE_NOT_UNDER_TEST_MS);
|
||||
|
||||
const ready = decodeProviderGuardianReady(
|
||||
await within(readyPayload, PROCESS_HANDSHAKE_WATCHDOG_MS, "guardian READY"),
|
||||
@@ -243,7 +263,7 @@ describe("provider guardian transaction protocol", () => {
|
||||
await expect(lstat(rawPath)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await expect(lstat(sealedTempPath)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await expect(readdir(path.join(workspace, "provider-evidence/untrusted"))).resolves.toEqual([]);
|
||||
}, childProcessTimeout(253));
|
||||
}, childProcessTimeout(854));
|
||||
|
||||
it("creates no evidence for empty or truncated guard input", async () => {
|
||||
const { startProviderGuardian } = await import(
|
||||
@@ -271,7 +291,7 @@ describe("provider guardian transaction protocol", () => {
|
||||
const retry = await startProviderGuardian({
|
||||
kind: "vulnerability",
|
||||
workspaceRoot: workspace,
|
||||
leaseMs: 3_000,
|
||||
leaseMs: LEASE_NOT_UNDER_TEST_MS,
|
||||
guardianScript: path.resolve("scripts/lib/provider-raw-guardian.ts"),
|
||||
});
|
||||
await retry.abort();
|
||||
@@ -280,7 +300,7 @@ describe("provider guardian transaction protocol", () => {
|
||||
await expect(readdir(path.join(workspace, "provider-evidence")))
|
||||
.resolves.toEqual(["untrusted"]);
|
||||
}
|
||||
}, childProcessTimeout(1_226));
|
||||
}, childProcessTimeout(1_516));
|
||||
|
||||
it.each(["raw", "sealed"] as const)(
|
||||
"cleans the independently bound %s private alias when peer bootstrap validation fails",
|
||||
@@ -317,7 +337,7 @@ describe("provider guardian transaction protocol", () => {
|
||||
const nonce = Buffer.alloc(32, 0x2b);
|
||||
const child = spawnGuardian(workspace, nonce);
|
||||
const completion = waitForChild(child);
|
||||
await sendGuard(child, nonce, 2_000, { closeInput: true });
|
||||
await sendGuard(child, nonce, LEASE_NOT_UNDER_TEST_MS, { closeInput: true });
|
||||
|
||||
await expect(within(completion, PROCESS_HANDSHAKE_WATCHDOG_MS, "guardian parent startup death"))
|
||||
.resolves.toEqual({ code: 125, signal: null });
|
||||
@@ -334,7 +354,7 @@ describe("provider guardian transaction protocol", () => {
|
||||
const completion = waitForChild(child);
|
||||
try {
|
||||
// A different nonce than the inherited private aliases were named for.
|
||||
await sendGuard(child, Buffer.alloc(32, 0x3e), 2_000);
|
||||
await sendGuard(child, Buffer.alloc(32, 0x3e), LEASE_NOT_UNDER_TEST_MS);
|
||||
await expect(within(completion, PROCESS_HANDSHAKE_WATCHDOG_MS, "rejected-guard guardian exit"))
|
||||
.resolves.toEqual({ code: 126, signal: null });
|
||||
} finally {
|
||||
@@ -359,7 +379,7 @@ describe("provider guardian transaction protocol", () => {
|
||||
const child = spawnGuardian(workspace, nonce);
|
||||
const completion = waitForChild(child);
|
||||
const readyPayload = readFrame(child.stdout!);
|
||||
await sendGuard(child, nonce, 3_000);
|
||||
await sendGuard(child, nonce, LEASE_NOT_UNDER_TEST_MS);
|
||||
const ready = decodeProviderGuardianReady(
|
||||
await within(readyPayload, PROCESS_HANDSHAKE_WATCHDOG_MS, "guardian READY"),
|
||||
nonce,
|
||||
@@ -438,7 +458,7 @@ describe("provider guardian transaction protocol", () => {
|
||||
const lease = await start({
|
||||
kind: "vulnerability",
|
||||
workspaceRoot: workspace,
|
||||
leaseMs: 3_000,
|
||||
leaseMs: LEASE_NOT_UNDER_TEST_MS,
|
||||
guardianScript,
|
||||
});
|
||||
expect((await readFile(`/proc/${lease.pid}/cmdline`)).toString("utf8")
|
||||
@@ -479,13 +499,13 @@ describe("provider guardian transaction protocol", () => {
|
||||
const retry = await startProviderGuardian({
|
||||
kind: "vulnerability",
|
||||
workspaceRoot: workspace,
|
||||
leaseMs: 2_000,
|
||||
leaseMs: LEASE_NOT_UNDER_TEST_MS,
|
||||
guardianScript: path.resolve("scripts/lib/provider-raw-guardian.ts"),
|
||||
});
|
||||
await retry.abort();
|
||||
await expect(readdir(path.join(workspace, "provider-evidence/untrusted"))).resolves.toEqual([]);
|
||||
await expect(readdir(path.join(workspace, "provider-evidence"))).resolves.toEqual(["untrusted"]);
|
||||
}, childProcessTimeout(429));
|
||||
}, childProcessTimeout(2_455));
|
||||
|
||||
it("fails closed when a separate trailing frame arrives after commit", async () => {
|
||||
const { encodeProviderGuardianCommit } = await import(
|
||||
@@ -513,7 +533,7 @@ describe("provider guardian transaction protocol", () => {
|
||||
const lease = await startProviderGuardian({
|
||||
kind: "vulnerability",
|
||||
workspaceRoot: workspace,
|
||||
leaseMs: 3_000,
|
||||
leaseMs: LEASE_NOT_UNDER_TEST_MS,
|
||||
guardianScript: path.resolve("scripts/lib/provider-raw-guardian.ts"),
|
||||
});
|
||||
|
||||
@@ -539,7 +559,7 @@ describe("provider guardian transaction protocol", () => {
|
||||
const starting = startProviderGuardian({
|
||||
kind: "vulnerability",
|
||||
workspaceRoot: workspace,
|
||||
leaseMs: 3_000,
|
||||
leaseMs: LEASE_NOT_UNDER_TEST_MS,
|
||||
guardianScript,
|
||||
});
|
||||
const guardianPid = await waitForNewDirectChild(existingChildren);
|
||||
@@ -561,7 +581,7 @@ describe("provider guardian transaction protocol", () => {
|
||||
const retry = await startProviderGuardian({
|
||||
kind: "vulnerability",
|
||||
workspaceRoot: workspace,
|
||||
leaseMs: 3_000,
|
||||
leaseMs: LEASE_NOT_UNDER_TEST_MS,
|
||||
guardianScript: path.resolve("scripts/lib/provider-raw-guardian.ts"),
|
||||
});
|
||||
expect(path.basename(retry.sealedTempPath)).toMatch(
|
||||
@@ -585,7 +605,7 @@ describe("provider guardian transaction protocol", () => {
|
||||
const starting = startProviderGuardian({
|
||||
kind: "vulnerability",
|
||||
workspaceRoot: workspace,
|
||||
leaseMs: 3_000,
|
||||
leaseMs: LEASE_NOT_UNDER_TEST_MS,
|
||||
guardianScript,
|
||||
});
|
||||
const guardianPid = Number(await waitForFile(markerPath));
|
||||
@@ -612,7 +632,7 @@ describe("provider guardian transaction protocol", () => {
|
||||
const retry = await startProviderGuardian({
|
||||
kind: "vulnerability",
|
||||
workspaceRoot: workspace,
|
||||
leaseMs: 3_000,
|
||||
leaseMs: LEASE_NOT_UNDER_TEST_MS,
|
||||
guardianScript: path.resolve("scripts/lib/provider-raw-guardian.ts"),
|
||||
});
|
||||
await retry.abort();
|
||||
@@ -630,7 +650,7 @@ describe("provider guardian transaction protocol", () => {
|
||||
const input = {
|
||||
kind: "vulnerability" as const,
|
||||
workspaceRoot: workspace,
|
||||
leaseMs: 10_000,
|
||||
leaseMs: LEASE_NOT_UNDER_TEST_MS,
|
||||
guardianScript: path.resolve("scripts/lib/provider-raw-guardian.ts"),
|
||||
};
|
||||
|
||||
@@ -670,7 +690,7 @@ describe("provider guardian transaction protocol", () => {
|
||||
const child = spawnGuardian(workspace, nonce);
|
||||
const completion = waitForChild(child);
|
||||
const readyPayload = readFrame(child.stdout!);
|
||||
await sendGuard(child, nonce, 3_000);
|
||||
await sendGuard(child, nonce, LEASE_NOT_UNDER_TEST_MS);
|
||||
const ready = decodeProviderGuardianReady(
|
||||
await within(readyPayload, PROCESS_HANDSHAKE_WATCHDOG_MS, "closed-stderr guardian READY"),
|
||||
nonce,
|
||||
@@ -685,7 +705,7 @@ describe("provider guardian transaction protocol", () => {
|
||||
.rejects.toMatchObject({ code: "ENOENT" });
|
||||
await expect(lstat(path.join(evidenceRoot, ready.sealedTempLeaf)))
|
||||
.rejects.toMatchObject({ code: "ENOENT" });
|
||||
}, childProcessTimeout(469));
|
||||
}, childProcessTimeout(576));
|
||||
|
||||
it("expires an uncommitted lease only after cleaning every owned object", async () => {
|
||||
const { decodeProviderGuardianReady } = await import(
|
||||
@@ -698,7 +718,7 @@ describe("provider guardian transaction protocol", () => {
|
||||
const child = spawnGuardian(workspace, nonce);
|
||||
const completion = waitForChild(child);
|
||||
const readyPayload = readFrame(child.stdout!);
|
||||
await sendGuard(child, nonce, 350);
|
||||
await sendGuard(child, nonce, EXPIRING_LEASE_MS);
|
||||
const ready = decodeProviderGuardianReady(
|
||||
await within(readyPayload, PROCESS_HANDSHAKE_WATCHDOG_MS, "deadline guardian READY"),
|
||||
nonce,
|
||||
@@ -710,32 +730,70 @@ describe("provider guardian transaction protocol", () => {
|
||||
await expect(lstat(rawPath)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await expect(lstat(sealedTempPath)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await expect(readdir(evidenceRoot)).resolves.toEqual(["untrusted"]);
|
||||
}, PROCESS_HEAVY_TIMEOUT_MS);
|
||||
}, childProcessTimeout(2_114));
|
||||
|
||||
it("still publishes near the lease deadline when post-processing completes in time", async () => {
|
||||
const { startProviderGuardian } = await import(
|
||||
"../../scripts/lib/provider-guardian-client.ts"
|
||||
);
|
||||
const {
|
||||
decodeProviderGuardianPublished,
|
||||
decodeProviderGuardianReady,
|
||||
encodeProviderGuardianCommit,
|
||||
encodeProviderGuardianPublish,
|
||||
} = await import("../../scripts/lib/provider-guardian-protocol.ts");
|
||||
const workspace = await createWorkspace("provider-guardian-near-timeout-");
|
||||
// The client stamps the deadline after it spawns the guardian, so the
|
||||
// deadline is at least 2s from here. Waking 1s from here, not 1s after
|
||||
// start-up returns, keeps publish and commit 1s on a slow host too.
|
||||
const startedAt = Date.now();
|
||||
const lease = await startProviderGuardian({
|
||||
kind: "vulnerability",
|
||||
workspaceRoot: workspace,
|
||||
leaseMs: 2_000,
|
||||
guardianScript: path.resolve("scripts/lib/provider-raw-guardian.ts"),
|
||||
});
|
||||
await new Promise<void>((resolve) =>
|
||||
setTimeout(resolve, Math.max(0, startedAt + 1_000 - Date.now())));
|
||||
const evidenceRoot = path.join(workspace, "provider-evidence");
|
||||
const rawPath = path.join(evidenceRoot, "untrusted/vulnerability-report.json");
|
||||
const sealedPath = path.join(evidenceRoot, "vulnerability-report.json");
|
||||
const nonce = Buffer.alloc(32, 0x5d);
|
||||
const child = spawnGuardian(workspace, nonce);
|
||||
const completion = waitForChild(child);
|
||||
const readyPayload = readFrame(child.stdout!);
|
||||
// The direct protocol starts the deadline once the guardian has loaded,
|
||||
// so start-up on a loaded host does not use up the lease. Only publish
|
||||
// and commit run on the real clock, inside NEAR_DEADLINE_MARGIN_MS.
|
||||
const deadlineEpochMs = await sendGuard(child, nonce, NEAR_DEADLINE_LEASE_MS);
|
||||
const ready = decodeProviderGuardianReady(
|
||||
await within(readyPayload, PROCESS_HANDSHAKE_WATCHDOG_MS, "near-deadline guardian READY"),
|
||||
nonce,
|
||||
);
|
||||
const bytes = Buffer.from('{"validated":"near-timeout"}\n');
|
||||
const handle = await open(
|
||||
path.join(evidenceRoot, ready.sealedTempLeaf),
|
||||
constants.O_WRONLY | constants.O_NOFOLLOW,
|
||||
);
|
||||
try {
|
||||
await handle.writeFile(bytes);
|
||||
await handle.chmod(0o400);
|
||||
await handle.sync();
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
// A timer can fire a millisecond before Date.now() reaches its target, so
|
||||
// wait again until it has.
|
||||
const nearDeadlineEpochMs = deadlineEpochMs - NEAR_DEADLINE_MARGIN_MS;
|
||||
while (Date.now() < nearDeadlineEpochMs) {
|
||||
await new Promise<void>((resolve) => setTimeout(resolve, nearDeadlineEpochMs - Date.now()));
|
||||
}
|
||||
expect(deadlineEpochMs - Date.now()).toBeLessThanOrEqual(NEAR_DEADLINE_MARGIN_MS);
|
||||
|
||||
await lease.publish(bytes);
|
||||
await lease.commit();
|
||||
await expect(lstat(lease.rawPath)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await expect(readFile(lease.sealedPath)).resolves.toEqual(bytes);
|
||||
}, childProcessTimeout(1_126));
|
||||
const publishedPayload = readFrame(child.stdout!);
|
||||
child.stdin!.write(encodeProviderGuardianPublish({
|
||||
nonce,
|
||||
sealedDev: ready.sealedDev,
|
||||
sealedIno: ready.sealedIno,
|
||||
size: bytes.byteLength,
|
||||
sha256: createHash("sha256").update(bytes).digest("hex"),
|
||||
}));
|
||||
decodeProviderGuardianPublished(
|
||||
await within(publishedPayload, PROCESS_HANDSHAKE_WATCHDOG_MS, "near-deadline guardian PUBLISHED"),
|
||||
nonce,
|
||||
{ dev: ready.sealedDev, ino: ready.sealedIno },
|
||||
);
|
||||
child.stdin!.end(encodeProviderGuardianCommit(nonce));
|
||||
await expect(within(completion, PROCESS_HANDSHAKE_WATCHDOG_MS, "near-deadline guardian commit EOF"))
|
||||
.resolves.toEqual({ code: 0, signal: null });
|
||||
await expect(lstat(rawPath)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await expect(readFile(sealedPath)).resolves.toEqual(bytes);
|
||||
}, childProcessTimeout(3_234));
|
||||
|
||||
it("signals guardian exit only while the provider scope latch is active", async () => {
|
||||
const lifecycle = await import("../../scripts/lib/provider-guardian-client.ts") as
|
||||
@@ -875,7 +933,7 @@ async function establishPublishedGuardian(
|
||||
const child = spawnGuardian(workspace, nonce);
|
||||
const completion = waitForChild(child);
|
||||
const readyPayload = readFrame(child.stdout!);
|
||||
await sendGuard(child, nonce, 3_000);
|
||||
await sendGuard(child, nonce, LEASE_NOT_UNDER_TEST_MS);
|
||||
const ready = decodeProviderGuardianReady(
|
||||
await within(readyPayload, PROCESS_HANDSHAKE_WATCHDOG_MS, "guardian READY"),
|
||||
nonce,
|
||||
@@ -1007,7 +1065,7 @@ function spawnGuardian(
|
||||
|
||||
/**
|
||||
* A guard carries an absolute deadline, so it goes out only once the guardian
|
||||
* reads stdin. Sent at spawn, the lease also pays for process start-up, which
|
||||
* reads stdin. Returns that deadline. Sent at spawn, the lease also pays for process start-up, which
|
||||
* has no bound on a loaded host: a 350ms lease then expires before the
|
||||
* guardian decodes the guard.
|
||||
*/
|
||||
@@ -1016,7 +1074,7 @@ async function sendGuard(
|
||||
nonce: Buffer,
|
||||
leaseMs: number,
|
||||
options: Readonly<{ closeInput?: boolean }> = {},
|
||||
): Promise<void> {
|
||||
): Promise<number> {
|
||||
const { encodeProviderGuardianGuard } = await import(
|
||||
"../../scripts/lib/provider-guardian-protocol.ts"
|
||||
);
|
||||
@@ -1025,13 +1083,11 @@ async function sendGuard(
|
||||
loaded.once("data", () => resolve());
|
||||
loaded.once("end", () => reject(new Error("guardian exited before it loaded")));
|
||||
}), PROCESS_HANDSHAKE_WATCHDOG_MS, "guardian start-up");
|
||||
const guard = encodeProviderGuardianGuard({
|
||||
kind: "vulnerability",
|
||||
nonce,
|
||||
deadlineEpochMs: Date.now() + leaseMs,
|
||||
});
|
||||
const deadlineEpochMs = Date.now() + leaseMs;
|
||||
const guard = encodeProviderGuardianGuard({ kind: "vulnerability", nonce, deadlineEpochMs });
|
||||
if (options.closeInput) child.stdin!.end(guard);
|
||||
else child.stdin!.write(guard);
|
||||
return deadlineEpochMs;
|
||||
}
|
||||
|
||||
function spawnGuardianWithPrivateModes(
|
||||
|
||||
Reference in New Issue
Block a user