From 7679ce65927d8fe7e8586bc45aa275e71fecd886 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Thu, 19 Nov 2026 16:32:00 +0900 Subject: [PATCH 1/2] =?UTF-8?q?revert:=20#55=20=EB=90=98=EB=8F=8C=EB=A6=BC?= =?UTF-8?q?(411c9338)=EC=9D=84=20=EB=90=98=EB=8F=8C=EB=A0=A4=20=EA=B7=B8?= =?UTF-8?q?=EB=A3=B9=20C=20=EA=B2=80=EC=82=AC=EA=B8=B0=20=EB=B3=80?= =?UTF-8?q?=EA=B2=BD=EC=9D=84=20=EB=8B=A4=EC=8B=9C=20=EB=84=A3=EB=8A=94?= =?UTF-8?q?=EB=8B=A4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This reverts commit 411c9338c6544590e3853d867190c02f3737486c. 문제: #55(b4895894)가 main run 376의 FE-GATE-030(browser file/storage 제거 드릴)을 깨서 411c9338(#64)로 통째로 되돌렸다. #55의 검사기 진단과 시험 빈틈 고침 12커밋이 main에 없다. 고른 방법: 되돌림을 되돌려 #55를 그대로 다시 넣고, 드릴을 깬 원인은 다음 커밋에서 따로 고친다. 리뷰어가 이 커밋이 #55와 같은지 트리로 볼 수 있다. 이 커밋의 트리는 b4895894 트리에 그 뒤 main에 들어온 #63·#65만 더한 것이다(git diff b4895894 <이 커밋>: docs/ledger 두 파일, cross-context-invalidation adapter 한 파일과 시험 두 파일). 버린 대안: #55 커밋 12개 cherry-pick. 71606ff1 merge의 충돌 해소분을 다시 풀어야 하고 결과 트리를 b4895894와 대조하기 어렵다. 위험·한계: 이 커밋만으로는 main 단계의 FE-GATE-030이 run 376처럼 실패한다(같은 트리 내용). PR 단계 게이트는 돈다. 검증: 다음 커밋 본문에 적는다. 그 커밋의 새 단언을 이 커밋의 검사기에 돌리면 실패한다(RED). Refs: #55, #64 --- docs/operations/ci-quality-gates.md | 30 +++- docs/operations/release-cache-rollback.md | 9 +- eslint.config.ts | 42 +++++ scripts/check-app-build-input.ts | 8 + scripts/check-optional-recipes.ts | 10 ++ scripts/check-request-fanout.ts | 96 ++++++++--- scripts/check-system-test-prerequisites.ts | 96 +++++++++-- scripts/check-test-evidence.ts | 46 ++++-- scripts/contracts/release-artifacts.ts | 56 ++++++- scripts/create-release-candidate.ts | 10 ++ scripts/drill-runbook.ts | 152 ++++++++++++++++-- scripts/generate-build-manifest.ts | 15 +- scripts/lib/build-manifest-outputs.ts | 49 ++++++ scripts/lib/hosting-probe.ts | 41 ++++- scripts/lib/local-release-evidence.ts | 13 +- scripts/lib/provider-cgroup.ts | 53 +++++- scripts/lib/test-evidence-artifact.ts | 2 + scripts/run-and-validate-provider.ts | 5 +- scripts/verify-documentation-readiness.ts | 38 ++++- scripts/verify-hosting-headers.ts | 2 +- scripts/verify-release.ts | 10 +- scripts/verify-reproducible-build.ts | 8 + tests/architecture/request-fanout.test.ts | 29 ++++ .../browser-capabilities/opfs-runtime.spec.ts | 23 ++- tests/e2e/preload-use.spec.ts | 19 +++ tests/e2e/service-worker-recovery.spec.ts | 25 ++- tests/e2e/support/fixture-builds.ts | 73 +++++++-- tests/e2e/support/service-worker-builds.ts | 86 +++------- .../allowed/reviewed-cursor-loop.ts | 18 +++ .../missing-forbid-only.fixture | 8 + tests/support/browser/strict-browser-test.ts | 77 ++++++--- .../browser-rpc-remediation.test.ts | 28 ++-- tests/unit/browser-spec-imports.test.ts | 51 ++++++ tests/unit/documentation-readiness.test.ts | 20 ++- tests/unit/e2e-build-helpers.test.ts | 36 +++++ tests/unit/hosting-probe.test.ts | 65 +++++++- tests/unit/route-chunks.test.ts | 74 +++++++++ tests/unit/run-isolation.test.ts | 24 +++ tests/unit/runbook-drill.test.ts | 106 +++++++++++- tests/unit/system-test-prerequisites.test.ts | 138 ++++++++++++++-- tests/unit/test-evidence-source-rules.test.ts | 44 ++++- tests/visual/scrollbar-gutter.visual.spec.ts | 4 +- 42 files changed, 1471 insertions(+), 268 deletions(-) create mode 100644 tests/fixtures/architecture/request-fanout/allowed/reviewed-cursor-loop.ts create mode 100644 tests/fixtures/test-evidence/playwright-config/missing-forbid-only.fixture create mode 100644 tests/unit/browser-spec-imports.test.ts create mode 100644 tests/unit/e2e-build-helpers.test.ts create mode 100644 tests/unit/route-chunks.test.ts diff --git a/docs/operations/ci-quality-gates.md b/docs/operations/ci-quality-gates.md index e39b888c..5fabe8af 100644 --- a/docs/operations/ci-quality-gates.md +++ b/docs/operations/ci-quality-gates.md @@ -230,6 +230,15 @@ namespace 점검을 돌아, 이 조건이 없는 runner에서 이름과 host 설 stderr, 위 sysctl 값, 고치는 방법을 적고 exit 1로 멈춘다. 전에는 이 검사가 PASS를 낸 뒤 provider 테스트가 bwrap 원문 오류로 실패했다(2026-11-05 desktop에서 114건 중 18건). +- 같은 검사가 provider의 cgroup 준비도 provider와 같은 호출로 해 본다. + `/usr/bin/prlimit`·`/usr/bin/systemd-run`이 실행 가능한지 보고, + `systemctl --user show --property=Version --value`로 user manager와 버전(254 + 이상)을 확인한 뒤, provider scope와 같은 속성·같은 cgroup 강제 확인으로 + `systemd-run --user --scope`에서 `/bin/true`를 한 번 띄운다 + (`scripts/lib/provider-cgroup.ts`). user bus가 없거나, 버전이 낮거나, memory·pids·cpu + controller가 위임되지 않은 호스트는 그 원인을 이름으로 대며 exit 1이다. 전에는 + system manager bus만 보고 PASS를 낸 뒤 provider 테스트가 하나씩 + `provider cgroup unavailable: ...`로 실패했다. - 고치는 방법은 `scripts/check-system-test-prerequisites.ts`가 실패할 때 출력한다. 호스트의 보안 설정을 바꾸는 일이므로 호스트 소유자가 고른다. 호스트를 바꾸는 설정 파일과 명령은 runner를 만드는 쪽의 인프라 repo가 갖는다. @@ -466,7 +475,18 @@ provider 확인이 필요하다고 표시한 것은 모두 `UNVERIFIED`다. `FAI drill은 `FAIL`(exit 1)이다. 그렇지 않고 `UNVERIFIED`가 하나라도 있으면 `UNVERIFIED`(exit 3)이고, 게이트는 `PASS`가 아니다. 그래서 provider 증거가 없으면 production 단계는 닫히지 않는다. 의도한 결과다. 로컬 drill은 승격한 release를 rollback할 -수 있다는 것을 보여 줄 수 없다. FE-GATE-016(`rollback-drill`)은 FE-GATE-025와 같은 +수 있다는 것을 보여 줄 수 없다. + +provider 증거는 `--provider-evidence <파일>`로 넘긴다(0929/11-02). 파일은 +`providerDrillEvidenceSchema`(`scripts/contracts/release-artifacts.ts`) 꼴이다: runbook ID, +release ID, 관찰 시각, provider 쪽 run ID, 그리고 항목마다 `PASS`/`FAIL`과 근거다. 항목 +이름은 `UNVERIFIED` 줄이 적는 이름(`trigger`, recovery assertion 이름, `provider +verification` 등)이다. drill은 runbook과 release가 같고, 관찰이 7일 안이고, 로컬 drill이 +보지 못한 항목만 답한 증거만 받는다. 받은 증거는 그 항목의 `UNVERIFIED`만 바꾸고, 로컬에서 +본 `PASS`·`FAIL`은 그대로 둔다. 모든 항목이 답을 얻고 `FAIL`이 없을 때만 drill이 `PASS`다. +받지 않은 증거는 이유와 함께 record의 `providerEvidence`에 남고 drill은 `UNVERIFIED`다. +템플릿의 production 게이트 명령은 아직 이 인자를 넘기지 않는다. provider를 연결한 제품이 +증거 파일을 만들고 drill 명령에 넘겨야 그 단계가 닫힌다. FE-GATE-016(`rollback-drill`)은 FE-GATE-025와 같은 FE-RB-005 drill을 돌렸으므로 폐기했다(`scripts/contracts/ci-gates.ts`의 `RETIRED_GATE_IDS`). @@ -505,6 +525,14 @@ gate의 realtime source/fixture 검사는 native SSE/WebSocket/Web Push API가 bootstrap/installed-feature 조합을 차단한다. optional recipe source gate는 uncomposed runtime 전체를 tree-shaking 없이 합성해 catalog의 gzip 예산도 blocking으로 검증한다. +같은 gate의 `check:request-fanout`은 `src/features`·`src/application`에서 루프나 배열 +순회 callback 안의 요청(`fetch()`, 이름이 transport인 receiver의 method)을 거절한다. +실패 줄은 두 탈출 방법을 함께 적는다: 기능 `adapters/`에서 +`createCursorPaginationRuntime`(`src/adapters/query-cache`)으로 페이지를 받거나, 상한이 +있고 검토한 반복이면 그 호출 줄에 `// fanout-reviewed: <이유>`를 적는다. 결과 줄은 +주석으로 넘긴 호출의 수와 위치를 적는다. 이름이 다른 receiver(`repository.load()`), +루프가 부르는 이름 있는 함수 안의 요청, 재귀, 순회 전에 만든 promise는 보지 못한다 +(`scripts/check-request-fanout.ts` 머리 주석). `FE-GATE-030`(main 단계)은 reference runtime source·전용 test·catalog metadata를 제거한 임시 repository에서도 base typecheck, architecture, test, build가 통과하는지 검증해 skeleton의 선택성을 유지한다. browser file/storage와 realtime runtime은 diff --git a/docs/operations/release-cache-rollback.md b/docs/operations/release-cache-rollback.md index f9377e17..79906c02 100644 --- a/docs/operations/release-cache-rollback.md +++ b/docs/operations/release-cache-rollback.md @@ -130,9 +130,12 @@ probe하지 않고 `UNVERIFIED`로 끝난다. 슬롯을 채우는 방법은 **주소와 연결** -1. 앱은 HTTPS origin의 루트(`/`)에서 내보낸다. 인증서가 유효하고, 어느 surface도 그 - origin 밖으로 redirect하지 않는다. — FE-GATE-019가 잰다. `HOSTING_BASE_URL`은 경로 없는 - HTTPS 루트 URL이어야 하고, surface마다 최종 origin을 본다. +1. 앱은 HTTPS origin의 루트(`/`)나 빌드한 하위 경로(`VITE_ROUTER_BASE_PATH`, 예 `/app/`)에서 + 내보낸다. 인증서가 유효하고, 어느 surface도 그 origin 밖으로 redirect하지 않는다. — + FE-GATE-019가 잰다. `HOSTING_BASE_URL`은 release의 기준 URL(루트 또는 `/`로 끝나는 그 + 하위 경로, query·fragment 없음)이다. probe는 surface 경로(`config/hosting/cache-policy.json`)를 + 그 아래에서 요청하고, 후보의 진입 script가 그 경로 아래에 있는지와 surface마다 최종 + origin을 본다. 2. TLS에서 HTTP/2를 맺는다(ALPN `h2`). — FE-GATE-019가 잰다(`transport.alpnProtocols`). **surface별 응답** (surface: probe가 요청하는 경로 하나) diff --git a/eslint.config.ts b/eslint.config.ts index fbdcd82a..f0079032 100644 --- a/eslint.config.ts +++ b/eslint.config.ts @@ -937,6 +937,48 @@ export default [ "react-hooks/rules-of-hooks": "off", }, }, + { + // 1112/03-04. A browser spec takes `test` from the strict fixture, which + // fails it on console errors, page errors, failed requests and unused + // preloads. FE-GATE-008 used to look for the fixture's path as text in + // tests/e2e only, so a path in a comment passed and the other browser + // folders were not looked at. tests/unit/browser-spec-imports.test.ts + // lints a fixture spec in each folder against this entry. + files: [`tests/{e2e,browser-capabilities,visual,storybook}/**/*.spec.ts`], + rules: { + "no-restricted-imports": [ + "error", + { + paths: [ + { + name: "@playwright/test", + importNames: ["test", "default"], + message: + "import test from tests/support/browser/strict-browser-test.ts; the strict fixture fails a spec on browser errors (.claude/skills/fe-qa).", + }, + ], + patterns: [ + ...deprecatedAliasImports, + { + group: ["**/support/browser/browser-test.ts", "**/support/browser/browser-test"], + importNames: ["test"], + message: + "import test from tests/support/browser/strict-browser-test.ts; the base test does not check browser errors.", + }, + ], + }, + ], + }, + }, + { + // 1112/03-04. The two specs that need the plain test on purpose: + // preload-use proves the strict fixture's own record, and chunk-load-error + // provokes the 404 a missing chunk gives to classify the engine's wording. + files: ["tests/e2e/preload-use.spec.ts", "tests/browser-capabilities/chunk-load-error.spec.ts"], + rules: { + "no-restricted-imports": ["error", { patterns: deprecatedAliasImports }], + }, + }, { files: [ `tests/fixtures/browser-file-storage-boundaries/forbidden/**/*.${sourceExtensions}`, diff --git a/scripts/check-app-build-input.ts b/scripts/check-app-build-input.ts index 9dc7d023..9193106a 100644 --- a/scripts/check-app-build-input.ts +++ b/scripts/check-app-build-input.ts @@ -5,6 +5,7 @@ import { createFileSystemDeclarationSource, type TopologyDeclarationSource, } from "./generate-product-feature-index.ts"; +import { checkoutBuildReadFailure } from "./lib/run-isolation.ts"; /** * scale-architecture.md §3 / §37. Build-input regression gate for deployable-app isolation. @@ -123,6 +124,13 @@ async function main(): Promise { // without `--require-bundle` the gate checks the generated indexes only, so a // stale inventory from another build can never pass or fail it. const requireBundle = process.argv.includes("--require-bundle"); + // 1106/12-02. The inventory is read where the checkout keeps it; under + // RUN_OUTPUT_ROOT the build wrote elsewhere and this would judge an earlier one. + const outputRootFailure = requireBundle ? checkoutBuildReadFailure("scripts/check-app-build-input.ts") : undefined; + if (outputRootFailure !== undefined) { + process.stderr.write(`${outputRootFailure}\n`); + process.exit(1); + } const report = await inspectAppBuildInput(process.cwd(), requireBundle ? {} : { bundleInventory: null }); if (requireBundle && !report.bundle.checked) { process.stderr.write("app build input: no production module inventory; run the build first\n"); diff --git a/scripts/check-optional-recipes.ts b/scripts/check-optional-recipes.ts index 8a7206f4..8c3f556d 100644 --- a/scripts/check-optional-recipes.ts +++ b/scripts/check-optional-recipes.ts @@ -11,6 +11,7 @@ import { validateRecipeCatalog, } from "./lib/optional-recipes.ts"; import { assertMatchesJsonSchema } from "./lib/json-schema.ts"; +import { checkoutBuildReadFailure } from "./lib/run-isolation.ts"; import { writeValidatedJsonArtifact } from "./lib/validated-json-artifact.ts"; type GateViolation = Readonly<{ @@ -36,6 +37,15 @@ const catalogPath = argument( ); const sourceRoot = argument("--source-root", "src"); const distRoot = argument("--dist-root", "dist"); +// 1106/12-02. Without --dist-root this scans the checkout's dist; under +// RUN_OUTPUT_ROOT the build wrote elsewhere, so it would judge an earlier build. +const outputRootFailure = process.argv.includes("--dist-root") + ? undefined + : checkoutBuildReadFailure("scripts/check-optional-recipes.ts"); +if (outputRootFailure !== undefined) { + process.stderr.write(`${outputRootFailure}\n`); + process.exit(1); +} const artifactPath = argument( "--artifact", "artifacts/quality/optional-recipes.json", diff --git a/scripts/check-request-fanout.ts b/scripts/check-request-fanout.ts index 3fcbe287..1a994599 100644 --- a/scripts/check-request-fanout.ts +++ b/scripts/check-request-fanout.ts @@ -17,6 +17,16 @@ import { parseAsync } from "@babel/core"; * The HTTP runtime itself (src/adapters/http) owns its bounded retry loop and is * out of scope. A reviewed exception is written on the call's line as * `// fanout-reviewed: `. + * + * 0929/12-01. A failure names both ways out: fetch page after page through + * `createCursorPaginationRuntime` (src/adapters/query-cache) assembled in the + * feature's `adapters/`, or mark a bounded, reviewed repetition on the call's + * line. Every reviewed exception is counted and listed in the result line. + * + * What it does not see: a dispatch through a receiver named otherwise + * (`repository.load()`, a destructured `get`), a dispatch in a named function + * that a loop calls, recursion, and promises built before an iteration and + * awaited inside it. Reviews still look for those. */ export const REQUEST_FANOUT_ROOTS = Object.freeze(["src/features", "src/application"]); @@ -44,6 +54,14 @@ const FUNCTION_TYPES = new Set(["ArrowFunctionExpression", "FunctionExpression"] type Node = { type: string; loc?: { start: { line: number } } } & Record; export type FanoutViolation = Readonly<{ file: string; line: number; call: string; context: string }>; +export type FanoutReviewedException = Readonly<{ file: string; line: number; call: string; reason: string }>; + +const REVIEWED = /fanout-reviewed:\s*(\S.*?)\s*$/u; + +/** How to fix a failure; both ways the source gate accepts. */ +export const FANOUT_REMEDY = + "load one page per request with createCursorPaginationRuntime (src/adapters/query-cache) assembled in the " + + "feature's adapters/, or, for a bounded repetition someone reviewed, write `// fanout-reviewed: ` on the call's line"; function receiverName(node: Node | undefined): string | null { if (!node) return null; @@ -81,6 +99,14 @@ export async function findRequestFanout( file: string, sourceText: string, ): Promise { + return (await inspectRequestFanout(file, sourceText)).violations; +} + +/** Per-row dispatches, and the ones a `fanout-reviewed:` comment let through. */ +export async function inspectRequestFanout( + file: string, + sourceText: string, +): Promise> { const plugins: ("jsx" | ["typescript", { dts: boolean }])[] = [["typescript", { dts: false }]]; if (file.endsWith(".tsx")) plugins.push("jsx"); const tree = (await parseAsync(sourceText, { @@ -93,6 +119,7 @@ export async function findRequestFanout( if (!tree) throw new Error(`cannot parse ${file}`); const lines = sourceText.split("\n"); const violations: FanoutViolation[] = []; + const reviewed: FanoutReviewedException[] = []; const visit = (node: Node, parent: Node | undefined, repeating: string | null): void => { let context = repeating; @@ -106,9 +133,9 @@ export async function findRequestFanout( if (node.type === "CallExpression" && context) { const call = dispatchDescription(node); const line = node.loc?.start.line ?? 0; - if (call && !/fanout-reviewed:\s*\S/u.test(lines[line - 1] ?? "")) { - violations.push({ file, line, call, context }); - } + const reason = REVIEWED.exec(lines[line - 1] ?? "")?.[1]; + if (call && reason !== undefined) reviewed.push({ file, line, call, reason }); + else if (call) violations.push({ file, line, call, context }); } for (const [key, value] of Object.entries(node)) { if (key === "loc" || key === "start" || key === "end" || key === "extra") continue; @@ -121,7 +148,7 @@ export async function findRequestFanout( } }; visit(tree, undefined, null); - return violations; + return { violations, reviewed }; } async function sourceFiles(directory: string): Promise { @@ -143,39 +170,62 @@ async function sourceFiles(directory: string): Promise { export async function scanRequestFanout( root: string, roots: readonly string[] = REQUEST_FANOUT_ROOTS, -): Promise> { +): Promise> { const files = (await Promise.all(roots.map((entry) => sourceFiles(path.join(root, entry))))).flat(); const violations: FanoutViolation[] = []; + const reviewed: FanoutReviewedException[] = []; for (const file of files) { - violations.push( - ...(await findRequestFanout(path.relative(root, file), await readFile(file, "utf8"))), - ); + const found = await inspectRequestFanout(path.relative(root, file), await readFile(file, "utf8")); + violations.push(...found.violations); + reviewed.push(...found.reviewed); } - return { files: files.length, violations }; + return { files: files.length, violations, reviewed }; +} + +export function formatFanoutViolation(violation: FanoutViolation): string { + return `FAIL ${violation.file}:${violation.line} ${violation.call} inside ${violation.context}: ${FANOUT_REMEDY}`; +} + +function formatReviewed(reviewed: readonly FanoutReviewedException[]): string { + if (reviewed.length === 0) return "0 reviewed exceptions"; + return `${reviewed.length} reviewed exception${reviewed.length === 1 ? "" : "s"} (` + + `${reviewed.map((entry) => `${entry.file}:${entry.line} ${entry.reason}`).join("; ")})`; +} + +/** + * The forbidden fixture must be rejected; the allowed fixture must pass, with + * its reviewed cursor loop counted as a reviewed exception. + */ +export async function checkRequestFanoutFixtures(root: string): Promise> { + const forbidden = await scanRequestFanout(root, ["tests/fixtures/architecture/request-fanout/forbidden"]); + const allowed = await scanRequestFanout(root, ["tests/fixtures/architecture/request-fanout/allowed"]); + const ok = forbidden.violations.length >= forbidden.files && forbidden.files > 0 && + allowed.violations.length === 0 && allowed.reviewed.length > 0; + return { + ok, + line: `request fan-out fixtures: forbidden ${forbidden.violations.length} rejected in ${forbidden.files} files, ` + + `allowed ${allowed.violations.length} violations and ${formatReviewed(allowed.reviewed)} ${ok ? "PASS" : "FAIL"}`, + }; } async function main(): Promise { const fixture = process.argv.includes("--fixture"); const root = process.cwd(); if (fixture) { - // The forbidden fixture must be rejected; the allowed fixture must pass. - const forbidden = await scanRequestFanout(root, ["tests/fixtures/architecture/request-fanout/forbidden"]); - const allowed = await scanRequestFanout(root, ["tests/fixtures/architecture/request-fanout/allowed"]); - const ok = forbidden.violations.length >= forbidden.files && forbidden.files > 0 && allowed.violations.length === 0; - process.stdout.write( - `request fan-out fixtures: forbidden ${forbidden.violations.length} rejected in ${forbidden.files} files, allowed ${allowed.violations.length} violations ${ok ? "PASS" : "FAIL"}\n`, - ); - if (!ok) process.exitCode = 1; + const result = await checkRequestFanoutFixtures(root); + process.stdout.write(`${result.line}\n`); + if (!result.ok) process.exitCode = 1; return; } const result = await scanRequestFanout(root); - for (const violation of result.violations) { - process.stderr.write( - `FAIL ${violation.file}:${violation.line} ${violation.call} inside ${violation.context}\n`, - ); - } + for (const violation of result.violations) process.stderr.write(`${formatFanoutViolation(violation)}\n`); process.stdout.write( - `request fan-out: ${result.files} files scanned, ${result.violations.length} per-row dispatches ${result.violations.length === 0 ? "PASS" : "FAIL"}\n`, + `request fan-out: ${result.files} files scanned, ${result.violations.length} per-row dispatches, ` + + `${formatReviewed(result.reviewed)} ${result.violations.length === 0 ? "PASS" : "FAIL"}\n`, ); if (result.violations.length > 0) process.exitCode = 1; } diff --git a/scripts/check-system-test-prerequisites.ts b/scripts/check-system-test-prerequisites.ts index 9a355026..96c4a52d 100644 --- a/scripts/check-system-test-prerequisites.ts +++ b/scripts/check-system-test-prerequisites.ts @@ -1,9 +1,16 @@ +import { spawnSync } from "node:child_process"; +import { randomBytes } from "node:crypto"; import { constants } from "node:fs"; import { access, readFile } from "node:fs/promises"; -import { spawnSync } from "node:child_process"; import path from "node:path"; import { fileURLToPath } from "node:url"; +import { + formatProviderPrerequisiteUnitName, + MINIMUM_PROVIDER_SYSTEMD_VERSION, + systemdRunPrerequisiteProbeArguments, + USER_MANAGER_VERSION_ARGUMENTS, +} from "./lib/provider-cgroup.ts"; import { PROVIDER_SANDBOX_NAMESPACE_ARGUMENTS } from "./lib/provider-sandbox-namespaces.ts"; /** What the check reads from the host; a test passes a fake. */ @@ -15,7 +22,9 @@ export type SystemTestHost = Readonly<{ command: string, args: readonly string[], timeoutMs: number, - ): Readonly<{ status: number | null; stderr: string; error?: Error | undefined }>; + ): Readonly<{ status: number | null; stdout?: string; stderr: string; error?: Error | undefined }>; + /** Names the provider-shaped scope the probe starts; a test passes a fixed one. */ + probeUnitName?: () => string; }>; export const nodeSystemTestHost: SystemTestHost = Object.freeze({ @@ -23,12 +32,18 @@ export const nodeSystemTestHost: SystemTestHost = Object.freeze({ readFile: async (target: string) => readFile(target, "utf8"), spawn: (command: string, args: readonly string[], timeoutMs: number) => { const result = spawnSync(command, [...args], { encoding: "utf8", timeout: timeoutMs }); - return { status: result.status, stderr: result.stderr ?? "", error: result.error }; + return { status: result.status, stdout: result.stdout ?? "", stderr: result.stderr ?? "", error: result.error }; }, + probeUnitName: defaultProbeUnitName, }); +function defaultProbeUnitName(): string { + return formatProviderPrerequisiteUnitName(process.pid, randomBytes(12).toString("hex")); +} + const BWRAP = "/usr/bin/bwrap"; const SYSTEMCTL = "/usr/bin/systemctl"; +const SYSTEMD_RUN = "/usr/bin/systemd-run"; /** * 1105/12-01. The provider sandbox's own namespace options on /bin/true. The @@ -50,6 +65,7 @@ const USER_NAMESPACE_SETTINGS = Object.freeze([ ] as const); const PROBE_TIMEOUT_MS = 5_000; +const SCOPE_PROBE_TIMEOUT_MS = 15_000; const MAX_REPORTED_STDERR_LINES = 4; export async function findSystemTestPrerequisiteFailures( @@ -59,6 +75,8 @@ export async function findSystemTestPrerequisiteFailures( const executable = new Map(); for (const [target, label] of [ [BWRAP, "bubblewrap"], + ["/usr/bin/prlimit", "prlimit"], + [SYSTEMD_RUN, "systemd-run"], [SYSTEMCTL, "systemctl"], ["/usr/bin/tar", "tar"], ] as const) { @@ -83,10 +101,18 @@ export async function findSystemTestPrerequisiteFailures( failures.push("cgroup v2 is required at /sys/fs/cgroup/cgroup.controllers"); } + // 1112/08-03. The provider asks the user manager (not the system manager) + // for its version and then starts its sandbox in a `systemd-run --user + // --scope` (scripts/run-and-validate-provider.ts). The check makes the same + // calls, so a host without a user bus, with an old manager, or without + // delegated controllers fails here by name instead of in each provider test. if (executable.get(SYSTEMCTL)) { - const probe = host.spawn(SYSTEMCTL, ["show-environment"], 3_000); - if (probe.error || probe.status !== 0) { - failures.push("a reachable systemd manager bus is required"); + const userManager = findUserManagerFailure(host); + if (userManager !== undefined) { + failures.push(userManager); + } else if (executable.get(SYSTEMD_RUN)) { + const scope = findProviderScopeFailure(host); + if (scope !== undefined) failures.push(scope); } } @@ -101,6 +127,52 @@ export async function findSystemTestPrerequisiteFailures( return failures; } +function findUserManagerFailure(host: SystemTestHost): string | undefined { + const probe = host.spawn(SYSTEMCTL, USER_MANAGER_VERSION_ARGUMENTS, 3_000); + const command = `${SYSTEMCTL} ${USER_MANAGER_VERSION_ARGUMENTS.join(" ")}`; + if (probe.error || probe.status !== 0) { + return [ + `a reachable systemd user manager is required (${command} ${outcomeOf(probe)}).`, + ...stderrLines(probe.stderr, "systemctl"), + " The provider runs in a `systemd-run --user --scope`; give the runner user a user", + " manager (for example `loginctl enable-linger `) and a user bus in XDG_RUNTIME_DIR.", + ].join("\n"); + } + const reported = (probe.stdout ?? "").trim(); + const match = /^([0-9]+)/u.exec(reported); + const version = match ? Number(match[1]) : Number.NaN; + if (!Number.isSafeInteger(version) || version < MINIMUM_PROVIDER_SYSTEMD_VERSION) { + return `systemd user manager ${MINIMUM_PROVIDER_SYSTEMD_VERSION} or newer is required (${command} reported ${JSON.stringify(reported)})`; + } + return undefined; +} + +function findProviderScopeFailure(host: SystemTestHost): string | undefined { + const args = systemdRunPrerequisiteProbeArguments((host.probeUnitName ?? defaultProbeUnitName)()); + const probe = host.spawn(SYSTEMD_RUN, args, SCOPE_PROBE_TIMEOUT_MS); + if (!probe.error && probe.status === 0) return undefined; + return [ + "systemd-run could not start the provider's cgroup scope around /bin/true " + + `(${SYSTEMD_RUN} --user --scope ... ${outcomeOf(probe)}).`, + ...stderrLines(probe.stderr, "systemd-run"), + " The scope needs the memory, pids and cpu controllers delegated to the user manager", + " (Delegate= in user@.service) and cgroup v2. Runner requirements: docs/operations/ci-quality-gates.md.", + ].join("\n"); +} + +function outcomeOf(probe: ReturnType): string { + return probe.error ? `did not finish: ${probe.error.message}` : `exited ${String(probe.status)}`; +} + +function stderrLines(stderr: string, label: string): string[] { + const lines = stderr + .split("\n") + .map((line) => line.trim()) + .filter((line) => line.length > 0) + .slice(0, MAX_REPORTED_STDERR_LINES); + return lines.length > 0 ? lines.map((line) => ` ${label} stderr: ${line}`) : [` ${label} stderr: (empty)`]; +} + /** * 1101/13-03. Whether this host lets the provider sandbox create its * namespaces, and if not, why and what to do. The release provider jobs run @@ -125,19 +197,11 @@ async function namespaceFailure( // Absent on this kernel; the others still say what the host allows. } } - const stderr = probe.stderr - .split("\n") - .map((line) => line.trim()) - .filter((line) => line.length > 0) - .slice(0, MAX_REPORTED_STDERR_LINES); - const outcome = probe.error - ? `did not finish: ${probe.error.message}` - : `exited ${String(probe.status)}`; return [ "this host does not allow unprivileged user and network namespaces, which the provider " + `sandbox needs (bwrap ${PROVIDER_SANDBOX_NAMESPACE_ARGUMENTS.join(" ")}).`, - ` probe: ${BWRAP} ${NAMESPACE_PROBE_ARGUMENTS.join(" ")} ${outcome}`, - ...(stderr.length > 0 ? stderr.map((line) => ` bwrap stderr: ${line}`) : [" bwrap stderr: (empty)"]), + ` probe: ${BWRAP} ${NAMESPACE_PROBE_ARGUMENTS.join(" ")} ${outcomeOf(probe)}`, + ...stderrLines(probe.stderr, "bwrap"), ...(settings.length > 0 ? [` host settings: ${settings.join(", ")}`] : []), " To fix, do one of these:", " - give /usr/bin/bwrap an AppArmor profile that allows `userns` (Ubuntu 23.10 and later),", diff --git a/scripts/check-test-evidence.ts b/scripts/check-test-evidence.ts index 6b16b483..4c9002b2 100644 --- a/scripts/check-test-evidence.ts +++ b/scripts/check-test-evidence.ts @@ -64,6 +64,12 @@ const policyPath = argumentValue( "config/testing/test-evidence.json", ); const explicitCatalogPath = optionalArgumentValue("--catalog"); +// 1112/03-03. The forbidOnly rule read only the repository's configs and ran +// only in the full check, so no fixture could show it still fires. +// `--playwright-config ` (repeatable) checks those files in any mode. +const explicitPlaywrightConfigs = process.argv.flatMap((argument, index) => + argument === "--playwright-config" && process.argv[index + 1] ? [process.argv[index + 1]!] : [], +); const explicitReceiptPath = optionalArgumentValue("--receipt"); const fixtureMode = sourceRoot !== "tests"; const sourceOnly = process.argv.includes("--source-only"); @@ -78,6 +84,7 @@ const facts = { sharedScenarios: 0, declaredScenarioExecutions: 0, executedScenarioExecutions: 0, + playwrightConfigs: [] as string[], }; async function filesBelow(target: string): Promise { @@ -365,6 +372,26 @@ if (parsedPolicy) { } } +async function missingForbidOnly(configs: readonly string[]): Promise { + const missing: string[] = []; + for (const config of configs) { + if (!(await readFile(config, "utf8")).includes("forbidOnly: true")) { + missing.push(`${config}: missing forbidOnly: true, so a committed test.only passes the gate`); + } + } + return missing; +} + +// The repository's configs unless the run names others; every mode but the +// scenario check reads them, and the report lists what it read. +if (!scenarioOnly) { + // playwright.dev.config.ts spreads playwright.config.ts. + facts.playwrightConfigs = explicitPlaywrightConfigs.length > 0 + ? explicitPlaywrightConfigs + : [...selectRepositoryPaths("playwright-configs")]; + failures.push(...(await missingForbidOnly(facts.playwrightConfigs))); +} + if (!fixtureMode && !scenarioOnly) { const e2eConfig = await readFile("playwright.config.ts", "utf8"); for (const token of [ @@ -381,22 +408,9 @@ if (!fixtureMode && !scenarioOnly) { failures.push(`playwright.config.ts missing release evidence token ${token}`); } } - // playwright.dev.config.ts spreads playwright.config.ts. - for (const config of selectRepositoryPaths("playwright-configs")) { - if (!(await readFile(config, "utf8")).includes("forbidOnly: true")) { - failures.push(`${config}: missing forbidOnly: true, so a committed test.only passes the gate`); - } - } - - const e2eFiles = (await filesBelow("tests/e2e")).filter((file) => - /\.spec\.ts$/.test(file), - ); - for (const file of e2eFiles) { - const source = await readFile(file, "utf8"); - if (!source.includes("support/browser/strict-browser-test")) { - failures.push(`${file}: bypasses strict browser fixture`); - } - } + // 1112/03-04. That a browser spec takes the strict fixture's `test` is an + // ESLint rule now (eslint.config.ts, tests/unit/browser-spec-imports.test.ts), + // which reads the import instead of looking for the path as text. const baselineFiles = (await filesBelow("tests/visual/__snapshots__")).filter( (file) => file.endsWith(".png"), diff --git a/scripts/contracts/release-artifacts.ts b/scripts/contracts/release-artifacts.ts index 13d30bdd..2b98f9df 100644 --- a/scripts/contracts/release-artifacts.ts +++ b/scripts/contracts/release-artifacts.ts @@ -746,16 +746,52 @@ const drillVerdict = z.enum(["PASS", "FAIL", "UNVERIFIED"]); export type DrillVerdict = z.infer; -/** FAIL beats UNVERIFIED beats PASS; provider scope alone keeps a drill UNVERIFIED. */ +/** + * FAIL beats UNVERIFIED beats PASS. A drill with provider scope stays + * UNVERIFIED until accepted provider evidence says the provider side passed + * (0929/11-02). + */ export function combineDrillVerdicts( verdicts: readonly DrillVerdict[], providerVerificationRequired: boolean, + providerVerification: DrillVerdict = "UNVERIFIED", ): DrillVerdict { - if (verdicts.includes("FAIL")) return "FAIL"; - if (verdicts.includes("UNVERIFIED") || providerVerificationRequired) return "UNVERIFIED"; + const all = providerVerificationRequired ? [...verdicts, providerVerification] : verdicts; + if (all.includes("FAIL")) return "FAIL"; + if (all.includes("UNVERIFIED")) return "UNVERIFIED"; return "PASS"; } +/** + * 0929/11-02. What the hosting provider observed for one drill of one + * release: the provider-side items a local drill cannot see (a smoke after the + * switch, live traffic, the trigger), named as the drill's UNVERIFIED line + * names them, plus `provider verification` itself. + */ +export const providerDrillEvidenceSchema = z + .object({ + schemaVersion: z.literal(1), + artifactType: z.literal("provider-drill-evidence"), + runbookId: z.string().regex(/^FE-RB-00[1-5]$/u), + releaseId: nonEmptyString, + observedAt: timestamp, + runId: nonEmptyString, + observations: z + .array( + z + .object({ item: nonEmptyString, result: z.enum(["PASS", "FAIL"]), evidence: nonEmptyString }) + .strict(), + ) + .min(1) + .max(32) + .refine((items) => new Set(items.map(({ item }) => item)).size === items.length, { + message: "must name each item once", + }), + }) + .strict(); + +export type ProviderDrillEvidence = z.infer; + export const runbookRecordArtifactSchema = z .object({ schemaVersion: z.literal(2), @@ -778,6 +814,19 @@ export const runbookRecordArtifactSchema = z negativeFixtureFailedAsExpected: drillVerdict, windowObservedBucket: nonEmptyString, providerVerificationRequired: z.boolean(), + providerEvidence: z + .object({ + accepted: z.boolean(), + reason: nonEmptyString.optional(), + runId: nonEmptyString.optional(), + observedAt: timestamp.optional(), + providerVerification: drillVerdict, + }) + .strict() + .refine((evidence) => evidence.accepted || (evidence.reason !== undefined && evidence.providerVerification === "UNVERIFIED"), { + message: "rejected provider evidence must say why and verify nothing", + }) + .optional(), result: drillVerdict, }) .strict() @@ -791,6 +840,7 @@ export const runbookRecordArtifactSchema = z record.negativeFixtureFailedAsExpected, ], record.providerVerificationRequired, + record.providerEvidence?.providerVerification, ); if (record.result !== expected) { context.addIssue({ diff --git a/scripts/create-release-candidate.ts b/scripts/create-release-candidate.ts index 7ac2e792..cd618d1b 100644 --- a/scripts/create-release-candidate.ts +++ b/scripts/create-release-candidate.ts @@ -1,5 +1,6 @@ import { mkdir } from "node:fs/promises"; +import { checkoutBuildReadFailure } from "./lib/run-isolation.ts"; import { writeValidatedJsonArtifact } from "./lib/validated-json-artifact.ts"; import { localEvidenceAssessmentArtifactSchema } from "./contracts/release-artifacts.ts"; import { createLocalEvidenceAssessment } from "./lib/local-release-evidence.ts"; @@ -10,6 +11,15 @@ import { releaseCandidateManifestSchema, } from "./lib/release-candidate.ts"; +// 1106/12-02. This digests dist and the evidence under artifacts/ where the +// checkout keeps them (scripts/lib/release-candidate.ts). Under RUN_OUTPUT_ROOT +// the build wrote elsewhere, so the candidate would package an earlier build. +const outputRootFailure = checkoutBuildReadFailure("scripts/create-release-candidate.ts"); +if (outputRootFailure !== undefined) { + process.stderr.write(`${outputRootFailure}\n`); + process.exit(1); +} + const assessment = await createLocalEvidenceAssessment(); await mkdir("artifacts/security", { recursive: true }); await writeValidatedJsonArtifact({ diff --git a/scripts/drill-runbook.ts b/scripts/drill-runbook.ts index c1b03c2d..c610a8a8 100644 --- a/scripts/drill-runbook.ts +++ b/scripts/drill-runbook.ts @@ -22,8 +22,10 @@ import { projectTelemetryEvent } from "../src/contracts/telemetry.ts"; import { EXPECTED_CONTRACT_SET_PACKAGES } from "../src/features/installed-contract-contributions.ts"; import { combineDrillVerdicts, + providerDrillEvidenceSchema, runbookRecordArtifactSchema, type DrillVerdict, + type ProviderDrillEvidence, } from "./contracts/release-artifacts.ts"; import { verifyReleaseRuntimeCoherence } from "./lib/release-runtime-coherence.ts"; import { writeValidatedJsonArtifact } from "./lib/validated-json-artifact.ts"; @@ -87,8 +89,25 @@ export type DrillOptions = Readonly<{ */ releaseRoot?: string; contractPackages?: readonly InstalledContractPackageIdentity[]; + /** + * 0929/11-02. The provider's record for this drill (`--provider-evidence`), + * parsed JSON. Without it a drill with provider scope cannot pass. + */ + providerEvidence?: unknown; + /** The clock evidence age is judged against; a test passes a fixed one. */ + now?: Date; }>; +/** Provider evidence older than this no longer describes the provider as it is. */ +export const PROVIDER_EVIDENCE_MAX_AGE_MS = 7 * 24 * 60 * 60 * 1_000; +/** How far ahead of this clock a provider's timestamp may be. */ +const PROVIDER_EVIDENCE_CLOCK_SKEW_MS = 5 * 60 * 1_000; +/** The item names a drill's UNVERIFIED line uses, and provider evidence answers. */ +const ITEM_TRIGGER = "trigger"; +const ITEM_CONTAINMENT = "containment"; +const ITEM_NEGATIVE_FIXTURE = "negative fixture"; +export const ITEM_PROVIDER_VERIFICATION = "provider verification"; + type DrillContext = Readonly<{ root: string; contractPackages: readonly InstalledContractPackageIdentity[]; @@ -490,14 +509,29 @@ export async function runRunbookDrill(runbookId: string, options: DrillOptions = const release = parseReleaseArtifact( await rootedReader(context.root)(DEFAULT_ROLLBACK_PATHS.release), ); - const result = await drill(context); + const local = await drill(context); + const now = options.now ?? new Date(); + const judged = options.providerEvidence === undefined + ? undefined + : judgeProviderEvidence(options.providerEvidence, runbookId, release.releaseId, openItems(local), now); + const result = judged?.accepted ? applyProviderEvidence(local, judged.evidence) : local; + const providerEvidence = judged === undefined + ? undefined + : judged.accepted + ? { + accepted: true as const, + runId: judged.evidence.runId, + observedAt: judged.evidence.observedAt, + providerVerification: observed(judged.evidence, ITEM_PROVIDER_VERIFICATION) ?? "UNVERIFIED", + } + : { accepted: false as const, reason: judged.reason, providerVerification: "UNVERIFIED" as const }; const escalationPathAsserted = verdict(specification.escalation.length >= 2); return { schemaVersion: 2 as const, runbookId, gateId: specification.gateId, releaseId: release.releaseId, - drillTimestamp: new Date().toISOString(), + drillTimestamp: now.toISOString(), triggerInjected: specification.triggerKinds[0] ?? "", triggerAsserted: result.triggerAsserted, containmentAsserted: result.containmentAsserted, @@ -506,6 +540,7 @@ export async function runRunbookDrill(runbookId: string, options: DrillOptions = negativeFixtureFailedAsExpected: result.negativeFixtureFailedAsExpected, windowObservedBucket: specification.window, providerVerificationRequired: result.providerVerificationRequired, + ...(providerEvidence === undefined ? {} : { providerEvidence }), result: combineDrillVerdicts( [ result.triggerAsserted, @@ -515,24 +550,117 @@ export async function runRunbookDrill(runbookId: string, options: DrillOptions = result.negativeFixtureFailedAsExpected, ], result.providerVerificationRequired, + providerEvidence?.providerVerification, ), }; } +/** The items the local drill left UNVERIFIED, by the names provider evidence uses. */ +function openItems(result: DrillResult): string[] { + return [ + ...(result.triggerAsserted === "UNVERIFIED" ? [ITEM_TRIGGER] : []), + ...(result.containmentAsserted === "UNVERIFIED" ? [ITEM_CONTAINMENT] : []), + ...result.recoveryAssertions.filter((item) => item.result === "UNVERIFIED").map((item) => item.assertion), + ...(result.negativeFixtureFailedAsExpected === "UNVERIFIED" ? [ITEM_NEGATIVE_FIXTURE] : []), + ...(result.providerVerificationRequired ? [ITEM_PROVIDER_VERIFICATION] : []), + ]; +} + +type JudgedProviderEvidence = + | Readonly<{ accepted: true; evidence: ProviderDrillEvidence }> + | Readonly<{ accepted: false; reason: string }>; + +/** + * 0929/11-02. Provider evidence counts only for this runbook, this release, + * a recent observation, and the items the local drill could not observe. + */ +export function judgeProviderEvidence( + value: unknown, + runbookId: string, + releaseId: string, + open: readonly string[], + now: Date, +): JudgedProviderEvidence { + const parsed = providerDrillEvidenceSchema.safeParse(value); + if (!parsed.success) { + const issue = parsed.error.issues[0]; + return { accepted: false, reason: `not provider drill evidence (${issue?.path.join(".") || "root"}: ${issue?.message ?? "invalid"})` }; + } + const evidence = parsed.data; + if (evidence.runbookId !== runbookId) { + return { accepted: false, reason: `evidence is for ${evidence.runbookId}, not ${runbookId}` }; + } + if (evidence.releaseId !== releaseId) { + return { accepted: false, reason: `evidence is for release ${evidence.releaseId}, not ${releaseId}` }; + } + const age = now.getTime() - Date.parse(evidence.observedAt); + if (age > PROVIDER_EVIDENCE_MAX_AGE_MS) { + return { accepted: false, reason: `evidence observed at ${evidence.observedAt} is older than ${PROVIDER_EVIDENCE_MAX_AGE_MS / 86_400_000} days` }; + } + if (age < -PROVIDER_EVIDENCE_CLOCK_SKEW_MS) { + return { accepted: false, reason: `evidence observed at ${evidence.observedAt} is in the future` }; + } + const extra = evidence.observations.map(({ item }) => item).filter((item) => !open.includes(item)); + if (extra.length > 0) { + return { accepted: false, reason: `evidence names items the local drill observed or does not have: ${extra.join(", ")}` }; + } + return { accepted: true, evidence }; +} + +function observed(evidence: ProviderDrillEvidence, item: string): DrillVerdict | undefined { + return evidence.observations.find((observation) => observation.item === item)?.result; +} + +/** Replaces each UNVERIFIED item the evidence answers; local PASS and FAIL stay. */ +function applyProviderEvidence(result: DrillResult, evidence: ProviderDrillEvidence): DrillResult { + const answer = (item: string, local: DrillVerdict): DrillVerdict => + local === "UNVERIFIED" ? observed(evidence, item) ?? local : local; + const source = (item: string) => + `provider evidence ${evidence.runId} at ${evidence.observedAt}: ${ + evidence.observations.find((observation) => observation.item === item)?.evidence ?? "" + }`; + return { + ...result, + triggerAsserted: answer(ITEM_TRIGGER, result.triggerAsserted), + containmentAsserted: answer(ITEM_CONTAINMENT, result.containmentAsserted), + recoveryAssertions: result.recoveryAssertions.map((item) => + item.result === "UNVERIFIED" && observed(evidence, item.assertion) !== undefined + ? { ...item, evidence: source(item.assertion), result: observed(evidence, item.assertion)! } + : item, + ), + negativeFixtureFailedAsExpected: answer(ITEM_NEGATIVE_FIXTURE, result.negativeFixtureFailedAsExpected), + }; +} + async function main(): Promise { const runbookId = process.argv .slice(2) .find((argument) => /^FE-RB-00[1-5]$/.test(argument)); const releaseRootFlag = process.argv.indexOf("--release-root"); const releaseRoot = releaseRootFlag >= 0 ? process.argv[releaseRootFlag + 1] : undefined; - if (!runbookId || (releaseRootFlag >= 0 && !releaseRoot)) { + const evidenceFlag = process.argv.indexOf("--provider-evidence"); + const evidencePath = evidenceFlag >= 0 ? process.argv[evidenceFlag + 1] : undefined; + if (!runbookId || (releaseRootFlag >= 0 && !releaseRoot) || (evidenceFlag >= 0 && !evidencePath)) { process.stderr.write( - "Usage: drill:runbook -- FE-RB-001..FE-RB-005 [--release-root ]\n", + "Usage: drill:runbook -- FE-RB-001..FE-RB-005 [--release-root ]" + + " [--provider-evidence ]\n", ); process.exit(2); } + let providerEvidence: unknown; + if (evidencePath !== undefined) { + try { + providerEvidence = JSON.parse(await readFile(evidencePath, "utf8")); + } catch (error) { + process.stderr.write(`${runbookId} drill: cannot read provider evidence ${evidencePath}: ${String(error)}\n`); + process.exit(2); + } + } - const { gateId, ...record } = await runRunbookDrill(runbookId, releaseRoot ? { releaseRoot } : {}); + const { gateId, ...record } = await runRunbookDrill(runbookId, { + ...(releaseRoot ? { releaseRoot } : {}), + ...(evidencePath !== undefined ? { providerEvidence } : {}), + }); const artifactDirectory = `artifacts/runbooks/${runbookId}`; await mkdir(artifactDirectory, { recursive: true }); await writeValidatedJsonArtifact({ @@ -547,14 +675,18 @@ async function main(): Promise { if (record.result === "UNVERIFIED") { // Not a pass: the gate stays red until provider evidence exists (D-14). const open = [ - ["trigger", record.triggerAsserted], - ["containment", record.containmentAsserted], + [ITEM_TRIGGER, record.triggerAsserted], + [ITEM_CONTAINMENT, record.containmentAsserted], ...record.recoveryAssertions.map((item) => [item.assertion, item.result] as const), - ["negative fixture", record.negativeFixtureFailedAsExpected], - ["provider verification", record.providerVerificationRequired ? "UNVERIFIED" : "PASS"], + [ITEM_NEGATIVE_FIXTURE, record.negativeFixtureFailedAsExpected], + [ + ITEM_PROVIDER_VERIFICATION, + record.providerVerificationRequired ? record.providerEvidence?.providerVerification ?? "UNVERIFIED" : "PASS", + ], ].filter(([, result]) => result === "UNVERIFIED").map(([name]) => name); + const rejected = record.providerEvidence?.accepted === false ? `; provider evidence rejected: ${record.providerEvidence.reason}` : ""; process.stderr.write( - `${runbookId} drill: UNVERIFIED (${gateId}; no local check failed; not observed locally: ${open.join(", ")})\n`, + `${runbookId} drill: UNVERIFIED (${gateId}; no local check failed; not observed locally: ${open.join(", ")}${rejected})\n`, ); process.exit(3); } diff --git a/scripts/generate-build-manifest.ts b/scripts/generate-build-manifest.ts index 2d47f7d7..58c1de8a 100644 --- a/scripts/generate-build-manifest.ts +++ b/scripts/generate-build-manifest.ts @@ -20,7 +20,7 @@ import { buildDate, } from "./lib/build-environment.ts"; import { writeValidatedJsonArtifact } from "./lib/validated-json-artifact.ts"; -import { CANONICAL_VITE_MANIFEST_PATH } from "./lib/build-manifest-outputs.ts"; +import { CANONICAL_VITE_MANIFEST_PATH, findRouteChunk } from "./lib/build-manifest-outputs.ts"; import { runOutputPath } from "./lib/run-isolation.ts"; assertCiBuildEnvironment(process.env); @@ -68,14 +68,13 @@ const routeChunks: Record = {}; const runtimeContracts: Readonly> = ROUTE_RUNTIME_CONTRACT; for (const definition of Object.values(ROUTE_REGISTRY)) { - const runtime = runtimeContracts[definition.routeId]; - const asset = Object.values(viteManifestObject).find( - (entry) => entry.name === runtime?.moduleId && entry.isDynamicEntry, + const chunk = findRouteChunk( + viteManifestObject, + definition.routeId, + runtimeContracts[definition.routeId]?.moduleId, ); - if (!runtime || !asset?.file) { - throw new Error(`Missing built route chunk: ${definition.routeId}`); - } - routeChunks[definition.chunkId] = asset.file; + if ("error" in chunk) throw new Error(chunk.error); + routeChunks[definition.chunkId] = chunk.file; } const runtimeConfigJsonSchema = z.toJSONSchema(runtimeConfigV2ArtifactSchema); diff --git a/scripts/lib/build-manifest-outputs.ts b/scripts/lib/build-manifest-outputs.ts index 0f7dea44..bdd8f268 100644 --- a/scripts/lib/build-manifest-outputs.ts +++ b/scripts/lib/build-manifest-outputs.ts @@ -186,3 +186,52 @@ export async function verifyBuildManifestOutputs( } return mismatches; } + +/** + * 0929/13-03. The built chunk of a route's lazy page, for the build manifest + * and the two release checks that compare its `routeChunks`. A route runtime's + * `moduleId` is the file name of the page it imports + * (`src/presentation/routes/route-runtime.tsx`, a generated feature's + * `-route-runtime.ts`), so the chunk is the dynamic entry whose source + * file — the Vite manifest key — has that name. The chunk's own `name` used to + * be matched instead, so a chunk-naming change in vite.config.ts stopped the + * build with a reason-less `Missing built route chunk`. + */ +export type RouteChunkLookup = Readonly<{ file: string }> | Readonly<{ error: string }>; + +const SOURCE_EXTENSION = /\.(?:tsx|ts|jsx|js)$/u; + +export function findRouteChunk( + manifest: Readonly>, + routeId: string, + moduleId: string | undefined, +): RouteChunkLookup { + if (moduleId === undefined) { + return { error: `Missing built route chunk: ${routeId} has no route runtime contract (ROUTE_RUNTIME_CONTRACT)` }; + } + const matches: { source: string; file: string }[] = []; + for (const [key, entry] of Object.entries(manifest)) { + if (!isManifestRecord(entry) || entry["isDynamicEntry"] !== true || typeof entry["file"] !== "string") continue; + const source = typeof entry["src"] === "string" ? entry["src"] : key; + if (path.posix.basename(source).replace(SOURCE_EXTENSION, "") === moduleId) { + matches.push({ source, file: entry["file"] }); + } + } + if (matches.length === 1) return { file: matches[0]!.file }; + if (matches.length === 0) { + return { + error: + `Missing built route chunk: ${routeId}: no lazily imported module in the Vite manifest has the source ` + + `file ${moduleId}.tsx (or .ts). A route runtime's moduleId must be the file name of the page it imports.`, + }; + } + return { + error: + `Ambiguous built route chunk: ${routeId}: the lazily imported modules ` + + `${matches.map(({ source }) => source).join(", ")} are all named ${moduleId}.`, + }; +} + +function isManifestRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} diff --git a/scripts/lib/hosting-probe.ts b/scripts/lib/hosting-probe.ts index 02a997b5..640fc236 100644 --- a/scripts/lib/hosting-probe.ts +++ b/scripts/lib/hosting-probe.ts @@ -142,9 +142,9 @@ export async function probeLiveRelease( const negotiateAlpn = transport.negotiateAlpn ?? negotiatedAlpnProtocol; const responses: ResponseHeaders = {}; const results: ProbeResult[] = []; - const probe = async (surface: string, pathname: string): Promise => { + const probe = async (surface: string, target: URL): Promise => { try { - const response = await fetcher(new URL(pathname, base)); + const response = await fetcher(target); const finalOrigin = new URL(response.url).origin; results.push( { @@ -191,11 +191,12 @@ export async function probeLiveRelease( const bodies: Record = {}; for (const [surface, pathname] of Object.entries(paths)) { - bodies[surface] = await probe(surface, pathname); + bodies[surface] = await probe(surface, surfaceUrl(pathname, base)); } const entryScript = servedEntryScript(bodies["index"] ?? ""); if (entryScript) { - await probe("hashedAsset", entryScript); + // The served index.html names it as the build wrote it (under the base path). + await probe("hashedAsset", new URL(entryScript, base)); } else { results.push({ surface: "hashedAsset", @@ -226,6 +227,21 @@ export async function probeLiveRelease( }); } const candidateEntryScript = servedEntryScript(candidate.indexHtml); + // 0929/13-06. The candidate was built for one base path; a HOSTING_BASE_URL + // with another path probes surfaces the release does not serve. + // The build writes its scripts to `assets/`, so the entry script's + // folder says which base the candidate was built for, in both directions. + const builtFor = candidateEntryScript?.startsWith(`${base.pathname}${BUILD_ASSET_DIRECTORY}`) === true; + results.push({ + surface: "deployment", + header: "base-path", + expected: `${base.pathname}${BUILD_ASSET_DIRECTORY}`, + observed: candidateEntryScript, + ...(builtFor + ? {} + : { reason: "the candidate was built for another base path than the HOSTING_BASE_URL path (VITE_ROUTER_BASE_PATH of the build)" }), + passed: builtFor, + }); results.push({ surface: "index", header: "candidate-entry-script", @@ -405,10 +421,19 @@ export function checkPublicServiceWorkers( }; } +/** + * 0929/13-06. The surfaces in config/hosting/cache-policy.json are written + * from the release's root (`/config.json`); a release served under a base + * path (`VITE_ROUTER_BASE_PATH=/app/`) serves them under that path. + */ +export function surfaceUrl(pathname: string, base: URL): URL { + return new URL(`.${pathname.startsWith("/") ? pathname : `/${pathname}`}`, base); +} + /** * A release gate must not promote a local preview server as live hosting - * evidence. - * + * evidence. 0929/13-06: the URL is the release's base URL, the root or the + * sub-path the build was made for (`/app/`). */ export function classifyLiveHostingBaseUrl(value: string): | { @@ -467,10 +492,10 @@ export function classifyLiveHostingBaseUrl(value: string): observedOrigin, }; } - if (url.pathname !== "/" || url.search || url.hash) { + if (!url.pathname.endsWith("/") || url.search || url.hash) { return { passed: false, - reason: "HOSTING_BASE_URL must be the canonical root URL", + reason: "HOSTING_BASE_URL must be the release's base URL: a path ending in \"/\", without query or fragment", url, observedOrigin, }; diff --git a/scripts/lib/local-release-evidence.ts b/scripts/lib/local-release-evidence.ts index 9149872e..3e1423c9 100644 --- a/scripts/lib/local-release-evidence.ts +++ b/scripts/lib/local-release-evidence.ts @@ -26,6 +26,7 @@ import { } from "../contracts/release-artifacts.ts"; import { CANONICAL_VITE_MANIFEST_PATH, + findRouteChunk, verifyBuildManifestOutputs, } from "./build-manifest-outputs.ts"; import { assertMatchesJsonSchema } from "./json-schema.ts"; @@ -1255,14 +1256,12 @@ async function verifyReleaseOutputs( const runtimeContracts: Readonly> = ROUTE_RUNTIME_CONTRACT; for (const definition of Object.values(ROUTE_REGISTRY)) { - const runtime = runtimeContracts[definition.routeId]; - const viteEntry = Object.values(viteManifest).find( - (entry) => - isRecord(entry) && - entry.name === runtime?.moduleId && - entry.isDynamicEntry === true, + const chunk = findRouteChunk( + viteManifest, + definition.routeId, + runtimeContracts[definition.routeId]?.moduleId, ); - const file = isRecord(viteEntry) ? viteEntry.file : null; + const file = "error" in chunk ? null : chunk.file; if ( typeof file !== "string" || release.routeChunks[definition.chunkId] !== file || diff --git a/scripts/lib/provider-cgroup.ts b/scripts/lib/provider-cgroup.ts index 74ae58d4..770f26b3 100644 --- a/scripts/lib/provider-cgroup.ts +++ b/scripts/lib/provider-cgroup.ts @@ -1,10 +1,24 @@ export type ProviderKind = "vulnerability" | "provenance"; +/** The oldest systemd user manager whose `--scope` properties the provider relies on. */ +export const MINIMUM_PROVIDER_SYSTEMD_VERSION = 254; + +/** `systemctl` arguments that ask the user manager its version; the provider and the prerequisite check both use them. */ +export const USER_MANAGER_VERSION_ARGUMENTS: readonly string[] = Object.freeze([ + "--user", + "show", + "--property=Version", + "--value", +]); + const MEMORY_MAX = 1_073_741_824; const TASKS_MAX = 64; const STOP_TIMEOUT_MS = 5_000; const RUNTIME_GRACE_MS = 10_000; const UNIT_NAME = /^ca-provider-(?:vulnerability|provenance)-[1-9][0-9]*-[0-9a-f]{24}\.scope$/u; +const PROBE_UNIT_NAME = /^ca-provider-prerequisite-[1-9][0-9]*-[0-9a-f]{24}\.scope$/u; +/** How long the prerequisite probe's scope may live; it runs /bin/true. */ +const PROBE_RUNTIME_MS = 5_000; const UNIT_NONCE = /^[0-9a-f]{24}$/u; const ENVIRONMENT_NAME = /^[A-Za-z_][A-Za-z0-9_]*$/u; const ENFORCEMENT_GATE = [ @@ -79,6 +93,37 @@ export function systemdRunProviderArguments( ) { throw new TypeError("provider scope wrapper path is invalid"); } + return [ + ...scopeArguments(unit, timeoutMs + RUNTIME_GRACE_MS), + nodeExecutable, + wrapperScript, + String(cpuSeconds), + reportPath, + String(reportDev), + String(reportIno), + ]; +} + +export function formatProviderPrerequisiteUnitName(supervisorPid: number, nonce: string): string { + if (!Number.isSafeInteger(supervisorPid) || supervisorPid <= 0 || !UNIT_NONCE.test(nonce)) { + throw new TypeError("provider cgroup unit identity is invalid"); + } + return `ca-provider-prerequisite-${supervisorPid}-${nonce}.scope`; +} + +/** + * 1112/08-03. The provider's own scope — same properties, same check that the + * unit's cgroup enforces them — around /bin/true. A host whose user manager + * cannot start the scope, or does not delegate the memory, pids and cpu + * controllers, fails here instead of in each provider test. + */ +export function systemdRunPrerequisiteProbeArguments(unit: string): string[] { + if (!PROBE_UNIT_NAME.test(unit)) throw new TypeError("provider cgroup unit is invalid"); + return [...scopeArguments(unit, PROBE_RUNTIME_MS), "/bin/true"]; +} + +/** `systemd-run` options and the enforcement gate up to its `$0`; the caller appends what the gate execs. */ +function scopeArguments(unit: string, runtimeMaxMs: number): string[] { return [ "--user", "--scope", @@ -94,19 +139,13 @@ export function systemdRunProviderArguments( "--property=KillMode=control-group", "--property=SendSIGKILL=yes", `--property=TimeoutStopSec=${STOP_TIMEOUT_MS}ms`, - `--property=RuntimeMaxSec=${timeoutMs + RUNTIME_GRACE_MS}ms`, + `--property=RuntimeMaxSec=${runtimeMaxMs}ms`, "--", "/bin/sh", "-eu", "-c", ENFORCEMENT_GATE, unit, - nodeExecutable, - wrapperScript, - String(cpuSeconds), - reportPath, - String(reportDev), - String(reportIno), ]; } diff --git a/scripts/lib/test-evidence-artifact.ts b/scripts/lib/test-evidence-artifact.ts index 1f86f3f7..dcc1de8c 100644 --- a/scripts/lib/test-evidence-artifact.ts +++ b/scripts/lib/test-evidence-artifact.ts @@ -12,6 +12,8 @@ export const testEvidenceReportSchema = z sharedScenarios: z.number().int().nonnegative(), declaredScenarioExecutions: z.number().int().nonnegative(), executedScenarioExecutions: z.number().int().nonnegative(), + // 1112/03-03. The Playwright configs the forbidOnly rule read. + playwrightConfigs: z.array(z.string().min(1)).optional(), }) .strict(), failures: z.array(z.string()), diff --git a/scripts/run-and-validate-provider.ts b/scripts/run-and-validate-provider.ts index f79b0092..699df5b5 100644 --- a/scripts/run-and-validate-provider.ts +++ b/scripts/run-and-validate-provider.ts @@ -20,8 +20,10 @@ import { encodeProviderBwrapInput, encodeProviderScopeFrame, formatProviderCgroupUnitName, + MINIMUM_PROVIDER_SYSTEMD_VERSION, systemctlKillProviderArguments, systemdRunProviderArguments, + USER_MANAGER_VERSION_ARGUMENTS, } from "./lib/provider-cgroup.ts"; import { assertProviderGuardianLeasePaths, @@ -50,7 +52,6 @@ const PROVIDER_REAP_TIMEOUT_MS = 5_000; const PROVIDER_CONTROL_TIMEOUT_MS = 5_000; const PROVIDER_CONTROL_MAX_OUTPUT_BYTES = 65_536; const PROVIDER_CONTROL_POLL_MS = 25; -const MINIMUM_PROVIDER_SYSTEMD_VERSION = 254; if (kind !== "vulnerability" && kind !== "provenance") { process.stderr.write("Usage: run-and-validate-provider --kind vulnerability|provenance\n"); process.exit(2); @@ -487,7 +488,7 @@ async function waitForProviderClose( async function assertProviderCgroupManagerAvailable(): Promise { const output = await runBoundedSystemctl( - ["--user", "show", "--property=Version", "--value"], + USER_MANAGER_VERSION_ARGUMENTS, "provider cgroup user manager probe", ); const match = /^([0-9]+)/u.exec(output.trim()); diff --git a/scripts/verify-documentation-readiness.ts b/scripts/verify-documentation-readiness.ts index 1032181d..3abdd9fe 100644 --- a/scripts/verify-documentation-readiness.ts +++ b/scripts/verify-documentation-readiness.ts @@ -304,16 +304,44 @@ function linkedPath(target: string, document: string): string | null { // never looked up. Leading `NAME=value` assignments are skipped, a // `--flag=value` argument is read as its value, and placeholders (``, // globs, `$VAR`) are left alone by namedPath. -const COMMAND_HEADS = new Set(["corepack", "pnpm", "npx", "node", "vitest", "playwright", "tsc", "eslint", "bash", "sh"]); -function commandPaths(command: string, document: string): (string | null)[] { +const COMMAND_HEADS = new Set([ + "corepack", "pnpm", "npx", "node", "vitest", "playwright", "tsc", "eslint", "bash", "sh", "python3", +]); +// 1112/08-06. `scripts/verify.sh` is this repository's main entry and most +// documented commands go through it: the entry is itself a path, `exec --` +// runs a command whose paths are looked up like any other, and `script` +// names package scripts (verifyScripts). +const VERIFY_ENTRY = /^(?:\.\/)?scripts\/verify\.sh$/u; +function commandWords(command: string): Readonly<{ words: string[]; head: number }> { const words = command.trim().replace(/^\$\s+/u, "").split(/\s+/u); let head = 0; while (/^[A-Z_][A-Z0-9_]*=/u.test(words[head] ?? "")) head += 1; - if (!COMMAND_HEADS.has(words[head] ?? "")) return []; + return { words, head }; +} +function commandPaths(command: string, document: string): (string | null)[] { + const { words, head } = commandWords(command); + const first = words[head] ?? ""; + if (VERIFY_ENTRY.test(first)) { + const separator = words.indexOf("--", head + 1); + const executed = words[head + 1] === "exec" && separator > head ? words.slice(separator + 1).join(" ") : ""; + return [namedPath(first, document), ...(executed === "" ? [] : commandPaths(executed, document))]; + } + if (!COMMAND_HEADS.has(first)) return []; return words .slice(head + 1) .map((word) => namedPath(word.replace(/^--?[\w.-]+=/u, "").replace(/^["']|["']$/gu, ""), document)); } +/** The package scripts `scripts/verify.sh script … [-- args]` runs; placeholders are left alone. */ +function verifyScripts(command: string): string[] { + const { words, head } = commandWords(command); + if (!VERIFY_ENTRY.test(words[head] ?? "") || words[head + 1] !== "script") return []; + const names: string[] = []; + for (const word of words.slice(head + 2)) { + if (word.startsWith("-")) break; + names.push(word); + } + return names.filter((name) => /^[a-z](?:[\w:-]*\w)?$/u.test(name)); +} // Lines of fenced shell blocks (```bash, ```sh, ```shell, ```console or no // language), which is where most documented commands are. function fencedCommands(text: string): string[] { @@ -398,6 +426,10 @@ for (const document of contractDocuments) { for (const [, name = ""] of text.matchAll(/`([a-z][\w-]*:[\w:-]*\w)(?=[` ])/gu)) { if (scriptPrefixes.has(name.split(":")[0]) && missingScript(name)) missingScripts.add(name); } + const commands = [...[...text.matchAll(/`([^`\n]+)`/gu)].map(([, token = ""]) => token), ...fencedCommands(text)]; + for (const name of commands.flatMap(verifyScripts)) { + if (missingScript(name)) missingScripts.add(name); + } documentReferences.push({ path: document, missingPaths: [...missingPaths].sort(), diff --git a/scripts/verify-hosting-headers.ts b/scripts/verify-hosting-headers.ts index 7ec12a74..e3ccb6af 100644 --- a/scripts/verify-hosting-headers.ts +++ b/scripts/verify-hosting-headers.ts @@ -121,7 +121,7 @@ if (liveTarget?.passed && !slotEmpty) { probeResults.push({ surface: "deployment", header: "base-url", - expected: "canonical non-loopback HTTPS root URL", + expected: "non-loopback HTTPS base URL of the release", observed: liveTarget.observedOrigin, reason: liveTarget.reason, passed: false, diff --git a/scripts/verify-release.ts b/scripts/verify-release.ts index c4eb5a9f..c332cd69 100644 --- a/scripts/verify-release.ts +++ b/scripts/verify-release.ts @@ -24,6 +24,7 @@ import { import { assertMatchesJsonSchema } from "./lib/json-schema.ts"; import { CANONICAL_VITE_MANIFEST_PATH, + findRouteChunk, verifyBuildManifestOutputs, } from "./lib/build-manifest-outputs.ts"; import { verifyReleaseRuntimeCoherence } from "./lib/release-runtime-coherence.ts"; @@ -168,12 +169,13 @@ for (const chunkId of actualChunkIds) { const runtimeContracts: Readonly> = ROUTE_RUNTIME_CONTRACT; for (const definition of Object.values(ROUTE_REGISTRY)) { - const runtime = runtimeContracts[definition.routeId]; - const viteEntry = Object.values(viteManifestObject).find( - (entry) => entry.name === runtime?.moduleId && entry.isDynamicEntry, + const viteEntry = findRouteChunk( + viteManifestObject, + definition.routeId, + runtimeContracts[definition.routeId]?.moduleId, ); const routeAsset = release.routeChunks[definition.chunkId]; - if (!runtime || !viteEntry || routeAsset !== viteEntry.file) { + if ("error" in viteEntry || routeAsset !== viteEntry.file) { artifactMismatches.push(`routeChunk:mismatch:${definition.chunkId}`); continue; } diff --git a/scripts/verify-reproducible-build.ts b/scripts/verify-reproducible-build.ts index dbf2ec31..79f15644 100644 --- a/scripts/verify-reproducible-build.ts +++ b/scripts/verify-reproducible-build.ts @@ -4,10 +4,18 @@ import path from "node:path"; import { reproducibleBuildArtifactSchema } from "./contracts/release-artifacts.ts"; import { assertCiBuildEnvironment } from "./lib/build-environment.ts"; +import { checkoutBuildReadFailure } from "./lib/run-isolation.ts"; import { supplyChainDigest } from "./lib/supply-chain.ts"; import { writeValidatedJsonArtifact } from "./lib/validated-json-artifact.ts"; assertCiBuildEnvironment(process.env); +// 1106/12-02. This builds twice and digests dist where the checkout keeps it. Under RUN_OUTPUT_ROOT the build wrote elsewhere, so +// this would judge the earlier build the checkout still holds. +const outputRootFailure = checkoutBuildReadFailure("scripts/verify-reproducible-build.ts"); +if (outputRootFailure !== undefined) { + process.stderr.write(`${outputRootFailure}\n`); + process.exit(1); +} async function filesWithin(directory: string): Promise { const entries = await readdir(directory, { withFileTypes: true }); diff --git a/tests/architecture/request-fanout.test.ts b/tests/architecture/request-fanout.test.ts index c764fc96..d0df201a 100644 --- a/tests/architecture/request-fanout.test.ts +++ b/tests/architecture/request-fanout.test.ts @@ -3,7 +3,9 @@ import path from "node:path"; import { describe, expect, it } from "vitest"; import { + checkRequestFanoutFixtures, findRequestFanout, + formatFanoutViolation, scanRequestFanout, } from "../../scripts/check-request-fanout.ts"; @@ -21,6 +23,13 @@ describe("no per-row transport loop (frontend N+1 source gate)", () => { expect(new Set(forbidden.violations.map((violation) => violation.file)).size).toBe(forbidden.files); expect(forbidden.files).toBe(3); expect(allowed.violations).toEqual([]); + expect(allowed.reviewed).toEqual([ + expect.objectContaining({ + file: "tests/fixtures/architecture/request-fanout/allowed/reviewed-cursor-loop.ts", + call: "gateway.page()", + reason: "cursor pages, bounded by maxPages", + }), + ]); }); it("names the dispatch and the repeating construct", async () => { @@ -40,4 +49,24 @@ describe("no per-row transport loop (frontend N+1 source gate)", () => { ); expect(violations).toEqual([]); }); + + // 0929/12-01. The failure line named only the call, so a product team saw + // neither the cursor runtime nor the reviewed-exception comment, and the + // result line hid how many calls a comment let through. + it("tells the author both ways out on the failure line", async () => { + const [violation] = await findRequestFanout( + "inline.ts", + "export const load = (api: { get(id: string): Promise }, ids: string[]) =>\n Promise.all(ids.map((id) => api.get(id)));\n", + ); + const line = formatFanoutViolation(violation!); + expect(line).toMatch(/^FAIL inline\.ts:2 api\.get\(\) inside \.map\(\) callback: /u); + expect(line).toContain("createCursorPaginationRuntime (src/adapters/query-cache)"); + expect(line).toContain("`// fanout-reviewed: ` on the call's line"); + }); + + it("counts and lists the reviewed exceptions in the fixture result line", async () => { + const result = await checkRequestFanoutFixtures(path.resolve(".")); + expect(result.ok).toBe(true); + expect(result.line).toMatch(/allowed 0 violations and 1 reviewed exception \(tests\/fixtures\/architecture\/request-fanout\/allowed\/reviewed-cursor-loop\.ts:\d+ cursor pages, bounded by maxPages\) PASS/u); + }); }); diff --git a/tests/browser-capabilities/opfs-runtime.spec.ts b/tests/browser-capabilities/opfs-runtime.spec.ts index 59d42d64..3a4849f9 100644 --- a/tests/browser-capabilities/opfs-runtime.spec.ts +++ b/tests/browser-capabilities/opfs-runtime.spec.ts @@ -337,6 +337,7 @@ async function removeOpfsAuthority( for (const committedObjects of [0, 3]) { test(`05-01 an approved purge in another tab waits for a live put and takes it with the partition (${committedObjects} committed objects)`, async ({ page, + browserName, }) => { const secondPage = await page.context().newPage(); const suffix = crypto.randomUUID().replaceAll("-", ""); @@ -349,7 +350,18 @@ for (const committedObjects of [0, 3]) { type Tab = import("../../src/adapters/storage/opfs/browser-opfs-runtime.ts").BrowserOpfsRuntime; try { await Promise.all([page.goto("/"), secondPage.goto("/")]); - if (!(await installOpfsTab(page, scope))) return; + // 1112/03-02. An early return here passed with no assertion on an engine + // the support check refused, while the evidence said zero skipped. + // Playwright's WebKit is refused (the first test asserts its UNSUPPORTED + // result), so there this asserts the refusal; every other engine must + // open the runtime. A skip is no way out: the capability evidence + // refuses skipped cases (scripts/verify-browser-capability-evidence.ts). + const installed = await installOpfsTab(page, scope); + if (browserName === "webkit") { + expect(installed, "the OPFS support check refuses Playwright's WebKit").toBe(false); + return; + } + expect(installed, "OPFS runtime supported in this engine").toBe(true); expect(await installOpfsTab(secondPage, scope, true)).toBe(true); const seeded = await secondPage.evaluate(async (count) => { @@ -455,6 +467,7 @@ for (const committedObjects of [0, 3]) { test("05-01 a reconcile in another tab skips a live put and the put stays readable", async ({ page, + browserName, }) => { const secondPage = await page.context().newPage(); const suffix = crypto.randomUUID().replaceAll("-", ""); @@ -467,7 +480,13 @@ test("05-01 a reconcile in another tab skips a live put and the put stays readab type Tab = import("../../src/adapters/storage/opfs/browser-opfs-runtime.ts").BrowserOpfsRuntime; try { await Promise.all([page.goto("/"), secondPage.goto("/")]); - if (!(await installOpfsTab(page, scope))) return; + // 1112/03-02. Asserted, not returned early: see the purge test above. + const installed = await installOpfsTab(page, scope); + if (browserName === "webkit") { + expect(installed, "the OPFS support check refuses Playwright's WebKit").toBe(false); + return; + } + expect(installed, "OPFS runtime supported in this engine").toBe(true); expect(await installOpfsTab(secondPage, scope)).toBe(true); // Tab A starts a put whose source stalls, so its journal row stays diff --git a/tests/e2e/preload-use.spec.ts b/tests/e2e/preload-use.spec.ts index 7f614f83..d748b098 100644 --- a/tests/e2e/preload-use.spec.ts +++ b/tests/e2e/preload-use.spec.ts @@ -298,6 +298,25 @@ recordedTest.describe("the strict fixture's console check", () => { ); }); +// 1112/03-05. The fixture watched its own page only, so a console error in +// a second tab of the same context (the cross-tab capability specs) passed. +recordedTest.describe("the strict fixture's other tabs", () => { + recordedTest.use({ + fixtureReports: { lines: [/^console:error:\[preload-use\] second tab$/u] }, + }); + + recordedTest( + "records a console error in another tab of the context", + async ({ page }) => { + await openHome(page); + const second = await page.context().newPage(); + await openHome(second); + + await second.evaluate(() => console.error("[preload-use] second tab")); + }, + ); +}); + /** * 1105/11-01. The verdict every other spec gets: a recorded failure fails the * test. The record is proven above; this proves the verdict. diff --git a/tests/e2e/service-worker-recovery.spec.ts b/tests/e2e/service-worker-recovery.spec.ts index 400ec503..561330fc 100644 --- a/tests/e2e/service-worker-recovery.spec.ts +++ b/tests/e2e/service-worker-recovery.spec.ts @@ -1,6 +1,11 @@ import path from "node:path"; -import { expect, test, type Page } from "../support/browser/strict-browser-test.ts"; +import { + expect, + test, + watchBrowserFailures, + type Page, +} from "../support/browser/strict-browser-test.ts"; import { prepareServiceWorkerBuilds, removeServiceWorkerBuilds, @@ -91,7 +96,7 @@ test("a REMOVE_REGISTRATION release unregisters a worker that blocks boot", asyn const server = await startBuildServer(builds.dist); // The broken phase fails requests on purpose, so this test drives its own // context instead of the strict `page` fixture and checks the recovered - // load for browser errors itself. + // load for browser errors with the fixture's own watcher (1112/03-05). const context = await browser.newContext(); try { const page = await context.newPage(); @@ -133,22 +138,16 @@ test("a REMOVE_REGISTRATION release unregisters a worker that blocks boot", asyn ) .toBe(true); - const failures: string[] = []; - page.on("console", (message) => { - if (["error", "warning"].includes(message.type())) { - failures.push(`console:${message.type()}:${message.text()}`); - } - }); - page.on("pageerror", (error) => failures.push(`pageerror:${error.name}`)); - page.on("requestfailed", (request) => - failures.push(`requestfailed:${new URL(request.url()).pathname}`), - ); + // No preload check here: this context loaded the origin twice already, + // and WebKit answers the recovered load's scripts from its cache, past + // the tracker's route, so every preload reads as unused (16 in a run). + const recovered = await watchBrowserFailures(page); await page.goto(`${server.origin}/`); expect(await startOutcome(page)).toEqual({ kind: "DISABLED" }); expect( await page.evaluate(() => navigator.serviceWorker.controller === null), ).toBe(true); - expect(failures).toEqual([]); + expect(await recovered.stop()).toEqual([]); } finally { await context.close(); await server.close(); diff --git a/tests/e2e/support/fixture-builds.ts b/tests/e2e/support/fixture-builds.ts index 23a95e3f..933fa022 100644 --- a/tests/e2e/support/fixture-builds.ts +++ b/tests/e2e/support/fixture-builds.ts @@ -65,17 +65,41 @@ function run( } } -export async function prepareFixtureBuilds( - prefix: string, - builds: readonly FixtureBuild[], -): Promise> { - // An edit's source text names the build (two specs never share a prefix). - const fingerprint = JSON.stringify( - builds.map((build) => [build.name, build.env ?? {}, build.edit?.toString() ?? ""]), - ); +export type EditedCopyBuild = Readonly<{ + name: Name; + /** VITE_BUILD_ID and RELEASE_ID of this build. */ + buildId: string; + edit?: (root: string) => Promise; + env?: Readonly>; +}>; + +/** + * 0929/11-07. The one way an e2e spec gets real production builds of an + * edited copy of the checkout; prepareFixtureBuilds and + * prepareServiceWorkerBuilds (service-worker-builds.ts) are its two shapes. + * It copies the checkout (without `checkout-copy-skip`), links node_modules, + * runs `prepare` once, then for each build runs `edit`, builds, moves `dist` + * to `dist-` and asks `collect` what the spec needs from that build. + * `key` and `fingerprint` name the result in the e2e run + * (shared-fixture-builds.ts), which removes the copy when preparing fails. + */ +export async function buildEditedCopies( + input: Readonly<{ + key: string; + fingerprint: string; + prepare?: (root: string) => Promise; + builds: readonly EditedCopyBuild[]; + collect?: (name: Name, root: string) => Promise; + }>, +): Promise>; + collected: Readonly>; + shared: boolean; +}>> { const { root, value, shared } = await sharedFixtureBuild( - `fixture-${prefix}`, - fingerprint, + input.key, + input.fingerprint, async (root) => { await cp(repoRoot, root, { recursive: true, @@ -83,19 +107,36 @@ export async function prepareFixtureBuilds( !skipped.has(path.relative(repoRoot, source).split(path.sep)[0]!), }); await linkFixtureNodeModules(root, repoRoot); + await input.prepare?.(root); const dist = {} as Record; - for (const build of builds) { + const collected = {} as Record; + for (const build of input.builds) { await build.edit?.(root); - const buildId = `${prefix}-${build.name}`; - run(root, "scripts/generate-product-feature-index.ts", buildId, build.env); - run(root, "scripts/build-frontend.ts", buildId, build.env); + run(root, "scripts/generate-product-feature-index.ts", build.buildId, build.env); + run(root, "scripts/build-frontend.ts", build.buildId, build.env); dist[build.name] = path.join(root, `dist-${build.name}`); await rename(path.join(root, "dist"), dist[build.name]); + collected[build.name] = input.collect ? await input.collect(build.name, root) : (null as Collected); } - return dist; + return { dist, collected }; }, ); - return { root, dist: value, shared }; + return { root, ...value, shared }; +} + +export async function prepareFixtureBuilds( + prefix: string, + builds: readonly FixtureBuild[], +): Promise> { + // An edit's source text names the build (two specs never share a prefix). + const { root, dist, shared } = await buildEditedCopies({ + key: `fixture-${prefix}`, + fingerprint: JSON.stringify( + builds.map((build) => [build.name, build.env ?? {}, build.edit?.toString() ?? ""]), + ), + builds: builds.map((build) => ({ ...build, buildId: `${prefix}-${build.name}` })), + }); + return { root, dist, shared }; } export async function removeFixtureBuilds( diff --git a/tests/e2e/support/service-worker-builds.ts b/tests/e2e/support/service-worker-builds.ts index c96b1961..ef4bc723 100644 --- a/tests/e2e/support/service-worker-builds.ts +++ b/tests/e2e/support/service-worker-builds.ts @@ -1,12 +1,5 @@ -import { spawnSync } from "node:child_process"; import { existsSync } from "node:fs"; -import { - cp, - readFile, - rename, - rm, - writeFile, -} from "node:fs/promises"; +import { readFile, rm, writeFile } from "node:fs/promises"; import { createServer, type Server } from "node:http"; import type { AddressInfo } from "node:net"; import path from "node:path"; @@ -14,11 +7,8 @@ import { pathToFileURL } from "node:url"; import { gzipSync } from "node:zlib"; import { SERVICE_WORKER_ASSETS_OUTPUT } from "../../../scripts/generate-service-worker-assets.ts"; -import { localBuildEnvironment } from "../../../scripts/lib/build-environment.ts"; -import { linkFixtureNodeModules } from "../../../scripts/lib/fixture-node-modules.ts"; -import { selectRepositoryPaths } from "../../../scripts/lib/repository-layout.ts"; import type { StaticAssetManifestV1 } from "../../../src/contracts/service-worker.ts"; -import { sharedFixtureBuild } from "./shared-fixture-builds.ts"; +import { buildEditedCopies, editOnce } from "./fixture-builds.ts"; /** * R3-6 test helper: two real production builds with the Service Worker on, @@ -54,11 +44,6 @@ import { sharedFixtureBuild } from "./shared-fixture-builds.ts"; * text it expected. */ -const repoRoot = path.resolve(import.meta.dirname, "../../.."); -// 1105/12-04. `.tmp` holds other runs' builds and results (.tmp/runs/), -// which change while this copy is made; fixture-builds.ts skips it too. -const skipped = new Set(selectRepositoryPaths("checkout-copy-skip")); - export type BuildName = "a" | "b"; export type ServiceWorkerBuilds = Readonly<{ root: string; @@ -101,17 +86,8 @@ export function createServiceWorkerRuntimeHost( } `; -export async function replaceOnce( - file: string, - search: string, - replacement: string, -): Promise { - const source = await readFile(file, "utf8"); - if (source.split(search).length !== 2) { - throw new Error(`Expected exactly one "${search}" in ${file}.`); - } - await writeFile(file, source.replace(search, replacement)); -} +/** 0929/11-07. The same one-occurrence edit as fixture-builds.ts, not a copy of it. */ +export const replaceOnce = editOnce; /** * Premise 1. Replaces the value of the one `SERVICE_WORKER` declaration in @@ -167,25 +143,12 @@ async function precachedUrls(root: string, name: BuildName): Promise { return SERVICE_WORKER_ASSETS.assets.map((asset) => asset.url).sort(); } -function run(cwd: string, script: string, buildId: string): void { - const result = spawnSync("node", [script], { - cwd, - encoding: "utf8", - // 1103/11-08. A local build, not the CI job's release build. - env: { ...localBuildEnvironment(), VITE_BUILD_ID: buildId, RELEASE_ID: buildId }, - }); - if (result.status !== 0) { - throw new Error( - `${script} failed for ${buildId}:\n${result.stdout}\n${result.stderr}`, - ); - } -} - /** * `beforeBuild` edits the temp copy right before build `name` (06-03: a * broken worker A, then a `REMOVE_REGISTRATION` release B). Edits persist * into the later build. `key` names this pair of builds in the e2e run - * (shared-fixture-builds.ts); each spec passes its own. + * (shared-fixture-builds.ts); each spec passes its own. The copy and the + * builds are fixture-builds.ts's `buildEditedCopies` (0929/11-07). */ export async function prepareServiceWorkerBuilds( options: Readonly<{ @@ -193,38 +156,27 @@ export async function prepareServiceWorkerBuilds( beforeBuild?: (name: BuildName, root: string) => Promise; }>, ): Promise { - const { root, value, shared } = await sharedFixtureBuild( - `sw-${options.key}`, - options.beforeBuild?.toString() ?? "", - async (root) => { - await cp(repoRoot, root, { - recursive: true, - filter: (source) => - !skipped.has(path.relative(repoRoot, source).split(path.sep)[0]!), - }); - await linkFixtureNodeModules(root, repoRoot); + const { root, dist, collected, shared } = await buildEditedCopies({ + key: `sw-${options.key}`, + fingerprint: options.beforeBuild?.toString() ?? "", + prepare: async (root) => { await turnOnServiceWorker(root); const hostFile = path.join(root, "src/bootstrap/register-service-worker.ts"); - await replaceOnce( + await editOnce( hostFile, "export function createServiceWorkerRuntimeHost(", "function createServiceWorkerRuntimeHostUnobserved(", ); await writeFile(hostFile, (await readFile(hostFile, "utf8")) + HOST_PROBE); - - const dist = { a: path.join(root, "dist-a"), b: path.join(root, "dist-b") }; - const precached: Record = { a: [], b: [] }; - for (const name of ["a", "b"] as const) { - await options.beforeBuild?.(name, root); - run(root, "scripts/generate-product-feature-index.ts", `sw-e2e-${name}`); - run(root, "scripts/build-frontend.ts", `sw-e2e-${name}`); - await rename(path.join(root, "dist"), dist[name]); - precached[name] = await precachedUrls(root, name); - } - return { dist, precached }; }, - ); - return { root, ...value, shared }; + builds: (["a", "b"] as const).map((name) => ({ + name, + buildId: `sw-e2e-${name}`, + edit: async (root: string) => options.beforeBuild?.(name, root), + })), + collect: (name, root) => precachedUrls(root, name), + }); + return { root, dist, precached: collected, shared }; } export async function removeServiceWorkerBuilds( diff --git a/tests/fixtures/architecture/request-fanout/allowed/reviewed-cursor-loop.ts b/tests/fixtures/architecture/request-fanout/allowed/reviewed-cursor-loop.ts new file mode 100644 index 00000000..aa3d5341 --- /dev/null +++ b/tests/fixtures/architecture/request-fanout/allowed/reviewed-cursor-loop.ts @@ -0,0 +1,18 @@ +// Allowed: a cursor loop someone reviewed. Each page needs the cursor the +// previous page returned, so the requests cannot be one bulk read; the count +// is bounded by the page cap. Feature code normally gets this from +// createCursorPaginationRuntime (src/adapters/query-cache). +type Page = Readonly<{ items: readonly string[]; nextCursor: string | null }>; +type Gateway = { page(cursor: string | null): Promise }; + +export async function loadAll(gateway: Gateway, maxPages: number) { + const items: string[] = []; + let cursor: string | null = null; + for (let page = 0; page < maxPages; page += 1) { + const next: Page = await gateway.page(cursor); // fanout-reviewed: cursor pages, bounded by maxPages + items.push(...next.items); + if (next.nextCursor === null) break; + cursor = next.nextCursor; + } + return items; +} diff --git a/tests/fixtures/test-evidence/playwright-config/missing-forbid-only.fixture b/tests/fixtures/test-evidence/playwright-config/missing-forbid-only.fixture new file mode 100644 index 00000000..ce5ed861 --- /dev/null +++ b/tests/fixtures/test-evidence/playwright-config/missing-forbid-only.fixture @@ -0,0 +1,8 @@ +// A Playwright config that would let a committed focused test run alone: +// check-test-evidence.ts --playwright-config must reject it (1112/03-03). +import { defineConfig } from "@playwright/test"; + +export default defineConfig({ + testDir: "tests/e2e", + reporter: [["junit", { outputFile: "artifacts/tests/e2e/results.xml" }]], +}); diff --git a/tests/support/browser/strict-browser-test.ts b/tests/support/browser/strict-browser-test.ts index 1c0b5a94..fdda5bc7 100644 --- a/tests/support/browser/strict-browser-test.ts +++ b/tests/support/browser/strict-browser-test.ts @@ -20,6 +20,48 @@ function ignoredConsole(message: ConsoleMessage) { ); } +/** + * 1112/03-05. Records a page's browser failures as `console:…`, `pageerror:…` + * and `requestfailed:…` lines, with the same rules the strict fixture judges + * by. The fixture watches its `page` and every other tab opened in the same + * context; a spec that drives a context of its own watches its pages with + * this. `preloads` also checks that every modulepreload ran (preload-use.ts) + * in the documents open when `stop` runs; attach it before the navigation + * that loads them. + */ +export async function watchBrowserFailures( + page: Page, + options: Readonly<{ preloads?: boolean }> = {}, +): Promise }>> { + const failures: string[] = []; + const onConsole = (message: ConsoleMessage) => { + if (["error", "warning"].includes(message.type()) && !ignoredConsole(message)) { + failures.push(`console:${message.type()}:${message.text()}`); + } + }; + const onPageError = (error: Error) => { + failures.push(`pageerror:${error.name}`); + }; + const onRequestFailed = (request: import("@playwright/test").Request) => { + failures.push( + `requestfailed:${request.method()}:${new URL(request.url()).pathname}:${request.failure()?.errorText ?? "unknown"}`, + ); + }; + page.on("console", onConsole); + page.on("pageerror", onPageError); + page.on("requestfailed", onRequestFailed); + const preloads = options.preloads ? await trackPreloadUse(page) : null; + return Object.freeze({ + async stop() { + if (preloads) failures.push(...(await preloads.check()).failures); + page.off("console", onConsole); + page.off("pageerror", onPageError); + page.off("requestfailed", onRequestFailed); + return [...failures]; + }, + }); +} + /** * 1105/11-01. What the strict `page` fixture does with every failure it * recorded, once the test has ended. @@ -62,32 +104,19 @@ export const test = base.extend<{ ); }, page: async ({ page, browserFailureVerdict }, use) => { - const preloads = await trackPreloadUse(page); - const failures: string[] = []; - const onConsole = (message: ConsoleMessage) => { - if ( - ["error", "warning"].includes(message.type()) && - !ignoredConsole(message) - ) { - failures.push(`console:${message.type()}:${message.text()}`); - } + const watched = [await watchBrowserFailures(page, { preloads: true })]; + // 1112/03-05. A second tab of the same context (the cross-tab specs) + // went unwatched. Its listeners are attached as the tab opens, before + // the spec can navigate it; preload use is checked on the first page only. + const pending: Promise[] = []; + const onPage = (opened: Page) => { + pending.push(watchBrowserFailures(opened).then((watcher) => void watched.push(watcher))); }; - const onPageError = (error: Error) => { - failures.push(`pageerror:${error.name}`); - }; - const onRequestFailed = (request: import("@playwright/test").Request) => { - failures.push( - `requestfailed:${request.method()}:${new URL(request.url()).pathname}:${request.failure()?.errorText ?? "unknown"}`, - ); - }; - page.on("console", onConsole); - page.on("pageerror", onPageError); - page.on("requestfailed", onRequestFailed); + page.context().on("page", onPage); await use(page); - failures.push(...(await preloads.check()).failures); - page.off("console", onConsole); - page.off("pageerror", onPageError); - page.off("requestfailed", onRequestFailed); + page.context().off("page", onPage); + await Promise.all(pending); + const failures = (await Promise.all(watched.map((watcher) => watcher.stop()))).flat(); browserFailureVerdict.judge(failures); }, }); diff --git a/tests/unit/browser-rpc/browser-rpc-remediation.test.ts b/tests/unit/browser-rpc/browser-rpc-remediation.test.ts index 7167788f..8048d718 100644 --- a/tests/unit/browser-rpc/browser-rpc-remediation.test.ts +++ b/tests/unit/browser-rpc/browser-rpc-remediation.test.ts @@ -377,6 +377,19 @@ describe("RPC-RR-04 transport results and frames are exactly decoded", () => { }); }); +/** + * 1101/10-07. The system clock with every wait capped at 30 ms. Without a + * positive receipt the stream cleanup waits out its 1 s bound; capped, a test + * passes that bound instead of sleeping through it. The operations' own 25 ms + * deadlines are below the cap and unchanged. Every stream test that ends + * without a positive receipt takes it, so none sleeps a real second. + */ +const cappedClock: ClockPort = Object.freeze({ + now: () => systemClock.now(), + sleep: (milliseconds: number, signal?: AbortSignal) => + systemClock.sleep(Math.min(milliseconds, 30), signal), +}); + /** * RPC-RR-01. A bare `AsyncIterable` gave the runtime no way to cancel the * physical stream or to learn when it actually closed, so a timed-out call left @@ -395,6 +408,7 @@ describe("RPC-RR-01 server stream leases and the DRAINING fence", () => { mappers: MAPPERS, requestEncoders: { RpcResourceStreamRequestEncoder: STREAM_ENCODER }, transports: { CONNECT_REFERENCE_STREAM: probe.transport }, + clock: cappedClock, }); const results = await collect( @@ -418,6 +432,7 @@ describe("RPC-RR-01 server stream leases and the DRAINING fence", () => { mappers: MAPPERS, requestEncoders: { RpcResourceStreamRequestEncoder: STREAM_ENCODER }, transports: { CONNECT_REFERENCE_STREAM: probe.transport }, + clock: cappedClock, }); const stream = runtime.bindServerStream( "WATCH_RPC_RESOURCES", @@ -541,18 +556,6 @@ describe("RPC-01 only a positive receipt confirms physical closure", () => { { label: "never settles", waitClosed: () => new Promise(() => {}) }, ]; - /** - * 1101/10-07. The system clock with every wait capped at 30 ms. Without a - * positive receipt the stream cleanup waits out its 1 s bound; capped, the - * test passes that bound instead of sleeping through it. The operation's own - * 25 ms deadlines are below the cap and unchanged. - */ - const cappedClock: ClockPort = Object.freeze({ - now: () => systemClock.now(), - sleep: (milliseconds: number, signal?: AbortSignal) => - systemClock.sleep(Math.min(milliseconds, 30), signal), - }); - for (const { label, waitClosed } of negativeReceipts) { it(`keeps the operation DRAINING when waitClosed ${label}`, async () => { const probe = leaseProbeWith(waitClosed); @@ -598,6 +601,7 @@ describe("RPC-01 only a positive receipt confirms physical closure", () => { mappers: MAPPERS, requestEncoders: { RpcResourceStreamRequestEncoder: STREAM_ENCODER }, transports: { CONNECT_REFERENCE_STREAM: probe.transport }, + clock: cappedClock, }); const stream = runtime.bindServerStream( "WATCH_RPC_RESOURCES", diff --git a/tests/unit/browser-spec-imports.test.ts b/tests/unit/browser-spec-imports.test.ts new file mode 100644 index 00000000..596db74f --- /dev/null +++ b/tests/unit/browser-spec-imports.test.ts @@ -0,0 +1,51 @@ +import path from "node:path"; + +import { ESLint } from "eslint"; +import { describe, expect, it } from "vitest"; + +import { childProcessTimeout } from "../helpers/child-process-timeout.ts"; + +const FOLDERS = ["e2e", "browser-capabilities", "visual", "storybook"] as const; + +const BARE = 'import { expect, test } from "@playwright/test";\n\ntest("probe", async () => { expect(1).toBe(1); });\n'; +const STRICT = + 'import { expect, test } from "../support/browser/strict-browser-test.ts";\n' + + 'import type { Page } from "@playwright/test";\n\n' + + 'test("probe", async ({ page }: { page: Page }) => { expect(page).toBeTruthy(); });\n'; +const DEFAULT_IMPORT = 'import test from "@playwright/test";\n\ntest("probe", async () => {});\n'; +const BASE_TEST = 'import { test } from "../support/browser/browser-test.ts";\n\ntest("probe", async () => {});\n'; + +async function restrictedImports(code: string, file: string): Promise { + const linter = new ESLint({ cwd: process.cwd() }); + const [result] = await linter.lintText(code, { filePath: path.join(process.cwd(), file), warnIgnored: true }); + return (result?.messages ?? []) + .filter((message) => message.ruleId === "no-restricted-imports") + .map((message) => message.message); +} + +// 1112/03-04. Strict fixture use was a text search for its path in +// tests/e2e/*.spec.ts: a path in a comment passed, a file could mix in the +// bare test, and tests/browser-capabilities, tests/visual and tests/storybook +// were not looked at. It is an import rule in each of the four folders now. +describe("browser specs take test from the strict fixture (1112/03-04)", () => { + it.each(FOLDERS)("rejects the bare Playwright test in tests/%s", async (folder) => { + const messages = await restrictedImports(BARE, `tests/${folder}/lint-probe.spec.ts`); + expect(messages).toEqual([expect.stringContaining("tests/support/browser/strict-browser-test.ts")]); + }, childProcessTimeout(683)); + + it("rejects the default import, the base test the strict fixture extends, and accepts the strict test", async () => { + expect(await restrictedImports(DEFAULT_IMPORT, "tests/visual/lint-probe.spec.ts")).toEqual([ + expect.stringContaining("tests/support/browser/strict-browser-test.ts"), + ]); + expect(await restrictedImports(BASE_TEST, "tests/e2e/lint-probe.spec.ts")).toEqual([ + expect.stringContaining("the base test does not check browser errors"), + ]); + expect(await restrictedImports(STRICT, "tests/browser-capabilities/lint-probe.spec.ts")).toEqual([]); + }, childProcessTimeout(683)); + + it("lets only the two specs that need the plain test import it", async () => { + for (const file of ["tests/e2e/preload-use.spec.ts", "tests/browser-capabilities/chunk-load-error.spec.ts"]) { + expect(await restrictedImports(BARE, file), file).toEqual([]); + } + }, childProcessTimeout(683)); +}); diff --git a/tests/unit/documentation-readiness.test.ts b/tests/unit/documentation-readiness.test.ts index dbc15222..9374e83e 100644 --- a/tests/unit/documentation-readiness.test.ts +++ b/tests/unit/documentation-readiness.test.ts @@ -250,11 +250,29 @@ describe("verify:documentation commands and sibling links (1103/12-05)", () => { const result = await verify({ [CI_DOC]: (text) => `${text}\n\`corepack pnpm exec vitest run tests/unit/documentation-readiness.test.ts -t "x"\`\n` + "```bash\ncorepack pnpm drill:runbook -- FE-RB-005 --release-root \n```\n" + - "Any script is run as `node scripts/…`.\n", + "Any script is run as `node scripts/…`.\n" + + "`scripts/verify.sh exec -- node scripts/run-vitest.ts run tests/unit/documentation-readiness.test.ts`, " + + "`scripts/verify.sh script verify:documentation lint -- --x`, `scripts/verify.sh script <이름>…`, " + + "`python3 scripts/check-work-docs.py --history <경로>`\n", }); expect(result.status, result.stderr).toBe(0); }, VERIFY_TIMEOUT); + // 1112/08-06. Commands through this repository's entry point, and python3 + // commands, were not read, so a stale path or script name in them passed. + it("reads commands run through scripts/verify.sh and python3", async () => { + const result = await verify({ + [CI_DOC]: (text) => `${text}\n\`scripts/verify.sh exec -- node scripts/no-such.ts\`\n` + + "```bash\nVERIFY_SNAPSHOT=1 scripts/verify.sh exec -- node tests/unit/no-such-shell.test.ts\n```\n" + + "`python3 scripts/no-such.py --history docs`\n`scripts/verify.sh script no:such-script -- --flag`\n", + }); + expect(result.status).toBe(1); + expect(result.stderr).toContain( + `${CI_DOC} names paths that do not exist: scripts/no-such.py, scripts/no-such.ts, tests/unit/no-such-shell.test.ts`, + ); + expect(result.stderr).toContain(`${CI_DOC} names package scripts that do not exist: no:such-script`); + }, VERIFY_TIMEOUT); + it("checks a link or a path into a paired repository checked out beside this one", async () => { const naming = (target: string) => (text: string) => `${text}\n[contract](../../../${target}) and \`${target}\`\n`; diff --git a/tests/unit/e2e-build-helpers.test.ts b/tests/unit/e2e-build-helpers.test.ts new file mode 100644 index 00000000..0a57b4bf --- /dev/null +++ b/tests/unit/e2e-build-helpers.test.ts @@ -0,0 +1,36 @@ +import { readFile, readdir } from "node:fs/promises"; +import path from "node:path"; + +import { describe, expect, it } from "vitest"; + +const SUPPORT = "tests/e2e/support"; + +// 0929/11-07. The Service Worker helper kept its own copy of the checkout +// copy, the build runner and the one-occurrence edit, so a fix to one (the +// local build env of 1103/11-08, the copy skip list of 1105/12-04) had to be +// made twice. Both helpers now go through fixture-builds.ts. +describe("e2e build helpers have one copy-and-build path (0929/11-07)", () => { + it("builds Service Worker fixtures through buildEditedCopies", async () => { + const source = await readFile(path.join(SUPPORT, "service-worker-builds.ts"), "utf8"); + + expect(source).toMatch(/import \{[^}]*\bbuildEditedCopies\b[^}]*\} from "\.\/fixture-builds\.ts";/u); + expect(source).toContain("buildEditedCopies({"); + for (const copy of ["spawnSync", "sharedFixtureBuild", "linkFixtureNodeModules", "await cp(", "function run("]) { + expect(source, copy).not.toContain(copy); + } + }); + + it("keeps the build runner and the one-occurrence edit in fixture-builds.ts only", async () => { + const files = (await readdir(SUPPORT)).filter((file) => file.endsWith(".ts")); + const defining = async (pattern: RegExp) => { + const found: string[] = []; + for (const file of files) { + if (pattern.test(await readFile(path.join(SUPPORT, file), "utf8"))) found.push(file); + } + return found; + }; + + expect(await defining(/^function run\(/mu)).toEqual(["fixture-builds.ts"]); + expect(await defining(/split\(search\)\.length !== 2/u)).toEqual(["fixture-builds.ts"]); + }); +}); diff --git a/tests/unit/hosting-probe.test.ts b/tests/unit/hosting-probe.test.ts index 7d82e34d..3dda020b 100644 --- a/tests/unit/hosting-probe.test.ts +++ b/tests/unit/hosting-probe.test.ts @@ -73,11 +73,18 @@ describe("live hosting evidence target", () => { }); }); + // 0929/13-06. A release built for VITE_ROUTER_BASE_PATH=/app/ is served + // under /app/; the probe refused any URL but the root, so it could not be used. + it("accepts the sub-path a release is built for", () => { + expect(classifyLiveHostingBaseUrl("https://frontend.example.test/app/")).toMatchObject({ passed: true }); + }); + it.each([ ["http://frontend.example.test/", "requires HTTPS"], ["https://localhost:4173/", "not live deployment evidence"], ["https://127.0.0.1/", "not live deployment evidence"], - ["https://frontend.example.test/app/", "canonical root URL"], + ["https://frontend.example.test/app", "path ending in \"/\""], + ["https://frontend.example.test/?preview=1", "without query or fragment"], ["https://user:secret@frontend.example.test/", "must not contain credentials"], ])("rejects %s", (url, reason) => { expect(classifyLiveHostingBaseUrl(url)).toMatchObject({ @@ -193,6 +200,62 @@ describe("SPA fallback and worker script surfaces (1101/13-04)", () => { expect(await failed({ ...RELEASE, "/__probe/deep-link": [200, INDEX] })).toEqual([]); }); + // 0929/13-06. Under a base path the surfaces are requested below it. + it("probes a release built for /app/ below /app/ and passes it", async () => { + const appIndex = ''; + const requested: string[] = []; + const routes: Record = { + "/app/": [200, appIndex], + "/app/config.json": [200, CONFIG], + "/app/release-manifest.json": [200, JSON.stringify({ schemaVersion: 2, ...IDENTITY })], + "/app/__probe/deep-link": [200, appIndex], + "/app/assets/index-A1.js": [200, ""], + }; + const { results } = await probeLiveRelease( + new URL("https://frontend.example.test/app/"), + PATHS, + { ...CANDIDATE, indexHtml: appIndex }, + { + fetcher: async (url) => { + requested.push(url.pathname); + return host(routes)(url); + }, + negotiateAlpn: async () => "h2", + alpnProtocols: ["h2"], + }, + ); + + expect(requested).toEqual([ + "/app/", + "/app/config.json", + "/app/release-manifest.json", + "/app/__probe/deep-link", + "/app/assets/index-A1.js", + ]); + expect(results.filter((result) => !result.passed)).toEqual([]); + }); + + it("fails a HOSTING_BASE_URL path the candidate was not built for", async () => { + expect( + (await probeLiveRelease(new URL("https://frontend.example.test/app/"), PATHS, CANDIDATE, { + fetcher: host({}), + negotiateAlpn: async () => "h2", + alpnProtocols: ["h2"], + })).results.find((result) => result.header === "base-path"), + ).toMatchObject({ expected: "/app/assets/", observed: "/assets/index-A1.js", passed: false }); + // And the other way: a release built for /app/ probed at the root. + expect( + (await probeLiveRelease(base, PATHS, { + ...CANDIDATE, + indexHtml: '', + }, { + fetcher: host({}), + negotiateAlpn: async () => "h2", + alpnProtocols: ["h2"], + })).results.find((result) => result.header === "base-path"), + ).toMatchObject({ expected: "/assets/", observed: "/app/assets/index-A1.js", passed: false }); + }); + it("fails a host without the fallback", async () => { expect(await failed(RELEASE)).toEqual(["appRoute http-status", "appRoute entry-script"]); }); diff --git a/tests/unit/route-chunks.test.ts b/tests/unit/route-chunks.test.ts new file mode 100644 index 00000000..99142cad --- /dev/null +++ b/tests/unit/route-chunks.test.ts @@ -0,0 +1,74 @@ +import { readFile } from "node:fs/promises"; + +import { describe, expect, it } from "vitest"; + +import { findRouteChunk } from "../../scripts/lib/build-manifest-outputs.ts"; + +// 0929/13-03. The route chunk was found by the chunk's `name`, so naming route +// chunks in vite.config.ts (the way tech-log did) stopped the build with +// `Missing built route chunk: APP_HOME` and no reason. It is found by the +// page's source file now, which is what the route runtime's moduleId names. +describe("route chunk lookup (0929/13-03)", () => { + const manifest = { + "index.html": { file: "assets/index-a.js", isEntry: true }, + "src/presentation/pages/home-page.tsx": { + file: "assets/route-home-page-b.js", + name: "route-home-page", + src: "src/presentation/pages/home-page.tsx", + isDynamicEntry: true, + }, + "src/features/order-feature/presentation/order-feature-list-page.tsx": { + file: "assets/order-feature-list-page-c.js", + name: "order-feature-list-page", + isDynamicEntry: true, + }, + "_home-page-shared.js": { file: "assets/home-page-d.js", name: "home-page" }, + }; + + it("finds the chunk by the page's source file even when the chunk has another name", () => { + expect(findRouteChunk(manifest, "APP_HOME", "home-page")).toEqual({ file: "assets/route-home-page-b.js" }); + expect(findRouteChunk(manifest, "ORDER_FEATURE_LIST", "order-feature-list-page")).toEqual({ + file: "assets/order-feature-list-page-c.js", + }); + }); + + it("names the reason and the rule when no lazily imported page has that file name", () => { + const lookup = findRouteChunk(manifest, "NOT_FOUND", "not-found-page"); + expect(lookup).toEqual({ + error: + "Missing built route chunk: NOT_FOUND: no lazily imported module in the Vite manifest has the source " + + "file not-found-page.tsx (or .ts). A route runtime's moduleId must be the file name of the page it imports.", + }); + }); + + it("refuses two lazily imported pages with the same file name", () => { + const lookup = findRouteChunk( + { ...manifest, "src/other/home-page.tsx": { file: "assets/home-page-e.js", isDynamicEntry: true } }, + "APP_HOME", + "home-page", + ); + expect(lookup).toEqual({ + error: + "Ambiguous built route chunk: APP_HOME: the lazily imported modules " + + "src/presentation/pages/home-page.tsx, src/other/home-page.tsx are all named home-page.", + }); + }); + + it("names a route without a runtime contract", () => { + expect(findRouteChunk(manifest, "GHOST", undefined)).toEqual({ + error: "Missing built route chunk: GHOST has no route runtime contract (ROUTE_RUNTIME_CONTRACT)", + }); + }); + + it("is the one lookup the manifest writer and both release checks use", async () => { + for (const file of [ + "scripts/generate-build-manifest.ts", + "scripts/verify-release.ts", + "scripts/lib/local-release-evidence.ts", + ]) { + const source = await readFile(file, "utf8"); + expect(source, file).toContain("findRouteChunk("); + expect(source, file).not.toMatch(/\.name === runtime\?\.moduleId/u); + } + }); +}); diff --git a/tests/unit/run-isolation.test.ts b/tests/unit/run-isolation.test.ts index b4bd7d94..873bb860 100644 --- a/tests/unit/run-isolation.test.ts +++ b/tests/unit/run-isolation.test.ts @@ -1,3 +1,4 @@ +import { spawnSync } from "node:child_process"; import { createServer, type Server } from "node:net"; import { afterEach, describe, expect, it, vi } from "vitest"; @@ -12,6 +13,7 @@ import { testRunPath, testRunRoot, } from "../../scripts/lib/run-isolation.ts"; +import { childProcessTimeout } from "../helpers/child-process-timeout.ts"; const servers: Server[] = []; @@ -110,6 +112,28 @@ describe("build output root (RUN_OUTPUT_ROOT)", () => { /^check-bundle: RUN_OUTPUT_ROOT must be/u, ); }); + + // 1106/12-02, the part the gates did not cover. Run alone, these four read + // the checkout's dist/ or artifacts/ and did not refuse another root: + // `RUN_OUTPUT_ROOT=.tmp/x corepack pnpm verify:reproducible-build` wrote + // `passed: true` about the earlier build. + it.each([ + ["scripts/verify-reproducible-build.ts"], + ["scripts/check-optional-recipes.ts", "--require-dist"], + ["scripts/check-app-build-input.ts", "--require-bundle"], + ["scripts/create-release-candidate.ts"], + ])("%s refuses to read the checkout's build under another RUN_OUTPUT_ROOT", (script, ...args) => { + const { CI: _ci, ...environment } = process.env; + const result = spawnSync(process.execPath, [script, ...args], { + encoding: "utf8", + env: { ...environment, RUN_OUTPUT_ROOT: ".tmp/run-output-refusal" }, + }); + + expect(result.status, result.stderr).toBe(1); + expect(result.stderr).toContain( + `${script} reads the build at this checkout's paths (dist/, artifacts/), but RUN_OUTPUT_ROOT=.tmp/run-output-refusal`, + ); + }, childProcessTimeout(499)); }); describe("server ports", () => { diff --git a/tests/unit/runbook-drill.test.ts b/tests/unit/runbook-drill.test.ts index 611d4598..afc78619 100644 --- a/tests/unit/runbook-drill.test.ts +++ b/tests/unit/runbook-drill.test.ts @@ -4,7 +4,8 @@ import path from "node:path"; import { afterEach, describe, expect, it } from "vitest"; -import { runRunbookDrill } from "../../scripts/drill-runbook.ts"; +import { runbookRecordArtifactSchema } from "../../scripts/contracts/release-artifacts.ts"; +import { PROVIDER_EVIDENCE_MAX_AGE_MS, runRunbookDrill } from "../../scripts/drill-runbook.ts"; import { computeContractSetDigest } from "../../src/contracts/contract-set-canonical.ts"; import type { InstalledContractPackageIdentity } from "../../src/contracts/external-contract-runtime.ts"; @@ -145,3 +146,106 @@ describe("runbook drill verdicts", () => { expect(source).not.toMatch(/verdict\(\s*true\s*\)/u); }); }); + +// 0929/11-02. The drills with provider scope took only a runbook ID and a +// release root, so no input could ever make them PASS and the production +// stage could not close. Provider evidence now answers the items the local +// drill cannot observe, for this runbook and release only, and while recent. +describe("runbook drill with provider evidence (0929/11-02)", () => { + const NOW = new Date("2026-11-13T09:00:00.000Z"); + const evidence = (overrides: Record = {}) => ({ + schemaVersion: 1, + artifactType: "provider-drill-evidence", + runbookId: "FE-RB-005", + releaseId: "release-a", + observedAt: "2026-11-13T08:30:00.000Z", + runId: "provider-run-41", + observations: [ + { item: "trigger", result: "PASS", evidence: "incident INC-7 declared the release-blocking defect" }, + { item: "핵심 smoke 시험", result: "PASS", evidence: "smoke against the prior release after the switch" }, + { item: "provider verification", result: "PASS", evidence: "provider serves release-a again" }, + ], + ...overrides, + }); + const drill = async (providerEvidence: unknown) => + runRunbookDrill("FE-RB-005", { releaseRoot: await builtDist(), contractPackages, providerEvidence, now: NOW }); + + it("passes FE-RB-005 when the provider observed every item the local drill could not", async () => { + const record = await drill(evidence()); + + expect(record.result).toBe("PASS"); + expect(record.triggerAsserted).toBe("PASS"); + expect(record.recoveryAssertions.find(({ assertion }) => assertion === "핵심 smoke 시험")).toEqual({ + assertion: "핵심 smoke 시험", + evidence: "provider evidence provider-run-41 at 2026-11-13T08:30:00.000Z: smoke against the prior release after the switch", + result: "PASS", + }); + expect(record.providerEvidence).toEqual({ + accepted: true, + runId: "provider-run-41", + observedAt: "2026-11-13T08:30:00.000Z", + providerVerification: "PASS", + }); + const { gateId: _gateId, ...artifact } = record; + expect(runbookRecordArtifactSchema.safeParse(artifact).success).toBe(true); + }); + + it("fails when the provider observed a failure", async () => { + const observations = evidence().observations.map((entry) => + entry.item === "핵심 smoke 시험" ? { ...entry, result: "FAIL" } : entry); + expect((await drill(evidence({ observations }))).result).toBe("FAIL"); + }); + + it("stays UNVERIFIED while an item the local drill could not observe has no answer", async () => { + const record = await drill(evidence({ observations: evidence().observations.slice(0, 2) })); + expect(record.result).toBe("UNVERIFIED"); + expect(record.providerEvidence?.providerVerification).toBe("UNVERIFIED"); + }); + + it.each([ + ["another release", { releaseId: "release-b" }, "evidence is for release release-b, not release-a"], + ["another runbook", { runbookId: "FE-RB-002" }, "evidence is for FE-RB-002, not FE-RB-005"], + [ + "an expired observation", + { observedAt: new Date(NOW.getTime() - PROVIDER_EVIDENCE_MAX_AGE_MS - 1_000).toISOString() }, + "is older than 7 days", + ], + ["a future observation", { observedAt: "2026-11-13T10:00:00.000Z" }, "is in the future"], + [ + "an answer for a check the local drill made", + { observations: [...evidence().observations, { item: "호환성 gate", result: "PASS", evidence: "x" }] }, + "evidence names items the local drill observed or does not have: 호환성 gate", + ], + ["a record that is not provider evidence", { artifactType: "runbook-record" }, "not provider drill evidence"], + ])("does not pass on evidence for %s", async (_case, overrides, reason) => { + const record = await drill(evidence(overrides)); + + expect(record.result).toBe("UNVERIFIED"); + expect(record.triggerAsserted).toBe("UNVERIFIED"); + expect(record.providerEvidence?.accepted).toBe(false); + expect(record.providerEvidence?.reason).toContain(reason); + const { gateId: _gateId, ...artifact } = record; + expect(runbookRecordArtifactSchema.safeParse(artifact).success).toBe(true); + }); + + it("rejects a record that claims PASS without accepted provider evidence", () => { + const record = { + schemaVersion: 2, + runbookId: "FE-RB-005", + releaseId: "release-a", + drillTimestamp: NOW.toISOString(), + triggerInjected: "RELEASE_BLOCKING_DEFECT", + triggerAsserted: "PASS", + containmentAsserted: "PASS", + escalationPathAsserted: "PASS", + recoveryAssertions: [], + negativeFixtureFailedAsExpected: "PASS", + windowObservedBucket: "TBD", + providerVerificationRequired: true, + providerEvidence: { accepted: false, reason: "expired", providerVerification: "UNVERIFIED" }, + result: "PASS", + }; + expect(runbookRecordArtifactSchema.safeParse(record).success).toBe(false); + expect(runbookRecordArtifactSchema.safeParse({ ...record, result: "UNVERIFIED" }).success).toBe(true); + }); +}); diff --git a/tests/unit/system-test-prerequisites.test.ts b/tests/unit/system-test-prerequisites.test.ts index 832a9b51..af623075 100644 --- a/tests/unit/system-test-prerequisites.test.ts +++ b/tests/unit/system-test-prerequisites.test.ts @@ -9,38 +9,58 @@ import { NAMESPACE_PROBE_ARGUMENTS, type SystemTestHost, } from "../../scripts/check-system-test-prerequisites.ts"; +import { + systemdRunPrerequisiteProbeArguments, + systemdRunProviderArguments, +} from "../../scripts/lib/provider-cgroup.ts"; import { PROVIDER_SANDBOX_NAMESPACE_ARGUMENTS } from "../../scripts/lib/provider-sandbox-namespaces.ts"; type SpawnCall = Readonly<{ command: string; args: readonly string[] }>; type SpawnResult = ReturnType; const RTM_NEWADDR = "bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted"; +const PROBE_UNIT = "ca-provider-prerequisite-4242-0123456789abcdef01234567.scope"; const USER_MANAGER_CONTROLLERS = "/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/cgroup.controllers"; -/** A host that has every prerequisite, except what `bwrap` and `files` say. */ +type HostOverrides = Readonly<{ + files?: Readonly>; + missing?: readonly string[]; + userManager?: SpawnResult; + systemdRun?: SpawnResult; +}>; + +/** A host that has every prerequisite, except what `bwrap` and the overrides say. */ function fakeHost( bwrap: SpawnResult, - files: Readonly> = {}, + options: HostOverrides = {}, ): { host: SystemTestHost; calls: SpawnCall[] } { const calls: SpawnCall[] = []; const host: SystemTestHost = { - access: async () => undefined, + access: async (target) => { + if (options.missing?.includes(target)) throw new Error(`EACCES: ${target}`); + }, readFile: async (target) => { - const content = { + const files: Readonly> = { "/sys/fs/cgroup/cgroup.controllers": "cpuset cpu io memory pids\n", "/proc/self/status": "Name:\tnode\nUid:\t1000\t1000\t1000\t1000\n", [USER_MANAGER_CONTROLLERS]: "cpu memory pids\n", - ...files, - }[target]; + ...options.files, + }; + const content = files[target]; if (content === undefined) throw new Error(`ENOENT: ${target}`); return content; }, spawn: (command, args) => { calls.push({ command, args }); - if (command === "/usr/bin/systemctl") return { status: 0, stderr: "" }; + if (command === "/usr/bin/systemctl" && args[0] === "--user") { + return options.userManager ?? { status: 0, stdout: "255\n", stderr: "" }; + } + if (command === "/usr/bin/systemctl") return { status: 0, stdout: "", stderr: "" }; + if (command === "/usr/bin/systemd-run") return options.systemdRun ?? { status: 0, stdout: "", stderr: "" }; if (command === "/usr/bin/bwrap") return bwrap; throw new Error(`unexpected spawn: ${command}`); }, + probeUnitName: () => PROBE_UNIT, }; return { host, calls }; } @@ -79,7 +99,7 @@ describe("system test prerequisites (1105/12-01)", () => { it("fails with the namespace restriction, bwrap's stderr and what to do when bwrap cannot set up loopback", async () => { const { host } = fakeHost( { status: 1, stderr: `${RTM_NEWADDR}\n` }, - { "/proc/sys/kernel/apparmor_restrict_unprivileged_userns": "1\n" }, + { files: { "/proc/sys/kernel/apparmor_restrict_unprivileged_userns": "1\n" } }, ); const failures = await findSystemTestPrerequisiteFailures(host); @@ -108,7 +128,7 @@ describe("system test prerequisites (1105/12-01)", () => { expect(failures[0]).toContain("did not finish: spawnSync /usr/bin/bwrap ETIMEDOUT"); }); - it("keeps the executable, cgroup v2 and systemd bus checks, and probes bwrap only when it is there", async () => { + it("keeps the executable, cgroup v2 and systemd checks, and probes bwrap only when it is there", async () => { const calls: SpawnCall[] = []; const host: SystemTestHost = { access: async (target) => { @@ -121,15 +141,98 @@ describe("system test prerequisites (1105/12-01)", () => { }, }; - expect(await findSystemTestPrerequisiteFailures(host)).toEqual([ + const failures = await findSystemTestPrerequisiteFailures(host); + expect(failures.slice(0, 2)).toEqual([ "bubblewrap is required at /usr/bin/bwrap", "cgroup v2 controllers are unavailable", - "a reachable systemd manager bus is required", ]); + expect(failures[2]).toMatch(/^a reachable systemd user manager is required/u); + expect(failures).toHaveLength(3); expect(calls.map(({ command }) => command)).toEqual(["/usr/bin/systemctl"]); }); }); +// 1112/08-03. The check looked at the system manager bus while the provider +// asks the user manager for its version and starts a `systemd-run --user +// --scope`, so a host without a user manager, with an old one, or without +// prlimit, systemd-run or delegated controllers printed PASS and then failed +// each provider test with `provider cgroup unavailable: ...`. +describe("system test prerequisites match the provider's own calls (1112/08-03)", () => { + it("fails by name when the system manager answers but the user manager does not", async () => { + const { host, calls } = fakeHost( + { status: 0, stderr: "" }, + { userManager: { status: 1, stdout: "", stderr: "Failed to connect to bus: No medium found\n" } }, + ); + + const failures = await findSystemTestPrerequisiteFailures(host); + + expect(failures).toHaveLength(1); + expect(failures[0]).toMatch(/^a reachable systemd user manager is required/u); + expect(failures[0]).toContain("/usr/bin/systemctl --user show --property=Version --value exited 1"); + expect(failures[0]).toContain("systemctl stderr: Failed to connect to bus: No medium found"); + expect(calls.some(({ command }) => command === "/usr/bin/systemd-run")).toBe(false); + }); + + it("fails when the user manager is older than the provider needs", async () => { + const { host } = fakeHost({ status: 0, stderr: "" }, { userManager: { status: 0, stdout: "253\n", stderr: "" } }); + + expect(await findSystemTestPrerequisiteFailures(host)).toEqual([ + 'systemd user manager 254 or newer is required (/usr/bin/systemctl --user show --property=Version --value reported "253")', + ]); + }); + + it("requires the prlimit and systemd-run executables the provider runs", async () => { + const { host } = fakeHost({ status: 0, stderr: "" }, { missing: ["/usr/bin/prlimit", "/usr/bin/systemd-run"] }); + + expect(await findSystemTestPrerequisiteFailures(host)).toEqual([ + "prlimit is required at /usr/bin/prlimit", + "systemd-run is required at /usr/bin/systemd-run", + ]); + }); + + it("starts the provider's own scope once and names a scope that cannot enforce its limits", async () => { + const { host, calls } = fakeHost( + { status: 0, stderr: "" }, + { systemdRun: { status: 125, stdout: "", stderr: "provider cgroup enforcement failed: cannot read memory.max\n" } }, + ); + + const failures = await findSystemTestPrerequisiteFailures(host); + + expect(failures).toHaveLength(1); + expect(failures[0]).toMatch(/^systemd-run could not start the provider's cgroup scope/u); + expect(failures[0]).toContain("exited 125"); + expect(failures[0]).toContain("systemd-run stderr: provider cgroup enforcement failed: cannot read memory.max"); + expect(failures[0]).toMatch(/memory, pids and cpu controllers delegated/u); + + const scopes = calls.filter(({ command }) => command === "/usr/bin/systemd-run"); + expect(scopes).toHaveLength(1); + expect(scopes[0]?.args).toEqual(systemdRunPrerequisiteProbeArguments(PROBE_UNIT)); + expect(scopes[0]?.args.at(-1)).toBe("/bin/true"); + }); + + it("probes with the provider scope's properties and enforcement gate, not a copy", () => { + const probe = systemdRunPrerequisiteProbeArguments(PROBE_UNIT); + const provider = systemdRunProviderArguments( + "ca-provider-vulnerability-4242-0123456789abcdef01234567.scope", + 60_000, + 60, + "/usr/bin/node", + "/opt/wrapper.ts", + "/tmp/report.json", + 1, + 1, + ); + const options = (args: readonly string[]) => + args.filter((arg) => arg.startsWith("--") && !arg.startsWith("--unit=") && !arg.startsWith("--property=RuntimeMaxSec=")); + expect(options(probe)).toEqual(options(provider)); + const gate = (args: readonly string[]) => args[args.indexOf("-c") + 1]; + expect(gate(probe)).toBe(gate(provider)); + expect(gate(probe)).toContain("require_cgroup_value memory.max"); + expect(() => systemdRunPrerequisiteProbeArguments("ca-provider-vulnerability-1-0123456789abcdef01234567.scope")) + .toThrow(/unit is invalid/u); + }); +}); + // 2026-11-14. In one system container in four, systemd delegated no cgroup // controller to the root user manager; this check printed PASS and then every // provider stopped with `provider cgroup enforcement failed: cannot read @@ -139,7 +242,7 @@ describe("system test prerequisites: provider scope controllers", () => { ["none", "\n", "cpu, memory, pids", "none"], ["memory", "cpu pids\n", "memory", "cpu pids"], ])("names the controllers the user manager lacks (%s missing)", async (_label, delegated, missing, has) => { - const { host } = fakeHost({ status: 0, stderr: "" }, { [USER_MANAGER_CONTROLLERS]: delegated }); + const { host } = fakeHost({ status: 0, stderr: "" }, { files: { [USER_MANAGER_CONTROLLERS]: delegated } }); expect(await findSystemTestPrerequisiteFailures(host)).toEqual([ "the systemd user manager at /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service " + @@ -148,13 +251,16 @@ describe("system test prerequisites: provider scope controllers", () => { }); it("reads the user manager only where cgroup v2 is there", async () => { - const { host } = fakeHost({ status: 0, stderr: "" }, { "/sys/fs/cgroup/cgroup.controllers": "\n", [USER_MANAGER_CONTROLLERS]: "\n" }); + const { host } = fakeHost( + { status: 0, stderr: "" }, + { files: { "/sys/fs/cgroup/cgroup.controllers": "\n", [USER_MANAGER_CONTROLLERS]: "\n" } }, + ); expect(await findSystemTestPrerequisiteFailures(host)).toEqual(["cgroup v2 controllers are unavailable"]); }); it("names an unreadable uid", async () => { - const { host } = fakeHost({ status: 0, stderr: "" }, { "/proc/self/status": "Name:\tnode\n" }); + const { host } = fakeHost({ status: 0, stderr: "" }, { files: { "/proc/self/status": "Name:\tnode\n" } }); expect(await findSystemTestPrerequisiteFailures(host)).toEqual([ "this process's uid is unreadable at /proc/self/status", @@ -162,7 +268,7 @@ describe("system test prerequisites: provider scope controllers", () => { }); it("requires a user manager for this process's own uid", async () => { - const { host } = fakeHost({ status: 0, stderr: "" }, { "/proc/self/status": "Uid:\t0\t0\t0\t0\n" }); + const { host } = fakeHost({ status: 0, stderr: "" }, { files: { "/proc/self/status": "Uid:\t0\t0\t0\t0\n" } }); expect(await findSystemTestPrerequisiteFailures(host)).toEqual([ "a systemd user manager is required at /sys/fs/cgroup/user.slice/user-0.slice/user@0.service", @@ -185,7 +291,7 @@ describe("provider sandbox namespace check (1101/13-03)", () => { it("names the restriction, bwrap's stderr and the host settings on a host that refuses them", async () => { const { host } = fakeHost( { status: 1, stderr: `${RTM_NEWADDR}\n` }, - { "/proc/sys/kernel/apparmor_restrict_unprivileged_userns": "1\n" }, + { files: { "/proc/sys/kernel/apparmor_restrict_unprivileged_userns": "1\n" } }, ); const failure = (await findProviderSandboxNamespaceFailure(host)) ?? ""; diff --git a/tests/unit/test-evidence-source-rules.test.ts b/tests/unit/test-evidence-source-rules.test.ts index 074cb705..53a13f59 100644 --- a/tests/unit/test-evidence-source-rules.test.ts +++ b/tests/unit/test-evidence-source-rules.test.ts @@ -1,9 +1,10 @@ import { spawnSync } from "node:child_process"; -import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; import { afterAll, describe, expect, it } from "vitest"; +import { selectRepositoryPaths } from "../../scripts/lib/repository-layout.ts"; import { childProcessTimeout } from "../helpers/child-process-timeout.ts"; const FORBIDDEN = "tests/fixtures/test-evidence/forbidden"; @@ -13,7 +14,7 @@ afterAll(async () => { await Promise.all(roots.map((root) => rm(root, { recursive: true, force: true }))); }); -function checkSource(sourceRoot: string) { +function checkSource(sourceRoot: string, extra: readonly string[] = []) { const result = spawnSync( process.execPath, [ @@ -23,6 +24,7 @@ function checkSource(sourceRoot: string) { sourceRoot, "--artifact", "artifacts/quality/test-evidence-source-rules.json", + ...extra, ], { encoding: "utf8" }, ); @@ -44,6 +46,44 @@ describe("test evidence source rules", () => { expect(result.stderr).toContain("quarantine expired on 2020-01-01"); }, childProcessTimeout(739)); + // 1112/03-03. The mask fixture had no test of its own: the directory run + // expects only the common prefix, so this rule could stop firing unseen. + it("rejects a screenshot mask over the whole UI", () => { + const result = checkSource(`${FORBIDDEN}/full-screen-mask.fixture`); + expect(result.status).toBe(1); + expect(result.stderr).toContain("screenshot mask may not cover the whole UI"); + }, childProcessTimeout(141)); + + // 1112/03-03. The forbidOnly rule ran only in the full check, on the + // repository's own configs, so nothing showed it still rejects a config. + it("rejects a Playwright config without forbidOnly and accepts the repository's", async () => { + const root = await mkdtemp(path.join(tmpdir(), "test-evidence-config-")); + roots.push(root); + await writeFile(path.join(root, "clean.fixture"), "export {};\n"); + + const missing = checkSource(root, [ + "--playwright-config", + "tests/fixtures/test-evidence/playwright-config/missing-forbid-only.fixture", + ]); + expect(missing.status).toBe(1); + expect(missing.stderr).toContain( + "tests/fixtures/test-evidence/playwright-config/missing-forbid-only.fixture: missing forbidOnly: true", + ); + + const present = checkSource(root, ["--playwright-config", "playwright.config.ts"]); + expect(present.stderr).toBe(""); + expect(present.status).toBe(0); + + // Without the flag the rule reads the repository's configs, every one. + const repository = checkSource(root); + expect(repository.status, repository.stderr).toBe(0); + const report = JSON.parse(await readFile("artifacts/quality/test-evidence-source-rules.json", "utf8")) as { + facts: { playwrightConfigs?: string[] }; + }; + expect(report.facts.playwrightConfigs).toEqual([...selectRepositoryPaths("playwright-configs")]); + expect(report.facts.playwrightConfigs?.length).toBeGreaterThan(0); + }, childProcessTimeout(296)); + it("rejects a skip without a quarantine", () => { const result = checkSource(`${FORBIDDEN}/unowned-skip.fixture`); expect(result.status).toBe(1); diff --git a/tests/visual/scrollbar-gutter.visual.spec.ts b/tests/visual/scrollbar-gutter.visual.spec.ts index 6ad4e93a..0df6d62b 100644 --- a/tests/visual/scrollbar-gutter.visual.spec.ts +++ b/tests/visual/scrollbar-gutter.visual.spec.ts @@ -66,7 +66,9 @@ async function stripColours( new Blob([bytes], { type: "image/png" }), ); const canvas = new OffscreenCanvas(bitmap.width, bitmap.height); - const context = canvas.getContext("2d"); + // Read back once per strip: without the hint Chromium warns on the + // console of this tab, which the strict fixture watches (1112/03-05). + const context = canvas.getContext("2d", { willReadFrequently: true }); if (context === null) throw new Error("no 2d canvas context"); context.drawImage(bitmap, 0, 0); return offsets.map(({ x, width }) => { -- 2.54.0 From 40a8d970dfb7a5825e7ebdea574f383730cc08e8 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Thu, 19 Nov 2026 16:32:00 +0900 Subject: [PATCH 2/2] =?UTF-8?q?fix(test-evidence):=20fixture=20=EC=8B=A4?= =?UTF-8?q?=ED=96=89=EC=9D=80=20=EB=B0=9B=EC=9D=80=20Playwright=20?= =?UTF-8?q?=EC=84=A4=EC=A0=95=EB=A7=8C=20=EC=9D=BD=EB=8A=94=EB=8B=A4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 문제: #55가 scripts/check-test-evidence.ts의 forbidOnly 검사를 `--playwright-config`가 없을 때도 fixture 모드에서 저장소 설정 (repository-layout의 playwright-configs)을 읽게 바꿨다. browser file/storage 제거 드릴 사본은 playwright.capabilities.config.ts를 지우지만 등록부 항목은 남긴다. 그래서 사본의 tests/unit/test-evidence-source-rules.test.ts가 fixture 모드로 부르는 검사 6건이 모두 ENOENT로 죽는다(main run 376, FE-GATE-030). 고른 방법 (가) 검사 쪽: 설정을 인자로 받지 않은 fixture 실행은 설정을 하나도 읽지 않는다. 전체 검사는 전처럼 등록부의 설정을 모두 읽는다. `--playwright-config`는 그대로 모든 모드(시나리오 검사 빼고)에서 돈다. 이유: 원인은 #55가 fixture 실행의 입력을 저장소 파일로 넓힌 것이다. 1112/03-03 지적의 최소 개선 방향은 설정을 인자로 받고 forbidOnly 없는 설정을 거절 fixture로 두는 것뿐이다. 기본값으로 저장소를 읽는 부분은 지적 밖이다. fixture 실행이 받은 파일에만 기대면 저장소 사본마다 어떤 설정이 남았는지와 상관없이 같은 결과를 낸다. 전체 검사 기준은 바뀌지 않는다. #55는 fixture 실행으로 「기본값은 등록부 설정을 모두 읽는다」를 단언했다. 그 단언은 fixture 실행이 설정을 읽지 않는다로 바꾸고, 전체 검사(`--source-only`)가 등록부 설정을 모두 읽는지는 새 시험 tests/unit/test-evidence-repository-configs.test.ts가 본다. 그 시험은 사본에서 지운 설정을 읽으므로, 드릴이 playwright.capabilities.config.ts와 그 run-isolation 시험을 지울 때 같이 이름으로 지운다. 버린 대안 (나) 드릴 쪽: 드릴이 사본의 config/repository-layout.json에서 그 항목도 지운다. 등록부를 고치는 드릴이 없고(recipes 드릴은 항목의 optional 표시에 기댄다), 다른 사본에서 fixture 실행이 저장소 설정을 읽는 문제는 남는다. 버린 대안 (가) 안의 건너뛰기: 없는 설정을 건너뛰면 설정이 사라져도 조용히 통과한다. 막으려면 등록부 항목을 optional로 바꿔야 하는데, 그 표시는 fixture-copy·secret-scan 집합에도 같이 걸리고 FE-GATE-008의 전체 검사도 설정이 빠진 것을 넘기게 된다. 위험·한계: 다운스트림이 이 드릴처럼 playwright.capabilities.config.ts를 지우고 등록부 항목을 남기면 전체 검사가 ENOENT로 죽는다. #55 이전부터 같다. 새 시험은 전체 검사 하나를 자식 프로세스로 돌린다(desktop 단독 3회 210~232ms). 검증(desktop, scripts/verify.sh, 공유 잠금, CI=true): - RED: tests/unit/test-evidence-source-rules.test.ts의 새 단언을 앞 커밋 검사기에 돌림 → rc=1, 1 failed(기대 [], 받은 값 설정 4개). 로그 /tmp/claude-gates/fe-reland-red-unit/ - 변이: 전체 모드 기본값을 []로 → 새 시험만 실패(expected [] to deeply equal [ …(4) ]), source-rules 6건 통과. /tmp/claude-gates/fe-reland-mutation1/ - 이 트리(7f1af01e)에서 모두 rc=0: - 두 시험 파일 /tmp/claude-gates/fe-reland2-unit/ - gate FE-GATE-030 371초. optional recipe·browser file/storage·realtime 드릴 모두 PASS. 새 시험은 recipe·realtime 사본에서 돌고 browser file 사본에서는 지워진다. /tmp/claude-gates/fe-reland2-gate030/ - gate FE-GATE-029 24초. /tmp/claude-gates/fe-reland2-gate029/ - merge 단계 14개(FE-GATE-013 시스템 컨테이너 안, FE-GATE-020 포함). /tmp/claude-gates/fe-reland2-merge/ Refs: #55, 1112/03-03 Fixes: 34a7627d3628 ("fix(test-evidence): 검사 규칙마다 혼자 도는 거절 시험을 두고 strict import를 lint로 본다") --- scripts/check-test-evidence.ts | 12 ++++++-- ...st-browser-file-storage-runtime-removal.ts | 5 +++- .../test-evidence-repository-configs.test.ts | 30 +++++++++++++++++++ tests/unit/test-evidence-source-rules.test.ts | 14 +++++---- 4 files changed, 51 insertions(+), 10 deletions(-) create mode 100644 tests/unit/test-evidence-repository-configs.test.ts diff --git a/scripts/check-test-evidence.ts b/scripts/check-test-evidence.ts index 4c9002b2..f7a5357b 100644 --- a/scripts/check-test-evidence.ts +++ b/scripts/check-test-evidence.ts @@ -382,13 +382,19 @@ async function missingForbidOnly(configs: readonly string[]): Promise return missing; } -// The repository's configs unless the run names others; every mode but the -// scenario check reads them, and the report lists what it read. +// The configs the run names, in every mode but the scenario check; without +// them the full check reads the repository's and a fixture run reads none. +// A fixture run depends only on what it is given: a removal drill's copy +// deletes playwright.capabilities.config.ts but keeps its registry entry, and +// every fixture run there failed on ENOENT when it read the registry (main run +// 376, FE-GATE-030). The report lists what was read. if (!scenarioOnly) { // playwright.dev.config.ts spreads playwright.config.ts. facts.playwrightConfigs = explicitPlaywrightConfigs.length > 0 ? explicitPlaywrightConfigs - : [...selectRepositoryPaths("playwright-configs")]; + : fixtureMode + ? [] + : [...selectRepositoryPaths("playwright-configs")]; failures.push(...(await missingForbidOnly(facts.playwrightConfigs))); } diff --git a/scripts/test-browser-file-storage-runtime-removal.ts b/scripts/test-browser-file-storage-runtime-removal.ts index 0607e916..f256bcba 100644 --- a/scripts/test-browser-file-storage-runtime-removal.ts +++ b/scripts/test-browser-file-storage-runtime-removal.ts @@ -134,10 +134,13 @@ try { // 1106/10-01. The capabilities config goes with this runtime, and so do its // run-isolation checks, by name: the template's test:unit runs them, this // copy has nothing for them to check. Without `force`, so a renamed file - // stops the drill here instead of failing its typecheck later. + // stops the drill here instead of failing its typecheck later. The test that + // the full test evidence check reads every registered Playwright config goes + // too: the registry still names the one deleted here. for (const capabilityConfigFile of [ "playwright.capabilities.config.ts", "tests/unit/playwright-capabilities-run-isolation.test.ts", + "tests/unit/test-evidence-repository-configs.test.ts", ]) { await rm(path.join(fixtureRoot, capabilityConfigFile)); } diff --git a/tests/unit/test-evidence-repository-configs.test.ts b/tests/unit/test-evidence-repository-configs.test.ts new file mode 100644 index 00000000..267861f7 --- /dev/null +++ b/tests/unit/test-evidence-repository-configs.test.ts @@ -0,0 +1,30 @@ +import { spawnSync } from "node:child_process"; +import { readFile, rm } from "node:fs/promises"; + +import { describe, expect, it } from "vitest"; +import { selectRepositoryPaths } from "../../scripts/lib/repository-layout.ts"; +import { childProcessTimeout } from "../helpers/child-process-timeout.ts"; + +const ARTIFACT = "artifacts/quality/test-evidence-repository-configs.json"; + +// The full check reads every Playwright config of the registry for forbidOnly; +// a fixture run reads only the configs it is given +// (tests/unit/test-evidence-source-rules.test.ts). Every config has forbidOnly, +// so nothing else fails if the full check reads none. The browser file/storage +// removal drill deletes this test with playwright.capabilities.config.ts. +describe("test evidence full check", () => { + it("reads every Playwright config of the registry", async () => { + await rm(ARTIFACT, { force: true }); + const result = spawnSync( + process.execPath, + ["scripts/check-test-evidence.ts", "--source-only", "--artifact", ARTIFACT], + { encoding: "utf8" }, + ); + expect(result.status, result.stderr).toBe(0); + const report = JSON.parse(await readFile(ARTIFACT, "utf8")) as { + facts: { playwrightConfigs?: string[] }; + }; + expect(report.facts.playwrightConfigs).toEqual([...selectRepositoryPaths("playwright-configs")]); + expect(report.facts.playwrightConfigs?.length).toBeGreaterThan(0); + }, childProcessTimeout(232)); +}); diff --git a/tests/unit/test-evidence-source-rules.test.ts b/tests/unit/test-evidence-source-rules.test.ts index 53a13f59..30e063cb 100644 --- a/tests/unit/test-evidence-source-rules.test.ts +++ b/tests/unit/test-evidence-source-rules.test.ts @@ -4,7 +4,6 @@ import { tmpdir } from "node:os"; import path from "node:path"; import { afterAll, describe, expect, it } from "vitest"; -import { selectRepositoryPaths } from "../../scripts/lib/repository-layout.ts"; import { childProcessTimeout } from "../helpers/child-process-timeout.ts"; const FORBIDDEN = "tests/fixtures/test-evidence/forbidden"; @@ -74,14 +73,17 @@ describe("test evidence source rules", () => { expect(present.stderr).toBe(""); expect(present.status).toBe(0); - // Without the flag the rule reads the repository's configs, every one. - const repository = checkSource(root); - expect(repository.status, repository.stderr).toBe(0); + // Without the flag a fixture run reads no config, so its result depends + // only on the files it names. A removal drill's copy keeps the registry + // but deletes playwright.capabilities.config.ts; when this run read the + // registry's configs, every fixture run there failed on ENOENT (main run + // 376, FE-GATE-030). The full check still reads every one. + const fixtureOnly = checkSource(root); + expect(fixtureOnly.status, fixtureOnly.stderr).toBe(0); const report = JSON.parse(await readFile("artifacts/quality/test-evidence-source-rules.json", "utf8")) as { facts: { playwrightConfigs?: string[] }; }; - expect(report.facts.playwrightConfigs).toEqual([...selectRepositoryPaths("playwright-configs")]); - expect(report.facts.playwrightConfigs?.length).toBeGreaterThan(0); + expect(report.facts.playwrightConfigs).toEqual([]); }, childProcessTimeout(296)); it("rejects a skip without a quarantine", () => { -- 2.54.0