fix(round9): 그룹 C 검사기 진단과 시험의 빈틈을 고친다 #55

Open
donghyeon.kang wants to merge 12 commits from donghyeon.kang/r9-c-checkers-tests AGit into main
42 changed files with 1461 additions and 262 deletions
+29 -1
View File
@@ -221,6 +221,15 @@ namespace 점검을 돌아, 이 조건이 없는 runner에서 이름과 host 설
stderr, 위 sysctl 값, 고치는 방법을 적고 exit 1로
멈춘다. 전에는 이 검사가 PASS를 낸 뒤 provider 테스트가 bwrap 원문 오류로
실패했다(2026-11-05 desktop에서 114건 중 18건).
- 같은 검사가 provider의 cgroup 준비도 provider와 같은 호출로 해 본다.
`/usr/bin/prlimit`·`/usr/bin/systemd-run`이 실행 가능한지 보고,
`systemctl --user show --property=Version --value`로 user manager와 버전(254
이상)을 확인한 뒤, provider scope와 같은 속성·같은 cgroup 강제 확인으로
`systemd-run --user --scope`에서 `/bin/true`를 한 번 띄운다
(`scripts/lib/provider-cgroup.ts`). user bus가 없거나, 버전이 낮거나, memory·pids·cpu
controller가 위임되지 않은 호스트는 그 원인을 이름으로 대며 exit 1이다. 전에는
system manager bus만 보고 PASS를 낸 뒤 provider 테스트가 하나씩
`provider cgroup unavailable: ...`로 실패했다.
- 고치는 방법은 `scripts/check-system-test-prerequisites.ts`가 실패할 때
출력한다. 호스트의 보안 설정을 바꾸는 일이므로 호스트 소유자가 고른다.
호스트를 바꾸는 설정 파일과 명령은 runner를 만드는 쪽의 인프라 repo가 갖는다.
@@ -443,7 +452,18 @@ provider 확인이 필요하다고 표시한 것은 모두 `UNVERIFIED`다. `FAI
drill은 `FAIL`(exit 1)이다. 그렇지 않고 `UNVERIFIED`가 하나라도 있으면
`UNVERIFIED`(exit 3)이고, 게이트는 `PASS`가 아니다. 그래서 provider 증거가 없으면
production 단계는 닫히지 않는다. 의도한 결과다. 로컬 drill은 승격한 release를 rollback할
수 있다는 것을 보여 줄 수 없다. FE-GATE-016(`rollback-drill`)은 FE-GATE-025와 같은
수 있다는 것을 보여 줄 수 없다.
provider 증거는 `--provider-evidence <파일>`로 넘긴다(0929/11-02). 파일은
`providerDrillEvidenceSchema`(`scripts/contracts/release-artifacts.ts`) 꼴이다: runbook ID,
release ID, 관찰 시각, provider 쪽 run ID, 그리고 항목마다 `PASS`/`FAIL`과 근거다. 항목
이름은 `UNVERIFIED` 줄이 적는 이름(`trigger`, recovery assertion 이름, `provider
verification` 등)이다. drill은 runbook과 release가 같고, 관찰이 7일 안이고, 로컬 drill이
보지 못한 항목만 답한 증거만 받는다. 받은 증거는 그 항목의 `UNVERIFIED`만 바꾸고, 로컬에서
본 `PASS`·`FAIL`은 그대로 둔다. 모든 항목이 답을 얻고 `FAIL`이 없을 때만 drill이 `PASS`다.
받지 않은 증거는 이유와 함께 record의 `providerEvidence`에 남고 drill은 `UNVERIFIED`다.
템플릿의 production 게이트 명령은 아직 이 인자를 넘기지 않는다. provider를 연결한 제품이
증거 파일을 만들고 drill 명령에 넘겨야 그 단계가 닫힌다. FE-GATE-016(`rollback-drill`)은 FE-GATE-025와 같은
FE-RB-005 drill을 돌렸으므로 폐기했다(`scripts/contracts/ci-gates.ts`의
`RETIRED_GATE_IDS`).
@@ -482,6 +502,14 @@ gate의 realtime source/fixture 검사는 native SSE/WebSocket/Web Push API가
bootstrap/installed-feature 조합을 차단한다. optional recipe source gate는
uncomposed runtime 전체를 tree-shaking 없이 합성해 catalog의 gzip 예산도
blocking으로 검증한다.
같은 gate의 `check:request-fanout`은 `src/features`·`src/application`에서 루프나 배열
순회 callback 안의 요청(`fetch()`, 이름이 transport인 receiver의 method)을 거절한다.
실패 줄은 두 탈출 방법을 함께 적는다: 기능 `adapters/`에서
`createCursorPaginationRuntime`(`src/adapters/query-cache`)으로 페이지를 받거나, 상한이
있고 검토한 반복이면 그 호출 줄에 `// fanout-reviewed: <이유>`를 적는다. 결과 줄은
주석으로 넘긴 호출의 수와 위치를 적는다. 이름이 다른 receiver(`repository.load()`),
루프가 부르는 이름 있는 함수 안의 요청, 재귀, 순회 전에 만든 promise는 보지 못한다
(`scripts/check-request-fanout.ts` 머리 주석).
`FE-GATE-030`(main 단계)은 reference runtime source·전용 test·catalog metadata를 제거한
임시 repository에서도 base typecheck, architecture, test, build가 통과하는지
검증해 skeleton의 선택성을 유지한다. browser file/storage와 realtime runtime은
+6 -3
View File
@@ -130,9 +130,12 @@ probe하지 않고 `UNVERIFIED`로 끝난다. 슬롯을 채우는 방법은
**주소와 연결**
1. 앱은 HTTPS origin의 루트(`/`)에서 내보낸다. 인증서가 유효하고, 어느 surface도 그
origin 밖으로 redirect하지 않는다. — FE-GATE-019가 잰다. `HOSTING_BASE_URL`은 경로 없는
HTTPS 루트 URL이어야 하고, surface마다 최종 origin을 본다.
1. 앱은 HTTPS origin의 루트(`/`)나 빌드한 하위 경로(`VITE_ROUTER_BASE_PATH`, 예 `/app/`)에서
내보낸다. 인증서가 유효하고, 어느 surface도 그 origin 밖으로 redirect하지 않는다. —
FE-GATE-019가 잰다. `HOSTING_BASE_URL`은 release의 기준 URL(루트 또는 `/`로 끝나는 그
하위 경로, query·fragment 없음)이다. probe는 surface 경로(`config/hosting/cache-policy.json`)를
그 아래에서 요청하고, 후보의 진입 script가 그 경로 아래에 있는지와 surface마다 최종
origin을 본다.
2. TLS에서 HTTP/2를 맺는다(ALPN `h2`). — FE-GATE-019가 잰다(`transport.alpnProtocols`).
**surface별 응답** (surface: probe가 요청하는 경로 하나)
+42
View File
@@ -937,6 +937,48 @@ export default [
"react-hooks/rules-of-hooks": "off",
},
},
{
// 1112/03-04. A browser spec takes `test` from the strict fixture, which
// fails it on console errors, page errors, failed requests and unused
// preloads. FE-GATE-008 used to look for the fixture's path as text in
// tests/e2e only, so a path in a comment passed and the other browser
// folders were not looked at. tests/unit/browser-spec-imports.test.ts
// lints a fixture spec in each folder against this entry.
files: [`tests/{e2e,browser-capabilities,visual,storybook}/**/*.spec.ts`],
rules: {
"no-restricted-imports": [
"error",
{
paths: [
{
name: "@playwright/test",
importNames: ["test", "default"],
message:
"import test from tests/support/browser/strict-browser-test.ts; the strict fixture fails a spec on browser errors (.claude/skills/fe-qa).",
},
],
patterns: [
...deprecatedAliasImports,
{
group: ["**/support/browser/browser-test.ts", "**/support/browser/browser-test"],
importNames: ["test"],
message:
"import test from tests/support/browser/strict-browser-test.ts; the base test does not check browser errors.",
},
],
},
],
},
},
{
// 1112/03-04. The two specs that need the plain test on purpose:
// preload-use proves the strict fixture's own record, and chunk-load-error
// provokes the 404 a missing chunk gives to classify the engine's wording.
files: ["tests/e2e/preload-use.spec.ts", "tests/browser-capabilities/chunk-load-error.spec.ts"],
rules: {
"no-restricted-imports": ["error", { patterns: deprecatedAliasImports }],
},
},
{
files: [
`tests/fixtures/browser-file-storage-boundaries/forbidden/**/*.${sourceExtensions}`,
+8
View File
@@ -5,6 +5,7 @@ import {
createFileSystemDeclarationSource,
type TopologyDeclarationSource,
} from "./generate-product-feature-index.ts";
import { checkoutBuildReadFailure } from "./lib/run-isolation.ts";
/**
* scale-architecture.md §3 / §37. Build-input regression gate for deployable-app isolation.
@@ -123,6 +124,13 @@ async function main(): Promise<void> {
// without `--require-bundle` the gate checks the generated indexes only, so a
// stale inventory from another build can never pass or fail it.
const requireBundle = process.argv.includes("--require-bundle");
// 1106/12-02. The inventory is read where the checkout keeps it; under
// RUN_OUTPUT_ROOT the build wrote elsewhere and this would judge an earlier one.
const outputRootFailure = requireBundle ? checkoutBuildReadFailure("scripts/check-app-build-input.ts") : undefined;
if (outputRootFailure !== undefined) {
process.stderr.write(`${outputRootFailure}\n`);
process.exit(1);
}
const report = await inspectAppBuildInput(process.cwd(), requireBundle ? {} : { bundleInventory: null });
if (requireBundle && !report.bundle.checked) {
process.stderr.write("app build input: no production module inventory; run the build first\n");
+10
View File
@@ -11,6 +11,7 @@ import {
validateRecipeCatalog,
} from "./lib/optional-recipes.ts";
import { assertMatchesJsonSchema } from "./lib/json-schema.ts";
import { checkoutBuildReadFailure } from "./lib/run-isolation.ts";
import { writeValidatedJsonArtifact } from "./lib/validated-json-artifact.ts";
type GateViolation = Readonly<{
@@ -36,6 +37,15 @@ const catalogPath = argument(
);
const sourceRoot = argument("--source-root", "src");
const distRoot = argument("--dist-root", "dist");
// 1106/12-02. Without --dist-root this scans the checkout's dist; under
// RUN_OUTPUT_ROOT the build wrote elsewhere, so it would judge an earlier build.
const outputRootFailure = process.argv.includes("--dist-root")
? undefined
: checkoutBuildReadFailure("scripts/check-optional-recipes.ts");
if (outputRootFailure !== undefined) {
process.stderr.write(`${outputRootFailure}\n`);
process.exit(1);
}
const artifactPath = argument(
"--artifact",
"artifacts/quality/optional-recipes.json",
+73 -23
View File
@@ -17,6 +17,16 @@ import { parseAsync } from "@babel/core";
* The HTTP runtime itself (src/adapters/http) owns its bounded retry loop and is
* out of scope. A reviewed exception is written on the call's line as
* `// fanout-reviewed: <reason>`.
*
* 0929/12-01. A failure names both ways out: fetch page after page through
* `createCursorPaginationRuntime` (src/adapters/query-cache) assembled in the
* feature's `adapters/`, or mark a bounded, reviewed repetition on the call's
* line. Every reviewed exception is counted and listed in the result line.
*
* What it does not see: a dispatch through a receiver named otherwise
* (`repository.load()`, a destructured `get`), a dispatch in a named function
* that a loop calls, recursion, and promises built before an iteration and
* awaited inside it. Reviews still look for those.
*/
export const REQUEST_FANOUT_ROOTS = Object.freeze(["src/features", "src/application"]);
@@ -44,6 +54,14 @@ const FUNCTION_TYPES = new Set(["ArrowFunctionExpression", "FunctionExpression"]
type Node = { type: string; loc?: { start: { line: number } } } & Record<string, unknown>;
export type FanoutViolation = Readonly<{ file: string; line: number; call: string; context: string }>;
export type FanoutReviewedException = Readonly<{ file: string; line: number; call: string; reason: string }>;
const REVIEWED = /fanout-reviewed:\s*(\S.*?)\s*$/u;
/** How to fix a failure; both ways the source gate accepts. */
export const FANOUT_REMEDY =
"load one page per request with createCursorPaginationRuntime (src/adapters/query-cache) assembled in the " +
"feature's adapters/, or, for a bounded repetition someone reviewed, write `// fanout-reviewed: <reason>` on the call's line";
function receiverName(node: Node | undefined): string | null {
if (!node) return null;
@@ -81,6 +99,14 @@ export async function findRequestFanout(
file: string,
sourceText: string,
): Promise<readonly FanoutViolation[]> {
return (await inspectRequestFanout(file, sourceText)).violations;
}
/** Per-row dispatches, and the ones a `fanout-reviewed:` comment let through. */
export async function inspectRequestFanout(
file: string,
sourceText: string,
): Promise<Readonly<{ violations: readonly FanoutViolation[]; reviewed: readonly FanoutReviewedException[] }>> {
const plugins: ("jsx" | ["typescript", { dts: boolean }])[] = [["typescript", { dts: false }]];
if (file.endsWith(".tsx")) plugins.push("jsx");
const tree = (await parseAsync(sourceText, {
@@ -93,6 +119,7 @@ export async function findRequestFanout(
if (!tree) throw new Error(`cannot parse ${file}`);
const lines = sourceText.split("\n");
const violations: FanoutViolation[] = [];
const reviewed: FanoutReviewedException[] = [];
const visit = (node: Node, parent: Node | undefined, repeating: string | null): void => {
let context = repeating;
@@ -106,9 +133,9 @@ export async function findRequestFanout(
if (node.type === "CallExpression" && context) {
const call = dispatchDescription(node);
const line = node.loc?.start.line ?? 0;
if (call && !/fanout-reviewed:\s*\S/u.test(lines[line - 1] ?? "")) {
violations.push({ file, line, call, context });
}
const reason = REVIEWED.exec(lines[line - 1] ?? "")?.[1];
if (call && reason !== undefined) reviewed.push({ file, line, call, reason });
else if (call) violations.push({ file, line, call, context });
}
for (const [key, value] of Object.entries(node)) {
if (key === "loc" || key === "start" || key === "end" || key === "extra") continue;
@@ -121,7 +148,7 @@ export async function findRequestFanout(
}
};
visit(tree, undefined, null);
return violations;
return { violations, reviewed };
}
async function sourceFiles(directory: string): Promise<string[]> {
@@ -143,39 +170,62 @@ async function sourceFiles(directory: string): Promise<string[]> {
export async function scanRequestFanout(
root: string,
roots: readonly string[] = REQUEST_FANOUT_ROOTS,
): Promise<Readonly<{ files: number; violations: readonly FanoutViolation[] }>> {
): Promise<Readonly<{
files: number;
violations: readonly FanoutViolation[];
reviewed: readonly FanoutReviewedException[];
}>> {
const files = (await Promise.all(roots.map((entry) => sourceFiles(path.join(root, entry))))).flat();
const violations: FanoutViolation[] = [];
const reviewed: FanoutReviewedException[] = [];
for (const file of files) {
violations.push(
...(await findRequestFanout(path.relative(root, file), await readFile(file, "utf8"))),
);
const found = await inspectRequestFanout(path.relative(root, file), await readFile(file, "utf8"));
violations.push(...found.violations);
reviewed.push(...found.reviewed);
}
return { files: files.length, violations };
return { files: files.length, violations, reviewed };
}
export function formatFanoutViolation(violation: FanoutViolation): string {
return `FAIL ${violation.file}:${violation.line} ${violation.call} inside ${violation.context}: ${FANOUT_REMEDY}`;
}
function formatReviewed(reviewed: readonly FanoutReviewedException[]): string {
if (reviewed.length === 0) return "0 reviewed exceptions";
return `${reviewed.length} reviewed exception${reviewed.length === 1 ? "" : "s"} (` +
`${reviewed.map((entry) => `${entry.file}:${entry.line} ${entry.reason}`).join("; ")})`;
}
/**
* The forbidden fixture must be rejected; the allowed fixture must pass, with
* its reviewed cursor loop counted as a reviewed exception.
*/
export async function checkRequestFanoutFixtures(root: string): Promise<Readonly<{ ok: boolean; line: string }>> {
const forbidden = await scanRequestFanout(root, ["tests/fixtures/architecture/request-fanout/forbidden"]);
const allowed = await scanRequestFanout(root, ["tests/fixtures/architecture/request-fanout/allowed"]);
const ok = forbidden.violations.length >= forbidden.files && forbidden.files > 0 &&
allowed.violations.length === 0 && allowed.reviewed.length > 0;
return {
ok,
line: `request fan-out fixtures: forbidden ${forbidden.violations.length} rejected in ${forbidden.files} files, ` +
`allowed ${allowed.violations.length} violations and ${formatReviewed(allowed.reviewed)} ${ok ? "PASS" : "FAIL"}`,
};
}
async function main(): Promise<void> {
const fixture = process.argv.includes("--fixture");
const root = process.cwd();
if (fixture) {
// The forbidden fixture must be rejected; the allowed fixture must pass.
const forbidden = await scanRequestFanout(root, ["tests/fixtures/architecture/request-fanout/forbidden"]);
const allowed = await scanRequestFanout(root, ["tests/fixtures/architecture/request-fanout/allowed"]);
const ok = forbidden.violations.length >= forbidden.files && forbidden.files > 0 && allowed.violations.length === 0;
process.stdout.write(
`request fan-out fixtures: forbidden ${forbidden.violations.length} rejected in ${forbidden.files} files, allowed ${allowed.violations.length} violations ${ok ? "PASS" : "FAIL"}\n`,
);
if (!ok) process.exitCode = 1;
const result = await checkRequestFanoutFixtures(root);
process.stdout.write(`${result.line}\n`);
if (!result.ok) process.exitCode = 1;
return;
}
const result = await scanRequestFanout(root);
for (const violation of result.violations) {
process.stderr.write(
`FAIL ${violation.file}:${violation.line} ${violation.call} inside ${violation.context}\n`,
);
}
for (const violation of result.violations) process.stderr.write(`${formatFanoutViolation(violation)}\n`);
process.stdout.write(
`request fan-out: ${result.files} files scanned, ${result.violations.length} per-row dispatches ${result.violations.length === 0 ? "PASS" : "FAIL"}\n`,
`request fan-out: ${result.files} files scanned, ${result.violations.length} per-row dispatches, ` +
`${formatReviewed(result.reviewed)} ${result.violations.length === 0 ? "PASS" : "FAIL"}\n`,
);
if (result.violations.length > 0) process.exitCode = 1;
}
+80 -16
View File
@@ -1,9 +1,16 @@
import { spawnSync } from "node:child_process";
import { randomBytes } from "node:crypto";
import { constants } from "node:fs";
import { access, readFile } from "node:fs/promises";
import { spawnSync } from "node:child_process";
import path from "node:path";
import { fileURLToPath } from "node:url";
import {
formatProviderPrerequisiteUnitName,
MINIMUM_PROVIDER_SYSTEMD_VERSION,
systemdRunPrerequisiteProbeArguments,
USER_MANAGER_VERSION_ARGUMENTS,
} from "./lib/provider-cgroup.ts";
import { PROVIDER_SANDBOX_NAMESPACE_ARGUMENTS } from "./lib/provider-sandbox-namespaces.ts";
/** What the check reads from the host; a test passes a fake. */
@@ -15,7 +22,9 @@ export type SystemTestHost = Readonly<{
command: string,
args: readonly string[],
timeoutMs: number,
): Readonly<{ status: number | null; stderr: string; error?: Error | undefined }>;
): Readonly<{ status: number | null; stdout?: string; stderr: string; error?: Error | undefined }>;
/** Names the provider-shaped scope the probe starts; a test passes a fixed one. */
probeUnitName?: () => string;
}>;
export const nodeSystemTestHost: SystemTestHost = Object.freeze({
@@ -23,12 +32,18 @@ export const nodeSystemTestHost: SystemTestHost = Object.freeze({
readFile: async (target: string) => readFile(target, "utf8"),
spawn: (command: string, args: readonly string[], timeoutMs: number) => {
const result = spawnSync(command, [...args], { encoding: "utf8", timeout: timeoutMs });
return { status: result.status, stderr: result.stderr ?? "", error: result.error };
return { status: result.status, stdout: result.stdout ?? "", stderr: result.stderr ?? "", error: result.error };
},
probeUnitName: defaultProbeUnitName,
});
function defaultProbeUnitName(): string {
return formatProviderPrerequisiteUnitName(process.pid, randomBytes(12).toString("hex"));
}
const BWRAP = "/usr/bin/bwrap";
const SYSTEMCTL = "/usr/bin/systemctl";
const SYSTEMD_RUN = "/usr/bin/systemd-run";
/**
* 1105/12-01. The provider sandbox's own namespace options on /bin/true. The
@@ -50,6 +65,7 @@ const USER_NAMESPACE_SETTINGS = Object.freeze([
] as const);
const PROBE_TIMEOUT_MS = 5_000;
const SCOPE_PROBE_TIMEOUT_MS = 15_000;
const MAX_REPORTED_STDERR_LINES = 4;
export async function findSystemTestPrerequisiteFailures(
@@ -59,6 +75,8 @@ export async function findSystemTestPrerequisiteFailures(
const executable = new Map<string, boolean>();
for (const [target, label] of [
[BWRAP, "bubblewrap"],
["/usr/bin/prlimit", "prlimit"],
[SYSTEMD_RUN, "systemd-run"],
[SYSTEMCTL, "systemctl"],
["/usr/bin/tar", "tar"],
] as const) {
@@ -80,10 +98,18 @@ export async function findSystemTestPrerequisiteFailures(
failures.push("cgroup v2 is required at /sys/fs/cgroup/cgroup.controllers");
}
// 1112/08-03. The provider asks the user manager (not the system manager)
// for its version and then starts its sandbox in a `systemd-run --user
// --scope` (scripts/run-and-validate-provider.ts). The check makes the same
// calls, so a host without a user bus, with an old manager, or without
// delegated controllers fails here by name instead of in each provider test.
if (executable.get(SYSTEMCTL)) {
const probe = host.spawn(SYSTEMCTL, ["show-environment"], 3_000);
if (probe.error || probe.status !== 0) {
failures.push("a reachable systemd manager bus is required");
const userManager = findUserManagerFailure(host);
if (userManager !== undefined) {
failures.push(userManager);
} else if (executable.get(SYSTEMD_RUN)) {
const scope = findProviderScopeFailure(host);
if (scope !== undefined) failures.push(scope);
}
}
@@ -94,6 +120,52 @@ export async function findSystemTestPrerequisiteFailures(
return failures;
}
function findUserManagerFailure(host: SystemTestHost): string | undefined {
const probe = host.spawn(SYSTEMCTL, USER_MANAGER_VERSION_ARGUMENTS, 3_000);
const command = `${SYSTEMCTL} ${USER_MANAGER_VERSION_ARGUMENTS.join(" ")}`;
if (probe.error || probe.status !== 0) {
return [
`a reachable systemd user manager is required (${command} ${outcomeOf(probe)}).`,
...stderrLines(probe.stderr, "systemctl"),
" The provider runs in a `systemd-run --user --scope`; give the runner user a user",
" manager (for example `loginctl enable-linger <user>`) and a user bus in XDG_RUNTIME_DIR.",
].join("\n");
}
const reported = (probe.stdout ?? "").trim();
const match = /^([0-9]+)/u.exec(reported);
const version = match ? Number(match[1]) : Number.NaN;
if (!Number.isSafeInteger(version) || version < MINIMUM_PROVIDER_SYSTEMD_VERSION) {
return `systemd user manager ${MINIMUM_PROVIDER_SYSTEMD_VERSION} or newer is required (${command} reported ${JSON.stringify(reported)})`;
}
return undefined;
}
function findProviderScopeFailure(host: SystemTestHost): string | undefined {
const args = systemdRunPrerequisiteProbeArguments((host.probeUnitName ?? defaultProbeUnitName)());
const probe = host.spawn(SYSTEMD_RUN, args, SCOPE_PROBE_TIMEOUT_MS);
if (!probe.error && probe.status === 0) return undefined;
return [
"systemd-run could not start the provider's cgroup scope around /bin/true " +
`(${SYSTEMD_RUN} --user --scope ... ${outcomeOf(probe)}).`,
...stderrLines(probe.stderr, "systemd-run"),
" The scope needs the memory, pids and cpu controllers delegated to the user manager",
" (Delegate= in user@.service) and cgroup v2. Runner requirements: docs/operations/ci-quality-gates.md.",
].join("\n");
}
function outcomeOf(probe: ReturnType<SystemTestHost["spawn"]>): string {
return probe.error ? `did not finish: ${probe.error.message}` : `exited ${String(probe.status)}`;
}
function stderrLines(stderr: string, label: string): string[] {
const lines = stderr
.split("\n")
.map((line) => line.trim())
.filter((line) => line.length > 0)
.slice(0, MAX_REPORTED_STDERR_LINES);
return lines.length > 0 ? lines.map((line) => ` ${label} stderr: ${line}`) : [` ${label} stderr: (empty)`];
}
/**
* 1101/13-03. Whether this host lets the provider sandbox create its
* namespaces, and if not, why and what to do. The release provider jobs run
@@ -118,19 +190,11 @@ async function namespaceFailure(
// Absent on this kernel; the others still say what the host allows.
}
}
const stderr = probe.stderr
.split("\n")
.map((line) => line.trim())
.filter((line) => line.length > 0)
.slice(0, MAX_REPORTED_STDERR_LINES);
const outcome = probe.error
? `did not finish: ${probe.error.message}`
: `exited ${String(probe.status)}`;
return [
"this host does not allow unprivileged user and network namespaces, which the provider " +
`sandbox needs (bwrap ${PROVIDER_SANDBOX_NAMESPACE_ARGUMENTS.join(" ")}).`,
` probe: ${BWRAP} ${NAMESPACE_PROBE_ARGUMENTS.join(" ")} ${outcome}`,
...(stderr.length > 0 ? stderr.map((line) => ` bwrap stderr: ${line}`) : [" bwrap stderr: (empty)"]),
` probe: ${BWRAP} ${NAMESPACE_PROBE_ARGUMENTS.join(" ")} ${outcomeOf(probe)}`,
...stderrLines(probe.stderr, "bwrap"),
...(settings.length > 0 ? [` host settings: ${settings.join(", ")}`] : []),
" To fix, do one of these:",
" - give /usr/bin/bwrap an AppArmor profile that allows `userns` (Ubuntu 23.10 and later),",
+30 -16
View File
@@ -64,6 +64,12 @@ const policyPath = argumentValue(
"config/testing/test-evidence.json",
);
const explicitCatalogPath = optionalArgumentValue("--catalog");
// 1112/03-03. The forbidOnly rule read only the repository's configs and ran
// only in the full check, so no fixture could show it still fires.
// `--playwright-config <file>` (repeatable) checks those files in any mode.
const explicitPlaywrightConfigs = process.argv.flatMap((argument, index) =>
argument === "--playwright-config" && process.argv[index + 1] ? [process.argv[index + 1]!] : [],
);
const explicitReceiptPath = optionalArgumentValue("--receipt");
const fixtureMode = sourceRoot !== "tests";
const sourceOnly = process.argv.includes("--source-only");
@@ -78,6 +84,7 @@ const facts = {
sharedScenarios: 0,
declaredScenarioExecutions: 0,
executedScenarioExecutions: 0,
playwrightConfigs: [] as string[],
};
async function filesBelow(target: string): Promise<string[]> {
@@ -365,6 +372,26 @@ if (parsedPolicy) {
}
}
async function missingForbidOnly(configs: readonly string[]): Promise<string[]> {
const missing: string[] = [];
for (const config of configs) {
if (!(await readFile(config, "utf8")).includes("forbidOnly: true")) {
missing.push(`${config}: missing forbidOnly: true, so a committed test.only passes the gate`);
}
}
return missing;
}
// The repository's configs unless the run names others; every mode but the
// scenario check reads them, and the report lists what it read.
if (!scenarioOnly) {
// playwright.dev.config.ts spreads playwright.config.ts.
facts.playwrightConfigs = explicitPlaywrightConfigs.length > 0
? explicitPlaywrightConfigs
: [...selectRepositoryPaths("playwright-configs")];
failures.push(...(await missingForbidOnly(facts.playwrightConfigs)));
}
if (!fixtureMode && !scenarioOnly) {
const e2eConfig = await readFile("playwright.config.ts", "utf8");
for (const token of [
@@ -381,22 +408,9 @@ if (!fixtureMode && !scenarioOnly) {
failures.push(`playwright.config.ts missing release evidence token ${token}`);
}
}
// playwright.dev.config.ts spreads playwright.config.ts.
for (const config of selectRepositoryPaths("playwright-configs")) {
if (!(await readFile(config, "utf8")).includes("forbidOnly: true")) {
failures.push(`${config}: missing forbidOnly: true, so a committed test.only passes the gate`);
}
}
const e2eFiles = (await filesBelow("tests/e2e")).filter((file) =>
/\.spec\.ts$/.test(file),
);
for (const file of e2eFiles) {
const source = await readFile(file, "utf8");
if (!source.includes("support/browser/strict-browser-test")) {
failures.push(`${file}: bypasses strict browser fixture`);
}
}
// 1112/03-04. That a browser spec takes the strict fixture's `test` is an
// ESLint rule now (eslint.config.ts, tests/unit/browser-spec-imports.test.ts),
// which reads the import instead of looking for the path as text.
const baselineFiles = (await filesBelow("tests/visual/__snapshots__")).filter(
(file) => file.endsWith(".png"),
+53 -3
View File
@@ -746,16 +746,52 @@ const drillVerdict = z.enum(["PASS", "FAIL", "UNVERIFIED"]);
export type DrillVerdict = z.infer<typeof drillVerdict>;
/** FAIL beats UNVERIFIED beats PASS; provider scope alone keeps a drill UNVERIFIED. */
/**
* FAIL beats UNVERIFIED beats PASS. A drill with provider scope stays
* UNVERIFIED until accepted provider evidence says the provider side passed
* (0929/11-02).
*/
export function combineDrillVerdicts(
verdicts: readonly DrillVerdict[],
providerVerificationRequired: boolean,
providerVerification: DrillVerdict = "UNVERIFIED",
): DrillVerdict {
if (verdicts.includes("FAIL")) return "FAIL";
if (verdicts.includes("UNVERIFIED") || providerVerificationRequired) return "UNVERIFIED";
const all = providerVerificationRequired ? [...verdicts, providerVerification] : verdicts;
if (all.includes("FAIL")) return "FAIL";
if (all.includes("UNVERIFIED")) return "UNVERIFIED";
return "PASS";
}
/**
* 0929/11-02. What the hosting provider observed for one drill of one
* release: the provider-side items a local drill cannot see (a smoke after the
* switch, live traffic, the trigger), named as the drill's UNVERIFIED line
* names them, plus `provider verification` itself.
*/
export const providerDrillEvidenceSchema = z
.object({
schemaVersion: z.literal(1),
artifactType: z.literal("provider-drill-evidence"),
runbookId: z.string().regex(/^FE-RB-00[1-5]$/u),
releaseId: nonEmptyString,
observedAt: timestamp,
runId: nonEmptyString,
observations: z
.array(
z
.object({ item: nonEmptyString, result: z.enum(["PASS", "FAIL"]), evidence: nonEmptyString })
.strict(),
)
.min(1)
.max(32)
.refine((items) => new Set(items.map(({ item }) => item)).size === items.length, {
message: "must name each item once",
}),
})
.strict();
export type ProviderDrillEvidence = z.infer<typeof providerDrillEvidenceSchema>;
export const runbookRecordArtifactSchema = z
.object({
schemaVersion: z.literal(2),
@@ -778,6 +814,19 @@ export const runbookRecordArtifactSchema = z
negativeFixtureFailedAsExpected: drillVerdict,
windowObservedBucket: nonEmptyString,
providerVerificationRequired: z.boolean(),
providerEvidence: z
.object({
accepted: z.boolean(),
reason: nonEmptyString.optional(),
runId: nonEmptyString.optional(),
observedAt: timestamp.optional(),
providerVerification: drillVerdict,
})
.strict()
.refine((evidence) => evidence.accepted || (evidence.reason !== undefined && evidence.providerVerification === "UNVERIFIED"), {
message: "rejected provider evidence must say why and verify nothing",
})
.optional(),
result: drillVerdict,
})
.strict()
@@ -791,6 +840,7 @@ export const runbookRecordArtifactSchema = z
record.negativeFixtureFailedAsExpected,
],
record.providerVerificationRequired,
record.providerEvidence?.providerVerification,
);
if (record.result !== expected) {
context.addIssue({
+10
View File
@@ -1,5 +1,6 @@
import { mkdir } from "node:fs/promises";
import { checkoutBuildReadFailure } from "./lib/run-isolation.ts";
import { writeValidatedJsonArtifact } from "./lib/validated-json-artifact.ts";
import { localEvidenceAssessmentArtifactSchema } from "./contracts/release-artifacts.ts";
import { createLocalEvidenceAssessment } from "./lib/local-release-evidence.ts";
@@ -10,6 +11,15 @@ import {
releaseCandidateManifestSchema,
} from "./lib/release-candidate.ts";
// 1106/12-02. This digests dist and the evidence under artifacts/ where the
// checkout keeps them (scripts/lib/release-candidate.ts). Under RUN_OUTPUT_ROOT
// the build wrote elsewhere, so the candidate would package an earlier build.
const outputRootFailure = checkoutBuildReadFailure("scripts/create-release-candidate.ts");
if (outputRootFailure !== undefined) {
process.stderr.write(`${outputRootFailure}\n`);
process.exit(1);
}
const assessment = await createLocalEvidenceAssessment();
await mkdir("artifacts/security", { recursive: true });
await writeValidatedJsonArtifact({
+142 -10
View File
@@ -22,8 +22,10 @@ import { projectTelemetryEvent } from "../src/contracts/telemetry.ts";
import { EXPECTED_CONTRACT_SET_PACKAGES } from "../src/features/installed-contract-contributions.ts";
import {
combineDrillVerdicts,
providerDrillEvidenceSchema,
runbookRecordArtifactSchema,
type DrillVerdict,
type ProviderDrillEvidence,
} from "./contracts/release-artifacts.ts";
import { verifyReleaseRuntimeCoherence } from "./lib/release-runtime-coherence.ts";
import { writeValidatedJsonArtifact } from "./lib/validated-json-artifact.ts";
@@ -87,8 +89,25 @@ export type DrillOptions = Readonly<{
*/
releaseRoot?: string;
contractPackages?: readonly InstalledContractPackageIdentity[];
/**
* 0929/11-02. The provider's record for this drill (`--provider-evidence`),
* parsed JSON. Without it a drill with provider scope cannot pass.
*/
providerEvidence?: unknown;
/** The clock evidence age is judged against; a test passes a fixed one. */
now?: Date;
}>;
/** Provider evidence older than this no longer describes the provider as it is. */
export const PROVIDER_EVIDENCE_MAX_AGE_MS = 7 * 24 * 60 * 60 * 1_000;
/** How far ahead of this clock a provider's timestamp may be. */
const PROVIDER_EVIDENCE_CLOCK_SKEW_MS = 5 * 60 * 1_000;
/** The item names a drill's UNVERIFIED line uses, and provider evidence answers. */
const ITEM_TRIGGER = "trigger";
const ITEM_CONTAINMENT = "containment";
const ITEM_NEGATIVE_FIXTURE = "negative fixture";
export const ITEM_PROVIDER_VERIFICATION = "provider verification";
type DrillContext = Readonly<{
root: string;
contractPackages: readonly InstalledContractPackageIdentity[];
@@ -490,14 +509,29 @@ export async function runRunbookDrill(runbookId: string, options: DrillOptions =
const release = parseReleaseArtifact(
await rootedReader(context.root)(DEFAULT_ROLLBACK_PATHS.release),
);
const result = await drill(context);
const local = await drill(context);
const now = options.now ?? new Date();
const judged = options.providerEvidence === undefined
? undefined
: judgeProviderEvidence(options.providerEvidence, runbookId, release.releaseId, openItems(local), now);
const result = judged?.accepted ? applyProviderEvidence(local, judged.evidence) : local;
const providerEvidence = judged === undefined
? undefined
: judged.accepted
? {
accepted: true as const,
runId: judged.evidence.runId,
observedAt: judged.evidence.observedAt,
providerVerification: observed(judged.evidence, ITEM_PROVIDER_VERIFICATION) ?? "UNVERIFIED",
}
: { accepted: false as const, reason: judged.reason, providerVerification: "UNVERIFIED" as const };
const escalationPathAsserted = verdict(specification.escalation.length >= 2);
return {
schemaVersion: 2 as const,
runbookId,
gateId: specification.gateId,
releaseId: release.releaseId,
drillTimestamp: new Date().toISOString(),
drillTimestamp: now.toISOString(),
triggerInjected: specification.triggerKinds[0] ?? "",
triggerAsserted: result.triggerAsserted,
containmentAsserted: result.containmentAsserted,
@@ -506,6 +540,7 @@ export async function runRunbookDrill(runbookId: string, options: DrillOptions =
negativeFixtureFailedAsExpected: result.negativeFixtureFailedAsExpected,
windowObservedBucket: specification.window,
providerVerificationRequired: result.providerVerificationRequired,
...(providerEvidence === undefined ? {} : { providerEvidence }),
result: combineDrillVerdicts(
[
result.triggerAsserted,
@@ -515,24 +550,117 @@ export async function runRunbookDrill(runbookId: string, options: DrillOptions =
result.negativeFixtureFailedAsExpected,
],
result.providerVerificationRequired,
providerEvidence?.providerVerification,
),
};
}
/** The items the local drill left UNVERIFIED, by the names provider evidence uses. */
function openItems(result: DrillResult): string[] {
return [
...(result.triggerAsserted === "UNVERIFIED" ? [ITEM_TRIGGER] : []),
...(result.containmentAsserted === "UNVERIFIED" ? [ITEM_CONTAINMENT] : []),
...result.recoveryAssertions.filter((item) => item.result === "UNVERIFIED").map((item) => item.assertion),
...(result.negativeFixtureFailedAsExpected === "UNVERIFIED" ? [ITEM_NEGATIVE_FIXTURE] : []),
...(result.providerVerificationRequired ? [ITEM_PROVIDER_VERIFICATION] : []),
];
}
type JudgedProviderEvidence =
| Readonly<{ accepted: true; evidence: ProviderDrillEvidence }>
| Readonly<{ accepted: false; reason: string }>;
/**
* 0929/11-02. Provider evidence counts only for this runbook, this release,
* a recent observation, and the items the local drill could not observe.
*/
export function judgeProviderEvidence(
value: unknown,
runbookId: string,
releaseId: string,
open: readonly string[],
now: Date,
): JudgedProviderEvidence {
const parsed = providerDrillEvidenceSchema.safeParse(value);
if (!parsed.success) {
const issue = parsed.error.issues[0];
return { accepted: false, reason: `not provider drill evidence (${issue?.path.join(".") || "root"}: ${issue?.message ?? "invalid"})` };
}
const evidence = parsed.data;
if (evidence.runbookId !== runbookId) {
return { accepted: false, reason: `evidence is for ${evidence.runbookId}, not ${runbookId}` };
}
if (evidence.releaseId !== releaseId) {
return { accepted: false, reason: `evidence is for release ${evidence.releaseId}, not ${releaseId}` };
}
const age = now.getTime() - Date.parse(evidence.observedAt);
if (age > PROVIDER_EVIDENCE_MAX_AGE_MS) {
return { accepted: false, reason: `evidence observed at ${evidence.observedAt} is older than ${PROVIDER_EVIDENCE_MAX_AGE_MS / 86_400_000} days` };
}
if (age < -PROVIDER_EVIDENCE_CLOCK_SKEW_MS) {
return { accepted: false, reason: `evidence observed at ${evidence.observedAt} is in the future` };
}
const extra = evidence.observations.map(({ item }) => item).filter((item) => !open.includes(item));
if (extra.length > 0) {
return { accepted: false, reason: `evidence names items the local drill observed or does not have: ${extra.join(", ")}` };
}
return { accepted: true, evidence };
}
function observed(evidence: ProviderDrillEvidence, item: string): DrillVerdict | undefined {
return evidence.observations.find((observation) => observation.item === item)?.result;
}
/** Replaces each UNVERIFIED item the evidence answers; local PASS and FAIL stay. */
function applyProviderEvidence(result: DrillResult, evidence: ProviderDrillEvidence): DrillResult {
const answer = (item: string, local: DrillVerdict): DrillVerdict =>
local === "UNVERIFIED" ? observed(evidence, item) ?? local : local;
const source = (item: string) =>
`provider evidence ${evidence.runId} at ${evidence.observedAt}: ${
evidence.observations.find((observation) => observation.item === item)?.evidence ?? ""
}`;
return {
...result,
triggerAsserted: answer(ITEM_TRIGGER, result.triggerAsserted),
containmentAsserted: answer(ITEM_CONTAINMENT, result.containmentAsserted),
recoveryAssertions: result.recoveryAssertions.map((item) =>
item.result === "UNVERIFIED" && observed(evidence, item.assertion) !== undefined
? { ...item, evidence: source(item.assertion), result: observed(evidence, item.assertion)! }
: item,
),
negativeFixtureFailedAsExpected: answer(ITEM_NEGATIVE_FIXTURE, result.negativeFixtureFailedAsExpected),
};
}
async function main(): Promise<void> {
const runbookId = process.argv
.slice(2)
.find((argument) => /^FE-RB-00[1-5]$/.test(argument));
const releaseRootFlag = process.argv.indexOf("--release-root");
const releaseRoot = releaseRootFlag >= 0 ? process.argv[releaseRootFlag + 1] : undefined;
if (!runbookId || (releaseRootFlag >= 0 && !releaseRoot)) {
const evidenceFlag = process.argv.indexOf("--provider-evidence");
const evidencePath = evidenceFlag >= 0 ? process.argv[evidenceFlag + 1] : undefined;
if (!runbookId || (releaseRootFlag >= 0 && !releaseRoot) || (evidenceFlag >= 0 && !evidencePath)) {
process.stderr.write(
"Usage: drill:runbook -- FE-RB-001..FE-RB-005 [--release-root <extracted release archive>]\n",
"Usage: drill:runbook -- FE-RB-001..FE-RB-005 [--release-root <extracted release archive>]" +
" [--provider-evidence <provider drill evidence JSON>]\n",
);
process.exit(2);
}
let providerEvidence: unknown;
if (evidencePath !== undefined) {
try {
providerEvidence = JSON.parse(await readFile(evidencePath, "utf8"));
} catch (error) {
process.stderr.write(`${runbookId} drill: cannot read provider evidence ${evidencePath}: ${String(error)}\n`);
process.exit(2);
}
}
const { gateId, ...record } = await runRunbookDrill(runbookId, releaseRoot ? { releaseRoot } : {});
const { gateId, ...record } = await runRunbookDrill(runbookId, {
...(releaseRoot ? { releaseRoot } : {}),
...(evidencePath !== undefined ? { providerEvidence } : {}),
});
const artifactDirectory = `artifacts/runbooks/${runbookId}`;
await mkdir(artifactDirectory, { recursive: true });
await writeValidatedJsonArtifact({
@@ -547,14 +675,18 @@ async function main(): Promise<void> {
if (record.result === "UNVERIFIED") {
// Not a pass: the gate stays red until provider evidence exists (D-14).
const open = [
["trigger", record.triggerAsserted],
["containment", record.containmentAsserted],
[ITEM_TRIGGER, record.triggerAsserted],
[ITEM_CONTAINMENT, record.containmentAsserted],
...record.recoveryAssertions.map((item) => [item.assertion, item.result] as const),
["negative fixture", record.negativeFixtureFailedAsExpected],
["provider verification", record.providerVerificationRequired ? "UNVERIFIED" : "PASS"],
[ITEM_NEGATIVE_FIXTURE, record.negativeFixtureFailedAsExpected],
[
ITEM_PROVIDER_VERIFICATION,
record.providerVerificationRequired ? record.providerEvidence?.providerVerification ?? "UNVERIFIED" : "PASS",
],
].filter(([, result]) => result === "UNVERIFIED").map(([name]) => name);
const rejected = record.providerEvidence?.accepted === false ? `; provider evidence rejected: ${record.providerEvidence.reason}` : "";
process.stderr.write(
`${runbookId} drill: UNVERIFIED (${gateId}; no local check failed; not observed locally: ${open.join(", ")})\n`,
`${runbookId} drill: UNVERIFIED (${gateId}; no local check failed; not observed locally: ${open.join(", ")}${rejected})\n`,
);
process.exit(3);
}
+7 -8
View File
@@ -20,7 +20,7 @@ import {
buildDate,
} from "./lib/build-environment.ts";
import { writeValidatedJsonArtifact } from "./lib/validated-json-artifact.ts";
import { CANONICAL_VITE_MANIFEST_PATH } from "./lib/build-manifest-outputs.ts";
import { CANONICAL_VITE_MANIFEST_PATH, findRouteChunk } from "./lib/build-manifest-outputs.ts";
import { runOutputPath } from "./lib/run-isolation.ts";
assertCiBuildEnvironment(process.env);
@@ -68,14 +68,13 @@ const routeChunks: Record<string, string> = {};
const runtimeContracts: Readonly<Record<string, { moduleId: string }>> =
ROUTE_RUNTIME_CONTRACT;
for (const definition of Object.values(ROUTE_REGISTRY)) {
const runtime = runtimeContracts[definition.routeId];
const asset = Object.values(viteManifestObject).find(
(entry) => entry.name === runtime?.moduleId && entry.isDynamicEntry,
const chunk = findRouteChunk(
viteManifestObject,
definition.routeId,
runtimeContracts[definition.routeId]?.moduleId,
);
if (!runtime || !asset?.file) {
throw new Error(`Missing built route chunk: ${definition.routeId}`);
}
routeChunks[definition.chunkId] = asset.file;
if ("error" in chunk) throw new Error(chunk.error);
routeChunks[definition.chunkId] = chunk.file;
}
const runtimeConfigJsonSchema = z.toJSONSchema(runtimeConfigV2ArtifactSchema);
+49
View File
@@ -186,3 +186,52 @@ export async function verifyBuildManifestOutputs(
}
return mismatches;
}
/**
* 0929/13-03. The built chunk of a route's lazy page, for the build manifest
* and the two release checks that compare its `routeChunks`. A route runtime's
* `moduleId` is the file name of the page it imports
* (`src/presentation/routes/route-runtime.tsx`, a generated feature's
* `<id>-route-runtime.ts`), so the chunk is the dynamic entry whose source
* file — the Vite manifest key — has that name. The chunk's own `name` used to
* be matched instead, so a chunk-naming change in vite.config.ts stopped the
* build with a reason-less `Missing built route chunk`.
*/
export type RouteChunkLookup = Readonly<{ file: string }> | Readonly<{ error: string }>;
const SOURCE_EXTENSION = /\.(?:tsx|ts|jsx|js)$/u;
export function findRouteChunk(
manifest: Readonly<Record<string, unknown>>,
routeId: string,
moduleId: string | undefined,
): RouteChunkLookup {
if (moduleId === undefined) {
return { error: `Missing built route chunk: ${routeId} has no route runtime contract (ROUTE_RUNTIME_CONTRACT)` };
}
const matches: { source: string; file: string }[] = [];
for (const [key, entry] of Object.entries(manifest)) {
if (!isManifestRecord(entry) || entry["isDynamicEntry"] !== true || typeof entry["file"] !== "string") continue;
const source = typeof entry["src"] === "string" ? entry["src"] : key;
if (path.posix.basename(source).replace(SOURCE_EXTENSION, "") === moduleId) {
matches.push({ source, file: entry["file"] });
}
}
if (matches.length === 1) return { file: matches[0]!.file };
if (matches.length === 0) {
return {
error:
`Missing built route chunk: ${routeId}: no lazily imported module in the Vite manifest has the source ` +
`file ${moduleId}.tsx (or .ts). A route runtime's moduleId must be the file name of the page it imports.`,
};
}
return {
error:
`Ambiguous built route chunk: ${routeId}: the lazily imported modules ` +
`${matches.map(({ source }) => source).join(", ")} are all named ${moduleId}.`,
};
}
function isManifestRecord(value: unknown): value is Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value);
}
+33 -8
View File
@@ -142,9 +142,9 @@ export async function probeLiveRelease(
const negotiateAlpn = transport.negotiateAlpn ?? negotiatedAlpnProtocol;
const responses: ResponseHeaders = {};
const results: ProbeResult[] = [];
const probe = async (surface: string, pathname: string): Promise<string | null> => {
const probe = async (surface: string, target: URL): Promise<string | null> => {
try {
const response = await fetcher(new URL(pathname, base));
const response = await fetcher(target);
const finalOrigin = new URL(response.url).origin;
results.push(
{
@@ -191,11 +191,12 @@ export async function probeLiveRelease(
const bodies: Record<string, string | null> = {};
for (const [surface, pathname] of Object.entries(paths)) {
bodies[surface] = await probe(surface, pathname);
bodies[surface] = await probe(surface, surfaceUrl(pathname, base));
}
const entryScript = servedEntryScript(bodies["index"] ?? "");
if (entryScript) {
await probe("hashedAsset", entryScript);
// The served index.html names it as the build wrote it (under the base path).
await probe("hashedAsset", new URL(entryScript, base));
} else {
results.push({
surface: "hashedAsset",
@@ -226,6 +227,21 @@ export async function probeLiveRelease(
});
}
const candidateEntryScript = servedEntryScript(candidate.indexHtml);
// 0929/13-06. The candidate was built for one base path; a HOSTING_BASE_URL
// with another path probes surfaces the release does not serve.
// The build writes its scripts to `<base>assets/`, so the entry script's
// folder says which base the candidate was built for, in both directions.
const builtFor = candidateEntryScript?.startsWith(`${base.pathname}${BUILD_ASSET_DIRECTORY}`) === true;
results.push({
surface: "deployment",
header: "base-path",
expected: `${base.pathname}${BUILD_ASSET_DIRECTORY}`,
observed: candidateEntryScript,
...(builtFor
? {}
: { reason: "the candidate was built for another base path than the HOSTING_BASE_URL path (VITE_ROUTER_BASE_PATH of the build)" }),
passed: builtFor,
});
results.push({
surface: "index",
header: "candidate-entry-script",
@@ -405,10 +421,19 @@ export function checkPublicServiceWorkers(
};
}
/**
* 0929/13-06. The surfaces in config/hosting/cache-policy.json are written
* from the release's root (`/config.json`); a release served under a base
* path (`VITE_ROUTER_BASE_PATH=/app/`) serves them under that path.
*/
export function surfaceUrl(pathname: string, base: URL): URL {
return new URL(`.${pathname.startsWith("/") ? pathname : `/${pathname}`}`, base);
}
/**
* A release gate must not promote a local preview server as live hosting
* evidence.
*
* evidence. 0929/13-06: the URL is the release's base URL, the root or the
* sub-path the build was made for (`/app/`).
*/
export function classifyLiveHostingBaseUrl(value: string):
| {
@@ -467,10 +492,10 @@ export function classifyLiveHostingBaseUrl(value: string):
observedOrigin,
};
}
if (url.pathname !== "/" || url.search || url.hash) {
if (!url.pathname.endsWith("/") || url.search || url.hash) {
return {
passed: false,
reason: "HOSTING_BASE_URL must be the canonical root URL",
reason: "HOSTING_BASE_URL must be the release's base URL: a path ending in \"/\", without query or fragment",
url,
observedOrigin,
};
+6 -7
View File
@@ -26,6 +26,7 @@ import {
} from "../contracts/release-artifacts.ts";
import {
CANONICAL_VITE_MANIFEST_PATH,
findRouteChunk,
verifyBuildManifestOutputs,
} from "./build-manifest-outputs.ts";
import { assertMatchesJsonSchema } from "./json-schema.ts";
@@ -1255,14 +1256,12 @@ async function verifyReleaseOutputs(
const runtimeContracts: Readonly<Record<string, { moduleId: string }>> =
ROUTE_RUNTIME_CONTRACT;
for (const definition of Object.values(ROUTE_REGISTRY)) {
const runtime = runtimeContracts[definition.routeId];
const viteEntry = Object.values(viteManifest).find(
(entry) =>
isRecord(entry) &&
entry.name === runtime?.moduleId &&
entry.isDynamicEntry === true,
const chunk = findRouteChunk(
viteManifest,
definition.routeId,
runtimeContracts[definition.routeId]?.moduleId,
);
const file = isRecord(viteEntry) ? viteEntry.file : null;
const file = "error" in chunk ? null : chunk.file;
if (
typeof file !== "string" ||
release.routeChunks[definition.chunkId] !== file ||
+46 -7
View File
@@ -1,10 +1,24 @@
export type ProviderKind = "vulnerability" | "provenance";
/** The oldest systemd user manager whose `--scope` properties the provider relies on. */
export const MINIMUM_PROVIDER_SYSTEMD_VERSION = 254;
/** `systemctl` arguments that ask the user manager its version; the provider and the prerequisite check both use them. */
export const USER_MANAGER_VERSION_ARGUMENTS: readonly string[] = Object.freeze([
"--user",
"show",
"--property=Version",
"--value",
]);
const MEMORY_MAX = 1_073_741_824;
const TASKS_MAX = 64;
const STOP_TIMEOUT_MS = 5_000;
const RUNTIME_GRACE_MS = 10_000;
const UNIT_NAME = /^ca-provider-(?:vulnerability|provenance)-[1-9][0-9]*-[0-9a-f]{24}\.scope$/u;
const PROBE_UNIT_NAME = /^ca-provider-prerequisite-[1-9][0-9]*-[0-9a-f]{24}\.scope$/u;
/** How long the prerequisite probe's scope may live; it runs /bin/true. */
const PROBE_RUNTIME_MS = 5_000;
const UNIT_NONCE = /^[0-9a-f]{24}$/u;
const ENVIRONMENT_NAME = /^[A-Za-z_][A-Za-z0-9_]*$/u;
const ENFORCEMENT_GATE = [
@@ -79,6 +93,37 @@ export function systemdRunProviderArguments(
) {
throw new TypeError("provider scope wrapper path is invalid");
}
return [
...scopeArguments(unit, timeoutMs + RUNTIME_GRACE_MS),
nodeExecutable,
wrapperScript,
String(cpuSeconds),
reportPath,
String(reportDev),
String(reportIno),
];
}
export function formatProviderPrerequisiteUnitName(supervisorPid: number, nonce: string): string {
if (!Number.isSafeInteger(supervisorPid) || supervisorPid <= 0 || !UNIT_NONCE.test(nonce)) {
throw new TypeError("provider cgroup unit identity is invalid");
}
return `ca-provider-prerequisite-${supervisorPid}-${nonce}.scope`;
}
/**
* 1112/08-03. The provider's own scope — same properties, same check that the
* unit's cgroup enforces them — around /bin/true. A host whose user manager
* cannot start the scope, or does not delegate the memory, pids and cpu
* controllers, fails here instead of in each provider test.
*/
export function systemdRunPrerequisiteProbeArguments(unit: string): string[] {
if (!PROBE_UNIT_NAME.test(unit)) throw new TypeError("provider cgroup unit is invalid");
return [...scopeArguments(unit, PROBE_RUNTIME_MS), "/bin/true"];
}
/** `systemd-run` options and the enforcement gate up to its `$0`; the caller appends what the gate execs. */
function scopeArguments(unit: string, runtimeMaxMs: number): string[] {
return [
"--user",
"--scope",
@@ -94,19 +139,13 @@ export function systemdRunProviderArguments(
"--property=KillMode=control-group",
"--property=SendSIGKILL=yes",
`--property=TimeoutStopSec=${STOP_TIMEOUT_MS}ms`,
`--property=RuntimeMaxSec=${timeoutMs + RUNTIME_GRACE_MS}ms`,
`--property=RuntimeMaxSec=${runtimeMaxMs}ms`,
"--",
"/bin/sh",
"-eu",
"-c",
ENFORCEMENT_GATE,
unit,
nodeExecutable,
wrapperScript,
String(cpuSeconds),
reportPath,
String(reportDev),
String(reportIno),
];
}
+2
View File
@@ -12,6 +12,8 @@ export const testEvidenceReportSchema = z
sharedScenarios: z.number().int().nonnegative(),
declaredScenarioExecutions: z.number().int().nonnegative(),
executedScenarioExecutions: z.number().int().nonnegative(),
// 1112/03-03. The Playwright configs the forbidOnly rule read.
playwrightConfigs: z.array(z.string().min(1)).optional(),
})
.strict(),
failures: z.array(z.string()),
+3 -2
View File
@@ -20,8 +20,10 @@ import {
encodeProviderBwrapInput,
encodeProviderScopeFrame,
formatProviderCgroupUnitName,
MINIMUM_PROVIDER_SYSTEMD_VERSION,
systemctlKillProviderArguments,
systemdRunProviderArguments,
USER_MANAGER_VERSION_ARGUMENTS,
} from "./lib/provider-cgroup.ts";
import {
assertProviderGuardianLeasePaths,
@@ -49,7 +51,6 @@ const PROVIDER_REAP_TIMEOUT_MS = 5_000;
const PROVIDER_CONTROL_TIMEOUT_MS = 5_000;
const PROVIDER_CONTROL_MAX_OUTPUT_BYTES = 65_536;
const PROVIDER_CONTROL_POLL_MS = 25;
const MINIMUM_PROVIDER_SYSTEMD_VERSION = 254;
if (kind !== "vulnerability" && kind !== "provenance") {
process.stderr.write("Usage: run-and-validate-provider --kind vulnerability|provenance\n");
process.exit(2);
@@ -501,7 +502,7 @@ async function waitForProviderClose(
async function assertProviderCgroupManagerAvailable(): Promise<void> {
const output = await runBoundedSystemctl(
["--user", "show", "--property=Version", "--value"],
USER_MANAGER_VERSION_ARGUMENTS,
"provider cgroup user manager probe",
);
const match = /^([0-9]+)/u.exec(output.trim());
+35 -3
View File
@@ -304,16 +304,44 @@ function linkedPath(target: string, document: string): string | null {
// never looked up. Leading `NAME=value` assignments are skipped, a
// `--flag=value` argument is read as its value, and placeholders (`<id>`,
// globs, `$VAR`) are left alone by namedPath.
const COMMAND_HEADS = new Set(["corepack", "pnpm", "npx", "node", "vitest", "playwright", "tsc", "eslint", "bash", "sh"]);
function commandPaths(command: string, document: string): (string | null)[] {
const COMMAND_HEADS = new Set([
"corepack", "pnpm", "npx", "node", "vitest", "playwright", "tsc", "eslint", "bash", "sh", "python3",
]);
// 1112/08-06. `scripts/verify.sh` is this repository's main entry and most
// documented commands go through it: the entry is itself a path, `exec --`
// runs a command whose paths are looked up like any other, and `script`
// names package scripts (verifyScripts).
const VERIFY_ENTRY = /^(?:\.\/)?scripts\/verify\.sh$/u;
function commandWords(command: string): Readonly<{ words: string[]; head: number }> {
const words = command.trim().replace(/^\$\s+/u, "").split(/\s+/u);
let head = 0;
while (/^[A-Z_][A-Z0-9_]*=/u.test(words[head] ?? "")) head += 1;
if (!COMMAND_HEADS.has(words[head] ?? "")) return [];
return { words, head };
}
function commandPaths(command: string, document: string): (string | null)[] {
const { words, head } = commandWords(command);
const first = words[head] ?? "";
if (VERIFY_ENTRY.test(first)) {
const separator = words.indexOf("--", head + 1);
const executed = words[head + 1] === "exec" && separator > head ? words.slice(separator + 1).join(" ") : "";
return [namedPath(first, document), ...(executed === "" ? [] : commandPaths(executed, document))];
}
if (!COMMAND_HEADS.has(first)) return [];
return words
.slice(head + 1)
.map((word) => namedPath(word.replace(/^--?[\w.-]+=/u, "").replace(/^["']|["']$/gu, ""), document));
}
/** The package scripts `scripts/verify.sh script <name>… [-- args]` runs; placeholders are left alone. */
function verifyScripts(command: string): string[] {
const { words, head } = commandWords(command);
if (!VERIFY_ENTRY.test(words[head] ?? "") || words[head + 1] !== "script") return [];
const names: string[] = [];
for (const word of words.slice(head + 2)) {
if (word.startsWith("-")) break;
names.push(word);
}
return names.filter((name) => /^[a-z](?:[\w:-]*\w)?$/u.test(name));
}
// Lines of fenced shell blocks (```bash, ```sh, ```shell, ```console or no
// language), which is where most documented commands are.
function fencedCommands(text: string): string[] {
@@ -398,6 +426,10 @@ for (const document of contractDocuments) {
for (const [, name = ""] of text.matchAll(/`([a-z][\w-]*:[\w:-]*\w)(?=[` ])/gu)) {
if (scriptPrefixes.has(name.split(":")[0]) && missingScript(name)) missingScripts.add(name);
}
const commands = [...[...text.matchAll(/`([^`\n]+)`/gu)].map(([, token = ""]) => token), ...fencedCommands(text)];
for (const name of commands.flatMap(verifyScripts)) {
if (missingScript(name)) missingScripts.add(name);
}
documentReferences.push({
path: document,
missingPaths: [...missingPaths].sort(),
+1 -1
View File
@@ -121,7 +121,7 @@ if (liveTarget?.passed && !slotEmpty) {
probeResults.push({
surface: "deployment",
header: "base-url",
expected: "canonical non-loopback HTTPS root URL",
expected: "non-loopback HTTPS base URL of the release",
observed: liveTarget.observedOrigin,
reason: liveTarget.reason,
passed: false,
+6 -4
View File
@@ -24,6 +24,7 @@ import {
import { assertMatchesJsonSchema } from "./lib/json-schema.ts";
import {
CANONICAL_VITE_MANIFEST_PATH,
findRouteChunk,
verifyBuildManifestOutputs,
} from "./lib/build-manifest-outputs.ts";
import { verifyReleaseRuntimeCoherence } from "./lib/release-runtime-coherence.ts";
@@ -168,12 +169,13 @@ for (const chunkId of actualChunkIds) {
const runtimeContracts: Readonly<Record<string, { moduleId: string }>> =
ROUTE_RUNTIME_CONTRACT;
for (const definition of Object.values(ROUTE_REGISTRY)) {
const runtime = runtimeContracts[definition.routeId];
const viteEntry = Object.values(viteManifestObject).find(
(entry) => entry.name === runtime?.moduleId && entry.isDynamicEntry,
const viteEntry = findRouteChunk(
viteManifestObject,
definition.routeId,
runtimeContracts[definition.routeId]?.moduleId,
);
const routeAsset = release.routeChunks[definition.chunkId];
if (!runtime || !viteEntry || routeAsset !== viteEntry.file) {
if ("error" in viteEntry || routeAsset !== viteEntry.file) {
artifactMismatches.push(`routeChunk:mismatch:${definition.chunkId}`);
continue;
}
+8
View File
@@ -4,10 +4,18 @@ import path from "node:path";
import { reproducibleBuildArtifactSchema } from "./contracts/release-artifacts.ts";
import { assertCiBuildEnvironment } from "./lib/build-environment.ts";
import { checkoutBuildReadFailure } from "./lib/run-isolation.ts";
import { supplyChainDigest } from "./lib/supply-chain.ts";
import { writeValidatedJsonArtifact } from "./lib/validated-json-artifact.ts";
assertCiBuildEnvironment(process.env);
// 1106/12-02. This builds twice and digests dist where the checkout keeps it. Under RUN_OUTPUT_ROOT the build wrote elsewhere, so
// this would judge the earlier build the checkout still holds.
const outputRootFailure = checkoutBuildReadFailure("scripts/verify-reproducible-build.ts");
if (outputRootFailure !== undefined) {
process.stderr.write(`${outputRootFailure}\n`);
process.exit(1);
}
async function filesWithin(directory: string): Promise<string[]> {
const entries = await readdir(directory, { withFileTypes: true });
+29
View File
@@ -3,7 +3,9 @@ import path from "node:path";
import { describe, expect, it } from "vitest";
import {
checkRequestFanoutFixtures,
findRequestFanout,
formatFanoutViolation,
scanRequestFanout,
} from "../../scripts/check-request-fanout.ts";
@@ -21,6 +23,13 @@ describe("no per-row transport loop (frontend N+1 source gate)", () => {
expect(new Set(forbidden.violations.map((violation) => violation.file)).size).toBe(forbidden.files);
expect(forbidden.files).toBe(3);
expect(allowed.violations).toEqual([]);
expect(allowed.reviewed).toEqual([
expect.objectContaining({
file: "tests/fixtures/architecture/request-fanout/allowed/reviewed-cursor-loop.ts",
call: "gateway.page()",
reason: "cursor pages, bounded by maxPages",
}),
]);
});
it("names the dispatch and the repeating construct", async () => {
@@ -40,4 +49,24 @@ describe("no per-row transport loop (frontend N+1 source gate)", () => {
);
expect(violations).toEqual([]);
});
// 0929/12-01. The failure line named only the call, so a product team saw
// neither the cursor runtime nor the reviewed-exception comment, and the
// result line hid how many calls a comment let through.
it("tells the author both ways out on the failure line", async () => {
const [violation] = await findRequestFanout(
"inline.ts",
"export const load = (api: { get(id: string): Promise<unknown> }, ids: string[]) =>\n Promise.all(ids.map((id) => api.get(id)));\n",
);
const line = formatFanoutViolation(violation!);
expect(line).toMatch(/^FAIL inline\.ts:2 api\.get\(\) inside \.map\(\) callback: /u);
expect(line).toContain("createCursorPaginationRuntime (src/adapters/query-cache)");
expect(line).toContain("`// fanout-reviewed: <reason>` on the call's line");
});
it("counts and lists the reviewed exceptions in the fixture result line", async () => {
const result = await checkRequestFanoutFixtures(path.resolve("."));
expect(result.ok).toBe(true);
expect(result.line).toMatch(/allowed 0 violations and 1 reviewed exception \(tests\/fixtures\/architecture\/request-fanout\/allowed\/reviewed-cursor-loop\.ts:\d+ cursor pages, bounded by maxPages\) PASS/u);
});
});
@@ -337,6 +337,7 @@ async function removeOpfsAuthority(
for (const committedObjects of [0, 3]) {
test(`05-01 an approved purge in another tab waits for a live put and takes it with the partition (${committedObjects} committed objects)`, async ({
page,
browserName,
}) => {
const secondPage = await page.context().newPage();
const suffix = crypto.randomUUID().replaceAll("-", "");
@@ -349,7 +350,18 @@ for (const committedObjects of [0, 3]) {
type Tab = import("../../src/adapters/storage/opfs/browser-opfs-runtime.ts").BrowserOpfsRuntime;
try {
await Promise.all([page.goto("/"), secondPage.goto("/")]);
if (!(await installOpfsTab(page, scope))) return;
// 1112/03-02. An early return here passed with no assertion on an engine
// the support check refused, while the evidence said zero skipped.
// Playwright's WebKit is refused (the first test asserts its UNSUPPORTED
// result), so there this asserts the refusal; every other engine must
// open the runtime. A skip is no way out: the capability evidence
// refuses skipped cases (scripts/verify-browser-capability-evidence.ts).
const installed = await installOpfsTab(page, scope);
if (browserName === "webkit") {
expect(installed, "the OPFS support check refuses Playwright's WebKit").toBe(false);
return;
}
expect(installed, "OPFS runtime supported in this engine").toBe(true);
expect(await installOpfsTab(secondPage, scope, true)).toBe(true);
const seeded = await secondPage.evaluate(async (count) => {
@@ -455,6 +467,7 @@ for (const committedObjects of [0, 3]) {
test("05-01 a reconcile in another tab skips a live put and the put stays readable", async ({
page,
browserName,
}) => {
const secondPage = await page.context().newPage();
const suffix = crypto.randomUUID().replaceAll("-", "");
@@ -467,7 +480,13 @@ test("05-01 a reconcile in another tab skips a live put and the put stays readab
type Tab = import("../../src/adapters/storage/opfs/browser-opfs-runtime.ts").BrowserOpfsRuntime;
try {
await Promise.all([page.goto("/"), secondPage.goto("/")]);
if (!(await installOpfsTab(page, scope))) return;
// 1112/03-02. Asserted, not returned early: see the purge test above.
const installed = await installOpfsTab(page, scope);
if (browserName === "webkit") {
expect(installed, "the OPFS support check refuses Playwright's WebKit").toBe(false);
return;
}
expect(installed, "OPFS runtime supported in this engine").toBe(true);
expect(await installOpfsTab(secondPage, scope)).toBe(true);
// Tab A starts a put whose source stalls, so its journal row stays
+19
View File
@@ -298,6 +298,25 @@ recordedTest.describe("the strict fixture's console check", () => {
);
});
// 1112/03-05. The fixture watched its own page only, so a console error in
// a second tab of the same context (the cross-tab capability specs) passed.
recordedTest.describe("the strict fixture's other tabs", () => {
recordedTest.use({
fixtureReports: { lines: [/^console:error:\[preload-use\] second tab$/u] },
});
recordedTest(
"records a console error in another tab of the context",
async ({ page }) => {
await openHome(page);
const second = await page.context().newPage();
await openHome(second);
await second.evaluate(() => console.error("[preload-use] second tab"));
},
);
});
/**
* 1105/11-01. The verdict every other spec gets: a recorded failure fails the
* test. The record is proven above; this proves the verdict.
+12 -13
View File
@@ -1,6 +1,11 @@
import path from "node:path";
import { expect, test, type Page } from "../support/browser/strict-browser-test.ts";
import {
expect,
test,
watchBrowserFailures,
type Page,
} from "../support/browser/strict-browser-test.ts";
import {
prepareServiceWorkerBuilds,
removeServiceWorkerBuilds,
@@ -91,7 +96,7 @@ test("a REMOVE_REGISTRATION release unregisters a worker that blocks boot", asyn
const server = await startBuildServer(builds.dist);
// The broken phase fails requests on purpose, so this test drives its own
// context instead of the strict `page` fixture and checks the recovered
// load for browser errors itself.
// load for browser errors with the fixture's own watcher (1112/03-05).
const context = await browser.newContext();
try {
const page = await context.newPage();
@@ -133,22 +138,16 @@ test("a REMOVE_REGISTRATION release unregisters a worker that blocks boot", asyn
)
.toBe(true);
const failures: string[] = [];
page.on("console", (message) => {
if (["error", "warning"].includes(message.type())) {
failures.push(`console:${message.type()}:${message.text()}`);
}
});
page.on("pageerror", (error) => failures.push(`pageerror:${error.name}`));
page.on("requestfailed", (request) =>
failures.push(`requestfailed:${new URL(request.url()).pathname}`),
);
// No preload check here: this context loaded the origin twice already,
// and WebKit answers the recovered load's scripts from its cache, past
// the tracker's route, so every preload reads as unused (16 in a run).
const recovered = await watchBrowserFailures(page);
await page.goto(`${server.origin}/`);
expect(await startOutcome(page)).toEqual({ kind: "DISABLED" });
expect(
await page.evaluate(() => navigator.serviceWorker.controller === null),
).toBe(true);
expect(failures).toEqual([]);
expect(await recovered.stop()).toEqual([]);
} finally {
await context.close();
await server.close();
+57 -16
View File
@@ -65,17 +65,41 @@ function run(
}
}
export async function prepareFixtureBuilds<Name extends string>(
prefix: string,
builds: readonly FixtureBuild<Name>[],
): Promise<FixtureBuilds<Name>> {
// An edit's source text names the build (two specs never share a prefix).
const fingerprint = JSON.stringify(
builds.map((build) => [build.name, build.env ?? {}, build.edit?.toString() ?? ""]),
);
export type EditedCopyBuild<Name extends string> = Readonly<{
name: Name;
/** VITE_BUILD_ID and RELEASE_ID of this build. */
buildId: string;
edit?: (root: string) => Promise<void>;
env?: Readonly<Record<string, string>>;
}>;
/**
* 0929/11-07. The one way an e2e spec gets real production builds of an
* edited copy of the checkout; prepareFixtureBuilds and
* prepareServiceWorkerBuilds (service-worker-builds.ts) are its two shapes.
* It copies the checkout (without `checkout-copy-skip`), links node_modules,
* runs `prepare` once, then for each build runs `edit`, builds, moves `dist`
* to `dist-<name>` and asks `collect` what the spec needs from that build.
* `key` and `fingerprint` name the result in the e2e run
* (shared-fixture-builds.ts), which removes the copy when preparing fails.
*/
export async function buildEditedCopies<Name extends string, Collected = null>(
input: Readonly<{
key: string;
fingerprint: string;
prepare?: (root: string) => Promise<void>;
builds: readonly EditedCopyBuild<Name>[];
collect?: (name: Name, root: string) => Promise<Collected>;
}>,
): Promise<Readonly<{
root: string;
dist: Readonly<Record<Name, string>>;
collected: Readonly<Record<Name, Collected>>;
shared: boolean;
}>> {
const { root, value, shared } = await sharedFixtureBuild(
`fixture-${prefix}`,
fingerprint,
input.key,
input.fingerprint,
async (root) => {
await cp(repoRoot, root, {
recursive: true,
@@ -83,19 +107,36 @@ export async function prepareFixtureBuilds<Name extends string>(
!skipped.has(path.relative(repoRoot, source).split(path.sep)[0]!),
});
await linkFixtureNodeModules(root, repoRoot);
await input.prepare?.(root);
const dist = {} as Record<Name, string>;
for (const build of builds) {
const collected = {} as Record<Name, Collected>;
for (const build of input.builds) {
await build.edit?.(root);
const buildId = `${prefix}-${build.name}`;
run(root, "scripts/generate-product-feature-index.ts", buildId, build.env);
run(root, "scripts/build-frontend.ts", buildId, build.env);
run(root, "scripts/generate-product-feature-index.ts", build.buildId, build.env);
run(root, "scripts/build-frontend.ts", build.buildId, build.env);
dist[build.name] = path.join(root, `dist-${build.name}`);
await rename(path.join(root, "dist"), dist[build.name]);
collected[build.name] = input.collect ? await input.collect(build.name, root) : (null as Collected);
}
return dist;
return { dist, collected };
},
);
return { root, dist: value, shared };
return { root, ...value, shared };
}
export async function prepareFixtureBuilds<Name extends string>(
prefix: string,
builds: readonly FixtureBuild<Name>[],
): Promise<FixtureBuilds<Name>> {
// An edit's source text names the build (two specs never share a prefix).
const { root, dist, shared } = await buildEditedCopies({
key: `fixture-${prefix}`,
fingerprint: JSON.stringify(
builds.map((build) => [build.name, build.env ?? {}, build.edit?.toString() ?? ""]),
),
builds: builds.map((build) => ({ ...build, buildId: `${prefix}-${build.name}` })),
});
return { root, dist, shared };
}
export async function removeFixtureBuilds(
+19 -67
View File
@@ -1,12 +1,5 @@
import { spawnSync } from "node:child_process";
import { existsSync } from "node:fs";
import {
cp,
readFile,
rename,
rm,
writeFile,
} from "node:fs/promises";
import { readFile, rm, writeFile } from "node:fs/promises";
import { createServer, type Server } from "node:http";
import type { AddressInfo } from "node:net";
import path from "node:path";
@@ -14,11 +7,8 @@ import { pathToFileURL } from "node:url";
import { gzipSync } from "node:zlib";
import { SERVICE_WORKER_ASSETS_OUTPUT } from "../../../scripts/generate-service-worker-assets.ts";
import { localBuildEnvironment } from "../../../scripts/lib/build-environment.ts";
import { linkFixtureNodeModules } from "../../../scripts/lib/fixture-node-modules.ts";
import { selectRepositoryPaths } from "../../../scripts/lib/repository-layout.ts";
import type { StaticAssetManifestV1 } from "../../../src/contracts/service-worker.ts";
import { sharedFixtureBuild } from "./shared-fixture-builds.ts";
import { buildEditedCopies, editOnce } from "./fixture-builds.ts";
/**
* R3-6 test helper: two real production builds with the Service Worker on,
@@ -54,11 +44,6 @@ import { sharedFixtureBuild } from "./shared-fixture-builds.ts";
* text it expected.
*/
const repoRoot = path.resolve(import.meta.dirname, "../../..");
// 1105/12-04. `.tmp` holds other runs' builds and results (.tmp/runs/<id>),
// which change while this copy is made; fixture-builds.ts skips it too.
const skipped = new Set(selectRepositoryPaths("checkout-copy-skip"));
export type BuildName = "a" | "b";
export type ServiceWorkerBuilds = Readonly<{
root: string;
@@ -101,17 +86,8 @@ export function createServiceWorkerRuntimeHost(
}
`;
export async function replaceOnce(
file: string,
search: string,
replacement: string,
): Promise<void> {
const source = await readFile(file, "utf8");
if (source.split(search).length !== 2) {
throw new Error(`Expected exactly one "${search}" in ${file}.`);
}
await writeFile(file, source.replace(search, replacement));
}
/** 0929/11-07. The same one-occurrence edit as fixture-builds.ts, not a copy of it. */
export const replaceOnce = editOnce;
/**
* Premise 1. Replaces the value of the one `SERVICE_WORKER` declaration in
@@ -167,25 +143,12 @@ async function precachedUrls(root: string, name: BuildName): Promise<string[]> {
return SERVICE_WORKER_ASSETS.assets.map((asset) => asset.url).sort();
}
function run(cwd: string, script: string, buildId: string): void {
const result = spawnSync("node", [script], {
cwd,
encoding: "utf8",
// 1103/11-08. A local build, not the CI job's release build.
env: { ...localBuildEnvironment(), VITE_BUILD_ID: buildId, RELEASE_ID: buildId },
});
if (result.status !== 0) {
throw new Error(
`${script} failed for ${buildId}:\n${result.stdout}\n${result.stderr}`,
);
}
}
/**
* `beforeBuild` edits the temp copy right before build `name` (06-03: a
* broken worker A, then a `REMOVE_REGISTRATION` release B). Edits persist
* into the later build. `key` names this pair of builds in the e2e run
* (shared-fixture-builds.ts); each spec passes its own.
* (shared-fixture-builds.ts); each spec passes its own. The copy and the
* builds are fixture-builds.ts's `buildEditedCopies` (0929/11-07).
*/
export async function prepareServiceWorkerBuilds(
options: Readonly<{
@@ -193,38 +156,27 @@ export async function prepareServiceWorkerBuilds(
beforeBuild?: (name: BuildName, root: string) => Promise<void>;
}>,
): Promise<ServiceWorkerBuilds> {
const { root, value, shared } = await sharedFixtureBuild(
`sw-${options.key}`,
options.beforeBuild?.toString() ?? "",
async (root) => {
await cp(repoRoot, root, {
recursive: true,
filter: (source) =>
!skipped.has(path.relative(repoRoot, source).split(path.sep)[0]!),
});
await linkFixtureNodeModules(root, repoRoot);
const { root, dist, collected, shared } = await buildEditedCopies({
key: `sw-${options.key}`,
fingerprint: options.beforeBuild?.toString() ?? "",
prepare: async (root) => {
await turnOnServiceWorker(root);
const hostFile = path.join(root, "src/bootstrap/register-service-worker.ts");
await replaceOnce(
await editOnce(
hostFile,
"export function createServiceWorkerRuntimeHost(",
"function createServiceWorkerRuntimeHostUnobserved(",
);
await writeFile(hostFile, (await readFile(hostFile, "utf8")) + HOST_PROBE);
const dist = { a: path.join(root, "dist-a"), b: path.join(root, "dist-b") };
const precached: Record<BuildName, readonly string[]> = { a: [], b: [] };
for (const name of ["a", "b"] as const) {
await options.beforeBuild?.(name, root);
run(root, "scripts/generate-product-feature-index.ts", `sw-e2e-${name}`);
run(root, "scripts/build-frontend.ts", `sw-e2e-${name}`);
await rename(path.join(root, "dist"), dist[name]);
precached[name] = await precachedUrls(root, name);
}
return { dist, precached };
},
);
return { root, ...value, shared };
builds: (["a", "b"] as const).map((name) => ({
name,
buildId: `sw-e2e-${name}`,
edit: async (root: string) => options.beforeBuild?.(name, root),
})),
collect: (name, root) => precachedUrls(root, name),
});
return { root, dist, precached: collected, shared };
}
export async function removeServiceWorkerBuilds(
@@ -0,0 +1,18 @@
// Allowed: a cursor loop someone reviewed. Each page needs the cursor the
// previous page returned, so the requests cannot be one bulk read; the count
// is bounded by the page cap. Feature code normally gets this from
// createCursorPaginationRuntime (src/adapters/query-cache).
type Page = Readonly<{ items: readonly string[]; nextCursor: string | null }>;
type Gateway = { page(cursor: string | null): Promise<Page> };
export async function loadAll(gateway: Gateway, maxPages: number) {
const items: string[] = [];
let cursor: string | null = null;
for (let page = 0; page < maxPages; page += 1) {
const next: Page = await gateway.page(cursor); // fanout-reviewed: cursor pages, bounded by maxPages
items.push(...next.items);
if (next.nextCursor === null) break;
cursor = next.nextCursor;
}
return items;
}
@@ -0,0 +1,8 @@
// A Playwright config that would let a committed focused test run alone:
// check-test-evidence.ts --playwright-config must reject it (1112/03-03).
import { defineConfig } from "@playwright/test";
export default defineConfig({
testDir: "tests/e2e",
reporter: [["junit", { outputFile: "artifacts/tests/e2e/results.xml" }]],
});
+53 -24
View File
@@ -20,6 +20,48 @@ function ignoredConsole(message: ConsoleMessage) {
);
}
/**
* 1112/03-05. Records a page's browser failures as `console:…`, `pageerror:…`
* and `requestfailed:…` lines, with the same rules the strict fixture judges
* by. The fixture watches its `page` and every other tab opened in the same
* context; a spec that drives a context of its own watches its pages with
* this. `preloads` also checks that every modulepreload ran (preload-use.ts)
* in the documents open when `stop` runs; attach it before the navigation
* that loads them.
*/
export async function watchBrowserFailures(
page: Page,
options: Readonly<{ preloads?: boolean }> = {},
): Promise<Readonly<{ stop(): Promise<string[]> }>> {
const failures: string[] = [];
const onConsole = (message: ConsoleMessage) => {
if (["error", "warning"].includes(message.type()) && !ignoredConsole(message)) {
failures.push(`console:${message.type()}:${message.text()}`);
}
};
const onPageError = (error: Error) => {
failures.push(`pageerror:${error.name}`);
};
const onRequestFailed = (request: import("@playwright/test").Request) => {
failures.push(
`requestfailed:${request.method()}:${new URL(request.url()).pathname}:${request.failure()?.errorText ?? "unknown"}`,
);
};
page.on("console", onConsole);
page.on("pageerror", onPageError);
page.on("requestfailed", onRequestFailed);
const preloads = options.preloads ? await trackPreloadUse(page) : null;
return Object.freeze({
async stop() {
if (preloads) failures.push(...(await preloads.check()).failures);
page.off("console", onConsole);
page.off("pageerror", onPageError);
page.off("requestfailed", onRequestFailed);
return [...failures];
},
});
}
/**
* 1105/11-01. What the strict `page` fixture does with every failure it
* recorded, once the test has ended.
@@ -62,32 +104,19 @@ export const test = base.extend<{
);
},
page: async ({ page, browserFailureVerdict }, use) => {
const preloads = await trackPreloadUse(page);
const failures: string[] = [];
const onConsole = (message: ConsoleMessage) => {
if (
["error", "warning"].includes(message.type()) &&
!ignoredConsole(message)
) {
failures.push(`console:${message.type()}:${message.text()}`);
}
const watched = [await watchBrowserFailures(page, { preloads: true })];
// 1112/03-05. A second tab of the same context (the cross-tab specs)
// went unwatched. Its listeners are attached as the tab opens, before
// the spec can navigate it; preload use is checked on the first page only.
const pending: Promise<void>[] = [];
const onPage = (opened: Page) => {
pending.push(watchBrowserFailures(opened).then((watcher) => void watched.push(watcher)));
};
const onPageError = (error: Error) => {
failures.push(`pageerror:${error.name}`);
};
const onRequestFailed = (request: import("@playwright/test").Request) => {
failures.push(
`requestfailed:${request.method()}:${new URL(request.url()).pathname}:${request.failure()?.errorText ?? "unknown"}`,
);
};
page.on("console", onConsole);
page.on("pageerror", onPageError);
page.on("requestfailed", onRequestFailed);
page.context().on("page", onPage);
await use(page);
failures.push(...(await preloads.check()).failures);
page.off("console", onConsole);
page.off("pageerror", onPageError);
page.off("requestfailed", onRequestFailed);
page.context().off("page", onPage);
await Promise.all(pending);
const failures = (await Promise.all(watched.map((watcher) => watcher.stop()))).flat();
browserFailureVerdict.judge(failures);
},
});
@@ -377,6 +377,19 @@ describe("RPC-RR-04 transport results and frames are exactly decoded", () => {
});
});
/**
* 1101/10-07. The system clock with every wait capped at 30 ms. Without a
* positive receipt the stream cleanup waits out its 1 s bound; capped, a test
* passes that bound instead of sleeping through it. The operations' own 25 ms
* deadlines are below the cap and unchanged. Every stream test that ends
* without a positive receipt takes it, so none sleeps a real second.
*/
const cappedClock: ClockPort = Object.freeze({
now: () => systemClock.now(),
sleep: (milliseconds: number, signal?: AbortSignal) =>
systemClock.sleep(Math.min(milliseconds, 30), signal),
});
/**
* RPC-RR-01. A bare `AsyncIterable` gave the runtime no way to cancel the
* physical stream or to learn when it actually closed, so a timed-out call left
@@ -395,6 +408,7 @@ describe("RPC-RR-01 server stream leases and the DRAINING fence", () => {
mappers: MAPPERS,
requestEncoders: { RpcResourceStreamRequestEncoder: STREAM_ENCODER },
transports: { CONNECT_REFERENCE_STREAM: probe.transport },
clock: cappedClock,
});
const results = await collect(
@@ -418,6 +432,7 @@ describe("RPC-RR-01 server stream leases and the DRAINING fence", () => {
mappers: MAPPERS,
requestEncoders: { RpcResourceStreamRequestEncoder: STREAM_ENCODER },
transports: { CONNECT_REFERENCE_STREAM: probe.transport },
clock: cappedClock,
});
const stream = runtime.bindServerStream(
"WATCH_RPC_RESOURCES",
@@ -541,18 +556,6 @@ describe("RPC-01 only a positive receipt confirms physical closure", () => {
{ label: "never settles", waitClosed: () => new Promise<void>(() => {}) },
];
/**
* 1101/10-07. The system clock with every wait capped at 30 ms. Without a
* positive receipt the stream cleanup waits out its 1 s bound; capped, the
* test passes that bound instead of sleeping through it. The operation's own
* 25 ms deadlines are below the cap and unchanged.
*/
const cappedClock: ClockPort = Object.freeze({
now: () => systemClock.now(),
sleep: (milliseconds: number, signal?: AbortSignal) =>
systemClock.sleep(Math.min(milliseconds, 30), signal),
});
for (const { label, waitClosed } of negativeReceipts) {
it(`keeps the operation DRAINING when waitClosed ${label}`, async () => {
const probe = leaseProbeWith(waitClosed);
@@ -598,6 +601,7 @@ describe("RPC-01 only a positive receipt confirms physical closure", () => {
mappers: MAPPERS,
requestEncoders: { RpcResourceStreamRequestEncoder: STREAM_ENCODER },
transports: { CONNECT_REFERENCE_STREAM: probe.transport },
clock: cappedClock,
});
const stream = runtime.bindServerStream(
"WATCH_RPC_RESOURCES",
+51
View File
@@ -0,0 +1,51 @@
import path from "node:path";
import { ESLint } from "eslint";
import { describe, expect, it } from "vitest";
import { childProcessTimeout } from "../helpers/child-process-timeout.ts";
const FOLDERS = ["e2e", "browser-capabilities", "visual", "storybook"] as const;
const BARE = 'import { expect, test } from "@playwright/test";\n\ntest("probe", async () => { expect(1).toBe(1); });\n';
const STRICT =
'import { expect, test } from "../support/browser/strict-browser-test.ts";\n' +
'import type { Page } from "@playwright/test";\n\n' +
'test("probe", async ({ page }: { page: Page }) => { expect(page).toBeTruthy(); });\n';
const DEFAULT_IMPORT = 'import test from "@playwright/test";\n\ntest("probe", async () => {});\n';
const BASE_TEST = 'import { test } from "../support/browser/browser-test.ts";\n\ntest("probe", async () => {});\n';
async function restrictedImports(code: string, file: string): Promise<string[]> {
const linter = new ESLint({ cwd: process.cwd() });
const [result] = await linter.lintText(code, { filePath: path.join(process.cwd(), file), warnIgnored: true });
return (result?.messages ?? [])
.filter((message) => message.ruleId === "no-restricted-imports")
.map((message) => message.message);
}
// 1112/03-04. Strict fixture use was a text search for its path in
// tests/e2e/*.spec.ts: a path in a comment passed, a file could mix in the
// bare test, and tests/browser-capabilities, tests/visual and tests/storybook
// were not looked at. It is an import rule in each of the four folders now.
describe("browser specs take test from the strict fixture (1112/03-04)", () => {
it.each(FOLDERS)("rejects the bare Playwright test in tests/%s", async (folder) => {
const messages = await restrictedImports(BARE, `tests/${folder}/lint-probe.spec.ts`);
expect(messages).toEqual([expect.stringContaining("tests/support/browser/strict-browser-test.ts")]);
}, childProcessTimeout(683));
it("rejects the default import, the base test the strict fixture extends, and accepts the strict test", async () => {
expect(await restrictedImports(DEFAULT_IMPORT, "tests/visual/lint-probe.spec.ts")).toEqual([
expect.stringContaining("tests/support/browser/strict-browser-test.ts"),
]);
expect(await restrictedImports(BASE_TEST, "tests/e2e/lint-probe.spec.ts")).toEqual([
expect.stringContaining("the base test does not check browser errors"),
]);
expect(await restrictedImports(STRICT, "tests/browser-capabilities/lint-probe.spec.ts")).toEqual([]);
}, childProcessTimeout(683));
it("lets only the two specs that need the plain test import it", async () => {
for (const file of ["tests/e2e/preload-use.spec.ts", "tests/browser-capabilities/chunk-load-error.spec.ts"]) {
expect(await restrictedImports(BARE, file), file).toEqual([]);
}
}, childProcessTimeout(683));
});
+19 -1
View File
@@ -250,11 +250,29 @@ describe("verify:documentation commands and sibling links (1103/12-05)", () => {
const result = await verify({
[CI_DOC]: (text) => `${text}\n\`corepack pnpm exec vitest run tests/unit/documentation-readiness.test.ts -t "x"\`\n` +
"```bash\ncorepack pnpm drill:runbook -- FE-RB-005 --release-root <release dir>\n```\n" +
"Any script is run as `node scripts/…`.\n",
"Any script is run as `node scripts/…`.\n" +
"`scripts/verify.sh exec -- node scripts/run-vitest.ts run tests/unit/documentation-readiness.test.ts`, " +
"`scripts/verify.sh script verify:documentation lint -- --x`, `scripts/verify.sh script <이름>…`, " +
"`python3 scripts/check-work-docs.py --history <경로>`\n",
});
expect(result.status, result.stderr).toBe(0);
}, VERIFY_TIMEOUT);
// 1112/08-06. Commands through this repository's entry point, and python3
// commands, were not read, so a stale path or script name in them passed.
it("reads commands run through scripts/verify.sh and python3", async () => {
const result = await verify({
[CI_DOC]: (text) => `${text}\n\`scripts/verify.sh exec -- node scripts/no-such.ts\`\n` +
"```bash\nVERIFY_SNAPSHOT=1 scripts/verify.sh exec -- node tests/unit/no-such-shell.test.ts\n```\n" +
"`python3 scripts/no-such.py --history docs`\n`scripts/verify.sh script no:such-script -- --flag`\n",
});
expect(result.status).toBe(1);
expect(result.stderr).toContain(
`${CI_DOC} names paths that do not exist: scripts/no-such.py, scripts/no-such.ts, tests/unit/no-such-shell.test.ts`,
);
expect(result.stderr).toContain(`${CI_DOC} names package scripts that do not exist: no:such-script`);
}, VERIFY_TIMEOUT);
it("checks a link or a path into a paired repository checked out beside this one", async () => {
const naming = (target: string) => (text: string) =>
`${text}\n[contract](../../../${target}) and \`${target}\`\n`;
+36
View File
@@ -0,0 +1,36 @@
import { readFile, readdir } from "node:fs/promises";
import path from "node:path";
import { describe, expect, it } from "vitest";
const SUPPORT = "tests/e2e/support";
// 0929/11-07. The Service Worker helper kept its own copy of the checkout
// copy, the build runner and the one-occurrence edit, so a fix to one (the
// local build env of 1103/11-08, the copy skip list of 1105/12-04) had to be
// made twice. Both helpers now go through fixture-builds.ts.
describe("e2e build helpers have one copy-and-build path (0929/11-07)", () => {
it("builds Service Worker fixtures through buildEditedCopies", async () => {
const source = await readFile(path.join(SUPPORT, "service-worker-builds.ts"), "utf8");
expect(source).toMatch(/import \{[^}]*\bbuildEditedCopies\b[^}]*\} from "\.\/fixture-builds\.ts";/u);
expect(source).toContain("buildEditedCopies({");
for (const copy of ["spawnSync", "sharedFixtureBuild", "linkFixtureNodeModules", "await cp(", "function run("]) {
expect(source, copy).not.toContain(copy);
}
});
it("keeps the build runner and the one-occurrence edit in fixture-builds.ts only", async () => {
const files = (await readdir(SUPPORT)).filter((file) => file.endsWith(".ts"));
const defining = async (pattern: RegExp) => {
const found: string[] = [];
for (const file of files) {
if (pattern.test(await readFile(path.join(SUPPORT, file), "utf8"))) found.push(file);
}
return found;
};
expect(await defining(/^function run\(/mu)).toEqual(["fixture-builds.ts"]);
expect(await defining(/split\(search\)\.length !== 2/u)).toEqual(["fixture-builds.ts"]);
});
});
+64 -1
View File
@@ -73,11 +73,18 @@ describe("live hosting evidence target", () => {
});
});
// 0929/13-06. A release built for VITE_ROUTER_BASE_PATH=/app/ is served
// under /app/; the probe refused any URL but the root, so it could not be used.
it("accepts the sub-path a release is built for", () => {
expect(classifyLiveHostingBaseUrl("https://frontend.example.test/app/")).toMatchObject({ passed: true });
});
it.each([
["http://frontend.example.test/", "requires HTTPS"],
["https://localhost:4173/", "not live deployment evidence"],
["https://127.0.0.1/", "not live deployment evidence"],
["https://frontend.example.test/app/", "canonical root URL"],
["https://frontend.example.test/app", "path ending in \"/\""],
["https://frontend.example.test/?preview=1", "without query or fragment"],
["https://user:secret@frontend.example.test/", "must not contain credentials"],
])("rejects %s", (url, reason) => {
expect(classifyLiveHostingBaseUrl(url)).toMatchObject({
@@ -193,6 +200,62 @@ describe("SPA fallback and worker script surfaces (1101/13-04)", () => {
expect(await failed({ ...RELEASE, "/__probe/deep-link": [200, INDEX] })).toEqual([]);
});
// 0929/13-06. Under a base path the surfaces are requested below it.
it("probes a release built for /app/ below /app/ and passes it", async () => {
const appIndex = '<script type="module" src="/app/assets/index-A1.js"></script>';
const requested: string[] = [];
const routes: Record<string, [number, string]> = {
"/app/": [200, appIndex],
"/app/config.json": [200, CONFIG],
"/app/release-manifest.json": [200, JSON.stringify({ schemaVersion: 2, ...IDENTITY })],
"/app/__probe/deep-link": [200, appIndex],
"/app/assets/index-A1.js": [200, ""],
};
const { results } = await probeLiveRelease(
new URL("https://frontend.example.test/app/"),
PATHS,
{ ...CANDIDATE, indexHtml: appIndex },
{
fetcher: async (url) => {
requested.push(url.pathname);
return host(routes)(url);
},
negotiateAlpn: async () => "h2",
alpnProtocols: ["h2"],
},
);
expect(requested).toEqual([
"/app/",
"/app/config.json",
"/app/release-manifest.json",
"/app/__probe/deep-link",
"/app/assets/index-A1.js",
]);
expect(results.filter((result) => !result.passed)).toEqual([]);
});
it("fails a HOSTING_BASE_URL path the candidate was not built for", async () => {
expect(
(await probeLiveRelease(new URL("https://frontend.example.test/app/"), PATHS, CANDIDATE, {
fetcher: host({}),
negotiateAlpn: async () => "h2",
alpnProtocols: ["h2"],
})).results.find((result) => result.header === "base-path"),
).toMatchObject({ expected: "/app/assets/", observed: "/assets/index-A1.js", passed: false });
// And the other way: a release built for /app/ probed at the root.
expect(
(await probeLiveRelease(base, PATHS, {
...CANDIDATE,
indexHtml: '<script type="module" src="/app/assets/index-A1.js"></script>',
}, {
fetcher: host({}),
negotiateAlpn: async () => "h2",
alpnProtocols: ["h2"],
})).results.find((result) => result.header === "base-path"),
).toMatchObject({ expected: "/assets/", observed: "/app/assets/index-A1.js", passed: false });
});
it("fails a host without the fallback", async () => {
expect(await failed(RELEASE)).toEqual(["appRoute http-status", "appRoute entry-script"]);
});
+74
View File
@@ -0,0 +1,74 @@
import { readFile } from "node:fs/promises";
import { describe, expect, it } from "vitest";
import { findRouteChunk } from "../../scripts/lib/build-manifest-outputs.ts";
// 0929/13-03. The route chunk was found by the chunk's `name`, so naming route
// chunks in vite.config.ts (the way tech-log did) stopped the build with
// `Missing built route chunk: APP_HOME` and no reason. It is found by the
// page's source file now, which is what the route runtime's moduleId names.
describe("route chunk lookup (0929/13-03)", () => {
const manifest = {
"index.html": { file: "assets/index-a.js", isEntry: true },
"src/presentation/pages/home-page.tsx": {
file: "assets/route-home-page-b.js",
name: "route-home-page",
src: "src/presentation/pages/home-page.tsx",
isDynamicEntry: true,
},
"src/features/order-feature/presentation/order-feature-list-page.tsx": {
file: "assets/order-feature-list-page-c.js",
name: "order-feature-list-page",
isDynamicEntry: true,
},
"_home-page-shared.js": { file: "assets/home-page-d.js", name: "home-page" },
};
it("finds the chunk by the page's source file even when the chunk has another name", () => {
expect(findRouteChunk(manifest, "APP_HOME", "home-page")).toEqual({ file: "assets/route-home-page-b.js" });
expect(findRouteChunk(manifest, "ORDER_FEATURE_LIST", "order-feature-list-page")).toEqual({
file: "assets/order-feature-list-page-c.js",
});
});
it("names the reason and the rule when no lazily imported page has that file name", () => {
const lookup = findRouteChunk(manifest, "NOT_FOUND", "not-found-page");
expect(lookup).toEqual({
error:
"Missing built route chunk: NOT_FOUND: no lazily imported module in the Vite manifest has the source " +
"file not-found-page.tsx (or .ts). A route runtime's moduleId must be the file name of the page it imports.",
});
});
it("refuses two lazily imported pages with the same file name", () => {
const lookup = findRouteChunk(
{ ...manifest, "src/other/home-page.tsx": { file: "assets/home-page-e.js", isDynamicEntry: true } },
"APP_HOME",
"home-page",
);
expect(lookup).toEqual({
error:
"Ambiguous built route chunk: APP_HOME: the lazily imported modules " +
"src/presentation/pages/home-page.tsx, src/other/home-page.tsx are all named home-page.",
});
});
it("names a route without a runtime contract", () => {
expect(findRouteChunk(manifest, "GHOST", undefined)).toEqual({
error: "Missing built route chunk: GHOST has no route runtime contract (ROUTE_RUNTIME_CONTRACT)",
});
});
it("is the one lookup the manifest writer and both release checks use", async () => {
for (const file of [
"scripts/generate-build-manifest.ts",
"scripts/verify-release.ts",
"scripts/lib/local-release-evidence.ts",
]) {
const source = await readFile(file, "utf8");
expect(source, file).toContain("findRouteChunk(");
expect(source, file).not.toMatch(/\.name === runtime\?\.moduleId/u);
}
});
});
+24
View File
@@ -1,3 +1,4 @@
import { spawnSync } from "node:child_process";
import { createServer, type Server } from "node:net";
import { afterEach, describe, expect, it, vi } from "vitest";
@@ -12,6 +13,7 @@ import {
testRunPath,
testRunRoot,
} from "../../scripts/lib/run-isolation.ts";
import { childProcessTimeout } from "../helpers/child-process-timeout.ts";
const servers: Server[] = [];
@@ -110,6 +112,28 @@ describe("build output root (RUN_OUTPUT_ROOT)", () => {
/^check-bundle: RUN_OUTPUT_ROOT must be/u,
);
});
// 1106/12-02, the part the gates did not cover. Run alone, these four read
// the checkout's dist/ or artifacts/ and did not refuse another root:
// `RUN_OUTPUT_ROOT=.tmp/x corepack pnpm verify:reproducible-build` wrote
// `passed: true` about the earlier build.
it.each([
["scripts/verify-reproducible-build.ts"],
["scripts/check-optional-recipes.ts", "--require-dist"],
["scripts/check-app-build-input.ts", "--require-bundle"],
["scripts/create-release-candidate.ts"],
])("%s refuses to read the checkout's build under another RUN_OUTPUT_ROOT", (script, ...args) => {
const { CI: _ci, ...environment } = process.env;
const result = spawnSync(process.execPath, [script, ...args], {
encoding: "utf8",
env: { ...environment, RUN_OUTPUT_ROOT: ".tmp/run-output-refusal" },
});
expect(result.status, result.stderr).toBe(1);
expect(result.stderr).toContain(
`${script} reads the build at this checkout's paths (dist/, artifacts/), but RUN_OUTPUT_ROOT=.tmp/run-output-refusal`,
);
}, childProcessTimeout(499));
});
describe("server ports", () => {
+105 -1
View File
@@ -4,7 +4,8 @@ import path from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import { runRunbookDrill } from "../../scripts/drill-runbook.ts";
import { runbookRecordArtifactSchema } from "../../scripts/contracts/release-artifacts.ts";
import { PROVIDER_EVIDENCE_MAX_AGE_MS, runRunbookDrill } from "../../scripts/drill-runbook.ts";
import { computeContractSetDigest } from "../../src/contracts/contract-set-canonical.ts";
import type { InstalledContractPackageIdentity } from "../../src/contracts/external-contract-runtime.ts";
@@ -145,3 +146,106 @@ describe("runbook drill verdicts", () => {
expect(source).not.toMatch(/verdict\(\s*true\s*\)/u);
});
});
// 0929/11-02. The drills with provider scope took only a runbook ID and a
// release root, so no input could ever make them PASS and the production
// stage could not close. Provider evidence now answers the items the local
// drill cannot observe, for this runbook and release only, and while recent.
describe("runbook drill with provider evidence (0929/11-02)", () => {
const NOW = new Date("2026-11-13T09:00:00.000Z");
const evidence = (overrides: Record<string, unknown> = {}) => ({
schemaVersion: 1,
artifactType: "provider-drill-evidence",
runbookId: "FE-RB-005",
releaseId: "release-a",
observedAt: "2026-11-13T08:30:00.000Z",
runId: "provider-run-41",
observations: [
{ item: "trigger", result: "PASS", evidence: "incident INC-7 declared the release-blocking defect" },
{ item: "핵심 smoke 시험", result: "PASS", evidence: "smoke against the prior release after the switch" },
{ item: "provider verification", result: "PASS", evidence: "provider serves release-a again" },
],
...overrides,
});
const drill = async (providerEvidence: unknown) =>
runRunbookDrill("FE-RB-005", { releaseRoot: await builtDist(), contractPackages, providerEvidence, now: NOW });
it("passes FE-RB-005 when the provider observed every item the local drill could not", async () => {
const record = await drill(evidence());
expect(record.result).toBe("PASS");
expect(record.triggerAsserted).toBe("PASS");
expect(record.recoveryAssertions.find(({ assertion }) => assertion === "핵심 smoke 시험")).toEqual({
assertion: "핵심 smoke 시험",
evidence: "provider evidence provider-run-41 at 2026-11-13T08:30:00.000Z: smoke against the prior release after the switch",
result: "PASS",
});
expect(record.providerEvidence).toEqual({
accepted: true,
runId: "provider-run-41",
observedAt: "2026-11-13T08:30:00.000Z",
providerVerification: "PASS",
});
const { gateId: _gateId, ...artifact } = record;
expect(runbookRecordArtifactSchema.safeParse(artifact).success).toBe(true);
});
it("fails when the provider observed a failure", async () => {
const observations = evidence().observations.map((entry) =>
entry.item === "핵심 smoke 시험" ? { ...entry, result: "FAIL" } : entry);
expect((await drill(evidence({ observations }))).result).toBe("FAIL");
});
it("stays UNVERIFIED while an item the local drill could not observe has no answer", async () => {
const record = await drill(evidence({ observations: evidence().observations.slice(0, 2) }));
expect(record.result).toBe("UNVERIFIED");
expect(record.providerEvidence?.providerVerification).toBe("UNVERIFIED");
});
it.each([
["another release", { releaseId: "release-b" }, "evidence is for release release-b, not release-a"],
["another runbook", { runbookId: "FE-RB-002" }, "evidence is for FE-RB-002, not FE-RB-005"],
[
"an expired observation",
{ observedAt: new Date(NOW.getTime() - PROVIDER_EVIDENCE_MAX_AGE_MS - 1_000).toISOString() },
"is older than 7 days",
],
["a future observation", { observedAt: "2026-11-13T10:00:00.000Z" }, "is in the future"],
[
"an answer for a check the local drill made",
{ observations: [...evidence().observations, { item: "호환성 gate", result: "PASS", evidence: "x" }] },
"evidence names items the local drill observed or does not have: 호환성 gate",
],
["a record that is not provider evidence", { artifactType: "runbook-record" }, "not provider drill evidence"],
])("does not pass on evidence for %s", async (_case, overrides, reason) => {
const record = await drill(evidence(overrides));
expect(record.result).toBe("UNVERIFIED");
expect(record.triggerAsserted).toBe("UNVERIFIED");
expect(record.providerEvidence?.accepted).toBe(false);
expect(record.providerEvidence?.reason).toContain(reason);
const { gateId: _gateId, ...artifact } = record;
expect(runbookRecordArtifactSchema.safeParse(artifact).success).toBe(true);
});
it("rejects a record that claims PASS without accepted provider evidence", () => {
const record = {
schemaVersion: 2,
runbookId: "FE-RB-005",
releaseId: "release-a",
drillTimestamp: NOW.toISOString(),
triggerInjected: "RELEASE_BLOCKING_DEFECT",
triggerAsserted: "PASS",
containmentAsserted: "PASS",
escalationPathAsserted: "PASS",
recoveryAssertions: [],
negativeFixtureFailedAsExpected: "PASS",
windowObservedBucket: "TBD",
providerVerificationRequired: true,
providerEvidence: { accepted: false, reason: "expired", providerVerification: "UNVERIFIED" },
result: "PASS",
};
expect(runbookRecordArtifactSchema.safeParse(record).success).toBe(false);
expect(runbookRecordArtifactSchema.safeParse({ ...record, result: "UNVERIFIED" }).success).toBe(true);
});
});
+112 -10
View File
@@ -9,33 +9,52 @@ import {
NAMESPACE_PROBE_ARGUMENTS,
type SystemTestHost,
} from "../../scripts/check-system-test-prerequisites.ts";
import {
systemdRunPrerequisiteProbeArguments,
systemdRunProviderArguments,
} from "../../scripts/lib/provider-cgroup.ts";
import { PROVIDER_SANDBOX_NAMESPACE_ARGUMENTS } from "../../scripts/lib/provider-sandbox-namespaces.ts";
type SpawnCall = Readonly<{ command: string; args: readonly string[] }>;
type SpawnResult = ReturnType<SystemTestHost["spawn"]>;
const RTM_NEWADDR = "bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted";
const PROBE_UNIT = "ca-provider-prerequisite-4242-0123456789abcdef01234567.scope";
/** A host that has every prerequisite, except what `bwrap` and `files` say. */
type HostOverrides = Readonly<{
files?: Readonly<Record<string, string>>;
missing?: readonly string[];
userManager?: SpawnResult;
systemdRun?: SpawnResult;
}>;
/** A host that has every prerequisite, except what `bwrap` and the overrides say. */
function fakeHost(
bwrap: SpawnResult,
files: Readonly<Record<string, string>> = {},
options: HostOverrides = {},
): { host: SystemTestHost; calls: SpawnCall[] } {
const calls: SpawnCall[] = [];
const host: SystemTestHost = {
access: async () => undefined,
access: async (target) => {
if (options.missing?.includes(target)) throw new Error(`EACCES: ${target}`);
},
readFile: async (target) => {
if (target === "/sys/fs/cgroup/cgroup.controllers") return "cpuset cpu io memory pids\n";
const content = files[target];
const content = options.files?.[target];
if (content === undefined) throw new Error(`ENOENT: ${target}`);
return content;
},
spawn: (command, args) => {
calls.push({ command, args });
if (command === "/usr/bin/systemctl") return { status: 0, stderr: "" };
if (command === "/usr/bin/systemctl" && args[0] === "--user") {
return options.userManager ?? { status: 0, stdout: "255\n", stderr: "" };
}
if (command === "/usr/bin/systemctl") return { status: 0, stdout: "", stderr: "" };
if (command === "/usr/bin/systemd-run") return options.systemdRun ?? { status: 0, stdout: "", stderr: "" };
if (command === "/usr/bin/bwrap") return bwrap;
throw new Error(`unexpected spawn: ${command}`);
},
probeUnitName: () => PROBE_UNIT,
};
return { host, calls };
}
@@ -74,7 +93,7 @@ describe("system test prerequisites (1105/12-01)", () => {
it("fails with the namespace restriction, bwrap's stderr and what to do when bwrap cannot set up loopback", async () => {
const { host } = fakeHost(
{ status: 1, stderr: `${RTM_NEWADDR}\n` },
{ "/proc/sys/kernel/apparmor_restrict_unprivileged_userns": "1\n" },
{ files: { "/proc/sys/kernel/apparmor_restrict_unprivileged_userns": "1\n" } },
);
const failures = await findSystemTestPrerequisiteFailures(host);
@@ -103,7 +122,7 @@ describe("system test prerequisites (1105/12-01)", () => {
expect(failures[0]).toContain("did not finish: spawnSync /usr/bin/bwrap ETIMEDOUT");
});
it("keeps the executable, cgroup v2 and systemd bus checks, and probes bwrap only when it is there", async () => {
it("keeps the executable, cgroup v2 and systemd checks, and probes bwrap only when it is there", async () => {
const calls: SpawnCall[] = [];
const host: SystemTestHost = {
access: async (target) => {
@@ -116,15 +135,98 @@ describe("system test prerequisites (1105/12-01)", () => {
},
};
expect(await findSystemTestPrerequisiteFailures(host)).toEqual([
const failures = await findSystemTestPrerequisiteFailures(host);
expect(failures.slice(0, 2)).toEqual([
"bubblewrap is required at /usr/bin/bwrap",
"cgroup v2 controllers are unavailable",
"a reachable systemd manager bus is required",
]);
expect(failures[2]).toMatch(/^a reachable systemd user manager is required/u);
expect(failures).toHaveLength(3);
expect(calls.map(({ command }) => command)).toEqual(["/usr/bin/systemctl"]);
});
});
// 1112/08-03. The check looked at the system manager bus while the provider
// asks the user manager for its version and starts a `systemd-run --user
// --scope`, so a host without a user manager, with an old one, or without
// prlimit, systemd-run or delegated controllers printed PASS and then failed
// each provider test with `provider cgroup unavailable: ...`.
describe("system test prerequisites match the provider's own calls (1112/08-03)", () => {
it("fails by name when the system manager answers but the user manager does not", async () => {
const { host, calls } = fakeHost(
{ status: 0, stderr: "" },
{ userManager: { status: 1, stdout: "", stderr: "Failed to connect to bus: No medium found\n" } },
);
const failures = await findSystemTestPrerequisiteFailures(host);
expect(failures).toHaveLength(1);
expect(failures[0]).toMatch(/^a reachable systemd user manager is required/u);
expect(failures[0]).toContain("/usr/bin/systemctl --user show --property=Version --value exited 1");
expect(failures[0]).toContain("systemctl stderr: Failed to connect to bus: No medium found");
expect(calls.some(({ command }) => command === "/usr/bin/systemd-run")).toBe(false);
});
it("fails when the user manager is older than the provider needs", async () => {
const { host } = fakeHost({ status: 0, stderr: "" }, { userManager: { status: 0, stdout: "253\n", stderr: "" } });
expect(await findSystemTestPrerequisiteFailures(host)).toEqual([
'systemd user manager 254 or newer is required (/usr/bin/systemctl --user show --property=Version --value reported "253")',
]);
});
it("requires the prlimit and systemd-run executables the provider runs", async () => {
const { host } = fakeHost({ status: 0, stderr: "" }, { missing: ["/usr/bin/prlimit", "/usr/bin/systemd-run"] });
expect(await findSystemTestPrerequisiteFailures(host)).toEqual([
"prlimit is required at /usr/bin/prlimit",
"systemd-run is required at /usr/bin/systemd-run",
]);
});
it("starts the provider's own scope once and names a scope that cannot enforce its limits", async () => {
const { host, calls } = fakeHost(
{ status: 0, stderr: "" },
{ systemdRun: { status: 125, stdout: "", stderr: "provider cgroup enforcement failed: cannot read memory.max\n" } },
);
const failures = await findSystemTestPrerequisiteFailures(host);
expect(failures).toHaveLength(1);
expect(failures[0]).toMatch(/^systemd-run could not start the provider's cgroup scope/u);
expect(failures[0]).toContain("exited 125");
expect(failures[0]).toContain("systemd-run stderr: provider cgroup enforcement failed: cannot read memory.max");
expect(failures[0]).toMatch(/memory, pids and cpu controllers delegated/u);
const scopes = calls.filter(({ command }) => command === "/usr/bin/systemd-run");
expect(scopes).toHaveLength(1);
expect(scopes[0]?.args).toEqual(systemdRunPrerequisiteProbeArguments(PROBE_UNIT));
expect(scopes[0]?.args.at(-1)).toBe("/bin/true");
});
it("probes with the provider scope's properties and enforcement gate, not a copy", () => {
const probe = systemdRunPrerequisiteProbeArguments(PROBE_UNIT);
const provider = systemdRunProviderArguments(
"ca-provider-vulnerability-4242-0123456789abcdef01234567.scope",
60_000,
60,
"/usr/bin/node",
"/opt/wrapper.ts",
"/tmp/report.json",
1,
1,
);
const options = (args: readonly string[]) =>
args.filter((arg) => arg.startsWith("--") && !arg.startsWith("--unit=") && !arg.startsWith("--property=RuntimeMaxSec="));
expect(options(probe)).toEqual(options(provider));
const gate = (args: readonly string[]) => args[args.indexOf("-c") + 1];
expect(gate(probe)).toBe(gate(provider));
expect(gate(probe)).toContain("require_cgroup_value memory.max");
expect(() => systemdRunPrerequisiteProbeArguments("ca-provider-vulnerability-1-0123456789abcdef01234567.scope"))
.toThrow(/unit is invalid/u);
});
});
// 1101/13-03. The release provider jobs run the same sandbox on whatever
// runner picks them up. The supervisor checked only that the executables
// exist, so a host that refuses the namespaces failed later with bwrap's raw
@@ -140,7 +242,7 @@ describe("provider sandbox namespace check (1101/13-03)", () => {
it("names the restriction, bwrap's stderr and the host settings on a host that refuses them", async () => {
const { host } = fakeHost(
{ status: 1, stderr: `${RTM_NEWADDR}\n` },
{ "/proc/sys/kernel/apparmor_restrict_unprivileged_userns": "1\n" },
{ files: { "/proc/sys/kernel/apparmor_restrict_unprivileged_userns": "1\n" } },
);
const failure = (await findProviderSandboxNamespaceFailure(host)) ?? "";
+42 -2
View File
@@ -1,9 +1,10 @@
import { spawnSync } from "node:child_process";
import { mkdtemp, rm, writeFile } from "node:fs/promises";
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { afterAll, describe, expect, it } from "vitest";
import { selectRepositoryPaths } from "../../scripts/lib/repository-layout.ts";
import { childProcessTimeout } from "../helpers/child-process-timeout.ts";
const FORBIDDEN = "tests/fixtures/test-evidence/forbidden";
@@ -13,7 +14,7 @@ afterAll(async () => {
await Promise.all(roots.map((root) => rm(root, { recursive: true, force: true })));
});
function checkSource(sourceRoot: string) {
function checkSource(sourceRoot: string, extra: readonly string[] = []) {
const result = spawnSync(
process.execPath,
[
@@ -23,6 +24,7 @@ function checkSource(sourceRoot: string) {
sourceRoot,
"--artifact",
"artifacts/quality/test-evidence-source-rules.json",
...extra,
],
{ encoding: "utf8" },
);
@@ -44,6 +46,44 @@ describe("test evidence source rules", () => {
expect(result.stderr).toContain("quarantine expired on 2020-01-01");
}, childProcessTimeout(739));
// 1112/03-03. The mask fixture had no test of its own: the directory run
// expects only the common prefix, so this rule could stop firing unseen.
it("rejects a screenshot mask over the whole UI", () => {
const result = checkSource(`${FORBIDDEN}/full-screen-mask.fixture`);
expect(result.status).toBe(1);
expect(result.stderr).toContain("screenshot mask may not cover the whole UI");
}, childProcessTimeout(141));
// 1112/03-03. The forbidOnly rule ran only in the full check, on the
// repository's own configs, so nothing showed it still rejects a config.
it("rejects a Playwright config without forbidOnly and accepts the repository's", async () => {
const root = await mkdtemp(path.join(tmpdir(), "test-evidence-config-"));
roots.push(root);
await writeFile(path.join(root, "clean.fixture"), "export {};\n");
const missing = checkSource(root, [
"--playwright-config",
"tests/fixtures/test-evidence/playwright-config/missing-forbid-only.fixture",
]);
expect(missing.status).toBe(1);
expect(missing.stderr).toContain(
"tests/fixtures/test-evidence/playwright-config/missing-forbid-only.fixture: missing forbidOnly: true",
);
const present = checkSource(root, ["--playwright-config", "playwright.config.ts"]);
expect(present.stderr).toBe("");
expect(present.status).toBe(0);
// Without the flag the rule reads the repository's configs, every one.
const repository = checkSource(root);
expect(repository.status, repository.stderr).toBe(0);
const report = JSON.parse(await readFile("artifacts/quality/test-evidence-source-rules.json", "utf8")) as {
facts: { playwrightConfigs?: string[] };
};
expect(report.facts.playwrightConfigs).toEqual([...selectRepositoryPaths("playwright-configs")]);
expect(report.facts.playwrightConfigs?.length).toBeGreaterThan(0);
}, childProcessTimeout(296));
it("rejects a skip without a quarantine", () => {
const result = checkSource(`${FORBIDDEN}/unowned-skip.fixture`);
expect(result.status).toBe(1);
+3 -1
View File
@@ -66,7 +66,9 @@ async function stripColours(
new Blob([bytes], { type: "image/png" }),
);
const canvas = new OffscreenCanvas(bitmap.width, bitmap.height);
const context = canvas.getContext("2d");
// Read back once per strip: without the hint Chromium warns on the
// console of this tab, which the strict fixture watches (1112/03-05).
const context = canvas.getContext("2d", { willReadFrequently: true });
if (context === null) throw new Error("no 2d canvas context");
context.drawImage(bitmap, 0, 0);
return offsets.map(({ x, width }) => {