Files
clean-architecture-frontend…/tests/unit/documentation-readiness.test.ts
T
DongHyeonka b4e4f98c03 fix(docs): 문서 검사가 명령 안 경로와 짝 저장소 참조를 본다
문제
verify:documentation(FE-GATE-017)은 공백 없는 backtick 경로만 본다. 그래서
`corepack pnpm exec vitest run tests/…` 같은 명령 안의 경로, 셸 코드 블록의
명령, 저장소 밖(`../`) 링크는 확인하지 않았다. 실제로 원장
docs/operations/adapter-remediation-ledger.md의 N-06~N-08 행이 V2와 함께
지운 테스트 세 개를 실행하라고 적고 있었다. 짝 백엔드 정본 링크가 깨져도
알 수 없었다. config/ci/ungated-scripts.json의 check:remediation-ledger
사유는 "It passes at this commit"이라는 시점 문장을 담았다.

고른 방법
- 인라인 코드와 셸 코드 블록(```bash/sh/shell/console/zsh, 언어 없음)에서
  명령(corepack, pnpm, npx, node, vitest, playwright, tsc, eslint, bash,
  sh)의 인자를 같은 경로 규칙으로 본다. 앞의 NAME=value는 건너뛰고,
  --flag=value는 값만 본다. 자리표시자(<id>, glob, $VAR)는 기존 규칙대로
  경로로 보지 않는다.
- 짝 백엔드 템플릿(clean-architecture-backend-template)을 가리키는 backtick
  경로와 저장소 밖 링크는 문서를 읽는 checkout 옆에서 찾는다. 그 checkout이
  있으면 없는 경로는 실패다. 없으면(CI, 중첩 worktree) 실패 대신 "not
  verified"로 출력하고 보고서 documentReferences[].unverifiedLinks에 남긴다.
  보고서 schema에 그 칸을 더했다.
- 원장 N-06~N-08 행의 명령을 LEG-01 행처럼 "V2와 함께 지움"으로 바꾸고,
  ungated 사유에서 시점 문장을 지웠다. docs/README.md 1절의 검사 범위 설명을
  고쳤다.
- 버린 대안: 지운 심볼 이름 검사. 지운 이름 목록을 따로 관리해야 하고
  오탐이 많다(리뷰 5.2절). 1절 밖 문서로 대상 넓히기. 기록 문서가 일부러 옛
  경로를 적어 예외가 늘어난다.

위험·한계
- 명령 머리 목록에 없는 명령(git, curl 등)의 인자는 여전히 보지 않는다.
- 짝 저장소 참조는 CI에서 늘 "not verified"다. 실제 확인은 옆에 checkout이
  있는 개발 환경에서만 된다.
- 1절 밖 문서(testing strategy :765, VD-23)는 그대로 검사 밖이다.

검증
- RED: 새 테스트 4건(인라인 명령, 셸 블록, 짝 저장소 있음·깨짐, 짝 저장소
  없음 보고) 실패.
- 수정 뒤 documentation-readiness 19건 통과. 저장소에서 수정 전 원장 행 3개가
  "names paths that do not exist"로 잡혔고, 행을 고친 뒤 verify:documentation
  PASS_SCOPED(worktree라 짝 저장소 참조 3건은 not verified).
  check:remediation-ledger PASS, check:ci PASS, check:types 통과, eslint 통과.

Refs: docs/reviews/2026-11-03-maturity-review/12-guardrails-ci.md 12-05
2026-11-04 21:15:03 +09:00

234 lines
11 KiB
TypeScript

import { spawnSync } from "node:child_process";
import { cp, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
const INPUTS = ["README.md", "docs", "artifacts/tests/a11y-manual"];
/**
* Runs verify:documentation on a copy of its inputs, each `changes` entry
* applied to its document and every `removed` file deleted. Code facts (src,
* package.json, gates.json) are read from this repository, not the copy. The
* copy is a `frontend` folder; `siblings` are files written beside it, where a
* checkout of the paired repositories would be.
*/
async function verify(
changes: Readonly<Record<string, (text: string) => string>>,
removed: readonly string[] = [],
siblings: Readonly<Record<string, string>> = {},
) {
const parent = await mkdtemp(path.join(tmpdir(), "documentation-readiness-"));
const root = path.join(parent, "frontend");
try {
for (const [file, content] of Object.entries(siblings)) {
await mkdir(path.dirname(path.join(parent, file)), { recursive: true });
await writeFile(path.join(parent, file), content);
}
for (const input of INPUTS) await cp(input, path.join(root, input), { recursive: true });
for (const [document, mutate] of Object.entries(changes)) {
const target = path.join(root, document);
await writeFile(target, mutate(await readFile(target, "utf8")));
}
for (const file of removed) await rm(path.join(root, file));
const result = spawnSync(process.execPath, [path.resolve("scripts/verify-documentation-readiness.ts")], {
cwd: root,
encoding: "utf8",
});
const report = await readFile(path.join(root, "artifacts/quality/documentation-review.json"), "utf8")
.then((json) => JSON.parse(json) as { documentReferences: { path: string; unverifiedLinks?: string[] }[] })
.catch(() => null);
return { ...result, report };
} finally {
await rm(parent, { recursive: true, force: true });
}
}
function verifyWithReadme(mutate: (readme: string) => string, document = "README.md") {
return verify({ [document]: mutate });
}
// 1101/14-06. A route missing from a scope document used to fail with only
// "canonical scoped-review evidence is incomplete", which points at the
// diagram review instead of the file to fix.
describe("verify:documentation failure output (1101/14-06)", () => {
it("passes the documents as they are", async () => {
const result = await verifyWithReadme((readme) => readme);
expect(result.status, result.stderr).toBe(0);
});
it("names the route ID that has no manual review record", async () => {
const result = await verify({}, ["artifacts/tests/a11y-manual/EXAMPLES_PLATFORM.md"]);
expect(result.status).toBe(1);
expect(result.stderr).toContain("artifacts/tests/a11y-manual/ has no record for route IDs: EXAMPLES_PLATFORM");
expect(result.stderr).toMatch(/FAIL_UNVERIFIED \(failed: route scope\)/u);
expect(result.stderr).not.toContain("scoped-review evidence");
});
it("names every failed check", async () => {
const result = await verify(
{ "README.md": (readme) => readme.replaceAll("`RELEASE_TARGET`", "") },
["artifacts/tests/a11y-manual/EXAMPLES_PLATFORM.md"],
);
expect(result.stderr).toContain("README.md does not name: RELEASE_TARGET");
expect(result.stderr).toMatch(/\(failed: route scope, runtime variables\)/u);
});
});
// 1101/14-05. README and the manual checklist listed every route ID, so each
// new feature edited two template-owned documents. The scope is the record
// files now, and the current contract documents (docs/README.md §1) may only
// name paths, package scripts and gate IDs that exist.
describe("verify:documentation current contract documents (1101/14-05)", () => {
const ROUTE_IDS = /`(?:APP_HOME|EXAMPLES_[A-Z]+|NOT_FOUND|REFERENCE_RESOURCE_[A-Z]+)`/gu;
it("does not need README or the checklist to list route IDs", async () => {
const result = await verify({
"README.md": (readme) => readme.replaceAll(ROUTE_IDS, ""),
"docs/accessibility/manual-checklist.md": (checklist) => checklist.replaceAll(ROUTE_IDS, ""),
});
expect(result.status, result.stderr).toBe(0);
});
it.each([
["README.md", "scripts/verify-documentation-readiness.ts", "scripts/verify-documentation-readyness.ts"],
["docs/contracts/compatibility.md", "scripts/check-compatibility.ts", "scripts/check-compatibilty.ts"],
// reached through the `operations/` directory link
["docs/operations/release-cache-rollback.md", "scripts/lib/hosting-probe.ts", "scripts/lib/hosting-prob.ts"],
])("fails when %s names a path that does not exist", async (document, before, after) => {
const result = await verifyWithReadme((text) => text.replace(`\`${before}\``, `\`${after}\``), document);
expect(result.status).toBe(1);
expect(result.stderr).toContain(`${document} names paths that do not exist: ${after}`);
expect(result.stderr).toMatch(/\(failed: document references\)/u);
});
it("fails when docs/README.md links a document that does not exist", async () => {
const result = await verifyWithReadme(
(index) => index.replace("(architecture/layers.md)", "(architecture/layer.md)"),
"docs/README.md",
);
expect(result.status).toBe(1);
expect(result.stderr).toContain("docs/README.md names paths that do not exist: docs/architecture/layer.md");
});
it("fails when a document names a package script or a gate ID that does not exist", async () => {
const result = await verify({
"README.md": (readme) => readme.replace("`verify:documentation` fails", "`verify:documentaton` fails"),
"docs/README.md": (index) => index.replace("(FE-GATE-017)", "(FE-GATE-099)"),
});
expect(result.status).toBe(1);
expect(result.stderr).toContain("README.md names package scripts that do not exist: verify:documentaton");
expect(result.stderr).toContain("docs/README.md names gate IDs that config/ci/gates.json does not define: FE-GATE-099");
});
it("does not ask generated output to exist", async () => {
const result = await verifyWithReadme((readme) => `${readme}\nSee \`artifacts/quality/not-generated-yet.json\`.\n`);
expect(result.status, result.stderr).toBe(0);
});
it("accepts a path under history a copy deleted, and only that", async () => {
const naming = (readme: string) => `${readme}\nSee \`docs/retired-notes/plan.md\`.\n`;
const deleted = await verifyWithReadme((readme) =>
naming(readme).replace("| Path | Contents |\n| --- | --- |\n", "$&| `docs/retired-notes/` | deleted |\n"),
);
expect(deleted.status, deleted.stderr).toBe(0);
const missing = await verifyWithReadme(naming);
expect(missing.status).toBe(1);
expect(missing.stderr).toContain("README.md names paths that do not exist: docs/retired-notes/plan.md");
});
});
// 1103/12-05. Paths inside a command and links into the paired repositories
// were never looked up: a deleted V2 test stayed in a `vitest run` command,
// and a broken link to the backend template's contract passed.
describe("verify:documentation commands and sibling links (1103/12-05)", () => {
const CI_DOC = "docs/operations/ci-quality-gates.md";
const BACKEND = "clean-architecture-backend-template/docs/web/http-contract.md";
it.each([
["an inline command", "`corepack pnpm exec vitest run tests/integration/no-such-client.test.ts`"],
["a shell block", "```bash\nnode tests/integration/no-such-client.test.ts --flag\n```"],
])("fails when %s names a path that does not exist", async (_shape, line) => {
const result = await verify({ [CI_DOC]: (text) => `${text}\n${line}\n` });
expect(result.status).toBe(1);
expect(result.stderr).toContain(`${CI_DOC} names paths that do not exist: tests/integration/no-such-client.test.ts`);
});
it("looks up a command's existing arguments and leaves placeholders alone", async () => {
const result = await verify({
[CI_DOC]: (text) => `${text}\n\`corepack pnpm exec vitest run tests/unit/documentation-readiness.test.ts -t "x"\`\n` +
"```bash\ncorepack pnpm drill:runbook -- FE-RB-005 --release-root <release dir>\n```\n",
});
expect(result.status, result.stderr).toBe(0);
});
it("checks a link or a path into a paired repository checked out beside this one", async () => {
const naming = (target: string) => (text: string) =>
`${text}\n[contract](../../../${target}) and \`${target}\`\n`;
const present = await verify(
{ [CI_DOC]: naming(BACKEND) },
[],
{ [BACKEND]: "# contract\n" },
);
expect(present.status, present.stderr).toBe(0);
const broken = await verify(
{ [CI_DOC]: naming("clean-architecture-backend-template/docs/web/no-such.md") },
[],
{ [BACKEND]: "# contract\n" },
);
expect(broken.status).toBe(1);
expect(broken.stderr).toContain(
`${CI_DOC} names paths that do not exist: ../clean-architecture-backend-template/docs/web/no-such.md`,
);
});
it("reports a link into a paired repository that is not checked out as unverified", async () => {
const result = await verify({
[CI_DOC]: (text) => `${text}\n[contract](../../../clean-architecture-backend-template/docs/web/no-such.md)\n`,
});
expect(result.status, result.stderr).toBe(0);
expect(result.stdout).toContain("not verified");
expect(result.report?.documentReferences.find((entry) => entry.path === CI_DOC)?.unverifiedLinks)
.toEqual(["../clean-architecture-backend-template/docs/web/no-such.md"]);
});
});
// 1101/14-03. The entry documents drew a top-level domain layer that does not
// exist, left out three of the nine src folders, and layers.md named a
// minority of the architecture rules while claiming it could not drift.
describe("verify:documentation source structure (1101/14-03)", () => {
it("fails when layers.md loses a rule name, as renaming a rule does", async () => {
const result = await verifyWithReadme(
(layers) => layers.replaceAll("`platform-kernel-is-product-neutral`", "`platform-kernel`"),
"docs/architecture/layers.md",
);
expect(result.status).toBe(1);
expect(result.stderr).toContain(
"docs/architecture/layers.md does not name .dependency-cruiser.json rules: platform-kernel-is-product-neutral",
);
expect(result.stderr).toMatch(/\(failed: source structure\)/u);
});
it("fails when a document names a top-level src folder that does not exist", async () => {
const result = await verifyWithReadme((readme) =>
readme.replace("presentation -> application -> contracts", "presentation -> application -> `src/domain`"),
);
expect(result.status).toBe(1);
expect(result.stderr).toContain("README.md names src folders that do not exist: domain");
});
it("fails when a document describes a folder differently from layers.md", async () => {
const result = await verifyWithReadme(
(overview) => overview.replace("- `src/platform` — product-neutral", "- `src/platform` — shared"),
"docs/architecture/overview.md",
);
expect(result.status).toBe(1);
expect(result.stderr).toContain(
"docs/architecture/overview.md does not describe src folders as the layers.md table does: platform",
);
});
});