diff --git a/docs/accessibility/manual-checklist.md b/docs/accessibility/manual-checklist.md index 2e4158db..61425334 100644 --- a/docs/accessibility/manual-checklist.md +++ b/docs/accessibility/manual-checklist.md @@ -74,7 +74,10 @@ What the automated run scans: no alert box is on screen, so a list page is scanned in its empty state. A read of one resource (a detail page) has no generic answer: that route is scanned in its failure state and the run records an `a11y-data-state` - annotation. + annotation. `a11y.json` lists it under `notChecked`. When a protected + route's `paramsSchema` is the input of such a read, the report expects that + record from the descriptors alone and fails without it + (`scripts/write-a11y-report.ts`). - The reference sample's list in its success, empty and error states, its detail, its form with submit errors and its status page, each after that state is on screen (`tests/e2e/reference-form.spec.ts`, removed with the diff --git a/docs/manual/01-add-a-feature.md b/docs/manual/01-add-a-feature.md index b790a13c..d8ff2cbf 100644 --- a/docs/manual/01-add-a-feature.md +++ b/docs/manual/01-add-a-feature.md @@ -432,6 +432,7 @@ projectRequest: (input) => Object.freeze({ - 이 spec은 등록된 모든 라우트를 열어 axe로 검사한다. 경로 파라미터는 이름과 상관없이 모두 `a11y-1`로 바꾼다. 파라미터 스키마가 이 값을 거부하면 기능 페이지 대신 잘못된 경로 화면을 검사하므로, id 스키마는 짧은 영문·숫자 문자열을 받게 둔다. - API 요청은 설치된 descriptor에서 method와 `pathTemplate`으로 찾아 답한다. 목록 조회에는 빈 목록을 그 operation의 모양(`responseEnvelope: "ENVELOPE_V1"`이면 Envelope)으로 답한다. 그 뒤 경고 상자(`role="alert"`)가 없는지 본다. 그래서 목록 페이지는 빈 상태로 검사된다. - 한 자원을 읽는 조회(경로 파라미터가 있고 빈 목록을 받지 않는 GET)에는 빈 목록 말고 답할 것이 없다. 그래서 상세 페이지는 실패 상태로 그려지고, 테스트 결과에 `a11y-data-state` annotation이 남는다. + - `a11y.json`은 그 라우트를 통과(`scope`)가 아니라 `notChecked`에 적는다. report(`scripts/write-a11y-report.ts`)는 라우트의 `paramsSchema`가 그런 GET의 입력 스키마이면 descriptor만 보고 그 라우트를 `notChecked`로 기대한다. spec이 기록을 남기지 않으면 `a11y.json not written:`으로 실패한다(1106/11-01). 생성 명령의 상세 라우트는 GET과 같은 `Params`를 쓰므로 이 규칙에 든다. - 설치된 GET이 아닌 요청을 그리기 전에 보내면 spec이 실패한다(`requests no installed list or single-resource read answers`). - spec은 h1이 보일 때까지 기다린다. 그래서 상세 페이지는 데이터가 없을 때도 h1을 그린다. 예: `heading={{ title: notice?.title ?? message("notice.detail.titlePending") }}`처럼 데이터가 없으면 고정 제목("공지")을 쓴다. - 기능이 앱 홈(`/`)을 대신하려면 route ID `APP_HOME`을 쓴다. `docs/architecture/starter-experience.md`의 "Product home and layouts" 절을 본다. diff --git a/docs/manual/reference/errors.md b/docs/manual/reference/errors.md index 910fa46c..90f2714b 100644 --- a/docs/manual/reference/errors.md +++ b/docs/manual/reference/errors.md @@ -178,6 +178,7 @@ lint / check:architecture / check:i18n → FE-GATE-002, FE-GATE-010 |---|---|---| | `artifacts/tests/a11y-manual/ has no record for route IDs: ` (`verify:documentation`) | 라우트의 a11y 기록 파일이 없다 | 기존 pending 기록을 복사해 제목과 `Route ID`를 바꾼다 | | `Manual accessibility record format is invalid:` 다음 `artifacts/tests/a11y-manual/.md: missing record` (FE-GATE-009) | 같은 원인 | 같은 곳 | +| `a11y.json not written:` 다음 `a11y: reads one resource () but has no failure-state record in artifacts/tests/a11y-data-state` (`test:a11y`, FE-GATE-009) | 라우트의 `paramsSchema`가 한 자원을 읽는 GET의 입력인데, 접근성 spec이 그 라우트를 실패 화면으로만 봤다는 기록을 남기지 않았다(1106/11-01). 그대로면 `a11y.json`이 그 라우트를 통과로 센다 | `tests/e2e/accessibility.spec.ts`의 기록 호출(`recordA11yDataState`). 기능 파일은 고치지 않는다 | | `Manual accessibility evidence is incomplete:` 다음 `: ` (FE-GATE-028) | 서명하지 않은 기록 | [03 문서](../03-outside-feature-folder.md#수동-접근성-검토-서명) | | `release IDs do not match` (FE-GATE-028) | 라우트마다 다른 릴리스로 검토했다 | 같은 릴리스 후보로 다시 검토한다 | | ` names paths that do not exist: ` (`verify:documentation`) | 문서가 없는 경로를 적었다 | 그 문서 | diff --git a/scripts/lib/a11y-data-state.ts b/scripts/lib/a11y-data-state.ts index 81dfd740..e3d8a2fa 100644 --- a/scripts/lib/a11y-data-state.ts +++ b/scripts/lib/a11y-data-state.ts @@ -14,6 +14,12 @@ import path from "node:path"; * only the failure state, no record otherwise. A route's latest run therefore * decides. `scripts/write-a11y-report.ts` reads the records and reports those * routes under `notChecked` instead of in the passed `scope`. + * + * 1106/11-01. No record used to mean "checked", and nothing but the spec said + * which routes needed one: a record call changed to `null` (review 11-R2) put + * the detail page back in `scope` and every check passed. The report now + * works the same routes out from the installed descriptors + * (`oneResourceReadRoutes`) and fails when one has no record. */ export const A11Y_DATA_STATE_DIRECTORY = "artifacts/tests/a11y-data-state"; @@ -74,6 +80,88 @@ export async function readA11yNotCheckedRoutes( return Object.freeze(notChecked); } +export type OneResourceRoute = Readonly<{ + routeId: string; + operationIds: readonly string[]; +}>; + +type RouteFacts = Readonly<{ + routeId: string; + access: string; + paramsSchema: string | null; +}>; + +type ReadFacts = Readonly<{ + contract: Readonly<{ + operationId: string; + method: string; + pathTemplate: string; + inputValidator: Readonly<{ schemaId: string }>; + outputValidator: Readonly<{ + safeParse(value: unknown): Readonly<{ success: boolean }>; + }>; + }>; +}>; + +/** + * 1106/11-01. The installed routes the spec can scan only on their failure + * state, from the descriptors alone: a protected route whose `paramsSchema` is + * the input of a GET that reads one resource. A GET reads one resource by the + * rule `tests/e2e/support/installed-api-answers.ts` answers with: it has a + * path parameter and its output rejects an empty list, so no generic body + * answers it. The spec signs in only on protected routes, so a public route is + * not listed. + * + * This finds the detail pages that take their params schema from their read + * (the reference sample and every `create:feature --with-create` slice). A + * detail page whose params schema differs is not found, and its record alone + * decides, as before. + */ +export function oneResourceReadRoutes( + routes: Iterable, + operations: Iterable, +): readonly OneResourceRoute[] { + const readsByInput = new Map(); + for (const { contract } of operations) { + if (contract.method !== "GET" || !contract.pathTemplate.includes("{")) continue; + if (contract.outputValidator.safeParse([]).success) continue; + const reads = readsByInput.get(contract.inputValidator.schemaId) ?? []; + reads.push(contract.operationId); + readsByInput.set(contract.inputValidator.schemaId, reads); + } + const found: OneResourceRoute[] = []; + for (const route of routes) { + if (route.access !== "session-required" || route.paramsSchema === null) continue; + const reads = readsByInput.get(route.paramsSchema); + if (reads === undefined) continue; + found.push( + Object.freeze({ routeId: route.routeId, operationIds: Object.freeze([...reads]) }), + ); + } + return Object.freeze(found); +} + +/** + * 1106/11-01. One line for each route in `expected` that `notChecked` leaves + * out, which `a11y.json` would count as passed. Empty when every route that + * reads one resource has its failure-state record. A record for a route not in + * `expected` claims less and is not refused. + */ +export function oneResourceRoutesCountedAsChecked( + expected: readonly OneResourceRoute[], + notChecked: readonly A11yNotCheckedRoute[], +): readonly string[] { + const recorded = new Set(notChecked.map(({ routeId }) => routeId)); + return Object.freeze( + expected + .filter(({ routeId }) => !recorded.has(routeId)) + .map( + ({ routeId, operationIds }) => + `${routeId} reads one resource (${operationIds.join(", ")}) but has no failure-state record in ${A11Y_DATA_STATE_DIRECTORY}; tests/e2e/accessibility.spec.ts must record why it scanned the route on its failure state only`, + ), + ); +} + /** * The `artifacts/tests/a11y.json` value. `automatedStatus: "passed"` speaks * for `scope` only: the installed routes axe scanned in their own state. diff --git a/scripts/write-a11y-report.ts b/scripts/write-a11y-report.ts index da9b06e2..8e51ff94 100644 --- a/scripts/write-a11y-report.ts +++ b/scripts/write-a11y-report.ts @@ -1,8 +1,12 @@ import { mkdir } from "node:fs/promises"; +import { ROUTE_REGISTRY } from "../src/features/installed-feature-contracts.ts"; +import { COMPOSED_CONTRACT_CONTRIBUTIONS } from "../src/features/loaded-feature-runtimes.ts"; import { automatedA11yArtifactSchema } from "./contracts/release-artifacts.ts"; import { automatedA11yReport, + oneResourceReadRoutes, + oneResourceRoutesCountedAsChecked, readA11yNotCheckedRoutes, } from "./lib/a11y-data-state.ts"; import { MANUAL_A11Y_ROUTE_IDS } from "./lib/manual-a11y-evidence.ts"; @@ -12,6 +16,22 @@ await mkdir("artifacts/tests", { recursive: true }); // 1103/11-01. Routes the spec scanned only on their failure state are not // reported as passed. const notChecked = await readA11yNotCheckedRoutes(MANUAL_A11Y_ROUTE_IDS); +// 1106/11-01. The records come from the spec alone. The descriptors say which +// routes need one, so a record call the spec lost fails here instead of +// counting a detail page as passed. +const countedAsChecked = oneResourceRoutesCountedAsChecked( + oneResourceReadRoutes( + Object.values(ROUTE_REGISTRY), + COMPOSED_CONTRACT_CONTRIBUTIONS.httpByOperationId.values(), + ), + notChecked, +); +if (countedAsChecked.length > 0) { + process.stderr.write( + `a11y.json not written:\n${countedAsChecked.map((line) => `a11y: ${line}`).join("\n")}\n`, + ); + process.exit(1); +} await writeValidatedJsonArtifact({ path: "artifacts/tests/a11y.json", schema: automatedA11yArtifactSchema, diff --git a/tests/e2e/accessibility.spec.ts b/tests/e2e/accessibility.spec.ts index e8264ff4..d4c797b1 100644 --- a/tests/e2e/accessibility.spec.ts +++ b/tests/e2e/accessibility.spec.ts @@ -40,6 +40,10 @@ for (const definition of ROUTES) { test(`@a11y ${route} has no critical or serious axe violations`, async ({ page, }) => { + // 1106/11-01. An earlier run's record never outlives this run, even when + // no record call below is reached. scripts/write-a11y-report.ts then + // fails on a route the descriptors say reads one resource. + await recordA11yDataState(definition.routeId, null); const heading = page.getByRole("heading", { level: 1 }); if (definition.access === "session-required") { // 1103/11-01. Feature data is not this spec's subject: a list read gets diff --git a/tests/features/reference-feature/reference-contract.test.ts b/tests/features/reference-feature/reference-contract.test.ts index 4f632f12..c388871f 100644 --- a/tests/features/reference-feature/reference-contract.test.ts +++ b/tests/features/reference-feature/reference-contract.test.ts @@ -23,6 +23,7 @@ import { REFERENCE_FEATURE_ROUTE_CODECS } from "../../../src/features/reference- import type { ReferenceFeatureInput } from "../../../src/features/reference-feature/application/reference-feature-api.ts"; import { createTestApplication } from "../../helpers/create-test-application.ts"; import { createReferenceFeatureInstalledInput } from "../../../src/features/reference-feature/adapters/create-reference-feature-input.ts"; +import { oneResourceReadRoutes } from "../../../scripts/lib/a11y-data-state.ts"; describe("reference feature boundary contracts", () => { it("propagates uncertain command effect certainty into the application failure", async () => { @@ -138,6 +139,23 @@ describe("reference feature boundary contracts", () => { ); }); + // 1106/11-01. scripts/write-a11y-report.ts fails when a11y.json would count + // these routes as passed: the accessibility spec has no generic answer for a + // read of one resource and scans the page on its failure state only. + it("has one detail page the a11y report expects under notChecked", () => { + expect( + oneResourceReadRoutes( + Object.values(REFERENCE_FEATURE_CONTRACT.routes), + REFERENCE_FEATURE_TEMPLATE_CONTRIBUTION.http, + ), + ).toEqual([ + { + routeId: "REFERENCE_RESOURCE_DETAIL", + operationIds: ["GET_REFERENCE_RESOURCE"], + }, + ]); + }); + it("contributes an identity-based invalidation graph and an explicit wire version", () => { expect(REFERENCE_FEATURE_CONTRACT.invalidation).toEqual({ topics: [REFERENCE_RESOURCE_INVALIDATION_TOPIC], diff --git a/tests/unit/a11y-report.test.ts b/tests/unit/a11y-report.test.ts index 330b46a9..bf23e4f7 100644 --- a/tests/unit/a11y-report.test.ts +++ b/tests/unit/a11y-report.test.ts @@ -7,6 +7,8 @@ import { afterEach, describe, expect, it } from "vitest"; import { automatedA11yArtifactSchema } from "../../scripts/contracts/release-artifacts.ts"; import { automatedA11yReport, + oneResourceReadRoutes, + oneResourceRoutesCountedAsChecked, readA11yNotCheckedRoutes, recordA11yDataState, } from "../../scripts/lib/a11y-data-state.ts"; @@ -99,3 +101,97 @@ describe("automated accessibility report", () => { ).toBe(false); }); }); + +/** + * 1106/11-01. Which routes need a failure-state record comes from the + * descriptors, not from the spec that writes the records. Review 11-R2 passed + * `null` for the detail page's record and every check passed. + */ +describe("routes the accessibility spec can scan only on their failure state", () => { + const read = ( + operationId: string, + method: string, + pathTemplate: string, + inputSchemaId: string, + acceptsEmptyList: boolean, + ) => ({ + contract: { + operationId, + method, + pathTemplate, + inputValidator: { schemaId: inputSchemaId }, + outputValidator: { + safeParse: (value: unknown) => ({ + success: acceptsEmptyList && Array.isArray(value), + }), + }, + }, + }); + const route = ( + routeId: string, + access: string, + paramsSchema: string | null, + ) => ({ routeId, access, paramsSchema }); + + it("finds a protected route whose params are the input of a read of one resource", () => { + expect( + oneResourceReadRoutes( + [ + route("THING_LIST", "session-required", null), + route("THING_DETAIL", "session-required", "ThingParams"), + route("THING_PARTS", "session-required", "ThingPartsParams"), + route("THING_EDIT", "session-required", "ThingEditParams"), + route("PUBLIC_THING", "public", "ThingParams"), + ], + [ + read("LIST_THINGS", "GET", "/api/things", "ThingListQuery", true), + read("GET_THING", "GET", "/api/things/{thingId}", "ThingParams", false), + // A list under a path parameter gets the empty list. + read("LIST_THING_PARTS", "GET", "/api/things/{thingId}/parts", "ThingPartsParams", true), + // Not a read. + read("DELETE_THING", "DELETE", "/api/things/{thingId}", "ThingEditParams", false), + ], + ), + ).toEqual([{ routeId: "THING_DETAIL", operationIds: ["GET_THING"] }]); + }); + + it("refuses records that count such a route as checked", async () => { + const directory = await recordDirectory(); + const expected = [ + { routeId: FAILURE_STATE_ROUTE, operationIds: ["TEST_GET_ENTITY"] }, + ]; + + // 11-R2: the spec recorded the route as scanned in its own state. + await recordA11yDataState(FAILURE_STATE_ROUTE, null, directory); + expect( + oneResourceRoutesCountedAsChecked( + expected, + await readA11yNotCheckedRoutes(ROUTES, directory), + ), + ).toEqual([ + expect.stringContaining( + `${FAILURE_STATE_ROUTE} reads one resource (TEST_GET_ENTITY) but has no failure-state record`, + ), + ]); + + await recordA11yDataState(FAILURE_STATE_ROUTE, REASON, directory); + expect( + oneResourceRoutesCountedAsChecked( + expected, + await readA11yNotCheckedRoutes(ROUTES, directory), + ), + ).toEqual([]); + }); + + it("does not refuse a record for a route the descriptors do not name", async () => { + const directory = await recordDirectory(); + await recordA11yDataState(FAILURE_STATE_ROUTE, REASON, directory); + + expect( + oneResourceRoutesCountedAsChecked( + [], + await readA11yNotCheckedRoutes(ROUTES, directory), + ), + ).toEqual([]); + }); +});