6.2 KiB
Test and evidence taxonomy
Each gate is blocking in its declared scope. Failures are not downgraded with
continue-on-error or warning-only scripts.
Executable levels
| Level | Command | Ownership / prerequisite | Evidence |
|---|---|---|---|
| runtime schema | pnpm test:runtime-schema |
pure runtime schema contracts | artifacts/tests/runtime-schema.xml |
| unit | pnpm test:unit |
Node-only domain/application/pure policy/runtime units; no systemd/bwrap/cgroup prerequisite | artifacts/tests/unit.xml |
| capability contract | pnpm test:contract |
reusable capability consumer contracts | artifacts/tests/contract.xml |
| component | pnpm test:component |
React/hook/UI behavior | artifacts/tests/component.xml |
| integration | pnpm test:integration |
HTTP/MSW, IndexedDB, composed browser-runtime boundaries, child-process/filesystem/IPC integration | artifacts/tests/integration.xml |
| system / CI runner | pnpm test:system |
compatible Linux host with systemd, bubblewrap, cgroup v2 and CI-provider process controls, where bubblewrap may create unprivileged user and network namespaces | artifacts/tests/system.xml |
| end-to-end | pnpm test:e2e |
pinned browser engines | artifacts/tests/e2e/ |
| accessibility | pnpm test:a11y |
pinned browser engines | artifacts/tests/a11y.json |
test:all is the normal product-development loop. It runs, in order,
test:runtime-schema, test:unit, test:contract, test:component,
test:integration, the rest of what test:coverage runs (tests/features,
the reference sample's tests among them, tests/scripts and
tests/scale/browser-soak.contract.test.ts), test:platform (tests/platform
and tests/architecture), check:release-coherence and test:recipes
(package.json test:all). So every Vitest path the merge gate's coverage
run covers also runs here (tests/unit/test-all-paths.test.ts, 0929/10-05).
It does not include test:system. CI-runner and
supply-chain assurance has different host prerequisites and is invoked
explicitly in the assurance path.
Deterministic test process
All Vitest package scripts launch through scripts/run-vitest.ts.
That runner:
- rejects Node versions outside the repository-supported
>=24.11.0 <25.0.0range before the suite starts, - owns
NODE_ENV=testrather than trusting the parent shell, - removes host-specific
npm_config_userconfig,npm_config_prefixandnpm_config_globalconfigvalues before Vitest starts.
vitest.config.ts also fixes NODE_ENV=test so a direct Vitest invocation
cannot accidentally select React's production behavior.
To run one test file, use the same runner:
node scripts/run-vitest.ts run tests/component/router.test.tsx
The runner renders the gitignored src/.generated/ product index first
(scripts/run-vitest.ts), so this works in a fresh clone. A direct
pnpm exec vitest run <file> skips that step and fails there with
Failed to resolve import "../.generated/product-feature-index.ts".
pnpm test:unit -- <file> does not narrow the run, because the script already
passes the whole tests/unit directory.
vitest.config.ts collects only tests/**/*.test.{ts,tsx}; Playwright
*.spec.ts files are not Vitest tests. tests/system/ is collected only when
the command names it, as test:system does, so a bare vitest run does not
start the CI-runner suite.
The system suite additionally runs
scripts/check-system-test-prerequisites.ts and fails with one prerequisite
report when the CI-runner host does not provide its required Linux facilities.
Besides the executables, cgroup v2 and the systemd bus, it runs bubblewrap once
with the provider sandbox's namespace options on /bin/true, so a host that
refuses unprivileged user namespaces (Ubuntu 23.10 and later with
kernel.apparmor_restrict_unprivileged_userns=1) fails there by name instead of
in the provider tests (1105/12-01, docs/operations/ci-quality-gates.md).
Development paths
Product feature:
- focused feature/unit/component test
- capability contract test when a reusable boundary changes
- type/lint/architecture
test:all
Reusable capability:
- focused unit tests
- capability contract tests
- integration tests
- type/lint/architecture
CI / release assurance:
test:system- supply-chain / promotion / release gates
A host-level process/sandbox test must not be placed in tests/unit merely
because it uses Vitest. The classification follows the system boundary and
prerequisites, not the test framework:
- Tests that need systemd, bubblewrap, cgroup v2 or CI-provider process
controls belong in
tests/system. - Tests of a long-lived child-process protocol belong in
tests/integration; for example the provider guardian transaction protocol lives attests/integration/provider-guardian-transaction.test.ts. Its handshake timeout is a test watchdog, not a production protocol deadline. - A script's own tests may start that script, Node or Vitest as a short-lived
child process to check its command-line contract, and stay in
tests/unitnext to the script's other tests. Such a suite declares its own longer timeout instead of raising the global one; for exampletests/unit/risk-coverage.test.tsusesPROCESS_HEAVY_TIMEOUT_MS.
End-to-end and automated accessibility scenarios run on the pinned Chromium, Firefox, and WebKit engines. The responsive contract explicitly exercises 320px reflow, a mobile navigation drawer, and a wide two-column gallery. Theme scenarios verify persistence, operating-system changes, and dark-surface axe results. Native modal focus and validation association are exercised in both component and browser tests.
A control is verified only when a positive fixture passes and its deliberately failing negative fixture is rejected. Generated evidence is retained by CI; the repository tracks only the evidence directory structure.
Promotion is an AND graph:
- merge gates
- merge gates plus release gates
- release gates plus rollback/runbook drills
- production promotion plus eligible field Web Vitals evidence
The frontend platform testing strategy contains the broader testing design. This file is the executable taxonomy for where a test belongs and which environment is allowed to run it.