Files
clean-architecture-frontend…/docs/testing/taxonomy.md
T
DongHyeonka 9655958bda fix(ci): test:system 사전 점검이 bwrap namespace를 실제로 만들어 본다
문제: FE-GATE-013의 provider sandbox는 bwrap을 --unshare-pid
--unshare-ipc --unshare-uts --unshare-net으로 띄운다. bwrap이 setuid가
아니므로 비특권 user namespace가 있어야 한다. 사전 점검
(scripts/check-system-test-prerequisites.ts)은 실행 파일·cgroup·systemd
bus만 보고 bwrap을 한 번도 실행하지 않는다. 그래서 이 제한이 있는 호스트
(Ubuntu 24.04, kernel.apparmor_restrict_unprivileged_userns=1)에서 PASS를
낸 뒤 provider 테스트 114건 중 18건이 "bwrap: loopback: Failed
RTM_NEWADDR: Operation not permitted" 또는 그 provider를 기다린 시간
초과로 실패한다. 이 조건은 runner 요구 목록, gate 정의, workflow 어디에도
없고 운영 원장의 일회성 메모에만 있다.

고른 방법:
- 사전 점검이 provider와 같은 namespace 옵션으로 bwrap에서 /bin/true를 한
  번 띄운다. 실패하면 "this host does not allow unprivileged user and
  network namespaces", bwrap stderr(최대 4줄), user namespace sysctl 값,
  고치는 방법 셋(AppArmor profile, sysctl, 다른 runner)을 적고 exit 1.
  기존 검사(실행 파일 3개, cgroup v2, systemd bus)는 그대로 둔다.
- 옵션 목록을 scripts/lib/provider-sandbox-namespaces.ts로 옮기고
  run-and-validate-provider.ts와 사전 점검이 같은 상수를 쓴다. 둘이
  갈라지면 점검이 다른 것을 시험하게 된다.
- 점검을 findSystemTestPrerequisiteFailures(host)로 내보내고 access·
  readFile·spawn을 주입받는다. CLI 동작(출력·exit 코드)은 같다.
- config/ci/gates.json의 FE-GATE-013에 hostRequirements 두 줄을 둔다.
  gate schema에 선택 필드를 더하고, generator가 그 gate를 도는 job의
  runs-on 바로 위에 "# FE-GATE-013 runner requirement: ..." 주석으로
  그린다. runner를 고르는 사람이 보는 자리다. 줄바꿈은 기존 nonEmpty
  규칙이 막으므로 주석 밖으로 나갈 수 없다. 이 필드는 gate의 명령·증거가
  아니라 runner 설명이라 기준선 gate shape 해시에는 넣지 않았다(기준선
  재기록 불필요).
- 요구 목록 세 곳(docs/operations/ci-quality-gates.md,
  docs/security/supply-chain.md, docs/testing/taxonomy.md)과 README merge
  게이트 절에 조건을 적었다. ci-quality-gates.md에 AppArmor profile 예시를
  두었다.

버린 대안:
- namespace가 없으면 provider 테스트를 skip: 거짓 초록이다.
- FE-GATE-013을 release 단계로 옮김: PR마다 sandbox 검사를 잃는다.
- probe를 systemd-run --user --scope 안에서 돌림: 리뷰 11-R10에서 scope
  안과 밖의 결과가 같았다. 직접 실행이 더 단순하다.
- workflow에 요구 사항을 runs-on label로 표현: runner label 체계가 아직
  없다. 주석이 지금 가능한 가장 작은 표시다.

위험·한계:
- probe는 bwrap에 --ro-bind / /만 준다. --size 같은 provider의 다른
  옵션 지원은 여기서 보지 않는다(기존과 같다). probe가 다른 이유로
  실패해도 namespace 문구가 나오지만 bwrap stderr를 같이 적는다.
- 호스트를 고칠지는 호스트 소유자가 정한다. 이 커밋은 조건을 적고 알리기만
  한다. 이 desktop에서 FE-GATE-013은 여전히 실패하고, 이제 첫 단계에서
  이유와 함께 실패한다.
- gate schema에 선택 필드 hostRequirements가 생겼다. 기존 gates.json과
  제품 overlay는 그대로 통과한다.

검증: desktop batch에서 확인
- node scripts/run-vitest.ts run tests/unit/system-test-prerequisites.test.ts tests/unit/ci-workflow-generation.test.ts
- corepack pnpm check:ci, check:types, lint, verify:documentation
- desktop에서 node scripts/check-system-test-prerequisites.ts가 exit 1과
  namespace 문구를 내는지, FE-GATE-013이 그 단계에서 멈추는지

Refs: 1105/12-01, 1105/11-02
2026-11-05 22:06:59 +09:00

127 lines
6.0 KiB
Markdown

# Test and evidence taxonomy
Each gate is blocking in its declared scope. Failures are not downgraded with
`continue-on-error` or warning-only scripts.
## Executable levels
| Level | Command | Ownership / prerequisite | Evidence |
| --- | --- | --- | --- |
| runtime schema | `pnpm test:runtime-schema` | pure runtime schema contracts | `artifacts/tests/runtime-schema.xml` |
| unit | `pnpm test:unit` | Node-only domain/application/pure policy/runtime units; no systemd/bwrap/cgroup prerequisite | `artifacts/tests/unit.xml` |
| capability contract | `pnpm test:contract` | reusable capability consumer contracts | `artifacts/tests/contract.xml` |
| component | `pnpm test:component` | React/hook/UI behavior | `artifacts/tests/component.xml` |
| integration | `pnpm test:integration` | HTTP/MSW, IndexedDB, composed browser-runtime boundaries, child-process/filesystem/IPC integration | `artifacts/tests/integration.xml` |
| system / CI runner | `pnpm test:system` | compatible Linux host with systemd, bubblewrap, cgroup v2 and CI-provider process controls, where bubblewrap may create unprivileged user and network namespaces | `artifacts/tests/system.xml` |
| end-to-end | `pnpm test:e2e` | pinned browser engines | `artifacts/tests/e2e/` |
| accessibility | `pnpm test:a11y` | pinned browser engines | `artifacts/tests/a11y.json` |
`test:all` is the normal product-development loop. It runs, in order,
`test:runtime-schema`, `test:unit`, `test:contract`, `test:component`,
`test:integration`, test:reference-feature (removed with the reference sample), `test:platform` (`tests/platform`
and `tests/architecture`), `check:release-coherence` and `test:recipes`
(`package.json` `test:all`). It does not include `test:system`. CI-runner and
supply-chain assurance has different host prerequisites and is invoked
explicitly in the assurance path.
## Deterministic test process
All Vitest package scripts launch through `scripts/run-vitest.ts`.
That runner:
1. rejects Node versions outside the repository-supported
`>=24.11.0 <25.0.0` range before the suite starts,
2. owns `NODE_ENV=test` rather than trusting the parent shell,
3. removes host-specific `npm_config_userconfig`, `npm_config_prefix` and
`npm_config_globalconfig` values before Vitest starts.
`vitest.config.ts` also fixes `NODE_ENV=test` so a direct Vitest invocation
cannot accidentally select React's production behavior.
To run one test file, use the same runner:
```sh
node scripts/run-vitest.ts run tests/component/router.test.tsx
```
The runner renders the gitignored `src/.generated/` product index first
(`scripts/run-vitest.ts`), so this works in a fresh clone. A direct
`pnpm exec vitest run <file>` skips that step and fails there with
`Failed to resolve import "../.generated/product-feature-index.ts"`.
`pnpm test:unit -- <file>` does not narrow the run, because the script already
passes the whole `tests/unit` directory.
`vitest.config.ts` collects only `tests/**/*.test.{ts,tsx}`; Playwright
`*.spec.ts` files are not Vitest tests. `tests/system/` is collected only when
the command names it, as `test:system` does, so a bare `vitest` run does not
start the CI-runner suite.
The system suite additionally runs
`scripts/check-system-test-prerequisites.ts` and fails with one prerequisite
report when the CI-runner host does not provide its required Linux facilities.
Besides the executables, cgroup v2 and the systemd bus, it runs bubblewrap once
with the provider sandbox's namespace options on `/bin/true`, so a host that
refuses unprivileged user namespaces (Ubuntu 23.10 and later with
`kernel.apparmor_restrict_unprivileged_userns=1`) fails there by name instead of
in the provider tests (1105/12-01, `docs/operations/ci-quality-gates.md`).
## Development paths
Product feature:
- focused feature/unit/component test
- capability contract test when a reusable boundary changes
- type/lint/architecture
- `test:all`
Reusable capability:
- focused unit tests
- capability contract tests
- integration tests
- type/lint/architecture
CI / release assurance:
- `test:system`
- supply-chain / promotion / release gates
A host-level process/sandbox test must not be placed in `tests/unit` merely
because it uses Vitest. The classification follows the system boundary and
prerequisites, not the test framework:
- Tests that need systemd, bubblewrap, cgroup v2 or CI-provider process
controls belong in `tests/system`.
- Tests of a long-lived child-process protocol belong in `tests/integration`;
for example the provider guardian transaction protocol lives at
`tests/integration/provider-guardian-transaction.test.ts`. Its handshake
timeout is a test watchdog, not a production protocol deadline.
- A script's own tests may start that script, Node or Vitest as a short-lived
child process to check its command-line contract, and stay in `tests/unit`
next to the script's other tests. Such a suite declares its own longer
timeout instead of raising the global one; for example
`tests/unit/risk-coverage.test.ts` uses `PROCESS_HEAVY_TIMEOUT_MS`.
End-to-end and automated accessibility scenarios run on the pinned Chromium,
Firefox, and WebKit engines. The responsive contract explicitly exercises
320px reflow, a mobile navigation drawer, and a wide two-column gallery.
Theme scenarios verify persistence, operating-system changes, and dark-surface
axe results. Native modal focus and validation association are exercised in
both component and browser tests.
A control is verified only when a positive fixture passes and its deliberately
failing negative fixture is rejected. Generated evidence is retained by CI;
the repository tracks only the evidence directory structure.
Promotion is an AND graph:
1. merge gates
2. merge gates plus release gates
3. release gates plus rollback/runbook drills
4. production promotion plus eligible field Web Vitals evidence
The [frontend platform testing strategy](./frontend-platform-testing-strategy.md)
contains the broader testing design. This file is the executable taxonomy for
where a test belongs and which environment is allowed to run it.